[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:09:52.398736 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:43031
I20260709 12:09:52.401113 11310 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:09:52.402791 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:09:52.427726 11316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:52.427968 11319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:52.429224 11317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:09:52.431020 11310 server_base.cc:1061] running on GCE node
I20260709 12:09:53.662577 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:09:53.662930 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:09:53.663172 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783598993663145 us; error 0 us; skew 500 ppm
I20260709 12:09:53.664155 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:09:53.671550 11310 webserver.cc:533] Webserver started at http://127.11.11.190:41717/ using document root <none> and password file <none>
I20260709 12:09:53.672712 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:09:53.673022 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:09:53.673605 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:09:53.681205 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "2707573fdb8c4ccfa794bb864d6ad3b8"
format_stamp: "Formatted at 2026-07-09 12:09:53 on dist-test-slave-4qvp"
I20260709 12:09:53.691690 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:09:53.699450 11325 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:53.701370 11310 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:09:53.702025 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "2707573fdb8c4ccfa794bb864d6ad3b8"
format_stamp: "Formatted at 2026-07-09 12:09:53 on dist-test-slave-4qvp"
I20260709 12:09:53.702555 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:09:53.767722 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:09:53.769649 11310 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:09:53.770191 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:09:53.860630 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:43031
I20260709 12:09:53.860728 11376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:43031 every 8 connection(s)
I20260709 12:09:53.870383 11377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:53.894491 11377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8: Bootstrap starting.
I20260709 12:09:53.900957 11377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:53.903106 11377 log.cc:826] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8: Log is configured to *not* fsync() on all Append() calls
I20260709 12:09:53.909168 11377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8: No bootstrap required, opened a new log
I20260709 12:09:53.933853 11377 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" member_type: VOTER }
I20260709 12:09:53.934530 11377 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:53.935075 11377 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2707573fdb8c4ccfa794bb864d6ad3b8, State: Initialized, Role: FOLLOWER
I20260709 12:09:53.936333 11377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" member_type: VOTER }
I20260709 12:09:53.936949 11377 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:09:53.937256 11377 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:09:53.937647 11377 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:53.942925 11377 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" member_type: VOTER }
I20260709 12:09:53.943843 11377 leader_election.cc:304] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2707573fdb8c4ccfa794bb864d6ad3b8; no voters: 
I20260709 12:09:53.945943 11377 leader_election.cc:290] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:09:53.946511 11380 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:53.949676 11380 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2707573fdb8c4ccfa794bb864d6ad3b8, State: Running, Role: LEADER
I20260709 12:09:53.950817 11380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" member_type: VOTER }
I20260709 12:09:53.952451 11377 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:09:53.963876 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2707573fdb8c4ccfa794bb864d6ad3b8. Latest consensus state: current_term: 1 leader_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" member_type: VOTER } }
I20260709 12:09:53.964882 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [sys.catalog]: This master's current role is: LEADER
I20260709 12:09:53.966243 11381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2707573fdb8c4ccfa794bb864d6ad3b8" member_type: VOTER } }
I20260709 12:09:53.967010 11381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [sys.catalog]: This master's current role is: LEADER
I20260709 12:09:53.971369 11390 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:09:53.984153 11390 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:09:53.986994 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:09:54.005297 11390 catalog_manager.cc:1383] Generated new cluster ID: 91a6128b522f455abd24ad47efba6c8e
I20260709 12:09:54.005704 11390 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:09:54.041450 11390 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:09:54.043582 11390 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:09:54.068310 11390 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8: Generated new TSK 0
I20260709 12:09:54.069744 11390 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:09:54.119359 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:09:54.131124 11399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:54.132601 11400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:54.138921 11402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:09:54.138994 11310 server_base.cc:1061] running on GCE node
I20260709 12:09:54.140439 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:09:54.140767 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:09:54.141043 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783598994141021 us; error 0 us; skew 500 ppm
I20260709 12:09:54.142047 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:09:54.147641 11310 webserver.cc:533] Webserver started at http://127.11.11.129:40909/ using document root <none> and password file <none>
I20260709 12:09:54.148476 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:09:54.148757 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:09:54.149129 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:09:54.150859 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "edea0c9ac6854273961ab944d9dd4664"
format_stamp: "Formatted at 2026-07-09 12:09:54 on dist-test-slave-4qvp"
I20260709 12:09:54.158946 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:09:54.165179 11407 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.166556 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20260709 12:09:54.166992 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "edea0c9ac6854273961ab944d9dd4664"
format_stamp: "Formatted at 2026-07-09 12:09:54 on dist-test-slave-4qvp"
I20260709 12:09:54.167407 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:09:54.208765 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:09:54.210752 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:09:54.213462 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:09:54.219628 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:09:54.220001 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.220345 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:09:54.220599 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.325753 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:38679
I20260709 12:09:54.325927 11469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:38679 every 8 connection(s)
I20260709 12:09:54.334473 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:09:54.353037 11474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:54.356321 11475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:54.360769 11477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:09:54.361766 11310 server_base.cc:1061] running on GCE node
I20260709 12:09:54.362859 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:09:54.363142 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:09:54.363354 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783598994363338 us; error 0 us; skew 500 ppm
I20260709 12:09:54.364198 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:09:54.368873 11470 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43031
I20260709 12:09:54.369458 11470 heartbeater.cc:461] Registering TS with master...
I20260709 12:09:54.370718 11470 heartbeater.cc:507] Master 127.11.11.190:43031 requested a full tablet report, sending...
I20260709 12:09:54.374361 11310 webserver.cc:533] Webserver started at http://127.11.11.130:37819/ using document root <none> and password file <none>
I20260709 12:09:54.374400 11342 ts_manager.cc:194] Registered new tserver with Master: edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679)
I20260709 12:09:54.375217 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:09:54.375523 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:09:54.375883 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:09:54.377259 11342 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40728
I20260709 12:09:54.377545 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "94ef9c5a5b0142d68529b466758bb9ce"
format_stamp: "Formatted at 2026-07-09 12:09:54 on dist-test-slave-4qvp"
I20260709 12:09:54.385142 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:09:54.391031 11482 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.392164 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:09:54.392586 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "94ef9c5a5b0142d68529b466758bb9ce"
format_stamp: "Formatted at 2026-07-09 12:09:54 on dist-test-slave-4qvp"
I20260709 12:09:54.392975 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:09:54.423291 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:09:54.425287 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:09:54.427421 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:09:54.432019 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:09:54.432353 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.432686 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:09:54.432926 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.555668 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:45643
I20260709 12:09:54.555997 11544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:45643 every 8 connection(s)
I20260709 12:09:54.564476 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:09:54.573509 11548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:09:54.579277 11549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:09:54.584527 11545 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43031
I20260709 12:09:54.585054 11545 heartbeater.cc:461] Registering TS with master...
I20260709 12:09:54.586054 11545 heartbeater.cc:507] Master 127.11.11.190:43031 requested a full tablet report, sending...
I20260709 12:09:54.588735 11342 ts_manager.cc:194] Registered new tserver with Master: 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:54.590648 11342 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40738
W20260709 12:09:54.596761 11551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:09:54.597820 11310 server_base.cc:1061] running on GCE node
I20260709 12:09:54.598958 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:09:54.599215 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:09:54.599406 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783598994599387 us; error 0 us; skew 500 ppm
I20260709 12:09:54.600065 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:09:54.603360 11310 webserver.cc:533] Webserver started at http://127.11.11.131:37211/ using document root <none> and password file <none>
I20260709 12:09:54.603979 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:09:54.604229 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:09:54.604580 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:09:54.605933 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
format_stamp: "Formatted at 2026-07-09 12:09:54 on dist-test-slave-4qvp"
I20260709 12:09:54.611546 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:09:54.616022 11556 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.617152 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:09:54.617543 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
format_stamp: "Formatted at 2026-07-09 12:09:54 on dist-test-slave-4qvp"
I20260709 12:09:54.617962 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:09:54.635623 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:09:54.637143 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:09:54.639739 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:09:54.643496 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:09:54.643760 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.644029 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:09:54.644207 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:09:54.762539 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:46173
I20260709 12:09:54.762682 11618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:46173 every 8 connection(s)
I20260709 12:09:54.784686 11619 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43031
I20260709 12:09:54.785161 11619 heartbeater.cc:461] Registering TS with master...
I20260709 12:09:54.786291 11619 heartbeater.cc:507] Master 127.11.11.190:43031 requested a full tablet report, sending...
I20260709 12:09:54.788933 11342 ts_manager.cc:194] Registered new tserver with Master: 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:54.789273 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.021016814s
I20260709 12:09:54.790851 11342 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40740
I20260709 12:09:54.860015 11342 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40748:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:09:54.865650 11342 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260709 12:09:54.983752 11389 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:09:55.010324 11435 tablet_service.cc:1511] Processing CreateTablet for tablet 11497649fc7f46f889043a6069fe0db4 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:09:55.011554 11433 tablet_service.cc:1511] Processing CreateTablet for tablet aac3cbc7a3564854b3ce7d2e74d902c8 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:09:55.012426 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11497649fc7f46f889043a6069fe0db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.013437 11433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac3cbc7a3564854b3ce7d2e74d902c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.015225 11434 tablet_service.cc:1511] Processing CreateTablet for tablet 2979f5770c1d486f91f7690339c626d6 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:09:55.015798 11583 tablet_service.cc:1511] Processing CreateTablet for tablet 2979f5770c1d486f91f7690339c626d6 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:09:55.016835 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2979f5770c1d486f91f7690339c626d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.017467 11583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2979f5770c1d486f91f7690339c626d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.033658 11581 tablet_service.cc:1511] Processing CreateTablet for tablet 207b53cef5cd41b488fe59d21bf4f010 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:09:55.032994 11432 tablet_service.cc:1511] Processing CreateTablet for tablet 207b53cef5cd41b488fe59d21bf4f010 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:09:55.037156 11581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 207b53cef5cd41b488fe59d21bf4f010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.034857 11431 tablet_service.cc:1511] Processing CreateTablet for tablet 4082122061a440c69a7019e8a5382b08 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:09:55.041316 11582 tablet_service.cc:1511] Processing CreateTablet for tablet aac3cbc7a3564854b3ce7d2e74d902c8 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:09:55.038545 11580 tablet_service.cc:1511] Processing CreateTablet for tablet 4082122061a440c69a7019e8a5382b08 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:09:55.050262 11430 tablet_service.cc:1511] Processing CreateTablet for tablet f93991cb0fa74f50980b9d0abb8afb0b (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:09:55.050442 11579 tablet_service.cc:1511] Processing CreateTablet for tablet f93991cb0fa74f50980b9d0abb8afb0b (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:09:55.047320 11584 tablet_service.cc:1511] Processing CreateTablet for tablet 11497649fc7f46f889043a6069fe0db4 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:09:55.052438 11429 tablet_service.cc:1511] Processing CreateTablet for tablet 1b44dbd6f6a34f7590ea121e2639b12a (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:09:55.063524 11432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 207b53cef5cd41b488fe59d21bf4f010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.074755 11429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b44dbd6f6a34f7590ea121e2639b12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.077049 11582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac3cbc7a3564854b3ce7d2e74d902c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.082170 11579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93991cb0fa74f50980b9d0abb8afb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.084977 11584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11497649fc7f46f889043a6069fe0db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.087795 11580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4082122061a440c69a7019e8a5382b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.089294 11430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93991cb0fa74f50980b9d0abb8afb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.096618 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4082122061a440c69a7019e8a5382b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.102867 11578 tablet_service.cc:1511] Processing CreateTablet for tablet 1b44dbd6f6a34f7590ea121e2639b12a (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:09:55.112493 11578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b44dbd6f6a34f7590ea121e2639b12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.128185 11509 tablet_service.cc:1511] Processing CreateTablet for tablet 2979f5770c1d486f91f7690339c626d6 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:09:55.131438 11509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2979f5770c1d486f91f7690339c626d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.133961 11428 tablet_service.cc:1511] Processing CreateTablet for tablet 3b8f4cd3211445cf90a682b27d40607f (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:09:55.140830 11428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f4cd3211445cf90a682b27d40607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.149237 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 11497649fc7f46f889043a6069fe0db4 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:09:55.179713 11506 tablet_service.cc:1511] Processing CreateTablet for tablet 4082122061a440c69a7019e8a5382b08 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:09:55.173029 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11497649fc7f46f889043a6069fe0db4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.205524 11427 tablet_service.cc:1511] Processing CreateTablet for tablet b9823d449d634ad08eadedf5c907bd92 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:09:55.215232 11577 tablet_service.cc:1511] Processing CreateTablet for tablet 3b8f4cd3211445cf90a682b27d40607f (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:09:55.215427 11505 tablet_service.cc:1511] Processing CreateTablet for tablet f93991cb0fa74f50980b9d0abb8afb0b (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:09:55.195118 11508 tablet_service.cc:1511] Processing CreateTablet for tablet aac3cbc7a3564854b3ce7d2e74d902c8 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:09:55.215225 11507 tablet_service.cc:1511] Processing CreateTablet for tablet 207b53cef5cd41b488fe59d21bf4f010 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:09:55.224591 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4082122061a440c69a7019e8a5382b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.230088 11577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f4cd3211445cf90a682b27d40607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.235495 11507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 207b53cef5cd41b488fe59d21bf4f010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.231393 11427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9823d449d634ad08eadedf5c907bd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.243232 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aac3cbc7a3564854b3ce7d2e74d902c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.248656 11504 tablet_service.cc:1511] Processing CreateTablet for tablet 1b44dbd6f6a34f7590ea121e2639b12a (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:09:55.269358 11504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b44dbd6f6a34f7590ea121e2639b12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.269451 11503 tablet_service.cc:1511] Processing CreateTablet for tablet 3b8f4cd3211445cf90a682b27d40607f (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:09:55.272326 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93991cb0fa74f50980b9d0abb8afb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.293569 11502 tablet_service.cc:1511] Processing CreateTablet for tablet b9823d449d634ad08eadedf5c907bd92 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:09:55.301755 11576 tablet_service.cc:1511] Processing CreateTablet for tablet b9823d449d634ad08eadedf5c907bd92 (DEFAULT_TABLE table=test-workload [id=97c864f1cc634d0b9aa89d0de6a352ba]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:09:55.304801 11503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8f4cd3211445cf90a682b27d40607f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.334776 11502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9823d449d634ad08eadedf5c907bd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.367017 11576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9823d449d634ad08eadedf5c907bd92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:09:55.385192 11470 heartbeater.cc:499] Master 127.11.11.190:43031 was elected leader, sending a full tablet report...
I20260709 12:09:55.428956 11637 tablet_bootstrap.cc:492] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:55.461210 11638 tablet_bootstrap.cc:492] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:55.562988 11638 tablet_bootstrap.cc:654] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.577986 11639 tablet_bootstrap.cc:492] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:55.586491 11637 tablet_bootstrap.cc:654] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.594151 11545 heartbeater.cc:499] Master 127.11.11.190:43031 was elected leader, sending a full tablet report...
I20260709 12:09:55.600993 11639 tablet_bootstrap.cc:654] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.612870 11638 tablet_bootstrap.cc:492] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:55.613940 11638 ts_tablet_manager.cc:1403] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.175s	user 0.015s	sys 0.004s
I20260709 12:09:55.617118 11637 tablet_bootstrap.cc:492] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:55.617707 11637 ts_tablet_manager.cc:1403] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.215s	user 0.013s	sys 0.014s
I20260709 12:09:55.621261 11637 raft_consensus.cc:359] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.621542 11638 raft_consensus.cc:359] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.622308 11637 raft_consensus.cc:385] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.622473 11638 raft_consensus.cc:385] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.622716 11637 raft_consensus.cc:740] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.622880 11638 raft_consensus.cc:740] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.623693 11637 consensus_queue.cc:260] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.623693 11638 consensus_queue.cc:260] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.628854 11639 tablet_bootstrap.cc:492] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:55.629580 11639 ts_tablet_manager.cc:1403] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.053s	user 0.015s	sys 0.013s
I20260709 12:09:55.633570 11639 raft_consensus.cc:359] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.634624 11639 raft_consensus.cc:385] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.634903 11638 ts_tablet_manager.cc:1434] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.010s
I20260709 12:09:55.634996 11639 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.636301 11639 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.637699 11619 heartbeater.cc:499] Master 127.11.11.190:43031 was elected leader, sending a full tablet report...
I20260709 12:09:55.644755 11637 ts_tablet_manager.cc:1434] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.013s
I20260709 12:09:55.645989 11637 tablet_bootstrap.cc:492] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:55.646601 11639 ts_tablet_manager.cc:1434] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.010s
I20260709 12:09:55.648077 11639 tablet_bootstrap.cc:492] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:55.657469 11639 tablet_bootstrap.cc:654] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.659240 11638 tablet_bootstrap.cc:492] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:55.663678 11637 tablet_bootstrap.cc:654] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.669934 11645 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:55.670572 11645 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.674597 11645 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:55.680672 11639 tablet_bootstrap.cc:492] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:55.680599 11643 raft_consensus.cc:493] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:55.681968 11639 ts_tablet_manager.cc:1403] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.034s	user 0.018s	sys 0.007s
I20260709 12:09:55.681900 11643 raft_consensus.cc:515] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.687425 11639 raft_consensus.cc:359] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.689612 11639 raft_consensus.cc:385] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.690094 11639 raft_consensus.cc:740] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.691219 11637 tablet_bootstrap.cc:492] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:55.692241 11637 ts_tablet_manager.cc:1403] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.046s	user 0.011s	sys 0.010s
I20260709 12:09:55.691929 11639 consensus_queue.cc:260] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.701007 11637 raft_consensus.cc:359] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.705441 11639 ts_tablet_manager.cc:1434] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.002s
I20260709 12:09:55.702394 11637 raft_consensus.cc:385] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.699136 11643 leader_election.cc:290] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:55.710387 11637 raft_consensus.cc:740] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.714417 11639 tablet_bootstrap.cc:492] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:55.727051 11637 consensus_queue.cc:260] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.736930 11637 ts_tablet_manager.cc:1434] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.044s	user 0.012s	sys 0.000s
I20260709 12:09:55.738304 11637 tablet_bootstrap.cc:492] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:55.744233 11638 tablet_bootstrap.cc:654] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.768100 11637 tablet_bootstrap.cc:654] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.769075 11639 tablet_bootstrap.cc:654] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
W20260709 12:09:55.790037 11620 tablet.cc:2406] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:09:55.816895 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:55.817476 11637 tablet_bootstrap.cc:492] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:55.822191 11637 ts_tablet_manager.cc:1403] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.084s	user 0.014s	sys 0.009s
W20260709 12:09:55.826231 11484 leader_election.cc:343] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:55.843490 11638 tablet_bootstrap.cc:492] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:55.846350 11638 ts_tablet_manager.cc:1403] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.187s	user 0.019s	sys 0.013s
I20260709 12:09:55.845969 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
W20260709 12:09:55.850647 11485 leader_election.cc:343] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:55.851830 11485 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce; no voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:55.853966 11645 raft_consensus.cc:2749] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:09:55.854643 11639 tablet_bootstrap.cc:492] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:55.855386 11639 ts_tablet_manager.cc:1403] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.143s	user 0.017s	sys 0.018s
I20260709 12:09:55.850199 11637 raft_consensus.cc:359] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.856866 11637 raft_consensus.cc:385] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.858155 11637 raft_consensus.cc:740] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.859686 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:55.861492 11445 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 0.
I20260709 12:09:55.862447 11638 raft_consensus.cc:359] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.863247 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:55.863521 11638 raft_consensus.cc:385] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.863538 11639 raft_consensus.cc:359] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.864051 11638 raft_consensus.cc:740] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.864084 11520 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 0.
I20260709 12:09:55.864485 11639 raft_consensus.cc:385] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.864882 11639 raft_consensus.cc:740] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.865712 11639 consensus_queue.cc:260] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.866101 11560 leader_election.cc:304] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:55.859732 11637 consensus_queue.cc:260] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.867661 11643 raft_consensus.cc:2804] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:55.868072 11643 raft_consensus.cc:493] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:55.868409 11643 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:55.868510 11637 ts_tablet_manager.cc:1434] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.046s	user 0.010s	sys 0.002s
I20260709 12:09:55.875332 11637 tablet_bootstrap.cc:492] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:55.875173 11643 raft_consensus.cc:515] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.865089 11638 consensus_queue.cc:260] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.871057 11639 ts_tablet_manager.cc:1434] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:09:55.880153 11638 ts_tablet_manager.cc:1434] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.000s
I20260709 12:09:55.880249 11639 tablet_bootstrap.cc:492] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:55.882180 11643 leader_election.cc:290] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:55.882894 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:55.883471 11638 tablet_bootstrap.cc:492] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:55.883746 11445 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:55.888900 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:55.890100 11520 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:55.893980 11639 tablet_bootstrap.cc:654] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.902520 11637 tablet_bootstrap.cc:654] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.904874 11520 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 1.
I20260709 12:09:55.906329 11560 leader_election.cc:304] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:55.908020 11643 raft_consensus.cc:2804] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:55.908569 11643 raft_consensus.cc:697] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:09:55.909440 11643 consensus_queue.cc:237] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:55.910224 11639 tablet_bootstrap.cc:492] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:55.910686 11639 ts_tablet_manager.cc:1403] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.005s
I20260709 12:09:55.913857 11639 raft_consensus.cc:359] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.914701 11639 raft_consensus.cc:385] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.915053 11639 raft_consensus.cc:740] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.915910 11639 consensus_queue.cc:260] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.917794 11638 tablet_bootstrap.cc:654] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.922104 11639 ts_tablet_manager.cc:1434] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:09:55.923328 11639 tablet_bootstrap.cc:492] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:55.933465 11639 tablet_bootstrap.cc:654] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.935693 11342 catalog_manager.cc:5697] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:55.940680 11637 tablet_bootstrap.cc:492] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:55.941387 11637 ts_tablet_manager.cc:1403] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.066s	user 0.008s	sys 0.011s
I20260709 12:09:55.945312 11638 tablet_bootstrap.cc:492] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:55.945083 11637 raft_consensus.cc:359] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.946087 11637 raft_consensus.cc:385] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.946581 11445 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 1.
I20260709 12:09:55.946482 11637 raft_consensus.cc:740] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.948098 11637 consensus_queue.cc:260] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.951568 11637 ts_tablet_manager.cc:1434] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:09:55.953011 11637 tablet_bootstrap.cc:492] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:55.946095 11638 ts_tablet_manager.cc:1403] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.063s	user 0.014s	sys 0.008s
I20260709 12:09:55.969990 11638 raft_consensus.cc:359] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.971632 11637 tablet_bootstrap.cc:654] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.974416 11638 raft_consensus.cc:385] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.974962 11638 raft_consensus.cc:740] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.979118 11639 tablet_bootstrap.cc:492] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:55.979838 11639 ts_tablet_manager.cc:1403] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.057s	user 0.021s	sys 0.034s
I20260709 12:09:55.977775 11638 consensus_queue.cc:260] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
W20260709 12:09:55.986622 11389 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:09:55.989820 11639 raft_consensus.cc:359] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:55.992136 11639 raft_consensus.cc:385] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:55.993930 11639 raft_consensus.cc:740] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:55.991669 11638 ts_tablet_manager.cc:1434] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.037s	user 0.009s	sys 0.000s
I20260709 12:09:55.996587 11638 tablet_bootstrap.cc:492] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:55.999048 11637 tablet_bootstrap.cc:492] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:55.999449 11637 ts_tablet_manager.cc:1403] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.016s
I20260709 12:09:56.002561 11637 raft_consensus.cc:359] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.003402 11637 raft_consensus.cc:385] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.003762 11637 raft_consensus.cc:740] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.004562 11637 consensus_queue.cc:260] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.007222 11637 ts_tablet_manager.cc:1434] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:09:56.008360 11637 tablet_bootstrap.cc:492] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:56.009264 11638 tablet_bootstrap.cc:654] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:55.994994 11639 consensus_queue.cc:260] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.015383 11639 ts_tablet_manager.cc:1434] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.004s
I20260709 12:09:56.016842 11639 tablet_bootstrap.cc:492] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:56.015383 11637 tablet_bootstrap.cc:654] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.027284 11639 tablet_bootstrap.cc:654] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.040103 11645 raft_consensus.cc:493] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.040706 11645 raft_consensus.cc:515] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.043272 11645 leader_election.cc:290] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.044844 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.045572 11445 raft_consensus.cc:2468] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 0.
I20260709 12:09:56.047104 11484 leader_election.cc:304] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.047942 11645 raft_consensus.cc:2804] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.048310 11645 raft_consensus.cc:493] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.048641 11645 raft_consensus.cc:3060] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.049661 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.051795 11638 tablet_bootstrap.cc:492] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
W20260709 12:09:56.051977 11485 leader_election.cc:343] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.052387 11638 ts_tablet_manager.cc:1403] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.056s	user 0.011s	sys 0.010s
I20260709 12:09:56.056313 11638 raft_consensus.cc:359] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.057132 11638 raft_consensus.cc:385] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.057471 11638 raft_consensus.cc:740] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.057920 11645 raft_consensus.cc:515] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.058509 11638 consensus_queue.cc:260] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.060186 11645 leader_election.cc:290] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.061151 11637 tablet_bootstrap.cc:492] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:56.061558 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:56.061811 11637 ts_tablet_manager.cc:1403] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.054s	user 0.021s	sys 0.023s
I20260709 12:09:56.063004 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:56.063630 11445 raft_consensus.cc:3060] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:09:56.066002 11485 leader_election.cc:343] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.065721 11637 raft_consensus.cc:359] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.066730 11637 raft_consensus.cc:385] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.067132 11637 raft_consensus.cc:740] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.068094 11637 consensus_queue.cc:260] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.071269 11637 ts_tablet_manager.cc:1434] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:09:56.072331 11639 tablet_bootstrap.cc:492] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:56.072925 11639 ts_tablet_manager.cc:1403] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.056s	user 0.019s	sys 0.012s
I20260709 12:09:56.073026 11637 tablet_bootstrap.cc:492] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:56.075853 11644 raft_consensus.cc:493] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.076431 11644 raft_consensus.cc:515] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.085063 11639 raft_consensus.cc:359] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.086081 11639 raft_consensus.cc:385] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.086416 11639 raft_consensus.cc:740] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.087067 11644 leader_election.cc:290] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.087286 11639 consensus_queue.cc:260] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.089977 11639 ts_tablet_manager.cc:1434] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.006s
I20260709 12:09:56.091559 11639 tablet_bootstrap.cc:492] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:56.093458 11645 raft_consensus.cc:493] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.094347 11637 tablet_bootstrap.cc:654] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.094158 11645 raft_consensus.cc:515] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.097196 11645 leader_election.cc:290] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.099035 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.099803 11594 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 0.
I20260709 12:09:56.102327 11445 raft_consensus.cc:2468] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 1.
I20260709 12:09:56.102876 11485 leader_election.cc:304] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:56.103369 11638 ts_tablet_manager.cc:1434] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 12:09:56.104676 11645 raft_consensus.cc:2804] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.105450 11645 raft_consensus.cc:493] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.106163 11645 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.107168 11638 tablet_bootstrap.cc:492] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:56.113898 11639 tablet_bootstrap.cc:654] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.122355 11644 raft_consensus.cc:493] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.123140 11644 raft_consensus.cc:515] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.131157 11644 leader_election.cc:290] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.129575 11645 raft_consensus.cc:515] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.138012 11484 leader_election.cc:304] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5
I20260709 12:09:56.138423 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.139158 11445 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 0.
I20260709 12:09:56.142436 11645 leader_election.cc:290] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.143559 11645 raft_consensus.cc:2804] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.142374 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:56.144146 11645 raft_consensus.cc:697] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:56.144331 11594 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.144724 11639 tablet_bootstrap.cc:492] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:56.145251 11645 consensus_queue.cc:237] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.145265 11639 ts_tablet_manager.cc:1403] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.054s	user 0.005s	sys 0.008s
I20260709 12:09:56.149957 11639 raft_consensus.cc:359] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.150910 11639 raft_consensus.cc:385] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.151196 11639 raft_consensus.cc:740] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.154073 11637 tablet_bootstrap.cc:492] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:56.154754 11637 ts_tablet_manager.cc:1403] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.082s	user 0.013s	sys 0.007s
I20260709 12:09:56.157258 11639 consensus_queue.cc:260] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.160234 11637 raft_consensus.cc:359] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.161481 11637 raft_consensus.cc:385] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.165880 11637 raft_consensus.cc:740] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.176344 11637 consensus_queue.cc:260] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.177883 11639 ts_tablet_manager.cc:1434] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.031s	user 0.013s	sys 0.008s
I20260709 12:09:56.178916 11594 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 1.
I20260709 12:09:56.179808 11639 tablet_bootstrap.cc:492] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:56.180058 11638 tablet_bootstrap.cc:654] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.180687 11485 leader_election.cc:304] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:56.185369 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:56.186182 11445 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.188378 11679 raft_consensus.cc:2804] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.189814 11679 raft_consensus.cc:697] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:56.191001 11637 ts_tablet_manager.cc:1434] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.005s
I20260709 12:09:56.190878 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.190114 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.192193 11519 raft_consensus.cc:2468] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 0.
I20260709 12:09:56.199954 11637 tablet_bootstrap.cc:492] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:56.200701 11411 leader_election.cc:304] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.201761 11644 raft_consensus.cc:2804] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.202186 11644 raft_consensus.cc:493] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.201226 11342 catalog_manager.cc:5697] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 1 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:56.202533 11644 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.193231 11679 consensus_queue.cc:237] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.225543 11644 raft_consensus.cc:515] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.228368 11677 raft_consensus.cc:493] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.228644 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.228076 11445 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 1.
I20260709 12:09:56.230245 11643 raft_consensus.cc:493] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.229449 11593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.230005 11594 raft_consensus.cc:2393] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate edea0c9ac6854273961ab944d9dd4664 in current term 1: Already voted for candidate 94ef9c5a5b0142d68529b466758bb9ce in this term.
I20260709 12:09:56.232769 11643 raft_consensus.cc:515] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.228973 11677 raft_consensus.cc:515] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.236021 11643 leader_election.cc:290] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
W20260709 12:09:56.236214 11410 leader_election.cc:343] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.236533 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:56.237318 11519 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.237353 11410 leader_election.cc:304] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edea0c9ac6854273961ab944d9dd4664; no voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:09:56.240060 11639 tablet_bootstrap.cc:654] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.244638 11638 tablet_bootstrap.cc:492] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:56.245215 11638 ts_tablet_manager.cc:1403] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.139s	user 0.011s	sys 0.027s
I20260709 12:09:56.247252 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.248497 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
W20260709 12:09:56.248695 11560 leader_election.cc:343] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.249454 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:56.249789 11445 raft_consensus.cc:2468] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 0.
I20260709 12:09:56.250074 11638 raft_consensus.cc:359] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
W20260709 12:09:56.253166 11410 leader_election.cc:343] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.246389 11644 leader_election.cc:290] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.251175 11638 raft_consensus.cc:385] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.253194 11558 leader_election.cc:304] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:09:56.257125 11644 raft_consensus.cc:2749] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:09:56.258913 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.256345 11638 raft_consensus.cc:740] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.260056 11643 raft_consensus.cc:2804] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.260828 11637 tablet_bootstrap.cc:654] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.260375 11638 consensus_queue.cc:260] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.260712 11643 raft_consensus.cc:493] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.255451 11677 leader_election.cc:290] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.261436 11643 raft_consensus.cc:3060] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.262352 11594 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 0.
I20260709 12:09:56.266695 11519 raft_consensus.cc:2468] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 1.
I20260709 12:09:56.270854 11638 ts_tablet_manager.cc:1434] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.005s
I20260709 12:09:56.272001 11638 tablet_bootstrap.cc:492] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:56.274508 11411 leader_election.cc:304] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5
I20260709 12:09:56.276059 11485 leader_election.cc:304] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:56.286924 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.287863 11445 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 0.
I20260709 12:09:56.289247 11677 raft_consensus.cc:2804] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.296010 11677 raft_consensus.cc:493] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.299363 11677 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.297667 11643 raft_consensus.cc:515] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.305879 11652 raft_consensus.cc:493] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.317318 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:56.318495 11445 raft_consensus.cc:3060] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.320348 11639 tablet_bootstrap.cc:492] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:56.320919 11639 ts_tablet_manager.cc:1403] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.141s	user 0.027s	sys 0.030s
I20260709 12:09:56.306471 11652 raft_consensus.cc:515] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.323990 11639 raft_consensus.cc:359] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.324744 11639 raft_consensus.cc:385] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.325029 11639 raft_consensus.cc:740] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.325845 11643 leader_election.cc:290] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:56.325762 11639 consensus_queue.cc:260] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.327454 11639 ts_tablet_manager.cc:1434] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:09:56.327320 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9823d449d634ad08eadedf5c907bd92" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.328498 11639 tablet_bootstrap.cc:492] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce: Bootstrap starting.
I20260709 12:09:56.329123 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:56.330468 11638 tablet_bootstrap.cc:654] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
W20260709 12:09:56.330822 11560 leader_election.cc:343] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.332100 11652 leader_election.cc:290] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:56.328099 11519 raft_consensus.cc:2468] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 0.
I20260709 12:09:56.331573 11342 catalog_manager.cc:5697] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 1 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:56.337225 11637 tablet_bootstrap.cc:492] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:56.333703 11681 raft_consensus.cc:2804] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.337747 11637 ts_tablet_manager.cc:1403] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.138s	user 0.007s	sys 0.019s
I20260709 12:09:56.337349 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9823d449d634ad08eadedf5c907bd92" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.338469 11444 raft_consensus.cc:2468] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 0.
I20260709 12:09:56.338652 11560 leader_election.cc:304] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:56.340462 11637 raft_consensus.cc:359] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.341125 11637 raft_consensus.cc:385] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.341426 11637 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.343060 11637 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.344985 11652 raft_consensus.cc:2804] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.346254 11652 raft_consensus.cc:493] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.346710 11652 raft_consensus.cc:3060] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.347841 11643 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.340226 11681 raft_consensus.cc:697] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:56.351536 11637 ts_tablet_manager.cc:1434] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 12:09:56.350806 11681 consensus_queue.cc:237] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.355813 11637 tablet_bootstrap.cc:492] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: Bootstrap starting.
I20260709 12:09:56.345309 11445 raft_consensus.cc:2468] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 1.
I20260709 12:09:56.358608 11677 raft_consensus.cc:515] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.363189 11558 leader_election.cc:304] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:09:56.364310 11643 raft_consensus.cc:2804] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.363198 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:56.365224 11594 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.374166 11637 tablet_bootstrap.cc:654] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.375325 11639 tablet_bootstrap.cc:654] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.384224 11594 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 1.
I20260709 12:09:56.387506 11485 leader_election.cc:304] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:56.388056 11652 raft_consensus.cc:515] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.388845 11679 raft_consensus.cc:2804] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.389554 11679 raft_consensus.cc:697] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:56.391007 11679 consensus_queue.cc:237] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.364584 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:56.394093 11445 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.368439 11677 leader_election.cc:290] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.403383 11652 leader_election.cc:290] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:56.398345 11643 raft_consensus.cc:697] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:09:56.404407 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9823d449d634ad08eadedf5c907bd92" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:56.405777 11444 raft_consensus.cc:3060] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.405519 11643 consensus_queue.cc:237] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.408454 11342 catalog_manager.cc:5697] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 0 to 1, leader changed from <none> to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:56.415076 11638 tablet_bootstrap.cc:492] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:56.415145 11444 raft_consensus.cc:2468] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 1.
I20260709 12:09:56.423979 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9823d449d634ad08eadedf5c907bd92" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:56.424716 11519 raft_consensus.cc:3060] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.417048 11638 ts_tablet_manager.cc:1403] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.145s	user 0.008s	sys 0.024s
I20260709 12:09:56.429997 11681 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.431025 11681 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.414073 11445 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 1.
I20260709 12:09:56.438301 11558 leader_election.cc:304] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.431234 11340 catalog_manager.cc:5697] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 1 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:56.453956 11679 raft_consensus.cc:493] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.454891 11679 raft_consensus.cc:515] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.462119 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.459990 11679 leader_election.cc:290] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.462926 11518 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 0.
I20260709 12:09:56.464306 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.459532 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.482195 11637 tablet_bootstrap.cc:492] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: No bootstrap required, opened a new log
I20260709 12:09:56.479048 11689 raft_consensus.cc:2804] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.483090 11637 ts_tablet_manager.cc:1403] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: Time spent bootstrapping tablet: real 0.127s	user 0.020s	sys 0.003s
I20260709 12:09:56.486343 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.481482 11681 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.481112 11411 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.489481 11689 raft_consensus.cc:697] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
W20260709 12:09:56.490116 11485 leader_election.cc:343] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.486332 11638 raft_consensus.cc:359] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
W20260709 12:09:56.492693 11410 leader_election.cc:343] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.491959 11694 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:09:56.493999 11519 raft_consensus.cc:2468] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 1.
I20260709 12:09:56.494992 11638 raft_consensus.cc:385] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:09:56.496392 11484 leader_election.cc:343] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.493947 11681 raft_consensus.cc:2804] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.497890 11639 tablet_bootstrap.cc:492] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce: No bootstrap required, opened a new log
I20260709 12:09:56.497206 11638 raft_consensus.cc:740] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.497922 11681 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.498623 11639 ts_tablet_manager.cc:1403] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce: Time spent bootstrapping tablet: real 0.170s	user 0.019s	sys 0.013s
I20260709 12:09:56.498749 11681 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.497486 11484 leader_election.cc:304] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce; no voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:56.499965 11689 consensus_queue.cc:237] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.501094 11679 raft_consensus.cc:2749] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:09:56.501744 11638 consensus_queue.cc:260] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.506526 11638 ts_tablet_manager.cc:1434] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.077s	user 0.002s	sys 0.007s
I20260709 12:09:56.511864 11639 raft_consensus.cc:359] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.512895 11639 raft_consensus.cc:385] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.513252 11639 raft_consensus.cc:740] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.518322 11639 consensus_queue.cc:260] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.522212 11639 ts_tablet_manager.cc:1434] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.002s
I20260709 12:09:56.525050 11681 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.525465 11637 raft_consensus.cc:359] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.526427 11637 raft_consensus.cc:385] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.526868 11637 raft_consensus.cc:740] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.527882 11637 consensus_queue.cc:260] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.530488 11681 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.531069 11638 tablet_bootstrap.cc:492] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:56.534246 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:56.535115 11519 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.536123 11637 ts_tablet_manager.cc:1434] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 12:09:56.538842 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:56.541751 11638 tablet_bootstrap.cc:654] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.545691 11519 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 1.
I20260709 12:09:56.547191 11411 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.548262 11681 raft_consensus.cc:2804] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.549052 11681 raft_consensus.cc:697] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:56.549978 11681 consensus_queue.cc:237] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
W20260709 12:09:56.555842 11410 leader_election.cc:343] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:09:56.552672 11342 catalog_manager.cc:5697] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260709 12:09:56.592541 11638 tablet_bootstrap.cc:492] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:56.593103 11638 ts_tablet_manager.cc:1403] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.062s	user 0.014s	sys 0.037s
I20260709 12:09:56.593410 11339 catalog_manager.cc:5697] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260709 12:09:56.596213 11638 raft_consensus.cc:359] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.596922 11638 raft_consensus.cc:385] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.597193 11638 raft_consensus.cc:740] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.597887 11638 consensus_queue.cc:260] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.599783 11638 ts_tablet_manager.cc:1434] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.007s
I20260709 12:09:56.600818 11638 tablet_bootstrap.cc:492] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:56.607903 11342 catalog_manager.cc:5697] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 0 to 1, leader changed from <none> to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:56.609006 11638 tablet_bootstrap.cc:654] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.618508 11638 tablet_bootstrap.cc:492] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:56.619000 11638 ts_tablet_manager.cc:1403] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20260709 12:09:56.621978 11638 raft_consensus.cc:359] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.622660 11638 raft_consensus.cc:385] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.622926 11638 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.623498 11594 raft_consensus.cc:3060] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.623565 11638 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.625564 11638 ts_tablet_manager.cc:1434] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
I20260709 12:09:56.626631 11638 tablet_bootstrap.cc:492] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Bootstrap starting.
I20260709 12:09:56.632911 11638 tablet_bootstrap.cc:654] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Neither blocks nor log segments found. Creating new log.
I20260709 12:09:56.633294 11679 consensus_queue.cc:1048] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.639007 11638 tablet_bootstrap.cc:492] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: No bootstrap required, opened a new log
I20260709 12:09:56.639667 11638 ts_tablet_manager.cc:1403] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.001s
I20260709 12:09:56.642423 11638 raft_consensus.cc:359] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.643242 11638 raft_consensus.cc:385] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:09:56.643602 11638 raft_consensus.cc:740] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Initialized, Role: FOLLOWER
I20260709 12:09:56.644783 11638 consensus_queue.cc:260] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.646939 11638 ts_tablet_manager.cc:1434] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:09:56.659144 11677 consensus_queue.cc:1048] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:09:56.659588 11681 raft_consensus.cc:493] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.660123 11681 raft_consensus.cc:515] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.662356 11681 leader_election.cc:290] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.665879 11593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.666592 11593 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 0.
I20260709 12:09:56.667699 11410 leader_election.cc:304] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.668462 11681 raft_consensus.cc:2804] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:09:56.668540 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.668861 11681 raft_consensus.cc:493] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:09:56.669293 11519 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 0.
I20260709 12:09:56.669309 11681 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.685946 11681 raft_consensus.cc:515] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.689101 11677 raft_consensus.cc:493] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.689332 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:56.690128 11519 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.689776 11677 raft_consensus.cc:515] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.690476 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:56.691061 11594 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.692821 11677 leader_election.cc:290] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.696687 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.697007 11593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" is_pre_election: true
I20260709 12:09:56.697849 11445 raft_consensus.cc:2393] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 94ef9c5a5b0142d68529b466758bb9ce in current term 1: Already voted for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in this term.
I20260709 12:09:56.699013 11594 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 1.
I20260709 12:09:56.700531 11410 leader_election.cc:304] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:56.689265 11681 leader_election.cc:290] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 1 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:56.701064 11484 leader_election.cc:304] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce; no voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:56.700994 11519 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 1.
I20260709 12:09:56.702519 11644 raft_consensus.cc:2804] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:09:56.704128 11677 raft_consensus.cc:3060] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.706207 11644 raft_consensus.cc:697] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:56.706032 11645 consensus_queue.cc:1048] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.707156 11644 consensus_queue.cc:237] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.714069 11677 raft_consensus.cc:2749] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:09:56.728375 11342 catalog_manager.cc:5697] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 0 to 1, leader changed from <none> to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:56.729555 11677 consensus_queue.cc:1048] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.828680 11652 raft_consensus.cc:493] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.829250 11652 raft_consensus.cc:515] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.831334 11652 leader_election.cc:290] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:56.832633 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:56.833328 11519 raft_consensus.cc:2393] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in current term 1: Already voted for candidate edea0c9ac6854273961ab944d9dd4664 in this term.
I20260709 12:09:56.832844 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.835176 11558 leader_election.cc:304] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5; no voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:56.836274 11652 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:56.841768 11652 raft_consensus.cc:2749] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:09:56.899289 11689 consensus_queue.cc:1048] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:09:56.900192 11652 consensus_queue.cc:1048] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:09:56.914264 11689 consensus_queue.cc:1048] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.936653 11644 consensus_queue.cc:1048] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.948915 11689 consensus_queue.cc:1048] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:56.960470 11644 consensus_queue.cc:1048] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:09:56.979162 11709 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:09:56.986886 11694 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:09:56.987587 11694 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:56.991161 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edea0c9ac6854273961ab944d9dd4664" is_pre_election: true
I20260709 12:09:56.998485 11694 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:57.014523 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce" is_pre_election: true
I20260709 12:09:57.015749 11518 raft_consensus.cc:2393] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in current term 1: Already voted for candidate edea0c9ac6854273961ab944d9dd4664 in this term.
I20260709 12:09:57.019539 11560 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5; no voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:57.020210 11709 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:09:57.021329 11694 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:09:57.048002 11592 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
mode: GRACEFUL
new_leader_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:38222
I20260709 12:09:57.048974 11592 raft_consensus.cc:606] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Received request to transfer leadership to TS edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:57.061219 11694 raft_consensus.cc:2749] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:09:57.096558 11644 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:57.120855 11518 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:48340
I20260709 12:09:57.121589 11518 raft_consensus.cc:606] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Received request to transfer leadership to TS edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:57.122298 11706 consensus_queue.cc:1048] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:57.122726 11644 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:09:57.137001 11520 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:48340
I20260709 12:09:57.137835 11520 raft_consensus.cc:606] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Received request to transfer leadership to TS edea0c9ac6854273961ab944d9dd4664
I20260709 12:09:57.150833 11706 consensus_queue.cc:1048] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:09:57.178424 11445 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4082122061a440c69a7019e8a5382b08"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:47722
I20260709 12:09:57.179229 11445 raft_consensus.cc:606] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:09:57.190073 11445 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:47722
I20260709 12:09:57.191044 11445 raft_consensus.cc:606] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:09:57.202847 11592 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:38222
I20260709 12:09:57.203771 11592 raft_consensus.cc:606] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:09:57.222625 11709 raft_consensus.cc:993] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:09:57.223306 11677 raft_consensus.cc:1081] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Signalling peer edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:09:57.225354 11445 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:47708
I20260709 12:09:57.225910 11445 raft_consensus.cc:493] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:09:57.226241 11445 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.231868 11445 raft_consensus.cc:515] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.234594 11445 leader_election.cc:290] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:57.234910 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:57.235165 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:57.235735 11520 raft_consensus.cc:3055] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:09:57.235867 11592 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.236213 11520 raft_consensus.cc:740] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:57.237234 11520 consensus_queue.cc:260] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.239428 11520 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.244967 11592 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:09:57.246469 11410 leader_election.cc:304] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:57.249686 11706 raft_consensus.cc:2804] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:09:57.250128 11706 raft_consensus.cc:697] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:57.250835 11706 consensus_queue.cc:237] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.255810 11520 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:09:57.259578 11341 catalog_manager.cc:5697] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 1 to 2, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:57.472867 11652 raft_consensus.cc:993] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:09:57.473428 11694 raft_consensus.cc:1081] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:09:57.475430 11520 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48314
I20260709 12:09:57.476044 11520 raft_consensus.cc:493] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:09:57.476437 11520 raft_consensus.cc:3060] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.481417 11520 raft_consensus.cc:515] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.484009 11520 leader_election.cc:290] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:57.484620 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:57.484684 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b44dbd6f6a34f7590ea121e2639b12a" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:57.485340 11445 raft_consensus.cc:3060] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.485450 11592 raft_consensus.cc:3055] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:09:57.485977 11592 raft_consensus.cc:740] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:09:57.486757 11592 consensus_queue.cc:260] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.487926 11592 raft_consensus.cc:3060] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.496989 11592 raft_consensus.cc:2468] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:09:57.497386 11445 raft_consensus.cc:2468] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:09:57.498639 11485 leader_election.cc:304] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:57.499912 11677 raft_consensus.cc:2804] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:09:57.500743 11677 raft_consensus.cc:697] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:57.502611 11677 consensus_queue.cc:237] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.511235 11341 catalog_manager.cc:5697] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 1 to 2, leader changed from 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 2 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:57.556424 11652 raft_consensus.cc:993] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: : Instructing follower edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:09:57.556934 11694 raft_consensus.cc:1081] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Signalling peer edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:09:57.558837 11445 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:47720
I20260709 12:09:57.559533 11445 raft_consensus.cc:493] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:09:57.559916 11445 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.567579 11445 raft_consensus.cc:515] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:57.570746 11445 leader_election.cc:290] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Requested vote from peers 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:09:57.572012 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:57.572012 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:57.572901 11520 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.572988 11592 raft_consensus.cc:3055] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:09:57.573503 11592 raft_consensus.cc:740] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:09:57.574976 11592 consensus_queue.cc:260] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:57.577049 11592 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.582465 11520 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:09:57.585558 11644 raft_consensus.cc:993] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:09:57.586185 11644 raft_consensus.cc:1081] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:09:57.587687 11411 leader_election.cc:304] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:57.588099 11519 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48326
I20260709 12:09:57.588569 11706 raft_consensus.cc:2804] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:09:57.588776 11519 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:09:57.589087 11706 raft_consensus.cc:697] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:57.589257 11519 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.590075 11706 consensus_queue.cc:237] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:09:57.600015 11519 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.604679 11519 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:57.607447 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:57.608964 11445 raft_consensus.cc:3055] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:09:57.609747 11445 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:57.608266 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:57.611652 11594 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.611225 11445 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.612836 11445 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.614737 11341 catalog_manager.cc:5697] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 1 to 2, leader changed from 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131) to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:57.620461 11594 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:09:57.611177 11592 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:09:57.621068 11445 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:09:57.622936 11485 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:57.623857 11677 raft_consensus.cc:2804] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:09:57.624393 11677 raft_consensus.cc:697] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:57.625222 11677 consensus_queue.cc:237] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.633502 11339 catalog_manager.cc:5697] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 1 to 2, leader changed from edea0c9ac6854273961ab944d9dd4664 (127.11.11.129) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 2 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:57.636855 11644 raft_consensus.cc:993] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:09:57.637365 11706 raft_consensus.cc:1081] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:09:57.639163 11519 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4082122061a440c69a7019e8a5382b08"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48326
I20260709 12:09:57.640816 11519 raft_consensus.cc:493] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:09:57.641181 11519 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.649111 11519 raft_consensus.cc:515] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.651657 11519 leader_election.cc:290] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:57.652935 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:57.653489 11592 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.653595 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:09:57.655138 11445 raft_consensus.cc:3055] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:09:57.655490 11445 raft_consensus.cc:740] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:57.656387 11445 consensus_queue.cc:260] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.657822 11445 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.660636 11592 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:09:57.662002 11485 leader_election.cc:304] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:09:57.662748 11677 raft_consensus.cc:2804] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:09:57.663198 11677 raft_consensus.cc:697] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:57.663952 11677 consensus_queue.cc:237] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.668959 11445 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:09:57.671752 11339 catalog_manager.cc:5697] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 1 to 2, leader changed from edea0c9ac6854273961ab944d9dd4664 (127.11.11.129) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 2 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:57.702157 11709 raft_consensus.cc:993] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:09:57.702757 11677 raft_consensus.cc:1081] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Signalling peer edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:09:57.703550 11592 raft_consensus.cc:1275] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:57.704428 11445 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:47708
I20260709 12:09:57.705120 11445 raft_consensus.cc:493] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:09:57.705565 11445 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.705257 11706 consensus_queue.cc:1048] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:57.712401 11445 raft_consensus.cc:515] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.717691 11445 leader_election.cc:290] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:09:57.717722 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:09:57.719000 11519 raft_consensus.cc:3055] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:09:57.719328 11519 raft_consensus.cc:740] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:09:57.720630 11519 consensus_queue.cc:260] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.722323 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:09:57.723074 11592 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.723318 11519 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:09:57.734705 11592 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:09:57.736331 11410 leader_election.cc:304] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:09:57.737583 11706 raft_consensus.cc:2804] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:09:57.738142 11706 raft_consensus.cc:697] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:09:57.739108 11519 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:09:57.739049 11706 consensus_queue.cc:237] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:09:57.742569 11520 raft_consensus.cc:1275] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:57.747663 11706 consensus_queue.cc:1048] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:57.749416 11339 catalog_manager.cc:5697] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 1 to 2, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:09:57.886884 11445 raft_consensus.cc:1275] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:57.889770 11677 consensus_queue.cc:1048] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:57.900959 11594 raft_consensus.cc:1275] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:57.902890 11735 consensus_queue.cc:1048] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:58.065322 11520 raft_consensus.cc:1275] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.066865 11706 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:09:58.077441 11594 raft_consensus.cc:1275] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.079633 11706 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:58.154122 11445 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.155694 11735 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:58.160382 11520 raft_consensus.cc:1275] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.163035 11644 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:09:58.166795 11594 raft_consensus.cc:1275] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.170342 11677 consensus_queue.cc:1048] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:09:58.171656 11594 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.173893 11444 raft_consensus.cc:1275] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.175463 11677 consensus_queue.cc:1048] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:09:58.178366 11709 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:09:58.199397 11592 raft_consensus.cc:1275] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:09:58.204324 11737 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:00.253067 11310 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:10:00.254361 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:00.280576 11445 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
mode: GRACEFUL
new_leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:43200
I20260709 12:10:00.281330 11445 raft_consensus.cc:606] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Received request to transfer leadership to TS 8f56f7c1df644b8ca2c123f6e4bc8fe5
I20260709 12:10:00.294184 11520 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:52082
I20260709 12:10:00.294860 11520 raft_consensus.cc:606] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Received request to transfer leadership to TS 8f56f7c1df644b8ca2c123f6e4bc8fe5
I20260709 12:10:00.296456 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:10:00.297137 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:00.808611 11709 raft_consensus.cc:993] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:00.809123 11709 raft_consensus.cc:1081] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Signalling peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:00.810778 11594 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:38196
I20260709 12:10:00.811262 11594 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:00.811561 11594 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:00.816730 11594 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:00.819331 11594 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 3 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:10:00.820478 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:00.820626 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:00.821138 11520 raft_consensus.cc:3055] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:00.821290 11445 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:00.821556 11520 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:00.822505 11520 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:00.823539 11520 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:00.826345 11445 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 3.
I20260709 12:10:00.827559 11558 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:00.828401 11694 raft_consensus.cc:2804] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:00.829655 11520 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 3.
I20260709 12:10:00.829881 11694 raft_consensus.cc:697] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:10:00.831882 11694 consensus_queue.cc:237] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:00.839707 11340 catalog_manager.cc:5697] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 2 to 3, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:00.901877 11737 raft_consensus.cc:993] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: : Instructing follower 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:00.902405 11747 raft_consensus.cc:1081] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Signalling peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:00.904671 11594 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:38208
I20260709 12:10:00.905297 11594 raft_consensus.cc:493] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:00.905673 11594 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:00.911013 11594 raft_consensus.cc:515] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:10:00.913549 11594 leader_election.cc:290] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 3 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:10:00.914954 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:00.914949 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:00.915637 11445 raft_consensus.cc:3055] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:00.915769 11520 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:00.915952 11445 raft_consensus.cc:740] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:00.916698 11445 consensus_queue.cc:260] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:10:00.917698 11445 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:00.921221 11520 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 3.
I20260709 12:10:00.922547 11560 leader_election.cc:304] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, 94ef9c5a5b0142d68529b466758bb9ce; no voters: 
I20260709 12:10:00.923074 11445 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 3.
I20260709 12:10:00.923393 11694 raft_consensus.cc:2804] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:00.923905 11694 raft_consensus.cc:697] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:10:00.924708 11694 consensus_queue.cc:237] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:10:00.932965 11340 catalog_manager.cc:5697] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 2 to 3, leader changed from edea0c9ac6854273961ab944d9dd4664 (127.11.11.129) to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:01.246982 11520 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:01.248672 11694 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:01.262859 11445 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:01.264670 11694 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:01.298581 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:01.304880 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:01.305420 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:01.450649 11445 raft_consensus.cc:1275] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:01.452066 11758 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:01.461795 11520 raft_consensus.cc:1275] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:01.463317 11758 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:02.306829 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:02.314126 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:02.314654 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:03.316025 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:03.321656 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:03.322094 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:04.323217 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:04.328994 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:04.329447 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:05.330682 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:05.336452 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:05.337001 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:06.338186 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:06.343674 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:06.344161 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:07.345244 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:07.350659 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:07.351137 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:08.352264 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:08.357966 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:08.358513 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:09.359825 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:09.365852 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:09.366402 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:10.407601 11310 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:10:10.414701 11594 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:38222
I20260709 12:10:10.415521 11594 raft_consensus.cc:606] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:10:10.423813 11445 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:47722
I20260709 12:10:10.424687 11445 raft_consensus.cc:606] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:10:10.432621 11444 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:47722
I20260709 12:10:10.433310 11444 raft_consensus.cc:606] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:10:10.440291 11444 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:47722
I20260709 12:10:10.440899 11444 raft_consensus.cc:606] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:10:10.452188 11594 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
mode: GRACEFUL
new_leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:38222
I20260709 12:10:10.452816 11594 raft_consensus.cc:606] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Received request to transfer leadership to TS 94ef9c5a5b0142d68529b466758bb9ce
I20260709 12:10:10.492141 11737 raft_consensus.cc:993] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.492738 11775 raft_consensus.cc:1081] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.494841 11519 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48326
I20260709 12:10:10.495609 11519 raft_consensus.cc:493] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:10.496086 11519 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:10.578452 11519 raft_consensus.cc:515] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.580852 11519 leader_election.cc:290] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:10.582123 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:10.582990 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:10.583313 11444 raft_consensus.cc:3055] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:10.583747 11594 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:10.583776 11444 raft_consensus.cc:740] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:10.585655 11444 consensus_queue.cc:260] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.587195 11444 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:10.630328 11737 raft_consensus.cc:993] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.630817 11775 raft_consensus.cc:1081] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.632629 11519 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48326
I20260709 12:10:10.633246 11519 raft_consensus.cc:493] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:10.633731 11519 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:10.634860 11775 raft_consensus.cc:993] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.635351 11737 raft_consensus.cc:1081] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.636986 11520 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48326
I20260709 12:10:10.637465 11520 raft_consensus.cc:493] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:10.637810 11520 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:10.647217 11776 raft_consensus.cc:993] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.647776 11776 raft_consensus.cc:1081] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.649897 11518 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48314
I20260709 12:10:10.650511 11518 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:10.650799 11518 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:10.679968 11519 raft_consensus.cc:515] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.680538 11520 raft_consensus.cc:515] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.679912 11444 raft_consensus.cc:2468] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:10:10.684705 11519 leader_election.cc:290] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 3 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:10.679912 11594 raft_consensus.cc:2468] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 2.
I20260709 12:10:10.684896 11520 leader_election.cc:290] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 3 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:10.686861 11484 leader_election.cc:304] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:10.687376 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:10.687920 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:10.688802 11592 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:10.688958 11774 raft_consensus.cc:2804] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:10.692160 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aac3cbc7a3564854b3ce7d2e74d902c8" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:10.692824 11445 raft_consensus.cc:3055] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:10.693151 11445 raft_consensus.cc:740] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:10.693771 11774 raft_consensus.cc:697] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:10.693995 11445 consensus_queue.cc:260] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.696501 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:10.695138 11774 consensus_queue.cc:237] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.694365 11594 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:10.697238 11444 raft_consensus.cc:3055] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:10.697847 11444 raft_consensus.cc:740] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:10.698757 11444 consensus_queue.cc:260] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.696950 11445 raft_consensus.cc:3060] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:10.700536 11444 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:10.713060 11340 catalog_manager.cc:5697] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 1 to 2, leader changed from edea0c9ac6854273961ab944d9dd4664 (127.11.11.129) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 2 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:10.749905 11771 raft_consensus.cc:993] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: : Instructing follower 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.750671 11771 raft_consensus.cc:1081] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Signalling peer 94ef9c5a5b0142d68529b466758bb9ce to start an election
I20260709 12:10:10.753213 11519 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
 from {username='slave'} at 127.0.0.1:48314
I20260709 12:10:10.753940 11519 raft_consensus.cc:493] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:10.754328 11519 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:10.774513 11444 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 3.
I20260709 12:10:10.774871 11445 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 3.
I20260709 12:10:10.776388 11594 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 3.
I20260709 12:10:10.774575 11518 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.774569 11592 raft_consensus.cc:2468] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 3.
I20260709 12:10:10.778513 11484 leader_election.cc:304] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:10.780496 11518 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 4 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:10.781034 11484 leader_election.cc:304] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:10.782450 11778 raft_consensus.cc:2804] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:10.782729 11774 raft_consensus.cc:2804] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:10.783010 11778 raft_consensus.cc:697] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:10.783130 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:10.783710 11445 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:10.783815 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:10.784039 11778 consensus_queue.cc:237] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.784806 11592 raft_consensus.cc:3055] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:10:10.785337 11592 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:10:10.786535 11592 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.788826 11592 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:10.794358 11774 raft_consensus.cc:697] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:10.795593 11774 consensus_queue.cc:237] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.805487 11340 catalog_manager.cc:5697] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 2 to 3, leader changed from edea0c9ac6854273961ab944d9dd4664 (127.11.11.129) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 3 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:10.806799 11340 catalog_manager.cc:5697] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 2 to 3, leader changed from edea0c9ac6854273961ab944d9dd4664 (127.11.11.129) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 3 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:10.876638 11592 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 4.
I20260709 12:10:10.876642 11445 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 4.
I20260709 12:10:10.876680 11519 raft_consensus.cc:515] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:10:10.878716 11484 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:10.879326 11519 leader_election.cc:290] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 4 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:10.880388 11774 raft_consensus.cc:2804] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:10:10.881222 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:10.881239 11774 raft_consensus.cc:697] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:10.881476 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11497649fc7f46f889043a6069fe0db4" candidate_uuid: "94ef9c5a5b0142d68529b466758bb9ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:10.882038 11445 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:10.882242 11592 raft_consensus.cc:3055] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:10:10.883533 11592 raft_consensus.cc:740] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:10:10.882732 11774 consensus_queue.cc:237] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:10.884509 11592 consensus_queue.cc:260] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:10:10.885934 11592 raft_consensus.cc:3060] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:10.892796 11340 catalog_manager.cc:5697] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 3 to 4, leader changed from 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 4 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:10.976715 11592 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 4.
I20260709 12:10:10.976606 11445 raft_consensus.cc:2468] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ef9c5a5b0142d68529b466758bb9ce in term 4.
I20260709 12:10:10.978698 11484 leader_election.cc:304] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:10.979800 11778 raft_consensus.cc:2804] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:10:10.980360 11778 raft_consensus.cc:697] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Becoming Leader. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:10.981376 11778 consensus_queue.cc:237] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } }
I20260709 12:10:10.990160 11340 catalog_manager.cc:5697] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce reported cstate change: term changed from 3 to 4, leader changed from 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131) to 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130). New cstate: current_term: 4 leader_uuid: "94ef9c5a5b0142d68529b466758bb9ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:11.204088 11445 raft_consensus.cc:1275] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:11.205574 11778 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:11.218351 11592 raft_consensus.cc:1275] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:11.220410 11784 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:11.233245 11592 raft_consensus.cc:1275] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:11.233945 11445 raft_consensus.cc:1275] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:11.235332 11784 consensus_queue.cc:1048] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:11.235559 11774 consensus_queue.cc:1048] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:11.251053 11594 raft_consensus.cc:1275] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:11.256628 11778 consensus_queue.cc:1048] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:11.272581 11444 raft_consensus.cc:1275] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:11.274391 11774 consensus_queue.cc:1048] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:10:11.285158 11592 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:11.287011 11784 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:11.301539 11444 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:11.303858 11784 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:11.503127 11444 raft_consensus.cc:1275] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:11.505035 11784 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:11.516736 11592 raft_consensus.cc:1275] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Refusing update from remote peer 94ef9c5a5b0142d68529b466758bb9ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:11.518867 11784 consensus_queue.cc:1048] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:13.507972 11310 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:10:13.509122 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:13.517706 11519 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:52082
I20260709 12:10:13.518646 11519 raft_consensus.cc:606] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Received request to transfer leadership to TS 8f56f7c1df644b8ca2c123f6e4bc8fe5
I20260709 12:10:13.522567 11519 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:52082
I20260709 12:10:13.523200 11519 raft_consensus.cc:606] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Received request to transfer leadership to TS 8f56f7c1df644b8ca2c123f6e4bc8fe5
I20260709 12:10:13.527096 11519 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:52082
I20260709 12:10:13.527733 11519 raft_consensus.cc:606] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Received request to transfer leadership to TS edea0c9ac6854273961ab944d9dd4664
I20260709 12:10:13.530565 11519 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4082122061a440c69a7019e8a5382b08"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:52082
I20260709 12:10:13.531147 11519 raft_consensus.cc:606] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Received request to transfer leadership to TS edea0c9ac6854273961ab944d9dd4664
I20260709 12:10:13.533952 11519 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
mode: GRACEFUL
new_leader_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:52082
I20260709 12:10:13.534590 11519 raft_consensus.cc:606] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Received request to transfer leadership to TS edea0c9ac6854273961ab944d9dd4664
I20260709 12:10:13.535807 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:10:13.536325 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:13.621591 11784 raft_consensus.cc:993] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:10:13.622313 11778 raft_consensus.cc:1081] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Signalling peer edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:10:13.624512 11444 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4082122061a440c69a7019e8a5382b08"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:47708
I20260709 12:10:13.625242 11444 raft_consensus.cc:493] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:13.625798 11444 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:13.647945 11778 raft_consensus.cc:993] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:10:13.648489 11784 raft_consensus.cc:1081] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Signalling peer edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:10:13.651051 11445 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:47708
I20260709 12:10:13.651638 11445 raft_consensus.cc:493] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:13.651947 11445 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:10:13.720479 11444 raft_consensus.cc:515] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.724465 11444 leader_election.cc:290] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 3 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:13.726123 11592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:13.726679 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4082122061a440c69a7019e8a5382b08" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:13.726984 11592 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:13.727463 11519 raft_consensus.cc:3055] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:13.727784 11519 raft_consensus.cc:740] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:13.728561 11519 consensus_queue.cc:260] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.729575 11519 raft_consensus.cc:3060] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:13.768215 11778 raft_consensus.cc:993] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:10:13.769737 11784 raft_consensus.cc:1081] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Signalling peer edea0c9ac6854273961ab944d9dd4664 to start an election
I20260709 12:10:13.771901 11443 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f"
dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
 from {username='slave'} at 127.0.0.1:47708
I20260709 12:10:13.772857 11443 raft_consensus.cc:493] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:13.773382 11443 raft_consensus.cc:3060] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:13.825731 11445 raft_consensus.cc:515] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.826938 11592 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 3.
I20260709 12:10:13.828147 11445 leader_election.cc:290] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 5 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:13.829102 11410 leader_election.cc:304] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:13.829968 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:13.830726 11594 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:10:13.831698 11519 raft_consensus.cc:2468] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 3.
I20260709 12:10:13.833966 11806 raft_consensus.cc:2804] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:13.834018 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f93991cb0fa74f50980b9d0abb8afb0b" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:13.834741 11519 raft_consensus.cc:3055] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:10:13.835121 11519 raft_consensus.cc:740] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:13.835907 11806 raft_consensus.cc:697] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 3 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:13.836261 11519 consensus_queue.cc:260] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.836884 11806 consensus_queue.cc:237] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.837956 11519 raft_consensus.cc:3060] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:10:13.846206 11340 catalog_manager.cc:5697] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 2 to 3, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:13.856680 11805 raft_consensus.cc:993] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:13.858330 11805 raft_consensus.cc:1081] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Signalling peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:13.860837 11592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:38196
I20260709 12:10:13.861528 11592 raft_consensus.cc:493] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:13.861985 11592 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:13.922214 11519 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 5.
I20260709 12:10:13.922227 11443 raft_consensus.cc:515] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.922214 11594 raft_consensus.cc:2468] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 5.
I20260709 12:10:13.925194 11443 leader_election.cc:290] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Requested vote from peers 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643), 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131:46173)
I20260709 12:10:13.926642 11593 raft_consensus.cc:1240] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 5 FOLLOWER]: Rejecting Update request from peer 94ef9c5a5b0142d68529b466758bb9ce for earlier term 4. Current term is 5. Ops: []
I20260709 12:10:13.926796 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:13.927201 11410 leader_election.cc:304] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:13.928903 11594 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8f4cd3211445cf90a682b27d40607f" candidate_uuid: "edea0c9ac6854273961ab944d9dd4664" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
I20260709 12:10:13.927500 11519 raft_consensus.cc:3055] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:13.929664 11594 raft_consensus.cc:3060] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:13.929834 11519 raft_consensus.cc:740] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:13.930074 11806 raft_consensus.cc:2804] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:10:13.930698 11806 raft_consensus.cc:697] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 5 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:13.931829 11519 consensus_queue.cc:260] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.931681 11806 consensus_queue.cc:237] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:13.933081 11519 raft_consensus.cc:3060] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:13.937505 11805 raft_consensus.cc:993] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: : Instructing follower 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:13.938123 11805 raft_consensus.cc:1081] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Signalling peer 8f56f7c1df644b8ca2c123f6e4bc8fe5 to start an election
I20260709 12:10:13.940090 11593 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6"
dest_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5"
 from {username='slave'} at 127.0.0.1:38196
I20260709 12:10:13.940677 11593 raft_consensus.cc:493] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:13.941093 11593 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:13.940971 11340 catalog_manager.cc:5697] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 4 to 5, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:14.015630 11519 raft_consensus.cc:2468] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:10:14.015798 11592 raft_consensus.cc:515] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.018067 11411 leader_election.cc:304] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ef9c5a5b0142d68529b466758bb9ce, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:14.018985 11592 leader_election.cc:290] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 4 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:10:14.020572 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:14.021282 11443 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:14.021610 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "207b53cef5cd41b488fe59d21bf4f010" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:14.022142 11519 raft_consensus.cc:3055] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:10:14.022440 11519 raft_consensus.cc:740] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:14.023254 11519 consensus_queue.cc:260] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.023983 11806 raft_consensus.cc:2804] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:14.024447 11519 raft_consensus.cc:3060] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:14.024756 11806 raft_consensus.cc:697] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Becoming Leader. State: Replica: edea0c9ac6854273961ab944d9dd4664, State: Running, Role: LEADER
I20260709 12:10:14.025915 11806 consensus_queue.cc:237] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.037569 11340 catalog_manager.cc:5697] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 reported cstate change: term changed from 1 to 2, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to edea0c9ac6854273961ab944d9dd4664 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "edea0c9ac6854273961ab944d9dd4664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:14.110044 11443 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 4.
I20260709 12:10:14.110071 11519 raft_consensus.cc:2468] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 4.
I20260709 12:10:14.110071 11594 raft_consensus.cc:2468] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edea0c9ac6854273961ab944d9dd4664 in term 2.
I20260709 12:10:14.111588 11593 raft_consensus.cc:515] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.112381 11558 leader_election.cc:304] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:14.113564 11771 raft_consensus.cc:2804] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:10:14.114559 11593 leader_election.cc:290] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 3 election: Requested vote from peers edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679), 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643)
I20260709 12:10:14.115739 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "94ef9c5a5b0142d68529b466758bb9ce"
I20260709 12:10:14.115630 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2979f5770c1d486f91f7690339c626d6" candidate_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "edea0c9ac6854273961ab944d9dd4664"
I20260709 12:10:14.116192 11771 raft_consensus.cc:697] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:10:14.116366 11519 raft_consensus.cc:3055] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:14.116539 11443 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:14.116791 11519 raft_consensus.cc:740] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 94ef9c5a5b0142d68529b466758bb9ce, State: Running, Role: LEADER
I20260709 12:10:14.117295 11771 consensus_queue.cc:237] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.117583 11519 consensus_queue.cc:260] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.118887 11519 raft_consensus.cc:3060] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:14.125701 11443 raft_consensus.cc:2468] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 3.
I20260709 12:10:14.126170 11519 raft_consensus.cc:2468] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f56f7c1df644b8ca2c123f6e4bc8fe5 in term 3.
I20260709 12:10:14.127135 11558 leader_election.cc:304] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f56f7c1df644b8ca2c123f6e4bc8fe5, edea0c9ac6854273961ab944d9dd4664; no voters: 
I20260709 12:10:14.128054 11771 raft_consensus.cc:2804] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:14.128803 11771 raft_consensus.cc:697] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Becoming Leader. State: Replica: 8f56f7c1df644b8ca2c123f6e4bc8fe5, State: Running, Role: LEADER
I20260709 12:10:14.127671 11340 catalog_manager.cc:5697] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 3 to 4, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 4 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:14.129900 11771 consensus_queue.cc:237] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } }
I20260709 12:10:14.143110 11340 catalog_manager.cc:5697] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 reported cstate change: term changed from 2 to 3, leader changed from 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130) to 8f56f7c1df644b8ca2c123f6e4bc8fe5 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:14.255767 11593 raft_consensus.cc:1275] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:14.257519 11806 consensus_queue.cc:1048] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:10:14.272178 11519 raft_consensus.cc:1275] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:14.274135 11820 consensus_queue.cc:1048] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:10:14.319280 11593 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 5 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:10:14.320986 11820 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:10:14.333642 11519 raft_consensus.cc:1275] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 5 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:10:14.335930 11807 consensus_queue.cc:1048] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:10:14.461324 11519 raft_consensus.cc:1275] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:14.462831 11807 consensus_queue.cc:1048] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:14.475956 11593 raft_consensus.cc:1275] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Refusing update from remote peer edea0c9ac6854273961ab944d9dd4664: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:14.477648 11820 consensus_queue.cc:1048] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f56f7c1df644b8ca2c123f6e4bc8fe5" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:14.536087 11443 raft_consensus.cc:1275] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:14.537871 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:14.537801 11771 consensus_queue.cc:1048] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:10:14.545554 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:14.546353 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:14.550474 11519 raft_consensus.cc:1275] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:14.552067 11815 consensus_queue.cc:1048] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:14.676911 11443 raft_consensus.cc:1275] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:14.678650 11829 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edea0c9ac6854273961ab944d9dd4664" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:10:14.680444 11519 raft_consensus.cc:1275] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Refusing update from remote peer 8f56f7c1df644b8ca2c123f6e4bc8fe5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:14.681915 11829 consensus_queue.cc:1048] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ef9c5a5b0142d68529b466758bb9ce" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:10:15.547791 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:15.554780 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:15.555338 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:16.556684 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:16.562000 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:16.562546 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:17.563876 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:17.569264 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:17.569876 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:18.570977 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:18.576278 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:18.576802 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:19.578190 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:19.584199 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:19.584800 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:20.661260 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:20.668336 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:20.668869 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:10:21.550114 11466 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6060 after lost signal to thread 11313
W20260709 12:10:21.551376 11466 debug-util.cc:398] Leaking SignalData structure 0x7b08000e6740 after lost signal to thread 11376
W20260709 12:10:21.552595 11466 debug-util.cc:398] Leaking SignalData structure 0x7b0800192940 after lost signal to thread 11469
W20260709 12:10:21.553740 11466 debug-util.cc:398] Leaking SignalData structure 0x7b08001ff560 after lost signal to thread 11544
W20260709 12:10:21.554768 11466 debug-util.cc:398] Leaking SignalData structure 0x7b08000aa2a0 after lost signal to thread 11618
I20260709 12:10:21.670349 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:21.679237 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:21.679939 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:22.682432 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:22.691746 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:22.692504 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:23.734174 11310 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:10:23.734994 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
I20260709 12:10:23.792188 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:10:23.793419 11310 tablet_replica.cc:333] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.794807 11310 raft_consensus.cc:2243] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:23.796002 11310 raft_consensus.cc:2272] T 207b53cef5cd41b488fe59d21bf4f010 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.799853 11310 tablet_replica.cc:333] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.800701 11310 raft_consensus.cc:2243] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:23.801353 11310 raft_consensus.cc:2272] T 1b44dbd6f6a34f7590ea121e2639b12a P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.804064 11310 tablet_replica.cc:333] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.804962 11310 raft_consensus.cc:2243] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:10:23.806175 11310 raft_consensus.cc:2272] T 4082122061a440c69a7019e8a5382b08 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.808727 11310 tablet_replica.cc:333] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.810070 11310 raft_consensus.cc:2243] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:10:23.811512 11310 raft_consensus.cc:2272] T f93991cb0fa74f50980b9d0abb8afb0b P edea0c9ac6854273961ab944d9dd4664 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.814678 11310 tablet_replica.cc:333] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.815610 11310 raft_consensus.cc:2243] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:23.816329 11310 raft_consensus.cc:2272] T 2979f5770c1d486f91f7690339c626d6 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.819429 11310 tablet_replica.cc:333] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.820393 11310 raft_consensus.cc:2243] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:23.821019 11310 raft_consensus.cc:2272] T b9823d449d634ad08eadedf5c907bd92 P edea0c9ac6854273961ab944d9dd4664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.823921 11310 tablet_replica.cc:333] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.824776 11310 raft_consensus.cc:2243] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:10:23.826110 11310 raft_consensus.cc:2272] T 3b8f4cd3211445cf90a682b27d40607f P edea0c9ac6854273961ab944d9dd4664 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.829201 11310 tablet_replica.cc:333] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.830029 11310 raft_consensus.cc:2243] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:23.830642 11310 raft_consensus.cc:2272] T aac3cbc7a3564854b3ce7d2e74d902c8 P edea0c9ac6854273961ab944d9dd4664 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:23.833457 11310 tablet_replica.cc:333] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664: stopping tablet replica
I20260709 12:10:23.834182 11310 raft_consensus.cc:2243] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:23.834791 11310 raft_consensus.cc:2272] T 11497649fc7f46f889043a6069fe0db4 P edea0c9ac6854273961ab944d9dd4664 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:10:23.962942 11558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38679: connect: Connection refused (error 111)
W20260709 12:10:23.968124 11558 consensus_peers.cc:597] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 -> Peer edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679): Couldn't send request to peer edea0c9ac6854273961ab944d9dd4664. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:10:23.968642 11558 consensus_peers.cc:597] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 -> Peer edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679): Couldn't send request to peer edea0c9ac6854273961ab944d9dd4664. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:24.033743 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
I20260709 12:10:24.056576 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
W20260709 12:10:24.076547 11484 consensus_peers.cc:597] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce -> Peer edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679): Couldn't send request to peer edea0c9ac6854273961ab944d9dd4664. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:10:24.095852 11484 consensus_peers.cc:597] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce -> Peer edea0c9ac6854273961ab944d9dd4664 (127.11.11.129:38679): Couldn't send request to peer edea0c9ac6854273961ab944d9dd4664. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:24.100472 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:10:24.101346 11310 tablet_replica.cc:333] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.102423 11310 raft_consensus.cc:2243] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.103246 11310 raft_consensus.cc:2272] T 1b44dbd6f6a34f7590ea121e2639b12a P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.105072 11310 tablet_replica.cc:333] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.105541 11310 raft_consensus.cc:2243] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.106032 11310 raft_consensus.cc:2272] T 207b53cef5cd41b488fe59d21bf4f010 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.107761 11310 tablet_replica.cc:333] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.108214 11310 raft_consensus.cc:2243] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.108595 11310 raft_consensus.cc:2272] T 4082122061a440c69a7019e8a5382b08 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.110327 11310 tablet_replica.cc:333] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.110786 11310 raft_consensus.cc:2243] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.111167 11310 raft_consensus.cc:2272] T f93991cb0fa74f50980b9d0abb8afb0b P 94ef9c5a5b0142d68529b466758bb9ce [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.112906 11310 tablet_replica.cc:333] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.113373 11310 raft_consensus.cc:2243] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.113819 11310 raft_consensus.cc:2272] T 2979f5770c1d486f91f7690339c626d6 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.115473 11310 tablet_replica.cc:333] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.115913 11310 raft_consensus.cc:2243] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.116266 11310 raft_consensus.cc:2272] T b9823d449d634ad08eadedf5c907bd92 P 94ef9c5a5b0142d68529b466758bb9ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.117913 11310 tablet_replica.cc:333] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.118378 11310 raft_consensus.cc:2243] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.118774 11310 raft_consensus.cc:2272] T 3b8f4cd3211445cf90a682b27d40607f P 94ef9c5a5b0142d68529b466758bb9ce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.120476 11310 tablet_replica.cc:333] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.120889 11310 raft_consensus.cc:2243] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.121734 11310 raft_consensus.cc:2272] T 11497649fc7f46f889043a6069fe0db4 P 94ef9c5a5b0142d68529b466758bb9ce [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.123472 11310 tablet_replica.cc:333] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce: stopping tablet replica
I20260709 12:10:24.123883 11310 raft_consensus.cc:2243] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.124632 11310 raft_consensus.cc:2272] T aac3cbc7a3564854b3ce7d2e74d902c8 P 94ef9c5a5b0142d68529b466758bb9ce [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:10:24.146889 11560 consensus_peers.cc:597] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 -> Peer 94ef9c5a5b0142d68529b466758bb9ce (127.11.11.130:45643): Couldn't send request to peer 94ef9c5a5b0142d68529b466758bb9ce. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:45643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:24.158582 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
I20260709 12:10:24.181555 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
I20260709 12:10:24.216274 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:10:24.217221 11310 tablet_replica.cc:333] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.218187 11310 raft_consensus.cc:2243] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.219282 11310 raft_consensus.cc:2272] T 2979f5770c1d486f91f7690339c626d6 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.221549 11310 tablet_replica.cc:333] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.222242 11310 raft_consensus.cc:2243] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.222783 11310 raft_consensus.cc:2272] T f93991cb0fa74f50980b9d0abb8afb0b P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.225101 11310 tablet_replica.cc:333] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.225755 11310 raft_consensus.cc:2243] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.226291 11310 raft_consensus.cc:2272] T 4082122061a440c69a7019e8a5382b08 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.228189 11310 tablet_replica.cc:333] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.228789 11310 raft_consensus.cc:2243] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.229846 11310 raft_consensus.cc:2272] T 207b53cef5cd41b488fe59d21bf4f010 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.232241 11310 tablet_replica.cc:333] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.233045 11310 raft_consensus.cc:2243] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.233669 11310 raft_consensus.cc:2272] T 1b44dbd6f6a34f7590ea121e2639b12a P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.236284 11310 tablet_replica.cc:333] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.237025 11310 raft_consensus.cc:2243] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.237615 11310 raft_consensus.cc:2272] T 3b8f4cd3211445cf90a682b27d40607f P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.239519 11310 tablet_replica.cc:333] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.240036 11310 raft_consensus.cc:2243] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.240909 11310 raft_consensus.cc:2272] T b9823d449d634ad08eadedf5c907bd92 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.242977 11310 tablet_replica.cc:333] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.243512 11310 raft_consensus.cc:2243] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.244006 11310 raft_consensus.cc:2272] T aac3cbc7a3564854b3ce7d2e74d902c8 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.245927 11310 tablet_replica.cc:333] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5: stopping tablet replica
I20260709 12:10:24.246433 11310 raft_consensus.cc:2243] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:24.246908 11310 raft_consensus.cc:2272] T 11497649fc7f46f889043a6069fe0db4 P 8f56f7c1df644b8ca2c123f6e4bc8fe5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.280143 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:10:24.304647 11310 master.cc:562] Master@127.11.11.190:43031 shutting down...
I20260709 12:10:24.350698 11397 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260709 12:10:24.452111 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:10:24.452797 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:24.453168 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2707573fdb8c4ccfa794bb864d6ad3b8: stopping tablet replica
I20260709 12:10:24.476588 11310 master.cc:584] Master@127.11.11.190:43031 shutdown complete.
8f56f7c1df644b8ca2c123f6e4bc8fe5  1
94ef9c5a5b0142d68529b466758bb9ce  4
edea0c9ac6854273961ab944d9dd4664  4
8f56f7c1df644b8ca2c123f6e4bc8fe5  3
94ef9c5a5b0142d68529b466758bb9ce  3
edea0c9ac6854273961ab944d9dd4664  3
8f56f7c1df644b8ca2c123f6e4bc8fe5  1
94ef9c5a5b0142d68529b466758bb9ce  8
edea0c9ac6854273961ab944d9dd4664  0
8f56f7c1df644b8ca2c123f6e4bc8fe5  3
94ef9c5a5b0142d68529b466758bb9ce  3
edea0c9ac6854273961ab944d9dd4664  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (32129 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:10:24.521586 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:39185
I20260709 12:10:24.522884 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:24.529965 11844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:24.529997 11843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:24.530393 11846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:24.531472 11310 server_base.cc:1061] running on GCE node
I20260709 12:10:24.532500 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:24.532681 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:24.532799 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599024532788 us; error 0 us; skew 500 ppm
I20260709 12:10:24.533265 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:24.536516 11310 webserver.cc:533] Webserver started at http://127.11.11.190:33543/ using document root <none> and password file <none>
I20260709 12:10:24.537020 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:24.537182 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:24.537431 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:24.538668 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "32474939a0ce4c568817895ac69ed49f"
format_stamp: "Formatted at 2026-07-09 12:10:24 on dist-test-slave-4qvp"
I20260709 12:10:24.543713 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:10:24.547518 11851 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:24.548422 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 12:10:24.548899 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "32474939a0ce4c568817895ac69ed49f"
format_stamp: "Formatted at 2026-07-09 12:10:24 on dist-test-slave-4qvp"
I20260709 12:10:24.549221 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:24.585417 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:24.586874 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:24.642293 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:39185
I20260709 12:10:24.642449 11902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:39185 every 8 connection(s)
I20260709 12:10:24.647131 11903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:24.659731 11903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f: Bootstrap starting.
I20260709 12:10:24.665748 11903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:24.670985 11903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f: No bootstrap required, opened a new log
I20260709 12:10:24.673631 11903 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32474939a0ce4c568817895ac69ed49f" member_type: VOTER }
I20260709 12:10:24.674196 11903 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:24.674463 11903 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32474939a0ce4c568817895ac69ed49f, State: Initialized, Role: FOLLOWER
I20260709 12:10:24.675127 11903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32474939a0ce4c568817895ac69ed49f" member_type: VOTER }
I20260709 12:10:24.675653 11903 raft_consensus.cc:399] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:10:24.675984 11903 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:10:24.676293 11903 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:24.681041 11903 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32474939a0ce4c568817895ac69ed49f" member_type: VOTER }
I20260709 12:10:24.681732 11903 leader_election.cc:304] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32474939a0ce4c568817895ac69ed49f; no voters: 
I20260709 12:10:24.683117 11903 leader_election.cc:290] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:10:24.683578 11906 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:24.685138 11906 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 1 LEADER]: Becoming Leader. State: Replica: 32474939a0ce4c568817895ac69ed49f, State: Running, Role: LEADER
I20260709 12:10:24.686002 11906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32474939a0ce4c568817895ac69ed49f" member_type: VOTER }
I20260709 12:10:24.686502 11903 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:10:24.691674 11907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32474939a0ce4c568817895ac69ed49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32474939a0ce4c568817895ac69ed49f" member_type: VOTER } }
I20260709 12:10:24.693161 11907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [sys.catalog]: This master's current role is: LEADER
I20260709 12:10:24.693285 11908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32474939a0ce4c568817895ac69ed49f. Latest consensus state: current_term: 1 leader_uuid: "32474939a0ce4c568817895ac69ed49f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32474939a0ce4c568817895ac69ed49f" member_type: VOTER } }
I20260709 12:10:24.693887 11908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [sys.catalog]: This master's current role is: LEADER
I20260709 12:10:24.703395 11914 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:10:24.709151 11914 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:10:24.711236 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:10:24.720118 11914 catalog_manager.cc:1383] Generated new cluster ID: 70f4a7ed6b58463fa0c1553a8fb29a2b
I20260709 12:10:24.720496 11914 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:10:24.745273 11914 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:10:24.746665 11914 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:10:24.762568 11914 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f: Generated new TSK 0
I20260709 12:10:24.763250 11914 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:10:24.779331 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:24.786676 11924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:24.788597 11925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:24.790359 11310 server_base.cc:1061] running on GCE node
W20260709 12:10:24.790483 11927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:24.791867 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:24.792096 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:24.792223 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599024792211 us; error 0 us; skew 500 ppm
I20260709 12:10:24.792797 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:24.795617 11310 webserver.cc:533] Webserver started at http://127.11.11.129:36119/ using document root <none> and password file <none>
I20260709 12:10:24.796121 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:24.796289 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:24.796537 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:24.797817 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "3c9b406eb70142dea09afbd266bd696f"
format_stamp: "Formatted at 2026-07-09 12:10:24 on dist-test-slave-4qvp"
I20260709 12:10:24.802999 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20260709 12:10:24.807529 11932 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:24.808544 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20260709 12:10:24.809000 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "3c9b406eb70142dea09afbd266bd696f"
format_stamp: "Formatted at 2026-07-09 12:10:24 on dist-test-slave-4qvp"
I20260709 12:10:24.809387 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:24.832824 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:24.834388 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:24.836261 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:10:24.838970 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:10:24.839239 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:24.839504 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:10:24.839711 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:24.891004 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:34093
I20260709 12:10:24.891173 11994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:34093 every 8 connection(s)
I20260709 12:10:24.896868 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:24.906428 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:24.907253 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:24.907621 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:24.910374 11310 server_base.cc:1061] running on GCE node
I20260709 12:10:24.912415 11995 heartbeater.cc:344] Connected to a master server at 127.11.11.190:39185
I20260709 12:10:24.912549 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:24.912822 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:24.912926 11995 heartbeater.cc:461] Registering TS with master...
I20260709 12:10:24.913039 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599024913015 us; error 0 us; skew 500 ppm
I20260709 12:10:24.913914 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:24.913988 11995 heartbeater.cc:507] Master 127.11.11.190:39185 requested a full tablet report, sending...
I20260709 12:10:24.917109 11868 ts_manager.cc:194] Registered new tserver with Master: 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093)
I20260709 12:10:24.917367 11310 webserver.cc:533] Webserver started at http://127.11.11.130:32909/ using document root <none> and password file <none>
I20260709 12:10:24.918339 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:24.918637 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:24.918990 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:24.919509 11868 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49980
I20260709 12:10:24.920581 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "997e54add8aa4783a09f5aa876db8fb3"
format_stamp: "Formatted at 2026-07-09 12:10:24 on dist-test-slave-4qvp"
I20260709 12:10:24.926955 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.004s
I20260709 12:10:24.930980 12007 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:24.932096 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 12:10:24.932430 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "997e54add8aa4783a09f5aa876db8fb3"
format_stamp: "Formatted at 2026-07-09 12:10:24 on dist-test-slave-4qvp"
I20260709 12:10:24.932744 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:24.954295 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:24.955899 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:24.957758 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:10:24.960598 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:10:24.960815 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:24.961201 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:10:24.961390 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:25.014226 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:46343
I20260709 12:10:25.014338 12069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:46343 every 8 connection(s)
I20260709 12:10:25.020162 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:25.029500 12073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:25.030095 12074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:25.032985 12070 heartbeater.cc:344] Connected to a master server at 127.11.11.190:39185
I20260709 12:10:25.033552 12070 heartbeater.cc:461] Registering TS with master...
W20260709 12:10:25.034096 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:25.034328 11310 server_base.cc:1061] running on GCE node
I20260709 12:10:25.034827 12070 heartbeater.cc:507] Master 127.11.11.190:39185 requested a full tablet report, sending...
I20260709 12:10:25.035610 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:25.035825 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:25.036055 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599025036035 us; error 0 us; skew 500 ppm
I20260709 12:10:25.036710 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:25.037480 11868 ts_manager.cc:194] Registered new tserver with Master: 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:25.039546 11868 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49986
I20260709 12:10:25.039912 11310 webserver.cc:533] Webserver started at http://127.11.11.131:40775/ using document root <none> and password file <none>
I20260709 12:10:25.040594 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:25.040868 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:25.041203 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:25.042549 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
format_stamp: "Formatted at 2026-07-09 12:10:25 on dist-test-slave-4qvp"
I20260709 12:10:25.047932 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:10:25.052062 12081 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:25.053262 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:10:25.053840 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
format_stamp: "Formatted at 2026-07-09 12:10:25 on dist-test-slave-4qvp"
I20260709 12:10:25.054157 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:25.075033 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:25.076356 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:25.078037 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:10:25.080538 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:10:25.080796 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:25.081056 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:10:25.081238 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:25.128324 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:37823
I20260709 12:10:25.128469 12143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:37823 every 8 connection(s)
I20260709 12:10:25.144687 12144 heartbeater.cc:344] Connected to a master server at 127.11.11.190:39185
I20260709 12:10:25.145226 12144 heartbeater.cc:461] Registering TS with master...
I20260709 12:10:25.146375 12144 heartbeater.cc:507] Master 127.11.11.190:39185 requested a full tablet report, sending...
I20260709 12:10:25.148713 11868 ts_manager.cc:194] Registered new tserver with Master: bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.149282 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017701309s
I20260709 12:10:25.150357 11868 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50002
I20260709 12:10:25.182001 11868 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50010:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:10:25.185293 11868 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:10:25.268960 12035 tablet_service.cc:1511] Processing CreateTablet for tablet 9751272f5592489a95216b06be6793ff (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:10:25.270483 12035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9751272f5592489a95216b06be6793ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.285516 12034 tablet_service.cc:1511] Processing CreateTablet for tablet c8d3b14c9ba14a48a6f7a51c571f49cc (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:10:25.287091 12034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d3b14c9ba14a48a6f7a51c571f49cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.287640 12109 tablet_service.cc:1511] Processing CreateTablet for tablet 9751272f5592489a95216b06be6793ff (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:10:25.289429 12109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9751272f5592489a95216b06be6793ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.289659 12030 tablet_service.cc:1511] Processing CreateTablet for tablet edf4111c57784cda9fd2e092fbd68847 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:10:25.291174 12030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf4111c57784cda9fd2e092fbd68847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.290733 12104 tablet_service.cc:1511] Processing CreateTablet for tablet edf4111c57784cda9fd2e092fbd68847 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:10:25.292378 12104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf4111c57784cda9fd2e092fbd68847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.292752 12031 tablet_service.cc:1511] Processing CreateTablet for tablet b1836d0ddf8c40b0a3961a899546dabb (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:10:25.294147 12031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1836d0ddf8c40b0a3961a899546dabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.294940 12029 tablet_service.cc:1511] Processing CreateTablet for tablet c722d809fe204d3f86b3045aff4a7fab (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:10:25.295159 12105 tablet_service.cc:1511] Processing CreateTablet for tablet b1836d0ddf8c40b0a3961a899546dabb (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:10:25.296283 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c722d809fe204d3f86b3045aff4a7fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.296814 12105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1836d0ddf8c40b0a3961a899546dabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.285516 12108 tablet_service.cc:1511] Processing CreateTablet for tablet c8d3b14c9ba14a48a6f7a51c571f49cc (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:10:25.299081 12106 tablet_service.cc:1511] Processing CreateTablet for tablet ac5c5d0844a545b4b511b3fd6239cb53 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:10:25.299805 12108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d3b14c9ba14a48a6f7a51c571f49cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.300629 12106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c5d0844a545b4b511b3fd6239cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.289676 12033 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3e7f70af184c4c8d2d2263e5a2c033 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:10:25.302623 12033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3e7f70af184c4c8d2d2263e5a2c033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.303072 12107 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3e7f70af184c4c8d2d2263e5a2c033 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:10:25.304760 12107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3e7f70af184c4c8d2d2263e5a2c033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.290855 12032 tablet_service.cc:1511] Processing CreateTablet for tablet ac5c5d0844a545b4b511b3fd6239cb53 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:10:25.305938 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c5d0844a545b4b511b3fd6239cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.329859 12103 tablet_service.cc:1511] Processing CreateTablet for tablet c722d809fe204d3f86b3045aff4a7fab (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:10:25.332100 12103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c722d809fe204d3f86b3045aff4a7fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.358424 12158 tablet_bootstrap.cc:492] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.360682 11960 tablet_service.cc:1511] Processing CreateTablet for tablet 9751272f5592489a95216b06be6793ff (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:10:25.363976 11959 tablet_service.cc:1511] Processing CreateTablet for tablet c8d3b14c9ba14a48a6f7a51c571f49cc (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:10:25.364876 11954 tablet_service.cc:1511] Processing CreateTablet for tablet c722d809fe204d3f86b3045aff4a7fab (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:10:25.365389 11960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9751272f5592489a95216b06be6793ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.367017 11956 tablet_service.cc:1511] Processing CreateTablet for tablet b1836d0ddf8c40b0a3961a899546dabb (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:10:25.367960 12107 tablet_service.cc:1511] Processing CreateTablet for tablet ce32af3d4f724f0db85f77eb35ba228c (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:10:25.368634 11956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1836d0ddf8c40b0a3961a899546dabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.369273 12107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce32af3d4f724f0db85f77eb35ba228c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.369956 11959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8d3b14c9ba14a48a6f7a51c571f49cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.370836 11957 tablet_service.cc:1511] Processing CreateTablet for tablet ac5c5d0844a545b4b511b3fd6239cb53 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:10:25.372243 11957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c5d0844a545b4b511b3fd6239cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.374022 11954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c722d809fe204d3f86b3045aff4a7fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.374902 11955 tablet_service.cc:1511] Processing CreateTablet for tablet edf4111c57784cda9fd2e092fbd68847 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:10:25.377285 11958 tablet_service.cc:1511] Processing CreateTablet for tablet 5a3e7f70af184c4c8d2d2263e5a2c033 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:10:25.377930 11952 tablet_service.cc:1511] Processing CreateTablet for tablet a6401f8748164c969e0dfe96d0cbf8dd (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:10:25.379724 12032 tablet_service.cc:1511] Processing CreateTablet for tablet a6401f8748164c969e0dfe96d0cbf8dd (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:10:25.379830 11953 tablet_service.cc:1511] Processing CreateTablet for tablet ce32af3d4f724f0db85f77eb35ba228c (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:10:25.381050 12032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6401f8748164c969e0dfe96d0cbf8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.382479 11955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edf4111c57784cda9fd2e092fbd68847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.384074 11958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3e7f70af184c4c8d2d2263e5a2c033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.385537 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6401f8748164c969e0dfe96d0cbf8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.388219 11953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce32af3d4f724f0db85f77eb35ba228c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.389008 12109 tablet_service.cc:1511] Processing CreateTablet for tablet a6401f8748164c969e0dfe96d0cbf8dd (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:10:25.390580 12109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6401f8748164c969e0dfe96d0cbf8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.398116 11951 tablet_service.cc:1511] Processing CreateTablet for tablet bae69c80fd414289b43cc35643e55ca4 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:10:25.398767 12035 tablet_service.cc:1511] Processing CreateTablet for tablet ce32af3d4f724f0db85f77eb35ba228c (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:10:25.399751 11951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae69c80fd414289b43cc35643e55ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.400027 12035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce32af3d4f724f0db85f77eb35ba228c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.408190 12159 tablet_bootstrap.cc:492] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:25.446710 12106 tablet_service.cc:1511] Processing CreateTablet for tablet bae69c80fd414289b43cc35643e55ca4 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:10:25.448297 12106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae69c80fd414289b43cc35643e55ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.446735 12158 tablet_bootstrap.cc:654] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.463929 12029 tablet_service.cc:1511] Processing CreateTablet for tablet bae69c80fd414289b43cc35643e55ca4 (DEFAULT_TABLE table=test-workload [id=4855fb3f635a477abffcab818f3299e8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:10:25.465447 12029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae69c80fd414289b43cc35643e55ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:25.474677 12160 tablet_bootstrap.cc:492] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.495273 12159 tablet_bootstrap.cc:654] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.514148 12158 tablet_bootstrap.cc:492] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:25.518360 12158 ts_tablet_manager.cc:1403] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.160s	user 0.024s	sys 0.013s
I20260709 12:10:25.522492 12158 raft_consensus.cc:359] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.523586 12158 raft_consensus.cc:385] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.526310 12158 raft_consensus.cc:740] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.527443 12158 consensus_queue.cc:260] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.530552 12160 tablet_bootstrap.cc:654] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.534850 12159 tablet_bootstrap.cc:492] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:25.535386 12158 ts_tablet_manager.cc:1434] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.005s
I20260709 12:10:25.535426 12159 ts_tablet_manager.cc:1403] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.128s	user 0.021s	sys 0.013s
I20260709 12:10:25.536332 12070 heartbeater.cc:499] Master 127.11.11.190:39185 was elected leader, sending a full tablet report...
I20260709 12:10:25.536662 12158 tablet_bootstrap.cc:492] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.538815 12159 raft_consensus.cc:359] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.539738 12159 raft_consensus.cc:385] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.540103 12159 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.541180 12159 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.544415 12158 tablet_bootstrap.cc:654] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.546342 12144 heartbeater.cc:499] Master 127.11.11.190:39185 was elected leader, sending a full tablet report...
I20260709 12:10:25.547077 12159 ts_tablet_manager.cc:1434] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:10:25.548563 12159 tablet_bootstrap.cc:492] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:25.557844 12159 tablet_bootstrap.cc:654] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.560106 12160 tablet_bootstrap.cc:492] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:25.560721 12160 ts_tablet_manager.cc:1403] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.086s	user 0.010s	sys 0.012s
I20260709 12:10:25.565032 12159 tablet_bootstrap.cc:492] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:25.565562 12159 ts_tablet_manager.cc:1403] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.003s
I20260709 12:10:25.566417 12158 tablet_bootstrap.cc:492] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:25.566992 12158 ts_tablet_manager.cc:1403] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.004s
I20260709 12:10:25.568594 12159 raft_consensus.cc:359] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.569388 12159 raft_consensus.cc:385] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.564563 12160 raft_consensus.cc:359] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.569753 12159 raft_consensus.cc:740] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.569985 12160 raft_consensus.cc:385] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.569799 12158 raft_consensus.cc:359] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.570549 12160 raft_consensus.cc:740] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.570829 12158 raft_consensus.cc:385] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.571211 12158 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.570889 12159 consensus_queue.cc:260] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.571777 12160 consensus_queue.cc:260] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.572008 12158 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.573516 12159 ts_tablet_manager.cc:1434] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260709 12:10:25.574657 12159 tablet_bootstrap.cc:492] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:25.577993 12158 ts_tablet_manager.cc:1434] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.005s
I20260709 12:10:25.579686 12158 tablet_bootstrap.cc:492] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.581125 12159 tablet_bootstrap.cc:654] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.584749 12164 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.585287 12164 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.598186 11995 heartbeater.cc:499] Master 127.11.11.190:39185 was elected leader, sending a full tablet report...
I20260709 12:10:25.598829 12158 tablet_bootstrap.cc:654] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.605212 12164 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093)
I20260709 12:10:25.605549 12158 tablet_bootstrap.cc:492] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:25.609200 12158 ts_tablet_manager.cc:1403] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.005s
I20260709 12:10:25.611215 12160 ts_tablet_manager.cc:1434] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.050s	user 0.010s	sys 0.011s
I20260709 12:10:25.610966 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:25.612033 12045 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 0.
I20260709 12:10:25.612358 12160 tablet_bootstrap.cc:492] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.612617 12158 raft_consensus.cc:359] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.613337 12158 raft_consensus.cc:385] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.613662 12158 raft_consensus.cc:740] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.608268 12159 tablet_bootstrap.cc:492] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:25.614506 12085 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:25.614924 12159 ts_tablet_manager.cc:1403] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.041s	user 0.013s	sys 0.018s
I20260709 12:10:25.615660 12164 raft_consensus.cc:2804] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:25.616282 12164 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:25.616663 12164 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.614509 12158 consensus_queue.cc:260] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.618306 12159 raft_consensus.cc:359] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.619058 12159 raft_consensus.cc:385] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.619369 12159 raft_consensus.cc:740] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.620214 12159 consensus_queue.cc:260] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.621724 12158 ts_tablet_manager.cc:1434] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:10:25.622996 12158 tablet_bootstrap.cc:492] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.626070 12164 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.627619 12163 raft_consensus.cc:493] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.628244 12163 raft_consensus.cc:515] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.630373 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:25.630746 12158 tablet_bootstrap.cc:654] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.631279 12045 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.642083 12163 leader_election.cc:290] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.643994 12158 tablet_bootstrap.cc:492] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:25.644699 12158 ts_tablet_manager.cc:1403] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.004s
I20260709 12:10:25.645435 12045 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 1.
I20260709 12:10:25.647147 12085 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:25.649214 12158 raft_consensus.cc:359] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.650313 12158 raft_consensus.cc:385] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.650750 12158 raft_consensus.cc:740] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.651851 12158 consensus_queue.cc:260] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.655118 12158 ts_tablet_manager.cc:1434] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.007s
I20260709 12:10:25.656613 12158 tablet_bootstrap.cc:492] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.659030 12164 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093)
I20260709 12:10:25.660347 12164 raft_consensus.cc:2804] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:25.660986 12164 raft_consensus.cc:697] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:25.664561 12160 tablet_bootstrap.cc:654] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.664639 12158 tablet_bootstrap.cc:654] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.668840 12164 consensus_queue.cc:237] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.676622 12159 ts_tablet_manager.cc:1434] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.061s	user 0.020s	sys 0.033s
I20260709 12:10:25.678078 12159 tablet_bootstrap.cc:492] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:25.680909 12160 tablet_bootstrap.cc:492] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:25.681841 12160 ts_tablet_manager.cc:1403] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.070s	user 0.005s	sys 0.012s
I20260709 12:10:25.685921 12159 tablet_bootstrap.cc:654] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.685454 12160 raft_consensus.cc:359] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.687224 12160 raft_consensus.cc:385] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.687559 12160 raft_consensus.cc:740] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.690343 12160 consensus_queue.cc:260] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.693353 12160 ts_tablet_manager.cc:1434] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.001s
I20260709 12:10:25.700623 12160 tablet_bootstrap.cc:492] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.706748 11970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:25.706996 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
W20260709 12:10:25.707376 11916 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:10:25.706072 12163 raft_consensus.cc:493] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:10:25.709053 12085 leader_election.cc:343] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.708914 12163 raft_consensus.cc:515] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
W20260709 12:10:25.710041 12085 leader_election.cc:343] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.711845 12163 leader_election.cc:290] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.717458 12159 tablet_bootstrap.cc:492] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:25.718310 12159 ts_tablet_manager.cc:1403] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.041s	user 0.008s	sys 0.013s
I20260709 12:10:25.719780 11868 catalog_manager.cc:5697] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:25.721544 12160 tablet_bootstrap.cc:654] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.738113 12159 raft_consensus.cc:359] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.739120 12159 raft_consensus.cc:385] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.739529 12159 raft_consensus.cc:740] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.740099 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:25.742791 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8d3b14c9ba14a48a6f7a51c571f49cc" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:25.743711 12158 tablet_bootstrap.cc:492] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:25.744355 12158 ts_tablet_manager.cc:1403] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.088s	user 0.015s	sys 0.028s
W20260709 12:10:25.744601 12011 leader_election.cc:343] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:25.745997 12011 leader_election.cc:343] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.746655 12160 tablet_bootstrap.cc:492] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:25.747218 12160 ts_tablet_manager.cc:1403] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.007s
I20260709 12:10:25.747773 12158 raft_consensus.cc:359] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.748606 12158 raft_consensus.cc:385] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.748910 12158 raft_consensus.cc:740] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.749750 12158 consensus_queue.cc:260] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.740597 12159 consensus_queue.cc:260] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.750469 12160 raft_consensus.cc:359] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.752794 12160 raft_consensus.cc:385] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.752429 12159 ts_tablet_manager.cc:1434] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.001s
I20260709 12:10:25.754125 12159 tablet_bootstrap.cc:492] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:25.754462 12158 ts_tablet_manager.cc:1434] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.005s
I20260709 12:10:25.756541 12160 raft_consensus.cc:740] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.757258 12158 tablet_bootstrap.cc:492] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.757448 12160 consensus_queue.cc:260] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.759398 12168 raft_consensus.cc:493] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.759858 12160 ts_tablet_manager.cc:1434] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.005s
I20260709 12:10:25.760185 12168 raft_consensus.cc:515] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.760953 12160 tablet_bootstrap.cc:492] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.766502 12168 leader_election.cc:290] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.769567 12158 tablet_bootstrap.cc:654] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.773183 12159 tablet_bootstrap.cc:654] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.782594 12160 tablet_bootstrap.cc:654] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.785151 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:25.785502 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8d3b14c9ba14a48a6f7a51c571f49cc" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:25.785984 12119 raft_consensus.cc:2468] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 0.
W20260709 12:10:25.787588 12008 leader_election.cc:343] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.788220 12008 leader_election.cc:304] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3; no voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:25.789960 12163 raft_consensus.cc:2749] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:25.791074 12008 leader_election.cc:304] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:25.792560 12163 raft_consensus.cc:2804] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:25.792997 12163 raft_consensus.cc:493] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:25.793391 12163 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.802369 12163 raft_consensus.cc:515] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.803683 12168 raft_consensus.cc:493] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.804526 12168 raft_consensus.cc:515] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.806933 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:25.807651 12119 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.813886 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
W20260709 12:10:25.815304 12011 leader_election.cc:343] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.815773 12119 raft_consensus.cc:2468] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 1.
I20260709 12:10:25.817024 12008 leader_election.cc:304] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:25.819790 12160 tablet_bootstrap.cc:492] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:25.807669 12168 leader_election.cc:290] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.822803 12160 ts_tablet_manager.cc:1403] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.062s	user 0.013s	sys 0.015s
I20260709 12:10:25.823042 12163 leader_election.cc:290] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.824779 12163 raft_consensus.cc:2804] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:25.826892 12164 raft_consensus.cc:493] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.825956 12163 raft_consensus.cc:697] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:25.827426 12164 raft_consensus.cc:515] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.831753 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:25.832728 12045 raft_consensus.cc:2468] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 0.
I20260709 12:10:25.829993 12164 leader_election.cc:290] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:25.833163 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:25.831032 12159 tablet_bootstrap.cc:492] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:25.834019 12044 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 0.
I20260709 12:10:25.834112 12085 leader_election.cc:304] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:25.834349 12159 ts_tablet_manager.cc:1403] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.080s	user 0.023s	sys 0.012s
I20260709 12:10:25.831753 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:25.838387 12159 raft_consensus.cc:359] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.839161 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:25.835884 11936 leader_election.cc:304] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
W20260709 12:10:25.839708 12085 leader_election.cc:343] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.835312 12164 raft_consensus.cc:2804] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:25.841012 12164 raft_consensus.cc:493] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:25.841100 12168 raft_consensus.cc:2804] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:25.841457 12164 raft_consensus.cc:3060] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.836246 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:25.841663 12168 raft_consensus.cc:493] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:25.843327 12168 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:25.847429 11933 leader_election.cc:343] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:25.848095 11936 leader_election.cc:343] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.848162 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:25.837479 12158 tablet_bootstrap.cc:492] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
W20260709 12:10:25.849804 11933 leader_election.cc:343] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.850270 12158 ts_tablet_manager.cc:1403] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.093s	user 0.041s	sys 0.022s
I20260709 12:10:25.829030 12160 raft_consensus.cc:359] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.851735 12160 raft_consensus.cc:385] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.852176 12160 raft_consensus.cc:740] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.851203 11936 leader_election.cc:304] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:25.851408 12164 raft_consensus.cc:515] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.839241 12159 raft_consensus.cc:385] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.830802 12163 consensus_queue.cc:237] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.853243 12160 consensus_queue.cc:260] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.854132 12159 raft_consensus.cc:740] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.866310 12159 consensus_queue.cc:260] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.867153 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
W20260709 12:10:25.869361 12085 leader_election.cc:343] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.871732 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:25.872437 12044 raft_consensus.cc:3060] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.872283 12158 raft_consensus.cc:359] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.875936 12158 raft_consensus.cc:385] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.876343 12158 raft_consensus.cc:740] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.875180 12168 raft_consensus.cc:515] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.877770 12164 leader_election.cc:290] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:25.881678 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
W20260709 12:10:25.883303 11933 leader_election.cc:343] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:25.884235 12145 tablet.cc:2406] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:10:25.878299 12158 consensus_queue.cc:260] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.901772 12168 leader_election.cc:290] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.894920 12194 raft_consensus.cc:2749] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:25.904608 12168 raft_consensus.cc:493] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.905174 12168 raft_consensus.cc:515] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.906418 12160 ts_tablet_manager.cc:1434] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.082s	user 0.023s	sys 0.033s
I20260709 12:10:25.908143 12160 tablet_bootstrap.cc:492] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.908562 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae69c80fd414289b43cc35643e55ca4" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
W20260709 12:10:25.910392 11933 leader_election.cc:343] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.911157 12168 leader_election.cc:290] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.899852 11866 catalog_manager.cc:5697] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:25.903350 12044 raft_consensus.cc:2468] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 1.
I20260709 12:10:25.913276 12085 leader_election.cc:304] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:25.914669 12164 raft_consensus.cc:2804] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:25.915148 12164 raft_consensus.cc:697] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:25.916097 12164 consensus_queue.cc:237] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.916999 12160 tablet_bootstrap.cc:654] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.892668 12159 ts_tablet_manager.cc:1434] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.058s	user 0.007s	sys 0.000s
I20260709 12:10:25.919220 12159 tablet_bootstrap.cc:492] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:25.904021 12158 ts_tablet_manager.cc:1434] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.053s	user 0.005s	sys 0.003s
I20260709 12:10:25.923008 12158 tablet_bootstrap.cc:492] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
W20260709 12:10:25.923853 11936 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:46343, user_credentials={real_user=slave}} blocked reactor thread for 81011.8us
I20260709 12:10:25.926594 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:25.927302 12044 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.929561 12159 tablet_bootstrap.cc:654] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.935786 12158 tablet_bootstrap.cc:654] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.939150 12160 tablet_bootstrap.cc:492] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:25.939755 12160 ts_tablet_manager.cc:1403] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.032s	user 0.015s	sys 0.005s
I20260709 12:10:25.940169 12044 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 1.
I20260709 12:10:25.940599 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae69c80fd414289b43cc35643e55ca4" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
W20260709 12:10:25.942729 11936 leader_election.cc:343] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:25.943231 12160 raft_consensus.cc:359] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.944084 12160 raft_consensus.cc:385] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.944446 12160 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.945453 12160 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.945657 11867 catalog_manager.cc:5697] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:25.943259 11936 leader_election.cc:304] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:25.948115 11936 leader_election.cc:304] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:25.950484 12193 raft_consensus.cc:2749] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:25.951578 12195 raft_consensus.cc:2804] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:25.952242 12195 raft_consensus.cc:697] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:25.953481 12195 consensus_queue.cc:237] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.959033 12193 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.959576 12193 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:25.960153 12160 ts_tablet_manager.cc:1434] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:10:25.961977 12160 tablet_bootstrap.cc:492] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.963074 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:25.963889 12044 raft_consensus.cc:2393] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c9b406eb70142dea09afbd266bd696f in current term 1: Already voted for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in this term.
I20260709 12:10:25.966279 12193 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:25.967228 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:25.969341 11933 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:25.970077 12160 tablet_bootstrap.cc:654] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.970883 12193 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:25.972872 12158 tablet_bootstrap.cc:492] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:25.973433 12158 ts_tablet_manager.cc:1403] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.051s	user 0.019s	sys 0.003s
I20260709 12:10:25.976476 12158 raft_consensus.cc:359] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.977288 12158 raft_consensus.cc:385] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.979866 12160 tablet_bootstrap.cc:492] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:25.977599 12158 raft_consensus.cc:740] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.981782 12160 ts_tablet_manager.cc:1403] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.008s
I20260709 12:10:25.982662 12159 tablet_bootstrap.cc:492] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:25.982271 12158 consensus_queue.cc:260] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.983289 12159 ts_tablet_manager.cc:1403] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.064s	user 0.024s	sys 0.009s
I20260709 12:10:25.984001 11867 catalog_manager.cc:5697] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 1 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:25.985167 12160 raft_consensus.cc:359] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.986163 12160 raft_consensus.cc:385] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:25.986513 12160 raft_consensus.cc:740] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:25.985180 12158 ts_tablet_manager.cc:1434] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.002s
I20260709 12:10:25.987437 12160 consensus_queue.cc:260] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.988930 12158 tablet_bootstrap.cc:492] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:25.990082 12160 ts_tablet_manager.cc:1434] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:10:25.991231 12160 tablet_bootstrap.cc:492] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:25.993078 12195 raft_consensus.cc:493] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:25.993600 12195 raft_consensus.cc:515] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:25.996299 12158 tablet_bootstrap.cc:654] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.997189 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8d3b14c9ba14a48a6f7a51c571f49cc" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:25.998171 12160 tablet_bootstrap.cc:654] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:25.998121 12044 raft_consensus.cc:2468] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 0.
I20260709 12:10:25.999866 11936 leader_election.cc:304] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:25.999979 12159 raft_consensus.cc:359] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.000782 12194 raft_consensus.cc:2804] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:26.001259 12194 raft_consensus.cc:493] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:25.980212 12193 raft_consensus.cc:2749] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:26.002653 12194 raft_consensus.cc:3060] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.003224 12164 raft_consensus.cc:493] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.003532 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8d3b14c9ba14a48a6f7a51c571f49cc" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:26.004297 12119 raft_consensus.cc:2468] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 0.
I20260709 12:10:26.003770 12164 raft_consensus.cc:515] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.007174 12164 leader_election.cc:290] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.007911 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
W20260709 12:10:26.010169 12085 leader_election.cc:343] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:26.011821 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.012662 12044 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 0.
I20260709 12:10:26.015375 12085 leader_election.cc:304] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.016608 12164 raft_consensus.cc:2804] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:26.013726 12195 leader_election.cc:290] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.001221 12159 raft_consensus.cc:385] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.017076 12164 raft_consensus.cc:493] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:26.018447 12164 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.012045 12194 raft_consensus.cc:515] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.022854 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8d3b14c9ba14a48a6f7a51c571f49cc" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:26.023455 12044 raft_consensus.cc:3060] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.025665 12194 leader_election.cc:290] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.028941 12158 tablet_bootstrap.cc:492] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:26.029337 12158 ts_tablet_manager.cc:1403] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.041s	user 0.018s	sys 0.015s
I20260709 12:10:26.029999 12160 tablet_bootstrap.cc:492] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:26.030594 12160 ts_tablet_manager.cc:1403] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.000s
I20260709 12:10:26.031626 12158 raft_consensus.cc:359] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.032212 12158 raft_consensus.cc:385] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.032444 12158 raft_consensus.cc:740] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.033097 12158 consensus_queue.cc:260] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.018731 12159 raft_consensus.cc:740] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.034132 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8d3b14c9ba14a48a6f7a51c571f49cc" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:26.033973 12160 raft_consensus.cc:359] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.034811 12160 raft_consensus.cc:385] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.034839 12119 raft_consensus.cc:3060] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.035210 12160 raft_consensus.cc:740] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.035221 12159 consensus_queue.cc:260] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.036890 12044 raft_consensus.cc:2468] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 1.
I20260709 12:10:26.038621 12159 ts_tablet_manager.cc:1434] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.003s
I20260709 12:10:26.039759 12159 tablet_bootstrap.cc:492] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:26.040223 11936 leader_election.cc:304] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:26.041119 12194 raft_consensus.cc:2804] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:26.041505 12160 consensus_queue.cc:260] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.042127 12158 ts_tablet_manager.cc:1434] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:10:26.042979 12194 raft_consensus.cc:697] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:26.044166 12160 ts_tablet_manager.cc:1434] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:10:26.043936 12194 consensus_queue.cc:237] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.045390 12160 tablet_bootstrap.cc:492] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:26.051997 12158 tablet_bootstrap.cc:492] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:26.056921 12159 tablet_bootstrap.cc:654] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.062852 12119 raft_consensus.cc:2468] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 1.
I20260709 12:10:26.063570 12160 tablet_bootstrap.cc:654] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.063845 12158 tablet_bootstrap.cc:654] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.064989 12164 raft_consensus.cc:515] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.070825 12164 leader_election.cc:290] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.071624 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:26.072197 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:26.072887 12044 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:26.073173 12085 leader_election.cc:343] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:26.072781 12159 tablet_bootstrap.cc:492] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:26.073030 11866 catalog_manager.cc:5697] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 1 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:26.074460 12159 ts_tablet_manager.cc:1403] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.005s
I20260709 12:10:26.080420 12044 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 1.
I20260709 12:10:26.081821 12085 leader_election.cc:304] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.082789 12164 raft_consensus.cc:2804] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:26.085036 12164 raft_consensus.cc:697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:26.085100 12159 raft_consensus.cc:359] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.085969 12159 raft_consensus.cc:385] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.086303 12159 raft_consensus.cc:740] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.086264 12164 consensus_queue.cc:237] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.088847 12160 tablet_bootstrap.cc:492] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:26.089360 12160 ts_tablet_manager.cc:1403] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.044s	user 0.013s	sys 0.001s
I20260709 12:10:26.091161 12158 tablet_bootstrap.cc:492] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:26.090736 12159 consensus_queue.cc:260] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.091739 12158 ts_tablet_manager.cc:1403] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.040s	user 0.020s	sys 0.005s
I20260709 12:10:26.094151 12160 raft_consensus.cc:359] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.094928 12160 raft_consensus.cc:385] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.095292 12160 raft_consensus.cc:740] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.096293 12159 ts_tablet_manager.cc:1434] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.021s	user 0.010s	sys 0.000s
I20260709 12:10:26.096065 12160 consensus_queue.cc:260] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.096552 12158 raft_consensus.cc:359] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.097295 12158 raft_consensus.cc:385] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.097491 12159 tablet_bootstrap.cc:492] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:26.097647 12158 raft_consensus.cc:740] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.098549 12158 consensus_queue.cc:260] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.102666 12160 ts_tablet_manager.cc:1434] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:10:26.103754 12160 tablet_bootstrap.cc:492] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:26.104780 12159 tablet_bootstrap.cc:654] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.108296 11867 catalog_manager.cc:5697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:26.110404 12158 ts_tablet_manager.cc:1434] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:10:26.115721 12160 tablet_bootstrap.cc:654] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.120908 12159 tablet_bootstrap.cc:492] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:26.121492 12159 ts_tablet_manager.cc:1403] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.012s
I20260709 12:10:26.123299 12158 tablet_bootstrap.cc:492] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3: Bootstrap starting.
I20260709 12:10:26.124362 12159 raft_consensus.cc:359] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.124908 12164 raft_consensus.cc:493] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.125087 12159 raft_consensus.cc:385] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.125517 12164 raft_consensus.cc:515] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.127898 12164 leader_election.cc:290] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.130023 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:26.130664 11969 raft_consensus.cc:2468] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 0.
I20260709 12:10:26.130517 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.132210 12085 leader_election.cc:304] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:26.133129 12164 raft_consensus.cc:2804] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:26.133580 12164 raft_consensus.cc:493] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:10:26.133733 12085 leader_election.cc:343] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:26.134047 12164 raft_consensus.cc:3060] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.134691 12158 tablet_bootstrap.cc:654] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.141040 12160 tablet_bootstrap.cc:492] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:26.141541 12160 ts_tablet_manager.cc:1403] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.010s
I20260709 12:10:26.125497 12159 raft_consensus.cc:740] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.142603 12159 consensus_queue.cc:260] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.144657 12160 raft_consensus.cc:359] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.145364 12164 raft_consensus.cc:515] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.147681 12159 ts_tablet_manager.cc:1434] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.001s
I20260709 12:10:26.145422 12160 raft_consensus.cc:385] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.148754 12159 tablet_bootstrap.cc:492] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:26.148680 12160 raft_consensus.cc:740] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.150089 12160 consensus_queue.cc:260] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.154436 12164 leader_election.cc:290] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.155786 12172 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.157015 12160 ts_tablet_manager.cc:1434] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260709 12:10:26.158879 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
W20260709 12:10:26.160446 12085 leader_election.cc:343] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:26.161250 12159 tablet_bootstrap.cc:654] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.162142 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:26.162793 11969 raft_consensus.cc:3060] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.175808 12160 tablet_bootstrap.cc:492] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: Bootstrap starting.
I20260709 12:10:26.180792 12158 tablet_bootstrap.cc:492] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3: No bootstrap required, opened a new log
I20260709 12:10:26.181644 12158 ts_tablet_manager.cc:1403] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3: Time spent bootstrapping tablet: real 0.059s	user 0.021s	sys 0.008s
I20260709 12:10:26.174695 11969 raft_consensus.cc:2468] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 1.
I20260709 12:10:26.186607 12085 leader_election.cc:304] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:26.187458 12172 raft_consensus.cc:2804] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:26.187992 12172 raft_consensus.cc:697] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:26.188689 12172 consensus_queue.cc:237] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.191023 12158 raft_consensus.cc:359] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.191829 12158 raft_consensus.cc:385] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.191978 12159 tablet_bootstrap.cc:492] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:26.192145 12158 raft_consensus.cc:740] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.192464 12159 ts_tablet_manager.cc:1403] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.044s	user 0.014s	sys 0.018s
I20260709 12:10:26.195437 12159 raft_consensus.cc:359] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.196193 12160 tablet_bootstrap.cc:654] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.196164 12159 raft_consensus.cc:385] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.196609 12159 raft_consensus.cc:740] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.197443 12159 consensus_queue.cc:260] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.197891 12158 consensus_queue.cc:260] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.204219 12163 raft_consensus.cc:493] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.205035 12163 raft_consensus.cc:515] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.209774 11970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
W20260709 12:10:26.212164 12011 leader_election.cc:343] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:26.213513 12159 ts_tablet_manager.cc:1434] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.021s	user 0.012s	sys 0.008s
I20260709 12:10:26.214308 12163 leader_election.cc:290] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.214569 12159 tablet_bootstrap.cc:492] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734: Bootstrap starting.
I20260709 12:10:26.215132 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:26.215924 12119 raft_consensus.cc:2468] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 0.
I20260709 12:10:26.216171 12158 ts_tablet_manager.cc:1434] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3: Time spent starting tablet: real 0.034s	user 0.003s	sys 0.004s
I20260709 12:10:26.217243 12008 leader_election.cc:304] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.218616 12163 raft_consensus.cc:2804] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:26.219055 12163 raft_consensus.cc:493] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:26.219769 12163 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.220037 11866 catalog_manager.cc:5697] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:26.222074 12159 tablet_bootstrap.cc:654] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:26.252225 12160 tablet_bootstrap.cc:492] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: No bootstrap required, opened a new log
I20260709 12:10:26.250836 12163 raft_consensus.cc:515] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.254418 12160 ts_tablet_manager.cc:1403] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: Time spent bootstrapping tablet: real 0.079s	user 0.017s	sys 0.004s
I20260709 12:10:26.256456 12163 leader_election.cc:290] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.256738 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:26.257913 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:26.258558 12119 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:26.258700 12011 leader_election.cc:343] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:26.254693 12159 tablet_bootstrap.cc:492] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734: No bootstrap required, opened a new log
I20260709 12:10:26.259680 12159 ts_tablet_manager.cc:1403] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent bootstrapping tablet: real 0.045s	user 0.025s	sys 0.008s
I20260709 12:10:26.259296 12160 raft_consensus.cc:359] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.260035 12160 raft_consensus.cc:385] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.260325 12160 raft_consensus.cc:740] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.261137 12215 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.266222 12164 raft_consensus.cc:493] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.266736 12164 raft_consensus.cc:515] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.267460 12160 consensus_queue.cc:260] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.268962 12164 leader_election.cc:290] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.262811 12159 raft_consensus.cc:359] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.269541 12159 raft_consensus.cc:385] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:26.269863 12159 raft_consensus.cc:740] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Initialized, Role: FOLLOWER
I20260709 12:10:26.270242 11970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:26.270623 12159 consensus_queue.cc:260] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.273219 12159 ts_tablet_manager.cc:1434] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:10:26.273335 12160 ts_tablet_manager.cc:1434] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260709 12:10:26.275681 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.276752 12045 raft_consensus.cc:2393] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb6e4947dc5e4a8595ab661f2ea6c734 in current term 1: Already voted for candidate 3c9b406eb70142dea09afbd266bd696f in this term.
I20260709 12:10:26.280524 12085 leader_election.cc:304] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:26.281800 12164 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.290336 12119 raft_consensus.cc:2468] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 1.
I20260709 12:10:26.291757 12008 leader_election.cc:304] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.293078 12163 raft_consensus.cc:2804] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:26.293649 12163 raft_consensus.cc:697] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:26.294641 12163 consensus_queue.cc:237] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.295773 12164 raft_consensus.cc:2749] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:26.296057 12194 raft_consensus.cc:493] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.296561 12194 raft_consensus.cc:515] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.300428 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.300628 12194 leader_election.cc:290] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.301730 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:26.302279 12045 raft_consensus.cc:2393] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c9b406eb70142dea09afbd266bd696f in current term 1: Already voted for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in this term.
I20260709 12:10:26.307077 11933 leader_election.cc:304] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:26.308893 12194 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.316139 12194 raft_consensus.cc:2749] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:26.322515 12194 raft_consensus.cc:493] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.322170 11868 catalog_manager.cc:5697] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:26.323082 12194 raft_consensus.cc:515] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.325497 12194 leader_election.cc:290] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.326318 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.326483 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:26.327358 12119 raft_consensus.cc:2393] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c9b406eb70142dea09afbd266bd696f in current term 1: Already voted for candidate 997e54add8aa4783a09f5aa876db8fb3 in this term.
I20260709 12:10:26.328969 11936 leader_election.cc:304] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:26.330399 12194 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.337332 12194 raft_consensus.cc:2749] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:26.379660 12164 consensus_queue.cc:1048] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.384578 11970 raft_consensus.cc:3060] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.386996 12164 raft_consensus.cc:493] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.387593 12164 raft_consensus.cc:515] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.389902 12164 leader_election.cc:290] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.391371 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae69c80fd414289b43cc35643e55ca4" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:26.392169 11969 raft_consensus.cc:2468] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 0.
I20260709 12:10:26.394497 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae69c80fd414289b43cc35643e55ca4" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.394543 12085 leader_election.cc:304] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:26.395310 12045 raft_consensus.cc:2468] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 0.
I20260709 12:10:26.396072 12172 raft_consensus.cc:2804] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:26.396486 12172 raft_consensus.cc:493] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:26.396844 12172 raft_consensus.cc:3060] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.398408 12164 consensus_queue.cc:1048] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.402240 12163 consensus_queue.cc:1048] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.405949 12172 raft_consensus.cc:515] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.412861 12172 leader_election.cc:290] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:26.414559 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae69c80fd414289b43cc35643e55ca4" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:26.415031 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bae69c80fd414289b43cc35643e55ca4" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:26.415381 11969 raft_consensus.cc:3060] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.415810 12044 raft_consensus.cc:3060] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.423744 12044 raft_consensus.cc:2468] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 1.
I20260709 12:10:26.425810 12085 leader_election.cc:304] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:26.426751 12192 consensus_queue.cc:1048] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.428134 12172 raft_consensus.cc:2804] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:26.426981 11969 raft_consensus.cc:2468] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 1.
I20260709 12:10:26.430169 12172 raft_consensus.cc:697] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:26.431290 12172 consensus_queue.cc:237] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.437510 12194 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.456802 12194 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.457942 11867 catalog_manager.cc:5697] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:26.464063 12163 raft_consensus.cc:493] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.464655 12163 raft_consensus.cc:515] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.469343 12163 leader_election.cc:290] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.470417 11970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:26.471264 11970 raft_consensus.cc:2468] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 0.
I20260709 12:10:26.472507 12011 leader_election.cc:304] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:26.473495 12163 raft_consensus.cc:2804] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:26.473875 12163 raft_consensus.cc:493] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:26.474213 12163 raft_consensus.cc:3060] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.474946 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:26.475683 12118 raft_consensus.cc:2468] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 0.
I20260709 12:10:26.490060 12163 raft_consensus.cc:515] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.493057 12163 leader_election.cc:290] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.493332 11970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:26.494220 11970 raft_consensus.cc:3060] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.496553 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:26.497270 12118 raft_consensus.cc:3060] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.502117 11970 raft_consensus.cc:2468] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 1.
I20260709 12:10:26.503527 12011 leader_election.cc:304] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:26.504477 12163 raft_consensus.cc:2804] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:26.505100 12163 raft_consensus.cc:697] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:26.505383 12118 raft_consensus.cc:2468] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 1.
I20260709 12:10:26.506098 12163 consensus_queue.cc:237] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.516021 11867 catalog_manager.cc:5697] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:26.557541 12194 consensus_queue.cc:1048] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.569751 12195 consensus_queue.cc:1048] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.594080 12172 consensus_queue.cc:1048] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.605327 12044 raft_consensus.cc:3060] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.609654 12215 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.617134 12172 consensus_queue.cc:1048] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.622010 12215 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.704818 12195 raft_consensus.cc:493] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:26.705358 12195 raft_consensus.cc:515] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.707333 12195 leader_election.cc:290] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.708743 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:26.708619 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:26.709812 12118 raft_consensus.cc:2393] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c9b406eb70142dea09afbd266bd696f in current term 1: Already voted for candidate 997e54add8aa4783a09f5aa876db8fb3 in this term.
I20260709 12:10:26.711826 11936 leader_election.cc:304] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:26.712814 12195 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:26.718701 12195 raft_consensus.cc:2749] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:26.748566 12163 consensus_queue.cc:1048] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.762853 12192 consensus_queue.cc:1048] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.798997 12118 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:49630
I20260709 12:10:26.800028 12118 raft_consensus.cc:606] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.812161 12118 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:49630
I20260709 12:10:26.812776 12118 raft_consensus.cc:606] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.830425 12045 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:41200
I20260709 12:10:26.831234 12045 raft_consensus.cc:606] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.838511 12118 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:49630
I20260709 12:10:26.839221 12118 raft_consensus.cc:606] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:26.862536 11970 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:36710
I20260709 12:10:26.863282 11970 raft_consensus.cc:606] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Received request to transfer leadership to TS 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:26.874873 12119 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
mode: GRACEFUL
new_leader_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:49630
I20260709 12:10:26.875605 12119 raft_consensus.cc:606] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Received request to transfer leadership to TS 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:26.905718 12192 raft_consensus.cc:993] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:26.906337 12163 raft_consensus.cc:1081] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:26.908568 11970 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36698
I20260709 12:10:26.909390 11970 raft_consensus.cc:493] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:26.909868 11970 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:26.915548 11970 raft_consensus.cc:515] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.917927 11970 leader_election.cc:290] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.918938 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:26.919139 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:26.919741 12045 raft_consensus.cc:3055] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:26.919831 12119 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:26.920204 12045 raft_consensus.cc:740] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:26.921182 12045 consensus_queue.cc:260] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.922401 12045 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:26.927064 12119 raft_consensus.cc:2468] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:26.928782 11933 leader_election.cc:304] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:26.929877 12195 raft_consensus.cc:2804] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:26.930543 12194 raft_consensus.cc:993] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:26.930624 12045 raft_consensus.cc:2468] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:26.931130 12194 raft_consensus.cc:1081] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Signalling peer 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:26.932037 12195 raft_consensus.cc:697] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:26.933422 12045 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:41184
I20260709 12:10:26.933310 12195 consensus_queue.cc:237] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.934440 12045 raft_consensus.cc:493] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:26.934890 12045 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:26.939219 12163 consensus_queue.cc:1048] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.941866 11867 catalog_manager.cc:5697] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 1 to 2, leader changed from 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 2 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:26.942791 12045 raft_consensus.cc:515] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.946327 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:26.946952 12119 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:26.947131 12045 leader_election.cc:290] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 2 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:26.947830 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:26.948473 11969 raft_consensus.cc:3055] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:26.948807 11969 raft_consensus.cc:740] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:26.949658 11969 consensus_queue.cc:260] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.957175 11969 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:26.957103 12192 consensus_queue.cc:1048] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:26.961254 12119 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 2.
I20260709 12:10:26.962920 12008 leader_election.cc:304] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:26.963976 12192 raft_consensus.cc:2804] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:26.964504 12192 raft_consensus.cc:697] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:26.965767 12192 consensus_queue.cc:237] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:26.971556 11969 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 2.
I20260709 12:10:26.975721 11867 catalog_manager.cc:5697] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 1 to 2, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:26.978384 12172 consensus_queue.cc:1048] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:26.997550 12216 raft_consensus.cc:993] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: : Instructing follower 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:26.998206 12172 raft_consensus.cc:1081] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Signalling peer 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:26.998215 12164 consensus_queue.cc:1048] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:27.000290 12045 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:41174
I20260709 12:10:27.000947 12045 raft_consensus.cc:493] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:27.001362 12045 raft_consensus.cc:3060] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.009157 12045 raft_consensus.cc:515] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.011646 12045 leader_election.cc:290] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 2 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:27.013496 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:27.014237 12118 raft_consensus.cc:3055] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:27.014256 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:27.014742 12118 raft_consensus.cc:740] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:27.015026 11969 raft_consensus.cc:3060] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.015653 12118 consensus_queue.cc:260] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.017168 12118 raft_consensus.cc:3060] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.023905 11969 raft_consensus.cc:2468] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 2.
I20260709 12:10:27.025002 12118 raft_consensus.cc:2468] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 2.
I20260709 12:10:27.025684 12011 leader_election.cc:304] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:27.026679 12192 raft_consensus.cc:2804] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:27.027052 12192 raft_consensus.cc:697] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:27.027845 12192 consensus_queue.cc:237] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.034477 11868 catalog_manager.cc:5697] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 1 to 2, leader changed from bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131) to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:27.042968 12164 raft_consensus.cc:993] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:27.043454 12164 raft_consensus.cc:1081] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:27.044754 12215 raft_consensus.cc:993] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:27.045203 12215 raft_consensus.cc:1081] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:27.045209 11969 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36696
I20260709 12:10:27.046452 11969 raft_consensus.cc:493] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:27.046947 11970 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36696
I20260709 12:10:27.047475 11969 raft_consensus.cc:3060] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.047696 11970 raft_consensus.cc:493] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:27.048192 11970 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.054382 11970 raft_consensus.cc:515] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.054864 11969 raft_consensus.cc:515] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.056674 11970 leader_election.cc:290] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:27.058239 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:27.058231 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:27.058681 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:27.058831 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1836d0ddf8c40b0a3961a899546dabb" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:27.059197 12045 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.059312 12044 raft_consensus.cc:3060] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.059527 12119 raft_consensus.cc:3055] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:27.059584 11969 leader_election.cc:290] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:27.060034 12119 raft_consensus.cc:740] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:27.059275 12118 raft_consensus.cc:3055] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:27.061026 12119 consensus_queue.cc:260] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.061878 12118 raft_consensus.cc:740] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:27.062620 12119 raft_consensus.cc:3060] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.063438 12118 consensus_queue.cc:260] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.064952 12118 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.067822 12045 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:27.068064 12044 raft_consensus.cc:2468] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:27.069296 11936 leader_election.cc:304] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:27.070290 12195 raft_consensus.cc:2804] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:27.070447 11936 leader_election.cc:304] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:27.070911 12195 raft_consensus.cc:697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:27.071614 12194 raft_consensus.cc:2804] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:27.071767 12119 raft_consensus.cc:2468] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:27.072224 12194 raft_consensus.cc:697] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:27.071826 12195 consensus_queue.cc:237] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.072474 12118 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:27.073207 12194 consensus_queue.cc:237] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:27.082103 11868 catalog_manager.cc:5697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 1 to 2, leader changed from bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 2 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:27.083717 11868 catalog_manager.cc:5697] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 1 to 2, leader changed from bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 2 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:27.258582 12164 raft_consensus.cc:993] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:27.259395 12164 raft_consensus.cc:1081] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:27.261267 11969 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36696
I20260709 12:10:27.262099 11969 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:27.262429 11969 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.267365 11969 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:27.269232 11969 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:27.270421 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:27.270725 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:27.271216 12045 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.271250 12118 raft_consensus.cc:3055] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:27.271845 12118 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:27.272832 12118 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:27.274192 12118 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:27.279930 12045 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:27.281487 11936 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:27.282224 12118 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:27.282395 12194 raft_consensus.cc:2804] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:27.283051 12194 raft_consensus.cc:697] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:27.283928 12194 consensus_queue.cc:237] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:27.291527 11868 catalog_manager.cc:5697] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 1 to 2, leader changed from bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 2 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:27.411494 11969 raft_consensus.cc:1275] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.413143 12192 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.423161 12118 raft_consensus.cc:1275] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.425029 12163 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.444104 11969 raft_consensus.cc:1275] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.445772 12163 consensus_queue.cc:1048] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.457191 12118 raft_consensus.cc:1275] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.459414 12242 consensus_queue.cc:1048] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.482933 12118 raft_consensus.cc:1275] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.484377 12194 consensus_queue.cc:1048] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:27.495342 12045 raft_consensus.cc:1275] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.497099 12195 consensus_queue.cc:1048] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.507385 12045 raft_consensus.cc:1275] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.509135 12248 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.517328 12118 raft_consensus.cc:1275] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.519043 12194 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.624006 12045 raft_consensus.cc:1275] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.625589 12193 consensus_queue.cc:1048] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.635479 12118 raft_consensus.cc:1275] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.637328 12194 consensus_queue.cc:1048] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:27.751237 12118 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.752738 12194 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:27.763448 12045 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:27.765244 12248 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:29.920337 11310 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:10:29.921087 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:29.948623 11969 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:37992
I20260709 12:10:29.949244 11969 raft_consensus.cc:606] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Received request to transfer leadership to TS bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:29.952607 11969 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:37992
I20260709 12:10:29.953219 11969 raft_consensus.cc:606] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Received request to transfer leadership to TS bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:29.954530 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:10:29.955052 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:29.965670 12262 raft_consensus.cc:993] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:29.966270 12248 raft_consensus.cc:1081] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Signalling peer bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:29.968020 12118 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:49628
I20260709 12:10:29.968559 12118 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:29.968894 12118 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:29.973789 12118 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:29.975540 12118 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 3 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093)
I20260709 12:10:29.976622 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:29.976943 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:29.977221 12045 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:29.977769 11969 raft_consensus.cc:3055] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:29.978142 11969 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:29.979094 11969 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:29.980248 11969 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:29.984001 12045 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 3.
I20260709 12:10:29.985176 12085 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:29.985954 12258 raft_consensus.cc:2804] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:29.987576 12258 raft_consensus.cc:697] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:29.987504 11969 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 3.
I20260709 12:10:29.988909 12258 consensus_queue.cc:237] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:29.998061 11867 catalog_manager.cc:5697] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 2 to 3, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:30.124184 12262 raft_consensus.cc:993] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:30.125078 12262 raft_consensus.cc:1081] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Signalling peer bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:30.127400 12118 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:49628
I20260709 12:10:30.128360 12118 raft_consensus.cc:493] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:30.128777 12118 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:30.135747 12118 raft_consensus.cc:515] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:30.137975 12118 leader_election.cc:290] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 3 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:30.139259 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:30.139755 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:30.139984 11969 raft_consensus.cc:3055] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:30.140570 12045 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:30.140571 11969 raft_consensus.cc:740] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:30.142213 11969 consensus_queue.cc:260] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:30.143651 11969 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:30.147091 12045 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 3.
I20260709 12:10:30.148449 12085 leader_election.cc:304] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:30.149303 12258 raft_consensus.cc:2804] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:30.149809 12258 raft_consensus.cc:697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:30.150538 11969 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 3.
I20260709 12:10:30.150765 12258 consensus_queue.cc:237] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:30.163648 11867 catalog_manager.cc:5697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 2 to 3, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:30.459317 12045 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:30.460997 12266 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:10:30.462626 11969 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:30.463987 12266 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:30.618361 11969 raft_consensus.cc:1275] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:30.619915 12265 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:30.630895 12044 raft_consensus.cc:1275] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:30.632467 12265 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:30.956434 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:30.962546 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:30.963157 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:31.964452 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:31.971136 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:31.971764 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:32.973322 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:32.981616 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:32.982295 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:33.983654 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:33.990350 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:33.990939 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:34.992372 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:34.999147 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:34.999784 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:36.001096 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:36.007241 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:36.007839 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:37.008968 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:37.015523 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:37.016119 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:38.017299 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:38.023496 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:38.024013 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:39.025417 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:39.032270 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:39.032788 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:40.906836 12299 raft_consensus.cc:493] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 997e54add8aa4783a09f5aa876db8fb3)
I20260709 12:10:40.907791 12299 raft_consensus.cc:515] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:40.912715 12299 leader_election.cc:290] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:40.938462 12300 raft_consensus.cc:493] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 997e54add8aa4783a09f5aa876db8fb3)
I20260709 12:10:40.939132 12300 raft_consensus.cc:515] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:40.941772 12300 leader_election.cc:290] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:40.948239 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" is_pre_election: true
I20260709 12:10:40.947919 11970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c9b406eb70142dea09afbd266bd696f" is_pre_election: true
I20260709 12:10:40.963960 12044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:40.967854 12043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6401f8748164c969e0dfe96d0cbf8dd" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "997e54add8aa4783a09f5aa876db8fb3" is_pre_election: true
I20260709 12:10:40.973766 11936 leader_election.cc:304] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f; no voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:40.974057 12085 leader_election.cc:304] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:40.975111 12298 raft_consensus.cc:2749] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:10:40.975158 12303 raft_consensus.cc:2749] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:10:40.987142 11310 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:10:40.993221 12119 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:49630
I20260709 12:10:40.993920 12119 raft_consensus.cc:606] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:41.003639 12119 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:49630
I20260709 12:10:41.004321 12119 raft_consensus.cc:606] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:41.016737 12043 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:41200
I20260709 12:10:41.017397 12043 raft_consensus.cc:606] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:41.023438 12043 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:41200
I20260709 12:10:41.024091 12043 raft_consensus.cc:606] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:41.029402 12043 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
mode: GRACEFUL
new_leader_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:41200
I20260709 12:10:41.030051 12043 raft_consensus.cc:606] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Received request to transfer leadership to TS 3c9b406eb70142dea09afbd266bd696f
I20260709 12:10:41.312510 12299 raft_consensus.cc:993] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.313019 12298 raft_consensus.cc:1081] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.314813 11970 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36696
I20260709 12:10:41.315341 11970 raft_consensus.cc:493] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:41.315667 11970 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:41.321944 11970 raft_consensus.cc:515] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.323863 11970 leader_election.cc:290] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 4 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:41.324934 12043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:41.325713 12043 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:41.325896 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a3e7f70af184c4c8d2d2263e5a2c033" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:41.326529 12119 raft_consensus.cc:3055] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:10:41.326851 12119 raft_consensus.cc:740] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:41.327946 12119 consensus_queue.cc:260] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.329457 12119 raft_consensus.cc:3060] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:41.335211 12043 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 4.
I20260709 12:10:41.336892 11936 leader_election.cc:304] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:41.337834 12303 raft_consensus.cc:2804] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:10:41.338068 12119 raft_consensus.cc:2468] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 4.
I20260709 12:10:41.338591 12303 raft_consensus.cc:697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:41.339720 12303 consensus_queue.cc:237] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.350214 12299 raft_consensus.cc:993] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.350833 12299 raft_consensus.cc:1081] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.352473 11970 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36696
I20260709 12:10:41.353060 11970 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:41.353458 11970 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:41.355602 11867 catalog_manager.cc:5697] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 3 to 4, leader changed from bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 4 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:41.360445 11970 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:41.362342 11970 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 4 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:41.363232 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:41.363801 12045 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:41.363715 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:41.364357 12119 raft_consensus.cc:3055] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:10:41.364691 12119 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:41.365789 12119 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:41.367234 12119 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:41.371657 12045 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 4.
I20260709 12:10:41.372159 12304 raft_consensus.cc:993] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.372848 12304 raft_consensus.cc:1081] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.373108 11936 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:41.374228 12303 raft_consensus.cc:2804] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:10:41.374686 12303 raft_consensus.cc:697] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:41.374989 12119 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 4.
I20260709 12:10:41.375507 11969 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36698
I20260709 12:10:41.376168 11969 raft_consensus.cc:493] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:41.376576 11969 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:41.375675 12303 consensus_queue.cc:237] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:41.385070 11969 raft_consensus.cc:515] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.387061 11969 leader_election.cc:290] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 3 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:41.387431 11867 catalog_manager.cc:5697] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 3 to 4, leader changed from bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 4 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:41.387981 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:41.388051 12119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:41.388890 12045 raft_consensus.cc:3055] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:41.389019 12119 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:41.389254 12045 raft_consensus.cc:740] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:41.390162 12045 consensus_queue.cc:260] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.391431 12045 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:41.396440 12119 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 3.
I20260709 12:10:41.398316 11933 leader_election.cc:304] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:41.399194 12300 raft_consensus.cc:2804] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:41.399469 12045 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 3.
I20260709 12:10:41.399643 12300 raft_consensus.cc:697] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:41.400620 12300 consensus_queue.cc:237] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.410504 11867 catalog_manager.cc:5697] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 2 to 3, leader changed from 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 3 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:41.441195 12301 raft_consensus.cc:993] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.441913 12302 raft_consensus.cc:1081] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.443684 11969 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36698
I20260709 12:10:41.444240 11969 raft_consensus.cc:493] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:41.444564 11969 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:41.449369 11969 raft_consensus.cc:515] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.451174 11969 leader_election.cc:290] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:41.452193 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:41.452478 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:41.452798 12045 raft_consensus.cc:3055] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:41.453169 12045 raft_consensus.cc:740] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:41.453208 12118 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:41.454214 12045 consensus_queue.cc:260] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.455240 12045 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:41.459590 12118 raft_consensus.cc:2468] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:41.459878 12045 raft_consensus.cc:2468] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:41.461135 11933 leader_election.cc:304] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:41.462103 12300 raft_consensus.cc:2804] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:41.462644 12300 raft_consensus.cc:697] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:41.463531 12300 consensus_queue.cc:237] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.471652 11867 catalog_manager.cc:5697] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 1 to 2, leader changed from 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 2 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:41.531878 12301 raft_consensus.cc:993] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: : Instructing follower 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.532444 12302 raft_consensus.cc:1081] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Signalling peer 3c9b406eb70142dea09afbd266bd696f to start an election
I20260709 12:10:41.534240 11969 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
 from {username='slave'} at 127.0.0.1:36698
I20260709 12:10:41.534749 11969 raft_consensus.cc:493] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:41.535010 11969 raft_consensus.cc:3060] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:41.539503 11969 raft_consensus.cc:515] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.541159 11969 leader_election.cc:290] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:41.542069 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:41.542340 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce32af3d4f724f0db85f77eb35ba228c" candidate_uuid: "3c9b406eb70142dea09afbd266bd696f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:41.542717 12045 raft_consensus.cc:3055] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:10:41.542804 12118 raft_consensus.cc:3060] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:41.543079 12045 raft_consensus.cc:740] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:41.543853 12045 consensus_queue.cc:260] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.544983 12045 raft_consensus.cc:3060] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:10:41.549420 12118 raft_consensus.cc:2468] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:41.550854 11933 leader_election.cc:304] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:41.551189 12045 raft_consensus.cc:2468] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9b406eb70142dea09afbd266bd696f in term 2.
I20260709 12:10:41.551796 12300 raft_consensus.cc:2804] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:10:41.552467 12300 raft_consensus.cc:697] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Leader. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:41.553413 12300 consensus_queue.cc:237] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:41.562530 11867 catalog_manager.cc:5697] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f reported cstate change: term changed from 1 to 2, leader changed from 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130) to 3c9b406eb70142dea09afbd266bd696f (127.11.11.129). New cstate: current_term: 2 leader_uuid: "3c9b406eb70142dea09afbd266bd696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:41.735147 12043 raft_consensus.cc:1275] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:41.736785 12300 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:41.750789 12118 raft_consensus.cc:1275] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:41.752477 12297 consensus_queue.cc:1048] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:10:41.922235 12118 raft_consensus.cc:1275] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:41.923803 12297 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:10:41.937922 12140 debug-util.cc:398] Leaking SignalData structure 0x7b08000aab00 after lost signal to thread 11313
I20260709 12:10:41.938807 12043 raft_consensus.cc:1275] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:10:41.939157 12140 debug-util.cc:398] Leaking SignalData structure 0x7b08000a99c0 after lost signal to thread 11902
W20260709 12:10:41.940521 12140 debug-util.cc:398] Leaking SignalData structure 0x7b08000dafa0 after lost signal to thread 11994
I20260709 12:10:41.940477 12300 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:10:41.941823 12140 debug-util.cc:398] Leaking SignalData structure 0x7b080013f440 after lost signal to thread 12069
W20260709 12:10:41.943010 12140 debug-util.cc:398] Leaking SignalData structure 0x7b08001e3e20 after lost signal to thread 12143
I20260709 12:10:41.942967 12118 raft_consensus.cc:1275] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:41.945564 12300 consensus_queue.cc:1048] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:41.971098 12119 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:41.971645 12043 raft_consensus.cc:1275] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:41.972730 12303 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:41.975132 12303 consensus_queue.cc:1048] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:10:41.989107 12045 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:41.993659 12300 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:10:42.090087 12045 raft_consensus.cc:1275] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:42.091493 12303 consensus_queue.cc:1048] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:42.101902 12119 raft_consensus.cc:1275] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Refusing update from remote peer 3c9b406eb70142dea09afbd266bd696f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:10:42.103856 12297 consensus_queue.cc:1048] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:10:44.086850 11310 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:10:44.088110 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:44.100924 11969 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:37992
I20260709 12:10:44.102049 11969 raft_consensus.cc:606] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Received request to transfer leadership to TS bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:44.105974 11969 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:37992
I20260709 12:10:44.106573 11969 raft_consensus.cc:606] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Received request to transfer leadership to TS 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:44.109480 11969 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:37992
I20260709 12:10:44.110033 11969 raft_consensus.cc:606] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 LEADER]: Received request to transfer leadership to TS 997e54add8aa4783a09f5aa876db8fb3
I20260709 12:10:44.112887 11969 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847"
dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
mode: GRACEFUL
new_leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:37992
I20260709 12:10:44.113387 11969 raft_consensus.cc:606] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Received request to transfer leadership to TS bb6e4947dc5e4a8595ab661f2ea6c734
I20260709 12:10:44.114778 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:10:44.115620 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:44.228282 12329 raft_consensus.cc:993] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:44.229107 12303 raft_consensus.cc:1081] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Signalling peer bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:44.231253 12118 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9751272f5592489a95216b06be6793ff"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:49628
I20260709 12:10:44.231968 12118 raft_consensus.cc:493] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:44.232407 12118 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:10:44.242466 12118 raft_consensus.cc:515] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:44.245314 12118 leader_election.cc:290] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 5 election: Requested vote from peers 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343), 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093)
I20260709 12:10:44.246860 12045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:44.247308 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9751272f5592489a95216b06be6793ff" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:44.247865 12045 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:10:44.248106 11969 raft_consensus.cc:3055] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:10:44.248538 11969 raft_consensus.cc:740] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:44.249955 11969 consensus_queue.cc:260] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:44.251595 11969 raft_consensus.cc:3060] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:10:44.255126 12045 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 5.
I20260709 12:10:44.256716 12085 leader_election.cc:304] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:44.257676 12320 raft_consensus.cc:2804] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:10:44.258881 12320 raft_consensus.cc:697] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 5 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:44.260071 12320 consensus_queue.cc:237] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } }
I20260709 12:10:44.261204 11969 raft_consensus.cc:2468] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 5.
I20260709 12:10:44.270149 11866 catalog_manager.cc:5697] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 4 to 5, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 5 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:44.421396 12329 raft_consensus.cc:993] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:44.422024 12329 raft_consensus.cc:1081] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Signalling peer 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:44.423874 12045 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:41184
I20260709 12:10:44.424456 12045 raft_consensus.cc:493] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:44.424829 12045 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:44.432565 12045 raft_consensus.cc:515] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.435361 12045 leader_election.cc:290] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 3 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:44.436893 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:44.436970 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5c5d0844a545b4b511b3fd6239cb53" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:44.437796 11969 raft_consensus.cc:3055] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:44.437865 12118 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:44.438371 11969 raft_consensus.cc:740] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:44.439262 11969 consensus_queue.cc:260] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.441246 11969 raft_consensus.cc:3060] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:44.442914 12329 raft_consensus.cc:993] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:44.443449 12329 raft_consensus.cc:1081] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 LEADER]: Signalling peer 997e54add8aa4783a09f5aa876db8fb3 to start an election
I20260709 12:10:44.445773 12043 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab"
dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
 from {username='slave'} at 127.0.0.1:41184
I20260709 12:10:44.446473 12043 raft_consensus.cc:493] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:44.446616 12118 raft_consensus.cc:2468] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 3.
I20260709 12:10:44.446938 12043 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:44.449692 12008 leader_election.cc:304] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:44.451215 12301 raft_consensus.cc:2804] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:44.452087 11969 raft_consensus.cc:2468] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 3.
I20260709 12:10:44.454659 12301 raft_consensus.cc:697] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:44.455926 12301 consensus_queue.cc:237] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.456283 12043 raft_consensus.cc:515] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.459705 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:44.460361 11969 raft_consensus.cc:3055] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:10:44.460675 11969 raft_consensus.cc:740] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:44.460530 12118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c722d809fe204d3f86b3045aff4a7fab" candidate_uuid: "997e54add8aa4783a09f5aa876db8fb3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
I20260709 12:10:44.461378 12118 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:44.462088 11969 consensus_queue.cc:260] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.463327 11969 raft_consensus.cc:3060] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:10:44.473942 11969 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 4.
I20260709 12:10:44.477516 12011 leader_election.cc:304] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9b406eb70142dea09afbd266bd696f, 997e54add8aa4783a09f5aa876db8fb3; no voters: 
I20260709 12:10:44.478643 12301 raft_consensus.cc:2804] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:10:44.479300 12301 raft_consensus.cc:697] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 4 LEADER]: Becoming Leader. State: Replica: 997e54add8aa4783a09f5aa876db8fb3, State: Running, Role: LEADER
I20260709 12:10:44.480371 12301 consensus_queue.cc:237] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.481737 11866 catalog_manager.cc:5697] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 2 to 3, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 3 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:44.482816 12118 raft_consensus.cc:2468] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 997e54add8aa4783a09f5aa876db8fb3 in term 4.
I20260709 12:10:44.483678 12043 leader_election.cc:290] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [CANDIDATE]: Term 4 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131:37823)
I20260709 12:10:44.496970 11866 catalog_manager.cc:5697] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 reported cstate change: term changed from 3 to 4, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130). New cstate: current_term: 4 leader_uuid: "997e54add8aa4783a09f5aa876db8fb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:44.655139 11969 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 5 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:10:44.656931 12320 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:10:44.669703 12043 raft_consensus.cc:1275] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 5 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:10:44.671530 12341 consensus_queue.cc:1048] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:10:44.712862 12329 raft_consensus.cc:993] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: : Instructing follower bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:44.713444 12329 raft_consensus.cc:1081] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Signalling peer bb6e4947dc5e4a8595ab661f2ea6c734 to start an election
I20260709 12:10:44.715804 12119 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847"
dest_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734"
 from {username='slave'} at 127.0.0.1:49628
I20260709 12:10:44.716490 12119 raft_consensus.cc:493] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:10:44.716944 12119 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:44.723091 12119 raft_consensus.cc:515] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.724849 12119 leader_election.cc:290] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 3 election: Requested vote from peers 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093), 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343)
I20260709 12:10:44.726171 11969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c9b406eb70142dea09afbd266bd696f"
I20260709 12:10:44.726320 12043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edf4111c57784cda9fd2e092fbd68847" candidate_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "997e54add8aa4783a09f5aa876db8fb3"
I20260709 12:10:44.726922 11969 raft_consensus.cc:3055] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:10:44.727077 12043 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:44.727286 11969 raft_consensus.cc:740] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c9b406eb70142dea09afbd266bd696f, State: Running, Role: LEADER
I20260709 12:10:44.728145 11969 consensus_queue.cc:260] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.729343 11969 raft_consensus.cc:3060] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:10:44.734630 12043 raft_consensus.cc:2468] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 3.
I20260709 12:10:44.736032 12085 leader_election.cc:304] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 997e54add8aa4783a09f5aa876db8fb3, bb6e4947dc5e4a8595ab661f2ea6c734; no voters: 
I20260709 12:10:44.736532 11969 raft_consensus.cc:2468] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6e4947dc5e4a8595ab661f2ea6c734 in term 3.
I20260709 12:10:44.737082 12341 raft_consensus.cc:2804] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:10:44.737847 12341 raft_consensus.cc:697] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Becoming Leader. State: Replica: bb6e4947dc5e4a8595ab661f2ea6c734, State: Running, Role: LEADER
I20260709 12:10:44.738868 12341 consensus_queue.cc:237] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } }
I20260709 12:10:44.747231 11867 catalog_manager.cc:5697] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 reported cstate change: term changed from 2 to 3, leader changed from 3c9b406eb70142dea09afbd266bd696f (127.11.11.129) to bb6e4947dc5e4a8595ab661f2ea6c734 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:44.931408 12119 raft_consensus.cc:1275] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:44.932958 12301 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:10:44.945469 11969 raft_consensus.cc:1275] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:10:44.947367 12348 consensus_queue.cc:1048] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:10:44.954869 11970 raft_consensus.cc:1275] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:44.956382 12348 consensus_queue.cc:1048] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:10:44.970682 12119 raft_consensus.cc:1275] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Refusing update from remote peer 997e54add8aa4783a09f5aa876db8fb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:44.972486 12301 consensus_queue.cc:1048] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6e4947dc5e4a8595ab661f2ea6c734" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:45.117106 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:45.123715 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:45.124323 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:45.216571 11969 raft_consensus.cc:1275] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:45.218317 12341 consensus_queue.cc:1048] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9b406eb70142dea09afbd266bd696f" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 34093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:10:45.233950 12043 raft_consensus.cc:1275] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Refusing update from remote peer bb6e4947dc5e4a8595ab661f2ea6c734: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:10:45.236021 12353 consensus_queue.cc:1048] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [LEADER]: Connected to new peer: Peer: permanent_uuid: "997e54add8aa4783a09f5aa876db8fb3" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:10:46.125507 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:46.131942 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:46.132493 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:47.133559 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:47.139238 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:47.139743 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:48.140950 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:48.148787 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:48.149386 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:49.150640 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:49.157106 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:49.157800 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:50.159067 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:50.165223 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:50.165748 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:51.166785 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:51.172607 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:51.173056 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:52.174088 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:52.180842 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:52.181892 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:53.183589 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:10:53.192256 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:10:53.192950 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:10:54.237018 11310 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:10:54.237901 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
W20260709 12:10:54.247318 12011 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20260709 12:10:54.253319 12011 consensus_peers.cc:597] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 -> Peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Couldn't send request to peer 3c9b406eb70142dea09afbd266bd696f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:54.282893 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:10:54.283859 11310 tablet_replica.cc:333] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.284662 11310 raft_consensus.cc:2243] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.285324 11310 raft_consensus.cc:2272] T edf4111c57784cda9fd2e092fbd68847 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.287559 11310 tablet_replica.cc:333] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.288223 11310 raft_consensus.cc:2243] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.289109 11310 raft_consensus.cc:2272] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.291114 11310 tablet_replica.cc:333] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.291703 11310 raft_consensus.cc:2243] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.292200 11310 raft_consensus.cc:2272] T a6401f8748164c969e0dfe96d0cbf8dd P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.294327 11310 tablet_replica.cc:333] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.294921 11310 raft_consensus.cc:2243] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.295419 11310 raft_consensus.cc:2272] T c722d809fe204d3f86b3045aff4a7fab P 3c9b406eb70142dea09afbd266bd696f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.297385 11310 tablet_replica.cc:333] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.297971 11310 raft_consensus.cc:2243] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.298509 11310 raft_consensus.cc:2272] T 9751272f5592489a95216b06be6793ff P 3c9b406eb70142dea09afbd266bd696f [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.300537 11310 tablet_replica.cc:333] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.301061 11310 raft_consensus.cc:2243] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.301514 11310 raft_consensus.cc:2272] T ac5c5d0844a545b4b511b3fd6239cb53 P 3c9b406eb70142dea09afbd266bd696f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.303555 11310 tablet_replica.cc:333] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.304059 11310 raft_consensus.cc:2243] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.304926 11310 raft_consensus.cc:2272] T c8d3b14c9ba14a48a6f7a51c571f49cc P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.306967 11310 tablet_replica.cc:333] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.307463 11310 raft_consensus.cc:2243] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.308302 11310 raft_consensus.cc:2272] T b1836d0ddf8c40b0a3961a899546dabb P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.310290 11310 tablet_replica.cc:333] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.310796 11310 raft_consensus.cc:2243] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.311300 11310 raft_consensus.cc:2272] T bae69c80fd414289b43cc35643e55ca4 P 3c9b406eb70142dea09afbd266bd696f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.313268 11310 tablet_replica.cc:333] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f: stopping tablet replica
I20260709 12:10:54.313876 11310 raft_consensus.cc:2243] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.314709 11310 raft_consensus.cc:2272] T ce32af3d4f724f0db85f77eb35ba228c P 3c9b406eb70142dea09afbd266bd696f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.352044 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
W20260709 12:10:54.357817 12085 consensus_peers.cc:597] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 -> Peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Couldn't send request to peer 3c9b406eb70142dea09afbd266bd696f. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:34093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:54.376959 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
W20260709 12:10:54.385766 12085 consensus_peers.cc:597] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 -> Peer 3c9b406eb70142dea09afbd266bd696f (127.11.11.129:34093): Couldn't send request to peer 3c9b406eb70142dea09afbd266bd696f. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:34093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:54.418449 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:10:54.419301 11310 tablet_replica.cc:333] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.420172 11310 raft_consensus.cc:2243] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.420962 11310 raft_consensus.cc:2272] T b1836d0ddf8c40b0a3961a899546dabb P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.423224 11310 tablet_replica.cc:333] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.423704 11310 raft_consensus.cc:2243] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.424088 11310 raft_consensus.cc:2272] T bae69c80fd414289b43cc35643e55ca4 P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.426412 11310 tablet_replica.cc:333] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.426959 11310 raft_consensus.cc:2243] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.427455 11310 raft_consensus.cc:2272] T edf4111c57784cda9fd2e092fbd68847 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.429248 11310 tablet_replica.cc:333] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.429808 11310 raft_consensus.cc:2243] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.430197 11310 raft_consensus.cc:2272] T ce32af3d4f724f0db85f77eb35ba228c P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.431854 11310 tablet_replica.cc:333] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.432296 11310 raft_consensus.cc:2243] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.432744 11310 raft_consensus.cc:2272] T 5a3e7f70af184c4c8d2d2263e5a2c033 P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.434602 11310 tablet_replica.cc:333] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.435101 11310 raft_consensus.cc:2243] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.435904 11310 raft_consensus.cc:2272] T c722d809fe204d3f86b3045aff4a7fab P 997e54add8aa4783a09f5aa876db8fb3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.437801 11310 tablet_replica.cc:333] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.438335 11310 raft_consensus.cc:2243] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.438778 11310 raft_consensus.cc:2272] T 9751272f5592489a95216b06be6793ff P 997e54add8aa4783a09f5aa876db8fb3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.440897 11310 tablet_replica.cc:333] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.441401 11310 raft_consensus.cc:2243] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.442749 11310 raft_consensus.cc:2272] T a6401f8748164c969e0dfe96d0cbf8dd P 997e54add8aa4783a09f5aa876db8fb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.444646 11310 tablet_replica.cc:333] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.445202 11310 raft_consensus.cc:2243] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.446080 11310 raft_consensus.cc:2272] T ac5c5d0844a545b4b511b3fd6239cb53 P 997e54add8aa4783a09f5aa876db8fb3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.447894 11310 tablet_replica.cc:333] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3: stopping tablet replica
I20260709 12:10:54.448366 11310 raft_consensus.cc:2243] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.448786 11310 raft_consensus.cc:2272] T c8d3b14c9ba14a48a6f7a51c571f49cc P 997e54add8aa4783a09f5aa876db8fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.488390 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
W20260709 12:10:54.503799 12085 consensus_peers.cc:597] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 -> Peer 997e54add8aa4783a09f5aa876db8fb3 (127.11.11.130:46343): Couldn't send request to peer 997e54add8aa4783a09f5aa876db8fb3. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:46343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:54.513054 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
I20260709 12:10:54.546839 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:10:54.547770 11310 tablet_replica.cc:333] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.548525 11310 raft_consensus.cc:2243] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.549047 11310 raft_consensus.cc:2272] T ce32af3d4f724f0db85f77eb35ba228c P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.551375 11310 tablet_replica.cc:333] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.551954 11310 raft_consensus.cc:2243] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.552875 11310 raft_consensus.cc:2272] T edf4111c57784cda9fd2e092fbd68847 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.555133 11310 tablet_replica.cc:333] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.555757 11310 raft_consensus.cc:2243] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.556695 11310 raft_consensus.cc:2272] T bae69c80fd414289b43cc35643e55ca4 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.558805 11310 tablet_replica.cc:333] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.559376 11310 raft_consensus.cc:2243] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.559856 11310 raft_consensus.cc:2272] T b1836d0ddf8c40b0a3961a899546dabb P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.562011 11310 tablet_replica.cc:333] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.562587 11310 raft_consensus.cc:2243] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.563072 11310 raft_consensus.cc:2272] T ac5c5d0844a545b4b511b3fd6239cb53 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.565056 11310 tablet_replica.cc:333] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.565582 11310 raft_consensus.cc:2243] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.566133 11310 raft_consensus.cc:2272] T c8d3b14c9ba14a48a6f7a51c571f49cc P bb6e4947dc5e4a8595ab661f2ea6c734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.568156 11310 tablet_replica.cc:333] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.568768 11310 raft_consensus.cc:2243] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.569278 11310 raft_consensus.cc:2272] T c722d809fe204d3f86b3045aff4a7fab P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.571499 11310 tablet_replica.cc:333] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.572242 11310 raft_consensus.cc:2243] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.573040 11310 raft_consensus.cc:2272] T 9751272f5592489a95216b06be6793ff P bb6e4947dc5e4a8595ab661f2ea6c734 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.575661 11310 tablet_replica.cc:333] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.576256 11310 raft_consensus.cc:2243] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.576756 11310 raft_consensus.cc:2272] T a6401f8748164c969e0dfe96d0cbf8dd P bb6e4947dc5e4a8595ab661f2ea6c734 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.578724 11310 tablet_replica.cc:333] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734: stopping tablet replica
I20260709 12:10:54.579190 11310 raft_consensus.cc:2243] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:10:54.579646 11310 raft_consensus.cc:2272] T 5a3e7f70af184c4c8d2d2263e5a2c033 P bb6e4947dc5e4a8595ab661f2ea6c734 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.631598 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:10:54.656970 11310 master.cc:562] Master@127.11.11.190:39185 shutting down...
I20260709 12:10:54.689831 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:10:54.690624 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:10:54.691004 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 32474939a0ce4c568817895ac69ed49f: stopping tablet replica
I20260709 12:10:54.713122 11310 master.cc:584] Master@127.11.11.190:39185 shutdown complete.
3c9b406eb70142dea09afbd266bd696f  5
997e54add8aa4783a09f5aa876db8fb3  4
bb6e4947dc5e4a8595ab661f2ea6c734  1
3c9b406eb70142dea09afbd266bd696f  3
997e54add8aa4783a09f5aa876db8fb3  4
bb6e4947dc5e4a8595ab661f2ea6c734  3
3c9b406eb70142dea09afbd266bd696f  8
997e54add8aa4783a09f5aa876db8fb3  1
bb6e4947dc5e4a8595ab661f2ea6c734  1
3c9b406eb70142dea09afbd266bd696f  4
997e54add8aa4783a09f5aa876db8fb3  3
bb6e4947dc5e4a8595ab661f2ea6c734  3
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30244 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:10:54.766089 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:35399
I20260709 12:10:54.767297 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:54.775389 12371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:54.775591 12373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:54.775632 12370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:54.776574 11310 server_base.cc:1061] running on GCE node
I20260709 12:10:54.777976 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:54.778194 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:54.778409 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599054778391 us; error 0 us; skew 500 ppm
I20260709 12:10:54.779105 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:54.782470 11310 webserver.cc:533] Webserver started at http://127.11.11.190:43905/ using document root <none> and password file <none>
I20260709 12:10:54.783104 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:54.783311 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:54.783648 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:54.785043 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "2d10571632e7445d99e8f1c750c96f70"
format_stamp: "Formatted at 2026-07-09 12:10:54 on dist-test-slave-4qvp"
I20260709 12:10:54.790434 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:10:54.794831 12378 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:54.795866 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:10:54.796146 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "2d10571632e7445d99e8f1c750c96f70"
format_stamp: "Formatted at 2026-07-09 12:10:54 on dist-test-slave-4qvp"
I20260709 12:10:54.796408 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:54.811816 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:54.813074 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:54.860761 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:35399
I20260709 12:10:54.860939 12429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:35399 every 8 connection(s)
I20260709 12:10:54.866357 12430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:54.879297 12430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Bootstrap starting.
I20260709 12:10:54.884868 12430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:54.890292 12430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: No bootstrap required, opened a new log
I20260709 12:10:54.892920 12430 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10571632e7445d99e8f1c750c96f70" member_type: VOTER }
I20260709 12:10:54.893417 12430 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:54.893766 12430 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10571632e7445d99e8f1c750c96f70, State: Initialized, Role: FOLLOWER
I20260709 12:10:54.894345 12430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10571632e7445d99e8f1c750c96f70" member_type: VOTER }
I20260709 12:10:54.894858 12430 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:10:54.895052 12430 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:10:54.895282 12430 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:54.900637 12430 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10571632e7445d99e8f1c750c96f70" member_type: VOTER }
I20260709 12:10:54.901243 12430 leader_election.cc:304] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d10571632e7445d99e8f1c750c96f70; no voters: 
I20260709 12:10:54.902802 12430 leader_election.cc:290] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:10:54.903316 12433 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:54.905097 12433 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 1 LEADER]: Becoming Leader. State: Replica: 2d10571632e7445d99e8f1c750c96f70, State: Running, Role: LEADER
I20260709 12:10:54.905975 12433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10571632e7445d99e8f1c750c96f70" member_type: VOTER }
I20260709 12:10:54.906284 12430 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:10:54.911406 12434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d10571632e7445d99e8f1c750c96f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10571632e7445d99e8f1c750c96f70" member_type: VOTER } }
I20260709 12:10:54.912283 12435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d10571632e7445d99e8f1c750c96f70. Latest consensus state: current_term: 1 leader_uuid: "2d10571632e7445d99e8f1c750c96f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10571632e7445d99e8f1c750c96f70" member_type: VOTER } }
I20260709 12:10:54.913064 12435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [sys.catalog]: This master's current role is: LEADER
I20260709 12:10:54.912303 12434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [sys.catalog]: This master's current role is: LEADER
I20260709 12:10:54.915318 12442 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:10:54.921825 12442 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:10:54.924477 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:10:54.932296 12442 catalog_manager.cc:1383] Generated new cluster ID: f2cef5811f3d4fb0a307411474f1a5d4
I20260709 12:10:54.932644 12442 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:10:54.958449 12442 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:10:54.959960 12442 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:10:54.974500 12442 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Generated new TSK 0
I20260709 12:10:54.975178 12442 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:10:54.992206 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:55.000644 12451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:55.002207 12452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:55.004753 12454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:55.005324 11310 server_base.cc:1061] running on GCE node
I20260709 12:10:55.006620 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:55.006901 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:55.007102 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599055007081 us; error 0 us; skew 500 ppm
I20260709 12:10:55.008294 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:55.011473 11310 webserver.cc:533] Webserver started at http://127.11.11.129:38823/ using document root <none> and password file <none>
I20260709 12:10:55.012108 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:55.012295 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:55.012681 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:55.014153 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "6d4ece942fd5414998215d26edf91001"
format_stamp: "Formatted at 2026-07-09 12:10:55 on dist-test-slave-4qvp"
I20260709 12:10:55.019702 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:10:55.024235 12459 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.025393 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:10:55.025756 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "6d4ece942fd5414998215d26edf91001"
format_stamp: "Formatted at 2026-07-09 12:10:55 on dist-test-slave-4qvp"
I20260709 12:10:55.026072 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:55.055516 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:55.057050 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:55.058957 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:10:55.062129 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:10:55.062350 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.062719 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:10:55.062952 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.118764 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:38859
I20260709 12:10:55.118878 12521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:38859 every 8 connection(s)
I20260709 12:10:55.125274 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:55.135699 12527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:55.137296 12526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:55.141234 11310 server_base.cc:1061] running on GCE node
I20260709 12:10:55.141351 12522 heartbeater.cc:344] Connected to a master server at 127.11.11.190:35399
I20260709 12:10:55.141970 12522 heartbeater.cc:461] Registering TS with master...
W20260709 12:10:55.142983 12529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:55.143091 12522 heartbeater.cc:507] Master 127.11.11.190:35399 requested a full tablet report, sending...
I20260709 12:10:55.144160 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:55.144429 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:55.144659 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599055144639 us; error 0 us; skew 500 ppm
I20260709 12:10:55.145802 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:55.146308 12395 ts_manager.cc:194] Registered new tserver with Master: 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:10:55.148820 11310 webserver.cc:533] Webserver started at http://127.11.11.130:38457/ using document root <none> and password file <none>
I20260709 12:10:55.149142 12395 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42850
I20260709 12:10:55.149374 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:55.149742 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:55.150048 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:55.151399 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "0a29b5d9f9d44298b02ffcd460811eef"
format_stamp: "Formatted at 2026-07-09 12:10:55 on dist-test-slave-4qvp"
I20260709 12:10:55.156473 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:10:55.160733 12534 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.161914 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 12:10:55.162262 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "0a29b5d9f9d44298b02ffcd460811eef"
format_stamp: "Formatted at 2026-07-09 12:10:55 on dist-test-slave-4qvp"
I20260709 12:10:55.162570 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:55.178486 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:55.180001 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:55.181864 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:10:55.184604 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:10:55.184811 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.185158 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:10:55.185313 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.237984 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:32927
I20260709 12:10:55.238193 12596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:32927 every 8 connection(s)
I20260709 12:10:55.243702 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:10:55.252226 12600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:10:55.253747 12601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:55.257751 11310 server_base.cc:1061] running on GCE node
W20260709 12:10:55.259294 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:10:55.260099 12597 heartbeater.cc:344] Connected to a master server at 127.11.11.190:35399
I20260709 12:10:55.260519 12597 heartbeater.cc:461] Registering TS with master...
I20260709 12:10:55.260633 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:10:55.260954 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:10:55.261142 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599055261109 us; error 0 us; skew 500 ppm
I20260709 12:10:55.261513 12597 heartbeater.cc:507] Master 127.11.11.190:35399 requested a full tablet report, sending...
I20260709 12:10:55.261905 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:10:55.264308 12395 ts_manager.cc:194] Registered new tserver with Master: 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:55.265195 11310 webserver.cc:533] Webserver started at http://127.11.11.131:35707/ using document root <none> and password file <none>
I20260709 12:10:55.265893 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:10:55.266136 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:10:55.266283 12395 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42864
I20260709 12:10:55.266458 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:10:55.268082 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "32889a507a6b4c74aec421ffbfd1e25d"
format_stamp: "Formatted at 2026-07-09 12:10:55 on dist-test-slave-4qvp"
I20260709 12:10:55.273432 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:10:55.279415 12608 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.280745 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:10:55.281296 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "32889a507a6b4c74aec421ffbfd1e25d"
format_stamp: "Formatted at 2026-07-09 12:10:55 on dist-test-slave-4qvp"
I20260709 12:10:55.281821 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:10:55.307051 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:10:55.308599 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:10:55.310102 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:10:55.312525 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:10:55.312770 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.313030 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:10:55.313212 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:10:55.363068 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:35471
I20260709 12:10:55.363215 12670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:35471 every 8 connection(s)
I20260709 12:10:55.380045 12671 heartbeater.cc:344] Connected to a master server at 127.11.11.190:35399
I20260709 12:10:55.380541 12671 heartbeater.cc:461] Registering TS with master...
I20260709 12:10:55.381428 12671 heartbeater.cc:507] Master 127.11.11.190:35399 requested a full tablet report, sending...
I20260709 12:10:55.383961 12395 ts_manager.cc:194] Registered new tserver with Master: 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:55.384444 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017783516s
I20260709 12:10:55.385577 12395 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42880
I20260709 12:10:55.418891 12395 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42886:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:10:55.426892 12395 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:10:55.667136 12487 tablet_service.cc:1511] Processing CreateTablet for tablet 6fad7500a4eb45afa94184a247d796b8 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:10:55.669152 12487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fad7500a4eb45afa94184a247d796b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.669476 12635 tablet_service.cc:1511] Processing CreateTablet for tablet c13f916d8b4d463182a1d1eb50150e31 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:10:55.669755 12558 tablet_service.cc:1511] Processing CreateTablet for tablet 43f1b7080b6a4c329281b7c405f5c998 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:10:55.671104 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13f916d8b4d463182a1d1eb50150e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.671581 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f1b7080b6a4c329281b7c405f5c998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.675071 12633 tablet_service.cc:1511] Processing CreateTablet for tablet a3a0781287514c758f107d79c1f96be4 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:10:55.676347 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 509f0430baf04e3393879d82fb9d10f9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:10:55.678282 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509f0430baf04e3393879d82fb9d10f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.678994 12630 tablet_service.cc:1511] Processing CreateTablet for tablet c126dc4ef9114297874fa66b1a0c56b0 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:10:55.679782 12486 tablet_service.cc:1511] Processing CreateTablet for tablet c13f916d8b4d463182a1d1eb50150e31 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:10:55.681383 12486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13f916d8b4d463182a1d1eb50150e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.681548 12632 tablet_service.cc:1511] Processing CreateTablet for tablet 43f1b7080b6a4c329281b7c405f5c998 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:10:55.682935 12561 tablet_service.cc:1511] Processing CreateTablet for tablet c13f916d8b4d463182a1d1eb50150e31 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:10:55.683475 12634 tablet_service.cc:1511] Processing CreateTablet for tablet 509f0430baf04e3393879d82fb9d10f9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:10:55.683913 12484 tablet_service.cc:1511] Processing CreateTablet for tablet a3a0781287514c758f107d79c1f96be4 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:10:55.684581 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13f916d8b4d463182a1d1eb50150e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.685370 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a0781287514c758f107d79c1f96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.684979 12483 tablet_service.cc:1511] Processing CreateTablet for tablet 43f1b7080b6a4c329281b7c405f5c998 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:10:55.686707 12483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f1b7080b6a4c329281b7c405f5c998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.687280 12557 tablet_service.cc:1511] Processing CreateTablet for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:10:55.688025 12485 tablet_service.cc:1511] Processing CreateTablet for tablet 509f0430baf04e3393879d82fb9d10f9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:10:55.675102 12559 tablet_service.cc:1511] Processing CreateTablet for tablet a3a0781287514c758f107d79c1f96be4 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:10:55.689706 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509f0430baf04e3393879d82fb9d10f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.664968 12562 tablet_service.cc:1511] Processing CreateTablet for tablet 6fad7500a4eb45afa94184a247d796b8 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:10:55.690157 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:10:55.691697 12629 tablet_service.cc:1511] Processing CreateTablet for tablet 54875fd261ee4c328e3d022d6a9418d8 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:10:55.692271 12556 tablet_service.cc:1511] Processing CreateTablet for tablet c126dc4ef9114297874fa66b1a0c56b0 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:10:55.667258 12636 tablet_service.cc:1511] Processing CreateTablet for tablet 6fad7500a4eb45afa94184a247d796b8 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:10:55.692763 12628 tablet_service.cc:1511] Processing CreateTablet for tablet b39f4bd5c55347ada5f73766216828ca (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:10:55.684844 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:10:55.697213 12627 tablet_service.cc:1511] Processing CreateTablet for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:10:55.697222 12480 tablet_service.cc:1511] Processing CreateTablet for tablet 54875fd261ee4c328e3d022d6a9418d8 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:10:55.698818 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a0781287514c758f107d79c1f96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.700028 12555 tablet_service.cc:1511] Processing CreateTablet for tablet 54875fd261ee4c328e3d022d6a9418d8 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:10:55.693961 12481 tablet_service.cc:1511] Processing CreateTablet for tablet c126dc4ef9114297874fa66b1a0c56b0 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:10:55.703991 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.704300 12554 tablet_service.cc:1511] Processing CreateTablet for tablet b39f4bd5c55347ada5f73766216828ca (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:10:55.706480 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c126dc4ef9114297874fa66b1a0c56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.707530 12479 tablet_service.cc:1511] Processing CreateTablet for tablet b39f4bd5c55347ada5f73766216828ca (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:10:55.708338 12553 tablet_service.cc:1511] Processing CreateTablet for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:10:55.709410 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.710319 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f1b7080b6a4c329281b7c405f5c998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.712783 12479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39f4bd5c55347ada5f73766216828ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.714668 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fad7500a4eb45afa94184a247d796b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.717226 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.717984 12480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54875fd261ee4c328e3d022d6a9418d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.718258 12629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54875fd261ee4c328e3d022d6a9418d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.720059 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509f0430baf04e3393879d82fb9d10f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.722143 12557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.725039 12559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a0781287514c758f107d79c1f96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.726076 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fad7500a4eb45afa94184a247d796b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.726716 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:10:55.729028 12481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c126dc4ef9114297874fa66b1a0c56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.729885 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.730016 12554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39f4bd5c55347ada5f73766216828ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.732038 12556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c126dc4ef9114297874fa66b1a0c56b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.733376 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b39f4bd5c55347ada5f73766216828ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.734786 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.738169 12555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54875fd261ee4c328e3d022d6a9418d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.795455 12479 tablet_service.cc:1511] Processing CreateTablet for tablet 40d0e235c01b4b7f89948ab708403a01 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:10:55.830782 12555 tablet_service.cc:1511] Processing CreateTablet for tablet 65037753870c4624995dde1173266f49 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:10:55.836602 12555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65037753870c4624995dde1173266f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.845273 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 40d0e235c01b4b7f89948ab708403a01 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:10:55.847134 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d0e235c01b4b7f89948ab708403a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.846905 12558 tablet_service.cc:1511] Processing CreateTablet for tablet 40d0e235c01b4b7f89948ab708403a01 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:10:55.847769 12484 tablet_service.cc:1511] Processing CreateTablet for tablet e3648047ca0742a2bfaa45ef2af517e7 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:10:55.848451 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d0e235c01b4b7f89948ab708403a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.856999 12561 tablet_service.cc:1511] Processing CreateTablet for tablet e3648047ca0742a2bfaa45ef2af517e7 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:10:55.858938 12479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d0e235c01b4b7f89948ab708403a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.861130 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 6f701a23559945868ce6e3da5866f95c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:10:55.862902 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f701a23559945868ce6e3da5866f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.863404 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 65037753870c4624995dde1173266f49 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:10:55.895565 12688 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:55.895749 12485 tablet_service.cc:1511] Processing CreateTablet for tablet 3eadf951b63a43fd993cea2c96d5214f (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:10:55.895565 12686 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:55.898409 12687 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:55.893712 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 6f701a23559945868ce6e3da5866f95c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:10:55.900275 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648047ca0742a2bfaa45ef2af517e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.901253 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eadf951b63a43fd993cea2c96d5214f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.891666 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648047ca0742a2bfaa45ef2af517e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.876951 12629 tablet_service.cc:1511] Processing CreateTablet for tablet e3648047ca0742a2bfaa45ef2af517e7 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:10:55.910883 12634 tablet_service.cc:1511] Processing CreateTablet for tablet 65037753870c4624995dde1173266f49 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
W20260709 12:10:55.924469 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:10:55.926060 12554 tablet_service.cc:1511] Processing CreateTablet for tablet 3eadf951b63a43fd993cea2c96d5214f (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:10:55.929064 12554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eadf951b63a43fd993cea2c96d5214f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.929445 12688 tablet_bootstrap.cc:654] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:55.919682 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65037753870c4624995dde1173266f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.945120 12686 tablet_bootstrap.cc:654] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:55.964092 12553 tablet_service.cc:1511] Processing CreateTablet for tablet c793ebd4de24492397303f9a6e6bc62a (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:10:55.965885 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c793ebd4de24492397303f9a6e6bc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.969956 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f701a23559945868ce6e3da5866f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.979494 12630 tablet_service.cc:1511] Processing CreateTablet for tablet 6f701a23559945868ce6e3da5866f95c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:10:55.980731 12635 tablet_service.cc:1511] Processing CreateTablet for tablet 3eadf951b63a43fd993cea2c96d5214f (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:10:55.986063 12687 tablet_bootstrap.cc:654] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:55.993144 12562 tablet_service.cc:1511] Processing CreateTablet for tablet 2dbcfa3bff034f5ab82e2ae802266099 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:10:55.994764 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbcfa3bff034f5ab82e2ae802266099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:55.999243 12632 tablet_service.cc:1511] Processing CreateTablet for tablet c793ebd4de24492397303f9a6e6bc62a (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:10:56.013784 12629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648047ca0742a2bfaa45ef2af517e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.018623 12627 tablet_service.cc:1511] Processing CreateTablet for tablet 2dbcfa3bff034f5ab82e2ae802266099 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:10:56.029382 12634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65037753870c4624995dde1173266f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.028376 12485 tablet_service.cc:1511] Processing CreateTablet for tablet c793ebd4de24492397303f9a6e6bc62a (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:10:56.031322 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbcfa3bff034f5ab82e2ae802266099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.031769 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c793ebd4de24492397303f9a6e6bc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.025755 12486 tablet_service.cc:1511] Processing CreateTablet for tablet 2dbcfa3bff034f5ab82e2ae802266099 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:10:56.034031 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 3066846eded04c878b1f03ecb43b1c79 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:10:56.037720 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3066846eded04c878b1f03ecb43b1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.042268 12486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbcfa3bff034f5ab82e2ae802266099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.034044 12483 tablet_service.cc:1511] Processing CreateTablet for tablet 3066846eded04c878b1f03ecb43b1c79 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:10:56.050871 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eadf951b63a43fd993cea2c96d5214f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.051118 12483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3066846eded04c878b1f03ecb43b1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.035454 12686 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:56.054837 12686 ts_tablet_manager.cc:1403] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.160s	user 0.008s	sys 0.017s
I20260709 12:10:56.055959 12688 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:56.056512 12688 ts_tablet_manager.cc:1403] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.161s	user 0.016s	sys 0.007s
I20260709 12:10:56.058049 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c793ebd4de24492397303f9a6e6bc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.059619 12688 raft_consensus.cc:359] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.060405 12688 raft_consensus.cc:385] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.060721 12688 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.062291 12688 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.062729 12686 raft_consensus.cc:359] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.063568 12686 raft_consensus.cc:385] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.063885 12686 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.064075 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f701a23559945868ce6e3da5866f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.074992 12480 tablet_service.cc:1511] Processing CreateTablet for tablet 79b687d20a534e70865ef902fbd0d4a5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:10:56.076632 12480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b687d20a534e70865ef902fbd0d4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.077471 12671 heartbeater.cc:499] Master 127.11.11.190:35399 was elected leader, sending a full tablet report...
I20260709 12:10:56.078660 12636 tablet_service.cc:1511] Processing CreateTablet for tablet 3066846eded04c878b1f03ecb43b1c79 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:10:56.078871 12558 tablet_service.cc:1511] Processing CreateTablet for tablet 79b687d20a534e70865ef902fbd0d4a5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:10:56.080152 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3066846eded04c878b1f03ecb43b1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.080157 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b687d20a534e70865ef902fbd0d4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.082636 12692 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:56.083202 12692 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.091907 12633 tablet_service.cc:1511] Processing CreateTablet for tablet 79b687d20a534e70865ef902fbd0d4a5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:10:56.094503 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b687d20a534e70865ef902fbd0d4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.092718 12688 ts_tablet_manager.cc:1434] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.036s	user 0.003s	sys 0.006s
I20260709 12:10:56.098869 12688 tablet_bootstrap.cc:492] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:56.100334 12687 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:56.100912 12687 ts_tablet_manager.cc:1403] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.203s	user 0.011s	sys 0.009s
I20260709 12:10:56.101572 12686 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.104563 12687 raft_consensus.cc:359] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.105393 12687 raft_consensus.cc:385] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.112344 12688 tablet_bootstrap.cc:654] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
W20260709 12:10:56.125976 12672 tablet.cc:2406] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:10:56.148437 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:56.128720 12484 tablet_service.cc:1511] Processing CreateTablet for tablet 59beb90f79b0450ea2fbb2b7e405261d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:10:56.135493 12553 tablet_service.cc:1511] Processing CreateTablet for tablet 59beb90f79b0450ea2fbb2b7e405261d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:10:56.147064 12687 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
W20260709 12:10:56.150673 12610 leader_election.cc:343] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:56.150916 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59beb90f79b0450ea2fbb2b7e405261d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.151677 12686 ts_tablet_manager.cc:1434] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.096s	user 0.006s	sys 0.005s
I20260709 12:10:56.153005 12686 tablet_bootstrap.cc:492] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:56.156265 12558 tablet_service.cc:1511] Processing CreateTablet for tablet bb9177ec2fb345c4bff68c8c8a132af6 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:10:56.147140 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 59beb90f79b0450ea2fbb2b7e405261d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:10:56.137804 12692 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:56.149405 12522 heartbeater.cc:499] Master 127.11.11.190:35399 was elected leader, sending a full tablet report...
I20260709 12:10:56.162736 12633 tablet_service.cc:1511] Processing CreateTablet for tablet bb9177ec2fb345c4bff68c8c8a132af6 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:10:56.162590 12687 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.177904 12485 tablet_service.cc:1511] Processing CreateTablet for tablet bb9177ec2fb345c4bff68c8c8a132af6 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:10:56.180735 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9177ec2fb345c4bff68c8c8a132af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.190582 12562 tablet_service.cc:1511] Processing CreateTablet for tablet feae6096d5e346a3b548be8b5e7488d4 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:10:56.184278 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59beb90f79b0450ea2fbb2b7e405261d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.191530 12688 tablet_bootstrap.cc:492] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:56.192274 12688 ts_tablet_manager.cc:1403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.094s	user 0.009s	sys 0.005s
I20260709 12:10:56.195981 12688 raft_consensus.cc:359] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.202155 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59beb90f79b0450ea2fbb2b7e405261d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.204717 12628 tablet_service.cc:1511] Processing CreateTablet for tablet feae6096d5e346a3b548be8b5e7488d4 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:10:56.206662 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9177ec2fb345c4bff68c8c8a132af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.209084 12688 raft_consensus.cc:385] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.207944 12482 tablet_service.cc:1511] Processing CreateTablet for tablet feae6096d5e346a3b548be8b5e7488d4 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:10:56.209501 12561 tablet_service.cc:1511] Processing CreateTablet for tablet 44b2b570b6dd4e36bc98e56e097afd62 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:10:56.213521 12688 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.210029 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feae6096d5e346a3b548be8b5e7488d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.214354 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feae6096d5e346a3b548be8b5e7488d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.215600 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b2b570b6dd4e36bc98e56e097afd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.218667 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:10:56.226222 12609 leader_election.cc:343] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:56.226773 12609 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:56.227710 12692 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:56.230350 12597 heartbeater.cc:499] Master 127.11.11.190:35399 was elected leader, sending a full tablet report...
I20260709 12:10:56.230823 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 44b2b570b6dd4e36bc98e56e097afd62 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:10:56.232281 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b2b570b6dd4e36bc98e56e097afd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.233125 12687 ts_tablet_manager.cc:1434] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.132s	user 0.001s	sys 0.014s
I20260709 12:10:56.238628 12687 tablet_bootstrap.cc:492] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:56.238862 12686 tablet_bootstrap.cc:654] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.243377 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feae6096d5e346a3b548be8b5e7488d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.245222 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9177ec2fb345c4bff68c8c8a132af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.267225 12632 tablet_service.cc:1511] Processing CreateTablet for tablet 44b2b570b6dd4e36bc98e56e097afd62 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:10:56.268639 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b2b570b6dd4e36bc98e56e097afd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.268146 12688 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.278987 12688 ts_tablet_manager.cc:1434] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.086s	user 0.000s	sys 0.007s
I20260709 12:10:56.279426 12484 tablet_service.cc:1511] Processing CreateTablet for tablet 05ad55cc3c04480abd0546c50226f9c1 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:10:56.282084 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ad55cc3c04480abd0546c50226f9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.282234 12553 tablet_service.cc:1511] Processing CreateTablet for tablet 05ad55cc3c04480abd0546c50226f9c1 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:10:56.283773 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ad55cc3c04480abd0546c50226f9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.287258 12688 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:56.300063 12686 tablet_bootstrap.cc:492] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:56.300823 12686 ts_tablet_manager.cc:1403] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.148s	user 0.011s	sys 0.000s
I20260709 12:10:56.303045 12630 tablet_service.cc:1511] Processing CreateTablet for tablet 05ad55cc3c04480abd0546c50226f9c1 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:10:56.303890 12560 tablet_service.cc:1511] Processing CreateTablet for tablet a1d35de928f546349f88cbea7818041c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:10:56.304657 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ad55cc3c04480abd0546c50226f9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.305435 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d35de928f546349f88cbea7818041c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.306803 12480 tablet_service.cc:1511] Processing CreateTablet for tablet a1d35de928f546349f88cbea7818041c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:10:56.308444 12480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d35de928f546349f88cbea7818041c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.309463 12687 tablet_bootstrap.cc:654] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.316577 12686 raft_consensus.cc:359] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.328982 12688 tablet_bootstrap.cc:654] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.317489 12686 raft_consensus.cc:385] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.337699 12686 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.338740 12686 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.343324 12630 tablet_service.cc:1511] Processing CreateTablet for tablet a1d35de928f546349f88cbea7818041c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:10:56.343739 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 4dc881215d9f4498b162561e79143f2d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:10:56.344969 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1d35de928f546349f88cbea7818041c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.345280 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dc881215d9f4498b162561e79143f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.355736 12558 tablet_service.cc:1511] Processing CreateTablet for tablet 4dc881215d9f4498b162561e79143f2d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:10:56.357218 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dc881215d9f4498b162561e79143f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.357303 12632 tablet_service.cc:1511] Processing CreateTablet for tablet 4dc881215d9f4498b162561e79143f2d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:10:56.357908 12694 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:56.358520 12694 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.359285 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dc881215d9f4498b162561e79143f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.348433 12686 ts_tablet_manager.cc:1434] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.047s	user 0.008s	sys 0.001s
I20260709 12:10:56.365502 12694 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:56.368188 12686 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:56.388127 12562 tablet_service.cc:1511] Processing CreateTablet for tablet 0b62b58d64bf4dc2850a3ea4a3cb128a (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:10:56.397181 12686 tablet_bootstrap.cc:654] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.397611 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b62b58d64bf4dc2850a3ea4a3cb128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.398602 12687 tablet_bootstrap.cc:492] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:56.399221 12687 ts_tablet_manager.cc:1403] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.161s	user 0.006s	sys 0.018s
I20260709 12:10:56.415287 12688 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:56.415874 12688 ts_tablet_manager.cc:1403] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.129s	user 0.007s	sys 0.020s
I20260709 12:10:56.415596 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 0b62b58d64bf4dc2850a3ea4a3cb128a (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:10:56.416983 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b62b58d64bf4dc2850a3ea4a3cb128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.395993 12484 tablet_service.cc:1511] Processing CreateTablet for tablet 0b62b58d64bf4dc2850a3ea4a3cb128a (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:10:56.418614 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b62b58d64bf4dc2850a3ea4a3cb128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.426789 12628 tablet_service.cc:1511] Processing CreateTablet for tablet 256d044fd2e244d4be827104688867ee (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:10:56.428457 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 256d044fd2e244d4be827104688867ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.430838 12561 tablet_service.cc:1511] Processing CreateTablet for tablet 472e786b46074d868d00df67a0985afd (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:10:56.431586 12687 raft_consensus.cc:359] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.432327 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e786b46074d868d00df67a0985afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.432377 12687 raft_consensus.cc:385] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.432766 12687 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.434605 12688 raft_consensus.cc:359] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.434926 12633 tablet_service.cc:1511] Processing CreateTablet for tablet 472e786b46074d868d00df67a0985afd (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:10:56.436555 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e786b46074d868d00df67a0985afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.438010 12558 tablet_service.cc:1511] Processing CreateTablet for tablet 256d044fd2e244d4be827104688867ee (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:10:56.439474 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 256d044fd2e244d4be827104688867ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.435389 12688 raft_consensus.cc:385] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.444890 12688 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.445911 12688 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.448129 12688 ts_tablet_manager.cc:1434] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:10:56.449148 12688 tablet_bootstrap.cc:492] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:56.452414 12627 tablet_service.cc:1511] Processing CreateTablet for tablet a7a8c7204d6b44c9b3d6ea6332e9f824 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:10:56.453812 12694 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:56.454370 12694 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.456909 12694 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:56.458128 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a8c7204d6b44c9b3d6ea6332e9f824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.463806 12480 tablet_service.cc:1511] Processing CreateTablet for tablet 256d044fd2e244d4be827104688867ee (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:10:56.464093 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 472e786b46074d868d00df67a0985afd (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:10:56.464413 12485 tablet_service.cc:1511] Processing CreateTablet for tablet a7a8c7204d6b44c9b3d6ea6332e9f824 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:10:56.465404 12480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 256d044fd2e244d4be827104688867ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.465946 12486 tablet_service.cc:1511] Processing CreateTablet for tablet 93e029db6a8e4a7db72ad88760a6b7d3 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:10:56.470590 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a8c7204d6b44c9b3d6ea6332e9f824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.471297 12486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e029db6a8e4a7db72ad88760a6b7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.473536 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e786b46074d868d00df67a0985afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.473982 12687 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.482292 12687 ts_tablet_manager.cc:1434] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.083s	user 0.007s	sys 0.002s
I20260709 12:10:56.483428 12687 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:56.486471 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:56.513581 12646 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:56.487980 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:56.461314 12560 tablet_service.cc:1511] Processing CreateTablet for tablet a7a8c7204d6b44c9b3d6ea6332e9f824 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:10:56.490043 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:56.515018 12572 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:56.491080 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:56.491225 12553 tablet_service.cc:1511] Processing CreateTablet for tablet 93e029db6a8e4a7db72ad88760a6b7d3 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:10:56.516033 12461 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:10:56.518178 12461 leader_election.cc:343] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:56.493595 12688 tablet_bootstrap.cc:654] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.504902 12686 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:56.518946 12694 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:56.519335 12694 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:56.519351 12686 ts_tablet_manager.cc:1403] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.152s	user 0.018s	sys 0.002s
W20260709 12:10:56.520211 12460 leader_election.cc:343] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:56.523222 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:56.519717 12694 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:56.525816 12556 tablet_service.cc:1511] Processing CreateTablet for tablet 8bd1dcd3e8a44986816c86ea6e580df5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:10:56.515373 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a8c7204d6b44c9b3d6ea6332e9f824. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.528111 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e029db6a8e4a7db72ad88760a6b7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.531327 12556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd1dcd3e8a44986816c86ea6e580df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.532703 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 93e029db6a8e4a7db72ad88760a6b7d3 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:10:56.534262 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e029db6a8e4a7db72ad88760a6b7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.538868 12482 tablet_service.cc:1511] Processing CreateTablet for tablet a30ed0986fa641bd8f88912b4b9ee320 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:10:56.544473 12559 tablet_service.cc:1511] Processing CreateTablet for tablet a30ed0986fa641bd8f88912b4b9ee320 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:10:56.553463 12687 tablet_bootstrap.cc:654] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.539880 12686 raft_consensus.cc:359] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:10:56.566998 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48649.7us
I20260709 12:10:56.567399 12706 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:56.539973 12484 tablet_service.cc:1511] Processing CreateTablet for tablet 8bd1dcd3e8a44986816c86ea6e580df5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:10:56.537508 12694 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.567874 12633 tablet_service.cc:1511] Processing CreateTablet for tablet a30ed0986fa641bd8f88912b4b9ee320 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:10:56.569554 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30ed0986fa641bd8f88912b4b9ee320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.570279 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30ed0986fa641bd8f88912b4b9ee320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.567463 12686 raft_consensus.cc:385] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.565800 12559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30ed0986fa641bd8f88912b4b9ee320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.570827 12686 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.571887 12686 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.574286 12686 ts_tablet_manager.cc:1434] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.005s
I20260709 12:10:56.575373 12686 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:56.547166 12630 tablet_service.cc:1511] Processing CreateTablet for tablet 8bd1dcd3e8a44986816c86ea6e580df5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:10:56.579128 12694 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:56.579878 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd1dcd3e8a44986816c86ea6e580df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.588959 12632 tablet_service.cc:1511] Processing CreateTablet for tablet 0b5f1e67a9754d3090a2610efe47c70f (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:10:56.592100 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:56.592777 12645 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:56.601758 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:56.602499 12571 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:56.602315 12561 tablet_service.cc:1511] Processing CreateTablet for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:10:56.603765 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.598990 12480 tablet_service.cc:1511] Processing CreateTablet for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:10:56.610823 12480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.611236 12688 tablet_bootstrap.cc:492] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:56.611853 12688 ts_tablet_manager.cc:1403] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.163s	user 0.016s	sys 0.002s
I20260709 12:10:56.615272 12688 raft_consensus.cc:359] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.616127 12688 raft_consensus.cc:385] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.616515 12688 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.616153 12571 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:56.617419 12688 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.624282 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:56.618722 12687 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:56.625569 12688 ts_tablet_manager.cc:1434] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.007s
I20260709 12:10:56.624996 12485 tablet_service.cc:1511] Processing CreateTablet for tablet 0b5f1e67a9754d3090a2610efe47c70f (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:10:56.625815 12687 ts_tablet_manager.cc:1403] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.143s	user 0.019s	sys 0.001s
I20260709 12:10:56.626988 12694 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:56.627790 12694 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:56.628806 12694 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.629360 12687 raft_consensus.cc:359] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.630292 12687 raft_consensus.cc:385] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.630635 12687 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.631974 12686 tablet_bootstrap.cc:654] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.631700 12687 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.636704 12688 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:56.623799 12627 tablet_service.cc:1511] Processing CreateTablet for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:10:56.599880 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bd1dcd3e8a44986816c86ea6e580df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.637490 12645 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:56.648350 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b5f1e67a9754d3090a2610efe47c70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.651999 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b5f1e67a9754d3090a2610efe47c70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.654973 12562 tablet_service.cc:1511] Processing CreateTablet for tablet 43fecbd2edc14ba2ab065395e5d48d1d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:10:56.656248 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 43fecbd2edc14ba2ab065395e5d48d1d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:10:56.657701 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fecbd2edc14ba2ab065395e5d48d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.659176 12687 ts_tablet_manager.cc:1434] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.002s
I20260709 12:10:56.660455 12687 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:56.663525 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.662685 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fecbd2edc14ba2ab065395e5d48d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.618170 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 0b5f1e67a9754d3090a2610efe47c70f (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:10:56.672626 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b5f1e67a9754d3090a2610efe47c70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.686682 12687 tablet_bootstrap.cc:654] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.706547 12688 tablet_bootstrap.cc:654] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.707178 12628 tablet_service.cc:1511] Processing CreateTablet for tablet 43fecbd2edc14ba2ab065395e5d48d1d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:10:56.708664 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fecbd2edc14ba2ab065395e5d48d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.709092 12482 tablet_service.cc:1511] Processing CreateTablet for tablet f02aa0ab5af34d0e9ef3a4956449c624 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:10:56.711329 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02aa0ab5af34d0e9ef3a4956449c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.713944 12692 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:56.714533 12692 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.717893 12692 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:56.720973 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:56.727737 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:10:56.728168 12610 leader_election.cc:343] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:56.729890 12609 leader_election.cc:343] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:56.730582 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:56.731633 12692 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:56.736140 12553 tablet_service.cc:1511] Processing CreateTablet for tablet f02aa0ab5af34d0e9ef3a4956449c624 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:10:56.736680 12394 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:56.746812 12627 tablet_service.cc:1511] Processing CreateTablet for tablet f02aa0ab5af34d0e9ef3a4956449c624 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:10:56.747102 12486 tablet_service.cc:1511] Processing CreateTablet for tablet a16c8cd5242e4ab6840a3a328938b5df (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:10:56.748371 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02aa0ab5af34d0e9ef3a4956449c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.749014 12486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a16c8cd5242e4ab6840a3a328938b5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.749749 12686 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:56.750324 12686 ts_tablet_manager.cc:1403] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.175s	user 0.010s	sys 0.011s
I20260709 12:10:56.750720 12698 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:56.751251 12698 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.751972 12558 tablet_service.cc:1511] Processing CreateTablet for tablet a16c8cd5242e4ab6840a3a328938b5df (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:10:56.753419 12558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a16c8cd5242e4ab6840a3a328938b5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.753747 12686 raft_consensus.cc:359] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.754168 12698 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:56.756476 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02aa0ab5af34d0e9ef3a4956449c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.765941 12686 raft_consensus.cc:385] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.766363 12686 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.768500 12686 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.781504 12686 ts_tablet_manager.cc:1434] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.003s
I20260709 12:10:56.807194 12480 tablet_service.cc:1511] Processing CreateTablet for tablet 979333b179ba4f49a65315a06d54e27e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:10:56.808665 12480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979333b179ba4f49a65315a06d54e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.808545 12483 tablet_service.cc:1511] Processing CreateTablet for tablet 81a9a92616494f46bc97254b53185664 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:10:56.810261 12483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a9a92616494f46bc97254b53185664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.810861 12633 tablet_service.cc:1511] Processing CreateTablet for tablet 979333b179ba4f49a65315a06d54e27e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:10:56.812247 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979333b179ba4f49a65315a06d54e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.814637 12686 tablet_bootstrap.cc:492] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:56.821905 12688 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:56.822180 12686 tablet_bootstrap.cc:654] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.822552 12630 tablet_service.cc:1511] Processing CreateTablet for tablet 81a9a92616494f46bc97254b53185664 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:10:56.823865 12688 ts_tablet_manager.cc:1403] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.197s	user 0.010s	sys 0.037s
I20260709 12:10:56.824309 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a9a92616494f46bc97254b53185664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.841298 12485 tablet_service.cc:1511] Processing CreateTablet for tablet 92ef671b99df423ea5602f8b7ccff307 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:10:56.843566 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ef671b99df423ea5602f8b7ccff307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.834043 12688 raft_consensus.cc:359] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.867372 12688 raft_consensus.cc:385] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.867690 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 92ef671b99df423ea5602f8b7ccff307 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:10:56.869401 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ef671b99df423ea5602f8b7ccff307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.870795 12478 tablet_service.cc:1511] Processing CreateTablet for tablet ddf2b3bf3e6344499adb020ce08a3abf (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:10:56.877450 12632 tablet_service.cc:1511] Processing CreateTablet for tablet a16c8cd5242e4ab6840a3a328938b5df (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:10:56.881148 12635 tablet_service.cc:1511] Processing CreateTablet for tablet ddf2b3bf3e6344499adb020ce08a3abf (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:10:56.883075 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf2b3bf3e6344499adb020ce08a3abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.867743 12688 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.885991 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf2b3bf3e6344499adb020ce08a3abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.887645 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a16c8cd5242e4ab6840a3a328938b5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.889602 12483 tablet_service.cc:1511] Processing CreateTablet for tablet 7816534d4a0942f0af150ea0a94392b9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:10:56.891076 12483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7816534d4a0942f0af150ea0a94392b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.891628 12686 tablet_bootstrap.cc:492] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:56.892244 12686 ts_tablet_manager.cc:1403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.092s	user 0.020s	sys 0.022s
I20260709 12:10:56.896727 12688 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.901706 12692 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:56.902369 12692 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.932225 12687 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:56.932885 12687 ts_tablet_manager.cc:1403] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.273s	user 0.008s	sys 0.009s
I20260709 12:10:56.933851 12688 ts_tablet_manager.cc:1434] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.110s	user 0.007s	sys 0.004s
I20260709 12:10:56.935411 12688 tablet_bootstrap.cc:492] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:56.938453 12692 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:56.939355 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:56.940091 12497 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:56.942266 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:56.945493 12692 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:56.948349 12686 raft_consensus.cc:359] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.949198 12686 raft_consensus.cc:385] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:56.949512 12686 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:56.949745 12559 tablet_service.cc:1511] Processing CreateTablet for tablet 979333b179ba4f49a65315a06d54e27e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:10:56.951341 12559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979333b179ba4f49a65315a06d54e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.952415 12628 tablet_service.cc:1511] Processing CreateTablet for tablet 7816534d4a0942f0af150ea0a94392b9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:10:56.953940 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7816534d4a0942f0af150ea0a94392b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.956367 12688 tablet_bootstrap.cc:654] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:56.945968 12692 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:56.969349 12482 tablet_service.cc:1511] Processing CreateTablet for tablet d03683a6ca224dc89740dc91f644b357 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:10:56.970798 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03683a6ca224dc89740dc91f644b357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:56.964277 12692 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:56.992092 12484 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbf126b18234b78b4f36bf9708ad96b (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:10:57.013325 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf126b18234b78b4f36bf9708ad96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.013235 12630 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbf126b18234b78b4f36bf9708ad96b (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20260709 12:10:57.014353 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:10:57.015292 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf126b18234b78b4f36bf9708ad96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.018774 12553 tablet_service.cc:1511] Processing CreateTablet for tablet 81a9a92616494f46bc97254b53185664 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:10:57.005757 12687 raft_consensus.cc:359] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.019207 12561 tablet_service.cc:1511] Processing CreateTablet for tablet 92ef671b99df423ea5602f8b7ccff307 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:10:57.020813 12687 raft_consensus.cc:385] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.022775 12687 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.022439 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a9a92616494f46bc97254b53185664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.024109 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ef671b99df423ea5602f8b7ccff307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.029045 12687 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.033964 12706 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.034545 12706 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:56.988008 12633 tablet_service.cc:1511] Processing CreateTablet for tablet d03683a6ca224dc89740dc91f644b357 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:10:57.036504 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03683a6ca224dc89740dc91f644b357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.036970 12706 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.007401 12686 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.038694 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.039594 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.040627 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.040922 12694 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.043637 12554 tablet_service.cc:1511] Processing CreateTablet for tablet d03683a6ca224dc89740dc91f644b357 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:10:57.043519 12556 tablet_service.cc:1511] Processing CreateTablet for tablet 7816534d4a0942f0af150ea0a94392b9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
W20260709 12:10:57.044160 12460 leader_election.cc:343] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.044462 12694 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.045327 12554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03683a6ca224dc89740dc91f644b357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.046319 12562 tablet_service.cc:1511] Processing CreateTablet for tablet ddf2b3bf3e6344499adb020ce08a3abf (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:10:57.047067 12557 tablet_service.cc:1511] Processing CreateTablet for tablet 9bbf126b18234b78b4f36bf9708ad96b (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:10:57.047941 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddf2b3bf3e6344499adb020ce08a3abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.048620 12557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf126b18234b78b4f36bf9708ad96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.049328 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:10:57.050092 12609 leader_election.cc:343] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.050370 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.052042 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:10:57.053872 12460 leader_election.cc:343] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:57.052147 12536 leader_election.cc:343] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.047067 12555 tablet_service.cc:1511] Processing CreateTablet for tablet 27144fedf0014de09f007ac6cc3f7275 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20260709 12:10:57.054754 12461 leader_election.cc:343] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.057796 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:57.059831 12706 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.054623 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 27144fedf0014de09f007ac6cc3f7275 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:10:57.055202 12694 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.038306 12687 ts_tablet_manager.cc:1434] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.105s	user 0.006s	sys 0.005s
I20260709 12:10:57.055044 12556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7816534d4a0942f0af150ea0a94392b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.060583 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.060881 12636 tablet_service.cc:1511] Processing CreateTablet for tablet 27144fedf0014de09f007ac6cc3f7275 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:10:57.074141 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:57.075107 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27144fedf0014de09f007ac6cc3f7275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.075870 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:57.077406 12555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27144fedf0014de09f007ac6cc3f7275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.080641 12694 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:57.081158 12694 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:57.065527 12706 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:57.081657 12694 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.095628 12687 tablet_bootstrap.cc:492] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:57.096043 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27144fedf0014de09f007ac6cc3f7275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.099030 12720 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.101189 12698 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.113592 12698 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.115897 12698 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.113461 12720 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.122431 12720 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:57.123948 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.124754 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:57.126078 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:10:57.135138 12686 ts_tablet_manager.cc:1434] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.243s	user 0.014s	sys 0.005s
I20260709 12:10:57.136502 12686 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:57.138064 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:57.140558 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 0c56dde469374f4da7440f134140670c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:10:57.142200 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c56dde469374f4da7440f134140670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.143234 12720 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:57.153903 12720 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:57.154376 12720 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:57.144433 12610 leader_election.cc:343] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.144676 12627 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0dce81bc984a0690d08335a4937b2c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:10:57.158221 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0dce81bc984a0690d08335a4937b2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:10:57.118695 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66759.7us
I20260709 12:10:57.148267 12706 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:57.148414 12692 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.163995 12692 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:57.165023 12688 tablet_bootstrap.cc:492] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:57.166038 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:57.167155 12688 ts_tablet_manager.cc:1403] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.230s	user 0.029s	sys 0.026s
W20260709 12:10:57.167613 12609 leader_election.cc:343] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.168398 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:57.169072 12497 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.172333 12483 tablet_service.cc:1511] Processing CreateTablet for tablet 4d3b77689c31445cbba405c2b232495e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:10:57.169142 12485 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0dce81bc984a0690d08335a4937b2c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:10:57.173795 12483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d3b77689c31445cbba405c2b232495e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.174657 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0dce81bc984a0690d08335a4937b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.174736 12688 raft_consensus.cc:359] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.175632 12688 raft_consensus.cc:385] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.175941 12688 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.189218 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:57.188877 12688 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.190811 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.191512 12646 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:57.189467 12635 tablet_service.cc:1511] Processing CreateTablet for tablet 0c56dde469374f4da7440f134140670c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:10:57.194276 12635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c56dde469374f4da7440f134140670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.195375 12555 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0dce81bc984a0690d08335a4937b2c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:10:57.196837 12555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0dce81bc984a0690d08335a4937b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.202064 12554 tablet_service.cc:1511] Processing CreateTablet for tablet 0c56dde469374f4da7440f134140670c (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:10:57.202414 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 4d3b77689c31445cbba405c2b232495e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:10:57.203627 12554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c56dde469374f4da7440f134140670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.204437 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d3b77689c31445cbba405c2b232495e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.207922 12497 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
W20260709 12:10:57.209755 12536 leader_election.cc:343] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.209568 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:57.210685 12692 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:57.215830 12687 tablet_bootstrap.cc:654] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.219259 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:57.221769 12694 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.226539 12720 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.229056 12631 tablet_service.cc:1511] Processing CreateTablet for tablet 4d3b77689c31445cbba405c2b232495e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:10:57.243006 12688 ts_tablet_manager.cc:1434] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.075s	user 0.009s	sys 0.003s
I20260709 12:10:57.253574 12698 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.265620 12631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d3b77689c31445cbba405c2b232495e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.281755 12688 tablet_bootstrap.cc:492] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:57.255534 12686 tablet_bootstrap.cc:654] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.293562 12692 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:57.313572 12633 tablet_service.cc:1511] Processing CreateTablet for tablet 07fe6c41f52649a88327e52e0352b171 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:10:57.315505 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fe6c41f52649a88327e52e0352b171. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:10:57.295009 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83568.2us
I20260709 12:10:57.325978 12485 tablet_service.cc:1511] Processing CreateTablet for tablet 07fe6c41f52649a88327e52e0352b171 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:10:57.327751 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fe6c41f52649a88327e52e0352b171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.332458 12636 tablet_service.cc:1511] Processing CreateTablet for tablet 1f92af3771e3446f8c3549eb19f5b2c9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:10:57.343844 12692 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.345468 12726 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.346101 12726 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.347759 12636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f92af3771e3446f8c3549eb19f5b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.348327 12726 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.351971 12628 tablet_service.cc:1511] Processing CreateTablet for tablet 7069a8d3584b466aab95934840523b8d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:10:57.356951 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 1f92af3771e3446f8c3549eb19f5b2c9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:10:57.359004 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f92af3771e3446f8c3549eb19f5b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.363401 12484 tablet_service.cc:1511] Processing CreateTablet for tablet 7069a8d3584b466aab95934840523b8d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:10:57.379119 12484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7069a8d3584b466aab95934840523b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.364812 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7069a8d3584b466aab95934840523b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.386358 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:10:57.360551 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48123.9us
I20260709 12:10:57.387650 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:10:57.395287 12610 leader_election.cc:343] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.397328 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.356190 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:10:57.398499 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57146.9us
I20260709 12:10:57.398749 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:57.400425 12698 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:57.402120 12698 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:57.402457 12698 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:57.408183 12461 leader_election.cc:343] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:57.409345 12460 leader_election.cc:343] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.409828 12460 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:57.410794 12729 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:10:57.411938 12460 leader_election.cc:343] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.415598 12485 tablet_service.cc:1511] Processing CreateTablet for tablet dd6a5b4068ea44d2ac6f61924dd38c0d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:10:57.417985 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 7b3c8a7a06a04c369587d6ecb196bb2e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:10:57.419633 12485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a5b4068ea44d2ac6f61924dd38c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.422272 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3c8a7a06a04c369587d6ecb196bb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.424921 12478 tablet_service.cc:1511] Processing CreateTablet for tablet e1f18763e7844e0994e2f11a6dd89290 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:10:57.423115 12633 tablet_service.cc:1511] Processing CreateTablet for tablet e1f18763e7844e0994e2f11a6dd89290 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:10:57.426496 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1f18763e7844e0994e2f11a6dd89290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.427572 12486 tablet_service.cc:1511] Processing CreateTablet for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:10:57.428031 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:57.428685 12646 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.428953 12486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f402dfd68ae4c8697765a76d9f1c4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.431558 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1f18763e7844e0994e2f11a6dd89290. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:10:57.400429 12536 leader_election.cc:343] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.431540 12694 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.472383 12630 tablet_service.cc:1511] Processing CreateTablet for tablet dd6a5b4068ea44d2ac6f61924dd38c0d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:10:57.474727 12630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a5b4068ea44d2ac6f61924dd38c0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:10:57.480432 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80026.3us
I20260709 12:10:57.483393 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 07fe6c41f52649a88327e52e0352b171 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:10:57.484910 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fe6c41f52649a88327e52e0352b171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.486018 12557 tablet_service.cc:1511] Processing CreateTablet for tablet 7069a8d3584b466aab95934840523b8d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:10:57.486958 12554 tablet_service.cc:1511] Processing CreateTablet for tablet 1f92af3771e3446f8c3549eb19f5b2c9 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:10:57.488122 12557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7069a8d3584b466aab95934840523b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.490329 12559 tablet_service.cc:1511] Processing CreateTablet for tablet 7b3c8a7a06a04c369587d6ecb196bb2e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:10:57.493288 12553 tablet_service.cc:1511] Processing CreateTablet for tablet e1f18763e7844e0994e2f11a6dd89290 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:10:57.495434 12725 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.496145 12725 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.488890 12554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f92af3771e3446f8c3549eb19f5b2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.484566 12561 tablet_service.cc:1511] Processing CreateTablet for tablet dd6a5b4068ea44d2ac6f61924dd38c0d (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:10:57.501593 12482 tablet_service.cc:1511] Processing CreateTablet for tablet 5d60fb5ad76741448a5376ca4d3f05df (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:10:57.503010 12482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d60fb5ad76741448a5376ca4d3f05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.506498 12627 tablet_service.cc:1511] Processing CreateTablet for tablet 7b3c8a7a06a04c369587d6ecb196bb2e (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:10:57.508109 12627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3c8a7a06a04c369587d6ecb196bb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.508411 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:57.509114 12571 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.450491 12720 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:57.511500 12687 tablet_bootstrap.cc:492] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:57.512208 12687 ts_tablet_manager.cc:1403] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.417s	user 0.015s	sys 0.008s
I20260709 12:10:57.514084 12632 tablet_service.cc:1511] Processing CreateTablet for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:10:57.515563 12632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f402dfd68ae4c8697765a76d9f1c4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.534929 12646 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:57.531701 12559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3c8a7a06a04c369587d6ecb196bb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.502803 12698 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.547148 12698 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.550132 12633 tablet_service.cc:1511] Processing CreateTablet for tablet 5d60fb5ad76741448a5376ca4d3f05df (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:10:57.552613 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d60fb5ad76741448a5376ca4d3f05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.536492 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:57.560567 12694 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:57.561292 12694 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:57.562151 12562 tablet_service.cc:1511] Processing CreateTablet for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:10:57.562522 12694 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.563647 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:10:57.565068 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.574350 12629 tablet_service.cc:1511] Processing CreateTablet for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:10:57.575965 12629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.581214 12561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a5b4068ea44d2ac6f61924dd38c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.587530 12394 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:57.589304 12725 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:57.591813 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.593165 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:10:57.595237 12610 leader_election.cc:343] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.591830 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:57.604074 12556 tablet_service.cc:1511] Processing CreateTablet for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
W20260709 12:10:57.605787 12609 leader_election.cc:343] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.606264 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:57.606798 12555 tablet_service.cc:1511] Processing CreateTablet for tablet 5d60fb5ad76741448a5376ca4d3f05df (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:10:57.607640 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:57.608268 12646 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.550135 12687 raft_consensus.cc:359] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.610093 12687 raft_consensus.cc:385] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.610419 12687 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.611243 12687 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.615022 12686 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:57.615541 12686 ts_tablet_manager.cc:1403] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.479s	user 0.033s	sys 0.031s
I20260709 12:10:57.613545 12687 ts_tablet_manager.cc:1434] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.101s	user 0.006s	sys 0.000s
I20260709 12:10:57.616139 12553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1f18763e7844e0994e2f11a6dd89290. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:10:57.640262 12536 leader_election.cc:343] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.643031 12687 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:57.643337 12571 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:57.644958 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:10:57.645972 12692 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:57.646476 12692 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:57.647461 12646 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:10:57.647289 12692 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.649786 12478 tablet_service.cc:1511] Processing CreateTablet for tablet 1033139a62f44d368f8443d665a2cf22 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:10:57.650243 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:10:57.651193 12478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1033139a62f44d368f8443d665a2cf22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.622473 12725 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.651722 12698 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:57.652244 12698 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:10:57.652838 12562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f402dfd68ae4c8697765a76d9f1c4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.653090 12698 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.665275 12556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.666949 12688 tablet_bootstrap.cc:654] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.677731 12560 tablet_service.cc:1511] Processing CreateTablet for tablet 1033139a62f44d368f8443d665a2cf22 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:10:57.667347 12686 raft_consensus.cc:359] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.688997 12686 raft_consensus.cc:385] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.689419 12686 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.689224 12633 tablet_service.cc:1511] Processing CreateTablet for tablet 1033139a62f44d368f8443d665a2cf22 (DEFAULT_TABLE table=test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:10:57.690642 12633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1033139a62f44d368f8443d665a2cf22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.682106 12555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d60fb5ad76741448a5376ca4d3f05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.705847 12686 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.709467 12686 ts_tablet_manager.cc:1434] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.093s	user 0.009s	sys 0.001s
I20260709 12:10:57.711290 12686 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:57.719384 12694 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.720026 12694 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.723817 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.724094 12686 tablet_bootstrap.cc:654] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
W20260709 12:10:57.725531 12460 leader_election.cc:343] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.729148 12560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1033139a62f44d368f8443d665a2cf22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:10:57.735216 12686 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:57.727905 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.730337 12694 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:10:57.742890 12461 leader_election.cc:343] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.743705 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:57.746133 12694 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.759469 12687 tablet_bootstrap.cc:654] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.760325 12688 tablet_bootstrap.cc:492] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:57.760987 12688 ts_tablet_manager.cc:1403] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.480s	user 0.010s	sys 0.012s
I20260709 12:10:57.755806 12686 ts_tablet_manager.cc:1403] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.045s	user 0.018s	sys 0.001s
I20260709 12:10:57.766743 12688 raft_consensus.cc:359] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.767635 12688 raft_consensus.cc:385] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.768075 12688 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.769971 12688 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.774628 12688 ts_tablet_manager.cc:1434] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.005s
I20260709 12:10:57.773881 12686 raft_consensus.cc:359] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.775995 12686 raft_consensus.cc:385] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.776378 12686 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.780686 12686 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.782467 12688 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:57.786316 12686 ts_tablet_manager.cc:1434] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 12:10:57.790027 12686 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:57.791186 12692 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:57.793488 12698 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.794067 12698 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.796381 12698 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.798542 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:57.799502 12687 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:57.800189 12687 ts_tablet_manager.cc:1403] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.157s	user 0.020s	sys 0.012s
I20260709 12:10:57.801136 12395 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:57.803309 12687 raft_consensus.cc:359] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.802923 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.804081 12687 raft_consensus.cc:385] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.804436 12687 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.809316 12686 tablet_bootstrap.cc:654] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.813094 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:57.819684 12698 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.831334 12688 tablet_bootstrap.cc:654] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.834961 12570 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.840728 12393 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:57.817595 12687 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.844506 12688 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:57.844951 12687 ts_tablet_manager.cc:1434] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.000s
I20260709 12:10:57.845083 12688 ts_tablet_manager.cc:1403] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.069s	user 0.010s	sys 0.009s
I20260709 12:10:57.846197 12687 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:57.849110 12688 raft_consensus.cc:359] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.849730 12692 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:57.850301 12688 raft_consensus.cc:385] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.814694 12394 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:57.854089 12688 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.857795 12687 tablet_bootstrap.cc:654] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.867274 12688 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.870232 12688 ts_tablet_manager.cc:1434] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.004s
I20260709 12:10:57.871273 12688 tablet_bootstrap.cc:492] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:57.878441 12688 tablet_bootstrap.cc:654] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.898452 12686 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:57.898981 12686 ts_tablet_manager.cc:1403] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.109s	user 0.046s	sys 0.033s
I20260709 12:10:57.902325 12686 raft_consensus.cc:359] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.903327 12686 raft_consensus.cc:385] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.903702 12686 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.904794 12686 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.910295 12694 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.910892 12694 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.911522 12686 ts_tablet_manager.cc:1434] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:10:57.912361 12688 tablet_bootstrap.cc:492] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:57.912899 12688 ts_tablet_manager.cc:1403] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.042s	user 0.021s	sys 0.005s
I20260709 12:10:57.913101 12686 tablet_bootstrap.cc:492] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:57.914795 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.915726 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.915751 12571 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:10:57.918224 12460 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:57.919044 12727 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.919605 12694 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.920253 12686 tablet_bootstrap.cc:654] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.928642 12688 raft_consensus.cc:359] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.930344 12688 raft_consensus.cc:385] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.932020 12686 tablet_bootstrap.cc:492] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:57.932399 12687 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:57.932696 12686 ts_tablet_manager.cc:1403] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.008s
I20260709 12:10:57.932478 12688 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.933014 12687 ts_tablet_manager.cc:1403] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.087s	user 0.028s	sys 0.003s
I20260709 12:10:57.933821 12688 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.934532 12727 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.935747 12686 raft_consensus.cc:359] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.936300 12687 raft_consensus.cc:359] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.937062 12687 raft_consensus.cc:385] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.938252 12688 ts_tablet_manager.cc:1434] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:10:57.937393 12687 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.939443 12688 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:57.939257 12687 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.936676 12686 raft_consensus.cc:385] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.940503 12686 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.941303 12686 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.944348 12686 ts_tablet_manager.cc:1434] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:10:57.946247 12686 tablet_bootstrap.cc:492] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:57.946638 12688 tablet_bootstrap.cc:654] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.950207 12687 ts_tablet_manager.cc:1434] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260709 12:10:57.951566 12687 tablet_bootstrap.cc:492] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:57.952692 12686 tablet_bootstrap.cc:654] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.959903 12687 tablet_bootstrap.cc:654] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.962368 12692 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.963215 12692 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.967968 12692 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:57.968573 12686 tablet_bootstrap.cc:492] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:57.969396 12686 ts_tablet_manager.cc:1403] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.013s
I20260709 12:10:57.971575 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:57.972662 12497 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:57.974190 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:57.975375 12571 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:57.975519 12686 raft_consensus.cc:359] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.977115 12686 raft_consensus.cc:385] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.977571 12686 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:57.978205 12698 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:57.978400 12686 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.978754 12698 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.981536 12686 ts_tablet_manager.cc:1434] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:10:57.982133 12687 tablet_bootstrap.cc:492] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:57.982920 12687 ts_tablet_manager.cc:1403] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.015s
I20260709 12:10:57.982967 12686 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:57.984539 12609 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:10:57.985786 12692 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:10:57.986181 12460 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:57.986253 12692 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:57.986783 12692 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:57.987514 12698 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:57.990958 12687 raft_consensus.cc:359] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.991359 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:57.991735 12687 raft_consensus.cc:385] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:57.992019 12687 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
W20260709 12:10:57.992904 12536 leader_election.cc:343] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.992707 12687 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:57.993757 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:57.994830 12687 ts_tablet_manager.cc:1434] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
W20260709 12:10:57.995625 12536 leader_election.cc:343] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:57.996053 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:57.997429 12698 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:57.996307 12687 tablet_bootstrap.cc:492] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.002143 12686 tablet_bootstrap.cc:654] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:57.997143 12692 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.005420 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.006359 12497 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.006841 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.007604 12570 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.011566 12727 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.012122 12727 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:10:58.017117 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:10:58.017375 12497 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:58.017757 12570 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:58.017974 12692 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.021274 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.019524 12687 tablet_bootstrap.cc:654] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.024364 12692 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.025040 12692 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:58.025440 12720 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.025931 12720 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.026294 12692 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.026752 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.027738 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:58.028684 12720 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.029201 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:58.029928 12727 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.033437 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.034363 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:58.039980 12686 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.040520 12686 ts_tablet_manager.cc:1403] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.058s	user 0.009s	sys 0.005s
I20260709 12:10:58.041666 12687 tablet_bootstrap.cc:492] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:58.042321 12687 ts_tablet_manager.cc:1403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.046s	user 0.014s	sys 0.006s
I20260709 12:10:58.044148 12686 raft_consensus.cc:359] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.045130 12686 raft_consensus.cc:385] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.045399 12686 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.045420 12687 raft_consensus.cc:359] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.046607 12687 raft_consensus.cc:385] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.046967 12687 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.046739 12686 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.047926 12687 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.050716 12687 ts_tablet_manager.cc:1434] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:10:58.030005 12497 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:58.029476 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.051838 12687 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.030730 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.052873 12570 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:58.054993 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:10:58.056277 12692 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.056727 12692 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.057072 12692 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.057945 12727 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:58.060535 12687 tablet_bootstrap.cc:654] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.063635 12686 ts_tablet_manager.cc:1434] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.000s
I20260709 12:10:58.064723 12729 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.067188 12692 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.068071 12729 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.070953 12729 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.073951 12686 tablet_bootstrap.cc:492] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.074110 12692 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.074432 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.075203 12570 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.078817 12687 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:58.079821 12687 ts_tablet_manager.cc:1403] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.028s	user 0.018s	sys 0.003s
I20260709 12:10:58.082022 12686 tablet_bootstrap.cc:654] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.082541 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.082755 12729 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.084264 12687 raft_consensus.cc:359] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.085093 12687 raft_consensus.cc:385] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.085508 12687 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.086356 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.086416 12687 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.089064 12687 ts_tablet_manager.cc:1434] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:10:58.089308 12688 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:58.089757 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:58.090690 12646 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:10:58.090755 12687 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.095053 12497 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:10:58.095379 12392 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:58.090040 12688 ts_tablet_manager.cc:1403] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.151s	user 0.033s	sys 0.054s
I20260709 12:10:58.099978 12687 tablet_bootstrap.cc:654] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.100730 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.105717 12688 raft_consensus.cc:359] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.107025 12688 raft_consensus.cc:385] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.107355 12688 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.107210 12698 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.108258 12698 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.109319 12729 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.110306 12729 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.113483 12698 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.114580 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.108558 12688 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.116503 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:10:58.117453 12536 leader_election.cc:343] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.117576 12497 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:58.118701 12688 ts_tablet_manager.cc:1434] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.002s
I20260709 12:10:58.119161 12570 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:58.119488 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.122160 12688 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:58.122464 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:10:58.124660 12692 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.125460 12692 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:58.128454 12698 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.129093 12698 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.129556 12698 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.126780 12692 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.148530 12686 tablet_bootstrap.cc:492] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.149094 12686 ts_tablet_manager.cc:1403] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.075s	user 0.028s	sys 0.017s
I20260709 12:10:58.151837 12686 raft_consensus.cc:359] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.152539 12686 raft_consensus.cc:385] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.152822 12686 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.153477 12686 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.154757 12687 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:58.155213 12687 ts_tablet_manager.cc:1403] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.065s	user 0.040s	sys 0.016s
I20260709 12:10:58.155871 12686 ts_tablet_manager.cc:1434] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20260709 12:10:58.156783 12686 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.158344 12687 raft_consensus.cc:359] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.159032 12687 raft_consensus.cc:385] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.159402 12687 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.160516 12687 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.160952 12688 tablet_bootstrap.cc:654] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.162134 12694 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.164669 12694 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.168572 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.169341 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:58.170593 12686 tablet_bootstrap.cc:654] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.172152 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.173298 12727 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.173743 12727 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.174106 12727 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.174175 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.175451 12694 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:10:58.176541 12460 leader_election.cc:343] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.163591 12698 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.185078 12686 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.185956 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.186311 12686 ts_tablet_manager.cc:1403] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.030s	user 0.007s	sys 0.016s
I20260709 12:10:58.186826 12497 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.191363 12686 raft_consensus.cc:359] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.192418 12686 raft_consensus.cc:385] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.192921 12686 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.194343 12686 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.198365 12686 ts_tablet_manager.cc:1434] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20260709 12:10:58.199815 12686 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.200276 12727 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.207969 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.208802 12686 tablet_bootstrap.cc:654] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
W20260709 12:10:58.209590 12460 leader_election.cc:343] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.212706 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:58.213374 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.214493 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:10:58.216094 12536 leader_election.cc:343] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.216708 12698 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.212837 12727 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.229683 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:58.233517 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:58.235479 12727 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.236861 12727 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:58.237811 12727 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.237847 12394 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:58.248047 12698 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.251413 12688 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:58.250396 12698 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.249774 12497 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:10:58.252144 12688 ts_tablet_manager.cc:1403] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.130s	user 0.015s	sys 0.003s
I20260709 12:10:58.248296 12686 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.254464 12686 ts_tablet_manager.cc:1403] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.055s	user 0.018s	sys 0.029s
I20260709 12:10:58.256006 12687 ts_tablet_manager.cc:1434] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.101s	user 0.033s	sys 0.067s
I20260709 12:10:58.256372 12729 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.256873 12729 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.257880 12698 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.259580 12729 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.260399 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.262240 12725 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.262419 12687 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.262950 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.262835 12725 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.263840 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:58.263652 12686 raft_consensus.cc:359] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.267068 12686 raft_consensus.cc:385] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.267177 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:58.267500 12686 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.267977 12496 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:58.268285 12725 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.268628 12686 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.266307 12688 raft_consensus.cc:359] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.269780 12688 raft_consensus.cc:385] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.271780 12688 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.268819 12725 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:58.273669 12686 ts_tablet_manager.cc:1434] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.001s
I20260709 12:10:58.274258 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.274837 12686 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.271122 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.270820 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.275134 12646 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:10:58.276930 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:10:58.270227 12687 tablet_bootstrap.cc:654] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.270818 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.275449 12571 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:58.277483 12692 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.272382 12698 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.283305 12461 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.283214 12688 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.291072 12729 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.287093 12692 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.297688 12763 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.317823 12763 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.318214 12763 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.320012 12686 tablet_bootstrap.cc:654] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.316738 12692 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.284978 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:58.323652 12497 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:58.326916 12393 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:58.332754 12763 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.338421 12727 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.339134 12727 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.341493 12688 ts_tablet_manager.cc:1434] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.079s	user 0.006s	sys 0.002s
I20260709 12:10:58.342733 12688 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:58.348973 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:10:58.350921 12460 leader_election.cc:343] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.352795 12727 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.297257 12725 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.056s
I20260709 12:10:58.354959 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:10:58.356653 12461 leader_election.cc:343] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.357239 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.357467 12692 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.359165 12694 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.359571 12768 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:10:58.360944 12698 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:10:58.362991 12729 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.362541 12698 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.365505 12763 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.366359 12725 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.367117 12725 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.371009 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:58.371757 12496 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:58.373590 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.374480 12720 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.374852 12720 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.375308 12720 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.379748 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:10:58.382053 12609 leader_election.cc:343] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.384032 12725 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.386559 12686 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.387348 12686 ts_tablet_manager.cc:1403] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.113s	user 0.010s	sys 0.009s
W20260709 12:10:58.388402 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70839.7us
I20260709 12:10:58.392491 12688 tablet_bootstrap.cc:654] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.393714 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.394455 12497 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.399387 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:58.400318 12645 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.400092 12686 raft_consensus.cc:359] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.400849 12686 raft_consensus.cc:385] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.401116 12686 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.408819 12770 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:58.419940 12686 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.397451 12720 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.423759 12720 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.434201 12497 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:10:58.436849 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.437783 12770 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.438290 12770 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:10:58.439253 12770 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.440831 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:10:58.442600 12609 leader_election.cc:343] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.445909 12686 ts_tablet_manager.cc:1434] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.058s	user 0.005s	sys 0.005s
I20260709 12:10:58.452034 12686 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.451892 12645 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:10:58.458150 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.459009 12495 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.458211 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.459678 12497 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.475219 12698 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.477489 12687 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:58.477232 12698 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.478005 12687 ts_tablet_manager.cc:1403] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.216s	user 0.019s	sys 0.018s
I20260709 12:10:58.480957 12698 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.494350 12688 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:58.494822 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.495133 12688 ts_tablet_manager.cc:1403] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.153s	user 0.021s	sys 0.016s
I20260709 12:10:58.495589 12570 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.489614 12692 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.503756 12688 raft_consensus.cc:359] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.508839 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:58.508857 12688 raft_consensus.cc:385] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.509991 12688 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.510776 12688 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.511459 12687 raft_consensus.cc:359] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.512087 12687 raft_consensus.cc:385] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.512328 12687 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.512905 12688 ts_tablet_manager.cc:1434] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.000s
I20260709 12:10:58.513017 12687 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.514755 12687 ts_tablet_manager.cc:1434] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.001s
I20260709 12:10:58.515543 12687 tablet_bootstrap.cc:492] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.501186 12779 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:10:58.519168 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:58.520627 12394 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:58.525223 12688 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:58.530575 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.531750 12779 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.532413 12779 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:58.532714 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.533344 12779 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.535072 12497 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:58.535504 12570 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:58.536437 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:58.537424 12692 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.581243 12687 tablet_bootstrap.cc:654] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.572830 12692 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:58.590131 12570 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.594302 12692 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.599756 12727 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.600378 12727 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.610250 12727 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.610718 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.610970 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:10:58.613603 12460 leader_election.cc:343] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.622584 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:10:58.624441 12461 leader_election.cc:343] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.625875 12461 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.629251 12778 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:10:58.630439 12727 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.642817 12688 tablet_bootstrap.cc:654] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.668859 12686 tablet_bootstrap.cc:654] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.681427 12768 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.682956 12687 tablet_bootstrap.cc:492] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:58.683532 12687 ts_tablet_manager.cc:1403] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.168s	user 0.022s	sys 0.015s
I20260709 12:10:58.687937 12687 raft_consensus.cc:359] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.689307 12687 raft_consensus.cc:385] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.690145 12687 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.695180 12687 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.699918 12687 ts_tablet_manager.cc:1434] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.001s
I20260709 12:10:58.701354 12687 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.689122 12394 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:58.710240 12720 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:58.714880 12686 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.715493 12686 ts_tablet_manager.cc:1403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.264s	user 0.026s	sys 0.027s
I20260709 12:10:58.718691 12686 raft_consensus.cc:359] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.719426 12686 raft_consensus.cc:385] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.719717 12686 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.720645 12686 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.723273 12686 ts_tablet_manager.cc:1434] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:10:58.725054 12686 tablet_bootstrap.cc:492] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.726020 12688 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:58.726614 12688 ts_tablet_manager.cc:1403] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.202s	user 0.032s	sys 0.017s
I20260709 12:10:58.729738 12688 raft_consensus.cc:359] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.730558 12688 raft_consensus.cc:385] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.731560 12688 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.732473 12688 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.735109 12688 ts_tablet_manager.cc:1434] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:10:58.736572 12688 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:58.738227 12720 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:58.739187 12392 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:58.741889 12687 tablet_bootstrap.cc:654] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.741598 12392 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:58.743278 12688 tablet_bootstrap.cc:654] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.764773 12686 tablet_bootstrap.cc:654] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.802737 12727 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.803607 12727 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.813876 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.814759 12571 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:58.815402 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.813457 12694 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:10:58.815984 12688 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:58.816176 12646 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:58.816231 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.816636 12688 ts_tablet_manager.cc:1403] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.080s	user 0.012s	sys 0.000s
I20260709 12:10:58.818023 12694 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.818483 12694 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.818862 12694 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.819694 12688 raft_consensus.cc:359] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.820253 12688 raft_consensus.cc:385] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.820504 12688 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.821218 12688 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.813957 12727 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.823380 12688 ts_tablet_manager.cc:1434] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:10:58.824558 12688 tablet_bootstrap.cc:492] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:58.827677 12571 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.831822 12686 tablet_bootstrap.cc:492] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:58.832304 12686 ts_tablet_manager.cc:1403] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.108s	user 0.035s	sys 0.019s
I20260709 12:10:58.839453 12768 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:58.840452 12694 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.843042 12694 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.843300 12727 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:58.845803 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.846375 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:58.846496 12570 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.847105 12645 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.847891 12686 raft_consensus.cc:359] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.848589 12686 raft_consensus.cc:385] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.848845 12686 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.849603 12686 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.851998 12687 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:58.853309 12687 ts_tablet_manager.cc:1403] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.152s	user 0.025s	sys 0.030s
I20260709 12:10:58.857431 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:58.859089 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.865237 12727 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:58.865957 12686 ts_tablet_manager.cc:1434] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.033s	user 0.004s	sys 0.005s
I20260709 12:10:58.865844 12727 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:58.863930 12688 tablet_bootstrap.cc:654] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.867322 12686 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:58.857545 12687 raft_consensus.cc:359] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.872295 12768 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.875319 12768 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.882055 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:58.872479 12727 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.872740 12687 raft_consensus.cc:385] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.883631 12687 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.884296 12768 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.884517 12687 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.886420 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.870601 12729 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.887226 12729 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.888185 12687 ts_tablet_manager.cc:1434] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.005s
W20260709 12:10:58.888535 12536 leader_election.cc:343] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.889784 12687 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:58.890738 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.891391 12571 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:58.893847 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:58.896701 12645 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:58.901078 12770 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:58.888605 12495 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:58.903710 12694 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.905843 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.907060 12694 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.907579 12694 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:58.908022 12461 leader_election.cc:343] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.910393 12770 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:58.911401 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.913363 12770 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:58.913782 12770 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:58.914125 12770 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:58.915076 12729 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.918431 12694 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.921516 12779 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.922032 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:58.922449 12779 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:10:58.923961 12461 leader_election.cc:343] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.927937 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:10:58.931051 12610 leader_election.cc:343] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.931572 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:10:58.933140 12609 leader_election.cc:343] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.933663 12609 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:58.933849 12694 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.935803 12720 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.936818 12779 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:58.938067 12770 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.938995 12687 tablet_bootstrap.cc:654] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
W20260709 12:10:58.939589 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44005.4us
I20260709 12:10:58.941540 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:58.941565 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:58.942642 12572 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:10:58.942654 12496 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:10:58.941565 12802 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:58.944501 12686 tablet_bootstrap.cc:654] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:58.944135 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:58.943457 12802 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.947491 12727 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:10:58.949189 12770 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.949929 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:58.950591 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:58.949280 12394 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:10:58.954547 12536 leader_election.cc:343] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:58.955156 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:58.958662 12688 tablet_bootstrap.cc:492] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:58.959152 12688 ts_tablet_manager.cc:1403] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.135s	user 0.006s	sys 0.013s
I20260709 12:10:58.957487 12698 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:10:58.974630 12460 leader_election.cc:343] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:10:58.980850 12536 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:10:58.990880 12688 raft_consensus.cc:359] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.991890 12802 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:58.991937 12688 raft_consensus.cc:385] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:58.993867 12688 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:58.994694 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:58.995779 12688 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:58.997296 12646 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:59.000530 12688 ts_tablet_manager.cc:1434] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.041s	user 0.004s	sys 0.004s
I20260709 12:10:59.007997 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.010183 12729 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.010643 12729 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.010983 12729 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.014074 12688 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.020735 12720 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:59.021618 12770 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.022239 12770 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.024403 12770 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.026145 12686 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.026283 12688 tablet_bootstrap.cc:654] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.026644 12686 ts_tablet_manager.cc:1403] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.160s	user 0.013s	sys 0.016s
I20260709 12:10:59.031096 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.029354 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.008038 12687 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.043082 12687 ts_tablet_manager.cc:1403] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.154s	user 0.015s	sys 0.017s
I20260709 12:10:59.035274 12536 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.046581 12770 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.051180 12686 raft_consensus.cc:359] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.052211 12686 raft_consensus.cc:385] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.052573 12686 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.048732 12687 raft_consensus.cc:359] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:10:59.054301 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:10:59.055342 12686 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.054531 12687 raft_consensus.cc:385] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.056838 12687 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.057883 12687 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.048663 12729 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.061209 12686 ts_tablet_manager.cc:1434] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.001s
I20260709 12:10:59.062002 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:59.062361 12686 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
W20260709 12:10:59.071643 12460 leader_election.cc:343] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.071902 12688 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.072507 12688 ts_tablet_manager.cc:1403] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.059s	user 0.018s	sys 0.017s
I20260709 12:10:59.072391 12692 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:59.074365 12572 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.075639 12688 raft_consensus.cc:359] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.076438 12688 raft_consensus.cc:385] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.076733 12688 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.077546 12688 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.079505 12687 ts_tablet_manager.cc:1434] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.036s	user 0.009s	sys 0.000s
I20260709 12:10:59.079807 12688 ts_tablet_manager.cc:1434] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:10:59.080628 12687 tablet_bootstrap.cc:492] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.080693 12688 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.087113 12686 tablet_bootstrap.cc:654] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.094992 12688 tablet_bootstrap.cc:654] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.102232 12687 tablet_bootstrap.cc:654] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.108727 12720 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:59.118299 12729 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.119534 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:59.120834 12645 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.124701 12688 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.125290 12688 ts_tablet_manager.cc:1403] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.005s
I20260709 12:10:59.127362 12720 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:59.128115 12687 tablet_bootstrap.cc:492] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.128575 12687 ts_tablet_manager.cc:1403] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.048s	user 0.014s	sys 0.004s
I20260709 12:10:59.128803 12686 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.129405 12686 ts_tablet_manager.cc:1403] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.067s	user 0.013s	sys 0.028s
I20260709 12:10:59.131232 12688 raft_consensus.cc:359] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.132347 12686 raft_consensus.cc:359] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.132985 12688 raft_consensus.cc:385] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.133554 12688 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.133059 12686 raft_consensus.cc:385] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.134363 12688 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.134975 12686 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.134384 12687 raft_consensus.cc:359] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.137027 12687 raft_consensus.cc:385] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.137394 12687 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.139020 12687 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.141279 12645 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:59.141645 12687 ts_tablet_manager.cc:1434] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:10:59.143002 12687 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.145133 12686 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.148530 12688 ts_tablet_manager.cc:1434] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:10:59.149089 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.149840 12688 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.154147 12802 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.154682 12802 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:59.155514 12802 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.169780 12686 ts_tablet_manager.cc:1434] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.040s	user 0.008s	sys 0.000s
I20260709 12:10:59.170833 12686 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:59.178696 12687 tablet_bootstrap.cc:654] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.189831 12688 tablet_bootstrap.cc:654] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.191385 12802 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.192252 12802 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.196818 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.197773 12686 tablet_bootstrap.cc:654] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
W20260709 12:10:59.198989 12460 leader_election.cc:343] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.200717 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.199059 12802 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:10:59.202891 12461 leader_election.cc:343] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.203414 12461 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:59.208385 12720 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.208949 12720 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.209884 12687 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.210405 12687 ts_tablet_manager.cc:1403] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.068s	user 0.013s	sys 0.002s
I20260709 12:10:59.211076 12688 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.211200 12720 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.211653 12688 ts_tablet_manager.cc:1403] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.062s	user 0.019s	sys 0.004s
I20260709 12:10:59.213069 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.213910 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.213083 12687 raft_consensus.cc:359] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.214605 12687 raft_consensus.cc:385] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.214766 12495 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:59.214545 12688 raft_consensus.cc:359] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.215255 12687 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.215642 12688 raft_consensus.cc:385] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.216104 12688 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.216231 12687 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:10:59.216588 12609 leader_election.cc:343] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.218719 12687 ts_tablet_manager.cc:1434] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:10:59.218389 12610 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.220131 12687 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.221517 12720 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.222143 12720 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.222656 12720 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.242337 12802 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.228528 12687 tablet_bootstrap.cc:654] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.217043 12688 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.248381 12391 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:59.256255 12720 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.257727 12687 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.258275 12687 ts_tablet_manager.cc:1403] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.038s	user 0.021s	sys 0.000s
I20260709 12:10:59.259949 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:59.260516 12495 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.262564 12770 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.263145 12688 ts_tablet_manager.cc:1434] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.051s	user 0.006s	sys 0.001s
I20260709 12:10:59.262970 12687 raft_consensus.cc:359] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.263784 12687 raft_consensus.cc:385] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.264067 12687 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.263146 12770 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.265865 12779 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.266996 12720 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.266557 12779 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.267694 12770 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.268635 12687 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.269222 12779 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.270094 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.278350 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:59.276103 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.276970 12688 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.280524 12570 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:59.281875 12687 ts_tablet_manager.cc:1434] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.023s	user 0.009s	sys 0.001s
I20260709 12:10:59.270896 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.284884 12645 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:10:59.282820 12609 leader_election.cc:343] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.283397 12495 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:59.286482 12609 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:10:59.287010 12610 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.287369 12720 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.287838 12720 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.288365 12687 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.287961 12779 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.287708 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.290409 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.290890 12496 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:10:59.290433 12779 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:59.293051 12779 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.293922 12770 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.288247 12720 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.305073 12770 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.307886 12688 tablet_bootstrap.cc:654] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.310148 12686 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.310835 12686 ts_tablet_manager.cc:1403] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.140s	user 0.034s	sys 0.036s
I20260709 12:10:59.315375 12391 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:59.317133 12686 raft_consensus.cc:359] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.317900 12720 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.321370 12686 raft_consensus.cc:385] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.321867 12686 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.320885 12687 tablet_bootstrap.cc:654] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.322837 12686 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.325989 12720 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.326597 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:59.327260 12496 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.327754 12686 ts_tablet_manager.cc:1434] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.001s
I20260709 12:10:59.328994 12686 tablet_bootstrap.cc:492] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:59.331177 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:59.331964 12570 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.335048 12688 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.335659 12688 ts_tablet_manager.cc:1403] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.059s	user 0.002s	sys 0.021s
I20260709 12:10:59.338995 12687 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.339586 12687 ts_tablet_manager.cc:1403] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.003s
I20260709 12:10:59.341575 12720 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.342829 12688 raft_consensus.cc:359] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.342255 12720 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.343743 12688 raft_consensus.cc:385] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.344085 12688 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.348588 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.349284 12570 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:59.350433 12496 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:59.352018 12688 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.352743 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.353886 12778 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.354403 12688 ts_tablet_manager.cc:1434] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.001s
I20260709 12:10:59.354496 12720 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.355504 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.357007 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.357440 12686 tablet_bootstrap.cc:654] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.354445 12778 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:59.358402 12779 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.359488 12778 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.360803 12687 raft_consensus.cc:359] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.361469 12687 raft_consensus.cc:385] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.361481 12688 tablet_bootstrap.cc:492] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.361806 12687 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.364259 12687 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.370395 12687 ts_tablet_manager.cc:1434] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.004s
I20260709 12:10:59.371447 12687 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.371416 12688 tablet_bootstrap.cc:654] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.373546 12802 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:59.381111 12688 tablet_bootstrap.cc:492] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.381206 12686 tablet_bootstrap.cc:492] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.382124 12688 ts_tablet_manager.cc:1403] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20260709 12:10:59.382969 12686 ts_tablet_manager.cc:1403] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.054s	user 0.007s	sys 0.006s
I20260709 12:10:59.383550 12802 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.384093 12802 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.385462 12688 raft_consensus.cc:359] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.386423 12688 raft_consensus.cc:385] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.386739 12688 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.386512 12686 raft_consensus.cc:359] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.387696 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.387604 12688 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.387985 12686 raft_consensus.cc:385] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.388428 12572 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:59.388491 12686 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.389186 12687 tablet_bootstrap.cc:654] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.390262 12688 ts_tablet_manager.cc:1434] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:10:59.389599 12686 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.390327 12460 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.391464 12688 tablet_bootstrap.cc:492] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.391593 12727 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.391963 12727 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.392311 12727 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.393501 12686 ts_tablet_manager.cc:1434] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.005s
I20260709 12:10:59.395043 12686 tablet_bootstrap.cc:492] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:59.402514 12688 tablet_bootstrap.cc:654] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.404632 12686 tablet_bootstrap.cc:654] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.406340 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.407167 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:59.409068 12802 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.414299 12687 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.414806 12687 ts_tablet_manager.cc:1403] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.044s	user 0.013s	sys 0.014s
I20260709 12:10:59.417877 12802 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:10:59.417616 12687 raft_consensus.cc:359] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.418707 12687 raft_consensus.cc:385] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.419049 12687 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.419884 12687 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.422344 12687 ts_tablet_manager.cc:1434] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:10:59.423681 12687 tablet_bootstrap.cc:492] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.435233 12686 tablet_bootstrap.cc:492] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.435874 12686 ts_tablet_manager.cc:1403] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.041s	user 0.019s	sys 0.000s
I20260709 12:10:59.436522 12688 tablet_bootstrap.cc:492] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.437093 12688 ts_tablet_manager.cc:1403] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.046s	user 0.027s	sys 0.000s
I20260709 12:10:59.438719 12686 raft_consensus.cc:359] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.439438 12686 raft_consensus.cc:385] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.439720 12686 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.439976 12687 tablet_bootstrap.cc:654] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.440053 12688 raft_consensus.cc:359] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.440981 12688 raft_consensus.cc:385] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.441377 12688 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.442157 12688 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.444386 12688 ts_tablet_manager.cc:1434] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.006s
I20260709 12:10:59.445422 12688 tablet_bootstrap.cc:492] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.440521 12686 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.450630 12686 ts_tablet_manager.cc:1434] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 12:10:59.452839 12688 tablet_bootstrap.cc:654] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.455722 12687 tablet_bootstrap.cc:492] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.457267 12687 ts_tablet_manager.cc:1403] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.005s
I20260709 12:10:59.466147 12688 tablet_bootstrap.cc:492] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.466738 12688 ts_tablet_manager.cc:1403] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.007s
I20260709 12:10:59.467101 12729 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.467665 12729 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.469722 12688 raft_consensus.cc:359] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.470489 12688 raft_consensus.cc:385] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.470603 12729 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.470809 12688 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.471809 12688 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.473719 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.474499 12570 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:59.476392 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.478030 12391 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:59.480468 12729 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.481120 12729 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.481796 12729 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.478092 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.480718 12686 tablet_bootstrap.cc:492] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:59.478322 12688 ts_tablet_manager.cc:1434] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:10:59.484239 12645 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:59.485291 12688 tablet_bootstrap.cc:492] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.417213 12727 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.491873 12727 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.493791 12778 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.494432 12778 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.494982 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:59.495707 12570 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.497925 12778 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.509414 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.512533 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.513312 12496 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
W20260709 12:10:59.514518 12609 leader_election.cc:343] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.519730 12570 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:59.521220 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.532738 12687 raft_consensus.cc:359] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.534595 12687 raft_consensus.cc:385] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.534974 12687 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.536402 12687 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.532246 12778 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.537392 12778 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.538841 12687 ts_tablet_manager.cc:1434] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.081s	user 0.006s	sys 0.001s
I20260709 12:10:59.539968 12687 tablet_bootstrap.cc:492] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.542455 12779 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.542896 12779 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.543247 12779 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.543362 12645 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.545683 12698 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.547552 12698 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.549456 12698 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:10:59.536093 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53789.9us
I20260709 12:10:59.551951 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.552368 12778 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.552754 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:59.550777 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.554808 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.555720 12496 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:59.557219 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.553494 12644 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.558068 12698 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.558491 12698 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:10:59.558629 12609 leader_election.cc:343] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.558897 12698 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.564817 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.567517 12720 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.568140 12720 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.569133 12834 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.571427 12720 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.576960 12834 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.579668 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.584182 12779 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.585909 12646 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:10:59.589298 12496 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
W20260709 12:10:59.584633 12609 leader_election.cc:343] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.589607 12768 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.592604 12686 tablet_bootstrap.cc:654] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.590257 12768 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.594595 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.596062 12768 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.600455 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.588385 12779 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:10:59.597669 12729 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.591455 12698 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.605753 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.608600 12835 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:10:59.608902 12460 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.619544 12688 tablet_bootstrap.cc:654] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.611301 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:10:59.620690 12720 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:10:59.614156 12536 leader_election.cc:343] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.622195 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:59.623378 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:59.625787 12694 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.629089 12720 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.637426 12572 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:10:59.635396 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.639997 12698 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.641913 12720 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:10:59.643036 12460 leader_election.cc:343] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.643610 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:10:59.638024 12495 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.644496 12570 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:10:59.645934 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:10:59.646898 12698 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.647287 12698 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.647657 12698 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.639245 12694 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:59.649035 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:59.649611 12694 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.651367 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:59.652334 12643 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:10:59.652668 12768 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.653724 12768 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.655217 12495 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:10:59.655882 12768 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.656548 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.657492 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:10:59.658910 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:59.658927 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:10:59.659359 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.658953 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.662170 12802 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:10:59.662882 12802 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.663256 12802 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.633359 12834 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.657492 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.666388 12496 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:10:59.669759 12770 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.672317 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:10:59.673269 12569 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.678072 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.680367 12729 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.700671 12687 tablet_bootstrap.cc:654] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.703364 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:59.704209 12645 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.706143 12698 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.694690 12770 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:10:59.735654 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121106us
W20260709 12:10:59.750506 12610 leader_election.cc:343] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.751119 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.751915 12686 tablet_bootstrap.cc:492] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.752499 12686 ts_tablet_manager.cc:1403] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.272s	user 0.023s	sys 0.006s
I20260709 12:10:59.754124 12394 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:10:59.762533 12688 tablet_bootstrap.cc:492] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:10:59.763123 12688 ts_tablet_manager.cc:1403] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.278s	user 0.018s	sys 0.020s
I20260709 12:10:59.770674 12686 raft_consensus.cc:359] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.771584 12686 raft_consensus.cc:385] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.772050 12686 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.759541 12729 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:10:59.775344 12686 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.766177 12779 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.772255 12698 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.794134 12686 ts_tablet_manager.cc:1434] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.003s
I20260709 12:10:59.795244 12686 tablet_bootstrap.cc:492] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:10:59.796895 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:10:59.797679 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.804695 12770 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:10:59.807098 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:10:59.789412 12802 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.809773 12645 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:59.815410 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.816476 12770 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.817185 12770 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:10:59.819456 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:59.792307 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.821602 12461 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:10:59.822923 12729 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.823523 12692 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:10:59.825850 12536 leader_election.cc:343] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.822913 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:10:59.827006 12844 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:10:59.828948 12779 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:10:59.829692 12779 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:10:59.830144 12569 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:10:59.831746 12779 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.834426 12729 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:10:59.835515 12729 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.837713 12692 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:10:59.838279 12692 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.844547 12688 raft_consensus.cc:359] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.846258 12688 raft_consensus.cc:385] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.846652 12688 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.847618 12688 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.850922 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:10:59.852170 12496 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:10:59.859804 12686 tablet_bootstrap.cc:654] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.860255 12802 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:10:59.867484 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:10:59.818125 12770 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.873164 12687 tablet_bootstrap.cc:492] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:10:59.873373 12688 ts_tablet_manager.cc:1434] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.110s	user 0.009s	sys 0.001s
I20260709 12:10:59.873687 12687 ts_tablet_manager.cc:1403] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.334s	user 0.012s	sys 0.010s
I20260709 12:10:59.874687 12688 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:10:59.876747 12778 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:59.905975 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:10:59.925549 12460 leader_election.cc:343] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:10:59.931078 12687 raft_consensus.cc:359] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.933223 12687 raft_consensus.cc:385] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.933598 12687 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.935732 12687 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.926296 12610 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:10:59.947772 12778 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:10:59.949661 12840 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:10:59.951258 12686 tablet_bootstrap.cc:492] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:10:59.951498 12687 ts_tablet_manager.cc:1434] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.077s	user 0.007s	sys 0.001s
I20260709 12:10:59.951792 12686 ts_tablet_manager.cc:1403] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.157s	user 0.013s	sys 0.002s
I20260709 12:10:59.953054 12687 tablet_bootstrap.cc:492] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:10:59.967689 12686 raft_consensus.cc:359] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.970165 12686 raft_consensus.cc:385] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:10:59.970551 12686 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:10:59.971376 12686 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:10:59.969214 12687 tablet_bootstrap.cc:654] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.969480 12844 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:10:59.978718 12686 ts_tablet_manager.cc:1434] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 12:10:59.987754 12688 tablet_bootstrap.cc:654] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:10:59.992301 12394 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:10:59.997566 12686 tablet_bootstrap.cc:492] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:00.010077 12692 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.012672 12646 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:00.015350 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:00.016206 12496 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:10:59.996675 12770 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:10:59.988032 12391 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:00.025947 12692 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:00.027439 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:00.029742 12729 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:00.031385 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:00.038414 12395 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:00.020527 12770 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:00.048187 12609 leader_election.cc:343] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:00.048996 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:00.058745 12840 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:00.062182 12686 tablet_bootstrap.cc:654] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.066007 12778 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:00.065501 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:00.085246 12687 tablet_bootstrap.cc:492] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.085855 12687 ts_tablet_manager.cc:1403] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.133s	user 0.026s	sys 0.005s
I20260709 12:11:00.088049 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.089346 12645 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:00.091698 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:00.093531 12698 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:00.093971 12698 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.094313 12698 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.094425 12687 raft_consensus.cc:359] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.095264 12687 raft_consensus.cc:385] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.095444 12687 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.074224 12770 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.096154 12687 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.098373 12687 ts_tablet_manager.cc:1434] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:11:00.099390 12687 tablet_bootstrap.cc:492] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:00.100566 12688 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:00.101127 12688 ts_tablet_manager.cc:1403] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.227s	user 0.009s	sys 0.018s
I20260709 12:11:00.078326 12497 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:00.105087 12688 raft_consensus.cc:359] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.107016 12688 raft_consensus.cc:385] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.107378 12688 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.108349 12688 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.110585 12687 tablet_bootstrap.cc:654] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.111109 12688 ts_tablet_manager.cc:1434] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:11:00.112242 12688 tablet_bootstrap.cc:492] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:00.125248 12729 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:00.126632 12688 tablet_bootstrap.cc:654] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.126395 12698 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.130817 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:00.132462 12646 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.137063 12698 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.152372 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:00.153256 12497 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.160104 12686 tablet_bootstrap.cc:492] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:00.161474 12686 ts_tablet_manager.cc:1403] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.164s	user 0.008s	sys 0.013s
I20260709 12:11:00.202245 12834 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:00.205677 12687 tablet_bootstrap.cc:492] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.206450 12687 ts_tablet_manager.cc:1403] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.107s	user 0.031s	sys 0.035s
I20260709 12:11:00.210041 12687 raft_consensus.cc:359] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.210963 12687 raft_consensus.cc:385] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.211313 12687 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.212210 12687 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.214519 12687 ts_tablet_manager.cc:1434] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260709 12:11:00.215996 12687 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:00.217921 12646 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:00.223923 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:00.227702 12698 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:00.229276 12834 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.230432 12834 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.232528 12687 tablet_bootstrap.cc:654] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.229269 12698 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:00.234809 12686 raft_consensus.cc:359] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.235471 12686 raft_consensus.cc:385] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.235710 12686 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.236274 12686 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.223157 12497 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:00.242198 12698 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.248502 12686 ts_tablet_manager.cc:1434] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.087s	user 0.004s	sys 0.002s
I20260709 12:11:00.245553 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:00.250521 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:00.250844 12688 tablet_bootstrap.cc:492] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:00.253134 12688 ts_tablet_manager.cc:1403] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.141s	user 0.037s	sys 0.036s
I20260709 12:11:00.261214 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.262192 12729 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:00.262588 12729 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.262648 12694 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:00.263031 12729 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.265010 12686 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:00.264719 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.265568 12646 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:00.267999 12834 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:00.276250 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:00.279779 12841 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:00.283164 12688 raft_consensus.cc:359] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.283954 12688 raft_consensus.cc:385] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.284229 12688 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.286481 12729 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.287040 12646 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.301816 12688 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.302212 12770 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.303050 12770 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.306064 12770 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.310581 12686 tablet_bootstrap.cc:654] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.310689 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.313570 12688 ts_tablet_manager.cc:1434] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.059s	user 0.007s	sys 0.000s
I20260709 12:11:00.323922 12768 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:11:00.325562 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:00.325770 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:00.326361 12642 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.326643 12494 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:00.326869 12688 tablet_bootstrap.cc:492] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:00.328099 12645 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:00.322145 12729 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.334578 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:00.335603 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:00.339666 12571 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.343428 12844 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:00.354099 12768 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.367592 12687 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.368245 12687 ts_tablet_manager.cc:1403] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.152s	user 0.024s	sys 0.029s
I20260709 12:11:00.382153 12687 raft_consensus.cc:359] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.383205 12687 raft_consensus.cc:385] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.383564 12687 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.395568 12768 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:00.396037 12642 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:00.399536 12687 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.400933 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.402048 12834 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:00.402591 12834 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:00.403509 12834 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.409302 12687 ts_tablet_manager.cc:1434] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.001s
I20260709 12:11:00.414111 12687 tablet_bootstrap.cc:492] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:00.422565 12688 tablet_bootstrap.cc:654] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.422435 12835 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:00.430027 12394 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:00.428848 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:00.445946 12770 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:00.447223 12686 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:00.447813 12686 ts_tablet_manager.cc:1403] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.183s	user 0.023s	sys 0.031s
I20260709 12:11:00.450095 12844 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.451364 12844 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.456025 12686 raft_consensus.cc:359] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.456892 12686 raft_consensus.cc:385] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.457218 12686 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.458081 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:00.457985 12686 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.466547 12686 ts_tablet_manager.cc:1434] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.001s
I20260709 12:11:00.468530 12686 tablet_bootstrap.cc:492] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:00.462479 12844 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:00.462039 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:00.478822 12687 tablet_bootstrap.cc:654] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.479480 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:00.440704 12392 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:00.484316 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:00.485046 12609 leader_election.cc:343] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:00.490430 12841 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:00.490933 12841 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.491315 12841 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.508713 12841 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.514552 12687 tablet_bootstrap.cc:492] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.515686 12687 ts_tablet_manager.cc:1403] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.102s	user 0.012s	sys 0.002s
I20260709 12:11:00.516954 12686 tablet_bootstrap.cc:654] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.522059 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:00.522691 12497 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.530220 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:00.531596 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:00.533453 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.533367 12687 raft_consensus.cc:359] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.534425 12687 raft_consensus.cc:385] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.534562 12840 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:00.534792 12687 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.534612 12841 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:00.535398 12840 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:00.535773 12687 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.536312 12840 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.536763 12571 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.539228 12688 tablet_bootstrap.cc:492] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:00.539762 12688 ts_tablet_manager.cc:1403] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.213s	user 0.017s	sys 0.010s
I20260709 12:11:00.542125 12687 ts_tablet_manager.cc:1434] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.003s
I20260709 12:11:00.543366 12687 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:00.544938 12688 raft_consensus.cc:359] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.548602 12688 raft_consensus.cc:385] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.549075 12688 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.550444 12688 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.553263 12688 ts_tablet_manager.cc:1434] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:11:00.555501 12688 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:00.559921 12571 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:00.565322 12688 tablet_bootstrap.cc:654] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.565485 12687 tablet_bootstrap.cc:654] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.570206 12686 tablet_bootstrap.cc:492] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:00.570652 12686 ts_tablet_manager.cc:1403] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.102s	user 0.017s	sys 0.003s
I20260709 12:11:00.572739 12686 raft_consensus.cc:359] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.573314 12686 raft_consensus.cc:385] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.573499 12686 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.574286 12686 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.574584 12391 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:00.576602 12687 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.577086 12687 ts_tablet_manager.cc:1403] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.034s	user 0.011s	sys 0.005s
I20260709 12:11:00.579725 12687 raft_consensus.cc:359] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.580361 12687 raft_consensus.cc:385] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.580600 12687 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.582165 12687 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.584698 12687 ts_tablet_manager.cc:1434] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.005s
I20260709 12:11:00.585609 12687 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:00.589329 12834 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.590260 12834 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.591672 12687 tablet_bootstrap.cc:654] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.594504 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:00.596215 12460 leader_election.cc:343] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:00.596712 12834 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.600500 12686 ts_tablet_manager.cc:1434] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 12:11:00.600967 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.601581 12688 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:00.601892 12686 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:00.601794 12645 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:00.602272 12688 ts_tablet_manager.cc:1403] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.017s
I20260709 12:11:00.604051 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:00.605542 12834 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:00.606103 12834 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.605959 12688 raft_consensus.cc:359] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.606725 12834 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.607069 12688 raft_consensus.cc:385] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.607589 12688 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.608525 12688 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.611702 12688 ts_tablet_manager.cc:1434] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:11:00.612957 12688 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:00.640775 12686 tablet_bootstrap.cc:654] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.643167 12834 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.644097 12688 tablet_bootstrap.cc:654] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.645464 12835 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.646157 12835 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.656076 12770 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.656839 12770 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.659511 12770 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.667131 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.667749 12770 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.668264 12770 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.670112 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:00.670284 12687 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.670919 12497 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:00.671291 12687 ts_tablet_manager.cc:1403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.086s	user 0.022s	sys 0.040s
I20260709 12:11:00.684393 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.690510 12686 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:00.691093 12686 ts_tablet_manager.cc:1403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.090s	user 0.013s	sys 0.022s
W20260709 12:11:00.691862 12536 leader_election.cc:343] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:00.687162 12770 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.695542 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:00.695720 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:00.696339 12494 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:00.696965 12571 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:00.701001 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.701809 12645 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:00.703965 12698 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:00.706331 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.719661 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:00.707377 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.720475 12644 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:00.721662 12770 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:00.722112 12770 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.722160 12688 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:00.722610 12770 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:00.722817 12835 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.722910 12688 ts_tablet_manager.cc:1403] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.110s	user 0.029s	sys 0.035s
I20260709 12:11:00.723423 12835 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:00.724614 12835 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.724992 12835 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:00.709678 12687 raft_consensus.cc:359] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.710040 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:00.726423 12687 raft_consensus.cc:385] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.726774 12687 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.726615 12642 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.728231 12688 raft_consensus.cc:359] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.729094 12688 raft_consensus.cc:385] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.711125 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:00.715126 12686 raft_consensus.cc:359] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.707768 12834 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.718366 12698 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:00.730106 12686 raft_consensus.cc:385] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.730195 12698 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.730504 12686 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.731542 12686 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.733816 12686 ts_tablet_manager.cc:1434] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.004s
I20260709 12:11:00.734844 12686 tablet_bootstrap.cc:492] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:00.731359 12770 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.729432 12688 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.737613 12688 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.740140 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:00.746045 12686 tablet_bootstrap.cc:654] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.746851 12698 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.752266 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:00.753489 12497 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:00.754668 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:00.755156 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:00.727574 12687 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.756481 12770 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.740257 12688 ts_tablet_manager.cc:1434] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
I20260709 12:11:00.767349 12688 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:00.764640 12645 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:00.782050 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61808us
I20260709 12:11:00.784276 12687 ts_tablet_manager.cc:1434] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.112s	user 0.008s	sys 0.004s
I20260709 12:11:00.785414 12687 tablet_bootstrap.cc:492] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:00.791200 12698 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.793928 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:00.794806 12642 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:00.794971 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72556.2us
W20260709 12:11:00.795996 12460 leader_election.cc:343] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:00.806552 12686 tablet_bootstrap.cc:492] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:00.807153 12686 ts_tablet_manager.cc:1403] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.073s	user 0.008s	sys 0.011s
I20260709 12:11:00.812203 12642 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:00.812654 12686 raft_consensus.cc:359] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.810027 12497 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:00.813483 12686 raft_consensus.cc:385] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.813827 12686 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.814570 12686 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.815794 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:00.816743 12698 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:00.817718 12770 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:00.818653 12698 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:00.820557 12686 ts_tablet_manager.cc:1434] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:11:00.820264 12835 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.821514 12494 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:00.821669 12686 tablet_bootstrap.cc:492] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:00.821878 12687 tablet_bootstrap.cc:654] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.819772 12698 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.823071 12835 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.826184 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:00.827466 12645 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:00.828214 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:00.829499 12835 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:00.829917 12569 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 2: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:00.831252 12835 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:00.832412 12835 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.835088 12688 tablet_bootstrap.cc:654] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.835726 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:00.836954 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:00.845536 12770 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:00.846184 12770 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:00.848516 12646 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 2: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:00.851872 12687 tablet_bootstrap.cc:492] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:00.852200 12686 tablet_bootstrap.cc:654] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:00.852476 12687 ts_tablet_manager.cc:1403] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.067s	user 0.009s	sys 0.014s
I20260709 12:11:00.855960 12770 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.853493 12884 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:11:00.862974 12768 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:00.896647 12461 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:00.899329 12884 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:00.900138 12884 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.900739 12884 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.928328 12840 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.928965 12840 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.968788 12890 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.969412 12890 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.971881 12890 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:00.972939 12890 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.984877 12687 raft_consensus.cc:359] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:00.986171 12687 raft_consensus.cc:385] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:00.986593 12687 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:00.990429 12835 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:00.990832 12687 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.002791 12688 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:01.003574 12688 ts_tablet_manager.cc:1403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.237s	user 0.011s	sys 0.025s
I20260709 12:11:01.013851 12687 ts_tablet_manager.cc:1434] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.161s	user 0.012s	sys 0.001s
I20260709 12:11:00.992142 12835 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.031278 12391 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:01.007239 12890 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.035485 12687 tablet_bootstrap.cc:492] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:01.036664 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.035034 12884 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.042886 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.050782 12569 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:01.051813 12835 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.052531 12686 tablet_bootstrap.cc:492] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:01.053159 12686 ts_tablet_manager.cc:1403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.232s	user 0.023s	sys 0.015s
I20260709 12:11:01.075047 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:01.095502 12688 raft_consensus.cc:359] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.096568 12688 raft_consensus.cc:385] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.096973 12688 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.103955 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:01.105645 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:01.105857 12840 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:01.073478 12610 leader_election.cc:343] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:01.108489 12688 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.113709 12688 ts_tablet_manager.cc:1434] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.110s	user 0.012s	sys 0.000s
I20260709 12:11:01.114919 12688 tablet_bootstrap.cc:492] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
W20260709 12:11:01.105870 12461 leader_election.cc:343] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:01.115507 12643 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
W20260709 12:11:01.143244 12892 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:11:01.148120 12609 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:01.149569 12840 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:11:01.149854 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77583.2us
I20260709 12:11:01.150301 12840 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:01.151511 12840 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:01.188027 12536 leader_election.cc:343] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:01.192888 12888 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:11:01.211762 12894 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.211208 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.197338 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:01.221791 12686 raft_consensus.cc:359] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.221782 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.231065 12686 raft_consensus.cc:385] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.233222 12686 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.234193 12686 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.243062 12395 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:01.233271 12890 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.241432 12883 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:01.250927 12883 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:01.251327 12883 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.250397 12686 ts_tablet_manager.cc:1434] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.197s	user 0.016s	sys 0.010s
I20260709 12:11:01.254233 12686 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:01.225761 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.259577 12687 tablet_bootstrap.cc:654] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.228178 12571 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:01.225955 12894 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.273329 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.278414 12894 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.280819 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:01.313516 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79084.3us
I20260709 12:11:01.324806 12694 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:01.327013 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.319650 12840 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.330269 12646 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:01.336993 12688 tablet_bootstrap.cc:654] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.336907 12840 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:01.338485 12460 leader_election.cc:343] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:01.341289 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:01.344700 12698 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:11:01.348065 12536 leader_election.cc:343] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:01.349393 12687 tablet_bootstrap.cc:492] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:01.349968 12687 ts_tablet_manager.cc:1403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.319s	user 0.013s	sys 0.013s
I20260709 12:11:01.338686 12883 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.351227 12494 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:01.350991 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:01.351962 12643 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:01.358336 12884 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:01.369829 12883 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.379195 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:01.381695 12894 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.383924 12687 raft_consensus.cc:359] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.384932 12687 raft_consensus.cc:385] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.385296 12687 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.387641 12687 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.391798 12686 tablet_bootstrap.cc:654] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.406842 12687 ts_tablet_manager.cc:1434] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.057s	user 0.002s	sys 0.007s
I20260709 12:11:01.407853 12687 tablet_bootstrap.cc:492] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:01.413024 12688 tablet_bootstrap.cc:492] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
W20260709 12:11:01.384141 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89896us
I20260709 12:11:01.415405 12905 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.413704 12688 ts_tablet_manager.cc:1403] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.299s	user 0.026s	sys 0.006s
I20260709 12:11:01.426059 12688 raft_consensus.cc:359] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.426715 12688 raft_consensus.cc:385] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.426909 12688 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.427440 12688 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.445129 12688 ts_tablet_manager.cc:1434] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:11:01.446367 12688 tablet_bootstrap.cc:492] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:01.450733 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:01.449784 12894 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:01.451462 12643 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.460521 12898 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:01.456810 12909 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:11:01.462394 12687 tablet_bootstrap.cc:654] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.423878 12905 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.465906 12643 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:01.471329 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:01.473673 12905 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.474666 12898 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:01.475479 12898 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:01.476567 12898 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.496196 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:01.500737 12686 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:01.501371 12686 ts_tablet_manager.cc:1403] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.248s	user 0.015s	sys 0.022s
I20260709 12:11:01.504577 12688 tablet_bootstrap.cc:654] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.504882 12686 raft_consensus.cc:359] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.506042 12686 raft_consensus.cc:385] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.506319 12686 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.506879 12686 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.512534 12686 ts_tablet_manager.cc:1434] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:11:01.514403 12686 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
W20260709 12:11:01.506021 12461 leader_election.cc:343] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:01.524354 12687 tablet_bootstrap.cc:492] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:01.524962 12687 ts_tablet_manager.cc:1403] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.117s	user 0.011s	sys 0.006s
I20260709 12:11:01.543452 12688 tablet_bootstrap.cc:492] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:01.545138 12688 ts_tablet_manager.cc:1403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.099s	user 0.013s	sys 0.006s
I20260709 12:11:01.545444 12686 tablet_bootstrap.cc:654] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.680685 12905 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.681335 12905 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.683951 12905 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.743824 12894 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.744515 12894 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.744899 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:01.746124 12496 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.747052 12894 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.753414 12910 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:01.759162 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:01.760828 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:01.762152 12910 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:01.762694 12910 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:01.763689 12910 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.765762 12688 raft_consensus.cc:359] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.766568 12688 raft_consensus.cc:385] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.766834 12688 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.766336 12687 raft_consensus.cc:359] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.767403 12687 raft_consensus.cc:385] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.767485 12688 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.770741 12688 ts_tablet_manager.cc:1434] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.225s	user 0.011s	sys 0.000s
I20260709 12:11:01.771186 12687 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.771948 12687 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.772507 12688 tablet_bootstrap.cc:492] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:01.777248 12687 ts_tablet_manager.cc:1434] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.252s	user 0.011s	sys 0.002s
I20260709 12:11:01.778611 12687 tablet_bootstrap.cc:492] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:01.780525 12688 tablet_bootstrap.cc:654] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.782625 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:01.783340 12570 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.783202 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.785013 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.785766 12569 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
W20260709 12:11:01.795909 12460 leader_election.cc:343] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:01.796551 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:01.797075 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:01.798970 12905 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:01.800904 12687 tablet_bootstrap.cc:654] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.806079 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:01.807291 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:01.818504 12688 tablet_bootstrap.cc:492] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:01.820072 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:01.820538 12905 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:11:01.819244 12688 ts_tablet_manager.cc:1403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.048s	user 0.016s	sys 0.013s
I20260709 12:11:01.824571 12834 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:01.825001 12834 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:01.827469 12834 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:01.833034 12841 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.833751 12841 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.840020 12688 raft_consensus.cc:359] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.841009 12688 raft_consensus.cc:385] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.841370 12688 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.842360 12688 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.858604 12688 ts_tablet_manager.cc:1434] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.000s
I20260709 12:11:01.860532 12688 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:01.864594 12896 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.867003 12896 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.888451 12896 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.883975 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.890005 12834 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.891183 12497 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:01.890376 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.892465 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:01.892838 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:01.894037 12902 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:01.894521 12902 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:01.894912 12902 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.895781 12834 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:01.881418 12841 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:01.901539 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:01.902376 12572 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:01.902801 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:01.903663 12570 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:01.911530 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 532453us
I20260709 12:11:01.912531 12536 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:01.917199 12570 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:01.919690 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:01.919988 12686 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:01.920713 12686 ts_tablet_manager.cc:1403] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.407s	user 0.035s	sys 0.027s
I20260709 12:11:01.920795 12907 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:01.921509 12907 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:01.922488 12907 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.924583 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:01.925967 12770 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:01.934809 12686 raft_consensus.cc:359] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.935986 12686 raft_consensus.cc:385] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.936344 12686 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.936856 12572 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:01.937467 12698 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:01.937216 12686 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.938939 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:01.943153 12643 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:01.958429 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:01.951534 12841 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:01.963339 12841 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.962577 12687 tablet_bootstrap.cc:492] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:01.965603 12687 ts_tablet_manager.cc:1403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.187s	user 0.011s	sys 0.022s
I20260709 12:11:01.966838 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.967052 12841 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:01.968420 12687 raft_consensus.cc:359] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.969158 12687 raft_consensus.cc:385] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:01.969426 12687 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:01.970548 12687 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:01.972316 12687 ts_tablet_manager.cc:1434] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:11:01.974097 12686 ts_tablet_manager.cc:1434] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.053s	user 0.011s	sys 0.000s
I20260709 12:11:01.974615 12687 tablet_bootstrap.cc:492] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:01.975436 12686 tablet_bootstrap.cc:492] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:01.972584 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.977378 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:01.978425 12490 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:01.984197 12644 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:11:01.984934 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:01.968009 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:01.951506 12902 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:01.969340 12610 leader_election.cc:343] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.005189 12688 tablet_bootstrap.cc:654] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:01.993315 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:02.009294 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.011345 12490 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:11:02.011713 12844 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:02.012482 12844 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:02.012971 12844 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.016978 12720 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:02.017549 12720 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.021346 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:02.023016 12720 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:02.026180 12687 tablet_bootstrap.cc:654] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:02.026577 12536 leader_election.cc:343] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.030820 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:02.039422 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:02.040452 12644 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:02.043562 12698 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:11:02.046504 12698 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:02.046901 12395 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 2, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:02.043782 12770 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:11:02.043905 12894 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:02.052909 12894 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:02.053316 12894 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.030656 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:02.056625 12609 leader_election.cc:343] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.069986 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:02.072187 12844 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.072889 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:02.073859 12495 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.074415 12645 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:11:02.084420 12610 leader_election.cc:343] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.087766 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.088428 12844 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:02.089190 12720 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:02.095968 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:02.097117 12568 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.066423 12894 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.092792 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:02.112501 12610 leader_election.cc:343] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.113577 12902 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:02.114563 12905 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:02.161371 12688 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:02.162137 12688 ts_tablet_manager.cc:1403] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.302s	user 0.020s	sys 0.019s
I20260709 12:11:02.163671 12686 tablet_bootstrap.cc:654] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:02.183655 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:02.172979 12688 raft_consensus.cc:359] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.184996 12688 raft_consensus.cc:385] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.185411 12688 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.193244 12688 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.202339 12894 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.203459 12688 ts_tablet_manager.cc:1434] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.041s	user 0.010s	sys 0.000s
I20260709 12:11:02.205278 12688 tablet_bootstrap.cc:492] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:02.209412 12568 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:02.220589 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.222213 12894 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:02.219974 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.224970 12902 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:02.226179 12902 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:02.212667 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:02.235881 12902 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.239239 12687 tablet_bootstrap.cc:492] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:02.240052 12687 ts_tablet_manager.cc:1403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.266s	user 0.028s	sys 0.010s
I20260709 12:11:02.242591 12569 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:02.241816 12894 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:02.245251 12894 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.290903 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:02.290643 12687 raft_consensus.cc:359] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.292415 12687 raft_consensus.cc:385] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.292490 12643 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:02.292986 12687 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.298853 12894 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.299485 12883 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:02.300400 12883 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.306804 12687 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.312264 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:02.313490 12687 ts_tablet_manager.cc:1434] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.073s	user 0.012s	sys 0.000s
W20260709 12:11:02.315250 12460 leader_election.cc:343] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.315953 12687 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:02.324523 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:02.325881 12496 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:02.280452 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:02.336685 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.356599 12931 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:02.379642 12901 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:02.380347 12931 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:02.381009 12931 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.381574 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:02.387503 12840 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260709 12:11:02.392338 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55738.1us
I20260709 12:11:02.393528 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:02.395778 12686 tablet_bootstrap.cc:492] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:02.396440 12686 ts_tablet_manager.cc:1403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.421s	user 0.020s	sys 0.020s
W20260709 12:11:02.397830 12536 leader_election.cc:343] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.386219 12883 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.410579 12931 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.380373 12901 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.412382 12688 tablet_bootstrap.cc:654] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:02.415556 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:02.406265 12770 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:02.416708 12770 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:02.417173 12770 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.419175 12901 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:02.419353 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:02.420092 12645 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:02.421397 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:02.421838 12643 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:02.423084 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:11:02.438045 12609 leader_election.cc:343] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.427383 12841 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:02.458534 12841 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:02.461123 12841 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.483842 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:02.445829 12901 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
I20260709 12:11:02.487375 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:02.488462 12490 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:02.495002 12686 raft_consensus.cc:359] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.496106 12686 raft_consensus.cc:385] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.496448 12686 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.497391 12686 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.502892 12686 ts_tablet_manager.cc:1434] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.106s	user 0.009s	sys 0.000s
W20260709 12:11:02.512107 12536 leader_election.cc:343] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.501607 12905 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:11:02.529532 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:02.531366 12840 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:02.531999 12840 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:02.532847 12840 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.535077 12645 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:11:02.543334 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.547178 12687 tablet_bootstrap.cc:654] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:02.553740 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080016aa20 after lost signal to thread 11313
I20260709 12:11:02.553900 12698 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:02.554867 12698 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:02.555450 12698 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:02.555868 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08000f82c0 after lost signal to thread 12429
I20260709 12:11:02.560007 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:02.562634 12898 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:02.563334 12898 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:02.563934 12898 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.567776 12392 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 2, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:11:02.576135 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080025b4e0 after lost signal to thread 12521
W20260709 12:11:02.577986 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080025acc0 after lost signal to thread 12596
W20260709 12:11:02.579721 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080024ab80 after lost signal to thread 12670
I20260709 12:11:02.579008 12392 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:02.539680 12770 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.562049 12686 tablet_bootstrap.cc:492] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:02.598264 12883 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:11:02.615363 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:02.616555 12495 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.606930 12770 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.628615 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:02.629910 12698 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.631489 12568 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:11:02.634472 12840 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.637401 12898 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.642175 12698 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.640228 12495 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:02.644654 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:02.650880 12496 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.659869 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:02.668470 12688 tablet_bootstrap.cc:492] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:02.642994 12898 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:02.669332 12688 ts_tablet_manager.cc:1403] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.464s	user 0.026s	sys 0.029s
I20260709 12:11:02.669662 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:02.663655 12491 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:02.671175 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:02.673615 12698 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:02.671480 12686 tablet_bootstrap.cc:654] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:02.675592 12698 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:02.676345 12698 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.667050 12840 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:02.681013 12609 leader_election.cc:343] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.681370 12496 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:02.682727 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:02.683563 12770 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:11:02.683732 12536 leader_election.cc:343] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.684161 12770 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:02.685128 12770 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.685985 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:02.686829 12643 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:02.690709 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:02.691730 12883 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:02.693372 12844 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:02.695070 12688 raft_consensus.cc:359] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.697841 12688 raft_consensus.cc:385] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.698092 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:02.698885 12643 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:02.698374 12688 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.717795 12688 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.712656 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:02.738245 12460 leader_election.cc:343] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:02.739041 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:02.741372 12883 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:02.739729 12391 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:02.743850 12688 ts_tablet_manager.cc:1434] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.000s
I20260709 12:11:02.745791 12688 tablet_bootstrap.cc:492] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:02.759676 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:02.760212 12687 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:02.760771 12687 ts_tablet_manager.cc:1403] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.445s	user 0.014s	sys 0.014s
I20260709 12:11:02.772943 12686 tablet_bootstrap.cc:492] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:02.773473 12686 ts_tablet_manager.cc:1403] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.212s	user 0.015s	sys 0.010s
I20260709 12:11:02.790109 12686 raft_consensus.cc:359] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.790925 12686 raft_consensus.cc:385] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.791194 12686 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.792147 12686 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:02.793833 12610 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:02.803617 12687 raft_consensus.cc:359] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.804713 12687 raft_consensus.cc:385] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.805064 12687 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.808954 12686 ts_tablet_manager.cc:1434] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:11:02.810091 12686 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:02.814062 12688 tablet_bootstrap.cc:654] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:02.822083 12687 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.824664 12687 ts_tablet_manager.cc:1434] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.064s	user 0.002s	sys 0.006s
I20260709 12:11:02.825840 12687 tablet_bootstrap.cc:492] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:02.831456 12686 tablet_bootstrap.cc:654] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:02.842631 12687 tablet_bootstrap.cc:654] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:02.847116 12395 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:02.849500 12395 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:02.857517 12840 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:02.862327 12688 tablet_bootstrap.cc:492] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:02.862838 12688 ts_tablet_manager.cc:1403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.118s	user 0.013s	sys 0.006s
I20260709 12:11:02.873544 12686 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:02.874190 12686 ts_tablet_manager.cc:1403] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.064s	user 0.008s	sys 0.013s
I20260709 12:11:02.877110 12687 tablet_bootstrap.cc:492] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:02.877712 12687 ts_tablet_manager.cc:1403] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.052s	user 0.016s	sys 0.000s
I20260709 12:11:02.880297 12688 raft_consensus.cc:359] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.881042 12688 raft_consensus.cc:385] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.881323 12688 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.881222 12687 raft_consensus.cc:359] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.882140 12687 raft_consensus.cc:385] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:02.882541 12687 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:02.882184 12688 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:02.961586 12391 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:02.976395 12687 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.007223 12687 ts_tablet_manager.cc:1434] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.129s	user 0.007s	sys 0.000s
I20260709 12:11:03.010258 12391 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:03.016299 12687 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:03.011097 12686 raft_consensus.cc:359] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.020541 12840 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.021334 12686 raft_consensus.cc:385] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.023577 12686 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.035025 12688 ts_tablet_manager.cc:1434] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.172s	user 0.009s	sys 0.005s
I20260709 12:11:03.035137 12840 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.037748 12688 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:03.038612 12686 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.112432 12840 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:03.119788 12686 ts_tablet_manager.cc:1434] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.245s	user 0.015s	sys 0.028s
I20260709 12:11:03.127538 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:03.122423 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:03.129292 12965 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:03.129964 12497 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:03.132063 12686 tablet_bootstrap.cc:492] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:03.135741 12687 tablet_bootstrap.cc:654] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.145273 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:03.151298 12966 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:03.151973 12966 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:03.152498 12966 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.174284 12898 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.174991 12898 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.178581 12898 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:03.188166 12609 leader_election.cc:343] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.214653 12841 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.215432 12841 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.219267 12644 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.220225 12688 tablet_bootstrap.cc:654] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.220798 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:03.223548 12460 leader_election.cc:343] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.225582 12841 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:03.228526 12946 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.229163 12946 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.229391 12840 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:03.232122 12946 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:03.245812 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:03.247098 12688 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:03.247756 12688 ts_tablet_manager.cc:1403] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.211s	user 0.021s	sys 0.016s
I20260709 12:11:03.249881 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:03.251106 12491 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:03.254061 12686 tablet_bootstrap.cc:654] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.254761 12688 raft_consensus.cc:359] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.257017 12698 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.257587 12698 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.258858 12896 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:03.260138 12698 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:03.256309 12688 raft_consensus.cc:385] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.260840 12688 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.261691 12688 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.262734 12687 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:03.263278 12687 ts_tablet_manager.cc:1403] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.254s	user 0.021s	sys 0.014s
I20260709 12:11:03.263507 12688 ts_tablet_manager.cc:1434] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.002s
I20260709 12:11:03.265072 12688 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:03.265777 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:03.266868 12495 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:03.280260 12687 raft_consensus.cc:359] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.282199 12687 raft_consensus.cc:385] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.282496 12687 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.277347 12966 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.298404 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:03.299640 12641 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:03.303655 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:03.308929 12687 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.314853 12720 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:03.318567 12966 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:03.318509 12687 ts_tablet_manager.cc:1434] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.055s	user 0.010s	sys 0.000s
I20260709 12:11:03.330446 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:03.330792 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:03.331576 12497 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:03.336750 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53738.9us
W20260709 12:11:03.350333 12609 leader_election.cc:343] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:03.350908 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:03.351409 12686 tablet_bootstrap.cc:492] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:03.352048 12686 ts_tablet_manager.cc:1403] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.231s	user 0.010s	sys 0.011s
I20260709 12:11:03.354782 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:03.355856 12646 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:03.358049 12687 tablet_bootstrap.cc:492] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:03.362848 12461 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:03.368830 12686 raft_consensus.cc:359] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.371227 12686 raft_consensus.cc:385] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.371650 12686 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.372547 12686 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.373536 12834 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:03.374053 12834 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:03.374384 12834 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.375284 12686 ts_tablet_manager.cc:1434] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:11:03.378070 12497 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:03.380367 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:03.394961 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:03.398007 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:03.388197 12966 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:03.399767 12686 tablet_bootstrap.cc:492] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:03.399469 12966 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:03.401458 12966 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.421782 12971 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.056s
W20260709 12:11:03.426110 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47973.9us
I20260709 12:11:03.429962 12834 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.441823 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:03.440728 12971 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260709 12:11:03.450091 12536 leader_election.cc:343] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:03.453055 12536 leader_election.cc:343] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.453534 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:03.455049 12395 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:03.457002 12946 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:03.451902 12971 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.464082 12770 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:11:03.468252 12688 tablet_bootstrap.cc:654] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.477694 12834 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:03.478829 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:03.479288 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:03.479877 12645 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.479545 12568 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.492043 12687 tablet_bootstrap.cc:654] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.495430 12645 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:03.507653 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:03.512811 12686 tablet_bootstrap.cc:654] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.516989 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:03.526978 12883 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:03.527482 12883 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:03.528447 12883 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:03.579041 12979 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:11:03.590552 12395 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:03.609956 12971 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:03.620970 12686 tablet_bootstrap.cc:492] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:03.621304 12688 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:03.622403 12686 ts_tablet_manager.cc:1403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.223s	user 0.010s	sys 0.009s
I20260709 12:11:03.622396 12688 ts_tablet_manager.cc:1403] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.358s	user 0.021s	sys 0.023s
I20260709 12:11:03.625560 12687 tablet_bootstrap.cc:492] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:03.625466 12686 raft_consensus.cc:359] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.626258 12687 ts_tablet_manager.cc:1403] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.290s	user 0.023s	sys 0.001s
I20260709 12:11:03.626325 12686 raft_consensus.cc:385] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.626722 12686 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.625600 12688 raft_consensus.cc:359] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.630545 12688 raft_consensus.cc:385] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.630842 12688 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.631748 12688 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.634315 12688 ts_tablet_manager.cc:1434] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20260709 12:11:03.635634 12688 tablet_bootstrap.cc:492] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:03.629422 12687 raft_consensus.cc:359] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.642302 12687 raft_consensus.cc:385] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.642616 12687 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.642969 12686 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.643328 12687 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.645790 12687 ts_tablet_manager.cc:1434] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:11:03.646816 12686 ts_tablet_manager.cc:1434] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260709 12:11:03.647411 12883 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.647778 12688 tablet_bootstrap.cc:654] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.648123 12686 tablet_bootstrap.cc:492] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:03.649442 12687 tablet_bootstrap.cc:492] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:03.647907 12883 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.664471 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:03.668308 12687 tablet_bootstrap.cc:654] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.669584 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:03.670351 12883 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:03.676764 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:03.677923 12907 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:03.690282 12686 tablet_bootstrap.cc:654] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.714563 12688 tablet_bootstrap.cc:492] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:03.715299 12688 ts_tablet_manager.cc:1403] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.080s	user 0.014s	sys 0.038s
I20260709 12:11:03.719678 12688 raft_consensus.cc:359] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.722029 12687 tablet_bootstrap.cc:492] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:03.722659 12687 ts_tablet_manager.cc:1403] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.074s	user 0.011s	sys 0.015s
I20260709 12:11:03.722029 12686 tablet_bootstrap.cc:492] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:03.725334 12686 ts_tablet_manager.cc:1403] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.078s	user 0.012s	sys 0.027s
I20260709 12:11:03.728394 12493 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.729221 12686 raft_consensus.cc:359] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.731652 12687 raft_consensus.cc:359] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.732431 12687 raft_consensus.cc:385] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.732705 12687 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.741186 12687 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.720587 12688 raft_consensus.cc:385] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.742692 12688 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.744174 12687 ts_tablet_manager.cc:1434] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:11:03.730054 12686 raft_consensus.cc:385] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.745281 12686 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.745591 12688 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.746282 12686 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.747434 12883 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.747908 12883 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.748838 12686 ts_tablet_manager.cc:1434] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:11:03.750121 12686 tablet_bootstrap.cc:492] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:03.749982 12883 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:03.754725 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:03.756423 12460 leader_election.cc:343] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.757550 12688 ts_tablet_manager.cc:1434] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.001s
I20260709 12:11:03.758718 12687 tablet_bootstrap.cc:492] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:03.759138 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:03.760038 12644 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:03.758725 12688 tablet_bootstrap.cc:492] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:03.762411 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:03.765864 12907 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:03.766311 12907 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:03.766628 12907 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.780283 12907 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.795225 12687 tablet_bootstrap.cc:654] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.796751 12688 tablet_bootstrap.cc:654] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.796062 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:03.800808 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:03.801614 12907 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:03.802727 12644 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:03.802685 12460 leader_election.cc:343] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.803326 12686 tablet_bootstrap.cc:654] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.812058 12688 tablet_bootstrap.cc:492] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:03.812676 12688 ts_tablet_manager.cc:1403] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.054s	user 0.007s	sys 0.006s
I20260709 12:11:03.816216 12688 raft_consensus.cc:359] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.817057 12688 raft_consensus.cc:385] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.817359 12688 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.818446 12688 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.826215 12644 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:03.829762 12688 ts_tablet_manager.cc:1434] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.007s
I20260709 12:11:03.837661 12688 tablet_bootstrap.cc:492] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:03.842259 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:03.843570 12883 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:03.843776 12720 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.844148 12883 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:03.844293 12720 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.845811 12688 tablet_bootstrap.cc:654] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.845366 12883 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.852172 12687 tablet_bootstrap.cc:492] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:03.852813 12687 ts_tablet_manager.cc:1403] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.098s	user 0.010s	sys 0.010s
I20260709 12:11:03.854954 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:03.855895 12568 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:03.856333 12687 raft_consensus.cc:359] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.857209 12687 raft_consensus.cc:385] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.857970 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:03.860044 12967 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:03.860520 12967 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:03.861001 12967 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.864619 12720 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:03.865410 12946 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:03.866151 12946 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.857499 12687 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.870487 12687 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.872076 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:03.872578 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:03.872927 12493 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:03.874691 12946 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:03.877589 12536 leader_election.cc:343] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.878568 12687 ts_tablet_manager.cc:1434] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.025s	user 0.003s	sys 0.005s
I20260709 12:11:03.879855 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:03.886790 12536 leader_election.cc:343] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:03.887398 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:03.888235 12971 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:03.888461 12568 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.890199 12720 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:03.892169 12687 tablet_bootstrap.cc:492] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:03.907636 12688 tablet_bootstrap.cc:492] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:03.907370 12966 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:11:03.906827 12394 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:03.919474 12686 tablet_bootstrap.cc:492] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:03.931114 12686 ts_tablet_manager.cc:1403] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.181s	user 0.015s	sys 0.002s
I20260709 12:11:03.934151 12686 raft_consensus.cc:359] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.934952 12686 raft_consensus.cc:385] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.935241 12686 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.935930 12686 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.920046 12688 ts_tablet_manager.cc:1403] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.083s	user 0.008s	sys 0.007s
I20260709 12:11:03.920583 12967 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.922097 12687 tablet_bootstrap.cc:654] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:03.940296 12967 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:03.942242 12686 ts_tablet_manager.cc:1434] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:11:03.942863 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:03.944962 12491 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.950176 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:03.950945 12569 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:03.951529 12686 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:03.943706 12688 raft_consensus.cc:359] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.962914 12688 raft_consensus.cc:385] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:03.963259 12688 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:03.963927 12688 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:03.970992 12834 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:03.971750 12688 ts_tablet_manager.cc:1434] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.028s	user 0.003s	sys 0.002s
I20260709 12:11:03.972986 12688 tablet_bootstrap.cc:492] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:03.983737 12569 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:03.994596 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:04.006472 12902 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:04.007073 12902 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:04.008370 12902 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.025699 12687 tablet_bootstrap.cc:492] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.026583 12687 ts_tablet_manager.cc:1403] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.135s	user 0.021s	sys 0.016s
I20260709 12:11:04.033926 12883 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:04.037497 12688 tablet_bootstrap.cc:654] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.065168 12686 tablet_bootstrap.cc:654] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.069203 12687 raft_consensus.cc:359] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.070302 12687 raft_consensus.cc:385] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.070658 12687 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.071534 12687 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.078367 12687 ts_tablet_manager.cc:1434] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.048s	user 0.005s	sys 0.003s
I20260709 12:11:04.079962 12687 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.082386 12966 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.082904 12966 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.086277 12491 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:04.087517 12966 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.088536 12688 tablet_bootstrap.cc:492] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.088989 12688 ts_tablet_manager.cc:1403] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.116s	user 0.005s	sys 0.007s
I20260709 12:11:04.090466 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.090274 12394 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:04.093254 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.093811 12688 raft_consensus.cc:359] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.094558 12688 raft_consensus.cc:385] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.094833 12688 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
W20260709 12:11:04.095474 12609 leader_election.cc:343] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.095491 12688 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.098526 12688 ts_tablet_manager.cc:1434] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
I20260709 12:11:04.099653 12688 tablet_bootstrap.cc:492] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
W20260709 12:11:04.096077 12610 leader_election.cc:343] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.104104 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.124356 12967 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.126063 12896 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.141774 12896 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.144191 12896 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.148031 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:04.151088 12536 leader_election.cc:343] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.162982 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:04.164314 12536 leader_election.cc:343] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.164737 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.165774 12971 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.166628 12688 tablet_bootstrap.cc:654] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.175752 12687 tablet_bootstrap.cc:654] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.178733 12686 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:04.191771 12686 ts_tablet_manager.cc:1403] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.241s	user 0.011s	sys 0.009s
I20260709 12:11:04.182053 12688 tablet_bootstrap.cc:492] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.192937 12688 ts_tablet_manager.cc:1403] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.094s	user 0.031s	sys 0.012s
I20260709 12:11:04.198557 12688 raft_consensus.cc:359] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.199275 12688 raft_consensus.cc:385] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.199515 12688 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.200206 12688 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.202629 12688 ts_tablet_manager.cc:1434] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.002s
I20260709 12:11:04.202852 12686 raft_consensus.cc:359] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.203450 12686 raft_consensus.cc:385] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.203733 12688 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.203771 12686 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.204879 12686 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.207080 12902 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.207790 12902 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.210911 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:04.214205 12610 leader_election.cc:343] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.215289 12902 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.207546 12686 ts_tablet_manager.cc:1434] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:11:04.217541 12686 tablet_bootstrap.cc:492] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:04.217808 12687 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.218354 12687 ts_tablet_manager.cc:1403] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.139s	user 0.020s	sys 0.001s
I20260709 12:11:04.221788 12896 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.222357 12896 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.222613 12687 raft_consensus.cc:359] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.223274 12687 raft_consensus.cc:385] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.223543 12687 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.224548 12687 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.226955 12687 ts_tablet_manager.cc:1434] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:11:04.226737 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.228242 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.228951 12490 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:04.230389 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:04.233209 12609 leader_election.cc:343] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.233762 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.234323 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:04.224749 12896 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.235690 12946 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:04.236407 12946 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.236963 12946 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.242076 12967 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:04.235608 12536 leader_election.cc:343] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.235538 12902 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.242720 12967 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.247167 12967 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.247761 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:04.249323 12610 leader_election.cc:343] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.228246 12687 tablet_bootstrap.cc:492] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.250715 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.212903 12688 tablet_bootstrap.cc:654] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:04.252805 12609 leader_election.cc:343] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.253327 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.254166 12967 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.275843 12686 tablet_bootstrap.cc:654] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.281284 12687 tablet_bootstrap.cc:654] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.296797 12688 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.297453 12688 ts_tablet_manager.cc:1403] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.094s	user 0.023s	sys 0.019s
W20260709 12:11:04.298197 12460 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:04.299522 12687 tablet_bootstrap.cc:492] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.300077 12687 ts_tablet_manager.cc:1403] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.072s	user 0.013s	sys 0.006s
I20260709 12:11:04.300526 12688 raft_consensus.cc:359] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.301358 12688 raft_consensus.cc:385] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.301702 12688 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.302642 12688 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.308859 12688 ts_tablet_manager.cc:1434] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:11:04.310088 12688 tablet_bootstrap.cc:492] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.312834 12687 raft_consensus.cc:359] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.313746 12687 raft_consensus.cc:385] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.314081 12687 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.314863 12687 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.317736 12967 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.318249 12967 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.273006 12946 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.320526 12967 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.321256 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:04.322989 12610 leader_election.cc:343] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.323637 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.324438 12569 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:04.325819 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.326964 12967 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:04.327410 12967 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.327795 12967 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.332723 12688 tablet_bootstrap.cc:654] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.332840 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:04.338670 12896 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.339298 12896 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.341419 12896 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.343070 12687 ts_tablet_manager.cc:1434] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.043s	user 0.009s	sys 0.000s
I20260709 12:11:04.344986 12687 tablet_bootstrap.cc:492] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.355981 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.356840 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:04.353900 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.354920 12967 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.359354 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:04.360474 12896 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:04.360865 12896 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.360898 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:04.361239 12896 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:04.362804 12610 leader_election.cc:343] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:04.366061 12536 leader_election.cc:343] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.367499 12967 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.368309 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:04.369002 12567 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.372035 12687 tablet_bootstrap.cc:654] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.373178 12896 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.378064 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:04.379652 12536 leader_election.cc:343] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.380991 12490 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.381824 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:04.382504 12646 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.398376 12946 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.400741 12896 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.402819 12687 tablet_bootstrap.cc:492] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.403947 12687 ts_tablet_manager.cc:1403] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.059s	user 0.015s	sys 0.004s
W20260709 12:11:04.404642 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:04.403882 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:04.408756 12687 raft_consensus.cc:359] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.414361 12687 raft_consensus.cc:385] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.414712 12687 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.416538 12686 tablet_bootstrap.cc:492] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:04.417188 12686 ts_tablet_manager.cc:1403] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.201s	user 0.047s	sys 0.012s
I20260709 12:11:04.420545 12686 raft_consensus.cc:359] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.421311 12686 raft_consensus.cc:385] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.421582 12686 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.422006 12687 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.422672 12686 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.425513 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:04.426019 12687 ts_tablet_manager.cc:1434] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 12:11:04.427007 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.427641 12687 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.428210 12967 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:04.428735 12967 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:04.429586 12967 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.439894 12686 ts_tablet_manager.cc:1434] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 12:11:04.451272 12999 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:04.453781 12687 tablet_bootstrap.cc:654] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.460582 12896 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.461231 12896 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.463806 12896 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:04.465790 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76359.2us
I20260709 12:11:04.472770 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:04.474961 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.472084 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.482460 12971 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:04.482959 12971 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:04.480357 12686 tablet_bootstrap.cc:492] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:04.483837 12971 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:04.490607 12536 leader_election.cc:343] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.473389 12999 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.523356 12999 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.524379 12395 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:04.473414 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.475822 12490 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:04.519550 12687 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.525604 12999 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.529403 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.526880 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.531337 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:04.531466 12688 tablet_bootstrap.cc:492] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.532167 12688 ts_tablet_manager.cc:1403] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.222s	user 0.037s	sys 0.064s
I20260709 12:11:04.534888 12688 raft_consensus.cc:359] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.535573 12688 raft_consensus.cc:385] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.535950 12688 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.535512 12391 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:11:04.536969 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46404.2us
I20260709 12:11:04.536705 12688 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:04.539808 12461 leader_election.cc:343] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.540952 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:04.542745 12883 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:04.543105 12883 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.543466 12883 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.545372 12688 ts_tablet_manager.cc:1434] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 12:11:04.545568 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:04.546684 12688 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.549724 12896 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:04.550314 12896 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:04.551469 12896 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.527225 12687 ts_tablet_manager.cc:1403] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.100s	user 0.019s	sys 0.000s
I20260709 12:11:04.548848 12686 tablet_bootstrap.cc:654] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.556309 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.558844 12883 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.561966 13012 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.566874 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:04.568282 12687 raft_consensus.cc:359] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.568994 12687 raft_consensus.cc:385] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.569252 12687 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.570366 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:04.571003 12569 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.569929 12687 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.574399 12687 ts_tablet_manager.cc:1434] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
I20260709 12:11:04.563915 12883 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:04.575536 12461 leader_election.cc:343] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.581056 12902 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:04.583786 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:04.575553 12687 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.585055 12688 tablet_bootstrap.cc:654] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.592299 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:04.596746 12391 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:04.598433 12999 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:04.599192 12999 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:04.599293 12907 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.600014 12907 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.600540 12999 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.599637 12687 tablet_bootstrap.cc:654] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.624274 13003 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:04.603832 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.630291 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:04.603852 12907 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.620558 12391 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:04.631762 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:04.633015 12999 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:04.618319 12902 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.633750 12902 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.636478 12902 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.628870 12688 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.637260 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.637599 12688 ts_tablet_manager.cc:1403] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.091s	user 0.014s	sys 0.019s
I20260709 12:11:04.638134 12496 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:04.606734 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.612587 12686 tablet_bootstrap.cc:492] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:04.638715 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:04.638860 12686 ts_tablet_manager.cc:1403] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.159s	user 0.014s	sys 0.005s
I20260709 12:11:04.640710 12999 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.642572 12686 raft_consensus.cc:359] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.638361 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.643321 12686 raft_consensus.cc:385] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.648186 12686 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.649185 12686 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.654170 12686 ts_tablet_manager.cc:1434] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.003s
I20260709 12:11:04.641302 12999 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:04.655722 12686 tablet_bootstrap.cc:492] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:04.665771 12687 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.666456 12687 ts_tablet_manager.cc:1403] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.091s	user 0.020s	sys 0.000s
I20260709 12:11:04.647028 12688 raft_consensus.cc:359] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.647574 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.670850 12966 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.674547 12688 raft_consensus.cc:385] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.678251 12688 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.679292 12688 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.676548 12686 tablet_bootstrap.cc:654] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.685034 12966 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.689527 12688 ts_tablet_manager.cc:1434] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.052s	user 0.004s	sys 0.003s
I20260709 12:11:04.690810 12688 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.693681 12687 raft_consensus.cc:359] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.694500 12687 raft_consensus.cc:385] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.694787 12687 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.695642 12687 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.696297 12999 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.699015 12999 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.700388 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:04.701174 12642 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:04.706373 12687 ts_tablet_manager.cc:1434] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.040s	user 0.008s	sys 0.000s
I20260709 12:11:04.707533 12687 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.730684 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:04.731405 12569 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:04.712611 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:04.720881 12907 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.722707 12686 tablet_bootstrap.cc:492] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:04.733462 12907 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.734124 12686 ts_tablet_manager.cc:1403] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.079s	user 0.005s	sys 0.012s
I20260709 12:11:04.724090 12688 tablet_bootstrap.cc:654] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.736291 12907 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.738754 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.739678 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:04.741250 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:04.743387 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:04.744280 12999 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:04.744791 12999 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:04.745800 12999 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.747061 12686 raft_consensus.cc:359] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.747828 12686 raft_consensus.cc:385] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.748073 12686 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.749441 12883 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:04.749428 12686 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.750042 12883 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.750454 12883 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.751677 12569 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:04.754894 12686 ts_tablet_manager.cc:1434] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.001s
I20260709 12:11:04.755985 12686 tablet_bootstrap.cc:492] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:04.765765 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:04.766669 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:04.767015 12686 tablet_bootstrap.cc:654] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.768564 12687 tablet_bootstrap.cc:654] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.773663 12688 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.774425 12688 ts_tablet_manager.cc:1403] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.084s	user 0.013s	sys 0.006s
I20260709 12:11:04.779450 12883 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.785987 12688 raft_consensus.cc:359] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.786757 12688 raft_consensus.cc:385] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.787016 12688 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.788499 12395 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 2, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:04.797192 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:04.798619 12569 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.801859 12688 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.803105 12687 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.803833 12687 ts_tablet_manager.cc:1403] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.096s	user 0.007s	sys 0.004s
I20260709 12:11:04.804618 12567 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.810269 12883 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.811458 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:04.812127 12643 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.814996 12688 ts_tablet_manager.cc:1434] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.040s	user 0.009s	sys 0.000s
I20260709 12:11:04.816188 12688 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.816401 12966 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:04.819474 12687 raft_consensus.cc:359] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.820190 12687 raft_consensus.cc:385] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.820451 12687 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.833942 12686 tablet_bootstrap.cc:492] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:04.834611 12686 ts_tablet_manager.cc:1403] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.079s	user 0.009s	sys 0.003s
I20260709 12:11:04.821076 12687 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.839727 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:04.841506 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:04.842511 12999 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:04.843020 12999 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:04.843948 12999 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.847541 12687 ts_tablet_manager.cc:1434] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 12:11:04.849275 12687 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.850602 12688 tablet_bootstrap.cc:654] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.862011 12686 raft_consensus.cc:359] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.864226 12496 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.865592 12896 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.872584 12896 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.878003 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.878315 12896 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.861467 12687 tablet_bootstrap.cc:654] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.889010 12688 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.889485 12688 ts_tablet_manager.cc:1403] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.074s	user 0.012s	sys 0.000s
I20260709 12:11:04.892445 12688 raft_consensus.cc:359] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.893157 12688 raft_consensus.cc:385] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.893424 12688 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.862977 12686 raft_consensus.cc:385] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.897161 12686 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.898919 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:04.896312 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.909950 12643 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:04.902144 12686 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.911532 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:04.911875 12493 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:04.912411 12896 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:04.912894 12896 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:04.913313 12896 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:04.900525 12688 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.907233 12687 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:04.915493 12687 ts_tablet_manager.cc:1403] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.000s
I20260709 12:11:04.916067 12966 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:11:04.918774 12688 ts_tablet_manager.cc:1434] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:11:04.920040 12688 tablet_bootstrap.cc:492] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.926856 12687 raft_consensus.cc:359] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.927587 12687 raft_consensus.cc:385] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.932695 12687 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.933300 12687 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.935283 12686 ts_tablet_manager.cc:1434] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.100s	user 0.007s	sys 0.002s
I20260709 12:11:04.936335 12686 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:04.938849 12971 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:04.939361 12971 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.942878 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:04.944167 12536 leader_election.cc:343] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.941458 12971 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:04.940281 12394 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:04.945178 12688 tablet_bootstrap.cc:654] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:04.966207 12967 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:04.952466 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:04.971203 12967 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.976116 12688 tablet_bootstrap.cc:492] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:04.976740 12688 ts_tablet_manager.cc:1403] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.057s	user 0.011s	sys 0.005s
I20260709 12:11:04.977365 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:04.977732 12967 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:04.978839 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:04.979424 12610 leader_election.cc:343] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.947990 12687 ts_tablet_manager.cc:1434] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.032s	user 0.004s	sys 0.001s
W20260709 12:11:04.980914 12609 leader_election.cc:343] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.969928 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:04.981526 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.981228 12687 tablet_bootstrap.cc:492] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:04.983255 13003 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:11:04.985052 12536 leader_election.cc:343] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:04.985492 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:04.986474 12971 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:04.990988 12688 raft_consensus.cc:359] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:04.992264 12688 raft_consensus.cc:385] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:04.992650 12688 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:04.994063 12688 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.002964 12688 ts_tablet_manager.cc:1434] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.001s
I20260709 12:11:05.007897 12688 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:04.957867 12686 tablet_bootstrap.cc:654] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:05.011922 12907 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:05.017434 12687 tablet_bootstrap.cc:654] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:05.018086 12688 tablet_bootstrap.cc:654] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:05.016767 12896 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.023588 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:05.024317 12896 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.024456 12496 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.031222 12687 tablet_bootstrap.cc:492] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:05.031421 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:05.035152 12643 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.037551 12896 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:05.039014 12496 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:05.053112 13012 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.053929 13012 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.059288 12643 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:05.079054 12686 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:05.079675 12686 ts_tablet_manager.cc:1403] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.144s	user 0.016s	sys 0.008s
I20260709 12:11:05.081715 12687 ts_tablet_manager.cc:1403] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.101s	user 0.018s	sys 0.005s
I20260709 12:11:05.082101 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:05.081358 12907 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.083717 12907 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.084285 12966 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.084823 12966 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.086982 12688 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:05.089732 12971 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:05.090433 12971 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:05.091434 12971 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.098349 12688 ts_tablet_manager.cc:1403] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.091s	user 0.016s	sys 0.009s
I20260709 12:11:05.099584 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:05.100201 12907 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.101578 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:05.110666 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:05.112602 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:05.113276 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:05.102609 13012 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.103192 12966 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:05.103314 12686 raft_consensus.cc:359] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.103003 12687 raft_consensus.cc:359] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.114140 12688 raft_consensus.cc:359] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.114770 12687 raft_consensus.cc:385] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.114770 12686 raft_consensus.cc:385] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.115000 12688 raft_consensus.cc:385] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.115227 12687 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.115485 12688 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.116020 12687 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.115468 12686 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.117108 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 0.
I20260709 12:11:05.117542 12686 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.118582 12496 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:05.119134 12687 ts_tablet_manager.cc:1434] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.004s
I20260709 12:11:05.120151 12687 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:05.120384 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:05.121424 12966 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:05.121821 12966 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:05.116514 12688 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.127511 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:05.128168 12686 ts_tablet_manager.cc:1434] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.048s	user 0.009s	sys 0.002s
I20260709 12:11:05.131455 12688 ts_tablet_manager.cc:1434] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:11:05.134259 12642 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.131557 12907 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:05.122200 12966 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.145838 12907 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:05.146370 12907 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.131295 12902 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.149287 12686 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:05.158990 12688 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:05.179787 12687 tablet_bootstrap.cc:654] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:05.170441 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:05.159296 13041 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.154018 12902 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.197270 12567 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:05.195605 13041 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:05.241500 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47617.2us
I20260709 12:11:05.223182 13043 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.252497 13047 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:05.246711 13044 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.286244 13044 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.256317 13043 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.288910 13045 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.061s
I20260709 12:11:05.346630 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:05.349229 12687 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:05.349785 12687 ts_tablet_manager.cc:1403] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.230s	user 0.013s	sys 0.024s
I20260709 12:11:05.352598 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:05.346060 12902 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:05.356835 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80385.4us
I20260709 12:11:05.369158 13043 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:05.369984 13044 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:05.378301 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110504us
I20260709 12:11:05.385751 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:05.395843 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:05.371670 13041 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.397500 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:05.399085 12907 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.400559 12688 tablet_bootstrap.cc:654] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:05.396534 12687 raft_consensus.cc:359] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:05.397068 12609 leader_election.cc:343] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.408687 12687 raft_consensus.cc:385] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.409128 12687 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.412798 12687 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.421270 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:05.425871 12687 ts_tablet_manager.cc:1434] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.076s	user 0.007s	sys 0.002s
I20260709 12:11:05.427158 12687 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
W20260709 12:11:05.460383 13048 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:11:05.461727 12686 tablet_bootstrap.cc:654] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:05.467777 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:05.463636 12609 leader_election.cc:343] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.522104 12643 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:11:05.563848 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148260us
I20260709 12:11:05.554569 12687 tablet_bootstrap.cc:654] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:05.600874 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 174230us
I20260709 12:11:05.553500 12567 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:11:05.591881 13058 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.522845 12966 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.640339 13058 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.654631 12907 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.662880 12644 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:05.672437 12686 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:05.673265 12686 ts_tablet_manager.cc:1403] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.535s	user 0.019s	sys 0.031s
I20260709 12:11:05.680044 12688 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:05.679978 13058 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.680745 12688 ts_tablet_manager.cc:1403] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.522s	user 0.016s	sys 0.004s
W20260709 12:11:05.687986 13062 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:11:05.691606 12688 raft_consensus.cc:359] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.692847 12688 raft_consensus.cc:385] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.693368 12688 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.693185 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:05.694964 12688 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:05.695701 12610 leader_election.cc:343] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.698995 12610 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:05.694407 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:05.710107 12568 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.707154 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:05.705848 12966 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:05.725704 12610 leader_election.cc:343] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.726257 12610 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:05.729995 12686 raft_consensus.cc:359] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.735340 12687 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:05.739372 12686 raft_consensus.cc:385] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.740041 12687 ts_tablet_manager.cc:1403] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.313s	user 0.022s	sys 0.010s
I20260709 12:11:05.742113 13054 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:11:05.739912 12686 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.750159 12686 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:05.756219 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:05.757745 12688 ts_tablet_manager.cc:1434] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.077s	user 0.012s	sys 0.007s
I20260709 12:11:05.758169 13074 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:05.738754 12907 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:11:05.768015 12688 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:05.757356 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:05.748153 13003 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:05.759645 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:05.770404 12571 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.735862 12568 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:05.780773 13047 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:05.781487 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:05.782593 12641 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.783174 13047 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:05.803934 12686 ts_tablet_manager.cc:1434] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.128s	user 0.007s	sys 0.015s
I20260709 12:11:05.814232 12686 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:05.792579 12491 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:05.826150 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 408755us
W20260709 12:11:05.841598 12536 leader_election.cc:343] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.852150 12687 raft_consensus.cc:359] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.851655 13041 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:11:05.855608 12687 raft_consensus.cc:385] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:05.866381 12687 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:05.849354 13054 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.869410 13054 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.870738 12496 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:05.886804 13054 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:05.886772 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:05.897228 12536 leader_election.cc:343] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.898090 12536 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:05.898303 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:05.902307 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:05.909793 13012 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:05.921814 12571 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
W20260709 12:11:05.923337 12536 leader_election.cc:343] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:05.926028 12536 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:05.928900 13068 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:11:05.933806 12391 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:05.935619 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:05.917810 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:05.938503 13003 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:05.939234 13003 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.942068 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:05.943204 13070 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:05.942633 13068 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:05.957824 12687 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:05.952102 12967 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:05.963970 13068 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:05.985275 12687 ts_tablet_manager.cc:1434] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.245s	user 0.026s	sys 0.024s
I20260709 12:11:05.977430 13068 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:06.028156 12687 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:06.054067 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:06.054876 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:06.052687 13003 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:06.090360 13078 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:11:06.108330 12688 tablet_bootstrap.cc:654] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:06.110693 12609 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:06.111513 13003 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:06.112006 13003 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:06.112054 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:06.112958 13003 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.113161 12967 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:06.115355 12967 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:06.115873 12967 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:06.118992 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:06.127310 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:06.128829 12491 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:06.130965 12686 tablet_bootstrap.cc:654] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:06.166376 12967 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.177459 12688 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:06.178242 12688 ts_tablet_manager.cc:1403] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.410s	user 0.033s	sys 0.049s
I20260709 12:11:06.185216 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:06.190241 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:06.215492 12967 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:06.186527 13012 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:06.219384 13012 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.198074 13077 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:06.343604 13068 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.316754 13077 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:11:06.388411 13087 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:11:06.442328 13077 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.447790 13012 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:06.464030 12687 tablet_bootstrap.cc:654] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:06.465757 12688 raft_consensus.cc:359] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.475790 12688 raft_consensus.cc:385] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:06.476056 12688 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:06.476718 12688 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.467132 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:06.482146 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 267443us
I20260709 12:11:06.485513 12975 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:06.490933 12686 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:06.492036 12686 ts_tablet_manager.cc:1403] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.678s	user 0.028s	sys 0.060s
I20260709 12:11:06.554224 12688 ts_tablet_manager.cc:1434] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.376s	user 0.022s	sys 0.016s
I20260709 12:11:06.554708 13068 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:06.555365 12688 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:06.549333 13094 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:06.557696 13094 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:06.567203 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70809.7us
I20260709 12:11:06.582511 13094 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:06.582773 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91303.6us
I20260709 12:11:06.563702 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:06.629527 12566 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:06.746129 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:06.751639 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122024us
I20260709 12:11:06.752542 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:06.753279 12687 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:06.754102 12687 ts_tablet_manager.cc:1403] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.751s	user 0.046s	sys 0.041s
I20260709 12:11:06.759847 12686 raft_consensus.cc:359] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.761562 12686 raft_consensus.cc:385] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:06.761920 12686 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:06.762648 12686 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.786286 12686 ts_tablet_manager.cc:1434] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.294s	user 0.008s	sys 0.024s
I20260709 12:11:06.794237 12686 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:06.791169 12687 raft_consensus.cc:359] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.797691 12687 raft_consensus.cc:385] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:06.798162 12687 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:06.808199 12687 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.824121 12688 tablet_bootstrap.cc:654] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:06.836129 12686 tablet_bootstrap.cc:654] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:06.825294 12687 ts_tablet_manager.cc:1434] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.001s
I20260709 12:11:06.846650 12687 tablet_bootstrap.cc:492] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:06.825857 12967 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:06.848558 12967 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.836309 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:06.850561 12490 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:06.851001 12967 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:06.831441 12566 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
W20260709 12:11:06.855881 12610 leader_election.cc:343] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:06.869436 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:06.869901 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:06.889196 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:06.890055 12640 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:06.893146 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:06.895175 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:06.907196 13068 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:06.869520 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:06.908934 12536 leader_election.cc:343] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:06.911069 12567 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:06.917289 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:06.945233 12609 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:06.938030 12967 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:06.953119 12967 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.955175 13095 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:06.955885 13095 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:06.956341 13095 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:06.956720 12967 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:06.969923 12391 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 88) took 1113 ms. Trace:
I20260709 12:11:06.952718 13063 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:11:07.025425 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:07.038923 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78660us
W20260709 12:11:07.047257 12610 leader_election.cc:343] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:07.048714 12610 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.061456 12687 tablet_bootstrap.cc:654] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:07.056186 13063 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:07.087364 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.089242 13095 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:07.090672 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:07.091790 13095 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:07.094988 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.067607 13044 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:07.105497 12686 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:07.114393 12686 ts_tablet_manager.cc:1403] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.320s	user 0.014s	sys 0.016s
I20260709 12:11:07.104568 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:07.073376 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.123171 13068 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:07.123760 13068 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:07.102277 12394 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:07.124600 13068 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.112460 13056 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:07.146198 12490 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:07.116103 13063 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:07.174060 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:07.162111 13056 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:07.195544 13061 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:07.195777 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:07.197428 13056 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:07.227629 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62129.4us
I20260709 12:11:07.179764 13063 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.230715 12688 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:07.231503 12688 ts_tablet_manager.cc:1403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.676s	user 0.033s	sys 0.050s
I20260709 12:11:07.196265 13061 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:06.970687 12391 rpcz_store.cc:276] 0709 12:11:05.856154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:05.860351 (+  4197us) service_pool.cc:224] Handling call
0709 12:11:06.969770 (+1109419us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:06.072646 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:06.099012 (+ 26366us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:06.099045 (+    33us) write_op.cc:435] Acquired schema lock
0709 12:11:06.099063 (+    18us) tablet.cc:662] Decoding operations
0709 12:11:06.099383 (+   320us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:06.099435 (+    52us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:06.099459 (+    24us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:06.119891 (+ 20432us) tablet.cc:701] Row locks acquired
0709 12:11:06.119906 (+    15us) write_op.cc:260] PREPARE: finished
0709 12:11:06.120045 (+   139us) write_op.cc:270] Start()
0709 12:11:06.120151 (+   106us) write_op.cc:276] Timestamp: P: 1783599066120007 usec, L: 0
0709 12:11:06.120170 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 12:11:06.120856 (+   686us) log.cc:844] Serialized 540 byte log entry
0709 12:11:06.748842 (+627986us) op_driver.cc:464] REPLICATION: finished
0709 12:11:06.810207 (+ 61365us) write_op.cc:301] APPLY: starting
0709 12:11:06.810308 (+   101us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:06.810424 (+   116us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:06.810615 (+   191us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:06.810913 (+   298us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:06.811123 (+   210us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:06.811169 (+    46us) write_op.cc:312] APPLY: finished
0709 12:11:06.814479 (+  3310us) log.cc:844] Serialized 22 byte log entry
0709 12:11:06.867790 (+ 53311us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:06.867913 (+   123us) write_op.cc:454] Released schema lock
0709 12:11:06.883158 (+ 15245us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:06.939708 (+ 56550us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:06.939758 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11140,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":60958,"apply.run_cpu_time_us":2300,"apply.run_wall_time_us":115011,"num_ops":1,"prepare.queue_time_us":93028,"prepare.run_cpu_time_us":34287,"prepare.run_wall_time_us":188702,"raft.queue_time_us":151406,"raft.run_cpu_time_us":23453,"raft.run_wall_time_us":48135,"replication_time_us":628535,"spinlock_wait_cycles":974720,"thread_start_us":320498,"threads_started":4,"wal-append.queue_time_us":432574}]]}
I20260709 12:11:07.174188 13053 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.114s
I20260709 12:11:07.301251 12688 raft_consensus.cc:359] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.302377 12688 raft_consensus.cc:385] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:07.302718 12688 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:07.310330 12688 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.330530 13061 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:07.327705 13047 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:11:07.346042 12688 ts_tablet_manager.cc:1434] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.114s	user 0.009s	sys 0.000s
I20260709 12:11:07.380198 13041 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:07.389008 13058 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:07.389123 13041 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.392483 13041 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:07.398267 12687 tablet_bootstrap.cc:492] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:07.398952 12687 ts_tablet_manager.cc:1403] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.553s	user 0.025s	sys 0.027s
I20260709 12:11:07.402460 12688 tablet_bootstrap.cc:492] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:07.389686 13058 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.388355 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:07.446478 12687 raft_consensus.cc:359] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.447480 12687 raft_consensus.cc:385] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:07.447779 12687 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:07.448552 12687 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.450296 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:07.451951 13071 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:07.452529 13071 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.454668 13071 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:07.456235 12687 ts_tablet_manager.cc:1434] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.057s	user 0.005s	sys 0.005s
I20260709 12:11:07.459836 12687 tablet_bootstrap.cc:492] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:07.427937 13058 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:07.466717 13052 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:07.470515 12688 tablet_bootstrap.cc:654] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:07.464860 13047 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:11:07.483969 12686 raft_consensus.cc:359] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.485563 12686 raft_consensus.cc:385] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:07.485982 12686 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:07.486791 12686 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.486975 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:07.505350 12686 ts_tablet_manager.cc:1434] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.391s	user 0.012s	sys 0.001s
I20260709 12:11:07.467305 13052 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.512591 12686 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:07.504902 13077 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:07.534065 12687 tablet_bootstrap.cc:654] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:07.527937 13056 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.540503 13050 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:07.541108 13050 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.558580 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.559593 13095 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:07.566653 13050 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:07.569746 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:07.570097 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.570502 12572 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:07.572257 12610 leader_election.cc:343] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:07.576668 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.577548 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:07.583870 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:07.584537 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:07.584628 12646 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
W20260709 12:11:07.597777 12536 leader_election.cc:343] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:07.579610 13050 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:07.581171 13052 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:07.637085 12493 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:07.649873 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.650184 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:07.624955 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.655525 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:07.664523 12494 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:07.662525 12572 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:07.678620 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:07.702576 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.683736 12564 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:07.705127 13058 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:07.706564 13058 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.712109 13058 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:07.671301 13056 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:07.726754 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:07.729902 12646 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:07.728940 13124 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
W20260709 12:11:07.730968 12536 leader_election.cc:343] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:07.733500 12610 leader_election.cc:343] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:07.652221 13050 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:07.755283 12609 leader_election.cc:343] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:07.755805 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.757009 12609 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:07.758971 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.754191 12494 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:07.771576 13073 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:07.765944 13074 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:07.785207 13073 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:07.785806 13073 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:07.758328 12967 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:07.768023 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.795475 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.797538 13068 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:07.776849 13122 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260709 12:11:07.809654 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50744.4us
I20260709 12:11:07.814239 13056 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:07.816160 13056 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:07.816531 13056 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:07.820142 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:07.825986 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:07.825960 13050 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:07.826041 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.833599 12686 tablet_bootstrap.cc:654] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:07.836249 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.831548 13058 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:07.835117 13074 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:07.843524 13071 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:07.917413 12687 tablet_bootstrap.cc:492] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:07.917933 12687 ts_tablet_manager.cc:1403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.458s	user 0.016s	sys 0.029s
I20260709 12:11:07.913959 13071 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:07.925019 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:07.933169 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:07.951464 12687 raft_consensus.cc:359] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.952225 12687 raft_consensus.cc:385] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:07.952487 12687 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:07.953219 12687 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.937534 13003 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:07.943929 13074 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.981547 12687 ts_tablet_manager.cc:1434] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.063s	user 0.007s	sys 0.000s
I20260709 12:11:07.953682 13073 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:07.995155 12687 tablet_bootstrap.cc:492] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:07.987867 13071 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.015929 13056 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.018393 13056 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:08.034515 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:08.035434 12645 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:08.037284 12688 tablet_bootstrap.cc:492] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:08.037918 13068 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:08.033169 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:08.040903 12688 ts_tablet_manager.cc:1403] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.653s	user 0.042s	sys 0.056s
W20260709 12:11:08.050405 12536 leader_election.cc:343] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.065320 12688 raft_consensus.cc:359] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.066452 12688 raft_consensus.cc:385] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.066747 12688 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.067404 12688 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.069764 12688 ts_tablet_manager.cc:1434] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:11:08.073895 12688 tablet_bootstrap.cc:492] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:08.080868 12686 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:08.081384 12686 ts_tablet_manager.cc:1403] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.569s	user 0.011s	sys 0.028s
I20260709 12:11:08.091554 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:08.096969 13073 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:08.101444 12395 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:11:08.117455 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62748.8us
I20260709 12:11:08.129734 12394 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 90) took 1270 ms. Trace:
W20260709 12:11:08.139113 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:08.123150 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:08.122850 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:08.138558 12536 leader_election.cc:343] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.141765 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:08.142827 13138 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.144791 12686 raft_consensus.cc:359] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.145709 12686 raft_consensus.cc:385] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.146134 12686 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.148764 12686 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.139945 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:08.143491 13063 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:08.152894 12571 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:08.156787 12688 tablet_bootstrap.cc:654] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:08.157401 12610 leader_election.cc:343] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:08.159340 12536 leader_election.cc:343] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:11:08.162474 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:08.162423 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:08.130115 13056 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:11:08.170463 12686 ts_tablet_manager.cc:1434] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.089s	user 0.010s	sys 0.000s
I20260709 12:11:08.175457 12687 tablet_bootstrap.cc:654] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.162796 13063 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.182061 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:08.171042 13071 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.189994 13094 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.195310 13063 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:08.199916 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:08.204702 12610 leader_election.cc:343] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.207381 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:08.215750 12686 tablet_bootstrap.cc:492] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
W20260709 12:11:08.217494 12609 leader_election.cc:343] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.218348 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:08.223768 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116045us
I20260709 12:11:08.130312 12394 rpcz_store.cc:276] 0709 12:11:06.859251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:06.859439 (+   188us) service_pool.cc:224] Handling call
0709 12:11:08.129657 (+1270218us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:07.218522 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:07.219467 (+   945us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:07.219514 (+    47us) write_op.cc:435] Acquired schema lock
0709 12:11:07.219554 (+    40us) tablet.cc:662] Decoding operations
0709 12:11:07.220049 (+   495us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:07.220153 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:07.220202 (+    49us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:07.220499 (+   297us) tablet.cc:701] Row locks acquired
0709 12:11:07.220526 (+    27us) write_op.cc:260] PREPARE: finished
0709 12:11:07.220738 (+   212us) write_op.cc:270] Start()
0709 12:11:07.220895 (+   157us) write_op.cc:276] Timestamp: P: 1783599067220680 usec, L: 0
0709 12:11:07.220925 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 12:11:07.242364 (+ 21439us) log.cc:844] Serialized 540 byte log entry
0709 12:11:07.306316 (+ 63952us) op_driver.cc:464] REPLICATION: finished
0709 12:11:07.331222 (+ 24906us) write_op.cc:301] APPLY: starting
0709 12:11:07.331301 (+    79us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:07.331401 (+   100us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:07.331516 (+   115us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:07.334940 (+  3424us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:07.335083 (+   143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:07.335125 (+    42us) write_op.cc:312] APPLY: finished
0709 12:11:07.335322 (+   197us) log.cc:844] Serialized 22 byte log entry
0709 12:11:07.335473 (+   151us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:07.335534 (+    61us) write_op.cc:454] Released schema lock
0709 12:11:07.335561 (+    27us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:07.335876 (+   315us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:07.335906 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":31949,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":1638,"apply.run_cpu_time_us":1483,"apply.run_wall_time_us":4636,"num_ops":1,"prepare.queue_time_us":87738,"prepare.run_cpu_time_us":4025,"prepare.run_wall_time_us":24246,"replication_time_us":79548,"thread_start_us":71698,"threads_started":1}]]}
I20260709 12:11:08.228652 13053 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.271431 12999 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:08.272034 12999 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.274544 12686 tablet_bootstrap.cc:654] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.275736 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:08.276010 12999 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:08.276592 12646 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:08.283154 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:08.285090 12460 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:08.286643 13053 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:08.290385 13123 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:08.292150 12687 tablet_bootstrap.cc:492] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:08.292620 12687 ts_tablet_manager.cc:1403] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.310s	user 0.008s	sys 0.007s
I20260709 12:11:08.307272 13138 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:08.309255 12688 tablet_bootstrap.cc:492] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:08.309876 12688 ts_tablet_manager.cc:1403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.236s	user 0.017s	sys 0.000s
I20260709 12:11:08.310042 12687 raft_consensus.cc:359] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.310719 12687 raft_consensus.cc:385] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.310978 12687 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.307901 13138 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.309130 13123 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.321960 12687 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.330307 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:08.332836 12687 ts_tablet_manager.cc:1434] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.002s
I20260709 12:11:08.334059 12687 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:08.326116 12688 raft_consensus.cc:359] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.329377 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:08.337692 12688 raft_consensus.cc:385] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:11:08.339586 12536 leader_election.cc:343] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.329876 13138 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:08.338033 12688 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.342314 12688 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.343327 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:08.343998 12688 ts_tablet_manager.cc:1434] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.000s
I20260709 12:11:08.345067 12688 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:08.347617 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:08.351500 13138 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:08.352144 12687 tablet_bootstrap.cc:654] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.352051 13138 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:08.352828 13138 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:08.374135 12688 tablet_bootstrap.cc:654] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.380927 12686 tablet_bootstrap.cc:492] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:08.381538 12686 ts_tablet_manager.cc:1403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.166s	user 0.012s	sys 0.010s
I20260709 12:11:08.384486 12686 raft_consensus.cc:359] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.385167 12686 raft_consensus.cc:385] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.388612 13138 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.393507 13138 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:08.385421 12686 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.394572 12686 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.399238 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:08.399675 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:08.400023 12495 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:08.400331 12637 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:08.401067 12686 ts_tablet_manager.cc:1434] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:11:08.408270 12686 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:08.408119 12391 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:08.421945 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:08.425493 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:08.427004 13056 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:08.427672 13056 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:08.428567 13056 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.447054 13132 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:08.447633 13132 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.477233 12394 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:08.485842 12686 tablet_bootstrap.cc:654] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.464218 12495 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:08.490741 13132 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:08.490010 12394 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:08.492609 12688 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:08.493444 12688 ts_tablet_manager.cc:1403] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.149s	user 0.022s	sys 0.022s
I20260709 12:11:08.494439 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:08.499430 12688 raft_consensus.cc:359] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.500243 12688 raft_consensus.cc:385] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.500530 12688 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.501161 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:08.501264 12688 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.502077 12637 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:08.489151 13071 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 12:11:08.510416 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:08.512154 12688 ts_tablet_manager.cc:1434] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
W20260709 12:11:08.512518 12536 leader_election.cc:343] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.508276 13075 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:08.517921 13113 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:08.518375 13113 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:08.518714 13113 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:08.519960 12688 tablet_bootstrap.cc:492] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:08.523272 12687 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:08.523878 12687 ts_tablet_manager.cc:1403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.190s	user 0.018s	sys 0.011s
I20260709 12:11:08.549875 13050 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:08.562065 13132 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:08.594331 12687 raft_consensus.cc:359] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.595254 12687 raft_consensus.cc:385] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.595595 12687 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.596597 12687 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.604486 12686 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:08.579533 12391 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:08.614329 12686 ts_tablet_manager.cc:1403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.206s	user 0.013s	sys 0.005s
W20260709 12:11:08.615505 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51667.9us
I20260709 12:11:08.619866 12391 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:08.635367 12687 ts_tablet_manager.cc:1434] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.111s	user 0.005s	sys 0.004s
I20260709 12:11:08.638012 12686 raft_consensus.cc:359] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.639446 12686 raft_consensus.cc:385] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.639858 12686 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.640962 12687 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:08.641505 12686 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.644341 12686 ts_tablet_manager.cc:1434] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.000s
I20260709 12:11:08.645548 12686 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:08.664350 13113 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.666715 12688 tablet_bootstrap.cc:654] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:08.683542 13153 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.004s	sys 0.000s
I20260709 12:11:08.767407 12686 tablet_bootstrap.cc:654] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.798485 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:08.810214 12687 tablet_bootstrap.cc:654] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.810494 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:08.811143 12646 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:08.795101 13113 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:08.820806 12536 leader_election.cc:343] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.835871 13054 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:08.836454 13054 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.838587 13054 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:08.840966 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:08.843617 12460 leader_election.cc:343] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.848480 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:08.851055 12461 leader_election.cc:343] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.851528 12461 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:08.852823 13054 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.861361 12687 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:08.861944 12687 ts_tablet_manager.cc:1403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.226s	user 0.005s	sys 0.017s
I20260709 12:11:08.884192 12646 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:08.885856 12688 tablet_bootstrap.cc:492] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:08.887061 12688 ts_tablet_manager.cc:1403] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.367s	user 0.020s	sys 0.001s
I20260709 12:11:08.865211 12687 raft_consensus.cc:359] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.889036 12687 raft_consensus.cc:385] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.889354 12687 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.900336 12687 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.898545 12688 raft_consensus.cc:359] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.901746 12688 raft_consensus.cc:385] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.902070 12688 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.903301 13073 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:08.903825 13073 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.908420 13073 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:08.911984 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:08.914453 12610 leader_election.cc:343] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.916188 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:08.918365 13101 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:08.923316 12688 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.929553 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:08.935328 12687 ts_tablet_manager.cc:1434] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.073s	user 0.007s	sys 0.000s
I20260709 12:11:08.935523 12686 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:08.934602 13101 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:08.936038 12686 ts_tablet_manager.cc:1403] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.291s	user 0.017s	sys 0.003s
I20260709 12:11:08.936702 12687 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:08.936462 13101 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.944710 12686 raft_consensus.cc:359] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.947803 12686 raft_consensus.cc:385] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:08.948207 12686 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:08.949605 12686 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:08.954119 12686 ts_tablet_manager.cc:1434] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.000s
I20260709 12:11:08.954913 12688 ts_tablet_manager.cc:1434] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.001s
I20260709 12:11:08.955425 12686 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:08.956099 12688 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
W20260709 12:11:08.960610 12609 leader_election.cc:343] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:08.963073 12609 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:08.968667 13044 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:08.973902 13113 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:11:08.981616 13113 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:08.984619 12686 tablet_bootstrap.cc:654] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:08.994581 12687 tablet_bootstrap.cc:654] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.004119 13113 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.004700 13113 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.010747 13113 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.016486 12395 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:09.019600 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:09.027137 12536 leader_election.cc:343] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.013927 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.034626 12644 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.033011 12688 tablet_bootstrap.cc:654] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.038265 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:09.040343 13071 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:09.040776 13071 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.041114 13071 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.061072 12495 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.087708 13071 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.089974 13071 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.090945 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:09.093364 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:09.093925 12536 leader_election.cc:343] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.094149 12637 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.094856 12687 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:09.098066 12687 ts_tablet_manager.cc:1403] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.162s	user 0.023s	sys 0.006s
I20260709 12:11:09.104933 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.109826 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:09.111222 13071 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:09.111724 13071 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:09.112520 13071 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.118763 12688 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:09.119343 12688 ts_tablet_manager.cc:1403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.163s	user 0.007s	sys 0.007s
I20260709 12:11:09.123164 12688 raft_consensus.cc:359] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.123950 12688 raft_consensus.cc:385] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.124222 12688 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.125201 12688 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.118505 12687 raft_consensus.cc:359] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.127377 12687 raft_consensus.cc:385] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.129180 12688 ts_tablet_manager.cc:1434] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:11:09.134707 12687 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.135632 12688 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:09.148223 12686 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.153698 12686 ts_tablet_manager.cc:1403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.199s	user 0.018s	sys 0.027s
I20260709 12:11:09.161808 12686 raft_consensus.cc:359] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.163956 12686 raft_consensus.cc:385] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.163398 12395 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:09.164337 12686 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.188838 12686 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.148970 12687 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.196077 12687 ts_tablet_manager.cc:1434] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.093s	user 0.006s	sys 0.002s
I20260709 12:11:09.196990 12686 ts_tablet_manager.cc:1434] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.038s	user 0.005s	sys 0.001s
I20260709 12:11:09.197361 12687 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:09.152226 12688 tablet_bootstrap.cc:654] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.207528 12687 tablet_bootstrap.cc:654] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.209486 12975 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.211081 12686 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:09.210821 12975 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.214509 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:09.217058 12975 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.218204 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.218968 12637 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:09.220135 12461 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:09.221084 12975 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.233650 12688 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:09.234279 12688 ts_tablet_manager.cc:1403] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.099s	user 0.017s	sys 0.010s
I20260709 12:11:09.237257 12688 raft_consensus.cc:359] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.238128 12688 raft_consensus.cc:385] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.238394 12688 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.233875 12687 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:09.239352 12687 ts_tablet_manager.cc:1403] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.000s
I20260709 12:11:09.239174 12688 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.240694 12975 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:09.240895 12688 ts_tablet_manager.cc:1434] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:11:09.242298 12688 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Bootstrap starting.
I20260709 12:11:09.246356 12686 tablet_bootstrap.cc:654] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
W20260709 12:11:09.247329 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:11:09.252266 12688 tablet_bootstrap.cc:654] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.281497 12687 raft_consensus.cc:359] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.282461 12687 raft_consensus.cc:385] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.285573 12687 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.286290 12687 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.288756 12687 ts_tablet_manager.cc:1434] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.000s
I20260709 12:11:09.290879 12687 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Bootstrap starting.
I20260709 12:11:09.292187 13073 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.292732 13073 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.297957 12687 tablet_bootstrap.cc:654] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.301438 12686 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.302045 12686 ts_tablet_manager.cc:1403] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.091s	user 0.037s	sys 0.016s
I20260709 12:11:09.304998 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:09.305591 12686 raft_consensus.cc:359] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.306298 12686 raft_consensus.cc:385] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.306563 12686 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
W20260709 12:11:09.306768 12610 leader_election.cc:343] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.307176 12686 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.309330 12686 ts_tablet_manager.cc:1434] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.003s
I20260709 12:11:09.309458 13073 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:09.309907 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:09.311522 13071 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:09.311707 12609 leader_election.cc:343] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.312150 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:09.312089 13071 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.312988 12687 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: No bootstrap required, opened a new log
I20260709 12:11:09.313582 12687 ts_tablet_manager.cc:1403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.004s
I20260709 12:11:09.314311 13071 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.316391 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:09.316664 13044 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:09.317121 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.316865 12687 raft_consensus.cc:359] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.317595 12687 raft_consensus.cc:385] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.317879 12687 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.318585 12687 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.321856 12686 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:09.322036 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:09.324446 13113 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:09.324847 13113 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.325080 12688 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: No bootstrap required, opened a new log
I20260709 12:11:09.325176 13113 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.325914 12688 ts_tablet_manager.cc:1403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent bootstrapping tablet: real 0.084s	user 0.013s	sys 0.008s
I20260709 12:11:09.323863 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.320829 12687 ts_tablet_manager.cc:1434] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:11:09.331365 12637 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.356667 12688 raft_consensus.cc:359] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.357807 12688 raft_consensus.cc:385] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.360424 12688 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.363888 12686 tablet_bootstrap.cc:654] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.364152 13071 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.369045 13071 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.362879 13113 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.374123 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:09.374643 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.374855 12489 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.375355 13071 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.373219 13113 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.375341 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:09.361878 12688 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.376586 12490 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.376787 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.383231 12688 ts_tablet_manager.cc:1434] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent starting tablet: real 0.057s	user 0.004s	sys 0.003s
I20260709 12:11:09.391594 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:09.392357 12637 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.399886 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.400137 12637 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.401569 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:09.403934 13071 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:09.404389 13071 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.404814 13071 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.411496 13044 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.412209 13044 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.414040 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:09.414569 13044 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:09.415619 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:09.416273 12568 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:09.417441 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:09.417912 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:09.418275 13044 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:09.418665 13044 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.418776 13101 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:09.419066 13044 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.419263 13101 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:09.420205 13101 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:09.431113 12610 leader_election.cc:343] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.413208 13071 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.437754 13071 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.439365 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:09.440407 12637 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.440652 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:09.441326 12495 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.442111 13044 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.445457 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:09.445708 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:09.446595 12564 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:11:09.447108 12610 leader_election.cc:343] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.450240 13044 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:09.455780 12536 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:09.457515 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.460927 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:09.462884 13138 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:09.463848 13138 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:09.464588 13138 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.470759 12686 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.471369 12686 ts_tablet_manager.cc:1403] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.150s	user 0.016s	sys 0.039s
I20260709 12:11:09.471997 12495 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.474891 12686 raft_consensus.cc:359] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.475826 12686 raft_consensus.cc:385] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.476104 12686 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.476794 12564 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:09.476909 12686 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.478295 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:09.479364 13044 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:09.480506 13044 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:09.481365 13044 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.492632 12686 ts_tablet_manager.cc:1434] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:11:09.505148 12686 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:09.531195 12394 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:09.549894 12391 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:09.565132 12686 tablet_bootstrap.cc:654] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.587163 13101 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:09.588471 12686 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.589001 12686 ts_tablet_manager.cc:1403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.084s	user 0.016s	sys 0.010s
I20260709 12:11:09.591676 12686 raft_consensus.cc:359] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.592240 12686 raft_consensus.cc:385] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.592406 12686 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.596431 12391 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:09.613111 13092 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.612790 12686 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.613745 13092 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.616956 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:09.618798 12610 leader_election.cc:343] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.620177 13092 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:09.621423 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:09.622406 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 0.
I20260709 12:11:09.623713 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:09.624791 13092 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:09.625821 13092 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.626164 13092 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.631239 12686 ts_tablet_manager.cc:1434] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.042s	user 0.009s	sys 0.000s
I20260709 12:11:09.644110 12686 tablet_bootstrap.cc:492] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:09.649603 13054 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.650244 13054 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.653501 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.653790 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:09.654388 12637 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:09.654515 12571 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 1: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:09.655757 12460 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:09.656826 12975 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.660871 12686 tablet_bootstrap.cc:654] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.655759 13054 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.668469 13138 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.669049 13138 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.671137 13138 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.673188 12686 tablet_bootstrap.cc:492] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.673774 12686 ts_tablet_manager.cc:1403] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.007s
I20260709 12:11:09.675825 12975 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:09.676864 12686 raft_consensus.cc:359] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.677683 12686 raft_consensus.cc:385] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.677959 12686 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.678908 12686 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.687417 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:09.688228 12489 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.687417 12686 ts_tablet_manager.cc:1434] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:11:09.690858 12686 tablet_bootstrap.cc:492] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:09.690996 13092 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.694460 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:09.695088 12571 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.694363 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:09.696691 12610 leader_election.cc:343] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:11:09.702965 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:09.703892 13092 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:09.704367 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:09.705931 12686 tablet_bootstrap.cc:654] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.706911 13092 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:09.707473 13092 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:09.708336 13092 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.710238 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.711026 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 0.
I20260709 12:11:09.714733 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:09.715543 13138 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:11:09.715919 13138 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.716291 13138 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.716183 13101 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:09.724920 12391 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:09.749315 13101 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:09.754998 12686 tablet_bootstrap.cc:492] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.755872 12686 ts_tablet_manager.cc:1403] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.065s	user 0.018s	sys 0.020s
I20260709 12:11:09.760247 12686 raft_consensus.cc:359] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.761096 12686 raft_consensus.cc:385] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.761433 12686 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.762302 13138 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.763446 12686 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.764822 13138 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.765717 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:09.768030 12490 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.766440 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:09.774031 12646 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.782356 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.786211 12686 ts_tablet_manager.cc:1434] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260709 12:11:09.787422 12686 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Bootstrap starting.
I20260709 12:11:09.787281 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:09.803923 12490 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.804586 13138 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:11:09.805092 13138 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:09.805985 13138 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.813109 12686 tablet_bootstrap.cc:654] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Neither blocks nor log segments found. Creating new log.
I20260709 12:11:09.833601 12686 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: No bootstrap required, opened a new log
I20260709 12:11:09.834645 12686 ts_tablet_manager.cc:1403] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.014s
I20260709 12:11:09.846127 12686 raft_consensus.cc:359] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.846931 12686 raft_consensus.cc:385] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:11:09.847245 12686 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Initialized, Role: FOLLOWER
I20260709 12:11:09.847996 12686 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.849910 12686 ts_tablet_manager.cc:1434] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:11:09.859313 12395 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:09.883466 13113 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:09.891274 13056 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:09.901593 13073 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:09.946213 12495 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.956983 13053 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:09.973088 13101 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:09.973683 13071 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.974216 13071 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.975399 13123 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:09.976092 13123 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:09.976491 13071 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.977705 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:09.978374 12489 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.979205 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:09.979674 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:09.979892 12637 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:09.980463 13113 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:09.980834 13113 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:09.981204 13113 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:09.984705 12497 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:09.992395 13123 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:09.993954 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:09.995060 13113 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:10.000063 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:10.001042 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:10.001849 12489 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:10.002230 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:10.076508 13113 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:10.086858 12489 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
W20260709 12:11:10.088393 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70689.4us
I20260709 12:11:10.090932 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:10.091779 13101 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:10.107093 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:10.108052 12646 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:10.114926 13101 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:10.116282 13101 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:10.118851 12646 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:10.126897 12999 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:10.127794 12999 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:10.131331 12999 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:10.122696 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105359us
I20260709 12:11:10.157544 13054 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:10.143450 13071 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:10.183107 12394 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 0 to 2, leader changed from <none> to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:10.185496 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:10.195322 13044 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:10.295068 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:10.297091 12497 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:11:10.304081 13101 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:10.305450 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:10.360780 12999 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:11:10.395720 13056 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:10.453476 13199 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:10.549176 12644 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:11:10.550971 13138 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:11:10.587946 12497 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:11:10.604260 13138 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:11:12.116863 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:11:12.130195 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:11:12.134702 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:11:12.139267 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:11:12.141330 11310 server_base.cc:1061] running on GCE node
I20260709 12:11:12.144182 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:11:12.144513 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:11:12.144759 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599072144738 us; error 0 us; skew 500 ppm
I20260709 12:11:12.145731 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:11:12.149829 11310 webserver.cc:533] Webserver started at http://127.11.11.132:35731/ using document root <none> and password file <none>
I20260709 12:11:12.150499 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:11:12.150813 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:11:12.151369 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:11:12.153039 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-3-root/instance:
uuid: "9485b0afaf144396adf854fff857414e"
format_stamp: "Formatted at 2026-07-09 12:11:12 on dist-test-slave-4qvp"
I20260709 12:11:12.163694 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.006s	sys 0.003s
I20260709 12:11:12.168337 13216 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:11:12.169431 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:11:12.169862 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-3-root
uuid: "9485b0afaf144396adf854fff857414e"
format_stamp: "Formatted at 2026-07-09 12:11:12 on dist-test-slave-4qvp"
I20260709 12:11:12.170226 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:11:12.213558 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:11:12.215794 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:11:12.220069 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:11:12.246834 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:11:12.247287 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:11:12.247696 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:11:12.248008 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
W20260709 12:11:12.371063 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75447.5us
I20260709 12:11:12.575268 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.132:33793
I20260709 12:11:12.575424 13286 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.132:33793 every 8 connection(s)
I20260709 12:11:12.601445 11310 auto_leader_rebalancer-test.cc:439] add a tserver: 9485b0afaf144396adf854fff857414e
I20260709 12:11:12.628131 13287 heartbeater.cc:344] Connected to a master server at 127.11.11.190:35399
I20260709 12:11:12.628720 13287 heartbeater.cc:461] Registering TS with master...
I20260709 12:11:12.630004 13287 heartbeater.cc:507] Master 127.11.11.190:35399 requested a full tablet report, sending...
I20260709 12:11:12.632997 12394 ts_manager.cc:194] Registered new tserver with Master: 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:12.635241 12394 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47684
I20260709 12:11:12.848832 12571 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:12.860355 12637 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:12.861827 13113 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:12.868103 12488 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:12.870729 13113 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:12.901531 13138 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:12.904760 12637 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:12.903640 12488 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:12.927898 12563 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:12.943063 12394 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:12.947221 12497 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:12.949071 13138 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:12.953959 12644 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:12.955647 13224 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:11:12.957445 12537 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b3c8a7a06a04c369587d6ecb196bb2e. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:11:12.960783 12537 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d60fb5ad76741448a5376ca4d3f05df. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:12.987156 13138 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.023276 12644 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.146983 12491 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:11:13.203598 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64094.2us
W20260709 12:11:13.265802 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48311.3us
I20260709 12:11:13.314862 12395 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:13.343128 13312 ts_tablet_manager.cc:933] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:13.448694 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64080.3us
I20260709 12:11:13.456238 13317 ts_tablet_manager.cc:933] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:13.499758 13317 tablet_copy_client.cc:323] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:13.512763 13312 tablet_copy_client.cc:323] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:13.560927 12644 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:13.603374 12582 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 5d60fb5ad76741448a5376ca4d3f05df from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:13.604173 12582 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 5d60fb5ad76741448a5376ca4d3f05df from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-5d60fb5ad76741448a5376ca4d3f05df
I20260709 12:11:13.611230 12494 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:13.611433 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 7b3c8a7a06a04c369587d6ecb196bb2e from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:13.612185 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 7b3c8a7a06a04c369587d6ecb196bb2e from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-7b3c8a7a06a04c369587d6ecb196bb2e
I20260709 12:11:13.613420 13316 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:13.613831 12563 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:13.616474 12582 tablet_copy_source_session.cc:215] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:13.619478 13316 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:13.620615 12581 tablet_copy_source_session.cc:215] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:13.624639 13317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d60fb5ad76741448a5376ca4d3f05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:13.633373 13312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3c8a7a06a04c369587d6ecb196bb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:13.641276 13287 heartbeater.cc:499] Master 127.11.11.190:35399 was elected leader, sending a full tablet report...
I20260709 12:11:13.667845 13316 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.671417 12494 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.700590 13317 tablet_copy_client.cc:806] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:13.701439 13317 tablet_copy_client.cc:670] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:13.709139 13317 tablet_copy_client.cc:538] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:13.701560 12391 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.711268 12564 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.721401 13312 tablet_copy_client.cc:806] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:13.722239 13312 tablet_copy_client.cc:670] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:13.729789 13312 tablet_copy_client.cc:538] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:11:13.737377 12611 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bbf126b18234b78b4f36bf9708ad96b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:13.756343 13317 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:13.760293 13312 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:13.775827 12494 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:13.807158 12642 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:13.816335 13317 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:13.816663 13235 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:13.820640 12564 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:13.821447 13317 tablet_bootstrap.cc:492] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:13.822373 13317 ts_tablet_manager.cc:1403] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.066s	user 0.022s	sys 0.012s
I20260709 12:11:13.826751 13227 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:11:13.881383 13312 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:13.882618 13312 tablet_bootstrap.cc:492] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:13.883378 13312 ts_tablet_manager.cc:1403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.130s	user 0.042s	sys 0.002s
I20260709 12:11:13.903765 13235 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.906018 12642 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.922921 12564 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:13.935331 13312 raft_consensus.cc:359] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:13.936578 13312 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:13.944738 13312 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:13.931720 13317 raft_consensus.cc:359] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:13.946064 13317 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:13.946898 13317 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:11:13.950863 12462 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dbcfa3bff034f5ab82e2ae802266099. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:13.982048 13317 ts_tablet_manager.cc:1434] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.159s	user 0.015s	sys 0.013s
I20260709 12:11:13.984882 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-5d60fb5ad76741448a5376ca4d3f05df received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:13.985455 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-5d60fb5ad76741448a5376ca4d3f05df on tablet 5d60fb5ad76741448a5376ca4d3f05df with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:13.990006 13262 raft_consensus.cc:1217] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:13.976614 12391 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:14.026517 13312 ts_tablet_manager.cc:1434] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.139s	user 0.028s	sys 0.042s
I20260709 12:11:14.029110 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-7b3c8a7a06a04c369587d6ecb196bb2e received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:14.029767 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-7b3c8a7a06a04c369587d6ecb196bb2e on tablet 7b3c8a7a06a04c369587d6ecb196bb2e with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:14.049088 13312 ts_tablet_manager.cc:933] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:14.065346 13312 tablet_copy_client.cc:323] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:14.253911 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 9bbf126b18234b78b4f36bf9708ad96b from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:14.254614 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 9bbf126b18234b78b4f36bf9708ad96b from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-9bbf126b18234b78b4f36bf9708ad96b
I20260709 12:11:14.265869 12656 tablet_copy_source_session.cc:215] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:14.276027 13312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbf126b18234b78b4f36bf9708ad96b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:14.312165 13312 tablet_copy_client.cc:806] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:14.312860 13312 tablet_copy_client.cc:670] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:14.326124 13317 ts_tablet_manager.cc:933] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:14.348640 13312 tablet_copy_client.cc:538] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:14.371821 13317 tablet_copy_client.cc:323] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:11:14.388268 13262 raft_consensus.cc:1217] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:14.411145 13312 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:14.414184 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 2dbcfa3bff034f5ab82e2ae802266099 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:11:14.414876 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 2dbcfa3bff034f5ab82e2ae802266099 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-2dbcfa3bff034f5ab82e2ae802266099
I20260709 12:11:14.429111 12507 tablet_copy_source_session.cc:215] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:14.432984 13317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dbcfa3bff034f5ab82e2ae802266099. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:14.459167 13317 tablet_copy_client.cc:806] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:14.459803 13317 tablet_copy_client.cc:670] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:14.461509 13312 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:14.462499 13312 tablet_bootstrap.cc:492] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:14.463178 13312 ts_tablet_manager.cc:1403] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.057s	user 0.025s	sys 0.009s
I20260709 12:11:14.466248 13312 raft_consensus.cc:359] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:14.467199 13312 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:14.467960 13312 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:14.478503 13317 tablet_copy_client.cc:538] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:14.479439 13312 ts_tablet_manager.cc:1434] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 12:11:14.489450 13317 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:14.495327 12656 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-9bbf126b18234b78b4f36bf9708ad96b received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:14.496039 12656 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-9bbf126b18234b78b4f36bf9708ad96b on tablet 9bbf126b18234b78b4f36bf9708ad96b with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:14.508850 13113 raft_consensus.cc:1064] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:14.512401 13113 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:14.524684 12638 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:14.527143 13262 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:14.530012 13309 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:11:14.531998 13337 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:14.534847 12493 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:14.542232 13337 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:14.566272 13317 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:14.567406 13317 tablet_bootstrap.cc:492] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:14.568128 13317 ts_tablet_manager.cc:1403] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.079s	user 0.021s	sys 0.008s
I20260709 12:11:14.584087 13317 raft_consensus.cc:359] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:14.585048 13317 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:14.596180 13113 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.602623 12493 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.614351 13317 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:14.621695 13317 ts_tablet_manager.cc:1434] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.053s	user 0.010s	sys 0.001s
I20260709 12:11:14.623155 12638 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.628810 13262 raft_consensus.cc:1217] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:14.630321 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-2dbcfa3bff034f5ab82e2ae802266099 received from {username='slave'} at 127.0.0.1:54234
I20260709 12:11:14.630829 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-2dbcfa3bff034f5ab82e2ae802266099 on tablet 2dbcfa3bff034f5ab82e2ae802266099 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:14.644452 13262 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.653560 12392 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:14.702787 12563 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:14.730167 13262 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:14.730664 12493 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:14.733012 13337 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:14.734688 13309 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:14.744988 13308 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.754420 13262 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.766397 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5d60fb5ad76741448a5376ca4d3f05df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:14.785351 12392 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.847347 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d60fb5ad76741448a5376ca4d3f05df with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:14.914364 13352 tablet_replica.cc:333] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:14.914737 12497 raft_consensus.cc:2955] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:14.936674 13352 raft_consensus.cc:2243] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:14.966202 13352 raft_consensus.cc:2272] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:14.968036 13337 raft_consensus.cc:1064] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:14.970676 13337 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:14.972342 13262 raft_consensus.cc:1217] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:14.976943 13352 ts_tablet_manager.cc:1916] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:15.007561 12638 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.008652 13262 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.010334 13337 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:15.011210 13308 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:15.013921 12492 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.020358 13309 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:15.054716 13309 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.059437 13262 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.059545 12638 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.059603 13352 ts_tablet_manager.cc:1929] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:15.065438 13352 log.cc:1199] T 5d60fb5ad76741448a5376ca4d3f05df P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/5d60fb5ad76741448a5376ca4d3f05df
I20260709 12:11:15.079131 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 5d60fb5ad76741448a5376ca4d3f05df (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:15.102623 12395 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.114025 12492 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.131295 13073 raft_consensus.cc:1064] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:15.136128 12563 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:15.141062 13073 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:15.176640 13261 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:15.183082 13308 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:15.198608 12646 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:15.218050 13113 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:15.219358 13261 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.251267 13359 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:11:15.312700 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96888.4us
I20260709 12:11:15.321372 12638 raft_consensus.cc:2955] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.383472 12394 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.400100 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b3c8a7a06a04c369587d6ecb196bb2e with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:15.422039 13361 tablet_replica.cc:333] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:15.425153 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7b3c8a7a06a04c369587d6ecb196bb2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:15.429481 13262 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.429816 12564 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.432190 13338 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:15.436028 13316 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:15.441288 13351 raft_consensus.cc:1064] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:15.444451 13351 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:15.445149 13361 raft_consensus.cc:2243] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:15.463493 13361 raft_consensus.cc:2272] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:15.471988 13361 ts_tablet_manager.cc:1916] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:15.477519 12492 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.480667 13073 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:15.507684 13262 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.509455 13235 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:15.510121 12564 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.513119 12644 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:15.513159 13194 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:11:15.516475 13194 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:15.546717 13261 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.540753 13338 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.572121 13361 ts_tablet_manager.cc:1929] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:15.573189 13361 log.cc:1199] T 7b3c8a7a06a04c369587d6ecb196bb2e P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/7b3c8a7a06a04c369587d6ecb196bb2e
I20260709 12:11:15.582108 12493 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.606329 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet 7b3c8a7a06a04c369587d6ecb196bb2e (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:15.613798 12569 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.617313 13194 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.631487 13260 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.642549 12569 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.649910 12394 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.676350 12644 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:15.675954 12646 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.697453 13261 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:15.700172 12494 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:15.703799 13357 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:15.706462 13357 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:15.716846 12394 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.728803 13073 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.739001 12493 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.775836 12491 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:15.781823 13260 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:15.783766 13194 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:15.790058 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9bbf126b18234b78b4f36bf9708ad96b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:15.792129 13351 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.801537 12394 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.827559 13260 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.833786 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bbf126b18234b78b4f36bf9708ad96b with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:15.852128 13371 tablet_replica.cc:333] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:15.853607 13371 raft_consensus.cc:2243] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:15.863586 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2dbcfa3bff034f5ab82e2ae802266099 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:15.865010 13371 raft_consensus.cc:2272] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:15.871229 13371 ts_tablet_manager.cc:1916] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:15.876267 12395 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.888700 12646 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:15.898386 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 2dbcfa3bff034f5ab82e2ae802266099 with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:15.905182 13260 raft_consensus.cc:2955] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.959892 13371 ts_tablet_manager.cc:1929] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:15.960495 13371 log.cc:1199] T 9bbf126b18234b78b4f36bf9708ad96b P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/9bbf126b18234b78b4f36bf9708ad96b
I20260709 12:11:15.962590 13371 tablet_replica.cc:333] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:15.963865 13371 raft_consensus.cc:2243] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:15.964634 13371 raft_consensus.cc:2272] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:15.965175 13369 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:15.970309 13371 ts_tablet_manager.cc:1916] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:15.982453 12638 raft_consensus.cc:2955] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:15.991272 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet 9bbf126b18234b78b4f36bf9708ad96b (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:15.995229 13371 ts_tablet_manager.cc:1929] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:15.995800 13371 log.cc:1199] T 2dbcfa3bff034f5ab82e2ae802266099 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/2dbcfa3bff034f5ab82e2ae802266099
I20260709 12:11:16.012807 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet 2dbcfa3bff034f5ab82e2ae802266099 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:17.034530 12563 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.049757 12638 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:17.051290 12537 consensus_peers.cc:597] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd6a5b4068ea44d2ac6f61924dd38c0d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:17.052121 12494 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:17.052428 13309 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:17.054603 13308 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:17.084249 13308 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.087960 12491 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.094503 12638 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.109531 12395 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.157718 12491 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.174049 12563 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:17.175835 13369 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:11:17.177351 12462 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65037753870c4624995dde1173266f49. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:17.178956 12638 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:17.180537 13369 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:17.249855 13235 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.257280 12564 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.262884 12644 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.298475 12564 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.311856 12394 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:17.323010 12646 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:17.323202 12491 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:17.324946 13308 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:11:17.330042 12537 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dc881215d9f4498b162561e79143f2d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:17.337278 13309 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:17.348393 13360 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.354509 12494 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.368863 12644 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.374157 12638 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.397065 12395 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.404001 12564 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:17.405493 12611 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bb3f87ad7904e7d97f68dbbd8c3ab60. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:17.406337 13375 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:17.414872 12493 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:17.416700 13357 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:17.435438 13229 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.451447 12494 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.453904 12569 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:17.477380 12395 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:17.489243 13391 ts_tablet_manager.cc:933] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:17.523538 13391 tablet_copy_client.cc:323] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:17.526213 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet dd6a5b4068ea44d2ac6f61924dd38c0d from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:17.526966 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet dd6a5b4068ea44d2ac6f61924dd38c0d from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-dd6a5b4068ea44d2ac6f61924dd38c0d
I20260709 12:11:17.544705 12581 tablet_copy_source_session.cc:215] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:17.549434 13391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a5b4068ea44d2ac6f61924dd38c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:17.574870 13393 ts_tablet_manager.cc:933] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:17.583438 13393 tablet_copy_client.cc:323] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:11:17.585937 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 65037753870c4624995dde1173266f49 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:11:17.586681 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 65037753870c4624995dde1173266f49 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-65037753870c4624995dde1173266f49
I20260709 12:11:17.590337 13391 tablet_copy_client.cc:806] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:17.590993 13391 tablet_copy_client.cc:670] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:17.606271 12507 tablet_copy_source_session.cc:215] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:17.608438 13391 tablet_copy_client.cc:538] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:17.610323 13393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65037753870c4624995dde1173266f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:17.626392 13391 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:17.637683 13393 tablet_copy_client.cc:806] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:17.638396 13393 tablet_copy_client.cc:670] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:17.649595 13393 tablet_copy_client.cc:538] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:17.669482 13393 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:17.672129 13391 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:17.673358 13391 tablet_bootstrap.cc:492] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:17.674721 13391 ts_tablet_manager.cc:1403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.049s	user 0.016s	sys 0.015s
I20260709 12:11:17.718181 13391 raft_consensus.cc:359] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.719198 13391 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:17.720053 13391 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.731673 13391 ts_tablet_manager.cc:1434] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.001s
I20260709 12:11:17.724671 13393 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:17.733043 13393 tablet_bootstrap.cc:492] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:17.734166 13393 ts_tablet_manager.cc:1403] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.065s	user 0.021s	sys 0.009s
I20260709 12:11:17.738112 13393 raft_consensus.cc:359] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.739813 13393 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:17.740830 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-dd6a5b4068ea44d2ac6f61924dd38c0d received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:17.740667 13393 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.741454 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-dd6a5b4068ea44d2ac6f61924dd38c0d on tablet dd6a5b4068ea44d2ac6f61924dd38c0d with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:17.802172 13393 ts_tablet_manager.cc:1434] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.067s	user 0.016s	sys 0.019s
I20260709 12:11:17.804458 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-65037753870c4624995dde1173266f49 received from {username='slave'} at 127.0.0.1:54234
I20260709 12:11:17.804967 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-65037753870c4624995dde1173266f49 on tablet 65037753870c4624995dde1173266f49 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:17.839175 13393 ts_tablet_manager.cc:933] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:17.841367 13393 tablet_copy_client.cc:323] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:17.843396 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 4dc881215d9f4498b162561e79143f2d from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:17.843940 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 4dc881215d9f4498b162561e79143f2d from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-4dc881215d9f4498b162561e79143f2d
I20260709 12:11:17.851182 12581 tablet_copy_source_session.cc:215] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:17.854512 13393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dc881215d9f4498b162561e79143f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:17.875962 13393 tablet_copy_client.cc:806] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:17.876636 13393 tablet_copy_client.cc:670] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:17.882794 13393 tablet_copy_client.cc:538] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:17.894250 13393 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:17.931295 13393 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:17.932348 13393 tablet_bootstrap.cc:492] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:17.933076 13393 ts_tablet_manager.cc:1403] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.019s
I20260709 12:11:17.936693 13393 raft_consensus.cc:359] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.937688 13393 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:17.938411 13393 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:17.941105 13393 ts_tablet_manager.cc:1434] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:11:17.943387 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-4dc881215d9f4498b162561e79143f2d received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:17.943917 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-4dc881215d9f4498b162561e79143f2d on tablet 4dc881215d9f4498b162561e79143f2d with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:17.948450 13391 ts_tablet_manager.cc:933] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:17.960469 13391 tablet_copy_client.cc:323] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:17.962420 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:17.963063 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-3bb3f87ad7904e7d97f68dbbd8c3ab60
I20260709 12:11:17.972267 13260 raft_consensus.cc:1217] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:17.985829 12656 tablet_copy_source_session.cc:215] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:17.996204 13391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:18.014832 13391 tablet_copy_client.cc:806] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:18.015519 13391 tablet_copy_client.cc:670] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:18.022898 13391 tablet_copy_client.cc:538] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:18.064427 13391 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:18.097760 13391 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:18.098737 13391 tablet_bootstrap.cc:492] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:18.099444 13391 ts_tablet_manager.cc:1403] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.035s	user 0.022s	sys 0.005s
I20260709 12:11:18.102964 13391 raft_consensus.cc:359] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:18.103947 13391 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:18.104645 13391 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:18.107930 13391 ts_tablet_manager.cc:1434] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:11:18.110139 12656 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-3bb3f87ad7904e7d97f68dbbd8c3ab60 received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:18.110664 12656 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-3bb3f87ad7904e7d97f68dbbd8c3ab60 on tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:18.184595 13260 raft_consensus.cc:1217] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:18.290069 13260 raft_consensus.cc:1217] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:18.343758 13260 raft_consensus.cc:1217] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:18.402537 13308 raft_consensus.cc:1064] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:18.405359 13308 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:18.416579 13260 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.419273 13360 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.420212 12644 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.423594 12493 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.423924 13308 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.430732 13308 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:18.437541 13360 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.442492 13260 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.455233 12494 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.460172 12394 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.497851 12638 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.528123 12564 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:18.534596 13260 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:18.534783 12638 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:18.536052 13309 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:18.537070 13410 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:18.545944 13309 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.548039 13260 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.550629 12638 raft_consensus.cc:2955] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.557960 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet dd6a5b4068ea44d2ac6f61924dd38c0d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:18.573485 12394 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.587962 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet dd6a5b4068ea44d2ac6f61924dd38c0d with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:18.594135 13413 tablet_replica.cc:333] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:18.608487 13383 raft_consensus.cc:1064] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:18.611672 13383 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:18.638199 13413 raft_consensus.cc:2243] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:18.639159 13413 raft_consensus.cc:2272] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:18.643311 12646 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.643847 13260 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.644851 13351 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.644694 13413 ts_tablet_manager.cc:1916] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:18.647686 13351 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:18.650456 12564 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.664484 13351 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.678264 13383 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.682536 13413 ts_tablet_manager.cc:1929] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:18.683158 13413 log.cc:1199] T dd6a5b4068ea44d2ac6f61924dd38c0d P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/dd6a5b4068ea44d2ac6f61924dd38c0d
I20260709 12:11:18.685595 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet dd6a5b4068ea44d2ac6f61924dd38c0d (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:18.696503 12564 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.695426 13261 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.714448 12646 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.712692 12395 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:18.743201 13308 raft_consensus.cc:1064] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:18.750573 12493 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:18.750411 13308 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:18.758834 13261 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:18.759639 12563 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:18.761513 13235 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:18.768168 13351 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:18.761866 13229 raft_consensus.cc:1064] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:18.774993 13229 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:18.780123 13383 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.789557 12638 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.792145 13260 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.786952 13261 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.789831 12493 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.786481 12563 raft_consensus.cc:2955] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.804847 13308 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:18.815994 12395 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.817322 13309 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.819453 13410 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.839337 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 65037753870c4624995dde1173266f49 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:18.847102 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 65037753870c4624995dde1173266f49 with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:18.859408 13425 tablet_replica.cc:333] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:18.867877 13425 raft_consensus.cc:2243] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:18.868728 13425 raft_consensus.cc:2272] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:18.872951 13260 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.874362 13425 ts_tablet_manager.cc:1916] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:18.876327 13421 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.878187 13410 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.880353 12638 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.887941 12494 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.889602 13403 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:11:18.908439 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77439.1us
I20260709 12:11:18.909507 12395 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:18.907423 12494 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:18.914798 12563 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:18.931543 13357 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:18.941651 13421 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.004127 13261 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.024664 13425 ts_tablet_manager.cc:1929] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:19.025264 13425 log.cc:1199] T 65037753870c4624995dde1173266f49 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/65037753870c4624995dde1173266f49
I20260709 12:11:19.011649 12493 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.029781 13261 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.051954 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 65037753870c4624995dde1173266f49 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:19.049388 12392 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:19.068794 12564 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.112275 12638 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:19.118727 13261 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:19.120821 13375 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:19.131249 12491 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:19.132982 13421 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:19.138664 13229 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.140092 13261 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.145267 12491 raft_consensus.cc:2955] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.153020 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:19.144518 12564 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:19.163887 12394 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:19.180613 13261 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:19.183424 13429 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:19.183851 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:19.184063 12638 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:19.186455 13429 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:19.198596 13430 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.200105 13260 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.203804 12638 raft_consensus.cc:2955] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.220600 13431 tablet_replica.cc:333] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:19.224407 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4dc881215d9f4498b162561e79143f2d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:19.230396 13431 raft_consensus.cc:2243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:19.231545 13431 raft_consensus.cc:2272] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:19.233068 12395 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:19.253918 13431 ts_tablet_manager.cc:1916] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:19.255244 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet 4dc881215d9f4498b162561e79143f2d with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:19.264597 13432 tablet_replica.cc:333] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:19.266119 13432 raft_consensus.cc:2243] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:19.266978 13432 raft_consensus.cc:2272] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:19.270810 13432 ts_tablet_manager.cc:1916] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:19.283012 13431 ts_tablet_manager.cc:1929] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:19.283645 13431 log.cc:1199] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/3bb3f87ad7904e7d97f68dbbd8c3ab60
I20260709 12:11:19.285929 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet 3bb3f87ad7904e7d97f68dbbd8c3ab60 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:19.292577 13432 ts_tablet_manager.cc:1929] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:19.293118 13432 log.cc:1199] T 4dc881215d9f4498b162561e79143f2d P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/4dc881215d9f4498b162561e79143f2d
I20260709 12:11:19.295346 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet 4dc881215d9f4498b162561e79143f2d (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:20.329820 12564 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:20.336665 12642 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:20.337522 12537 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59beb90f79b0450ea2fbb2b7e405261d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:20.337510 12491 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:20.338466 13430 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:20.339318 13410 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:20.351241 13430 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.354349 12642 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.358300 12493 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.376462 12642 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:11:20.385089 12611 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40d0e235c01b4b7f89948ab708403a01. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:20.387277 12563 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:20.373291 12392 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.389648 12493 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:20.392928 13375 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:20.394245 13229 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:20.439220 13441 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.442416 12491 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.457779 12564 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:11:20.480945 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47767.1us
I20260709 12:11:20.485033 12491 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:20.497452 12644 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:20.499284 12564 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:20.501221 13404 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:11:20.502377 12462 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79b687d20a534e70865ef902fbd0d4a5. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:20.503997 13448 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:20.511858 12394 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.518796 13446 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.524004 12642 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.524199 12569 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.576187 12392 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:20.570138 12569 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:20.600296 12491 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:20.600461 12642 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:20.601234 12537 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07fe6c41f52649a88327e52e0352b171. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:20.602047 13437 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:20.602970 13438 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:20.612361 13438 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.614306 12491 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.614591 12642 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:20.632910 12395 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:20.848272 13457 ts_tablet_manager.cc:933] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:20.850687 13457 tablet_copy_client.cc:323] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:20.852741 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 59beb90f79b0450ea2fbb2b7e405261d from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:20.853281 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 59beb90f79b0450ea2fbb2b7e405261d from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-59beb90f79b0450ea2fbb2b7e405261d
I20260709 12:11:20.859956 12581 tablet_copy_source_session.cc:215] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:20.863394 13457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59beb90f79b0450ea2fbb2b7e405261d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:20.879199 13457 tablet_copy_client.cc:806] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:20.879920 13457 tablet_copy_client.cc:670] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:20.884394 13457 tablet_copy_client.cc:538] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:20.893883 13457 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:20.913179 13457 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:20.914160 13457 tablet_bootstrap.cc:492] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:20.914883 13457 ts_tablet_manager.cc:1403] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.003s
I20260709 12:11:20.918296 13457 raft_consensus.cc:359] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:20.919263 13457 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:20.920020 13457 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:20.922911 13460 ts_tablet_manager.cc:933] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:20.928907 13457 ts_tablet_manager.cc:1434] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.004s
I20260709 12:11:20.931380 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-59beb90f79b0450ea2fbb2b7e405261d received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:20.932145 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-59beb90f79b0450ea2fbb2b7e405261d on tablet 59beb90f79b0450ea2fbb2b7e405261d with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:20.938951 13460 tablet_copy_client.cc:323] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:20.941004 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 40d0e235c01b4b7f89948ab708403a01 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:20.941568 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 40d0e235c01b4b7f89948ab708403a01 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-40d0e235c01b4b7f89948ab708403a01
I20260709 12:11:20.950277 12656 tablet_copy_source_session.cc:215] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:20.953917 13460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d0e235c01b4b7f89948ab708403a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:20.964478 13457 ts_tablet_manager.cc:933] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:20.974740 13460 tablet_copy_client.cc:806] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:20.975646 13460 tablet_copy_client.cc:670] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:20.989987 13460 tablet_copy_client.cc:538] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:21.001029 13457 tablet_copy_client.cc:323] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:11:21.003134 13460 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:21.021299 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 79b687d20a534e70865ef902fbd0d4a5 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:11:21.022029 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 79b687d20a534e70865ef902fbd0d4a5 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-79b687d20a534e70865ef902fbd0d4a5
I20260709 12:11:21.030684 12507 tablet_copy_source_session.cc:215] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:21.043417 13457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b687d20a534e70865ef902fbd0d4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:21.036974 13460 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:21.047134 13460 tablet_bootstrap.cc:492] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:21.056362 13460 ts_tablet_manager.cc:1403] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.054s	user 0.026s	sys 0.008s
I20260709 12:11:21.069473 13460 raft_consensus.cc:359] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:21.070600 13460 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:21.071398 13460 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:21.074491 13460 ts_tablet_manager.cc:1434] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20260709 12:11:21.076512 12656 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-40d0e235c01b4b7f89948ab708403a01 received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:21.076854 12656 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-40d0e235c01b4b7f89948ab708403a01 on tablet 40d0e235c01b4b7f89948ab708403a01 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:21.083854 13457 tablet_copy_client.cc:806] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:21.084543 13457 tablet_copy_client.cc:670] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:21.090713 13457 tablet_copy_client.cc:538] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:21.101172 13457 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:21.123915 13457 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:21.124970 13457 tablet_bootstrap.cc:492] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:21.125728 13457 ts_tablet_manager.cc:1403] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.016s
I20260709 12:11:21.128518 13457 raft_consensus.cc:359] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:21.129292 13457 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:21.129961 13457 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:21.133318 13457 ts_tablet_manager.cc:1434] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:11:21.136003 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-79b687d20a534e70865ef902fbd0d4a5 received from {username='slave'} at 127.0.0.1:54234
I20260709 12:11:21.136521 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-79b687d20a534e70865ef902fbd0d4a5 on tablet 79b687d20a534e70865ef902fbd0d4a5 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:21.169867 13457 ts_tablet_manager.cc:933] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:21.171972 13457 tablet_copy_client.cc:323] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:21.173832 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 07fe6c41f52649a88327e52e0352b171 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:21.174932 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 07fe6c41f52649a88327e52e0352b171 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-07fe6c41f52649a88327e52e0352b171
I20260709 12:11:21.186228 12581 tablet_copy_source_session.cc:215] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:21.190207 13457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07fe6c41f52649a88327e52e0352b171. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:21.209448 13457 tablet_copy_client.cc:806] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:21.210060 13457 tablet_copy_client.cc:670] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:21.215829 13457 tablet_copy_client.cc:538] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:21.224567 13457 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:21.243748 13457 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:21.244740 13457 tablet_bootstrap.cc:492] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:21.245364 13457 ts_tablet_manager.cc:1403] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.021s	user 0.021s	sys 0.000s
I20260709 12:11:21.251869 13457 raft_consensus.cc:359] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:21.252746 13260 raft_consensus.cc:1217] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:21.252779 13457 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:21.254370 13457 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:21.257221 13457 ts_tablet_manager.cc:1434] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:11:21.258988 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-07fe6c41f52649a88327e52e0352b171 received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:21.259495 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-07fe6c41f52649a88327e52e0352b171 on tablet 07fe6c41f52649a88327e52e0352b171 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:21.344136 13260 raft_consensus.cc:1217] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:21.482671 13260 raft_consensus.cc:1217] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:21.650290 13260 raft_consensus.cc:1217] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:21.779809 13410 raft_consensus.cc:1064] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:21.782943 13410 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:21.791373 13260 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:21.793658 13438 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:21.798807 12491 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:21.798955 12644 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:21.800730 13438 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:21.801772 13410 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:21.816120 12644 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.820927 12491 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.818655 13437 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.839972 12394 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.869242 13260 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.904220 12568 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:21.916600 13260 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:21.917680 12491 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:21.923374 13410 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:21.924227 13437 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:21.933035 13410 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.937741 12491 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.946700 13446 raft_consensus.cc:1064] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:21.949707 13446 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:21.950047 13260 raft_consensus.cc:2955] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.959118 13261 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:21.961468 13404 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:21.964775 12394 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:21.969805 12568 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:21.971057 12644 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:21.984535 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 59beb90f79b0450ea2fbb2b7e405261d with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:21.993360 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 59beb90f79b0450ea2fbb2b7e405261d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:21.998876 13479 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:22.000473 13482 tablet_replica.cc:333] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:22.001386 13446 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:22.001720 13482 raft_consensus.cc:2243] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:22.007040 13482 raft_consensus.cc:2272] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:22.010978 13482 ts_tablet_manager.cc:1916] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:22.026158 13375 raft_consensus.cc:1064] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:22.027796 13404 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.032738 13375 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:22.034268 12644 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.045132 13261 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.051843 12394 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:22.067386 13482 ts_tablet_manager.cc:1929] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:22.067997 13482 log.cc:1199] T 59beb90f79b0450ea2fbb2b7e405261d P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/59beb90f79b0450ea2fbb2b7e405261d
I20260709 12:11:22.069818 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 59beb90f79b0450ea2fbb2b7e405261d (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:22.072497 13260 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:22.074189 13441 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:22.079528 12494 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:22.081036 12568 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:22.081101 13441 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:11:22.089231 12569 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.092533 13229 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:22.104964 12494 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:22.109892 13410 raft_consensus.cc:1064] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:22.118943 13262 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:22.121182 13480 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:22.112396 13410 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:22.137272 13446 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.150403 13261 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:22.157850 13469 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:22.163688 12644 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:22.162843 13262 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.170504 13438 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:22.223296 12493 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:22.226773 13410 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:22.233851 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 79b687d20a534e70865ef902fbd0d4a5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:22.240933 12563 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:22.251545 13479 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:22.253482 13229 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.255548 12395 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.276384 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 79b687d20a534e70865ef902fbd0d4a5 with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:22.277392 13482 tablet_replica.cc:333] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:22.278467 13482 raft_consensus.cc:2243] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:22.278707 13437 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.279281 13482 raft_consensus.cc:2272] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:22.285844 13261 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.292922 13482 ts_tablet_manager.cc:1916] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:22.301115 12568 raft_consensus.cc:2955] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.302516 12569 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.298573 13262 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.308216 12494 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.311039 12493 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.376663 12392 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:22.337812 12644 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.392549 12395 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:22.416885 13482 ts_tablet_manager.cc:1929] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:22.417514 13482 log.cc:1199] T 79b687d20a534e70865ef902fbd0d4a5 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/79b687d20a534e70865ef902fbd0d4a5
I20260709 12:11:22.423856 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 79b687d20a534e70865ef902fbd0d4a5 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:22.442335 12568 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:22.463322 12644 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:22.477123 13262 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:22.479053 13438 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:22.482340 12638 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:22.484007 13438 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:22.489988 13261 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:22.491667 12494 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:22.492705 13375 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:22.494959 13229 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:22.494814 13437 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.505173 13487 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.506752 12638 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.507503 13261 raft_consensus.cc:2955] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.517311 13260 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.528501 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 07fe6c41f52649a88327e52e0352b171 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:22.530871 12491 raft_consensus.cc:2955] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.538288 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 40d0e235c01b4b7f89948ab708403a01 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:22.548475 12391 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:22.553333 12394 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:22.567430 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 40d0e235c01b4b7f89948ab708403a01 with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:22.589184 13496 tablet_replica.cc:333] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:22.590225 13496 raft_consensus.cc:2243] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:22.590471 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet 07fe6c41f52649a88327e52e0352b171 with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:22.591181 13496 raft_consensus.cc:2272] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:22.605320 13497 tablet_replica.cc:333] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:22.606662 13496 ts_tablet_manager.cc:1916] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:22.606913 13497 raft_consensus.cc:2243] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:22.607728 13497 raft_consensus.cc:2272] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:22.614071 13497 ts_tablet_manager.cc:1916] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:22.630430 13496 ts_tablet_manager.cc:1929] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:22.631197 13496 log.cc:1199] T 40d0e235c01b4b7f89948ab708403a01 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/40d0e235c01b4b7f89948ab708403a01
I20260709 12:11:22.633325 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet 40d0e235c01b4b7f89948ab708403a01 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:22.634272 13497 ts_tablet_manager.cc:1929] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:22.634877 13497 log.cc:1199] T 07fe6c41f52649a88327e52e0352b171 P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/07fe6c41f52649a88327e52e0352b171
I20260709 12:11:22.638461 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet 07fe6c41f52649a88327e52e0352b171 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:23.699651 12569 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:23.711903 12638 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:11:23.713627 12537 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f701a23559945868ce6e3da5866f95c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:23.714229 13438 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.714881 12492 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:23.716310 13410 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.723881 13410 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.725603 12638 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.730330 12492 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.749989 12392 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.765969 12638 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:23.784675 12568 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:23.786677 12611 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3a0781287514c758f107d79c1f96be4. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:23.788532 13487 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.791311 12492 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:23.795469 13487 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.815454 13498 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.822762 12492 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.821118 12569 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.840134 12642 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:23.852072 12491 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:23.854068 13487 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.855981 12569 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:23.859525 13487 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:11:23.872313 12611 consensus_peers.cc:597] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c13f916d8b4d463182a1d1eb50150e31. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:23.889561 12391 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:23.896263 13441 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.899613 12494 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.926584 12569 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:23.944655 12391 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:11:23.958297 12537 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d0dce81bc984a0690d08335a4937b2c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:23.960145 12642 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:23.962298 13507 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.964725 12494 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:23.972110 12569 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.975306 13507 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:23.991222 13438 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:23.992939 12644 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:24.009963 12494 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:24.015822 12394 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:24.261102 13523 ts_tablet_manager.cc:933] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:24.264391 13523 tablet_copy_client.cc:323] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:24.268640 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 6f701a23559945868ce6e3da5866f95c from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:24.269361 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 6f701a23559945868ce6e3da5866f95c from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-6f701a23559945868ce6e3da5866f95c
I20260709 12:11:24.277768 12581 tablet_copy_source_session.cc:215] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:24.281081 13523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f701a23559945868ce6e3da5866f95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:24.297281 13523 tablet_copy_client.cc:806] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:24.298036 13523 tablet_copy_client.cc:670] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:24.302909 13523 tablet_copy_client.cc:538] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:24.313467 13523 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:24.335281 13523 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:24.336195 13523 tablet_bootstrap.cc:492] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:24.336899 13523 ts_tablet_manager.cc:1403] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.024s	user 0.023s	sys 0.000s
I20260709 12:11:24.340065 13523 raft_consensus.cc:359] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.340938 13523 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:24.341496 13523 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.344776 13523 ts_tablet_manager.cc:1434] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.003s
I20260709 12:11:24.346879 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-6f701a23559945868ce6e3da5866f95c received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:24.347373 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-6f701a23559945868ce6e3da5866f95c on tablet 6f701a23559945868ce6e3da5866f95c with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:24.395953 13523 ts_tablet_manager.cc:933] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:24.399379 13523 tablet_copy_client.cc:323] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:24.401713 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet a3a0781287514c758f107d79c1f96be4 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:24.402367 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet a3a0781287514c758f107d79c1f96be4 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-a3a0781287514c758f107d79c1f96be4
I20260709 12:11:24.408661 13528 ts_tablet_manager.cc:933] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:24.411643 12656 tablet_copy_source_session.cc:215] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:24.415464 13528 tablet_copy_client.cc:323] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:24.417021 13523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a0781287514c758f107d79c1f96be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:24.420542 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet c13f916d8b4d463182a1d1eb50150e31 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:24.421193 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet c13f916d8b4d463182a1d1eb50150e31 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-c13f916d8b4d463182a1d1eb50150e31
I20260709 12:11:24.435587 12656 tablet_copy_source_session.cc:215] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:24.441112 13528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13f916d8b4d463182a1d1eb50150e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:24.473058 13523 tablet_copy_client.cc:806] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:24.474013 13523 tablet_copy_client.cc:670] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:24.482442 13523 tablet_copy_client.cc:538] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:24.485805 13530 ts_tablet_manager.cc:933] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:24.491160 13528 tablet_copy_client.cc:806] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:24.491797 13528 tablet_copy_client.cc:670] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:24.498389 13523 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:24.503130 13530 tablet_copy_client.cc:323] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:24.505774 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 5d0dce81bc984a0690d08335a4937b2c from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:24.506371 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 5d0dce81bc984a0690d08335a4937b2c from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-5d0dce81bc984a0690d08335a4937b2c
I20260709 12:11:24.507170 13528 tablet_copy_client.cc:538] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:24.518256 13528 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:24.540709 12581 tablet_copy_source_session.cc:215] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:24.544797 13530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0dce81bc984a0690d08335a4937b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:24.580606 13530 tablet_copy_client.cc:806] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:24.581445 13530 tablet_copy_client.cc:670] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:24.586194 13530 tablet_copy_client.cc:538] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:24.593469 13523 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:24.594501 13523 tablet_bootstrap.cc:492] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:24.595188 13523 ts_tablet_manager.cc:1403] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.097s	user 0.014s	sys 0.047s
I20260709 12:11:24.600116 13530 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:24.602958 13523 raft_consensus.cc:359] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.603986 13523 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:24.604794 13523 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.609066 13528 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:24.610339 13528 tablet_bootstrap.cc:492] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:24.611177 13528 ts_tablet_manager.cc:1403] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.093s	user 0.023s	sys 0.012s
I20260709 12:11:24.615067 13528 raft_consensus.cc:359] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.616036 13528 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:24.616775 13528 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.620811 13523 ts_tablet_manager.cc:1434] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:11:24.622741 12656 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-a3a0781287514c758f107d79c1f96be4 received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:24.623243 12656 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-a3a0781287514c758f107d79c1f96be4 on tablet a3a0781287514c758f107d79c1f96be4 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:24.625768 13528 ts_tablet_manager.cc:1434] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260709 12:11:24.627909 12656 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-c13f916d8b4d463182a1d1eb50150e31 received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:24.628337 12656 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-c13f916d8b4d463182a1d1eb50150e31 on tablet c13f916d8b4d463182a1d1eb50150e31 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:24.651557 13530 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:24.652375 13530 tablet_bootstrap.cc:492] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:24.652957 13530 ts_tablet_manager.cc:1403] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.053s	user 0.016s	sys 0.006s
I20260709 12:11:24.655256 13530 raft_consensus.cc:359] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.655885 13530 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:24.656426 13530 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:24.658943 13530 ts_tablet_manager.cc:1434] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:11:24.660862 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-5d0dce81bc984a0690d08335a4937b2c received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:24.661305 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-5d0dce81bc984a0690d08335a4937b2c on tablet 5d0dce81bc984a0690d08335a4937b2c with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:24.850216 13260 raft_consensus.cc:1217] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:24.857198 13260 raft_consensus.cc:1217] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:11:24.943172 13260 raft_consensus.cc:1217] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:24.995368 13260 raft_consensus.cc:1217] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:25.331216 13520 raft_consensus.cc:1064] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:25.336953 13520 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:25.358985 13260 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.363653 12569 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.365481 13520 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:25.368211 13441 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:25.369874 12494 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.377276 13438 raft_consensus.cc:1064] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:25.380378 13536 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:25.380380 13438 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:25.403195 13262 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.406127 13410 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:25.415831 13441 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.418361 12492 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.420010 13438 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:25.431058 12644 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.438169 12569 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.438398 13260 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.459276 13410 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:25.462817 12394 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:25.468436 13507 raft_consensus.cc:1064] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:25.471988 13507 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:25.490898 13410 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.506059 12493 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.507688 13262 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.572306 13261 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:11:25.574694 12646 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:11:25.578691 12494 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:11:25.584914 13438 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:25.586464 13542 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:25.587201 13441 raft_consensus.cc:1064] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:25.588418 13507 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:25.590009 13441 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:25.592900 12646 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.593971 12493 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:11:25.651533 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69332.3us
I20260709 12:11:25.714604 12492 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.753127 13262 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.755167 12392 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.759471 13539 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:11:25.803551 12566 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:25.807902 13410 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.811750 13259 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.820823 13539 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:25.823397 12643 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:25.833519 13260 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:25.837083 12491 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:25.838698 13520 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:25.845234 13567 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:25.852521 13441 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.841190 13520 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:25.863224 13567 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.864522 13260 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.876168 12494 raft_consensus.cc:2955] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.878376 12493 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.882350 13262 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.895681 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a3a0781287514c758f107d79c1f96be4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:25.908033 12646 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.910023 12563 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:25.915916 12392 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.919064 12395 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.935227 12494 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.945955 12566 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:25.969112 13259 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:25.973171 13410 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:25.973030 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet a3a0781287514c758f107d79c1f96be4 with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:25.991109 13574 tablet_replica.cc:333] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:25.993836 13574 raft_consensus.cc:2243] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:25.995474 13574 raft_consensus.cc:2272] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:26.004606 12643 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:26.011565 13507 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.013250 13574 ts_tablet_manager.cc:1916] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:26.014075 13260 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.007695 12395 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.037935 12643 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:26.038616 13562 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:26.043656 12563 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:26.052599 12646 raft_consensus.cc:2955] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.058218 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5d0dce81bc984a0690d08335a4937b2c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:26.059850 13260 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:11:26.059546 12393 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.063944 13562 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:11:26.070976 13260 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:26.074612 12493 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:11:26.076761 13568 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:26.079753 12563 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:26.083916 13568 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:26.087213 13543 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:26.088735 13574 ts_tablet_manager.cc:1929] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:26.089865 13574 log.cc:1199] T a3a0781287514c758f107d79c1f96be4 P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/a3a0781287514c758f107d79c1f96be4
I20260709 12:11:26.090124 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d0dce81bc984a0690d08335a4937b2c with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:26.092989 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet a3a0781287514c758f107d79c1f96be4 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:26.098584 13520 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.116237 13410 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.118806 13260 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.124786 13575 tablet_replica.cc:333] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:26.126113 13575 raft_consensus.cc:2243] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:26.137866 13575 raft_consensus.cc:2272] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:26.126638 13259 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.139137 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6f701a23559945868ce6e3da5866f95c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:26.131536 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c13f916d8b4d463182a1d1eb50150e31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:26.146189 13575 ts_tablet_manager.cc:1916] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:26.157186 12393 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:26.157451 12493 raft_consensus.cc:2955] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.204759 12392 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:26.224141 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f701a23559945868ce6e3da5866f95c with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:26.228210 13575 ts_tablet_manager.cc:1929] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:26.228856 13575 log.cc:1199] T 5d0dce81bc984a0690d08335a4937b2c P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/5d0dce81bc984a0690d08335a4937b2c
I20260709 12:11:26.206410 12563 raft_consensus.cc:2955] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:26.230935 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet 5d0dce81bc984a0690d08335a4937b2c (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:26.237131 13576 tablet_replica.cc:333] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:26.238492 13576 raft_consensus.cc:2243] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:26.239137 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet c13f916d8b4d463182a1d1eb50150e31 with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:26.245306 13575 tablet_replica.cc:333] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:26.246934 13575 raft_consensus.cc:2243] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:26.247884 13575 raft_consensus.cc:2272] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:26.249197 13576 raft_consensus.cc:2272] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:26.253440 13576 ts_tablet_manager.cc:1916] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:26.254459 13575 ts_tablet_manager.cc:1916] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:26.278597 13575 ts_tablet_manager.cc:1929] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:26.279126 13575 log.cc:1199] T c13f916d8b4d463182a1d1eb50150e31 P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/c13f916d8b4d463182a1d1eb50150e31
I20260709 12:11:26.281093 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet c13f916d8b4d463182a1d1eb50150e31 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:26.291621 13576 ts_tablet_manager.cc:1929] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:11:26.292182 13576 log.cc:1199] T 6f701a23559945868ce6e3da5866f95c P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/6f701a23559945868ce6e3da5866f95c
I20260709 12:11:26.293982 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 6f701a23559945868ce6e3da5866f95c (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:27.341789 12646 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:27.349330 12563 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:27.350801 12611 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c56dde469374f4da7440f134140670c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:27.352598 13567 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:27.354600 12491 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:27.356510 13567 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:27.369174 12566 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.367447 13520 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.370550 12493 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.391492 12393 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.396565 12491 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:11:27.408131 12462 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 509f0430baf04e3393879d82fb9d10f9. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:27.412299 12563 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:27.413133 12646 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:27.414431 13559 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:27.416221 13503 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:27.435277 13559 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.436749 12563 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.454298 12643 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.471488 12391 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.488452 12643 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:11:27.496574 12611 consensus_peers.cc:597] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3648047ca0742a2bfaa45ef2af517e7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:27.496806 12491 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:27.500041 13520 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:27.502661 12563 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:27.505049 13567 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:27.516366 13520 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.518422 12491 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.520399 12563 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.532001 12643 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:27.535895 12395 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:27.566423 12491 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:27.568552 12611 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 472e786b46074d868d00df67a0985afd. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:27.569136 13520 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:27.570767 12563 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:27.572582 13520 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:27.593951 12494 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.591728 13520 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.603089 12563 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.614243 12395 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:27.831457 13598 ts_tablet_manager.cc:933] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:27.833382 13598 tablet_copy_client.cc:323] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:11:27.835481 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 509f0430baf04e3393879d82fb9d10f9 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:11:27.836053 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 509f0430baf04e3393879d82fb9d10f9 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-509f0430baf04e3393879d82fb9d10f9
I20260709 12:11:27.844436 12507 tablet_copy_source_session.cc:215] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:27.848091 13598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509f0430baf04e3393879d82fb9d10f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:27.867069 13598 tablet_copy_client.cc:806] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:27.867810 13598 tablet_copy_client.cc:670] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:27.872682 13598 tablet_copy_client.cc:538] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:27.882738 13598 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:27.909611 13598 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:27.910960 13598 tablet_bootstrap.cc:492] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:27.911757 13598 ts_tablet_manager.cc:1403] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.029s	user 0.017s	sys 0.008s
I20260709 12:11:27.914786 13598 raft_consensus.cc:359] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:27.915710 13598 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:27.916415 13598 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:27.933139 13598 ts_tablet_manager.cc:1434] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.021s	user 0.011s	sys 0.008s
I20260709 12:11:27.935828 13602 ts_tablet_manager.cc:933] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:27.936704 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-509f0430baf04e3393879d82fb9d10f9 received from {username='slave'} at 127.0.0.1:54234
I20260709 12:11:27.937431 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-509f0430baf04e3393879d82fb9d10f9 on tablet 509f0430baf04e3393879d82fb9d10f9 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:27.943113 13602 tablet_copy_client.cc:323] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:27.946408 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 0c56dde469374f4da7440f134140670c from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:27.947469 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 0c56dde469374f4da7440f134140670c from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-0c56dde469374f4da7440f134140670c
I20260709 12:11:27.958186 12656 tablet_copy_source_session.cc:215] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:27.963529 13602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c56dde469374f4da7440f134140670c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:27.986835 13602 tablet_copy_client.cc:806] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:27.987613 13602 tablet_copy_client.cc:670] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:27.993551 13602 tablet_copy_client.cc:538] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:28.009864 13602 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:28.043663 13598 ts_tablet_manager.cc:933] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:28.045356 13602 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:28.046473 13602 tablet_bootstrap.cc:492] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:28.047204 13602 ts_tablet_manager.cc:1403] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.038s	user 0.020s	sys 0.004s
I20260709 12:11:28.050729 13602 raft_consensus.cc:359] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:28.051651 13602 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:28.052362 13602 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:28.056901 13602 ts_tablet_manager.cc:1434] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:11:28.059247 12656 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-0c56dde469374f4da7440f134140670c received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:28.059777 12656 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-0c56dde469374f4da7440f134140670c on tablet 0c56dde469374f4da7440f134140670c with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:28.062532 13598 tablet_copy_client.cc:323] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:28.068056 13606 ts_tablet_manager.cc:933] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:28.068377 12656 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 472e786b46074d868d00df67a0985afd from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:28.069005 12656 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 472e786b46074d868d00df67a0985afd from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-472e786b46074d868d00df67a0985afd
I20260709 12:11:28.071169 13606 tablet_copy_client.cc:323] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:28.075764 12655 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet e3648047ca0742a2bfaa45ef2af517e7 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:28.076395 12655 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet e3648047ca0742a2bfaa45ef2af517e7 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-e3648047ca0742a2bfaa45ef2af517e7
I20260709 12:11:28.091599 12656 tablet_copy_source_session.cc:215] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:28.092581 12655 tablet_copy_source_session.cc:215] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:28.095708 13598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472e786b46074d868d00df67a0985afd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:28.099009 13606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3648047ca0742a2bfaa45ef2af517e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:28.118796 13606 tablet_copy_client.cc:806] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:28.119405 13606 tablet_copy_client.cc:670] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:28.122611 13598 tablet_copy_client.cc:806] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:28.122952 13606 tablet_copy_client.cc:538] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:28.123850 13598 tablet_copy_client.cc:670] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:28.128429 13598 tablet_copy_client.cc:538] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:28.135064 13606 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:28.141873 13598 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:28.205114 13598 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:28.207501 13598 tablet_bootstrap.cc:492] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:28.208328 13598 ts_tablet_manager.cc:1403] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.067s	user 0.029s	sys 0.006s
I20260709 12:11:28.213382 13606 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:28.214473 13606 tablet_bootstrap.cc:492] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:28.215181 13606 ts_tablet_manager.cc:1403] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.080s	user 0.040s	sys 0.022s
I20260709 12:11:28.217516 13598 raft_consensus.cc:359] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:28.218583 13598 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:28.218950 13606 raft_consensus.cc:359] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:28.219867 13606 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:28.219338 13598 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:28.220572 13606 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:28.223585 13606 ts_tablet_manager.cc:1434] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:11:28.225519 12655 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-e3648047ca0742a2bfaa45ef2af517e7 received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:28.225996 12655 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-e3648047ca0742a2bfaa45ef2af517e7 on tablet e3648047ca0742a2bfaa45ef2af517e7 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:28.228886 13598 ts_tablet_manager.cc:1434] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.004s
I20260709 12:11:28.230999 12655 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-472e786b46074d868d00df67a0985afd received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:28.231673 12655 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-472e786b46074d868d00df67a0985afd on tablet 472e786b46074d868d00df67a0985afd with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:28.431936 13259 raft_consensus.cc:1217] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:28.494233 13259 raft_consensus.cc:1217] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:28.540292 13259 raft_consensus.cc:1217] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:28.550786 13259 raft_consensus.cc:1217] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:29.012003 13520 raft_consensus.cc:1064] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:29.014770 13520 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.027115 13260 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.029728 13612 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.028333 12563 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.033649 12491 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.035367 13520 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:29.040910 13520 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.046540 13567 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.056169 13559 raft_consensus.cc:1064] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:29.061940 13520 raft_consensus.cc:1064] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:29.062817 12494 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.059630 13559 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.069170 13520 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.089907 13260 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.092118 12563 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.095563 13590 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.097206 13589 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.131278 13259 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.142886 12391 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:29.163919 12566 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.177889 12643 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.209824 12494 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.211846 12563 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.211983 13618 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.213578 13612 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.214548 13621 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.234292 12568 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.243738 13590 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.257496 13260 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.266391 13567 raft_consensus.cc:1064] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:29.269541 13567 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.269941 13612 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.275677 13617 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.278214 13259 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.293258 12568 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.290653 12494 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.331935 12644 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.341126 13260 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.348742 12568 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.352862 13612 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:29.355458 12492 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.358327 13617 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.361202 13259 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:29.372090 13617 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:29.401289 13567 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.413697 12494 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.414220 13260 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.415491 12393 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.417863 12393 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.435752 12563 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.450152 12493 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.457522 13259 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.460028 13589 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:29.469271 13559 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.483671 12646 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.489847 13261 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.492719 12392 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.495779 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 509f0430baf04e3393879d82fb9d10f9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:29.495854 12563 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.504365 13628 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:29.504515 13259 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.510236 12568 raft_consensus.cc:2955] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.511759 13618 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:29.518088 12566 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.521997 13617 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:29.553436 13617 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.575055 12644 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.576892 12563 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.579075 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e3648047ca0742a2bfaa45ef2af517e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:29.584584 12393 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:29.586236 12493 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.591215 13617 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:29.600301 13612 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.603577 12493 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.616825 13259 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.627919 13261 raft_consensus.cc:2955] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.631879 13567 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:11:29.636499 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 509f0430baf04e3393879d82fb9d10f9 with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:29.644369 13260 raft_consensus.cc:2955] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.662179 12395 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.674402 12393 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:29.677327 13639 tablet_replica.cc:333] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:29.678711 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 0c56dde469374f4da7440f134140670c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:29.696609 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet e3648047ca0742a2bfaa45ef2af517e7 with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:29.697875 13639 raft_consensus.cc:2243] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:29.699242 13640 tablet_replica.cc:333] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:29.699926 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c56dde469374f4da7440f134140670c with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:29.702263 13640 raft_consensus.cc:2243] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:29.700071 13639 raft_consensus.cc:2272] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:29.716426 13640 raft_consensus.cc:2272] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:29.719962 13641 tablet_replica.cc:333] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:29.721560 13641 raft_consensus.cc:2243] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:29.721975 13640 ts_tablet_manager.cc:1916] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:29.722817 13641 raft_consensus.cc:2272] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:29.723973 13639 ts_tablet_manager.cc:1916] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:29.795852 13641 ts_tablet_manager.cc:1916] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:29.801209 13640 ts_tablet_manager.cc:1929] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:29.801795 13640 log.cc:1199] T e3648047ca0742a2bfaa45ef2af517e7 P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/e3648047ca0742a2bfaa45ef2af517e7
I20260709 12:11:29.801381 13639 ts_tablet_manager.cc:1929] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:29.803685 13639 log.cc:1199] T 509f0430baf04e3393879d82fb9d10f9 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/509f0430baf04e3393879d82fb9d10f9
I20260709 12:11:29.803651 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet e3648047ca0742a2bfaa45ef2af517e7 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:29.810413 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 509f0430baf04e3393879d82fb9d10f9 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
W20260709 12:11:29.817382 13612 consensus_queue.cc:1175] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20260709 12:11:29.818636 12644 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:29.831207 13261 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.834059 12493 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:29.835901 13567 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:29.837240 13612 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:29.849251 13568 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.852070 13641 ts_tablet_manager.cc:1929] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:29.852834 13641 log.cc:1199] T 0c56dde469374f4da7440f134140670c P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/0c56dde469374f4da7440f134140670c
I20260709 12:11:29.854801 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet 0c56dde469374f4da7440f134140670c (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:29.861451 13261 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.865757 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 472e786b46074d868d00df67a0985afd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:29.871423 12493 raft_consensus.cc:2955] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:29.874857 12392 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:29.907550 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 472e786b46074d868d00df67a0985afd with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:29.908699 13641 tablet_replica.cc:333] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:29.927310 13641 raft_consensus.cc:2243] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:29.928418 13641 raft_consensus.cc:2272] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:29.932861 13641 ts_tablet_manager.cc:1916] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:29.954377 13641 ts_tablet_manager.cc:1929] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:29.954813 13641 log.cc:1199] T 472e786b46074d868d00df67a0985afd P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/472e786b46074d868d00df67a0985afd
I20260709 12:11:29.956727 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet 472e786b46074d868d00df67a0985afd (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:31.402398 12563 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:31.411120 12646 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:31.411243 12492 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:31.411285 12537 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54875fd261ee4c328e3d022d6a9418d8. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:31.412652 13643 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.413546 13611 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.423002 13643 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.424829 12492 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.425940 12646 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.441836 12638 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:31.444160 12393 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:11:31.454507 12611 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c793ebd4de24492397303f9a6e6bc62a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:31.457937 12492 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:31.458233 12563 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:31.459685 13646 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:31.460723 13655 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.475332 13651 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.480706 12566 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.486045 12492 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.501251 12492 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:31.509563 12638 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:11:31.510198 12462 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43fecbd2edc14ba2ab065395e5d48d1d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:31.508790 12395 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.511353 13650 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.512621 12566 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:31.514189 13650 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.522603 13628 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.527276 12638 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.527899 12566 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.539783 12563 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:31.543486 12392 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:11:31.549010 12537 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1f18763e7844e0994e2f11a6dd89290. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:31.550771 12638 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:31.554159 12492 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:31.554805 13643 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.555950 13653 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:31.569792 13653 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.571894 12492 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.574714 12638 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:31.606544 12395 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:31.848742 13673 ts_tablet_manager.cc:933] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:31.850734 13673 tablet_copy_client.cc:323] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:11:31.852850 12655 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet c793ebd4de24492397303f9a6e6bc62a from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:11:31.853468 12655 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet c793ebd4de24492397303f9a6e6bc62a from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-c793ebd4de24492397303f9a6e6bc62a
I20260709 12:11:31.867040 12655 tablet_copy_source_session.cc:215] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:31.870936 13673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c793ebd4de24492397303f9a6e6bc62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:31.900779 13673 tablet_copy_client.cc:806] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:31.901449 13673 tablet_copy_client.cc:670] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:31.908260 13673 tablet_copy_client.cc:538] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:31.918514 13675 ts_tablet_manager.cc:933] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:31.920763 13673 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:31.923452 13675 tablet_copy_client.cc:323] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:31.926582 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 54875fd261ee4c328e3d022d6a9418d8 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:31.927337 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 54875fd261ee4c328e3d022d6a9418d8 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-54875fd261ee4c328e3d022d6a9418d8
I20260709 12:11:31.937549 12581 tablet_copy_source_session.cc:215] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:31.941999 13675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54875fd261ee4c328e3d022d6a9418d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:31.945269 13673 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:31.946362 13673 tablet_bootstrap.cc:492] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:31.947024 13673 ts_tablet_manager.cc:1403] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.027s	user 0.022s	sys 0.004s
I20260709 12:11:31.951504 13673 raft_consensus.cc:359] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:31.952498 13673 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:31.953243 13673 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:31.959774 13673 ts_tablet_manager.cc:1434] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20260709 12:11:31.964152 13675 tablet_copy_client.cc:806] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:31.964850 13675 tablet_copy_client.cc:670] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:31.964171 12655 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-c793ebd4de24492397303f9a6e6bc62a received from {username='slave'} at 127.0.0.1:35404
I20260709 12:11:31.966377 12655 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-c793ebd4de24492397303f9a6e6bc62a on tablet c793ebd4de24492397303f9a6e6bc62a with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:31.970219 13675 tablet_copy_client.cc:538] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:31.984965 13675 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:32.010023 13675 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:32.010990 13675 tablet_bootstrap.cc:492] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:32.011687 13675 ts_tablet_manager.cc:1403] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.027s	user 0.027s	sys 0.000s
I20260709 12:11:32.014849 13675 raft_consensus.cc:359] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:32.015733 13675 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:32.016412 13675 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:32.021590 13675 ts_tablet_manager.cc:1434] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:11:32.023837 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-54875fd261ee4c328e3d022d6a9418d8 received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:32.024405 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-54875fd261ee4c328e3d022d6a9418d8 on tablet 54875fd261ee4c328e3d022d6a9418d8 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:32.061579 13675 ts_tablet_manager.cc:933] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:32.063571 13675 tablet_copy_client.cc:323] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:11:32.065310 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 43fecbd2edc14ba2ab065395e5d48d1d from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:11:32.065953 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 43fecbd2edc14ba2ab065395e5d48d1d from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-43fecbd2edc14ba2ab065395e5d48d1d
I20260709 12:11:32.074764 12507 tablet_copy_source_session.cc:215] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:32.078095 13675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43fecbd2edc14ba2ab065395e5d48d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:32.097324 13675 tablet_copy_client.cc:806] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:32.098142 13675 tablet_copy_client.cc:670] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:32.102984 13675 tablet_copy_client.cc:538] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:32.117712 13675 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:32.145159 13675 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:32.146097 13675 tablet_bootstrap.cc:492] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:32.146829 13675 ts_tablet_manager.cc:1403] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.012s
I20260709 12:11:32.151886 13675 raft_consensus.cc:359] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:32.152824 13675 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:32.153477 13675 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:32.155864 13675 ts_tablet_manager.cc:1434] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.004s
I20260709 12:11:32.158398 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-43fecbd2edc14ba2ab065395e5d48d1d received from {username='slave'} at 127.0.0.1:54234
I20260709 12:11:32.158838 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-43fecbd2edc14ba2ab065395e5d48d1d on tablet 43fecbd2edc14ba2ab065395e5d48d1d with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:32.164844 13673 ts_tablet_manager.cc:933] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:32.182457 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6060 after lost signal to thread 11313
W20260709 12:11:32.183818 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6740 after lost signal to thread 12429
W20260709 12:11:32.185197 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080033ebe0 after lost signal to thread 12521
W20260709 12:11:32.186853 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08003374c0 after lost signal to thread 12596
W20260709 12:11:32.188103 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdec0 after lost signal to thread 12670
W20260709 12:11:32.189411 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002b1860 after lost signal to thread 13286
I20260709 12:11:32.208788 13673 tablet_copy_client.cc:323] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:32.224020 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet e1f18763e7844e0994e2f11a6dd89290 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:32.224771 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet e1f18763e7844e0994e2f11a6dd89290 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-e1f18763e7844e0994e2f11a6dd89290
I20260709 12:11:32.242170 12581 tablet_copy_source_session.cc:215] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:32.254082 13673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1f18763e7844e0994e2f11a6dd89290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:32.305646 13257 raft_consensus.cc:1217] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:32.314330 13673 tablet_copy_client.cc:806] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:32.339969 13673 tablet_copy_client.cc:670] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:32.344992 13673 tablet_copy_client.cc:538] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:32.356479 13673 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:32.385810 13673 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:32.386739 13673 tablet_bootstrap.cc:492] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:32.387391 13673 ts_tablet_manager.cc:1403] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.031s	user 0.015s	sys 0.016s
I20260709 12:11:32.390420 13673 raft_consensus.cc:359] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:32.391251 13673 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:32.391920 13673 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:32.401995 13673 ts_tablet_manager.cc:1434] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.002s
I20260709 12:11:32.403985 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-e1f18763e7844e0994e2f11a6dd89290 received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:32.404454 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-e1f18763e7844e0994e2f11a6dd89290 on tablet e1f18763e7844e0994e2f11a6dd89290 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:32.424181 13257 raft_consensus.cc:1217] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:32.588874 13259 raft_consensus.cc:1217] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:11:32.646595 13257 raft_consensus.cc:1217] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:11:32.849972 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:32.857065 13653 raft_consensus.cc:1064] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:32.860153 13653 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:32.874898 13257 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:32.876739 12492 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:32.878700 13653 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:32.880198 13685 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:32.931479 12646 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:32.937898 13683 raft_consensus.cc:1064] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:32.940621 13683 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:32.970641 13257 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:32.973308 13651 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:32.974301 12564 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:32.977761 13563 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:32.986644 13651 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:32.990293 13650 raft_consensus.cc:1064] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:32.993348 13650 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:32.999243 13687 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.042760 12494 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.045298 13651 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.058439 12492 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:33.064764 13257 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.072659 13259 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.078558 12393 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.083606 12646 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.090296 13683 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:33.091367 13258 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:33.092911 12638 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:33.094882 13649 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:33.104588 13649 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:33.106530 12564 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.106881 12395 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.153350 12564 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:11:33.156587 13650 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:33.171116 13649 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.164595 12564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:33.176821 12564 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:11:33.184340 12638 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.209677 12563 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.209681 12564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:33.216382 12564 raft_consensus.cc:606] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:33.279407 12493 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:11:33.333704 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92101.7us
W20260709 12:11:33.335819 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67961.2us
W20260709 12:11:33.338203 13704 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:11:33.417893 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 186142us
I20260709 12:11:33.494511 13687 raft_consensus.cc:1064] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:11:33.517783 13687 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:33.493465 13258 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.579555 12563 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:33.594937 13653 raft_consensus.cc:993] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:33.595959 13653 raft_consensus.cc:1081] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:33.633574 13257 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:33.637212 13257 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:33.637890 13257 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:33.640189 13643 raft_consensus.cc:993] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:33.641003 13685 raft_consensus.cc:1081] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:33.641069 12391 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:33.644304 13261 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:33.669270 12497 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:34492
I20260709 12:11:33.671233 12497 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:33.671734 12497 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:33.687148 12491 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:33.712311 12643 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:33.713296 12643 raft_consensus.cc:606] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:33.674503 13653 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 12:11:33.742053 12497 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:33.750492 13685 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:11:33.800001 13257 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:33.796072 12497 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:33.814920 13259 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:11:33.816387 12563 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:33.817250 12563 raft_consensus.cc:606] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:33.823097 13718 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.824800 12489 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:11:33.832993 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116402us
I20260709 12:11:33.832522 13687 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:33.834985 13685 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:33.842296 13259 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.850596 12566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:33.850430 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:33.851495 12569 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:33.851375 12566 raft_consensus.cc:606] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:11:33.852022 12569 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:33.861685 12569 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:33.863255 12569 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:33.870635 12497 raft_consensus.cc:2955] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:33.883208 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:33.884018 12643 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:33.919088 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:33.925686 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:33.926509 12489 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:33.927583 13257 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:33.929144 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:33.929875 12564 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:33.930187 12564 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:33.931056 12564 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:33.932394 12564 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:33.940812 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 54875fd261ee4c328e3d022d6a9418d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:33.962620 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:33.962620 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:33.966328 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:33.943387 12569 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:33.970734 13678 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:33.981492 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:33.990765 13711 raft_consensus.cc:993] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:33.998941 13651 raft_consensus.cc:1081] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:34.003113 13721 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:34.021003 12569 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32884
I20260709 12:11:34.023571 12569 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:34.024051 12569 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.018860 13678 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:34.011340 13721 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:34.036089 12642 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:11:34.050657 13721 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.026216 13678 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.078949 12569 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.080957 12569 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:34.080749 13643 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:34.088140 13653 raft_consensus.cc:993] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:34.088773 13653 raft_consensus.cc:1081] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:34.092900 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:34.093531 12497 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.106995 12489 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:34492
I20260709 12:11:34.107849 12489 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:34.108340 12489 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.129583 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:11:34.130514 12492 raft_consensus.cc:606] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:34.129752 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:34.131487 13687 raft_consensus.cc:993] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:34.131620 12645 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:34.132252 12645 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:34.132311 13687 raft_consensus.cc:1081] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:34.139000 13653 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:34.141911 12497 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:34.151666 13258 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:34.152449 13258 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:34.152878 13258 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.133167 12645 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.157804 12645 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.170091 13261 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:34.183140 12492 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.195354 12569 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:34.196265 12569 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:11:34.205757 12645 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:34.227778 13724 raft_consensus.cc:993] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:34.226338 13258 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.227914 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260709 12:11:34.196700 12489 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.230412 12393 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:34.239104 12569 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:34.256901 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:11:34.267328 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:34.268466 12569 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:34.269112 12569 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:34.271499 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:34.272416 12637 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.249900 13258 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:34.273600 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:34.274607 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129717us
I20260709 12:11:34.275620 12645 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.273209 12489 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:34.278152 13563 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:34.278816 13563 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:34.281358 13261 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:34.296784 13714 raft_consensus.cc:1081] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:34.298475 13563 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.305054 13710 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260709 12:11:34.304517 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:34.297863 12491 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:34.307904 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:34.311767 12566 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:34.311947 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:34.312335 12568 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:11:34.312826 12640 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.312985 12568 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:34.313532 12568 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:34.312163 12566 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:34.324132 12566 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.330623 12569 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.335548 12644 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:34.336351 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:34.338792 13678 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:34.339510 13678 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:34.340692 12640 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:34.342362 12566 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.343708 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:34.349161 13719 raft_consensus.cc:993] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:34.350756 13724 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:34.351256 13719 raft_consensus.cc:1081] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:34.351542 13724 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:34.351441 13720 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 12:11:34.340750 13678 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.378454 12569 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.376559 12568 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.353127 13724 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.411870 12492 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:34492
I20260709 12:11:34.412544 12492 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:34.412954 12492 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:34.411975 12568 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:34.417270 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:34.418013 12640 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:34.407745 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84575.8us
W20260709 12:11:34.426262 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82784.4us
I20260709 12:11:34.379794 12646 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:34.462283 12566 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:34.471303 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:34.479115 12640 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:11:34.516294 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76739.5us
I20260709 12:11:34.523963 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:34.559319 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74880.2us
I20260709 12:11:34.559263 13718 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:34.571604 13718 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:34.578672 13718 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.544557 13721 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:11:34.584869 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 94753.1us
I20260709 12:11:34.585361 13726 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:34.586030 13726 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:34.590725 12492 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:34.606729 12492 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:34.615013 12569 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:34.617172 13678 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:11:34.632000 13726 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:34.635651 13258 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:34.640030 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:34.642545 13713 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:11:34.663856 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:34.820724 12379 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 54875fd261ee4c328e3d022d6a9418d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20260709 12:11:35.189908 12494 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:35.195065 13685 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260709 12:11:34.578884 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:35.224321 12393 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 214) took 1020 ms. Trace:
I20260709 12:11:35.224789 12393 rpcz_store.cc:276] 0709 12:11:34.204254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.204432 (+   178us) service_pool.cc:224] Handling call
0709 12:11:35.224244 (+1019812us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:34.391200 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:34.410658 (+ 19458us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:34.410699 (+    41us) write_op.cc:435] Acquired schema lock
0709 12:11:34.410734 (+    35us) tablet.cc:662] Decoding operations
0709 12:11:34.417156 (+  6422us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:34.417255 (+    99us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:34.417297 (+    42us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:34.417550 (+   253us) tablet.cc:701] Row locks acquired
0709 12:11:34.417570 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:11:34.417797 (+   227us) write_op.cc:270] Start()
0709 12:11:34.417963 (+   166us) write_op.cc:276] Timestamp: P: 1783599094417722 usec, L: 0
0709 12:11:34.417994 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:11:34.450883 (+ 32889us) log.cc:844] Serialized 535 byte log entry
0709 12:11:34.453028 (+  2145us) op_driver.cc:464] REPLICATION: finished
0709 12:11:34.453521 (+   493us) write_op.cc:301] APPLY: starting
0709 12:11:34.453605 (+    84us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:34.453754 (+   149us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:34.453884 (+   130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:34.454195 (+   311us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:34.454367 (+   172us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:34.454420 (+    53us) write_op.cc:312] APPLY: finished
0709 12:11:34.460363 (+  5943us) log.cc:844] Serialized 22 byte log entry
0709 12:11:34.485573 (+ 25210us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:34.497504 (+ 11931us) write_op.cc:454] Released schema lock
0709 12:11:34.499209 (+  1705us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:34.592950 (+ 93741us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eee28 00007ffbf9229581 00007ffbf9228d58 00007ffbf922d9f1 00007ffbf922d9a9 00007ffbf922d939 00007ffbf922d901 00007ffbf922cb2d 0000000000388634 0000000000388479
0709 12:11:34.593170 (+   220us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:34.593206 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":257,"apply.run_cpu_time_us":2201,"apply.run_wall_time_us":63497,"num_ops":1,"prepare.queue_time_us":138867,"prepare.run_cpu_time_us":8409,"prepare.run_wall_time_us":201963,"replication_time_us":34875,"spinlock_wait_cycles":5311872,"thread_start_us":27681,"threads_started":1}]]}
I20260709 12:11:35.244729 12566 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:35.248698 13745 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:35.249995 12566 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:35.253931 13745 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:35.260098 12493 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:35.279273 13757 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:11:35.282936 13757 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:35.283396 13757 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:35.288492 13757 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.291756 13757 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:35.329852 12570 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:35.336172 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:35.336838 13745 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:35.338402 13745 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:11:35.339171 13745 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:35.340530 13745 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:35.341960 13745 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.355450 13745 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:35.364831 12391 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 2 to 3, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:35.370927 12565 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:35.370692 12392 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:35.390599 12564 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:35.416483 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:35.417850 12489 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:11:35.444564 13711 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:35.448174 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:35.450320 13722 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:35.450898 13722 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:35.451884 13722 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.459990 13256 raft_consensus.cc:2955] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:35.472378 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:35.478760 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1218) took 1028 ms. Trace:
I20260709 12:11:35.479306 12491 rpcz_store.cc:276] 0709 12:11:34.441060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.441487 (+   427us) service_pool.cc:224] Handling call
0709 12:11:35.468969 (+1027482us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:35.469177 (+   208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:35.469348 (+   171us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:35.469371 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:35.469408 (+    37us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:35.469434 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:35.469489 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:35.469580 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42693376}
W20260709 12:11:35.464938 13643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.011s	sys 0.011s
I20260709 12:11:35.482833 12497 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:35.492687 12492 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:35.492497 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1219) took 1037 ms. Trace:
W20260709 12:11:35.490474 13643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.011s	sys 0.012s
I20260709 12:11:35.499202 12497 rpcz_store.cc:276] 0709 12:11:34.452688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.452855 (+   167us) service_pool.cc:224] Handling call
0709 12:11:35.472185 (+1019330us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:35.490370 (+ 18185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16426240}
I20260709 12:11:35.445343 13711 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:35.513250 13711 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:35.525795 12640 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:35.538182 13656 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:35.558050 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:35.561795 13757 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.562559 13757 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.564904 13757 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.565609 12609 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:35.566735 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:35.583405 13770 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:35.584426 12497 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:35.565444 13757 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.538260 13723 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.539139 13656 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.589546 13717 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:35.593212 13656 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.589800 13651 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:35.609702 13651 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.612918 13651 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:35.581468 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80664.7us
I20260709 12:11:35.671023 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:35.685511 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:35.590914 13649 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.590060 13711 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:35.686260 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:35.688369 13649 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:35.687513 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:35.705983 13756 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:35.706951 13756 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.601296 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:35.589003 13718 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.101s
I20260709 12:11:35.714704 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:35.686728 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:35.588755 13757 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:35.589553 13766 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.196s
I20260709 12:11:35.602521 13717 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:35.736330 13775 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.002s	sys 0.000s
I20260709 12:11:35.590772 13723 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.704768 13711 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:35.741663 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08000673a0 after lost signal to thread 11313
I20260709 12:11:35.745177 13751 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:35.599264 13726 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.147s
I20260709 12:11:35.745398 13756 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.751139 13771 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.746167 13751 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.735301 13721 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.752147 13771 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.770231 13714 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.769482 13721 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:35.741722 13649 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:35.782395 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 54875fd261ee4c328e3d022d6a9418d8 with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
W20260709 12:11:35.780676 12426 debug-util.cc:398] Leaking SignalData structure 0x7b0800067a80 after lost signal to thread 12429
I20260709 12:11:35.801524 13752 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.772140 13714 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.802501 13752 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.831799 13723 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.834571 13757 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:35.852229 13714 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.833361 13717 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:35.857986 13751 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.809808 13711 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:35.863245 13721 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:35.864980 13771 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.868110 13752 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.870405 13766 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:35.874997 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:35.878357 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080033b420 after lost signal to thread 12521
I20260709 12:11:35.878472 12565 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
W20260709 12:11:35.878916 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080033b2c0 after lost signal to thread 12522
I20260709 12:11:35.878801 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:35.879549 12563 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:35.918833 13721 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.919665 13721 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.922209 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:35.926461 13649 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:35.927007 13649 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:35.927475 13649 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:35.936851 12426 debug-util.cc:398] Leaking SignalData structure 0x7b0800348b00 after lost signal to thread 12596
I20260709 12:11:35.947135 13721 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:35.956271 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:35.982923 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 298399us
W20260709 12:11:35.984130 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 282009us
I20260709 12:11:35.984450 13726 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:35.984999 13726 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.987605 13726 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:35.988109 13711 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:35.988766 13711 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:35.991542 13711 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:35.996160 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 43fecbd2edc14ba2ab065395e5d48d1d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:35.987936 13774 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.010640 12563 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:11:36.014081 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 323570us
I20260709 12:11:36.022399 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:35.979846 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 311340us
I20260709 12:11:36.020851 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.027237 13772 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.028645 13770 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.029668 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.036007 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:36.018474 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080034eea0 after lost signal to thread 12670
I20260709 12:11:36.061007 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:36.048941 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080034eee0 after lost signal to thread 12671
W20260709 12:11:36.064456 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080034e580 after lost signal to thread 13286
W20260709 12:11:36.064847 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080034e540 after lost signal to thread 13287
I20260709 12:11:36.073835 13769 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.079633 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:36.083267 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:36.078344 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.080444 12642 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:36.090348 12637 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:36.108811 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.114677 13784 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:36.116793 13783 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:35.944972 13767 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:36.048156 13770 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.024544 13774 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.139099 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.131395 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.155984 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.130676 13766 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:36.174829 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.176227 12637 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:36.176301 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.088601 13769 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.177423 13259 raft_consensus.cc:1813] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Handling vote request from an unknown peer 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:36.185926 13259 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:11:36.183969 12643 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:36.184849 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.189939 13261 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:36.196689 13710 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:36.197436 13710 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.200278 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:36.191515 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 175813us
I20260709 12:11:36.117704 13783 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.203614 13781 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.203750 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:36.173341 13756 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.208159 12609 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.204509 13770 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.115708 13784 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.209949 12638 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:36.183832 13714 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:36.187098 13752 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:36.227442 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.227509 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:36.251827 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43909.5us
I20260709 12:11:36.208338 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.183468 13774 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.141575 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:36.262941 13780 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:11:36.094543 13772 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.258805 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.288341 13789 tablet_replica.cc:333] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:36.289233 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.180635 13766 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.268343 13713 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.296406 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.302765 13683 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.302928 13788 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:36.263010 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.272661 13774 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.257222 13752 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.254887 13714 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.313279 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.320173 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.312575 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.303578 13683 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.348022 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.348934 13713 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.349231 13726 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:36.303557 13788 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.348271 13766 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.262425 13786 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.267333 13801 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.280725 13779 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.397953 13778 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.418898 13796 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.419803 13796 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.358078 13788 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:36.327960 13774 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:36.453943 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 297105us
W20260709 12:11:36.461506 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44413.8us
I20260709 12:11:36.462728 12536 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.414875 13786 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.245268 13781 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.473019 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.349481 13713 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.491932 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.361543 13726 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.402505 13766 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 12:11:36.495163 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.498958 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.425170 13649 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.488803 13744 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.445791 13801 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.527045 12493 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:36.505221 13723 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.487926 13771 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 12:11:36.531822 13796 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.532578 13796 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.532820 13746 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:36.513295 13787 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.533098 13796 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.536180 13795 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.533427 13746 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.513383 13783 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.536044 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.506580 13683 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.512030 13784 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.539328 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.444128 13778 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.459440 13779 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.558348 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.561728 12638 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:36.563592 12643 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:36.563851 13803 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:36.551751 13746 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:36.564689 13803 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.565474 13769 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.538834 13795 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.586583 13796 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.602262 13786 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.613106 13259 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:36.621091 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:36.595494 13714 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:36.592061 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.628636 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.630290 13779 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:36.633200 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.643105 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.591786 13772 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.649207 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:36.625109 13774 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.662981 13792 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:36.670500 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.671288 13710 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:36.683728 13803 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.661502 13649 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.666759 13779 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:36.694303 13801 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.707854 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.663826 13792 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:36.719285 13795 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.691334 13713 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.723551 13778 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.719375 13781 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:36.701576 13779 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.755414 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.753295 13651 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:36.776275 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.788396 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.791078 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.794302 13802 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.794535 13792 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:36.794647 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.796010 13683 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:11:36.797034 13767 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.642s	user 0.007s	sys 0.007s
I20260709 12:11:36.789944 13715 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:36.786970 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.007s	sys 0.010s
W20260709 12:11:36.802738 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.007s	sys 0.010s
W20260709 12:11:36.803081 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 539557us
I20260709 12:11:36.806604 13752 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:36.806345 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.811892 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.819247 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:36.832898 12488 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:36.833909 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.838867 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:36.839875 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 170856us
I20260709 12:11:36.841904 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.770459 13651 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.845224 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:36.851890 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 309825us
W20260709 12:11:36.855309 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 201104us
I20260709 12:11:36.837590 13752 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:36.860741 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 345451us
I20260709 12:11:36.862740 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.860052 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:36.908982 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.903616 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:36.911928 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51784.9us
I20260709 12:11:36.866118 12570 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:36.847880 13769 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.915921 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:36.917076 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57103.1us
I20260709 12:11:36.857445 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.921999 13649 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.922752 13789 raft_consensus.cc:2243] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:36.926486 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.852144 13656 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260709 12:11:36.929364 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:36.857803 13779 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:36.864751 12641 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:36.860370 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:36.945685 13793 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.946604 13793 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.971799 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.972357 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:36.981479 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:36.994329 13811 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.994383 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:36.995352 13790 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.487s	user 0.004s	sys 0.003s
I20260709 12:11:36.903139 13752 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:36.933609 13767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34818 (request call id 73) took 2697 ms. Trace:
I20260709 12:11:37.003955 13807 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:36.988561 13687 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:36.877594 13808 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260709 12:11:37.005101 13807 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.017517 13714 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:11:37.025425 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109588us
I20260709 12:11:36.992106 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.030187 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.043804 13651 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.047207 13789 raft_consensus.cc:2272] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:37.051179 13813 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:37.060196 13797 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:37.060640 13810 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
W20260709 12:11:37.054358 13788 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.570s	user 0.015s	sys 0.000s
I20260709 12:11:37.055280 12496 raft_consensus.cc:2955] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:37.026350 13746 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:37.074400 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103613us
I20260709 12:11:37.079006 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.072081 13779 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.082984 13779 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.086819 13802 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.053519 13813 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.087554 13802 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.089641 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.090498 13779 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.092501 13802 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.101756 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.107537 13781 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.116349 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.063007 13770 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.121724 13726 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.131886 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:37.140635 13793 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:37.142303 13752 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:37.072325 13788 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.143709 13816 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:37.144407 13816 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:37.144721 13788 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.144874 13816 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:37.145840 12391 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 211) took 2516 ms. Trace:
I20260709 12:11:37.148589 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:37.146236 12391 rpcz_store.cc:276] 0709 12:11:34.629469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.629671 (+   202us) service_pool.cc:224] Handling call
0709 12:11:37.145743 (+2516072us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:35.871542 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:35.872185 (+   643us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:35.872229 (+    44us) write_op.cc:435] Acquired schema lock
0709 12:11:35.872249 (+    20us) tablet.cc:662] Decoding operations
0709 12:11:35.872672 (+   423us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:35.872741 (+    69us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:35.872771 (+    30us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:35.872980 (+   209us) tablet.cc:701] Row locks acquired
0709 12:11:35.873000 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:11:35.873168 (+   168us) write_op.cc:270] Start()
0709 12:11:35.873298 (+   130us) write_op.cc:276] Timestamp: P: 1783599095873120 usec, L: 0
0709 12:11:35.873320 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:11:35.880972 (+  7652us) log.cc:844] Serialized 598 byte log entry
0709 12:11:36.524075 (+643103us) op_driver.cc:464] REPLICATION: finished
0709 12:11:36.876770 (+352695us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b200007f3f8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf53a1b91 00007ffbf53c3a19 00007ffbf538e8c3 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169 00007ffbf5397131 00007ffbf539642d 0000000000388634
0709 12:11:36.922059 (+ 45289us) write_op.cc:301] APPLY: starting
0709 12:11:36.922176 (+   117us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:36.922302 (+   126us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:36.922328 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:36.922635 (+   307us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:36.922801 (+   166us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:36.922847 (+    46us) write_op.cc:312] APPLY: finished
0709 12:11:36.923062 (+   215us) log.cc:844] Serialized 22 byte log entry
0709 12:11:36.923225 (+   163us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:36.923308 (+    83us) write_op.cc:454] Released schema lock
0709 12:11:36.923354 (+    46us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:36.923723 (+   369us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:36.923752 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":590586,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":392330,"apply.run_cpu_time_us":1670,"apply.run_wall_time_us":1670,"num_ops":1,"prepare.queue_time_us":503729,"prepare.run_cpu_time_us":6001,"prepare.run_wall_time_us":139037,"raft.queue_time_us":392341,"raft.run_cpu_time_us":41177,"raft.run_wall_time_us":372995,"replication_time_us":627520,"spinlock_wait_cycles":10558336,"thread_start_us":635227,"threads_started":3,"wal-append.queue_time_us":129618}]]}
I20260709 12:11:37.130337 13719 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:37.141029 13770 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.140690 13746 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:37.162470 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.166127 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.167779 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.171341 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.010886 13767 rpcz_store.cc:276] 0709 12:11:34.220628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.228540 (+  7912us) service_pool.cc:224] Handling call
0709 12:11:34.334898 (+106358us) log.cc:844] Serialized 501 byte log entry
0709 12:11:36.917517 (+2582619us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1208,"raft.queue_time_us":1690,"raft.run_cpu_time_us":1783,"raft.run_wall_time_us":1783,"spinlock_wait_cycles":85376,"thread_start_us":122693,"threads_started":1,"wal-append.queue_time_us":149832,"wal-append.run_cpu_time_us":1906,"wal-append.run_wall_time_us":1003965}
I20260709 12:11:37.189584 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.223639 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.164383 13786 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:37.251526 13789 ts_tablet_manager.cc:1916] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:11:37.247764 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72977.6us
I20260709 12:11:37.262943 13813 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.258929 13786 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.276651 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.277612 13813 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.267334 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.268321 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.271057 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.271576 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.272248 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.272439 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.273200 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:37.291067 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.293032 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:11:37.251859 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44386.6us
I20260709 12:11:37.295316 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c793ebd4de24492397303f9a6e6bc62a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:37.295163 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:37.305373 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.306171 12569 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:37.309886 12567 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:11:37.361109 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66012.7us
W20260709 12:11:37.361936 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58265.2us
W20260709 12:11:37.362942 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50590.7us
I20260709 12:11:37.292316 13715 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.368018 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:37.368759 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60684.3us
I20260709 12:11:37.368340 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.378278 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.301831 13818 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260709 12:11:37.364434 13802 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:37.360030 13822 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:37.367307 13770 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:37.396730 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.395530 13813 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.402998 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:37.393136 13683 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.391774 13802 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:37.384202 13726 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:37.427907 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62899.4us
I20260709 12:11:37.410202 13813 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:37.438548 13826 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:11:37.438691 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.413695 13802 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:37.439952 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:37.444447 13717 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:37.444895 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.447168 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.454145 12639 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:37.458338 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:37.447954 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.413820 13649 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 12:11:37.452239 13788 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.409955 13793 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 12:11:37.461164 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1129) took 1319 ms. Trace:
I20260709 12:11:37.467144 12642 rpcz_store.cc:276] 0709 12:11:36.141161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:36.144054 (+  2893us) service_pool.cc:224] Handling call
0709 12:11:36.183004 (+ 38950us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:36.183234 (+   230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:36.584373 (+401139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:36.584427 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:36.584493 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:36.584541 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:37.460058 (+875517us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:37.460111 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:37.460237 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":943769600}
I20260709 12:11:37.433156 13791 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.464444 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:37.472833 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.474521 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.476415 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.454888 13782 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:37.465124 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:37.481217 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.487263 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:37.406401 13816 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.494971 13782 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:37.495462 13782 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:37.499351 13781 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:37.499138 13766 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.507309 13785 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:37.517230 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:37.504911 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45794.9us
I20260709 12:11:37.509320 13649 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:37.526023 13786 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:37.513967 13781 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.531461 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.538416 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.538817 13816 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:37.540030 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.541240 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:37.513743 13782 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.570024 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.533929 13781 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:37.580909 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.573031 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.580111 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:37.588491 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:37.518641 13766 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.523299 13785 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.602665 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:37.603410 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 114477us
I20260709 12:11:37.605808 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.591984 13803 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:11:37.609807 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 94759.4us
I20260709 12:11:37.604823 13813 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.617998 13813 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.626514 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.627339 13745 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.627902 13745 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.634074 13816 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:37.634779 13816 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.638863 13814 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:37.639489 13814 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.640027 12462 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:37.642946 13789 ts_tablet_manager.cc:1929] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:37.643579 13789 log.cc:1199] T 54875fd261ee4c328e3d022d6a9418d8 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/54875fd261ee4c328e3d022d6a9418d8
I20260709 12:11:37.644955 13789 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34818 (request call id 74) took 2088 ms. Trace:
I20260709 12:11:37.600575 13722 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.657096 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.658038 12566 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:37.658387 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.667789 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:37.649513 13827 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:37.655768 13814 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.648036 13803 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.774737 13839 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:37.715129 13719 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.000s
I20260709 12:11:37.808781 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.719720 13745 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:37.736012 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120419us
I20260709 12:11:37.814549 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:37.839118 13802 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.800963 13719 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:37.776268 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 149873us
I20260709 12:11:37.801142 13751 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:37.803742 13813 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:37.840653 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119148us
I20260709 12:11:37.857287 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:11:37.857966 13840 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:11:37.860934 13837 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:11:37.645335 13789 rpcz_store.cc:276] 0709 12:11:35.589098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:35.773815 (+184717us) service_pool.cc:224] Handling call
0709 12:11:36.251092 (+477277us) ts_tablet_manager.cc:1151] Deleting tablet 54875fd261ee4c328e3d022d6a9418d8
0709 12:11:36.271562 (+ 20470us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:11:37.618436 (+1346874us) tablet_metadata.cc:665] Metadata flushed
0709 12:11:37.642770 (+ 24334us) tablet_metadata.cc:665] Metadata flushed
0709 12:11:37.644866 (+  2096us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":617336,"tablet-delete.queue_time_us":417872,"thread_start_us":332707,"threads_started":1}
I20260709 12:11:37.866015 13766 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.868852 13802 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:37.873924 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:37.877810 12571 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:37.841575 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.884595 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.891716 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.871490 13803 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:37.906747 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.878662 13751 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.895464 13814 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:37.897019 13802 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.919116 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.924166 13751 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:37.930577 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.891139 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.935369 13781 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:37.937001 13259 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:37.938787 13814 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:37.905963 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.951347 12644 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:37.954527 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72295.1us
I20260709 12:11:37.955567 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:37.955716 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:37.958457 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103551us
I20260709 12:11:37.958999 13807 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:37.941407 13814 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:37.887382 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 54875fd261ee4c328e3d022d6a9418d8 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:37.963882 12392 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 207) took 3737 ms. Trace:
W20260709 12:11:37.968815 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81445.5us
I20260709 12:11:37.971607 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:37.950122 13786 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:37.975847 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.950706 13803 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:37.965277 13726 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:37.981576 12644 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:37.995332 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:37.995457 12395 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:38.005565 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:37.968406 13842 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:38.016695 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:38.026384 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.034940 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:38.053128 13785 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:38.053006 13723 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:38.054174 13723 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:38.054525 13723 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:38.055393 13723 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.055580 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.059403 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:38.059962 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.060645 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.061268 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.061460 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.061932 13813 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.062152 13786 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.062791 13813 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.063910 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.059506 13724 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:37.967123 12392 rpcz_store.cc:276] 0709 12:11:34.224700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.229389 (+  4689us) service_pool.cc:224] Handling call
0709 12:11:34.343851 (+114462us) spinlock_profiling.cc:243] Waited 5.89 ms on lock 0x7b5400290090. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cf640 00007ffbfb5a6846 00007ffbfb6b090e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:37.961548 (+3617697us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:36.010766 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:36.011380 (+   614us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:36.011426 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:11:36.011449 (+    23us) tablet.cc:662] Decoding operations
0709 12:11:36.011847 (+   398us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:36.011894 (+    47us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:36.011927 (+    33us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:36.012183 (+   256us) tablet.cc:701] Row locks acquired
0709 12:11:36.012203 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:11:36.012355 (+   152us) write_op.cc:270] Start()
0709 12:11:36.012479 (+   124us) write_op.cc:276] Timestamp: P: 1783599096012306 usec, L: 0
0709 12:11:36.012510 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:11:36.276319 (+263809us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b5400268ca0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5770520 00007ffbf57f9b42 00007ffbf57eef67 00007ffbf57eeda4 00007ffbf9229581 00007ffbf9228d58 00007ffbf922d9f1 00007ffbf922d9a9 00007ffbf922d939 00007ffbf922d901
0709 12:11:36.362517 (+ 86198us) log.cc:844] Serialized 541 byte log entry
0709 12:11:36.997506 (+634989us) op_driver.cc:464] REPLICATION: finished
0709 12:11:37.013353 (+ 15847us) write_op.cc:301] APPLY: starting
0709 12:11:37.026215 (+ 12862us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:37.036362 (+ 10147us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:37.039416 (+  3054us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:37.062319 (+ 22903us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:37.062465 (+   146us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:37.062506 (+    41us) write_op.cc:312] APPLY: finished
0709 12:11:37.062737 (+   231us) log.cc:844] Serialized 23 byte log entry
0709 12:11:37.062916 (+   179us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:37.063007 (+    91us) write_op.cc:454] Released schema lock
0709 12:11:37.063070 (+    63us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:37.089826 (+ 26756us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:37.113987 (+ 24161us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":81242,"reported_tablets":2,"spinlock_wait_cycles":13544320,"child_traces":[["op",{"apply.queue_time_us":1317,"apply.run_cpu_time_us":2043,"apply.run_wall_time_us":56927,"num_ops":1,"prepare.queue_time_us":631979,"prepare.run_cpu_time_us":14215,"prepare.run_wall_time_us":392770,"replication_time_us":984864,"spinlock_wait_cycles":124468096}]]}
I20260709 12:11:38.067377 13807 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:38.067932 13807 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.065727 13723 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:38.068943 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.069528 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.070098 13807 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:38.070344 13787 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:38.070641 13807 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.066860 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:38.071832 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.071857 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.072701 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:38.073258 13808 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.074203 13818 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.075866 13745 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:38.078015 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:38.065913 13724 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:38.078543 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.078763 13258 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:38.079437 13724 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.077807 13842 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.080559 13724 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:38.081940 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.084223 13781 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:38.086190 13813 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.089522 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 17) took 1879 ms. Trace:
I20260709 12:11:38.089951 12490 rpcz_store.cc:276] 0709 12:11:36.210553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:36.223207 (+ 12654us) service_pool.cc:224] Handling call
0709 12:11:36.223586 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:36.353192 (+129606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:36.353245 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:36.415816 (+ 62571us) log.cc:844] Serialized 26 byte log entry
0709 12:11:36.765894 (+350078us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:36.864168 (+ 98274us) log.cc:844] Serialized 14 byte log entry
0709 12:11:36.865758 (+  1590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:38.089151 (+1223393us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:38.089196 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:38.089234 (+    38us) raft_consensus.cc:1685] finished
0709 12:11:38.089316 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:38.089436 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13600,"prepare.queue_time_us":263399,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":21961,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":36003,"spinlock_wait_cycles":291785984,"thread_start_us":367484,"threads_started":2,"wal-append.queue_time_us":380716,"wal-append.run_cpu_time_us":5092,"wal-append.run_wall_time_us":1206223}
I20260709 12:11:38.097726 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:38.101852 13745 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:38.102402 13745 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:38.103497 13745 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.103909 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.104769 13745 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:38.105712 12565 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:38.106483 13715 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.107951 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:38.082989 13717 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:38.112058 13717 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:38.115351 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:38.105419 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.128942 12492 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:38.131414 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:38.133615 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55161us
I20260709 12:11:38.134261 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.135545 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.137131 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:38.156276 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:38.155004 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.114940 13717 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:38.133751 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:38.154131 13842 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.146237 13802 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.154316 13751 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:38.196709 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.191002 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1456) took 1248 ms. Trace:
I20260709 12:11:38.202497 12493 rpcz_store.cc:276] 0709 12:11:36.933114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:36.933224 (+   110us) service_pool.cc:224] Handling call
0709 12:11:37.265801 (+332577us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:38.154828 (+889027us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cee9c 00007ffbf26cea5b 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:38.181706 (+ 26878us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88927872}
I20260709 12:11:38.185889 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.206807 13751 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.223405 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:38.244705 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65464.2us
I20260709 12:11:38.244211 13751 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:38.249025 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.249920 13788 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:38.251579 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.253993 13788 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:38.255565 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:38.257926 13788 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:38.258373 13788 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:38.259603 13788 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:38.260313 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69847.1us
W20260709 12:11:38.264355 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97685us
I20260709 12:11:38.266456 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.266624 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.269299 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:38.271559 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.271840 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.263269 13782 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:38.278970 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.281211 13726 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:38.281852 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.282195 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.282546 13726 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:38.282948 13726 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:38.283721 13756 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:38.285530 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.287048 13715 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.292182 13842 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:38.293411 13842 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:38.293776 13842 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:11:38.294051 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.590s (SENT) [suppressed 4 similar messages]
I20260709 12:11:38.294687 13842 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.296180 13842 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:38.300439 13858 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.282578 13816 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:38.308790 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:38.311967 13788 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.316399 12646 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:38.319005 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.320286 12488 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:38.320878 13788 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:38.326972 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60606.6us
I20260709 12:11:38.328028 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.332866 13808 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.337852 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:38.341166 13861 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:38.341913 13861 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:38.342338 13861 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:38.334597 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:38.345317 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51295us
I20260709 12:11:38.346429 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.347353 13781 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.347514 13710 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.348474 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.350595 13781 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.351437 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:38.352260 13710 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.360324 12610 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.362464 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:38.372290 13788 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.415481 13563 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:38.362984 13726 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.423074 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:38.399536 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:38.426491 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.427834 13829 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:38.428238 13829 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:38.429142 13829 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.431650 13829 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:38.432775 13726 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:38.436492 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.439622 13834 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.445840 12395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 114) took 4243 ms. Trace:
I20260709 12:11:38.446494 12395 rpcz_store.cc:276] 0709 12:11:34.195061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:34.197678 (+  2617us) service_pool.cc:224] Handling call
0709 12:11:38.438700 (+4241022us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:38.039058 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:38.039709 (+   651us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:38.039752 (+    43us) write_op.cc:435] Acquired schema lock
0709 12:11:38.039777 (+    25us) tablet.cc:662] Decoding operations
0709 12:11:38.040210 (+   433us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:38.040294 (+    84us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:38.040333 (+    39us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:38.040541 (+   208us) tablet.cc:701] Row locks acquired
0709 12:11:38.040560 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:11:38.040751 (+   191us) write_op.cc:270] Start()
0709 12:11:38.040877 (+   126us) write_op.cc:276] Timestamp: P: 1783599098040703 usec, L: 0
0709 12:11:38.040900 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:11:38.056261 (+ 15361us) log.cc:844] Serialized 536 byte log entry
0709 12:11:38.223839 (+167578us) op_driver.cc:464] REPLICATION: finished
0709 12:11:38.343032 (+119193us) write_op.cc:301] APPLY: starting
0709 12:11:38.343154 (+   122us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:38.343304 (+   150us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:38.343490 (+   186us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:38.362504 (+ 19014us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:38.362685 (+   181us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:38.362730 (+    45us) write_op.cc:312] APPLY: finished
0709 12:11:38.373685 (+ 10955us) log.cc:844] Serialized 23 byte log entry
0709 12:11:38.409527 (+ 35842us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:38.413788 (+  4261us) write_op.cc:454] Released schema lock
0709 12:11:38.415945 (+  2157us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:38.425212 (+  9267us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:38.425240 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":818683,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":118839,"apply.run_cpu_time_us":2936,"apply.run_wall_time_us":82158,"num_ops":1,"prepare.queue_time_us":38838,"prepare.run_cpu_time_us":3090,"prepare.run_wall_time_us":19550,"replication_time_us":182760,"thread_start_us":63874,"threads_started":2}]]}
I20260709 12:11:38.443470 13745 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:38.452733 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.454488 13785 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.453922 12392 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:11:38.464277 12392 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:38.476158 12392 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 3, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 3 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:38.481843 12392 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 3 to 4, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 1 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:11:38.499393 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84279us
I20260709 12:11:38.535933 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.538591 13861 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.542037 13861 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:38.547004 12638 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:38.548372 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 133832us
I20260709 12:11:38.531426 12392 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:38.566519 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:38.571051 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:38.549062 13755 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:38.616235 12492 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:38.578382 13755 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.639122 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:38.577143 13803 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.642097 13859 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:38.642869 13859 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:38.643960 13859 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.645838 12641 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:38.646839 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:38.647420 12490 raft_consensus.cc:1813] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Handling vote request from an unknown peer 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:38.647944 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:38.647955 12490 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:11:38.657752 12536 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.658689 13751 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:38.641146 13755 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:38.674853 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52281.6us
I20260709 12:11:38.695683 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1140) took 1108 ms. Trace:
I20260709 12:11:38.696973 12638 rpcz_store.cc:276] 0709 12:11:37.587060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:37.588160 (+  1100us) service_pool.cc:224] Handling call
0709 12:11:38.695445 (+1107285us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:38.695578 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2179954944}
I20260709 12:11:38.709128 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:38.710176 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.711714 13842 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:38.712177 13842 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:38.712522 13842 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:38.718376 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:38.725888 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:38.726735 12566 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:38.727697 13710 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:38.740718 13842 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.729518 13710 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:38.746374 13710 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:38.756827 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:38.758603 12488 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:38.758806 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:38.758582 13842 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:38.787335 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 43fecbd2edc14ba2ab065395e5d48d1d with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34818
I20260709 12:11:38.791639 12566 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.794258 12391 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 4, VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) evicted. New cstate: current_term: 1 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:38.793560 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:38.812403 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:38.817003 13842 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:38.817166 13869 tablet_replica.cc:333] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:11:38.817720 13842 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:38.818822 12395 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:38.827370 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet c793ebd4de24492397303f9a6e6bc62a with delete_type TABLET_DATA_TOMBSTONED (TS 0a29b5d9f9d44298b02ffcd460811eef not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32876
I20260709 12:11:38.826929 13842 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:38.844245 13260 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:11:38.856178 13744 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:11:38.889060 12638 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260709 12:11:38.896870 13868 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.000s	sys 0.002s
I20260709 12:11:38.908459 13869 raft_consensus.cc:2243] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:38.913774 13871 tablet_replica.cc:333] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:11:38.935431 13871 raft_consensus.cc:2243] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:38.936497 13871 raft_consensus.cc:2272] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:38.905177 13751 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:11:38.953241 13869 raft_consensus.cc:2272] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:38.966645 13869 ts_tablet_manager.cc:1916] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:38.964723 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:38.987780 12391 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:39.010946 12393 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 2 to 3, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 3 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:11:39.023950 13756 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:39.047153 13257 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:39.062069 12395 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:39.076769 13871 ts_tablet_manager.cc:1916] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:39.097414 12395 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:39.108022 12488 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:39.111758 12379 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e1f18763e7844e0994e2f11a6dd89290 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:11:39.141564 13864 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:39.158632 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet e1f18763e7844e0994e2f11a6dd89290 with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34480
I20260709 12:11:39.160554 12638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:39.161446 12638 raft_consensus.cc:606] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:39.161537 13876 tablet_replica.cc:333] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:11:39.168292 12566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:39.169142 12566 raft_consensus.cc:606] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:39.174382 12566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:39.175369 12566 raft_consensus.cc:606] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:39.181824 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:11:39.182684 12488 raft_consensus.cc:606] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:39.186822 12566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:39.187585 12566 raft_consensus.cc:606] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:39.196836 12640 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:39.197589 12640 raft_consensus.cc:606] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:39.199389 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:11:39.201542 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:11:39.201079 12638 raft_consensus.cc:2955] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:39.202656 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:39.226662 13876 raft_consensus.cc:2243] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:11:39.227711 13876 raft_consensus.cc:2272] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:11:39.231189 13876 ts_tablet_manager.cc:1916] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:11:39.231005 12566 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:39.235230 13864 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:39.263087 13876 ts_tablet_manager.cc:1929] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:11:39.268078 13876 log.cc:1199] T e1f18763e7844e0994e2f11a6dd89290 P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/e1f18763e7844e0994e2f11a6dd89290
I20260709 12:11:39.270519 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet e1f18763e7844e0994e2f11a6dd89290 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:39.277563 12571 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:39.286125 13815 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:11:39.307524 13818 raft_consensus.cc:993] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:39.308132 13751 raft_consensus.cc:1081] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:39.310016 13257 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:39.310693 13257 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:39.311156 13257 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.331147 13869 ts_tablet_manager.cc:1929] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:11:39.331743 13869 log.cc:1199] T 43fecbd2edc14ba2ab065395e5d48d1d P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/43fecbd2edc14ba2ab065395e5d48d1d
I20260709 12:11:39.332526 13745 raft_consensus.cc:993] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:39.332381 12488 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:39.333145 13710 raft_consensus.cc:1081] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:39.341040 13842 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:39.343979 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 43fecbd2edc14ba2ab065395e5d48d1d (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:39.351651 12567 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:11:39.354324 12567 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:39.355103 12567 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.367228 13871 ts_tablet_manager.cc:1929] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20260709 12:11:39.369200 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57674.7us
I20260709 12:11:39.386281 13871 log.cc:1199] T c793ebd4de24492397303f9a6e6bc62a P 0a29b5d9f9d44298b02ffcd460811eef: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/wals/c793ebd4de24492397303f9a6e6bc62a
I20260709 12:11:39.370818 13257 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.393579 13257 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:39.388294 12567 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.394368 12379 catalog_manager.cc:5028] TS 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): tablet c793ebd4de24492397303f9a6e6bc62a (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:11:39.394593 13563 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:39.395285 13563 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.396134 12567 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.398891 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:39.399720 12488 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:39.400071 12488 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:39.401041 12488 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.403815 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:39.409950 12490 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.397439 13563 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:39.405822 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:39.404520 12488 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.412295 12566 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:39.412933 12566 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:39.413970 12638 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:39.421430 13842 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:39.423877 13865 raft_consensus.cc:993] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:39.424437 13865 raft_consensus.cc:1081] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:39.416952 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:39.430722 13257 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:39.434382 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:39.434681 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:39.434856 12567 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32884
I20260709 12:11:39.435079 12641 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.435787 12640 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:39.436033 12567 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:39.436697 12567 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.440990 12643 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:39.442847 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:39.443799 13818 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:39.444187 13818 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:39.444566 13818 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.444492 12641 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:39.454718 13744 raft_consensus.cc:993] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:39.455117 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:39.455341 13756 raft_consensus.cc:1081] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:39.456423 13756 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:39.456861 13756 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:39.458417 13756 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.462755 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:39.464373 13818 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.465256 13788 raft_consensus.cc:993] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:39.465867 13865 raft_consensus.cc:1081] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:39.462513 12488 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:39.465006 12567 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.433653 12566 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.473759 13262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50708
I20260709 12:11:39.474458 13262 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:39.474843 13262 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.477733 12566 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.481534 13257 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:39.482486 13257 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:39.483239 13257 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.485970 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:39.477674 13827 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 12:11:39.488978 13842 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:39.498636 13257 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.522342 13257 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.538175 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:39.539086 12565 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:39.539448 12565 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:39.540325 12565 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.543537 12567 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.536326 12392 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 3, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 3 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:39.545850 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:39.554651 12492 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.555992 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:39.556740 12644 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:39.548821 13262 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:39.570410 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91176.2us
I20260709 12:11:39.571686 12566 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:39.577380 13262 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.541568 12565 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.569494 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:39.554546 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67309.4us
I20260709 12:11:39.572988 13818 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:39.587738 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:39.590107 12492 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:39.588650 13842 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:39.585009 12641 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:11:39.608233 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46188.3us
I20260709 12:11:39.597987 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:39.619700 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:39.623973 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:39.626714 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:39.627686 12643 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:39.638443 13744 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:39.639317 13744 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.646085 13744 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:39.638407 13886 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:11:39.610869 12641 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:39.652192 12641 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.653136 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:39.653846 13888 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:39.654296 12640 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:39.654486 13888 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.657333 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:39.658192 12492 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:39.651887 13842 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.659325 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:39.621665 13755 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:39.667481 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:39.670948 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:39.674903 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:39.676075 12492 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.673107 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:39.678901 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:39.643769 13818 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:39.684182 12641 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.684784 13563 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:39.685885 13563 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.693280 12492 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:39.668784 13744 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:39.701267 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:39.702369 12643 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:39.702852 12643 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:39.657388 13888 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.708645 12643 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.693454 13818 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:39.698506 13744 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:39.717975 13744 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.719745 13818 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.723831 13563 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:39.713224 13893 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:39.730257 13893 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:39.730703 13893 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.734926 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:39.735939 12640 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:39.722986 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:39.737528 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:39.723058 13885 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.111s	user 0.004s	sys 0.000s
I20260709 12:11:39.738430 13891 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:39.751377 13888 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:39.756328 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:39.776889 12643 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:39.787110 12497 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:39.750279 13755 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:39.802145 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:39.809784 13257 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:39.782673 13891 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:39.833312 12643 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:39.800021 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
W20260709 12:11:39.851709 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67877.2us
W20260709 12:11:39.854507 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53086.2us
I20260709 12:11:39.857965 13755 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.871731 13891 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.855986 13744 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:39.878260 13744 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.894591 13893 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:39.898335 12641 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:39.902043 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:39.908604 13893 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:39.913450 12565 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
W20260709 12:11:39.966913 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46948.5us
I20260709 12:11:39.970285 12567 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:39.972980 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:39.977023 13893 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:39.977600 13893 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:39.978845 13893 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:39.980084 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124932us
I20260709 12:11:39.986377 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:39.985476 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 193820us
I20260709 12:11:39.998219 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:39.998735 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:39.999636 12643 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:40.017781 12497 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:40.000551 13896 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:40.036542 13905 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:40.039819 13756 raft_consensus.cc:993] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:40.041011 13751 raft_consensus.cc:1081] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:40.020532 13896 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:40.041934 13896 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:40.041617 12393 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:40.042707 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:40.044142 13257 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:40.044745 13257 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:40.044895 12643 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:40.045120 13257 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:40.037232 13905 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.054915 13905 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:40.056137 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:40.060089 13903 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:40.060626 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:40.071105 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:40.071872 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:40.069384 13257 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.010118 13888 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:40.088104 13896 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.060683 13903 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:40.092412 13903 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:40.100119 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:40.100824 12497 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:40.084975 13257 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:40.104866 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:40.107946 13257 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:40.112599 13782 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:40.113128 13782 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:40.113590 13782 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:40.099635 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:40.119715 12565 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:40.120272 12565 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:40.121955 12565 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.123428 12565 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:40.120709 12643 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:11:40.126700 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:40.121021 13896 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:40.128098 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:40.128782 13257 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:40.127422 12646 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:11:40.142998 12490 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:11:40.150665 12497 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:40.163247 12646 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:40.165283 13257 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:40.158471 13751 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:11:40.166097 13888 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.167847 13755 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:40.170910 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:40.173611 13818 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:40.174299 13818 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:40.175307 13818 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.177201 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:40.178251 13893 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:40.178951 13893 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:40.179949 13893 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.192232 13782 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.256865 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:40.301916 13782 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:40.302843 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:40.305739 12572 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.313978 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:40.314754 13257 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:40.324158 13257 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:40.334870 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:40.356215 13782 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:40.357108 13262 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:40.356868 13782 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:40.359808 13782 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:40.328636 12565 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:40.333091 13893 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:11:40.379591 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:40.346005 13803 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:40.388553 12497 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.384352 12572 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:40.389926 12643 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.391652 12572 raft_consensus.cc:606] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:40.414623 13803 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:40.414845 12497 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:40.430522 13893 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:40.382565 13744 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:11:40.462764 12567 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:40.463758 12567 raft_consensus.cc:606] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:40.484805 12391 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:40.508245 12391 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 3, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 3 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:40.439221 13897 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:11:40.520396 12391 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:40.472712 13803 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.562685 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:40.563721 12497 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:40.596037 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:40.540148 13893 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:11:40.604753 13563 raft_consensus.cc:993] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:40.610505 12497 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.622087 12564 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.628095 13891 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:11:40.629935 13563 raft_consensus.cc:1081] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:40.630946 13891 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:11:40.627866 13788 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:40.634280 13262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:40.642732 13262 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:40.643332 13262 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:40.637807 13935 raft_consensus.cc:993] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:40.669387 13262 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.672353 12497 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.689910 13935 raft_consensus.cc:1081] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:40.638913 13788 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:40.727013 13262 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:40.715662 13260 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:40.729337 13260 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:40.729923 13260 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:40.731537 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68462.9us
I20260709 12:11:40.671051 12565 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:40.742995 13262 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:40.747129 12496 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:40.721302 13788 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:40.756049 12646 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
W20260709 12:11:40.807788 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 202087us
W20260709 12:11:40.809484 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58477.9us
I20260709 12:11:40.750154 12565 raft_consensus.cc:606] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:40.831688 12565 raft_consensus.cc:614] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:11:40.809823 13818 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:11:40.877979 13943 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:40.878674 13943 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.881834 13260 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.897668 13260 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:40.908332 12392 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 3, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:40.910281 13943 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:40.911047 13890 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:11:40.910928 12392 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:40.924816 13909 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:40.925523 13909 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:40.926687 13909 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.928364 13909 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:40.931079 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:40.932058 12564 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:40.932485 12564 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:40.935840 12570 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:40.942752 12570 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:40.950058 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:40.949496 12392 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:40.951419 12570 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:40.952033 12570 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:40.959968 13842 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:40.960517 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:40.968688 12644 raft_consensus.cc:606] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:40.963665 12564 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:40.969903 12564 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:40.964977 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:40.979410 13257 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:40.970500 13940 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:11:41.006645 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:11:41.007361 12488 raft_consensus.cc:606] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:40.958323 13890 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:11:40.970995 12570 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:41.015753 12564 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:41.023226 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:41.023852 12572 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:41.053613 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:41.050697 13888 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:41.073745 12643 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:41.096848 13888 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:41.098124 13888 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:41.102644 12643 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:41.110838 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:41.111690 12640 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:41.112286 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:41.113385 12644 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:41.116542 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50416.7us
I20260709 12:11:41.129392 13788 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:41.131948 13788 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:41.139710 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:41.140241 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:41.159994 12638 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:41.165369 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:41.167018 13909 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:41.177562 12645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:41.180961 12645 raft_consensus.cc:606] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:41.182410 12645 raft_consensus.cc:614] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:11:41.190522 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:11:41.139515 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150163us
I20260709 12:11:41.195963 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:11:41.197978 13910 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:11:41.201375 13910 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:11:41.201663 13951 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:41.206904 12641 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:41.216775 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:11:41.218086 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:11:41.228134 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:41.236045 13914 raft_consensus.cc:993] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:41.236641 13914 raft_consensus.cc:1081] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:41.250979 12646 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:41.260937 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:41.261914 12571 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:11:41.262398 12571 pending_rounds.cc:85] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 5
I20260709 12:11:41.262758 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:41.266530 12571 pending_rounds.cc:107] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:11:41.266965 12571 raft_consensus.cc:2889] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:11:41.269122 12566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32884
I20260709 12:11:41.270028 12566 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:41.270533 12566 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:41.251693 12640 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:41.276010 12646 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:41.279037 12393 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:41.335774 12570 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:41.338471 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:41.343346 12566 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:41.349776 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:41.350478 12494 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:41.291111 13803 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:11:41.354377 13715 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:41.355064 13715 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:41.354477 13934 raft_consensus.cc:993] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:41.357879 13934 raft_consensus.cc:1081] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:11:41.383036 12566 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:41.386787 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:41.420657 12566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:11:41.421480 12566 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:41.421990 12566 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:41.398722 13744 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:11:41.424190 13715 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:41.426797 13949 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:11:41.427251 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:41.436012 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:41.438392 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:41.410593 13948 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:41.381793 13909 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 12:11:41.464463 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:41.466058 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:41.468348 12566 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:41.473886 13953 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:41.467998 13935 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:41.478466 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:41.481513 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:41.467809 12640 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:41.520563 12494 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:41.541481 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:41.570444 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92102.8us
I20260709 12:11:41.576434 13260 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:11:41.581168 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:41.582193 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:41.562273 12566 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:41.590364 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:41.591030 12496 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:41.591346 12496 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:41.569314 13935 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:41.573940 13953 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:41.569962 12640 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:41.577350 13951 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:41.610383 13803 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:11:41.677733 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
W20260709 12:11:41.695477 13960 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:11:41.704908 12638 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:41.705410 13953 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:41.729826 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.012s	sys 0.038s
W20260709 12:11:41.747952 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.012s	sys 0.039s
W20260709 12:11:41.748355 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 550554us
I20260709 12:11:41.617708 13908 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260709 12:11:41.914844 13881 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.139s
I20260709 12:11:42.002763 12496 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:41.683533 13935 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:42.019596 13961 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:11:42.020145 12496 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:42.021092 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 48) took 1029 ms. Trace:
I20260709 12:11:42.021545 12497 rpcz_store.cc:276] 0709 12:11:40.991523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:40.991672 (+   149us) service_pool.cc:224] Handling call
0709 12:11:40.992018 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:40.992411 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:40.992442 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:40.993027 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:11:41.109436 (+116409us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b180000e1a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:11:41.374908 (+265472us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:41.374956 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:41.375027 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:42.020723 (+645696us) raft_consensus.cc:1685] finished
0709 12:11:42.020752 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:42.021016 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6024,"raft.queue_time_us":145,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":13121,"spinlock_wait_cycles":9520384,"thread_start_us":265276,"threads_started":1,"wal-append.queue_time_us":716328}
W20260709 12:11:42.002308 13890 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.010s	sys 0.036s
I20260709 12:11:42.034255 12496 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
W20260709 12:11:42.038280 13890 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.011s	sys 0.037s
W20260709 12:11:42.006054 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91213.7us
W20260709 12:11:42.064515 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.021s	sys 0.090s
W20260709 12:11:42.304453 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.021s	sys 0.090s
W20260709 12:11:42.297330 13958 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.002s	sys 0.011s
W20260709 12:11:42.289229 13970 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 12:11:42.307885 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 71) took 1214 ms. Trace:
I20260709 12:11:42.308308 12567 rpcz_store.cc:276] 0709 12:11:41.093370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.114182 (+ 20812us) service_pool.cc:224] Handling call
0709 12:11:41.114778 (+   596us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:41.135401 (+ 20623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:41.135435 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:41.135945 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:11:41.258200 (+122255us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b180018c8e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:11:41.637133 (+378933us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:41.637177 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:41.637260 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:42.307534 (+670274us) raft_consensus.cc:1685] finished
0709 12:11:42.307558 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:42.307787 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9818,"raft.queue_time_us":20650,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9831,"spinlock_wait_cycles":8410368,"thread_start_us":378743,"threads_started":1,"wal-append.queue_time_us":802752}
W20260709 12:11:42.271454 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.006s	sys 0.026s
W20260709 12:11:42.125384 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.010s	sys 0.024s
W20260709 12:11:42.219081 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.000s	sys 0.030s
W20260709 12:11:42.311139 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.001s	sys 0.033s
W20260709 12:11:42.310047 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.006s	sys 0.027s
W20260709 12:11:42.290158 13903 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.007s	sys 0.003s
W20260709 12:11:42.318490 13903 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.007s	sys 0.003s
W20260709 12:11:42.310612 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.015s	sys 0.038s
I20260709 12:11:42.051070 13935 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:42.323593 13892 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:11:42.270910 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.004s	sys 0.028s
I20260709 12:11:42.325919 13930 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:11:42.326947 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.005s	sys 0.028s
I20260709 12:11:42.327080 13930 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:42.329255 13930 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:42.326485 13892 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:42.332294 13892 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:42.343331 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:42.359099 12643 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:42.374140 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:42.555985 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.019s	sys 0.060s
W20260709 12:11:43.099318 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.503s	user 0.019s	sys 0.075s
W20260709 12:11:43.100129 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.005s	sys 0.013s
W20260709 12:11:43.100615 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.005s	sys 0.013s
I20260709 12:11:42.393537 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:42.593997 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 247387us
I20260709 12:11:42.748854 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:42.887492 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.007s	sys 0.020s
W20260709 12:11:42.395087 13974 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 12:11:42.904582 13985 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:11:42.596817 13984 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:11:42.840818 13958 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.005s	sys 0.032s
W20260709 12:11:42.860217 13909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.002s	sys 0.017s
W20260709 12:11:43.082798 13976 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:11:42.593878 13893 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:42.790657 13982 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.090385 13905 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:42.394307 13892 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:42.558805 13864 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:42.720677 13980 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:42.925848 13981 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:43.347779 13892 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.348345 13980 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.348837 13934 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.348501 13981 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.349436 13934 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.350140 13892 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.351047 13981 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.350795 13864 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:42.379385 13930 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:42.700604 13977 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.328982 13988 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.329715 13935 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:43.330849 13907 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:42.839422 13756 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:42.848544 13953 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:42.603976 13931 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:11:43.334913 13958 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.005s	sys 0.038s
I20260709 12:11:43.347051 13905 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.257489 13751 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:43.351656 13981 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:11:43.337471 13909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.002s	sys 0.022s
I20260709 12:11:43.391739 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1455) took 2034 ms. Trace:
I20260709 12:11:43.601713 13909 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:43.600988 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:43.350528 13980 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:43.584408 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 835624us
I20260709 12:11:43.344190 13992 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.356045 13978 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.602038 12571 rpcz_store.cc:276] 0709 12:11:41.357227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.357408 (+   181us) service_pool.cc:224] Handling call
0709 12:11:41.366101 (+  8693us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:41.366510 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:41.366539 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:41.367041 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:11:41.808034 (+440993us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:41.808099 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:41.808204 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:43.391460 (+1583256us) raft_consensus.cc:1685] finished
0709 12:11:43.391486 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:43.391670 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2547,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":5333,"thread_start_us":434258,"threads_started":1,"wal-append.queue_time_us":764363}
I20260709 12:11:43.341251 13893 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.340474 13982 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.374146 14001 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:43.359663 13934 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:43.612564 13976 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
W20260709 12:11:43.655766 12646 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.000s	sys 0.028s
I20260709 12:11:43.601274 13755 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.515166 13993 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.515015 13996 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:43.372130 13930 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:43.653256 13989 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:11:43.623183 13931 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:43.658916 13260 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.088s	sys 0.149s
I20260709 12:11:43.373600 13977 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.605118 13909 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:43.668041 12646 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.001s	sys 0.028s
I20260709 12:11:43.662946 13998 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.695319 14004 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:43.703817 13998 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.704064 14004 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.707381 14004 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:43.648305 13994 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.391597 13935 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.633939 13751 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.611833 13756 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.674371 13958 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:43.714126 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1409) took 2253 ms. Trace:
I20260709 12:11:43.645831 13953 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:43.723107 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80691.4us
I20260709 12:11:43.515511 13907 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.721822 12646 rpcz_store.cc:276] 0709 12:11:41.457073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.457205 (+   132us) service_pool.cc:224] Handling call
0709 12:11:41.457530 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:41.462601 (+  5071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:41.462651 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:41.469372 (+  6721us) log.cc:844] Serialized 26 byte log entry
0709 12:11:42.304930 (+835558us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:42.844231 (+539301us) log.cc:844] Serialized 14 byte log entry
0709 12:11:43.709939 (+865708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:43.710187 (+   248us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:43.710318 (+   131us) raft_consensus.cc:1685] finished
0709 12:11:43.710341 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:43.710789 (+   448us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8863,"prepare.queue_time_us":1295526,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":490899,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":9789,"thread_start_us":2170387,"threads_started":3,"wal-append.queue_time_us":1430869}
I20260709 12:11:43.657126 13981 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.390949 13988 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.707563 13998 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:43.718238 13997 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.674661 13755 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.647256 13992 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.713873 13994 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.771694 14004 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.788425 14002 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:43.791687 13751 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.791565 14002 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.791069 14004 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.792356 13751 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:43.792863 13751 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.665434 14001 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.778129 13986 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.793761 14015 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:43.660959 13978 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.689350 13996 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.794119 14011 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:43.813958 14011 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.814231 14015 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.826272 14016 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:43.828205 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 312) took 1314 ms. Trace:
I20260709 12:11:43.678174 13993 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.807909 13864 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.827754 14001 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.827244 14016 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:43.838819 13991 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:11:43.790119 13997 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.809942 13986 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.839728 13992 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.877848 13993 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.914909 14015 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:43.915607 13992 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:43.916460 13992 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.915033 13953 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.857959 14016 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.857846 13977 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.861094 13978 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.840555 13982 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.878926 13909 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.878448 13958 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:43.914644 13988 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.835683 13930 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.922721 13958 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:43.921108 13996 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.914548 14004 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:43.918965 13992 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:43.920167 13997 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.917825 13986 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.914535 14002 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:44.077126 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:44.255712 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.000s	sys 0.002s
W20260709 12:11:44.256464 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.000s	sys 0.003s
W20260709 12:11:44.257896 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.000s	sys 0.015s
W20260709 12:11:44.258365 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.000s	sys 0.015s
I20260709 12:11:44.267412 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 60) took 2057 ms. Trace:
I20260709 12:11:44.268327 12645 rpcz_store.cc:276] 0709 12:11:42.210244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:42.232151 (+ 21907us) service_pool.cc:224] Handling call
0709 12:11:42.232631 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:42.309832 (+ 77201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:42.309872 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:42.310374 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:11:42.740569 (+430195us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:43.326173 (+585604us) log.cc:844] Serialized 14 byte log entry
0709 12:11:43.332919 (+  6746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:43.333073 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:44.266443 (+933370us) raft_consensus.cc:1685] finished
0709 12:11:44.266511 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:44.267052 (+   541us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117510,"prepare.queue_time_us":893718,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":457229,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":13998,"thread_start_us":634274,"threads_started":2,"wal-append.queue_time_us":1045133}
W20260709 12:11:44.289000 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.021s	sys 0.006s
W20260709 12:11:44.289909 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.022s	sys 0.006s
W20260709 12:11:44.290534 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 564473us
W20260709 12:11:44.294885 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 490184us
I20260709 12:11:43.828732 13260 rpcz_store.cc:276] 0709 12:11:42.500820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:42.501017 (+   197us) service_pool.cc:224] Handling call
0709 12:11:42.501494 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:42.539274 (+ 37780us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:42.539318 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:43.102978 (+563660us) log.cc:844] Serialized 26 byte log entry
0709 12:11:43.683205 (+580227us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:43.683250 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:43.683323 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:43.815063 (+131740us) raft_consensus.cc:1685] finished
0709 12:11:43.815084 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:43.815231 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":826765,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":579723,"threads_started":1,"wal-append.queue_time_us":703335}
I20260709 12:11:44.309708 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:44.315062 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:44.329401 13755 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:43.833721 13756 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:44.335289 13756 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:44.336611 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:44.335103 13755 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.343791 13755 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.344920 13864 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:44.346405 13864 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.348011 13755 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:44.336433 13756 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.331032 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:44.332000 13751 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:44.341676 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:44.350981 13756 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:44.351725 13864 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.352692 13864 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.353662 13864 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:43.836210 13893 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:44.429694 13756 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.430995 13952 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:44.433903 13893 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.436554 13893 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.431013 13756 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.444533 13893 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:44.457894 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:44.469888 13756 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:44.782737 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.004s	sys 0.009s
W20260709 12:11:44.783350 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.004s	sys 0.009s
I20260709 12:11:44.829829 14001 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.830369 13988 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.831189 14001 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.831452 13988 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.837704 14001 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:44.841370 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.052s	user 0.023s	sys 0.034s
I20260709 12:11:44.839710 13864 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.856535 13755 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:44.857372 14017 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.857788 13755 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:44.856721 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:44.858477 14017 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:44.863802 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.076s	user 0.023s	sys 0.036s
I20260709 12:11:44.864097 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1318) took 3408 ms. Trace:
I20260709 12:11:44.864727 14024 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:44.865219 12563 rpcz_store.cc:276] 0709 12:11:41.436906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.437081 (+   175us) service_pool.cc:224] Handling call
0709 12:11:41.437419 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:41.572393 (+134974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:44.837746 (+3265353us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54003e0154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:44.842609 (+  4863us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238902144}
I20260709 12:11:44.865687 14024 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.868531 14024 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:44.870610 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.221s	user 0.013s	sys 0.007s
W20260709 12:11:44.871109 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.269s	user 0.013s	sys 0.007s
I20260709 12:11:44.873265 13996 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.875209 13996 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.877861 14027 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:44.878437 13992 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.879106 14027 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.879932 13992 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.881884 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:44.882977 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:44.883744 14027 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:44.884395 13756 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.885458 13756 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.851845 14012 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:44.889004 13756 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:44.889386 14012 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.891929 13996 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:44.895376 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.012s	sys 0.045s
I20260709 12:11:44.892786 14012 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.878131 14004 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:44.848115 13997 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:11:44.895896 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.012s	sys 0.045s
I20260709 12:11:44.842238 13952 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.906183 12571 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:44.905862 13997 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:44.907589 14004 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:44.909868 13983 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.121s	user 0.003s	sys 0.012s
I20260709 12:11:44.910102 13996 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:44.913324 14001 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.914377 14001 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.917109 14001 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.918258 13988 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.922055 13755 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:44.927124 14004 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:44.932307 14004 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.933238 14004 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.935534 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1469) took 1581 ms. Trace:
I20260709 12:11:44.895428 13992 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:44.936055 12565 rpcz_store.cc:276] 0709 12:11:43.353549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:43.356989 (+  3440us) service_pool.cc:224] Handling call
0709 12:11:43.649546 (+292557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:44.235379 (+585833us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:44.235476 (+    97us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:44.235585 (+   109us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:44.235652 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:44.935010 (+699358us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:44.935074 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:44.935314 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333141888}
I20260709 12:11:44.940867 14017 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.925508 13952 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.972200 13908 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:44.972929 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:44.973870 13908 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.964461 13996 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.974290 12565 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:44.977922 13996 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:44.977386 13908 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.973145 14029 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:44.981766 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:44.982971 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:44.984967 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:44.985875 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:44.988391 13910 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:44.981848 14029 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:44.997845 14029 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:44.999014 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:44.970099 13952 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:45.020198 13952 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:45.020504 12571 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
W20260709 12:11:45.004436 13975 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.731s	user 0.004s	sys 0.014s
W20260709 12:11:45.009681 13979 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.221s	user 0.009s	sys 0.002s
I20260709 12:11:45.010623 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:45.014578 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1464) took 1692 ms. Trace:
I20260709 12:11:45.023180 12568 rpcz_store.cc:276] 0709 12:11:43.322305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:43.322766 (+   461us) service_pool.cc:224] Handling call
0709 12:11:43.333663 (+ 10897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:43.780630 (+446967us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:43.780676 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:43.780750 (+    74us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:43.780802 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:44.945492 (+1164690us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:44.945586 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:45.014301 (+ 68715us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1027247872}
I20260709 12:11:45.028368 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:45.031005 13910 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:45.031821 13910 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:45.032317 13910 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:45.032961 14017 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:45.030874 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1408) took 3573 ms. Trace:
I20260709 12:11:45.033849 12640 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:45.039425 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.040799 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.041129 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.042255 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.033533 14017 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.043291 14017 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:45.041203 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.043502 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.038764 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.047372 12568 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:45.039670 13952 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.034018 12641 rpcz_store.cc:276] 0709 12:11:41.456669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.462057 (+  5388us) service_pool.cc:224] Handling call
0709 12:11:41.488590 (+ 26533us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:41.489019 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:41.489063 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:41.489683 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:11:41.830399 (+340716us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:41.893245 (+ 62846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:41.893365 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:45.002908 (+3109543us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:11:45.029512 (+ 26604us) raft_consensus.cc:1685] finished
0709 12:11:45.029546 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:45.029995 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2683,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":148,"spinlock_wait_cycles":32547712,"thread_start_us":229032,"threads_started":1,"wal-append.queue_time_us":685085}
I20260709 12:11:45.037051 12640 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:45.053298 13905 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.063321 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.063941 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.065811 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.248557 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:45.064635 12568 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:45.249208 13905 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:45.250349 13905 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:45.251061 13992 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.251873 13992 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:45.253237 13905 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.128927 13996 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:45.254937 13996 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.259063 13996 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:45.275821 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.277657 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.278501 12564 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:45.275921 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.288715 13903 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.068517 12565 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:45.340374 13903 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.341563 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.342669 12565 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:45.342782 13992 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.343768 13903 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:45.343988 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.345160 12565 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:45.069482 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.345860 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.347249 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.069440 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:45.371062 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:45.440368 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:45.442737 13988 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.425427 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:45.448562 13988 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.460988 13988 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:45.465242 12537 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 2.054s (SENT)
I20260709 12:11:45.466169 14017 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.466605 13907 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:45.494462 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:45.503875 13908 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:45.513365 13910 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.520758 14017 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.537801 13908 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:45.540092 13988 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:45.542457 13908 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.543464 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:45.546096 13931 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:45.550990 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49818us
I20260709 12:11:45.552444 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:45.554030 12640 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:45.555436 14014 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:45.556129 14014 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:45.560449 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1407) took 4104 ms (client timeout 2213 ms). Trace:
W20260709 12:11:45.561378 12640 rpcz_store.cc:269] 0709 12:11:41.456312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.461942 (+  5630us) service_pool.cc:224] Handling call
0709 12:11:45.000261 (+3538319us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b4c004a92c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3a96644 00007ffbf3abdf2d 00007ffbf3abdb93 00007ffbf57ee18b 00007ffbf57eecb5 00007ffbf57e9f30 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17
0709 12:11:45.560343 (+560082us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4009856}
I20260709 12:11:45.561723 13990 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:45.568722 13988 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.569805 13988 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.572942 13988 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.573045 13981 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.573807 13988 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:45.576404 13990 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.578469 13988 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.584688 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.585577 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.585738 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:45.588933 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.589509 13988 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.591591 13910 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:45.586205 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 206) took 2838 ms (client timeout 1864 ms). Trace:
I20260709 12:11:45.590586 13988 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:45.605935 13988 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:45.606534 13988 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:45.608376 14004 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.597051 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:45.592470 13258 rpcz_store.cc:269] 0709 12:11:42.748068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:42.789806 (+ 41738us) service_pool.cc:224] Handling call
0709 12:11:45.584266 (+2794460us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:45.586097 (+  1831us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133996800}
I20260709 12:11:45.578408 13990 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.603772 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.615615 13262 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:45.617429 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.618152 13990 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.620258 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.621536 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:45.621575 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 204) took 3063 ms. Trace:
I20260709 12:11:45.630975 13259 rpcz_store.cc:276] 0709 12:11:42.581331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:42.581522 (+   191us) service_pool.cc:224] Handling call
0709 12:11:45.620526 (+3039004us) spinlock_profiling.cc:243] Waited 7.13 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:45.620841 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:45.621066 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:45.621090 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:45.621162 (+    72us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:45.621212 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:45.621261 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:45.621367 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16407808}
I20260709 12:11:45.621592 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:45.622381 12646 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:45.627765 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.628049 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:45.637728 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.639621 13256 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
W20260709 12:11:45.641776 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.657s	user 0.026s	sys 0.061s
W20260709 12:11:45.642665 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.658s	user 0.026s	sys 0.062s
I20260709 12:11:45.645231 13932 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.645475 13910 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:45.646170 13932 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:45.651152 13931 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:45.652016 13931 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:45.649329 13932 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:45.662112 13931 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:45.666016 12460 connection.cc:256] RPC call timeout handler was delayed by -1.6227843090000533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260709 12:11:45.672672 12460 connection.cc:256] RPC call timeout handler was delayed by -1.3149843090000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22060000000000002s for a response
W20260709 12:11:45.673246 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76421.3us
I20260709 12:11:45.669858 13945 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:45.677419 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:45.677924 13945 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:45.678365 13945 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:45.679240 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:45.680604 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:45.680893 13962 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 3.872s	user 0.007s	sys 0.045s
I20260709 12:11:45.702328 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:11:45.710805 13951 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:11:45.719940 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.720917 12567 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:45.738849 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1466) took 2402 ms. Trace:
I20260709 12:11:45.684010 13988 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:45.739043 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52012us
I20260709 12:11:45.674909 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.741777 12572 rpcz_store.cc:276] 0709 12:11:43.336838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:43.337038 (+   200us) service_pool.cc:224] Handling call
0709 12:11:43.337488 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:43.780601 (+443113us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:43.780672 (+    71us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:43.780751 (+    79us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:43.780807 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:45.738217 (+1957410us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:45.738317 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:45.738615 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1018280832}
I20260709 12:11:45.746336 13988 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.122733 13947 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.124138 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:45.784811 13935 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.788807 14011 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:45.816308 14045 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:46.126519 14045 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.126581 13947 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.129334 13947 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:45.816943 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.550s	user 0.007s	sys 0.034s
I20260709 12:11:45.826488 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1461) took 2606 ms. Trace:
I20260709 12:11:46.130275 12569 rpcz_store.cc:276] 0709 12:11:43.220449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:43.220599 (+   150us) service_pool.cc:224] Handling call
0709 12:11:43.220930 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:43.675208 (+454278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:43.675238 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:43.675284 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:43.675316 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:45.825684 (+2150368us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:45.825792 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:45.826215 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1044182784}
I20260709 12:11:46.131893 14045 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.833511 14049 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.834201 14048 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.132041 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:45.837107 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4213432239998838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
W20260709 12:11:46.134208 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4049649230000796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
W20260709 12:11:46.134676 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3711690600000566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
I20260709 12:11:46.134589 14049 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.135370 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.135530 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:46.136538 13992 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:46.136749 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:45.839038 14051 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.125463 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.137178 13992 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:46.138057 14051 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.138573 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:46.138706 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:46.139372 12569 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:46.139046 13992 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.140453 14051 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.846686 14052 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:45.860646 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1468) took 2515 ms. Trace:
I20260709 12:11:46.141104 13890 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:11:45.846335 12462 connection.cc:256] RPC call timeout handler was delayed by -1.7830989239998871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
I20260709 12:11:46.141314 14052 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.144629 12497 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:11:45.922456 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.138132 12641 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:46.146289 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.146865 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.147647 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.908413 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:45.936193 14053 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.949820 13994 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.967764 12393 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:45.976971 14014 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:45.987022 13887 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:45.988233 13932 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:45.988287 14054 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:46.007712 13256 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:46.009004 14056 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:46.007015 14055 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:46.011629 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.013824 14057 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.014868 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.015051 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.020202 13931 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.024524 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.028945 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.148658 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.029350 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.031587 14058 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.036159 14059 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.042189 13910 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.044348 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.046044 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.052528 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.053407 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.054277 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.055423 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.056143 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.039706 13945 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.061205 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.063351 13997 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:46.068423 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.069681 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.149160 14053 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.084051 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.100821 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.114868 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.116639 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:46.130379 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.864s	user 0.008s	sys 0.034s
I20260709 12:11:45.829813 14047 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:45.835018 14050 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.137241 14049 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:46.140347 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:45.902776 13903 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.144233 14052 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:45.924676 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.071535 13896 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:11:46.165181 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 484692us
I20260709 12:11:46.430011 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:46.430796 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:46.141227 12566 rpcz_store.cc:276] 0709 12:11:43.344941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:43.345079 (+   138us) service_pool.cc:224] Handling call
0709 12:11:45.859435 (+2514356us) spinlock_profiling.cc:243] Waited 7.83 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:45.859776 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:45.860108 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:45.860139 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:45.860201 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:45.860231 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:45.860296 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:45.860432 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18005760}
I20260709 12:11:46.431547 13262 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.244801 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 67) took 1763 ms. Trace:
I20260709 12:11:46.156172 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.342585 12569 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:46.440415 12570 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:46.439668 12571 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.261199 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:46.437990 12637 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.429877 12564 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:46.448675 13952 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.439280 13261 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.448599 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:46.449412 13952 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.449978 13998 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.450356 13998 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.450726 13998 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:46.451949 13952 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:46.452620 13952 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.453001 13952 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.453433 13952 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:46.439838 12645 rpcz_store.cc:276] 0709 12:11:44.482008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:44.741537 (+259529us) service_pool.cc:224] Handling call
0709 12:11:44.825189 (+ 83652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:44.910296 (+ 85107us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:44.910355 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:44.911714 (+  1359us) log.cc:844] Serialized 26 byte log entry
0709 12:11:44.970987 (+ 59273us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:44.979965 (+  8978us) log.cc:844] Serialized 14 byte log entry
0709 12:11:44.980905 (+   940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:46.171676 (+1190771us) spinlock_profiling.cc:243] Waited 84.4 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:46.171782 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:46.171863 (+    81us) raft_consensus.cc:1685] finished
0709 12:11:46.244519 (+ 72656us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:46.244733 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8376,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":1705,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":5713,"spinlock_wait_cycles":194225024,"thread_start_us":58725,"threads_started":1,"wal-append.queue_time_us":62163,"wal-append.run_cpu_time_us":4572,"wal-append.run_wall_time_us":1010012}
I20260709 12:11:46.466145 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.467196 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.467758 12645 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:46.468459 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.469074 12645 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:46.459232 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:46.310961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 158967us
I20260709 12:11:46.431933 13928 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:46.474851 13928 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:46.475370 13928 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:46.475575 12497 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:46.149840 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.477596 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.477763 12564 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:46.477885 14055 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:46.481407 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 113569us
I20260709 12:11:46.482492 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.464327 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.483229 12563 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.483531 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.156191 13256 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:46.484920 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.261978 13914 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.487186 13914 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.432324 13945 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:46.493217 14054 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.494731 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:46.496577 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:46.156559 13932 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.165391 14004 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.165867 14035 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:46.273025 13788 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.491724 13914 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:46.485381 14058 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.498368 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.498519 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.165786 14038 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.493085 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:46.482692 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.507042 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.484974 13907 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:46.519050 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:46.464753 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 311576us
I20260709 12:11:46.519040 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.463608 12565 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:46.535506 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:46.520407 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87466.1us
I20260709 12:11:46.537081 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.501075 14054 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.499157 12646 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:46.485232 13255 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.154405 14048 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.514812 12644 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.514645 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.539076 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.533249 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:46.536094 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:46.539714 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:46.538862 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.545024 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.152220 13887 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.501513 13955 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:46.480309 13910 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.475584 13931 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.576565 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:46.501258 14004 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.489388 13903 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.548444 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.489710 14047 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.550659 14055 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:46.551117 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.508704 14057 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.610791 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.485594 13896 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.553015 14058 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.530941 13890 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:46.431205 14059 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.555341 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.555464 14053 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.556664 13998 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.545526 13997 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.608817 14060 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:46.534049 14052 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.549038 13907 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.563789 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.530444 14050 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.530355 14051 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.530169 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.550652 13932 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.566202 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.569247 13952 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.513365 14035 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:11:46.579730 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43734.6us
I20260709 12:11:46.518342 14038 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.657645 14038 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:46.549389 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.658777 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.658898 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.596295 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.546770 13788 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.660465 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.661144 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:46.661726 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.662014 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.662258 13788 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:46.570612 13928 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.599326 12490 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.655067 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:46.609782 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.611439 12570 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:46.591967 14014 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.617884 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.482913 14056 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:46.598436 14004 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:46.615904 12537 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27144fedf0014de09f007ac6cc3f7275. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:11:46.674384 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 97187.9us
I20260709 12:11:46.613309 13988 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.561772 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.645613 13890 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:11:46.642308 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66901us
I20260709 12:11:46.651000 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.648273 12497 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:46.649406 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:46.682516 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.670094 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:46.663715 14035 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:46.671005 12491 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:46.639919 13997 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.644356 14052 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.676457 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.677065 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:46.673062 14014 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.644950 13914 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:46.668893 14060 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.677842 14070 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:46.680195 13951 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:46.667078 12571 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:46.683252 12637 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:46.635951 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73172.7us
I20260709 12:11:46.687767 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.688414 12643 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:46.691617 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.677187 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.680382 13932 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:46.678110 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.652007 14051 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.701048 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.680491 13988 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:46.617715 13955 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:46.696904 13907 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:46.703751 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:46.697363 14052 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:46.707170 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.704720 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:46.708096 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.728091 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.725448 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.055172 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:46.825510 12570 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:46.710367 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:11:47.058357 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 201876us
I20260709 12:11:46.694978 13997 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:46.690783 14061 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:46.692750 14067 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:46.901527 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 220122us
I20260709 12:11:47.265885 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:47.110134 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 399805us
I20260709 12:11:47.106408 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.265885 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.101154 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:47.224404 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.118024 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1246) took 1201 ms. Trace:
I20260709 12:11:47.150287 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:11:47.313163 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 212091us
I20260709 12:11:47.311693 12643 rpcz_store.cc:276] 0709 12:11:45.917027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:46.601802 (+684775us) service_pool.cc:224] Handling call
0709 12:11:46.619900 (+ 18098us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:47.060603 (+440703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:47.117673 (+ 57070us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:47.117934 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":138488576}
I20260709 12:11:47.153440 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:46.729847 14058 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:46.731182 13994 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:47.331866 13994 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:47.332559 13994 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:47.096208 13952 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.335310 13928 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:47.128561 13932 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.338219 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.339511 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.340957 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.342934 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:46.770401 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800192060 after lost signal to thread 11313
I20260709 12:11:46.816147 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:46.823246 13998 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.129210 13992 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:47.131373 14070 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:47.106235 14047 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.160250 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.323344 14050 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.318421 13945 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:11:47.323408 14077 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:11:47.329370 14048 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:46.708612 13951 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:47.064486 13887 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.824322 13910 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.064980 14051 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:47.344857 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.004s	sys 0.032s
W20260709 12:11:47.392822 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.004s	sys 0.032s
W20260709 12:11:47.393330 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 688898us
I20260709 12:11:46.709751 13914 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.094596 14056 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:47.044003 14060 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:46.841995 13931 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:47.267503 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.198372 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.219632 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.065243 13988 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.345966 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:46.701578 14014 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.074472 13903 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.195938 14074 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:47.286008 13955 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.308010 14067 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:47.221529 14059 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.365892 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.129935 14045 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:11:47.365154 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56184.2us
I20260709 12:11:47.365818 12493 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:47.386502 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.130173 14071 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:47.427838 14071 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.430626 14071 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.398151 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1431) took 1548 ms. Trace:
I20260709 12:11:47.398586 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.398584 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.361557 14057 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:47.309964 14061 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:47.411466 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65592.3us
I20260709 12:11:47.440864 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:47.414057 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:47.389549 13951 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.421200 13947 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:47.403050 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79838.9us
I20260709 12:11:46.900748 13890 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.423024 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:47.414013 14038 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.750s	user 0.019s	sys 0.004s
I20260709 12:11:47.346480 14058 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.399188 13896 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.424190 13952 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:47.425412 14045 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.431290 12645 rpcz_store.cc:276] 0709 12:11:45.849550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:46.469484 (+619934us) service_pool.cc:224] Handling call
0709 12:11:46.555204 (+ 85720us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:47.397956 (+842752us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b54000a6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:47.398074 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1204658944}
I20260709 12:11:47.434504 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1421) took 1586 ms. Trace:
I20260709 12:11:47.436357 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:47.410923 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3be0 after lost signal to thread 12429
I20260709 12:11:47.414722 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:47.385288 14070 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.432555 14067 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.464027 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:47.479923 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:47.479717 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.453306 13903 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:47.463920 12644 rpcz_store.cc:276] 0709 12:11:45.847666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:46.579399 (+731733us) service_pool.cc:224] Handling call
0709 12:11:46.597904 (+ 18505us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:46.618320 (+ 20416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:46.618339 (+    19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:46.675375 (+ 57036us) log.cc:844] Serialized 506 byte log entry
0709 12:11:46.837728 (+162353us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:46.837797 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:46.837921 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:47.434170 (+596249us) raft_consensus.cc:1685] finished
0709 12:11:47.434199 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:47.434420 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":100647,"threads_started":1,"wal-append.queue_time_us":562497}
I20260709 12:11:47.461199 14057 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:47.488655 14057 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:47.489084 14057 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.455972 13947 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:47.486567 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:11:47.478261 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3ca0 after lost signal to thread 12443
I20260709 12:11:47.485739 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:47.546129 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103707us
W20260709 12:11:47.547148 14079 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:11:47.549132 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.467756 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:47.556092 12496 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:47.559147 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.548213 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:47.494757 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:47.566727 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.567843 12645 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:11:47.567894 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63472.7us
W20260709 12:11:47.572567 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86508.2us
I20260709 12:11:47.484249 13928 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:47.485409 13903 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:47.575368 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155859us
I20260709 12:11:47.438652 14074 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.486874 14082 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:47.548832 13947 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.584270 13903 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.588649 13914 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:47.588950 14061 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.483072 14047 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
W20260709 12:11:47.589619 14004 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.889s	user 0.017s	sys 0.005s
I20260709 12:11:47.584991 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:47.588972 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.590632 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.461319 14038 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.567320 13987 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:11:47.595582 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110448us
I20260709 12:11:47.572477 13945 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:47.612299 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.613775 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:47.606698 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:47.606060 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 125386us
I20260709 12:11:47.584625 13955 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:47.609696 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:47.610368 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.609346 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.619589 12568 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.613696 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.589787 14052 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.621050 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:47.593309 12491 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:47.615686 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.594688 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.610985 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:11:47.583030 13258 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:47.625692 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:47.628692 12646 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:47.602033 13887 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:47.586840 14082 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:47.636436 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.636550 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:47.611498 13987 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:47.638698 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:47.610575 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.645465 14059 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:47.645566 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.659240 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.659745 12489 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:11:47.631546 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800363c20 after lost signal to thread 12521
I20260709 12:11:47.663756 12494 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:47.612552 14004 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.650106 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:47.650590 14048 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:47.674911 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:47.632310 14078 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20260709 12:11:47.658784 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.630982 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:47.646575 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.619328 13955 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:47.670522 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.651376 14074 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.630955 14088 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:47.676110 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.634070 13932 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.678344 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.703603 12495 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:47.660809 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.663295 14059 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:47.705106 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:47.663749 12518 debug-util.cc:398] Leaking SignalData structure 0x7b080029fe60 after lost signal to thread 12522
I20260709 12:11:47.707398 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.678331 14048 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:47.711988 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.686018 12497 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:47.658766 13987 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:47.715840 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.710103 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.717192 12496 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:47.622180 13945 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.713259 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:47.684846 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48363.3us
I20260709 12:11:47.693423 13955 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.721764 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.638722 14082 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.715677 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.658524 13994 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.706789 14059 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.738689 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.703413 14088 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:11:47.727702 14080 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.181s	user 0.002s	sys 0.006s
I20260709 12:11:47.730508 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:47.730226 13947 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:47.734651 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118825us
I20260709 12:11:47.734385 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:47.740358 12493 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:47.741050 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53043.9us
I20260709 12:11:47.716329 14048 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:47.753378 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.753548 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:47.756738 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:47.756932 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.746439 14088 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:47.756353 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08002f7a00 after lost signal to thread 12596
I20260709 12:11:47.751027 14070 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.774744 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:47.777503 13994 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.776912 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:47.755656 14058 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.786062 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.790112 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.790170 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.793973 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.781250 13887 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:47.782649 14089 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:47.799608 12495 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:47.780565 14060 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:47.800151 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:47.801052 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1263) took 1157 ms. Trace:
I20260709 12:11:47.802246 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.804755 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1384) took 1255 ms. Trace:
I20260709 12:11:47.833848 12494 rpcz_store.cc:276] 0709 12:11:46.544015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:46.578550 (+ 34535us) service_pool.cc:224] Handling call
0709 12:11:47.799237 (+1220687us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:47.799403 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2280981376}
I20260709 12:11:47.835167 12646 rpcz_store.cc:276] 0709 12:11:46.643764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.124233 (+480469us) service_pool.cc:224] Handling call
0709 12:11:47.162286 (+ 38053us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:47.629149 (+466863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:47.800782 (+171633us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:47.800959 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1072123136}
I20260709 12:11:47.836674 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.837983 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.839393 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.840018 12646 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:47.840898 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:47.843406 14082 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.846661 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:47.848304 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:47.850153 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51039.5us
I20260709 12:11:47.851178 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:47.833130 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800366600 after lost signal to thread 12670
W20260709 12:11:47.854700 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84643.1us
I20260709 12:11:47.854790 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.798089 14058 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:47.801199 13887 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:47.860213 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:47.857914 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:47.806318 14052 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.832553 14038 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.851217 14082 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.852138 14056 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:11:47.859598 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800366560 after lost signal to thread 12671
I20260709 12:11:47.879187 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:47.880324 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:47.880954 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.879528 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.879925 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.881347 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.833950 14051 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.889874 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.892961 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:47.888613 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:47.803987 13947 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:47.880790 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.865001 13887 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:47.860244 14091 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:47.874429 14004 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:47.902856 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.903160 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.860383 13932 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:11:47.866214 14060 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:47.850801 14067 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.856765 14096 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:47.897919 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.904726 12488 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:47.905866 13988 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:47.893863 14084 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:47.908406 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.915644 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.917989 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.916675 12493 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:47.922088 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.923238 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.924222 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.875200 14057 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:47.899906 14091 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:47.919957 14084 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:11:47.920300 12518 debug-util.cc:398] Leaking SignalData structure 0x7b080037a9c0 after lost signal to thread 13286
I20260709 12:11:47.966643 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.966849 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.973961 14084 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:47.918838 14075 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:47.975199 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:47.976817 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.977555 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:47.981846 13255 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:47.983717 14057 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:47.983834 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:47.979240 14075 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:47.926831 13947 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:47.987146 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.003446 12489 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:47.987432 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.986845 14007 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:48.011972 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.002929 13947 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:48.013937 12491 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:11:48.010816 12641 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:47.981693 14101 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:48.024554 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:47.978649 14091 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.017084 13947 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:48.027722 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.031849 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.032716 12641 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:48.032972 14045 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:48.033602 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.034303 12641 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:48.019819 14051 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.035883 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:48.036798 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145421us
I20260709 12:11:48.036525 12491 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:48.043907 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.046687 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.035283 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.023797 14075 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.068073 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.074379 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.075238 12493 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:48.076272 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:48.077800 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.045373 13988 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:48.081933 14106 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:48.083603 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.082662 14106 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.085990 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.087088 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.045197 13903 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.095183 13896 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:11:48.101480 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:48.101941 13997 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.746s	user 0.008s	sys 0.023s
I20260709 12:11:48.106832 14059 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.107774 13896 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.108457 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.118638 12491 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:48.117744 14059 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.118275 13997 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.130107 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.130868 12489 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:48.131230 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.132977 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.133865 12490 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:48.127350 13997 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:48.368408 12641 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:48.137375 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.204329 14084 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.172031 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.210115 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:48.376837 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.518s (SENT)
I20260709 12:11:48.172676 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:48.452885 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 200903us
W20260709 12:11:48.453390 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 280840us
I20260709 12:11:48.248562 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.454665 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.454890 14045 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:48.455031 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 206620us
I20260709 12:11:48.205422 13987 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:48.424335 14088 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.641s	user 0.015s	sys 0.004s
I20260709 12:11:48.478598 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:48.460821 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:48.464968 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:48.134568 14035 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.414s	user 0.011s	sys 0.026s
I20260709 12:11:48.482535 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.163281 13903 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:48.374688 13887 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:48.449103 14048 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.684s	user 0.004s	sys 0.014s
I20260709 12:11:48.488981 14088 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.500499 13259 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:48.456822 13997 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:48.487267 14084 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.502854 14048 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.487810 14094 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:48.488270 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.476454 14070 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:48.506567 14048 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:48.488973 14091 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.507302 12565 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:48.487263 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.490338 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.125593 14089 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:48.482705 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105936us
I20260709 12:11:48.230963 14106 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:48.512221 12610 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.523s (SENT)
I20260709 12:11:48.495402 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.496753 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.516091 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.372577 13788 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:48.460376 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.523321 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.523769 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.486573 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.525538 12497 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:48.510637 12563 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.519870 14091 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.512763 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:48.534288 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.535421 12491 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:48.517591 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:48.536643 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:48.537853 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77101.2us
W20260709 12:11:48.540357 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:48.542063 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:48.544593 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.546866 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.522872 14051 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:48.523138 14059 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:48.531875 13951 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.050s	user 0.020s	sys 0.010s
I20260709 12:11:48.125593 13955 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.501596 14045 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 12:11:48.558807 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:48.565693 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53506.5us
W20260709 12:11:48.567487 12610 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.537s (SENT)
I20260709 12:11:48.567754 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.560920 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:48.569806 14014 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.128s	user 0.001s	sys 0.025s
I20260709 12:11:48.551481 13896 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.568935 12494 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:48.570281 14014 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.571547 12638 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.571527 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.572969 14014 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:48.550092 14038 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:48.580524 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.581508 12489 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
W20260709 12:11:48.564498 13947 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.527s	user 0.011s	sys 0.002s
I20260709 12:11:48.557516 13914 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:48.499553 14035 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.536274 13788 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.593727 14038 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:48.606745 14038 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.611142 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1593) took 1278 ms. Trace:
I20260709 12:11:48.611691 12488 rpcz_store.cc:276] 0709 12:11:47.344379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.391436 (+ 47057us) service_pool.cc:224] Handling call
0709 12:11:47.411554 (+ 20118us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:47.914787 (+503233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.602720 (+687933us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:48.603004 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1129213824}
I20260709 12:11:48.625190 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:48.565026 13951 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.630030 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.600514 14067 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.593195 14109 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:48.640260 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149607us
I20260709 12:11:48.622412 14051 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:48.592751 13947 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.644382 12642 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:48.644411 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.819s (SENT)
I20260709 12:11:48.645172 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:48.645448 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 83) took 1053 ms. Trace:
I20260709 12:11:48.646071 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.641955 14051 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:48.647068 14051 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:48.640715 12641 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:48.645854 12637 rpcz_store.cc:276] 0709 12:11:47.592102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.443616 (+851514us) service_pool.cc:224] Handling call
0709 12:11:48.443936 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.444143 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:11:48.569328 (+125185us) log.cc:844] Serialized 14 byte log entry
0709 12:11:48.643988 (+ 74660us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:48.644058 (+    70us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:48.644113 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.645237 (+  1124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:48.645384 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":144,"thread_start_us":191071,"threads_started":2}
I20260709 12:11:48.641402 14109 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:11:48.669991 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128007us
I20260709 12:11:48.671870 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.665298 13890 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:48.677439 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127236us
I20260709 12:11:48.678377 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.673489 14067 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:48.681406 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.673724 13887 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.672912 14109 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.680305 13987 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.717504 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.683353 14101 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:50128 (request call id 10) took 7710 ms. Trace:
W20260709 12:11:48.696743 14110 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:11:48.703207 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.703625 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:48.704936 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.711730 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.709743 14109 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.712339 14061 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:48.713543 13788 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:48.714007 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.715262 13947 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.716214 13951 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:48.731237 14061 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:48.686443 14115 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 12:11:48.732687 12645 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:48.755339 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71924.8us
I20260709 12:11:48.757040 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1277) took 1171 ms. Trace:
I20260709 12:11:48.757421 12645 rpcz_store.cc:276] 0709 12:11:47.585065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.605535 (+ 20470us) service_pool.cc:224] Handling call
0709 12:11:48.713844 (+1108309us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:48.755970 (+ 42126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12349696}
I20260709 12:11:48.758728 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.758858 14035 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:48.759300 12645 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:48.760524 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:48.735877 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.734647 14056 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:48.762014 13928 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:48.745769 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1262) took 2102 ms. Trace:
W20260709 12:11:48.732340 13253 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 1.122s	user 0.002s	sys 0.027s
I20260709 12:11:48.763528 14056 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:48.768283 13928 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:48.734269 14061 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.771168 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.737764 14115 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:48.783948 14115 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:48.772418 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:48.784813 14115 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.736035 14109 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.785845 14115 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:48.786440 14120 ts_tablet_manager.cc:933] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:48.786814 14056 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.787896 14056 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.788264 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.772642 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.772744 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:48.747221 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 82) took 1155 ms. Trace:
I20260709 12:11:48.771353 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.774487 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1272) took 2017 ms. Trace:
I20260709 12:11:48.793300 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 106) took 1008 ms. Trace:
I20260709 12:11:48.793826 12565 rpcz_store.cc:276] 0709 12:11:47.784875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.787589 (+  2714us) service_pool.cc:224] Handling call
0709 12:11:47.833009 (+ 45420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.507740 (+674731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.793063 (+285323us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:48.793244 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1550242048}
I20260709 12:11:48.775799 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.776479 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.753509 13955 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:48.792695 12638 rpcz_store.cc:276] 0709 12:11:46.756924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.102983 (+346059us) service_pool.cc:224] Handling call
0709 12:11:47.150851 (+ 47868us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.572514 (+1421663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.774226 (+201712us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:48.774423 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3266409344}
I20260709 12:11:48.790820 14088 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:48.780709 13253 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.173s	user 0.002s	sys 0.028s
I20260709 12:11:48.778564 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:48.778914 13887 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:48.799453 12646 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.799822 13887 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:48.734269 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.800390 13887 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.767966 12642 rpcz_store.cc:276] 0709 12:11:46.643441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.102675 (+459234us) service_pool.cc:224] Handling call
0709 12:11:47.160201 (+ 57526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.662630 (+1502429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.745416 (+ 82786us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:48.745695 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3412728704}
I20260709 12:11:48.803304 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.804363 12638 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:48.791412 13928 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:48.808522 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.792449 12641 rpcz_store.cc:276] 0709 12:11:47.591915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.443571 (+851656us) service_pool.cc:224] Handling call
0709 12:11:48.444010 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.669451 (+225441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.746922 (+ 77471us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:48.747152 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":429680768}
I20260709 12:11:48.811038 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.811255 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.812260 12641 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:48.812637 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.785377 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.791585 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:11:48.791946 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.795534 12637 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:48.816150 14075 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.816721 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.817482 12637 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:48.725564 14101 rpcz_store.cc:276] 0709 12:11:40.971446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:40.972869 (+  1423us) service_pool.cc:224] Handling call
0709 12:11:41.085026 (+112157us) log.cc:844] Serialized 506 byte log entry
0709 12:11:41.894012 (+808986us) spinlock_profiling.cc:243] Waited 4.25 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fba0a 00007ffbf40fb693 00007ffbf40fb631 00007ffbf582fb52 00007ffbf582fcf5 00007ffbf576c691 00007ffbf574ceaa 00007ffbf5753d53 00007ffbf5753cd9 00007ffbf5753c69
0709 12:11:48.678509 (+6784497us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12474,"raft.queue_time_us":50424,"raft.run_cpu_time_us":7040,"raft.run_wall_time_us":472073,"spinlock_wait_cycles":9767296,"thread_start_us":438023,"threads_started":1,"wal-append.queue_time_us":846639,"wal-append.run_cpu_time_us":10701,"wal-append.run_wall_time_us":1104773}
I20260709 12:11:48.818738 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1451) took 1173 ms. Trace:
I20260709 12:11:48.819155 12640 rpcz_store.cc:276] 0709 12:11:47.644880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.817743 (+1172863us) service_pool.cc:224] Handling call
0709 12:11:48.818106 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.818334 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:48.818364 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:48.818422 (+    58us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:48.818456 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.818526 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:48.818651 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:48.821170 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1452) took 1176 ms. Trace:
W20260709 12:11:48.822327 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43853us
I20260709 12:11:48.823634 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.825117 14071 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:48.825881 14067 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:48.826355 14067 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:48.826252 14071 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:48.826812 14067 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.830230 13255 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.830444 13256 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.822371 14050 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:48.833091 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1614) took 1093 ms. Trace:
I20260709 12:11:48.833700 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:48.834239 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 396) took 1369 ms (client timeout 1696 ms). Trace:
W20260709 12:11:48.834661 13259 rpcz_store.cc:269] 0709 12:11:47.464747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.465081 (+   334us) service_pool.cc:224] Handling call
0709 12:11:48.833997 (+1368916us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:48.834144 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2136064768}
I20260709 12:11:48.835774 13260 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:11:48.816162 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.829445 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:48.837875 13945 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:48.837991 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 86) took 1065 ms. Trace:
I20260709 12:11:48.813249 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:48.838737 14050 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:48.833755 14071 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.839386 14100 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:48.839030 12641 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.839879 14050 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:48.840956 14050 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:48.841291 14050 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:48.842991 14084 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:48.826272 14059 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:48.846064 14059 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:48.847122 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.846976 14059 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.847713 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:48.848394 14059 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.848727 13254 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 2: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:48.851752 12643 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:48.852648 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:48.854561 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:48.855315 13262 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:48.852725 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 326) took 1306 ms. Trace:
I20260709 12:11:48.856982 13260 rpcz_store.cc:276] 0709 12:11:47.546030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.549271 (+  3241us) service_pool.cc:224] Handling call
0709 12:11:48.835602 (+1286331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.836363 (+   761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.852480 (+ 16117us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400578ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:48.852626 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2895570688}
I20260709 12:11:48.857656 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:48.848006 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1442) took 1356 ms. Trace:
I20260709 12:11:48.805298 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:48.809345 12644 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.868731 12638 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:48.881068 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:48.797966 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:48.746435 14045 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:48.884922 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:48.838418 12645 rpcz_store.cc:276] 0709 12:11:47.772820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.837162 (+1064342us) service_pool.cc:224] Handling call
0709 12:11:48.837495 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.837698 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:48.837721 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:48.837753 (+    32us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:48.837776 (+    23us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.837827 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:48.837925 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:48.871026 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.886444 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.873543 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.873868 13945 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:48.888334 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.833505 12495 rpcz_store.cc:276] 0709 12:11:47.742221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.742859 (+   638us) service_pool.cc:224] Handling call
0709 12:11:47.799410 (+ 56551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:47.803922 (+  4512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.832906 (+1028984us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b5400191c50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:48.833032 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99687808}
I20260709 12:11:48.884052 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:48.874029 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.874912 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.875093 13258 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.875473 14071 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:11:48.873607 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:48.892661 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.893450 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:48.893715 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95845.1us
I20260709 12:11:48.878355 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:48.879222 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:48.843055 14118 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:48.879827 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:48.867611 12642 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:48.851819 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1276) took 1285 ms. Trace:
I20260709 12:11:48.896826 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:48.839900 14100 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:48.882915 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:48.896262 12569 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:48.896692 12646 rpcz_store.cc:276] 0709 12:11:47.566955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.158281 (+591326us) service_pool.cc:224] Handling call
0709 12:11:48.749444 (+591163us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:48.749693 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.800137 (+ 50444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.851613 (+ 51476us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:48.851756 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146691712}
I20260709 12:11:48.897890 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.783003 14035 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20260709 12:11:48.751073 14051 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.970741 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.983847 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.900924 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:48.971583 12393 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 218) took 7398 ms. Trace:
I20260709 12:11:48.982817 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.980552 12646 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.986730 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:48.985085 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100230us
I20260709 12:11:48.983723 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.984450 13262 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:48.990252 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.979468 12644 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:48.991135 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.994496 12569 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 2: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:48.995478 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.998879 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.000000 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.000573 12568 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:49.000872 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65569.7us
I20260709 12:11:49.000640 12495 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:49.003545 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.004339 12565 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:48.888568 14101 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:48.846922 14067 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.008136 12645 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:49.005136 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106939us
I20260709 12:11:48.894532 14047 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:49.008201 12495 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:49.004968 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:48.969483 14050 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.010198 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.027251 12571 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:48.995939 14070 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:11:48.972299 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:49.010943 14035 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:49.011291 12395 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:11:48.968129 13997 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:48.967957 13951 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:48.900843 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:49.000604 13947 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 12:11:49.016160 14051 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:48.975760 13945 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:49.017409 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.051303 12564 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.016213 13887 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.033912 14035 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:11:49.018712 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116815us
I20260709 12:11:49.021765 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.010262 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:49.023826 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:49.024196 12490 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:49.009078 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.026194 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.057047 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:48.993784 14100 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.010751 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.025563 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:48.974334 12641 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:48.895318 14090 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.134s
I20260709 12:11:49.076875 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:48.974678 14120 tablet_copy_client.cc:323] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
W20260709 12:11:49.040122 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:48.866291 12639 rpcz_store.cc:276] 0709 12:11:47.491848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.558729 (+ 66881us) service_pool.cc:224] Handling call
0709 12:11:48.846923 (+1288194us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:48.847940 (+  1017us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14839168}
I20260709 12:11:49.077410 12643 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:48.901280 14118 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:49.050330 13997 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.083850 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:49.084254 14118 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:48.988706 12393 rpcz_store.cc:276] 0709 12:11:41.572839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:41.590085 (+ 17246us) service_pool.cc:224] Handling call
0709 12:11:48.971450 (+7381365us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:46.637772 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:46.864404 (+226632us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:46.864493 (+    89us) write_op.cc:435] Acquired schema lock
0709 12:11:46.864536 (+    43us) tablet.cc:662] Decoding operations
0709 12:11:46.872903 (+  8367us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:46.873048 (+   145us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:46.873109 (+    61us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:46.873642 (+   533us) tablet.cc:701] Row locks acquired
0709 12:11:46.873712 (+    70us) write_op.cc:260] PREPARE: finished
0709 12:11:46.874015 (+   303us) write_op.cc:270] Start()
0709 12:11:46.874247 (+   232us) write_op.cc:276] Timestamp: P: 1783599106873910 usec, L: 0
0709 12:11:46.874295 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 12:11:46.875397 (+  1102us) log.cc:844] Serialized 536 byte log entry
0709 12:11:47.802999 (+927602us) op_driver.cc:464] REPLICATION: finished
0709 12:11:48.004617 (+201618us) write_op.cc:301] APPLY: starting
0709 12:11:48.004755 (+   138us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:48.004927 (+   172us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:48.005164 (+   237us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:48.005606 (+   442us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:48.039907 (+ 34301us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:48.052360 (+ 12453us) write_op.cc:312] APPLY: finished
0709 12:11:48.103960 (+ 51600us) log.cc:844] Serialized 23 byte log entry
0709 12:11:48.131246 (+ 27286us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:48.149123 (+ 17877us) write_op.cc:454] Released schema lock
0709 12:11:48.155037 (+  5914us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:48.155794 (+   757us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:48.155851 (+    57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16621,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":154126,"apply.run_cpu_time_us":3708,"apply.run_wall_time_us":151017,"num_ops":1,"prepare.queue_time_us":437216,"prepare.run_cpu_time_us":38196,"prepare.run_wall_time_us":639571,"raft.queue_time_us":115380,"raft.run_cpu_time_us":53995,"raft.run_wall_time_us":214586,"replication_time_us":907394,"thread_start_us":751520,"threads_started":4,"wal-append.queue_time_us":445393}]]}
I20260709 12:11:49.085670 14118 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.043937 14125 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:11:49.087131 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.087797 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.088502 12495 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:48.821538 12640 rpcz_store.cc:276] 0709 12:11:47.645166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.820160 (+1174994us) service_pool.cc:224] Handling call
0709 12:11:48.820542 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.820759 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:48.820789 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:48.820866 (+    77us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:48.820916 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:48.820983 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:48.821110 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:49.027693 14047 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.018151 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.090238 14047 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.062673 13945 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:49.070230 14085 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.052047 14070 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:49.094548 14070 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:49.095851 12564 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:49.049535 13951 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:49.064670 12644 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:11:49.097117 13951 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.077190 14093 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:49.029424 14038 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:11:49.103807 12489 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:49.088014 12497 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:49.104826 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.095669 14070 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.054893 14004 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.101383 14093 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:49.093130 13928 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.085280 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.089566 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:49.066083 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44446.2us
W20260709 12:11:49.114500 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48547.6us
W20260709 12:11:49.118844 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109844us
I20260709 12:11:49.128351 12494 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:49.129336 12571 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:49.143759 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:49.144789 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.145007 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:49.151727 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.152064 12645 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.101405 14101 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:11:49.153280 12638 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:49.154052 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.154381 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:49.154992 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83211us
W20260709 12:11:49.158483 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 257680us
W20260709 12:11:49.150102 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 175458us
I20260709 12:11:49.155390 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.156419 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:49.160718 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.183492 12497 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:49.184243 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.184442 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.182044 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:11:49.183485 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.187095 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.191442 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.192201 12495 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:49.193212 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.194720 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.196514 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.196836 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.197904 12489 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:49.198341 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:49.187101 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 157122us
I20260709 12:11:49.192034 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.206751 12564 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:11:49.142988 14124 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.207207 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:49.208202 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.208866 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:49.213441 14007 raft_consensus.cc:2764] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:11:49.213778 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.194237 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.215910 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.184307 13998 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:49.182891 14056 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.201017 14109 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.218223 14007 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:49.219762 14007 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:49.220233 14007 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:49.185302 14051 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.223766 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.224759 12495 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:49.221539 14007 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:49.229825 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.230669 14093 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.216044 14126 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.234925 13928 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:49.237423 14129 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.190124 14070 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:49.250737 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.254343 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.255487 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.238801 14047 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.211361 14090 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:49.239787 14113 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.187112 13945 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:49.211218 14125 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.218309 14101 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:49.264933 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76588.4us
I20260709 12:11:49.218619 13998 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:49.266436 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:49.266801 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64435.5us
W20260709 12:11:49.267688 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75723.8us
I20260709 12:11:49.277240 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.277472 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.277957 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:49.268512 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81500.5us
I20260709 12:11:49.269498 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.280238 12493 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:49.270913 14007 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.282083 14093 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.283593 12565 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.250872 13947 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:49.284690 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.285686 12571 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.287480 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.263005 14085 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.288631 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.270417 14051 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.289963 14129 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.289938 14051 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.283080 14091 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:49.291432 14091 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:49.292577 14051 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:49.261920 14090 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.293422 14090 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.292920 12566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:49.283289 13896 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.294138 12566 raft_consensus.cc:606] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.298210 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:49.303725 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.306604 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.274417 14126 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:49.310678 12570 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:49.315007 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.280373 13887 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:49.328775 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.292670 14091 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.279733 13945 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:49.287194 13947 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:49.331513 12568 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.295279 14071 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:49.327891 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138411us
I20260709 12:11:49.327566 14133 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.316401 14131 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.311173 14038 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.337114 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:49.315804 14126 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:49.325660 14129 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.350875 14143 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:49.351842 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72755.1us
I20260709 12:11:49.353227 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.354219 14143 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:49.354821 14143 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.326514 14130 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.357847 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.358671 12488 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:49.339128 13947 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:49.354955 12489 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:11:49.368455 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.338755 14071 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:49.383415 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54713.3us
I20260709 12:11:49.384783 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.356966 13955 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.356112 14126 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.406397 12563 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.333666 14137 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:11:49.414615 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.431668 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.432848 14045 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:49.433492 14045 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:49.433655 13928 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:11:49.434572 14045 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.434916 14140 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:49.329558 14139 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260709 12:11:49.388201 14071 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.436465 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.433672 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.438863 12488 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:49.435154 13928 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:49.399698 14130 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.439945 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.387952 14145 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.416332 13887 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:11:49.439373 14140 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:49.383401 13998 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:49.466506 14127 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:11:49.442776 13928 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:49.462323 14100 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.467372 13887 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:49.473237 13887 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:49.456812 14125 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 12:11:49.482607 14125 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:49.483094 14125 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:49.468664 14047 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:49.471444 14146 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:49.478152 13896 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:49.484820 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 147820us
I20260709 12:11:49.491053 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:11:49.492652 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.494840 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet 27144fedf0014de09f007ac6cc3f7275 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:49.445015 13945 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.495618 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet 27144fedf0014de09f007ac6cc3f7275 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-27144fedf0014de09f007ac6cc3f7275
I20260709 12:11:49.468767 14051 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:11:49.498883 13928 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.503615 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:49.504040 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:49.505067 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:49.511338 13255 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.512173 14061 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.487066 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:49.502779 14146 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:49.496863 14139 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.523799 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:49.520846 14146 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.528739 12571 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:49.526659 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:49.530254 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:49.493769 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.484053 14125 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.531742 12646 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:11:49.532370 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.513671 13955 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:49.535552 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:49.535791 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95952.8us
I20260709 12:11:49.536543 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.537889 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.523161 14051 consensus_queue.cc:1243] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:11:49.540936 14067 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:49.544639 14125 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.544276 14035 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.479175 13887 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:49.546972 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53332.4us
W20260709 12:11:49.556980 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 172304us
I20260709 12:11:49.558770 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:49.560601 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.472327 14140 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.566089 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1455) took 1519 ms. Trace:
I20260709 12:11:49.570036 12643 rpcz_store.cc:276] 0709 12:11:48.046906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.864071 (+817165us) service_pool.cc:224] Handling call
0709 12:11:49.077197 (+213126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:49.082698 (+  5501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:49.565823 (+483125us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540018d3d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:49.566004 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489039744}
I20260709 12:11:49.546643 13928 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.546278 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:49.574555 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.577741 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.556532 14056 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.557061 14139 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.551548 14124 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.583563 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.590248 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.575659 13887 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.590227 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.550644 13955 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.595247 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.581226 14014 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:49.587023 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.597436 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.601365 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.596246 14056 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.592679 14091 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.611853 12493 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:11:49.612241 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.588480 14047 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.611953 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.614894 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.615406 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.597179 13788 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:11:49.619774 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.598145 14014 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.616549 14056 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:11:49.619037 14067 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.619251 14115 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.585306 14145 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:11:49.605338 14124 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.626114 14100 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:49.628165 14014 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.648715 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1463) took 1167 ms. Trace:
I20260709 12:11:49.648993 12496 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:11:49.639142 13788 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.645471 14130 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:11:49.651494 14130 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:49.649245 12644 rpcz_store.cc:276] 0709 12:11:48.481539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.992522 (+510983us) service_pool.cc:224] Handling call
0709 12:11:49.013747 (+ 21225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:49.064528 (+ 50781us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:11:49.129068 (+ 64540us) log.cc:844] Serialized 14 byte log entry
0709 12:11:49.264883 (+135815us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:49.264945 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:49.265007 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:49.648443 (+383436us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:49.648497 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:49.648647 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":49620,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":116174208,"thread_start_us":131087,"threads_started":2,"wal-append.queue_time_us":90921}
I20260709 12:11:49.652813 14126 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:49.655236 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.656729 12581 tablet_copy_source_session.cc:215] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:49.632757 14137 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:49.626757 14154 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.663767 13253 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.688068 14153 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:49.652547 14130 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.688721 13955 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:49.689409 14130 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:11:49.695089 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104923us
I20260709 12:11:49.695960 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.697710 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:49.698063 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83323.9us
I20260709 12:11:49.698884 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.698964 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.665343 14137 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.707329 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.708331 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.708755 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.709719 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.700361 14113 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.701117 13992 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.712196 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:49.706188 12567 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:49.720697 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.704823 14137 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.726505 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.687274 14090 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.706377 14088 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.720628 14136 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:49.712126 14154 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:49.718878 14007 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.728263 13955 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:49.735898 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.737888 14148 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.738649 14120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27144fedf0014de09f007ac6cc3f7275. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:49.730623 14067 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 12:11:49.718983 14145 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:49.741101 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:49.745512 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.728083 14124 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.775794 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:49.740422 14154 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:49.779269 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.747032 14113 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:49.783356 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:49.749191 13955 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.746699 14136 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.786892 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.786950 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:11:49.787360 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66728.6us
I20260709 12:11:49.789444 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.790977 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:49.774492 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:49.790288 13955 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.753445 14145 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.773888 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:49.793856 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1459) took 1650 ms. Trace:
I20260709 12:11:49.809515 12642 rpcz_store.cc:276] 0709 12:11:48.143131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.918346 (+775215us) service_pool.cc:224] Handling call
0709 12:11:48.990953 (+ 72607us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:48.991493 (+   540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:49.793648 (+802155us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b54002e5750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:49.793790 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39829376}
I20260709 12:11:49.790450 13945 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:11:49.790588 14136 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:49.804557 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.806962 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.782708 14154 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.786157 14113 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:49.833065 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:49.790371 13788 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.791664 14056 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.813465 14145 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:49.845930 14166 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
W20260709 12:11:49.848382 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59007.5us
I20260709 12:11:49.830030 14132 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.831759 14113 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.856307 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.847194 13788 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:49.784107 14124 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.847857 14153 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.858062 14148 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.852115 14164 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:49.875936 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69091.8us
I20260709 12:11:49.860554 14126 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:49.880415 12488 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:49.881743 14156 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.866503 13788 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:49.890342 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:49.890686 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:49.891680 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:49.892103 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:49.892590 12495 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.882212 14163 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.864992 13947 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.915433 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:49.918064 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:49.918871 12644 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:49.902908 14153 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:49.886699 14162 raft_consensus.cc:2764] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:11:49.926252 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:49.933228 12563 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:49.910259 14150 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:49.918138 14090 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:49.936386 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.922012 14132 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:49.924335 14126 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.942088 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:49.952652 14050 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:49.953950 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:49.934789 13896 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:49.936419 14153 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:49.954944 13887 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:49.977468 12570 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:49.986164 12493 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:49.978897 14071 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.525s	user 0.013s	sys 0.004s
I20260709 12:11:49.969781 13887 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.943717 14137 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:49.967928 14014 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:49.987371 14122 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.008315 14153 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.012341 13987 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:50.012835 13987 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:50.013844 13987 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.015740 13987 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:50.017796 14091 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:50.018913 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1660) took 1146 ms. Trace:
I20260709 12:11:50.019150 14091 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:50.019660 14091 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.019376 12494 rpcz_store.cc:276] 0709 12:11:48.872789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.872893 (+   104us) service_pool.cc:224] Handling call
0709 12:11:50.018722 (+1145829us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:50.018845 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438045440}
I20260709 12:11:50.020819 14091 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:49.993214 14071 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.046192 12461 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:11:50.046589 12461 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:11:50.006723 14122 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:50.007633 13887 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.047698 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:11:50.049707 14146 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.512s	user 0.009s	sys 0.006s
W20260709 12:11:50.045117 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156723us
I20260709 12:11:50.011937 13947 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.056865 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.057542 12493 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.058715 12493 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:11:50.060299 14143 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.702s	user 0.015s	sys 0.000s
I20260709 12:11:50.059892 12493 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.061259 12493 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.061808 13261 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.045985 14088 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:50.060781 14143 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.057148 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:50.052793 13928 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:11:50.073855 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:11:50.069731 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.062275 14091 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:50.057817 14061 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.082752 12572 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.059895 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:50.061482 14071 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.072754 14056 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:50.085112 14100 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:50.052502 14124 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.068845 14085 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.096413 14014 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:50.080380 13254 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:11:50.062188 14146 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.105998 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.107268 12489 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.108361 13253 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:50.112551 13992 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:50.119194 14146 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:50.122800 13951 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.124984 12495 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:50.125687 12494 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.123278 13951 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:50.126744 14172 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:11:50.129864 13951 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.140004 13955 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.131314 13951 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.148674 13992 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:50.142989 12489 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:50.150249 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.151024 13992 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.151574 13992 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.152693 14156 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.153309 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.144320 12495 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:50.144860 12536 leader_election.cc:336] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.608s (SENT)
I20260709 12:11:50.153759 13992 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.156693 13992 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.150307 14172 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.158030 14137 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:50.146420 12493 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:50.147652 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:50.160764 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1296) took 1400 ms. Trace:
I20260709 12:11:50.161194 12645 rpcz_store.cc:276] 0709 12:11:48.760675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:49.013122 (+252447us) service_pool.cc:224] Handling call
0709 12:11:49.013375 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:49.152651 (+139276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:50.160534 (+1007883us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:50.160705 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318314880}
I20260709 12:11:50.159160 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.158243 14172 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.159065 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:50.136999 14154 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.123955 14100 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.166198 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.168591 13955 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:50.168673 14179 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.170035 14163 raft_consensus.cc:993] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 6d4ece942fd5414998215d26edf91001 to start an election
W20260709 12:11:50.167083 14118 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:11:50.147454 14164 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.180883 12637 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:50.187259 12569 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.186990 14100 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.187074 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 91) took 2071 ms. Trace:
I20260709 12:11:50.190179 12638 rpcz_store.cc:276] 0709 12:11:48.115466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.881187 (+765721us) service_pool.cc:224] Handling call
0709 12:11:49.055041 (+173854us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:49.182798 (+127757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:49.781952 (+599154us) spinlock_profiling.cc:243] Waited 98 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:50.186850 (+404898us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54003b01d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:50.187004 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624482432}
I20260709 12:11:50.192788 12640 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.195359 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:11:50.200521 14130 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.511s	user 0.006s	sys 0.012s
I20260709 12:11:50.202860 14163 raft_consensus.cc:1081] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Signalling peer 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:11:50.203048 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:50.208243 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.209743 12492 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:50.209578 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.211462 12569 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:50.214255 12569 raft_consensus.cc:606] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:50.214773 12569 raft_consensus.cc:614] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:11:50.210851 12493 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:50.217015 14143 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.219574 14126 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.220633 12566 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:50.221829 14165 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.396s	user 0.000s	sys 0.005s
I20260709 12:11:50.223680 14004 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:50.187815 14136 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.172161 14172 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.233948 14014 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:50.164623 14113 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.237044 14004 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:50.246917 14004 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:50.237632 14007 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.240775 12493 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:34492
I20260709 12:11:50.256966 12493 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:50.257308 12493 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.258116 12570 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.236022 14145 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:50.261765 12537 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a30ed0986fa641bd8f88912b4b9ee320. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:11:50.264285 12569 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:50.267645 14059 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.268402 14059 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:50.270365 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 216719us
W20260709 12:11:50.274623 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.273379 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:11:50.255409 14172 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:50.281876 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.283308 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.284591 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:11:50.285226 12494 raft_consensus.cc:606] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:50.285748 12494 raft_consensus.cc:614] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:11:50.287230 14124 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:50.288560 12640 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:50.289016 14124 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:11:50.290990 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.240046 14163 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:11:50.293249 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.296167 12495 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.274837 14132 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:11:50.295142 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.303360 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:50.303586 12639 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:50.275012 14146 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 12:11:50.311739 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:50.290293 14148 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.316053 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.316542 12646 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:50.324167 12495 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:50.324317 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:50.324931 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:50.325150 12493 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.313035 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:50.318210 12643 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:50.326951 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:50.319106 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.329308 14154 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.289580 14124 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:50.281473 14166 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:11:50.335386 14124 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.338302 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.340301 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:50.341253 14045 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:50.341887 14045 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:50.342357 14045 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:50.344169 14109 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.344710 14109 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:50.348721 14004 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.352871 12641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:50.337304 14124 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.361807 12638 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:50.364089 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.365005 12567 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:50.366712 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1657) took 1495 ms (client timeout 1945 ms). Trace:
W20260709 12:11:50.367448 12492 rpcz_store.cc:269] 0709 12:11:48.871707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.871816 (+   109us) service_pool.cc:224] Handling call
0709 12:11:50.366536 (+1494720us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:50.366639 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2973428608}
W20260709 12:11:50.370827 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.367949 14109 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.374028 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:50.378031 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:50.379244 14109 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.385946 14118 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:50.389709 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.391197 12567 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:50.393374 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.394467 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.380419 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:50.380739 13914 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.382036 14136 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:50.396773 12494 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.397904 12493 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.397967 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.399984 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:50.403661 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.427961 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:50.428781 12492 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:50.428579 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60883.6us
I20260709 12:11:50.430436 12495 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:50.433450 12494 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.439092 14113 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:50.440866 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64421.2us
W20260709 12:11:50.445046 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1412) took 1461 ms (client timeout 1608 ms). Trace:
W20260709 12:11:50.445542 12491 rpcz_store.cc:269] 0709 12:11:48.983471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:48.983595 (+   124us) service_pool.cc:224] Handling call
0709 12:11:50.444849 (+1461254us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:50.444987 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220248448}
I20260709 12:11:50.448868 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.450328 12497 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:50.450559 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.451642 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 81) took 1412 ms. Trace:
I20260709 12:11:50.451924 12497 rpcz_store.cc:276] 0709 12:11:49.039652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:49.221906 (+182254us) service_pool.cc:224] Handling call
0709 12:11:50.449938 (+1228032us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:50.450777 (+   839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:50.451429 (+   652us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000f5ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:50.451578 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2823151872}
I20260709 12:11:50.453857 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1443) took 1096 ms. Trace:
I20260709 12:11:50.388800 14101 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:50.454481 12496 rpcz_store.cc:276] 0709 12:11:49.356933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:49.456505 (+ 99572us) service_pool.cc:224] Handling call
0709 12:11:49.648724 (+192219us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:49.649545 (+   821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:50.453545 (+804000us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540034e250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:50.453769 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":440951552}
I20260709 12:11:50.456749 12492 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.460340 12568 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.456607 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:50.458230 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63971.6us
I20260709 12:11:50.461586 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.422019 13987 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:11:50.399034 13914 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:50.424489 14118 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.470149 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:50.472412 12642 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.422674 13928 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:50.477489 12571 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.484176 12567 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.478317 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.487684 12638 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:50.442971 14125 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.457167 14081 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:50.430208 14051 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260709 12:11:50.456064 14090 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:50.499630 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.468868 14162 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.502399 12488 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:50.507061 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.476840 13998 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.481293 14118 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:50.491852 13997 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.468585 14130 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:11:50.496930 14038 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:50.533093 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.473613 14101 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.512485 14081 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:50.519853 14145 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:50.547616 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 234253us
I20260709 12:11:50.508742 14014 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:11:50.550701 12572 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:50.552772 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.562019 12637 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.566140 12493 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.568758 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.568826 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.572976 12493 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:11:50.525132 14125 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
W20260709 12:11:50.582741 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132242us
I20260709 12:11:50.585988 12643 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:11:50.587198 12492 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:11:50.524300 14138 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:11:50.591198 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:50.596781 12643 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.600683 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:50.548105 13914 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.550678 14038 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:50.537760 14136 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:11:50.635535 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:50.636749 12642 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:11:50.640086 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102903us
I20260709 12:11:50.632983 14113 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:50.632244 13987 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:50.742779 13987 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.743862 12639 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:50.624089 14081 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.745802 14148 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:50.746925 14148 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.747282 14148 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.637431 13928 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.748071 14148 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.748495 13928 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.748389 14113 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.749219 14148 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:50.732887 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55192.8us
I20260709 12:11:50.750377 14169 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:50.750169 14113 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.758360 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:50.751677 13928 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.757228 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:50.768326 12566 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:50.769836 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92599.7us
I20260709 12:11:50.732674 14187 ts_tablet_manager.cc:933] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:50.633497 14090 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20260709 12:11:50.632886 14143 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.716982 14142 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:50.676872 14146 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260709 12:11:50.798141 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:50.799796 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.767365 14169 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:50.801199 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.803233 13928 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.803745 13928 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.804468 14132 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:50.805872 14132 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.806633 14152 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260709 12:11:50.808107 14157 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:11:50.804085 14014 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:50.788647 14143 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.810652 14014 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:50.811734 14014 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.813066 14143 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.796355 14174 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:11:50.777045 14179 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:11:50.814942 13928 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.816979 14035 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:50.821762 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.837972 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:11:50.840094 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:50.827505 12644 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:50.841431 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:50.765528 14154 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260709 12:11:50.881954 14146 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:50.882598 14146 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:50.883664 14146 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.885321 14146 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.822464 13254 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:50.895920 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94809.2us
W20260709 12:11:50.895478 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58106.7us
W20260709 12:11:50.906638 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65343.5us
I20260709 12:11:50.893078 14085 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.908696 14084 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:11:50.906252 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:50.886234 13997 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.912192 12488 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:50.890713 14126 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.817922 14122 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.901265 14130 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.894183 14192 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.931325 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.931730 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:11:50.917356 14045 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.572s	user 0.008s	sys 0.005s
I20260709 12:11:50.932092 14136 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:50.895153 14075 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:11:50.911777 14152 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:50.920244 13997 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.960932 14174 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:50.961680 12641 raft_consensus.cc:606] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:50.915580 14169 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.962232 12641 raft_consensus.cc:614] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:11:50.962502 13788 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.963322 14177 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:50.963896 13788 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:50.963786 14177 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:50.911137 14085 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.962895 14169 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:50.912024 14142 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:11:50.944597 14109 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.005s
I20260709 12:11:50.885921 14038 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:50.951606 14120 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.717s	user 0.008s	sys 0.003s
I20260709 12:11:50.918524 14004 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:50.909310 14101 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:50.938182 14154 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:50.976223 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:50.974402 14085 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.957530 14162 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:50.982671 12572 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:56486
I20260709 12:11:50.962486 13997 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:50.984619 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71757us
W20260709 12:11:50.984822 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86018.6us
W20260709 12:11:50.976748 14124 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.621s	user 0.012s	sys 0.000s
I20260709 12:11:50.986063 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.983974 12572 raft_consensus.cc:606] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:11:50.993140 14174 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:11:50.973361 14045 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:50.989513 14004 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:50.994793 14174 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:50.997257 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:50.998171 14059 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:50.955221 14126 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.002945 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:51.015349 13261 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:51.004060 12488 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:51.005249 12492 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:50.989970 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.970824 14142 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.006317 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.019867 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:51.008684 14172 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.724s	user 0.010s	sys 0.000s
I20260709 12:11:50.969209 14088 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260709 12:11:50.982844 13992 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.826s	user 0.011s	sys 0.006s
I20260709 12:11:51.009373 12489 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.010514 12641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:11:51.017064 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:51.023372 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:51.014493 14085 raft_consensus.cc:993] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: : Instructing follower 9485b0afaf144396adf854fff857414e to start an election
I20260709 12:11:51.030562 12565 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:51.030340 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:50.967886 14192 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.032187 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.033466 12644 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:51.036653 14120 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50696 (request call id 395) took 3589 ms. Trace:
I20260709 12:11:51.031154 14091 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:51.029798 14157 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:51.037931 14157 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:51.037103 14120 rpcz_store.cc:276] 0709 12:11:47.459358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.467117 (+  7759us) service_pool.cc:224] Handling call
0709 12:11:48.713461 (+1246344us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbfa805682
0709 12:11:48.786779 (+ 73318us) ts_tablet_manager.cc:934] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
0709 12:11:49.682775 (+895996us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:11:50.233863 (+551088us) tablet_metadata.cc:665] Metadata flushed
0709 12:11:50.234014 (+   151us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:11:50.987948 (+753934us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:11:51.035996 (+ 48048us) tablet_replica.cc:206] Creating consensus instance
0709 12:11:51.036405 (+   409us) ts_tablet_manager.cc:989] Replica registered.
0709 12:11:51.036564 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":104245,"spinlock_wait_cycles":12456320,"tablet-copy.queue_time_us":1227236,"thread_start_us":1279324,"threads_started":2}
I20260709 12:11:51.038597 14157 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:51.024955 14101 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:11:51.029677 14200 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:51.039086 14120 tablet_copy_client.cc:806] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:51.142630 12638 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:51.082428 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.039590 13992 raft_consensus.cc:1081] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Signalling peer 9485b0afaf144396adf854fff857414e to start an election
W20260709 12:11:51.142728 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91963.2us
I20260709 12:11:51.032474 12641 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:51.039750 14091 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:51.043370 14085 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:11:51.153604 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.156963 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.157766 12644 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:51.149802 12641 raft_consensus.cc:614] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:11:51.051626 14200 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:51.145823 14050 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.048596 14045 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.148393 13928 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.151181 14091 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:51.150942 14125 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.177709 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:51.178431 12494 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:51.179046 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.174937 12489 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.154558 14187 tablet_copy_client.cc:323] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
I20260709 12:11:51.176697 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:51.182086 12644 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:51.173885 14035 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:11:51.187805 12638 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:11:51.189025 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.172392 14081 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:11:51.190902 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:51.173324 14120 tablet_copy_client.cc:670] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:51.167850 14200 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.152385 14205 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:11:51.182250 14101 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260709 12:11:51.202587 14192 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.162557 14113 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:11:51.205986 12644 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.170688 14088 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:11:51.207157 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.214058 12563 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:51.170440 14014 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:11:51.215788 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:51.217192 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.228983 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.173250 14050 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.180828 13928 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.231531 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:11:51.234725 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.184145 14125 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.242889 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:51.212647 14045 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:51.234198 14192 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:51.233533 14109 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.249359 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.237507 14162 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:51.237428 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.212663 14004 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:11:51.253242 12642 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:51.250842 14081 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:51.247143 14126 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.251080 14192 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:51.185912 14204 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260709 12:11:51.253782 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:51.255187 14162 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:51.239414 14140 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:11:51.272164 14081 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:51.278338 13887 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:11:51.286893 14214 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.276621 14192 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:51.256520 14109 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.257050 14045 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:51.297020 12491 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:51.285825 14148 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
I20260709 12:11:51.293041 14081 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.316825 14200 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.322006 14215 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.329435 14145 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:51.329972 14145 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:51.330832 14145 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.332113 14157 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.302569 14214 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.323083 14137 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:11:51.361074 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:51.362306 14100 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:51.362104 14148 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:11:51.363090 14100 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:51.363608 14100 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.355825 14050 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.372423 13998 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:51.374752 14212 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:51.374838 14211 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:11:51.337585 14162 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.327531 14215 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.376731 13928 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.377461 13998 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:51.405575 14200 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.406001 13928 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.406577 14214 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.406718 13928 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.408325 14157 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.401960 14140 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:51.409833 14140 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.410476 13928 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.385401 14109 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:51.418855 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 184898us
I20260709 12:11:51.427801 14215 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.404801 14162 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:51.440333 14100 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.441216 14118 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.441902 14140 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.442457 14118 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:51.449489 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:51.458001 13928 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.458709 13928 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:51.472299 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.476591 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:51.477289 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:51.478226 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:51.480082 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.404366 14212 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.488292 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:51.492507 12640 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.492671 14216 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:51.493319 14216 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:51.493687 14216 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:51.494004 13928 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.496433 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1408) took 1181 ms. Trace:
I20260709 12:11:51.497571 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:51.496930 12565 rpcz_store.cc:276] 0709 12:11:50.315274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:50.315374 (+   100us) service_pool.cc:224] Handling call
0709 12:11:50.315630 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:51.030990 (+715360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:51.496128 (+465138us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:51.496357 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1643351936}
I20260709 12:11:51.498490 12493 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.475546 14125 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.488484 14219 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.464741 13998 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.514009 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:51.492513 13914 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:51.505344 13928 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:51.529439 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:51.530867 12491 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:51.532824 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.519726 13256 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.553306 12492 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:11:51.496218 14192 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.554961 12493 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:51.522004 14091 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.556209 13896 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.525609 14219 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.571924 13896 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:51.662382 14223 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.125s	user 0.002s	sys 0.000s
I20260709 12:11:51.663069 13896 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.664189 14220 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.664820 14220 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.539189 13928 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.680789 12566 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:51.686147 12492 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.681281 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:51.684247 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:51.559439 14100 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.692646 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:51.698933 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:51.694089 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 196855us
W20260709 12:11:51.702565 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.703262 12642 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:11:51.702754 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:51.560724 13788 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.146s
I20260709 12:11:51.695290 14118 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.687198 14222 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:51.704742 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet a30ed0986fa641bd8f88912b4b9ee320 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:11:51.707814 13896 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:51.704418 12639 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:51.730896 13928 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.731073 14007 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:51.736070 14007 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:51.738381 12489 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:11:51.739833 13261 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:50696
I20260709 12:11:51.740419 13261 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:11:51.740757 12493 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:51.740796 13261 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.725741 13896 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:51.741513 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:11:51.729211 14156 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:11:51.751469 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48804us
W20260709 12:11:51.745340 14218 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 12:11:51.728822 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet a30ed0986fa641bd8f88912b4b9ee320 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-a30ed0986fa641bd8f88912b4b9ee320
I20260709 12:11:51.737159 14007 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:51.739341 14220 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.739511 14212 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:51.724839 14222 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.747279 13910 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.763803 13910 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.724575 14118 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.763031 14230 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:51.762905 14142 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:51.753821 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:51.752797 14156 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:51.750878 13896 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:51.775981 14230 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:51.777175 14222 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.767129 14177 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.779593 12491 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:11:51.781548 14230 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.794214 14192 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.796427 14081 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.790508 14224 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.799587 14224 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.789434 14227 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.810344 13255 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:51.814342 14175 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:51.814850 14175 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:51.815165 14203 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:11:51.816200 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.779851 14233 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:51.822199 12488 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.822278 14177 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:51.823494 14166 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:51.791455 14225 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:11:51.823971 14166 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:51.816349 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:51.825958 14045 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.801577 14224 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.826488 12639 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:51.775004 14232 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:11:51.833292 14119 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:51.834430 14204 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.870144 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.830535 14142 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:51.837337 14101 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.804260 14219 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.805047 14118 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.841341 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:51.845180 12571 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.864311 13254 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:51.846225 12563 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:51.830899 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:51.805047 14227 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.849059 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:51.845896 14100 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:51.853127 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:51.854429 13910 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:51.815814 14203 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:51.871560 14142 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.867655 14179 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.868166 14238 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:51.909353 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78611.1us
I20260709 12:11:51.871783 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:51.877390 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:51.880541 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.910537 12491 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:51.910959 14167 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:51.911214 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.882589 12564 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:51.884696 12565 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:51.911960 14167 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:51.881774 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:51.912916 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.887462 14227 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.884272 12489 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:51.776742 14156 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.913462 14154 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:51.914222 14156 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:51.914487 14154 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:51.913431 14167 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:51.888558 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47348.8us
I20260709 12:11:51.815785 14175 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.870677 14204 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.868470 14169 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.878993 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.862229 14216 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.912007 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:51.924726 12571 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
W20260709 12:11:51.917261 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.874732 14075 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.825739 14166 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.925338 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.876516 13788 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:51.888398 14004 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:51.891098 13947 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.934060 13261 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.916139 14219 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.936858 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.929373 14220 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:11:51.939680 14216 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.923964 13992 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:11:51.936754 13788 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.945448 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.945972 13788 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:51.919286 14045 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.947708 13260 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:51.934468 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.916107 14038 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.890408 14100 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.884404 14101 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.956324 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.957075 13261 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:51.953338 13947 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.966936 12571 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:11:51.959086 12563 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:51.959491 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:51.923537 14169 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.956262 14220 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.962649 14038 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.951071 14113 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:51.971729 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.972754 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:51.874167 14233 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.952757 14174 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:51.985662 12569 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:51.890697 14119 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.966195 14224 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:51.934217 14075 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.989794 12564 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:51.984795 13887 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:51.994175 13887 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:51.974419 14113 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:11:52.009847 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.948486 14004 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.953001 14219 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.964596 14045 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:51.977344 14124 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:51.932984 14154 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.014626 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:51.916391 14179 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.997546 13887 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:51.928032 14238 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.998405 14081 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:52.027469 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.028029 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:51.978906 14220 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:51.988524 14174 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.035475 12567 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
W20260709 12:11:52.037055 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:51.966395 14203 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.018278 14038 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:51.999071 14224 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.046667 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.047561 12571 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.037249 14118 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.061954 14174 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.062309 14101 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.062624 14179 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.062847 14101 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:52.063306 14179 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.063845 14122 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:52.064335 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.075677 13261 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:52.064950 14119 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.065148 14220 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.041828 12639 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:52.065214 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.068250 14224 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:52.068848 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1412) took 1315 ms (client timeout 1622 ms). Trace:
W20260709 12:11:52.085374 12568 rpcz_store.cc:269] 0709 12:11:50.753755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:50.758060 (+  4305us) service_pool.cc:224] Handling call
0709 12:11:52.068601 (+1310541us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:52.068739 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1277999872}
I20260709 12:11:52.086966 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.087677 12568 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.088644 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.089468 12568 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:11:52.071229 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:52.077996 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.059083 14142 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.092509 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.089859 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.096588 14220 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:52.097138 14239 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:11:52.095635 12571 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.094523 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:52.094238 12460 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.622s (SENT) [suppressed 71 similar messages]
I20260709 12:11:52.074533 14191 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.097206 14220 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.091979 14119 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.090878 14204 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.108049 14061 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:52.053217 14113 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:52.119131 14061 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.120189 14045 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.120854 14238 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.121331 14061 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.077814 14101 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.111260 14169 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.128698 14061 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:52.128968 13947 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.122344 14233 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.107663 14124 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.129555 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.133297 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.135288 12571 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:11:52.121270 14236 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:11:52.098343 14242 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.111313 14220 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.111940 14100 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.129408 14061 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.096848 14179 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:52.150242 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:52.160198 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:52.169842 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53630.8us
I20260709 12:11:52.161332 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:52.155861 14101 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.144181 14242 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.149889 14241 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:11:52.176908 14242 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:52.183772 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89597.4us
I20260709 12:11:52.184581 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:52.184778 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72306us
I20260709 12:11:52.185393 14125 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.185510 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:52.185927 14125 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.186825 14125 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.187104 14045 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.187541 14045 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.187911 14045 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.190174 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.111744 14191 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.191520 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.191680 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.192389 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.156415 14172 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:52.132357 14204 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:52.193158 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.130244 14119 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.194921 12496 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:52.150403 13896 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.194960 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 225) took 1747 ms. Trace:
I20260709 12:11:52.193022 14138 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:52.160789 14219 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.199870 14145 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:52.201758 14091 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.155759 13914 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.203636 14172 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:52.204128 12491 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:52.194703 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.178117 14050 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:52.177506 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.178284 14075 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.209417 13254 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:52.211247 12570 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:52.180518 14004 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:52.212239 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.144516 14124 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.192988 12488 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:52.211057 14138 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:52.222486 14119 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.225682 14249 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.226013 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:52.238377 12566 raft_consensus.cc:2376] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2. Re-sending same reply.
I20260709 12:11:52.206072 14204 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.243505 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.230157 14219 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.245792 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:52.247081 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.246546 14250 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.247998 12566 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:52.248265 14250 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.249377 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:52.252472 12566 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.238662 13896 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.227746 13914 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:52.256669 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85713us
I20260709 12:11:52.253607 13253 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:52.244005 14081 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:52.261154 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.282006 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:52.282402 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.283175 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.283459 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.273223 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.274415 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.282104 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.286140 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.287137 12644 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:52.288331 14191 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.286177 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.290077 12637 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:52.291931 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.292707 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:52.286792 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.288527 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.293658 12643 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.260360 14120 tablet_copy_client.cc:538] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:52.207489 13256 rpcz_store.cc:276] 0709 12:11:50.448031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:50.449455 (+  1424us) service_pool.cc:224] Handling call
0709 12:11:51.371440 (+921985us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:51.371704 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:51.543673 (+171969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:52.173777 (+630104us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:52.193240 (+ 19463us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":303143424}
I20260709 12:11:52.297752 12644 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:52.320415 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:52.342389 12643 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:52.338579 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.343997 13262 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.267158 14050 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.347450 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:52.362514 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.280759 14101 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:52.347512 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.284512 14119 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.267272 14091 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:52.267158 13896 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:52.257517 14249 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.378198 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:52.280754 14075 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.366393 14244 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:52.359362 14172 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.349154 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.346765 14122 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:52.281272 14252 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.349720 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.257104 14219 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.260238 14232 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260709 12:11:52.290848 14084 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.291357 14216 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.291218 14081 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.431315 14175 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.432055 14125 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.346756 14222 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:52.394747 13896 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.434245 14222 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:52.397384 14091 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.434886 14222 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.250546 14138 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.436177 14250 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.438244 14205 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:52.438562 14138 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:52.438759 14205 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:52.397259 13259 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:52.405404 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.368877 14050 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.410288 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.410389 14246 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:52.411842 14219 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.413785 14084 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.414112 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.458606 12643 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:52.415704 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.460965 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:52.419409 14172 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:52.420547 14244 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.423238 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:52.424464 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.426633 13914 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.404985 14061 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.427078 14248 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:11:52.428179 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.482600 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.434336 13887 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.483836 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.431119 14249 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.283208 14059 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.486028 14059 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.439484 14090 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:52.487274 14249 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:52.487970 14249 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:52.487533 14090 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.433519 14081 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.451882 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.429178 14252 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.436569 14216 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.491989 14242 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.492749 14216 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.381435 14101 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.479007 14204 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.495628 14101 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.496076 14199 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:52.496690 14199 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:52.459816 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:52.497437 12643 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:52.497910 12564 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:52.497517 12642 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.502070 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.503010 13259 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:52.456343 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.398419 14075 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.435437 14154 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:11:52.478947 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.389508 14119 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:52.483078 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54959.3us
I20260709 12:11:52.483196 13947 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:52.516955 13947 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.517418 13947 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:52.485433 13887 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.519127 13896 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.457798 13890 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.520859 13890 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.429023 14179 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.521476 14119 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.477137 14219 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:52.408898 14122 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.523217 13890 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.523134 14090 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.524284 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:52.524679 14122 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.525058 12637 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.512666 12640 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.525887 12638 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:51.919006 14136 consensus_queue.cc:1175] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:11:52.527484 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.477582 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.507884 14242 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.489064 14249 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.509860 14124 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.509235 14216 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.511463 14244 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.513067 12644 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:52.514014 14091 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.514094 14007 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:52.524489 14125 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.467036 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.531911 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.542775 12566 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
W20260709 12:11:52.532713 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.547492 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.548493 12490 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:52.552644 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:52.552920 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.554435 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.555248 12492 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:52.556828 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.557971 12492 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:52.535079 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
W20260709 12:11:52.537984 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 257836us
I20260709 12:11:52.493325 14252 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.563432 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:52.564455 13259 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:52.566103 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.569278 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.572206 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.572741 14216 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.573257 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:52.562088 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:52.574869 14220 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.578033 14050 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.577783 14199 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.512468 14222 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.527511 14075 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.585455 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:52.580027 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.509370 14084 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.577503 14220 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:52.583281 12566 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:11:52.504802 14246 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.584355 14132 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:52.584263 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.592852 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:52.579867 13257 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
I20260709 12:11:52.581303 12493 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:52.588533 14050 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.546177 14148 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:52.589653 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.594290 12489 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:52.595288 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:52.568216 14205 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:52.604553 12492 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.603324 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.605926 14075 raft_consensus.cc:2764] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:11:52.615408 14045 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.618008 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:52.618755 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:52.620328 14045 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:52.599771 14050 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.625293 12644 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:52.628590 14120 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:11:52.644393 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.645991 13890 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:52.646456 13890 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.646800 13890 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.611562 14081 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:52.675933 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.629397 14050 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:52.683007 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 335582us
I20260709 12:11:52.694491 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:52.679409 14081 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.662220 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:52.699095 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.701810 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.698031 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:11:52.699792 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.679004 14214 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:52.707834 12492 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
W20260709 12:11:52.723341 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.718724 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:52.682468 14145 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:11:52.734611 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:11:52.775815 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57210.1us
I20260709 12:11:52.704833 14081 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:52.780870 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46192.9us
I20260709 12:11:52.788615 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:52.684725 13947 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:52.699317 14246 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:52.700083 14125 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:52.724402 14124 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:52.723980 13255 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:11:52.774624 14220 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:52.952603 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166539us
I20260709 12:11:52.780316 14200 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:52.955332 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:52.981251 12565 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:52.786464 14222 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:52.962210 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:52.830552 14251 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:52.733178 14214 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:53.031390 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.035707 12639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.054s	sys 0.102s
I20260709 12:11:53.058990 14258 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:53.035852 14061 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.059721 14258 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.060140 14061 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.062434 14045 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:53.063064 14045 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:53.063519 14045 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.030905 14261 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:53.073232 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.073864 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.074314 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.075759 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:52.949889 14132 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:52.950088 14125 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:52.830936 14257 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:53.053398 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91310.1us
W20260709 12:11:53.059322 12639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.054s	sys 0.102s
I20260709 12:11:53.080433 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:53.087272 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:52.960381 14252 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:52.971871 14220 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:53.102981 12565 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:11:52.733740 14174 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:53.105232 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.079s	sys 0.136s
W20260709 12:11:53.124996 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.079s	sys 0.145s
W20260709 12:11:53.126092 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 643911us
W20260709 12:11:53.137876 13259 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.556s	user 0.012s	sys 0.000s
I20260709 12:11:53.138603 13259 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:53.112864 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:53.045821 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.155773 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:53.156636 12642 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.124250 14220 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:53.195369 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.197172 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119563us
I20260709 12:11:52.968519 14124 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:53.189070 12640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.010s	sys 0.052s
I20260709 12:11:53.088064 13914 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:53.217760 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.095103 14125 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:53.224238 14258 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:53.227315 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.189325 14232 raft_consensus.cc:2764] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:11:53.193382 13992 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:53.237918 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.205039 14222 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:53.230680 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:53.195700 14241 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:11:53.220906 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.220707 12640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.011s	sys 0.053s
I20260709 12:11:53.238154 12572 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:11:53.233013 14172 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.756s	user 0.012s	sys 0.000s
I20260709 12:11:53.188406 14252 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.250258 12572 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:53.229246 14156 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.257730 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:53.260825 12565 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.104086 14132 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.251077 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.229907 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:53.228184 14259 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:53.232509 14243 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.126483 14257 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.237108 14177 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.210752 14174 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:53.276051 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46233.4us
I20260709 12:11:53.262547 12572 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.303577 12572 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:53.126147 14251 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.225464 14230 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:11:53.271023 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:53.233242 13914 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.339079 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.340054 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:53.342645 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:53.335084 12488 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.004s	sys 0.009s
I20260709 12:11:53.341243 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.348160 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.350869 12568 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:53.353029 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:53.345903 12488 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.005s	sys 0.010s
I20260709 12:11:53.357043 12568 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:53.333909 14200 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:53.360008 14028 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:53.363112 14028 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:53.363624 14028 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:53.364114 14262 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:11:53.244725 13992 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.360584 13947 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.367038 13947 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:53.367652 13947 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.273485 14156 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.262329 14172 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:53.372725 14172 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.375860 14172 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:53.273020 14259 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:53.250396 14222 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.272392 14241 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:11:53.387291 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166866us
I20260709 12:11:53.276871 14260 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:53.394933 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 193855us
W20260709 12:11:53.395861 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.335258 14250 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.008s	sys 0.004s
W20260709 12:11:53.406003 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 137479us
W20260709 12:11:53.406100 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 213646us
I20260709 12:11:53.408979 12568 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:53.410044 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:53.413244 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.416445 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1423) took 1543 ms (client timeout 1956 ms). Trace:
I20260709 12:11:53.410367 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:53.416915 12569 rpcz_store.cc:269] 0709 12:11:51.872934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:51.945078 (+ 72144us) service_pool.cc:224] Handling call
0709 12:11:53.416228 (+1471150us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:53.416374 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90616576}
I20260709 12:11:51.764602 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:53.383675 14259 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.309527 14177 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.415710 14061 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:53.419595 12395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 222) took 7856 ms. Trace:
I20260709 12:11:53.443527 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.303865 14243 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.403543 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1326) took 1119 ms. Trace:
I20260709 12:11:53.444578 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:53.405862 14156 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.415279 14263 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.431906 14059 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.415526 14132 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.407912 14277 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:53.438743 13992 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:53.481869 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143821us
I20260709 12:11:53.402899 14241 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.485764 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:53.495107 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84853.7us
I20260709 12:11:53.488436 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:11:53.500345 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.501171 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.501413 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:53.501603 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.439143 12395 rpcz_store.cc:276] 0709 12:11:45.563652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:46.145357 (+581705us) service_pool.cc:224] Handling call
0709 12:11:53.419499 (+7274142us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:49.316850 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:49.369785 (+ 52935us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:49.369821 (+    36us) write_op.cc:435] Acquired schema lock
0709 12:11:49.369840 (+    19us) tablet.cc:662] Decoding operations
0709 12:11:49.370261 (+   421us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:49.370325 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:49.370349 (+    24us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:49.370588 (+   239us) tablet.cc:701] Row locks acquired
0709 12:11:49.370602 (+    14us) write_op.cc:260] PREPARE: finished
0709 12:11:49.370758 (+   156us) write_op.cc:270] Start()
0709 12:11:49.370866 (+   108us) write_op.cc:276] Timestamp: P: 1783599109370717 usec, L: 0
0709 12:11:49.370881 (+    15us) op_driver.cc:348] REPLICATION: starting
0709 12:11:49.371592 (+   711us) log.cc:844] Serialized 536 byte log entry
0709 12:11:49.612436 (+240844us) op_driver.cc:464] REPLICATION: finished
0709 12:11:49.616108 (+  3672us) write_op.cc:301] APPLY: starting
0709 12:11:49.616207 (+    99us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:49.616360 (+   153us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:49.621579 (+  5219us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:49.691494 (+ 69915us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:49.699738 (+  8244us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:49.704081 (+  4343us) write_op.cc:312] APPLY: finished
0709 12:11:49.730482 (+ 26401us) log.cc:844] Serialized 23 byte log entry
0709 12:11:49.741603 (+ 11121us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:49.747908 (+  6305us) write_op.cc:454] Released schema lock
0709 12:11:49.748564 (+   656us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:49.789974 (+ 41410us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:49.795224 (+  5250us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3480,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":3399,"apply.run_cpu_time_us":3433,"apply.run_wall_time_us":166112,"num_ops":1,"prepare.queue_time_us":219272,"prepare.run_cpu_time_us":18334,"prepare.run_wall_time_us":196722,"raft.queue_time_us":127978,"raft.run_cpu_time_us":2931,"raft.run_wall_time_us":4141,"replication_time_us":241357,"thread_start_us":347977,"threads_started":4,"wal-append.queue_time_us":110535,"wal-append.run_cpu_time_us":34798,"wal-append.run_wall_time_us":1256471}]]}
I20260709 12:11:53.403079 14260 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.470840 14061 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:53.468043 14214 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:53.507431 12567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1425) took 1630 ms (client timeout 1876 ms). Trace:
W20260709 12:11:53.514984 12567 rpcz_store.cc:269] 0709 12:11:51.876866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:51.922551 (+ 45685us) service_pool.cc:224] Handling call
0709 12:11:53.507210 (+1584659us) spinlock_profiling.cc:243] Waited 96.5 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:53.507362 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221840640}
I20260709 12:11:53.489061 14059 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:53.488988 13992 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:53.517931 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107985us
I20260709 12:11:53.486724 14280 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:53.493422 14132 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:53.515718 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:53.491282 14252 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.490868 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:53.488930 14124 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:53.560446 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1497) took 1270 ms. Trace:
I20260709 12:11:53.561074 12637 rpcz_store.cc:276] 0709 12:11:52.289637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.525828 (+236191us) service_pool.cc:224] Handling call
0709 12:11:52.526235 (+   407us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:11:52.528834 (+  2599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:52.528867 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:11:52.541900 (+ 13033us) log.cc:844] Serialized 52 byte log entry
0709 12:11:52.966981 (+425081us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:52.967047 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:52.967163 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:53.560040 (+592877us) raft_consensus.cc:1685] finished
0709 12:11:53.560064 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:53.560352 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20137,"raft.queue_time_us":1015,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":11709,"thread_start_us":381629,"threads_started":1,"wal-append.queue_time_us":709362}
I20260709 12:11:53.570057 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.571158 12569 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:53.495786 14174 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.572863 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.580322 12637 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:53.518543 14257 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.518266 13914 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:53.560804 14132 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:53.577353 12392 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:11:53.589865 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.590653 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:53.529877 14059 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:53.585602 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:53.559342 13992 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:53.582072 14268 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 12:11:53.571533 14282 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:11:53.487108 14263 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:53.601756 14028 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.515549 14275 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:11:53.585963 13890 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.939s	user 0.007s	sys 0.008s
I20260709 12:11:53.564783 14276 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:53.488933 14277 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.515285 14061 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:53.573032 14280 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.625530 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:53.593115 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:53.599035 14007 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.037s	user 0.007s	sys 0.006s
I20260709 12:11:53.455417 12641 rpcz_store.cc:276] 0709 12:11:52.281180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.284095 (+  2915us) service_pool.cc:224] Handling call
0709 12:11:52.405188 (+121093us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:52.427512 (+ 22324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:52.427548 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:52.428154 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 12:11:52.778296 (+350142us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:52.781879 (+  3583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:52.829952 (+ 48073us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:53.381433 (+551481us) raft_consensus.cc:1685] finished
0709 12:11:53.382588 (+  1155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:53.399857 (+ 17269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":57883,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"thread_start_us":289809,"threads_started":1,"wal-append.queue_time_us":784729}
I20260709 12:11:53.627071 14059 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:53.667344 14272 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:11:53.669057 13992 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:53.636541 14081 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.682s	user 0.013s	sys 0.006s
I20260709 12:11:53.658154 12572 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:53.674326 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 152) took 1415 ms. Trace:
I20260709 12:11:53.674701 12572 rpcz_store.cc:276] 0709 12:11:52.259288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.422927 (+163639us) service_pool.cc:224] Handling call
0709 12:11:53.237950 (+815023us) spinlock_profiling.cc:243] Waited 59.1 ms on lock 0x7b54004f1120. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6e18 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:53.674250 (+436300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135932288}
I20260709 12:11:53.676055 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.677129 12572 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:11:53.583050 14278 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:53.598412 14132 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.579198 14124 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
W20260709 12:11:53.687877 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:53.603828 14175 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.172s	user 0.015s	sys 0.000s
I20260709 12:11:53.497025 14200 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:53.631819 14265 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
W20260709 12:11:53.606607 13255 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.554s	user 0.013s	sys 0.000s
W20260709 12:11:53.603084 14138 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.164s	user 0.011s	sys 0.004s
I20260709 12:11:53.589370 14257 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:53.697822 13947 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:53.656466 14276 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:11:53.643565 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1713) took 1454 ms (client timeout 1765 ms). Trace:
I20260709 12:11:53.643764 14263 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.644488 14275 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:53.624610 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.669865 14177 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.670558 14243 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.714533 14166 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:53.626286 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:53.696175 14050 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.998s	user 0.022s	sys 0.002s
I20260709 12:11:53.627421 12565 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:53.656229 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:53.718748 12567 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.666101 14157 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:53.648635 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.684746 14280 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:53.659618 14267 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260709 12:11:53.693828 12566 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:11:53.681231 13896 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 12:11:53.687254 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:53.697548 13255 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:53.747772 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 235) took 1100 ms. Trace:
W20260709 12:11:53.642045 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 95) took 2154 ms (client timeout 1671 ms). Trace:
I20260709 12:11:53.669370 14260 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.650166 14061 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:53.748245 13255 rpcz_store.cc:276] 0709 12:11:52.646936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.660412 (+ 13476us) service_pool.cc:224] Handling call
0709 12:11:53.747692 (+1087280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:53.731102 14157 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:53.698246 14257 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.751307 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.752308 14257 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.667366 14222 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:53.659462 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:53.748569 12497 rpcz_store.cc:269] 0709 12:11:51.487806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:51.495760 (+  7954us) service_pool.cc:224] Handling call
0709 12:11:53.641743 (+2145983us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:53.641912 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1758735232}
W20260709 12:11:53.702620 12493 rpcz_store.cc:269] 0709 12:11:52.189174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.191256 (+  2082us) service_pool.cc:224] Handling call
0709 12:11:53.643369 (+1452113us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:53.643507 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":889075584}
I20260709 12:11:53.706118 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1324) took 1433 ms. Trace:
I20260709 12:11:53.755127 12638 rpcz_store.cc:276] 0709 12:11:52.272272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.272466 (+   194us) service_pool.cc:224] Handling call
0709 12:11:52.415516 (+143050us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:53.705869 (+1290353us) spinlock_profiling.cc:243] Waited 65.3 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:53.706036 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159516416}
I20260709 12:11:53.660446 14241 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.686616 14038 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:53.689713 14278 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.683245 14081 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.693895 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:53.759162 12572 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.648981 13890 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.723827 14050 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.728590 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1572) took 1865 ms. Trace:
W20260709 12:11:53.734602 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47483.8us
I20260709 12:11:53.736109 14276 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:53.716257 14275 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.734823 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:53.739830 14118 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:53.734632 14084 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.755322 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.756582 12563 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:11:53.720417 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:53.756819 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.768309 12637 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:53.770493 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.757344 14038 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:53.752724 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:53.725551 14045 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.003s
I20260709 12:11:53.752554 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1498) took 1462 ms. Trace:
I20260709 12:11:53.773346 12569 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:53.774886 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1578) took 1066 ms. Trace:
I20260709 12:11:53.775811 12569 rpcz_store.cc:276] 0709 12:11:52.708856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.752435 (+1043579us) service_pool.cc:224] Handling call
0709 12:11:53.774804 (+ 22369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:53.776839 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:53.764364 12570 rpcz_store.cc:276] 0709 12:11:51.863664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:51.863852 (+   188us) service_pool.cc:224] Handling call
0709 12:11:51.864352 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:52.229190 (+364838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:53.728257 (+1499067us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:53.728511 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":797148416}
I20260709 12:11:53.785902 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:53.790573 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.793231 12567 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:53.796460 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1443) took 1092 ms. Trace:
I20260709 12:11:53.772472 13254 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 3. Ops: []
I20260709 12:11:53.765828 13992 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.791566 12638 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:53.756700 14259 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.799368 12567 rpcz_store.cc:276] 0709 12:11:52.704316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.655835 (+951519us) service_pool.cc:224] Handling call
0709 12:11:53.796377 (+140542us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:54.019701 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 150) took 1768 ms (client timeout 1970 ms). Trace:
W20260709 12:11:53.997390 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137862us
W20260709 12:11:54.103600 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.061s	sys 0.039s
I20260709 12:11:53.997799 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1440) took 1296 ms. Trace:
W20260709 12:11:53.933992 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 199229us
I20260709 12:11:54.094430 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.175611 12566 rpcz_store.cc:276] 0709 12:11:52.701204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.701466 (+   262us) service_pool.cc:224] Handling call
0709 12:11:53.997510 (+1296044us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:53.997717 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270626176}
I20260709 12:11:53.864254 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:11:53.920437 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:54.175037 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.062s	sys 0.040s
W20260709 12:11:53.947952 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:53.961495 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1582) took 1007 ms. Trace:
I20260709 12:11:53.855826 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:54.179688 12570 rpcz_store.cc:276] 0709 12:11:52.790527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.779807 (+989280us) service_pool.cc:224] Handling call
0709 12:11:53.797302 (+ 17495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:53.866238 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:53.767817 14118 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:54.034025 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.183169 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:11:54.031163 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1431) took 2074 ms (client timeout 1715 ms). Trace:
I20260709 12:11:54.146641 12572 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:53.873581 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:53.837646 12646 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:53.834342 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:53.939139 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1433) took 1981 ms. Trace:
I20260709 12:11:54.013171 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1502) took 1669 ms. Trace:
W20260709 12:11:54.061739 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:54.187103 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.178718 13261 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
W20260709 12:11:54.184566 12563 rpcz_store.cc:269] 0709 12:11:51.956648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:51.971426 (+ 14778us) service_pool.cc:224] Handling call
0709 12:11:54.030827 (+2059401us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:54.031058 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4090960640}
I20260709 12:11:54.059134 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:54.129343 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.007s	sys 0.020s
W20260709 12:11:54.014104 13257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 419) took 1547 ms (client timeout 1858 ms). Trace:
I20260709 12:11:53.887025 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:54.174237 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.185986 12569 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 3
W20260709 12:11:53.907148 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.190515 12565 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:11:54.174882 12564 rpcz_store.cc:269] 0709 12:11:52.251063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.407870 (+156807us) service_pool.cc:224] Handling call
0709 12:11:54.019398 (+1611528us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:54.019598 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108863872}
I20260709 12:11:54.020289 12489 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:53.772892 14276 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:54.184988 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1445) took 1452 ms. Trace:
I20260709 12:11:54.021198 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:54.192974 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:54.189849 13257 rpcz_store.cc:269] 0709 12:11:52.466532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.466701 (+   169us) service_pool.cc:224] Handling call
0709 12:11:54.013842 (+1547141us) spinlock_profiling.cc:243] Waited 66.6 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:54.014028 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153284352}
W20260709 12:11:54.197257 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.192494 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:54.194554 12567 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:54.186746 12644 rpcz_store.cc:276] 0709 12:11:52.343673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.541774 (+198101us) service_pool.cc:224] Handling call
0709 12:11:52.542111 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:52.625750 (+ 83639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.012754 (+1387004us) spinlock_profiling.cc:243] Waited 81.7 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:54.013091 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187876608}
I20260709 12:11:54.198957 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1581) took 1419 ms. Trace:
I20260709 12:11:54.199404 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:54.199354 12567 rpcz_store.cc:276] 0709 12:11:52.779354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.194119 (+1414765us) service_pool.cc:224] Handling call
0709 12:11:54.194411 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.198689 (+  4278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.198904 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.024662 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.194200 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.196064 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1321) took 1924 ms. Trace:
W20260709 12:11:53.802521 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.200917 12646 rpcz_store.cc:276] 0709 12:11:52.271293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.272822 (+  1529us) service_pool.cc:224] Handling call
0709 12:11:54.195855 (+1923033us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:54.195993 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3570622848}
I20260709 12:11:54.180632 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:53.840266 14261 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.769s	user 0.011s	sys 0.002s
I20260709 12:11:54.043011 12581 tablet_copy_source_session.cc:215] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:11:54.200420 12495 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:54.103529 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:53.963534 14125 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.727s	user 0.009s	sys 0.003s
I20260709 12:11:54.203053 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:53.838749 14220 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.620s	user 0.006s	sys 0.013s
I20260709 12:11:54.201071 12493 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:11:53.829164 13256 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:11:54.039279 14271 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.699s	user 0.008s	sys 0.000s
I20260709 12:11:54.204634 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:53.828482 14038 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:54.196195 12572 rpcz_store.cc:276] 0709 12:11:52.732833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.693506 (+960673us) service_pool.cc:224] Handling call
0709 12:11:54.184902 (+491396us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.204809 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:53.830032 14081 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:54.203831 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1586) took 1238 ms. Trace:
I20260709 12:11:54.185895 12571 rpcz_store.cc:276] 0709 12:11:51.957223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.136417 (+179194us) service_pool.cc:224] Handling call
0709 12:11:52.136894 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:52.204310 (+ 67416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:52.204347 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:52.204911 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:11:52.259498 (+ 54587us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:52.259527 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:53.917413 (+1657886us) spinlock_profiling.cc:243] Waited 67 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:53.917468 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:53.917526 (+    58us) raft_consensus.cc:1685] finished
0709 12:11:53.917538 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:53.939031 (+ 21493us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2013,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":7159,"spinlock_wait_cycles":154084736,"thread_start_us":30358,"threads_started":1,"wal-append.queue_time_us":197062,"wal-append.run_cpu_time_us":1950,"wal-append.run_wall_time_us":1005159}
I20260709 12:11:53.874220 14124 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.848021 14205 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:54.203583 14261 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:53.850023 13998 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:54.016535 14203 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:53.799629 14028 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:54.236835 12642 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
W20260709 12:11:54.189885 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.008s	sys 0.022s
W20260709 12:11:54.325765 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 810051us
I20260709 12:11:54.005239 14113 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:54.284368 14260 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:54.288797 14256 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.874621 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:53.876638 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:53.800967 14273 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:53.801802 13890 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.259796 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.260550 14220 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.022373 14258 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:54.261888 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.345670 12493 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:54.350734 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.351680 12493 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:54.353049 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.262373 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.873095 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1320) took 1602 ms. Trace:
I20260709 12:11:54.262483 12495 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:54.357093 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:54.203176 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.358551 12637 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:54.358752 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.359797 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.360601 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:53.878177 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.262650 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:54.256183 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.372885 12491 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:54.374332 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.375170 12491 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:54.376237 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.265074 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.205129 12563 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:54.264866 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:54.267432 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1592) took 1067 ms. Trace:
I20260709 12:11:53.916754 14224 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:53.932906 14251 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:53.828586 14278 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:54.139328 14285 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:54.190511 14248 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.801543 14277 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:53.773332 14084 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.275168 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:53.860633 14222 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:53.955569 14157 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.278154 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:54.389848 12489 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:54.279508 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:54.279812 12497 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:53.792135 14045 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.391130 14222 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.395706 14045 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.396032 14222 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.396517 14045 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:54.396751 14222 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:54.282303 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:54.397884 14285 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.399119 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.400620 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.400933 14285 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.281397 12570 rpcz_store.cc:276] 0709 12:11:52.965496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.191907 (+1226411us) service_pool.cc:224] Handling call
0709 12:11:54.192295 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.203545 (+ 11250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.203776 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.391400 14278 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:54.403308 14278 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.391580 14248 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.286228 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:54.370445 14172 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.856173 14059 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.915683 14249 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:54.427408 14084 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:53.990038 14132 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.431903 14132 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:54.009493 14061 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.289090 13955 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:54.434178 13955 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.391605 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 351) took 1962 ms. Trace:
I20260709 12:11:54.436004 13254 rpcz_store.cc:276] 0709 12:11:52.428745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.437968 (+  9223us) service_pool.cc:224] Handling call
0709 12:11:52.637297 (+199329us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:52.637590 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.193768 (+1556178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.371989 (+178221us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:54.390328 (+ 18339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2576955008}
I20260709 12:11:54.193884 13992 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.774453 12640 rpcz_store.cc:276] 0709 12:11:52.289999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.526439 (+236440us) service_pool.cc:224] Handling call
0709 12:11:52.527307 (+   868us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:52.527681 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:52.527711 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:52.528297 (+   586us) log.cc:844] Serialized 26 byte log entry
0709 12:11:53.751916 (+1223619us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b180037c420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1
0709 12:11:53.752090 (+   174us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:53.752135 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:53.752207 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:53.752251 (+    44us) raft_consensus.cc:1685] finished
0709 12:11:53.752270 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:53.752491 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1561,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":9928,"spinlock_wait_cycles":5037952,"thread_start_us":715581,"threads_started":1,"wal-append.queue_time_us":712417}
I20260709 12:11:54.295632 14276 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:54.297536 14259 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:54.211172 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.014s	sys 0.014s
I20260709 12:11:54.260684 14125 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:53.937609 14241 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:54.326589 14261 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:54.337348 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1505) took 1091 ms. Trace:
I20260709 12:11:54.331589 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:53.860874 14257 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:53.940495 14050 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:54.354064 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:54.303049 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:54.377580 12494 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
W20260709 12:11:54.379379 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116828us
W20260709 12:11:54.378464 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113802us
I20260709 12:11:54.380051 12569 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:54.382093 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.378592 12572 rpcz_store.cc:276] 0709 12:11:53.199677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.266087 (+1066410us) service_pool.cc:224] Handling call
0709 12:11:54.266786 (+   699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.267012 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:54.267049 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:54.267102 (+    53us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:54.267146 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.267218 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:54.267361 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.379997 12491 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:54.378419 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1589) took 1344 ms. Trace:
W20260709 12:11:54.362061 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.387429 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1514) took 1111 ms. Trace:
I20260709 12:11:54.287143 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:54.408746 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:11:54.401350 12644 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:54.347337 14256 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.408707 13253 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:54.292219 14124 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.346995 14260 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.380913 14258 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.793176 12637 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:54.793438 12638 rpcz_store.cc:276] 0709 12:11:53.277111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.274722 (+997611us) service_pool.cc:224] Handling call
0709 12:11:54.387354 (+112632us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.794093 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1519) took 1302 ms. Trace:
W20260709 12:11:54.692387 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.242s	user 0.010s	sys 0.074s
W20260709 12:11:54.627465 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:54.796223 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.345s	user 0.015s	sys 0.076s
I20260709 12:11:54.544507 12492 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:11:54.797307 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:54.797940 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.437505 12495 raft_consensus.cc:2368] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4.
W20260709 12:11:54.797300 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.35266e+06us
W20260709 12:11:54.597332 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.796237 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1527) took 1068 ms. Trace:
I20260709 12:11:54.386395 13987 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:54.799633 12644 rpcz_store.cc:276] 0709 12:11:53.727760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.400277 (+672517us) service_pool.cc:224] Handling call
0709 12:11:54.796181 (+395904us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.795918 12642 rpcz_store.cc:276] 0709 12:11:53.246199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.270705 (+ 24506us) service_pool.cc:224] Handling call
0709 12:11:54.337001 (+1066296us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:54.337249 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2063113472}
I20260709 12:11:54.816609 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:54.796360 12643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.009s	sys 0.013s
W20260709 12:11:54.817803 12643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.009s	sys 0.013s
I20260709 12:11:54.795667 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1531) took 1066 ms. Trace:
I20260709 12:11:54.794869 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.794591 12637 rpcz_store.cc:276] 0709 12:11:53.492130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.381677 (+889547us) service_pool.cc:224] Handling call
0709 12:11:54.794018 (+412341us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:54.803158 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.803556 12563 rpcz_store.cc:276] 0709 12:11:53.033805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.196949 (+1163144us) service_pool.cc:224] Handling call
0709 12:11:54.197279 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.378135 (+180856us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.378356 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.804818 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 157) took 1402 ms. Trace:
W20260709 12:11:54.805965 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.338s	user 0.017s	sys 0.029s
W20260709 12:11:54.799912 12610 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.006s	sys 0.013s
W20260709 12:11:54.818179 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.836025 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:54.839239 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:54.840516 12638 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:54.841235 12646 rpcz_store.cc:276] 0709 12:11:53.728985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.262050 (+533065us) service_pool.cc:224] Handling call
0709 12:11:54.795587 (+533537us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:54.572618 14275 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.021s	sys 0.055s
I20260709 12:11:54.397203 14045 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.386197 13896 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:54.397286 14222 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:54.397815 14286 raft_consensus.cc:2764] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
W20260709 12:11:54.633029 14199 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.038s	sys 0.037s
I20260709 12:11:54.843685 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.859604 12642 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:54.798063 14241 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:54.602056 12581 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:50142 (request call id 142) took 3007 ms. Trace:
W20260709 12:11:54.807868 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 476317us
I20260709 12:11:54.436666 13955 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:54.846694 12610 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.007s	sys 0.014s
I20260709 12:11:54.845894 12569 rpcz_store.cc:276] 0709 12:11:53.402412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.777454 (+375042us) service_pool.cc:224] Handling call
0709 12:11:54.804707 (+1027253us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:54.871496 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 610709us
I20260709 12:11:54.872488 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:54.607134 14061 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.465157 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1515) took 1110 ms. Trace:
I20260709 12:11:54.835817 14264 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:11:54.854421 13896 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:54.857043 12496 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:11:54.680094 14282 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.033s	user 0.030s	sys 0.064s
I20260709 12:11:54.861487 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.792934 14278 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.863782 14081 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:54.855546 14275 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.017s	user 0.022s	sys 0.068s
I20260709 12:11:54.826052 13987 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:54.845265 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.37768e+06us
I20260709 12:11:54.849359 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.870272 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:54.851699 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1526) took 1124 ms. Trace:
W20260709 12:11:54.858951 14199 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.044s	sys 0.043s
I20260709 12:11:54.817965 14291 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:54.899546 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:11:54.929250 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:54.939831 13217 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.011s	sys 0.007s
I20260709 12:11:54.940050 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:54.858565 14249 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:54.858748 14251 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:54.902678 12642 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:54.390443 14224 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:54.868254 14241 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:54.436856 14172 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.836143 14261 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.825790 14257 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:54.868616 12581 rpcz_store.cc:276] 0709 12:11:51.563421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:51.572537 (+  9116us) service_pool.cc:224] Handling call
0709 12:11:54.042403 (+2469866us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57fd3f7 00007ffbfa793036 00007ffbfa79ca3e 00007ffbf206027b 00007ffbfa79c9ba 00007ffbfa797acc 00007ffbfa792c7b 00007ffbfa7878d1 00007ffbf983b4b4 00007ffbf983b442
0709 12:11:54.570492 (+528089us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3919537536}
W20260709 12:11:54.901695 14282 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.268s	user 0.035s	sys 0.065s
W20260709 12:11:54.940541 13217 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.011s	sys 0.008s
W20260709 12:11:54.942090 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.038s	sys 0.094s
I20260709 12:11:54.858435 14222 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:54.945079 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:55.004361 14258 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.008713 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 158) took 1606 ms. Trace:
I20260709 12:11:55.009318 12565 rpcz_store.cc:276] 0709 12:11:53.402685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.340028 (+937343us) service_pool.cc:224] Handling call
0709 12:11:54.340552 (+   524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.995837 (+655285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:54.995880 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:54.995929 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:54.995959 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.008278 (+ 12319us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:55.008433 (+   155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.008634 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1506344704}
I20260709 12:11:54.951181 14084 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:11:55.015626 14007 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:55.016247 14007 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:55.016937 14125 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.018517 13992 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:54.943887 14120 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.004s	sys 0.020s
I20260709 12:11:55.019428 14264 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:55.020032 14264 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:11:55.019467 14120 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.004s	sys 0.021s
I20260709 12:11:54.949126 14249 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.022660 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.022980 14257 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.023411 12644 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:55.023805 12497 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:55.024448 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.025091 12644 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:55.025359 14124 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.026731 14059 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:54.950632 14061 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:55.027881 14061 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:55.030555 14045 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.037129 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.038096 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:11:54.901881 14259 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:54.946525 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1518) took 1484 ms. Trace:
I20260709 12:11:54.948693 14292 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:54.947584 12537 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
I20260709 12:11:55.052989 12537 leader_election.cc:403] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:11:54.908054 14220 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:55.054777 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.001s	sys 0.045s
I20260709 12:11:54.949966 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:54.942943 12638 rpcz_store.cc:276] 0709 12:11:53.727469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.794598 (+1067129us) service_pool.cc:224] Handling call
0709 12:11:54.851600 (+ 57002us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.909410 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:54.937304 13987 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:54.937777 14200 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.946s	user 0.000s	sys 0.029s
I20260709 12:11:54.898159 12641 rpcz_store.cc:276] 0709 12:11:53.354607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.877854 (+523247us) service_pool.cc:224] Handling call
0709 12:11:54.453399 (+575545us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:54.942386 13955 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:54.991505 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.039s	sys 0.094s
W20260709 12:11:55.067032 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.20075e+06us
I20260709 12:11:54.916152 14081 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.068049 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:54.991452 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 722483us
I20260709 12:11:55.027997 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 238) took 1163 ms. Trace:
I20260709 12:11:55.043766 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.047868 12637 rpcz_store.cc:276] 0709 12:11:53.462567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.849020 (+1386453us) service_pool.cc:224] Handling call
0709 12:11:54.946422 (+ 97402us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:54.945657 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75432.6us
W20260709 12:11:55.044459 13998 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.720s	user 0.013s	sys 0.000s
I20260709 12:11:55.048795 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.050431 14261 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.051425 14224 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:55.078792 14220 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:55.079684 14294 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:55.079795 12644 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:55.080495 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:55.080549 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:11:55.052232 12567 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.009s	sys 0.027s
I20260709 12:11:54.944536 14291 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:55.053269 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:55.054440 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 112004us
I20260709 12:11:54.901439 13896 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:55.055781 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.001s	sys 0.045s
I20260709 12:11:55.059672 14084 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
I20260709 12:11:55.061484 13992 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:55.058305 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108383us
I20260709 12:11:55.083858 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.035950 14007 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.074437 12642 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:54.948865 14251 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:55.063283 14200 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.000s	sys 0.030s
I20260709 12:11:55.056406 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.107710 12536 leader_election.cc:400] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
I20260709 12:11:55.108363 12536 leader_election.cc:403] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:11:55.109043 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:55.079805 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 111) took 2298 ms. Trace:
I20260709 12:11:55.110759 12643 rpcz_store.cc:276] 0709 12:11:52.781913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.782273 (+   360us) service_pool.cc:224] Handling call
0709 12:11:52.782967 (+   694us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:52.786425 (+  3458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:52.786470 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:54.032299 (+1245829us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b18001d88a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:11:54.034034 (+  1735us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:11:54.034830 (+   796us) log.cc:844] Serialized 26 byte log entry
0709 12:11:54.818458 (+783628us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:54.818532 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:54.818652 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:55.079518 (+260866us) raft_consensus.cc:1685] finished
0709 12:11:55.079541 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.079727 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58128,"raft.queue_time_us":35420,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":123966,"spinlock_wait_cycles":104412288,"thread_start_us":783154,"threads_started":1,"wal-append.queue_time_us":1042912}
I20260709 12:11:55.084282 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.114168 12644 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:55.087136 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1507) took 1840 ms. Trace:
I20260709 12:11:55.115525 12642 rpcz_store.cc:276] 0709 12:11:53.246791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.048344 (+1801553us) service_pool.cc:224] Handling call
0709 12:11:55.087066 (+ 38722us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:55.081579 12567 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.010s	sys 0.027s
I20260709 12:11:55.086732 12637 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:55.088256 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1328) took 2806 ms (client timeout 1514 ms). Trace:
I20260709 12:11:55.066604 13955 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:11:55.091207 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:55.073884 14118 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.864s	user 0.011s	sys 0.003s
W20260709 12:11:55.117895 12645 rpcz_store.cc:269] 0709 12:11:52.281987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.282105 (+   118us) service_pool.cc:224] Handling call
0709 12:11:55.088010 (+2805905us) spinlock_profiling.cc:243] Waited 2.8 s on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:55.088179 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6433317120}
I20260709 12:11:55.029397 14061 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.081382 14246 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.127488 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:55.079442 14220 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:55.129685 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:55.063752 14292 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:55.082252 12646 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:55.109758 14132 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.152878 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 121) took 1750 ms. Trace:
I20260709 12:11:55.085826 14261 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:55.152627 12638 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:55.143546 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.132012 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.072837 13253 rpcz_store.cc:276] 0709 12:11:53.864170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.874166 (+  9996us) service_pool.cc:224] Handling call
0709 12:11:55.027779 (+1153613us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b540056a9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:55.027926 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111803776}
I20260709 12:11:55.088462 13992 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:55.148607 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.150751 12645 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:55.107302 14248 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:55.128640 14203 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.803s	user 0.014s	sys 0.000s
I20260709 12:11:55.175326 14090 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.175431 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:55.176071 14090 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.177742 13260 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:55.107738 14256 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:55.178411 14248 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:55.178764 14256 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:11:55.127068 13955 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.179445 14248 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:55.179872 14248 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:55.126988 14282 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.153554 14246 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.186061 14282 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.167153 14132 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.188493 14282 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.172158 14261 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:55.189978 14261 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.188695 14132 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:55.194082 14132 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:55.194521 14132 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:55.195463 14132 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.198413 14172 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.173430 13992 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.201359 13955 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.144523 14118 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.203977 13955 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:55.207419 14081 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.152289 14220 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.223876 14251 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:55.155339 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:55.126577 13998 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:55.130478 14224 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
I20260709 12:11:55.083729 14294 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.158949 14007 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:55.248687 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:55.257203 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 357) took 1058 ms. Trace:
I20260709 12:11:55.257930 13260 rpcz_store.cc:276] 0709 12:11:54.199001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.199153 (+   152us) service_pool.cc:224] Handling call
0709 12:11:55.256880 (+1057727us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54005e1354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:55.257093 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2248152448}
I20260709 12:11:55.234727 13998 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:55.264113 13998 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:55.180917 14248 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.237784 14291 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:55.286415 14113 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.944s	user 0.016s	sys 0.000s
W20260709 12:11:55.366581 12392 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.760s	user 0.006s	sys 0.119s
I20260709 12:11:55.081959 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:55.369326 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.188s	user 0.030s	sys 0.054s
W20260709 12:11:55.469388 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.288s	user 0.031s	sys 0.054s
W20260709 12:11:55.370074 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.985s	user 0.018s	sys 0.028s
W20260709 12:11:55.474457 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.089s	user 0.019s	sys 0.028s
I20260709 12:11:55.074880 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:55.479425 12392 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.874s	user 0.008s	sys 0.119s
W20260709 12:11:55.121276 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.120085 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:55.498554 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 378415us
I20260709 12:11:55.500741 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:55.372856 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:55.373135 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.010s	user 0.005s	sys 0.051s
I20260709 12:11:55.374466 12490 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:54.356047 12639 rpcz_store.cc:276] 0709 12:11:52.270932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:52.280301 (+  9369us) service_pool.cc:224] Handling call
0709 12:11:52.368479 (+ 88178us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:52.520600 (+152121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:52.520647 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:52.521231 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:11:53.203658 (+682427us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:53.208282 (+  4624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:53.863181 (+654899us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:53.863252 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:53.863320 (+    68us) raft_consensus.cc:1685] finished
0709 12:11:53.863340 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:53.872966 (+  9626us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1412,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"spinlock_wait_cycles":335352576,"thread_start_us":582023,"threads_started":1,"wal-append.queue_time_us":712301,"wal-append.run_cpu_time_us":13600,"wal-append.run_wall_time_us":1597747}
W20260709 12:11:55.502362 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:55.505401 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.507077 12639 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:55.382676 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 437938us
W20260709 12:11:55.383970 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.281749 14222 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.387712 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 155) took 1986 ms. Trace:
W20260709 12:11:55.139896 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.524142 14224 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:55.299791 14248 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.526091 14224 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:55.155485 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:55.155556 12643 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:11:55.153376 12646 rpcz_store.cc:276] 0709 12:11:53.402171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.052907 (+1650736us) service_pool.cc:224] Handling call
0709 12:11:55.152795 (+ 99888us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.276082 14118 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.285386 14249 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:55.283512 14292 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.332096 14294 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:55.549785 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1339) took 1265 ms. Trace:
W20260709 12:11:55.430945 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.550467 12643 rpcz_store.cc:276] 0709 12:11:54.281199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.119699 (+838500us) service_pool.cc:224] Handling call
0709 12:11:55.143147 (+ 23448us) spinlock_profiling.cc:243] Waited 6.53 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:11:55.545826 (+402679us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15012608}
I20260709 12:11:55.551527 14222 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:55.552158 14090 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:55.433897 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.985s	user 0.030s	sys 0.069s
W20260709 12:11:55.554060 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.146s	user 0.031s	sys 0.069s
W20260709 12:11:55.434250 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.671s (SENT)
I20260709 12:11:55.433270 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 118) took 2194 ms. Trace:
I20260709 12:11:55.444864 14273 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:55.558303 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124423us
I20260709 12:11:55.558017 14273 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.545311 14045 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:55.520224 12571 rpcz_store.cc:276] 0709 12:11:53.401277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.282067 (+880790us) service_pool.cc:224] Handling call
0709 12:11:54.299311 (+ 17244us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:54.299914 (+   603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:54.299958 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:54.300667 (+   709us) log.cc:844] Serialized 26 byte log entry
0709 12:11:55.084892 (+784225us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:55.084944 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.085034 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:55.386130 (+301096us) raft_consensus.cc:1685] finished
0709 12:11:55.386421 (+   291us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.387130 (+   709us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":202,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":284,"thread_start_us":729476,"threads_started":1,"wal-append.queue_time_us":979104}
W20260709 12:11:55.460976 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 332549us
W20260709 12:11:55.494321 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.495661 14260 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:55.503041 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.140s	user 0.006s	sys 0.051s
I20260709 12:11:55.377030 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:55.508126 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.508275 14246 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:55.526856 13254 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:11:55.122864 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:55.593418 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:55.594635 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1342) took 1285 ms. Trace:
I20260709 12:11:55.595145 12641 rpcz_store.cc:276] 0709 12:11:54.309591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.082352 (+772761us) service_pool.cc:224] Handling call
0709 12:11:55.594463 (+512111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.562722 14113 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.599474 12641 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:55.600009 14113 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.600968 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1509) took 2353 ms. Trace:
W20260709 12:11:55.503703 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 421796us
I20260709 12:11:55.597136 14132 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:55.603855 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:55.604856 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1726) took 1547 ms. Trace:
I20260709 12:11:55.528396 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1530) took 1760 ms. Trace:
I20260709 12:11:55.504256 14261 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:55.606060 12638 rpcz_store.cc:276] 0709 12:11:53.728691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.055939 (+1327248us) service_pool.cc:224] Handling call
0709 12:11:55.473443 (+417504us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:55.489406 (+ 15963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105616640}
W20260709 12:11:55.606912 14205 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.314s	user 0.010s	sys 0.010s
I20260709 12:11:55.608027 12568 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:55.609148 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1510) took 2361 ms. Trace:
I20260709 12:11:55.536556 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1346) took 1134 ms. Trace:
I20260709 12:11:55.530117 14224 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.537019 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1349) took 1151 ms. Trace:
I20260709 12:11:55.557104 14172 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:55.537438 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1504) took 2287 ms. Trace:
I20260709 12:11:55.562402 14045 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:55.651196 14045 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:55.653030 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.557189 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 160) took 2095 ms. Trace:
W20260709 12:11:55.562616 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 487726us
I20260709 12:11:55.583230 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1508) took 2336 ms. Trace:
I20260709 12:11:55.656359 12643 rpcz_store.cc:276] 0709 12:11:53.247063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.551985 (+2304922us) service_pool.cc:224] Handling call
0709 12:11:55.552703 (+   718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.582559 (+ 29856us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:55.582606 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:55.582678 (+    72us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:55.582740 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.582818 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.583150 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.563582 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1591) took 2364 ms. Trace:
I20260709 12:11:55.556509 12644 rpcz_store.cc:276] 0709 12:11:53.239085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.168869 (+1929784us) service_pool.cc:224] Handling call
0709 12:11:55.431424 (+262555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.432210 (+   786us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:55.432289 (+    79us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:55.432404 (+   115us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:55.432502 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.432643 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.432906 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.575702 12640 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.580542 14293 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.542510 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:55.576603 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1506) took 2330 ms. Trace:
I20260709 12:11:55.641723 12642 rpcz_store.cc:276] 0709 12:11:54.385703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.131627 (+745924us) service_pool.cc:224] Handling call
0709 12:11:55.536723 (+405096us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.641007 12638 rpcz_store.cc:276] 0709 12:11:53.247615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.607625 (+2360010us) service_pool.cc:224] Handling call
0709 12:11:55.608223 (+   598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.608590 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:55.608620 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:55.608692 (+    72us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:55.608797 (+   105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.608875 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.609043 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.643142 12641 rpcz_store.cc:276] 0709 12:11:53.247327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.597166 (+2349839us) service_pool.cc:224] Handling call
0709 12:11:55.598460 (+  1294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.600439 (+  1979us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.600823 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.641114 12645 rpcz_store.cc:276] 0709 12:11:54.346627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.129200 (+782573us) service_pool.cc:224] Handling call
0709 12:11:55.480716 (+351516us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.660156 12646 rpcz_store.cc:276] 0709 12:11:53.246512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.545213 (+2298701us) service_pool.cc:224] Handling call
0709 12:11:55.575217 (+ 30004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.575930 (+   713us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:55.575964 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:55.576065 (+   101us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:11:55.576104 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.576185 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.576372 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:55.612195 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1444) took 2898 ms (client timeout 2228 ms). Trace:
I20260709 12:11:55.605540 12490 rpcz_store.cc:276] 0709 12:11:54.057717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.057872 (+   155us) service_pool.cc:224] Handling call
0709 12:11:54.058208 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.503462 (+1445254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.604359 (+100897us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:55.604692 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3024638848}
I20260709 12:11:55.670089 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.674316 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:55.675575 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1513) took 2427 ms. Trace:
I20260709 12:11:55.676311 12638 rpcz_store.cc:276] 0709 12:11:53.248390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.672818 (+2424428us) service_pool.cc:224] Handling call
0709 12:11:55.673750 (+   932us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.675268 (+  1518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.675487 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.676820 12643 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:55.684002 12644 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:55.561617 14273 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.616369 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.610756 14118 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:55.690409 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1511) took 2442 ms. Trace:
I20260709 12:11:55.694931 12643 rpcz_store.cc:276] 0709 12:11:53.247876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.667743 (+2419867us) service_pool.cc:224] Handling call
0709 12:11:55.676151 (+  8408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.683317 (+  7166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.690335 (+  7018us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.618062 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.611194 14084 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:55.666504 12568 rpcz_store.cc:269] 0709 12:11:52.714234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:53.437804 (+723570us) service_pool.cc:224] Handling call
0709 12:11:55.611791 (+2173987us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:55.612037 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4974314240}
I20260709 12:11:55.696210 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.640522 14172 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:55.732709 14172 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:55.703626 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1512) took 2447 ms. Trace:
I20260709 12:11:55.741671 12644 rpcz_store.cc:276] 0709 12:11:53.248134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.667841 (+2419707us) service_pool.cc:224] Handling call
0709 12:11:55.678541 (+ 10700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.691976 (+ 13435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.695076 (+  3100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.658152 12567 rpcz_store.cc:276] 0709 12:11:53.199389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.200172 (+1000783us) service_pool.cc:224] Handling call
0709 12:11:54.200463 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:54.200664 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:11:55.117216 (+916552us) log.cc:844] Serialized 14 byte log entry
0709 12:11:55.562409 (+445193us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:55.562560 (+   151us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:55.562714 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.562880 (+   166us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.563349 (+   469us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1095764,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":238,"raft.run_cpu_time_us":438,"raft.run_wall_time_us":438,"thread_start_us":1265836,"threads_started":2,"wal-append.queue_time_us":447466}
I20260709 12:11:55.709596 14293 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:11:55.712663 13987 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.644s	user 0.016s	sys 0.006s
I20260709 12:11:55.713682 13992 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:55.740342 12638 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:11:55.722630 12646 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:55.724354 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.731173 14118 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:55.655862 12566 rpcz_store.cc:276] 0709 12:11:53.461599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.394822 (+933223us) service_pool.cc:224] Handling call
0709 12:11:54.395161 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:54.407244 (+ 12083us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:54.407287 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:54.407780 (+   493us) log.cc:844] Serialized 26 byte log entry
0709 12:11:55.555122 (+1147342us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:55.555449 (+   327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.555660 (+   211us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:55.555770 (+   110us) raft_consensus.cc:1685] finished
0709 12:11:55.555792 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.557108 (+  1316us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11590,"raft.queue_time_us":12553,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":45956,"thread_start_us":1146688,"threads_started":1,"wal-append.queue_time_us":906016}
I20260709 12:11:55.716274 14113 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.748463 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:55.688630 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 436449us
I20260709 12:11:55.732790 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:11:55.777135 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 122) took 2343 ms. Trace:
I20260709 12:11:55.758509 13987 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.761785 14084 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:55.767120 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 232944us
W20260709 12:11:55.768409 14241 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.783s	user 0.011s	sys 0.019s
I20260709 12:11:55.787439 14241 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.792424 12645 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:55.798038 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1593) took 2588 ms. Trace:
W20260709 12:11:55.798714 14317 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:11:55.798638 12564 rpcz_store.cc:276] 0709 12:11:53.209278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.266255 (+1056977us) service_pool.cc:224] Handling call
0709 12:11:54.284555 (+ 18300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:54.285236 (+   681us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:54.285282 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:54.322792 (+ 37510us) log.cc:844] Serialized 26 byte log entry
0709 12:11:55.576345 (+1253553us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:55.577198 (+   853us) log.cc:844] Serialized 14 byte log entry
0709 12:11:55.745810 (+168612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.773728 (+ 27918us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:55.773798 (+    70us) raft_consensus.cc:1685] finished
0709 12:11:55.773821 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.797907 (+ 24086us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1097,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":6134,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":13611,"thread_start_us":1372187,"threads_started":2,"wal-append.queue_time_us":974270}
I20260709 12:11:55.801597 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 119) took 2537 ms. Trace:
I20260709 12:11:55.802146 12641 rpcz_store.cc:276] 0709 12:11:53.259379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.662451 (+2403072us) service_pool.cc:224] Handling call
0709 12:11:55.770304 (+107853us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.790397 (+ 20093us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:55.790428 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:55.790471 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:11:55.790628 (+   157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.790692 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:55.790842 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:55.815594 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.658838 12639 rpcz_store.cc:276] 0709 12:11:53.245864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.505089 (+2259225us) service_pool.cc:224] Handling call
0709 12:11:55.505975 (+   886us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.519927 (+ 13952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.533081 (+ 13154us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:55.913058 14007 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.670s	user 0.024s	sys 0.013s
I20260709 12:11:55.820986 13955 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:55.914997 13955 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:55.915870 13955 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:55.822484 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73412.9us
I20260709 12:11:55.823396 14273 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.922351 14273 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.928529 14311 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:55.826939 13987 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.827888 14241 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:55.828492 13258 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:55.777753 12646 rpcz_store.cc:276] 0709 12:11:53.426886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.665785 (+2238899us) service_pool.cc:224] Handling call
0709 12:11:55.722227 (+ 56442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.751055 (+ 28828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.769552 (+ 18497us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:11:55.805315 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145085us
I20260709 12:11:55.786042 12569 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:55.884506 12572 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:11:55.806061 14118 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:55.894435 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71407us
I20260709 12:11:55.930317 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:55.931485 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:55.933835 13992 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:55.824033 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:55.935822 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:55.950925 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:55.952936 13896 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.846s	user 0.014s	sys 0.006s
I20260709 12:11:55.953658 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.953838 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.957028 14276 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:55.916723 14172 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:55.966449 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:55.967850 12567 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:11:55.976634 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157479us
W20260709 12:11:55.984074 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 358) took 1763 ms (client timeout 1998 ms). Trace:
W20260709 12:11:55.985586 13258 rpcz_store.cc:269] 0709 12:11:54.220353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:54.234783 (+ 14430us) service_pool.cc:224] Handling call
0709 12:11:55.983676 (+1748893us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:55.983938 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2371676416}
I20260709 12:11:55.984977 14222 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:11:55.986734 14220 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.778s	user 0.008s	sys 0.022s
I20260709 12:11:55.986801 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:55.992089 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.010540 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:55.991623 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:55.976104 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:56.008915 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:56.021493 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1529) took 2293 ms. Trace:
I20260709 12:11:56.013247 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:56.026572 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:56.028090 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.028496 14220 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.030300 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.031298 12644 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:56.033983 14316 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.201s	user 0.001s	sys 0.007s
I20260709 12:11:56.033815 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.034855 12493 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.035594 14172 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.035159 14045 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:56.036327 14120 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:11:56.037931 12495 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:56.038604 14120 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:11:56.039852 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.040747 12565 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.040483 14222 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.042378 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.043226 14222 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.044292 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:56.044487 12490 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.044900 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:56.040099 14120 tablet_bootstrap.cc:492] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
W20260709 12:11:56.023592 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48203.5us
W20260709 12:11:56.036173 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97836.7us
I20260709 12:11:56.043401 12570 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:11:56.022209 12645 rpcz_store.cc:276] 0709 12:11:53.728386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.677348 (+1948962us) service_pool.cc:224] Handling call
0709 12:11:55.678046 (+   698us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.793099 (+115053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:56.020804 (+227705us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:56.021220 (+   416us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":259642624}
I20260709 12:11:56.030334 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.052038 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:11:56.052069 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59348.1us
I20260709 12:11:56.054322 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1344) took 1721 ms. Trace:
I20260709 12:11:56.055948 12638 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 2: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:56.054975 12640 rpcz_store.cc:276] 0709 12:11:54.327735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.134677 (+806942us) service_pool.cc:224] Handling call
0709 12:11:56.024060 (+889383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:56.061954 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:55.970706 13896 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.063436 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.069072 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.069437 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.071146 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.047343 14246 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.074365 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.080791 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.076875 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.084617 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1535) took 1141 ms. Trace:
I20260709 12:11:56.080875 14246 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:56.086273 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.076476 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.093971 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.094816 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:56.088176 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.085350 12643 rpcz_store.cc:276] 0709 12:11:54.942903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.696146 (+753243us) service_pool.cc:224] Handling call
0709 12:11:55.733414 (+ 37268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:55.734058 (+   644us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:11:55.945691 (+211633us) log.cc:844] Serialized 14 byte log entry
0709 12:11:56.080889 (+135198us) log.cc:844] Serialized 14 byte log entry
0709 12:11:56.084165 (+  3276us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:56.084237 (+    72us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:11:56.084289 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:56.084377 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:56.084539 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11527,"prepare.queue_time_us":302609,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":293825,"threads_started":2}
I20260709 12:11:56.079527 14045 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.069051 14090 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:56.101471 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.105613 12563 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.109975 14084 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.092845 14246 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.126570 12646 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.114084 12641 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:11:56.114877 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.119495 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.127456 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.127215 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.128592 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:56.046639 14276 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.142006 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:56.115602 12565 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:56.146377 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.141947 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:56.154805 14261 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.548s	user 0.018s	sys 0.005s
I20260709 12:11:56.155522 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.159515 12497 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:56.161912 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60343.8us
I20260709 12:11:56.164264 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:56.113942 14273 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.162014 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:56.188653 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137397us
I20260709 12:11:56.196167 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.169955 12566 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:11:56.191804 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103941us
W20260709 12:11:56.124061 13998 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.831s	user 0.023s	sys 0.000s
I20260709 12:11:56.199806 12570 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.202049 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.202613 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.123394 13896 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.155460 14261 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:56.210295 12497 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:56.210493 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.211580 12645 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:56.213539 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:56.235873 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:56.237859 13992 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:11:56.238914 13992 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:56.239375 13992 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.239733 13992 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:56.166235 14264 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.146s	user 0.002s	sys 0.021s
I20260709 12:11:56.126250 14090 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.242219 14264 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.243980 14090 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.219967 12572 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.127002 14293 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.245469 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.246171 12639 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.246910 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.248797 12493 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:56.249780 12563 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.252010 12497 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:11:56.253583 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.254640 12497 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.255515 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.255674 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.255539 13992 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.257347 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.257779 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.258378 12497 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:56.258821 12490 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:11:56.259763 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57503.1us
I20260709 12:11:56.223480 14248 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:11:56.224270 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.225335 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.226456 14007 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:56.121833 14320 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.234069 12492 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 1.
I20260709 12:11:56.217993 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:56.272876 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:56.258925 14324 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:11:56.207278 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.277181 12564 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.201445 14220 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.237928 14120 ts_tablet_manager.cc:1403] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 3.610s	user 0.042s	sys 0.035s
I20260709 12:11:56.313409 12565 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.261777 14084 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:56.298105 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.315310 12644 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:56.324898 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.324632 12491 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
W20260709 12:11:56.326440 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59603.4us
I20260709 12:11:56.326728 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.271385 14007 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.332573 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.334411 12493 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.258833 13992 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.340677 12570 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 2: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:56.337781 12496 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
I20260709 12:11:56.275337 14320 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:56.347867 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:56.348881 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.351401 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.338826 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:56.332716 14224 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.630s	user 0.013s	sys 0.003s
I20260709 12:11:56.355321 12491 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:56.343233 12490 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.315709 14264 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.343978 12566 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:56.352996 12563 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:56.353349 12565 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:56.338373 14007 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:56.360045 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.361707 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.349427 12568 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.269763 13998 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.362898 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:56.359006 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105713us
I20260709 12:11:56.367748 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.368430 12565 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.351809 14320 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.338339 14084 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.379385 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:56.372184 14132 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.769s	user 0.018s	sys 0.000s
I20260709 12:11:56.381940 12563 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:11:56.381573 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.385167 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.387327 12566 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:56.390688 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.384438 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:56.391582 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.392016 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.391922 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.392215 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.397665 12497 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.398526 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.400238 12491 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:56.403383 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:56.402056 12569 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:56.359333 14224 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:56.410125 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48467.2us
I20260709 12:11:56.410823 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:56.410061 12570 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:56.413075 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.415853 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:56.417200 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:56.417603 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 253527us
I20260709 12:11:56.418067 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.422514 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.426152 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.426537 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.429050 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:56.406217 12494 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:56.433847 12563 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:11:56.433483 14220 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.435537 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.435673 14220 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:56.436024 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.436372 12494 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:11:56.394301 14276 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.437929 12490 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.394702 13955 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.437207 12492 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:11:56.442030 12460 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
W20260709 12:11:56.442833 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.442862 12460 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:11:56.443763 14224 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.444146 12571 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:56.444985 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:56.447834 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:11:56.445945 14118 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:11:56.448575 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.448918 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.441951 13955 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.449229 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.389595 14187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a30ed0986fa641bd8f88912b4b9ee320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:11:56.455974 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:56.457216 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.458099 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:56.436602 13998 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.459115 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.460602 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.455883 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.462419 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.462963 12491 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 1.
I20260709 12:11:56.463454 12489 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.464178 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.464491 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.465212 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.465653 12491 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:56.461421 12494 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:56.467058 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.475618 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.477063 12564 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:56.436403 14220 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.476542 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:11:56.481452 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48446.9us
W20260709 12:11:56.479584 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 114965us
I20260709 12:11:56.480170 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:56.488382 12491 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.488548 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.479282 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.484500 12493 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:11:56.486711 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110561us
I20260709 12:11:56.492074 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:11:56.482558 14118 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.492879 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:56.441959 14246 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.497207 12489 raft_consensus.cc:2368] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3.
I20260709 12:11:56.499663 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:11:56.501070 12568 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:11:56.507285 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:56.494589 14118 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:56.500874 12639 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:56.514678 12491 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:11:56.517869 14291 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.518597 14291 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:56.519100 14291 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.520928 14118 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.524694 14084 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:56.512560 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64920.1us
W20260709 12:11:56.510425 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2542874480000137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260709 12:11:56.504053 14332 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.533357 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:56.483510 14120 raft_consensus.cc:359] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:56.541085 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:11:56.547214 14335 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.159s	user 0.002s	sys 0.000s
I20260709 12:11:56.541041 14045 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:56.550882 14246 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.551311 14061 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:56.557090 14118 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.537376 14332 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.561219 14332 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.536194 14241 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.563071 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:56.564100 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56942.9us
I20260709 12:11:56.538898 14120 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:11:56.566843 12492 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.571306 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.557613 14045 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:56.577759 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:56.578336 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:56.586494 12492 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:56.565364 14241 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:56.567437 14118 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.594370 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.597785 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:56.599196 14220 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.600482 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:56.601527 14305 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.601948 14305 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:56.602401 14305 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.573882 14061 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.605456 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:56.606493 13260 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.610332 14061 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:56.612200 14061 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.612489 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:56.611728 13258 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:56.617801 14261 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.620074 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.589241 14045 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:11:56.626588 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2416874479999933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
I20260709 12:11:56.591394 14241 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:56.635269 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2321698249998008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
W20260709 12:11:56.635881 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2056874480001625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21190000000000003s for a response
W20260709 12:11:56.636262 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1741874480001115s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2154s for a response
W20260709 12:11:56.636634 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1525874480000766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2178s for a response
W20260709 12:11:56.636999 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1300874480000402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
W20260709 12:11:56.637360 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1057874480000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260709 12:11:56.623535 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:56.649473 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:56.650342 13258 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:56.650679 14321 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.532s	user 0.001s	sys 0.005s
I20260709 12:11:56.653043 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:56.654109 13254 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:56.604566 14084 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
W20260709 12:11:56.677206 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48481.6us
I20260709 12:11:56.654692 14224 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:56.678676 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55266.3us
I20260709 12:11:56.657321 14246 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.672649 14260 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:11:56.686532 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.622864 14120 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:56.694238 14224 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.683347 14246 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.696697 14320 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.684953 14220 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:56.697789 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.698753 14305 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.706008 14346 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:11:56.714104 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65089.3us
I20260709 12:11:56.654534 14061 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.714684 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:56.719798 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 138661us
W20260709 12:11:56.723635 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:56.729839 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.655313 14007 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:56.726783 13260 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:56.715232 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:56.733824 14350 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:56.734541 14350 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:56.735517 14350 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.736155 14305 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.729944 14260 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:56.734016 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.738682 14260 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.739394 14007 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.740561 13955 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:56.741329 13955 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.739554 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.742859 12565 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:56.722222 14246 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.729105 14261 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:56.741787 13955 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.758543 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:56.759456 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.741840 14224 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.760989 14045 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.774094 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:56.779034 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.780158 14132 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.780723 14132 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.781111 14132 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.754504 14305 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:56.757105 13998 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:56.800976 14081 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:56.805090 14081 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:56.802379 14260 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.812081 14081 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.812736 14081 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.810956 13955 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.768671 14332 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.796211 13998 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:11:56.836416 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62536.5us
I20260709 12:11:56.838054 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:56.841666 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 187346us
I20260709 12:11:56.843266 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:56.844455 14061 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.845189 14120 ts_tablet_manager.cc:1434] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.517s	user 0.013s	sys 0.000s
I20260709 12:11:56.837814 13998 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:56.857782 13955 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:56.858119 14320 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.859923 14090 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:56.852530 14120 ts_tablet_manager.cc:1467] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Tablet startup took 540ms
I20260709 12:11:56.870373 14045 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:56.871362 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156898us
W20260709 12:11:56.872310 14331 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.482s	user 0.001s	sys 0.001s
I20260709 12:11:56.873692 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:56.860450 14090 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:11:56.876415 12461 connection.cc:256] RPC call timeout handler was delayed by -1.2281071499999143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2782s for a response
I20260709 12:11:56.849452 14241 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:56.879029 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.882726 14276 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:56.883219 14276 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.883558 14276 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.883788 14350 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.880091 14090 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:11:56.886370 14348 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 12:11:56.889453 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161310us
I20260709 12:11:56.894325 13218 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
I20260709 12:11:56.894848 13218 leader_election.cc:403] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:11:56.850242 14291 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:56.901939 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.893364 14355 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:56.903424 14355 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.903839 14355 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:11:56.903527 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.902531 12497 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:11:56.898779 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:56.906991 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.908779 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.910044 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.908918 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.914760 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.911643 12639 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:11:56.920652 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:56.924430 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0076071499997852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3027s for a response
I20260709 12:11:56.924974 14045 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.925513 14045 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:56.925891 14045 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.928732 14332 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:56.929549 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.931159 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:56.932121 14248 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.933208 14248 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.933862 14248 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.898312 14132 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.932355 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:56.939049 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96601.8us
I20260709 12:11:56.939934 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.940362 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.941095 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.901479 14291 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.946228 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.947127 12566 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:11:56.942108 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 160585us
I20260709 12:11:56.945279 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.946033 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.949286 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.951746 12568 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 1.
I20260709 12:11:56.950199 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.953343 12566 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.955515 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.956061 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.956779 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.957487 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:56.958065 12567 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.958904 14358 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:11:56.961880 12640 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:56.961970 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:56.962848 12570 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:56.963639 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:56.964202 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:56.965134 12646 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:56.966383 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.966559 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:56.974534 13998 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.980229 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:56.975842 14357 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:56.981213 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 114) took 3782 ms. Trace:
I20260709 12:11:56.981546 14357 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:11:56.977061 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:56.967229 12644 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:56.941196 14354 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
I20260709 12:11:56.992259 14248 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:56.994223 14352 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:11:56.993799 14081 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.984556 14332 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:56.996357 14081 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:56.996493 14332 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:56.997040 14332 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:56.998092 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:56.998715 12641 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3. Re-sending same reply.
I20260709 12:11:56.999610 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.000682 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.001220 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.001842 12642 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:57.002540 12638 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:57.003263 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.003669 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.004454 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.005522 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:57.008420 14329 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:57.008865 14329 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:57.009291 14329 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:57.013971 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:57.016076 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:57.017352 13955 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:56.964864 12567 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:56.968606 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:56.961709 14220 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:56.983476 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43569.3us
I20260709 12:11:57.019969 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:57.024286 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:57.025929 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57139.2us
I20260709 12:11:57.028424 12570 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:56.944811 14246 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:56.948379 14276 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:56.982054 14357 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:56.984103 14260 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:57.034152 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54089.7us
I20260709 12:11:56.981784 12637 rpcz_store.cc:276] 0709 12:11:53.199037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.151752 (+1952715us) service_pool.cc:224] Handling call
0709 12:11:55.473907 (+322155us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:55.516832 (+ 42925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:55.516867 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:55.609238 (+ 92371us) log.cc:844] Serialized 26 byte log entry
0709 12:11:55.776488 (+167250us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:55.776536 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:55.780210 (+  3674us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:56.591159 (+810949us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:11:56.652125 (+ 60966us) raft_consensus.cc:1685] finished
0709 12:11:56.652148 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:56.980803 (+328655us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54000b45d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:56.980974 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5043,"raft.queue_time_us":14443,"raft.run_cpu_time_us":667,"raft.run_wall_time_us":29999,"spinlock_wait_cycles":789050880,"thread_start_us":166682,"threads_started":1,"wal-append.queue_time_us":437391}
I20260709 12:11:57.038468 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.043934 12646 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:57.048358 12640 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:57.023240 14220 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:57.017987 14187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50696 (request call id 406) took 6684 ms. Trace:
I20260709 12:11:57.059242 12644 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:57.057380 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:57.033452 14291 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:57.053717 14264 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:11:57.082893 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66904.3us
I20260709 12:11:57.084515 12610 leader_election.cc:400] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4.
I20260709 12:11:57.084913 12610 leader_election.cc:403] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:11:57.057619 13258 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:57.058612 14220 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:56.868736 14120 ts_tablet_manager.cc:1469] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Trace:
0709 12:11:47.459358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.467117 (+  7759us) service_pool.cc:224] Handling call
0709 12:11:48.713461 (+1246344us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbfa805682
0709 12:11:48.786779 (+ 73318us) ts_tablet_manager.cc:934] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
0709 12:11:49.682775 (+895996us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:11:50.233863 (+551088us) tablet_metadata.cc:665] Metadata flushed
0709 12:11:50.234014 (+   151us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:11:50.987948 (+753934us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:11:51.035996 (+ 48048us) tablet_replica.cc:206] Creating consensus instance
0709 12:11:51.036405 (+   409us) ts_tablet_manager.cc:989] Replica registered.
0709 12:11:51.036564 (+   159us) inbound_call.cc:177] Queueing success response
0709 12:11:52.628152 (+1591588us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:11:56.354927 (+3726775us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:11:56.398995 (+ 44068us) tablet_replica.cc:261] Starting instrumentation
0709 12:11:56.399765 (+   770us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":104245,"raft.queue_time_us":10249,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":3345,"spinlock_wait_cycles":12456320,"tablet-copy.queue_time_us":1227236,"tablet-download-pool.queue_time_us":463,"tablet-download-pool.run_cpu_time_us":2936,"tablet-download-pool.run_wall_time_us":1066141,"thread_start_us":1279324,"threads_started":2}
I20260709 12:11:57.043308 14260 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.094990 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:57.099329 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1453) took 1262 ms. Trace:
I20260709 12:11:57.077860 14132 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:57.099977 12571 rpcz_store.cc:276] 0709 12:11:55.831271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:55.883701 (+ 52430us) service_pool.cc:224] Handling call
0709 12:11:57.093170 (+1209469us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:57.093355 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1122794240}
I20260709 12:11:57.088950 14061 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:57.081453 13998 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:57.110174 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:11:57.109896 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 3.358s (SENT) [suppressed 48 similar messages]
I20260709 12:11:57.089883 14291 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:11:57.128922 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48689.3us
W20260709 12:11:57.107293 14273 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.008s	sys 0.008s
W20260709 12:11:57.131884 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58241.7us
I20260709 12:11:57.142908 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1462) took 1030 ms. Trace:
I20260709 12:11:57.143379 12565 rpcz_store.cc:276] 0709 12:11:56.112903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:56.376287 (+263384us) service_pool.cc:224] Handling call
0709 12:11:56.396445 (+ 20158us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:56.743254 (+346809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:57.142644 (+399390us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:57.142838 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":795938688}
I20260709 12:11:57.146342 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:57.110754 14241 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:11:57.152869 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.160331 12493 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:57.161285 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.162070 12490 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:57.167313 14362 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4.
I20260709 12:11:57.139413 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:11:57.144135 14273 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.009s	sys 0.008s
I20260709 12:11:57.163633 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.177214 12496 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:57.140856 14132 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:57.172503 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:11:57.173414 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75863.8us
I20260709 12:11:57.175901 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.177029 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.168596 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.192104 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:57.192469 12497 raft_consensus.cc:2376] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4. Re-sending same reply.
I20260709 12:11:57.140645 14061 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:57.195279 12494 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:57.193997 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75591.1us
I20260709 12:11:57.128549 14090 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.192139 12489 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 4
W20260709 12:11:57.206525 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59551.9us
I20260709 12:11:57.160719 14351 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:57.169600 14305 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:57.209095 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.215828 12493 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:57.209949 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.216882 12497 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:57.210170 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.178328 14241 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:57.197059 14132 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:57.222832 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.224051 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.221324 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:57.228360 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.224869 12494 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3. Re-sending same reply.
I20260709 12:11:57.229564 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.231748 12490 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:57.231709 12493 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:57.058422 14187 rpcz_store.cc:276] 0709 12:11:50.333463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:50.333674 (+   211us) service_pool.cc:224] Handling call
0709 12:11:50.783158 (+449484us) ts_tablet_manager.cc:934] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
0709 12:11:56.375390 (+5592232us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:11:56.694708 (+319318us) tablet_metadata.cc:665] Metadata flushed
0709 12:11:56.694855 (+   147us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:11:56.853343 (+158488us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:11:56.941486 (+ 88143us) tablet_replica.cc:206] Creating consensus instance
0709 12:11:56.986711 (+ 45225us) ts_tablet_manager.cc:989] Replica registered.
0709 12:11:57.017819 (+ 31108us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":117543,"tablet-copy.queue_time_us":206589,"thread_start_us":242579,"threads_started":2}
I20260709 12:11:57.208389 14045 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.222340 14132 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:57.218526 14246 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:57.237455 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45486.1us
W20260709 12:11:57.238891 12611 connection.cc:256] RPC call timeout handler was delayed by -1.0884961279998606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
I20260709 12:11:57.242558 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:57.252686 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:57.257377 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:57.253517 14241 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:57.233546 14276 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:57.258401 13255 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:57.258806 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:57.258456 14090 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:57.268548 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.272248 12645 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:57.270428 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.273006 12640 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:57.275913 14045 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:57.276611 14045 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:57.277087 14045 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:57.278118 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.278918 12637 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:57.279670 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:57.220805 14351 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.296741 14248 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:57.298030 14061 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:57.299911 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:57.300385 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 120) took 11332 ms (client timeout 15000 ms). Trace:
I20260709 12:11:57.266279 14353 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:57.302532 14353 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:57.303460 14353 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.260172 14187 tablet_copy_client.cc:806] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:11:57.305917 14187 tablet_copy_client.cc:670] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:11:57.265239 14366 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260709 12:11:57.307188 12646 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:11:57.281147 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:57.310722 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-27144fedf0014de09f007ac6cc3f7275 received from {username='slave'} at 127.0.0.1:50142
I20260709 12:11:57.311484 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-27144fedf0014de09f007ac6cc3f7275 on tablet 27144fedf0014de09f007ac6cc3f7275 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:11:57.315342 14260 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:57.317050 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:57.322854 14367 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:11:57.327054 14187 tablet_copy_client.cc:538] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:11:57.332159 14045 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:57.336397 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:57.338828 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.348809 12496 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:11:57.353968 12566 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:11:57.254774 14261 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:57.356956 12569 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:57.258515 14241 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.398370 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.368921 12489 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
W20260709 12:11:57.300994 12392 rpcz_store.cc:269] 0709 12:11:45.968175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:45.968392 (+   217us) service_pool.cc:224] Handling call
0709 12:11:57.300225 (+11331833us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:55.319505 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:55.321166 (+  1661us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:55.321226 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:11:55.321256 (+    30us) tablet.cc:662] Decoding operations
0709 12:11:55.321769 (+   513us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:55.321869 (+   100us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:55.321920 (+    51us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:55.322220 (+   300us) tablet.cc:701] Row locks acquired
0709 12:11:55.322249 (+    29us) write_op.cc:260] PREPARE: finished
0709 12:11:55.322478 (+   229us) write_op.cc:270] Start()
0709 12:11:55.322645 (+   167us) write_op.cc:276] Timestamp: P: 1783599115322408 usec, L: 0
0709 12:11:55.322678 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 12:11:55.324039 (+  1361us) log.cc:844] Serialized 536 byte log entry
0709 12:11:56.350743 (+1026704us) op_driver.cc:464] REPLICATION: finished
0709 12:11:56.602626 (+251883us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf9229ef0
0709 12:11:56.840687 (+238061us) write_op.cc:301] APPLY: starting
0709 12:11:56.850905 (+ 10218us) tablet.cc:1367] starting BulkCheckPresence
0709 12:11:56.858913 (+  8008us) tablet.cc:1370] finished BulkCheckPresence
0709 12:11:56.858954 (+    41us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:11:56.859363 (+   409us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:11:56.859676 (+   313us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:11:56.859722 (+    46us) write_op.cc:312] APPLY: finished
0709 12:11:56.860122 (+   400us) log.cc:844] Serialized 23 byte log entry
0709 12:11:56.860319 (+   197us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:56.860450 (+   131us) write_op.cc:454] Released schema lock
0709 12:11:56.860507 (+    57us) write_op.cc:341] FINISH: Updating metrics
0709 12:11:56.967047 (+106540us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:11:56.983062 (+ 16015us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3252066,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":432859,"apply.run_cpu_time_us":3407,"apply.run_wall_time_us":115642,"num_ops":1,"prepare.queue_time_us":1712276,"prepare.run_cpu_time_us":13118,"prepare.run_wall_time_us":224467,"raft.queue_time_us":391531,"raft.run_cpu_time_us":78628,"raft.run_wall_time_us":513498,"replication_time_us":967795,"spinlock_wait_cycles":41265024,"thread_start_us":2410757,"threads_started":4,"wal-append.queue_time_us":211611}]]}
W20260709 12:11:57.395424 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122580us
I20260709 12:11:57.406107 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:57.231603 14291 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:57.407380 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120351us
I20260709 12:11:57.425536 12645 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:11:57.394104 14007 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
W20260709 12:11:57.430075 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.432197 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:57.339690 14351 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:57.370651 13955 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:57.451325 14377 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:57.458066 14320 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:11:57.459941 14241 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:57.461340 14350 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.577s	user 0.012s	sys 0.000s
I20260709 12:11:57.463766 14357 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.281311 14332 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:57.499866 14200 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
W20260709 12:11:57.505285 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 376271us
I20260709 12:11:57.439821 14366 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:11:57.538694 12611 connection.cc:256] RPC call timeout handler was delayed by -1.0047961279999527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
I20260709 12:11:57.538890 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.505575 14320 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:11:57.554394 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115382us
I20260709 12:11:57.505494 14377 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:57.513309 14351 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:57.558743 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:57.561095 14364 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:57.561822 14364 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:57.562391 14364 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:57.563583 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.564445 12640 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
W20260709 12:11:57.568938 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.574232 14351 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.575055 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:57.576879 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:57.578265 14368 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:57.578816 14368 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:57.507593 14305 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:57.578855 14081 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:11:57.552788 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:11:57.550592 12645 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:57.605046 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.556943 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:57.590257 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:57.582017 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:57.583811 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.607263 12489 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:57.466684 14329 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.546300 14366 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:11:57.622022 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 473308us
W20260709 12:11:57.623878 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84060.9us
W20260709 12:11:57.628466 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71790.3us
I20260709 12:11:57.632390 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:57.591614 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:57.610440 14373 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:57.638482 12492 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:57.641592 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:57.582242 14355 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.678s	user 0.012s	sys 0.003s
W20260709 12:11:57.647395 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66275.1us
I20260709 12:11:57.580551 14377 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:57.547839 14132 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.668408 12391 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:11:57.578282 14351 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:57.610860 14354 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:57.666882 14084 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:57.649647 14007 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:57.644191 13998 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:57.674674 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:57.659245 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69087.8us
I20260709 12:11:57.645648 14373 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:57.710482 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.709216 14372 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:57.699527 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85129.8us
I20260709 12:11:57.663076 14261 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:57.710008 14329 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:57.710690 14386 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:57.712086 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.735179 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:57.742533 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.665263 14332 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:57.784821 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.843339 12644 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:57.705364 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:57.845846 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 194131us
I20260709 12:11:57.846200 12643 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:11:57.846732 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:57.847275 12645 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:11:57.848769 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207653us
I20260709 12:11:57.845055 12637 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:11:57.711840 14367 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:11:57.860586 12489 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:11:57.861595 12641 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:57.682291 14246 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:57.862344 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.707592 14187 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
W20260709 12:11:57.865181 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 185472us
I20260709 12:11:57.865836 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:11:57.867173 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.868400 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.692446 14084 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:57.847999 14007 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:57.869822 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.870522 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.871207 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.871845 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:57.871618 14305 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.842809 14354 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:57.661120 14355 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.711089 14320 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.873772 14357 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:57.987411 14320 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:57.895674 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.989156 12638 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:11:57.977062 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:57.990759 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:57.992291 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.941567 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:57.994045 12645 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:57.941172 12646 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:11:57.994908 12642 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:57.993358 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.995433 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.996320 12637 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:58.000788 12643 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:58.001403 12567 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:58.002171 12644 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:11:58.002815 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.997603 12572 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:57.712968 14362 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:11:57.919462 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:57.946004 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.942533 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:58.007699 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.995434 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:57.947041 14384 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.212s	user 0.000s	sys 0.000s
I20260709 12:11:58.006296 12567 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:57.996934 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:58.004434 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:58.007810 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.011634 12570 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:11:57.945686 14389 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:11:58.012074 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.947258 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:57.705273 14368 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:57.849130 14386 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:58.006299 14246 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:57.855185 14373 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:58.066476 12641 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:58.016971 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.996663 14007 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:57.996802 14084 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:57.946153 14351 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:58.070024 12569 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:57.943926 14379 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:11:57.866549 14353 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.287s
I20260709 12:11:58.070748 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:57.706032 13955 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.152375 12645 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:57.852203 14372 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:57.918221 14273 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:58.155607 12642 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:58.149103 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.156972 12564 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:58.158308 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.012588 14261 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:11:58.161690 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 219205us
W20260709 12:11:58.171653 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 322092us
I20260709 12:11:58.017619 14241 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:57.873899 13998 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:58.017387 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:58.179780 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:58.206943 12571 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:58.191453 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:58.215387 12641 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:58.215632 12637 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.166249 14061 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:58.150596 14386 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:58.211305 12567 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:11:58.189893 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 122569us
W20260709 12:11:58.188089 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.214694 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:58.226239 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:58.219099 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:11:58.224738 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.233253 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:11:58.236722 12493 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.999s	user 0.005s	sys 0.004s
I20260709 12:11:58.237156 12493 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:11:58.237874 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1536) took 1025 ms. Trace:
I20260709 12:11:58.238149 12493 rpcz_store.cc:276] 0709 12:11:57.212718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.223925 (+ 11207us) service_pool.cc:224] Handling call
0709 12:11:58.237557 (+1013632us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:58.242803 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:58.243068 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.246687 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.207314 14261 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:58.192894 14372 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:58.253490 12640 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:11:58.256187 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:11:58.225363 14061 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:58.231201 14090 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:11:58.267125 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.181735 14379 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:11:58.244186 14332 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:11:58.249366 14248 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:58.271178 14061 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.298221 14248 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:58.298714 14248 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.301781 13998 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:58.302017 14291 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:58.246949 14329 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:11:58.211580 14273 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:58.255625 14261 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:58.216120 14351 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.216451 14241 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.297648 13256 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:11:58.231428 14045 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260709 12:11:58.318691 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:11:58.339874 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97132.5us
W20260709 12:11:58.340145 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97490.6us
W20260709 12:11:58.326344 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.001s	sys 0.011s
W20260709 12:11:58.344129 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.001s	sys 0.011s
I20260709 12:11:58.330896 14248 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.333276 12493 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:58.327081 12391 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.004s	sys 0.008s
I20260709 12:11:58.274597 14132 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:58.346840 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 717204us
W20260709 12:11:58.351578 12391 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.005s	sys 0.008s
I20260709 12:11:58.352113 14248 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:58.354238 14404 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:58.363824 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.364638 12564 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.369472 14372 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.373610 14372 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:58.297667 14090 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:58.383790 14090 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:58.344341 14332 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:11:58.386014 14081 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:11:58.386790 14220 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.257s	user 0.010s	sys 0.003s
I20260709 12:11:58.386657 14081 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:58.387198 14220 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.388931 14081 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:58.389144 14220 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:58.389583 14132 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:11:58.288713 14379 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:58.391405 14350 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:58.391948 14350 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:11:58.394415 14388 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 12:11:58.382794 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64247us
I20260709 12:11:58.332541 14291 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:11:58.271445 14383 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:58.382889 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:58.397172 14241 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:58.399947 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:58.403213 12491 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:11:58.412073 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.443392 14353 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:11:58.445828 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.446513 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:58.447185 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:58.447810 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47995.6us
I20260709 12:11:58.450139 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.451170 12567 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
W20260709 12:11:58.457926 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75121.5us
I20260709 12:11:58.239526 14354 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:58.465144 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.399962 14355 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:58.486970 14391 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.220s	user 0.001s	sys 0.001s
W20260709 12:11:58.594877 12492 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.948s	user 0.011s	sys 0.007s
I20260709 12:11:58.595561 12492 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:58.596194 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 136) took 1013 ms. Trace:
I20260709 12:11:58.596606 12492 rpcz_store.cc:276] 0709 12:11:57.582713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.591250 (+  8537us) service_pool.cc:224] Handling call
0709 12:11:58.596117 (+1004867us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:11:58.492700 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.597679 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.598052 12563 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.599090 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.599788 12492 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:58.600592 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.493547 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.447055 14383 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:11:58.499914 14381 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.692s	user 0.007s	sys 0.000s
I20260709 12:11:58.500735 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.602624 14383 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.501835 14241 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:11:58.502718 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56277.9us
W20260709 12:11:58.605895 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.503141 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.606352 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:58.560556 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 127) took 1291 ms. Trace:
I20260709 12:11:58.609671 13998 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:58.608343 12639 rpcz_store.cc:276] 0709 12:11:57.268875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.276352 (+  7477us) service_pool.cc:224] Handling call
0709 12:11:57.276717 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:57.537836 (+261119us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:57.540985 (+  3149us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:11:57.548784 (+  7799us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:57.551311 (+  2527us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:57.804738 (+253427us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b180035e0c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:11:58.560279 (+755541us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:58.560335 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:58.560471 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":163,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":42978,"spinlock_wait_cycles":424873600}
I20260709 12:11:58.610181 13998 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:58.610599 13998 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:58.605952 14241 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:11:58.611457 14241 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:58.613436 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 131) took 1465 ms. Trace:
W20260709 12:11:58.580163 14377 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.893s	user 0.006s	sys 0.004s
W20260709 12:11:58.578770 14364 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.016s	user 0.009s	sys 0.005s
I20260709 12:11:58.507119 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.508088 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.510193 14246 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:11:58.510462 14397 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:11:58.512681 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:11:58.627161 12489 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:11:58.628036 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115463us
W20260709 12:11:58.513293 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85866.3us
I20260709 12:11:58.458145 14248 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:11:58.515389 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.516628 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.519272 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:11:58.519481 12461 connection.cc:256] RPC call timeout handler was delayed by -1.2275684340002044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
W20260709 12:11:58.530320 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71251.4us
I20260709 12:11:58.537314 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.561472 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.601555 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.634384 12572 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:58.558187 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 304732us
I20260709 12:11:58.392951 14350 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:58.586206 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:58.587549 12571 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:11:58.578946 14320 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.591s	user 0.010s	sys 0.001s
I20260709 12:11:58.381232 14373 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.503566 14366 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.654129 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:11:58.617342 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.655948 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:58.656294 12496 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.504170 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.658373 12493 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.618829 12563 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:58.558001 14386 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:58.617306 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 263426us
I20260709 12:11:58.661590 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.646867 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.615872 12495 rpcz_store.cc:276] 0709 12:11:57.148484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.148662 (+   178us) service_pool.cc:224] Handling call
0709 12:11:57.169785 (+ 21123us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:11:57.170248 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:57.170290 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:11:57.175265 (+  4975us) log.cc:844] Serialized 26 byte log entry
0709 12:11:57.487477 (+312212us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:11:57.487539 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:57.487648 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:58.484119 (+996471us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:11:58.605411 (+121292us) raft_consensus.cc:1685] finished
0709 12:11:58.605443 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:11:58.613352 (+  7909us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":57254,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5719,"spinlock_wait_cycles":92573056,"thread_start_us":261998,"threads_started":1,"wal-append.queue_time_us":427953}
I20260709 12:11:58.653136 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.667011 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:58.533977 14061 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.668334 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:11:58.670261 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.535616 14372 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
I20260709 12:11:58.671253 12567 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 2: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:58.671841 13260 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:58.670274 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:58.668617 12488 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:58.672632 12489 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.673715 14359 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:11:58.674738 14406 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:58.671419 14118 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:11:58.709138 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.709180 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:58.676935 14045 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:58.664064 14246 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:58.655750 13987 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:11:58.681721 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:58.684389 14261 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.684731 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.686144 14329 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:58.628405 14377 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:58.686885 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.673664 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.673161 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:58.695631 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:11:58.698127 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:11:58.703501 14084 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.517s	user 0.010s	sys 0.000s
I20260709 12:11:58.705416 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.706312 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.706663 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:58.707572 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:58.681499 12643 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.680s	user 0.016s	sys 0.000s
I20260709 12:11:58.709765 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:58.710435 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:58.711566 14386 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:58.706817 14007 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.525s	user 0.006s	sys 0.011s
I20260709 12:11:58.713475 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.714555 12497 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.676501 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.628592 14364 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:58.806120 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.835181 12563 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:11:58.858579 12641 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:11:58.836192 12495 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:11:58.716696 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48125.4us
I20260709 12:11:58.859578 13987 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:58.681747 14248 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:58.835320 12638 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:11:58.837060 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.786540 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:58.715718 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:58.835819 12645 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:58.782670 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 134) took 1118 ms. Trace:
I20260709 12:11:58.862462 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:11:58.836725 12564 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:11:58.864609 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.859795 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:58.869669 12492 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.840263 12496 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:11:58.802850 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88517.7us
I20260709 12:11:58.836651 12643 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:11:58.755198 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:58.835382 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141993us
I20260709 12:11:58.716130 12489 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:11:58.759212 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.872773 12493 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:11:58.784919 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.872695 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1545) took 1199 ms. Trace:
I20260709 12:11:58.867318 12640 rpcz_store.cc:276] 0709 12:11:57.663790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.663998 (+   208us) service_pool.cc:224] Handling call
0709 12:11:57.664291 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:58.254134 (+589843us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:58.782357 (+528223us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:58.782584 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1331030784}
I20260709 12:11:58.794591 14305 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:58.861714 12490 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:58.713095 14351 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:58.713752 14273 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:58.881366 12570 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:58.871933 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.879758 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:58.881141 12643 rpcz_store.cc:276] 0709 12:11:57.673427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.867436 (+194009us) service_pool.cc:224] Handling call
0709 12:11:58.872538 (+1005102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49280}
I20260709 12:11:58.913681 12496 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.882616 12494 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:58.882434 12488 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:58.798862 14373 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:58.888564 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.890054 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:58.900928 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.865097 13987 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.864233 14007 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.891600 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.864128 14359 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:58.899108 14353 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:11:58.895498 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:11:58.898068 14379 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.508s	user 0.004s	sys 0.012s
I20260709 12:11:58.901597 14362 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:58.863365 14248 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:58.926731 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.928844 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:58.936841 12640 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:11:58.939977 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.938437 12460 leader_election.cc:400] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:11:58.959110 12460 leader_election.cc:403] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:11:58.959260 12566 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:11:58.961851 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:58.930757 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.962916 12496 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:58.875890 14045 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:58.799945 13998 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.860514 14084 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.860709 14329 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:58.952100 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:58.863627 14118 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:58.908162 14413 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:58.955363 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.955940 12638 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:11:58.865360 14241 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:58.956398 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:58.965971 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:58.967486 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87866.2us
I20260709 12:11:58.961110 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:58.970856 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:58.999045 12492 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:58.959635 14359 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:11:59.004544 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:58.965360 12490 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:11:58.945055 14399 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:58.992184 12640 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:58.874810 14246 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:58.967267 14332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.027s	sys 0.011s
W20260709 12:11:58.972119 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 339192us
I20260709 12:11:59.030987 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:11:59.032315 12461 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.511s (SENT)
I20260709 12:11:59.032931 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:59.002614 12489 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:59.005848 12565 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:59.025825 12496 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:58.984107 14362 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:59.002038 12637 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:11:59.039496 12496 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:59.029428 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 346805us
I20260709 12:11:59.041656 12493 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:58.956712 14379 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:59.029703 14332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.028s	sys 0.011s
I20260709 12:11:58.924265 14305 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:59.032941 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:11:59.026803 14386 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.026455 14351 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:11:59.056154 14362 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.059857 12640 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:59.056802 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:11:59.054330 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68427.8us
W20260709 12:11:59.013096 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:59.075474 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:59.047694 14357 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:59.048870 14224 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.000610 14118 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:11:59.098146 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100390us
I20260709 12:11:59.095820 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:59.101194 13260 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:11:59.101248 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:59.119280 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:11:59.119866 14353 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.120816 12488 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:59.119519 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:11:59.134967 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:11:59.139482 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107184us
W20260709 12:11:59.140617 12461 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.511s (SENT)
I20260709 12:11:59.143029 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.141227 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:59.149679 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:59.149340 12493 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:59.150408 12640 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:11:59.150708 12567 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:59.155607 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.156279 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:11:59.158025 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:11:59.158737 12489 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:11:59.161528 12493 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:59.124761 14406 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:11:59.164758 12461 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.518s (SENT)
I20260709 12:11:59.138993 14357 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:59.155504 14413 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:59.170305 14399 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.171649 14305 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.172214 13987 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.175262 14084 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.176522 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 247) took 2039 ms. Trace:
I20260709 12:11:59.178201 14007 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.124313 14248 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.180440 13987 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.181568 14364 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.182011 12563 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:11:59.183279 14118 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.161208 14359 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.183972 14413 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.186355 13998 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:59.191609 14366 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:59.192915 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:11:59.193562 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:59.194165 14422 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.194794 14350 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:59.197264 14241 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:59.198707 14246 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.159050 14261 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:11:59.206007 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50488.7us
I20260709 12:11:59.207019 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.208577 12489 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:11:59.193004 12564 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:11:59.197479 12568 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:59.212687 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:59.213522 12490 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:59.212463 13254 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:59.176996 13256 rpcz_store.cc:276] 0709 12:11:57.137614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.196844 (+ 59230us) service_pool.cc:224] Handling call
0709 12:11:57.313055 (+116211us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:11:57.313276 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:58.349222 (+1035946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:59.176221 (+826999us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:11:59.176421 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1622161408}
I20260709 12:11:59.218947 14379 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.219120 14118 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:11:59.158053 14332 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.165556 14377 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.220778 14273 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:59.221786 14273 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.145723 14045 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.224269 14273 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:11:59.225715 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1640) took 1052 ms. Trace:
I20260709 12:11:59.226460 14329 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.162992 14353 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.228092 14357 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.222101 14220 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:59.240775 12567 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:11:59.226171 12570 rpcz_store.cc:276] 0709 12:11:58.182956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:58.183339 (+   383us) service_pool.cc:224] Handling call
0709 12:11:59.225332 (+1041993us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:59.225515 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1460005120}
I20260709 12:11:59.242308 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:59.245239 12564 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:11:59.245157 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:59.246582 12488 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:11:59.245574 14332 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:11:59.245939 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:59.247756 12568 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:59.246071 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:59.244127 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:59.248349 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.248631 12570 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:11:59.245059 14373 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.249457 13260 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:11:59.258679 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:59.260347 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:59.261653 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:59.263850 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:11:59.265615 12460 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:11:59.266072 12460 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260709 12:11:59.276710 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120619us
I20260709 12:11:59.278239 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.281816 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:11:59.242823 14220 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:59.275420 14379 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:11:59.290618 14291 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:11:59.321712 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:59.291462 14305 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.332917 14375 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:11:59.258203 14420 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:11:59.302440 14320 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:11:59.352335 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:11:59.279371 14424 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 12:11:59.383105 14377 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:59.383961 14416 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:11:59.383962 14377 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.384884 14220 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.386066 14220 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:59.394649 14428 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:11:59.395783 14420 consensus_queue.cc:1243] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:11:59.396500 14420 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:11:59.396888 14420 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:11:59.398051 14420 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.399789 14420 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:11:59.401790 14383 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:11:59.403257 14291 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:11:59.403659 14291 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:11:59.404567 14291 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.408461 12567 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:11:59.394946 14377 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:11:59.431380 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 153268us
W20260709 12:11:59.498920 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 233481us
I20260709 12:11:59.405547 14329 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:59.405784 14261 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:11:59.350922 14406 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.566494 14428 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:11:59.567129 14428 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:11:59.569805 14405 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.996s	user 0.013s	sys 0.046s
W20260709 12:11:59.570339 14405 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.239s	user 0.014s	sys 0.047s
I20260709 12:11:59.573933 14291 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:59.594311 12492 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
W20260709 12:11:59.605994 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:59.618449 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 359868us
I20260709 12:11:59.620352 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:59.622648 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:59.623713 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1530) took 2439 ms (client timeout 1721 ms). Trace:
W20260709 12:11:59.624717 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:59.625571 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:11:59.631070 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 382775us
W20260709 12:11:59.624229 12491 rpcz_store.cc:269] 0709 12:11:57.154753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.191353 (+ 36600us) service_pool.cc:224] Handling call
0709 12:11:59.593810 (+2402457us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:11:59.594038 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2715944192}
I20260709 12:11:59.659086 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:11:59.649062 14187 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:11:59.664908 12567 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:11:59.648221 14428 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:59.664055 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.665359 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:11:59.670228 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49961.4us
I20260709 12:11:59.671116 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:11:59.660400 14377 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260709 12:11:59.677525 14351 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:59.666843 14366 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:11:59.678336 14366 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:11:59.678233 14351 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.678875 14366 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:11:59.681921 14351 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.684841 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:11:59.687381 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:11:59.651185 14386 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:11:59.663612 14261 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:59.688611 14404 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:11:59.688889 14261 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.689270 14404 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:11:59.689909 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:59.691517 14261 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.687762 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:59.695982 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:11:59.696321 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:59.697522 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:11:59.700908 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:59.701802 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:11:59.705958 12567 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 1. Current term is 2. Ops: []
I20260709 12:11:59.708956 14260 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:11:59.691522 14404 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:11:59.713477 14404 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:11:59.714244 14404 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:11:59.715464 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:11:59.715660 14404 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:11:59.707643 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:59.703779 14355 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:11:59.793715 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:11:59.793773 14357 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:59.805895 14260 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:11:59.808255 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:11:59.808481 14260 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:11:59.810819 14260 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:11:59.709090 14377 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:11:59.717799 14320 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:11:59.661651 14439 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:11:59.814226 13253 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
W20260709 12:11:59.805895 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134868us
I20260709 12:11:59.815572 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:11:59.820475 14261 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:11:59.796344 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:11:59.831153 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111081us
W20260709 12:11:59.826695 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116447us
I20260709 12:11:59.836369 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:11:59.837610 12492 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:11:59.843923 12567 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:11:59.845700 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:11:59.827221 13998 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:11:59.934933 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119496us
W20260709 12:12:00.002713 14435 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:12:00.003542 14187 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20260709 12:12:00.003742 14438 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.299s	user 0.000s	sys 0.000s
I20260709 12:11:59.928375 14305 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:11:59.901225 14358 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.155s
I20260709 12:11:59.902989 14260 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:11:59.929674 14426 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.735s	user 0.009s	sys 0.010s
I20260709 12:11:59.900331 14375 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:00.013342 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 311434us
I20260709 12:12:00.039741 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:11:59.938583 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93044.8us
I20260709 12:12:00.041651 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:00.018312 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:00.049542 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.023711 14187 tablet_bootstrap.cc:492] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:11:59.917325 14357 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
W20260709 12:12:00.026185 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87859.6us
I20260709 12:12:00.016609 14332 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:00.017098 14420 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:00.030848 14305 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:00.016165 14351 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:12:00.055747 12461 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:12:00.076475 12461 leader_election.cc:403] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:00.076507 12566 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:00.077857 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1495) took 1219 ms. Trace:
I20260709 12:12:00.078287 12566 rpcz_store.cc:276] 0709 12:11:58.858825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:59.187350 (+328525us) service_pool.cc:224] Handling call
0709 12:12:00.066856 (+879506us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:00.067224 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:00.077385 (+ 10161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:00.077763 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54869248}
I20260709 12:12:00.084357 12567 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:00.086615 12567 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:00.086715 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1773) took 1356 ms. Trace:
I20260709 12:12:00.063912 12568 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:00.045742 13998 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:00.054901 14373 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:00.090703 12488 rpcz_store.cc:276] 0709 12:11:58.711389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:58.995652 (+284263us) service_pool.cc:224] Handling call
0709 12:12:00.067682 (+1072030us) spinlock_profiling.cc:243] Waited 88.9 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:00.067900 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204509952}
I20260709 12:12:00.057276 14260 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:12:00.105675 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.066499 14305 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:00.095232 14187 ts_tablet_manager.cc:1403] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 2.424s	user 0.048s	sys 0.024s
W20260709 12:12:00.121659 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66020.7us
W20260709 12:12:00.099440 14248 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.918s	user 0.010s	sys 0.026s
I20260709 12:12:00.134011 12646 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:12:00.134305 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1489) took 1446 ms. Trace:
I20260709 12:12:00.128682 12461 leader_election.cc:400] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:12:00.053583 14375 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.122386 14320 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:00.118148 14357 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:00.134634 12646 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:00.141490 14248 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.011s	sys 0.027s
I20260709 12:12:00.134222 14377 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:00.145387 12646 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.147022 12646 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:00.068243 14420 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.135061 12569 rpcz_store.cc:276] 0709 12:11:58.688284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:58.905278 (+216994us) service_pool.cc:224] Handling call
0709 12:11:58.905811 (+   533us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:11:58.906326 (+   515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:11:58.906359 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:11:58.907103 (+   744us) log.cc:844] Serialized 52 byte log entry
0709 12:11:59.157893 (+250790us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:11:59.157940 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:11:59.158017 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:11:59.825287 (+667270us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:00.129295 (+304008us) raft_consensus.cc:1685] finished
0709 12:12:00.129332 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:00.134211 (+  4879us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17226,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":12826,"spinlock_wait_cycles":21436288,"thread_start_us":239421,"threads_started":1,"wal-append.queue_time_us":287485}
I20260709 12:12:00.119189 13998 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.143630 14332 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.159576 14403 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:12:00.161010 14332 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.162755 14445 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:00.171407 14355 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:12:00.183974 14366 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.504s	user 0.011s	sys 0.001s
W20260709 12:12:00.185853 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.146950 14377 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.139658 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:00.204412 12646 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:12:00.239459 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1397) took 1545 ms. Trace:
W20260709 12:12:00.212462 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.216378 14351 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:00.223378 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.152933 14357 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:12:00.283752 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 244154us
W20260709 12:12:00.240558 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101015us
I20260709 12:12:00.307931 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:00.240113 12646 rpcz_store.cc:276] 0709 12:11:58.696872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:58.954972 (+258100us) service_pool.cc:224] Handling call
0709 12:11:58.987067 (+ 32095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:00.133895 (+1146828us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b54001bc720. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:12:00.238926 (+105031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:00.239357 (+   431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111444736}
I20260709 12:12:00.312252 12492 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:00.317354 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:00.318722 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:00.326160 12496 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:00.327785 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.330524 12496 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:00.266096 14401 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:12:00.336593 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.342387 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:00.250222 14329 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.680s	user 0.011s	sys 0.001s
I20260709 12:12:00.341216 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.341771 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.245998 14403 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.349900 12496 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
W20260709 12:12:00.347162 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.349754 12488 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5. Re-sending same reply.
I20260709 12:12:00.316768 14353 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.341531 14220 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:12:00.318809 14357 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:00.290304 14420 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:00.356057 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.327142 14439 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.361619 12641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "9485b0afaf144396adf854fff857414e"
 from {username='slave'} at 127.0.0.1:58586
W20260709 12:12:00.362258 14448 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:12:00.197932 14366 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.368294 14375 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:00.369589 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1390) took 1680 ms. Trace:
I20260709 12:12:00.323730 14454 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:00.373927 12641 raft_consensus.cc:606] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Received request to transfer leadership to TS 9485b0afaf144396adf854fff857414e
I20260709 12:12:00.390161 12641 raft_consensus.cc:614] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:12:00.382659 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1635) took 2402 ms (client timeout 2707 ms). Trace:
W20260709 12:12:00.391162 12565 rpcz_store.cc:269] 0709 12:11:57.980146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.995853 (+ 15707us) service_pool.cc:224] Handling call
0709 12:12:00.382389 (+2386536us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:00.382588 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2286670592}
I20260709 12:12:00.355985 14329 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:12:00.387008 12637 rpcz_store.cc:276] 0709 12:11:58.688852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:58.689350 (+   498us) service_pool.cc:224] Handling call
0709 12:11:58.715641 (+ 26291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:11:59.036580 (+320939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:00.369315 (+1332735us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:00.369526 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":657914624}
I20260709 12:12:00.371331 14305 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.371855 14445 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.325023 14351 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.343163 14358 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:12:00.387409 12492 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:12:00.372175 14353 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:00.404888 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 355439us
I20260709 12:12:00.405894 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.407212 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.408862 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.338861 14320 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.416622 12489 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:00.414182 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.396394 14061 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:00.419090 12493 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:12:00.396838 14439 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:12:00.399196 13998 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:00.412915 14332 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:00.389460 14187 raft_consensus.cc:359] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 4 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.397307 14454 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.442234 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1507) took 1198 ms. Trace:
W20260709 12:12:00.422629 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 364309us
I20260709 12:12:00.442866 12572 rpcz_store.cc:276] 0709 12:11:59.243997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:59.922582 (+678585us) service_pool.cc:224] Handling call
0709 12:12:00.005030 (+ 82448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:00.244895 (+239865us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:00.244944 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:00.267033 (+ 22089us) log.cc:844] Serialized 26 byte log entry
0709 12:12:00.412551 (+145518us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:00.412593 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:00.412679 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:00.441932 (+ 29253us) raft_consensus.cc:1685] finished
0709 12:12:00.441957 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:00.442133 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":156700,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":326,"thread_start_us":352535,"threads_started":2,"wal-append.queue_time_us":149379}
I20260709 12:12:00.445868 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:00.450146 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:00.427125 14248 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.455318 14351 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:00.456610 14187 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
W20260709 12:12:00.456986 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.457731 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:00.459010 12641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:12:00.498401 12641 raft_consensus.cc:606] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:00.460419 14305 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:00.462013 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.498803 12641 raft_consensus.cc:614] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:00.463338 14454 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:00.464375 12565 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:00.445111 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:00.466809 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:00.481495 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:00.482193 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73653.4us
I20260709 12:12:00.505115 12495 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 2: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:00.434572 14061 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.481302 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:00.496196 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:00.511816 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:00.512241 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:00.435811 13998 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.467525 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:00.507014 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:00.509233 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1782) took 1782 ms (client timeout 1601 ms). Trace:
I20260709 12:12:00.510821 12570 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
W20260709 12:12:00.515836 12495 rpcz_store.cc:269] 0709 12:11:58.727260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:58.859895 (+132635us) service_pool.cc:224] Handling call
0709 12:12:00.461828 (+1601933us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:00.509112 (+ 47284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2733312}
W20260709 12:12:00.512789 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98683.7us
W20260709 12:12:00.514073 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7049358530000518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20260709 12:12:00.521247 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6707358529999965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260709 12:12:00.521672 12536 connection.cc:256] RPC call timeout handler was delayed by -2.59873585299988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 12:12:00.522023 12536 connection.cc:256] RPC call timeout handler was delayed by -2.58793585300009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
W20260709 12:12:00.522382 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4232358530000511s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
W20260709 12:12:00.522819 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4016358530000161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
I20260709 12:12:00.518705 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:00.523142 12536 connection.cc:256] RPC call timeout handler was delayed by -2.27563585300004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
W20260709 12:12:00.523463 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1595358530000794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21200000000000002s for a response
W20260709 12:12:00.523758 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0722358529999383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
W20260709 12:12:00.524096 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9948358530000405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2303s for a response
W20260709 12:12:00.524425 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7393319380000776s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20260709 12:12:00.524744 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7141319380000368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
W20260709 12:12:00.525043 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6493319379999321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
W20260709 12:12:00.525348 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5863319380000576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21650000000000003s for a response
I20260709 12:12:00.526001 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:00.527981 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2461319379999622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2543s for a response
W20260709 12:12:00.514405 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137873us
I20260709 12:12:00.463966 14453 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:12:00.531401 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:00.533864 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66626.1us
I20260709 12:12:00.533236 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 370) took 1452 ms. Trace:
I20260709 12:12:00.537748 14375 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:00.538368 14375 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:00.539196 14457 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:12:00.539379 14375 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.516623 14383 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:00.542452 14383 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:00.542610 13998 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:00.542869 14383 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:00.543278 13998 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.544240 13998 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.517395 14061 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.550041 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:00.551083 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:00.560098 14453 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:12:00.560746 14453 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.561003 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:00.559929 14353 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:00.529346 14329 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:12:00.533386 14305 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 12:12:00.547299 14403 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
W20260709 12:12:00.573040 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44181.4us
I20260709 12:12:00.573963 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.534106 14357 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.537521 13258 rpcz_store.cc:276] 0709 12:11:59.080447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:59.121712 (+ 41265us) service_pool.cc:224] Handling call
0709 12:12:00.457529 (+1335817us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:00.532000 (+ 74471us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23476608}
I20260709 12:12:00.561597 14332 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:00.561956 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 369) took 1484 ms. Trace:
I20260709 12:12:00.584721 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:00.565326 14366 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:00.533246 14187 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:00.582945 13255 rpcz_store.cc:276] 0709 12:11:59.077680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:59.118912 (+ 41232us) service_pool.cc:224] Handling call
0709 12:12:00.560865 (+1441953us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:00.561880 (+  1015us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19042688}
I20260709 12:12:00.533492 14456 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.556443 14463 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:12:00.575098 14455 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:12:00.577472 14445 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:12:00.594439 14061 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:00.607491 14132 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:00.617300 14132 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.618157 14357 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:00.619113 14187 ts_tablet_manager.cc:1434] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.489s	user 0.011s	sys 0.001s
I20260709 12:12:00.618884 14357 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:00.603762 14469 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:00.626880 14377 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:00.627871 14452 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:00.606177 14454 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:12:00.613857 14470 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:00.639340 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.631492 13258 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:00.605095 14453 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.634804 14466 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:12:00.671933 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.682796 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:00.685034 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71091.2us
I20260709 12:12:00.630477 14332 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:12:00.606459 14456 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.695909 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.647820 14452 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.665750 14470 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:00.693368 14305 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.689422 14466 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.693033 14463 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:00.712008 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:00.703409 14224 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:00.698987 14447 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:00.711402 14454 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:12:00.709800 14445 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:00.720842 14224 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:00.721936 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.711956 14473 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.729519 14332 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:00.733110 14332 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.706758 12536 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:00.676884 14446 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:12:00.732015 14454 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:00.739557 12536 leader_election.cc:403] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:12:00.762313 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:00.802567 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 191924us
I20260709 12:12:00.718884 14305 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:00.804273 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 273156us
I20260709 12:12:00.806553 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:00.804939 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:00.733407 14424 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:12:00.807039 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100518us
I20260709 12:12:00.808173 12491 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
W20260709 12:12:00.808701 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 231665us
I20260709 12:12:00.809271 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.810379 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:00.733950 14224 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:00.823506 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:00.819468 14482 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:00.820500 14445 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:12:00.821374 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:00.745316 14455 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:00.743948 14454 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:00.828269 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:00.818296 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:00.857312 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50855.5us
I20260709 12:12:00.858994 12490 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:00.859686 14445 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:00.860379 12492 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:00.817771 14424 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:00.817657 14446 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:00.807971 14456 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:00.728240 14463 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.881701 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.829475 14452 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:00.854426 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:00.855646 14479 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.888985 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:00.883329 13998 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:00.915840 12492 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:00.919855 14424 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.921561 14424 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:00.735504 14332 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:00.734838 14473 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.924154 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:00.737604 14466 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.737980 14470 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.923405 14383 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:00.923074 14446 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:00.970984 12494 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:00.933696 14456 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260709 12:12:00.956771 13256 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:00.933097 14445 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:00.998795 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:00.957921 14446 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:00.895782 14452 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.001559 12492 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:00.996606 14359 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.810s	user 0.000s	sys 0.037s
I20260709 12:12:00.925108 14479 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.015005 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.015791 12488 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:01.016219 14479 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:00.925848 13998 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.016911 14224 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.018884 14473 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:00.996671 14466 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:00.922636 14454 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:00.893532 14482 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 12:12:00.998106 14383 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.022481 14446 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:01.003492 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 375) took 1763 ms (client timeout 1540 ms). Trace:
I20260709 12:12:00.895043 14483 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
W20260709 12:12:01.021020 14359 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.834s	user 0.000s	sys 0.037s
W20260709 12:12:01.043653 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 238760us
I20260709 12:12:01.046372 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:12:01.053316 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:01.055032 14478 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.050148 14472 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:01.060047 14481 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:12:01.055229 14488 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.061493 14479 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:01.060396 14472 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:01.062075 14479 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:01.062309 14472 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:01.062654 14479 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:01.047662 13256 rpcz_store.cc:269] 0709 12:11:59.240383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:59.242805 (+  2422us) service_pool.cc:224] Handling call
0709 12:12:00.826417 (+1583612us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:01.001042 (+174625us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28781824}
I20260709 12:12:01.060456 14224 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.072376 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.073350 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:01.075641 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:01.074991 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:01.077400 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:01.077528 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.080070 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:01.096065 14495 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.058614 14305 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.072121 14454 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:01.114297 14463 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.059686 14478 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.129035 12567 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:01.110819 14495 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.136087 14455 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:12:01.138417 13955 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.145449 14332 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.145996 14453 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.061265 14488 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.150854 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:01.153847 13955 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.157178 14478 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.159838 13998 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:01.163756 14479 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.140513 14496 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:01.167373 14132 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.169586 14495 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.137308 14446 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.174692 14320 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:01.189265 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.199416 12568 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:01.193222 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.201117 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:01.205058 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:01.216006 14446 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.195528 14496 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:01.217209 14496 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:01.198261 13998 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:01.198745 14320 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:01.183741 14453 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.180581 14332 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:01.228204 14362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.051s	user 0.031s	sys 0.034s
I20260709 12:12:01.237095 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.237489 13998 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:01.237774 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.238209 14420 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.238683 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.238582 13998 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.238751 14420 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.239531 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:01.241799 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.242444 12643 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:01.231407 14132 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:01.236131 14362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.062s	user 0.032s	sys 0.035s
I20260709 12:12:01.244706 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.244930 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.245605 12642 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:12:01.245750 12639 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:01.247303 12645 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:12:01.233201 14455 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.254570 12639 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:01.233361 14495 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:01.249195 12641 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:01.255491 14320 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.250708 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.257128 14320 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:01.257697 12637 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
I20260709 12:12:01.252334 14453 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:01.259023 13955 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:01.253610 14479 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:01.254222 14455 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.255250 14495 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:01.250264 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.259982 14495 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:01.260233 12645 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:01.262223 12645 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:12:01.264288 14332 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.265746 14329 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.266170 12645 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:01.266358 14329 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.268680 14329 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.301793 14451 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.302649 12497 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:01.304194 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.305047 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.305828 12565 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:01.306598 14420 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.306869 12565 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:01.308373 12565 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:12:01.302407 14451 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.291651 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:01.275473 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:01.288383 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.953s	user 0.007s	sys 0.006s
W20260709 12:12:01.313832 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.979s	user 0.007s	sys 0.006s
I20260709 12:12:01.288465 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.305074 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:01.315034 12496 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:01.290872 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.290082 12492 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:01.296461 12643 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:01.288841 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.294636 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.318440 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.273283 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:01.273454 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.319353 12490 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:01.319720 12567 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:01.320123 12570 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:01.275979 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:01.314179 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 981751us
I20260709 12:12:01.323273 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.283788 12489 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 1. Current term is 2. Ops: []
I20260709 12:12:01.317236 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:01.320923 12563 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:01.327111 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.294246 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:01.312752 12565 pending_rounds.cc:85] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 4
I20260709 12:12:01.334012 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:01.335299 13218 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
I20260709 12:12:01.338084 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:01.334030 14377 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:01.344559 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:01.346626 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.346962 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:01.336355 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:01.267781 14455 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.332122 12565 pending_rounds.cc:107] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:12:01.360179 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.370078 12565 raft_consensus.cc:2889] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:01.363844 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.365283 12645 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:01.343354 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-a30ed0986fa641bd8f88912b4b9ee320 received from {username='slave'} at 127.0.0.1:50142
I20260709 12:12:01.343350 14377 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.286757 14488 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.393879 14061 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.394485 14061 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.395676 12646 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:12:01.396209 12570 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:01.396733 14061 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.397513 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.398401 14483 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.398944 14483 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.401234 14483 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.383720 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-a30ed0986fa641bd8f88912b4b9ee320 on tablet a30ed0986fa641bd8f88912b4b9ee320 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:12:01.402101 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.383841 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:01.403340 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.403752 13258 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:01.384654 13253 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:12:01.405169 12495 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:01.405468 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.406379 12646 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:01.385861 14451 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:01.410737 12497 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:12:01.411320 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.411695 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.411954 12565 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:01.412216 12497 raft_consensus.cc:2376] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2. Re-sending same reply.
I20260709 12:12:01.413043 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:01.413146 12565 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:12:01.370108 14498 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.415470 14498 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:01.417929 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73444.2us
I20260709 12:12:01.418336 14452 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.418792 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:01.419123 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.420063 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:01.420210 12536 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:01.420939 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.355931 14375 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260709 12:12:01.386509 12563 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:01.386759 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:01.388233 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.425850 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.426741 12640 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:12:01.391041 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.428438 12567 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:01.428577 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81694.3us
I20260709 12:12:01.429648 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:01.430270 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:01.431120 12497 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:01.431209 12643 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:01.370836 12641 raft_consensus.cc:2376] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4. Re-sending same reply.
I20260709 12:12:01.436378 12564 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:01.441892 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:01.442337 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.343982 14359 raft_consensus.cc:2764] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:12:01.356401 14332 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:01.462895 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.476195 12494 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:01.477182 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:01.477917 12494 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:01.478353 12563 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:01.478673 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.479964 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:01.480036 14496 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.480693 12494 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:01.482184 12645 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:01.482323 12637 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:01.462929 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.470639 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.484629 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.425307 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:01.481536 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:01.497912 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56158.4us
I20260709 12:12:01.356145 14408 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20260709 12:12:01.385402 14377 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:01.472908 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:01.473198 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.357993 14453 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.460016 13259 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:01.331637 14329 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260709 12:12:01.474063 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.474480 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:01.485723 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.485702 12646 raft_consensus.cc:2376] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3. Re-sending same reply.
I20260709 12:12:01.493590 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:01.388710 14462 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 12:12:01.498904 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:01.509037 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:01.837024 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.838189 12640 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
W20260709 12:12:01.845158 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 414906us
I20260709 12:12:01.846364 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:01.390388 14456 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:01.388513 14355 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:01.856693 12496 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 2: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:01.575249 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:01.574837 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:01.498473 12494 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3. Re-sending same reply.
I20260709 12:12:01.883066 12497 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:01.871675 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.812288 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:01.389794 14478 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:01.796272 14460 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.267s	user 0.000s	sys 0.001s
I20260709 12:12:01.436910 14452 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:01.884637 14478 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:01.885142 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:01.886221 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:12:01.430902 14472 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:01.888355 14357 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:12:01.888975 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.889863 12646 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:01.890314 14472 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:01.890476 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.890908 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.891314 12496 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:01.891899 12646 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:01.892251 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.893112 12646 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:01.893289 13955 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:01.893735 13955 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:01.893919 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.894214 13955 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:01.894140 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:01.506842 14305 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:01.497063 14451 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:12:01.895360 12494 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:01.895799 14451 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:01.896567 12494 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:01.894769 12495 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:01.713349 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:01.460033 14455 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.486s
I20260709 12:12:01.896853 14451 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:01.893412 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47171.6us
W20260709 12:12:02.035027 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.004s	sys 0.034s
I20260709 12:12:01.436769 14362 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:01.716317 14479 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:01.898450 12494 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:12:02.119148 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:01.943222 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:01.928514 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 429643us
I20260709 12:12:01.897568 12496 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
W20260709 12:12:01.883090 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 351135us
I20260709 12:12:01.927816 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:01.986794 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:02.130759 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 313642us
I20260709 12:12:01.498574 14488 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:01.898626 12563 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:02.123241 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:01.902518 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:02.139814 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.010s	sys 0.018s
W20260709 12:12:02.179638 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.011s	sys 0.019s
W20260709 12:12:02.140664 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.000s	sys 0.011s
I20260709 12:12:01.887029 13262 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 4 LEARNER]: Advancing to term 5
I20260709 12:12:01.826026 14498 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:02.131425 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 637875us
W20260709 12:12:02.130105 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.008s	sys 0.038s
I20260709 12:12:02.174536 12494 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:02.175701 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:02.137651 14305 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:02.180120 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 814281us
I20260709 12:12:02.131289 14383 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:02.075397 14495 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.815s	user 0.008s	sys 0.004s
I20260709 12:12:02.131527 14473 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:02.215029 12496 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:02.218783 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.253314 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.064891 14404 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:12:02.190773 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67686.3us
I20260709 12:12:02.274857 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:02.200511 12495 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:12:01.988026 14462 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:02.079996 14469 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:12:01.958505 14353 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:02.201586 14454 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.008s	sys 0.010s
I20260709 12:12:02.199194 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:02.180864 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.001s	sys 0.013s
I20260709 12:12:01.945142 14061 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:01.903779 14386 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:01.901463 14478 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:02.276883 14404 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:02.307459 14454 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.009s	sys 0.010s
I20260709 12:12:02.283062 14496 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:02.196542 14488 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:02.215541 12494 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:02.264854 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.239213 12565 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:02.295020 14469 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:02.287972 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.369282 12637 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:02.370651 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:01.883193 14453 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:02.011511 14359 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.302779 12497 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:12:02.191277 14479 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:02.307862 12641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.006s	sys 0.038s
I20260709 12:12:02.220340 14332 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:02.204633 14466 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:02.221096 14473 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:02.387116 14473 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:02.338382 14452 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:02.351758 14386 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.318787 13998 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:02.394800 14386 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.394868 14404 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:02.396226 14505 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.216s	user 0.000s	sys 0.000s
I20260709 12:12:02.338277 14509 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260709 12:12:02.326140 14358 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:02.211025 14503 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:01.882422 14355 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:02.338265 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:02.278262 14462 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:02.205681 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:02.350443 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44636.8us
W20260709 12:12:02.351311 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 177916us
W20260709 12:12:02.269325 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65093.9us
I20260709 12:12:02.295439 14353 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:02.205897 14515 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.164s
I20260709 12:12:02.211555 14305 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.210932 14403 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.139s
I20260709 12:12:02.378733 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:02.379190 12641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.006s	sys 0.038s
I20260709 12:12:02.266115 14383 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.265304 14495 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:02.383736 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 184690us
I20260709 12:12:02.277136 14482 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.091s
W20260709 12:12:02.398329 14514 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:12:02.424131 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:02.424178 12640 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:02.422354 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:02.422633 14462 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:02.452729 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:02.424589 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 219014us
I20260709 12:12:02.427781 14408 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:02.362753 14478 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:02.415612 14355 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:02.467296 14383 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.432463 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1518) took 1174 ms. Trace:
I20260709 12:12:02.374325 14061 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.468055 12571 rpcz_store.cc:276] 0709 12:12:01.258444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.263413 (+  4969us) service_pool.cc:224] Handling call
0709 12:12:01.263692 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:01.263904 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:01.503939 (+240035us) log.cc:844] Serialized 14 byte log entry
0709 12:12:02.431910 (+927971us) log.cc:844] Serialized 14 byte log entry
0709 12:12:02.432115 (+   205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:02.432166 (+    51us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:02.432217 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.432285 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.432400 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46194,"prepare.queue_time_us":514381,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":118,"raft.queue_time_us":32721,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":22855,"thread_start_us":1120692,"threads_started":3,"wal-append.queue_time_us":897952}
I20260709 12:12:02.469410 14061 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.474377 14355 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:02.476194 12461 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.581s (SENT)
I20260709 12:12:02.476282 14305 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.476747 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:02.477890 12646 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:02.479516 14355 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:02.480005 13955 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.585s	user 0.008s	sys 0.002s
I20260709 12:12:02.480482 13955 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.482577 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 158) took 1083 ms. Trace:
I20260709 12:12:02.483096 12490 rpcz_store.cc:276] 0709 12:12:01.398848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.459620 (+ 60772us) service_pool.cc:224] Handling call
0709 12:12:01.507636 (+ 48016us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:01.853258 (+345622us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:01.853294 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:01.853815 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:12:02.314798 (+460983us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:02.314850 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.314941 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:02.482289 (+167348us) raft_consensus.cc:1685] finished
0709 12:12:02.482314 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.482501 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":507420,"raft.queue_time_us":346258,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":30562,"thread_start_us":371476,"threads_started":1,"wal-append.queue_time_us":537272}
I20260709 12:12:02.484531 14061 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.484886 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:02.486150 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 379) took 1175 ms. Trace:
I20260709 12:12:02.486292 14479 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.486685 13254 rpcz_store.cc:276] 0709 12:12:01.311182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.332557 (+ 21375us) service_pool.cc:224] Handling call
0709 12:12:02.484696 (+1152139us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:02.486061 (+  1365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92733184}
I20260709 12:12:02.432467 12645 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:02.396307 14404 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:02.490379 14482 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:02.490769 14453 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:02.492355 14488 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:02.405717 14358 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:02.415926 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 384) took 1058 ms. Trace:
I20260709 12:12:02.449681 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 152) took 1185 ms. Trace:
I20260709 12:12:02.416265 14456 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:02.435076 14332 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:02.429491 14329 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.455061 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:02.450886 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:02.456306 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:02.485186 14061 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:12:02.436965 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 229836us
I20260709 12:12:02.404379 14359 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:02.416716 14503 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:02.586748 12461 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.675s (SENT)
I20260709 12:12:02.494712 12638 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:02.570734 14353 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.599809 14353 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:02.600557 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:02.602242 14386 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:12:02.602550 14523 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:02.603068 14386 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:02.603222 14523 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:02.603735 14523 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:02.604239 14386 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:02.494552 12642 rpcz_store.cc:276] 0709 12:12:01.264256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.265077 (+   821us) service_pool.cc:224] Handling call
0709 12:12:01.265512 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:01.265804 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:01.265893 (+    89us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:01.266497 (+   604us) log.cc:844] Serialized 26 byte log entry
0709 12:12:01.569022 (+302525us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:02.013411 (+444389us) log.cc:844] Serialized 14 byte log entry
0709 12:12:02.172993 (+159582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.173074 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:02.449401 (+276327us) raft_consensus.cc:1685] finished
0709 12:12:02.449425 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.449588 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":159251,"prepare.queue_time_us":530700,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":159600,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":8401,"thread_start_us":559292,"threads_started":2,"wal-append.queue_time_us":830642}
W20260709 12:12:02.605772 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 149523us
I20260709 12:12:02.576687 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 156) took 1178 ms. Trace:
I20260709 12:12:02.602198 12461 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3.
I20260709 12:12:02.609257 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.610005 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:02.610416 14305 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.540307 14488 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.611557 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.611039 14305 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.612298 12563 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:02.523590 13262 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 5 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:02.613315 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 441) took 1116 ms. Trace:
W20260709 12:12:02.520129 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97149.6us
I20260709 12:12:02.613476 14463 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:02.610078 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1419) took 1127 ms. Trace:
I20260709 12:12:02.614724 14470 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:02.615187 14470 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:02.615229 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.616551 14359 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.610936 12568 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:02.557339 14522 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
I20260709 12:12:02.618032 12568 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:02.614724 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.614789 12640 rpcz_store.cc:276] 0709 12:12:01.483077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.871333 (+388256us) service_pool.cc:224] Handling call
0709 12:12:02.609904 (+738571us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:02.610010 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241912704}
I20260709 12:12:02.614576 14463 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:02.619053 12568 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.619867 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:02.620193 14463 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.616127 14470 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.621482 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:02.620204 12568 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:02.612128 14488 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.622691 13262 rpcz_store.cc:276] 0709 12:12:01.497420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.886476 (+389056us) service_pool.cc:224] Handling call
0709 12:12:01.886860 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:02.613030 (+726170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.613255 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:02.623728 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3011873580001065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260709 12:12:02.625362 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170366us
I20260709 12:12:02.579973 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 222) took 1279 ms. Trace:
I20260709 12:12:02.534561 12637 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:02.628391 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 177558us
I20260709 12:12:02.493929 13260 rpcz_store.cc:276] 0709 12:12:01.357615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.358106 (+   491us) service_pool.cc:224] Handling call
0709 12:12:02.338032 (+979926us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:02.415850 (+ 77818us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159628160}
I20260709 12:12:02.541496 12645 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:02.497155 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1672) took 1421 ms. Trace:
I20260709 12:12:02.608036 12493 rpcz_store.cc:276] 0709 12:12:01.398265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.409752 (+ 11487us) service_pool.cc:224] Handling call
0709 12:12:02.345587 (+935835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:02.345886 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.345925 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:02.345987 (+    62us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:02.346043 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.346122 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.576402 (+230280us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b540012b1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:02.576593 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2151343488}
I20260709 12:12:02.638830 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.628409 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:02.699193 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:02.688275 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.605s (SENT) [suppressed 13 similar messages]
I20260709 12:12:02.703066 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:02.703979 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:02.706470 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.458590 14351 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:12:02.699680 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65220.1us
W20260709 12:12:02.691668 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:02.691541 12536 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:02.717729 12536 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:12:02.708968 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:02.586093 14526 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:02.693312 14487 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:02.496469 14357 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:12:02.715628 14320 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.458s	user 0.013s	sys 0.001s
I20260709 12:12:02.689224 12566 rpcz_store.cc:276] 0709 12:12:01.076588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.076739 (+   151us) service_pool.cc:224] Handling call
0709 12:12:01.077046 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:01.098661 (+ 21615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:01.098690 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:01.108215 (+  9525us) log.cc:844] Serialized 26 byte log entry
0709 12:12:01.424467 (+316252us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b1800099ac0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:12:01.856842 (+432375us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:01.856908 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:01.857021 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:02.463027 (+606006us) raft_consensus.cc:1685] finished
0709 12:12:02.463046 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.463267 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":459,"raft.queue_time_us":21290,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":12950,"spinlock_wait_cycles":3426176,"thread_start_us":282765,"threads_started":1,"wal-append.queue_time_us":953079}
I20260709 12:12:02.689146 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.690811 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.692392 12564 rpcz_store.cc:276] 0709 12:12:01.300958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.329016 (+ 28058us) service_pool.cc:224] Handling call
0709 12:12:01.428964 (+ 99948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:01.437846 (+  8882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.579704 (+1141858us) spinlock_profiling.cc:243] Waited 99.6 ms on lock 0x7b54003ab450. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:02.579890 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":229047552}
W20260709 12:12:02.726744 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98501us
I20260709 12:12:02.697175 14377 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:02.695820 14359 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:12:02.691780 14329 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.633354 14332 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:02.775728 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.790627 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:02.787415 12572 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:02.632016 14439 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.710166 14466 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260709 12:12:02.797911 14439 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.736454 13998 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:02.800422 14439 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.457578 14530 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.321s
I20260709 12:12:02.774655 14260 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.806137 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:02.807034 12495 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:02.807469 12488 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:02.808059 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:02.809007 14524 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:12:02.810408 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1417) took 1421 ms. Trace:
I20260709 12:12:02.810897 12639 rpcz_store.cc:276] 0709 12:12:01.388987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.893259 (+504272us) service_pool.cc:224] Handling call
0709 12:12:01.898568 (+  5309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:01.899339 (+   771us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:01.899374 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:01.900034 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 12:12:02.213986 (+313952us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:02.262039 (+ 48053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.290070 (+ 28031us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:02.810091 (+520021us) raft_consensus.cc:1685] finished
0709 12:12:02.810117 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.810251 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":282000,"threads_started":1,"wal-append.queue_time_us":624853}
I20260709 12:12:02.736305 13253 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 1 LEARNER]: Advancing to term 3
I20260709 12:12:02.522137 14478 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.804898 14260 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.815673 14478 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.700867 14515 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:12:02.816255 14260 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:02.735759 14305 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.780596 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:02.595031 14534 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.133s
I20260709 12:12:02.818866 14530 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:02.819332 14530 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:02.820281 14530 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:02.825233 14404 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:02.829200 14495 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:02.791885 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:02.737237 14061 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:02.775418 14496 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.737155 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:02.795495 14516 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:12:02.796015 12597 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:12:02.459023 14469 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.810200 14488 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:02.702703 14447 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.070s
I20260709 12:12:02.884938 12494 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:02.853787 14428 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.831521 14487 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:02.687781 14456 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.741142 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:02.841886 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:02.889268 12495 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:02.847568 12497 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:02.854768 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:02.773527 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1414) took 1410 ms. Trace:
I20260709 12:12:02.792744 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:02.861667 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:02.842743 14530 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:02.737067 14453 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.747643 14357 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:02.793455 12640 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:02.751616 14366 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:02.868942 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:02.747809 14498 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.872655 12568 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260709 12:12:02.915499 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54275.7us
I20260709 12:12:02.915851 12568 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:12:02.916128 12568 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:12:02.916481 12568 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:02.916523 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:02.796342 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:02.917811 14495 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:02.918123 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 385) took 1557 ms. Trace:
I20260709 12:12:02.918437 14495 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.918628 14454 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:02.918998 14454 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:02.918941 14495 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.919443 14454 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:02.921146 14495 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:02.918465 13257 rpcz_store.cc:276] 0709 12:12:01.360458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.414838 (+ 54380us) service_pool.cc:224] Handling call
0709 12:12:02.577254 (+1162416us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:02.918062 (+340808us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24243072}
W20260709 12:12:02.845350 14509 consensus_queue.cc:1175] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 5 status { last_received { term: 5 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 5 index: 7 } }
I20260709 12:12:02.877506 12488 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:12:02.835870 14332 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:02.924990 14526 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:12:02.592559 14529 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:12:02.934705 14477 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.935495 14477 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.936215 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1428) took 1062 ms. Trace:
I20260709 12:12:02.935992 14523 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.937260 12489 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:12:02.936599 12646 rpcz_store.cc:276] 0709 12:12:01.874069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.478725 (+604656us) service_pool.cc:224] Handling call
0709 12:12:02.479078 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:02.479494 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.479526 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:02.480061 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:12:02.722587 (+242526us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:02.722630 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.722718 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:02.935980 (+213262us) raft_consensus.cc:1685] finished
0709 12:12:02.935998 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.936141 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3002,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":242089,"threads_started":1,"wal-append.queue_time_us":386737}
I20260709 12:12:02.868398 13998 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:12:02.937969 13253 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 3 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:02.938418 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.855927 14132 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 12:12:02.940702 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:02.941644 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:02.892421 12643 rpcz_store.cc:276] 0709 12:12:01.362711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.461356 (+ 98645us) service_pool.cc:224] Handling call
0709 12:12:01.461767 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:01.462197 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:01.462255 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:01.497880 (+ 35625us) log.cc:844] Serialized 26 byte log entry
0709 12:12:02.208440 (+710560us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:02.208478 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.208557 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:02.773263 (+564706us) raft_consensus.cc:1685] finished
0709 12:12:02.773294 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:02.773461 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":138017,"raft.queue_time_us":1209,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":43032,"thread_start_us":426920,"threads_started":1,"wal-append.queue_time_us":765517}
W20260709 12:12:02.896100 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155111us
W20260709 12:12:02.897219 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104604us
I20260709 12:12:02.944463 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:02.787815 14377 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.954187 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:02.902887 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:02.870834 14320 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:02.877182 14420 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:02.894697 13955 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:02.971262 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.971927 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:02.972465 12568 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:02.972764 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:02.975939 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:02.719018 14358 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:02.961388 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:02.964452 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 276191us
I20260709 12:12:03.134667 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:02.952997 14487 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.914835 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:03.155654 12642 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:02.970783 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:03.156697 12488 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:02.911906 14357 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:02.912249 14061 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:12:02.957423 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:02.885690 14362 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 12:12:02.977998 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:03.185971 12566 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:02.979241 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:02.976599 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:03.187377 12564 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:02.878330 14496 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:02.966069 13254 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:03.126897 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:03.136374 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:03.219099 12643 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:02.965059 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 332899us
W20260709 12:12:03.148401 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 457000us
I20260709 12:12:03.149520 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:02.964910 14351 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:02.959878 14386 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:02.966253 14260 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:02.899302 14428 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.912436 14453 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.927450 14495 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260709 12:12:03.178493 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:03.265318 12497 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:03.183785 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:03.266670 12563 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:03.188649 12568 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:02.914333 14462 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:03.191143 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 233819us
I20260709 12:12:03.274576 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:03.191778 12646 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:12:03.192273 14355 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:12:03.200557 14408 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:02.940233 14366 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:03.200604 14535 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:03.208716 14539 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.250s	user 0.000s	sys 0.000s
I20260709 12:12:03.183284 14537 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:12:03.210124 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1415) took 1847 ms. Trace:
W20260709 12:12:03.215070 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1682) took 1380 ms (client timeout 1775 ms). Trace:
W20260709 12:12:03.148831 14473 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.761s	user 0.004s	sys 0.009s
I20260709 12:12:03.216444 12569 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:12:03.301486 12569 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:03.301574 12572 rpcz_store.cc:269] 0709 12:12:01.834601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.834899 (+   298us) service_pool.cc:224] Handling call
0709 12:12:03.214784 (+1379885us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b54001b2bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:03.214974 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1416209792}
I20260709 12:12:03.304844 12564 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:03.225262 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:03.307267 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82201.3us
I20260709 12:12:03.308564 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:02.952246 14498 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:03.190366 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:03.315572 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:03.316475 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 126231us
I20260709 12:12:03.150206 14439 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:03.235673 12640 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:03.250952 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 268819us
I20260709 12:12:03.300871 12641 rpcz_store.cc:276] 0709 12:12:01.362985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.460238 (+ 97253us) service_pool.cc:224] Handling call
0709 12:12:01.460727 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:02.432516 (+971789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.432557 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:02.454304 (+ 21747us) log.cc:844] Serialized 26 byte log entry
0709 12:12:02.871023 (+416719us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:02.871069 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:02.871138 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:03.209736 (+338598us) raft_consensus.cc:1685] finished
0709 12:12:03.209766 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:03.210027 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73739,"raft.queue_time_us":253747,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":53636,"thread_start_us":1179536,"threads_started":2,"wal-append.queue_time_us":440544}
I20260709 12:12:03.329828 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:03.257491 12642 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:02.969094 14320 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:02.989176 14463 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.212s
I20260709 12:12:03.195725 14447 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:03.204898 14456 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:03.310037 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:03.355788 12572 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:12:03.320809 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:03.357649 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:03.358568 12568 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:03.317255 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:03.331235 12641 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:03.331667 12642 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:12:03.171737 14487 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.237206 14477 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.268316 14466 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:03.343824 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:03.350286 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:03.351071 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:03.352953 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155324us
I20260709 12:12:03.353451 12566 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:03.303068 14535 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:03.403039 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85911us
I20260709 12:12:03.284060 14353 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:03.327024 14439 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:03.406093 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:03.407438 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:03.405457 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:03.411330 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:03.342963 14132 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:03.334791 14220 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:03.345561 12569 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:03.273201 14260 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.327340 14473 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:03.768574 12569 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:03.570457 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1426) took 1981 ms. Trace:
W20260709 12:12:03.769229 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 363911us
I20260709 12:12:03.666349 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:03.768285 12566 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:03.771963 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:03.769265 12640 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:03.577509 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 379639us
I20260709 12:12:03.773300 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:03.337141 14061 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:03.774209 12568 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:03.571242 12643 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:03.395563 14355 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:03.796417 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:03.768412 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 424615us
W20260709 12:12:03.788094 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:03.781138 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1564) took 1225 ms. Trace:
I20260709 12:12:03.770077 12638 rpcz_store.cc:276] 0709 12:12:01.588869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.877219 (+288350us) service_pool.cc:224] Handling call
0709 12:12:03.570176 (+1692957us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:03.570359 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":709465344}
I20260709 12:12:03.395274 14404 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:03.798805 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:03.800105 12640 rpcz_store.cc:276] 0709 12:12:02.555753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.327549 (+771796us) service_pool.cc:224] Handling call
0709 12:12:03.769028 (+441479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:03.780754 (+ 11726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.781046 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:03.403039 14466 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:03.549679 14547 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 12:12:03.613199 14523 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:03.808197 14248 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:03.830766 14523 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:03.830936 14248 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:03.569546 14552 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:03.832566 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1810) took 1008 ms. Trace:
I20260709 12:12:03.682441 14452 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:03.567235 14462 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.833362 14452 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:03.631877 14540 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:12:03.412052 14535 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:03.681670 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:03.826594 14546 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:12:03.722944 14377 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:03.679996 14550 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:03.566082 13253 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:03.425834 14305 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:03.563225 14366 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.680670 13254 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:12:03.568954 14551 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:12:03.833045 12489 rpcz_store.cc:276] 0709 12:12:02.824019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.839471 (+ 15452us) service_pool.cc:224] Handling call
0709 12:12:02.839948 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:02.937597 (+ 97649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.832266 (+894669us) spinlock_profiling.cc:243] Waited 97.1 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:03.832493 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223272704}
I20260709 12:12:03.838142 12563 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:03.839059 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:03.429030 14353 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:03.754907 14358 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:03.558282 14375 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:03.569373 14554 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:03.439262 14477 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:03.565842 14428 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.791580 14439 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:03.823513 12495 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:03.796083 14488 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:03.844697 12569 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:12:03.845152 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.001s	sys 0.026s
W20260709 12:12:03.892602 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.002s	sys 0.027s
I20260709 12:12:03.413305 14329 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:03.846204 12536 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:03.630141 14495 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:03.782460 14132 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:03.859054 14305 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:03.796651 14483 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:03.798547 14355 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:03.848961 12568 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:03.398851 14447 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:03.854326 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:03.819087 14404 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:03.821528 14466 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:03.511953 14359 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.334s
W20260709 12:12:03.868552 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 430587us
W20260709 12:12:03.883844 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 471309us
W20260709 12:12:03.885568 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 479565us
I20260709 12:12:03.859037 14453 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.802642 14061 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:03.894246 12566 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:03.928773 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:03.885345 14320 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:03.858964 14567 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:03.859815 14463 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:03.861505 14377 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:03.491825 14351 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.870059 14353 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:03.967124 14332 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.043s	user 0.008s	sys 0.004s
I20260709 12:12:03.967614 14332 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:03.943092 14530 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.039s	user 0.007s	sys 0.005s
I20260709 12:12:03.943238 14567 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.970383 14377 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:03.901964 14329 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:03.869127 14375 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:03.869303 14554 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:03.909934 12564 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:03.923025 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:03.908638 14498 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.924110 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:03.924908 13218 connection.cc:256] RPC call timeout handler was delayed by -1.2449968589999116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31610000000000005s for a response
I20260709 12:12:03.929437 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:03.894412 14132 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:03.896263 14488 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:04.015007 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:03.984519 14357 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.817s	user 0.007s	sys 0.006s
I20260709 12:12:03.895797 14220 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:03.935788 14559 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
W20260709 12:12:03.897166 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 378) took 2574 ms (client timeout 2291 ms). Trace:
I20260709 12:12:03.938347 14260 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:03.941850 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1822) took 1014 ms. Trace:
I20260709 12:12:03.901187 14495 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.944150 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1824) took 1001 ms. Trace:
I20260709 12:12:03.973021 14567 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:03.951275 13253 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:12:03.952690 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:03.984546 14447 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.027341 14488 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:03.861570 14550 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.942797 14463 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.957759 14503 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:03.987689 14329 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.069078 14260 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:04.083395 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154075us
I20260709 12:12:04.038702 14357 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.930994 14061 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:03.962348 14452 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:03.974148 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:03.974725 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:04.093152 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:03.832582 14552 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:03.983211 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:03.926630 14404 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:03.987497 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64531us
W20260709 12:12:03.923493 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:04.027774 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:03.885015 14477 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.037068 12488 rpcz_store.cc:276] 0709 12:12:02.927310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.193243 (+265933us) service_pool.cc:224] Handling call
0709 12:12:03.193726 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:03.227193 (+ 33467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:03.227246 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:03.227910 (+   664us) log.cc:844] Serialized 26 byte log entry
0709 12:12:03.426541 (+198631us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:03.426595 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.426687 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:03.941492 (+514805us) raft_consensus.cc:1685] finished
0709 12:12:03.941523 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:03.941769 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32891,"raft.queue_time_us":41304,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":6634,"thread_start_us":147156,"threads_started":1,"wal-append.queue_time_us":594717}
I20260709 12:12:03.900869 14305 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:04.041395 12492 rpcz_store.cc:276] 0709 12:12:02.942528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.265486 (+322958us) service_pool.cc:224] Handling call
0709 12:12:03.266059 (+   573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:03.266612 (+   553us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:03.266660 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:03.283574 (+ 16914us) log.cc:844] Serialized 26 byte log entry
0709 12:12:03.795371 (+511797us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:03.795442 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.795566 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:03.943807 (+148241us) raft_consensus.cc:1685] finished
0709 12:12:03.943843 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:03.944060 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":709,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":20796,"thread_start_us":439758,"threads_started":1,"wal-append.queue_time_us":602458}
I20260709 12:12:04.119656 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:04.121922 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1817) took 1264 ms. Trace:
I20260709 12:12:03.895182 14456 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:04.037400 14377 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:04.136059 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 212018us
I20260709 12:12:04.067798 14473 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:04.037832 13256 rpcz_store.cc:269] 0709 12:12:01.310890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.313319 (+  2429us) service_pool.cc:224] Handling call
0709 12:12:03.681441 (+2368122us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:03.881978 (+200537us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32125824}
I20260709 12:12:04.115059 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:03.943428 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105970us
I20260709 12:12:04.117717 12488 pending_rounds.cc:85] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 4
I20260709 12:12:04.141439 12488 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:12:04.141749 12488 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:03.954329 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1813) took 1129 ms. Trace:
I20260709 12:12:04.122455 12496 rpcz_store.cc:276] 0709 12:12:02.857336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.857794 (+   458us) service_pool.cc:224] Handling call
0709 12:12:02.881096 (+ 23302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:02.881580 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.883189 (+  1609us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:02.900288 (+ 17099us) log.cc:844] Serialized 26 byte log entry
0709 12:12:03.288306 (+388018us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:03.288509 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.288899 (+   390us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:04.121692 (+832793us) raft_consensus.cc:1685] finished
0709 12:12:04.121708 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:04.121852 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":158,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":31275,"thread_start_us":308613,"threads_started":1,"wal-append.queue_time_us":545650}
W20260709 12:12:04.133189 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:04.143718 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:04.141769 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:04.133921 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:04.134038 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:04.131497 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:12:04.086743 14260 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.140966 12566 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:04.056490 14355 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.142946 12493 rpcz_store.cc:276] 0709 12:12:02.825075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.836903 (+ 11828us) service_pool.cc:224] Handling call
0709 12:12:02.837286 (+   383us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:02.869962 (+ 32676us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.870014 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:02.878835 (+  8821us) log.cc:844] Serialized 52 byte log entry
0709 12:12:03.327123 (+448288us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:03.327187 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.953915 (+626728us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:03.953962 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:03.954020 (+    58us) raft_consensus.cc:1685] finished
0709 12:12:03.954035 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:03.954262 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4593,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":7507,"spinlock_wait_cycles":71989632,"thread_start_us":386875,"threads_started":1,"wal-append.queue_time_us":956842}
I20260709 12:12:04.143491 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:03.960891 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1529) took 1747 ms. Trace:
I20260709 12:12:04.138455 12395 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:04.093670 14332 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:04.165869 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:04.156008 12570 rpcz_store.cc:276] 0709 12:12:02.213974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.214169 (+   195us) service_pool.cc:224] Handling call
0709 12:12:02.296274 (+ 82105us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:02.296544 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:02.354542 (+ 57998us) log.cc:844] Serialized 14 byte log entry
0709 12:12:03.946442 (+1591900us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:03.960114 (+ 13672us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b18001edae0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:12:03.960310 (+   196us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:03.960370 (+    60us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:03.960422 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.960512 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:03.960816 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50205,"prepare.queue_time_us":65245,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":81,"raft.queue_time_us":114181,"raft.run_cpu_time_us":3225,"raft.run_wall_time_us":1032878,"spinlock_wait_cycles":44224512,"thread_start_us":608044,"threads_started":2,"wal-append.queue_time_us":963905}
I20260709 12:12:04.179056 14584 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:12:04.183286 14510 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.332s	user 0.010s	sys 0.000s
I20260709 12:12:04.191807 14260 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:04.168979 14377 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:04.201982 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 153) took 2937 ms. Trace:
I20260709 12:12:04.202766 14477 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:04.204741 14357 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.207028 14550 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.209287 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.213310 14359 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:04.202730 12644 rpcz_store.cc:276] 0709 12:12:01.264633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.270559 (+  5926us) service_pool.cc:224] Handling call
0709 12:12:01.270937 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:01.271343 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:01.271368 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:01.292684 (+ 21316us) log.cc:844] Serialized 26 byte log entry
0709 12:12:01.380222 (+ 87538us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:01.382983 (+  2761us) log.cc:844] Serialized 14 byte log entry
0709 12:12:01.691576 (+308593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:01.691721 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:04.142426 (+2450705us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:04.185251 (+ 42825us) raft_consensus.cc:1685] finished
0709 12:12:04.185419 (+   168us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:04.201819 (+ 16400us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":328371,"prepare.queue_time_us":661909,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":67,"raft.queue_time_us":422416,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":51973,"spinlock_wait_cycles":71790848,"thread_start_us":38394,"threads_started":2,"wal-append.queue_time_us":138391}
I20260709 12:12:04.074940 14554 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.220995 14552 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:04.222316 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45407.5us
I20260709 12:12:04.169435 14585 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:04.079504 14220 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.229663 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:04.229925 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88292.9us
I20260709 12:12:04.235359 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:04.236474 14580 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.163s	user 0.000s	sys 0.002s
I20260709 12:12:04.236862 12536 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:04.245463 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:04.162256 14577 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:04.223078 14473 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:12:04.257346 14420 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.288s	user 0.006s	sys 0.008s
I20260709 12:12:04.244412 14552 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:04.271538 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:04.253914 14509 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:04.209405 14353 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.256309 14355 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:04.263154 14359 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:04.232033 14404 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.278965 14570 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:12:04.347179 14570 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:04.347648 14570 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:04.347340 14061 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.349399 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:04.280057 14420 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:04.350980 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1542) took 1661 ms. Trace:
I20260709 12:12:04.351574 12571 rpcz_store.cc:276] 0709 12:12:02.689606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.689798 (+   192us) service_pool.cc:224] Handling call
0709 12:12:02.696280 (+  6482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:02.980249 (+283969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.980394 (+   145us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:03.013174 (+ 32780us) log.cc:844] Serialized 26 byte log entry
0709 12:12:03.893148 (+879974us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:03.893200 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.893279 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:04.350712 (+457433us) raft_consensus.cc:1685] finished
0709 12:12:04.350736 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:04.350910 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":84747,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":68430,"thread_start_us":947617,"threads_started":2,"wal-append.queue_time_us":1092527}
W20260709 12:12:04.285303 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:04.287364 14403 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:04.279533 14447 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:04.358084 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.314807 14329 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:04.192994 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:04.323489 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1815) took 1474 ms. Trace:
I20260709 12:12:04.327644 12395 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index -1 to 2, term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to <none>, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 2 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:04.331048 12572 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: [4.5-4.5]
W20260709 12:12:04.347541 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1820) took 1489 ms (client timeout 1901 ms). Trace:
W20260709 12:12:04.360198 12495 rpcz_store.cc:269] 0709 12:12:02.858585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.902522 (+ 43937us) service_pool.cc:224] Handling call
0709 12:12:04.347297 (+1444775us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:04.347468 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993600896}
I20260709 12:12:04.330708 14535 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.332015 14495 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.333261 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:04.335664 14466 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.267757 14452 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260709 12:12:04.378321 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.388927 13262 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 1.
I20260709 12:12:04.322791 14359 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:04.341439 14487 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.269418 14473 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:04.349213 14570 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:04.356741 14454 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.437s	user 0.014s	sys 0.000s
W20260709 12:12:04.291988 14558 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.437s	user 0.003s	sys 0.003s
I20260709 12:12:04.358927 12494 rpcz_store.cc:276] 0709 12:12:02.848765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.850142 (+  1377us) service_pool.cc:224] Handling call
0709 12:12:02.850964 (+   822us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:02.885582 (+ 34618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.323118 (+1437536us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:04.323395 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77237632}
W20260709 12:12:04.370478 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105358us
I20260709 12:12:04.378636 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.378984 14452 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:12:04.370831 12395 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:04.358425 14403 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:04.380807 12646 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:04.391105 14359 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:04.280315 14585 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:04.277998 14351 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.098s
I20260709 12:12:04.381819 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:04.381723 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.382362 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1561) took 1827 ms. Trace:
I20260709 12:12:04.455657 12643 rpcz_store.cc:276] 0709 12:12:02.554842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.272176 (+717334us) service_pool.cc:224] Handling call
0709 12:12:03.571020 (+298844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:03.800545 (+229525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.382109 (+581564us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54002a49d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:04.382295 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":597500800}
I20260709 12:12:04.384500 14377 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:12:04.384867 14466 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:04.386188 14353 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:04.489977 14473 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.387995 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1536) took 1780 ms. Trace:
I20260709 12:12:04.395659 14132 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.441972 14585 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.496583 12565 rpcz_store.cc:276] 0709 12:12:02.607975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.608125 (+   150us) service_pool.cc:224] Handling call
0709 12:12:02.608493 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:02.953150 (+344657us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:03.334636 (+381486us) log.cc:844] Serialized 14 byte log entry
0709 12:12:03.762953 (+428317us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:03.763039 (+    86us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:03.763127 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.387693 (+624566us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:04.387747 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:04.387917 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10371,"prepare.queue_time_us":799165,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":205177,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":10592,"spinlock_wait_cycles":792036736,"thread_start_us":738339,"threads_started":3,"wal-append.queue_time_us":527270}
I20260709 12:12:04.277215 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.322458 14439 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.500749 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:04.330776 14404 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.396288 12490 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260709 12:12:04.396225 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 276329us
I20260709 12:12:04.504190 14329 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:04.505178 14329 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:04.406462 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:04.505581 14329 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:04.506068 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:04.337163 14463 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:04.411873 12395 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:12:04.442993 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 224) took 16926 ms (client timeout 15000 ms). Trace:
I20260709 12:12:04.506475 14329 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.477037 14509 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:04.477964 13262 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:04.507352 14569 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:04.508777 14329 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:04.510581 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.510941 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:04.507644 12395 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:04.473824 14561 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:04.410952 14487 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:12:04.481837 14305 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.481940 14403 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.482726 13253 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:04.483625 14488 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.406981 14454 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.407414 14452 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:04.485328 14358 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.001s
I20260709 12:12:04.495987 14478 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:04.507392 12489 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:04.474403 14549 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260709 12:12:04.479791 14061 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:04.516188 12395 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 2 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:12:04.576500 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 439445us
I20260709 12:12:04.617488 13261 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:12:04.545929 14404 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:04.618093 12395 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 2 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:04.620285 12395 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:04.618209 14537 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:04.634533 12642 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:12:04.622869 12395 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 2 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:04.645785 12395 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:04.647403 14447 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:04.647904 12395 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:04.614573 14478 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:04.547633 14561 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.675199 14478 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:04.677163 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.008s	sys 0.047s
W20260709 12:12:04.677788 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.008s	sys 0.047s
I20260709 12:12:04.634229 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:04.647902 14447 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:04.649372 14537 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.634717 14404 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:04.828146 14220 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.016s	sys 0.028s
W20260709 12:12:04.871183 14220 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.018s	sys 0.029s
W20260709 12:12:04.881078 12611 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.005s	sys 0.009s
W20260709 12:12:04.881686 12611 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.006s	sys 0.009s
W20260709 12:12:04.882055 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 531761us
I20260709 12:12:04.884410 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:04.819984 14404 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:04.871793 14375 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:04.973382 14561 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:04.943286 14447 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:04.990180 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1828) took 1014 ms. Trace:
I20260709 12:12:04.990741 12490 rpcz_store.cc:276] 0709 12:12:03.976233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.096286 (+120053us) service_pool.cc:224] Handling call
0709 12:12:04.096882 (+   596us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:04.502657 (+405775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.989779 (+487122us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:04.990084 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":643588480}
W20260709 12:12:04.993379 12492 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.008s	sys 0.026s
W20260709 12:12:04.993999 12492 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.008s	sys 0.026s
W20260709 12:12:05.020730 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:04.973421 14454 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:05.224464 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:05.249289 14470 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:05.215229 14447 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:05.217728 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 166) took 1098 ms. Trace:
I20260709 12:12:05.051645 14488 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.160511 14537 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.119289 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1829) took 1142 ms. Trace:
W20260709 12:12:05.223713 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 605678us
I20260709 12:12:05.263399 12489 rpcz_store.cc:276] 0709 12:12:03.976498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.111524 (+135026us) service_pool.cc:224] Handling call
0709 12:12:05.118926 (+1007402us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:05.119177 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":809621760}
I20260709 12:12:05.264672 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:05.264859 14570 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:05.265591 14523 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:05.240978 12566 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:12:05.266554 14523 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:05.267164 14523 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:05.247934 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 237) took 1632 ms. Trace:
I20260709 12:12:05.224040 14061 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:12:05.232331 14535 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.269480 14469 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:05.233860 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.271695 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1544) took 2301 ms. Trace:
W20260709 12:12:05.213348 12609 connection.cc:256] RPC call timeout handler was delayed by -1.0565437059997294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260709 12:12:05.275372 12567 rpcz_store.cc:276] 0709 12:12:02.970331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.970474 (+   143us) service_pool.cc:224] Handling call
0709 12:12:02.970908 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:03.682988 (+712080us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:03.683039 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:03.683103 (+    64us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:03.683138 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:05.271400 (+1588262us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:05.271446 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:05.271603 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1636825344}
I20260709 12:12:05.239923 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.223277 14554 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:05.274974 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1563) took 2719 ms. Trace:
I20260709 12:12:05.280265 12646 rpcz_store.cc:276] 0709 12:12:02.555445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.276906 (+721461us) service_pool.cc:224] Handling call
0709 12:12:03.280991 (+  4085us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:04.409907 (+1128916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:05.274712 (+864805us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:05.274901 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2528764800}
W20260709 12:12:05.211066 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:04.507179 12391 rpcz_store.cc:269] 0709 12:11:47.488247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:47.488532 (+   285us) service_pool.cc:224] Handling call
0709 12:12:04.414391 (+16925859us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:11:58.327623 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:11:58.328459 (+   836us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:11:58.328533 (+    74us) write_op.cc:435] Acquired schema lock
0709 12:11:58.328562 (+    29us) tablet.cc:662] Decoding operations
0709 12:11:58.329116 (+   554us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:11:58.329228 (+   112us) write_op.cc:641] Partition lock acquired for write op
0709 12:11:58.329269 (+    41us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:11:58.329697 (+   428us) tablet.cc:701] Row locks acquired
0709 12:11:58.329762 (+    65us) write_op.cc:260] PREPARE: finished
0709 12:11:58.329979 (+   217us) write_op.cc:270] Start()
0709 12:11:58.330166 (+   187us) write_op.cc:276] Timestamp: P: 1783599118329895 usec, L: 0
0709 12:11:58.330193 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:11:58.331153 (+   960us) log.cc:844] Serialized 541 byte log entry
0709 12:11:59.190352 (+859199us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:12:00.170116 (+979764us) op_driver.cc:464] REPLICATION: finished
0709 12:12:00.455872 (+285756us) write_op.cc:301] APPLY: starting
0709 12:12:00.455958 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:00.456062 (+   104us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:00.456086 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:00.456354 (+   268us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:00.456540 (+   186us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:12:00.456581 (+    41us) write_op.cc:312] APPLY: finished
0709 12:12:00.456791 (+   210us) log.cc:844] Serialized 23 byte log entry
0709 12:12:00.456958 (+   167us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:00.457038 (+    80us) write_op.cc:454] Released schema lock
0709 12:12:00.457080 (+    42us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:00.457501 (+   421us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:00.457527 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4813,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":285465,"apply.run_cpu_time_us":1614,"apply.run_wall_time_us":1615,"num_ops":1,"prepare.queue_time_us":521330,"prepare.run_cpu_time_us":62879,"prepare.run_wall_time_us":1243316,"raft.queue_time_us":460976,"raft.run_cpu_time_us":29834,"raft.run_wall_time_us":247445,"replication_time_us":1839722,"spinlock_wait_cycles":165952896,"thread_start_us":2352373,"threads_started":4,"wal-append.queue_time_us":1251138,"wal-append.run_cpu_time_us":80663,"wal-append.run_wall_time_us":1865769}]]}
I20260709 12:12:05.283216 12570 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:05.260286 14454 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:05.282251 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:05.281514 14567 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:05.282393 14456 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.287959 14351 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:05.277251 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.346266 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:05.268832 12563 rpcz_store.cc:276] 0709 12:12:03.615146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.889856 (+274710us) service_pool.cc:224] Handling call
0709 12:12:03.905185 (+ 15329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:04.078338 (+173153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:04.078384 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:04.082960 (+  4576us) log.cc:844] Serialized 26 byte log entry
0709 12:12:04.264435 (+181475us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:04.264496 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:05.247404 (+982908us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:05.247476 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:05.247538 (+    62us) raft_consensus.cc:1685] finished
0709 12:12:05.247561 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:05.247848 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397394304,"thread_start_us":117558,"threads_started":1,"wal-append.queue_time_us":363839}
I20260709 12:12:05.336885 14496 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.348834 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.350091 12567 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:05.351162 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.291105 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 156) took 2540 ms. Trace:
I20260709 12:12:05.291160 14577 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.351797 14523 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.352626 14577 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.344476 14220 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 12:12:05.354465 14220 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:05.293534 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:05.354630 14523 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.354871 14220 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:05.337143 14532 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.016s	sys 0.024s
I20260709 12:12:05.294148 12571 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:12:05.356328 14532 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.016s	sys 0.024s
I20260709 12:12:05.356438 14220 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.357982 14220 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:05.295890 14592 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.296166 12493 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:05.359004 14456 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:05.359092 14592 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:05.355296 14577 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:05.299218 14582 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.180s	user 0.004s	sys 0.012s
I20260709 12:12:05.361701 14305 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.361882 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 155) took 2612 ms. Trace:
I20260709 12:12:05.300295 14375 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:05.363202 13256 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:12:05.362282 12641 rpcz_store.cc:276] 0709 12:12:02.749163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.360490 (+611327us) service_pool.cc:224] Handling call
0709 12:12:03.360990 (+   500us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:03.369672 (+  8682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:03.369718 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:03.940349 (+570631us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:12:03.940897 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:12:04.040017 (+ 99120us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:04.040079 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.040196 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:05.298441 (+1258245us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:05.361647 (+ 63206us) raft_consensus.cc:1685] finished
0709 12:12:05.361672 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:05.361813 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8089,"raft.queue_time_us":9022,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":31294,"spinlock_wait_cycles":36575232,"thread_start_us":98629,"threads_started":1,"wal-append.queue_time_us":122036}
I20260709 12:12:05.341605 14567 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:05.214366 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 807904us
I20260709 12:12:05.294766 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 239) took 1660 ms. Trace:
W20260709 12:12:05.213960 13262 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.706s	user 0.011s	sys 0.030s
I20260709 12:12:05.334198 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1819) took 2356 ms. Trace:
I20260709 12:12:05.356583 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:05.336802 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:05.349363 14496 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.357388 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.365607 14132 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:05.359778 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:05.369863 14132 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.370092 12493 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:05.370386 14132 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.299530 12495 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:05.372455 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:05.373608 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:05.366241 13262 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:12:05.261663 12492 rpcz_store.cc:276] 0709 12:12:04.119552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.134382 (+ 14830us) service_pool.cc:224] Handling call
0709 12:12:04.134741 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:04.392336 (+257595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:04.392385 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:04.395856 (+  3471us) log.cc:844] Serialized 26 byte log entry
0709 12:12:04.994526 (+598670us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:04.994593 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.994704 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:05.217409 (+222705us) raft_consensus.cc:1685] finished
0709 12:12:05.217437 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:05.217644 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17016,"raft.queue_time_us":120239,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":10762,"thread_start_us":827290,"threads_started":2,"wal-append.queue_time_us":809897}
I20260709 12:12:05.364701 14567 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.374706 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:05.375558 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:05.375955 14567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:05.376580 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:05.358136 14452 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.369306 12571 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:12:05.377928 14452 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:12:05.378734 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.294909 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:05.367623 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:05.380664 12567 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:05.379683 12571 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:05.381037 12490 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:05.361315 14456 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.357774 12642 rpcz_store.cc:276] 0709 12:12:02.750949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.362108 (+611159us) service_pool.cc:224] Handling call
0709 12:12:03.362542 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:04.634980 (+1272438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:05.290824 (+655844us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:05.291036 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2924692736}
I20260709 12:12:05.365510 12569 rpcz_store.cc:276] 0709 12:12:03.634233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.892333 (+258100us) service_pool.cc:224] Handling call
0709 12:12:03.895173 (+  2840us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:03.942015 (+ 46842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:03.942051 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:03.942712 (+   661us) log.cc:844] Serialized 26 byte log entry
0709 12:12:04.053079 (+110367us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:04.053144 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:05.294325 (+1241181us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:05.294381 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:05.294434 (+    53us) raft_consensus.cc:1685] finished
0709 12:12:05.294447 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:05.294696 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11848,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":150,"spinlock_wait_cycles":106649472,"thread_start_us":103852,"threads_started":1,"wal-append.queue_time_us":237328,"wal-append.run_cpu_time_us":2180,"wal-append.run_wall_time_us":1007456}
I20260709 12:12:05.376206 12495 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:12:05.369354 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:05.269078 14478 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:05.393033 12493 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:05.366456 12491 rpcz_store.cc:276] 0709 12:12:02.858347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:02.891515 (+ 33168us) service_pool.cc:224] Handling call
0709 12:12:02.891951 (+   436us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:02.898258 (+  6307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:02.898304 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:02.913402 (+ 15098us) log.cc:844] Serialized 52 byte log entry
0709 12:12:03.203356 (+289954us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b4800080058. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:03.347478 (+144122us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:03.347546 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:03.348885 (+  1339us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:04.291021 (+942136us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:05.214497 (+923476us) raft_consensus.cc:1685] finished
0709 12:12:05.214532 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:05.214774 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":136226,"raft.queue_time_us":7326,"raft.run_cpu_time_us":3038,"raft.run_wall_time_us":292960,"spinlock_wait_cycles":18825472,"thread_start_us":376653,"threads_started":1,"wal-append.queue_time_us":809764}
I20260709 12:12:05.412712 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.422145 12569 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:12:05.431754 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:05.432842 12571 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:05.443297 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:05.444620 12567 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:05.447696 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:05.418795 13253 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.866s	user 0.009s	sys 0.003s
I20260709 12:12:05.417454 14577 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:05.419005 14248 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:05.380064 14574 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.456934 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:05.368417 14305 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:05.368410 14496 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.466727 14596 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260709 12:12:05.466866 14577 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:05.507229 14577 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:05.507593 14596 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:05.508293 14596 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:05.512322 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 270) took 1002 ms. Trace:
I20260709 12:12:05.513156 13256 rpcz_store.cc:276] 0709 12:12:04.510211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.547861 (+ 37650us) service_pool.cc:224] Handling call
0709 12:12:05.511949 (+964088us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:05.512221 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1403186944}
I20260709 12:12:05.480322 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:05.485797 14353 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:12:05.517153 14404 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.001s
I20260709 12:12:05.517408 14353 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:05.518067 14353 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:05.518013 14404 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:05.460587 14386 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:05.522328 14578 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:05.523872 14305 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:05.526103 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:05.487368 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:05.569346 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:05.569902 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:05.529470 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:05.573592 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44264.5us
I20260709 12:12:05.459129 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:05.535727 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:05.588455 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52848.4us
I20260709 12:12:05.536528 14220 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.450639 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:05.560621 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:05.592252 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:05.441885 14456 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.594929 12571 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:05.578270 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:05.435715 14439 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.390362 14351 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.623456 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 236) took 2008 ms. Trace:
I20260709 12:12:05.624914 12572 rpcz_store.cc:276] 0709 12:12:03.614908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:03.807862 (+192954us) service_pool.cc:224] Handling call
0709 12:12:03.830139 (+ 22277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:04.359789 (+529650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:05.623060 (+1263271us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:05.623372 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1151304576}
W20260709 12:12:05.626392 14401 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.255s	user 0.050s	sys 0.062s
I20260709 12:12:05.490048 14496 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:05.587502 14591 raft_consensus.cc:2764] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:12:05.587711 14305 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:05.503998 14592 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.587496 14386 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:05.613247 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 393) took 1216 ms. Trace:
I20260709 12:12:05.627466 14523 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:05.627233 14470 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.671388 14132 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:05.586539 14574 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.638362 14585 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.613695 14456 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.622831 14220 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:05.647351 14401 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s	user 0.050s	sys 0.062s
I20260709 12:12:05.660610 13253 rpcz_store.cc:276] 0709 12:12:04.380843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.381583 (+   740us) service_pool.cc:224] Handling call
0709 12:12:05.596662 (+1215079us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:05.673799 14386 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:05.587189 14578 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:05.657642 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:05.720574 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:05.637420 14596 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.718757 14578 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:05.749356 14359 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.294s	user 0.008s	sys 0.006s
I20260709 12:12:05.694029 14470 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:05.792227 13256 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260709 12:12:05.792276 14570 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.003s
I20260709 12:12:05.794939 14359 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:05.811939 13257 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:05.834363 14456 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.835196 14456 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:05.835765 14456 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.821493 14591 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:05.839130 14591 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:05.842888 14577 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:05.847174 14470 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.851552 14452 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:05.820027 14404 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.855567 14351 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:05.872646 13257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 448) took 1515 ms (client timeout 1859 ms). Trace:
W20260709 12:12:05.874127 13257 rpcz_store.cc:269] 0709 12:12:04.357501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.357765 (+   264us) service_pool.cc:224] Handling call
0709 12:12:05.872314 (+1514549us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:05.872535 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3341195648}
I20260709 12:12:05.869467 14403 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:05.823945 14359 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.870540 14591 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:05.897790 14598 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:12:05.896539 14577 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:05.900942 14220 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:05.929082 14447 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.668s	user 0.007s	sys 0.003s
W20260709 12:12:05.932379 14602 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:12:05.905251 14404 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:05.913611 14424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.252s	user 0.012s	sys 0.007s
W20260709 12:12:05.940752 14424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.279s	user 0.013s	sys 0.007s
I20260709 12:12:05.913233 14456 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:05.937455 14523 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:05.937150 14404 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:05.952296 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.017s	sys 0.033s
W20260709 12:12:05.952873 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.017s	sys 0.033s
W20260709 12:12:05.953292 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 614128us
I20260709 12:12:05.951948 14404 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:05.961539 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.016s	sys 0.039s
I20260709 12:12:05.961510 13257 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
W20260709 12:12:05.965440 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.017s	sys 0.040s
I20260709 12:12:05.965832 13257 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 4
W20260709 12:12:05.965996 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 733072us
I20260709 12:12:05.966331 13257 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260709 12:12:05.966758 13257 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:05.968166 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:05.957273 14567 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.581s	user 0.007s	sys 0.009s
I20260709 12:12:05.958223 14359 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:05.976605 14488 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.021s	sys 0.015s
W20260709 12:12:06.011713 12395 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.015s	sys 0.016s
W20260709 12:12:06.012892 12395 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.017s	sys 0.016s
I20260709 12:12:06.023377 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 163) took 1906 ms. Trace:
I20260709 12:12:06.024273 12488 rpcz_store.cc:276] 0709 12:12:04.116868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:04.117106 (+   238us) service_pool.cc:224] Handling call
0709 12:12:04.117472 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:04.148568 (+ 31096us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:04.148602 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:04.158084 (+  9482us) log.cc:844] Serialized 26 byte log entry
0709 12:12:04.411210 (+253126us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:04.411247 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:04.411312 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:06.022406 (+1611094us) raft_consensus.cc:1685] finished
0709 12:12:06.022502 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:06.023040 (+   538us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":252772,"threads_started":1,"wal-append.queue_time_us":1641174}
W20260709 12:12:06.026237 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:06.029278 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 450843us
W20260709 12:12:06.037046 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:06.037918 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.000s	sys 0.015s
W20260709 12:12:06.038528 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.000s	sys 0.016s
W20260709 12:12:06.038961 12568 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.028s	sys 0.024s
W20260709 12:12:06.039599 12568 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.028s	sys 0.024s
W20260709 12:12:06.041985 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.002s	sys 0.013s
W20260709 12:12:06.042605 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.002s	sys 0.013s
W20260709 12:12:06.043026 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 678211us
W20260709 12:12:06.044809 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:06.046119 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.044806 12565 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20260709 12:12:05.943737 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.007s	sys 0.016s
W20260709 12:12:06.045176 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.006s	sys 0.011s
W20260709 12:12:06.049361 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.006s	sys 0.011s
W20260709 12:12:06.049474 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.008s	sys 0.016s
W20260709 12:12:06.048038 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.005s	sys 0.007s
I20260709 12:12:06.046763 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.046685 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.044400 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:06.059510 14482 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:05.973165 14359 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:06.067580 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:06.063186 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.005s	sys 0.007s
I20260709 12:12:06.047508 12565 pending_rounds.cc:85] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 1
I20260709 12:12:06.074083 12565 pending_rounds.cc:107] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:12:06.074764 12565 raft_consensus.cc:2889] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:06.075349 13262 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:06.076197 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.077557 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:06.078258 14591 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:06.083495 14132 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.084530 14132 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.082749 14359 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.088665 14567 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:06.089028 14132 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.090339 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:05.981110 14488 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.021s	sys 0.016s
I20260709 12:12:06.091248 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:06.091629 13253 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:06.092706 14401 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:06.090276 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.015s	sys 0.012s
I20260709 12:12:06.090492 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.093669 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.093834 14401 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:06.094403 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69243.9us
I20260709 12:12:06.095248 14469 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:06.095592 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:06.096843 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.107352 12563 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:06.110596 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.115227 14574 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.091645 14496 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.117728 14496 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:06.106031 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.890s	user 0.015s	sys 0.012s
I20260709 12:12:06.124893 12566 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:06.130118 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86116.9us
I20260709 12:12:06.133387 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:06.134083 14454 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.719s	user 0.011s	sys 0.005s
I20260709 12:12:06.134490 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.134827 14454 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:06.135509 12564 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.152958 14596 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:06.159153 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.27463e+06us
I20260709 12:12:06.159631 14610 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.162740 14611 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:06.164124 14607 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.166684 14608 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:06.168865 14603 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.170817 12497 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:06.171319 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44797.6us
I20260709 12:12:06.101490 14567 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.159760 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.174182 14469 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:06.174618 12572 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.174885 14469 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:06.175552 14469 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:06.175781 14567 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.179953 12565 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:06.181458 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.181807 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.182140 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.182703 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:06.183307 12563 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:06.184192 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.185595 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:06.186448 14478 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.006s
I20260709 12:12:06.183238 12570 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:06.192067 14305 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.006s
I20260709 12:12:06.191711 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.159207 14619 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:06.193656 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:06.194322 14619 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:06.205260 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:06.209000 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81603.6us
I20260709 12:12:06.204572 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:06.210170 12569 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:06.200456 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.161199 14610 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.230340 12567 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:06.230839 14401 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.231822 14610 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.231707 14401 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:06.165467 14607 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.222463 14404 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.167863 14608 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.170090 14603 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.224421 14578 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:06.225777 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 240306us
I20260709 12:12:06.222857 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:06.193145 14305 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.143661 14454 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 12:12:06.248106 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.249980 14607 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:06.250445 14608 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:06.196048 14619 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.252074 14404 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.252681 14496 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.255836 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:06.260015 14603 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:06.242911 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69833.2us
I20260709 12:12:06.270180 12496 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:06.270812 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.271871 12571 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:06.274226 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.276183 14607 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:06.241813 14132 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:12:06.278469 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1701) took 1045 ms. Trace:
I20260709 12:12:06.279029 12568 rpcz_store.cc:276] 0709 12:12:05.236839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:05.245778 (+  8939us) service_pool.cc:224] Handling call
0709 12:12:05.246348 (+   570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:05.246651 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:06.040446 (+793795us) log.cc:844] Serialized 14 byte log entry
0709 12:12:06.243943 (+203497us) log.cc:844] Serialized 14 byte log entry
0709 12:12:06.256366 (+ 12423us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:06.261924 (+  5558us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:06.266738 (+  4814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:06.278148 (+ 11410us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:06.278382 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68759,"prepare.queue_time_us":732123,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":114,"raft.queue_time_us":78538,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":7631,"thread_start_us":786871,"threads_started":2,"wal-append.queue_time_us":114046}
I20260709 12:12:06.296069 12567 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:06.298581 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50991.8us
I20260709 12:12:06.300151 14496 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:06.307633 14578 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.308897 14624 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:06.309406 14624 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:06.310612 14624 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.312242 12497 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:06.312422 14614 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:06.320478 12563 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:06.322705 12569 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:06.334213 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.335572 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:06.336125 14404 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.336822 14404 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.338160 14570 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:06.339455 14532 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:06.339475 14404 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.340447 14532 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:06.340864 12490 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:06.342340 14532 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.344669 14532 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.347489 14577 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.348383 14577 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.351567 14305 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.356787 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:06.359614 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.360069 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:06.362361 12566 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:06.363337 12572 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:06.372429 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57338.7us
W20260709 12:12:06.374353 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 151563us
I20260709 12:12:06.376580 12564 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:06.312305 14624 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:06.374266 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:06.377728 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64029.4us
I20260709 12:12:06.360699 14577 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:06.381783 14353 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.728s	user 0.013s	sys 0.014s
I20260709 12:12:06.382383 14353 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.382902 14625 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:06.385069 14353 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.339564 14614 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:06.388170 12488 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 3. Ops: []
I20260709 12:12:06.389142 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:06.351646 12536 connection.cc:256] RPC call timeout handler was delayed by -3.8377797840000767s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2209s for a response
I20260709 12:12:06.397228 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.404999 12565 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:06.388640 14404 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:06.390133 14614 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:06.406781 14567 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:06.425841 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66394.1us
I20260709 12:12:06.423729 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.433074 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:06.435181 14386 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.667s	user 0.019s	sys 0.003s
I20260709 12:12:06.435513 14614 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.436524 14537 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:06.437033 14537 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:06.437415 14537 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.445957 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.450500 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:06.452473 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:06.425268 14607 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:12:06.455768 14496 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:06.457767 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.435863 14386 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.424285 14404 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:06.425597 14619 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:06.466524 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.437439 14614 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.480026 14567 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.483736 14386 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.484381 14567 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.486343 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.487392 12496 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
W20260709 12:12:06.488819 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.490715 12497 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:12:06.485248 14454 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:06.494094 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.477851 14633 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.477878 14132 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:06.497798 13261 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:06.499276 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.500310 12493 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:06.501717 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.503904 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.505048 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.506049 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.506594 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.506889 12563 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:12:06.507028 12567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:06.507951 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:06.506099 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.506228 14633 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.510468 12568 raft_consensus.cc:2376] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3. Re-sending same reply.
I20260709 12:12:06.512084 14633 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:06.515019 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.516353 14386 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:06.517078 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:06.491464 14496 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:06.521251 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68948.5us
I20260709 12:12:06.466485 14469 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.533350 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:06.534859 14132 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:06.537489 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:06.537882 14629 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:12:06.540105 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.540599 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.541311 12497 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:12:06.494302 14454 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:06.542824 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.552150 12493 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
W20260709 12:12:06.556416 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46794us
I20260709 12:12:06.562376 14610 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:06.565544 14596 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:06.566248 12489 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:06.566378 14596 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:06.567101 14596 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.552186 14454 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:06.497869 14619 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.576180 14619 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.577603 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:06.577497 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:06.582624 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48504.3us
I20260709 12:12:06.582518 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.585302 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:06.586314 14401 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:06.587141 14401 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:06.587709 14132 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.588371 14401 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.485324 14404 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.580629 14496 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:06.590024 12491 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:06.596706 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 190702us
W20260709 12:12:06.597668 12536 connection.cc:256] RPC call timeout handler was delayed by -3.4535989050000353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260709 12:12:06.598058 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3110150400000293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
I20260709 12:12:06.580670 14469 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:06.598407 12536 connection.cc:256] RPC call timeout handler was delayed by -3.0106797840001036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3128s for a response
W20260709 12:12:06.598368 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 204491us
W20260709 12:12:06.599660 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5622150400001829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29350000000000004s for a response
I20260709 12:12:06.565423 14537 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.601230 14635 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:06.602617 14635 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:06.602979 14635 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:06.603479 14537 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.604017 14635 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.597337 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.607493 14353 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:06.616998 14473 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.617408 14624 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:06.617777 14424 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:06.618024 14473 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.618327 14624 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.619535 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:06.620968 14473 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.626444 14496 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.622462 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.624503 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.626356 14220 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:06.624909 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.648138 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.623836 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:06.628144 14603 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:06.625762 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.620987 14624 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.645917 14635 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:06.643321 14220 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:06.658303 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:06.650421 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:06.659894 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:06.658295 14532 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:06.683946 14633 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:12:06.704473 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60474.2us
I20260709 12:12:06.686105 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.690232 13257 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:06.694288 13256 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:06.681015 14496 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.618417 14424 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.671126 14469 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:06.695199 14641 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.695627 14577 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:06.694289 14404 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:06.697914 14603 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.698942 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.678030 14220 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:06.701102 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:06.726967 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:06.732829 12566 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.726967 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.709575 14532 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:06.731211 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.731987 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:06.739144 12563 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:06.715152 14633 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:06.704937 14132 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:12:06.729013 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:06.740614 14353 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:06.766114 14353 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:06.766820 14353 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:06.752023 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53217.6us
I20260709 12:12:06.770202 14424 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.770514 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:06.773772 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160591us
I20260709 12:12:06.731302 14608 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:12:06.727298 14619 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.756376 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:06.736089 14532 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.776369 12642 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:12:06.774780 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:06.778762 12640 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:12:06.749482 14469 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.774667 14403 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:06.785374 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.744827 14567 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:06.780743 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:06.777926 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:06.807713 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88841.1us
I20260709 12:12:06.777801 14386 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:06.784296 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.605s (SENT)
I20260709 12:12:06.804586 14403 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:06.816725 12643 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:06.779397 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.738334 14641 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.778170 14473 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:06.778781 14577 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.802820 14404 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:06.834453 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:06.846048 12644 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:06.847862 14596 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.849604 12646 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 1. Current term is 2. Ops: []
I20260709 12:12:06.828469 14578 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:06.850320 12494 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
W20260709 12:12:06.851687 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109091us
W20260709 12:12:06.852298 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:06.854223 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79574.4us
I20260709 12:12:06.830200 14403 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:06.861678 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:06.828433 14482 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:06.780651 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:06.865320 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 245947us
I20260709 12:12:06.870330 14633 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.836476 14619 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.877971 14554 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:12:06.877848 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:06.880296 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:06.901160 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.834196 14473 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:06.891361 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.899044 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.835070 14386 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.907259 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.838363 14567 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.838028 14469 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:06.922102 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82721.5us
I20260709 12:12:06.911479 14619 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.911096 14473 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:06.912936 12637 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:06.915416 14496 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:06.916148 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:06.918934 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.911531 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.906281 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.924124 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:06.908954 12643 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:06.926192 12639 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:12:06.927908 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46718us
I20260709 12:12:06.930711 12566 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
W20260709 12:12:06.933447 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.839061 14454 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:06.928969 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 144735us
W20260709 12:12:06.926810 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44192us
I20260709 12:12:06.851853 14577 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.931797 12644 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:06.944500 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.949268 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:06.952131 12646 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:06.954198 12640 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:06.955667 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:06.926468 14375 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:06.943624 14607 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:12:06.958341 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:06.958722 12642 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:12:06.959308 14649 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:12:06.959621 12637 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:06.965591 12641 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
W20260709 12:12:06.971441 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65186.6us
W20260709 12:12:06.974148 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:06.976464 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.951895 14482 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:12:06.979928 13218 leader_election.cc:336] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.652s (SENT)
I20260709 12:12:06.956476 14404 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:12:06.971060 14642 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:12:06.966431 14375 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:06.961012 14424 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:06.971297 14537 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:06.972049 14596 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.993695 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:06.994138 14596 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:06.994590 13258 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:06.995100 14577 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.994831 14596 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:06.995690 14577 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:06.996089 14577 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:06.996443 14577 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:06.997116 14305 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:06.953059 14624 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:12:06.976307 14578 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:06.993208 14496 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:07.022604 14454 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:07.001917 14424 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.028159 14424 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:06.990715 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:06.998435 14625 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:07.004765 14386 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:07.005219 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115732us
I20260709 12:12:06.987569 14375 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:06.990800 14473 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:07.014060 14469 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:07.014946 14641 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:07.015478 14537 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:07.016588 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:06.997915 14305 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:07.017701 14567 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:06.986521 14603 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:07.068382 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.069962 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.070050 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.070730 12637 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:07.070927 12644 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:07.071640 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.071789 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.072254 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:07.072702 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.073971 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.075713 12646 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:07.077240 12643 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:07.018836 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:07.020841 14578 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:07.022214 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48218.8us
I20260709 12:12:06.997296 14596 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:07.039420 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87004.3us
I20260709 12:12:06.990886 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.040992 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.029563 12638 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:07.013001 14353 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.057204 12566 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:07.061245 14643 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:07.062248 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:07.097687 14447 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:07.098381 14567 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:07.050976 14625 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:07.105649 14625 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.081983 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:07.110013 14454 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:07.110646 14454 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:07.111163 14454 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:07.114269 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62563.4us
I20260709 12:12:07.115624 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:07.117972 14469 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:07.118600 14469 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:07.119127 14469 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:07.002205 14220 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.087858 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:07.089479 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71154.5us
I20260709 12:12:07.117522 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:07.083947 14482 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:07.136260 14403 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.141701 14644 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:07.144069 12643 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:07.144786 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.150770 14532 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.154821 14403 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:07.155980 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.154397 14482 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.158690 14657 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:07.159293 14657 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.160000 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.160732 12641 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.130601 14607 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:07.124403 14643 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:07.163193 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.162457 14607 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:07.161728 14657 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:07.164615 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.164299 14643 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.161131 14375 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.165014 14657 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:07.139351 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:07.165354 12646 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:07.165380 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.162844 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:07.211068 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85044.6us
W20260709 12:12:07.211520 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94148.1us
I20260709 12:12:07.215137 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.211154 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:07.216243 12638 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:07.142735 14641 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:07.166406 14305 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:07.216713 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 261103us
W20260709 12:12:07.223569 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136700us
I20260709 12:12:07.231552 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:07.218257 14659 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.217s	user 0.004s	sys 0.001s
I20260709 12:12:07.235935 12641 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:07.169167 14220 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:07.221956 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76645.2us
I20260709 12:12:07.182067 14353 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:07.239281 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:07.245195 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.247447 12491 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:07.247689 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.248595 12494 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:07.255209 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.239055 14403 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:07.262710 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.264385 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:07.166908 14537 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.250329 14447 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:12:07.282971 14447 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:07.283569 14447 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:07.270849 13260 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:12:07.274065 14567 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:07.287261 14567 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:07.287735 14567 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:07.289916 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.278033 14635 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:12:07.290412 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.291493 14635 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:07.292287 14635 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:07.293889 14403 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:07.295047 14656 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:12:07.295238 14403 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.164623 14607 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:07.279472 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128719us
I20260709 12:12:07.298070 12395 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 219) took 10030 ms. Trace:
I20260709 12:12:07.260196 14663 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:07.302353 14663 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:07.303071 14663 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.304358 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.305193 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:07.259598 14496 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:07.267527 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:07.288973 14567 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.310693 14404 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:07.310987 14567 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.311904 14577 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.266471 14643 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:07.313879 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:07.314806 13253 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:07.318025 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93064.7us
I20260709 12:12:07.319516 14666 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:07.320036 14666 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:07.320375 14666 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:07.320773 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:07.253810 14641 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.324266 14641 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:07.327867 12497 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:12:07.328652 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:07.330160 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.337046 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:07.337750 12564 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.336383 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:07.298523 12395 rpcz_store.cc:276] 0709 12:11:57.267983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:57.268719 (+   736us) service_pool.cc:224] Handling call
0709 12:12:07.297885 (+10029166us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:05.983105 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:05.983611 (+   506us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:05.983636 (+    25us) write_op.cc:435] Acquired schema lock
0709 12:12:05.983652 (+    16us) tablet.cc:662] Decoding operations
0709 12:12:06.082125 (+ 98473us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:06.082304 (+   179us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:06.082386 (+    82us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:12:06.083989 (+  1603us) tablet.cc:701] Row locks acquired
0709 12:12:06.084054 (+    65us) write_op.cc:260] PREPARE: finished
0709 12:12:06.084431 (+   377us) write_op.cc:270] Start()
0709 12:12:06.084653 (+   222us) write_op.cc:276] Timestamp: P: 1783599126084301 usec, L: 0
0709 12:12:06.084700 (+    47us) op_driver.cc:348] REPLICATION: starting
0709 12:12:06.122258 (+ 37558us) log.cc:844] Serialized 4431 byte log entry
0709 12:12:06.609278 (+487020us) op_driver.cc:464] REPLICATION: finished
0709 12:12:06.845226 (+235948us) write_op.cc:301] APPLY: starting
0709 12:12:06.845379 (+   153us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:06.845761 (+   382us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:06.845935 (+   174us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:06.847699 (+  1764us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:06.848135 (+   436us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 12:12:06.848185 (+    50us) write_op.cc:312] APPLY: finished
0709 12:12:06.848631 (+   446us) log.cc:844] Serialized 83 byte log entry
0709 12:12:06.848839 (+   208us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:06.848985 (+   146us) write_op.cc:454] Released schema lock
0709 12:12:06.849063 (+    78us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:06.870378 (+ 21315us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f032a 00007ffbf5775f14 00007ffbf5775ec3 00007ffbf5775e38 00007ffbf5775df3 00007ffbf57750dc 00007ffbf579ee16 00007ffbf5786cbb 00007ffbf577139d 00007ffbf5774d75
0709 12:12:06.878735 (+  8357us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:06.878775 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3830686,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":194070,"apply.run_cpu_time_us":4441,"apply.run_wall_time_us":33501,"num_ops":11,"prepare.queue_time_us":679978,"prepare.run_cpu_time_us":10318,"prepare.run_wall_time_us":177090,"raft.queue_time_us":113807,"raft.run_cpu_time_us":23169,"raft.run_wall_time_us":315171,"replication_time_us":505726,"spinlock_wait_cycles":2968448,"thread_start_us":860308,"threads_started":4,"wal-append.queue_time_us":172485}]]}
I20260709 12:12:07.284871 14447 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.790318 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.530056 12646 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:07.366405 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:07.366135 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.408245 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:07.716372 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:07.716415 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57217.6us
I20260709 12:12:07.397354 12641 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:07.809006 12642 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.805336 12644 raft_consensus.cc:2376] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4. Re-sending same reply.
W20260709 12:12:07.806680 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 547415us
W20260709 12:12:07.398377 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62106.5us
I20260709 12:12:07.396306 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.360741 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:07.808830 12568 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
W20260709 12:12:07.455891 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127317us
I20260709 12:12:07.603322 12564 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:07.363395 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.815392 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:07.821456 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.824414 12638 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3.
I20260709 12:12:07.825165 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.826573 12646 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
W20260709 12:12:07.829846 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.831663 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:07.515312 14386 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:07.836872 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.836968 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.516943 14633 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:07.359789 14496 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:07.378712 14532 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:07.856235 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:07.857158 12644 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.399873 14644 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:07.410382 14578 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:07.758145 14375 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:07.848153 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:07.852900 12641 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:07.407402 14570 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:07.532827 14619 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:07.649318 14537 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:07.839565 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:07.858485 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.858734 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.887727 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:07.798123 14663 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.862244 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.853152 14633 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:07.856690 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.529080 12392 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:07.874311 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.900012 12496 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:12:07.538178 14404 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:07.535130 14454 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:07.836778 14386 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:07.853343 14403 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:07.925243 12493 raft_consensus.cc:2368] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:07.336272 14661 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.522s
I20260709 12:12:07.887859 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:07.850122 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.142s (SENT) [suppressed 16 similar messages]
I20260709 12:12:07.893744 12646 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:07.905733 14643 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.592s	user 0.008s	sys 0.005s
I20260709 12:12:07.325783 14635 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:07.864307 14447 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:07.896224 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:07.900777 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:07.883270 14469 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.764s	user 0.013s	sys 0.001s
I20260709 12:12:07.899339 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:07.910972 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81802.7us
I20260709 12:12:07.888557 12494 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:07.972172 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156975us
I20260709 12:12:07.939178 14401 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:07.942466 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:07.945694 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.950295 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:07.958253 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:07.895164 14619 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:07.336539 14353 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.502s
I20260709 12:12:07.961539 12641 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:07.924965 14386 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:08.046022 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82518.9us
I20260709 12:12:07.891166 14537 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.043843 14661 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:12:08.043362 14607 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:07.919624 14404 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.162861 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:08.042721 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 192611us
I20260709 12:12:08.030821 14633 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.049652 12495 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:08.049721 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:07.907424 14578 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:08.165172 12492 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:12:08.066221 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:08.169333 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.108445 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.104014 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:08.111485 12568 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:08.170679 14567 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.859s	user 0.008s	sys 0.004s
I20260709 12:12:08.171381 14567 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.172000 14567 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:08.172868 14668 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:12:08.112673 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.048199 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.173975 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.174446 12639 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:08.175197 14454 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:08.115722 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.175706 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.176030 12566 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:08.176626 12641 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:08.177101 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1468) took 1004 ms. Trace:
I20260709 12:12:08.126832 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.116393 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.116650 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:08.127055 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.127281 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:08.045059 14469 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.061942 14403 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:08.232250 12563 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
W20260709 12:12:08.154222 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.128758 14619 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.157341 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:08.127060 14401 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.158746 14607 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.161141 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.116111 14496 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.047657 12392 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.237694 12572 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:08.181373 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.239590 12638 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.187021 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.126842 14596 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.177320 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:08.243481 12566 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.132364 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.240454 12643 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.244163 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:08.238946 12565 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:08.229593 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:08.242116 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:08.238533 12490 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:08.238453 12567 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:08.250873 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196215us
W20260709 12:12:08.220933 13253 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.905s	user 0.013s	sys 0.001s
W20260709 12:12:08.263971 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 136793us
I20260709 12:12:08.174043 12564 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:08.184489 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.251533 12392 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 4 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 5 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.115721 14679 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:08.282651 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 230432us
W20260709 12:12:08.242772 14669 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.191s	user 0.001s	sys 0.001s
I20260709 12:12:08.185209 14633 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.294631 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.283926 12645 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:08.282456 12492 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:08.285369 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.275543 12491 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:08.242894 14607 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.300037 12490 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5.
I20260709 12:12:08.177385 12641 rpcz_store.cc:276] 0709 12:12:07.173139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.175306 (+1002167us) service_pool.cc:224] Handling call
0709 12:12:08.177042 (+  1736us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96512}
I20260709 12:12:08.288331 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.290194 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:08.245021 14680 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:08.243743 14678 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.141s	user 0.000s	sys 0.002s
I20260709 12:12:08.175661 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.184296 14537 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:08.305130 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.272406 14305 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.306155 12491 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.301647 12392 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.313107 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:08.168035 14482 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20260709 12:12:08.282878 14454 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:08.304096 14679 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.294849 14625 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.333534 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:08.339121 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.340471 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.341176 12493 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.342018 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.342932 12571 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:08.343114 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1467) took 1170 ms. Trace:
I20260709 12:12:08.264173 14596 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.343691 12641 rpcz_store.cc:276] 0709 12:12:07.172783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.324770 (+1151987us) service_pool.cc:224] Handling call
0709 12:12:08.343032 (+ 18262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.344460 12571 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:08.327665 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.349900 12497 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:08.350854 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:08.329015 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.351971 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:08.329881 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:08.330181 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.344995 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.353955 12641 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.346105 14596 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.345824 12392 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:08.284744 13253 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:08.357594 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.347246 14684 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:08.356912 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.362273 12497 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:08.360704 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.362730 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:08.186321 14578 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.365852 12491 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:08.367326 12490 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.371284 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:08.325311 14454 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.331104 14567 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.317523 14680 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:08.187836 14570 consensus_queue.cc:1175] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 3 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: [2.5-2.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:12:08.372329 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:08.398557 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 166) took 1110 ms. Trace:
I20260709 12:12:08.383584 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1451) took 1247 ms. Trace:
I20260709 12:12:08.398926 12644 rpcz_store.cc:276] 0709 12:12:07.288267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.830178 (+541911us) service_pool.cc:224] Handling call
0709 12:12:08.398490 (+568312us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:08.394001 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60592.3us
I20260709 12:12:08.397886 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:08.406369 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:08.324147 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:08.352020 14603 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:08.378836 12392 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.358610 14643 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:08.404224 12645 rpcz_store.cc:276] 0709 12:12:07.136614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.138599 (+  1985us) service_pool.cc:224] Handling call
0709 12:12:08.383178 (+1244579us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:08.383430 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2502803840}
I20260709 12:12:08.430174 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:08.386888 14674 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.143s	user 0.000s	sys 0.003s
W20260709 12:12:08.438714 12642 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.616s	user 0.013s	sys 0.001s
W20260709 12:12:08.419191 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56610.1us
I20260709 12:12:08.373342 14684 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.424918 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.378014 14567 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:08.390034 14663 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:08.439855 12642 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:08.440543 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.441408 12644 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:08.453193 14677 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
I20260709 12:12:08.374890 14454 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:08.456490 12489 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:08.428114 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
W20260709 12:12:08.461962 14676 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 12:12:08.462101 14683 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.462808 14683 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.424342 14686 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.442330 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.442091 12392 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.403761 14661 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.449769 12641 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:08.331566 14305 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.427417 14404 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.485430 14684 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:08.435356 14687 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:08.450194 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.449769 12567 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:08.456060 14679 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.465708 13254 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:08.466781 14683 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.560357 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:08.561141 12570 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.469080 14596 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.471869 14603 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:08.473923 14577 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:08.475486 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:08.477208 14675 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:12:08.445825 14619 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:08.442739 14403 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.424952 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1460) took 1254 ms. Trace:
I20260709 12:12:08.577139 12639 rpcz_store.cc:276] 0709 12:12:07.170498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.175314 (+1004816us) service_pool.cc:224] Handling call
0709 12:12:08.421864 (+246550us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.578318 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.578992 12639 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:08.579519 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1481) took 1066 ms. Trace:
I20260709 12:12:08.579948 12639 rpcz_store.cc:276] 0709 12:12:07.512814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.577988 (+1065174us) service_pool.cc:224] Handling call
0709 12:12:08.579458 (+  1470us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:08.483433 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53362.1us
I20260709 12:12:08.583349 14687 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:08.485249 12491 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:08.486532 14447 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.003s
I20260709 12:12:08.398025 14386 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.588250 14386 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:08.488502 14673 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.375s	user 0.000s	sys 0.001s
I20260709 12:12:08.589468 12488 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.489078 14643 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:08.591058 14643 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.591584 14643 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:08.591979 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1595) took 1228 ms (client timeout 1568 ms). Trace:
W20260709 12:12:08.592574 12488 rpcz_store.cc:269] 0709 12:12:07.363138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.363845 (+   707us) service_pool.cc:224] Handling call
0709 12:12:07.856524 (+492679us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:08.591622 (+735098us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:08.591851 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1593657088}
I20260709 12:12:08.491205 14680 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.402741 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 285) took 1145 ms. Trace:
I20260709 12:12:08.593873 13253 rpcz_store.cc:276] 0709 12:12:07.226268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.234766 (+  8498us) service_pool.cc:224] Handling call
0709 12:12:07.313523 (+ 78757us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:08.371317 (+1057794us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8280320}
I20260709 12:12:08.461962 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1449) took 1392 ms. Trace:
I20260709 12:12:08.596127 12488 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:08.595242 12637 rpcz_store.cc:276] 0709 12:12:07.069427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.072893 (+  3466us) service_pool.cc:224] Handling call
0709 12:12:07.073276 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:07.079702 (+  6426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:07.079747 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:07.084526 (+  4779us) log.cc:844] Serialized 26 byte log entry
0709 12:12:07.306632 (+222106us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:07.566134 (+259502us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:08.337988 (+771854us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b1800436000. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:12:08.338187 (+   199us) log.cc:844] Serialized 14 byte log entry
0709 12:12:08.351760 (+ 13573us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.351846 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:08.461701 (+109855us) raft_consensus.cc:1685] finished
0709 12:12:08.461718 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:08.461870 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37639,"prepare.queue_time_us":308819,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3491,"raft.queue_time_us":19712,"raft.run_cpu_time_us":3805,"raft.run_wall_time_us":238088,"spinlock_wait_cycles":32006144,"thread_start_us":526066,"threads_started":2,"wal-append.queue_time_us":838103}
I20260709 12:12:08.498860 12569 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:12:08.482685 14469 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:08.500440 12392 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.599099 14305 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.599400 12392 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.461458 12492 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:08.502429 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.505939 12644 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:08.419039 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1475) took 1106 ms. Trace:
I20260709 12:12:08.602356 12565 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 2.
I20260709 12:12:08.512748 12572 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:12:08.545533 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:08.381306 14220 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:08.602768 14454 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.502491 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1478) took 1189 ms. Trace:
I20260709 12:12:08.640645 12642 rpcz_store.cc:276] 0709 12:12:07.313263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.347245 (+ 33982us) service_pool.cc:224] Handling call
0709 12:12:08.502415 (+1155170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.549425 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.511754 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:08.558668 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1477) took 1245 ms. Trace:
I20260709 12:12:08.642362 12641 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:08.564512 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:08.642786 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131152us
I20260709 12:12:08.643392 13261 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:12:08.508107 14644 raft_consensus.cc:2764] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:12:08.602413 12643 rpcz_store.cc:276] 0709 12:12:07.312173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.038964 (+726791us) service_pool.cc:224] Handling call
0709 12:12:08.186851 (+147887us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:08.418223 (+231372us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9948032}
W20260709 12:12:08.576102 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 301914us
W20260709 12:12:08.488646 14671 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.582s	user 0.000s	sys 0.002s
I20260709 12:12:08.594280 12490 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:12:08.660604 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.603295 14220 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:08.547379 14666 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.227s	user 0.011s	sys 0.002s
W20260709 12:12:08.603430 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58038.9us
I20260709 12:12:08.546429 14653 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:08.603785 14634 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:08.609382 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:08.664925 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 170) took 1375 ms. Trace:
I20260709 12:12:08.601027 12392 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 2, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 2 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.574714 14596 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.574489 14401 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:08.602499 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1472) took 1333 ms. Trace:
I20260709 12:12:08.642494 12645 rpcz_store.cc:276] 0709 12:12:07.312899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.430496 (+1117597us) service_pool.cc:224] Handling call
0709 12:12:08.558558 (+128062us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.643950 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:08.646024 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.679417 12644 rpcz_store.cc:276] 0709 12:12:07.269593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.405477 (+1135884us) service_pool.cc:224] Handling call
0709 12:12:08.602424 (+196947us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.570583 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:08.645185 12638 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:08.680949 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1476) took 1368 ms. Trace:
I20260709 12:12:08.648906 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:08.682214 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.683365 12569 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.510871 14661 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.580976 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.659858 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:08.714270 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.715406 12567 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:08.716535 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:08.720129 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:08.665335 12642 rpcz_store.cc:276] 0709 12:12:07.289413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.660243 (+1370830us) service_pool.cc:224] Handling call
0709 12:12:08.664840 (+  4597us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.665424 12637 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:08.681324 12638 rpcz_store.cc:276] 0709 12:12:07.312524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.039070 (+726546us) service_pool.cc:224] Handling call
0709 12:12:08.181166 (+142096us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:08.680888 (+499722us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6238080}
I20260709 12:12:08.509493 14686 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.729070 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1471) took 1484 ms. Trace:
I20260709 12:12:08.729808 12643 rpcz_store.cc:276] 0709 12:12:07.244606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.659350 (+1414744us) service_pool.cc:224] Handling call
0709 12:12:08.728967 (+ 69617us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.731523 12643 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:08.732436 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1469) took 1559 ms. Trace:
I20260709 12:12:08.732857 12643 rpcz_store.cc:276] 0709 12:12:07.173454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.730719 (+1557265us) service_pool.cc:224] Handling call
0709 12:12:08.731238 (+   519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.732049 (+   811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.732358 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.659368 14532 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:08.738200 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.738600 14661 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:08.752170 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 163) took 1464 ms. Trace:
I20260709 12:12:08.752820 12639 rpcz_store.cc:276] 0709 12:12:07.287425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.580666 (+1293241us) service_pool.cc:224] Handling call
0709 12:12:08.752064 (+171398us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.739856 12642 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:08.755419 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1462) took 1584 ms. Trace:
I20260709 12:12:08.754925 12392 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 3, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.755977 12642 rpcz_store.cc:276] 0709 12:12:07.171123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.722178 (+1551055us) service_pool.cc:224] Handling call
0709 12:12:08.739631 (+ 17453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.755059 (+ 15428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.755341 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:08.755935 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 322473us
I20260709 12:12:08.753834 12645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:12:08.770452 12642 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:08.770537 14635 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.771387 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 159) took 1485 ms. Trace:
I20260709 12:12:08.771768 12642 rpcz_store.cc:276] 0709 12:12:07.286183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.756887 (+1470704us) service_pool.cc:224] Handling call
0709 12:12:08.770013 (+ 13126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.771002 (+   989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.771281 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.772805 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1455) took 1612 ms. Trace:
I20260709 12:12:08.773254 12640 rpcz_store.cc:276] 0709 12:12:07.160737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.165768 (+  5031us) service_pool.cc:224] Handling call
0709 12:12:07.166162 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:07.166583 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:07.166614 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:07.167135 (+   521us) log.cc:844] Serialized 26 byte log entry
0709 12:12:07.328024 (+160889us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:07.366346 (+ 38322us) log.cc:844] Serialized 14 byte log entry
0709 12:12:07.802190 (+435844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:07.802321 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:07.932504 (+130183us) spinlock_profiling.cc:243] Waited 5.52 ms on lock 0x7b180032d400. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:08.487839 (+555335us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:08.772551 (+284712us) raft_consensus.cc:1685] finished
0709 12:12:08.772580 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:08.772735 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5135,"prepare.queue_time_us":68590,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"raft.queue_time_us":458999,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":19051,"spinlock_wait_cycles":332489728,"thread_start_us":160454,"threads_started":1,"wal-append.queue_time_us":733201}
I20260709 12:12:08.775669 12640 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 5. Ops: []
I20260709 12:12:08.776307 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 162) took 1489 ms. Trace:
I20260709 12:12:08.776593 12640 rpcz_store.cc:276] 0709 12:12:07.287162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.775125 (+1487963us) service_pool.cc:224] Handling call
0709 12:12:08.775492 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.776037 (+   545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.776250 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.778084 12642 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:08.778815 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 165) took 1490 ms. Trace:
I20260709 12:12:08.779374 12642 rpcz_store.cc:276] 0709 12:12:07.287974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.777402 (+1489428us) service_pool.cc:224] Handling call
0709 12:12:08.777783 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.778575 (+   792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.778761 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.780791 14625 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:08.781205 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 169) took 1492 ms. Trace:
I20260709 12:12:08.678936 14634 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:08.785317 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141491us
I20260709 12:12:08.790514 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
W20260709 12:12:08.782177 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 220070us
I20260709 12:12:08.792562 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1482) took 1049 ms. Trace:
I20260709 12:12:08.794940 12637 rpcz_store.cc:276] 0709 12:12:07.743311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.597621 (+854310us) service_pool.cc:224] Handling call
0709 12:12:08.792385 (+194764us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.791589 12641 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:08.784655 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:08.797379 12643 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:12:08.797901 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1473) took 1486 ms. Trace:
I20260709 12:12:08.799333 12643 pending_rounds.cc:85] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 2
I20260709 12:12:08.781868 12642 rpcz_store.cc:276] 0709 12:12:07.289140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.780078 (+1490938us) service_pool.cc:224] Handling call
0709 12:12:08.780418 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.780763 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:08.780793 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:08.780841 (+    48us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:08.780887 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.781033 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:08.781147 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.799631 12641 rpcz_store.cc:276] 0709 12:12:07.311255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.533574 (+1222319us) service_pool.cc:224] Handling call
0709 12:12:08.797806 (+264232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.801188 12565 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:08.740952 14537 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:08.811359 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 370945us
I20260709 12:12:08.809158 12392 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 3, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.810701 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.799930 12643 pending_rounds.cc:107] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:12:08.796619 12638 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:08.815603 12643 raft_consensus.cc:2889] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:08.815893 12640 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:12:08.816833 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.822031 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.823000 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.793905 14447 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:08.825745 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1463) took 1644 ms. Trace:
I20260709 12:12:08.833140 12638 rpcz_store.cc:276] 0709 12:12:07.171474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.722307 (+1550833us) service_pool.cc:224] Handling call
0709 12:12:08.796409 (+ 74102us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.815285 (+ 18876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.815517 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.825872 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.794871 14532 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:08.830950 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1470) took 1618 ms. Trace:
I20260709 12:12:08.798195 14634 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:08.815245 14537 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.840149 14537 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.821512 14386 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:12:08.844038 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.845495 14664 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20260709 12:12:08.847803 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.839303 12643 rpcz_store.cc:276] 0709 12:12:07.212353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.737019 (+1524666us) service_pool.cc:224] Handling call
0709 12:12:08.795083 (+ 58064us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.830287 (+ 35204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:08.830855 (+   568us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:08.834430 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:08.714144 14653 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:08.835747 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 158874us
I20260709 12:12:08.783727 14680 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
W20260709 12:12:08.837949 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 559456us
I20260709 12:12:08.798206 14596 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.808072 14661 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:08.872618 14661 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:08.872829 14653 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:08.873113 14386 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:08.878243 14386 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:08.879096 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.879398 14386 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.835300 12392 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:12:08.854952 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:08.825251 14473 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:08.869853 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:08.870173 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52156us
I20260709 12:12:08.866487 14537 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.849978 12645 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:08.882823 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.883059 12645 raft_consensus.cc:614] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:08.873180 14661 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.874761 14454 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:08.826645 14684 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:08.876822 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54661.4us
I20260709 12:12:08.877336 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:08.878129 14694 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:08.687166 14666 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:08.884128 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.870606 14641 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:08.848991 14696 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:12:08.889700 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78216.9us
I20260709 12:12:08.890532 12610 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5.
I20260709 12:12:08.891212 12392 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:08.902215 12610 leader_election.cc:403] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:08.894183 12493 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:12:08.903314 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:12:08.904942 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50708.4us
I20260709 12:12:08.908526 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:08.910724 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98190.5us
I20260709 12:12:08.892356 14661 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:08.905298 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:08.889392 14473 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:08.919011 14701 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:12:08.916769 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:08.907676 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.916515 14578 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:08.920583 14703 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.920854 14447 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:08.917004 12565 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:12:08.921186 14578 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:08.920332 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.922564 14578 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:08.922971 14447 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:08.923358 14447 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:08.923786 14578 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.924214 14709 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:08.924324 14447 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.925594 14653 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:08.926079 14699 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:08.926215 14653 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.926573 14699 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:08.926890 14653 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.928284 14689 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:08.918439 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.930035 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.898422 14570 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:08.930488 14697 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:12:08.931138 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.930945 14570 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.931905 14447 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.932080 14404 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:08.932996 14404 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:08.934068 14404 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.934365 14570 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.919301 14473 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:08.934995 12565 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:12:08.936321 14683 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:08.936811 14683 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:08.937160 12565 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:08.937204 14683 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.930541 12568 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:08.920292 13254 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.940297 14537 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.891096 14663 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:08.929554 14619 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:12:08.921123 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.922418 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:08.941102 14663 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.941174 14619 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:08.927067 14699 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:08.941587 14663 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.941792 14701 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:08.942416 14701 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:08.942523 14619 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.943778 14619 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.943415 14701 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.893302 14684 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:08.945988 14684 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:08.946370 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:08.947366 14353 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:12:08.952176 14607 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.625s	user 0.006s	sys 0.003s
I20260709 12:12:08.952690 14607 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.914124 14693 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:12:08.954041 14596 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:08.957576 14596 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:08.957849 14686 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:08.958738 14596 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.900478 14386 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:08.957010 14532 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:08.910593 14695 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:12:08.955652 12495 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:12:08.956773 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:08.955657 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.880640 14220 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:08.956784 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.078398 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:08.953244 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:09.085723 12570 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:09.086050 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.060935 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.081189 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:08.956490 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:09.086481 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.086567 12564 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:12:09.088068 13257 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:09.043226 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137964us
I20260709 12:12:08.898887 14679 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:09.038136 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:08.904881 12639 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:12:09.092213 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136190us
I20260709 12:12:09.099754 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:09.093837 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140745us
W20260709 12:12:09.095975 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 57989us
I20260709 12:12:08.904729 14305 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:09.098380 12565 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.982933 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:09.099095 14679 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.250178 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:09.115082 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 257802us
W20260709 12:12:09.129798 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.253754 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.254969 12569 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:08.955332 14666 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.113102 14220 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:09.116719 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1459) took 1946 ms. Trace:
I20260709 12:12:09.081211 14607 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:09.087077 14643 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.107512 14694 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:09.201861 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.182596 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.119831 14702 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:09.072985 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:09.336091 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:09.341614 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.342355 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:09.168741 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.093331 14482 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:09.187146 12564 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:09.118954 14634 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.218700 14661 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.220748 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:09.162114 14567 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:09.221696 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1485) took 1374 ms. Trace:
I20260709 12:12:09.355872 14607 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:09.224419 13254 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:09.356814 14714 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:09.357342 14714 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:09.357978 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.358453 14714 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:09.359424 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.360018 12567 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.360060 14714 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:09.227365 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:09.362496 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 261368us
I20260709 12:12:09.362607 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:09.363519 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:09.364758 14715 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:12:09.227805 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.016168 14699 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:09.146373 14705 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.207s	user 0.005s	sys 0.000s
W20260709 12:12:09.234539 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134861us
I20260709 12:12:09.233306 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1843) took 1221 ms. Trace:
I20260709 12:12:08.912882 14641 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.370223 14695 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:09.370430 12494 rpcz_store.cc:276] 0709 12:12:08.103646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.112110 (+  8464us) service_pool.cc:224] Handling call
0709 12:12:09.233063 (+1120953us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:09.233227 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1497602816}
I20260709 12:12:09.242091 14709 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:09.371171 13260 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:09.371275 14447 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:09.371361 14695 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:09.372068 14447 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.372609 14709 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.123258 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:09.373534 14634 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:09.346375 14702 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:09.374092 12493 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:09.374819 12493 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:09.116287 12570 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:09.243120 12572 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:12:09.376269 12572 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 1
I20260709 12:12:09.376539 12572 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:12:09.376772 12572 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:09.116747 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.131455 14692 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:09.145077 14619 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:08.987546 14644 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.218s
I20260709 12:12:09.131942 14686 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:09.422873 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.248525 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:09.423174 14686 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.423723 12497 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
W20260709 12:12:09.250324 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 267528us
W20260709 12:12:09.110237 14706 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.178s	user 0.000s	sys 0.002s
I20260709 12:12:09.367842 14482 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:09.249583 14305 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.257611 14684 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.426265 14305 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.426637 14482 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.428567 14684 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.258487 14577 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:09.429710 14577 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.430238 14577 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:09.431499 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:09.432996 14712 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:09.433343 14643 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:09.433565 14712 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.434046 14712 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:09.356026 12639 rpcz_store.cc:276] 0709 12:12:07.847729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.808942 (+961213us) service_pool.cc:224] Handling call
0709 12:12:08.854922 (+ 45980us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:09.098933 (+244011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:09.221545 (+122612us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:09.177165 14653 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.387549 14692 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:09.436672 14653 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:09.129516 14473 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.372052 14695 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:09.262485 14700 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:09.270957 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.277034 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:09.273685 14708 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:09.273685 14679 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.444960 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:09.445331 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 168381us
I20260709 12:12:09.425393 14686 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.445767 12497 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.274422 14701 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:09.322113 14663 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.323975 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:09.325295 14454 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:12:09.152963 14683 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.333269 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1461) took 2162 ms. Trace:
I20260709 12:12:09.231536 14703 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5.
I20260709 12:12:09.280905 12646 rpcz_store.cc:276] 0709 12:12:07.170144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.287857 (+1117713us) service_pool.cc:224] Handling call
0709 12:12:09.116531 (+828674us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:09.116642 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1132856064}
I20260709 12:12:09.455925 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:12:09.333992 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:09.458734 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.124393 14404 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.354015 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.346688 13258 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:09.365440 14689 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:09.375085 14702 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:09.381397 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.384619 14661 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.394846 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:09.398000 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:09.418628 14713 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:12:09.398765 14403 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:12:09.434078 14699 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.269053 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.522068 12565 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 2.
I20260709 12:12:09.438580 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:09.524605 12494 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:09.440389 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 402) took 1585 ms. Trace:
W20260709 12:12:09.440550 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 271859us
I20260709 12:12:09.442812 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:09.394177 14715 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:09.534891 12563 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:09.525743 13260 rpcz_store.cc:276] 0709 12:12:07.855380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:07.855575 (+   195us) service_pool.cc:224] Handling call
0709 12:12:09.440218 (+1584643us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:09.440330 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3244185472}
I20260709 12:12:09.297843 14680 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:09.393947 14386 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:09.538771 12568 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:09.455404 12644 rpcz_store.cc:276] 0709 12:12:07.170822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.712991 (+1542169us) service_pool.cc:224] Handling call
0709 12:12:08.713340 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:08.795088 (+ 81748us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:12:08.910182 (+115094us) log.cc:844] Serialized 14 byte log entry
0709 12:12:09.332597 (+422415us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:09.332708 (+   111us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:09.332794 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:09.332925 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:09.333161 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11392,"prepare.queue_time_us":260066,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":196,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":2190,"thread_start_us":452739,"threads_started":2,"wal-append.queue_time_us":434376}
I20260709 12:12:09.450213 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:09.466249 13256 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:12:09.467005 12572 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:09.467131 14607 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:09.543717 14683 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.512768 12493 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:12:09.544620 14683 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:09.545583 14703 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:12:09.485746 14454 raft_consensus.cc:1068] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:12:09.446859 14643 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:09.528945 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:09.530903 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.557720 12493 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:09.558207 12566 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.450915 14701 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.451244 14625 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.101s
W20260709 12:12:09.537046 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 139139us
I20260709 12:12:09.542343 14386 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:09.553208 14715 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.604069 13258 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:09.604636 12564 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:09.604206 14386 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.552850 14664 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:09.606004 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:09.491904 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.549585 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:09.550386 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101893us
I20260709 12:12:09.554917 13257 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:12:09.454766 14700 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:09.471710 14641 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:09.455084 14723 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:12:09.472337 14473 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.606670 12565 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.620026 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.621352 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:09.620060 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
W20260709 12:12:09.561895 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115566us
W20260709 12:12:09.571722 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.579258 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.590581 14447 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.474099 14728 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.171s
I20260709 12:12:09.644109 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.594661 14454 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:09.558362 14607 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:09.606248 14664 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:09.662757 12567 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 12:12:09.594079 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:12:09.665612 12637 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260709 12:12:09.665658 12644 raft_consensus.cc:606] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:09.665493 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 222771us
I20260709 12:12:09.667399 12644 raft_consensus.cc:614] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:09.649386 12569 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:09.668371 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:09.650203 12568 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:09.641752 12566 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:09.670727 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:09.654314 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:09.655859 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:09.595292 14625 consensus_queue.cc:1243] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:12:09.658419 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:09.629590 12564 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:09.557817 14643 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:09.632277 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103438us
I20260709 12:12:09.659667 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:09.684878 12646 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:12:09.650502 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.685102 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.687641 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.689749 14567 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:09.690398 14567 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:09.690891 14567 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.695068 14700 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.688709 12567 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:09.677949 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:09.619150 14686 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:09.574769 14701 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.682873 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.701298 12536 leader_election.cc:400] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:09.712663 12536 leader_election.cc:403] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:09.754032 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:09.754866 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:09.758277 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.556252 14680 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.659610 14607 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:09.699501 14447 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:09.758941 12570 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 6
W20260709 12:12:09.786355 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85275.3us
I20260709 12:12:09.787249 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:09.789043 12565 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
W20260709 12:12:09.774720 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 153827us
I20260709 12:12:09.611886 14679 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.147s
I20260709 12:12:09.612303 14724 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.270s
I20260709 12:12:09.469051 14695 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.815390 12565 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:09.826824 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:09.809150 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124169us
I20260709 12:12:09.711869 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:09.726598 14728 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:12:09.821225 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2983860379999896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260709 12:12:09.832581 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:09.845820 12536 connection.cc:256] RPC call timeout handler was delayed by -2.231786037999882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20260709 12:12:09.834046 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:09.823546 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.725442 14689 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:09.848842 12491 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
W20260709 12:12:09.822326 14596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.020s	sys 0.019s
I20260709 12:12:09.824358 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:09.851917 14596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.021s	sys 0.019s
I20260709 12:12:09.852361 14695 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.853183 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:09.846350 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:09.822556 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 265365us
I20260709 12:12:09.796236 14715 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:09.835176 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:09.811183 14447 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.847040 13257 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:09.847393 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:09.819154 14699 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
W20260709 12:12:09.846675 12536 connection.cc:256] RPC call timeout handler was delayed by -2.187686038000038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
W20260709 12:12:09.866580 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1813860380000278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
I20260709 12:12:09.821923 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.854015 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:09.870316 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:09.828729 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.854413 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:09.759100 14686 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.869005 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:09.877669 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:09.874053 12568 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:09.883553 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:09.848953 14689 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.868934 14447 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:09.873742 12639 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:09.892439 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70638.8us
I20260709 12:12:09.891705 14404 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.905699 14709 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:09.906272 14709 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:09.906628 14709 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:09.909749 14716 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:09.910061 14625 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:09.910821 14625 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:09.910162 14716 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.911420 14716 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:09.912693 13257 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:09.846465 14732 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 12:12:09.921168 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:09.868471 14664 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.927789 14634 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:09.928368 14634 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:09.909085 14305 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.929247 14634 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.931420 14305 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.933194 12495 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:09.875698 14482 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:12:09.934042 14482 consensus_queue.cc:1243] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:12:09.830863 14643 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:09.878041 14663 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.942576 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:09.896921 14689 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:09.956314 13257 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:09.959239 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:09.961567 12640 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
W20260709 12:12:09.971642 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143051us
I20260709 12:12:09.975490 14739 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:09.976382 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.976246 14625 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:09.977427 14739 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:09.978117 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.979339 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:09.979799 14739 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:09.980754 14739 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:09.981031 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.982182 14744 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:09.982681 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:09.983769 14744 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:09.984539 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:09.985339 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64301.6us
I20260709 12:12:09.986467 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:09.986802 14625 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:09.987846 14746 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:09.987702 14625 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:09.988183 14744 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:12:09.988423 14746 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:09.988938 14746 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:09.998028 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:10.001252 14635 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:10.001955 14635 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:10.002851 14635 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:09.952847 14567 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.006332 14680 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.008150 14567 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.999763 14746 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:10.008828 14712 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.574s	user 0.010s	sys 0.000s
I20260709 12:12:10.009569 14712 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:10.005895 14577 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.575s	user 0.012s	sys 0.000s
I20260709 12:12:10.012387 14712 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.012569 14577 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.013830 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.014714 12639 raft_consensus.cc:2376] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5. Re-sending same reply.
I20260709 12:12:09.960327 14728 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:10.015040 14577 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:10.016975 14746 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:09.961297 14643 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:10.018779 12536 connection.cc:256] RPC call timeout handler was delayed by -2.11478603799992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20260709 12:12:10.021737 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1111860379999143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
W20260709 12:12:10.022073 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0373860380000224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260709 12:12:10.022413 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0076860379999744s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20260709 12:12:10.022699 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9671860379999089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
W20260709 12:12:10.023017 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9617860379999001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
W20260709 12:12:10.023424 12536 connection.cc:256] RPC call timeout handler was delayed by -1.921286038000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1955s for a response
W20260709 12:12:10.023798 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7871860379998452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20260709 12:12:10.024163 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6800860379998994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
W20260709 12:12:10.024541 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5864860379999755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23270000000000002s for a response
W20260709 12:12:10.024914 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1359148890001052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260709 12:12:10.025249 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0216148890001477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
W20260709 12:12:10.025606 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0207148890001463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20260709 12:12:10.028194 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:10.031572 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:10.016310 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.032899 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:12:10.019219 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.019767 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.020426 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.035562 12643 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:10.036782 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.038511 12643 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:10.041090 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.041641 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:10.034648 12640 raft_consensus.cc:2376] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5. Re-sending same reply.
I20260709 12:12:10.048358 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:10.049001 13257 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:10.049500 12646 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:10.053969 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.054024 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:10.054780 12461 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.515s (SENT)
I20260709 12:12:10.055377 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.046626 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:10.054955 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:10.056429 12637 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:09.942348 14607 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:10.059770 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:10.018410 14724 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:10.061398 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.072579 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:10.040694 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.041023 14732 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:10.045761 14746 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:12:10.077657 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90773us
I20260709 12:12:10.081722 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.082444 12640 raft_consensus.cc:2376] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3. Re-sending same reply.
I20260709 12:12:10.082679 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:10.061095 14663 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:10.061527 14386 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:10.086656 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.090545 12643 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:10.091902 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.070771 14482 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:10.064154 14712 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:12:10.066250 14724 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:10.093654 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.095245 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:10.080366 14404 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:10.098229 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.097949 12643 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:10.015897 14728 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.083935 14694 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:10.090179 14663 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:10.078611 14699 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:12:10.086818 14732 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.092060 14679 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:12:10.108297 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:10.108970 14734 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:10.104441 12639 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:10.101883 14664 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:12:10.161537 14692 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.075251 14661 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.025s
W20260709 12:12:10.162691 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54469.2us
I20260709 12:12:10.163381 14692 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:10.163827 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:10.091037 14386 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:10.165467 14386 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.165129 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:10.166033 14734 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:10.166500 14734 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:10.167309 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.167487 14734 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.168718 14734 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:10.169310 14714 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:10.169871 14714 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:10.170276 14714 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:10.170224 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.170921 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.171257 14728 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:10.131712 14694 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:10.172784 14694 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.173882 14404 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:10.122270 14473 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:12:10.181885 12494 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:10.124688 14447 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.099850 14625 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:10.091717 14607 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.101428 14482 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:10.194612 12490 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:10.148571 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.151654 14732 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:10.087080 14703 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 12:12:10.127610 14739 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:10.152648 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:10.101538 14746 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:10.149322 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.204591 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.205276 12640 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
I20260709 12:12:10.203548 14746 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:10.206444 14746 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:10.115612 14709 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.207726 14689 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.209518 14709 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.174568 12563 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:12:10.210228 14689 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.173293 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:12:10.211068 14749 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.112s	user 0.000s	sys 0.002s
I20260709 12:12:10.211863 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.166512 12488 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:10.180212 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.206449 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.130110 14663 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.215735 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.217005 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.195662 12643 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:10.213613 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:10.221081 14699 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:10.221487 14699 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:10.222028 14699 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.222088 14732 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:10.223217 14732 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:10.224232 14732 consensus_queue.cc:1243] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:12:10.224897 14732 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:10.162791 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.197187 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.225996 12639 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
I20260709 12:12:10.226601 12492 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:10.175274 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.210457 12563 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 1
I20260709 12:12:10.228102 12563 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:12:10.228461 12563 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:10.229502 14482 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.230712 14386 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.231808 12496 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:10.227154 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.233532 14701 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.233753 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.235445 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.236192 14596 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:10.237272 12493 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:10.238224 14567 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:10.239118 14567 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:10.201817 14716 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.239480 14567 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:10.239951 14567 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:10.200656 12494 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:10.235731 14701 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.225337 14732 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:10.247658 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166632us
I20260709 12:12:10.250754 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.256461 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:10.254860 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 168239us
I20260709 12:12:10.205093 14739 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:10.241048 14716 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.233894 14386 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.266769 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.276757 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.253603 14689 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:10.279363 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:10.241954 14625 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:10.204978 14661 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260709 12:12:10.254871 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.255119 14715 raft_consensus.cc:2764] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:12:10.277478 12494 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:10.266170 14404 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.281813 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.300961 12490 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:10.301407 14577 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:10.283151 12642 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:10.271216 14739 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:10.263681 12643 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:10.286989 14695 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:10.309480 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.310344 12490 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:10.310001 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.311071 12643 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:10.309295 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 187367us
I20260709 12:12:10.312371 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.313079 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:10.336926 12491 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:10.338140 14751 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:12:10.317351 14643 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:12:10.340757 14644 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:10.343515 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:10.302039 14577 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:10.348610 14714 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.350170 14447 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.351737 14663 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.391930 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.392917 12491 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.355090 12568 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:10.312378 14695 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:10.320273 14703 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:12:10.355943 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69141us
I20260709 12:12:10.364333 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:10.298241 14732 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:10.202049 14724 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:10.366122 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 290) took 1348 ms. Trace:
I20260709 12:12:10.360639 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.387041 12643 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:10.377236 14220 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:10.390714 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.353802 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:10.402381 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.856s (SENT)
I20260709 12:12:10.406103 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:12:10.419956 12638 raft_consensus.cc:2376] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
I20260709 12:12:10.414546 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.420975 12640 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:10.421084 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.422562 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.376531 14644 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:10.417189 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.380000 14577 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.420038 12646 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:10.427078 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.427868 12646 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:10.416610 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 281) took 1135 ms. Trace:
I20260709 12:12:10.431434 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:10.401981 14695 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.438480 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.429826 12564 rpcz_store.cc:276] 0709 12:12:09.270327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:09.353590 (+ 83263us) service_pool.cc:224] Handling call
0709 12:12:10.405672 (+1052082us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:10.405816 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330674944}
I20260709 12:12:10.442634 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.438333 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:10.448149 12489 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:10.451391 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:10.423760 14684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.013s	sys 0.009s
I20260709 12:12:10.369506 14625 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.458072 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.464637 12494 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:10.467747 12461 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.517s (SENT)
I20260709 12:12:10.444319 14220 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.486107 12641 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:10.489600 12646 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:10.445932 14732 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:10.459887 14684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.014s	sys 0.009s
I20260709 12:12:10.507151 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.508416 12640 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:10.508770 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.512331 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.513028 12641 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4. Re-sending same reply.
I20260709 12:12:10.514086 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.514822 12640 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:10.459475 14761 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:10.484246 14447 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:10.522941 14663 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:10.416103 13256 rpcz_store.cc:276] 0709 12:12:09.015844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:09.034266 (+ 18422us) service_pool.cc:224] Handling call
0709 12:12:10.357468 (+1323202us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:10.363952 (+  6484us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189744000}
W20260709 12:12:10.526126 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58487.8us
I20260709 12:12:10.527364 14567 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:10.530405 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128059us
W20260709 12:12:10.531607 14755 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
W20260709 12:12:10.555593 12461 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.546s (SENT)
I20260709 12:12:10.540395 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.557189 12641 raft_consensus.cc:2376] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3. Re-sending same reply.
I20260709 12:12:10.510396 14703 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:10.552587 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:10.569979 12536 leader_election.cc:403] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:12:10.532361 12646 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
W20260709 12:12:10.533386 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122630us
I20260709 12:12:10.515292 14577 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:10.560642 12490 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:10.565084 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
W20260709 12:12:10.579677 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161907us
I20260709 12:12:10.571813 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:10.582798 14757 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
I20260709 12:12:10.587385 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:10.587867 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:10.589414 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.589788 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.590332 12646 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:10.591713 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.599287 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:10.600041 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.601936 12646 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6. Re-sending same reply.
I20260709 12:12:10.603027 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.584909 14766 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:10.608567 12488 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
W20260709 12:12:10.585482 14700 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.004s	sys 0.014s
I20260709 12:12:10.584484 14447 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:10.594286 14772 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:10.610713 12640 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:10.621945 14703 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.532182 14644 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.538424 14761 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.596949 14689 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:10.647835 14769 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.352s
I20260709 12:12:10.999922 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.000208 14689 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:11.000779 12646 raft_consensus.cc:2376] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4. Re-sending same reply.
I20260709 12:12:10.645288 12564 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:12:10.955251 14739 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.614s	user 0.012s	sys 0.000s
W20260709 12:12:10.646459 14764 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:12:10.969605 14695 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.511s	user 0.006s	sys 0.006s
I20260709 12:12:10.814355 12491 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:10.644476 12563 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:10.655099 14699 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:10.639850 14447 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:11.007665 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:10.638550 12461 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.681s (SENT)
I20260709 12:12:11.009366 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:11.010061 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 371562us
W20260709 12:12:11.010769 12461 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.684s (SENT)
I20260709 12:12:11.011168 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:10.992754 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.005s	sys 0.007s
W20260709 12:12:11.018780 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.006s	sys 0.007s
I20260709 12:12:11.005432 14772 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:10.881484 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 294212us
I20260709 12:12:11.010210 12640 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.020589 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.026930 12646 raft_consensus.cc:2376] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6. Re-sending same reply.
I20260709 12:12:10.627853 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.003001 14644 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.040374 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.006855 14699 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.024600 14782 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:10.612540 14700 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.005s	sys 0.014s
W20260709 12:12:11.026386 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:11.028383 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:11.028348 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:11.051131 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:11.051383 12646 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6. Re-sending same reply.
W20260709 12:12:11.051831 12461 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.531s (SENT)
I20260709 12:12:11.052385 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:11.052779 12461 leader_election.cc:336] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.778s (SENT)
I20260709 12:12:11.053257 12646 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:11.043345 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:10.935132 12639 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:11.000953 14689 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:10.954898 14692 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.791s	user 0.016s	sys 0.000s
I20260709 12:12:11.003269 14739 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:10.960450 14404 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.661s	user 0.012s	sys 0.000s
I20260709 12:12:11.025451 14403 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.055853 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:11.055975 12461 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.808s (SENT)
I20260709 12:12:11.061188 12565 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:11.086767 12495 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
W20260709 12:12:11.067512 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:10.629345 14220 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.080190 12492 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:11.006862 14447 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:10.956815 14734 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.788s	user 0.008s	sys 0.007s
I20260709 12:12:10.614122 14766 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.084621 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1500) took 1039 ms. Trace:
I20260709 12:12:10.650043 14746 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.088126 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:11.093402 12461 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.821s (SENT)
I20260709 12:12:11.094434 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.099589 12637 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:11.100128 12488 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:12:11.101727 12461 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.612s (SENT)
I20260709 12:12:11.054142 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 184) took 1037 ms. Trace:
W20260709 12:12:11.085541 14577 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.501s	user 0.012s	sys 0.000s
I20260709 12:12:11.004379 14695 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.087217 12495 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 4
I20260709 12:12:11.097085 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.097829 12641 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.107242 12495 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:12:11.107599 12495 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:11.097354 12646 rpcz_store.cc:276] 0709 12:12:10.045343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.052095 (+1006752us) service_pool.cc:224] Handling call
0709 12:12:11.084548 (+ 32453us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:11.102903 12640 rpcz_store.cc:276] 0709 12:12:10.016577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.043075 (+1026498us) service_pool.cc:224] Handling call
0709 12:12:11.054072 (+ 10997us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:11.106352 12565 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:11.111204 12646 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:11.088478 14404 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.091291 14700 raft_consensus.cc:2764] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:12:11.087821 14777 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.111896 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1487) took 1076 ms. Trace:
I20260709 12:12:11.112542 12640 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:12:11.117424 12640 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 2
I20260709 12:12:11.117292 12646 rpcz_store.cc:276] 0709 12:12:10.035601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.110459 (+1074858us) service_pool.cc:224] Handling call
0709 12:12:11.110787 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.111672 (+   885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.111843 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:11.119640 12640 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:12:11.120015 12640 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:11.090564 14699 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:12:11.096199 14447 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.133249 14403 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:11.134270 14403 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.137203 14403 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.139067 14746 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.101251 14776 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:11.139797 14766 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.086404 14782 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.154804 12637 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:11.106357 14567 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.149336 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1497) took 1105 ms. Trace:
I20260709 12:12:11.163283 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.167233 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.164218 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:11.168334 12488 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:11.169395 14454 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:11.175535 14454 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:11.176736 14454 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.174966 12645 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:11.179011 12640 rpcz_store.cc:276] 0709 12:12:10.044355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.111913 (+1067558us) service_pool.cc:224] Handling call
0709 12:12:11.112215 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.148932 (+ 36717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.149238 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:11.166195 12637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:12:11.181473 12637 raft_consensus.cc:606] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.154876 14770 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:11.191045 12488 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:11.193547 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:12:11.196580 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:12:11.120806 14577 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:11.203758 14577 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.205888 14567 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:11.207001 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1592) took 1550 ms (client timeout 1856 ms). Trace:
W20260709 12:12:11.179855 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 552078us
I20260709 12:12:11.206578 14567 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:11.211691 12566 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:12:11.209923 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:11.207490 12642 rpcz_store.cc:269] 0709 12:12:09.656809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:09.657093 (+   284us) service_pool.cc:224] Handling call
0709 12:12:10.170062 (+512969us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:11.206753 (+1036691us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:11.206931 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272508544}
W20260709 12:12:11.184736 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1589) took 1531 ms (client timeout 1696 ms). Trace:
W20260709 12:12:11.213263 12645 rpcz_store.cc:269] 0709 12:12:09.653368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:09.653526 (+   158us) service_pool.cc:224] Handling call
0709 12:12:11.167049 (+1513523us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:11.184653 (+ 17604us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14431616}
I20260709 12:12:11.108671 14772 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.211925 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:11.178418 14783 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:12:11.215687 12572 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:11.214929 12566 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:12:11.217082 12565 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:11.217229 12564 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:12:11.218143 12570 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:11.219724 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:11.220429 12461 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.695s (SENT)
I20260709 12:12:11.223404 12567 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.224771 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.246658 12642 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:12:11.245534 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:11.209064 14776 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:11.250370 12572 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.250447 12563 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:11.207075 14577 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.229503 14771 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:11.211463 14699 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:11.123456 14777 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.226327 14447 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:11.265918 12461 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.848s (SENT)
I20260709 12:12:11.268769 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:11.269054 12461 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.029s (SENT)
I20260709 12:12:11.260499 12639 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:12:11.259531 14607 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.276026 12642 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:11.263469 14754 raft_consensus.cc:2764] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:12:11.281527 14785 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.282203 14785 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.284680 14744 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.285305 14744 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.288327 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1506) took 1235 ms. Trace:
I20260709 12:12:11.288509 14776 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.288846 12640 rpcz_store.cc:276] 0709 12:12:10.053178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.180552 (+1127374us) service_pool.cc:224] Handling call
0709 12:12:11.181017 (+   465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:11.187845 (+  6828us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:11.187875 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:11.188492 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 12:12:11.252210 (+ 63718us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:11.257044 (+  4834us) log.cc:844] Serialized 14 byte log entry
0709 12:12:11.270340 (+ 13296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.270480 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:11.288001 (+ 17521us) raft_consensus.cc:1685] finished
0709 12:12:11.288028 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:11.288239 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16337,"prepare.queue_time_us":7842,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":17416,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":3811,"thread_start_us":63267,"threads_started":1,"wal-append.queue_time_us":97174}
I20260709 12:12:11.294943 14567 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:11.310828 12461 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.142s (SENT)
I20260709 12:12:11.272392 14771 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:11.328907 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.325918 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.329113 13261 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:12:11.329896 13257 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:11.326597 12642 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:11.267405 14699 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.333369 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.338039 14771 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.339215 14694 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.335040 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 189) took 1034 ms. Trace:
I20260709 12:12:11.331372 14716 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.342694 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.341171 12642 rpcz_store.cc:276] 0709 12:12:10.300407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.277960 (+977553us) service_pool.cc:224] Handling call
0709 12:12:11.318656 (+ 40696us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.330367 (+ 11711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.334980 (+  4613us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:11.347822 14777 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.349885 14728 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.353474 14712 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.358930 14716 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.359263 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.359340 14728 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.359678 14712 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.360208 12488 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:11.361184 14716 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.361840 14733 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:11.362169 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:11.362962 14733 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:11.363068 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:11.363389 14733 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:11.363679 14644 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:12:11.364029 14694 raft_consensus.cc:2764] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:12:11.364315 14664 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:11.364938 14664 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:11.350060 14695 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.350585 12646 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:11.366046 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1521) took 1281 ms. Trace:
I20260709 12:12:11.366466 12639 rpcz_store.cc:276] 0709 12:12:10.084934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.188130 (+1103196us) service_pool.cc:224] Handling call
0709 12:12:11.212340 (+ 24210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.277080 (+ 64740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.365817 (+ 88737us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b54000a87d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:11.365983 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51243392}
I20260709 12:12:11.367117 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.367517 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1504) took 1315 ms. Trace:
I20260709 12:12:11.367780 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.368005 12646 rpcz_store.cc:276] 0709 12:12:10.052511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.120008 (+1067497us) service_pool.cc:224] Handling call
0709 12:12:11.120347 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.366083 (+245736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.367271 (+  1188us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:11.367465 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":528547200}
I20260709 12:12:11.369153 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.370111 14761 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.371393 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1527) took 1129 ms. Trace:
I20260709 12:12:11.371717 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.372162 14664 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.372933 14567 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:11.352368 14785 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.373459 14567 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.374264 14473 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.371781 12641 rpcz_store.cc:276] 0709 12:12:10.242611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:10.598886 (+356275us) service_pool.cc:224] Handling call
0709 12:12:11.371200 (+772314us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:11.371339 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1140779392}
I20260709 12:12:11.375897 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.377102 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.373391 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.378913 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.380697 14712 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.380208 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.313153 14607 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.382114 12642 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:11.384397 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.385970 14686 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.386641 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 203) took 1372 ms. Trace:
I20260709 12:12:11.371747 14695 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:11.387050 12495 rpcz_store.cc:276] 0709 12:12:10.014396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:10.014625 (+   229us) service_pool.cc:224] Handling call
0709 12:12:10.014968 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.110633 (+1095665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.386426 (+275793us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:11.386586 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2463909248}
I20260709 12:12:11.387785 14699 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.387912 14695 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.350605 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.351150 14782 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.352249 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1513) took 1269 ms. Trace:
I20260709 12:12:11.398666 12638 rpcz_store.cc:276] 0709 12:12:10.082528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:10.451091 (+368563us) service_pool.cc:224] Handling call
0709 12:12:11.352007 (+900916us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:11.352171 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1373823232}
I20260709 12:12:11.399391 12642 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:11.399742 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.353096 14744 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.390868 14695 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.353405 14751 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:11.404839 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:11.354599 14737 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:11.355484 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.349458 13256 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:12:11.349439 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1867) took 1180 ms. Trace:
I20260709 12:12:11.357401 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:11.395416 12497 raft_consensus.cc:2368] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:11.396811 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:11.401011 12646 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
W20260709 12:12:11.402480 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76660.1us
I20260709 12:12:11.449795 14780 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:11.450273 14780 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:11.403072 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.451699 12638 raft_consensus.cc:2376] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4. Re-sending same reply.
I20260709 12:12:11.404811 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.405251 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.405591 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:11.407079 13260 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.407377 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.454073 12567 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:11.458390 14774 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.821s	user 0.008s	sys 0.012s
I20260709 12:12:11.462394 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.463176 12638 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:11.463549 14714 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.407991 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.412142 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:11.467489 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55417.5us
I20260709 12:12:11.467585 14732 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.468302 14732 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.468626 14709 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:11.374518 14724 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:11.431680 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1599) took 1607 ms (client timeout 1589 ms). Trace:
W20260709 12:12:11.469718 12497 rpcz_store.cc:269] 0709 12:12:09.824938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:09.825091 (+   153us) service_pool.cc:224] Handling call
0709 12:12:11.350417 (+1525326us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:11.418051 (+ 67634us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3164032}
I20260709 12:12:11.470280 14739 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.432657 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.472324 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.417239 14634 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:11.472615 14633 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:11.473107 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.473331 14634 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:11.473809 14625 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:11.403497 14728 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.476522 14728 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:11.409440 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.435519 14567 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.477612 14742 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:11.478921 14732 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.440557 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.477696 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:11.480063 12563 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:11.441994 14771 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:11.442210 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.480502 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.480799 14771 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:11.480911 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.443099 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.481789 13256 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:11.443586 14694 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.482640 14703 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.482785 14694 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.483208 14703 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.484467 14635 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:12:11.485111 14694 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:11.436297 14794 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:12:11.481209 12564 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:11.482213 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.410182 14607 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.364621 14644 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:11.444808 14699 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:11.446046 14695 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.482286 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.447063 14772 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.481135 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:11.417577 12493 rpcz_store.cc:276] 0709 12:12:10.168805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:10.174990 (+  6185us) service_pool.cc:224] Handling call
0709 12:12:11.349204 (+1174214us) spinlock_profiling.cc:243] Waited 9.16 ms on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:11.349360 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21079296}
I20260709 12:12:11.470741 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.485605 14703 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:11.485733 14712 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:11.495869 14712 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.496650 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:11.453279 14702 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:11.505674 14702 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.506628 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:11.505533 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:11.507645 12492 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:11.508122 14702 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.461267 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1872) took 1282 ms. Trace:
I20260709 12:12:11.508652 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.508771 12496 rpcz_store.cc:276] 0709 12:12:10.179121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:10.179240 (+   119us) service_pool.cc:224] Handling call
0709 12:12:10.193793 (+ 14553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:10.232194 (+ 38401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.460980 (+1228786us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:11.461190 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86925568}
I20260709 12:12:11.469261 14709 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.510787 14709 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.474011 14634 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.406847 14751 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:11.489804 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.514406 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:11.514792 12571 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:11.514523 14751 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.516224 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:11.496928 14712 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.444855 14744 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:12:11.381431 14689 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.517014 12571 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:11.517168 14634 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.490964 14482 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:11.518347 14482 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.519482 14712 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.520480 14482 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:11.487823 14695 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.451234 14780 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:11.499791 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44822.5us
I20260709 12:12:11.488211 14733 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.521545 14780 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:11.522635 14695 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:11.522432 14643 consensus_queue.cc:1175] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 4 status { last_received { term: 3 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:12:11.491398 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:11.402967 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.524338 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.487491 14644 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:11.525059 12642 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:11.525981 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.526064 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.526762 12642 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:11.491096 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.527522 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.528756 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.502424 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.525738 14644 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.492434 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 191) took 1174 ms. Trace:
I20260709 12:12:11.493803 12646 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:11.473228 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.531808 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.522361 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.503808 14661 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:11.503707 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.544226 12497 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:12:11.504664 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107928us
I20260709 12:12:11.516330 14751 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:11.518709 14689 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.486584 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.548871 12564 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:11.520360 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.488273 14607 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.529111 14643 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:11.488060 14699 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:11.487923 12570 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
W20260709 12:12:11.532071 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58902.7us
I20260709 12:12:11.553066 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.553361 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.554771 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:11.503175 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64354.2us
I20260709 12:12:11.556185 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.557000 12638 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:11.558435 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.532948 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:12:11.530409 12642 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.494921 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.517190 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.502193 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.488381 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.537755 12646 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:11.538527 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:11.563199 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.545439 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:11.531076 12640 rpcz_store.cc:276] 0709 12:12:10.317893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.291025 (+973132us) service_pool.cc:224] Handling call
0709 12:12:11.291479 (+   454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:11.292399 (+   920us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:12:11.293183 (+   784us) log.cc:844] Serialized 14 byte log entry
0709 12:12:11.316253 (+ 23070us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:11.317278 (+  1025us) log.cc:844] Serialized 52 byte log entry
0709 12:12:11.317450 (+   172us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:11.318116 (+   666us) log.cc:844] Serialized 14 byte log entry
0709 12:12:11.318370 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.318451 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:11.492216 (+173765us) raft_consensus.cc:1685] finished
0709 12:12:11.492236 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:11.492374 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":374,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":94,"raft.queue_time_us":305,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":6470,"spinlock_wait_cycles":85376,"thread_start_us":21867,"threads_started":1,"wal-append.queue_time_us":22597}
I20260709 12:12:11.564563 13260 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:11.546442 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.563406 12644 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.565120 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.567480 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.550500 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.569715 12492 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:11.561326 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:11.571336 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.572327 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.572167 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:11.575382 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 187) took 1285 ms. Trace:
I20260709 12:12:11.575637 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.575806 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.576148 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.577338 13257 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:11.577433 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.577733 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.578748 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:11.579730 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.580889 12563 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:11.573868 12492 raft_consensus.cc:2368] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:11.494151 14454 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:11.522357 14733 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:11.574611 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:11.586580 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69492.7us
I20260709 12:12:11.588253 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.587147 12567 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:12:11.581820 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:11.591540 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:11.591303 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.525060 13256 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:11.596612 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:11.598482 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.603509 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.590524 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:11.607774 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61452.1us
I20260709 12:12:11.614061 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:11.547907 14742 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:11.616621 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.551829 14643 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:11.618461 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:11.616885 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122494us
I20260709 12:12:11.631182 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.626329 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.635224 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.578661 14644 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.587963 14567 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:11.594256 14596 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.568531 14695 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:11.629406 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:11.627029 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.569005 14661 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:11.579272 14686 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:11.625785 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:11.718888 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93288.5us
I20260709 12:12:11.636854 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.492131 14771 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.597750 14714 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.707455 12569 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:11.569355 14607 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.579399 14746 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:12:11.708586 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71888.1us
I20260709 12:12:11.659157 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:11.702536 12644 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:11.606884 14625 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:11.551548 14386 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:12:11.715544 12571 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:11.720254 12646 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:12:11.639227 14700 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:11.631992 14769 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.543013 14702 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
W20260709 12:12:11.723270 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93952.3us
I20260709 12:12:11.768168 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:11.617522 14728 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:11.774349 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:11.631572 14305 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:11.780972 14305 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:11.781647 14305 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:11.627435 14715 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.761272 14700 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:11.607774 14684 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.115s
I20260709 12:12:11.638091 14643 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:11.621321 14220 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:12:11.654829 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:11.816413 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:11.606423 14689 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:11.758811 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.773553 14746 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.823398 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:11.824677 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.749836 12563 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:11.621457 14766 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.618217 14716 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:11.578186 14577 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 12:12:11.832989 14577 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:11.627056 14782 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:11.763788 12569 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:11.764190 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.711090 14742 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.833482 14577 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:11.835310 14742 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.706661 14694 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20260709 12:12:11.836589 14577 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.838413 14577 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:11.617810 14473 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.149s
I20260709 12:12:11.628310 14641 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.120s
I20260709 12:12:11.607530 14754 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.141s
I20260709 12:12:11.727037 14664 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:11.759284 14596 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:11.841719 14596 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:11.796088 14695 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.756217 14770 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:11.798413 12643 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260709 12:12:11.772090 13253 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:12:11.780815 14756 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:11.827817 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.706382 14785 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260709 12:12:11.825531 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:11.846045 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.575801 12645 rpcz_store.cc:276] 0709 12:12:10.290085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.214189 (+924104us) service_pool.cc:224] Handling call
0709 12:12:11.245532 (+ 31343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:11.255931 (+ 10399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:11.255983 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:11.266563 (+ 10580us) log.cc:844] Serialized 26 byte log entry
0709 12:12:11.358284 (+ 91721us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:11.363440 (+  5156us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:11.415347 (+ 51907us) log.cc:844] Serialized 14 byte log entry
0709 12:12:11.551387 (+136040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.551460 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:11.551498 (+    38us) raft_consensus.cc:1685] finished
0709 12:12:11.575180 (+ 23682us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:11.575321 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141017,"prepare.queue_time_us":56430,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":154734,"raft.run_cpu_time_us":2432,"raft.run_wall_time_us":114341,"spinlock_wait_cycles":3762944,"thread_start_us":38597,"threads_started":1,"wal-append.queue_time_us":136582}
I20260709 12:12:11.828972 12566 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:11.810135 14715 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.842267 14782 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:11.787254 14744 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:11.596400 14633 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
W20260709 12:12:11.865599 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 228126us
I20260709 12:12:11.869146 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:11.823138 14689 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:11.847720 14770 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:11.843052 14695 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.844892 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:11.894387 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.811322 14700 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:11.861984 12566 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:12:11.859385 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 242853us
I20260709 12:12:11.882153 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.899689 12645 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.887261 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.901878 12641 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:11.903049 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1533) took 1545 ms. Trace:
W20260709 12:12:11.898978 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 272581us
I20260709 12:12:11.902843 12567 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:12:11.905972 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186269us
I20260709 12:12:11.906924 12641 rpcz_store.cc:276] 0709 12:12:10.360878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.901061 (+1540183us) service_pool.cc:224] Handling call
0709 12:12:11.901478 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.902332 (+   854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:11.902986 (+   654us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:11.922583 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.849155 14716 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.927412 12564 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:11.845329 14761 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.892963 14689 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:11.893297 14770 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:11.929317 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:11.927645 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:11.863310 14744 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:11.941360 12642 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 3: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:11.825229 14661 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:11.926827 14788 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:12:11.842921 14596 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:11.848647 14305 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:11.845850 14482 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
W20260709 12:12:11.963085 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47222.3us
W20260709 12:12:11.963532 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:11.963591 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.916366 14724 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:11.935751 14758 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.350s	user 0.023s	sys 0.077s
W20260709 12:12:11.982888 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:11.965483 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:11.969662 12493 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:11.962205 14744 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:11.939939 14703 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:11.980237 12497 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:12:11.980486 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:11.989979 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:11.927819 14641 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:12:11.992102 12492 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:12:11.966449 14699 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:11.969861 14775 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:12.004998 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:12.006565 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:12.006772 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:12.008186 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:11.999269 12495 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:12.009963 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:11.990845 14758 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.561s	user 0.023s	sys 0.078s
I20260709 12:12:12.017053 12494 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:12.014298 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:12.010720 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:12.027154 12642 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:11.961869 14664 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260709 12:12:11.966384 14663 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:12:11.961649 14754 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 12:12:11.979039 14756 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:11.993211 14709 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:11.971470 14386 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
W20260709 12:12:12.043001 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 173953us
I20260709 12:12:12.000349 14809 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:12.054041 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:11.991180 14724 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:12.059196 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 136707us
I20260709 12:12:11.961782 14761 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:12.062363 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94389.6us
I20260709 12:12:11.990971 14694 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 12:12:12.013713 14811 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:12.068874 12642 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:12.060582 14709 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:11.970551 14746 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.041592 14702 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:12.058148 14712 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.004s
W20260709 12:12:12.070148 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179807us
W20260709 12:12:12.115809 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:12.071355 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:12.019023 14220 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:12:12.007149 14703 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.098937 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1539) took 1513 ms. Trace:
I20260709 12:12:12.122767 12642 rpcz_store.cc:276] 0709 12:12:10.585952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.068282 (+1482330us) service_pool.cc:224] Handling call
0709 12:12:12.068656 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:12.098599 (+ 29943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:12.098868 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:12.108366 13253 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:12:12.031072 14634 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:12:12.098443 14811 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:12.101075 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:12.130573 13257 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:12.066628 14653 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:12:12.071065 14684 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:12.044142 14635 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
W20260709 12:12:12.113420 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 132976us
I20260709 12:12:12.115976 14702 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:12.142105 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:12.142921 14808 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:12:12.116789 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45583.2us
I20260709 12:12:12.145569 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:12.136535 14684 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:12.146396 14684 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:12.146932 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:12.148149 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:12.149263 14807 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:12:12.150529 14796 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260709 12:12:12.119879 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98364.4us
I20260709 12:12:12.151090 14724 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:12.057924 12570 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:12:12.111724 14709 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:12.110836 14715 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:12.132879 12564 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:12:12.132058 14811 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:12.062345 14782 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:12.153904 12570 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 1
I20260709 12:12:12.159336 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:12.167174 12564 pending_rounds.cc:85] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:12:12.123754 14818 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:12.066000 14716 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.193737 12570 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 3.2
W20260709 12:12:12.172334 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:12.194835 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:12.118808 14728 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:12:12.200882 12570 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:12.171455 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75688.4us
I20260709 12:12:12.151113 14807 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:12.170565 14809 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:12.171514 14699 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:12.193533 14780 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:12.213644 14715 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:12.214111 14695 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:12.245039 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:12.248356 12564 pending_rounds.cc:107] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:12:12.248687 12564 raft_consensus.cc:2889] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:12.250869 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 263621us
I20260709 12:12:12.236099 14809 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:12.234913 14739 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:12.243259 14715 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:12.262130 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117679us
I20260709 12:12:12.237519 14780 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:12.317080 12489 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:12.318914 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:12.327378 12565 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:12:12.154850 14702 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:12.332991 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 205282us
I20260709 12:12:12.252159 14473 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:12:12.254884 14695 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:12.360229 12564 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:12.196084 14756 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.316309 14715 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:12.340155 14454 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.734s	user 0.007s	sys 0.006s
I20260709 12:12:12.314016 14635 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:12.266711 14809 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:12.329950 14780 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:12.253543 14634 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.105s
I20260709 12:12:12.368762 14695 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:12.313494 14739 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:12.464967 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:12.492852 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:12.463936 14644 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.756s	user 0.011s	sys 0.005s
I20260709 12:12:12.339751 14816 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.310s
I20260709 12:12:12.455717 14454 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:12.632179 12644 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:12.491729 14695 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:12.629544 14386 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:12.673162 14386 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:12.677942 14810 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
W20260709 12:12:12.679646 14577 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.841s	user 0.010s	sys 0.005s
I20260709 12:12:12.681618 14386 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:12.517187 14699 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:12.654870 14818 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:12.655282 14827 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:12.683935 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 489168us
I20260709 12:12:12.728127 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:12.731513 14694 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 12:12:12.732059 14826 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:12.732610 14826 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:12.733312 14694 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:12.735031 14826 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:12.743604 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 390762us
I20260709 12:12:12.727424 14827 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:12.659207 14454 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.746553 14644 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:12.682516 14825 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.525s
W20260709 12:12:12.785431 14830 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.441s	user 0.001s	sys 0.001s
W20260709 12:12:12.786204 14833 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.441s	user 0.000s	sys 0.000s
W20260709 12:12:12.787165 14820 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.394s	user 0.000s	sys 0.001s
I20260709 12:12:12.790396 14776 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:12.770073 14834 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.446s
I20260709 12:12:12.817410 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 299) took 1209 ms. Trace:
I20260709 12:12:12.817982 12569 rpcz_store.cc:276] 0709 12:12:11.607615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.834616 (+227001us) service_pool.cc:224] Handling call
0709 12:12:11.835040 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:12.021135 (+186095us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:12.021166 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:12.040037 (+ 18871us) log.cc:844] Serialized 26 byte log entry
0709 12:12:12.115283 (+ 75246us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:12.115316 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:12.115381 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:12.817005 (+701624us) raft_consensus.cc:1685] finished
0709 12:12:12.817043 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:12.817314 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7600,"raft.queue_time_us":8137,"raft.run_cpu_time_us":839,"raft.run_wall_time_us":96947,"thread_start_us":219052,"threads_started":2,"wal-append.queue_time_us":175117}
W20260709 12:12:12.805112 14733 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.199s	user 0.011s	sys 0.007s
I20260709 12:12:12.846906 14840 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:12.847674 14840 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:12.850169 14689 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.885s	user 0.011s	sys 0.006s
I20260709 12:12:12.850718 14689 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:12.853426 14827 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.855216 14827 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:12.793448 14644 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:12.865839 14607 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:12.883322 14700 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.955s	user 0.016s	sys 0.000s
I20260709 12:12:12.889359 12638 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:12.838804 14703 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:12.879302 14643 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.028s	user 0.007s	sys 0.003s
W20260709 12:12:12.909283 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.017s	sys 0.006s
I20260709 12:12:12.823596 14776 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:12.887727 12639 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.266s	user 0.009s	sys 0.009s
I20260709 12:12:12.824209 14699 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.893074 14772 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
W20260709 12:12:12.946063 14751 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.386s	user 0.015s	sys 0.000s
W20260709 12:12:12.926831 14684 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.772s	user 0.011s	sys 0.002s
I20260709 12:12:12.918668 14739 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:12.893648 14607 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:12:12.824951 14818 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:12.946641 14837 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:12.910874 14754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.017s	sys 0.063s
W20260709 12:12:12.957173 14664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.007s	sys 0.015s
W20260709 12:12:12.983122 14664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.007s	sys 0.016s
I20260709 12:12:12.960002 14739 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:12.986805 14739 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:12.988685 14305 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.010s	sys 0.036s
W20260709 12:12:12.988795 12571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.015s	sys 0.007s
W20260709 12:12:12.989837 12571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.016s	sys 0.007s
W20260709 12:12:12.996804 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.020s	sys 0.009s
W20260709 12:12:12.997430 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.020s	sys 0.009s
W20260709 12:12:12.989372 14305 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.010s	sys 0.036s
I20260709 12:12:12.883961 14700 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:13.027791 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 307000us
I20260709 12:12:13.100164 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:12.931756 14643 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.941311 12639 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:13.023016 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 297) took 1520 ms. Trace:
W20260709 12:12:13.112161 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.013s	sys 0.011s
W20260709 12:12:13.112733 14596 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.011s	sys 0.023s
I20260709 12:12:12.957290 14684 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:12.957434 14751 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:12.957840 14840 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:13.139537 14473 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:13.116194 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.013s	sys 0.011s
I20260709 12:12:13.119771 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:12.982470 14754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.017s	sys 0.064s
W20260709 12:12:13.130424 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:13.130499 12637 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.010s	sys 0.010s
W20260709 12:12:13.135614 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.013s	sys 0.027s
I20260709 12:12:13.103118 14772 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.141706 12646 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:13.115887 12571 rpcz_store.cc:276] 0709 12:12:11.502550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.757973 (+255423us) service_pool.cc:224] Handling call
0709 12:12:11.779905 (+ 21932us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:11.780179 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:11.939201 (+159022us) log.cc:844] Serialized 14 byte log entry
0709 12:12:13.022345 (+1083144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:13.022459 (+   114us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:13.022538 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:13.022654 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:13.022879 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11776,"prepare.queue_time_us":61665,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":572725,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":60,"thread_start_us":1201200,"threads_started":3,"wal-append.queue_time_us":507554}
I20260709 12:12:13.163116 12494 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:13.162415 12495 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 12:12:13.310520 12637 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.014s	sys 0.015s
I20260709 12:12:13.163311 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:13.301813 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:13.311900 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:13.324182 14776 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:13.329380 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1760) took 1932 ms. Trace:
I20260709 12:12:13.012526 14825 consensus_queue.cc:1243] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260709 12:12:13.136421 14661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s	user 0.015s	sys 0.045s
I20260709 12:12:13.346351 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:13.325287 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 218) took 1327 ms. Trace:
I20260709 12:12:13.319416 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:13.344322 12570 rpcz_store.cc:276] 0709 12:12:11.397142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.595120 (+197978us) service_pool.cc:224] Handling call
0709 12:12:11.595434 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:12.215091 (+619657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:13.329051 (+1113960us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:13.329297 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002260224}
W20260709 12:12:13.338259 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.000s	sys 0.016s
W20260709 12:12:13.383920 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.000s	sys 0.019s
W20260709 12:12:13.384791 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 656721us
W20260709 12:12:13.348454 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.015s	sys 0.034s
W20260709 12:12:13.134229 14596 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.012s	sys 0.023s
W20260709 12:12:13.140162 14809 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.001s
I20260709 12:12:13.136499 14841 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:13.155805 14806 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.177s	user 0.003s	sys 0.021s
I20260709 12:12:13.372989 12493 rpcz_store.cc:276] 0709 12:12:11.997600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.856401 (+858801us) service_pool.cc:224] Handling call
0709 12:12:12.856848 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:12.857308 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:12.857342 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:12.859919 (+  2577us) log.cc:844] Serialized 26 byte log entry
0709 12:12:13.025600 (+165681us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:13.025680 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:13.025795 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:13.325037 (+299242us) raft_consensus.cc:1685] finished
0709 12:12:13.325059 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:13.325225 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6144,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152,"thread_start_us":114366,"threads_started":1,"wal-append.queue_time_us":431058}
W20260709 12:12:13.279315 14746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.003s	sys 0.013s
I20260709 12:12:13.160408 14831 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.382513 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:12.985343 14837 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.450240 12571 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:13.351785 14644 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:13.389077 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:13.385030 12570 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:13.397480 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:13.139864 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:13.139819 14454 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:13.325954 14863 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.169054 14689 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:13.372855 14661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.120s	user 0.030s	sys 0.065s
I20260709 12:12:13.161605 14473 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:13.253922 14800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.012s	sys 0.024s
I20260709 12:12:13.326382 14663 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.344249 14733 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:13.450973 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 306) took 1122 ms. Trace:
I20260709 12:12:13.387988 12494 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:12:13.284425 14607 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:13.345674 14567 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:13.458539 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1656) took 1117 ms. Trace:
I20260709 12:12:13.791708 12570 rpcz_store.cc:276] 0709 12:12:12.340868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.384164 (+1043296us) service_pool.cc:224] Handling call
0709 12:12:13.384709 (+   545us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:13.458240 (+ 73531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:13.458479 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:13.375306 14825 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.156339 14751 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.412825 14739 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:13.531569 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 142684us
I20260709 12:12:13.803510 14845 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.804281 14845 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:13.467700 14800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.021s	user 0.014s	sys 0.026s
I20260709 12:12:13.796406 14607 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:13.806053 14825 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:13.536623 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:13.549501 14844 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.790596 14733 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.807933 13255 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:13.808312 14844 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:13.813385 14771 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:13.521701 14454 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.817102 14454 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.521922 14863 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:13.819730 14814 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.754s	user 0.009s	sys 0.009s
I20260709 12:12:13.560019 14386 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.560168 14689 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:13.561285 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1608) took 2258 ms (client timeout 1945 ms). Trace:
I20260709 12:12:13.562538 14473 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:13.568549 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:12:13.562577 14842 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260709 12:12:13.564850 14644 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.574836 14482 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:13.577849 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:13.578505 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1605) took 2279 ms (client timeout 2024 ms). Trace:
I20260709 12:12:13.579520 14850 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:13.587224 14709 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.391s	user 0.012s	sys 0.001s
W20260709 12:12:13.590667 14848 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:12:13.595026 14661 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:13.597177 14625 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:13.599934 14684 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:13.443279 14841 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.601032 14776 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:13.602372 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1507) took 3548 ms (client timeout 1946 ms). Trace:
I20260709 12:12:13.603595 14837 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:13.458534 14831 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:13.604687 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1604) took 2306 ms (client timeout 2989 ms). Trace:
W20260709 12:12:13.606159 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1602) took 2308 ms (client timeout 1593 ms). Trace:
W20260709 12:12:13.606730 14715 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.165s	user 0.016s	sys 0.004s
I20260709 12:12:13.607342 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:13.609149 14761 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:13.611778 14823 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:13.612082 14780 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.142s	user 0.011s	sys 0.001s
W20260709 12:12:13.614353 14807 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.380s	user 0.015s	sys 0.000s
W20260709 12:12:13.615110 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1614) took 2099 ms (client timeout 1935 ms). Trace:
I20260709 12:12:13.414207 14809 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.465503 14664 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.316334 14772 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:13.403467 14821 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:13.655078 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1526) took 3428 ms. Trace:
W20260709 12:12:13.780383 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 194) took 2145 ms (client timeout 2125 ms). Trace:
I20260709 12:12:13.775327 14663 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:13.779470 12641 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:13.623947 12571 rpcz_store.cc:276] 0709 12:12:12.328682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.380347 (+1051665us) service_pool.cc:224] Handling call
0709 12:12:13.449889 (+ 69542us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:13.450712 (+   823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:13.450913 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:13.789455 12568 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:12:13.794391 14702 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.429s	user 0.005s	sys 0.011s
I20260709 12:12:13.550110 14849 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:13.551303 14851 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:12:13.824927 14386 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:13.825101 12638 rpcz_store.cc:269] 0709 12:12:11.304150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.575884 (+271734us) service_pool.cc:224] Handling call
0709 12:12:13.560994 (+1985110us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.561167 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2900636032}
W20260709 12:12:14.166646 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.032s	user 0.038s	sys 0.050s
W20260709 12:12:14.167395 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.032s	user 0.038s	sys 0.050s
W20260709 12:12:14.168774 12488 rpcz_store.cc:269] 0709 12:12:11.519832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.520053 (+   221us) service_pool.cc:224] Handling call
0709 12:12:13.614751 (+2094698us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.614997 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4549092608}
I20260709 12:12:14.167461 12637 rpcz_store.cc:276] 0709 12:12:10.190314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.189254 (+998940us) service_pool.cc:224] Handling call
0709 12:12:11.189814 (+   560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:13.339034 (+2149220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:13.339068 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:13.339110 (+    42us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:13.339156 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:13.609603 (+270447us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:13.618568 (+  8965us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:13.618767 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4350,"raft.queue_time_us":275662,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7164,"spinlock_wait_cycles":2877916928,"thread_start_us":892981,"threads_started":1}
W20260709 12:12:14.166587 12643 rpcz_store.cc:269] 0709 12:12:10.080787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:10.416796 (+336009us) service_pool.cc:224] Handling call
0709 12:12:13.602084 (+3185288us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.602253 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3156279040}
W20260709 12:12:14.166090 12646 rpcz_store.cc:269] 0709 12:12:11.635030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.763738 (+128708us) service_pool.cc:224] Handling call
0709 12:12:13.659443 (+1895705us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.780240 (+120797us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3005656832}
W20260709 12:12:14.164397 12640 rpcz_store.cc:269] 0709 12:12:11.298577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.567131 (+268554us) service_pool.cc:224] Handling call
0709 12:12:13.578261 (+2011130us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.578405 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3192191232}
W20260709 12:12:14.166322 12644 rpcz_store.cc:269] 0709 12:12:11.298238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.845740 (+547502us) service_pool.cc:224] Handling call
0709 12:12:13.604362 (+1758622us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.604587 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1805113088}
I20260709 12:12:13.824923 14689 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:14.192453 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.915s	user 0.036s	sys 0.063s
W20260709 12:12:14.202389 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.925s	user 0.036s	sys 0.063s
W20260709 12:12:13.800091 14695 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.103s	user 0.014s	sys 0.004s
W20260709 12:12:13.795673 14744 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.785s	user 0.016s	sys 0.008s
I20260709 12:12:14.196564 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 304) took 1160 ms. Trace:
W20260709 12:12:14.197649 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7762094819997856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20260709 12:12:14.229785 12536 connection.cc:256] RPC call timeout handler was delayed by -1.705109481999898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
I20260709 12:12:14.228566 13260 rpcz_store.cc:276] 0709 12:12:13.036487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.036649 (+   162us) service_pool.cc:224] Handling call
0709 12:12:13.577565 (+540916us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:14.196491 (+618926us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36937600}
W20260709 12:12:14.227239 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1553) took 2072 ms (client timeout 1946 ms). Trace:
W20260709 12:12:13.991065 14770 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 2.020s	user 0.031s	sys 0.020s
I20260709 12:12:14.172528 14844 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.051152 14700 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.214519 12640 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:14.070314 14868 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.167193 14482 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:14.089455 14832 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.355s	user 0.003s	sys 0.009s
I20260709 12:12:14.216840 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 220) took 1414 ms. Trace:
I20260709 12:12:14.109387 14858 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.164382 14766 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:12:14.231628 12638 rpcz_store.cc:269] 0709 12:12:12.154667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.226949 (+2072282us) service_pool.cc:211] Skipping call since client already timed out
0709 12:12:14.227144 (+   195us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:12:14.101475 14826 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:14.199429 12643 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
W20260709 12:12:14.198285 14403 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.322s	user 0.029s	sys 0.072s
I20260709 12:12:13.778697 14714 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:14.049160 14777 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.374475 12638 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:14.375267 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1548) took 2979 ms. Trace:
I20260709 12:12:14.375731 12638 rpcz_store.cc:276] 0709 12:12:11.395696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.373459 (+2977763us) service_pool.cc:224] Handling call
0709 12:12:14.374102 (+   643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.374972 (+   870us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.375206 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.377511 12638 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:14.213428 12644 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:14.379714 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1609) took 3051 ms. Trace:
I20260709 12:12:14.165308 14625 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:14.148051 14643 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:14.240367 14716 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.462s	user 0.016s	sys 0.038s
I20260709 12:12:14.193168 14821 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.311839 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1606) took 3010 ms. Trace:
I20260709 12:12:14.193739 14807 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:14.319434 14852 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:14.234499 12571 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
W20260709 12:12:14.305079 12491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.004s	user 0.022s	sys 0.043s
W20260709 12:12:14.291725 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:14.404003 12491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.121s	user 0.023s	sys 0.044s
I20260709 12:12:14.326507 12489 rpcz_store.cc:276] 0709 12:12:12.802460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.802703 (+   243us) service_pool.cc:224] Handling call
0709 12:12:12.803255 (+   552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:12.803807 (+   552us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:12.803848 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:12.804460 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:12:12.988173 (+183713us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:12.988242 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:12.988357 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:14.216568 (+1228211us) raft_consensus.cc:1685] finished
0709 12:12:14.216597 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:14.216775 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1350,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":16267,"thread_start_us":183179,"threads_started":1,"wal-append.queue_time_us":598898}
W20260709 12:12:14.331864 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.059s	user 0.019s	sys 0.099s
I20260709 12:12:14.331092 14766 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:14.336601 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1541) took 3710 ms. Trace:
I20260709 12:12:14.192903 14850 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.413981 14482 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:13.826618 14644 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.218108 14596 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.219756 14689 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:14.227468 14684 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:14.198342 14811 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.985s	user 0.005s	sys 0.016s
I20260709 12:12:14.378549 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1551) took 2982 ms. Trace:
I20260709 12:12:14.449288 12638 rpcz_store.cc:276] 0709 12:12:11.396370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.376715 (+2980345us) service_pool.cc:224] Handling call
0709 12:12:14.377103 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.378224 (+  1121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.378481 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.451404 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:14.452497 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 195) took 2558 ms. Trace:
I20260709 12:12:14.193367 14849 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.380208 12644 rpcz_store.cc:276] 0709 12:12:11.328377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.212660 (+2884283us) service_pool.cc:224] Handling call
0709 12:12:14.213159 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.379300 (+166141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.379632 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.192541 14664 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.192771 14661 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.395143 12640 rpcz_store.cc:276] 0709 12:12:11.301868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.213910 (+2912042us) service_pool.cc:224] Handling call
0709 12:12:14.214269 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.311138 (+ 96869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.311754 (+   616us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.290167 14825 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:14.314904 14782 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.560s	user 0.042s	sys 0.094s
I20260709 12:12:14.234237 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1542) took 3607 ms. Trace:
I20260709 12:12:14.312768 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 307) took 1199 ms. Trace:
I20260709 12:12:14.181473 14780 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:14.164325 12639 rpcz_store.cc:269] 0709 12:12:11.297570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.367481 (+ 69911us) service_pool.cc:224] Handling call
0709 12:12:13.605879 (+2238398us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:13.606047 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":501566336}
I20260709 12:12:14.510177 14751 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:14.511435 12639 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:14.512578 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1554) took 2357 ms. Trace:
W20260709 12:12:14.374162 14403 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.516s	user 0.029s	sys 0.073s
I20260709 12:12:14.193027 14715 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:14.321426 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.498s	user 0.006s	sys 0.054s
W20260709 12:12:14.514828 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.691s	user 0.007s	sys 0.054s
I20260709 12:12:14.515722 14843 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:14.516404 14715 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.182068 14709 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.517133 14715 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:14.517514 13253 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:14.519775 14709 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.192476 14776 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:14.411706 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:14.521029 12536 connection.cc:256] RPC call timeout handler was delayed by -1.301951329999838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2989s for a response
W20260709 12:12:14.521430 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2285061929999301s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
W20260709 12:12:14.521772 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0964563150000686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
W20260709 12:12:14.522064 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0957137569998849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260709 12:12:14.327579 14686 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:14.523052 14686 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.524101 14843 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.332695 14864 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.526084 14864 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.526772 14850 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.527392 14850 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.527812 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1659) took 1813 ms. Trace:
I20260709 12:12:14.527921 14850 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.528700 12571 rpcz_store.cc:276] 0709 12:12:12.714347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.198081 (+1483734us) service_pool.cc:224] Handling call
0709 12:12:14.220638 (+ 22557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.412401 (+191763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.527454 (+115053us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:14.527723 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31226112}
I20260709 12:12:14.191774 14702 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:14.413962 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.141s	user 0.020s	sys 0.099s
I20260709 12:12:14.392063 14868 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.414229 12643 rpcz_store.cc:276] 0709 12:12:10.626498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.198666 (+3572168us) service_pool.cc:224] Handling call
0709 12:12:14.199210 (+   544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.336205 (+136995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.336522 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.556818 12643 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:12:14.315424 14851 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:12:14.290938 14856 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:14.227837 14695 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.335934 14826 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:14.565966 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.155s	user 0.017s	sys 0.033s
I20260709 12:12:14.337858 14607 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:14.424518 12461 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.946s (SENT)
I20260709 12:12:14.566669 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:14.567656 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143206us
I20260709 12:12:13.822506 14567 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.432546 14852 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.320678 14770 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.232558 14744 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.211042 14823 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.399832 14714 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.396057 14777 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.431149 14733 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:14.450650 14447 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:14.453061 12638 rpcz_store.cc:276] 0709 12:12:11.894524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.450615 (+2556091us) service_pool.cc:224] Handling call
0709 12:12:14.451132 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.452109 (+   977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.452408 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.594794 12564 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:14.470603 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.534s	user 0.026s	sys 0.062s
W20260709 12:12:14.595794 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.660s	user 0.026s	sys 0.063s
I20260709 12:12:14.599318 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.599421 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.453727 14596 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.414707 14766 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.600478 12494 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:14.395113 14858 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.452004 14482 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:14.495446 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1536) took 4050 ms. Trace:
I20260709 12:12:14.498427 12637 rpcz_store.cc:276] 0709 12:12:10.626826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.194479 (+3567653us) service_pool.cc:224] Handling call
0709 12:12:14.233436 (+ 38957us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.233759 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.233798 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:14.233862 (+    64us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:14.233911 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.233991 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:14.234151 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.611708 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1652) took 2290 ms. Trace:
W20260709 12:12:14.499046 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.155s	user 0.046s	sys 0.088s
W20260709 12:12:14.612617 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.270s	user 0.046s	sys 0.088s
I20260709 12:12:14.612128 12564 rpcz_store.cc:276] 0709 12:12:12.321289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.713471 (+392182us) service_pool.cc:224] Handling call
0709 12:12:12.730382 (+ 16911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.595354 (+1864972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.611411 (+ 16057us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:14.611639 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4286918528}
I20260709 12:12:14.501456 14454 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:14.504033 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1538) took 3964 ms. Trace:
I20260709 12:12:14.530402 14821 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.530797 14776 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.525612 14686 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.324399 14844 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:14.622807 14852 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.623129 14844 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.624240 14809 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.524883 14843 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:14.544049 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:14.528544 14864 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.499428 13254 rpcz_store.cc:276] 0709 12:12:13.090775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.101122 (+ 10347us) service_pool.cc:224] Handling call
0709 12:12:13.607075 (+505953us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:14.290448 (+683373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39937920}
I20260709 12:12:14.547660 14386 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.548255 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 301) took 1545 ms. Trace:
I20260709 12:12:14.622844 14850 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.548523 14841 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.537758 14715 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.642172 14844 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.642635 14702 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.643855 14715 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.554062 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1660) took 1835 ms. Trace:
I20260709 12:12:14.644433 12571 rpcz_store.cc:276] 0709 12:12:12.718793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.552835 (+1834042us) service_pool.cc:224] Handling call
0709 12:12:14.553310 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.553574 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.553613 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:14.553700 (+    87us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:14.553746 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.553826 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:14.553981 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.557492 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1555) took 2402 ms. Trace:
I20260709 12:12:14.557955 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1765) took 3122 ms. Trace:
I20260709 12:12:14.645900 12643 rpcz_store.cc:276] 0709 12:12:12.155403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.556064 (+2400661us) service_pool.cc:224] Handling call
0709 12:12:14.556432 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.557216 (+   784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.557430 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.646179 12568 rpcz_store.cc:276] 0709 12:12:11.411061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.604813 (+193752us) service_pool.cc:224] Handling call
0709 12:12:14.522139 (+2917326us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:14.533711 (+ 11572us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4345879936}
W20260709 12:12:14.500799 14782 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.743s	user 0.043s	sys 0.094s
I20260709 12:12:14.513053 12639 rpcz_store.cc:276] 0709 12:12:12.155083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.510566 (+2355483us) service_pool.cc:224] Handling call
0709 12:12:14.511130 (+   564us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.512148 (+  1018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.512496 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.454466 14811 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.630520 14831 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.565614 12572 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20260709 12:12:14.566362 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.155s	user 0.017s	sys 0.033s
I20260709 12:12:14.654847 14772 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:14.655349 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.244e+06us
I20260709 12:12:14.346478 14863 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.658150 14827 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:14.662575 14849 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.663765 14644 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.664141 13260 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:14.665759 14695 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.666639 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:14.667677 14714 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.668414 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:14.671535 14766 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:14.673063 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:14.580552 12642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.138s	user 0.007s	sys 0.023s
I20260709 12:12:14.447345 14625 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.674636 14625 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:14.454141 14684 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:14.678877 14861 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
I20260709 12:12:14.679942 14684 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:14.591300 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.572s (SENT) [suppressed 34 similar messages]
I20260709 12:12:14.561543 14845 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:14.609113 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.611407 12645 rpcz_store.cc:276] 0709 12:12:10.445042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.903265 (+1458223us) service_pool.cc:224] Handling call
0709 12:12:11.935541 (+ 32276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:12.043166 (+107625us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:12.053964 (+ 10798us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:12.054793 (+   829us) log.cc:844] Serialized 26 byte log entry
0709 12:12:12.113969 (+ 59176us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:12.114015 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:12.114100 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:13.646314 (+1532214us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:14.465871 (+819557us) raft_consensus.cc:1685] finished
0709 12:12:14.465903 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:14.495341 (+ 29438us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":104503,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":11816,"spinlock_wait_cycles":17278336,"thread_start_us":147099,"threads_started":2,"wal-append.queue_time_us":439459}
I20260709 12:12:14.682543 12645 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:12:14.683182 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1560) took 2524 ms. Trace:
I20260709 12:12:14.683483 12645 rpcz_store.cc:276] 0709 12:12:12.158594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.681927 (+2523333us) service_pool.cc:224] Handling call
0709 12:12:14.682299 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:14.682918 (+   619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.683120 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:14.615886 12641 rpcz_store.cc:276] 0709 12:12:10.539315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:11.916521 (+1377206us) service_pool.cc:224] Handling call
0709 12:12:13.779230 (+1862709us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.172022 (+392792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.503584 (+331562us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400278b50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:14.503952 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4282773248}
I20260709 12:12:14.609220 14607 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.585502 14807 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:14.615394 14447 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.615376 14733 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.622155 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.624348 14661 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:14.391029 14716 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.613s	user 0.017s	sys 0.038s
I20260709 12:12:14.629468 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.729875 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.730701 12493 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:12:14.731789 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.732446 12493 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:14.630003 12646 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:14.630003 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.630924 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.630470 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:14.630505 14751 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.739482 14751 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.641829 13253 rpcz_store.cc:276] 0709 12:12:13.002987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.004728 (+  1741us) service_pool.cc:224] Handling call
0709 12:12:13.005290 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.517946 (+1512656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.547895 (+ 29949us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:14.548132 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3477197696}
I20260709 12:12:14.653641 14811 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.607795 14785 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.741925 14663 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.742142 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1653) took 2416 ms. Trace:
I20260709 12:12:14.742110 14785 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.685076 14878 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.743090 12565 rpcz_store.cc:276] 0709 12:12:12.326014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.710250 (+384236us) service_pool.cc:224] Handling call
0709 12:12:12.714974 (+  4724us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:12.969920 (+254946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:12.969952 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:12.970716 (+   764us) log.cc:844] Serialized 52 byte log entry
0709 12:12:14.168233 (+1197517us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:14.168455 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.168578 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:14.741915 (+573337us) raft_consensus.cc:1685] finished
0709 12:12:14.741934 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:14.742079 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":379515,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":10940,"thread_start_us":1287387,"threads_started":2,"wal-append.queue_time_us":1474945}
I20260709 12:12:14.744431 14878 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:14.744844 14785 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:14.618922 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.745950 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:14.621436 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:14.748060 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1654) took 2421 ms. Trace:
I20260709 12:12:14.577739 14856 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:14.687894 12642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.246s	user 0.007s	sys 0.024s
W20260709 12:12:14.717196 14851 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:12:14.719306 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.738749 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.717430 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:14.762874 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:14.767411 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:14.748554 12572 rpcz_store.cc:276] 0709 12:12:12.326350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.710125 (+383775us) service_pool.cc:224] Handling call
0709 12:12:12.729902 (+ 19777us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:14.655269 (+1925367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.747757 (+ 92488us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:14.747974 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4221006592}
I20260709 12:12:14.685906 14876 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:14.825593 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 108237us
I20260709 12:12:14.824947 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:14.890266 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:14.826718 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1540) took 4200 ms. Trace:
W20260709 12:12:14.815405 14874 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:12:14.741982 14664 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.754518 14724 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:14.815480 14777 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:14.761803 14733 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:15.064314 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.700s	user 0.005s	sys 0.046s
I20260709 12:12:15.054183 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:15.121591 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.758s	user 0.006s	sys 0.050s
W20260709 12:12:15.122005 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.95861e+06us
W20260709 12:12:14.999686 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 121) took 13761 ms (client timeout 15000 ms). Trace:
W20260709 12:12:15.089417 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.646s	user 0.013s	sys 0.053s
W20260709 12:12:15.126150 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 236042us
I20260709 12:12:14.888782 14663 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:15.126883 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.673s	user 0.022s	sys 0.035s
W20260709 12:12:15.127345 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.674s	user 0.022s	sys 0.035s
I20260709 12:12:15.127701 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:14.718720 14826 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.119449 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:15.126469 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.684s	user 0.015s	sys 0.054s
I20260709 12:12:15.137297 12495 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:15.129109 12641 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:15.056566 14868 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:15.131021 13262 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:14.822276 14780 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.894407 12642 rpcz_store.cc:276] 0709 12:12:10.626064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.124356 (+1498292us) service_pool.cc:224] Handling call
0709 12:12:12.124931 (+   575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:12.134163 (+  9232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:12.134201 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:12.134948 (+   747us) log.cc:844] Serialized 26 byte log entry
0709 12:12:12.874558 (+739610us) spinlock_profiling.cc:243] Waited 949 us on lock 0x7b180046ada0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:12:13.010748 (+136190us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:13.407375 (+396627us) log.cc:844] Serialized 14 byte log entry
0709 12:12:14.754323 (+1346948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.754780 (+   457us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:14.754869 (+    89us) raft_consensus.cc:1685] finished
0709 12:12:14.826016 (+ 71147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:14.826653 (+   637us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21450,"prepare.queue_time_us":1230154,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4809,"raft.queue_time_us":225652,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":19993,"spinlock_wait_cycles":2182528,"thread_start_us":1842323,"threads_started":3,"wal-append.queue_time_us":1373585,"wal-append.run_cpu_time_us":2632,"wal-append.run_wall_time_us":1013252}
I20260709 12:12:15.143551 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 198) took 2744 ms. Trace:
I20260709 12:12:15.154984 12641 rpcz_store.cc:276] 0709 12:12:12.399137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.685578 (+2286441us) service_pool.cc:224] Handling call
0709 12:12:14.685896 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.143091 (+457195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.143481 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:15.156724 12641 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:15.122064 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
W20260709 12:12:15.125705 12392 rpcz_store.cc:269] 0709 12:12:01.214452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.214680 (+   228us) service_pool.cc:224] Handling call
0709 12:12:14.975338 (+13760658us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:09.674951 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:09.689724 (+ 14773us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:09.689791 (+    67us) write_op.cc:435] Acquired schema lock
0709 12:12:09.689818 (+    27us) tablet.cc:662] Decoding operations
0709 12:12:09.691374 (+  1556us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:09.691451 (+    77us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:09.691497 (+    46us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:12:09.693025 (+  1528us) tablet.cc:701] Row locks acquired
0709 12:12:09.693049 (+    24us) write_op.cc:260] PREPARE: finished
0709 12:12:09.693237 (+   188us) write_op.cc:270] Start()
0709 12:12:09.693385 (+   148us) write_op.cc:276] Timestamp: P: 1783599129693175 usec, L: 0
0709 12:12:09.693407 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:12:09.874717 (+181310us) log.cc:844] Serialized 5516 byte log entry
0709 12:12:10.383184 (+508467us) op_driver.cc:464] REPLICATION: finished
0709 12:12:11.444450 (+1061266us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf9229ef0
0709 12:12:11.979173 (+534723us) write_op.cc:301] APPLY: starting
0709 12:12:11.994680 (+ 15507us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:12.071293 (+ 76613us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:12.073962 (+  2669us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:12.360082 (+286120us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:12.360486 (+   404us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 12:12:12.360561 (+    75us) write_op.cc:312] APPLY: finished
0709 12:12:12.361173 (+   612us) log.cc:844] Serialized 101 byte log entry
0709 12:12:13.806052 (+1444879us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:13.806282 (+   230us) write_op.cc:454] Released schema lock
0709 12:12:13.806364 (+    82us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:13.806868 (+   504us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:13.806904 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2905406,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":1569392,"apply.run_cpu_time_us":49498,"apply.run_wall_time_us":1836294,"num_ops":14,"prepare.queue_time_us":284358,"prepare.run_cpu_time_us":21679,"prepare.run_wall_time_us":442916,"raft.queue_time_us":134034,"raft.run_cpu_time_us":105720,"raft.run_wall_time_us":1685624,"replication_time_us":671659,"spinlock_wait_cycles":58916608,"thread_start_us":3387276,"threads_started":5,"wal-append.queue_time_us":2000043,"wal-append.run_cpu_time_us":16206,"wal-append.run_wall_time_us":1204369}]]}
W20260709 12:12:15.011068 14875 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:12:15.169080 12642 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:12:15.174535 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:15.108734 14808 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.268s	user 0.019s	sys 0.034s
I20260709 12:12:15.196892 12497 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:12:15.205538 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78045.6us
I20260709 12:12:15.207348 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:15.187713 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:15.291815 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:15.212694 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:14.895542 14661 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:15.203478 14807 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:14.968200 14876 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.214026 14386 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:12:15.218719 14859 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.252s	user 0.000s	sys 0.012s
I20260709 12:12:15.308336 14807 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.203715 14826 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:15.311086 14807 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:15.223037 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1543) took 4595 ms. Trace:
I20260709 12:12:15.313850 12646 rpcz_store.cc:276] 0709 12:12:10.627135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.198862 (+3571727us) service_pool.cc:224] Handling call
0709 12:12:14.200068 (+  1206us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.733351 (+533283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.222714 (+489363us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:15.222961 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":987891072}
I20260709 12:12:15.227092 14823 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:15.229050 14770 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:15.229748 14644 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:15.231153 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1661) took 2127 ms. Trace:
I20260709 12:12:15.234133 14700 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:15.235013 14845 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:15.236773 14482 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:15.237311 12493 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:15.239002 12492 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:15.240429 12494 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
W20260709 12:12:15.241473 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105921us
I20260709 12:12:15.243251 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:15.244285 14785 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:15.245209 14879 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:15.246187 14596 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:15.246737 14744 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:15.148075 14843 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:14.747795 14607 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:14.976080 14878 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:14.991186 14447 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:15.057967 14856 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:15.236318 14567 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:15.250721 14625 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.576s	user 0.006s	sys 0.006s
I20260709 12:12:15.135757 14724 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.183449 14811 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:15.295150 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:15.397961 14746 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:15.261765 14751 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:15.278684 14766 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.606s	user 0.009s	sys 0.001s
W20260709 12:12:15.288723 14808 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.468s	user 0.020s	sys 0.037s
I20260709 12:12:15.297645 14664 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:12:15.316802 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:15.262633 14818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.941s	user 0.018s	sys 0.058s
I20260709 12:12:15.361855 12564 rpcz_store.cc:276] 0709 12:12:13.103501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.640996 (+1537495us) service_pool.cc:224] Handling call
0709 12:12:14.764030 (+123034us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.119950 (+355920us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.119999 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:15.120100 (+   101us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:15.120158 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.230853 (+110695us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:15.230908 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:15.231081 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315606272}
W20260709 12:12:15.349736 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.015s	user 0.028s	sys 0.049s
W20260709 12:12:15.370570 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 127411us
W20260709 12:12:15.434069 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.105s	user 0.029s	sys 0.049s
I20260709 12:12:15.387204 14482 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.437907 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:15.438731 12646 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
W20260709 12:12:15.415506 14403 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s	user 0.023s	sys 0.008s
W20260709 12:12:15.390501 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:15.267031 13258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.489s	user 0.013s	sys 0.036s
W20260709 12:12:15.318200 14756 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.228s	user 0.023s	sys 0.030s
I20260709 12:12:15.401206 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:15.405206 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1655) took 3071 ms. Trace:
I20260709 12:12:15.439396 14733 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:15.461262 12569 rpcz_store.cc:276] 0709 12:12:12.326613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.820231 (+493618us) service_pool.cc:224] Handling call
0709 12:12:12.820707 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:12.821147 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:12.821187 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:12.821986 (+   799us) log.cc:844] Serialized 26 byte log entry
0709 12:12:14.515265 (+1693279us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:14.515321 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:14.515415 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:15.397935 (+882520us) raft_consensus.cc:1685] finished
0709 12:12:15.397966 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:15.398184 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24007,"raft.queue_time_us":452708,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":54845,"thread_start_us":1691609,"threads_started":1,"wal-append.queue_time_us":1789962}
I20260709 12:12:15.296088 14663 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.105s
W20260709 12:12:15.407749 12461 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.015s	sys 0.016s
W20260709 12:12:15.408744 12640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s	user 0.025s	sys 0.037s
W20260709 12:12:15.467589 12461 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.015s	sys 0.016s
W20260709 12:12:15.467700 12640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.026s	sys 0.037s
W20260709 12:12:15.467734 12566 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.827s	user 0.008s	sys 0.011s
W20260709 12:12:15.468384 12566 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.008s	sys 0.011s
W20260709 12:12:15.468086 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 584174us
I20260709 12:12:15.420492 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:15.264472 14699 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.670s	user 0.030s	sys 0.062s
I20260709 12:12:15.421175 12641 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:12:15.470767 12641 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:12:15.449049 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:15.343923 14386 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 12:12:15.389572 14882 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:15.424784 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:15.408074 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:15.411572 14876 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:15.388965 14785 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.440055 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:15.482767 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.003s	sys 0.018s
I20260709 12:12:15.364432 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:15.415618 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:15.385730 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.187s	user 0.015s	sys 0.011s
W20260709 12:12:15.484434 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.286s	user 0.015s	sys 0.012s
I20260709 12:12:15.386260 14700 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:15.436570 14818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.122s	user 0.019s	sys 0.058s
I20260709 12:12:15.435019 14840 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:15.443001 14447 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:15.431879 14567 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:15.566079 12567 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.825s	user 0.014s	sys 0.011s
I20260709 12:12:15.465461 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:15.468794 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 197) took 3573 ms. Trace:
W20260709 12:12:15.416503 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.016s	sys 0.016s
I20260709 12:12:15.424650 14811 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.471138 12641 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:12:15.339943 14684 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.091s
W20260709 12:12:15.457319 13258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.679s	user 0.014s	sys 0.041s
I20260709 12:12:15.475219 12565 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:15.451843 14772 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:15.460265 14756 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.370s	user 0.024s	sys 0.031s
W20260709 12:12:15.457804 14403 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.024s	sys 0.008s
I20260709 12:12:15.578023 14772 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.480655 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:15.481756 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:15.579712 14700 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:15.580240 12569 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:15.580694 14607 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:15.567608 14840 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:15.474933 14882 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:15.585952 14882 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:15.430097 14746 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:15.586699 14482 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:15.483111 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.003s	sys 0.018s
I20260709 12:12:15.573572 14567 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:15.472831 14699 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.877s	user 0.030s	sys 0.070s
I20260709 12:12:15.483111 12646 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:15.589978 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:15.591142 12646 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:15.592252 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:15.592912 12646 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:15.546998 14876 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:15.594537 13254 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:15.595219 14876 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:15.597024 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:15.598065 12646 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:15.523497 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159132us
I20260709 12:12:15.524734 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:15.532187 12642 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:12:15.533543 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47847.9us
W20260709 12:12:15.476802 14709 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.001s	sys 0.018s
W20260709 12:12:15.536675 13261 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.417s	user 0.017s	sys 0.004s
W20260709 12:12:15.559190 12644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s	user 0.017s	sys 0.021s
W20260709 12:12:15.602748 12644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.017s	sys 0.021s
I20260709 12:12:15.525539 14845 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:15.439729 14856 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:15.606351 12569 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:15.570041 12641 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:15.607235 14888 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:12:15.570909 12567 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.830s	user 0.015s	sys 0.011s
I20260709 12:12:15.568913 12640 rpcz_store.cc:276] 0709 12:12:11.895100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.490038 (+2594938us) service_pool.cc:224] Handling call
0709 12:12:14.490454 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.490710 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:14.491061 (+   351us) log.cc:844] Serialized 14 byte log entry
0709 12:12:15.468405 (+977344us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:15.468476 (+    71us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:15.468522 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.468596 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:15.468731 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":157,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2428,"raft.queue_time_us":906,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4894,"thread_start_us":976905,"threads_started":1}
I20260709 12:12:15.578356 14403 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:15.477907 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.015s	sys 0.006s
W20260709 12:12:15.609990 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.015s	sys 0.006s
W20260709 12:12:15.430655 14689 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.969s	user 0.013s	sys 0.000s
I20260709 12:12:15.594616 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:15.520908 12609 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.016s	sys 0.011s
W20260709 12:12:15.618095 12609 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.016s	sys 0.011s
W20260709 12:12:15.618625 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.02726e+06us
I20260709 12:12:15.602665 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:15.522166 12491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.007s	sys 0.021s
W20260709 12:12:15.620299 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.007s	sys 0.021s
I20260709 12:12:15.482477 13255 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:15.657821 14607 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:15.658396 14607 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:15.601365 13259 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:15.569235 14889 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.107s
W20260709 12:12:15.604056 14709 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.002s	sys 0.018s
I20260709 12:12:15.597282 14567 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:15.606048 14856 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:15.569823 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.047s	user 0.017s	sys 0.016s
I20260709 12:12:15.569393 14447 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.620360 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:15.603386 13261 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s	user 0.018s	sys 0.005s
W20260709 12:12:16.230995 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.000s	sys 0.008s
I20260709 12:12:16.061969 12565 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:16.563218 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1670) took 1377 ms (client timeout 1594 ms). Trace:
W20260709 12:12:16.437237 12488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.007s	user 0.010s	sys 0.005s
W20260709 12:12:16.564466 12488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.139s	user 0.011s	sys 0.006s
W20260709 12:12:16.207450 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.001s	sys 0.017s
W20260709 12:12:16.565335 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.006s	sys 0.026s
W20260709 12:12:16.280479 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.006s	sys 0.015s
W20260709 12:12:15.690148 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:16.578635 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s	user 0.012s	sys 0.020s
W20260709 12:12:15.929157 12638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.041s	sys 0.058s
I20260709 12:12:15.694200 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:16.606523 12564 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:16.202556 12637 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.006s	sys 0.013s
W20260709 12:12:16.607402 12637 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.019s	user 0.016s	sys 0.018s
W20260709 12:12:16.563215 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1677) took 1302 ms (client timeout 1735 ms). Trace:
W20260709 12:12:16.608201 12569 rpcz_store.cc:269] 0709 12:12:15.260622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.463235 (+202613us) service_pool.cc:224] Handling call
0709 12:12:16.563133 (+1099898us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:15.703368 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.578518 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 472) took 1145 ms. Trace:
I20260709 12:12:16.569268 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 308) took 3461 ms. Trace:
W20260709 12:12:16.578126 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.11266e+06us
W20260709 12:12:16.610821 12638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.354s	user 0.049s	sys 0.080s
W20260709 12:12:15.707978 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:16.612236 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:16.562664 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.000s	sys 0.012s
I20260709 12:12:16.609856 13259 rpcz_store.cc:276] 0709 12:12:15.433156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.457489 (+ 24333us) service_pool.cc:224] Handling call
0709 12:12:16.578442 (+1120953us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:16.616171 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57596.4us
W20260709 12:12:16.283535 14808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s	user 0.013s	sys 0.011s
I20260709 12:12:16.558421 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1629) took 1928 ms. Trace:
W20260709 12:12:16.564116 12565 rpcz_store.cc:269] 0709 12:12:15.185907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.422409 (+236502us) service_pool.cc:224] Handling call
0709 12:12:16.563124 (+1140715us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:16.541863 12640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.000s	sys 0.020s
W20260709 12:12:16.632992 12640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.000s	sys 0.026s
W20260709 12:12:16.210916 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.005s	sys 0.014s
W20260709 12:12:15.844497 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.003s	sys 0.019s
W20260709 12:12:16.634872 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.340s	user 0.009s	sys 0.023s
W20260709 12:12:16.634773 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.069s	user 0.012s	sys 0.024s
W20260709 12:12:16.635358 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.34083e+06us
W20260709 12:12:16.635566 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.07037e+06us
I20260709 12:12:16.638824 14845 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:16.639554 14845 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:15.704257 14482 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:16.643055 14845 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.645052 14826 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:16.645754 14826 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:16.646317 14826 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:15.668239 14716 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:16.650811 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.03044e+06us
I20260709 12:12:16.652817 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:15.610782 14403 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:16.659785 14403 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.661362 14403 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:15.844244 12639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.032s	sys 0.068s
W20260709 12:12:16.665505 12639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.417s	user 0.037s	sys 0.072s
I20260709 12:12:16.671494 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 305) took 3634 ms. Trace:
W20260709 12:12:16.275262 12644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.005s	sys 0.007s
I20260709 12:12:15.658690 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.678692 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 474) took 1195 ms. Trace:
I20260709 12:12:16.679178 13255 rpcz_store.cc:276] 0709 12:12:15.483271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.658336 (+175065us) service_pool.cc:224] Handling call
0709 12:12:16.678603 (+1020267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.680305 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.680552 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1561) took 4451 ms. Trace:
I20260709 12:12:16.558754 14305 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:16.682046 14305 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.602203 14567 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:16.684130 14567 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.619294 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.686444 12569 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:12:16.686955 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1793) took 1344 ms (client timeout 1708 ms). Trace:
W20260709 12:12:16.687352 12569 rpcz_store.cc:269] 0709 12:12:15.342356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.618902 (+1276546us) service_pool.cc:224] Handling call
0709 12:12:16.686866 (+ 67964us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.688494 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.689361 12569 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:16.609833 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:16.689952 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1792) took 1389 ms (client timeout 1566 ms). Trace:
W20260709 12:12:16.690361 12572 rpcz_store.cc:269] 0709 12:12:15.300814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.436152 (+135338us) service_pool.cc:224] Handling call
0709 12:12:16.689864 (+1253712us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.691506 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.692348 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1679) took 1425 ms. Trace:
I20260709 12:12:16.692689 12572 rpcz_store.cc:276] 0709 12:12:15.266661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.691128 (+1424467us) service_pool.cc:224] Handling call
0709 12:12:16.692287 (+  1159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.621441 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 473) took 1155 ms. Trace:
I20260709 12:12:15.658777 14607 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:16.693905 13259 rpcz_store.cc:276] 0709 12:12:15.466157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.620100 (+1153943us) service_pool.cc:224] Handling call
0709 12:12:16.620525 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:16.620963 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:16.621002 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:16.621058 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:16.621105 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.621178 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.621338 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.624562 14856 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:16.696280 14856 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:16.696319 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:15.702971 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.697053 12572 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:16.697351 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 466) took 1302 ms. Trace:
I20260709 12:12:16.697530 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1787) took 1552 ms. Trace:
I20260709 12:12:16.697772 13253 rpcz_store.cc:276] 0709 12:12:15.394740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.397967 (+  3227us) service_pool.cc:224] Handling call
0709 12:12:16.697282 (+1299315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.697902 12572 rpcz_store.cc:276] 0709 12:12:15.145535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.696045 (+1550510us) service_pool.cc:224] Handling call
0709 12:12:16.697472 (+  1427us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.569173 14840 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:16.626849 14909 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:16.698846 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:15.839969 14876 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:16.699488 12572 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:16.699734 14876 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:16.699986 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1794) took 1306 ms. Trace:
I20260709 12:12:16.700237 14876 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.681196 12645 rpcz_store.cc:276] 0709 12:12:12.229150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.684210 (+2455060us) service_pool.cc:224] Handling call
0709 12:12:14.684531 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:15.225970 (+541439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.225999 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:15.226475 (+   476us) log.cc:844] Serialized 26 byte log entry
0709 12:12:15.534562 (+308087us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:15.535328 (+   766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.535409 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:16.680196 (+1144787us) raft_consensus.cc:1685] finished
0709 12:12:16.680227 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.680462 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":524785,"raft.queue_time_us":545379,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"thread_start_us":307737,"threads_started":1,"wal-append.queue_time_us":1317325}
I20260709 12:12:16.702369 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:16.703346 12645 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:16.703792 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1573) took 1012 ms. Trace:
I20260709 12:12:16.704130 12645 rpcz_store.cc:276] 0709 12:12:15.691254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.701968 (+1010714us) service_pool.cc:224] Handling call
0709 12:12:16.703723 (+  1755us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.712600 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1675) took 1456 ms. Trace:
I20260709 12:12:16.713376 12569 rpcz_store.cc:276] 0709 12:12:15.255796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.688133 (+1432337us) service_pool.cc:224] Handling call
0709 12:12:16.712504 (+ 24371us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.715473 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:16.736783 14892 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.895s	user 0.010s	sys 0.010s
I20260709 12:12:16.705277 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:16.740692 14906 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.767s	user 0.004s	sys 0.008s
W20260709 12:12:16.741904 14901 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.224s	user 0.003s	sys 0.000s
I20260709 12:12:16.742830 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1558) took 4584 ms. Trace:
I20260709 12:12:16.743454 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1895) took 3557 ms. Trace:
I20260709 12:12:16.743475 12643 rpcz_store.cc:276] 0709 12:12:12.157931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.647367 (+2489436us) service_pool.cc:224] Handling call
0709 12:12:14.647872 (+   505us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:15.133838 (+485966us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.133889 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:15.134568 (+   679us) log.cc:844] Serialized 26 byte log entry
0709 12:12:15.450128 (+315560us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:15.450194 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.450313 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:16.739878 (+1289565us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:16.742046 (+  2168us) raft_consensus.cc:1685] finished
0709 12:12:16.742089 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.742713 (+   624us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":485331,"raft.queue_time_us":555471,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":26855,"spinlock_wait_cycles":108787072,"thread_start_us":234624,"threads_started":1,"wal-append.queue_time_us":635219}
I20260709 12:12:16.743957 12491 rpcz_store.cc:276] 0709 12:12:13.186066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.186243 (+   177us) service_pool.cc:224] Handling call
0709 12:12:13.282781 (+ 96538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:14.414677 (+1131896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.414711 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:14.745889 (+331178us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:12:14.746382 (+   493us) log.cc:844] Serialized 26 byte log entry
0709 12:12:15.620780 (+874398us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:15.620844 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.620968 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:16.741228 (+1120260us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:16.743133 (+  1905us) raft_consensus.cc:1685] finished
0709 12:12:16.743166 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.743342 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":248418,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":24202,"spinlock_wait_cycles":277518848,"thread_start_us":2005148,"threads_started":2,"wal-append.queue_time_us":1374017}
I20260709 12:12:16.746819 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:16.747575 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:16.748116 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1577) took 1055 ms. Trace:
I20260709 12:12:16.748449 12643 rpcz_store.cc:276] 0709 12:12:15.692400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.746487 (+1054087us) service_pool.cc:224] Handling call
0709 12:12:16.748049 (+  1562us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:16.743597 12644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.136s	user 0.010s	sys 0.007s
W20260709 12:12:16.752138 14893 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.291s	user 0.001s	sys 0.002s
W20260709 12:12:16.753774 14902 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.192s	user 0.002s	sys 0.000s
I20260709 12:12:16.755239 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 223) took 2104 ms. Trace:
W20260709 12:12:16.756011 14882 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.169s	user 0.013s	sys 0.004s
I20260709 12:12:16.755828 12489 rpcz_store.cc:276] 0709 12:12:14.650998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.235553 (+584555us) service_pool.cc:224] Handling call
0709 12:12:15.235953 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.314631 (+ 78678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.314668 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:15.314721 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:15.314775 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.754825 (+1440050us) spinlock_profiling.cc:243] Waited 78.4 ms on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:16.754900 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.755138 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":180307328}
I20260709 12:12:16.749460 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:16.759522 12643 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:16.760214 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1574) took 1068 ms. Trace:
I20260709 12:12:16.761044 12643 rpcz_store.cc:276] 0709 12:12:15.691556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.749143 (+1057587us) service_pool.cc:224] Handling call
0709 12:12:16.760122 (+ 10979us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.762543 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:16.763772 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.764639 12643 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:16.765278 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1576) took 1073 ms. Trace:
I20260709 12:12:16.765796 12643 rpcz_store.cc:276] 0709 12:12:15.692130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.762198 (+1070068us) service_pool.cc:224] Handling call
0709 12:12:16.765217 (+  3019us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.766908 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:16.767575 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1566) took 1331 ms. Trace:
I20260709 12:12:16.767910 12643 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:16.723491 12569 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:12:15.663681 14887 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:12:16.770576 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1791) took 1481 ms. Trace:
I20260709 12:12:16.771014 12569 rpcz_store.cc:276] 0709 12:12:15.288830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.715002 (+1426172us) service_pool.cc:224] Handling call
0709 12:12:16.770491 (+ 55489us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:15.666462 14756 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:16.643821 14845 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:16.631578 14808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.192s	user 0.017s	sys 0.012s
I20260709 12:12:16.768220 12640 rpcz_store.cc:276] 0709 12:12:15.436427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.609302 (+172875us) service_pool.cc:224] Handling call
0709 12:12:15.616308 (+  7006us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:15.616889 (+   581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.616935 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:15.695603 (+ 78668us) log.cc:844] Serialized 26 byte log entry
0709 12:12:16.633783 (+938180us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:16.633847 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.633954 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:16.767235 (+133281us) raft_consensus.cc:1685] finished
0709 12:12:16.767267 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.767493 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":169,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":174,"thread_start_us":810156,"threads_started":1,"wal-append.queue_time_us":1000011}
I20260709 12:12:15.664428 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 306) took 2618 ms. Trace:
I20260709 12:12:16.700323 12572 rpcz_store.cc:276] 0709 12:12:15.393753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.698601 (+1304848us) service_pool.cc:224] Handling call
0709 12:12:16.699928 (+  1327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.611699 13262 rpcz_store.cc:276] 0709 12:12:13.107759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.107955 (+   196us) service_pool.cc:224] Handling call
0709 12:12:13.135943 (+ 27988us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.143395 (+2007452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.568926 (+1425531us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:16.569185 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4589476864}
I20260709 12:12:15.666203 14689 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:16.630538 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:16.782799 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:15.731488 14785 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.783547 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.796769 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1559) took 4638 ms. Trace:
I20260709 12:12:16.650094 14716 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:16.631824 12496 rpcz_store.cc:276] 0709 12:12:14.632167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.170723 (+538556us) service_pool.cc:224] Handling call
0709 12:12:15.250010 (+ 79287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.433011 (+183001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.433058 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:15.433123 (+    65us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:15.433181 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.557859 (+1124678us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:16.558024 (+   165us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.558317 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3409,"raft.queue_time_us":8594,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":9079,"spinlock_wait_cycles":411792128}
I20260709 12:12:16.674203 13261 rpcz_store.cc:276] 0709 12:12:13.036850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.037001 (+   151us) service_pool.cc:224] Handling call
0709 12:12:13.652889 (+615888us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:13.778562 (+125673us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:13.795443 (+ 16881us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:13.795481 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:13.796051 (+   570us) log.cc:844] Serialized 26 byte log entry
0709 12:12:16.596253 (+2800202us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:16.596310 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.596397 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:16.670045 (+ 73648us) raft_consensus.cc:1685] finished
0709 12:12:16.670110 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.671138 (+  1028us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15219,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":238896,"spinlock_wait_cycles":17642880,"thread_start_us":2445931,"threads_started":1,"wal-append.queue_time_us":2394651}
I20260709 12:12:16.785125 14856 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.787158 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1580) took 1093 ms. Trace:
I20260709 12:12:16.595708 14811 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.789086 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:16.643353 14482 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.790618 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1578) took 1098 ms. Trace:
I20260709 12:12:16.858565 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.858940 12640 rpcz_store.cc:276] 0709 12:12:15.692692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.782309 (+1089617us) service_pool.cc:224] Handling call
0709 12:12:16.790549 (+  8240us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:16.765148 13260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 314) took 1302 ms (client timeout 1565 ms). Trace:
W20260709 12:12:16.860404 13260 rpcz_store.cc:269] 0709 12:12:15.463280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.763262 (+1299982us) service_pool.cc:224] Handling call
0709 12:12:16.765065 (+  1803us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:16.791136 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1676) took 1533 ms (client timeout 1787 ms). Trace:
I20260709 12:12:16.788738 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:16.792869 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:16.861223 12564 rpcz_store.cc:269] 0709 12:12:15.258137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.611858 (+353721us) service_pool.cc:224] Handling call
0709 12:12:16.791065 (+1179207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.861646 13262 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:16.794102 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:16.862747 12572 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:16.794405 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.794744 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.797096 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.781344 14845 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.869285 12565 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
W20260709 12:12:16.870055 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1774) took 1746 ms (client timeout 1947 ms). Trace:
I20260709 12:12:16.859949 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:16.870482 12565 rpcz_store.cc:269] 0709 12:12:15.123598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.633010 (+1509412us) service_pool.cc:224] Handling call
0709 12:12:16.869967 (+236957us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":636800}
I20260709 12:12:16.865104 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.798846 12643 rpcz_store.cc:276] 0709 12:12:15.693259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.766529 (+1073270us) service_pool.cc:224] Handling call
0709 12:12:16.787079 (+ 20550us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.873210 12643 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:16.826623 12570 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:12:16.864115 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.874529 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:16.875154 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1795) took 1255 ms. Trace:
I20260709 12:12:16.859858 14811 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:16.867571 12569 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:16.876986 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1789) took 1624 ms (client timeout 2159 ms). Trace:
W20260709 12:12:16.877674 12569 rpcz_store.cc:269] 0709 12:12:15.253063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.771915 (+1518852us) service_pool.cc:224] Handling call
0709 12:12:16.876896 (+104981us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.885766 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1579) took 1192 ms. Trace:
I20260709 12:12:16.886313 12640 rpcz_store.cc:276] 0709 12:12:15.692967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.859653 (+1166686us) service_pool.cc:224] Handling call
0709 12:12:16.885683 (+ 26030us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.883790 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1571) took 1439 ms. Trace:
I20260709 12:12:16.887809 12643 rpcz_store.cc:276] 0709 12:12:15.444074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.872340 (+1428266us) service_pool.cc:224] Handling call
0709 12:12:16.872786 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:16.883362 (+ 10576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.883690 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.875535 12564 rpcz_store.cc:276] 0709 12:12:15.620157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.863823 (+1243666us) service_pool.cc:224] Handling call
0709 12:12:16.875078 (+ 11255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.827165 14785 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:16.786624 14882 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:16.893643 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.895130 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.896884 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:16.897964 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1673) took 1687 ms. Trace:
I20260709 12:12:16.896729 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.898495 12569 rpcz_store.cc:276] 0709 12:12:15.211057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.878459 (+1667402us) service_pool.cc:224] Handling call
0709 12:12:16.897884 (+ 19425us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.862630 13253 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:16.900539 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:16.901226 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:16.903573 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:16.903731 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.905073 13262 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:16.904645 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.907925 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1678) took 1643 ms. Trace:
I20260709 12:12:16.908648 12565 rpcz_store.cc:276] 0709 12:12:15.264859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.871305 (+1606446us) service_pool.cc:224] Handling call
0709 12:12:16.907829 (+ 36524us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.913437 12646 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:16.866652 14567 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.893564 12640 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:12:16.885032 14826 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.908058 14840 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:16.910106 14772 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.921520 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1572) took 1230 ms. Trace:
I20260709 12:12:16.923085 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.924213 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.903290 14858 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.797283 12639 rpcz_store.cc:276] 0709 12:12:12.158274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.649092 (+2490818us) service_pool.cc:224] Handling call
0709 12:12:14.649531 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:15.182509 (+532978us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.209929 (+ 27420us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:15.247738 (+ 37809us) log.cc:844] Serialized 26 byte log entry
0709 12:12:16.669119 (+1421381us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:16.669498 (+   379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.783901 (+114403us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:16.787650 (+  3749us) raft_consensus.cc:1685] finished
0709 12:12:16.791499 (+  3849us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:16.795294 (+  3795us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":482301,"raft.queue_time_us":577491,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":135,"thread_start_us":1419969,"threads_started":1,"wal-append.queue_time_us":1511690}
I20260709 12:12:16.910722 14923 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:16.894474 14811 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:16.921981 14567 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:16.937412 12640 rpcz_store.cc:276] 0709 12:12:15.690808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.887321 (+1196513us) service_pool.cc:224] Handling call
0709 12:12:16.887738 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:16.919505 (+ 31767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.921445 (+  1940us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:16.900033 14785 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:16.921380 14305 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.945041 14724 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.945676 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:16.943586 14567 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:16.937271 14840 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:16.941203 14772 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:16.951318 14858 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:16.943773 14923 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.948537 14724 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:16.957098 14895 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:12:16.957340 14724 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:16.957547 14882 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:16.957854 14724 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:16.958192 14923 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.787621 13258 rpcz_store.cc:276] 0709 12:12:13.046402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.047856 (+  1454us) service_pool.cc:224] Handling call
0709 12:12:13.650510 (+602654us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:13.776087 (+125577us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:13.776706 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:13.776741 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:13.777467 (+   726us) log.cc:844] Serialized 52 byte log entry
0709 12:12:15.611298 (+1833831us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:15.614743 (+  3445us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.663988 (+ 49245us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:15.664044 (+    56us) raft_consensus.cc:1685] finished
0709 12:12:15.664060 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:15.664333 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14368,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":6365,"spinlock_wait_cycles":37761408,"thread_start_us":1798265,"threads_started":1,"wal-append.queue_time_us":1680120,"wal-append.run_cpu_time_us":3010,"wal-append.run_wall_time_us":1154983}
I20260709 12:12:16.953668 14840 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:16.952239 14567 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:16.957111 14689 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:16.954849 14447 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.952198 14876 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.967518 14482 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:16.409767 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:12:17.036051 12461 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.026s	sys 0.026s
W20260709 12:12:17.062292 12461 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.026s	sys 0.027s
W20260709 12:12:17.063136 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 654498us
I20260709 12:12:17.041851 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 221) took 2409 ms. Trace:
I20260709 12:12:16.913154 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1796) took 1292 ms. Trace:
I20260709 12:12:17.066346 12564 rpcz_store.cc:276] 0709 12:12:15.620522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.889973 (+1269451us) service_pool.cc:224] Handling call
0709 12:12:16.913051 (+ 23078us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.065924 12488 rpcz_store.cc:276] 0709 12:12:14.632506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.385069 (+752563us) service_pool.cc:224] Handling call
0709 12:12:15.385577 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:15.386075 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.386125 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:15.398028 (+ 11903us) log.cc:844] Serialized 26 byte log entry
0709 12:12:16.565260 (+1167232us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:16.565323 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.565426 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:17.040828 (+475402us) raft_consensus.cc:1685] finished
0709 12:12:17.040938 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.041504 (+   566us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3826,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":9636,"thread_start_us":1139967,"threads_started":1,"wal-append.queue_time_us":1507811}
I20260709 12:12:17.070518 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:17.071842 13258 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:17.049211 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.049211 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:16.988763 14756 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:16.988327 14785 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:16.955852 14772 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:16.988128 14826 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.996402 14840 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:16.970686 14845 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:16.960119 14858 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:17.225814 12637 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.000s	sys 0.012s
W20260709 12:12:17.226526 12637 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.000s	sys 0.013s
I20260709 12:12:17.228541 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1557) took 5071 ms. Trace:
W20260709 12:12:16.974576 14903 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.001s	sys 0.001s
W20260709 12:12:17.229789 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.008s	sys 0.020s
W20260709 12:12:17.230538 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.008s	sys 0.020s
W20260709 12:12:17.231081 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 594166us
W20260709 12:12:17.054486 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.069301 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.235332 12564 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:17.163755 12565 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:17.238070 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1786) took 2092 ms. Trace:
I20260709 12:12:17.238678 12565 rpcz_store.cc:276] 0709 12:12:15.145257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.048405 (+1903148us) service_pool.cc:224] Handling call
0709 12:12:17.237978 (+189573us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.172070 12569 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:17.148157 14876 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260709 12:12:17.242527 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1669) took 2057 ms. Trace:
I20260709 12:12:17.243183 12569 rpcz_store.cc:276] 0709 12:12:15.185500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.911176 (+1725676us) service_pool.cc:224] Handling call
0709 12:12:17.242362 (+331186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.240469 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:17.245766 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.007s	sys 0.005s
W20260709 12:12:17.246405 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.007s	sys 0.005s
I20260709 12:12:17.246459 12565 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:17.247179 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 594374us
W20260709 12:12:17.249991 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.008s	sys 0.019s
I20260709 12:12:17.220827 14756 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:17.217087 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.005s	sys 0.020s
W20260709 12:12:17.217087 12566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.003s	sys 0.023s
W20260709 12:12:17.225539 14709 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.010s	sys 0.031s
W20260709 12:12:17.229815 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.011s	sys 0.001s
W20260709 12:12:17.258436 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.006s	sys 0.020s
I20260709 12:12:17.245183 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:17.259452 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.009s	sys 0.009s
W20260709 12:12:17.259757 12567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.004s	sys 0.016s
W20260709 12:12:17.260042 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.009s	sys 0.009s
W20260709 12:12:17.260310 12567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.005s	sys 0.016s
W20260709 12:12:17.260646 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 624462us
W20260709 12:12:17.258545 12566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.005s	sys 0.023s
I20260709 12:12:17.264408 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1657) took 4580 ms. Trace:
I20260709 12:12:17.267055 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 196) took 5372 ms. Trace:
I20260709 12:12:17.235373 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1662) took 4118 ms. Trace:
I20260709 12:12:17.272380 14931 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:17.267372 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 201) took 2669 ms. Trace:
I20260709 12:12:17.274173 14932 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:17.273991 12641 rpcz_store.cc:276] 0709 12:12:14.597341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.268409 (+671068us) service_pool.cc:224] Handling call
0709 12:12:15.268776 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.607326 (+338550us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.266647 (+1659321us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:17.267104 (+   457us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":349543808}
I20260709 12:12:17.275120 14932 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.267055 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1556) took 5111 ms. Trace:
I20260709 12:12:17.275944 14772 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:17.254851 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.003s	sys 0.015s
W20260709 12:12:17.276886 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.004s	sys 0.015s
I20260709 12:12:17.270049 12571 rpcz_store.cc:276] 0709 12:12:13.116469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.645390 (+1528921us) service_pool.cc:224] Handling call
0709 12:12:14.645772 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:14.746713 (+100941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.746969 (+   256us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:14.747537 (+   568us) log.cc:844] Serialized 26 byte log entry
0709 12:12:16.559236 (+1811699us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:16.559286 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:16.559385 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:17.234277 (+674892us) raft_consensus.cc:1685] finished
0709 12:12:17.234475 (+   198us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.234976 (+   501us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100491,"raft.queue_time_us":102252,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":62189,"thread_start_us":635406,"threads_started":1,"wal-append.queue_time_us":2002877}
W20260709 12:12:17.256417 14709 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.011s	sys 0.031s
W20260709 12:12:17.286216 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.011s	sys 0.002s
I20260709 12:12:17.265035 12570 rpcz_store.cc:276] 0709 12:12:12.683387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:13.810869 (+1127482us) service_pool.cc:224] Handling call
0709 12:12:13.811352 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:16.873845 (+3062493us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.263583 (+389738us) spinlock_profiling.cc:243] Waited 3.01 s on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:17.264172 (+   589us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6933439360}
W20260709 12:12:17.291739 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.013s	sys 0.004s
W20260709 12:12:17.292289 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.013s	sys 0.005s
I20260709 12:12:17.292147 14876 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:17.292809 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:17.293488 14934 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:17.293195 14876 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.294107 12564 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.295876 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.296743 14876 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:17.297880 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.005s	sys 0.013s
W20260709 12:12:17.298585 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.006s	sys 0.013s
I20260709 12:12:17.297920 14934 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.300460 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.299528 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:17.301838 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98699.1us
I20260709 12:12:17.302222 14826 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:17.303288 12570 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:12:17.304958 14876 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:17.301395 14934 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:17.314343 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:12:17.318427 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85769.4us
W20260709 12:12:17.321199 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.320919 14876 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:12:17.321952 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.309919 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:17.322171 14876 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.323151 14858 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.320766 14914 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:17.308153 14840 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:17.324842 14935 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:17.305979 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1778) took 2180 ms. Trace:
I20260709 12:12:17.267982 12644 rpcz_store.cc:276] 0709 12:12:11.894813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.464708 (+2569895us) service_pool.cc:224] Handling call
0709 12:12:14.465534 (+   826us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:14.535674 (+ 70140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.535716 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:14.562600 (+ 26884us) log.cc:844] Serialized 52 byte log entry
0709 12:12:15.603275 (+1040675us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:16.750327 (+1147052us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.266076 (+515749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.266353 (+   277us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:17.266454 (+   101us) raft_consensus.cc:1685] finished
0709 12:12:17.266476 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.266724 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1193603,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":209063,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":6548,"thread_start_us":2605706,"threads_started":3,"wal-append.queue_time_us":2202737}
W20260709 12:12:17.318367 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48890.1us
I20260709 12:12:17.325350 14876 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:17.313661 14821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.014s	sys 0.006s
I20260709 12:12:17.305253 14826 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:12:17.339427 14821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.015s	sys 0.006s
I20260709 12:12:17.340250 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:17.341719 14876 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:17.342278 14931 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:17.343343 14931 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:17.343878 14931 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.356340 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.349150 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:17.357154 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:17.357508 12569 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:17.358071 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1783) took 2213 ms. Trace:
I20260709 12:12:17.358425 12569 rpcz_store.cc:276] 0709 12:12:15.144371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.355952 (+2211581us) service_pool.cc:224] Handling call
0709 12:12:17.357987 (+  2035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.360193 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.361476 14937 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:17.361963 14937 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:17.349705 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:17.362879 14937 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:17.363905 14826 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:17.345592 14931 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.350060 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:17.229051 12637 rpcz_store.cc:276] 0709 12:12:12.157505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.611764 (+2454259us) service_pool.cc:224] Handling call
0709 12:12:14.612152 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:14.616937 (+  4785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.616966 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:14.617439 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:12:14.857611 (+240172us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b180039f700. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:12:15.587771 (+730160us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:16.613295 (+1025524us) log.cc:844] Serialized 14 byte log entry
0709 12:12:16.753042 (+139747us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:17.227452 (+474410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.227732 (+   280us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:17.227886 (+   154us) raft_consensus.cc:1685] finished
0709 12:12:17.227904 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.228286 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":577048,"prepare.queue_time_us":1208818,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":336968,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":12505,"spinlock_wait_cycles":200723968,"thread_start_us":2331518,"threads_started":3,"wal-append.queue_time_us":1799188}
W20260709 12:12:17.354333 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58700.1us
I20260709 12:12:17.373690 12570 rpcz_store.cc:276] 0709 12:12:15.124973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.299003 (+2174030us) service_pool.cc:224] Handling call
0709 12:12:17.305685 (+  6682us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.365506 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:17.353341 12643 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:12:17.376845 12643 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 2
I20260709 12:12:17.350771 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:17.383966 12640 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:17.383955 12565 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:12:17.348359 14858 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:17.386304 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 467) took 1990 ms (client timeout 1897 ms). Trace:
W20260709 12:12:17.386893 13254 rpcz_store.cc:269] 0709 12:12:15.395206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.398047 (+  2841us) service_pool.cc:224] Handling call
0709 12:12:17.385803 (+1987756us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:17.386001 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176119168}
I20260709 12:12:17.366262 14931 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:17.377118 12643 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:12:17.393851 12643 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:17.395237 12639 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:17.351442 14939 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:17.396924 14939 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.399922 14939 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.386407 14858 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.401917 14567 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.403486 14858 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:17.406041 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:17.407593 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:17.414646 12645 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.513s	user 0.019s	sys 0.000s
W20260709 12:12:17.426759 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84348.3us
I20260709 12:12:17.276400 12638 rpcz_store.cc:276] 0709 12:12:12.155721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.594452 (+2438731us) service_pool.cc:224] Handling call
0709 12:12:14.618265 (+ 23813us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:15.129820 (+511555us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:15.129884 (+    64us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:15.150820 (+ 20936us) log.cc:844] Serialized 52 byte log entry
0709 12:12:16.611393 (+1460573us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:17.264608 (+653215us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.265553 (+   945us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.266023 (+   470us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.266312 (+   289us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:17.266395 (+    83us) raft_consensus.cc:1685] finished
0709 12:12:17.266419 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.266724 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51066,"prepare.queue_time_us":186009,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":83,"raft.queue_time_us":517437,"raft.run_cpu_time_us":718,"raft.run_wall_time_us":34510,"thread_start_us":1818693,"threads_started":2,"wal-append.queue_time_us":1589785}
I20260709 12:12:17.413288 14567 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:17.430892 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1788) took 2221 ms (client timeout 2233 ms). Trace:
W20260709 12:12:17.435849 12572 rpcz_store.cc:269] 0709 12:12:15.206157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.793782 (+1587625us) service_pool.cc:224] Handling call
0709 12:12:17.424168 (+630386us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.409530 14856 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:17.441363 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76066.8us
I20260709 12:12:17.433195 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:17.414058 14884 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:12:17.460304 14607 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.766s	user 0.013s	sys 0.013s
I20260709 12:12:17.441143 14709 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:12:17.413125 14878 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:17.276926 14932 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:17.466063 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106056us
I20260709 12:12:17.428516 14756 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:17.454294 14909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.012s	sys 0.010s
I20260709 12:12:17.478487 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:17.479322 14858 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:17.479687 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1563) took 2044 ms. Trace:
W20260709 12:12:17.478703 14904 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.030s	user 0.008s	sys 0.009s
I20260709 12:12:17.480003 14858 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.480847 14858 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:17.481933 14909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.013s	sys 0.010s
I20260709 12:12:17.482206 14716 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.482674 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 307) took 4356 ms. Trace:
I20260709 12:12:17.483155 12568 rpcz_store.cc:276] 0709 12:12:13.126259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.647644 (+1521385us) service_pool.cc:224] Handling call
0709 12:12:14.648020 (+   376us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:14.747809 (+ 99789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.747842 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:14.859235 (+111393us) log.cc:844] Serialized 52 byte log entry
0709 12:12:15.420375 (+561140us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:15.420421 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.420498 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:15.660278 (+239780us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:17.439710 (+1779432us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:17.482151 (+ 42441us) raft_consensus.cc:1685] finished
0709 12:12:17.482176 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.482438 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99358,"raft.queue_time_us":113421,"raft.run_cpu_time_us":19191,"raft.run_wall_time_us":867452,"spinlock_wait_cycles":99186688,"thread_start_us":406929,"threads_started":1,"wal-append.queue_time_us":865757}
W20260709 12:12:17.489522 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49530.6us
I20260709 12:12:17.488193 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.417801 14937 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.464795 14884 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.491281 14386 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:17.480139 12642 rpcz_store.cc:276] 0709 12:12:15.435118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.440609 (+  5491us) service_pool.cc:224] Handling call
0709 12:12:15.440928 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:15.601289 (+160361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.479125 (+1877836us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:17.479482 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208902528}
I20260709 12:12:17.508373 13262 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:17.472260 14878 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.488174 14607 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.469869 14926 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
W20260709 12:12:17.531268 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 123916us
I20260709 12:12:17.526258 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.535053 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.544405 14305 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:12:17.530305 14772 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:17.546057 14772 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.546546 14772 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:17.554279 14884 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.555819 14607 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.556280 14884 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:17.557227 12571 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:17.540478 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:17.570978 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1663) took 4441 ms. Trace:
I20260709 12:12:17.571382 12571 rpcz_store.cc:276] 0709 12:12:13.129514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.556246 (+4426732us) service_pool.cc:224] Handling call
0709 12:12:17.556855 (+   609us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.570716 (+ 13861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.570921 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.558859 14811 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:17.560694 14890 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.024s	user 0.019s	sys 0.025s
W20260709 12:12:17.561888 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 127102us
I20260709 12:12:17.564922 14766 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:17.527894 14386 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.581260 14766 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.583249 14766 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.565954 12569 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:17.587277 14885 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.056s	user 0.012s	sys 0.041s
I20260709 12:12:17.587643 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:17.586349 14403 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.924s	user 0.018s	sys 0.037s
I20260709 12:12:17.567224 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.591763 12570 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:12:17.592483 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1664) took 4457 ms. Trace:
I20260709 12:12:17.592876 12570 rpcz_store.cc:276] 0709 12:12:13.135349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.590628 (+4455279us) service_pool.cc:224] Handling call
0709 12:12:17.591186 (+   558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.592207 (+  1021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.592425 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.594028 12572 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:17.595803 12572 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:17.595950 14766 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:17.596444 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 309) took 4431 ms. Trace:
I20260709 12:12:17.596931 12572 rpcz_store.cc:276] 0709 12:12:13.165075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.594897 (+4429822us) service_pool.cc:224] Handling call
0709 12:12:17.595426 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.596195 (+   769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.596388 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.598335 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 308) took 4465 ms. Trace:
I20260709 12:12:17.598803 12568 rpcz_store.cc:276] 0709 12:12:13.133103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.485421 (+4352318us) service_pool.cc:224] Handling call
0709 12:12:17.525586 (+ 40165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.597954 (+ 72368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.597991 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.598056 (+    65us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:17.598120 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.598178 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.598283 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.608162 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.608683 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.609077 12494 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:17.609360 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.609974 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.610423 12496 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:17.611023 12488 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:12:17.610973 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:17.510628 14909 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:12:17.611805 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:17.606585 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 125888us
W20260709 12:12:17.550531 14891 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.978s	user 0.026s	sys 0.020s
I20260709 12:12:17.615808 13258 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:12:17.616384 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:17.617319 12569 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:17.617654 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 305) took 5360 ms. Trace:
I20260709 12:12:17.619103 12569 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:17.618068 12563 rpcz_store.cc:276] 0709 12:12:12.257119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.258068 (+   949us) service_pool.cc:224] Handling call
0709 12:12:12.681620 (+423552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:14.111580 (+1429960us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b180008c5c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57ebbfb 00007ffbf57f664b
0709 12:12:14.195321 (+ 83741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:14.195363 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:14.210718 (+ 15355us) log.cc:844] Serialized 26 byte log entry
0709 12:12:15.434765 (+1224047us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:15.434826 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:15.434961 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:17.465419 (+2030458us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:17.617154 (+151735us) raft_consensus.cc:1685] finished
0709 12:12:17.617176 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.617414 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":274532,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":23817,"spinlock_wait_cycles":260903936,"thread_start_us":1417022,"threads_started":2,"wal-append.queue_time_us":1267400}
I20260709 12:12:17.619797 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 310) took 4397 ms. Trace:
I20260709 12:12:17.620220 12569 rpcz_store.cc:276] 0709 12:12:13.222833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.618303 (+4395470us) service_pool.cc:224] Handling call
0709 12:12:17.618787 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.619540 (+   753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.619743 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.620743 12563 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:17.583284 14386 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.624114 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:17.565485 14808 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:17.608018 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:17.533192 13253 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.529s	user 0.000s	sys 0.021s
I20260709 12:12:17.580135 14826 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:17.625873 14826 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:17.626397 14826 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:17.589316 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72906.2us
I20260709 12:12:17.630122 13253 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:12:17.575567 14746 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:17.625255 14567 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:12:17.612192 12491 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.616384 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:17.643150 13259 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:17.644212 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65091.9us
I20260709 12:12:17.631512 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.645594 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:17.633136 14756 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:17.646724 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.647338 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:17.647948 14939 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:17.648442 14939 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.648727 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:17.649422 14607 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:17.650367 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:17.634496 14403 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:17.651057 14403 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.651507 14403 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.652477 14938 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:17.652972 14938 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.653321 14938 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.654042 14940 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:17.654572 14940 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.654899 14940 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:17.634320 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123480us
I20260709 12:12:17.634943 14808 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.658545 14808 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:17.660528 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1575) took 1968 ms (client timeout 1735 ms). Trace:
W20260709 12:12:17.661049 12645 rpcz_store.cc:269] 0709 12:12:15.691847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.704986 (+1013139us) service_pool.cc:224] Handling call
0709 12:12:17.660285 (+955299us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:17.660427 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":371485568}
I20260709 12:12:17.652386 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:17.651374 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:17.675268 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.676488 14785 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:17.677119 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:17.678066 12565 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:17.678524 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1776) took 2554 ms. Trace:
I20260709 12:12:17.678853 12565 rpcz_store.cc:276] 0709 12:12:15.124326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.676700 (+2552374us) service_pool.cc:224] Handling call
0709 12:12:17.678428 (+  1728us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.680595 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:17.681581 12565 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:17.681730 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1649) took 5363 ms. Trace:
I20260709 12:12:17.682725 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.683509 14914 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:17.683637 12492 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:17.683990 14914 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:17.686105 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:17.687043 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1567) took 2250 ms. Trace:
I20260709 12:12:17.677155 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.687441 12646 rpcz_store.cc:276] 0709 12:12:15.436748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:15.612452 (+175704us) service_pool.cc:224] Handling call
0709 12:12:15.612757 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:16.913983 (+1301226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.686611 (+772628us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:17.686853 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2990640000}
I20260709 12:12:17.682121 12567 rpcz_store.cc:276] 0709 12:12:12.318774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:12.318906 (+   132us) service_pool.cc:224] Handling call
0709 12:12:14.522330 (+2203424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.740208 (+217878us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:15.647526 (+907318us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.260922 (+1613396us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.261092 (+   170us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:17.261275 (+   183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.668721 (+407446us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:17.668785 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.681507 (+ 12722us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54003ad750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:17.681663 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2010983,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":85337,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":5566480128,"thread_start_us":2519287,"threads_started":3,"wal-append.queue_time_us":1115227}
I20260709 12:12:17.689108 13257 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:17.689260 14785 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:17.573284 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:17.662396 14936 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:17.662813 14915 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:17.648882 14939 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.666114 14932 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.703259 13259 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:17.667320 12491 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:17.667804 14938 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.650159 14607 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.672483 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1658) took 4983 ms. Trace:
I20260709 12:12:17.675127 14845 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:17.621443 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1666) took 4351 ms. Trace:
I20260709 12:12:17.707782 12563 rpcz_store.cc:276] 0709 12:12:13.270144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.619966 (+4349822us) service_pool.cc:224] Handling call
0709 12:12:17.620395 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.621198 (+   803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.621392 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:17.690668 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66703.4us
W20260709 12:12:17.638715 14724 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.680s	user 0.017s	sys 0.004s
I20260709 12:12:17.636277 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:17.685364 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
W20260709 12:12:17.711460 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64307.1us
I20260709 12:12:17.714092 14724 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:17.715713 14909 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:17.707482 14845 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.716668 14845 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.698390 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.685295 14914 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.726810 12571 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:17.727815 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1777) took 2603 ms. Trace:
I20260709 12:12:17.728274 12571 rpcz_store.cc:276] 0709 12:12:15.124643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.701756 (+2577113us) service_pool.cc:224] Handling call
0709 12:12:17.726181 (+ 24425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.727456 (+  1275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.727742 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.730057 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.730351 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1667) took 4455 ms. Trace:
I20260709 12:12:17.629616 14858 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.715889 12644 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:17.730840 12569 rpcz_store.cc:276] 0709 12:12:13.274875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.621115 (+4346240us) service_pool.cc:224] Handling call
0709 12:12:17.621503 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.729952 (+108449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.729978 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.730024 (+    46us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:17.730061 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.730132 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.730273 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.689888 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:17.718010 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81873.7us
I20260709 12:12:17.727979 14914 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:17.736950 14853 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:17.714627 14937 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:17.710497 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:17.720925 14607 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.721438 14724 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.722865 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:17.724413 12563 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:17.725764 14811 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:17.700444 14915 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:17.704818 12566 rpcz_store.cc:276] 0709 12:12:12.689189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:14.114168 (+1424979us) service_pool.cc:224] Handling call
0709 12:12:14.114628 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:14.614600 (+499972us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:15.469047 (+854447us) log.cc:844] Serialized 14 byte log entry
0709 12:12:16.596921 (+1127874us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.373208 (+776287us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.395182 (+ 21974us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.395285 (+   103us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:17.395455 (+   170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.672040 (+276585us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:17.672088 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.672410 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17492,"prepare.queue_time_us":2367936,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":145,"raft.queue_time_us":220184,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":405,"spinlock_wait_cycles":1149224192,"thread_start_us":2709766,"threads_started":4,"wal-append.queue_time_us":1441472}
I20260709 12:12:17.720141 14936 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:17.739924 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75612.2us
I20260709 12:12:17.733471 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1784) took 2588 ms. Trace:
I20260709 12:12:17.754312 12569 rpcz_store.cc:276] 0709 12:12:15.144686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.732069 (+2587383us) service_pool.cc:224] Handling call
0709 12:12:17.732595 (+   526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.732987 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.733016 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.733098 (+    82us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:17.733190 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.733253 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.733407 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:17.739635 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96296.8us
I20260709 12:12:17.737382 14756 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:12:17.758862 12493 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:12:17.759986 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70225.4us
I20260709 12:12:17.760557 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:17.738606 14853 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:17.728643 14889 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:12:17.763091 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52706us
I20260709 12:12:17.767381 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1780) took 2641 ms. Trace:
I20260709 12:12:17.770633 12563 rpcz_store.cc:276] 0709 12:12:15.125572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.717428 (+2591856us) service_pool.cc:224] Handling call
0709 12:12:17.723992 (+  6564us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.767026 (+ 43034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.767303 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.778519 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:17.783037 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.768819 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:17.753129 14811 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:17.793291 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:17.796136 14607 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:17.797581 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:17.798163 14403 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.698683 14878 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.799674 14878 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:17.800588 14403 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:17.769618 14567 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:17.801949 14567 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:17.802431 14567 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:17.777518 14856 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:17.766307 14853 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:17.806357 12567 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:17.807297 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1785) took 2662 ms. Trace:
I20260709 12:12:17.798617 14938 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.807682 12567 rpcz_store.cc:276] 0709 12:12:15.144975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.749830 (+2604855us) service_pool.cc:224] Handling call
0709 12:12:17.805938 (+ 56108us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.806990 (+  1052us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.807236 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:17.815941 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.808017 14858 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:17.817464 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:17.819264 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.779395 14889 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:17.781678 14909 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:17.831183 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 313) took 3160 ms. Trace:
I20260709 12:12:17.831364 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:17.790028 14811 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:17.801223 14939 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.838267 12565 rpcz_store.cc:276] 0709 12:12:14.670893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.682314 (+3011421us) service_pool.cc:224] Handling call
0709 12:12:17.682748 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.683141 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:12:17.684735 (+  1594us) log.cc:844] Serialized 14 byte log entry
0709 12:12:17.830500 (+145765us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.830571 (+    71us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:17.830625 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.830710 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.830912 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5440,"prepare.queue_time_us":654,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":98245,"threads_started":1,"wal-append.queue_time_us":144490}
I20260709 12:12:17.790750 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:17.809064 14876 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:17.820102 14607 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:17.832144 14889 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:17.861783 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1668) took 2676 ms. Trace:
I20260709 12:12:17.862448 12569 rpcz_store.cc:276] 0709 12:12:15.185022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.755198 (+2570176us) service_pool.cc:224] Handling call
0709 12:12:17.819675 (+ 64477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.861162 (+ 41487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.861205 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.861272 (+    67us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:17.861316 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.861401 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.861573 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:17.866954 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69453.6us
I20260709 12:12:17.867924 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.870337 12393 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:17.874991 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 316) took 2406 ms. Trace:
I20260709 12:12:17.875741 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:17.878415 14928 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:12:17.879984 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 315) took 2535 ms. Trace:
I20260709 12:12:17.880558 12567 rpcz_store.cc:276] 0709 12:12:15.344764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.851181 (+2506417us) service_pool.cc:224] Handling call
0709 12:12:17.879293 (+ 28112us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.879630 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.879653 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.879695 (+    42us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:17.879741 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.879792 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.879915 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.881281 14482 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:17.869482 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:17.883422 12566 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:17.866496 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.875548 12565 rpcz_store.cc:276] 0709 12:12:15.468447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.873670 (+2405223us) service_pool.cc:224] Handling call
0709 12:12:17.874182 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.874576 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.874603 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.874681 (+    78us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:17.874717 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.874775 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.874914 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.867975 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:17.884729 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1674) took 2631 ms. Trace:
I20260709 12:12:17.887987 12566 rpcz_store.cc:276] 0709 12:12:15.253683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.777871 (+2524188us) service_pool.cc:224] Handling call
0709 12:12:17.841911 (+ 64040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.884439 (+ 42528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.884670 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.847183 14856 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:17.896303 12491 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:17.826900 14756 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:12:17.841656 14785 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:17.903815 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 181094us
I20260709 12:12:17.910235 12536 leader_election.cc:400] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
I20260709 12:12:17.910473 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:17.904549 14915 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.912048 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1790) took 2649 ms. Trace:
I20260709 12:12:17.912230 14937 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:17.912799 14937 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:17.912539 12563 rpcz_store.cc:276] 0709 12:12:15.263177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.830154 (+2566977us) service_pool.cc:224] Handling call
0709 12:12:17.897291 (+ 67137us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.911680 (+ 14389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.911717 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:17.911765 (+    48us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:17.911809 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.911877 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:17.911989 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.913331 14853 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.914691 14936 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:17.915863 12567 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:17.916555 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 317) took 1293 ms. Trace:
W20260709 12:12:17.918592 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135777us
I20260709 12:12:17.919741 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:17.915508 14853 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:17.910687 12536 leader_election.cc:403] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:17.917062 12567 rpcz_store.cc:276] 0709 12:12:16.623065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.894739 (+1271674us) service_pool.cc:224] Handling call
0709 12:12:17.915653 (+ 20914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.916297 (+   644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.916494 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:17.914196 14937 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.896585 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.921876 14941 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:17.931357 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 192133us
I20260709 12:12:17.922178 12393 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 1 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:17.939306 14941 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:17.954778 14941 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:17.958813 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.967142 14941 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:17.897141 14876 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:17.973404 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:17.980631 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61075.1us
I20260709 12:12:17.986670 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:17.908648 14772 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:17.989116 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:17.979818 14930 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.158s
I20260709 12:12:17.938663 14938 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:12:17.972949 14482 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:18.025125 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 217085us
I20260709 12:12:17.993780 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:18.016939 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 106978us
I20260709 12:12:18.017475 12490 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:17.997381 12491 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:18.016548 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.023193 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:18.032259 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 254655us
I20260709 12:12:18.033383 12493 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:18.034742 14684 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:18.035108 14952 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.248s	user 0.002s	sys 0.000s
I20260709 12:12:18.034842 14856 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.035390 14684 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:18.036198 14876 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:18.036266 14684 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:18.036975 12393 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 1 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:18.039112 14948 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:18.039691 14948 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:18.040166 14948 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:18.041715 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68988.3us
I20260709 12:12:18.044427 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1779) took 2919 ms. Trace:
I20260709 12:12:18.044953 12572 rpcz_store.cc:276] 0709 12:12:15.125269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.703176 (+2577907us) service_pool.cc:224] Handling call
0709 12:12:17.723091 (+ 19915us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.723678 (+   587us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.723712 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:17.724352 (+   640us) log.cc:844] Serialized 26 byte log entry
0709 12:12:17.835821 (+111469us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:17.835883 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.835961 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:18.044151 (+208190us) raft_consensus.cc:1685] finished
0709 12:12:18.044175 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:18.044350 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5398,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":132,"thread_start_us":110998,"threads_started":1,"wal-append.queue_time_us":301721}
I20260709 12:12:17.976397 14940 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:18.049578 14957 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.113s	user 0.002s	sys 0.000s
I20260709 12:12:18.050457 14607 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.028118 14808 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:17.997426 14909 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.065737 14941 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:18.067608 14845 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.068935 14876 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:18.069404 14876 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:18.069913 14876 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:18.070225 14939 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:18.016230 14724 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:12:18.069566 14716 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:18.073997 12644 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:18.071373 14684 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.051110 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:17.980069 14826 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:18.091122 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48365.9us
I20260709 12:12:18.093870 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:18.095152 14840 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:18.097398 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:18.098021 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:18.098821 12644 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:18.099457 14845 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:18.100709 12497 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:12:18.101922 14305 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:18.101976 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.103171 14927 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:18.103660 14927 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:18.114240 14840 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:18.072057 14724 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:18.116123 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93492.5us
I20260709 12:12:18.054216 14940 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:18.027789 14482 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:18.129344 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:18.129604 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:18.130350 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.131249 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:18.132505 14840 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:18.133116 14840 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:18.133549 14840 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:18.134716 14951 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.248s	user 0.003s	sys 0.000s
W20260709 12:12:18.136163 14959 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 12:12:18.139070 14915 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:18.139349 14772 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:18.140659 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60532.3us
W20260709 12:12:18.148138 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45832us
I20260709 12:12:18.147733 14772 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:12:18.149802 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 426) took 1255 ms. Trace:
I20260709 12:12:18.150331 13260 rpcz_store.cc:276] 0709 12:12:16.894704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:16.894845 (+   141us) service_pool.cc:224] Handling call
0709 12:12:17.388467 (+493622us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.437461 (+ 48994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.437504 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:17.461689 (+ 24185us) log.cc:844] Serialized 26 byte log entry
0709 12:12:17.543138 (+ 81449us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:17.543181 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.571641 (+ 28460us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:18.133862 (+562221us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:18.139650 (+  5788us) raft_consensus.cc:1685] finished
0709 12:12:18.139682 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:18.149331 (+  9649us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b5400578ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:18.149525 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45322,"raft.queue_time_us":48839,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":3655,"spinlock_wait_cycles":1119574528,"thread_start_us":66146,"threads_started":1,"wal-append.queue_time_us":346687}
I20260709 12:12:18.140161 14811 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.157995 14684 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:18.115303 14927 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.164352 14927 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:18.165269 14482 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.166918 14482 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:18.167724 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:18.168517 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.168895 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 314) took 3497 ms. Trace:
I20260709 12:12:18.169369 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.170002 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.170238 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.170917 12572 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:18.171285 12566 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:18.157829 14967 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:18.172320 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.172348 12492 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:18.173789 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.174489 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:18.176450 14403 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:18.177346 14766 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:18.179440 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.180347 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:18.185377 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.185599 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.186370 12642 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:18.188422 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:18.170075 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.185783 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.169384 12564 rpcz_store.cc:276] 0709 12:12:14.671857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.694500 (+3022643us) service_pool.cc:224] Handling call
0709 12:12:17.753531 (+ 59031us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.818049 (+ 64518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.818082 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:17.818734 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 12:12:17.974106 (+155372us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:17.974187 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.974283 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:18.168550 (+194267us) raft_consensus.cc:1685] finished
0709 12:12:18.168573 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:18.168779 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25759,"raft.queue_time_us":10207,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":171,"thread_start_us":153117,"threads_started":1,"wal-append.queue_time_us":334425}
I20260709 12:12:18.185411 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.204422 12563 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:18.123814 14724 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:18.205238 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56181.5us
I20260709 12:12:18.216382 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:18.216430 12489 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:18.219753 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:18.223750 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.223857 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.220769 12488 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:18.230559 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.231633 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.207369 14785 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:18.230580 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.243944 12640 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:18.243968 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:18.265128 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:18.216565 14811 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:18.248301 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:18.283347 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:18.286510 12569 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:18.286942 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 145613us
I20260709 12:12:18.215772 14935 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4.
W20260709 12:12:18.296334 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48216.9us
I20260709 12:12:18.287353 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:18.299240 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60369.4us
I20260709 12:12:18.301296 12572 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:18.306404 14966 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:18.307018 12642 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:18.307224 14948 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.313434 12640 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:12:18.314680 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1782) took 3170 ms. Trace:
I20260709 12:12:18.293538 14766 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:18.315516 14766 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.314602 14937 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:18.315240 12571 rpcz_store.cc:276] 0709 12:12:15.144058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.740537 (+2596479us) service_pool.cc:224] Handling call
0709 12:12:17.785442 (+ 44905us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.973069 (+187627us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.973192 (+   123us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:17.988165 (+ 14973us) log.cc:844] Serialized 26 byte log entry
0709 12:12:18.213928 (+225763us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:18.213992 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:18.214098 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:18.314410 (+100312us) raft_consensus.cc:1685] finished
0709 12:12:18.314434 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:18.314602 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28199,"raft.queue_time_us":42247,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":22798,"thread_start_us":232506,"threads_started":2,"wal-append.queue_time_us":288763}
I20260709 12:12:18.320588 12638 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:12:18.297592 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:18.323395 12497 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:18.342037 12564 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:18.346103 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:18.322072 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:18.354192 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.355118 12640 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:18.359225 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61782.9us
I20260709 12:12:18.327319 14937 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:18.361603 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.307075 14966 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:18.366218 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65362.2us
I20260709 12:12:18.346709 14826 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:18.373064 12640 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260709 12:12:18.383014 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72092.4us
I20260709 12:12:18.380471 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1665) took 5212 ms. Trace:
W20260709 12:12:18.368613 14932 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.655s	user 0.011s	sys 0.004s
I20260709 12:12:18.327275 14386 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:12:18.366828 12572 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:18.367789 14937 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:18.394289 12568 rpcz_store.cc:276] 0709 12:12:13.168561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.599903 (+4431342us) service_pool.cc:224] Handling call
0709 12:12:17.600416 (+   513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:17.601049 (+   633us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:17.601103 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:17.673935 (+ 72832us) log.cc:844] Serialized 26 byte log entry
0709 12:12:17.745505 (+ 71570us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:17.745554 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:17.746039 (+   485us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:18.380197 (+634158us) raft_consensus.cc:1685] finished
0709 12:12:18.380218 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:18.380401 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":205,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":145,"thread_start_us":71055,"threads_started":1,"wal-append.queue_time_us":541491}
I20260709 12:12:18.368551 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:18.425395 12637 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:18.334843 14931 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260709 12:12:18.397416 14756 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:12:18.371202 14966 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:18.432905 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:18.437711 14386 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:12:18.423746 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:18.399958 14567 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.597s	user 0.015s	sys 0.000s
I20260709 12:12:18.385808 14785 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:18.371634 14716 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:12:18.418848 14724 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.375763 14934 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 12:12:18.460996 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:18.433992 14889 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.537s	user 0.006s	sys 0.008s
I20260709 12:12:18.431775 14948 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:18.428665 14766 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:18.446269 14808 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:18.457067 14785 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:18.521507 12637 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:18.520186 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.542348 12638 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:12:18.545141 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.546115 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.547938 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.487049 14386 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 12:12:18.550050 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.493453 14876 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.523952 14785 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:18.552942 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:18.555637 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.559574 12569 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:18.458802 14567 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.557497 12497 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:18.561604 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:18.486691 14889 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:18.574226 12565 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:18.575780 12641 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:18.578949 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:18.583498 12491 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:18.553691 14968 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:18.601840 12564 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:18.602581 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:18.606024 12490 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.005s
I20260709 12:12:18.606647 12490 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:18.612799 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:18.614360 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.617657 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.618511 12563 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:18.628712 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.631309 12495 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:18.631019 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:18.628187 14915 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:18.633554 14876 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:18.634752 14932 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:18.635197 14932 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:18.636096 14932 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:18.587688 14709 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:12:18.570897 14840 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:18.642436 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:18.655026 12646 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:18.666571 12497 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:19.008685 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:19.009059 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.993348 14915 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:18.994305 14826 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:18.996289 14937 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.545s	user 0.012s	sys 0.000s
I20260709 12:12:18.942549 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.997182 14567 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:18.884291 12645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.010s	sys 0.008s
I20260709 12:12:18.682160 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:18.884023 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:19.058178 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.010s	sys 0.017s
I20260709 12:12:18.619793 14926 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:19.003304 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:18.637421 14932 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:18.674300 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:18.673050 14970 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
I20260709 12:12:19.007140 12572 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:18.703595 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.022775 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:18.683265 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.066596 12565 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:19.072440 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 441498us
I20260709 12:12:19.073882 12492 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.078581 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:19.054214 14482 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.886s	user 0.016s	sys 0.007s
W20260709 12:12:19.063493 14977 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.583s	user 0.005s	sys 0.005s
I20260709 12:12:19.084540 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:19.086977 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.093917 12568 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:19.090175 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.089375 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.096174 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.099324 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.100418 12572 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:19.068023 14926 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:19.107026 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:19.100374 12567 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.117827 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:19.114907 14840 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:19.102433 14915 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:12:19.070371 14826 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.115603 12565 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:19.132990 12495 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:19.133378 12493 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:19.138692 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:19.145558 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56307us
I20260709 12:12:19.104053 14909 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:19.064049 14937 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.155277 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.096030 14716 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
W20260709 12:12:19.162676 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159490us
I20260709 12:12:19.127187 14926 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:19.146867 14987 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:12:19.169546 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:19.170288 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63537.2us
I20260709 12:12:19.171116 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:19.173754 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 233) took 1180 ms. Trace:
I20260709 12:12:19.176411 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:19.178099 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.178938 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.174454 12496 rpcz_store.cc:276] 0709 12:12:17.993318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.123787 (+130469us) service_pool.cc:224] Handling call
0709 12:12:18.150144 (+ 26357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:18.150534 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:18.150564 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:18.151176 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:12:18.332203 (+181027us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:18.332243 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:18.332317 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:19.173332 (+841015us) raft_consensus.cc:1685] finished
0709 12:12:19.173372 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:19.173653 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":180515,"threads_started":1,"wal-append.queue_time_us":244081}
I20260709 12:12:19.182811 12646 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:19.164271 14482 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:19.197265 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.204924 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:19.208088 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:19.169152 14990 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:19.210409 13260 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6.
I20260709 12:12:19.167827 14709 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
W20260709 12:12:19.196512 14785 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.004s
I20260709 12:12:19.182329 14889 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:19.217548 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:19.204514 14976 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.173s	user 0.001s	sys 0.001s
I20260709 12:12:19.196750 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:19.185029 14986 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:19.229604 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:19.220770 14990 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:19.221184 14909 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:19.222649 14785 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.242703 13257 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:12:19.243168 13257 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 5
I20260709 12:12:19.243371 13257 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:12:19.243574 13257 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:19.245496 14785 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.246181 14709 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:19.246878 14709 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:19.257047 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52231.8us
I20260709 12:12:19.259841 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1589) took 1020 ms. Trace:
W20260709 12:12:19.256067 14966 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.795s	user 0.028s	sys 0.035s
I20260709 12:12:19.260386 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.261261 12643 rpcz_store.cc:276] 0709 12:12:18.235803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.235941 (+   138us) service_pool.cc:224] Handling call
0709 12:12:18.281163 (+ 45222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:18.301841 (+ 20678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:18.301873 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:18.317948 (+ 16075us) log.cc:844] Serialized 26 byte log entry
0709 12:12:18.528823 (+210875us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:18.528889 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:18.529001 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:19.255723 (+726722us) raft_consensus.cc:1685] finished
0709 12:12:19.255743 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:19.255892 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":210393,"threads_started":1,"wal-append.queue_time_us":319728}
W20260709 12:12:19.256331 14927 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.090s	user 0.016s	sys 0.006s
I20260709 12:12:19.246292 14785 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:19.265930 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:19.280818 14986 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:19.281394 14986 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:19.281894 14986 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.256081 14709 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.296598 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:19.274158 14826 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.301216 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:19.279975 14785 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:19.308524 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:19.309314 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49022.4us
I20260709 12:12:19.312098 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:19.300063 14993 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.219s	user 0.000s	sys 0.002s
W20260709 12:12:19.316952 14988 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 12:12:19.325958 15010 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:19.324631 14986 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.329466 14986 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:19.329869 15006 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:19.330459 15006 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:19.331688 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:19.276410 14966 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.308370 14927 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:19.315487 14785 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.330866 15006 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.310601 14909 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:19.342576 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.321656 15005 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:19.349577 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:19.387246 12567 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:19.364473 12491 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.372694 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.365108 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.371917 12492 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:19.368830 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.374675 12566 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:19.379204 13259 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:19.377036 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:19.374920 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.359504 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:19.366101 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.394332 12644 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:19.393968 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.403324 12567 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.350039 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.405573 12643 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:19.357009 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.408048 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.408133 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.410457 12642 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
W20260709 12:12:19.350219 14996 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
W20260709 12:12:19.380474 15002 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:12:19.357980 14937 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.374593 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:19.375898 15003 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.129s	user 0.003s	sys 0.000s
I20260709 12:12:19.420665 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1814) took 1096 ms. Trace:
I20260709 12:12:19.423327 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.430545 12571 rpcz_store.cc:276] 0709 12:12:18.324863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.356578 (+1031715us) service_pool.cc:224] Handling call
0709 12:12:19.420563 (+ 63985us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:19.423969 12569 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:19.423327 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.427168 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.433008 12643 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.434192 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.361835 14927 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:19.435710 14927 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.436125 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.437178 12563 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:19.438918 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.439719 12637 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:19.359709 14826 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:19.441321 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.442899 12496 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:12:19.444175 12497 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 12:12:19.444577 14972 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.068s	user 0.003s	sys 0.000s
W20260709 12:12:19.450198 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85168.6us
I20260709 12:12:19.405222 15005 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:19.350308 14990 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.452211 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:19.374150 14966 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:19.455253 14926 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.459159 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.460788 13257 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:19.461865 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 240) took 1019 ms. Trace:
W20260709 12:12:19.463217 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103780us
I20260709 12:12:19.465934 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.462338 12489 rpcz_store.cc:276] 0709 12:12:18.442410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.442623 (+   213us) service_pool.cc:224] Handling call
0709 12:12:18.487591 (+ 44968us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:18.497621 (+ 10030us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:18.497706 (+    85us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:18.556893 (+ 59187us) log.cc:844] Serialized 26 byte log entry
0709 12:12:18.781237 (+224344us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:18.781297 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:18.781404 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:19.461552 (+680148us) raft_consensus.cc:1685] finished
0709 12:12:19.461572 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:19.461774 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":177821,"threads_started":1,"wal-append.queue_time_us":633601}
I20260709 12:12:19.472446 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.473500 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:19.476693 12611 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6.
I20260709 12:12:19.477267 12611 leader_election.cc:403] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:12:19.482576 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.479259 15013 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:19.487377 12491 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:19.490835 12646 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.491758 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.495008 12644 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:19.495123 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.496290 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:19.502179 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.478052 14937 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:19.425091 14567 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 12:12:19.511240 12571 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.508612 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.509276 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.483136 15005 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.516873 12567 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:19.494225 14826 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:19.515822 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:19.517618 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.519443 12492 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:19.521860 12489 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:19.523600 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:19.524644 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.510581 14937 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:19.526146 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:19.528528 12644 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:19.500845 15000 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:12:19.518410 15013 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:19.534580 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.535930 12567 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:19.538486 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.539079 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.541779 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.543318 15006 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.545604 15006 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.547468 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:19.550927 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.551318 12492 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:19.557042 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.562777 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.564604 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.532542 14937 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.528882 14926 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:19.564462 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.569578 12567 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:19.533773 15008 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:19.568234 14607 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:19.574290 14607 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:19.575558 14607 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.583946 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:19.575591 14785 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.585264 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.587180 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.588047 12567 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.587128 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.589854 12646 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:19.589246 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.590955 12637 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:19.590946 12568 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.595176 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.595783 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.597110 14926 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:19.597965 14926 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:19.598472 15013 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:19.599436 15013 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:19.599052 14926 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.601173 14915 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:19.601591 14785 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.603338 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.601706 14915 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:19.604894 14914 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:19.605480 14915 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:19.605911 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.945s (SENT) [suppressed 8 similar messages]
W20260709 12:12:19.607065 14971 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.232s	user 0.001s	sys 0.001s
I20260709 12:12:19.605532 14914 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.608688 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.613023 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.613298 12643 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:19.607024 14915 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:19.624725 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.627072 12644 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:19.636126 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.644054 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:19.622051 14968 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:19.649469 12492 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:19.650425 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.651320 14930 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:19.652004 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.670534 12569 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:19.636619 14932 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:19.636761 14811 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:19.678260 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72366.6us
I20260709 12:12:19.679807 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.680841 12492 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:19.653472 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:19.654073 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.682196 12495 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:19.683073 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.654834 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.657860 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:19.684974 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.658504 14785 raft_consensus.cc:2764] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:12:19.666361 12568 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:19.687618 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.688715 12493 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:12:19.690438 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.690837 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.691838 12644 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:19.693564 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.695706 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.696373 12568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.698707 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.700778 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:19.703035 12493 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:19.654075 15007 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:19.705538 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1600) took 1101 ms. Trace:
I20260709 12:12:19.670034 14930 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:19.709877 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:19.711707 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.699355 14709 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:12:19.713852 15025 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:19.678596 14968 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.715126 15025 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:19.715641 15025 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:19.716004 15025 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.717037 14968 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:19.707216 12641 rpcz_store.cc:276] 0709 12:12:18.603863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.604023 (+   160us) service_pool.cc:224] Handling call
0709 12:12:19.598747 (+994724us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:19.598983 (+   236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.599279 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:19.599627 (+   348us) log.cc:844] Serialized 14 byte log entry
0709 12:12:19.705126 (+105499us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:19.705190 (+    64us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:19.705244 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:19.705315 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:19.705458 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2995,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":94,"spinlock_wait_cycles":5006080,"thread_start_us":105063,"threads_started":1}
I20260709 12:12:19.689867 14932 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.711210 15007 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.722940 15007 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:19.724188 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.725739 14932 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:19.707207 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:19.731736 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.733101 12639 raft_consensus.cc:2368] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
W20260709 12:12:19.734241 12536 connection.cc:256] RPC call timeout handler was delayed by -2.91895631400007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
W20260709 12:12:19.734730 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7794563140000719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
W20260709 12:12:19.735090 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7452563140000166s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2096s for a response
W20260709 12:12:19.735513 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2214563140000791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2678s for a response
I20260709 12:12:19.690003 14826 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:19.738260 14926 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.666494 14927 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:19.730844 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 325) took 1537 ms. Trace:
I20260709 12:12:19.741420 12567 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:19.753105 12571 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:19.752960 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65155.5us
I20260709 12:12:19.755810 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:19.763790 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1599) took 1183 ms (client timeout 1563 ms). Trace:
I20260709 12:12:19.754869 12566 rpcz_store.cc:276] 0709 12:12:18.193492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.204607 (+ 11115us) service_pool.cc:224] Handling call
0709 12:12:18.217233 (+ 12626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.394380 (+1177147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:19.730538 (+336158us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:19.730768 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2661372544}
I20260709 12:12:19.781672 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:19.778925 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:19.779834 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.787554 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.787879 12567 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:19.788633 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:19.782689 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:19.790344 12489 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:19.790989 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81188.9us
I20260709 12:12:19.791859 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:19.765339 12639 rpcz_store.cc:269] 0709 12:12:18.571561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.579549 (+  7988us) service_pool.cc:224] Handling call
0709 12:12:19.731526 (+1151977us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:19.755253 (+ 23727us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6330624}
I20260709 12:12:19.704603 14811 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.774564 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.742627 14914 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:19.806833 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.810348 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:19.814937 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.818406 12641 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:12:19.819873 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:19.820441 12641 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:19.820677 12639 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:19.821868 14982 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:19.767627 14756 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
W20260709 12:12:19.824573 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73347.9us
I20260709 12:12:19.743520 14930 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:19.827096 14930 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:19.834843 12566 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:19.836812 12566 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:19.838173 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.821892 12641 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:19.800729 15005 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6.
I20260709 12:12:19.851377 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.924448 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:19.940686 12567 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:12:19.942379 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91119.7us
W20260709 12:12:19.950258 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 112249us
I20260709 12:12:19.951650 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:19.803224 14889 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.122s
I20260709 12:12:19.956609 12642 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:19.953970 14756 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:19.945900 14840 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:19.950778 14990 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:12:19.952545 14966 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
W20260709 12:12:19.979007 14978 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.013s	sys 0.041s
W20260709 12:12:19.838090 14968 consensus_queue.cc:1175] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 4 status { last_received { term: 4 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 4 index: 6 } }
W20260709 12:12:20.012449 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 153092us
I20260709 12:12:20.017105 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1671) took 4830 ms. Trace:
I20260709 12:12:20.019517 14966 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:20.020314 14966 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:20.024165 14966 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:20.014678 14978 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.014s	sys 0.042s
I20260709 12:12:20.017573 12570 rpcz_store.cc:276] 0709 12:12:15.186276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.763150 (+2576874us) service_pool.cc:224] Handling call
0709 12:12:17.789479 (+ 26329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:17.828857 (+ 39378us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:17.894138 (+ 65281us) log.cc:844] Serialized 14 byte log entry
0709 12:12:20.016583 (+2122445us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b180034fcc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:12:20.016692 (+   109us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:20.016758 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:20.016806 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:20.016890 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:20.017037 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":51493,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":78757,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":2921,"spinlock_wait_cycles":109474176,"thread_start_us":424207,"threads_started":2,"wal-append.queue_time_us":442618}
I20260709 12:12:20.050879 12570 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:20.016436 14940 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:19.996706 14840 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.057477 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.058557 12570 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:20.044077 14567 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:20.063037 12571 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:20.059712 14927 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:20.078826 15028 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
W20260709 12:12:20.111097 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159486us
I20260709 12:12:20.113405 12569 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:20.120919 14811 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:20.122166 15035 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.065s	user 0.002s	sys 0.000s
W20260709 12:12:20.127308 14979 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.520s	user 0.001s	sys 0.005s
W20260709 12:12:20.130996 15029 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.293s	user 0.000s	sys 0.002s
I20260709 12:12:20.131182 14840 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:20.131569 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.132539 12571 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5. Re-sending same reply.
I20260709 12:12:20.056753 14940 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:20.134131 12536 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:20.134591 12536 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:20.134923 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.136937 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:20.136010 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.138775 12571 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:20.143814 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.145689 12569 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:20.139581 14937 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.561s	user 0.009s	sys 0.004s
I20260709 12:12:20.150501 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.152629 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.156451 15012 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:20.156191 14937 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.157094 15012 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:20.158370 14926 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:12:20.159641 14926 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:20.159698 15012 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:20.160758 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.152221 15042 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:20.160225 14926 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:20.167438 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.169399 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:20.169900 15034 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:12:20.172315 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.173161 12567 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:20.174233 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.175083 13260 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:20.175582 14982 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:20.176291 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:20.179347 14940 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:20.162657 14937 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:20.183339 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:20.179098 15025 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.191969 15025 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:20.194376 12569 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:20.196111 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.196975 14756 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:20.197952 13256 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:20.198844 14982 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:20.184938 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.200183 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:20.206938 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.207847 12564 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 5
W20260709 12:12:20.209201 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:20.213033 12609 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.947s (SENT)
I20260709 12:12:20.213783 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:20.349244 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.350350 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.351118 12571 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:20.354677 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143337us
I20260709 12:12:20.361315 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.178287 14926 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:20.370740 12461 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.142s	user 0.013s	sys 0.025s
W20260709 12:12:20.385944 12461 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.157s	user 0.013s	sys 0.025s
W20260709 12:12:20.386639 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.17385e+06us
I20260709 12:12:20.359519 14966 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:20.360952 14937 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:20.389987 12643 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:12:20.390386 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.398860 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:20.400098 12643 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:20.400707 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:20.401355 12644 raft_consensus.cc:2376] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6. Re-sending same reply.
I20260709 12:12:20.402334 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:20.406953 12564 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:20.404383 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:20.409883 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 196909us
I20260709 12:12:20.408783 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:20.374456 14982 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:20.424365 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:20.425252 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:20.425727 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:20.425916 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:20.426518 15008 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:20.426837 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:20.427567 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1940) took 1886 ms (client timeout 1810 ms). Trace:
W20260709 12:12:20.428066 12491 rpcz_store.cc:269] 0709 12:12:18.559165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.599677 (+ 40512us) service_pool.cc:224] Handling call
0709 12:12:20.427325 (+1827648us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:20.427427 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1722781440}
W20260709 12:12:20.400507 14930 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.005s
I20260709 12:12:20.430994 14982 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:20.431631 14982 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:20.432155 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 484) took 1066 ms. Trace:
I20260709 12:12:20.432734 13257 rpcz_store.cc:276] 0709 12:12:19.366075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.366244 (+   169us) service_pool.cc:224] Handling call
0709 12:12:19.366561 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.461260 (+ 94699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:20.431825 (+970565us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:20.432076 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":216065408}
I20260709 12:12:20.434409 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:20.435659 12638 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:20.439234 12461 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.205s (SENT)
I20260709 12:12:20.440173 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.442157 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:20.444550 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.426946 15008 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:20.447640 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.448423 12569 raft_consensus.cc:2376] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6. Re-sending same reply.
I20260709 12:12:20.410050 15042 raft_consensus.cc:2764] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:12:20.451982 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.452972 12567 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:20.454386 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.455245 12564 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5. Re-sending same reply.
I20260709 12:12:20.413959 14926 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:20.499641 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:20.583319 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104439us
W20260709 12:12:20.597954 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 188390us
I20260709 12:12:20.610355 12638 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:20.611405 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:20.585347 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1926) took 2712 ms (client timeout 1769 ms). Trace:
I20260709 12:12:20.614323 12564 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:20.602078 15025 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:20.618963 15025 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:20.622879 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.624159 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.625087 15000 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:12:20.614471 12494 rpcz_store.cc:269] 0709 12:12:17.765934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.777757 (+ 11823us) service_pool.cc:224] Handling call
0709 12:12:20.478023 (+2700266us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:20.478193 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1336487296}
I20260709 12:12:20.631487 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:20.633574 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.020s	sys 0.038s
W20260709 12:12:20.634250 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.020s	sys 0.038s
I20260709 12:12:20.594277 14821 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:12:20.645298 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:20.582166 14937 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:20.670197 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:20.679060 12564 raft_consensus.cc:2376] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5. Re-sending same reply.
I20260709 12:12:20.678248 14821 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:20.683244 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.636108 14940 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:20.633775 14826 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:20.704721 14821 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:20.706053 14821 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:20.682170 14966 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:20.632577 15008 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.684254 14937 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.734530 12644 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:20.736248 15025 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:20.734561 15052 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:20.722584 14982 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.738765 15041 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:20.756150 14937 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:20.770678 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109091us
I20260709 12:12:20.771042 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:20.772233 12570 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:20.753155 15041 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:20.769285 15000 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:20.769086 15053 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:20.778618 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.777671 15041 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:20.784750 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:20.784512 15051 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
W20260709 12:12:20.795094 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1930) took 2945 ms (client timeout 1800 ms). Trace:
I20260709 12:12:20.768342 14966 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:20.795531 12490 rpcz_store.cc:269] 0709 12:12:17.850082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:17.875325 (+ 25243us) service_pool.cc:224] Handling call
0709 12:12:20.794873 (+2919548us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:20.795017 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":280303488}
I20260709 12:12:20.798367 15057 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:12:20.798815 14997 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.685s	user 0.005s	sys 0.034s
I20260709 12:12:20.800239 13257 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:20.800729 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:20.800309 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1590) took 2515 ms. Trace:
I20260709 12:12:20.804768 12570 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:12:20.803350 12645 rpcz_store.cc:276] 0709 12:12:18.285088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.285319 (+   231us) service_pool.cc:224] Handling call
0709 12:12:18.318763 (+ 33444us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:18.319126 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:18.319153 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:18.327411 (+  8258us) log.cc:844] Serialized 52 byte log entry
0709 12:12:19.058771 (+731360us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:19.108589 (+ 49818us) log.cc:844] Serialized 14 byte log entry
0709 12:12:19.294127 (+185538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:19.294219 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:20.780173 (+1485954us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:20.799974 (+ 19801us) raft_consensus.cc:1685] finished
0709 12:12:20.800092 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:20.800240 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39742,"prepare.queue_time_us":81363,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":164255,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":34589,"spinlock_wait_cycles":52882688,"thread_start_us":913093,"threads_started":2,"wal-append.queue_time_us":733043}
I20260709 12:12:20.802930 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:20.806610 13253 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:12:20.788563 15053 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:20.807914 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1653) took 1368 ms. Trace:
I20260709 12:12:20.787160 15056 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:20.812855 12637 rpcz_store.cc:276] 0709 12:12:19.439716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.440559 (+   843us) service_pool.cc:224] Handling call
0709 12:12:19.440891 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.591478 (+150587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:20.807633 (+1216155us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:20.807840 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344602496}
W20260709 12:12:20.819924 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1631) took 2178 ms (client timeout 2061 ms). Trace:
W20260709 12:12:20.820541 12497 rpcz_store.cc:269] 0709 12:12:18.642079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.008788 (+366709us) service_pool.cc:224] Handling call
0709 12:12:20.819651 (+1810863us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:20.819826 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":998087552}
I20260709 12:12:20.817066 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.843851 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:20.821827 15000 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.826642 15055 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:12:20.854269 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.815076 15056 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:20.857128 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.858961 12645 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:20.848747 14785 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:20.856005 15052 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:20.815868 15057 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:12:20.856602 15008 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:20.874222 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.859477 15056 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:20.878804 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:20.855189 14937 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:12:20.881369 12645 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:12:20.866623 15055 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:20.860411 14826 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:12:20.862636 14940 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:12:20.858067 15053 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.878599 14982 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:20.905824 15057 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:20.923656 15057 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:20.924604 15057 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:20.927772 14914 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:20.928357 14914 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:20.877391 15059 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:12:20.895424 15052 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:20.934396 15025 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:20.949191 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92112.6us
I20260709 12:12:20.950860 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.951839 15055 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:20.951447 15052 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.952556 15055 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:20.953542 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:20.955297 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81130.5us
I20260709 12:12:20.958647 15055 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:20.959137 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:20.960274 15008 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:20.960779 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.961778 15008 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:20.961882 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:20.962484 15008 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:20.963146 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:20.930392 15062 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:12:20.971060 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1232466100000238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 12:12:20.971824 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0071466100000634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
I20260709 12:12:20.990701 15000 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:20.991571 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:20.993078 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:20.994004 15000 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:20.995723 15000 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.000774 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 250) took 1209 ms. Trace:
W20260709 12:12:21.000964 12460 connection.cc:256] RPC call timeout handler was delayed by -1.0861878719999822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
W20260709 12:12:21.001547 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:21.002980 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:20.947777 15025 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:21.004817 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:21.003665 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.005764 12572 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:20.967031 14937 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:20.963565 15008 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.038225 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1651) took 1599 ms. Trace:
I20260709 12:12:21.034601 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:21.027594 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 613055us
W20260709 12:12:21.042219 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.044515 12488 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:21.044090 12642 rpcz_store.cc:276] 0709 12:12:19.438790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.465449 (+ 26659us) service_pool.cc:224] Handling call
0709 12:12:21.037971 (+1572522us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:21.038136 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087739648}
W20260709 12:12:21.044579 12461 connection.cc:256] RPC call timeout handler was delayed by -1.9588001530000838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20260709 12:12:21.027406 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.035804 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:21.025761 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.047221 12494 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:21.013698 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:21.013231 13260 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:21.040388 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:21.034101 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:21.049851 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86764.8us
W20260709 12:12:21.050380 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57388.1us
I20260709 12:12:21.050522 15070 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:21.050767 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:21.051450 15070 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:21.052488 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.013146 15062 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:21.054353 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.042064 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:21.055984 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:21.057107 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:21.059680 14785 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:21.059443 15070 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.060204 14785 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:21.060762 15070 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:21.035388 12490 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:21.062799 15073 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:21.063012 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.063838 12490 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:21.063737 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.064872 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.037163 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.070592 12489 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:21.005475 14982 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:21.080730 12497 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:21.078718 12643 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:21.006304 15017 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.073374 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:21.092000 12644 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:21.067533 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:21.094331 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.010879 14826 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:21.030546 14926 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.566s	user 0.009s	sys 0.004s
W20260709 12:12:21.086707 12461 connection.cc:256] RPC call timeout handler was delayed by -1.9147001530000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20260709 12:12:21.106163 12461 connection.cc:256] RPC call timeout handler was delayed by -1.8400001530001191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20260709 12:12:21.033143 14986 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:21.050323 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.118853 13257 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:21.118976 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:21.028739 15042 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:21.088660 15066 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:21.087637 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:21.096064 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:21.099434 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:21.030516 14386 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.085309 14937 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:21.027884 15063 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.100309 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:21.106252 14889 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:21.143553 12645 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:21.101830 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:21.120327 14927 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:21.147927 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97220.6us
I20260709 12:12:21.065616 12490 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:21.108726 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.119558 13259 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:21.119853 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:21.079936 14909 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.121413 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:21.153786 12639 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:21.083918 14884 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.098207 14982 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.023984 14914 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.129161 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.178047 12495 raft_consensus.cc:2376] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4. Re-sending same reply.
I20260709 12:12:21.179036 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.179694 12495 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:21.132062 12646 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:21.135337 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.100255 12644 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:21.099768 14930 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:21.099514 14826 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:21.027979 14876 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:21.061189 14785 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:21.124240 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60615.1us
I20260709 12:12:21.176636 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.138000 14926 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:21.145742 14889 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:21.144198 15007 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:12:21.182754 14914 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:21.187611 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52365.7us
I20260709 12:12:21.206657 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:21.213423 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.147877 14927 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:21.231552 12497 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5. Re-sending same reply.
I20260709 12:12:21.284484 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:21.238407 12494 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:21.133175 14986 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.109270 15025 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.291508 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:21.282959 12495 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:21.281351 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:21.282174 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 132598us
I20260709 12:12:21.294361 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:21.295616 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 168995us
I20260709 12:12:21.231349 15059 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:21.230976 14826 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:21.222994 15041 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.156193 15053 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:12:21.291352 15060 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:12:21.258369 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:21.287002 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:21.344525 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:21.344688 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57810.5us
I20260709 12:12:21.104700 15017 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.143106 14386 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.297504 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.296631 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:21.329802 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 246554us
I20260709 12:12:21.239257 14889 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:21.162477 14909 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.288795 14927 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:21.359012 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 126317us
I20260709 12:12:21.220227 14884 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.148434 15063 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:21.357613 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61079.4us
I20260709 12:12:21.287416 15061 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:21.352870 12494 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:21.369499 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.370445 12497 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5. Re-sending same reply.
I20260709 12:12:21.310576 15059 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:21.302678 14966 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:21.379428 12490 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:12:21.328126 15072 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:21.378100 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:21.386737 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:21.351503 14811 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.386407 15072 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.387737 12490 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:21.401715 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.387573 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:21.398305 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 252184us
I20260709 12:12:21.419616 12494 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:12:21.328217 15075 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:12:21.359335 15007 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:21.466621 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.378772 15059 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:21.485407 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:21.351387 14937 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.401996 15025 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:21.494503 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115986us
I20260709 12:12:21.404151 15086 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:21.417371 15082 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:21.505064 12495 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:21.502730 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122520us
I20260709 12:12:21.518710 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:21.480562 15066 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:21.478816 15007 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:21.544423 15018 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:21.547005 15010 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:21.562026 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 246) took 1898 ms. Trace:
I20260709 12:12:21.563376 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:21.566375 12494 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:12:21.568460 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.359691 15077 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.140s
I20260709 12:12:21.422747 14811 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:21.534121 15056 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.650s	user 0.014s	sys 0.000s
I20260709 12:12:21.409312 15061 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.125s
I20260709 12:12:21.523810 15042 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:12:21.620823 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 180754us
I20260709 12:12:21.622159 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:21.514034 15086 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:21.577664 12492 rpcz_store.cc:276] 0709 12:12:19.663294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.702315 (+ 39021us) service_pool.cc:224] Handling call
0709 12:12:19.702690 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.839146 (+136456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:19.839187 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:19.839232 (+    45us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:19.839267 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:21.561612 (+1722345us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:21.561716 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:21.561940 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":313323264}
I20260709 12:12:21.588404 12490 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:12:21.626571 12490 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 5
I20260709 12:12:21.626951 12490 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:12:21.627277 12490 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:21.502717 15085 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:12:21.578603 15057 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.653s	user 0.014s	sys 0.000s
I20260709 12:12:21.640130 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.503296 15075 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20260709 12:12:21.577386 15007 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:21.643364 12497 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:21.535210 15082 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.629109 15056 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:21.649125 15081 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
W20260709 12:12:21.649999 15083 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:12:21.650404 15079 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:21.651067 15088 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.658082 14909 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:21.659221 15072 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:21.645962 15085 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:21.577519 15010 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:21.666405 12488 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.621s	user 0.012s	sys 0.000s
I20260709 12:12:21.704663 12488 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:21.589429 14982 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:21.675298 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:21.636631 15041 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:21.583287 15077 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20260709 12:12:21.632402 15086 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:21.689435 12571 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:21.577272 15084 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.170s
I20260709 12:12:21.694832 12565 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:21.697211 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.675767 14966 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:12:21.694931 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 267367us
I20260709 12:12:21.679015 15074 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:21.670837 12639 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:21.677515 14884 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:21.680675 14930 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.717844 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1849) took 1271 ms. Trace:
I20260709 12:12:21.725207 12571 rpcz_store.cc:276] 0709 12:12:20.446464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:20.449839 (+  3375us) service_pool.cc:224] Handling call
0709 12:12:21.563158 (+1113319us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:21.717765 (+154607us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4363520}
I20260709 12:12:21.608310 15018 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.685766 15095 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.728761 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:21.728964 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53921.1us
W20260709 12:12:21.748713 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.695621 15079 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:21.703588 14386 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:21.663715 15057 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:21.785758 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57134.4us
I20260709 12:12:21.793161 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.769236 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1829) took 2406 ms. Trace:
I20260709 12:12:21.788836 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:21.773484 14982 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:12:21.798408 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.788946 15099 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.776368 14840 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:21.795213 12566 rpcz_store.cc:276] 0709 12:12:19.362525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.837598 (+475073us) service_pool.cc:224] Handling call
0709 12:12:19.939404 (+101806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.939680 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:12:21.515119 (+1575439us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b1800436f00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:12:21.515316 (+   197us) log.cc:844] Serialized 14 byte log entry
0709 12:12:21.768655 (+253339us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:21.768757 (+   102us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:21.768832 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:21.768941 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:21.769137 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1128,"prepare.queue_time_us":37130,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":6758,"raft.queue_time_us":55839,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":4960,"spinlock_wait_cycles":2555776,"thread_start_us":262534,"threads_started":2,"wal-append.queue_time_us":271869}
W20260709 12:12:21.860080 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 341457us
W20260709 12:12:21.855015 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 275348us
I20260709 12:12:21.001272 12493 rpcz_store.cc:276] 0709 12:12:19.791654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.806748 (+ 15094us) service_pool.cc:224] Handling call
0709 12:12:19.807185 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:19.807465 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:12:19.958998 (+151533us) log.cc:844] Serialized 14 byte log entry
0709 12:12:21.000327 (+1041329us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:21.000395 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:21.000449 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:21.000547 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:21.000710 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1040959,"prepare.queue_time_us":228222,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":1042775,"thread_start_us":132988,"threads_started":1}
W20260709 12:12:21.851876 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63172.9us
I20260709 12:12:21.863631 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:21.852864 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58196.9us
I20260709 12:12:21.864759 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:21.853842 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:21.865828 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:21.889175 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:21.892437 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:21.893074 12461 connection.cc:256] RPC call timeout handler was delayed by -1.5214001530000587s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
I20260709 12:12:21.852384 15061 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.789476 15041 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20260709 12:12:21.783186 14930 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.862497 15042 raft_consensus.cc:2764] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:12:21.927729 14909 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.955685 15100 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:21.934643 15062 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:21.956698 15100 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.956277 15107 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:21.955963 15089 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
W20260709 12:12:21.958036 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94507.2us
I20260709 12:12:21.962752 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:21.937326 15055 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:12:21.947862 15080 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.271s	user 0.001s	sys 0.000s
I20260709 12:12:21.695371 15088 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.973661 15041 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:21.975068 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:21.957993 15094 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:21.944043 15074 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:21.977010 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.001s	sys 0.010s
W20260709 12:12:21.984052 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62398.9us
I20260709 12:12:21.976043 14724 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:21.957373 15100 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.958382 15109 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.968341 14716 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:21.944214 14909 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.975577 15108 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.161s
I20260709 12:12:21.784231 15095 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.957894 15042 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:21.926499 15061 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.867280 15099 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.956903 15062 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.863821 15010 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.957587 15107 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:21.936977 15079 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:21.939738 15085 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:22.099404 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210314us
I20260709 12:12:22.114331 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1708) took 1113 ms. Trace:
I20260709 12:12:22.271494 12565 rpcz_store.cc:276] 0709 12:12:21.000982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:21.001875 (+   893us) service_pool.cc:224] Handling call
0709 12:12:22.114033 (+1112158us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:22.114245 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1486216960}
W20260709 12:12:22.272010 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.006s	sys 0.018s
W20260709 12:12:22.274996 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 974923us
W20260709 12:12:22.077934 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102923us
I20260709 12:12:22.279165 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:22.201678 15098 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
W20260709 12:12:22.154313 15115 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:12:22.209929 15007 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.528s	user 0.011s	sys 0.002s
I20260709 12:12:22.179019 12571 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:22.092881 15092 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:12:22.271885 14927 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.903s	user 0.010s	sys 0.002s
I20260709 12:12:22.115018 15057 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.128459 14986 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.075716 15056 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:22.100673 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.314456 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:22.297502 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1655) took 1259 ms. Trace:
W20260709 12:12:22.086259 14914 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.790s	user 0.013s	sys 0.000s
W20260709 12:12:22.076884 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140506us
I20260709 12:12:22.285916 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:22.024266 15113 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:12:22.076287 15053 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:22.172498 14811 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.154618 15106 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.280961 14840 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:22.122768 15069 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:22.124655 15025 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.610s	user 0.010s	sys 0.000s
W20260709 12:12:22.327917 14889 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.959s	user 0.013s	sys 0.001s
I20260709 12:12:22.086833 15070 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.342550 14889 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.343253 15070 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.345983 15122 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.346830 15122 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:22.347330 14826 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.998s	user 0.009s	sys 0.004s
I20260709 12:12:22.115703 15018 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.067278 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 323) took 1909 ms. Trace:
I20260709 12:12:22.348282 14826 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.348544 13256 rpcz_store.cc:276] 0709 12:12:20.158010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:20.163808 (+  5798us) service_pool.cc:224] Handling call
0709 12:12:20.171315 (+  7507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:20.198458 (+ 27143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:22.020034 (+1821576us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:22.029214 (+  9180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60617600}
I20260709 12:12:22.129379 15088 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:22.229869 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.162128 14930 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.025818 15082 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.297345 15001 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.298120 14927 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:22.301708 15086 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.000s
I20260709 12:12:22.303339 15107 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.280754 14941 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.309739 15094 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:22.310592 15121 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.024068 14567 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.270740 15074 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:22.320698 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.407231 12492 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:22.316495 12489 rpcz_store.cc:276] 0709 12:12:21.037822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:21.071289 (+ 33467us) service_pool.cc:224] Handling call
0709 12:12:21.071730 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:21.295171 (+223441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:21.295221 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:21.349797 (+ 54576us) log.cc:844] Serialized 26 byte log entry
0709 12:12:21.694647 (+344850us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:21.694690 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:21.694761 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:22.297278 (+602517us) raft_consensus.cc:1685] finished
0709 12:12:22.297306 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:22.297445 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15711,"raft.queue_time_us":197505,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":36932,"thread_start_us":446118,"threads_started":2,"wal-append.queue_time_us":534708}
I20260709 12:12:22.156509 15017 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.406112 14567 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:22.414299 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:22.415323 12571 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:22.416469 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.417171 12571 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:22.325136 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44927.6us
I20260709 12:12:22.324321 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:22.418525 12563 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:22.168445 15063 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.319399 14785 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:22.330458 15106 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.331250 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.332167 14986 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:12:22.333458 14811 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:22.333897 15120 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:22.333770 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:22.336292 15100 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.283820 15111 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:12:22.322808 14926 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:12:22.349877 15070 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.341814 15069 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.355417 14909 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.356889 14889 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.280823 15109 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:22.360039 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 326) took 1355 ms (client timeout 1594 ms). Trace:
I20260709 12:12:22.428609 15063 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:22.361102 15062 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.363286 14826 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.436223 14811 consensus_queue.cc:1243] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:12:22.368196 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.450814 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:22.277565 14716 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.373494 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:22.453115 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:22.374768 12564 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:12:22.380831 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.457096 12490 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:22.387367 15122 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.388665 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.325811 15008 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:12:22.333590 15119 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:12:22.397711 12572 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:22.476117 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.476900 12572 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:22.478319 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.478436 15010 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:22.479031 12572 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:22.480118 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.398074 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.480824 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:22.482127 12567 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:22.481884 15074 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.483341 15074 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:22.292147 15007 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.485868 15007 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.410625 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.486337 15099 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.419116 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:22.487222 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68190.6us
I20260709 12:12:22.423321 12536 leader_election.cc:400] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:22.490044 12536 leader_election.cc:403] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:12:22.488648 12572 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:22.409894 15059 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.905s	user 0.007s	sys 0.008s
I20260709 12:12:22.430657 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.430442 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.491991 12571 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:12:22.432313 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.396693 14930 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:12:22.422583 14968 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.495501 14968 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.496521 15082 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:22.397280 14941 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.497903 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.460000 15102 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:22.441216 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.498726 14968 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:22.499751 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76547.3us
I20260709 12:12:22.499876 15062 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:22.417860 15094 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.501083 14941 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.498853 15102 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:22.502699 15094 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.360656 14821 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:22.503368 14821 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:22.503679 15007 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:22.504268 15007 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:22.443460 13259 rpcz_store.cc:269] 0709 12:12:21.004916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:21.005154 (+   238us) service_pool.cc:224] Handling call
0709 12:12:22.359814 (+1354660us) spinlock_profiling.cc:243] Waited 70.5 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:22.359964 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162139008}
I20260709 12:12:22.504860 15007 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:22.505367 14967 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:22.504886 14821 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.506307 15059 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.506752 14821 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:22.446216 13262 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 5 LEARNER]: Advancing to term 7
I20260709 12:12:22.509958 13256 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:22.363539 15061 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.397729 15086 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.512472 15086 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.430871 15088 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:22.438264 14884 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:22.457708 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.397477 15001 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.459548 13255 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:12:22.459745 15110 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:12:22.459834 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.460584 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.462630 14716 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.462996 15069 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.463479 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.465416 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:22.453035 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:22.465853 15095 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.438342 15120 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:22.467842 14927 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.425876 14785 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.418097 15121 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:22.420325 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96096.6us
I20260709 12:12:22.536015 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.536988 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:22.493734 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:22.491143 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:22.492677 12563 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:22.498188 13253 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:22.500589 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:22.503299 15102 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.540302 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:22.430189 14937 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
W20260709 12:12:22.442036 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108409us
I20260709 12:12:22.445278 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.498730 12493 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:22.522593 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:22.542150 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:22.525578 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.546636 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.548413 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.549983 13254 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:12:22.550474 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.551155 14927 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:22.543049 15086 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:22.554664 14927 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.554710 12569 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:12:22.556943 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.557655 14785 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.558174 12569 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:22.560424 15121 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.549515 15001 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.562042 15120 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.565148 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.567375 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:22.568195 13253 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:22.544457 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:22.570601 12564 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:22.526489 13262 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 7 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:12:22.532209 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66894.5us
I20260709 12:12:22.532300 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:22.529810 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:22.532907 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.456418 15063 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:22.531484 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:22.536710 12572 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:22.505887 14967 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.529386 15088 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.583792 14967 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:22.584556 15017 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:22.585105 15017 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:22.585377 15063 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.585752 15088 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:22.586138 15086 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:12:22.586723 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43915us
I20260709 12:12:22.580869 15109 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.580112 15069 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:12:22.541391 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47801.8us
I20260709 12:12:22.587685 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:22.573611 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73105.5us
I20260709 12:12:22.552057 12571 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:22.522303 15106 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.569991 14937 consensus_queue.cc:1243] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:12:22.597924 14941 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:22.570271 15095 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:22.579962 15100 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:12:22.583204 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.591508 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:22.606642 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:22.595721 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:22.596377 14876 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:22.597114 15111 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:12:22.536362 15085 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:22.600005 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70289.9us
I20260709 12:12:22.557459 14927 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.614693 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:22.591395 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:22.592011 14884 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:22.616802 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5090629920000538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2959s for a response
I20260709 12:12:22.611177 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:22.580868 15102 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:22.624835 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:22.622174 12643 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:12:22.628343 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:22.629273 12643 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:22.632470 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:22.633649 12646 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:22.598968 14941 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:22.630673 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:22.636245 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.656073 12644 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:22.636785 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:22.639428 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66745.3us
I20260709 12:12:22.623907 12564 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:22.599514 15109 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:22.605746 15095 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:22.662240 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.609932 14785 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:22.663805 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.585776 15062 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:12:22.676484 12637 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:22.612785 14966 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:22.670931 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.662034 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:22.619640 14986 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:22.620047 15111 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:12:22.587646 15017 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.676231 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.678669 12638 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:22.655361 14937 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:22.679175 12641 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:22.673446 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:22.680162 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62259.4us
I20260709 12:12:22.706054 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.655596 14941 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:22.612565 15085 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:22.714529 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:22.683249 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.683322 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:22.685593 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.623646 15094 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:22.687306 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.689020 12645 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:22.689281 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.690768 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:22.676738 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70199.4us
I20260709 12:12:22.692914 12497 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:22.717945 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:22.677080 14785 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.690210 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:22.700400 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.700536 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:22.701665 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.671052 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.623368 14884 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:22.724020 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58576.3us
I20260709 12:12:22.724467 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:22.725098 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.709945 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.710755 12643 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:22.676820 14927 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:22.704742 15075 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:22.718492 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:22.654027 15091 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:22.721488 12639 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:22.737138 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.623575 15056 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260709 12:12:22.742064 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.706729 14937 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:22.726361 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.672472 15095 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:22.685829 14986 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:22.724875 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:22.730213 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.735931 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.748788 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:22.681056 15082 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:12:22.738379 12642 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:22.708580 15122 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.720633 15085 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:22.754442 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.769137 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.769537 12643 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:22.768425 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.761735 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.719416 15094 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:22.770936 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.771401 15094 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:22.772020 12643 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:22.737493 14716 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:22.762871 12496 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:22.722765 14785 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:22.724102 15109 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:22.723652 15069 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:22.731132 15008 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:22.720726 15006 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.775262 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.768491 15057 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.776441 12568 pending_rounds.cc:85] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:12:22.789216 12568 pending_rounds.cc:107] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:12:22.777436 12488 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:22.734338 15075 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.744933 15091 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:22.737109 15106 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:22.751659 14937 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:22.761057 14986 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:22.731594 15097 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:12:22.760147 14966 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:22.773509 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61129.5us
I20260709 12:12:22.791714 12568 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:22.791888 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:22.795369 12536 leader_election.cc:400] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:22.805558 12536 leader_election.cc:403] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:22.807243 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:22.808557 12492 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:12:22.809062 12492 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 2
I20260709 12:12:22.809314 12492 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:12:22.809557 12492 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:22.790565 15075 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:22.777997 14716 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:22.774950 15122 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.810812 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:22.796176 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71371.6us
I20260709 12:12:22.790726 15091 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:22.821970 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.823163 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.824227 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.826236 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:22.827482 12492 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:22.785122 15069 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:22.828815 15069 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:22.829067 12492 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:22.787390 15006 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.776850 14941 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:22.809748 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.811899 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:22.818454 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:22.766258 15136 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:12:22.800932 15074 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.092s
I20260709 12:12:22.791178 15008 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:22.801317 14909 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:22.872054 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.874814 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:22.903721 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:22.952437 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:22.953976 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:22.904775 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:22.955354 13261 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:22.906077 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:22.907294 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95480.4us
I20260709 12:12:22.955374 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:22.819550 15130 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:12:22.799677 15057 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:22.810979 15062 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.155s
I20260709 12:12:22.874446 14811 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:22.857213 14567 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.009927 14811 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:23.010838 14811 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.011670 14567 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.012276 14811 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:22.830355 14966 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.013682 12570 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:22.882304 15129 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.141s	user 0.000s	sys 0.006s
I20260709 12:12:22.947868 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:22.876519 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:22.957949 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:22.958622 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:22.878633 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83446us
I20260709 12:12:23.020839 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:22.879042 15056 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:22.959170 14930 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:23.004664 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:23.025496 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:23.005563 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:23.026867 12493 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:23.014392 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105882us
I20260709 12:12:22.924767 12488 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:22.955394 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 236739us
I20260709 12:12:22.866761 14884 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:22.814729 15100 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:22.814572 15111 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:23.046058 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:23.050202 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:22.800897 15106 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.022665 12494 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:12:23.062517 12488 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260709 12:12:22.854305 15128 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 12:12:22.950693 14909 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:23.037726 12393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 223) took 21593 ms (client timeout 15000 ms). Trace:
W20260709 12:12:23.045361 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 128322us
I20260709 12:12:23.046023 15142 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:23.045662 15136 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:22.817982 14716 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:23.054869 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:23.055099 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:22.868080 15059 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:22.888602 14967 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.009125 15091 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.105306 12569 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:22.946975 15108 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:22.892930 15007 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.127816 12493 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:23.025014 14930 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.121817 12488 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:23.126713 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:23.051250 14941 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.059834 14567 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:23.236205 14567 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.236727 14567 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.240909 15006 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.245103 15109 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.245759 15109 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:23.251353 14724 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:23.251976 14724 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.254645 14930 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.010906 15069 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.121578 15142 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:23.121179 15100 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:23.192227 15057 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.035219 15079 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260709 12:12:23.194128 15147 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:23.196710 12571 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:23.121529 14909 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.138195 13256 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:23.183216 14821 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:23.218873 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186585us
I20260709 12:12:23.295470 12536 leader_election.cc:400] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
W20260709 12:12:23.223711 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.296048 12536 leader_election.cc:403] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:12:23.221297 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:23.212378 15008 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.212805 15107 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:23.230206 15007 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.210258 14884 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:23.212272 15141 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
W20260709 12:12:23.306551 15095 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.532s	user 0.006s	sys 0.006s
I20260709 12:12:23.315279 15095 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.231259 13257 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:23.079674 15102 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:23.266336 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:23.275230 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.029s	sys 0.055s
I20260709 12:12:23.150234 15056 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:23.278980 12496 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:12:23.280109 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.044s	sys 0.069s
I20260709 12:12:23.333573 13259 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:12:23.333662 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.045s	sys 0.069s
I20260709 12:12:23.300390 14821 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.303004 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:23.335275 13261 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:23.214596 14967 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:23.120939 12393 rpcz_store.cc:269] 0709 12:12:01.385284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:01.473457 (+ 88173us) service_pool.cc:224] Handling call
0709 12:12:22.978342 (+21504885us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:18.614187 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:18.678573 (+ 64386us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:18.678647 (+    74us) write_op.cc:435] Acquired schema lock
0709 12:12:18.678691 (+    44us) tablet.cc:662] Decoding operations
0709 12:12:19.000513 (+321822us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:19.000629 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:19.000679 (+    50us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:12:19.030783 (+ 30104us) tablet.cc:701] Row locks acquired
0709 12:12:19.077712 (+ 46929us) write_op.cc:260] PREPARE: finished
0709 12:12:19.112849 (+ 35137us) write_op.cc:270] Start()
0709 12:12:19.114725 (+  1876us) write_op.cc:276] Timestamp: P: 1783599139098978 usec, L: 0
0709 12:12:19.114753 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:12:19.178307 (+ 63554us) log.cc:844] Serialized 1317 byte log entry
0709 12:12:19.745146 (+566839us) spinlock_profiling.cc:243] Waited 8.08 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:12:20.598715 (+853569us) op_driver.cc:464] REPLICATION: finished
0709 12:12:20.791479 (+192764us) write_op.cc:301] APPLY: starting
0709 12:12:20.796354 (+  4875us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:20.811479 (+ 15125us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:20.811549 (+    70us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:20.814719 (+  3170us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:20.861060 (+ 46341us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:12:20.865062 (+  4002us) write_op.cc:312] APPLY: finished
0709 12:12:20.882949 (+ 17887us) log.cc:844] Serialized 35 byte log entry
0709 12:12:20.883196 (+   247us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:20.883310 (+   114us) write_op.cc:454] Released schema lock
0709 12:12:20.883376 (+    66us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:20.883853 (+   477us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:20.883882 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2306443,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":181412,"apply.run_cpu_time_us":4030,"apply.run_wall_time_us":100002,"num_ops":3,"prepare.queue_time_us":411085,"prepare.run_cpu_time_us":69789,"prepare.run_wall_time_us":1461423,"raft.queue_time_us":420048,"raft.run_cpu_time_us":68759,"raft.run_wall_time_us":140579,"replication_time_us":1483746,"spinlock_wait_cycles":18586496,"thread_start_us":1419264,"threads_started":4,"wal-append.queue_time_us":769021,"wal-append.run_cpu_time_us":42201,"wal-append.run_wall_time_us":1963711}]]}
I20260709 12:12:23.211524 15106 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.268718 15052 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:23.311975 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.295557 15099 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:23.328936 15102 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:23.318133 15107 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.349958 15102 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.351136 15102 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:23.352000 15107 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:23.324230 15085 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.000s
I20260709 12:12:23.353155 15085 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.328969 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:23.319036 15079 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:23.330438 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.030s	sys 0.055s
W20260709 12:12:23.302541 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.201560 14986 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.280323 12493 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:23.317451 15095 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.273258 15006 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 12:12:23.310287 14876 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:23.382226 14876 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:23.383500 14876 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.384917 14876 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:23.346000 14821 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.273167 15108 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.274611 15094 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.310024 13256 raft_consensus.cc:606] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:23.370635 15079 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:23.348913 15142 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:23.358392 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:23.468242 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.020s	sys 0.073s
W20260709 12:12:23.468647 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.021s	sys 0.073s
I20260709 12:12:23.310580 14884 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:12:23.329818 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63578us
I20260709 12:12:23.361558 15085 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:23.362625 15149 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:12:23.348965 15052 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.349380 15099 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.364094 15070 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:23.368698 15062 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:23.321527 15069 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:23.332405 15057 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 12:12:23.375767 14967 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:23.377038 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:23.516426 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 492050us
I20260709 12:12:23.321461 15073 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5.
I20260709 12:12:23.292762 15100 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:23.379776 15075 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.000s
I20260709 12:12:23.388413 12571 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:23.347319 14724 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:23.467762 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.333235 15082 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:12:23.432453 14927 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.651s	user 0.007s	sys 0.007s
I20260709 12:12:23.471305 12489 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:23.469532 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1718) took 1045 ms. Trace:
I20260709 12:12:23.476843 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:23.478111 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 423748us
I20260709 12:12:23.575244 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:23.369145 15102 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.097s
I20260709 12:12:23.467325 13256 raft_consensus.cc:614] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:23.574276 12563 rpcz_store.cc:276] 0709 12:12:22.423840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.553616 (+129776us) service_pool.cc:224] Handling call
0709 12:12:22.554111 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:22.554394 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:12:22.714184 (+159790us) log.cc:844] Serialized 14 byte log entry
0709 12:12:23.469116 (+754932us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:23.469181 (+    65us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:23.469231 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:23.469317 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:23.469464 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13731,"prepare.queue_time_us":196560,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":4009,"raft.queue_time_us":2413,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":35375,"thread_start_us":913848,"threads_started":2}
I20260709 12:12:23.579071 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:23.534482 15053 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:23.538504 14986 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.543334 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 330) took 1109 ms. Trace:
I20260709 12:12:23.580678 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:23.581072 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:23.518899 15145 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.001s	sys 0.001s
I20260709 12:12:23.582131 15099 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.545385 15142 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:23.586268 13253 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 3 LEARNER]: Advancing to term 5
I20260709 12:12:23.548264 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:23.588040 15122 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:23.588584 15053 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:23.589668 15053 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.591297 15053 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:23.592679 12563 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:23.573523 15079 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.594831 15095 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:23.595367 15094 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:23.548453 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:23.596097 14724 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:23.596506 12572 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.598064 12495 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:23.552196 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:23.602038 12564 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
W20260709 12:12:23.602131 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1646) took 4231 ms (client timeout 1553 ms). Trace:
W20260709 12:12:23.602816 12640 rpcz_store.cc:269] 0709 12:12:19.370381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:19.405574 (+ 35193us) service_pool.cc:224] Handling call
0709 12:12:23.601856 (+4196282us) spinlock_profiling.cc:243] Waited 3.27 s on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:23.602033 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7519447936}
I20260709 12:12:23.603492 12564 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:12:23.555436 15070 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:23.606734 15070 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:23.508273 14937 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.698s	user 0.006s	sys 0.011s
I20260709 12:12:23.608945 14937 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.577426 12566 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:12:23.579162 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102407us
I20260709 12:12:23.531242 15091 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.577337 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:23.580809 13257 rpcz_store.cc:276] 0709 12:12:22.433755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.433904 (+   149us) service_pool.cc:224] Handling call
0709 12:12:23.543051 (+1109147us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54005c4c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:23.543259 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1581309184}
I20260709 12:12:23.581589 13259 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:23.583387 15100 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.594187 12563 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:23.550123 15063 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:23.554175 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:23.544214 15108 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.611305 14821 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:23.614403 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66263.5us
I20260709 12:12:23.483471 14884 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:23.578209 14927 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:23.621127 12572 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:23.623354 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46118.5us
I20260709 12:12:23.626957 15122 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:23.510499 15109 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:23.635155 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.649542 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:23.631174 15142 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:23.639355 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:23.658722 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:23.632649 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:23.619005 14811 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:23.618410 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:23.667570 12461 leader_election.cc:400] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:23.643046 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:23.616389 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:23.571187 15075 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.662900 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:23.643819 12643 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
W20260709 12:12:23.674422 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.012s	sys 0.030s
W20260709 12:12:23.641120 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.005s	sys 0.032s
W20260709 12:12:23.671000 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52715.3us
W20260709 12:12:23.675031 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.013s	sys 0.031s
W20260709 12:12:23.675318 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.005s	sys 0.032s
W20260709 12:12:23.675568 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 555865us
W20260709 12:12:23.676025 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 717381us
W20260709 12:12:23.679890 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:23.680609 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.683300 12461 leader_election.cc:403] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260709 12:12:23.664753 14785 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.877s	user 0.011s	sys 0.003s
W20260709 12:12:23.688608 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.692713 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:23.694068 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:23.695045 12566 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 3.
I20260709 12:12:23.696717 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:23.697599 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:23.644362 14567 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.699674 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:23.700779 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84502.7us
I20260709 12:12:23.703182 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:23.705516 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:23.708364 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 340064us
I20260709 12:12:23.638713 15063 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.672278 15095 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:23.732604 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:23.732239 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:23.735489 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:23.736548 12567 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20260709 12:12:23.686312 14990 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:23.737468 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:23.665459 15156 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260709 12:12:23.740451 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:23.743597 12495 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:23.747208 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104271us
W20260709 12:12:23.750008 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46915.7us
I20260709 12:12:23.749470 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:23.707543 14967 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:23.760548 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:23.770262 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64860.8us
I20260709 12:12:23.762930 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:23.772126 12488 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 2.
I20260709 12:12:23.769997 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:23.771194 12495 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:23.730549 15095 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.780134 12494 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:23.783144 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:23.781008 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:23.778848 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:23.786839 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:23.787765 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:23.788394 12640 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.784523 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:23.736253 15062 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:23.736148 15056 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:23.706763 14785 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:23.740887 14990 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.758798 15085 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:23.828928 14990 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.831871 15109 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.834460 15052 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.835245 15052 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:23.835876 15052 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.842518 14821 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.843173 14567 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.843912 14567 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:23.844333 14937 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.770188 15073 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:23.845996 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:23.786772 12496 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:23.801976 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:23.848057 12493 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:23.771057 15094 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:23.739588 15057 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.076s
I20260709 12:12:23.849319 15094 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.849961 15094 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:23.850188 14930 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.595s	user 0.011s	sys 0.002s
I20260709 12:12:23.733739 15041 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 12:12:23.852284 15075 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:23.803390 14941 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.569s	user 0.010s	sys 0.004s
I20260709 12:12:23.822489 14967 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:23.854266 15063 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:23.854738 14716 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.000s
I20260709 12:12:23.855225 14716 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:23.856413 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:23.825279 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65163.6us
I20260709 12:12:23.858340 15056 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.860986 15056 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.825981 15010 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:12:23.861832 15062 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.864419 15062 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.864108 15063 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:12:23.867583 13259 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6.
I20260709 12:12:23.742317 15107 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:23.846360 15073 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:23.850759 14930 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.869189 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:23.861187 15075 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:12:23.805137 15114 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 12:12:23.877045 12395 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 3, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:12:23.798909 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66420.2us
I20260709 12:12:23.815093 15025 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:23.821604 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50115.6us
I20260709 12:12:23.791126 12494 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:23.790253 12637 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:23.791870 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:23.859031 15052 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:23.792138 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:23.879293 13253 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 5 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:23.803802 15095 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:23.897927 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:23.815515 15156 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:23.921289 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:23.797554 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:23.896415 14876 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.511s	user 0.013s	sys 0.000s
W20260709 12:12:23.915205 12497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.043s	sys 0.028s
I20260709 12:12:23.894261 15063 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:23.898891 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:23.800863 15146 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:12:23.789592 15154 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:23.923162 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125723us
I20260709 12:12:23.914934 13262 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:23.888623 15062 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:23.815622 15086 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:23.894913 13258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.014s	sys 0.038s
I20260709 12:12:23.798687 12495 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:12:23.880199 15061 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.012s	sys 0.013s
W20260709 12:12:23.930272 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 138212us
I20260709 12:12:23.895478 14716 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.936501 15069 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:23.939781 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:23.922477 15156 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:23.946429 15162 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:23.914193 15025 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:23.950676 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:23.930160 12497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.043s	sys 0.029s
W20260709 12:12:23.958684 15008 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.003s	sys 0.041s
I20260709 12:12:23.937772 15059 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:23.929479 14937 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:23.952894 15063 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:23.971017 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:24.046729 12565 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:23.973174 12563 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:23.977747 12566 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:23.979997 12572 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:23.947006 15018 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
W20260709 12:12:23.982174 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.983389 14930 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.965771 15086 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:23.966239 13258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.015s	sys 0.038s
I20260709 12:12:24.018733 15091 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:23.951470 15121 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:12:23.966025 15062 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:23.991261 15160 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.250s	user 0.003s	sys 0.000s
W20260709 12:12:23.991694 14909 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.688s	user 0.013s	sys 0.000s
I20260709 12:12:24.083400 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 1986) took 1274 ms. Trace:
I20260709 12:12:23.938550 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:24.083875 12497 rpcz_store.cc:276] 0709 12:12:22.809035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.054371 (+245336us) service_pool.cc:224] Handling call
0709 12:12:23.116192 (+ 61821us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:23.339329 (+223137us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:23.339357 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:23.354318 (+ 14961us) log.cc:844] Serialized 26 byte log entry
0709 12:12:24.014212 (+659894us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:24.014274 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.014389 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:24.083128 (+ 68739us) raft_consensus.cc:1685] finished
0709 12:12:24.083152 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:24.083319 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76141,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":154,"thread_start_us":881764,"threads_started":2,"wal-append.queue_time_us":721739}
W20260709 12:12:23.949819 14966 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.006s	sys 0.015s
W20260709 12:12:23.955643 15136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.020s	sys 0.024s
I20260709 12:12:23.921556 15100 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20260709 12:12:24.014181 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63680.9us
I20260709 12:12:23.996788 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:24.092621 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:24.013047 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73348.1us
W20260709 12:12:24.001334 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.018029 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:24.092330 15136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.865s	user 0.020s	sys 0.024s
I20260709 12:12:24.099650 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 234) took 1389 ms. Trace:
I20260709 12:12:24.106122 12638 rpcz_store.cc:276] 0709 12:12:22.710232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.710354 (+   122us) service_pool.cc:224] Handling call
0709 12:12:22.731290 (+ 20936us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:22.747682 (+ 16392us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:22.747715 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:23.243264 (+495549us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:12:23.243793 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 12:12:23.369959 (+126166us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:23.370002 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:23.370111 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:23.990341 (+620230us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:24.099396 (+109055us) raft_consensus.cc:1685] finished
0709 12:12:24.099420 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:24.099575 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15978,"raft.queue_time_us":43143,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":4521,"spinlock_wait_cycles":51959936,"thread_start_us":125719,"threads_started":1,"wal-append.queue_time_us":425442}
I20260709 12:12:24.107532 14990 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.083535 15052 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:24.090493 14966 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.006s	sys 0.015s
I20260709 12:12:24.102722 14785 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:24.102041 15063 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.112116 14785 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:24.112916 14785 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.086994 14937 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:24.017143 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:24.116312 14785 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:24.117007 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:24.119570 14876 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:24.122014 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.091434 15156 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.122893 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:24.123493 15156 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:24.126240 14930 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:24.127229 14930 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:24.127794 14930 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.086045 14909 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.129230 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:24.129956 14909 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:24.130100 12492 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:24.131729 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:24.132407 12564 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.048442 15073 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.127539 15052 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.143415 15052 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:24.145292 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.145969 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.146879 12645 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 12:12:24.150529 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.950042 15154 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:24.151069 14930 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:23.958060 12395 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:24.172619 12395 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 4, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:24.174062 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.174782 12645 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.175511 12395 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:12:23.992209 15061 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.013s	sys 0.013s
W20260709 12:12:24.020627 15008 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.003s	sys 0.041s
W20260709 12:12:24.048972 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:23.947299 15000 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 12:12:24.018977 15025 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:24.013603 15162 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:24.020210 14986 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:24.012601 14811 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:24.180760 15162 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:24.181473 14811 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.183959 12638 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:24.184715 14811 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:23.984323 15006 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:24.025696 15059 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:24.185444 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.185950 15059 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:24.023237 14821 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:24.078558 15017 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.189913 14821 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:24.190374 14821 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.191087 12395 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:24.069703 15086 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:24.080224 15062 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:24.088238 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 149798us
I20260709 12:12:24.151259 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:24.152187 15154 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:23.950843 15147 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:24.153350 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.154974 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:12:24.161343 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.168193 12564 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:23.962906 13262 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:24.192878 12493 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:24.193019 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:24.198297 12645 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:24.232478 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:24.202625 12640 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:12:24.206805 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:24.237152 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.238489 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.239570 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.240552 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.240968 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 241) took 1466 ms. Trace:
I20260709 12:12:24.241596 12643 rpcz_store.cc:276] 0709 12:12:22.774820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.775011 (+   191us) service_pool.cc:224] Handling call
0709 12:12:24.240565 (+1465554us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:24.240888 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1973044480}
I20260709 12:12:24.242059 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:24.236380 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:24.242744 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.243247 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:24.243468 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:24.244098 12643 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:24.244360 12495 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:24.244493 14811 raft_consensus.cc:2764] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:12:24.245334 12643 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:24.210966 12496 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:24.262699 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:24.263463 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:24.200127 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:24.223809 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:24.208171 12494 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:24.244282 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:24.266894 15162 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.267920 12493 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:12:24.226073 12637 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:24.245477 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:24.270648 12495 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:24.234527 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:24.247324 12645 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:24.248605 12640 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:24.245994 15099 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:24.248582 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 333748us
I20260709 12:12:24.257310 14990 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:24.255160 15094 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.256307 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 242) took 1473 ms. Trace:
I20260709 12:12:24.282117 12642 rpcz_store.cc:276] 0709 12:12:22.783056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.783203 (+   147us) service_pool.cc:224] Handling call
0709 12:12:24.256096 (+1472893us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:24.256233 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2414488320}
I20260709 12:12:24.284723 12642 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:24.220903 12395 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 3 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:24.261317 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 272) took 1405 ms. Trace:
I20260709 12:12:24.286475 12492 rpcz_store.cc:276] 0709 12:12:22.855655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.947456 (+ 91801us) service_pool.cc:224] Handling call
0709 12:12:24.261064 (+1313608us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:24.261231 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2555984768}
I20260709 12:12:24.262816 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:24.289042 12570 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.288527 12395 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:24.291214 12395 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:12:24.279281 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48978.2us
I20260709 12:12:24.294500 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:24.280272 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80299.7us
W20260709 12:12:24.253173 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:24.299032 12568 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.013s	sys 0.060s
W20260709 12:12:24.299392 12568 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.013s	sys 0.061s
W20260709 12:12:24.250108 14884 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.595s	user 0.011s	sys 0.001s
I20260709 12:12:24.301471 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1878) took 1684 ms. Trace:
I20260709 12:12:24.301528 12645 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
W20260709 12:12:24.303584 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69157us
I20260709 12:12:24.307674 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:24.309901 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.027s	sys 0.020s
I20260709 12:12:24.311970 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:24.319159 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.027s	sys 0.020s
I20260709 12:12:24.319394 12488 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:24.319535 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 546882us
I20260709 12:12:24.314548 13257 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:24.320317 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:24.321573 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:24.324584 12490 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:12:24.305325 12568 rpcz_store.cc:276] 0709 12:12:22.616944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.743131 (+126187us) service_pool.cc:224] Handling call
0709 12:12:22.776185 (+ 33054us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:22.811698 (+ 35513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:22.811732 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:22.812386 (+   654us) log.cc:844] Serialized 52 byte log entry
0709 12:12:23.527418 (+715032us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:24.299817 (+772399us) log.cc:844] Serialized 14 byte log entry
0709 12:12:24.301028 (+  1211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.301133 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:24.301201 (+    68us) raft_consensus.cc:1685] finished
0709 12:12:24.301220 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:24.301398 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34215,"raft.queue_time_us":7531,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":6639,"thread_start_us":1182306,"threads_started":2,"wal-append.queue_time_us":782860}
I20260709 12:12:24.332584 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.334590 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:24.335274 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:24.269574 15162 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:24.336494 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:24.333462 12642 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:24.342422 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.343392 12645 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:24.344153 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.293292 12395 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 3 to 5, now has a pending config: VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129), VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 5 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:24.348882 12641 raft_consensus.cc:2368] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:24.323565 13253 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:24.398495 12637 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:24.273804 15156 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:12:24.401433 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:24.389444 12488 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:24.300251 15147 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.105s
I20260709 12:12:24.303673 15099 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:24.423923 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.333761 15062 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:24.446126 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:24.421738 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86576.4us
I20260709 12:12:24.411084 12395 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 3, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 3 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:24.409963 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:24.324132 14811 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:24.513607 12642 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:24.519459 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.520393 12642 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.511996 12640 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.488178 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:24.523470 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:24.348541 15052 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:24.343137 15074 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:12:24.455695 15151 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:12:24.541234 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:24.473531 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:24.476464 15082 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.023s	sys 0.031s
I20260709 12:12:24.348340 14909 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:24.347950 14990 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:12:24.549908 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140163us
I20260709 12:12:24.586207 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:24.546789 12395 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:24.336493 15167 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:24.402287 14966 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:24.569280 15082 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.023s	sys 0.032s
I20260709 12:12:24.566339 12488 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:24.403884 14927 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.179s
W20260709 12:12:24.567504 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 166179us
I20260709 12:12:24.529117 12645 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:24.419041 14937 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:24.454879 15017 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:24.578342 12642 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:24.580838 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 273006us
I20260709 12:12:24.411509 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:24.324695 14884 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.303148 15095 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.562592 15155 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:24.514114 15070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.013s	sys 0.030s
I20260709 12:12:24.513756 15136 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260709 12:12:24.568370 14967 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:24.411965 15056 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.174s
I20260709 12:12:24.620746 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:24.708689 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.613438 12640 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:24.620500 12395 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 3 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:24.712617 12395 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 2, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 2 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:24.542704 15113 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.092s
I20260709 12:12:24.645509 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:24.673384 14876 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:24.716032 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93976.2us
I20260709 12:12:24.717916 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:24.580747 15173 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:24.719044 15173 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:24.719493 15173 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.716670 14876 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.717516 12395 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 5 to 6, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 6 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:24.721829 15074 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:24.722664 15074 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:24.722849 12395 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 1 to 3, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:24.724228 15074 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:24.725004 15074 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:12:24.725836 15073 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:24.726359 15073 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:24.447009 15169 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.203s
I20260709 12:12:24.726608 14876 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:24.653033 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:24.727952 12640 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:24.728385 12645 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:24.606638 14909 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:24.590101 15162 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:24.729789 15162 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:24.730324 14909 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.730783 15162 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.732002 15162 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:24.617213 15010 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:24.601816 14811 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:24.614435 14966 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:24.618757 14990 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:24.675602 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:24.620129 15167 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:24.747000 15167 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.749972 15165 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:12:24.750221 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74979.1us
I20260709 12:12:24.750587 15165 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:24.751242 14884 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:24.751871 15157 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:24.656626 15099 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.751525 15165 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.752390 15157 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:24.752977 15165 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.753367 15157 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.754210 14990 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.675848 15088 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:24.686811 12644 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:24.725207 12395 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 4 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:24.751617 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:24.760329 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.964s (SENT) [suppressed 20 similar messages]
I20260709 12:12:24.766698 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 274) took 1024 ms. Trace:
W20260709 12:12:24.679670 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 237693us
W20260709 12:12:24.692155 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.693280 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:24.769794 12642 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.621593 15018 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260709 12:12:24.736920 12491 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:24.739995 13258 raft_consensus.cc:606] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:24.771677 13258 raft_consensus.cc:614] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:24.737931 14967 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:24.744241 15183 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:12:24.744565 15181 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260709 12:12:24.704821 15070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.014s	sys 0.030s
I20260709 12:12:24.726756 15073 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.678248 15025 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.734067 15155 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:24.755396 15136 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:12:24.756822 14966 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:24.782655 14967 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:24.794449 15153 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:24.799475 14876 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 12:12:24.792703 15094 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:24.803264 12569 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:24.768795 15158 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:24.807247 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1687) took 1053 ms. Trace:
W20260709 12:12:24.806432 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 1973) took 2123 ms (client timeout 2823 ms). Trace:
W20260709 12:12:24.851711 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.793469 15074 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:24.827361 14821 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.637s	user 0.005s	sys 0.008s
I20260709 12:12:24.851756 12641 rpcz_store.cc:276] 0709 12:12:23.754227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.754354 (+   127us) service_pool.cc:224] Handling call
0709 12:12:24.807047 (+1052693us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:24.807180 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262000512}
W20260709 12:12:24.851902 12490 rpcz_store.cc:269] 0709 12:12:22.683107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:22.683264 (+   157us) service_pool.cc:224] Handling call
0709 12:12:24.806134 (+2122870us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:24.806348 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3675552000}
I20260709 12:12:24.783581 15088 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.780089 12489 rpcz_store.cc:276] 0709 12:12:23.742473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.744352 (+  1879us) service_pool.cc:224] Handling call
0709 12:12:23.744860 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:23.745159 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:12:24.281097 (+535938us) log.cc:844] Serialized 14 byte log entry
0709 12:12:24.766189 (+485092us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:24.766261 (+    72us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:24.766326 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.766438 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:24.766612 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":666826,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":2618,"spinlock_wait_cycles":13440,"thread_start_us":979023,"threads_started":2}
I20260709 12:12:24.856741 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:24.793262 14884 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20260709 12:12:24.891980 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 318569us
I20260709 12:12:24.838146 15153 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:24.895100 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:24.822156 15156 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
W20260709 12:12:24.878211 15160 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:12:24.919416 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 201559us
W20260709 12:12:24.918586 15180 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.295s	user 0.001s	sys 0.000s
I20260709 12:12:24.917140 14876 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:24.935350 14876 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:24.936498 14876 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.938060 15095 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:24.940408 15173 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:24.944097 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.943797 15073 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:24.946547 15158 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:24.946919 15073 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:24.947089 15158 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.948539 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:24.950075 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:24.950960 13261 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:24.953246 14876 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.953940 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:24.955611 15179 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:12:24.955967 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:24.957423 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:24.958254 13257 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:24.959460 15088 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:24.960366 15177 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.308s	user 0.000s	sys 0.002s
I20260709 12:12:24.964021 15079 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:24.968253 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:24.891069 14821 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.972852 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1735) took 1597 ms. Trace:
I20260709 12:12:24.920925 15136 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:12:24.974064 15138 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:24.974705 15138 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:24.975824 15138 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.992537 15138 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:24.982309 15086 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.747s	user 0.012s	sys 0.002s
I20260709 12:12:24.987622 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:25.052896 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:25.052578 12570 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:24.990576 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1690) took 1235 ms. Trace:
I20260709 12:12:24.890050 15074 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:24.955629 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:24.999090 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.067229 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:24.978387 12494 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:24.911206 15190 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:12:25.071180 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44100.8us
I20260709 12:12:24.973479 12569 rpcz_store.cc:276] 0709 12:12:23.374952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.377592 (+  2640us) service_pool.cc:224] Handling call
0709 12:12:23.377945 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:24.847494 (+1469549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.972452 (+124958us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:24.972753 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3277287168}
I20260709 12:12:25.076875 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:24.988894 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:25.054109 12646 rpcz_store.cc:276] 0709 12:12:23.755221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.755335 (+   114us) service_pool.cc:224] Handling call
0709 12:12:23.755686 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:23.806187 (+ 50501us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:12:23.806236 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:23.827891 (+ 21655us) log.cc:844] Serialized 26 byte log entry
0709 12:12:24.310355 (+482464us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:24.310400 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.310470 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:24.989958 (+679488us) raft_consensus.cc:1685] finished
0709 12:12:24.990018 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:24.990238 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49945,"raft.queue_time_us":57062,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":22217,"thread_start_us":482021,"threads_started":1,"wal-append.queue_time_us":731420}
I20260709 12:12:25.077883 12564 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:25.055389 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55378.1us
I20260709 12:12:25.081172 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:25.057241 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:25.058073 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 297726us
I20260709 12:12:25.059262 12567 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:25.068719 12494 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:25.070403 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:25.083495 12571 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:25.083738 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.020753 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:25.084954 12566 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:25.056195 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:25.087819 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:25.096019 12570 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:25.086473 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:25.099675 12563 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:25.100803 13259 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:25.100858 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:25.102959 13257 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:12:25.104326 12536 leader_election.cc:400] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:25.052578 15056 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.059262 15073 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:25.064265 15070 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:25.070007 15190 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:25.104410 15181 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
W20260709 12:12:25.133421 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:25.141077 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76850.1us
W20260709 12:12:25.143384 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50066.2us
I20260709 12:12:25.059228 14811 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
W20260709 12:12:25.160506 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56320us
I20260709 12:12:25.161365 12564 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:25.123064 14567 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:25.156529 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:25.066397 15086 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.069463 14876 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
W20260709 12:12:25.149413 15163 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:12:24.933826 15155 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:25.083009 15136 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:12:25.197438 12569 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:25.152096 15167 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:25.205509 15173 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:25.206853 12638 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:25.151641 15190 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:25.107806 15111 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
W20260709 12:12:25.194798 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114030us
I20260709 12:12:25.194463 15212 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:25.200556 15176 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20260709 12:12:25.162832 15102 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:25.201256 15086 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:25.194334 14567 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:25.185566 15025 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:25.202468 14876 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:12:25.203210 15059 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.017s	user 0.009s	sys 0.004s
W20260709 12:12:25.203754 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47314.1us
W20260709 12:12:25.176023 15191 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:12:25.203966 14990 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:25.196394 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:25.192523 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:25.240664 12564 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:25.123906 15091 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260709 12:12:25.244798 15159 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:25.245428 15159 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:25.247175 15107 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:25.246882 15159 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.247826 15107 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:25.248564 15159 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:25.250262 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:25.231909 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:25.249166 15107 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:25.256999 15195 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:12:25.258388 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:25.211817 15165 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:25.127563 15056 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.188432 14811 consensus_queue.cc:1243] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:12:25.270310 15069 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.271061 15069 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.272027 15094 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.274869 15006 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.151364 15073 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.276698 14937 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:25.272842 15094 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.275480 15006 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.278453 15073 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:25.279836 15073 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.280378 15073 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:25.280735 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 332298us
I20260709 12:12:25.150806 15070 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:25.251562 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:25.281908 15063 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:25.283365 15070 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:25.283972 15070 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:25.284917 15070 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:25.267436 12494 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.288789 12494 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:25.291885 14785 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.292460 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:25.293359 12646 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:25.291885 15196 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:12:25.295957 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1623) took 1117 ms. Trace:
I20260709 12:12:25.296416 12638 rpcz_store.cc:276] 0709 12:12:24.178988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.185137 (+  6149us) service_pool.cc:224] Handling call
0709 12:12:25.295748 (+1110611us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:25.295896 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2346228480}
I20260709 12:12:25.253367 15147 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:25.297540 15056 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:25.297940 15147 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:25.275391 12611 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6.
I20260709 12:12:25.298707 12611 leader_election.cc:403] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:12:25.298931 15147 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.300800 15147 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:25.301826 14724 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6.
I20260709 12:12:25.302300 14930 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:25.303553 14724 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
W20260709 12:12:25.304123 14724 raft_consensus.cc:1068] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:12:25.278482 15082 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:25.305310 15082 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:25.306267 15082 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.307344 15082 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:25.296089 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2009) took 1045 ms. Trace:
I20260709 12:12:25.310207 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.311424 15069 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:25.310374 12495 rpcz_store.cc:276] 0709 12:12:24.250975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.271424 (+ 20449us) service_pool.cc:224] Handling call
0709 12:12:24.271796 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:24.303510 (+ 31714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:24.303555 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:24.304122 (+   567us) log.cc:844] Serialized 26 byte log entry
0709 12:12:24.903215 (+599093us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:24.903300 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.903414 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:25.291119 (+387705us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:25.295787 (+  4668us) raft_consensus.cc:1685] finished
0709 12:12:25.295807 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:25.296028 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":155468,"raft.queue_time_us":36996,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":19408,"spinlock_wait_cycles":6494976,"thread_start_us":481379,"threads_started":1,"wal-append.queue_time_us":680225}
I20260709 12:12:25.292577 14785 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.312420 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.312995 12495 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:25.314214 14785 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:25.315023 15113 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:25.315279 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83478.2us
I20260709 12:12:25.315414 15113 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:25.316089 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:25.316227 15113 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:25.316910 15056 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:25.317309 15113 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:25.319007 15165 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.322175 15165 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:25.323462 14567 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.326424 14821 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:25.327198 14884 consensus_queue.cc:1175] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 4 status { last_received { term: 3 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:12:25.327850 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 250) took 1025 ms. Trace:
I20260709 12:12:25.328450 12645 rpcz_store.cc:276] 0709 12:12:24.302657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.729259 (+426602us) service_pool.cc:224] Handling call
0709 12:12:24.729820 (+   561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:24.909719 (+179899us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:24.909770 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:24.909859 (+    89us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:24.909910 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:25.327499 (+417589us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54002e5254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:25.327556 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:25.327760 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413187200}
I20260709 12:12:25.330457 15074 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:25.331700 15136 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:25.332124 15136 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:25.332945 15136 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.098690 15153 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:25.213599 15184 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.324s	user 0.000s	sys 0.002s
I20260709 12:12:25.334113 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2004) took 1114 ms. Trace:
W20260709 12:12:25.335919 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92197.1us
I20260709 12:12:25.365578 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1685) took 1612 ms. Trace:
I20260709 12:12:25.337710 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:25.338265 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:25.368538 12646 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:25.338867 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.373154 12641 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9.
I20260709 12:12:25.339491 12643 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:12:25.340281 12640 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:25.342705 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.211421 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:25.344396 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.346340 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:25.376695 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.376989 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.377596 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:25.351569 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:25.234781 15186 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.328s	user 0.000s	sys 0.001s
I20260709 12:12:25.211496 15173 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:25.196039 15061 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:25.281500 15006 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:25.230772 15198 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.350785 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:25.237810 14876 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:25.367839 12639 rpcz_store.cc:276] 0709 12:12:23.753308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.753830 (+   522us) service_pool.cc:224] Handling call
0709 12:12:23.786262 (+ 32432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:23.823460 (+ 37198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:23.823508 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:23.824176 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:12:24.253022 (+428846us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:24.298994 (+ 45972us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:24.579946 (+280952us) log.cc:844] Serialized 14 byte log entry
0709 12:12:24.581496 (+  1550us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.587379 (+  5883us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:25.334497 (+747118us) raft_consensus.cc:1685] finished
0709 12:12:25.365321 (+ 30824us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:25.365511 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38799,"prepare.queue_time_us":504857,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":39351,"raft.run_cpu_time_us":3542,"raft.run_wall_time_us":443488,"spinlock_wait_cycles":84220672,"thread_start_us":728799,"threads_started":2,"wal-append.queue_time_us":977212}
I20260709 12:12:25.379418 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:25.385159 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.385582 12489 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:25.375406 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:25.387123 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:25.390690 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:25.392650 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:25.395767 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.234926 15212 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:25.405316 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209244us
I20260709 12:12:25.477336 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2006) took 1257 ms. Trace:
I20260709 12:12:25.441458 12564 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:25.401760 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:25.365172 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2008) took 1141 ms. Trace:
I20260709 12:12:25.403539 12646 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:25.485903 12568 raft_consensus.cc:2368] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7.
I20260709 12:12:25.400899 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:25.486819 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:25.487169 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.487696 12646 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:25.489400 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:25.485719 12493 rpcz_store.cc:276] 0709 12:12:24.224058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.268741 (+ 44683us) service_pool.cc:224] Handling call
0709 12:12:24.939264 (+670523us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:24.939570 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:24.940002 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:24.940053 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:24.940613 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:12:25.107760 (+167147us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:25.107806 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:25.107896 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:25.334160 (+226264us) raft_consensus.cc:1685] finished
0709 12:12:25.334186 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:25.365081 (+ 30895us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12406,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":8117504,"thread_start_us":166716,"threads_started":1,"wal-append.queue_time_us":384207}
W20260709 12:12:25.475826 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:25.425779 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:25.492542 12641 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:25.493470 12640 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:25.477684 15111 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:25.356521 15074 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:25.498996 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:25.499331 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:25.379889 15085 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:25.380192 15070 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:25.241954 15102 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.335340 15059 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:25.503460 12641 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:25.581096 12639 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [4.2-6.3]
W20260709 12:12:25.798380 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 430271us
W20260709 12:12:25.393566 14967 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.550s	user 0.003s	sys 0.009s
W20260709 12:12:25.886889 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:26.058104 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 846698us
I20260709 12:12:25.367619 12496 rpcz_store.cc:276] 0709 12:12:24.219247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.265402 (+ 46155us) service_pool.cc:224] Handling call
0709 12:12:24.271314 (+  5912us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:24.309366 (+ 38052us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:24.309411 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:24.399249 (+ 89838us) log.cc:844] Serialized 52 byte log entry
0709 12:12:24.735959 (+336710us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:24.797142 (+ 61183us) log.cc:844] Serialized 14 byte log entry
0709 12:12:24.797349 (+   207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:24.797422 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:25.333848 (+536426us) raft_consensus.cc:1685] finished
0709 12:12:25.333870 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:25.334043 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84074,"prepare.queue_time_us":61417,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":86,"raft.queue_time_us":67297,"raft.run_cpu_time_us":869,"raft.run_wall_time_us":79897,"thread_start_us":220306,"threads_started":1,"wal-append.queue_time_us":464811}
W20260709 12:12:25.721834 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:25.737998 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:25.680579 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:25.554340 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:25.485298 12494 rpcz_store.cc:276] 0709 12:12:24.219905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.266931 (+ 47026us) service_pool.cc:224] Handling call
0709 12:12:24.977906 (+710975us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:24.978213 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:25.267023 (+288810us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400299b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577106f 00007ffbf57e9f60 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62
0709 12:12:25.476773 (+209750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:25.476808 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:25.476948 (+   140us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:25.476985 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:25.477068 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:25.477255 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91178,"raft.queue_time_us":92508,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":434535168}
I20260709 12:12:25.504179 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:26.081974 12570 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:26.082065 12646 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:26.083364 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:26.082864 15074 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.083920 15198 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.084721 12643 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:25.401561 15094 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:25.475679 14930 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:25.391968 15056 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:26.082901 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:26.155751 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:26.159761 12495 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:12:26.158857 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.156559 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:26.161147 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:25.860445 15159 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.611s	user 0.014s	sys 0.000s
W20260709 12:12:25.856388 15062 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.317s	user 0.009s	sys 0.006s
W20260709 12:12:25.887876 15203 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:12:26.088227 15085 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.091542 15208 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:12:26.170143 15208 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.171444 15208 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.846912 15073 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:26.112383 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1617) took 1967 ms (client timeout 2009 ms). Trace:
W20260709 12:12:26.223371 12644 rpcz_store.cc:269] 0709 12:12:24.145155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.145743 (+   588us) service_pool.cc:224] Handling call
0709 12:12:26.111955 (+1966212us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:26.112281 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1242228608}
I20260709 12:12:26.116675 14811 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:26.225339 14811 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:26.226744 14811 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.229224 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2012) took 1977 ms. Trace:
I20260709 12:12:26.230129 12488 rpcz_store.cc:276] 0709 12:12:24.251775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.736561 (+484786us) service_pool.cc:224] Handling call
0709 12:12:24.737015 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:24.765363 (+ 28348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:24.765405 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:24.784139 (+ 18734us) log.cc:844] Serialized 26 byte log entry
0709 12:12:25.087244 (+303105us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:25.087286 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:25.087362 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:25.252640 (+165278us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b4800080058. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:26.228711 (+976071us) raft_consensus.cc:1685] finished
0709 12:12:26.228749 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.229120 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":129819,"raft.queue_time_us":28587,"raft.run_cpu_time_us":7710,"raft.run_wall_time_us":483942,"spinlock_wait_cycles":11625216,"thread_start_us":265548,"threads_started":1,"wal-append.queue_time_us":592167}
I20260709 12:12:26.232645 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:25.652560 14966 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.814s	user 0.011s	sys 0.001s
I20260709 12:12:25.617498 15212 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:25.707597 14876 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:25.402099 15173 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.133951 15107 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:26.128366 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:26.130462 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.029s	sys 0.045s
W20260709 12:12:26.325706 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.030s	sys 0.045s
I20260709 12:12:26.135540 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1749) took 1076 ms. Trace:
W20260709 12:12:26.142452 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59160.7us
I20260709 12:12:26.143846 12490 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:26.146675 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:26.327597 12570 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:26.327732 12536 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9.
W20260709 12:12:26.147647 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.148098 12564 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:12:26.084998 15138 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.023s	user 0.005s	sys 0.005s
I20260709 12:12:26.085093 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:25.380358 15169 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.380159 14785 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.086714 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:25.476059 15156 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:25.618319 15158 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.671s	user 0.011s	sys 0.000s
I20260709 12:12:25.481956 14990 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:25.390489 15017 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.156666 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:26.273221 15082 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.293797 15111 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:25.574159 15095 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.271502 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:26.261797 12569 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:12:26.233284 12488 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:26.320492 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.301781 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 288) took 1527 ms. Trace:
I20260709 12:12:26.154953 15222 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:25.536206 15157 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.133003 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37834 (request call id 1) took 2005 ms. Trace:
I20260709 12:12:26.328269 12536 leader_election.cc:403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:26.124758 14927 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:26.281682 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:26.423930 12643 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 7
W20260709 12:12:26.294379 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 790311us
I20260709 12:12:26.303962 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1682) took 1250 ms. Trace:
I20260709 12:12:26.223403 15073 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:25.677522 15224 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:26.255777 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2005) took 2036 ms. Trace:
I20260709 12:12:26.341702 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:26.233470 12566 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:12:26.328600 12572 rpcz_store.cc:276] 0709 12:12:25.060561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.060889 (+   328us) service_pool.cc:224] Handling call
0709 12:12:25.061397 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:25.098625 (+ 37228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:25.098660 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:25.099320 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 12:12:25.235196 (+135876us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:25.235238 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:25.235322 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:26.135318 (+899996us) raft_consensus.cc:1685] finished
0709 12:12:26.135336 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.135479 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43655,"raft.queue_time_us":41645,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":17781,"thread_start_us":61124,"threads_started":1,"wal-append.queue_time_us":450380}
I20260709 12:12:26.442328 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:26.443676 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:26.443926 12490 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:26.444757 12489 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:12:26.444907 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:26.423440 14927 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:26.289155 15190 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.054s	user 0.007s	sys 0.003s
W20260709 12:12:26.295032 15154 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 2.018s	user 0.016s	sys 0.000s
I20260709 12:12:26.286743 13256 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
W20260709 12:12:26.291872 15113 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.937s	user 0.011s	sys 0.000s
W20260709 12:12:26.289852 15197 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.110s	user 0.007s	sys 0.008s
W20260709 12:12:26.273190 15225 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:12:26.358610 12639 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:26.358141 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
W20260709 12:12:26.161474 15147 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.860s	user 0.012s	sys 0.004s
I20260709 12:12:26.362401 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:26.364328 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:26.339512 15079 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.375s	user 0.014s	sys 0.000s
I20260709 12:12:26.317569 15018 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.159137 15056 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:26.256038 15094 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.278203 15167 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.378226 12641 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:12:26.296386 15102 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.291029 15198 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.318446 14811 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:26.391502 12496 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:26.235666 14937 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.645821 15088 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:25.406737 15061 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:26.656159 15236 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.528s	user 0.000s	sys 0.009s
W20260709 12:12:26.657684 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 338) took 2348 ms (client timeout 2222 ms). Trace:
I20260709 12:12:26.214485 15062 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:26.395244 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:26.318120 15209 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:26.658420 15165 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:26.658310 13256 rpcz_store.cc:269] 0709 12:12:24.308928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.323164 (+ 14236us) service_pool.cc:224] Handling call
0709 12:12:26.657358 (+2334194us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:26.657545 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3607458688}
I20260709 12:12:26.262193 14930 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:26.659155 15165 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:26.642159 15220 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260709 12:12:26.660538 14930 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:26.654035 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1683) took 1592 ms (client timeout 2043 ms). Trace:
I20260709 12:12:26.661934 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:26.658640 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1898) took 1306 ms. Trace:
I20260709 12:12:26.662603 12563 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:26.662722 12564 rpcz_store.cc:276] 0709 12:12:25.351942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.495551 (+143609us) service_pool.cc:224] Handling call
0709 12:12:26.658567 (+1163016us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.663632 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:26.664113 12565 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:26.664529 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1637) took 1174 ms. Trace:
I20260709 12:12:26.665669 15113 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:12:26.662233 12492 rpcz_store.cc:269] 0709 12:12:25.061564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.083403 (+ 21839us) service_pool.cc:224] Handling call
0709 12:12:26.653835 (+1570432us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:26.653967 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2928802560}
I20260709 12:12:26.666289 15113 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:25.837383 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1676) took 2058 ms. Trace:
I20260709 12:12:26.357389 15085 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.413389 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1681) took 1463 ms. Trace:
W20260709 12:12:26.647940 15230 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.517s	user 0.005s	sys 0.003s
I20260709 12:12:26.414485 12569 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:12:26.417255 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:26.690328 12569 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:12:26.690848 12569 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:26.691182 12568 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:12:26.421269 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93658us
I20260709 12:12:26.563220 15056 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:26.386406 14876 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:26.425902 12495 rpcz_store.cc:276] 0709 12:12:25.053242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.257075 (+1203833us) service_pool.cc:224] Handling call
0709 12:12:26.257531 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.303403 (+ 45872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:26.303460 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:26.303532 (+    72us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:26.303584 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.303682 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.303875 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45272,"raft.queue_time_us":57050,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":20791}
W20260709 12:12:26.428606 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.419683 13258 rpcz_store.cc:276] 0709 12:12:24.127692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.129046 (+  1354us) service_pool.cc:224] Handling call
0709 12:12:26.132670 (+2003624us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:26.132858 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1402814720}
I20260709 12:12:26.445070 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:26.054726 15136 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:26.345656 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1678) took 1418 ms. Trace:
I20260709 12:12:26.733146 12493 rpcz_store.cc:276] 0709 12:12:24.927038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.086448 (+1159410us) service_pool.cc:224] Handling call
0709 12:12:26.345481 (+259033us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.734067 12643 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:26.734447 12493 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:26.735000 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2017) took 1353 ms. Trace:
I20260709 12:12:26.735328 12493 rpcz_store.cc:276] 0709 12:12:25.381124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.733960 (+1352836us) service_pool.cc:224] Handling call
0709 12:12:26.734245 (+   285us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:26.734784 (+   539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.734950 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.737006 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2018) took 1355 ms. Trace:
I20260709 12:12:26.446633 12572 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:26.338008 14966 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.358338 14785 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.367326 15169 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.390185 15158 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.378254 15156 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.438869 12497 rpcz_store.cc:276] 0709 12:12:24.219576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:24.266717 (+ 47141us) service_pool.cc:224] Handling call
0709 12:12:24.952955 (+686238us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:24.953171 (+   216us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:24.953410 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:25.207487 (+254077us) log.cc:844] Serialized 14 byte log entry
0709 12:12:25.598399 (+390912us) log.cc:844] Serialized 14 byte log entry
0709 12:12:26.235108 (+636709us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b1800321dc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:12:26.255327 (+ 20219us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:26.255392 (+    65us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:26.255444 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.255547 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.255706 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":733770,"prepare.queue_time_us":156876,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":113,"raft.queue_time_us":661750,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":64917,"spinlock_wait_cycles":18304512,"thread_start_us":461157,"threads_started":2,"wal-append.queue_time_us":1205171}
I20260709 12:12:26.393975 15152 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.396394 15017 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.464025 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2016) took 1083 ms. Trace:
I20260709 12:12:26.579870 15094 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.753403 12490 rpcz_store.cc:276] 0709 12:12:25.380802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.680292 (+299490us) service_pool.cc:224] Handling call
0709 12:12:26.463911 (+783619us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:26.419341 15155 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.021s	sys 0.021s
I20260709 12:12:26.422160 15138 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:26.423064 14909 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:26.519114 12642 raft_consensus.cc:2376] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6. Re-sending same reply.
W20260709 12:12:26.519327 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155498us
I20260709 12:12:26.582396 15167 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:26.527444 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.958s	user 0.004s	sys 0.022s
I20260709 12:12:26.454795 13253 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:26.537395 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:26.534423 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 378712us
I20260709 12:12:26.563093 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:26.564338 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.565356 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:26.494628 15010 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:26.409093 15082 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.412519 15095 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.441758 15073 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:26.526183 15086 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:26.562888 15052 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.992s	user 0.016s	sys 0.001s
I20260709 12:12:26.652410 15102 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:26.641173 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1636) took 1250 ms. Trace:
I20260709 12:12:26.807997 12646 rpcz_store.cc:276] 0709 12:12:25.391587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.490626 (+ 99039us) service_pool.cc:224] Handling call
0709 12:12:26.054295 (+563669us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.141396 (+ 87101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.640876 (+499480us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b54000a6ed0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:26.641098 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4810624}
I20260709 12:12:26.516791 15111 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.641422 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:26.655067 14937 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.763367 14811 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.657759 15061 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.658955 15209 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:26.139042 15070 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.469035 15193 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:26.802982 15114 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.667531 12491 rpcz_store.cc:276] 0709 12:12:23.778667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:23.820374 (+ 41707us) service_pool.cc:224] Handling call
0709 12:12:23.820811 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:24.771320 (+950509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:25.836947 (+1065627us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:25.837279 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2106227968}
I20260709 12:12:26.152956 14967 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.396183 15208 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:26.668429 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.665452 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1743) took 1648 ms. Trace:
I20260709 12:12:26.668900 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2019) took 1287 ms. Trace:
I20260709 12:12:26.864153 12496 rpcz_store.cc:276] 0709 12:12:25.381666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.156463 (+774797us) service_pool.cc:224] Handling call
0709 12:12:26.668836 (+512373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.529528 15099 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:26.594511 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:26.640656 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:26.760855 14909 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.535995 14986 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.676920 12488 rpcz_store.cc:276] 0709 12:12:24.949909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.232486 (+1282577us) service_pool.cc:224] Handling call
0709 12:12:26.233005 (+   519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.413085 (+180080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.413330 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.413875 12494 rpcz_store.cc:276] 0709 12:12:24.774203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.259773 (+1485570us) service_pool.cc:224] Handling call
0709 12:12:26.300946 (+ 41173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.301194 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:26.301234 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:26.301291 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:26.301331 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.301401 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.301561 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.435436 15157 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:26.276404 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:26.564786 15079 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.861462 15241 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.879043 15099 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:26.563982 15018 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.879292 15241 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.880208 15099 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.881254 15018 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:26.703442 15232 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20260709 12:12:26.692585 15214 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:26.883093 15214 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.692823 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:26.885481 15214 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.859401 15209 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:26.886183 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 193477us
I20260709 12:12:26.887225 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:26.888725 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:26.698633 15165 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:26.889642 15223 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:26.890093 15165 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:26.890317 15223 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:26.690205 15088 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.891444 15223 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.535640 15147 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:26.894018 15147 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.896721 15147 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.519462 15190 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.898460 15082 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:26.737425 12493 rpcz_store.cc:276] 0709 12:12:25.381375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.736041 (+1354666us) service_pool.cc:224] Handling call
0709 12:12:26.736385 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.736609 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:26.736648 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:26.736705 (+    57us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:26.736748 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.736817 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.736943 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.899798 15190 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.531766 15154 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.653810 15059 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:26.745932 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1896) took 1394 ms. Trace:
I20260709 12:12:26.901018 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1693) took 1563 ms. Trace:
I20260709 12:12:26.438874 15224 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.693233 15074 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.748531 12572 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:26.902590 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1902) took 1549 ms. Trace:
I20260709 12:12:26.901301 12567 rpcz_store.cc:276] 0709 12:12:25.351413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.581120 (+229707us) service_pool.cc:224] Handling call
0709 12:12:25.610771 (+ 29651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.260435 (+649664us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:12:26.318194 (+ 57759us) log.cc:844] Serialized 14 byte log entry
0709 12:12:26.520198 (+202004us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:26.520294 (+    96us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:26.520365 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.745555 (+225190us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:26.745619 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.745860 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":57243,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"raft.queue_time_us":120,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4855,"spinlock_wait_cycles":1493291904,"thread_start_us":201516,"threads_started":1,"wal-append.queue_time_us":445008}
I20260709 12:12:26.902969 12572 rpcz_store.cc:276] 0709 12:12:25.353213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.747696 (+1394483us) service_pool.cc:224] Handling call
0709 12:12:26.748220 (+   524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:26.902245 (+154025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.902524 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:26.903561 15224 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.903880 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:26.904183 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:26.904862 12572 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:26.755693 12490 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:26.754940 15152 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.732599 15085 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:26.906193 15085 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.906585 15070 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:26.765794 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1892) took 1422 ms. Trace:
I20260709 12:12:26.793361 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:26.423707 15159 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:26.640950 14927 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.907649 15159 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.908066 14927 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:26.908452 15085 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:26.759975 15155 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.098s	user 0.021s	sys 0.035s
I20260709 12:12:26.762876 15244 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.909927 15244 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.911290 15085 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:26.911504 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:26.798864 13262 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:26.756508 15242 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:26.800745 15010 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.864658 12645 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7. Re-sending same reply.
I20260709 12:12:26.669693 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1684) took 1602 ms. Trace:
I20260709 12:12:26.864054 12565 rpcz_store.cc:276] 0709 12:12:25.016847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.018450 (+  1603us) service_pool.cc:224] Handling call
0709 12:12:25.018922 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.664569 (+1645647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.665193 (+   624us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:26.665388 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3783011200}
I20260709 12:12:26.810746 15052 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:26.760401 15138 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.811096 15086 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.875005 13260 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 4: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:12:26.877481 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:27.014665 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 126035us
I20260709 12:12:26.907275 12569 rpcz_store.cc:276] 0709 12:12:25.343363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.349052 (+  5689us) service_pool.cc:224] Handling call
0709 12:12:25.349584 (+   532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.691601 (+1342017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.765498 (+ 73897us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54003ac354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:26.765723 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2096667904}
I20260709 12:12:26.878782 15079 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:26.702376 15234 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 12:12:26.574607 14990 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.727918 12566 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:26.898602 12492 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20260709 12:12:26.904485 14986 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:26.743850 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:26.852021 13254 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:26.909370 15094 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:26.764995 15173 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:26.790978 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.222s	user 0.005s	sys 0.030s
I20260709 12:12:26.765336 15113 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:26.664911 12640 rpcz_store.cc:276] 0709 12:12:25.490645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.494181 (+  3536us) service_pool.cc:224] Handling call
0709 12:12:25.494560 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:25.560454 (+ 65894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:25.560501 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:25.561088 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:12:26.151086 (+589998us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:26.151146 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.151247 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:26.664290 (+513043us) raft_consensus.cc:1685] finished
0709 12:12:26.664309 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.664447 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41891,"raft.queue_time_us":104612,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":3079,"thread_start_us":491291,"threads_started":1,"wal-append.queue_time_us":1023245}
I20260709 12:12:26.813951 15102 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.912432 12567 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:12:26.912043 15244 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.010686 15157 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:27.078912 15241 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.082909 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:27.083611 12494 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:27.083719 14937 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.084637 15156 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:26.907315 15152 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:27.085670 14785 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:27.086911 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2020) took 1705 ms. Trace:
I20260709 12:12:27.087029 14967 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:27.087304 12497 rpcz_store.cc:276] 0709 12:12:25.381922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.751407 (+1369485us) service_pool.cc:224] Handling call
0709 12:12:26.756001 (+  4594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.895358 (+139357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:26.895388 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:26.895428 (+    40us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:26.895458 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.086685 (+191227us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:27.086733 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.086849 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319905024}
I20260709 12:12:27.088230 15169 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:26.912443 15073 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:26.862751 14811 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:27.011847 15017 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:26.912359 13258 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:27.016718 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:27.095378 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78755.8us
I20260709 12:12:27.095763 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1685) took 2009 ms. Trace:
I20260709 12:12:27.096462 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:26.952322 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.017720 12489 rpcz_store.cc:276] 0709 12:12:25.067163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.295477 (+1228314us) service_pool.cc:224] Handling call
0709 12:12:26.444508 (+149031us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:26.445174 (+   666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.669494 (+224320us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540012cad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:26.669620 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341713792}
I20260709 12:12:26.901582 12637 rpcz_store.cc:276] 0709 12:12:25.337388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.337531 (+   143us) service_pool.cc:224] Handling call
0709 12:12:25.338169 (+   638us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:12:25.366133 (+ 27964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:25.366167 (+    34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:12:25.391565 (+ 25398us) log.cc:844] Serialized 558 byte log entry
0709 12:12:26.431931 (+1040366us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:26.431978 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.432163 (+   185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:26.900663 (+468500us) raft_consensus.cc:1685] finished
0709 12:12:26.900696 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:26.900925 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16509,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":25796,"thread_start_us":173606,"threads_started":1,"wal-append.queue_time_us":627881}
W20260709 12:12:26.907624 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114392us
I20260709 12:12:26.948840 12638 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:27.099643 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:26.913168 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2021) took 1531 ms. Trace:
I20260709 12:12:27.100680 12490 rpcz_store.cc:276] 0709 12:12:25.382169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.754417 (+1372248us) service_pool.cc:224] Handling call
0709 12:12:26.755199 (+   782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.905389 (+150190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:26.913095 (+  7706us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:27.027096 15190 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.017215 12569 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:26.911895 15085 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:27.026788 12566 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:27.063575 12462 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:12:27.064292 12572 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:27.017215 12565 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:12:27.065045 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.016s	sys 0.039s
I20260709 12:12:26.893481 15088 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:26.975275 15006 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:27.070832 15052 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:26.964522 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.103282 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:27.123739 13259 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:27.014058 15159 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.105645 12490 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:27.109367 12462 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20260709 12:12:27.128233 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92952.7us
W20260709 12:12:27.109843 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.017s	sys 0.039s
I20260709 12:12:27.110188 12569 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:12:27.132875 12644 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:27.112455 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:27.030565 15222 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9.
W20260709 12:12:27.113314 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.019s	sys 0.036s
I20260709 12:12:27.104700 15241 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:27.117105 12489 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:27.113065 15073 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:27.113270 15017 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:26.874169 15114 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:27.135116 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.137192 12644 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:12:27.139839 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.019s	sys 0.036s
I20260709 12:12:27.156056 13258 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:27.140357 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:27.157620 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:27.145800 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:27.114773 15062 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:12:27.261739 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.012s	sys 0.018s
I20260709 12:12:27.113411 15085 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:27.148186 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1900) took 1795 ms. Trace:
I20260709 12:12:27.114351 14966 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.096273 12492 rpcz_store.cc:276] 0709 12:12:25.086618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.666712 (+1580094us) service_pool.cc:224] Handling call
0709 12:12:26.667250 (+   538us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:12:27.029303 (+362053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.095384 (+ 66081us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:27.095697 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":530953472}
I20260709 12:12:27.024228 15147 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:27.102337 15056 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.157603 15224 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:27.117086 15190 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.128468 15158 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:27.228581 12571 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.901s	user 0.012s	sys 0.022s
I20260709 12:12:27.127121 15006 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:27.263394 15154 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.277659 15095 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:27.195940 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:27.134927 15052 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:27.141961 14716 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:12:27.236792 12644 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
W20260709 12:12:27.234863 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.906s	user 0.007s	sys 0.027s
W20260709 12:12:27.262046 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.950s	user 0.004s	sys 0.030s
I20260709 12:12:27.246723 13257 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:27.255108 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:27.227141 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.257982 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:27.262091 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.016s	sys 0.015s
I20260709 12:12:27.238327 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1691) took 1937 ms. Trace:
I20260709 12:12:27.113991 15173 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.024552 15242 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:27.277806 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.015s	sys 0.018s
I20260709 12:12:27.277658 12564 rpcz_store.cc:276] 0709 12:12:25.352595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.663708 (+1311113us) service_pool.cc:224] Handling call
0709 12:12:26.664065 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.664288 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:26.664620 (+   332us) log.cc:844] Serialized 14 byte log entry
0709 12:12:27.147644 (+483024us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:27.147725 (+    81us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:27.147800 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.147899 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.148096 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":596,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":4479,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":21627,"thread_start_us":482465,"threads_started":1}
I20260709 12:12:27.281705 12494 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:27.283262 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:27.293776 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.245594 14785 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:27.228096 12639 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 12:12:27.320573 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93870.7us
W20260709 12:12:27.322137 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.995s	user 0.012s	sys 0.022s
W20260709 12:12:27.305681 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 372615us
I20260709 12:12:27.585999 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.588248 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1901) took 2235 ms. Trace:
I20260709 12:12:27.588667 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.308643 15062 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:27.588891 12568 rpcz_store.cc:276] 0709 12:12:25.352906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.746110 (+1393204us) service_pool.cc:224] Handling call
0709 12:12:26.767862 (+ 21752us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:26.791346 (+ 23484us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:12:26.814795 (+ 23449us) log.cc:844] Serialized 14 byte log entry
0709 12:12:27.307187 (+492392us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:27.307813 (+   626us) log.cc:844] Serialized 26 byte log entry
0709 12:12:27.307999 (+   186us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:27.308044 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.308136 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:27.587924 (+279788us) raft_consensus.cc:1685] finished
0709 12:12:27.587953 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.588171 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28937,"prepare.queue_time_us":19683,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":26360,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":8546,"thread_start_us":337698,"threads_started":1,"wal-append.queue_time_us":530501}
I20260709 12:12:27.590323 15062 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.592361 15062 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:27.592684 15209 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.706s	user 0.010s	sys 0.001s
W20260709 12:12:27.281366 15212 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.009s	sys 0.025s
W20260709 12:12:27.593835 15212 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.009s	sys 0.026s
I20260709 12:12:27.594461 15242 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:27.595108 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.283s	user 0.004s	sys 0.040s
I20260709 12:12:27.595301 12638 rpcz_store.cc:276] 0709 12:12:25.301356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.309886 (+  8530us) service_pool.cc:224] Handling call
0709 12:12:27.178927 (+1869041us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:27.238257 (+ 59330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3766323072}
I20260709 12:12:27.596956 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:27.480309 12493 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.008s	sys 0.012s
I20260709 12:12:27.438071 12563 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:27.394716 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 261) took 1053 ms. Trace:
W20260709 12:12:27.599015 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.270s	user 0.016s	sys 0.028s
I20260709 12:12:27.599357 12645 rpcz_store.cc:276] 0709 12:12:26.341000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.341253 (+   253us) service_pool.cc:224] Handling call
0709 12:12:27.394485 (+1053232us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:27.394628 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":977835264}
I20260709 12:12:27.597316 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 264) took 1161 ms. Trace:
W20260709 12:12:27.600849 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.601439 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:27.598126 12493 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.009s	sys 0.017s
I20260709 12:12:27.603263 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:27.599843 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.604532 12569 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:27.604707 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:27.605470 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:27.557611 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:27.596694 13261 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:27.597306 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1686) took 2510 ms. Trace:
I20260709 12:12:27.580816 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:27.442039 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 368) took 1109 ms. Trace:
I20260709 12:12:27.623255 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.677285 12572 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:12:27.601030 12641 rpcz_store.cc:276] 0709 12:12:26.435981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.575876 (+139895us) service_pool.cc:224] Handling call
0709 12:12:26.576150 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:27.034545 (+458395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:27.034577 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:27.034646 (+    69us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:27.034679 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.410699 (+376020us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:27.410750 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.597245 (+186495us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1053602048}
I20260709 12:12:27.619365 12567 rpcz_store.cc:276] 0709 12:12:26.332441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:27.090740 (+758299us) service_pool.cc:224] Handling call
0709 12:12:27.123286 (+ 32546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:27.285854 (+162568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:27.285895 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:27.285954 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:27.286005 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.441758 (+155753us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:27.441812 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.441969 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373203328}
I20260709 12:12:27.686003 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.686930 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:27.687808 12564 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:12:27.688863 12568 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:12:27.594708 15085 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.692891 13260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65037753870c4624995dde1173266f49"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:27.616338 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:27.698623 13260 raft_consensus.cc:606] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:12:27.699146 13259 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:12:27.605335 14990 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.011s	sys 0.019s
I20260709 12:12:27.593115 15209 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.309028 14966 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.309283 15198 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.410683 14567 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:27.304526 15074 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.674850 13255 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:27.393576 15250 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:12:27.328853 15086 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.603720 15241 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:27.330701 15070 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.437374 15056 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.607021 15169 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 3. Current term is 4.
I20260709 12:12:27.606498 15255 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:27.370759 15138 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.446442 15224 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:27.441118 15079 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.626492 15173 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.716401 15114 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:27.716825 14909 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.380179 15010 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.603238 14785 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:27.678826 15212 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:27.307284 15208 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:27.379058 15102 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.606292 14811 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:27.899190 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.164s	user 0.020s	sys 0.017s
W20260709 12:12:27.903036 12488 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.872s	user 0.029s	sys 0.012s
I20260709 12:12:27.470057 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:27.606547 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:27.915194 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.621994 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:27.469844 13254 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:27.843951 15052 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.607615 12495 rpcz_store.cc:276] 0709 12:12:25.086963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.730699 (+1643736us) service_pool.cc:224] Handling call
0709 12:12:26.732587 (+  1888us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:26.732956 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:26.737854 (+  4898us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:26.738396 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:12:27.596992 (+858596us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:27.597034 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.597097 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:27.597134 (+    37us) raft_consensus.cc:1685] finished
0709 12:12:27.597145 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.597246 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29649,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":849647,"threads_started":1,"wal-append.queue_time_us":629683}
I20260709 12:12:27.917270 12496 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
W20260709 12:12:27.795236 15073 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.644s	user 0.012s	sys 0.000s
I20260709 12:12:27.844961 15085 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.321462 15147 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.022012 13260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:28.022235 15111 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:28.023675 13260 raft_consensus.cc:606] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:28.024394 13260 raft_consensus.cc:614] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:27.716719 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 464) took 1065 ms. Trace:
I20260709 12:12:27.752741 14567 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:28.026144 15154 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:28.027226 14567 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.685464 14930 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:12:28.030197 15114 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:28.031002 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:27.824781 14986 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.727646 14966 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:27.393565 15006 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.032790 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:27.759366 15099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.012s	sys 0.020s
I20260709 12:12:28.033864 12570 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:12:28.034471 15099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.022s	user 0.013s	sys 0.021s
W20260709 12:12:28.035185 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.703958 15113 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:28.010412 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 462) took 1420 ms (client timeout 1814 ms). Trace:
I20260709 12:12:28.036849 14966 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.036358 15113 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.038504 15147 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:27.744889 15136 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.001s	user 0.011s	sys 0.002s
W20260709 12:12:28.036091 13254 rpcz_store.cc:269] 0709 12:12:26.590339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.641052 (+ 50713us) service_pool.cc:224] Handling call
0709 12:12:28.010041 (+1368989us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:27.747934 15074 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.810702 15208 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.842952 15209 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.041950 14876 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.043346 14876 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.026183 13256 rpcz_store.cc:276] 0709 12:12:26.651245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.659768 (+  8523us) service_pool.cc:224] Handling call
0709 12:12:26.660144 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.660390 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:27.166808 (+506418us) log.cc:844] Serialized 14 byte log entry
0709 12:12:27.696207 (+529399us) log.cc:844] Serialized 14 byte log entry
0709 12:12:27.716273 (+ 20066us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:27.716344 (+    71us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:27.716410 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:27.716496 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:27.716651 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19421,"prepare.queue_time_us":778645,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":109,"raft.queue_time_us":54184,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":9615,"thread_start_us":716136,"threads_started":2,"wal-append.queue_time_us":576147}
I20260709 12:12:28.044530 15208 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.698601 15167 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.840819 15173 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.734498 15198 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.054637 15198 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.326985 15216 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.385s
I20260709 12:12:28.055547 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 463) took 1405 ms. Trace:
I20260709 12:12:28.056228 13255 rpcz_store.cc:276] 0709 12:12:26.650625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.651387 (+   762us) service_pool.cc:224] Handling call
0709 12:12:26.651908 (+   521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:27.762382 (+1110474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.054989 (+292607us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:28.055317 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2172039552}
I20260709 12:12:28.056768 15074 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.796922 15224 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.060186 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:27.842377 15253 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.421s	user 0.002s	sys 0.000s
I20260709 12:12:27.735987 15271 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.064340 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 499) took 1532 ms. Trace:
I20260709 12:12:27.754145 15070 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.064973 13257 rpcz_store.cc:276] 0709 12:12:26.521483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.521695 (+   212us) service_pool.cc:224] Handling call
0709 12:12:26.803509 (+281814us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:27.596486 (+792977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.052277 (+455791us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:28.053159 (+   882us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400113fd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:28.053424 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546510592}
W20260709 12:12:28.008941 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 386822us
I20260709 12:12:28.068154 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:28.069180 15070 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:27.751060 15221 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:28.075896 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:28.077260 12492 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:28.079623 12494 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:28.000072 15169 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:28.003222 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.003566 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.014513 15102 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.015698 15248 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:28.016067 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:28.017916 12491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.011s	sys 0.004s
I20260709 12:12:28.025769 14811 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.091828 15169 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.092404 14811 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.754880 15190 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:27.814528 15212 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:28.095762 15169 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.098739 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1916) took 1805 ms. Trace:
I20260709 12:12:28.041409 14567 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:28.063750 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:27.769236 15255 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.069784 15278 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:28.099254 12563 rpcz_store.cc:276] 0709 12:12:26.293217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.663392 (+370175us) service_pool.cc:224] Handling call
0709 12:12:28.093559 (+1430167us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:28.098513 (+  4954us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1778964864}
I20260709 12:12:28.086223 15113 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.106688 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.107352 15255 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:28.108259 12494 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:28.108683 15102 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.110479 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:28.112963 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 370) took 1777 ms. Trace:
I20260709 12:12:28.113722 12566 rpcz_store.cc:276] 0709 12:12:26.335003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:27.108655 (+773652us) service_pool.cc:224] Handling call
0709 12:12:27.109001 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:27.693337 (+584336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:27.693373 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:27.693420 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:27.693454 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.112311 (+418857us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b54001a27d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:28.112396 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:28.112719 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1343312128}
I20260709 12:12:28.064917 15271 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:27.804039 14785 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.115091 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:28.115464 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 445509us
I20260709 12:12:28.070581 12569 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:12:28.116470 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 386418us
I20260709 12:12:28.116937 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.075966 13257 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:28.098974 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:28.102134 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.114722 15198 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:27.967139 12488 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.030s	sys 0.012s
W20260709 12:12:28.145993 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.147965 15209 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:28.115170 15074 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20260709 12:12:28.114287 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46427.9us
I20260709 12:12:28.120200 15271 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.084895 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:28.123382 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.01591e+06us
I20260709 12:12:28.123890 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:28.124689 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.125174 14811 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.126909 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2026) took 2615 ms. Trace:
W20260709 12:12:28.128100 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1913) took 1872 ms (client timeout 2025 ms). Trace:
I20260709 12:12:28.129243 12490 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:28.123423 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:28.131670 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:28.133054 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:28.096558 15212 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.134325 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:28.135008 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.012s	sys 0.005s
I20260709 12:12:28.133457 13259 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 7 LEARNER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:28.136206 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:28.137953 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 504) took 1412 ms. Trace:
I20260709 12:12:28.138171 15102 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.141454 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:28.142623 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120953us
I20260709 12:12:28.145025 13257 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:28.145110 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2029) took 1695 ms. Trace:
I20260709 12:12:28.160652 12496 rpcz_store.cc:276] 0709 12:12:25.511867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.865548 (+1353681us) service_pool.cc:224] Handling call
0709 12:12:26.866011 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:28.010091 (+1144080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.126534 (+116443us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:28.126796 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2415786240}
W20260709 12:12:28.161141 12570 rpcz_store.cc:269] 0709 12:12:26.255437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.594098 (+338661us) service_pool.cc:224] Handling call
0709 12:12:28.127838 (+1533740us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:28.128028 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2668386048}
I20260709 12:12:28.159166 12489 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:28.162883 13261 rpcz_store.cc:276] 0709 12:12:26.725492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.725652 (+   160us) service_pool.cc:224] Handling call
0709 12:12:28.137448 (+1411796us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:28.137751 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1303375744}
I20260709 12:12:28.180826 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:28.184302 12495 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:12:28.186455 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.187144 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:28.189936 12495 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:28.191365 15274 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.128s	user 0.004s	sys 0.000s
I20260709 12:12:28.172932 12493 rpcz_store.cc:276] 0709 12:12:26.449232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.899473 (+450241us) service_pool.cc:224] Handling call
0709 12:12:26.899888 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.900144 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:27.603090 (+702946us) log.cc:844] Serialized 14 byte log entry
0709 12:12:28.143075 (+539985us) log.cc:844] Serialized 14 byte log entry
0709 12:12:28.143733 (+   658us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:28.143854 (+   121us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:28.144006 (+   152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.144182 (+   176us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:28.144444 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":867443,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":908,"raft.queue_time_us":309,"spinlock_wait_cycles":84105984,"thread_start_us":1149443,"threads_started":2,"wal-append.queue_time_us":415642}
I20260709 12:12:28.185410 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.194442 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:28.185225 15074 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:28.203351 12494 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:12:28.216457 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1694) took 1071 ms. Trace:
I20260709 12:12:28.219489 12497 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260709 12:12:28.214442 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97794.1us
I20260709 12:12:28.219172 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.220247 12494 rpcz_store.cc:276] 0709 12:12:27.141581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:28.109297 (+967716us) service_pool.cc:224] Handling call
0709 12:12:28.109978 (+   681us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:28.209602 (+ 99624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.212909 (+  3307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:28.202507 15074 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:28.215287 14785 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.236471 15099 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:28.239498 15099 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:28.239961 15099 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.226141 15255 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:28.248169 12609 leader_election.cc:400] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7.
I20260709 12:12:28.248560 12609 leader_election.cc:403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:28.242215 15255 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:28.250833 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:12:28.251878 15255 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.236739 15165 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:28.253916 15242 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:12:28.254734 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:28.256930 15222 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:28.257459 15222 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:28.257839 15222 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:28.258342 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:28.252207 15074 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.240144 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:28.260013 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2027) took 1834 ms. Trace:
I20260709 12:12:28.238406 15113 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.274827 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:28.265167 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.279035 15074 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.269253 15209 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:28.279922 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2024) took 2896 ms. Trace:
I20260709 12:12:28.269011 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:28.281128 15278 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7.
I20260709 12:12:28.267191 12488 rpcz_store.cc:276] 0709 12:12:26.413908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.948936 (+535028us) service_pool.cc:224] Handling call
0709 12:12:26.949376 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.949683 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:28.146194 (+1196511us) log.cc:844] Serialized 14 byte log entry
0709 12:12:28.227634 (+ 81440us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:28.233165 (+  5531us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:28.236183 (+  3018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.236404 (+   221us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:28.247731 (+ 11327us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55694,"prepare.queue_time_us":855403,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"raft.queue_time_us":2036,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":8587,"thread_start_us":1183509,"threads_started":2}
I20260709 12:12:28.283644 14876 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.294993 15209 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:28.314740 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68837.2us
W20260709 12:12:28.312168 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66746.2us
I20260709 12:12:28.295153 14567 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:28.294154 15169 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:28.295320 12491 rpcz_store.cc:276] 0709 12:12:25.382988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:26.864424 (+1481436us) service_pool.cc:224] Handling call
0709 12:12:26.878690 (+ 14266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:26.913482 (+ 34792us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:27.375161 (+461679us) log.cc:844] Serialized 14 byte log entry
0709 12:12:28.169993 (+794832us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:28.170091 (+    98us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:28.170177 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.190346 (+ 20169us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:28.279427 (+ 89081us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:28.279507 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:28.279822 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":558,"prepare.queue_time_us":883898,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":48,"raft.queue_time_us":1101,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"spinlock_wait_cycles":134224512,"thread_start_us":1156236,"threads_started":2,"wal-append.queue_time_us":687519}
I20260709 12:12:28.297669 15113 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.317759 15209 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:28.335045 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95038.7us
I20260709 12:12:28.311610 14811 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:12:28.341233 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1897) took 2895 ms. Trace:
I20260709 12:12:28.328576 15190 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:28.346439 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99003.4us
I20260709 12:12:28.344077 12571 rpcz_store.cc:276] 0709 12:12:25.351676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:25.620536 (+268860us) service_pool.cc:224] Handling call
0709 12:12:25.620990 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:25.621282 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:25.621706 (+   424us) log.cc:844] Serialized 14 byte log entry
0709 12:12:26.647190 (+1025484us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:27.602911 (+955721us) log.cc:844] Serialized 14 byte log entry
0709 12:12:28.162026 (+559115us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:28.203121 (+ 41095us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:28.218120 (+ 14999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:28.227259 (+  9139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:28.246908 (+ 19649us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":115143,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":114,"raft.queue_time_us":214842,"raft.run_cpu_time_us":428,"raft.run_wall_time_us":37445,"spinlock_wait_cycles":55249664,"thread_start_us":2423175,"threads_started":3,"wal-append.queue_time_us":508973}
I20260709 12:12:28.367851 15136 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:28.368673 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.368845 15136 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:28.370035 15281 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.284s	user 0.004s	sys 0.000s
I20260709 12:12:28.372195 15136 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.356498 15169 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.346225 15283 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.375661 15006 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:28.376078 15283 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.377833 15010 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:28.378031 14967 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:28.378548 15010 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:28.379078 15010 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.380029 15278 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:28.380510 15278 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:28.380733 15079 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:28.381672 15278 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:28.386006 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122503us
I20260709 12:12:28.387102 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:28.365792 15190 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.360628 15216 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:12:28.395836 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 216098us
I20260709 12:12:28.357025 14785 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:28.364410 15282 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.278s	user 0.004s	sys 0.000s
I20260709 12:12:28.349546 15224 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.395720 15278 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:28.422797 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.964s (SENT)
I20260709 12:12:28.423013 15190 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.389554 15167 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:28.437295 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 175144us
I20260709 12:12:28.438696 15283 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.397917 15079 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:28.356426 14567 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.446339 15167 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:28.478756 15079 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.480131 15167 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.457198 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.483532 12488 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:28.484793 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:28.457798 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.458236 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.489610 12564 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:28.489770 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.483565 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:28.458315 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.459024 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:28.459167 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.485545 12488 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.459816 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.495107 15271 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.495442 12568 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:28.459923 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.495812 15271 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.460242 12572 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20260709 12:12:28.460862 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.461721 12571 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:12:28.462217 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.462324 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.462661 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.463246 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.469257 15169 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:28.473176 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.473932 14811 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.474645 15136 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.494609 15113 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:28.495926 15006 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:28.496687 12563 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:28.572119 12571 pending_rounds.cc:85] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 3
I20260709 12:12:28.571786 15136 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.572607 12571 pending_rounds.cc:107] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:12:28.507539 15224 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.573050 12571 raft_consensus.cc:2889] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:28.573392 12497 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:28.574560 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:28.516402 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.575941 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.576800 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:28.576893 12638 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:28.577080 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:28.520267 15242 raft_consensus.cc:993] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:12:28.578212 15242 raft_consensus.cc:1081] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:12:28.578761 15006 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:28.579254 15006 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.578164 12493 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:28.512620 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.584388 12491 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:28.509544 14567 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.539140 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:12:28.587965 12496 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:28.516481 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.598498 12488 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:28.503429 15271 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.601006 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.514622 15165 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:28.503417 15277 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:12:28.538662 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.607968 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.607968 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.532128 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.608860 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.611717 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:28.588289 14785 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.513911 15190 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.614857 12570 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:28.620371 12646 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:28.614810 12497 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:28.612299 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:28.622143 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 208045us
I20260709 12:12:28.591605 15169 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:28.601330 15224 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.617100 12645 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:28.628563 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.657451 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:28.658474 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 181303us
W20260709 12:12:28.660382 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58641.4us
I20260709 12:12:28.551990 15099 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.674381 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:28.674893 15136 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.675645 15099 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:28.676016 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.677806 15283 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:28.678862 15283 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:12:28.679471 15283 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:28.680303 15283 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:12:28.680927 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:28.682665 13218 leader_election.cc:336] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.972s (SENT)
I20260709 12:12:28.683295 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:28.660780 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:28.664547 15299 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:12:28.664948 12572 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:28.665342 15224 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:28.690256 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:28.665982 12392 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 5, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:28.693197 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.671253 14876 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:28.610083 15165 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:28.615180 15167 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:28.683589 15190 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.691448 12570 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:28.668385 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:28.612576 14811 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.680696 15283 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:28.707077 12569 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:28.712236 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.713847 15147 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:12:28.714470 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.714756 15147 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:28.715144 15147 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:28.715508 15147 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.723167 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:28.726193 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:28.700438 15165 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.628345 15010 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.761380 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.766543 14811 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.767046 12495 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:28.727958 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:28.768098 12571 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.768200 15010 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:28.662174 14785 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.773059 14785 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:28.774502 15297 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:12:28.777493 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:28.732008 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.781596 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:28.736502 15209 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.704198 15099 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:28.702240 15169 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.738142 12642 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:12:28.738193 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46985.4us
I20260709 12:12:28.739256 15278 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:28.739835 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.741809 12639 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:12:28.743448 15222 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.744099 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.793682 12563 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:28.797677 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:28.747300 15198 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.006s
I20260709 12:12:28.747956 12570 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.749125 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.750159 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:28.750664 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.757843 14876 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:28.760502 15224 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.762096 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:28.762781 15190 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:28.818320 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:28.818409 15010 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.818936 12497 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:28.762362 12392 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 4, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 4 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:28.819353 15010 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.820761 12392 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:28.822109 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.822345 12392 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 3, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:28.823411 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.720360 15283 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.791247 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:28.803912 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54903.4us
I20260709 12:12:28.805801 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:28.805511 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.806351 15111 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:12:28.827076 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.825842 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:28.835067 12641 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.805687 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:28.837007 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 154382us
I20260709 12:12:28.825794 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.806595 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:28.806540 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.807188 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.809508 15209 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.811543 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:28.841519 12569 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:28.824641 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.842108 12563 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.821394 15113 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:28.765128 15277 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.825150 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.835999 12489 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:28.841609 13259 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.811921 12490 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:28.803340 15198 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:28.797385 15102 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:28.823810 12392 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:28.823694 15010 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.851434 12638 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:28.854102 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:28.791163 15006 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.850035 13257 raft_consensus.cc:2376] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9. Re-sending same reply.
I20260709 12:12:28.858235 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.859010 12565 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:28.859633 15006 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:28.859975 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:28.869719 12572 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:12:28.870182 12572 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 5
I20260709 12:12:28.870532 12572 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:12:28.870769 12572 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:28.872468 12572 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:28.873250 12572 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:12:28.877878 15113 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.882635 12644 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:28.921535 12641 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:12:28.883154 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.846719 15074 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:28.885842 15102 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:28.887451 12392 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 6, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:28.890542 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:28.891062 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 141043us
I20260709 12:12:28.896766 12569 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:28.845496 15190 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:28.906572 12497 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:28.907302 14876 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:28.907904 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:28.908638 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 225) took 20870 ms (client timeout 15000 ms). Trace:
I20260709 12:12:28.909406 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:28.858253 12488 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:28.913244 12571 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:28.860579 12565 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:28.918684 12572 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:12:28.936229 12572 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:28.921999 12563 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:28.923136 12570 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:12:28.923810 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86850.6us
I20260709 12:12:28.924919 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:28.928663 13259 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:28.938508 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:28.939522 15209 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:28.940182 15209 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:28.943549 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:28.947479 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.924436 15074 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:28.957047 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:28.960798 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81836.2us
W20260709 12:12:28.968217 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60428.3us
I20260709 12:12:28.953918 15111 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:28.965987 12392 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:28.976295 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:28.969348 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:28.979355 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:28.980772 12646 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:28.982594 14716 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:28.964658 15010 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:28.985692 12489 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:12:28.989288 14930 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:28.990845 14930 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:28.966672 15074 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:28.929986 15147 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:28.975595 14876 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:28.986105 15006 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:29.027347 12495 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:28.967829 15224 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:12:28.964418 15099 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:12:28.996711 15111 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:29.030937 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 226897us
I20260709 12:12:29.033473 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.037540 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.005900 15278 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:29.038378 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62249.1us
I20260709 12:12:29.046872 12646 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:29.050613 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.052672 12488 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:29.016212 14876 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:29.086633 14876 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:29.028844 15006 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.049403 15278 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.058604 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.093080 12641 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:29.059376 15302 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:12:29.059720 15010 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:29.063416 15242 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:29.076538 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:29.077081 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:29.077768 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.110740 12488 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:29.078634 15222 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:29.080464 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.048659 15111 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:28.934137 12395 rpcz_store.cc:269] 0709 12:12:08.049779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:08.050247 (+   468us) service_pool.cc:224] Handling call
0709 12:12:28.908378 (+20858131us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:25.481818 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:25.707917 (+226099us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:25.791572 (+ 83655us) write_op.cc:435] Acquired schema lock
0709 12:12:25.794550 (+  2978us) tablet.cc:662] Decoding operations
0709 12:12:25.924805 (+130255us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:25.924904 (+    99us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:25.924948 (+    44us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:12:25.926600 (+  1652us) tablet.cc:701] Row locks acquired
0709 12:12:25.926627 (+    27us) write_op.cc:260] PREPARE: finished
0709 12:12:25.926842 (+   215us) write_op.cc:270] Start()
0709 12:12:25.926993 (+   151us) write_op.cc:276] Timestamp: P: 1783599145926773 usec, L: 0
0709 12:12:25.927021 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:12:25.927949 (+   928us) log.cc:844] Serialized 6598 byte log entry
0709 12:12:26.654496 (+726547us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:27.703578 (+1049082us) op_driver.cc:464] REPLICATION: finished
0709 12:12:27.857428 (+153850us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b18003d4ec0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:27.857655 (+   227us) write_op.cc:301] APPLY: starting
0709 12:12:27.857793 (+   138us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:27.858291 (+   498us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:27.858326 (+    35us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:27.860995 (+  2669us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:27.861448 (+   453us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0709 12:12:27.861525 (+    77us) write_op.cc:312] APPLY: finished
0709 12:12:27.862433 (+   908us) log.cc:844] Serialized 113 byte log entry
0709 12:12:27.863682 (+  1249us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:27.863966 (+   284us) write_op.cc:454] Released schema lock
0709 12:12:27.864055 (+    89us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:27.865129 (+  1074us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:27.865177 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":43,"child_traces":[["op",{"apply.queue_time_us":144309,"apply.run_cpu_time_us":5621,"apply.run_wall_time_us":7428,"num_ops":16,"prepare.queue_time_us":134514,"prepare.run_cpu_time_us":9005,"prepare.run_wall_time_us":744891,"raft.queue_time_us":902062,"raft.run_cpu_time_us":25160,"raft.run_wall_time_us":270694,"replication_time_us":1747147,"spinlock_wait_cycles":64739072,"thread_start_us":826640,"threads_started":4,"wal-append.queue_time_us":199309,"wal-append.run_cpu_time_us":80218,"wal-append.run_wall_time_us":2736407}]]}
I20260709 12:12:29.082365 12497 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20260709 12:12:29.083372 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.085222 12496 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:12:29.112324 12493 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:29.055992 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54778us
W20260709 12:12:29.098157 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47597.4us
I20260709 12:12:29.116125 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:29.113306 12497 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 1
I20260709 12:12:29.118433 12497 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260709 12:12:29.118848 12497 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:29.101584 12572 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:29.127348 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:29.127377 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:29.138749 12392 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 6, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.143931 12497 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:12:29.144415 12497 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:29.101042 15006 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:29.147058 12572 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:29.148712 12392 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:29.099555 15278 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.144219 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.112483 15010 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.112336 15167 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:29.158174 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.158898 12641 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:29.179852 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.184092 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.143340 15079 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:29.152776 12392 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.142809 15242 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:29.189299 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.190433 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.149948 14785 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:29.147432 15221 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:50128 (request call id 11) took 39925 ms. Trace:
I20260709 12:12:29.195909 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:29.159821 15167 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.225315 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.228866 12641 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:29.229812 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.231175 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:29.231837 12488 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:29.232816 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.232939 15169 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:29.197188 15113 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:29.234143 15113 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:29.192303 15165 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.236742 15165 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:29.236950 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.238123 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.196127 15283 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:29.238914 12495 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:29.233382 12488 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:29.089448 15209 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:29.197966 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2363815839999006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3164s for a response
I20260709 12:12:29.245383 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.251967 12496 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:29.248849 15190 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:29.238229 15221 rpcz_store.cc:276] 0709 12:11:49.020872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:11:49.021052 (+   180us) service_pool.cc:224] Handling call
0709 12:11:49.697226 (+676174us) spinlock_profiling.cc:243] Waited 8.87 ms on lock 0x7b54002745a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5770520 00007ffbf57f9b42 00007ffbf57f8800 00007ffbfa7b9d29 00007ffbf3d381d4 00007ffbf3d38162 00007ffbf3d38091 00007ffbf3d3800c 00007ffbf3d372c2 00007ffbf3b2e48c
0709 12:11:49.723486 (+ 26260us) log.cc:844] Serialized 506 byte log entry
0709 12:11:50.369861 (+646375us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b54002745a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5773b26 00007ffbf576d49a 00007ffbf576c714 00007ffbf574ceaa 00007ffbf5753d53 00007ffbf5753cd9 00007ffbf5753c69 00007ffbf5753c31 00007ffbf5752dfd 0000000000388634
0709 12:12:28.935070 (+38565209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21369,"raft.queue_time_us":134452,"raft.run_cpu_time_us":12904,"raft.run_wall_time_us":616386,"spinlock_wait_cycles":24261376,"thread_start_us":297081,"threads_started":3,"wal-append.queue_time_us":149239,"wal-append.run_cpu_time_us":2253,"wal-append.run_wall_time_us":1012029}
I20260709 12:12:29.238394 15167 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:29.263234 15283 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:29.264196 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68368.2us
I20260709 12:12:29.265586 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.270581 12489 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
W20260709 12:12:29.269212 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 281282us
I20260709 12:12:29.157807 15010 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.266973 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.275563 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.277357 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:29.284351 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 157105us
I20260709 12:12:29.300118 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.299252 12493 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:29.315855 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.318467 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.314323 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:29.322751 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:29.201640 12392 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 3, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.274262 15190 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:29.281309 15102 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:29.335412 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.280225 14716 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:29.345432 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:29.329238 15169 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:29.350133 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49272.2us
W20260709 12:12:29.352924 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70758.2us
I20260709 12:12:29.331956 15079 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:29.334239 15147 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:29.353729 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:29.355244 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.352674 12392 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.333283 15190 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:29.366562 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:29.367489 12641 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:29.372236 14785 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:29.372870 14785 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:29.373922 14785 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:29.375702 15074 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.376209 15242 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:29.377799 15314 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:12:29.357317 15102 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 12:12:29.386747 12392 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.356477 15169 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:29.338305 15222 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:12:29.358973 15079 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.411334 15325 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:29.411728 15079 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:29.412961 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:29.413651 13262 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:29.413967 13262 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:29.414033 15312 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:12:29.414855 13262 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.416134 13262 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:29.396481 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 285560us
I20260709 12:12:29.365648 15190 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:29.403169 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52749.9us
I20260709 12:12:29.387744 15136 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:29.429955 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111610us
I20260709 12:12:29.424947 12646 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
W20260709 12:12:29.437294 15307 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260709 12:12:29.427764 15085 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:29.439368 15085 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.439740 15085 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.441938 12392 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.430404 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.444408 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:29.442324 15309 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:12:29.450259 15313 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:12:29.449900 14716 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:29.451522 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:29.454349 15147 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:29.455042 15147 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.457320 15147 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:29.457922 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79249.9us
I20260709 12:12:29.463917 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.401358 15169 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:29.459371 15323 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:12:29.460762 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.420953 15165 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:29.477965 13259 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:12:29.462930 15324 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.003s	sys 0.000s
I20260709 12:12:29.465381 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.481168 12392 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 3, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.483630 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.471581 13262 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:29.435765 15136 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:29.504688 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44027.5us
W20260709 12:12:29.511410 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 153589us
I20260709 12:12:29.515440 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:29.516306 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:29.480630 15209 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:29.483644 15074 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:29.487774 15147 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:12:29.503743 15136 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:29.531847 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.497838 15154 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:29.486435 15326 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:12:29.548548 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 118265us
I20260709 12:12:29.486816 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:29.502571 15165 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.550067 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:29.552150 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120253us
I20260709 12:12:29.560544 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.562451 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:29.561480 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 207831us
I20260709 12:12:29.563670 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:29.565212 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.564687 12392 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 2, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 2 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:29.566184 12646 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:29.517100 14876 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:29.569947 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49507.8us
I20260709 12:12:29.564579 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.582041 12392 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.530028 15209 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.567011 15074 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:29.601882 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.579805 14785 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:29.610368 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.579785 15331 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:29.609614 15327 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.059s
I20260709 12:12:29.616892 12392 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 5, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:12:29.618816 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46815.2us
I20260709 12:12:29.624661 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.624176 12392 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 4 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:12:29.631711 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.630597 12392 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 6, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:12:29.632933 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69332.3us
I20260709 12:12:29.633787 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.635682 12392 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:12:29.640321 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78252.9us
I20260709 12:12:29.618626 15327 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:29.625965 15074 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:29.657886 15074 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.658402 15074 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:29.662163 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.642635 12637 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:29.665148 15283 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:29.646032 15331 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:29.646318 15337 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:29.647768 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:29.649392 15329 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.125s	user 0.002s	sys 0.000s
I20260709 12:12:29.652971 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:29.601090 15167 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:29.654405 15328 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:12:29.696918 15102 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:29.654800 15333 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:29.697607 15102 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:29.697805 15333 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:29.688699 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.698822 15102 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.698928 15333 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:29.656647 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.616636 15332 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:12:29.672978 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111559us
W20260709 12:12:29.654702 15322 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.242s	user 0.003s	sys 0.000s
W20260709 12:12:29.685909 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52405.8us
I20260709 12:12:29.691548 15277 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:29.692250 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:29.692504 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 81a9a92616494f46bc97254b53185664: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:29.713351 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:29.640285 15222 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:29.720532 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:29.721946 13259 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:29.665699 15283 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:29.696982 15336 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:12:29.708431 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.700131 15325 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:29.706055 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.690531 15337 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.686561 15331 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.732681 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.752156 12637 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:29.688491 15327 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:29.697821 15167 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:29.754148 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49703.5us
W20260709 12:12:29.754213 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49647.5us
I20260709 12:12:29.705417 15277 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.759747 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
W20260709 12:12:29.761559 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59663.2us
I20260709 12:12:29.769008 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:29.769735 12572 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.770467 15277 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:29.772229 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:29.772992 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:29.769519 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:29.751430 15337 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:29.774070 12565 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:29.737533 15283 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:29.779633 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:29.780923 12565 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:29.784520 12564 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:12:29.785492 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 132611us
I20260709 12:12:29.739761 15325 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:29.812773 12572 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:29.758105 15331 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:29.816843 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.825017 12637 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:29.774744 15079 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:29.782287 15113 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:29.837158 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:29.838954 12488 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.764601 15340 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:29.866214 15327 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:29.870924 15291 raft_consensus.cc:993] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: : Instructing follower 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:12:29.872092 14876 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:29.776248 15332 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
W20260709 12:12:29.853273 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.506s (SENT) [suppressed 5 similar messages]
I20260709 12:12:29.850342 12497 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:12:29.846899 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 190393us
I20260709 12:12:29.809932 15325 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:29.887600 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:29.881067 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1623532859998704s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20260709 12:12:29.881745 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:29.881916 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:29.885846 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 182786us
I20260709 12:12:29.889951 12570 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:29.890127 12568 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:29.849395 15307 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:12:29.902557 15079 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:29.916910 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.917888 12646 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:29.918967 15291 raft_consensus.cc:1081] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Signalling peer 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:12:29.919451 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.922878 12489 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:29.923619 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:29.923965 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:29.929589 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 229401us
I20260709 12:12:29.946190 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.947420 12646 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:12:29.894061 15343 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:29.921237 15222 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:29.961370 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:29.932126 15147 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:29.987108 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63630.9us
W20260709 12:12:29.988173 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149601us
I20260709 12:12:29.989471 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:29.990665 12488 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:29.926932 15169 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:29.975622 15006 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.784s	user 0.008s	sys 0.007s
I20260709 12:12:29.998624 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:29.945859 15291 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:12:29.932329 15085 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:29.966881 15349 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:30.005297 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.007333 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.006691 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:29.995213 15242 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:30.017275 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.012303 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:29.975142 15222 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:30.021030 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:30.026595 12537 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:30.027088 12537 leader_election.cc:403] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:12:29.990063 15190 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.037504 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.013180 15346 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:30.013594 14876 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:29.997906 14716 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:12:29.994804 15147 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:30.017287 15113 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:29.999922 15006 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.049476 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.051785 15165 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:30.027601 15340 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:30.052711 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.037837 15349 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:12:30.054179 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.055018 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.020428 15351 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:30.057497 12568 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:30.057497 12488 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:30.069859 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:30.094534 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 160248us
W20260709 12:12:30.074697 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179443us
I20260709 12:12:30.208995 12639 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:30.209524 12641 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:30.230970 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 329914us
W20260709 12:12:30.234189 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 207798us
W20260709 12:12:30.207587 15074 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.543s	user 0.008s	sys 0.004s
W20260709 12:12:30.194227 15010 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.862s	user 0.005s	sys 0.010s
W20260709 12:12:30.172931 15278 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.981s	user 0.014s	sys 0.000s
I20260709 12:12:30.184283 15169 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:30.254987 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:30.279735 15209 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:30.283754 12640 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:30.208117 15361 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:30.299485 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:30.300523 12564 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:30.050736 14876 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:30.050092 15346 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:30.259013 12637 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:30.071802 15147 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:30.262224 15278 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.266608 15085 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.269861 14986 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:30.270816 15113 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.051002 15216 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.224s
W20260709 12:12:30.323243 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73891.7us
I20260709 12:12:30.140120 15277 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:30.245927 15111 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.095s	user 0.011s	sys 0.005s
I20260709 12:12:30.290504 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:30.255445 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.253713 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:30.289374 12639 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:30.292155 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.306267 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:30.332659 12572 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:30.065670 15343 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
W20260709 12:12:30.308707 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 265168us
W20260709 12:12:30.337993 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82654.7us
I20260709 12:12:30.316798 12641 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:30.318552 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:30.342530 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
W20260709 12:12:30.287955 14785 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.668s	user 0.013s	sys 0.000s
W20260709 12:12:30.273270 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:30.166960 15337 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.358019 15337 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.359915 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:30.320634 15331 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.368294 14986 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:30.373965 15340 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.377444 15278 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.324855 15190 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.378448 12644 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:30.379005 14785 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:30.379813 15006 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.253170 15074 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.338807 15111 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.383477 15074 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.320508 15346 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.384517 15111 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.385416 12490 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:30.352265 15147 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:30.388616 15147 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:30.388991 15147 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:30.326327 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 318985us
I20260709 12:12:30.286931 15319 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:30.390173 15319 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:30.390599 15319 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:30.390842 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:30.312144 14876 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.319908 15361 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.355681 15277 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.416147 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:30.416895 12568 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.395210 12563 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:12:30.393592 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.418499 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.418792 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:30.422523 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.395388 12564 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:30.411374 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69570.7us
I20260709 12:12:30.424242 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.258608 15010 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.426594 15010 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.428984 12564 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:30.431309 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:12:30.394605 15283 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.571s	user 0.011s	sys 0.001s
I20260709 12:12:30.434124 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:30.442495 12571 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:30.435006 13257 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:30.395480 15136 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.840s	user 0.014s	sys 0.007s
I20260709 12:12:30.444094 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:30.369472 15331 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.447698 13261 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:30.448104 15358 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:12:30.449640 13253 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 5 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:12:30.446233 15190 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:30.414459 15363 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:12:30.453265 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1708) took 1374 ms (client timeout 1601 ms). Trace:
I20260709 12:12:30.455397 15346 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.458353 15349 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:30.414422 15074 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:30.458503 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.459242 15102 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:12:30.526383 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 187714us
W20260709 12:12:30.530159 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105992us
W20260709 12:12:30.514458 12490 rpcz_store.cc:269] 0709 12:12:29.078784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.079988 (+  1204us) service_pool.cc:224] Handling call
0709 12:12:30.453003 (+1373015us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:30.453165 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2928242560}
W20260709 12:12:30.536603 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183932us
I20260709 12:12:30.433924 15343 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:30.600571 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.451248 14986 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:30.443991 15085 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:30.659274 15085 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:30.614711 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196459us
I20260709 12:12:30.663435 15361 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.615223 15363 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:30.665112 15346 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.665012 15363 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.614305 12565 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:30.674714 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 421333us
W20260709 12:12:30.649917 15362 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.297s	user 0.001s	sys 0.000s
I20260709 12:12:30.393859 15325 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:30.653895 15369 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20260709 12:12:30.654666 15366 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:30.656913 12568 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:30.553879 15349 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:30.647895 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 189442us
I20260709 12:12:30.460546 15222 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.701086 15326 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:30.706758 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.722954 15325 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.723716 15074 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:30.724381 15074 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:30.714352 15366 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:30.724076 15356 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:12:30.725428 15356 consensus_queue.cc:1243] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:12:30.725301 15366 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:30.724915 15074 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:30.615724 15343 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:30.730089 15343 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:30.732365 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:30.704416 15357 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 12:12:30.734861 15357 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:30.459761 15165 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.246s
I20260709 12:12:30.735276 15357 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:30.615062 15169 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:30.736330 15357 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.459877 15283 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.738662 12566 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:12:30.715651 15349 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.744367 15283 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.460942 15136 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.750514 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:30.625473 15102 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:30.756284 15102 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.749868 15085 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.758565 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.759441 12492 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:30.761376 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.762320 12488 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:30.751700 12490 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:12:30.766372 15325 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:30.766963 15325 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:30.769047 15136 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:30.769407 15325 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:30.758167 12495 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:30.765251 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46044.6us
I20260709 12:12:30.774806 12536 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:30.775267 12536 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:12:30.779364 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.780459 12641 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:30.784248 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.786432 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.791745 12638 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:12:30.773901 14785 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:12:30.794498 15283 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6.
I20260709 12:12:30.794660 14785 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:30.795610 14785 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:30.796021 12640 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:30.796573 14785 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:30.805157 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 204702us
I20260709 12:12:30.807199 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.681244 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:30.814317 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:30.816417 12492 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:30.819010 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:30.824752 12488 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:30.826781 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:30.826879 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121262us
I20260709 12:12:30.833987 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.834628 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:30.834698 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.828558 12495 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65037753870c4624995dde1173266f49"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:54234
I20260709 12:12:30.834736 12641 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:30.835703 12638 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:30.836140 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1790) took 1049 ms. Trace:
I20260709 12:12:30.835693 12495 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:12:30.835144 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.850021 12495 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:30.806528 15357 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.868687 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.844017 15346 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:30.845234 15222 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:30.809309 13257 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:12:30.841174 15353 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.955s	user 0.006s	sys 0.009s
I20260709 12:12:30.872459 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.852005 15079 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:30.872536 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.873219 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.879148 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:30.880993 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.882616 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.812346 15169 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:30.883776 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77791.8us
I20260709 12:12:30.849979 12563 rpcz_store.cc:276] 0709 12:12:29.786855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.787009 (+   154us) service_pool.cc:224] Handling call
0709 12:12:29.918812 (+131803us) spinlock_profiling.cc:243] Waited 66.7 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cef8c 00007ffbf26ceef6 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:29.919029 (+   217us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:30.417562 (+498533us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:30.835736 (+418174us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:30.836028 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1247736960}
W20260709 12:12:30.888792 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200809us
W20260709 12:12:30.894941 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60418.8us
I20260709 12:12:30.865761 15283 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:30.909271 12488 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.857723 15278 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:30.906672 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:30.922534 12643 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:12:30.882719 14785 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:30.935909 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.938544 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.939688 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:30.954514 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.954900 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.955412 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:30.955730 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:30.955962 12492 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:30.949285 12495 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:30.907697 15340 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:30.964995 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.965997 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
W20260709 12:12:30.936688 15377 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:12:30.976842 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:30.969761 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 312) took 1157 ms. Trace:
W20260709 12:12:30.984536 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 157222us
I20260709 12:12:30.985442 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:30.968281 12488 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:30.924136 15167 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:30.935112 15079 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:30.986404 12492 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:30.973948 12645 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:30.992040 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:30.991546 15006 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:30.981925 12494 rpcz_store.cc:276] 0709 12:12:29.812784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.812931 (+   147us) service_pool.cc:224] Handling call
0709 12:12:29.883258 (+ 70327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:29.883652 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:29.884161 (+   509us) log.cc:844] Serialized 14 byte log entry
0709 12:12:29.889518 (+  5357us) log.cc:844] Serialized 14 byte log entry
0709 12:12:29.889665 (+   147us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:29.932432 (+ 42767us) log.cc:844] Serialized 26 byte log entry
0709 12:12:29.932624 (+   192us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:30.235339 (+302715us) log.cc:844] Serialized 14 byte log entry
0709 12:12:30.236722 (+  1383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:30.236821 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:30.840134 (+603313us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:30.969386 (+129252us) raft_consensus.cc:1685] finished
0709 12:12:30.969420 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:30.969668 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40995,"prepare.queue_time_us":92772,"prepare.run_cpu_time_us":173,"prepare.run_wall_time_us":2555,"raft.queue_time_us":15582,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":4819,"spinlock_wait_cycles":15672704,"thread_start_us":245716,"threads_started":2,"wal-append.queue_time_us":1756}
I20260709 12:12:30.910127 15378 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 12:12:30.906145 15346 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:30.901809 15222 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.013494 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:31.014751 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:31.016218 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:31.019784 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:31.024324 12488 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:30.982898 15165 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:31.014209 15327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.015s	user 0.009s	sys 0.025s
I20260709 12:12:30.981663 15136 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:30.944999 15278 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:30.937314 15354 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
W20260709 12:12:31.029887 15190 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.504s	user 0.014s	sys 0.001s
I20260709 12:12:31.045204 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:31.045281 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:30.922453 15343 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.057543 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:31.094465 12494 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:31.095726 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43540.6us
W20260709 12:12:31.032285 15333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.121s	user 0.036s	sys 0.041s
I20260709 12:12:31.101176 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:31.102541 12645 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:30.964789 15375 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260709 12:12:31.031055 15010 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:12:31.043746 15319 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.653s	user 0.014s	sys 0.003s
I20260709 12:12:31.031981 15351 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:31.092526 15340 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:31.011801 15167 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.128551 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:31.129680 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:31.139977 12646 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:31.118467 15147 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.729s	user 0.013s	sys 0.009s
W20260709 12:12:31.140136 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124025us
I20260709 12:12:31.130306 12492 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:12:31.134663 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:31.041440 15006 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:31.136327 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:31.117957 15333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.207s	user 0.036s	sys 0.046s
W20260709 12:12:31.055469 15327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.010s	sys 0.026s
I20260709 12:12:31.141443 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:31.143958 12642 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:31.161890 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:31.162963 12642 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:31.163985 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:31.164044 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:31.164731 12642 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:31.145100 12645 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:31.165503 15361 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:31.166118 15361 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.166582 15361 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:31.169903 15085 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:31.124750 15136 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.175392 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:31.035717 15368 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260709 12:12:31.183145 12490 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:12:31.159142 12492 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20260709 12:12:31.143467 15340 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.127820 15283 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:31.191210 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:31.248322 12645 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:31.141013 15190 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:31.177047 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:31.175870 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 316) took 1287 ms. Trace:
I20260709 12:12:31.267463 12567 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:12:31.270941 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79825.5us
I20260709 12:12:31.272746 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:31.265537 12492 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:31.278193 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:31.044888 15165 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:31.289844 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:31.265182 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120842us
I20260709 12:12:31.282755 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1789) took 1496 ms. Trace:
I20260709 12:12:31.295187 12492 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:31.297472 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 162942us
I20260709 12:12:31.298395 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:31.303962 12490 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:31.293896 12567 rpcz_store.cc:276] 0709 12:12:29.786554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.786723 (+   169us) service_pool.cc:224] Handling call
0709 12:12:30.937072 (+1150349us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:31.282663 (+345591us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11409664}
W20260709 12:12:31.306658 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173462us
I20260709 12:12:31.179936 15136 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:31.309896 12488 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:31.311798 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1791) took 1524 ms. Trace:
I20260709 12:12:31.250314 12491 rpcz_store.cc:276] 0709 12:12:29.888628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.888859 (+   231us) service_pool.cc:224] Handling call
0709 12:12:29.922055 (+ 33196us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:29.936229 (+ 14174us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:12:30.306515 (+370286us) log.cc:844] Serialized 14 byte log entry
0709 12:12:30.790502 (+483987us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:30.821879 (+ 31377us) log.cc:844] Serialized 52 byte log entry
0709 12:12:30.822085 (+   206us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:30.822121 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:30.822195 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:31.175597 (+353402us) raft_consensus.cc:1685] finished
0709 12:12:31.175618 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:31.175800 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":493644,"prepare.queue_time_us":127567,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":47,"raft.queue_time_us":424632,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":7223,"thread_start_us":352109,"threads_started":2,"wal-append.queue_time_us":102418}
I20260709 12:12:31.152184 15325 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.129s
I20260709 12:12:31.246594 15085 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:31.264886 15340 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:31.117272 15169 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.137820 15319 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.153066 15147 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.339977 12492 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
W20260709 12:12:31.312923 15349 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.570s	user 0.013s	sys 0.000s
I20260709 12:12:31.289147 15358 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:31.312417 12569 rpcz_store.cc:276] 0709 12:12:29.787530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.787709 (+   179us) service_pool.cc:224] Handling call
0709 12:12:29.788038 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:29.788402 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:29.788433 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:29.791522 (+  3089us) log.cc:844] Serialized 26 byte log entry
0709 12:12:30.846946 (+1055424us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b24001522b8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:12:31.266683 (+419737us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b1800341d40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1
0709 12:12:31.266903 (+   220us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:31.311026 (+ 44123us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.311359 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.311465 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:31.311528 (+    63us) raft_consensus.cc:1685] finished
0709 12:12:31.311547 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:31.311715 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6312,"prepare.queue_time_us":60454,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"raft.queue_time_us":128462,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":12235,"spinlock_wait_cycles":68245120,"thread_start_us":90782,"threads_started":1,"wal-append.queue_time_us":172630}
I20260709 12:12:31.249536 15375 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
W20260709 12:12:31.375967 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59531.3us
I20260709 12:12:31.308205 15367 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:31.370417 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 382984us
I20260709 12:12:31.402864 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:31.424494 12646 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:31.357913 15085 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:31.180522 15079 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.424508 12536 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:31.426548 12536 leader_election.cc:403] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260709 12:12:31.395067 15350 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 12:12:31.418385 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90368.4us
W20260709 12:12:31.419691 15360 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.041s	user 0.005s	sys 0.009s
I20260709 12:12:31.420158 13259 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:31.421170 15319 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:31.402951 15358 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:31.422075 15391 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.447625 15358 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:31.447783 15391 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.422636 15345 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:12:31.434376 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161697us
I20260709 12:12:31.468802 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:31.474468 15378 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:31.474687 15404 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:12:31.475297 15378 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.441187 15085 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:31.440274 15354 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:31.445353 15337 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:31.497241 12461 connection.cc:256] RPC call timeout handler was delayed by -1.2751761549998264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
I20260709 12:12:31.403618 15349 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.467999 15190 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:31.477528 15378 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:31.561424 15398 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:12:31.425043 15006 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.515367 15403 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20260709 12:12:31.565526 12642 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:31.566957 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:31.477599 15391 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:31.624369 12490 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:31.453263 15367 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.474562 15397 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:31.615151 15278 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:31.645426 12640 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:12:31.558825 15337 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:31.549842 15346 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:31.567466 15277 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:31.568861 15368 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:12:31.657879 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 359560us
I20260709 12:12:31.657953 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:31.657267 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1795) took 1367 ms (client timeout 1774 ms). Trace:
I20260709 12:12:31.659505 12642 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7. Re-sending same reply.
I20260709 12:12:31.652568 15165 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:31.659971 12565 rpcz_store.cc:269] 0709 12:12:30.290038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.290199 (+   161us) service_pool.cc:224] Handling call
0709 12:12:31.657040 (+1366841us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:31.657193 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":650834688}
I20260709 12:12:31.548066 15190 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.485569 15283 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.687981 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:31.509541 15354 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:31.699048 15074 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.971s	user 0.014s	sys 0.008s
I20260709 12:12:31.574479 15402 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.094s
I20260709 12:12:31.654421 15277 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:31.661516 15346 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:31.711835 15277 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.713161 15277 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:31.704147 14876 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.310s	user 0.016s	sys 0.008s
I20260709 12:12:31.688048 15167 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:31.739147 15400 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:31.739733 15400 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:31.743673 15400 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.745044 15400 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:31.723351 14876 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.747655 12489 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:31.749845 14876 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:31.751240 15190 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:31.758344 15190 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:31.758890 15190 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:31.751638 15408 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.760769 15408 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.763576 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:31.654716 15361 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:31.709998 15074 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:31.786506 15074 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:31.788108 12642 pending_rounds.cc:85] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 4
I20260709 12:12:31.788508 12642 pending_rounds.cc:107] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260709 12:12:31.788569 15361 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:31.788900 12642 raft_consensus.cc:2889] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:31.713928 15346 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:31.809026 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 291745us
W20260709 12:12:31.812691 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:31.876447 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1786) took 2093 ms. Trace:
I20260709 12:12:31.808518 15221 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:31.878908 12566 rpcz_store.cc:276] 0709 12:12:29.783329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.783498 (+   169us) service_pool.cc:224] Handling call
0709 12:12:29.882033 (+ 98535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:30.739162 (+857129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.876173 (+1137011us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:31.876377 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1970130944}
W20260709 12:12:31.883494 15347 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.001s	user 0.022s	sys 0.038s
W20260709 12:12:31.884163 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1666) took 2044 ms (client timeout 2298 ms). Trace:
I20260709 12:12:31.814152 15074 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:31.824092 15391 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:31.890275 12637 rpcz_store.cc:269] 0709 12:12:29.837508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.849857 (+ 12349us) service_pool.cc:224] Handling call
0709 12:12:31.875668 (+2025811us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:31.875843 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":835550464}
I20260709 12:12:31.861593 15402 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:31.888409 15364 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.873327 15411 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:31.885982 15221 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:31.874756 15010 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.029s	sys 0.019s
I20260709 12:12:31.891867 15420 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:31.931965 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:31.963660 12494 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.868s	user 0.008s	sys 0.018s
I20260709 12:12:31.964372 12494 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:31.922508 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:31.965606 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2070) took 1079 ms. Trace:
I20260709 12:12:31.966684 12494 rpcz_store.cc:276] 0709 12:12:30.886250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.011538 (+125288us) service_pool.cc:224] Handling call
0709 12:12:31.965517 (+953979us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:32.034953 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:31.892705 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 281) took 1016 ms. Trace:
I20260709 12:12:31.974972 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:32.047041 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75637.1us
W20260709 12:12:32.044760 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116557us
I20260709 12:12:32.045516 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.047098 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:31.935530 15416 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:12:32.046737 12646 rpcz_store.cc:276] 0709 12:12:30.876281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.444725 (+568444us) service_pool.cc:224] Handling call
0709 12:12:31.445236 (+   511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:31.445535 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:31.445573 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:31.466977 (+ 21404us) log.cc:844] Serialized 26 byte log entry
0709 12:12:31.860157 (+393180us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:31.891677 (+ 31520us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.892236 (+   559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.892360 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:31.892421 (+    61us) raft_consensus.cc:1685] finished
0709 12:12:31.892440 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:31.892623 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":44883,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":71,"raft.queue_time_us":14130,"spinlock_wait_cycles":8179584,"thread_start_us":344331,"threads_started":1,"wal-append.queue_time_us":349627}
I20260709 12:12:32.050426 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:31.953954 14785 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.004s	user 0.010s	sys 0.005s
I20260709 12:12:32.060932 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:31.964788 15010 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.029s	sys 0.020s
I20260709 12:12:32.068907 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:31.950261 15349 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:31.959729 15354 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.070538 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.072681 12643 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:31.911561 15074 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.076478 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:32.079954 12494 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:31.918109 15412 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:31.920786 15391 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.061543 15190 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.066782 15010 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:32.088630 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.101542 12565 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:32.073807 15167 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:31.917865 15397 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.161s
I20260709 12:12:32.095741 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:32.102589 15366 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.373s	user 0.020s	sys 0.002s
W20260709 12:12:32.109506 15358 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.661s	user 0.010s	sys 0.005s
W20260709 12:12:32.114033 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 191751us
I20260709 12:12:32.088550 15349 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:32.115044 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:31.925338 15402 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.116389 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:32.107764 15102 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.351s	user 0.014s	sys 0.006s
I20260709 12:12:31.929044 15364 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.119570 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:32.121248 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.122736 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.123148 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:32.103112 15346 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:32.126868 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:32.127233 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.127523 12536 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:32.127915 12494 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:31.938060 15411 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.129221 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:32.130384 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.132001 12567 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:32.051608 15420 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.116001 15167 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:32.125928 15357 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.241s	user 0.010s	sys 0.004s
I20260709 12:12:32.102475 15412 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.145545 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.145881 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:32.128966 15397 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:32.133428 15346 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:32.181213 15397 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:32.144649 15349 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:32.156077 15409 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.081s
I20260709 12:12:32.157059 15415 raft_consensus.cc:2764] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:12:32.185113 15411 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:32.189131 15420 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:32.159399 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2075) took 1133 ms. Trace:
I20260709 12:12:32.126287 15102 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:32.161033 15370 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.807s	user 0.014s	sys 0.008s
I20260709 12:12:32.211592 12571 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:32.163357 15402 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:32.164336 15389 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.706s	user 0.001s	sys 0.000s
I20260709 12:12:32.165035 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:32.165972 15385 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.706s	user 0.002s	sys 0.000s
W20260709 12:12:32.167546 15384 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.867s	user 0.002s	sys 0.000s
W20260709 12:12:32.169185 15386 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.871s	user 0.000s	sys 0.002s
W20260709 12:12:32.170765 15380 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.152s	user 0.000s	sys 0.003s
I20260709 12:12:32.115240 15358 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.130048 15425 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.098s
I20260709 12:12:32.174328 15167 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.113077 15366 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.214610 15415 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:32.183147 15074 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.189908 15349 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:32.156684 15010 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.191454 15364 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:32.181825 15397 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:32.208237 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.277750 12565 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:32.278038 12488 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:32.202229 12492 rpcz_store.cc:276] 0709 12:12:31.026184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.404512 (+378328us) service_pool.cc:224] Handling call
0709 12:12:31.405198 (+   686us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:31.549445 (+144247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:31.549475 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:31.550040 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:12:31.912797 (+362757us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:31.912833 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.912905 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:32.159070 (+246165us) raft_consensus.cc:1685] finished
0709 12:12:32.159099 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:32.159313 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19292,"raft.queue_time_us":30037,"spinlock_wait_cycles":13808000,"thread_start_us":324412,"threads_started":2,"wal-append.queue_time_us":418659}
I20260709 12:12:32.221655 12571 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 7. Ops: []
I20260709 12:12:32.248497 15221 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:32.283376 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 165706us
I20260709 12:12:32.284688 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:32.214394 15346 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.232857 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.233417 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.234210 15366 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.234627 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.290540 12572 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:32.236840 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.237504 12569 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:32.239149 15102 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:32.242074 15277 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.004s
W20260709 12:12:32.242774 15337 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.588s	user 0.013s	sys 0.006s
I20260709 12:12:32.337878 15337 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:32.339195 12564 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:32.339609 15349 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.244746 15358 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.340310 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:32.245359 15388 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.911s	user 0.000s	sys 0.003s
W20260709 12:12:32.252291 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:32.273025 15415 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:32.342000 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:32.342316 12494 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:32.272228 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.344017 15415 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:32.344611 12646 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:12:32.346058 15222 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:32.332962 15389 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:12:32.333832 15209 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:32.285830 15333 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:32.348147 15333 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:32.334296 15074 consensus_queue.cc:1175] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 7 status { last_received { term: 7 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 7 index: 5 } }
I20260709 12:12:32.333705 15346 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.357837 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.358721 12567 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:32.270288 15136 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.907s	user 0.014s	sys 0.011s
I20260709 12:12:32.283181 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.152452 15165 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 12:12:32.286652 15318 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.286564 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.415539 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:32.416006 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:32.350620 15333 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:32.290076 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:32.302755 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:32.257264 15412 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.287341 15364 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:12:32.336081 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.424394 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.336562 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.346837 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.426044 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.426752 12572 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:32.427134 12568 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:32.427486 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:32.417936 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.436791 12492 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:32.431841 15136 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.430716 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.486954 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:32.477725 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:32.478209 12564 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:32.506232 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.541098 12492 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:32.541786 12565 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:12:32.538897 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:32.478703 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144195us
I20260709 12:12:32.514580 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.543324 12488 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:32.545104 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:32.479245 15410 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:32.549248 12571 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 12:12:32.417325 15393 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.995s	user 0.003s	sys 0.000s
I20260709 12:12:32.550154 12645 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:32.550740 12572 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 7. Ops: []
I20260709 12:12:32.551652 12571 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:32.482161 15085 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.953s	user 0.022s	sys 0.000s
I20260709 12:12:32.555790 12568 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:32.562914 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:32.526901 15412 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.565461 12565 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:12:32.566044 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.566701 12488 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:32.568802 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:32.571169 12571 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:32.572535 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.573761 12492 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:32.504709 15402 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:12:32.582424 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2076) took 1125 ms. Trace:
I20260709 12:12:32.586036 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 326) took 1122 ms. Trace:
I20260709 12:12:32.586081 12492 rpcz_store.cc:276] 0709 12:12:31.457398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.553652 (+1096254us) service_pool.cc:224] Handling call
0709 12:12:32.569482 (+ 15830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:32.574247 (+  4765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:32.582368 (+  8121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:32.586653 12488 rpcz_store.cc:276] 0709 12:12:31.463678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.567411 (+1103733us) service_pool.cc:224] Handling call
0709 12:12:32.585285 (+ 17874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:32.585598 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:32.585684 (+    86us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:32.585748 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:32.585804 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:32.585868 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:32.585978 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:32.575419 15340 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.181s	user 0.022s	sys 0.006s
I20260709 12:12:32.591593 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:32.571557 15165 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.594973 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:32.574484 15412 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.571776 15079 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:12:32.572309 15325 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.126s	user 0.026s	sys 0.032s
I20260709 12:12:32.598008 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:32.575111 15346 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.588828 15410 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:32.605564 12488 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:32.568401 15358 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 12:12:32.600078 15165 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.479836 15318 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.629163 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:32.569869 15085 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.571915 15337 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.026s
W20260709 12:12:32.597874 15400 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.852s	user 0.016s	sys 0.005s
W20260709 12:12:32.602514 15325 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.188s	user 0.026s	sys 0.032s
I20260709 12:12:32.599680 15392 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.578938 15409 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:12:32.575067 15010 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
W20260709 12:12:32.631987 12611 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items
I20260709 12:12:32.678318 15409 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:32.678488 15010 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:32.678920 15409 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:32.679188 15010 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:32.680311 15409 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.660538 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:32.614959 15412 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:32.673190 12488 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:32.680410 15010 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.667747 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:32.757274 12640 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:12:32.597564 15340 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.644323 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:32.761675 13257 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:32.751586 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 149624us
I20260709 12:12:32.634524 15136 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:32.682488 15433 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:12:32.758031 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:32.661572 15346 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.752327 12491 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [2.2-7.3]
I20260709 12:12:32.762778 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:32.759238 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1659) took 3259 ms (client timeout 2092 ms). Trace:
W20260709 12:12:32.770092 12644 rpcz_store.cc:269] 0709 12:12:29.499785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:29.499946 (+   161us) service_pool.cc:224] Handling call
0709 12:12:32.756591 (+3256645us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:32.759082 (+  2491us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1982515584}
W20260709 12:12:32.767812 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 157288us
I20260709 12:12:32.634891 12610 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35404 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1751)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9)
I20260709 12:12:32.759951 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:32.784650 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:32.792336 12488 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:32.793401 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:32.802742 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:32.811987 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.766145 15333 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.768795 15364 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.818346 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.822513 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:32.817548 15400 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:32.780947 15409 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:32.686453 15392 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.840229 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:32.827581 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.792176 15325 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:32.842535 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.846825 15415 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:32.847337 15415 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:32.852522 13260 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:12:32.853205 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 217116us
I20260709 12:12:32.818025 15410 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:32.855834 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.853694 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 280) took 1984 ms. Trace:
I20260709 12:12:32.781860 15346 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.858038 12639 rpcz_store.cc:276] 0709 12:12:30.859691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.859832 (+   141us) service_pool.cc:224] Handling call
0709 12:12:30.885555 (+ 25723us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:30.885848 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:30.885875 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:30.903512 (+ 17637us) log.cc:844] Serialized 26 byte log entry
0709 12:12:31.185006 (+281494us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:31.327960 (+142954us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.469921 (+141961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.470020 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:32.352880 (+882860us) spinlock_profiling.cc:243] Waited 85.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:32.839649 (+486769us) raft_consensus.cc:1685] finished
0709 12:12:32.844043 (+  4394us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:32.844257 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":112662,"prepare.queue_time_us":166950,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":146011,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":197103872,"thread_start_us":309978,"threads_started":2,"wal-append.queue_time_us":476317}
I20260709 12:12:32.861101 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:32.872925 15337 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.847772 15277 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:32.852206 15415 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:32.879868 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 321) took 2065 ms. Trace:
I20260709 12:12:32.856189 15400 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:32.831841 15364 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.825992 15333 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.880472 12493 rpcz_store.cc:276] 0709 12:12:30.814340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.814481 (+   141us) service_pool.cc:224] Handling call
0709 12:12:30.858809 (+ 44328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:30.859077 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:30.859107 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:32.139600 (+1280493us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:12:32.155932 (+ 16332us) log.cc:844] Serialized 26 byte log entry
0709 12:12:32.308971 (+153039us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:32.309040 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:32.310215 (+  1175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:32.870219 (+560004us) raft_consensus.cc:1685] finished
0709 12:12:32.870255 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:32.879778 (+  9523us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25411968,"thread_start_us":152531,"threads_started":1,"wal-append.queue_time_us":360113}
I20260709 12:12:32.877355 15337 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:32.886955 12567 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.001s
I20260709 12:12:32.891567 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:32.904894 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:32.870716 15325 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:32.868954 15010 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:32.898124 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117148us
I20260709 12:12:32.894888 15410 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:12:32.893940 15397 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.004s
W20260709 12:12:32.906152 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119186us
I20260709 12:12:32.880071 15277 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:32.911640 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:32.903393 15436 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:12:32.924136 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:32.927057 12639 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:32.930190 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1750) took 1066 ms. Trace:
I20260709 12:12:32.930759 12639 rpcz_store.cc:276] 0709 12:12:31.863707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.905684 (+1041977us) service_pool.cc:224] Handling call
0709 12:12:32.930108 (+ 24424us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:32.932281 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:32.933236 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1751) took 1069 ms. Trace:
I20260709 12:12:32.934473 12639 rpcz_store.cc:276] 0709 12:12:31.864027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.931991 (+1067964us) service_pool.cc:224] Handling call
0709 12:12:32.933165 (+  1174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:32.935187 15222 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:32.930938 15400 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:32.938457 15410 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:32.938951 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:32.938484 12493 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:32.907331 15165 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.959410 12493 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:32.929607 15277 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:32.937609 15397 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.938113 15434 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:32.949878 15340 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:32.973140 12493 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.936486 15318 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:32.966432 15352 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.607s	user 0.041s	sys 0.044s
I20260709 12:12:32.979849 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:32.955093 15439 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:12:32.983435 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:32.939244 15325 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260709 12:12:32.984879 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1693) took 1032 ms. Trace:
I20260709 12:12:32.985371 12493 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:32.942312 15222 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.985616 12639 rpcz_store.cc:276] 0709 12:12:31.952252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.935496 (+983244us) service_pool.cc:224] Handling call
0709 12:12:32.984756 (+ 49260us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:32.982887 15352 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.627s	user 0.041s	sys 0.044s
I20260709 12:12:32.970172 15400 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:12:32.994781 15392 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.995393 15403 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:32.995918 15403 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:32.994868 15410 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:32.996744 15403 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:32.982724 15085 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:32.677958 12611 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1750)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1751)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9)
I20260709 12:12:33.014652 15346 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.015338 15391 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.016489 15364 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.000567 15403 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.003109 12572 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:12:32.992180 15402 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 12:12:33.004621 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 520) took 2653 ms. Trace:
I20260709 12:12:33.006762 15337 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.020895 13259 rpcz_store.cc:276] 0709 12:12:30.350743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.351315 (+   572us) service_pool.cc:224] Handling call
0709 12:12:30.351689 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:31.446789 (+1095100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.004246 (+1557457us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:33.004517 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2456771328}
I20260709 12:12:33.007323 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.008230 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:33.024427 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.025256 12496 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.026170 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.027370 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.028613 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:33.031121 12644 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:12:33.022207 12611 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1706)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1707)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9)
I20260709 12:12:32.999298 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:33.040771 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117263us
I20260709 12:12:33.042219 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:33.087507 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45355.5us
I20260709 12:12:33.047195 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:33.088681 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.088568 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:33.043030 12611 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1706)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1707)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9)
I20260709 12:12:33.088927 12490 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:33.089454 15439 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:33.089962 15439 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:33.090392 15439 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:33.090560 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:33.053682 15383 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.950s	user 0.004s	sys 0.018s
I20260709 12:12:33.045590 15408 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.052073 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1739) took 2160 ms. Trace:
W20260709 12:12:33.043247 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1682) took 1201 ms (client timeout 1578 ms). Trace:
I20260709 12:12:33.086525 15277 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.043211 15147 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.055152 12496 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 12:12:33.064671 12461 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.575s (SENT)
I20260709 12:12:33.064901 15400 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.099419 15439 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:33.100780 15395 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.773s	user 0.003s	sys 0.010s
W20260709 12:12:33.093854 12644 rpcz_store.cc:269] 0709 12:12:31.841559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.006966 (+1165407us) service_pool.cc:224] Handling call
0709 12:12:33.043153 (+ 36187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.102221 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2061) took 2346 ms. Trace:
I20260709 12:12:33.095454 15400 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:33.103215 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1680) took 1262 ms (client timeout 1575 ms). Trace:
W20260709 12:12:33.097023 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.103638 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 278) took 2244 ms. Trace:
I20260709 12:12:33.104521 12641 rpcz_store.cc:276] 0709 12:12:30.858837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.859047 (+   210us) service_pool.cc:224] Handling call
0709 12:12:30.885345 (+ 26298us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:30.948773 (+ 63428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:30.948806 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:30.964732 (+ 15926us) log.cc:844] Serialized 52 byte log entry
0709 12:12:31.159753 (+195021us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:31.297894 (+138141us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.332790 (+ 34896us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.657210 (+324420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.657313 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:32.179883 (+522570us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b18004337e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:33.100031 (+920148us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:33.103371 (+  3340us) raft_consensus.cc:1685] finished
0709 12:12:33.103397 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:33.103556 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":138579,"prepare.queue_time_us":295281,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":114,"raft.queue_time_us":332598,"raft.run_cpu_time_us":520,"raft.run_wall_time_us":35031,"spinlock_wait_cycles":426400384,"thread_start_us":460069,"threads_started":2,"wal-append.queue_time_us":286105}
I20260709 12:12:33.098474 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.099303 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:33.103647 12646 rpcz_store.cc:269] 0709 12:12:31.840955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.082884 (+241929us) service_pool.cc:224] Handling call
0709 12:12:33.102954 (+1020070us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.103115 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":544304896}
W20260709 12:12:33.110635 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.111009 15209 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:33.093946 15396 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.671s	user 0.021s	sys 0.016s
I20260709 12:12:33.102669 12489 rpcz_store.cc:276] 0709 12:12:30.755696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.755849 (+   153us) service_pool.cc:224] Handling call
0709 12:12:30.756172 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:31.748085 (+991913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.101859 (+1353774us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:33.102097 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2279690624}
W20260709 12:12:33.100353 15390 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.698s	user 0.015s	sys 0.023s
I20260709 12:12:33.125466 15400 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.090202 12611 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1706)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1707)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9)
W20260709 12:12:33.086380 15438 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.450s	user 0.001s	sys 0.000s
I20260709 12:12:33.128944 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1920) took 1082 ms. Trace:
I20260709 12:12:33.129514 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.129907 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.129391 12569 rpcz_store.cc:276] 0709 12:12:32.046630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.046785 (+   155us) service_pool.cc:224] Handling call
0709 12:12:33.116049 (+1069264us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.128874 (+ 12825us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":431154048}
I20260709 12:12:33.129498 15354 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:33.130985 13259 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:33.131479 15354 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.132059 15354 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.134054 15343 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.128161 12611 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1708)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1706)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1707)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9)
I20260709 12:12:33.117986 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.093715 12645 rpcz_store.cc:276] 0709 12:12:30.891245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.454763 (+563518us) service_pool.cc:224] Handling call
0709 12:12:31.471760 (+ 16997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:32.550785 (+1079025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.051776 (+500991us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:33.051991 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2479388928}
I20260709 12:12:33.091567 12490 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:33.120714 15221 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.137105 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.138536 12645 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:12:33.138818 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:33.139040 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.097474 15277 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:33.139184 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1752) took 1263 ms (client timeout 1615 ms). Trace:
W20260709 12:12:33.141053 12645 rpcz_store.cc:269] 0709 12:12:31.875528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.136650 (+1261122us) service_pool.cc:224] Handling call
0709 12:12:33.139100 (+  2450us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.142778 15354 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.124856 15209 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:33.143504 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55095.8us
I20260709 12:12:33.122153 12497 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:33.098047 15391 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.096557 15366 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:12:33.123148 15167 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
W20260709 12:12:33.187613 15407 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.563s	user 0.001s	sys 0.011s
I20260709 12:12:33.158260 15397 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:33.095826 12461 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.578s (SENT)
I20260709 12:12:33.189935 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:33.095108 15434 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:33.190478 15333 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.172914 15343 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:12:33.191972 15434 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:33.123839 15432 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.575s	user 0.001s	sys 0.000s
I20260709 12:12:33.124509 12493 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:33.143433 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.146962 12564 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:33.197273 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.147584 15363 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:33.198207 15363 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:33.198014 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.199641 15363 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:33.201431 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105624us
I20260709 12:12:33.148888 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.204947 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:33.205070 15409 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:33.206123 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.206557 15402 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.207096 12490 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:12:33.149184 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1742) took 2170 ms. Trace:
I20260709 12:12:33.207166 15402 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.207975 12640 rpcz_store.cc:276] 0709 12:12:30.979033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.646466 (+667433us) service_pool.cc:224] Handling call
0709 12:12:32.756905 (+1110439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:32.758118 (+  1213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.148948 (+390830us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54001f7cd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.149117 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2552786176}
I20260709 12:12:33.209342 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.180009 15400 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.210122 12640 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:33.210614 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1753) took 1334 ms. Trace:
I20260709 12:12:33.210474 15400 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.211014 12640 rpcz_store.cc:276] 0709 12:12:31.875851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.209058 (+1333207us) service_pool.cc:224] Handling call
0709 12:12:33.210551 (+  1493us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.211370 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.134114 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 277) took 2281 ms. Trace:
I20260709 12:12:33.151036 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.152334 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.213215 12563 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:33.153095 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.214355 12571 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:33.198033 12564 raft_consensus.cc:2376] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8. Re-sending same reply.
I20260709 12:12:33.154111 15364 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:33.155180 13256 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:33.145747 15209 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.158974 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.158241 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:33.160153 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.121109 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.163686 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:33.221557 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 57986.9us
I20260709 12:12:33.170647 15453 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.172349 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1743) took 1243 ms. Trace:
I20260709 12:12:33.223083 12494 rpcz_store.cc:276] 0709 12:12:31.929453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.129432 (+199979us) service_pool.cc:224] Handling call
0709 12:12:33.172091 (+1042659us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.172268 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":450302720}
I20260709 12:12:33.173126 12492 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 7. Ops: [5.5-5.5]
I20260709 12:12:33.227058 12488 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:33.220535 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:33.228268 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:33.228994 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.173812 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.178968 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:33.181499 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:33.230129 13257 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8. Re-sending same reply.
I20260709 12:12:33.230432 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:33.075485 15410 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.185415 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.186280 15332 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.192836 15343 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:33.204180 12461 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.673s (SENT)
I20260709 12:12:33.155239 15346 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:33.243263 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:33.238445 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88455.2us
I20260709 12:12:33.201315 15363 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:33.251978 12495 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.214383 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.252878 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.180925 15366 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:33.217993 12645 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:33.220264 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:33.238072 15397 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:33.212432 12638 rpcz_store.cc:276] 0709 12:12:30.853258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.853856 (+   598us) service_pool.cc:224] Handling call
0709 12:12:30.854371 (+   515us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:30.872011 (+ 17640us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:12:31.006889 (+134878us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.187626 (+180737us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:31.335425 (+147799us) log.cc:844] Serialized 52 byte log entry
0709 12:12:31.335679 (+   254us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:31.368257 (+ 32578us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.534070 (+165813us) log.cc:844] Serialized 14 byte log entry
0709 12:12:31.568634 (+ 34564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:31.568723 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:33.093282 (+1524559us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:33.133872 (+ 40590us) raft_consensus.cc:1685] finished
0709 12:12:33.133895 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:33.134049 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115796,"prepare.queue_time_us":279949,"prepare.run_cpu_time_us":178,"prepare.run_wall_time_us":4648,"raft.queue_time_us":392795,"raft.run_cpu_time_us":457,"raft.run_wall_time_us":457,"spinlock_wait_cycles":176685952,"thread_start_us":408424,"threads_started":3,"wal-append.queue_time_us":403216}
W20260709 12:12:33.264528 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1688) took 1414 ms (client timeout 1736 ms). Trace:
I20260709 12:12:33.264691 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:33.265043 12645 rpcz_store.cc:269] 0709 12:12:31.849738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.142227 (+1292489us) service_pool.cc:224] Handling call
0709 12:12:33.264462 (+122235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.249475 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.266265 12641 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7. Re-sending same reply.
I20260709 12:12:33.266443 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:33.266952 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1683) took 1419 ms (client timeout 1673 ms). Trace:
W20260709 12:12:33.267436 12641 rpcz_store.cc:269] 0709 12:12:31.847920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.227632 (+1379712us) service_pool.cc:224] Handling call
0709 12:12:33.266871 (+ 39239us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:33.267639 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1755) took 1391 ms (client timeout 1832 ms). Trace:
I20260709 12:12:33.248973 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:33.268419 12645 rpcz_store.cc:269] 0709 12:12:31.876354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.266066 (+1389712us) service_pool.cc:224] Handling call
0709 12:12:33.267563 (+  1497us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.269073 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:33.270423 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1691) took 1336 ms (client timeout 1749 ms). Trace:
W20260709 12:12:33.094005 15351 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.916s	user 0.020s	sys 0.029s
I20260709 12:12:33.215904 15402 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.245128 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:33.257488 13258 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:12:33.258848 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1754) took 1382 ms (client timeout 1801 ms). Trace:
W20260709 12:12:33.278574 12640 rpcz_store.cc:269] 0709 12:12:31.876119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.211758 (+1335639us) service_pool.cc:224] Handling call
0709 12:12:33.258782 (+ 47024us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.279912 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.212709 15400 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.067167 15325 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:33.212666 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1685) took 1364 ms (client timeout 1741 ms). Trace:
W20260709 12:12:33.288816 12646 rpcz_store.cc:269] 0709 12:12:31.848674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.210846 (+1362172us) service_pool.cc:224] Handling call
0709 12:12:33.212572 (+  1726us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.270640 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:54234 (request call id 318) took 2446 ms. Trace:
I20260709 12:12:33.248138 15343 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:33.270975 12638 rpcz_store.cc:269] 0709 12:12:31.933999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.264298 (+1330299us) service_pool.cc:224] Handling call
0709 12:12:33.270358 (+  6060us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.269620 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:33.275930 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45630.8us
I20260709 12:12:33.276176 13257 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:33.295665 12645 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:33.296520 12495 rpcz_store.cc:276] 0709 12:12:30.813230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.828411 (+ 15181us) service_pool.cc:224] Handling call
0709 12:12:33.251840 (+2423429us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b4c002c70c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3ad113f 00007ffbf3a952ab 00007ffbf3ac2f3c 00007ffbf3ac33fe 00007ffbf3d5088b 00007ffbf574f495 00007ffbf57a030f 00007ffbf57eaa55 00007ffbfa7bacd0
0709 12:12:33.252420 (+   580us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26353024}
I20260709 12:12:33.297080 15366 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.298334 15450 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:33.298781 15450 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:33.299615 15450 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.300186 12566 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:33.293920 15406 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.301069 15406 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.289467 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.218899 15391 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.270771 15209 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:33.301594 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.247308 13254 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:33.316866 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1687) took 1467 ms. Trace:
I20260709 12:12:33.317226 12646 rpcz_store.cc:276] 0709 12:12:31.849385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.289802 (+1440417us) service_pool.cc:224] Handling call
0709 12:12:33.316800 (+ 26998us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:33.305478 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.311061 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.308269 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.307274 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1735) took 2411 ms. Trace:
I20260709 12:12:33.314563 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.314987 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.323031 12565 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:33.263928 15397 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:33.302801 12641 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:33.274323 15351 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.099s	user 0.020s	sys 0.029s
I20260709 12:12:33.315598 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.336297 12564 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:33.308706 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1732) took 2435 ms (client timeout 1801 ms). Trace:
I20260709 12:12:33.325109 12571 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:33.325030 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.340399 12638 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.326480 12570 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:33.340893 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.341207 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1936) took 1097 ms. Trace:
I20260709 12:12:33.341562 12570 rpcz_store.cc:276] 0709 12:12:32.243472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.527615 (+284143us) service_pool.cc:224] Handling call
0709 12:12:33.314363 (+786748us) spinlock_profiling.cc:243] Waited 888 us on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:33.341151 (+ 26788us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2041728}
I20260709 12:12:33.320709 12491 rpcz_store.cc:276] 0709 12:12:30.895368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.339087 (+443719us) service_pool.cc:224] Handling call
0709 12:12:31.395142 (+ 56055us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:32.769807 (+1374665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.306934 (+537127us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:33.307195 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3120420096}
I20260709 12:12:33.328543 13261 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:12:33.332633 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:33.334020 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1756) took 1443 ms (client timeout 1898 ms). Trace:
I20260709 12:12:33.340920 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1684) took 1492 ms. Trace:
I20260709 12:12:33.242435 15410 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.345702 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:33.338244 12643 rpcz_store.cc:269] 0709 12:12:30.873462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.964960 (+ 91498us) service_pool.cc:224] Handling call
0709 12:12:33.308477 (+2343517us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.308630 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2444484480}
I20260709 12:12:33.347970 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.344058 12638 rpcz_store.cc:276] 0709 12:12:31.848329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.324671 (+1476342us) service_pool.cc:224] Handling call
0709 12:12:33.340847 (+ 16176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.351164 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.262686 15165 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 12:12:33.295743 15167 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:33.360985 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1759) took 1469 ms. Trace:
I20260709 12:12:33.361824 12638 rpcz_store.cc:276] 0709 12:12:31.891261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.350794 (+1459533us) service_pool.cc:224] Handling call
0709 12:12:33.360907 (+ 10113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.363432 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.303536 15406 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.309887 15209 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:33.318200 15400 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:33.308993 15221 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.059s
I20260709 12:12:33.359480 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.317906 15343 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.382025 12565 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7. Re-sending same reply.
I20260709 12:12:33.383239 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.383817 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:33.384505 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.385178 12565 raft_consensus.cc:2376] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6. Re-sending same reply.
I20260709 12:12:33.386260 15442 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:33.386698 15442 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:33.387028 15442 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.389962 15074 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:33.390467 15074 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:33.391417 15074 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.250269 15332 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.394469 15332 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.365687 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.396034 12643 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:33.396752 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1676) took 1557 ms. Trace:
I20260709 12:12:33.360178 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.353926 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.337997 15397 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:33.410758 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:33.410710 12638 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:12:33.411435 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1677) took 1571 ms (client timeout 1980 ms). Trace:
I20260709 12:12:33.373759 15222 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.373957 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.376945 15165 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:33.346094 15443 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:12:33.314651 15415 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.378358 15209 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.316648 15392 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:12:33.366664 15167 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:33.437016 12641 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:33.366430 15333 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:12:33.304529 15391 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.398240 12564 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:12:33.364843 15345 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:33.298327 15366 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:33.397135 12643 rpcz_store.cc:276] 0709 12:12:31.839578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.347471 (+1507893us) service_pool.cc:224] Handling call
0709 12:12:33.396669 (+ 49198us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:33.358074 12645 rpcz_store.cc:269] 0709 12:12:31.890597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.269323 (+1378726us) service_pool.cc:224] Handling call
0709 12:12:33.333938 (+ 64615us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.406293 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.498615 12563 raft_consensus.cc:2376] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
W20260709 12:12:33.411871 12640 rpcz_store.cc:269] 0709 12:12:31.839943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.279540 (+1439597us) service_pool.cc:224] Handling call
0709 12:12:33.411369 (+131829us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.336490 15351 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:33.414101 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1760) took 1522 ms. Trace:
I20260709 12:12:33.414595 12570 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:33.499891 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.415769 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.242810 15453 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:33.437736 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.510782 12565 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
I20260709 12:12:33.416779 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2077) took 1795 ms. Trace:
I20260709 12:12:33.512037 12492 rpcz_store.cc:276] 0709 12:12:31.621108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.587562 (+966454us) service_pool.cc:224] Handling call
0709 12:12:33.172670 (+585108us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:33.224167 (+ 51497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.416554 (+192387us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54000f78d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.416717 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320806400}
I20260709 12:12:33.411826 15346 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.416640 12639 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:33.440582 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 282) took 2414 ms. Trace:
I20260709 12:12:33.514019 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:33.423187 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.354820 12646 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:33.516242 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1758) took 1625 ms (client timeout 2000 ms). Trace:
W20260709 12:12:33.516608 12646 rpcz_store.cc:269] 0709 12:12:31.891095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.345228 (+1454133us) service_pool.cc:224] Handling call
0709 12:12:33.516181 (+170953us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:33.517570 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.517505 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.518323 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1697) took 1425 ms. Trace:
I20260709 12:12:33.518635 12646 rpcz_store.cc:276] 0709 12:12:32.093259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.517249 (+1423990us) service_pool.cc:224] Handling call
0709 12:12:33.518270 (+  1021us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.283334 15440 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:12:33.416194 15165 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:33.448015 15415 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.448452 15167 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.450110 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.450867 12567 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:33.413146 15445 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:12:33.410897 15439 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.318382 15325 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.410729 15409 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.443710 15450 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 7
W20260709 12:12:33.490801 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 231518us
I20260709 12:12:33.542029 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:33.497196 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1679) took 1656 ms (client timeout 1886 ms). Trace:
I20260709 12:12:33.499756 12638 rpcz_store.cc:276] 0709 12:12:31.891431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.363073 (+1471642us) service_pool.cc:224] Handling call
0709 12:12:33.413573 (+ 50500us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.501050 12571 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:33.521489 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:33.543236 12641 rpcz_store.cc:269] 0709 12:12:31.840622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.268459 (+1427837us) service_pool.cc:224] Handling call
0709 12:12:33.497128 (+228669us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.521674 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.543844 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:33.523329 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.544489 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:33.544615 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.513947 12642 rpcz_store.cc:276] 0709 12:12:31.026467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:31.719904 (+693437us) service_pool.cc:224] Handling call
0709 12:12:31.787845 (+ 67941us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:12:31.815566 (+ 27721us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:31.815614 (+    48us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:12:31.816541 (+   927us) log.cc:844] Serialized 78 byte log entry
0709 12:12:33.084347 (+1267806us) spinlock_profiling.cc:243] Waited 8.39 ms on lock 0x7b1800139d80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:12:33.270461 (+186114us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:33.270519 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:33.270598 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:33.395040 (+124442us) raft_consensus.cc:1685] finished
0709 12:12:33.395074 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:33.440474 (+ 45400us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8802,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":48476,"spinlock_wait_cycles":19303168,"thread_start_us":185883,"threads_started":1,"wal-append.queue_time_us":1516260}
I20260709 12:12:33.524456 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.531668 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.523591 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1692) took 1551 ms. Trace:
I20260709 12:12:33.546788 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.545616 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1767) took 1029 ms. Trace:
I20260709 12:12:33.548974 12641 rpcz_store.cc:276] 0709 12:12:32.516375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.544279 (+1027904us) service_pool.cc:224] Handling call
0709 12:12:33.545538 (+  1259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.547042 12645 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.548202 12640 rpcz_store.cc:276] 0709 12:12:31.949735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.499615 (+1549880us) service_pool.cc:224] Handling call
0709 12:12:33.500590 (+   975us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.550807 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.551491 12640 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5. Re-sending same reply.
I20260709 12:12:33.551520 12570 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:33.552800 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.497053 15345 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:33.554847 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.554298 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.555557 12565 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:33.549068 12643 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
W20260709 12:12:33.532441 15403 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.008s	sys 0.004s
W20260709 12:12:33.556748 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1761) took 1650 ms (client timeout 2157 ms). Trace:
I20260709 12:12:33.547677 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
W20260709 12:12:33.557286 12643 rpcz_store.cc:269] 0709 12:12:31.906152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.522239 (+1616087us) service_pool.cc:224] Handling call
0709 12:12:33.556668 (+ 34429us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.558542 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1772) took 1010 ms. Trace:
I20260709 12:12:33.547412 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:33.559127 12646 rpcz_store.cc:276] 0709 12:12:32.548167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.524205 (+976038us) service_pool.cc:224] Handling call
0709 12:12:33.558453 (+ 34248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.560717 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.561451 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:33.547402 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.562768 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1763) took 1084 ms. Trace:
I20260709 12:12:33.563478 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:33.563732 12646 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.568310 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260709 12:12:33.564010 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.515869 15455 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.570097 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1769) took 1022 ms. Trace:
I20260709 12:12:33.570606 12645 rpcz_store.cc:276] 0709 12:12:32.547292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.521391 (+974099us) service_pool.cc:224] Handling call
0709 12:12:33.570008 (+ 48617us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.572041 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.573354 12645 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.574116 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1764) took 1072 ms. Trace:
I20260709 12:12:33.574648 12645 rpcz_store.cc:276] 0709 12:12:32.501989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.571658 (+1069669us) service_pool.cc:224] Handling call
0709 12:12:33.574042 (+  2384us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.578253 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.490878 15332 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20260709 12:12:33.525813 15400 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:33.580381 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.582965 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.583837 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.572384 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.587356 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1757) took 1696 ms. Trace:
I20260709 12:12:33.587663 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1766) took 1073 ms. Trace:
I20260709 12:12:33.587896 12642 rpcz_store.cc:276] 0709 12:12:31.890878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.546697 (+1655819us) service_pool.cc:224] Handling call
0709 12:12:33.587261 (+ 40564us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29056}
I20260709 12:12:33.588392 12646 rpcz_store.cc:276] 0709 12:12:32.514528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.572058 (+1057530us) service_pool.cc:224] Handling call
0709 12:12:33.587591 (+ 15533us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.583891 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.589779 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.590564 12570 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:33.581555 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.593710 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.590288 12640 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.595449 12491 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:33.595903 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1701) took 1008 ms. Trace:
I20260709 12:12:33.596269 12640 rpcz_store.cc:276] 0709 12:12:32.588012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.578900 (+990888us) service_pool.cc:224] Handling call
0709 12:12:33.595824 (+ 16924us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.595271 12645 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:33.597201 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:33.598891 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.599207 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.599897 12495 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:33.594656 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.563232 12638 rpcz_store.cc:276] 0709 12:12:32.478729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.547183 (+1068454us) service_pool.cc:224] Handling call
0709 12:12:33.562693 (+ 15510us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.602555 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.603132 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:33.603166 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.603965 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.604535 12638 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:33.594450 12488 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:12:33.605561 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.606383 12638 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:33.597214 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.569522 15439 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:33.545488 15406 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:33.579638 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.610488 12489 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:33.599906 12564 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:33.530684 15346 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.555114 15409 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.611130 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.720186 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.642874 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.663661 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.721315 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:33.681571 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1919) took 1667 ms (client timeout 1717 ms). Trace:
W20260709 12:12:33.725003 12566 rpcz_store.cc:269] 0709 12:12:32.014592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.014828 (+   236us) service_pool.cc:224] Handling call
0709 12:12:33.681283 (+1666455us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:33.681465 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2843468672}
I20260709 12:12:33.675330 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.728768 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:33.653369 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:33.731104 12494 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:33.653859 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:33.732640 12570 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:33.578028 15415 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:33.687948 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1694) took 1735 ms (client timeout 1639 ms). Trace:
W20260709 12:12:33.758424 12644 rpcz_store.cc:269] 0709 12:12:31.952617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.687712 (+1735095us) service_pool.cc:211] Skipping call since client already timed out
0709 12:12:33.687862 (+   150us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:12:33.735752 12641 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:33.585186 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:33.737491 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.720232 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:33.748644 12565 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:33.723948 12569 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
W20260709 12:12:33.690402 15460 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 12:12:33.778051 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.779042 12571 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:33.779350 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:33.701839 15462 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:12:33.678783 15467 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:33.607838 15463 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:33.791353 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.757964 15415 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.663147 15332 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:12:33.785156 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1699) took 1236 ms. Trace:
I20260709 12:12:33.839257 12641 rpcz_store.cc:276] 0709 12:12:32.548451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.549909 (+1001458us) service_pool.cc:224] Handling call
0709 12:12:33.785071 (+235162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:33.784422 13256 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:33.722141 15325 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.722605 15439 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:33.687064 15457 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.783026 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:33.786407 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.850474 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:33.829883 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.763499 15410 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.582831 15455 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.835223 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.768566 15367 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:33.611124 15400 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.840626 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.786430 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:33.843381 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:33.846753 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 283451us
I20260709 12:12:33.847607 12570 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
W20260709 12:12:33.848285 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:33.848773 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 249679us
I20260709 12:12:33.839416 15415 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:33.867254 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.805281 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:33.794106 15463 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:33.852448 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:33.689970 15345 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.676991 15351 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.872406 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:33.652320 15165 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:33.861866 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:33.877765 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:33.877688 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:33.860445 12563 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:12:33.880319 12563 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:33.875429 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:33.864171 12571 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:33.883123 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:33.865574 12641 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:33.866098 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:33.850978 12638 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:33.839891 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:33.886407 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1768) took 1359 ms. Trace:
I20260709 12:12:33.886826 12638 rpcz_store.cc:276] 0709 12:12:32.526937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.786081 (+1259144us) service_pool.cc:224] Handling call
0709 12:12:33.886346 (+100265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9600}
I20260709 12:12:33.888499 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:33.888001 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.897221 12643 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:33.897459 12564 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:12:33.858918 15391 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:33.903676 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.904946 12642 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:33.873317 15147 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:33.872501 15403 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.878669 15434 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:33.906919 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:33.878314 15463 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.896117 15209 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:33.892684 15367 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:33.925516 12564 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:12:33.898655 15165 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.925994 12564 pending_rounds.cc:85] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 3
I20260709 12:12:33.926297 12564 pending_rounds.cc:107] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:12:33.926534 12564 raft_consensus.cc:2889] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:33.927940 12564 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:33.913045 15147 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:33.929062 15147 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.901284 15473 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:33.930927 15448 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:33.902621 15453 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:33.857098 15457 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:33.906778 15345 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:33.958917 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:33.884289 15410 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.048s
W20260709 12:12:33.987455 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 99080.6us
I20260709 12:12:33.988116 12641 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:33.990047 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:33.992254 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55375us
I20260709 12:12:34.001780 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:33.935528 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:34.003755 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:34.008662 12642 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:33.938651 15448 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:33.964351 15391 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:34.024648 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:34.029754 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 152167us
I20260709 12:12:33.963706 15475 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:33.922912 15403 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.056077 12641 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:33.917943 15439 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.010151 15367 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:34.016155 15477 raft_consensus.cc:476] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Not starting pre-election -- already a leader
I20260709 12:12:34.063531 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.026130 15391 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:34.066390 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:34.058832 15442 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.000s
I20260709 12:12:34.034076 15409 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:33.959195 15473 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.082244 15448 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.067571 12489 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:34.037735 15346 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.036121 15475 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:34.089959 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:34.101272 12641 raft_consensus.cc:2376] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6. Re-sending same reply.
I20260709 12:12:34.090054 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:34.092087 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.099395 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:34.104434 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:34.062948 15209 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:34.112694 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:34.121007 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119289us
W20260709 12:12:34.127606 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.088433 15391 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:34.098621 15477 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 12:12:34.098925 15442 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.148043 15457 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.148376 15221 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:34.148845 15455 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:34.149367 15480 consensus_queue.cc:1175] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:12:34.129416 15486 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:34.150664 12642 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:34.146816 15450 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.589s	user 0.008s	sys 0.005s
I20260709 12:12:34.068498 15483 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:34.120414 15346 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:34.182130 13257 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:34.179327 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:34.112295 15475 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:34.187116 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.168092 15410 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.161226 15487 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:34.194271 12646 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:12:34.201395 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:34.169483 15448 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:34.199673 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95330.5us
I20260709 12:12:34.203792 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:34.204830 12642 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:34.208205 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:34.210297 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:34.190618 15472 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.272s	user 0.002s	sys 0.000s
I20260709 12:12:34.176789 15486 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:34.211530 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:34.181988 15221 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.174093 15482 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:34.242451 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.262908 12642 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
W20260709 12:12:34.275854 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:34.282929 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.221138 15409 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.211148 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:34.215576 15477 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:34.286418 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:34.294586 12642 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:34.287058 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.297592 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 286) took 2661 ms. Trace:
I20260709 12:12:34.298038 12642 rpcz_store.cc:276] 0709 12:12:31.636517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.295613 (+2659096us) service_pool.cc:224] Handling call
0709 12:12:34.297094 (+  1481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.297282 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:34.297304 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:34.297346 (+    42us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:34.297374 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.297423 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:34.297537 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:34.225843 15400 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:34.274111 15482 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:34.248098 15486 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:34.301047 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:34.304862 12642 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
W20260709 12:12:34.308468 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 125099us
I20260709 12:12:34.207690 15410 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:34.291352 15167 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.748s	user 0.008s	sys 0.005s
I20260709 12:12:34.301848 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:34.293680 15364 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.067s	user 0.009s	sys 0.005s
I20260709 12:12:34.285132 15450 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.300217 15477 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:34.323253 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:34.274662 15209 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.214792 15367 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.317260 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1689) took 2467 ms. Trace:
I20260709 12:12:34.214133 15487 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:34.333280 15481 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:12:34.274111 15221 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.314446 15482 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:34.343147 12642 rpcz_store.cc:276] 0709 12:12:31.850090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.304306 (+2454216us) service_pool.cc:224] Handling call
0709 12:12:34.304690 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.316701 (+ 12011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.317181 (+   480us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:34.335214 15364 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:34.346578 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:34.302670 15409 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.334841 15499 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:34.357424 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1690) took 2507 ms. Trace:
I20260709 12:12:34.348919 15483 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:34.344344 15497 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:34.369997 15495 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.022s
W20260709 12:12:34.384177 15478 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.376s	user 0.002s	sys 0.000s
W20260709 12:12:34.388054 15493 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:12:34.393910 15473 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.331491 15167 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.374027 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:34.398514 12572 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:34.411723 12495 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:34.379967 15504 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:34.382148 15488 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:12:34.373467 15489 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:34.355592 15496 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:34.372762 15494 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.207s	user 0.001s	sys 0.001s
I20260709 12:12:34.396610 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:34.379549 15366 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.860s	user 0.011s	sys 0.002s
I20260709 12:12:34.340271 15450 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:34.433792 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 191521us
I20260709 12:12:34.372189 15499 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:34.438413 12494 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260709 12:12:34.443735 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:34.435601 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:34.447193 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50641.6us
I20260709 12:12:34.384207 15495 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.418800 15364 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:34.451509 12489 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:12:34.451865 12536 connection.cc:256] RPC call timeout handler was delayed by -1.161149665000039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1615s for a response
W20260709 12:12:34.454809 12536 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.544s (SENT)
I20260709 12:12:34.455499 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.452090 15499 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:34.444195 15496 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.464295 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 283) took 3429 ms. Trace:
W20260709 12:12:34.465971 15415 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.580s	user 0.004s	sys 0.009s
I20260709 12:12:34.470502 15415 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.441576 15497 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:34.473519 15497 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:34.463423 15487 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:34.478241 15147 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.549s	user 0.001s	sys 0.012s
I20260709 12:12:34.461468 15496 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.478909 15147 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.478263 15415 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.482756 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:34.483513 15397 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.045s	user 0.012s	sys 0.002s
I20260709 12:12:34.487063 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:34.461349 15165 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.002s
I20260709 12:12:34.489308 15403 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.470036 12645 rpcz_store.cc:276] 0709 12:12:31.035848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.879884 (+2844036us) service_pool.cc:224] Handling call
0709 12:12:33.880193 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:33.880584 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:33.880618 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:33.886377 (+  5759us) log.cc:844] Serialized 26 byte log entry
0709 12:12:34.014968 (+128591us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:34.202099 (+187131us) log.cc:844] Serialized 14 byte log entry
0709 12:12:34.203013 (+   914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.203142 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:34.463996 (+260854us) raft_consensus.cc:1685] finished
0709 12:12:34.464024 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:34.464218 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":561,"prepare.queue_time_us":301531,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3026,"raft.queue_time_us":5185,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":9263,"thread_start_us":237177,"threads_started":2,"wal-append.queue_time_us":223217}
I20260709 12:12:34.490401 15457 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:34.414382 15502 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20260709 12:12:34.433276 15483 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.437244 15442 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.493964 15409 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.494132 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1810) took 1340 ms. Trace:
I20260709 12:12:34.554500 12571 rpcz_store.cc:276] 0709 12:12:33.153268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.785032 (+631764us) service_pool.cc:224] Handling call
0709 12:12:33.863909 (+ 78877us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:33.882594 (+ 18685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.493882 (+611288us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54001b17d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:34.494059 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69986048}
I20260709 12:12:34.556572 12571 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:34.377987 15477 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:34.495992 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1705) took 1483 ms (client timeout 1600 ms). Trace:
W20260709 12:12:34.497190 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.463277 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:34.499712 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44944.4us
W20260709 12:12:34.565994 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0099496650000219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
I20260709 12:12:34.510182 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:34.496170 15410 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:34.489915 15400 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.531317 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:34.492573 15074 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:34.536216 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.536659 12564 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:34.538962 12492 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:34.513965 15364 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:34.510268 15167 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.542093 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:34.515007 15467 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.710s	user 0.005s	sys 0.010s
I20260709 12:12:34.511824 15221 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.545334 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 481) took 1245 ms. Trace:
I20260709 12:12:34.551219 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:34.564951 12643 rpcz_store.cc:269] 0709 12:12:33.012141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.558857 (+546716us) service_pool.cc:224] Handling call
0709 12:12:34.495774 (+936917us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:34.495913 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":707154048}
I20260709 12:12:34.539633 12495 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:34.457036 15495 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.522058 15397 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.511690 15366 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 12:12:34.732010 12566 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:12:34.731469 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.622581 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:34.733264 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 197193us
W20260709 12:12:34.725713 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:34.683769 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:34.695170 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 208223us
I20260709 12:12:34.634537 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1958) took 1626 ms. Trace:
I20260709 12:12:34.739059 12567 rpcz_store.cc:276] 0709 12:12:33.008584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.008803 (+   219us) service_pool.cc:224] Handling call
0709 12:12:33.009140 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:33.526338 (+517198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.634128 (+1107790us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54002cfbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:34.634428 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1015363968}
W20260709 12:12:34.737367 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.749974 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.753882 15469 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:12:34.719560 15363 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.459s	user 0.015s	sys 0.000s
I20260709 12:12:34.760270 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:34.581744 15463 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.620s	user 0.011s	sys 0.003s
W20260709 12:12:34.723876 15391 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.553s	user 0.011s	sys 0.000s
I20260709 12:12:34.793589 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:34.767853 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.693970 15450 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:34.572569 15345 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.000s
W20260709 12:12:34.573827 15501 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:12:34.796988 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
W20260709 12:12:34.820953 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:34.827910 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78020.6us
W20260709 12:12:34.782518 15445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.197s	user 0.022s	sys 0.066s
I20260709 12:12:34.794643 12565 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:34.853137 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:34.872438 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:34.830150 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
W20260709 12:12:34.690851 13256 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.836s	user 0.012s	sys 0.001s
I20260709 12:12:34.609023 15403 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:34.737771 15467 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:12:34.832000 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:34.735880 15437 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:34.636190 15502 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:34.840863 15480 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:34.839648 12536 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7.
I20260709 12:12:34.838757 12644 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:34.840541 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:34.827090 15167 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:34.898960 12563 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:34.875306 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:34.541698 15457 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.758432 15503 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:34.570271 15475 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.880636 12638 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:34.910761 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71310.6us
I20260709 12:12:34.911065 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1695) took 2958 ms. Trace:
I20260709 12:12:34.911429 12638 rpcz_store.cc:276] 0709 12:12:31.952955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.880023 (+2927068us) service_pool.cc:224] Handling call
0709 12:12:34.880418 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.910750 (+ 30332us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.911000 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:34.912175 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.912931 12638 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:12:34.752070 15499 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.541342 15455 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:34.831007 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 360) took 1154 ms. Trace:
I20260709 12:12:34.942497 15467 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:34.942900 13254 rpcz_store.cc:276] 0709 12:12:33.676515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.679498 (+  2983us) service_pool.cc:224] Handling call
0709 12:12:34.550987 (+871489us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:34.830922 (+279935us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5831552}
I20260709 12:12:34.943578 15495 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.891343 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:34.894737 15480 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:34.757156 15400 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.834548 15366 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:34.906812 12643 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:34.948124 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1696) took 2858 ms. Trace:
I20260709 12:12:34.948587 12643 rpcz_store.cc:276] 0709 12:12:32.089914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.906232 (+2816318us) service_pool.cc:224] Handling call
0709 12:12:34.906618 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.947822 (+ 41204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.948059 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:34.828881 15479 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:34.878686 15445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.294s	user 0.022s	sys 0.075s
I20260709 12:12:34.909260 15503 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:34.884255 15403 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:34.951355 15480 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:34.956041 15403 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.956908 15479 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:34.944197 15486 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.615s	user 0.012s	sys 0.000s
I20260709 12:12:34.959987 15486 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:34.962738 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.600s (SENT) [suppressed 57 similar messages]
I20260709 12:12:34.964141 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:34.834810 15484 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:12:34.936506 15448 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.722s	user 0.006s	sys 0.008s
I20260709 12:12:34.913551 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1698) took 2796 ms. Trace:
I20260709 12:12:34.913471 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:34.731474 13253 rpcz_store.cc:276] 0709 12:12:33.299755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.301344 (+  1589us) service_pool.cc:224] Handling call
0709 12:12:34.482529 (+1181185us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:34.545239 (+ 62710us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38062848}
I20260709 12:12:34.835507 15391 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.879204 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:12:34.920985 15439 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:34.924422 15397 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.930985 15499 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:34.931284 15343 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:34.890075 15147 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:34.893270 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 156409us
I20260709 12:12:34.966506 15484 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:34.909385 15465 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:34.951776 12643 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:34.967738 12638 rpcz_store.cc:276] 0709 12:12:32.117379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.912233 (+2794854us) service_pool.cc:224] Handling call
0709 12:12:34.912599 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.913278 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.913491 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:34.976830 13261 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 3.
I20260709 12:12:34.999193 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1765) took 2494 ms. Trace:
I20260709 12:12:34.945374 15367 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:34.979182 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65767us
I20260709 12:12:35.003237 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.004276 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:34.986506 13259 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:12:34.986644 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:34.978960 15222 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:34.908984 15167 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.961822 15495 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:35.009582 15499 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:35.009744 15209 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:35.010715 15520 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260709 12:12:35.011171 15495 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.013937 12641 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:35.014729 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1706) took 2002 ms. Trace:
I20260709 12:12:35.015058 12641 rpcz_store.cc:276] 0709 12:12:33.012419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.013206 (+2000787us) service_pool.cc:224] Handling call
0709 12:12:35.013567 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.014379 (+   812us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.014665 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.018352 15468 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:35.018807 15468 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:34.998580 15465 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:35.022110 15448 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.022704 15448 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.023494 15479 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.024247 15391 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.024837 15448 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.025799 15397 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:35.022658 15465 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.027186 15465 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:35.027041 15484 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.028187 15484 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:35.030081 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:35.030807 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.032440 12567 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:35.033643 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.034248 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.035646 12640 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:12:35.036401 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1777) took 1738 ms. Trace:
I20260709 12:12:35.036623 15403 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:35.036768 12640 rpcz_store.cc:276] 0709 12:12:33.297650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.035070 (+1737420us) service_pool.cc:224] Handling call
0709 12:12:35.035395 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:35.036109 (+   714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.036321 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:34.983098 15486 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:34.898694 15502 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:34.947328 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 419) took 1655 ms. Trace:
I20260709 12:12:34.975555 15408 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:34.998106 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1762) took 3086 ms (client timeout 3228 ms). Trace:
W20260709 12:12:35.088035 12646 rpcz_store.cc:269] 0709 12:12:31.911767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.589435 (+1677668us) service_pool.cc:224] Handling call
0709 12:12:34.997242 (+1407807us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:34.998016 (+   774us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1386789120}
I20260709 12:12:34.999900 12638 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [2.2-7.3]
I20260709 12:12:34.999720 12643 rpcz_store.cc:276] 0709 12:12:32.504978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.949735 (+2444757us) service_pool.cc:224] Handling call
0709 12:12:34.950195 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.998824 (+ 48629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.999114 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.096417 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1770) took 2548 ms. Trace:
I20260709 12:12:35.097043 12638 rpcz_store.cc:276] 0709 12:12:32.547618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.999423 (+2451805us) service_pool.cc:224] Handling call
0709 12:12:34.999722 (+   299us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:35.090545 (+ 90823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.096343 (+  5798us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.083456 12568 rpcz_store.cc:276] 0709 12:12:33.292061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.916244 (+624183us) service_pool.cc:224] Handling call
0709 12:12:33.916633 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:34.095826 (+179193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:34.095875 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:34.096622 (+   747us) log.cc:844] Serialized 26 byte log entry
0709 12:12:34.288053 (+191431us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:34.288098 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.288168 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:34.947122 (+658954us) raft_consensus.cc:1685] finished
0709 12:12:34.947138 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:34.947273 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":582,"raft.queue_time_us":49708,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":11862,"thread_start_us":295736,"threads_started":2,"wal-append.queue_time_us":268049}
I20260709 12:12:35.006254 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:34.995091 15482 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.627s	user 0.012s	sys 0.001s
W20260709 12:12:35.131399 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:35.007421 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1686) took 3158 ms (client timeout 1630 ms). Trace:
I20260709 12:12:35.007413 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:34.997660 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 535) took 1483 ms. Trace:
I20260709 12:12:35.100775 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1780) took 1791 ms. Trace:
W20260709 12:12:35.133617 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 126258us
W20260709 12:12:35.132942 12639 rpcz_store.cc:269] 0709 12:12:31.849026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.990336 (+1141310us) service_pool.cc:224] Handling call
0709 12:12:35.007219 (+2016883us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:35.007348 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":870334848}
I20260709 12:12:35.134541 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:34.991622 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 536) took 1475 ms. Trace:
I20260709 12:12:34.990157 15473 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:35.037683 15403 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:35.038280 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1839) took 1831 ms. Trace:
I20260709 12:12:35.149123 12563 rpcz_store.cc:276] 0709 12:12:33.207101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.881117 (+674016us) service_pool.cc:224] Handling call
0709 12:12:33.881402 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.899555 (+1018153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.038064 (+138509us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:35.038224 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2339521920}
I20260709 12:12:34.367422 12642 rpcz_store.cc:276] 0709 12:12:31.850462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:34.356309 (+2505847us) service_pool.cc:224] Handling call
0709 12:12:34.356675 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.357038 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:34.357067 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:34.357119 (+    52us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:34.357147 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.357211 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:34.357345 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":592,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":3813}
I20260709 12:12:35.049305 15167 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.054533 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1967) took 1557 ms. Trace:
W20260709 12:12:35.055680 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1773) took 1759 ms (client timeout 2330 ms). Trace:
I20260709 12:12:35.058626 15496 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:12:35.059948 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1749) took 3196 ms (client timeout 1544 ms). Trace:
W20260709 12:12:35.159761 12637 rpcz_store.cc:269] 0709 12:12:31.863366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:32.076170 (+212804us) service_pool.cc:224] Handling call
0709 12:12:35.059588 (+2983418us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:35.059816 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5151184128}
W20260709 12:12:35.063060 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.064658 15503 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.065793 15506 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:35.066847 15147 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:35.071524 15349 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.072721 15367 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.073961 15495 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.007296 15455 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260709 12:12:35.126276 15502 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:35.075311 15515 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:12:35.076381 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:35.076638 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:35.076803 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:35.078742 12640 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:34.861737 15463 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.082882 12641 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:35.189412 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1707) took 2176 ms. Trace:
I20260709 12:12:35.189307 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1778) took 1891 ms. Trace:
I20260709 12:12:35.189829 12641 rpcz_store.cc:276] 0709 12:12:33.012737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.015843 (+2003106us) service_pool.cc:224] Handling call
0709 12:12:35.016210 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.189195 (+172985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.189368 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.090163 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:35.086961 15443 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:12:35.110852 15457 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:35.112484 15475 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.118103 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 417) took 1842 ms. Trace:
I20260709 12:12:35.159544 15496 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:35.123132 12645 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:35.220718 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 291) took 1687 ms. Trace:
I20260709 12:12:35.221252 12645 rpcz_store.cc:276] 0709 12:12:33.533165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.122383 (+1589218us) service_pool.cc:224] Handling call
0709 12:12:35.122852 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.220344 (+ 97492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.220639 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.082509 15345 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
W20260709 12:12:35.126629 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.086201 15465 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:12:35.136109 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 294) took 1559 ms. Trace:
I20260709 12:12:35.136324 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.143071 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:35.133844 12646 rpcz_store.cc:276] 0709 12:12:33.298409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.089282 (+1790873us) service_pool.cc:224] Handling call
0709 12:12:35.089684 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.089913 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:35.089940 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:35.089990 (+    50us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:35.090033 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.090092 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:35.090238 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.145087 15482 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.133262 13260 rpcz_store.cc:276] 0709 12:12:33.514199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.516442 (+  2243us) service_pool.cc:224] Handling call
0709 12:12:33.533478 (+ 17036us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:34.610285 (+1076807us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b18003d2400. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57ebbfb 00007ffbf57f664b
0709 12:12:34.610478 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:34.610508 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:34.611169 (+   661us) log.cc:844] Serialized 52 byte log entry
0709 12:12:34.839383 (+228214us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:34.967450 (+128067us) log.cc:844] Serialized 14 byte log entry
0709 12:12:34.989030 (+ 21580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:34.989117 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:34.989159 (+    42us) raft_consensus.cc:1685] finished
0709 12:12:34.993679 (+  4520us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:34.997578 (+  3899us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42552,"prepare.queue_time_us":256996,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":284226,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":18333,"spinlock_wait_cycles":89525120,"thread_start_us":566957,"threads_started":3,"wal-append.queue_time_us":286990}
I20260709 12:12:35.155217 15449 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:12:35.151593 12642 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:35.231163 12571 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:35.231750 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 295) took 1641 ms. Trace:
I20260709 12:12:35.232333 12645 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:35.232280 12642 rpcz_store.cc:276] 0709 12:12:33.590833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.150867 (+1560034us) service_pool.cc:224] Handling call
0709 12:12:35.151334 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.231429 (+ 80095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.231679 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.153889 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:35.155401 12644 rpcz_store.cc:269] 0709 12:12:33.296558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.784452 (+487894us) service_pool.cc:224] Handling call
0709 12:12:35.055506 (+1271054us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54002e5754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:35.055616 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2258660224}
W20260709 12:12:35.161734 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.155179 12572 rpcz_store.cc:276] 0709 12:12:33.496710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.931817 (+435107us) service_pool.cc:224] Handling call
0709 12:12:33.932148 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:34.415079 (+482931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.054281 (+639202us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:35.054467 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1071866752}
I20260709 12:12:35.176120 15477 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:35.245924 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:35.149690 15222 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:35.145463 15391 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:35.019688 15468 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.249711 15222 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:35.230593 15467 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.104149 12638 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:12:35.250478 15222 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:35.194550 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 118007us
I20260709 12:12:35.250671 15467 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:35.252079 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:35.253100 15467 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.159276 15408 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:35.213706 12565 rpcz_store.cc:276] 0709 12:12:33.275505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.778930 (+503425us) service_pool.cc:224] Handling call
0709 12:12:35.117862 (+1338932us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:35.118028 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2331968384}
I20260709 12:12:35.119035 12570 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:12:35.098681 15480 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:35.183925 15514 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.301s	user 0.001s	sys 0.003s
W20260709 12:12:35.217824 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 141080us
W20260709 12:12:35.146854 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140632us
I20260709 12:12:35.255930 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:35.169922 15479 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:12:35.224946 12639 rpcz_store.cc:276] 0709 12:12:33.576607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.135205 (+1558598us) service_pool.cc:224] Handling call
0709 12:12:35.135584 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.135788 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:35.135811 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:35.135853 (+    42us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:35.135886 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.135935 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:35.136044 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:35.228247 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92001.3us
I20260709 12:12:35.268035 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.270342 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.235186 12641 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:12:35.236167 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89908.7us
W20260709 12:12:35.234409 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:35.237490 15521 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 12:12:35.238580 15482 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:35.180091 15346 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.013s	sys 0.031s
I20260709 12:12:35.205999 15502 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:12:35.243753 15367 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.178566 15349 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.189934 12640 rpcz_store.cc:276] 0709 12:12:33.297911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.078325 (+1780414us) service_pool.cc:224] Handling call
0709 12:12:35.078592 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.189039 (+110447us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.189248 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.145325 13256 rpcz_store.cc:276] 0709 12:12:33.515655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.520542 (+  4887us) service_pool.cc:224] Handling call
0709 12:12:34.990884 (+1470342us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.289417 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:35.269003 15074 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.527s	user 0.008s	sys 0.004s
W20260709 12:12:35.258016 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 148916us
I20260709 12:12:35.293995 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.294806 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.297117 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:35.260911 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.261922 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.262215 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.250996 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 290) took 1724 ms. Trace:
I20260709 12:12:35.277384 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.298844 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.298581 12638 rpcz_store.cc:276] 0709 12:12:33.527014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.098392 (+1571378us) service_pool.cc:224] Handling call
0709 12:12:35.098851 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.250726 (+151875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.250937 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.299212 12496 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:35.299734 12566 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:12:35.277601 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:35.278503 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.280578 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1722) took 1400 ms. Trace:
I20260709 12:12:35.300408 12641 rpcz_store.cc:276] 0709 12:12:33.880473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.191012 (+1310539us) service_pool.cc:224] Handling call
0709 12:12:35.227072 (+ 36060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.271328 (+ 44256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.280521 (+  9193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.282132 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:35.277490 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:35.284058 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 218731us
I20260709 12:12:35.290668 12494 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:12:35.283545 15346 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.013s	sys 0.032s
I20260709 12:12:35.307070 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:35.291335 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.294781 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:35394 (request call id 9) took 3071 ms. Trace:
W20260709 12:12:35.284251 15501 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:12:35.256839 12564 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:35.315557 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:35.276257 15492 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.205s	user 0.019s	sys 0.043s
I20260709 12:12:35.294221 12493 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:35.322194 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44803us
I20260709 12:12:35.298157 12492 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:35.236737 15475 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:35.231487 15363 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:35.309481 12489 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:35.308589 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:35.310465 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.295975 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:35.310889 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.311224 15366 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.283190 15391 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:35.311789 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.288156 15510 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:35.308617 12646 rpcz_store.cc:276] 0709 12:12:32.283095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.226209 (+2943114us) service_pool.cc:224] Handling call
0709 12:12:35.276530 (+ 50321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:35.295557 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:35.290343 15457 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:35.293718 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.317560 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:35.308251 15496 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.320084 12572 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:12:35.364511 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56051.2us
W20260709 12:12:35.320854 15492 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.255s	user 0.020s	sys 0.044s
I20260709 12:12:35.285476 15147 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:35.325868 12565 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:35.285831 15403 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.310401 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.357318 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:35.358073 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.358163 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:35.283391 15495 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:35.323938 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.359326 12495 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7.
I20260709 12:12:35.361697 15391 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:35.331789 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:35.294066 15465 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.294361 15497 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:35.324780 12569 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:35.363973 12567 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:35.321254 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43748.3us
W20260709 12:12:35.420241 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135365us
I20260709 12:12:35.421059 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:35.421589 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:35.422498 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:35.368697 15345 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.373839 15346 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:35.369216 15074 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:35.355934 15409 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 12:12:35.369491 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.371938 15510 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:35.372850 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.354959 15366 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.352702 15525 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:35.353814 15343 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:35.324483 15486 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:35.324707 15475 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:35.353497 15363 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:35.387293 12494 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:35.414896 12496 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:35.387857 12489 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:35.361042 15400 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:35.417004 12492 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:35.304795 15443 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 12:12:35.422808 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:35.368819 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:35.436793 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:35.429137 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105270us
I20260709 12:12:35.418395 15506 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.371681 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.389245 15349 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.364420 15457 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.623046 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:35.557598 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.624769 12567 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:35.558130 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.612391 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.630115 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:35.630535 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207778us
I20260709 12:12:35.381866 15367 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.162s
I20260709 12:12:35.527669 15345 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.487795 15221 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:35.612468 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:35.535362 15318 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:35.525985 15391 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:35.631719 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.527061 15526 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:35.419920 15403 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.448112 15410 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:35.652446 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:35.657781 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.672354 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:35.644572 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:35.547339 15346 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:35.659603 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:35.679575 12567 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7. Re-sending same reply.
I20260709 12:12:35.680699 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.615011 15363 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:35.443022 15465 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.534237 15074 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:35.674247 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51561.6us
I20260709 12:12:35.676869 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:35.628209 15455 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.628593 15349 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:35.616144 15475 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:35.684762 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.680827 15221 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:35.608160 15525 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.649178 15343 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:35.690510 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.656040 15318 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:35.657838 15408 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:35.634325 15506 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:35.700759 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140290us
I20260709 12:12:35.708820 12563 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:35.708984 12640 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7.
W20260709 12:12:35.710018 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 340574us
I20260709 12:12:35.692948 15400 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:35.727836 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:12:35.741766 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47966.3us
I20260709 12:12:35.730675 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.685724 15410 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.717175 15497 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:35.755342 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.754724 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:35.727600 15318 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:35.763568 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.726819 15221 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:35.726049 15408 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:35.724251 15541 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:35.729069 15400 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:35.746353 15343 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:35.701846 15455 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:35.756558 15484 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.728s	user 0.010s	sys 0.005s
W20260709 12:12:35.790781 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 114145us
I20260709 12:12:35.705161 15349 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.791810 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:35.782040 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109802us
I20260709 12:12:35.752856 15539 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:35.785959 15400 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:35.795217 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:35.774487 15527 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:35.803146 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:35.774994 15408 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:35.766091 15538 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:35.695946 15510 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:35.799528 15534 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:12:35.790199 15343 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:35.782151 15542 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:35.823354 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1470728660001441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.233s for a response
I20260709 12:12:35.819826 15343 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:35.821591 15542 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:35.812908 15538 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:35.818228 15533 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:12:35.814620 15484 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:35.779170 15346 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.816107 15549 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:35.766943 15543 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:12:35.844802 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1827) took 2669 ms. Trace:
I20260709 12:12:35.862290 12570 rpcz_store.cc:276] 0709 12:12:33.175869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:33.863219 (+687350us) service_pool.cc:224] Handling call
0709 12:12:35.118750 (+1255531us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:35.254905 (+136155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.844553 (+589648us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003aaf50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:35.844721 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2886211968}
I20260709 12:12:35.846616 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:35.864643 15548 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:35.867667 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.869614 12570 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:35.871171 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.872058 12570 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:35.823918 15209 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:12:35.873397 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.874118 12570 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:35.836108 15351 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:12:35.880579 15531 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:12:35.882246 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1715) took 2006 ms. Trace:
I20260709 12:12:35.882751 12637 rpcz_store.cc:276] 0709 12:12:33.875625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.160695 (+1285070us) service_pool.cc:224] Handling call
0709 12:12:35.161000 (+   305us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:35.169573 (+  8573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:35.169602 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:35.178280 (+  8678us) log.cc:844] Serialized 52 byte log entry
0709 12:12:35.365048 (+186768us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:35.365090 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:35.365159 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:35.881929 (+516770us) raft_consensus.cc:1685] finished
0709 12:12:35.881963 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:35.882170 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21487,"raft.queue_time_us":15960,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":5013,"thread_start_us":134963,"threads_started":1,"wal-append.queue_time_us":490421}
I20260709 12:12:35.835207 15403 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:35.879629 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.827546 15457 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:35.875794 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:35.830497 15349 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.826725 15074 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.837139 15410 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:35.838413 15465 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:35.852813 15497 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:35.840129 15551 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:35.840518 15538 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:35.898847 15469 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:35.827781 15550 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:35.998699 15469 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:35.861263 15527 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:35.999804 15527 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:36.001082 15527 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:35.843798 15525 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.831866 15542 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.004112 12570 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:35.845515 15549 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:36.009073 15549 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.011085 15455 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:36.013134 12569 raft_consensus.cc:1217] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-5.3]   Dedup: 3.2->[5.3-5.3]
I20260709 12:12:36.013728 12569 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260709 12:12:35.896278 15345 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:35.834198 15221 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:35.896468 15536 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:12:35.843274 15366 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.924554 15465 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.021883 15465 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.024827 15465 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:35.880894 15351 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:36.025949 15550 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:36.026150 15351 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.026823 15550 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:35.892760 15477 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.645s	user 0.011s	sys 0.002s
I20260709 12:12:36.028344 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:36.028513 15480 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.025987 13218 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7.
I20260709 12:12:35.908332 15551 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:35.930742 15391 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.030861 15551 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.032069 15391 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:35.915060 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:35.878281 15363 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:35.913013 15497 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:36.034158 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119272us
I20260709 12:12:36.035516 12645 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:35.920451 15222 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.669s	user 0.010s	sys 0.002s
I20260709 12:12:36.035221 15363 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.036134 15222 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.037415 12568 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:36.037590 15167 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:36.038748 15222 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:35.938755 15408 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.040136 12568 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:36.041416 12568 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:36.041567 15408 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:35.833676 15397 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 12:12:36.042598 12566 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:12:36.043035 12566 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 5
I20260709 12:12:36.043267 12566 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260709 12:12:35.910612 12564 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:36.043476 15475 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:35.939859 15555 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260709 12:12:35.982779 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.035262 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.046669 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:35.988713 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.000s	sys 0.004s
W20260709 12:12:36.047412 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.000s	sys 0.004s
I20260709 12:12:35.919893 15486 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:12:36.048205 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 690089us
I20260709 12:12:36.048467 15486 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.049809 15167 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:36.050266 15167 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.050601 15167 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:35.994649 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 191586us
I20260709 12:12:36.029516 13218 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:36.052049 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.052762 15510 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.053706 15483 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:36.054216 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.055091 15483 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:35.996028 15526 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:36.055620 15483 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:36.055964 15526 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:35.995419 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 180821us
I20260709 12:12:36.056669 15483 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:36.025007 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:35.997871 13220 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.000s	sys 0.005s
I20260709 12:12:35.940747 15400 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.058017 15561 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:36.058355 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:36.058789 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.059306 15561 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:36.035188 15563 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:36.059617 15400 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:36.060175 13220 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.000s	sys 0.005s
I20260709 12:12:36.027518 15351 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.060451 15400 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:36.059662 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.061869 15492 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:36.062278 15492 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.062665 15492 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:36.063437 15439 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:12:36.064366 15439 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:36.064720 15439 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:36.056061 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:35.844836 15484 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.059669 15561 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:36.066177 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.069499 15484 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.056952 15526 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.070451 15526 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:36.059841 15363 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:12:35.978762 15539 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:36.035862 15497 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.049201 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:36.069087 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.080783 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.081792 12488 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:35.979478 15343 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.067786 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.066816 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:36.090906 12643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.030s	sys 0.042s
I20260709 12:12:36.055523 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.043495 12566 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:36.070482 15561 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.084520 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.084892 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.096745 12497 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:36.097014 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45967.9us
I20260709 12:12:36.085181 12491 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 5
I20260709 12:12:36.098050 12491 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:12:36.098398 12491 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:36.099579 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50497.2us
I20260709 12:12:36.084520 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.100370 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.096556 12490 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:36.100811 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:36.091421 12643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.030s	sys 0.042s
I20260709 12:12:36.089150 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.102430 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.103253 12488 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:36.104091 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:36.096032 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:36.095870 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 963204us
I20260709 12:12:36.105535 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.100893 12494 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:36.066017 15439 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.115195 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.070924 15397 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.117776 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:36.127265 12493 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [2.2-7.3]
I20260709 12:12:36.128114 15551 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:36.136974 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.144975 12563 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:35.933279 15318 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.152206 12492 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.152969 15551 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:36.153419 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:36.154213 15318 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:36.149721 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48965.7us
I20260709 12:12:36.104563 12488 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 2.
I20260709 12:12:36.158756 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.163964 12490 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.158870 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:36.165012 12488 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: [5.6-7.7]
W20260709 12:12:36.167171 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61737.1us
I20260709 12:12:36.168310 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:36.101128 15563 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:36.172375 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.184062 12640 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:36.174363 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.176915 12646 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:36.179538 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.179608 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.186527 12638 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.187279 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.187992 12642 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:36.182123 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:36.137609 15345 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:36.206921 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.196969 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.208223 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.075671 15147 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 12:12:36.136865 15222 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.211457 15222 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.214452 15538 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.216324 15455 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:36.217070 15455 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:36.148948 15465 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.080s
W20260709 12:12:36.208252 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49467.9us
I20260709 12:12:36.219244 12638 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:36.206964 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:36.221796 15439 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.222095 12488 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.222735 12488 raft_consensus.cc:614] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:36.116194 15554 raft_consensus.cc:2764] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:12:36.183104 15563 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:36.226243 15563 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.144001 15550 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:36.228765 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.208997 12646 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:36.230608 12637 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:12:36.206981 12490 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:36.209414 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.231670 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.232035 12640 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7. Re-sending same reply.
I20260709 12:12:36.137173 15400 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:36.217589 15455 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.237267 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.240770 12641 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:36.099396 15408 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
W20260709 12:12:36.245097 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.250052 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.251158 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.183485 15561 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.198447 15221 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.116868 15495 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
I20260709 12:12:36.246474 12494 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:36.197817 15403 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:12:36.247151 15536 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:12:36.241458 15410 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:36.157701 15539 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20260709 12:12:36.262874 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.236286 15345 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.263783 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.274991 12637 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:12:36.247781 15409 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.029s	sys 0.042s
I20260709 12:12:36.265936 12641 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:36.157142 15397 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.239112 15400 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:36.272747 12497 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.273032 12644 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:36.207501 15450 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 12:12:36.279526 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:36.272267 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:12:36.289275 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.216480 15542 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.289443 12638 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: [5.6-7.7]
W20260709 12:12:36.278662 15409 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.029s	sys 0.043s
I20260709 12:12:36.290695 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.291033 15222 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.291440 12640 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.292353 15397 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.294804 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.295719 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:36.270335 15410 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:36.299772 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 197379us
I20260709 12:12:36.274708 15473 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.298050 15410 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.305246 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:36.276473 15345 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.274237 15475 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.319208 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:36.320576 13256 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:12:36.262243 15221 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.207218 15469 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.309612 12497 raft_consensus.cc:606] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.338469 12497 raft_consensus.cc:614] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:36.332046 12494 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:36.326877 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.304541 13254 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:36.344254 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.344884 12642 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.345475 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:36.346315 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.332118 12640 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:36.342546 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.347864 12646 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:36.347914 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.342765 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.348560 12645 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.280807 15484 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.348929 12637 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.344053 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.350217 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.351348 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.303073 15209 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:12:36.353046 15475 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:36.359184 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1854) took 1001 ms. Trace:
I20260709 12:12:36.356539 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.370399 15574 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:36.356779 15343 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.348769 15550 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:12:36.373979 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.398655 12644 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.375571 15473 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:36.384459 12568 rpcz_store.cc:276] 0709 12:12:35.357405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.042216 (+684811us) service_pool.cc:224] Handling call
0709 12:12:36.042528 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:36.042802 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:12:36.042831 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:36.043407 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 12:12:36.115786 (+ 72379us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:36.115823 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:36.115896 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:36.358888 (+242992us) raft_consensus.cc:1685] finished
0709 12:12:36.358920 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:36.359108 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":71995,"threads_started":1,"wal-append.queue_time_us":225441}
I20260709 12:12:36.359321 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:36.359679 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:36.405242 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.406679 12568 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.381227 12637 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:36.390339 12642 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.392021 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.395632 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.343261 15527 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:12:36.367677 13255 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:36.395704 15222 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.352217 15469 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:36.402360 12564 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:36.389425 15397 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:12:36.365849 15221 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.383826 15403 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:36.420612 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 363099us
I20260709 12:12:36.355358 15483 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:36.357286 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:36.359725 15559 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:12:36.432340 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.368110 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:36.357076 15575 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:36.377487 15495 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:12:36.384164 15475 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.392979 15510 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7.
I20260709 12:12:36.394474 15409 raft_consensus.cc:2764] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:12:36.401468 15484 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.434191 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.502106 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:36.499245 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:36.499140 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:36.499925 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108034us
I20260709 12:12:36.499902 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.521001 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.515301 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.521001 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.520642 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.522634 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.523115 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.523355 12644 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:36.525660 12644 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9. Ops: [5.6-8.8]
I20260709 12:12:36.527423 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.512817 15575 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:36.546989 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.500129 15408 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:36.528913 15450 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:12:36.522856 15549 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:36.529120 15564 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:36.569195 15564 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.550247 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.517441 15475 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:36.552136 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:36.441205 15222 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:36.554525 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 256d044fd2e244d4be827104688867ee: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:36.558398 13260 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:36.549439 13258 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:12:36.432024 15492 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.497648 15167 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.574962 13258 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 4
I20260709 12:12:36.548648 15483 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:36.554397 15486 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.006s
I20260709 12:12:36.582168 13258 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.5
I20260709 12:12:36.575646 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.586704 13258 raft_consensus.cc:2917] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Aborting config change with OpId 5.5: Aborted: Op aborted by new leader
I20260709 12:12:36.587229 13258 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:36.582721 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.514081 15403 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.588158 12568 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.598901 12568 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:36.600131 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.600706 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.601425 12568 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:36.582728 15343 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.603226 15351 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:36.604111 15351 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:36.603358 15343 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.602463 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:36.605278 15351 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.613734 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.614614 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:36.615978 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.616706 12568 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:36.618076 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.618606 12568 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.627283 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:36.628891 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.630555 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.631351 12568 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:36.632797 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.634672 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.635607 12568 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:36.636956 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.637562 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:36.638770 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.640583 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:12:36.641167 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1858) took 1282 ms. Trace:
I20260709 12:12:36.641460 12568 rpcz_store.cc:276] 0709 12:12:35.358534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.640009 (+1281475us) service_pool.cc:224] Handling call
0709 12:12:36.640349 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:36.640922 (+   573us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:36.641105 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:36.643774 12568 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:12:36.645435 12568 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:36.648159 15410 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:36.649700 15486 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:36.651368 15563 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.652060 15403 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:36.653080 15403 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:36.574513 15564 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:36.661775 12495 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
W20260709 12:12:36.675776 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 154839us
I20260709 12:12:36.676964 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.658701 15167 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.653590 15563 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.679219 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:36.680459 15577 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:12:36.681442 15450 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.683161 15503 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.658818 15465 raft_consensus.cc:2764] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:12:36.665519 15549 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:36.659792 15221 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:36.716163 15221 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.718900 15221 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.719071 15410 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:36.653901 15484 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:36.720144 15484 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.674907 15542 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:36.722209 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.722397 15167 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:36.712611 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:36.723263 15167 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:36.711009 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108618us
I20260709 12:12:36.724149 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.724259 15167 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.725178 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.725531 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:36.726049 15495 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:36.726541 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.726852 15222 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.727092 15583 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:36.670504 15575 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.728266 15542 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:36.728765 15542 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.729151 15542 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.732028 12646 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:36.733289 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:36.710424 15549 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.734076 12489 raft_consensus.cc:606] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.727731 15583 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:36.722654 15484 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.734519 12489 raft_consensus.cc:614] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:36.714072 15345 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.734614 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.691368 15483 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:12:36.738821 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.742369 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.744567 12645 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:12:36.747769 15587 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:36.748299 15587 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.748629 15587 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.669492 15403 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.726895 15495 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:36.734097 15549 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.746320 15484 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:36.757437 12640 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:12:36.757504 12641 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:12:36.755571 15483 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:36.767021 15484 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.767102 15483 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.746101 15492 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.708551 15450 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.694298 15574 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 12:12:36.774019 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.779512 12641 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:36.780336 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:36.780987 12489 raft_consensus.cc:606] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.781324 12489 raft_consensus.cc:614] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:36.757793 15351 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:36.759558 15495 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:36.737373 15455 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.792289 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:36.771330 15222 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:36.771385 15484 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.796873 15583 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:36.818050 15345 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:36.818091 15496 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.821650 15540 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:36.822301 15540 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:36.823266 15540 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.823637 15542 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.824673 15540 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:36.826508 15343 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.828306 15506 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:36.828969 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.831913 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81a9a92616494f46bc97254b53185664"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:36.852834 12489 raft_consensus.cc:606] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:36.856943 15455 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:36.857389 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:36.854933 12489 raft_consensus.cc:614] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:36.860296 12638 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:12:36.787256 15351 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:36.829802 15520 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:36.795193 15495 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:36.866261 15343 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:36.874948 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1974) took 1571 ms (client timeout 1663 ms). Trace:
W20260709 12:12:36.875475 12572 rpcz_store.cc:269] 0709 12:12:35.303108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.419740 (+116632us) service_pool.cc:224] Handling call
0709 12:12:36.874739 (+1454999us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:36.874876 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917926272}
I20260709 12:12:36.878684 15448 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:36.876748 15343 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.882205 15343 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.886231 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.887117 12565 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.866641 15520 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:36.890030 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1857) took 1531 ms. Trace:
I20260709 12:12:36.890607 12564 rpcz_store.cc:276] 0709 12:12:35.358282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.044408 (+686126us) service_pool.cc:224] Handling call
0709 12:12:36.044748 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:36.440287 (+395539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:36.889700 (+449413us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:36.889945 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":803856768}
I20260709 12:12:36.891140 15520 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.892549 15520 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:36.879236 15448 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:36.899302 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:36.896261 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.865844 15351 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:36.920464 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:36.921371 12572 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:36.920495 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:36.928349 12564 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:36.935382 15463 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:36.941442 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2102) took 1652 ms (client timeout 1869 ms). Trace:
W20260709 12:12:36.942070 12496 rpcz_store.cc:269] 0709 12:12:35.288627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.311484 (+ 22857us) service_pool.cc:224] Handling call
0709 12:12:36.941114 (+1629630us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:36.941343 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70393600}
I20260709 12:12:36.943136 15468 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:36.944465 15538 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:36.946624 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:36.854368 15222 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.991215 12637 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 2. (index mismatch)
I20260709 12:12:36.961485 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:36.974354 12565 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:36.979369 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:37.025280 12564 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:36.996701 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 430) took 1967 ms. Trace:
I20260709 12:12:37.015093 12637 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:12:37.066481 12571 rpcz_store.cc:276] 0709 12:12:35.028954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.031484 (+  2530us) service_pool.cc:224] Handling call
0709 12:12:35.031829 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.231552 (+199723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:36.989482 (+1757930us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:36.996611 (+  7129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457941760}
W20260709 12:12:37.037062 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66487.9us
I20260709 12:12:37.065898 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:37.069196 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:37.087512 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.116061 12571 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:36.987919 15533 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260709 12:12:37.025525 15526 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.949s	user 0.011s	sys 0.003s
I20260709 12:12:37.118297 12572 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [5.3-6.4]
I20260709 12:12:36.999084 15455 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:36.919862 15564 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:36.989182 15463 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.119766 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 442) took 1046 ms. Trace:
I20260709 12:12:37.136142 12565 rpcz_store.cc:276] 0709 12:12:36.072969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.065323 (+992354us) service_pool.cc:224] Handling call
0709 12:12:37.119363 (+ 54040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.119522 (+   159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:37.119541 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:37.119581 (+    40us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:37.119601 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.119638 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.119716 (+    78us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:37.124958 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1863) took 1029 ms. Trace:
I20260709 12:12:36.947156 15475 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.138072 12572 rpcz_store.cc:276] 0709 12:12:36.095801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.067346 (+971545us) service_pool.cc:224] Handling call
0709 12:12:37.117988 (+ 50642us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:37.124609 (+  6621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.124890 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:37.115350 15589 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:37.120870 15590 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:37.139268 15502 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:37.143240 15502 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:37.144375 15592 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:12:37.144200 15502 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:37.152040 15571 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.811s	user 0.000s	sys 0.002s
I20260709 12:12:37.158927 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:37.161557 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:37.162099 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.163867 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.164608 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:37.164552 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.165935 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.167842 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.168814 12640 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:37.169071 12490 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:37.169461 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.169878 12489 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:37.170274 12497 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:37.172027 12565 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:37.173525 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.174679 13218 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7.
I20260709 12:12:37.176249 12490 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:37.137503 15542 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.029119 15448 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.180559 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:37.187304 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.190089 12646 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:12:37.133869 15455 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:37.195526 12491 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:37.189596 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:37.152891 15549 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.202030 15549 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.212692 12638 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:12:37.220628 15564 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.222460 15222 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.173394 15590 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:37.225803 15517 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:12:37.181309 15526 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:12:37.171408 15589 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:37.226989 15595 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:12:37.224864 15587 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.224507 15590 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:12:37.239197 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49672.1us
I20260709 12:12:37.248200 15542 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.003s
W20260709 12:12:37.261752 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.190981 15502 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:37.252801 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1774) took 1175 ms. Trace:
I20260709 12:12:37.254061 15526 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:12:37.251101 15589 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.264521 12491 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:37.238667 15455 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:37.266261 12495 rpcz_store.cc:276] 0709 12:12:36.077342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.138572 (+ 61230us) service_pool.cc:224] Handling call
0709 12:12:37.252593 (+1114021us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:37.252728 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087397120}
I20260709 12:12:37.136260 15463 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:37.278362 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.279702 15542 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:37.264060 15473 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.237223 15449 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:37.248560 15448 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:37.284004 15495 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.275993 15526 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:37.288928 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:37.289254 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.294878 13260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:37.296123 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:37.263615 15469 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:37.307104 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.297089 15542 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:37.311199 15542 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.311883 15463 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:37.312445 15542 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:37.313369 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:37.313709 15318 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.314344 15318 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.297493 15364 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:37.311962 15495 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.317988 15364 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:37.319103 15364 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.319928 15474 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.320117 15364 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:37.320580 15474 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.319113 15526 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.324507 15526 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:37.314065 12645 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:37.332077 12495 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:37.309396 15449 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:37.300590 15473 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.334240 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:37.335213 15449 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.336400 15473 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.338143 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:37.339291 13260 raft_consensus.cc:606] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:37.340145 13260 raft_consensus.cc:614] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:37.355050 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.356429 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.357396 12638 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:37.339560 15468 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:37.362385 15468 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:37.363639 15517 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.363627 15468 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.364426 15517 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.366721 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1866) took 1019 ms. Trace:
I20260709 12:12:37.370662 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:37.371887 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.372457 12638 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6. Re-sending same reply.
I20260709 12:12:37.369140 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:37.367148 12564 rpcz_store.cc:276] 0709 12:12:36.347139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.069105 (+721966us) service_pool.cc:224] Handling call
0709 12:12:37.121283 (+ 52178us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:37.121702 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:37.121733 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:37.122395 (+   662us) log.cc:844] Serialized 52 byte log entry
0709 12:12:37.309991 (+187596us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:37.310056 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.310157 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:37.366490 (+ 56333us) raft_consensus.cc:1685] finished
0709 12:12:37.366511 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.366657 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":171,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5293,"thread_start_us":168494,"threads_started":1,"wal-append.queue_time_us":226051}
I20260709 12:12:37.427726 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.429764 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.428848 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.428438 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.432421 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.434242 12488 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:37.431277 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.408661 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:37.408555 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.435134 12495 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:37.382704 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.435942 12490 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:37.431012 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:37.430617 12497 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:37.437091 12497 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:37.435014 12637 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:37.384568 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:37.439803 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.440689 12564 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:37.441165 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.442114 12496 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:37.441735 12491 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:37.438092 12497 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.443907 12497 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:37.444257 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.445295 12645 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:37.450721 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.327389 15469 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.456415 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.403950 15473 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:37.384754 15495 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.461421 15400 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.465970 15403 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:37.467448 15455 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.472445 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.410597 15517 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.489341 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.490053 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:37.494572 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:37.495080 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.498277 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.498723 12645 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:37.495724 12497 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 12:12:37.497543 12564 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:37.499507 12492 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:37.499570 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.500283 12638 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:37.472301 15473 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:37.500301 12497 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 3
I20260709 12:12:37.501432 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:37.508694 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.508797 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:37.509333 12497 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:12:37.479512 15448 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:37.509778 12497 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:37.513609 12495 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:37.498440 15400 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:37.509614 15473 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:37.490415 15351 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.511215 15448 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:37.488690 15495 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:37.529852 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157234us
I20260709 12:12:37.536964 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:37.539088 15469 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.543982 15403 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:37.550099 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:37.554935 15403 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:37.556684 12646 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:12:37.557798 15463 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:37.558429 15463 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.559738 12646 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:37.560575 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1765) took 1500 ms. Trace:
I20260709 12:12:37.560726 15463 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.561023 12493 rpcz_store.cc:276] 0709 12:12:36.060394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.066669 (+  6275us) service_pool.cc:224] Handling call
0709 12:12:36.096485 (+ 29816us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:36.140066 (+ 43581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.560266 (+1420200us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:37.560506 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69946752}
W20260709 12:12:37.563778 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.564059 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:37.565348 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1988) took 1037 ms. Trace:
I20260709 12:12:37.565740 12572 rpcz_store.cc:276] 0709 12:12:36.527540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.139046 (+611506us) service_pool.cc:224] Handling call
0709 12:12:37.166978 (+ 27932us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:37.167557 (+   579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:37.167606 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:37.168146 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:12:37.370389 (+202243us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:37.370432 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.370503 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:37.565141 (+194638us) raft_consensus.cc:1685] finished
0709 12:12:37.565162 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.565292 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1890,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":43371,"thread_start_us":142710,"threads_started":1,"wal-append.queue_time_us":245771}
W20260709 12:12:37.569909 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.572082 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:37.574110 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1848) took 2274 ms. Trace:
I20260709 12:12:37.575510 12570 rpcz_store.cc:276] 0709 12:12:35.299958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.874855 (+574897us) service_pool.cc:224] Handling call
0709 12:12:37.573827 (+1698972us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:37.574039 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261578112}
I20260709 12:12:37.567137 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:37.580713 13256 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:37.580758 13261 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:12:37.582544 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:37.573606 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1969) took 2272 ms. Trace:
I20260709 12:12:37.586632 12569 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:37.586390 12566 rpcz_store.cc:276] 0709 12:12:35.301310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.911412 (+610102us) service_pool.cc:224] Handling call
0709 12:12:35.911907 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:36.092113 (+180206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.573335 (+1481222us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:37.573537 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":299639168}
I20260709 12:12:37.588718 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.590842 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:37.594712 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:37.595649 12566 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:37.597468 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:37.598405 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:37.586175 15483 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:37.599498 15465 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:37.600225 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:37.600409 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:37.601070 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.639420 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:12:37.600929 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:37.602193 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1976) took 2298 ms (client timeout 2256 ms). Trace:
W20260709 12:12:37.641160 12563 rpcz_store.cc:269] 0709 12:12:35.303716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.730347 (+426631us) service_pool.cc:224] Handling call
0709 12:12:37.601923 (+1871576us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:37.602113 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":897335552}
I20260709 12:12:37.602391 12568 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:12:37.603475 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:37.604912 15448 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.606098 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.589488 15445 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.576784 15221 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:37.607784 15495 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.591588 15403 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:12:37.596694 15564 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:37.609706 15450 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.573268 15589 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:12:37.610100 15502 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.610292 15475 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:37.610836 12567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 1977) took 2306 ms (client timeout 2000 ms). Trace:
W20260709 12:12:37.687666 12567 rpcz_store.cc:269] 0709 12:12:35.303991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.684022 (+380031us) service_pool.cc:224] Handling call
0709 12:12:37.610582 (+1926560us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:37.610752 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":943842048}
I20260709 12:12:37.592916 15497 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:12:37.550657 15364 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.617918 15520 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:37.609328 15410 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:37.623858 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:37.703474 13256 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:37.628829 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:37.630339 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:37.634625 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:37.706521 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.637894 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:37.637892 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 296) took 2357 ms. Trace:
I20260709 12:12:37.586447 15587 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.687147 15448 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:37.607376 13253 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:12:37.708568 12639 rpcz_store.cc:276] 0709 12:12:35.280527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.289418 (+  8891us) service_pool.cc:224] Handling call
0709 12:12:35.289770 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.290024 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:35.362842 (+ 72818us) log.cc:844] Serialized 14 byte log entry
0709 12:12:35.838780 (+475938us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b18002d7a80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1
0709 12:12:35.840118 (+  1338us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b18002d7a80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:36.112217 (+272099us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b18002d7a80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:12:36.115501 (+  3284us) log.cc:844] Serialized 14 byte log entry
0709 12:12:36.415243 (+299742us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:37.524250 (+1109007us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.637440 (+113190us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:37.637505 (+    65us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:37.637568 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.637671 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.637820 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3149378,"prepare.queue_time_us":2342380,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":155,"raft.queue_time_us":258105,"raft.run_cpu_time_us":2572,"raft.run_wall_time_us":261357,"spinlock_wait_cycles":139952768,"thread_start_us":221813,"threads_started":3,"wal-append.queue_time_us":338213,"wal-append.run_cpu_time_us":5008,"wal-append.run_wall_time_us":1568018}
I20260709 12:12:37.642453 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:37.651309 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.638733 15483 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.643316 15542 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:37.680639 12572 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:37.832152 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1995) took 1281 ms. Trace:
I20260709 12:12:37.638931 15537 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:37.704819 15564 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:37.696092 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:37.842198 13262 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:37.643819 12571 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:37.643062 15473 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:37.711243 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82515.4us
W20260709 12:12:37.713228 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:37.716774 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78976.5us
I20260709 12:12:37.662348 15445 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.651408 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:37.850869 13260 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:37.719126 12563 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:37.721455 12496 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:37.739645 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:37.779862 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.832355 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1991) took 1283 ms. Trace:
I20260709 12:12:37.856858 12572 rpcz_store.cc:276] 0709 12:12:36.548705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.653584 (+1104879us) service_pool.cc:224] Handling call
0709 12:12:37.680413 (+ 26829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.832130 (+151717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.832307 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:37.857882 13261 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:37.858371 15410 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:37.832182 15455 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.861896 13257 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
W20260709 12:12:37.694725 15552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.416s	user 0.005s	sys 0.009s
I20260709 12:12:37.832729 12564 rpcz_store.cc:276] 0709 12:12:36.551341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.827794 (+1276453us) service_pool.cc:224] Handling call
0709 12:12:37.831634 (+  3840us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.831815 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:37.831840 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:37.831880 (+    40us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:37.831911 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.831961 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.832094 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:37.863802 15552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.587s	user 0.005s	sys 0.009s
I20260709 12:12:37.864794 12564 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:12:37.696687 15502 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.831700 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.865785 15351 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:37.866345 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:37.867689 15502 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:37.865497 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1997) took 1276 ms. Trace:
W20260709 12:12:37.868515 15526 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.001s
I20260709 12:12:37.868793 12564 rpcz_store.cc:276] 0709 12:12:36.589301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.864222 (+1274921us) service_pool.cc:224] Handling call
0709 12:12:37.864547 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:37.865212 (+   665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.865443 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:37.870388 12564 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:12:37.718299 13253 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 5
I20260709 12:12:37.870882 13253 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:12:37.871196 13253 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:37.871573 13253 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 12:12:37.871838 13253 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:37.828316 15403 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:37.718487 15573 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.873968 15573 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.875032 15483 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:37.875085 12564 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:37.719709 15590 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:37.876293 15573 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:37.877207 15449 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.703064 15364 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:37.877825 15449 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.847788 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:37.879928 15503 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:37.880470 15449 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.880538 15503 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:37.882000 15517 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.882251 15503 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.882604 15517 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.887494 15364 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:37.890828 12572 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:37.695998 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:37.867660 12570 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:37.901839 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2001) took 1085 ms. Trace:
I20260709 12:12:37.902350 12570 rpcz_store.cc:276] 0709 12:12:36.816951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.867177 (+1050226us) service_pool.cc:224] Handling call
0709 12:12:37.867479 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.901435 (+ 33956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.901758 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:37.911868 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:37.914875 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1748) took 1167 ms. Trace:
I20260709 12:12:37.920825 13255 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 4.
I20260709 12:12:37.980189 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.921975 15622 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:37.983892 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
W20260709 12:12:37.922644 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.931003 12563 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:37.936832 12571 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:37.836925 15542 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.006956 15542 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:37.847249 15564 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:37.837081 15587 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:37.862321 15607 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 12:12:37.974373 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1749) took 1225 ms. Trace:
W20260709 12:12:37.976075 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:37.977876 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:12:37.980022 12496 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:37.981600 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:37.982966 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.002983 12393 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 4 to 6, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 6 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.004760 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:38.005227 15445 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.005496 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.032840 12493 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:38.005879 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.005463 15520 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:38.012152 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.045830 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:38.018692 13261 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:12:38.018962 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:37.989230 13260 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:38.019817 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:38.050663 15408 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.050874 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.051277 15408 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.043004 15345 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.052613 15345 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.054371 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:37.991055 15222 raft_consensus.cc:2764] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:12:38.055080 15345 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:38.056104 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.043330 15409 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:12:38.057466 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:38.058118 12489 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.056764 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:38.060319 13259 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:38.061676 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:38.061864 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:37.983371 15573 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:38.062554 15408 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.062965 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.063297 15408 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:38.063800 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:38.064409 12490 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:38.064795 15520 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:38.065236 15520 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.027045 12488 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:38.065615 15520 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:38.034443 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:38.035871 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.055166 12494 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:38.069995 12639 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:38.037927 12642 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:38.071321 12494 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:38.024427 15622 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.041307 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.072412 15628 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:38.072665 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:38.073936 15622 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.043397 15587 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:37.981498 12644 rpcz_store.cc:276] 0709 12:12:36.747680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.748214 (+   534us) service_pool.cc:224] Handling call
0709 12:12:36.748579 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:36.748828 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:37.687762 (+938934us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.913868 (+226106us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.914364 (+   496us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:37.914432 (+    68us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:37.914500 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.914598 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.914787 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":778336,"prepare.queue_time_us":795368,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":86,"raft.queue_time_us":248,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":231,"thread_start_us":377494,"threads_started":2}
I20260709 12:12:38.039712 12646 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:38.072914 15628 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.075413 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:38.076010 15587 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.076186 13259 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:38.033161 15539 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:12:38.076471 15628 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.051745 12640 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:38.020664 12491 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:38.113976 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:38.054289 15502 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:38.117297 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.119671 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:38.120431 12491 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:38.120124 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:38.028848 12641 rpcz_store.cc:276] 0709 12:12:36.749333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.783763 (+ 34430us) service_pool.cc:224] Handling call
0709 12:12:36.784123 (+   360us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:36.947982 (+163859us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:36.948021 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:36.968048 (+ 20027us) log.cc:844] Serialized 52 byte log entry
0709 12:12:37.117595 (+149547us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:37.662363 (+544768us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.973797 (+311434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:37.973943 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:37.974025 (+    82us) raft_consensus.cc:1685] finished
0709 12:12:37.974042 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:37.974277 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":434403,"prepare.queue_time_us":426684,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":68713,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":3971,"thread_start_us":619765,"threads_started":4,"wal-append.queue_time_us":189946}
I20260709 12:12:38.102145 15575 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:38.104200 15583 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:38.110488 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1979) took 2804 ms. Trace:
I20260709 12:12:38.082628 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:38.129323 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.130360 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.129395 12569 rpcz_store.cc:276] 0709 12:12:35.305950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.039626 (+733676us) service_pool.cc:224] Handling call
0709 12:12:37.574209 (+1534583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.587297 (+ 13088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.110251 (+522954us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54003acd50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:38.110411 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3528336256}
I20260709 12:12:38.131115 12489 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 4: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:38.023653 15483 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.136922 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1996) took 1552 ms. Trace:
I20260709 12:12:38.131034 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:38.140461 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 289) took 4622 ms. Trace:
W20260709 12:12:38.127955 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 491020us
I20260709 12:12:38.141013 12572 rpcz_store.cc:276] 0709 12:12:36.584713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.863393 (+1278680us) service_pool.cc:224] Handling call
0709 12:12:37.863790 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.891407 (+ 27617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.136531 (+245124us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:38.136834 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61113728}
I20260709 12:12:38.141530 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:38.148082 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.142751 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:38.082502 15551 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260709 12:12:38.062742 15573 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.125551 15537 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:12:38.141242 15560 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:38.162433 15400 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:38.162988 15400 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:38.163995 15400 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.167443 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:38.168685 15517 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.145920 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:38.145046 15483 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.157545 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 543) took 1920 ms. Trace:
I20260709 12:12:38.168687 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:38.143806 15583 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.056149 15564 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.143388 12643 rpcz_store.cc:276] 0709 12:12:33.518500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:35.091581 (+1573081us) service_pool.cc:224] Handling call
0709 12:12:35.092018 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:35.092353 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:35.323707 (+231354us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.523379 (+2199672us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b18000bafa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:12:37.671201 (+147822us) log.cc:844] Serialized 14 byte log entry
0709 12:12:38.139967 (+468766us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:38.140035 (+    68us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:38.140105 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.140214 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:38.140379 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":630488,"prepare.queue_time_us":1699755,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":93,"raft.queue_time_us":78957,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":35241,"spinlock_wait_cycles":55213312,"thread_start_us":1553816,"threads_started":4,"wal-append.queue_time_us":830236,"wal-append.run_cpu_time_us":1747,"wal-append.run_wall_time_us":1012402}
I20260709 12:12:38.168875 15584 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:38.193064 13258 rpcz_store.cc:276] 0709 12:12:36.237670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.259480 (+ 21810us) service_pool.cc:224] Handling call
0709 12:12:36.259874 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:36.587760 (+327886us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.157101 (+1569341us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:38.157431 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":618136960}
I20260709 12:12:38.149609 15455 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:38.170508 15400 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:38.171465 15474 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.149722 15410 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:12:38.206411 15549 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.196369 15495 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:38.213439 15455 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:38.220377 15564 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.144196 15503 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.139004 15502 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.231295 15503 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.359830 15410 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.360672 15410 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.270361 12489 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:38.362994 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.364069 12489 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:38.364917 15345 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:38.272078 12494 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
W20260709 12:12:38.273767 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 238975us
I20260709 12:12:38.275025 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:38.281594 12497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.035s	sys 0.054s
I20260709 12:12:38.288208 12488 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:38.291882 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:38.294715 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:38.294607 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:38.297557 12571 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:12:38.357400 15502 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:38.310709 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 141887us
I20260709 12:12:38.179090 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.178812 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.176918 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:38.313053 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166992us
I20260709 12:12:38.180470 12640 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:12:38.397205 15551 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.313186 12492 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:12:38.202545 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:38.220084 15549 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.237870 15573 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.223253 15583 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:38.226356 15611 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:12:38.403502 15474 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:38.403847 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:38.404385 12642 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:38.344914 15533 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:38.348742 15584 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:38.401607 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 198044us
I20260709 12:12:38.406186 15584 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.350055 15483 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.365448 15345 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.376166 15480 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.408742 15345 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:38.409068 15480 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.409914 15584 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.412441 15480 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.372486 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:38.378129 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.361536 15503 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.384708 12566 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:38.414960 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:12:38.391206 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.398576 15551 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.421137 15551 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:38.234455 15625 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:12:38.399641 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:38.388116 12497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.036s	sys 0.054s
I20260709 12:12:38.432241 15455 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.433382 15483 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.434300 15483 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.026157 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:38.404243 15474 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.422394 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:38.423012 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1861) took 2547 ms. Trace:
I20260709 12:12:38.423243 12490 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:38.441969 15584 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:38.446337 15410 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.441510 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:38.452760 15549 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.454577 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.424988 12393 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.426987 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:38.428118 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.437005 15483 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.465847 15474 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.472568 12393 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 7, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 7 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.474807 15527 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:38.475375 15527 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:38.476824 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76043.7us
I20260709 12:12:38.480029 12495 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
W20260709 12:12:38.480796 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49444.9us
W20260709 12:12:38.482095 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.481920 15483 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:12:38.482743 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:38.483527 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.484107 15533 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:38.485226 15533 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:12:38.486375 13253 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 7 LEARNER]: Advancing to term 9
I20260709 12:12:38.485226 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:38.484165 15537 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:38.490273 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:38.490803 15537 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:38.491690 15638 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:38.492385 15537 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.494247 15622 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:38.492457 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 1989) took 1962 ms. Trace:
W20260709 12:12:38.510376 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83736.6us
I20260709 12:12:38.441448 12568 rpcz_store.cc:276] 0709 12:12:35.875390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.646532 (+771142us) service_pool.cc:224] Handling call
0709 12:12:37.602084 (+955552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:37.643088 (+ 41004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.422371 (+779283us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54002cf6d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:38.422735 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196400384}
I20260709 12:12:38.475426 12393 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.524262 15397 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:38.524915 15397 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.525283 15397 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:38.526383 15463 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:38.528447 15467 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:38.483593 15502 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:38.529657 15467 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.530890 15628 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:38.534648 15442 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:12:38.536369 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:38.537889 15635 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:38.538532 15635 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.540064 15635 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.541819 15622 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.544412 15467 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.529687 15502 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.547606 15502 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.552245 15318 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.553777 15408 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:12:38.492451 15638 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:38.569046 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:38.569952 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.573256 12641 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:12:38.574466 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.574770 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:38.575644 12646 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:38.575908 12642 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:38.578441 12643 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 12:12:38.578994 12643 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 6
I20260709 12:12:38.579231 12640 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:38.538432 15473 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.579285 12643 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:12:38.581774 12643 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:38.509800 12565 rpcz_store.cc:276] 0709 12:12:36.529875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:37.193848 (+663973us) service_pool.cc:224] Handling call
0709 12:12:37.260454 (+ 66606us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:37.260728 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:37.393686 (+132958us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.587425 (+193739us) log.cc:844] Serialized 14 byte log entry
0709 12:12:37.900610 (+313185us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b1800480b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:38.125470 (+224860us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b1800480b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:38.491393 (+365923us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b1800480b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:12:38.491741 (+   348us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:38.491846 (+   105us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:38.491934 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.492066 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:38.492256 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":397233,"prepare.queue_time_us":265925,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":99,"raft.queue_time_us":931452,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":50431,"spinlock_wait_cycles":159356928,"thread_start_us":292324,"threads_started":2,"wal-append.queue_time_us":267207}
I20260709 12:12:38.585135 15527 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.587275 15527 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:38.569252 15638 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.544839 15622 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.610831 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:38.617908 15502 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.635887 15345 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:38.636559 15542 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.614s	user 0.012s	sys 0.004s
I20260709 12:12:38.637794 12645 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:38.638705 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47056us
I20260709 12:12:38.640322 15584 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:12:38.655680 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46557.2us
I20260709 12:12:38.656639 12642 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:38.664109 13253 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 9 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:38.692601 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:38.684975 15587 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.699954 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:38.671765 15480 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 12:12:38.685470 15620 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:38.664418 15542 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.677819 15584 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:12:38.705713 15575 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20260709 12:12:38.711544 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 175487us
W20260709 12:12:38.734112 15520 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.668s	user 0.013s	sys 0.007s
I20260709 12:12:38.734772 15520 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.690052 15474 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:38.703447 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 1.06472e+06us
I20260709 12:12:38.718080 15644 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.699263 15637 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:12:38.719638 15620 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.754684 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:38.755678 15620 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.722342 15345 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.756757 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:38.723980 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:38.726145 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 680247us
I20260709 12:12:38.758157 13253 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:38.739060 12641 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 5: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:38.762164 15615 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:38.763403 15615 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:12:38.765154 15615 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:12:38.774701 15473 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:38.773850 12393 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 4 to 6, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 6 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:12:38.775151 15473 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.775619 15473 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:38.775797 15615 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:38.746892 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50129.3us
I20260709 12:12:38.747210 15409 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.777822 15615 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.751113 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.750218 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.783394 15474 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.784230 15520 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.785869 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:38.751806 15644 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.786602 13258 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:38.788398 15644 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:38.752982 15403 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:38.765832 15408 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:38.789433 15403 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:38.770752 15587 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.789815 15408 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.790858 15408 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.791029 15403 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.791901 15587 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:38.774030 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91363.4us
I20260709 12:12:38.745036 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:38.794813 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:38.744458 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74800.8us
I20260709 12:12:38.778824 15409 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.769868 15575 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:38.797717 15409 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:38.798290 15575 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:38.798477 15221 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:38.798966 15575 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.799484 15575 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:38.799882 15221 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:38.801290 15584 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.778391 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:38.789709 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.02827e+06us
I20260709 12:12:38.802306 15584 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.803637 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:38.792429 15408 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.805280 15442 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:38.805603 15318 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:38.805864 15442 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.806562 15442 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:38.806356 15318 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:38.795445 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94426.6us
I20260709 12:12:38.807579 15318 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.809028 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:38.809508 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:38.810145 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:12:38.810467 12568 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:38.811179 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:38.812072 15551 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:12:38.812403 15564 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.581s	user 0.006s	sys 0.010s
I20260709 12:12:38.812587 15551 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:38.812959 12563 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:38.812995 15551 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:38.810134 15527 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:38.816947 15527 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:38.785094 12393 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.810945 15637 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:38.818749 12393 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.798339 15409 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:38.820731 15409 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:38.822118 15409 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.824658 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:38.825762 15400 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.005s
I20260709 12:12:38.817407 15527 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:38.826469 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.827119 15480 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:12:38.828073 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 220) took 20191 ms (client timeout 15000 ms). Trace:
W20260709 12:12:38.828851 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.828853 15474 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.830627 15463 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:38.830917 15642 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:12:38.844224 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87655.6us
I20260709 12:12:38.845220 15542 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:38.888507 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116048us
W20260709 12:12:38.882507 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81980.5us
I20260709 12:12:38.894048 15584 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:38.892066 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:38.862983 12489 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 2
I20260709 12:12:38.891719 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:38.857535 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46478.6us
I20260709 12:12:38.899703 12494 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 12:12:38.900355 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:38.848320 12393 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 1 to 3, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 3 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:38.950564 12489 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:12:38.920495 13253 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:12:38.940234 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115705us
I20260709 12:12:38.950992 12489 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:38.940444 13258 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:38.895879 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:38.947086 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.900631 15409 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:38.955725 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:38.925141 15397 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:38.967635 15583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.000s	sys 0.014s
I20260709 12:12:38.896179 15615 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:38.958725 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:38.968223 15583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.000s	sys 0.014s
I20260709 12:12:38.986567 12563 raft_consensus.cc:2368] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:38.987588 12571 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:12:38.993908 12569 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:38.994669 12490 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:38.906461 15318 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.003784 12568 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:39.003140 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.011648 15318 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:39.012177 13261 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:39.012496 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:39.013087 15437 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:39.013756 15437 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:39.014217 15437 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:39.016428 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:39.017285 12495 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:39.017987 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122248us
W20260709 12:12:39.018374 15647 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:12:39.019119 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:39.020047 15445 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:12:39.003702 15587 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:12:39.022294 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:39.022979 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:39.012650 15318 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
W20260709 12:12:39.024688 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63482.3us
I20260709 12:12:39.025146 15363 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:38.979337 15584 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:39.026193 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:38.976955 15474 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.025981 15363 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.051419 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.045233 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:39.046593 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:39.047501 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.048359 15587 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:39.049993 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:39.050522 15533 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:39.051326 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:39.039909 15628 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:12:39.052912 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.064873 12495 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:39.039783 15622 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.066799 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.068336 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:39.076597 12393 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:39.076232 13262 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:39.022681 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:39.050309 15552 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:12:39.086197 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.054330 15502 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:39.082763 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.085878 13258 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:12:39.101928 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.103767 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:39.053982 15560 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:12:39.093035 12496 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
W20260709 12:12:39.108635 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.106904 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.120991 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.122562 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.123628 13257 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:39.128345 15622 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:39.130519 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84065.2us
I20260709 12:12:39.131953 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:39.132763 12645 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:39.134085 13254 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:12:39.134864 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:39.134137 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.087818 15221 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:39.113356 12393 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 3, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 3 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:39.118225 15345 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:39.170583 12645 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:12:39.175109 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.173028 15551 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.143059 15642 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.134140 15590 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:12:38.928694 12392 rpcz_store.cc:269] 0709 12:12:18.636863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:18.637169 (+   306us) service_pool.cc:224] Handling call
0709 12:12:38.827661 (+20190492us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:29.883111 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:29.883741 (+   630us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:29.883782 (+    41us) write_op.cc:435] Acquired schema lock
0709 12:12:29.883803 (+    21us) tablet.cc:662] Decoding operations
0709 12:12:29.885534 (+  1731us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:29.885745 (+   211us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:29.885782 (+    37us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:12:30.072067 (+186285us) tablet.cc:701] Row locks acquired
0709 12:12:30.072134 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:12:30.172137 (+100003us) write_op.cc:270] Start()
0709 12:12:30.198473 (+ 26336us) write_op.cc:276] Timestamp: P: 1783599150139700 usec, L: 0
0709 12:12:30.200932 (+  2459us) op_driver.cc:348] REPLICATION: starting
0709 12:12:30.355395 (+154463us) log.cc:844] Serialized 8269 byte log entry
0709 12:12:32.805882 (+2450487us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:12:33.296477 (+490595us) op_driver.cc:464] REPLICATION: finished
0709 12:12:33.882065 (+585588us) write_op.cc:301] APPLY: starting
0709 12:12:33.898958 (+ 16893us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:34.059745 (+160787us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:34.068683 (+  8938us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:34.376461 (+307778us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:34.376948 (+   487us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=21
0709 12:12:34.377001 (+    53us) write_op.cc:312] APPLY: finished
0709 12:12:34.386113 (+  9112us) log.cc:844] Serialized 145 byte log entry
0709 12:12:34.510988 (+124875us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:34.511223 (+   235us) write_op.cc:454] Released schema lock
0709 12:12:34.511309 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:34.511770 (+   461us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:34.511803 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":48365,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":549823,"apply.run_cpu_time_us":50456,"apply.run_wall_time_us":658666,"num_ops":21,"prepare.queue_time_us":66721,"prepare.run_cpu_time_us":95066,"prepare.run_wall_time_us":3116575,"raft.queue_time_us":248503,"raft.run_cpu_time_us":67142,"raft.run_wall_time_us":360808,"replication_time_us":3075752,"spinlock_wait_cycles":93619072,"thread_start_us":3153752,"threads_started":5,"wal-append.queue_time_us":3082972,"wal-append.run_cpu_time_us":11053,"wal-append.run_wall_time_us":2273822}]]}
I20260709 12:12:39.116808 15552 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:39.199683 13258 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:12:39.203053 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.204035 15622 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:39.221204 12571 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:39.190691 12393 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:39.221297 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:39.223089 12490 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:39.164003 15542 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.174046 15345 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:39.205770 15363 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:39.207796 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.159569 15502 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.200848 15520 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:39.244858 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.248257 15345 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.253785 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.254424 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:39.255158 12642 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:39.258152 12638 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:39.257712 12393 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 5 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 6 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:12:39.258631 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:39.258940 12565 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:39.259847 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.265254 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:39.265738 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.266108 12639 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:39.266422 13259 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:39.213884 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
W20260709 12:12:39.293140 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 330758us
W20260709 12:12:39.272743 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88161.3us
I20260709 12:12:39.298713 12492 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:39.282863 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:39.226574 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.179872 15463 raft_consensus.cc:2764] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:12:39.228422 15628 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.229377 12563 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:39.204213 15551 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:39.234468 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.304440 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:39.307871 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87256.8us
I20260709 12:12:39.303216 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.315641 12569 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:39.271173 15397 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:39.323730 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.306634 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:39.328657 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:39.258558 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.328960 15437 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.330431 13255 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:39.330113 15552 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.332048 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.330274 15620 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:39.337373 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.333244 12570 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:39.272558 15520 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:39.271020 15590 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:39.355085 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.351747 12568 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:39.291496 15542 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.272015 15502 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.336867 15620 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:39.379487 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.345084 15437 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:39.368702 15473 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.593s	user 0.017s	sys 0.000s
I20260709 12:12:39.395591 12570 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:39.380211 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.381371 12564 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:39.382211 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:39.394634 15620 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:39.391237 15502 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:39.223310 15587 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.294080 15642 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.358263 15551 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:39.349989 15520 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:39.364400 15537 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:12:39.358114 15445 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:12:39.398990 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47557.8us
I20260709 12:12:39.407367 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:39.440121 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.413931 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.441617 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.414187 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.415374 12568 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 2. (term mismatch)
I20260709 12:12:39.415205 15635 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:39.416194 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.421959 15397 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:39.422628 15656 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.423528 15590 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.404196 15638 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:12:39.367776 15408 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.426090 15480 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:39.407229 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100784us
I20260709 12:12:39.456239 15635 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:39.364861 15560 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:12:39.375171 15474 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:39.352257 15628 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:39.435634 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.443058 12570 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:39.443693 12568 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 1
I20260709 12:12:39.594524 12568 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:12:39.594867 12568 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:39.436583 15551 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:39.541818 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.486488 12569 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:39.445588 15445 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:39.567847 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98097.2us
I20260709 12:12:39.395565 15473 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.470579 15397 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:39.427475 15542 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:39.570094 15483 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:39.433328 15527 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.602586 15551 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:39.573159 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115444us
I20260709 12:12:39.676115 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.593236 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:39.579485 15575 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.779s	user 0.012s	sys 0.004s
I20260709 12:12:39.595288 12568 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.3
I20260709 12:12:39.639752 15397 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:39.683418 12568 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:39.599956 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.602778 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:39.602280 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.584084 15408 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.640270 15469 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.289s
I20260709 12:12:39.640552 15656 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:39.642923 15655 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.000s	sys 0.005s
I20260709 12:12:39.579075 15502 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:12:39.650106 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.580379 12565 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:39.589728 15480 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:39.639201 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:39.702937 15442 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.896s	user 0.016s	sys 0.003s
I20260709 12:12:39.741484 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.677574 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:39.751305 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156627us
W20260709 12:12:39.681203 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 299053us
I20260709 12:12:39.685141 13256 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:39.687747 12491 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:39.754390 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.755981 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:39.708163 15635 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.760150 15473 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:39.755981 15642 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:39.761660 15642 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:12:39.762209 15642 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:39.762730 15642 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:39.765899 15463 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:39.766408 15463 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:39.766817 15463 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:39.774147 12570 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:39.775511 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.776516 12570 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:39.778268 15400 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:39.779163 15400 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.780603 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:39.781868 15622 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.777459 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.638543 15445 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:39.696753 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.694777 15575 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:39.789513 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 486335us
I20260709 12:12:39.666703 15363 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:39.704548 15455 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.697805 15628 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:39.713164 15437 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:39.761761 12569 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:39.790621 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:39.797000 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:39.662230 15483 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:39.782243 15658 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.116s	user 0.001s	sys 0.001s
I20260709 12:12:39.569602 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:39.796203 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:39.810261 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 108252us
W20260709 12:12:39.581377 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103828us
I20260709 12:12:39.814435 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:39.824839 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.746227 15560 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:39.828877 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 188482us
I20260709 12:12:39.793141 15400 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:39.829725 15409 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.861s	user 0.018s	sys 0.001s
W20260709 12:12:39.847628 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91818.3us
I20260709 12:12:39.854032 12460 leader_election.cc:400] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:39.854482 12460 leader_election.cc:403] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:12:39.847841 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.813406 15483 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:39.855763 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:39.745106 15442 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.865598 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:39.868491 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.872923 12497 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:39.822088 15474 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:12:39.878592 12564 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:39.877691 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:39.874778 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:39.874985 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:39.884325 12563 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:39.884536 12639 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:39.847752 15638 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:39.842716 15363 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:39.894049 13261 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:39.839831 15628 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:12:39.839736 15622 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.867444 15437 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:39.887897 15654 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.348s	user 0.000s	sys 0.001s
I20260709 12:12:39.914934 15679 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:12:39.916115 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.914486 12644 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:39.887004 15638 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:39.841966 15455 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.916879 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 458) took 1108 ms. Trace:
W20260709 12:12:39.921125 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52790.2us
I20260709 12:12:39.866900 15564 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:12:39.865166 15502 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:12:39.933000 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:39.940354 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.862403 15542 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 12:12:39.927878 12566 rpcz_store.cc:276] 0709 12:12:38.808872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:38.811129 (+  2257us) service_pool.cc:224] Handling call
0709 12:12:38.811554 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:38.831851 (+ 20297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:38.831902 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:38.832753 (+   851us) log.cc:844] Serialized 26 byte log entry
0709 12:12:38.972059 (+139306us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:38.972127 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:38.972232 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:39.916471 (+944239us) raft_consensus.cc:1685] finished
0709 12:12:39.916504 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:39.916792 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19630,"raft.queue_time_us":22482,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":222,"thread_start_us":138668,"threads_started":1,"wal-append.queue_time_us":530293}
W20260709 12:12:39.947762 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 123572us
I20260709 12:12:39.880720 15480 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:39.948611 15408 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:39.882946 15560 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:39.950551 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2153) took 1146 ms. Trace:
I20260709 12:12:39.961186 12564 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:12:39.931043 15638 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:39.963958 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:39.962378 12570 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:39.947597 15527 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:39.954423 12489 rpcz_store.cc:276] 0709 12:12:38.804533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:38.804936 (+   403us) service_pool.cc:224] Handling call
0709 12:12:38.805536 (+   600us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:12:38.984594 (+179058us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:38.984646 (+    52us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:12:39.061773 (+ 77127us) log.cc:844] Serialized 584 byte log entry
0709 12:12:39.349553 (+287780us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:39.349615 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:39.350449 (+   834us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:39.950247 (+599798us) raft_consensus.cc:1685] finished
0709 12:12:39.950267 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:39.950461 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10828,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":7355,"thread_start_us":248266,"threads_started":1,"wal-append.queue_time_us":474475}
W20260709 12:12:39.977386 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86583us
I20260709 12:12:39.975904 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:39.979629 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:39.981068 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.954054 15552 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:39.987280 12569 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:39.984071 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:39.957015 15584 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:39.998047 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107890us
I20260709 12:12:39.931346 15437 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.006960 12646 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:39.984968 15409 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:39.992964 15474 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:12:40.017828 12641 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:40.018597 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:39.998422 15552 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.014189 15666 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:40.025183 15686 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:12:40.025499 15409 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:40.015672 15656 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:40.026553 12646 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:12:40.030082 15502 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:39.997452 15408 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:40.024981 15463 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:40.052842 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52135.9us
I20260709 12:12:39.996924 15363 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.053673 15397 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.074128 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:40.065471 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1870) took 1040 ms. Trace:
I20260709 12:12:40.063174 15502 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:40.065737 15681 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:40.136184 15681 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:40.139747 15687 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:40.058164 15409 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:40.021204 15584 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.076546 12572 rpcz_store.cc:276] 0709 12:12:38.979515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:38.979664 (+   149us) service_pool.cc:224] Handling call
0709 12:12:39.054547 (+ 74883us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:39.163493 (+108946us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:39.668365 (+504872us) log.cc:844] Serialized 14 byte log entry
0709 12:12:39.879350 (+210985us) log.cc:844] Serialized 14 byte log entry
0709 12:12:40.019590 (+140240us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:40.019642 (+    52us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:40.019686 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:40.019755 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:40.019868 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":639035,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":122,"raft.queue_time_us":167872,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":5874,"thread_start_us":641079,"threads_started":3,"wal-append.queue_time_us":228106}
W20260709 12:12:40.082630 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 153729us
I20260709 12:12:40.066613 15666 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:40.105291 15502 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:40.105921 15656 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:40.075860 15575 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.139804 15681 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:40.204663 15678 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.141669 15685 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:40.162386 15679 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:40.209374 15437 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:40.212054 15675 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:12:40.213836 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2164) took 1186 ms. Trace:
I20260709 12:12:40.220011 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:40.220824 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:40.222074 15635 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:40.224442 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 388) took 1172 ms. Trace:
W20260709 12:12:40.225528 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:40.226887 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 247364us
I20260709 12:12:40.247798 12496 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:40.264599 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:40.205998 15656 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:40.245352 12494 rpcz_store.cc:276] 0709 12:12:39.027516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.027718 (+   202us) service_pool.cc:224] Handling call
0709 12:12:39.045169 (+ 17451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:39.143239 (+ 98070us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:39.143273 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:39.143317 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:39.143371 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:40.213549 (+1070178us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:40.213597 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:40.213761 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":224904064}
I20260709 12:12:40.254799 13256 rpcz_store.cc:276] 0709 12:12:39.052682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.089600 (+ 36918us) service_pool.cc:224] Handling call
0709 12:12:40.224164 (+1134564us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:40.224332 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1271829760}
I20260709 12:12:40.313316 12490 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:40.310822 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:40.322377 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:40.386144 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64821.8us
W20260709 12:12:40.447736 12563 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.558s	user 0.007s	sys 0.003s
I20260709 12:12:40.370684 12495 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260709 12:12:40.408663 15483 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.527s	user 0.005s	sys 0.005s
I20260709 12:12:40.476343 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2023) took 1001 ms. Trace:
I20260709 12:12:40.396363 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:40.379465 13260 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:12:40.471359 12563 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
W20260709 12:12:40.350766 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 251461us
W20260709 12:12:40.444222 12639 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.000s
I20260709 12:12:40.495937 12639 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:12:40.421573 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 470716us
I20260709 12:12:40.475487 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:40.411123 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1771) took 1156 ms. Trace:
I20260709 12:12:40.497514 13258 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:40.318939 15656 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:40.481478 12566 rpcz_store.cc:276] 0709 12:12:39.475268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.956500 (+481232us) service_pool.cc:224] Handling call
0709 12:12:39.974734 (+ 18234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:39.996690 (+ 21956us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:39.997083 (+   393us) log.cc:844] Serialized 14 byte log entry
0709 12:12:40.475931 (+478848us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:40.476000 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:40.476051 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:40.476129 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:40.476277 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3891,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":53,"raft.queue_time_us":318805,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":440434,"threads_started":2}
W20260709 12:12:40.377815 15690 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:12:40.479777 15483 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.318948 15221 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:12:40.534722 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:40.377964 15669 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.364s	user 0.002s	sys 0.000s
I20260709 12:12:40.554093 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.552093 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:40.554921 12567 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:40.552552 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.311265 15686 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:40.473567 15437 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.384604 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:40.396096 15688 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:40.378517 15463 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.548480 13258 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:40.541742 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 486495us
I20260709 12:12:40.549710 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:40.549901 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.550114 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:40.551011 12571 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:12:40.554376 15659 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:40.460580 15687 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20260709 12:12:40.245229 15678 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.540753 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.541213 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:12:40.378819 15696 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:40.497426 12640 rpcz_store.cc:276] 0709 12:12:39.254919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.257745 (+  2826us) service_pool.cc:224] Handling call
0709 12:12:39.258154 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:39.366878 (+108724us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:39.366911 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:39.436086 (+ 69175us) log.cc:844] Serialized 26 byte log entry
0709 12:12:39.825615 (+389529us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:39.826237 (+   622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:39.826323 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:40.410525 (+584202us) raft_consensus.cc:1685] finished
0709 12:12:40.410559 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:40.410957 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":292790,"raft.queue_time_us":108435,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":25344,"thread_start_us":185381,"threads_started":1,"wal-append.queue_time_us":481066}
W20260709 12:12:40.542460 15445 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.716s	user 0.008s	sys 0.005s
I20260709 12:12:40.538264 15679 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:40.579239 15679 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.556898 12567 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:40.581179 15678 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:40.559864 15520 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.995s	user 0.006s	sys 0.007s
I20260709 12:12:40.575709 15689 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.579665 15659 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.580443 15686 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.528863 15691 raft_consensus.cc:2764] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:12:40.408386 15502 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.577427 12496 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:40.586596 12495 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:12:40.537608 15409 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.561342 15397 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.562306 15666 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.615926 15409 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:40.618769 15502 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:40.619812 12488 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:40.579239 15696 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.624665 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1770) took 1370 ms. Trace:
I20260709 12:12:40.625134 12644 rpcz_store.cc:276] 0709 12:12:39.254615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.255869 (+  1254us) service_pool.cc:224] Handling call
0709 12:12:39.256209 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:39.916891 (+660682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:40.624353 (+707462us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:40.624590 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312605056}
I20260709 12:12:40.589273 15445 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.626133 15699 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 12:12:40.542404 15685 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260709 12:12:40.628675 15445 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:40.607668 15659 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.571825 15437 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.596668 15678 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.600409 15474 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:40.583374 15221 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:40.647112 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 368) took 1705 ms. Trace:
W20260709 12:12:40.663928 15408 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.586s	user 0.011s	sys 0.000s
I20260709 12:12:40.581605 15679 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.583482 15688 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:40.666844 12491 rpcz_store.cc:276] 0709 12:12:38.942213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:38.942463 (+   250us) service_pool.cc:224] Handling call
0709 12:12:38.952514 (+ 10051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:39.753413 (+800899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:40.646759 (+893346us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:40.647028 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1690125696}
I20260709 12:12:40.646040 15533 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:40.638062 15584 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:40.637581 15659 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.595786 15520 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.644564 15437 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:40.676263 15620 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.240s	user 0.012s	sys 0.005s
I20260709 12:12:40.667201 15685 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:40.625556 15696 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.674191 15408 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:40.643388 15678 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:40.706578 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118731us
I20260709 12:12:40.605830 15689 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.686932 15533 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:40.694854 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.838s (SENT) [suppressed 10 similar messages]
I20260709 12:12:40.687788 15397 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:12:40.684775 15688 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.708189 15699 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:12:40.731958 15642 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.969s	user 0.017s	sys 0.000s
W20260709 12:12:40.755604 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:40.758245 15699 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:40.759586 15699 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.698194 15620 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:40.766601 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48910.3us
W20260709 12:12:40.787040 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67729.7us
I20260709 12:12:40.716310 15685 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.794931 15699 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:12:40.803005 15704 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.125s	user 0.002s	sys 0.000s
W20260709 12:12:40.786883 15551 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.083s	user 0.008s	sys 0.008s
I20260709 12:12:40.811941 15542 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:40.818859 15542 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.821171 15542 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:40.821992 15408 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.824223 15408 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.825783 15700 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:40.826208 15700 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:40.826406 15678 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.827006 15408 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.765991 15642 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.828114 15620 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:40.830091 15642 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.830914 12569 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
W20260709 12:12:40.822484 15706 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:12:40.810093 15681 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.791343 15483 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:40.838069 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143251us
W20260709 12:12:40.830145 15638 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.831s	user 0.004s	sys 0.010s
I20260709 12:12:40.785804 15533 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:40.842180 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49407.4us
I20260709 12:12:40.863130 13262 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:40.842736 15622 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:40.845285 15680 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.780s	user 0.003s	sys 0.000s
I20260709 12:12:40.770092 15221 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.846320 15615 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:40.847390 15681 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:40.848518 15520 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:40.849018 15684 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:40.851879 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 384) took 1802 ms. Trace:
I20260709 12:12:40.879050 15684 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:40.879240 13261 rpcz_store.cc:276] 0709 12:12:39.049925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.061126 (+ 11201us) service_pool.cc:224] Handling call
0709 12:12:40.851638 (+1790512us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54005e1354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:40.851810 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1823454464}
W20260709 12:12:40.852267 15635 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.002s
I20260709 12:12:40.854312 15685 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.880262 15684 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.827020 15700 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.863299 15638 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.857518 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:40.858825 15628 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.919s	user 0.002s	sys 0.040s
I20260709 12:12:40.815522 15551 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.857882 15483 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:12:40.878284 15689 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.878611 12568 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:40.891669 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:40.910362 15520 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:40.910622 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.911275 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:40.912165 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.912204 15520 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:12:40.912832 12568 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:40.914278 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:40.915513 15520 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:40.875825 15542 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:40.899164 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:40.898540 15628 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.002s	sys 0.041s
I20260709 12:12:40.904630 15551 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:40.881608 15684 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:40.905903 15635 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:12:40.897322 12570 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:40.877667 15622 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:40.884408 15638 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:40.917766 15685 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:40.949204 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.915901 15520 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:40.882400 15700 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:40.904881 15703 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:40.941831 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:40.971274 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:40.944012 12638 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:12:40.899508 15533 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:40.919564 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.946573 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:40.935938 15706 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:12:40.949860 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:40.951150 13261 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:12:40.939337 15587 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:40.930791 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:40.924582 15620 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:12:40.979259 12571 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:40.958066 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:40.956040 15622 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.948369 15542 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.953401 15699 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 12:12:40.961839 12645 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:12:40.967396 15707 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:40.970146 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:40.970487 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.989020 12644 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:40.972093 15656 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:40.974428 12638 pending_rounds.cc:85] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 4
W20260709 12:12:40.993484 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47020.9us
I20260709 12:12:40.974851 15703 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:40.979862 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:40.996006 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:40.969930 12570 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:40.997485 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.986280 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:40.993408 12638 pending_rounds.cc:107] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:12:40.974005 12643 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:40.998768 12638 raft_consensus.cc:2889] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:40.999915 12570 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.000125 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.989506 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:40.976580 15520 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.011564 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.048228 15527 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:41.049925 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.051414 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.052170 12642 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:41.053157 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.053706 15707 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:12:41.054024 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.054392 15707 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:12:41.055629 15551 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:41.056355 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.057380 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.057806 15442 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:41.058871 15657 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.630s	user 0.003s	sys 0.007s
I20260709 12:12:41.053696 15560 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:41.062031 15703 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.062398 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.063735 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.064342 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:41.064946 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.066584 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.014919 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.049469 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.047465 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:41.065313 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.073652 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.073817 12642 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:41.074940 12572 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.075947 12643 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:41.073392 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45538.8us
W20260709 12:12:41.085515 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73275.5us
I20260709 12:12:41.100493 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:41.011449 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.048730 15527 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:41.002566 15403 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:41.137385 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:40.996767 15397 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:41.137210 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:41.141036 12570 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.027611 15549 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.142s
I20260709 12:12:41.151865 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.152830 12571 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.149761 15403 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.154491 15403 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:41.154870 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:41.157346 15681 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:41.153820 15718 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:41.157831 15681 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:41.158100 15718 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.158310 15681 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:41.158632 15718 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:40.993067 15587 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.144279 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:41.162336 15587 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:41.162400 15397 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.163712 15397 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:41.163815 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:40.988849 15685 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.153s
I20260709 12:12:41.166891 15685 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:41.166937 15622 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:41.167402 15685 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.150671 15583 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.167822 15685 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.168223 15583 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.142637 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.169934 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.170778 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.171482 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.171916 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.172335 12571 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:41.173003 15408 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:41.173482 15408 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.173887 12568 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:41.173978 15408 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:41.174993 15656 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:41.175263 12568 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:12:41.176195 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.176509 15679 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:41.177393 15679 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.177907 15679 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.178387 15542 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:41.180807 15638 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:41.181393 15638 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.181803 15638 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:41.176622 12568 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:12:41.165438 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.137409 15684 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:12:41.142390 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.186261 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:41.136420 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.185527 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.190789 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:41.173273 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:41.195698 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51551.8us
I20260709 12:12:41.199541 12571 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.203092 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.204046 12571 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.206648 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:41.115123 15363 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:41.206082 15480 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:41.210762 15480 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:41.210999 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:41.211760 12488 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:41.211865 15480 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:41.214324 15455 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:41.208277 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:41.210772 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:41.217026 12494 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:41.209379 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:41.218695 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.197978 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.220412 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.220788 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:41.221655 12571 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:12:41.221966 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.219703 12568 raft_consensus.cc:2376] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7. Re-sending same reply.
I20260709 12:12:41.224810 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.202343 15587 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:41.203383 15635 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:41.230224 12571 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:12:41.230209 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.142778 15551 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 12:12:41.207994 15684 consensus_queue.cc:1243] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:12:41.215431 15659 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:41.250955 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.252480 12644 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:12:41.253376 12564 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.254541 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.255481 12568 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6. Re-sending same reply.
I20260709 12:12:41.257330 12637 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:41.256647 15684 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:12:41.262238 15698 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:41.262665 12571 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:41.262771 15698 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:41.263105 12571 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:41.253010 12644 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 4
I20260709 12:12:41.248044 15635 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.276646 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.272951 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.251385 15659 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:41.223942 15363 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:41.298209 12637 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:41.293354 12564 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.263924 12571 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.308044 12571 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:41.269311 12644 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.5
I20260709 12:12:41.309942 12570 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:41.297724 12568 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:41.313548 12644 raft_consensus.cc:2917] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Aborting config change with OpId 5.5: Aborted: Op aborted by new leader
I20260709 12:12:41.320585 15637 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:41.321177 15637 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:41.322324 15637 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.324523 15637 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:41.326620 12644 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:41.291307 15635 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:41.331985 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:41.257031 15397 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.334798 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:41.301915 15659 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:41.338239 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:41.332132 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.263792 15698 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:41.349907 12571 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:41.352912 12638 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:41.361311 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:41.362741 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:41.364665 12563 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:41.366921 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.368901 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:12:41.375677 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2159) took 2367 ms (client timeout 1698 ms). Trace:
I20260709 12:12:41.351274 15575 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.376392 12565 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 5. Current term is 6. Ops: [4.4-5.5]
I20260709 12:12:41.376133 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:41.378428 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1884) took 1990 ms. Trace:
I20260709 12:12:41.378769 12565 rpcz_store.cc:276] 0709 12:12:39.388431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.830974 (+442543us) service_pool.cc:224] Handling call
0709 12:12:41.369072 (+1538098us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:41.369376 (+   304us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:41.378106 (+  8730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:41.378370 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6216448}
I20260709 12:12:41.380532 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 506) took 2054 ms. Trace:
I20260709 12:12:41.364414 15666 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:41.381042 13262 rpcz_store.cc:276] 0709 12:12:39.320924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.321187 (+   263us) service_pool.cc:224] Handling call
0709 12:12:40.862909 (+1541722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:40.863692 (+   783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:41.375354 (+511662us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54005e1350. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:41.375554 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3471710464}
W20260709 12:12:41.376217 12497 rpcz_store.cc:269] 0709 12:12:38.999711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.007805 (+  8094us) service_pool.cc:224] Handling call
0709 12:12:41.366727 (+2358922us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:41.366859 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1876419968}
I20260709 12:12:41.384065 15667 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:12:41.384446 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:41.404844 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:41.405249 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:41.380774 15533 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.349685 15620 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:12:41.371621 15345 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.386531 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:41.350632 15718 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:41.411163 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 202005us
I20260709 12:12:41.374928 15221 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.377013 15698 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.367270 15400 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:12:41.364778 15455 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:12:41.385152 15702 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.374527 15520 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.396278 12569 raft_consensus.cc:1217] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-7.4]   Dedup: 3.2->[5.3-7.4]
I20260709 12:12:41.374972 15542 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:12:41.455235 12640 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:41.458700 12638 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:12:41.401463 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 215157us
I20260709 12:12:41.385721 15622 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:41.412825 15703 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.411784 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.415077 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.405823 15666 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.403268 15583 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.403870 15575 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.397194 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.477432 12643 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:12:41.452456 12569 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:12:41.477960 12569 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:12:41.478283 12569 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:41.412132 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.473559 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:41.479840 12644 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:12:41.413671 15667 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:41.450402 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:41.473084 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58104.7us
I20260709 12:12:41.484850 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:41.400717 15638 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.462724 15542 consensus_queue.cc:1243] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:12:41.486526 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.481742 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:41.412467 15533 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:41.474666 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88237us
I20260709 12:12:41.500746 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:41.457075 15587 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:12:41.492580 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:41.470227 15666 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:41.503865 12644 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: [6.5-9.6]
I20260709 12:12:41.462361 15622 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:41.519285 15622 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:41.468906 15560 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:41.526225 15560 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:41.527691 12536 connection.cc:256] RPC call timeout handler was delayed by -3.852408559999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 12:12:41.530536 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:41.474771 15615 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:41.534950 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:41.532835 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48151.2us
W20260709 12:12:41.533865 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52243.4us
I20260709 12:12:41.537007 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:41.537286 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:41.490736 15583 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:41.543845 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:41.461577 15702 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.450745 15221 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.544651 12645 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:41.477746 15551 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:12:41.529160 15455 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:41.570720 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.476537 15703 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:41.534981 15615 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:41.543951 15696 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.541443 15678 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:41.557731 15684 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:41.516604 15403 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:41.637243 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.570698 15455 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:41.636099 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135426us
I20260709 12:12:41.583043 15615 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:41.647553 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103797us
I20260709 12:12:41.583859 15363 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:41.648057 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:41.597839 15397 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.682966 12642 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:12:41.686620 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:41.654963 15724 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.352s	user 0.000s	sys 0.002s
W20260709 12:12:41.704680 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 158680us
I20260709 12:12:41.654335 15455 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:41.654662 15684 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:41.709056 12638 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:41.683686 15678 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:12:41.714468 12536 connection.cc:256] RPC call timeout handler was delayed by -3.5884009709998281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
W20260709 12:12:41.715409 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 265106us
W20260709 12:12:41.748118 12536 connection.cc:256] RPC call timeout handler was delayed by -3.4062315969999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260709 12:12:41.752048 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3979085599999053s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
I20260709 12:12:41.719936 15520 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:41.724747 15628 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:41.753719 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3925085599998965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
I20260709 12:12:41.684171 15667 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:41.915320 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:41.851895 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:41.936976 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3601085599998441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
I20260709 12:12:41.749545 15587 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:41.918237 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:41.939504 13261 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:41.939918 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2171105689999422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24220000000000003s for a response
W20260709 12:12:41.947629 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7500105689998691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2941s for a response
W20260709 12:12:41.761188 12462 connection.cc:256] RPC call timeout handler was delayed by -1.4390044280000893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
I20260709 12:12:41.940435 13259 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:12:41.940889 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:41.942564 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107317us
I20260709 12:12:41.817848 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.949041 12564 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:41.954021 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7284105689998341s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2965s for a response
W20260709 12:12:41.954558 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4071570059998066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16080000000000003s for a response
W20260709 12:12:41.954849 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4008570059997965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1615s for a response
W20260709 12:12:41.955144 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2821748660001049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32200000000000006s for a response
W20260709 12:12:41.955740 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1902570059999107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
W20260709 12:12:41.956171 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9040570059999027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
W20260709 12:12:41.899286 15743 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:12:41.747498 15678 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:42.003137 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1904) took 1028 ms. Trace:
I20260709 12:12:42.003901 15678 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.003630 12570 rpcz_store.cc:276] 0709 12:12:40.975344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.319811 (+344467us) service_pool.cc:224] Handling call
0709 12:12:41.320240 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:41.383441 (+ 63201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:41.383489 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:41.398095 (+ 14606us) log.cc:844] Serialized 26 byte log entry
0709 12:12:41.483165 (+ 85070us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:12:41.483214 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:41.483294 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:42.002841 (+519547us) raft_consensus.cc:1685] finished
0709 12:12:42.002865 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:42.003062 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32056,"raft.queue_time_us":41684,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":83509,"threads_started":1,"wal-append.queue_time_us":355018}
W20260709 12:12:42.006408 15738 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:12:42.006588 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:41.844630 15403 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:41.755300 15718 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.820606 15684 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.861505 15622 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.019421 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1889) took 1145 ms. Trace:
I20260709 12:12:42.020170 15622 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:41.959492 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:41.960362 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:41.962122 15737 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:12:41.961925 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:41.800737 15533 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:41.961213 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:41.962953 15742 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:12:41.949946 15628 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.059273 13259 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:12:41.968288 15720 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.894s	user 0.004s	sys 0.000s
I20260709 12:12:41.970328 15587 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:41.971428 15681 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.813s	user 0.009s	sys 0.004s
I20260709 12:12:41.972041 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:41.973284 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:42.068957 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:41.980176 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.981043 15700 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.925814 15702 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.985806 15758 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:41.986585 15750 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:12:41.938239 15221 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.990322 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:41.800100 15638 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.993932 15463 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:41.994648 15397 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:41.997483 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:42.094103 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:42.095165 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:41.998371 15751 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:12:42.096951 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:41.998927 15746 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260709 12:12:42.000519 15753 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:42.098943 15753 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:41.893808 15703 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:41.868453 15752 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.101378 15752 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.102941 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:42.104194 15752 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.104373 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:42.104924 15752 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:41.914240 15659 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.000s
I20260709 12:12:42.105933 15659 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.108546 15659 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:41.949215 15583 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:42.111109 15583 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:42.007596 15739 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:12:42.112298 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 480221us
I20260709 12:12:42.114151 12489 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:42.114840 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:42.116569 12644 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:42.117115 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1890) took 1243 ms. Trace:
I20260709 12:12:41.999207 13260 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
W20260709 12:12:41.956629 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8482570059998125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22290000000000001s for a response
W20260709 12:12:42.117977 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7573570059998929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.233s for a response
I20260709 12:12:42.008769 15718 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:42.021395 13253 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:42.022758 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:42.023331 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1789) took 1025 ms. Trace:
I20260709 12:12:42.019901 12563 rpcz_store.cc:276] 0709 12:12:40.873830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.064596 (+190766us) service_pool.cc:224] Handling call
0709 12:12:42.019199 (+954603us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:42.019352 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":686818688}
I20260709 12:12:42.030508 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:41.797189 15483 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:42.024926 13257 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
W20260709 12:12:41.976832 15345 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.550s	user 0.007s	sys 0.002s
I20260709 12:12:42.072109 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:42.084275 15760 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:42.085112 15734 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:42.085294 15679 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.907s	user 0.013s	sys 0.002s
I20260709 12:12:42.087082 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.089887 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:42.070946 15698 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.002s
I20260709 12:12:42.090049 12568 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:42.118167 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:42.134771 15551 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.138167 12570 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:12:42.139067 12645 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:42.140064 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:42.141036 12568 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:42.141902 15437 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:42.126760 15583 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:42.142781 15345 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:42.143206 15583 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.133944 15768 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:42.144094 15583 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.145066 15768 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.145699 15679 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.148322 15583 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.148500 15679 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.149746 12496 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:42.126169 15753 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.152539 15483 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:42.139175 15551 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.155371 15483 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.155594 15551 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:42.156746 15718 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.105562 15752 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.157933 15437 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:42.159400 15667 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:42.159837 12490 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:42.160388 12494 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:42.159437 15718 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.159865 15667 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:42.159833 15752 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.161849 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:42.161720 12640 rpcz_store.cc:276] 0709 12:12:40.998510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:40.998936 (+   426us) service_pool.cc:224] Handling call
0709 12:12:42.023120 (+1024184us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:42.023261 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":930030848}
I20260709 12:12:42.162088 15667 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.163873 15667 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:42.165066 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.166106 12488 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:12:42.141947 15734 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.167647 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.168229 15734 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.160073 15760 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.117504 12571 rpcz_store.cc:276] 0709 12:12:40.874126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.246829 (+372703us) service_pool.cc:224] Handling call
0709 12:12:41.247259 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:41.353094 (+105835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:42.116876 (+763782us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:42.117052 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34965888}
I20260709 12:12:42.170333 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.171386 12571 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:12:42.171849 15221 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:42.172454 15221 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.127240 15552 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:42.174824 15221 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.176482 12641 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:42.118346 12536 connection.cc:256] RPC call timeout handler was delayed by -1.60255700599987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2502s for a response
W20260709 12:12:42.177953 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5404570059997695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25710000000000005s for a response
W20260709 12:12:42.178325 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4054570059997786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2721s for a response
W20260709 12:12:42.178637 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2614570059997732s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28809999999999997s for a response
I20260709 12:12:42.179963 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:42.182453 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.184545 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:42.190244 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.192687 13256 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c56dde469374f4da7440f134140670c"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:12:42.192826 15473 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:12:42.194538 13253 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:12:42.200642 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:42.201560 13253 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:42.202152 15628 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.203708 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:42.204353 13260 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 12:12:42.204617 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:42.213482 12568 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:12:42.214146 12568 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:12:42.214565 12568 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:12:42.215473 12568 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:42.213510 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:42.216704 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:42.205302 13256 raft_consensus.cc:606] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:42.217712 12564 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:42.237780 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 200565us
W20260709 12:12:42.234530 15666 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.694s	user 0.009s	sys 0.005s
I20260709 12:12:42.240520 15666 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.242928 13253 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8.
I20260709 12:12:42.253952 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1796) took 1204 ms. Trace:
I20260709 12:12:42.254251 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1891) took 1379 ms. Trace:
W20260709 12:12:42.257077 12638 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.001s
I20260709 12:12:42.254974 12572 rpcz_store.cc:276] 0709 12:12:40.874424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.247694 (+373270us) service_pool.cc:224] Handling call
0709 12:12:41.258804 (+ 11110us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:41.293292 (+ 34488us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:41.294431 (+  1139us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:41.312405 (+ 17974us) log.cc:844] Serialized 26 byte log entry
0709 12:12:41.482662 (+170257us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:41.482713 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:41.482791 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:42.253887 (+771096us) raft_consensus.cc:1685] finished
0709 12:12:42.253919 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:42.254149 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4167,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":3148,"thread_start_us":151025,"threads_started":1,"wal-append.queue_time_us":458319}
I20260709 12:12:42.258194 12638 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:42.080885 15681 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:42.218216 15637 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.893s	user 0.018s	sys 0.000s
I20260709 12:12:42.214807 15689 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:42.088375 15700 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.093374 15702 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.144s
I20260709 12:12:42.184262 15463 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.092849 15758 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.260790 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.376816 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:42.643666 12490 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:42.274307 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 543413us
I20260709 12:12:42.411742 12494 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:42.254586 12646 rpcz_store.cc:276] 0709 12:12:41.049153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.055043 (+  5890us) service_pool.cc:224] Handling call
0709 12:12:41.055339 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:41.055545 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:12:41.361063 (+305518us) log.cc:844] Serialized 14 byte log entry
0709 12:12:41.535389 (+174326us) log.cc:844] Serialized 14 byte log entry
0709 12:12:42.253193 (+717804us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b1800143c80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:12:42.253420 (+   227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:42.253495 (+    75us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:42.253550 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:42.253685 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:42.253864 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46159,"prepare.queue_time_us":514594,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":7182,"raft.queue_time_us":74707,"raft.run_wall_time_us":6861,"spinlock_wait_cycles":43745920,"thread_start_us":438564,"threads_started":2,"wal-append.queue_time_us":484189}
I20260709 12:12:42.644831 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:42.219197 15221 raft_consensus.cc:2764] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260709 12:12:42.148525 15768 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.261233 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.735395 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:42.734963 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1825) took 1210 ms. Trace:
I20260709 12:12:42.288409 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.285156 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:42.744980 12492 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:12:42.346243 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:42.339176 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:42.590638 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:42.404729 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:42.744424 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 396713us
I20260709 12:12:42.275601 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2060) took 1045 ms. Trace:
W20260709 12:12:42.691175 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46437.5us
I20260709 12:12:42.275553 12493 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:42.317503 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:42.261641 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 613759us
I20260709 12:12:42.551600 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.482669 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 342) took 1070 ms. Trace:
I20260709 12:12:42.750422 12495 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:12:42.311480 12488 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:42.732718 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.747346 12565 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
W20260709 12:12:42.335639 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 417793us
I20260709 12:12:42.489614 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 471) took 1080 ms. Trace:
I20260709 12:12:42.748508 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:42.490111 15455 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.758s	user 0.009s	sys 0.004s
I20260709 12:12:42.745784 12645 rpcz_store.cc:276] 0709 12:12:41.524988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.030071 (+505083us) service_pool.cc:224] Handling call
0709 12:12:42.734889 (+704818us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:42.757617 12568 rpcz_store.cc:276] 0709 12:12:41.409137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.162893 (+753756us) service_pool.cc:224] Handling call
0709 12:12:42.163278 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:42.244779 (+ 81501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:42.489327 (+244548us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:42.489534 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":114573184}
W20260709 12:12:42.760030 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:42.758800 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:42.760560 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:42.760934 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:42.238404 15685 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.070s	user 0.017s	sys 0.000s
W20260709 12:12:42.390676 15635 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.037s	user 0.015s	sys 0.000s
I20260709 12:12:42.747956 12567 rpcz_store.cc:276] 0709 12:12:41.230748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.393931 (+163183us) service_pool.cc:224] Handling call
0709 12:12:41.394359 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:41.394842 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:41.394871 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:41.395472 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:12:41.537928 (+142456us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:41.537970 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:41.538046 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:42.275255 (+737209us) raft_consensus.cc:1685] finished
0709 12:12:42.275278 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:42.275505 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1492,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":142018,"threads_started":1,"wal-append.queue_time_us":331680}
W20260709 12:12:42.321589 15720 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:12:42.764554 12572 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 3: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:42.219616 15397 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:42.406162 15408 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.232s	user 0.014s	sys 0.005s
W20260709 12:12:42.687757 15730 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.053s	user 0.000s	sys 0.012s
W20260709 12:12:42.528461 15678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.010s	sys 0.022s
I20260709 12:12:42.773289 12569 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:42.773546 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.261046 15564 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:42.540146 15549 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:42.657379 15735 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.022s	user 0.000s	sys 0.007s
I20260709 12:12:42.366869 15760 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.684165 15552 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.750566 12637 rpcz_store.cc:276] 0709 12:12:41.411920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.412632 (+   712us) service_pool.cc:224] Handling call
0709 12:12:41.449461 (+ 36829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:41.449720 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:12:41.541735 (+ 92015us) log.cc:844] Serialized 14 byte log entry
0709 12:12:42.482045 (+940310us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:42.482138 (+    93us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:42.482209 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:42.482336 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:42.482564 (+   228us) inbound_call.cc:177] Queueing success response
0709 12:12:42.677381 (+194817us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
Metrics: {"mutex_wait_us":710129,"prepare.queue_time_us":82107,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":10065,"raft.queue_time_us":730846,"raft.run_cpu_time_us":3344,"raft.run_wall_time_us":389400,"spinlock_wait_cycles":23322880,"thread_start_us":319947,"threads_started":2}
I20260709 12:12:42.655900 15703 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.398789 15442 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.288858 15400 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:42.783245 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.643187 15689 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:42.672851 15749 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:42.786221 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 195735us
I20260709 12:12:42.837047 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:42.838680 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59257.3us
I20260709 12:12:42.770305 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.646575 15560 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:42.257812 15698 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.522s
W20260709 12:12:42.845278 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65193.8us
W20260709 12:12:42.845885 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62597.1us
I20260709 12:12:42.777098 15637 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:42.855454 12495 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 3.
I20260709 12:12:42.857038 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.858083 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:42.859859 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 574936us
I20260709 12:12:42.690302 15575 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:42.338404 15520 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.504s
I20260709 12:12:42.885068 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:43.093096 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:12:42.844007 15678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.012s	sys 0.025s
I20260709 12:12:42.954959 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:42.954300 12492 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:42.841050 15770 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.840425 15408 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.847440 15363 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.181s
I20260709 12:12:42.894898 15667 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.336472 15527 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.774550 15455 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.864437 15691 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:42.785391 15768 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.123s
I20260709 12:12:42.742389 15769 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:42.860036 15549 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:42.963306 15781 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.842052 15533 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260709 12:12:43.145231 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:43.098385 12490 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:43.101320 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:42.864179 15702 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:43.103482 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:43.104434 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:43.104597 15560 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:43.104976 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:43.106062 12567 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:43.222206 12565 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:43.015489 15552 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:42.964423 15703 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:42.777192 15685 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.099423 12568 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:43.111555 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:43.266039 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1918) took 1040 ms. Trace:
I20260709 12:12:42.958395 12488 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
W20260709 12:12:43.021965 15696 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.391s	user 0.014s	sys 0.002s
I20260709 12:12:42.779364 15635 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.125310 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:42.966203 15689 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.211689 15527 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:43.137732 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:43.278374 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:43.203954 12569 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:42.964092 15442 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:43.166626 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:43.284263 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:43.284843 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:43.285234 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:43.210481 12492 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:43.131278 15678 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:42.967693 15707 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.197s
I20260709 12:12:43.212244 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:42.881989 15473 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.237s
I20260709 12:12:43.223470 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:43.294387 12565 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.097412 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2038) took 2485 ms. Trace:
W20260709 12:12:43.227088 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.039s	sys 0.058s
I20260709 12:12:43.138428 15769 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:43.230670 12489 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8.
I20260709 12:12:43.266120 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1920) took 1039 ms. Trace:
I20260709 12:12:43.298385 12568 rpcz_store.cc:276] 0709 12:12:42.226841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.760783 (+533942us) service_pool.cc:224] Handling call
0709 12:12:43.266025 (+505242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.266518 12572 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:43.266631 12567 rpcz_store.cc:276] 0709 12:12:42.225284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.859657 (+634373us) service_pool.cc:224] Handling call
0709 12:12:43.265943 (+406286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.271261 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:43.234026 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 137509us
W20260709 12:12:43.218143 15622 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.028s	sys 0.074s
W20260709 12:12:43.027136 15730 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:12:43.278002 12493 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:43.106251 15575 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:43.275141 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:43.279747 12490 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:43.235560 15615 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.539s	user 0.015s	sys 0.008s
W20260709 12:12:43.280684 15776 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.407s	user 0.000s	sys 0.001s
W20260709 12:12:43.310942 15622 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.029s	sys 0.074s
I20260709 12:12:43.218600 15637 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:43.311298 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:43.311805 15552 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.275341 15696 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:43.312150 15637 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.138825 15667 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:43.297873 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.040s	sys 0.058s
I20260709 12:12:43.317288 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:43.269397 15560 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:43.295665 12566 rpcz_store.cc:276] 0709 12:12:40.541200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:40.595042 (+ 53842us) service_pool.cc:224] Handling call
0709 12:12:40.608039 (+ 12997us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:40.746238 (+138199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:40.746302 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:40.779924 (+ 33622us) log.cc:844] Serialized 26 byte log entry
0709 12:12:40.866799 (+ 86875us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:40.867137 (+   338us) log.cc:844] Serialized 14 byte log entry
0709 12:12:40.872420 (+  5283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:40.872495 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:41.967547 (+1095052us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:42.946452 (+978905us) raft_consensus.cc:1685] finished
0709 12:12:43.025908 (+ 79456us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:43.026159 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56355,"prepare.queue_time_us":15163,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":81123,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":17739,"spinlock_wait_cycles":17709440,"thread_start_us":49588,"threads_started":1,"wal-append.queue_time_us":147377}
W20260709 12:12:43.313570 15772 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
W20260709 12:12:43.314546 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.005s	sys 0.008s
I20260709 12:12:43.314373 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:43.312029 12492 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.318460 12568 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:12:43.315035 12497 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
W20260709 12:12:43.318710 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.021s	sys 0.021s
I20260709 12:12:43.316555 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:43.324337 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.021s	sys 0.021s
W20260709 12:12:43.326278 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:43.328061 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:43.329093 12488 raft_consensus.cc:2376] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9. Re-sending same reply.
W20260709 12:12:43.323683 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.005s	sys 0.008s
W20260709 12:12:43.333956 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58896us
W20260709 12:12:43.334832 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 569572us
I20260709 12:12:43.336215 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:43.314309 15678 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.346735 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 467) took 1938 ms. Trace:
I20260709 12:12:43.346573 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:43.348114 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1917) took 1139 ms. Trace:
I20260709 12:12:43.349393 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:43.353904 15799 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:43.354846 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:43.355496 15552 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:43.206537 15781 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.122340 15770 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.397943 12488 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:43.402957 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:43.320148 15575 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.473989 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1805) took 1175 ms. Trace:
W20260709 12:12:43.500653 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:43.356840 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 398869us
I20260709 12:12:43.412046 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.497573 12568 rpcz_store.cc:276] 0709 12:12:42.209013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.299817 (+1090804us) service_pool.cc:224] Handling call
0709 12:12:43.348016 (+ 48199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.502187 12640 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:43.440474 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115937us
I20260709 12:12:43.395660 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:43.412961 12644 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260709 12:12:43.418326 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:43.415743 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 477) took 1224 ms. Trace:
I20260709 12:12:43.309067 15769 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:43.399325 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:43.442539 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:43.403862 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.325520 15363 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:43.395722 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:43.472447 12495 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.496554 12646 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:43.349421 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 476) took 1160 ms. Trace:
W20260709 12:12:43.477169 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 384196us
I20260709 12:12:43.503624 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.495546 12563 rpcz_store.cc:276] 0709 12:12:41.408128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.142577 (+734449us) service_pool.cc:224] Handling call
0709 12:12:42.143018 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:42.143592 (+   574us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:42.143619 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:42.144154 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:12:42.395250 (+251096us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:42.395312 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:42.395419 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:43.346444 (+951025us) raft_consensus.cc:1685] finished
0709 12:12:43.346469 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:43.346656 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":250539,"threads_started":1,"wal-append.queue_time_us":802159}
I20260709 12:12:43.553263 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:43.547036 12567 rpcz_store.cc:276] 0709 12:12:42.191752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.354980 (+1163228us) service_pool.cc:224] Handling call
0709 12:12:43.413573 (+ 58593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.415278 (+  1705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:43.415318 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:43.415381 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:43.415427 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.415502 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:43.415660 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.554911 12565 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:43.500078 12488 rpcz_store.cc:276] 0709 12:12:42.298912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.347733 (+1048821us) service_pool.cc:224] Handling call
0709 12:12:43.397701 (+ 49968us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.473613 (+ 75912us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.473905 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:43.473485 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 137385us
I20260709 12:12:43.547629 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:43.547387 12638 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:43.551968 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:43.549998 12637 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.547387 12566 rpcz_store.cc:276] 0709 12:12:42.188011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.319032 (+1131021us) service_pool.cc:224] Handling call
0709 12:12:43.348212 (+ 29180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.348440 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:43.348477 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:43.348534 (+    57us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:43.348579 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.348649 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:43.348788 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.504813 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:43.561685 12490 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 4.
I20260709 12:12:43.548720 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:43.476596 15773 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.257s	user 0.001s	sys 0.000s
W20260709 12:12:43.571660 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 176190us
I20260709 12:12:43.572252 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:43.446939 15777 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.356s	user 0.000s	sys 0.001s
W20260709 12:12:43.567407 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.015s	sys 0.012s
W20260709 12:12:43.446499 15771 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.235s	user 0.001s	sys 0.001s
I20260709 12:12:43.347082 15684 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:43.578899 12488 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:12:43.318112 15615 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.575945 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.431444 15463 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.350025 15400 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:43.346550 15707 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:43.589855 12494 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8.
I20260709 12:12:43.524987 15758 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.589886 12490 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:12:43.574023 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.015s	sys 0.012s
I20260709 12:12:43.572603 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:43.370265 15691 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:43.577741 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.524463 15542 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:43.475322 15442 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:43.409798 15703 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:43.498600 15760 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:43.581727 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:43.586805 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.356554 15787 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:43.574887 15437 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:43.576829 15757 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:43.557542 15702 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:43.395817 15552 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.612706 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:43.652364 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.580893 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 378) took 1119 ms. Trace:
I20260709 12:12:43.706472 12488 rpcz_store.cc:276] 0709 12:12:42.461156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.556557 (+1095401us) service_pool.cc:224] Handling call
0709 12:12:43.578631 (+ 22074us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:43.580570 (+  1939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.580812 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:43.694202 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 569063us
W20260709 12:12:43.654515 15784 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:12:43.709616 12568 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:43.698287 12644 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:43.695911 12492 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:43.516497 15799 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.700590 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.578718 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:43.711444 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 379) took 1190 ms. Trace:
I20260709 12:12:43.581743 15666 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:43.582453 15408 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:43.618933 12495 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:43.695261 15473 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:43.590270 15707 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:43.719700 12492 rpcz_store.cc:276] 0709 12:12:42.521347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.292806 (+771459us) service_pool.cc:224] Handling call
0709 12:12:43.711362 (+418556us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.603241 15788 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:43.716260 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.722278 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:43.713131 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.752125 12495 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:43.746222 12490 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:12:43.613811 15790 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:12:43.748610 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 142719us
I20260709 12:12:43.603237 15455 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:43.752853 12640 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:43.755928 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1913) took 1549 ms. Trace:
W20260709 12:12:43.756376 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 177949us
I20260709 12:12:43.757691 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:12:43.758594 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:43.757145 12565 rpcz_store.cc:276] 0709 12:12:42.206731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.580611 (+1373880us) service_pool.cc:224] Handling call
0709 12:12:43.581455 (+   844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.581725 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:43.606815 (+ 25090us) log.cc:844] Serialized 14 byte log entry
0709 12:12:43.755503 (+148688us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:43.755567 (+    64us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:43.755622 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.755697 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:43.755858 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38048,"prepare.queue_time_us":29697,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":60,"thread_start_us":110122,"threads_started":1}
I20260709 12:12:43.607031 15696 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.131s
I20260709 12:12:43.763309 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.700748 15795 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:43.709815 15684 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:43.766297 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:43.771147 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:43.719033 15794 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:43.769477 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 196961us
I20260709 12:12:43.770522 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:43.770861 12640 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:43.719465 15685 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:43.756399 15667 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:43.738039 15437 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:43.775254 12568 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:43.768581 12565 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 12:12:43.774464 12488 raft_consensus.cc:606] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:43.789378 12488 raft_consensus.cc:614] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:43.757990 15707 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:43.798141 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:43.798475 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:43.787940 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:43.799647 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1915) took 1591 ms (client timeout 2116 ms). Trace:
I20260709 12:12:43.800103 12462 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8.
I20260709 12:12:43.791465 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 479) took 1017 ms. Trace:
I20260709 12:12:43.805383 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:43.805970 12568 rpcz_store.cc:276] 0709 12:12:42.773907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.771424 (+997517us) service_pool.cc:224] Handling call
0709 12:12:43.771813 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.791190 (+ 19377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.791407 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.791841 12565 pending_rounds.cc:85] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:12:43.755692 15757 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:43.772938 15796 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:43.812208 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:43.800501 12572 rpcz_store.cc:269] 0709 12:12:42.207890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.095567 (+887677us) service_pool.cc:224] Handling call
0709 12:12:43.799539 (+703972us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.812835 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:43.802858 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80774.1us
I20260709 12:12:43.807960 12565 pending_rounds.cc:107] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 12:12:43.809109 12568 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:43.744825 15760 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 12:12:43.814154 12565 raft_consensus.cc:2889] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:43.820329 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 480) took 1036 ms. Trace:
I20260709 12:12:43.820757 12565 rpcz_store.cc:276] 0709 12:12:42.784170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.767418 (+983248us) service_pool.cc:224] Handling call
0709 12:12:43.768041 (+   623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.820030 (+ 51989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.820259 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:43.800698 12462 leader_election.cc:403] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:12:43.771634 15622 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:43.849828 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:43.775554 15788 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:43.850319 15622 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.852108 15788 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.854082 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:43.774125 15684 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:43.815687 12644 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:43.815619 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:43.855599 15684 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.780174 15698 raft_consensus.cc:2764] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:12:43.707794 15787 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:43.823096 15575 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.815078 15560 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:43.879624 12646 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:43.783344 15794 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.801893 15781 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.838111 15615 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.778020 15795 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.822708 15455 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:43.821808 15533 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:43.839672 15770 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.843366 15799 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:43.844509 15678 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:43.845135 15527 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.557s	user 0.006s	sys 0.006s
I20260709 12:12:43.815465 15702 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.814527 15400 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:43.846791 12640 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:43.847257 15552 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.857520 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:43.864012 12638 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 12:12:43.870230 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57522.2us
I20260709 12:12:43.970831 15760 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:43.758286 15473 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:43.945230 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:43.802182 15463 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:12:43.948060 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 148130us
I20260709 12:12:43.816483 15748 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:43.860278 15778 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:12:43.857738 15796 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:43.857473 15622 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:43.859180 15397 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:43.978775 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1806) took 1656 ms. Trace:
I20260709 12:12:43.944216 15696 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:12:43.988029 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 169511us
I20260709 12:12:43.992265 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:43.857306 15810 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:43.993040 12491 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
W20260709 12:12:43.993805 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136361us
I20260709 12:12:43.970592 15455 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:43.993705 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 263368us
I20260709 12:12:43.817934 15757 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:43.801414 15437 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.037961 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:44.040027 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:44.037459 12491 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 6
I20260709 12:12:44.040817 12646 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:44.044039 12491 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 12:12:44.045238 12491 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:44.054798 15637 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:44.066506 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74343.6us
I20260709 12:12:44.037983 12493 rpcz_store.cc:276] 0709 12:12:42.322226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.497478 (+1175252us) service_pool.cc:224] Handling call
0709 12:12:43.569685 (+ 72207us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:43.611589 (+ 41904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:43.611641 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:43.655148 (+ 43507us) log.cc:844] Serialized 26 byte log entry
0709 12:12:43.782046 (+126898us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:43.782106 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.782186 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:43.978362 (+196176us) raft_consensus.cc:1685] finished
0709 12:12:43.978401 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:43.978667 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30248,"raft.queue_time_us":48018,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":29742,"thread_start_us":48338,"threads_started":1,"wal-append.queue_time_us":149843}
I20260709 12:12:44.105782 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:43.987267 15678 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:44.107837 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:44.110464 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72589.2us
I20260709 12:12:44.111805 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:44.001863 15533 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:44.115036 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75023.8us
I20260709 12:12:44.116497 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:44.125291 12637 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.554s	user 0.007s	sys 0.000s
I20260709 12:12:43.999214 15760 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:44.115350 15689 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.815s	user 0.010s	sys 0.003s
I20260709 12:12:44.115542 15810 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:44.118739 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74258.3us
I20260709 12:12:44.035158 15527 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8.
I20260709 12:12:44.127851 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:44.127318 15805 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
W20260709 12:12:44.122457 15807 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:12:43.998142 15552 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:44.126701 12637 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:44.127432 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.001513 15748 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:43.998859 15520 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:44.001889 15778 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:44.129597 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:44.116788 15797 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:44.041934 15455 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:43.874831 15691 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.120537 15757 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.125757 15696 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:44.116353 15678 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:44.125538 15622 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:44.163156 15748 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:44.131500 15760 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:44.190414 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 146882us
W20260709 12:12:44.187445 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57909.5us
I20260709 12:12:44.114104 15637 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.177265 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:44.197165 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.211800 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.217618 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.218770 12572 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:12:44.225692 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.226893 12564 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:44.213196 12568 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:44.153913 15689 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.157896 15810 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.233353 15819 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:44.270037 15819 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.272485 15819 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.234427 15787 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:44.235651 15808 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
W20260709 12:12:44.235888 15806 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:12:44.190951 15696 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.269376 15810 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.279235 15696 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.237274 15698 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:12:44.238446 15813 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:12:44.238624 15473 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:44.239667 15820 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:12:44.240937 15821 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:44.245822 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:44.251536 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:44.194656 15757 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.103s
I20260709 12:12:44.311223 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2181) took 2360 ms. Trace:
W20260709 12:12:44.252794 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.182372 15778 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.335357 15778 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:12:44.336412 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 175685us
I20260709 12:12:44.188424 15520 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.263391 15637 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.264230 15816 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:44.339571 15816 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:44.340097 15816 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:44.340934 15520 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.341676 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:44.343044 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2180) took 2393 ms. Trace:
I20260709 12:12:44.343456 12491 rpcz_store.cc:276] 0709 12:12:41.949870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.950103 (+   233us) service_pool.cc:224] Handling call
0709 12:12:41.961377 (+ 11274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:44.115483 (+2154106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:44.342786 (+227303us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:44.342978 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4653569408}
I20260709 12:12:44.264752 15824 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:44.345939 15824 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:44.346453 15824 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:44.194470 15575 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:44.349500 15700 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.349902 15575 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:44.350683 15575 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:44.351439 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.265736 15678 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.192195 15527 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:44.187179 15797 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.283620 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 478) took 2064 ms. Trace:
I20260709 12:12:44.284751 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.370573 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.290050 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.372524 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.372603 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.163182 15552 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.194865 15622 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:44.425917 15622 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:44.297161 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120015us
I20260709 12:12:44.298313 15821 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:44.299574 15782 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 1.255s	user 0.003s	sys 0.006s
I20260709 12:12:44.187714 15822 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:44.301297 15810 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:44.302161 15689 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:44.303546 15560 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:12:44.300755 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49346.9us
I20260709 12:12:44.251957 15812 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:44.314836 12497 rpcz_store.cc:276] 0709 12:12:41.950436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:41.950565 (+   129us) service_pool.cc:224] Handling call
0709 12:12:41.971662 (+ 21097us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.325707 (+1354045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:44.310081 (+984374us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:44.310321 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3088831232}
I20260709 12:12:44.323659 12572 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:12:44.333194 15775 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.227s	user 0.001s	sys 0.001s
I20260709 12:12:44.352857 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.355578 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:44.382302 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.349996 15793 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:12:44.390487 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.370016 12563 rpcz_store.cc:276] 0709 12:12:42.218978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.582672 (+1363694us) service_pool.cc:224] Handling call
0709 12:12:43.609261 (+ 26589us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:43.653571 (+ 44310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:43.653613 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:43.654276 (+   663us) log.cc:844] Serialized 26 byte log entry
0709 12:12:43.717672 (+ 63396us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:43.717737 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.718729 (+   992us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:44.281709 (+562980us) raft_consensus.cc:1685] finished
0709 12:12:44.281730 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:44.283544 (+  1814us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84585,"raft.queue_time_us":72235,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":6028,"thread_start_us":62882,"threads_started":1,"wal-append.queue_time_us":320887}
I20260709 12:12:44.452646 13253 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:44.452445 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.454145 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.395474 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.455611 12568 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:12:44.396991 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.398234 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.392742 15527 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:44.407224 15769 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.836s	user 0.008s	sys 0.005s
I20260709 12:12:44.407622 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.407459 15760 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.405889 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.409735 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:44.467522 13256 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:44.412189 15552 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.412426 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.397409 15637 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:44.397562 15408 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:44.382580 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:44.422412 15797 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.497274 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:44.353551 15635 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.423503 15678 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:44.500744 15818 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:12:44.458496 15560 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:44.506050 15821 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.507573 15825 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:44.508183 15825 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:44.508562 15825 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:44.466568 15769 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.511704 15769 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.476692 15812 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.516356 15812 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.518164 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 472) took 3085 ms. Trace:
I20260709 12:12:44.518462 15748 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.519430 15760 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.519623 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:44.518605 12570 rpcz_store.cc:276] 0709 12:12:41.432505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.163009 (+730504us) service_pool.cc:224] Handling call
0709 12:12:42.163598 (+   589us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:42.164315 (+   717us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:12:42.172309 (+  7994us) log.cc:844] Serialized 14 byte log entry
0709 12:12:43.126617 (+954308us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:43.138750 (+ 12133us) log.cc:844] Serialized 26 byte log entry
0709 12:12:43.138931 (+   181us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:43.138970 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.139068 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:44.332448 (+1193380us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:44.509377 (+176929us) raft_consensus.cc:1685] finished
0709 12:12:44.509505 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:44.518084 (+  8579us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":7357,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":143323,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":222,"spinlock_wait_cycles":32605824,"thread_start_us":678121,"threads_started":3,"wal-append.queue_time_us":689618}
W20260709 12:12:44.476977 15793 raft_consensus.cc:1068] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:12:44.520699 15748 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:44.520893 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.424487 15816 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:44.433537 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 319988us
I20260709 12:12:44.463225 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.525811 12564 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:44.521581 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:44.444693 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:44.468753 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.469199 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.470189 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.411418 15455 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.438105 15822 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:44.472862 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.470796 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.440501 15810 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:44.489758 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107330us
W20260709 12:12:44.442400 15707 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.622s	user 0.009s	sys 0.005s
I20260709 12:12:44.487739 13257 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:12:44.494627 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.543623 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.273850 15684 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.432821 15696 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:44.563251 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.564495 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.565538 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.566226 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.532683 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:44.542917 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.542213 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:44.532683 12492 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:44.546977 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.499608 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:44.572427 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:44.574862 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75406.7us
I20260709 12:12:44.572184 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.573220 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.576731 12493 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:44.574940 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130380us
I20260709 12:12:44.578666 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:44.578918 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:44.579617 13258 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:44.580788 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:44.580857 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.582252 12491 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:44.525082 15552 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:44.583536 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.587175 13253 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:44.584962 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:44.594589 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.595602 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:44.548506 15810 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:44.602371 12492 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 12:12:44.603226 12492 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 4
I20260709 12:12:44.603448 12492 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:12:44.603657 12492 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:44.604122 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.600076 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.597437 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.606726 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.607116 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.521700 15575 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.526893 15533 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:44.547652 15822 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:44.628794 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.630024 12645 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:44.613655 12493 raft_consensus.cc:2368] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:12:44.597046 15520 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:44.599071 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.622571 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.579145 15397 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:44.584137 15748 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:12:44.587209 15542 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:44.588286 15760 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:12:44.614178 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:44.601773 15778 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:44.643886 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:44.644187 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:44.499542 15549 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.685226 15549 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:44.686700 15824 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:44.612251 15790 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:12:44.688024 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:44.689003 12566 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:44.647188 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:44.691171 12646 raft_consensus.cc:2376] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8. Re-sending same reply.
I20260709 12:12:44.648344 12568 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 6: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:44.692770 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.599454 15832 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:44.651405 12490 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:44.652865 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.703140 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.703847 12488 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:44.630510 15696 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:44.569576 15707 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.623940 15408 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:44.666352 12645 raft_consensus.cc:2376] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10. Re-sending same reply.
I20260709 12:12:44.712391 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:44.646072 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.666707 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:44.666806 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:44.639744 15564 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:44.715888 15564 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:44.665902 15778 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:44.716740 15778 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:44.669152 12380 connection.cc:573] client connection to 127.11.11.131:35471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260709 12:12:44.641139 15520 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:44.721172 15520 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:44.724475 15797 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:44.725173 15797 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.678833 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:44.589232 15560 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.682565 15455 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.683867 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:44.684458 15816 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:44.696976 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.700664 12568 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:12:44.700927 12491 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:44.708098 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.709379 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.714895 12495 raft_consensus.cc:606] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:44.546469 15587 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:44.676960 15533 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.689241 15684 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.749279 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.753268 12495 raft_consensus.cc:614] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:12:44.768991 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.685081 15825 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:44.773254 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.773900 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.779610 12495 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:44.780843 12492 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:44.794567 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.808743 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:44.809350 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.809695 12646 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:44.812291 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.813645 12492 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:44.811261 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:44.817435 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:44.819094 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140487us
I20260709 12:12:44.819144 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.820034 12495 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:44.790707 15564 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.828066 15564 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:44.829077 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:44.830746 12645 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:44.832424 12645 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:44.836054 12495 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:12:44.837815 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:44.839754 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1796) took 2703 ms (client timeout 2015 ms). Trace:
I20260709 12:12:44.840008 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:44.808483 15787 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:44.851500 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 475) took 2663 ms. Trace:
I20260709 12:12:44.851998 12571 rpcz_store.cc:276] 0709 12:12:42.187703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.215940 (+ 28237us) service_pool.cc:224] Handling call
0709 12:12:42.216273 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:42.880672 (+664399us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:12:43.308143 (+427471us) log.cc:844] Serialized 14 byte log entry
0709 12:12:43.756376 (+448233us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:43.756740 (+   364us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:43.756803 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:44.851219 (+1094416us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:44.851270 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:44.851433 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60302,"prepare.queue_time_us":529221,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":61021,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":14330,"spinlock_wait_cycles":1527112960,"thread_start_us":681192,"threads_started":2,"wal-append.queue_time_us":660007}
W20260709 12:12:44.855626 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.856257 15688 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:44.858162 15688 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:44.859131 15749 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:44.859069 15688 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.859952 15749 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:44.860746 12489 rpcz_store.cc:269] 0709 12:12:42.136185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.182030 (+ 45845us) service_pool.cc:224] Handling call
0709 12:12:44.839459 (+2657429us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:44.839667 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2219948416}
I20260709 12:12:44.861274 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:44.865238 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:44.866772 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:44.867499 13262 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:44.815842 15707 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:44.895849 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.899372 12646 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:12:44.880654 12492 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:44.901193 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:44.903105 13257 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:44.815654 15622 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.919538 15622 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:44.920912 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:44.922536 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:44.908039 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46911.1us
I20260709 12:12:44.912204 15549 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:44.916867 15825 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.929339 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:44.931011 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:44.931819 15758 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:44.942298 15575 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:44.884827 15702 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:44.946933 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 488824us
I20260709 12:12:44.949338 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:44.885922 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:44.950981 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:44.953269 15825 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:44.961498 12570 pending_rounds.cc:85] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:12:44.961969 12570 pending_rounds.cc:107] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:12:44.962373 12570 raft_consensus.cc:2889] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:44.883282 15397 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:12:44.968775 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:44.953469 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:44.901790 15810 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.953444 15691 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:44.951980 15702 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:44.954489 15575 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:44.951954 15795 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:44.982344 12493 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:44.982200 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1801) took 2761 ms (client timeout 1710 ms). Trace:
W20260709 12:12:44.985358 12496 rpcz_store.cc:269] 0709 12:12:42.221087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.221231 (+   144us) service_pool.cc:224] Handling call
0709 12:12:44.981883 (+2760652us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:44.982073 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1313586560}
I20260709 12:12:44.997896 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:44.999815 12565 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.5-9.5]
W20260709 12:12:45.000336 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:45.001374 12380 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 280848us
I20260709 12:12:45.007498 15778 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.978474 15520 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:44.993188 15702 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:44.966092 15825 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:45.024253 15825 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:44.980141 15691 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:45.026319 15691 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:45.027238 15810 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.030104 12488 raft_consensus.cc:2368] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7.
I20260709 12:12:45.034103 12645 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:44.990674 15696 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:45.035502 12495 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:12:45.035737 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1802) took 3324 ms (client timeout 2227 ms). Trace:
W20260709 12:12:45.036204 12643 rpcz_store.cc:269] 0709 12:12:41.711763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.104075 (+392312us) service_pool.cc:224] Handling call
0709 12:12:45.035493 (+2931418us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:45.035622 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2545996544}
I20260709 12:12:45.036706 15520 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:45.037799 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:45.034379 15844 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:12:45.038782 15778 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:45.038936 15832 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:45.039511 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1823) took 1252 ms. Trace:
I20260709 12:12:45.039680 15832 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:45.043283 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:45.048620 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117699us
I20260709 12:12:45.054358 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.039934 12638 rpcz_store.cc:276] 0709 12:12:43.786812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.786991 (+   179us) service_pool.cc:224] Handling call
0709 12:12:43.787338 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:43.990279 (+202941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:45.039248 (+1048969us) spinlock_profiling.cc:243] Waited 76.3 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:45.039450 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175530240}
I20260709 12:12:45.056097 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:12:45.056734 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:45.057421 12563 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:45.068848 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:45.069866 12566 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:45.071718 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:45.081770 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 469) took 3652 ms (client timeout 2330 ms). Trace:
W20260709 12:12:45.084013 12569 rpcz_store.cc:269] 0709 12:12:41.408664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.077065 (+668401us) service_pool.cc:224] Handling call
0709 12:12:45.061301 (+2984236us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:45.061460 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553336192}
I20260709 12:12:45.107412 12563 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:45.108912 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:45.111580 12563 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:45.110492 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:45.040798 15832 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.069244 15397 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:45.093972 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:45.118176 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.115846 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:45.124653 12569 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:45.127779 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:45.121393 12566 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:45.129469 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.127748 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:45.131112 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:45.131353 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:45.133260 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:45.139240 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:45.140779 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:45.116629 15684 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:45.141888 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:45.143184 12563 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:45.143716 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.140995 15819 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:45.146989 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:45.166180 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
W20260709 12:12:45.158784 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:45.164600 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:45.168532 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:45.169476 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:45.161861 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:45.200547 12497 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:45.201582 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:45.211565 12489 raft_consensus.cc:2368] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11.
I20260709 12:12:45.216342 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.217175 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
W20260709 12:12:45.221536 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103481us
I20260709 12:12:45.148542 15575 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:45.224014 12495 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:12:45.163321 15760 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:45.148132 15817 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:45.227226 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:45.228331 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.174940 15819 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:45.219514 15804 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:12:45.230181 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:45.230849 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:45.160723 15520 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:45.231588 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:45.226374 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82055.3us
W20260709 12:12:45.263532 13218 connection.cc:256] RPC call timeout handler was delayed by -2.0573291179998705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260709 12:12:45.265183 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3938830260001396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260709 12:12:45.266518 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3938830260001396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260709 12:12:45.266947 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3367908640000223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
W20260709 12:12:45.267323 13218 connection.cc:256] RPC call timeout handler was delayed by -1.2503537099998994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
I20260709 12:12:45.231994 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.233049 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:45.215852 15637 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:45.197691 15684 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:45.228360 15856 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.001s	sys 0.001s
W20260709 12:12:45.239420 15783 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.946s	user 0.006s	sys 0.000s
I20260709 12:12:45.242246 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:45.239455 15437 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 12:12:45.246687 12568 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:45.255898 15760 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:45.252246 15822 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.599s	user 0.013s	sys 0.000s
I20260709 12:12:45.260659 15587 raft_consensus.cc:1064] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:12:45.261178 15758 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:45.262310 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:45.269152 12571 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:45.270159 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47670.6us
I20260709 12:12:45.278302 15836 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:45.255165 15819 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:45.227689 15843 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:12:45.257962 15817 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:45.296332 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:45.297174 12568 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:45.280972 12566 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 2. Current term is 3. Ops: []
I20260709 12:12:45.325579 12566 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:45.326910 12566 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:12:45.328024 12566 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:45.328943 12566 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:45.329255 12566 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:45.274462 15853 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:45.309505 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51868.6us
I20260709 12:12:45.335089 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:45.310281 12571 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:45.282433 15637 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:45.314201 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:45.317342 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.290341 15520 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:45.330202 12566 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.356284 12566 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:45.351713 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:45.352587 15817 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.353957 15397 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:45.321398 15620 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:12:45.360008 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161210us
I20260709 12:12:45.341985 15637 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:45.355062 15455 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:12:45.370311 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53137.5us
I20260709 12:12:45.302757 15822 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.376590 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.379446 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.382139 12566 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:12:45.382652 12566 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:12:45.382896 12566 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:12:45.383531 15533 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:45.366102 15549 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:12:45.388054 15707 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:45.388492 15707 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:45.380920 15564 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 12:12:45.389763 15794 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:45.389390 15707 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.390831 15707 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:45.393692 15691 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:45.390928 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1824) took 1586 ms (client timeout 1982 ms). Trace:
W20260709 12:12:45.394690 12640 rpcz_store.cc:269] 0709 12:12:43.804891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:43.805056 (+   165us) service_pool.cc:224] Handling call
0709 12:12:45.390767 (+1585711us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:45.390869 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92916992}
I20260709 12:12:45.396526 15810 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:45.399102 15815 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:45.400067 15691 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.399827 15815 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:45.400904 15691 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:12:45.403149 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1804) took 3123 ms (client timeout 1744 ms). Trace:
W20260709 12:12:45.403625 12494 rpcz_store.cc:269] 0709 12:12:42.280265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.736281 (+456016us) service_pool.cc:224] Handling call
0709 12:12:45.402938 (+2666657us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:45.403086 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1720508672}
I20260709 12:12:45.404433 12571 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:45.388617 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1826) took 1349 ms. Trace:
I20260709 12:12:45.407893 12644 rpcz_store.cc:276] 0709 12:12:44.039340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.045513 (+  6173us) service_pool.cc:224] Handling call
0709 12:12:45.388407 (+1342894us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:45.388520 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":427933440}
I20260709 12:12:45.397428 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.412341 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:45.383183 12566 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:45.316135 15836 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:45.413960 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139715us
I20260709 12:12:45.415021 12566 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:45.417359 12569 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:12:45.365789 15816 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:12:45.397200 15810 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.374126 15620 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:45.440182 12492 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9.
I20260709 12:12:45.403168 15841 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:12:45.436252 15749 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.554s	user 0.013s	sys 0.001s
I20260709 12:12:45.463765 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2084) took 1052 ms. Trace:
I20260709 12:12:45.416958 15825 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:45.472244 15868 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:12:45.472347 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84573us
I20260709 12:12:45.413491 15455 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:45.437649 15790 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:45.365872 15684 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.498231 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:45.502779 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:45.461845 15408 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:12:45.499948 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:45.369366 15760 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:45.500799 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117932us
I20260709 12:12:45.504439 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:45.507756 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:45.514849 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.515612 15853 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:45.516657 12566 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:12:45.516887 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.518672 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:45.452910 15575 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:12:45.520632 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:45.527535 12643 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:45.462363 15696 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 12:12:45.520848 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:45.531486 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:45.532927 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 169087us
I20260709 12:12:45.525162 15853 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:45.531194 15843 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:12:45.476203 15749 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.552295 12643 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:45.552752 15819 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:45.546684 15821 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:45.557901 12489 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:45.565570 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106919us
I20260709 12:12:45.573616 12646 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
W20260709 12:12:45.563848 15702 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.013s	sys 0.000s
I20260709 12:12:45.540196 15879 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:12:45.578222 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73311.1us
I20260709 12:12:45.558136 15853 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:45.583009 15853 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:45.583928 15853 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.555433 15843 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:12:45.553855 15815 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.557863 15821 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:45.586671 15681 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:45.546244 15871 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:45.582397 15786 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.237s	user 0.002s	sys 0.007s
W20260709 12:12:45.588294 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50427.9us
W20260709 12:12:45.589653 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:45.594446 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45898.7us
W20260709 12:12:45.600309 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 198200us
W20260709 12:12:45.601233 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:45.603552 12610 leader_election.cc:400] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:12:45.604245 12610 leader_election.cc:403] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:12:45.605898 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:45.608191 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:45.556346 15620 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:45.617247 12488 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:45.564779 15455 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:45.602173 15520 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:45.581254 15702 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.623308 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.623582 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.611156 15816 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:45.627952 12495 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:45.631757 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.630520 12497 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:12:45.631141 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.625330 15836 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.625422 15810 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.619954 15684 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:45.639103 15865 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:45.676194 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1800) took 3981 ms. Trace:
I20260709 12:12:45.628963 15707 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:45.639468 15749 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:45.649533 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.641765 15702 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.586664 15821 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:45.628238 15822 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:45.650127 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:45.655516 12488 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:45.671054 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.675809 15533 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:45.699981 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.679311 12641 rpcz_store.cc:276] 0709 12:12:41.694970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.177367 (+482397us) service_pool.cc:224] Handling call
0709 12:12:42.575727 (+398360us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cee9c 00007ffbf26cea5b 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:42.576037 (+   310us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:12:42.678427 (+102390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:42.678458 (+    31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:12:42.679514 (+  1056us) log.cc:844] Serialized 78 byte log entry
0709 12:12:43.132885 (+453371us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:43.205870 (+ 72985us) log.cc:844] Serialized 14 byte log entry
0709 12:12:43.504347 (+298477us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b18003fda20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:43.505146 (+   799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.505268 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:45.579725 (+2074457us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:45.675870 (+ 96145us) raft_consensus.cc:1685] finished
0709 12:12:45.675905 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:45.676084 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101610,"prepare.queue_time_us":78949,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":71,"raft.queue_time_us":179119,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":12010,"spinlock_wait_cycles":28259456,"thread_start_us":751712,"threads_started":2,"wal-append.queue_time_us":665626}
W20260709 12:12:45.677165 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46090.8us
I20260709 12:12:45.691247 12491 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:45.676194 15564 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:45.695066 12497 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:45.703985 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.705821 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.707984 12488 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:45.711504 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:45.721062 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:45.700848 15707 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:45.706557 15867 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.147s	user 0.001s	sys 0.001s
W20260709 12:12:45.755563 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9905763620001835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260709 12:12:45.768404 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.768992 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:45.772177 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8609763620002013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260709 12:12:45.772379 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:45.772780 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8447763620001751s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20260709 12:12:45.773378 12495 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:45.771348 12488 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:12:45.773283 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8141763620001257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
W20260709 12:12:45.774389 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7862763620000806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20260709 12:12:45.775022 12536 connection.cc:256] RPC call timeout handler was delayed by -2.76737636200005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260709 12:12:45.775425 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6989763620001668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18720000000000003s for a response
I20260709 12:12:45.687498 15684 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.752308 15821 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:45.778842 12610 leader_election.cc:400] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7.
I20260709 12:12:45.751953 15882 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:12:45.777473 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5702763620001861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260709 12:12:45.795846 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3398763620002683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2271s for a response
I20260709 12:12:45.710348 15564 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:45.751291 15881 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:45.789880 15888 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.003s	sys 0.000s
I20260709 12:12:45.751276 15520 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:12:45.753990 15871 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:45.767283 15868 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:45.784838 12610 leader_election.cc:403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:12:45.775846 15685 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:45.799898 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.768069 15749 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
W20260709 12:12:45.804581 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:45.807523 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1827) took 1767 ms (client timeout 1993 ms). Trace:
W20260709 12:12:45.816426 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:45.722959 15702 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:45.829268 12491 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:45.802644 15564 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:45.831545 12639 rpcz_store.cc:269] 0709 12:12:44.039636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.039779 (+   143us) service_pool.cc:224] Handling call
0709 12:12:45.807263 (+1767484us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:45.807438 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944829184}
I20260709 12:12:45.751488 15872 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:12:45.802719 15797 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:45.870014 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:45.881705 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.871804 15870 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:45.890013 12497 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:45.880362 15868 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:45.885725 15893 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:45.814281 15881 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:45.917541 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 350920us
I20260709 12:12:45.829766 15685 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:45.809973 15707 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.872881 15892 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:45.922676 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:45.927187 12491 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:45.929322 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129584us
I20260709 12:12:45.930428 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.874392 15702 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.870097 15882 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 12:12:45.955971 12491 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:45.956676 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:45.809991 15871 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.938352 15853 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:12:45.959573 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:45.971141 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:45.932742 15793 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:45.967046 15749 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:45.971347 15853 raft_consensus.cc:1068] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:12:46.031842 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.993s (SENT) [suppressed 32 similar messages]
W20260709 12:12:46.032181 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61882.5us
I20260709 12:12:46.032267 15907 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:12:45.984138 15637 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.602s	user 0.012s	sys 0.004s
I20260709 12:12:45.971884 15897 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:12:45.926354 15893 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.962711 15883 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 12:12:45.979571 15900 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:46.034946 15615 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:46.142427 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:46.035873 15901 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:12:45.931623 15892 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:45.985554 15473 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.091s
W20260709 12:12:46.243959 15885 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.357s	user 0.000s	sys 0.001s
W20260709 12:12:46.244549 15785 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.720s	user 0.004s	sys 0.000s
I20260709 12:12:46.076785 15684 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:46.246157 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 214343us
I20260709 12:12:46.036269 15749 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:46.264287 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1801) took 4555 ms. Trace:
I20260709 12:12:46.265027 12642 rpcz_store.cc:276] 0709 12:12:41.708916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:42.177374 (+468458us) service_pool.cc:224] Handling call
0709 12:12:42.219170 (+ 41796us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:42.372458 (+153288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:42.372505 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:42.670762 (+298257us) log.cc:844] Serialized 26 byte log entry
0709 12:12:43.318024 (+647262us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:43.318102 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:43.318210 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:46.074613 (+2756403us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:46.256448 (+181835us) raft_consensus.cc:1685] finished
0709 12:12:46.256485 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:46.264164 (+  7679us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":407446,"raft.queue_time_us":168937,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":15632,"spinlock_wait_cycles":45119744,"thread_start_us":646700,"threads_started":1,"wal-append.queue_time_us":684530}
I20260709 12:12:46.054781 15637 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:46.274303 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 302978us
I20260709 12:12:46.044226 15853 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.180s
I20260709 12:12:46.248929 15901 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:46.286792 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:46.214926 15615 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.042758 15868 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.299906 15810 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.300791 15810 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.304853 15810 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.307030 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:46.314620 15685 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.319520 15883 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:46.349442 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2204) took 1669 ms. Trace:
I20260709 12:12:46.350139 12493 rpcz_store.cc:276] 0709 12:12:44.680388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.707256 (+ 26868us) service_pool.cc:224] Handling call
0709 12:12:44.982073 (+274817us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:44.985190 (+  3117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:46.349012 (+1363822us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400191c50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:46.349261 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":540433664}
W20260709 12:12:46.341526 15894 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.379s	user 0.002s	sys 0.000s
I20260709 12:12:46.337548 15836 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:12:46.336635 15875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.001s	sys 0.030s
I20260709 12:12:46.343302 15702 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.337407 15896 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:12:46.353953 15881 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:46.370903 15875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.002s	sys 0.030s
W20260709 12:12:46.401417 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:46.359094 15363 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:46.371428 15836 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:46.412488 15363 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:46.417980 15527 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.423471 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:46.425041 15865 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:46.425514 15865 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:46.426388 15865 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:46.434019 15908 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:12:46.429315 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2196) took 1898 ms. Trace:
I20260709 12:12:46.435551 12490 rpcz_store.cc:276] 0709 12:12:44.531390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.532038 (+   648us) service_pool.cc:224] Handling call
0709 12:12:44.572746 (+ 40708us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:44.701863 (+129117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:46.428922 (+1727059us) spinlock_profiling.cc:243] Waited 78.4 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:46.429211 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":180242688}
I20260709 12:12:46.437717 15688 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:46.437934 15895 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
W20260709 12:12:46.444247 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135356us
I20260709 12:12:46.411818 15836 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:46.449880 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:46.437333 15824 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.403620 15901 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:46.474942 15911 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:12:46.460067 15865 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:46.461223 15892 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.464924 15897 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
W20260709 12:12:46.557421 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107640us
I20260709 12:12:46.558930 13218 leader_election.cc:400] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11.
I20260709 12:12:46.559386 13218 leader_election.cc:403] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:12:46.579641 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:46.552855 15910 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:12:46.527369 15912 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
W20260709 12:12:46.563165 15816 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.907s	user 0.008s	sys 0.003s
I20260709 12:12:46.525921 15637 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.561924 15615 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.526685 15893 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:46.594795 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:46.584555 13255 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:46.512311 15363 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.582417 15917 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11.
I20260709 12:12:46.592121 15816 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
W20260709 12:12:46.586992 15564 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.701s	user 0.014s	sys 0.000s
I20260709 12:12:46.509557 15875 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20260709 12:12:46.585466 15883 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:12:46.619130 15922 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:12:46.600971 15915 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:46.622316 15915 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:46.623133 15915 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:46.623751 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1942) took 1656 ms. Trace:
I20260709 12:12:46.624185 12571 rpcz_store.cc:276] 0709 12:12:44.967418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:45.337095 (+369677us) service_pool.cc:224] Handling call
0709 12:12:45.337402 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:45.404802 (+ 67400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:46.623409 (+1218607us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:46.623670 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153778560}
I20260709 12:12:46.622416 15825 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.651520 15363 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.640166 15825 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.628459 15917 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:12:46.670141 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:46.676280 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81602.8us
I20260709 12:12:46.678812 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:46.679632 12488 raft_consensus.cc:2376] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8. Re-sending same reply.
I20260709 12:12:46.682212 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:46.684894 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.685771 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:46.687232 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:46.688045 12490 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:46.688374 12642 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:12:46.642795 15869 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:12:46.695313 12488 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:46.681804 15870 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.777s	user 0.009s	sys 0.008s
I20260709 12:12:46.616884 15564 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.699220 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:46.708452 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:46.715962 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:46.714974 15922 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:46.706858 15870 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7.
I20260709 12:12:46.722616 15825 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:46.730476 15924 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:12:46.732023 15821 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.950s	user 0.010s	sys 0.006s
I20260709 12:12:46.735978 15564 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.741086 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1843) took 1956 ms. Trace:
I20260709 12:12:46.742550 15915 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.747490 12646 rpcz_store.cc:276] 0709 12:12:44.784800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.904700 (+119900us) service_pool.cc:224] Handling call
0709 12:12:44.905108 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:45.576675 (+671567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:46.740677 (+1164002us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:46.740985 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1536741248}
W20260709 12:12:46.748389 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:46.749123 15871 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.749519 15915 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.733151 15819 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:46.752421 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:46.751055 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.774919 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.775604 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:46.776031 12645 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:46.746565 12642 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:12:46.754158 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:46.777285 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:46.750568 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:46.754400 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:46.778144 12570 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:46.778523 12564 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:46.774487 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.779940 12639 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:46.780663 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:46.783178 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:46.790091 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.755134 15875 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:46.806749 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:46.820313 12566 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:46.807514 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.823145 12644 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:46.823937 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.824571 12639 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:46.807936 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:46.809686 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:46.808068 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:46.826769 12564 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:46.825446 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.749691 15871 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:46.753038 15397 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.830072 12569 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:12:46.754647 15473 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:46.828157 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:46.841846 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:46.842593 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92117.5us
I20260709 12:12:46.841512 12645 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:46.845983 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:46.844604 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:46.846352 12642 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:46.842041 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:12:46.846614 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109605us
I20260709 12:12:46.848338 12644 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:46.848845 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:46.846886 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:46.829618 15875 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:46.828112 15684 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.855165 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.864005 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:46.855906 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.853790 15816 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:46.870213 15816 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:46.870684 15816 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:46.871042 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:46.872402 12495 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:46.856989 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.874912 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.876611 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.858233 15560 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:46.819497 15824 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.877436 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.878115 15824 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:46.880784 15824 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.861796 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:46.820806 15922 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:46.853930 15821 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:46.882817 15821 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:46.882508 15922 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.883251 15821 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:46.886139 15819 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:46.887988 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.890695 12639 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:46.866178 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:46.859730 15527 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:46.863974 15875 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:46.845293 15397 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:46.867269 15836 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.867990 15871 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.869519 15363 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:12:46.896883 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.899374 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.902166 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.888167 15684 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:46.914242 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.915681 15684 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.917876 12493 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:46.916316 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.918844 15797 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:46.919341 15797 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:46.919601 12493 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:12:46.920280 15797 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:46.922147 12572 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:12:46.923714 15397 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:46.924059 12564 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:46.925851 15836 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:46.919449 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.924831 15560 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:46.932245 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2066) took 2572 ms. Trace:
I20260709 12:12:46.932693 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:46.935642 12639 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:46.932858 12565 rpcz_store.cc:276] 0709 12:12:44.359484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.934345 (+574861us) service_pool.cc:224] Handling call
0709 12:12:44.999156 (+ 64811us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:45.000774 (+  1618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:46.931958 (+1931184us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b54003abe50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:46.932163 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145984768}
I20260709 12:12:46.936416 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.938344 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.940136 12644 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:46.943212 12639 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33480
I20260709 12:12:46.944193 12639 raft_consensus.cc:606] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Received request to transfer leadership
I20260709 12:12:46.944676 12639 raft_consensus.cc:614] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:12:46.949092 15898 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.872s	user 0.000s	sys 0.002s
I20260709 12:12:46.887696 15757 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
W20260709 12:12:46.975144 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109057us
I20260709 12:12:46.976210 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:46.971676 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:46.941550 12644 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:46.927906 15796 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:47.034417 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58296.7us
I20260709 12:12:46.899242 15527 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:46.960415 15768 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:46.943931 15870 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.115s
W20260709 12:12:47.134027 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.045971 15796 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.184548 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 396) took 1971 ms. Trace:
I20260709 12:12:47.185108 12489 rpcz_store.cc:276] 0709 12:12:45.212858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:45.213020 (+   162us) service_pool.cc:224] Handling call
0709 12:12:47.184228 (+1971208us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:47.184428 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":737165568}
I20260709 12:12:47.052271 15797 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:47.193791 12642 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
W20260709 12:12:47.197618 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.052659 15685 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.154s
I20260709 12:12:47.206068 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.199576 12643 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:47.203183 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2217) took 2282 ms (client timeout 1806 ms). Trace:
W20260709 12:12:47.209322 12492 rpcz_store.cc:269] 0709 12:12:44.921096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:44.921930 (+   834us) service_pool.cc:224] Handling call
0709 12:12:47.202927 (+2280997us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:47.203089 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185996544}
I20260709 12:12:47.044533 15549 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:47.210933 13262 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:47.052382 15790 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:47.218557 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:47.215428 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.219512 13260 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:47.225159 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.227213 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:47.228638 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.219043 15790 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:47.211139 15768 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:12:47.241351 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:47.242101 13257 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:47.239598 13260 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:47.242358 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.243431 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:47.248080 15821 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.250391 15821 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.250998 15794 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:47.251660 15794 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:47.254880 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.261086 13256 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:47.252712 15794 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.220625 15549 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.210724 15875 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.238020 15790 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:47.231233 13254 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:47.223505 15816 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.240425 15819 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:47.275933 12569 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:47.270319 15757 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:47.275274 13254 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:47.304703 13261 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:47.268528 15345 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:47.271804 15827 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.937s	user 0.020s	sys 0.029s
W20260709 12:12:47.312242 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.315297 13254 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.316182 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:47.316645 13254 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:47.322774 15549 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:47.323956 15527 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:47.324757 13257 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:47.325883 15819 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.274194 15363 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:12:47.328440 15816 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:47.311340 15827 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.977s	user 0.022s	sys 0.033s
I20260709 12:12:47.274595 15821 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:12:47.307910 15794 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:47.271229 15685 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:12:47.340126 15937 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:47.353065 15937 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.355690 15937 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:47.361575 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 495886us
I20260709 12:12:47.364342 15437 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:47.362990 13254 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:12:47.368018 15937 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:47.369355 15685 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:47.369800 15685 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:47.307328 15757 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.371294 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:47.374382 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:47.384467 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:47.385010 15843 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:47.385426 15843 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:47.386500 15843 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.388553 15843 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:47.365208 15345 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:47.392628 15615 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:47.393458 15615 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:47.393942 15615 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:47.395601 15913 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.918s	user 0.005s	sys 0.017s
I20260709 12:12:47.395748 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:47.399916 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:47.403266 12490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.013s	sys 0.021s
I20260709 12:12:47.386831 15937 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:12:47.404855 12490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.013s	sys 0.021s
I20260709 12:12:47.405874 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:47.424194 15684 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.010s	sys 0.037s
I20260709 12:12:47.446018 15943 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:47.447961 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.383231 15685 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.2, Last appended by leader: 2, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.412766 15937 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:12:47.473831 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:47.474763 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:47.475364 12609 connection.cc:256] RPC call timeout handler was delayed by -2.0539544089997435s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260709 12:12:47.475800 12609 connection.cc:256] RPC call timeout handler was delayed by -1.7398544089999177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24460000000000004s for a response
W20260709 12:12:47.476162 12609 connection.cc:256] RPC call timeout handler was delayed by -1.6596584109997821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20260709 12:12:47.476498 12609 connection.cc:256] RPC call timeout handler was delayed by -1.59737297699985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260709 12:12:47.476858 12609 connection.cc:256] RPC call timeout handler was delayed by -1.4031584109998221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20440000000000003s for a response
W20260709 12:12:47.477222 12609 connection.cc:256] RPC call timeout handler was delayed by -1.2087584109997351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22599999999999998s for a response
I20260709 12:12:47.481508 15870 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:12:47.428068 15825 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:47.452689 15684 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.011s	sys 0.038s
W20260709 12:12:47.477562 12609 connection.cc:256] RPC call timeout handler was delayed by -1.0938948699999855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20260709 12:12:47.483875 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 447197us
I20260709 12:12:45.466953 12568 rpcz_store.cc:276] 0709 12:12:44.410832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:45.322588 (+911756us) service_pool.cc:224] Handling call
0709 12:12:45.323060 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:45.323370 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:12:45.338333 (+ 14963us) log.cc:844] Serialized 14 byte log entry
0709 12:12:45.458289 (+119956us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:45.458371 (+    82us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:45.458418 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:45.458496 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:45.463686 (+  5190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15274,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":3620,"raft.queue_time_us":99,"thread_start_us":119473,"threads_started":1,"wal-append.queue_time_us":125708}
W20260709 12:12:47.504284 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.500941 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:47.472800 15943 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:47.519579 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:47.482069 15937 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:47.523109 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.557403 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:47.478251 15863 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:12:47.558962 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.488190 15870 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:47.516971 15685 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:47.522212 15363 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:47.551719 15875 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.522946 15943 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:47.579551 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:47.580565 12643 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:47.552366 15946 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:12:47.587105 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67672.4us
I20260709 12:12:47.566586 15863 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:47.550451 15821 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:12:47.597575 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:47.592038 15760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.956s	user 0.013s	sys 0.015s
I20260709 12:12:47.584502 15564 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:47.588774 15397 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:47.625247 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 198732us
I20260709 12:12:47.584842 15944 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:47.590927 15876 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:47.626027 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:12:47.623759 15760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.987s	user 0.014s	sys 0.015s
W20260709 12:12:47.630386 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 265472us
I20260709 12:12:47.637195 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:47.640411 12568 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:12:47.599876 15870 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:47.644265 15942 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:12:47.632068 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:47.650195 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.627595 15797 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:47.657789 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161962us
I20260709 12:12:47.644675 15870 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:12:47.652886 15934 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:12:47.648064 15819 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:47.660553 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.675129 13262 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:47.671320 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:47.673719 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:47.680033 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.628961 15397 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.658452 15797 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:47.660418 15757 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:47.671970 15363 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:47.684708 13254 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:12:47.701128 15948 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
W20260709 12:12:47.703191 15935 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:12:47.704471 15397 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:47.707872 15873 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.524s	user 0.001s	sys 0.001s
I20260709 12:12:47.711849 15615 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:47.715355 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48932.3us
I20260709 12:12:47.628597 15863 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.685178 15951 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:12:47.691475 15871 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:47.687851 15760 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:47.682773 15819 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.691037 15797 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:47.727639 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54036.6us
I20260709 12:12:47.727756 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:47.731624 15684 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:47.735550 15871 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.672614 15950 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:12:47.763800 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:47.765462 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49115.7us
I20260709 12:12:47.762567 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:47.767132 12490 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 3.
I20260709 12:12:47.767504 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:47.763392 12491 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:47.769026 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:47.770411 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:47.768561 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:47.780761 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:47.698231 15946 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:47.786021 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:47.787602 15956 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:12:47.790546 12644 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:12:47.790616 15871 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:47.738294 15953 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:12:47.792956 12644 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:12:47.794232 12646 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 5. Ops: []
I20260709 12:12:47.794493 12645 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:47.797240 12638 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:12:47.749359 15954 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:12:47.803764 12637 pending_rounds.cc:85] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 4
I20260709 12:12:47.804358 12637 pending_rounds.cc:107] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:12:47.804670 12637 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:47.808001 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62522.8us
I20260709 12:12:47.768330 15944 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:47.801259 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:47.812637 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:47.751494 15906 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:47.751120 15960 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:12:47.813239 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85555.7us
I20260709 12:12:47.755915 15749 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:12:47.818372 12492 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:47.751690 15971 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:47.841475 12489 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
W20260709 12:12:47.813879 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46486.8us
I20260709 12:12:47.822108 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:12:47.842831 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.806890 15684 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:47.832841 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:47.853288 12492 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:47.738579 15549 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.080s
I20260709 12:12:47.823892 15963 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:47.765789 15760 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.854031 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:47.856750 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:47.859831 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50726.4us
I20260709 12:12:47.860154 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:47.861174 12492 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 12:12:47.862829 12492 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:12:47.864563 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2218) took 2845 ms. Trace:
I20260709 12:12:47.781708 15790 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.827735 15906 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:47.865166 12494 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:47.870033 12494 raft_consensus.cc:614] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:12:47.866884 12496 rpcz_store.cc:276] 0709 12:12:45.019141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:45.061353 (+ 42212us) service_pool.cc:224] Handling call
0709 12:12:45.061786 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:45.062159 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:45.090430 (+ 28271us) log.cc:844] Serialized 14 byte log entry
0709 12:12:47.191491 (+2101061us) spinlock_profiling.cc:243] Waited 4.64 ms on lock 0x7b18000eb5a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:12:47.487414 (+295923us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b18000eb5a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:47.617558 (+130144us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:47.623260 (+  5702us) log.cc:844] Serialized 26 byte log entry
0709 12:12:47.623484 (+   224us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:47.626144 (+  2660us) log.cc:844] Serialized 14 byte log entry
0709 12:12:47.763756 (+137612us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b18000eb5a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:12:47.864206 (+100450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:47.864313 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:47.864354 (+    41us) raft_consensus.cc:1685] finished
0709 12:12:47.864369 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:47.864501 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23331,"prepare.queue_time_us":30720,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":285918,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":17926,"spinlock_wait_cycles":39206272,"thread_start_us":496890,"threads_started":3,"wal-append.queue_time_us":2484696}
I20260709 12:12:47.871935 15397 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:47.872813 15397 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.843390 15962 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:12:47.879805 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.826741 15560 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:12:47.858768 15684 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:47.858578 15963 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:47.866829 15966 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:47.896406 15966 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.896848 15790 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.890635 15970 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:47.898213 12492 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:47.898629 15966 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.898312 15970 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.887277 15794 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
W20260709 12:12:47.901839 12443 auto_rebalancer.cc:893] Removing replace marker failed: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:12:47.902462 12443 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 12:12:47.902918 12443 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:12:47.868692 15870 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:12:47.903877 15587 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 5, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.905193 15749 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:47.906985 15974 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:47.907507 15974 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:47.908571 15974 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.894786 15946 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:47.910130 15906 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:47.911393 15906 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:47.875701 15397 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.886899 15816 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:12:47.843369 15939 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.063s
W20260709 12:12:47.961392 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 104746us
I20260709 12:12:47.964607 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:47.885789 15965 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:47.966046 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120798us
I20260709 12:12:47.977185 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:47.914731 15963 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:47.952904 15615 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.961026 15816 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:47.992857 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:47.967409 15819 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:47.993724 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111863us
I20260709 12:12:48.001585 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:48.002020 12640 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:47.851920 15971 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.002363 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.007004 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:47.957940 15966 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:12:48.003069 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:48.002516 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.011543 12638 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:48.008646 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:48.021296 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.022149 12566 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:47.999418 15816 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:48.034584 12568 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:48.038561 12565 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:48.005887 15821 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:48.045922 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.008811 15962 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:12:48.041703 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.042759 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.049657 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:48.050763 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:48.039829 15520 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:48.065303 15520 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:48.038239 15970 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:47.883343 15944 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.005736 15960 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:12:48.040894 15819 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:48.094589 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:47.988168 15965 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.102609 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:48.097448 15685 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.521s	user 0.014s	sys 0.000s
I20260709 12:12:48.081125 15819 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:48.123380 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.127144 12572 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:48.128612 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.129413 12570 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:48.131893 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.132941 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.098472 15794 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:12:48.111289 15549 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.172981 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.180754 12570 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:48.127029 15988 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:12:48.121048 15962 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:48.126452 15969 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:12:48.116768 15819 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:48.188149 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.128098 15945 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.123263 15976 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:48.120819 15821 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.067s
I20260709 12:12:48.206961 15975 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:48.210494 15975 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.212674 15975 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:48.213234 15789 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.839s	user 0.024s	sys 0.037s
W20260709 12:12:48.213593 15789 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.954s	user 0.024s	sys 0.037s
I20260709 12:12:48.215023 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.215698 12567 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:48.218763 15983 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.219357 15983 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:48.221830 15981 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:12:48.186736 15549 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.215301 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:48.228636 12570 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:48.227061 12572 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:48.230826 15549 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:48.240994 15957 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:48.242029 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.241685 15957 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:48.242530 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.251111 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.246805 15972 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
W20260709 12:12:48.263717 15943 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.677s	user 0.010s	sys 0.003s
I20260709 12:12:48.307498 15943 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.308394 15803 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:48.288089 15760 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:48.299791 15962 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.303098 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.308911 15803 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:48.315315 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.244798 15957 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:48.255620 15967 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:12:48.309477 12497 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:48.301420 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:48.313952 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:48.322456 12492 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:48.322845 12637 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:48.285365 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 413) took 1065 ms. Trace:
I20260709 12:12:48.323633 13261 rpcz_store.cc:276] 0709 12:12:47.219571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:47.220226 (+   655us) service_pool.cc:224] Handling call
0709 12:12:48.284961 (+1064735us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:48.285256 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":169954176}
I20260709 12:12:48.204358 15945 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:48.326234 15987 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:12:48.327661 15945 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:48.180820 15520 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.295410 15971 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:48.331563 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.336355 13259 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:12:48.338451 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 541) took 1137 ms. Trace:
W20260709 12:12:48.318112 15925 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.490s	user 0.004s	sys 0.000s
I20260709 12:12:48.338893 13259 rpcz_store.cc:276] 0709 12:12:47.201408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:47.211222 (+  9814us) service_pool.cc:224] Handling call
0709 12:12:48.330691 (+1119469us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:48.330957 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:48.336846 (+  5889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:48.338370 (+  1524us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3822464}
I20260709 12:12:48.292222 15965 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:48.347455 15993 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.094s	user 0.001s	sys 0.001s
I20260709 12:12:48.221578 15983 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:48.255458 15979 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.251s
W20260709 12:12:48.358235 15797 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.003s
W20260709 12:12:48.354244 15937 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.784s	user 0.016s	sys 0.000s
I20260709 12:12:48.389456 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:48.418177 12492 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:48.419845 12492 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:12:48.358894 15797 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.427765 15797 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:48.425468 15937 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:48.438592 15963 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.440902 15963 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:48.448640 12639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.050s	sys 0.079s
W20260709 12:12:48.449244 12639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.050s	sys 0.079s
W20260709 12:12:48.461071 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.463683 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.464697 12643 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:48.460536 15758 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:12:48.480811 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 13) took 2972 ms. Trace:
I20260709 12:12:48.317915 15803 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.481673 12494 rpcz_store.cc:276] 0709 12:12:45.493268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:45.494009 (+   741us) service_pool.cc:224] Handling call
0709 12:12:48.358834 (+2864825us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:48.465862 (+107028us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5425401088}
I20260709 12:12:48.486270 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.492700 12641 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:12:48.488456 15758 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 4, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:48.493778 15947 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.869s	user 0.008s	sys 0.002s
I20260709 12:12:48.496346 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2227) took 1644 ms. Trace:
I20260709 12:12:48.496872 12495 rpcz_store.cc:276] 0709 12:12:46.852248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:46.888659 (+ 36411us) service_pool.cc:224] Handling call
0709 12:12:46.900114 (+ 11455us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:46.942899 (+ 42785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:46.942942 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:46.957421 (+ 14479us) log.cc:844] Serialized 52 byte log entry
0709 12:12:47.413518 (+456097us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:47.413574 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:47.415817 (+  2243us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:48.489263 (+1073446us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:12:48.495876 (+  6613us) raft_consensus.cc:1685] finished
0709 12:12:48.495915 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:48.496248 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27848,"raft.queue_time_us":53746,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":17037,"spinlock_wait_cycles":5792128,"thread_start_us":234553,"threads_started":1,"wal-append.queue_time_us":492757}
I20260709 12:12:48.507776 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:48.497277 15684 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.547s	user 0.003s	sys 0.009s
I20260709 12:12:48.509078 15684 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.510262 15796 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:48.511487 15684 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:48.515627 12492 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:48.519976 15959 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:48.520530 15959 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:48.497053 15803 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:48.524115 12492 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 4. Current term is 5. Ops: []
W20260709 12:12:48.531208 15897 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:12:48.526598 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.533986 12489 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:12:48.498245 15937 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:48.558269 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.570102 12641 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:48.572671 15980 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.573536 15980 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.588479 15865 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.497484 15816 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:48.590255 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.592147 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.589313 15865 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.598698 15865 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:48.602715 15893 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:48.603482 15893 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:48.610821 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.611910 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.614902 15863 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:12:48.620046 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.622072 15796 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:48.622833 15795 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:12:48.638841 15961 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.865s	user 0.005s	sys 0.000s
I20260709 12:12:48.643886 15974 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:48.644411 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.664901 15949 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.030s	user 0.001s	sys 0.000s
I20260709 12:12:48.684121 15691 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:48.688092 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.689239 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.763177 12643 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
W20260709 12:12:48.766670 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.767395 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2236) took 1003 ms. Trace:
I20260709 12:12:48.769479 12491 rpcz_store.cc:276] 0709 12:12:47.763547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:47.810402 (+ 46855us) service_pool.cc:224] Handling call
0709 12:12:47.810811 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:47.885427 (+ 74616us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:47.885483 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:47.885549 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:47.885596 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:48.693350 (+807754us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:48.767093 (+ 73743us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:48.767312 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170853120}
I20260709 12:12:48.657948 15363 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.597208 15825 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:48.798219 15825 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:48.604766 15893 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:48.775429 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.807134 15893 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:48.772168 15221 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:48.809465 15221 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.811336 12639 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:48.812302 15221 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:48.783600 15871 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:48.814141 15871 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:48.814867 15816 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:12:48.815001 15871 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:48.521443 15959 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.685074 15691 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:48.786715 15819 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.586s	user 0.008s	sys 0.003s
W20260709 12:12:48.791747 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.830031 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.830336 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.792860 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:48.812876 15520 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:48.824738 15984 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:48.849941 15984 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.850178 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:48.824764 15897 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:48.828469 15914 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:48.835299 15221 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:48.836169 15749 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:48.818125 15959 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:48.838913 15691 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:48.843197 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:48.876446 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.847872 13258 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:12:48.834028 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71389.4us
I20260709 12:12:48.850183 13259 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 5. Current term is 6. Ops: []
I20260709 12:12:48.881446 12570 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:48.854034 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 260168us
I20260709 12:12:48.503136 15943 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:48.866255 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.869678 15865 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:48.901435 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.889735 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.891563 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.903887 12572 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:48.904011 12563 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9.
I20260709 12:12:48.870783 15897 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.869277 15984 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:48.897171 15951 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.858161 15816 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:48.947407 15951 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.827150 15363 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.875164 15749 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.952172 15749 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:48.892246 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.905445 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:48.953207 12571 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:48.952384 15363 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:48.900295 15980 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:48.852742 15819 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.824833 15871 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.961452 15871 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:48.961851 15819 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:48.904884 15795 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.962839 12563 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:48.962937 15795 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.964010 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.938825 15796 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.966174 15796 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.940140 13259 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 9 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:12:48.942512 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:48.969923 12567 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:48.903004 15827 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.943857 15914 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:48.945019 15907 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:48.947520 15816 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:48.858560 15876 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:48.937001 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:48.975255 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 63136.9us
I20260709 12:12:48.999024 12572 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:48.976133 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91609.9us
I20260709 12:12:49.006615 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:49.008380 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:49.009069 15804 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:49.009457 13254 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:12:48.971087 15827 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.009102 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:48.957280 15897 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:49.022699 15796 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:49.027415 15914 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:49.030423 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:49.033769 15816 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:48.997850 15980 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
W20260709 12:12:49.039132 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.990381 15907 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:49.045332 15827 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:49.011528 15876 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:49.052912 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.057572 15795 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:49.086758 15915 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:49.087473 15915 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:49.092844 15951 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:49.111861 15803 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 12:12:49.061823 15907 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.168584 15907 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:49.088655 15915 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.174578 15915 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:49.052273 15876 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:49.143311 15778 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:12:49.180601 15622 raft_consensus.cc:1027] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: attempt to promote peer 9485b0afaf144396adf854fff857414e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260709 12:12:49.310499 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.302486 12395 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:12:49.320200 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:48.972855 12563 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:49.322279 12563 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:49.325187 12563 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.329141 12563 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:49.332602 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.334384 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:49.336504 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:49.338279 12567 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:49.339550 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.341476 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.014221 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:49.368927 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:49.369692 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 355360us
W20260709 12:12:49.370747 12393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 122) took 21089 ms (client timeout 15000 ms). Trace:
I20260709 12:12:49.373858 12571 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:12:49.376114 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:49.360071 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.155514 15825 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.393935 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:49.394697 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.394951 12571 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:49.396003 15915 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:49.397028 15915 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:49.166785 15863 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.002s
I20260709 12:12:49.394291 15871 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:49.162329 15974 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.518s	user 0.012s	sys 0.003s
W20260709 12:12:49.528824 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.532322 12495 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
W20260709 12:12:49.553781 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.317202 12395 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:49.555282 15871 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:49.559082 12572 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.559s	user 0.013s	sys 0.005s
I20260709 12:12:49.559886 12572 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:12:49.371510 12393 rpcz_store.cc:269] 0709 12:12:28.280718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:28.280997 (+   279us) service_pool.cc:224] Handling call
0709 12:12:49.370283 (+21089286us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:40.212772 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:40.213462 (+   690us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:40.213516 (+    54us) write_op.cc:435] Acquired schema lock
0709 12:12:40.213541 (+    25us) tablet.cc:662] Decoding operations
0709 12:12:40.214820 (+  1279us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:40.214904 (+    84us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:40.214935 (+    31us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:12:40.216249 (+  1314us) tablet.cc:701] Row locks acquired
0709 12:12:40.216301 (+    52us) write_op.cc:260] PREPARE: finished
0709 12:12:40.216509 (+   208us) write_op.cc:270] Start()
0709 12:12:40.216693 (+   184us) write_op.cc:276] Timestamp: P: 1783599160216436 usec, L: 0
0709 12:12:40.216708 (+    15us) op_driver.cc:348] REPLICATION: starting
0709 12:12:40.217460 (+   752us) log.cc:844] Serialized 4813 byte log entry
0709 12:12:41.064938 (+847478us) op_driver.cc:464] REPLICATION: finished
0709 12:12:41.478756 (+413818us) write_op.cc:301] APPLY: starting
0709 12:12:41.504420 (+ 25664us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:41.631321 (+126901us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:41.700989 (+ 69668us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:42.452465 (+751476us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:42.540246 (+ 87781us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0709 12:12:42.540304 (+    58us) write_op.cc:312] APPLY: finished
0709 12:12:42.540753 (+   449us) log.cc:844] Serialized 89 byte log entry
0709 12:12:42.773019 (+232266us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:42.842241 (+ 69222us) write_op.cc:454] Released schema lock
0709 12:12:42.876139 (+ 33898us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:43.134759 (+258620us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:43.142422 (+  7663us) write_op.cc:454] Released schema lock
0709 12:12:44.280010 (+1137588us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
Metrics: {"mutex_wait_us":984183,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":380972,"apply.run_cpu_time_us":24111,"apply.run_wall_time_us":1605826,"num_ops":12,"prepare.queue_time_us":783021,"prepare.run_cpu_time_us":18032,"prepare.run_wall_time_us":267822,"raft.queue_time_us":206703,"raft.run_cpu_time_us":24709,"raft.run_wall_time_us":229694,"replication_time_us":848030,"spinlock_wait_cycles":56735488,"thread_start_us":943072,"threads_started":5,"wal-append.queue_time_us":655507,"wal-append.run_cpu_time_us":26625,"wal-append.run_wall_time_us":3948522}]]}
I20260709 12:12:49.567180 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:49.567534 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2249) took 1745 ms. Trace:
I20260709 12:12:49.568183 12495 rpcz_store.cc:276] 0709 12:12:47.825147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.499760 (+674613us) service_pool.cc:224] Handling call
0709 12:12:48.682252 (+182492us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:48.682594 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:49.566012 (+883418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:49.566974 (+   962us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:49.567321 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1662385152}
I20260709 12:12:49.569437 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:49.574198 15825 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:49.582563 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.582145 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.583184 15520 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:49.584353 15520 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.584755 15915 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.585525 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:49.586660 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:49.588582 15520 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:49.588519 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:49.591266 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.578756 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.590660 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:49.595491 15622 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:49.596230 13254 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:49.596540 15622 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.589519 15870 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:49.606411 15816 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:49.606662 15870 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.607550 15816 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.581069 12395 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:49.610958 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:49.612002 13254 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:49.617133 15816 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:49.606921 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.616994 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:49.623986 15870 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:49.613212 12395 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20260709 12:12:49.635068 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:49.636742 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:49.637849 13254 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:49.636966 15816 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:49.639243 15816 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.640241 12572 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:12:49.646251 15816 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:49.647238 15816 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:49.647710 15816 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:49.648140 15816 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:49.654482 15871 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:49.658442 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.658753 15871 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:12:49.661121 12643 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:49.663743 15622 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:49.665150 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:49.668524 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:49.669783 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:49.670658 15870 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:49.671587 15870 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:49.672091 15870 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:49.669757 12395 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:49.685918 15876 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:49.708334 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.713081 12567 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:49.590660 15520 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:49.686918 15876 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:49.735881 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.773167 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:49.774514 12571 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:12:49.775241 15893 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:49.776106 15893 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.777849 15863 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:49.778721 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.778663 15863 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.781474 15871 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:49.792339 15871 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.793853 15876 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:49.791070 12563 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20260709 12:12:49.813808 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.796572 12563 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 3
I20260709 12:12:49.759028 15622 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:49.788249 12395 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 5 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 6 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:12:49.846195 12563 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:12:49.856699 12563 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:49.866124 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.846779 15520 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.857288 15871 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:49.865502 15863 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:49.871649 15907 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:49.872720 15907 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:49.875600 15907 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:49.876472 15622 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:49.875643 12572 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:49.908766 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:49.915877 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2099) took 1017 ms. Trace:
I20260709 12:12:49.916601 12563 rpcz_store.cc:276] 0709 12:12:48.893609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.965154 (+ 71545us) service_pool.cc:224] Handling call
0709 12:12:48.965504 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:49.906176 (+940672us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:49.910652 (+  4476us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:49.881035 15871 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:49.931910 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:49.945286 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.224s	user 0.019s	sys 0.026s
W20260709 12:12:49.946147 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s	user 0.020s	sys 0.026s
W20260709 12:12:49.955642 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:49.955840 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.748s	user 0.068s	sys 0.085s
I20260709 12:12:49.949254 15893 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:49.975773 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.793s	user 0.068s	sys 0.086s
W20260709 12:12:49.976905 15946 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.015s	user 0.067s	sys 0.118s
W20260709 12:12:49.977775 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.820s	user 0.021s	sys 0.115s
W20260709 12:12:49.978313 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.866s	user 0.021s	sys 0.116s
W20260709 12:12:49.979714 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.526s	user 0.038s	sys 0.118s
W20260709 12:12:49.980180 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.576s	user 0.038s	sys 0.118s
I20260709 12:12:49.981884 12563 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: [4.4-6.5]
W20260709 12:12:49.982518 13258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.014s	sys 0.019s
W20260709 12:12:49.983093 13258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.014s	sys 0.019s
I20260709 12:12:49.984614 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 580) took 1074 ms. Trace:
W20260709 12:12:49.985157 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.857s	user 0.048s	sys 0.153s
W20260709 12:12:49.985881 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.920s	user 0.050s	sys 0.154s
W20260709 12:12:49.995824 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.025s	sys 0.017s
W20260709 12:12:49.996466 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.941s	user 0.026s	sys 0.017s
I20260709 12:12:50.003732 12395 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:50.013697 15871 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:50.015198 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.019469 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.024953 15946 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.110s	user 0.068s	sys 0.118s
I20260709 12:12:49.985110 13258 rpcz_store.cc:276] 0709 12:12:48.909760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.909926 (+   166us) service_pool.cc:224] Handling call
0709 12:12:48.910463 (+   537us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:48.910762 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:12:49.002006 (+ 91244us) log.cc:844] Serialized 14 byte log entry
0709 12:12:49.983870 (+981864us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:49.984009 (+   139us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:49.984109 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:49.984247 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:49.984498 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":102904,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":10752,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":200,"thread_start_us":981161,"threads_started":1}
W20260709 12:12:50.042934 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.833s	user 0.026s	sys 0.021s
W20260709 12:12:50.054229 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.979s	user 0.022s	sys 0.023s
W20260709 12:12:50.054991 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s	user 0.022s	sys 0.023s
I20260709 12:12:50.056514 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2093) took 1169 ms. Trace:
W20260709 12:12:50.057694 12462 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.750s	user 0.009s	sys 0.036s
W20260709 12:12:50.058205 12462 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.858s	user 0.009s	sys 0.037s
W20260709 12:12:50.058686 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.85901e+06us
W20260709 12:12:50.060465 15995 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.080s	user 0.003s	sys 0.000s
W20260709 12:12:50.070170 12488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.797s	user 0.010s	sys 0.051s
W20260709 12:12:50.071189 12488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.826s	user 0.010s	sys 0.052s
W20260709 12:12:50.072417 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.920s	user 0.007s	sys 0.041s
W20260709 12:12:50.073297 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.954s	user 0.007s	sys 0.041s
W20260709 12:12:50.084683 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.894s	user 0.027s	sys 0.022s
I20260709 12:12:50.056939 12570 rpcz_store.cc:276] 0709 12:12:48.887500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.888059 (+   559us) service_pool.cc:224] Handling call
0709 12:12:48.888560 (+   501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:48.888883 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:48.972607 (+ 83724us) log.cc:844] Serialized 14 byte log entry
0709 12:12:50.055794 (+1083187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:50.055932 (+   138us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:50.056026 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.056162 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.056416 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":88835,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":9599,"raft.queue_time_us":189,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":147,"thread_start_us":1082273,"threads_started":1}
I20260709 12:12:50.122419 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:50.118686 13262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.173s	user 0.020s	sys 0.019s
W20260709 12:12:50.124596 15757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.335s	user 0.023s	sys 0.035s
W20260709 12:12:50.124799 13262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.254s	user 0.021s	sys 0.019s
W20260709 12:12:50.122740 13257 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.033s	user 0.004s	sys 0.035s
I20260709 12:12:50.128059 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:50.120005 15520 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:50.138564 13257 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.128s	user 0.005s	sys 0.036s
W20260709 12:12:50.140187 12496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.124s	user 0.015s	sys 0.038s
I20260709 12:12:50.140129 15803 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.141003 15803 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:50.144356 15803 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:50.140056 12609 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.114s	user 0.027s	sys 0.024s
W20260709 12:12:50.140098 15922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.298s	user 0.036s	sys 0.042s
W20260709 12:12:50.145397 12609 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.166s	user 0.027s	sys 0.025s
W20260709 12:12:50.145792 15922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.303s	user 0.036s	sys 0.042s
W20260709 12:12:50.145998 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.16689e+06us
W20260709 12:12:50.149773 12496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.187s	user 0.015s	sys 0.038s
I20260709 12:12:50.153599 15973 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:12:50.156222 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.137478 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1965) took 2116 ms. Trace:
I20260709 12:12:50.122157 12395 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 4, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:12:50.151621 15757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.352s	user 0.023s	sys 0.036s
W20260709 12:12:50.138212 15790 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.790s	user 0.009s	sys 0.025s
I20260709 12:12:50.157363 12569 rpcz_store.cc:276] 0709 12:12:48.018736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.018949 (+   213us) service_pool.cc:224] Handling call
0709 12:12:48.019268 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:48.036609 (+ 17341us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:48.119893 (+ 83284us) log.cc:844] Serialized 14 byte log entry
0709 12:12:50.112737 (+1992844us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:50.118795 (+  6058us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:50.118871 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.126244 (+  7373us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.134495 (+  8251us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":239980,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":26596,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":14034,"thread_start_us":1953544,"threads_started":2}
I20260709 12:12:50.171798 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.172824 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:12:50.173733 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.034s	user 0.003s	sys 0.051s
W20260709 12:12:50.178712 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.100s	user 0.003s	sys 0.051s
I20260709 12:12:50.155949 15622 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:50.179216 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:50.175812 13259 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.030s	user 0.012s	sys 0.028s
I20260709 12:12:50.182132 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:50.184067 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.10538e+06us
W20260709 12:12:50.180737 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.656s	user 0.014s	sys 0.040s
W20260709 12:12:50.189025 15564 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.534s	user 0.029s	sys 0.043s
W20260709 12:12:50.190383 13259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.118s	user 0.013s	sys 0.028s
W20260709 12:12:50.192317 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.744s	user 0.015s	sys 0.040s
I20260709 12:12:50.193710 15946 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:12:50.194995 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:50.196089 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.195928 15946 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:12:50.199966 15520 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:12:50.172683 15803 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
W20260709 12:12:50.205808 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.208472 12569 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:12:50.201886 15870 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.529s	user 0.017s	sys 0.000s
I20260709 12:12:50.209790 12570 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:50.210031 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.210230 15870 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.210965 12569 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:50.212507 12569 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:50.190745 15758 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.610s	user 0.024s	sys 0.048s
W20260709 12:12:50.203208 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.999s	user 0.052s	sys 0.045s
I20260709 12:12:50.211392 12570 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:12:50.225124 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.106s	user 0.053s	sys 0.045s
W20260709 12:12:50.205497 15564 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.620s	user 0.030s	sys 0.043s
W20260709 12:12:50.230618 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.227936 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:50.228557 15758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.714s	user 0.025s	sys 0.048s
I20260709 12:12:50.204972 15871 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:50.236970 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.78872e+06us
I20260709 12:12:50.242038 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.247159 12570 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
W20260709 12:12:50.214587 15790 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.866s	user 0.010s	sys 0.026s
I20260709 12:12:50.234913 15863 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.258095 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:50.255707 15922 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.260538 15922 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.262022 15870 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:50.263717 15922 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:50.264533 15922 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.264832 16011 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:50.237742 12395 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 5, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:50.265345 15922 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.272886 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:50.244550 15622 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.283865 13254 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:12:50.265573 16011 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:50.289510 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.307057 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.291643 12395 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 2 to 5, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:50.312495 12570 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:50.253311 15973 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:50.315802 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.317886 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.289915 15871 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.293193 15988 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:50.307585 16011 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:50.330654 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.311249 15832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 4.819s	user 0.027s	sys 0.045s
W20260709 12:12:50.343880 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.317030 15973 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:12:50.341750 12609 connection.cc:256] RPC call timeout handler was delayed by -1.0782551640002112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
I20260709 12:12:50.346496 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.330350 15988 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:50.345436 15622 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.358596 12395 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:50.333068 12569 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:50.297919 15863 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.360677 12395 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 5 to 7, pending config changed. New cstate: current_term: 7 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:12:50.392247 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2115) took 1038 ms. Trace:
I20260709 12:12:50.392796 12569 rpcz_store.cc:276] 0709 12:12:49.327429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:50.212039 (+884610us) service_pool.cc:224] Handling call
0709 12:12:50.212387 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:50.365656 (+153269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.366109 (+   453us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:50.404199 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.408586 15622 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:50.409493 15622 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:12:50.410903 15944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.018s	user 0.037s	sys 0.036s
W20260709 12:12:50.411504 15944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.065s	user 0.037s	sys 0.036s
I20260709 12:12:50.412569 15944 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:12:50.414939 15999 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:12:50.417294 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1976) took 2317 ms. Trace:
I20260709 12:12:50.418565 15893 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:50.419865 15893 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:50.417923 12568 rpcz_store.cc:276] 0709 12:12:48.104043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.104267 (+   224us) service_pool.cc:224] Handling call
0709 12:12:48.104623 (+   356us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:48.110225 (+  5602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:48.110258 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:48.124078 (+ 13820us) log.cc:844] Serialized 52 byte log entry
0709 12:12:48.631013 (+506935us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:12:49.995396 (+1364383us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:49.995474 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:49.995638 (+   164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:50.416695 (+421057us) raft_consensus.cc:1685] finished
0709 12:12:50.416730 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.417178 (+   448us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1578,"raft.queue_time_us":1747,"raft.run_cpu_time_us":3130,"raft.run_wall_time_us":522391,"spinlock_wait_cycles":3641728,"thread_start_us":1812519,"threads_started":1,"wal-append.queue_time_us":2072685}
I20260709 12:12:50.421799 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.422783 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:50.424301 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.357672 15988 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:50.426100 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.426937 12568 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:50.428085 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.348294 15871 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:50.398334 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77269.9us
I20260709 12:12:50.438637 13258 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:12:50.439214 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.439601 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.429528 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.441862 12568 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:50.443667 15816 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.795s	user 0.014s	sys 0.004s
I20260709 12:12:50.443799 13254 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:12:50.444267 15816 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.447387 15816 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:50.449179 15907 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:50.452682 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2121) took 1122 ms. Trace:
I20260709 12:12:50.453049 15863 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:50.453250 12568 rpcz_store.cc:276] 0709 12:12:49.330538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:50.451443 (+1120905us) service_pool.cc:224] Handling call
0709 12:12:50.451869 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:50.452263 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:50.452292 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:50.452350 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:50.452398 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.452461 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.452588 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:50.453089 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:50.456941 12568 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:50.458500 15922 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:50.457392 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2235) took 2694 ms (client timeout 1649 ms). Trace:
W20260709 12:12:50.461181 12493 rpcz_store.cc:269] 0709 12:12:47.763263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:47.780293 (+ 17030us) service_pool.cc:224] Handling call
0709 12:12:50.456955 (+2676662us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:50.457145 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6040998272}
I20260709 12:12:50.466465 12395 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 3 to 4, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 4 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:50.468362 12395 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 5 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 6 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:12:50.470505 12569 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:50.471280 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:12:50.353157 15832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 5.007s	user 0.036s	sys 0.047s
W20260709 12:12:50.488561 15962 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.022s	user 0.011s	sys 0.048s
W20260709 12:12:50.485044 12536 connection.cc:256] RPC call timeout handler was delayed by -3.44388737700001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260709 12:12:50.492370 15915 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.030s	sys 0.029s
W20260709 12:12:50.489009 15962 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.168s	user 0.011s	sys 0.049s
I20260709 12:12:50.487906 15758 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:50.498667 12395 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 6 to 7, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 7 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:12:50.491569 12536 connection.cc:256] RPC call timeout handler was delayed by -3.44388737700001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260709 12:12:50.502097 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.517439 12536 connection.cc:256] RPC call timeout handler was delayed by -3.4384873770000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260709 12:12:50.520936 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3088873770000191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20260709 12:12:50.522287 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3043873770000118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260709 12:12:50.496656 15915 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.030s	sys 0.029s
W20260709 12:12:50.528786 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2575873769999362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
I20260709 12:12:50.513777 15893 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:50.533696 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2386873769999056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
I20260709 12:12:50.538143 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1975) took 2444 ms. Trace:
W20260709 12:12:50.542200 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.529768 15907 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:50.556201 12494 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
W20260709 12:12:50.555739 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2386873769999056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260709 12:12:50.558454 12536 connection.cc:256] RPC call timeout handler was delayed by -3.1936873770000602s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18330000000000002s for a response
W20260709 12:12:50.559274 12536 connection.cc:256] RPC call timeout handler was delayed by -3.0973873769999045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
W20260709 12:12:50.559742 12536 connection.cc:256] RPC call timeout handler was delayed by -3.0208873770000082s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2025s for a response
I20260709 12:12:50.557170 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1974) took 2478 ms. Trace:
I20260709 12:12:50.552186 12565 rpcz_store.cc:276] 0709 12:12:48.093868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.094154 (+   286us) service_pool.cc:224] Handling call
0709 12:12:48.094673 (+   519us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:48.095227 (+   554us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:48.095268 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:48.095937 (+   669us) log.cc:844] Serialized 26 byte log entry
0709 12:12:49.979179 (+1883242us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:49.979272 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:49.979445 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:50.537883 (+558438us) raft_consensus.cc:1685] finished
0709 12:12:50.537901 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.538084 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3267,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":5278,"thread_start_us":1866681,"threads_started":1,"wal-append.queue_time_us":2412080}
W20260709 12:12:50.564852 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9812873769999442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260709 12:12:50.569114 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9812873769999442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260709 12:12:50.571080 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9722873769999296s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
I20260709 12:12:50.563902 12564 rpcz_store.cc:276] 0709 12:12:48.079558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.079732 (+   174us) service_pool.cc:224] Handling call
0709 12:12:48.103463 (+ 23731us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:48.103820 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:48.103848 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:48.107215 (+  3367us) log.cc:844] Serialized 26 byte log entry
0709 12:12:48.673542 (+566327us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b1800468040. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:12:49.946916 (+1273374us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:12:49.947017 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:49.947166 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:50.541718 (+594552us) raft_consensus.cc:1685] finished
0709 12:12:50.541749 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.557077 (+ 15328us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1925,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":2005,"spinlock_wait_cycles":3443584,"thread_start_us":1272874,"threads_started":1,"wal-append.queue_time_us":2412850}
I20260709 12:12:50.526623 15758 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:50.605933 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9722873769999296s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
W20260709 12:12:50.613015 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9587692090001383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260709 12:12:50.615893 12536 connection.cc:256] RPC call timeout handler was delayed by -2.9587692090001383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
I20260709 12:12:50.576292 15907 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:50.619899 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2247) took 2798 ms. Trace:
I20260709 12:12:50.611596 15871 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:50.623824 12494 rpcz_store.cc:276] 0709 12:12:47.821085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.495527 (+674442us) service_pool.cc:224] Handling call
0709 12:12:48.495866 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:50.561374 (+2065508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.615345 (+ 53971us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:50.619778 (+  4433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4709604736}
I20260709 12:12:50.647205 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2241) took 2882 ms. Trace:
W20260709 12:12:50.653585 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.660557 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 557) took 1600 ms. Trace:
W20260709 12:12:50.632725 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8957692090002638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260709 12:12:50.664382 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8957692090002638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260709 12:12:50.668779 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7634692090002773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
W20260709 12:12:50.669271 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7634692090002773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
W20260709 12:12:50.669787 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7319692090002263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18530000000000002s for a response
W20260709 12:12:50.670250 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7319692090002263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18530000000000002s for a response
W20260709 12:12:50.670625 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7148692090001987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18720000000000003s for a response
W20260709 12:12:50.671141 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7148692090001987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18720000000000003s for a response
W20260709 12:12:50.671603 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6977873769999405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2384s for a response
W20260709 12:12:50.673745 13218 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.024s	sys 0.031s
I20260709 12:12:50.662053 13259 rpcz_store.cc:276] 0709 12:12:49.060197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:49.060431 (+   234us) service_pool.cc:224] Handling call
0709 12:12:49.070325 (+  9894us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:49.070906 (+   581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:49.070947 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:49.071873 (+   926us) log.cc:844] Serialized 52 byte log entry
0709 12:12:50.254315 (+1182442us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:50.276349 (+ 22034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.287812 (+ 11463us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:50.660142 (+372330us) raft_consensus.cc:1685] finished
0709 12:12:50.660173 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.660466 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1888,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":1744,"thread_start_us":1119094,"threads_started":1,"wal-append.queue_time_us":1585083}
W20260709 12:12:50.702443 13218 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.025s	sys 0.031s
I20260709 12:12:50.648675 12496 rpcz_store.cc:276] 0709 12:12:47.764791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:47.872084 (+107293us) service_pool.cc:224] Handling call
0709 12:12:47.872434 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:47.961423 (+ 88989us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:47.961529 (+   106us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:47.962643 (+  1114us) log.cc:844] Serialized 26 byte log entry
0709 12:12:50.153020 (+2190377us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:50.153105 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.154391 (+  1286us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:50.646449 (+492058us) raft_consensus.cc:1685] finished
0709 12:12:50.646473 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.647126 (+   653us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8192,"raft.queue_time_us":93417,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":2665,"thread_start_us":2273861,"threads_started":2,"wal-append.queue_time_us":2658427}
W20260709 12:12:50.665889 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.012s	sys 0.048s
W20260709 12:12:50.704118 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.012s	sys 0.048s
W20260709 12:12:50.704576 12571 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.004s	sys 0.053s
W20260709 12:12:50.705701 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.004s	sys 0.053s
W20260709 12:12:50.701094 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6977873769999405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2384s for a response
W20260709 12:12:50.707546 12536 connection.cc:256] RPC call timeout handler was delayed by -2.677969209000139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260709 12:12:50.709486 12536 connection.cc:256] RPC call timeout handler was delayed by -2.677969209000139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260709 12:12:50.711189 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3791692090001106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22450000000000003s for a response
W20260709 12:12:50.703714 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 951587us
W20260709 12:12:50.722452 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3791692090001106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22450000000000003s for a response
W20260709 12:12:50.724742 12536 connection.cc:256] RPC call timeout handler was delayed by -2.371969209000099s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22530000000000003s for a response
W20260709 12:12:50.727005 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2405873770001108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2892s for a response
W20260709 12:12:50.729905 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9570692090001103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27140000000000003s for a response
W20260709 12:12:50.732350 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9570692090001103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27140000000000003s for a response
W20260709 12:12:50.732653 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.736289 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8355873769999107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33420000000000005s for a response
W20260709 12:12:50.739303 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8355873769999107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33420000000000005s for a response
W20260709 12:12:50.742048 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6906873769999038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.35030000000000006s for a response
W20260709 12:12:50.743702 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6906873769999038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.35030000000000006s for a response
W20260709 12:12:50.745654 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5169692090000808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32030000000000003s for a response
W20260709 12:12:50.747349 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5169692090000808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32030000000000003s for a response
I20260709 12:12:50.676594 15962 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:50.802186 12572 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.018s	sys 0.032s
W20260709 12:12:50.804175 12572 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.018s	sys 0.032s
W20260709 12:12:50.804035 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84067.7us
W20260709 12:12:50.807785 12642 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.022s	sys 0.023s
W20260709 12:12:50.809183 12642 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.022s	sys 0.023s
W20260709 12:12:50.808605 12566 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.014s	sys 0.014s
W20260709 12:12:50.812373 12566 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.014s	sys 0.014s
I20260709 12:12:50.667457 15973 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:50.808210 12563 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.013s	sys 0.013s
W20260709 12:12:50.810588 12567 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.000s	sys 0.025s
W20260709 12:12:50.820369 12563 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.014s	sys 0.014s
W20260709 12:12:50.821899 12567 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.000s	sys 0.026s
W20260709 12:12:50.824132 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.825316 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1972) took 2788 ms. Trace:
I20260709 12:12:50.828438 12566 rpcz_store.cc:276] 0709 12:12:48.050817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.063768 (+ 12951us) service_pool.cc:224] Handling call
0709 12:12:48.064211 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:48.064652 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:48.064687 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:48.065266 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 12:12:49.986693 (+1921427us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:50.812845 (+826152us) log.cc:844] Serialized 14 byte log entry
0709 12:12:50.820670 (+  7825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.824655 (+  3985us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:50.824706 (+    51us) raft_consensus.cc:1685] finished
0709 12:12:50.824718 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.825252 (+   534us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7802,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":7870,"thread_start_us":2746106,"threads_started":2,"wal-append.queue_time_us":2291716}
I20260709 12:12:50.844934 15871 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:50.851619 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.853205 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.003s	sys 0.008s
W20260709 12:12:50.853493 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.003s	sys 0.036s
W20260709 12:12:50.853726 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.004s	sys 0.008s
W20260709 12:12:50.853844 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.004s	sys 0.036s
I20260709 12:12:50.810652 16011 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:50.855067 15757 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.005s	sys 0.006s
W20260709 12:12:50.880762 15757 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.005s	sys 0.006s
I20260709 12:12:50.882633 16011 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:50.841035 15988 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:50.877005 13257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.005s	sys 0.010s
W20260709 12:12:50.845810 15790 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.005s	sys 0.013s
W20260709 12:12:50.852136 16021 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.134s	user 0.001s	sys 0.001s
I20260709 12:12:50.819010 16004 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
W20260709 12:12:50.858021 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 658317us
W20260709 12:12:50.885767 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 478270us
I20260709 12:12:50.846882 15758 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:50.887650 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.005s	sys 0.013s
W20260709 12:12:50.889775 13257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.006s	sys 0.010s
I20260709 12:12:50.846585 15907 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:50.858889 15946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.004s	sys 0.008s
W20260709 12:12:50.888367 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.889988 15790 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.005s	sys 0.013s
W20260709 12:12:50.919263 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.009s	sys 0.027s
W20260709 12:12:50.929682 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.010s	sys 0.027s
W20260709 12:12:50.879222 15803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.000s	sys 0.016s
I20260709 12:12:50.892306 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:50.938539 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.005s	sys 0.023s
W20260709 12:12:50.944766 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.005s	sys 0.023s
W20260709 12:12:50.921280 15946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.004s	sys 0.009s
W20260709 12:12:50.938592 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.008s	sys 0.020s
W20260709 12:12:50.948257 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95389.9us
W20260709 12:12:50.954501 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.009s	sys 0.020s
I20260709 12:12:50.944682 13259 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
W20260709 12:12:50.927764 15564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.003s	sys 0.023s
I20260709 12:12:50.960871 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2108) took 1638 ms. Trace:
I20260709 12:12:50.850400 15962 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:50.964850 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46950.1us
W20260709 12:12:50.942880 15803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.000s	sys 0.022s
I20260709 12:12:50.961655 12571 rpcz_store.cc:276] 0709 12:12:49.322901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:49.844604 (+521703us) service_pool.cc:224] Handling call
0709 12:12:49.861309 (+ 16705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:49.878221 (+ 16912us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:50.707058 (+828837us) log.cc:844] Serialized 14 byte log entry
0709 12:12:50.960394 (+253336us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:50.960474 (+    80us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:50.960534 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.960633 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:50.960803 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1009514,"thread_start_us":967207,"threads_started":2}
W20260709 12:12:50.966735 12609 connection.cc:256] RPC call timeout handler was delayed by -1.0287551640001311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18610000000000002s for a response
W20260709 12:12:50.971107 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.979015 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.979774 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.924787 15757 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:50.855598 15915 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.092s
W20260709 12:12:50.960482 15564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.004s	sys 0.023s
W20260709 12:12:50.983726 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:50.983867 16015 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:12:50.973017 15893 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:50.991734 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:50.998299 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:50.891440 15871 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:50.997576 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:50.960788 15907 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.004292 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:51.006989 16018 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.310s	user 0.000s	sys 0.002s
I20260709 12:12:51.007453 16027 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.007535 15973 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:51.008076 16027 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:51.008579 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 551) took 2162 ms. Trace:
I20260709 12:12:51.008728 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.009093 13262 rpcz_store.cc:276] 0709 12:12:48.846405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.846578 (+   173us) service_pool.cc:224] Handling call
0709 12:12:48.847069 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:48.858841 (+ 11772us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:48.858890 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:48.859552 (+   662us) log.cc:844] Serialized 26 byte log entry
0709 12:12:50.149206 (+1289654us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:50.149288 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.149397 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:51.008304 (+858907us) raft_consensus.cc:1685] finished
0709 12:12:51.008328 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:51.008488 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90557,"raft.queue_time_us":27095,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":138,"thread_start_us":1254233,"threads_started":1,"wal-append.queue_time_us":1747744}
I20260709 12:12:51.011956 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.012411 15893 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.013168 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.013892 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.014385 12646 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:12:51.014772 12639 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:51.015058 15962 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:51.015865 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.016034 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.016996 12638 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:12:51.017513 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.018252 12643 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:51.020583 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:51.021459 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:51.022158 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:51.022727 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:51.024575 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.026974 12637 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 11 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:12:51.023597 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:50.987412 15757 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:51.022746 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:51.018267 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:51.070966 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.013059 15816 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:51.085841 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:51.123138 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.109853 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:51.128244 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:12:51.085201 16008 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:12:51.051858 12571 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:51.129397 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:51.044751 12640 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:51.050343 16047 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:51.020583 16000 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
W20260709 12:12:51.106076 16044 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 12:12:51.105783 16019 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
W20260709 12:12:51.139700 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90324.6us
I20260709 12:12:51.140841 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:51.152025 12638 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:51.142035 15832 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:12:51.136212 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:51.155756 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 897791us
I20260709 12:12:51.042124 15870 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:51.158151 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.144270 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:51.050442 16038 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.164757 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:12:51.159386 15622 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:51.166067 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:51.085506 16035 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.189s
W20260709 12:12:51.147610 12610 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.233s (SENT) [suppressed 15 similar messages]
W20260709 12:12:51.203243 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 152275us
W20260709 12:12:51.204799 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:51.256660 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104545us
I20260709 12:12:51.300276 15520 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:51.321431 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:51.322239 15520 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:51.240622 12638 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:12:51.299258 16003 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:51.247382 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:51.322010 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 299407us
W20260709 12:12:51.325209 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77933.7us
I20260709 12:12:51.286185 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:51.264842 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 118919us
I20260709 12:12:51.168996 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
I20260709 12:12:51.216144 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1847) took 2880 ms. Trace:
I20260709 12:12:51.242616 15757 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:51.347939 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:51.299520 16056 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:12:51.344889 16062 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:51.166731 15907 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.239135 16045 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.190s
I20260709 12:12:51.297271 15870 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:51.159825 15832 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:51.236539 16039 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
W20260709 12:12:51.352349 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 204751us
I20260709 12:12:51.242511 13254 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 4. Current term is 6. Ops: []
I20260709 12:12:51.355608 12609 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9.
I20260709 12:12:51.433710 12609 leader_election.cc:403] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:12:51.329116 12642 rpcz_store.cc:276] 0709 12:12:48.303272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.697642 (+394370us) service_pool.cc:224] Handling call
0709 12:12:48.765078 (+ 67436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:48.852391 (+ 87313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:48.852442 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:48.991174 (+138732us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:12:48.991782 (+   608us) log.cc:844] Serialized 26 byte log entry
0709 12:12:49.997163 (+1005381us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:51.134999 (+1137836us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b1800357700. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:12:51.135144 (+   145us) log.cc:844] Serialized 14 byte log entry
0709 12:12:51.149352 (+ 14208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:51.159343 (+  9991us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:51.159404 (+    61us) raft_consensus.cc:1685] finished
0709 12:12:51.159418 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:51.183442 (+ 24024us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67068,"prepare.queue_time_us":812108,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":2402,"raft.queue_time_us":160054,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":183,"spinlock_wait_cycles":117053440,"thread_start_us":1753993,"threads_started":2,"wal-append.queue_time_us":1276940}
I20260709 12:12:51.402462 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:51.354383 12494 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:51.358206 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:51.360517 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 288414us
I20260709 12:12:51.408634 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:12:51.405155 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47628.2us
I20260709 12:12:51.410530 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:51.435642 13261 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:51.413849 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92056.7us
I20260709 12:12:51.245853 15622 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:51.324649 15816 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:51.242087 16038 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:51.238085 15893 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:51.357223 16046 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.415357 12570 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:51.360146 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:51.356563 15757 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:51.330847 16044 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:12:51.187073 15876 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:51.299530 15944 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:51.357298 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:51.431972 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.239428 16042 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.297410 15988 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:51.430835 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.241771 16053 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:51.439742 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5.
I20260709 12:12:51.469434 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:51.470155 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:51.435948 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:51.471709 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.444129 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:51.445096 16036 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.445815 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:51.446400 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:51.446050 15816 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.477578 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.478603 16036 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:51.447155 15870 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:51.347011 16027 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:51.450361 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:51.451614 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.456292 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:51.421060 16062 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:51.461043 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:51.458930 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 552) took 2612 ms. Trace:
I20260709 12:12:51.457126 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:51.474539 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.475090 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.475967 12641 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:51.476912 15564 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9.
I20260709 12:12:51.474010 12495 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:51.485677 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:51.361207 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.480006 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:51.481184 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.458493 13255 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:51.477542 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.481171 12640 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:51.479249 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:51.226558 16047 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:51.477505 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122282us
I20260709 12:12:51.487797 12568 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:51.493103 16041 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.495996 12497 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:51.496694 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:51.496840 16059 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:51.496469 16041 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.502425 13262 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:51.473006 16038 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:51.497526 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:51.496312 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:51.437497 16003 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:51.512741 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56621.6us
I20260709 12:12:51.484563 13257 rpcz_store.cc:276] 0709 12:12:48.846740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.846893 (+   153us) service_pool.cc:224] Handling call
0709 12:12:48.871907 (+ 25014us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:12:48.872124 (+   217us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:48.938145 (+ 66021us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:12:50.181324 (+1243179us) log.cc:844] Serialized 14 byte log entry
0709 12:12:50.959407 (+778083us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:51.149786 (+190379us) log.cc:844] Serialized 26 byte log entry
0709 12:12:51.159712 (+  9926us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:51.159755 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:51.183692 (+ 23937us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:51.449516 (+265824us) raft_consensus.cc:1685] finished
0709 12:12:51.458658 (+  9142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:51.458856 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73755,"prepare.queue_time_us":84609,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":74597,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":15022,"spinlock_wait_cycles":3062016,"thread_start_us":1815535,"threads_started":2,"wal-append.queue_time_us":801859}
I20260709 12:12:51.485718 16062 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:51.518352 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68066.2us
I20260709 12:12:51.518448 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.519232 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.494093 15907 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:51.522511 16035 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:51.522969 16035 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:51.523344 16035 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:51.526070 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:51.534267 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:51.483438 15876 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.544822 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.565153 12495 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:12:51.539402 15915 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:51.566752 15915 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:51.568225 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.567214 15915 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:51.579303 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:51.535348 16006 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:51.485458 15944 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:51.602111 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:51.606019 16003 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:51.610952 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161537us
I20260709 12:12:51.494304 16042 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:51.483642 16046 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.609236 12490 raft_consensus.cc:2376] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 5. Re-sending same reply.
I20260709 12:12:51.616003 16041 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:51.618369 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:51.634003 13261 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:51.636075 16003 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:51.637123 16003 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:51.638880 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124174us
I20260709 12:12:51.621956 16069 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:51.641489 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:51.647318 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 254090us
I20260709 12:12:51.654035 16003 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:51.642505 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:51.656205 12491 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:12:51.513924 15832 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.659215 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:51.659207 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57483.6us
I20260709 12:12:51.662484 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.662534 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:51.663245 12495 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:51.667357 12490 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:51.655774 16069 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:51.668315 16071 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:51.670071 16071 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:51.670282 16075 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:51.670501 16071 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:51.670877 16075 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:51.672065 16075 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:51.677772 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:51.678514 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:51.679018 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:51.681344 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:51.694067 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:51.698153 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.699586 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:51.700716 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:51.702719 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63881us
I20260709 12:12:51.704835 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:51.706027 16062 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.701318 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:51.718962 12564 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:12:51.721820 16062 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:51.722092 16071 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:51.689014 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:51.727758 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:51.764055 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 320971us
I20260709 12:12:51.768016 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:51.771766 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2107) took 2449 ms. Trace:
W20260709 12:12:52.242221 13218 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.034s	sys 0.072s
W20260709 12:12:52.224417 12568 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.732s	user 0.007s	sys 0.006s
W20260709 12:12:51.953289 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.243203 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
W20260709 12:12:51.855171 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87304.3us
W20260709 12:12:52.243906 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:52.243119 13218 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.034s	sys 0.072s
W20260709 12:12:52.244673 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 563288us
W20260709 12:12:52.238626 12570 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.781s	user 0.079s	sys 0.129s
W20260709 12:12:52.224113 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 557180us
I20260709 12:12:52.246244 12570 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 12:12:52.225139 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 564077us
W20260709 12:12:52.246706 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.223780 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:51.678318 16069 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:52.224987 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 1982) took 1206 ms. Trace:
W20260709 12:12:52.111805 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.051766 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:52.248983 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:52.263185 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 558397us
I20260709 12:12:52.257987 12638 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:52.248318 12566 rpcz_store.cc:276] 0709 12:12:51.018691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.019241 (+   550us) service_pool.cc:224] Handling call
0709 12:12:51.019591 (+   350us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:51.430493 (+410902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:51.430553 (+    60us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:51.472305 (+ 41752us) log.cc:844] Serialized 52 byte log entry
0709 12:12:51.609831 (+137526us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:51.773900 (+164069us) log.cc:844] Serialized 14 byte log entry
0709 12:12:52.224623 (+450723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.224698 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:52.224747 (+    49us) raft_consensus.cc:1685] finished
0709 12:12:52.224757 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.224930 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":171341,"prepare.queue_time_us":163992,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":195419,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":6887,"thread_start_us":388934,"threads_started":2,"wal-append.queue_time_us":164119}
I20260709 12:12:52.266054 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:52.267124 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.159329 12642 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:12:52.219112 15915 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.639s	user 0.009s	sys 0.006s
I20260709 12:12:51.762342 16003 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:12:51.862118 16070 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:12:51.863495 16072 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:12:52.275859 12638 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:12:51.887862 16015 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.817s	user 0.003s	sys 0.000s
I20260709 12:12:51.851279 16041 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:12:52.172719 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:51.738163 16047 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:52.269099 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:52.268708 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:51.772333 12567 rpcz_store.cc:276] 0709 12:12:49.322367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:49.755636 (+433269us) service_pool.cc:224] Handling call
0709 12:12:49.853766 (+ 98130us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:49.896890 (+ 43124us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:12:50.823235 (+926345us) log.cc:844] Serialized 14 byte log entry
0709 12:12:51.139804 (+316569us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b18003f4900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:12:51.771060 (+631256us) log.cc:844] Serialized 14 byte log entry
0709 12:12:51.771311 (+   251us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:51.771378 (+    67us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:12:51.771430 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:51.771530 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:51.771692 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1298170,"prepare.queue_time_us":1596019,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":87,"raft.queue_time_us":158837,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":200,"spinlock_wait_cycles":15177600,"thread_start_us":1056565,"threads_started":3,"wal-append.queue_time_us":191969}
I20260709 12:12:51.855419 15816 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:52.292069 13260 raft_consensus.cc:2368] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:12:52.213956 16046 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:52.292788 16046 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:52.293246 16046 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:52.293599 16046 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:52.295435 16049 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.129s	user 0.000s	sys 0.012s
I20260709 12:12:52.295660 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:52.296048 16062 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:52.296692 12643 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
W20260709 12:12:52.247437 16061 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.032s	sys 0.067s
W20260709 12:12:52.289239 12492 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.002s
I20260709 12:12:52.285125 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:51.776428 15973 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:52.236896 16042 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:51.865810 16064 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:12:51.772760 16036 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:52.297722 12492 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:52.307783 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2112) took 2983 ms. Trace:
I20260709 12:12:52.308204 12563 rpcz_store.cc:276] 0709 12:12:49.324805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:49.983711 (+658906us) service_pool.cc:224] Handling call
0709 12:12:49.984346 (+   635us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:50.006181 (+ 21835us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:12:50.822490 (+816309us) log.cc:844] Serialized 14 byte log entry
0709 12:12:50.999460 (+176970us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:51.037797 (+ 38337us) log.cc:844] Serialized 52 byte log entry
0709 12:12:51.123786 (+ 85989us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:51.771160 (+647374us) log.cc:844] Serialized 14 byte log entry
0709 12:12:51.771754 (+   594us) log.cc:844] Serialized 14 byte log entry
0709 12:12:51.771987 (+   233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:51.772108 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:52.294822 (+522714us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:12:52.307481 (+ 12659us) raft_consensus.cc:1685] finished
0709 12:12:52.307507 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.307689 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1409966,"prepare.queue_time_us":1635023,"prepare.run_cpu_time_us":204,"prepare.run_wall_time_us":1871,"raft.queue_time_us":15959,"raft.run_cpu_time_us":671,"raft.run_wall_time_us":20499,"spinlock_wait_cycles":2893696,"thread_start_us":966103,"threads_started":2,"wal-append.queue_time_us":293789}
I20260709 12:12:52.300318 12537 leader_election.cc:400] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:12:52.283473 16003 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:52.311172 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:52.311686 12537 leader_election.cc:403] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260709 12:12:52.301956 16061 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.032s	sys 0.068s
I20260709 12:12:52.319347 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:52.323640 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:52.321353 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 404) took 5565 ms. Trace:
I20260709 12:12:52.286783 15915 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:52.322940 16071 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:52.314888 16003 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:52.330714 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51071.2us
I20260709 12:12:52.336268 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:52.363626 12644 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:52.366333 12492 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:12:52.364943 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 413) took 1015 ms. Trace:
I20260709 12:12:52.375986 12493 rpcz_store.cc:276] 0709 12:12:51.323395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.323584 (+   189us) service_pool.cc:224] Handling call
0709 12:12:51.323965 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:51.597577 (+273612us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:51.597612 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:51.597684 (+    72us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:12:51.597715 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.338998 (+741283us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:52.339054 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.339225 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560709376}
I20260709 12:12:52.319375 15973 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:52.391521 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 526) took 1024 ms. Trace:
I20260709 12:12:52.392216 12564 rpcz_store.cc:276] 0709 12:12:51.367393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.460340 (+ 92947us) service_pool.cc:224] Handling call
0709 12:12:52.391246 (+930906us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:52.391428 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":533771648}
I20260709 12:12:52.404855 15871 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:52.430423 15871 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:52.437237 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:52.423923 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.414419 12489 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:52.453195 15622 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:52.453307 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:52.455861 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:52.447482 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:52.449316 15944 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:52.459889 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 423) took 1101 ms. Trace:
I20260709 12:12:52.420748 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 517) took 2256 ms. Trace:
I20260709 12:12:52.460440 13258 rpcz_store.cc:276] 0709 12:12:51.358427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.401855 (+ 43428us) service_pool.cc:224] Handling call
0709 12:12:52.447223 (+1045368us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:52.459775 (+ 12552us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3588480}
I20260709 12:12:52.460444 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1993) took 1293 ms. Trace:
I20260709 12:12:52.462857 12570 rpcz_store.cc:276] 0709 12:12:51.182502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.215450 (+ 32948us) service_pool.cc:224] Handling call
0709 12:12:52.460192 (+1244742us) spinlock_profiling.cc:243] Waited 88.7 ms on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:52.460341 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":203966208}
I20260709 12:12:52.456645 15816 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:52.464298 15816 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:52.464800 15816 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:52.467859 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2251) took 1015 ms. Trace:
I20260709 12:12:52.468454 12489 rpcz_store.cc:276] 0709 12:12:51.452353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.477689 (+ 25336us) service_pool.cc:224] Handling call
0709 12:12:51.478249 (+   560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:52.452642 (+974393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.467462 (+ 14820us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:52.467771 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2151173376}
I20260709 12:12:52.470599 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.451052 15520 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:52.460964 12569 rpcz_store.cc:276] 0709 12:12:50.165188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:50.538014 (+372826us) service_pool.cc:224] Handling call
0709 12:12:50.538364 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:50.538755 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:50.928394 (+389639us) log.cc:844] Serialized 14 byte log entry
0709 12:12:52.411685 (+1483291us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b18004a16a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:52.419959 (+  8274us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b18004a16a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:12:52.420245 (+   286us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:52.420322 (+    77us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:52.420386 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.420484 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.420665 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6212,"prepare.queue_time_us":1227978,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":103003,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":24657,"spinlock_wait_cycles":87249024,"thread_start_us":622853,"threads_started":3,"wal-append.queue_time_us":165488,"wal-append.run_cpu_time_us":1502,"wal-append.run_wall_time_us":1022088}
I20260709 12:12:52.474056 15520 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:52.474123 15622 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:12:52.478992 15922 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:52.479633 15922 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:52.323323 16042 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:52.490284 16042 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:52.491607 16042 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:52.491837 15922 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:52.493028 15922 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:52.494061 12643 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 4: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:52.495675 15915 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:52.497414 16076 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:12:52.501813 15973 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:52.503033 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 415) took 1178 ms. Trace:
I20260709 12:12:52.503652 12491 rpcz_store.cc:276] 0709 12:12:51.324132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.360696 (+ 36564us) service_pool.cc:224] Handling call
0709 12:12:52.502682 (+1141986us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:52.502904 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":389717888}
I20260709 12:12:52.506150 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1891) took 1052 ms. Trace:
I20260709 12:12:52.506312 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.507602 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2263) took 1050 ms. Trace:
I20260709 12:12:52.508411 12497 rpcz_store.cc:276] 0709 12:12:51.456740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.497000 (+ 40260us) service_pool.cc:224] Handling call
0709 12:12:52.506131 (+1009131us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:12:52.507515 (+  1384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11586944}
W20260709 12:12:52.513924 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.496045 12643 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:52.505218 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.478994 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.324767 12488 rpcz_store.cc:276] 0709 12:12:46.751960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:46.754906 (+  2946us) service_pool.cc:224] Handling call
0709 12:12:46.756994 (+  2088us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:46.757434 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:46.757461 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:48.243622 (+1486161us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:12:48.244338 (+   716us) log.cc:844] Serialized 26 byte log entry
0709 12:12:50.071957 (+1827619us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:50.072032 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:50.072218 (+   186us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:52.282385 (+2210167us) spinlock_profiling.cc:243] Waited 78.3 ms on lock 0x7b24001215f8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:12:52.313026 (+ 30641us) raft_consensus.cc:1685] finished
0709 12:12:52.313049 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.317325 (+  4276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34210,"raft.queue_time_us":109,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":19858,"spinlock_wait_cycles":188858624,"thread_start_us":1826823,"threads_started":1,"wal-append.queue_time_us":2455655}
I20260709 12:12:52.323117 16047 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:12:52.555824 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.375581 15988 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:52.563776 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:52.565428 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 265349us
I20260709 12:12:52.556463 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.567518 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:52.567214 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 304529us
W20260709 12:12:52.579054 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:52.581151 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 315143us
I20260709 12:12:52.603933 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:52.605127 12489 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:52.612175 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2256) took 1158 ms. Trace:
I20260709 12:12:52.612857 12488 rpcz_store.cc:276] 0709 12:12:51.454180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.554820 (+1100640us) service_pool.cc:224] Handling call
0709 12:12:52.612065 (+ 57245us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:52.586055 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 362454us
I20260709 12:12:52.625854 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:52.635627 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136333us
I20260709 12:12:52.506906 12639 rpcz_store.cc:276] 0709 12:12:51.453604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.460303 (+  6699us) service_pool.cc:224] Handling call
0709 12:12:51.486493 (+ 26190us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:51.532879 (+ 46386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:51.532925 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:51.533823 (+   898us) log.cc:844] Serialized 52 byte log entry
0709 12:12:51.604303 (+ 70480us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:51.604661 (+   358us) log.cc:844] Serialized 14 byte log entry
0709 12:12:51.604874 (+   213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:51.604949 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:52.505769 (+900820us) raft_consensus.cc:1685] finished
0709 12:12:52.505803 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.506025 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":610,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":3171,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":58095,"spinlock_wait_cycles":7936,"thread_start_us":69896,"threads_started":1,"wal-append.queue_time_us":124980}
I20260709 12:12:52.578105 15832 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:52.648283 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.650887 12637 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:52.661965 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.654621 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2271) took 1036 ms. Trace:
I20260709 12:12:52.663044 12491 rpcz_store.cc:276] 0709 12:12:51.618078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.612664 (+994586us) service_pool.cc:224] Handling call
0709 12:12:52.654513 (+ 41849us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:52.666049 16035 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.142s	user 0.015s	sys 0.008s
I20260709 12:12:52.666649 16035 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:52.671424 12496 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:52.675324 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.672241 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:52.662568 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:52.674494 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2268) took 1057 ms. Trace:
I20260709 12:12:52.679389 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2250) took 1230 ms. Trace:
I20260709 12:12:52.679512 12497 rpcz_store.cc:276] 0709 12:12:51.617152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.611688 (+994536us) service_pool.cc:224] Handling call
0709 12:12:52.674421 (+ 62733us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.663594 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:52.680006 12494 rpcz_store.cc:276] 0709 12:12:51.448970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.458434 (+  9464us) service_pool.cc:224] Handling call
0709 12:12:51.486211 (+ 27777us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:51.518204 (+ 31993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:51.518236 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:51.530504 (+ 12268us) log.cc:844] Serialized 26 byte log entry
0709 12:12:52.242674 (+712170us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b18003aeac0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:12:52.363009 (+120335us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:52.363071 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.363163 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:52.671015 (+307852us) raft_consensus.cc:1685] finished
0709 12:12:52.671036 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.679308 (+  8272us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3560,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":11634,"spinlock_wait_cycles":13832576,"thread_start_us":120040,"threads_started":1,"wal-append.queue_time_us":1135093}
W20260709 12:12:52.699671 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 388574us
W20260709 12:12:52.696729 16075 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.005s	user 0.031s	sys 0.058s
I20260709 12:12:52.689280 16082 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:12:52.693778 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:52.727847 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.704210 13258 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:52.700717 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.729617 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2283) took 1039 ms. Trace:
I20260709 12:12:52.750362 12491 rpcz_store.cc:276] 0709 12:12:51.690537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.683129 (+992592us) service_pool.cc:224] Handling call
0709 12:12:52.729550 (+ 46421us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.688657 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2266) took 1080 ms. Trace:
I20260709 12:12:52.751890 12488 rpcz_store.cc:276] 0709 12:12:51.608100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.629294 (+1021194us) service_pool.cc:224] Handling call
0709 12:12:52.688562 (+ 59268us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:12:52.747965 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 165749us
W20260709 12:12:52.687445 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 227497us
I20260709 12:12:52.763914 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:52.725610 16075 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.031s	sys 0.058s
I20260709 12:12:52.768617 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:52.723258 16035 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:52.783336 12492 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:12:52.793429 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.695590 16089 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:52.802176 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:52.803414 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 150875us
I20260709 12:12:52.809197 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2277) took 1178 ms. Trace:
I20260709 12:12:52.726514 16082 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:52.820766 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2289) took 1128 ms. Trace:
I20260709 12:12:52.821594 12491 rpcz_store.cc:276] 0709 12:12:51.692432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.770420 (+1077988us) service_pool.cc:224] Handling call
0709 12:12:52.820665 (+ 50245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.726503 16084 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:52.810871 12492 rpcz_store.cc:276] 0709 12:12:51.619719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.321133 (+701414us) service_pool.cc:224] Handling call
0709 12:12:52.797915 (+476782us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.800218 15915 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:52.835891 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:52.839074 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65578.1us
I20260709 12:12:52.839653 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.814947 15622 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:52.845062 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2288) took 1153 ms. Trace:
I20260709 12:12:52.847740 12491 rpcz_store.cc:276] 0709 12:12:51.692039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.822630 (+1130591us) service_pool.cc:224] Handling call
0709 12:12:52.844956 (+ 22326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.846498 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1827) took 1168 ms. Trace:
I20260709 12:12:52.849293 12492 rpcz_store.cc:276] 0709 12:12:51.681049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.839269 (+1158220us) service_pool.cc:224] Handling call
0709 12:12:52.846410 (+  7141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.850704 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:52.852708 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2110) took 3529 ms. Trace:
W20260709 12:12:52.844028 16046 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.014s
I20260709 12:12:52.838280 15915 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:52.855600 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:52.856987 16083 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:52.856987 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.859500 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:52.863498 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:52.866034 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.867795 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.866999 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2261) took 1411 ms. Trace:
I20260709 12:12:52.869606 12491 rpcz_store.cc:276] 0709 12:12:51.456044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.853910 (+1397866us) service_pool.cc:224] Handling call
0709 12:12:52.866945 (+ 13035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.873122 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2262) took 1416 ms. Trace:
I20260709 12:12:52.853235 12572 rpcz_store.cc:276] 0709 12:12:49.323819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:49.930904 (+607085us) service_pool.cc:224] Handling call
0709 12:12:49.971764 (+ 40860us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:49.994058 (+ 22294us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:50.808284 (+814226us) log.cc:844] Serialized 14 byte log entry
0709 12:12:52.449862 (+1641578us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b1800466840. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:12:52.684007 (+234145us) log.cc:844] Serialized 14 byte log entry
0709 12:12:52.852286 (+168279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:52.852352 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:52.852409 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.852485 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:52.852626 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55839,"prepare.queue_time_us":1357992,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":1921,"raft.queue_time_us":123776,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":18788,"spinlock_wait_cycles":118575872,"thread_start_us":1484098,"threads_started":5,"wal-append.queue_time_us":489944,"wal-append.run_cpu_time_us":1624,"wal-append.run_wall_time_us":1043317}
I20260709 12:12:52.860534 15915 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:52.873771 12492 rpcz_store.cc:276] 0709 12:12:51.456410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.854275 (+1397865us) service_pool.cc:224] Handling call
0709 12:12:52.873044 (+ 18769us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.878001 16082 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:52.889122 16077 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.596s	user 0.007s	sys 0.000s
I20260709 12:12:52.887035 15962 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:12:52.892349 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:52.892701 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.892038 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:52.899735 12641 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.601s	user 0.016s	sys 0.008s
I20260709 12:12:52.902118 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:52.907774 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1828) took 1137 ms. Trace:
I20260709 12:12:52.908213 12491 rpcz_store.cc:276] 0709 12:12:51.770477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.870654 (+1100177us) service_pool.cc:224] Handling call
0709 12:12:52.907711 (+ 37057us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.912195 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:52.914225 16064 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:52.914883 16064 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:52.915345 16064 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:52.872112 15757 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:52.917011 16089 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:12:52.917232 15757 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:52.917548 16089 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:52.918256 16089 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:52.922683 16006 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:12:52.925606 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:52.927445 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 275088us
W20260709 12:12:52.937364 12644 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.550s	user 0.012s	sys 0.006s
I20260709 12:12:52.938805 12644 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:52.914254 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:52.957957 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:52.958951 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2258) took 1504 ms. Trace:
I20260709 12:12:52.859903 16046 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:52.962877 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:52.959548 12489 rpcz_store.cc:276] 0709 12:12:51.454888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.603533 (+1148645us) service_pool.cc:224] Handling call
0709 12:12:52.958843 (+355310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.961071 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:52.924816 15816 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:52.907313 16087 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:12:52.962945 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:52.971160 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56957.9us
I20260709 12:12:52.971648 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.900514 15962 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:52.965252 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:52.975986 12489 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:52.976584 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2276) took 1357 ms. Trace:
W20260709 12:12:52.926473 16103 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:12:52.975332 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2259) took 1520 ms. Trace:
I20260709 12:12:52.980628 12491 rpcz_store.cc:276] 0709 12:12:51.455233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.959125 (+1503892us) service_pool.cc:224] Handling call
0709 12:12:52.975254 (+ 16129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.976945 12489 rpcz_store.cc:276] 0709 12:12:51.619443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.964830 (+1345387us) service_pool.cc:224] Handling call
0709 12:12:52.976519 (+ 11689us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:52.957559 15757 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:52.979174 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:52.982442 13260 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:52.982863 12493 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:52.866477 16083 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:52.986685 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:52.990377 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:52.993326 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:52.976910 15962 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:52.998967 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:52.999172 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.001415 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.008915 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:52.990159 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79802us
I20260709 12:12:52.998474 16098 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 12:12:53.004079 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:53.014652 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1893) took 1508 ms. Trace:
I20260709 12:12:53.015522 16083 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:52.980003 15622 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.017763 16082 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:53.004739 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:53.009187 12644 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 12:12:53.019825 12462 connection.cc:256] RPC call timeout handler was delayed by -1.47959277200016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 12:12:53.003932 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:52.988379 16096 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:12:53.021143 12462 connection.cc:256] RPC call timeout handler was delayed by -1.3337927720001517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
I20260709 12:12:52.979619 16011 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:12:53.023391 12461 leader_election.cc:400] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:53.031832 12461 leader_election.cc:403] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:12:53.031845 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.035032 15915 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.037340 15915 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:52.994158 16078 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:53.037719 15944 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:53.038213 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.015125 12646 rpcz_store.cc:276] 0709 12:12:51.506279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.565217 (+1058938us) service_pool.cc:224] Handling call
0709 12:12:52.625447 (+ 60230us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:52.664919 (+ 39472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:52.664947 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:52.665501 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:12:52.814865 (+149364us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:52.814904 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:52.814995 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:53.014367 (+199372us) raft_consensus.cc:1685] finished
0709 12:12:53.014397 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:53.014580 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":192333,"spinlock_wait_cycles":4256512,"thread_start_us":133013,"threads_started":1,"wal-append.queue_time_us":294297}
W20260709 12:12:53.060213 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49972.4us
I20260709 12:12:53.039093 13257 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:53.039281 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.040129 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.042778 16046 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:53.043349 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.047516 12637 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:53.048005 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:53.048305 16064 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:52.995924 16062 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 12:12:53.030575 16096 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:53.052345 16082 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:53.053298 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2274) took 1434 ms. Trace:
I20260709 12:12:53.030092 16004 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:12:53.055780 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:53.055689 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129161us
W20260709 12:12:53.055930 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97802.5us
I20260709 12:12:53.066329 12641 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:53.062424 12569 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:53.063009 12640 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:53.066170 12492 rpcz_store.cc:276] 0709 12:12:51.618871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.985733 (+1366862us) service_pool.cc:224] Handling call
0709 12:12:53.053195 (+ 67462us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:53.069444 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.054039 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:53.070696 12641 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:12:53.076232 16089 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.078704 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.072649 12495 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 5.
I20260709 12:12:53.085549 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.063432 16035 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:53.086441 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:53.081447 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:53.059000 16078 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:53.090782 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2285) took 1399 ms. Trace:
I20260709 12:12:53.088591 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.091243 12492 rpcz_store.cc:276] 0709 12:12:51.691135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.069305 (+1378170us) service_pool.cc:224] Handling call
0709 12:12:53.090709 (+ 21404us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:53.092499 12639 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:53.093454 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:53.094017 12645 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:12:53.097060 16092 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
W20260709 12:12:53.098197 15520 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.623s	user 0.015s	sys 0.000s
W20260709 12:12:53.100636 12610 connection.cc:256] RPC call timeout handler was delayed by -1.6019715469999483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20260709 12:12:53.101783 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61721us
I20260709 12:12:53.102593 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:53.068652 16096 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:12:53.106164 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66972.5us
I20260709 12:12:53.106802 12491 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:12:53.078353 16046 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:53.110532 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:53.057788 16011 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.108143 12640 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:53.114023 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.095278 16078 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:53.065414 16084 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
W20260709 12:12:53.116874 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2280) took 1496 ms (client timeout 1777 ms). Trace:
W20260709 12:12:53.119202 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 418) took 1670 ms (client timeout 1881 ms). Trace:
W20260709 12:12:53.123061 12493 rpcz_store.cc:269] 0709 12:12:51.620541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.478490 (+857949us) service_pool.cc:224] Handling call
0709 12:12:53.116588 (+638098us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:53.116779 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1014426496}
W20260709 12:12:53.123109 12496 rpcz_store.cc:269] 0709 12:12:51.454123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.507483 (+ 53360us) service_pool.cc:224] Handling call
0709 12:12:53.118944 (+1611461us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:53.119130 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2647266688}
I20260709 12:12:53.130141 13255 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:12:53.130380 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:53.106544 16069 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.820s	user 0.011s	sys 0.009s
W20260709 12:12:53.100672 15871 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.666s	user 0.010s	sys 0.003s
I20260709 12:12:53.125079 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2269) took 1507 ms. Trace:
I20260709 12:12:53.149318 12491 rpcz_store.cc:276] 0709 12:12:51.617429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.107487 (+1490058us) service_pool.cc:224] Handling call
0709 12:12:53.124993 (+ 17506us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:53.100703 15876 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:53.138784 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:53.139948 16117 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:53.124630 16095 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:12:53.157097 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155925us
W20260709 12:12:53.142823 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.140498 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:53.158582 12493 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:53.148147 12645 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:12:53.138284 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:53.145210 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:53.151098 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.155663 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.155550 12569 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:12:53.131052 16036 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:53.081683 15944 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.155965 16069 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.163125 12646 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:53.120361 16004 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:53.168357 15816 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:53.171073 13259 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:12:53.199980 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54896.8us
I20260709 12:12:53.173825 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:53.202042 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.175889 16036 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:53.191015 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.161502 15871 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:12:53.197094 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:53.240780 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.212136 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:53.174166 16046 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:53.213333 16069 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:53.218302 15944 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:53.219419 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:53.226591 16109 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:12:53.230516 16004 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:53.232057 16064 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:53.254036 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133426us
I20260709 12:12:53.233703 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:12:53.208880 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:53.251576 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1736421880000307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20470000000000002s for a response
W20260709 12:12:53.268774 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1198714219999601s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 12:12:53.269215 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1074353799999699s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
W20260709 12:12:53.252887 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51769.5us
I20260709 12:12:53.254125 12496 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:53.267556 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 165065us
I20260709 12:12:53.209235 16117 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:53.272917 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.239954 16036 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:53.256446 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.266995 13255 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:53.280263 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:53.286047 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147533us
I20260709 12:12:53.283978 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:53.289565 15520 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:53.293059 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2255) took 1839 ms. Trace:
I20260709 12:12:53.255656 16084 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:53.294894 12488 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:53.268532 16046 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:53.293612 12497 rpcz_store.cc:276] 0709 12:12:51.453852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.291577 (+1837725us) service_pool.cc:224] Handling call
0709 12:12:53.292108 (+   531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:53.292360 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:53.292420 (+    60us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:53.292485 (+    65us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:53.292521 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:53.292586 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:53.292718 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:53.247494 16045 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:12:53.272558 15816 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:53.288897 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:12:53.300689 16058 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:12:53.301782 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.301883 15944 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:53.275661 16082 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:53.304384 16011 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:53.275033 16117 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:53.314172 16089 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:53.319212 16117 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:53.319628 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:53.320330 13255 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:53.306605 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1895) took 1239 ms. Trace:
W20260709 12:12:53.289505 16114 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.118s	user 0.002s	sys 0.000s
I20260709 12:12:53.308554 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2257) took 1844 ms. Trace:
I20260709 12:12:53.317713 15944 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:53.322966 12488 rpcz_store.cc:276] 0709 12:12:51.454530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.294171 (+1839641us) service_pool.cc:224] Handling call
0709 12:12:53.294675 (+   504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:53.298813 (+  4138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:53.299035 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:53.324272 15944 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.311431 16058 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:53.326699 16058 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:53.328822 16106 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.315s	user 0.002s	sys 0.000s
I20260709 12:12:53.329120 13255 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:53.298703 16084 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:53.316167 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.313856 15816 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.321056 16046 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.322039 12637 rpcz_store.cc:276] 0709 12:12:52.067745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.436631 (+368886us) service_pool.cc:224] Handling call
0709 12:12:53.306443 (+869812us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:53.306545 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456515968}
I20260709 12:12:53.309723 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.330735 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:53.315127 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:53.338925 12493 raft_consensus.cc:606] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:53.338938 16064 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:53.339370 12493 raft_consensus.cc:614] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:12:53.337029 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 137542us
I20260709 12:12:53.329485 15757 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.342834 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 14) took 1675 ms. Trace:
I20260709 12:12:53.343602 12493 rpcz_store.cc:276] 0709 12:12:51.667024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.282888 (+1615864us) service_pool.cc:224] Handling call
0709 12:12:53.340103 (+ 57215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:53.346427 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:53.340384 16003 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.840s	user 0.015s	sys 0.006s
I20260709 12:12:53.336922 16084 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:53.336560 16089 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:12:53.357728 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64609.9us
I20260709 12:12:53.387706 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:53.390475 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52594.6us
I20260709 12:12:53.389289 12572 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:12:53.396874 12563 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:53.339859 16064 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:53.349786 16058 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:53.402861 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.402209 15871 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:53.353228 15988 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:53.415472 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 116876us
I20260709 12:12:53.396291 16085 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:53.429963 12462 leader_election.cc:400] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7.
I20260709 12:12:53.434854 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:53.437700 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50161.6us
I20260709 12:12:53.441856 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:53.442806 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:53.444532 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.423110 15988 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:53.449783 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108825us
I20260709 12:12:53.455695 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:53.391597 16003 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.458689 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.454800 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:53.422402 16083 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:12:53.455240 15520 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 12:12:53.479984 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.481581 12494 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:53.483672 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:53.484604 12563 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:53.486821 13259 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:12:53.461719 16094 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:12:53.488389 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92473.5us
I20260709 12:12:53.405085 16004 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.467295 15832 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:12:53.497506 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89602.8us
W20260709 12:12:53.502333 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96116.2us
I20260709 12:12:53.503990 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:53.505046 12572 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:53.512275 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:53.513779 12497 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:53.513847 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.515252 15520 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:12:53.520395 15962 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.521056 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:53.521917 12572 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:53.522080 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.525084 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.506245 16089 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:12:53.542507 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76103us
I20260709 12:12:53.548694 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.549893 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.522789 16094 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:53.528690 16120 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:53.514146 16069 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:12:53.555596 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:53.528723 16045 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:53.544322 16011 raft_consensus.cc:2764] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:12:53.529461 16089 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:53.543818 15816 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:53.528318 15962 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:53.562395 12572 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 6: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:53.566056 12563 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:53.569612 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.497506 16064 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.551975 16097 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:53.559805 16120 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:53.560833 15757 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:53.562099 16045 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:53.593218 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.565230 15871 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:12:53.584612 15922 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.091s	user 0.019s	sys 0.018s
I20260709 12:12:53.577603 16089 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:53.604415 16045 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:53.616098 16127 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:12:53.616865 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104651us
I20260709 12:12:53.596401 16120 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:53.624220 16125 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:12:53.584295 16110 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:53.591672 16094 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:12:53.628795 16098 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:12:53.605041 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77467.6us
I20260709 12:12:53.666538 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2275) took 2012 ms. Trace:
I20260709 12:12:53.670254 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.589785 16085 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:53.673902 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89846.5us
W20260709 12:12:53.680738 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.695051 12489 rpcz_store.cc:276] 0709 12:12:51.619152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.330146 (+1710994us) service_pool.cc:224] Handling call
0709 12:12:53.330545 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:53.336410 (+  5865us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:53.352957 (+ 16547us) log.cc:844] Serialized 14 byte log entry
0709 12:12:53.461843 (+108886us) log.cc:844] Serialized 14 byte log entry
0709 12:12:53.630920 (+169077us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:53.630983 (+    63us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:53.631030 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:53.631101 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:53.631219 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21298,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":91,"raft.queue_time_us":93404,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":17564,"spinlock_wait_cycles":29577472,"thread_start_us":228510,"threads_started":2,"wal-append.queue_time_us":142812}
W20260709 12:12:53.683116 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.632920 16133 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:53.706878 16110 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:53.707191 15922 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:12:53.707453 16126 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:12:53.708233 16110 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.697377 16098 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:53.710594 16098 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.711920 16098 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:12:53.712857 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.714324 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2254) took 2260 ms. Trace:
I20260709 12:12:53.717439 12496 rpcz_store.cc:276] 0709 12:12:51.453492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.296216 (+1842724us) service_pool.cc:224] Handling call
0709 12:12:53.296748 (+   532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:53.298895 (+  2147us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:53.298948 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:53.312367 (+ 13419us) log.cc:844] Serialized 26 byte log entry
0709 12:12:53.441859 (+129492us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:53.441918 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:53.442016 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:53.713937 (+271921us) raft_consensus.cc:1685] finished
0709 12:12:53.713968 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:53.714228 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1562,"raft.queue_time_us":4871,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":124,"thread_start_us":97088,"threads_started":1,"wal-append.queue_time_us":162065}
W20260709 12:12:53.729336 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:53.732788 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.698601 16003 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:53.745009 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.746820 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.753796 12638 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:12:53.713795 16123 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:53.701172 16133 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:53.767135 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.770133 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.767135 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.768867 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:53.781865 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69111.4us
I20260709 12:12:53.773168 12496 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:53.784612 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.744174 16130 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:12:53.713552 16117 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.791181 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:53.791973 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.797159 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:53.786937 16083 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:53.799249 12496 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 6: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:53.799623 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:53.795482 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:53.717774 16085 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.784009 15973 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:53.805792 16011 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:12:53.808077 16133 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.851981 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:53.811651 16096 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.687s	user 0.013s	sys 0.001s
I20260709 12:12:53.853075 12496 raft_consensus.cc:2376] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7. Re-sending same reply.
W20260709 12:12:53.860502 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60261.9us
I20260709 12:12:53.813418 16117 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:53.813702 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1825) took 2203 ms. Trace:
I20260709 12:12:53.870784 12497 rpcz_store.cc:276] 0709 12:12:51.610266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.309420 (+1699154us) service_pool.cc:224] Handling call
0709 12:12:53.318658 (+  9238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:53.514345 (+195687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:53.813394 (+299049us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:53.813608 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444269440}
I20260709 12:12:53.811707 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:53.788380 15871 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:53.791891 15757 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:12:53.840337 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:53.811203 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2272) took 2192 ms (client timeout 2207 ms). Trace:
W20260709 12:12:53.879185 12494 rpcz_store.cc:269] 0709 12:12:51.618328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:52.727666 (+1109338us) service_pool.cc:224] Handling call
0709 12:12:53.810890 (+1083224us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:53.811124 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640796032}
I20260709 12:12:53.855557 15816 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:12:53.869462 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117153us
I20260709 12:12:53.885841 12572 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:53.833392 16083 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:53.895944 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151020us
I20260709 12:12:53.892978 12567 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:53.892650 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:12:53.881585 16078 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.725s	user 0.006s	sys 0.008s
I20260709 12:12:53.898746 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.901387 12637 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:53.904477 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:53.905543 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:12:53.906184 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.882874 15816 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:53.850895 15973 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.873948 16096 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.885175 16069 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:53.917104 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.918922 12572 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:53.919627 16161 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:12:53.926481 16136 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:12:53.926740 16085 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:53.928838 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:53.929142 16143 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:12:53.931080 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:53.896837 15988 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:53.988699 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:53.935446 16112 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.923s	user 0.006s	sys 0.000s
I20260709 12:12:53.933470 16097 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:12:53.946245 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2278) took 2326 ms (client timeout 1688 ms). Trace:
I20260709 12:12:53.969205 15816 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:53.941391 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:53.971102 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:54.001410 12490 rpcz_store.cc:269] 0709 12:12:51.619991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.700476 (+ 80485us) service_pool.cc:224] Handling call
0709 12:12:53.946044 (+2245568us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:53.946162 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4604274432}
I20260709 12:12:53.942186 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:53.999341 12571 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.005358 12568 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:12:54.001502 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70647.1us
W20260709 12:12:53.947576 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2267) took 2330 ms (client timeout 1603 ms). Trace:
W20260709 12:12:54.006913 12495 rpcz_store.cc:269] 0709 12:12:51.616778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:51.668118 (+ 51340us) service_pool.cc:224] Handling call
0709 12:12:53.947418 (+2279300us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:53.947514 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3115711872}
I20260709 12:12:54.008004 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.003394 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:53.945647 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2273) took 2327 ms. Trace:
I20260709 12:12:54.000336 16011 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:53.939414 16089 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.013190 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:54.019539 12493 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:54.019717 12496 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:53.911681 16078 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:53.981439 16133 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:54.019444 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:54.020328 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:54.018092 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:54.026338 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 186091us
I20260709 12:12:54.028818 12490 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:54.032961 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:54.034329 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:54.037286 16085 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:54.039484 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:54.041692 16069 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:12:54.043943 16096 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:54.046703 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154173us
I20260709 12:12:54.008976 12488 rpcz_store.cc:276] 0709 12:12:51.618601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.324055 (+1705454us) service_pool.cc:224] Handling call
0709 12:12:53.324516 (+   461us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:53.324945 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:53.324986 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:53.334639 (+  9653us) log.cc:844] Serialized 52 byte log entry
0709 12:12:53.742848 (+408209us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:12:53.742909 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:53.743023 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:53.945380 (+202357us) raft_consensus.cc:1685] finished
0709 12:12:53.945410 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:53.945556 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1013,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":99200,"thread_start_us":403887,"threads_started":1,"wal-append.queue_time_us":604522}
W20260709 12:12:54.057209 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:54.063854 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.064144 15988 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:54.066130 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:54.067854 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:54.068485 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1832) took 1442 ms. Trace:
I20260709 12:12:54.068782 12490 rpcz_store.cc:276] 0709 12:12:52.638421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.067291 (+1428870us) service_pool.cc:224] Handling call
0709 12:12:54.067639 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.068222 (+   583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.068427 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:54.077577 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:54.080556 12571 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:54.074967 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.088755 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:54.090499 12488 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:54.092717 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.093468 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.090001 16047 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:54.098469 12496 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:54.099543 16144 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:54.103977 16144 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:54.104451 16144 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:54.110003 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1831) took 1716 ms. Trace:
I20260709 12:12:54.110562 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.110460 12488 rpcz_store.cc:276] 0709 12:12:52.386495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.056871 (+1670376us) service_pool.cc:224] Handling call
0709 12:12:54.089704 (+ 32833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.101857 (+ 12153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.102339 (+   482us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:54.088371 16069 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:54.114741 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1838) took 1247 ms. Trace:
I20260709 12:12:54.109299 16150 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:54.115644 16161 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:54.115986 16167 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:54.147019 16045 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.148516 12638 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:12:54.150629 16078 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:54.152292 16036 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.865s	user 0.014s	sys 0.000s
I20260709 12:12:54.152709 12497 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:54.152760 16036 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.117911 16153 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:12:54.118824 16111 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:12:54.170454 15816 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.119573 15922 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.175364 15816 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:54.031141 16083 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.178014 15871 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:54.179405 16117 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:12:54.179551 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 172875us
I20260709 12:12:54.180220 16117 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.180622 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.180769 16117 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.182150 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:54.182528 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.182823 12569 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.108135 16047 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:54.122556 16089 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:54.118717 16084 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.719s	user 0.006s	sys 0.006s
I20260709 12:12:54.130776 12496 rpcz_store.cc:276] 0709 12:12:52.861487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.086350 (+1224863us) service_pool.cc:224] Handling call
0709 12:12:54.098300 (+ 11950us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.108486 (+ 10186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.108700 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:54.161969 12492 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:54.164408 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.169492 12488 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:12:54.172547 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79907us
W20260709 12:12:54.122552 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.011s	sys 0.010s
I20260709 12:12:54.185256 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:54.195674 12568 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:54.186151 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:54.198232 12643 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:54.198837 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.012s	sys 0.011s
I20260709 12:12:54.199762 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.200845 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:54.136210 16161 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:54.221781 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.222110 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 422) took 1199 ms. Trace:
I20260709 12:12:54.225914 12488 rpcz_store.cc:276] 0709 12:12:53.022357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.112264 (+1089907us) service_pool.cc:224] Handling call
0709 12:12:54.137807 (+ 25543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.194445 (+ 56638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.222040 (+ 27595us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:54.171418 16036 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:54.244558 12569 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:12:54.199275 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 744536us
I20260709 12:12:54.097509 16120 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:54.225261 16165 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:12:54.224135 16078 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:54.193374 16047 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:54.273146 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.192307 16064 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.192412 16096 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:54.284379 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:54.250083 15871 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:54.225371 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:54.290028 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:54.250892 15915 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:12:54.262786 15973 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:54.277087 15757 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:54.250576 16161 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.286645 16167 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.292569 16094 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.172163 16011 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.265429 15988 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 12:12:54.308916 16041 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:12:54.328946 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48095.4us
I20260709 12:12:54.371795 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.220695 16084 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:54.343832 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59538.2us
I20260709 12:12:54.353229 12568 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:54.326658 15757 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:54.370580 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188150us
I20260709 12:12:54.310278 16064 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.345876 15520 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:54.348450 16135 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:54.384018 12570 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:54.283377 16144 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.407485 12489 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:54.402678 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.409914 12564 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:12:54.410904 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.384722 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.375428 16110 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:54.379077 16041 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:54.380762 15757 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:54.382718 15520 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:54.404747 16157 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:54.370097 16094 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.405580 16135 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:54.405292 16004 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:54.470460 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:54.353099 16069 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:54.479214 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45640.4us
I20260709 12:12:54.422544 15816 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:54.420934 16110 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:54.417903 16153 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:54.425833 15520 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:54.494027 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.357100 16167 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:54.503847 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119199us
I20260709 12:12:54.507283 16167 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:54.510070 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:54.497710 16110 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:12:54.518453 16169 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:12:54.522377 16095 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:54.526111 16064 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:54.526264 16135 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.488420 16176 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:54.529942 16120 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:54.539353 16174 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 12:12:54.447476 16157 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.544399 16144 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:54.493312 16117 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:54.545670 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:12:54.546597 12495 raft_consensus.cc:606] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:54.548383 12495 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:54.485209 16041 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.565943 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:54.566375 16167 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:54.567459 12490 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:54.571326 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 1998) took 1225 ms. Trace:
I20260709 12:12:54.571808 12571 rpcz_store.cc:276] 0709 12:12:53.346002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.346137 (+   135us) service_pool.cc:224] Handling call
0709 12:12:54.571141 (+1225004us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:54.571259 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1499813632}
I20260709 12:12:54.411887 16158 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 12:12:54.500334 15816 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:54.576386 15816 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:54.578431 16164 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.229s	user 0.003s	sys 0.000s
I20260709 12:12:54.580461 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2282) took 2890 ms. Trace:
I20260709 12:12:54.580901 12493 rpcz_store.cc:276] 0709 12:12:51.690158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.357229 (+1667071us) service_pool.cc:224] Handling call
0709 12:12:53.398934 (+ 41705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.020165 (+621231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.580211 (+560046us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:54.580398 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1426787072}
W20260709 12:12:54.586741 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:54.588093 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2291) took 1488 ms. Trace:
I20260709 12:12:54.588492 12491 rpcz_store.cc:276] 0709 12:12:53.099828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.178002 (+ 78174us) service_pool.cc:224] Handling call
0709 12:12:54.587891 (+1409889us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:54.588023 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1275091328}
I20260709 12:12:54.590817 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:54.590446 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:54.598950 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:54.599848 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:54.606796 13261 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:54.605818 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:54.629698 12645 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:54.620581 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:54.652845 13253 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:54.609184 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:54.522449 16094 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:54.674947 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.677284 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:54.677295 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:54.680804 12641 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:12:54.678246 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:54.681571 12645 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:54.681948 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:54.601600 16153 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:54.687083 12639 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:54.688874 13259 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:12:54.609370 15915 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:54.611110 16170 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:12:54.684373 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:54.694970 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:54.695665 12643 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:54.619483 16157 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:54.609460 16117 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.083s
W20260709 12:12:54.718499 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 423) took 1695 ms (client timeout 2091 ms). Trace:
I20260709 12:12:54.717984 12645 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260709 12:12:54.718986 12492 rpcz_store.cc:269] 0709 12:12:53.022663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.173281 (+150618us) service_pool.cc:224] Handling call
0709 12:12:54.718285 (+1545004us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:54.718430 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053328768}
I20260709 12:12:54.603792 16004 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:54.734409 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 126226us
I20260709 12:12:54.693701 16153 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.744206 16084 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:54.745397 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.746371 16157 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:54.746888 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:12:54.747260 16058 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.333s	user 0.004s	sys 0.013s
W20260709 12:12:54.748047 15922 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.573s	user 0.013s	sys 0.000s
I20260709 12:12:54.747985 16157 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:54.749102 16157 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:54.750123 16157 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.2, Last appended by leader: 2, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.750711 16004 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:54.751699 16133 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:54.752126 16133 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:54.752645 15757 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.754621 12572 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:12:54.720436 16177 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:54.758008 16177 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:54.758175 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.758481 16177 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:54.758400 15832 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.759696 15832 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:54.767167 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2284) took 3076 ms. Trace:
I20260709 12:12:54.767664 12489 rpcz_store.cc:276] 0709 12:12:51.690832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.716070 (+2025238us) service_pool.cc:224] Handling call
0709 12:12:53.716663 (+   593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.408066 (+691403us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.766943 (+358877us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:54.767106 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588062976}
I20260709 12:12:54.739845 15915 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:54.693259 16064 raft_consensus.cc:2764] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:12:54.748432 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.747915 12566 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.769903 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:54.780458 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:54.772830 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.773093 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:54.785223 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:12:54.742023 12497 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.589s	user 0.003s	sys 0.013s
I20260709 12:12:54.790197 12497 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:12:54.791188 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2293) took 1690 ms. Trace:
I20260709 12:12:54.791683 12497 rpcz_store.cc:276] 0709 12:12:53.100707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.152090 (+1051383us) service_pool.cc:224] Handling call
0709 12:12:54.152557 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.790809 (+638252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.791105 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:54.798163 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:54.801560 12568 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:54.801939 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:12:54.812767 12637 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:54.820412 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:54.800118 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.822670 12646 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:12:54.822902 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:54.823341 12638 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:54.826915 12638 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:12:54.828863 12570 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:54.831171 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:54.881429 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:54.881863 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:54.882721 12565 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:54.883826 13259 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 12:12:54.882052 12564 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
W20260709 12:12:54.824232 16078 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.536s	user 0.005s	sys 0.009s
I20260709 12:12:54.887130 12564 pending_rounds.cc:85] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:12:54.887532 12564 pending_rounds.cc:107] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.8
W20260709 12:12:54.887975 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87912.4us
I20260709 12:12:54.888350 12572 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:12:54.881444 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:54.880134 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:54.895427 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:54.798419 16158 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:54.800850 15922 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:54.825913 16123 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:54.887323 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:54.899732 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 405767us
W20260709 12:12:54.900473 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69378.4us
I20260709 12:12:54.919691 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:54.769311 15915 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:54.934998 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 344702us
I20260709 12:12:54.798657 15520 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:54.822135 16094 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:54.898747 12564 raft_consensus.cc:2889] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:54.952199 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:54.958707 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2310) took 1069 ms. Trace:
I20260709 12:12:54.974584 12488 rpcz_store.cc:276] 0709 12:12:53.888966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.569255 (+680289us) service_pool.cc:224] Handling call
0709 12:12:54.569595 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:54.750673 (+181078us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:54.750716 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:54.750761 (+    45us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:54.750796 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.958341 (+207545us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540037e454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:54.958414 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:54.958629 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":415692928}
I20260709 12:12:54.921237 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:54.925791 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:12:54.885814 16084 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:12:54.926765 16108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.009s	sys 0.014s
W20260709 12:12:54.927882 16047 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.610s	user 0.013s	sys 0.001s
I20260709 12:12:54.928472 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:54.930097 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:54.884141 16085 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:12:54.907565 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159258us
W20260709 12:12:54.941321 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 350904us
I20260709 12:12:54.898904 16011 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.902712 16078 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:54.949427 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:55.029640 12642 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:54.799657 16130 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.104s
W20260709 12:12:54.949457 13218 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.091s (SENT)
I20260709 12:12:54.929503 15988 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:54.799849 16128 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.135s
I20260709 12:12:54.905805 15832 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:54.983393 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.006700 16004 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:12:54.998327 12563 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260709 12:12:55.002390 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:54.899262 16042 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
W20260709 12:12:55.007202 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86062.1us
I20260709 12:12:54.972600 15922 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:54.916509 16035 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
I20260709 12:12:54.908383 16006 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260709 12:12:54.911592 16095 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.085s
I20260709 12:12:54.946983 16158 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.061s
I20260709 12:12:55.032212 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:54.902165 16133 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:55.017894 16108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.009s	sys 0.015s
I20260709 12:12:55.006927 16120 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:12:54.997372 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77834.3us
W20260709 12:12:55.036616 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87197.2us
I20260709 12:12:54.985328 16097 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:55.005868 16084 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:55.122573 16084 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:55.104905 16035 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:55.126281 16035 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:55.127378 16035 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:55.051895 16166 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.908s	user 0.000s	sys 0.006s
I20260709 12:12:55.054318 15757 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:55.047784 16181 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:55.046057 12564 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:55.068042 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.070983 13253 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
W20260709 12:12:55.091154 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:55.037518 15988 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:55.101847 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:55.102265 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:55.021733 16047 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:55.077567 16083 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.897s	user 0.027s	sys 0.070s
I20260709 12:12:55.049618 16004 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:55.111123 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:12:55.154904 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86972.5us
I20260709 12:12:55.156468 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.111861 16177 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.113773 16186 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:12:55.114220 16138 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.136s
W20260709 12:12:55.115017 16176 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.585s	user 0.014s	sys 0.000s
I20260709 12:12:55.118830 16041 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:55.178306 16176 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:55.178418 16041 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:55.121954 16119 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:55.179131 16041 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:55.179502 16119 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:55.180074 16119 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:55.035243 16011 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.182857 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:12:55.184019 16011 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:55.099169 16085 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:55.135767 16181 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:55.172416 16058 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:55.172739 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033375
I20260709 12:12:55.071058 15922 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:55.193133 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91399.8us
I20260709 12:12:55.185465 16085 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:55.161429 16083 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.028s	sys 0.070s
I20260709 12:12:55.144074 16117 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:55.164633 16004 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:55.148881 15988 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:55.136749 16120 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:55.131141 16133 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:55.222999 16045 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:55.293073 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2305) took 1584 ms. Trace:
I20260709 12:12:55.235006 15922 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:55.238658 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.244570 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:55.153182 16069 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 12:12:55.135718 16134 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
W20260709 12:12:55.216665 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 327397us
I20260709 12:12:55.267408 12572 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:55.251255 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 299144us
W20260709 12:12:55.290194 16188 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:12:55.291139 16085 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:55.164693 16035 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:55.311249 15816 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.734s	user 0.017s	sys 0.008s
I20260709 12:12:55.311914 15520 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:55.293565 12495 rpcz_store.cc:276] 0709 12:12:53.708202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.549296 (+841094us) service_pool.cc:224] Handling call
0709 12:12:54.549556 (+   260us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:54.590866 (+ 41310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:54.590897 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:54.656047 (+ 65150us) log.cc:844] Serialized 26 byte log entry
0709 12:12:55.073092 (+417045us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:55.073144 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:55.073240 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:55.292807 (+219567us) raft_consensus.cc:1685] finished
0709 12:12:55.292828 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:55.293004 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":461,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6128,"thread_start_us":359540,"threads_started":1,"wal-append.queue_time_us":588613}
I20260709 12:12:55.262070 16130 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:12:55.249902 16181 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:55.365895 13259 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
W20260709 12:12:55.368333 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:55.378681 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 162217us
W20260709 12:12:55.363010 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46889.8us
W20260709 12:12:55.385221 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:55.379868 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 326400us
I20260709 12:12:55.397531 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:55.398463 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.381199 13259 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 5
I20260709 12:12:55.417968 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:55.426505 13257 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:12:55.356524 16117 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:55.407776 13259 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 7.6
W20260709 12:12:55.469934 16182 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 12:12:55.401530 16110 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.887s	user 0.013s	sys 0.001s
I20260709 12:12:55.379396 15816 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:55.501765 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 232828us
I20260709 12:12:55.493254 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.504455 12567 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:55.483085 16185 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:55.495327 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.422544 12564 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:55.500510 13259 raft_consensus.cc:2917] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Aborting config change with OpId 7.6: Aborted: Op aborted by new leader
I20260709 12:12:55.492847 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.506417 13259 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:12:55.509174 12565 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.626s	user 0.014s	sys 0.000s
I20260709 12:12:55.509668 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:12:55.511111 12566 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.731s	user 0.012s	sys 0.000s
I20260709 12:12:55.513644 12566 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:12:55.521888 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.539840 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.540737 12564 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:12:55.547039 12563 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:55.540664 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:12:55.560863 16153 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.738s	user 0.005s	sys 0.005s
I20260709 12:12:55.562485 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.563414 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:12:55.563483 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 146030us
I20260709 12:12:55.565054 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:55.562485 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.566222 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:55.566499 12567 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:55.569377 16176 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:55.570300 16176 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:55.570787 16176 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:55.575204 16157 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:55.575800 16157 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:55.578397 16138 raft_consensus.cc:993] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:12:55.579043 16138 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:55.579661 16138 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:55.580132 16138 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:55.584889 15816 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:55.585410 16202 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:12:55.592474 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 276278us
W20260709 12:12:55.586623 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64837.6us
I20260709 12:12:55.612175 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:55.544569 16128 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:55.393265 16163 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.162s
I20260709 12:12:55.510660 16185 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:12:55.672283 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107301us
I20260709 12:12:55.687711 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.681185 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.689116 12563 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:55.573052 16176 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:55.551316 16123 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:55.673417 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.674556 15520 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:55.604902 16134 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260709 12:12:55.543434 16110 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.592116 16153 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.610608 15832 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20260709 12:12:55.608891 16158 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260709 12:12:55.712314 16176 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:12:55.686311 16058 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:55.702016 16157 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:55.589495 16120 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:55.786231 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 435) took 1623 ms. Trace:
I20260709 12:12:55.681028 16135 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 12:12:55.786737 13255 rpcz_store.cc:276] 0709 12:12:54.163038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.171418 (+  8380us) service_pool.cc:224] Handling call
0709 12:12:55.785985 (+1614567us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:55.786165 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1321380096}
I20260709 12:12:55.787467 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.776799 16047 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:55.768319 16177 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:55.798205 16195 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.255s	user 0.000s	sys 0.000s
I20260709 12:12:55.799357 15757 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:55.799840 15757 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:55.800791 15757 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:55.802381 16192 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20260709 12:12:55.779345 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:55.780055 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:55.819072 16041 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.639s	user 0.012s	sys 0.000s
I20260709 12:12:55.819635 16041 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.820343 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.802551 16153 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:55.803308 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.821699 12567 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:55.822322 16184 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.328s	user 0.000s	sys 0.001s
I20260709 12:12:55.823797 16210 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:12:55.825060 16210 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:12:55.804764 16187 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:12:55.826560 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.807196 12646 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:55.827069 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 434) took 1235 ms. Trace:
W20260709 12:12:55.803364 16189 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:12:55.804816 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.829195 12568 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:55.829790 12564 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:55.806414 16163 raft_consensus.cc:1081] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:12:55.728711 15520 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:55.811336 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 214962us
I20260709 12:12:55.850771 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.681646 16128 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.811825 16058 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:55.813238 16161 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.440s	user 0.012s	sys 0.005s
I20260709 12:12:55.856040 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.856950 12569 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:55.814430 16110 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:55.610023 16196 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20260709 12:12:55.826773 16213 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:55.827747 16212 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:12:55.723766 16204 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:55.837797 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:55.839460 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1920) took 1210 ms. Trace:
I20260709 12:12:55.868052 12564 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:12:55.839967 16004 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.546s	user 0.006s	sys 0.003s
I20260709 12:12:55.777995 16185 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.872790 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:55.842167 16041 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:55.844444 16138 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.768566 15816 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:55.851667 12571 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:12:55.777688 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 379717us
I20260709 12:12:55.827605 12491 rpcz_store.cc:276] 0709 12:12:54.591512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.591648 (+   136us) service_pool.cc:224] Handling call
0709 12:12:54.596802 (+  5154us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:54.597155 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:54.597193 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:54.613759 (+ 16566us) log.cc:844] Serialized 26 byte log entry
0709 12:12:54.984821 (+371062us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:12:54.984858 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:54.984930 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:55.826826 (+841896us) raft_consensus.cc:1685] finished
0709 12:12:55.826850 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:55.827002 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":576,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":11074,"thread_start_us":370661,"threads_started":1,"wal-append.queue_time_us":937614}
I20260709 12:12:55.855046 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:12:55.882907 12563 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:12:55.884752 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:55.885910 12567 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:12:55.885587 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:55.887864 12644 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:55.870900 12494 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:12:55.859406 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 247364us
I20260709 12:12:55.873893 12569 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:12:55.873363 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 185771us
I20260709 12:12:55.709108 16123 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:55.892071 12572 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.518s	user 0.012s	sys 0.000s
I20260709 12:12:55.892526 12572 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:12:55.893303 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2030) took 1063 ms. Trace:
I20260709 12:12:55.894268 12572 rpcz_store.cc:276] 0709 12:12:54.829943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.898140 (+ 68197us) service_pool.cc:224] Handling call
0709 12:12:55.267146 (+369006us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b54003acba8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5d05f4 00007ffbfa7b9227 00007ffbfa7b8ab8 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:55.893220 (+626074us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12611968}
I20260709 12:12:55.884711 12563 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260709 12:12:55.851076 16084 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.710s	user 0.010s	sys 0.003s
I20260709 12:12:55.865556 16150 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:55.874536 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:55.901330 12489 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:55.867444 15757 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:55.867754 12640 rpcz_store.cc:276] 0709 12:12:54.629514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.629694 (+   180us) service_pool.cc:224] Handling call
0709 12:12:54.630017 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:54.686286 (+ 56269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:54.686318 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:54.694483 (+  8165us) log.cc:844] Serialized 26 byte log entry
0709 12:12:55.031636 (+337153us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:55.031679 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:55.031758 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:55.830929 (+799171us) raft_consensus.cc:1685] finished
0709 12:12:55.830956 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:55.839368 (+  8412us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3885,"raft.queue_time_us":3978,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":8988,"thread_start_us":336687,"threads_started":1,"wal-append.queue_time_us":698994}
W20260709 12:12:55.878405 16133 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.577s	user 0.012s	sys 0.000s
I20260709 12:12:55.899369 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:55.867261 16212 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:55.903681 12645 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:55.903929 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:55.870342 16204 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:55.904313 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:55.905396 12568 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:12:55.905519 12644 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:55.906611 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.908699 12641 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:55.909471 12494 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:55.872848 16213 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:55.909395 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:55.862998 16144 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:12:55.914455 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:55.908879 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:55.899076 16185 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:55.916648 16215 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 12:12:55.916833 16212 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:55.918048 16138 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:55.907867 16084 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.913065 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:55.923503 12638 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:55.865600 16161 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:55.893167 16110 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:12:55.893810 16117 raft_consensus.cc:2764] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:12:55.936647 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:55.936738 16212 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:12:55.937455 16209 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:12:55.938949 16181 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.508s	user 0.010s	sys 0.000s
I20260709 12:12:55.872882 16004 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:55.899606 16158 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:12:55.941002 12568 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:12:55.963862 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2018) took 2023 ms (client timeout 2486 ms). Trace:
W20260709 12:12:56.142145 12566 rpcz_store.cc:269] 0709 12:12:53.940782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:53.940975 (+   193us) service_pool.cc:224] Handling call
0709 12:12:55.963530 (+2022555us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:55.963769 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1709461888}
I20260709 12:12:56.143605 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:55.914664 16176 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:56.149561 12645 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:12:56.150034 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:56.162940 12641 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:56.150315 12495 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:55.961736 16186 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.193s
I20260709 12:12:56.151115 12565 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:55.904883 16042 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:55.915033 15915 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.238s
W20260709 12:12:56.017607 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 449885us
I20260709 12:12:56.176978 12568 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:56.152534 16004 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:56.157651 15520 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:56.158887 12646 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:55.966347 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58469.9us
I20260709 12:12:55.965814 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:55.905762 16150 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:55.934288 16119 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.754s	user 0.014s	sys 0.001s
I20260709 12:12:56.163532 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:55.958981 12639 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
W20260709 12:12:56.181213 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 268237us
I20260709 12:12:55.911793 16204 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:56.183835 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:56.138738 16128 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:56.152346 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107492us
I20260709 12:12:56.182713 12639 pending_rounds.cc:85] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 3
I20260709 12:12:56.188822 12639 pending_rounds.cc:107] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:12:56.189205 12639 raft_consensus.cc:2889] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:55.964119 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 587) took 1078 ms. Trace:
I20260709 12:12:56.171947 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:12:56.024611 15988 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.658s	user 0.008s	sys 0.004s
I20260709 12:12:56.174810 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:56.152179 16084 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:56.192739 13253 rpcz_store.cc:276] 0709 12:12:54.885898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:54.886099 (+   201us) service_pool.cc:224] Handling call
0709 12:12:54.886520 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:55.132502 (+245982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:55.963862 (+831360us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54005c4c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:55.964056 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423758592}
I20260709 12:12:56.178849 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:56.129340 16069 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:56.005405 12569 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:12:56.185719 13257 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:56.198575 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:56.180825 16231 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:56.216095 16231 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.217595 13261 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:12:56.223047 13258 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 9 LEARNER]: Advancing to term 11
I20260709 12:12:56.227160 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:56.181687 16150 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:56.231690 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:56.185442 15816 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:56.237185 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 271448us
I20260709 12:12:56.238729 13262 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 12:12:56.240981 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:56.245110 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.260327 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:12:56.260908 12569 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:56.265560 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.266672 12564 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:56.268874 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:12:56.269443 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86213.7us
I20260709 12:12:56.275162 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:56.279327 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:56.285557 12566 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:56.285851 16134 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:56.286619 16134 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:56.287165 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:56.287771 16134 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.279874 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.292313 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:56.280067 16217 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:12:56.280071 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.294021 16217 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:56.294157 16235 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:56.294973 16235 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.295166 16217 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.295600 16206 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:12:56.297686 16206 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:56.298225 16206 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.282817 16231 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:56.212770 15520 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:56.203125 16058 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:56.287981 12566 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:56.213691 16176 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:56.167330 16181 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.237569 16205 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:56.181865 16042 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.265479 16212 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:12:56.352965 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109303us
I20260709 12:12:56.352993 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:56.305150 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:56.328428 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:56.358418 12571 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:56.197880 16215 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:12:56.266026 16157 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:56.317008 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48276.5us
I20260709 12:12:56.379813 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:56.215001 16130 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
I20260709 12:12:56.371400 12566 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260709 12:12:56.263001 15973 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:56.185981 16119 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.303990 16196 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:56.263693 16222 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:56.387463 12643 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:56.224606 16128 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.135s
I20260709 12:12:56.303634 16110 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:56.214721 15988 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:12:56.359277 16205 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:56.266215 16004 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.118s
I20260709 12:12:56.219177 16069 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.403429 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:56.406338 12571 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:12:56.446352 12568 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:56.358286 16235 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:56.165483 16213 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.277563 15816 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.393728 15973 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:56.395336 16231 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:56.399466 16158 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:56.488885 16231 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:56.358903 16200 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260709 12:12:56.362008 15520 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.489681 16231 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:56.454502 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.455493 15973 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:56.395829 16157 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:56.442492 16205 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:56.466416 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.424855 16176 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:12:56.429289 16218 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:56.411157 16212 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:12:56.477598 15757 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.003s
I20260709 12:12:56.478489 15816 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:12:56.424094 16222 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:56.481834 16185 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.537s	user 0.006s	sys 0.006s
I20260709 12:12:56.485832 16208 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:56.489006 16158 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:56.398505 16196 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:56.304219 16220 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.172s
I20260709 12:12:56.493242 16128 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:12:56.508636 12569 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:12:56.516885 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:56.599853 12610 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.902s (SENT) [suppressed 14 similar messages]
W20260709 12:12:56.523241 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 218413us
I20260709 12:12:56.505966 13258 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 LEARNER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:12:56.553723 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:12:56.578131 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 180219us
I20260709 12:12:56.494577 16234 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:12:56.605794 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52418.2us
W20260709 12:12:56.696235 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84605.4us
I20260709 12:12:56.698951 12638 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:56.748399 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:56.755249 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155415us
I20260709 12:12:56.591787 15757 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:56.598524 16196 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:56.591144 16158 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:56.777380 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:56.778635 12563 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:12:56.786587 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:56.619028 16130 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:12:56.609743 16128 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:56.835285 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:56.947575 12637 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:56.618520 16004 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.219s
I20260709 12:12:56.826747 16041 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:12:57.131584 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:56.598129 16218 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:56.758549 16217 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.280s
I20260709 12:12:56.836841 16064 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.276s
I20260709 12:12:56.609647 16208 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.040904 15757 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:56.667464 16234 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.142045 16215 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.171s
W20260709 12:12:57.201589 12564 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.934s	user 0.027s	sys 0.017s
W20260709 12:12:57.197041 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 817235us
I20260709 12:12:57.334827 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:57.342278 12644 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260709 12:12:57.395632 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.396508 12571 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:12:57.396973 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.397256 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2039) took 1090 ms. Trace:
I20260709 12:12:57.397775 12571 rpcz_store.cc:276] 0709 12:12:56.307223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.396002 (+1088779us) service_pool.cc:224] Handling call
0709 12:12:57.396292 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:57.396990 (+   698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.397198 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:57.398361 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.395103 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:12:57.346905 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:57.361761 13260 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.753s	user 0.009s	sys 0.005s
I20260709 12:12:57.400769 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:12:57.399942 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2158) took 1135 ms. Trace:
I20260709 12:12:57.402213 12564 rpcz_store.cc:276] 0709 12:12:56.264586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.265159 (+   573us) service_pool.cc:224] Handling call
0709 12:12:57.399856 (+1134697us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:57.399945 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:57.396229 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.410302 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:57.417814 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:57.369894 16058 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.011s	sys 0.010s
W20260709 12:12:57.326460 16230 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.810s	user 0.000s	sys 0.000s
I20260709 12:12:57.425159 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:57.443496 12644 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:57.309157 15973 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.803s	user 0.005s	sys 0.001s
I20260709 12:12:57.431603 12641 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:12:57.206130 16224 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.843s	user 0.002s	sys 0.000s
I20260709 12:12:57.424214 12639 raft_consensus.cc:2368] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:12:57.188956 16181 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:12:57.392302 16176 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.006s	sys 0.009s
I20260709 12:12:57.161054 16177 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.241s
W20260709 12:12:57.392830 16232 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.883s	user 0.001s	sys 0.000s
I20260709 12:12:57.411232 15520 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:12:57.372833 16035 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.980s	user 0.010s	sys 0.006s
I20260709 12:12:57.196791 15988 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.164369 16041 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.278510 16119 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.445845 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.451704 16206 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.452881 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.453564 16208 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.207053 16130 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.248s
I20260709 12:12:57.040132 16212 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.453256 16234 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.460222 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:57.435125 16058 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.012s	sys 0.011s
I20260709 12:12:57.418743 16069 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.419165 16242 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:57.199100 16128 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.470928 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:57.600175 12642 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 12:12:57.515834 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:57.478899 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:57.605479 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.908s	user 0.000s	sys 0.004s
W20260709 12:12:57.606550 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.000s	sys 0.004s
W20260709 12:12:57.607136 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 909998us
I20260709 12:12:57.608460 16213 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:57.613071 16213 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:12:57.548026 16255 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.624061 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1946) took 1636 ms. Trace:
I20260709 12:12:57.624508 12640 rpcz_store.cc:276] 0709 12:12:55.988106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:55.988328 (+   222us) service_pool.cc:224] Handling call
0709 12:12:55.988886 (+   558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:56.008926 (+ 20040us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:56.008966 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:56.009648 (+   682us) log.cc:844] Serialized 26 byte log entry
0709 12:12:56.301130 (+291482us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:56.301177 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:56.301260 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:57.623838 (+1322578us) raft_consensus.cc:1685] finished
0709 12:12:57.623860 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.624005 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19843,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":6678,"thread_start_us":159368,"threads_started":1,"wal-append.queue_time_us":401880}
I20260709 12:12:57.627132 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:57.446575 16231 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.956s	user 0.014s	sys 0.000s
W20260709 12:12:57.629576 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 229670us
I20260709 12:12:57.642141 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.601481 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:57.644608 12640 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8.
I20260709 12:12:57.669756 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:57.669572 12645 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:12:57.671187 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:57.673239 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.676426 15520 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:57.632210 16255 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.678365 12645 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 8. Ops: [6.3-6.3]
I20260709 12:12:57.649858 16222 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.679926 12640 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:57.547446 16244 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.680804 16244 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.683452 16244 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.370362 15757 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.656213 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.591744 16256 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:57.685702 15757 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.686327 15757 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:12:57.686185 16256 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.646628 16177 consensus_queue.cc:1243] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:12:57.686443 12642 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:57.686964 15757 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:57.461858 16176 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.884s	user 0.006s	sys 0.010s
I20260709 12:12:57.677143 16128 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:57.688812 16256 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.680239 16255 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.665500 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2042) took 1303 ms. Trace:
I20260709 12:12:57.689389 15757 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.690111 12572 rpcz_store.cc:276] 0709 12:12:56.362286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.664472 (+1302186us) service_pool.cc:224] Handling call
0709 12:12:57.664884 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:57.665122 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:57.665151 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:57.665201 (+    50us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:57.665236 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.665295 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.665431 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:57.660844 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 445) took 1478 ms. Trace:
I20260709 12:12:57.677980 16257 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:12:57.679831 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.485898 15973 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.692487 12572 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:57.674252 12639 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:57.624387 16157 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.619931 12641 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:57.644062 16035 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:57.733482 16035 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.734247 16004 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:57.734758 16004 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:57.735590 16004 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.736080 16035 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.631214 16242 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.707458 16041 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.700788 16218 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.716465 16257 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:57.738456 16242 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.738818 16257 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:57.740203 16183 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 2.045s	user 0.021s	sys 0.026s
W20260709 12:12:57.692453 16150 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.416s	user 0.011s	sys 0.003s
I20260709 12:12:57.742236 16150 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.744545 16150 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.745445 12638 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:57.710970 15973 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.692742 13254 rpcz_store.cc:276] 0709 12:12:56.182233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.183649 (+  1416us) service_pool.cc:224] Handling call
0709 12:12:56.254046 (+ 70397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:56.589918 (+335872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:56.589968 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:57.282758 (+692790us) log.cc:844] Serialized 26 byte log entry
0709 12:12:57.566942 (+284184us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:57.623021 (+ 56079us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.643584 (+ 20563us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:57.660507 (+ 16923us) raft_consensus.cc:1685] finished
0709 12:12:57.660536 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.660739 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81,"raft.queue_time_us":120028,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":16128,"thread_start_us":294093,"threads_started":2,"wal-append.queue_time_us":314830}
I20260709 12:12:57.709319 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:57.703390 16197 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.866s	user 0.017s	sys 0.004s
I20260709 12:12:57.728335 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2038) took 1421 ms. Trace:
I20260709 12:12:57.731635 16259 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:57.732165 16158 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.889s	user 0.012s	sys 0.000s
I20260709 12:12:57.747900 16259 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.748237 16158 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.750928 16158 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.747288 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1862) took 1140 ms. Trace:
I20260709 12:12:57.753252 12637 rpcz_store.cc:276] 0709 12:12:56.606553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.618642 (+ 12089us) service_pool.cc:224] Handling call
0709 12:12:56.760193 (+141551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.948467 (+188274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.747010 (+798543us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:57.747220 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":430038272}
W20260709 12:12:57.713553 16205 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.197s	user 0.013s	sys 0.001s
I20260709 12:12:57.736928 16004 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:57.752233 16190 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.506s	user 0.007s	sys 0.011s
I20260709 12:12:57.747581 12569 rpcz_store.cc:276] 0709 12:12:56.306894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.839063 (+532169us) service_pool.cc:224] Handling call
0709 12:12:57.045261 (+206198us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:57.114223 (+ 68962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:57.114273 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:57.372998 (+258725us) log.cc:844] Serialized 26 byte log entry
0709 12:12:57.638641 (+265643us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:57.638702 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.638808 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:57.728054 (+ 89246us) raft_consensus.cc:1685] finished
0709 12:12:57.728084 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.728262 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16897,"raft.queue_time_us":493162,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":176,"thread_start_us":166175,"threads_started":1,"wal-append.queue_time_us":211996}
I20260709 12:12:57.733984 16264 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.696355 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2043) took 1326 ms. Trace:
I20260709 12:12:57.768077 12572 rpcz_store.cc:276] 0709 12:12:56.370517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.691721 (+1321204us) service_pool.cc:224] Handling call
0709 12:12:57.692220 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:57.696108 (+  3888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.696300 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:57.758718 16157 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:57.773166 16157 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.774231 16222 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:57.774478 16176 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.775043 16176 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.775530 16157 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.776041 16258 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:57.776160 16157 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:57.776784 16064 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:57.777411 16064 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.773175 16177 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.777720 16264 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.778353 16177 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.776939 16157 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.759364 16150 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.766856 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.774917 16222 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.751022 16259 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.759009 16153 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:57.771612 16196 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.928s	user 0.011s	sys 0.000s
W20260709 12:12:57.788269 16094 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.685s	user 0.026s	sys 0.029s
I20260709 12:12:57.695112 16231 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:57.790176 16094 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.691s	user 0.026s	sys 0.029s
I20260709 12:12:57.791389 16222 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.792084 16231 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.779524 16256 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:57.792904 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:57.792712 16256 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.793488 12569 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:57.794003 16265 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:57.794803 16265 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.797294 16153 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.790169 16196 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.810026 16229 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:57.810619 16229 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:57.765136 16242 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:12:57.774134 16191 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.528s	user 0.009s	sys 0.028s
W20260709 12:12:57.759550 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83336us
I20260709 12:12:57.841392 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:57.777503 16176 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.771957 16123 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.854624 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2040) took 1547 ms. Trace:
I20260709 12:12:57.816234 16242 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.847366 16094 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:57.765070 16205 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:57.854012 16204 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.644s	user 0.014s	sys 0.000s
I20260709 12:12:57.855222 12571 rpcz_store.cc:276] 0709 12:12:56.307545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.398730 (+1091185us) service_pool.cc:224] Handling call
0709 12:12:57.399038 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:57.399259 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:12:57.470269 (+ 71010us) log.cc:844] Serialized 14 byte log entry
0709 12:12:57.854199 (+383930us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:57.854268 (+    69us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:57.854318 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.854402 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.854553 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":280405,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":1292,"thread_start_us":432243,"threads_started":2}
I20260709 12:12:57.871018 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2037) took 1564 ms. Trace:
I20260709 12:12:57.788499 16150 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:57.936967 12566 rpcz_store.cc:276] 0709 12:12:56.306556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.387113 (+ 80557us) service_pool.cc:224] Handling call
0709 12:12:56.412196 (+ 25083us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:12:56.757723 (+345527us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:56.757758 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:12:56.837442 (+ 79684us) log.cc:844] Serialized 52 byte log entry
0709 12:12:57.471073 (+633631us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:57.852930 (+381857us) log.cc:844] Serialized 14 byte log entry
0709 12:12:57.870544 (+ 17614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.870668 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:57.870734 (+    66us) raft_consensus.cc:1685] finished
0709 12:12:57.870753 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.870946 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11111,"prepare.queue_time_us":208065,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":1061096,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":51997,"thread_start_us":567550,"threads_started":3,"wal-append.queue_time_us":278754}
I20260709 12:12:57.877272 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:12:57.844493 16120 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.005s	sys 0.039s
W20260709 12:12:57.958448 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81321.9us
I20260709 12:12:57.960019 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:57.963258 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:57.964552 12569 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:57.963946 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:57.977497 16196 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.901979 16094 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:57.957733 16120 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.111s	user 0.006s	sys 0.039s
I20260709 12:12:57.989506 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2047) took 1543 ms. Trace:
I20260709 12:12:57.965409 12638 raft_consensus.cc:2376] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10. Re-sending same reply.
I20260709 12:12:57.845552 16258 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:57.952369 16204 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.014436 12566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:12:57.994119 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:58.069227 12566 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:12:58.070351 12566 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:12:58.071821 12569 rpcz_store.cc:276] 0709 12:12:56.424623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.965430 (+1540807us) service_pool.cc:224] Handling call
0709 12:12:57.967746 (+  2316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:57.967921 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:57.967939 (+    18us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:12:57.967973 (+    34us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:12:57.967993 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:57.968032 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:57.968130 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:58.069944 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:58.075076 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:58.069752 12637 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:58.068972 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:58.064857 16261 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:12:57.962643 16177 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:57.955560 16064 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:58.081034 16150 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.966270 16158 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:12:58.009739 16259 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:12:57.994403 16268 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:12:58.112239 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.113242 12493 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:12:58.084614 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:58.113875 16158 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:58.114347 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.115067 16158 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.113265 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.114384 12638 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:58.118221 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:58.118898 16238 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:58.119443 16238 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.113888 12491 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:12:58.067873 16094 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:58.065338 16242 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:57.956453 16153 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:57.981467 15832 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:58.089159 16264 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.089262 16204 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:58.090204 16161 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.089835 16262 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.110s
I20260709 12:12:58.091121 16205 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:58.083770 16273 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:58.092008 16256 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.084017 15757 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:58.094776 16258 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.120579 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:58.148986 12638 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:58.084790 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:58.123580 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.129047 16036 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:58.129199 16238 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:58.131319 16042 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.131606 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.132120 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:58.154423 12489 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:58.132856 16203 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:58.132990 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.129514 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.138800 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.140386 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.143455 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:58.147599 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:58.132275 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.173024 12492 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 4.
I20260709 12:12:58.156312 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.151073 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:12:58.147452 15832 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.141863 16273 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:58.169561 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:58.164814 12566 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.184660 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:58.184535 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:58.151706 16036 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:58.187130 16006 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:58.172106 15757 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:58.167845 16203 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:58.178807 16273 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:58.190062 16084 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:12:58.230118 12566 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:58.239487 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:12:58.294342 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69828.3us
I20260709 12:12:58.253013 16045 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:12:58.252571 16004 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:58.408811 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:32896 (request call id 2045) took 1867 ms. Trace:
I20260709 12:12:58.412912 12566 rpcz_store.cc:276] 0709 12:12:56.396793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.965063 (+1568270us) service_pool.cc:224] Handling call
0709 12:12:58.264051 (+298988us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:58.264700 16271 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:58.257416 16084 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:58.257176 16161 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:12:58.494604 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 197140us
I20260709 12:12:58.391496 16006 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:12:58.508946 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.398958 16045 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:58.526289 16272 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.271s	user 0.000s	sys 0.000s
I20260709 12:12:58.530213 16084 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.531646 16045 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:58.417812 16036 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.512413 16208 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:58.550354 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2050) took 1998 ms. Trace:
I20260709 12:12:58.550817 12569 rpcz_store.cc:276] 0709 12:12:56.552406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.073351 (+1520945us) service_pool.cc:224] Handling call
0709 12:12:58.088084 (+ 14733us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:58.126295 (+ 38211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:58.126371 (+    76us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:58.134544 (+  8173us) log.cc:844] Serialized 26 byte log entry
0709 12:12:58.188892 (+ 54348us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:12:58.188936 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.189020 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:58.550090 (+361070us) raft_consensus.cc:1685] finished
0709 12:12:58.550116 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:58.550287 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20630,"raft.queue_time_us":30143,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":36576,"threads_started":1,"wal-append.queue_time_us":407860}
I20260709 12:12:58.554102 16084 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:58.558028 16036 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:58.562083 16045 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:58.553505 12569 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:58.573935 12569 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:58.598820 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.512122 16161 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:58.612409 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:58.613531 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:58.508855 16271 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:58.616039 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.625238 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:58.603790 12569 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.692323 12489 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:58.693727 12567 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:12:58.700802 12489 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:58.705921 12569 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:58.515836 16203 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.537212 15757 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:58.633108 16283 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:12:58.578646 16208 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.621904 15832 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:58.717579 12493 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
W20260709 12:12:58.697077 16281 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.427s	user 0.000s	sys 0.010s
I20260709 12:12:58.730135 12567 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 12. Ops: []
W20260709 12:12:58.714998 16282 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:12:58.737789 16203 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:58.748643 16094 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.608s	user 0.009s	sys 0.003s
W20260709 12:12:58.768074 12494 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.598s	user 0.013s	sys 0.000s
I20260709 12:12:58.785511 12494 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:58.718273 16006 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.2, Last appended by leader: 2, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.776533 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2044) took 2392 ms. Trace:
I20260709 12:12:58.788481 12571 rpcz_store.cc:276] 0709 12:12:56.384469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.961308 (+1576839us) service_pool.cc:224] Handling call
0709 12:12:57.961701 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:57.962074 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:57.962107 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:57.990251 (+ 28144us) log.cc:844] Serialized 26 byte log entry
0709 12:12:58.142128 (+151877us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:58.142174 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.142269 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:58.776201 (+633932us) raft_consensus.cc:1685] finished
0709 12:12:58.776243 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:58.776446 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":27019,"thread_start_us":54024,"threads_started":1,"wal-append.queue_time_us":117305}
I20260709 12:12:58.791785 12569 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:12:58.792435 12569 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 4
I20260709 12:12:58.792774 12569 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:12:58.793011 12569 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:58.784921 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2057) took 1026 ms. Trace:
W20260709 12:12:58.773617 12496 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.015s	sys 0.018s
W20260709 12:12:58.793802 12496 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.015s	sys 0.018s
I20260709 12:12:58.793597 12567 rpcz_store.cc:276] 0709 12:12:57.758999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.709764 (+950765us) service_pool.cc:224] Handling call
0709 12:12:58.729857 (+ 20093us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:58.779445 (+ 49588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.784856 (+  5411us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:58.807194 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2052) took 2229 ms. Trace:
I20260709 12:12:58.808167 12569 rpcz_store.cc:276] 0709 12:12:56.577563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.552977 (+1975414us) service_pool.cc:224] Handling call
0709 12:12:58.553377 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:58.806735 (+253358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.807080 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:58.783857 16094 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:58.832474 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:58.810423 16161 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.835842 12641 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:58.834079 16238 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:58.870880 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 398) took 2984 ms. Trace:
I20260709 12:12:58.871457 12646 rpcz_store.cc:276] 0709 12:12:55.886432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:55.887200 (+   768us) service_pool.cc:224] Handling call
0709 12:12:55.887528 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.178176 (+290648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.870494 (+2692318us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:58.870784 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623607168}
I20260709 12:12:58.873473 12646 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:12:58.894520 12646 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:58.909880 16208 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:58.890234 16167 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:58.917407 16167 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:58.920235 16167 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:58.855628 16238 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:58.944972 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 435) took 3570 ms. Trace:
I20260709 12:12:58.945689 12495 rpcz_store.cc:276] 0709 12:12:55.374276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:55.374417 (+   141us) service_pool.cc:224] Handling call
0709 12:12:55.374731 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.173716 (+798985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.944559 (+2770843us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:58.944861 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1783118080}
W20260709 12:12:58.950992 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:58.958302 16208 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:58.958935 16208 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:58.961205 16208 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.904489 16271 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:58.965929 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2051) took 2388 ms. Trace:
I20260709 12:12:58.966361 12566 rpcz_store.cc:276] 0709 12:12:56.577259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.503193 (+1925934us) service_pool.cc:224] Handling call
0709 12:12:58.503598 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:58.506351 (+  2753us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:58.506398 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:58.517485 (+ 11087us) log.cc:844] Serialized 26 byte log entry
0709 12:12:58.780344 (+262859us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:12:58.780401 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.780505 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:58.965703 (+185198us) raft_consensus.cc:1685] finished
0709 12:12:58.965724 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:58.965868 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2427,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":4263,"thread_start_us":177077,"threads_started":1,"wal-append.queue_time_us":326898}
W20260709 12:12:58.978914 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 167179us
W20260709 12:12:58.980824 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:58.983453 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:58.982781 16094 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:58.975893 16257 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:12:58.994922 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2140) took 3569 ms. Trace:
I20260709 12:12:58.995507 12565 rpcz_store.cc:276] 0709 12:12:55.425234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:55.514584 (+ 89350us) service_pool.cc:224] Handling call
0709 12:12:55.893810 (+379226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.171916 (+278106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:58.993602 (+2821686us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:58.994569 (+   967us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54003acd50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:58.994728 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1462423296}
W20260709 12:12:58.996879 16273 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.736s	user 0.010s	sys 0.001s
W20260709 12:12:59.008496 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:59.027117 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:59.018096 12462 connection.cc:256] RPC call timeout handler was delayed by -2.6711831530001291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2429s for a response
I20260709 12:12:58.998600 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.031399 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:12:59.032747 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.033159 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:12:59.037489 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.049271 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:12:59.015275 16280 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.304s	user 0.002s	sys 0.000s
W20260709 12:12:59.053360 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:59.058315 12462 connection.cc:256] RPC call timeout handler was delayed by -1.2923256280000714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
I20260709 12:12:58.994901 16257 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:12:59.056088 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.061259 16257 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:59.058804 12462 connection.cc:256] RPC call timeout handler was delayed by -1.2225181430001157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
I20260709 12:12:59.061558 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
W20260709 12:12:59.061831 12462 connection.cc:256] RPC call timeout handler was delayed by -1.1598943419999159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260709 12:12:59.062418 16257 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.063757 16257 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:12:59.063830 16210 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:12:59.054517 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:12:59.060135 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.066244 12637 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:12:59.000855 16238 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.068032 16238 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:59.068608 16238 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.063395 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:59.070879 16238 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.071058 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:59.072177 16036 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:12:59.017554 16204 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:59.072973 16036 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:59.074216 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.074862 12569 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:59.077674 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.072902 16204 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:12:59.079752 16204 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.080932 16204 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:59.081171 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.082358 12495 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 3. Current term is 4. Ops: []
I20260709 12:12:59.083231 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.084249 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.084856 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:59.083477 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.086086 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.078559 12496 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 8. Ops: []
I20260709 12:12:59.086946 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.088212 12489 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 12:12:59.088451 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.088838 16238 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:12:59.089006 12460 connection.cc:256] RPC call timeout handler was delayed by -1.1550343999999768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
I20260709 12:12:59.089488 16238 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.093938 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.095106 12488 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:59.090170 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:59.090308 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.096704 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:12:59.092496 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.098171 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:12:59.092110 12565 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:12:59.092113 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.100750 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.101562 12493 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:59.101668 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.102495 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.100749 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.103084 12493 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:12:59.097316 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.099138 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.105188 16210 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:12:59.106527 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.107553 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.108927 12494 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:59.107419 12495 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:59.111241 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:12:59.113708 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.114674 16238 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.116376 12496 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:12:59.116822 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.118121 12495 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:59.120292 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.121323 16303 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:59.122000 16303 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.123133 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.124182 12495 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:12:59.124317 16303 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.125190 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.019448 16117 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.016441 16157 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.127468 12536 leader_election.cc:400] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:12:59.143301 16238 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.133445 12646 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:59.136202 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.012519 16167 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.115s
I20260709 12:12:58.997608 16273 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.014925 16083 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260709 12:12:59.125916 16220 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.060976 16294 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.144176 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.129698 16210 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:12:59.082386 16300 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.068s
I20260709 12:12:59.129577 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.163334 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:12:59.167855 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154073us
I20260709 12:12:59.165220 12488 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:59.201973 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:59.202718 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.203101 12488 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:59.168298 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:59.204312 12537 connection.cc:256] RPC call timeout handler was delayed by -1.1906032799997774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2231s for a response
I20260709 12:12:59.177234 16257 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.177814 16085 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:59.165580 16303 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:59.204530 12571 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:59.176589 12567 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:12:59.215432 12489 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:59.208355 12640 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 12:12:59.209136 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45862.1us
I20260709 12:12:59.232187 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:12:59.204982 12537 connection.cc:256] RPC call timeout handler was delayed by -1.0515258979999089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20440000000000003s for a response
I20260709 12:12:59.234117 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:59.239156 12493 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:12:59.239532 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.240450 12571 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:12:59.243695 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43589us
I20260709 12:12:59.241811 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:59.205986 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.253471 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8.
I20260709 12:12:59.254850 12536 leader_election.cc:403] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:12:59.258109 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.258704 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.260636 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.161890 16117 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.262948 12571 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:12:59.272559 12496 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:12:59.180186 16238 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.277827 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.280242 12488 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:12:59.285938 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:12:59.199589 16220 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.252151 16303 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:12:59.294909 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 60930us
I20260709 12:12:59.279672 16309 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:12:59.206661 16294 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.240713 16257 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.300330 12567 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:12:59.294797 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:12:59.306802 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53568.6us
W20260709 12:12:59.309528 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.315475 12494 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:12:59.296762 16314 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:12:59.341511 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.342525 12488 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:12:59.322962 16309 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:59.305913 16310 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:12:59.327435 16307 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:59.352773 16317 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8.
I20260709 12:12:59.352974 16298 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:59.348555 16309 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:12:59.353781 16117 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.347033 16157 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:12:59.337967 16314 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:59.355742 16310 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:12:59.361883 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:59.366593 16299 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:12:59.367537 16311 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
W20260709 12:12:59.380399 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98128.7us
I20260709 12:12:59.306770 12571 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:12:59.526571 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:59.383997 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.577849 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:59.549171 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.581243 12569 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:59.484427 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.582233 12567 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:59.578924 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:12:59.589687 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.008s	sys 0.011s
I20260709 12:12:59.598882 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2036) took 3292 ms. Trace:
W20260709 12:12:59.602631 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.010s	sys 0.011s
W20260709 12:12:59.602837 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:59.603166 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 570098us
I20260709 12:12:59.602639 12568 rpcz_store.cc:276] 0709 12:12:56.306202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.352899 (+ 46697us) service_pool.cc:224] Handling call
0709 12:12:56.446139 (+ 93240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.446804 (+   665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.598633 (+3151829us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b5400102f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:12:59.598815 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139835776}
W20260709 12:12:59.604765 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.603749 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1947) took 3598 ms. Trace:
I20260709 12:12:59.381726 16238 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:59.509958 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 265340us
I20260709 12:12:59.608781 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:59.610278 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:59.605504 12643 rpcz_store.cc:276] 0709 12:12:56.005655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.005945 (+   290us) service_pool.cc:224] Handling call
0709 12:12:56.024730 (+ 18785us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.400554 (+375824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.603466 (+3202912us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:59.603679 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":833641216}
I20260709 12:12:59.611996 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:12:59.612778 12488 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:59.372731 16307 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:59.422809 16294 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.424049 16273 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:59.497061 16300 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:12:59.577253 16157 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:59.614809 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:59.622156 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:59.622052 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.629457 12568 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:59.629917 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:12:59.624044 13220 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.008s	sys 0.024s
I20260709 12:12:59.624202 12491 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:12:59.630728 13220 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.008s	sys 0.024s
I20260709 12:12:59.616537 16167 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:12:59.589514 16314 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:12:59.600819 16257 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:59.624926 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.618316 16301 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:12:59.648979 12567 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:12:59.638016 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.628172 16083 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.645615 12643 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:59.646984 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.650300 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.630044 16157 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:12:59.633916 16213 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:12:59.654913 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:12:59.652904 12644 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:12:59.656453 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.638551 16167 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:12:59.663596 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 335170us
I20260709 12:12:59.381865 16298 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.669049 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.669148 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:12:59.670181 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1955) took 1862 ms. Trace:
I20260709 12:12:59.671442 12643 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:12:59.671707 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.658205 16307 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.672580 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.673660 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:59.674005 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.674769 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:12:59.675195 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.675808 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:12:59.675988 16325 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:12:59.676692 16325 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:12:59.676753 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:12:59.677152 16325 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:59.677510 16325 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:59.677891 16294 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:59.678251 12564 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:12:59.678344 16294 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:59.678818 12564 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:12:59.661715 16213 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.679662 12564 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.680846 12564 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:59.627714 16316 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:12:59.686429 16325 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.687033 16321 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:59.673326 12643 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:12:59.688481 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.677435 12645 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:12:59.691479 12645 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:12:59.693962 12536 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.710s (SENT)
I20260709 12:12:59.694590 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:59.671337 12638 rpcz_store.cc:276] 0709 12:12:57.807820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.186571 (+378751us) service_pool.cc:224] Handling call
0709 12:12:58.255761 (+ 69190us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:58.482553 (+226792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:58.482598 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:12:58.498041 (+ 15443us) log.cc:844] Serialized 26 byte log entry
0709 12:12:58.809906 (+311865us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:58.809951 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.660309 (+850358us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:59.660347 (+    38us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:12:59.660407 (+    60us) raft_consensus.cc:1685] finished
0709 12:12:59.660415 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:12:59.670097 (+  9682us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6899,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":13423,"spinlock_wait_cycles":520029568,"thread_start_us":215587,"threads_started":1,"wal-append.queue_time_us":406841}
I20260709 12:12:59.701237 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.702044 12567 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:12:59.702416 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.648531 16301 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:59.704419 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.705709 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.714993 12644 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:12:59.716243 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.692314 12645 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.718713 12645 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:12:59.720588 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.744s (SENT)
I20260709 12:12:59.722007 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:12:59.722582 12571 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:59.669755 16167 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:12:59.659997 16083 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.700610 16217 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:59.732275 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.721709 16213 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:59.732273 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:12:59.735270 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.015s (SENT)
I20260709 12:12:59.727768 12643 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:12:59.678962 16294 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:12:59.730347 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 361624us
I20260709 12:12:59.731638 12638 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:12:59.738339 12644 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:12:59.705176 16238 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:12:59.716706 16301 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:59.717432 16303 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.738534 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1970) took 1621 ms. Trace:
I20260709 12:12:59.733521 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.736001 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:59.720254 16316 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:12:59.732336 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.729297 16327 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:12:59.746297 12564 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:12:59.756323 12564 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 5
I20260709 12:12:59.756767 12564 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 7.6
W20260709 12:12:59.750833 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136113us
I20260709 12:12:59.750747 12638 rpcz_store.cc:276] 0709 12:12:58.116670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.712437 (+1595767us) service_pool.cc:224] Handling call
0709 12:12:59.731387 (+ 18950us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:59.738257 (+  6870us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.738474 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:12:59.727550 16085 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:12:59.778355 16085 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.780820 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.782435 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.786815 12570 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:12:59.758038 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:12:59.789085 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:59.745396 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:12:59.728461 16220 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.752704 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.752535 12567 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:12:59.752933 12568 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:12:59.755530 12645 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:12:59.758015 16327 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:12:59.759488 16319 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:12:59.760093 16298 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.774121 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.791981 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:12:59.816589 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.817485 13261 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:12:59.804257 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.801312 12564 raft_consensus.cc:2917] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Aborting config change with OpId 7.6: Aborted: Op aborted by new leader
I20260709 12:12:59.704903 16321 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.822556 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.810504 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 545) took 4241 ms. Trace:
I20260709 12:12:59.826160 12563 rpcz_store.cc:276] 0709 12:12:55.569226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:55.897115 (+327889us) service_pool.cc:224] Handling call
0709 12:12:55.918393 (+ 21278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:56.786261 (+867868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.810077 (+3023816us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:12:59.810407 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1977111424}
I20260709 12:12:59.816902 12645 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:12:59.818917 12564 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:12:59.805353 12644 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:12:59.825606 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.830355 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.831347 12568 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:12:59.831552 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2041) took 3474 ms. Trace:
I20260709 12:12:59.832340 12564 rpcz_store.cc:276] 0709 12:12:56.357605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:57.418822 (+1061217us) service_pool.cc:224] Handling call
0709 12:12:57.419191 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:59.678159 (+2258968us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54000035a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:12:59.831155 (+152996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.831468 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616383744}
I20260709 12:12:59.836524 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:12:59.738684 16217 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.781548 16085 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.840761 12564 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9. Re-sending same reply.
W20260709 12:12:59.833169 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97921.2us
I20260709 12:12:59.803278 16247 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:12:59.850710 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.808779 16322 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:12:59.781337 16294 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.846364 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.847725 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:59.858803 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.861737 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:12:59.851605 12563 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:12:59.813025 16238 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:12:59.863646 13259 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 LEARNER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: []
W20260709 12:12:59.846439 16058 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:12:59.871395 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.873761 16321 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:59.880347 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:12:59.881667 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.883445 12644 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:12:59.885166 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.893949 12568 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:12:59.897388 12568 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 12:12:59.839845 16213 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:12:59.869361 16298 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:12:59.917390 16157 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.900243 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:12:59.923107 12570 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:12:59.856065 16247 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:12:59.848738 16328 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:12:59.869586 16318 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:12:59.904884 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.843629 16215 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:12:59.906147 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.861297 16325 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.911741 16167 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.917691 16217 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.921423 16303 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:12:59.899241 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:12:59.942364 12569 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:12:59.929934 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.945604 12568 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: []
I20260709 12:12:59.951654 13257 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:12:59.934975 16247 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:12:59.956825 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.958364 13259 raft_consensus.cc:2368] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11.
I20260709 12:12:59.951022 16303 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:12:59.941157 16328 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:12:59.963183 16327 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.963560 16294 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.964119 16157 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:12:59.978477 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:12:59.980273 16117 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:12:59.980782 16117 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:12:59.981987 16243 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:12:59.967236 16167 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.954851 16083 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:12:59.968592 16301 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.969368 16303 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:12:59.970378 16265 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:12:59.974166 16297 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.909s	user 0.008s	sys 0.006s
W20260709 12:12:59.975386 16330 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:12:59.987937 16307 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:12:59.985476 16328 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.968044 16314 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:12:59.983372 16117 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.3, Last appended by leader: 3, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:12:59.995069 16117 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:12:59.977020 16309 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260709 12:12:59.996598 16314 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:12:59.988584 16229 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.006472 16301 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.000044 16309 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:00.052342 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 231247us
I20260709 12:13:00.056372 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:00.060775 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:12:59.915397 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.065598 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:00.067487 12489 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:00.067026 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.068843 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.069761 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:00.070448 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2329) took 1946 ms (client timeout 1902 ms). Trace:
I20260709 12:13:00.070940 12489 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
W20260709 12:13:00.071218 12491 rpcz_store.cc:269] 0709 12:12:58.123498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.138448 (+ 14950us) service_pool.cc:224] Handling call
0709 12:13:00.069889 (+1931441us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:00.070176 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2756904832}
I20260709 12:13:00.072882 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:12:59.912209 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.074566 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.074723 12495 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:00.076273 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.076975 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.077277 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.078261 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.078366 12489 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:00.079319 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.080387 12563 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:00.083587 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.084278 12570 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:12:59.952790 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.100098 13261 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:00.102820 12564 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:00.104403 13258 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:00.105126 16229 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:00.108350 16265 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.122591 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.124655 16217 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:00.125553 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.126339 16286 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:00.127810 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.134426 12495 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:00.140527 16294 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.142144 16294 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.140477 12567 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:00.148538 12570 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:13:00.125284 16217 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:00.153268 12497 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:13:00.158825 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.158393 16265 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.160478 12489 raft_consensus.cc:2376] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8. Re-sending same reply.
I20260709 12:13:00.151168 16309 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.173586 16247 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.155181 16217 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:00.186434 16247 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.187296 16243 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.189981 16309 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.190999 16243 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:13:00.194521 13260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 444) took 4020 ms (client timeout 1807 ms). Trace:
I20260709 12:13:00.145529 16202 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:00.219604 12496 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:13:00.150813 16294 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.249555 16245 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:00.276973 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:00.278657 12495 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:00.288897 16334 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:00.299027 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136837us
W20260709 12:13:00.303923 13260 rpcz_store.cc:269] 0709 12:12:56.174254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:56.174538 (+   284us) service_pool.cc:224] Handling call
0709 12:13:00.193869 (+4019331us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:00.194173 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":939048192}
W20260709 12:13:00.333329 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.305223 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:00.335134 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:00.335353 16316 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:13:00.336859 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:00.306682 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.337496 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.347585 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:00.349013 16309 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.351851 16243 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.352517 16309 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.354055 16328 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:00.354645 16328 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:00.355074 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2164) took 1052 ms. Trace:
I20260709 12:13:00.355262 16328 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:00.355643 12564 rpcz_store.cc:276] 0709 12:12:59.303102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.846018 (+542916us) service_pool.cc:224] Handling call
0709 12:13:00.354754 (+508736us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:00.354962 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562847488}
W20260709 12:13:00.360702 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110560us
I20260709 12:13:00.338171 12493 raft_consensus.cc:2376] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6. Re-sending same reply.
I20260709 12:13:00.357407 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:00.304684 16036 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.222s	user 0.013s	sys 0.005s
I20260709 12:13:00.352660 16247 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:13:00.364705 16202 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:00.373857 16085 raft_consensus.cc:1064] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:13:00.374096 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:00.375454 16202 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.377942 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:00.378254 16085 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:00.382658 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.386160 16265 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.387656 16265 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:00.388285 16265 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:00.388832 16265 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:00.391165 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2166) took 1087 ms. Trace:
I20260709 12:13:00.391402 12571 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:13:00.391811 12569 rpcz_store.cc:276] 0709 12:12:59.303626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.629565 (+325939us) service_pool.cc:224] Handling call
0709 12:13:00.390644 (+761079us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:00.390945 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":716195712}
I20260709 12:13:00.337172 16245 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:00.395124 16245 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:00.396415 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1869) took 1263 ms (client timeout 1536 ms). Trace:
I20260709 12:13:00.407877 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.400315 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:00.414979 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.397034 16245 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:00.454970 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:00.455931 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.461244 16217 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.426160 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.469996 16217 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.470973 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.472442 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.476264 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:00.366918 16344 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:13:00.429440 16286 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:00.480604 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.481051 16286 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:00.481580 16286 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:00.481829 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:00.482620 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.482978 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:00.483992 13258 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:00.484459 16243 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:00.484926 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:00.431917 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1969) took 2315 ms. Trace:
I20260709 12:13:00.485901 13260 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 LEARNER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11. Re-sending same reply.
I20260709 12:13:00.439723 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1972) took 1266 ms. Trace:
I20260709 12:13:00.486347 13253 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:13:00.486599 12638 rpcz_store.cc:276] 0709 12:12:59.173556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.757515 (+583959us) service_pool.cc:224] Handling call
0709 12:12:59.769964 (+ 12449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:59.770213 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:12:59.840378 (+ 70165us) log.cc:844] Serialized 14 byte log entry
0709 12:13:00.438953 (+598575us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:00.439093 (+   140us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:00.439172 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.439301 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:00.439512 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":508765,"prepare.queue_time_us":135549,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":516730,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":183,"thread_start_us":158716,"threads_started":2,"wal-append.queue_time_us":139023}
I20260709 12:13:00.439893 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1975) took 1261 ms. Trace:
I20260709 12:13:00.486620 16309 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:00.480099 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:00.489681 12645 rpcz_store.cc:276] 0709 12:12:59.177917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.828294 (+650377us) service_pool.cc:224] Handling call
0709 12:12:59.849336 (+ 21042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:59.849548 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:12:59.936693 (+ 87145us) log.cc:844] Serialized 14 byte log entry
0709 12:13:00.439112 (+502419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:00.439255 (+   143us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:00.439310 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.439437 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:00.439642 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":352985,"prepare.queue_time_us":56411,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":361122,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":87936,"thread_start_us":235411,"threads_started":2,"wal-append.queue_time_us":74632}
W20260709 12:13:00.433822 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.492063 16314 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:00.492543 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.493968 16110 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:00.494484 16181 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.494784 16110 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.495362 16181 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.441865 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 413) took 1411 ms. Trace:
I20260709 12:13:00.490278 16309 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:00.442720 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:00.444262 15757 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.497949 16309 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:00.498834 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56247.5us
I20260709 12:13:00.498584 15757 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.465466 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.444729 15816 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:00.445714 16045 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:00.500906 15816 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.501498 16045 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.502986 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:00.447654 16247 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:00.503832 16335 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:00.504649 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.505107 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1953) took 2697 ms. Trace:
I20260709 12:13:00.500924 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.506067 16138 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:13:00.507330 16138 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:13:00.509013 16312 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:00.509465 16312 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:00.510064 16181 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:00.510615 16312 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.512213 16312 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:00.397042 12494 rpcz_store.cc:269] 0709 12:12:59.133319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.133459 (+   140us) service_pool.cc:224] Handling call
0709 12:13:00.393650 (+1260191us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:00.396207 (+  2557us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285968128}
I20260709 12:13:00.514189 15816 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:00.514410 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.515442 12494 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 4.
I20260709 12:13:00.516702 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.519318 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1967) took 2403 ms. Trace:
I20260709 12:13:00.519831 12640 rpcz_store.cc:276] 0709 12:12:58.115853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.062593 (+946740us) service_pool.cc:224] Handling call
0709 12:12:59.063010 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:12:59.230798 (+167788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.518578 (+1287780us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:00.519094 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333394176}
I20260709 12:13:00.362710 16334 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.505777 16110 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:00.485116 16243 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:00.505649 12641 rpcz_store.cc:276] 0709 12:12:57.807200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.185978 (+378778us) service_pool.cc:224] Handling call
0709 12:12:58.238950 (+ 52972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:58.836673 (+597723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.504557 (+1667884us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:00.504885 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1371943680}
I20260709 12:13:00.485831 12637 rpcz_store.cc:276] 0709 12:12:58.116418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.067173 (+950755us) service_pool.cc:224] Handling call
0709 12:12:59.068119 (+   946us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:59.068361 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:12:59.121504 (+ 53143us) log.cc:844] Serialized 14 byte log entry
0709 12:12:59.190071 (+ 68567us) log.cc:844] Serialized 14 byte log entry
0709 12:13:00.412406 (+1222335us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b18003bde80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:00.430841 (+ 18435us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:00.430976 (+   135us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:00.431102 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.431273 (+   171us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:00.431554 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1229242,"prepare.queue_time_us":52786,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":105,"raft.queue_time_us":1278347,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":18339,"spinlock_wait_cycles":79325312,"thread_start_us":94892,"threads_started":2,"wal-append.queue_time_us":86162}
I20260709 12:13:00.437898 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.524667 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:00.524937 15757 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:00.457327 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1965) took 2341 ms. Trace:
I20260709 12:13:00.525842 16243 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.526170 12646 rpcz_store.cc:276] 0709 12:12:58.115291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:58.895597 (+780306us) service_pool.cc:224] Handling call
0709 12:12:58.895903 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:59.165013 (+269110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.456576 (+1291563us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54000a6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:00.457031 (+   455us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":536161024}
I20260709 12:13:00.492666 16314 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:00.527549 16314 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:00.528020 16307 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.014s
I20260709 12:13:00.480181 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:00.524699 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2163) took 1222 ms. Trace:
I20260709 12:13:00.525904 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:00.482911 16305 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.374s	user 0.005s	sys 0.006s
W20260709 12:13:00.447254 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.533473 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2059) took 1461 ms. Trace:
W20260709 12:13:00.533598 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.533927 12566 rpcz_store.cc:276] 0709 12:12:59.072039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.075118 (+  3079us) service_pool.cc:224] Handling call
0709 12:12:59.075493 (+   375us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:12:59.075954 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:12:59.075984 (+    30us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:12:59.076777 (+   793us) log.cc:844] Serialized 78 byte log entry
0709 12:12:59.108152 (+ 31375us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:12:59.120083 (+ 11931us) log.cc:844] Serialized 14 byte log entry
0709 12:12:59.243406 (+123323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:12:59.243512 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:00.481864 (+1238352us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:00.532962 (+ 51098us) raft_consensus.cc:1685] finished
0709 12:13:00.533057 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:00.533292 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2619,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":87270,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":15169,"spinlock_wait_cycles":38678272,"thread_start_us":165123,"threads_started":3,"wal-append.queue_time_us":31015}
I20260709 12:13:00.536134 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:00.530277 12563 rpcz_store.cc:276] 0709 12:12:59.302735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.886423 (+583688us) service_pool.cc:224] Handling call
0709 12:12:59.886687 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.108605 (+221918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.524269 (+415664us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:00.524604 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":443449216}
I20260709 12:13:00.537433 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:00.538273 12488 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.925s	user 0.016s	sys 0.009s
I20260709 12:13:00.521715 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:00.507493 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.542099 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.554589 12491 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:00.476377 16229 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 12:13:00.545936 16120 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:00.537179 13254 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:00.536181 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:00.543089 16117 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:00.567009 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:00.567973 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:00.568259 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:00.366151 16036 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s	user 0.014s	sys 0.005s
I20260709 12:13:00.572577 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:00.575591 16309 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.575924 16265 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.582345 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:00.592262 12489 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:13:00.583199 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:00.599936 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 454) took 1245 ms. Trace:
W20260709 12:13:00.585572 12610 connection.cc:256] RPC call timeout handler was delayed by -1.0910478460000377s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
I20260709 12:13:00.603825 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.604864 12495 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:00.600364 12496 rpcz_store.cc:276] 0709 12:12:59.354020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.473489 (+1119469us) service_pool.cc:224] Handling call
0709 12:13:00.599855 (+126366us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.600481 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 442) took 1448 ms. Trace:
I20260709 12:13:00.607579 12489 rpcz_store.cc:276] 0709 12:12:59.152462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.338894 (+1186432us) service_pool.cc:224] Handling call
0709 12:13:00.600417 (+261523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.614939 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.615859 12491 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:00.577018 16314 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.497871 12643 rpcz_store.cc:276] 0709 12:12:59.030332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.737327 (+706995us) service_pool.cc:224] Handling call
0709 12:12:59.737698 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:12:59.737920 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:12:59.832249 (+ 94329us) log.cc:844] Serialized 14 byte log entry
0709 12:13:00.440930 (+608681us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:00.441095 (+   165us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:00.441234 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.441366 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:00.441591 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":518425,"prepare.queue_time_us":167591,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":525182,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":6400,"thread_start_us":163673,"threads_started":2,"wal-append.queue_time_us":124392}
I20260709 12:13:00.619649 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.569754 16045 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:00.624240 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 88169.6us
W20260709 12:13:00.625742 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:00.626617 12610 connection.cc:256] RPC call timeout handler was delayed by -1.0673095750000812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
W20260709 12:13:00.627012 12610 connection.cc:256] RPC call timeout handler was delayed by -1.0520095750000564s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260709 12:13:00.627336 12610 connection.cc:256] RPC call timeout handler was delayed by -1.0370478460001777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
I20260709 12:13:00.634594 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.621366 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 450) took 1323 ms. Trace:
I20260709 12:13:00.636617 12488 rpcz_store.cc:276] 0709 12:12:59.298257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.347950 (+ 49693us) service_pool.cc:224] Handling call
0709 12:13:00.614443 (+1266493us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.642457 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 441) took 1477 ms. Trace:
I20260709 12:13:00.650089 15757 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.650956 15757 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.653993 15757 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:00.659765 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:00.660789 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1873) took 1511 ms. Trace:
I20260709 12:13:00.661298 12491 rpcz_store.cc:276] 0709 12:12:59.149401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.580821 (+1431420us) service_pool.cc:224] Handling call
0709 12:13:00.660686 (+ 79865us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.646656 12492 rpcz_store.cc:276] 0709 12:12:59.150331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.586332 (+1436001us) service_pool.cc:224] Handling call
0709 12:13:00.613009 (+ 26677us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.619173 (+  6164us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:00.619220 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:00.627209 (+  7989us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:00.627275 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.627373 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:00.627566 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.608736 16294 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:00.617297 16217 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:00.673100 13256 raft_consensus.cc:2368] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11.
I20260709 12:13:00.668231 12496 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:13:00.685552 12493 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:00.686899 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 448) took 1467 ms. Trace:
I20260709 12:13:00.687491 12496 rpcz_store.cc:276] 0709 12:12:59.213292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.606908 (+1393616us) service_pool.cc:224] Handling call
0709 12:13:00.637110 (+ 30202us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.680283 (+ 43173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.680609 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.677991 16138 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:00.699481 16138 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:00.701783 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.702108 16138 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:00.702235 16351 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.192s	user 0.000s	sys 0.003s
W20260709 12:13:00.716208 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 327290us
I20260709 12:13:00.719699 12489 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:00.721097 12496 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:00.695956 16085 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:00.722992 12488 raft_consensus.cc:1217] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Deduplicated request from leader. Original: 5.4->[7.5-7.6]   Dedup: 7.6->[]
I20260709 12:13:00.724715 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 455) took 1033 ms. Trace:
I20260709 12:13:00.725926 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 592) took 1063 ms. Trace:
I20260709 12:13:00.727540 12488 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [7.5-7.6]
I20260709 12:13:00.726461 12497 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:00.728438 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1878) took 1023 ms. Trace:
I20260709 12:13:00.728685 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2346) took 1009 ms. Trace:
I20260709 12:13:00.728879 12488 rpcz_store.cc:276] 0709 12:12:59.705388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.664162 (+958774us) service_pool.cc:224] Handling call
0709 12:13:00.722869 (+ 58707us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:00.728156 (+  5287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.728373 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.727624 12489 rpcz_store.cc:276] 0709 12:12:59.690848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.619043 (+928195us) service_pool.cc:224] Handling call
0709 12:13:00.698419 (+ 79376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.724308 (+ 25889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.724620 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:00.734786 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 109380us
I20260709 12:13:00.727847 13254 rpcz_store.cc:276] 0709 12:12:59.654509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:59.654673 (+   164us) service_pool.cc:224] Handling call
0709 12:13:00.698793 (+1044120us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:00.718177 (+ 19384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2008100224}
I20260709 12:13:00.729342 12497 rpcz_store.cc:276] 0709 12:12:59.719260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.673942 (+954682us) service_pool.cc:224] Handling call
0709 12:13:00.722084 (+ 48142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.728298 (+  6214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.728606 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.733554 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:00.750694 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178302us
W20260709 12:13:00.758374 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71961us
I20260709 12:13:00.698830 16328 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.727371 16085 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:00.760900 12491 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:13:00.773702 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1877) took 1081 ms. Trace:
I20260709 12:13:00.774209 12491 rpcz_store.cc:276] 0709 12:12:59.691712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.666807 (+975095us) service_pool.cc:224] Handling call
0709 12:13:00.746831 (+ 80024us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:00.772898 (+ 26067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.773592 (+   694us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.776600 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.794335 12489 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:00.798903 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 179913us
I20260709 12:13:00.764153 16309 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:00.808032 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44824.7us
I20260709 12:13:00.808678 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.811092 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2354) took 1051 ms. Trace:
I20260709 12:13:00.814677 12488 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:00.810247 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.821309 12491 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:13:00.823603 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.827625 12537 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:00.828065 12537 leader_election.cc:403] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:00.833091 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.813112 16045 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:00.821938 12489 rpcz_store.cc:276] 0709 12:12:59.759727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.743157 (+983430us) service_pool.cc:224] Handling call
0709 12:13:00.794057 (+ 50900us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.810708 (+ 16651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.811007 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:00.819547 16335 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:00.843137 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:00.840152 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 345048us
I20260709 12:13:00.829541 16265 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:00.850912 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 261016us
I20260709 12:13:00.857713 12394 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:00.829949 16334 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.838851 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:00.866854 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 248178us
I20260709 12:13:00.872596 12497 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:13:00.872933 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.873812 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.839111 16045 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:00.875524 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:00.878928 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.874500 15757 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.886384 15757 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:00.888842 15757 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:00.865077 16202 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:00.885083 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115048us
I20260709 12:13:00.889829 16202 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.893152 16202 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.892925 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:00.874619 16286 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:00.874894 16220 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
W20260709 12:13:00.896360 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67416.9us
I20260709 12:13:00.896555 16335 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:00.902182 16328 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:00.848454 16245 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
W20260709 12:13:00.917977 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50176.9us
I20260709 12:13:00.928756 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:00.926573 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.931663 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:00.872398 16314 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:00.908977 16045 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:00.935322 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.937088 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:00.927536 16303 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.783s	user 0.008s	sys 0.014s
I20260709 12:13:00.937919 12492 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:00.941183 12497 raft_consensus.cc:2368] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:00.942065 12496 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:00.932713 16265 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:00.838732 16085 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:00.949182 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 312767us
I20260709 12:13:00.950912 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:00.927208 15757 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:00.931864 16334 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:00.956832 16334 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:00.961192 16334 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:00.938552 16335 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:00.966003 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60511us
I20260709 12:13:00.967473 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:00.968575 16245 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:00.969220 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:00.969947 12496 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:00.943637 16309 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:00.972314 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:00.976387 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:00.983475 16220 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:13:00.991030 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2353) took 1225 ms. Trace:
I20260709 12:13:00.991642 12488 rpcz_store.cc:276] 0709 12:12:59.759398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.730229 (+970831us) service_pool.cc:224] Handling call
0709 12:13:00.766687 (+ 36458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.820701 (+ 54014us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:00.984529 (+163828us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b540037e454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:00.984739 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109720320}
W20260709 12:13:00.998481 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 105694us
I20260709 12:13:01.000913 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:01.006546 15816 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:01.014003 12497 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:01.006608 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:00.952656 16303 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.012111 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:01.025432 12563 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:00.965189 16335 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.003942 16085 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:01.018851 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:01.050223 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65684.2us
W20260709 12:13:01.055696 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60116us
I20260709 12:13:01.055913 12570 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:13:01.059116 16303 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:01.060961 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.062292 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:01.079715 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:01.066381 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:01.085191 12491 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:13:01.091198 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:01.093593 12491 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 5: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:01.010954 16309 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.094496 16181 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.107914 16181 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.110960 16181 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:01.081501 16220 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:13:01.107071 16312 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:13:01.117409 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128864us
I20260709 12:13:01.121979 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:01.122731 12491 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:01.024777 15757 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.145603 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 560) took 1324 ms. Trace:
I20260709 12:13:01.147831 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2350) took 1373 ms. Trace:
I20260709 12:13:01.148571 12492 rpcz_store.cc:276] 0709 12:12:59.743155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.689107 (+945952us) service_pool.cc:224] Handling call
0709 12:13:00.742927 (+ 53820us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:00.946356 (+203429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:01.116096 (+169740us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b540020e4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:01.116354 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":445729152}
I20260709 12:13:01.127940 16138 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.117892 12394 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:01.132963 12491 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:13:01.134333 16360 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:01.135306 16312 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:01.135756 16084 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.136443 16354 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.135s
I20260709 12:13:01.136587 16235 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:13:01.138545 16349 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.365s	user 0.003s	sys 0.000s
I20260709 12:13:01.141988 12462 leader_election.cc:400] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11.
I20260709 12:13:01.183430 12462 leader_election.cc:403] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:13:01.142908 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85444.7us
I20260709 12:13:01.185134 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:01.186935 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:01.144001 16358 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:01.188388 16303 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:01.188879 16303 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:01.189301 16303 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:01.126646 16215 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:13:01.146010 12570 rpcz_store.cc:276] 0709 12:12:59.820913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.177560 (+356647us) service_pool.cc:224] Handling call
0709 12:13:00.179420 (+  1860us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:01.056637 (+877217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:01.145335 (+ 88698us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:01.145540 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2012332288}
I20260709 12:13:01.162067 12488 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:01.165676 16312 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:01.172492 16084 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.173832 16354 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:01.179291 12490 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:01.203382 16360 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:01.203976 16360 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:01.191268 16110 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:01.225204 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:01.204648 16286 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:01.220239 16217 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.524s	user 0.012s	sys 0.002s
W20260709 12:13:01.230361 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:01.233052 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 91304us
I20260709 12:13:01.224586 16220 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11.
W20260709 12:13:01.258824 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71956.1us
I20260709 12:13:01.259256 16220 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:01.259806 16220 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:01.260079 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:01.260712 16220 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.262787 16181 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.263564 16181 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.264122 16138 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.266199 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:01.266726 12565 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:01.266947 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:01.268038 16138 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:01.224803 16360 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.268843 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 451) took 1945 ms. Trace:
I20260709 12:13:01.269256 12490 rpcz_store.cc:276] 0709 12:12:59.323818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.525561 (+1201743us) service_pool.cc:224] Handling call
0709 12:13:01.268602 (+743041us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:01.268761 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789739392}
I20260709 12:13:01.269791 16360 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:01.203477 16120 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:13:01.270907 12462 leader_election.cc:400] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11.
I20260709 12:13:01.271283 12462 leader_election.cc:403] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:13:01.231593 16110 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:01.268546 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:01.273403 16312 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.274685 15757 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:01.279217 16312 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:01.276367 16120 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:01.233304 16217 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:01.277400 16110 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:01.290891 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.263345 12394 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:01.274147 12493 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
W20260709 12:13:01.298310 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.307044 12563 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:01.318585 12497 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:01.294586 12567 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11. Re-sending same reply.
I20260709 12:13:01.320820 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:01.327306 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 262182us
I20260709 12:13:01.231331 16328 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:01.338100 12570 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:01.338872 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:01.340006 12488 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:01.343230 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.346693 16202 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:01.347344 16202 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:01.349619 16309 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.353503 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:01.321830 16245 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:01.275549 16084 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.354789 12491 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:01.329844 16117 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:01.362375 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:01.369042 12492 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:01.330096 15816 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:01.357146 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 444) took 2197 ms. Trace:
I20260709 12:13:01.335525 16328 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.391222 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 449) took 2116 ms. Trace:
I20260709 12:13:01.373409 12494 rpcz_store.cc:276] 0709 12:12:59.159935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.613145 (+1453210us) service_pool.cc:224] Handling call
0709 12:13:00.626107 (+ 12962us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:00.871700 (+245593us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:00.871779 (+    79us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:00.886551 (+ 14772us) log.cc:844] Serialized 26 byte log entry
0709 12:13:01.101871 (+215320us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:01.101982 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:01.102133 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:01.356801 (+254668us) raft_consensus.cc:1685] finished
0709 12:13:01.356824 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:01.357039 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25004,"raft.queue_time_us":53500,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":17906,"thread_start_us":440090,"threads_started":2,"wal-append.queue_time_us":458636}
W20260709 12:13:01.407089 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.390051 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:01.394994 12495 rpcz_store.cc:276] 0709 12:12:59.275359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.614691 (+1339332us) service_pool.cc:224] Handling call
0709 12:13:00.678321 (+ 63630us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:00.720058 (+ 41737us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:00.720091 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:00.739978 (+ 19887us) log.cc:844] Serialized 52 byte log entry
0709 12:13:01.085393 (+345415us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:01.106595 (+ 21202us) log.cc:844] Serialized 14 byte log entry
0709 12:13:01.174829 (+ 68234us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b18002fb1e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f
0709 12:13:01.176321 (+  1492us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b4800080058. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:01.390824 (+214503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:01.390912 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:01.390965 (+    53us) raft_consensus.cc:1685] finished
0709 12:13:01.390978 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:01.391148 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86145,"prepare.queue_time_us":21434,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":100545,"raft.run_cpu_time_us":6296,"raft.run_wall_time_us":433575,"spinlock_wait_cycles":17741312,"thread_start_us":569322,"threads_started":2,"wal-append.queue_time_us":485661}
W20260709 12:13:01.413662 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 339629us
I20260709 12:13:01.414981 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:01.419373 12570 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:13:01.428124 12489 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:01.348323 16202 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.384057 16181 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:01.389109 16309 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:01.430917 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:01.435104 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 126764us
I20260709 12:13:01.409790 16243 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:01.444792 13254 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 LEARNER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:13:01.450627 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.417294 16245 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.464074 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:01.462638 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100991us
W20260709 12:13:01.474617 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65322us
I20260709 12:13:01.487326 12491 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
W20260709 12:13:01.489107 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:01.493058 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103103us
W20260709 12:13:01.496537 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 490035us
W20260709 12:13:01.494527 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75681.4us
W20260709 12:13:01.501668 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.499614 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:01.500079 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 268020us
I20260709 12:13:01.520294 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:01.435487 15816 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:01.511023 12496 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.529s	user 0.012s	sys 0.000s
I20260709 12:13:01.549376 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.609447 12567 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260709 12:13:01.590425 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.613571 12496 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:01.612493 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115001us
W20260709 12:13:01.614144 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.592429 12570 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
W20260709 12:13:01.619010 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120526us
I20260709 12:13:01.616302 12570 pending_rounds.cc:85] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 4
I20260709 12:13:01.626957 16235 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.627295 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:01.627880 16235 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:01.582886 16265 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.001s
I20260709 12:13:01.525965 16202 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:01.629433 16370 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:01.622507 12570 pending_rounds.cc:107] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:13:01.640950 12570 raft_consensus.cc:2889] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:01.639060 16376 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:01.662369 16368 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:13:01.643929 16370 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:01.500706 16309 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.669320 16235 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.670097 16309 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.670756 16309 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.671312 16309 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:13:01.672187 12563 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:01.505947 16245 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.675405 16245 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:01.677029 16245 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:01.633347 16335 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:01.678304 16220 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:01.678144 16335 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.680936 16335 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:01.681742 16243 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:01.614898 16367 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.241s
I20260709 12:13:01.682020 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.683264 16367 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:01.683715 16367 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.684005 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.684705 16367 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.666033 16036 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:01.526392 16328 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.633774 16383 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:01.686283 16369 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:13:01.686457 16381 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:13:01.686825 16376 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.614672 16314 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:13:01.694054 16110 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:01.696014 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.696867 16366 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:01.665904 16370 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:01.689373 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2352) took 1930 ms. Trace:
W20260709 12:13:01.698375 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100122us
I20260709 12:13:01.694386 12394 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:13:01.703769 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 204253us
I20260709 12:13:01.709568 16265 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:01.741492 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:01.761343 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.608206 16303 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:01.711673 16334 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.750s	user 0.007s	sys 0.008s
W20260709 12:13:01.759186 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:01.741940 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 224) took 31478 ms (client timeout 15000 ms). Trace:
W20260709 12:13:01.764101 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.709048 16036 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:01.764788 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 207879us
W20260709 12:13:01.762437 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:01.761901 16045 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.809s	user 0.015s	sys 0.000s
I20260709 12:13:01.760944 12493 rpcz_store.cc:276] 0709 12:12:59.759019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:00.694424 (+935405us) service_pool.cc:224] Handling call
0709 12:13:01.273940 (+579516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:01.320748 (+ 46808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:01.689142 (+368394us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b540012a7d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:01.689296 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1202724608}
I20260709 12:13:01.776705 16045 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:01.778708 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:01.765690 16294 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.083s	user 0.010s	sys 0.006s
I20260709 12:13:01.761422 16383 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.774616 16120 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11.
I20260709 12:13:01.780368 12394 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 5 to 9, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 9 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:13:01.790628 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 302811us
I20260709 12:13:01.791682 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:01.780468 16367 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:01.767031 16369 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:01.793879 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:01.799537 12394 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:01.785729 15757 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:13:01.796149 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100231us
I20260709 12:13:01.798940 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:01.787297 16294 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:01.761169 16243 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:01.787127 16314 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:01.796779 13256 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:01.805924 12570 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:01.806309 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.806463 16366 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:01.807875 16303 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:01.793879 16328 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:01.813454 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101521us
I20260709 12:13:01.774757 16334 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.785530 16235 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:13:01.822746 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:01.786389 16202 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:13:01.795743 16389 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:01.802273 16309 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.830260 12566 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:01.834282 16045 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:01.864568 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53691.9us
I20260709 12:13:01.833986 12394 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 5 to 7, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 7 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:13:01.821051 16383 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.837981 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:01.838537 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:01.839849 16369 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:01.842672 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.847381 16334 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:01.821015 16314 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.852116 12569 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 3. Preceding OpId from leader: term: 10 index: 3. (term mismatch)
I20260709 12:13:01.882726 12569 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:13:01.883121 12569 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 9.3
I20260709 12:13:01.883384 12569 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:01.852945 16376 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.855414 16243 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.856067 16036 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:01.857782 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66235.5us
I20260709 12:13:01.826201 16393 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:01.882917 12394 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 11, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 11 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:13:01.884238 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58409.6us
W20260709 12:13:01.873416 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.647s (SENT) [suppressed 38 similar messages]
W20260709 12:13:01.884276 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46449.3us
I20260709 12:13:01.892072 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:01.905069 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.905784 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.908664 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:01.880722 16309 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:01.889671 16229 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:01.898612 16366 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:01.889663 16389 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:01.923120 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.917925 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:01.926843 12394 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:01.899452 16393 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:01.929121 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.926999 12569 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:01.885802 16314 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:01.939849 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66511.2us
I20260709 12:13:01.914834 16391 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:01.945690 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:01.944515 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:01.914353 16045 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:13:01.936653 16036 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:01.928196 16309 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:01.960747 12394 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 4, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 4 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:13:01.963829 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48534.1us
I20260709 12:13:01.928478 16366 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:01.934800 16334 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:01.958488 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:01.934465 16229 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.980353 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:01.763862 12395 rpcz_store.cc:269] 0709 12:12:30.263658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:30.269876 (+  6218us) service_pool.cc:224] Handling call
0709 12:12:33.136468 (+2866592us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:01.741540 (+28605072us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:12:51.390347 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:12:51.391081 (+   734us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:12:51.391145 (+    64us) write_op.cc:435] Acquired schema lock
0709 12:12:51.391160 (+    15us) tablet.cc:662] Decoding operations
0709 12:12:51.434748 (+ 43588us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:12:51.442948 (+  8200us) write_op.cc:641] Partition lock acquired for write op
0709 12:12:51.456924 (+ 13976us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:12:51.720480 (+263556us) tablet.cc:701] Row locks acquired
0709 12:12:51.720545 (+    65us) write_op.cc:260] PREPARE: finished
0709 12:12:51.720833 (+   288us) write_op.cc:270] Start()
0709 12:12:51.721064 (+   231us) write_op.cc:276] Timestamp: P: 1783599171720741 usec, L: 0
0709 12:12:51.721096 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:12:51.722228 (+  1132us) log.cc:844] Serialized 6278 byte log entry
0709 12:12:52.836574 (+1114346us) op_driver.cc:464] REPLICATION: finished
0709 12:12:54.017027 (+1180453us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b18003d4e60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:12:54.044852 (+ 27825us) write_op.cc:301] APPLY: starting
0709 12:12:54.045006 (+   154us) tablet.cc:1367] starting BulkCheckPresence
0709 12:12:54.045397 (+   391us) tablet.cc:1370] finished BulkCheckPresence
0709 12:12:54.045445 (+    48us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:12:54.065063 (+ 19618us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:12:54.065479 (+   416us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0709 12:12:54.065536 (+    57us) write_op.cc:312] APPLY: finished
0709 12:12:54.143756 (+ 78220us) log.cc:844] Serialized 107 byte log entry
0709 12:12:54.576424 (+432668us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:12:55.165804 (+589380us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:55.219575 (+ 53771us) write_op.cc:454] Released schema lock
0709 12:12:55.265267 (+ 45692us) write_op.cc:341] FINISH: Updating metrics
0709 12:12:55.473690 (+208423us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:12:55.473756 (+    66us) write_op.cc:454] Released schema lock
0709 12:12:57.739495 (+2265739us) spinlock_profiling.cc:243] Waited 8.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
Metrics: {"mutex_wait_us":22158,"reported_tablets":49,"spinlock_wait_cycles":6083441536,"child_traces":[["op",{"apply.queue_time_us":128199,"apply.run_cpu_time_us":31474,"apply.run_wall_time_us":1351396,"num_ops":15,"prepare.queue_time_us":303506,"prepare.run_cpu_time_us":110118,"prepare.run_wall_time_us":949697,"raft.queue_time_us":483060,"raft.run_cpu_time_us":27745,"raft.run_wall_time_us":211258,"replication_time_us":1074716,"spinlock_wait_cycles":125411968,"thread_start_us":2045940,"threads_started":5,"wal-append.queue_time_us":2003465,"wal-append.run_cpu_time_us":77970,"wal-append.run_wall_time_us":4534757}]]}
I20260709 12:13:01.934207 16294 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:01.982729 16397 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:01.988725 16397 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:01.989122 16397 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:01.990847 16334 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:01.992285 16334 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:01.985786 16307 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:01.994674 16307 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:01.995101 16307 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:01.985688 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:01.994009 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:01.989152 16360 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:13:01.998581 16360 consensus_queue.cc:1243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:13:01.998903 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:01.999528 12565 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:02.000001 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:01.927809 15757 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:13:01.958597 16391 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:02.007311 16391 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.008193 16294 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.008953 16294 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.009866 16372 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:02.011662 16294 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:01.986708 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:01.955154 16358 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:02.022486 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135923us
W20260709 12:13:02.023963 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.028167 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:01.981081 16366 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:01.963145 12394 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index -1 to 3, term changed from 7 to 9, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added, no longer has a pending config: VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129), VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 9 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:13:02.048485 15757 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:13:02.048837 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62221.8us
I20260709 12:13:02.049160 15757 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:02.050119 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:02.050997 16181 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:13:02.050578 15757 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:02.052186 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 97526.2us
I20260709 12:13:02.053540 16309 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.052409 15757 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:02.061084 12394 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 4, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 4 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:02.037781 16045 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:02.033147 16217 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
W20260709 12:13:02.083496 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.034121 16360 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:13:02.089417 12394 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:02.039827 16358 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:02.094449 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.098186 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48261.3us
I20260709 12:13:02.085270 16181 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:13:02.115177 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.116207 12643 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:13:02.118294 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.119370 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:02.124686 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.115149 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.128113 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.129014 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:02.048938 16229 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.130899 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.123598 12394 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:02.135862 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.140412 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:02.145126 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.127007 16294 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:02.088294 16389 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.150355 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:02.153275 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.154958 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.147650 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.153985 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:02.158406 12640 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:02.158950 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.156965 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.165638 12564 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:02.155968 12641 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
W20260709 12:13:02.171429 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.143433 16265 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:02.196393 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:02.199182 16405 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:13:02.200574 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:02.268116 12570 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:13:02.203827 16395 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.427s	user 0.003s	sys 0.000s
W20260709 12:13:02.209177 16220 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.003s
I20260709 12:13:02.268733 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.215803 16243 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:02.269860 16243 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:02.270351 16243 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:02.271682 16370 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.216732 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:02.226011 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92046.9us
I20260709 12:13:02.228338 12565 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:02.229707 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60524.1us
I20260709 12:13:02.148391 16294 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:02.158744 16398 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:13:02.252130 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106791us
I20260709 12:13:02.169203 16045 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:02.253914 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.145000 16303 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:13:02.258991 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.265944 12461 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.710s (SENT)
I20260709 12:13:02.146767 16309 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:02.155099 16393 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:02.274771 12572 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:02.277885 12565 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:02.304522 12637 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:02.279877 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:02.283201 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:02.285382 13259 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:02.274467 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:13:02.262981 16110 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:02.294414 16294 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:02.280261 16265 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:02.308267 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:02.317571 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.319032 16411 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:02.319589 16181 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.324482 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:02.323172 12640 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:02.329152 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.330179 12637 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:02.326259 16265 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:02.335991 16411 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:02.337126 16181 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.338304 16373 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:02.338981 16373 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.339743 16181 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:02.340286 12567 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:13:02.312316 16220 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.354141 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.351950 12641 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: [9.3-9.3]
I20260709 12:13:02.360078 12643 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:02.364169 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.368242 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:02.326341 16303 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:02.370431 13254 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:02.365046 12646 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 6.
I20260709 12:13:02.374660 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:02.379580 16309 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:02.388258 16309 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.388849 12645 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260709 12:13:02.380184 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:02.386433 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:02.386843 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.393904 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 574) took 1398 ms. Trace:
I20260709 12:13:02.394407 12566 rpcz_store.cc:276] 0709 12:13:00.995133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:01.003450 (+  8317us) service_pool.cc:224] Handling call
0709 12:13:01.024023 (+ 20573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:01.862804 (+838781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:02.393646 (+530842us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:02.393836 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1850714496}
I20260709 12:13:02.393965 12489 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:02.397405 12639 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:02.368966 16202 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.350250 16307 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:02.404388 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84474.1us
W20260709 12:13:02.413130 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103612us
I20260709 12:13:02.360558 16220 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.424688 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.425482 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.397927 16376 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:02.436668 12639 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:02.440809 12641 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:13:02.341145 16373 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:02.461971 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.475425 12645 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:13:02.478891 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:02.479693 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:02.480577 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.425374 16358 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:02.476902 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.482282 12566 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:02.441242 16370 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:02.416332 16202 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.490655 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.492772 12568 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:13:02.494578 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.402024 16045 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.498401 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.500066 12566 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:02.501551 12638 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 2. Preceding OpId from leader: term: 10 index: 3. (index mismatch)
I20260709 12:13:02.493454 16393 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:02.502696 16397 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.513s	user 0.011s	sys 0.003s
I20260709 12:13:02.511447 12646 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:13:02.516669 16202 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:02.517984 12566 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
W20260709 12:13:02.518149 16413 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 12:13:02.518435 16408 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:13:02.519105 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:02.519868 12461 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.104s (SENT)
W20260709 12:13:02.520780 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.518474 12566 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:13:02.522176 12566 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:02.503597 16374 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:02.490056 16366 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.507901 16394 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:02.530668 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90171.5us
I20260709 12:13:02.507957 16417 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:02.532954 12638 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:13:02.540297 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
W20260709 12:13:02.541585 16416 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:13:02.542198 12572 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:13:02.542555 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.544399 16366 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:02.544898 16328 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.716s	user 0.009s	sys 0.004s
I20260709 12:13:02.540925 16110 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:02.549767 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.620185 12461 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.123s (SENT)
I20260709 12:13:02.538192 16394 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:02.553691 16265 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:13:02.524263 16397 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:02.556443 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.559005 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.559592 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.560593 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:02.563541 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 463) took 1263 ms (client timeout 1576 ms). Trace:
I20260709 12:13:02.566069 12491 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:02.566404 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:02.625520 12490 rpcz_store.cc:269] 0709 12:13:01.303177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:01.320472 (+ 17295us) service_pool.cc:224] Handling call
0709 12:13:02.563370 (+1242898us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:02.563479 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2072598400}
I20260709 12:13:02.566809 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:02.626237 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:13:02.567291 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:02.567480 16335 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.627158 12571 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:02.567721 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:02.627583 16335 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.628504 16417 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:02.629590 16417 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.630316 16335 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:02.568102 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:02.569343 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.568825 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.570187 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.569900 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:02.632304 16307 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:02.632473 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:02.563578 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:02.569497 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.633318 12491 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:02.612733 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5.
I20260709 12:13:02.618242 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:02.634241 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:02.634636 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.634814 12491 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:02.635121 12643 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:13:02.635707 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:02.589910 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.620592 12637 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:02.625238 12644 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9. Re-sending same reply.
W20260709 12:13:02.631294 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4579769360000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260709 12:13:02.637853 12641 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
W20260709 12:13:02.638032 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4210769359999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
I20260709 12:13:02.632864 12497 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:02.636193 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:02.621388 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100878us
I20260709 12:13:02.639376 12645 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:02.628101 12489 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:02.624655 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.641541 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:02.498546 16303 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.626271 12492 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
W20260709 12:13:02.640341 16245 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.963s	user 0.013s	sys 0.003s
W20260709 12:13:02.655236 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2925860660000126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
W20260709 12:13:02.655432 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 823615us
I20260709 12:13:02.625218 16394 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:02.662513 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2761769359999562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
I20260709 12:13:02.657202 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:02.673178 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2554769359999227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260709 12:13:02.675321 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2079860659998758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 12:13:02.675693 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0853769360001024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
W20260709 12:13:02.676072 12536 connection.cc:256] RPC call timeout handler was delayed by -1.072249735000014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260709 12:13:02.676441 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0603860659998645s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
W20260709 12:13:02.676829 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0394769360000282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20190000000000002s for a response
I20260709 12:13:02.538533 16374 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.659952 16036 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.678618 16360 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:02.679334 16360 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:02.679720 16360 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:02.680331 15816 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:02.678758 16036 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.637795 16202 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.639122 16370 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:02.682345 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.743319 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65677.7us
W20260709 12:13:02.788183 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141627us
I20260709 12:13:02.820895 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:02.687144 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.646100 16138 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:02.822850 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135796us
W20260709 12:13:02.660418 15757 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.605s	user 0.009s	sys 0.005s
I20260709 12:13:02.680465 16374 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:02.828882 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:02.673157 16393 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:02.791865 16243 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.002s
I20260709 12:13:02.696568 16245 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:02.681977 16217 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.146s
I20260709 12:13:02.795495 16411 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:02.814909 16220 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:02.848392 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:02.854939 16314 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:02.858805 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:02.861948 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.815784 15816 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:02.867760 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.864420 12461 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.330s (SENT)
I20260709 12:13:02.868525 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:02.866261 12489 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:02.862708 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.645985 16369 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.186s
I20260709 12:13:02.681910 16375 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.872437 16427 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:02.826750 16358 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:02.875727 13259 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:13:02.656518 16366 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.183s
W20260709 12:13:02.879403 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:02.885673 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66460.4us
W20260709 12:13:02.888723 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49750us
W20260709 12:13:02.930778 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:02.916603 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.933318 12566 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:02.934911 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.937827 16372 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:02.838328 16217 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:13:02.916750 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52361.7us
I20260709 12:13:02.818709 16202 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.864069 16411 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:02.923856 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:02.923892 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103488us
I20260709 12:13:02.932451 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.832290 16138 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.960523 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:02.890044 16417 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:13:02.955572 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.967823 12497 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:13:02.965051 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:02.864013 16243 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:02.927861 16220 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:02.988376 12489 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:13:02.927569 16358 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:02.971828 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:02.929083 16235 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.865409 16393 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:02.941912 16309 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:02.995579 16378 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:02.948647 16217 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:02.942732 16427 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:02.962631 16411 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:02.961346 16335 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:03.003376 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.919530 16245 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260709 12:13:02.891575 15757 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260709 12:13:03.005506 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:03.018649 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:02.839445 16110 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.035328 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:03.026463 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60542us
I20260709 12:13:03.043829 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:03.003557 16378 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:02.916302 16375 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.017709 16309 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.003834 16235 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:03.155071 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:03.018834 12492 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:13:03.197922 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 192475us
I20260709 12:13:03.199082 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:03.084041 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:03.152030 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:02.997257 15816 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:03.230904 12567 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:03.073987 16120 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:03.233243 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:03.245455 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:03.140177 16426 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
I20260709 12:13:03.047925 16335 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:03.265875 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:03.434130 16120 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:03.236090 12564 raft_consensus.cc:2376] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11. Re-sending same reply.
I20260709 12:13:03.446969 12644 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:13:03.141369 16430 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:13:03.196781 16036 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:03.074632 16397 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:03.435793 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:03.448879 12644 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:03.456724 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:03.444912 12569 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.563s	user 0.009s	sys 0.002s
W20260709 12:13:03.449182 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 375940us
I20260709 12:13:03.457803 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:03.103299 16265 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:03.120342 16376 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:03.118842 16045 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:03.235268 16303 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:13:03.205794 16378 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:03.486222 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 481397us
W20260709 12:13:03.497187 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.031s	sys 0.113s
W20260709 12:13:03.489279 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 482813us
W20260709 12:13:03.505574 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.031s	sys 0.115s
W20260709 12:13:03.506367 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 307346us
W20260709 12:13:03.446815 16422 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.025s	sys 0.042s
I20260709 12:13:03.479684 12644 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.512352 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:03.519987 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:03.512612 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:03.514158 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:03.454092 16120 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:03.468415 12567 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:03.191578 16358 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.525171 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:13:03.481775 15816 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:03.480559 16419 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:03.542213 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:03.518236 16422 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.025s	sys 0.043s
I20260709 12:13:03.146147 16217 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.460542 16418 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:03.566016 12637 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:03.518052 16374 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:03.551219 12566 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:03.442968 16444 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.306s
I20260709 12:13:03.575371 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:03.525585 16394 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.847s	user 0.012s	sys 0.000s
W20260709 12:13:03.565148 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 243163us
W20260709 12:13:03.578724 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:03.572561 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114151us
I20260709 12:13:03.571080 12566 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:13:03.464546 16245 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:03.588342 12564 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:03.583410 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 461) took 1215 ms. Trace:
I20260709 12:13:03.196861 16427 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:03.604068 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:03.499198 16376 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:03.626158 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:03.627378 12644 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:03.592123 13256 rpcz_store.cc:276] 0709 12:13:02.349778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.350020 (+   242us) service_pool.cc:224] Handling call
0709 12:13:02.357212 (+  7192us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:02.383730 (+ 26518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:02.383751 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:02.565327 (+181576us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b4c0020a280. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3a96644 00007ffbf3abdf2d 00007ffbf3abdb93 00007ffbf57ee18b 00007ffbf57eecb5 00007ffbf57f0004 00007ffbf57ef729 00007ffbf57f4d96 00007ffbf57e9adb
0709 12:13:02.566175 (+   848us) log.cc:844] Serialized 564 byte log entry
0709 12:13:02.796292 (+230117us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:02.796338 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:02.796434 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:03.555103 (+758669us) raft_consensus.cc:1685] finished
0709 12:13:03.555127 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:03.565387 (+ 10260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8049408,"thread_start_us":178637,"threads_started":1,"wal-append.queue_time_us":324139}
I20260709 12:13:03.551540 16419 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:03.661473 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77581.9us
I20260709 12:13:03.547312 16443 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:03.557510 16265 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:03.590237 16084 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:13:03.672552 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 166347us
I20260709 12:13:03.573608 16202 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:03.574246 16243 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:03.522630 16446 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.115s
I20260709 12:13:03.676285 12645 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:03.679791 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:03.688418 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:03.680969 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:03.660847 16217 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:03.688369 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170166us
I20260709 12:13:03.568624 16450 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.128s
W20260709 12:13:03.674886 16428 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.678s	user 0.002s	sys 0.000s
I20260709 12:13:03.658496 16434 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:03.693800 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:03.696518 16439 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.546s	user 0.000s	sys 0.002s
I20260709 12:13:03.698185 16084 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:03.701618 16452 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.141s
I20260709 12:13:03.705757 16448 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:03.689174 16265 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:03.732795 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:03.706991 16438 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.576s	user 0.001s	sys 0.000s
I20260709 12:13:03.708356 16309 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:03.710026 16442 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:13:03.711004 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:03.655776 16245 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:03.776980 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:03.710711 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120550us
I20260709 12:13:03.673904 16443 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:03.584052 16374 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.831947 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:03.587016 16394 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:03.675622 16419 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:03.715346 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:03.724831 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:03.726670 12564 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:13:03.743081 12461 leader_election.cc:336] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.647s (SENT)
I20260709 12:13:03.693899 16202 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:03.828651 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:03.831207 12646 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:03.834255 12566 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:13:04.158470 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1992) took 1597 ms. Trace:
W20260709 12:13:04.366367 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:04.366845 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:04.366809 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 651735us
I20260709 12:13:04.367121 16036 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:13:04.367758 16358 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:04.367942 16036 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:04.368403 16358 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.034403 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 579) took 2135 ms. Trace:
I20260709 12:13:04.370128 16394 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:04.370265 12565 rpcz_store.cc:276] 0709 12:13:01.900463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.304349 (+403886us) service_pool.cc:224] Handling call
0709 12:13:02.320522 (+ 16173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:03.041966 (+721444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.034019 (+992053us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:04.034307 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1604824832}
I20260709 12:13:04.370975 16358 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.371325 12568 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:04.368660 16036 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:04.373064 12565 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:04.393208 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:03.882942 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77778.7us
I20260709 12:13:03.934533 16335 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:04.106314 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2087) took 2113 ms (client timeout 1691 ms). Trace:
W20260709 12:13:04.400329 12569 rpcz_store.cc:269] 0709 12:13:01.918700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:01.961009 (+ 42309us) service_pool.cc:224] Handling call
0709 12:13:04.031913 (+2070904us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:04.032136 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2032342400}
I20260709 12:13:04.365298 12643 rpcz_store.cc:276] 0709 12:13:02.560946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.635823 (+ 74877us) service_pool.cc:224] Handling call
0709 12:13:02.636128 (+   305us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:02.636533 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:02.636560 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:02.637176 (+   616us) log.cc:844] Serialized 52 byte log entry
0709 12:13:02.938431 (+301255us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:13:02.938575 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:02.938961 (+   386us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:04.158071 (+1219110us) raft_consensus.cc:1685] finished
0709 12:13:04.158101 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:04.158375 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79640,"raft.queue_time_us":2199,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":9995,"thread_start_us":298455,"threads_started":1,"wal-append.queue_time_us":347924}
W20260709 12:13:03.839555 12572 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.006s	sys 0.008s
W20260709 12:13:04.406308 12572 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.331s	user 0.044s	sys 0.034s
I20260709 12:13:03.847652 12645 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:13:03.877285 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 211739us
I20260709 12:13:04.146459 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:04.387526 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 609275us
W20260709 12:13:04.208650 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.011s	sys 0.051s
W20260709 12:13:03.978914 16307 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.012s	sys 0.013s
W20260709 12:13:04.417109 16307 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.038s	sys 0.039s
I20260709 12:13:04.417866 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.400080 16202 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.419462 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.420496 16202 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.420634 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.399256 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 467) took 1998 ms. Trace:
I20260709 12:13:03.832321 16444 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:04.423611 12640 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:04.435372 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1999) took 1200 ms. Trace:
I20260709 12:13:04.435878 12640 rpcz_store.cc:276] 0709 12:13:03.234749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:03.234974 (+   225us) service_pool.cc:224] Handling call
0709 12:13:03.458265 (+223291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:04.423402 (+965137us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b5400237920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:13:04.435002 (+ 11600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.435290 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7363328}
I20260709 12:13:04.153067 16461 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.511s
I20260709 12:13:04.421119 16036 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:03.834013 12642 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:04.409374 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.019s	sys 0.058s
W20260709 12:13:04.014803 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:04.152803 16436 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.689s	user 0.001s	sys 0.001s
W20260709 12:13:04.176050 16447 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:13:04.437582 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:04.410904 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 264579us
W20260709 12:13:04.411532 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:03.776171 16448 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.190205 16265 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.432065 12646 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:13:04.422907 12490 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260709 12:13:04.561024 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:04.468263 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56029.1us
W20260709 12:13:04.421324 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 678276us
I20260709 12:13:04.481513 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:04.444100 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:04.477960 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:04.485880 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:04.562970 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 996517us
I20260709 12:13:03.841725 16443 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:04.566807 12494 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:04.567142 12489 raft_consensus.cc:2376] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11. Re-sending same reply.
I20260709 12:13:04.436751 12495 rpcz_store.cc:276] 0709 12:13:02.366377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.366540 (+   163us) service_pool.cc:224] Handling call
0709 12:13:02.387431 (+ 20891us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:02.491519 (+104088us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:02.491554 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:02.495101 (+  3547us) log.cc:844] Serialized 52 byte log entry
0709 12:13:02.640400 (+145299us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:02.673165 (+ 32765us) log.cc:844] Serialized 14 byte log entry
0709 12:13:02.684931 (+ 11766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:03.734263 (+1049332us) spinlock_profiling.cc:243] Waited 95.7 ms on lock 0x7b540020bcd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:03.734339 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:03.734416 (+    77us) raft_consensus.cc:1685] finished
0709 12:13:03.770467 (+ 36051us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:04.364558 (+594091us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19043,"prepare.queue_time_us":37570,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":17622,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":6120,"spinlock_wait_cycles":220182272,"thread_start_us":126986,"threads_started":1,"wal-append.queue_time_us":423513,"wal-append.run_cpu_time_us":3949,"wal-append.run_wall_time_us":1535788}
I20260709 12:13:04.333180 16309 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:04.426558 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.919s	user 0.003s	sys 0.017s
W20260709 12:13:04.586684 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.614s	user 0.003s	sys 0.027s
I20260709 12:13:04.574811 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:04.539337 12644 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.866s	user 0.012s	sys 0.000s
I20260709 12:13:04.587975 12644 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:13:04.588593 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 1998) took 1509 ms. Trace:
I20260709 12:13:03.831909 16245 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:04.409936 16454 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:04.576053 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.475449 16383 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.101s
I20260709 12:13:04.578609 16335 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:04.428706 16427 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:04.579726 16202 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:04.580307 16463 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:04.579927 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135877us
W20260709 12:13:04.427261 16294 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 2.085s	user 0.015s	sys 0.001s
I20260709 12:13:04.583340 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:04.612124 13256 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:04.566129 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:04.590462 12610 leader_election.cc:400] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:04.076004 16417 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:04.565114 16110 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.009s	sys 0.038s
I20260709 12:13:04.588871 12644 rpcz_store.cc:276] 0709 12:13:03.078981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:03.229101 (+150120us) service_pool.cc:224] Handling call
0709 12:13:03.446795 (+217694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:04.588328 (+1141533us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.588542 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:04.457572 16360 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.777s	user 0.005s	sys 0.009s
W20260709 12:13:04.564795 16314 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.689s	user 0.008s	sys 0.007s
I20260709 12:13:04.594003 12646 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:04.596531 12565 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:13:04.423331 16374 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:04.598371 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:04.578609 16229 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:04.598717 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:04.599777 13260 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:04.562640 16444 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:04.473920 16376 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:04.602837 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:04.610497 13253 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
W20260709 12:13:04.596387 16412 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:13:04.615334 12610 leader_election.cc:403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:13:04.616376 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:04.616297 16358 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:04.620096 16417 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.620645 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.307603 16459 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.667s
I20260709 12:13:04.610082 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:04.624539 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.603662 16469 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:04.636837 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.638151 13254 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:04.189955 16084 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.680509 16110 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:04.682209 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.681874 16469 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.682979 12563 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:04.684005 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.684749 12563 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:04.662636 12565 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:04.685968 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.449369 16419 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.687646 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:04.688709 16378 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.149s	user 0.007s	sys 0.006s
I20260709 12:13:04.691669 12569 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:13:04.630550 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:04.604835 16427 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:04.692659 16427 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:04.592996 16309 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.651696 13257 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:04.628510 16468 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:04.687172 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:04.694578 16217 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.980s	user 0.010s	sys 0.002s
I20260709 12:13:04.693994 16468 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.694998 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:04.695323 16309 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:04.695797 12563 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:04.697105 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2002) took 1182 ms. Trace:
I20260709 12:13:04.697532 12639 rpcz_store.cc:276] 0709 12:13:03.514787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:03.514992 (+   205us) service_pool.cc:224] Handling call
0709 12:13:03.548681 (+ 33689us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:03.587763 (+ 39082us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:03.587820 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:03.672375 (+ 84555us) log.cc:844] Serialized 26 byte log entry
0709 12:13:03.892132 (+219757us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:13:03.892189 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:03.892280 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:04.696917 (+804637us) raft_consensus.cc:1685] finished
0709 12:13:04.696937 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:04.697052 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14564,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":64205,"thread_start_us":153436,"threads_started":1,"wal-append.queue_time_us":953747}
I20260709 12:13:04.601948 16454 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.699649 16419 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:04.701143 16454 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.702865 16468 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.703063 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:04.703801 12565 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
W20260709 12:13:04.704048 16437 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.565s	user 0.000s	sys 0.009s
I20260709 12:13:04.622437 16294 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.643571 16229 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.711400 16448 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.621616 16463 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.716734 16446 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:04.643591 16444 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:04.718755 16036 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.643738 16455 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:04.722131 16358 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.754894 16469 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.709043 16427 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:04.724115 15816 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.166s	user 0.007s	sys 0.007s
I20260709 12:13:04.725973 16417 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:04.727170 16120 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.166s	user 0.010s	sys 0.002s
I20260709 12:13:04.659778 16394 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:04.610993 16335 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.634044 16360 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:04.682039 16473 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
W20260709 12:13:04.728119 16411 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.656s	user 0.015s	sys 0.000s
W20260709 12:13:04.747344 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:04.755825 12565 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:04.762228 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:04.823012 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:04.731880 16461 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:04.689137 16378 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.794559 12566 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:04.743830 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:04.802917 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 230576us
I20260709 12:13:04.805531 13257 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:04.805944 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.088s	user 0.002s	sys 0.013s
W20260709 12:13:04.813021 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 227271us
I20260709 12:13:04.787186 16446 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:04.819818 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196617us
W20260709 12:13:04.827739 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.136s	user 0.003s	sys 0.013s
W20260709 12:13:04.829010 13218 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.590s (SENT)
I20260709 12:13:04.835364 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:04.812193 16393 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:04.844696 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.591s (SENT)
I20260709 12:13:04.845263 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:04.856261 12492 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:13:04.871439 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:04.876812 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:04.878921 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:04.901846 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 577) took 2977 ms. Trace:
W20260709 12:13:04.902364 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 327574us
I20260709 12:13:04.831301 16354 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:04.902918 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75476.6us
I20260709 12:13:04.903941 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:04.850028 16383 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:04.905777 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:04.847155 16461 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:04.826223 16394 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.814843 16120 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.922537 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:04.941112 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:04.926635 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81963.7us
W20260709 12:13:04.942436 13218 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.690s (SENT)
I20260709 12:13:04.806722 16455 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:04.929641 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.902993 12571 rpcz_store.cc:276] 0709 12:13:01.887848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.275315 (+387467us) service_pool.cc:224] Handling call
0709 12:13:02.626983 (+351668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:02.627601 (+   618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.864422 (+2236821us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54001a22d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:04.864649 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":742490880}
I20260709 12:13:04.905861 16307 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:04.936322 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.936920 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:04.938287 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.868209 16202 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:04.845646 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:04.944119 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:04.832788 16411 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:04.905212 15816 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:04.918875 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63857.4us
I20260709 12:13:04.956059 16110 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:04.956753 16450 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:13:04.963410 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:04.980415 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:04.970450 12639 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:05.020001 16110 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:13:05.020797 16470 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
W20260709 12:13:05.021693 13287 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:13:05.021858 16110 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:05.022748 16110 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:13:05.023980 16477 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:13:05.025421 16360 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.025749 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2210) took 1088 ms. Trace:
I20260709 12:13:05.026257 16418 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:13:05.026623 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.027500 12639 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:05.028258 16418 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:05.028822 16418 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:05.026150 12567 rpcz_store.cc:276] 0709 12:13:03.937242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:03.937544 (+   302us) service_pool.cc:224] Handling call
0709 12:13:04.125083 (+187539us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:04.125720 (+   637us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:04.125773 (+    53us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:13:04.403257 (+277484us) log.cc:844] Serialized 78 byte log entry
0709 12:13:04.778692 (+375435us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:13:04.778740 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.778824 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:05.025472 (+246648us) raft_consensus.cc:1685] finished
0709 12:13:05.025498 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:05.025685 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":860,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":18897,"thread_start_us":348174,"threads_started":1,"wal-append.queue_time_us":385885}
I20260709 12:13:05.030110 16383 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:05.031773 16335 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.028549 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.032953 12639 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:05.033425 15816 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.033906 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.034770 12639 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:05.035691 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.035920 15816 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:05.036543 12639 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:13:04.974442 12646 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:04.989668 12642 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:05.039254 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:05.040477 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.041098 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:04.926266 16181 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:05.041738 16294 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.041904 16181 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:05.042387 16181 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:04.925969 16354 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:05.043458 16358 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:04.949807 16448 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:05.045904 16448 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:05.046006 16490 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:05.046437 16448 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:04.947060 16036 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:05.047159 16036 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.045207 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:05.050237 12640 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:05.051348 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.052232 12640 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:13:05.038046 16411 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:04.992446 16481 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:05.054155 16481 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:05.054608 16481 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:05.054814 16463 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:05.055501 16354 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.055944 12646 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260709 12:13:05.056306 12646 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:04.996496 16459 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:13:05.018700 16450 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:05.058720 16440 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.863s	user 0.009s	sys 0.000s
I20260709 12:13:05.058867 16450 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:05.060734 12563 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:05.063019 12563 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:05.056516 12646 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:13:05.063670 12646 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:05.064930 16453 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260709 12:13:05.042490 16427 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:05.066706 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50404.2us
I20260709 12:13:05.037426 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:05.068020 12639 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:05.046511 16490 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:05.070307 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.085552 12644 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:05.082468 12494 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.000s
I20260709 12:13:05.086261 12494 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
W20260709 12:13:05.083140 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 177489us
I20260709 12:13:05.096772 12563 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:13:05.303432 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.388084 12563 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:13:05.150991 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63031.9us
W20260709 12:13:05.350157 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 395014us
W20260709 12:13:05.389475 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:05.209311 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 220641us
I20260709 12:13:05.409443 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:05.424770 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:05.425573 12494 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 6.
I20260709 12:13:05.426540 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:05.411389 16372 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.022s	sys 0.057s
I20260709 12:13:05.186270 16378 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:05.122359 16443 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.511s	user 0.000s	sys 0.012s
I20260709 12:13:05.187435 16120 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.301241 16469 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:05.150388 16229 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:05.122418 16036 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.185737 16217 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.275s
I20260709 12:13:05.323626 16394 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:05.439606 16370 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
W20260709 12:13:05.461067 16475 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.448s	user 0.001s	sys 0.000s
I20260709 12:13:05.450974 16480 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:05.464332 16495 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:13:05.323257 16448 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:05.454972 16245 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.852s	user 0.009s	sys 0.004s
I20260709 12:13:05.369038 16202 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.540014 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 466) took 3167 ms. Trace:
I20260709 12:13:05.540585 12493 rpcz_store.cc:276] 0709 12:13:02.365896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.366059 (+   163us) service_pool.cc:224] Handling call
0709 12:13:02.390644 (+ 24585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:02.688204 (+297560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:02.688243 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:02.688299 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:02.688350 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.532890 (+2844540us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:05.532966 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:05.533199 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":683746688}
I20260709 12:13:05.329259 16450 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:05.462021 16444 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.646s	user 0.012s	sys 0.006s
W20260709 12:13:05.510146 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.105775 16490 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:05.197665 16418 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.457850 16443 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:05.568420 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.567723 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:05.568472 13218 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.057s (SENT)
I20260709 12:13:05.484272 16469 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:05.501776 16181 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.507375 16481 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:05.596328 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 659425us
W20260709 12:13:05.594460 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.151172 16461 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:05.632649 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.600986 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:05.570340 16444 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:05.556787 16245 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.636266 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:05.601440 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:05.635076 12639 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.004s
I20260709 12:13:05.612030 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:05.650904 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.152s (SENT)
I20260709 12:13:05.534404 16480 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.657307 12639 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:13:05.679910 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:05.610347 16217 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:05.637589 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:05.616082 16448 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.624163 16469 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:05.657974 16503 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:05.614331 16451 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.075s
I20260709 12:13:05.690155 16358 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:05.613976 16036 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:13:05.696233 16481 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.607875 16450 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:05.697692 16120 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:05.699944 16181 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:05.700800 16488 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:05.704041 16480 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:05.706296 16443 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:05.638377 16220 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:13:05.709043 16444 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:05.636978 16394 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:05.693723 16378 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:13:05.603814 16370 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:05.689337 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 1996) took 3027 ms (client timeout 1959 ms). Trace:
I20260709 12:13:05.769685 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.770538 12637 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:05.730160 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:05.656061 16504 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:13:05.709067 16502 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.658s	user 0.001s	sys 0.000s
I20260709 12:13:05.711597 16464 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:05.743692 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:05.779354 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179427us
I20260709 12:13:05.731423 16217 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.656196 16335 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:13:05.612335 16500 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.122s
I20260709 12:13:05.605525 16501 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.105s
W20260709 12:13:05.770062 12641 rpcz_store.cc:269] 0709 12:13:02.662048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.689009 (+3026961us) service_pool.cc:211] Skipping call since client already timed out
0709 12:13:05.689221 (+   212us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:13:05.714185 12565 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:13:05.774605 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:05.784188 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:05.740257 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2127) took 1109 ms. Trace:
I20260709 12:13:05.840180 12563 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:05.832024 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48784.2us
W20260709 12:13:05.798003 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.027s	sys 0.059s
W20260709 12:13:05.842193 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.036s	sys 0.060s
I20260709 12:13:05.844610 12567 rpcz_store.cc:276] 0709 12:13:04.631175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.030602 (+399427us) service_pool.cc:224] Handling call
0709 12:13:05.030972 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:05.031210 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:05.569239 (+538029us) log.cc:844] Serialized 14 byte log entry
0709 12:13:05.739782 (+170543us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:05.739835 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:05.739897 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.739995 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:05.740179 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115523,"prepare.queue_time_us":97173,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":3608,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":10172,"spinlock_wait_cycles":326912,"thread_start_us":586714,"threads_started":2}
I20260709 12:13:05.770290 16480 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:05.772954 16504 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:05.859068 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:05.861515 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:05.782437 16450 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:05.863945 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:05.750099 16358 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.789551 16181 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:05.772789 16394 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:05.780263 16370 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:05.917708 12490 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:13:05.883737 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:05.880206 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196480us
W20260709 12:13:05.870429 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 269773us
W20260709 12:13:05.874804 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.893744 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:05.860652 16504 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:05.974538 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:05.934532 12567 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:05.894146 16181 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:05.865144 16502 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:13:05.976425 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:05.980939 12567 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:05.893211 16484 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.843s	user 0.005s	sys 0.001s
I20260709 12:13:05.867005 16501 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:13:06.017017 16245 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:06.019966 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:05.870993 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
W20260709 12:13:06.056245 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:06.087157 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:05.870870 16444 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:13:06.124002 13218 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.560s (SENT)
I20260709 12:13:06.099193 12644 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:06.133890 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:06.124289 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113832us
I20260709 12:13:05.914736 16389 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
I20260709 12:13:06.128726 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2103) took 3772 ms. Trace:
I20260709 12:13:05.911127 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:06.127194 16520 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:06.129015 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.146430 12563 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:06.134464 12644 raft_consensus.cc:614] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:06.135185 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:06.138242 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:06.151743 16138 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:06.153584 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 218482us
I20260709 12:13:06.152271 16520 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:06.136740 12572 rpcz_store.cc:276] 0709 12:13:02.356478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.642438 (+285960us) service_pool.cc:224] Handling call
0709 12:13:02.858281 (+215843us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:04.407195 (+1548914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:04.407245 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:04.548123 (+140878us) log.cc:844] Serialized 26 byte log entry
0709 12:13:04.901325 (+353202us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:13:04.901416 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.458905 (+557489us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54003e0154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:05.458963 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:05.731569 (+272606us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:06.128473 (+396904us) raft_consensus.cc:1685] finished
0709 12:13:06.128501 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:06.128661 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29183,"raft.queue_time_us":188231,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":208500,"spinlock_wait_cycles":552776320,"thread_start_us":1568107,"threads_started":2,"wal-append.queue_time_us":454947}
I20260709 12:13:05.872212 16450 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:06.157866 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.158727 12572 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:06.142258 16375 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.161578 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2211) took 2060 ms. Trace:
I20260709 12:13:06.164701 13262 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:06.165179 12566 rpcz_store.cc:276] 0709 12:13:04.100954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.365727 (+264773us) service_pool.cc:224] Handling call
0709 12:13:04.366059 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:04.826265 (+460206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:06.161210 (+1334945us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:06.161486 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984771840}
W20260709 12:13:06.164470 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 153579us
W20260709 12:13:06.172035 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:06.130070 16501 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:06.167593 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:06.182466 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.132194 16444 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:06.200758 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:06.170159 16519 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.248s	user 0.003s	sys 0.008s
I20260709 12:13:06.201745 12566 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:13:06.210362 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:06.211162 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43666.4us
I20260709 12:13:06.211383 16378 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:13:06.211889 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:06.212463 16527 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:06.212898 16378 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:06.213137 16527 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:06.214041 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2123) took 1586 ms. Trace:
I20260709 12:13:06.214112 16527 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.194022 16481 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:06.216116 16378 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:06.182363 16120 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:06.214752 12564 rpcz_store.cc:276] 0709 12:13:04.627868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.007723 (+379855us) service_pool.cc:224] Handling call
0709 12:13:05.023980 (+ 16257us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:05.024450 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:05.410101 (+385651us) log.cc:844] Serialized 14 byte log entry
0709 12:13:05.888774 (+478673us) log.cc:844] Serialized 14 byte log entry
0709 12:13:05.890073 (+  1299us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:05.935545 (+ 45472us) log.cc:844] Serialized 26 byte log entry
0709 12:13:05.935719 (+   174us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:05.935755 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.935825 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:06.213779 (+277954us) raft_consensus.cc:1685] finished
0709 12:13:06.213804 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:06.213961 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106117,"prepare.queue_time_us":109201,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":88,"raft.queue_time_us":47280,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"spinlock_wait_cycles":57471104,"thread_start_us":697250,"threads_started":3,"wal-append.queue_time_us":157642}
I20260709 12:13:06.199252 16501 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.161979 16450 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:06.130117 15757 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:06.212634 16490 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.615s	user 0.008s	sys 0.017s
I20260709 12:13:06.227980 12641 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 9. Ops: []
I20260709 12:13:06.228868 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:06.202831 16458 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:06.241374 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:06.244982 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.257329 12637 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:06.261346 12564 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:06.259723 12563 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:06.259811 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:06.182068 16488 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:13:06.216418 16481 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:06.223963 16378 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:13:06.264333 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100517us
I20260709 12:13:06.271054 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.275281 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.276319 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:06.279018 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.279724 12637 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:06.196723 16504 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:06.258257 16217 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.234565 16202 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:13:06.264585 16374 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:06.412467 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.414703 12641 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:06.270285 16181 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:06.269207 16481 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:06.272934 16045 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:06.274641 16378 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:06.272344 16358 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.278820 16036 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:06.263818 15757 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.436089 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:06.444957 16508 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.016s	sys 0.017s
I20260709 12:13:06.277706 16235 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.418852 16374 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:06.452863 16516 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.324s	user 0.000s	sys 0.000s
I20260709 12:13:06.454255 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.418095 16370 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:06.263468 16490 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.496937 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:06.436371 16045 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:06.413690 16529 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:06.569286 12572 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:06.445403 16531 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:13:06.429922 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.574764 12563 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:06.516871 16374 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:06.518465 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2122) took 1890 ms. Trace:
I20260709 12:13:06.520882 12643 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: [9.7-11.8]
I20260709 12:13:06.521653 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.577597 12640 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:06.479086 16458 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:06.523350 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:06.579000 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:06.497838 16508 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.016s	sys 0.018s
I20260709 12:13:06.525929 16501 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:06.528810 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:06.585938 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57261.1us
I20260709 12:13:06.586290 16045 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.587349 16501 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:06.587647 16045 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:06.588047 16370 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:06.588587 16370 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.589427 16482 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:06.510018 12641 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:06.589879 16482 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:06.590763 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.591001 16501 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:06.591439 12641 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12. Re-sending same reply.
I20260709 12:13:06.592454 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.592204 16501 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.572856 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.593978 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:06.268658 16444 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:06.583274 13257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 620) took 1880 ms (client timeout 1893 ms). Trace:
I20260709 12:13:06.583930 13254 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:06.583770 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.584812 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.590909 16482 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:06.575304 12571 rpcz_store.cc:276] 0709 12:13:04.627667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.994942 (+367275us) service_pool.cc:224] Handling call
0709 12:13:05.017362 (+ 22420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:05.083928 (+ 66566us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:05.083964 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:05.122269 (+ 38305us) log.cc:844] Serialized 26 byte log entry
0709 12:13:05.844296 (+722027us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:05.844354 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.844439 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:06.518206 (+673767us) raft_consensus.cc:1685] finished
0709 12:13:06.518233 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:06.518399 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":574,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":28685,"thread_start_us":650163,"threads_started":1,"wal-append.queue_time_us":781098}
I20260709 12:13:06.602761 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.603180 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.616597 12641 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:13:06.498106 16520 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.621958 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.629834 13262 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:13:06.615530 16458 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:06.603502 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.610092 12640 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:06.610214 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.637619 12563 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20260709 12:13:06.639017 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:06.612008 12642 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:13:06.608585 13257 rpcz_store.cc:269] 0709 12:13:04.702565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.702751 (+   186us) service_pool.cc:224] Handling call
0709 12:13:06.583037 (+1880286us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:06.583201 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131070848}
I20260709 12:13:06.641386 12572 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:06.615326 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:06.597441 16394 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.597259 16529 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:06.869406 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.863204 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 470) took 1020 ms. Trace:
I20260709 12:13:06.877138 13254 rpcz_store.cc:276] 0709 12:13:05.843010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.843149 (+   139us) service_pool.cc:224] Handling call
0709 12:13:06.583718 (+740569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:06.609234 (+ 25516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:06.862958 (+253724us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b540019d2d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:06.863132 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667649280}
I20260709 12:13:06.686721 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.880172 12644 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:06.686152 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.881803 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:06.676017 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79542.9us
I20260709 12:13:06.720314 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.818110 12494 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:06.736469 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:06.860749 12564 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.000s
I20260709 12:13:06.884812 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 481) took 1386 ms. Trace:
I20260709 12:13:06.885108 12564 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:06.885425 12494 rpcz_store.cc:276] 0709 12:13:05.498546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.499003 (+   457us) service_pool.cc:224] Handling call
0709 12:13:05.522127 (+ 23124us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:06.817856 (+1295729us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b5400297d20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:13:06.884419 (+ 66563us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:06.884715 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2403712}
I20260709 12:13:06.886312 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.887209 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:06.688621 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.648959 12641 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:06.888882 12640 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:13:06.700753 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.761190 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:06.753010 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:06.890230 12571 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:06.715842 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.892086 12563 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:13:06.808710 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2098) took 4782 ms. Trace:
I20260709 12:13:06.728704 12637 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:13:06.802533 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 618) took 2153 ms. Trace:
I20260709 12:13:06.894318 13256 rpcz_store.cc:276] 0709 12:13:04.648714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.648909 (+   195us) service_pool.cc:224] Handling call
0709 12:13:04.649386 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:04.798286 (+148900us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:04.798331 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:04.798377 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:04.798407 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:06.802100 (+2003693us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:06.802172 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:06.802402 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341848448}
I20260709 12:13:06.896116 16537 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:06.896368 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.861734 16235 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:13:06.799227 16500 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:06.896407 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.897882 16389 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:06.898468 12497 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:06.899524 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:06.897766 16490 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:06.827778 12567 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.705s	user 0.012s	sys 0.000s
I20260709 12:13:06.897768 16500 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:06.775331 16427 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:06.900815 12567 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:06.897792 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.821327 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:06.813297 16497 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.769s	user 0.004s	sys 0.007s
I20260709 12:13:06.892712 12568 rpcz_store.cc:276] 0709 12:13:02.026515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:02.495971 (+469456us) service_pool.cc:224] Handling call
0709 12:13:02.508730 (+ 12759us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:04.372367 (+1863637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:06.808329 (+2435962us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:06.808596 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4282999040}
I20260709 12:13:06.613322 16450 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:06.890185 12643 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:06.735906 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.920980 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.915102 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:06.928436 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2135) took 1250 ms. Trace:
I20260709 12:13:06.928699 16369 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:06.808005 16314 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:13:06.912184 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90984.6us
W20260709 12:13:06.714706 16531 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:13:06.920748 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2220) took 1324 ms. Trace:
I20260709 12:13:06.929791 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.914884 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:06.921550 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:06.682317 16444 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:06.935742 12566 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:06.860319 16504 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:06.936311 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:06.787802 16458 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:06.650002 16370 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:06.999963 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2110) took 2391 ms. Trace:
I20260709 12:13:06.988427 12568 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:07.049520 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2214) took 1461 ms. Trace:
I20260709 12:13:06.991621 16527 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:13:07.027746 12494 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:06.934291 12563 rpcz_store.cc:276] 0709 12:13:05.677732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:06.920535 (+1242803us) service_pool.cc:224] Handling call
0709 12:13:06.928340 (+  7805us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.020740 12567 rpcz_store.cc:276] 0709 12:13:05.596270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.976091 (+379821us) service_pool.cc:224] Handling call
0709 12:13:06.920675 (+944584us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:06.735456 16482 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:06.959764 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.012924 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:06.947268 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.029242 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.026064 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.027164 12564 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:13:06.930765 16490 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:13:07.118637 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 484) took 1516 ms. Trace:
I20260709 12:13:07.167572 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2216) took 1572 ms. Trace:
I20260709 12:13:07.126904 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.029826 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.125325 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:06.937459 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.115300 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.169854 12645 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:07.170688 13262 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 7: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:07.144001 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:07.050904 12495 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:07.148469 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.171476 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:07.171833 13256 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:07.170506 12496 raft_consensus.cc:606] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:07.172761 12496 raft_consensus.cc:614] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:07.147951 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:07.036533 16120 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.796s	user 0.008s	sys 0.008s
I20260709 12:13:06.939361 16500 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:06.997862 15757 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:06.937850 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.173663 12492 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:07.168032 12491 rpcz_store.cc:276] 0709 12:13:05.602063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.605324 (+  3261us) service_pool.cc:224] Handling call
0709 12:13:05.605810 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:05.610880 (+  5070us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:05.738785 (+127905us) log.cc:844] Serialized 14 byte log entry
0709 12:13:05.910892 (+172107us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:05.923762 (+ 12870us) log.cc:844] Serialized 26 byte log entry
0709 12:13:05.923953 (+   191us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:05.923994 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.924078 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:07.118383 (+1194305us) raft_consensus.cc:1685] finished
0709 12:13:07.118405 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.118561 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25272,"prepare.queue_time_us":11000,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":26425,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":6203,"thread_start_us":232280,"threads_started":2,"wal-append.queue_time_us":433077}
W20260709 12:13:07.123559 12537 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 2.391s (SENT) [suppressed 23 similar messages]
I20260709 12:13:07.168262 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.028023 16314 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:06.998919 16489 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:07.063393 12568 rpcz_store.cc:276] 0709 12:13:05.587924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:06.948029 (+1360105us) service_pool.cc:224] Handling call
0709 12:13:06.988171 (+ 40142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.049183 (+ 61012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.049448 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.031940 12569 rpcz_store.cc:276] 0709 12:13:04.608222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.608387 (+   165us) service_pool.cc:224] Handling call
0709 12:13:06.999627 (+2391240us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:06.999851 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":67846528}
I20260709 12:13:07.190666 16489 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:07.191648 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.192394 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:06.937119 16369 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.178407 12566 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:07.178853 12564 rpcz_store.cc:276] 0709 12:13:05.595164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.026486 (+1431322us) service_pool.cc:224] Handling call
0709 12:13:07.026858 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.167273 (+140415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.167507 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:07.124739 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:07.110623 16545 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.182s	user 0.006s	sys 0.000s
I20260709 12:13:07.180436 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:07.185878 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62370.6us
I20260709 12:13:06.936601 16450 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.197597 12643 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:13:06.933874 16427 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.187677 12642 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:07.203635 12642 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.204033 12645 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:13:07.121862 16476 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.192s	user 0.009s	sys 0.004s
I20260709 12:13:07.182667 16529 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:07.190063 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.189342 16490 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:07.193142 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:07.214124 16378 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.719s	user 0.011s	sys 0.001s
W20260709 12:13:06.930356 16461 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.163s	user 0.016s	sys 0.020s
I20260709 12:13:06.939556 12490 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
W20260709 12:13:07.111996 16491 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.086s	user 0.000s	sys 0.008s
I20260709 12:13:07.106930 16455 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:07.060678 16394 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:07.201493 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.201828 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:07.205116 12640 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:07.206027 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.206887 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.185861 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:07.226783 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.228083 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.228998 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:07.184046 16120 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:13:07.212235 16374 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.592s	user 0.010s	sys 0.003s
I20260709 12:13:07.215301 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.209848 12645 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:07.192440 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.209848 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.217095 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:07.219070 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:07.194175 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.197357 16369 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:07.201594 16045 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.613s	user 0.014s	sys 0.000s
I20260709 12:13:07.238396 12564 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:07.204499 12642 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.222687 12643 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:07.239125 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2217) took 1643 ms. Trace:
I20260709 12:13:07.232229 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 5.
W20260709 12:13:07.221187 16461 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.464s	user 0.023s	sys 0.021s
I20260709 12:13:07.236239 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.241873 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:07.234495 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.234494 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.243813 12563 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
I20260709 12:13:07.245723 12563 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:07.246519 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2218) took 1650 ms. Trace:
I20260709 12:13:07.246920 12563 rpcz_store.cc:276] 0709 12:13:05.595753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.244990 (+1649237us) service_pool.cc:224] Handling call
0709 12:13:07.245442 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.246206 (+   764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.246447 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.248729 12563 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:07.220914 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.239473 12564 rpcz_store.cc:276] 0709 12:13:05.595485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.237731 (+1642246us) service_pool.cc:224] Handling call
0709 12:13:07.238196 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.238815 (+   619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.239009 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.216413 16417 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:13:07.214022 16489 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.261466 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:07.285097 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.68093e+06us
W20260709 12:13:07.255460 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.375501 12494 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:13:07.264484 12645 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:07.377045 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.332417 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.249403 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2222) took 1652 ms. Trace:
I20260709 12:13:07.334905 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.263801 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.378176 12563 rpcz_store.cc:276] 0709 12:13:05.597124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.247869 (+1650745us) service_pool.cc:224] Handling call
0709 12:13:07.248242 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.249111 (+   869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.249328 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.281591 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.302716 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:07.362342 13256 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:07.380798 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78259.5us
I20260709 12:13:07.250938 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:07.348765 12497 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:07.379506 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:07.281591 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.277284 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.249452 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.383244 12643 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:07.323549 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2131) took 1647 ms. Trace:
I20260709 12:13:07.377190 12645 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.072450 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 11.606s	user 0.252s	sys 0.074s Metrics: {"bytes_written":29585,"cfile_init":1,"dirs.queue_time_us":201420,"dirs.run_cpu_time_us":622,"dirs.run_wall_time_us":38880,"drs_written":1,"lbm_read_time_us":827,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557200,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":18,"peak_mem_usage":0,"rows_written":63,"thread_start_us":149460,"threads_started":1}
W20260709 12:13:07.304421 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88131us
I20260709 12:13:07.384373 12563 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:07.333177 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:07.265700 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.383965 12572 rpcz_store.cc:276] 0709 12:13:05.676539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.322517 (+1645978us) service_pool.cc:224] Handling call
0709 12:13:07.322935 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.323171 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.323198 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.323246 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:07.323279 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.323349 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.323474 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.258680 12642 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:07.386421 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53377.8us
I20260709 12:13:07.386979 12564 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:07.375532 12571 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:07.236126 16481 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.756s	user 0.013s	sys 0.000s
W20260709 12:13:07.400653 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151257us
I20260709 12:13:07.390540 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:07.398993 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:07.404104 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.404855 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.405534 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:07.330957 16517 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.194s	user 0.002s	sys 0.006s
I20260709 12:13:07.406548 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.407501 12491 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:07.408649 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 157841us
W20260709 12:13:07.316156 16498 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.264s	user 0.004s	sys 0.006s
I20260709 12:13:07.331995 16450 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:07.277653 16465 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:07.258353 16527 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:07.411758 12569 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:07.260519 16501 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.410744 16045 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.413602 16455 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:07.414040 16417 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:07.450661 16417 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.451561 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7922 bytes on disk
I20260709 12:13:07.452984 16417 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:07.240170 16314 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.414523 16370 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:13:07.414628 16412 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:07.416118 15757 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:13:07.417223 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.312314 16490 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.258031 16374 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.495332 16374 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:07.228487 16394 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.420883 12491 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:13:07.264513 16523 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:07.496833 16523 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.497668 12491 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
W20260709 12:13:07.498230 16494 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.455s	user 0.012s	sys 0.003s
I20260709 12:13:07.497931 16523 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.499119 16523 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:07.500422 16314 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:07.500744 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2114) took 2875 ms. Trace:
I20260709 12:13:07.501187 12570 rpcz_store.cc:276] 0709 12:13:04.625458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.720845 (+ 95387us) service_pool.cc:224] Handling call
0709 12:13:04.728248 (+  7403us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:04.729102 (+   854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:04.729149 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:04.755421 (+ 26272us) log.cc:844] Serialized 590 byte log entry
0709 12:13:04.908143 (+152722us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:05.141389 (+233246us) log.cc:844] Serialized 14 byte log entry
0709 12:13:05.147070 (+  5681us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.147168 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:06.594057 (+1446889us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b18003fafc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db
0709 12:13:07.497379 (+903322us) spinlock_profiling.cc:243] Waited 60.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:07.500499 (+  3120us) raft_consensus.cc:1685] finished
0709 12:13:07.500525 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.500675 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":219484,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":3864,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":31190,"spinlock_wait_cycles":184664320,"thread_start_us":331689,"threads_started":2,"wal-append.queue_time_us":249158}
I20260709 12:13:07.503491 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.421044 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:07.504711 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.313881 16427 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:07.505620 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:07.391280 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.389649 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2143) took 1261 ms. Trace:
I20260709 12:13:07.506491 12571 rpcz_store.cc:276] 0709 12:13:06.127964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:06.689806 (+561842us) service_pool.cc:224] Handling call
0709 12:13:07.389556 (+699750us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.507505 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:07.508208 16487 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.457s	user 0.009s	sys 0.004s
I20260709 12:13:07.509198 12571 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:13:07.392799 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2133) took 1715 ms. Trace:
I20260709 12:13:07.427400 16521 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:13:07.435221 12642 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:07.510039 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1916) took 3031 ms. Trace:
I20260709 12:13:07.510274 16521 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:07.510715 16521 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.510885 16411 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:07.511361 16411 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.511837 16521 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.512559 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2140) took 1792 ms. Trace:
I20260709 12:13:07.390208 16120 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:07.512931 16235 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:07.439429 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:07.387681 16500 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.410531 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.441226 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2132) took 1764 ms. Trace:
I20260709 12:13:07.407330 16461 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.421022 16489 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:07.448179 16455 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.427925 16537 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:07.513723 12489 raft_consensus.cc:606] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:07.516948 12569 rpcz_store.cc:276] 0709 12:13:05.676883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.387608 (+1710725us) service_pool.cc:224] Handling call
0709 12:13:07.411541 (+ 23933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.440819 (+ 29278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.441141 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.519117 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.519402 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2000) took 4072 ms. Trace:
I20260709 12:13:07.513252 16521 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:07.460948 12496 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:07.223757 16378 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.520260 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:07.530699 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 140215us
W20260709 12:13:07.532578 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122279us
I20260709 12:13:07.533459 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.469588 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.017s	user 0.002s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":347,"lbm_reads_lt_1ms":4}
I20260709 12:13:07.406895 16481 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.458420 16444 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:07.475838 12497 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:07.534780 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:07.477530 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.539000 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:07.539471 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:07.540132 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.541266 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2225) took 1759 ms. Trace:
I20260709 12:13:07.541831 12566 rpcz_store.cc:276] 0709 12:13:05.781843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:06.594690 (+812847us) service_pool.cc:224] Handling call
0709 12:13:07.540997 (+946307us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54002477d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:07.541174 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":736879360}
I20260709 12:13:07.482743 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.490938 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 482) took 1923 ms. Trace:
I20260709 12:13:07.543808 12491 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:13:07.543376 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2144) took 1412 ms. Trace:
W20260709 12:13:07.491379 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2215) took 1903 ms (client timeout 2362 ms). Trace:
I20260709 12:13:07.492352 16412 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:13:07.404382 12645 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.512261 16411 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.440864 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:07.571822 12640 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:07.518294 12571 rpcz_store.cc:276] 0709 12:13:05.719856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.508417 (+1788561us) service_pool.cc:224] Handling call
0709 12:13:07.508778 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.509596 (+   818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.512501 (+  2905us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.572494 12570 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
W20260709 12:13:07.515266 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133318us
I20260709 12:13:07.510568 12488 rpcz_store.cc:276] 0709 12:13:04.478596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:04.478778 (+   182us) service_pool.cc:224] Handling call
0709 12:13:04.479296 (+   518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:04.479803 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:04.479839 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:04.480559 (+   720us) log.cc:844] Serialized 26 byte log entry
0709 12:13:04.830337 (+349778us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:04.830402 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.830533 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:07.507535 (+2677002us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:07.509677 (+  2142us) raft_consensus.cc:1685] finished
0709 12:13:07.509719 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.509964 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1964,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":127,"spinlock_wait_cycles":72182016,"thread_start_us":349198,"threads_started":1,"wal-append.queue_time_us":569589}
I20260709 12:13:07.573208 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2226) took 1790 ms. Trace:
I20260709 12:13:07.509770 12572 rpcz_store.cc:276] 0709 12:13:05.677146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.388977 (+1711831us) service_pool.cc:224] Handling call
0709 12:13:07.392127 (+  3150us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.392394 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.392432 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.392493 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:07.392536 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.392607 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.392730 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.525164 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:07.573483 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:07.573594 12570 rpcz_store.cc:276] 0709 12:13:05.782543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.518187 (+1735644us) service_pool.cc:224] Handling call
0709 12:13:07.572313 (+ 54126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.572958 (+   645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.573154 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.460000 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:07.574295 12563 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:07.574926 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115003us
I20260709 12:13:07.513391 16120 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:07.478636 16418 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:07.544669 12564 rpcz_store.cc:276] 0709 12:13:06.130885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.542411 (+1411526us) service_pool.cc:224] Handling call
0709 12:13:07.542776 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.542996 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.543028 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.543081 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:07.543124 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.543185 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.543305 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:07.544889 12565 rpcz_store.cc:269] 0709 12:13:05.592634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.600895 (+  8261us) service_pool.cc:224] Handling call
0709 12:13:07.491158 (+1890263us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b54001d6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:07.491311 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141608704}
I20260709 12:13:07.543689 12568 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:07.531524 16383 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.552896 12645 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:07.552026 13259 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:07.569854 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.519881 12638 rpcz_store.cc:276] 0709 12:13:03.447229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:03.768521 (+321292us) service_pool.cc:224] Handling call
0709 12:13:04.387178 (+618657us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:04.400574 (+ 13396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:04.400607 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:04.411351 (+ 10744us) log.cc:844] Serialized 26 byte log entry
0709 12:13:04.748819 (+337468us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:04.748891 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:04.749013 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:07.121077 (+2372064us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:07.504950 (+383873us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b24003e3738. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:13:07.519195 (+ 14245us) raft_consensus.cc:1685] finished
0709 12:13:07.519216 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.519333 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":97,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":72993,"spinlock_wait_cycles":30660224,"thread_start_us":326550,"threads_started":1,"wal-append.queue_time_us":419035}
W20260709 12:13:07.525903 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:07.592445 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1922961980001219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1653s for a response
I20260709 12:13:07.539533 16427 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.552350 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:07.613422 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70145.9us
I20260709 12:13:07.463841 16370 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:07.534945 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.616025 16370 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:13:07.616572 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81734us
I20260709 12:13:07.618778 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.531973 16455 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:07.578711 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:07.622025 13262 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:07.622890 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.578790 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.524585 16537 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.543526 12493 rpcz_store.cc:276] 0709 12:13:05.567999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:05.568850 (+   851us) service_pool.cc:224] Handling call
0709 12:13:05.589935 (+ 21085us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:05.590313 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:05.590341 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:05.595914 (+  5573us) log.cc:844] Serialized 26 byte log entry
0709 12:13:05.776719 (+180805us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:05.776780 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:05.776885 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:07.330124 (+1553239us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:07.490401 (+160277us) raft_consensus.cc:1685] finished
0709 12:13:07.490438 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.490821 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8891,"raft.queue_time_us":847,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":2839,"spinlock_wait_cycles":9688448,"thread_start_us":164493,"threads_started":1,"wal-append.queue_time_us":326037}
I20260709 12:13:07.623545 16537 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:07.624285 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.625027 12493 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:07.626350 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.513383 16235 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:07.627869 16235 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.551967 12489 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:07.630901 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.585222 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:07.537926 16444 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.634516 16460 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:07.634912 16444 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:07.635426 12571 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:07.635921 16383 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:07.588750 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.593047 12488 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:07.557788 16374 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:07.543066 16463 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:07.513820 16480 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:13:07.582432 16500 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:07.609170 13260 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:13:07.590320 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.587272 13258 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:07.587678 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48814.2us
I20260709 12:13:07.652482 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:07.637358 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2148) took 1394 ms. Trace:
I20260709 12:13:07.636135 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2147) took 1406 ms. Trace:
I20260709 12:13:07.639654 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:07.588270 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.586395 16411 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:07.663273 12571 rpcz_store.cc:276] 0709 12:13:06.230130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.594247 (+1364117us) service_pool.cc:224] Handling call
0709 12:13:07.635260 (+ 41013us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.635879 (+   619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.636075 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.663107 12570 rpcz_store.cc:276] 0709 12:13:06.242854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.597318 (+1354464us) service_pool.cc:224] Handling call
0709 12:13:07.636850 (+ 39532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.637049 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.637073 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.637114 (+    41us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:07.637156 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.637203 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.637306 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.649905 16412 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:13:07.635113 16460 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.649379 16110 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 12:13:07.667773 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.668123 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:07.724542 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56876.1us
I20260709 12:13:07.724814 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.649515 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:07.671270 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.747725 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.662460 16419 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:07.753525 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.762867 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.650842 16427 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.101s
I20260709 12:13:07.666700 16045 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:07.650126 16503 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:07.716534 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:07.659209 16374 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.632750 16482 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:07.649051 16417 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:07.815485 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2223) took 2218 ms. Trace:
I20260709 12:13:07.816093 12567 rpcz_store.cc:276] 0709 12:13:05.597433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.276851 (+1679418us) service_pool.cc:224] Handling call
0709 12:13:07.295825 (+ 18974us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.299135 (+  3310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.299163 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:07.306535 (+  7372us) log.cc:844] Serialized 26 byte log entry
0709 12:13:07.546008 (+239473us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:07.546055 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.546119 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:07.815213 (+269094us) raft_consensus.cc:1685] finished
0709 12:13:07.815239 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.815418 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7651,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":3935,"thread_start_us":162421,"threads_started":1,"wal-append.queue_time_us":413489}
I20260709 12:13:07.656805 16463 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.726011 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:07.649682 16461 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.670961 13262 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:07.638731 16481 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:07.723194 16460 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:07.835106 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 164098us
W20260709 12:13:07.840382 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.772226 13259 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:07.742815 16450 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:07.660543 16383 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.787035 13253 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:07.787931 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.852377 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:07.798911 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:07.854557 13258 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:07.674417 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.798511 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2151) took 1351 ms. Trace:
I20260709 12:13:07.776932 16045 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.796389 16374 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:07.805593 16482 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.647938 16120 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.761986 16419 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:07.768266 16551 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:07.826035 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.899833 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:07.847158 16480 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:07.833513 13257 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:13:07.909111 13257 pending_rounds.cc:85] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 7
W20260709 12:13:07.887051 16369 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.608s	user 0.013s	sys 0.000s
I20260709 12:13:07.832186 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.846290 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:07.862138 12564 rpcz_store.cc:276] 0709 12:13:06.447370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.797550 (+1350180us) service_pool.cc:224] Handling call
0709 12:13:07.797958 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.798169 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.798192 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.798242 (+    50us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:07.798288 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.798341 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:07.798449 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.917701 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.836357 16451 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
W20260709 12:13:07.870730 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 144814us
I20260709 12:13:07.796543 16552 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:07.825585 16463 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:07.930265 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:07.829756 16527 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:13:07.831552 16378 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:07.831339 12563 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:07.933828 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2233) took 1036 ms. Trace:
I20260709 12:13:07.934239 12563 rpcz_store.cc:276] 0709 12:13:06.897179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.830664 (+933485us) service_pool.cc:224] Handling call
0709 12:13:07.831063 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.933448 (+102385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.933757 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:07.909489 13257 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:13:07.834965 13254 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:13:07.874285 16045 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:07.887567 16427 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:07.917644 12638 raft_consensus.cc:2376] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10. Re-sending same reply.
W20260709 12:13:07.921219 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75051.9us
I20260709 12:13:07.914254 16503 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:07.922521 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 285424us
I20260709 12:13:07.922820 12564 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:07.925530 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:07.830950 16465 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:07.951112 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:07.927130 16481 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:07.935281 13257 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:07.931886 16552 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.915273 16551 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:07.945107 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2234) took 1047 ms. Trace:
I20260709 12:13:07.946494 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:07.960049 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:07.867297 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.897770 16450 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:13:07.921582 12497 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:07.931165 12642 raft_consensus.cc:2376] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 5. Re-sending same reply.
I20260709 12:13:07.954056 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:07.932807 16501 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:07.958506 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:07.961845 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.946584 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.962796 12564 rpcz_store.cc:276] 0709 12:13:06.897485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.922251 (+1024766us) service_pool.cc:224] Handling call
0709 12:13:07.922640 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.944756 (+ 22116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.945029 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:07.971698 12536 connection.cc:256] RPC call timeout handler was delayed by -3.7361088059999474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
I20260709 12:13:07.972065 12488 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:07.941125 16427 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.959626 16294 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:07.972105 12536 connection.cc:256] RPC call timeout handler was delayed by -3.7064088059998994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
I20260709 12:13:07.969784 12638 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12. Re-sending same reply.
I20260709 12:13:07.957760 16529 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:07.984859 12536 connection.cc:256] RPC call timeout handler was delayed by -3.6785088059998543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
I20260709 12:13:07.921051 16480 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:07.967939 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.953985 16521 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:07.965354 16552 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:07.988878 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:07.991007 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:07.957756 15757 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:07.963569 16527 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:13:07.985481 12536 connection.cc:256] RPC call timeout handler was delayed by -3.6623088059998281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
W20260709 12:13:07.996527 12536 connection.cc:256] RPC call timeout handler was delayed by -3.4067088059998696s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1896s for a response
I20260709 12:13:07.973261 16314 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:07.988509 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:07.969036 16501 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:07.981081 16503 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:07.925489 16369 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.984567 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:07.997128 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3005088059999252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
I20260709 12:13:07.966833 16558 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
W20260709 12:13:08.024966 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2789088059998903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20379999999999998s for a response
W20260709 12:13:08.025524 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2195088059997943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
I20260709 12:13:07.965850 16335 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:13:08.029040 12639 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 10. Ops: []
I20260709 12:13:07.984670 16370 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:07.998889 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.002010 16419 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.002645 16378 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Committing config change with OpId 8.7: config changed from index 2 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:07.968079 12641 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:07.984489 16110 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:13:07.954483 16455 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:08.025923 12536 connection.cc:256] RPC call timeout handler was delayed by -3.1619088059999285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
I20260709 12:13:08.024029 16521 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.004985 16314 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:07.972843 16563 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:13:08.036034 12638 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:07.989231 16500 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:07.989514 16529 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.007436 16527 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:07.988724 16294 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:08.043937 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6283010219999596s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
W20260709 12:13:08.084555 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4141010219998407s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
W20260709 12:13:08.085157 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1161088059998292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.333s for a response
W20260709 12:13:08.085551 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4889010219999363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.292s for a response
W20260709 12:13:08.085951 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3349088059999303s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.41980000000000006s for a response
W20260709 12:13:08.086313 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3053010219998669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3124s for a response
W20260709 12:13:08.086669 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3053010219998669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3124s for a response
W20260709 12:13:07.991748 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 406563us
I20260709 12:13:08.088971 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:08.050606 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75189.6us
I20260709 12:13:08.037042 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2152) took 1571 ms. Trace:
I20260709 12:13:08.091251 12567 rpcz_store.cc:276] 0709 12:13:06.465440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.817961 (+1352521us) service_pool.cc:224] Handling call
0709 12:13:07.824871 (+  6910us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.825101 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:13:07.954657 (+129556us) log.cc:844] Serialized 14 byte log entry
0709 12:13:08.036611 (+ 81954us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:08.036682 (+    71us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:13:08.036741 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.036818 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.036974 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":246284,"raft.queue_time_us":105,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":6824,"thread_start_us":210672,"threads_started":2}
I20260709 12:13:08.055563 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.056519 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.090683 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:08.056890 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2153) took 1559 ms. Trace:
I20260709 12:13:08.094166 12568 rpcz_store.cc:276] 0709 12:13:06.497418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.828001 (+1330583us) service_pool.cc:224] Handling call
0709 12:13:07.828379 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.828635 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:08.004634 (+175999us) log.cc:844] Serialized 14 byte log entry
0709 12:13:08.056315 (+ 51681us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:08.056407 (+    92us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:08.056488 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.056592 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.056797 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15950,"raft.queue_time_us":17727,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":5860,"thread_start_us":210800,"threads_started":2}
I20260709 12:13:08.060771 16521 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.096046 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.058301 16314 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:08.060184 12644 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:08.027191 16417 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:08.030994 16566 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:08.067240 12565 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:08.071617 13260 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:08.081911 16500 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.028594 16503 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:08.101980 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:08.200052 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 212496us
W20260709 12:13:08.200438 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.103395 12645 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:13:08.108208 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:08.065855 16561 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:08.068778 16383 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.203516 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.097932 16481 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:08.186633 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:08.186740 16564 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:13:08.025895 16427 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.187805 12567 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:08.149883 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:08.206113 12565 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:08.218348 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:08.220537 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70779.9us
I20260709 12:13:08.222386 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.223316 12641 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 12:13:08.210179 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 374885us
I20260709 12:13:08.097550 16521 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.226133 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:13:08.226344 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.107144 16557 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:13:08.228080 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.189278 16523 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:08.033797 16551 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:08.234570 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 253873us
I20260709 12:13:08.234827 12642 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:08.227455 16568 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.247433 16568 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.249387 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.249684 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2224) took 2468 ms. Trace:
I20260709 12:13:08.250283 12641 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:13:08.217247 16567 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260709 12:13:08.218616 16489 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.686s	user 0.012s	sys 0.000s
I20260709 12:13:08.204676 16561 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:08.213976 16461 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:08.243129 16370 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.244249 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.066047 16501 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:08.236927 16555 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.297s	user 0.000s	sys 0.002s
I20260709 12:13:08.245865 16503 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.204030 16451 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:13:08.250059 12569 rpcz_store.cc:276] 0709 12:13:05.781059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.517871 (+1736812us) service_pool.cc:224] Handling call
0709 12:13:07.553008 (+ 35137us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.582939 (+ 29931us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:07.582972 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:07.595855 (+ 12883us) log.cc:844] Serialized 26 byte log entry
0709 12:13:07.877057 (+281202us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:07.877117 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.877230 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:08.249436 (+372206us) raft_consensus.cc:1685] finished
0709 12:13:08.249459 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.249603 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23330,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":7239,"thread_start_us":198486,"threads_started":1,"wal-append.queue_time_us":305439}
I20260709 12:13:08.123572 16419 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260709 12:13:08.256345 12642 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:08.257187 12493 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:13:08.258081 12645 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:08.289027 12645 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:13:08.289439 12645 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:08.228133 16360 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:13:08.261868 12490 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:08.262197 12567 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
W20260709 12:13:08.236810 16553 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
I20260709 12:13:08.263303 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.271492 16569 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:08.253959 16481 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 12:13:08.232142 16110 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:08.228449 16443 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:08.276430 16427 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:08.291337 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2039) took 1092 ms. Trace:
I20260709 12:13:08.371748 12645 rpcz_store.cc:276] 0709 12:13:07.199376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.264039 (+ 64663us) service_pool.cc:224] Handling call
0709 12:13:07.264344 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.290496 (+1026152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.291272 (+   776us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:08.374010 12645 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:13:08.296867 16501 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:08.273073 16582 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:08.302773 12567 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:13:08.376675 12567 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 12:13:08.307579 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:08.318024 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92049.2us
I20260709 12:13:08.246398 16455 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.379678 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:08.370368 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.378342 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:08.274256 16417 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.385712 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.387058 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.374441 12645 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 7
I20260709 12:13:08.388912 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2387) took 1321 ms. Trace:
I20260709 12:13:08.389329 12645 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:13:08.389986 12645 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:08.390666 12497 rpcz_store.cc:276] 0709 12:13:07.067626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.538709 (+471083us) service_pool.cc:224] Handling call
0709 12:13:07.571476 (+ 32767us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.966529 (+395053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.388634 (+422105us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:08.388839 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804725504}
I20260709 12:13:08.198351 16566 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.392156 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:08.396085 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:08.251819 16523 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.398211 12497 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:08.400691 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.317873 16360 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:08.398353 12492 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:08.296958 16369 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.280185 16521 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.104s
I20260709 12:13:08.399968 12495 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:08.316854 16110 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:08.414004 12638 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:08.398945 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 290795us
I20260709 12:13:08.400374 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2043) took 1166 ms. Trace:
I20260709 12:13:08.251296 16527 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.280447 16461 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.416157 12643 rpcz_store.cc:276] 0709 12:13:07.233569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.494674 (+261105us) service_pool.cc:224] Handling call
0709 12:13:07.495501 (+   827us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.496040 (+   539us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.496066 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:07.497066 (+  1000us) log.cc:844] Serialized 564 byte log entry
0709 12:13:07.737116 (+240050us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:07.737159 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:07.737279 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:08.400110 (+662831us) raft_consensus.cc:1685] finished
0709 12:13:08.400133 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.400309 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":239618,"threads_started":1,"wal-append.queue_time_us":391396}
I20260709 12:13:08.393936 16480 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.395035 16443 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:08.400337 16579 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:08.425917 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47671.6us
I20260709 12:13:08.422029 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:08.407035 16360 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:13:08.444036 16529 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.420315 16110 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:08.444154 16461 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.447794 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.451562 12645 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:08.451885 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.453176 12641 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:08.451885 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:13:08.438613 16460 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.568s	user 0.004s	sys 0.007s
I20260709 12:13:08.431259 16369 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:08.423415 16586 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:08.463544 16294 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:08.472856 16374 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.576s	user 0.009s	sys 0.000s
W20260709 12:13:08.478610 12461 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.959s (SENT)
I20260709 12:13:08.482218 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:08.478852 16459 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 12:13:08.483618 16568 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.440723 16579 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:08.485709 16580 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:13:08.487850 16579 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:08.479483 16560 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 12:13:08.490547 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.491755 12572 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:13:08.492008 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2045) took 1228 ms. Trace:
I20260709 12:13:08.492687 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:08.493235 16045 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.001s
I20260709 12:13:08.492357 12646 rpcz_store.cc:276] 0709 12:13:07.263760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.517424 (+253664us) service_pool.cc:224] Handling call
0709 12:13:07.552184 (+ 34760us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.552596 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.552624 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:07.553164 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:13:08.000237 (+447073us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:08.000283 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.000349 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:08.491759 (+491410us) raft_consensus.cc:1685] finished
0709 12:13:08.491779 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.491952 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18676,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6712,"thread_start_us":366666,"threads_started":1,"wal-append.queue_time_us":535317}
I20260709 12:13:08.493852 16045 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.494346 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:08.494828 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.494570 16314 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.495458 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.496800 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:08.498003 16444 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.862s	user 0.013s	sys 0.000s
I20260709 12:13:08.498462 16444 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.499475 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.501021 12494 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:08.501794 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.502508 13259 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:08.502907 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2389) took 1434 ms. Trace:
I20260709 12:13:08.503264 12494 rpcz_store.cc:276] 0709 12:13:07.068199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.377158 (+308959us) service_pool.cc:224] Handling call
0709 12:13:08.502757 (+1125599us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:08.502861 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2287024896}
I20260709 12:13:08.504334 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:08.504869 12494 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:08.505195 12494 raft_consensus.cc:614] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:08.505381 16523 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.498407 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.506448 12567 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:08.498878 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2046) took 1213 ms. Trace:
I20260709 12:13:08.507149 12639 rpcz_store.cc:276] 0709 12:13:07.285284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.542877 (+257593us) service_pool.cc:224] Handling call
0709 12:13:07.582681 (+ 39804us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.029488 (+446807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.498657 (+469169us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:08.498825 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1025995136}
I20260709 12:13:08.500990 13253 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:08.510756 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.511413 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:08.516088 12497 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:08.519078 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:08.519286 12645 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:08.523836 12492 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:08.525928 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:08.555743 12569 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:08.481539 16411 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.801s	user 0.011s	sys 0.001s
I20260709 12:13:08.479975 16460 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.540129 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.525710 16563 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:08.558079 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.599520 12567 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:08.577209 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.597374 16459 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.597249 16588 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:08.479251 16586 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:08.626672 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46809.5us
I20260709 12:13:08.496887 16314 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.628971 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:08.525022 16443 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:13:08.606580 16463 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.002s
I20260709 12:13:08.640702 16576 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.109s
I20260709 12:13:08.641441 16460 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.641594 12491 raft_consensus.cc:2368] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:08.641865 16581 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:08.646593 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:08.646315 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.647099 12646 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:08.652179 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:08.510506 16045 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.509989 16374 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:08.651234 16418 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.058s	user 0.013s	sys 0.000s
I20260709 12:13:08.603343 16579 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:13:08.520710 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.726850 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:08.669862 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2377) took 1620 ms (client timeout 2118 ms). Trace:
W20260709 12:13:08.654214 16537 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.030s	user 0.010s	sys 0.008s
W20260709 12:13:08.690719 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61877us
I20260709 12:13:08.729244 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:08.698966 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.611856 16459 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.721359 12567 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:08.732775 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.699795 12569 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
W20260709 12:13:08.727990 12496 rpcz_store.cc:269] 0709 12:13:07.049707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.234241 (+184534us) service_pool.cc:224] Handling call
0709 12:13:08.669595 (+1435354us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:08.669787 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471368064}
I20260709 12:13:08.734541 12646 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:08.737073 12637 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:13:08.695889 16314 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:08.737295 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.699954 16411 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:08.738950 16411 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.741750 16526 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.742406 16526 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.743472 16374 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.733755 16419 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:13:08.697715 16463 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.738386 12641 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:13:08.748083 13261 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:08.751184 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.752069 12638 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:08.752110 12641 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:08.755291 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.759846 12646 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
W20260709 12:13:08.765309 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 315403us
I20260709 12:13:08.767480 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:08.788516 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:08.803103 16412 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:08.818228 16412 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:08.819286 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:08.819723 16463 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.819150 16045 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:08.728588 16369 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:13:08.761298 16314 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:13:08.823765 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94625.5us
I20260709 12:13:08.826313 16417 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:08.737159 16537 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.837167 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.839444 12569 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:08.841215 16594 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:08.843778 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 587) took 1813 ms. Trace:
I20260709 12:13:08.844241 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.845158 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 487) took 1257 ms. Trace:
I20260709 12:13:08.845664 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.845564 12493 rpcz_store.cc:276] 0709 12:13:07.587273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.626019 (+ 38746us) service_pool.cc:224] Handling call
0709 12:13:08.844944 (+1218925us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:08.845071 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":746327936}
I20260709 12:13:08.846783 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.848060 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.848783 12493 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:08.725530 16586 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.804035 16566 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.823135 16411 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.850667 16586 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:08.851055 16446 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.852023 16394 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.852160 16446 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:08.849812 16529 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:08.846786 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.854005 12492 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 6. Current term is 7. Ops: []
W20260709 12:13:08.844439 16590 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:13:08.844262 12563 rpcz_store.cc:276] 0709 12:13:07.030868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.940395 (+909527us) service_pool.cc:224] Handling call
0709 12:13:07.964803 (+ 24408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.965090 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:08.111622 (+146532us) log.cc:844] Serialized 14 byte log entry
0709 12:13:08.843129 (+731507us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b18004a2240. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:13:08.843373 (+   244us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:08.843445 (+    72us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:08.843488 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.843575 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.843706 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6019,"prepare.queue_time_us":262787,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":9551,"raft.queue_time_us":9359,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":202836992,"thread_start_us":283842,"threads_started":2,"wal-append.queue_time_us":285558}
I20260709 12:13:08.803859 16418 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:13:08.899840 16418 consensus_queue.cc:1243] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:13:08.903395 16537 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.904675 13256 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 11. Ops: []
I20260709 12:13:08.905781 12646 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:08.906458 16552 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.908s	user 0.012s	sys 0.001s
I20260709 12:13:08.906956 16552 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.907269 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.845808 16585 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.910063 12568 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:08.827987 16419 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 12:13:08.892510 16411 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.897189 16110 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.911578 16411 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.914518 16552 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:08.914968 16378 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.000s
I20260709 12:13:08.892594 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2145) took 2756 ms. Trace:
I20260709 12:13:08.898622 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1928) took 1092 ms. Trace:
I20260709 12:13:08.821290 16360 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.909425 16585 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.916756 12493 rpcz_store.cc:276] 0709 12:13:07.806406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.849284 (+1042878us) service_pool.cc:224] Handling call
0709 12:13:08.849569 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.898178 (+ 48609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:08.898216 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:08.898273 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:08.898307 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.898385 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.898549 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8946,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":108}
I20260709 12:13:08.922439 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.835098 16523 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
W20260709 12:13:08.916399 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97187.8us
I20260709 12:13:08.843837 16314 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:13:08.917959 12497 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:08.909420 12646 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 12:13:08.926285 12646 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 7
I20260709 12:13:08.926671 12646 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 12:13:08.916620 12566 rpcz_store.cc:276] 0709 12:13:06.136501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.542776 (+1406275us) service_pool.cc:224] Handling call
0709 12:13:07.579675 (+ 36899us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:07.580220 (+   545us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:08.248670 (+668450us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b18002fc0e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:08.428960 (+180290us) log.cc:844] Serialized 14 byte log entry
0709 12:13:08.668981 (+240021us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:08.699381 (+ 30400us) log.cc:844] Serialized 26 byte log entry
0709 12:13:08.699558 (+   177us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:08.699604 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:08.699698 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:08.854600 (+154902us) raft_consensus.cc:1685] finished
0709 12:13:08.892281 (+ 37681us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:08.892506 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27833,"prepare.queue_time_us":448847,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":29282,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":8482,"spinlock_wait_cycles":49705216,"thread_start_us":398702,"threads_started":2,"wal-append.queue_time_us":371382}
I20260709 12:13:08.923619 12493 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:08.928859 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:08.929556 12493 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:08.926993 12646 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:08.929148 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:13:08.933683 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111593us
I20260709 12:13:08.934729 12496 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 6. Ops: []
I20260709 12:13:08.911588 16419 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:13:08.938386 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:08.939244 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:08.940368 16566 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:08.942116 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:08.944630 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2394) took 1875 ms (client timeout 2491 ms). Trace:
I20260709 12:13:08.945061 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.945878 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:08.946642 16598 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:08.936686 16419 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:08.947592 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:08.947981 16360 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:08.948549 12645 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:08.948482 16419 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:08.957993 12645 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
W20260709 12:13:08.945097 12490 rpcz_store.cc:269] 0709 12:13:07.069617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.241535 (+171918us) service_pool.cc:224] Handling call
0709 12:13:08.944474 (+1702939us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:08.944572 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2027839872}
I20260709 12:13:08.959379 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:08.960130 12645 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:08.958130 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.962453 13259 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:08.964671 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:08.966141 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.967577 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:08.968140 12490 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:08.968923 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:08.969977 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.971463 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.972455 16598 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:08.974304 16427 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:08.975173 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.974984 16427 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:08.977562 16427 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.978132 16585 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:08.979960 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.978147 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:08.981204 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:08.974179 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:08.982440 13253 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:08.983409 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:08.981810 16598 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:08.984306 16598 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:08.984930 16598 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:08.986260 16411 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.986936 16419 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:08.987691 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:08.988373 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:08.934825 16110 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:08.993896 16594 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:08.998683 16501 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.004s
I20260709 12:13:08.998875 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:08.999822 12642 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:09.000058 16394 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:08.943979 16566 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:09.006274 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:08.937100 16523 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:13:09.008279 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:09.015321 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:09.015651 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:09.016763 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:09.028008 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:09.025529 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:09.030849 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:09.031230 12493 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:09.032832 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:09.033206 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:09.034237 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:09.032419 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:09.036019 12493 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9. Re-sending same reply.
I20260709 12:13:09.036060 16443 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:09.034456 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:09.037436 12490 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:09.037554 12493 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:09.039856 12646 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:13:09.045830 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:09.048207 13253 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:09.048840 13253 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:09.038977 16501 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:09.060976 13253 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.041023 16594 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:09.038241 16369 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:09.028316 16394 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:13:09.041812 15757 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:09.068279 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:09.064054 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1933) took 1200 ms. Trace:
I20260709 12:13:09.074762 12490 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:09.074888 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 619) took 1442 ms. Trace:
I20260709 12:13:09.108584 13260 rpcz_store.cc:276] 0709 12:13:07.632216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.832169 (+199953us) service_pool.cc:224] Handling call
0709 12:13:07.832483 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.190231 (+357748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.074570 (+884339us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.074814 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":549197056}
I20260709 12:13:09.073968 12643 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:13:09.102159 13253 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:09.103727 16585 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:09.108053 15757 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:09.078327 16369 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:09.059165 16566 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:09.063259 16501 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:09.111935 12643 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:09.251442 12643 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.253036 12643 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:09.254356 16411 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.255131 16411 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.258324 16411 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:09.259114 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:09.107700 12493 rpcz_store.cc:276] 0709 12:13:07.867008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.036932 (+1169924us) service_pool.cc:224] Handling call
0709 12:13:09.037264 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:09.045963 (+  8699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.046545 (+   582us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:09.121613 16585 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.264765 16585 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:09.122195 15757 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:09.122781 16369 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:09.141691 16529 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:09.274362 16529 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:09.142637 16528 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.278493 16528 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.145439 16417 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:09.149884 16501 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:09.150514 16110 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.282475 16110 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.150956 16586 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.133s
I20260709 12:13:09.151124 16566 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:09.156987 16598 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.166488 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 483) took 1578 ms. Trace:
W20260709 12:13:09.286098 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:09.186151 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2041) took 1968 ms. Trace:
I20260709 12:13:09.287206 16598 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:09.230536 16503 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.007s	sys 0.013s
I20260709 12:13:09.187948 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2378) took 2123 ms. Trace:
W20260709 12:13:09.214921 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:09.154961 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 6. Current term is 7. Ops: []
W20260709 12:13:09.238142 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:09.241724 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:09.180580 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:09.290073 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:09.218675 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1920) took 1545 ms. Trace:
I20260709 12:13:09.291025 16383 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:09.291560 16383 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:09.291049 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:09.185606 16482 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:09.292536 16383 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:09.204394 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2022) took 2331 ms (client timeout 1959 ms). Trace:
I20260709 12:13:09.293566 16462 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:09.225441 16526 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:09.285704 16566 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:09.288499 16503 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.007s	sys 0.013s
I20260709 12:13:09.199927 16557 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:09.295377 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:09.294500 16462 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:09.295509 12645 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:09.291179 12492 rpcz_store.cc:276] 0709 12:13:07.673578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.692480 (+1018902us) service_pool.cc:224] Handling call
0709 12:13:08.853812 (+161332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.857811 (+  3999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.218387 (+360576us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54000f5ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:09.218579 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369776896}
I20260709 12:13:09.296732 16462 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.292547 16586 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.295595 16557 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:09.300622 16586 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.301035 16557 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:09.294092 12637 rpcz_store.cc:269] 0709 12:13:06.873159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:06.930492 (+ 57333us) service_pool.cc:224] Handling call
0709 12:13:09.204090 (+2273598us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:09.204278 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3601189632}
I20260709 12:13:09.302356 16557 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:09.302887 16586 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:09.286422 13255 rpcz_store.cc:276] 0709 12:13:07.587795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.814127 (+226332us) service_pool.cc:224] Handling call
0709 12:13:07.871545 (+ 57418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.915131 (+ 43586us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.920690 (+  5559us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.928133 (+  7443us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:07.930722 (+  2589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.166060 (+1235338us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.166141 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:09.166370 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47466752}
I20260709 12:13:09.291617 12495 rpcz_store.cc:276] 0709 12:13:07.064767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.465429 (+400662us) service_pool.cc:224] Handling call
0709 12:13:07.466945 (+  1516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.414970 (+948025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.187693 (+772723us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.187883 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2145232128}
I20260709 12:13:09.305085 12495 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:13:09.305292 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:09.306133 16482 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:09.306592 16482 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:09.306986 16482 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:09.308637 16314 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:09.309157 16314 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:09.309552 16314 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:09.286809 12644 rpcz_store.cc:276] 0709 12:13:07.217962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.469201 (+251239us) service_pool.cc:224] Handling call
0709 12:13:07.469614 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.107337 (+637723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.185864 (+1078527us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.186083 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1357473024}
I20260709 12:13:09.109191 12490 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:09.329232 12493 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:13:09.285395 16110 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:09.343217 16554 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.306s	user 0.007s	sys 0.002s
I20260709 12:13:09.346376 16314 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.355933 16314 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:09.356289 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:09.356599 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:09.357314 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 1926) took 1551 ms. Trace:
I20260709 12:13:09.357474 12637 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:09.357483 12640 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:09.357980 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 489) took 1770 ms. Trace:
I20260709 12:13:09.358491 12497 rpcz_store.cc:276] 0709 12:13:07.587794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.520012 (+932218us) service_pool.cc:224] Handling call
0709 12:13:08.520317 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.925905 (+405588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.357743 (+431838us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.357918 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":914051968}
I20260709 12:13:09.359061 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2230) took 3100 ms. Trace:
I20260709 12:13:09.359129 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:09.359632 12570 rpcz_store.cc:276] 0709 12:13:06.258936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.753741 (+1494805us) service_pool.cc:224] Handling call
0709 12:13:07.782268 (+ 28527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.926046 (+143778us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:07.926079 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:07.926134 (+    55us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:07.926204 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.358831 (+1432627us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54002cfbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.358881 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:09.359002 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330017536}
I20260709 12:13:09.365824 16110 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.366580 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:09.366457 16110 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.367830 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:09.368458 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:09.369585 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:09.369742 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:09.370206 12640 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:09.371186 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:09.371866 12640 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:09.371474 16561 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.372874 16561 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:09.374724 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:09.375628 16418 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:09.376053 16418 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:09.376261 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1931) took 1704 ms (client timeout 2237 ms). Trace:
I20260709 12:13:09.376914 16418 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.377859 12639 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:09.349983 16383 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:09.376649 12638 rpcz_store.cc:269] 0709 12:13:07.671926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.067428 (+395502us) service_pool.cc:224] Handling call
0709 12:13:09.376022 (+1308594us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:09.376174 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":494020864}
I20260709 12:13:09.379330 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:09.379689 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:09.380515 16383 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:09.379908 12638 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:13:09.377974 16418 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:09.381844 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:09.383036 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:09.383320 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:09.383730 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:09.357904 12496 rpcz_store.cc:276] 0709 12:13:07.805731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.735206 (+929475us) service_pool.cc:224] Handling call
0709 12:13:08.735708 (+   502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.935226 (+199518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.356954 (+421728us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.357138 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457221888}
I20260709 12:13:09.385538 16569 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:09.386014 16569 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:09.387048 16569 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.388270 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 481) took 1939 ms. Trace:
I20260709 12:13:09.388667 13256 rpcz_store.cc:276] 0709 12:13:07.448583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.448740 (+   157us) service_pool.cc:224] Handling call
0709 12:13:07.449076 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.905169 (+1456093us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:09.388000 (+482831us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:09.388203 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3346961792}
I20260709 12:13:09.390192 16370 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:09.391399 16417 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:09.391815 16417 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:09.392740 16370 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:09.393914 16370 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.395045 13260 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: [8.10-8.10]
I20260709 12:13:09.388674 16569 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:09.401916 16374 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.402938 16374 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.403515 16412 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.404934 16412 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:09.392799 16417 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:09.406157 16374 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:09.407833 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:09.410197 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:09.408818 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.175209 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1931) took 2367 ms (client timeout 1820 ms). Trace:
W20260709 12:13:10.145884 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.526360 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.528035 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.528352 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.192963 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:10.532328 16335 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:10.534029 16374 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:10.533896 16335 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:13:10.535029 16374 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.986873 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:10.538936 12566 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:10.539049 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2064) took 1182 ms. Trace:
I20260709 12:13:10.539835 12637 rpcz_store.cc:276] 0709 12:13:09.356122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.361239 (+  5117us) service_pool.cc:224] Handling call
0709 12:13:09.361485 (+   246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:10.537787 (+1176302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:10.537858 (+    71us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:10.537950 (+    92us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:10.538010 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:10.538204 (+   194us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:10.538551 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:09.567674 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:10.166095 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.543140 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.160840 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.543679 12491 rpcz_store.cc:269] 0709 12:13:07.807423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.899507 (+ 92084us) service_pool.cc:224] Handling call
0709 12:13:10.174838 (+2275331us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:10.175076 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633536256}
W20260709 12:13:10.545269 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.545933 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:10.546070 12570 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 11
W20260709 12:13:10.546725 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.185456 16369 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.914s	user 0.009s	sys 0.004s
I20260709 12:13:10.548344 16369 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:10.145920 16378 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:10.545957 16503 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:10.550448 16378 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:10.552207 16418 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:09.435153 16110 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:09.409379 16411 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:09.449295 16501 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:10.170368 16523 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:09.351656 16529 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:10.480758 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 488) took 2892 ms (client timeout 1796 ms). Trace:
I20260709 12:13:10.615480 16620 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:10.600368 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:10.574666 12491 raft_consensus.cc:1217] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Deduplicated request from leader. Original: 6.4->[8.5-10.6]   Dedup: 8.5->[10.6-10.6]
I20260709 12:13:11.239144 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:11.239550 12491 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20260709 12:13:10.586526 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:10.604107 12495 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:13:11.257983 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:11.259218 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:10.578217 12496 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:13:10.623360 16622 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.261843 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:10.628089 16619 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:10.644425 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:11.262729 16619 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:10.580804 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:10.504802 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.267690 12563 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:10.518007 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 596) took 3452 ms. Trace:
I20260709 12:13:11.271255 12564 rpcz_store.cc:276] 0709 12:13:07.065128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.973450 (+908322us) service_pool.cc:224] Handling call
0709 12:13:07.973786 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:08.190870 (+217084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:08.190911 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:08.206940 (+ 16029us) log.cc:844] Serialized 26 byte log entry
0709 12:13:08.512536 (+305596us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:08.512576 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:10.509014 (+1996438us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:10.509138 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:10.509289 (+   151us) raft_consensus.cc:1685] finished
0709 12:13:10.509306 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:10.517770 (+  8464us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":494,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":15253,"spinlock_wait_cycles":498281856,"thread_start_us":259554,"threads_started":1,"wal-append.queue_time_us":423793,"wal-append.run_cpu_time_us":10745,"wal-append.run_wall_time_us":1795640}
I20260709 12:13:10.626081 12566 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:11.275882 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 608) took 1358 ms (client timeout 1533 ms). Trace:
W20260709 12:13:11.276309 12566 rpcz_store.cc:269] 0709 12:13:09.917780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.986314 (+ 68534us) service_pool.cc:224] Handling call
0709 12:13:11.275777 (+1289463us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:10.511735 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2236) took 3094 ms. Trace:
I20260709 12:13:11.278414 12568 rpcz_store.cc:276] 0709 12:13:07.417575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.118536 (+700961us) service_pool.cc:224] Handling call
0709 12:13:08.892644 (+774108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:08.910592 (+ 17948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:10.511117 (+1600525us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b5400044f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:10.511463 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1779297152}
I20260709 12:13:11.279928 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.280992 12568 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:10.500567 12569 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: [9.7-11.8]
I20260709 12:13:11.283529 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.284174 12569 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:10.502141 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:10.618180 12570 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:11.287421 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 605) took 1829 ms. Trace:
I20260709 12:13:11.287866 12570 rpcz_store.cc:276] 0709 12:13:09.458072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.567175 (+109103us) service_pool.cc:224] Handling call
0709 12:13:11.287301 (+1720126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:11.289083 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.289898 12570 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:10.623155 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.295305 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:10.606339 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2053) took 2171 ms. Trace:
I20260709 12:13:11.296778 12639 rpcz_store.cc:276] 0709 12:13:08.435214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.508152 (+ 72938us) service_pool.cc:224] Handling call
0709 12:13:09.377677 (+869525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:09.378404 (+   727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:10.605795 (+1227391us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b540029c550. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:10.606093 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1998873344}
I20260709 12:13:11.297302 16411 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:10.563683 12641 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.202s	user 0.007s	sys 0.007s
W20260709 12:13:10.634140 12643 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.380s	user 0.008s	sys 0.028s
W20260709 12:13:10.852829 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 204939us
I20260709 12:13:11.301477 12643 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:13:10.866415 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:11.302336 16450 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:13:11.302407 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2057) took 2578 ms. Trace:
I20260709 12:13:11.302865 12643 rpcz_store.cc:276] 0709 12:13:08.724212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.916719 (+192507us) service_pool.cc:224] Handling call
0709 12:13:08.917095 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:09.073846 (+156751us) spinlock_profiling.cc:243] Waited 90.4 ms on lock 0x7b54001bc720. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:13:11.301976 (+2228130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:11.302317 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207823104}
I20260709 12:13:11.304684 16450 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:10.877564 12460 connection.cc:256] RPC call timeout handler was delayed by -1.1715028610001355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
I20260709 12:13:10.888567 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 1938) took 3211 ms. Trace:
W20260709 12:13:10.908051 12638 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.526s	user 0.021s	sys 0.015s
I20260709 12:13:10.909296 16629 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:10.934258 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:11.307106 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:13:11.306891 12642 rpcz_store.cc:276] 0709 12:13:07.676654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.287531 (+610877us) service_pool.cc:224] Handling call
0709 12:13:08.287902 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:09.000400 (+712498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:10.887733 (+1887333us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:10.888299 (+   566us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1636241280}
I20260709 12:13:10.971765 12492 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:11.308257 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2066) took 1945 ms. Trace:
I20260709 12:13:11.308632 12638 rpcz_store.cc:276] 0709 12:13:09.362502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.379072 (+ 16570us) service_pool.cc:224] Handling call
0709 12:13:11.308112 (+1929040us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:11.309801 16526 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.305450 16450 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.311242 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2399) took 2471 ms. Trace:
I20260709 12:13:11.311836 12492 rpcz_store.cc:276] 0709 12:13:08.840136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.297204 (+457068us) service_pool.cc:224] Handling call
0709 12:13:09.297598 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:11.308580 (+2010982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:11.310763 (+  2183us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:11.311115 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3848711552}
I20260709 12:13:11.312702 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:10.998113 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:11.006870 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.314167 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.314251 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:11.314558 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.315068 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.315586 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.023389 16631 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.316684 12567 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:11.316620 16631 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:11.027233 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800107620 after lost signal to thread 11310
W20260709 12:13:11.318923 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080024ade0 after lost signal to thread 11313
I20260709 12:13:11.319701 16629 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:13:11.320343 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08001bbc20 after lost signal to thread 12429
I20260709 12:13:11.320416 16629 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.028363 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.322305 12565 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:11.322669 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.323513 12567 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:11.029260 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 450) took 2091 ms. Trace:
I20260709 12:13:11.324189 12645 rpcz_store.cc:276] 0709 12:13:08.937273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.026981 (+ 89708us) service_pool.cc:224] Handling call
0709 12:13:09.027443 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:09.296278 (+268835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:11.004685 (+1708407us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:11.029002 (+ 24317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616010496}
I20260709 12:13:10.897209 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:10.655856 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:10.805749 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.330775 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.331197 12492 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:10.375918 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 475) took 3253 ms. Trace:
W20260709 12:13:10.664299 13218 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.277s	user 0.004s	sys 0.031s
W20260709 12:13:11.077725 13261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 614) took 3533 ms (client timeout 3346 ms). Trace:
W20260709 12:13:11.337052 13261 rpcz_store.cc:269] 0709 12:13:07.544074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.544514 (+   440us) service_pool.cc:224] Handling call
0709 12:13:11.077072 (+3532558us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:11.077415 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2581166336}
W20260709 12:13:10.516167 15757 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.247s	user 0.013s	sys 0.017s
I20260709 12:13:11.341085 15757 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.342645 12492 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:10.665182 16084 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.347725 16084 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:10.818634 16370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.414s	user 0.009s	sys 0.005s
W20260709 12:13:11.352098 16370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.947s	user 0.011s	sys 0.006s
I20260709 12:13:10.559206 16374 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:10.842727 16235 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:10.557860 16378 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:10.586725 16335 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.025135 16417 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:10.553139 16418 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.353379 16370 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:10.797044 16412 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.392s	user 0.005s	sys 0.028s
I20260709 12:13:10.459604 16444 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:10.678587 16459 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:10.498682 16482 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.191s	user 0.008s	sys 0.026s
I20260709 12:13:10.859419 16490 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:11.374680 16490 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.362155 12392 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 6 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:11.375906 16490 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.376106 16526 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.377158 16490 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:11.030510 16501 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:10.837541 16528 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.026088 16523 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.380981 16450 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.381251 16526 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:13:11.110432 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.382695 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080035a0c0 after lost signal to thread 12521
W20260709 12:13:11.383052 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08002902c0 after lost signal to thread 12522
I20260709 12:13:11.383697 16523 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.111166 16626 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:11.384284 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800314760 after lost signal to thread 12596
W20260709 12:13:10.870577 16566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.562s	user 0.026s	sys 0.008s
W20260709 12:13:10.979369 16561 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.606s	user 0.014s	sys 0.014s
W20260709 12:13:11.386252 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 489671us
I20260709 12:13:11.386067 16370 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:09.408347 16586 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.384838 16626 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.388447 16586 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:10.989483 16587 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.546s	user 0.014s	sys 0.009s
W20260709 12:13:11.392603 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91136.6us
I20260709 12:13:11.179508 16620 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.396239 16636 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.396799 16636 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.390723 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:11.397838 16620 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:11.178931 12489 rpcz_store.cc:269] 0709 12:13:07.587564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.630567 (+ 43003us) service_pool.cc:224] Handling call
0709 12:13:10.479881 (+2849314us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:10.480309 (+   428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2603956096}
I20260709 12:13:11.182543 16529 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.399192 16636 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.199559 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.400087 12571 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:11.207369 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.241192 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:11.402661 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.249495 12493 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:11.261461 16622 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.406118 16455 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:11.406832 16455 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:11.407637 16622 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.408295 16586 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.408726 12494 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:11.407858 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.409871 13261 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:11.411248 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:11.264065 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 1924) took 3459 ms (client timeout 1893 ms). Trace:
W20260709 12:13:10.316219 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.289s	user 0.012s	sys 0.013s
I20260709 12:13:10.647555 16621 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.390450 12570 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:11.414557 16084 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.414252 16621 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.415441 15757 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.416606 16621 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.265189 16619 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.286684 12572 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:11.417848 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.418588 16558 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.296216 16503 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.419934 16503 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.421061 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.421836 12572 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7.
I20260709 12:13:11.422190 16503 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.298357 16411 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.299850 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:13:11.423179 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2060) took 2068 ms (client timeout 1687 ms). Trace:
W20260709 12:13:11.423606 12641 rpcz_store.cc:269] 0709 12:13:09.354947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:09.355993 (+  1046us) service_pool.cc:224] Handling call
0709 12:13:11.423028 (+2067035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:11.424289 16411 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.323393 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.293668 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.426505 12565 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 11. Ops: []
I20260709 12:13:11.427253 12566 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:13:11.427999 13217 connection.cc:256] RPC call timeout handler was delayed by -1.0376428669999314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17250000000000001s for a response
I20260709 12:13:11.428524 12565 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:11.428663 16459 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:11.040445 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.429565 16459 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.332631 16546 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:11.332842 13254 rpcz_store.cc:276] 0709 12:13:07.122107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.124053 (+  1946us) service_pool.cc:224] Handling call
0709 12:13:07.124429 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:07.935452 (+811023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:10.374849 (+2439397us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:10.375549 (+   700us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633053952}
I20260709 12:13:11.430656 16546 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.432906 16459 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.333907 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:11.334082 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.335258 13260 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20260709 12:13:11.335337 13218 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.948s	user 0.006s	sys 0.031s
I20260709 12:13:11.355445 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.358517 12568 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:11.359807 12497 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:11.423439 12572 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:10.556690 16369 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.373975 16418 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.373171 16482 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.434279 13254 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:13:11.436517 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.433605 16546 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.378578 16631 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:10.691205 16551 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:13:11.384830 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08002a0340 after lost signal to thread 12597
W20260709 12:13:11.385769 16566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.078s	user 0.026s	sys 0.010s
W20260709 12:13:10.454844 16583 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.019s	user 0.011s	sys 0.013s
W20260709 12:13:11.447592 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080035d540 after lost signal to thread 12670
I20260709 12:13:11.387235 16557 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.447965 12572 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:11.448340 16557 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:11.448989 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08002b7f00 after lost signal to thread 13286
I20260709 12:13:11.439463 13256 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:13:11.451505 16557 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.390444 16626 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.452651 12490 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:11.454586 16370 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:11.385823 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 451799us
I20260709 12:13:11.193936 16632 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.455475 12568 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:11.455946 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:11.455758 16632 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.407835 16455 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.263149 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:11.458001 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:11.459903 16626 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:11.460588 16626 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.412350 16559 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:11.461751 16626 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:11.411967 12488 rpcz_store.cc:269] 0709 12:13:07.804839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.980482 (+175643us) service_pool.cc:224] Handling call
0709 12:13:11.263783 (+3283301us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:11.263983 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6080612224}
I20260709 12:13:11.418762 12570 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:11.462666 16632 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.464234 12570 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:11.465819 12570 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:13:11.467271 12570 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:11.467669 12570 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:11.468667 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:11.419279 16558 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:11.430037 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.470394 12611 connection.cc:256] RPC call timeout handler was delayed by -1.1573608819999208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3346s for a response
I20260709 12:13:11.471307 16369 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.472299 16558 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.472199 16369 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.473076 16418 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.473850 16418 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.474418 12572 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:11.474627 16369 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.475178 16369 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.475878 16369 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.476277 16418 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.476861 16418 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.477053 16503 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.477798 16411 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.477358 16418 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.478385 16411 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.480623 16418 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.482270 16369 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.444854 13254 raft_consensus.cc:606] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:11.481667 12392 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:13:11.468832 12570 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:11.379117 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72577.9us
I20260709 12:13:11.483668 12392 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:13:11.485054 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:11.439576 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.413s	user 0.017s	sys 0.013s
I20260709 12:13:11.485806 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
W20260709 12:13:11.486773 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.4713e+06us
I20260709 12:13:11.487473 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:11.488008 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:11.488379 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:11.446645 16551 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.489101 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.489837 16619 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:13:11.490592 16551 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.490749 16411 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.447049 16482 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.492240 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:11.492838 12495 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:11.493321 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.494041 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.494210 13253 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:11.439868 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:11.496022 16619 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:11.483505 12570 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:11.496579 16619 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.462263 16559 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:11.436122 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 2.0488e+06us
I20260709 12:13:11.473009 16621 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.498311 16621 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.498487 16559 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.500890 16621 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.328127 16579 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:11.416972 12571 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:11.481926 16503 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.501839 16482 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:11.502779 12571 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:11.503767 16503 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.325222 16629 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.504554 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:11.502833 16482 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:11.485611 12392 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 6 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:11.505219 13254 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:11.506610 12392 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 8, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 8 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:11.478310 13255 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 2 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:11.492000 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:11.510809 13256 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:11.547825 13254 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:11.394428 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
W20260709 12:13:11.553953 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 604) took 2184 ms (client timeout 1725 ms). Trace:
I20260709 12:13:11.545399 12568 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:11.557905 12495 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
W20260709 12:13:11.565963 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74062.9us
W20260709 12:13:11.554976 12563 rpcz_store.cc:269] 0709 12:13:09.369546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:10.503900 (+1134354us) service_pool.cc:224] Handling call
0709 12:13:11.553890 (+1049990us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:11.578949 12569 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:11.508126 12392 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:11.588682 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:11.589509 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:11.591154 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82806.4us
I20260709 12:13:11.588702 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:11.590736 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:11.595325 12489 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260709 12:13:11.597420 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109678us
I20260709 12:13:11.509930 16631 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:11.505228 16482 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:11.429239 12566 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 2 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:11.603901 12567 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:11.605927 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49370.8us
I20260709 12:13:11.584226 16369 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:11.599079 12392 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 7, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 7 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:13:11.612680 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.613772 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.615708 16463 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.616333 16463 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.617883 12496 raft_consensus.cc:2376] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 7. Re-sending same reply.
I20260709 12:13:11.618678 16626 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.619227 16626 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:11.629915 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.631944 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.632863 12570 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:11.632757 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 176969us
I20260709 12:13:11.634011 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:11.638552 12490 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:11.635223 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:11.641692 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.641954 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.640403 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.640321 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:11.511579 16619 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.644848 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.645849 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.579064 16503 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.658291 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:11.658903 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.659739 12563 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:11.666461 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.667837 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.634790 16369 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:11.674345 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:11.675001 13218 connection.cc:256] RPC call timeout handler was delayed by -1.0584102339998935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17900000000000002s for a response
I20260709 12:13:11.685472 16621 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:11.687134 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:11.689399 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:13:11.676635 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.631950 16588 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.645175 16529 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:11.714401 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72659.7us
I20260709 12:13:11.675750 12392 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 7, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 7 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:11.705117 13256 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:11.706918 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.717199 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.720875 13261 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:11.722086 13260 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:13:11.676981 16369 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:11.733227 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48274.6us
I20260709 12:13:11.732450 12567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:11.693485 16370 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:11.677853 16235 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.694072 16084 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:11.738971 12563 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:11.740342 16649 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:11.741027 16649 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:11.741448 16649 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:11.726956 12609 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.362s (SENT)
W20260709 12:13:11.756953 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 80274.8us
I20260709 12:13:11.677162 16557 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:13:11.746019 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:11.760561 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.767985 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:13:11.693722 16620 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 12:13:11.700758 16503 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.769351 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:11.738615 16551 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:11.755666 16370 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:11.771494 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113411us
I20260709 12:13:11.776643 12570 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:11.779742 16444 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:11.779956 12569 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:13:11.780361 16444 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.780520 16557 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:11.781605 16444 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.784904 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:11.785871 12639 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:11.786033 16619 consensus_queue.cc:1175] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 8 status { last_received { term: 8 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 8 index: 4 } }
I20260709 12:13:11.786870 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.787768 13259 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:11.732960 16629 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
W20260709 12:13:11.792069 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.764081 16084 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.756039 16528 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:13:11.783093 16503 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:11.799733 16551 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.795097 16632 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 12:13:11.769166 16546 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:13:11.803045 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:11.803273 16546 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:11.803720 16546 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:11.803913 12637 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:11.788619 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:11.806068 16620 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:13:11.808329 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:11.809211 16335 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:13:11.809677 16335 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:11.785866 16444 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:11.810767 16335 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.812444 16335 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:11.765390 16482 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.760610 16235 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.814316 16561 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:11.814952 16561 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:11.815404 16561 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:11.816414 16235 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.816788 16417 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:11.814416 12638 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: [9.8-11.9]
W20260709 12:13:11.821944 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144478us
I20260709 12:13:11.824910 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:11.827071 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:11.828325 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:11.804916 16546 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:11.829447 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.761758 16529 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:11.831985 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:11.837735 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.838312 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.839735 16653 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
W20260709 12:13:11.841351 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68348.6us
I20260709 12:13:11.842012 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:11.802455 16551 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.842952 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:11.746146 16588 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.849239 16084 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:11.853729 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.845986 16557 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:11.859928 16370 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:11.860653 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.860972 16557 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:11.861606 16557 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:11.862188 16558 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:11.859931 16621 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:11.865648 16235 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:13:11.866115 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.866262 16235 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:11.871485 12639 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 12:13:11.873656 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51667us
I20260709 12:13:11.887025 16631 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:11.887212 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:11.888052 16631 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:11.888541 16631 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:11.888921 16631 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:11.864298 16620 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:13:11.863943 16417 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:13:11.867823 16551 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:13:11.895627 16559 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:11.896315 16559 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:11.896731 16559 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:11.897876 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.898442 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:11.906898 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.914338 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2166) took 1282 ms. Trace:
I20260709 12:13:11.933954 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:11.933395 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52417.5us
I20260709 12:13:11.915175 12565 rpcz_store.cc:276] 0709 12:13:10.623732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:11.431420 (+807688us) service_pool.cc:224] Handling call
0709 12:13:11.431850 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:11.432179 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:11.578180 (+146001us) log.cc:844] Serialized 14 byte log entry
0709 12:13:11.647417 (+ 69237us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b1800115900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:13:11.905250 (+257833us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:11.905355 (+   105us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:11.905426 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:11.905526 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:11.905746 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":145934,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"spinlock_wait_cycles":11897728,"thread_start_us":393846,"threads_started":3}
W20260709 12:13:11.940414 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.931092 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:11.907871 16546 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:11.888006 16529 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:11.961349 16659 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:11.961930 16659 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:11.963677 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:11.964800 12637 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:11.967226 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:11.968314 16621 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:11.867352 16235 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:11.970484 16235 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:11.927173 16455 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:13:11.974754 16455 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:13:11.979779 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:11.884328 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:13:11.987587 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 300578us
I20260709 12:13:11.955184 16459 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
W20260709 12:13:11.978667 16558 consensus_queue.cc:1175] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 12 status { last_received { term: 11 index: 10 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:13:12.010213 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74997us
I20260709 12:13:12.011103 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:11.985316 16626 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:11.988315 16463 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:12.016487 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:12.017297 12637 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:12.018707 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.017800 16459 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:12.022280 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96926.1us
I20260709 12:13:12.008052 16579 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:12.028683 16619 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:12.029372 16619 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:12.031757 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115387us
I20260709 12:13:12.032639 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:12.040544 16619 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.042830 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:12.053907 12643 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:12.025911 16459 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:12.055487 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 190772us
I20260709 12:13:12.056584 16444 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:13:12.027408 16579 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:12.064445 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.069967 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:12.070137 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:12.072242 12568 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13. Re-sending same reply.
I20260709 12:13:12.075409 12569 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:12.086090 16490 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.087198 16620 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:12.087891 16482 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:12.089567 16632 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:12.091881 16632 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:12.092298 16632 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:12.059985 16459 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:12.094789 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:12.116967 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:12.122912 12610 connection.cc:256] RPC call timeout handler was delayed by -1.7730015770000591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
I20260709 12:13:12.093343 16632 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:12.130447 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1626221310000346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260709 12:13:12.131771 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:12.134385 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1131221309999546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
I20260709 12:13:12.125669 12565 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:12.135241 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:12.135452 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1077221309999459s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
I20260709 12:13:12.135972 12497 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:12.136760 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.133405 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.140548 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.130488 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.131166 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:12.147634 12667 debug-util.cc:398] Leaking SignalData structure 0x7b0800114500 after lost signal to thread 11310
I20260709 12:13:11.963300 16659 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.150066 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.159901 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2165) took 1539 ms. Trace:
W20260709 12:13:12.159001 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89127.8us
I20260709 12:13:12.160421 12566 rpcz_store.cc:276] 0709 12:13:10.620690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:11.428199 (+807509us) service_pool.cc:224] Handling call
0709 12:13:11.428568 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:11.429053 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:11.715694 (+286641us) log.cc:844] Serialized 14 byte log entry
0709 12:13:12.159219 (+443525us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:12.159295 (+    76us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:12.159366 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:12.159487 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:12.159720 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":181704,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":395062,"threads_started":2,"wal-append.queue_time_us":96880}
I20260709 12:13:12.151302 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:12.167049 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5640 after lost signal to thread 11313
W20260709 12:13:12.168825 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08003e9a00 after lost signal to thread 12429
W20260709 12:13:12.173053 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf3a0 after lost signal to thread 12521
W20260709 12:13:12.173784 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156990us
I20260709 12:13:12.175757 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.178086 16658 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:13:12.182785 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfba0 after lost signal to thread 12522
W20260709 12:13:12.183463 12667 debug-util.cc:398] Leaking SignalData structure 0x7b080029b880 after lost signal to thread 12536
I20260709 12:13:12.184650 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:12.182031 12490 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:13:12.188660 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08001f1540 after lost signal to thread 12596
W20260709 12:13:12.189975 12667 debug-util.cc:398] Leaking SignalData structure 0x7b0800142900 after lost signal to thread 12597
I20260709 12:13:12.187089 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:12.151047 16586 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:12.171434 16588 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:12.192574 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:12.198519 12667 debug-util.cc:398] Leaking SignalData structure 0x7b0800272f00 after lost signal to thread 12670
W20260709 12:13:12.211202 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:12.212481 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49705us
I20260709 12:13:12.214227 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.216130 16417 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:12.216840 16417 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:12.217797 16417 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.221990 16586 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.222699 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:12.224470 16586 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:12.225257 13255 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
W20260709 12:13:12.230600 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe440 after lost signal to thread 13286
W20260709 12:13:12.232273 12667 debug-util.cc:398] Leaking SignalData structure 0x7b0800371820 after lost signal to thread 16586
I20260709 12:13:12.239003 12488 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:13:12.252166 16459 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.254271 12496 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 6.
W20260709 12:13:12.226639 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47184.3us
I20260709 12:13:12.166711 16631 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.270274 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:12.274046 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:12.179214 16369 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:12.281494 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.268620 16662 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:12.272974 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:12.275309 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48154.2us
I20260709 12:13:12.229827 16588 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:12.279332 16561 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.222075 16632 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:12.283855 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73052.4us
I20260709 12:13:12.286010 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:12.287267 16631 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:12.312855 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:12.287904 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:12.308918 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131335us
I20260709 12:13:12.275385 16659 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:12.313771 12494 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:12.315784 12392 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 8, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:12.320384 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.321372 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.322260 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.325801 16369 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:12.333031 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.309993 16658 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:13:12.332080 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.333823 12493 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:12.329676 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.334591 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.268666 16629 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
W20260709 12:13:12.320915 16649 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.579s	user 0.013s	sys 0.000s
I20260709 12:13:12.342427 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:12.353719 13255 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:12.345197 12492 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:12.322846 16370 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:12.315403 16586 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:12.364714 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.341616 16620 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:13:12.331599 16657 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:13:12.333400 16335 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:13:12.398347 16664 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260709 12:13:12.399490 16656 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 12:13:12.401667 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.403327 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.404381 16631 raft_consensus.cc:2764] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
I20260709 12:13:12.405342 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.409106 12565 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:12.409806 16503 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:13:12.412556 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.413333 12565 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:12.414368 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:12.397608 16557 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.535s	user 0.002s	sys 0.008s
I20260709 12:13:12.418080 16557 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.428740 16557 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:12.377404 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.378991 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.432163 12568 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:12.434594 16561 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:12.437389 13258 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:12.437923 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:12.439129 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65301us
I20260709 12:13:12.440289 16369 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:13:12.194027 16579 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.384876 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.343091 16658 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.442328 16658 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:12.442405 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:12.444142 16579 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:12.385687 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.385908 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.385159 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.362046 16370 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.386525 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:12.446146 16370 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:12.394006 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:12.446522 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:12.447491 12492 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:13:12.448120 16459 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:12.439245 16503 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:12.446283 12571 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:13:12.445389 12567 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:12.446139 12564 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:12.327906 16634 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
W20260709 12:13:12.429224 16559 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.000s
I20260709 12:13:12.446043 12566 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260709 12:13:12.409423 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 182396us
W20260709 12:13:12.482597 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.457487 16503 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.491580 16503 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:12.441993 16369 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:13:12.498633 16586 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.405311 16631 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.341738 16588 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.466137 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.533831 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:12.534829 12639 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:12.374752 16649 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.484160 16620 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:12.542001 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:12.536912 12392 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 7 to 8, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:12.550359 12564 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:13:12.545814 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:12.375089 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:12.564932 13255 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:12.567517 12497 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:12.556320 16529 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:13:12.570302 12642 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:12.571619 12643 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:12.515724 16559 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.547238 16335 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:12.603124 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.023s	sys 0.034s
W20260709 12:13:12.622151 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.023s	sys 0.035s
I20260709 12:13:12.616600 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:12.622565 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 610547us
I20260709 12:13:12.626408 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:12.585916 16546 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.000s
I20260709 12:13:12.631014 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:12.642863 13255 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:13:12.640404 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:12.643553 12571 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:13:12.631147 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:12.644946 13253 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:13:12.646988 13259 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:13:12.634054 12638 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
W20260709 12:13:12.650327 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.645310 16662 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:12.654001 13260 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:13:12.666528 13260 pending_rounds.cc:85] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 8
I20260709 12:13:12.654520 12392 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 6, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:13:12.669392 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:12.656388 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 251674us
W20260709 12:13:12.661125 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:12.672827 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.663535 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:12.664290 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.674845 12646 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:12.676455 12564 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 12:13:12.678081 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80320.1us
I20260709 12:13:12.679263 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:12.675930 16546 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:13:12.681241 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55087.9us
I20260709 12:13:12.682329 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:12.684614 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.686661 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.686748 16444 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:12.686246 16662 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.687630 16444 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.666932 13260 pending_rounds.cc:107] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:13:12.688802 13260 raft_consensus.cc:2889] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:12.677023 12392 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 9, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:12.693703 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.647861 16528 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:12.696687 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:12.625236 16634 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
W20260709 12:13:12.702026 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 260783us
I20260709 12:13:12.646937 12639 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Committing config change with OpId 8.7: config changed from index 2 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:12.708758 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.676147 16586 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:12.725333 12569 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:12.728324 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.704950 16084 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:12.737684 16674 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:13:12.743891 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:12.744415 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47818.9us
W20260709 12:13:12.745302 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81151.8us
I20260709 12:13:12.745594 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:12.752007 16444 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:12.752275 16551 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:12.752929 16551 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.753607 16551 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:12.752878 12392 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 1 to 8. New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:12.754860 12392 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 9, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 9 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:13:12.756635 16675 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:13:12.758433 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.760628 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:12.761986 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 123) took 24335 ms (client timeout 15000 ms). Trace:
I20260709 12:13:12.748746 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:12.720557 16667 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.162s	user 0.002s	sys 0.000s
I20260709 12:13:12.763278 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:12.701167 16671 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.709932 16546 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:12.741290 16631 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:12.718448 16669 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:12.721400 16649 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:12.770998 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:12.784407 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102206us
I20260709 12:13:12.746790 16561 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:12.749797 16586 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.763144 16482 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:12.787180 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:12.793795 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.796542 12569 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:13:12.712324 16528 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.775101 16444 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:12.769942 16369 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.798257 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.800264 13260 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:12.798831 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:12.813191 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.793175 12392 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:12.696270 16620 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:12.864229 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:12.860999 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:12.779629 16686 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:12.817596 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129373us
I20260709 12:13:12.874776 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:12.876332 12392 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:13:12.786854 16669 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:12.780467 16084 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:12.796954 16687 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:12.865694 16619 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.806643 16444 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:12.964710 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:12.863586 16586 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:12.865417 16529 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.796677 16626 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:12.800130 16459 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.134s
I20260709 12:13:12.963059 12637 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:12.979081 12392 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 3 to 5, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 5 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:12.966325 13260 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:12.879197 16686 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.964932 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 454) took 1204 ms. Trace:
I20260709 12:13:12.941414 16687 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.984611 16687 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:12.992416 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 230223us
I20260709 12:13:12.883777 16676 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:12.817090 16558 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.151s
I20260709 12:13:12.995707 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:12.888988 16665 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:13:12.998704 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:12.999672 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:12.969769 16444 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:12.935743 16669 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:12.939106 16631 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:12.890033 16682 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:12.804436 16557 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.160s
I20260709 12:13:13.002799 12392 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:13.007691 16665 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.982385 12641 rpcz_store.cc:276] 0709 12:13:11.782295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:11.785869 (+  3574us) service_pool.cc:224] Handling call
0709 12:13:11.786558 (+   689us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:11.787040 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:11.787082 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:11.788034 (+   952us) log.cc:844] Serialized 26 byte log entry
0709 12:13:12.153181 (+365147us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:12.153248 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:12.153349 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:12.964608 (+811259us) raft_consensus.cc:1685] finished
0709 12:13:12.964635 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:12.964846 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3549,"raft.queue_time_us":2729,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":31031,"thread_start_us":364383,"threads_started":1,"wal-append.queue_time_us":571082}
I20260709 12:13:12.969257 16588 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:12.970082 16586 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:12.973853 16559 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:12.975257 16626 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:12.818747 16335 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:13.069380 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:13.015327 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 149345us
W20260709 12:13:13.027900 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 149366us
I20260709 12:13:13.081970 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:13.037199 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 224067us
I20260709 12:13:13.032307 12392 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 5 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:12.993498 16686 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:13.022415 16658 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.003s
I20260709 12:13:12.992818 16370 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.024806 16665 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:13.089342 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90722.3us
I20260709 12:13:13.093219 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:13:13.101679 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:13.090654 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.094611 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:13.105119 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:13.042544 16586 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:13.047993 16653 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:13:13.012612 16619 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.083s
I20260709 12:13:12.995877 16546 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.086927 16631 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:13.129154 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:13.130198 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160188us
W20260709 12:13:13.131789 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.086154 16626 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:13.141571 12641 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:13.143117 16528 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:13.143435 16694 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
W20260709 12:13:12.762498 12394 rpcz_store.cc:269] 0709 12:12:48.426555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:48.426916 (+   361us) service_pool.cc:224] Handling call
0709 12:13:12.761686 (+24334770us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:02.639519 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:02.794054 (+154535us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:02.794133 (+    79us) write_op.cc:435] Acquired schema lock
0709 12:13:02.794163 (+    30us) tablet.cc:662] Decoding operations
0709 12:13:02.860140 (+ 65977us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:02.860222 (+    82us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:02.860256 (+    34us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:13:02.861445 (+  1189us) tablet.cc:701] Row locks acquired
0709 12:13:02.861498 (+    53us) write_op.cc:260] PREPARE: finished
0709 12:13:02.873448 (+ 11950us) write_op.cc:270] Start()
0709 12:13:02.873656 (+   208us) write_op.cc:276] Timestamp: P: 1783599182862090 usec, L: 0
0709 12:13:02.873693 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 12:13:02.874486 (+   793us) log.cc:844] Serialized 5642 byte log entry
0709 12:13:04.744709 (+1870223us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b180043e520. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:05.711088 (+966379us) op_driver.cc:464] REPLICATION: finished
0709 12:13:06.417945 (+706857us) write_op.cc:301] APPLY: starting
0709 12:13:06.457918 (+ 39973us) tablet.cc:1367] starting BulkCheckPresence
0709 12:13:06.897828 (+439910us) tablet.cc:1370] finished BulkCheckPresence
0709 12:13:06.897878 (+    50us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:13:08.696375 (+1798497us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:13:08.743788 (+ 47413us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=28
0709 12:13:08.743853 (+    65us) write_op.cc:312] APPLY: finished
0709 12:13:08.744435 (+   582us) log.cc:844] Serialized 188 byte log entry
0709 12:13:08.932569 (+188134us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:08.942476 (+  9907us) write_op.cc:454] Released schema lock
0709 12:13:08.978600 (+ 36124us) write_op.cc:341] FINISH: Updating metrics
0709 12:13:08.979122 (+   522us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:08.979155 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":839729,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":560310,"apply.run_cpu_time_us":87335,"apply.run_wall_time_us":2569160,"cfile_cache_hit":26,"cfile_cache_hit_bytes":27144,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6215,"cfile_init":1,"lbm_read_time_us":108858,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":6,"num_ops":14,"prepare.queue_time_us":239092,"prepare.run_cpu_time_us":74779,"prepare.run_wall_time_us":1012959,"raft.queue_time_us":612751,"raft.run_cpu_time_us":37428,"raft.run_wall_time_us":911124,"replication_time_us":2837081,"spinlock_wait_cycles":11580672,"thread_start_us":2260486,"threads_started":5,"wal-append.queue_time_us":910201,"wal-append.run_cpu_time_us":71448,"wal-append.run_wall_time_us":3949110}]]}
I20260709 12:13:13.150012 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:13.155747 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.161655 16673 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:13.162374 16503 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.670s	user 0.005s	sys 0.009s
I20260709 12:13:13.162500 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:13.170563 12642 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:13.170756 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:13.139899 16682 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:13.164369 16579 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.720s	user 0.013s	sys 0.000s
I20260709 12:13:13.165185 16463 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:13.168749 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52972.8us
I20260709 12:13:13.118888 16662 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.089706 16658 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:13.173657 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68598.6us
I20260709 12:13:13.150537 16619 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:13.206867 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:13.207226 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 498491us
I20260709 12:13:13.211886 16706 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:13.180056 16621 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:13.176957 16411 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.216943 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:13.272185 12642 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:13.278132 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106811us
W20260709 12:13:13.280894 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64058.6us
I20260709 12:13:13.212504 16463 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.285259 16709 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:13:13.285645 16708 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:13.286252 16708 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:13.286691 16708 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:13.287827 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:13.286594 16463 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:13.302476 16690 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
I20260709 12:13:13.175037 16503 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:13.302657 16696 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:13:13.324023 12565 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:13.260888 16459 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.288522 16709 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.268136 16619 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:13.339720 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 249512us
W20260709 12:13:13.289486 16693 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:13:13.330709 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:13.216766 16682 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:13:13.283449 16706 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:13.329011 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.533s (SENT) [suppressed 36 similar messages]
I20260709 12:13:13.215540 16684 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.355079 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:13.269389 16528 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:13:13.356300 16703 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.357759 12645 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
W20260709 12:13:13.390836 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60210.2us
W20260709 12:13:13.336680 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 252765us
I20260709 12:13:13.380038 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:13.276211 16621 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:13.305477 16084 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:13.386832 16631 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:13.387887 16710 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:13:13.388842 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:13.396874 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:13.398640 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127112us
I20260709 12:13:13.392941 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:13.389559 16703 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:13.390962 16551 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.629s	user 0.006s	sys 0.007s
I20260709 12:13:13.412410 12639 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 12:13:13.278947 16662 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.418462 16370 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:13.422142 16551 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.425735 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:13.426481 12642 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:13.427366 16662 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:13.428558 16551 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:13.396112 16682 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:13.351047 16482 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:13.358839 16459 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.443500 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:13.447643 12492 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:13:13.438223 16713 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:13.431103 16588 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:13.454059 16632 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.141s	user 0.012s	sys 0.003s
I20260709 12:13:13.396382 16444 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.421149 16621 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:13.463148 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110826us
I20260709 12:13:13.468370 12639 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:13:13.471066 12496 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:13:13.475028 16679 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.429s	user 0.001s	sys 0.001s
I20260709 12:13:13.476697 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:13.477648 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:13.477842 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:13.409943 16709 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:13.487641 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.484373 12642 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:13:13.484825 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73139.5us
I20260709 12:13:13.488796 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:13.484606 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.450305 16503 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:13.490026 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:13.491310 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:13.491581 12492 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:13.492751 16722 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:13.491314 16695 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.359s	user 0.001s	sys 0.001s
I20260709 12:13:13.494199 16719 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:13:13.495954 16701 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:13.493324 16722 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:13.496563 16701 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:13.496893 16722 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:13.496991 16701 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:13.495970 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2193) took 1130 ms. Trace:
I20260709 12:13:13.498014 16482 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.498597 12570 rpcz_store.cc:276] 0709 12:13:12.365559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.447778 (+ 82219us) service_pool.cc:224] Handling call
0709 12:13:12.448250 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:12.499812 (+ 51562us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:12.624202 (+124390us) log.cc:844] Serialized 14 byte log entry
0709 12:13:12.817605 (+193403us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:12.884908 (+ 67303us) log.cc:844] Serialized 26 byte log entry
0709 12:13:12.885128 (+   220us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:12.885183 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:12.885280 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:13.495680 (+610400us) raft_consensus.cc:1685] finished
0709 12:13:13.495709 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:13.495889 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26701,"prepare.queue_time_us":288311,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":52365,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":15664,"spinlock_wait_cycles":13708544,"thread_start_us":264416,"threads_started":2,"wal-append.queue_time_us":344783}
I20260709 12:13:13.199492 16680 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.385s
I20260709 12:13:13.506007 16588 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:13.506410 16658 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:13.508589 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:13.508908 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:13.493355 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:13.522102 12536 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:13.522578 12536 leader_election.cc:403] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:13.523659 16588 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.524626 16588 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:13.526762 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:13.467785 16713 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:13.470587 16653 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.452092 16682 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:13.544361 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:13.552006 12642 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:13:13.520033 16621 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.493508 16711 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:13.464684 16684 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:13:13.481575 16528 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260709 12:13:13.545964 16503 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:13:13.586475 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59797us
I20260709 12:13:13.547851 16658 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.550928 16713 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:13.605345 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:13.467432 16619 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:13.617154 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.561947 16653 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.619412 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:13.620065 12642 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:13.620453 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 291376us
W20260709 12:13:13.630883 16712 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:13:13.636462 16621 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:13.596210 16503 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:13.639915 12536 leader_election.cc:400] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:13.483388 16706 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.649984 12536 leader_election.cc:403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20260709 12:13:13.652117 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.621826 16711 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:13.663386 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58229.7us
W20260709 12:13:13.666085 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:13.666595 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.670563 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.671288 12572 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:13.673827 16684 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.661310 16629 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:13.682085 16629 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:13.651314 16503 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:13.686609 16586 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.545s	user 0.012s	sys 0.000s
I20260709 12:13:13.687062 16653 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:13.687201 16586 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.663156 16335 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:13.688107 16335 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:13.689915 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65593.9us
I20260709 12:13:13.691668 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:13.693092 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:13.695194 16676 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.688s	user 0.012s	sys 0.000s
I20260709 12:13:13.695654 16676 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:13.664460 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:13.697866 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:13.701741 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.659941 16706 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:13.658528 16658 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
W20260709 12:13:13.696333 16669 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.666s	user 0.010s	sys 0.000s
I20260709 12:13:13.719756 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:13.720716 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:13.708190 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.722885 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:13.708885 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44483.9us
I20260709 12:13:13.734283 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:13.734670 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:13.735229 12570 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:13.735671 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.736334 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:13.727546 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:13.740481 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.741219 16631 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:13.741381 12567 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:13.744289 16669 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.746357 16709 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:13.746393 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.747977 12563 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:13.750089 16724 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:13:13.749240 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.755690 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:13.710745 16653 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.755903 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.760511 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.761271 12566 raft_consensus.cc:2376] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12. Re-sending same reply.
I20260709 12:13:13.747027 16669 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:13.762753 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:13.763852 12572 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:13.763777 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.770534 16459 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:13.696871 16619 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:13.762390 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.724030 16719 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:13.777051 12563 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:13.710158 16561 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.910s	user 0.010s	sys 0.005s
I20260709 12:13:13.781805 12642 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:13.778473 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.786389 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:13.791312 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2079) took 1123 ms. Trace:
W20260709 12:13:13.791927 16686 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.680s	user 0.011s	sys 0.000s
I20260709 12:13:13.792387 16686 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.791847 12638 rpcz_store.cc:276] 0709 12:13:12.667743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.668561 (+   818us) service_pool.cc:224] Handling call
0709 12:13:12.682136 (+ 13575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:12.682479 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:12.682903 (+   424us) log.cc:844] Serialized 14 byte log entry
0709 12:13:13.689323 (+1006420us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b1800464620. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:13:13.689936 (+   613us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b1800464620. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:13.790792 (+100856us) spinlock_profiling.cc:243] Waited 9.69 ms on lock 0x7b1800464620. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:13:13.790916 (+   124us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:13.790979 (+    63us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:13.791029 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:13.791106 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:13.791232 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37592,"prepare.queue_time_us":61898,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":68,"raft.queue_time_us":142256,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":4626,"spinlock_wait_cycles":70908800,"thread_start_us":390287,"threads_started":2,"wal-append.queue_time_us":347566}
W20260709 12:13:13.797492 16688 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.919s	user 0.003s	sys 0.000s
W20260709 12:13:13.800107 16691 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.634s	user 0.001s	sys 0.001s
I20260709 12:13:13.801051 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:13.797258 16658 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.803683 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2080) took 1089 ms. Trace:
I20260709 12:13:13.807080 12572 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 7.
I20260709 12:13:13.807776 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.810328 16503 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.814846 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2071) took 1315 ms. Trace:
I20260709 12:13:13.864576 12645 rpcz_store.cc:276] 0709 12:13:12.499179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.505735 (+  6556us) service_pool.cc:224] Handling call
0709 12:13:12.533322 (+ 27587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:13.389603 (+856281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:13.814541 (+424938us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:13.814766 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1895377792}
I20260709 12:13:13.816967 16682 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.822185 16335 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.825204 16686 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:13.825958 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132966us
I20260709 12:13:13.834829 16561 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:13.836570 13260 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 12:13:13.847263 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2205) took 1107 ms. Trace:
I20260709 12:13:13.847561 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:13.708026 16586 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:13.723428 16370 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:13.851624 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:13.781438 16417 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:13.862030 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:13.860797 16703 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:13.857239 12646 rpcz_store.cc:276] 0709 12:13:12.714664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.721645 (+  6981us) service_pool.cc:224] Handling call
0709 12:13:12.740131 (+ 18486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:12.963192 (+223061us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:12.996465 (+ 33273us) log.cc:844] Serialized 14 byte log entry
0709 12:13:13.207791 (+211326us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:13.219384 (+ 11593us) log.cc:844] Serialized 26 byte log entry
0709 12:13:13.219557 (+   173us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:13.219592 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:13.803135 (+583543us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:13.803195 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:13.803267 (+    72us) raft_consensus.cc:1685] finished
0709 12:13:13.803391 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:13.803600 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85634,"prepare.queue_time_us":28921,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":5535,"raft.queue_time_us":176231,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":5943,"spinlock_wait_cycles":341329280,"thread_start_us":101259,"threads_started":1,"wal-append.queue_time_us":395787}
I20260709 12:13:13.835592 12570 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:13.876523 13260 pending_rounds.cc:85] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 8
I20260709 12:13:13.928372 13260 pending_rounds.cc:107] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 12:13:13.928817 13260 raft_consensus.cc:2889] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:13.876573 12571 rpcz_store.cc:276] 0709 12:13:12.726682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.727009 (+   327us) service_pool.cc:224] Handling call
0709 12:13:12.727413 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:12.869441 (+142028us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:12.869470 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:12.869524 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:12.869552 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:13.833994 (+964442us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:13.834062 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:13.834241 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":326120704}
I20260709 12:13:13.880605 12563 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:13.870048 12566 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:13.872270 16634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.009s	sys 0.000s
I20260709 12:13:13.882457 16335 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:13.882737 16370 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.882018 13262 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:13:13.909826 16711 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:13.911620 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:13.912895 16626 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.739s	user 0.006s	sys 0.006s
I20260709 12:13:13.915009 16503 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:13.892388 16734 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:13.915395 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2191) took 1550 ms. Trace:
W20260709 12:13:13.916380 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 178637us
I20260709 12:13:13.917734 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2195) took 1541 ms. Trace:
I20260709 12:13:13.920841 16620 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:13.923776 16559 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:13.924875 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82370.3us
I20260709 12:13:13.870136 16658 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.942579 12565 rpcz_store.cc:276] 0709 12:13:12.375939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.414048 (+ 38109us) service_pool.cc:224] Handling call
0709 12:13:13.917458 (+1503410us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b54001a27d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:13.917658 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2089981312}
I20260709 12:13:13.786026 16720 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:13:13.862432 16619 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:13.929548 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 203002us
W20260709 12:13:13.931488 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:13.933983 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:13.932797 16634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.010s	sys 0.000s
W20260709 12:13:13.932397 16687 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.947s	user 0.014s	sys 0.001s
I20260709 12:13:13.922359 16417 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:13.941870 12568 rpcz_store.cc:276] 0709 12:13:12.364942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.433033 (+ 68091us) service_pool.cc:224] Handling call
0709 12:13:12.433470 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:12.497731 (+ 64261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:12.497766 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:12.498377 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:13:12.706550 (+208173us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:13:12.706612 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:12.706725 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:12.962935 (+256210us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:13.796704 (+833769us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:13.915125 (+118421us) raft_consensus.cc:1685] finished
0709 12:13:13.915153 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:13.915320 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18477,"raft.queue_time_us":66555,"raft.run_cpu_time_us":2275,"raft.run_wall_time_us":455054,"spinlock_wait_cycles":21748480,"thread_start_us":207594,"threads_started":1,"wal-append.queue_time_us":302053}
I20260709 12:13:13.955346 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.950848 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:13.955521 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.949405 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:13.958022 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:13.954066 16720 consensus_queue.cc:1243] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:13.973743 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.978732 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.979132 13255 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:13.983144 16417 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.985805 12646 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:13:13.986894 12646 raft_consensus.cc:606] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:13.987390 12646 raft_consensus.cc:614] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:13:13.986074 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74569.9us
I20260709 12:13:13.944983 16620 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:13.990643 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:13.994127 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44830us
I20260709 12:13:13.994964 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:13.995092 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.996213 12571 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 10 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
W20260709 12:13:13.999418 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.006532 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.003429 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.008183 12570 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:14.008126 12565 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:13.959013 16653 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:13.941792 16717 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:13:14.011684 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:13.942148 16626 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.002502 16561 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:14.018393 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.017755 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:13.952350 16559 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:13.956817 16705 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:13:14.054328 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.055097 12570 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:14.019793 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:13.959347 16687 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.090340 16546 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:14.171077 16546 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:14.095077 16708 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.808s	user 0.007s	sys 0.003s
I20260709 12:13:14.099452 16658 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:14.100752 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:14.102190 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.173852 12567 raft_consensus.cc:2376] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10. Re-sending same reply.
I20260709 12:13:14.104983 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.175845 12566 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:13:14.106702 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:14.151285 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.176621 13260 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:14.108235 16586 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.110543 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.119292 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.060458 16684 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:13:14.128371 16620 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:14.134792 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80763us
I20260709 12:13:14.142397 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:13.996246 16444 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:14.144287 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.012043 16720 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:14.147115 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 203219us
I20260709 12:13:14.221846 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:14.222021 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.222711 12637 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:14.056809 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:14.160288 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:14.150033 16631 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:14.150403 16671 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:13:14.163028 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:14.150223 16529 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:14.009522 16634 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:13:14.021476 16722 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.524s	user 0.017s	sys 0.006s
W20260709 12:13:14.178664 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70896.7us
I20260709 12:13:14.182631 12643 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:13:14.184082 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:14.238981 12642 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:14.185194 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.187213 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.184198 16715 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.200794 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:14.205866 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
W20260709 12:13:14.215461 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.221341 16503 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.220865 12568 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:14.232967 16716 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:14.225916 12563 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:14.021847 16561 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:14.119308 16626 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:14.232005 13261 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:14.232393 12566 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:14.208487 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.235183 12641 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:13:14.234979 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:14.258008 12646 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 10 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 12:13:14.234115 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:14.231783 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:14.240810 12569 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:14.258921 12644 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11. Re-sending same reply.
I20260709 12:13:14.259176 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:14.105077 16370 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:14.241767 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 185078us
I20260709 12:13:14.223124 16369 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
W20260709 12:13:14.242123 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43550.5us
I20260709 12:13:14.196841 16708 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.224861 12570 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:14.253059 12565 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:14.272254 12565 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:13:14.253342 12568 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 5
W20260709 12:13:14.253743 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53068.2us
W20260709 12:13:14.256052 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 241265us
I20260709 12:13:14.256313 12572 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:13:14.272716 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50760.3us
I20260709 12:13:14.260548 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:14.274140 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.274712 13255 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:14.274887 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.267848 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:14.219651 12496 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:14.223371 16586 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.272948 12568 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260709 12:13:14.234534 16634 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:14.277807 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.251295 16528 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:14.257618 16706 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.531s	user 0.012s	sys 0.000s
I20260709 12:13:14.280884 12568 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:14.282372 12565 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:14.295030 12565 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:14.287472 16686 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:14.256866 16561 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:14.287540 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.236884 16529 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.292666 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:14.296737 12493 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:13:14.234083 16631 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.298385 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:14.247109 16722 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.299760 12496 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:13:14.276878 16732 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:14.302006 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.302492 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:14.277125 16370 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.290438 16634 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:14.308954 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.309111 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.255685 16503 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.307050 12490 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:13:14.290673 16528 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:14.306116 16701 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.804s	user 0.010s	sys 0.004s
I20260709 12:13:14.247226 16546 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.298295 16705 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:14.316762 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:14.340039 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.331944 12491 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:14.300172 16686 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:14.301860 16369 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:14.252657 16715 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:14.349143 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75090.6us
I20260709 12:13:14.298565 16559 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:14.303490 16417 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:14.308326 16557 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:14.312398 16732 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:14.328270 12565 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.326814 16734 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:14.357324 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:14.358987 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150594us
I20260709 12:13:14.333961 16528 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:14.360146 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.335145 16720 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:14.367102 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:14.364806 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.372789 12491 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:14.374109 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.374486 12491 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:14.372608 12565 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:14.311501 16370 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:14.362807 16665 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.244s	user 0.011s	sys 0.004s
W20260709 12:13:14.399224 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97293.2us
I20260709 12:13:14.398713 12495 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:14.351270 16703 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.402294 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:14.353688 16417 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:14.408810 12565 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:14.413386 12640 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:14.339746 16701 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.366771 16732 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:14.418247 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.418486 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.384128 16720 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:14.422755 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:14.424705 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:14.423379 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106758us
I20260709 12:13:14.412541 16665 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.418900 16703 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.383574 16671 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:14.431783 16529 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:14.397464 16721 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:14.439597 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:14.421864 16737 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
I20260709 12:13:14.406314 16482 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:13:14.435014 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:14.446224 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:14.384367 16669 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:13:14.418550 16687 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:14.383450 16559 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:13:14.436409 16720 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:14.452162 16738 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
W20260709 12:13:14.505793 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105804us
I20260709 12:13:14.508415 16619 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:13:14.506222 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:14.506234 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80399.3us
I20260709 12:13:14.508955 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.511137 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.434582 16705 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:13:14.524185 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.503080 16731 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.533710 16731 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.535045 16734 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:13:14.535413 12494 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:13:14.535648 16734 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:14.536610 16731 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:14.536796 16734 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.512625 16529 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:14.542068 16529 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:14.543176 16529 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:14.543516 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160158us
I20260709 12:13:14.446314 16671 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.520990 16621 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:14.547060 16671 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:14.547663 16671 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:13:14.512400 16708 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:14.515069 16687 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:14.554060 16687 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:14.554446 16665 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:14.528256 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.558629 16744 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:14.558830 16701 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:14.559275 16744 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.559814 16701 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:14.516903 16557 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:14.561715 16557 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:14.563077 16744 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:14.563026 16557 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.564980 16722 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:14.528033 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:14.560945 16706 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:14.554571 16687 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:14.571164 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.539827 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:14.509702 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69157.2us
I20260709 12:13:14.594741 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.596496 12495 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:14.596791 13260 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 12:13:14.602010 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74214.5us
W20260709 12:13:14.560487 16335 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.624s	user 0.010s	sys 0.000s
I20260709 12:13:14.612186 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:14.547976 16741 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
W20260709 12:13:14.611658 16760 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.000s	sys 0.003s
W20260709 12:13:14.626586 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81865.1us
I20260709 12:13:14.618476 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.620067 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:14.525735 16369 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:14.607550 16739 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
W20260709 12:13:14.607847 16740 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 12:13:14.643548 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.644613 12568 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:14.643909 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:14.661280 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:14.624260 16713 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.002s	user 0.015s	sys 0.001s
I20260709 12:13:14.677784 12568 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:14.679028 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.682411 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.616601 16665 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:14.687243 12565 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:14.687997 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:14.677402 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:14.691355 16743 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.330s	user 0.003s	sys 0.000s
I20260709 12:13:14.560560 16701 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:14.707439 16721 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:14.707952 16559 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:13:14.714090 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 381584us
W20260709 12:13:14.734107 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115736us
I20260709 12:13:14.679695 16708 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:14.695924 16631 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:14.606730 16706 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.616374 16744 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
W20260709 12:13:14.751720 16750 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
W20260709 12:13:14.752902 16748 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:13:14.755020 16721 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.756662 16370 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:14.756855 16762 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:14.759397 16761 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:14.733173 16705 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:13:14.761673 16701 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:14.740144 16369 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:14.766884 16482 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:14.700471 16713 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.769734 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:14.770943 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.772053 16559 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:14.776739 16626 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:14.550872 16417 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.778254 12489 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:14.780362 16629 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:14.812290 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 273033us
I20260709 12:13:14.885851 16482 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:14.733505 16704 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:14.783521 16746 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:13:14.733721 16715 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:14.830960 13255 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:14.866492 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.868940 12565 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:14.954663 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.955426 12565 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:13:14.956614 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:14.957058 12565 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:14.883095 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:14.886021 12496 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:13:14.958370 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:14.959041 12570 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 12:13:14.886530 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:14.909943 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140313us
W20260709 12:13:14.960088 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.913914 12571 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 8: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:14.914990 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:14.889110 16369 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:14.882606 16703 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:14.906569 16742 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.243s	user 0.002s	sys 0.000s
I20260709 12:13:14.681771 16335 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:13:14.921358 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:15.014176 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92907.6us
I20260709 12:13:15.014971 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.016203 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:14.924873 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:14.888792 16761 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:14.949916 12637 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 5. Current term is 7. Ops: []
I20260709 12:13:15.017119 12643 raft_consensus.cc:2368] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9.
I20260709 12:13:14.909245 16722 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:14.866488 16546 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:13:14.972558 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 508) took 1097 ms. Trace:
I20260709 12:13:15.013149 16411 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:13:15.026098 13253 rpcz_store.cc:276] 0709 12:13:13.856084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:13.856287 (+   203us) service_pool.cc:224] Handling call
0709 12:13:14.170116 (+313829us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:13:14.170417 (+   301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:14.170941 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.170984 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:14.171639 (+   655us) log.cc:844] Serialized 26 byte log entry
0709 12:13:14.344557 (+172918us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:14.344625 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:14.344729 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:14.937211 (+592482us) raft_consensus.cc:1685] finished
0709 12:13:14.937238 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:14.952878 (+ 15640us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":323,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":8961792,"thread_start_us":172363,"threads_started":1,"wal-append.queue_time_us":319933}
I20260709 12:13:15.027567 16734 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:14.960479 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:15.028584 16701 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:13:15.029090 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68727.7us
I20260709 12:13:15.029524 16722 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:15.030315 16335 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:13:14.974001 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:15.033059 16744 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:15.033442 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72489.8us
W20260709 12:13:15.034162 16732 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.590s	user 0.011s	sys 0.000s
I20260709 12:13:15.034672 16732 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.035164 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:15.026134 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:15.036859 12639 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13.
I20260709 12:13:15.037698 16335 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:13:14.984477 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.936295 16482 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:14.992619 16682 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:13:14.942166 16417 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:14.995669 16759 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:13:15.039042 16335 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:15.047209 16335 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:15.049194 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2215) took 1049 ms. Trace:
I20260709 12:13:15.049751 12572 rpcz_store.cc:276] 0709 12:13:13.999359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.274413 (+275054us) service_pool.cc:224] Handling call
0709 12:13:14.274840 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:14.313925 (+ 39085us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.313985 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:14.344208 (+ 30223us) log.cc:844] Serialized 26 byte log entry
0709 12:13:14.519804 (+175596us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:14.519877 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:14.519991 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:14.753264 (+233273us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:15.048817 (+295553us) raft_consensus.cc:1685] finished
0709 12:13:15.048842 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:15.049115 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50337,"raft.queue_time_us":30594,"raft.run_cpu_time_us":3388,"raft.run_wall_time_us":496473,"spinlock_wait_cycles":6593280,"thread_start_us":108927,"threads_started":1,"wal-append.queue_time_us":262129}
I20260709 12:13:14.998072 16713 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.004333 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:14.940300 16704 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:15.054836 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50595.4us
W20260709 12:13:14.954128 16716 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.697s	user 0.008s	sys 0.004s
I20260709 12:13:15.006801 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
W20260709 12:13:15.009318 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:14.935514 16665 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:15.018991 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:14.919284 16490 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:13:15.025097 16686 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.020002 16761 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:15.037886 16752 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:14.990465 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:14.982105 16629 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:14.974254 16711 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:15.135340 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116395us
I20260709 12:13:15.138485 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.140596 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.151919 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.157809 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.159844 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.169438 12572 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:13:15.169948 12572 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:15.065177 16701 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:15.133970 16716 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:15.131326 16722 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:15.150312 16370 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:15.146221 16411 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:15.234359 12494 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:13:15.220805 12572 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.144304 16744 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:15.221931 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2295) took 1460 ms. Trace:
I20260709 12:13:15.239374 12564 rpcz_store.cc:276] 0709 12:13:13.761811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.190938 (+429127us) service_pool.cc:224] Handling call
0709 12:13:14.191409 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:14.203113 (+ 11704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.203163 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:14.230036 (+ 26873us) log.cc:844] Serialized 26 byte log entry
0709 12:13:14.335677 (+105641us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:14.335742 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:14.335852 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:15.220703 (+884851us) raft_consensus.cc:1685] finished
0709 12:13:15.220726 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:15.221852 (+  1126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11117,"raft.queue_time_us":11306,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9060,"thread_start_us":73525,"threads_started":1,"wal-append.queue_time_us":120711}
W20260709 12:13:15.172463 16720 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.651s	user 0.012s	sys 0.003s
I20260709 12:13:15.247053 16720 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.249691 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.250082 12491 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:15.154852 16717 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.251842 16717 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.254420 16717 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.228291 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:15.255373 12565 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:15.134423 16682 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:13:15.141394 16713 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:13:15.178176 16621 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:15.176872 16708 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:15.143538 16703 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 12:13:15.237427 12571 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:13:15.237167 16768 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:15.181497 16752 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:15.160800 16711 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:15.277505 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.286602 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.289830 12572 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:13:15.262821 16561 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.947s	user 0.015s	sys 0.000s
I20260709 12:13:15.227716 16762 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:15.297448 12494 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:15.262918 16732 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.263286 16551 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:15.306543 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 146830us
W20260709 12:13:15.298300 16634 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.954s	user 0.013s	sys 0.000s
W20260709 12:13:15.297091 16687 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.680s	user 0.011s	sys 0.000s
W20260709 12:13:15.304206 16776 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:13:15.322679 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.301753 16704 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.303975 16768 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:15.343281 12572 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:13:15.345906 12572 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 2
I20260709 12:13:15.346242 12572 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:13:15.346263 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:15.346705 12572 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:15.347489 12572 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:13:15.347891 12572 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:15.300410 16561 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.292611 16665 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.362460 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.315441 16551 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.365118 16703 consensus_queue.cc:1243] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:15.370388 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:15.379475 16771 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:13:15.311206 16722 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.381923 16708 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:15.382095 16682 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:15.324990 16687 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.387367 12492 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:15.320081 16634 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.357476 16716 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:15.396842 16716 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:15.397297 16716 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:15.398252 16634 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.398689 16503 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:15.399024 12642 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:15.400290 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 480) took 1008 ms. Trace:
I20260709 12:13:15.400625 12642 rpcz_store.cc:276] 0709 12:13:14.391662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.394503 (+  2841us) service_pool.cc:224] Handling call
0709 12:13:14.394942 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:15.399445 (+1004503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.400038 (+   593us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54002e5254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:15.400223 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2308656384}
I20260709 12:13:15.402801 16720 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.405164 16619 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:15.405723 16561 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.406139 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.406896 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.408304 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.408910 12496 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:15.410053 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.409955 16768 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.411631 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.413039 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.407083 12489 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:15.406132 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:15.423569 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.384430 16708 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:15.384609 16732 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:13:15.421808 16785 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:13:15.441177 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
W20260709 12:13:15.435640 16528 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.039s	user 0.014s	sys 0.000s
I20260709 12:13:15.427413 16619 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:15.358575 16794 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.088s
W20260709 12:13:15.456525 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 189628us
I20260709 12:13:15.462576 12496 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:15.464347 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.468349 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.471594 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:15.440557 16708 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:15.472888 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:15.416743 16335 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.477913 16803 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:15.478161 16711 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:15.480684 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.457417 16235 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.460891 16705 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:15.489409 12496 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
W20260709 12:13:15.492882 16787 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
W20260709 12:13:15.493407 16791 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:13:15.495713 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.385586 16752 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.498980 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.502537 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:15.502668 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:15.504146 16720 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:15.504483 16753 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:15.505070 16720 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:15.505707 16720 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:15.506516 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:15.506702 16753 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:13:15.514143 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:15.515367 12571 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:15.516788 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:15.499636 12488 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:15.387764 16762 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.467403 16528 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.533674 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.535612 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.536477 12496 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:15.499486 16803 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:15.537900 16708 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:15.545051 16801 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:13:15.554623 16687 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.497251 16705 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.568189 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.570889 16705 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:15.573609 16417 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.530s	user 0.010s	sys 0.002s
I20260709 12:13:15.556210 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.575541 12496 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:15.576613 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.557016 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.579336 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.580168 12488 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:13:15.491185 16235 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.582387 13253 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:15.582962 16411 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:15.583398 16411 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:15.584007 16235 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.584332 16411 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.584908 16335 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.585891 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 517) took 1163 ms. Trace:
I20260709 12:13:15.543344 16716 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.586342 12491 rpcz_store.cc:276] 0709 12:13:14.422368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.422731 (+   363us) service_pool.cc:224] Handling call
0709 12:13:15.585612 (+1162881us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:15.585792 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1802519808}
I20260709 12:13:15.588213 16716 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.589036 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:15.590036 12496 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:13:15.591252 13253 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:15.557821 16551 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.558357 16620 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:15.559005 16761 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.595903 12496 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:13:15.531106 16444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.019s	sys 0.026s
I20260709 12:13:15.485102 16482 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.574785 16708 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:15.610474 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 651) took 1325 ms. Trace:
I20260709 12:13:15.611218 13255 rpcz_store.cc:276] 0709 12:13:14.285503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.285793 (+   290us) service_pool.cc:224] Handling call
0709 12:13:14.306153 (+ 20360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:14.953285 (+647132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.610056 (+656771us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:15.610369 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1206304000}
I20260709 12:13:15.614773 16704 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:15.615484 16704 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:15.616371 16803 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.598069 12489 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:15.600993 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:15.605083 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:15.585541 16411 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:15.601778 16444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.019s	sys 0.026s
I20260709 12:13:15.629113 13261 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:15.623723 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:15.599885 16680 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.636590 13262 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:15.616760 16704 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:15.644336 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2450) took 1350 ms (client timeout 1614 ms). Trace:
W20260709 12:13:15.644989 12492 rpcz_store.cc:269] 0709 12:13:14.293447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.315257 (+ 21810us) service_pool.cc:224] Handling call
0709 12:13:15.644039 (+1328782us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:15.644261 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2400542080}
W20260709 12:13:15.641084 16800 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.002s	sys 0.000s
I20260709 12:13:15.648640 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:15.649406 13254 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:15.603906 16720 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.667474 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:15.668668 13254 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:15.672420 16369 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.647231 16680 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.690915 16482 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:15.692023 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.703796 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:15.675153 16761 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.715194 16335 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 12:13:15.721331 16335 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:15.721796 16335 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:15.722131 16720 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.722688 16335 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.723973 16335 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:15.710981 16766 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:13:15.737215 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2456) took 1323 ms (client timeout 1749 ms). Trace:
I20260709 12:13:15.737694 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:15.739220 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:15.742995 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.747179 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:15.745519 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:15.748198 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:15.737749 12495 rpcz_store.cc:269] 0709 12:13:14.413663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.413840 (+   177us) service_pool.cc:224] Handling call
0709 12:13:15.736992 (+1323152us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:15.737139 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76401024}
I20260709 12:13:15.751084 16716 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:15.753366 16370 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:15.753952 16370 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:15.754837 16370 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.753185 16444 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:15.756976 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:15.757470 16765 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.840s	user 0.004s	sys 0.000s
I20260709 12:13:15.758077 16084 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:15.758551 16084 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:15.758790 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:15.758922 16084 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:15.763697 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.763681 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:15.778169 16411 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:15.780880 16716 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:15.780403 16369 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:15.796525 12492 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:15.786816 16370 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:15.797276 16716 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:15.784700 16619 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.812729 12488 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:15.804540 16634 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.801425 16411 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:15.824716 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.808580 16369 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:15.813851 16733 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.815318 16716 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:15.835194 12491 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:15.837682 12494 raft_consensus.cc:606] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:15.838166 12494 raft_consensus.cc:614] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:15.842078 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2219) took 1837 ms. Trace:
I20260709 12:13:15.821259 16417 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:13:15.848333 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2226) took 1153 ms. Trace:
I20260709 12:13:15.848831 12568 rpcz_store.cc:276] 0709 12:13:14.694684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.695203 (+   519us) service_pool.cc:224] Handling call
0709 12:13:14.695618 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:14.695873 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:14.696189 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.847921 (+1151732us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:15.847977 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:15.848019 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.848096 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:15.848279 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1005843,"prepare.queue_time_us":6928,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"thread_start_us":145367,"threads_started":1,"wal-append.queue_time_us":460004}
I20260709 12:13:15.849676 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:15.849820 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:15.850759 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.842585 12567 rpcz_store.cc:276] 0709 12:13:14.004246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.288158 (+283912us) service_pool.cc:224] Handling call
0709 12:13:14.288676 (+   518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:14.288946 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:14.452562 (+163616us) log.cc:844] Serialized 14 byte log entry
0709 12:13:14.703218 (+250656us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.579096 (+875878us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b180042ae40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:15.841417 (+262321us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b180042ae40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:15.841586 (+   169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:15.841685 (+    99us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:15.841749 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.841844 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:15.842010 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1074152,"prepare.queue_time_us":196798,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":118,"raft.queue_time_us":1172736,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":50064,"spinlock_wait_cycles":110250368,"thread_start_us":352522,"threads_started":2,"wal-append.queue_time_us":376687}
W20260709 12:13:15.857606 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.839339 16733 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.860960 16417 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:15.861459 16417 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:15.861120 16411 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.862416 16417 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:15.863191 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.863547 16417 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:15.863698 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:15.825960 16634 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.842844 16653 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:15.860533 16619 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.869130 12497 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:15.876281 12570 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:13:15.876988 12564 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 12:13:15.890462 12564 pending_rounds.cc:85] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:13:15.891470 12564 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:13:15.893097 12564 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:15.857925 16762 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:15.912658 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:15.914628 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:15.915211 12564 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:15.913085 16503 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:15.920066 16503 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:15.922542 16757 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.163s	user 0.008s	sys 0.000s
I20260709 12:13:15.922717 16503 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.926494 16761 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:15.927127 16482 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:13:15.902806 16814 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:13:15.897850 16671 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:15.943531 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:15.947906 12536 connection.cc:256] RPC call timeout handler was delayed by -5.5091210820000924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
W20260709 12:13:15.948505 12536 connection.cc:256] RPC call timeout handler was delayed by -5.4434210819999862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
W20260709 12:13:15.948909 12536 connection.cc:256] RPC call timeout handler was delayed by -4.6163210820000131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2812s for a response
W20260709 12:13:15.949319 12536 connection.cc:256] RPC call timeout handler was delayed by -4.4660210819999975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2979s for a response
I20260709 12:13:15.949599 16084 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.950878 16631 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.931025 16529 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:15.962553 16631 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.962910 16529 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:15.963853 12488 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:15.961334 16658 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:15.965899 16713 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:15.966463 16658 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:15.967648 16818 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:13:15.949287 16369 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.951234 16671 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:15.971592 16084 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:15.974052 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:15.975526 16671 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.969591 16658 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.978004 16658 raft_consensus.cc:2764] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 12:13:15.917732 16762 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:15.980674 16658 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:15.983865 12536 leader_election.cc:400] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:15.984272 12536 leader_election.cc:403] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:13:15.986129 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:15.986550 16819 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:13:15.987926 16764 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:15.988121 16634 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:15.989356 16764 raft_consensus.cc:2764] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:13:15.993338 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:15.965339 16631 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:15.996901 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:15.996225 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 519) took 1246 ms. Trace:
I20260709 12:13:16.000100 16762 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.004516 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.016008 12570 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:16.029680 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:16.030867 12495 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:15.996340 16671 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:15.998579 12490 rpcz_store.cc:276] 0709 12:13:14.750211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.750367 (+   156us) service_pool.cc:224] Handling call
0709 12:13:14.750811 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:14.768779 (+ 17968us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.768812 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:15.619685 (+850873us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:15.620399 (+   714us) log.cc:844] Serialized 26 byte log entry
0709 12:13:15.790334 (+169935us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:15.790371 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.790440 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:15.990749 (+200309us) raft_consensus.cc:1685] finished
0709 12:13:15.990774 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:15.996150 (+  5376us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17511,"raft.queue_time_us":74352,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5496,"spinlock_wait_cycles":32956672,"thread_start_us":169399,"threads_started":1,"wal-append.queue_time_us":203088}
I20260709 12:13:15.978999 16706 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.037763 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.048434 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:16.047657 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.059265 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.060537 12643 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:13:16.061267 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.064009 12642 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:16.064958 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.072105 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:16.074085 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.074810 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.078790 12643 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:13:16.085033 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.086597 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.087893 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.088781 12645 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:16.088582 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.091679 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.094118 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.059366 16706 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.086232 12638 pending_rounds.cc:85] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:16.095621 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:16.096552 12646 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 6. Current term is 7. Ops: []
I20260709 12:13:16.095876 12638 pending_rounds.cc:107] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:13:16.097184 12638 raft_consensus.cc:2889] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:16.084942 16830 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:16.099115 16830 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:16.100368 16830 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.104395 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:16.108201 16758 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.360s	user 0.003s	sys 0.003s
I20260709 12:13:16.042085 16671 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:16.109665 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.109853 16825 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:16.110448 16825 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:16.110841 16825 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:16.111289 16671 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.112531 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.113843 16828 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:13:16.114048 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.115324 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.115940 12570 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:16.117206 16754 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.361s	user 0.002s	sys 0.005s
I20260709 12:13:16.059113 16713 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.105403 13253 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:16.123030 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.123862 12643 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:16.125649 12564 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:16.127084 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 626) took 2295 ms. Trace:
I20260709 12:13:16.128038 12646 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:16.128607 12644 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:16.129324 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:16.132303 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.127645 12569 rpcz_store.cc:276] 0709 12:13:13.831538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.259422 (+427884us) service_pool.cc:224] Handling call
0709 12:13:14.259800 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:14.288842 (+ 29042us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.288877 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:14.289376 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:13:14.442154 (+152778us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:14.442200 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:14.442306 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:16.116419 (+1674113us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:16.126710 (+ 10291us) raft_consensus.cc:1685] finished
0709 12:13:16.126733 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:16.127008 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2010,"raft.queue_time_us":30199,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":8335,"spinlock_wait_cycles":128066944,"thread_start_us":108731,"threads_started":1,"wal-append.queue_time_us":295761}
I20260709 12:13:16.136066 12564 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:16.137528 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:16.138090 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44856.3us
I20260709 12:13:16.138460 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.138321 12564 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:13:16.144277 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:16.121971 16770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.118s	user 0.002s	sys 0.012s
I20260709 12:13:16.145895 12645 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:16.155879 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:16.139264 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.175076 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:16.162928 16770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.238s	user 0.003s	sys 0.012s
I20260709 12:13:16.179667 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:16.185302 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.134522 16713 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.179152 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:16.128590 16706 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.198464 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.204805 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 479) took 1813 ms. Trace:
I20260709 12:13:16.192672 12564 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:16.208026 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:16.194062 13253 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:16.210975 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.194365 12570 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:16.204794 16586 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:16.214424 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.217113 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.215838 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:16.218274 16826 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:13:16.226977 12569 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:16.205324 16831 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:13:16.227771 12564 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:13:16.228176 12565 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:16.228783 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.190308 16631 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:13:16.229756 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.232085 12571 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 7. Current term is 8. Ops: []
I20260709 12:13:16.247792 16828 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.248544 16828 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.250008 16671 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.250613 16671 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.250890 16770 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:16.252339 16770 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:13:16.253568 12639 pending_rounds.cc:85] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:16.254036 12639 pending_rounds.cc:107] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:13:16.254320 12639 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:16.247948 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.205857 12641 rpcz_store.cc:276] 0709 12:13:14.391223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.391849 (+   626us) service_pool.cc:224] Handling call
0709 12:13:14.392381 (+   532us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:14.392883 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.392922 (+    39us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:13:14.393871 (+   949us) log.cc:844] Serialized 78 byte log entry
0709 12:13:14.455896 (+ 62025us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:14.506420 (+ 50524us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.045172 (+538752us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b180033f9a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:15.641814 (+596642us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b180033f9a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:15.642049 (+   235us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.642458 (+   409us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.642610 (+   152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.642691 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:15.743203 (+100512us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b180033f9a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f
0709 12:13:15.743663 (+   460us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:16.107192 (+363529us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:16.204195 (+ 97003us) raft_consensus.cc:1685] finished
0709 12:13:16.204246 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:16.204689 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1119504,"prepare.queue_time_us":1227981,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":116,"raft.queue_time_us":15892,"raft.run_cpu_time_us":2576,"raft.run_wall_time_us":97782,"spinlock_wait_cycles":143511296,"thread_start_us":61448,"threads_started":1,"wal-append.queue_time_us":227322}
I20260709 12:13:16.252821 16770 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:16.270323 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.271327 12641 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:16.252835 16671 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.278919 16558 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:16.260550 16716 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:16.330319 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82464.7us
I20260709 12:13:16.331615 12536 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13.
I20260709 12:13:16.332068 12536 leader_election.cc:403] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:13:16.329044 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.333777 12641 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13.
I20260709 12:13:16.275315 16827 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:16.339552 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.285240 16829 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.298286 16558 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:16.350160 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.350983 12645 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:16.353826 12640 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:13:16.351620 12641 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:13:16.341706 16634 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.351504 16834 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.357191 16658 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:16.361097 16827 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13.
I20260709 12:13:16.373481 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.382690 12641 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 12:13:16.371198 16823 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.313s	user 0.001s	sys 0.001s
W20260709 12:13:16.394721 16837 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:13:16.395406 16706 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.382297 16834 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.396688 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.398517 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.398798 16834 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.357283 16829 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.401889 16829 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.403208 12567 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:13:16.408849 16528 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:16.385233 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53821.9us
I20260709 12:13:16.346796 16770 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.387629 16716 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.368311 16634 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.391661 16825 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.392589 16713 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:16.393747 16767 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.478s	user 0.006s	sys 0.000s
W20260709 12:13:16.443785 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.453657 12536 leader_election.cc:400] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9.
I20260709 12:13:16.460578 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:16.471727 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:16.487828 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:16.488858 12536 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13.
I20260709 12:13:16.489977 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:16.491531 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:16.492542 16653 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:16.493294 16653 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:16.495905 16634 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.464782 16770 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:16.498749 16825 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.484488 16369 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.483304 16713 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.417145 16558 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.528638 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.533743 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:16.535318 16558 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:16.527652 16369 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.531189 16713 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.540844 12567 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:13:16.586074 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:16.652451 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.664935 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2241) took 1155 ms. Trace:
I20260709 12:13:16.665477 12572 rpcz_store.cc:276] 0709 12:13:15.509465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.513423 (+  3958us) service_pool.cc:224] Handling call
0709 12:13:15.518396 (+  4973us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:16.044482 (+526086us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:16.059842 (+ 15360us) log.cc:844] Serialized 14 byte log entry
0709 12:13:16.342800 (+282958us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:16.354360 (+ 11560us) log.cc:844] Serialized 26 byte log entry
0709 12:13:16.354534 (+   174us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:16.354797 (+   263us) log.cc:844] Serialized 14 byte log entry
0709 12:13:16.354990 (+   193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:16.355082 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:16.664642 (+309560us) raft_consensus.cc:1685] finished
0709 12:13:16.664674 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:16.664852 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":396966,"prepare.queue_time_us":19205,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":60,"raft.queue_time_us":468990,"raft.run_cpu_time_us":526,"raft.run_wall_time_us":17250,"spinlock_wait_cycles":7897344,"thread_start_us":310408,"threads_started":2,"wal-append.queue_time_us":300692}
W20260709 12:13:16.669239 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.700335 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.701278 12569 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:16.704975 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.705137 12567 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:16.706373 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:16.707459 12564 raft_consensus.cc:2376] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9. Re-sending same reply.
I20260709 12:13:16.707588 12572 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:16.709071 12569 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:13:16.709718 16369 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:16.710090 16841 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:13:16.712416 12567 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 12:13:16.729959 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.733371 16369 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.734071 16369 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.736748 16369 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:16.770936 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.771808 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:16.772852 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.781896 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 625) took 2950 ms. Trace:
I20260709 12:13:16.782475 12563 rpcz_store.cc:276] 0709 12:13:13.831139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.262198 (+431059us) service_pool.cc:224] Handling call
0709 12:13:14.284164 (+ 21966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:14.329577 (+ 45413us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:14.478664 (+149087us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.719732 (+1241068us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:15.866386 (+146654us) log.cc:844] Serialized 14 byte log entry
0709 12:13:15.913954 (+ 47568us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:15.914024 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:15.914080 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.962360 (+ 48280us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:16.781531 (+819171us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:16.781593 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:16.781813 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1159374,"prepare.queue_time_us":149341,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":98,"raft.queue_time_us":1208009,"raft.run_cpu_time_us":1225,"raft.run_wall_time_us":116117,"spinlock_wait_cycles":177136640,"thread_start_us":395626,"threads_started":3,"wal-append.queue_time_us":353435}
I20260709 12:13:16.787371 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.788499 12637 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
W20260709 12:13:16.790328 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.792531 12570 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:16.792791 16626 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.793272 16626 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.795583 16626 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.796716 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:16.797890 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3444895780000934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260709 12:13:16.798635 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1113895779999439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
I20260709 12:13:16.728636 16713 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:16.793392 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.801990 16731 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:16.802726 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:16.803887 16731 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:13:16.804494 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.038s	sys 0.049s
W20260709 12:13:16.804956 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.038s	sys 0.049s
I20260709 12:13:16.805042 16626 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:13:16.806075 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.806896 12637 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:16.805292 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:16.807612 16763 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:13:16.731132 16833 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:13:16.810537 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.809079 16731 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:13:16.823900 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 214767us
W20260709 12:13:16.823951 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 155799us
I20260709 12:13:16.834625 16733 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.825891 16761 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 12:13:16.815472 16626 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:13:16.839214 16626 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:13:16.839607 16626 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:16.840410 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.841274 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:16.819270 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.844131 16772 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:13:16.845563 16825 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.852859 16825 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.843813 12637 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:16.856521 16825 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.860117 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:16.862471 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:16.865067 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49548.3us
I20260709 12:13:16.866370 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:16.869556 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.872228 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.872815 12637 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:16.875617 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.869007 16671 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.864095 16369 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.877712 16825 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.878021 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.885206 12565 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:13:16.871481 16733 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:16.886123 16680 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.875931 16528 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:16.887396 16825 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.883059 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:16.840451 16626 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.890298 16626 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:13:16.890527 16831 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:16.890942 16831 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:16.891291 16831 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:16.894701 16769 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.006s	user 0.007s	sys 0.008s
I20260709 12:13:16.895422 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.896193 12643 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:16.896560 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 627) took 3064 ms. Trace:
I20260709 12:13:16.897519 16369 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.900506 16369 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:16.901311 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.902271 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2468) took 1157 ms. Trace:
I20260709 12:13:16.903147 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.910189 12492 rpcz_store.cc:276] 0709 12:13:15.744742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.744919 (+   177us) service_pool.cc:224] Handling call
0709 12:13:15.745397 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:15.799188 (+ 53791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:16.901942 (+1102754us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:16.902184 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":116887808}
I20260709 12:13:16.903864 16777 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:16.896996 12566 rpcz_store.cc:276] 0709 12:13:13.831849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:14.260526 (+428677us) service_pool.cc:224] Handling call
0709 12:13:14.280271 (+ 19745us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:14.290140 (+  9869us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:14.290187 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:14.299682 (+  9495us) log.cc:844] Serialized 26 byte log entry
0709 12:13:14.626229 (+326547us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:14.626287 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:14.626384 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:16.875097 (+2248713us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:16.896163 (+ 21066us) raft_consensus.cc:1685] finished
0709 12:13:16.896202 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:16.896494 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12645,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":13107,"spinlock_wait_cycles":95891200,"thread_start_us":292352,"threads_started":1,"wal-append.queue_time_us":418974}
W20260709 12:13:16.907464 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:16.907701 16850 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:13:16.872097 16731 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260709 12:13:16.908937 16851 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:13:16.908353 16671 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:16.897207 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.890082 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:16.921851 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.922550 12643 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:16.922403 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.922758 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:16.923653 12570 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:16.923971 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.925001 12570 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:16.925602 12566 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:16.927160 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:16.928474 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:16.929352 13254 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13. Re-sending same reply.
I20260709 12:13:16.931530 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.922904 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:16.916464 12637 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:16.909467 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:16.917821 16369 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:16.911944 16777 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:16.919477 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2466) took 1511 ms. Trace:
I20260709 12:13:16.920632 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:16.934073 12638 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:16.935678 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.936270 12491 rpcz_store.cc:276] 0709 12:13:15.407950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.587347 (+179397us) service_pool.cc:224] Handling call
0709 12:13:15.587717 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:15.837484 (+249767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:16.919227 (+1081743us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:16.919424 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":568573824}
I20260709 12:13:16.913194 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.971737 16828 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:16.970853 16671 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.942528 16827 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:16.946835 16417 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:16.936292 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91317.2us
I20260709 12:13:16.946291 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.948114 16825 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:16.921802 16761 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:13:16.964542 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84674.4us
I20260709 12:13:16.933184 16649 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:16.938133 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:16.985898 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47845.8us
I20260709 12:13:16.987035 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:16.964959 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.939900 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:16.935283 16369 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:16.943305 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:16.989722 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:16.991037 12637 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:16.984211 16731 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:16.994889 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.982297 16827 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.984542 16761 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:16.982458 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69369.7us
I20260709 12:13:16.987205 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:16.988108 12638 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:16.949416 16777 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.005578 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:17.006379 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.006975 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:16.988484 16713 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:17.016463 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:16.993882 16825 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:17.017720 12637 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:16.990815 16768 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.028882 16768 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.031260 16768 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.033207 16777 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:17.019623 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:17.035075 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.014659 16671 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:17.027748 16369 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:17.036044 16369 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.036681 16369 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.000552 16827 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:16.983700 16417 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.023619 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:17.036095 16671 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:17.038335 12645 raft_consensus.cc:2368] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:17.027916 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.000470 16706 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:16.989566 16649 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:16.995622 16558 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:17.047547 12638 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:17.079615 16649 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:17.080826 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.020233 16828 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:17.042138 16731 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:17.049472 16369 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.022410 16825 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.048635 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.084458 16825 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:17.084676 12645 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:17.039340 16713 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.072963 16417 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.075973 16716 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:17.098066 16716 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.099274 16713 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:17.100366 16716 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:17.075634 16761 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.077838 16831 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.079512 12642 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:13:17.083475 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.048804 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.105896 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.086097 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:17.107577 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.109220 12494 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:17.109997 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:17.113879 16417 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:17.114454 16417 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:17.114876 16846 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.318s	user 0.002s	sys 0.000s
I20260709 12:13:17.115150 12637 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:17.116791 12637 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20260709 12:13:17.118180 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2111) took 1051 ms. Trace:
I20260709 12:13:17.118589 12642 rpcz_store.cc:276] 0709 12:13:16.066799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:17.078588 (+1011789us) service_pool.cc:224] Handling call
0709 12:13:17.079093 (+   505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:17.101873 (+ 22780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:17.118116 (+ 16243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:17.120024 12642 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:17.120708 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:17.122776 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.123293 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.124069 12637 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:13:17.124143 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.125329 12637 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:17.126680 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.121443 12642 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:13:17.127698 12497 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:17.127733 12637 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:17.129227 12642 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:17.131734 12637 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:13:17.133584 12637 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:17.138407 12495 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:17.140267 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.141360 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:17.102180 16828 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:17.101709 16761 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:17.097138 16860 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:17.114872 16417 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:17.211223 12645 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:17.212278 12637 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:17.142395 16713 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:13:17.213943 13260 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:17.219229 13259 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:13:17.222667 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:17.223424 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 216497us
I20260709 12:13:17.223846 13254 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 8: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:17.211246 12492 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:17.122165 16858 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.231407 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:17.229921 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 655) took 1718 ms. Trace:
I20260709 12:13:17.232631 13253 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 12:13:17.236109 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 231522us
I20260709 12:13:17.232591 13261 rpcz_store.cc:276] 0709 12:13:15.501899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.502094 (+   195us) service_pool.cc:224] Handling call
0709 12:13:17.178795 (+1676701us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:17.220305 (+ 41510us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287848832}
I20260709 12:13:17.238440 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.225797 16721 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
W20260709 12:13:17.229404 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 231324us
I20260709 12:13:17.239254 16827 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.239760 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:17.240342 12497 raft_consensus.cc:606] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:17.240782 12497 raft_consensus.cc:614] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:17.240170 16828 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.241707 16827 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.242452 16831 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.242786 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.236971 16858 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.244839 12488 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:17.246636 13255 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:17.250751 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.251693 12566 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:17.246591 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:17.246568 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.264402 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.274858 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:17.276446 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:17.277694 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120087us
I20260709 12:13:17.236802 16713 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.291041 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:17.290082 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.290272 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.292194 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.296689 12566 raft_consensus.cc:2376] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10. Re-sending same reply.
I20260709 12:13:17.297919 12570 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:17.244664 16706 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:17.303367 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2117) took 1227 ms (client timeout 1566 ms). Trace:
W20260709 12:13:17.308363 12640 rpcz_store.cc:269] 0709 12:13:16.076523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.091372 (+ 14849us) service_pool.cc:224] Handling call
0709 12:13:17.303121 (+1211749us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:17.303286 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":599914880}
I20260709 12:13:17.255121 16860 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:17.312203 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2104) took 1252 ms. Trace:
I20260709 12:13:17.312842 12639 rpcz_store.cc:276] 0709 12:13:16.059849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.060006 (+   157us) service_pool.cc:224] Handling call
0709 12:13:16.071441 (+ 11435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:16.261002 (+189561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:16.261049 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:16.261613 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:13:16.542147 (+280534us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:16.542186 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:17.114161 (+571975us) spinlock_profiling.cc:243] Waited 45.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:17.311711 (+197550us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:17.311793 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:17.311876 (+    83us) raft_consensus.cc:1685] finished
0709 12:13:17.311898 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:17.312116 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522495744,"thread_start_us":249288,"threads_started":1,"wal-append.queue_time_us":445034}
I20260709 12:13:17.274147 16831 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.316779 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.320598 12488 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 12:13:17.324544 12488 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 8
I20260709 12:13:17.324952 12488 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 12:13:17.325268 12488 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:17.326366 13255 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:13:17.327221 13255 raft_consensus.cc:606] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:17.327654 13255 raft_consensus.cc:614] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:17.339449 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.340061 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.340286 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:17.277284 16770 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:13:17.316138 16704 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:17.353188 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78422.2us
I20260709 12:13:17.317689 16865 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:17.310089 16777 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:13:17.331765 16716 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:17.303280 16713 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.356248 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:13:17.404220 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:17.410064 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.349382 16706 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:17.474033 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:17.475606 13261 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:17.468106 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 221598us
I20260709 12:13:17.481905 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:17.330798 16831 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:17.469853 16843 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.641s	user 0.004s	sys 0.004s
I20260709 12:13:17.488524 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:17.492353 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.496644 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:17.461493 16862 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:13:17.500586 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.501121 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:17.495517 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2123) took 1408 ms. Trace:
I20260709 12:13:17.501678 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 1979) took 1099 ms. Trace:
I20260709 12:13:17.506196 12641 rpcz_store.cc:276] 0709 12:13:16.401902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.797589 (+395687us) service_pool.cc:224] Handling call
0709 12:13:17.492190 (+694601us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:13:17.501566 (+  9376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24621952}
I20260709 12:13:17.505431 12638 rpcz_store.cc:276] 0709 12:13:16.086636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:17.081959 (+995323us) service_pool.cc:224] Handling call
0709 12:13:17.082303 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:17.082537 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:17.082809 (+   272us) log.cc:844] Serialized 14 byte log entry
0709 12:13:17.495080 (+412271us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:17.495140 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:17.495193 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:17.495287 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:17.495449 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9914,"prepare.queue_time_us":746,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"thread_start_us":401882,"threads_started":2}
W20260709 12:13:17.500051 16848 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.676s	user 0.001s	sys 0.007s
I20260709 12:13:17.499059 16828 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.506192 16777 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.509900 16863 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:17.540710 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 184544us
I20260709 12:13:17.526213 16716 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:17.508265 16649 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.555269 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:17.566473 12536 leader_election.cc:400] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:17.567166 12536 leader_election.cc:403] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:17.573436 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.574923 16764 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:17.576399 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:17.577421 16529 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:17.584399 13261 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:17.586055 16780 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:17.586663 16780 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:17.586818 16861 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.287s	user 0.002s	sys 0.000s
I20260709 12:13:17.587735 16780 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.594236 16827 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:17.594491 16686 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:17.547863 16877 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:13:17.596899 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2120) took 1515 ms. Trace:
I20260709 12:13:17.601204 16557 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:17.543085 16777 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.601938 16557 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:17.604853 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:17.548758 16863 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.609385 16777 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.610550 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.611377 12566 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:17.623073 16768 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:17.595563 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2247) took 2072 ms. Trace:
I20260709 12:13:17.732566 12568 rpcz_store.cc:276] 0709 12:13:15.523395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.850135 (+326740us) service_pool.cc:224] Handling call
0709 12:13:15.850468 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:15.960549 (+110081us) spinlock_profiling.cc:243] Waited 4.44 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:16.183299 (+222750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:16.183350 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:16.184049 (+   699us) log.cc:844] Serialized 26 byte log entry
0709 12:13:16.529953 (+345904us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:13:16.530018 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:16.530128 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:17.215200 (+685072us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:17.595292 (+380092us) raft_consensus.cc:1685] finished
0709 12:13:17.595319 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:17.595493 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9658,"raft.run_cpu_time_us":2441,"raft.run_wall_time_us":100344,"spinlock_wait_cycles":67377280,"thread_start_us":623721,"threads_started":2,"wal-append.queue_time_us":534163}
I20260709 12:13:17.734850 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:17.736116 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:17.739030 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:17.719158 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 376633us
I20260709 12:13:17.599225 16831 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:17.715448 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.632133 16877 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:13:17.649485 16849 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.838s	user 0.000s	sys 0.005s
I20260709 12:13:17.658342 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:17.745865 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:17.601125 12646 rpcz_store.cc:276] 0709 12:13:16.081238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.140704 (+ 59466us) service_pool.cc:224] Handling call
0709 12:13:16.158565 (+ 17861us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:16.159141 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:16.159184 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:16.167533 (+  8349us) log.cc:844] Serialized 52 byte log entry
0709 12:13:16.690372 (+522839us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:16.721316 (+ 30944us) log.cc:844] Serialized 14 byte log entry
0709 12:13:16.721692 (+   376us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:16.721795 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:17.075589 (+353794us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:13:17.235537 (+159948us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:17.596606 (+361069us) raft_consensus.cc:1685] finished
0709 12:13:17.596639 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:17.596829 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18824,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":1858,"raft.queue_time_us":3865,"raft.run_cpu_time_us":7390,"raft.run_wall_time_us":361229,"spinlock_wait_cycles":40247424,"thread_start_us":485083,"threads_started":1,"wal-append.queue_time_us":627741}
I20260709 12:13:17.629590 16671 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:17.596959 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.747406 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:17.626770 16808 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:17.680884 16529 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.622236 16875 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:17.727491 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.767916 12566 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:17.619264 16649 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.656383 16863 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:17.633540 16858 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.603049 16557 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.488837 16860 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.659860 16713 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.539702 16828 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.636154 16827 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:13:17.552695 16847 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.728s	user 0.000s	sys 0.004s
W20260709 12:13:17.698100 16417 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.001s
W20260709 12:13:17.716915 16825 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.632s	user 0.010s	sys 0.001s
W20260709 12:13:17.730381 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:17.801414 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.802184 12569 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:13:17.802379 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:17.619966 16879 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:17.752745 16808 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:17.599009 12570 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:17.737242 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:17.740903 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:17.816509 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75693.5us
I20260709 12:13:17.753592 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:17.786886 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2113) took 1711 ms (client timeout 1886 ms). Trace:
I20260709 12:13:17.817649 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:17.745113 16877 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:17.792991 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 511) took 2202 ms. Trace:
I20260709 12:13:17.784946 16671 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:17.728566 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:17.808490 16827 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 11.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:13:17.744728 16878 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:13:17.809484 16629 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.810016 16528 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:17.810881 12572 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:17.810786 16825 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.812081 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 643) took 1880 ms. Trace:
I20260709 12:13:17.812412 16626 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:17.813232 16828 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.850929 16865 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:17.815541 12567 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:17.817512 12644 rpcz_store.cc:269] 0709 12:13:16.075147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.086356 (+ 11209us) service_pool.cc:224] Handling call
0709 12:13:17.785744 (+1699388us) spinlock_profiling.cc:243] Waited 41.1 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:17.786791 (+  1047us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94560640}
I20260709 12:13:17.841143 12571 rpcz_store.cc:276] 0709 12:13:15.931533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.235761 (+304228us) service_pool.cc:224] Handling call
0709 12:13:16.236119 (+   358us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:16.255477 (+ 19358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:16.255518 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:16.263580 (+  8062us) log.cc:844] Serialized 52 byte log entry
0709 12:13:16.700450 (+436870us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:16.827038 (+126588us) log.cc:844] Serialized 14 byte log entry
0709 12:13:16.829784 (+  2746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:16.829883 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:17.493499 (+663616us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:17.811743 (+318244us) raft_consensus.cc:1685] finished
0709 12:13:17.811861 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:17.812020 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25486,"prepare.queue_time_us":74856,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":21522,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":1693,"spinlock_wait_cycles":47238400,"thread_start_us":480583,"threads_started":2,"wal-append.queue_time_us":516080}
I20260709 12:13:17.821554 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.822233 16713 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:17.856566 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.007s	sys 0.007s
W20260709 12:13:17.826325 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97849.5us
I20260709 12:13:17.814901 16764 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.753803 16551 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.061s
I20260709 12:13:17.835157 12570 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:13:17.860502 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.007s	sys 0.008s
I20260709 12:13:17.746016 16831 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:17.861110 16831 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.864512 16831 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.781065 16529 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.820188 13259 rpcz_store.cc:276] 0709 12:13:15.590828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.598922 (+  8094us) service_pool.cc:224] Handling call
0709 12:13:17.218877 (+1619955us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:17.219859 (+   982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:17.792727 (+572868us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b540058d750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:17.792913 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3724139264}
I20260709 12:13:17.868314 16529 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.833914 16761 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:17.825258 16671 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:17.832254 16557 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:17.823438 16877 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.804891 16417 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.872857 16768 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:17.875211 16808 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.894586 16881 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:13:17.895808 16704 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.537s	user 0.010s	sys 0.001s
I20260709 12:13:17.901995 16768 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:17.896162 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:17.903373 16768 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.881803 16761 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:17.905453 16761 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:17.906530 12492 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:17.853883 16825 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.903849 16716 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:17.908209 16716 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:17.908347 16775 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.908730 16716 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:17.910341 16704 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:13:17.895910 16879 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:17.908987 16775 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:17.888027 16817 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.957s	user 0.014s	sys 0.043s
W20260709 12:13:17.914095 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.915059 16586 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.915612 16808 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:17.915261 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 656) took 2413 ms. Trace:
I20260709 12:13:17.915927 16586 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:17.916388 13258 rpcz_store.cc:276] 0709 12:13:15.502225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.503078 (+   853us) service_pool.cc:224] Handling call
0709 12:13:15.503419 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:15.633081 (+129662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:15.633123 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:15.647001 (+ 13878us) log.cc:844] Serialized 26 byte log entry
0709 12:13:15.793583 (+146582us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:15.793619 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:15.793979 (+   360us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:17.784018 (+1990039us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:17.914899 (+130881us) raft_consensus.cc:1685] finished
0709 12:13:17.914928 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:17.915198 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":44391,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":150,"spinlock_wait_cycles":10716928,"thread_start_us":244491,"threads_started":2,"wal-append.queue_time_us":161584}
W20260709 12:13:17.918210 16842 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.179s	user 0.004s	sys 0.009s
I20260709 12:13:17.918743 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.920095 12492 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:17.919991 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:13:17.920730 12644 raft_consensus.cc:606] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:17.921131 12644 raft_consensus.cc:614] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:17.922225 16877 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:17.871244 16713 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:17.927652 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.011s	sys 0.014s
W20260709 12:13:17.937695 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.011s	sys 0.014s
W20260709 12:13:17.931427 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:17.941061 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.892174 16777 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:17.885125 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 384041us
I20260709 12:13:17.945523 12644 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:17.874257 16706 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:17.848462 16629 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:17.956211 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.832733 16875 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.094s
I20260709 12:13:17.852629 16865 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:17.949678 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.973696 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:17.979537 12492 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:17.971271 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:17.983734 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:17.985886 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:17.910352 16369 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:17.857275 16626 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.990818 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.980852 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:17.946316 16859 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.001s	sys 0.016s
I20260709 12:13:17.915648 16775 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:17.987720 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:17.998247 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:17.988232 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:17.940723 16713 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:18.008704 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.032428 12642 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:18.033953 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.034780 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:18.036821 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:17.883988 16764 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.047173 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:18.048365 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:18.049880 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:18.050626 12638 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.051816 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.052634 12646 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:18.053210 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:18.038031 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50780.6us
I20260709 12:13:18.040364 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.055814 12495 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:17.943734 16879 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:17.968374 16586 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:17.967387 16862 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.082515 16862 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:18.020409 16859 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.001s	sys 0.017s
W20260709 12:13:18.087822 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.058972 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.090147 12642 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:18.091225 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.057729 16883 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:13:17.998217 16634 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:13:18.065954 16680 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:18.066391 16873 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:18.067126 16529 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:18.067620 16828 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:17.971267 16706 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.069818 16686 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:18.070806 16869 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.571s	user 0.000s	sys 0.003s
I20260709 12:13:18.038781 16831 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:17.990883 16557 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:18.075407 16626 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.075887 16875 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:18.079273 13258 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:13:17.977412 16777 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:17.992758 16825 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.017554 16369 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:18.019699 16882 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:18.162189 16828 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:18.163048 16882 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.080616 16879 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.046748 16370 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:18.167922 16879 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:18.081537 16880 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:18.059378 16827 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:13:18.169169 16827 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 11.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:13:18.092183 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:18.170622 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:18.093899 12488 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.024s	sys 0.016s
I20260709 12:13:18.093818 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.113921 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:18.172017 12488 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.024s	sys 0.016s
I20260709 12:13:18.113106 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:18.038355 16417 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:18.195269 12641 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:18.119541 16780 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:18.115427 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.085335 16862 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:18.122149 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.211542 12646 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:13:18.176164 16711 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.006s	sys 0.021s
I20260709 12:13:18.159127 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:18.159258 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 99620.3us
I20260709 12:13:18.059454 16649 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
W20260709 12:13:18.179752 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.038424 16551 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:18.105098 12640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.008s	sys 0.034s
I20260709 12:13:18.196521 12642 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:13:18.115028 16680 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.107533 16770 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:13:18.062384 16716 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.168447 16629 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.214089 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:18.264319 12639 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:18.142560 16831 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.265163 16831 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.214573 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:18.130908 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.215871 16764 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.147749 16686 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.217583 12641 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:13:18.153964 16557 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.274809 16686 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:18.275501 12490 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:18.275616 16557 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.204613 16634 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.027s
W20260709 12:13:18.227200 16872 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.704s	user 0.000s	sys 0.003s
I20260709 12:13:18.227952 16830 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:18.230203 16863 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:18.215132 16780 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.231972 12645 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:18.197357 16762 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:18.254050 12640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.893s	user 0.009s	sys 0.035s
I20260709 12:13:18.204319 12495 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:18.246536 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87501.1us
I20260709 12:13:18.295261 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:18.295648 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.152880 16875 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.259322 16882 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:18.260062 16649 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:18.261111 16877 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:18.261833 16897 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:18.262387 16671 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.272684 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.280144 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2485) took 1156 ms. Trace:
W20260709 12:13:18.329375 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 498099us
I20260709 12:13:18.278729 13259 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:13:18.280125 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:18.289243 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.305190 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:18.331111 12491 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:18.302598 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:18.292505 12394 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 8 to 12, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 12 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:13:18.246034 16711 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.006s	sys 0.021s
I20260709 12:13:18.329384 12494 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:13:18.333333 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53323us
I20260709 12:13:18.334357 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:18.317783 16764 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:18.293092 16863 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:18.310643 16649 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:18.320370 16529 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.320990 16828 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:18.321933 16827 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:18.253996 16551 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.198638 16825 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.323513 12645 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:13:18.329972 12490 rpcz_store.cc:276] 0709 12:13:17.123686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:17.123818 (+   132us) service_pool.cc:224] Handling call
0709 12:13:18.279970 (+1156152us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:18.280081 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2645636480}
I20260709 12:13:18.286474 12638 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:18.246031 16868 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:18.306269 16875 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:18.335597 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.342831 16883 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:18.289736 16830 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:18.319370 12639 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:18.339591 16887 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:18.328776 16897 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.430110 16897 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:18.335693 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:18.342672 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 435647us
I20260709 12:13:18.337703 16764 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.434156 16764 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:18.434564 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98951.6us
I20260709 12:13:18.435708 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:18.436769 16370 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:18.436964 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:18.437232 16370 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.437852 16879 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:18.438189 16370 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.345994 16880 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:18.439561 16880 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:18.440599 16880 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.351517 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.353148 16671 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.353492 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.442826 12496 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.348795 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:18.355144 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.362676 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:18.444902 13258 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:18.367128 16828 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.367180 16529 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.367744 13259 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:13:18.450033 13259 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 7
I20260709 12:13:18.373111 12645 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:18.450615 12645 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:13:18.450978 12645 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:18.376298 16827 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.376701 12641 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:18.376914 16761 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.377476 16868 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:18.384824 12642 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:18.392289 12644 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:13:18.349570 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 310232us
I20260709 12:13:18.407265 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:18.423959 16763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.593s	user 0.043s	sys 0.037s
I20260709 12:13:18.421346 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.438304 16879 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.310277 16762 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.419667 16775 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:18.354187 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:18.448428 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.378548 12394 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.450347 13259 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:13:18.521166 13259 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:18.521829 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.452821 12638 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:13:18.522842 12638 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:18.523135 12638 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:13:18.461982 12489 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:18.419389 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.465459 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:18.524072 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.766s (SENT) [suppressed 18 similar messages]
I20260709 12:13:18.523365 12638 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:18.524427 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59474.8us
I20260709 12:13:18.468007 12491 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:18.329754 16369 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.506738 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:18.508173 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 159490us
I20260709 12:13:18.538410 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:18.509191 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65548.1us
I20260709 12:13:18.349570 16716 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.525251 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:18.511451 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51133.9us
I20260709 12:13:18.540587 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:18.517278 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 163219us
W20260709 12:13:18.516305 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211182us
I20260709 12:13:18.531528 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.454313 16884 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:18.535300 12394 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 10, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 10 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.549671 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:18.555295 12491 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:13:18.556473 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 224306us
I20260709 12:13:18.561699 12496 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:18.505234 16763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.676s	user 0.044s	sys 0.037s
I20260709 12:13:18.543951 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:18.544024 12490 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:18.563108 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.518759 16879 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:18.554757 12394 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 12, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 12 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.569430 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.572451 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:18.570043 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.536953 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.574057 12491 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9.
I20260709 12:13:18.573377 12394 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.579671 12394 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.583881 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:18.575590 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 14. Ops: [12.11-12.11]
I20260709 12:13:18.586293 12496 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:18.586127 12394 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 8 to 10, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:18.587677 12394 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 3 to 13. New cstate: current_term: 13 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.588280 12496 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260709 12:13:18.589598 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65555.3us
I20260709 12:13:18.515733 16753 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:18.515977 16882 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:18.602219 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.596837 12492 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:18.573007 12493 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:18.545650 16883 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:13:18.570745 16906 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:13:18.557780 16877 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:13:18.668279 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124435us
W20260709 12:13:18.642506 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70180.8us
I20260709 12:13:18.651588 12489 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:18.654872 16878 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:18.657594 12496 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:18.657431 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:18.663901 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80149.3us
I20260709 12:13:18.676393 12494 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:18.628315 12495 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:18.569307 16634 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:18.534577 16775 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:18.670967 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112115us
I20260709 12:13:18.690371 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:18.672700 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:18.692718 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.641098 16883 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.611752 16884 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:18.676937 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:18.669412 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:18.678166 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 221) took 38989 ms (client timeout 15000 ms). Trace:
W20260709 12:13:18.678653 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.571152 16777 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:18.510700 16886 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.119s
W20260709 12:13:18.714303 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44992.7us
I20260709 12:13:18.616959 16558 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20260709 12:13:18.563998 16865 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.681435 16634 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.704324 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.644052 16903 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:18.702544 16899 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:18.715298 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:18.694288 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.743628 12489 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:18.533159 16649 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.675333 16878 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.515226 16863 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:13:18.697793 12491 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:18.725978 16894 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.361s	user 0.000s	sys 0.001s
I20260709 12:13:18.726600 16827 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:18.616175 12394 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 2 to 7, term changed from 7 to 8, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129), 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:13:18.739179 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 199604us
I20260709 12:13:18.787474 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.788165 12491 raft_consensus.cc:2376] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9. Re-sending same reply.
I20260709 12:13:18.737375 16634 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.688433 16918 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:18.705763 16883 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:18.703080 16680 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.705889 16780 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.644901 16882 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.741569 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.799466 12497 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
W20260709 12:13:18.765969 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.721995 16586 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:18.781239 16878 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:18.781219 16903 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:18.776894 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:18.729688 16884 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:18.704666 16711 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:18.818483 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.788372 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.819921 16780 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:18.811473 16827 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:18.820525 16780 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:18.820667 16827 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.821448 16780 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.823292 16827 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.823889 16827 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.824369 16827 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:18.824908 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.790796 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:18.775750 16898 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.235s	user 0.002s	sys 0.000s
I20260709 12:13:18.788355 12394 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 10, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 10 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.827719 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:18.828280 16886 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:18.828532 12563 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:18.828992 12394 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 4 to 12. New cstate: current_term: 12 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:18.829939 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:18.830561 12568 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:13:18.775610 16905 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 12:13:18.788174 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:18.832327 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44256us
I20260709 12:13:18.833069 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:18.795830 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.742856 16899 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.794821 16918 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.797747 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:18.780061 16713 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.721s	user 0.014s	sys 0.000s
W20260709 12:13:18.748971 16895 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.286s	user 0.001s	sys 0.001s
I20260709 12:13:18.622816 16868 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:18.706952 12392 rpcz_store.cc:269] 0709 12:12:39.642439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:39.650268 (+  7829us) service_pool.cc:224] Handling call
0709 12:13:18.631327 (+38981059us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:13.777058 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:13.777941 (+   883us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:13.778008 (+    67us) write_op.cc:435] Acquired schema lock
0709 12:13:13.778027 (+    19us) tablet.cc:662] Decoding operations
0709 12:13:13.944269 (+166242us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:13.946171 (+  1902us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:13.950584 (+  4413us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:13:14.163967 (+213383us) tablet.cc:701] Row locks acquired
0709 12:13:14.164041 (+    74us) write_op.cc:260] PREPARE: finished
0709 12:13:14.164379 (+   338us) write_op.cc:270] Start()
0709 12:13:14.164601 (+   222us) write_op.cc:276] Timestamp: P: 1783599194164209 usec, L: 0
0709 12:13:14.164653 (+    52us) op_driver.cc:348] REPLICATION: starting
0709 12:13:14.165913 (+  1260us) log.cc:844] Serialized 7506 byte log entry
0709 12:13:15.237545 (+1071632us) op_driver.cc:464] REPLICATION: finished
0709 12:13:15.479451 (+241906us) write_op.cc:301] APPLY: starting
0709 12:13:15.491461 (+ 12010us) tablet.cc:1367] starting BulkCheckPresence
0709 12:13:15.558560 (+ 67099us) spinlock_profiling.cc:243] Waited 5.13 ms on lock 0x7b18000b1d60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:15.699340 (+140780us) tablet.cc:1370] finished BulkCheckPresence
0709 12:13:15.699392 (+    52us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:13:15.958987 (+259595us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:13:15.970088 (+ 11101us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0709 12:13:15.970158 (+    70us) write_op.cc:312] APPLY: finished
0709 12:13:15.970694 (+   536us) log.cc:844] Serialized 172 byte log entry
0709 12:13:15.970917 (+   223us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:15.971120 (+   203us) write_op.cc:454] Released schema lock
0709 12:13:15.971193 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 12:13:15.974951 (+  3758us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:15.975005 (+    54us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":920282,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":187858,"apply.run_cpu_time_us":33298,"apply.run_wall_time_us":511868,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83471,"num_ops":19,"prepare.queue_time_us":250366,"prepare.run_cpu_time_us":20801,"prepare.run_wall_time_us":502749,"raft.queue_time_us":384569,"raft.run_cpu_time_us":64805,"raft.run_wall_time_us":290004,"replication_time_us":1009594,"spinlock_wait_cycles":11807104,"thread_start_us":518578,"threads_started":4,"wal-append.queue_time_us":144684,"wal-append.run_cpu_time_us":24456,"wal-append.run_wall_time_us":2659976}]]}
W20260709 12:13:18.810124 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137487us
I20260709 12:13:18.748030 16891 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:13:18.759547 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.809300 16926 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:18.822183 16909 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
W20260709 12:13:18.867030 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50162.1us
I20260709 12:13:18.831367 16827 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:18.844152 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:18.895411 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.844710 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:18.896924 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52326.7us
I20260709 12:13:18.899125 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.873485 16911 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:18.748056 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:18.591621 16830 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.803699 16825 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:18.882481 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:18.802183 16913 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:13:18.882936 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:18.921887 12536 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:13:18.830844 12394 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 11, now has a pending config: VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129), VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 11 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:18.884395 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.813095 16649 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:18.793715 16912 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.090s
I20260709 12:13:18.807148 16920 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:18.841461 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.873293 16629 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:13:18.900884 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:18.933180 12394 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 8 to 11, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 11 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:18.737860 16865 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.905043 16868 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:18.887687 16894 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:13:18.873375 16899 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.904246 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:18.907382 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 159441us
I20260709 12:13:18.906662 16900 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:18.899935 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:18.874306 16918 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.875273 16924 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:18.911587 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:18.963920 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66225.7us
I20260709 12:13:18.881148 16529 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:18.915408 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.889256 16761 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:18.807226 16551 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.925055 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:18.933133 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:18.904193 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:18.882676 16713 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:18.906061 16891 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:18.945290 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62603.8us
I20260709 12:13:18.953198 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:18.925877 16686 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:18.937557 16893 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
W20260709 12:13:18.957409 16910 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:13:18.897617 16926 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.002880 16929 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:19.005478 16926 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:18.958374 16831 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.693s	user 0.012s	sys 0.000s
I20260709 12:13:19.006892 16831 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.971119 16924 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.948902 16827 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:18.940028 16629 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:18.947467 16830 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:18.916859 16887 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:13:18.937873 16921 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 12:13:18.920228 16711 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:18.965132 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:18.955972 16900 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:19.009529 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2492) took 1666 ms. Trace:
I20260709 12:13:18.968927 12642 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:18.914566 16626 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:13:18.979148 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:18.919976 16680 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:13:18.990815 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:18.991557 16713 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:18.992206 12637 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
W20260709 12:13:18.994664 16764 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.011s
I20260709 12:13:18.924952 16825 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 12:13:18.952399 16912 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 11.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:13:19.018419 16908 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:13:18.966500 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:19.021390 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.019258 12488 rpcz_store.cc:276] 0709 12:13:17.344383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:17.345111 (+   728us) service_pool.cc:224] Handling call
0709 12:13:17.345530 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:17.345829 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:18.172551 (+826722us) log.cc:844] Serialized 14 byte log entry
0709 12:13:19.008643 (+836092us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b180043a260. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:13:19.008863 (+   220us) log.cc:844] Serialized 14 byte log entry
0709 12:13:19.009133 (+   270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:19.009192 (+    59us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:19.009237 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:19.009312 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:19.009460 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78410,"prepare.queue_time_us":916857,"prepare.run_cpu_time_us":202,"prepare.run_wall_time_us":15604,"raft.queue_time_us":109119,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":52955,"spinlock_wait_cycles":50071168,"thread_start_us":1154181,"threads_started":3,"wal-append.queue_time_us":441891}
I20260709 12:13:19.009385 16831 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.028870 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.029593 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:18.998600 16586 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.028463 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.014050 16882 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.039094 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.041320 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.041496 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.056810 13259 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:13:19.012559 16629 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:19.060359 16828 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:18.962535 16884 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.057765 16586 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.066639 16369 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:19.068248 16370 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.629s	user 0.011s	sys 0.000s
I20260709 12:13:19.068854 16370 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:19.010562 16915 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:13:19.027957 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.047559 12563 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
W20260709 12:13:19.037310 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58249.5us
I20260709 12:13:19.018708 16925 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:19.107853 16925 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:19.108361 16925 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:19.108873 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2140) took 1095 ms. Trace:
I20260709 12:13:19.049963 12569 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:19.012099 16865 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.052335 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 645) took 2709 ms. Trace:
I20260709 12:13:19.117230 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.118438 12563 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:19.117411 12564 rpcz_store.cc:276] 0709 12:13:16.343289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:16.799343 (+456054us) service_pool.cc:224] Handling call
0709 12:13:16.810665 (+ 11322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:17.215583 (+404918us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:17.230548 (+ 14965us) log.cc:844] Serialized 14 byte log entry
0709 12:13:18.235030 (+1004482us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b18002194c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:13:18.576350 (+341320us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:18.576418 (+    68us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:18.576468 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:19.052040 (+475572us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:19.052117 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:19.052268 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13149,"prepare.queue_time_us":14846,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":4610,"raft.queue_time_us":227564,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":38481,"spinlock_wait_cycles":938793472,"thread_start_us":340950,"threads_started":2,"wal-append.queue_time_us":1603752}
I20260709 12:13:18.997076 16935 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:19.010397 16924 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:19.042593 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125236us
I20260709 12:13:19.129338 16940 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:19.129427 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:19.130219 16940 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.131184 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:19.044663 16711 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.132462 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:19.132687 16940 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.133136 12639 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:19.133790 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:19.109337 12640 rpcz_store.cc:276] 0709 12:13:18.013655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:18.308362 (+294707us) service_pool.cc:224] Handling call
0709 12:13:18.308835 (+   473us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:18.329563 (+ 20728us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:18.329601 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:18.344297 (+ 14696us) log.cc:844] Serialized 52 byte log entry
0709 12:13:18.553446 (+209149us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:18.553507 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:18.553613 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:19.108624 (+555011us) raft_consensus.cc:1685] finished
0709 12:13:19.108646 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:19.108803 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4945,"raft.queue_time_us":5157,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"thread_start_us":166831,"threads_started":1,"wal-append.queue_time_us":414945}
I20260709 12:13:19.135294 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:19.036913 16551 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:13:19.047039 16926 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:19.042964 16764 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:18.941989 16903 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.010049 16827 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.154776 16827 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:19.072124 16891 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.156744 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.074316 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.024907 16900 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:19.160163 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:18.983919 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:19.162509 12567 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:19.163172 12570 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:19.163378 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.163666 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.164173 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.164428 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:19.164855 12570 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:19.164572 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.076131 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:19.077493 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.166925 13255 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:19.050838 16897 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.620s	user 0.009s	sys 0.005s
I20260709 12:13:19.167254 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.090574 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.166008 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.096168 16629 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.097506 16918 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.102218 12572 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:19.106884 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.107038 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.177690 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.139555 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.178359 12563 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:19.178629 12488 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:19.179337 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.179502 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:19.180176 12488 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:19.141089 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:19.142495 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.143218 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:19.181372 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105395us
I20260709 12:13:19.181715 12489 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:19.147258 12492 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:13:19.150135 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.127493 16626 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.127892 16935 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.168996 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.147029 16558 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.170924 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:19.171893 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.173362 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:19.173940 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.174046 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.200181 12565 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:19.200635 12497 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:13:19.165516 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.176229 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.182433 12609 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:19.182901 12569 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:19.150282 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:19.168915 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.201239 12567 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:19.184969 16731 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:19.204069 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:19.194190 16634 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.195353 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.149076 16893 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.068255 16586 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.167778 16897 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.200425 12493 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:19.179874 16926 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:19.206753 12494 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:19.185221 16865 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:19.208525 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.208997 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.209527 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:19.260893 12489 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:19.211887 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:19.213047 12572 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:19.192114 16900 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.237201 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:19.239328 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:19.201838 16935 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:19.240854 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:19.198035 16370 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.284021 12569 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:19.207396 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:19.205384 16731 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:19.225782 16711 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:19.295038 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:19.238515 16940 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:19.302282 13255 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
W20260709 12:13:19.302665 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95336.6us
I20260709 12:13:19.302933 12569 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:13:19.303714 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:19.201611 16626 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.312471 13253 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:19.303644 12569 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:13:19.313836 12569 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:13:19.212488 16558 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:19.310496 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 183799us
I20260709 12:13:19.325881 12565 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:19.314921 12569 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:19.257360 16893 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.330974 12571 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:19.298146 16711 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:19.330186 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.327373 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:19.336354 12565 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:19.312350 16370 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:19.307309 16940 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:19.309453 16827 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:19.338801 12569 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:19.342501 13254 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:19.343984 13254 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:19.347561 13254 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.331248 16944 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:19.370414 12569 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:19.372750 12569 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
W20260709 12:13:19.374302 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.343950 16940 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:19.343271 16370 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.384093 12489 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:19.339064 16711 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:19.394594 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:19.374459 12569 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:19.332108 16768 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.047s
W20260709 12:13:19.389968 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.396579 12639 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:13:19.398198 12565 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:19.400313 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.355742 16649 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.407131 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:19.450325 12639 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
W20260709 12:13:19.451290 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44297.4us
I20260709 12:13:19.452834 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:19.376434 16830 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.458338 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:19.461930 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:19.394732 16825 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.356387 16827 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:19.584787 16878 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.768s	user 0.015s	sys 0.000s
I20260709 12:13:19.344226 16731 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.389577 16370 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:19.460640 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:19.454218 16828 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:19.462391 16772 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:19.637169 16772 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:19.638343 16772 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.526840 16934 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.640682 16934 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.643441 16934 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:19.537178 16775 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.481820 16830 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.645054 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:19.645798 16830 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.645946 13260 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:19.464604 16860 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 12:13:19.648154 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:19.539264 16761 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:19.535553 16626 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:19.650040 16874 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:19.650983 16874 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:19.651572 16874 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:19.651885 16893 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:19.654928 16951 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
W20260709 12:13:19.535524 16947 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
W20260709 12:13:19.559770 16948 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
I20260709 12:13:19.560453 16529 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:19.478287 16763 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:19.672081 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.672435 16763 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.673015 16763 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:19.514004 16671 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.060s	user 0.012s	sys 0.003s
I20260709 12:13:19.453758 16649 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.512676 16629 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.569767 16764 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.378720 16926 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.567190 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:19.681450 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114394us
W20260709 12:13:19.571220 16911 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.642s	user 0.012s	sys 0.001s
I20260709 12:13:19.449218 16762 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.462749 12496 raft_consensus.cc:2376] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10. Re-sending same reply.
I20260709 12:13:19.589145 16586 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:19.643574 16875 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.194s	user 0.012s	sys 0.000s
I20260709 12:13:19.456337 16770 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:19.556016 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.650267 16731 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:19.558606 16900 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:19.601784 16879 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.971s	user 0.011s	sys 0.005s
I20260709 12:13:19.512436 16920 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.136s
I20260709 12:13:19.602449 16897 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.633333 16878 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:19.670688 16899 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.704s	user 0.011s	sys 0.000s
I20260709 12:13:19.679649 12563 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:13:19.682540 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:19.678795 16558 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.629915 16825 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.705920 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:19.698318 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 165834us
I20260709 12:13:19.700950 13260 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:19.693614 16911 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:19.690932 16459 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:19.663928 16828 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.672188 16529 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.721542 12566 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
W20260709 12:13:19.714911 16901 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.013s	sys 0.019s
I20260709 12:13:19.695255 16586 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.707710 16671 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.709587 16860 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:19.716732 16459 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:19.730033 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 269485us
I20260709 12:13:19.717701 16900 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:19.704977 16879 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.698176 16875 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:19.729477 16901 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.000s	user 0.013s	sys 0.020s
I20260709 12:13:19.739349 16459 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:19.740067 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:19.748193 16949 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 12:13:19.734965 16860 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:19.712046 16899 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.743294 16900 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:19.739154 16671 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:19.823733 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117887us
W20260709 12:13:19.841781 16956 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.241s	user 0.006s	sys 0.000s
I20260709 12:13:19.842151 12643 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:19.748652 16963 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:13:19.842854 16768 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.843750 16879 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.902951 16586 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:19.903821 12569 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:19.842891 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:19.906694 12563 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:19.847625 16957 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.265s	user 0.000s	sys 0.006s
I20260709 12:13:19.848142 16944 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:19.849051 16878 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.850009 16649 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:19.850800 16883 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.038s	user 0.000s	sys 0.027s
I20260709 12:13:19.909894 16883 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.912386 16883 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.852021 16828 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:19.853202 16925 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.744s	user 0.007s	sys 0.011s
I20260709 12:13:19.853541 16529 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:19.852524 16935 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.551s	user 0.004s	sys 0.014s
I20260709 12:13:19.853767 16874 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.914129 16935 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.915771 16874 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:19.854712 16369 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.787s	user 0.011s	sys 0.011s
W20260709 12:13:19.854405 16958 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.171s	user 0.000s	sys 0.005s
I20260709 12:13:19.856806 16827 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:19.857110 16950 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.255s	user 0.000s	sys 0.006s
I20260709 12:13:19.896487 16711 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:19.819391 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103833us
W20260709 12:13:19.864801 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:19.919420 12638 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:19.850139 16825 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.849078 16875 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.873306 16686 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:19.920199 16925 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:19.874420 16960 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.168s	user 0.001s	sys 0.004s
I20260709 12:13:19.894920 16830 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:19.886489 16860 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:19.922330 13259 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:19.922762 16925 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:19.901041 16370 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:19.930217 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 190245us
I20260709 12:13:19.901427 16768 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.917698 12642 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:19.917232 16935 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.920701 16686 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:19.941535 16921 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:19.942088 16921 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:19.943562 16768 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.943497 16921 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.944799 16921 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:19.900204 16899 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:19.931519 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:19.954234 12563 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:19.888543 16877 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.947890 16863 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:19.949860 16900 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:19.951659 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 496) took 2007 ms. Trace:
I20260709 12:13:19.952132 16968 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:19.921172 16711 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.929653 16875 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.937832 16911 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:19.958577 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:19.961319 12637 rpcz_store.cc:276] 0709 12:13:17.944484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:17.945830 (+  1346us) service_pool.cc:224] Handling call
0709 12:13:17.946140 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:19.024678 (+1078538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:19.951388 (+926710us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:19.951590 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2405155712}
W20260709 12:13:19.966858 16940 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.565s	user 0.005s	sys 0.007s
I20260709 12:13:19.967748 16634 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:19.956691 16877 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.980181 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:19.975905 16911 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:19.979910 16925 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:20.008293 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:20.009835 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74533.7us
I20260709 12:13:20.014057 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:20.014906 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:20.019179 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:19.977999 16875 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.999342 16825 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:19.973718 16968 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:19.979224 16940 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.023360 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.010337 16911 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:20.032099 16968 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.015774 16827 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:20.020735 16529 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:20.006513 16860 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.037760 16875 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.024821 16828 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:20.023959 16825 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.043452 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.044551 12563 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:20.027204 16649 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:20.046445 16825 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:20.064484 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50967.2us
I20260709 12:13:20.068405 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.071723 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:20.072050 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.073577 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.074371 12563 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:20.082058 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.082741 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:20.040205 16772 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:13:20.060304 16761 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:20.081028 16925 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:20.093600 16982 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:20.088462 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:20.068813 16370 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.050751 16649 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.083528 16529 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:13:20.136894 16955 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.776s	user 0.005s	sys 0.000s
I20260709 12:13:20.089699 16828 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:20.135878 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2155) took 1238 ms. Trace:
I20260709 12:13:20.091784 16860 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.138671 12638 rpcz_store.cc:276] 0709 12:13:18.897697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:18.966223 (+ 68526us) service_pool.cc:224] Handling call
0709 12:13:18.998011 (+ 31788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:19.919842 (+921831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:20.135552 (+215710us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:20.135794 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118524288}
I20260709 12:13:20.123075 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:20.097035 16981 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:20.105665 16968 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:20.148304 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2350) took 1163 ms. Trace:
W20260709 12:13:20.149519 16978 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.064s	user 0.002s	sys 0.000s
I20260709 12:13:20.150192 16936 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:20.150856 16936 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.149125 12567 rpcz_store.cc:276] 0709 12:13:18.984493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.203550 (+219057us) service_pool.cc:224] Handling call
0709 12:13:19.203952 (+   402us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:19.204378 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:19.204405 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:19.205058 (+   653us) log.cc:844] Serialized 52 byte log entry
0709 12:13:19.299937 (+ 94879us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:19.561327 (+261390us) log.cc:844] Serialized 14 byte log entry
0709 12:13:19.561706 (+   379us) log.cc:844] Serialized 14 byte log entry
0709 12:13:19.684512 (+122806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:19.684593 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:20.134041 (+449448us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:20.148002 (+ 13961us) raft_consensus.cc:1685] finished
0709 12:13:20.148031 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:20.148237 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19448,"prepare.queue_time_us":555396,"prepare.run_cpu_time_us":447,"prepare.run_wall_time_us":512,"raft.queue_time_us":112174,"raft.run_cpu_time_us":480,"raft.run_wall_time_us":31616,"spinlock_wait_cycles":14280064,"thread_start_us":418714,"threads_started":3,"wal-append.queue_time_us":146261}
I20260709 12:13:20.155119 16982 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:20.155593 16982 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:20.156212 16763 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.156726 16982 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.159219 16828 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:20.159052 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53688.9us
I20260709 12:13:20.161767 16649 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.162091 16925 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:20.164989 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:20.166307 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:20.169055 16370 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:20.169909 16912 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:20.172802 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2154) took 1278 ms. Trace:
I20260709 12:13:20.175055 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:20.175388 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:20.177069 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 528) took 1096 ms. Trace:
I20260709 12:13:20.177640 12568 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:13:20.173310 12643 rpcz_store.cc:276] 0709 12:13:18.894772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:18.895025 (+   253us) service_pool.cc:224] Handling call
0709 12:13:20.172552 (+1277527us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:20.172727 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2008464256}
I20260709 12:13:20.177577 13253 rpcz_store.cc:276] 0709 12:13:19.080989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.090218 (+  9229us) service_pool.cc:224] Handling call
0709 12:13:20.176803 (+1086585us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:20.176983 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":331074304}
I20260709 12:13:20.185596 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:20.187202 16686 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.188778 16940 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.189309 16918 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.182464 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.192199 16982 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:20.195520 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:20.199846 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:20.200332 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:20.204572 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:20.103646 16761 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.207729 16877 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:20.210362 16976 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.177s	user 0.003s	sys 0.000s
I20260709 12:13:20.211776 16761 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.212471 16761 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:20.159744 16828 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:20.213583 16828 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:20.218263 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.223496 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:20.224510 13258 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.227703 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.228673 12640 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 7: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:20.229838 16686 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.231966 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.232724 12640 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:20.234184 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:20.234730 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:20.235391 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.235543 13255 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:20.236032 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:20.249112 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2344) took 1284 ms. Trace:
I20260709 12:13:20.249677 12568 rpcz_store.cc:276] 0709 12:13:18.964658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:18.964804 (+   146us) service_pool.cc:224] Handling call
0709 12:13:20.248807 (+1284003us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:20.249021 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2663684992}
I20260709 12:13:20.195969 16981 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:20.255159 16981 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.256055 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2284) took 1104 ms. Trace:
I20260709 12:13:20.259085 16763 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.216063 16459 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.263083 16459 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:20.264607 16970 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.420s	user 0.000s	sys 0.006s
I20260709 12:13:20.267231 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:20.268234 16370 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:20.268831 16370 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:20.269205 16370 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:20.256562 12564 rpcz_store.cc:276] 0709 12:13:19.151764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.678926 (+527162us) service_pool.cc:224] Handling call
0709 12:13:19.683536 (+  4610us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:19.724139 (+ 40603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:19.724188 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:19.727023 (+  2835us) log.cc:844] Serialized 26 byte log entry
0709 12:13:19.981160 (+254137us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:19.981228 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:20.193200 (+211972us) spinlock_profiling.cc:243] Waited 952 us on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:20.255685 (+ 62485us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:20.255745 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:20.255800 (+    55us) raft_consensus.cc:1685] finished
0709 12:13:20.255816 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:20.255992 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3170,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":4875,"spinlock_wait_cycles":94376064,"thread_start_us":138953,"threads_started":1,"wal-append.queue_time_us":272108}
I20260709 12:13:20.277037 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:20.286536 16975 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.274s	user 0.000s	sys 0.003s
I20260709 12:13:20.288122 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2004) took 1164 ms. Trace:
I20260709 12:13:20.288815 12642 rpcz_store.cc:276] 0709 12:13:19.123286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.123902 (+   616us) service_pool.cc:224] Handling call
0709 12:13:20.287799 (+1163897us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:20.288015 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1801008512}
I20260709 12:13:20.290347 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 657) took 1463 ms. Trace:
I20260709 12:13:20.290830 12566 rpcz_store.cc:276] 0709 12:13:18.826451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:18.829231 (+  2780us) service_pool.cc:224] Handling call
0709 12:13:18.829610 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:19.722053 (+892443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:20.290085 (+568032us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:20.290277 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2050700544}
I20260709 12:13:20.295006 12568 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:20.298892 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.302909 12566 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.310633 13258 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:20.271736 16926 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:20.318984 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:20.275738 16880 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:20.274820 16911 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.269564 16825 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.300501 16877 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:13:20.342497 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.344532 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.345391 12640 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:20.349470 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.351877 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:20.353811 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.354823 12638 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:20.365900 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.366357 12566 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:20.367367 12642 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:20.341182 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.368816 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:20.369910 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.336457 16716 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:20.373747 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.377971 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:20.333592 16880 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:20.423730 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45838.6us
I20260709 12:13:20.372105 16830 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:20.374692 16984 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:20.377619 16918 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:20.443089 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2352) took 1458 ms. Trace:
I20260709 12:13:20.421936 16716 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:20.447829 16716 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:20.452749 16979 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.419s	user 0.000s	sys 0.002s
I20260709 12:13:20.454272 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2283) took 1302 ms. Trace:
I20260709 12:13:20.454722 12565 rpcz_store.cc:276] 0709 12:13:19.151525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.530393 (+378868us) service_pool.cc:224] Handling call
0709 12:13:19.579282 (+ 48889us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:19.730558 (+151276us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:19.730611 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:19.742074 (+ 11463us) log.cc:844] Serialized 26 byte log entry
0709 12:13:19.893924 (+151850us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:19.894004 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:19.894107 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:20.452083 (+557976us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:20.453855 (+  1772us) raft_consensus.cc:1685] finished
0709 12:13:20.453883 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:20.454195 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":66771,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":127,"spinlock_wait_cycles":4220160,"thread_start_us":168836,"threads_started":2,"wal-append.queue_time_us":232292}
I20260709 12:13:20.456794 16936 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:20.467593 13254 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:20.467782 16911 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.470810 16671 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:20.475262 16984 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:20.479377 16868 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:20.480172 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:20.480054 16868 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.483891 16880 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.483935 12565 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
W20260709 12:13:20.486720 16983 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.386s	user 0.003s	sys 0.000s
I20260709 12:13:20.491957 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.492740 12568 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:20.494277 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2346) took 1511 ms. Trace:
I20260709 12:13:20.494697 12571 rpcz_store.cc:276] 0709 12:13:18.983181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:18.983323 (+   142us) service_pool.cc:224] Handling call
0709 12:13:20.494026 (+1510703us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:20.494215 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":704637184}
I20260709 12:13:20.497118 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.497982 12569 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:20.498169 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.499027 12564 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:20.420996 16827 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:20.497364 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:20.433930 16968 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:20.501261 12566 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:20.444133 16918 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.502032 16918 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.503044 12565 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:20.473382 16825 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.509974 16880 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.510520 16880 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.513340 16880 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:20.514307 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:20.514235 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.444559 12572 rpcz_store.cc:276] 0709 12:13:18.984899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.277682 (+292783us) service_pool.cc:224] Handling call
0709 12:13:19.278030 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:19.278404 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:19.676501 (+398097us) log.cc:844] Serialized 14 byte log entry
0709 12:13:19.869898 (+193397us) log.cc:844] Serialized 14 byte log entry
0709 12:13:20.243829 (+373931us) spinlock_profiling.cc:243] Waited 6.53 ms on lock 0x7b18000f4300. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:20.244096 (+   267us) log.cc:844] Serialized 14 byte log entry
0709 12:13:20.244324 (+   228us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:20.244898 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 12:13:20.245056 (+   158us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:20.245112 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:20.245294 (+   182us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:20.264112 (+ 18818us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:13:20.442834 (+178722us) raft_consensus.cc:1685] finished
0709 12:13:20.442858 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:20.443020 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":193452,"prepare.queue_time_us":420959,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":133,"raft.queue_time_us":150702,"raft.run_cpu_time_us":617,"raft.run_wall_time_us":14159,"spinlock_wait_cycles":30695296,"thread_start_us":412002,"threads_started":2,"wal-append.queue_time_us":158643}
I20260709 12:13:20.509056 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.516199 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2506) took 1308 ms. Trace:
I20260709 12:13:20.516639 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:20.517268 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.30531e+06us
I20260709 12:13:20.517802 12565 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.518342 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:20.516662 12491 rpcz_store.cc:276] 0709 12:13:19.207570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:19.208526 (+   956us) service_pool.cc:224] Handling call
0709 12:13:19.208786 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:19.208982 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:19.213337 (+  4355us) log.cc:844] Serialized 14 byte log entry
0709 12:13:20.515603 (+1302266us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b1800304180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:20.515776 (+   173us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:20.515864 (+    88us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:20.515915 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:20.515989 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:20.516131 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28064,"prepare.queue_time_us":170,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4377,"raft.queue_time_us":1303396,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":89,"spinlock_wait_cycles":83874560}
I20260709 12:13:20.520627 12569 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:20.515007 12566 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:20.521893 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 303542us
I20260709 12:13:20.545269 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.546278 12563 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.556365 13254 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:20.557353 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 523) took 3073 ms. Trace:
I20260709 12:13:20.557751 13254 rpcz_store.cc:276] 0709 12:13:17.483797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:17.484011 (+   214us) service_pool.cc:224] Handling call
0709 12:13:17.504734 (+ 20723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:19.342372 (+1837638us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540056b1a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:13:20.467112 (+1124740us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b580086ded0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:13:20.556981 (+ 89869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:20.557275 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4928844800}
I20260709 12:13:20.560889 12564 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:20.563958 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:20.564412 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.564731 12564 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:20.565315 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:20.566566 12572 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:20.476647 16984 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:20.572037 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45144.6us
I20260709 12:13:20.574692 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:20.575874 13253 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:20.579630 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:20.580528 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:20.581787 12571 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:20.582334 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:20.584386 12565 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:20.574725 16833 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:13:20.593500 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:20.594345 12571 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:20.584439 16926 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:20.591843 16971 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:20.580269 16828 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.619697 16716 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.618548 16971 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.641230 16716 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.645983 16716 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.652549 16828 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:20.653976 16918 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.662966 16918 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.708772 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:20.710273 12570 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:20.711356 12491 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:20.712062 12491 raft_consensus.cc:614] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:20.715004 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:20.717171 12565 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:20.730340 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:20.735334 12565 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:13:20.739454 12565 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:13:20.746169 12565 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:13:20.748413 16528 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:20.750385 16529 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:20.753569 12565 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:20.762404 16528 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:20.773452 16528 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.775910 16626 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:20.790138 16529 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.794435 16828 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:20.795194 16828 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:20.795636 16828 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:20.827675 16877 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.834280 16877 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.835330 16762 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:20.829358 12572 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:20.831025 12566 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:13:20.828254 16918 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:20.844230 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 261774us
I20260709 12:13:20.844866 16918 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.847673 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:20.848064 16868 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.848325 16918 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.822464 16529 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.860841 16971 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:20.862737 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 344300us
I20260709 12:13:20.864194 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:20.866154 16551 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:20.866971 16551 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:20.867134 16899 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:20.867516 16551 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:20.870244 16899 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:20.861069 16912 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:20.871748 16557 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:20.871825 16912 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.872507 16557 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:20.872720 16768 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:20.873302 16768 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:20.876873 16971 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:20.879381 16868 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:20.880486 16877 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:20.881379 16680 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:20.886950 16557 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.874872 16768 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.890280 16768 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:20.891686 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:20.900682 12566 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:13:20.872684 16912 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:20.890473 16557 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:20.927749 12565 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:20.943269 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:20.959131 12495 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [11.9-12.10]
I20260709 12:13:20.945034 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:20.961405 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:20.968232 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:20.972718 16918 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:20.973750 16918 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.975334 16762 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:20.976989 16762 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:13:20.978417 13259 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:13:20.969651 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:20.978971 13261 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:20.995788 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:20.978425 16868 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:21.007051 16868 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:21.008672 16868 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.011193 12644 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:21.010982 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:20.989370 16918 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:20.977350 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:21.013818 12644 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:20.994586 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.021286 16713 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:21.022329 16713 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:20.986090 17005 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:13:21.027130 17006 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:21.029538 16711 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:21.030313 12491 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:20.996215 16935 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:21.031368 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.032428 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.032369 16935 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:20.972788 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:21.034902 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.037978 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.038882 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:20.996259 16886 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.039810 16918 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:21.040418 16918 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:21.040014 16886 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.040947 16918 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:21.011538 16868 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:21.023887 16713 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:20.989737 17007 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:21.036290 16935 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.065847 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:21.078246 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132111us
I20260709 12:13:21.048169 16528 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.063807 16886 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:21.083664 16370 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.803s	user 0.009s	sys 0.017s
I20260709 12:13:21.084313 16370 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.085304 16940 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.086126 16940 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.086851 16370 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.088300 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.089119 16940 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.090766 12495 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:21.092051 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.092253 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.092796 12495 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:21.093083 12489 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:21.084506 16945 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:21.100159 16528 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.101118 12488 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:21.081910 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:21.103083 16770 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.103623 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.103813 16770 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.081571 12646 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: [9.6-11.7]
I20260709 12:13:21.102464 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.079546 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.115176 12492 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:21.119449 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 7.
I20260709 12:13:21.134438 12494 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 12:13:21.138693 12563 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.584s	user 0.012s	sys 0.015s
I20260709 12:13:21.105543 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:21.145316 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50536.4us
I20260709 12:13:21.157125 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:21.105340 17007 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:21.160858 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:21.159129 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:21.161799 16828 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.149657 16626 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:21.170284 16370 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 12:13:21.177102 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:21.187611 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138111us
W20260709 12:13:21.188659 16990 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:13:21.192380 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.766s (SENT)
I20260709 12:13:21.197196 12637 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:21.198042 17006 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:21.198581 16634 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.199371 16634 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.200312 12568 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:21.201895 12497 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:21.203797 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.204850 12641 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:21.176188 12644 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.212554 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.216152 12494 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:13:21.213898 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:21.227190 16557 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:21.228332 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:21.228461 16680 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:21.229208 13254 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:21.233773 16649 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:21.239352 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.239722 16984 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:21.240459 16984 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.243010 16984 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.209893 16935 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:13:21.244077 16935 consensus_queue.cc:1243] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:13:21.246755 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 176070us
W20260709 12:13:21.247972 17010 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.206s	user 0.000s	sys 0.004s
I20260709 12:13:21.248725 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:13:21.250957 17014 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.224s	user 0.004s	sys 0.001s
I20260709 12:13:21.255069 17007 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:21.166021 16945 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.256309 16770 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:21.239490 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:21.244099 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:21.261365 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84444.4us
I20260709 12:13:21.262249 16945 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.280089 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.283601 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.284994 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.287784 17004 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:21.288333 17004 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:21.309514 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 178251us
I20260709 12:13:21.277993 16634 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:21.292335 16980 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.010s
W20260709 12:13:21.322597 13218 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.785s (SENT)
I20260709 12:13:21.324150 12495 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:21.330036 13254 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:21.331864 12488 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 8. Current term is 9. Ops: []
W20260709 12:13:21.331831 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66846.6us
I20260709 12:13:21.339633 16808 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.340518 16808 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:21.343515 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119189us
I20260709 12:13:21.345273 16711 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:21.345942 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.346416 16711 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:13:21.346920 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:21.345364 12641 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:21.361243 12494 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12.
I20260709 12:13:21.359735 12488 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:21.366637 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:21.360800 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:21.357942 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:21.357093 16901 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:21.372053 16901 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:21.372236 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:21.374161 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:21.374372 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.335134 16940 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.358340 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:21.395094 16649 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:21.395959 16649 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:21.396430 16649 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:21.396643 17019 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:13:21.399109 16926 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:21.399253 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.399729 16926 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:21.399976 12563 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:13:21.400179 16926 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:21.401080 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.357014 16808 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.401878 12497 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:21.402398 16808 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:21.402952 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.403644 12497 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:21.404609 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.405660 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.406518 13254 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:13:21.373273 16901 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.407992 16901 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:21.408255 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51519.2us
I20260709 12:13:21.409925 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:21.395251 17018 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
I20260709 12:13:21.289430 17004 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.370410 16634 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:21.356882 17002 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:21.423228 12495 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:21.415851 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.428268 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.440284 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:21.451326 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.453198 12494 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:21.453869 12494 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:21.358057 16945 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:13:21.426749 17002 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:21.390228 16940 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.455312 13253 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:21.461596 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 164560us
I20260709 12:13:21.471752 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:21.432199 16634 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:21.425464 16370 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:13:21.442684 16828 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.415984 16912 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.389272 16557 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
W20260709 12:13:21.463845 17017 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.305s	user 0.000s	sys 0.002s
I20260709 12:13:21.501480 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:21.495956 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50256.9us
W20260709 12:13:21.496670 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43717.6us
I20260709 12:13:21.475160 17004 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:21.464185 16713 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:21.462939 17000 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:21.516896 16912 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:21.519891 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78306.2us
I20260709 12:13:21.527330 16945 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:21.528991 17002 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.3, Last appended by leader: 3, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:21.505602 16551 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.006s
I20260709 12:13:21.560469 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:21.566685 16899 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:21.567294 16899 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:21.568450 16899 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.569938 12644 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:21.572590 16833 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:21.573081 16833 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:21.507026 16935 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:21.574047 16935 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:21.574527 16935 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:21.578838 12637 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:21.461818 16780 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.120s
I20260709 12:13:21.539285 17000 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:21.596318 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:21.601871 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:21.603816 16918 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.562s	user 0.011s	sys 0.002s
I20260709 12:13:21.603935 17002 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:21.604417 16918 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.544629 16770 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:21.604878 17006 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:21.605573 17006 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:21.606072 17006 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:21.610268 17021 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:13:21.611676 16899 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:21.611856 16649 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.612823 16926 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.613935 16918 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.615242 16649 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.616518 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.617375 12495 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:21.574450 16833 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:21.586622 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:21.622044 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.623507 13253 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:21.624543 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.587709 16713 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:21.618050 17000 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:21.594751 12494 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:21.630559 12489 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.513s	user 0.014s	sys 0.002s
I20260709 12:13:21.577728 16551 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.641121 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:21.639747 12489 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:21.643164 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.645161 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.646320 12492 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:21.655687 13260 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
W20260709 12:13:21.661850 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115619us
I20260709 12:13:21.632083 17006 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.666548 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:21.671216 12490 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:21.664836 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.667977 12494 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:21.666078 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.670702 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:21.672416 16634 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:21.667833 16370 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:21.663013 16940 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.670326 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:21.679319 16918 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.692246 16918 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.692512 16713 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.694135 16877 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:21.695192 16877 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:21.694912 16918 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:21.696316 16877 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:21.697996 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83211.9us
I20260709 12:13:21.670661 16833 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:21.689157 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.668334 16551 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.718024 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:21.719728 12638 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:21.720875 16981 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:21.721513 16981 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:21.721947 16981 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:21.710423 16370 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:21.723632 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:21.724751 16711 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:21.724798 16370 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.725850 16711 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:21.727238 16711 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.691874 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.729501 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.724936 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:21.757807 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87721.6us
I20260709 12:13:21.765259 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.761523 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
W20260709 12:13:21.788340 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:21.788949 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.771549 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:21.788985 12489 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 12:13:21.799566 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:21.790638 12488 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13. Re-sending same reply.
I20260709 12:13:21.712119 16940 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:13:21.824741 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:21.827471 12644 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:13:21.828274 12644 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 8
I20260709 12:13:21.828619 12644 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:13:21.828917 12644 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:21.830727 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.831692 16926 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.832168 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:21.836412 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 231954us
I20260709 12:13:21.833434 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.847527 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.849470 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.850358 12569 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:21.825953 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.772325 16918 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:21.868288 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.878044 12570 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:21.871858 16762 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:21.888850 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:21.890019 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.890681 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.891836 12567 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:21.893479 17025 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.283s	user 0.001s	sys 0.003s
I20260709 12:13:21.893780 16557 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:21.848903 16886 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:13:21.895056 16557 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:21.895823 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.896790 12491 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 7.
I20260709 12:13:21.900832 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.902572 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.904493 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:21.906621 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.907388 12566 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:21.908414 17006 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:21.909263 17006 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:21.910445 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.895823 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.911901 12642 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:21.912204 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.913024 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.908432 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:21.895713 16557 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:21.916961 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.911139 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:21.907477 17036 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:13:21.931543 12497 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:21.932521 12643 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:21.933264 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.934338 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.937708 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.938112 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.939427 12570 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:21.939937 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:21.943928 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.944885 12571 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:21.946353 12646 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
W20260709 12:13:21.958015 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47142.5us
I20260709 12:13:21.957052 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:21.963241 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:21.964555 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:21.965736 12492 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:21.914630 16935 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:21.970163 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 169083us
I20260709 12:13:21.964569 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:21.968640 12569 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 6.
I20260709 12:13:21.970456 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:21.975317 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.975988 12641 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:21.967620 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:22.003818 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:22.007166 12490 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:21.966918 16926 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:22.013350 12571 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:22.014456 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:21.932421 16971 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:13:21.984514 12642 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:21.974428 12567 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:22.017007 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:22.022925 12566 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:22.008242 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 250061us
I20260709 12:13:22.018600 12644 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:22.027819 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.029809 12569 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:22.030282 12491 raft_consensus.cc:2368] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:13:22.026775 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.039958 12489 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:22.041329 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:21.962777 16918 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:22.030092 16901 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.603s	user 0.010s	sys 0.003s
I20260709 12:13:22.061874 12570 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:22.068850 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.070051 16629 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:22.070523 17034 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:22.076864 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.078156 12569 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:22.093124 12569 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:22.048832 16716 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:22.093168 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.095458 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:22.102387 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82110.5us
I20260709 12:13:22.102777 17053 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:22.103626 17053 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:22.104056 17053 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:22.105381 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:22.106017 12563 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:22.107401 12563 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:13:22.088236 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:22.113310 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:22.114879 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:22.116797 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:22.117589 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:22.119848 12569 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260709 12:13:22.124300 17038 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:13:22.126368 17035 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:22.127198 17035 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:22.133031 16634 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:22.133585 16634 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:22.134023 16634 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:22.135152 16557 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:22.137815 16713 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:22.090626 16629 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:22.140077 16629 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.147070 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:22.147961 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.110574 16981 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:22.150142 12490 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:13:22.150208 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 261513us
I20260709 12:13:22.151366 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:22.151782 12490 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:22.155948 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:22.157064 16971 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:22.157387 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60314.9us
I20260709 12:13:22.141397 16629 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:22.148231 12564 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:22.049063 16926 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:13:22.166174 16828 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:22.146956 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:22.171962 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:22.177762 12572 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:22.122017 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.153409 12490 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:22.151440 12644 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
W20260709 12:13:22.241485 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68504.8us
I20260709 12:13:22.251168 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:22.257797 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111001us
I20260709 12:13:22.110034 17034 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:22.263521 12494 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.573s	user 0.018s	sys 0.007s
I20260709 12:13:22.120524 16370 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.149s
I20260709 12:13:22.270563 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.270349 16828 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.274621 12494 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:22.275161 12492 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:22.279227 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:22.279441 12566 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:22.279778 12493 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:22.292296 12489 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:13:22.292980 12489 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 6
I20260709 12:13:22.293215 12489 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 11.7
W20260709 12:13:22.254732 17004 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.719s	user 0.013s	sys 0.000s
I20260709 12:13:22.293504 12489 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:22.277845 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:22.299103 12569 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:22.301728 12566 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:22.310477 12495 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:22.312805 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:22.313591 12643 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:22.308323 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:22.271987 16901 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:22.317555 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:22.329550 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.313375 12563 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:13:22.332782 12563 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:13:22.333178 12563 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:22.335656 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136970us
W20260709 12:13:22.343096 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 191823us
I20260709 12:13:22.261183 16833 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:13:22.307893 16935 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:22.355594 12567 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:22.356647 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 245302us
W20260709 12:13:22.359624 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 293307us
W20260709 12:13:22.365499 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62964.9us
I20260709 12:13:22.372008 12566 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:22.378106 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101746us
I20260709 12:13:22.378144 12572 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:13:22.380045 13218 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12.
I20260709 12:13:22.380390 13218 leader_election.cc:403] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:13:22.382975 12564 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:13:22.331557 17035 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:22.313692 17007 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:22.392328 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.407946 12567 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:22.400310 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:22.335440 16716 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:22.402846 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:22.434929 12643 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:22.337152 16901 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:22.361750 17000 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.702s	user 0.008s	sys 0.005s
I20260709 12:13:22.464738 17007 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:22.381803 16417 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:22.426954 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:22.435878 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48398.6us
I20260709 12:13:22.439795 17040 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:22.439949 16557 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:22.440534 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:22.441814 17002 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.837s	user 0.015s	sys 0.000s
I20260709 12:13:22.441504 17048 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:22.400070 16808 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12.
I20260709 12:13:22.440058 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:22.394556 17004 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:13:22.446568 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:22.447927 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.454181 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.461855 16901 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:22.467732 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.468096 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.535023 12494 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:22.403599 16918 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:22.470567 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:22.468046 16721 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:22.442714 17053 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:22.474323 17047 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:13:22.551416 16808 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:22.552062 16808 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:22.462633 17000 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.536536 12497 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:22.470870 16868 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:22.536875 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:22.536988 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55029.4us
I20260709 12:13:22.467209 17006 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:13:22.434158 16770 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:22.468369 16417 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:22.544728 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74264.1us
I20260709 12:13:22.566787 12491 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:22.478171 17052 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:13:22.552181 17039 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.537s	user 0.001s	sys 0.002s
I20260709 12:13:22.520519 13259 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:22.535421 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:22.555783 12567 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:22.480795 16981 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:22.480669 16971 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:13:22.559751 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:22.561757 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:22.563920 12488 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:22.465607 17007 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.552536 16808 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:22.529603 17035 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:22.541174 16721 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:22.561863 16899 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.950s	user 0.006s	sys 0.010s
I20260709 12:13:22.533525 16649 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:13:22.576020 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:22.569252 17059 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:22.567471 16770 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:22.569839 17052 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:22.581326 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134894us
I20260709 12:13:22.585384 16926 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:22.580245 17048 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:13:22.597939 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:22.608067 12496 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:22.602344 16918 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.026s
W20260709 12:13:22.628523 17049 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:13:22.618513 12643 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:22.618724 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54427.9us
I20260709 12:13:22.634021 17059 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.637517 16721 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:22.638787 16721 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:22.557835 17034 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:22.648870 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:13:22.622795 17035 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.609196 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:22.579495 17040 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:22.612787 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:22.627964 17057 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.149s
I20260709 12:13:22.605470 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
W20260709 12:13:22.666224 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99259.1us
W20260709 12:13:22.664247 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 148972us
W20260709 12:13:22.668987 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:22.603011 17053 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:22.672415 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 232432us
I20260709 12:13:22.672678 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:22.674732 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:22.623492 17052 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:22.680382 12497 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:22.677693 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70648.7us
I20260709 12:13:22.681258 12491 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:22.682600 12643 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
W20260709 12:13:22.682674 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117246us
I20260709 12:13:22.663370 16828 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:22.663846 17040 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:22.694752 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:22.701823 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:22.674194 17048 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:22.730263 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48759.1us
I20260709 12:13:22.720408 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:22.731246 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:22.696144 17007 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:22.730386 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:22.697958 17040 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:22.698061 16926 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:22.697710 16899 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:22.741204 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:22.652105 16716 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.685335 16629 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
W20260709 12:13:22.746001 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114890us
W20260709 12:13:22.788381 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64835.8us
I20260709 12:13:22.723091 17006 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:22.794603 17006 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:13:22.795286 17065 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:22.796342 17006 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:22.796550 17058 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:22.797083 17006 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:22.797734 17050 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.100s	user 0.001s	sys 0.004s
I20260709 12:13:22.798358 17006 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:22.802340 17062 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 12:13:22.792042 17066 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:22.810396 17066 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:22.813079 17066 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:22.814652 17048 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.680650 16417 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.815871 13254 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:13:22.820173 17035 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:22.820222 17070 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:22.797080 17058 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:22.822396 17058 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:22.823530 16551 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:22.823931 16551 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:22.824820 16551 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:22.827226 17051 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.307s	user 0.000s	sys 0.006s
I20260709 12:13:22.827492 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2527) took 1280 ms. Trace:
I20260709 12:13:22.828182 17061 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:22.828609 17004 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:22.827982 12496 rpcz_store.cc:276] 0709 12:13:21.550888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:21.551343 (+   455us) service_pool.cc:224] Handling call
0709 12:13:22.827253 (+1275910us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:22.827396 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2255207168}
I20260709 12:13:22.790369 12646 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:13:22.741690 16926 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:22.779230 17000 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:22.794970 17043 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.125s	user 0.006s	sys 0.000s
I20260709 12:13:22.738502 16828 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:22.732080 16918 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:13:22.832624 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101475us
I20260709 12:13:22.793131 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:22.840435 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:22.841926 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:22.841342 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:22.843835 12496 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:22.841064 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:22.829196 17004 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.846067 12646 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:22.828691 17061 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:22.844970 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:22.849923 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:22.856626 12637 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:22.860838 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67810.3us
I20260709 12:13:22.861799 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:22.858131 16828 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:22.864933 17070 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:22.865557 17070 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:22.860554 16918 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:22.865593 16828 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.845211 17066 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:22.852988 16634 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.718s	user 0.012s	sys 0.004s
W20260709 12:13:22.887665 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:22.960955 12637 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:22.912539 17085 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:22.924799 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:22.870117 17066 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:22.866484 17070 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.855187 17035 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:22.854633 17077 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:13:22.923213 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61510.3us
I20260709 12:13:22.966073 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:22.909992 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120875us
I20260709 12:13:23.063908 12642 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:22.854055 17061 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:22.907445 16901 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:22.999835 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:22.940104 17000 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:13:23.054780 17083 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:13:23.002882 12640 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:22.862816 16528 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:23.005177 12497 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:13:22.924136 16940 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:13:23.079882 17091 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:23.080211 16940 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:23.081962 16528 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.083425 16528 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.084189 16808 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.010792 17060 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:23.086033 17060 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:23.039125 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 189280us
I20260709 12:13:23.063372 17077 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.090585 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:22.867347 16918 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.094983 17066 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:23.075333 17035 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:23.050639 17088 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.054630 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:23.096796 17035 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.097760 16770 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:23.058226 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.065969 13258 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:22.906849 16634 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.065397 17085 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:13:23.099665 12644 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:23.100291 17085 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:23.100725 17085 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.100822 16634 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:23.072027 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 351749us
I20260709 12:13:23.102226 17004 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.071301 17000 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:23.103221 17000 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:22.961481 16926 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:23.064694 17081 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260709 12:13:23.077234 17005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.186s	user 0.017s	sys 0.008s
I20260709 12:13:23.085507 13253 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:13:23.076174 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.106237 16926 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:23.098255 17089 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:23.092724 16808 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:23.123245 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 218101us
I20260709 12:13:23.123972 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:23.113595 17005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.222s	user 0.018s	sys 0.008s
I20260709 12:13:23.140703 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:13:23.142661 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70470.8us
W20260709 12:13:23.144485 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.122421 17066 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.153115 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.163612 12567 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:13:23.154304 12566 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:23.164023 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99520.4us
I20260709 12:13:23.164963 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.155236 17090 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:13:23.156036 17080 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:23.110927 17060 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:23.101568 17085 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.097452 17088 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.159446 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.167743 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.141822 16770 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:23.178536 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70503.4us
I20260709 12:13:23.181849 17071 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.182559 17071 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.183220 17088 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.139227 17052 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.165848 17089 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:23.137259 17040 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.151286 17057 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.199918 12569 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:23.180081 17070 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:23.183059 17058 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.203961 12646 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:13:23.205041 12638 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:23.206043 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:23.197475 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:23.206136 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:23.209656 12564 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:23.212965 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:23.214545 12637 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:23.215349 17057 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.217182 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:23.218312 12639 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:23.218580 12642 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: [9.6-11.7]
I20260709 12:13:23.219412 17057 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:23.142526 17077 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:23.217274 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93221.1us
I20260709 12:13:23.168262 17066 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:23.223625 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46144.9us
I20260709 12:13:23.230428 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:23.231936 17040 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.235489 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2026) took 1334 ms. Trace:
I20260709 12:13:23.236020 12641 rpcz_store.cc:276] 0709 12:13:21.900679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:21.902137 (+  1458us) service_pool.cc:224] Handling call
0709 12:13:23.235144 (+1333007us) spinlock_profiling.cc:243] Waited 72.6 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:23.235356 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166962944}
I20260709 12:13:23.200794 17005 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:23.237914 17005 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.240495 17005 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.245935 17052 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.212188 17088 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.247313 17088 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:13:23.250887 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:23.253790 12566 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:23.209340 17070 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:23.266469 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.272189 12567 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:23.272927 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:23.273169 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50219.8us
I20260709 12:13:23.273723 12564 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 6 FOLLOWER]: Advancing to term 8
W20260709 12:13:23.273921 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84884.2us
I20260709 12:13:23.275615 17005 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:13:23.276154 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.228996 17057 raft_consensus.cc:2764] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:13:23.263147 17098 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 12:13:23.277578 17057 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:23.280440 17098 consensus_queue.cc:1243] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:13:23.281143 17098 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:23.279297 17097 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:13:23.280032 17000 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.282981 17097 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:23.283447 17097 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.283483 17058 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.253206 17091 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.279911 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.280224 17052 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:23.286183 17052 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.287683 17052 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.274132 17088 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.278025 17040 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:23.289865 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.284633 17097 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.291612 17097 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:23.284531 17000 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.284757 17091 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.296519 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.277544 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.266547 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:13:23.291522 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:23.302366 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.303608 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:23.291802 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:23.308208 16808 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:23.308555 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.310101 17057 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:23.310282 16808 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:23.312048 17095 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:23.310024 12564 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:23.312654 17095 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:23.313269 17095 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:23.315248 16417 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:23.318796 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.320014 17074 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:23.320453 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:23.321859 17074 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:13:23.322000 17037 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.449s	user 0.016s	sys 0.028s
I20260709 12:13:23.327972 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:23.329006 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:23.333039 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80309.5us
I20260709 12:13:23.322692 17074 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:23.281781 17098 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.337538 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:23.343156 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:23.331089 17057 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:13:23.291754 16626 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:23.346292 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:23.346351 12496 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:23.368436 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.333849 16721 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:23.348479 17000 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:23.325049 17058 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:13:23.377552 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85873us
W20260709 12:13:23.378091 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50235.3us
I20260709 12:13:23.285204 17070 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.334097 16971 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:13:23.394280 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:23.395951 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.402927 16940 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:23.403407 16926 raft_consensus.cc:2764] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:13:23.403818 16940 consensus_queue.cc:1243] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:23.404132 16926 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:23.404743 17002 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:23.405292 17002 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.406692 17002 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.410777 17066 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:23.375113 16626 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.418576 16626 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.345124 17074 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.420158 16626 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:23.404776 16926 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.423992 16926 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.428769 17080 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:23.429749 12643 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
W20260709 12:13:23.429883 17103 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:13:23.432314 16711 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:23.433105 16711 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:23.442214 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54522us
I20260709 12:13:23.443094 16762 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:23.444597 16762 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:13:23.372852 17060 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:23.446631 17060 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:23.447083 17060 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:23.341915 16886 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:13:23.375234 17088 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:23.481612 12565 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:23.455607 16926 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.448554 17071 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:23.387427 17090 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:13:23.491376 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 691) took 2202 ms. Trace:
I20260709 12:13:23.506687 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:23.521961 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.523401 12496 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:23.526240 13259 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:23.491423 16982 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.527832 16982 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.480314 16711 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:23.534099 17070 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:23.392021 17061 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.540851 16982 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.540632 17095 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.543145 12536 leader_election.cc:400] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:13:23.543579 12536 leader_election.cc:403] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:13:23.545236 17052 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:23.590121 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:23.590979 13260 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:23.514338 12568 rpcz_store.cc:276] 0709 12:13:21.288764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:21.290605 (+  1841us) service_pool.cc:224] Handling call
0709 12:13:21.291059 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:21.291600 (+   541us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:21.291635 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:21.292358 (+   723us) log.cc:844] Serialized 26 byte log entry
0709 12:13:21.631122 (+338764us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:21.631864 (+   742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:21.631951 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:23.267328 (+1635377us) spinlock_profiling.cc:243] Waited 39 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:23.457439 (+190111us) raft_consensus.cc:1685] finished
0709 12:13:23.457464 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:23.491227 (+ 33763us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"spinlock_wait_cycles":89602304,"thread_start_us":338345,"threads_started":1,"wal-append.queue_time_us":469599}
I20260709 12:13:23.380923 17000 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:23.597868 17000 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:23.374290 17078 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260709 12:13:23.499063 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:23.603359 13253 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:23.603521 16780 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:23.500249 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:23.476107 12637 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:23.506204 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.389207 16721 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:23.512759 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.606896 16721 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.545529 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.610976 16551 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:23.546036 13254 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:23.545694 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.546612 17058 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:23.627224 12496 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:23.413360 17059 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.547683 16981 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:23.547552 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:23.549175 16926 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:13:23.551925 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:23.647065 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.423398 17098 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:23.557034 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2540) took 1866 ms (client timeout 1991 ms). Trace:
I20260709 12:13:23.557952 16901 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:23.559675 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.578475 17089 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.579136 17060 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.579419 16417 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:23.580155 16945 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:23.547607 17061 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:23.581281 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:23.691190 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 111291us
I20260709 12:13:23.588891 17091 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:23.481352 16557 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.108s
I20260709 12:13:23.604081 16780 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.554224 17095 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:23.643918 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 273621us
I20260709 12:13:23.646942 12490 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:23.626299 17085 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:23.653503 12488 rpcz_store.cc:269] 0709 12:13:21.703778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:21.882382 (+178604us) service_pool.cc:224] Handling call
0709 12:13:23.556799 (+1674417us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:23.556959 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1423913728}
I20260709 12:13:23.674448 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:23.639364 16981 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:23.679595 16945 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.671494 17098 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:23.654156 17059 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:23.704216 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:23.721174 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 173703us
I20260709 12:13:23.723023 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:23.698535 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:23.705179 12610 connection.cc:256] RPC call timeout handler was delayed by -1.3195896199997605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22540000000000002s for a response
W20260709 12:13:23.727624 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53319.4us
I20260709 12:13:23.728796 12496 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:23.710956 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:23.717258 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:23.731948 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:23.733121 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.704672 16886 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:23.741870 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.734061 12497 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:23.729043 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 284136us
W20260709 12:13:23.736227 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.736281 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:23.736917 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:23.738231 13258 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:23.745458 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.745783 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.675631 17088 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:13:23.737761 16557 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:23.749091 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:23.748065 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:23.748821 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.771391 12492 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:23.702317 16899 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:13:23.720613 17059 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.736682 17109 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:23.746716 12496 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:23.738027 17070 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.743688 16886 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:23.784540 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51327.3us
I20260709 12:13:23.786147 12497 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:13:23.805889 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.806862 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.807834 12497 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:23.713765 16417 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:13:23.676795 17089 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.795409 12567 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:23.737461 17091 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 12:13:23.809664 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.806911 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:23.790330 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79460.4us
I20260709 12:13:23.798756 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:23.799201 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:23.812340 12496 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:23.813755 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:23.800551 12491 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:23.754611 17005 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:23.804325 12571 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:13:23.818602 12567 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:23.811823 12564 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:23.762830 16918 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:23.789348 16634 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:23.794211 17059 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:23.825484 12568 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:23.826057 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.834964 12567 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:23.828879 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.794113 17109 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:23.837185 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:23.832473 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134077us
I20260709 12:13:23.833235 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.839877 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.840764 12572 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:23.841411 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.844954 12497 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:23.810952 16886 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:23.847023 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:23.784184 17095 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:13:23.821928 17046 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:23.823482 16629 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:13:23.867010 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.819793 17115 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.884788 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:23.886715 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:23.869613 12496 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
W20260709 12:13:23.893456 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46543.6us
I20260709 12:13:23.893563 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:23.889735 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:23.848153 17109 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:23.900470 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:23.809929 17070 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.852181 17090 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:23.869174 17091 consensus_queue.cc:1243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:13:23.861793 17034 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:23.862257 16762 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.915823 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:23.921566 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:23.790709 16780 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:23.879796 17046 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:23.883056 17005 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.918649 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:23.910300 17117 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:13:23.906242 16721 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:23.892208 16629 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:23.914814 16828 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:23.912667 16649 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:23.956921 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:23.929025 12497 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:13:23.965227 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43769.6us
I20260709 12:13:23.924023 17123 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:13:23.966536 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 99647.7us
I20260709 12:13:23.906384 17058 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
W20260709 12:13:23.991408 17108 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:13:23.992452 17125 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:23.994205 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:23.955165 17091 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:23.962999 16649 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:23.957332 17116 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:24.035372 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:24.036965 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:23.983727 17060 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:23.986311 17123 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:23.931385 16762 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.956547 17046 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:23.994506 16716 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.040835 16649 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:24.049064 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:23.983296 16901 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:13:23.907481 17115 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.957520 17005 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.964082 17074 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:24.051645 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 303625us
W20260709 12:13:24.071869 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83833.6us
I20260709 12:13:24.101851 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:24.090415 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:24.091892 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92220.4us
I20260709 12:13:24.094771 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:24.058097 16828 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:24.106137 16828 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:24.107419 16828 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.097599 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:24.097886 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:24.105211 16557 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:24.113035 17127 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:24.113211 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.113306 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:24.077657 17112 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.322s	user 0.000s	sys 0.002s
I20260709 12:13:24.035562 17091 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.068149 17060 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:24.129830 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:24.128861 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 144687us
I20260709 12:13:24.144927 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:24.167861 12639 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:24.164155 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:24.168744 12644 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:13:24.115809 16721 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:24.164832 12640 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:24.166942 12491 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 15 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:13:24.164009 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:24.118606 17122 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:13:24.114737 16901 consensus_queue.cc:1243] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260709 12:13:24.184499 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54778.1us
I20260709 12:13:24.111533 16557 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.118767 17128 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:24.074839 16716 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:24.210495 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 106252us
I20260709 12:13:24.245380 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:24.215986 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:24.197496 12610 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.991s (SENT) [suppressed 16 similar messages]
W20260709 12:13:24.260303 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44388.6us
I20260709 12:13:24.122012 17074 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:24.279655 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:24.117838 16629 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.288332 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:24.180721 17060 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.212864 17128 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:24.278355 17121 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:13:24.195439 17125 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 12:13:24.270257 17089 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.193487 17124 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:13:24.263335 17058 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:24.301426 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 147161us
W20260709 12:13:24.307011 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61700.5us
I20260709 12:13:24.313886 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:24.316570 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:24.319846 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:24.340255 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:24.289422 17141 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:24.357434 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 8.
I20260709 12:13:24.298455 17128 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:24.372362 12568 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.001s
I20260709 12:13:24.372929 12568 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:24.376098 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.381062 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:24.352367 17141 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:24.364728 17111 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.378s	user 0.000s	sys 0.002s
W20260709 12:13:24.368501 17052 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.779s	user 0.005s	sys 0.008s
I20260709 12:13:24.386291 12646 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:24.185725 17123 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.393584 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:24.395229 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 197775us
I20260709 12:13:24.396945 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.398486 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.402319 12645 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:24.403363 13258 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:24.403923 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.374483 17035 raft_consensus.cc:2764] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 12:13:24.383101 16980 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:24.356896 17058 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.418725 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.430217 12646 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:24.386691 17141 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:24.431406 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
W20260709 12:13:24.440649 17113 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.381s	user 0.001s	sys 0.001s
I20260709 12:13:24.446923 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.421572 17123 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:24.422526 16762 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.450793 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:24.414364 16980 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:24.368233 16721 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.468616 17145 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:24.469127 17153 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.477804 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:24.458971 16980 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.460060 17115 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:24.461031 17135 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:24.461499 17130 consensus_queue.cc:1175] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 10 status { last_received { term: 10 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 10 index: 11 } }
I20260709 12:13:24.462286 17156 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:24.466130 12639 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:24.453927 17091 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.468497 17139 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:24.504597 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 571) took 1139 ms. Trace:
I20260709 12:13:24.517771 12493 rpcz_store.cc:276] 0709 12:13:23.340248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.343905 (+  3657us) service_pool.cc:224] Handling call
0709 12:13:23.344403 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:23.344810 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:23.344837 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:23.345377 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:13:23.675378 (+330001us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:23.675434 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:23.675528 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:24.479732 (+804204us) raft_consensus.cc:1685] finished
0709 12:13:24.479767 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:24.479991 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3453,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":13801,"thread_start_us":329532,"threads_started":1,"wal-append.queue_time_us":546486}
I20260709 12:13:24.525552 12643 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:24.526113 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:24.472618 17153 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.468281 17147 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:24.476001 17005 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:24.507690 17061 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.014s	sys 0.029s
I20260709 12:13:24.546850 17150 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.556252 17150 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.557749 16528 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:24.558207 16528 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:24.558665 17150 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:24.491354 17136 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.551244 17073 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:24.538686 16980 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:24.550637 17061 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.015s	sys 0.029s
I20260709 12:13:24.555377 17143 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:24.550707 17120 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.344s	user 0.001s	sys 0.000s
I20260709 12:13:24.543833 17115 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260709 12:13:24.530083 17146 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:13:24.554795 12569 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:24.559084 16528 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:24.589607 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:24.642428 17058 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.597527 16716 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:24.580222 17074 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:24.533556 17144 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.647104 17144 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.519325 17135 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.652287 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:24.653098 12572 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:13:24.591996 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:24.609934 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132219us
W20260709 12:13:24.641396 16551 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.016s	user 0.008s	sys 0.007s
I20260709 12:13:24.583174 17060 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:24.669288 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:24.664813 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:24.679118 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:24.683713 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:24.685366 12572 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:13:24.658120 17006 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.654134 17073 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:24.668471 17074 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.703943 12564 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:24.536043 17156 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.682261 17005 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.714133 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:24.709714 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:24.729681 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:24.730279 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:24.686749 16634 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.855s	user 0.010s	sys 0.003s
W20260709 12:13:24.664292 17116 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.002s	sys 0.024s
I20260709 12:13:24.717540 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:24.690688 16528 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:24.715658 17060 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:24.719630 17135 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:24.731765 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:24.583374 17136 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.782018 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:24.783382 12568 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:24.740885 17115 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:24.743412 17053 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:24.653503 17143 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.721939 17139 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:24.750603 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:24.739806 17116 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.003s	sys 0.025s
I20260709 12:13:24.694474 17144 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:24.731603 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57221.2us
W20260709 12:13:24.765130 17000 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.166s	user 0.016s	sys 0.001s
I20260709 12:13:24.805159 17000 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.805560 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:24.807420 17061 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:24.807597 17000 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.807960 17061 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:24.704212 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:24.808584 12489 raft_consensus.cc:2368] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11.
W20260709 12:13:24.772603 17046 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.694s	user 0.008s	sys 0.005s
I20260709 12:13:24.809347 17046 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.808477 17061 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:24.728704 17159 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:13:24.745092 16634 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:24.814833 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:24.741429 17006 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.741750 17005 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.746274 17074 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.799476 12572 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:24.740670 17167 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260709 12:13:24.797928 17060 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:24.793433 17115 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:13:24.816821 17146 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:13:24.802703 17116 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:24.812017 17165 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:24.816157 17091 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:24.821427 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100136us
I20260709 12:13:24.830801 17000 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:24.857985 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:24.793361 17148 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:24.838109 17115 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:24.878959 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2388) took 2969 ms. Trace:
W20260709 12:13:24.895609 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:24.934337 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:24.882596 17059 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.025s	user 0.009s	sys 0.005s
I20260709 12:13:24.879488 12563 rpcz_store.cc:276] 0709 12:13:21.909353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:22.107955 (+198602us) service_pool.cc:224] Handling call
0709 12:13:22.108428 (+   473us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:22.610311 (+501883us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:22.610361 (+    50us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:13:22.676585 (+ 66224us) log.cc:844] Serialized 616 byte log entry
0709 12:13:22.872640 (+196055us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:22.900796 (+ 28156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:24.855823 (+1955027us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:24.855891 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:24.855944 (+    53us) raft_consensus.cc:1685] finished
0709 12:13:24.855953 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:24.878873 (+ 22920us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4030,"raft.queue_time_us":156638,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":7713,"spinlock_wait_cycles":390562048,"thread_start_us":347760,"threads_started":2,"wal-append.queue_time_us":333364,"wal-append.run_cpu_time_us":2739,"wal-append.run_wall_time_us":1008488}
I20260709 12:13:24.939494 17153 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.943653 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:24.837103 17128 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:24.948325 16918 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.100s	user 0.014s	sys 0.000s
I20260709 12:13:24.951022 17074 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:24.881943 17060 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:24.956264 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:24.957218 17153 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:24.957492 12639 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:13:24.952309 16649 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.865s	user 0.015s	sys 0.000s
W20260709 12:13:24.939561 17166 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:13:24.811946 17164 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.150s
I20260709 12:13:24.860711 17091 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.964903 12639 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:24.965345 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:24.883613 17165 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.961388 16918 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:24.976894 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:24.973170 12640 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:24.955237 17135 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 12:13:24.986799 17135 consensus_queue.cc:1243] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:13:24.990959 17128 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:24.992098 12639 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.942895 17059 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:24.965727 17094 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:24.967905 17000 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:24.968071 17168 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:24.968137 17149 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.011667 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2558) took 1429 ms. Trace:
W20260709 12:13:24.969364 17169 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:13:24.972608 16649 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:24.968410 17176 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:25.016085 17175 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:13:24.977514 16557 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:25.023473 17161 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
I20260709 12:13:25.023717 17136 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:25.025883 17163 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
I20260709 12:13:24.856297 17073 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.028195 17091 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.022174 17060 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.036492 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 573) took 1261 ms. Trace:
I20260709 12:13:25.018730 17006 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.039619 16981 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.036537 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2318) took 1888 ms. Trace:
I20260709 12:13:25.053570 16981 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:25.052872 16557 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:25.064103 17098 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.328s	user 0.008s	sys 0.005s
I20260709 12:13:25.012121 12497 rpcz_store.cc:276] 0709 12:13:23.583164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.846185 (+263021us) service_pool.cc:224] Handling call
0709 12:13:23.846996 (+   811us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:23.869568 (+ 22572us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:24.076959 (+207391us) log.cc:844] Serialized 14 byte log entry
0709 12:13:24.373674 (+296715us) log.cc:844] Serialized 14 byte log entry
0709 12:13:24.435940 (+ 62266us) log.cc:844] Serialized 14 byte log entry
0709 12:13:24.452476 (+ 16536us) log.cc:844] Serialized 14 byte log entry
0709 12:13:24.452810 (+   334us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:13:24.533318 (+ 80508us) log.cc:844] Serialized 616 byte log entry
0709 12:13:24.533551 (+   233us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:24.533593 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:24.533722 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:25.011365 (+477643us) raft_consensus.cc:1685] finished
0709 12:13:25.011388 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:25.011593 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120909,"prepare.queue_time_us":699222,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":111,"raft.queue_time_us":244641,"raft.run_cpu_time_us":1096,"raft.run_wall_time_us":44122,"spinlock_wait_cycles":6885632,"thread_start_us":166623,"threads_started":2,"wal-append.queue_time_us":230926}
I20260709 12:13:25.053071 17156 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.065555 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:25.066350 17088 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:25.066831 17088 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:25.019066 17000 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:25.067763 17088 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.068842 17088 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:25.070178 17165 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:25.017259 17094 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:25.049561 12496 rpcz_store.cc:276] 0709 12:13:23.774727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.914647 (+139920us) service_pool.cc:224] Handling call
0709 12:13:23.915132 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:23.915643 (+   511us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:24.580771 (+665128us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b180037e6a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:24.721841 (+141070us) log.cc:844] Serialized 14 byte log entry
0709 12:13:24.860078 (+138237us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:24.876204 (+ 16126us) log.cc:844] Serialized 26 byte log entry
0709 12:13:24.876399 (+   195us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:24.876711 (+   312us) log.cc:844] Serialized 14 byte log entry
0709 12:13:24.882872 (+  6161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:24.882951 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:25.036240 (+153289us) raft_consensus.cc:1685] finished
0709 12:13:25.036263 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:25.036421 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6825,"prepare.queue_time_us":533420,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":12011,"raft.queue_time_us":68152,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":50875,"spinlock_wait_cycles":31022464,"thread_start_us":205567,"threads_started":2,"wal-append.queue_time_us":216709}
I20260709 12:13:25.078434 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.053745 12566 rpcz_store.cc:276] 0709 12:13:23.147612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.174862 (+ 27250us) service_pool.cc:224] Handling call
0709 12:13:23.185984 (+ 11122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:23.262759 (+ 76775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:25.036314 (+1773555us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:25.036457 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155304832}
I20260709 12:13:25.082952 12496 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:25.051942 17002 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:25.052412 17168 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:24.978791 17116 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.094762 17143 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.095274 17046 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.096856 16649 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.098553 17059 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.099779 17005 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.100159 17061 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.100899 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:25.088183 17085 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.362s	user 0.015s	sys 0.000s
I20260709 12:13:25.103772 16945 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.105438 16716 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:25.105315 17115 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.051573 17149 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:25.111560 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:25.180124 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:25.180570 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.077811 16557 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.115885 16982 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.131003 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.184713 12643 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
W20260709 12:13:25.133832 12610 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.712s (SENT)
W20260709 12:13:25.143049 16886 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.249s	user 0.015s	sys 0.000s
I20260709 12:13:25.148360 17178 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.068842 17000 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.106949 17164 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Committing config change with OpId 9.7: config changed from index 3 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:25.148594 16945 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:25.117252 17168 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.206784 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.208161 12566 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:25.178298 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.145731 12570 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:13:25.112655 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:25.178644 17085 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:25.182649 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:25.185653 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:25.140635 13253 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:13:25.194904 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.209076 13258 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:25.195549 16886 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:25.200510 17184 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:13:25.071507 17176 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.147361 16716 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:25.212960 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:25.134615 17094 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.223050 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:25.221944 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88164.2us
I20260709 12:13:25.229907 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:25.210675 17141 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.757s	user 0.013s	sys 0.001s
I20260709 12:13:25.231931 17168 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.234486 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:25.242810 12643 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:13:25.235363 17077 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:25.236702 17109 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.285s	user 0.015s	sys 0.000s
I20260709 12:13:25.238552 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:25.239029 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.208987 17115 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:25.239642 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.242142 12644 raft_consensus.cc:2368] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:25.228427 17061 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.222590 16528 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:25.194090 16982 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.199250 17178 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.250566 17005 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:25.251428 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:25.255604 12567 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:25.241361 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.241093 16626 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.265712 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.265583 12497 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:25.243460 17094 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:25.271682 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:25.255415 17128 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:25.219497 17085 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.255137 17180 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.279446 17139 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:25.282382 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.249876 17073 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:25.283635 12567 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:25.284566 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:25.284847 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:25.261186 16528 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:25.286640 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:25.244833 17141 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.288560 12494 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:13:25.288210 17097 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:25.277287 17128 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:25.258600 17109 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.292069 17149 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:25.296152 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.296223 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:25.299118 12488 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:25.300392 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.289290 16528 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:25.285465 17180 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.304140 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2287) took 3624 ms. Trace:
I20260709 12:13:25.304829 17176 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:25.304971 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:25.309397 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2546) took 1803 ms (client timeout 1712 ms). Trace:
W20260709 12:13:25.326076 12489 rpcz_store.cc:269] 0709 12:13:23.505681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.505871 (+   190us) service_pool.cc:224] Handling call
0709 12:13:25.309099 (+1803228us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:25.309300 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2765277568}
I20260709 12:13:25.311515 16780 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:25.290347 17098 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:25.297786 17128 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:25.290722 17123 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.834s	user 0.010s	sys 0.005s
I20260709 12:13:25.276526 16626 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.320643 12565 rpcz_store.cc:276] 0709 12:13:21.680303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:21.682110 (+  1807us) service_pool.cc:224] Handling call
0709 12:13:23.457875 (+1775765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:23.482280 (+ 24405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:25.303876 (+1821596us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54001a36d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:25.304067 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4058267008}
I20260709 12:13:25.296628 17097 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:25.322316 16982 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.324024 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.328240 17098 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:25.337221 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.340782 16626 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.346760 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2020) took 1064 ms. Trace:
I20260709 12:13:25.347308 12489 rpcz_store.cc:276] 0709 12:13:24.281857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.336922 (+1055065us) service_pool.cc:224] Handling call
0709 12:13:25.346680 (+  9758us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.351187 17180 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:25.352777 12495 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:25.349442 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.354773 12565 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:25.356678 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.358386 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.362288 12488 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:25.339236 16680 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.364418 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.338989 17097 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:25.342916 17098 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:25.366811 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2017) took 1085 ms. Trace:
I20260709 12:13:25.367300 12488 rpcz_store.cc:276] 0709 12:13:24.281121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.335697 (+1054576us) service_pool.cc:224] Handling call
0709 12:13:25.366728 (+ 31031us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.367708 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.359082 17004 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.369422 17004 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:25.375566 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:25.384474 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.374253 16680 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.389573 17002 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.390401 17002 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.392678 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.393996 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.394798 17141 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:25.395516 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.396193 12565 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:25.397171 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.399014 12488 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:13:25.400043 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.403581 17085 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.405326 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2027) took 1095 ms. Trace:
I20260709 12:13:25.295099 16716 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.405870 12488 rpcz_store.cc:276] 0709 12:13:24.309318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.392345 (+1083027us) service_pool.cc:224] Handling call
0709 12:13:25.405058 (+ 12713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.408139 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.409947 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:25.410256 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.393940 17002 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.411264 12489 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
W20260709 12:13:25.417925 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.424860 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2021) took 1134 ms. Trace:
I20260709 12:13:25.425345 12489 rpcz_store.cc:276] 0709 12:13:24.282091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.409908 (+1127817us) service_pool.cc:224] Handling call
0709 12:13:25.416823 (+  6915us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.426472 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.427204 17109 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.427821 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.432677 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:25.433377 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.408927 16886 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.435014 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:25.434036 13261 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
W20260709 12:13:25.436206 17126 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.153s	user 0.002s	sys 0.002s
I20260709 12:13:25.435827 16716 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:25.437367 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:25.427255 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.434719 16528 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.434217 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:25.450227 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:25.456642 13253 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:25.456266 13258 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15.
I20260709 12:13:25.456946 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:25.427210 17053 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:25.458268 17053 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:25.463440 17188 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:13:25.465085 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2408) took 1179 ms. Trace:
I20260709 12:13:25.466655 17060 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.467344 17060 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.453522 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.465550 12571 rpcz_store.cc:276] 0709 12:13:24.286010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:24.296911 (+ 10901us) service_pool.cc:224] Handling call
0709 12:13:24.320791 (+ 23880us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:24.366389 (+ 45598us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:24.366430 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:24.377541 (+ 11111us) log.cc:844] Serialized 26 byte log entry
0709 12:13:25.248456 (+870915us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b18000a6ae0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:13:25.295144 (+ 46688us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:25.407029 (+111885us) log.cc:844] Serialized 14 byte log entry
0709 12:13:25.407410 (+   381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:25.407507 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:25.464684 (+ 57177us) raft_consensus.cc:1685] finished
0709 12:13:25.464823 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:25.465000 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2009,"raft.queue_time_us":87183,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":8269,"spinlock_wait_cycles":40558848,"thread_start_us":133786,"threads_started":2,"wal-append.queue_time_us":964493}
I20260709 12:13:25.426460 17178 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.472878 12571 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:25.473021 17178 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.474979 12572 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:25.451418 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.465525 13254 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:25.475804 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:25.456153 17186 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.180s	user 0.002s	sys 0.000s
I20260709 12:13:25.482967 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.453742 17109 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.485112 12565 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:25.487623 12637 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:13:25.488183 12571 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:13:25.489344 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.490172 12637 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:25.490778 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:25.492161 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.493993 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.494839 12637 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:25.473605 17178 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.502086 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.502657 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.503870 12646 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:13:25.504391 17119 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.518s	user 0.003s	sys 0.000s
I20260709 12:13:25.503901 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:25.508556 17153 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.509155 17153 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:25.509822 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.510682 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:25.511053 16528 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:25.511487 12639 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:25.511669 12643 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:25.493021 17109 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.459348 17053 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.516166 17002 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.517189 17053 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:25.520530 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.521313 12496 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:25.525022 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:25.526688 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:25.526988 12639 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:25.521884 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2019) took 1240 ms. Trace:
I20260709 12:13:25.528121 12496 rpcz_store.cc:276] 0709 12:13:24.281597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.520200 (+1238603us) service_pool.cc:224] Handling call
0709 12:13:25.521808 (+  1608us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.529228 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:25.516868 17002 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.531836 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:25.524905 12488 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:25.533674 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.535214 12639 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:25.540246 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [12.11-14.12]
I20260709 12:13:25.541225 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2561) took 1415 ms. Trace:
I20260709 12:13:25.511847 17046 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.542022 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.542065 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:25.493865 17060 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:25.544286 12496 rpcz_store.cc:276] 0709 12:13:24.125954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.538672 (+1412718us) service_pool.cc:224] Handling call
0709 12:13:25.539972 (+  1300us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:25.540940 (+   968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:25.541163 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.493695 17164 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:13:25.542004 17058 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.549183 17164 consensus_queue.cc:1243] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:13:25.549417 17058 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.551764 17058 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:25.537283 17190 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:13:25.553895 17178 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:25.534597 17071 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:13:25.530557 17079 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:25.534916 16634 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:25.558058 12639 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:25.558414 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2039) took 1182 ms. Trace:
I20260709 12:13:25.558935 12644 rpcz_store.cc:276] 0709 12:13:24.375659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:24.375800 (+   141us) service_pool.cc:224] Handling call
0709 12:13:25.557941 (+1182141us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:25.558086 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987098496}
I20260709 12:13:25.559774 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:25.560994 12568 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:25.561990 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.486591 16370 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:13:25.511993 16528 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.569779 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.579739 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.580761 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:25.545466 17141 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:25.582974 17141 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:25.583532 17141 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:25.543282 17077 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.597486 12568 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:25.532732 17097 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.538290 17098 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.609223 13253 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:25.614802 12496 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:25.615825 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2563) took 1489 ms. Trace:
I20260709 12:13:25.616278 12496 rpcz_store.cc:276] 0709 12:13:24.126623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.601553 (+1474930us) service_pool.cc:224] Handling call
0709 12:13:25.614563 (+ 13010us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.615411 (+   848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:25.615727 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.619180 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.623121 12565 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:25.568085 17046 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:25.565322 17128 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:25.603567 17005 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:25.675483 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.578843 16528 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:25.776386 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.777737 17058 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:25.778733 17160 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:25.780818 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.782400 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:25.968338 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.968606 12639 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:25.720074 16780 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:25.784970 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:25.789677 17153 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:25.934747 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:25.969496 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:25.991955 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:25.972455 12489 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
W20260709 12:13:25.973359 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2565) took 1846 ms (client timeout 1710 ms). Trace:
W20260709 12:13:25.993012 12497 rpcz_store.cc:269] 0709 12:13:24.127460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.065202 (+937742us) service_pool.cc:224] Handling call
0709 12:13:25.972873 (+907671us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:25.973196 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319674752}
I20260709 12:13:25.605376 17125 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.117s
I20260709 12:13:25.618799 17109 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:25.602654 17077 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:26.024906 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2555) took 2472 ms (client timeout 1595 ms). Trace:
W20260709 12:13:26.025820 12490 rpcz_store.cc:269] 0709 12:13:23.552274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.726046 (+173772us) service_pool.cc:224] Handling call
0709 12:13:26.024657 (+2298611us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:26.024819 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2056776960}
I20260709 12:13:26.024821 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:25.975579 12496 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:25.974821 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2394) took 2188 ms. Trace:
I20260709 12:13:25.976588 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.043414 12569 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:25.992486 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 575) took 2192 ms (client timeout 2324 ms). Trace:
I20260709 12:13:26.014348 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 13. Ops: []
I20260709 12:13:26.045614 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 579) took 1765 ms. Trace:
I20260709 12:13:26.046005 12497 rpcz_store.cc:276] 0709 12:13:24.280607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.013302 (+1732695us) service_pool.cc:224] Handling call
0709 12:13:26.014011 (+   709us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.045276 (+ 31265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.045550 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.042332 12570 rpcz_store.cc:276] 0709 12:13:23.786724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.793729 (+  7005us) service_pool.cc:224] Handling call
0709 12:13:23.837596 (+ 43867us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.212296 (+1374700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:25.974478 (+762182us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:25.974729 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2984376192}
W20260709 12:13:26.044497 12491 rpcz_store.cc:269] 0709 12:13:23.783341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.824217 (+ 40876us) service_pool.cc:224] Handling call
0709 12:13:25.975183 (+2150966us) spinlock_profiling.cc:243] Waited 56.3 ms on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:25.975390 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129487744}
W20260709 12:13:26.065716 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.071548 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.072610 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:25.992475 16982 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:26.086764 12490 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:26.087935 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 580) took 1807 ms. Trace:
I20260709 12:13:26.088383 12490 rpcz_store.cc:276] 0709 12:13:24.280877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.036627 (+1755750us) service_pool.cc:224] Handling call
0709 12:13:26.077370 (+ 40743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.087492 (+ 10122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.087849 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.013612 16770 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:26.024585 16901 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.118324 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:26.127099 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.129861 12639 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:13:26.077698 17053 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.560s	user 0.010s	sys 0.001s
I20260709 12:13:26.138229 12490 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:13:26.139253 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2029) took 1829 ms. Trace:
I20260709 12:13:26.139622 12490 rpcz_store.cc:276] 0709 12:13:24.309838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.089401 (+1779563us) service_pool.cc:224] Handling call
0709 12:13:26.137889 (+ 48488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:26.138863 (+   974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.139172 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.141615 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 581) took 1818 ms. Trace:
I20260709 12:13:25.974494 17058 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.143869 12490 rpcz_store.cc:276] 0709 12:13:24.322783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.140514 (+1817731us) service_pool.cc:224] Handling call
0709 12:13:26.140982 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.141204 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.141239 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.141301 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:26.141335 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.141411 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.141547 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:25.981705 17160 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.174822 12639 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:13:26.057229 16886 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:25.994051 16780 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.194788 12490 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:13:26.219805 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2568) took 1360 ms. Trace:
I20260709 12:13:26.220350 12490 rpcz_store.cc:276] 0709 12:13:24.859492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.159616 (+1300124us) service_pool.cc:224] Handling call
0709 12:13:26.194297 (+ 34681us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.219323 (+ 25026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.219712 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.196718 17000 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:26.198133 17191 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.589s	user 0.001s	sys 0.000s
I20260709 12:13:26.208285 12491 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:26.234627 12490 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:26.237118 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2337) took 1046 ms. Trace:
W20260709 12:13:26.237735 12572 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.749s	user 0.011s	sys 0.002s
I20260709 12:13:26.237816 12563 rpcz_store.cc:276] 0709 12:13:25.190849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.470755 (+279906us) service_pool.cc:224] Handling call
0709 12:13:25.471198 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.474641 (+  3443us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:25.607548 (+132907us) log.cc:844] Serialized 14 byte log entry
0709 12:13:26.236690 (+629142us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.236756 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:26.236809 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.236892 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.237050 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15762,"prepare.queue_time_us":69397,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"thread_start_us":735582,"threads_started":3}
I20260709 12:13:26.238372 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.247820 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 583) took 1003 ms. Trace:
I20260709 12:13:26.252477 12569 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:26.259488 12572 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 12:13:26.178925 17094 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.901s	user 0.013s	sys 0.002s
I20260709 12:13:26.276484 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.300873 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.299060 12490 rpcz_store.cc:276] 0709 12:13:25.244088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.221437 (+977349us) service_pool.cc:224] Handling call
0709 12:13:26.224929 (+  3492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.235193 (+ 10264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.247708 (+ 12515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.308182 12644 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:26.309537 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2339) took 1118 ms. Trace:
I20260709 12:13:26.310102 12572 rpcz_store.cc:276] 0709 12:13:25.191223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.456499 (+265276us) service_pool.cc:224] Handling call
0709 12:13:26.309469 (+852970us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.163873 16770 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.249892 17109 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:26.160974 16901 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.273751 17153 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:26.317354 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.146500 16982 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.365371 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.366279 16982 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.366875 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.236276 16886 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.367800 17058 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.340287 17098 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.358385 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.368538 17058 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:26.359301 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:26.360069 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.369285 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2024) took 2060 ms. Trace:
I20260709 12:13:26.369912 12642 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:26.251761 17000 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.369313 12644 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:13:26.339362 17153 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:26.372876 16716 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.917s	user 0.010s	sys 0.000s
I20260709 12:13:26.363623 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:26.373678 17097 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.369800 12488 rpcz_store.cc:276] 0709 12:13:24.308622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.480677 (+1172055us) service_pool.cc:224] Handling call
0709 12:13:26.369071 (+888394us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:26.369226 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50096384}
I20260709 12:13:26.374745 17097 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:26.375140 17097 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.375499 17097 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:26.376915 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:26.334404 17094 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:26.377980 17094 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.378214 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.367760 12643 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:26.373771 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.369228 16886 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:26.379494 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:26.366959 16982 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.371064 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:26.380542 12642 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:26.380167 16982 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.380744 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:26.372506 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.373049 17153 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:26.381475 12646 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:26.378410 17094 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:26.378571 17098 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:26.386152 17098 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.386564 17098 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:26.386713 16528 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.347146 17109 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.390497 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:26.380002 12645 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:13:26.348160 12639 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:26.360596 16770 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.399120 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:26.399879 12643 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:13:26.327972 17005 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:26.407424 13253 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:26.410159 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:26.410741 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.398995 16770 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:26.418443 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.369827 17046 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:26.423801 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.424605 12572 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:26.390977 16982 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.413729 17005 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.369179 17058 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.412952 16528 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:26.457881 17199 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.471810 12643 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:26.474687 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.475419 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:26.476305 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.476926 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:26.434900 13261 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 3 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:26.412302 16770 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.458321 16557 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:26.493772 16770 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:26.495175 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 584) took 1250 ms. Trace:
I20260709 12:13:26.495630 12490 rpcz_store.cc:276] 0709 12:13:25.244293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.424886 (+1180593us) service_pool.cc:224] Handling call
0709 12:13:26.494663 (+ 69777us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.494842 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.494869 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.494911 (+    42us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:26.494938 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.494998 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.495119 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.496183 12646 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:26.497779 17199 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.497484 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 587) took 1252 ms. Trace:
I20260709 12:13:26.499346 12490 rpcz_store.cc:276] 0709 12:13:25.244922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.496569 (+1251647us) service_pool.cc:224] Handling call
0709 12:13:26.496904 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.497122 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.497157 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.497211 (+    54us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:26.497247 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.497313 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.497427 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.500236 17199 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.502326 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:26.502671 12565 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:26.503904 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:26.505084 12611 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15.
I20260709 12:13:26.505427 12611 leader_election.cc:403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:26.508975 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:26.459255 17005 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:26.509917 17046 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:26.525851 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 426425us
I20260709 12:13:26.510478 16886 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15.
I20260709 12:13:26.520704 16716 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:26.536636 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.524544 16557 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.551610 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.553364 17070 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.551610 17199 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.554302 17199 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.556514 17199 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.556763 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 723) took 1048 ms. Trace:
I20260709 12:13:26.557782 16716 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:26.558287 17197 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:13:26.558876 16557 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.558849 16716 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:23.344825 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:26.561198 12565 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:26.553390 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.562299 12641 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:26.561585 12565 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:26.532560 17053 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:13:26.517041 17153 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.549544 17198 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.567343 12565 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.557296 12566 rpcz_store.cc:276] 0709 12:13:25.508382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.120682 (+612300us) service_pool.cc:224] Handling call
0709 12:13:26.555864 (+435182us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:13:26.556135 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.556355 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.556389 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.556442 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:26.556478 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.556541 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.556691 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9019648}
I20260709 12:13:26.573668 12565 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:26.576082 16557 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.567109 16770 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:26.573916 17141 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.990s	user 0.015s	sys 0.000s
W20260709 12:13:26.596189 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.599990 17141 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.590904 17203 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:26.570849 17198 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.624006 12565 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
W20260709 12:13:26.631246 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 330423us
I20260709 12:13:26.635221 12565 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 5
I20260709 12:13:26.635687 12565 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:13:26.636008 12565 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:26.595588 16770 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.616966 16886 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:26.647958 17153 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.651374 17198 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.601182 17094 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.655795 17094 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.658499 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.659533 17058 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:26.623211 17203 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.671015 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.674625 16557 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:26.675107 16557 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.675244 16770 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.675551 16557 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:26.678422 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:26.659957 16886 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.668298 17153 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:26.698971 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.705263 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:26.706046 12572 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:13:26.713366 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:26.714681 12488 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:26.683051 17153 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.716876 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:26.718256 12565 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:26.723909 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:26.724746 12566 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:26.729077 17203 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:26.734459 17202 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:13:26.736294 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2353) took 1411 ms. Trace:
I20260709 12:13:26.736742 12570 rpcz_store.cc:276] 0709 12:13:25.325227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.060544 (+735317us) service_pool.cc:224] Handling call
0709 12:13:26.102857 (+ 42313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:26.103189 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.103222 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:26.134143 (+ 30921us) log.cc:844] Serialized 26 byte log entry
0709 12:13:26.501983 (+367840us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:26.502109 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.502756 (+   647us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:26.736007 (+233251us) raft_consensus.cc:1685] finished
0709 12:13:26.736031 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.736219 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":367060,"threads_started":1,"wal-append.queue_time_us":488624}
I20260709 12:13:26.738582 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:26.739215 12570 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:26.742547 17124 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:26.744107 17124 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:26.746150 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.750751 17205 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.751508 17205 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.711216 16629 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.754491 16629 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.757076 16629 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.759980 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:26.760812 12488 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:26.762388 12488 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:26.723156 17153 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:26.711251 17094 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
W20260709 12:13:26.765141 17195 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.242s	user 0.000s	sys 0.002s
I20260709 12:13:26.794864 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 589) took 1351 ms. Trace:
I20260709 12:13:26.803975 12488 rpcz_store.cc:276] 0709 12:13:25.445373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.761599 (+1316226us) service_pool.cc:224] Handling call
0709 12:13:26.762025 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.793344 (+ 31319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.793606 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:26.794255 17141 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:26.808220 17004 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.808904 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2562) took 2682 ms. Trace:
I20260709 12:13:26.761528 17097 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.726203 17098 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.821025 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:26.821233 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2350) took 1590 ms. Trace:
I20260709 12:13:26.779628 12572 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:26.822714 12565 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:26.825150 12566 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:26.844954 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.845709 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:26.844974 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.847187 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:26.850530 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.845674 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:26.821168 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 411095us
I20260709 12:13:26.852434 12566 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:26.848505 12572 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:26.853001 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:26.855016 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:26.838531 12493 rpcz_store.cc:276] 0709 12:13:24.126311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.556251 (+1429940us) service_pool.cc:224] Handling call
0709 12:13:25.556615 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:25.557040 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:25.557076 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:25.567157 (+ 10081us) log.cc:844] Serialized 52 byte log entry
0709 12:13:26.035434 (+468277us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:26.405792 (+370358us) log.cc:844] Serialized 14 byte log entry
0709 12:13:26.411829 (+  6037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.413565 (+  1736us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:26.764336 (+350771us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:26.808462 (+ 44126us) raft_consensus.cc:1685] finished
0709 12:13:26.808590 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.808811 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5546,"prepare.queue_time_us":535475,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":10375,"raft.queue_time_us":58575,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":9433,"spinlock_wait_cycles":4488064,"thread_start_us":782780,"threads_started":2,"wal-append.queue_time_us":930796}
I20260709 12:13:26.763279 16886 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:26.860132 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.862500 12488 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:13:26.813153 17205 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.866693 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:26.868005 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:26.869827 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:26.794588 17005 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.874089 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2551) took 3327 ms. Trace:
I20260709 12:13:26.874554 12495 rpcz_store.cc:276] 0709 12:13:23.545212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.550485 (+  5273us) service_pool.cc:224] Handling call
0709 12:13:23.550908 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.354119 (+1803211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.872628 (+1518509us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540037e454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:26.872869 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4142972672}
I20260709 12:13:26.875423 17097 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:26.875891 16680 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:26.853691 17058 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.878811 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2564) took 2751 ms. Trace:
W20260709 12:13:26.878947 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.873675 12572 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:26.879940 12496 rpcz_store.cc:276] 0709 12:13:24.126924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.778095 (+1651171us) service_pool.cc:224] Handling call
0709 12:13:25.975374 (+197279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.047497 (+ 72123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.878489 (+830992us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540020c6d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:26.878671 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452454144}
I20260709 12:13:26.881558 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.882391 12565 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:26.884635 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 344093us
I20260709 12:13:26.885972 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:26.889699 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.890179 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.890954 12565 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:26.892884 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.895347 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.896694 12572 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:26.885442 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:26.898687 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:26.901582 12572 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:26.904372 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:26.905731 12488 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:13:26.906580 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.911567 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:26.912462 13253 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:26.877454 17207 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:26.884155 17097 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:26.897058 17094 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:26.918133 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:26.889783 16680 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:26.920106 12640 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:26.889068 16982 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:26.920744 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.921375 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.931644 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
W20260709 12:13:26.930380 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73897us
I20260709 12:13:26.920162 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:26.893400 17124 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:26.932804 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.924717 12640 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:26.925915 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:26.927543 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:26.921703 12495 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:26.889780 17057 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.933859 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:26.942660 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:26.932962 12570 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:26.945369 12564 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:13:26.908638 17046 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:26.937790 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82888.9us
I20260709 12:13:26.933178 12638 raft_consensus.cc:2376] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15. Re-sending same reply.
I20260709 12:13:26.884421 17058 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.930106 17207 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:26.940575 17097 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:26.917124 17125 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.937582 16680 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:26.964164 17227 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:26.964499 16680 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.842747 12571 rpcz_store.cc:276] 0709 12:13:25.230677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.491284 (+260607us) service_pool.cc:224] Handling call
0709 12:13:25.522631 (+ 31347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.522948 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:26.364263 (+841315us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b18002dfd00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:13:26.364453 (+   190us) log.cc:844] Serialized 14 byte log entry
0709 12:13:26.810743 (+446290us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.810829 (+    86us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:26.810896 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.820989 (+ 10093us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:26.821162 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":80770,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":73,"raft.queue_time_us":719,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":29460,"spinlock_wait_cycles":3593472,"thread_start_us":507989,"threads_started":2}
I20260709 12:13:26.940349 17216 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:26.965095 17227 consensus_queue.cc:1243] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:26.961994 17212 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:13:26.970696 17125 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:26.974861 17046 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.976182 17125 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:26.947263 17057 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:26.980348 17057 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:26.945039 17214 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:26.984445 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:26.989416 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:26.994565 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:26.997027 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:26.962847 12640 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:27.004668 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:27.004550 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 588) took 1757 ms. Trace:
I20260709 12:13:26.970559 17219 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:13:27.010854 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105943us
I20260709 12:13:27.012529 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2547) took 3506 ms. Trace:
W20260709 12:13:27.015632 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:27.016038 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83319.5us
I20260709 12:13:27.017030 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:26.988219 17214 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:26.991361 17057 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:27.018468 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58607.1us
I20260709 12:13:27.022552 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.013376 12494 rpcz_store.cc:276] 0709 12:13:23.506049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:23.521771 (+ 15722us) service_pool.cc:224] Handling call
0709 12:13:23.524156 (+  2385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.289000 (+1764844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:27.012258 (+1723258us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:27.012464 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4054057728}
I20260709 12:13:27.019984 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:27.020771 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.023679 12570 raft_consensus.cc:2376] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10. Re-sending same reply.
I20260709 12:13:27.003999 17034 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:27.005023 17097 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:26.961717 16557 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.006726 12490 rpcz_store.cc:276] 0709 12:13:25.246999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.500446 (+1253447us) service_pool.cc:224] Handling call
0709 12:13:26.502066 (+  1620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.502993 (+   927us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:13:26.856499 (+353506us) log.cc:844] Serialized 14 byte log entry
0709 12:13:27.004017 (+147518us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:27.004089 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:13:27.004173 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:27.004278 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:27.004471 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9936,"prepare.queue_time_us":93541,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"thread_start_us":472334,"threads_started":2}
I20260709 12:13:27.027580 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:27.032147 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:27.010075 17125 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:27.030648 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.028151 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:27.040769 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.041445 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.034164 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:27.042116 12571 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:13:27.016968 17094 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:27.005376 16716 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:13:26.988301 17216 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.072247 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.113519 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:27.079383 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:27.077561 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:27.121891 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95543.3us
W20260709 12:13:27.078195 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44094.2us
I20260709 12:13:27.124142 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:27.021572 17219 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:27.014297 17206 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:27.014144 17215 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:27.125501 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0016634999999496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
W20260709 12:13:27.078497 17217 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:13:27.077814 17209 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:13:27.024611 17098 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.192335 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.087420 17222 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:27.024993 17214 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:27.042973 17220 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:27.031476 17057 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.169950 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:27.282797 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 240418us
W20260709 12:13:27.283512 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113706us
I20260709 12:13:27.284327 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:27.284984 17057 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:27.120607 16982 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:27.285854 16982 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:27.287598 16982 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.078918 16557 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.043466 17097 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:27.290187 16557 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:27.265486 12536 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10.
I20260709 12:13:27.270864 12571 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:27.291940 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2559) took 3167 ms. Trace:
I20260709 12:13:27.279206 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.082378 17227 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.183s
I20260709 12:13:27.071673 17005 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:27.264034 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 236537us
I20260709 12:13:27.133284 17198 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:27.126165 17226 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:27.290606 16557 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.130824 17225 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:27.257906 17215 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.034844 17034 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.078353 17228 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.202s
I20260709 12:13:27.123342 17125 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.152s
I20260709 12:13:27.293318 12492 rpcz_store.cc:276] 0709 12:13:24.124318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:24.124508 (+   190us) service_pool.cc:224] Handling call
0709 12:13:24.124995 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:25.464913 (+1339918us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:25.464946 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:25.464990 (+    44us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:25.465020 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:27.282442 (+1817422us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:27.282497 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:27.291877 (+  9380us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1158,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":159,"spinlock_wait_cycles":3080588672}
I20260709 12:13:27.311267 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:27.351684 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 227458us
I20260709 12:13:27.326251 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:27.352020 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:27.345482 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61211.3us
I20260709 12:13:27.295909 17220 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:27.347939 12565 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:27.353392 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:27.314947 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:27.352897 12571 raft_consensus.cc:2368] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12.
I20260709 12:13:27.313992 17098 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:27.261456 17206 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:27.359529 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48358.5us
I20260709 12:13:27.363914 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:27.357456 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:27.366540 17205 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:27.367044 17205 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.367389 17205 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:27.078569 17207 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.370981 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.335461 16557 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:27.335299 17225 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:27.364826 12609 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.672s (SENT)
W20260709 12:13:27.373399 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58480.9us
I20260709 12:13:27.377578 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.382879 12564 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:27.350121 17226 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:27.384888 12571 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:27.316092 17155 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:13:27.348973 17215 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:27.354921 16680 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:27.290598 17097 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.358803 17210 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:27.328545 17094 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.387090 17225 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:27.363016 17005 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20260709 12:13:27.355345 17220 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:27.333546 17198 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.388036 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:27.375550 17232 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:27.449390 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107586us
W20260709 12:13:27.446177 17217 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:13:27.421198 17236 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:27.454725 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97284.6us
W20260709 12:13:27.456718 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:27.457727 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:27.459595 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:27.448606 17232 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.452207 17210 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:27.464841 17232 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:27.476728 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91334.8us
I20260709 12:13:27.349436 17219 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:27.505681 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119819us
I20260709 12:13:27.506979 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:27.463548 17216 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:27.470202 17198 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.466034 17210 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.508358 17241 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:27.515954 17141 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:27.508989 17006 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.013s
W20260709 12:13:27.524652 17233 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:13:27.503580 17231 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:27.547765 17198 raft_consensus.cc:2764] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:13:27.552785 17034 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.547194 17210 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:27.558579 17198 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:27.561098 17241 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.564853 17241 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:27.569512 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106234us
I20260709 12:13:27.570369 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:27.574681 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81350.5us
I20260709 12:13:27.578634 12536 leader_election.cc:400] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:27.546017 17141 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.464011 17236 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:27.568290 17153 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.770s	user 0.010s	sys 0.004s
W20260709 12:13:27.581883 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74973.2us
I20260709 12:13:27.618798 12609 leader_election.cc:400] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12.
I20260709 12:13:27.619453 12609 leader_election.cc:403] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:13:27.566002 17214 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.581674 12536 leader_election.cc:403] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:27.589519 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:27.660112 16716 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:27.678124 17235 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
W20260709 12:13:27.666061 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:27.681488 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:27.680612 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102138us
W20260709 12:13:27.683495 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94114.1us
I20260709 12:13:27.687870 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.553530 17205 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.689438 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:27.689441 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.692696 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:27.695777 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.716100 16780 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.723734 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.724328 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.724893 12642 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:27.725108 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.580195 17231 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:27.726145 12641 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:27.732849 12644 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:27.733707 17206 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.747145 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:27.735826 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2571) took 2776 ms. Trace:
I20260709 12:13:27.476892 17226 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.736670 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.749660 12563 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:13:27.749143 12489 rpcz_store.cc:276] 0709 12:13:24.959718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.200030 (+1240312us) service_pool.cc:224] Handling call
0709 12:13:26.369678 (+169648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:26.370060 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.370092 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:26.370821 (+   729us) log.cc:844] Serialized 26 byte log entry
0709 12:13:26.683428 (+312607us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:26.683479 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:26.683552 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:27.349168 (+665616us) raft_consensus.cc:1685] finished
0709 12:13:27.349189 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:27.735545 (+386356us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b5400191c50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:27.735761 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4491,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":388873984,"thread_start_us":312172,"threads_started":1,"wal-append.queue_time_us":525228}
I20260709 12:13:27.755856 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.736917 17225 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.724749 12638 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:27.738628 17236 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.739624 12490 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:27.659425 17098 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:27.774469 17098 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.765586 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.778713 12638 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:27.781992 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.743048 17215 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.731448 17216 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:27.662146 17219 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:27.584381 17141 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:27.597496 17046 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:27.787492 12641 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:27.756669 12489 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:27.758914 12494 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:27.761821 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:27.761721 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.762662 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:27.790002 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.790266 12496 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:27.757302 17244 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:13:27.684908 17246 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:27.765864 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.745170 17005 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:27.745296 17130 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:27.778000 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.782770 12644 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:27.733687 16557 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.671118 17153 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.793514 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.804433 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.808357 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.821924 12642 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:27.764353 17250 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:27.815434 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:27.812708 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.817425 12494 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:27.781687 17214 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:27.791863 17219 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.784612 16680 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:27.814914 12493 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:27.753069 17231 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.830049 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.831090 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.835419 12641 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:27.827296 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.813620 17046 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:27.837484 12642 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:27.792115 17216 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.846076 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.848039 12493 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:27.850395 12536 connection.cc:256] RPC call timeout handler was delayed by -2.084114326999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260709 12:13:27.850840 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0067143270000543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 12:13:27.851109 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7106143270000302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:13:27.851356 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6890143269999953s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2041s for a response
W20260709 12:13:27.851615 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3497143269999015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24180000000000001s for a response
W20260709 12:13:27.851889 12536 connection.cc:256] RPC call timeout handler was delayed by -1.296614327000043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2477s for a response
W20260709 12:13:27.852160 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0608143270001165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2739s for a response
I20260709 12:13:27.852756 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:27.853411 12563 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:27.831986 17219 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:27.857934 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.830631 17250 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.860582 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.861269 12641 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:27.861827 17250 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.861958 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.863493 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.864615 16557 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.867352 17153 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.869082 17225 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.856177 17220 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.870466 17205 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.870225 17046 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.820079 17130 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.872318 17220 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.872831 17130 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:27.875419 17216 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:27.876603 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.876971 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.877265 12644 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:27.877735 12641 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:27.879006 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.880399 17231 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.881475 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.864614 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.883906 12642 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:27.889565 17206 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:27.891150 17002 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.882126 12567 raft_consensus.cc:2376] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14. Re-sending same reply.
I20260709 12:13:27.893507 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.894315 12493 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:27.897385 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:27.899206 17255 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:13:27.899099 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.899997 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.900774 12568 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:27.904666 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.905678 12489 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:27.907466 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.908743 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.910037 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:27.910300 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:27.910606 17248 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:13:27.911404 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.912056 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.855746 17035 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:27.909420 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:27.912734 12568 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:27.913043 17035 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.913412 12494 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:27.913858 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.901600 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.914542 12568 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:27.914788 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.882519 12641 pending_rounds.cc:85] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:27.920135 12641 pending_rounds.cc:107] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:13:27.920188 12563 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:27.920711 12641 raft_consensus.cc:2889] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:27.921886 12638 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:27.922525 17260 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:27.923041 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.923882 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:27.929834 12638 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:27.931447 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.932866 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.935483 16716 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12.
I20260709 12:13:27.923180 17260 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.942730 17215 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:27.945118 17035 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:27.945858 17130 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:27.945019 17249 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:27.947090 16886 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:27.967790 16886 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.968075 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.968863 12567 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:27.970104 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.948633 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.970870 12567 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:27.971201 12565 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:13:27.971747 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.950728 17000 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.951880 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.974795 12572 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:27.975628 17219 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:27.980079 17249 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.981873 12640 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:27.982915 17249 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.983548 17130 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.953012 12496 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:27.956033 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:27.956686 12644 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:27.945197 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:27.973310 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:27.958478 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:27.965341 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:27.972194 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.949606 17128 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:27.989809 12567 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:27.989955 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:27.995460 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:27.995745 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:27.989456 12565 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:27.996284 12568 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:27.991703 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:27.998239 16716 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:27.998728 16716 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.966148 17210 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.952360 17232 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:27.999084 16716 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.002851 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:28.003566 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:28.004580 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:27.997139 17000 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:28.006892 17000 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:28.007308 17000 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.008646 17210 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.009882 17232 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.005556 12565 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:28.012662 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:27.997495 17053 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:27.996160 12563 raft_consensus.cc:2376] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15. Re-sending same reply.
I20260709 12:13:28.009938 12495 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:28.016450 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.012233 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:28.014686 12640 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:13:28.015219 16551 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:28.002202 17215 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:28.018617 17215 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:27.999150 16901 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.019064 17215 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:28.020373 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2417) took 1011 ms. Trace:
I20260709 12:13:28.018316 16551 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.020839 12564 rpcz_store.cc:276] 0709 12:13:27.008589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:27.014645 (+  6056us) service_pool.cc:224] Handling call
0709 12:13:28.020166 (+1005521us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:28.020298 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779781504}
I20260709 12:13:28.022990 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:27.993114 12572 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:27.997617 17198 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:28.019786 12496 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 3 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:28.027829 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:28.032886 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:27.993880 12646 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:28.035496 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 555) took 1117 ms. Trace:
I20260709 12:13:28.034561 12565 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:13:27.985623 17141 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.040393 12494 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 14. Ops: []
I20260709 12:13:28.040531 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2427) took 1022 ms. Trace:
W20260709 12:13:28.038991 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74244us
I20260709 12:13:28.046639 12567 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:28.046806 12565 rpcz_store.cc:276] 0709 12:13:27.018749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.033824 (+1015075us) service_pool.cc:224] Handling call
0709 12:13:28.034194 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.040220 (+  6026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.040457 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:28.042078 12572 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:28.040068 12640 rpcz_store.cc:276] 0709 12:13:26.900075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.901247 (+  1172us) service_pool.cc:224] Handling call
0709 12:13:26.919965 (+ 18718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:27.981658 (+1061693us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b58005429d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:13:28.017299 (+ 35641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.017471 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2245567744}
I20260709 12:13:28.047652 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.043658 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.044507 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:28.037072 17060 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:13:28.048528 12494 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:28.048514 12490 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:28.047345 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:28.051433 12493 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:28.031541 17053 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:28.051399 17034 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:28.056730 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:28.056872 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:28.057859 17249 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:28.058892 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.060928 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.062196 16551 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.069765 17260 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.073002 17141 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:28.049178 17210 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:28.074700 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:28.074587 17210 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.075469 12640 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:28.077239 17251 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:28.078506 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2026) took 3769 ms. Trace:
I20260709 12:13:28.078984 17251 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:28.079360 12637 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:28.080060 12637 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:28.080410 16680 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:28.039846 17130 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.045600 17152 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:28.081597 17152 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:28.082392 17077 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.082695 17152 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.079030 12491 rpcz_store.cc:276] 0709 12:13:24.309091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.054890 (+1745799us) service_pool.cc:224] Handling call
0709 12:13:26.207680 (+152790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.232735 (+ 25055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.078188 (+1845453us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400191250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:28.078391 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338737920}
I20260709 12:13:28.094473 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:28.095896 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:28.086860 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:28.107033 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:28.108129 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47295.1us
I20260709 12:13:28.081095 12637 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.109586 12637 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:28.114158 12640 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:28.115537 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:28.079481 17251 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:28.135785 17246 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:13:28.117671 17205 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:13:28.119266 17071 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:13:28.117759 17206 raft_consensus.cc:2764] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:13:28.145849 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:28.106070 17053 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:28.103200 17046 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.084s
I20260709 12:13:28.187625 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:28.210795 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:28.188575 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.212071 12493 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:28.086886 17210 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.113024 17228 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:28.207456 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:28.116870 17203 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:28.118888 16901 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:28.135885 17207 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:28.120289 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:28.191277 16680 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:28.205214 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:28.212431 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66732.4us
I20260709 12:13:28.200803 17034 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:28.226928 12637 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:28.200524 17249 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:28.192679 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.222859 12641 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.136682 16982 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:13:28.218835 17232 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:28.223640 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:28.227684 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:28.235965 12640 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:28.236744 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 551) took 2731 ms. Trace:
I20260709 12:13:28.228019 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:28.109732 17219 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:28.234917 17164 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 12:13:28.240407 17214 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:28.241854 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49299.4us
I20260709 12:13:28.242637 17210 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:28.242830 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.231689 17034 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:28.253604 12493 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:28.105751 17000 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.244865 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:28.270004 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:28.237402 12637 rpcz_store.cc:276] 0709 12:13:25.500030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:25.500185 (+   155us) service_pool.cc:224] Handling call
0709 12:13:25.500967 (+   782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.079233 (+2578266us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b5400292d20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:13:28.230724 (+151491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.231545 (+   821us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111569152}
I20260709 12:13:28.271497 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:28.231413 16901 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:28.248799 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2420) took 1237 ms. Trace:
I20260709 12:13:28.221745 17203 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:28.253911 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:28.247267 17261 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:13:28.237423 17207 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:28.277592 12568 rpcz_store.cc:276] 0709 12:13:27.011514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.033697 (+1022183us) service_pool.cc:224] Handling call
0709 12:13:28.034258 (+   561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.034543 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:28.034878 (+   335us) log.cc:844] Serialized 14 byte log entry
0709 12:13:28.248163 (+213285us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:28.248257 (+    94us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:28.248320 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.248408 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:28.248670 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36606,"prepare.queue_time_us":690,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":5460,"raft.queue_time_us":113,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":16503,"spinlock_wait_cycles":20352,"thread_start_us":212718,"threads_started":1,"wal-append.queue_time_us":221337}
I20260709 12:13:28.280354 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 3 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:28.281767 17214 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:28.272998 17210 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:28.246819 17239 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:28.252398 17216 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:28.294824 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:28.286954 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:28.255796 17035 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:28.263269 17034 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:28.289482 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:28.284358 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.280516 17203 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:28.313587 17214 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:28.318711 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.318943 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:28.318799 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.320433 17128 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:28.324142 17266 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:13:28.290364 17236 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:28.329582 17236 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:28.329916 12493 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:28.334234 12493 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:28.335081 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:28.285147 16680 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.336629 12642 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:28.337687 17160 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:28.345814 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.354179 17214 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.354776 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 552) took 1475 ms. Trace:
I20260709 12:13:28.355490 13260 rpcz_store.cc:276] 0709 12:13:26.879141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.879408 (+   267us) service_pool.cc:224] Handling call
0709 12:13:26.891769 (+ 12361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.931336 (+ 39567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:26.931388 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:26.931455 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:26.931503 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.354389 (+1422886us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:28.354461 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:28.354667 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90237824}
I20260709 12:13:28.361586 17000 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.362670 17035 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:28.363176 17035 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:28.366187 12644 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20260709 12:13:28.320731 17239 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.373723 17239 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.375631 16716 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.328181 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
W20260709 12:13:28.389043 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66052.4us
I20260709 12:13:28.330072 17236 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:28.379529 17207 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.370651 12571 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Committing config change with OpId 9.7: config changed from index 3 to 7, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:28.380434 17251 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.370617 16901 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:28.386713 17262 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.336s	user 0.000s	sys 0.001s
I20260709 12:13:28.389323 17128 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:28.398638 17128 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.400444 17210 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.406176 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:28.407672 12640 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:13:28.412281 12637 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:28.462905 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 347438us
I20260709 12:13:28.464267 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.469332 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:28.470292 12640 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.408164 17239 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:28.482710 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.361418 17053 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.488027 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.461542 17133 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:13:28.479676 17216 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:28.464847 17215 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.517493 17251 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.487435 17271 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:13:28.491644 17249 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:13:28.531837 12640 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:28.542918 12638 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:28.491971 17239 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.546396 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2351) took 3288 ms. Trace:
I20260709 12:13:28.549130 12569 rpcz_store.cc:276] 0709 12:13:25.258068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.059816 (+801748us) service_pool.cc:224] Handling call
0709 12:13:26.076318 (+ 16502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:26.299148 (+222830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.545461 (+2246313us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:28.546305 (+   844us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400084224}
I20260709 12:13:28.552459 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:28.519567 17216 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.555804 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:13:28.559381 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.535645 17034 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.567096 12569 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:13:28.567536 17273 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:13:28.581286 17249 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:28.582000 17249 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:28.582598 17249 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.582007 17203 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.591509 17203 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.598023 17215 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:28.600297 12639 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:13:28.600477 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:28.601663 17216 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.602815 17216 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:13:28.603974 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 598) took 1238 ms. Trace:
I20260709 12:13:28.604038 12489 raft_consensus.cc:1217] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Deduplicated request from leader. Original: 10.7->[12.8-12.8]   Dedup: 12.8->[]
I20260709 12:13:28.605297 12489 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:13:28.606266 17239 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.554526 17275 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:28.607468 17275 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:28.585466 17035 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.607882 17275 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:28.608582 12643 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:28.610081 16716 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:28.614607 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.618427 12639 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:28.619325 12639 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:28.620415 12639 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.622292 12639 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:28.604425 12492 rpcz_store.cc:276] 0709 12:13:27.365953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:27.366590 (+   637us) service_pool.cc:224] Handling call
0709 12:13:27.367019 (+   429us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:27.448471 (+ 81452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:27.448520 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:27.451424 (+  2904us) log.cc:844] Serialized 52 byte log entry
0709 12:13:27.567348 (+115924us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:27.848813 (+281465us) log.cc:844] Serialized 14 byte log entry
0709 12:13:27.849076 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.603411 (+754335us) spinlock_profiling.cc:243] Waited 80.9 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:28.603461 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:28.603507 (+    46us) raft_consensus.cc:1685] finished
0709 12:13:28.603728 (+   221us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:28.603917 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":321355,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":22846,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":34525,"spinlock_wait_cycles":186031488,"thread_start_us":354667,"threads_started":2,"wal-append.queue_time_us":356380}
I20260709 12:13:28.630707 12640 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:28.633747 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2384) took 1080 ms. Trace:
I20260709 12:13:28.634276 12572 rpcz_store.cc:276] 0709 12:13:27.553598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.087685 (+534087us) service_pool.cc:224] Handling call
0709 12:13:28.171369 (+ 83684us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:28.171852 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:13:28.171892 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:28.368383 (+196491us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:28.368838 (+   455us) log.cc:844] Serialized 26 byte log entry
0709 12:13:28.530976 (+162138us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:13:28.531026 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.531103 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:28.612660 (+ 81557us) raft_consensus.cc:1685] finished
0709 12:13:28.612680 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:28.633601 (+ 20921us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":619,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":14307,"spinlock_wait_cycles":9321088,"thread_start_us":153508,"threads_started":1,"wal-append.queue_time_us":241926}
I20260709 12:13:28.637312 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:28.677488 12643 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:13:28.627080 17035 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.688491 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:28.689287 12572 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.690619 12639 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 12:13:28.696069 12639 pending_rounds.cc:85] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 9
I20260709 12:13:28.696152 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2431) took 1616 ms. Trace:
I20260709 12:13:28.700006 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2236) took 1012 ms. Trace:
I20260709 12:13:28.701285 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:28.696326 12639 pending_rounds.cc:107] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:13:28.702368 12639 raft_consensus.cc:2889] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:28.700503 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.696586 12567 rpcz_store.cc:276] 0709 12:13:27.079346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.025583 (+946237us) service_pool.cc:224] Handling call
0709 12:13:28.026024 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.049565 (+ 23541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.695870 (+646305us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:28.696082 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30956800}
I20260709 12:13:28.710500 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:28.711179 12567 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:28.578028 17137 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.104s
I20260709 12:13:28.700589 12646 rpcz_store.cc:276] 0709 12:13:27.687329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:27.687520 (+   191us) service_pool.cc:224] Handling call
0709 12:13:28.699732 (+1012212us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:28.699920 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":652752256}
W20260709 12:13:28.717124 13218 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.813s (SENT)
I20260709 12:13:28.714861 13258 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:13:28.713515 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.720031 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:28.720809 12567 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:28.681594 17053 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:28.724745 13260 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [12.9-14.10]
W20260709 12:13:28.723814 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 235855us
I20260709 12:13:28.729808 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.728282 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2439) took 1105 ms. Trace:
I20260709 12:13:28.711989 17242 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:28.734009 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.710269 17205 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:28.731376 12565 rpcz_store.cc:276] 0709 12:13:27.623221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.319271 (+696050us) service_pool.cc:224] Handling call
0709 12:13:28.319550 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.319747 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:28.393009 (+ 73262us) log.cc:844] Serialized 14 byte log entry
0709 12:13:28.727942 (+334933us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:28.727989 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:28.728019 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.728113 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:28.728226 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":76815,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":54496,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":334414,"threads_started":2,"wal-append.queue_time_us":192720}
I20260709 12:13:28.721803 16780 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:28.749145 13254 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:28.750485 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.731178 17222 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:28.734760 17246 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:28.789513 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:28.760910 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:28.738500 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.764967 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.765422 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
W20260709 12:13:28.803112 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.766232 13254 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:13:28.739286 17271 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:28.770001 12570 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:28.776374 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2421) took 1764 ms. Trace:
I20260709 12:13:28.777693 17285 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:13:28.732739 17077 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:13:28.778681 17034 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:28.740743 16629 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:13:28.814291 17077 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:28.814723 17077 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:28.815523 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.815807 17077 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.816998 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.734570 17251 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 12:13:28.749495 17242 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.732774 17160 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260709 12:13:28.786964 12646 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:28.792954 12492 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:28.796031 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:28.798027 17133 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:28.721572 17260 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:13:28.755169 17058 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.805156 12572 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:28.869449 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:28.804651 12563 rpcz_store.cc:276] 0709 12:13:27.012374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.033754 (+1021380us) service_pool.cc:224] Handling call
0709 12:13:28.050959 (+ 17205us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:28.093991 (+ 43032us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:28.094027 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:28.596518 (+502491us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b18003973c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:13:28.597189 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:13:28.732346 (+135157us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:28.732537 (+   191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.732602 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:28.775988 (+ 43386us) raft_consensus.cc:1685] finished
0709 12:13:28.776019 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:28.776294 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38273,"raft.queue_time_us":38391,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":15933,"spinlock_wait_cycles":26895616,"thread_start_us":134658,"threads_started":1,"wal-append.queue_time_us":176449}
I20260709 12:13:28.811280 17290 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:28.813032 17280 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:13:28.823815 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.819636 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.822093 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.817909 13253 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 10: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:28.818401 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:28.897794 13254 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:28.898059 13261 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:28.818776 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:28.866206 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70269.3us
W20260709 12:13:28.804795 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66413.1us
I20260709 12:13:28.819401 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:28.822566 17282 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:13:28.745119 17097 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 12:13:28.817512 17077 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:28.821887 17098 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:28.749663 17205 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.911046 13258 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:28.914495 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:28.922152 17053 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:28.930073 17035 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:13:28.930744 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.888682 17152 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:28.949007 17152 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:28.949560 17152 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:28.892668 17290 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:28.895991 17228 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:28.947309 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.954391 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.958114 13260 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:28.958282 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:28.962298 12492 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:13:28.960163 17228 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:28.963718 17236 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.559s	user 0.006s	sys 0.007s
I20260709 12:13:28.964444 17236 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:28.965216 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.966822 17109 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:28.968847 17275 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:28.954551 17058 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:28.986570 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:28.991022 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:28.991557 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:28.992535 12490 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:28.961021 17290 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:28.998912 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:28.937266 17098 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:29.000773 12569 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:29.002897 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2066) took 1099 ms. Trace:
I20260709 12:13:29.005524 12642 rpcz_store.cc:276] 0709 12:13:27.903486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.337359 (+433873us) service_pool.cc:224] Handling call
0709 12:13:28.387157 (+ 49798us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:28.387965 (+   808us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:28.388005 (+    40us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:13:28.392064 (+  4059us) log.cc:844] Serialized 78 byte log entry
0709 12:13:28.632614 (+240550us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:28.687244 (+ 54630us) log.cc:844] Serialized 14 byte log entry
0709 12:13:28.687511 (+   267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:28.687619 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:29.002569 (+314950us) raft_consensus.cc:1685] finished
0709 12:13:29.002595 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:29.002814 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":55360,"raft.queue_time_us":768,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":27448,"spinlock_wait_cycles":10159360,"thread_start_us":164743,"threads_started":1,"wal-append.queue_time_us":329916}
I20260709 12:13:29.011303 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:29.012148 12492 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:28.984510 17035 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:28.988291 17285 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:29.019307 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109197us
I20260709 12:13:29.022503 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:29.000734 17058 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:29.004472 17289 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:29.004148 17288 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:28.964468 17228 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:29.013317 17034 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:29.013689 16780 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:29.076759 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 207401us
I20260709 12:13:29.077883 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:29.082377 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57231us
I20260709 12:13:29.083235 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:29.022763 17035 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.032011 17222 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:29.100028 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108578us
I20260709 12:13:29.101383 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:29.032985 17289 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:29.106223 12492 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:29.157171 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:29.025328 17285 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.065999 17288 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.103693 17228 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:29.109118 17249 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.526s	user 0.016s	sys 0.000s
I20260709 12:13:29.172483 17304 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:29.176292 12492 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 12:13:29.173405 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90250.3us
I20260709 12:13:29.181617 17308 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:29.165560 17222 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:29.182472 17308 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.159777 17242 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:29.159030 17205 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:29.165601 17236 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:29.195545 17152 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.218767 17296 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:29.174099 16780 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.166455 17058 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.3, Last appended by leader: 3, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:29.224095 17301 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:13:29.219631 17296 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.166683 17137 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20260709 12:13:29.253786 12460 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.514s (SENT) [suppressed 13 similar messages]
I20260709 12:13:29.211386 17305 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:29.189072 17312 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:29.281978 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:29.176966 17216 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:13:29.210798 17275 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:29.179507 17249 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:29.296550 17303 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:13:29.469305 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 311948us
I20260709 12:13:29.441169 17098 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:29.466145 17275 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:29.509258 17239 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:29.514156 17306 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
W20260709 12:13:29.312999 17297 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260709 12:13:29.517578 17290 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.003s
I20260709 12:13:29.466004 17305 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.151s
W20260709 12:13:29.520416 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 340986us
I20260709 12:13:29.678773 17198 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:29.590126 17239 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:29.523806 17275 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:29.638286 17290 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:29.776602 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2459) took 1169 ms. Trace:
I20260709 12:13:29.767815 17198 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:30.078738 12565 rpcz_store.cc:276] 0709 12:13:28.607089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.740384 (+133295us) service_pool.cc:224] Handling call
0709 12:13:28.741299 (+   915us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:28.756630 (+ 15331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:28.756708 (+    78us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:28.810143 (+ 53435us) log.cc:844] Serialized 26 byte log entry
0709 12:13:29.029100 (+218957us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:29.029153 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:29.029239 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:29.776207 (+746968us) raft_consensus.cc:1685] finished
0709 12:13:29.776241 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:29.776490 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14653,"raft.queue_time_us":42674,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":178,"thread_start_us":161535,"threads_started":1,"wal-append.queue_time_us":295186}
I20260709 12:13:29.979579 17035 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.081929 17275 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:30.094278 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2253) took 2162 ms (client timeout 1847 ms). Trace:
W20260709 12:13:30.094880 12638 rpcz_store.cc:269] 0709 12:13:27.931570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:27.997389 (+ 65819us) service_pool.cc:224] Handling call
0709 12:13:30.093990 (+2096601us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:30.094188 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1182960512}
I20260709 12:13:30.094370 17035 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:30.098692 17035 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.101882 17035 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:29.797374 17302 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.332s	user 0.000s	sys 0.000s
I20260709 12:13:30.107244 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2270) took 1280 ms. Trace:
I20260709 12:13:30.107766 12639 rpcz_store.cc:276] 0709 12:13:28.826488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.826729 (+   241us) service_pool.cc:224] Handling call
0709 12:13:28.827265 (+   536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:28.827801 (+   536us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:28.827848 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:28.828533 (+   685us) log.cc:844] Serialized 26 byte log entry
0709 12:13:29.035425 (+206892us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:29.036186 (+   761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:29.036299 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:30.106936 (+1070637us) raft_consensus.cc:1685] finished
0709 12:13:30.106964 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:30.107165 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38038,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":24479,"thread_start_us":206044,"threads_started":1,"wal-append.queue_time_us":440045}
I20260709 12:13:30.077308 17296 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:29.910782 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2464) took 1130 ms. Trace:
I20260709 12:13:30.113554 12564 rpcz_store.cc:276] 0709 12:13:28.780537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.790464 (+  9927us) service_pool.cc:224] Handling call
0709 12:13:28.790845 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:28.791240 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:28.791270 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:28.791907 (+   637us) log.cc:844] Serialized 26 byte log entry
0709 12:13:29.011577 (+219670us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:29.011623 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:29.011695 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:29.910448 (+898753us) raft_consensus.cc:1685] finished
0709 12:13:29.910478 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:29.910696 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13497,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":2673,"thread_start_us":219194,"threads_started":1,"wal-append.queue_time_us":378432}
W20260709 12:13:30.049540 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 338754us
W20260709 12:13:30.118357 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:29.932766 17289 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:29.934007 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 832638us
I20260709 12:13:30.121227 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:30.123545 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:30.125960 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:30.087522 12391 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 6 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:30.135675 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:30.146787 12490 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.153s	user 0.020s	sys 0.004s
I20260709 12:13:30.147656 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:30.148537 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2051) took 1176 ms. Trace:
I20260709 12:13:30.148907 12490 rpcz_store.cc:276] 0709 12:13:28.971999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.972298 (+   299us) service_pool.cc:224] Handling call
0709 12:13:30.148428 (+1176130us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:30.153990 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.012s	sys 0.014s
W20260709 12:13:30.154752 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.012s	sys 0.014s
I20260709 12:13:30.088996 17290 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:30.080498 17198 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.049314 17303 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:30.049019 17309 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.130363 17289 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:30.086650 17152 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:30.157425 13218 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.006s	sys 0.006s
I20260709 12:13:29.400754 17312 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:30.146396 17046 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.038s	sys 0.038s
W20260709 12:13:30.153748 17304 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.007s	sys 0.013s
W20260709 12:13:30.186808 13218 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.007s	sys 0.006s
W20260709 12:13:30.189507 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 668190us
I20260709 12:13:30.206297 17239 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.210836 17312 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:30.192267 17304 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.914s	user 0.007s	sys 0.014s
I20260709 12:13:30.211902 17304 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:30.185242 17046 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.077s	user 0.039s	sys 0.038s
I20260709 12:13:30.182282 17198 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:30.186298 17035 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:30.186058 17290 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:30.222437 16886 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.192001 17249 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:30.225659 17164 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.227201 17206 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:30.230897 17285 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:30.231469 17228 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.055s	user 0.009s	sys 0.004s
W20260709 12:13:30.234807 17252 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.416s	user 0.035s	sys 0.022s
W20260709 12:13:30.240935 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2457) took 1641 ms (client timeout 1981 ms). Trace:
W20260709 12:13:30.250105 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.259474 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:30.259632 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:30.262032 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:30.262032 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:30.263437 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:30.248585 17035 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:30.275440 12566 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:30.256204 17239 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.262284 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:30.276145 12572 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:30.276600 12563 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:13:30.274101 12570 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:30.266952 12565 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:13:30.279693 12568 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:30.252323 17290 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:30.272994 12569 rpcz_store.cc:269] 0709 12:13:28.599888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.600214 (+   326us) service_pool.cc:224] Handling call
0709 12:13:30.240629 (+1640415us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:30.240823 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917349120}
I20260709 12:13:30.274029 12567 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:30.137318 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:30.282920 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:30.286285 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149058us
I20260709 12:13:30.268103 17206 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:30.291216 12564 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:30.285540 12565 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:13:30.305470 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:30.279590 16528 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.307507 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:30.298612 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:30.308887 12569 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:30.309022 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:30.311096 12567 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:30.322115 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:30.323422 12490 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:13:30.328735 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.307077 17285 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:30.269894 17164 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.292361 17290 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:30.279362 17210 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:30.292503 17035 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:30.338701 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:30.339627 12572 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:13:30.349794 12569 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:13:30.350010 12610 connection.cc:256] RPC call timeout handler was delayed by -2.2587064750000536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
W20260709 12:13:30.350368 12610 connection.cc:256] RPC call timeout handler was delayed by -2.2281064750000041s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260709 12:13:30.350663 12610 connection.cc:256] RPC call timeout handler was delayed by -2.1552064750001136s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20260709 12:13:30.350741 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:30.351879 12610 connection.cc:256] RPC call timeout handler was delayed by -2.1318064750000758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20260709 12:13:30.352137 12572 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:30.352329 12610 connection.cc:256] RPC call timeout handler was delayed by -2.0141852069998549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20260709 12:13:30.352744 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:30.340298 12567 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:13:30.343125 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:30.378055 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:30.384647 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.395610 12566 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:13:30.292186 17239 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.347276 17034 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:30.307171 16528 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.416483 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:30.377038 17249 raft_consensus.cc:2764] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:13:30.418602 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:30.418848 12567 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:13:30.352870 12610 connection.cc:256] RPC call timeout handler was delayed by -1.6836064750000332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
W20260709 12:13:30.423362 12610 connection.cc:256] RPC call timeout handler was delayed by -1.4019064750000325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25470000000000004s for a response
I20260709 12:13:30.423503 12566 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:13:30.423781 12610 connection.cc:256] RPC call timeout handler was delayed by -1.3974064750000252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25520000000000004s for a response
I20260709 12:13:30.403231 17034 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:30.351305 17210 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.397766 17289 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
W20260709 12:13:30.426893 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.431159 12490 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:30.436972 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 966416us
W20260709 12:13:30.438053 12460 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.996s	user 0.007s	sys 0.015s
W20260709 12:13:30.439698 12460 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.998s	user 0.007s	sys 0.015s
W20260709 12:13:30.441579 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.18776e+06us
I20260709 12:13:30.177599 17309 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.446831 17309 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.222821 17313 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.225s
I20260709 12:13:30.224288 17304 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.270074 17303 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:30.436640 17216 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.012s	sys 0.010s
I20260709 12:13:30.427480 16886 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.429083 17034 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:30.426790 17320 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:30.469822 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143039us
I20260709 12:13:30.464659 17058 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:30.470693 12460 connection.cc:256] RPC call timeout handler was delayed by -1.0954902700000275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
W20260709 12:13:30.438529 17305 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.010s	sys 0.012s
I20260709 12:13:30.420136 17206 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:30.492580 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.000s	sys 0.023s
W20260709 12:13:30.493233 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.000s	sys 0.023s
I20260709 12:13:30.496004 12567 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:13:30.499044 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:30.500059 12567 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:13:30.438949 17249 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:13:30.469151 17309 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:30.481772 17216 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.023s	user 0.013s	sys 0.011s
W20260709 12:13:30.522722 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86538.4us
I20260709 12:13:30.527192 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:30.528424 12569 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:30.531828 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92356.1us
W20260709 12:13:30.490403 17305 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.011s	sys 0.013s
I20260709 12:13:30.535353 17228 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:30.479475 17323 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:30.491007 17164 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.542955 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:30.493265 17330 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:30.542833 17206 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:30.488955 16886 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.489959 17058 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:30.556563 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.566433 17289 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:30.544031 17249 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.695389 12569 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:30.493292 17320 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.560130 17309 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.578800 17323 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.603061 17330 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:30.815130 13217 connection.cc:256] RPC call timeout handler was delayed by -2.0694362480003292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
I20260709 12:13:30.706462 17339 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 12:13:30.579598 17322 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.200s
I20260709 12:13:30.738314 17343 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 12:13:30.706938 17344 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:13:30.561560 17124 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
W20260709 12:13:30.818436 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.815433 17246 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:30.818897 17289 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:30.829919 12567 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:30.830431 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 269193us
I20260709 12:13:30.832580 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.832830 17321 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:30.834569 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.839202 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:30.844194 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 355937us
I20260709 12:13:30.814937 12572 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:13:30.837591 17331 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.305s	user 0.006s	sys 0.010s
W20260709 12:13:30.826604 13217 connection.cc:256] RPC call timeout handler was delayed by -1.8327362480001739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
W20260709 12:13:30.860441 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:30.868546 13217 connection.cc:256] RPC call timeout handler was delayed by -1.794573119000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
W20260709 12:13:30.869010 13217 connection.cc:256] RPC call timeout handler was delayed by -1.5074731189999966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20510000000000003s for a response
I20260709 12:13:30.870623 13261 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:30.870932 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.871359 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.874276 13258 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:30.875902 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52449.7us
I20260709 12:13:30.846524 17336 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.858860 17246 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:30.847326 17210 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.894193 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.895344 12639 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:30.896584 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.828318 17249 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.900161 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:30.903642 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.904546 12646 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:30.904801 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:30.904954 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.905875 12639 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
W20260709 12:13:30.907078 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:30.909476 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.910208 12646 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:30.911032 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:30.911516 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.912070 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.912557 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.867018 17304 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.915992 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:30.917239 12639 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:30.921188 12646 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:30.866452 17239 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:30.921082 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:30.923384 12641 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:30.924813 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.963454 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.964221 12644 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:30.964551 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64482.9us
I20260709 12:13:30.967849 12638 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:30.969441 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.968290 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:30.964416 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:30.966548 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:30.970410 12640 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:30.977101 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:30.978253 12643 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:30.866011 17349 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:13:30.984211 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:30.992383 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:30.993868 17348 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.994608 17348 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.914239 17215 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.921039 17305 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:30.921237 17338 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:31.004407 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.007876 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:30.883957 17321 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.008731 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.008792 13256 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:31.007962 17215 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:30.997820 17349 consensus_queue.cc:1243] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:13:30.975805 17325 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:31.018690 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:31.019562 12643 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:13:31.019907 12640 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:30.926620 17304 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:30.893617 17336 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.025234 13258 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:13:31.004920 17006 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:31.026996 13261 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:31.028290 12639 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:31.031414 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.026382 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.036252 13262 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:31.047969 17336 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.051645 16886 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:31.034689 17006 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.013903 17305 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.012416 17338 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.059894 17330 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.234932 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2574) took 3216 ms. Trace:
I20260709 12:13:31.240731 12642 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:31.247468 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:31.402046 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:31.247069 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:31.248061 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 279634us
I20260709 12:13:31.250743 12638 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:13:31.256861 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.263446 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2578) took 3238 ms. Trace:
W20260709 12:13:31.253844 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.408862 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:31.264559 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 601) took 3359 ms (client timeout 1650 ms). Trace:
I20260709 12:13:31.270943 17058 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.271672 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:31.272460 17198 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.052s	user 0.021s	sys 0.000s
I20260709 12:13:31.278867 17215 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.282557 17226 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.284031 17228 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.263476 16886 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:31.300302 17006 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:31.321671 17275 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.227s	user 0.006s	sys 0.018s
W20260709 12:13:31.325697 17290 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.975s	user 0.015s	sys 0.005s
W20260709 12:13:31.326581 17342 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.308s	user 0.001s	sys 0.008s
I20260709 12:13:31.326874 17341 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:31.327611 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:31.332926 17309 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:31.317286 17035 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.966s	user 0.012s	sys 0.018s
W20260709 12:13:31.414078 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 319213us
I20260709 12:13:31.336580 17304 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:31.285354 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.426839 17290 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.428104 17226 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:31.428794 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144129us
I20260709 12:13:31.348403 17338 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.355388 17321 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.357813 17249 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:31.367712 13260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 670) took 2550 ms (client timeout 2089 ms). Trace:
I20260709 12:13:31.410547 12489 rpcz_store.cc:276] 0709 12:13:28.024480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.024595 (+   115us) service_pool.cc:224] Handling call
0709 12:13:28.603919 (+579324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:28.605749 (+  1830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.262840 (+2657091us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b540037da50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:31.263211 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1331445632}
I20260709 12:13:31.368947 13256 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:31.372936 17348 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.374702 17350 raft_consensus.cc:2764] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 12:13:31.376008 12609 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.006s	sys 0.013s
I20260709 12:13:31.394284 17305 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:31.410166 12495 rpcz_store.cc:269] 0709 12:13:27.905351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:27.909746 (+  4395us) service_pool.cc:224] Handling call
0709 12:13:31.264044 (+3354298us) spinlock_profiling.cc:243] Waited 98.9 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:31.264300 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227574656}
W20260709 12:13:31.334082 17327 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.320s	user 0.005s	sys 0.023s
I20260709 12:13:31.423142 17275 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:31.442901 12609 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.006s	sys 0.014s
I20260709 12:13:31.425931 17341 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.426265 17228 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.427413 12491 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:31.347481 17246 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.451587 17246 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:31.451836 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 632618us
W20260709 12:13:31.434990 13260 rpcz_store.cc:269] 0709 12:13:28.816710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.823492 (+  6782us) service_pool.cc:224] Handling call
0709 12:13:31.367169 (+2543677us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:31.367393 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140365568}
I20260709 12:13:31.436561 17006 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:31.459853 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.461807 12490 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:31.447733 17215 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:31.451620 17275 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.475217 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.470786 17006 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:31.476536 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2596) took 1150 ms. Trace:
I20260709 12:13:31.477003 12489 rpcz_store.cc:276] 0709 12:13:30.325815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.464571 (+1138756us) service_pool.cc:224] Handling call
0709 12:13:31.476411 (+ 11840us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.436609 17035 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.478024 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:31.478417 12488 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:31.477999 17006 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:31.479240 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.402444 12496 rpcz_store.cc:276] 0709 12:13:28.018749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.019045 (+   296us) service_pool.cc:224] Handling call
0709 12:13:28.019423 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.019651 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:28.115351 (+ 95700us) log.cc:844] Serialized 14 byte log entry
0709 12:13:30.207122 (+2091771us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b180042c1c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:30.428923 (+221801us) log.cc:844] Serialized 14 byte log entry
0709 12:13:31.004292 (+575369us) log.cc:844] Serialized 14 byte log entry
0709 12:13:31.004575 (+   283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:31.004638 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:31.004690 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.005701 (+  1011us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:31.234360 (+228659us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103172,"prepare.queue_time_us":936407,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":2537,"raft.queue_time_us":121691,"raft.run_cpu_time_us":494,"raft.run_wall_time_us":26574,"spinlock_wait_cycles":28448640,"thread_start_us":688737,"threads_started":4,"wal-append.queue_time_us":862866,"wal-append.run_cpu_time_us":2804,"wal-append.run_wall_time_us":1196175}
W20260709 12:13:31.482520 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74057.9us
I20260709 12:13:31.481165 12391 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 5, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 5 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:31.487123 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:31.483688 17206 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.844s	user 0.012s	sys 0.012s
I20260709 12:13:31.475543 12646 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:31.465945 17228 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.491333 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2062) took 1060 ms. Trace:
I20260709 12:13:31.514170 12495 rpcz_store.cc:276] 0709 12:13:30.426582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.459362 (+1032780us) service_pool.cc:224] Handling call
0709 12:13:31.486669 (+ 27307us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:31.494688 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43515.5us
I20260709 12:13:31.495638 17341 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.496234 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:31.517963 12492 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:13:31.520788 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.521286 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2055) took 2539 ms. Trace:
W20260709 12:13:31.494585 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 124) took 16000 ms (client timeout 15000 ms). Trace:
I20260709 12:13:31.522701 12496 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:31.524833 12490 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:31.524220 12492 rpcz_store.cc:276] 0709 12:13:28.982354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:29.178085 (+195731us) service_pool.cc:224] Handling call
0709 12:13:29.190837 (+ 12752us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:31.518794 (+2327957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.520846 (+  2052us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:31.521206 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5347531520}
I20260709 12:13:31.529309 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2060) took 1101 ms. Trace:
I20260709 12:13:31.530344 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2058) took 1121 ms. Trace:
I20260709 12:13:31.530938 12496 rpcz_store.cc:276] 0709 12:13:30.404790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.519988 (+1115198us) service_pool.cc:224] Handling call
0709 12:13:31.526064 (+  6076us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:31.497816 17034 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.978s	user 0.014s	sys 0.018s
I20260709 12:13:31.537812 13256 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:13:31.528438 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54135.3us
I20260709 12:13:31.494168 17035 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.499737 17320 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.529711 12490 rpcz_store.cc:276] 0709 12:13:30.425140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.255833 (+830693us) service_pool.cc:224] Handling call
0709 12:13:31.526220 (+270387us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.542346 17060 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:31.568080 17060 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:31.568930 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.546960 17152 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:31.548422 17137 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:31.574018 17152 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:31.574179 17137 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:31.575551 17288 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.574018 17323 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:13:31.575910 17137 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.534713 12391 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 5, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 5 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:31.577456 12490 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:31.549748 12489 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:31.524632 12394 rpcz_store.cc:269] 0709 12:13:15.414513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:15.414745 (+   232us) service_pool.cc:224] Handling call
0709 12:13:15.759378 (+344633us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:31.393969 (+15634591us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:19.567941 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:19.708257 (+140316us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:19.711272 (+  3015us) write_op.cc:435] Acquired schema lock
0709 12:13:19.714590 (+  3318us) tablet.cc:662] Decoding operations
0709 12:13:19.844739 (+130149us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:19.844832 (+    93us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:19.844863 (+    31us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:13:19.846064 (+  1201us) tablet.cc:701] Row locks acquired
0709 12:13:19.846079 (+    15us) write_op.cc:260] PREPARE: finished
0709 12:13:19.846226 (+   147us) write_op.cc:270] Start()
0709 12:13:19.846342 (+   116us) write_op.cc:276] Timestamp: P: 1783599199846176 usec, L: 0
0709 12:13:19.846363 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:13:19.847135 (+   772us) log.cc:844] Serialized 5510 byte log entry
0709 12:13:20.448194 (+601059us) op_driver.cc:464] REPLICATION: finished
0709 12:13:20.643480 (+195286us) write_op.cc:301] APPLY: starting
0709 12:13:20.643587 (+   107us) tablet.cc:1367] starting BulkCheckPresence
0709 12:13:20.667680 (+ 24093us) tablet.cc:1370] finished BulkCheckPresence
0709 12:13:20.667879 (+   199us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:13:21.532857 (+864978us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:13:21.533390 (+   533us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0709 12:13:21.533470 (+    80us) write_op.cc:312] APPLY: finished
0709 12:13:21.711852 (+178382us) log.cc:844] Serialized 121 byte log entry
0709 12:13:22.070540 (+358688us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:22.070968 (+   428us) write_op.cc:454] Released schema lock
0709 12:13:22.071057 (+    89us) write_op.cc:341] FINISH: Updating metrics
0709 12:13:22.133850 (+ 62793us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:22.133889 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":398512,"reported_tablets":26,"spinlock_wait_cycles":5910400,"child_traces":[["op",{"apply.queue_time_us":192796,"apply.run_cpu_time_us":48422,"apply.run_wall_time_us":1455837,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58415,"num_ops":13,"prepare.queue_time_us":443079,"prepare.run_cpu_time_us":113172,"prepare.run_wall_time_us":494699,"raft.queue_time_us":296944,"raft.run_cpu_time_us":19621,"raft.run_wall_time_us":146319,"replication_time_us":601662,"thread_start_us":996633,"threads_started":5,"wal-append.queue_time_us":905649,"wal-append.run_cpu_time_us":22114,"wal-append.run_wall_time_us":2748059}]]}
I20260709 12:13:31.554476 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.555323 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:31.556480 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.494606 17290 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.523819 17226 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.499117 17271 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:31.570034 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.569824 17060 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.546909 17207 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:31.598815 17350 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:13:31.571228 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.571362 17308 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.559427 17034 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.552151 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.577546 17137 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:31.579501 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2597) took 1248 ms. Trace:
I20260709 12:13:31.591400 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:31.575963 17152 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.618324 17152 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:31.618777 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.601243 17271 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:31.578307 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.602653 17350 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:31.595109 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.608948 12489 rpcz_store.cc:276] 0709 12:13:30.331251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.485131 (+1153880us) service_pool.cc:224] Handling call
0709 12:13:31.579396 (+ 94265us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:31.586546 17360 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:13:31.595697 17005 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:13:31.612934 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:31.614154 17060 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:31.592497 17002 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:31.595789 17288 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.613025 17348 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:13:31.622067 12491 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:31.621292 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:31.632200 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2601) took 1277 ms (client timeout 1669 ms). Trace:
I20260709 12:13:31.625937 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:31.630673 12494 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260709 12:13:31.632802 12491 rpcz_store.cc:269] 0709 12:13:30.355080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.270778 (+915698us) service_pool.cc:224] Handling call
0709 12:13:31.632093 (+361315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.635663 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2604) took 1218 ms. Trace:
I20260709 12:13:31.635020 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.621554 17338 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:31.639647 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
W20260709 12:13:31.640232 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2590) took 1319 ms (client timeout 1720 ms). Trace:
I20260709 12:13:31.640800 17207 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:13:31.648250 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120537us
I20260709 12:13:31.649219 17254 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.649610 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:31.650149 17254 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.652303 16716 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:31.653024 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:31.641896 12563 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:31.642122 17288 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.641932 17317 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:31.626821 17350 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:31.672747 17315 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:13:31.674229 17271 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.675792 17271 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:31.679872 17288 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:31.681038 17311 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.209s	user 0.004s	sys 0.010s
I20260709 12:13:31.684839 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.645211 17338 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:31.689890 17035 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:31.690511 17035 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:31.691017 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 602) took 3416 ms. Trace:
I20260709 12:13:31.692711 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2404) took 1296 ms. Trace:
I20260709 12:13:31.691696 12493 rpcz_store.cc:276] 0709 12:13:28.274618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.335430 (+ 60812us) service_pool.cc:224] Handling call
0709 12:13:28.336074 (+   644us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:28.404026 (+ 67952us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:28.404071 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:13:31.190966 (+2786895us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:31.277277 (+ 86311us) log.cc:844] Serialized 78 byte log entry
0709 12:13:31.438001 (+160724us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:31.470317 (+ 32316us) log.cc:844] Serialized 14 byte log entry
0709 12:13:31.486334 (+ 16017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.489355 (+  3021us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:31.690337 (+200982us) raft_consensus.cc:1685] finished
0709 12:13:31.690495 (+   158us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:31.690746 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":112912,"prepare.queue_time_us":35674,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":64,"raft.queue_time_us":71621,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":6380,"spinlock_wait_cycles":11065216,"thread_start_us":88766,"threads_started":1,"wal-append.queue_time_us":196975}
I20260709 12:13:31.696019 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.696844 12493 raft_consensus.cc:2376] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12. Re-sending same reply.
I20260709 12:13:31.697687 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2591) took 1376 ms. Trace:
I20260709 12:13:31.698297 12493 rpcz_store.cc:276] 0709 12:13:30.321242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.695578 (+1374336us) service_pool.cc:224] Handling call
0709 12:13:31.697437 (+  1859us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.699399 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.700608 12493 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:31.653259 17254 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:31.645684 12494 rpcz_store.cc:269] 0709 12:13:30.320923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.551579 (+1230656us) service_pool.cc:224] Handling call
0709 12:13:31.640153 (+ 88574us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.655673 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.656276 17348 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:13:31.641158 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46314.2us
I20260709 12:13:31.658067 17339 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:31.660020 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:31.660454 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2046) took 2715 ms. Trace:
I20260709 12:13:31.579711 17289 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.734201 13260 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:13:31.734771 17289 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:31.736053 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:31.666635 17320 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:31.691013 17035 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:31.737937 12537 connection.cc:256] RPC call timeout handler was delayed by -2.3206251690000954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20260709 12:13:31.738487 12537 connection.cc:256] RPC call timeout handler was delayed by -2.3074296649999724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
W20260709 12:13:31.738849 12537 connection.cc:256] RPC call timeout handler was delayed by -1.9579894470000454s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
I20260709 12:13:31.740356 17254 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:31.705601 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.742331 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2059) took 1330 ms. Trace:
I20260709 12:13:31.742365 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.742781 12489 rpcz_store.cc:276] 0709 12:13:30.412328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.635187 (+1222859us) service_pool.cc:224] Handling call
0709 12:13:31.742249 (+107062us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204032}
I20260709 12:13:31.685524 12492 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11.
I20260709 12:13:31.647166 12536 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:13:31.743919 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2599) took 1396 ms. Trace:
I20260709 12:13:31.637089 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2064) took 1208 ms. Trace:
I20260709 12:13:31.653087 16716 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:31.744374 12492 rpcz_store.cc:276] 0709 12:13:30.347672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.591530 (+1243858us) service_pool.cc:224] Handling call
0709 12:13:31.743837 (+152307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.746026 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.746205 16716 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.748062 16716 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:31.747120 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2592) took 1425 ms. Trace:
I20260709 12:13:31.722460 17314 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:31.750708 12492 rpcz_store.cc:276] 0709 12:13:30.321544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.745543 (+1423999us) service_pool.cc:224] Handling call
0709 12:13:31.747035 (+  1492us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.730501 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.757488 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2061) took 1332 ms. Trace:
I20260709 12:13:31.757881 12494 rpcz_store.cc:276] 0709 12:13:30.425578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.730075 (+1304497us) service_pool.cc:224] Handling call
0709 12:13:31.757415 (+ 27340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.758877 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.759505 12494 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13. Re-sending same reply.
I20260709 12:13:31.760399 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.704821 12491 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:13:31.728459 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.727252 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:31.657320 16528 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.729486 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2600) took 1379 ms. Trace:
I20260709 12:13:31.730290 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.637104 12490 rpcz_store.cc:276] 0709 12:13:30.417383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.568355 (+1150972us) service_pool.cc:224] Handling call
0709 12:13:31.634228 (+ 65873us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.736179 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.737709 17320 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.715943 17348 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:31.740501 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.727677 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.728920 17034 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.761574 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.765316 12493 rpcz_store.cc:276] 0709 12:13:30.349933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.699056 (+1349123us) service_pool.cc:224] Handling call
0709 12:13:31.729404 (+ 30348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.764267 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.764844 16528 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:31.689342 17338 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:31.743934 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.762986 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2605) took 1334 ms. Trace:
I20260709 12:13:31.760869 17035 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.766448 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:31.769870 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 117149us
W20260709 12:13:31.770448 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101012us
I20260709 12:13:31.785418 12495 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:31.744563 12496 rpcz_store.cc:276] 0709 12:13:30.428650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.538985 (+1110335us) service_pool.cc:224] Handling call
0709 12:13:31.637006 (+ 98021us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:31.776543 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137169us
I20260709 12:13:31.771420 12488 rpcz_store.cc:276] 0709 12:13:28.944956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.959067 (+ 14111us) service_pool.cc:224] Handling call
0709 12:13:28.959563 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:31.493016 (+2533453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.659942 (+166926us) spinlock_profiling.cc:243] Waited 2.51 s on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:31.660342 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5778412416}
I20260709 12:13:31.786162 12494 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:31.693485 12563 rpcz_store.cc:276] 0709 12:13:30.395716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:30.395948 (+   232us) service_pool.cc:224] Handling call
0709 12:13:30.418131 (+ 22183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:31.673925 (+1255794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.692001 (+ 18076us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:31.692482 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2802816384}
W20260709 12:13:31.777463 12537 connection.cc:256] RPC call timeout handler was delayed by -1.0142592039999272s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
I20260709 12:13:31.782269 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.769942 13253 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 8.
I20260709 12:13:31.791893 12491 rpcz_store.cc:276] 0709 12:13:30.428719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.644226 (+1215507us) service_pool.cc:224] Handling call
0709 12:13:31.762916 (+118690us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.773864 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:31.793097 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2602) took 1396 ms (client timeout 1776 ms). Trace:
W20260709 12:13:31.797363 12495 rpcz_store.cc:269] 0709 12:13:30.396881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.526662 (+1129781us) service_pool.cc:224] Handling call
0709 12:13:31.793024 (+266362us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.778303 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2448) took 3903 ms. Trace:
I20260709 12:13:31.789345 12489 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:31.770181 17317 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:31.804093 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2057) took 1416 ms. Trace:
I20260709 12:13:31.799835 16528 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:31.801792 17271 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.807235 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:31.807606 17369 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.145s	user 0.003s	sys 0.000s
I20260709 12:13:31.799404 12571 rpcz_store.cc:276] 0709 12:13:27.874963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:28.369974 (+495011us) service_pool.cc:224] Handling call
0709 12:13:28.370309 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:28.370531 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:28.508367 (+137836us) log.cc:844] Serialized 14 byte log entry
0709 12:13:28.694577 (+186210us) log.cc:844] Serialized 14 byte log entry
0709 12:13:29.508736 (+814159us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b180036af60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:30.158445 (+649709us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b180036af60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:31.672980 (+1514535us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b180036af60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:31.673405 (+   425us) log.cc:844] Serialized 14 byte log entry
0709 12:13:31.721728 (+ 48323us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:31.721854 (+   126us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:31.721957 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.777865 (+ 55908us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:31.778071 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2962699,"prepare.queue_time_us":3021992,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":1753,"raft.queue_time_us":1463,"raft.run_cpu_time_us":449,"raft.run_wall_time_us":8365,"spinlock_wait_cycles":42337408,"thread_start_us":215416,"threads_started":2,"wal-append.queue_time_us":272081,"wal-append.run_cpu_time_us":2531,"wal-append.run_wall_time_us":1146546}
I20260709 12:13:31.802628 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2593) took 1474 ms. Trace:
I20260709 12:13:31.837982 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.838137 12492 rpcz_store.cc:276] 0709 12:13:30.321906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.753161 (+1431255us) service_pool.cc:224] Handling call
0709 12:13:31.796420 (+ 43259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.838673 12493 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:13:31.839250 17128 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:31.839542 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2595) took 1517 ms. Trace:
I20260709 12:13:31.839738 12492 raft_consensus.cc:1217] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 11.10->[12.11-14.12]   Dedup: 14.12->[]
I20260709 12:13:31.840207 17128 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:31.810184 12494 rpcz_store.cc:276] 0709 12:13:30.380086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.761288 (+1381202us) service_pool.cc:224] Handling call
0709 12:13:31.796672 (+ 35384us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.776300 17290 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.841384 17290 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.842672 17128 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:31.844460 17290 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.821573 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.844925 17006 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:31.846223 16780 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:31.824255 17227 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:31.825145 12497 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:31.847848 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:31.826040 17058 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:31.849023 17058 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:31.849380 17058 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:31.850526 17058 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.851048 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.852164 17058 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:31.810066 17317 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:31.828876 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.854161 17317 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:31.829608 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:31.855912 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.857872 17343 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:31.831627 17271 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.777083 17288 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:31.769455 17348 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.861127 17348 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:31.776707 17308 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.862397 17308 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.866865 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:31.840358 12492 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [12.11-14.12]
I20260709 12:13:31.868891 17152 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:31.869043 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2598) took 1522 ms. Trace:
I20260709 12:13:31.869606 12492 rpcz_store.cc:276] 0709 12:13:30.346560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.839088 (+1492528us) service_pool.cc:224] Handling call
0709 12:13:31.839595 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:31.868705 (+ 29110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.868946 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.865388 17308 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:31.841959 12494 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:13:31.871227 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2056) took 1492 ms. Trace:
I20260709 12:13:31.871621 12494 rpcz_store.cc:276] 0709 12:13:30.379214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.841214 (+1462000us) service_pool.cc:224] Handling call
0709 12:13:31.841727 (+   513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:31.870829 (+ 29102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.871148 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.846673 16780 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:31.825569 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110021us
W20260709 12:13:31.873885 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71181.2us
I20260709 12:13:31.873442 16780 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:31.875753 16780 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:31.876350 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:31.768451 17320 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:31.827987 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:31.836241 16528 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.777591 17034 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.840039 12493 rpcz_store.cc:276] 0709 12:13:30.322515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.837659 (+1515144us) service_pool.cc:224] Handling call
0709 12:13:31.838215 (+   556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:31.839214 (+   999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.839444 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:31.879947 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.885275 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2063) took 1457 ms. Trace:
I20260709 12:13:31.861331 17271 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:31.886296 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58475.4us
I20260709 12:13:31.885793 12490 rpcz_store.cc:276] 0709 12:13:30.427500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.852760 (+1425260us) service_pool.cc:224] Handling call
0709 12:13:31.853218 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:31.884950 (+ 31732us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:31.884976 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:31.885016 (+    40us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:13:31.885037 (+    21us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.885093 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:31.885220 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28486}
I20260709 12:13:31.885480 12491 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:31.889658 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:31.895488 12642 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:13:31.899457 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2025) took 7590 ms. Trace:
I20260709 12:13:31.895267 17006 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:31.899968 12497 rpcz_store.cc:276] 0709 12:13:24.308865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:26.046817 (+1737952us) service_pool.cc:224] Handling call
0709 12:13:28.694763 (+2647946us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:13:28.694992 (+   229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:31.847623 (+3152631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.899014 (+ 51391us) spinlock_profiling.cc:243] Waited 3.12 s on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:31.899343 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7196380800}
I20260709 12:13:31.908499 12639 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:31.916092 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.809242 17350 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.928012 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:31.886662 17034 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:31.929780 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:31.930737 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.935225 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:31.907155 17320 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:31.938014 12563 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:31.938400 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:31.939278 12644 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:31.940565 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:31.941427 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.945528 17326 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:31.946195 17326 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:31.946669 17326 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:31.949996 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75827.8us
I20260709 12:13:31.950476 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:31.951591 13253 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
I20260709 12:13:31.958544 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.958899 12569 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:31.959262 12572 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:31.949976 17254 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:31.960143 17254 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:31.960330 12570 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:31.960374 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:31.960902 13259 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:31.961372 12494 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:31.962716 12566 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:31.964119 12565 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:31.946195 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2271) took 1035 ms. Trace:
I20260709 12:13:31.914724 17308 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:31.975237 13260 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:31.979189 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:31.979713 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.969929 13258 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:31.966048 12641 rpcz_store.cc:276] 0709 12:13:30.910853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:30.924445 (+ 13592us) service_pool.cc:224] Handling call
0709 12:13:30.924883 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:31.358229 (+433346us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:31.458028 (+ 99799us) log.cc:844] Serialized 14 byte log entry
0709 12:13:31.611133 (+153105us) log.cc:844] Serialized 14 byte log entry
0709 12:13:31.626155 (+ 15022us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:31.682501 (+ 56346us) log.cc:844] Serialized 26 byte log entry
0709 12:13:31.682675 (+   174us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:31.682713 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:31.806633 (+123920us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:31.847908 (+ 41275us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b1800329f80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:31.943020 (+ 95112us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:31.943062 (+    42us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:31.943100 (+    38us) raft_consensus.cc:1685] finished
0709 12:13:31.943108 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:31.945997 (+  2889us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61696,"prepare.queue_time_us":48114,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":101,"raft.queue_time_us":19659,"raft.run_cpu_time_us":761,"raft.run_wall_time_us":18600,"spinlock_wait_cycles":996844160,"thread_start_us":153490,"threads_started":2,"wal-append.queue_time_us":187436}
I20260709 12:13:31.960628 17254 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:31.991094 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143441us
I20260709 12:13:31.982890 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:31.984010 12643 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:31.997711 12490 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
W20260709 12:13:32.006024 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:31.970301 17006 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:31.971534 17308 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:32.010404 12637 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:32.008697 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:32.020623 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.001223 17271 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:32.012465 17320 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.023546 12569 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:32.023206 17271 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.015513 17308 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:32.034852 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.035068 12563 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:32.036103 17222 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:32.036528 17379 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:13:32.036854 17222 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:32.040697 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:32.046561 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 386636us
I20260709 12:13:32.028223 17035 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.057614 12569 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
W20260709 12:13:32.066217 17380 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:13:32.067042 17340 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 12:13:32.068164 17058 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:32.068926 17058 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:32.070076 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.069082 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:32.027324 17006 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.031733 17226 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:13:32.090143 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:32.090945 17340 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:32.031605 17207 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:13:32.094744 17320 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:32.094923 17271 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:32.095554 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.071827 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:32.106593 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:32.114845 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.049566 17137 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:13:32.101016 17340 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:32.119102 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.124825 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:32.128607 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63850.8us
I20260709 12:13:32.131605 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.132308 12644 raft_consensus.cc:2376] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11. Re-sending same reply.
I20260709 12:13:32.136874 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.144779 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:32.147027 17034 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:32.147873 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:32.148128 17034 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:32.148931 17034 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:32.149168 17290 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:32.149915 17290 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:32.150991 17290 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.162024 17058 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.169701 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:32.169821 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135433us
I20260709 12:13:32.170727 17314 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:32.170936 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:32.171245 17314 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:32.171298 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:32.171860 17314 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:32.171986 17285 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:32.182920 12643 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:32.184379 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.195521 12637 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:32.200742 12494 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:32.129362 17340 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:32.212235 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.218030 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:32.218109 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.194291 17207 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:32.215870 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:32.219812 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.210268 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.227568 12569 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:32.234215 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63382us
I20260709 12:13:32.234606 12488 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
W20260709 12:13:32.223767 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 272333us
I20260709 12:13:32.215935 17035 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:13:32.243695 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 189157us
I20260709 12:13:32.215840 17058 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:32.245865 12565 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:32.217417 17350 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.248900 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:32.225370 17207 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:32.262557 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.267767 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.268585 17207 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:32.276254 12641 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:32.244786 17035 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:32.281394 17035 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:32.282637 17226 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:32.282789 17207 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.284008 17254 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.285552 17207 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.286389 17254 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:32.243146 17271 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:32.295485 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:32.296293 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.299387 12563 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:32.304404 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.305310 12565 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:32.309019 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:32.300617 17288 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:32.314950 12637 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260709 12:13:32.306775 17289 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.571s	user 0.002s	sys 0.017s
I20260709 12:13:32.306972 17006 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:32.324646 12641 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:13:32.331894 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67856.8us
I20260709 12:13:32.303274 17271 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.320048 17222 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:32.326932 17338 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.524s	user 0.014s	sys 0.011s
I20260709 12:13:32.340907 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.300076 17326 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.344794 12642 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:13:32.351688 13260 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:32.306728 17034 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.357640 12565 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 8.
W20260709 12:13:32.345371 17381 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:13:32.384490 17326 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:32.394258 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46526.4us
I20260709 12:13:32.394462 12565 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:13:32.398130 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.403968 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:32.404995 12563 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:32.442715 12569 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:13:32.443233 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 230723us
I20260709 12:13:32.339461 17288 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.385401 17326 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
W20260709 12:13:32.455729 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 326653us
W20260709 12:13:32.465508 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:32.441569 17383 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
W20260709 12:13:32.487988 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.399991 17006 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:32.400328 17289 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:32.489089 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:32.353869 17338 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:32.507902 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45527.7us
W20260709 12:13:32.512761 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 358653us
I20260709 12:13:32.441484 17035 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.497242 17289 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:32.524806 12565 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:32.525807 12569 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:13:32.511814 17222 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:13:32.500519 17006 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.530187 17289 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:32.517891 17137 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
W20260709 12:13:32.562341 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73768.6us
I20260709 12:13:32.564604 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.579999 12569 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:32.577262 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 325932us
W20260709 12:13:32.549820 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 240843us
I20260709 12:13:32.584709 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:32.525723 17285 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:32.547652 17271 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.611017 17391 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:32.612452 13255 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
W20260709 12:13:32.618052 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103316us
W20260709 12:13:32.620291 17386 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:13:32.623657 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.627184 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.604669 16711 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:32.630036 16711 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:32.630429 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:32.631032 17382 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.353s	user 0.004s	sys 0.000s
W20260709 12:13:32.631316 17384 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:13:32.630488 16711 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:32.638944 17006 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:32.641355 17390 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:13:32.643227 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:32.683198 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:32.543421 17339 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:13:32.590485 17005 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
W20260709 12:13:32.699421 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94401.9us
I20260709 12:13:32.704063 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.641036 16711 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.580828 17397 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.116s
I20260709 12:13:32.698326 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.709266 12641 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:32.705747 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104601us
I20260709 12:13:32.703400 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:32.709182 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.711716 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:32.716096 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109247us
I20260709 12:13:32.719920 12646 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 3. Preceding OpId from leader: term: 12 index: 4. (index mismatch)
W20260709 12:13:32.730690 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 118080us
W20260709 12:13:32.742995 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:32.743500 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.763547 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.712846 17338 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.713042 17288 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:32.768558 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.720448 17005 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:13:32.772322 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:32.768955 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:32.771818 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.782413 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.783766 12639 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:32.697100 17285 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:32.775578 17317 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.921s	user 0.009s	sys 0.006s
I20260709 12:13:32.788901 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:32.768479 17034 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:32.780073 17340 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.548s	user 0.007s	sys 0.006s
I20260709 12:13:32.771585 17271 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:32.773128 17035 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:32.788722 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.768209 17408 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:13:32.921223 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:32.819160 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.924289 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.905033 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:32.924580 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 180353us
I20260709 12:13:32.923095 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.793776 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.807336 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:32.818883 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:32.806885 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:32.924000 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.909936 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:32.792882 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:32.916883 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134458us
I20260709 12:13:32.820135 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:32.952617 12566 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:13:32.948290 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:32.822990 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119797us
I20260709 12:13:32.900080 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2085) took 1025 ms. Trace:
W20260709 12:13:32.953135 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.807421 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:32.953792 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:32.956632 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.951460 12637 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:32.957759 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:32.955971 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:32.833149 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:32.810197 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.950986 12497 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:32.961216 12495 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10. Re-sending same reply.
I20260709 12:13:32.960906 12646 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:32.923895 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.955951 12638 rpcz_store.cc:276] 0709 12:13:31.875162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.882485 (+  7323us) service_pool.cc:224] Handling call
0709 12:13:31.894096 (+ 11611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:31.929458 (+ 35362us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:31.992072 (+ 62614us) log.cc:844] Serialized 14 byte log entry
0709 12:13:32.834933 (+842861us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b1800446ce0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:32.899483 (+ 64550us) log.cc:844] Serialized 14 byte log entry
0709 12:13:32.899678 (+   195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:32.899733 (+    55us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:32.899792 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:32.899868 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:32.900010 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8920,"prepare.queue_time_us":662473,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":118,"raft.queue_time_us":862,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":1886,"spinlock_wait_cycles":2435840,"thread_start_us":278921,"threads_started":1,"wal-append.queue_time_us":520944}
I20260709 12:13:32.966141 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:32.966449 17320 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:32.966711 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.966952 17320 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:32.967646 12638 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:32.968943 12489 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
W20260709 12:13:32.818666 17393 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.145s	user 0.002s	sys 0.000s
I20260709 12:13:32.834510 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.967005 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:32.968456 17320 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:32.968459 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:32.977478 12564 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:32.977950 12644 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:32.956411 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:32.974093 12567 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:32.976183 12642 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:32.925290 17389 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
W20260709 12:13:32.833058 17348 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.971s	user 0.010s	sys 0.007s
I20260709 12:13:32.787030 17137 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:32.820703 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:32.834256 17314 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.662s	user 0.014s	sys 0.001s
I20260709 12:13:33.001958 17314 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:33.005281 17308 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.944s	user 0.016s	sys 0.001s
I20260709 12:13:33.005753 17314 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.005869 17308 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.006716 17320 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:33.007504 12572 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:33.010061 17308 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:32.979594 17271 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.016822 17271 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.983937 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:33.018134 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:33.018522 12563 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:33.018942 12571 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:33.019555 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:33.020584 12563 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 12:13:33.020816 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:33.022342 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:33.023102 12492 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:33.023733 12568 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:32.997359 17034 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.024498 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:33.025710 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:33.026288 12492 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:33.026378 17034 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:32.994928 17137 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:32.806995 17317 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:32.821404 17406 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:13:33.002583 12564 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
W20260709 12:13:33.003268 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 198624us
I20260709 12:13:33.029826 12564 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:13:33.030162 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:33.031402 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.022481 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:32.815305 17340 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.035449 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:33.036321 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 8.
I20260709 12:13:33.037015 12567 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:33.035779 17340 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.000919 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:33.031592 12570 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:33.038080 12571 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:13:33.038884 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.046154 12640 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
W20260709 12:13:33.051702 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95828.2us
I20260709 12:13:33.047372 12646 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:33.056200 17035 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:33.030192 12564 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:13:33.062813 12564 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:32.907023 17417 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:32.917276 17398 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:32.999576 17404 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:32.819774 17405 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.062181 17215 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.069988 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:33.087543 17215 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:32.819375 17403 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:33.088609 17340 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:33.089001 17403 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:33.003139 17415 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:33.089663 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:33.089867 17415 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:33.090301 17415 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:33.090799 17340 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:32.819247 17399 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:33.037616 17285 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.038779 17034 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:33.083997 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:33.085533 12536 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:33.062239 17035 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.104915 12569 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:33.105228 12536 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:33.075111 17417 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:33.108112 12642 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:13:33.094566 16780 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.218s	user 0.020s	sys 0.000s
W20260709 12:13:33.112740 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156416us
I20260709 12:13:33.078715 17404 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:33.114198 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.089582 17403 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:33.127303 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51101us
I20260709 12:13:33.130209 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.106509 17035 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:33.132550 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:33.133080 17058 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.872s	user 0.008s	sys 0.008s
W20260709 12:13:33.133675 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88701.7us
I20260709 12:13:33.134099 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.134145 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.135421 13257 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:33.135582 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:33.134447 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.137173 12610 leader_election.cc:400] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11.
I20260709 12:13:33.137501 12610 leader_election.cc:403] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:33.142143 12496 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:13:33.108722 17340 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.106628 17034 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:33.142925 12496 pending_rounds.cc:85] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 6
I20260709 12:13:33.145047 12496 pending_rounds.cc:107] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:13:33.145804 12496 raft_consensus.cc:2889] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:33.148759 12644 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:33.150954 12637 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:13:33.151485 13260 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:33.151849 12569 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:33.151885 12639 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:33.114812 17410 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:33.111310 17398 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:33.111727 17417 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:33.154759 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.153851 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:33.166893 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.119525 17404 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:33.096381 17405 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.169283 13257 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:33.161886 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.171206 13256 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:33.171492 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:33.173259 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:33.131461 17271 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:33.116350 17222 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:13:33.173988 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.155818 17034 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:33.180444 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.183796 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.168042 17058 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.186713 17423 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:13:33.190886 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.172462 16780 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
W20260709 12:13:33.192307 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83424.3us
I20260709 12:13:33.199679 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.195317 12571 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 12:13:33.198004 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.200280 12571 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:13:33.200639 12571 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 12:13:33.200800 12571 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:33.172609 17005 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:33.173411 17285 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.196534 17410 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:13:33.173606 17408 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11.
I20260709 12:13:33.179423 17308 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260709 12:13:33.210278 13257 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:33.211231 12641 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.001s
I20260709 12:13:33.211201 12638 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:13:33.211602 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84840.1us
I20260709 12:13:33.213189 12641 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
W20260709 12:13:33.216770 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79780.8us
I20260709 12:13:33.217782 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:33.218546 12564 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:33.157075 17340 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.202401 17314 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:33.195422 17391 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:33.222478 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:33.224262 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 60106.2us
I20260709 12:13:33.207909 16780 consensus_queue.cc:1243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 12:13:33.229686 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46235.9us
W20260709 12:13:33.230435 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50057.9us
I20260709 12:13:33.185259 17006 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:13:33.233961 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.241338 13257 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:33.241521 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.215149 17412 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:33.242442 13259 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:33.239487 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:33.223094 17397 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:33.247565 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:33.248718 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:33.251554 17397 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.252000 17397 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:33.253317 12568 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:33.227902 17423 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:33.262009 17423 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:33.263187 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.263055 17423 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.264709 17423 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:33.216457 17285 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.268164 17222 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:13:33.268811 17285 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.268610 17403 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.210116 17058 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.271028 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.271577 17058 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.272691 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.225888 17314 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.274222 17314 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:33.275898 17415 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.278148 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.278982 13261 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 9: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:33.279845 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.281013 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:33.277470 13254 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:33.283943 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:33.284739 17425 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:13:33.285017 13257 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:33.289122 12567 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 12:13:33.293543 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43746.3us
I20260709 12:13:33.293795 12567 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:13:33.295120 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.295631 12567 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:13:33.296615 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.300156 13262 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:33.300520 13260 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
W20260709 12:13:33.301288 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70175.2us
I20260709 12:13:33.300644 12567 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:13:33.305313 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:33.306504 17415 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.307075 17403 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.299185 17227 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:33.307921 17227 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:33.287482 17401 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:13:33.302852 12567 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:33.310497 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59749.6us
I20260709 12:13:33.291934 17271 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:13:33.299254 17289 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.706s	user 0.013s	sys 0.000s
I20260709 12:13:33.308975 17227 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.321758 17427 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:33.322642 17409 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:13:33.315291 17006 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:33.264822 17398 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.321851 17271 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.324745 17340 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:33.326097 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92310.9us
I20260709 12:13:33.321341 17323 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:33.326874 17405 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.322934 17227 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:33.328011 16711 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.332860 12565 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:33.327039 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:33.334962 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.336607 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:33.337867 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 753) took 1114 ms. Trace:
I20260709 12:13:33.338196 17315 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:33.339095 17315 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:33.338307 13258 rpcz_store.cc:276] 0709 12:13:32.223176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:32.223323 (+   147us) service_pool.cc:224] Handling call
0709 12:13:32.249393 (+ 26070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:32.261454 (+ 12061us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:13:32.582446 (+320992us) log.cc:844] Serialized 14 byte log entry
0709 12:13:33.337254 (+754808us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b180033c520. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:33.337466 (+   212us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:33.337522 (+    56us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:13:33.337569 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:33.337661 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:33.337784 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8315,"prepare.queue_time_us":439053,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"spinlock_wait_cycles":49325312,"thread_start_us":458696,"threads_started":2,"wal-append.queue_time_us":238488}
W20260709 12:13:33.342262 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 599543us
I20260709 12:13:33.301724 17005 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:13:33.344435 17350 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:33.344941 17005 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:33.345336 17005 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:33.346246 17005 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.347647 17005 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:33.351858 12564 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:33.324879 17271 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:33.337211 17323 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:33.327003 17427 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:33.359627 17399 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:33.366525 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:33.377609 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82623.2us
I20260709 12:13:33.383658 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.325366 17289 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.367123 17417 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.395151 17315 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:33.395040 17405 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.395517 17397 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.395965 17405 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.378633 17427 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:33.399326 17405 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.399901 17427 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.401307 17427 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:33.401757 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.394722 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.402191 17399 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.382527 16711 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 12:13:33.404234 17387 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:33.418941 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.420070 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:33.441176 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:33.393221 17323 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:33.469782 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49842.6us
I20260709 12:13:33.465849 17405 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:13:33.473611 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52431us
W20260709 12:13:33.476481 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60567.7us
I20260709 12:13:33.477190 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.404553 17006 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:33.387385 17423 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20260709 12:13:33.465714 17397 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.468319 17403 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:33.468001 17222 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:33.486855 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:33.503944 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137527us
I20260709 12:13:33.475311 17285 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:33.507779 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:33.422003 17410 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 12:13:33.465731 17035 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.525967 17315 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:13:33.553758 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66996.8us
I20260709 12:13:33.499862 17308 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:33.564836 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:33.568924 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61493.7us
I20260709 12:13:33.570490 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 611) took 1858 ms. Trace:
I20260709 12:13:33.571282 12488 rpcz_store.cc:276] 0709 12:13:31.715890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.854891 (+139001us) service_pool.cc:224] Handling call
0709 12:13:31.855271 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:32.242658 (+387387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:33.566435 (+1323777us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:33.566770 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":871457024}
I20260709 12:13:33.515736 17285 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:33.575515 12564 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:13:33.556715 17315 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.515933 17403 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.605247 12564 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:33.572655 17417 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.582294 17285 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:33.588526 17315 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:33.596693 17308 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.627560 17308 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.573918 17340 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:33.634099 17340 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:33.635246 17340 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.608778 12566 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:13:33.586236 17409 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:33.557862 17404 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.656785 17423 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:13:33.623580 17323 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.727532 12571 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:33.729507 17409 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.730319 12643 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:13:33.593276 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:33.748344 17408 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:33.765688 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:33.766882 12644 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:33.749434 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 745) took 2095 ms. Trace:
I20260709 12:13:33.756273 12564 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:33.725930 17435 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.100s	user 0.004s	sys 0.000s
W20260709 12:13:33.758230 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 286635us
I20260709 12:13:33.736006 12638 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:33.767418 13255 rpcz_store.cc:276] 0709 12:13:31.654323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.654747 (+   424us) service_pool.cc:224] Handling call
0709 12:13:31.655418 (+   671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:32.612909 (+957491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:33.749049 (+1136140us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:33.749348 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987246464}
W20260709 12:13:33.772578 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179405us
W20260709 12:13:33.770406 17034 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.563s	user 0.014s	sys 0.000s
I20260709 12:13:33.769718 17409 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:33.769975 17415 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.784135 17423 consensus_queue.cc:1243] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:13:33.814080 12564 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:33.788123 17419 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:33.822502 17397 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:33.824441 16711 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:33.824597 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:33.825457 17207 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:33.800981 17289 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.830736 12566 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:33.806211 17431 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:33.779470 17408 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:33.827095 17314 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.002s
W20260709 12:13:33.850222 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67409.5us
W20260709 12:13:33.852818 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70322.9us
W20260709 12:13:33.854395 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 386810us
I20260709 12:13:33.860142 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:33.833384 17419 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:33.835516 17397 consensus_queue.cc:1243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:13:33.836247 16711 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:33.878664 17226 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:13:33.878942 16711 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.842679 17207 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:33.811285 17034 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.851320 17431 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:33.827466 17415 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:33.885941 17388 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.440s	user 0.016s	sys 0.011s
I20260709 12:13:33.880761 16711 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:33.899377 12567 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:33.896042 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:33.897085 12568 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.000s	sys 0.023s
I20260709 12:13:33.882218 17419 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:33.900861 12568 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.000s	sys 0.024s
I20260709 12:13:33.858700 17314 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:33.881543 17308 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.888823 17035 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.917114 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:33.918224 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:33.924678 12566 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:33.896463 17226 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.895987 17432 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:33.889607 17431 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:33.920634 13218 connection.cc:256] RPC call timeout handler was delayed by -1.0656038479999097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
W20260709 12:13:33.899659 17388 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.468s	user 0.016s	sys 0.011s
I20260709 12:13:33.926199 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2603) took 3525 ms. Trace:
I20260709 12:13:33.929824 12496 rpcz_store.cc:276] 0709 12:13:30.400485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:31.846272 (+1445787us) service_pool.cc:224] Handling call
0709 12:13:31.846679 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:33.168852 (+1322173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:33.925906 (+757054us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:33.926110 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2921817472}
W20260709 12:13:33.929737 13218 connection.cc:256] RPC call timeout handler was delayed by -1.0242038480000701s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
I20260709 12:13:33.923434 17415 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.935353 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:33.937922 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:33.938735 12643 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:33.941820 17403 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.945194 17432 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.948094 17432 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.950508 12566 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:33.953490 17137 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:33.955312 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.650s (SENT)
I20260709 12:13:33.955569 17137 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:33.955936 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:33.957194 17387 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:33.911880 17227 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:33.958156 17227 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:33.959779 13218 leader_election.cc:336] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.323s (SENT)
I20260709 12:13:33.960829 17227 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.962743 17314 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.907222 16528 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:13:33.964918 17226 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.931149 17423 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:13:33.967801 17423 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:33.968250 17226 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:33.970681 17285 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:33.929755 17207 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:33.910295 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56739us
I20260709 12:13:33.973994 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:33.973587 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:33.974745 12646 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 9
I20260709 12:13:33.979163 12646 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:13:33.979303 12639 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:33.905295 12641 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9.
I20260709 12:13:33.979498 12646 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:33.961483 17271 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.592s	user 0.004s	sys 0.005s
W20260709 12:13:33.979545 17427 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
I20260709 12:13:33.991950 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:33.997491 12645 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:34.002431 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:33.977739 17399 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:33.968416 17423 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:34.015583 17005 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.668s	user 0.008s	sys 0.005s
I20260709 12:13:34.018158 17315 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.019959 17423 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.020330 17215 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:34.020990 17315 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:34.021790 17320 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:34.022429 17320 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:34.022464 17005 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:34.023579 17320 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.024333 17005 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:13:34.024797 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.027601 17440 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:34.025121 17320 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:34.029853 12637 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:34.033681 17215 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.034287 17215 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.035459 17034 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:34.034397 17315 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.036603 17215 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.038707 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:34.037134 17315 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.041921 17399 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.043537 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2298) took 1085 ms. Trace:
I20260709 12:13:34.044001 12640 rpcz_store.cc:276] 0709 12:13:32.958199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:32.958363 (+   164us) service_pool.cc:224] Handling call
0709 12:13:33.005773 (+ 47410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:33.057799 (+ 52026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.043188 (+985389us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:34.043437 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81777024}
I20260709 12:13:34.044494 17399 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:34.045655 17447 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:13:33.992969 17404 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:33.990861 17207 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:34.046203 17392 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:34.087774 17392 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.047148 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:34.045938 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.091022 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:34.091876 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:34.055977 17408 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.093582 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:34.048667 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2477) took 2006 ms (client timeout 1611 ms). Trace:
W20260709 12:13:34.094509 12572 rpcz_store.cc:269] 0709 12:13:32.042616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:32.048768 (+  6152us) service_pool.cc:224] Handling call
0709 12:13:34.048355 (+1999587us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:34.048554 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053472640}
I20260709 12:13:33.991436 17271 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.096091 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.050304 17445 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:13:34.052390 12569 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
W20260709 12:13:34.100157 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.038223 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.058920 17387 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:34.055836 17222 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:34.101121 17222 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:34.092979 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.051362 17315 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.062970 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.063794 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.065140 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.105721 13260 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:34.105795 13253 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:34.106462 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 749) took 1020 ms. Trace:
I20260709 12:13:34.106873 12569 rpcz_store.cc:276] 0709 12:13:33.086380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:33.559866 (+473486us) service_pool.cc:224] Handling call
0709 12:13:34.052125 (+492259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:34.098825 (+ 46700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.106235 (+  7410us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54002459d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:34.106399 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130822400}
I20260709 12:13:34.065800 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.110000 13261 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:34.065903 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.067445 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.113292 13259 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:34.114387 13262 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:34.068529 17440 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:34.026149 17227 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
W20260709 12:13:34.070057 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.070099 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.115785 17271 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:34.116256 13257 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:34.115386 17392 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.117519 17387 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:34.118109 17387 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:34.119115 17387 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.120527 17387 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:34.121528 17399 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:34.072614 17317 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:34.073525 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.123422 12639 raft_consensus.cc:2376] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 9. Re-sending same reply.
I20260709 12:13:34.126019 13260 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:34.128031 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.128346 13261 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:34.129202 17285 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:34.129616 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.129976 17285 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:34.098112 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.102141 17222 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.132069 13253 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:34.132470 17222 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:34.102550 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:34.061390 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.134262 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:13:34.116410 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.136840 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.138530 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.139489 12640 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:34.139699 17315 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:13:34.140676 17408 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:34.141177 17408 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:34.141984 17408 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:34.142105 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.142352 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.143069 12495 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:34.070758 17417 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.144711 17417 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.145435 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.147532 17417 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.141113 17315 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:34.148922 17315 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:34.149941 17315 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.151710 17315 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:34.152936 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.133404 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.154474 17392 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:34.155359 17392 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:34.155870 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.143959 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.130491 17404 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:34.142298 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.143168 12496 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:13:34.159832 17034 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:34.160558 17034 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:34.156955 12641 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:34.161684 17034 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.162561 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.164868 12641 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:34.159108 17404 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:34.167279 17404 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.172180 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:34.075043 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65569.5us
I20260709 12:13:34.173652 12639 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:34.174942 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.176121 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.177315 12637 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:34.178546 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 133172us
I20260709 12:13:34.179296 17254 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:34.180398 13257 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:34.182006 17227 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.182283 17226 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.261001 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:34.189406 13258 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:13:34.190510 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.190709 12568 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: [11.5-11.5]
I20260709 12:13:34.191977 12571 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:34.192483 17431 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.262630 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:34.193840 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.194494 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:34.197600 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.198146 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.200273 12569 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:13:34.192013 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.278018 12572 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:34.278167 12569 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:13:34.278924 12569 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:34.280217 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85810.2us
I20260709 12:13:34.213071 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.281437 12639 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:34.218533 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 160045us
I20260709 12:13:34.131436 17285 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:34.237131 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.291040 17431 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:34.238658 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63802.5us
I20260709 12:13:34.283403 17432 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:34.292135 17432 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:34.292017 17285 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.184911 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.292811 17432 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:34.256573 13259 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:34.277586 12645 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:34.284296 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:34.270265 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.285809 17388 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:34.287408 12496 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:34.287798 17254 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:34.274302 17427 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:13:34.288889 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54782.4us
I20260709 12:13:34.292783 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.260967 17222 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.284276 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.320169 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:34.294239 12570 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:34.322324 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.311465 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.297127 12637 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:34.314244 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:34.156744 17392 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.324747 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.324299 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:34.331876 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.338546 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.134099 17419 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:34.350589 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54833.3us
I20260709 12:13:34.318375 16780 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.278154 17227 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.362425 12637 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:34.362969 12566 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:13:34.363544 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.363598 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.369030 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.375281 12637 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:34.379742 12496 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:13:34.283643 17226 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.309902 16528 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:13:34.387259 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:34.389189 12495 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:34.318706 17399 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:13:34.406845 12642 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:34.410584 12645 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:34.391269 12637 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
W20260709 12:13:34.391460 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 278192us
W20260709 12:13:34.393672 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.382345 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:34.359776 17396 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:34.359032 17317 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:34.358430 17456 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:34.421396 12641 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:34.423794 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.428373 12637 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:34.372457 17397 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:34.321557 17409 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.371258 17254 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:34.434132 12491 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:13:34.412940 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.804s (SENT) [suppressed 36 similar messages]
I20260709 12:13:34.324851 17222 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.426934 17456 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:34.337692 17350 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:13:34.368441 17440 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:13:34.447822 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.446092 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:34.468475 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:34.469192 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:34.455726 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 133455us
I20260709 12:13:34.457834 12645 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:13:34.458216 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94748.3us
I20260709 12:13:34.367858 17454 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 12:13:34.385332 17128 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:13:34.433405 17392 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:34.480329 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 226) took 58102 ms (client timeout 15000 ms). Trace:
I20260709 12:13:34.383275 17428 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:13:34.480788 12637 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:13:34.462905 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:34.493595 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.430374 17433 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:34.457598 12393 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:34.466233 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:34.435235 17427 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:34.433107 17397 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:34.382822 16780 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.499388 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.503326 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
W20260709 12:13:34.510213 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123057us
I20260709 12:13:34.454643 17456 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:34.558249 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:34.560017 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 88374.7us
I20260709 12:13:34.501008 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.481168 17137 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:34.562994 12643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.031s	sys 0.017s
I20260709 12:13:34.566748 12571 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:34.566509 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 184286us
I20260709 12:13:34.496457 17396 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:13:34.576792 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.502568 17397 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:34.579368 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.559643 17417 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.573030 12393 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 6 to 8, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:13:34.588104 12643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.032s	sys 0.017s
W20260709 12:13:34.487847 12391 rpcz_store.cc:269] 0709 12:12:36.360294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:12:36.360617 (+   323us) service_pool.cc:224] Handling call
0709 12:13:34.458407 (+58097790us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:31.766693 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:31.767500 (+   807us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:31.767558 (+    58us) write_op.cc:435] Acquired schema lock
0709 12:13:31.767578 (+    20us) tablet.cc:662] Decoding operations
0709 12:13:31.768299 (+   721us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:31.768399 (+   100us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:31.768428 (+    29us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:13:31.769010 (+   582us) tablet.cc:701] Row locks acquired
0709 12:13:31.769067 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:13:31.769308 (+   241us) write_op.cc:270] Start()
0709 12:13:31.769507 (+   199us) write_op.cc:276] Timestamp: P: 1783599211769222 usec, L: 0
0709 12:13:31.769557 (+    50us) op_driver.cc:348] REPLICATION: starting
0709 12:13:31.808032 (+ 38475us) log.cc:844] Serialized 1313 byte log entry
0709 12:13:32.442711 (+634679us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:33.011094 (+568383us) op_driver.cc:464] REPLICATION: finished
0709 12:13:33.240439 (+229345us) write_op.cc:301] APPLY: starting
0709 12:13:33.269705 (+ 29266us) tablet.cc:1367] starting BulkCheckPresence
0709 12:13:33.273144 (+  3439us) tablet.cc:1370] finished BulkCheckPresence
0709 12:13:33.273189 (+    45us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:13:33.305019 (+ 31830us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:13:33.305305 (+   286us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 12:13:33.305376 (+    71us) write_op.cc:312] APPLY: finished
0709 12:13:33.337819 (+ 32443us) log.cc:844] Serialized 41 byte log entry
0709 12:13:33.338142 (+   323us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:33.338289 (+   147us) write_op.cc:454] Released schema lock
0709 12:13:33.338370 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 12:13:33.340115 (+  1745us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:33.340163 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7136393,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":202477,"apply.run_cpu_time_us":8311,"apply.run_wall_time_us":116223,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16655,"num_ops":3,"prepare.queue_time_us":49205,"prepare.run_cpu_time_us":91755,"prepare.run_wall_time_us":336404,"raft.queue_time_us":161055,"raft.run_cpu_time_us":52958,"raft.run_wall_time_us":327315,"replication_time_us":1168044,"spinlock_wait_cycles":35305856,"thread_start_us":391937,"threads_started":4,"wal-append.queue_time_us":452012,"wal-append.run_cpu_time_us":43372,"wal-append.run_wall_time_us":2088496}]]}
I20260709 12:13:34.604094 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.605139 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:34.605903 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.598887 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.608289 12495 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:34.609506 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:34.623812 17340 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:34.631554 17314 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:34.634356 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 171541us
I20260709 12:13:34.664892 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.665189 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:34.638646 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 152106us
I20260709 12:13:34.665977 12495 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:34.561231 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.652369 17340 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:34.662268 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.668383 12496 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:34.670388 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:34.679940 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.684737 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.685153 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.667186 12393 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:34.591969 17005 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:34.686980 12637 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:34.695708 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.696368 12570 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:34.695163 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.687793 12572 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:34.673175 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:34.689968 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.663669 17417 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.583057 17433 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:34.583263 17396 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:34.661554 17323 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:34.662668 17271 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:34.720603 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.579435 17137 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:34.723515 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50447.1us
I20260709 12:13:34.730643 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:34.719179 12494 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:34.681725 17034 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:34.762246 12637 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:13:34.598300 17440 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.114s
I20260709 12:13:34.694167 17005 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:34.773223 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 205943us
I20260709 12:13:34.774262 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:13:34.721026 17449 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.315s	user 0.000s	sys 0.001s
I20260709 12:13:34.780704 12497 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:13:34.777379 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 197833us
I20260709 12:13:34.762848 12637 pending_rounds.cc:85] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:34.786929 12637 pending_rounds.cc:107] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:13:34.787276 12637 raft_consensus.cc:2889] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:34.784986 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 185996us
I20260709 12:13:34.714181 17396 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:13:34.781948 12536 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.249s (SENT)
I20260709 12:13:34.804263 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:34.739933 17317 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:34.809712 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:34.714123 17457 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:34.805110 12570 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:13:34.682458 17314 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.758788 17323 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:34.818593 12496 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:34.835340 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:34.820492 12570 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:34.824992 12637 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 4
I20260709 12:13:34.837828 12637 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:13:34.825690 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:34.839032 12637 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:34.858853 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:34.859565 12497 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:34.812831 17465 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:34.759682 17271 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.860918 12393 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:34.865092 12570 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:34.834539 17451 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.413s	user 0.001s	sys 0.001s
I20260709 12:13:34.707662 17254 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.815264 17404 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:34.904104 17340 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:34.905558 12496 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 3
I20260709 12:13:34.906193 12496 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 12:13:34.906491 12496 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:34.853048 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51802.5us
I20260709 12:13:34.817760 17483 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:34.691145 17285 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:34.907930 17483 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:34.868301 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86381us
I20260709 12:13:34.908555 17483 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:34.892391 17409 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:34.909660 17409 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:13:34.895184 17468 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:13:34.874387 17473 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:34.899976 12393 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 6 to 8, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:13:34.901751 17462 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.447s	user 0.002s	sys 0.000s
I20260709 12:13:34.815770 17388 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
W20260709 12:13:34.902171 17464 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:13:34.902486 17490 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:13:34.903570 17459 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.312s	user 0.000s	sys 0.001s
W20260709 12:13:34.903874 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78277.1us
I20260709 12:13:34.970191 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:34.904656 17404 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:34.905403 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.908133 17285 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.893549 17485 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:13:34.877576 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78174.9us
W20260709 12:13:34.973521 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 147892us
I20260709 12:13:34.817425 17227 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:34.868705 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 307516us
I20260709 12:13:34.837595 17317 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:34.839399 17034 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:34.911111 17207 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.823s	user 0.012s	sys 0.000s
I20260709 12:13:34.947516 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.947554 12393 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 9, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:34.866592 17465 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:34.911665 17226 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:34.869837 17454 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
W20260709 12:13:34.949076 17461 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.460s	user 0.000s	sys 0.002s
I20260709 12:13:34.821396 17457 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.938254 17473 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:34.985940 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:35.022733 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.966091 17467 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:35.028261 12639 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.742s	user 0.015s	sys 0.000s
I20260709 12:13:35.065721 12639 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:13:35.032545 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62419.5us
I20260709 12:13:34.979648 17404 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:35.050820 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:35.054917 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:34.964140 17490 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:35.033159 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:35.070519 12645 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:35.068785 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 398456us
I20260709 12:13:34.974895 16780 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:35.072847 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87008.7us
W20260709 12:13:35.073561 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50933us
W20260709 12:13:35.073686 12641 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.010s	sys 0.004s
I20260709 12:13:34.910449 17005 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:35.078336 12641 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.011s	sys 0.004s
I20260709 12:13:34.996083 17476 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:35.130609 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:35.090323 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:34.979507 17440 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20260709 12:13:35.080526 17490 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:35.093940 17486 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.094797 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 572) took 1123 ms. Trace:
I20260709 12:13:35.094290 17465 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.095842 17496 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:35.096699 17471 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:35.022112 17323 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.104102 17005 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:35.108111 17469 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.263s	user 0.001s	sys 0.000s
I20260709 12:13:35.108230 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:35.111917 17137 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:35.112692 17452 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.494s	user 0.000s	sys 0.002s
I20260709 12:13:35.113873 17457 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:35.169605 17456 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260709 12:13:35.170997 17456 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.158741 17417 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:35.176275 17315 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.995s	user 0.005s	sys 0.009s
W20260709 12:13:35.178323 17408 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.036s	user 0.014s	sys 0.001s
I20260709 12:13:35.053324 17034 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:34.943790 17433 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.184154 17034 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:35.185254 17432 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.867s	user 0.007s	sys 0.006s
I20260709 12:13:35.115499 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:35.153779 17496 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:35.187862 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72457.7us
I20260709 12:13:35.165156 16780 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:13:35.115566 17222 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.116520 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 757) took 1048 ms. Trace:
I20260709 12:13:35.056603 17439 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:35.132985 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:35.133570 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55243.6us
I20260709 12:13:35.131964 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:35.081277 17464 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.104s	user 0.000s	sys 0.007s
W20260709 12:13:35.149093 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98459.8us
I20260709 12:13:35.131970 12643 rpcz_store.cc:276] 0709 12:13:33.971998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:33.982061 (+ 10063us) service_pool.cc:224] Handling call
0709 12:13:33.994041 (+ 11980us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:34.664425 (+670384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:34.664470 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:34.724703 (+ 60233us) log.cc:844] Serialized 26 byte log entry
0709 12:13:35.068469 (+343766us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:35.069245 (+   776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:35.069333 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.094516 (+ 25183us) raft_consensus.cc:1685] finished
0709 12:13:35.094542 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.094725 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10658,"raft.queue_time_us":14975,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":142253,"thread_start_us":855175,"threads_started":2,"wal-append.queue_time_us":297953}
I20260709 12:13:35.070210 17207 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:35.155929 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:35.146706 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:34.981686 17472 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.153s
I20260709 12:13:35.079195 17398 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.232340 17398 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.150841 17317 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.235777 17440 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.146154 17486 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.236416 17440 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.238966 17486 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.162418 17485 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.240015 17485 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.241417 17440 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.244277 17398 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.247205 17485 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.129204 12497 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:35.120657 17454 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:13:35.185804 17465 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:35.060423 17445 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:35.188107 17496 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:35.109966 12393 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 6, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 6 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:35.146772 17467 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:35.189224 17458 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.620s	user 0.002s	sys 0.000s
I20260709 12:13:35.118237 17396 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:35.212050 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80213.9us
I20260709 12:13:35.190404 16780 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:35.193915 13254 rpcz_store.cc:276] 0709 12:13:34.067603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.067912 (+   309us) service_pool.cc:224] Handling call
0709 12:13:34.068316 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:34.211810 (+143494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:34.211846 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:34.212675 (+   829us) log.cc:844] Serialized 26 byte log entry
0709 12:13:34.315275 (+102600us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:34.338456 (+ 23181us) log.cc:844] Serialized 14 byte log entry
0709 12:13:34.444355 (+105899us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b18002d2b60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:34.654023 (+209668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.654398 (+   375us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.116184 (+461786us) raft_consensus.cc:1685] finished
0709 12:13:35.116283 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.116445 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1518,"prepare.queue_time_us":23001,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":7450,"raft.queue_time_us":369521,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":200,"spinlock_wait_cycles":40170368,"thread_start_us":454022,"threads_started":3,"wal-append.queue_time_us":155782}
I20260709 12:13:35.259383 12393 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:35.189177 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:35.192642 17397 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.586s	user 0.007s	sys 0.002s
I20260709 12:13:35.262626 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:35.267234 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:35.218070 17392 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.717s	user 0.012s	sys 0.003s
I20260709 12:13:35.267622 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:35.280501 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:35.137544 17340 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:35.171000 17285 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.217954 17439 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:35.284626 12393 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 2 to 5, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 5 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:35.230868 17315 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:35.218183 17137 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:35.278405 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.010s	sys 0.029s
I20260709 12:13:35.229933 17034 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:35.179538 17408 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.269248 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2308) took 1026 ms. Trace:
I20260709 12:13:35.284557 12488 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:35.297703 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:35.297003 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107939us
W20260709 12:13:35.303618 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.011s	sys 0.030s
W20260709 12:13:35.311168 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 708671us
I20260709 12:13:35.309840 12641 rpcz_store.cc:276] 0709 12:13:34.243067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.427858 (+184791us) service_pool.cc:224] Handling call
0709 12:13:34.432258 (+  4400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:34.432526 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:35.081285 (+648759us) log.cc:844] Serialized 14 byte log entry
0709 12:13:35.268802 (+187517us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:35.268870 (+    68us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:35.268927 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:35.269010 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.269176 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3772,"prepare.queue_time_us":669291,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":4097,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":13868,"thread_start_us":802851,"threads_started":2,"wal-append.queue_time_us":217425}
I20260709 12:13:35.217703 17432 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.270134 16780 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:35.280189 17417 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:13:35.317904 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:35.319684 12568 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:35.164160 17471 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.284691 17317 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:35.279846 17398 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.282049 17254 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.282390 17468 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.264482 17207 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.281244 17397 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.269389 17477 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:13:35.331243 17392 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:35.349445 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:35.357506 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:35.367937 17398 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.368650 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:35.303175 17137 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:35.371407 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109045us
I20260709 12:13:35.369747 17503 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 12:13:35.373862 17474 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:35.374475 17470 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:35.341225 12571 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:13:35.411710 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:35.384126 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2437) took 1484 ms. Trace:
I20260709 12:13:35.297465 17439 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:35.385957 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 756) took 1783 ms. Trace:
I20260709 12:13:35.334254 17417 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:35.353256 17317 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:35.408010 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:13:35.400645 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:35.378813 12567 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
W20260709 12:13:35.415285 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65922.1us
I20260709 12:13:35.412599 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 575) took 1439 ms. Trace:
I20260709 12:13:35.414839 12564 rpcz_store.cc:276] 0709 12:13:33.899323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:33.899754 (+   431us) service_pool.cc:224] Handling call
0709 12:13:33.917280 (+ 17526us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:33.933027 (+ 15747us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:33.933083 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:33.979949 (+ 46866us) log.cc:844] Serialized 26 byte log entry
0709 12:13:34.349482 (+369533us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:13:34.349540 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.352752 (+  3212us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.187941 (+835189us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:35.377905 (+189964us) raft_consensus.cc:1685] finished
0709 12:13:35.377932 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.384046 (+  6114us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11731,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":211,"spinlock_wait_cycles":17413888,"thread_start_us":340456,"threads_started":1,"wal-append.queue_time_us":442649}
I20260709 12:13:35.299315 17340 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.426494 17502 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:13:35.328125 17034 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.433611 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:35.436410 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:35.383651 17396 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:35.433112 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75638.4us
I20260709 12:13:35.440896 17005 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:35.421543 12563 rpcz_store.cc:276] 0709 12:13:33.602292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:33.822342 (+220050us) service_pool.cc:224] Handling call
0709 12:13:33.822805 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:33.823252 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:33.982258 (+159006us) log.cc:844] Serialized 14 byte log entry
0709 12:13:34.487709 (+505451us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:34.488333 (+   624us) log.cc:844] Serialized 26 byte log entry
0709 12:13:34.488469 (+   136us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:34.488503 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.488737 (+   234us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.385674 (+896937us) raft_consensus.cc:1685] finished
0709 12:13:35.385702 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.385882 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70582,"prepare.queue_time_us":230652,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":2175,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":14708,"thread_start_us":381576,"threads_started":2,"wal-append.queue_time_us":588212}
I20260709 12:13:35.452080 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:35.429987 17315 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:35.446568 17454 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:35.434093 17317 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:35.447005 17509 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:13:35.423074 17285 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:35.413691 17137 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:35.413532 17470 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:35.376508 17254 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.463166 12393 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:35.434288 12646 rpcz_store.cc:276] 0709 12:13:33.973082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:33.974094 (+  1012us) service_pool.cc:224] Handling call
0709 12:13:33.974486 (+   392us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:34.012421 (+ 37935us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:34.012455 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:34.013139 (+   684us) log.cc:844] Serialized 52 byte log entry
0709 12:13:34.306030 (+292891us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:34.306101 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.306376 (+   275us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.111946 (+805570us) spinlock_profiling.cc:243] Waited 64 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:35.409061 (+297115us) raft_consensus.cc:1685] finished
0709 12:13:35.409087 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.412516 (+  3429us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4865,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"spinlock_wait_cycles":147254016,"thread_start_us":292307,"threads_started":1,"wal-append.queue_time_us":441473}
I20260709 12:13:35.429292 17417 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:35.475723 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:35.450484 17427 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.878s	user 0.011s	sys 0.003s
I20260709 12:13:35.438123 17387 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:35.487613 17227 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
W20260709 12:13:35.492432 17506 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:13:35.503309 17510 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:35.463960 17285 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:35.504354 17481 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:13:35.506669 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:35.599817 17404 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.507148 17006 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:35.602734 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2313) took 1282 ms. Trace:
I20260709 12:13:35.603132 17006 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.603766 17006 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.603207 12638 rpcz_store.cc:276] 0709 12:13:34.320767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.680218 (+359451us) service_pool.cc:224] Handling call
0709 12:13:34.680568 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:34.860182 (+179614us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:35.217537 (+357355us) log.cc:844] Serialized 14 byte log entry
0709 12:13:35.345859 (+128322us) log.cc:844] Serialized 14 byte log entry
0709 12:13:35.346296 (+   437us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:35.347074 (+   778us) log.cc:844] Serialized 26 byte log entry
0709 12:13:35.347258 (+   184us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:35.347307 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:35.347411 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.602488 (+255077us) raft_consensus.cc:1685] finished
0709 12:13:35.602510 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:35.602668 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16785,"prepare.queue_time_us":249921,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":80,"raft.queue_time_us":89521,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":31402,"spinlock_wait_cycles":41401344,"thread_start_us":517757,"threads_started":3,"wal-append.queue_time_us":162951}
I20260709 12:13:35.482197 17433 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:13:35.607206 17471 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.511672 17482 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:35.608258 17419 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:35.457466 17340 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:35.482534 17503 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:13:35.609532 17503 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:35.610073 17503 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:35.610466 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 757) took 1552 ms. Trace:
W20260709 12:13:35.553498 17466 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.643s	user 0.001s	sys 0.001s
I20260709 12:13:35.531132 12393 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:35.382462 17468 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:35.538622 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102297us
W20260709 12:13:35.530644 17483 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.003s
I20260709 12:13:35.645169 13258 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:13:35.462718 17477 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.413074 17467 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.544886 17470 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.545917 17507 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 12:13:35.551499 17410 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:35.563580 17254 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.576685 17440 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:35.577338 17398 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.580823 17397 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:35.530755 17315 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:35.488626 17485 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:35.487797 17472 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:35.412650 16780 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:35.592474 17285 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:35.594038 17227 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:13:35.596892 17497 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.597256 17474 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.538095 17387 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:35.597664 17457 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.542650 17427 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:35.598466 17271 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.487290 17454 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:35.458546 17445 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 12:13:35.618563 17432 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:35.529472 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53891.9us
I20260709 12:13:35.931851 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:35.619421 16528 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:35.620451 17006 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.611011 17503 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:35.624672 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:35.610915 12567 rpcz_store.cc:276] 0709 12:13:34.057727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.060138 (+  2411us) service_pool.cc:224] Handling call
0709 12:13:35.378418 (+1318280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:35.432189 (+ 53771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:35.610203 (+178014us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001a36d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:35.610386 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3030890752}
I20260709 12:13:35.640555 13261 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:13:35.655673 17314 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:35.957871 12568 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.067s	sys 0.092s
W20260709 12:13:35.958423 12568 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.067s	sys 0.093s
I20260709 12:13:35.952963 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:35.960281 12571 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.010s	sys 0.020s
W20260709 12:13:35.643337 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186828us
I20260709 12:13:35.645299 12393 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 5 to 10, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 10 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:35.649255 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:35.670565 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:35.671954 12646 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:35.763286 12495 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:35.885766 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:35.652086 17388 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:35.660934 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:35.928020 17485 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:35.648618 13255 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:35.826190 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:35.826378 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:35.968765 12644 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:35.862609 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:35.787667 12642 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:13:35.827816 12492 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:35.970359 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:35.971191 12642 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10.
I20260709 12:13:35.652992 17408 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:35.864491 12490 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:13:35.967864 12393 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 3 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 6 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:35.664657 13257 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:13:35.672165 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:35.665303 13260 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:13:35.661994 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2312) took 1351 ms. Trace:
I20260709 12:13:35.648453 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:35.648453 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:35.672690 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:35.690478 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:35.663497 13259 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: [12.9-15.10]
W20260709 12:13:35.854173 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64816.8us
I20260709 12:13:35.929500 12489 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:35.935596 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:35.637691 17428 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.047s	user 0.016s	sys 0.018s
W20260709 12:13:35.979583 17428 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.396s	user 0.016s	sys 0.018s
I20260709 12:13:35.963052 12637 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:35.782632 17340 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.724102 17476 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:35.979616 12640 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 12:13:35.827064 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:35.981395 12494 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:35.982565 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:35.983290 12494 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:35.985690 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50296.8us
I20260709 12:13:35.986989 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:35.656373 17399 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:35.964470 12643 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 12:13:35.966001 12496 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.006s	sys 0.017s
W20260709 12:13:35.964382 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.010s	sys 0.020s
I20260709 12:13:35.667174 17468 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:35.762661 17489 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:35.650669 17034 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:35.899324 17410 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:35.900632 17440 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:35.649175 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:35.664399 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:35.989061 12496 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.006s	sys 0.018s
I20260709 12:13:35.923430 17227 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:35.953526 17419 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:35.945350 16528 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:35.997751 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:35.974519 12645 rpcz_store.cc:276] 0709 12:13:34.310752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.568332 (+257580us) service_pool.cc:224] Handling call
0709 12:13:35.068730 (+500398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:35.080105 (+ 11375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:35.661658 (+581553us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54000a87d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:35.661905 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080270208}
I20260709 12:13:35.648773 17465 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.323s
I20260709 12:13:36.000428 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:35.951896 17474 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.006107 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:36.007385 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:35.980877 17486 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.039781 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:36.014600 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.014400 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:35.648044 17483 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:36.019062 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 221668us
W20260709 12:13:36.045183 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47611us
I20260709 12:13:36.028162 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.028385 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.028999 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:35.998138 17207 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.030331 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:35.981482 17485 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:35.967859 17388 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:36.040802 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:36.093288 13261 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:36.093801 13255 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:36.094875 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2439) took 1911 ms. Trace:
I20260709 12:13:36.095304 12569 rpcz_store.cc:276] 0709 12:13:34.183522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.199545 (+ 16023us) service_pool.cc:224] Handling call
0709 12:13:34.199978 (+   433us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:34.297219 (+ 97241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:34.297257 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:34.312172 (+ 14915us) log.cc:844] Serialized 52 byte log entry
0709 12:13:34.635675 (+323503us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:34.635709 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.635773 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:35.553080 (+917307us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b24001522b8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:13:36.075241 (+522161us) raft_consensus.cc:1685] finished
0709 12:13:36.075266 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.094795 (+ 19529us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97,"raft.queue_time_us":105,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8241,"spinlock_wait_cycles":7740672,"thread_start_us":275731,"threads_started":1,"wal-append.queue_time_us":467650}
I20260709 12:13:36.041105 12497 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:35.998436 17399 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:35.980590 17006 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:35.950205 17457 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.022712 13258 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
I20260709 12:13:35.996279 17427 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:36.029187 12393 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:36.043504 17465 consensus_queue.cc:1243] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:13:36.056751 17314 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:36.046451 13218 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.604s (SENT)
W20260709 12:13:36.115686 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 128781us
I20260709 12:13:36.116252 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:36.057206 17419 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:35.952289 17477 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.008877 17503 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:36.058761 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:36.059238 17439 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.624s	user 0.007s	sys 0.005s
I20260709 12:13:36.008103 17340 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.018200 17404 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.017526 17456 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.060441 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:36.062006 17440 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.030870 17408 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:36.062938 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.031008 17489 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.063700 17486 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.064122 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.065546 17483 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:36.068339 17005 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.000s
I20260709 12:13:36.069087 12565 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:36.057240 13253 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:36.073103 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:35.943980 17497 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.029832 17387 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:36.033696 17496 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.765s	user 0.014s	sys 0.000s
W20260709 12:13:36.101788 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.181502 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:36.112452 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.006230 17510 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
W20260709 12:13:36.121783 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.004s	sys 0.004s
I20260709 12:13:36.183789 12569 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:36.021522 17454 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.122165 12393 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:13:36.184312 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2455) took 1086 ms. Trace:
I20260709 12:13:36.126912 13262 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:36.127723 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2628) took 1279 ms. Trace:
I20260709 12:13:36.157387 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:36.151243 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104817us
I20260709 12:13:36.170323 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:36.122506 17314 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:36.188397 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.246s (SENT)
I20260709 12:13:36.128703 17419 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:36.150373 17465 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:36.190407 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.190979 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:36.191560 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:36.187352 12496 rpcz_store.cc:276] 0709 12:13:34.848636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.863128 (+ 14492us) service_pool.cc:224] Handling call
0709 12:13:34.905199 (+ 42071us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:35.213067 (+307868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:35.213099 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:35.213828 (+   729us) log.cc:844] Serialized 52 byte log entry
0709 12:13:35.379521 (+165693us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:36.009750 (+630229us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.046268 (+ 36518us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.127263 (+ 80995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.127372 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:36.127423 (+    51us) raft_consensus.cc:1685] finished
0709 12:13:36.127436 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.127644 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":243591,"prepare.queue_time_us":65682,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":91,"raft.queue_time_us":237126,"raft.run_cpu_time_us":600,"raft.run_wall_time_us":71719,"thread_start_us":970893,"threads_started":3,"wal-append.queue_time_us":271276}
I20260709 12:13:36.195439 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:36.196583 12569 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:36.186933 12563 rpcz_store.cc:276] 0709 12:13:35.098143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.452530 (+354387us) service_pool.cc:224] Handling call
0709 12:13:35.463273 (+ 10743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:35.463528 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:35.591010 (+127482us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.183935 (+592925us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:36.184014 (+    79us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:36.184060 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.184128 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.184253 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":108190,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":2422,"thread_start_us":675524,"threads_started":2}
I20260709 12:13:36.132336 17350 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:13:36.058789 16528 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.174849 17440 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:36.206475 12639 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:36.122692 17006 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.242683 12645 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:13:36.203491 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:36.262972 12644 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:36.217073 12393 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 4 to 9, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:36.202450 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.175182 17005 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:36.205164 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:36.202046 17516 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.247929 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:36.200901 17419 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:36.251398 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.240617 17323 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.253680 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.137136 17439 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.193849 17507 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.190035 17314 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:36.217077 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 552767us
I20260709 12:13:36.311058 12563 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:36.312141 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.200915 17526 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.242379 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:36.122005 17227 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.325691 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:36.358404 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.188892 17496 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.316039 17427 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.133410 17410 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.366515 12572 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:36.343915 12490 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:36.347427 12569 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:36.337358 17137 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.799s	user 0.008s	sys 0.004s
I20260709 12:13:36.367941 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.368916 12569 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:36.328359 16528 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:36.377465 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.370042 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:36.379025 12643 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14. Re-sending same reply.
I20260709 12:13:36.370103 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.380016 12569 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:36.376724 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:36.381743 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.383703 12497 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:36.383181 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:36.336176 17445 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.384330 12569 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:13:36.388136 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.357614 17526 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.395078 12496 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:36.397327 17526 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:36.398316 17526 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:13:36.399314 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.399907 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.400296 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.400823 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.401317 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.402056 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.402837 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:36.404269 12572 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:36.404937 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 763) took 1216 ms. Trace:
I20260709 12:13:36.405766 17488 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:36.406383 17488 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:36.407334 17488 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.389475 17397 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.408885 17397 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:36.410758 17531 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
I20260709 12:13:36.411558 17397 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.389194 17227 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:36.412485 17315 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:36.406121 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:36.416580 12643 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:36.406009 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.418109 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.419101 12492 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:36.405256 12572 rpcz_store.cc:276] 0709 12:13:35.189246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.403619 (+1214373us) service_pool.cc:224] Handling call
0709 12:13:36.404058 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:36.404689 (+   631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.404880 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:36.430060 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:36.431157 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:36.434772 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:36.442509 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.443234 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.460064 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:36.462419 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:36.464277 12491 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:36.212132 17388 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.334668 17516 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.475946 12644 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:36.476363 12644 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:36.477150 12644 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.477816 17516 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.482028 17457 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:36.482697 17457 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.483209 17457 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:36.483866 17417 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.895s	user 0.010s	sys 0.000s
I20260709 12:13:36.351071 17323 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.486183 17477 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.487332 17323 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:36.487903 17323 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:36.487332 17497 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:36.489769 17497 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:13:36.490828 17490 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.309s	user 0.011s	sys 0.003s
I20260709 12:13:36.491271 17490 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.492470 12642 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:13:36.493129 17317 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.978s	user 0.013s	sys 0.000s
I20260709 12:13:36.386409 17137 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.495790 17137 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:36.496491 17137 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:36.436950 17226 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.497059 17137 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:36.497269 17226 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.502992 17006 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:36.387285 17533 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:36.504755 17533 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:36.506315 17528 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:13:36.507939 17533 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:36.419400 17445 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.432031 17410 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:36.508639 17222 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:13:36.376310 17427 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.515599 17476 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:36.488351 17323 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.522194 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.527738 12572 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:36.513926 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.529008 12496 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:36.530936 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68644.3us
I20260709 12:13:36.535530 12488 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:36.470809 17474 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.542452 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.543289 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:36.543711 17484 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.445s	user 0.008s	sys 0.002s
I20260709 12:13:36.544718 17489 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:36.545933 17535 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:13:36.455464 17397 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.547277 17439 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:36.546926 17137 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.548902 17340 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:36.551383 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.552076 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2449) took 1469 ms. Trace:
I20260709 12:13:36.571805 12490 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:36.360725 17507 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.559876 12572 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:36.562914 12461 leader_election.cc:400] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10.
I20260709 12:13:36.536753 17476 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:13:36.565569 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.581588 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:36.582607 12490 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:36.484961 17417 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:36.570255 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 220434us
I20260709 12:13:36.537222 17445 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:36.571645 12571 rpcz_store.cc:276] 0709 12:13:35.082702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.427917 (+345215us) service_pool.cc:224] Handling call
0709 12:13:35.428292 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:35.428660 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:35.988889 (+560229us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.324565 (+335676us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:36.340078 (+ 15513us) log.cc:844] Serialized 26 byte log entry
0709 12:13:36.340228 (+   150us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:36.340261 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.340324 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:36.551797 (+211473us) raft_consensus.cc:1685] finished
0709 12:13:36.551822 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.552010 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2229,"prepare.queue_time_us":142856,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":4088,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":30775,"thread_start_us":834112,"threads_started":2,"wal-append.queue_time_us":528450}
W20260709 12:13:36.602209 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 411284us
I20260709 12:13:36.602167 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:36.604130 12572 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:36.614581 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 186600us
I20260709 12:13:36.577605 12461 leader_election.cc:403] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:13:36.493605 17317 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:36.647833 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.648653 12490 raft_consensus.cc:2376] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10. Re-sending same reply.
W20260709 12:13:36.644207 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111680us
I20260709 12:13:36.649981 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:36.650274 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:36.651841 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:36.568809 17397 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:36.661221 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:36.662693 12571 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:13:36.781752 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.817199 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.842782 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.951s	user 0.009s	sys 0.021s
W20260709 12:13:36.843591 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.010s	sys 0.022s
I20260709 12:13:36.662554 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:36.848416 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 185989us
I20260709 12:13:36.878383 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:36.884418 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 232157us
I20260709 12:13:36.888617 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2495) took 2414 ms. Trace:
I20260709 12:13:36.892062 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 623) took 1195 ms. Trace:
I20260709 12:13:36.892524 12493 rpcz_store.cc:276] 0709 12:13:35.697156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.762422 (+ 65266us) service_pool.cc:224] Handling call
0709 12:13:35.862251 (+ 99829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:35.862489 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:13:35.862787 (+   298us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.891425 (+1028638us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.891684 (+   259us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:36.891745 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:36.891790 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.891870 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.892009 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3283,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":7820,"raft.queue_time_us":253,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":107,"thread_start_us":980994,"threads_started":1}
I20260709 12:13:36.894138 17541 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:36.894435 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.889055 12566 rpcz_store.cc:276] 0709 12:13:34.474494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:34.477468 (+  2974us) service_pool.cc:224] Handling call
0709 12:13:34.498815 (+ 21347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:34.631706 (+132891us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:34.631742 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:34.632357 (+   615us) log.cc:844] Serialized 26 byte log entry
0709 12:13:34.635544 (+  3187us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:34.635618 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:34.635740 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:36.543286 (+1907546us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b24001522b8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:13:36.669378 (+126092us) raft_consensus.cc:1685] finished
0709 12:13:36.669418 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.888534 (+219116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69065,"raft.queue_time_us":94749,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":23629184,"thread_start_us":2408,"threads_started":1,"wal-append.queue_time_us":355370}
I20260709 12:13:36.655318 17005 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:36.900050 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.003s	sys 0.029s
I20260709 12:13:36.912243 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2094) took 1113 ms. Trace:
W20260709 12:13:36.900360 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.011s	sys 0.004s
I20260709 12:13:36.645915 17445 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:36.904830 12646 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 12:13:36.853380 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.663462 17472 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:36.915110 12494 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:13:36.879065 17477 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:36.916222 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.003s	sys 0.029s
I20260709 12:13:36.885874 17340 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:36.913825 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.920606 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:36.657498 17207 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:13:36.974048 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.022s	sys 0.007s
I20260709 12:13:36.931514 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:36.663820 17457 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:36.888037 17489 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:36.935524 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2456) took 1837 ms. Trace:
W20260709 12:13:36.916818 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.012s	sys 0.004s
I20260709 12:13:36.935681 12566 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260709 12:13:37.067431 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 898143us
I20260709 12:13:37.068328 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2506) took 1505 ms. Trace:
I20260709 12:13:36.960565 17477 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:36.878422 17476 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:13:36.900434 17533 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:36.947628 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:37.096626 13255 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:36.657792 17350 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.272s
I20260709 12:13:36.969796 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:36.957669 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:36.667353 17433 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.268s
I20260709 12:13:36.936156 17543 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:13:36.952123 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:36.971148 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54673.4us
W20260709 12:13:37.106246 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:36.916667 12492 rpcz_store.cc:276] 0709 12:13:35.798201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.001438 (+203237us) service_pool.cc:224] Handling call
0709 12:13:36.895965 (+894527us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:36.911638 (+ 15673us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":787191040}
I20260709 12:13:36.946667 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:37.110317 13257 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:36.905992 17314 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.004s
W20260709 12:13:36.975548 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.023s	sys 0.008s
W20260709 12:13:37.060671 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140153us
I20260709 12:13:36.945365 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2458) took 1711 ms. Trace:
I20260709 12:13:37.068904 12566 rpcz_store.cc:276] 0709 12:13:35.563226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.917502 (+1354276us) service_pool.cc:224] Handling call
0709 12:13:36.931575 (+ 14073us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:37.067630 (+136055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:37.068217 (+   587us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:36.933153 17445 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:37.063969 17454 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.032s	sys 0.135s
I20260709 12:13:37.112041 12493 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:13:37.065994 12570 rpcz_store.cc:276] 0709 12:13:35.098464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.551867 (+453403us) service_pool.cc:224] Handling call
0709 12:13:35.552421 (+   554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:35.552679 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:36.024476 (+471797us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.934927 (+910451us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:36.934998 (+    71us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:36.935054 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.935134 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.935441 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19236,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":1355,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":110,"thread_start_us":1360805,"threads_started":2}
I20260709 12:13:37.127970 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.115784 12571 rpcz_store.cc:276] 0709 12:13:35.234309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.909788 (+1675479us) service_pool.cc:224] Handling call
0709 12:13:36.944603 (+ 34815us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:36.944870 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:36.944912 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:36.944972 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:36.945020 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.945099 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:36.945275 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.120884 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.128652 12570 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:37.121398 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:37.129665 12492 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:37.122066 17397 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:37.123378 17445 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:37.125032 17485 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.015s	user 0.005s	sys 0.006s
I20260709 12:13:37.126216 17317 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.115268 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:37.089596 17489 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:36.951517 17340 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.099965 17490 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.095777 17222 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:37.095607 17482 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:37.139048 17227 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.723s	user 0.012s	sys 0.000s
W20260709 12:13:37.133879 12393 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.056s	sys 0.092s
W20260709 12:13:37.151301 12393 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.056s	sys 0.092s
I20260709 12:13:37.142921 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:37.126951 17454 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.032s	sys 0.135s
I20260709 12:13:37.116549 17323 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.152145 12566 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:37.105823 17543 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:37.149010 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.162279 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.167135 12493 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:37.168030 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2097) took 1309 ms. Trace:
I20260709 12:13:37.168452 12493 rpcz_store.cc:276] 0709 12:13:35.859059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.139919 (+1280860us) service_pool.cc:224] Handling call
0709 12:13:37.167944 (+ 28025us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.142364 17489 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:37.139268 17445 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.146865 17397 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:37.159277 17539 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.584s	user 0.002s	sys 0.000s
I20260709 12:13:37.184115 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 620) took 1492 ms. Trace:
W20260709 12:13:37.240985 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:37.240063 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:37.165786 17285 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.215s	user 0.014s	sys 0.000s
I20260709 12:13:37.239511 17397 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:37.244802 17507 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.185559 17445 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.165673 17482 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.165319 17541 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:37.272890 16528 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.873s	user 0.009s	sys 0.000s
W20260709 12:13:37.286659 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:37.287412 17340 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:37.165309 17222 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:37.127077 17314 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.143491 17485 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.579653 17485 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:37.290284 17542 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.648s	user 0.002s	sys 0.000s
W20260709 12:13:37.495273 17553 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:13:37.314878 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.555318 12570 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 12:13:37.303984 17419 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.951s	user 0.011s	sys 0.000s
I20260709 12:13:37.588840 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:37.301304 17417 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:13:37.360545 17410 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.851s	user 0.011s	sys 0.002s
I20260709 12:13:37.290536 17473 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.591820 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 696) took 1553 ms. Trace:
I20260709 12:13:37.592128 17473 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:37.592846 12489 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
W20260709 12:13:37.414911 13218 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.037s	sys 0.017s
W20260709 12:13:37.597942 13218 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.037s	sys 0.018s
W20260709 12:13:37.598408 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 932885us
W20260709 12:13:37.599047 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:37.601924 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 617) took 1911 ms (client timeout 1549 ms). Trace:
I20260709 12:13:37.602775 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2335) took 1532 ms. Trace:
I20260709 12:13:37.603325 12638 rpcz_store.cc:276] 0709 12:13:36.070731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.071072 (+   341us) service_pool.cc:224] Handling call
0709 12:13:36.116213 (+ 45141us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:36.116727 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:36.116768 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:36.127406 (+ 10638us) log.cc:844] Serialized 52 byte log entry
0709 12:13:36.339072 (+211666us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:36.339133 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.339212 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:37.602447 (+1263235us) raft_consensus.cc:1685] finished
0709 12:13:37.602474 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:37.602681 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3834,"raft.queue_time_us":109,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":8027,"spinlock_wait_cycles":1318144,"thread_start_us":152492,"threads_started":1,"wal-append.queue_time_us":396014}
I20260709 12:13:37.605878 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:37.605759 17419 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.607227 12637 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:37.606762 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:37.608745 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:37.609597 12638 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:37.602603 12488 rpcz_store.cc:269] 0709 12:13:35.690199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.690775 (+   576us) service_pool.cc:224] Handling call
0709 12:13:37.601585 (+1910810us) spinlock_profiling.cc:243] Waited 69.5 ms on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:37.601783 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159907072}
I20260709 12:13:37.611104 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.611958 12488 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:37.612635 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2114) took 1249 ms. Trace:
I20260709 12:13:37.113497 17533 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:37.417798 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 331035us
I20260709 12:13:37.414331 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:37.420783 12496 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.891s	user 0.008s	sys 0.004s
I20260709 12:13:37.560168 17490 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:37.496567 17440 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.184s	user 0.006s	sys 0.008s
I20260709 12:13:37.567263 17541 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.343587 17477 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:37.495815 17550 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.307s	user 0.001s	sys 0.000s
W20260709 12:13:37.569396 17503 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.409s	user 0.008s	sys 0.004s
I20260709 12:13:37.570266 17479 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:13:37.571770 17549 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.213s	user 0.000s	sys 0.000s
I20260709 12:13:37.573834 17285 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.574879 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:37.615906 17538 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.575342 17397 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:37.760804 17314 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:37.652303 17417 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:37.581970 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.765697 12570 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:37.461876 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:37.556133 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59373.5us
W20260709 12:13:37.583376 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:37.444108 12572 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.834s	user 0.013s	sys 0.001s
W20260709 12:13:37.454366 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 311217us
I20260709 12:13:37.767444 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:37.584081 12491 rpcz_store.cc:276] 0709 12:13:35.691750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.692358 (+   608us) service_pool.cc:224] Handling call
0709 12:13:35.797578 (+105220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:36.464971 (+667393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:37.183821 (+718850us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:37.184036 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1532629376}
I20260709 12:13:37.581182 12571 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:37.345484 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.590219 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2109) took 1540 ms. Trace:
I20260709 12:13:37.769861 12492 rpcz_store.cc:276] 0709 12:13:36.049360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.109651 (+1060291us) service_pool.cc:224] Handling call
0709 12:13:37.590143 (+480492us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.771270 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.772284 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2127) took 1290 ms. Trace:
I20260709 12:13:37.772696 12492 rpcz_store.cc:276] 0709 12:13:36.481611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.770852 (+1289241us) service_pool.cc:224] Handling call
0709 12:13:37.772200 (+  1348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.613058 12488 rpcz_store.cc:276] 0709 12:13:36.363247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.610820 (+1247573us) service_pool.cc:224] Handling call
0709 12:13:37.612552 (+  1732us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.592490 13262 rpcz_store.cc:276] 0709 12:13:36.038805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.039315 (+   510us) service_pool.cc:224] Handling call
0709 12:13:36.114812 (+ 75497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:36.187287 (+ 72475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:37.591479 (+1404192us) spinlock_profiling.cc:243] Waited 54.7 ms on lock 0x7b540056bdd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:37.591723 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125848960}
I20260709 12:13:37.648762 12563 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:13:37.651073 12496 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:37.774344 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2091) took 1990 ms (client timeout 2133 ms). Trace:
I20260709 12:13:37.774595 12493 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:37.774777 12496 rpcz_store.cc:269] 0709 12:13:35.783916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.432916 (+649000us) service_pool.cc:224] Handling call
0709 12:13:37.774276 (+1341360us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:37.775187 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2634) took 1636 ms (client timeout 2015 ms). Trace:
W20260709 12:13:37.775533 12493 rpcz_store.cc:269] 0709 12:13:36.139222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.169396 (+1030174us) service_pool.cc:224] Handling call
0709 12:13:37.775123 (+605727us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.775856 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.776564 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2115) took 1360 ms. Trace:
I20260709 12:13:37.776753 12496 rpcz_store.cc:276] 0709 12:13:36.415866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.775530 (+1359664us) service_pool.cc:224] Handling call
0709 12:13:37.776519 (+   989us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:37.657240 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4034935250001581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1684s for a response
I20260709 12:13:37.577491 17445 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.660763 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:37.522959 17207 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:37.608774 17419 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.648888 17547 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:13:37.655601 17396 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.287s	user 0.044s	sys 0.060s
I20260709 12:13:37.450069 17323 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.741933 12495 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:37.818768 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.819373 12571 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 9: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:37.819283 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.820670 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2119) took 1341 ms. Trace:
I20260709 12:13:37.821043 12496 rpcz_store.cc:276] 0709 12:13:36.479343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.777153 (+1297810us) service_pool.cc:224] Handling call
0709 12:13:37.820601 (+ 43448us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.821336 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.822093 12492 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:37.822489 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2640) took 1656 ms (client timeout 2077 ms). Trace:
I20260709 12:13:37.822400 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:37.823273 12492 rpcz_store.cc:269] 0709 12:13:36.166015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.791260 (+1625245us) service_pool.cc:224] Handling call
0709 12:13:37.822409 (+ 31149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.653788 17546 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.655200 17317 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.567773 17543 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.624065 17556 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.121s
I20260709 12:13:37.656497 17490 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.754804 12564 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:13:37.755895 17285 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:37.833503 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.757779 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2457) took 2523 ms. Trace:
I20260709 12:13:37.834317 12569 rpcz_store.cc:276] 0709 12:13:35.233995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.414669 (+1180674us) service_pool.cc:224] Handling call
0709 12:13:36.415027 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:36.456359 (+ 41332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:36.456414 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:36.514148 (+ 57734us) log.cc:844] Serialized 26 byte log entry
0709 12:13:36.919346 (+405198us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:36.919417 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:36.919862 (+   445us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:37.757347 (+837485us) raft_consensus.cc:1685] finished
0709 12:13:37.757384 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:37.757679 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5460,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":68316,"spinlock_wait_cycles":1606528,"thread_start_us":404362,"threads_started":1,"wal-append.queue_time_us":727993}
I20260709 12:13:37.836520 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.837239 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:37.758446 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.838688 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.839881 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.840472 12566 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:37.579483 17399 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.843014 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:37.843633 17489 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.000s
W20260709 12:13:37.763055 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183108us
I20260709 12:13:37.615142 17454 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:37.767048 12645 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:37.844861 17454 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:37.777379 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4007935250001537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
W20260709 12:13:37.826551 17005 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.902s	user 0.008s	sys 0.006s
I20260709 12:13:37.784047 12638 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:37.785926 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.786697 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.823927 17440 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.855131 12570 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:37.855160 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3697717659999853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3295s for a response
I20260709 12:13:37.806180 17533 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:37.814330 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 190247us
I20260709 12:13:37.816633 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.858552 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:37.859107 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2124) took 1378 ms (client timeout 1766 ms). Trace:
W20260709 12:13:37.859745 12488 rpcz_store.cc:269] 0709 12:13:36.480795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.773858 (+1293063us) service_pool.cc:224] Handling call
0709 12:13:37.858801 (+ 84943us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:37.826159 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:37.826833 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:37.860831 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:37.861220 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:37.862226 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:37.864148 17496 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:37.828559 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:37.786953 17479 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:37.866235 12496 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:37.657825 17507 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:37.824749 17396 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.463s	user 0.048s	sys 0.073s
I20260709 12:13:37.831751 17546 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.867980 13262 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:37.868101 17507 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.857213 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2141) took 1200 ms. Trace:
I20260709 12:13:37.816320 17222 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.869765 12638 rpcz_store.cc:276] 0709 12:13:36.657381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.615370 (+957989us) service_pool.cc:224] Handling call
0709 12:13:37.857119 (+241749us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.870939 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:37.845263 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2137) took 1198 ms (client timeout 1591 ms). Trace:
W20260709 12:13:37.873531 12645 rpcz_store.cc:269] 0709 12:13:36.647113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.195978 (+548865us) service_pool.cc:224] Handling call
0709 12:13:37.845184 (+649206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.874547 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:37.874410 17473 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:37.875412 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2140) took 1227 ms. Trace:
I20260709 12:13:37.783089 17538 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.875571 17473 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:13:37.875833 12645 rpcz_store.cc:276] 0709 12:13:36.648248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.874259 (+1226011us) service_pool.cc:224] Handling call
0709 12:13:37.875356 (+  1097us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.767501 12572 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:37.841459 17399 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:37.877092 17563 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:13:37.846521 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79162.5us
I20260709 12:13:37.877657 17563 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:37.876829 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:37.879207 12638 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:37.878944 17563 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.879817 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2135) took 1234 ms. Trace:
I20260709 12:13:37.817466 17207 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.880472 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:37.881399 17479 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:37.882355 17207 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:37.883316 12462 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
I20260709 12:13:37.883689 12462 leader_election.cc:403] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:13:37.884281 17207 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:37.886663 17128 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:37.887193 17128 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:37.844036 17489 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.847570 17465 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:37.888361 17465 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:37.888171 17128 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:37.848814 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2452) took 2763 ms (client timeout 2452 ms). Trace:
I20260709 12:13:37.849689 12571 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:37.889312 17489 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:37.890069 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.890774 12571 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:37.785779 17417 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:37.783362 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:37.889233 12565 rpcz_store.cc:269] 0709 12:13:35.085730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.146286 (+ 60556us) service_pool.cc:224] Handling call
0709 12:13:37.848529 (+2702243us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:37.848696 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1924609920}
I20260709 12:13:37.844846 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2509) took 2280 ms. Trace:
I20260709 12:13:37.852415 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.896075 12569 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:37.856411 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2954935250002109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260709 12:13:37.857225 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2116) took 1378 ms (client timeout 1704 ms). Trace:
W20260709 12:13:37.897507 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2837935250001919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1817s for a response
I20260709 12:13:37.859397 12570 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:37.735208 17557 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:37.865267 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:37.866698 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2118) took 1387 ms. Trace:
I20260709 12:13:37.900403 12496 rpcz_store.cc:276] 0709 12:13:36.479085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.828324 (+1349239us) service_pool.cc:224] Handling call
0709 12:13:37.866640 (+ 38316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.832671 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.866149 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:37.860844 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.873051 17546 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:37.877310 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2478) took 1310 ms (client timeout 1638 ms). Trace:
I20260709 12:13:37.888746 17465 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:37.848016 17490 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:37.892964 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
W20260709 12:13:37.923758 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2636) took 1783 ms (client timeout 2205 ms). Trace:
I20260709 12:13:37.855667 17005 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10.
W20260709 12:13:37.895280 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:37.925954 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59775.1us
I20260709 12:13:37.804983 17445 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:37.897730 12491 rpcz_store.cc:269] 0709 12:13:36.478440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.769034 (+1290594us) service_pool.cc:224] Handling call
0709 12:13:37.857148 (+ 88114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.895710 12566 rpcz_store.cc:276] 0709 12:13:35.564094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.844010 (+2279916us) service_pool.cc:224] Handling call
0709 12:13:37.844339 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:37.844512 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:37.844535 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:37.844585 (+    50us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:37.844614 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:37.844664 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:37.844766 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.891343 17516 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:37.898155 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2726910290000433s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31200000000000006s for a response
I20260709 12:13:37.900496 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:37.871210 17507 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:37.918196 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2130) took 1297 ms. Trace:
W20260709 12:13:37.917737 12572 rpcz_store.cc:269] 0709 12:13:36.566580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.587850 (+ 21270us) service_pool.cc:224] Handling call
0709 12:13:37.877245 (+1289395us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.918448 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.901151 12570 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:37.918917 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.941983 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.942665 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2131) took 1056 ms. Trace:
I20260709 12:13:37.942914 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:37.943128 12496 rpcz_store.cc:276] 0709 12:13:36.886491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.918608 (+1032117us) service_pool.cc:224] Handling call
0709 12:13:37.942593 (+ 23985us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.919246 12492 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:37.824195 17340 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
W20260709 12:13:37.924314 12493 rpcz_store.cc:269] 0709 12:13:36.140074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.787392 (+1647318us) service_pool.cc:224] Handling call
0709 12:13:37.923648 (+136256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.880331 12638 rpcz_store.cc:276] 0709 12:13:36.645347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.870655 (+1225308us) service_pool.cc:224] Handling call
0709 12:13:37.879749 (+  9094us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.825824 17323 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.932579 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
W20260709 12:13:37.934628 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:37.934881 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:37.920107 17128 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:37.880512 17563 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:37.940224 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2603935250001541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
I20260709 12:13:37.944209 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2117) took 1465 ms. Trace:
I20260709 12:13:37.944367 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:37.970233 12536 connection.cc:256] RPC call timeout handler was delayed by -1.207898398999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
I20260709 12:13:37.940662 12488 rpcz_store.cc:276] 0709 12:13:36.620772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.860516 (+1239744us) service_pool.cc:224] Handling call
0709 12:13:37.918132 (+ 57616us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.947716 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:37.970769 12492 rpcz_store.cc:276] 0709 12:13:36.478801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.823811 (+1345010us) service_pool.cc:224] Handling call
0709 12:13:37.944132 (+120321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.972014 12493 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:37.972659 12569 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:37.972774 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2129) took 1380 ms. Trace:
I20260709 12:13:37.965734 17397 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.932250 17417 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:37.955952 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:37.939615 17546 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:37.964058 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:37.933044 17482 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:37.985352 12490 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:37.964058 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:37.981032 12565 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9. Re-sending same reply.
I20260709 12:13:37.986915 12491 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:37.944969 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2513) took 2379 ms. Trace:
I20260709 12:13:37.987358 12570 rpcz_store.cc:276] 0709 12:13:35.565116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.900685 (+2335569us) service_pool.cc:224] Handling call
0709 12:13:37.900992 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:37.944765 (+ 43773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:37.944923 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.949090 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.891738 17440 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:37.971135 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1942717659999289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.34900000000000003s for a response
W20260709 12:13:37.995997 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0389935250000235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
I20260709 12:13:37.854552 17317 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.974128 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:37.997431 12565 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:37.997936 12638 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:13:37.998512 17222 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:37.998703 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2142) took 1331 ms. Trace:
I20260709 12:13:37.999267 12638 rpcz_store.cc:276] 0709 12:13:36.667108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.947476 (+1280368us) service_pool.cc:224] Handling call
0709 12:13:37.998623 (+ 51147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.999274 17222 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.000926 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:37.978942 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119778us
I20260709 12:13:38.002786 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2461) took 2225 ms. Trace:
I20260709 12:13:38.003345 12565 rpcz_store.cc:276] 0709 12:13:35.777845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.987273 (+2209428us) service_pool.cc:224] Handling call
0709 12:13:37.997155 (+  9882us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.002363 (+  5208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.002705 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.828562 17547 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:37.930341 17465 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.975095 12493 rpcz_store.cc:276] 0709 12:13:36.592666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.947334 (+1354668us) service_pool.cc:224] Handling call
0709 12:13:37.972679 (+ 25345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.973802 17564 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:37.994925 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2132) took 1048 ms. Trace:
I20260709 12:13:37.988637 12566 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:38.019081 12491 rpcz_store.cc:276] 0709 12:13:36.946863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.940950 (+994087us) service_pool.cc:224] Handling call
0709 12:13:37.994816 (+ 53866us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:37.977978 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.004360 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.020244 12569 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:38.004797 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.021396 12565 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:38.011340 12496 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15.
I20260709 12:13:37.980826 17546 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:38.017343 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.021361 12569 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:38.014219 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2144) took 1115 ms. Trace:
I20260709 12:13:38.022099 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:38.027062 12638 rpcz_store.cc:276] 0709 12:13:36.898860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.000542 (+1101682us) service_pool.cc:224] Handling call
0709 12:13:38.014142 (+ 13600us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.027477 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:37.958825 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:38.026093 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:38.028678 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69977.5us
W20260709 12:13:38.029776 13218 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.549s (SENT)
I20260709 12:13:38.010797 17556 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:38.030297 17222 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:38.030490 17556 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:37.994184 17541 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:37.919480 17557 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.023401 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:37.956331 17516 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.024685 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:38.024959 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:38.025384 12645 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15. Re-sending same reply.
I20260709 12:13:37.946250 17445 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.026815 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2468) took 2030 ms. Trace:
I20260709 12:13:38.022884 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:38.048903 12569 rpcz_store.cc:276] 0709 12:13:35.996060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.020914 (+2024854us) service_pool.cc:224] Handling call
0709 12:13:38.021184 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.026531 (+  5347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.026755 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:38.035034 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.036365 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.051821 12638 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:38.052551 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2139) took 1404 ms. Trace:
I20260709 12:13:38.052855 12638 rpcz_store.cc:276] 0709 12:13:36.647931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.035962 (+1388031us) service_pool.cc:224] Handling call
0709 12:13:38.052484 (+ 16522us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.054216 17285 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:38.054708 17285 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:38.055073 17285 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:38.055579 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2635) took 1915 ms (client timeout 1750 ms). Trace:
W20260709 12:13:38.056018 12495 rpcz_store.cc:269] 0709 12:13:36.139776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.388687 (+248911us) service_pool.cc:224] Handling call
0709 12:13:38.055344 (+1666657us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:38.055503 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3000331648}
I20260709 12:13:38.057245 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.059597 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2122) took 1579 ms. Trace:
I20260709 12:13:38.060029 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 766) took 2109 ms. Trace:
I20260709 12:13:38.060736 12566 rpcz_store.cc:276] 0709 12:13:35.951024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.019627 (+2068603us) service_pool.cc:224] Handling call
0709 12:13:38.019998 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.059702 (+ 39704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.059963 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.026842 17546 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.022716 17538 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.033318 17556 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:38.049014 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:38.072050 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.058511 17445 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.059955 12495 rpcz_store.cc:276] 0709 12:13:36.480180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.056913 (+1576733us) service_pool.cc:224] Handling call
0709 12:13:38.059509 (+  2596us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.035578 17465 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:38.029446 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:38.062626 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.074425 12638 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:38.074925 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2136) took 1429 ms. Trace:
I20260709 12:13:38.075287 12638 rpcz_store.cc:276] 0709 12:13:36.646485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.053486 (+1407001us) service_pool.cc:224] Handling call
0709 12:13:38.074859 (+ 21373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.076216 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.076795 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:38.046667 17207 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
I20260709 12:13:38.078023 13259 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:13:38.044991 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2470) took 1997 ms. Trace:
I20260709 12:13:38.018867 17564 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.086759 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:38.018584 17482 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:13:38.064666 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.068094 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.091586 12569 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:38.026544 17340 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:38.035699 17005 raft_consensus.cc:2764] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260709 12:13:38.079068 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49707.6us
I20260709 12:13:38.077250 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2134) took 1432 ms. Trace:
I20260709 12:13:38.079208 12565 rpcz_store.cc:276] 0709 12:13:36.047702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.043772 (+1996070us) service_pool.cc:224] Handling call
0709 12:13:38.044356 (+   584us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.044607 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:38.044635 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.044683 (+    48us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:38.044721 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.044781 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.044925 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.079735 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.095938 12638 rpcz_store.cc:276] 0709 12:13:36.645000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.075925 (+1430925us) service_pool.cc:224] Handling call
0709 12:13:38.077197 (+  1272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.097084 12491 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:38.085246 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:38.065937 12571 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:38.087581 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:38.067356 12567 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:38.043313 17454 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:38.043509 17323 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.060627 17317 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.045936 17397 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:38.095062 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.099494 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 767) took 2032 ms. Trace:
I20260709 12:13:38.101938 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:38.112622 12567 rpcz_store.cc:276] 0709 12:13:36.067436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.061501 (+1994065us) service_pool.cc:224] Handling call
0709 12:13:38.061892 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.099248 (+ 37356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.099447 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.113482 13262 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:38.104066 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.104774 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:38.105505 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82160.8us
I20260709 12:13:38.114948 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:38.115999 13262 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:38.107312 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.088698 17314 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:38.088770 17222 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:38.116930 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:38.117055 17222 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.117482 17222 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:38.118407 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.119331 12638 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:38.120239 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.090853 17538 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:38.120800 12638 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:38.109051 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:38.124163 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.090080 17556 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:38.127694 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.128608 12638 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:38.113610 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.114948 12569 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:38.129181 12495 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:38.037240 17566 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
W20260709 12:13:38.129683 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.125099 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.129781 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.130713 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2092) took 2346 ms. Trace:
I20260709 12:13:38.130999 12569 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:38.131052 12491 rpcz_store.cc:276] 0709 12:13:35.784224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.107123 (+2322899us) service_pool.cc:224] Handling call
0709 12:13:38.130639 (+ 23516us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.088464 17564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:38.127113 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:38.133674 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:38.134308 13262 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:38.092306 17445 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:38.127247 12609 leader_election.cc:400] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:38.130154 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.135130 12609 leader_election.cc:403] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:38.135955 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.121160 17538 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.037696 17571 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.084s
I20260709 12:13:38.137759 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.132777 17317 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:38.143033 17557 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.145669 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:38.150909 17317 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:38.151762 17317 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.148339 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:38.107847 17340 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:38.153899 17340 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.151131 12491 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:13:38.159828 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2095) took 2333 ms. Trace:
I20260709 12:13:38.160300 12491 rpcz_store.cc:276] 0709 12:13:35.826178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.137452 (+2311274us) service_pool.cc:224] Handling call
0709 12:13:38.159726 (+ 22274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.162984 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:38.164390 12491 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14. Re-sending same reply.
I20260709 12:13:38.165495 17317 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.167999 12638 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:38.168932 17317 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.173661 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.176726 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2514) took 1069 ms. Trace:
I20260709 12:13:38.177423 12569 rpcz_store.cc:276] 0709 12:13:37.107267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.175532 (+1068265us) service_pool.cc:224] Handling call
0709 12:13:38.175997 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.176246 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:38.176285 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.176345 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:38.176407 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.176489 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.176641 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.177173 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2123) took 1696 ms. Trace:
I20260709 12:13:38.178660 12491 rpcz_store.cc:276] 0709 12:13:36.480456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.167001 (+1686545us) service_pool.cc:224] Handling call
0709 12:13:38.177079 (+ 10078us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.180943 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.182611 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.184540 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2637) took 2033 ms. Trace:
I20260709 12:13:38.152468 17397 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:38.185130 12495 rpcz_store.cc:276] 0709 12:13:36.151192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.180510 (+2029318us) service_pool.cc:224] Handling call
0709 12:13:38.184433 (+  3923us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.192728 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:38.189013 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.595s (SENT)
I20260709 12:13:38.193485 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.109544 17417 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 12:13:38.187708 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.137209 17445 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.149214 17538 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:38.197744 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.203151 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.204149 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:38.205555 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.182276 17323 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:13:38.207958 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.185338 17566 consensus_queue.cc:1243] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:13:38.209082 12638 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:38.200736 17490 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:38.174715 17314 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:38.213701 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:38.189181 17547 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.190285 17397 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:38.266261 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.217751 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2145) took 1311 ms. Trace:
I20260709 12:13:38.218506 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.206391 12495 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:38.266932 12638 rpcz_store.cc:276] 0709 12:13:36.906782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.148029 (+1241247us) service_pool.cc:224] Handling call
0709 12:13:38.217666 (+ 69637us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.268370 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.269243 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:38.270164 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.271896 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.220010 17419 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:38.196925 17557 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:38.196622 17556 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:38.272521 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:38.241772 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.200032 17445 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:38.265275 12491 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:13:38.196199 17573 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:38.225302 17490 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.221743 17497 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:38.272676 12495 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:38.308866 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 236890us
W20260709 12:13:38.282356 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63945.3us
I20260709 12:13:38.311122 17497 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.309819 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2474) took 1901 ms. Trace:
I20260709 12:13:38.279165 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.314003 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2473) took 1928 ms. Trace:
I20260709 12:13:38.314064 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:38.316331 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.274745 17397 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:38.312294 12567 rpcz_store.cc:276] 0709 12:13:36.408152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.114354 (+1706202us) service_pool.cc:224] Handling call
0709 12:13:38.116481 (+  2127us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.116714 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:38.207586 (+ 90872us) log.cc:844] Serialized 14 byte log entry
0709 12:13:38.309452 (+101866us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.309511 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:38.309543 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.309593 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.309759 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":82197,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":110,"thread_start_us":170366,"threads_started":2}
W20260709 12:13:38.328157 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49096.4us
W20260709 12:13:38.331507 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:38.339267 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.277522 17557 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.310060 17417 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:38.311743 17497 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:38.303779 17419 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:38.277525 17556 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.310364 17490 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:38.339656 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:38.331504 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:38.338052 17567 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
I20260709 12:13:38.353097 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.311255 17314 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.356312 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:38.369920 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 586) took 2176 ms. Trace:
I20260709 12:13:38.370386 12638 rpcz_store.cc:276] 0709 12:13:36.194121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.337781 (+2143660us) service_pool.cc:224] Handling call
0709 12:13:38.356070 (+ 18289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.369585 (+ 13515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.369847 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.356464 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2472) took 2194 ms. Trace:
I20260709 12:13:38.349584 17557 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:38.349727 17573 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:38.372937 12566 rpcz_store.cc:276] 0709 12:13:36.161951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.069221 (+1907270us) service_pool.cc:224] Handling call
0709 12:13:38.069562 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.069820 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:13:38.307601 (+237781us) log.cc:844] Serialized 14 byte log entry
0709 12:13:38.354645 (+ 47044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.354737 (+    92us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:13:38.354805 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.356234 (+  1429us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.356397 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2530,"prepare.queue_time_us":128887,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"thread_start_us":260217,"threads_started":2}
I20260709 12:13:38.361843 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:38.350428 17417 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:38.355242 17340 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:38.355481 17419 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:38.378259 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61917.8us
W20260709 12:13:38.383060 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.392795 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:38.361733 17556 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.395865 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.360594 17488 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:38.367378 17399 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:38.397683 12491 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:38.373294 16780 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.355937 17285 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.388170 17340 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.446641 17340 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:38.450052 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:38.452250 17579 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:13:38.454210 17574 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:13:38.454841 17530 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:38.455507 17530 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:38.358583 17576 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:38.389065 17454 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:38.406036 17585 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:13:38.456331 17530 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.478083 17470 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:38.381072 17417 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.481293 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2450) took 3397 ms. Trace:
I20260709 12:13:38.481613 17486 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:38.481887 12564 rpcz_store.cc:276] 0709 12:13:35.083542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.435070 (+351528us) service_pool.cc:224] Handling call
0709 12:13:35.459187 (+ 24117us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:37.832586 (+2373399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.480913 (+648327us) spinlock_profiling.cc:243] Waited 2.29 s on lock 0x7b54003ab954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:38.481194 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5277752064}
I20260709 12:13:38.483238 17427 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.472774 12638 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260709 12:13:38.465597 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125998us
I20260709 12:13:38.471223 12491 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:38.489555 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.490154 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:38.490630 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.491071 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2143) took 1611 ms. Trace:
I20260709 12:13:38.491127 12491 raft_consensus.cc:2376] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13. Re-sending same reply.
I20260709 12:13:38.491652 12638 rpcz_store.cc:276] 0709 12:13:36.879802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.396567 (+1516765us) service_pool.cc:224] Handling call
0709 12:13:38.464551 (+ 67984us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.486473 (+ 21922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.490991 (+  4518us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:38.516024 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.539804 17591 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:38.540827 17591 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:38.554059 12493 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:38.558928 12638 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 8. Current term is 9. Ops: []
I20260709 12:13:38.565760 17419 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.501153 17427 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:38.567476 17427 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:38.570319 17586 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260709 12:13:38.572049 17589 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:13:38.603585 13260 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260709 12:13:38.604024 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.604645 17419 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:38.610477 12492 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:38.617080 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 694) took 2585 ms. Trace:
I20260709 12:13:38.618131 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2347) took 1032 ms. Trace:
I20260709 12:13:38.619068 12638 rpcz_store.cc:276] 0709 12:13:37.585275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.559897 (+974622us) service_pool.cc:224] Handling call
0709 12:13:38.617357 (+ 57460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.617645 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:38.617685 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.617748 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:38.617794 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.617878 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.618044 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.618043 13259 rpcz_store.cc:276] 0709 12:13:36.027860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.028283 (+   423us) service_pool.cc:224] Handling call
0709 12:13:36.043795 (+ 15512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.078495 (+2034700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.612401 (+533906us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:38.612701 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4677285248}
I20260709 12:13:38.604235 17474 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:38.628250 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.634140 17227 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:38.641182 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.022s	sys 0.019s
W20260709 12:13:38.641714 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.022s	sys 0.019s
W20260709 12:13:38.642174 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 533121us
I20260709 12:13:38.643590 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:38.646499 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 196558us
I20260709 12:13:38.647809 12638 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:38.647935 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.649961 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:38.637015 17546 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.549s	user 0.015s	sys 0.001s
I20260709 12:13:38.607236 17490 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.581528 17445 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.634806 17427 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:38.658748 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.659570 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:38.659896 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 588) took 1074 ms. Trace:
I20260709 12:13:38.660434 12638 rpcz_store.cc:276] 0709 12:13:37.585851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.627702 (+1041851us) service_pool.cc:224] Handling call
0709 12:13:38.647556 (+ 19854us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.651483 (+  3927us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.659817 (+  8334us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:38.635289 17408 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.382s	user 0.034s	sys 0.056s
W20260709 12:13:38.703889 17408 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.500s	user 0.035s	sys 0.056s
I20260709 12:13:38.704236 17408 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:38.705343 17408 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.653769 12643 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:38.656649 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:38.707430 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2342) took 2515 ms. Trace:
W20260709 12:13:38.707671 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54738.1us
I20260709 12:13:38.707909 12643 rpcz_store.cc:276] 0709 12:13:36.192161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.420208 (+228047us) service_pool.cc:224] Handling call
0709 12:13:36.458435 (+ 38227us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.653548 (+2195113us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b5400238820. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:13:38.707088 (+ 53540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.707356 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7961472}
I20260709 12:13:38.709172 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:38.711047 12536 leader_election.cc:400] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:38.711495 12536 leader_election.cc:403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:13:38.667507 17445 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.650632 17592 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:38.674304 17454 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:38.650463 17285 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:38.676695 17323 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:38.637893 17474 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.645910 17227 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:38.682499 17564 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.537s	user 0.011s	sys 0.000s
I20260709 12:13:38.684250 17557 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:38.728395 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.026s	sys 0.019s
I20260709 12:13:38.661192 17482 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:38.661029 17546 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:38.687044 17222 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.569s	user 0.006s	sys 0.015s
W20260709 12:13:38.657229 17563 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.009s
W20260709 12:13:38.688917 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2102) took 2828 ms (client timeout 2065 ms). Trace:
W20260709 12:13:38.743347 12494 rpcz_store.cc:269] 0709 12:13:35.860522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.542123 (+681601us) service_pool.cc:224] Handling call
0709 12:13:38.688615 (+2146492us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:38.688816 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":824329472}
I20260709 12:13:38.690814 17388 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.691818 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2339) took 2500 ms. Trace:
I20260709 12:13:38.692651 17497 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.695295 17490 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.696943 17226 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:38.702015 17417 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.703450 12638 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:38.756430 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2349) took 1145 ms. Trace:
I20260709 12:13:38.756857 12638 rpcz_store.cc:276] 0709 12:13:37.610664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.661650 (+1050986us) service_pool.cc:224] Handling call
0709 12:13:38.703225 (+ 41575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.756085 (+ 52860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.756360 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.708014 13257 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:13:38.716672 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:38.724056 12564 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:38.719278 17591 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:13:38.729604 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.027s	sys 0.019s
I20260709 12:13:38.684131 17340 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:38.730436 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:38.734519 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.720227 17470 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:38.721575 17285 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:38.715008 17576 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:38.740525 12491 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:13:38.719784 17592 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.748292 12646 rpcz_store.cc:276] 0709 12:13:36.191034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.192560 (+  1526us) service_pool.cc:224] Handling call
0709 12:13:36.192953 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:36.927910 (+734957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.691479 (+1763569us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:38.691709 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1618537216}
I20260709 12:13:38.766310 17570 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:38.759946 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.772292 12569 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:38.746620 17465 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.669s	user 0.005s	sys 0.013s
I20260709 12:13:38.735265 17454 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:38.780925 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64348.4us
I20260709 12:13:38.809854 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:38.781111 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.782295 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 765) took 2837 ms. Trace:
I20260709 12:13:38.811472 12570 rpcz_store.cc:276] 0709 12:13:35.944685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.003677 (+2058992us) service_pool.cc:224] Handling call
0709 12:13:38.003994 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.004205 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:38.087161 (+ 82956us) log.cc:844] Serialized 14 byte log entry
0709 12:13:38.781734 (+694573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.781833 (+    99us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:38.781901 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.782011 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.782209 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12962,"prepare.queue_time_us":194341,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":25002,"thread_start_us":763371,"threads_started":2}
I20260709 12:13:38.813740 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:38.814496 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:38.784150 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:38.772825 17591 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.785122 17470 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.785614 17497 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:38.788787 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:38.821444 12568 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:38.731525 17397 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.745285 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.734190 17564 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.747228 17222 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.808744 17595 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:38.819149 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:38.823411 12494 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:38.835836 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.836328 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.741349 17556 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.838883 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:38.837699 12494 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:38.802994 17592 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:38.842640 12564 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:38.830297 17576 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:38.816619 17482 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:38.821133 17591 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:38.870071 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:38.871271 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.852578 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:38.865403 12565 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:38.840663 17314 raft_consensus.cc:2764] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
W20260709 12:13:38.871847 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.842640 17497 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:38.875622 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:38.820010 17570 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:38.888248 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:38.820701 17454 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:38.901908 17534 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.570s	user 0.020s	sys 0.006s
I20260709 12:13:38.867604 17482 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:38.906193 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2334) took 2907 ms. Trace:
I20260709 12:13:38.875190 17387 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:38.904487 17538 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.690s	user 0.013s	sys 0.004s
W20260709 12:13:38.887792 17485 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.142s	user 0.002s	sys 0.012s
W20260709 12:13:38.902802 17477 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.154s	user 0.022s	sys 0.034s
I20260709 12:13:38.905334 17483 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:38.904170 17314 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:38.927698 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:38.908632 12640 rpcz_store.cc:276] 0709 12:13:35.999176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.999412 (+   236us) service_pool.cc:224] Handling call
0709 12:13:36.000310 (+   898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:36.132269 (+131959us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:36.165539 (+ 33270us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.345571 (+180032us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:36.345678 (+   107us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:36.345745 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.905859 (+2560114us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:38.905923 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.906110 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":38897,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":66630,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":301872512,"thread_start_us":177182,"threads_started":1,"wal-append.queue_time_us":308655,"wal-append.run_cpu_time_us":1242,"wal-append.run_wall_time_us":1023167}
W20260709 12:13:38.930606 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 200274us
W20260709 12:13:38.932008 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.933097 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:38.910141 17482 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:38.934019 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2105) took 3072 ms. Trace:
I20260709 12:13:38.937480 12640 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
W20260709 12:13:38.925783 17485 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.180s	user 0.002s	sys 0.012s
I20260709 12:13:38.939426 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2351) took 1290 ms. Trace:
I20260709 12:13:38.940174 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 593) took 1287 ms. Trace:
I20260709 12:13:38.944643 12640 rpcz_store.cc:276] 0709 12:13:37.650991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.931950 (+1280959us) service_pool.cc:224] Handling call
0709 12:13:38.932297 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.937979 (+  5682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.938193 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.936618 12494 rpcz_store.cc:276] 0709 12:13:35.861436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.926704 (+3065268us) service_pool.cc:224] Handling call
0709 12:13:38.933951 (+  7247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:38.888398 17497 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:38.942373 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2132) took 2294 ms (client timeout 1600 ms). Trace:
W20260709 12:13:38.950839 12642 rpcz_store.cc:269] 0709 12:13:36.643820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.413805 (+769985us) service_pool.cc:224] Handling call
0709 12:13:38.937924 (+1524119us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:38.938043 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":634059008}
I20260709 12:13:38.952584 12642 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:38.956888 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:38.926445 17477 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.178s	user 0.023s	sys 0.034s
I20260709 12:13:38.944517 12638 rpcz_store.cc:276] 0709 12:13:37.648936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.757903 (+1108967us) service_pool.cc:224] Handling call
0709 12:13:38.789835 (+ 31932us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.790085 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:38.790352 (+   267us) log.cc:844] Serialized 14 byte log entry
0709 12:13:38.938813 (+148461us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.938887 (+    74us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:38.938963 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.939190 (+   227us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.939345 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6069,"prepare.queue_time_us":1333,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":3539,"thread_start_us":144508,"threads_started":1}
W20260709 12:13:39.011870 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.946641 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.015204 12494 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:39.016250 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.017086 12570 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:13:39.017727 12644 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:38.949183 17559 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.190s	user 0.003s	sys 0.000s
W20260709 12:13:38.969024 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.042482 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.043680 12566 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:39.047878 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 222176us
W20260709 12:13:39.048853 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:38.937893 17557 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:39.048522 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:38.942785 17556 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:39.053004 12644 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:13:39.059559 12644 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:38.928789 17538 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.056573 17340 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.025017 17587 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:38.942741 17546 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.059864 12644 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:13:39.128299 12644 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:39.129380 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2344) took 2936 ms. Trace:
W20260709 12:13:39.055325 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2330) took 3256 ms (client timeout 1668 ms). Trace:
I20260709 12:13:39.063793 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:38.950578 17465 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.089916 17477 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:39.090158 17594 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:13:39.110539 17591 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.107975 17595 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:39.124720 17454 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:39.126372 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 282655us
W20260709 12:13:39.126696 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.054385 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:39.039572 17417 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.133481 13259 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.130040 12644 rpcz_store.cc:276] 0709 12:13:36.192836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.431975 (+239139us) service_pool.cc:224] Handling call
0709 12:13:36.458592 (+ 26617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:39.017237 (+2558645us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b58008a98d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:13:39.128870 (+111633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.129192 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":212187136}
W20260709 12:13:39.131708 12639 rpcz_store.cc:269] 0709 12:13:35.787970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.969003 (+181033us) service_pool.cc:224] Handling call
0709 12:13:39.044022 (+3075019us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.044270 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":382934784}
I20260709 12:13:39.143923 12494 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:39.166326 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2140) took 1788 ms. Trace:
I20260709 12:13:39.166946 12494 rpcz_store.cc:276] 0709 12:13:37.377984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.946423 (+1568439us) service_pool.cc:224] Handling call
0709 12:13:39.166255 (+219832us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:39.168560 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.162079 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:39.169893 12494 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:39.143329 17597 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.363s	user 0.000s	sys 0.002s
W20260709 12:13:39.179329 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5169404829998712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
I20260709 12:13:39.134429 17317 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.198148 17317 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.199625 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 770) took 1602 ms. Trace:
W20260709 12:13:39.200052 17599 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.295s	user 0.002s	sys 0.000s
I20260709 12:13:39.200279 12569 rpcz_store.cc:276] 0709 12:13:37.597384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.178786 (+581402us) service_pool.cc:224] Handling call
0709 12:13:38.179211 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.803802 (+624591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.199176 (+395374us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54001a27d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:39.199520 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1363397376}
I20260709 12:13:39.202625 17538 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:39.205747 17559 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:13:39.206246 17606 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.002s	sys 0.000s
I20260709 12:13:39.156841 17465 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:39.184576 12536 connection.cc:256] RPC call timeout handler was delayed by -1.510640482999861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
W20260709 12:13:39.210707 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4440404829999807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
W20260709 12:13:39.211244 12536 connection.cc:256] RPC call timeout handler was delayed by -1.417040482999937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260709 12:13:39.211748 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4134404829999312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18810000000000002s for a response
I20260709 12:13:39.212616 12568 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:13:39.214480 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4080404829999225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20260709 12:13:39.214951 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4080404829999225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20260709 12:13:39.185514 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.163290 17397 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:39.163743 17222 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:39.188812 17595 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:13:39.190209 17419 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.585s	user 0.012s	sys 0.000s
I20260709 12:13:39.072876 17600 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.122s
I20260709 12:13:39.155982 17470 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.190851 17591 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:39.191800 12462 service_pool.cc:136] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34492 dropped due to backpressure: service queue is full with 50 items [suppressed 5 similar messages]
I20260709 12:13:39.192466 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2350) took 1576 ms. Trace:
I20260709 12:13:39.226212 12643 rpcz_store.cc:276] 0709 12:13:37.616160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.709367 (+1093207us) service_pool.cc:224] Handling call
0709 12:13:38.724588 (+ 15221us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:38.733851 (+  9263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:38.733895 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:38.753403 (+ 19508us) log.cc:844] Serialized 26 byte log entry
0709 12:13:38.943383 (+189980us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:38.943452 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.943538 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:39.192184 (+248646us) raft_consensus.cc:1685] finished
0709 12:13:39.192209 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:39.192400 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5150,"raft.queue_time_us":9639,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":4676,"thread_start_us":132481,"threads_started":1,"wal-append.queue_time_us":368133}
I20260709 12:13:39.134429 17587 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:39.208592 17564 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.197022 17570 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:39.239770 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211884us
W20260709 12:13:39.218914 12536 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.591s (SENT)
I20260709 12:13:39.219659 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:39.220724 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2108) took 3222 ms (client timeout 1684 ms). Trace:
I20260709 12:13:39.241367 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:13:39.241770 12497 rpcz_store.cc:269] 0709 12:13:35.998714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.125021 (+126307us) service_pool.cc:224] Handling call
0709 12:13:39.220524 (+3095503us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.220650 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333641088}
I20260709 12:13:39.245064 12644 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 12:13:39.224836 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:39.195083 17497 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.220882 17595 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:39.216640 17538 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.235827 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.253875 12639 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:39.236467 12572 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:39.142973 17607 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.093s
I20260709 12:13:39.159060 17477 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.215713 17593 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:39.235446 17457 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.772s	user 0.007s	sys 0.018s
I20260709 12:13:39.242601 17571 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.248260 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.227634 17317 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.237334 17587 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.266176 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.251250 12643 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:13:39.281716 17604 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.191s	user 0.003s	sys 0.000s
I20260709 12:13:39.283033 12644 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:39.288378 17477 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:39.273064 17457 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.810s	user 0.007s	sys 0.018s
I20260709 12:13:39.292222 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 592) took 1641 ms. Trace:
I20260709 12:13:39.295702 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:39.259382 17465 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:39.294893 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2337) took 3121 ms (client timeout 1985 ms). Trace:
W20260709 12:13:39.295329 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 241041us
I20260709 12:13:39.250653 17591 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:39.300093 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.321244 12638 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:39.309897 12646 rpcz_store.cc:276] 0709 12:13:37.650653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.796101 (+1145448us) service_pool.cc:224] Handling call
0709 12:13:38.796673 (+   572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.797023 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:13:38.824198 (+ 27175us) log.cc:844] Serialized 14 byte log entry
0709 12:13:39.155131 (+330933us) log.cc:844] Serialized 14 byte log entry
0709 12:13:39.291668 (+136537us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b18003659e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:39.291848 (+   180us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:39.291904 (+    56us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:39.291953 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.292025 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:39.292152 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":140923,"prepare.queue_time_us":35217,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":2401,"raft.queue_time_us":166736,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":15406,"spinlock_wait_cycles":15838976,"thread_start_us":302682,"threads_started":1,"wal-append.queue_time_us":412674}
I20260709 12:13:39.255110 17595 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:39.299651 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:39.333209 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.317279 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.296217 17600 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:39.315235 17556 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:13:39.315407 12637 rpcz_store.cc:269] 0709 12:13:36.174005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.181184 (+  7179us) service_pool.cc:224] Handling call
0709 12:13:39.294643 (+3113459us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.294814 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3276537728}
I20260709 12:13:39.338053 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:39.294206 17587 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.338624 17556 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:39.339637 17581 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.889s	user 0.003s	sys 0.000s
I20260709 12:13:39.341140 17587 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.340698 17350 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.342026 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.342705 12637 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.342576 17350 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.344777 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.345561 12644 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:38.314991 12571 rpcz_store.cc:276] 0709 12:13:36.386004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.108714 (+1722710us) service_pool.cc:224] Handling call
0709 12:13:38.134596 (+ 25882us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.134827 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:13:38.198958 (+ 64131us) log.cc:844] Serialized 14 byte log entry
0709 12:13:38.313329 (+114371us) log.cc:844] Serialized 14 byte log entry
0709 12:13:38.313513 (+   184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:38.313569 (+    56us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:38.313657 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.313756 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:38.313927 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12005,"prepare.queue_time_us":79954,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":29,"raft.queue_time_us":123,"thread_start_us":128731,"threads_started":2}
I20260709 12:13:39.346685 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.347179 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:39.347807 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2134) took 2241 ms (client timeout 1887 ms). Trace:
I20260709 12:13:39.347970 12643 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:39.350057 17543 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.350267 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.351246 17323 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:39.351797 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.352663 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:39.295552 17427 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:13:39.355497 17603 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:13:39.294260 17608 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:13:39.357769 17602 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.267s	user 0.001s	sys 0.000s
I20260709 12:13:39.357900 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:39.359066 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:39.360699 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.338140 17600 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:39.362401 17323 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:39.362928 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2641) took 3192 ms (client timeout 1922 ms). Trace:
I20260709 12:13:39.363059 17600 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.340086 17556 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:39.363466 12490 rpcz_store.cc:269] 0709 12:13:36.170238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.879646 (+709408us) service_pool.cc:224] Handling call
0709 12:13:39.362670 (+2483024us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.362824 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2400391552}
I20260709 12:13:39.364413 17600 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:39.365087 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.365238 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.366088 17477 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:39.328859 17611 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:13:39.366506 17477 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:39.353232 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.331358 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 603) took 1373 ms. Trace:
W20260709 12:13:39.334545 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 626) took 3599 ms (client timeout 1872 ms). Trace:
I20260709 12:13:39.264312 17538 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.368046 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 762) took 4179 ms. Trace:
I20260709 12:13:39.368357 12497 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:39.368682 12563 rpcz_store.cc:276] 0709 12:13:35.188704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.414836 (+1226132us) service_pool.cc:224] Handling call
0709 12:13:36.415097 (+   261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:37.787820 (+1372723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.367774 (+1579954us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001b2bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:39.367988 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2836333952}
I20260709 12:13:39.369737 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.370160 17538 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.370520 12497 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:39.371419 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.367231 13255 rpcz_store.cc:276] 0709 12:13:37.958188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.958376 (+   188us) service_pool.cc:224] Handling call
0709 12:13:37.965927 (+  7551us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:38.017149 (+ 51222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:38.017183 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:38.028790 (+ 11607us) log.cc:844] Serialized 52 byte log entry
0709 12:13:38.108773 (+ 79983us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:39.320124 (+1211351us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b1800285040. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:13:39.320324 (+   200us) log.cc:844] Serialized 14 byte log entry
0709 12:13:39.330882 (+ 10558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.330970 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:39.331013 (+    43us) raft_consensus.cc:1685] finished
0709 12:13:39.331110 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:39.331284 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12405,"prepare.queue_time_us":89749,"raft.queue_time_us":58549,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":3298,"spinlock_wait_cycles":24095360,"thread_start_us":164394,"threads_started":3,"wal-append.queue_time_us":158625}
I20260709 12:13:39.372040 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2459) took 3689 ms. Trace:
I20260709 12:13:39.366918 17477 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:39.372586 12572 rpcz_store.cc:276] 0709 12:13:35.683073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.951221 (+2268148us) service_pool.cc:224] Handling call
0709 12:13:37.980455 (+ 29234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:39.254863 (+1274408us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.370818 (+115955us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b5400201aa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6a02 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:39.371798 (+   980us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:39.371985 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2992218624}
I20260709 12:13:39.369750 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 594) took 1718 ms. Trace:
I20260709 12:13:39.374970 17323 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:39.330137 17465 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:13:39.375941 17465 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:39.376339 17465 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:39.377328 17465 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.379369 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2323) took 3708 ms. Trace:
W20260709 12:13:39.380573 17582 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.930s	user 0.005s	sys 0.001s
I20260709 12:13:39.379843 12641 rpcz_store.cc:276] 0709 12:13:35.671164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.671291 (+   127us) service_pool.cc:224] Handling call
0709 12:13:35.671657 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:35.958241 (+286584us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:35.958497 (+   256us) log.cc:844] Serialized 14 byte log entry
0709 12:13:36.104953 (+146456us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:36.105032 (+    79us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:36.105096 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:38.893294 (+2788198us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b1800427480. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419
0709 12:13:39.379118 (+485824us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:39.379171 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:39.379310 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":115,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":110,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":14289,"spinlock_wait_cycles":665350912,"thread_start_us":145961,"threads_started":1,"wal-append.queue_time_us":355762,"wal-append.run_cpu_time_us":2002,"wal-append.run_wall_time_us":1190531}
I20260709 12:13:39.289531 17314 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:39.329360 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:39.378712 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2138) took 2731 ms (client timeout 1887 ms). Trace:
W20260709 12:13:39.348249 12493 rpcz_store.cc:269] 0709 12:13:37.127511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.014480 (+886969us) service_pool.cc:224] Handling call
0709 12:13:39.347609 (+1333129us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540020e4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.347732 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":987004160}
I20260709 12:13:39.374091 12640 rpcz_store.cc:276] 0709 12:13:37.651274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.946096 (+1294822us) service_pool.cc:224] Handling call
0709 12:13:38.946446 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:38.946801 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:38.946830 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:38.947615 (+   785us) log.cc:844] Serialized 26 byte log entry
0709 12:13:39.223419 (+275804us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:39.223474 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:39.223544 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:39.369468 (+145924us) raft_consensus.cc:1685] finished
0709 12:13:39.369492 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:39.369678 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":71233,"thread_start_us":111418,"threads_started":1,"wal-append.queue_time_us":222932}
W20260709 12:13:39.382368 12645 rpcz_store.cc:269] 0709 12:13:36.647575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.876545 (+1228970us) service_pool.cc:224] Handling call
0709 12:13:39.378464 (+1501919us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.378616 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337118592}
I20260709 12:13:39.383352 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.383976 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 630) took 1495 ms. Trace:
I20260709 12:13:39.383471 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.384887 12645 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:39.384918 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.387782 13254 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:39.388121 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.389041 12646 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:39.390313 13254 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:39.391940 13255 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:13:39.393203 12490 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:39.347323 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:39.388950 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.394348 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.394536 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.395287 12494 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:39.394655 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.396221 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.396953 12637 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:39.396421 12641 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 12:13:39.398326 13260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 765) took 2485 ms (client timeout 2889 ms). Trace:
I20260709 12:13:39.398962 12641 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:13:39.399652 13253 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:39.384410 12491 rpcz_store.cc:276] 0709 12:13:37.888937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.492233 (+603296us) service_pool.cc:224] Handling call
0709 12:13:39.383750 (+891517us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.383897 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253007232}
W20260709 12:13:39.367369 12489 rpcz_store.cc:269] 0709 12:13:35.734671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.999967 (+265296us) service_pool.cc:224] Handling call
0709 12:13:39.334294 (+3334327us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.334452 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3554143488}
I20260709 12:13:39.400861 12639 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:39.294438 17440 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.402107 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:13:39.395522 12644 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:39.225800 12462 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34492 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2651)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2654)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2658)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2662)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2660)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2661)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2668)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 638)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2151)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2150)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2646)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2664)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 22)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2103)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2110)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2639)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2111)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2642)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2643)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2125)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 631)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2145)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2659)
I20260709 12:13:39.390079 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:39.414382 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55398.5us
W20260709 12:13:39.398792 13260 rpcz_store.cc:269] 0709 12:13:36.913173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:36.913519 (+   346us) service_pool.cc:224] Handling call
0709 12:13:39.397974 (+2484455us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.398263 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3742477696}
W20260709 12:13:39.421216 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 174927us
I20260709 12:13:39.409106 12645 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:39.423288 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:39.317315 17285 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.531s	user 0.011s	sys 0.002s
I20260709 12:13:39.420218 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.433030 12639 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:39.432425 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.655s (SENT) [suppressed 49 similar messages]
I20260709 12:13:39.378696 17465 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:39.435439 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:39.438649 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.437016 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:39.481456 13254 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
W20260709 12:13:39.467303 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2586404829999083s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
I20260709 12:13:39.467434 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:39.473856 17596 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:39.479640 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.491190 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.457273 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:39.482563 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2109404829998311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
I20260709 12:13:39.488161 12494 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9.
I20260709 12:13:39.491417 12489 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:13:39.485078 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:39.492707 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0426404830000138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22930000000000003s for a response
W20260709 12:13:39.493846 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73744.4us
W20260709 12:13:39.493969 12611 connection.cc:256] RPC call timeout handler was delayed by -1.4495652370001153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20260709 12:13:39.494515 12611 connection.cc:256] RPC call timeout handler was delayed by -1.1948652370001582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22050000000000003s for a response
I20260709 12:13:39.496105 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.496995 12646 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:13:39.491918 12569 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.501303 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.502601 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:39.492118 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:39.504731 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.495294 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.507050 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.518208 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.383352 17516 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.522074 12494 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:39.522055 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.766s (SENT)
I20260709 12:13:39.429862 17323 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.427965 17427 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:39.562674 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.566286 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.472718 17477 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:39.561137 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2121) took 3081 ms (client timeout 2402 ms). Trace:
W20260709 12:13:39.572746 12496 rpcz_store.cc:269] 0709 12:13:36.479906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.944100 (+1464194us) service_pool.cc:224] Handling call
0709 12:13:39.512586 (+1568486us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.561011 (+ 48425us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93143296}
I20260709 12:13:39.564747 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.567147 12491 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:39.561687 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.571164 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:39.384009 17226 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.143s
I20260709 12:13:39.527056 17388 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:39.486445 17285 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:39.322391 17571 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:39.572073 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50086.6us
I20260709 12:13:39.480504 17474 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.605506 13259 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 12:13:39.590286 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.526309 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:39.615650 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:39.529404 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:39.433099 17557 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:39.536815 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.501835 17533 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:39.526935 17387 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:39.629884 12569 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:39.569945 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.615669 12494 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:39.630055 17533 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.631173 12489 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:39.631630 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.632465 12569 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:39.630252 17387 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:39.633901 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:39.634892 17497 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:39.635406 17497 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:39.636075 17474 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.637017 17474 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.639685 12494 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:39.587702 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:39.616613 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.847s (SENT)
I20260709 12:13:39.616652 12566 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:13:39.618937 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:39.642181 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.887s (SENT)
I20260709 12:13:39.623612 12563 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.564580 17005 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:13:39.560560 17596 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:39.642373 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.624780 17137 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:39.642943 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.887s (SENT)
I20260709 12:13:39.564791 17476 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:13:39.648037 17476 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:39.648464 17476 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:39.627009 17388 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:39.628455 17592 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.763s	user 0.006s	sys 0.005s
I20260709 12:13:39.650290 17592 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.650703 17388 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.651816 12642 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 16. Ops: []
I20260709 12:13:39.652976 17592 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.527649 17564 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:39.654210 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.654088 17564 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.654834 17427 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.649307 17476 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.656762 17005 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:39.657014 17476 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:39.657294 17005 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:39.658031 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:39.658926 13254 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:39.658277 17005 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.659801 17005 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.660137 17533 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:39.628996 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.569872 17397 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
W20260709 12:13:39.625157 17482 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.682s	user 0.012s	sys 0.000s
I20260709 12:13:39.675199 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:39.644867 17226 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:39.646224 17596 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:39.661608 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.663309 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:39.685819 12570 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:39.637084 12572 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:39.669731 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:39.687150 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:39.687513 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.671370 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:39.672964 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:39.595844 17417 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:39.683028 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.402s (SENT)
W20260709 12:13:39.693665 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.735450 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:39.736045 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:39.736853 12460 connection.cc:256] RPC call timeout handler was delayed by -2.651139128000068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
I20260709 12:13:39.695998 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.738385 12493 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.682018 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:39.511786 17456 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:13:39.684072 17568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.045s	sys 0.104s
I20260709 12:13:39.747342 12567 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:39.700364 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.583461 17619 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:39.745820 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:39.746592 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:39.763118 13255 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:39.757097 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:39.759138 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76116.9us
I20260709 12:13:39.764745 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:39.772574 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:39.683542 17226 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:39.775163 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:39.692260 17285 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:39.691345 17557 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:39.778642 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.948s (SENT)
W20260709 12:13:39.760730 17568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.045s	sys 0.105s
I20260709 12:13:39.735219 17417 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:39.751076 17456 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:39.569108 12463 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34482 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2668)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 641)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 638)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2151)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 642)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 643)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2150)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 639)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 640)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2646)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2153)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2664)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2157)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 22)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 625)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2090)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2103)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2110)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2639)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2111)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2642)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2643)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2125)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 631)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2145)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 635)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 636)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 637)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2645)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2650)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2659)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2158)
I20260709 12:13:39.774397 17576 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:39.779191 17592 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:39.636283 17497 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:39.793411 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:39.685446 17482 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:39.634907 17387 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:39.792855 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2106) took 3931 ms (client timeout 3120 ms). Trace:
W20260709 12:13:39.817399 12492 rpcz_store.cc:269] 0709 12:13:35.875285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:38.002222 (+2126937us) service_pool.cc:224] Handling call
0709 12:13:39.792653 (+1790431us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b540020bcd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:39.792783 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1317145856}
I20260709 12:13:39.818892 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.790254 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:39.808473 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 334488us
I20260709 12:13:39.809347 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.812160 12571 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:39.814193 12460 connection.cc:256] RPC call timeout handler was delayed by -2.3119272860001274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2452s for a response
I20260709 12:13:39.815481 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.792848 17350 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:39.840119 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 272607us
W20260709 12:13:39.841143 12610 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.649s (SENT)
I20260709 12:13:39.800344 17417 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:39.802613 17456 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:39.823644 12643 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:39.814754 12493 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:39.829433 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.865221 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2662) took 1024 ms. Trace:
I20260709 12:13:39.865345 12494 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:39.865900 12493 rpcz_store.cc:276] 0709 12:13:38.840474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.382848 (+542374us) service_pool.cc:224] Handling call
0709 12:13:39.695778 (+312930us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:13:39.865106 (+169328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36398976}
I20260709 12:13:39.837697 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:39.838338 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.847672 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.860028 12570 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:13:39.867614 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.785899 17614 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:13:39.878271 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:39.898723 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.900399 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:39.901257 12641 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:39.906107 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.823592 17592 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:39.914078 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.924068 12493 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:39.915553 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:39.926483 12644 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:39.919124 12646 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:39.927752 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.932106 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.916889 12571 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:39.934847 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:39.935081 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:39.935936 12495 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15.
I20260709 12:13:39.938721 12489 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:39.940191 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.936832 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.945896 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.946658 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:39.947175 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2663) took 1103 ms. Trace:
I20260709 12:13:39.947542 12492 rpcz_store.cc:276] 0709 12:13:38.843928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.899908 (+1055980us) service_pool.cc:224] Handling call
0709 12:13:39.947110 (+ 47202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:39.889411 17600 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:39.886924 17473 raft_consensus.cc:2764] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:13:39.983341 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.961550 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:39.966427 17624 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
W20260709 12:13:39.970592 17623 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:13:39.973100 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:39.974104 17477 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:39.894888 17397 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:39.977869 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.000528 12646 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:39.941179 17587 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:39.988595 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:39.988868 12536 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.133s (SENT)
I20260709 12:13:39.988274 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.004372 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.006525 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.006639 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:40.009339 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2669) took 1095 ms. Trace:
W20260709 12:13:40.008967 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.219s (SENT)
I20260709 12:13:40.006377 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:40.010447 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.010247 12492 rpcz_store.cc:276] 0709 12:13:38.914394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.987944 (+1073550us) service_pool.cc:224] Handling call
0709 12:13:40.009250 (+ 21306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.011368 17619 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.012104 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.013124 12492 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:39.840651 17226 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:40.020311 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.021268 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.022061 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:40.022850 12495 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:40.023658 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 634) took 1925 ms. Trace:
I20260709 12:13:40.023995 12495 rpcz_store.cc:276] 0709 12:13:38.097933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.003982 (+1906049us) service_pool.cc:224] Handling call
0709 12:13:40.023582 (+ 19600us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.025204 12494 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:39.956624 17555 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.028115 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:40.041966 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.018064 17591 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.029686 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.053843 12493 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 9. Re-sending same reply.
I20260709 12:13:40.031965 17474 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:40.017144 17543 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.666s	user 0.005s	sys 0.005s
W20260709 12:13:40.034516 17625 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:13:40.037329 17619 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:40.029687 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:40.055800 17619 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.056362 12563 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:40.058678 17619 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.029917 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:39.986348 17457 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:13:40.041334 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.061364 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:40.062480 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.062633 12570 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:40.042191 12564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:40.042773 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:40.066141 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.066928 12492 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:40.068033 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.025164 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.021215 13255 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:40.049886 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:39.987100 17635 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:40.082438 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 240489us
I20260709 12:13:40.083549 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:40.078363 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:13:40.085570 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.086864 12564 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:13:40.087934 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.088903 12564 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 9: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:40.091001 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2647) took 1285 ms. Trace:
I20260709 12:13:40.091742 12492 rpcz_store.cc:276] 0709 12:13:38.805542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.067761 (+1262219us) service_pool.cc:224] Handling call
0709 12:13:40.090894 (+ 23133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.061033 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.093067 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2646) took 1287 ms. Trace:
I20260709 12:13:40.080356 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.093129 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.094434 12493 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:13:40.080427 12495 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:40.071686 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.008802 17408 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:40.098803 13259 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:13:40.093832 12494 rpcz_store.cc:276] 0709 12:13:38.805248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.059484 (+1254236us) service_pool.cc:224] Handling call
0709 12:13:40.092965 (+ 33481us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:39.902381 17557 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.095094 12492 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:40.098687 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.104724 12495 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:40.096369 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57692.2us
I20260709 12:13:40.105932 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:13:40.108335 17633 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:13:40.064656 17474 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:13:40.112854 17457 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:40.113711 17592 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.105861 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:40.081836 12460 connection.cc:256] RPC call timeout handler was delayed by -2.2252251990000786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260709 12:13:40.068325 17507 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:40.131441 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.132320 12494 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:40.080482 17226 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:40.133829 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.009088 17397 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.145743 17507 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.146152 17639 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:40.146744 17323 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.564s	user 0.011s	sys 0.000s
I20260709 12:13:40.146906 17639 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.147612 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.147706 17397 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.148301 12570 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:40.148571 17397 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:40.149089 17397 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:40.149609 17323 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:40.150269 17595 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.818s	user 0.009s	sys 0.005s
I20260709 12:13:40.150982 12645 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:13:40.053170 17432 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:13:40.135782 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2664) took 1291 ms. Trace:
I20260709 12:13:40.151723 12492 rpcz_store.cc:276] 0709 12:13:38.844228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.092721 (+1248493us) service_pool.cc:224] Handling call
0709 12:13:40.135708 (+ 42987us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.153162 12492 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:40.153782 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2090) took 4370 ms. Trace:
I20260709 12:13:40.136562 17628 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:13:40.154125 12492 rpcz_store.cc:276] 0709 12:13:35.783587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.152613 (+4369026us) service_pool.cc:224] Handling call
0709 12:13:40.152991 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.153481 (+   490us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.153706 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.155704 12492 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:40.156242 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2103) took 4295 ms. Trace:
I20260709 12:13:40.156790 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:40.149349 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:40.157960 17477 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:13:40.158690 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:40.158991 17588 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.705s	user 0.003s	sys 0.001s
I20260709 12:13:40.052819 17600 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 12:13:40.159960 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.161191 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.114406 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:40.115273 17457 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:13:40.161908 12570 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:40.162118 12489 raft_consensus.cc:606] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:40.162564 12489 raft_consensus.cc:614] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:40.149253 17639 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.163111 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 22) took 1289 ms. Trace:
I20260709 12:13:40.163573 12489 rpcz_store.cc:276] 0709 12:13:38.884301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.114202 (+1229901us) service_pool.cc:224] Handling call
0709 12:13:40.162928 (+ 48726us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.164438 17323 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:40.080865 17543 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
I20260709 12:13:40.144348 17456 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.150828 17595 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.166522 12568 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:40.167766 17595 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.135859 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.095738 17635 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.136960 12495 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 14. Ops: []
I20260709 12:13:40.173799 17640 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:40.174190 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 625) took 4440 ms. Trace:
I20260709 12:13:40.099247 17408 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:40.174544 12495 rpcz_store.cc:276] 0709 12:13:35.734357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.136446 (+4402089us) service_pool.cc:224] Handling call
0709 12:13:40.136773 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.173892 (+ 37119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.174130 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.175040 17408 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:40.176162 12495 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:40.176937 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2111) took 4012 ms. Trace:
I20260709 12:13:40.179404 17456 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.180737 12570 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:13:40.149501 17397 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:40.165247 12489 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:40.133067 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:40.183059 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2110) took 4069 ms. Trace:
I20260709 12:13:40.174438 17640 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:40.176301 17465 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.667s	user 0.012s	sys 0.000s
W20260709 12:13:40.183683 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50885.9us
I20260709 12:13:40.183490 12489 rpcz_store.cc:276] 0709 12:13:36.113555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.164552 (+4050997us) service_pool.cc:224] Handling call
0709 12:13:40.164883 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.182734 (+ 17851us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.182996 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.186131 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:40.156543 12492 rpcz_store.cc:276] 0709 12:13:35.860826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.155052 (+4294226us) service_pool.cc:224] Handling call
0709 12:13:40.155365 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.156009 (+   644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.156191 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.177338 12495 rpcz_store.cc:276] 0709 12:13:36.164545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.175486 (+4010941us) service_pool.cc:224] Handling call
0709 12:13:40.175795 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.176662 (+   867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.176883 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.135586 12564 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:13:40.131592 12460 connection.cc:256] RPC call timeout handler was delayed by -2.1427250560002449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
I20260709 12:13:40.126494 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:40.185429 12494 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
W20260709 12:13:40.199258 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:40.219583 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.196732 12489 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 10. Current term is 11. Ops: [9.6-10.7]
W20260709 12:13:40.220714 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.151841 17432 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:13:40.220741 12460 connection.cc:256] RPC call timeout handler was delayed by -1.977077195999982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260709 12:13:40.229460 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.173521 17635 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:40.229351 12460 connection.cc:256] RPC call timeout handler was delayed by -1.8426331340001525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2783s for a response
W20260709 12:13:40.283406 12460 connection.cc:256] RPC call timeout handler was delayed by -1.7628771960000904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24580000000000002s for a response
W20260709 12:13:40.283962 12460 connection.cc:256] RPC call timeout handler was delayed by -1.7611488360000749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260709 12:13:40.257285 17640 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:40.198551 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 163809us
I20260709 12:13:40.231911 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2642) took 4059 ms. Trace:
I20260709 12:13:40.223003 17639 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:40.192772 17222 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:40.256659 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 234381us
W20260709 12:13:40.283279 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97248.4us
I20260709 12:13:40.190687 12646 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
W20260709 12:13:40.240334 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 220505us
I20260709 12:13:40.272553 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 600) took 2409 ms. Trace:
I20260709 12:13:40.229501 17432 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:40.284349 12460 connection.cc:256] RPC call timeout handler was delayed by -1.5676608229998692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
I20260709 12:13:40.310578 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:40.311136 12460 connection.cc:256] RPC call timeout handler was delayed by -1.5162846990001526s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2795s for a response
W20260709 12:13:40.311450 12460 connection.cc:256] RPC call timeout handler was delayed by -1.2410676320000675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
W20260709 12:13:40.311790 12460 connection.cc:256] RPC call timeout handler was delayed by -1.1599608229998921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20260709 12:13:40.292625 13259 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:13:40.290298 12489 rpcz_store.cc:276] 0709 12:13:36.170545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.195233 (+4024688us) service_pool.cc:224] Handling call
0709 12:13:40.195630 (+   397us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:40.229921 (+ 34291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.230274 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:40.284054 12610 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.922s (SENT)
I20260709 12:13:40.313834 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.297523 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:40.302392 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.313881 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.309546 12571 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 4
I20260709 12:13:40.326215 12571 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:13:40.326633 12571 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:40.327011 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.327672 12489 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:40.327816 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:40.303012 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:40.294088 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.330101 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:40.191949 17595 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:40.297333 13257 rpcz_store.cc:276] 0709 12:13:37.863108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.863258 (+   150us) service_pool.cc:224] Handling call
0709 12:13:37.863573 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:38.760470 (+896897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.272230 (+1511760us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:40.272482 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1917306752}
I20260709 12:13:40.331557 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.324558 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:40.332811 17457 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:13:40.333140 17641 raft_consensus.cc:2764] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:13:40.323998 17596 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:40.334371 17645 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:40.334931 17596 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.337258 17457 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:40.276026 17571 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.338020 17427 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:40.293711 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:40.341102 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:40.341271 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47664us
W20260709 12:13:40.260604 17476 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.603s	user 0.008s	sys 0.005s
W20260709 12:13:40.365768 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71742.5us
I20260709 12:13:40.378212 12611 leader_election.cc:400] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:40.378902 12611 leader_election.cc:403] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:13:40.380071 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.345528 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:40.367563 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64696.6us
W20260709 12:13:40.305935 17137 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.661s	user 0.010s	sys 0.003s
I20260709 12:13:40.338600 17427 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:40.289757 17465 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:40.387862 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:40.381261 17317 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:40.388697 17226 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:40.389009 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:40.367821 17600 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:40.388697 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.389925 17547 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:40.390249 17600 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:40.380311 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:40.391239 17564 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.391626 17600 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.391986 17226 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:40.392100 17564 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:40.392794 17564 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:40.393198 17564 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:40.364253 17596 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:40.393731 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:40.390400 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:40.399032 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:40.402643 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61609.9us
I20260709 12:13:40.406379 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.410386 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.305060 17507 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:40.325814 17555 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:40.439338 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:40.437857 12565 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:40.296310 17227 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.344942 17323 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:40.439707 12489 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:40.323881 17222 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:40.375447 17571 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:40.339912 17456 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:40.441785 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.416955 17408 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:40.443066 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.445168 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.269145 17417 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.308597 17432 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.412935 17427 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:40.338636 17457 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.447075 17477 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:13:40.449712 17483 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:40.482605 17619 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.374364 17473 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:13:40.464308 17323 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:40.490074 17323 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:40.301734 17646 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.150s
I20260709 12:13:40.491503 17646 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:40.491930 17646 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:40.492972 17646 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.494228 17646 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:40.495613 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.444561 17583 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 12:13:40.498327 17583 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:40.498847 17583 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:40.499516 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.500169 17408 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.499899 17583 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.345649 17595 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.290940 17639 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.432400 17547 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.366083 17647 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:40.454685 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48269.9us
I20260709 12:13:40.339562 17285 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.113s
I20260709 12:13:40.497280 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.440712 17226 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:40.439510 17465 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:40.438570 17317 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.489009 17483 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.507473 12644 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:40.516183 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.519049 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.520336 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.539737 12639 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:40.523891 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.542677 12492 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:40.543426 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2125) took 4062 ms. Trace:
I20260709 12:13:40.543798 12492 rpcz_store.cc:276] 0709 12:13:36.481078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.541997 (+4060919us) service_pool.cc:224] Handling call
0709 12:13:40.542448 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.543131 (+   683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.543348 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.524392 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.440171 17596 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:40.462890 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.553972 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.467311 12491 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:40.470978 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:40.471169 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.554543 12489 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:40.444577 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45639.8us
I20260709 12:13:40.555536 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:40.560702 17564 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.454202 17555 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:40.472725 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:40.562333 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91533.7us
I20260709 12:13:40.562507 12640 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:40.564126 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.564791 12640 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:40.564915 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:40.473778 12637 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:40.565824 17576 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:40.567090 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.468205 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:40.481081 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.569137 12642 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11. Re-sending same reply.
I20260709 12:13:40.569344 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2676) took 1495 ms. Trace:
I20260709 12:13:40.481482 12572 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:13:40.464740 17497 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:40.466528 17222 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.465811 17571 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:40.466813 17456 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:40.545531 12492 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:13:40.480371 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100126us
I20260709 12:13:40.569909 12491 rpcz_store.cc:276] 0709 12:13:39.073950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.384327 (+1310377us) service_pool.cc:224] Handling call
0709 12:13:40.569278 (+184951us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:40.570544 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.530014 17483 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.574591 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:40.576862 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108741us
I20260709 12:13:40.577877 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 601) took 2713 ms. Trace:
I20260709 12:13:40.580435 17571 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:40.599781 13253 rpcz_store.cc:276] 0709 12:13:37.864056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:37.864172 (+   116us) service_pool.cc:224] Handling call
0709 12:13:39.399302 (+1535130us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:39.400104 (+   802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.577509 (+1177405us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54000d11d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:40.577773 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3529707904}
I20260709 12:13:40.602623 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2128) took 4038 ms. Trace:
I20260709 12:13:40.603749 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:40.604708 12563 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:40.605404 12489 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:40.606189 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:40.603072 12492 rpcz_store.cc:276] 0709 12:13:36.563953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.544708 (+3980755us) service_pool.cc:224] Handling call
0709 12:13:40.545096 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.600668 (+ 55572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.602515 (+  1847us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.590099 17456 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:40.501579 17583 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:40.502827 17417 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:40.580159 17323 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.549018 17317 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:40.504259 17595 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.528079 17639 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.584087 17644 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:40.576261 17497 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:40.584545 17645 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:40.522675 17465 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.518172 17285 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:40.586498 12494 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:40.666635 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 631) took 2777 ms. Trace:
I20260709 12:13:40.667074 12494 rpcz_store.cc:276] 0709 12:13:37.889176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.585862 (+2696686us) service_pool.cc:224] Handling call
0709 12:13:40.586286 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.666293 (+ 80007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.666569 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.508147 17397 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.580725 17555 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:40.586544 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.678820 12496 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:40.519342 17226 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:40.506713 17432 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:40.547890 17314 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:40.590492 12490 raft_consensus.cc:2376] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10. Re-sending same reply.
I20260709 12:13:40.586988 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.615732 17557 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
W20260709 12:13:40.597771 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.685374 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 636) took 2116 ms. Trace:
I20260709 12:13:40.572839 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:40.687229 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114441us
I20260709 12:13:40.688324 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:40.657188 17497 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:40.578126 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2639) took 4417 ms. Trace:
I20260709 12:13:40.618708 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.582592 17408 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:40.668164 17644 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:40.693890 17408 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:40.694435 17408 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:40.694842 17644 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.624557 12639 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:40.697597 17005 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:40.698346 12639 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:13:40.680617 17226 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:40.625610 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2479) took 1011 ms. Trace:
I20260709 12:13:40.699349 12567 rpcz_store.cc:276] 0709 12:13:39.614563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.615855 (+  1292us) service_pool.cc:224] Handling call
0709 12:13:39.747026 (+131171us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:39.765612 (+ 18586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.625360 (+859748us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400044f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:40.625528 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250641664}
I20260709 12:13:40.625527 12488 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
W20260709 12:13:40.700878 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2649) took 1894 ms (client timeout 2032 ms). Trace:
I20260709 12:13:40.626758 12563 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:40.619832 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 241270us
I20260709 12:13:40.689534 12493 rpcz_store.cc:276] 0709 12:13:36.161221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.169191 (+4007970us) service_pool.cc:224] Handling call
0709 12:13:40.169615 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:40.170234 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:13:40.170269 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:40.170983 (+   714us) log.cc:844] Serialized 26 byte log entry
0709 12:13:40.512997 (+342014us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b5400297320. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf576a6b8 00007ffbf57f392b 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c
0709 12:13:40.513089 (+    92us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:13:40.513173 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.513285 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:40.513353 (+    68us) raft_consensus.cc:1685] finished
0709 12:13:40.513373 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:40.578047 (+ 64674us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1538,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"spinlock_wait_cycles":113220480,"thread_start_us":101163,"threads_started":1,"wal-append.queue_time_us":95940}
I20260709 12:13:40.704579 17227 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:40.705195 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 637) took 2109 ms. Trace:
I20260709 12:13:40.705438 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.641551 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2146) took 1146 ms. Trace:
I20260709 12:13:40.706691 12645 rpcz_store.cc:276] 0709 12:13:39.495153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.495401 (+   248us) service_pool.cc:224] Handling call
0709 12:13:39.495820 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:40.151414 (+655594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.641235 (+489821us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:40.641473 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1506102144}
W20260709 12:13:40.701279 12488 rpcz_store.cc:269] 0709 12:13:38.806161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.382418 (+576257us) service_pool.cc:224] Handling call
0709 12:13:40.524218 (+1141800us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:13:40.700806 (+176588us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56738048}
I20260709 12:13:40.709164 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:40.710983 12567 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:13:40.715900 17397 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.717437 12497 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:40.718685 17222 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.722066 17482 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:40.722466 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105072us
I20260709 12:13:40.728070 12641 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:40.729148 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:40.729825 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.646454 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 602) took 1303 ms. Trace:
I20260709 12:13:40.621455 17507 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:40.660107 17427 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.662369 17323 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:40.668982 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:40.672523 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.594810 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:40.770900 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 176054us
I20260709 12:13:40.772083 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:40.682085 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2668) took 1773 ms (client timeout 2212 ms). Trace:
I20260709 12:13:40.773361 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.598898 12566 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:13:40.774567 12489 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:40.666064 17285 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:40.691039 17641 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:40.689254 17314 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.691435 17635 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 12:13:40.706211 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:40.686041 12494 rpcz_store.cc:276] 0709 12:13:38.561807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.667959 (+2106152us) service_pool.cc:224] Handling call
0709 12:13:40.676775 (+  8816us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.677171 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:40.677205 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:40.677256 (+    51us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:40.677288 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.677367 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:40.677564 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5187,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":5988}
I20260709 12:13:40.734967 12638 rpcz_store.cc:276] 0709 12:13:39.343407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.360378 (+ 16971us) service_pool.cc:224] Handling call
0709 12:13:40.646202 (+1285824us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:40.646366 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2363058560}
I20260709 12:13:40.784277 13261 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:40.785058 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.735440 12640 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:40.735370 12493 rpcz_store.cc:276] 0709 12:13:38.596048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.704327 (+2108279us) service_pool.cc:224] Handling call
0709 12:13:40.704675 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.704853 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:40.704880 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:40.704920 (+    40us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:40.704949 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.705000 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:40.705113 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.785683 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.747473 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2643) took 4561 ms. Trace:
I20260709 12:13:40.757439 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:40.786900 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.787249 13254 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:40.787611 12640 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:40.787837 13255 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
W20260709 12:13:40.789778 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.757180 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:40.790602 13258 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:13:40.737087 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:40.792428 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.792656 17456 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.793836 17317 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:40.783535 17507 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:40.740684 17476 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.788916 12640 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:40.794941 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.796348 17388 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:13:40.796805 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.797386 12494 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:40.802901 12640 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 12:13:40.767453 12491 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 9. Current term is 10. Ops: []
W20260709 12:13:40.804885 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2672) took 1860 ms (client timeout 1892 ms). Trace:
W20260709 12:13:40.773267 12496 rpcz_store.cc:269] 0709 12:13:38.908618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.586231 (+1677613us) service_pool.cc:224] Handling call
0709 12:13:40.682010 (+ 95779us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.691241 17440 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:40.780995 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.798774 17314 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.799969 17285 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.801487 17635 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:40.823550 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.786617 12495 rpcz_store.cc:276] 0709 12:13:36.186338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.222448 (+4036110us) service_pool.cc:224] Handling call
0709 12:13:40.222825 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:40.226193 (+  3368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:40.226240 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:40.226854 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:13:40.526927 (+300073us) spinlock_profiling.cc:243] Waited 6.15 ms on lock 0x7b1800333520. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:13:40.738762 (+211835us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:40.738808 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.738894 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:40.738945 (+    51us) raft_consensus.cc:1685] finished
0709 12:13:40.738957 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:40.747392 (+  8435us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":11427,"spinlock_wait_cycles":14139264,"thread_start_us":211627,"threads_started":1,"wal-append.queue_time_us":475066}
I20260709 12:13:40.825747 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.826521 12495 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:40.827306 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:40.805356 12497 rpcz_store.cc:269] 0709 12:13:38.945030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.372237 (+427207us) service_pool.cc:224] Handling call
0709 12:13:40.499381 (+1127144us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:13:40.804664 (+305283us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:40.804825 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560118400}
W20260709 12:13:40.736136 12461 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.985s (SENT)
W20260709 12:13:40.829581 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93486.6us
I20260709 12:13:40.810534 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:40.766670 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2670) took 1839 ms (client timeout 1922 ms). Trace:
I20260709 12:13:40.741503 17482 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:40.816833 12492 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:40.691656 17564 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:40.816915 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:40.820281 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:40.841140 13256 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:40.841919 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:40.842618 13255 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:40.843338 17397 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:13:40.784148 17645 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:13:40.746704 17547 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:40.767740 17457 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:40.822254 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.831605 12494 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:40.831405 12490 rpcz_store.cc:269] 0709 12:13:38.927649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:39.393934 (+466285us) service_pool.cc:224] Handling call
0709 12:13:40.518808 (+1124874us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:13:40.766428 (+247620us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:40.766591 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144350848}
I20260709 12:13:40.834048 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 635) took 2735 ms. Trace:
I20260709 12:13:40.834841 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.867611 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.868302 12497 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:40.837206 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.868479 12494 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:40.838441 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:40.757512 17387 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 12:13:40.847038 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.874408 12493 raft_consensus.cc:2376] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13. Re-sending same reply.
I20260709 12:13:40.821863 17619 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:40.850251 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.880416 12495 raft_consensus.cc:2376] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13. Re-sending same reply.
I20260709 12:13:40.860170 13256 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:40.861997 12643 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:40.832356 17657 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:40.867395 12491 rpcz_store.cc:276] 0709 12:13:38.098159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.620833 (+2522674us) service_pool.cc:224] Handling call
0709 12:13:40.767245 (+146412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.805135 (+ 37890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.833748 (+ 28613us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54001900d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:40.833959 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264278272}
I20260709 12:13:40.881933 12495 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:40.850746 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:40.875619 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.847769 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.884318 13255 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:40.884905 12497 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:40.887564 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:40.877116 12496 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:40.892668 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2645) took 2087 ms. Trace:
I20260709 12:13:40.882743 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2655) took 2065 ms. Trace:
W20260709 12:13:40.883484 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45117.6us
I20260709 12:13:40.893838 12495 rpcz_store.cc:276] 0709 12:13:38.817241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.881246 (+2064005us) service_pool.cc:224] Handling call
0709 12:13:40.881690 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.882479 (+   789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.882679 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.895305 12495 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:40.895861 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2158) took 1383 ms. Trace:
I20260709 12:13:40.896152 12495 rpcz_store.cc:276] 0709 12:13:39.512679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.894640 (+1381961us) service_pool.cc:224] Handling call
0709 12:13:40.894960 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.895623 (+   663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.895803 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.893116 12496 rpcz_store.cc:276] 0709 12:13:38.804971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.876440 (+2071469us) service_pool.cc:224] Handling call
0709 12:13:40.876868 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.892366 (+ 15498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.892603 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.899996 12496 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:13:40.901968 12496 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:13:40.903600 12496 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:40.905148 12496 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:40.823192 17635 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:40.911468 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.907393 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2650) took 2079 ms. Trace:
I20260709 12:13:40.836447 17440 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:40.907457 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59854.5us
I20260709 12:13:40.906718 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:40.911290 12493 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:13:40.913161 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:40.945474 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:40.944077 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:40.849917 17564 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:40.946599 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95911.6us
I20260709 12:13:40.946520 17564 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.879253 17645 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:40.945616 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2156) took 1441 ms. Trace:
I20260709 12:13:40.907821 17457 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:40.947129 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:40.874425 17388 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:40.948102 17645 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.948882 17457 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.948994 17388 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:40.948572 12640 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:40.952119 17388 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:40.952844 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 253681us
I20260709 12:13:40.912402 12497 rpcz_store.cc:276] 0709 12:13:38.806441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.878411 (+2071970us) service_pool.cc:224] Handling call
0709 12:13:40.884724 (+  6313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.885323 (+   599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.885564 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.954002 12493 rpcz_store.cc:276] 0709 12:13:39.504146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.892679 (+1388533us) service_pool.cc:224] Handling call
0709 12:13:40.893079 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.945283 (+ 52204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.945544 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:40.953190 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:40.897485 12495 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:40.956772 13255 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:40.956274 17564 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:40.959281 12491 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:13:40.870023 17483 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:40.959724 12491 pending_rounds.cc:85] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 10
W20260709 12:13:40.924338 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:40.961709 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:40.965713 12491 pending_rounds.cc:107] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:13:40.836555 17614 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:13:40.966848 12491 raft_consensus.cc:2889] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:40.974687 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2659) took 2140 ms. Trace:
I20260709 12:13:40.975075 12491 rpcz_store.cc:276] 0709 12:13:38.833919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.882453 (+2048534us) service_pool.cc:224] Handling call
0709 12:13:40.958916 (+ 76463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.974372 (+ 15456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:40.974618 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:40.970659 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84042.4us
I20260709 12:13:40.906658 17222 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:40.939584 12496 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.977833 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:40.978652 12566 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:40.849596 17482 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.875460 17595 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 12:13:40.869935 17323 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.099s
I20260709 12:13:40.959686 17543 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:41.000780 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:40.982327 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:40.960889 17314 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:40.977540 13258 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:41.014621 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:41.015527 12496 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:41.022014 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:41.052394 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:41.023010 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:41.023856 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.028617 17635 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.029022 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.058166 12642 raft_consensus.cc:2376] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13. Re-sending same reply.
I20260709 12:13:41.059499 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.060236 12642 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:41.030733 17226 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:40.984676 17005 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:13:41.078876 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.035547 17465 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:41.042395 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:41.030752 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2145) took 3136 ms. Trace:
W20260709 12:13:41.087901 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87189.3us
I20260709 12:13:41.092440 12492 rpcz_store.cc:276] 0709 12:13:37.894453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.608698 (+2714245us) service_pool.cc:224] Handling call
0709 12:13:40.806110 (+197412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.846000 (+ 39890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:41.030464 (+184464us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54000c44d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:41.030666 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453121920}
I20260709 12:13:41.043035 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:41.093392 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:41.044226 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.091634 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.045228 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.050695 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.046456 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.095781 12497 raft_consensus.cc:2376] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:41.052706 13261 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [12.9-16.11]
I20260709 12:13:41.055114 13256 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
W20260709 12:13:41.098305 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116090us
I20260709 12:13:41.055619 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:40.985054 17222 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:41.063192 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.063587 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:41.064311 17440 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.065017 17635 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:40.979262 17507 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.065712 17497 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.120296 17497 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:41.065924 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:41.121380 13255 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:13:41.122376 12644 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:41.122010 17668 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:41.066254 17456 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.067198 17427 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:41.068547 17666 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:13:41.073467 17571 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:41.070842 17576 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.504s	user 0.007s	sys 0.004s
I20260709 12:13:41.101209 17432 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:41.125846 17576 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:41.073374 17646 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.579s	user 0.011s	sys 0.000s
I20260709 12:13:41.126717 12570 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:41.079984 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.127182 13254 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20260709 12:13:41.046372 17658 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:13:41.099996 13258 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:41.094825 12489 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:41.070745 17482 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.107774 13260 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:41.093947 17543 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:41.051365 17314 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:41.115324 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.117663 12488 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:41.118199 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.146948 12640 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:41.127825 12639 raft_consensus.cc:606] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:41.147080 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:41.109567 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46132.9us
I20260709 12:13:41.148666 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.094233 17465 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:41.109869 17222 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:41.147360 12639 raft_consensus.cc:614] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:41.086582 17473 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:13:41.090951 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45942.5us
W20260709 12:13:41.158106 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.154058 17314 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:41.128973 17477 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:41.166618 17571 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:41.154922 17387 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:41.169561 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.170981 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.171736 12640 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:41.172492 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.128638 17432 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:13:41.173491 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.161517 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:41.106058 17669 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:41.154278 17657 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:13:41.195116 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.173874 17408 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:41.176705 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75644.3us
I20260709 12:13:41.177656 13253 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:41.180595 12640 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:41.158082 12496 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:41.155100 17646 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:13:41.162276 12637 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:41.175494 17473 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:41.210634 12645 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
W20260709 12:13:41.211129 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117837us
W20260709 12:13:41.204490 17674 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:13:41.197163 17571 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:41.207764 17669 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:41.216132 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 12:13:41.164065 12642 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:13:41.231556 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:41.233268 12642 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:13:41.242866 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85285.7us
I20260709 12:13:41.244191 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:41.230291 17669 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:41.300675 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56551.9us
I20260709 12:13:41.302695 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.303100 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.288358 17669 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:41.246054 17432 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:41.248020 17686 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:13:41.250654 12638 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:13:41.308101 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.308766 12496 raft_consensus.cc:2376] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14. Re-sending same reply.
I20260709 12:13:41.255291 17408 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:41.256445 17583 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.622s	user 0.008s	sys 0.003s
I20260709 12:13:41.227597 17543 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.258682 17314 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.312633 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.261214 17635 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:41.313932 17635 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:41.314545 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.315320 17635 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.316344 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.316985 12495 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:41.263545 17226 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:41.267082 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.281899 17473 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.237056 17456 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:41.246510 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:41.253942 17684 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:41.310515 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.327318 12496 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:41.310544 17692 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:13:41.328814 17684 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:41.329190 17684 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:41.332363 12638 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:13:41.322562 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.333913 12638 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:41.305742 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.320768 17226 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:41.324007 17314 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.317394 17635 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:41.325245 17456 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:41.354859 17684 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.355183 17456 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.356617 17684 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:41.360654 17285 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:41.313617 12494 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:13:41.373891 17619 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:41.374574 17619 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.375624 17678 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:41.376255 17678 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:41.376149 17222 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.377020 17483 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:41.334884 12489 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:41.340340 17226 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:41.347582 12638 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:13:41.353909 17564 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:41.375020 12536 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:34492 dropped due to backpressure: service queue is full with 50 items
I20260709 12:13:41.308743 17432 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:41.365298 17657 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:41.344815 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.382236 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:41.426834 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82184.7us
I20260709 12:13:41.427075 12563 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:13:41.378635 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.387408 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:41.432334 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:41.432699 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:41.433216 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:41.431512 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:41.434008 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:41.434612 12563 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:41.436865 12569 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:13:41.379202 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 336612us
I20260709 12:13:41.440824 12568 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:41.414709 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:41.443331 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:41.443951 12564 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:13:41.433496 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
W20260709 12:13:41.448499 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.449764 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:41.452943 12496 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 7. (term mismatch)
I20260709 12:13:41.453382 12496 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 6
I20260709 12:13:41.453598 12496 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 15.7
I20260709 12:13:41.453846 12496 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:41.455854 12496 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:41.457415 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.458092 12496 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:41.465858 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:41.466815 12490 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:41.467767 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:41.471164 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56606.7us
I20260709 12:13:41.472150 12536 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15.
I20260709 12:13:41.441958 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:41.474511 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:41.475704 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:41.476687 12489 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:41.473510 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.481753 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:41.464704 17222 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.483197 12495 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:41.488718 17432 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.490159 17432 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:41.378672 17595 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:41.493196 17595 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:41.494201 17595 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.495512 17595 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:41.379436 17576 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:41.408361 17639 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:41.431618 17657 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:41.497938 17639 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:41.498909 17639 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.500118 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:41.500721 12567 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:41.501919 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:41.502671 12490 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:41.503198 12496 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:41.408432 17227 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:41.505985 17227 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:41.497949 17576 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:41.510210 17576 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:41.486615 17614 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:13:41.514304 17583 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:41.514717 17583 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:41.515591 17583 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.516825 17583 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:41.524348 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48706.9us
I20260709 12:13:41.528258 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:41.529273 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:41.378927 17677 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:41.530289 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 312474us
I20260709 12:13:41.441512 17564 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:41.377580 17619 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:41.536968 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.484431 17600 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:41.544606 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:41.547667 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.551311 12646 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:13:41.557838 12641 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14. Re-sending same reply.
I20260709 12:13:41.483742 17645 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:41.561352 12641 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:41.485831 17635 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:41.486181 17340 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:41.519186 17686 raft_consensus.cc:2764] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 12:13:41.536362 17687 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.227s	user 0.003s	sys 0.000s
I20260709 12:13:41.570876 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.386857 17669 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:41.577970 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.579288 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:41.594404 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.594460 12567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:41.596829 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.597405 12641 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:41.538046 17684 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:41.414162 17465 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.411864 17226 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.564038 17387 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:41.604862 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:41.605484 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.577127 17564 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:41.378067 17483 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.593040 17645 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:41.533252 17690 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.089s
I20260709 12:13:41.626768 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:41.600425 17684 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:41.568724 17482 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:13:41.569646 17600 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.377602 17678 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.553536 17677 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.613710 17387 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:41.647156 17340 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:41.647898 17340 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.650081 17645 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.601239 17619 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:13:41.628674 17465 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:41.616619 17564 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:41.610266 17556 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:13:41.632894 17684 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:41.637866 17689 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:41.662114 17387 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:41.663880 17564 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:41.666958 17222 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:41.741487 12565 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:13:41.744760 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 163430us
I20260709 12:13:41.754453 17600 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:41.754555 17340 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.758831 17314 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:41.762400 17227 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.765489 17227 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.766786 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:41.767539 17677 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:41.767617 12565 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:41.772642 17576 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:41.774638 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:41.787415 12565 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:41.791239 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:41.791580 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.738792 17689 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:41.798389 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:41.798694 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 254136us
I20260709 12:13:41.801112 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:41.802181 12567 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:41.801362 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.803707 12641 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:41.750154 17692 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:13:41.809383 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.811882 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.812184 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.812728 12641 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:41.812881 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:41.813431 12637 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:41.814198 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.815268 12641 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:13:41.818120 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.799245 17227 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:41.826085 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.747632 17222 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:41.841748 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:41.853897 12646 raft_consensus.cc:2376] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10. Re-sending same reply.
I20260709 12:13:41.799065 17483 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:13:41.802271 17677 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:41.846341 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.803215 17285 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:41.798926 17340 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
W20260709 12:13:41.874022 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:41.868587 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:41.808130 17477 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:13:42.008613 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140090us
I20260709 12:13:41.869606 17678 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:42.017136 17706 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.017413 17314 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:41.871562 17227 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:42.017485 17678 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.030766 17678 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.033273 17678 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.021522 17477 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:42.001639 17645 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:42.034403 17677 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:13:42.074206 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:42.029104 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:42.076416 12646 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:41.871258 17285 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:42.071381 17704 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:13:42.072060 17477 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:42.086611 17595 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.591s	user 0.012s	sys 0.002s
I20260709 12:13:42.049814 17689 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.088755 17576 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:42.089102 17432 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.588s	user 0.014s	sys 0.000s
W20260709 12:13:42.065266 17657 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.567s	user 0.013s	sys 0.000s
I20260709 12:13:42.103039 17285 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:42.137256 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:42.144167 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:42.121253 17645 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.131247 17677 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:42.138274 17583 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.621s	user 0.017s	sys 0.000s
I20260709 12:13:42.028077 17706 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.167100 12646 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:42.156967 17645 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.142951 17657 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.161681 17677 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.195613 12645 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:42.203718 17706 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:42.204154 17657 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:42.206225 17677 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.210188 17222 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.156486 17227 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:42.231197 17465 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.000s
I20260709 12:13:42.224658 17564 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.223783 17387 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.306277 17483 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:42.306154 17657 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:42.357424 17564 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.351425 17657 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.350632 17483 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.364868 17483 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:42.366416 17684 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.647s	user 0.013s	sys 0.000s
I20260709 12:13:42.361419 17657 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:42.367339 17684 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.374379 17684 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:42.374960 17547 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.099s	sys 0.180s
W20260709 12:13:42.375833 17547 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.099s	sys 0.181s
W20260709 12:13:42.421207 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:42.422570 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:42.435236 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:42.443760 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.448587 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:42.449797 12564 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:42.462595 17222 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:42.464193 17222 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.482402 17432 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.483547 17432 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.485536 17483 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.486802 17483 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.488785 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.490358 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:42.492625 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:42.494072 17476 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.495195 17476 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.500018 17689 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.501618 17689 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.515172 17692 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:42.515893 17692 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:42.516300 17692 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:42.531041 17483 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.534523 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.536177 17689 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.526207 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:42.538086 17432 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.547050 17476 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:42.547533 17697 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.790s	user 0.018s	sys 0.079s
I20260709 12:13:42.550027 17657 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.550467 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 796) took 1169 ms. Trace:
I20260709 12:13:42.551028 17657 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.550982 12572 rpcz_store.cc:276] 0709 12:13:41.381055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:41.382326 (+  1271us) service_pool.cc:224] Handling call
0709 12:13:41.382779 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:41.383320 (+   541us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:41.383362 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:41.383955 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 12:13:41.554745 (+170790us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:41.554809 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:41.554917 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:42.546212 (+991295us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:42.549461 (+  3249us) raft_consensus.cc:1685] finished
0709 12:13:42.549483 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:42.550273 (+   790us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":147,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"spinlock_wait_cycles":40715648,"thread_start_us":170278,"threads_started":1,"wal-append.queue_time_us":335878}
I20260709 12:13:42.549093 17222 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.540282 17387 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:42.565567 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.570830 12572 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 13 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 5.
I20260709 12:13:42.572551 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2644) took 3768 ms. Trace:
I20260709 12:13:42.573132 12488 rpcz_store.cc:276] 0709 12:13:38.804629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.708982 (+1904353us) service_pool.cc:224] Handling call
0709 12:13:40.709301 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:41.146421 (+437120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:42.571795 (+1425374us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:42.572270 (+   475us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938469760}
I20260709 12:13:42.610558 17483 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.611723 17483 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.616926 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:58586 (request call id 15) took 2026 ms. Trace:
I20260709 12:13:42.617841 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:42.618552 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2374) took 2141 ms. Trace:
W20260709 12:13:42.618887 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56746us
I20260709 12:13:42.619004 12643 rpcz_store.cc:276] 0709 12:13:40.476751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.477042 (+   291us) service_pool.cc:224] Handling call
0709 12:13:40.477886 (+   844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:40.881677 (+403791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:42.617924 (+1736247us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:42.618331 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":882328448}
I20260709 12:13:42.605808 17657 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.618871 13253 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:42.622709 17690 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:42.623302 17690 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:42.623678 17690 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:42.622754 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:42.627115 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:42.617465 12639 rpcz_store.cc:276] 0709 12:13:40.599424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.698179 (+ 98755us) service_pool.cc:224] Handling call
0709 12:13:42.616327 (+1918148us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:42.616641 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":985948416}
I20260709 12:13:42.639525 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:42.640785 13259 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:42.643522 17677 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.641366 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 605) took 1886 ms. Trace:
I20260709 12:13:42.647387 17657 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.644455 17677 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.648589 17657 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:42.650158 17708 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.577s	user 0.004s	sys 0.006s
I20260709 12:13:42.650665 17677 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:42.653766 17701 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.896s	user 0.033s	sys 0.062s
I20260709 12:13:42.644809 13259 rpcz_store.cc:276] 0709 12:13:40.754621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.754804 (+   183us) service_pool.cc:224] Handling call
0709 12:13:42.638963 (+1884159us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:13:42.641247 (+  2284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47530368}
I20260709 12:13:42.660660 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:42.673533 17483 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.673903 13262 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:42.678210 12643 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:42.711082 12564 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:42.719374 17350 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.721180 17350 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.721870 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:42.726342 17350 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:42.728917 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 778) took 1942 ms. Trace:
I20260709 12:13:42.729534 13261 rpcz_store.cc:276] 0709 12:13:40.786509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.786970 (+   461us) service_pool.cc:224] Handling call
0709 12:13:41.052288 (+265318us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:13:41.096623 (+ 44335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:42.728328 (+1631705us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540058d750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:42.728655 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562627840}
I20260709 12:13:42.730252 17688 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:42.736950 17692 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.774402 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:42.793715 17688 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:42.794516 17727 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:42.796473 17727 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:42.803822 17350 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:42.804747 17350 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.808382 17685 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.809470 17685 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.812439 17685 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.827029 17689 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.828227 17685 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.829214 17685 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.829967 17005 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.830950 17005 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.788044 17432 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:42.834021 17005 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.834746 17432 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.835714 17350 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.836602 17645 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.839154 17727 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.841746 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.843415 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.844272 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
W20260709 12:13:42.844653 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2568) took 2241 ms (client timeout 1665 ms). Trace:
W20260709 12:13:42.845229 12570 rpcz_store.cc:269] 0709 12:13:40.603088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.605826 (+  2738us) service_pool.cc:224] Handling call
0709 12:13:42.844018 (+2238192us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:42.844363 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1194994560}
I20260709 12:13:42.845785 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.847126 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:42.847215 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:42.849296 12564 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:13:42.828075 17689 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.856199 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:42.857209 17350 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:42.857857 17657 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.858748 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:42.859207 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:42.859771 17564 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:42.862164 17657 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:42.862803 17657 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:42.857344 12488 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:42.863055 12496 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:13:42.865283 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.865909 17669 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.866340 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 9.
I20260709 12:13:42.867434 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:42.868330 12490 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:13:42.868779 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:42.868981 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:42.869680 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.855969 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.856215 13259 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:13:42.873842 17685 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:42.874325 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2377) took 2395 ms (client timeout 2783 ms). Trace:
I20260709 12:13:42.874795 17482 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:42.875890 17482 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:42.877866 17614 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:42.878280 17614 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:42.880057 17703 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:42.880493 17703 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:42.886528 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:42.887398 17226 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.888237 17226 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:42.874816 12644 rpcz_store.cc:269] 0709 12:13:40.478615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.644446 (+165831us) service_pool.cc:224] Handling call
0709 12:13:42.873907 (+2229461us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:42.874108 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":809389312}
I20260709 12:13:42.897097 12494 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:13:42.900302 17226 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:42.879536 17614 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.900619 12495 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [10.6-11.7]
I20260709 12:13:42.901942 17614 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:42.903167 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.903630 17323 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.903920 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:42.904472 17323 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.904783 17703 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.4, Last appended by leader: 4, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.906527 17703 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:42.906819 12495 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 12:13:42.908258 12572 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:13:42.908751 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:42.908207 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:42.910892 17722 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:13:42.911209 12494 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:42.912708 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.913493 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:42.915071 12488 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:42.917709 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.918910 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:42.919952 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:42.920450 12488 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:42.921715 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:42.925315 17669 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:42.925540 17476 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.926299 17476 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:42.911382 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:42.929371 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:13:42.930406 12644 raft_consensus.cc:606] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:42.930639 17600 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.930935 12644 raft_consensus.cc:614] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:42.932349 17647 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:42.933284 17647 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.926146 17669 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.936909 17417 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:42.938633 17706 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.939563 17706 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.942889 17692 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.943643 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:13:42.944502 12644 raft_consensus.cc:606] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:42.944830 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:42.945513 12564 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:42.951777 17596 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.952618 17596 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.951982 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2502) took 1323 ms. Trace:
I20260709 12:13:42.954524 12567 rpcz_store.cc:276] 0709 12:13:41.628158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:41.628418 (+   260us) service_pool.cc:224] Handling call
0709 12:13:41.641942 (+ 13524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:41.802679 (+160737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:42.951468 (+1148789us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:42.951809 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":368025472}
I20260709 12:13:42.958433 17668 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:42.959985 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:42.963022 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:42.959276 17668 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:42.969187 17340 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:42.969884 17340 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:42.971312 17340 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.972755 17727 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:42.973620 17705 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:42.973031 17340 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:42.974346 17705 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:42.979368 17705 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:42.981868 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:42.982007 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:42.982816 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:13:42.983772 17709 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.913s	user 0.008s	sys 0.003s
I20260709 12:13:42.987224 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:42.988131 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:42.995311 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:43.000795 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79185.3us
I20260709 12:13:43.006539 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:43.006910 17476 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.007939 17705 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.009091 17705 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.011950 17705 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:43.012337 12643 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:43.013072 17706 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:42.931360 17600 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.016371 12490 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:43.017335 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:43.017912 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.018380 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:42.850032 17432 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:42.853086 17689 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.028208 12492 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
W20260709 12:13:43.037649 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.033072 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.043161 13262 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:42.937877 17417 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.020258 17692 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:43.057879 13253 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:42.864025 17657 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.044368 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:43.049467 17669 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.050307 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.050973 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:43.051434 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:43.052263 12493 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:42.796851 17688 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.088212 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.089143 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:43.001905 17668 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.059518 17692 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.090349 17692 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:43.063266 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.072917 17685 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.075030 17716 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.076668 12492 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:13:43.034600 17476 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.078545 17596 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:43.078984 17647 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:43.079659 17417 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:43.083062 17600 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:43.111616 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.084983 12610 leader_election.cc:400] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
W20260709 12:13:43.089179 17285 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.938s	user 0.016s	sys 0.016s
I20260709 12:13:43.074150 17689 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:43.108877 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:43.102775 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:43.083974 17323 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:43.108217 17556 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.117467 17689 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:43.118562 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.124821 17668 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.125090 17685 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.125475 17668 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.127550 17668 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.122941 17227 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:13:43.105988 17226 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:43.128799 17716 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.129467 17689 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:43.129314 17226 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.131477 17716 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.124023 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.131916 17596 raft_consensus.cc:993] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:43.123227 12565 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:43.099977 17476 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.146096 12567 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:43.146198 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.127621 17685 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.137743 17556 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.148414 12569 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:43.123617 17285 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.150790 13253 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:43.143903 12494 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:43.135735 17323 raft_consensus.cc:1081] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:43.156138 12497 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:43.156297 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:43.156962 13257 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:13:43.155802 12564 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 16 index: 7. (index mismatch)
I20260709 12:13:43.158390 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.164117 12572 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:13:43.170445 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85715.6us
I20260709 12:13:43.172102 17727 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:13:43.173274 17456 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:43.173866 17456 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:43.174254 17456 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:43.176895 17727 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:43.177384 17727 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:43.178316 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:43.178568 17727 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.180106 17727 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:43.190335 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:43.191360 13260 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:43.191766 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:43.192693 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:43.193121 17387 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:43.193472 13258 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:43.208333 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:43.162686 17690 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.538s	user 0.003s	sys 0.013s
W20260709 12:13:43.214475 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.215606 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.216715 12566 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:43.217075 13253 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:43.215699 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:43.221400 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:43.222076 12564 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:43.233946 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:43.238860 12568 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:43.240942 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.242229 17685 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.242812 17685 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.244150 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.245224 12566 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:43.246241 17716 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:43.246767 17716 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.246891 12571 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:43.247191 17716 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:43.248024 17571 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.248600 17571 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.251117 17571 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.252257 12567 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:13:43.252529 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:43.252923 17556 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.234277 17005 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.253652 17227 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:43.253959 17005 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.255025 17227 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:43.255656 17227 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:43.256446 17005 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:43.257015 17477 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:13:43.257359 17668 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:43.257850 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:43.259358 17669 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:13:43.259666 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64822.1us
I20260709 12:13:43.260735 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:43.262692 17476 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.263219 17408 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:43.264240 17408 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:43.265746 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:43.245236 17685 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.256709 17227 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:43.269677 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46417.5us
I20260709 12:13:43.270829 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.271051 17705 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:43.253082 12565 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:13:43.279707 12493 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:43.280885 13262 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:43.284366 13261 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:13:43.220539 17689 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.285934 12572 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:43.257896 17668 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:43.251375 17285 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.300962 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.222579 17690 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.311048 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:43.316535 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:43.320701 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60077.6us
I20260709 12:13:43.299252 17669 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:43.329481 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:43.334903 13260 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:43.328629 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:43.338181 12495 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:43.336426 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.336238 13257 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:13:43.338265 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72685us
I20260709 12:13:43.341861 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:43.344199 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:43.345781 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:43.346351 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:43.349457 17731 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.267s	user 0.000s	sys 0.002s
I20260709 12:13:43.352238 12570 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:13:43.358795 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.366719 13258 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:13:43.368160 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.377655 17417 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:43.356135 17703 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:43.382023 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.383209 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:43.380266 17417 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:43.385974 17417 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:43.393981 17739 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:13:43.396662 17724 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.332s	user 0.000s	sys 0.003s
W20260709 12:13:43.399137 17721 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.334s	user 0.001s	sys 0.002s
I20260709 12:13:43.402076 17226 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:43.403388 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.405882 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.406890 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:43.409458 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:43.402212 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:43.416553 13260 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:13:43.418499 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 155858us
I20260709 12:13:43.418605 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.419559 12566 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:43.421869 17689 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:43.422817 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81041.8us
I20260709 12:13:43.425007 12610 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:43.425357 12610 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:13:43.341344 17323 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:13:43.431749 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:43.433087 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:43.433444 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71428.2us
W20260709 12:13:43.442699 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:43.448783 17728 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.384s	user 0.002s	sys 0.000s
I20260709 12:13:43.450688 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.416821 17226 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:43.454731 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:43.458513 12643 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:43.456527 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.433359 17647 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:43.434223 17703 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:43.469135 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:43.453099 17564 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:43.472443 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:43.454188 17689 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:13:43.466584 17614 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.564s	user 0.014s	sys 0.001s
I20260709 12:13:43.453265 17226 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:43.476552 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 113359us
I20260709 12:13:43.505614 17226 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:43.509217 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:43.512552 17689 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:43.493842 17647 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.465616 17227 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:43.416136 17668 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.486240 17702 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:43.466109 17596 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10.
I20260709 12:13:43.471793 17476 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:13:43.488613 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:43.553604 17614 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:43.553918 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65436.9us
I20260709 12:13:43.555017 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:43.556779 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:43.558116 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 654) took 2071 ms. Trace:
I20260709 12:13:43.558598 12489 rpcz_store.cc:276] 0709 12:13:41.486487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:41.487115 (+   628us) service_pool.cc:224] Handling call
0709 12:13:41.487452 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:41.487665 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:41.487755 (+    90us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:42.806650 (+1318895us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:42.807573 (+   923us) log.cc:844] Serialized 26 byte log entry
0709 12:13:43.061519 (+253946us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:43.068289 (+  6770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:43.105294 (+ 37005us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:43.447974 (+342680us) spinlock_profiling.cc:243] Waited 9.83 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:43.557695 (+109721us) raft_consensus.cc:1685] finished
0709 12:13:43.557719 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:43.557911 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109836288,"thread_start_us":253447,"threads_started":1,"wal-append.queue_time_us":250020}
I20260709 12:13:43.489398 17285 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:13:43.490037 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88162.6us
I20260709 12:13:43.560968 17483 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:43.562009 17483 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:43.562580 17483 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:43.562887 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:43.563645 17483 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.564939 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:43.490147 17645 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:13:43.490446 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.491127 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:43.491477 17703 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.492900 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:43.493436 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133029us
I20260709 12:13:43.467278 17727 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:13:43.493755 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68921.8us
I20260709 12:13:43.479758 17690 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.494472 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:43.463438 17340 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:43.499259 17716 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.451438 17669 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:13:43.500339 17556 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:13:43.502521 17743 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:13:43.585978 12644 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 12:13:43.586244 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77242.6us
I20260709 12:13:43.500705 17692 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.586700 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.588560 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.589160 12497 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:43.590806 12495 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:43.591249 12493 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:43.591918 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.592276 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.592773 12496 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:43.592484 17340 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.592870 12494 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:43.504930 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:43.594144 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89396.2us
I20260709 12:13:43.594835 17340 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.594931 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:43.497081 17640 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:43.517345 17647 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:43.545677 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:43.580322 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.581034 17727 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:43.585050 17705 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:43.585376 12568 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:43.588749 17692 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.589463 12492 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:43.503713 17456 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.589448 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.605434 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:43.540622 17227 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.576229 17564 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:43.622597 12563 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:43.581018 17657 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:43.633095 12489 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:43.665337 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:43.667604 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110917us
I20260709 12:13:43.668186 12567 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:43.668208 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.665337 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:43.613523 17689 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:43.667639 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:43.613523 17600 raft_consensus.cc:2764] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:13:43.655364 17227 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:43.654901 17657 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.658844 17477 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:43.669891 17596 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:43.612100 17408 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
W20260709 12:13:43.698215 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 126588us
I20260709 12:13:43.623492 17706 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:13:43.687716 17690 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:43.713047 12571 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:43.717351 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.721275 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:43.724474 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:43.694031 17657 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:43.693949 17477 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.693984 17564 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:43.727986 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114132us
I20260709 12:13:43.730842 12565 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:13:43.731945 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52588.8us
I20260709 12:13:43.727779 17596 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:43.743002 17596 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.748234 12566 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:43.754815 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.721844 17692 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.750452 17690 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:43.758493 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61385.1us
I20260709 12:13:43.759645 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:43.750219 17748 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:43.762202 17748 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.764544 17748 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.766781 17408 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:13:43.740691 17477 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:43.771081 17730 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.701s	user 0.009s	sys 0.004s
I20260709 12:13:43.775558 17340 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:43.776007 17753 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:43.753335 17285 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:43.754345 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54945.4us
I20260709 12:13:43.779934 12571 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:13:43.782253 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49385.7us
I20260709 12:13:43.765403 17748 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:43.790948 12565 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 12:13:43.761410 17755 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:43.807034 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55573.9us
W20260709 12:13:43.798131 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.791503 12565 pending_rounds.cc:85] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:13:43.810014 12565 pending_rounds.cc:107] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 12:13:43.810451 12565 raft_consensus.cc:2889] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:43.814108 17751 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:43.814857 17751 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.815337 17716 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:43.816057 17285 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.776185 17340 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:43.820647 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.808020 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:43.821921 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 196072us
I20260709 12:13:43.822458 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.822721 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:43.824843 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.822839 17764 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:43.831282 12563 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:43.818955 17285 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:43.837380 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76052.7us
I20260709 12:13:43.823591 17748 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:43.805029 17755 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.786866 17692 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.849964 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:43.850209 17614 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:43.850751 12571 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:43.852137 17757 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:43.854141 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:43.854423 12569 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:43.855782 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:43.857954 17547 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:43.858685 17547 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.860288 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:43.820770 17703 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:43.827440 17417 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:13:43.831060 17323 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:13:43.864120 17323 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:43.864251 17417 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:43.834904 17716 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:43.865269 17716 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:43.858096 17745 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:13:43.861903 17226 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.861804 17703 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.815361 17751 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:43.781394 17727 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.845695 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:43.826964 17764 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:43.892084 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69453.3us
I20260709 12:13:43.861411 17547 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:43.915059 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:43.891862 12492 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:43.870558 17716 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:43.873929 17614 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:43.922463 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:43.759701 17690 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:43.921010 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:43.819399 17340 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:43.928776 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.929024 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:43.945973 12567 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:43.866124 17755 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:43.871685 17417 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:43.936676 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:43.923406 17323 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:43.866073 17669 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
W20260709 12:13:43.952373 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77800.5us
W20260709 12:13:43.954586 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97124.1us
I20260709 12:13:43.958823 13259 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: []
I20260709 12:13:43.926328 17764 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:43.981683 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45125.1us
I20260709 12:13:43.941571 17456 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:43.996187 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:44.002246 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:43.960996 17323 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:43.960749 17547 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:43.980583 17645 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:44.019692 17645 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:44.020131 17645 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:44.021174 17749 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:13:44.021445 17387 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.022188 17759 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:44.022328 17387 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:44.024720 17741 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.546s	user 0.002s	sys 0.002s
I20260709 12:13:43.956796 17614 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:44.026620 17614 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.029354 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.030877 17397 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:44.033970 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.013885 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.040520 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.044730 17432 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:44.018970 17761 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.019188 17556 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:44.047062 17668 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:44.049715 17748 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.053464 17703 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:44.054394 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62034us
I20260709 12:13:44.057649 17417 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.058414 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.060855 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.061787 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.062613 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.063421 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.064409 17477 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.014919 17323 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:44.065243 17647 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:44.016194 12563 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:13:44.071156 17556 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:44.069610 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:44.100147 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.015285 17547 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:44.107023 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.147217 12567 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
W20260709 12:13:44.115275 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:44.118760 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
W20260709 12:13:44.132296 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 149500us
I20260709 12:13:44.098757 17340 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:44.151494 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:44.070200 17432 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:44.158804 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 198766us
I20260709 12:13:44.159125 12642 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:44.160372 12572 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:44.163269 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:44.104650 17759 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:44.174556 17702 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:44.175508 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.152846 17547 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:44.187002 12641 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:13:44.139200 17647 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:44.187772 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.124953 17564 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:44.199123 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68010.9us
I20260709 12:13:44.187350 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.201232 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.210213 12640 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:44.213253 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.065606 17764 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.211673 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.216553 12563 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:44.220067 13253 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:44.200179 17702 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:44.100986 17761 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.218722 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.224339 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.228797 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:44.232246 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:44.225044 13258 raft_consensus.cc:2376] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14. Re-sending same reply.
I20260709 12:13:44.235646 12571 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:13:44.278653 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55112.8us
W20260709 12:13:44.278946 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85323.6us
I20260709 12:13:44.279132 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:44.279228 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.280119 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:44.280607 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:13:44.279153 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.281394 13259 raft_consensus.cc:606] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:44.282032 13259 raft_consensus.cc:614] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:44.285023 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 9.
I20260709 12:13:44.283228 12563 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:13:44.289047 13257 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16.
I20260709 12:13:44.281549 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:44.293817 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:44.203419 17757 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:13:44.303582 12572 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 12:13:44.306854 12572 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:13:44.307215 12572 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:13:44.307452 12572 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:44.309115 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.237257 17483 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.230532 12564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32884
I20260709 12:13:44.311625 12564 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:13:44.312297 12564 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:44.313944 17742 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.599s	user 0.004s	sys 0.007s
I20260709 12:13:44.314627 17766 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:44.315407 17766 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:44.316541 17766 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.317780 17684 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.318517 17684 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.309993 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:44.294459 17476 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:44.335626 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:44.317823 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.292693 17706 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:13:44.349376 12572 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:13:44.340742 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:44.343818 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183228us
W20260709 12:13:44.357117 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 232737us
W20260709 12:13:44.359084 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136474us
I20260709 12:13:44.340438 17476 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:44.373322 17657 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.002s
I20260709 12:13:44.374262 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.375285 12644 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:44.374423 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.373971 17657 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.293721 17752 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:13:44.388943 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.389848 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.394248 17760 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:44.380617 17759 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.320241 17770 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.078s
W20260709 12:13:44.409298 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 246114us
I20260709 12:13:44.409677 12642 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:44.392738 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:44.394080 17762 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:13:44.372882 17476 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.415642 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.317865 17773 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
I20260709 12:13:44.415773 12643 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 11.
I20260709 12:13:44.407543 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.302404 17556 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.424924 17556 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:44.426121 17774 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:13:44.426465 17645 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.429234 17645 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.423925 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.435094 12645 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:44.430150 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.394969 17760 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.408854 17600 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:44.354108 17483 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:44.508675 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69937.5us
I20260709 12:13:44.430151 17685 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:44.518239 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:44.520222 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135258us
W20260709 12:13:44.463176 17716 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.512s	user 0.009s	sys 0.004s
I20260709 12:13:44.367986 17564 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.522639 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.566176 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.575738 17760 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.380581 17647 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:44.584626 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 274201us
I20260709 12:13:44.594133 17777 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:13:44.594982 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:44.386843 17432 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.460888 17476 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:44.568197 17785 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:44.603075 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 76236.9us
I20260709 12:13:44.452597 17600 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:44.604943 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.643221 12645 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:44.618330 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:44.643683 12638 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:13:44.647039 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:44.623368 12564 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:44.599617 17751 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.675s	user 0.015s	sys 0.000s
I20260709 12:13:44.655555 12572 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:44.658507 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.659443 12646 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:44.608019 17482 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.105s	user 0.025s	sys 0.033s
I20260709 12:13:44.607250 17684 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:44.643018 17755 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.003s
I20260709 12:13:44.675419 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:44.613722 17716 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:44.668747 17482 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.159s	user 0.025s	sys 0.033s
I20260709 12:13:44.661010 17432 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:44.686612 12564 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:44.691053 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89516.6us
I20260709 12:13:44.686724 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.664331 17778 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.700297 17600 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.701946 17600 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:44.704454 17387 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.704766 12643 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:44.705129 12643 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:44.705848 12643 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.706897 12643 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:44.708564 17639 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:44.695431 17323 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.532s	user 0.015s	sys 0.000s
I20260709 12:13:44.714622 17323 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.716953 17323 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:44.719870 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2539) took 1027 ms. Trace:
I20260709 12:13:44.720381 12569 rpcz_store.cc:276] 0709 12:13:43.692622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:43.713184 (+ 20562us) service_pool.cc:224] Handling call
0709 12:13:44.719668 (+1006484us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:44.719802 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296610176}
I20260709 12:13:44.665081 17751 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.723359 17226 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.700922 17483 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.684715 17785 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:44.736136 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74938.4us
I20260709 12:13:44.696048 17684 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:44.759562 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.674132 17755 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.766181 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:44.766434 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.587s (SENT) [suppressed 44 similar messages]
I20260709 12:13:44.767704 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:44.753929 17778 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.767143 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.729816 17759 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:44.771199 12644 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:44.771685 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:44.769536 13257 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:44.778595 12637 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:13:44.787840 12643 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:13:44.788357 12643 pending_rounds.cc:85] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 7
I20260709 12:13:44.788635 12643 pending_rounds.cc:107] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:13:44.788919 12643 raft_consensus.cc:2889] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:44.763475 17761 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.795037 12488 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:13:44.768857 17785 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.799618 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:44.804589 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:44.781153 17751 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.798075 17645 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:44.773772 17684 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.808801 17596 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.808177 17323 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 12:13:44.818540 17417 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:44.813462 17802 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:44.852986 17705 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.858044 17705 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.860917 17705 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:44.862021 17786 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:44.862876 17786 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.866246 17786 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:44.870533 17716 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.872532 17778 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:44.879338 17227 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.182s	user 0.015s	sys 0.000s
I20260709 12:13:44.879873 17227 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:44.884346 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71412.6us
I20260709 12:13:44.886617 17786 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:44.887845 17802 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:44.889114 17786 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:13:44.890861 17797 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:13:44.847741 17596 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.894408 17596 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:44.895522 17755 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:44.813856 17785 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:44.841717 17793 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:44.902282 17793 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.903504 17785 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:44.904687 17793 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:44.888006 17792 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:13:44.905418 17684 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:44.842137 17417 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:44.908066 17417 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:44.917606 17787 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.329s	user 0.000s	sys 0.002s
W20260709 12:13:44.919716 17781 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.331s	user 0.000s	sys 0.002s
W20260709 12:13:44.917313 17547 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.698s	user 0.009s	sys 0.006s
I20260709 12:13:44.921582 17547 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:44.925098 17547 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:44.926198 17477 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:44.932377 17668 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.831s	user 0.012s	sys 0.000s
I20260709 12:13:44.937834 12645 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:44.965847 12395 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 8, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:13:44.968173 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:44.939179 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:44.942770 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:44.946446 17340 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.001s
W20260709 12:13:44.946274 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 311995us
I20260709 12:13:44.948297 17764 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.975602 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:44.978240 12563 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:44.948493 12643 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:13:44.887384 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:44.950264 12637 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:13:44.952078 12646 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:44.952255 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.953469 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:44.955721 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:44.956175 17640 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.957244 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:44.959308 17757 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:44.960302 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:44.960686 17702 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.666s	user 0.013s	sys 0.000s
I20260709 12:13:44.963783 17227 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:44.975884 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:44.978084 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:44.989369 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:44.998181 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.000157 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:45.007447 12638 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:45.007625 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:44.978315 17005 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:44.969305 12395 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:45.017196 13258 raft_consensus.cc:2368] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
W20260709 12:13:45.031006 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 264605us
I20260709 12:13:44.988076 17668 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:45.042042 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.007108 17432 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.047504 12644 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260709 12:13:45.047027 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159727us
I20260709 12:13:45.012384 17689 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:45.050936 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.018373 17706 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:45.030876 17547 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:45.007714 17600 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:13:45.037014 17668 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:45.010959 17702 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.082185 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:45.046598 17764 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:45.040159 17476 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:13:45.032977 17005 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.077054 17684 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.007000 17757 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.077265 17805 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.017787 17640 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.082048 17668 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:45.075323 17432 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.087800 17476 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:45.090922 17692 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.111819 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:45.095160 17764 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:45.121826 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 146144us
I20260709 12:13:45.087529 17689 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:45.127765 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48836.2us
I20260709 12:13:45.125284 17688 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 12:13:45.095715 17779 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:45.098953 17766 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:45.107921 17793 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:45.085492 17547 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.133973 17476 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:45.159637 17702 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.096745 17684 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.163237 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:45.170792 12462 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16.
I20260709 12:13:45.171305 12462 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:13:45.152092 17692 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:13:45.170830 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.148808 17716 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:45.174479 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2602) took 1120 ms. Trace:
I20260709 12:13:45.164510 17684 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.131904 17417 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.204237 12572 rpcz_store.cc:276] 0709 12:13:44.054585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:44.364048 (+309463us) service_pool.cc:224] Handling call
0709 12:13:44.374058 (+ 10010us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:44.667073 (+293015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:45.174188 (+507115us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:45.174406 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641510272}
I20260709 12:13:45.209993 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.169109 17779 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:45.154024 17766 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.213156 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:45.135522 17805 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.212073 17790 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:45.216859 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.219978 12639 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:13:45.205220 17716 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:45.176528 17702 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.224789 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.212309 17639 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:45.228643 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58000.8us
I20260709 12:13:45.229331 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:45.231859 17432 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.219746 17408 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:45.224229 17547 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.219430 17684 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:45.239195 17766 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.241899 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:45.223591 17716 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:45.247802 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:45.253183 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.254681 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:45.249936 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.253196 13261 raft_consensus.cc:2376] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14. Re-sending same reply.
I20260709 12:13:45.257261 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.227624 17778 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:13:45.259071 17817 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:13:45.268133 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:45.253399 17685 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.249478 17779 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:45.230196 17689 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.251924 17408 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:45.272233 17645 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:45.273422 17784 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.690s	user 0.002s	sys 0.001s
I20260709 12:13:45.270742 17758 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:13:45.261313 17564 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:45.283214 17564 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:45.283865 17564 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:45.268991 17227 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:45.265738 17805 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.277851 17639 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:45.290685 17805 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.281558 17685 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.291299 17805 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.230777 17702 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.225046 17764 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.286723 17788 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:45.287091 17775 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.707s	user 0.003s	sys 0.000s
I20260709 12:13:45.301507 17788 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:45.302168 17788 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:45.305384 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.289839 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:45.299247 17657 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.274396 17789 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:13:45.288594 17793 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:45.314095 13262 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:45.271193 17476 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.309474 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.317523 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 10.
I20260709 12:13:45.310081 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.319124 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:45.310290 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.319954 12646 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:45.311065 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.274708 17547 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.021s
W20260709 12:13:45.325985 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:45.311688 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.282248 17645 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:45.311774 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.324983 17005 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.312062 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.327013 12637 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:45.318558 12641 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:45.294801 17640 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.311578 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.294595 17758 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:13:45.344797 12490 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 9. Current term is 10. Ops: []
I20260709 12:13:45.342343 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.346201 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.342733 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.314394 17793 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:45.327628 17706 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:45.346956 12492 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:45.340833 17805 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:45.354094 17820 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:13:45.355964 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.356326 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 668) took 1529 ms. Trace:
I20260709 12:13:45.356807 12494 rpcz_store.cc:276] 0709 12:13:43.826775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:43.828778 (+  2003us) service_pool.cc:224] Handling call
0709 12:13:43.842305 (+ 13527us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:43.873088 (+ 30783us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:43.873165 (+    77us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:44.044178 (+171013us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b1800434b00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:45.129309 (+1085131us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:45.130551 (+  1242us) log.cc:844] Serialized 52 byte log entry
0709 12:13:45.210539 (+ 79988us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:45.223121 (+ 12582us) log.cc:844] Serialized 14 byte log entry
0709 12:13:45.223366 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:45.223433 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:45.356087 (+132654us) raft_consensus.cc:1685] finished
0709 12:13:45.356107 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:45.356262 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14139,"raft.queue_time_us":8768,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":36073,"spinlock_wait_cycles":64064384,"thread_start_us":79460,"threads_started":1,"wal-append.queue_time_us":127762}
I20260709 12:13:45.356026 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.359382 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.362270 17792 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:45.362766 17792 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:45.363802 17792 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.365200 17792 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:45.378134 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 666) took 1552 ms. Trace:
I20260709 12:13:45.392012 12488 rpcz_store.cc:276] 0709 12:13:43.826042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:43.826247 (+   205us) service_pool.cc:224] Handling call
0709 12:13:43.837143 (+ 10896us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:44.798398 (+961255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:45.377774 (+579376us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:45.378044 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2191405440}
I20260709 12:13:45.379285 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.338353 17417 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:45.341043 17790 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16.
I20260709 12:13:45.417584 12646 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:45.426970 12638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:58586
I20260709 12:13:45.427802 12638 raft_consensus.cc:606] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:45.428143 12638 raft_consensus.cc:614] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:45.434046 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.441720 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.442415 12496 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:45.442610 12490 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:45.444589 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.445290 17668 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.448635 12496 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:45.448788 17793 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.454668 17793 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:45.456411 17753 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:45.456614 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 807) took 1739 ms. Trace:
W20260709 12:13:45.457664 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 720209us
I20260709 12:13:45.458921 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:45.459862 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:45.460125 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.462225 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.462455 17747 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:45.462921 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:45.458096 17483 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:13:45.464361 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.465029 12572 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:45.465085 17483 raft_consensus.cc:2764] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:13:45.378316 17685 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.466323 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.466598 17808 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:45.467273 12493 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:13:45.467427 17808 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.468735 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.469305 12570 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:45.469848 17340 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.469964 17748 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:45.470546 17340 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.471628 17748 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:13:45.472751 17340 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.469504 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:45.473697 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.474400 12567 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:45.474952 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:45.476553 17815 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.321s	user 0.001s	sys 0.000s
W20260709 12:13:45.482340 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:45.483157 17614 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:45.483737 17614 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:45.484129 17614 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:45.463259 12490 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:45.476637 12569 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:45.486490 17285 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:45.487530 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.488227 12494 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:45.449807 17668 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.489403 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2606) took 1325 ms. Trace:
I20260709 12:13:45.489298 17703 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:45.490377 17703 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:45.489995 12568 rpcz_store.cc:276] 0709 12:13:44.164502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:44.438442 (+273940us) service_pool.cc:224] Handling call
0709 12:13:44.456882 (+ 18440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:44.831653 (+374771us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:44.831679 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:44.875461 (+ 43782us) log.cc:844] Serialized 26 byte log entry
0709 12:13:45.093385 (+217924us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:45.113885 (+ 20500us) log.cc:844] Serialized 14 byte log entry
0709 12:13:45.489052 (+375167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:45.489137 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:45.489183 (+    46us) raft_consensus.cc:1685] finished
0709 12:13:45.489198 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:45.489329 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":374898,"prepare.queue_time_us":27994,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":73844,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":5445,"thread_start_us":585172,"threads_started":2,"wal-append.queue_time_us":243236}
I20260709 12:13:45.491914 17703 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.493505 17703 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:45.466323 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.506630 12640 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:45.467929 17808 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.511536 17757 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.463174 17747 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.512060 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:45.504773 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.542528 12639 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:13:45.457576 12566 rpcz_store.cc:276] 0709 12:13:43.717205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:43.749538 (+ 32333us) service_pool.cc:224] Handling call
0709 12:13:44.231210 (+481672us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:13:44.282495 (+ 51285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:44.309690 (+ 27195us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:13:44.354649 (+ 44959us) log.cc:844] Serialized 14 byte log entry
0709 12:13:44.662930 (+308281us) log.cc:844] Serialized 14 byte log entry
0709 12:13:44.950374 (+287444us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b18003be420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:45.456055 (+505681us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b18003be420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:45.456251 (+   196us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:45.456303 (+    52us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:13:45.456347 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:45.456415 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:45.456549 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":769262,"prepare.queue_time_us":118835,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":103,"raft.queue_time_us":805657,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":14264,"spinlock_wait_cycles":24860928,"thread_start_us":277232,"threads_started":1,"wal-append.queue_time_us":358586}
I20260709 12:13:45.550544 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.551362 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.553449 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:45.549516 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.562096 12570 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:45.564316 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.567878 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:45.565313 12570 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:45.569401 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.571154 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:45.539539 17690 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.958s	user 0.018s	sys 0.012s
I20260709 12:13:45.573210 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.574568 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.544782 17482 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:45.578037 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.581430 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.544350 17556 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.549676 17340 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:45.584506 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.588778 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:45.585035 12494 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:13:45.573943 17690 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.019s	sys 0.012s
W20260709 12:13:45.599463 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 261379us
I20260709 12:13:45.600880 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.605551 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.609758 12640 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:13:45.392957 17408 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.603864 17556 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.630569 12494 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:45.602350 17340 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:45.636704 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.638383 12567 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:45.641924 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.648552 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.650863 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.652774 12566 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:45.649832 13256 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:45.655148 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.656966 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:45.628677 17476 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.641461 17340 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:45.638675 17808 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.672814 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:45.674643 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.674036 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.676046 12640 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:45.593353 17482 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.682367 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:45.660204 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2063297189999957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
I20260709 12:13:45.683665 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.683882 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.685359 13253 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:13:45.686038 12490 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:45.688020 12494 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
W20260709 12:13:45.685986 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1127297190000718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.168s for a response
I20260709 12:13:45.688902 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:45.689246 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0857297190000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
I20260709 12:13:45.690006 12568 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:45.690188 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:45.700385 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:45.702593 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.670768 17702 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:45.709015 12490 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [16.7-16.7]
I20260709 12:13:45.683180 17799 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:45.712935 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.713322 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:45.714280 17285 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:13:45.714649 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.706501 17483 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:45.715938 17285 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:45.716323 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.717375 12570 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:45.731045 17556 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.709860 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:45.733234 17799 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:45.734753 17799 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:45.714848 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:45.743685 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:45.744638 13261 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:45.746654 12638 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:45.758559 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:45.728820 17564 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.703114 17808 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:45.762823 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.766119 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:45.755581 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.695781 17716 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.768827 17689 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.760288 17556 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:45.754379 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:45.708941 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:45.776793 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.780627 12639 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:45.757516 17753 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:45.775995 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.747197 17684 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:13:45.784251 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:45.786322 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:45.789450 12494 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:45.706187 17788 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.775115 17690 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.828560 17690 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:45.826342 17828 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:13:45.775352 17748 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.799651 12489 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:45.832170 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.826325 17805 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:45.827827 17716 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:45.832005 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132163us
I20260709 12:13:45.775223 17285 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:45.757268 17747 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.833425 12488 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:45.783234 17556 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:45.862643 17829 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:13:45.862445 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.831928 17639 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:45.876590 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:45.876874 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.877311 12563 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:45.863593 17748 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:45.880287 17748 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:45.880894 17748 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:13:45.883857 12572 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:45.877620 12568 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:13:45.889025 12563 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:45.834336 17647 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:45.892861 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.886701 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:45.895452 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.832847 17790 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:45.896819 12563 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:45.887955 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:45.898620 12572 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:45.777173 17645 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.903338 12646 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260709 12:13:45.905305 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107096us
I20260709 12:13:45.870153 17556 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:45.907533 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75476.1us
I20260709 12:13:45.910240 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.929982 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:45.928920 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:45.786242 17689 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.938912 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:45.897411 17790 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:45.940944 12495 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:45.933502 12572 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [16.7-16.7]
I20260709 12:13:45.950129 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:45.897760 17766 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:45.890961 17747 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:13:45.884050 17285 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:45.960114 12563 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:45.932745 17769 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:45.962666 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.900430 17824 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:45.939839 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:13:45.933017 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 141055us
W20260709 12:13:45.983528 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:45.970422 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:45.950948 17706 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:45.953533 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47150.3us
I20260709 12:13:45.934594 17684 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:45.986282 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:45.988343 12572 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:45.956173 17790 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:45.959584 17690 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.950244 17805 consensus_queue.cc:1243] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:13:45.941900 17600 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:13:45.942023 17456 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:13:45.973049 17668 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:45.985675 17685 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:45.986790 17788 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:46.006632 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:45.990520 17706 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:46.008364 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:46.015892 12564 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:13:46.018280 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:46.024521 17689 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:46.025931 17830 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
W20260709 12:13:46.027189 17832 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:13:46.028980 17831 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:13:46.029878 17476 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:13:46.030414 17476 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:46.030833 17476 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:46.031251 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:46.031389 12565 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:13:46.034255 17482 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:45.968650 17657 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:13:45.995169 17824 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:46.052235 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:46.053416 12641 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:13:46.038123 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107672us
I20260709 12:13:45.965391 17766 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.058924 12567 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:13:46.060247 12643 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:13:46.060654 12643 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 10
I20260709 12:13:45.996522 17792 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:46.061656 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.061705 17792 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:13:46.062302 17792 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:46.062497 12492 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:46.063702 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.063482 17792 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.040477 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:45.968937 17799 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.067684 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:46.068360 12644 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:13:46.068528 17799 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.064033 17639 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:46.056699 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.057898 12497 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:13:46.060873 12643 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 12:13:46.087604 12643 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:46.066314 17827 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.319s	user 0.002s	sys 0.000s
I20260709 12:13:46.044324 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.060879 17766 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:46.064574 17788 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:46.071455 17837 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.104s	user 0.000s	sys 0.003s
I20260709 12:13:46.073026 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:46.065368 17792 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:13:46.080852 17614 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.596s	user 0.009s	sys 0.003s
I20260709 12:13:46.081012 17764 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:46.064814 17689 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:46.065596 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.064792 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:46.082901 17685 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:46.083811 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:46.170753 13253 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:46.064515 12492 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:46.035946 17684 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:46.092644 12495 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12.
I20260709 12:13:46.019448 17706 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:46.101691 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57398.3us
I20260709 12:13:46.102838 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:46.177528 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74755.5us
W20260709 12:13:46.103507 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 311660us
I20260709 12:13:46.091653 17808 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:46.112206 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:46.080915 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:46.143489 12564 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:46.091557 17564 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:46.058924 17483 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:46.148949 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210118us
I20260709 12:13:46.167109 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.190788 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:46.171454 12643 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:13:46.168521 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:46.188392 12645 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:13:46.190748 12536 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:13:46.173272 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.192039 12536 leader_election.cc:403] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260709 12:13:46.195775 12640 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.510s	user 0.006s	sys 0.005s
I20260709 12:13:46.196501 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:46.200067 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:46.173274 17793 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.718s	user 0.014s	sys 0.000s
I20260709 12:13:46.094754 17639 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.204169 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.208230 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.204584 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:46.145321 17764 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:46.209137 12491 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:46.208683 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:46.216115 12637 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:13:46.217442 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.218166 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:46.219043 12644 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:46.223896 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:46.144960 17788 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.197585 17716 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:46.147418 17614 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.226480 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.208773 17839 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11.
I20260709 12:13:46.205355 17766 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:46.249555 12495 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:13:46.229660 13253 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:46.236510 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.236604 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.236826 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:46.227123 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:46.174247 17689 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.255075 12489 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:13:46.190500 17799 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:13:46.190272 17482 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:13:46.252089 12492 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:46.198549 17808 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:46.258337 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76856.1us
I20260709 12:13:46.253564 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:46.207707 17600 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:13:46.261128 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:46.261569 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 285495us
I20260709 12:13:46.262192 12644 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:46.228996 17668 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:46.241806 17847 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:46.225863 17793 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:13:46.280143 12497 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:13:46.283174 12644 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:13:46.269595 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.270263 17408 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:46.260911 17227 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:46.259487 17851 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:46.283982 17483 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:46.294684 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:46.301854 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.302978 12497 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:13:46.304056 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.305738 17227 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.308883 17227 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.309970 17483 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:46.310256 17600 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:46.311290 17483 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.311319 17600 consensus_queue.cc:1243] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:46.282948 17657 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:46.313215 17600 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:46.313531 17657 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.316855 17227 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:46.317754 17227 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:46.319119 17227 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.325438 17706 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.203841 17841 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
W20260709 12:13:46.331501 17703 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.835s	user 0.013s	sys 0.004s
I20260709 12:13:46.327368 17689 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.327981 17645 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.275692 17684 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.329438 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:46.333901 13253 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:13:46.335309 12395 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 8, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:46.337723 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:13:46.338094 17808 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:46.339020 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:46.344897 17706 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:46.346967 17748 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:46.347666 17748 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.349108 17841 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:46.292361 17340 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.003s
I20260709 12:13:46.349855 17841 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.350535 17340 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.350977 17285 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.299563 12489 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:13:46.357735 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:46.352828 13253 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:46.328241 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.353735 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:46.349195 17835 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:13:46.292285 17408 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:46.396507 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68320.4us
I20260709 12:13:46.224264 17824 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.345108 17564 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:13:46.350039 17556 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.353029 17858 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:46.298871 17417 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.103s
I20260709 12:13:46.401263 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.436239 12491 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:46.434248 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.438778 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:46.260681 17764 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.428594 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:46.396867 17851 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:13:46.414984 17844 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
W20260709 12:13:46.411051 17842 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
W20260709 12:13:46.410085 17838 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
W20260709 12:13:46.410576 17843 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:13:46.444270 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:46.405819 17408 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:46.412231 17753 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:46.449600 13259 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:46.389297 17716 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:46.451049 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.406051 17859 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:46.418990 17788 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.410699 17841 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:46.456868 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 267451us
W20260709 12:13:46.461104 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 193027us
I20260709 12:13:46.500020 12563 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:13:46.450956 17851 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:46.420493 17790 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.411214 17668 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.489444 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:46.454895 17748 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:46.452105 17852 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:46.468084 12497 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:46.501739 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 220104us
W20260709 12:13:46.487761 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43622.9us
I20260709 12:13:46.518177 12568 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:13:46.526373 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:46.489748 17753 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:46.529676 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 258807us
I20260709 12:13:46.498929 17859 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:46.487142 17340 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:46.544400 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.545617 12495 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:13:46.549360 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.528456 17639 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:46.544795 17647 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.645s	user 0.014s	sys 0.000s
I20260709 12:13:46.532930 17841 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:46.523243 17866 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:46.566519 12496 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:13:46.569458 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:46.574033 12488 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:13:46.539299 17861 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:46.569607 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:46.554666 17476 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.004s
I20260709 12:13:46.546597 17859 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:46.582194 12490 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:46.588863 12395 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 7 to 8, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:46.585777 12642 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:13:46.596184 12642 pending_rounds.cc:85] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 10
I20260709 12:13:46.596468 12642 pending_rounds.cc:107] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 12:13:46.596760 12642 raft_consensus.cc:2889] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:13:46.610613 17845 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:13:46.611720 17685 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:46.615317 17793 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:46.615953 17793 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:46.636093 12497 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:13:46.594990 17614 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:46.579846 17841 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:46.580708 17866 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:46.554001 17863 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
I20260709 12:13:46.666703 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2557) took 1040 ms. Trace:
I20260709 12:13:46.655212 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:46.664597 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:46.660015 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:46.674794 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:46.675246 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:46.677197 17861 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:46.629558 17716 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.678403 17456 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:46.632061 17854 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:46.674003 12489 raft_consensus.cc:2376] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11. Re-sending same reply.
I20260709 12:13:46.617080 17793 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.673342 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:46.657902 17841 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:46.696007 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:46.595433 17476 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.699046 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:46.796073 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97123.9us
I20260709 12:13:46.797266 17769 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:46.797667 12491 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: [17.14-17.14]
W20260709 12:13:46.701321 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 172415us
I20260709 12:13:46.704054 17456 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:46.668109 17866 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:46.652621 17788 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:13:46.652499 17747 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:13:46.681195 17808 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:46.807268 17808 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:46.807861 17808 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:46.739521 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:46.657763 17647 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:13:46.765141 17340 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:46.666174 17766 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:13:46.697772 17849 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:46.680446 17870 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:46.791600 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 242312us
I20260709 12:13:46.835343 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:46.792241 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 256137us
I20260709 12:13:46.690627 17799 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:46.821810 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 125912us
I20260709 12:13:46.822057 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:46.679195 17614 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:13:46.825676 12494 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:46.686100 17871 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:46.797154 17476 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:46.838276 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:46.858860 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:46.859809 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:46.687620 17703 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20260709 12:13:46.687340 17645 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 12:13:46.905226 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:46.906093 12569 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:46.870183 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2425) took 2194 ms. Trace:
I20260709 12:13:46.906402 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:46.907058 12642 rpcz_store.cc:276] 0709 12:13:44.675888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:44.676102 (+   214us) service_pool.cc:224] Handling call
0709 12:13:44.690648 (+ 14546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:46.630952 (+1940304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:46.849925 (+218973us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:46.870074 (+ 20149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4346405120}
I20260709 12:13:46.787220 17852 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:46.826364 17867 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:46.875576 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:46.878625 12566 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:13:46.855306 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:46.850818 17870 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:46.934808 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79587.5us
W20260709 12:13:46.900892 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62693.4us
I20260709 12:13:46.788705 17689 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:46.817417 17793 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:46.665172 17851 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:46.921005 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 394687us
I20260709 12:13:46.686231 17854 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.912562 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:46.850378 17340 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:46.939633 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:46.982817 12611 leader_election.cc:400] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:47.058313 12611 leader_election.cc:403] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:13:46.947144 17871 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:13:46.955700 17792 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.006s
I20260709 12:13:46.953336 17790 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:47.014734 17864 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.357s	user 0.001s	sys 0.000s
I20260709 12:13:46.946552 17600 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.115s
W20260709 12:13:47.075699 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 120100us
W20260709 12:13:47.068233 17847 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.772s	user 0.009s	sys 0.005s
W20260709 12:13:47.018345 17874 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.330s	user 0.001s	sys 0.000s
I20260709 12:13:47.128506 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:47.025485 17861 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:47.012923 17851 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:47.072203 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:47.075431 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135864us
I20260709 12:13:47.025104 17753 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:46.908715 17684 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:47.001860 17227 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.062178 17871 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:47.030743 17805 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:47.090489 13257 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:13:47.181679 13257 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 9
I20260709 12:13:47.025104 17866 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:47.030604 17873 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:13:47.091380 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:47.018204 17852 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 12.4, Last appended by leader: 4, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:47.188913 17859 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.558s	user 0.007s	sys 0.002s
I20260709 12:13:47.079481 17881 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14.
I20260709 12:13:47.092361 17600 consensus_queue.cc:1243] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 17.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:13:47.092379 12638 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:13:46.942380 17870 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:47.091686 12490 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:47.019006 17888 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:13:46.983597 17863 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.109s
W20260709 12:13:47.149037 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 213249us
I20260709 12:13:47.107889 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:47.277000 13256 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:47.107161 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:47.037712 17408 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.505s	user 0.013s	sys 0.000s
I20260709 12:13:47.155498 13254 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:47.159898 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:47.160249 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 742) took 1068 ms. Trace:
W20260709 12:13:47.151604 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 306844us
I20260709 12:13:47.079182 17858 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:47.124893 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:47.176383 13258 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:47.181964 13257 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 12:13:47.156364 17861 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:47.162011 17847 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:47.187652 17866 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:47.190585 12570 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:47.420485 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:47.102792 17824 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.000s	sys 0.015s
W20260709 12:13:47.194949 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103625us
I20260709 12:13:47.181119 17871 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:47.217301 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:47.278765 13260 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:47.329958 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:47.448813 12495 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:47.330706 17600 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:47.332939 17753 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:47.334611 17890 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:13:47.338892 17340 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:47.343034 12493 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:47.344251 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:47.495326 12497 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:47.133507 17894 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
W20260709 12:13:47.351336 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 191536us
I20260709 12:13:47.182060 17884 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.345866 13261 rpcz_store.cc:276] 0709 12:13:46.092064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.092206 (+   142us) service_pool.cc:224] Handling call
0709 12:13:46.103450 (+ 11244us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:46.110385 (+  6935us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:46.110428 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:46.166345 (+ 55917us) log.cc:844] Serialized 52 byte log entry
0709 12:13:46.503920 (+337575us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:13:46.531743 (+ 27823us) log.cc:844] Serialized 14 byte log entry
0709 12:13:46.532277 (+   534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:46.532351 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:47.159925 (+627574us) raft_consensus.cc:1685] finished
0709 12:13:47.159954 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:47.160173 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":924,"prepare.queue_time_us":27840,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":1232,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":38606,"spinlock_wait_cycles":2944,"thread_start_us":286366,"threads_started":1,"wal-append.queue_time_us":364619}
I20260709 12:13:47.104408 17702 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:13:47.145977 17882 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:13:47.388136 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:47.387019 13257 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:47.392145 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:47.388473 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 247662us
I20260709 12:13:47.441401 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:47.444202 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:47.421518 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:47.189390 17859 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.420122 17861 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:47.527129 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:47.528180 12489 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:47.528065 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:47.420204 17645 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.488850 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:47.526826 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:47.384430 17408 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.181910 17227 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.384565 17881 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:47.506986 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:47.194717 17805 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.523327 12492 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:47.537305 12495 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:47.536803 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92699.8us
I20260709 12:13:47.525942 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:47.526747 17894 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:47.529464 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:47.496273 17863 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:47.440623 17824 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.016s	user 0.000s	sys 0.019s
I20260709 12:13:47.530748 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:47.490110 13259 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
W20260709 12:13:47.531288 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 139253us
I20260709 12:13:47.486732 17858 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.508255 17888 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.533407 12496 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:47.549783 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:47.561723 12497 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:47.550493 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:47.506575 17753 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:47.527005 17899 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.578666 12490 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260709 12:13:47.579567 12490 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 5
I20260709 12:13:47.579963 12490 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:13:47.580292 12490 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:47.555972 17476 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:47.556404 17894 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:47.533186 17645 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.581960 17564 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:46.671552 12572 rpcz_store.cc:276] 0709 12:13:45.626240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:45.989465 (+363225us) service_pool.cc:224] Handling call
0709 12:13:45.989772 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:45.990265 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:13:46.287193 (+296928us) log.cc:844] Serialized 14 byte log entry
0709 12:13:46.666237 (+379044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:46.666300 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:46.666346 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:46.666451 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:46.666610 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8160,"prepare.queue_time_us":508436,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":8269,"thread_start_us":664016,"threads_started":2}
W20260709 12:13:47.615763 17896 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:13:47.616137 17645 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:47.616541 17887 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.434s	user 0.001s	sys 0.001s
I20260709 12:13:47.617383 17408 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:47.629343 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:47.632544 12495 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
W20260709 12:13:47.637058 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138757us
I20260709 12:13:47.594128 17881 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:47.671470 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:47.579130 17790 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:13:47.558925 17893 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
W20260709 12:13:47.658187 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100741us
I20260709 12:13:47.578666 17753 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:47.650714 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98892.3us
I20260709 12:13:47.559252 17903 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 12:13:47.762274 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:47.767246 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.770174 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.773787 12640 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260709 12:13:47.733515 12393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 227) took 39634 ms (client timeout 15000 ms). Trace:
I20260709 12:13:47.681505 17456 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:47.778083 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.779443 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.782444 12640 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:47.783588 12395 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 8, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:13:47.788851 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:47.792047 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 261450us
I20260709 12:13:47.795827 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:47.797345 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.799537 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:47.798301 12637 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:47.805174 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.588733 17888 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:47.801553 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 126558us
I20260709 12:13:47.564383 17863 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.781478 17456 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:47.836596 12637 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:47.840193 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.830715 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.590792 17899 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.846740 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.849915 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:47.853084 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:47.515847 17884 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.841797 12643 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:47.573203 17858 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.861826 17858 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:47.843648 12395 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:47.823112 17866 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:47.856693 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:47.874185 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:47.834797 17564 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:47.876117 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:47.882082 12641 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:47.883163 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:47.883708 12641 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:47.825322 17706 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:47.876816 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 205459us
I20260709 12:13:47.880880 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:47.885812 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:47.824782 17912 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:47.893213 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97479us
I20260709 12:13:47.903120 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:47.865980 17895 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.369s	user 0.002s	sys 0.000s
I20260709 12:13:47.913393 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:47.862692 17645 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:47.876215 17866 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:47.901654 17841 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.192s	user 0.011s	sys 0.005s
I20260709 12:13:47.775916 17894 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.902137 17858 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:47.874480 17848 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:13:47.874240 17882 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:13:47.898288 17921 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:47.911247 17859 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:47.915359 17918 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:47.897130 17706 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:47.930186 17866 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:47.925109 17227 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:47.935081 17841 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.997442 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2746) took 2092 ms. Trace:
I20260709 12:13:47.997773 17841 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:47.997917 12494 rpcz_store.cc:276] 0709 12:13:45.905036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:45.905387 (+   351us) service_pool.cc:224] Handling call
0709 12:13:45.938009 (+ 32622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:46.878234 (+940225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:47.997183 (+1118949us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:47.997361 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2040756608}
W20260709 12:13:47.956418 17905 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:13:47.999151 17863 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:47.907605 17912 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.002914 17899 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:47.930554 17483 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:48.004000 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2571) took 1146 ms. Trace:
I20260709 12:13:48.004427 17483 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.004745 12568 rpcz_store.cc:276] 0709 12:13:46.857673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.876199 (+ 18526us) service_pool.cc:224] Handling call
0709 12:13:46.876576 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:46.876866 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:46.876891 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:46.877446 (+   555us) log.cc:844] Serialized 26 byte log entry
0709 12:13:47.107657 (+230211us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:47.107708 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:47.107790 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:48.003682 (+895892us) raft_consensus.cc:1685] finished
0709 12:13:48.003716 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:48.003918 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":229696,"threads_started":1,"wal-append.queue_time_us":314401}
I20260709 12:13:48.007119 17483 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.007866 17908 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:48.008551 17908 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:48.009291 17904 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
I20260709 12:13:47.936179 17914 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:48.010135 17914 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:47.777076 12393 rpcz_store.cc:269] 0709 12:13:08.098551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:08.098823 (+   272us) service_pool.cc:224] Handling call
0709 12:13:47.733206 (+39634383us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:37.402802 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:37.568677 (+165875us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:37.568711 (+    34us) write_op.cc:435] Acquired schema lock
0709 12:13:37.568728 (+    17us) tablet.cc:662] Decoding operations
0709 12:13:37.570331 (+  1603us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:37.570420 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:37.570448 (+    28us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:13:37.571747 (+  1299us) tablet.cc:701] Row locks acquired
0709 12:13:37.571770 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:13:37.571935 (+   165us) write_op.cc:270] Start()
0709 12:13:37.572095 (+   160us) write_op.cc:276] Timestamp: P: 1783599217571876 usec, L: 0
0709 12:13:37.572121 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:13:37.572922 (+   801us) log.cc:844] Serialized 6343 byte log entry
0709 12:13:37.927744 (+354822us) op_driver.cc:464] REPLICATION: finished
0709 12:13:38.029716 (+101972us) write_op.cc:301] APPLY: starting
0709 12:13:38.029828 (+   112us) tablet.cc:1367] starting BulkCheckPresence
0709 12:13:38.380171 (+350343us) tablet.cc:1370] finished BulkCheckPresence
0709 12:13:38.402855 (+ 22684us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:13:39.087934 (+685079us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:13:39.088421 (+   487us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:13:39.088495 (+    74us) write_op.cc:312] APPLY: finished
0709 12:13:39.089116 (+   621us) log.cc:844] Serialized 148 byte log entry
0709 12:13:39.360638 (+271522us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:13:39.801889 (+441251us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:39.813599 (+ 11710us) write_op.cc:454] Released schema lock
0709 12:13:39.840009 (+ 26410us) write_op.cc:341] FINISH: Updating metrics
0709 12:13:39.948344 (+108335us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:39.954294 (+  5950us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26157,"reported_tablets":51,"spinlock_wait_cycles":3840,"child_traces":[["op",{"apply.queue_time_us":101641,"apply.run_cpu_time_us":184671,"apply.run_wall_time_us":1873613,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70943,"num_ops":16,"prepare.queue_time_us":828464,"prepare.run_cpu_time_us":66762,"prepare.run_wall_time_us":420082,"raft.queue_time_us":70504,"raft.run_cpu_time_us":62227,"raft.run_wall_time_us":90890,"replication_time_us":355401,"spinlock_wait_cycles":122432896,"thread_start_us":1639598,"threads_started":5,"wal-append.queue_time_us":1039606,"wal-append.run_cpu_time_us":12431,"wal-append.run_wall_time_us":2244661}]]}
I20260709 12:13:47.959153 12395 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 9, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:48.014290 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:48.015000 17898 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.422s	user 0.000s	sys 0.000s
I20260709 12:13:47.964483 17858 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:47.965000 17917 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.964843 17476 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:47.966280 17799 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:13:47.968345 17808 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.160s	user 0.014s	sys 0.000s
W20260709 12:13:47.972034 17862 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.302s	user 0.012s	sys 0.027s
I20260709 12:13:47.952116 17847 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:13:47.978595 17910 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 12:13:47.978964 17888 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:47.942332 17907 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.983608 17706 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:47.984548 17753 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:47.940951 17456 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.989010 17884 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:47.926066 17645 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.990566 17859 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:47.990839 17928 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:47.991614 17918 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:47.992079 17911 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:13:47.992569 17920 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:47.992717 17564 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:47.949906 17921 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:48.004053 17912 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:48.020726 17793 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.034s	user 0.014s	sys 0.000s
I20260709 12:13:48.020907 17858 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.012930 17914 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:48.022557 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 333966us
I20260709 12:13:48.056574 17907 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.073400 17920 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.088920 17920 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.024603 17917 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.081430 17929 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:48.090209 17929 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.090725 12494 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:48.023854 12395 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 9, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:48.091734 17917 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.093081 17929 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.094538 12395 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 8, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:48.035754 12641 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:48.062287 17859 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.099071 17859 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.072248 17921 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.102140 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:48.102908 12490 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.086208 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2444) took 2074 ms. Trace:
I20260709 12:13:48.026018 17799 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:48.104671 12646 rpcz_store.cc:276] 0709 12:13:46.011734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.013940 (+  2206us) service_pool.cc:224] Handling call
0709 12:13:46.014430 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:46.188278 (+173848us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:46.188543 (+   265us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:46.192823 (+  4280us) log.cc:844] Serialized 26 byte log entry
0709 12:13:46.463378 (+270555us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:46.504316 (+ 40938us) log.cc:844] Serialized 14 byte log entry
0709 12:13:46.548478 (+ 44162us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:46.548565 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:47.971143 (+1422578us) spinlock_profiling.cc:243] Waited 9.31 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:48.085857 (+114714us) raft_consensus.cc:1685] finished
0709 12:13:48.085968 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:48.086117 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101629,"prepare.queue_time_us":40659,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":6198,"raft.queue_time_us":136887,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":15001,"spinlock_wait_cycles":22363648,"thread_start_us":441342,"threads_started":2,"wal-append.queue_time_us":392938}
I20260709 12:13:48.027372 17476 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:48.106460 17799 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.108651 17799 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:48.110855 17909 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:48.111333 17909 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:48.112224 17909 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:48.049996 17851 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.883s	user 0.014s	sys 0.000s
I20260709 12:13:48.050693 17847 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:48.058954 17881 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:48.051335 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 368812us
I20260709 12:13:48.025362 17848 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:48.040159 17808 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.102851 17854 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:48.055583 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 152537us
I20260709 12:13:48.116844 12496 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:13:48.118180 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.121418 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:48.096714 17871 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.002s
I20260709 12:13:48.123430 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:48.021860 17908 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.096860 12395 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 6, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 6 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:48.135875 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.136636 12497 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:48.137835 12489 raft_consensus.cc:2368] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:48.138119 12495 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:13:48.139293 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.108700 17907 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.145833 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:48.142185 12489 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:48.146552 12568 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:48.143494 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:48.147497 12565 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:48.144771 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.140203 12492 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:13:48.147969 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:48.150123 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 672) took 1904 ms. Trace:
I20260709 12:13:48.150619 12491 rpcz_store.cc:276] 0709 12:13:46.245943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.437105 (+191162us) service_pool.cc:224] Handling call
0709 12:13:46.437502 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:46.826388 (+388886us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:48.149764 (+1323376us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:48.150046 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":827588480}
I20260709 12:13:48.152156 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:48.153316 13261 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:13:48.154713 17863 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:48.155443 17863 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.135389 17848 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.159096 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:48.160105 12566 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:48.160498 17848 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.162696 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:48.164135 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:48.165009 12567 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:48.077751 17689 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:48.168973 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.170665 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.170743 12646 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:48.172127 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:48.173134 17859 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:48.173910 17859 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.170836 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.175839 17854 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:48.176151 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:48.176548 17854 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.176985 12572 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
W20260709 12:13:48.204289 12536 connection.cc:256] RPC call timeout handler was delayed by -1.094435244999886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16110000000000002s for a response
I20260709 12:13:48.123888 17909 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:48.188766 17928 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:48.192348 17863 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.142628 17871 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.195927 17689 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.132104 17408 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:48.200104 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2452) took 2097 ms (client timeout 1993 ms). Trace:
I20260709 12:13:48.200809 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:48.217497 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.218181 13262 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:48.201579 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:48.180902 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.205260 12572 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:13:48.220609 12395 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:13:48.217386 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:48.177600 12564 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:48.205391 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:48.219334 12638 rpcz_store.cc:269] 0709 12:13:46.102185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.167935 (+ 65750us) service_pool.cc:224] Handling call
0709 12:13:48.199821 (+2031886us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:48.199988 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2070510848}
I20260709 12:13:48.227035 12490 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:48.227471 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:48.228528 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.229046 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:48.186619 17886 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:48.206452 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.235359 12645 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 12:13:48.236553 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186193us
I20260709 12:13:48.221935 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.176154 17859 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:48.206256 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0593352450000566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
I20260709 12:13:48.235306 12488 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:48.208534 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.252082 12610 leader_election.cc:400] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:48.252483 12610 leader_election.cc:403] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
W20260709 12:13:48.253327 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:48.253158 17863 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:48.253893 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.254113 17863 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:48.254729 17863 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.254807 17757 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:48.256109 17757 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:48.256541 17757 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:48.254544 17899 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:48.256574 17851 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:48.257014 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:48.257248 17824 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:13:48.258144 17824 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:13:48.258658 17824 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:13:48.259001 17824 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:48.259553 17757 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:48.257448 17477 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:13:48.259867 17824 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.256640 17914 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:48.261348 17824 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:48.261694 17914 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:48.262634 17914 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.264652 17914 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.207286 17907 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:48.286690 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.292891 12638 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
W20260709 12:13:48.231454 17870 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.789s	user 0.006s	sys 0.006s
I20260709 12:13:48.288288 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:48.288482 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:48.294349 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.233091 12395 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:13:48.303479 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46553.4us
I20260709 12:13:48.305986 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:48.309536 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:48.312475 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:48.308750 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.218236 17881 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 12:13:48.166812 17930 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.157s
I20260709 12:13:48.235018 17886 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:48.320879 12488 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:13:48.167234 17934 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.141s
I20260709 12:13:48.307477 17848 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:48.315083 17639 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:48.231096 17408 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.328204 12492 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:48.315971 17841 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:48.319399 17858 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:48.237274 17476 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:48.354058 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45398.3us
W20260709 12:13:48.355605 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 133766us
I20260709 12:13:48.310657 17483 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:13:48.334204 17888 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:48.367920 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:48.348040 17861 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.821s	user 0.014s	sys 0.000s
I20260709 12:13:48.369388 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.373584 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.373725 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.374810 12490 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:48.375825 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:13:48.366871 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.368009 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.376817 12488 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:48.306759 17907 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.357451 17930 consensus_queue.cc:1243] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:13:48.357138 17639 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:48.387797 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:48.387941 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.389776 12497 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:48.378975 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.371446 17888 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:48.349743 17851 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:13:48.367571 17790 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:48.408209 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:48.371682 17808 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.219116 17928 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.432973 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.433151 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:48.391029 17639 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:48.434214 12490 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:48.416412 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.433851 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:48.376966 17645 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.384650 17841 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:48.358280 17917 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:13:48.380267 17858 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:48.311954 17870 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.417894 17888 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.382261 17861 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.449793 12490 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:13:48.458721 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.317176 17847 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.439581 17841 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:48.482570 12640 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:13:48.430902 17790 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.488346 12489 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [15.14-15.14]
W20260709 12:13:48.490350 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:48.465649 17808 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:13:48.419070 17921 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.453928 17645 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:13:48.434293 17891 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.257s
I20260709 12:13:48.483464 17841 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:48.535544 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.534994 12492 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:48.494217 17917 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:48.488157 17689 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.492203 17854 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.518909 17808 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:48.559223 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 191283us
I20260709 12:13:48.560937 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.538414 17824 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:48.542687 17917 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:48.583803 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:48.466348 17886 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 16.7, Last appended by leader: 7, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.534927 17863 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.597375 12488 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:48.565049 17851 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:13:48.630147 17941 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
W20260709 12:13:48.630445 17939 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
W20260709 12:13:48.637742 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:48.639197 17855 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:48.640734 17863 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.641645 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:48.572539 17858 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.642815 17861 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.580744 17476 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.590145 17824 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.590389 17689 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:13:48.643528 17408 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.644026 17881 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:48.609094 17954 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:48.588874 17917 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.648778 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:48.629559 17927 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:48.696216 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:48.646941 17911 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:13:48.704790 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:48.747455 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.747822 12639 raft_consensus.cc:2376] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14. Re-sending same reply.
I20260709 12:13:48.748334 12641 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:48.613129 17851 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:48.750066 17899 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
W20260709 12:13:48.750937 17952 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.001s	sys 0.001s
I20260709 12:13:48.750475 17851 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:48.752398 17948 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.215s	user 0.000s	sys 0.005s
I20260709 12:13:48.734647 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.728477 17888 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.710261 17907 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.747409 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.760071 12491 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:48.761675 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:48.752576 17851 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:48.741694 17954 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:48.769483 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:48.776896 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 414267us
I20260709 12:13:48.747684 17855 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:48.778964 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.666307 17808 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.784924 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:48.788380 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.789243 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:48.789673 12491 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:48.794652 17969 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:48.795392 17969 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:48.795877 17969 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:48.797775 17955 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.066s	user 0.004s	sys 0.000s
I20260709 12:13:48.803032 17769 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:48.803619 17769 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:48.804620 17769 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:48.808946 17866 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.823s	user 0.014s	sys 0.002s
I20260709 12:13:48.816993 17866 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.818650 12488 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:48.819895 17866 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.768095 17954 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:48.808467 17799 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.699s	user 0.012s	sys 0.002s
I20260709 12:13:48.801041 17639 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.785282 17870 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.788792 17907 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:48.810112 17928 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.754139 17927 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:48.837947 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76413us
I20260709 12:13:48.814884 17753 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:48.822398 17790 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:48.856251 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:48.835615 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46444.8us
I20260709 12:13:48.857491 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:48.877130 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:48.848824 12491 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:13:48.861941 12493 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:48.864115 17888 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:48.855064 17900 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.160s	user 0.006s	sys 0.009s
I20260709 12:13:48.872749 17556 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.875061 17639 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.805105 17841 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.876356 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:48.836743 17808 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:48.859732 17753 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:48.858539 17947 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260709 12:13:48.839092 17907 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:48.858222 17657 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.890462 17964 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:48.859202 17799 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:13:48.883162 17937 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.304s	user 0.000s	sys 0.005s
W20260709 12:13:48.904022 17951 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.201s	user 0.001s	sys 0.001s
I20260709 12:13:48.920030 12638 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:13:48.905874 17764 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.907938 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:48.909124 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.910584 17647 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:48.914619 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 829) took 1929 ms. Trace:
I20260709 12:13:48.915120 12496 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:13:48.916378 17841 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:48.927431 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:48.930464 12569 rpcz_store.cc:276] 0709 12:13:46.984957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.985163 (+   206us) service_pool.cc:224] Handling call
0709 12:13:46.985482 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:46.985863 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:46.985900 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:47.031223 (+ 45323us) log.cc:844] Serialized 26 byte log entry
0709 12:13:47.161200 (+129977us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:47.161244 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:47.161315 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:48.579178 (+1417863us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:48.914214 (+335036us) raft_consensus.cc:1685] finished
0709 12:13:48.914236 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:48.914540 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8636,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":39876,"spinlock_wait_cycles":4968320,"thread_start_us":129496,"threads_started":1,"wal-append.queue_time_us":455660}
I20260709 12:13:48.934079 12489 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:48.929287 12646 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:13:48.897986 17753 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:48.932763 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:48.936683 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2226) took 1110 ms. Trace:
I20260709 12:13:48.901095 17907 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:48.883693 17870 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:13:48.900424 17966 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:13:48.921375 17757 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.008s
I20260709 12:13:48.954813 17969 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:48.956055 17927 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.957870 17456 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:48.930274 17647 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:48.966281 17647 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.967352 17647 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:48.962539 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83692.7us
I20260709 12:13:48.967424 17969 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:48.968381 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:48.938934 12643 rpcz_store.cc:276] 0709 12:13:47.825959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.858127 (+ 32168us) service_pool.cc:224] Handling call
0709 12:13:47.858665 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:47.890012 (+ 31347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:47.890067 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:47.890864 (+   797us) log.cc:844] Serialized 26 byte log entry
0709 12:13:48.098198 (+207334us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b180025c600. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:13:48.178001 (+ 79803us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:13:48.178042 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:48.178122 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:48.920296 (+742174us) raft_consensus.cc:1685] finished
0709 12:13:48.920322 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:48.936589 (+ 16267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49894,"raft.queue_time_us":38715,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":5477,"spinlock_wait_cycles":7591680,"thread_start_us":79587,"threads_started":1,"wal-append.queue_time_us":447659}
I20260709 12:13:48.970150 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:48.972005 17969 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:48.973538 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:48.974428 17969 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:48.975431 17969 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:48.886821 17855 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:48.945873 17966 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:48.944849 17824 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:48.989795 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:48.992667 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:48.975793 17969 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:48.995436 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:48.961110 17757 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.003298 17757 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.005995 17556 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:13:49.006636 17973 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:13:48.990368 17966 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:48.996665 17841 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:48.997874 17969 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:48.996384 17950 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:13:49.010574 17909 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.801s	user 0.013s	sys 0.003s
I20260709 12:13:49.027280 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2224) took 1239 ms. Trace:
I20260709 12:13:49.027740 12645 rpcz_store.cc:276] 0709 12:13:47.788340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.824356 (+ 36016us) service_pool.cc:224] Handling call
0709 12:13:47.824786 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:48.247783 (+422997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.026954 (+779171us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.027207 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":943452544}
I20260709 12:13:49.021142 17799 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260709 12:13:48.998473 17977 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
W20260709 12:13:49.033963 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.035488 17931 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.036157 17931 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.036620 17792 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:49.037189 17792 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:49.037819 17974 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:49.039202 17792 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.056155 12494 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:49.097867 17917 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.101732 12639 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:49.107414 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2578) took 1719 ms (client timeout 2086 ms). Trace:
W20260709 12:13:49.107998 12564 rpcz_store.cc:269] 0709 12:13:47.387645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.387808 (+   163us) service_pool.cc:224] Handling call
0709 12:13:49.107103 (+1719295us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:49.107295 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1487261568}
I20260709 12:13:49.068846 17907 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.112342 17907 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.114271 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:49.116703 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:49.120098 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:49.133685 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.134332 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:49.127910 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.139793 12564 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 10: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:49.141307 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 125942us
I20260709 12:13:49.142967 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.154910 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.157055 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.168704 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.180019 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2458) took 1411 ms. Trace:
I20260709 12:13:49.180488 12644 rpcz_store.cc:276] 0709 12:13:47.768824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.769815 (+   991us) service_pool.cc:224] Handling call
0709 12:13:49.179841 (+1410026us) spinlock_profiling.cc:243] Waited 85.6 ms on lock 0x7b54002a44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:49.179951 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":196928256}
I20260709 12:13:49.121891 17917 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.182971 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.186440 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 747) took 2479 ms. Trace:
I20260709 12:13:49.186896 13259 rpcz_store.cc:276] 0709 12:13:46.702189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.702408 (+   219us) service_pool.cc:224] Handling call
0709 12:13:46.702890 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:47.561023 (+858133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.181517 (+1620494us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.181749 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1803793792}
I20260709 12:13:49.146317 17851 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.195130 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.229468 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:49.232270 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.185910 17841 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:49.186107 17977 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:13:49.237792 12566 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:49.243006 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.254303 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.166109 17907 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 12:13:49.268784 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.282462 12645 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:49.258033 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.283046 12641 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:49.228586 17799 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:49.257663 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:49.268074 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.287566 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.288110 12645 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:49.289202 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:49.286427 12461 connection.cc:256] RPC call timeout handler was delayed by -1.7200497110002289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260709 12:13:49.289850 12461 connection.cc:256] RPC call timeout handler was delayed by -1.6831497110001692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
W20260709 12:13:49.290155 12461 connection.cc:256] RPC call timeout handler was delayed by -1.6327497110000877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.209s for a response
W20260709 12:13:49.290459 12461 connection.cc:256] RPC call timeout handler was delayed by -1.323800601999892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
W20260709 12:13:49.290719 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1505538199999137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
W20260709 12:13:49.291033 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0745006019999437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260709 12:13:49.292133 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:49.284017 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.271096 12644 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:49.293294 12641 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:49.293754 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.191648 17824 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.285809 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.297355 12643 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:13:49.188079 17964 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:13:49.316895 17949 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:49.301154 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.308961 17931 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.310618 17690 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:13:49.311089 17689 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.312903 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:49.251432 17909 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.317384 17949 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:49.291476 17799 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:49.313889 17967 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:49.288995 17942 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.329185 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.299006 17764 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:49.228381 17851 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.311218 17977 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:49.323417 17863 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:49.371232 17863 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.372839 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2573) took 2514 ms. Trace:
I20260709 12:13:49.318449 17858 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.323097 17408 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:49.373940 17858 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.374150 17408 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.376579 17408 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.378218 17991 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:49.380995 17954 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.006s	sys 0.005s
I20260709 12:13:49.381448 17954 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.357704 17942 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.321204 17870 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:49.385968 17870 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.386370 17949 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.388414 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:49.388955 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:49.389067 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2454) took 1627 ms (client timeout 1936 ms). Trace:
I20260709 12:13:49.389714 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:49.390112 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:49.391543 12494 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:49.344027 17753 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.393723 12637 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:13:49.394244 17753 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.373317 12570 rpcz_store.cc:276] 0709 12:13:46.858427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:46.874688 (+ 16261us) service_pool.cc:224] Handling call
0709 12:13:46.875120 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:47.420854 (+545734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.372537 (+1951683us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.372735 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":724669696}
I20260709 12:13:49.395713 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2228) took 1569 ms. Trace:
I20260709 12:13:49.396101 12637 rpcz_store.cc:276] 0709 12:13:47.826599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.923614 (+ 97015us) service_pool.cc:224] Handling call
0709 12:13:49.393469 (+1469855us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:49.394158 (+   689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.395515 (+  1357us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400155250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:49.395652 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3379363712}
I20260709 12:13:49.399528 17942 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.400095 17966 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.400561 17870 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.319566 17917 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:49.403452 17917 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.404264 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2758) took 1284 ms. Trace:
I20260709 12:13:49.404659 12497 rpcz_store.cc:276] 0709 12:13:48.122449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:48.137249 (+ 14800us) service_pool.cc:224] Handling call
0709 12:13:49.404028 (+1266779us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:49.404172 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":492590976}
I20260709 12:13:49.328589 17689 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.408684 13257 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 12:13:49.409129 13257 pending_rounds.cc:85] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 9
I20260709 12:13:49.409337 13257 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:13:49.409570 13257 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:49.346148 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:13:49.410753 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64698.5us
I20260709 12:13:49.413726 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:49.389540 12642 rpcz_store.cc:269] 0709 12:13:47.761577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.761826 (+   249us) service_pool.cc:224] Handling call
0709 12:13:49.388884 (+1627058us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:49.389008 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880588672}
I20260709 12:13:49.419162 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:49.390132 12496 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:13:49.420572 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:49.421443 12494 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:49.397459 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.343191 12646 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:49.358259 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.424705 12643 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:13:49.334832 17867 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:13:49.327029 17931 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:49.441233 12637 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:49.357012 17685 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:49.363559 17908 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:49.367096 17921 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.384204 17969 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.319803 17987 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:13:49.358091 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128728us
I20260709 12:13:49.334920 17799 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:49.442565 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:49.441105 17942 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:13:49.483278 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.486768 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.446774 12646 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:49.494812 12645 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:49.472615 12489 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:49.450626 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.441102 17657 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.333214 17909 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:49.446977 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 207332us
I20260709 12:13:49.475735 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:49.402467 17966 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.478458 12637 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:13:49.474663 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:13:49.425544 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.320732 17824 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.478590 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:49.516212 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.516338 13262 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:49.444777 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.518169 12642 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:49.445672 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:49.522804 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.471604 17917 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.526639 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.445103 17764 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:49.527719 12646 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:49.528121 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.526116 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.529132 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.533540 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.442684 17858 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.534512 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:49.485771 17689 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.536088 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:49.566650 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.499938 17897 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:49.521201 17685 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:13:49.571853 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45863.9us
I20260709 12:13:49.573392 12494 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:49.499703 17931 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:49.575587 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:49.589174 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.589572 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:49.598335 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.535630 17987 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:13:49.578087 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.536357 17870 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:49.601395 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:49.601768 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:49.611368 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:49.576630 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47592.4us
W20260709 12:13:49.594455 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 228977us
I20260709 12:13:49.606333 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.412094 17991 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.569938 17966 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:13:49.609669 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:49.571400 17917 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:49.613123 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.581521 17977 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:49.580749 17897 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:49.592213 17685 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:49.654472 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:49.617974 17907 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:49.652846 17847 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.660749 17907 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:49.661237 17907 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:49.661273 17847 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.662549 17805 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.585237 17858 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:13:49.663753 17909 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.622459 17987 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:49.664197 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 646) took 1464 ms. Trace:
I20260709 12:13:49.664789 13257 rpcz_store.cc:276] 0709 12:13:48.200047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:48.209265 (+  9218us) service_pool.cc:224] Handling call
0709 12:13:48.209617 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:49.410261 (+1200644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.663843 (+253582us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.664110 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2756463872}
I20260709 12:13:49.665907 17556 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.537142 17657 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.655789 17685 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:49.690447 17863 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.699906 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2222) took 1916 ms. Trace:
I20260709 12:13:49.703451 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.704221 12643 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:49.707127 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94403.5us
W20260709 12:13:49.715161 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 217360us
I20260709 12:13:49.720288 17824 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.718711 17847 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.653482 17897 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:49.738862 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:49.664835 17992 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:13:49.700487 12640 rpcz_store.cc:276] 0709 12:13:47.783163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.806923 (+ 23760us) service_pool.cc:224] Handling call
0709 12:13:47.807362 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:48.483114 (+675752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.699540 (+1216426us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.699814 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1548458240}
I20260709 12:13:49.660749 17977 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.655333 17995 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:49.665278 17987 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:49.731644 17456 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:49.745287 12537 connection.cc:256] RPC call timeout handler was delayed by -1.1989704960001291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:13:49.744271 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:49.662637 17966 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:49.771941 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:49.773064 12570 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:49.773825 17769 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:49.764698 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 685) took 1117 ms. Trace:
I20260709 12:13:49.743373 18001 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:49.778258 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 188825us
I20260709 12:13:49.702989 17556 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.701783 17994 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:13:49.778700 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:49.783545 12565 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:13:49.743701 17996 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.752655 17871 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.782261 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.792223 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.793785 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.795825 12640 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:13:49.797829 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:49.768229 17997 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:49.776360 12491 rpcz_store.cc:276] 0709 12:13:48.647737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:48.884573 (+236836us) service_pool.cc:224] Handling call
0709 12:13:48.884935 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:48.885113 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:48.885410 (+   297us) log.cc:844] Serialized 14 byte log entry
0709 12:13:49.408607 (+523197us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b18004ce700. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:13:49.764135 (+355528us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:49.764238 (+   103us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:49.764307 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.764411 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:49.764607 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":147,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":4021,"raft.queue_time_us":124,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":10973,"spinlock_wait_cycles":4886784,"thread_start_us":355160,"threads_started":1}
I20260709 12:13:49.803791 12563 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:49.776741 17851 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.804097 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:49.810232 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.811124 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:13:49.811219 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.813130 12566 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:49.802321 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.807956 12567 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:49.753044 17824 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:13:49.804863 12569 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:49.789688 17994 consensus_queue.cc:1243] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:13:49.814270 12565 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:49.818019 12568 raft_consensus.cc:2368] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:49.813529 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:49.804685 17841 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.564s	user 0.011s	sys 0.010s
I20260709 12:13:49.802579 17702 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:49.751827 17977 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:49.861657 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:49.865712 12568 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:49.842571 17991 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:49.805809 18007 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:13:49.852465 12566 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [16.7-16.7]
I20260709 12:13:49.854698 12572 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:13:49.855666 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.856200 17849 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:49.856745 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:49.857239 18000 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:13:49.881181 12567 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:49.857985 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.806286 17997 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:49.861598 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117693us
I20260709 12:13:49.966972 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:49.969244 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:49.978336 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47425us
I20260709 12:13:49.980691 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.983033 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:49.985009 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.985232 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:49.986990 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.988760 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:49.986076 13257 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:13:49.991032 13260 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:13:49.871115 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:49.882663 12569 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:13:49.885581 17792 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:49.889842 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2767) took 1570 ms. Trace:
I20260709 12:13:49.899376 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:49.902968 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 656) took 1235 ms. Trace:
I20260709 12:13:49.929522 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:50.028854 12492 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:49.932937 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:50.030054 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:49.933854 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:49.934731 12496 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 10. Current term is 11. Ops: [10.5-10.5]
W20260709 12:13:49.938160 17972 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.039s	user 0.000s	sys 0.005s
I20260709 12:13:49.945024 17886 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:49.939231 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:49.933579 17855 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:13:49.977752 17964 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:49.949558 17657 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:49.940297 17894 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:49.950601 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:49.820775 17969 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:49.799548 17966 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:13:49.785882 18001 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.112661 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.113770 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:50.114192 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:49.964287 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.115192 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:50.009857 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.047608 17851 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.015175 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.027949 12639 rpcz_store.cc:276] 0709 12:13:48.667094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:48.760468 (+ 93374us) service_pool.cc:224] Handling call
0709 12:13:48.779427 (+ 18959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:49.102287 (+322860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.902555 (+800268us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.902873 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":740612992}
W20260709 12:13:50.034138 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49435.9us
I20260709 12:13:49.993251 17991 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:13:50.025911 12493 rpcz_store.cc:276] 0709 12:13:48.319537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:48.319693 (+   156us) service_pool.cc:224] Handling call
0709 12:13:48.350298 (+ 30605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:48.878287 (+527989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:49.889501 (+1011214us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:49.889758 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175635840}
I20260709 12:13:50.128356 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.052160 12496 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:50.053943 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:50.129729 12491 raft_consensus.cc:2368] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:50.056417 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:50.059068 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:49.916687 12460 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.086s (SENT) [suppressed 15 similar messages]
W20260709 12:13:50.131695 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 215075us
I20260709 12:13:50.131928 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:49.962688 17702 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:50.134456 17702 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.135903 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.136616 12565 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:50.069073 12637 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:13:50.140108 12637 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:49.979043 17483 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.070524 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.980932 17997 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:50.072821 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:50.001788 17883 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:50.074437 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:49.802280 17996 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.083678 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.086941 12567 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:49.862828 17841 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:50.114276 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:50.120087 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:50.162218 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.037591 12566 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:13:50.121219 12572 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
W20260709 12:13:50.078078 18000 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:13:50.122617 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:50.133301 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:50.175985 12644 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:50.143797 12642 raft_consensus.cc:2368] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:50.131860 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:50.151401 13262 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:50.152076 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:50.206554 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
W20260709 12:13:50.207405 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84876.8us
I20260709 12:13:50.207556 12496 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:50.066470 17964 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:50.151468 12565 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:50.215723 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.217301 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:49.979440 17849 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.166344 17477 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:50.167577 17556 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.170082 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.122592 17987 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.119796 17657 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:50.146598 17972 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:13:50.203431 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 343487us
W20260709 12:13:50.203954 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46676.6us
I20260709 12:13:50.122227 17932 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:50.144485 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 177584us
I20260709 12:13:50.213774 12643 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 4. Preceding OpId from leader: term: 14 index: 5. (index mismatch)
I20260709 12:13:50.219485 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.222292 12564 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:13:50.222673 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.154906 17969 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.215250 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:50.222380 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:50.130556 17851 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.284228 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:50.160327 18010 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:50.143229 17483 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.250783 17850 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:13:50.207127 17914 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:13:50.295162 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72939.1us
W20260709 12:13:50.288484 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73326.1us
I20260709 12:13:50.145054 17918 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 12:13:50.297294 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:50.300087 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:50.224251 17476 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:50.300990 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.307888 12641 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260709 12:13:50.308677 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 357027us
I20260709 12:13:50.309253 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.304677 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:50.310411 12563 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:50.296602 17969 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.312119 17991 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:50.312904 17969 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.314373 17964 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:50.315219 18011 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:13:50.315712 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.209218 17870 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:13:50.317126 12569 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:50.223927 17867 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:13:50.294746 17855 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:50.295442 18010 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:50.320654 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:50.231664 17923 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 12:13:50.302479 17477 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:13:50.345108 13257 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:13:50.316120 17841 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:50.362265 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.363350 12569 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:50.316877 17227 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:50.351996 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.280493 17987 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.366761 12566 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:50.368512 12572 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:13:50.324529 17476 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:50.326958 17964 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:50.364708 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:50.297070 17657 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.308377 18006 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:13:50.301891 17914 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.341694 17969 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:13:50.391615 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.344287 18010 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:50.349290 17477 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:50.416616 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.349555 17855 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:50.421162 12568 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.555s	user 0.012s	sys 0.000s
I20260709 12:13:50.427111 12568 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:50.427961 12566 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 17 index: 7. (index mismatch)
I20260709 12:13:50.400665 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:50.306375 17932 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.369109 17227 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:50.397264 17848 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:50.405102 17859 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:50.397200 17867 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:13:50.486371 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59721.7us
I20260709 12:13:50.445345 17477 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:50.443488 17977 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.458969 17227 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:50.528272 12572 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:13:50.548095 12568 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:50.552637 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:13:50.533710 17685 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.849s	user 0.017s	sys 0.000s
I20260709 12:13:50.524407 17859 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:50.577906 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.559280 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:50.563212 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:50.567097 12643 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:50.567065 12495 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:13:50.496937 17848 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.514005 17867 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:50.324368 17556 consensus_queue.cc:1175] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 17 status { last_received { term: 16 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:13:50.532178 17476 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:50.568814 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.599262 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:50.640151 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141074us
I20260709 12:13:50.650941 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:50.650187 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52599.8us
I20260709 12:13:50.655068 12643 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:50.655462 12567 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:50.572364 12568 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:50.667128 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.668625 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.597340 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:50.673060 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.674986 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:13:50.654331 18016 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:13:50.673900 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:50.682507 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.623557 17483 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.569927 18001 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.570950 17685 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.696909 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:13:50.676594 17899 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.608274 17702 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:50.698170 17899 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:50.702610 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 302048us
I20260709 12:13:50.703472 18013 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:50.704353 12643 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:50.704852 17954 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:50.705168 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.705541 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:50.706274 18023 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:13:50.707690 17932 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:50.708420 17932 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:50.561883 17918 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:50.710062 17918 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.707583 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:50.711963 17716 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:50.712436 17716 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:50.712446 17929 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:50.713346 17907 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.051s	user 0.014s	sys 0.003s
I20260709 12:13:50.713866 17907 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.715118 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:50.716249 17907 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.717293 17929 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:50.720212 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:50.721325 17949 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:50.556324 17855 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.713555 17716 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.717512 18014 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:50.725943 17716 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:50.727581 17997 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.551s	user 0.010s	sys 0.000s
I20260709 12:13:50.608534 17996 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.731326 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:50.732548 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.748549 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:50.736156 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:50.736508 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.737705 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:50.753572 13258 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:13:50.735586 17954 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:50.755300 17841 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:50.755759 17841 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:50.756695 17841 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.757813 17657 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.761264 12570 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:13:50.762104 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:50.763708 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:50.763993 12570 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
W20260709 12:13:50.764951 17908 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.279s	user 0.011s	sys 0.012s
I20260709 12:13:50.766149 12639 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 12:13:50.767061 18024 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:13:50.767370 17871 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.768520 17788 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:50.768170 17871 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.769423 17788 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:13:50.769896 17788 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:50.770282 17788 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:50.758067 17647 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:13:50.778122 17880 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:50.781265 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:50.782641 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:50.784164 17932 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:50.784981 17954 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:50.785866 17954 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:50.786127 17932 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:50.787137 17954 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.752283 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:50.713362 12569 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:13:50.726616 18014 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:50.808868 17884 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:50.809437 17884 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:50.810461 17884 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:50.812551 17792 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.751s	user 0.008s	sys 0.004s
I20260709 12:13:50.835552 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.808562 18014 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:50.842058 12572 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:13:50.842526 12638 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:13:50.740485 17987 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:50.678923 18021 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.080s
W20260709 12:13:50.856424 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2785) took 1469 ms (client timeout 1770 ms). Trace:
W20260709 12:13:50.856917 12489 rpcz_store.cc:269] 0709 12:13:49.387301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:49.389432 (+  2131us) service_pool.cc:224] Handling call
0709 12:13:50.856236 (+1466804us) spinlock_profiling.cc:243] Waited 82.2 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:50.856355 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189057408}
W20260709 12:13:50.697500 17799 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.198s	user 0.018s	sys 0.003s
I20260709 12:13:50.857775 17799 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.859983 17799 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:50.818630 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.820897 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:50.834081 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.807264 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:50.739686 17914 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:13:50.806277 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75071.9us
I20260709 12:13:50.866637 12566 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:50.848789 12646 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:13:50.845047 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:50.808579 17918 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:50.841756 17841 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:50.854372 18014 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:50.861980 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:50.830808 17995 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.080s	user 0.006s	sys 0.009s
I20260709 12:13:50.874392 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:50.823354 17657 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:50.953421 12495 pending_rounds.cc:85] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 9
I20260709 12:13:50.953917 12495 pending_rounds.cc:107] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:13:50.954229 12495 raft_consensus.cc:2889] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:50.847659 17863 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:50.829355 17908 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:13:50.856449 18026 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:13:50.741791 17997 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:50.885084 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:50.952219 12572 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:50.966106 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80183.2us
W20260709 12:13:50.878201 17886 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.814s	user 0.013s	sys 0.001s
I20260709 12:13:50.835758 17668 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:13:50.978070 12646 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:50.836417 17684 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.116s
I20260709 12:13:50.881428 17880 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:13:50.952672 17657 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:50.963625 17995 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:51.010588 12644 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 7. Preceding OpId from leader: term: 17 index: 7. (term mismatch)
I20260709 12:13:50.964323 17908 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:51.020813 12644 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 6
I20260709 12:13:51.021315 12644 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 16.7
I20260709 12:13:51.021654 12644 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:51.017561 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:51.013352 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.962911 17918 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:13:51.032886 12566 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
W20260709 12:13:51.036319 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:50.989368 17859 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:51.041297 13260 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:13:51.001030 17932 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:51.043910 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.007s	sys 0.037s
I20260709 12:13:50.983894 17852 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:13:51.022603 17657 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:51.022460 17792 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:50.974618 17863 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.035019 17894 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.951s	user 0.013s	sys 0.003s
I20260709 12:13:51.019793 17899 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:51.079957 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2465) took 1830 ms. Trace:
I20260709 12:13:51.142508 12638 rpcz_store.cc:276] 0709 12:13:49.249161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:49.256979 (+  7818us) service_pool.cc:224] Handling call
0709 12:13:51.055801 (+1798822us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:51.079085 (+ 23284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3575676800}
W20260709 12:13:51.056751 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.007s	sys 0.037s
W20260709 12:13:51.148500 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62991.8us
I20260709 12:13:51.058626 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2625) took 1279 ms. Trace:
I20260709 12:13:51.151377 13253 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:13:51.032187 17870 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.151423 12571 rpcz_store.cc:276] 0709 12:13:49.779698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:49.785581 (+  5883us) service_pool.cc:224] Handling call
0709 12:13:49.805385 (+ 19804us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:50.235794 (+430409us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:13:50.298206 (+ 62412us) log.cc:844] Serialized 14 byte log entry
0709 12:13:50.638134 (+339928us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:50.638882 (+   748us) log.cc:844] Serialized 26 byte log entry
0709 12:13:50.639064 (+   182us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:50.639121 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:50.639263 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:51.058361 (+419098us) raft_consensus.cc:1685] finished
0709 12:13:51.058385 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:51.058540 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96169,"prepare.queue_time_us":54584,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":51,"raft.queue_time_us":184761,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":31915,"thread_start_us":700643,"threads_started":2,"wal-append.queue_time_us":554733}
W20260709 12:13:51.091281 17477 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.561s	user 0.011s	sys 0.001s
I20260709 12:13:51.096753 17987 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:51.192338 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.007s	sys 0.024s
W20260709 12:13:51.197474 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.007s	sys 0.024s
W20260709 12:13:51.197988 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 551055us
I20260709 12:13:51.085991 17848 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:51.198876 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.056721 17934 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.140s
I20260709 12:13:51.182776 17477 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.206310 17934 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:51.206869 17934 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:51.207875 17934 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.210273 18028 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:13:51.209259 17934 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:51.212598 17964 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.795s	user 0.006s	sys 0.005s
I20260709 12:13:51.213992 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2486) took 1058 ms. Trace:
I20260709 12:13:51.214418 12644 rpcz_store.cc:276] 0709 12:13:50.155503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.194914 (+ 39411us) service_pool.cc:224] Handling call
0709 12:13:50.199948 (+  5034us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:51.029405 (+829457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.213718 (+184313us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:51.213924 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1862787328}
I20260709 12:13:51.216331 12567 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:13:51.166957 17564 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:51.228075 12637 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.010s	sys 0.022s
I20260709 12:13:51.093885 17950 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:13:51.054168 17932 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.051955 17859 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.156832 17792 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:13:51.230962 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.120666 17908 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.228619 12637 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.010s	sys 0.022s
I20260709 12:13:51.248420 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:51.250594 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.012s	sys 0.026s
W20260709 12:13:51.252300 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46664.5us
I20260709 12:13:51.262007 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:51.261960 13259 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.015s	sys 0.023s
W20260709 12:13:51.266716 13259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.015s	sys 0.023s
I20260709 12:13:51.267259 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:51.268792 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.012s	sys 0.026s
I20260709 12:13:51.262156 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2479) took 1186 ms. Trace:
I20260709 12:13:51.270345 12637 rpcz_store.cc:276] 0709 12:13:50.075718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.140948 (+ 65230us) service_pool.cc:224] Handling call
0709 12:13:50.141279 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:50.202116 (+ 60837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:50.202148 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:50.202767 (+   619us) log.cc:844] Serialized 26 byte log entry
0709 12:13:50.633610 (+430843us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:51.261300 (+627690us) log.cc:844] Serialized 14 byte log entry
0709 12:13:51.261798 (+   498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.261872 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:51.261907 (+    35us) raft_consensus.cc:1685] finished
0709 12:13:51.261930 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:51.262096 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74816,"prepare.queue_time_us":365280,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":61477,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5964,"thread_start_us":922078,"threads_started":2,"wal-append.queue_time_us":636281}
I20260709 12:13:51.221827 17894 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.171903 17870 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.215541 17477 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.295300 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:51.273782 17912 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.008s	sys 0.030s
I20260709 12:13:51.300431 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:51.310791 12643 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.013s	sys 0.031s
W20260709 12:13:51.311343 12643 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.013s	sys 0.032s
W20260709 12:13:51.298168 17855 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.007s	sys 0.020s
I20260709 12:13:51.215929 17987 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.229171 17564 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:51.321249 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:51.309118 17912 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.008s	sys 0.031s
W20260709 12:13:51.306504 18007 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.295s	user 0.037s	sys 0.037s
I20260709 12:13:51.297179 18005 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:51.306636 17997 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.307529 17908 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:51.315881 17855 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.007s	sys 0.021s
I20260709 12:13:51.293956 17417 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:13:51.316167 18033 raft_consensus.cc:2764] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:13:51.312546 18022 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
W20260709 12:13:51.331441 18007 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.326s	user 0.037s	sys 0.037s
I20260709 12:13:51.295964 17894 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.330644 18005 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:51.317466 17824 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
W20260709 12:13:51.359223 17931 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.757s	user 0.012s	sys 0.004s
I20260709 12:13:51.359951 17931 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:51.360875 12497 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:51.361217 17788 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.590s	user 0.013s	sys 0.001s
W20260709 12:13:51.382387 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.018s	sys 0.013s
W20260709 12:13:51.386322 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.018s	sys 0.013s
W20260709 12:13:51.389504 12639 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.008s	sys 0.010s
W20260709 12:13:51.389945 12639 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.009s	sys 0.010s
I20260709 12:13:51.347476 18033 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:51.354601 18004 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:13:51.394032 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103537us
W20260709 12:13:51.367421 17456 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.590s	user 0.012s	sys 0.004s
W20260709 12:13:51.397090 12489 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.010s	sys 0.011s
W20260709 12:13:51.401453 12489 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.011s	sys 0.011s
I20260709 12:13:51.400362 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.333453 18037 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:13:51.412334 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.333895 17792 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.382131 17859 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.396289 17848 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:13:51.436111 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.367417 17788 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.399245 18004 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:51.410661 17456 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:13:51.443325 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.007s	sys 0.022s
I20260709 12:13:51.415793 18022 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:13:51.441378 18051 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:13:51.426636 18037 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 12:13:51.437095 17883 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.271s	user 0.014s	sys 0.000s
W20260709 12:13:51.447515 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.007s	sys 0.022s
W20260709 12:13:51.475278 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 668106us
W20260709 12:13:51.477011 18034 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:13:51.477836 18046 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.478411 18046 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.479465 18036 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:13:51.480737 18046 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.429818 18033 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:51.485366 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:51.409456 18005 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:51.488652 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 264245us
W20260709 12:13:51.491262 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79045.9us
I20260709 12:13:51.494699 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:51.495873 12637 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:51.497119 17894 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.498016 18050 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:51.498469 18050 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:51.499588 18050 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.481711 17994 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:51.501994 17994 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:51.502373 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:51.503352 17897 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.756s	user 0.015s	sys 0.002s
I20260709 12:13:51.503129 17994 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.503818 17897 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.504812 17994 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:51.505146 12492 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.6-12.6]
W20260709 12:13:51.506173 18054 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
W20260709 12:13:51.506501 18039 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:13:51.482885 17841 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.507208 17841 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.510641 18010 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.015s	user 0.013s	sys 0.000s
I20260709 12:13:51.504499 18050 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:51.511152 18010 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.512631 18040 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:13:51.505797 18004 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:51.525017 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.503170 13255 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:51.527845 12637 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:13:51.527534 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2491) took 1288 ms. Trace:
I20260709 12:13:51.529911 12643 rpcz_store.cc:276] 0709 12:13:50.238942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.709039 (+470097us) service_pool.cc:224] Handling call
0709 12:13:50.709341 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:50.709541 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:51.311836 (+602295us) log.cc:844] Serialized 14 byte log entry
0709 12:13:51.527057 (+215221us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:51.527117 (+    60us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:51.527179 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.527291 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:51.527458 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65210,"prepare.queue_time_us":346740,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":9280,"thread_start_us":735836,"threads_started":2}
W20260709 12:13:51.532028 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.485823 17883 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:51.520181 17932 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.603415 12491 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:51.520025 17841 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:51.528710 17227 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.995s	user 0.007s	sys 0.006s
I20260709 12:13:51.525786 17456 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:51.526361 18005 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:51.612926 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.533043 17914 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:51.602345 17716 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.875s	user 0.003s	sys 0.011s
W20260709 12:13:51.623454 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 323099us
W20260709 12:13:51.529273 18022 consensus_queue.cc:1175] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 15 status { last_received { term: 13 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: [14.13-14.13]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:13:51.632969 17849 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.635910 18053 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:51.636763 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.603574 17987 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.603034 17934 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.534323 17657 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.660727 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.662869 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:51.621464 18014 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.735s	user 0.007s	sys 0.006s
I20260709 12:13:51.613894 17417 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.677529 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:51.673231 17841 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.692433 13261 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
I20260709 12:13:51.615715 17227 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.689543 17476 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:51.703400 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:51.704282 12639 raft_consensus.cc:2376] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12. Re-sending same reply.
I20260709 12:13:51.623255 18057 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:13:51.690253 17849 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:13:51.688088 17897 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:51.717760 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93390.1us
I20260709 12:13:51.718773 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.685022 17934 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.699563 17476 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:51.724869 18063 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:51.750417 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.750839 18063 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.751621 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.752979 18063 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:51.697844 17788 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:51.755707 17934 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:51.730974 17657 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.688136 17417 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.727416 17926 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:51.734833 17227 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.698518 17841 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.748034 18010 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.750233 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:51.756752 17476 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.725615 18047 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:13:51.794334 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.738765 17995 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:51.764050 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.798319 12638 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:51.799345 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:51.764240 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.800282 12641 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:51.801506 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:51.802122 12641 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:51.766999 13254 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:51.770481 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.773788 12639 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 18. Ops: [16.11-16.11]
I20260709 12:13:51.812338 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.813715 12493 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:13:51.818178 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.819166 12639 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 10.
I20260709 12:13:51.822297 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.822629 12536 leader_election.cc:400] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:51.822933 12536 leader_election.cc:403] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:13:51.824293 12639 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:13:51.679731 18053 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.828154 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:51.779652 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.741339 17908 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.825532 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.778146 18014 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:51.795099 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:51.794799 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.834393 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:51.834457 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:51.759877 17417 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:51.797350 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.798161 18058 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:51.797408 17863 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:51.791774 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73068us
I20260709 12:13:51.860653 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:51.861459 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.863902 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.766242 17926 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:51.837198 12639 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:13:51.838382 12494 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:51.865440 17926 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.826930 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:51.831610 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.845894 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.846503 12641 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:13:51.857700 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.816488 17995 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:51.865846 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:51.840008 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.871021 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.871984 12646 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:51.867574 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.869035 12638 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:51.867633 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:51.875813 12639 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 14 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:13:51.837368 17987 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:51.814667 18046 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:13:51.883725 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.880445 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2618) took 1047 ms. Trace:
I20260709 12:13:51.884555 12569 rpcz_store.cc:276] 0709 12:13:50.833083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.858931 (+ 25848us) service_pool.cc:224] Handling call
0709 12:13:50.859313 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:50.859714 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:13:50.859742 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:50.860262 (+   520us) log.cc:844] Serialized 26 byte log entry
0709 12:13:51.322645 (+462383us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:51.322727 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.322844 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:51.880145 (+557301us) raft_consensus.cc:1685] finished
0709 12:13:51.880180 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:51.880362 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":908,"spinlock_wait_cycles":13593088,"thread_start_us":461878,"threads_started":1,"wal-append.queue_time_us":622233}
I20260709 12:13:51.837041 17950 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:51.835647 17971 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:51.886051 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:51.891816 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:51.893568 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:51.887135 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.895361 12493 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:51.876411 18062 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:13:51.896668 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.908030 17971 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.908828 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:51.909514 12494 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:51.896872 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:13:51.910154 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.910707 17971 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.911255 18070 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:51.912642 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:51.898854 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:51.916410 12493 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:51.896341 18058 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:51.899713 17987 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:51.899729 18063 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:51.902925 18059 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
I20260709 12:13:51.826623 18010 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:13:51.903386 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2606) took 1579 ms. Trace:
I20260709 12:13:51.940760 12563 rpcz_store.cc:276] 0709 12:13:50.324300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.408868 (+ 84568us) service_pool.cc:224] Handling call
0709 12:13:50.410276 (+  1408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:51.028627 (+618351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.903105 (+874478us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b54003e0154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:51.903316 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1372140800}
I20260709 12:13:51.906726 17863 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:51.942056 17863 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:51.942523 17863 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:51.942600 18053 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:51.943718 17851 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:51.874692 18014 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.921241 17227 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:51.946058 18014 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:51.946269 17227 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.839143 17908 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:51.943003 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 654) took 1828 ms (client timeout 1856 ms). Trace:
I20260709 12:13:51.950138 17908 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:51.950034 13254 rpcz_store.cc:269] 0709 12:13:50.114191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.114960 (+   769us) service_pool.cc:224] Handling call
0709 12:13:51.942816 (+1827856us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:51.942937 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3155050880}
I20260709 12:13:51.952437 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.953116 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.921542 12489 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:51.954253 12570 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:13:51.953585 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.954550 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.955017 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:51.914772 17556 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:51.953194 12568 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:51.955976 17556 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:51.956530 17556 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.957962 12494 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:51.924460 17987 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:51.959226 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.925407 17647 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:51.896487 17788 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:51.934687 17950 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:51.963757 13260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 652) took 1850 ms (client timeout 2106 ms). Trace:
I20260709 12:13:51.934937 17841 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:51.935549 18004 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:51.964285 13260 rpcz_store.cc:269] 0709 12:13:50.113562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.113945 (+   383us) service_pool.cc:224] Handling call
0709 12:13:51.963477 (+1849532us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:51.963685 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2129407232}
I20260709 12:13:51.883379 17417 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 12:13:51.965724 17867 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:51.938823 17799 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:51.939105 17995 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.937347 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 696) took 1911 ms. Trace:
I20260709 12:13:51.970177 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.970492 12497 rpcz_store.cc:276] 0709 12:13:50.025974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.173917 (+147943us) service_pool.cc:224] Handling call
0709 12:13:50.222258 (+ 48341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:51.361339 (+1139081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.937032 (+575693us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:51.937268 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2618049280}
I20260709 12:13:51.883108 17848 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:51.954104 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.921306 18070 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.928058 18063 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:51.959632 17987 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.879302 18073 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:51.963544 17950 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:51.923511 17766 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:51.953421 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:51.932935 18058 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:51.990351 18063 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:51.995254 18058 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.897985 17849 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:51.996747 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.997570 12497 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:51.997239 17849 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:51.999046 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:51.999351 17950 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.000118 17766 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:51.968320 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136816us
I20260709 12:13:52.000636 12563 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:13:52.001104 12563 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:13:52.001354 12563 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:13:52.001576 12563 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:52.001866 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:52.002368 17766 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.002676 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:52.003224 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.003932 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.969245 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 692) took 2021 ms. Trace:
I20260709 12:13:51.971257 12564 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:52.005276 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.006016 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.006513 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:51.974328 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 10.
I20260709 12:13:52.006847 17227 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.007741 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.008407 12494 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:51.974234 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.010141 12571 raft_consensus.cc:2376] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16. Re-sending same reply.
I20260709 12:13:52.011211 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.004891 12491 rpcz_store.cc:276] 0709 12:13:49.947889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.130388 (+182499us) service_pool.cc:224] Handling call
0709 12:13:50.130717 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:51.603976 (+1473259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.968911 (+364935us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:51.969168 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3386241920}
I20260709 12:13:52.012887 18058 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:51.972790 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.006093 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.013556 12491 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:13:52.014082 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:51.955260 12489 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:52.016038 12489 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:51.993242 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:52.017710 12570 raft_consensus.cc:2368] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:52.018484 17950 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:52.006721 18014 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:52.019748 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:52.019948 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:52.003568 12497 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:52.021027 18014 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:52.022112 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.022605 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:52.022936 12569 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:52.023900 18014 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:52.026000 12493 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:52.007035 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:52.031021 12536 leader_election.cc:400] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14.
I20260709 12:13:52.035166 12572 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:52.038455 18014 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:52.023555 17863 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.046614 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.072117 17950 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:51.989780 18073 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.022125 17556 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.045753 17987 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:52.086583 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:51.978358 17799 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.088425 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:52.095402 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.095660 12567 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:13:52.095857 12566 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:52.095618 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.087141 17995 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:52.086340 18073 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:52.090297 17556 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:52.118412 17753 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:52.123931 17799 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.126408 17871 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:52.130127 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:52.130806 17926 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:52.135661 17685 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.136191 17476 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:52.137008 17564 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.091159 18058 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.130946 12569 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:13:52.155332 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59807.8us
I20260709 12:13:52.119089 17753 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:52.141373 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.145509 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.159580 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:52.152705 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.154438 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.161731 12567 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:52.156337 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.162957 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.163116 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:52.102391 18010 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:52.170506 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.171232 12572 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:52.160791 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:52.176460 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:52.164465 18075 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:13:52.114862 17556 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.178251 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:52.183224 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:52.183588 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:52.184238 13260 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:13:52.184515 13254 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:13:52.143086 17792 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:52.192937 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:13:52.162858 17871 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:52.163467 17859 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:52.168922 17996 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:52.165470 17417 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:52.198406 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:52.190248 18014 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:52.201743 17931 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:52.201946 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:52.202692 17908 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:52.169245 17799 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:52.203518 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:52.165123 17564 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.169912 18010 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.172320 17899 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.203037 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:52.185539 18027 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:52.174559 18081 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:52.179252 18076 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:52.213217 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:52.185446 18033 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:52.165797 17227 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:13:52.167665 18058 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.193904 17556 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:52.182579 18084 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:52.175450 17685 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:13:52.176621 17912 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:13:52.197355 17971 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:13:52.235181 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.236169 13261 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16. Re-sending same reply.
I20260709 12:13:52.197392 17994 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:13:52.256536 17923 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:52.280521 17912 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.280931 17923 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.281366 17923 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:52.281927 12570 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 12:13:52.226768 17859 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.283051 17645 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:52.283798 17645 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:52.284567 17859 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.214754 17863 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:52.227764 17996 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.272485 17841 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.277135 18076 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.286865 18076 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:52.286664 17841 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:52.287442 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 803) took 4407 ms (client timeout 2115 ms). Trace:
W20260709 12:13:52.287823 13253 rpcz_store.cc:269] 0709 12:13:47.880432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:47.880612 (+   180us) service_pool.cc:224] Handling call
0709 12:13:52.287150 (+4406538us) spinlock_profiling.cc:243] Waited 3.27 s on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:52.287330 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7510162176}
I20260709 12:13:52.289088 17841 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.273115 12639 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:52.290091 13258 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:13:52.289773 18084 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.243754 18010 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:52.291550 17848 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:52.227902 17417 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.276830 17685 consensus_queue.cc:1243] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:52.242952 18014 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.204017 17792 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.251549 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2483) took 2139 ms. Trace:
I20260709 12:13:52.292728 17417 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:52.280715 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 651) took 2167 ms. Trace:
I20260709 12:13:52.277035 18033 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:52.281287 17912 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.244973 12646 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:13:52.243326 12609 leader_election.cc:400] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 12. Current term is 13.
I20260709 12:13:52.237599 18007 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.238390 17871 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:52.295819 12639 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:13:52.279481 18058 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.292160 18084 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.222208 17753 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.290522 13258 pending_rounds.cc:85] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 10
I20260709 12:13:52.312742 13258 pending_rounds.cc:107] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:13:52.313148 13258 raft_consensus.cc:2889] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:52.284850 17645 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.297469 12642 rpcz_store.cc:276] 0709 12:13:50.112570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.176869 (+ 64299us) service_pool.cc:224] Handling call
0709 12:13:50.177242 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:50.201761 (+ 24519us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:51.812267 (+1610506us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b18002997a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:51.967352 (+155085us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.169773 (+202421us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:52.170260 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:13:52.170397 (+   137us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:52.170645 (+   248us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.170848 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.170924 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:52.251276 (+ 80352us) raft_consensus.cc:1685] finished
0709 12:13:52.251301 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:52.251472 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82865,"prepare.queue_time_us":540651,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":78,"raft.queue_time_us":27545,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":30762,"spinlock_wait_cycles":63408896,"thread_start_us":272186,"threads_started":2,"wal-append.queue_time_us":223654}
I20260709 12:13:52.295012 18081 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:52.303732 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.318349 12646 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:52.318492 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2238) took 2381 ms. Trace:
I20260709 12:13:52.319037 12492 rpcz_store.cc:276] 0709 12:13:49.935758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.067855 (+132097us) service_pool.cc:224] Handling call
0709 12:13:50.068250 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:51.505592 (+1437342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.316582 (+810990us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:52.316801 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3303894400}
I20260709 12:13:52.312320 17848 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.323674 17848 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.325217 18037 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:52.325708 18037 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:52.326735 18037 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.327472 17753 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:52.327915 17639 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.328542 17639 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:52.330281 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:52.330825 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:52.310791 18083 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:13:52.252030 17934 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:52.298501 13259 rpcz_store.cc:276] 0709 12:13:50.113220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.113384 (+   164us) service_pool.cc:224] Handling call
0709 12:13:50.718094 (+604710us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:50.718459 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:50.718533 (+    74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:50.719025 (+   492us) log.cc:844] Serialized 26 byte log entry
0709 12:13:51.267344 (+548319us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:51.267414 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.267521 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:51.513682 (+246161us) raft_consensus.cc:1685] finished
0709 12:13:51.513707 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:52.280404 (+766697us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54006d1ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:52.280653 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2050,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":1389611392,"thread_start_us":547624,"threads_started":1,"wal-append.queue_time_us":691727}
I20260709 12:13:52.285789 17996 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:52.244040 17799 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:52.346607 17799 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:52.299382 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57991us
I20260709 12:13:52.327956 18037 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:52.350737 17848 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.351919 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 758) took 1201 ms. Trace:
I20260709 12:13:52.352373 13258 rpcz_store.cc:276] 0709 12:13:51.150582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:51.152208 (+  1626us) service_pool.cc:224] Handling call
0709 12:13:52.289726 (+1137518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:52.313885 (+ 24159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.351669 (+ 37784us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54005c3fd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:52.351834 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2615010048}
W20260709 12:13:52.354399 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.330749 17639 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.356549 18014 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.357143 18014 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:52.357882 17841 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:52.357769 18014 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.358453 17841 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.362814 17914 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:52.301369 17564 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.364054 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:52.369710 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.371337 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.301133 18033 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:52.302593 12490 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:52.333925 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:52.334206 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.267671 18070 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.252595 17899 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.252836 17852 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:13:52.306242 17871 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.338249 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:52.339557 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 706) took 1470 ms. Trace:
I20260709 12:13:52.341115 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:52.344045 17912 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.316634 17645 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:52.366647 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.422618 12566 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:13:52.391247 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.354905 17792 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.308495 18007 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.415469 12495 rpcz_store.cc:276] 0709 12:13:50.869698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.869918 (+   220us) service_pool.cc:224] Handling call
0709 12:13:50.953147 (+ 83229us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:51.001661 (+ 48514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:51.001693 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:51.002415 (+   722us) log.cc:844] Serialized 52 byte log entry
0709 12:13:51.487113 (+484698us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:51.532111 (+ 44998us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.332086 (+799975us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b18004114c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:52.332311 (+   225us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.332527 (+   216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.332604 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:52.339297 (+  6693us) raft_consensus.cc:1685] finished
0709 12:13:52.339321 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:52.339496 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98942,"prepare.queue_time_us":52024,"raft.queue_time_us":44488,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":40964,"spinlock_wait_cycles":11165824,"thread_start_us":438686,"threads_started":1,"wal-append.queue_time_us":588143}
W20260709 12:13:52.419651 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 150361us
I20260709 12:13:52.370464 17914 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:52.428563 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:52.397537 18032 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:52.437563 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.391522 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:52.440613 13260 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:13:52.398037 17848 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:52.411815 18037 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:52.419807 17871 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:52.452890 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110958us
I20260709 12:13:52.441919 18032 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.439456 17912 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:52.458209 12646 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:13:52.463201 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.463510 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.463791 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.464087 12492 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:52.464661 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:52.465489 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.467074 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:52.468627 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.442863 17848 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:52.481663 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.478526 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:52.483798 12491 raft_consensus.cc:606] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:52.482300 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.456398 17792 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.509373 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.468113 18032 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:52.509492 12496 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [12.8-14.9]
I20260709 12:13:52.513617 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:52.508747 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.473202 17863 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.514403 18027 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
W20260709 12:13:52.545329 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.545805 18027 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:52.453003 18081 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.549114 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.549414 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.574584 12569 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:52.551179 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.551971 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:52.552937 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:52.575722 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.576301 12564 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:52.516357 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.577167 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.577972 12491 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:52.517319 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.580806 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:52.532992 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.582504 12492 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:13:52.482403 12495 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
W20260709 12:13:52.489526 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.560619 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:52.584453 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:52.559010 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.532166 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.530419 12637 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:52.533033 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.575407 12566 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:52.536232 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2597) took 2382 ms. Trace:
I20260709 12:13:52.589342 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.531628 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:52.590037 12497 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:52.590391 12639 raft_consensus.cc:2376] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15. Re-sending same reply.
I20260709 12:13:52.559211 12571 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:52.591307 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:52.585925 12494 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:52.473868 17799 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.480041 17848 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:52.476688 18037 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:52.573201 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.596045 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.596738 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.589798 12565 rpcz_store.cc:276] 0709 12:13:50.153416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.320745 (+167329us) service_pool.cc:224] Handling call
0709 12:13:50.321093 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:50.424902 (+103809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:50.424940 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:50.425549 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 12:13:51.145672 (+720123us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:51.145735 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.535765 (+1390030us) spinlock_profiling.cc:243] Waited 88.8 ms on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:52.535832 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:52.535893 (+    61us) raft_consensus.cc:1685] finished
0709 12:13:52.535910 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:52.536160 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14524,"raft.queue_time_us":15485,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":15756,"spinlock_wait_cycles":204216704,"thread_start_us":625733,"threads_started":1,"wal-append.queue_time_us":844834}
I20260709 12:13:52.616948 18084 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:52.618042 18084 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:52.617844 17867 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:13:52.620064 17867 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:52.620481 17867 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:52.528494 17870 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:52.479216 17912 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:52.479547 18086 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:13:52.600441 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.563696 18027 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:52.530004 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:52.609175 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.677043 12492 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:52.623936 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.607492 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:52.678790 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71366.5us
I20260709 12:13:52.678964 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:52.592023 12639 raft_consensus.cc:2376] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15. Re-sending same reply.
I20260709 12:13:52.573877 17639 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:52.516976 17841 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.506413 17908 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.108s
I20260709 12:13:52.561174 17852 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:52.603348 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:52.655166 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.673487 12489 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:52.674263 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 10.
I20260709 12:13:52.680403 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.702764 12639 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:52.625483 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.532124 17863 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.683125 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.683676 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.685092 12637 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:52.540596 18014 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.686959 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.710959 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.711712 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:13:52.678459 17685 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:52.758137 12646 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:52.691026 12564 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:52.618479 18084 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:52.514992 17792 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.630997 12462 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
I20260709 12:13:52.768270 12462 leader_election.cc:403] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:13:52.646057 18050 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.626663 18092 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:52.771350 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:52.772085 12495 raft_consensus.cc:606] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:52.585923 18093 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.706571 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:52.748200 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:52.767741 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88138.2us
W20260709 12:13:52.774646 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171404us
I20260709 12:13:52.775970 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.782171 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.781229 12637 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:52.694777 18087 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:52.785832 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.621258 17867 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 14.5, Last appended by leader: 5, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.706430 18086 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:52.791767 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 160916us
I20260709 12:13:52.792762 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:52.804798 12639 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:52.691258 17417 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.809353 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.809547 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.810123 12495 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:13:52.814481 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:52.815847 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.696568 17639 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.816768 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:52.815865 17824 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:13:52.818305 17824 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.818797 17824 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:13:52.786844 18046 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.819898 18046 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.817798 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.626215 17799 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.821369 12637 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:13:52.822134 18046 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.822551 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.822841 17799 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.823307 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:52.823601 17867 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:52.824198 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.825340 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.802703 18037 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:52.826151 18037 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.826586 18037 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:52.830884 17923 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.000s
I20260709 12:13:52.831439 17923 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.831998 17417 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:52.832996 17824 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.834846 17996 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.835382 17639 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.836139 17639 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:52.833765 17934 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
I20260709 12:13:52.804194 17564 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:52.837826 17934 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:52.838402 17934 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.775893 18092 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.810380 12492 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:52.843200 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2480) took 2767 ms. Trace:
W20260709 12:13:52.793967 18063 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.798s	user 0.014s	sys 0.000s
I20260709 12:13:52.843724 12645 rpcz_store.cc:276] 0709 12:13:50.075966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:50.155739 (+ 79773us) service_pool.cc:224] Handling call
0709 12:13:50.213748 (+ 58009us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:50.245375 (+ 31627us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:50.542182 (+296807us) log.cc:844] Serialized 14 byte log entry
0709 12:13:51.058535 (+516353us) log.cc:844] Serialized 14 byte log entry
0709 12:13:51.058772 (+   237us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:51.087949 (+ 29177us) log.cc:844] Serialized 52 byte log entry
0709 12:13:51.088133 (+   184us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:13:51.088375 (+   242us) log.cc:844] Serialized 14 byte log entry
0709 12:13:51.106619 (+ 18244us) log.cc:844] Serialized 14 byte log entry
0709 12:13:51.392665 (+286046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:51.392797 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:51.931888 (+539091us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b1800453460. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:52.348867 (+416979us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b1800453460. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:52.817880 (+469013us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b1800453460. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:52.842866 (+ 24986us) raft_consensus.cc:1685] finished
0709 12:13:52.842903 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:52.843123 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20923,"prepare.queue_time_us":1280099,"prepare.run_cpu_time_us":217,"prepare.run_wall_time_us":12501,"raft.queue_time_us":428989,"raft.run_cpu_time_us":912,"raft.run_wall_time_us":79956,"spinlock_wait_cycles":20213888,"thread_start_us":978436,"threads_started":3,"wal-append.queue_time_us":745732}
I20260709 12:13:52.814851 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:52.825963 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:52.850178 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.796347 17926 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:52.837272 17996 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:52.851624 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.714103 17852 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.783260 18050 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.791436 18093 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.807485 18027 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:52.856475 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2249) took 1107 ms. Trace:
W20260709 12:13:53.246016 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.005s	sys 0.009s
W20260709 12:13:53.247117 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.006s	sys 0.009s
W20260709 12:13:53.247659 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 567706us
W20260709 12:13:53.244541 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:53.245983 12488 rpcz_store.cc:276] 0709 12:13:51.749456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:51.751907 (+  2451us) service_pool.cc:224] Handling call
0709 12:13:51.752201 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:51.752417 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:52.339701 (+587284us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.855364 (+515663us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b18004b4420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:13:52.855684 (+   320us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.856031 (+   347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:52.856103 (+    72us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:52.856163 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.856260 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:52.856408 (+   148us) inbound_call.cc:177] Queueing success response
0709 12:13:53.054620 (+198212us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b18004b4420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
Metrics: {"mutex_wait_us":595081,"prepare.queue_time_us":1000409,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":95,"raft.queue_time_us":440,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":35451,"spinlock_wait_cycles":21952128,"thread_start_us":120900,"threads_started":1,"wal-append.queue_time_us":142757}
W20260709 12:13:53.246313 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2256) took 1416 ms (client timeout 1775 ms). Trace:
W20260709 12:13:53.259006 12490 rpcz_store.cc:269] 0709 12:13:51.830438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:51.834183 (+  3745us) service_pool.cc:224] Handling call
0709 12:13:53.245563 (+1411380us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.246000 (+   437us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":970142976}
I20260709 12:13:53.260730 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:53.262698 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:53.263692 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:53.273643 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:53.279682 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:53.265043 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:53.287799 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:53.290809 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:53.293135 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:53.000496 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2257) took 1167 ms. Trace:
I20260709 12:13:53.295369 12640 rpcz_store.cc:276] 0709 12:13:51.832812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:51.867411 (+ 34599us) service_pool.cc:224] Handling call
0709 12:13:53.000263 (+1132852us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.000409 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183710592}
I20260709 12:13:53.297757 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:53.298735 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 494062us
I20260709 12:13:53.299100 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:53.271504 12490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:53.068101 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:53.070461 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:53.269326 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:52.873826 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:52.900573 17824 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:53.015197 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 200779us
I20260709 12:13:53.063302 17884 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:53.316695 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.066330 17923 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.318745 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:52.987758 18073 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.823s	user 0.008s	sys 0.003s
I20260709 12:13:53.320338 13254 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:53.320986 18073 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.323676 12644 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
W20260709 12:13:52.900038 17753 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.572s	user 0.000s	sys 0.018s
I20260709 12:13:52.799894 18086 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:52.982185 13258 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:53.348939 17849 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:53.093053 17645 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.658s	user 0.012s	sys 0.000s
I20260709 12:13:52.805784 17870 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:53.316781 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.360316 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:53.071336 18033 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.650s	user 0.012s	sys 0.000s
I20260709 12:13:53.361874 13255 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:53.097146 17685 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:13:52.894150 17995 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.787s	user 0.009s	sys 0.004s
I20260709 12:13:53.364449 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:52.906242 17564 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:52.895241 17863 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:52.958302 17987 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.870s	user 0.010s	sys 0.003s
I20260709 12:13:53.010404 18007 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:52.895980 18010 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.605s	user 0.006s	sys 0.005s
I20260709 12:13:53.366992 17564 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.368541 18010 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:53.015563 17871 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.001s
I20260709 12:13:53.370623 17871 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.371238 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2496) took 1675 ms. Trace:
I20260709 12:13:52.956869 17899 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:53.371790 12644 rpcz_store.cc:276] 0709 12:13:51.695391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:51.695528 (+   137us) service_pool.cc:224] Handling call
0709 12:13:51.908943 (+213415us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:13:51.909158 (+   215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.340843 (+1431685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.370730 (+ 29887us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:53.371088 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3255716992}
W20260709 12:13:52.954726 18076 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.667s	user 0.006s	sys 0.006s
I20260709 12:13:53.373697 17788 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.374632 18076 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.374713 17788 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:53.108726 17914 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.660s	user 0.012s	sys 0.000s
W20260709 12:13:53.378129 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117697us
I20260709 12:13:53.378399 17926 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:53.378628 17914 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.380354 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:53.381937 18085 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.383055 18085 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.383525 18004 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:53.384164 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:53.384606 18004 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.385579 12639 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:53.389079 18093 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.389430 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:53.395507 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:53.090720 18101 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.287s	user 0.002s	sys 0.000s
I20260709 12:13:52.900193 17792 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.401618 12566 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:53.360592 12569 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:53.394556 12389 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:53.403826 12646 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:52.985384 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.404856 12563 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:13:53.405222 13261 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:53.406311 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:52.981235 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.407789 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:53.408236 13256 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:53.410153 12642 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
I20260709 12:13:53.410995 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 762) took 1077 ms. Trace:
I20260709 12:13:53.411653 13256 rpcz_store.cc:276] 0709 12:13:52.333767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.333894 (+   127us) service_pool.cc:224] Handling call
0709 12:13:53.410576 (+1076682us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.410880 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2252067584}
I20260709 12:13:53.021869 18037 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.414201 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.415650 18073 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:52.953524 17556 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.671s	user 0.009s	sys 0.001s
I20260709 12:13:53.416782 18037 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.417295 18085 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.417728 17556 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.420336 17871 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.422422 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:53.423455 12488 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:53.425999 17870 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:53.426321 13255 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:53.426651 17870 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:53.428329 17859 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:53.428984 17859 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:52.873136 17934 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.429571 17639 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:53.430224 17639 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:53.430761 17639 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:53.431854 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:52.852075 18063 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:53.434098 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69687.3us
I20260709 12:13:53.434165 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.434468 18076 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:53.110143 18032 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.001s
I20260709 12:13:53.436405 18032 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.436941 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:53.438163 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:53.437839 18107 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:52.874509 17912 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.440045 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:53.442286 17850 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:53.443173 17850 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:53.443902 12489 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:53.386945 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:53.446025 17645 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:53.446527 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:53.447124 17645 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.107340 18087 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:53.090575 18104 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.148s	user 0.000s	sys 0.003s
I20260709 12:13:53.450490 12638 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:53.114451 17848 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.452375 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2266) took 1012 ms. Trace:
I20260709 12:13:53.452723 17914 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:52.847716 18092 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:53.454567 18119 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.455575 18119 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.361949 18081 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.415696 13254 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 16 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
W20260709 12:13:52.930493 18102 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:13:53.421306 17556 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.452822 12638 rpcz_store.cc:276] 0709 12:13:52.440340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.826466 (+386126us) service_pool.cc:224] Handling call
0709 12:13:52.826781 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.451139 (+624358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.451980 (+   841us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:53.452290 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1432882432}
I20260709 12:13:53.441394 12497 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:13:53.485651 17645 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.486037 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:53.429495 17859 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:53.381768 17867 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:53.408253 12493 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:53.416884 18073 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.500451 12639 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:13:53.510615 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:53.513947 12567 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:53.515003 18119 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.516105 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:53.510262 17645 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:53.527238 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:53.545099 17950 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:53.545984 17950 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.354127 17849 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.556006 12639 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:13:53.561798 18093 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:53.562144 12489 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:13:53.563720 18110 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:13:53.567858 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113565us
I20260709 12:13:53.568478 18032 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.445345 17850 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.570587 18063 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:53.563733 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 685) took 1046 ms. Trace:
I20260709 12:13:53.610008 12638 rpcz_store.cc:276] 0709 12:13:52.517049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.502590 (+985541us) service_pool.cc:224] Handling call
0709 12:13:53.562766 (+ 60176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.563268 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:53.563303 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:53.563371 (+    68us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:13:53.563407 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.563496 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:53.563641 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:53.540938 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:53.547565 17852 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.598202 18093 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:53.624542 17788 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.612957 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:53.626899 18033 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.631191 17995 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:53.662055 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95534.1us
I20260709 12:13:53.632797 17848 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:53.480708 17926 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.678462 17995 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.594887 18037 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.633388 12569 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:53.638227 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 162834us
I20260709 12:13:53.632982 12637 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:13:53.631718 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 163452us
I20260709 12:13:53.690164 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2500) took 1011 ms. Trace:
I20260709 12:13:53.655386 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:53.691816 12644 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:13:53.666880 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:53.656397 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2263) took 1052 ms. Trace:
I20260709 12:13:53.656034 18093 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:53.645409 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2264) took 1055 ms. Trace:
I20260709 12:13:53.609364 18081 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.677323 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.678632 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:53.634037 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 679) took 1125 ms. Trace:
W20260709 12:13:53.690762 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:53.655493 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:53.694483 12638 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:53.692729 12493 rpcz_store.cc:276] 0709 12:13:52.589100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.605739 (+ 16639us) service_pool.cc:224] Handling call
0709 12:13:53.641308 (+1035569us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b540034e754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.641587 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582150784}
I20260709 12:13:53.695511 12637 rpcz_store.cc:276] 0709 12:13:52.678127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.298631 (+620504us) service_pool.cc:224] Handling call
0709 12:13:53.689847 (+391216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:53.671501 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:53.639757 17824 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.647253 18004 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.578490 17867 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:52.897189 18084 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.719713 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
W20260709 12:13:53.724817 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69496.3us
I20260709 12:13:53.644964 18073 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.714177 12566 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:53.717401 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:53.766419 12490 raft_consensus.cc:606] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:53.767174 12490 raft_consensus.cc:614] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:53.709682 12641 rpcz_store.cc:276] 0709 12:13:52.508262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.509169 (+   907us) service_pool.cc:224] Handling call
0709 12:13:52.509475 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:52.509893 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:52.509915 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:52.510361 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:13:52.629535 (+119174us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:52.629600 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:52.629745 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:53.633474 (+1003729us) raft_consensus.cc:1685] finished
0709 12:13:53.633524 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:53.633793 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1060,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":3120,"thread_start_us":101373,"threads_started":1,"wal-append.queue_time_us":182373}
W20260709 12:13:53.777140 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51971.4us
I20260709 12:13:53.707046 12491 rpcz_store.cc:276] 0709 12:13:52.589388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.594136 (+  4748us) service_pool.cc:224] Handling call
0709 12:13:52.594469 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:52.594690 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:13:53.255631 (+660941us) log.cc:844] Serialized 14 byte log entry
0709 12:13:53.493810 (+238179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:53.507660 (+ 13850us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:53.548267 (+ 40607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.603780 (+ 55513us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:53.645163 (+ 41383us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":684637,"prepare.queue_time_us":716607,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":6878,"raft.queue_time_us":42158,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":5108,"thread_start_us":96725,"threads_started":1}
I20260709 12:13:53.746891 17824 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.745191 17788 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:53.763691 18107 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:53.795904 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69374.7us
I20260709 12:13:53.780777 17914 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:53.758498 18037 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.792217 17788 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:53.811354 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:53.818084 12637 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:53.819051 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2508) took 1024 ms. Trace:
I20260709 12:13:53.819443 12637 rpcz_store.cc:276] 0709 12:13:52.794440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.711484 (+917044us) service_pool.cc:224] Handling call
0709 12:13:53.736696 (+ 25212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.818702 (+ 82006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.818986 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:53.825999 12641 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:13:53.830480 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:53.835243 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:53.835628 18107 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.836169 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:53.836323 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:13:53.839455 18107 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.841804 17564 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.842265 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.843400 13254 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:53.843541 12566 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:13:53.845048 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:53.847375 18081 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:53.850818 17884 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.009s
W20260709 12:13:53.851490 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:53.851926 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:53.852705 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:53.853286 18050 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:53.853940 18010 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.854210 18120 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:53.854797 18010 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:53.854907 18120 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:53.855413 17995 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.855816 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:53.856977 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:13:53.857690 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2265) took 1489 ms (client timeout 1723 ms). Trace:
I20260709 12:13:53.855647 18120 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:53.858176 12643 rpcz_store.cc:269] 0709 12:13:52.368215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.414052 (+ 45837us) service_pool.cc:224] Handling call
0709 12:13:53.857455 (+1443403us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.857591 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3258743040}
I20260709 12:13:53.859853 18084 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:53.860648 12646 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.5-14.5]
I20260709 12:13:53.861436 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2509) took 1047 ms. Trace:
I20260709 12:13:53.861775 12646 rpcz_store.cc:276] 0709 12:13:52.814290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.769949 (+955659us) service_pool.cc:224] Handling call
0709 12:13:53.812304 (+ 42355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:53.861139 (+ 48835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.861378 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:53.863320 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2504) took 1141 ms. Trace:
I20260709 12:13:53.863541 12646 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:53.863740 12638 rpcz_store.cc:276] 0709 12:13:52.721617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.626098 (+904481us) service_pool.cc:224] Handling call
0709 12:13:53.690984 (+ 64886us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.710677 (+ 19693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.863089 (+152412us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b54000a82d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.863239 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65375488}
I20260709 12:13:53.864895 12646 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:53.865105 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:53.866339 12563 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:53.851624 18014 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:53.835795 17914 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:53.860064 18086 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.002s
W20260709 12:13:53.871136 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 337489us
I20260709 12:13:53.865167 12638 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:53.880964 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:53.881384 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79946.1us
I20260709 12:13:53.824190 17788 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:53.897890 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:53.897994 18033 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:13:53.899680 18118 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:13:53.900858 18037 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.902176 12641 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 5
I20260709 12:13:53.902601 12641 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:13:53.902901 12641 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:53.855134 18073 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.892673 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:53.940496 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 277145us
I20260709 12:13:53.905545 17859 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.943691 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:53.855430 18010 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:53.944814 18010 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:53.893307 18086 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.946456 12640 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:53.948915 17859 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:53.950763 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:53.951881 12493 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:53.953671 12642 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:53.954500 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 680) took 1446 ms. Trace:
I20260709 12:13:53.954871 12642 rpcz_store.cc:276] 0709 12:13:52.508547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.952833 (+1444286us) service_pool.cc:224] Handling call
0709 12:13:53.953320 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.954214 (+   894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.954437 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:53.957094 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:53.958149 12489 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:53.959671 17950 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:53.960929 18131 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.111s	user 0.003s	sys 0.000s
I20260709 12:13:53.962477 17849 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.905879 18014 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.967193 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:53.968446 18014 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:53.915855 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:53.913314 17914 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.917568 17867 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:53.924780 17799 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:53.931854 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 214494us
I20260709 12:13:53.972124 17799 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:53.932277 18127 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:53.973742 18127 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:53.974166 18127 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:53.970296 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:53.933831 18050 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:53.968847 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2497) took 1436 ms. Trace:
I20260709 12:13:53.934988 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:53.922187 12639 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:13:53.978111 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2510) took 1128 ms. Trace:
I20260709 12:13:53.932951 12569 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:53.975934 12640 rpcz_store.cc:276] 0709 12:13:52.532530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.297089 (+764559us) service_pool.cc:224] Handling call
0709 12:13:53.968643 (+671554us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:53.968786 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1489353600}
W20260709 12:13:53.916265 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43557.4us
I20260709 12:13:53.978399 12639 rpcz_store.cc:276] 0709 12:13:52.850233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.777046 (+926813us) service_pool.cc:224] Handling call
0709 12:13:53.872260 (+ 95214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:53.977838 (+105578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:53.978054 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:13:53.988602 12488 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.564s	user 0.014s	sys 0.000s
I20260709 12:13:53.989136 12488 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:53.990983 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:53.996260 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:13:53.975409 17799 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.004392 17914 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:53.895783 17639 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.006187 12496 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:54.007472 17639 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:53.984673 18050 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.008451 18050 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.018151 12646 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:13:54.018859 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:54.018936 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:54.020180 17867 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:54.020601 17867 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:54.021010 17867 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:54.029779 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.030480 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:54.030970 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54474.8us
I20260709 12:13:54.032013 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.032826 17849 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:54.033876 17227 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:54.034293 17227 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:54.034606 17897 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:54.035593 17897 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.035717 17476 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:54.036070 17897 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:54.037042 18014 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.029815 18092 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.038942 17919 raft_consensus.cc:993] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:54.037881 18014 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.041416 18092 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.034727 17227 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:54.041910 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.042563 12563 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:54.043398 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.044214 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:54.044409 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100825us
I20260709 12:13:54.041529 18014 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.033391 17849 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:54.036267 17476 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:54.045887 17476 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:54.046320 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58091.5us
I20260709 12:13:54.046412 17849 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:53.983870 18081 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:54.031981 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.009224 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:54.018086 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:53.871596 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98763.3us
I20260709 12:13:53.990420 18085 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:54.060597 12489 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:54.045606 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.039693 17919 raft_consensus.cc:1081] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:54.063015 13259 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:54.063220 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:54.068779 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.068886 13255 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:13:54.071235 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.028903 18120 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.001449 17788 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.088881 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:54.089499 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:54.091758 18093 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.078722 17914 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.096920 12494 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:54.060755 18014 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:13:54.100906 12497 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:54.101435 18086 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:54.102949 17870 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.004s
I20260709 12:13:54.063616 17848 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:13:54.119953 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:53.918242 18107 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.160s
I20260709 12:13:54.149035 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.158985 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.159905 12488 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:54.164088 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:54.164467 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:54.091823 18027 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.185459 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:54.177560 12644 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:54.194067 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.190938 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.164453 18085 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:54.202314 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:54.225391 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:54.225612 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.226231 12565 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:13:54.226411 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.229177 18086 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:54.229864 18086 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:54.229965 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.230630 12563 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:54.230820 18086 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.231884 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.232306 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:54.232451 12563 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.156021 17870 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.237600 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.241559 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.210773 18085 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:54.240504 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 151802us
I20260709 12:13:54.192927 17914 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.256409 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:54.256596 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:54.261917 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:54.264101 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48416.8us
I20260709 12:13:54.265414 12645 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:54.264912 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.261807 12491 raft_consensus.cc:606] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:54.255195 18085 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.281489 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.283922 12569 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.257799 18092 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.263569 18073 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:54.290548 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.304188 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.313354 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.314946 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:54.316500 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2625) took 1835 ms. Trace:
I20260709 12:13:54.253214 18127 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:54.330241 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 146277us
I20260709 12:13:54.333165 18124 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.317353 12572 rpcz_store.cc:276] 0709 12:13:52.356175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:52.361538 (+  5363us) service_pool.cc:224] Handling call
0709 12:13:52.362014 (+   476us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:52.362560 (+   546us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:52.362934 (+   374us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.676115 (+313181us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.676613 (+   498us) log.cc:844] Serialized 14 byte log entry
0709 12:13:52.817610 (+140997us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b180044f980. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:53.055588 (+237978us) spinlock_profiling.cc:243] Waited 8.41 ms on lock 0x7b180044f980. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:13:53.297249 (+241661us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b180044f980. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:13:53.662769 (+365520us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:53.795919 (+133150us) log.cc:844] Serialized 52 byte log entry
0709 12:13:53.851899 (+ 55980us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:53.940969 (+ 89070us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b5400001f20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5768dbe 00007ffbf5773930 00007ffbf57738d8 00007ffbf5773858 00007ffbf5773813 00007ffbf5772a9c 00007ffbfb65e2f6 00007ffbfb658aa8 00007ffbf57af7cb 00007ffbf57b22c0
0709 12:13:54.153423 (+212454us) log.cc:844] Serialized 14 byte log entry
0709 12:13:54.167667 (+ 14244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:54.180496 (+ 12829us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:54.180563 (+    67us) raft_consensus.cc:1685] finished
0709 12:13:54.180582 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:54.191951 (+ 11369us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1107,"prepare.queue_time_us":377710,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":143,"raft.queue_time_us":63285,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":71903,"spinlock_wait_cycles":125395968,"thread_start_us":924674,"threads_started":3,"wal-append.queue_time_us":418029}
I20260709 12:13:54.337409 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.273026 18087 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:54.343863 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.295794 18033 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:13:54.354924 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 244636us
I20260709 12:13:54.356161 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.359419 17919 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.360347 17919 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.366178 17919 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.289992 17886 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:54.289696 18058 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.169s
I20260709 12:13:54.380992 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.341722 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.281839 18114 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.268s
I20260709 12:13:54.386907 13259 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.386654 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:54.388692 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47176.7us
I20260709 12:13:54.328439 18037 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:54.327240 17848 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:54.321763 17912 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:54.295737 18092 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.401892 12496 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:13:54.402285 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.402849 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.407239 12489 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:13:54.409734 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.312361 17227 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.413745 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
W20260709 12:13:54.415875 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 169873us
I20260709 12:13:54.392218 17788 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.398494 17848 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.399220 17849 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:54.395687 18033 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:54.461009 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:54.465090 12566 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:54.340893 18124 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.472783 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.473542 12572 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.475610 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.477375 17914 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.477576 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.478008 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.478653 12495 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:13:54.458202 18135 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.002s	sys 0.000s
I20260709 12:13:54.479154 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:54.484628 18033 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.488763 17788 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:54.489677 17788 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:54.490878 18139 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:13:54.495069 17824 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.417583 17886 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:54.502252 18138 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:13:54.503079 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 681) took 1994 ms. Trace:
I20260709 12:13:54.457487 18087 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.503487 12645 rpcz_store.cc:276] 0709 12:13:52.508797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.916351 (+1407554us) service_pool.cc:224] Handling call
0709 12:13:53.917021 (+   670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:54.265960 (+348939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:54.502728 (+236768us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:54.502981 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":800633728}
I20260709 12:13:54.504088 17886 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.504573 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2516) took 1555 ms. Trace:
W20260709 12:13:54.504935 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.418368 17867 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.505923 17897 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.507045 18127 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.507997 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:54.508266 17867 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.463194 17848 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:54.510387 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:54.510466 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:54.511513 18050 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.008s	sys 0.005s
I20260709 12:13:54.511871 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.513046 18086 raft_consensus.cc:2764] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election decision vote started in defunct term 18: won
W20260709 12:13:54.464327 18010 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.003s
I20260709 12:13:54.513921 18086 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:54.514003 18010 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.516321 18010 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.472160 18120 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.517781 12490 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.518661 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.521940 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:54.511971 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:54.512095 18050 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.414717 18037 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:54.532785 12567 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:54.534029 13259 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:54.505010 12643 rpcz_store.cc:276] 0709 12:13:52.949422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.859070 (+909648us) service_pool.cc:224] Handling call
0709 12:13:53.859612 (+   542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:53.860272 (+   660us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:53.860298 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:53.860891 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 12:13:54.176675 (+315784us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:54.176725 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:54.176809 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:54.504333 (+327524us) raft_consensus.cc:1685] finished
0709 12:13:54.504353 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:54.504506 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7982,"raft.queue_time_us":7075,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":228,"thread_start_us":315374,"threads_started":1,"wal-append.queue_time_us":393818}
I20260709 12:13:54.456758 17919 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:13:54.463984 17849 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.535595 12565 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.535908 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:54.538323 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 138735us
I20260709 12:13:54.538839 17476 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.546020 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:54.547700 18143 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:13:54.517781 18114 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.479506 17931 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.497334 17912 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.550911 18037 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.552116 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:13:54.554195 18124 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.554708 17897 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.533057 17556 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:54.620437 17227 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.552142 18092 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.568608 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2515) took 1637 ms. Trace:
W20260709 12:13:54.568728 18146 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.077s	user 0.003s	sys 0.000s
I20260709 12:13:54.571564 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.572467 17848 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:54.573560 18145 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:13:54.689019 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:54.574734 18144 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
W20260709 12:13:54.576839 18136 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.414s	user 0.003s	sys 0.000s
I20260709 12:13:54.577333 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.694370 13256 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:54.577886 17870 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.694650 12564 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12.
I20260709 12:13:54.622313 12563 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:54.696089 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.597131 12489 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:54.697125 12564 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:13:54.697355 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 185514us
I20260709 12:13:54.697552 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.698273 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.579746 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.698654 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.699102 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:54.699932 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.700026 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.700687 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:54.700929 12564 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:54.636698 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.702102 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.702734 12567 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:54.703143 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.639123 12496 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:13:54.703830 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:54.704161 12568 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:13:54.704411 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.704634 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:13:54.705243 12496 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:54.622745 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:54.705961 18027 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:13:54.624567 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147326us
I20260709 12:13:54.647058 12492 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:13:54.707365 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:13:54.573568 18147 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:13:54.650194 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.645898 12569 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.655820 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.709479 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:54.709957 12491 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 11.
I20260709 12:13:54.710035 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2513) took 1794 ms. Trace:
I20260709 12:13:54.622399 18050 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.711088 18050 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:54.710510 12637 rpcz_store.cc:276] 0709 12:13:52.915796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.820361 (+904565us) service_pool.cc:224] Handling call
0709 12:13:53.932587 (+112226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:53.933386 (+   799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:53.933424 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:53.934161 (+   737us) log.cc:844] Serialized 26 byte log entry
0709 12:13:54.215126 (+280965us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:54.215179 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:54.215257 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:54.572639 (+357382us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:54.709704 (+137065us) raft_consensus.cc:1685] finished
0709 12:13:54.709725 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:54.709961 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1902,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":172,"spinlock_wait_cycles":36554624,"thread_start_us":279235,"threads_started":1,"wal-append.queue_time_us":407139}
I20260709 12:13:54.620323 17919 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:54.713438 17919 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.666833 17931 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.715042 17919 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.716519 17931 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.656919 12494 raft_consensus.cc:606] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:13:54.664008 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.660153 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.660403 18058 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:54.639219 17849 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:54.660048 18114 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.719849 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.701296 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.630179 17788 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:54.644785 18152 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:13:54.682705 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:54.657469 17886 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.574116 17799 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:54.696148 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.698366 12489 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.703270 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:54.724166 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.729022 12564 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
I20260709 12:13:54.721491 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:54.706429 18027 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.708339 17912 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.639576 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:54.731814 12565 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:13:54.732177 12488 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:54.565579 18093 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.591679 17824 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:54.720824 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.625196 18087 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.705495 12568 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:54.690255 18086 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:54.672008 12641 rpcz_store.cc:276] 0709 12:13:52.931066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:53.834347 (+903281us) service_pool.cc:224] Handling call
0709 12:13:53.834958 (+   611us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:53.903796 (+ 68838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:53.903832 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:53.904613 (+   781us) log.cc:844] Serialized 52 byte log entry
0709 12:13:54.237402 (+332789us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:13:54.237511 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:54.482197 (+244686us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b54002e5754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:54.482258 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:54.568251 (+ 85993us) raft_consensus.cc:1685] finished
0709 12:13:54.568274 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:54.568525 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1742,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":168,"spinlock_wait_cycles":113633152,"thread_start_us":330839,"threads_started":1,"wal-append.queue_time_us":661207}
I20260709 12:13:54.723265 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:54.639014 18085 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.734532 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:54.728430 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.738481 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:54.747980 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:13:54.751470 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 211322us
I20260709 12:13:54.739997 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:54.727685 18092 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.745893 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.740336 12565 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 16. Ops: []
I20260709 12:13:54.718295 17476 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.758991 12497 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 17
W20260709 12:13:54.759599 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 365779us
I20260709 12:13:54.761559 12571 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:54.739239 17995 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:54.804698 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47098.9us
I20260709 12:13:54.805444 12567 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:54.733872 18027 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.765173 12563 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:54.799074 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53276.9us
I20260709 12:13:54.807010 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:54.810976 12637 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.745180 17824 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:54.742609 17931 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:54.815797 13217 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
I20260709 12:13:54.760396 18081 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:54.818325 13217 leader_election.cc:403] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:13:54.745813 17926 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:54.822271 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:54.820729 12571 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:54.827997 12643 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:54.828557 12565 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:13:54.758147 17848 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:54.830567 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148079us
I20260709 12:13:54.817793 17870 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:54.836297 17870 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.821692 17824 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:54.839253 17870 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:54.808744 18087 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:54.841271 18087 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:54.843721 18087 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.584007 12389 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 13 to 14, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 14 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:54.804090 18120 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:54.755985 17950 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:54.846205 18120 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.846899 18120 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:54.846560 17950 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.841889 17931 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:54.850173 18114 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.811074 17995 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:54.851094 17995 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:54.850606 17931 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.853781 17926 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.757954 17799 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:54.855629 17799 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.4, Last appended by leader: 4, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.856664 12572 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:13:54.857484 12572 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:13:54.857894 12572 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:54.853899 18081 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:54.854494 17926 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:54.866300 12572 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:54.870972 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 129538us
I20260709 12:13:54.872118 12567 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:54.873124 12563 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:54.866881 13254 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:54.874714 18158 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:54.875365 18158 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:54.875834 18158 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:54.877975 17923 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
W20260709 12:13:54.879441 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46951.2us
I20260709 12:13:54.867715 12637 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:54.849520 17950 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:54.863368 13257 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:54.733453 17886 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.887286 12563 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:54.890684 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:54.894071 12568 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:54.899736 12497 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:13:54.911250 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.926946 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:54.927932 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.818482 18092 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:54.932948 12572 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:54.940676 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 233453us
I20260709 12:13:54.946627 12489 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:54.899564 18114 raft_consensus.cc:2764] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
W20260709 12:13:54.929797 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.058s (SENT) [suppressed 37 similar messages]
I20260709 12:13:54.961351 17886 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:54.963239 18092 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:54.964176 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.966602 18076 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:54.970741 12572 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:54.976070 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:54.981226 12497 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
W20260709 12:13:54.989353 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:54.930328 17848 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:54.877257 18081 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:54.994113 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:55.003599 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90252us
W20260709 12:13:55.004487 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50860.4us
I20260709 12:13:55.005330 12572 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:13:54.900436 17923 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.074s
W20260709 12:13:55.016319 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.049950 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.053531 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:55.065660 17848 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:55.051852 18092 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:13:54.973729 18120 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.114291 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.114607 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.001106 17995 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.117015 18014 raft_consensus.cc:2764] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260709 12:13:55.118314 18161 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:13:55.123119 17848 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.125613 17788 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:55.127866 18120 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.127698 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.128593 18085 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.159897 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:55.160656 13255 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.075541 18092 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:55.130867 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2655) took 1036 ms. Trace:
I20260709 12:13:55.133740 12572 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:55.051223 17870 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
W20260709 12:13:55.148847 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78792.5us
I20260709 12:13:55.174372 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:55.169232 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
W20260709 12:13:55.174451 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.170810 12568 rpcz_store.cc:276] 0709 12:13:54.094057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:54.836117 (+742060us) service_pool.cc:224] Handling call
0709 12:13:54.886222 (+ 50105us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:54.910757 (+ 24535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:55.130640 (+219883us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x7b54003e1050. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:55.130804 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":114483968}
I20260709 12:13:55.185518 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.186326 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:55.188179 18129 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:13:55.191039 18162 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:55.191974 12609 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12.
I20260709 12:13:55.192423 12609 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:13:55.193833 18162 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:55.195016 18162 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.199334 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.199715 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.207554 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2677) took 1281 ms. Trace:
I20260709 12:13:55.208043 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.188256 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.175746 18092 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:55.211866 17886 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:13:55.210923 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.212864 18050 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:55.213127 12564 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:55.222002 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.221364 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.224262 12568 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:55.233562 17926 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.206671 17870 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:55.258170 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.258852 12567 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 17
W20260709 12:13:55.240866 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55469.1us
I20260709 12:13:55.207913 12566 rpcz_store.cc:276] 0709 12:13:53.926230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:54.709279 (+783049us) service_pool.cc:224] Handling call
0709 12:13:54.843293 (+134014us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:13:54.843540 (+   247us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:54.844098 (+   558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:54.844132 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:54.844799 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:13:55.019829 (+175030us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:55.118878 (+ 99049us) log.cc:844] Serialized 14 byte log entry
0709 12:13:55.119138 (+   260us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:55.119229 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:55.207181 (+ 87952us) raft_consensus.cc:1685] finished
0709 12:13:55.207307 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:55.207479 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68957,"prepare.queue_time_us":99328,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":17537,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":14330,"spinlock_wait_cycles":34024704,"thread_start_us":174499,"threads_started":1,"wal-append.queue_time_us":312548}
I20260709 12:13:55.262257 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.263937 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.261384 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:55.233014 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.266822 12565 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:13:55.266017 12389 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:55.234228 17914 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:55.268239 13256 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:55.266100 17950 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:55.269748 17950 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:55.270219 17950 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.271157 12497 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:13:55.272858 12493 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:55.276901 17919 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:55.277766 17919 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.279829 18033 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:55.240674 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:55.282310 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.241334 18085 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:55.246061 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.279825 17919 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:55.284489 12565 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:55.287043 17859 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:55.287858 17859 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.290670 17931 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:55.291263 17931 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.235051 18027 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.293946 12637 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:13:55.245859 12563 raft_consensus.cc:2376] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17. Re-sending same reply.
I20260709 12:13:55.247427 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.248004 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.250952 17995 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:55.151253 17788 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:55.309091 17788 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.309697 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.310387 12639 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:55.310654 17788 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:55.254760 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.255085 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.226742 18172 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:55.311801 18172 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:55.313591 18172 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:55.257133 17849 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.538s	user 0.002s	sys 0.008s
I20260709 12:13:55.279982 17914 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:13:55.315539 17849 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.251910 17886 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:55.317117 13257 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13.
I20260709 12:13:55.317873 12568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.317440 17886 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.320398 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.235236 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.323024 17914 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:13:55.323513 17914 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:55.323912 17914 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:55.321408 17849 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:55.326534 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65593.7us
I20260709 12:13:55.276706 12389 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:55.329862 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.329272 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:55.338140 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102990us
I20260709 12:13:55.346810 13255 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:13:55.346195 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:55.347488 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.333015 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.351550 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:55.346925 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 457583us
I20260709 12:13:55.357808 18120 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:55.358186 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:55.359602 12566 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:55.360548 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.352953 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:55.359438 12497 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:55.375842 12564 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:55.377323 12567 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:55.366729 18158 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.399658 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.399921 12572 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:55.404966 12569 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:13:55.366340 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 222) took 48246 ms (client timeout 15000 ms). Trace:
I20260709 12:13:55.401878 12462 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13.
I20260709 12:13:55.406502 12462 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:13:55.403275 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.367457 18014 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:55.409493 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.358915 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:55.411862 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.416155 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.417045 12564 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
W20260709 12:13:55.427198 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74348.4us
I20260709 12:13:55.429742 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:55.423806 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.057s	sys 0.081s
W20260709 12:13:55.452864 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.057s	sys 0.081s
I20260709 12:13:55.371057 17926 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:55.453222 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 571865us
I20260709 12:13:55.366079 18033 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:55.454412 12569 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:13:55.455233 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.462555 12637 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:55.364190 17859 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.329262 17824 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.457219 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:55.472769 13259 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:55.360493 18027 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.460670 18114 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:55.415701 17556 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:55.488286 18081 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:55.490129 12645 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:55.460439 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:55.493216 12497 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:13:55.471235 18037 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.005s	sys 0.006s
I20260709 12:13:55.453110 17867 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 13
W20260709 12:13:55.505761 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45459.3us
I20260709 12:13:55.500427 17926 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:13:55.453576 17850 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:13:55.498618 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 138184us
I20260709 12:13:55.453944 18014 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:55.494172 13257 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:13:55.504192 13254 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:55.360529 12389 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:13:55.503104 18037 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.006s	sys 0.006s
I20260709 12:13:55.550405 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
W20260709 12:13:55.551652 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50376.4us
I20260709 12:13:55.499361 17556 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:55.563668 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.488627 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:13:55.564647 12490 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:55.505149 18114 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.569911 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:55.554787 18164 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:13:55.570991 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:55.573066 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68943.5us
I20260709 12:13:55.581930 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.582808 12497 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:55.582645 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.584704 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.585602 12495 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:55.506183 17950 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.566512 17556 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:55.594839 12494 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:13:55.596830 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.599005 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:55.505445 18081 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.601866 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.606276 12497 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:13:55.606061 17564 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.005s	sys 0.017s
I20260709 12:13:55.606737 18014 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.610581 18092 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.611105 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.612059 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.613511 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:55.616256 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.618513 18178 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:55.623701 18158 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.604637 18037 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:13:55.630780 12494 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:13:55.664464 12497 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:55.633127 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.667533 12490 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:13:55.595077 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.605350 17886 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.674232 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.675487 12494 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:55.680038 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85056.5us
W20260709 12:13:55.680708 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 251673us
I20260709 12:13:55.681141 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.681459 12565 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
W20260709 12:13:55.682752 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.605036 17931 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.683641 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:13:55.642606 17564 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.006s	sys 0.017s
I20260709 12:13:55.666416 18177 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:13:55.690730 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 225946us
I20260709 12:13:55.700030 13217 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
I20260709 12:13:55.700495 13217 leader_election.cc:403] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:13:55.706542 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.706833 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.707340 12491 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:55.701193 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:55.707963 12491 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.714246 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:55.715250 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:13:55.714596 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207024us
I20260709 12:13:55.718678 12563 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.686754 17886 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:55.746938 17919 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:13:55.710569 17850 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:13:55.748881 17850 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:13:55.749655 17850 consensus_queue.cc:1243] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:13:55.750944 17850 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:55.754002 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.754433 17919 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:13:55.754668 12490 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.755065 17919 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:55.755461 17919 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.756372 17931 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:13:55.753276 18107 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 12:13:55.758180 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:55.758280 18107 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:55.756562 17919 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.758679 17931 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:55.702039 18162 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:13:55.405753 12395 rpcz_store.cc:269] 0709 12:13:07.120300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:07.120539 (+   239us) service_pool.cc:224] Handling call
0709 12:13:55.366078 (+48245539us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:48.889390 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:48.906956 (+ 17566us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:48.907013 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:13:48.907036 (+    23us) tablet.cc:662] Decoding operations
0709 12:13:48.908200 (+  1164us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:48.908293 (+    93us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:48.908323 (+    30us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:13:48.909417 (+  1094us) tablet.cc:701] Row locks acquired
0709 12:13:48.909471 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:13:48.922024 (+ 12553us) write_op.cc:270] Start()
0709 12:13:48.943619 (+ 21595us) write_op.cc:276] Timestamp: P: 1783599228909569 usec, L: 0
0709 12:13:48.946373 (+  2754us) op_driver.cc:348] REPLICATION: starting
0709 12:13:48.948775 (+  2402us) log.cc:844] Serialized 4791 byte log entry
0709 12:13:49.665837 (+717062us) op_driver.cc:464] REPLICATION: finished
0709 12:13:50.177663 (+511826us) write_op.cc:301] APPLY: starting
0709 12:13:50.217314 (+ 39651us) tablet.cc:1367] starting BulkCheckPresence
0709 12:13:51.362750 (+1145436us) tablet.cc:1370] finished BulkCheckPresence
0709 12:13:51.362926 (+   176us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:13:51.699908 (+336982us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:13:51.738941 (+ 39033us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=0,mrs_lookups=0
0709 12:13:51.739004 (+    63us) write_op.cc:312] APPLY: finished
0709 12:13:51.739459 (+   455us) log.cc:844] Serialized 113 byte log entry
0709 12:13:51.869334 (+129875us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:51.901459 (+ 32125us) write_op.cc:454] Released schema lock
0709 12:13:51.901547 (+    88us) write_op.cc:341] FINISH: Updating metrics
0709 12:13:51.904964 (+  3417us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:13:51.904999 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2697305,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":447422,"apply.run_cpu_time_us":42651,"apply.run_wall_time_us":1774918,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54239,"num_ops":12,"prepare.queue_time_us":190525,"prepare.run_cpu_time_us":25439,"prepare.run_wall_time_us":200577,"raft.queue_time_us":431330,"raft.run_cpu_time_us":30379,"raft.run_wall_time_us":150915,"replication_time_us":717562,"thread_start_us":699320,"threads_started":5,"wal-append.queue_time_us":361854,"wal-append.run_cpu_time_us":16705,"wal-append.run_wall_time_us":2464168}]]}
I20260709 12:13:55.760025 12494 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:55.763870 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:55.764695 13259 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:55.715317 12491 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:55.767040 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.702392 18120 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:55.765843 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48982.6us
I20260709 12:13:55.775926 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:55.713665 18176 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:55.710754 18050 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:55.716336 18027 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:55.783479 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:55.744717 18177 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:55.696430 18158 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:55.793479 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91864us
I20260709 12:13:55.762703 18085 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:55.794088 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:13:55.761945 17931 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.763624 18162 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:55.823347 18162 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:55.823889 18172 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:55.813769 18190 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:55.826144 18190 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.827247 18190 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.828739 18190 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:55.832628 13259 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:55.798386 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.791083 17850 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:13:55.814162 18050 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.772725 17886 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:55.806136 18176 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:55.817833 17919 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.818636 18081 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:55.819136 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:55.819741 13256 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:55.820010 18177 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.758736 12389 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:55.822319 17931 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.879009 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.879632 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:55.834084 17788 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.523s	user 0.015s	sys 0.000s
I20260709 12:13:55.839453 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.842689 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:55.881980 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.883044 12638 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:55.883905 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.884905 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.884963 12497 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:55.886034 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:55.852125 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:55.852861 12494 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:55.890203 12645 raft_consensus.cc:2376] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15. Re-sending same reply.
I20260709 12:13:55.707623 17923 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:55.853142 18194 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:55.859189 12563 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
W20260709 12:13:55.863174 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141535us
I20260709 12:13:55.812037 17859 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:13:55.874665 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76387.4us
I20260709 12:13:55.898430 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:55.845369 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:55.880879 12646 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:55.835274 17564 raft_consensus.cc:2764] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:13:55.898531 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:55.904537 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:55.789232 18120 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.764837 18114 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.978462 17886 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.836146 18085 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.982231 18120 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.983703 18085 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:55.984680 17867 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13.
I20260709 12:13:55.985841 12641 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:55.873814 18027 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:55.989717 17950 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:55.990448 17897 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:55.990183 18027 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:55.991071 17897 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.992151 17897 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:55.992992 18027 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:55.877156 18073 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:55.985877 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86653.1us
I20260709 12:13:55.908200 12497 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:13:55.994890 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:55.998248 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:13:56.003355 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:55.930574 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:55.933773 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.007949 12639 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
W20260709 12:13:55.945986 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:56.008791 12495 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:55.901650 17859 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.859405 18158 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:55.902659 18037 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:55.909843 18033 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:56.021113 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.025612 12639 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:56.026886 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.015937 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.028504 12637 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:56.027570 12639 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16. Re-sending same reply.
I20260709 12:13:56.036993 12490 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
W20260709 12:13:56.039608 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 289502us
W20260709 12:13:56.040683 12496 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.016s	sys 0.016s
W20260709 12:13:56.041081 12496 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.016s	sys 0.016s
I20260709 12:13:56.046571 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.050783 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.021847 17848 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:56.071376 12641 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:13:56.058290 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149992us
I20260709 12:13:56.059389 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2845) took 1418 ms. Trace:
I20260709 12:13:56.057116 12644 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:13:56.060139 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161809us
I20260709 12:13:56.081377 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.077603 12637 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:56.014761 18073 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.084435 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.085722 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.089206 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:56.039245 17788 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:56.086550 17931 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:56.078853 12496 rpcz_store.cc:276] 0709 12:13:54.640556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:54.706032 (+ 65476us) service_pool.cc:224] Handling call
0709 12:13:54.706435 (+   403us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:54.706891 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:54.706921 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:54.718273 (+ 11352us) log.cc:844] Serialized 52 byte log entry
0709 12:13:54.825876 (+107603us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:55.371627 (+545751us) log.cc:844] Serialized 14 byte log entry
0709 12:13:56.059011 (+687384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:56.059108 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:56.059159 (+    51us) raft_consensus.cc:1685] finished
0709 12:13:56.059171 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:56.059326 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27720,"prepare.queue_time_us":84755,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":112869,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":8363264,"thread_start_us":1108951,"threads_started":3,"wal-append.queue_time_us":185138}
W20260709 12:13:56.078269 12536 connection.cc:256] RPC call timeout handler was delayed by -1.83611557800009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260709 12:13:56.093173 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.014832 17867 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:13:56.039500 18196 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:13:56.100189 12646 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 12:13:56.102067 18186 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:13:56.102303 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:56.103098 18195 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:13:56.103538 18190 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:56.058337 18092 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:56.084278 12639 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:56.107247 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:56.051517 18037 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.108115 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:13:56.088104 18161 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
W20260709 12:13:56.157433 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:56.160297 12572 pending_rounds.cc:85] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:13:56.160846 12572 pending_rounds.cc:107] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:13:56.161199 12572 raft_consensus.cc:2889] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:13:56.126698 12644 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
W20260709 12:13:56.117365 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4374155780001274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260709 12:13:56.162761 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3035652739999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
W20260709 12:13:56.163266 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2984518670000398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
W20260709 12:13:56.163513 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1493864609999491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
W20260709 12:13:56.163883 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0953864610000892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
W20260709 12:13:56.164283 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0809864610000659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20260709 12:13:56.100420 18182 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 12:13:56.166038 18190 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:56.167639 18189 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.001s	sys 0.001s
I20260709 12:13:56.169849 17639 raft_consensus.cc:993] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:56.170367 17867 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:56.171900 18037 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:56.172577 12491 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:56.174517 18073 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:56.176023 12638 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:56.177670 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.178463 12638 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 11: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:56.179256 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.179967 12638 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:56.085923 17848 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:56.142716 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 147888us
W20260709 12:13:56.187485 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50051.4us
I20260709 12:13:56.188820 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.196064 12639 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:56.137804 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:56.196615 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.190004 12567 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:13:56.147787 17870 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.886s	user 0.015s	sys 0.005s
I20260709 12:13:56.202775 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.199576 12641 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:13:56.200484 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:13:56.201136 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 302690us
I20260709 12:13:56.204265 12646 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:56.179148 18050 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:56.219350 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.219350 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.223261 12644 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:56.169351 18190 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:56.082718 18176 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:56.229748 12637 raft_consensus.cc:2376] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14. Re-sending same reply.
I20260709 12:13:56.236487 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:56.243103 18197 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:56.243629 18196 raft_consensus.cc:1081] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:56.248106 12567 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:13:56.262017 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:56.263218 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.264509 12646 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:56.265823 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:56.266366 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.202183 17867 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:56.219815 17870 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12.
W20260709 12:13:56.234108 17556 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.627s	user 0.013s	sys 0.000s
I20260709 12:13:56.114487 18158 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.228466 17995 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:56.304203 12643 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:56.240113 18194 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:13:56.310938 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56144.1us
I20260709 12:13:56.247892 18210 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:56.243723 18197 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:56.266713 18014 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:56.285420 18050 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:56.305291 17923 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
I20260709 12:13:56.366240 12491 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:56.315565 18085 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:56.313923 17995 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:56.286250 18073 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:13:56.320909 18199 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:56.363029 17870 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:56.357270 18194 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:56.302713 17556 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.357020 18092 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:13:56.378563 18172 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.554s	user 0.015s	sys 0.002s
I20260709 12:13:56.392877 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:56.373082 18085 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:56.434006 13257 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:56.434672 13257 raft_consensus.cc:614] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:13:56.357437 18014 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:56.479696 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98028.4us
I20260709 12:13:56.452780 18085 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:56.631307 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.633462 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:56.635566 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 154894us
I20260709 12:13:56.636396 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:56.435293 17870 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:56.461853 18120 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:56.650910 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 388911us
I20260709 12:13:56.640859 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:56.647550 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:56.646072 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:13:56.510092 17914 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.186s	user 0.011s	sys 0.011s
I20260709 12:13:56.388646 18199 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:56.667068 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.668407 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.669179 12565 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:56.670863 12563 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:56.109434 12389 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 3 to 7, term changed from 11 to 14, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to <none>, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129), VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 14 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:13:56.675964 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.679535 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.680342 12569 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:56.681262 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:56.433068 18092 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:56.683172 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 436177us
I20260709 12:13:56.393584 17867 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:56.676632 18162 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.849s	user 0.012s	sys 0.001s
I20260709 12:13:56.699733 18210 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:56.702387 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.698271 18222 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:56.651350 18213 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:56.704655 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.705998 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.714845 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.714924 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.725692 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:56.727140 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.728428 12565 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:56.727329 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.461190 18197 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.744668 13258 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 11: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:56.737763 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.669898 18120 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:56.672535 17914 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.658263 17995 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:56.739786 12646 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:56.638072 18194 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:56.764251 12639 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.562s	user 0.006s	sys 0.002s
W20260709 12:13:56.768100 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131764us
I20260709 12:13:56.768421 12639 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:56.758438 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.712706 18162 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.770772 13258 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:56.747769 17848 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:56.787496 12642 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:13:56.783213 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.788688 18227 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:56.789840 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:56.789569 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:56.793706 12567 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:13:56.794153 12567 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:56.794203 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.734684 18177 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:13:56.802951 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:56.798959 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:56.800045 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37834 (request call id 10) took 1391 ms. Trace:
I20260709 12:13:56.806248 12565 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:13:56.805811 13257 rpcz_store.cc:276] 0709 12:13:55.448787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.457987 (+  9200us) service_pool.cc:224] Handling call
0709 12:13:56.791314 (+1333327us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:56.791451 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2025363840}
I20260709 12:13:56.814478 12638 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:56.838429 18217 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:13:56.807435 12389 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 10 to 13, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 13 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:56.716993 18196 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
W20260709 12:13:56.788718 18215 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260709 12:13:56.852708 13256 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
W20260709 12:13:56.853760 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.011s	sys 0.015s
W20260709 12:13:56.854503 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.011s	sys 0.015s
W20260709 12:13:56.850972 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 146611us
I20260709 12:13:56.849141 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:56.855233 12566 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:13:56.854919 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 652161us
W20260709 12:13:56.857213 12640 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.007s	sys 0.002s
W20260709 12:13:56.859141 12640 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.007s	sys 0.002s
W20260709 12:13:56.857134 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54294us
I20260709 12:13:56.860020 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 697) took 1011 ms. Trace:
W20260709 12:13:56.858701 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.004s	sys 0.005s
W20260709 12:13:56.860630 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.004s	sys 0.006s
W20260709 12:13:56.831248 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59714.6us
I20260709 12:13:56.860423 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:56.840029 12389 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 9, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 9 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:13:56.862166 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 484902us
I20260709 12:13:56.722069 18213 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:56.744419 18200 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.385s
I20260709 12:13:56.792711 18227 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:56.869481 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:56.870335 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:56.872525 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.880065 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:56.846212 18014 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:56.882398 13258 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:56.874315 18199 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:56.884347 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:56.885509 13257 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:56.887142 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:56.853725 17556 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:56.879383 17931 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.764s	user 0.009s	sys 0.005s
I20260709 12:13:56.863592 17848 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:13:56.720131 18210 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.720652 18222 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.913228 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:56.874954 18037 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:56.860436 12640 rpcz_store.cc:276] 0709 12:13:55.848913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.858629 (+  9716us) service_pool.cc:224] Handling call
0709 12:13:55.896468 (+ 37839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:55.896776 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:13:56.116442 (+219666us) log.cc:844] Serialized 14 byte log entry
0709 12:13:56.859547 (+743105us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:56.859626 (+    79us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:56.859674 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:56.859765 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:56.859955 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14851,"prepare.queue_time_us":389580,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":69764,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":8429,"thread_start_us":838093,"threads_started":3,"wal-append.queue_time_us":259766}
W20260709 12:13:56.902910 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.017s	sys 0.032s
I20260709 12:13:56.883028 18227 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:56.902547 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:56.905254 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:56.925112 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.017s	sys 0.032s
I20260709 12:13:56.899318 18014 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:56.955010 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:13:56.954056 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:56.904490 12567 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:56.900238 18230 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:56.884145 18190 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:13:56.868434 17897 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:13:56.786935 17870 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:56.961697 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1430754120001438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
I20260709 12:13:56.907768 13253 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:56.921367 18050 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:56.966962 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:56.977926 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116134us
I20260709 12:13:56.922319 18240 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:56.956859 17848 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:56.981243 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111974us
I20260709 12:13:56.970108 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:56.978416 13218 connection.cc:256] RPC call timeout handler was delayed by -1.0521754119999969s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16200000000000003s for a response
I20260709 12:13:57.004616 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:56.892266 18220 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 12:13:56.991930 12565 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:13:56.992890 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:56.992929 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:56.968864 18014 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:56.999154 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:56.971333 18092 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:56.978998 18231 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.016011 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.002910 12640 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:57.023661 12568 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:57.023838 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:57.027458 12565 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:13:57.020255 18033 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.986s	user 0.010s	sys 0.005s
I20260709 12:13:57.031788 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:57.017467 18014 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:57.039237 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:57.035636 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 122478us
I20260709 12:13:57.001250 17848 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:57.038306 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:57.036691 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.037123 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:57.037498 12536 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:13:57.037842 18037 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:57.019378 18176 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:57.052834 12536 leader_election.cc:403] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:13:57.053016 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:57.030439 17556 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.055583 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:13:57.055173 17848 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.038861 18120 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:57.055083 17556 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.057914 17556 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.058835 18250 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:57.059293 18250 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:57.059645 18250 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.040963 18162 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:57.061713 18236 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20260709 12:13:57.040699 18033 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:57.035107 18200 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:57.063570 18200 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:57.063705 18211 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:13:57.064039 18200 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:57.064853 12641 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:57.039080 18092 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:57.065759 18092 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:57.066221 18092 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:57.057793 18238 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 12:13:57.035969 17914 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:57.038491 18225 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.068426 18225 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:57.073734 18221 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
W20260709 12:13:57.078840 18234 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
W20260709 12:13:57.080543 18232 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260709 12:13:57.081974 18228 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260709 12:13:57.085389 18225 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:57.022961 12389 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 11 to 14, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 14 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:13:57.089862 18248 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:13:57.091125 18216 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.278s	user 0.001s	sys 0.000s
W20260709 12:13:57.092413 18219 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.331s	user 0.002s	sys 0.000s
I20260709 12:13:57.054282 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:57.097349 18238 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:57.097802 18238 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:57.098265 18238 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.041100 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:13:57.100589 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.101929 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.102639 12640 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:13:57.103364 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.104426 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.104967 12640 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:57.057786 18206 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:13:57.106451 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52246.7us
I20260709 12:13:57.106186 18206 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:57.107259 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:57.026558 18231 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.110162 18206 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.088738 18240 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:57.111985 18224 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:13:57.114804 18250 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.115466 18235 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:57.117218 18250 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.119232 12566 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:13:57.120041 18244 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:13:57.056231 18213 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.123406 18092 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:57.124132 17919 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.246s	user 0.013s	sys 0.001s
I20260709 12:13:57.124907 17897 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:57.125712 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72796.6us
I20260709 12:13:57.127950 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.128645 12640 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.130144 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.134159 18200 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.137090 18200 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.144030 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:57.131458 18230 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:57.131551 17897 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.131021 12644 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:57.150250 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113164us
I20260709 12:13:57.151705 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:57.152993 18037 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:57.154857 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:57.179044 17556 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:13:57.147771 18236 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:13:57.150099 18240 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:13:57.198282 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114020us
I20260709 12:13:57.154815 17919 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:57.149397 18230 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:57.210104 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:57.211002 12491 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:13:57.206925 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.196354 18200 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:57.215082 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:57.197023 18242 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:57.219882 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:57.228516 12493 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:57.199093 18211 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:57.205503 17556 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:57.208499 18092 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.250806 17867 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:57.215183 18200 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:13:57.258663 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45553.6us
W20260709 12:13:57.258359 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48366.9us
I20260709 12:13:57.260861 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:57.243381 18231 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:57.262949 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:57.252843 18245 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:57.265316 18178 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:13:57.267524 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70722.1us
I20260709 12:13:57.252111 18242 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:57.253474 18210 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:57.213490 18252 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:13:57.211858 18240 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.267741 18178 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:13:57.267752 18243 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:13:57.267961 18213 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:13:57.373525 18178 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:57.374658 18256 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:13:57.278410 18230 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.396811 18240 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:57.263979 18222 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:57.288039 18211 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.361691 18238 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.412200 18197 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:57.413416 18197 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.359288 18176 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:57.259202 18200 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.265667 18227 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.420876 18227 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.421970 12568 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 12:13:57.391373 18242 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:57.296021 18085 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.645s	user 0.012s	sys 0.001s
I20260709 12:13:57.426137 18085 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.427265 18194 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:57.427901 18194 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.428745 18085 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.279870 17556 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.399503 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.439368 18227 raft_consensus.cc:2764] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:13:57.446208 18238 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.449692 17824 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.449415 18014 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.452637 17859 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.453895 12536 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
W20260709 12:13:57.456681 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2661) took 2933 ms (client timeout 2039 ms). Trace:
W20260709 12:13:57.457302 12570 rpcz_store.cc:269] 0709 12:13:54.523362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:54.698268 (+174906us) service_pool.cc:224] Handling call
0709 12:13:57.456388 (+2758120us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54003e0154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:57.456567 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2167587200}
I20260709 12:13:57.463299 17919 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:13:57.463617 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200727us
I20260709 12:13:57.465513 12461 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
I20260709 12:13:57.466480 12640 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:57.473785 17859 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:57.474176 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.475245 18238 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.409109 18235 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:57.457427 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.483606 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:57.484705 18194 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.495738 12640 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 18 because replica has last-logged OpId of term: 17 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 12:13:57.466735 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:57.481498 18197 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.521891 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.522907 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:13:57.525799 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:13:57.526225 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 146379us
I20260709 12:13:57.527669 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.528957 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:57.530093 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.531657 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.531980 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.532898 12640 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:13:57.534008 12639 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:13:57.534301 12640 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.11-13.11]
W20260709 12:13:57.540218 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82896.7us
I20260709 12:13:57.541006 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.524569 18120 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:57.524250 18014 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.530300 18107 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:57.536715 18178 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:57.558336 18178 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:57.558945 18178 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:57.515092 18238 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:57.543227 18263 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260709 12:13:57.543758 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77111us
I20260709 12:13:57.537592 18235 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:13:57.581775 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:13:57.597689 18187 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.883s	user 0.001s	sys 0.023s
W20260709 12:13:57.598428 18188 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.840s	user 0.001s	sys 0.008s
I20260709 12:13:57.594089 18120 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:57.603612 18120 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:57.604045 18120 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:57.600045 18238 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.609350 12640 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: []
I20260709 12:13:57.610663 12640 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:13:57.613536 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:13:57.614940 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73996.1us
I20260709 12:13:57.615438 17859 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:57.615944 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.616140 18197 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:13:57.616818 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:57.617722 17859 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:57.618067 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:57.618311 17859 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:57.618196 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:57.618705 18179 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.874s	user 0.012s	sys 0.000s
I20260709 12:13:57.619453 18197 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:57.619876 18197 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:57.620095 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2701) took 2935 ms. Trace:
I20260709 12:13:57.620258 18197 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.620478 12571 rpcz_store.cc:276] 0709 12:13:54.684814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.005783 (+320969us) service_pool.cc:224] Handling call
0709 12:13:55.059456 (+ 53673us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:55.070561 (+ 11105us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:55.070611 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:55.117065 (+ 46454us) log.cc:844] Serialized 26 byte log entry
0709 12:13:55.467566 (+350501us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:13:55.467616 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:55.467709 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:57.617794 (+2150085us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:13:57.619712 (+  1918us) raft_consensus.cc:1685] finished
0709 12:13:57.619743 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:57.620004 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8200,"raft.queue_time_us":8400,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":59549,"spinlock_wait_cycles":45900160,"thread_start_us":348984,"threads_started":1,"wal-append.queue_time_us":554017}
I20260709 12:13:57.618831 17859 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:57.617396 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.628028 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.629698 12640 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
W20260709 12:13:57.629675 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0067855220002002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2276s for a response
I20260709 12:13:57.632323 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.633878 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.599620 18200 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.632944 18242 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:57.633584 18238 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:57.647132 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.634413 18014 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:57.658352 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.653040 17556 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:13:57.639372 18257 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:57.681563 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.639068 18225 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:13:57.635334 18194 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:13:57.689580 12640 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:57.693780 12536 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:13:57.697161 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.675505 18085 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:57.702334 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.696702 18242 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:13:57.714375 18185 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.965s	user 0.004s	sys 0.008s
I20260709 12:13:57.719682 18197 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.726604 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.701363 18257 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:13:57.732188 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.735107 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.705068 18014 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:57.741163 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.742560 18196 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:57.744112 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.745970 18247 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:57.745602 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.748224 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:57.749119 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.749166 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2523) took 1869 ms. Trace:
I20260709 12:13:57.749706 17556 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:57.750389 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.750643 17556 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:57.750972 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.751360 17556 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:13:57.751724 12646 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.750989 12640 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:57.754542 18242 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:13:57.756153 18236 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:13:57.758000 18238 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:57.758478 18238 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:57.759371 18238 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.760704 18238 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:57.762334 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2305) took 1075 ms. Trace:
I20260709 12:13:57.762784 12644 rpcz_store.cc:276] 0709 12:13:56.686909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.875061 (+188152us) service_pool.cc:224] Handling call
0709 12:13:57.762115 (+887054us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:57.762257 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345521408}
I20260709 12:13:57.764196 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.764843 12644 raft_consensus.cc:2376] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14. Re-sending same reply.
I20260709 12:13:57.766296 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2722) took 1075 ms. Trace:
I20260709 12:13:57.766734 12568 rpcz_store.cc:276] 0709 12:13:56.690842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.024608 (+333766us) service_pool.cc:224] Handling call
0709 12:13:57.035249 (+ 10641us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:57.422682 (+387433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:57.766031 (+343349us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:57.766236 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":888584064}
I20260709 12:13:57.768360 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.769331 12644 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:13:57.770725 18247 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:57.749782 12642 rpcz_store.cc:276] 0709 12:13:55.879286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.881262 (+  1976us) service_pool.cc:224] Handling call
0709 12:13:55.881702 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:56.788013 (+906311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:57.748840 (+960827us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:57.749088 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2082562816}
I20260709 12:13:57.729403 18085 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:57.777940 12646 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
W20260709 12:13:57.783541 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 729) took 1900 ms (client timeout 1671 ms). Trace:
I20260709 12:13:57.783308 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:13:57.785158 12490 rpcz_store.cc:269] 0709 12:13:55.883312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.885731 (+  2419us) service_pool.cc:224] Handling call
0709 12:13:57.783298 (+1897567us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:57.783455 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":246334848}
I20260709 12:13:57.786213 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.786834 12640 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:13:57.783447 18268 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:13:57.796550 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.773149 18014 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:57.783793 18230 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:57.800127 12642 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:57.802474 17886 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.803313 18203 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:13:57.806212 12640 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:57.825228 12536 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.224s (SENT)
I20260709 12:13:57.825421 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.806859 18237 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:13:57.808198 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2545) took 1786 ms. Trace:
I20260709 12:13:57.814931 18197 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:57.819585 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:57.819914 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.820416 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:57.826632 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.828302 12641 rpcz_store.cc:276] 0709 12:13:56.021718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.204104 (+182386us) service_pool.cc:224] Handling call
0709 12:13:56.234919 (+ 30815us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:57.065326 (+830407us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:57.807835 (+742509us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:57.808118 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1908010880}
I20260709 12:13:57.831840 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.833899 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.834211 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:57.834842 12640 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:57.834954 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:57.836836 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 773) took 2885 ms. Trace:
I20260709 12:13:57.776898 18247 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.839432 18014 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.841185 18247 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:57.843185 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:57.844677 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:57.846704 12566 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:13:57.847030 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:57.837424 13254 rpcz_store.cc:276] 0709 12:13:54.944860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:54.951158 (+  6298us) service_pool.cc:224] Handling call
0709 12:13:54.951508 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:55.553320 (+601812us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:57.829716 (+2276396us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:57.829924 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1270558080}
I20260709 12:13:57.854249 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:57.849206 12461 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.276s (SENT)
I20260709 12:13:57.857890 12568 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:57.859078 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:57.824667 18203 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:57.861909 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.870520 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:57.871519 12571 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:57.860809 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:57.868937 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:57.875101 12572 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:57.828120 18237 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:57.870819 12640 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:57.866963 12566 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:57.877177 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:57.864060 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:57.879288 12641 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:13:57.880733 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.834101 18222 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:57.882303 12641 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:57.834272 18200 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.884663 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:57.884985 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53214.4us
I20260709 12:13:57.834702 17556 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:57.842538 18250 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.890818 18273 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:13:57.893512 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:57.895248 12565 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:57.859275 18197 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.900871 18203 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.907711 18197 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.908627 18203 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:57.847908 18257 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:57.910044 18257 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.912819 18257 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.848711 18190 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:57.913594 18190 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.858883 18193 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.858446 18242 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:57.824352 17886 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.832520 18120 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.901594 12639 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:13:57.902325 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71941.6us
I20260709 12:13:57.897521 18196 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:13:57.902765 12568 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:13:57.906119 12641 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
W20260709 12:13:57.905963 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105515us
I20260709 12:13:57.914021 17897 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:57.868278 18236 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:58.016680 18236 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:58.017352 18236 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:57.827257 18230 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:57.915014 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:58.022302 12640 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:58.023869 18230 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:58.024550 18230 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:13:57.916251 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:57.927095 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:58.026537 18230 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:13:58.030436 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.033114 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:58.035118 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:58.035838 12641 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:57.894797 17859 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.054203 12563 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:58.054526 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:57.899472 17556 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.083139 12639 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:13:58.091686 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:58.043797 18225 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:57.903569 18250 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.920958 18200 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.132s
I20260709 12:13:58.010522 18273 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.060235 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:58.043680 17897 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:13:58.100919 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:58.304402 12639 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:58.103137 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:13:58.066203 18268 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:13:58.111507 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:58.306291 12640 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:58.112516 12496 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:58.113221 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:57.997743 18242 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:57.924763 18193 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:58.212286 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 351516us
I20260709 12:13:58.069805 18177 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.068967 18227 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:58.306782 18107 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.752s	user 0.018s	sys 0.005s
I20260709 12:13:58.398182 12640 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:58.399843 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:58.400695 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.013s	sys 0.064s
W20260709 12:13:58.401294 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.013s	sys 0.064s
W20260709 12:13:58.401649 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 512287us
I20260709 12:13:58.401846 18107 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:58.293807 12494 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:13:58.065744 18257 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.408497 18257 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.053121 18204 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.300372 12641 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:13:58.411002 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:58.412660 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:58.311385 12488 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:13:58.315778 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:58.414531 12639 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:13:58.415192 18242 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:58.415541 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:58.417114 12639 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:13:58.326361 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.327374 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.418219 12491 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:13:58.326450 17886 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:58.008594 18237 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:58.339802 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 731) took 1131 ms. Trace:
I20260709 12:13:58.112068 18225 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:58.341463 12642 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.001s
I20260709 12:13:58.343247 17639 raft_consensus.cc:993] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:58.346690 12567 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:58.346946 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:58.297470 18271 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.319514 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:58.406606 12497 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:13:58.400766 12640 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:58.509826 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:13:58.509037 12495 rpcz_store.cc:276] 0709 12:13:57.208804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.208978 (+   174us) service_pool.cc:224] Handling call
0709 12:13:57.783482 (+574504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:57.783806 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:57.783834 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:57.783886 (+    52us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:57.783914 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:57.783970 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:58.339496 (+555526us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b540034d850. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:58.339699 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1071,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":2313,"spinlock_wait_cycles":1301189504}
W20260709 12:13:58.509397 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.016s	sys 0.008s
I20260709 12:13:58.548583 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:32896 (request call id 2681) took 2267 ms. Trace:
W20260709 12:13:58.554761 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.019s	sys 0.008s
I20260709 12:13:58.549325 12567 rpcz_store.cc:276] 0709 12:13:56.247698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.247938 (+   240us) service_pool.cc:224] Handling call
0709 12:13:58.344846 (+2096908us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eaa3d 00007ffbfa7bacd0 00007ffbf3d3f254 00007ffbf3d3f1e2 00007ffbf3d3f111 00007ffbf3d3f08c 00007ffbf3d3e342 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:58.514991 (+170145us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":235,"spinlock_wait_cycles":1253169024}
I20260709 12:13:58.558897 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.559681 12491 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 11.
I20260709 12:13:58.407071 18197 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:58.566619 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 247173us
I20260709 12:13:58.568032 12489 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:13:58.571563 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:58.572963 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.591329 12494 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
W20260709 12:13:58.566385 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.049s	sys 0.102s
W20260709 12:13:58.594415 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.049s	sys 0.102s
W20260709 12:13:58.596555 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 532875us
I20260709 12:13:58.600323 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:58.570967 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:58.633479 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.636533 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:58.639799 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:58.640403 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.641227 12491 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:13:58.650454 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:58.376367 18177 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:58.395617 18227 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:58.654747 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.532006 17886 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:58.558163 17850 raft_consensus.cc:1081] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:13:58.301594 18273 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:58.666757 12494 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
W20260709 12:13:58.667344 12492 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.017s	sys 0.021s
W20260709 12:13:58.680464 12492 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.017s	sys 0.022s
I20260709 12:13:58.674078 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.567559 18237 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:58.639673 18196 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:13:58.694905 12489 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:13:58.684700 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.706303 12491 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:13:58.409950 18204 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.738459 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:58.739436 12494 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:58.680315 18200 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:58.681901 17886 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.559247 18271 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.685611 18275 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:58.712654 18279 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.717377 17859 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:58.794083 12571 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.908s	user 0.008s	sys 0.004s
I20260709 12:13:58.680397 18277 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:13:58.798656 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.723977 18196 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:58.800567 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 462637us
I20260709 12:13:58.802081 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.805943 12571 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:13:58.772588 18120 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:58.772184 18200 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:58.791522 18242 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.790396 17556 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:58.829744 13256 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:58.837446 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:58.838485 12495 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:58.805382 18257 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:58.805910 17788 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:58.848304 12493 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:13:58.813868 18250 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:13:58.864192 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 263989us
I20260709 12:13:58.865244 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:58.866417 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:58.867331 17556 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:13:58.857131 18257 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:13:58.876116 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:58.805944 18279 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:58.951233 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.014s	sys 0.033s
W20260709 12:13:58.942183 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.047s	sys 0.065s
W20260709 12:13:58.964748 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.047s	sys 0.066s
W20260709 12:13:58.964500 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.015s	sys 0.034s
I20260709 12:13:58.942793 18120 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:13:58.865581 18193 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:58.983599 18238 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.222s	user 0.015s	sys 0.000s
I20260709 12:13:58.985508 18271 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:13:58.943019 18196 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:58.988687 18274 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:13:58.840381 18242 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:58.991966 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2729) took 1147 ms. Trace:
I20260709 12:13:58.992218 18242 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:58.992551 12563 rpcz_store.cc:276] 0709 12:13:57.844291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.844424 (+   133us) service_pool.cc:224] Handling call
0709 12:13:58.991658 (+1147234us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:58.991854 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":468278144}
I20260709 12:13:58.995384 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2730) took 1150 ms. Trace:
I20260709 12:13:58.996467 18193 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:58.997275 18193 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.001230 18193 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.003248 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:59.006172 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2539) took 3017 ms (client timeout 2276 ms). Trace:
W20260709 12:13:59.006819 12637 rpcz_store.cc:269] 0709 12:13:55.988295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.098996 (+110701us) service_pool.cc:224] Handling call
0709 12:13:59.005832 (+2906836us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.006011 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176137472}
I20260709 12:13:59.013969 12490 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:13:58.959640 18200 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.016788 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2860) took 1101 ms. Trace:
I20260709 12:13:58.995874 12570 rpcz_store.cc:276] 0709 12:13:57.844605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.844765 (+   160us) service_pool.cc:224] Handling call
0709 12:13:57.878806 (+ 34041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:58.296771 (+417965us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:58.296809 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:58.296851 (+    42us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:13:58.296881 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:58.995103 (+698222us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:58.995156 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:58.995321 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":541606656}
I20260709 12:13:59.017324 12490 rpcz_store.cc:276] 0709 12:13:57.915688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.915883 (+   195us) service_pool.cc:224] Handling call
0709 12:13:59.016451 (+1100568us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.016692 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270314880}
I20260709 12:13:58.947446 12495 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:13:58.874590 18286 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:58.866700 18190 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:58.858306 17788 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:58.954998 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:58.890432 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.035012 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:13:59.035202 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 144896us
W20260709 12:13:58.965164 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 533358us
I20260709 12:13:59.036361 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:58.944677 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:58.940383 17556 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:58.970026 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:13:58.955837 18236 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.938s	user 0.017s	sys 0.004s
I20260709 12:13:58.972983 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:13:58.978482 13256 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:58.965593 18275 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:13:58.980127 18283 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:13:58.981781 18273 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:58.982460 18227 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:58.982640 17886 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:58.989385 18274 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:59.034246 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2862) took 1116 ms. Trace:
I20260709 12:13:59.036391 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.073203 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:13:59.072240 12492 rpcz_store.cc:276] 0709 12:13:57.918211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.918388 (+   177us) service_pool.cc:224] Handling call
0709 12:13:57.918834 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:57.919068 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:13:58.681191 (+762123us) log.cc:844] Serialized 14 byte log entry
0709 12:13:59.033449 (+352258us) log.cc:844] Serialized 14 byte log entry
0709 12:13:59.033747 (+   298us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:59.033952 (+   205us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:13:59.033997 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.034063 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:59.034187 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1126,"prepare.queue_time_us":936198,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":2796,"raft.queue_time_us":168,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":9731,"spinlock_wait_cycles":5241088,"thread_start_us":1024874,"threads_started":2}
I20260709 12:13:59.074999 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.047910 12493 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:13:59.075824 18227 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:59.045868 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.068773 12490 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [15.5-15.5]
W20260709 12:13:59.069056 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 207207us
I20260709 12:13:59.075163 18120 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:13:59.045837 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:59.086259 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.087762 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.091326 18292 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:13:59.044018 18177 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.045083 18242 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:13:59.047403 18225 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:59.103061 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54247.6us
I20260709 12:13:59.083278 17850 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.044435 18073 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:13:59.090180 18238 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:13:59.107195 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.089936 18120 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:59.096745 18257 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:13:59.122349 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.067595 18210 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 12:13:59.098862 18190 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.130280 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:59.051975 18236 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.108095 18292 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:59.104051 18242 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:59.120572 17995 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:59.127043 18271 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:13:59.069314 18290 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.078s
I20260709 12:13:59.133163 18291 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:59.109802 18204 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.136920 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.154625 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.155841 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:13:59.127513 18120 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:59.158097 18289 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:13:59.158929 18289 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:13:59.108261 17886 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:13:59.140630 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:59.130115 18238 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:59.129099 12495 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:13:59.134634 18257 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:59.144757 18073 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:13:59.178216 18073 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:59.147285 18291 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:13:59.123219 17850 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:59.152395 18222 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.258s	user 0.007s	sys 0.007s
I20260709 12:13:59.041679 18286 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.179358 18073 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.192401 18286 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.146626 18288 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:13:59.144981 18178 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.585s	user 0.016s	sys 0.002s
I20260709 12:13:59.173714 12491 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:13:59.174047 18305 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.175493 18257 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.143378 18242 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:59.160198 18290 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:13:59.184389 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.199311 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.200065 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:59.200292 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:59.201098 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:59.180056 18289 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:13:59.201326 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:59.141264 18190 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:59.189145 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95584.3us
I20260709 12:13:59.202293 12646 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:13:59.202857 18238 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.127562 17556 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.213521 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2866) took 1162 ms. Trace:
I20260709 12:13:59.214147 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.195688 18222 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:59.213986 12488 rpcz_store.cc:276] 0709 12:13:58.051688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.051995 (+   307us) service_pool.cc:224] Handling call
0709 12:13:58.069431 (+ 17436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:58.413841 (+344410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.213252 (+799411us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b540012c5d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:59.213450 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":555558656}
I20260709 12:13:59.218582 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:59.228704 12563 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:59.230803 12646 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:59.222352 18014 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:13:59.234372 18014 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:13:59.234820 18236 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.234933 18014 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:13:59.235708 18236 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:13:59.240628 12563 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:13:59.142526 18275 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.141319 18274 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.245333 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:59.219344 18294 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:13:59.210798 18210 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:13:59.236919 18203 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.328s	user 0.014s	sys 0.002s
W20260709 12:13:59.248246 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45136.3us
I20260709 12:13:59.249261 12570 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:59.271840 12567 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:13:59.236169 18236 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:59.252571 12495 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:13:59.259182 13257 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:59.275374 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:59.209412 18027 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:13:59.244930 18279 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:13:59.270174 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2281) took 4023 ms (client timeout 2224 ms). Trace:
W20260709 12:13:59.284026 12643 rpcz_store.cc:269] 0709 12:13:55.247011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.247166 (+   155us) service_pool.cc:224] Handling call
0709 12:13:59.265809 (+4018643us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b540018c4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.270108 (+  4299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2293489024}
I20260709 12:13:59.265929 18273 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:13:59.270807 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2861) took 1353 ms (client timeout 1681 ms). Trace:
I20260709 12:13:59.291724 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2554) took 1666 ms. Trace:
W20260709 12:13:59.291864 12496 rpcz_store.cc:269] 0709 12:13:57.917896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.918151 (+   255us) service_pool.cc:224] Handling call
0709 12:13:59.265609 (+1347458us) spinlock_profiling.cc:243] Waited 71.2 ms on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.270757 (+  5148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163757952}
I20260709 12:13:59.271214 12489 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:13:59.251098 12563 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:13:59.230091 18196 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:13:59.259857 18191 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.279628 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:13:59.199702 18178 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.210662 18289 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:13:59.286223 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:13:59.305255 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.203307 18305 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.278410 18233 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:13:59.292192 12641 rpcz_store.cc:276] 0709 12:13:57.625615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.416044 (+790429us) service_pool.cc:224] Handling call
0709 12:13:58.416477 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:58.835938 (+419461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:58.835972 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:58.983618 (+147646us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:58.984130 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:13:59.123918 (+139788us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:59.130109 (+  6191us) log.cc:844] Serialized 14 byte log entry
0709 12:13:59.203324 (+ 73215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.203406 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:59.276266 (+ 72860us) raft_consensus.cc:1685] finished
0709 12:13:59.291466 (+ 15200us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:59.291638 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38813,"prepare.queue_time_us":8411,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":384721,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":276,"spinlock_wait_cycles":3280768,"thread_start_us":592354,"threads_started":3,"wal-append.queue_time_us":205795}
I20260709 12:13:59.250872 18210 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:59.298380 12571 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:13:59.290650 18107 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.273810 18236 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:13:59.285701 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.671s (SENT)
I20260709 12:13:59.303120 12563 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:13:59.273698 17788 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.320034 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.274066 18214 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.295212 18301 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.370072 18301 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.326515 18235 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:13:59.371521 18120 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.295032 18274 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:59.322604 18277 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.328121 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.328230 18306 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:59.328872 18203 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:13:59.331533 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:13:59.332276 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46663.4us
I20260709 12:13:59.332406 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.334537 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:59.498950 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.335198 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:59.500306 12496 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:13:59.298100 18196 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:59.339164 13255 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:59.510469 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:59.339109 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:59.341889 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 835) took 2647 ms (client timeout 2348 ms). Trace:
I20260709 12:13:59.330657 18210 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:13:59.333981 17931 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:59.343163 13260 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:13:59.560729 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 228382us
I20260709 12:13:59.343309 18242 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.345219 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:59.345578 18238 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.346869 18310 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.350199 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:59.352172 12490 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:13:59.209719 18291 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.242619 18292 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.364939 17788 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.366923 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2558) took 1740 ms. Trace:
I20260709 12:13:59.581748 12640 rpcz_store.cc:276] 0709 12:13:57.626731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.530619 (+903888us) service_pool.cc:224] Handling call
0709 12:13:58.531151 (+   532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:58.565307 (+ 34156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:58.565359 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:58.632922 (+ 67563us) log.cc:844] Serialized 26 byte log entry
0709 12:13:58.895483 (+262561us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:59.097518 (+202035us) log.cc:844] Serialized 14 byte log entry
0709 12:13:59.121474 (+ 23956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.121561 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:59.366608 (+245047us) raft_consensus.cc:1685] finished
0709 12:13:59.366634 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:59.366823 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64142,"prepare.queue_time_us":16708,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":265559,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":9447,"thread_start_us":343585,"threads_started":2,"wal-append.queue_time_us":396076}
I20260709 12:13:59.372270 18235 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:59.327075 18307 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.374219 18178 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.375488 17850 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.304083 18191 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.324468 18233 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:59.376830 18214 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:59.380815 18237 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.680s	user 0.005s	sys 0.005s
I20260709 12:13:59.383479 18190 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.383715 18227 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:13:59.339362 18107 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.485584 18297 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.491698 18305 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.338188 18273 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.118s
W20260709 12:13:59.484462 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156438us
I20260709 12:13:59.482929 18277 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.494624 18289 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:59.502072 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:13:59.512848 13253 rpcz_store.cc:269] 0709 12:13:56.694349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.694496 (+   147us) service_pool.cc:224] Handling call
0709 12:13:59.341598 (+2647102us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.341778 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432158976}
W20260709 12:13:59.567312 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51965us
I20260709 12:13:59.574607 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.503975 18076 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.516383 18196 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:13:59.516808 18247 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.680016 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:59.595508 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2567) took 1861 ms. Trace:
I20260709 12:13:59.569228 18306 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:13:59.610564 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 300156us
I20260709 12:13:59.576081 18288 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:59.694052 18296 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:13:59.695544 17824 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:13:59.612936 18307 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.698396 18307 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.698997 18301 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:13:59.614415 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 395930us
I20260709 12:13:59.622968 12569 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:59.628461 12571 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:13:59.700054 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:13:59.629328 12563 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:59.590808 18235 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:13:59.587950 18310 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.711592 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:59.711961 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.712986 12563 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:13:59.653728 18073 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.627707 17850 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.662325 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.716717 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:13:59.696084 18237 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.665470 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.573691 17931 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.585832 18238 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.672793 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2530) took 3765 ms. Trace:
I20260709 12:13:59.620100 18178 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.681159 12640 rpcz_store.cc:276] 0709 12:13:57.734604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.594206 (+1859602us) service_pool.cc:224] Handling call
0709 12:13:59.594722 (+   516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:59.595011 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:59.595053 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:59.595124 (+    71us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:13:59.595174 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.595255 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:59.595421 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:59.681972 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:13:59.691606 12643 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:13:59.689368 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:59.602212 17788 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:13:59.671676 17995 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.520s	user 0.010s	sys 0.001s
I20260709 12:13:59.722373 18073 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.738572 17788 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.615420 18279 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:13:59.739832 18073 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:59.696010 18306 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.743137 18306 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.744056 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.744751 12572 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:13:59.717048 17556 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:13:59.746095 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.746191 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:59.746822 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:59.610810 18172 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:59.718297 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.772374 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:59.716581 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:13:59.726729 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:59.774300 12640 raft_consensus.cc:2376] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13. Re-sending same reply.
I20260709 12:13:59.775633 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:59.776602 12640 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:13:59.724639 12638 rpcz_store.cc:276] 0709 12:13:55.900620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.180624 (+280004us) service_pool.cc:224] Handling call
0709 12:13:56.180930 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:56.850285 (+669355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.648330 (+2798045us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:59.666408 (+ 18078us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455762688}
W20260709 12:13:59.728057 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46218.9us
I20260709 12:13:59.728559 18301 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:13:59.728065 17859 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:13:59.733603 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2308) took 1957 ms. Trace:
I20260709 12:13:59.735423 18257 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:13:59.781769 12643 rpcz_store.cc:276] 0709 12:13:57.776603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.669682 (+1893079us) service_pool.cc:224] Handling call
0709 12:13:59.685187 (+ 15505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:59.733215 (+ 48028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.733511 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:59.668897 18297 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.747570 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.747745 18280 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:13:59.748054 12568 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:13:59.786682 12568 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:13:59.787195 12568 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:59.748788 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.788166 12569 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:59.788235 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.750130 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.789286 12570 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:13:59.750838 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.750488 18178 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.754179 18238 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.811210 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.811219 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:59.816028 12572 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:13:59.707834 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 145406us
I20260709 12:13:59.755079 18120 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.756157 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.793061 17556 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:59.844396 12644 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:13:59.844872 17556 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.846721 12570 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:13:59.760211 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2566) took 2053 ms. Trace:
I20260709 12:13:59.848356 12570 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:13:59.847353 12637 rpcz_store.cc:276] 0709 12:13:57.706648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.008061 (+1301413us) service_pool.cc:224] Handling call
0709 12:13:59.008547 (+   486us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:13:59.009091 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:59.009133 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:13:59.264444 (+255311us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:59.282806 (+ 18362us) log.cc:844] Serialized 52 byte log entry
0709 12:13:59.572300 (+289494us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:59.572362 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.572586 (+   224us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:13:59.759859 (+187273us) raft_consensus.cc:1685] finished
0709 12:13:59.759883 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:13:59.760139 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35615,"raft.queue_time_us":6883,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":17006,"spinlock_wait_cycles":45081856,"thread_start_us":281876,"threads_started":1,"wal-append.queue_time_us":467806}
I20260709 12:13:59.849113 18282 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:13:59.762982 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:59.849597 18282 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:13:59.850339 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87486.7us
I20260709 12:13:59.781744 18301 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:13:59.849449 12637 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 11. Current term is 12. Ops: []
I20260709 12:13:59.765555 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:13:59.852277 17931 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.852612 12637 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:13:59.852926 17931 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:13:59.767118 12567 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:59.769138 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2302) took 1200 ms. Trace:
I20260709 12:13:59.853791 12491 rpcz_store.cc:276] 0709 12:13:58.568636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.716048 (+147412us) service_pool.cc:224] Handling call
0709 12:13:58.799755 (+ 83707us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.195940 (+396185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.768774 (+572834us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:59.769069 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":859050880}
I20260709 12:13:59.854621 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2301) took 1303 ms. Trace:
I20260709 12:13:59.716197 18014 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.855094 12489 rpcz_store.cc:276] 0709 12:13:58.551151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.715778 (+164627us) service_pool.cc:224] Handling call
0709 12:13:59.271000 (+555222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.293004 (+ 22004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.854401 (+561397us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b540012cad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.854552 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1111369088}
I20260709 12:13:59.857170 12646 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:13:59.784644 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.858554 18107 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.722208 17850 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.859560 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:59.861183 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:13:59.725160 18237 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.701924 18247 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.866755 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:13:59.796559 13259 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:13:59.679574 18076 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.870144 18315 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.870594 17564 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.871088 17564 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.870880 18315 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.812366 18257 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.873936 18257 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.768831 18114 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:59.874810 18227 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.875452 18114 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.875629 18227 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.877899 12563 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:13:59.781068 18193 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:13:59.878902 18193 consensus_queue.cc:1243] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:13:59.879480 18193 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:13:59.880158 18236 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.520s	user 0.006s	sys 0.003s
I20260709 12:13:59.880700 18236 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.765870 18233 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.840565 18235 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.831082 18310 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.842947 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 839) took 3089 ms. Trace:
I20260709 12:13:59.843142 18277 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.758167 18200 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:59.841487 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2865) took 1810 ms. Trace:
I20260709 12:13:59.861830 18274 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:13:59.862383 18107 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:13:59.783823 12565 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:13:59.867789 12637 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:13:59.868009 12569 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:13:59.874660 17859 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:13:59.877935 18315 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.769968 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:13:59.909839 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 139956us
I20260709 12:13:59.913025 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.885856 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.850759 18178 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:13:59.895340 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:13:59.890359 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.931649 12565 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:59.932082 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.885370 13257 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:13:59.884522 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.932719 18236 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.907773 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.896111 17859 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:13:59.933483 17859 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:13:59.896740 18301 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:59.932606 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.914693 17850 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.886416 18247 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.935590 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.933766 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.936908 13255 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:13:59.937249 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.857460 18014 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.895514 18107 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:13:59.882156 18235 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:13:59.884130 18210 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.850744 18282 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:13:59.881574 18193 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:59.882424 18233 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:13:59.920287 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.886636 13260 rpcz_store.cc:276] 0709 12:13:56.754011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.754170 (+   159us) service_pool.cc:224] Handling call
0709 12:13:59.342691 (+2588521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:59.560587 (+217896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.842688 (+282101us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b5400579750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.842876 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5884809088}
I20260709 12:13:59.870137 18076 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.922102 17867 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:13:59.886865 18242 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.887485 18191 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.887532 18200 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:13:59.923297 18310 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:00.027593 18310 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.028101 18310 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:00.028952 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.030474 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.031384 12567 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:14:00.031759 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:00.032223 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:00.033488 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.034449 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.034823 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.035301 13260 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:00.035405 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2527) took 4139 ms. Trace:
I20260709 12:14:00.035772 12564 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:14:00.036486 18276 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:13:59.925395 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.926887 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.037406 12564 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:14:00.038347 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:00.033376 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:13:59.926721 18244 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:13:59.835160 18297 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.916455 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.887653 12497 rpcz_store.cc:276] 0709 12:13:58.031451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.050411 (+ 18960us) service_pool.cc:224] Handling call
0709 12:13:58.050974 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:58.529918 (+478944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.841193 (+1311275us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:13:59.841415 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":816765824}
I20260709 12:14:00.042341 18244 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:00.042954 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:00.043143 18244 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:13:59.921299 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:13:59.932188 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.044515 18244 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.045270 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.046470 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.035825 12645 rpcz_store.cc:276] 0709 12:13:55.895726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:55.905949 (+ 10223us) service_pool.cc:224] Handling call
0709 12:13:55.982254 (+ 76305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:56.000932 (+ 18678us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:56.192347 (+191415us) log.cc:844] Serialized 14 byte log entry
0709 12:13:58.868201 (+2675854us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b18003bde80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:13:58.881683 (+ 13482us) log.cc:844] Serialized 14 byte log entry
0709 12:13:59.042594 (+160911us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:13:59.042647 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:13:59.042695 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:13:59.290233 (+247538us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b18003bde80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:00.035164 (+744931us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:00.035214 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:00.035346 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25872,"prepare.queue_time_us":362184,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":50589,"raft.queue_time_us":88045,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":4292,"spinlock_wait_cycles":43886720,"thread_start_us":424411,"threads_started":3,"wal-append.queue_time_us":488990,"wal-append.run_cpu_time_us":3063,"wal-append.run_wall_time_us":1606275}
I20260709 12:14:00.046558 18291 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:13:59.938385 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 832) took 3302 ms (client timeout 1980 ms). Trace:
W20260709 12:14:00.047272 13259 rpcz_store.cc:269] 0709 12:13:56.636432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:56.640444 (+  4012us) service_pool.cc:224] Handling call
0709 12:13:59.938180 (+3297736us) spinlock_profiling.cc:243] Waited 3.09 s on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:13:59.938307 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7100179200}
I20260709 12:13:59.939237 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.048484 12569 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:13:59.939458 12638 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:13:59.940462 12572 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:13:59.895695 18237 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:13:59.933533 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.940747 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.944191 13257 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:13:59.888455 12643 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:14:00.056507 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2326) took 2206 ms. Trace:
I20260709 12:14:00.056880 12643 rpcz_store.cc:276] 0709 12:13:57.850091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.867081 (+2016990us) service_pool.cc:224] Handling call
0709 12:13:59.888131 (+ 21050us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:00.056220 (+168089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.056447 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:13:59.952266 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:13:59.965440 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52505.1us
I20260709 12:14:00.063185 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:13:59.966634 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:13:59.916486 18227 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:13:59.940852 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.071106 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.071523 12646 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:14:00.072523 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.073076 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.074148 12646 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:13:59.969915 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:00.076050 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 106242us
I20260709 12:14:00.077100 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.078056 12495 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:13:59.964623 12568 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:13:59.945407 18114 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:00.082832 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161630us
I20260709 12:14:00.075150 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:13:59.989848 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:13:59.955644 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 285111us
I20260709 12:13:59.998601 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:13:59.998227 18193 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:00.084417 12491 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:14:00.026598 18172 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:00.007633 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:13:59.952948 18107 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:13:59.999414 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.001183 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:00.013098 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:13:59.950935 18315 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:13:59.984395 18274 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:00.001953 18238 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:13:59.935218 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.042349 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:00.050550 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2325) took 2200 ms. Trace:
W20260709 12:14:00.051111 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.005141 18236 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:00.065024 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98519.5us
I20260709 12:14:00.068327 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:00.069864 12572 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
W20260709 12:14:00.070353 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129711us
I20260709 12:14:00.081136 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.081682 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.084806 12646 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:14:00.092353 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.084496 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:00.093870 12568 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:00.095897 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:00.007004 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.096026 12566 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:14:00.023681 17867 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:00.025419 18200 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:00.111514 12490 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:00.169788 12645 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
W20260709 12:14:00.170241 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85834.8us
I20260709 12:14:00.170657 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.111143 12494 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 11.
I20260709 12:14:00.171690 12638 rpcz_store.cc:276] 0709 12:13:57.849732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.867224 (+2017492us) service_pool.cc:224] Handling call
0709 12:13:59.939178 (+ 71954us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:00.050249 (+111071us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.050490 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:00.111690 12493 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:00.173215 12493 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:00.075878 18320 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:00.176239 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.171540 12490 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:00.179132 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:00.181471 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 113249us
I20260709 12:14:00.182626 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:00.184818 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.186604 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.080420 18184 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:00.192229 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:00.188292 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.194098 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:00.196436 13257 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:14:00.198475 18230 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:00.199204 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.258724 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.096880 17639 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:00.186517 17931 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:00.202525 12645 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:14:00.204067 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.079978 18276 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:14:00.188594 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92757.5us
I20260709 12:14:00.235266 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.111034 18244 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:00.109930 18214 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:00.260087 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46734.4us
I20260709 12:14:00.267328 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.291113 13259 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:14:00.103595 18210 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:00.298285 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.097527 18247 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:00.302452 12494 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:00.204584 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.325960 12572 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:14:00.326503 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.326853 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.270202 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:00.306159 12488 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:14:00.308048 12641 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.006s	sys 0.026s
I20260709 12:14:00.257572 12637 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:14:00.327548 12572 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:14:00.310524 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:00.328231 12641 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.006s	sys 0.027s
I20260709 12:14:00.329010 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:00.329856 12637 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:00.330842 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.332265 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.176478 17556 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:00.319937 13255 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:00.333292 17556 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:00.336416 17556 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:00.110195 18277 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:00.327442 13259 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:14:00.329701 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.337570 12572 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:00.337281 18277 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.338572 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.339507 18277 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:00.339744 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:00.318938 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.259632 18227 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:14:00.346592 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.113875 18127 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.199s
I20260709 12:14:00.110507 18222 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.215s
I20260709 12:14:00.290393 17931 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.301273 18280 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:00.348510 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:00.109962 18278 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.190s
I20260709 12:14:00.106436 18310 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:00.343824 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:00.352304 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:00.204537 18274 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:00.353837 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83732.8us
I20260709 12:14:00.323920 17639 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:00.341574 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.342185 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.264770 18114 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 12:14:00.109572 18301 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:00.301180 18276 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.343549 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:00.376964 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55504us
I20260709 12:14:00.189178 18236 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.451272 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:00.450202 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.512826 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:00.325569 18247 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.267748 17950 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.100s
I20260709 12:14:00.353922 18239 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:00.168174 18076 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.151s
I20260709 12:14:00.264122 18120 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 12:14:00.265728 18257 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.090s
I20260709 12:14:00.556501 12495 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:00.480572 12491 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:14:00.301643 18289 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:00.467433 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:00.499164 13258 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:00.574698 13258 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:00.574839 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107563us
I20260709 12:14:00.257645 18327 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.190s
W20260709 12:14:00.375754 18196 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.673s	user 0.010s	sys 0.001s
I20260709 12:14:00.263341 18085 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260709 12:14:00.319308 18204 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.131s
I20260709 12:14:00.372329 17931 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:14:00.599509 18335 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:14:00.575497 13258 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:00.606462 13258 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:00.586194 18076 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:00.607374 18076 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.607870 18076 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.586620 18289 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.586030 18257 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:00.614298 18257 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:00.616995 18257 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:00.620357 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 705) took 2852 ms. Trace:
I20260709 12:14:00.621615 12637 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:00.300433 18297 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:14:00.486451 18211 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:00.350801 18213 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.150s
I20260709 12:14:00.545007 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:00.663924 12645 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.553402 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:00.542629 18239 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:00.474359 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130620us
I20260709 12:14:00.563947 17639 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.613443 18289 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:00.482203 18292 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:00.572917 13261 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:00.402868 18321 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:00.586051 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:00.531113 18276 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.547621 18247 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.539462 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:00.732369 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 193016us
I20260709 12:14:00.733791 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:00.620922 12641 rpcz_store.cc:276] 0709 12:13:57.768295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.666408 (+1898113us) service_pool.cc:224] Handling call
0709 12:13:59.666908 (+   500us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.689077 (+ 22169us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:13:59.689115 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:59.705489 (+ 16374us) log.cc:844] Serialized 26 byte log entry
0709 12:14:00.328820 (+623331us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:00.328879 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.329006 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:00.620001 (+290995us) raft_consensus.cc:1685] finished
0709 12:14:00.620034 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:00.620262 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4389,"raft.queue_time_us":2431,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":11814,"thread_start_us":587959,"threads_started":1,"wal-append.queue_time_us":846718}
I20260709 12:14:00.739651 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:00.740607 12641 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:00.677919 18239 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:00.457024 18230 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.109s
I20260709 12:14:00.636037 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:00.543710 18027 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:00.665427 12494 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:14:00.356050 18193 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.530859 18315 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.117s
I20260709 12:14:00.578301 18272 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.670742 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:00.571728 18324 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.119s
I20260709 12:14:00.689881 12491 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:00.696630 18274 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:00.634004 18257 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:00.752431 18240 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:14:00.724447 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.806000 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:00.780805 17859 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.841s	user 0.006s	sys 0.009s
I20260709 12:14:00.767198 18278 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:00.624289 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.670001 18213 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:14:00.750722 17886 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:00.621201 18196 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.689513 18292 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:00.770023 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134075us
W20260709 12:14:00.775928 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:00.783110 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 112447us
I20260709 12:14:00.624886 18114 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:00.825961 12491 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: []
I20260709 12:14:00.803284 17849 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:00.828224 17564 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:00.829452 18345 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 12:14:00.830499 18310 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:00.855014 18233 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.837s	user 0.011s	sys 0.001s
I20260709 12:14:00.804436 18320 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:00.775413 17476 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260709 12:14:00.815860 18240 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.854313 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:00.891567 12494 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:00.851730 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:00.893966 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:00.800310 18257 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:00.834550 17859 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.806885 17867 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:00.904186 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2560) took 3277 ms. Trace:
I20260709 12:14:00.862401 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:00.841352 17886 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:00.911002 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:00.911621 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.889889 12491 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:14:00.859390 17849 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:00.897894 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.773931 18289 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.064s
W20260709 12:14:00.890252 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.012s	sys 0.012s
I20260709 12:14:00.728545 13258 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:14:00.900173 17564 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:00.904727 12642 rpcz_store.cc:276] 0709 12:13:57.627182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.545738 (+918556us) service_pool.cc:224] Handling call
0709 12:13:58.546162 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:13:58.805997 (+259835us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:13:58.832380 (+ 26383us) log.cc:844] Serialized 14 byte log entry
0709 12:14:00.070452 (+1238072us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b1800323620. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:14:00.474143 (+403691us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:00.474392 (+   249us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:00.474465 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.903863 (+429398us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:00.903938 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:00.904118 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37039,"prepare.queue_time_us":23846,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":107075,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":25389,"spinlock_wait_cycles":606824064,"thread_start_us":300028,"threads_started":1,"wal-append.queue_time_us":1920175}
I20260709 12:14:00.908902 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2701) took 1166 ms. Trace:
I20260709 12:14:00.879153 18085 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 14.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:14:00.857721 18292 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.848045 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:14:00.883857 18320 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:00.914009 18196 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:00.914700 18236 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:00.841883 18278 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.916726 17859 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:00.918412 18235 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.957s	user 0.011s	sys 0.000s
I20260709 12:14:00.889639 18301 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:00.919843 18343 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:14:00.887037 18352 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:14:00.867959 18114 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:00.920433 18291 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.873s	user 0.012s	sys 0.000s
I20260709 12:14:00.903857 18127 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:00.923923 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2746) took 1001 ms. Trace:
I20260709 12:14:00.933112 17564 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:00.954512 12567 rpcz_store.cc:276] 0709 12:13:59.922809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.041244 (+118435us) service_pool.cc:224] Handling call
0709 12:14:00.041611 (+   367us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:00.056186 (+ 14575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:00.056218 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:00.056946 (+   728us) log.cc:844] Serialized 52 byte log entry
0709 12:14:00.293562 (+236616us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:00.526785 (+233223us) log.cc:844] Serialized 14 byte log entry
0709 12:14:00.527225 (+   440us) log.cc:844] Serialized 14 byte log entry
0709 12:14:00.553118 (+ 25893us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.553195 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:00.923622 (+370427us) raft_consensus.cc:1685] finished
0709 12:14:00.923652 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:00.923844 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6369,"prepare.queue_time_us":924707,"prepare.run_cpu_time_us":323,"prepare.run_wall_time_us":5339,"raft.queue_time_us":157417,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":135,"spinlock_wait_cycles":7201024,"thread_start_us":405688,"threads_started":2,"wal-append.queue_time_us":375943}
I20260709 12:14:00.925045 12645 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:00.958189 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:00.905593 18191 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:14:00.925693 12494 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:00.926327 12563 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 12: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:00.895282 18240 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:00.931209 12566 rpcz_store.cc:276] 0709 12:13:59.742666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.749222 (+  6556us) service_pool.cc:224] Handling call
0709 12:13:59.749527 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:00.176236 (+426709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.908548 (+732312us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:00.908812 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796192512}
W20260709 12:14:00.947089 18334 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:14:00.926185 17886 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:00.928972 17849 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:00.951808 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:01.013913 12642 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:00.964823 18114 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:00.953325 18333 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:14:00.922204 17931 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:00.940649 18347 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:00.968227 12640 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:00.977818 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:00.905359 17950 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:01.010421 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.060729 12638 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:01.061717 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 707) took 1196 ms. Trace:
I20260709 12:14:01.062109 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 709) took 1186 ms. Trace:
I20260709 12:14:01.024015 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:01.063019 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:00.942245 18320 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:01.055614 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:01.087839 18344 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.216s	user 0.005s	sys 0.001s
I20260709 12:14:01.064016 12567 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:01.062160 12638 rpcz_store.cc:276] 0709 12:13:59.865209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.013660 (+1148451us) service_pool.cc:224] Handling call
0709 12:14:01.060456 (+ 46796us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.061332 (+   876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.061619 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.090396 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2588) took 1122 ms. Trace:
I20260709 12:14:01.090721 12638 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:01.090859 12640 rpcz_store.cc:276] 0709 12:13:59.967940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.046220 (+ 78280us) service_pool.cc:224] Handling call
0709 12:14:01.090213 (+1043993us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:01.090328 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118190976}
I20260709 12:14:01.091519 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 710) took 1215 ms. Trace:
I20260709 12:14:01.062793 12641 rpcz_store.cc:276] 0709 12:13:59.875878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.060964 (+1185086us) service_pool.cc:224] Handling call
0709 12:14:01.061388 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.061660 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.061700 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:01.061758 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:01.061803 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.061877 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.062013 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.091841 18278 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:01.092451 12640 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:00.955485 18127 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.093168 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 713) took 1216 ms. Trace:
I20260709 12:14:01.070986 18347 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:01.072342 18351 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:14:00.973703 18213 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.111074 18213 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.112805 18347 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:00.973985 18235 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.078071 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 848) took 1829 ms. Trace:
I20260709 12:14:01.021486 18353 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.078058 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:01.079706 12566 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:01.118899 12566 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:14:01.119205 12566 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:00.972712 17564 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:01.093168 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 714) took 1214 ms. Trace:
I20260709 12:14:01.123198 12641 rpcz_store.cc:276] 0709 12:13:59.878853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.092384 (+1213531us) service_pool.cc:224] Handling call
0709 12:14:01.092670 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.092864 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.092890 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:01.092940 (+    50us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:01.092973 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.093025 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.093110 (+    85us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.116734 13256 rpcz_store.cc:276] 0709 12:13:59.249004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.249212 (+   208us) service_pool.cc:224] Handling call
0709 12:13:59.275938 (+ 26726us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.289830 (+ 13892us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:59.289866 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:00.047792 (+757926us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b180047ee40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:14:00.048381 (+   589us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b2c00007be8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:00.048964 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:14:00.317251 (+268287us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:00.317301 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.317381 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.077713 (+760332us) raft_consensus.cc:1685] finished
0709 12:14:01.077741 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.077948 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4391,"raft.queue_time_us":13659,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":6024,"spinlock_wait_cycles":29460992,"thread_start_us":254928,"threads_started":1,"wal-append.queue_time_us":923922}
W20260709 12:14:01.095813 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.097239 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 789) took 1333 ms. Trace:
I20260709 12:14:01.093566 12640 rpcz_store.cc:276] 0709 12:13:59.878493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.091961 (+1213468us) service_pool.cc:224] Handling call
0709 12:14:01.092271 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.092932 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.093110 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.091890 12638 rpcz_store.cc:276] 0709 12:13:59.876181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.090164 (+1213983us) service_pool.cc:224] Handling call
0709 12:14:01.090523 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.091203 (+   680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.091422 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.128453 12640 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:14:01.052377 18273 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.056808 18291 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.105129 12567 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:01.093566 18127 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:01.095042 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:01.138844 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:01.126783 13261 rpcz_store.cc:276] 0709 12:13:59.770736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.770987 (+   251us) service_pool.cc:224] Handling call
0709 12:13:59.823744 (+ 52757us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:00.686206 (+862462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.096898 (+410692us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:01.097156 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1687588224}
I20260709 12:14:01.123267 18280 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.144704 12638 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:14:01.113396 18213 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.154116 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2327) took 3303 ms. Trace:
I20260709 12:14:01.161435 12644 rpcz_store.cc:276] 0709 12:13:57.850386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.845242 (+1994856us) service_pool.cc:224] Handling call
0709 12:13:59.845727 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.845974 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:59.846001 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:59.846546 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 12:14:00.174159 (+327613us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:14:00.174203 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.174287 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.153767 (+979480us) raft_consensus.cc:1685] finished
0709 12:14:01.153803 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.154025 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":327211,"threads_started":1,"wal-append.queue_time_us":998425}
I20260709 12:14:01.161041 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.120821 18289 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:01.169077 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.144368 17950 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:01.176095 18274 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:14:01.176360 17950 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.178133 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2738) took 1305 ms. Trace:
I20260709 12:14:01.178814 17950 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.179751 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:01.178579 12570 rpcz_store.cc:276] 0709 12:13:59.872842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.925731 (+ 52889us) service_pool.cc:224] Handling call
0709 12:13:59.926165 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.927546 (+  1381us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:00.211138 (+283592us) log.cc:844] Serialized 14 byte log entry
0709 12:14:00.764703 (+553565us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:00.824488 (+ 59785us) log.cc:844] Serialized 26 byte log entry
0709 12:14:00.824698 (+   210us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:00.824750 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.824859 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.177901 (+353042us) raft_consensus.cc:1685] finished
0709 12:14:01.177927 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.178068 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":263,"prepare.queue_time_us":852637,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":23398,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":16966,"thread_start_us":702756,"threads_started":2,"wal-append.queue_time_us":598057}
I20260709 12:14:01.180830 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:01.181185 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:01.182430 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:01.183508 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:01.185974 12567 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:01.186661 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2716) took 1001 ms. Trace:
I20260709 12:14:01.186988 12567 rpcz_store.cc:276] 0709 12:14:00.184745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.185409 (+1000664us) service_pool.cc:224] Handling call
0709 12:14:01.185763 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.186416 (+   653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.186604 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.188292 12567 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:01.188864 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2718) took 1003 ms. Trace:
I20260709 12:14:01.189181 12567 rpcz_store.cc:276] 0709 12:14:00.185259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.187778 (+1002519us) service_pool.cc:224] Handling call
0709 12:14:01.188097 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.188626 (+   529us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.188812 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:01.142266 18273 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.126957 18353 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.193006 18353 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.195727 18120 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:01.196483 18120 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:01.201464 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 752) took 1201 ms (client timeout 1507 ms). Trace:
W20260709 12:14:01.202188 12488 rpcz_store.cc:269] 0709 12:13:59.999733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.999962 (+   229us) service_pool.cc:224] Handling call
0709 12:14:01.201151 (+1201189us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:01.201360 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":303609088}
I20260709 12:14:01.167685 17919 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:01.206830 17919 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.212775 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.213085 18027 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.213979 18027 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.214804 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.215859 18014 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.216378 18014 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.216643 18027 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.217217 18355 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:01.220018 18014 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.223697 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:01.225292 18191 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:01.225906 18191 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.226361 18191 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:01.230587 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:01.126823 18235 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.216356 17950 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:01.236130 18235 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.181362 12570 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:01.153409 18233 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:14:01.239112 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.245606 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.252746 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:01.250572 18354 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.167461 17867 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:01.250007 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77534.2us
I20260709 12:14:01.146117 18291 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.240164 18196 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:01.245157 18233 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.246379 17859 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:01.272053 17859 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.228331 18257 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.157558 18280 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.267825 18196 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.269795 17556 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:01.271569 18233 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:01.258394 17950 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:14:01.276569 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.278571 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.237713 18289 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.312630 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:01.287957 18114 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:01.276821 12493 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.338258 12493 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:01.266755 17919 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.256623 18239 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.273087 18291 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:01.316939 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:01.366770 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49962.3us
I20260709 12:14:01.369156 18355 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:01.369947 18355 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:01.251257 18120 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.372052 18120 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:01.311411 17950 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:01.373984 17950 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:14:01.375362 18280 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:01.375684 18239 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.376551 18239 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:14:01.379273 18289 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:01.382478 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.383388 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2893) took 1301 ms. Trace:
I20260709 12:14:01.383790 12497 rpcz_store.cc:276] 0709 12:14:00.082069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.964290 (+882221us) service_pool.cc:224] Handling call
0709 12:14:00.964748 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:00.965324 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:00.965358 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.014625 (+ 49267us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.275694 (+261069us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:01.275756 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.275853 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.383096 (+107243us) raft_consensus.cc:1685] finished
0709 12:14:01.383118 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.383311 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8426,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12883,"thread_start_us":185606,"threads_started":1,"wal-append.queue_time_us":317302}
I20260709 12:14:01.386685 17859 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.386957 17849 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:01.387971 18276 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.636s	user 0.014s	sys 0.000s
I20260709 12:14:01.269945 17867 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.293702 18272 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:01.301867 18196 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:01.349236 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:01.270332 18354 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.359823 17886 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.352178 17931 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.302435 17639 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:14:01.294919 18353 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:14:01.338891 18293 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:01.440282 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.441149 12492 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 12: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:01.390271 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:01.411465 18244 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.093s	user 0.014s	sys 0.000s
W20260709 12:14:01.403656 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.388458 18276 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.404498 17919 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:14:01.431496 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:01.342466 17556 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:01.433887 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2888) took 1361 ms. Trace:
I20260709 12:14:01.452365 12491 rpcz_store.cc:276] 0709 12:14:00.072836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.923254 (+850418us) service_pool.cc:224] Handling call
0709 12:14:00.940662 (+ 17408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:00.941165 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:00.941198 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:00.941967 (+   769us) log.cc:844] Serialized 52 byte log entry
0709 12:14:01.245053 (+303086us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:01.430193 (+185140us) log.cc:844] Serialized 14 byte log entry
0709 12:14:01.433486 (+  3293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.433570 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.433617 (+    47us) raft_consensus.cc:1685] finished
0709 12:14:01.433652 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.433816 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7364,"raft.queue_time_us":17000,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":8350,"thread_start_us":471792,"threads_started":2,"wal-append.queue_time_us":368779}
W20260709 12:14:01.388880 18076 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.780s	user 0.014s	sys 0.001s
I20260709 12:14:01.442636 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.455402 12497 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:01.315886 18257 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:01.445493 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:01.459969 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.453225 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:01.427644 12493 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:01.440253 18293 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:01.453315 12568 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:14:01.389746 13261 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:01.455909 12497 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:01.464605 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2313) took 1464 ms. Trace:
I20260709 12:14:01.394277 12570 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:01.471994 12641 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:01.471822 12493 rpcz_store.cc:276] 0709 12:14:00.000566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.000795 (+   229us) service_pool.cc:224] Handling call
0709 12:14:00.607979 (+607184us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b4c002c6f00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3a96644 00007ffbf3abdf2d 00007ffbf3abdb93 00007ffbf57ee18b 00007ffbf57eecb5 00007ffbf57e9f30 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17
0709 12:14:01.276168 (+668189us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b54000ee0a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577106f 00007ffbf57e9f60 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2
0709 12:14:01.464542 (+188374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266318464}
I20260709 12:14:01.473839 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.474601 12493 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:01.456750 17919 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:01.485114 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.487902 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.483745 12497 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.7, Last appended by leader: 7, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.491319 12497 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:01.461223 17867 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.441517 18320 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.481163 18293 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:01.500398 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:01.504594 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 163577us
I20260709 12:14:01.506712 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.508579 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:01.462841 18076 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.525944 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:01.531694 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:01.535110 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 174383us
I20260709 12:14:01.498276 18237 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:01.565511 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.567698 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.595114 12497 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:01.528601 17639 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:01.565655 18353 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:01.591007 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:01.572177 17849 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.510905 12493 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:14:01.569617 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2884) took 1497 ms. Trace:
I20260709 12:14:01.598704 12495 rpcz_store.cc:276] 0709 12:14:00.071880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.191912 (+120032us) service_pool.cc:224] Handling call
0709 12:14:01.569392 (+1377480us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:01.569541 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":835795712}
I20260709 12:14:01.600020 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.601686 12495 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:01.552475 17564 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.596359 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.602545 12497 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:01.497268 18240 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.604878 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:01.609807 18364 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:14:01.612923 12495 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:01.603397 12493 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 13
I20260709 12:14:01.613618 12491 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:01.616796 12497 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:01.617221 12493 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:14:01.618048 12493 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:01.581312 18374 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:14:01.620069 18320 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.601200 18353 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.622073 18353 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:01.623862 12491 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:14:01.624542 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2895) took 1531 ms. Trace:
I20260709 12:14:01.624925 12494 rpcz_store.cc:276] 0709 12:14:00.093525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.971966 (+878441us) service_pool.cc:224] Handling call
0709 12:14:01.054918 (+ 82952us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:01.108976 (+ 54058us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.109030 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.109783 (+   753us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.492988 (+383205us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:01.493032 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.493107 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.624316 (+131209us) raft_consensus.cc:1685] finished
0709 12:14:01.624338 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.624477 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23161,"raft.queue_time_us":14603,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":19140,"thread_start_us":337885,"threads_started":1,"wal-append.queue_time_us":469820}
I20260709 12:14:01.627794 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2757) took 1702 ms. Trace:
I20260709 12:14:01.628207 12566 rpcz_store.cc:276] 0709 12:13:59.928492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.078999 (+1150507us) service_pool.cc:224] Handling call
0709 12:14:01.079465 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:01.159691 (+ 80226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.159726 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:01.160384 (+   658us) log.cc:844] Serialized 52 byte log entry
0709 12:14:01.465062 (+304678us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:01.486956 (+ 21894us) log.cc:844] Serialized 14 byte log entry
0709 12:14:01.487290 (+   334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.487392 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.627482 (+140090us) raft_consensus.cc:1685] finished
0709 12:14:01.627509 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.627727 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15147,"prepare.queue_time_us":21678,"raft.queue_time_us":28608,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":11860,"spinlock_wait_cycles":12244352,"thread_start_us":213959,"threads_started":1,"wal-append.queue_time_us":388381}
I20260709 12:14:01.629536 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2896) took 1535 ms. Trace:
I20260709 12:14:01.630090 12488 rpcz_store.cc:276] 0709 12:14:00.093955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.203676 (+1109721us) service_pool.cc:224] Handling call
0709 12:14:01.204267 (+   591us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:01.204807 (+   540us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.204852 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.205548 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.456345 (+250797us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:01.456386 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.456450 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.629279 (+172829us) raft_consensus.cc:1685] finished
0709 12:14:01.629301 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.629464 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13371,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":239,"thread_start_us":196993,"threads_started":1,"wal-append.queue_time_us":281054}
I20260709 12:14:01.631449 12566 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:01.631191 17950 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.632623 12488 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:14:01.634274 18233 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:01.635068 18370 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:14:01.597069 17639 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:01.637953 17639 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:01.614675 18373 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:01.639297 18373 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.639751 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 909) took 1467 ms. Trace:
I20260709 12:14:01.640666 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2892) took 1563 ms. Trace:
I20260709 12:14:01.640832 17564 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.583168 17931 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:14:01.641086 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.642159 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:01.642532 18354 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.642644 17931 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:14:01.522094 17867 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
I20260709 12:14:01.646567 18233 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.649111 18276 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:01.662768 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71830.7us
I20260709 12:14:01.614807 17886 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.661964 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.669091 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:01.640237 12563 rpcz_store.cc:276] 0709 12:14:00.171994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.084630 (+912636us) service_pool.cc:224] Handling call
0709 12:14:01.085064 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:01.116979 (+ 31915us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:14:01.152841 (+ 35862us) log.cc:844] Serialized 14 byte log entry
0709 12:14:01.434540 (+281699us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.443863 (+  9323us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.444014 (+   151us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:01.444046 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.444115 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.639530 (+195415us) raft_consensus.cc:1685] finished
0709 12:14:01.639551 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.639692 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31313,"prepare.queue_time_us":32178,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":34176,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":15023,"thread_start_us":175899,"threads_started":1,"wal-append.queue_time_us":266525}
I20260709 12:14:01.663045 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:01.641146 12489 rpcz_store.cc:276] 0709 12:14:00.077356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.944115 (+866759us) service_pool.cc:224] Handling call
0709 12:14:00.944579 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:00.945073 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:00.945125 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:00.950281 (+  5156us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.274549 (+324268us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:01.274617 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.274745 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.640430 (+365685us) raft_consensus.cc:1685] finished
0709 12:14:01.640455 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.640601 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2411,"raft.queue_time_us":131,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":8062,"thread_start_us":250827,"threads_started":1,"wal-append.queue_time_us":490996}
I20260709 12:14:01.643445 17931 consensus_queue.cc:1243] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:14:01.666239 12497 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [15.14-17.15]
I20260709 12:14:01.680721 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.666143 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.666823 12646 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
W20260709 12:14:01.617658 18247 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.848s	user 0.004s	sys 0.016s
I20260709 12:14:01.681470 12638 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
W20260709 12:14:01.685081 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:01.685681 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:01.664489 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68897.2us
I20260709 12:14:01.643285 18354 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:01.687039 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.526640 17919 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.687703 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 914) took 1111 ms. Trace:
I20260709 12:14:01.688280 12644 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:01.690930 12567 rpcz_store.cc:276] 0709 12:14:00.576549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.190067 (+613518us) service_pool.cc:224] Handling call
0709 12:14:01.190491 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:01.191280 (+   789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.191333 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.191883 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.457760 (+265877us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:01.457813 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.457897 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.687429 (+229532us) raft_consensus.cc:1685] finished
0709 12:14:01.687452 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.687598 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":131,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"thread_start_us":265516,"threads_started":1,"wal-append.queue_time_us":384975}
I20260709 12:14:01.662051 17950 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.689548 12488 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:14:01.683732 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.664906 17556 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:01.639792 18373 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:01.683763 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:01.699790 18293 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.690605 18247 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.705984 17931 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:14:01.691090 18354 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:01.706811 18380 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:01.707314 17919 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:01.696300 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.712493 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:01.732656 12563 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:01.691169 18127 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.532s	user 0.007s	sys 0.011s
I20260709 12:14:01.687887 18276 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:01.676977 17564 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:14:01.720561 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:01.720889 12495 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:14:01.724082 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:01.725193 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:01.737672 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:01.738363 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.738600 18282 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:01.738526 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.740518 18320 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:14:01.726680 18354 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:01.715313 17931 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:01.713512 18272 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:01.744213 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.754115 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:01.713820 17556 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:01.752370 18292 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.807s	user 0.014s	sys 0.000s
W20260709 12:14:01.763265 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102787us
W20260709 12:14:01.763550 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.842s (SENT) [suppressed 13 similar messages]
I20260709 12:14:01.736734 18374 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:01.762064 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.742563 18242 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:01.763715 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:01.766196 12497 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:01.755100 18377 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:14:01.747020 18272 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:01.772594 18191 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.004s
I20260709 12:14:01.768152 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.744872 18114 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:01.768051 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:01.788090 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:01.746924 17886 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:01.766438 12567 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 12:14:01.798900 12567 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:14:01.799165 12567 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:14:01.799588 12567 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:01.767321 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:01.757620 18257 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:01.821339 18348 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:01.758517 18076 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.736438 18127 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.821933 18348 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:01.822908 18348 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 15.5, Last appended by leader: 5, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.823921 18348 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:01.824092 12489 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [15.5-15.5]
I20260709 12:14:01.809413 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.826903 12490 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:14:01.827615 17639 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.829887 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2336) took 1613 ms. Trace:
I20260709 12:14:01.830542 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2585) took 1900 ms. Trace:
I20260709 12:14:01.831022 12641 rpcz_store.cc:276] 0709 12:13:59.930355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.124929 (+1194574us) service_pool.cc:224] Handling call
0709 12:14:01.125432 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.472712 (+347280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.830302 (+357590us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:01.830480 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796467968}
I20260709 12:14:01.823939 18127 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.796010 18272 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:01.838063 12568 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260709 12:14:01.838536 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.840811 17556 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.765439 18244 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:01.841786 18244 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:01.842829 18244 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.796458 12488 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:01.797883 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:01.846822 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.848398 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.804050 18247 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.804659 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:01.850281 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.807986 18107 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.851259 18107 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:01.852321 18339 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.879s	user 0.000s	sys 0.004s
I20260709 12:14:01.765713 18374 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:01.853884 18374 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:01.855468 18190 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:01.855988 18190 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:01.799415 18240 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:01.793208 18377 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:01.858623 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.858876 18377 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.860014 12572 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:01.860555 18014 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.861109 18014 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.862076 13255 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
W20260709 12:14:01.862896 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 912) took 1690 ms (client timeout 2205 ms). Trace:
W20260709 12:14:01.863312 12572 rpcz_store.cc:269] 0709 12:14:00.172930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.341077 (+168147us) service_pool.cc:224] Handling call
0709 12:14:01.862729 (+1521652us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001a27d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:01.862840 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3173785856}
I20260709 12:14:01.864408 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:01.865864 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:01.866657 13262 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:01.866807 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:01.867419 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:01.754789 18381 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:14:01.796496 18235 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:01.869738 18315 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.830394 12645 rpcz_store.cc:276] 0709 12:14:00.216168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.957824 (+741656us) service_pool.cc:224] Handling call
0709 12:14:01.829614 (+871790us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:01.829808 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1118734208}
I20260709 12:14:01.870401 18315 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:01.766185 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77258.1us
I20260709 12:14:01.872933 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 748) took 1883 ms. Trace:
I20260709 12:14:01.840265 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.873157 18315 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:01.813752 18200 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.878155 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 916) took 1025 ms. Trace:
I20260709 12:14:01.878319 18353 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.878968 18315 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:01.879181 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:01.886407 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:01.818885 18230 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:14:01.819341 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.792608 17931 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.825444 18196 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.850533 13258 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:01.806679 18242 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:01.857558 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.908247 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:01.908869 12646 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:01.813760 18191 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.873487 12496 rpcz_store.cc:276] 0709 12:13:59.989058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:59.992639 (+  3581us) service_pool.cc:224] Handling call
0709 12:13:59.993146 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.993550 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:59.993594 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:59.994295 (+   701us) log.cc:844] Serialized 26 byte log entry
0709 12:14:00.306769 (+312474us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:00.306804 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.306867 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.851628 (+1544761us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:01.872570 (+ 20942us) raft_consensus.cc:1685] finished
0709 12:14:01.872599 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.872866 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13390720,"thread_start_us":286253,"threads_started":1,"wal-append.queue_time_us":921777}
I20260709 12:14:01.911605 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.814828 18293 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:01.874894 18193 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:01.952544 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.877115 18235 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:01.953557 12638 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:01.802142 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:01.768774 18292 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.881165 12488 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:01.878626 12570 rpcz_store.cc:276] 0709 12:14:00.852851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.472381 (+619530us) service_pool.cc:224] Handling call
0709 12:14:01.483069 (+ 10688us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:01.502438 (+ 19369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.502486 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:01.503234 (+   748us) log.cc:844] Serialized 52 byte log entry
0709 12:14:01.577971 (+ 74737us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:01.578036 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.877728 (+299692us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:01.877777 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:01.877830 (+    53us) raft_consensus.cc:1685] finished
0709 12:14:01.877847 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:01.878091 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3745,"raft.queue_time_us":5405,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":15532,"spinlock_wait_cycles":34643200,"thread_start_us":74134,"threads_started":1,"wal-append.queue_time_us":117121}
I20260709 12:14:01.968449 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.891891 18230 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:01.970104 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.970640 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:01.882412 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.896714 17556 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.974267 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:01.888396 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:01.885226 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:01.976868 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63890.6us
I20260709 12:14:01.880457 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:01.891070 12568 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:01.892604 13260 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:01.856906 18190 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.893384 12490 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:14:01.981875 12488 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:01.894431 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:01.990150 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95857.3us
I20260709 12:14:01.898726 12641 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:14:01.992065 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:01.992725 12641 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:14:01.899454 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135929us
I20260709 12:14:01.901957 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.992273 12488 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:14:01.904121 13254 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:14:01.907281 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:01.989339 12490 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:14:01.910611 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.878644 18200 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:01.803144 18114 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:01.912304 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:01.879668 18315 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.912015 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.922647 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:01.977877 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 175784us
I20260709 12:14:02.005607 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.007503 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.008283 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:01.791654 18385 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:01.981957 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:01.977109 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:01.904714 17886 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:01.984428 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.911463 18203 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:01.908006 18240 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:01.994580 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.996250 12490 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 11. Current term is 12. Ops: []
I20260709 12:14:01.997876 13255 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:01.997462 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:02.000890 12640 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:14:02.003989 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:01.978785 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:02.011536 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.024531 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:02.012709 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57365.8us
I20260709 12:14:02.014991 12497 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:01.880909 18354 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:01.903679 12645 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:14:02.025305 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:01.911105 18257 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 12:14:01.974689 18289 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.912458 18213 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.964771 17639 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:02.027504 18190 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:01.964699 18280 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:01.905426 18297 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:14:01.968948 18242 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:01.982936 17556 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:01.972191 17931 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:02.053470 12643 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:02.032017 12572 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:02.054865 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:02.056264 12638 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:02.056412 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.053831 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.057232 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.069530 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:02.120234 12640 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:02.013757 17926 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:02.004982 18293 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:02.077800 12641 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:02.122532 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.107170 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:02.123538 12641 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:14:01.906713 18282 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:14:02.025856 18203 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:02.098496 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:02.131129 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:02.131771 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:02.132241 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:01.917438 18276 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.141s
I20260709 12:14:01.906497 18381 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 12:14:02.010978 18211 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
W20260709 12:14:02.122967 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68200.8us
I20260709 12:14:02.074270 18320 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:14:02.117564 18385 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:02.031503 18355 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.079s
I20260709 12:14:02.125854 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:02.146152 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 235628us
I20260709 12:14:02.145787 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:02.147655 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:02.147699 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:02.127120 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:02.030948 17886 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.089335 18184 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:02.129702 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.136832 12563 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:02.140446 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.155530 12643 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:14:02.118716 18176 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:02.066637 18240 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.147862 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.156373 12641 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:14:02.124380 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.150084 12638 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:14:02.158142 12638 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:02.134081 18203 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:02.083333 18289 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.121528 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:02.150774 12640 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:02.160081 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.172837 12566 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:02.160535 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 247468us
W20260709 12:14:02.163587 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143410us
I20260709 12:14:02.158967 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.167245 12643 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:02.137471 18293 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:02.140478 17926 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:02.086222 18213 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.114847 18280 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:02.174502 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53247.7us
I20260709 12:14:02.174855 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:02.183811 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.184629 12570 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:02.151999 18385 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:02.185885 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.156508 18244 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:02.187646 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:02.188383 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 172418us
I20260709 12:14:02.143054 18282 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:02.190380 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:02.192106 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51806us
I20260709 12:14:02.192719 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.194242 12570 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:02.121719 18235 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.119508 18230 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:02.203655 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.203060 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.175503 18176 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:02.214623 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.182190 18384 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:02.190126 18244 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:02.231076 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.233053 12566 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:02.212646 12570 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:02.213120 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:02.190070 18242 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.7, Last appended by leader: 7, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.232416 18374 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:02.239521 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.244421 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.244920 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.250294 12566 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:02.187940 18373 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:02.255829 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 108024us
I20260709 12:14:02.260219 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.226045 18230 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:02.226257 18353 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:02.238087 18390 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:14:02.209806 18383 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:14:02.240622 18380 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:14:02.267366 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:02.287442 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.243500 18276 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:14:02.222800 18399 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:02.241387 17919 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.508s	user 0.010s	sys 0.000s
I20260709 12:14:02.245352 18211 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.262528 18196 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:02.306670 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 324834us
I20260709 12:14:02.286130 18381 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:02.235442 18402 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.089s
I20260709 12:14:02.311765 13253 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:02.263106 18257 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:14:02.314982 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:02.245008 18191 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:02.318697 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119881us
I20260709 12:14:02.240236 18374 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:02.246906 18384 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:14:02.346575 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.347307 12566 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:14:02.328490 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.244823 18292 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:02.333411 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120359us
I20260709 12:14:02.253357 18355 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:14:02.290107 18354 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:02.291412 18408 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:02.240775 18321 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:14:02.254045 18272 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.312862 18315 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.349264 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:02.267112 18405 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:02.370572 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2335) took 1011 ms. Trace:
I20260709 12:14:02.370998 12494 rpcz_store.cc:276] 0709 12:14:01.359029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.626549 (+267520us) service_pool.cc:224] Handling call
0709 12:14:01.626929 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:01.718575 (+ 91646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.718627 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.719380 (+   753us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.892302 (+172922us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:01.892336 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.892402 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:02.370330 (+477928us) raft_consensus.cc:1685] finished
0709 12:14:02.370351 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:02.370511 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48678,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":12737,"thread_start_us":246927,"threads_started":2,"wal-append.queue_time_us":354036}
W20260709 12:14:02.355489 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66495.9us
I20260709 12:14:02.256013 18293 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.360056 12566 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
W20260709 12:14:02.347222 18389 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.231s	user 0.002s	sys 0.000s
I20260709 12:14:02.265444 17926 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.369371 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:02.376334 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:02.376649 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:02.387072 13255 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:02.376823 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.333527 18211 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.376823 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2557) took 4750 ms. Trace:
W20260709 12:14:02.371758 18391 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 12:14:02.286866 18385 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.387674 12646 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:02.364022 18408 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:02.392292 12640 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:14:02.369109 17886 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:02.336583 18399 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:14:02.395895 12643 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:14:02.316407 18176 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:02.420131 18341 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.577s	user 0.008s	sys 0.000s
I20260709 12:14:02.366353 17556 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.374508 18405 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:14:02.371861 18374 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:14:02.382330 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.424476 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:02.428215 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51942.7us
I20260709 12:14:02.444797 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.395675 12639 rpcz_store.cc:276] 0709 12:13:57.626481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:58.425511 (+799030us) service_pool.cc:224] Handling call
0709 12:13:59.009443 (+583932us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:13:59.009962 (+   519us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:13:59.010003 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:13:59.915583 (+905580us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:13:59.916192 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 12:14:00.262289 (+346097us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:00.573259 (+310970us) log.cc:844] Serialized 14 byte log entry
0709 12:14:00.573537 (+   278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:00.573668 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:00.917732 (+344064us) raft_consensus.cc:1685] finished
0709 12:14:00.917854 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:02.376613 (+1458759us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54000a82d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:02.376757 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1757,"prepare.queue_time_us":652403,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":13237,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":52410,"spinlock_wait_cycles":1384564352,"thread_start_us":625260,"threads_started":2,"wal-append.queue_time_us":966894,"wal-append.run_cpu_time_us":2419,"wal-append.run_wall_time_us":1030693}
W20260709 12:14:02.419691 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50402.4us
I20260709 12:14:02.447242 12566 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:02.428494 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:02.436287 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 248716us
I20260709 12:14:02.388151 18240 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:02.442905 12643 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:14:02.445988 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.427852 18408 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:02.427242 18289 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:02.434799 18222 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:02.451833 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:02.450876 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 472206us
W20260709 12:14:02.454210 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.437006 18213 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.460805 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:02.481254 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.462531 12640 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:02.448470 18384 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.465112 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:02.474408 18409 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:14:02.474958 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:02.474838 18393 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 12:14:02.461215 18257 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:02.475741 18416 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:02.517611 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.517959 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:02.477792 18348 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.653s	user 0.008s	sys 0.004s
W20260709 12:14:02.457963 18107 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.600s	user 0.013s	sys 0.000s
I20260709 12:14:02.464175 18374 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:02.469931 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.518332 12565 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:02.473387 18272 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:02.477160 18396 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 12:14:02.522964 13257 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:14:02.477540 18422 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:02.524557 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.536029 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:02.521576 12566 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:14:02.526651 13255 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:02.491096 12640 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:14:02.474489 18373 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.512275 18196 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:02.527279 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:02.541024 13262 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:02.528399 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:02.534986 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.484956 18227 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.526469 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:02.533151 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:02.542938 12494 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:02.543206 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2742) took 2652 ms. Trace:
I20260709 12:14:02.533567 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:02.531193 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:02.544158 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:02.534228 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:02.543665 12571 rpcz_store.cc:276] 0709 12:13:59.891138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.038607 (+147469us) service_pool.cc:224] Handling call
0709 12:14:00.038944 (+   337us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:00.039354 (+   410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:00.039383 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:00.040065 (+   682us) log.cc:844] Serialized 52 byte log entry
0709 12:14:00.364261 (+324196us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:00.940540 (+576279us) log.cc:844] Serialized 14 byte log entry
0709 12:14:01.012168 (+ 71628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:01.012267 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:02.377180 (+1364913us) spinlock_profiling.cc:243] Waited 8.57 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:02.542903 (+165723us) raft_consensus.cc:1685] finished
0709 12:14:02.542997 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:02.543141 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106804,"prepare.queue_time_us":512932,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":229,"raft.queue_time_us":74224,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":57120,"spinlock_wait_cycles":19708288,"thread_start_us":871919,"threads_started":2,"wal-append.queue_time_us":719096}
I20260709 12:14:02.523427 18289 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:02.546657 12488 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:02.528709 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:02.548242 13260 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20260709 12:14:02.551261 13257 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:02.558370 12565 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:02.559544 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.560901 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.567967 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.568854 12566 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:02.560148 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:02.570099 12565 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:02.518502 18405 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:14:02.572687 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:02.573832 13257 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:02.584324 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:02.585108 12571 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:02.588243 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.589666 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:02.512569 18384 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.550609 18289 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:02.593000 18289 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:14:02.619879 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140752us
W20260709 12:14:02.621081 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.638741 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:02.571326 18200 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.566s	user 0.013s	sys 0.000s
W20260709 12:14:02.669983 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:02.671051 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.593755 17919 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:02.677197 18411 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.263s	user 0.000s	sys 0.002s
I20260709 12:14:02.585158 18196 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
I20260709 12:14:02.516605 18416 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:02.722076 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130474us
I20260709 12:14:02.560375 18227 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.659143 18374 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.560348 18257 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.719743 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:02.723376 12566 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 15 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 12:14:02.684484 18341 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:14:02.725397 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 755) took 1469 ms. Trace:
W20260709 12:14:02.731154 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92242.1us
I20260709 12:14:02.744657 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.731828 13262 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:02.747269 13255 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:14:02.732486 12492 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:14:02.742162 12493 rpcz_store.cc:276] 0709 12:14:01.255725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.602657 (+346932us) service_pool.cc:224] Handling call
0709 12:14:01.603035 (+   378us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:01.752697 (+149662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:01.752739 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:01.760399 (+  7660us) log.cc:844] Serialized 52 byte log entry
0709 12:14:02.013196 (+252797us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:02.013235 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:02.013321 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:02.725140 (+711819us) raft_consensus.cc:1685] finished
0709 12:14:02.725162 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:02.725334 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30522,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":113,"thread_start_us":323214,"threads_started":2,"wal-append.queue_time_us":385288}
I20260709 12:14:02.684463 18419 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:02.761376 18419 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.722863 17919 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:02.722769 18422 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:02.721101 18242 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.773773 12645 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:02.729897 18354 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.719810 18120 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:02.775146 18428 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.075s	user 0.000s	sys 0.004s
W20260709 12:14:02.770924 18394 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
W20260709 12:14:02.774451 18193 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.805s	user 0.015s	sys 0.004s
W20260709 12:14:02.768738 18190 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.658s	user 0.009s	sys 0.008s
I20260709 12:14:02.754736 18377 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:14:02.773203 18433 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.784647 18211 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:02.800184 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:02.767531 18384 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:02.783892 18114 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.753s	user 0.020s	sys 0.005s
I20260709 12:14:02.774113 18415 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:02.770709 18416 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.765522 18419 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.788542 18230 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:14:02.811589 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78719.5us
I20260709 12:14:02.840721 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:02.785038 18354 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.845392 18375 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.846051 18375 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.847715 12491 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:14:02.839810 18272 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:14:02.820001 18415 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:02.864784 18415 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.868546 18273 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:02.869740 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:02.818485 18114 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.786165 18242 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.869776 18422 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:02.788630 18120 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:02.874673 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2326) took 1594 ms (client timeout 2088 ms). Trace:
W20260709 12:14:02.879791 12492 rpcz_store.cc:269] 0709 12:14:01.280401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.489154 (+208753us) service_pool.cc:224] Handling call
0709 12:14:02.874480 (+1385326us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:02.874604 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1878333184}
I20260709 12:14:02.876406 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2747) took 2953 ms. Trace:
I20260709 12:14:02.882076 12569 rpcz_store.cc:276] 0709 12:13:59.923058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.049240 (+126182us) service_pool.cc:224] Handling call
0709 12:14:00.049678 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:00.054896 (+  5218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:00.054929 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.707707 (+1652778us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:14:02.338360 (+630653us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:02.350790 (+ 12430us) log.cc:844] Serialized 26 byte log entry
0709 12:14:02.665256 (+314466us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:02.665320 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:02.665447 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:02.876077 (+210630us) raft_consensus.cc:1685] finished
0709 12:14:02.876113 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:02.876319 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4751,"raft.queue_time_us":5116,"raft.run_cpu_time_us":8683,"raft.run_wall_time_us":1634042,"spinlock_wait_cycles":22235776,"thread_start_us":243248,"threads_started":1,"wal-append.queue_time_us":417488}
W20260709 12:14:02.889024 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48390.2us
I20260709 12:14:02.890542 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:02.893383 18355 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:14:02.894029 18355 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:02.895291 18355 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.900068 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.903733 12569 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:02.874065 18242 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:02.911984 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:02.892380 17931 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.767s	user 0.012s	sys 0.001s
I20260709 12:14:02.872520 18237 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:02.925041 12571 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:02.925880 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2765) took 1025 ms. Trace:
I20260709 12:14:02.926235 12571 rpcz_store.cc:276] 0709 12:14:01.900318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.812860 (+912542us) service_pool.cc:224] Handling call
0709 12:14:02.875040 (+ 62180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:02.925549 (+ 50509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:02.925817 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:02.913410 12569 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:14:02.812489 18433 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.928999 12569 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:14:02.915124 18230 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.915690 18419 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:02.917153 18315 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:02.925715 18237 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:02.920817 18429 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:02.956004 18230 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.899945 18408 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.917475 18321 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:14:02.890203 17919 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:02.969934 18242 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.009s
I20260709 12:14:02.977933 18412 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:14:02.998476 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 217011us
I20260709 12:14:02.959811 17931 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.009397 12565 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 11. Current term is 12. Ops: []
I20260709 12:14:02.995911 18193 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.025327 12496 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:03.001991 18203 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.825s	user 0.019s	sys 0.013s
I20260709 12:14:03.001219 18227 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:03.032295 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102996us
I20260709 12:14:03.033443 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:02.999420 18384 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:14:03.041922 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.057307 12492 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:03.043723 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.044533 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:03.050819 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.040570 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.045722 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.060168 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:03.061698 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.029320 18412 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:02.999816 18429 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.011642 18190 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:14:03.068619 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.027135 18315 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:14:03.072831 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:03.027307 17931 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.061002 18398 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:03.063850 18292 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:03.034224 18203 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.030716 18193 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.085681 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.038013 18273 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:14:03.077427 17849 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:14:03.038467 18425 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
W20260709 12:14:03.094738 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:03.118966 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.072752 18211 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:03.097398 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.097858 18451 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:03.098114 18230 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:03.083429 18374 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:03.080075 18419 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:03.077674 18114 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.100068 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:03.083340 18433 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:03.101295 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.094553 18292 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.102154 18412 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.142148 18419 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.094116 18242 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:03.103344 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.104977 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2763) took 1205 ms. Trace:
I20260709 12:14:03.145033 12566 rpcz_store.cc:276] 0709 12:14:01.899702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.767217 (+867515us) service_pool.cc:224] Handling call
0709 12:14:02.767715 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:02.768170 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:02.768203 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:02.768780 (+   577us) log.cc:844] Serialized 26 byte log entry
0709 12:14:02.878587 (+109807us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:02.878633 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:02.878711 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:03.104723 (+226012us) raft_consensus.cc:1685] finished
0709 12:14:03.104747 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:03.104915 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1660,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":113,"thread_start_us":109315,"threads_started":1,"wal-append.queue_time_us":266581}
I20260709 12:14:03.106775 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:03.107373 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.090900 18321 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:03.112674 18408 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:03.113837 18444 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:14:03.114621 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.116335 18315 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:03.108245 18429 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.123715 17931 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.124156 18326 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:03.125372 18448 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:14:03.132840 13259 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:14:03.137468 12642 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:03.147908 12497 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:03.077673 18257 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 12:14:03.149595 12494 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 13: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:03.149246 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2338) took 2929 ms (client timeout 1876 ms). Trace:
W20260709 12:14:03.179363 12637 rpcz_store.cc:269] 0709 12:14:00.219650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.331966 (+112316us) service_pool.cc:224] Handling call
0709 12:14:03.148949 (+2816983us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:03.149139 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":663766400}
W20260709 12:14:03.153939 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50651.1us
I20260709 12:14:03.179790 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.155879 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:03.158305 18227 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:14:03.161257 18433 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.181566 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:03.182874 18433 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.184593 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:03.162349 18244 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.186197 18244 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:03.163045 18193 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:03.164808 18289 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.507s	user 0.011s	sys 0.002s
I20260709 12:14:03.166673 18424 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.136592 18374 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:03.181134 12497 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:03.149173 18425 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:03.186578 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.168383 18230 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:03.200332 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:03.205307 12638 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:14:03.172761 18326 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:03.208267 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:03.178464 18448 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:03.209041 18422 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:03.196874 18429 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.167909 18450 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:14:03.194280 18289 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.196943 18425 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:03.196736 18227 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:03.231432 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.256960 18448 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.211843 18456 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:03.258904 18448 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:14:03.267297 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.269541 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.270022 12646 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:14:03.272661 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.273613 12646 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:03.275743 18453 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:03.277052 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.277143 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:03.278716 12646 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:03.278788 12638 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:03.229391 18433 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.294492 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:03.198993 18424 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.295511 12640 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:03.295953 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:03.297833 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:03.256987 18429 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.299154 18422 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:03.300536 18422 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:03.300752 18429 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.302270 18455 raft_consensus.cc:2764] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:03.213568 18412 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:14:03.298357 18289 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:14:03.277642 18292 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.282579 18315 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:03.329751 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.304163 18257 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.312247 18433 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:03.303550 18203 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.290338 18420 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.122s
I20260709 12:14:03.312273 17919 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.336233 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:03.332526 18380 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:03.332497 18315 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:03.338536 18433 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:03.378183 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2721) took 2247 ms. Trace:
I20260709 12:14:03.337591 18292 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:03.378753 12567 rpcz_store.cc:276] 0709 12:14:01.138066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.693134 (+555068us) service_pool.cc:224] Handling call
0709 12:14:01.693596 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:01.800319 (+106723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.377898 (+1577579us) spinlock_profiling.cc:243] Waited 70.4 ms on lock 0x7b54003ac354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:03.378114 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161872768}
W20260709 12:14:03.354578 18452 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.182s	user 0.002s	sys 0.000s
I20260709 12:14:03.355595 18227 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:03.277997 18374 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:03.412496 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58385.8us
I20260709 12:14:03.425761 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:03.342231 17886 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.351548 18380 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:03.351465 18454 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:03.339725 18257 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.376648 18315 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.384711 18433 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.421860 12640 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:03.343770 18429 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:03.336366 18326 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.398101 18355 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.348042 17919 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.429322 18451 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:03.429806 17886 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.442243 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:03.410496 18412 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:03.461061 12639 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 7. Preceding OpId from leader: term: 18 index: 8. (index mismatch)
I20260709 12:14:03.417521 18310 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:14:03.453394 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:03.335143 18425 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.464097 18203 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:03.464663 18203 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:03.464721 18425 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:03.465965 18203 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.478374 18310 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:03.479799 18310 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.482309 18450 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.483064 18450 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.486511 18450 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.489060 18310 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:03.495914 18462 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:14:03.496555 17886 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:03.497735 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.498776 12640 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:03.500181 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.500980 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:03.396715 18422 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.506063 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:03.507092 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:03.509657 18470 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.509753 18326 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:03.510351 18470 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.519217 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:03.538892 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:03.509377 18385 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.493417 18374 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:03.459903 18451 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:03.559274 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.512921 18470 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:03.558996 18310 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:14:03.564327 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.567728 12640 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:14:03.491693 18429 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.515443 18355 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.558677 18450 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:03.517931 18412 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.493269 18454 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.585381 18454 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.588512 17867 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:03.589247 17867 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:03.560777 18451 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:03.547363 18380 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.575475 18450 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:03.605756 18191 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:03.606370 18450 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.606603 18191 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:03.607604 18415 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.605921 17919 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.608040 18191 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.570479 18310 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:03.596400 18412 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:03.596264 18237 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.597s	user 0.011s	sys 0.000s
I20260709 12:14:03.595817 18355 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.610014 18380 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.613562 18310 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.632030 18456 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:03.642675 18429 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.643806 18433 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.646493 18433 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:03.648058 18451 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.650419 18451 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.692509 12494 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:03.697356 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:03.707290 12642 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:14:03.727923 12640 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:14:03.739079 17867 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.593464 18120 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.749092 18257 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.619947 18237 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.632647 18382 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:03.612509 18191 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:03.655143 18450 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.801364 18120 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:03.802222 18120 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:03.802670 18120 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:03.798182 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99349.4us
I20260709 12:14:03.821069 18382 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:03.822127 17867 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:03.822901 18382 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.825137 18382 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:03.825999 18315 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.837182 18315 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.848068 18237 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:03.864933 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:03.865677 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.867259 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2723) took 2728 ms. Trace:
I20260709 12:14:03.867568 18257 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.868513 18257 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.870028 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.870458 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2761) took 1971 ms. Trace:
I20260709 12:14:03.870805 12565 rpcz_store.cc:276] 0709 12:14:01.899088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.720942 (+821854us) service_pool.cc:224] Handling call
0709 12:14:02.721286 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:03.012433 (+291147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.869907 (+857474us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:03.870266 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":661994880}
I20260709 12:14:03.871496 18257 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.864964 18237 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
W20260709 12:14:03.876156 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 925) took 1988 ms (client timeout 2203 ms). Trace:
W20260709 12:14:03.876520 18230 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.006s
W20260709 12:14:03.876694 12572 rpcz_store.cc:269] 0709 12:14:01.887814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.888002 (+   188us) service_pool.cc:224] Handling call
0709 12:14:03.875701 (+1987699us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:03.875930 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115752192}
I20260709 12:14:03.877245 18230 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:03.881197 12639 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:14:03.882916 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2604) took 1980 ms. Trace:
I20260709 12:14:03.883500 12641 rpcz_store.cc:276] 0709 12:14:01.902000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.157291 (+255291us) service_pool.cc:224] Handling call
0709 12:14:02.158515 (+  1224us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:02.178130 (+ 19615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:02.178166 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:02.178225 (+    59us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:02.178262 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.882190 (+1703928us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:03.882307 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:03.882644 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44554496}
I20260709 12:14:03.886497 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2592) took 1987 ms. Trace:
I20260709 12:14:03.886891 12644 rpcz_store.cc:276] 0709 12:14:01.898545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.899306 (+   761us) service_pool.cc:224] Handling call
0709 12:14:03.149360 (+1250054us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:03.153679 (+  4319us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:03.153710 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:03.190752 (+ 37042us) log.cc:844] Serialized 52 byte log entry
0709 12:14:03.282417 (+ 91665us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:14:03.282538 (+   121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.282674 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:03.396893 (+114219us) raft_consensus.cc:1685] finished
0709 12:14:03.396917 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:03.886056 (+489139us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54001f7cd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:03.886306 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3189,"raft.queue_time_us":4239,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2516,"spinlock_wait_cycles":2752005504,"thread_start_us":83622,"threads_started":1,"wal-append.queue_time_us":178781}
W20260709 12:14:03.889822 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2608) took 1986 ms (client timeout 1932 ms). Trace:
W20260709 12:14:03.890363 12645 rpcz_store.cc:269] 0709 12:14:01.902973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.053547 (+150574us) service_pool.cc:224] Handling call
0709 12:14:03.889426 (+1835879us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:03.889592 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1503532800}
I20260709 12:14:03.892976 18450 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.895967 12493 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:03.898201 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:03.900436 12570 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260709 12:14:03.901033 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.902100 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.906394 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:03.907262 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.908272 12497 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:03.867832 12563 rpcz_store.cc:276] 0709 12:14:01.138901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.797310 (+658409us) service_pool.cc:224] Handling call
0709 12:14:01.797609 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:02.155546 (+357937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.866644 (+1711098us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:03.867034 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779115264}
I20260709 12:14:03.908849 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:03.910071 18415 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:14:03.910877 12489 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:14:03.911485 18415 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:03.912158 18415 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:03.912603 18415 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:03.914237 18014 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.910069 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:03.921423 18451 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.922434 18451 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.922632 18310 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:03.925493 12493 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:03.931481 18296 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:03.932440 18296 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.936612 18324 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:03.937395 18433 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:03.938061 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:03.938448 18433 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:03.939010 18433 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:03.939574 18296 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:03.939066 12494 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:03.942273 18433 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:03.943657 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:03.943917 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 803) took 1578 ms. Trace:
I20260709 12:14:03.945080 18014 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.947956 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:03.955948 12567 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:03.958297 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:03.978546 18014 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.979426 18191 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:03.979674 12569 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 15 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 5.
I20260709 12:14:03.963843 12492 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:14:03.980172 18191 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:03.981063 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:03.981988 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.982596 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.983618 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.984337 13259 rpcz_store.cc:276] 0709 12:14:02.365010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.365182 (+   172us) service_pool.cc:224] Handling call
0709 12:14:02.365666 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:03.178827 (+813161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.943283 (+764456us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:03.943651 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1763377408}
I20260709 12:14:03.985742 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.986191 18014 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.987946 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.990473 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.991623 13257 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:14:03.977952 18433 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:03.994115 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:03.945195 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 805) took 1574 ms. Trace:
I20260709 12:14:03.995688 13260 rpcz_store.cc:276] 0709 12:14:02.371071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.372575 (+  1504us) service_pool.cc:224] Handling call
0709 12:14:02.372952 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:02.548752 (+175800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.944570 (+1395818us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:03.944982 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":402172288}
I20260709 12:14:03.998530 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:03.963704 18184 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:03.999562 18184 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:04.003489 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:03.971084 18451 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:03.976341 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:03.976658 18120 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.004891 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:04.006767 18274 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:14:04.008736 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:03.914463 18244 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.727s	user 0.001s	sys 0.029s
I20260709 12:14:04.011564 18354 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.012070 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.016422 18320 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:04.024533 13257 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:14:04.016745 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:03.968895 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:04.017305 18014 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:14:04.014312 18450 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:04.022342 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.029184 18274 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:14:04.029196 18320 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.039827 12567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:04.029574 18451 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.044307 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.044168 18451 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.045284 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:04.046145 12570 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 15. (term mismatch)
I20260709 12:14:04.046343 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:04.046553 12570 pending_rounds.cc:85] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 14
I20260709 12:14:04.046877 12570 pending_rounds.cc:107] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260709 12:14:04.046933 18451 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.057261 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.058288 12644 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:14:04.059757 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:04.051059 18375 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.051357 18320 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.047189 12570 raft_consensus.cc:2889] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:04.052800 18280 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:04.048699 18450 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.054064 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:04.055341 18310 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.071666 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.073140 18244 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:14:04.075104 18460 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.628s	user 0.002s	sys 0.000s
W20260709 12:14:04.075623 18441 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.052s	user 0.002s	sys 0.000s
W20260709 12:14:04.076094 18440 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.210s	user 0.002s	sys 0.000s
I20260709 12:14:04.082141 18227 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.078192 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.093489 13257 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:14:04.052053 18191 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.097198 17564 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:14:04.098114 18292 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.671s	user 0.007s	sys 0.014s
I20260709 12:14:04.098845 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2735) took 2088 ms. Trace:
I20260709 12:14:04.100018 18421 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:04.100373 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2354) took 1659 ms. Trace:
I20260709 12:14:04.111816 18421 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:04.112285 12643 rpcz_store.cc:276] 0709 12:14:02.440822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.492252 (+ 51430us) service_pool.cc:224] Handling call
0709 12:14:02.492614 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:02.492863 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:02.492888 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:02.513135 (+ 20247us) log.cc:844] Serialized 26 byte log entry
0709 12:14:02.755015 (+241880us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:02.755355 (+   340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:02.755452 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:04.055670 (+1300218us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:04.099786 (+ 44116us) raft_consensus.cc:1685] finished
0709 12:14:04.099821 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:04.100179 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26934656,"thread_start_us":203426,"threads_started":1,"wal-append.queue_time_us":307302}
I20260709 12:14:04.115548 18421 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:04.116693 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.110672 12571 rpcz_store.cc:276] 0709 12:14:02.010475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.927143 (+916668us) service_pool.cc:224] Handling call
0709 12:14:02.929608 (+  2465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:03.062247 (+132639us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:03.062278 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:03.079454 (+ 17176us) log.cc:844] Serialized 26 byte log entry
0709 12:14:03.256251 (+176797us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:14:03.256291 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:03.256364 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:04.051475 (+795111us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:04.098174 (+ 46699us) raft_consensus.cc:1685] finished
0709 12:14:04.098200 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:04.098608 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":64194,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":1720,"spinlock_wait_cycles":4794624,"thread_start_us":226163,"threads_started":2,"wal-append.queue_time_us":239338}
I20260709 12:14:04.118451 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.118891 12571 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:04.119447 12563 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:14:04.123543 18375 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:04.125180 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:04.126142 12566 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:04.129288 12571 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:04.129787 12571 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:14:04.130106 12571 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:04.130999 18120 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:04.119397 18434 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.103s	user 0.004s	sys 0.000s
I20260709 12:14:04.125761 18375 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:04.133229 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.134284 18375 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.090248 18244 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:04.131014 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:04.143177 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.148296 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:04.149730 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.135705 18227 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:04.154932 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 186356us
I20260709 12:14:04.155867 18184 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.158078 18172 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:04.160290 18191 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:04.160740 18191 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:04.162861 18415 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.162081 18191 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.166909 18191 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:04.169507 18184 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.176839 18450 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.177846 12569 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:04.179510 18415 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:04.182292 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.187458 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:04.145649 18292 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:04.190691 12563 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 12:14:04.194013 12565 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 12:14:04.196238 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:04.200037 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46319us
I20260709 12:14:04.203351 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2745) took 4280 ms. Trace:
I20260709 12:14:04.212642 13257 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:14:04.212971 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.221151 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:04.225167 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.203902 12564 rpcz_store.cc:276] 0709 12:13:59.922437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:00.039119 (+116682us) service_pool.cc:224] Handling call
0709 12:14:00.039422 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:00.598040 (+558618us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:00.900189 (+302149us) log.cc:844] Serialized 14 byte log entry
0709 12:14:01.165969 (+265780us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:01.170437 (+  4468us) log.cc:844] Serialized 26 byte log entry
0709 12:14:01.170614 (+   177us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:01.170648 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:04.202170 (+3031522us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:04.202278 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:04.202358 (+    80us) raft_consensus.cc:1685] finished
0709 12:14:04.202473 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:04.203104 (+   631us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57064,"prepare.queue_time_us":284691,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":5357,"raft.queue_time_us":100545,"raft.run_cpu_time_us":503,"raft.run_wall_time_us":16313,"spinlock_wait_cycles":1239602432,"thread_start_us":527957,"threads_started":2,"wal-append.queue_time_us":383171,"wal-append.run_cpu_time_us":2254,"wal-append.run_wall_time_us":1064446}
I20260709 12:14:04.208583 18292 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.239830 18191 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:04.236685 18274 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.241576 18274 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.246948 18274 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.247941 18292 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.250633 18292 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.252671 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.254113 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:04.266451 18424 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.267789 12563 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
W20260709 12:14:04.270298 18395 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.050s	user 0.004s	sys 0.004s
W20260709 12:14:04.273599 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.280310 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.257175 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:04.283586 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:04.281302 12564 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:14:04.284322 18191 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:14:04.293391 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:04.294888 13260 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:04.296066 17867 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.297004 17867 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.277068 18382 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.318542 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.319710 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
W20260709 12:14:04.326128 18482 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.128s	user 0.002s	sys 0.000s
I20260709 12:14:04.332355 12571 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260709 12:14:04.333719 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.322875 17564 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:14:04.341472 18481 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.211s	user 0.000s	sys 0.002s
I20260709 12:14:04.341697 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:04.342083 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:04.344116 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:04.346244 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.348047 12566 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:04.327172 18375 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.354893 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:04.361337 12569 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:04.353240 18382 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.370532 18402 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:04.343777 17867 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.372370 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:04.350332 18292 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:14:04.380458 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.381088 12494 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:04.381579 18375 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:04.397923 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50948.2us
I20260709 12:14:04.327004 18244 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.402233 12490 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
W20260709 12:14:04.402428 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 182016us
W20260709 12:14:04.392686 12461 connection.cc:256] RPC call timeout handler was delayed by -1.6024999750000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
W20260709 12:14:04.404179 12461 connection.cc:256] RPC call timeout handler was delayed by -1.4759395650000897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1539s for a response
W20260709 12:14:04.405537 12461 connection.cc:256] RPC call timeout handler was delayed by -1.4602999749999981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
W20260709 12:14:04.405898 12461 connection.cc:256] RPC call timeout handler was delayed by -1.3268000579998898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
W20260709 12:14:04.406183 12461 connection.cc:256] RPC call timeout handler was delayed by -1.269499974999917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
W20260709 12:14:04.406499 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1214294960000188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
W20260709 12:14:04.406872 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1105763609998576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1932s for a response
W20260709 12:14:04.407245 12461 connection.cc:256] RPC call timeout handler was delayed by -1.094129662000114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
I20260709 12:14:04.403848 17867 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.417416 18402 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.434010 18289 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:04.434077 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:04.434696 18289 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:04.435091 13256 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:04.437503 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:04.439658 18483 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.313s	user 0.002s	sys 0.001s
I20260709 12:14:04.443424 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2379) took 1014 ms. Trace:
I20260709 12:14:04.444761 18402 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.445659 18382 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.446914 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2334) took 3088 ms. Trace:
I20260709 12:14:04.447692 18380 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.448252 18237 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:04.448863 18486 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.318s	user 0.001s	sys 0.000s
W20260709 12:14:04.449925 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.435170 18289 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:04.459723 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.459486 12491 rpcz_store.cc:276] 0709 12:14:01.358733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:01.624728 (+265995us) service_pool.cc:224] Handling call
0709 12:14:01.625073 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:02.848141 (+1223068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:04.446318 (+1598177us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:04.446676 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2810720000}
I20260709 12:14:04.421931 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:04.451676 12640 rpcz_store.cc:276] 0709 12:14:03.428580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:03.798956 (+370376us) service_pool.cc:224] Handling call
0709 12:14:03.845081 (+ 46125us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:03.849363 (+  4282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:03.849422 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:03.850594 (+  1172us) log.cc:844] Serialized 52 byte log entry
0709 12:14:04.028761 (+178167us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:04.070132 (+ 41371us) log.cc:844] Serialized 14 byte log entry
0709 12:14:04.128104 (+ 57972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:04.128184 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:04.332987 (+204803us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:04.442967 (+109980us) raft_consensus.cc:1685] finished
0709 12:14:04.443127 (+   160us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:04.443348 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57164,"prepare.queue_time_us":37090,"raft.queue_time_us":45524,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":264,"spinlock_wait_cycles":44322816,"thread_start_us":177565,"threads_started":1,"wal-append.queue_time_us":216360}
I20260709 12:14:04.423372 18436 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:04.465042 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:04.479100 13256 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:14:04.480942 12494 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:04.486405 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:04.481804 17867 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.493983 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.489177 18415 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.495762 18415 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.497267 18222 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:04.497848 18222 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:04.498232 18404 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 12:14:04.499562 18414 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:04.499979 18414 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:04.500355 18414 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:04.488869 17859 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:04.483285 18436 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:04.505985 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.502569 18237 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:04.513525 13261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 712) took 1996 ms (client timeout 2100 ms). Trace:
I20260709 12:14:04.513940 17859 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.516301 17859 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:04.514048 13261 rpcz_store.cc:269] 0709 12:14:02.517078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:02.517637 (+   559us) service_pool.cc:224] Handling call
0709 12:14:04.513117 (+1995480us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:04.513319 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553072384}
I20260709 12:14:04.519578 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:04.520799 12569 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:14:04.466429 18380 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.525717 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:04.507397 18436 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:04.506359 18382 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:04.534938 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:04.538478 18415 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:04.539687 18211 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:04.540252 18211 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:04.540711 18211 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:04.541803 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:04.541954 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120198us
I20260709 12:14:04.547420 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:04.549533 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46678.7us
I20260709 12:14:04.550596 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:04.551334 17859 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:04.551818 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:04.531041 18230 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:04.534219 18375 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.554724 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:04.561169 18272 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:04.563274 18272 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.566509 18405 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:04.567219 18405 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:04.568158 18272 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.568346 18405 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.565116 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:04.578951 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:04.555927 18230 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:04.560650 18415 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:14:04.584158 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50063.4us
I20260709 12:14:04.581859 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:04.586905 12491 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:04.584977 18280 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:04.584589 18230 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:04.580973 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.578137 18227 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.615890 12569 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:14:04.589387 18272 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.618498 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.617215 12494 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:04.593721 18415 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:04.618146 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.628181 12640 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:04.609714 18382 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:04.631422 12643 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:14:04.631785 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.632790 12645 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:14:04.632979 12638 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 12:14:04.633756 18191 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:04.634433 18230 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:04.637801 18494 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 12:14:04.638142 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.639315 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:04.639858 18354 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:04.640291 18354 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:04.642268 18431 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:04.642712 18431 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:04.645234 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2370) took 1309 ms. Trace:
I20260709 12:14:04.616772 18399 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.095s
W20260709 12:14:04.626204 18421 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.510s	user 0.004s	sys 0.012s
I20260709 12:14:04.654376 18421 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.655083 18380 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.655719 18380 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:14:04.656636 12571 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:14:04.658453 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.659893 17867 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.660174 18421 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:04.630925 18382 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:04.661374 18382 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:04.661485 18237 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.662915 18415 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.642268 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:04.671345 18464 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.251s	user 0.000s	sys 0.004s
I20260709 12:14:04.672173 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:04.682726 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.683800 12638 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 15 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 5.
I20260709 12:14:04.685276 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.687609 18244 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:04.694053 18421 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:04.694725 18272 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.696933 18272 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:04.701269 12642 rpcz_store.cc:276] 0709 12:14:03.336280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:03.336419 (+   139us) service_pool.cc:224] Handling call
0709 12:14:03.336754 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:03.799660 (+462906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:04.644865 (+845205us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:04.645128 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":849818368}
I20260709 12:14:04.706487 12640 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:04.709085 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:04.710263 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:04.710909 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.719539 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:04.718519 12609 connection.cc:256] RPC call timeout handler was delayed by -2.0829479770000034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260709 12:14:04.720348 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
W20260709 12:14:04.730305 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:04.730912 12609 connection.cc:256] RPC call timeout handler was delayed by -1.9344445740000538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260709 12:14:04.731465 12609 connection.cc:256] RPC call timeout handler was delayed by -1.2196444569999585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20260709 12:14:04.731768 12609 connection.cc:256] RPC call timeout handler was delayed by -1.0567118750000191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
I20260709 12:14:04.732621 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:04.733860 18273 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:04.735555 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:04.736615 18273 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:04.737152 18273 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:04.738173 18273 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.744465 18222 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.745885 12641 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:04.745826 18466 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:04.746097 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.746796 18466 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.709354 18431 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.749054 18466 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.641276 18354 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.713276 18237 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:04.749655 18431 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:04.750473 18354 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:04.750816 18222 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.707662 17867 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:14:04.749735 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.747272 12642 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 13 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
W20260709 12:14:04.763520 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.777671 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.777537 18414 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.778816 12572 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:14:04.779920 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.781659 17867 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:04.782333 17867 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.779984 18414 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:04.787215 18382 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:04.789126 12646 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:04.784670 17867 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.791637 12638 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:04.846762 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 174693us
I20260709 12:14:04.789608 18382 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:04.867551 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.790668 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:04.869478 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.834223 17859 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:04.767968 18211 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:04.884176 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 363587us
W20260709 12:14:04.886991 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96445.5us
I20260709 12:14:04.889432 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.890008 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:04.903398 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 351616us
W20260709 12:14:04.881139 18247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.657s	user 0.010s	sys 0.014s
I20260709 12:14:04.917107 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.918025 12643 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:04.894354 18382 raft_consensus.cc:2764] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:14:04.904989 18238 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:14:04.912551 18247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.704s	user 0.010s	sys 0.015s
I20260709 12:14:04.924908 12645 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:14:04.923772 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:04.931962 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:04.872006 18289 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:04.935073 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.935303 12638 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:14:04.938671 12565 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 15. Ops: []
I20260709 12:14:04.946497 12640 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:04.947590 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:04.948190 12640 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:04.948949 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:04.950071 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.951413 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.951999 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:04.952874 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.953917 18417 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:04.954658 12637 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:14:04.955749 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:04.950240 18496 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:04.957341 12637 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:14:04.960130 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:04.961038 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:04.961817 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:04.979278 12491 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:04.962365 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 523163us
I20260709 12:14:04.963902 18289 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:04.963934 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:04.967075 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:04.953583 12642 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:04.969576 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71566.5us
I20260709 12:14:04.978184 12490 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:14:04.928665 18238 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:05.019248 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.020208 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:04.964426 18348 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 12:14:05.032068 12568 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:05.035059 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.037267 12565 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:05.038455 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:05.033473 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48778.2us
W20260709 12:14:05.041920 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.042327 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173028us
I20260709 12:14:04.930809 18436 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.045483 12494 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:05.039263 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:05.051589 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.028908 18238 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:05.063668 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.034170 18230 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:05.074172 12572 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:05.081902 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 189333us
I20260709 12:14:05.033560 17867 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:05.090538 18497 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.096073 12645 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:05.091785 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 156566us
W20260709 12:14:05.093767 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.094465 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:05.100149 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.104215 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:05.074787 18230 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:05.113404 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.048552 18273 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:14:05.117197 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:05.118773 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61219.7us
I20260709 12:14:05.092993 18237 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:05.067879 18382 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:14:05.082312 18354 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.120070 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:05.119141 12563 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:05.126029 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:05.128062 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.113579 18247 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:05.115651 18230 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:05.129722 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:05.146745 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:05.147495 12490 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:05.142220 12645 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:14:05.145344 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.149887 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.150655 12638 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:05.152675 12646 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:14:05.153934 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.154652 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:14:05.155673 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:05.156365 12642 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:05.157042 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.158090 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:05.158761 18355 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:05.159659 18355 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.153947 18415 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:05.162663 18415 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:05.126340 18380 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:05.153167 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67646us
I20260709 12:14:05.102700 18497 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.188956 18497 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:05.190737 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.195945 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:05.145931 18273 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:05.092561 17859 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.204970 18273 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:05.211866 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:05.212700 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:05.212996 18505 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:14:05.136472 18354 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.218797 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.217104 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:05.219995 12642 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:14:05.228098 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.228297 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:05.210687 18380 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:05.217375 18273 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:05.199121 17867 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:14:05.268759 12643 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:05.276211 12640 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:05.279300 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.285944 18414 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:05.286685 18414 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:05.294996 18244 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.607s	user 0.008s	sys 0.004s
I20260709 12:14:05.296135 18354 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:05.296726 18276 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:14:05.298210 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.296775 18354 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:05.163727 18415 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.267680 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:05.297318 18276 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:05.305663 12566 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:14:05.306442 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:05.324106 12566 pending_rounds.cc:85] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 13
I20260709 12:14:05.265110 18496 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:05.277751 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 141905us
I20260709 12:14:05.324404 12566 pending_rounds.cc:107] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:14:05.324967 12566 raft_consensus.cc:2889] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:05.330116 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.331118 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.337527 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:05.343304 12642 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:14:05.344406 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 203659us
I20260709 12:14:05.348578 12572 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:14:05.350404 12490 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [12.6-14.7]
I20260709 12:14:05.355060 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:05.344899 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:05.357173 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:05.362504 12494 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:05.361428 18496 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.372017 18496 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:05.373769 18405 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:05.376631 12491 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:14:05.377033 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 474807us
W20260709 12:14:05.378578 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.380514 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141992us
I20260709 12:14:05.381794 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:05.382929 18405 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:05.384182 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:05.385166 13257 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:05.386385 18421 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.386152 18380 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.387094 18421 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.390091 18421 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:05.392355 18399 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:05.392927 18399 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:05.394176 18400 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:14:05.394562 18400 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:05.395550 18400 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.396910 18400 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:05.398916 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:05.400713 18498 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.398s	user 0.003s	sys 0.000s
I20260709 12:14:05.346338 18355 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:05.408059 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:05.402616 18421 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:14:05.403828 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:05.401674 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:05.405042 18399 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.365309 18276 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.420426 13260 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:05.422708 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:05.439059 12563 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:05.423966 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.397305 18273 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.409788 18211 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:05.428666 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.433346 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.459146 12492 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
W20260709 12:14:05.460827 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53037.9us
W20260709 12:14:05.466401 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 336750us
I20260709 12:14:05.470381 12493 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [15.14-17.15]
I20260709 12:14:05.477897 12491 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:05.479597 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:05.480448 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.468098 18382 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:14:05.481477 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.468458 18417 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:05.489717 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:05.488144 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.492975 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.486631 12494 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:05.488499 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.495657 12495 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
W20260709 12:14:05.496596 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111927us
I20260709 12:14:05.477943 18399 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:05.496101 12488 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:05.472654 18405 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.470273 18191 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
W20260709 12:14:05.503245 18509 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:14:05.512816 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.470453 17859 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:14:05.500701 18514 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:05.526911 18191 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:05.533900 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.534226 18191 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:05.535495 18372 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:05.535221 18191 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.536041 18372 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:05.536512 18372 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:05.536741 18191 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:05.512943 18276 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:05.511570 18405 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.539736 18405 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:05.511204 17867 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 12:14:05.541567 18398 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:05.542742 18373 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:14:05.544819 18515 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:05.545449 18515 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:05.546464 18515 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.548790 18273 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:05.500272 18172 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:05.530838 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:05.550175 12572 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:05.533412 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.550623 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:05.505831 18375 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:14:05.555799 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.518807 18503 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:14:05.557782 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
W20260709 12:14:05.519086 18499 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
W20260709 12:14:05.535063 18506 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:14:05.494693 18230 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:05.568325 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108428us
I20260709 12:14:05.551345 18247 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:05.573184 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 457894us
I20260709 12:14:05.528313 18514 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:05.580055 12491 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 12:14:05.560556 18375 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:05.589962 12490 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:14:05.565680 18273 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:14:05.592763 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80040.1us
W20260709 12:14:05.593168 18518 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:14:05.593582 18382 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:05.588500 18172 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:05.659111 18172 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.660777 18436 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:05.598160 18402 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.662142 18402 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.663719 18495 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:05.664202 18495 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:05.598965 18517 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:14:05.599300 18422 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.667604 18422 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.667903 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:05.667963 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77456.9us
I20260709 12:14:05.603721 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 943) took 1214 ms. Trace:
I20260709 12:14:05.623210 17950 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:05.672425 12566 rpcz_store.cc:276] 0709 12:14:04.389062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:04.398500 (+  9438us) service_pool.cc:224] Handling call
0709 12:14:04.398846 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:05.353416 (+954570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:05.603383 (+249967us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54002cfbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:05.603631 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066684160}
I20260709 12:14:05.581560 18355 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:05.625319 18500 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.669s	user 0.000s	sys 0.003s
W20260709 12:14:05.628284 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.630097 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.623315 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.580694 12491 pending_rounds.cc:85] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 8
I20260709 12:14:05.707706 12491 pending_rounds.cc:107] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:14:05.708051 12491 raft_consensus.cc:2889] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:05.572679 18211 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:05.648841 18506 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:14:05.672087 12563 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:05.665293 18402 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:05.575874 18373 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:05.691413 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 204745us
I20260709 12:14:05.583593 18514 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:05.588721 18380 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:14:05.710578 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87391.6us
I20260709 12:14:05.727667 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:14:05.720670 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.688792 18238 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.609s	user 0.007s	sys 0.005s
W20260709 12:14:05.646353 12389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 125) took 22872 ms (client timeout 15000 ms). Trace:
I20260709 12:14:05.735730 12490 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:14:05.677793 18375 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:05.739727 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.688386 18382 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:05.746296 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:05.747107 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75674.4us
I20260709 12:14:05.747542 18422 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:05.745543 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:05.748457 12493 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:05.723668 18399 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:05.724968 18237 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:05.749607 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:05.749640 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 242247us
I20260709 12:14:05.755024 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:05.665213 18495 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:05.758795 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.760643 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:05.766004 18247 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:05.766194 18516 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:05.710501 18400 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:14:05.726624 18380 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:05.771106 18456 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:05.771513 18512 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.772724 18372 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:05.804814 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135082us
I20260709 12:14:05.775408 18531 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:05.776258 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:05.782258 18382 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.711555 17867 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
W20260709 12:14:05.768769 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 181296us
I20260709 12:14:05.710906 18355 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:05.791615 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.722576 18211 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.718541 17950 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.807700 12495 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:14:05.807830 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:05.811625 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.821862 12645 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:14:05.822350 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46220.6us
I20260709 12:14:05.809059 18512 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.824249 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:05.822398 12637 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:14:05.826102 12494 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 12:14:05.827275 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:05.811659 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:05.787161 18244 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:14:05.813868 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:05.770102 18422 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:05.786572 18273 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:05.733549 18373 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.743508 18238 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.786092 18237 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:05.828483 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.828660 12640 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
W20260709 12:14:05.829092 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.830039 12569 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:05.818125 18274 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:14:05.825469 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.798449 18399 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:05.840561 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:05.815618 18456 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:05.844291 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:05.845966 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.851389 12563 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:05.846896 12497 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:05.830513 12490 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:14:05.848011 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:05.853036 12569 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:05.833386 17859 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:05.856293 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:05.852643 12490 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 8
I20260709 12:14:05.857585 12490 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:14:05.857913 12490 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:05.857216 12492 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:05.863112 18399 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.866003 18414 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:05.841565 18400 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:05.849063 18273 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:05.879645 18523 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:14:05.856809 17859 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:05.842839 18422 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:05.894948 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:05.903074 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.908852 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:05.998492 12568 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:14:06.003330 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.004047 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:05.897971 17859 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:05.890038 18536 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:05.900756 18402 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:06.026458 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131626us
I20260709 12:14:06.036409 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:06.028004 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157233us
I20260709 12:14:06.000438 18274 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:05.878026 18400 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.004298 18237 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:05.904968 18421 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.134s
I20260709 12:14:06.045996 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:06.046661 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:05.860430 18380 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:05.906147 18497 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 5, Time since last communication: 0.133s
I20260709 12:14:06.049216 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.038370 18402 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.045506 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:06.059249 12645 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:06.060878 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:06.002866 18414 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:14:06.064653 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.069756 13261 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:06.047448 18274 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.073176 18274 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:06.067556 18534 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:06.078524 18534 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.078626 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:06.079015 18534 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:06.086998 12572 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.519s	user 0.012s	sys 0.000s
I20260709 12:14:06.088363 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.089586 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:06.090065 12492 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:06.050779 18289 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:06.050540 18525 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.094461 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:06.099309 12493 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:06.070308 18402 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:06.102195 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 203380us
I20260709 12:14:06.093444 18536 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:06.025943 18273 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.118479 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.116168 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:06.105484 18521 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
W20260709 12:14:06.143899 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54553.2us
I20260709 12:14:06.071121 18348 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:14:06.137764 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:06.196942 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59368us
I20260709 12:14:06.143590 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:06.158649 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.143304 18516 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:06.160797 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.198756 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:06.161785 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.200435 12490 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:14:06.166096 18289 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:06.167496 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.106065 18414 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.177505 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.111008 18525 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.177457 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.184404 18230 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:06.177976 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:06.194922 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:06.196549 12492 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
W20260709 12:14:06.202018 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87215.3us
W20260709 12:14:06.217753 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 309010us
I20260709 12:14:06.064901 18456 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.218632 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.171679 18400 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.186447 18238 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.188302 18355 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.185475 18512 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.223811 12490 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:14:06.221704 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:06.209311 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 354948us
I20260709 12:14:06.228302 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:06.205824 17950 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.092490 18380 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:06.226058 18536 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.201588 18237 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:14:06.204406 18497 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:14:06.223431 18516 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:06.221201 18544 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:06.255692 12496 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 7. Preceding OpId from leader: term: 18 index: 8. (index mismatch)
I20260709 12:14:06.256850 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:06.259403 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.260373 12490 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:06.212378 18414 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.262811 12497 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
W20260709 12:14:06.286906 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 208374us
I20260709 12:14:06.291482 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:14:06.293041 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:06.294361 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99528.7us
I20260709 12:14:06.246666 18422 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.298843 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.300652 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 12.
I20260709 12:14:06.261010 18544 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:06.307107 18540 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:06.310667 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:06.305030 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:06.256074 18355 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:14:06.312526 18540 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.247253 18536 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.316417 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:06.320351 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.307642 18238 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:06.338812 18238 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:06.339318 18238 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:06.342367 18533 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:06.343137 18533 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:06.299716 17950 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.022s
W20260709 12:14:05.741128 12389 rpcz_store.cc:269] 0709 12:13:42.744792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:42.745146 (+   354us) service_pool.cc:224] Handling call
0709 12:14:05.617282 (+22872136us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:13:57.639961 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:13:57.799688 (+159727us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:13:57.803669 (+  3981us) write_op.cc:435] Acquired schema lock
0709 12:13:57.803693 (+    24us) tablet.cc:662] Decoding operations
0709 12:13:57.804829 (+  1136us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:13:57.804954 (+   125us) write_op.cc:641] Partition lock acquired for write op
0709 12:13:57.804981 (+    27us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:13:57.805991 (+  1010us) tablet.cc:701] Row locks acquired
0709 12:13:57.806047 (+    56us) write_op.cc:260] PREPARE: finished
0709 12:13:57.806261 (+   214us) write_op.cc:270] Start()
0709 12:13:57.806458 (+   197us) write_op.cc:276] Timestamp: P: 1783599237806171 usec, L: 0
0709 12:13:57.806566 (+   108us) op_driver.cc:348] REPLICATION: starting
0709 12:13:57.919944 (+113378us) log.cc:844] Serialized 4013 byte log entry
0709 12:13:59.699559 (+1779615us) op_driver.cc:464] REPLICATION: finished
0709 12:13:59.847464 (+147905us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eee28 00007ffbf9229581 00007ffbf9228d58 00007ffbf922d9f1 00007ffbf922d9a9 00007ffbf922d939 00007ffbf922d901 00007ffbf922cb2d 0000000000388634 0000000000388479
0709 12:14:00.300890 (+453426us) write_op.cc:301] APPLY: starting
0709 12:14:00.323066 (+ 22176us) tablet.cc:1367] starting BulkCheckPresence
0709 12:14:01.086158 (+763092us) tablet.cc:1370] finished BulkCheckPresence
0709 12:14:01.086222 (+    64us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:14:01.648165 (+561943us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:14:01.648534 (+   369us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 12:14:01.648595 (+    61us) write_op.cc:312] APPLY: finished
0709 12:14:01.649063 (+   468us) log.cc:844] Serialized 97 byte log entry
0709 12:14:01.746702 (+ 97639us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:01.758146 (+ 11444us) write_op.cc:454] Released schema lock
0709 12:14:01.791933 (+ 33787us) write_op.cc:341] FINISH: Updating metrics
0709 12:14:01.836862 (+ 44929us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:01.836901 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2375762,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":413398,"apply.run_cpu_time_us":42919,"apply.run_wall_time_us":1649998,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45887,"num_ops":10,"prepare.queue_time_us":200825,"prepare.run_cpu_time_us":68705,"prepare.run_wall_time_us":2247337,"raft.queue_time_us":504971,"raft.run_cpu_time_us":41500,"raft.run_wall_time_us":264573,"replication_time_us":1892772,"spinlock_wait_cycles":42139008,"thread_start_us":1238847,"threads_started":5,"wal-append.queue_time_us":949531,"wal-append.run_cpu_time_us":14366,"wal-append.run_wall_time_us":2779508}]]}
I20260709 12:14:06.326256 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.348088 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.349860 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.325153 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.342943 18559 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:14:06.359596 18546 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:14:06.303967 18558 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:06.346343 17950 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:06.326833 18372 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.336236 18515 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:14:06.378605 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:06.383460 18524 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.601s	user 0.008s	sys 0.004s
I20260709 12:14:06.310994 18544 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:06.360631 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:06.392963 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68215.9us
I20260709 12:14:06.322623 18540 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:06.399806 18556 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.400660 18556 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.404033 18556 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:06.396478 17950 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:06.406941 17950 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.414301 17950 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:06.414824 18414 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.361788 12496 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:06.423426 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.424806 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:06.370293 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59709.5us
W20260709 12:14:06.399140 18514 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.652s	user 0.008s	sys 0.004s
I20260709 12:14:06.426013 18514 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.313695 18348 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:14:06.428874 18514 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.378464 18525 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.375981 13261 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:06.425472 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
I20260709 12:14:06.440260 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.441370 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.447983 18495 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:06.449337 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:06.442480 18402 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.451368 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.451444 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.451933 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.452101 12569 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:06.452978 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.453364 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.442477 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.453737 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.453933 18526 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:06.454125 12563 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:06.454856 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.352952 18274 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.395496 13256 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:06.344579 18533 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.399489 18558 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:06.375442 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.484371 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:06.499923 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:06.500833 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.501394 13256 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:14:06.432652 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:06.396754 18515 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:06.434681 12488 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15.
I20260709 12:14:06.435392 18531 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:06.467723 18247 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.453395 13253 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:06.456029 18526 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:06.455978 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.517011 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.519227 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.522644 12568 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:06.458211 18375 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.001s
I20260709 12:14:06.397264 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:06.322785 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:06.476679 18276 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:06.553742 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.527405 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:06.526719 12496 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:14:06.464171 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:06.434731 18528 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.660s	user 0.010s	sys 0.001s
I20260709 12:14:06.466080 18422 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.498477 18238 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.465128 18540 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.476583 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.431213 18211 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.504592 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:06.594105 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89553.1us
I20260709 12:14:06.595273 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:06.505730 18414 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:06.505993 18398 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:14:06.506145 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73579us
I20260709 12:14:06.421419 18534 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.624482 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.438337 12489 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:14:06.511303 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:06.511988 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.517366 12497 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:06.522189 13259 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:06.567695 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 243551us
I20260709 12:14:06.569752 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:06.569679 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.503111 18549 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.503471 18558 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:06.517542 18348 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:14:06.573216 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 250827us
I20260709 12:14:06.574539 18402 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.580497 12569 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:06.509428 18547 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:06.399065 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:06.508014 18531 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:06.516584 18274 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.636264 18514 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:06.639405 18382 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:06.517189 18289 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:14:06.569567 18556 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:06.517252 18230 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 12:14:06.508764 18525 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:14:06.567289 18276 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.517100 18559 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:14:06.634954 12566 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:06.560040 18247 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:06.707105 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 308157us
I20260709 12:14:06.507121 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:06.640749 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.659395 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:06.646188 13258 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:14:06.663121 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
W20260709 12:14:06.696597 18563 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:14:06.582536 18560 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:06.675089 12567 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:06.622931 17867 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:06.679471 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.679165 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.695587 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.782436 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 12.
W20260709 12:14:06.473732 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.699712 12490 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:06.677160 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.591128 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:06.660920 18424 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.546703 18526 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.699033 18348 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:06.710078 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:06.641479 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.790127 12564 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:14:06.666508 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155297us
W20260709 12:14:06.719484 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 212511us
W20260709 12:14:06.720431 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61129.4us
I20260709 12:14:06.604271 18422 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.779112 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.793251 12568 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:06.646821 18550 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:06.733109 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:06.702492 17859 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.651s	user 0.005s	sys 0.009s
I20260709 12:14:06.706228 18556 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.576316 18515 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:06.801347 18568 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20260709 12:14:06.801604 18515 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:06.802497 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.803784 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.804966 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.805097 18276 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.670285 18414 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:06.675918 18522 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:06.582186 18375 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.676155 18549 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.787849 18398 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:14:06.798203 18550 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.708664 18547 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.799496 17859 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.800416 18556 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:06.785545 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
W20260709 12:14:06.847517 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.825502 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:06.814124 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.805581 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
W20260709 12:14:06.836297 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:06.829686 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.807663 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:06.875244 13261 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:06.831260 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.805979 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.816334 12564 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:14:06.835206 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.827689 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.904227 12646 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:14:06.828408 12642 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:06.829056 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.843209 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.877074 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.906484 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:06.907115 12494 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
W20260709 12:14:06.869526 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:06.811362 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101361us
I20260709 12:14:06.812963 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.902118 12495 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:06.782445 17950 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:14:06.799561 18567 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:14:06.782025 18274 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:06.927639 12567 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:14:06.814589 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 755) took 1022 ms. Trace:
I20260709 12:14:06.877072 12637 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
W20260709 12:14:06.917303 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2356) took 1394 ms (client timeout 1562 ms). Trace:
I20260709 12:14:06.917757 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2362) took 1080 ms. Trace:
W20260709 12:14:06.918282 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 110684us
I20260709 12:14:06.930688 12489 rpcz_store.cc:276] 0709 12:14:05.817519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:05.817708 (+   189us) service_pool.cc:224] Handling call
0709 12:14:06.879551 (+1061843us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:06.898204 (+ 18653us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359985024}
I20260709 12:14:06.931483 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.910550 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:06.932303 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:06.932890 12644 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:06.929360 13253 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:06.934507 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.833441 18238 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:06.925860 12491 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:06.920910 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:06.813144 18355 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:06.903108 18560 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.936388 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:06.808923 18422 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.929047 12641 rpcz_store.cc:276] 0709 12:14:05.792478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:05.792713 (+   235us) service_pool.cc:224] Handling call
0709 12:14:06.064103 (+271390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:06.191833 (+127730us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:06.191892 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:06.203158 (+ 11266us) log.cc:844] Serialized 26 byte log entry
0709 12:14:06.312035 (+108877us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:06.312087 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:06.312156 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:06.814283 (+502127us) raft_consensus.cc:1685] finished
0709 12:14:06.814314 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:06.814514 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":81448,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":8875,"thread_start_us":148344,"threads_started":2,"wal-append.queue_time_us":303911}
I20260709 12:14:06.832286 18534 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.875449 18522 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.940382 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:06.940568 12643 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:14:06.937258 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:06.954421 12646 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:06.812008 18247 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:06.899435 18551 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.944746 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:06.929811 12488 rpcz_store.cc:269] 0709 12:14:05.503904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:05.504071 (+   167us) service_pool.cc:224] Handling call
0709 12:14:05.667733 (+163662us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:14:06.879350 (+1211617us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:06.898096 (+ 18746us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1776076544}
I20260709 12:14:06.958765 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:06.948303 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.959977 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:06.950846 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.945112 12645 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [15.12-16.13]
I20260709 12:14:06.960621 12492 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:06.956370 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:06.957011 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:06.961448 12644 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:14:06.961593 12497 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:06.917870 18398 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:06.959384 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:06.964617 12567 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:06.929764 18230 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:06.927909 18274 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:06.990242 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:06.946862 18544 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.541s	user 0.013s	sys 0.000s
I20260709 12:14:06.936949 18540 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:06.985200 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:06.949378 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 157641us
I20260709 12:14:06.796099 18424 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.949882 18276 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.956151 18522 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:06.952859 18542 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:06.990044 18274 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:06.949009 18560 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:06.991135 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:06.957477 18380 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:06.838429 18348 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:06.990468 18247 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:07.003756 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53279.8us
I20260709 12:14:06.957652 18515 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:06.809476 18531 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:06.956961 18355 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:07.014405 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.027252 12494 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:07.015169 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:07.003710 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:07.010809 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.001194 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:06.988106 17950 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:07.025839 12646 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:07.030157 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.003319 18544 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.032297 12640 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:06.983106 18551 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.036939 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:07.037106 12567 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:07.052372 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:07.034868 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:07.055680 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:07.034332 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.063117 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:07.089093 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:07.092072 12567 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:14:07.093703 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:07.097347 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63096us
I20260709 12:14:07.098681 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.094726 12489 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:14:07.035136 18542 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:07.017789 18276 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.044463 18405 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.069026 18543 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:07.069862 18516 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:07.066025 18575 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:14:07.104028 18495 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.655s	user 0.009s	sys 0.004s
I20260709 12:14:07.043423 18238 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:07.069733 18541 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:07.091580 18531 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:07.095278 18422 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:07.073814 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.026247 18380 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.134021 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.017422 18414 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.030385 18247 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.147006 18550 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.131186 18402 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:14:07.150418 18570 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:14:07.153756 18398 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.158011 18273 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.161839 18375 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.162653 18230 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:07.162868 18554 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:07.163843 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:07.164716 18544 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.230286 18575 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:07.165726 18542 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.167364 18549 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:07.167512 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:07.168956 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:07.169189 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2361) took 1370 ms. Trace:
I20260709 12:14:07.237011 18422 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:07.171535 18276 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:07.172864 12568 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 12:14:07.267457 12568 pending_rounds.cc:85] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:14:07.267841 12568 pending_rounds.cc:107] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:14:07.174666 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:14:07.268093 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:07.175971 18244 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:07.182996 18551 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.183534 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.183763 18424 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:07.185422 18355 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.186595 12493 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 12:14:07.189764 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 205010us
I20260709 12:14:07.197957 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:07.182304 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.211000 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:07.212244 18543 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:07.222339 12495 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:07.147017 18516 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:14:07.284577 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86759.9us
I20260709 12:14:07.283253 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:07.240878 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 185308us
I20260709 12:14:07.246467 12646 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:14:07.165989 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:07.247998 12565 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
W20260709 12:14:07.249331 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 150789us
I20260709 12:14:07.251104 12644 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:07.236130 18402 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:07.286325 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.291518 12566 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:07.288698 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:07.289495 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:07.257831 12491 rpcz_store.cc:276] 0709 12:14:05.798623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:05.798865 (+   242us) service_pool.cc:224] Handling call
0709 12:14:05.815479 (+ 16614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:06.942370 (+1126891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.168871 (+226501us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:07.169085 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2553126784}
I20260709 12:14:07.090194 18515 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.257444 18554 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:07.259614 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:07.146291 18405 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.261816 12494 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:07.256618 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:07.263741 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:07.265074 12643 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:07.268102 12568 raft_consensus.cc:2889] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:07.361052 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2384) took 1034 ms. Trace:
I20260709 12:14:07.227331 18238 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.272432 12488 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:07.278682 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:07.363616 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:07.364485 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:07.254851 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:07.299976 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.300968 12644 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:07.235762 17950 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:07.358458 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69038.9us
W20260709 12:14:07.359014 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 99542.1us
I20260709 12:14:07.271461 18244 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:07.370384 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77251.5us
I20260709 12:14:07.377758 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.379182 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:07.246981 18273 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.373196 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:07.380952 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:07.361506 12494 rpcz_store.cc:276] 0709 12:14:06.326844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.245450 (+918606us) service_pool.cc:224] Handling call
0709 12:14:07.261596 (+ 16146us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.360764 (+ 99168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.360990 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:07.299494 18558 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.601s	user 0.012s	sys 0.000s
I20260709 12:14:07.380986 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.285226 18516 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:07.384933 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85047.6us
I20260709 12:14:07.383630 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 768) took 1055 ms. Trace:
I20260709 12:14:07.289969 18424 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:07.396014 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 771) took 1067 ms. Trace:
I20260709 12:14:07.297654 18402 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:07.390825 12491 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:07.401921 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:07.363144 18526 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:07.402434 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 769) took 1074 ms. Trace:
I20260709 12:14:07.398669 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:07.365139 18276 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:07.372318 18375 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:07.360729 18551 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:07.394340 12571 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:07.392256 12495 rpcz_store.cc:276] 0709 12:14:06.327925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.289591 (+961666us) service_pool.cc:224] Handling call
0709 12:14:07.370097 (+ 80506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.383137 (+ 13040us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:07.383183 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:07.383254 (+    71us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:07.383306 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.383392 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:07.383538 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:07.415706 12571 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:07.403234 12488 rpcz_store.cc:276] 0709 12:14:06.328770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.395223 (+1066453us) service_pool.cc:224] Handling call
0709 12:14:07.395541 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.395706 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:07.395731 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:07.395785 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:07.395818 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.395868 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:07.395963 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:07.405272 12491 rpcz_store.cc:276] 0709 12:14:06.328182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.389864 (+1061682us) service_pool.cc:224] Handling call
0709 12:14:07.390301 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.402147 (+ 11846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.402373 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:07.382872 18372 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:07.384442 18583 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.394017 18516 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:07.388825 18585 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.371843 18348 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:14:07.385906 18589 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.400532 18495 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:07.395978 18552 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:07.399930 18514 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.419492 12571 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.408293 18354 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:07.428639 18424 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:14:07.405231 18526 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:07.435815 12495 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 15. Ops: []
I20260709 12:14:07.385742 18549 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:14:07.406905 18399 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:07.444798 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:07.452646 12571 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:07.451056 18405 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:07.508543 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 772) took 1090 ms. Trace:
I20260709 12:14:07.529256 12495 rpcz_store.cc:276] 0709 12:14:06.418609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.422383 (+1003774us) service_pool.cc:224] Handling call
0709 12:14:07.435602 (+ 13219us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.507628 (+ 72026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.508461 (+   833us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:07.394075 18558 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:07.503980 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104813us
I20260709 12:14:07.538432 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:07.469928 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:07.442369 18552 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:07.554914 12565 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:07.431783 18583 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.645210 12565 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 15. Ops: []
I20260709 12:14:07.557646 12495 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:14:07.671679 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 777) took 1251 ms. Trace:
I20260709 12:14:07.672132 12495 rpcz_store.cc:276] 0709 12:14:06.420115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.531339 (+1111224us) service_pool.cc:224] Handling call
0709 12:14:07.557259 (+ 25920us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.671341 (+114082us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.671610 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:07.455875 18354 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:07.464474 18276 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:07.508107 18536 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.464721 18551 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:07.529117 18399 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:07.540467 18380 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.454327 18373 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:07.645998 18574 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.361s	user 0.001s	sys 0.000s
I20260709 12:14:07.451455 18589 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.439153 18422 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.685606 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2956) took 1249 ms. Trace:
I20260709 12:14:07.538764 18526 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:07.464740 18514 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.556385 18552 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:07.691045 18580 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.290s	user 0.000s	sys 0.000s
W20260709 12:14:07.697395 18573 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.413s	user 0.001s	sys 0.000s
W20260709 12:14:07.691933 18584 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.249s	user 0.000s	sys 0.000s
W20260709 12:14:07.693776 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 238519us
I20260709 12:14:07.557060 18247 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:07.700309 12495 rpcz_store.cc:276] 0709 12:14:06.436171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.682956 (+1246785us) service_pool.cc:224] Handling call
0709 12:14:07.683282 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.683736 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:07.683832 (+    96us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:07.683873 (+    41us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:07.683910 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.685442 (+  1532us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:07.685552 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:07.664659 18375 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:07.701926 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:07.707185 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:07.682765 18354 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:07.435681 18575 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.602734 18544 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:07.540256 18495 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.156s
I20260709 12:14:07.578640 18244 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.691840 18583 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.688791 18497 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.694700 18399 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:07.716368 18515 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:07.957723 18541 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:07.463313 18554 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.981206 18541 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.981771 18127 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:07.982324 18127 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:07.983249 18127 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.984611 18127 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:07.783245 18514 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:07.746086 12491 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:07.991082 18589 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:07.772584 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 380698us
I20260709 12:14:07.991842 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 776) took 1572 ms. Trace:
W20260709 12:14:07.772002 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 373388us
I20260709 12:14:07.992336 12491 rpcz_store.cc:276] 0709 12:14:06.419853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.417643 (+997790us) service_pool.cc:224] Handling call
0709 12:14:07.420554 (+  2911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.990065 (+569511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.991579 (+  1514us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:07.991775 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748008192}
I20260709 12:14:07.453248 18585 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.994287 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 769) took 1012 ms. Trace:
I20260709 12:14:07.994700 12643 rpcz_store.cc:276] 0709 12:14:06.982442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.035613 (+ 53171us) service_pool.cc:224] Handling call
0709 12:14:07.994052 (+958439us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:07.994188 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461993216}
I20260709 12:14:07.995807 18585 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:07.996054 18514 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:14:07.996886 12488 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:14:07.727885 18556 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.809s	user 0.011s	sys 0.001s
I20260709 12:14:07.997985 18556 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:07.998657 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 774) took 1579 ms. Trace:
W20260709 12:14:07.998894 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:07.999078 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.999121 12488 rpcz_store.cc:276] 0709 12:14:06.419267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.420300 (+1001033us) service_pool.cc:224] Handling call
0709 12:14:07.420630 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.997361 (+576731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.998364 (+  1003us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:07.998583 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1324263808}
W20260709 12:14:07.793551 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2373) took 1632 ms (client timeout 1696 ms). Trace:
I20260709 12:14:08.000464 18556 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:08.000636 12493 rpcz_store.cc:269] 0709 12:14:06.192487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.204985 (+ 12498us) service_pool.cc:224] Handling call
0709 12:14:07.793348 (+1588363us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:07.793487 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2218467584}
I20260709 12:14:08.001436 12638 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:08.002208 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:08.002835 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 762) took 1125 ms. Trace:
I20260709 12:14:08.002956 12493 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:07.710613 18276 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:07.783964 18375 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.003480 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.003284 12638 rpcz_store.cc:276] 0709 12:14:06.877844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.906964 (+ 29120us) service_pool.cc:224] Handling call
0709 12:14:08.002619 (+1095655us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:08.002772 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2428372352}
I20260709 12:14:08.005465 18276 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:07.687218 18551 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.007198 18551 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:07.806452 18573 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:14:07.788575 18273 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:08.011072 18547 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:08.011453 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2778) took 1789 ms. Trace:
I20260709 12:14:07.788316 18238 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:07.837765 12571 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:14:07.720101 18591 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:14:08.014317 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2809) took 1079 ms. Trace:
I20260709 12:14:08.014599 18558 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:08.014742 12571 rpcz_store.cc:276] 0709 12:14:06.935366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.299375 (+364009us) service_pool.cc:224] Handling call
0709 12:14:07.383469 (+ 84094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.013928 (+630459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.014234 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:08.015575 18589 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:08.016426 18589 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:08.000487 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:07.443312 18543 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.020072 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:07.744863 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:07.943372 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:08.028299 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85089.2us
I20260709 12:14:08.028858 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:07.951450 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:08.029735 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:07.956806 12563 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:07.958755 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:07.961822 18398 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:07.968863 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2378) took 1650 ms (client timeout 1824 ms). Trace:
I20260709 12:14:07.979552 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2957) took 1066 ms. Trace:
W20260709 12:14:07.741855 18590 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.075s	user 0.000s	sys 0.003s
I20260709 12:14:08.000138 12491 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:07.832464 12641 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:14:08.020635 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:07.951864 18382 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:07.999677 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:08.011957 12572 rpcz_store.cc:276] 0709 12:14:06.222422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.222631 (+   209us) service_pool.cc:224] Handling call
0709 12:14:06.223039 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:06.260380 (+ 37341us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:06.467651 (+207271us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b18002b2280. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:14:06.777877 (+310226us) log.cc:844] Serialized 14 byte log entry
0709 12:14:07.005347 (+227470us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:07.023187 (+ 17840us) log.cc:844] Serialized 26 byte log entry
0709 12:14:07.023905 (+   718us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:07.023944 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.024029 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:08.011070 (+987041us) raft_consensus.cc:1685] finished
0709 12:14:08.011188 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:08.011377 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48517,"prepare.queue_time_us":44698,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":4165,"raft.queue_time_us":29742,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":16321,"spinlock_wait_cycles":2377984,"thread_start_us":446371,"threads_started":2,"wal-append.queue_time_us":438892}
W20260709 12:14:08.044638 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85967.2us
W20260709 12:14:07.952095 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128074us
W20260709 12:14:07.973934 18531 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.747s	user 0.009s	sys 0.005s
I20260709 12:14:08.032500 12495 rpcz_store.cc:276] 0709 12:14:06.912961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.756780 (+843819us) service_pool.cc:224] Handling call
0709 12:14:07.757254 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.757551 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:07.786865 (+ 29314us) log.cc:844] Serialized 14 byte log entry
0709 12:14:07.978989 (+192124us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:07.979042 (+    53us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:07.979088 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:07.979343 (+   255us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:07.979486 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32072,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":5672,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":128,"thread_start_us":186388,"threads_started":1}
I20260709 12:14:08.038661 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:08.045998 13254 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:08.046346 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:08.032338 12490 rpcz_store.cc:269] 0709 12:14:06.318111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.318326 (+   215us) service_pool.cc:224] Handling call
0709 12:14:07.968620 (+1650294us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:07.968791 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":801943296}
I20260709 12:14:08.046725 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:08.047505 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:08.047698 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:08.048228 12490 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:08.049057 12495 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:14:08.050275 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 305515us
I20260709 12:14:08.047727 12491 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:08.053786 18582 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:08.055347 18582 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:08.055838 18582 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:08.057350 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:08.059125 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:08.059757 13257 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:08.059922 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:08.060482 13258 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:08.040814 12493 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:08.050275 13261 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:08.063334 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2958) took 1116 ms. Trace:
I20260709 12:14:07.970033 18598 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:08.024632 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:08.054361 13260 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:14:08.042273 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:08.064536 12495 rpcz_store.cc:276] 0709 12:14:06.940378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.048440 (+1108062us) service_pool.cc:224] Handling call
0709 12:14:08.048723 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.049414 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.049605 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:08.098588 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58566.5us
I20260709 12:14:08.017894 18244 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:08.113379 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.114281 12569 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 7. (term mismatch)
I20260709 12:14:08.114853 12569 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:14:08.115134 12569 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 12:14:08.115401 12569 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:08.074746 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45141.4us
I20260709 12:14:08.043651 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 770) took 1049 ms. Trace:
I20260709 12:14:08.119697 12641 rpcz_store.cc:276] 0709 12:14:06.994556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.045645 (+ 51089us) service_pool.cc:224] Handling call
0709 12:14:07.831702 (+786057us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:07.831998 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.039585 (+207587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.043574 (+  3989us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21739264}
W20260709 12:14:08.104244 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:08.121538 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:08.073889 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53910.3us
W20260709 12:14:08.108724 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:08.110030 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:08.111104 12490 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:08.122602 12495 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:08.120361 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:07.983876 18541 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.124536 12495 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:14:08.128594 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.043370 18422 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:08.043179 18585 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:08.098740 17867 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:08.126699 18586 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:08.095131 18382 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.135447 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:08.136127 18531 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.158805 18531 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.139217 18542 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:14:08.161381 18531 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.126874 18534 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:14:07.976361 18600 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.169s
I20260709 12:14:08.096577 18536 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:14:08.167594 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 771) took 1142 ms. Trace:
I20260709 12:14:08.168126 12642 rpcz_store.cc:276] 0709 12:14:07.025307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.384602 (+359295us) service_pool.cc:224] Handling call
0709 12:14:07.410665 (+ 26063us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:07.415721 (+  5056us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:07.415755 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:07.415816 (+    61us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:07.415848 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.167281 (+751433us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:08.167347 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:08.167503 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11097984}
I20260709 12:14:08.141489 18422 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.170308 18422 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:08.171240 12496 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:14:08.126484 18598 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.134516 18230 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:08.179104 18230 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:08.142328 18522 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.118s	user 0.006s	sys 0.009s
I20260709 12:14:08.180011 18522 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.181886 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2782) took 1276 ms. Trace:
I20260709 12:14:08.182436 12564 rpcz_store.cc:276] 0709 12:14:06.905008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.920625 (+ 15617us) service_pool.cc:224] Handling call
0709 12:14:08.181574 (+1260949us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:08.181803 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1946440960}
I20260709 12:14:08.185613 17859 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.147454 18424 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:14:08.095381 18515 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.126s
I20260709 12:14:08.147634 18606 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:08.189577 18606 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.148104 18556 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.192152 18522 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:08.192721 18556 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.193786 18585 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.194270 18585 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:08.196534 13258 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:14:08.200858 18588 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.598s	user 0.000s	sys 0.002s
I20260709 12:14:08.148423 18276 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.205058 18276 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.207073 13255 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:08.148751 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.148777 18558 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.018428 18497 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.210274 18558 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.149039 18547 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:08.212958 18497 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:08.200465 18422 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.213501 18547 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.149869 18415 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:14:08.150278 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:08.217181 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66989.1us
I20260709 12:14:08.217864 18424 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:08.218328 18424 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.218502 18422 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:08.218751 18424 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:08.134610 18603 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:08.219851 18603 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.220230 18603 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:08.042995 18398 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.181898 13254 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:08.224371 18398 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:08.146898 18526 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.188453 18595 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:14:08.226344 18456 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:08.147569 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.179522 18230 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:08.228140 18526 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.228287 12572 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:08.144927 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.229804 12563 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:08.231738 18560 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.196s	user 0.011s	sys 0.002s
I20260709 12:14:08.232317 18560 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.233726 12492 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 18. Ops: []
W20260709 12:14:08.234764 18274 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.208s	user 0.014s	sys 0.000s
I20260709 12:14:08.235357 18274 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.102149 17950 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:08.237250 17950 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.242453 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:08.242599 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:08.251015 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130747us
I20260709 12:14:08.254925 13257 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:08.031965 18414 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.106s
I20260709 12:14:08.176815 12394 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:08.263651 12489 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:08.261355 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147600us
I20260709 12:14:08.265201 12490 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:14:08.273725 13253 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:08.273743 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 214967us
I20260709 12:14:08.277683 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.284193 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:08.280342 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:08.288152 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.292936 18373 raft_consensus.cc:2764] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:14:08.294782 18422 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:08.301437 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.302418 12640 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:08.248927 17950 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:08.262709 18273 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:08.300689 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.299345 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:08.328213 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 221565us
I20260709 12:14:08.328820 12637 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:08.257366 18415 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:08.298620 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:08.326679 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:08.331341 12572 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:08.346412 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47976.3us
I20260709 12:14:08.287683 17867 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:14:08.356778 12496 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:08.225152 18398 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 12:14:08.348033 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:08.349824 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.148684 18354 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.056843 18582 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.392256 18582 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:14:08.392861 18495 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.393546 18495 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:08.401718 18373 consensus_queue.cc:1175] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 15 status { last_received { term: 13 index: 12 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 15. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:14:08.249389 18541 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:14:08.370723 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:08.298820 18600 consensus_queue.cc:1243] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:14:08.326251 18552 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.582s	user 0.008s	sys 0.004s
I20260709 12:14:08.343181 18273 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:08.273572 18531 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
W20260709 12:14:08.348733 18402 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.936s	user 0.014s	sys 0.000s
I20260709 12:14:08.400516 18495 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:08.458012 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 184648us
W20260709 12:14:08.467157 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 186912us
I20260709 12:14:08.348834 12394 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:14:08.475778 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94515.5us
I20260709 12:14:08.389750 18380 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.531359 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:08.516700 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:08.314539 18422 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.145s
W20260709 12:14:08.545083 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:08.560514 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 461857us
I20260709 12:14:08.523357 18273 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:08.458171 18348 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:08.571978 12394 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:08.574698 18622 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:08.562206 18600 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:08.575750 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44468.6us
I20260709 12:14:08.576330 18619 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:08.647748 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:08.583140 18247 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.586592 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:08.658965 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72507.3us
I20260709 12:14:08.591264 18560 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:08.593600 18382 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.597659 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:08.599535 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 728) took 1238 ms. Trace:
W20260709 12:14:08.671505 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69786.4us
I20260709 12:14:08.672534 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:08.668509 13253 rpcz_store.cc:276] 0709 12:14:07.361332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.361480 (+   148us) service_pool.cc:224] Handling call
0709 12:14:07.394041 (+ 32561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.283502 (+889461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.599273 (+315771us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:08.599466 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2018879232}
I20260709 12:14:08.413959 18549 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.189s
I20260709 12:14:08.514123 18554 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:08.483680 18583 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 12:14:08.677284 12488 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:08.678270 12572 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:08.521196 18456 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.140s
I20260709 12:14:08.515043 18552 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.651526 18619 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.617019 18604 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:08.665180 18534 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:08.542825 18402 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:08.721870 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:08.658864 18516 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.220s	user 0.013s	sys 0.001s
I20260709 12:14:08.679791 18595 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:08.759377 12643 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:08.723282 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:08.717765 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:08.566592 18380 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:08.723037 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50642.8us
I20260709 12:14:08.761890 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:08.762603 12568 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:08.762629 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:08.731238 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:08.707584 18622 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:08.767088 12566 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:08.738518 12565 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:08.670812 18348 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:08.775183 18603 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.554s	user 0.012s	sys 0.000s
I20260709 12:14:08.775682 18603 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.776139 18348 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.777881 12394 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:08.780141 12394 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 13 to 14, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 14 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:14:08.783242 18399 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.748s	user 0.009s	sys 0.001s
I20260709 12:14:08.783481 18402 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.783684 18399 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.785995 12645 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:08.668901 18600 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.665050 18247 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.788929 18600 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:08.792968 18244 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.657s	user 0.008s	sys 0.000s
W20260709 12:14:08.796200 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78585.4us
I20260709 12:14:08.797914 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:08.778589 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:08.774596 18620 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.725368 18619 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:08.729683 18534 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:08.885087 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.885087 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:08.888028 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:08.861129 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63306.4us
I20260709 12:14:08.726584 18415 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:08.887934 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:08.606256 18618 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.176s
W20260709 12:14:08.888123 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:08.817005 12394 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 10 to 13, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 13 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:08.889936 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:08.775698 18585 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.581s	user 0.009s	sys 0.002s
I20260709 12:14:08.902316 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:08.766672 18274 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:08.906925 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.908990 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2650) took 1869 ms. Trace:
I20260709 12:14:08.900360 12394 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 13 to 14, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 14 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:08.909276 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:08.795791 18595 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:08.900208 12563 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.002s	sys 0.010s
I20260709 12:14:08.909510 12645 rpcz_store.cc:276] 0709 12:14:07.040039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.793134 (+753095us) service_pool.cc:224] Handling call
0709 12:14:07.828255 (+ 35121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.786470 (+958215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.908675 (+122205us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:08.908909 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2202014592}
W20260709 12:14:08.899786 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:08.917804 12563 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.003s	sys 0.010s
I20260709 12:14:08.916468 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:08.919359 13258 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:08.890656 18414 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:08.915786 12394 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:08.891531 18534 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:08.794304 18589 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.997809 18629 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:14:08.999425 18639 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:09.000530 18639 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:09.000833 18637 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:09.000861 18639 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:09.002003 18623 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.002262 18639 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.002691 18623 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.003685 18639 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:09.008004 18230 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.780s	user 0.009s	sys 0.003s
I20260709 12:14:09.008551 18230 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.864277 18620 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.010193 18534 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.903038 18544 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:08.988078 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:08.891938 18244 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:08.988384 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:08.730751 18604 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:08.763257 18516 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:08.994252 18627 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:08.995399 12570 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:14:08.996017 18354 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.012092 18620 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:08.921612 18495 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.060927 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:09.063335 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.023s	sys 0.024s
W20260709 12:14:09.064332 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.023s	sys 0.024s
W20260709 12:14:09.064801 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 803823us
W20260709 12:14:08.983179 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:08.921366 18631 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:08.927723 18414 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:09.013640 18399 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.090648 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.091490 12567 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:14:09.001549 18637 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.005303 18623 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.020157 18544 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.020687 18534 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.033080 18415 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.039218 18380 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.054983 12394 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:09.058730 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:08.910913 18585 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:09.012702 18630 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.095s	user 0.001s	sys 0.001s
I20260709 12:14:09.091511 18603 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:08.980350 18541 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:09.082329 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.099038 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.101349 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:09.107975 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 345423us
W20260709 12:14:09.082362 18398 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.011s	sys 0.052s
I20260709 12:14:09.113255 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.052973 18639 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:09.124219 12637 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.795s	user 0.012s	sys 0.000s
I20260709 12:14:09.052242 18244 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.091073 18414 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:09.085726 18495 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.132553 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.076599 12497 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:09.098390 18552 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.137310 12394 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 15. New cstate: current_term: 15 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:14:09.140414 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:09.199030 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.141355 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.200811 12568 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:09.127874 18399 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:09.143805 12639 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:09.153496 12637 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:14:09.130826 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 228562us
I20260709 12:14:09.147351 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:09.208003 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.158130 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.158782 18639 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.211874 18639 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.213806 18497 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:09.214927 18607 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.103s	user 0.013s	sys 0.009s
W20260709 12:14:09.147388 18424 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.928s	user 0.012s	sys 0.001s
I20260709 12:14:09.141543 18595 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.215608 18424 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.167498 18544 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.171674 18604 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.219919 18495 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.103091 18622 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.175305 18230 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.220225 12642 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17.
I20260709 12:14:09.102124 18560 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:14:09.223958 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:09.224296 18637 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.179085 18645 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:14:09.181831 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.183722 18516 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.184697 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.239730 12570 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:09.212764 18414 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:09.144829 18398 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.011s	sys 0.052s
I20260709 12:14:09.193552 12565 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:14:09.128438 18247 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.126215 18620 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:14:09.226694 18495 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:09.214547 18497 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.159121 18585 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.252156 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.228289 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.208984 12637 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:09.232605 12567 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:14:09.231874 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:09.173110 18598 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:09.188566 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:09.242973 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 318107us
I20260709 12:14:09.245051 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.246443 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.154160 18244 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:09.249100 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136638us
W20260709 12:14:09.227260 18551 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.219s	user 0.015s	sys 0.000s
I20260709 12:14:09.061589 18627 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.264298 12639 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:09.239933 12394 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 5 to 12, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 12 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:09.227011 18623 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:09.257871 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.267648 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.261262 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.267853 12565 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:09.267907 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:09.268560 12564 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:09.199062 18603 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 12:14:09.268934 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.268586 12567 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:09.271057 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:09.271610 12637 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
W20260709 12:14:09.275970 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129032us
I20260709 12:14:09.276605 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.275929 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:09.257258 18620 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:09.290006 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:09.265789 18273 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.700s	user 0.013s	sys 0.000s
I20260709 12:14:09.283984 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:09.286721 18424 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.291083 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:09.285945 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:09.283416 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.294123 13254 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:09.311578 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:09.307426 12394 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 15 to 16, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 16 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:09.307188 13262 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:09.274027 18230 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.284739 18598 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:09.289839 18620 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:09.283102 18516 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:09.268290 18247 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:14:09.279907 18543 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:09.350479 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:09.350222 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66383.2us
W20260709 12:14:09.362946 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77144.1us
I20260709 12:14:09.351994 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.325289 12567 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 12:14:09.364207 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.371049 12568 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:09.358884 18544 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.372079 18544 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.372387 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.364490 18653 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:09.316668 18639 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.354014 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2820) took 1009 ms. Trace:
I20260709 12:14:09.326277 12394 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 10 to 12, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 12 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:14:09.358675 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:09.402149 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43551.2us
W20260709 12:14:09.375033 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.392931 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.352290 18650 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:09.324357 18598 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:09.347823 18230 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.349047 18643 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:09.351217 18554 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:09.293891 18647 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:14:09.349649 18632 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:14:09.399876 12563 rpcz_store.cc:276] 0709 12:14:08.344417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.344612 (+   195us) service_pool.cc:224] Handling call
0709 12:14:08.345180 (+   568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:08.918790 (+573610us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:08.918836 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:08.919789 (+   953us) log.cc:844] Serialized 26 byte log entry
0709 12:14:09.196160 (+276371us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:09.196199 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.196268 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:09.353747 (+157479us) raft_consensus.cc:1685] finished
0709 12:14:09.353771 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:09.353942 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28930,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":52210,"thread_start_us":749096,"threads_started":2,"wal-append.queue_time_us":355979}
I20260709 12:14:09.381662 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:09.269299 18551 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.407222 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:09.316998 18626 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.373368 18653 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.294003 18636 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.371080 18516 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:09.315501 18273 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.426653 12646 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:09.425384 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.424360 12563 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:09.427598 12563 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:09.427858 12567 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:09.413067 18650 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.434382 12394 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 11, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 11 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:09.435443 12637 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:09.412817 18598 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:09.428503 12563 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:09.417927 12610 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.523s (SENT) [suppressed 8 similar messages]
I20260709 12:14:09.443974 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.417759 18554 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:09.412669 18643 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.458040 18643 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:09.449456 12563 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:09.466305 18414 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.466696 12394 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:09.433205 18626 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.470356 18626 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.470439 18544 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.417219 18543 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:14:09.407636 18639 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.473184 18514 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:09.455488 18644 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:14:09.456204 18655 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.490087 18655 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.491089 18628 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:09.491925 18628 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:09.492666 18655 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.470887 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 767) took 3143 ms. Trace:
I20260709 12:14:09.494333 12492 rpcz_store.cc:276] 0709 12:14:06.327658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.284157 (+956499us) service_pool.cc:224] Handling call
0709 12:14:07.361488 (+ 77331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.234158 (+872670us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.470616 (+1236458us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:09.470821 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005002112}
I20260709 12:14:09.495762 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.496379 12492 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:09.497997 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:09.499218 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:09.442405 18646 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 12:14:09.500358 12492 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:09.442790 18636 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.502879 18636 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.502959 18497 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.455241 18650 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.452943 12496 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:09.476056 18639 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.473197 18273 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.480882 18514 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:09.490296 18642 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:14:09.522073 18636 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:09.522418 18642 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:09.522858 18642 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:09.522795 18636 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.525266 18636 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.522104 18514 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.528457 18514 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:09.447587 18619 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.552s	user 0.007s	sys 0.009s
I20260709 12:14:09.531713 18619 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.534266 18619 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.534893 18244 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.536041 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2382) took 3209 ms. Trace:
I20260709 12:14:09.536517 12489 rpcz_store.cc:276] 0709 12:14:06.326245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.116618 (+790373us) service_pool.cc:224] Handling call
0709 12:14:09.535743 (+2419125us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:09.535930 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2380312448}
I20260709 12:14:09.542511 12495 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:09.542980 12495 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:09.543722 18551 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.545388 12494 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 18. Ops: []
I20260709 12:14:09.546413 18230 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.546936 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 770) took 3218 ms. Trace:
I20260709 12:14:09.547408 12494 rpcz_store.cc:276] 0709 12:14:06.328485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.392509 (+1064024us) service_pool.cc:224] Handling call
0709 12:14:09.545183 (+2152674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:09.545933 (+   750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.546740 (+   807us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b5400191750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:09.546877 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4903449984}
I20260709 12:14:09.548592 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.549194 12494 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:09.552481 18543 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.553112 18543 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.554702 18649 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:09.555141 18649 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:09.446731 18647 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:09.555642 18543 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:09.515326 17950 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.148s	user 0.015s	sys 0.000s
I20260709 12:14:09.555977 18649 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.556871 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:09.557873 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.507290 18620 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.559159 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:09.559266 18495 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:09.560050 18495 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.561460 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.561955 18380 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:09.562281 12494 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:14:09.511729 12394 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:09.436988 18516 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:09.436882 18415 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:09.426513 18653 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:09.560412 18551 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:09.577108 18551 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:09.577291 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:09.577729 18585 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:09.578152 18585 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:09.578985 18585 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.579428 12637 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:09.579295 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2966) took 2398 ms. Trace:
I20260709 12:14:09.579943 12637 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:09.498590 18598 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.580164 12488 rpcz_store.cc:276] 0709 12:14:07.181074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.135376 (+954302us) service_pool.cc:224] Handling call
0709 12:14:08.135720 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.719439 (+583719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.578911 (+859472us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:09.579149 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1244800384}
I20260709 12:14:09.580850 12637 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.581954 18636 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.582324 18598 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:09.583421 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.543699 12495 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.562904 18380 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:09.574097 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:09.575812 18415 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.576160 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:09.594538 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.583338 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2796) took 2920 ms. Trace:
I20260709 12:14:09.587122 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 772) took 1584 ms. Trace:
I20260709 12:14:09.598510 12394 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 14 to 13, leader changed from <none> to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 13 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:09.604569 12643 rpcz_store.cc:276] 0709 12:14:08.003058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.003184 (+   126us) service_pool.cc:224] Handling call
0709 12:14:09.586815 (+1583631us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:09.587027 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1735984384}
I20260709 12:14:09.605334 18380 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.592147 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 798) took 1243 ms. Trace:
I20260709 12:14:09.605888 12394 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:09.608335 18556 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.499313 18606 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.572541 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:09.493182 18628 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.575609 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:09.584951 12637 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:09.624825 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2963) took 2444 ms. Trace:
I20260709 12:14:09.625283 12490 rpcz_store.cc:276] 0709 12:14:07.180316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.123218 (+942902us) service_pool.cc:224] Handling call
0709 12:14:08.123596 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:08.265963 (+142367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.624543 (+1358580us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:09.624764 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":324741888}
I20260709 12:14:09.556284 18543 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.626434 18543 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.629118 18556 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:09.638321 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62843.2us
W20260709 12:14:09.638871 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2960) took 2459 ms (client timeout 1523 ms). Trace:
W20260709 12:14:09.639565 12497 rpcz_store.cc:269] 0709 12:14:07.179617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:07.181967 (+  2350us) service_pool.cc:224] Handling call
0709 12:14:09.638536 (+2456569us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:09.638773 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4142256384}
I20260709 12:14:09.640359 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:09.579948 18585 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:09.644573 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:09.645282 12494 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:09.645730 18375 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:09.646178 18375 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:09.646518 18375 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:09.556905 17950 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.649924 18556 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:09.652048 17950 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:09.653304 18244 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.561969 12567 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:14:09.660604 12567 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:09.584918 12495 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:09.603125 12569 rpcz_store.cc:276] 0709 12:14:06.663198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.913761 (+250563us) service_pool.cc:224] Handling call
0709 12:14:06.914314 (+   553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.133437 (+1219123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.583009 (+1449572us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:09.583264 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2758491392}
I20260709 12:14:09.608552 12394 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 9, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 9 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:09.581908 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.662871 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.581933 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.606153 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:09.519018 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:09.591215 18558 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.614523 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:09.605367 18627 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.677784 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.679085 12569 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:14:09.557296 18649 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:09.582587 18636 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:09.618974 18642 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.619997 18586 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:09.623646 18415 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.641029 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:09.641880 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69439.9us
I20260709 12:14:09.590893 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:09.626626 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:09.655732 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.721953 12644 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:09.656513 12572 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17.
I20260709 12:14:09.656587 18375 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:09.658622 18495 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:09.663859 12643 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:09.606820 12496 rpcz_store.cc:276] 0709 12:14:08.348701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.566199 (+217498us) service_pool.cc:224] Handling call
0709 12:14:09.591885 (+1025686us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:09.592072 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1803772160}
I20260709 12:14:09.680171 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:09.755750 12492 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:09.682592 12494 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:09.758312 12494 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:14:09.669131 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:09.760551 12565 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:09.692414 12637 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 9. (term mismatch)
I20260709 12:14:09.761834 12637 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 8
I20260709 12:14:09.762210 12637 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 18.9
I20260709 12:14:09.762501 12637 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:09.763969 12571 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
W20260709 12:14:09.711912 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97568.8us
I20260709 12:14:09.715668 12563 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:14:09.769862 12563 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:09.770228 12563 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:14:09.770524 12563 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:09.716574 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:09.719413 18586 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:09.749046 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:09.777554 12496 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:14:09.716797 12567 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:09.752907 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.670150 18402 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.783679 12461 leader_election.cc:400] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17.
I20260709 12:14:09.784111 12461 leader_election.cc:403] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:14:09.788115 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:09.719396 18372 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:14:09.791680 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71020.8us
I20260709 12:14:09.792068 12567 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:14:09.792264 12563 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:14:09.792618 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.794239 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:09.809381 12646 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:09.790982 18373 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:14:09.803498 18547 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.830422 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:09.833302 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:09.835139 18514 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:14:09.889287 12637 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:09.835757 12495 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:14:09.890852 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 793) took 2601 ms. Trace:
I20260709 12:14:09.839306 18372 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
W20260709 12:14:09.872802 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.845520 18276 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:09.806815 18402 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:09.898978 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 483895us
I20260709 12:14:09.899153 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.899978 12643 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:09.821012 18495 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17.
I20260709 12:14:09.856607 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:09.841125 18515 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:14:09.891391 12495 rpcz_store.cc:276] 0709 12:14:07.289311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.143850 (+854539us) service_pool.cc:224] Handling call
0709 12:14:08.144241 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:09.542401 (+1398160us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54001c41a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:14:09.890499 (+348098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.890785 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":739617152}
I20260709 12:14:09.886473 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
W20260709 12:14:09.902061 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 494914us
I20260709 12:14:09.860867 18549 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:14:09.912565 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:09.913254 12646 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:09.905066 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.915823 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.912408 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:09.918910 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:09.903901 12642 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:09.930055 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:09.896569 12567 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 12:14:09.935520 12567 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:14:09.936053 12567 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:14:09.936400 12567 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:09.954964 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:09.958515 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:09.960264 12643 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:14:09.963151 12637 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:09.965090 12637 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 12. Current term is 13. Ops: []
I20260709 12:14:09.970247 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:09.975371 12643 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:14:09.906256 18606 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:09.905810 18551 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:09.902925 18415 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:09.985400 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:09.995438 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.948830 17950 raft_consensus.cc:2764] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:14:09.967769 18244 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:09.966828 18238 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:14:10.028535 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.010093 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.009119 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.991195 12646 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
W20260709 12:14:10.045359 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 258136us
W20260709 12:14:10.046307 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:09.983038 18515 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:14:10.050370 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64164.9us
I20260709 12:14:09.992529 18355 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:10.063077 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:10.081663 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.082340 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.082970 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.083400 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.083871 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.092308 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.094890 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:10.096117 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2788) took 1365 ms (client timeout 1537 ms). Trace:
W20260709 12:14:10.097092 12572 rpcz_store.cc:269] 0709 12:14:08.730424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.730972 (+   548us) service_pool.cc:224] Handling call
0709 12:14:10.095910 (+1364938us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:10.096054 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2048971136}
I20260709 12:14:10.099882 12642 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:10.103189 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:10.109936 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:10.110994 12563 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:10.058621 18604 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.000s	sys 0.027s
I20260709 12:14:10.133555 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.159786 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:10.180023 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79608us
I20260709 12:14:10.056672 18515 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:10.187426 12536 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15.
I20260709 12:14:10.188299 12536 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:10.191526 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:10.194067 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:10.096563 18622 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.023s	sys 0.077s
I20260709 12:14:10.042814 18415 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.212908 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:10.103186 18348 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.009s	sys 0.021s
I20260709 12:14:10.006884 18551 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.001428 18606 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:10.162256 18604 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.000s	sys 0.028s
I20260709 12:14:10.161545 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.096529 18642 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:10.096055 18542 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:10.214901 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.107870 18276 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:10.238322 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:10.215519 18348 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.011s	sys 0.027s
I20260709 12:14:10.197134 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:10.201297 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 417242us
I20260709 12:14:10.201864 18636 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:10.248906 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51939.6us
I20260709 12:14:10.230095 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:10.186025 18375 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.239305 12567 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:10.201422 18244 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15.
W20260709 12:14:10.216382 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111084us
I20260709 12:14:10.258220 18636 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:14:10.259859 18636 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:14:10.242980 18522 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:10.210398 18547 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
W20260709 12:14:10.243178 18664 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.361s	user 0.001s	sys 0.000s
I20260709 12:14:10.252444 18276 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:10.206084 18622 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.023s	sys 0.081s
I20260709 12:14:10.252235 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.256492 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.278680 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2962) took 3098 ms. Trace:
I20260709 12:14:10.349253 12493 rpcz_store.cc:276] 0709 12:14:07.180103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.062436 (+882333us) service_pool.cc:224] Handling call
0709 12:14:08.062773 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:08.100500 (+ 37727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:08.100529 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:08.111952 (+ 11423us) log.cc:844] Serialized 26 byte log entry
0709 12:14:09.601968 (+1490016us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b18001fc360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:09.903883 (+301915us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:09.903917 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.904015 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:10.278312 (+374297us) raft_consensus.cc:1685] finished
0709 12:14:10.278331 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:10.278603 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":156979,"raft.queue_time_us":38075,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":21440,"spinlock_wait_cycles":12071808,"thread_start_us":301684,"threads_started":1,"wal-append.queue_time_us":2121229}
I20260709 12:14:10.266139 18402 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:10.344195 12488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.046s	sys 0.064s
I20260709 12:14:10.284237 18677 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:14:10.280046 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:10.357892 12488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.046s	sys 0.064s
W20260709 12:14:10.269245 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.362751 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82780.4us
I20260709 12:14:10.370767 12563 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:14:10.287336 18662 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
W20260709 12:14:10.267875 18554 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.760s	user 0.012s	sys 0.001s
W20260709 12:14:10.265607 18653 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.687s	user 0.009s	sys 0.000s
W20260709 12:14:10.372534 12564 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.000s
I20260709 12:14:10.274179 18522 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:10.397970 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.407295 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:14:10.256708 18515 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.244443 18542 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.256466 18456 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:10.414033 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.414959 12564 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:10.295610 18606 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:10.299271 18643 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.840s	user 0.004s	sys 0.009s
W20260709 12:14:10.289526 18663 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.442s	user 0.002s	sys 0.000s
I20260709 12:14:10.303357 18620 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:10.380703 18516 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.795s	user 0.006s	sys 0.007s
W20260709 12:14:10.449134 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0214703869999084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4622s for a response
I20260709 12:14:10.280382 18244 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.147s
W20260709 12:14:10.458053 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.459331 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:10.421093 18276 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:10.448622 18585 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.804s	user 0.016s	sys 0.000s
I20260709 12:14:10.478420 12490 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:10.452961 18456 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:10.480686 18456 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:10.485574 18586 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.669s	user 0.018s	sys 0.003s
I20260709 12:14:10.486431 18586 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.488302 13255 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:10.490206 18586 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.392962 18554 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:10.530793 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.531457 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2807) took 1312 ms. Trace:
I20260709 12:14:10.532029 12566 rpcz_store.cc:276] 0709 12:14:09.219487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.269908 (+ 50421us) service_pool.cc:224] Handling call
0709 12:14:09.270247 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:09.323554 (+ 53307us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:09.323590 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:09.360560 (+ 36970us) log.cc:844] Serialized 26 byte log entry
0709 12:14:09.781184 (+420624us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:09.781253 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:09.781364 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:10.531186 (+749822us) raft_consensus.cc:1685] finished
0709 12:14:10.531213 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:10.531387 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36842,"raft.queue_time_us":67213,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":126,"thread_start_us":409761,"threads_started":1,"wal-append.queue_time_us":534700}
I20260709 12:14:10.524431 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:10.535106 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.539142 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
W20260709 12:14:10.526832 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.528934 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.001s	sys 0.021s
W20260709 12:14:10.544210 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.002s	sys 0.022s
W20260709 12:14:10.560781 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.499094 18522 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.425096 18653 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.445973 18643 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.455598 18516 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.572181 12493 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
W20260709 12:14:10.561578 18514 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.015s	sys 0.018s
W20260709 12:14:10.549819 18373 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.026s	sys 0.032s
I20260709 12:14:10.579898 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.596556 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.580508 13262 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:10.527840 18515 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:10.573657 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.549185 18665 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:10.562031 18495 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:10.561718 18585 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:10.603631 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.010s	sys 0.023s
W20260709 12:14:10.608067 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.011s	sys 0.023s
I20260709 12:14:10.607981 12637 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:10.561497 18375 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:10.606287 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:10.598242 18514 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.016s	sys 0.018s
I20260709 12:14:10.580683 18244 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:10.604458 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.613294 12568 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:10.600934 18373 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.027s	sys 0.032s
I20260709 12:14:10.582252 18595 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:10.615756 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.614516 12489 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:10.604251 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:10.619166 12492 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18.
I20260709 12:14:10.620396 12493 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:14:10.622074 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:10.623041 12495 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:10.603341 18598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.009s	sys 0.025s
I20260709 12:14:10.625785 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:10.634258 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.635124 12496 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:10.637300 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:10.639823 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.605067 18415 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:10.631600 18598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.009s	sys 0.025s
I20260709 12:14:10.643390 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.628638 18531 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:10.614059 18495 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.644709 12496 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 12:14:10.636272 18649 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.917s	user 0.015s	sys 0.005s
I20260709 12:14:10.629459 18244 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:10.671089 18668 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:14:10.672434 18244 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:10.673000 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 731) took 1797 ms. Trace:
I20260709 12:14:10.673763 18653 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:10.674837 18551 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.673390 13253 rpcz_store.cc:276] 0709 12:14:08.875169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.875339 (+   170us) service_pool.cc:224] Handling call
0709 12:14:08.906421 (+ 31082us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:09.408293 (+501872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:09.408335 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:09.433169 (+ 24834us) log.cc:844] Serialized 52 byte log entry
0709 12:14:09.894431 (+461262us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:09.894499 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.280886 (+386387us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:10.280971 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:10.672629 (+391658us) raft_consensus.cc:1685] finished
0709 12:14:10.672655 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:10.672920 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4377,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":6207,"spinlock_wait_cycles":1130831744,"thread_start_us":460659,"threads_started":1,"wal-append.queue_time_us":947398}
I20260709 12:14:10.685688 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2811) took 1375 ms. Trace:
I20260709 12:14:10.686208 12568 rpcz_store.cc:276] 0709 12:14:09.309919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.371975 (+ 62056us) service_pool.cc:224] Handling call
0709 12:14:10.685372 (+1313397us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:10.685560 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2845080448}
I20260709 12:14:10.687639 12569 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:10.691751 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:10.687624 12497 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:10.698625 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 991) took 1289 ms. Trace:
I20260709 12:14:10.699306 12569 rpcz_store.cc:276] 0709 12:14:09.409367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.778634 (+369267us) service_pool.cc:224] Handling call
0709 12:14:10.687378 (+908744us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:10.688270 (+   892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.698288 (+ 10018us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54003ab450. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:10.698510 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2087491456}
I20260709 12:14:10.630918 18595 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.615312 18585 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.644857 18583 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:10.616073 18375 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.661060 12493 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:10.602212 18628 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s	user 0.029s	sys 0.064s
W20260709 12:14:10.644317 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.014s	sys 0.005s
W20260709 12:14:10.722482 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.014s	sys 0.005s
W20260709 12:14:10.644006 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.004s	sys 0.016s
I20260709 12:14:10.706241 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:10.712046 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:10.716172 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 470241us
W20260709 12:14:10.719292 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.014s	sys 0.018s
W20260709 12:14:10.725078 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.014s	sys 0.019s
I20260709 12:14:10.725653 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 773) took 2711 ms. Trace:
I20260709 12:14:10.726754 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:10.724048 13253 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:10.726887 12494 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:10.734202 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:10.726207 12638 rpcz_store.cc:276] 0709 12:14:08.005730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.009780 (+  4050us) service_pool.cc:224] Handling call
0709 12:14:08.010240 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:08.239004 (+228764us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:14:08.239457 (+   453us) log.cc:844] Serialized 14 byte log entry
0709 12:14:08.658005 (+418548us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:08.658064 (+    59us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:08.658112 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.717043 (+2058931us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:10.717110 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:10.717305 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5904,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":8060,"raft.queue_time_us":124047,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":139,"spinlock_wait_cycles":504019840,"thread_start_us":418110,"threads_started":1,"wal-append.queue_time_us":523683,"wal-append.run_cpu_time_us":2493,"wal-append.run_wall_time_us":1378928}
I20260709 12:14:10.713642 18583 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:10.755299 17950 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:10.755941 17950 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:10.756372 17950 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:10.755566 18583 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.752899 18554 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.758618 18516 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.759434 18355 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:10.759792 18375 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:10.760695 18585 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:10.749043 18542 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:10.757300 18583 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:10.740609 18348 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:10.769384 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2993) took 1372 ms. Trace:
I20260709 12:14:10.769974 12497 rpcz_store.cc:276] 0709 12:14:09.397468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.640609 (+243141us) service_pool.cc:224] Handling call
0709 12:14:10.769157 (+1128548us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:10.769307 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2225390464}
I20260709 12:14:10.771482 18679 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:10.772353 18679 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:10.773444 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 891) took 1131 ms. Trace:
I20260709 12:14:10.774058 13262 rpcz_store.cc:276] 0709 12:14:09.642009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.642163 (+   154us) service_pool.cc:224] Handling call
0709 12:14:09.642486 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:10.602308 (+959822us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.773146 (+170838us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b54005e1354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:10.773364 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2156507520}
W20260709 12:14:10.774977 18616 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.308s	user 0.007s	sys 0.005s
I20260709 12:14:10.776448 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2814) took 1298 ms. Trace:
W20260709 12:14:10.776857 18684 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:14:10.777114 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2984) took 1511 ms. Trace:
I20260709 12:14:10.778100 12490 rpcz_store.cc:276] 0709 12:14:09.265653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.626316 (+360663us) service_pool.cc:224] Handling call
0709 12:14:10.776873 (+1150557us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b540020e4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:10.777040 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1739583360}
I20260709 12:14:10.776965 12570 rpcz_store.cc:276] 0709 12:14:09.477942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.791679 (+313737us) service_pool.cc:224] Handling call
0709 12:14:09.838531 (+ 46852us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:09.839032 (+   501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:09.839072 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:09.839694 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:14:10.725773 (+886079us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:10.725816 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.728505 (+  2689us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:10.776146 (+ 47641us) raft_consensus.cc:1685] finished
0709 12:14:10.776170 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:10.776345 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3343,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":7332,"thread_start_us":885571,"threads_started":1,"wal-append.queue_time_us":934185}
I20260709 12:14:10.779520 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.781286 12490 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
W20260709 12:14:10.781674 18536 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:14:10.782611 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:10.771551 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:10.783619 18559 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:14:10.784384 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:10.785326 12497 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:10.786067 12571 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:14:10.759899 18355 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:10.787619 18355 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.788439 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.782400 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:10.790149 18682 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:14:10.790072 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:10.718772 18620 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:10.790812 12570 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:10.790913 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:10.721936 18628 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.081s	user 0.030s	sys 0.064s
I20260709 12:14:10.792299 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:10.794329 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.793681 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:10.794912 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.796650 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.797726 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:10.798064 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.746783 18681 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:14:10.801087 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.807094 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 2959) took 3627 ms. Trace:
I20260709 12:14:10.796612 18559 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:10.752014 18683 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:10.824733 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.827412 12568 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:10.796113 18536 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:10.810863 12491 rpcz_store.cc:276] 0709 12:14:07.179323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.054536 (+875213us) service_pool.cc:224] Handling call
0709 12:14:08.054854 (+   318us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:08.055301 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:14:08.055615 (+   314us) log.cc:844] Serialized 14 byte log entry
0709 12:14:08.311257 (+255642us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:08.345795 (+ 34538us) log.cc:844] Serialized 52 byte log entry
0709 12:14:08.345981 (+   186us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:14:08.346017 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:08.346086 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:10.774122 (+2428036us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:10.806640 (+ 32518us) raft_consensus.cc:1685] finished
0709 12:14:10.806764 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:10.807004 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44272,"prepare.queue_time_us":1007,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":157236,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":26389,"spinlock_wait_cycles":38475520,"thread_start_us":245467,"threads_started":2,"wal-append.queue_time_us":218734}
I20260709 12:14:10.801576 18627 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:10.833063 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:10.821524 18373 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 12:14:10.833285 12460 leader_election.cc:400] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17.
I20260709 12:14:10.820118 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:10.817112 18559 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:10.836163 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:10.826520 18636 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:10.837910 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:10.847270 12497 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:10.828792 18542 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:10.852929 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:10.855582 12493 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:10.842403 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:10.724704 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.005s	sys 0.016s
W20260709 12:14:10.858161 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:10.837824 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:10.863957 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:10.855412 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:10.865942 12491 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:10.872607 13253 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:10.879639 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:10.866143 12570 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:10.841225 18604 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:10.841904 18559 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:10.888648 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:10.906862 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:10.870791 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:10.897046 12493 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:10.900557 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.836136 18536 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:10.855242 18620 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:14:10.923810 12493 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:10.923643 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:10.909478 13260 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:10.924350 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
W20260709 12:14:10.903244 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 340256us
I20260709 12:14:10.983353 13253 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:14:10.971632 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
W20260709 12:14:10.884477 18631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.012s	sys 0.011s
I20260709 12:14:11.004983 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.014310 12495 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:14:10.918519 18680 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:14:10.920841 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:10.891963 18456 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:10.909897 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:11.024451 13258 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:10.920354 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:11.025555 12568 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:10.911433 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:11.008756 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85183.1us
I20260709 12:14:11.027377 13254 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:10.926390 13257 raft_consensus.cc:1217] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Deduplicated request from leader. Original: 15.12->[16.13-16.13]   Dedup: 16.13->[]
I20260709 12:14:11.018129 12495 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:14:11.028134 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.028563 13257 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:14:11.019037 12493 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.8-18.8]
I20260709 12:14:11.020336 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:10.918828 18627 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:10.906205 18683 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.031932 12495 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:10.873857 18542 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:11.013203 18631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.012s	sys 0.017s
I20260709 12:14:11.037653 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.007490 18681 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:10.981853 18495 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:11.069344 12563 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:11.083750 12563 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:14:11.017223 18636 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:11.066340 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 828) took 1004 ms. Trace:
I20260709 12:14:11.053678 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.067808 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:11.088562 12572 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:11.062881 12490 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:11.090595 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.091576 12490 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:11.054383 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:11.055488 12495 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:11.002861 18516 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.038069 12491 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:14:10.919435 18679 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.035240 18456 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:11.007833 18620 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:11.076040 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:11.107573 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.065335 18559 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.096903 18595 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:11.097465 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44008.9us
I20260709 12:14:11.051838 18627 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:11.068109 18629 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:11.100464 12570 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:11.087257 13255 rpcz_store.cc:276] 0709 12:14:10.061485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:10.061864 (+   379us) service_pool.cc:224] Handling call
0709 12:14:11.066088 (+1004224us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:11.066269 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":975338880}
I20260709 12:14:11.081308 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.108604 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.046403 18604 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:14:11.127148 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.013s	sys 0.022s
W20260709 12:14:11.128502 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.014s	sys 0.022s
W20260709 12:14:11.111254 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 377228us
I20260709 12:14:11.059321 18536 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.118045 18629 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:11.131043 12536 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18.
I20260709 12:14:11.108369 18603 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:14:11.114591 18680 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:14:11.110714 18694 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
W20260709 12:14:11.116087 18689 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:14:11.134220 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.106536 18373 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:11.108136 18456 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:11.141549 12572 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:14:11.142931 13255 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:14:11.116451 18244 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:11.145432 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:11.117586 18627 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:11.133358 18679 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.107311 18649 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:14:11.115581 18636 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:11.157097 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.145787 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.126209 18604 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:11.076918 18683 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.148551 18665 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:11.138389 18531 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:11.110545 18415 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:14:11.110713 18516 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.206250 18620 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.099263 18681 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.144348 18558 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:11.211414 18558 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:11.212044 18558 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:11.169737 12641 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 13: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:11.214154 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2655) took 2340 ms (client timeout 1618 ms). Trace:
W20260709 12:14:11.214692 12641 rpcz_store.cc:269] 0709 12:14:08.873791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:08.876515 (+  2724us) service_pool.cc:224] Handling call
0709 12:14:11.145242 (+2268727us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cee9c 00007ffbf26cea5b 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:14:11.214006 (+ 68764us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2822528}
W20260709 12:14:11.172748 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54801.9us
I20260709 12:14:11.178658 12496 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:14:11.172166 18604 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:11.164007 18603 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:11.217466 18373 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:11.217702 18603 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.218132 18373 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.219156 18355 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:11.189766 18636 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.223471 18542 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.169798 18679 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:11.226661 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.196151 18696 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:14:11.227373 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.228193 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.231053 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.235798 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90096.2us
I20260709 12:14:11.237612 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.215875 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 990) took 1806 ms. Trace:
I20260709 12:14:11.238497 12565 rpcz_store.cc:276] 0709 12:14:09.409123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.774967 (+365844us) service_pool.cc:224] Handling call
0709 12:14:09.775376 (+   409us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:09.775855 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:09.834388 (+ 58533us) log.cc:844] Serialized 14 byte log entry
0709 12:14:10.609320 (+774932us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:10.625191 (+ 15871us) log.cc:844] Serialized 52 byte log entry
0709 12:14:10.625353 (+   162us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:10.625385 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.625457 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:11.215627 (+590170us) raft_consensus.cc:1685] finished
0709 12:14:11.215649 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:11.215811 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":58366,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":448,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":21466,"thread_start_us":738301,"threads_started":1,"wal-append.queue_time_us":880791}
I20260709 12:14:11.243353 12495 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:11.233484 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.246647 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.259397 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.137406 18690 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.073s
I20260709 12:14:11.264374 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.267978 13254 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:11.264611 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.274389 12496 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:11.204104 18665 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:11.265349 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.287695 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.249094 18705 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:11.250147 18709 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:11.303121 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.276120 18244 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:14:11.301365 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 220153us
I20260709 12:14:11.292227 18665 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:11.325073 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.330166 12492 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:14:11.266291 18628 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:14:11.302161 18705 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:11.306041 18583 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:11.351217 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.352129 12496 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:14:11.289582 18635 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:14:11.305294 18244 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:11.345661 12490 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:14:11.296445 18542 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
W20260709 12:14:11.349334 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79566.3us
I20260709 12:14:11.332573 18681 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.357686 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:11.341144 18373 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.364313 12645 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:11.357686 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:11.373467 18701 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:14:11.384599 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:11.385059 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:11.364218 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.385864 12641 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:11.374697 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:11.379158 18674 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.918s	user 0.002s	sys 0.000s
W20260709 12:14:11.381122 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.387094 12495 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:11.388029 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.299787 18629 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.389425 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.391840 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:11.397684 18687 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:11.399290 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.400516 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.401602 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:11.401902 18671 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.807s	user 0.004s	sys 0.000s
I20260709 12:14:11.402355 12492 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:11.402990 18629 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:11.403209 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.403950 12492 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:11.350740 18705 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:11.362017 18244 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:11.398046 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:11.392378 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:11.423463 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:11.407431 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.430545 18679 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.422458 18541 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
W20260709 12:14:11.434046 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.434893 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.411321 18715 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:14:11.435369 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.410916 18619 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.450798 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:11.451138 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.410959 18722 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:11.412593 18707 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:14:11.414764 18718 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:14:11.417762 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.460934 18722 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:11.461328 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 13.
I20260709 12:14:11.461989 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:11.462723 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:11.462260 18722 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.462937 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.463611 12491 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:11.463508 18679 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:11.436575 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.466085 18348 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:11.466571 18348 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:11.466485 18688 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:11.467638 18348 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.468993 18348 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:11.449506 18628 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:14:11.469820 13257 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:11.471624 18678 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.819s	user 0.002s	sys 0.000s
I20260709 12:14:11.473788 18415 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:11.474771 18415 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.475360 18627 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.476487 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 799) took 3127 ms. Trace:
W20260709 12:14:11.478193 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 2981) took 2213 ms (client timeout 2131 ms). Trace:
W20260709 12:14:11.440148 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.478721 12489 rpcz_store.cc:269] 0709 12:14:09.264832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.560109 (+295277us) service_pool.cc:224] Handling call
0709 12:14:11.477899 (+1917790us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:11.478101 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2327522560}
I20260709 12:14:11.482995 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:11.459573 18373 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:11.471540 18643 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:11.404887 18559 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:11.477237 18415 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:11.459381 18721 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:11.455066 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.486220 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:11.495796 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.476913 12488 rpcz_store.cc:276] 0709 12:14:08.348983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.712716 (+1363733us) service_pool.cc:224] Handling call
0709 12:14:09.713210 (+   494us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:09.713711 (+   501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:09.713759 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:09.783475 (+ 69716us) log.cc:844] Serialized 52 byte log entry
0709 12:14:10.360234 (+576759us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:10.363864 (+  3630us) log.cc:844] Serialized 14 byte log entry
0709 12:14:10.365861 (+  1997us) log.cc:844] Serialized 14 byte log entry
0709 12:14:10.366162 (+   301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:10.366250 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:11.060841 (+694591us) spinlock_profiling.cc:243] Waited 5.2 ms on lock 0x7b18003a3360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:11.471010 (+410169us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:11.476227 (+  5217us) raft_consensus.cc:1685] finished
0709 12:14:11.476252 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:11.476419 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":9019,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":108,"raft.queue_time_us":6003,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":19595,"spinlock_wait_cycles":16897152,"thread_start_us":555978,"threads_started":1,"wal-append.queue_time_us":806447}
W20260709 12:14:11.498668 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62180us
I20260709 12:14:11.483147 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:11.503269 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:11.514792 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59835.4us
I20260709 12:14:11.515605 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.515950 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.514693 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:11.497237 18711 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:14:11.518147 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.521175 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.405408 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.522709 12492 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:14:11.505479 18628 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:11.502894 18627 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:11.517292 18415 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:11.478557 18619 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:11.552942 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188312us
I20260709 12:14:11.497263 18629 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:14:11.559063 12494 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:11.528333 18559 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:11.524271 18688 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:11.528903 18649 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:11.569028 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.542096 18522 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:11.541873 18653 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:11.584100 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 783) took 1675 ms. Trace:
I20260709 12:14:11.547080 18628 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:11.594748 13254 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:14:11.552795 18552 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:11.596493 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:11.565127 17950 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.808s	user 0.015s	sys 0.001s
I20260709 12:14:11.590103 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.597266 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.597959 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.599843 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:11.600828 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:11.596889 18686 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.926s	user 0.003s	sys 0.000s
I20260709 12:14:11.542542 18721 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:11.521056 18652 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:14:11.547842 18604 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.551165 18635 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:11.606384 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.600781 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.559518 18627 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.630890 18583 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:11.631712 18627 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:11.609951 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50470.1us
I20260709 12:14:11.558967 18711 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:11.614497 18543 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:11.646013 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124532us
I20260709 12:14:11.603149 12643 rpcz_store.cc:276] 0709 12:14:09.908788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:09.995412 (+ 86624us) service_pool.cc:224] Handling call
0709 12:14:09.996078 (+   666us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:10.240377 (+244299us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:10.240415 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:10.240460 (+    45us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:10.240489 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:11.583736 (+1343247us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:11.583809 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:11.584012 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":561242496}
I20260709 12:14:11.624058 17950 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.608042 18628 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:11.633031 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.614126 18681 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:14:11.635932 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114831us
I20260709 12:14:11.632733 18515 raft_consensus.cc:2764] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:11.641986 18721 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 15.5, Last appended by leader: 5, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.643869 18727 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:11.644088 18619 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:11.614501 18552 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:11.621821 18542 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:14:11.616734 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:11.694556 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.619714 18522 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:11.658476 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:11.662928 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 144846us
I20260709 12:14:11.665874 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.625325 18635 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:11.667764 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.669284 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.617146 18495 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:14:11.620971 18531 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:14:11.697878 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:11.709618 12570 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:14:11.711194 18604 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:11.716243 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:11.719976 17950 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:11.720316 18515 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:11.620647 18665 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.655902 18708 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:11.721670 18708 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:11.722189 18708 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:14:11.722353 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.722982 18665 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.665174 18730 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
I20260709 12:14:11.724951 18730 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 15.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:14:11.725412 18730 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:11.725781 18730 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:11.700680 18635 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:11.726863 18534 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:11.730101 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:11.731014 12572 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:11.732571 18705 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:11.733726 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64574.6us
I20260709 12:14:11.734032 12568 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:11.735549 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:11.736168 18705 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.701232 18522 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.736528 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.688311 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:11.671804 18649 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:11.732583 18708 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.737465 18649 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.641402 18627 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:14:11.739193 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.737903 18598 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:14:11.701710 18732 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:14:11.726898 18730 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.720479 18619 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:11.742144 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:11.743672 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77861.3us
W20260709 12:14:11.744398 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:11.748387 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.758553 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:11.772706 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:11.771344 13255 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:11.770530 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.664459 18558 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.742972 18495 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:11.663777 18595 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:14:11.782539 18244 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:11.744040 18598 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:11.769145 18619 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:11.812305 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.769335 18627 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:11.776160 18620 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:11.673913 18737 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.118s
I20260709 12:14:11.709450 18355 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:14:11.773728 18649 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:11.799525 18730 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:11.824779 12563 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
W20260709 12:14:11.832655 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200446us
I20260709 12:14:11.834079 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.810950 12496 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:14:11.814988 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44581.6us
I20260709 12:14:11.793876 18722 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:11.812877 18708 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:11.815080 18627 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:11.835803 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.811966 18598 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:11.815766 18619 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:11.845778 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:11.845070 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:11.823227 18737 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:14:11.831624 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 187099us
I20260709 12:14:11.828117 18415 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:11.801487 18552 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:11.852309 18575 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:11.835664 18355 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:14:11.853399 18558 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:11.864879 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.887516 18620 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:11.890316 18415 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:11.846071 18725 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:14:11.815487 18709 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:14:11.843885 18515 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:11.847092 18627 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:11.889276 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.901412 18575 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:11.920688 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:11.899298 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 201218us
I20260709 12:14:11.868134 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:11.871431 18744 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:11.926853 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58882.4us
I20260709 12:14:11.845142 18722 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:11.901608 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57505.2us
W20260709 12:14:11.904903 18456 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.738s	user 0.012s	sys 0.001s
I20260709 12:14:11.915552 18649 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.911890 18711 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.958858 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:11.926052 18736 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.108s	user 0.002s	sys 0.000s
I20260709 12:14:11.922134 18620 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:11.916112 18415 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:11.940915 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.911511 18558 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:11.913873 18552 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:11.969419 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:11.997787 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87492us
I20260709 12:14:11.970283 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.999372 12494 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:11.999643 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:11.928642 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:11.973943 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.974208 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:11.987519 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:11.989338 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:11.929199 18355 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:14:12.004062 18348 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.014s	sys 0.001s
I20260709 12:14:12.004259 18355 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:11.978963 18534 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:12.005930 18534 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:12.006603 18534 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:11.926733 18276 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:14:12.007789 18534 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.009907 18558 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.012809 18558 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:12.013131 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2830) took 1112 ms. Trace:
I20260709 12:14:12.016242 18541 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:12.016769 18541 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:12.017830 18541 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.018999 18708 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:12.019263 18541 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:12.021513 18244 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.022037 18643 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:12.019052 18683 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:12.022624 18643 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:12.023468 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.024286 12496 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:12.027469 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:12.030504 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:11.973304 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:12.032486 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:12.032987 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.033093 12494 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:11.983520 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:11.987298 18744 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:12.013576 12570 rpcz_store.cc:276] 0709 12:14:10.900708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:11.128851 (+228143us) service_pool.cc:224] Handling call
0709 12:14:11.132845 (+  3994us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:11.710107 (+577262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:12.012829 (+302722us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:12.013060 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1325521152}
I20260709 12:14:12.044627 12643 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:12.046125 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.036866 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:11.981184 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 290725us
W20260709 12:14:12.040634 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112085us
I20260709 12:14:11.979414 18749 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:12.059908 12645 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:12.060140 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.059919 12644 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:14:12.060322 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.064059 12640 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:14:12.060580 12646 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:11.991865 18649 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.061137 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:11.999742 18738 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:14:11.979102 18746 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:11.988844 18729 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:12.075099 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47757.8us
W20260709 12:14:12.075413 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92033us
I20260709 12:14:12.076742 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:12.067354 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:12.086498 12639 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.8-18.8]
W20260709 12:14:12.100739 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.102020 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:12.100618 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97093.9us
I20260709 12:14:12.039932 18737 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:12.107115 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:12.048447 18276 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:12.054997 18645 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:12.113694 12644 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:14:12.114944 12640 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:14:12.115818 12646 raft_consensus.cc:1217] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Deduplicated request from leader. Original: 12.6->[14.7-14.7]   Dedup: 14.7->[]
I20260709 12:14:11.993969 18711 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.120851 12646 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:14:12.082643 18749 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:12.005012 18355 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.098124 18348 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:12.141768 18531 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:12.142582 18244 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:12.042930 18740 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:14:12.092151 18746 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:12.101677 18688 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:11.989244 18456 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.179407 12645 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 18. Ops: []
I20260709 12:14:12.215408 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:12.023684 18643 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.169288 18631 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:14:12.212924 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122078us
I20260709 12:14:12.111361 18744 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:14:12.220505 18688 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:12.127377 18495 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.098s
W20260709 12:14:12.213037 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 225645us
I20260709 12:14:12.234789 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.170946 18348 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:12.169801 18749 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:12.102543 18729 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.088328 18575 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:12.227530 18635 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.500s	user 0.013s	sys 0.000s
I20260709 12:14:12.289393 18635 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.291464 18729 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.292495 18635 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.247756 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:12.294265 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46684.8us
I20260709 12:14:12.295359 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.297386 18595 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:12.298180 18595 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:12.299333 18595 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.301990 18761 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:12.303001 18761 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:14:12.307852 18739 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.404s	user 0.003s	sys 0.000s
I20260709 12:14:12.312929 18683 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:14:12.313457 18683 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:12.260761 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.220696 18746 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:12.223927 18516 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.223626 18620 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.7, Last appended by leader: 7, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.230180 18649 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:12.207705 18531 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.231788 18631 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:12.264595 18619 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:12.270804 18628 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.602s	user 0.014s	sys 0.000s
I20260709 12:14:12.275522 18355 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:12.276499 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.277725 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.278167 12642 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:12.356796 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:14:12.282197 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:12.283685 18753 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:14:12.315693 12640 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:12.335755 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 259005us
I20260709 12:14:12.336256 12646 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:12.364233 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.316803 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:12.258054 18348 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:12.337800 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 246425us
I20260709 12:14:12.214355 18244 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:12.373322 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56618.1us
W20260709 12:14:12.351154 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 123128us
I20260709 12:14:12.351227 18631 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:12.382882 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:12.383231 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.357789 12639 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:12.338440 18754 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:12.365603 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.336508 18756 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:12.375319 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:12.244458 18688 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:12.379717 12488 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:14:12.380805 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.314364 18683 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.381790 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.391956 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.403580 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:12.407936 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.409802 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.410647 12639 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:12.412089 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.419554 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.393527 18516 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:12.427899 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.430007 12644 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:12.427121 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.389699 18754 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:12.405270 18495 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:12.358635 18628 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.446010 12642 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:12.447992 18665 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:12.450309 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.452884 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.416189 18744 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:12.431234 18558 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:12.454612 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61989.7us
I20260709 12:14:12.461117 12646 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:12.468869 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.469776 12642 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:12.427184 18711 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:12.404626 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:12.472292 12489 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:12.480829 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97941us
I20260709 12:14:12.481992 12644 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:12.427847 18516 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.488097 12572 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:14:12.493225 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.457172 18665 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:12.459141 18495 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:12.496094 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.444180 18754 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:12.472106 18683 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:14:12.484124 18627 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.554s	user 0.010s	sys 0.004s
I20260709 12:14:12.515081 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.514884 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.517989 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.518891 12642 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:14:12.522517 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.471097 18558 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.508718 18456 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:12.524222 18687 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:12.547358 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.560576 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.565671 12642 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:14:12.567142 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 331829us
I20260709 12:14:12.570135 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.514950 18627 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.546365 18687 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:12.597807 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.599370 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:14:12.564211 18730 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.733s	user 0.009s	sys 0.004s
W20260709 12:14:12.572690 18770 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:14:12.568886 18722 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:12.604535 18772 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:14:12.605087 18541 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:12.613091 18687 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.671541 18541 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.672935 18687 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:12.598407 18495 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.617898 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2863) took 1135 ms. Trace:
I20260709 12:14:12.677038 12563 rpcz_store.cc:276] 0709 12:14:11.482210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:11.485447 (+  3237us) service_pool.cc:224] Handling call
0709 12:14:11.485809 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:11.826675 (+340866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:12.617571 (+790896us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:12.617818 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":778956160}
I20260709 12:14:12.625511 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:12.678148 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166501us
I20260709 12:14:12.678787 12638 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
I20260709 12:14:12.626344 12639 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:12.626874 12646 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:14:12.680114 12646 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 7
I20260709 12:14:12.680382 12646 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:14:12.680616 12646 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:12.628671 18627 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:12.629128 18415 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.000s
I20260709 12:14:12.629343 12640 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:12.629390 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:12.686735 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.633734 18595 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:12.635121 18534 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:12.635589 18760 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:12.636700 18598 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.751s	user 0.007s	sys 0.004s
I20260709 12:14:12.642474 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:12.643276 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:12.645795 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.689841 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.562417 18665 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.558640 18751 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:14:12.647917 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:12.649199 12494 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [18.12-20.13]
I20260709 12:14:12.648000 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:12.649989 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:12.652050 18734 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.932s	user 0.003s	sys 0.000s
I20260709 12:14:12.653151 13257 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20260709 12:14:12.655673 17950 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:12.658924 18558 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:12.659730 18688 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.660349 18531 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.660750 18244 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.661051 18631 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.730154 12461 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
I20260709 12:14:12.732053 18754 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.733359 13258 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260709 12:14:12.710686 18598 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.690817 18415 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.738085 12497 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:14:12.739681 18415 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:12.687600 18722 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:12.665328 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.740936 18643 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:12.701550 18534 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.741523 12642 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:12.602496 18619 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.742264 18515 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:12.742851 18515 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:12.742998 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.713748 18752 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:12.743912 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:12.744237 18752 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:12.745087 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.691993 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.745236 18752 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.746649 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.747319 18583 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:14:12.747857 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.748329 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.694744 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.749289 12566 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:12.749779 18541 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.739806 12643 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:12.753127 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.754355 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.755388 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.756387 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.757094 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.698012 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.701946 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.762252 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.702153 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.703176 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:12.705178 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62899.1us
I20260709 12:14:12.705931 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.660422 18516 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:12.687925 18723 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.842s	user 0.003s	sys 0.000s
I20260709 12:14:12.718125 13253 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:14:12.769865 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.770561 13253 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.771180 13253 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:12.721148 13254 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:14:12.772539 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.773245 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:12.726240 12641 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:12.724916 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:12.703712 18595 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.720364 17950 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:12.666061 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.743333 18665 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:12.748611 12572 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:12.786083 17950 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 18.8, Last appended by leader: 8, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.693343 12565 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:12.787523 17950 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:12.710724 18760 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.791123 18760 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.791208 18598 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:12.798218 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.757472 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.667570 13255 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:12.697829 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.664585 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.801155 12645 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:12.762425 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.702600 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.803967 12639 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
I20260709 12:14:12.762951 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:12.765172 12563 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:14:12.765852 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.762941 12566 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:12.805166 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.806057 12567 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:14:12.772989 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:12.808858 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.685729 18603 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:14:12.705010 18729 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:12.816210 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.780687 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:12.781757 12494 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:12.822736 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: [11.7-15.8]
I20260709 12:14:12.823263 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.823973 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:12.730194 18631 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:12.783651 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 789) took 1427 ms. Trace:
I20260709 12:14:12.784231 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.810758 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.812071 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.812542 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:12.686904 18749 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.776826 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:12.840597 12637 rpcz_store.cc:276] 0709 12:14:11.356294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:11.356416 (+   122us) service_pool.cc:224] Handling call
0709 12:14:11.382296 (+ 25880us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:14:11.390101 (+  7805us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:11.390143 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:14:11.390935 (+   792us) log.cc:844] Serialized 78 byte log entry
0709 12:14:11.652838 (+261903us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:11.687252 (+ 34414us) log.cc:844] Serialized 14 byte log entry
0709 12:14:11.687569 (+   317us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:11.687683 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:12.661113 (+973430us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:12.783315 (+122202us) raft_consensus.cc:1685] finished
0709 12:14:12.783432 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:12.783588 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46158,"prepare.queue_time_us":34798,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":48896,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":5421,"spinlock_wait_cycles":43954048,"thread_start_us":261414,"threads_started":1,"wal-append.queue_time_us":397719}
I20260709 12:14:12.787570 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.783030 18516 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:12.825328 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:12.826206 18348 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.828024 12645 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:14:12.833039 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:12.814352 18683 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:12.833119 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:12.835522 18688 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.820679 18598 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:12.837157 18760 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:12.839907 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:12.888103 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48254.6us
I20260709 12:14:12.843576 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:12.891557 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.891649 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:12.862221 12643 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:14:12.863591 12638 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:12.740983 18722 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.875653 18603 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:12.877856 18348 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.879949 18749 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:12.880761 18516 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:14:12.882081 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49050.5us
I20260709 12:14:12.858409 18761 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:12.894125 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 303538us
I20260709 12:14:12.900282 12646 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:12.900105 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:12.926141 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.882222 12564 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:12.914855 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:12.916743 12566 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:12.943679 12565 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:12.948412 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.958382 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:12.962921 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.969732 12566 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:12.972905 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:12.919587 18516 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:12.899065 18598 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:12.894357 18244 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:12.813516 18746 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.995952 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:12.908797 18619 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:12.909729 18583 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:12.921272 18631 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:12.950362 18543 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:12.865226 18541 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.119s
I20260709 12:14:12.845547 18729 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.018747 18598 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:13.008097 12641 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: [11.9-13.10]
I20260709 12:14:13.010643 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:13.052677 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56844.7us
I20260709 12:14:13.053673 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.002744 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:12.983064 18709 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:12.881597 18683 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:12.984067 18749 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:13.015893 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:12.990617 18760 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:12.990934 18730 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:13.019858 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:12.918756 18603 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:12.975919 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:13.001416 18681 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.282s	user 0.003s	sys 0.018s
W20260709 12:14:13.066593 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56077.6us
I20260709 12:14:13.054823 12492 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:13.062058 12571 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:13.011005 12638 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:14:13.004195 18558 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
W20260709 12:14:13.094717 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119052us
I20260709 12:14:13.102046 12461 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 13. Current term is 14.
W20260709 12:14:13.069300 18681 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.381s	user 0.003s	sys 0.018s
I20260709 12:14:13.109272 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:13.110033 12495 raft_consensus.cc:606] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:13.066983 18740 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:13.066164 18631 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:13.115665 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.119405 12641 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18. Re-sending same reply.
I20260709 12:14:13.094022 18754 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.067232 18749 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:13.095062 18543 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:13.128322 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:13.135150 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.109078 18681 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:13.140337 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71804.6us
I20260709 12:14:13.022934 18744 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.113s
I20260709 12:14:13.149142 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.093268 18730 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.159854 18730 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:13.125172 18749 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:13.163120 18534 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.164266 18631 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.168272 18722 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.068960 18516 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:13.155511 12597 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (5 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:14:13.185925 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:13.186848 12495 raft_consensus.cc:606] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:13.156108 18788 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:14:13.174397 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:13.154335 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100779us
W20260709 12:14:13.194177 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50046.1us
I20260709 12:14:13.194945 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.177608 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.195925 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.177773 12567 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:14:13.187548 12495 raft_consensus.cc:614] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:13.136873 18760 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:13.198762 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:13.144826 18688 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:13.200172 12567 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:13.141342 18761 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:13.208220 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:13.216301 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:13.124045 18515 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.218742 12567 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:14:13.214174 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.178417 18558 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:13.234784 18604 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:13.235239 18604 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:13.236492 18786 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:14:13.230466 18761 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.238231 18583 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:13.238816 18583 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:13.231300 18771 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:13.239730 18771 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:13.239956 18583 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.233186 18754 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.236142 18604 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.238875 18761 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.228121 12567 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:14:13.210222 18688 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:13.249768 12567 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:14:13.255388 12567 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:13.201938 18779 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:13.240674 18771 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.227662 18649 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.261682 18688 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:13.248811 18711 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.226190 18534 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:13.250314 18516 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:13.297179 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 382420us
I20260709 12:14:13.212342 18629 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 12:14:13.261682 18558 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:13.303861 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.210999 18690 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:14:13.311190 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.283114 12641 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:14:13.327355 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:13.261729 18760 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 12:14:13.292701 18595 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.347633 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.261993 18754 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.306671 18534 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:13.356220 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:13.356850 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:13.317404 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:13.294571 18792 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:13.351629 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:13.541337 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 189780us
I20260709 12:14:13.542445 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.354249 18721 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.323899 18730 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:13.325212 18722 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:13.307104 18779 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:14:13.310856 18711 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.364728 18534 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:13.551666 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 254446us
I20260709 12:14:13.564343 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:13.296607 18649 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.303138 18681 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:13.558099 18797 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.250s	user 0.000s	sys 0.008s
I20260709 12:14:13.550313 18746 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:13.669912 12645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.000s	sys 0.067s
W20260709 12:14:13.700459 12645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.000s	sys 0.067s
W20260709 12:14:13.698380 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 503461us
W20260709 12:14:13.652817 18789 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.419s	user 0.001s	sys 0.001s
W20260709 12:14:13.706496 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:13.710888 12638 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:13.712714 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.562641 18778 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:13.572063 18690 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:13.717404 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:13.652637 18373 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:13.726373 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:13.730154 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.024s	sys 0.039s
I20260709 12:14:13.668933 18683 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:13.563161 18721 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.705063 18244 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:13.706636 18752 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:13.715456 18628 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:13.719233 18791 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.430s	user 0.000s	sys 0.001s
I20260709 12:14:13.718355 18746 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:13.558458 18792 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.710184 18761 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.668025 18802 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.749747 18752 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:13.729553 18559 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:13.755443 18746 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:13.780915 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3023) took 1133 ms. Trace:
I20260709 12:14:13.723558 18778 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:13.755462 18729 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:13.782878 12488 rpcz_store.cc:276] 0709 12:14:12.647078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.647649 (+   571us) service_pool.cc:224] Handling call
0709 12:14:13.780624 (+1132975us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:13.780813 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":256487168}
W20260709 12:14:13.788664 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:13.789320 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:13.757663 18730 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.761554 18515 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.763795 18636 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:13.763324 18802 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:13.824361 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.018s	sys 0.010s
W20260709 12:14:13.825147 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.018s	sys 0.010s
W20260709 12:14:13.828047 12644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.014s	sys 0.014s
W20260709 12:14:13.828593 12644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.014s	sys 0.014s
I20260709 12:14:13.791062 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:13.769774 18761 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:13.822793 18801 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.438s	user 0.000s	sys 0.002s
W20260709 12:14:13.731258 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.024s	sys 0.039s
W20260709 12:14:13.825497 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 583674us
I20260709 12:14:13.836109 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:13.819198 18598 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.750s	user 0.016s	sys 0.000s
I20260709 12:14:13.848835 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:13.874676 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2699) took 1237 ms. Trace:
W20260709 12:14:13.877107 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.026s	sys 0.036s
I20260709 12:14:13.878592 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.879546 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.879841 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.881320 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.881836 12646 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:13.847352 18740 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.042s	sys 0.064s
I20260709 12:14:13.891541 12497 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:13.875629 12644 rpcz_store.cc:276] 0709 12:14:12.637531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.859504 (+221973us) service_pool.cc:224] Handling call
0709 12:14:13.287652 (+428148us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:13.287886 (+   234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:13.288160 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:13.288524 (+   364us) log.cc:844] Serialized 14 byte log entry
0709 12:14:13.874092 (+585568us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:13.874201 (+   109us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:13.874270 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:13.874375 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:13.874583 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34000,"prepare.queue_time_us":5746,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"spinlock_wait_cycles":3993472,"thread_start_us":551024,"threads_started":1}
W20260709 12:14:13.948745 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 406372us
I20260709 12:14:13.830792 18754 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:13.885852 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.884538 12641 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:13.952589 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.956390 12643 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:13.812917 18730 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:13.958766 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.960099 12645 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:13.959908 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:13.952898 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 214557us
I20260709 12:14:13.962069 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.962270 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:13.963243 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 391623us
I20260709 12:14:13.962973 12643 raft_consensus.cc:2376] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14. Re-sending same reply.
I20260709 12:14:13.964286 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.967752 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.968871 12641 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:13.969964 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:13.972921 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:13.882364 18515 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:14:13.891067 18740 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.042s	sys 0.064s
I20260709 12:14:13.885264 18779 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:13.979277 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2701) took 1326 ms. Trace:
I20260709 12:14:13.975526 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:13.986676 12644 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:13.979935 12637 rpcz_store.cc:276] 0709 12:14:12.652995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.865833 (+212838us) service_pool.cc:224] Handling call
0709 12:14:12.901439 (+ 35606us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:12.908924 (+  7485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:12.908978 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:12.921834 (+ 12856us) log.cc:844] Serialized 26 byte log entry
0709 12:14:13.131232 (+209398us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:13.569210 (+437978us) log.cc:844] Serialized 14 byte log entry
0709 12:14:13.573059 (+  3849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:13.573172 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:13.978898 (+405726us) raft_consensus.cc:1685] finished
0709 12:14:13.978931 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:13.979184 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29794,"prepare.queue_time_us":19275,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":101581,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":67835,"thread_start_us":626904,"threads_started":2,"wal-append.queue_time_us":374411}
W20260709 12:14:13.964192 18806 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.083s	user 0.001s	sys 0.001s
I20260709 12:14:13.954989 18711 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:13.825771 18752 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.007859 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.830363 18746 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:13.992457 18793 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
W20260709 12:14:14.019274 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 365808us
I20260709 12:14:13.977687 18515 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:14.028674 18515 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:14.030354 18575 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:14.030965 18802 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:14.039132 18803 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 12:14:14.039525 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 900) took 1384 ms. Trace:
I20260709 12:14:14.040172 13260 rpcz_store.cc:276] 0709 12:14:12.655389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.655942 (+   553us) service_pool.cc:224] Handling call
0709 12:14:12.656420 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:12.822993 (+166573us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.039121 (+1216128us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:14.039428 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285337088}
I20260709 12:14:14.040827 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 796) took 1041 ms. Trace:
I20260709 12:14:14.041427 12642 rpcz_store.cc:276] 0709 12:14:13.000230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.000583 (+   353us) service_pool.cc:224] Handling call
0709 12:14:13.002495 (+  1912us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:13.002851 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:13.002885 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:13.003519 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:14:13.243503 (+239984us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:13.244608 (+  1105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:13.244687 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:14.040498 (+795811us) raft_consensus.cc:1685] finished
0709 12:14:14.040533 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:14.040732 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":903,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":10786,"thread_start_us":201293,"threads_started":1,"wal-append.queue_time_us":645903}
I20260709 12:14:14.043604 18687 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.044374 18687 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.050704 18515 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.051491 18779 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:14.056213 18779 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:14.058430 18778 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.052884 18773 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:14.027899 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.065649 12644 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:14.017969 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.999955 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:13.979846 18754 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:14:14.071441 18730 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.077044 18373 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:13.882716 18598 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.078915 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.079550 13258 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:14:14.079947 18815 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:14:14.080291 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.080613 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:14.087370 18687 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.063400 18773 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:14.096118 18787 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.043s	sys 0.068s
I20260709 12:14:14.089150 18649 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.090781 18763 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:14:14.090915 18749 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.930s	user 0.013s	sys 0.000s
W20260709 12:14:14.069849 18665 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.278s	user 0.016s	sys 0.000s
I20260709 12:14:14.060082 18721 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.106863 18598 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.063797 18746 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:14.106886 18604 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:14.109290 18787 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.044s	sys 0.068s
I20260709 12:14:14.100528 18792 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.113590 18737 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:14.112275 18763 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:14.122222 18737 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:14.138412 18761 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.116797 18749 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.141592 18779 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.141882 18763 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.143146 18763 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:14.143558 18749 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.143987 18779 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.126444 18773 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.146041 18773 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:14.186990 12489 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.8-18.8]
I20260709 12:14:14.187836 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:13.884177 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.190681 12496 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:13.888870 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.198896 12488 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:13.887948 12494 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:13.950249 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:14.205863 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 255524us
I20260709 12:14:14.206589 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.207887 12493 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:14.209398 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:13.883698 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.026s	sys 0.037s
I20260709 12:14:14.215044 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:14.217111 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2843) took 1456 ms. Trace:
W20260709 12:14:14.217971 12563 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.004s	sys 0.017s
W20260709 12:14:14.218734 12563 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.004s	sys 0.017s
I20260709 12:14:14.217938 12566 rpcz_store.cc:276] 0709 12:14:12.760986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.056577 (+295591us) service_pool.cc:224] Handling call
0709 12:14:13.057089 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:13.057424 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:13.057896 (+   472us) log.cc:844] Serialized 14 byte log entry
0709 12:14:14.215874 (+1157978us) log.cc:844] Serialized 14 byte log entry
0709 12:14:14.216230 (+   356us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:14.216324 (+    94us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:14.216413 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.216563 (+   150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:14.216828 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12092,"prepare.queue_time_us":5943,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"thread_start_us":1156350,"threads_started":1,"wal-append.queue_time_us":1077294}
I20260709 12:14:14.220340 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2880) took 1436 ms. Trace:
I20260709 12:14:14.222136 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:14.222468 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.222307 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:14.224078 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.225183 12566 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:14.282312 18721 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:14.226614 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1023) took 1276 ms. Trace:
I20260709 12:14:14.230458 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.231886 12569 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
W20260709 12:14:14.232228 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 406872us
I20260709 12:14:14.234787 18244 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:14.098944 18373 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:14.105124 17950 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.317s	user 0.017s	sys 0.000s
I20260709 12:14:14.242226 18531 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:14.246207 18542 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:14.247783 18583 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.248708 18598 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.249938 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.248597 18355 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.003s
I20260709 12:14:14.255707 18415 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:14.257608 18559 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:14.260416 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2431) took 1480 ms. Trace:
I20260709 12:14:14.262043 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2422) took 2238 ms. Trace:
I20260709 12:14:14.220757 12563 rpcz_store.cc:276] 0709 12:14:12.784250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.100180 (+315930us) service_pool.cc:224] Handling call
0709 12:14:13.100474 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:13.100683 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:13.120022 (+ 19339us) log.cc:844] Serialized 14 byte log entry
0709 12:14:14.219551 (+1099529us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:14.219717 (+   166us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:14.219854 (+   137us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.220004 (+   150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:14.220276 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11691,"prepare.queue_time_us":21354,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":136107,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8731,"thread_start_us":1093791,"threads_started":2,"wal-append.queue_time_us":709830}
I20260709 12:14:14.245131 18515 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.271873 18631 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.271934 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2875) took 1571 ms. Trace:
I20260709 12:14:14.276963 18665 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:14.276731 18727 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:14:14.282497 18643 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.541s	user 0.011s	sys 0.008s
I20260709 12:14:14.284411 18348 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:14.288751 18244 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:14.313868 12489 rpcz_store.cc:276] 0709 12:14:12.779865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.780032 (+   167us) service_pool.cc:224] Handling call
0709 12:14:12.780539 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:14.188836 (+1408297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.259962 (+ 71126us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:14.260304 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3231840000}
I20260709 12:14:14.283870 18805 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:14.319624 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:14.318951 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.314330 12492 rpcz_store.cc:276] 0709 12:14:12.025980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.048927 (+ 22947us) service_pool.cc:224] Handling call
0709 12:14:12.098271 (+ 49344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:13.100295 (+1002024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.261413 (+1161118us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:14.261809 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2189116288}
I20260709 12:14:14.330097 18792 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:14.330549 18730 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.331239 18792 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.331776 18730 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.334635 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2871) took 1635 ms. Trace:
I20260709 12:14:14.335711 18727 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:14.315587 12565 rpcz_store.cc:276] 0709 12:14:12.700490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.788447 (+ 87957us) service_pool.cc:224] Handling call
0709 12:14:12.943340 (+154893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:12.944304 (+   964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.271451 (+1327147us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54002472d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:14.271716 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355111808}
I20260709 12:14:14.336557 18727 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:14.341168 18802 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:14:14.342743 18727 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 13, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.343815 18729 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.318166 18348 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:14.344990 18727 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:14.344710 18729 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.284652 12567 rpcz_store.cc:276] 0709 12:14:12.950574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.261852 (+311278us) service_pool.cc:224] Handling call
0709 12:14:13.262244 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:13.262850 (+   606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:13.262878 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:13.263413 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:14:13.751713 (+488300us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:13.751761 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:13.751825 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:14.225922 (+474097us) raft_consensus.cc:1685] finished
0709 12:14:14.225970 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:14.226347 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4880,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":96,"thread_start_us":451549,"threads_started":1,"wal-append.queue_time_us":686160}
I20260709 12:14:14.353266 18558 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:14.354008 18709 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:14.355037 18516 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.043s	user 0.009s	sys 0.017s
I20260709 12:14:14.358065 13260 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:14.360162 18348 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.363451 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.365692 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:14.368101 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.334791 18355 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:14.370043 18355 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:14.370757 18779 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.371766 18779 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.374114 18778 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:14.374411 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145092us
I20260709 12:14:14.375011 18778 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.375756 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:14.378739 18631 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.322336 18244 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:14.335484 12564 rpcz_store.cc:276] 0709 12:14:12.699172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.699420 (+   248us) service_pool.cc:224] Handling call
0709 12:14:12.723022 (+ 23602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:12.927015 (+203993us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.333851 (+1406836us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:14.334377 (+   526us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365310848}
I20260709 12:14:14.402940 18541 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.403782 18541 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:14.405009 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.392822 18583 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.406332 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 797) took 1402 ms. Trace:
I20260709 12:14:14.406971 12638 rpcz_store.cc:276] 0709 12:14:13.004905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.173995 (+169090us) service_pool.cc:224] Handling call
0709 12:14:14.405708 (+1231713us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:14.406026 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1110092032}
I20260709 12:14:14.409058 18583 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.409997 18559 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:14.410864 18792 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.412469 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:14.415637 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.354041 18558 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:14.412711 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:14.365422 18603 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.412356 18729 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.411010 18559 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.422837 18665 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:14.419843 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:14.430248 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.437749 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:14.438953 18735 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.439544 18559 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.439895 18735 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.441847 18833 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.442523 12568 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:14:14.442836 18833 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.445832 18816 raft_consensus.cc:993] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:14:14.446580 12568 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:14.446868 18583 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:14.447803 18583 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.449710 12568 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:14:14.449965 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63359.1us
I20260709 12:14:14.452884 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:14.492209 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98095us
I20260709 12:14:14.454388 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:14.427855 18631 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:14.495136 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:14.495500 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120028us
I20260709 12:14:14.496948 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:14.498471 18583 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.499636 18583 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:14.500586 18603 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:14.502105 18681 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.503265 18765 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:14.502882 18681 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.503809 18765 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:14.504312 18595 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:14.505264 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.537s (SENT) [suppressed 26 similar messages]
I20260709 12:14:14.505113 18595 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.507977 18816 raft_consensus.cc:1081] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:14:14.504325 18765 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:14.527189 18595 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.508849 12568 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260709 12:14:14.528796 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.446540 18810 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.534689 18810 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.536892 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:14.541966 18559 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:14.542069 18812 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:14.542979 18559 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.545836 18456 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:14.546677 18456 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.547914 18735 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.455744 18348 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:14.509656 18833 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.515686 18727 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:14.516835 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:14.519079 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.519870 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.520880 12565 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:14.522045 12494 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:14:14.529168 12568 pending_rounds.cc:85] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:14.553444 12568 pending_rounds.cc:107] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:14:14.553833 12568 raft_consensus.cc:2889] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:14.508562 18516 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:14.538658 18653 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.555022 18730 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.355098 18709 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.555768 18653 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.557670 18516 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:14.539471 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86878.7us
I20260709 12:14:14.558290 18709 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.539794 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:14.540665 18495 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:14.541100 18832 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.500342 18583 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.398626 18598 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.592787 18583 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.487707 18778 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.555078 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.571971 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.572983 18832 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.574160 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.574465 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.575541 18558 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.576525 18541 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.543051 18812 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.577700 18629 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:14.577848 12563 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 8. (term mismatch)
I20260709 12:14:14.578603 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:14.579993 18516 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.330758 18737 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.583293 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:14.586058 18373 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.599385 12643 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:14.602376 12567 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:14.606801 12568 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:14.613029 12563 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:14.668464 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:14.614593 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117675us
I20260709 12:14:14.670513 12637 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 12:14:14.501364 18603 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.670789 18531 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:14.675571 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:14.676512 12644 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:14.676069 18531 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.676702 18603 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.678177 18737 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:14.681257 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2841) took 1921 ms. Trace:
I20260709 12:14:14.607306 18653 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:14.620795 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115725us
W20260709 12:14:14.706552 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128109us
I20260709 12:14:14.668329 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.686152 12563 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 16.8
I20260709 12:14:14.712365 12563 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:14.682040 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2846) took 1920 ms. Trace:
I20260709 12:14:14.568132 18495 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:14.710741 18834 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.611598 18595 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.715703 13260 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:14.713681 12569 rpcz_store.cc:276] 0709 12:14:12.761917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.069202 (+307285us) service_pool.cc:224] Handling call
0709 12:14:13.095830 (+ 26628us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:14.299772 (+1203942us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.614505 (+314733us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:14.681937 (+ 67432us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2610827008}
W20260709 12:14:14.670615 18746 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.550s	user 0.007s	sys 0.010s
W20260709 12:14:14.724715 12493 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.516s	user 0.006s	sys 0.009s
I20260709 12:14:14.728596 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:14.713819 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:14.729580 12489 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: [9.8-13.10]
I20260709 12:14:14.725356 12642 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:14.621385 18629 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:14.739697 18629 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.740684 18558 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:14.743012 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2842) took 1982 ms. Trace:
I20260709 12:14:14.743474 12570 rpcz_store.cc:276] 0709 12:14:12.760660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.869053 (+108393us) service_pool.cc:224] Handling call
0709 12:14:12.903695 (+ 34642us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:12.905043 (+  1348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:12.905088 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:12.905788 (+   700us) log.cc:844] Serialized 26 byte log entry
0709 12:14:14.332513 (+1426725us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b180001b220. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:14.502610 (+170097us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:14.502722 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.502861 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:14.742430 (+239569us) raft_consensus.cc:1685] finished
0709 12:14:14.742454 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:14.742803 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":879,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":41797,"spinlock_wait_cycles":27770240,"thread_start_us":68215,"threads_started":1,"wal-append.queue_time_us":1804667}
I20260709 12:14:14.744987 18456 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:14.747392 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.727535 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.750010 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:14.750707 18779 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.752135 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:14.752702 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 213258us
I20260709 12:14:14.753223 12493 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:14.752887 18816 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:14.759232 18456 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:14.760218 18812 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.760385 18456 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:14.730872 18516 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:14.760933 18456 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:14.761101 18516 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:14.731084 18534 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.054s	user 0.008s	sys 0.022s
I20260709 12:14:14.762034 18534 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.765553 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:14.717922 18355 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:14.732448 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.765782 18595 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.767947 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:14.768167 18832 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.768898 12490 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:14.769968 18805 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:14.770310 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.771307 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.593525 18598 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.604560 18765 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.753958 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:14.754626 18681 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.755823 18515 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.741400 18583 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:14:14.757656 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.723829 18603 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:14.601413 18810 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.741187 18629 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:14.731600 12494 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:14.763831 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.737102 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:14.682632 12572 rpcz_store.cc:276] 0709 12:14:12.760294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:12.858319 (+ 98025us) service_pool.cc:224] Handling call
0709 12:14:12.858706 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:12.860928 (+  2222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:12.860963 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:12.871612 (+ 10649us) log.cc:844] Serialized 26 byte log entry
0709 12:14:14.336610 (+1464998us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b180044c440. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:14.515257 (+178647us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:14.515321 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.515471 (+   150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:14.680700 (+165229us) raft_consensus.cc:1685] finished
0709 12:14:14.680723 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:14.681048 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1749,"raft.queue_time_us":5252,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":9250,"spinlock_wait_cycles":4562816,"thread_start_us":97266,"threads_started":1,"wal-append.queue_time_us":1806438}
I20260709 12:14:14.735061 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.772320 12489 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
W20260709 12:14:14.753772 12461 connection.cc:256] RPC call timeout handler was delayed by -1.029824551000047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.269s for a response
I20260709 12:14:14.780025 12565 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:14.761616 18516 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:14.720368 18495 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:14.767951 18534 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.728390 18559 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.803902 12488 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [18.12-20.13]
I20260709 12:14:14.787185 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:14.764699 18542 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:14:14.811012 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48177.6us
I20260709 12:14:14.819604 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.780918 18244 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.798437 18730 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:14.823109 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.821139 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:14.827342 12497 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:14.768285 18595 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:14.831861 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.834694 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.835072 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:14.763190 18834 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.842623 12572 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:14.843842 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.841466 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.845906 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.849182 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.849438 18456 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.850128 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:14.851271 18456 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:14.851444 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.852206 18531 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:14.793866 18515 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:14.851661 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:14.855664 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.860466 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:14.861776 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.862641 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:14.862892 12565 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:14:14.795162 18583 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:14:14.867111 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.849267 12494 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:14.864378 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.837682 18244 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.838042 18730 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:14.872705 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.851912 18355 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:14.878885 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.879741 12564 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
W20260709 12:14:14.880273 18688 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.495s	user 0.012s	sys 0.012s
W20260709 12:14:14.880147 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47714.8us
I20260709 12:14:14.875290 18559 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:14.890195 18604 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.769s	user 0.020s	sys 0.000s
I20260709 12:14:14.891342 12646 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:14.894490 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:14.895462 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.882383 18735 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.896198 18735 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:14.897100 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62198.7us
I20260709 12:14:14.898064 18244 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:14.884176 12566 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:14.854205 18681 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.895382 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:14.881402 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.889550 12568 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
W20260709 12:14:14.902760 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.904369 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.914971 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:14.880170 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:14.898860 18244 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.922199 18681 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:14.923012 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:14.924295 18244 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:14.924810 18839 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:14:14.925544 18515 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:14.926084 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 172274us
I20260709 12:14:14.926311 18811 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:14.927918 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:14.929751 18559 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:14.931066 18765 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.932430 13261 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:14.866063 18531 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:14.934233 18531 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.943792 12494 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:14:14.944413 12494 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 13
I20260709 12:14:14.944720 12494 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:14:14.886113 18730 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:14.956485 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.868103 18595 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:14.957700 18595 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:14.960361 18595 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.912899 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.937286 18531 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:14.963393 12565 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 12.
I20260709 12:14:14.945264 12494 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:14.968349 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.969321 12572 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:14:14.898903 18735 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:14.910885 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:14.970988 12490 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18.
W20260709 12:14:14.983928 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108105us
I20260709 12:14:14.986119 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:14.985493 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:14.989169 12495 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:14:14.990896 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:14.998327 13262 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:14.963300 18542 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:15.001860 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:15.007802 12566 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 12.
I20260709 12:14:14.924836 18681 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.003156 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:14.999112 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.919955 18604 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.085s
I20260709 12:14:15.014279 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:15.014919 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 277970us
I20260709 12:14:15.015264 12565 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:15.010509 18559 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 12:14:15.023512 12490 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:14:15.005955 18534 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:15.010970 18244 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:15.032256 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117579us
I20260709 12:14:14.962783 18688 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.040932 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:15.038966 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:14.990051 18811 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:14:15.050509 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.045578 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:15.047029 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.047665 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:15.034107 18810 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:14:15.049141 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 126313us
W20260709 12:14:15.065147 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.039911 18534 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:15.047992 18763 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.712s	user 0.015s	sys 0.003s
I20260709 12:14:15.070935 18834 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:15.020058 18515 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:14:15.000358 18595 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:14:15.094990 13255 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:14:15.082907 12565 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:14:15.086995 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.080165 12638 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:14:15.098443 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:15.097467 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:15.099252 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:14:15.095599 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44197.7us
I20260709 12:14:15.076305 18810 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.102339 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:15.103390 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:15.104168 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:15.108701 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:15.083442 18681 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:15.117792 13253 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:15.124058 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.125146 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:15.127038 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:15.132396 12566 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:15.136644 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.109513 18495 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.146575 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.147069 12493 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:15.110726 18810 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:15.112509 18515 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:14:15.128868 18681 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:15.153992 18773 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.956s	user 0.010s	sys 0.006s
I20260709 12:14:15.152174 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:15.170641 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 145446us
I20260709 12:14:15.143355 18604 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:14:15.192786 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90562.9us
I20260709 12:14:15.189960 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:15.167092 18792 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:15.173120 18681 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:15.196765 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90951.8us
I20260709 12:14:15.197422 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:15.200587 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115935us
I20260709 12:14:15.183524 18763 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:15.235999 12569 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:14:15.203315 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.208474 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:15.209690 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:15.210806 18534 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.211344 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.213819 18542 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:15.216353 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:15.216544 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.219316 12565 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:15.224767 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.225344 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.225685 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:15.226290 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.226611 18688 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.233799 18773 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:15.242359 13261 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:14:15.240072 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:15.250471 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:15.245960 12567 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.520s	user 0.013s	sys 0.000s
I20260709 12:14:15.253659 12567 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:15.262028 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56995.7us
I20260709 12:14:15.235692 18763 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.263147 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:15.269549 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53350.6us
I20260709 12:14:15.258083 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.265666 18456 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:15.226665 18760 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.178s
W20260709 12:14:15.288524 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:15.289863 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73485.8us
W20260709 12:14:15.286542 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75336.9us
I20260709 12:14:15.297849 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2453) took 1336 ms. Trace:
W20260709 12:14:15.303306 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.291260 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2883) took 2366 ms. Trace:
I20260709 12:14:15.255163 18541 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:15.310892 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:15.307411 12646 rpcz_store.cc:276] 0709 12:14:13.961846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.961996 (+   150us) service_pool.cc:224] Handling call
0709 12:14:15.297587 (+1335591us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:15.297781 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2134279552}
W20260709 12:14:15.320959 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:15.321368 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44526.2us
I20260709 12:14:15.322425 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:15.324944 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.328189 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.299688 18832 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:15.332043 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:15.290096 18812 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:15.333374 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.336510 12566 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:14:15.346174 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:15.257529 18730 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.275414 18763 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:15.285580 18534 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:15.274171 18709 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:15.285916 18595 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:15.290936 18811 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:15.258363 18542 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.342733 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.347867 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:15.350106 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.308624 12571 rpcz_store.cc:276] 0709 12:14:12.883699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:13.176936 (+293237us) service_pool.cc:224] Handling call
0709 12:14:13.177385 (+   449us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:13.199909 (+ 22524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:13.199942 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:14.452491 (+1252549us) spinlock_profiling.cc:243] Waited 9.28 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:14.456343 (+  3852us) log.cc:844] Serialized 52 byte log entry
0709 12:14:14.844085 (+387742us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:14.861376 (+ 17291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:14.869225 (+  7849us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:15.193514 (+324289us) raft_consensus.cc:1685] finished
0709 12:14:15.193538 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:15.248994 (+ 55456us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5698,"raft.queue_time_us":22179,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":31785,"spinlock_wait_cycles":21353728,"thread_start_us":136808,"threads_started":1,"wal-append.queue_time_us":508166}
I20260709 12:14:15.289971 18531 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:15.332672 18832 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:15.357544 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.332643 18559 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:15.333582 18792 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:15.357903 13257 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:15.369812 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:15.301026 18604 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:15.365463 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:15.340068 18816 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:15.326998 18495 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:15.366665 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.343685 18812 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.308354 18516 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.374779 12563 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:15.384019 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.351359 18779 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:15.379750 18792 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.302573 12569 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:14:15.398816 12569 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:14:15.399257 12569 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:15.398226 18812 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:15.375470 18559 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.381175 18534 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:15.393507 18348 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.836s	user 0.014s	sys 0.002s
I20260709 12:14:15.428319 12640 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:14:15.430040 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:15.438558 18761 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:15.439024 18730 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.439368 18761 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:15.440668 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.442042 12497 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:15.415279 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:15.443567 12646 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:15.374025 18709 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:15.419516 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67133.4us
I20260709 12:14:15.422158 18604 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.425952 18534 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:15.443819 12569 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.401796 18792 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:15.457469 12570 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:15.456806 12566 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 17. Ops: []
I20260709 12:14:15.472100 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.473147 12569 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.463034 18761 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.441399 18811 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:15.476169 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.477264 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:15.412281 18595 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:15.411808 18531 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:15.464913 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.475710 18761 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:15.484018 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.486954 12572 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:14:15.418550 18559 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:15.473181 12640 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:15.483239 18495 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:14:15.511513 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 127055us
I20260709 12:14:15.517398 12641 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:15.522629 12567 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:14:15.525013 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.527033 18773 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:15.529690 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.531953 18603 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:15.532856 18603 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:15.534400 18773 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:14:15.534650 18847 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:14:15.536346 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.495085 18816 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:14:15.552038 18816 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:15.552709 18816 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.554260 18534 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.555534 18816 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.556226 18816 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:15.556581 18816 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:15.541543 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3036) took 1346 ms. Trace:
I20260709 12:14:15.559401 12496 rpcz_store.cc:276] 0709 12:14:14.195567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:14.299264 (+103697us) service_pool.cc:224] Handling call
0709 12:14:14.324466 (+ 25202us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:14.412170 (+ 87704us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:14.671717 (+259547us) log.cc:844] Serialized 14 byte log entry
0709 12:14:14.902037 (+230320us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:14.902118 (+    81us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:14.902184 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:15.541297 (+639113us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:15.541354 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:15.541473 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":131611,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":56,"raft.queue_time_us":566,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":74065,"spinlock_wait_cycles":33311488,"thread_start_us":429806,"threads_started":2,"wal-append.queue_time_us":419161}
I20260709 12:14:15.541887 18737 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:15.561434 18516 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.561676 18737 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:15.562990 18737 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.564061 18737 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:15.543845 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 756) took 1180 ms. Trace:
I20260709 12:14:15.565779 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:15.534829 18773 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:15.568065 13255 rpcz_store.cc:276] 0709 12:14:14.363590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:14.367507 (+  3917us) service_pool.cc:224] Handling call
0709 12:14:15.543626 (+1176119us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:15.543757 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1651118848}
I20260709 12:14:15.490226 18515 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:15.569918 18773 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.571791 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.576537 18515 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:14:15.572940 18244 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.003s
I20260709 12:14:15.512907 18495 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.518873 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:15.577574 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:15.583132 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67420.3us
I20260709 12:14:15.582232 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:15.580590 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44323.8us
I20260709 12:14:15.490790 18531 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:15.595469 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.578322 18495 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:15.599046 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.598693 13255 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:15.602051 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.579178 18773 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:15.610067 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:15.611533 12610 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18.
I20260709 12:14:15.612001 12610 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:14:15.613456 12638 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 7. Preceding OpId from leader: term: 19 index: 8. (index mismatch)
I20260709 12:14:15.598572 18558 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:15.580791 18603 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:15.578096 18244 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:15.495807 18595 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.624176 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.491325 18779 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.600342 18688 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:14:15.616058 18558 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.627036 18774 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:15.583972 18832 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.628576 18834 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:15.608827 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.601677 18531 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:15.635429 12640 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:15.646346 18832 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:15.649796 18834 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.650424 18834 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:15.653795 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48660.9us
I20260709 12:14:15.655153 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:15.674125 18709 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.679078 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:15.680668 18629 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.876s	user 0.003s	sys 0.010s
I20260709 12:14:15.682858 18541 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:15.683533 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:15.683835 18541 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.684347 12497 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:15.686717 18541 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.688515 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:15.689371 12488 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:15.692628 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:15.693208 18811 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.693459 12488 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:15.694619 18688 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:15.694864 18542 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.695142 18688 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:15.696260 18688 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.696862 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.697741 18558 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:15.699299 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44347.5us
I20260709 12:14:15.700105 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:15.701406 18635 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:15.705487 12641 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:15.710667 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:15.709126 18515 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:15.713335 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.715088 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:15.700510 18629 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:15.708307 18688 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:15.746313 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.747270 12646 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:15.749815 18516 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:15.750420 18604 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.750531 18516 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.753506 18516 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.579689 18816 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.722474 18603 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18.
I20260709 12:14:15.724960 18774 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.767194 18542 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:15.767921 18542 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.773415 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.744088 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.776332 18542 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.774184 13255 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:15.722962 18629 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:15.779706 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49347.7us
I20260709 12:14:15.753064 18515 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.785722 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:15.781756 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45141.9us
I20260709 12:14:15.794360 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:15.790918 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 825) took 1058 ms. Trace:
I20260709 12:14:15.798446 12488 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:15.785595 18629 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:15.795547 12489 rpcz_store.cc:276] 0709 12:14:14.732406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:14.878965 (+146559us) service_pool.cc:224] Handling call
0709 12:14:14.879300 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:14.879561 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:14:14.896001 (+ 16440us) log.cc:844] Serialized 14 byte log entry
0709 12:14:15.790552 (+894551us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:15.790612 (+    60us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:15.790664 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:15.790732 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:15.790848 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8527,"prepare.queue_time_us":19687,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2750,"raft.queue_time_us":30947,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":10687,"thread_start_us":879773,"threads_started":2,"wal-append.queue_time_us":715561}
I20260709 12:14:15.828202 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:15.794155 18542 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:14:15.862372 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:15.809484 18456 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.003s
W20260709 12:14:15.873459 18810 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.000s
I20260709 12:14:15.878314 12497 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:15.875056 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.884554 12641 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 15 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 5.
I20260709 12:14:15.887773 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:15.889549 18774 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.867215 18456 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:15.890992 18456 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:15.892671 18516 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:15.893404 18516 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:15.895649 18516 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.896544 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:15.897272 12643 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:15.898123 12644 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:14:15.837534 18542 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:15.903935 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:15.904819 12642 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:15.906018 12646 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:14:15.904214 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 426478us
I20260709 12:14:15.912009 12639 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:14:15.914860 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.883244 18595 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.920342 18812 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.922973 18737 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:15.882254 18810 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:15.929317 18681 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.674s	user 0.005s	sys 0.005s
I20260709 12:14:15.930956 12637 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:15.939438 18760 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:14:15.939976 18515 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:15.964846 12638 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:15.902945 18542 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:15.967370 18812 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.942852 18779 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:15.946300 12645 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:14:15.948011 18559 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:15.951339 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:15.941383 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:15.933003 18495 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:15.955509 12492 raft_consensus.cc:606] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:15.951263 18763 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.584s	user 0.010s	sys 0.002s
I20260709 12:14:15.951963 18541 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:15.969031 12645 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
I20260709 12:14:15.925467 18816 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:15.976455 12496 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:15.981577 12645 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:14:15.982079 12645 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:15.983822 12640 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:15.959436 18737 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:16.003185 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 382013us
W20260709 12:14:16.002439 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.006s	sys 0.006s
I20260709 12:14:15.981734 18348 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:16.002396 18760 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:16.021883 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.058653 12496 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:15.989085 18355 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.024636 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:15.977545 18681 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:16.025534 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 436210us
I20260709 12:14:16.066807 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:16.027987 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 198995us
W20260709 12:14:16.029285 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.006s	sys 0.006s
I20260709 12:14:15.991379 18541 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.006927 18515 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:14:16.004019 18761 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.500s	user 0.013s	sys 0.000s
I20260709 12:14:15.982604 18792 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.076964 12489 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 5. Preceding OpId from leader: term: 17 index: 6. (index mismatch)
W20260709 12:14:16.044042 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.047081 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:15.978909 18763 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.057070 18373 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:16.029718 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88411.4us
W20260709 12:14:16.069043 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 730296us
I20260709 12:14:16.087545 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:16.073940 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.088160 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.083115 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:16.090487 12494 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:16.090582 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.083592 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.095103 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:16.095839 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:16.096199 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 292059us
I20260709 12:14:16.112722 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:16.096354 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:16.068610 18348 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.119185 18792 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.120131 13259 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:16.074041 18515 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:16.120954 18515 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:16.122397 18559 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.068653 18355 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.045486 18558 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:16.135594 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:16.138985 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55986.7us
W20260709 12:14:16.140003 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:16.147550 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51287.5us
I20260709 12:14:16.147550 12642 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:16.150009 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.150902 12566 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:16.149600 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62516us
I20260709 12:14:16.108687 18604 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.067374 18760 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.175872 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.176826 12491 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:16.177735 18760 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.177811 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.125433 18681 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.194221 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.194047 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:16.162997 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:16.165198 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.094913 18761 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 12:14:16.164992 18792 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:16.167920 18746 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:16.115728 18855 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:16.127635 18737 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.151994 18583 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:16.087193 18834 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.171286 18531 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.172775 18810 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.246122 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.146634 18773 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.173839 18629 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.119185 18763 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.134840 18861 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:14:16.175432 18872 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:16.180630 12491 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:16.203008 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.204061 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.269409 12496 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17.
I20260709 12:14:16.154076 18635 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:14:16.293540 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:16.294024 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:16.206748 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5159671760000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260709 12:14:16.228456 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.201695 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1040) took 1121 ms. Trace:
W20260709 12:14:16.295034 12536 connection.cc:256] RPC call timeout handler was delayed by -1.285567176000086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260709 12:14:16.295513 12536 connection.cc:256] RPC call timeout handler was delayed by -1.263967176000051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260709 12:14:16.239851 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.202771 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:16.241953 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 163657us
W20260709 12:14:16.242620 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48127.3us
I20260709 12:14:16.180662 18812 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.295248 12570 rpcz_store.cc:276] 0709 12:14:15.079843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:15.495620 (+415777us) service_pool.cc:224] Handling call
0709 12:14:15.520941 (+ 25321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:15.605910 (+ 84969us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:15.979760 (+373850us) log.cc:844] Serialized 14 byte log entry
0709 12:14:16.192043 (+212283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:16.192108 (+    65us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:16.200409 (+  8301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:16.200779 (+   370us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:16.201376 (+   597us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15300,"prepare.queue_time_us":509604,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":141696,"thread_start_us":535048,"threads_started":3}
I20260709 12:14:16.181015 18832 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:16.244987 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.245884 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.173496 18495 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.243559 18516 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:16.191149 18604 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.254643 18752 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:16.255578 18870 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.258100 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.264941 12566 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:16.243745 18855 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:16.199908 18792 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:16.268246 12491 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:16.313390 12497 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:16.336735 12637 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.199558 18709 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.095s
I20260709 12:14:16.262575 18583 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:16.269940 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.263078 18773 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.202696 18760 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
I20260709 12:14:16.230013 18811 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
I20260709 12:14:16.383962 18811 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:16.384349 18811 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:16.252310 18598 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:16.385830 18598 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:16.386240 18598 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:16.318255 12642 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.336916 12639 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 5. Preceding OpId from leader: term: 17 index: 6. (index mismatch)
W20260709 12:14:16.312500 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161537us
I20260709 12:14:16.344601 12645 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:16.321951 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.293478 18635 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:16.316123 18355 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.367213 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.342821 18792 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:16.431044 12639 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 13. Current term is 14. Ops: [11.9-13.10]
I20260709 12:14:16.371974 18773 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.372418 18752 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.375608 18531 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.376981 18558 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.441689 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:16.378469 18866 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:14:16.343551 18870 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.379257 18373 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:16.443276 18373 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:16.382884 18834 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.402101 18880 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:16.443629 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2877) took 1489 ms. Trace:
I20260709 12:14:16.402501 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:16.444684 18870 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.445268 13253 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:16.403895 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:16.446013 13261 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:16.402343 18559 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:16.405397 18541 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.446862 18559 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.369377 18872 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:16.368047 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 187103us
I20260709 12:14:16.421240 18629 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.449698 18348 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.422590 18832 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.450554 18832 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.449398 18559 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.413777 18635 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.423095 18604 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.424703 18583 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.453904 18856 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:16.426055 18855 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:16.435708 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 254696us
W20260709 12:14:16.439747 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.442059 12643 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:14:16.420601 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:16.457783 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:16.462208 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.462023 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.444118 12569 rpcz_store.cc:276] 0709 12:14:14.954104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:15.474309 (+520205us) service_pool.cc:224] Handling call
0709 12:14:15.474782 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:15.494037 (+ 19255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:15.494083 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:15.516563 (+ 22480us) log.cc:844] Serialized 26 byte log entry
0709 12:14:16.019653 (+503090us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:16.019718 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:16.019828 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:16.443344 (+423516us) raft_consensus.cc:1685] finished
0709 12:14:16.443374 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:16.443553 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2999,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10763,"thread_start_us":364673,"threads_started":1,"wal-append.queue_time_us":562705}
I20260709 12:14:16.337322 18812 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.456339 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.471729 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:16.477818 18542 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.512s	user 0.013s	sys 0.000s
I20260709 12:14:16.478405 18542 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.478907 18812 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.479851 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.480854 18542 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.454591 18856 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.482143 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.483158 12643 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.483574 18856 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.484016 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:16.486272 18515 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.487041 12643 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:16.488250 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.489050 12564 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:16.490545 12571 raft_consensus.cc:1217] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Deduplicated request from leader. Original: 8.5->[10.6-16.9]   Dedup: 10.6->[12.7-16.9]
I20260709 12:14:16.491150 12571 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:14:16.491434 12571 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:14:16.491770 12571 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:16.492136 12571 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:14:16.492383 12571 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:16.493306 18541 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:16.493477 13261 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:16.497620 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:16.473608 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
W20260709 12:14:16.503618 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45989.3us
I20260709 12:14:16.506580 12640 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:16.507313 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.510303 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.510672 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.512101 12567 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:16.512444 12567 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 18.8
I20260709 12:14:16.512673 12567 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:16.516542 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.520844 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.524966 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.385272 18811 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.526597 12646 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:16.527691 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.449849 18872 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:16.524231 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:16.536298 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:16.526377 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67560.7us
I20260709 12:14:16.544196 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.544914 18865 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.545185 18885 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.545822 18883 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.534623 18495 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 12:14:16.546031 18885 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.546638 18883 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.526321 18856 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:16.548653 18856 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:16.549072 18856 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.548681 18885 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.550303 12489 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:16.550566 18883 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.545542 18865 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.554060 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 832) took 1489 ms. Trace:
I20260709 12:14:16.554630 12494 rpcz_store.cc:276] 0709 12:14:15.064928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:15.085964 (+ 21036us) service_pool.cc:224] Handling call
0709 12:14:16.553736 (+1467772us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:16.553933 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2248918400}
W20260709 12:14:16.558490 18876 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:14:16.558835 18737 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.560276 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2899) took 1725 ms. Trace:
I20260709 12:14:16.534786 18629 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.560702 12565 rpcz_store.cc:276] 0709 12:14:14.835006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:15.331985 (+496979us) service_pool.cc:224] Handling call
0709 12:14:15.357196 (+ 25211us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:15.464156 (+106960us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:15.978122 (+513966us) log.cc:844] Serialized 14 byte log entry
0709 12:14:16.365159 (+387037us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:16.379615 (+ 14456us) log.cc:844] Serialized 26 byte log entry
0709 12:14:16.379795 (+   180us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:16.379835 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:16.379913 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:16.559989 (+180076us) raft_consensus.cc:1685] finished
0709 12:14:16.560016 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:16.560203 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50490,"prepare.queue_time_us":621719,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":238197,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":23148,"spinlock_wait_cycles":67968,"thread_start_us":830600,"threads_started":3,"wal-append.queue_time_us":504741}
I20260709 12:14:16.562316 18629 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.563119 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.564276 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.564756 18629 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.537881 18534 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:16.565941 18534 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:16.565470 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.566098 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:16.567725 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 147268us
W20260709 12:14:16.521689 18816 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.539s	user 0.011s	sys 0.000s
I20260709 12:14:16.546635 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:16.573086 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.522673 18516 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.580463 18516 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.571246 12565 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:16.573201 13259 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.581707 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.582949 12497 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:16.569676 12566 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:14:16.585930 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.587019 12488 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:16.590339 18832 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.591296 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.592338 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.592746 18752 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.593124 12564 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:14:16.593961 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.594398 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.594805 12493 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:16.595182 12565 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:16.592394 12490 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:16.596218 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.597155 12565 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:14:16.598443 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:16.603608 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.606153 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.611238 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:16.614262 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.615096 12565 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:16.615506 18885 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:16.615875 18516 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.616269 18885 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:16.616492 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.617373 12565 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:16.575153 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.493981 18541 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.577713 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.571883 18534 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.576126 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:16.497100 18746 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:14:16.596781 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.579535 12494 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:16.622349 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.654618 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:16.656530 12638 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.620291 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.619961 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.658236 12570 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:14:16.657315 13259 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:16.636329 12572 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:14:16.659291 12572 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 12:14:16.523957 18542 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:16.659744 12572 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:16.659885 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:16.619899 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.619501 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.662071 12641 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:16.630816 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:16.657613 12645 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:16.644825 12563 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:16.657294 12564 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:16.661031 12640 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:16.618211 18879 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.230s	user 0.003s	sys 0.000s
I20260709 12:14:16.618947 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.679247 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:16.684754 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:16.685292 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.686142 12641 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.687482 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.688530 12565 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 13.
I20260709 12:14:16.619014 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.683719 12495 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:16.493522 18792 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.655122 12492 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:14:16.693552 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122786us
I20260709 12:14:16.640059 12491 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.696569 12491 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:14:16.700855 12497 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:16.702648 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.693758 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:16.706777 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.712574 12569 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:16.706532 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87662us
I20260709 12:14:16.714119 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.714679 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:16.714824 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:14:16.715848 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.553557 18865 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:16.719098 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.572433 18709 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:16.726584 12641 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.726481 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.727377 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:16.728549 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:16.730149 12489 raft_consensus.cc:606] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:16.730559 12489 raft_consensus.cc:614] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:16.732673 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.733940 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.734135 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.734876 12643 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:16.735752 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.734840 13261 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:16.619077 18515 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:16.735296 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.728098 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.744232 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:16.744416 12639 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:16.734261 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:16.741690 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.741492 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.742585 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.745751 12493 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:16.747135 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.705286 18812 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.749114 12645 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:16.750672 12568 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:14:16.746446 12494 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:16.713253 18495 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:16.695006 18516 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:16.711683 18752 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:16.755699 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.760001 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:16.765280 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.756381 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:16.749467 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.769654 12641 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:16.717900 18603 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:14:16.759618 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 128989us
W20260709 12:14:16.761438 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.640620 18856 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:16.757098 18879 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:14:16.741499 18872 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:16.780630 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:16.741964 18874 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:16.617514 18885 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.814517 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:16.814762 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.763178 18559 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:16.772028 18603 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:16.802392 12489 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
W20260709 12:14:16.812273 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.783090 18834 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:16.755019 18373 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.800695 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.718551 18635 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:16.836263 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.789696 18872 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:16.816969 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:16.800287 18861 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:16.764213 18812 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.742125 18709 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.798357 18874 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:16.821282 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:16.776696 18495 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:16.826781 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:16.832597 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:16.862455 13258 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.790162 18752 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:16.864423 13253 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:16.803367 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:16.865316 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.834172 12645 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 14.
I20260709 12:14:16.841356 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:16.775398 18558 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:16.849913 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.850840 12493 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:16.851174 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.871127 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:16.776065 18516 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:16.856791 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:16.827744 18603 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:16.860626 12641 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:14:16.867776 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.874102 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.876549 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:16.963620 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 270937us
I20260709 12:14:16.906847 18515 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:16.967116 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93082.4us
I20260709 12:14:16.961822 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:16.969185 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.912920 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.838949 18348 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
W20260709 12:14:16.935205 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93949.8us
I20260709 12:14:16.930262 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:16.933888 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:16.903785 12566 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:14:16.902987 18811 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:16.919199 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:16.861101 18779 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:16.855151 18834 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:16.905792 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43606.6us
I20260709 12:14:16.983217 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:16.848147 18861 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:16.847446 18872 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:16.987030 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:16.987677 12489 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:16.972352 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:16.860678 18874 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:16.971794 12493 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:16.976583 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:16.983919 13261 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:16.982051 13255 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:16.985512 12568 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 17. Ops: []
I20260709 12:14:16.963320 18604 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:16.995720 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:16.978024 18598 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.591s	user 0.010s	sys 0.003s
I20260709 12:14:16.962688 18688 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:16.966185 18515 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:16.999640 12536 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:17.018543 12536 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:17.000349 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:17.003504 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 223) took 36031 ms (client timeout 15000 ms). Trace:
I20260709 12:14:16.986682 18861 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:16.982144 18542 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:17.013790 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.014298 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:16.990013 18541 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.021669 12497 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:17.033619 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:16.993301 18887 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:17.056921 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.057103 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.059181 12640 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:17.036082 12491 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:17.061213 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.061753 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:17.061261 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69476.1us
I20260709 12:14:17.062041 12638 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:17.057765 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.017825 18816 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.074652 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.075013 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.013667 18760 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:17.081589 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:17.017702 18598 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.093266 18598 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.057039 18779 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.021521 18495 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.094661 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.095743 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:17.095968 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.068603 18348 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:17.097204 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.097668 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.099030 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:17.099720 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 389558us
I20260709 12:14:17.115298 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:17.116189 18882 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.042258 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:17.091312 18893 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:14:17.103595 12643 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:17.068487 18558 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:17.073189 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:14:17.106184 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.096478 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.111241 12495 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:14:17.069850 18761 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:14:17.064816 18792 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.045384 12641 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:17.026470 18832 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:17.027017 18515 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:17.087067 18373 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.148038 18812 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.103696 18541 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:17.148936 18541 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.119006 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:17.119072 12640 raft_consensus.cc:2376] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20. Re-sending same reply.
I20260709 12:14:17.150632 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:17.150934 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.151466 18541 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.151655 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:17.112910 18760 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:17.152964 18896 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:14:17.154326 18760 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.154915 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.155634 12491 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:17.155710 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.155953 18534 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.157248 12490 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:14:17.158136 18516 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.059734 18542 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.160408 18542 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:17.161995 18892 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
I20260709 12:14:17.163185 18534 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.167124 18861 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.168368 18515 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.077491 18856 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.100493 18746 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:17.179153 18746 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:17.019910 12394 rpcz_store.cc:269] 0709 12:13:40.794134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:40.797874 (+  3740us) service_pool.cc:224] Handling call
0709 12:14:16.825649 (+36027775us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:14:10.839509 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:14:10.920592 (+ 81083us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:14:10.920659 (+    67us) write_op.cc:435] Acquired schema lock
0709 12:14:10.920687 (+    28us) tablet.cc:662] Decoding operations
0709 12:14:11.098071 (+177384us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:14:11.098187 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 12:14:11.098216 (+    29us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:14:11.374284 (+276068us) tablet.cc:701] Row locks acquired
0709 12:14:11.374350 (+    66us) write_op.cc:260] PREPARE: finished
0709 12:14:11.374585 (+   235us) write_op.cc:270] Start()
0709 12:14:11.374790 (+   205us) write_op.cc:276] Timestamp: P: 1783599251374504 usec, L: 0
0709 12:14:11.374832 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 12:14:11.375783 (+   951us) log.cc:844] Serialized 7550 byte log entry
0709 12:14:11.953950 (+578167us) op_driver.cc:464] REPLICATION: finished
0709 12:14:12.297681 (+343731us) write_op.cc:301] APPLY: starting
0709 12:14:12.297893 (+   212us) tablet.cc:1367] starting BulkCheckPresence
0709 12:14:12.831284 (+533391us) tablet.cc:1370] finished BulkCheckPresence
0709 12:14:12.831430 (+   146us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:14:14.598851 (+1767421us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:14:14.599493 (+   642us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0709 12:14:14.599572 (+    79us) write_op.cc:312] APPLY: finished
0709 12:14:14.600474 (+   902us) log.cc:844] Serialized 172 byte log entry
0709 12:14:15.055431 (+454957us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:15.209451 (+154020us) write_op.cc:454] Released schema lock
0709 12:14:15.209609 (+   158us) write_op.cc:341] FINISH: Updating metrics
0709 12:14:15.210343 (+   734us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:15.210379 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":285998,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":264037,"apply.run_cpu_time_us":100860,"apply.run_wall_time_us":2970419,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83471,"num_ops":19,"prepare.queue_time_us":604291,"prepare.run_cpu_time_us":27064,"prepare.run_wall_time_us":733815,"raft.queue_time_us":261624,"raft.run_cpu_time_us":96430,"raft.run_wall_time_us":201614,"replication_time_us":541173,"thread_start_us":1311062,"threads_started":5,"wal-append.queue_time_us":800970,"wal-append.run_cpu_time_us":16352,"wal-append.run_wall_time_us":2400311}]]}
I20260709 12:14:17.180258 18746 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.103040 18456 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:17.181885 18456 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.182380 18456 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:17.183671 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.183910 12639 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:17.134881 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.186098 12493 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:17.135689 12389 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 7 to 16. New cstate: current_term: 16 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:17.190497 12389 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 12 to 18, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 18 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:17.191759 18883 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:17.192518 18603 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:17.139869 18891 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 12:14:17.140210 18737 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.581s	user 0.008s	sys 0.004s
I20260709 12:14:17.118002 18635 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.140897 18348 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:17.202418 18603 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.108583 18904 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:17.122577 18779 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.205415 18904 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:17.205917 18904 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:17.207525 18779 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.142086 18709 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.027169 18688 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.026634 18855 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.108s
I20260709 12:14:17.210117 18688 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.210381 18855 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:17.210816 18855 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:17.212098 12644 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:14:17.211726 18855 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.213001 18855 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:17.214107 18542 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:17.152819 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.215348 18542 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:17.215770 18542 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:16.994449 18752 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.216715 18542 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.133997 18882 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:14:17.217800 18541 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.218433 18531 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:17.218381 18541 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.218859 18903 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:17.124655 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:17.219689 18542 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:17.220280 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95748.8us
I20260709 12:14:17.220932 12638 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 12:14:17.221156 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.221868 18604 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:17.223341 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:17.160329 13258 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:17.139060 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:14:17.076458 18681 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.170725 18861 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.117550 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.126921 18834 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.200068 18559 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.169853 18373 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:14:17.216976 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:17.227249 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.230782 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
W20260709 12:14:17.235146 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 220904us
I20260709 12:14:17.238889 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:17.244632 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.218998 18531 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:17.245283 12570 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
W20260709 12:14:17.247275 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 251628us
I20260709 12:14:17.247498 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:17.251336 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.005s	sys 0.018s
I20260709 12:14:17.243845 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:17.251839 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.005s	sys 0.018s
I20260709 12:14:17.266912 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:17.218457 18882 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:14:17.294577 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 144040us
I20260709 12:14:17.296072 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.260381 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:17.238763 18884 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.241142 18373 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:17.306596 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 242775us
I20260709 12:14:17.246488 18832 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:17.303690 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.000s	sys 0.007s
W20260709 12:14:17.319048 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52265.3us
W20260709 12:14:17.319758 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59487us
W20260709 12:14:17.318605 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.000s	sys 0.008s
I20260709 12:14:17.303289 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:17.292306 18681 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:17.322127 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:17.339821 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:17.294087 18882 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:17.336540 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.317526 18373 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:17.313932 18779 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:17.342288 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:17.350031 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:17.301667 18856 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:14:17.358398 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:17.341816 18603 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:17.342424 18681 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.341470 18737 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:17.344571 18688 raft_consensus.cc:2764] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:14:17.371753 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.373313 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.374779 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.380283 12570 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:17.386718 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47630.8us
I20260709 12:14:17.335417 18884 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.365409 18885 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:17.394618 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:17.399478 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84208.8us
W20260709 12:14:17.400116 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56988.4us
I20260709 12:14:17.343722 18832 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.364382 18760 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:17.366571 18603 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.378726 18681 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:17.404955 18885 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:17.436307 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:17.346817 18907 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:14:17.399000 18869 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:17.387253 18737 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.428133 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:17.428462 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:17.366288 18558 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.432855 12564 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:17.469064 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:17.376622 18752 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:17.437289 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:17.435683 18603 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:17.401609 18709 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:17.230199 12389 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 11 to 15, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 15 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:14:17.462962 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:17.364432 18883 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:14:17.470435 18541 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.472762 18874 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.436769 18792 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.360107 18531 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.477355 18737 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:17.477540 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83569.3us
I20260709 12:14:17.479138 12645 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:17.485261 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.488366 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.476202 18604 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:17.466007 18746 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:17.513268 18746 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.513742 18746 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:17.491266 18373 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.496505 18515 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.495285 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.517187 18373 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.517710 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.518604 12569 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:17.485365 18869 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:17.501719 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 278437us
I20260709 12:14:17.475783 18884 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.504050 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:17.505578 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:17.510974 12564 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:17.510766 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.509217 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.490226 18709 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:17.500100 18760 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:17.464897 18907 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:17.519354 12568 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:17.525852 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.493350 18752 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.527232 18901 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.528061 18897 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:14:17.535310 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.506785 18541 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:17.537313 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.562664 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.564711 12564 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 12:14:17.541965 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:17.491472 18558 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.527565 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:17.547879 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186586us
I20260709 12:14:17.549400 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:17.552364 18912 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:14:17.542649 18760 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.540231 18709 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:17.536484 18515 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:17.444278 18882 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.555038 18832 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.555510 12563 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:17.542687 18922 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:17.557664 18927 raft_consensus.cc:2764] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 12:14:17.558409 18915 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:14:17.559144 18456 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.564749 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.571542 18541 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:17.534607 18920 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:17.575757 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:17.572352 18909 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:14:17.563699 18919 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.581875 18779 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:17.571445 18583 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:17.505867 18792 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.640136 18792 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.607630 18906 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.606925 12389 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 7 to 14. New cstate: current_term: 14 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:17.610380 18874 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.560923 18885 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.613962 18903 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:17.615446 18603 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.621568 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
W20260709 12:14:17.635187 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59520.5us
W20260709 12:14:17.635762 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58420.2us
I20260709 12:14:17.656369 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:17.642465 18812 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:17.645617 18919 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.659983 18919 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.661100 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2884) took 1198 ms. Trace:
I20260709 12:14:17.661474 12567 rpcz_store.cc:276] 0709 12:14:16.462917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:16.471092 (+  8175us) service_pool.cc:224] Handling call
0709 12:14:16.511871 (+ 40779us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:16.513243 (+  1372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:16.513276 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:16.513892 (+   616us) log.cc:844] Serialized 26 byte log entry
0709 12:14:17.252341 (+738449us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:17.252405 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:17.252512 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:17.660875 (+408363us) raft_consensus.cc:1685] finished
0709 12:14:17.660898 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:17.661041 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":737930,"threads_started":1,"wal-append.queue_time_us":879121}
I20260709 12:14:17.662350 18918 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:14:17.664413 18603 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.664335 18737 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.667044 18760 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.669116 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.669791 12572 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:17.671252 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.671730 18737 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.674012 18541 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.674738 18746 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.633003 18922 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:17.678346 18922 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.679142 18558 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:17.682247 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 186577us
I20260709 12:14:17.683290 12572 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:17.684463 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:17.644176 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:17.686455 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.688815 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.690682 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:17.646015 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 813) took 1209 ms. Trace:
I20260709 12:14:17.632916 18920 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.659839 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.601769 18904 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.659406 12389 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 6 to 17. New cstate: current_term: 17 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:17.625126 18515 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:14:17.698220 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:17.697517 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:17.694948 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 236405us
W20260709 12:14:17.696161 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122871us
I20260709 12:14:17.707249 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.681600 18456 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:17.734100 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.688563 18760 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:17.743462 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:17.738221 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.736760 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:17.696699 18872 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.688s	user 0.008s	sys 0.004s
I20260709 12:14:17.692342 18516 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:17.744848 12643 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:17.745954 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:17.746671 12493 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:17.744120 12494 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:17.705542 12637 rpcz_store.cc:276] 0709 12:14:16.424895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:16.481188 (+ 56293us) service_pool.cc:224] Handling call
0709 12:14:16.481553 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:16.482070 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:14:16.689862 (+207792us) log.cc:844] Serialized 14 byte log entry
0709 12:14:17.425912 (+736050us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b18001c3420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:17.633097 (+207185us) log.cc:844] Serialized 14 byte log entry
0709 12:14:17.633471 (+   374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:17.633521 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:17.633569 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:17.633665 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:17.633795 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12049,"prepare.queue_time_us":258566,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":10251,"raft.queue_time_us":37994,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":16832,"spinlock_wait_cycles":6549248,"thread_start_us":1079866,"threads_started":3,"wal-append.queue_time_us":631469}
I20260709 12:14:17.741782 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.750393 12572 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:17.751308 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.692865 18918 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:17.791944 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:17.745352 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:17.693039 18903 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:17.754428 12640 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:17.753605 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63003.3us
I20260709 12:14:17.756347 18930 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.756520 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.756793 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.757335 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:17.752679 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:17.784919 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:17.697099 18906 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:17.705320 18629 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:17.786042 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.792694 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:17.696331 18920 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:17.795516 12565 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:17.798272 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:17.799144 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:17.800786 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73632.2us
I20260709 12:14:17.801822 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:17.740600 18583 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:17.645668 18869 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:17.741245 18792 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:14:17.875131 12490 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:17.875649 12638 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:17.876302 12643 raft_consensus.cc:2368] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
W20260709 12:14:17.814146 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:17.915903 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.916730 12643 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:17.873207 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127988us
I20260709 12:14:17.873864 12567 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 18
W20260709 12:14:17.891819 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90083.2us
W20260709 12:14:17.907549 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154937us
I20260709 12:14:17.919494 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:17.804332 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:17.922910 18752 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:17.922448 12389 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:17.924652 18904 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:17.924738 12389 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 18, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 18 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:14:17.926482 18348 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.724s	user 0.012s	sys 0.000s
I20260709 12:14:17.927013 18348 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:14:17.927224 18937 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:14:17.927485 12389 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 10 to 17. New cstate: current_term: 17 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:17.928633 18938 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:17.929427 18938 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:17.930584 12637 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:14:17.801687 18832 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:14:17.951233 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:17.956656 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:17.760108 18872 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:17.967389 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.972854 12567 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:17.974114 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.909734 18603 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:17.979578 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:17.982465 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 284269us
I20260709 12:14:17.984223 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:17.942777 18760 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:17.986334 12567 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:17.943359 18903 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:17.985029 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:17.958756 18832 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:17.955994 18737 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.958250 18919 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:17.805825 18934 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.186s
I20260709 12:14:17.989580 12642 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15.
I20260709 12:14:17.965900 18918 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:18.023628 18939 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:18.010977 18832 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:18.024340 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:17.931454 18874 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:14:18.029564 12643 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:17.980526 18603 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:18.101867 18603 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:18.102337 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:18.103087 12643 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:18.040982 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:18.106015 13253 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:18.043422 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:17.985962 18906 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:14:18.050629 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:18.028538 18870 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:18.098506 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:18.118363 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.118877 12643 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:17.998737 18746 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:18.045310 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:18.111984 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:17.905860 18930 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.121286 12646 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:18.121574 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.037606 18737 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.125126 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
W20260709 12:14:18.127094 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 142161us
W20260709 12:14:18.129462 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:18.102289 18874 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 14.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 12:14:18.120707 18891 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:14:18.114661 18456 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:18.098955 18939 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:18.122366 18856 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:14:18.140208 12639 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:18.064673 18919 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.140580 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:18.142863 12389 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 11 to 19, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 19 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:14:18.146528 18943 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:14:18.147285 18737 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.149964 18872 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:18.150561 18681 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.696s	user 0.012s	sys 0.000s
I20260709 12:14:18.151647 18681 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.154033 18595 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:18.154507 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.154690 18595 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.155966 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.156775 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:18.141418 12646 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:18.159901 18940 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.356s	user 0.001s	sys 0.000s
I20260709 12:14:18.119939 18903 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:18.163636 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.163856 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.149386 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2756) took 1131 ms. Trace:
I20260709 12:14:18.174600 12495 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:18.165894 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.174863 12640 rpcz_store.cc:276] 0709 12:14:17.018029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.215778 (+197749us) service_pool.cc:224] Handling call
0709 12:14:17.216114 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:17.801126 (+585012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:18.149104 (+347978us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:18.149318 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1237544832}
I20260709 12:14:18.160252 18681 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:18.194535 12494 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:14:18.195648 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.196818 12491 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:18.196998 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.197613 12496 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:18.145223 12389 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:18.199982 18939 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.200824 12389 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 16 to 19, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 19 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:18.205478 18919 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.204458 12389 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:18.208094 12389 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:14:18.209867 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 276473us
I20260709 12:14:18.046125 18938 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:18.160526 18709 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.561s	user 0.009s	sys 0.000s
I20260709 12:14:18.210197 12637 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:14:18.210991 12637 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 9
I20260709 12:14:18.211180 12637 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:14:18.211326 12637 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:18.214473 12536 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17.
I20260709 12:14:18.214828 12536 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:18.216939 12638 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:18.218325 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:18.099189 18918 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.167181 18872 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:18.189459 18927 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:18.217923 18940 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:14:18.230870 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.238528 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.218897 18348 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:18.243199 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.239357 12646 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:18.258224 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.261077 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.239303 18932 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17.
I20260709 12:14:18.218894 18709 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.277964 18709 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:18.279760 18916 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.808s	user 0.005s	sys 0.005s
I20260709 12:14:18.281196 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:18.282603 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:18.283231 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:18.284015 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.288370 12572 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:18.288822 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.263226 18348 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:18.287113 18456 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:18.287642 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:18.250950 18884 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:18.265126 12389 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 9 to 13, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 13 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:18.254969 18870 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:18.261279 18542 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:18.238826 18872 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.312172 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.328330 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.336798 18959 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:18.337726 18515 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:18.338666 18957 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:18.339167 18957 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:18.338866 18870 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.339555 18957 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:18.340814 18595 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:18.335497 18542 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:18.342198 18542 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.342769 18516 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:18.343297 18516 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:18.344437 18516 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.347467 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.349347 18516 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:18.311305 18456 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:18.344333 18542 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:18.245069 18927 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.312335 18884 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:18.362584 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.363885 18920 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.365468 18348 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.322703 18709 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:18.245883 18760 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:18.362424 18972 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:14:18.343134 18930 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.375701 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:14:18.382897 18891 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:14:18.445468 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 207032us
I20260709 12:14:18.445928 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:18.446857 12572 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:18.448101 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:18.269161 18903 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.382300 18884 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:18.387756 18918 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.389258 18872 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:18.448843 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:18.515771 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 227498us
I20260709 12:14:18.451762 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.515976 18760 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.516753 12491 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:18.453171 12567 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:18.453735 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:18.518105 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67547.2us
I20260709 12:14:18.518867 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:18.520712 18956 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:18.521564 18956 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:18.521878 18951 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:18.516207 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67503.6us
W20260709 12:14:18.462273 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:18.463037 18965 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:18.499094 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:18.481722 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 261216us
W20260709 12:14:18.548239 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:18.552352 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.553911 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.556703 12495 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
W20260709 12:14:18.556981 18977 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.077s	user 0.003s	sys 0.000s
W20260709 12:14:18.498459 18887 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.968s	user 0.007s	sys 0.008s
I20260709 12:14:18.560696 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.561126 12495 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:18.522503 18951 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:18.550982 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:18.562541 12495 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:14:18.385100 18832 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.566033 12496 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:14:18.568104 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.795s (SENT)
I20260709 12:14:18.571050 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:18.571542 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.571542 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.572409 12491 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:18.569288 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2747) took 1554 ms. Trace:
I20260709 12:14:18.558955 18542 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:18.560654 18887 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.047s	user 0.007s	sys 0.008s
W20260709 12:14:18.581401 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:18.600235 12637 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:14:18.573455 12641 rpcz_store.cc:276] 0709 12:14:17.015292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.184919 (+169627us) service_pool.cc:224] Handling call
0709 12:14:17.185337 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:17.246291 (+ 60954us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:17.301176 (+ 54885us) log.cc:844] Serialized 14 byte log entry
0709 12:14:18.462450 (+1161274us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b1800367360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:14:18.476966 (+ 14516us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:18.477865 (+   899us) log.cc:844] Serialized 26 byte log entry
0709 12:14:18.478083 (+   218us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:18.563913 (+ 85830us) log.cc:844] Serialized 14 byte log entry
0709 12:14:18.564220 (+   307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:18.564298 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:18.568940 (+  4642us) raft_consensus.cc:1685] finished
0709 12:14:18.568966 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:18.569225 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61402,"prepare.queue_time_us":54970,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":13973,"raft.queue_time_us":125357,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":62112,"spinlock_wait_cycles":2307328,"thread_start_us":99476,"threads_started":2,"wal-append.queue_time_us":1134232}
I20260709 12:14:18.546654 18603 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.563472 18951 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:18.620469 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.621399 12495 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:18.569272 18861 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:18.611339 18960 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:18.612363 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.612788 18920 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:18.604094 18542 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.573671 18595 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:18.646051 18595 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.637274 18960 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.651868 18629 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:18.653241 18960 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.628063 18861 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:18.654238 18861 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:18.654887 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2934) took 1411 ms. Trace:
I20260709 12:14:18.655431 12571 rpcz_store.cc:276] 0709 12:14:17.243329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.243598 (+   269us) service_pool.cc:224] Handling call
0709 12:14:18.654659 (+1411061us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:18.654789 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":402857856}
I20260709 12:14:18.630623 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.633265 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.603461 18963 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:14:18.639276 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.828s (SENT)
I20260709 12:14:18.641553 18603 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.560856 18965 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.663012 12491 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:18.665974 12495 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 4
I20260709 12:14:18.681209 12495 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 14.5
I20260709 12:14:18.681640 12495 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:18.683061 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:18.687469 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119437us
I20260709 12:14:18.688242 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.688675 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:18.689313 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:18.692165 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66702.4us
I20260709 12:14:18.690696 12571 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:18.694162 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.694417 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.695013 12491 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:18.696810 18939 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:18.699617 18456 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.703395 12490 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:14:18.726608 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2742) took 1713 ms (client timeout 1534 ms). Trace:
I20260709 12:14:18.728839 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.736693 12497 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:14:18.730170 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:18.729266 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2915) took 1231 ms. Trace:
W20260709 12:14:18.734709 12645 rpcz_store.cc:269] 0709 12:14:17.013664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.014283 (+   619us) service_pool.cc:224] Handling call
0709 12:14:18.726385 (+1712102us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:18.726516 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":348293504}
I20260709 12:14:18.740075 12569 rpcz_store.cc:276] 0709 12:14:17.498291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.527236 (+ 28945us) service_pool.cc:224] Handling call
0709 12:14:18.714864 (+1187628us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:18.729188 (+ 14324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055840128}
I20260709 12:14:18.731570 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2916) took 1233 ms. Trace:
I20260709 12:14:18.743886 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:18.746783 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4954949330001455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
W20260709 12:14:18.747483 12536 connection.cc:256] RPC call timeout handler was delayed by -1.45499493300008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 12:14:18.748706 12536 connection.cc:256] RPC call timeout handler was delayed by -1.415394933000016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
W20260709 12:14:18.749148 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4090949330000058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
W20260709 12:14:18.749547 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3982949329999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
W20260709 12:14:18.749951 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3334949330001109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 12:14:18.750342 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2038949330001287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18320000000000003s for a response
W20260709 12:14:18.750739 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1840949330000967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1854s for a response
W20260709 12:14:18.751160 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1102949329999774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
W20260709 12:14:18.751543 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0148949330000505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2042s for a response
I20260709 12:14:18.730201 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:18.753362 12496 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:18.755832 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:18.747476 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:18.686290 18963 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:18.743498 12563 rpcz_store.cc:276] 0709 12:14:17.498552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.599793 (+101241us) service_pool.cc:224] Handling call
0709 12:14:17.600277 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:17.600557 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:17.818907 (+218350us) log.cc:844] Serialized 14 byte log entry
0709 12:14:17.819750 (+   843us) log.cc:844] Serialized 14 byte log entry
0709 12:14:18.159311 (+339561us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b18004373e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf96c758f 00007ffbf2706571 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd
0709 12:14:18.730978 (+571667us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b18004373e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf53a9d70 00007ffbf539d293 00007ffbf5392cea 00007ffbf539323a 00007ffbf57fda2a 00007ffbf580d0bf 00007ffbf580d038
0709 12:14:18.731224 (+   246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:18.731272 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:18.731325 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:18.731401 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:18.731512 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93973,"prepare.queue_time_us":180693,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":77,"raft.queue_time_us":2727,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":122,"spinlock_wait_cycles":180247168,"thread_start_us":127951,"threads_started":1}
I20260709 12:14:18.697435 18939 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:18.710503 18917 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.200s	user 0.004s	sys 0.000s
I20260709 12:14:18.703393 18542 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:18.705542 18973 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:18.728462 18927 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:18.776152 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.605871 18956 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.727813 18595 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:18.780784 12566 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:14:18.787019 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.795045 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.797787 18542 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:18.819233 18456 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.820791 12497 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:14:18.821384 18947 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.659s	user 0.003s	sys 0.000s
I20260709 12:14:18.821672 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1062) took 1450 ms. Trace:
I20260709 12:14:18.822125 12565 rpcz_store.cc:276] 0709 12:14:17.371200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.371354 (+   154us) service_pool.cc:224] Handling call
0709 12:14:18.821363 (+1450009us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:18.821555 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":621657984}
I20260709 12:14:18.826956 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:18.832531 12566 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:18.838764 12536 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
W20260709 12:14:18.850448 18910 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.337s	user 0.000s	sys 0.005s
I20260709 12:14:18.785226 18973 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.850317 12566 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:14:18.814507 18542 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
W20260709 12:14:18.870114 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 162742us
I20260709 12:14:18.888475 18927 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:18.890772 18516 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.534s	user 0.012s	sys 0.000s
I20260709 12:14:18.891153 18973 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:18.840484 18957 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.840117 18979 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
W20260709 12:14:18.895365 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56830.3us
I20260709 12:14:18.894894 18541 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:18.920205 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 833) took 2218 ms (client timeout 1795 ms). Trace:
W20260709 12:14:18.920807 12492 rpcz_store.cc:269] 0709 12:14:16.699274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:16.699466 (+   192us) service_pool.cc:224] Handling call
0709 12:14:18.907269 (+2207803us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:18.917830 (+ 10561us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3682210688}
W20260709 12:14:18.922863 18984 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:14:18.933658 18988 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:18.933946 12461 leader_election.cc:400] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:18.937218 12461 leader_election.cc:403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:18.937461 18965 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:18.938484 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.939868 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:18.870899 18939 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.941016 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:18.943246 18595 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:18.944025 18595 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:18.944753 18595 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:18.945565 18970 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:18.946480 18970 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:18.947151 18957 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.947646 18970 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.945330 18595 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:18.857468 18963 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.949693 18595 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.13, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.952706 18541 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:18.953508 18832 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:18.955893 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55510.5us
I20260709 12:14:18.957031 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:18.958922 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:18.960637 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.961777 12639 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:18.962026 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.962779 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:18.963376 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.963595 12641 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:18.963534 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.964332 12637 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 18 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:14:18.964622 12643 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:18.965548 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.967028 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.967110 12643 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:18.967929 12497 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:18.969522 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:18.969739 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.970161 12646 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:18.970274 18541 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:18.971274 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.972502 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.973235 12645 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:18.903772 18927 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.974159 12491 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:18.974812 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:18.975564 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:18.975862 18887 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:18.976397 18927 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.973773 18951 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:18.977366 18760 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:18.977794 18760 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:18.977978 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.978729 12571 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:18.978935 18951 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:18.979617 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:18.965312 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.966150 18516 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:18.969003 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:18.988018 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.989790 12646 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:18.984962 18761 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.057998 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2923) took 1074 ms. Trace:
I20260709 12:14:19.027055 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:18.988432 12641 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:18.969076 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:18.990199 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:19.061535 12641 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:18.994076 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 2924) took 1006 ms. Trace:
I20260709 12:14:19.062497 12567 rpcz_store.cc:276] 0709 12:14:17.987405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:18.117955 (+130550us) service_pool.cc:224] Handling call
0709 12:14:18.993833 (+875878us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:18.994003 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767302912}
W20260709 12:14:19.046823 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 175360us
I20260709 12:14:19.081516 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:18.995055 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.082515 12643 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:19.058353 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:19.050999 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.053865 12497 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:19.058586 12572 rpcz_store.cc:276] 0709 12:14:17.983905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:18.449078 (+465173us) service_pool.cc:224] Handling call
0709 12:14:18.550248 (+101170us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:18.550473 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:18.565288 (+ 14815us) log.cc:844] Serialized 14 byte log entry
0709 12:14:19.057441 (+492153us) log.cc:844] Serialized 14 byte log entry
0709 12:14:19.057678 (+   237us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:19.057733 (+    55us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:19.057777 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:19.057839 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:19.057949 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6028,"prepare.queue_time_us":19201,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":21393,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":484693,"threads_started":3,"wal-append.queue_time_us":212628}
W20260709 12:14:19.086711 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91899.3us
I20260709 12:14:19.005839 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.087875 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.011619 18583 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:19.066272 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97303.6us
I20260709 12:14:18.986991 18907 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:19.103717 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.060001 12637 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:19.092295 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86567.3us
I20260709 12:14:18.989472 18541 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:19.106040 18541 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.106251 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:19.108325 18973 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:19.108657 18541 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.011001 18832 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.109094 18973 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:19.109869 18973 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:19.109760 18832 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:19.112310 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.037364 18603 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:19.111824 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.112972 18603 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:19.113545 18603 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:19.118579 18541 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:19.120525 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.112443 18832 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:19.038371 12645 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:18.991285 18861 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:19.131175 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.009922 18986 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:19.084513 18920 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.079010 18887 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:19.125561 18832 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:14:19.064227 18927 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.047008 18954 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:14:19.095146 18913 raft_consensus.cc:2764] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:14:19.156116 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:18.994084 18516 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.064435 18761 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:19.151538 18984 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:14:19.190537 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70149.4us
I20260709 12:14:19.195662 12646 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:14:19.138234 18869 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:19.137910 18986 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:19.194236 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:19.153487 18988 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:19.062070 18760 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.204231 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:19.127821 18959 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:14:19.127048 18583 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.231161 18913 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:19.201056 18869 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:19.255658 12496 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:14:19.255800 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:19.205897 12645 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:19.260545 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:19.269790 12493 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:14:19.157331 18927 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.276932 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:19.290419 13260 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:19.277607 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.277735 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.275022 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.275375 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.287731 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:19.294133 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:19.294946 12638 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:19.155047 18920 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.260919 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.276602 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:19.298424 13254 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:19.287731 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:19.300278 12493 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:19.274199 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.202700 18986 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:19.269646 18990 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.175s	user 0.000s	sys 0.006s
I20260709 12:14:19.309990 13259 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:19.314245 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50921.3us
I20260709 12:14:19.318081 13258 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:19.270741 18907 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:19.319259 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:19.288885 18884 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.740s	user 0.000s	sys 0.021s
I20260709 12:14:19.383926 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:19.261209 12492 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:19.322708 18760 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:19.365948 18998 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:19.374563 19000 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:14:19.375787 18861 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.272375 18542 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:19.308768 18913 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:19.377944 18963 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.378316 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:19.410738 12563 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:19.379128 18516 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.274466 18988 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.379470 18583 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.380760 18920 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:19.382637 18991 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:14:19.271530 18869 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:19.384709 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.430939 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:19.431188 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:19.385198 12496 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:19.431808 12491 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:19.432255 12572 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
W20260709 12:14:19.390105 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 260374us
I20260709 12:14:19.393600 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:19.434798 12646 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16. Re-sending same reply.
I20260709 12:14:19.428503 12645 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:19.387413 18887 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:19.406373 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 300238us
I20260709 12:14:19.395324 12571 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:19.446233 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.446990 12571 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:14:19.440415 18595 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:14:19.441825 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.446746 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.440999 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
W20260709 12:14:19.433212 19001 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:14:19.458974 12640 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:14:19.393128 18884 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.430413 18761 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.482606 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:19.483587 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.495697 12638 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:14:19.451570 18959 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:19.414429 18603 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.401397 18998 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.504012 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.499887 12640 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:14:19.504905 12641 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:19.500352 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.428632 18963 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.432157 18913 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.521445 12563 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
W20260709 12:14:19.526384 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.511137 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:19.515275 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:19.508585 12638 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:19.497239 12491 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:19.531227 18884 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.533551 18986 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.535686 18761 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 12:14:19.544425 18812 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:19.545014 18812 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:19.534458 13260 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:19.545979 18986 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:19.538769 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92629.3us
W20260709 12:14:19.551050 18942 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.510s	user 0.007s	sys 0.001s
I20260709 12:14:19.554116 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.555429 18927 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:19.543272 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:19.558941 18992 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:14:19.559785 18903 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:19.578327 13254 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:19.540520 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:19.498068 18960 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:14:19.565163 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49983.4us
I20260709 12:14:19.570035 19012 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:19.586915 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46462.9us
I20260709 12:14:19.592387 12571 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:19.596681 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:19.611752 18812 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.618867 18603 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.537242 18907 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:19.620395 18541 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.501s	user 0.013s	sys 0.000s
I20260709 12:14:19.629959 18541 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:19.562639 18887 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.621393 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2925) took 1633 ms. Trace:
I20260709 12:14:19.631223 12565 rpcz_store.cc:276] 0709 12:14:17.987777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:18.823226 (+835449us) service_pool.cc:224] Handling call
0709 12:14:18.823750 (+   524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:18.828021 (+  4271us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:18.837317 (+  9296us) log.cc:844] Serialized 14 byte log entry
0709 12:14:19.061016 (+223699us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:19.062548 (+  1532us) log.cc:844] Serialized 26 byte log entry
0709 12:14:19.062778 (+   230us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:19.062832 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:19.062955 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:19.621117 (+558162us) raft_consensus.cc:1685] finished
0709 12:14:19.621143 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:19.621314 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8079,"prepare.queue_time_us":9784,"raft.queue_time_us":5151,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":17928,"spinlock_wait_cycles":9397120,"thread_start_us":223084,"threads_started":1,"wal-append.queue_time_us":366930}
I20260709 12:14:19.506702 18959 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.601497 18761 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:19.609966 18963 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.615044 18913 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.625003 18903 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.627887 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:19.628293 12567 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:19.598788 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:19.576459 18986 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.542372 19010 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.106s
W20260709 12:14:19.635645 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81784.9us
I20260709 12:14:19.643893 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:19.644608 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.645115 12641 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:19.673030 12638 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:19.646081 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.625595 18884 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:19.657725 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:19.665035 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:19.665443 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66791.1us
I20260709 12:14:19.633502 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:19.675135 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:19.646356 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.677174 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.611788 18998 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.687845 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:19.693905 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:19.704455 12611 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.963s (SENT) [suppressed 5 similar messages]
I20260709 12:14:19.652544 18541 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:19.645332 18920 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:19.706604 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.710232 12640 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 12:14:19.710460 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83821.3us
I20260709 12:14:19.713639 12495 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:19.724244 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:19.683317 18761 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:19.683460 18927 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:19.716404 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:19.728269 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:19.738523 18913 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:19.687733 18903 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.712155 19016 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:19.733551 12640 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:19.703191 18986 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.758026 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:19.743495 13254 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:19.678611 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2928) took 1076 ms. Trace:
I20260709 12:14:19.717752 18541 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:19.731555 18920 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.760392 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:19.765352 18960 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:19.761593 18927 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:19.766641 18946 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:19.766132 18541 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.767179 18963 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:14:19.729298 19028 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:14:19.769176 19022 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:19.789677 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46061.3us
I20260709 12:14:19.804041 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:19.773948 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.802263 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.774583 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:19.769528 18973 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.659s	user 0.008s	sys 0.006s
I20260709 12:14:19.762976 18913 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:19.808609 13260 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:19.761798 18603 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:19.816108 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:19.816187 13258 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:19.771507 12572 rpcz_store.cc:276] 0709 12:14:18.602641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.444944 (+842303us) service_pool.cc:224] Handling call
0709 12:14:19.478177 (+ 33233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:19.678100 (+199923us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:19.678148 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:19.678214 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:19.678259 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:19.678359 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:19.678542 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":98075,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":7751,"thread_start_us":199162,"threads_started":1}
I20260709 12:14:19.769852 18761 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:19.823053 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120238us
I20260709 12:14:19.770278 18884 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:14:19.802208 18946 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:19.830535 19026 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:19.830816 19025 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:19.817495 18913 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:19.826375 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132595us
I20260709 12:14:19.762782 18887 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:14:19.825554 18907 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:19.827677 18963 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:19.802948 18595 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:14:19.828536 19012 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:14:19.828156 18906 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:19.823422 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:19.791688 18920 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.831535 18516 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.843905 18913 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:19.850349 18516 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.850801 18946 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 13.7, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.851912 18903 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.852596 18903 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:19.854115 19029 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:19.855026 19029 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:19.861045 18920 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.816251 18973 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.864033 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.846324 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:14:19.870577 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97136.6us
I20260709 12:14:19.873059 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:19.844475 18907 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:19.856266 19029 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.850696 18986 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.846733 18963 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:19.900861 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55487.2us
I20260709 12:14:19.910759 12572 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:14:19.910916 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:19.897992 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67462.1us
I20260709 12:14:19.900329 18970 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:19.892620 12563 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:19.867866 18920 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:19.863523 18812 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.821900 19022 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.915482 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:19.929716 13260 raft_consensus.cc:2376] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16. Re-sending same reply.
I20260709 12:14:19.853071 18516 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.862912 18906 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:19.925946 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:19.914757 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:19.891868 18907 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:19.939532 12491 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:19.941672 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3087) took 1778 ms. Trace:
I20260709 12:14:19.894374 19012 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:19.915987 19010 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:19.948488 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:19.869094 19027 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
W20260709 12:14:19.947685 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:19.945159 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:19.943457 12488 rpcz_store.cc:276] 0709 12:14:18.162934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:18.191456 (+ 28522us) service_pool.cc:224] Handling call
0709 12:14:18.191791 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:18.985455 (+793664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:18.985490 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:18.985539 (+    49us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:18.985578 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:19.941347 (+955769us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:19.941405 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:19.941574 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1824685824}
I20260709 12:14:19.916327 18970 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:19.885283 19028 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:14:19.929363 18986 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:19.957785 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:19.937086 18959 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:19.959434 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:19.941541 18906 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:19.961939 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:19.862522 18927 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.956979 18903 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:19.909219 19025 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 12:14:19.977198 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:19.924554 18761 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.966486 19027 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:19.978546 12491 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:19.974220 18903 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:19.982860 18903 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:19.985519 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72697.4us
W20260709 12:14:19.986779 19033 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:14:19.954655 19010 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.991479 19043 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:19.992153 19043 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:19.992393 18959 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:19.992553 19043 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:19.992976 18959 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:19.966166 19034 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:14:19.995223 19032 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.153s	user 0.000s	sys 0.002s
I20260709 12:14:19.964258 19039 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:19.957751 19040 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:14:20.015218 19048 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:20.015831 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2931) took 2979 ms. Trace:
I20260709 12:14:19.965783 18986 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:20.019028 18973 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:19.981881 19027 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:20.033921 19027 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.035406 19027 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:20.037076 18869 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.586s	user 0.006s	sys 0.005s
I20260709 12:14:20.021813 19022 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:20.010643 18963 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 2, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.981513 18970 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:19.977062 19021 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:20.026048 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:19.979156 18913 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.042826 18913 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.026827 19039 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:20.043960 18595 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:20.044572 18595 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:20.044216 19039 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.027230 18882 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:20.055576 18970 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:20.028904 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.028505 12568 rpcz_store.cc:276] 0709 12:14:17.029610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.030042 (+   432us) service_pool.cc:224] Handling call
0709 12:14:17.030352 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:17.546423 (+516071us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.008652 (+2462229us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:20.008879 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1124034816}
I20260709 12:14:20.031638 18959 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.032629 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.039969 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:20.061563 12645 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:20.062680 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.063829 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.056870 19021 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.065503 19043 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.066583 19021 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.047133 19010 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.064421 12645 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:20.059834 12491 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:20.073431 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1072) took 1420 ms. Trace:
W20260709 12:14:20.081822 18760 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.003s
I20260709 12:14:20.082758 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.083776 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.084375 12646 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:20.085292 12638 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:14:20.030277 18986 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:20.075654 12571 rpcz_store.cc:276] 0709 12:14:18.652615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.488635 (+836020us) service_pool.cc:224] Handling call
0709 12:14:19.489011 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:19.593027 (+104016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.073096 (+480069us) spinlock_profiling.cc:243] Waited 84 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:20.073350 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193158528}
I20260709 12:14:20.074613 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.089068 12645 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:20.028335 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66501.7us
I20260709 12:14:20.117087 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:20.117866 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:20.037608 18869 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.136103 18869 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:20.142213 12639 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.6-17.6]
W20260709 12:14:20.143896 18926 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.629s	user 0.004s	sys 0.000s
I20260709 12:14:20.148366 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2755) took 3131 ms. Trace:
I20260709 12:14:20.123070 19028 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.086694 18760 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:20.148888 12644 rpcz_store.cc:276] 0709 12:14:17.017739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.214165 (+196426us) service_pool.cc:224] Handling call
0709 12:14:17.214583 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:17.316547 (+101964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:17.316592 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:17.342322 (+ 25730us) log.cc:844] Serialized 26 byte log entry
0709 12:14:17.441662 (+ 99340us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:17.441715 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:17.441799 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:18.719455 (+1277656us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:14:20.143062 (+1423607us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:20.147986 (+  4924us) raft_consensus.cc:1685] finished
0709 12:14:20.148025 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:20.148245 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90143,"raft.queue_time_us":99192,"raft.run_cpu_time_us":11460,"raft.run_wall_time_us":1403232,"spinlock_wait_cycles":74986112,"thread_start_us":55705,"threads_started":1,"wal-append.queue_time_us":147090}
W20260709 12:14:20.169502 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 195688us
W20260709 12:14:20.170680 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132667us
I20260709 12:14:20.167500 19043 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:20.126943 18595 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.164017 19028 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:20.180606 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 206615us
I20260709 12:14:20.165300 18760 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:20.195284 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:20.199753 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.201483 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.203051 19028 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:20.205147 18973 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:20.205574 12646 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [18.10-19.11]
I20260709 12:14:20.205662 18973 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:20.206451 18973 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:20.209769 18760 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.194819 18874 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.210942 18874 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.207506 18986 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.214000 19040 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.214228 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "256d044fd2e244d4be827104688867ee"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:20.214453 18986 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.179490 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.218686 12644 raft_consensus.cc:1217] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-19.11]   Dedup: 17.10->[19.11-19.11]
I20260709 12:14:20.219545 12644 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:14:20.221091 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.194698 18913 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.223313 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:20.194546 18883 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.684s	user 0.016s	sys 0.021s
I20260709 12:14:20.223552 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:20.225476 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.232327 18752 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:14:20.235656 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63659us
I20260709 12:14:20.239742 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.240140 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.240772 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:20.241621 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:20.246502 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.247988 12490 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:20.214633 19040 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.249442 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:20.250092 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:20.236479 18883 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.729s	user 0.016s	sys 0.022s
I20260709 12:14:20.250357 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.252346 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.274972 12568 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:20.251824 12490 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:20.263185 19017 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.278108 19017 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.267055 19050 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:14:20.267663 19053 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:14:20.245587 18874 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.280967 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.283675 12491 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:20.276995 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.267397 18901 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.195241 18903 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.251225 18959 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 12:14:20.269222 18752 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:20.214263 18907 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.269892 19040 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.237782 19043 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:14:20.236848 18927 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:20.252230 12492 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:20.270838 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:20.251820 18973 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.272256 19047 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:20.340762 19047 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.342478 13260 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:14:20.275195 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.342872 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2935) took 1664 ms. Trace:
I20260709 12:14:20.277004 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:20.343712 12493 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:20.343046 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:20.277427 19054 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:14:20.292812 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.299015 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.236020 18986 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:14:20.290285 19025 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:20.337523 18907 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.337673 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.238941 18913 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.350701 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.350939 12572 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
W20260709 12:14:20.339372 19055 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:14:20.340003 18973 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.343392 12563 rpcz_store.cc:276] 0709 12:14:18.678301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.524900 (+846599us) service_pool.cc:224] Handling call
0709 12:14:19.525402 (+   502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:19.917125 (+391723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.342532 (+425407us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:20.342796 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843780992}
I20260709 12:14:20.247905 18541 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 15.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 12:14:20.331802 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:20.348681 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73624.5us
I20260709 12:14:20.353183 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.294064 18752 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:20.426874 18752 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.428308 18752 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:20.330857 18869 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:20.429414 18869 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.431804 18869 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.432022 19042 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:20.432358 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.432775 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.433594 13258 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:20.433925 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.434811 18816 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:20.435552 18816 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:20.436753 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.438616 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.320237 18959 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:20.360626 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.362387 19058 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.320000 18760 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:20.420106 12568 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:14:20.422580 12491 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:20.293793 18688 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:14:20.425889 12644 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:20.447053 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.447512 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:20.449260 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:20.454988 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:20.462417 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:20.463578 12568 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:14:20.474331 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:20.456424 12563 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 12:14:20.475885 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:20.455519 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:20.465791 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:20.485675 12566 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:20.502879 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1069) took 2041 ms. Trace:
I20260709 12:14:20.503206 12566 rpcz_store.cc:276] 0709 12:14:18.461198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:18.858153 (+396955us) service_pool.cc:224] Handling call
0709 12:14:20.485149 (+1626996us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:20.485421 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:20.502623 (+ 17202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.502819 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14177152}
I20260709 12:14:20.485675 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.453784 18865 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:14:20.316977 18901 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.490391 18936 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:20.432911 19010 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:14:20.526194 12489 raft_consensus.cc:606] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:20.526868 12489 raft_consensus.cc:614] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:20.489919 18907 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:20.485100 18541 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:20.495869 18903 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.499595 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:20.546425 12492 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:20.438038 18921 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.505508 12491 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 5.
I20260709 12:14:20.478652 18752 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:20.507548 18792 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:20.544584 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70382.9us
I20260709 12:14:20.550285 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:20.553488 18752 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:20.553743 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.524320 18901 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.554652 18752 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 17.6, Last appended by leader: 6, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.554838 18921 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.556041 18752 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:20.559852 12637 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:14:20.559264 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.460569 18760 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.565778 19052 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:20.566390 19052 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:20.566761 19052 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:20.570734 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.572985 19017 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.573843 18921 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.574198 18760 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.542336 18907 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:20.589998 18907 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:20.591073 18913 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.594062 18989 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 12:14:20.576117 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.595958 12491 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:20.578203 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.549336 19025 raft_consensus.cc:2764] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:20.457535 19058 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.582587 18986 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:20.580889 19027 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:20.560470 18973 raft_consensus.cc:2764] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:14:20.559963 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.648670 12493 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:20.596692 12569 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:20.597762 12641 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:20.600258 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.558499 18938 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:14:20.560719 18792 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.579252 19047 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:20.646338 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2761) took 2866 ms (client timeout 2168 ms). Trace:
W20260709 12:14:20.663177 12642 rpcz_store.cc:269] 0709 12:14:17.733612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.733812 (+   200us) service_pool.cc:224] Handling call
0709 12:14:20.597265 (+2863453us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:20.600215 (+  2950us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":569902848}
I20260709 12:14:20.581565 18516 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.650362 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2926) took 2080 ms. Trace:
I20260709 12:14:20.664577 12569 rpcz_store.cc:276] 0709 12:14:18.570005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:18.886368 (+316363us) service_pool.cc:224] Handling call
0709 12:14:20.596286 (+1709918us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:20.596496 (+   210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:20.649871 (+ 53375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.650271 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14736640}
W20260709 12:14:20.651628 19035 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.724s	user 0.006s	sys 0.000s
I20260709 12:14:20.656757 12491 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:14:20.657109 18906 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.687s	user 0.012s	sys 0.000s
I20260709 12:14:20.677572 18906 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.646169 18913 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:14:20.659622 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59494.3us
I20260709 12:14:20.585752 18993 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
I20260709 12:14:20.666576 18531 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:20.662766 19042 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:20.677591 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.686820 12493 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:14:20.670500 18938 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:20.674727 18792 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.679363 12496 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:20.680929 12637 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:14:20.683367 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.684059 18973 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:14:20.696797 18973 consensus_queue.cc:1243] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:14:20.701494 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.688419 18913 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:14:20.702602 18752 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:20.703197 18752 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:20.703586 18752 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:20.693392 19058 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:20.695752 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.705341 12491 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:14:20.676707 19047 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:20.706022 19047 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.707577 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.708391 19047 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:20.664583 18516 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.709718 18973 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:20.711298 18516 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.688766 18938 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:20.714620 18907 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.714519 18938 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.715677 18938 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:20.716856 18907 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.722622 18906 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.724598 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.725351 12563 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:20.726272 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.727736 12569 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:14:20.732339 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:20.732630 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.713388 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:20.740183 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:20.739868 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.741601 12640 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:20.746340 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:20.742175 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.753542 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.758015 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.759093 13258 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:20.763890 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:20.765236 13256 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:20.766004 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2963) took 1140 ms. Trace:
I20260709 12:14:20.769176 12567 rpcz_store.cc:276] 0709 12:14:19.626063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.686276 (+ 60213us) service_pool.cc:224] Handling call
0709 12:14:19.686747 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:19.818812 (+132065us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:19.818855 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:19.827280 (+  8425us) log.cc:844] Serialized 26 byte log entry
0709 12:14:20.087349 (+260069us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:20.087413 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.721238 (+633825us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:20.721322 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:20.721410 (+    88us) raft_consensus.cc:1685] finished
0709 12:14:20.721433 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:20.765874 (+ 44441us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9352,"raft.queue_time_us":30645,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8061,"spinlock_wait_cycles":271372544,"thread_start_us":255711,"threads_started":1,"wal-append.queue_time_us":339790}
I20260709 12:14:20.774721 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.768392 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:20.780233 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:20.781469 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.787572 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.778445 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.790230 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:20.749827 18932 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:20.756652 18956 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.792419 18816 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:20.818156 18816 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:20.818689 18816 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:20.807847 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:20.779673 18542 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:20.803625 18874 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:20.823233 18874 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:20.824471 18874 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.792696 18913 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:20.812824 19042 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.829147 19042 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.829787 18956 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.830860 18816 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.832590 19042 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.813987 12564 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:14:20.814144 12567 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:14:20.835400 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2914) took 3337 ms. Trace:
I20260709 12:14:20.836279 18348 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.783895 19034 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:20.837057 18348 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.839902 12567 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:14:20.836019 12564 rpcz_store.cc:276] 0709 12:14:17.498037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.565581 (+ 67544us) service_pool.cc:224] Handling call
0709 12:14:17.567955 (+  2374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:20.833967 (+3266012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:20.835044 (+  1077us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:20.835321 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7464163712}
I20260709 12:14:20.848414 18604 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:20.790445 18973 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:20.833381 18816 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.821364 18542 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:20.850445 18973 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:20.850817 18542 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:20.851080 18973 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.857980 12563 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:14:20.859889 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:20.860771 12639 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:20.799291 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.848263 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:20.870570 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71401.8us
I20260709 12:14:20.873864 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.874459 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:20.875547 12645 raft_consensus.cc:1217] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-19.11]   Dedup: 17.10->[19.11-19.11]
I20260709 12:14:20.876125 12645 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:14:20.877823 18903 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:14:20.879005 18903 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:20.879781 18903 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:20.880159 18903 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:20.881142 18903 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:20.883725 18855 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:20.795260 18541 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:20.885574 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.838025 19034 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:20.891769 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.893710 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:20.887928 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:20.899390 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:20.887090 12564 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:20.832729 18956 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:20.902925 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:20.861294 18348 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:20.797775 18907 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 2, Time since last communication: 0.103s
I20260709 12:14:20.910178 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:20.884932 19021 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.065s	sys 0.093s
I20260709 12:14:20.813333 18752 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:20.887650 18930 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:20.913340 19021 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.065s	sys 0.094s
I20260709 12:14:20.891180 18855 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:20.914558 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:20.896152 19034 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:20.894970 18986 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:20.884498 19004 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:14:20.899205 18921 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:20.893929 18927 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:20.915243 18956 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:20.889227 18911 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:14:20.903043 18903 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:20.945691 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:20.997478 12489 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:20.946833 19004 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:20.950738 19016 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:20.909760 18792 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:20.921115 18901 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:20.935137 18348 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:21.009656 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:21.008987 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:20.940827 18541 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:20.996510 18855 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:20.951596 18921 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:21.007125 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:21.010562 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.013048 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:21.030282 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:21.014110 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:21.031555 12643 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:21.016178 12642 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [18.10-19.11]
I20260709 12:14:21.030355 12640 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:21.034960 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:21.041328 19042 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:21.046519 12489 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:14:21.049952 19025 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:21.052703 18913 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:21.055351 19060 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.391s	user 0.000s	sys 0.002s
W20260709 12:14:21.056628 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 2965) took 1430 ms (client timeout 1858 ms). Trace:
I20260709 12:14:21.057920 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:21.064962 18963 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:21.061774 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:21.071908 12637 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:14:21.021452 18348 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:21.072669 18348 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:21.074497 18925 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:21.074926 18925 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:21.075845 18925 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.077121 18925 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:21.078256 19052 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.006s
I20260709 12:14:21.078753 19052 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.079258 19050 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:21.081172 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:21.019559 12493 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:14:21.084475 12638 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:21.025043 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.085091 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:21.021698 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.002377 19004 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:21.082340 13260 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:21.073554 12637 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [12.6-14.7]
I20260709 12:14:20.996788 18956 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:21.093773 18956 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:21.095144 19004 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.096658 19004 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:21.057090 12565 rpcz_store.cc:269] 0709 12:14:19.626641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.633165 (+  6524us) service_pool.cc:224] Handling call
0709 12:14:21.056390 (+1423225us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:21.056555 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1239223168}
I20260709 12:14:21.089605 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:21.099449 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2917) took 3576 ms. Trace:
I20260709 12:14:21.029894 18792 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:21.100226 12570 rpcz_store.cc:276] 0709 12:14:17.522879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:17.648732 (+125853us) service_pool.cc:224] Handling call
0709 12:14:17.649375 (+   643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:17.649767 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:18.026376 (+376609us) log.cc:844] Serialized 14 byte log entry
0709 12:14:18.226869 (+200493us) log.cc:844] Serialized 14 byte log entry
0709 12:14:18.242543 (+ 15674us) log.cc:844] Serialized 14 byte log entry
0709 12:14:18.718578 (+476035us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x7b1800416b00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:20.885736 (+2167158us) spinlock_profiling.cc:243] Waited 8.08 ms on lock 0x7b1800416b00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:14:21.098359 (+212623us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:21.098447 (+    88us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:21.098520 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:21.098766 (+   246us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:21.099365 (+   599us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":243912,"prepare.queue_time_us":165631,"prepare.run_cpu_time_us":172,"prepare.run_wall_time_us":2323,"raft.queue_time_us":212395,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":14810,"spinlock_wait_cycles":40373376,"thread_start_us":548184,"threads_started":2}
I20260709 12:14:21.003859 19016 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:21.037663 18921 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:21.132995 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:21.133996 12489 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:21.100791 18965 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:21.088125 12642 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:21.037473 18855 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.012132 18973 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.115850 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.209839 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:21.109193 12643 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:21.210913 12568 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:21.017554 12639 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:21.115630 18913 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:21.178314 19016 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.190995 19052 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:21.205891 19050 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.126371 18531 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:21.128520 18920 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:21.124830 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 236996us
I20260709 12:14:21.100925 18884 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 12:14:21.211916 19072 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:21.221895 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.222601 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:21.213070 18907 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.247375 12489 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:21.258806 19081 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:21.259347 19081 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:21.247605 18956 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.260380 19081 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.226156 18531 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.263553 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2790) took 1444 ms. Trace:
I20260709 12:14:21.267927 12641 rpcz_store.cc:276] 0709 12:14:19.819197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.819350 (+   153us) service_pool.cc:224] Handling call
0709 12:14:20.597487 (+778137us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:20.655442 (+ 57955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:21.263230 (+607788us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b54002c6850. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:21.263458 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1750817024}
I20260709 12:14:21.269940 18956 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:21.270339 18855 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.239032 18907 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.272822 18907 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.230466 19072 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.280377 19072 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.239414 19016 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:21.223851 19050 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.283558 19050 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.266641 19081 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:21.230563 18792 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.300593 18958 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:21.300720 18920 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:21.301394 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:21.302654 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:21.303880 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.304315 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:21.305768 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:21.306239 18913 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:21.306758 18913 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:21.308936 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:21.318145 12571 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:21.301491 18920 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:21.320434 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 426784us
I20260709 12:14:21.323161 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.315543 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:21.327384 12489 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:21.343748 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:21.359834 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:21.360909 12489 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:21.361349 12489 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:21.339779 18931 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:21.382119 12488 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:21.348559 18635 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:14:21.370653 19050 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.301492 18958 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.388689 18635 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:21.374964 18921 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.373714 18931 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:21.391183 18531 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.391960 18348 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:21.400686 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:21.379766 18542 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.005s
I20260709 12:14:21.376515 18936 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:21.412168 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:21.412952 12488 raft_consensus.cc:606] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:21.413345 12488 raft_consensus.cc:614] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:21.368636 19034 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.416083 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.416016 13255 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
W20260709 12:14:21.412792 19079 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.087s	user 0.003s	sys 0.000s
W20260709 12:14:21.438119 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:21.483155 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:21.472813 19080 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:14:21.473281 18921 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:21.473697 19078 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:14:21.415383 18542 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.406278 18635 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:21.574136 18583 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.578316 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2478) took 2022 ms. Trace:
I20260709 12:14:21.581378 13262 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:14:21.406833 19050 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.583768 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 936) took 1462 ms. Trace:
I20260709 12:14:21.579620 12495 rpcz_store.cc:276] 0709 12:14:19.556168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:19.556374 (+   206us) service_pool.cc:224] Handling call
0709 12:14:19.556747 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:19.727902 (+171155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:21.577853 (+1849951us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:21.578215 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360046464}
I20260709 12:14:21.585800 13261 rpcz_store.cc:276] 0709 12:14:20.111854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:20.112173 (+   319us) service_pool.cc:224] Handling call
0709 12:14:20.112507 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:20.112888 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:20.112917 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:20.121389 (+  8472us) log.cc:844] Serialized 26 byte log entry
0709 12:14:21.071409 (+950020us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b180024c7c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:21.336542 (+265133us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:21.336604 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:21.336702 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:21.477837 (+141135us) raft_consensus.cc:1685] finished
0709 12:14:21.477875 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:21.574240 (+ 96365us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14583,"raft.queue_time_us":633,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":5166,"spinlock_wait_cycles":30494976,"thread_start_us":264798,"threads_started":1,"wal-append.queue_time_us":1262942}
I20260709 12:14:21.477517 19040 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:21.585348 18963 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.520s	user 0.016s	sys 0.000s
I20260709 12:14:21.579031 18583 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.414880 18936 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.417770 18931 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:21.610909 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.607766 18531 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.620384 19004 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:21.637734 18936 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.638303 19004 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.641551 19034 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.604789 19040 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.617373 18635 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:21.661244 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.684206 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:21.719033 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.637734 19050 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.747175 18973 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.747201 18635 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:21.751952 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.644659 18531 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.746447 18936 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.747695 18542 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.747509 18920 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:21.767148 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.677500 19004 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.771129 18348 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.788581 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:21.789772 12568 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:21.765836 18583 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.755172 18973 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.804392 12565 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:21.804557 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.805198 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.805827 12567 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:21.806008 12564 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:21.806538 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.808820 12565 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:14:21.810580 18348 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:21.810910 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.811249 18348 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.811970 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.813278 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.814028 12566 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 13.
I20260709 12:14:21.807078 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:21.815333 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.816089 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:21.817886 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:21.818622 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.818770 12495 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:21.820326 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.765558 18936 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.823076 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:21.826087 12460 connection.cc:256] RPC call timeout handler was delayed by -1.9245189910000136s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
I20260709 12:14:21.826539 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:21.827266 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:21.828027 19072 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.829288 19072 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.833765 19072 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.816792 18920 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:21.839432 19081 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.004s
I20260709 12:14:21.840091 18884 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.840628 18884 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.842346 12570 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:14:21.843030 18595 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:21.847553 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.848690 12488 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 13.
I20260709 12:14:21.850699 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:21.851123 18936 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.855772 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:21.856436 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:21.858243 12489 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:21.858582 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:21.859347 18541 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:21.859517 12488 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:21.859738 18541 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:21.865015 18541 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:21.845420 19040 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.868458 19040 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:21.869021 19040 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:21.856587 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:21.817523 18348 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.860787 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:21.860937 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.861924 12497 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:14:21.862143 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.862986 12489 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:14:21.863443 18958 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:21.863270 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:21.756747 18913 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.870383 13254 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:21.869594 19040 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:21.896699 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:21.897547 12497 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:21.871111 18973 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.878588 12492 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:21.879240 12490 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:21.881762 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.884995 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.885236 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:21.906359 12489 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:21.882038 19081 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:21.881104 19010 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:14:21.910158 19081 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:21.911092 19010 consensus_queue.cc:1243] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:14:21.880673 18936 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:21.912217 18936 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:21.911815 19081 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.913146 18936 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.916520 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:21.886479 18913 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.917474 12568 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:21.906518 12488 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:21.911646 19072 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:21.926937 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:21.928341 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:21.920902 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:21.929863 12568 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:14:21.933785 13254 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:21.948030 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:21.957378 12563 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:21.957852 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:21.950770 18752 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:21.940109 19009 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:21.961500 13260 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:21.961701 19009 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:21.934867 18583 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:21.970376 18913 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:21.971350 18913 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:21.955513 18541 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:21.986400 19039 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.291s	user 0.016s	sys 0.028s
W20260709 12:14:21.987020 19039 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.292s	user 0.016s	sys 0.028s
W20260709 12:14:21.989271 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:21.991050 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2484) took 1965 ms. Trace:
I20260709 12:14:21.991851 12496 rpcz_store.cc:276] 0709 12:14:20.025869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:20.028323 (+  2454us) service_pool.cc:224] Handling call
0709 12:14:20.034264 (+  5941us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:20.691171 (+656907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:21.990648 (+1299477us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:21.990942 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1483090304}
I20260709 12:14:21.996745 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:21.997887 18973 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:21.998422 18973 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.000497 18348 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:22.002887 18920 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:22.004094 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:22.053316 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49354.5us
I20260709 12:14:22.001546 18348 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:22.036602 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
W20260709 12:14:22.063000 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 179890us
I20260709 12:14:22.063112 12491 raft_consensus.cc:606] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:22.063683 13218 connection.cc:256] RPC call timeout handler was delayed by -2.1430246729999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
W20260709 12:14:22.064268 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120734us
W20260709 12:14:22.064844 13218 connection.cc:256] RPC call timeout handler was delayed by -2.1268246729998737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
I20260709 12:14:22.064821 12491 raft_consensus.cc:614] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:22.066452 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:22.063817 12495 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:22.040385 18884 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.069801 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:22.010062 19009 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:21.924155 19072 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:22.044893 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:22.050633 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.051092 12571 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:14:22.058094 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:22.070631 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:22.073673 12568 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:21.944605 19010 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.125s
I20260709 12:14:22.075843 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:22.076810 19072 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.078374 19072 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:22.080065 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.080775 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.081058 12568 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:22.082108 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.069159 19027 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:22.046451 18541 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:22.087571 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:22.068361 18752 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:22.095060 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:22.087987 18884 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:22.087769 19050 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:22.096539 19081 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:22.097535 19050 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:22.098608 18861 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:22.098744 19050 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.088500 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:22.101879 18973 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.085752 18920 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:22.087129 18816 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.096653 18884 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.104404 18884 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:22.108891 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:22.109730 18970 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:22.110846 18970 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:22.111153 18970 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:22.124804 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.125882 12568 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:14:22.122259 12564 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:22.086717 19027 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.094431 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:22.128134 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:22.128698 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.099421 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:22.103930 18816 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.111492 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:22.112071 18920 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:14:22.046226 18913 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:22.114128 18542 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:14:22.116580 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.140357 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:22.140529 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:22.141098 12495 raft_consensus.cc:606] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:22.092334 12563 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
W20260709 12:14:22.121820 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.142140 12563 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:14:22.142478 12563 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:14:22.142740 12563 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:22.122869 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.085956 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.128249 12496 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:22.094462 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.145875 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 14.
I20260709 12:14:22.132792 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:22.134934 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:22.147239 13260 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:14:22.147675 13260 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 13
I20260709 12:14:22.147902 13260 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:14:22.148151 13260 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:22.131534 19065 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.014s	sys 0.011s
I20260709 12:14:22.144814 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
W20260709 12:14:22.150611 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65014.1us
I20260709 12:14:22.143477 18595 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 12:14:22.151326 13253 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:22.158479 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:22.153874 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.163566 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:22.161722 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:22.164969 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.165406 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:22.168255 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.169569 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:22.171550 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:14:22.144210 18973 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:22.174007 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:14:22.152246 19065 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.015s	sys 0.011s
I20260709 12:14:22.204784 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:22.143779 18542 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:22.148839 18688 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:22.152201 18988 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:22.156174 18541 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:22.215548 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.217398 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:22.151707 18883 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.061s
I20260709 12:14:22.196048 18913 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:22.160967 18760 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.202167 19081 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:22.172751 18958 raft_consensus.cc:2764] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:14:22.234202 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.230752 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:14:22.338603 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.149331 18887 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:14:22.208230 18973 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.207963 18855 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:14:22.161013 18994 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 12:14:22.211227 18925 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:22.206708 18348 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:22.171682 19010 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 12:14:22.209409 19030 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.073s
W20260709 12:14:22.351266 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117191us
I20260709 12:14:22.338030 19081 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.232717 18595 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:22.218870 18541 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:22.227342 18761 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:14:22.263722 19052 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.964s	user 0.020s	sys 0.028s
I20260709 12:14:22.167850 18948 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.182s
I20260709 12:14:22.167599 18931 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.175s
I20260709 12:14:22.389199 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:22.235352 18583 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:22.350575 18920 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.369057 18973 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:22.406917 18348 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:22.407963 19098 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:14:22.407766 18348 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.367017 18855 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.211089 19004 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.175s
I20260709 12:14:22.367285 18959 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:22.363201 18913 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:22.366619 18635 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.393446 19081 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:22.396773 18541 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:22.379891 18925 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.405332 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.341501 18760 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:22.400787 19052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.097s	user 0.021s	sys 0.029s
I20260709 12:14:22.432194 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.410530 18595 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:22.410360 19030 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.411919 18855 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:22.385825 18542 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.428965 18925 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:22.481223 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.485357 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.443421 19010 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:22.443532 18959 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:22.491581 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:22.407050 18920 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.501986 18761 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:22.450649 18988 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:22.512225 18988 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.491869 19027 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:22.515208 18913 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.491164 18959 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.518119 18959 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:22.490875 18542 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:22.524573 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2526) took 1467 ms. Trace:
I20260709 12:14:22.525750 19083 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:22.525107 12645 rpcz_store.cc:276] 0709 12:14:21.057207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:21.065849 (+  8642us) service_pool.cc:224] Handling call
0709 12:14:21.066106 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:21.066276 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:21.218916 (+152640us) log.cc:844] Serialized 14 byte log entry
0709 12:14:22.524090 (+1305174us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:22.524170 (+    80us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:22.524235 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:22.524324 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:22.524493 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1304742,"prepare.queue_time_us":2094,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":152178,"threads_started":1}
W20260709 12:14:22.526335 19100 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:14:22.526475 19083 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.527118 19021 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:22.527650 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.20433e+06us
W20260709 12:14:22.528409 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.30602e+06us
W20260709 12:14:22.528685 19104 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:14:22.529084 19083 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:22.529420 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.530398 18904 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:22.530853 18904 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.531232 18920 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.531751 18921 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:22.532145 18779 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:22.533699 18985 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:22.534363 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.527755 19021 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.536734 19021 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:22.537106 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:22.538380 19021 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:22.539357 19052 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:22.539924 19052 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:22.534165 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:22.527812 19101 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:14:22.511106 19050 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.487424 18595 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.529273 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.560155 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:22.560027 19052 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.561882 19028 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:22.562633 19103 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:14:22.562755 19006 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:22.563589 19006 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:22.565842 18927 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:22.566604 18927 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.564775 19006 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.570991 18779 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:22.571444 18779 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:22.571851 18779 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:22.572302 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:22.572566 19099 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.341s	user 0.001s	sys 0.000s
I20260709 12:14:22.574252 19026 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:22.574911 18760 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.478878 18887 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 2, Time since last communication: 0.062s
W20260709 12:14:22.544514 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:22.587311 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101988us
W20260709 12:14:22.588290 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.462344 19030 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.489343 19091 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:22.531181 18904 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:22.491729 18816 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.543547 12572 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:22.577244 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:22.578164 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.578859 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.742156 12639 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:22.743662 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.579217 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.745345 12643 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:22.586141 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.426373 18635 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.746706 12640 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 21 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:14:22.534231 18985 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:22.652590 12563 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:22.671089 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:22.591073 19040 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.697s	user 0.011s	sys 0.005s
I20260709 12:14:22.748837 12564 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:22.509155 18761 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.717012 12646 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:22.717413 12642 raft_consensus.cc:1217] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-19.11]   Dedup: 17.10->[19.11-19.11]
I20260709 12:14:22.752506 12569 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:14:22.752660 12642 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:14:22.488363 18516 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:14:22.722978 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:22.728801 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156571us
I20260709 12:14:22.553401 18920 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.569514 18927 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:22.748564 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 866) took 1446 ms (client timeout 1603 ms). Trace:
W20260709 12:14:22.687422 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:22.759047 13262 rpcz_store.cc:269] 0709 12:14:21.301852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:21.304052 (+  2200us) service_pool.cc:224] Handling call
0709 12:14:22.732573 (+1428521us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:22.748476 (+ 15903us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":634948864}
I20260709 12:14:22.759550 12489 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:14:22.760043 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:22.604157 18887 consensus_queue.cc:1243] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:14:22.756553 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.748482 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.761088 12496 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
I20260709 12:14:22.596210 19034 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:14:22.755239 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 178075us
I20260709 12:14:22.774555 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:22.738237 19105 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260709 12:14:22.715713 19091 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.590713 18884 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.734711 18965 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:22.778774 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.780135 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:22.585453 18595 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.793447 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:22.750840 18901 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:22.727998 19028 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:22.758617 18761 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:22.753868 18348 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.779800 19111 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.576267 19050 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:22.781363 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 266559us
I20260709 12:14:22.797340 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.802958 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.772262 19058 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:22.758817 19040 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.793447 18816 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:22.788502 19047 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:22.787647 19034 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:22.870136 12566 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:22.826568 19112 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:22.828519 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.848071 12563 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:22.830677 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:22.849277 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58460.7us
I20260709 12:14:22.831774 12565 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:14:22.878294 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.878572 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.878965 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.877321 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:22.879783 12565 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:22.793617 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71081.2us
I20260709 12:14:22.786607 19091 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:22.854403 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.875579 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:22.893167 12645 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:22.765525 18920 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.779474 18906 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:14:22.818392 18956 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:22.792924 18965 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:22.873342 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.826189 18957 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:22.880584 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:22.881731 12563 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:22.831153 19081 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.917531 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.914804 12642 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:22.887575 12565 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:22.818399 19083 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:14:22.896871 12646 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:14:22.922272 12645 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:22.864133 18760 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:22.867118 18635 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.825091 18930 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:14:22.911078 18956 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:22.943673 12570 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:22.944751 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:22.911908 18939 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:14:22.912937 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.921051 12495 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:22.952052 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.881435 18348 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.880751 18870 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.924103 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.926357 12565 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:22.896202 19029 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.931008 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.925228 19115 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:22.907114 19003 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.821445 18985 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.943989 18773 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.905078 19082 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.945544 12496 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:22.946803 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:22.949564 12572 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:22.949966 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:22.976593 12489 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:22.977794 19114 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:14:22.978142 18884 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.956286 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:22.980139 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.980883 12645 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:22.934718 19072 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.856s	user 0.015s	sys 0.001s
I20260709 12:14:22.872139 18816 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.837847 19111 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.962338 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:22.963405 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:22.984959 12642 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:22.985491 19111 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.961834 18913 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:22.896337 19027 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:22.970358 19003 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.964079 19115 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.988610 19040 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:22.949316 18956 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:22.956382 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.890954 19112 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.990785 12640 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:22.975415 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:22.999487 12639 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:22.993151 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:22.969556 19121 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.988484 19115 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:22.935483 18760 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.968596 18516 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.991721 18913 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:22.960927 18348 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.962016 18870 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.977850 18985 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:23.035521 12643 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:14:23.035717 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:22.980455 18773 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.040517 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:23.009589 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
W20260709 12:14:22.993234 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82841.9us
I20260709 12:14:22.995862 18884 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:22.934849 19028 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:23.053252 12639 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:22.988397 19082 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:22.964874 19029 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:22.988072 19027 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.068646 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:23.006531 18583 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260709 12:14:23.073918 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:23.074776 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:23.060575 18595 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.105168 18930 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.053727 18516 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.012411 19030 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.055836 18913 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:22.965982 19034 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:23.102205 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:23.077180 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.103152 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:23.041687 19121 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:23.092283 19021 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.001s	sys 0.017s
I20260709 12:14:23.078325 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:23.102010 19047 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.138514 13255 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:23.138787 12490 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:23.094348 19003 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.113137 18583 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:23.117020 19050 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.091775 18884 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.188637 18884 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.190245 19027 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.190651 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.190971 19029 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:23.192277 19082 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.157411 18920 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.179684 19130 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:14:23.180764 19132 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
W20260709 12:14:23.184832 19126 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
W20260709 12:14:23.151717 19021 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.001s	sys 0.018s
I20260709 12:14:23.164953 19047 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:23.192972 18779 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.621s	user 0.002s	sys 0.006s
I20260709 12:14:23.120810 18930 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.165198 19081 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.214260 19027 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.279414 19027 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.205958 19050 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.281678 19050 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.282770 19027 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.284241 19050 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.272642 18930 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.248499 18779 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.256049 18870 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.161433 18816 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:23.256892 18855 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.774s	user 0.007s	sys 0.004s
I20260709 12:14:23.301689 18515 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:23.302292 18515 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:23.303227 18515 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.8, Last appended by leader: 8, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.303879 18970 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.304553 18515 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:23.305097 18970 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:23.306200 12641 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:23.214100 18986 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260709 12:14:23.307405 18779 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.307739 18986 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:23.308125 18986 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:23.261068 18925 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.770s	user 0.008s	sys 0.002s
I20260709 12:14:23.266567 18773 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:23.268002 19138 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:14:23.268751 19140 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:23.268888 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:23.297212 18541 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.853s	user 0.014s	sys 0.000s
I20260709 12:14:23.274712 19121 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.216699 19129 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.121s	user 0.000s	sys 0.003s
I20260709 12:14:23.276551 19047 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.277472 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 835) took 3025 ms. Trace:
W20260709 12:14:23.278481 18542 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.756s	user 0.010s	sys 0.001s
I20260709 12:14:23.294524 13261 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:14:23.405428 13261 pending_rounds.cc:85] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 14
I20260709 12:14:23.405773 13261 pending_rounds.cc:107] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:14:23.406018 13261 raft_consensus.cc:2889] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:23.297999 13260 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:23.301141 18957 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:23.302018 18855 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.258141 19112 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.307976 18904 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.555s	user 0.005s	sys 0.009s
W20260709 12:14:23.313248 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 235035us
I20260709 12:14:23.316526 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:23.414402 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:23.414680 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98261.6us
W20260709 12:14:23.322139 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 205664us
I20260709 12:14:23.418502 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:23.419359 12640 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:23.328269 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.270179 18920 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.358650 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:23.363029 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:23.365391 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.423802 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:23.353653 19091 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.436771 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:23.437089 18855 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.438570 19091 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.371261 12495 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 12:14:23.439484 12495 pending_rounds.cc:85] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 9
I20260709 12:14:23.439749 12495 pending_rounds.cc:107] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:14:23.439996 12495 raft_consensus.cc:2889] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:23.372150 12492 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:23.441712 18920 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.442623 12494 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:23.444715 18920 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.445037 18760 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:23.398378 18965 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.446851 12569 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:23.448335 18965 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.357431 18541 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.451318 18541 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.338599 18925 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:23.453229 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:23.454082 18925 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.458844 18761 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.633s	user 0.010s	sys 0.003s
I20260709 12:14:23.459435 18761 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.459980 12494 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:23.462555 12637 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:23.423116 18904 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.379323 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.466909 12489 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:23.468536 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 783) took 1336 ms. Trace:
I20260709 12:14:23.469024 13254 rpcz_store.cc:276] 0709 12:14:22.132372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:22.132522 (+   150us) service_pool.cc:224] Handling call
0709 12:14:23.468316 (+1335794us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:23.468469 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1693755136}
I20260709 12:14:23.381426 19115 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:23.539065 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:23.539153 19096 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.453s	user 0.014s	sys 0.027s
I20260709 12:14:23.540985 19115 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:23.386137 19118 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.291s	user 0.000s	sys 0.002s
I20260709 12:14:23.386930 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.544116 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.391162 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:23.394268 18516 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.395121 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.309062 18986 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.399119 12644 rpcz_store.cc:276] 0709 12:14:20.252595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:20.491049 (+238454us) service_pool.cc:224] Handling call
0709 12:14:20.491622 (+   573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:20.525997 (+ 34375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:20.526030 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:20.554058 (+ 28028us) log.cc:844] Serialized 26 byte log entry
0709 12:14:21.970571 (+1416513us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b18004bd420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:22.167353 (+196782us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:22.167402 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:23.277078 (+1109676us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:23.277137 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:23.277191 (+    54us) raft_consensus.cc:1685] finished
0709 12:14:23.277208 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:23.277383 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4629,"raft.queue_time_us":1886,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":7130,"spinlock_wait_cycles":92570880,"thread_start_us":196548,"threads_started":1,"wal-append.queue_time_us":1834291}
W20260709 12:14:23.401978 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.398006 19089 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.422907 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:23.369689 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:23.562013 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.015s	sys 0.028s
I20260709 12:14:23.562090 12639 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 16: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:23.426920 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:23.562657 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.015s	sys 0.028s
W20260709 12:14:23.565920 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 614575us
I20260709 12:14:23.428120 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:23.428792 12642 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:14:23.567179 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 139154us
I20260709 12:14:23.568099 12642 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:23.569614 12642 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:23.432192 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.435552 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:23.570763 12641 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:23.379724 18531 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.471519 18348 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:23.472864 19120 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.256s	user 0.003s	sys 0.000s
I20260709 12:14:23.474529 13253 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:23.477917 12646 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:23.478701 12494 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:23.649945 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.478915 18904 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.650300 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:23.603516 18986 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:23.654206 12639 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:23.489197 19009 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:23.656250 12641 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:23.656865 12494 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:23.491876 18966 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:23.493145 12496 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:23.498454 18761 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:23.500434 18956 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:23.506543 18811 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:23.508432 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:23.701185 12640 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: [20.13-22.14]
I20260709 12:14:23.510895 12492 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:23.514582 19052 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:23.516520 12638 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 12:14:23.520287 19133 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.429s	user 0.002s	sys 0.000s
W20260709 12:14:23.526939 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.446844 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.537230 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:23.549070 12495 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:23.548219 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.731431 12563 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:23.550139 12491 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:23.434566 18973 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.027s	user 0.003s	sys 0.014s
I20260709 12:14:23.573134 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:23.434083 19006 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:23.575364 12642 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:23.607401 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:23.609232 12645 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: [14.13-18.14]
W20260709 12:14:23.619769 12564 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.019s	sys 0.040s
W20260709 12:14:23.737195 12564 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.019s	sys 0.040s
I20260709 12:14:23.739208 12645 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:23.753257 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:23.633319 12493 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:23.751331 19006 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:23.786018 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:23.790738 12642 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:23.792116 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.573468 13254 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:14:23.799019 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:23.806339 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.807231 12494 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:23.808322 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.809823 12572 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:14:23.811801 12495 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: [14.11-16.12]
I20260709 12:14:23.813289 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.814518 12495 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:23.815690 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.615258 12637 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:23.433422 18927 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.171s
I20260709 12:14:23.642835 13218 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 15 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16.
I20260709 12:14:23.598747 19003 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:23.646844 12488 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:23.433704 18834 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.173s
I20260709 12:14:23.650892 12569 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:14:23.655958 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:23.900079 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 244229us
I20260709 12:14:23.902031 12496 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 13. Current term is 14. Ops: []
I20260709 12:14:23.640525 18906 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:14:23.676498 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.784813 18973 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:23.678586 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:23.681056 19123 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:23.686503 13255 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:23.510237 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:23.653836 18959 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.135s	user 0.014s	sys 0.001s
I20260709 12:14:23.704321 18811 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.922758 19123 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:23.923415 19123 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:23.925155 18811 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:23.934161 18927 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:23.934820 18927 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:23.841188 18812 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
W20260709 12:14:23.935905 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.935930 18927 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.943459 18956 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:23.702139 19026 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 12:14:23.718084 19151 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.718698 19112 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:23.718478 18904 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:23.719475 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2962) took 1046 ms. Trace:
W20260709 12:14:23.718500 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 304225us
I20260709 12:14:23.725976 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:23.983472 12568 rpcz_store.cc:276] 0709 12:14:22.657883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:22.658050 (+   167us) service_pool.cc:224] Handling call
0709 12:14:22.658548 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:22.658883 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:22.958835 (+299952us) log.cc:844] Serialized 14 byte log entry
0709 12:14:23.330962 (+372127us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:23.331681 (+   719us) log.cc:844] Serialized 26 byte log entry
0709 12:14:23.331828 (+   147us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:23.332085 (+   257us) log.cc:844] Serialized 14 byte log entry
0709 12:14:23.685913 (+353828us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b180019b1c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:14:23.686316 (+   403us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:23.686475 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:23.703887 (+ 17412us) raft_consensus.cc:1685] finished
0709 12:14:23.703918 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:23.704091 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":351553,"prepare.queue_time_us":315036,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":99,"raft.queue_time_us":373709,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":32571,"spinlock_wait_cycles":9129728,"thread_start_us":562918,"threads_started":2,"wal-append.queue_time_us":526414}
I20260709 12:14:23.732790 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:23.914093 18595 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:23.816608 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.996757 12491 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:14:23.818051 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.819482 18456 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:23.820565 18957 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:23.796855 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:24.010103 12639 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:23.830138 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.013931 12572 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:14:23.661572 19091 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.122s
I20260709 12:14:23.847898 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:23.750303 19115 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.851178 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:23.788689 19121 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.999329 18456 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:24.034266 18456 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:24.034889 18904 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:14:24.035539 18904 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:24.039969 18973 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:24.040884 19143 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:14:23.985098 19151 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:23.967799 18959 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.051420 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2990) took 1222 ms. Trace:
I20260709 12:14:24.055384 18973 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:24.055641 18811 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:24.056059 18811 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:24.056133 18973 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.060984 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:24.051898 12567 rpcz_store.cc:276] 0709 12:14:22.829529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:22.891656 (+ 62127us) service_pool.cc:224] Handling call
0709 12:14:22.892078 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:22.897492 (+  5414us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:22.991026 (+ 93534us) log.cc:844] Serialized 14 byte log entry
0709 12:14:23.399858 (+408832us) log.cc:844] Serialized 14 byte log entry
0709 12:14:23.721947 (+322089us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:23.722662 (+   715us) log.cc:844] Serialized 26 byte log entry
0709 12:14:23.722904 (+   242us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:23.723210 (+   306us) log.cc:844] Serialized 14 byte log entry
0709 12:14:23.723440 (+   230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:23.723531 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:23.944439 (+220908us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:14:24.051147 (+106708us) raft_consensus.cc:1685] finished
0709 12:14:24.051183 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.051348 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":340982,"prepare.queue_time_us":79216,"prepare.run_cpu_time_us":167,"prepare.run_wall_time_us":2794,"raft.queue_time_us":467015,"raft.run_cpu_time_us":4128,"raft.run_wall_time_us":288780,"spinlock_wait_cycles":74740864,"thread_start_us":481090,"threads_started":3,"wal-append.queue_time_us":416178}
I20260709 12:14:24.069415 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:24.081775 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:24.092701 12569 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:24.036662 18904 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.099560 12567 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:23.634516 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:24.107826 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 260045us
I20260709 12:14:24.108829 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:23.790423 18792 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:23.796064 19006 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:24.110133 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:23.879972 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:24.113597 12571 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:24.114540 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 328570us
I20260709 12:14:23.789059 19145 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.106s
I20260709 12:14:23.824474 19149 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:23.785070 19142 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.111s
I20260709 12:14:23.896938 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2966) took 1099 ms. Trace:
I20260709 12:14:23.673323 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:24.127072 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:23.902348 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:23.683027 18966 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:23.893500 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 251087us
I20260709 12:14:23.912154 12570 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:23.599134 19089 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:23.735038 19107 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:23.917057 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:23.705165 18946 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:14:23.971786 12497 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.001s	sys 0.068s
I20260709 12:14:23.973052 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:23.881875 18752 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:23.971526 18688 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.678s	user 0.030s	sys 0.046s
I20260709 12:14:23.986235 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:23.733129 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:23.898471 19003 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:23.888229 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:24.115589 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.110210 19111 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:24.118018 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 133222us
I20260709 12:14:24.118651 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:24.119210 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 198164us
W20260709 12:14:24.120556 13258 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.073s	sys 0.130s
W20260709 12:14:24.235277 13258 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.074s	sys 0.130s
I20260709 12:14:24.129146 18834 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:24.132277 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:24.126302 12564 rpcz_store.cc:276] 0709 12:14:22.797395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:22.797559 (+   164us) service_pool.cc:224] Handling call
0709 12:14:22.798065 (+   506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:22.818913 (+ 20848us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:22.818943 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:22.819485 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:14:22.929609 (+110124us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:23.519404 (+589795us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:23.737819 (+218415us) log.cc:844] Serialized 14 byte log entry
0709 12:14:23.738136 (+   317us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:23.738255 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:23.875700 (+137445us) raft_consensus.cc:1685] finished
0709 12:14:23.896565 (+ 20865us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:23.896839 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1425,"prepare.queue_time_us":46701,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":24411,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":67420,"spinlock_wait_cycles":8959872,"thread_start_us":823464,"threads_started":2,"wal-append.queue_time_us":149885}
W20260709 12:14:24.236979 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105202us
I20260709 12:14:24.237507 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:24.117151 19026 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:24.220389 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2996) took 1224 ms. Trace:
I20260709 12:14:24.225818 13253 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:24.244791 12569 rpcz_store.cc:276] 0709 12:14:22.995778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.219081 (+1223303us) service_pool.cc:224] Handling call
0709 12:14:24.219634 (+   553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.219919 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.219958 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.220024 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:24.220071 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.220144 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.220300 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:24.247608 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2997) took 1189 ms. Trace:
I20260709 12:14:24.248138 12569 rpcz_store.cc:276] 0709 12:14:23.058473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.246332 (+1187859us) service_pool.cc:224] Handling call
0709 12:14:24.246814 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.247104 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.247143 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.247205 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:24.247251 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.247326 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.247501 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:24.148854 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:24.140522 13259 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.034s	sys 0.046s
W20260709 12:14:24.250259 13259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.036s	sys 0.049s
W20260709 12:14:24.227129 12497 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.005s	sys 0.071s
W20260709 12:14:24.227933 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:24.124847 18913 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.968s	user 0.013s	sys 0.000s
I20260709 12:14:24.240043 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:24.017072 19147 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:14:24.221977 18688 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.946s	user 0.031s	sys 0.047s
I20260709 12:14:24.057104 18811 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.121234 18956 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:14:24.125983 19133 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:14:24.123364 19091 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:14:24.141266 19146 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.525s	user 0.000s	sys 0.007s
W20260709 12:14:24.134253 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 231961us
I20260709 12:14:24.087658 19115 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.130136 18812 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.125s
I20260709 12:14:24.228374 18456 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.233525 18541 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.256208 19151 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:24.258996 18515 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.954s	user 0.018s	sys 0.000s
W20260709 12:14:24.261332 19140 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.901s	user 0.009s	sys 0.005s
I20260709 12:14:24.261020 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:24.262939 18959 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.264420 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:24.235046 18986 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.000s
I20260709 12:14:24.265687 18887 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:24.235556 18904 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:14:24.242605 18957 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:24.362005 18957 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:24.362599 18957 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:24.233290 19112 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.269807 19021 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 12:14:24.270364 19156 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:14:24.138557 19089 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.277773 18752 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.234162 19003 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:24.110272 18792 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.282402 19147 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:24.301668 19154 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.372658 18986 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:24.208309 19107 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:24.302578 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62603.3us
I20260709 12:14:24.302732 18956 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.304352 19153 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:24.242291 19111 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:24.281814 19009 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.000s
I20260709 12:14:24.117394 19121 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.328430 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:24.334635 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73943.8us
I20260709 12:14:24.415359 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:24.415822 18959 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:24.416329 18959 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:24.416886 18959 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:24.336617 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:24.419225 18542 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:24.419827 18542 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:24.420233 18542 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:24.417325 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:24.340885 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:24.426982 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86248.1us
I20260709 12:14:24.295982 18688 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:24.359656 18456 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:24.332257 19052 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.569s	user 0.007s	sys 0.008s
I20260709 12:14:24.370751 12563 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:14:24.329727 19145 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.343366 19151 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.348043 19140 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:24.354017 18811 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:24.129382 18966 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.130781 19149 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.301980 19142 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.465935 19142 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.466656 19107 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:24.467365 19107 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.467873 19107 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.468537 19142 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.294833 18913 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.380275 18812 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:24.377317 19112 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.398720 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:24.412076 18956 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.422230 19147 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.497998 19147 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:24.498869 18752 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.396896 18986 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:24.499899 18986 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:24.506131 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1113) took 1314 ms. Trace:
I20260709 12:14:24.506639 12564 rpcz_store.cc:276] 0709 12:14:23.192061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.505001 (+1312940us) service_pool.cc:224] Handling call
0709 12:14:24.505468 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.505770 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.505798 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.505846 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:24.505877 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.505934 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.506057 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:24.510776 19155 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.756s	user 0.001s	sys 0.003s
I20260709 12:14:24.511032 19111 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.463029 18834 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:24.516201 18595 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:24.516683 18595 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:24.459133 19052 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.393329 18973 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.424368 18816 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.462337 12566 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:14:24.455331 19145 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.573727 18816 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.576092 19145 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.334473 18541 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.578529 18541 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.485440 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.579627 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
W20260709 12:14:24.489100 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90463us
I20260709 12:14:24.455192 18931 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.583350 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:24.494892 18456 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:14:24.495699 19161 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.227s	user 0.001s	sys 0.001s
I20260709 12:14:24.435534 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:24.585875 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 150467us
I20260709 12:14:24.586983 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:24.398173 19154 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.590204 18931 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.488139 18812 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:24.520902 18966 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:24.452260 19151 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:24.604751 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:24.571676 18752 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:24.616961 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:24.618278 19160 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260709 12:14:24.598642 19147 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:24.597558 18834 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:24.606103 18904 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.604000 19121 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.582967 19052 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.623070 18792 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.517596 18595 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:24.623592 19123 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.700s	user 0.039s	sys 0.034s
I20260709 12:14:24.606103 19082 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:24.622776 19112 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.620995 18969 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.678581 19149 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.613790 19107 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.639917 19154 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.609843 18456 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:24.619179 18956 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:24.638301 19115 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.681231 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 786) took 1291 ms. Trace:
I20260709 12:14:24.678643 18969 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.595069 18688 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:24.595134 18906 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.000s	sys 0.052s
I20260709 12:14:24.638998 18957 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.677320 19123 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.684425 18913 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.685607 18887 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:24.688521 18927 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:24.675964 18904 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.641953 18752 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:24.698308 18906 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.001s	sys 0.052s
I20260709 12:14:24.691911 13259 rpcz_store.cc:276] 0709 12:14:23.390242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.394297 (+  4055us) service_pool.cc:224] Handling call
0709 12:14:23.394745 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:23.395160 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:23.395188 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:23.395734 (+   546us) log.cc:844] Serialized 26 byte log entry
0709 12:14:24.287155 (+891421us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:24.287230 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.287333 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:24.680874 (+393541us) raft_consensus.cc:1685] finished
0709 12:14:24.680902 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.681129 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1728,"raft.queue_time_us":3437,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":1791,"thread_start_us":854679,"threads_started":1,"wal-append.queue_time_us":750324}
I20260709 12:14:24.691475 18456 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.718048 19151 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.718446 18595 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:24.718983 18456 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:24.720782 19006 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.000s
I20260709 12:14:24.713291 18906 consensus_queue.cc:1243] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:24.722097 18816 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:24.675957 18812 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.721287 19006 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.717355 19003 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:24.728653 18855 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:14:24.729943 18542 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:24.751943 18946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.011s	sys 0.022s
I20260709 12:14:24.746177 18812 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:14:24.757375 18946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.012s	sys 0.022s
I20260709 12:14:24.779902 18957 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:24.762986 19052 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:24.785380 18904 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:24.789399 18969 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:24.802076 19026 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.525s	user 0.014s	sys 0.002s
I20260709 12:14:24.792846 18931 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:24.793394 19123 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:24.854684 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.012s	sys 0.025s
W20260709 12:14:24.855852 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.012s	sys 0.025s
W20260709 12:14:24.858814 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:24.415439 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87189.3us
I20260709 12:14:24.871487 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:24.878010 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.001s	sys 0.015s
W20260709 12:14:24.878571 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.001s	sys 0.015s
I20260709 12:14:24.880564 12489 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:14:24.883611 12494 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:14:24.883424 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3128) took 1521 ms. Trace:
W20260709 12:14:24.884377 12491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.014s	sys 0.005s
W20260709 12:14:24.884927 12491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.014s	sys 0.006s
I20260709 12:14:24.886562 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3147) took 1054 ms. Trace:
W20260709 12:14:24.888453 12497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.001s	sys 0.005s
W20260709 12:14:24.889084 12497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.002s	sys 0.005s
I20260709 12:14:24.888916 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:24.890899 12488 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:24.887282 12493 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:14:24.894229 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:24.884745 12490 rpcz_store.cc:276] 0709 12:14:23.361350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.367413 (+  6063us) service_pool.cc:224] Handling call
0709 12:14:23.367869 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:23.368253 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:23.368286 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:23.368833 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 12:14:23.759763 (+390930us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:23.759828 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:23.760559 (+   731us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:24.882175 (+1121616us) raft_consensus.cc:1685] finished
0709 12:14:24.882278 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.883000 (+   722us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5013,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":390465,"threads_started":1,"wal-append.queue_time_us":1013467}
W20260709 12:14:24.898310 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.005s	sys 0.003s
W20260709 12:14:24.899048 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.005s	sys 0.003s
W20260709 12:14:24.901521 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 484097us
I20260709 12:14:24.908492 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:24.915302 12563 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:24.916343 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1117) took 1312 ms. Trace:
I20260709 12:14:24.916687 12563 rpcz_store.cc:276] 0709 12:14:23.604443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.913825 (+1309382us) service_pool.cc:224] Handling call
0709 12:14:24.914808 (+   983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.915938 (+  1130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.916234 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:24.918797 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.919641 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:24.921798 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.922665 12563 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:24.924291 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.533974 12564 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:24.926815 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1115) took 1631 ms. Trace:
I20260709 12:14:24.927249 12564 rpcz_store.cc:276] 0709 12:14:23.295352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.530637 (+1235285us) service_pool.cc:224] Handling call
0709 12:14:24.533684 (+  3047us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.926170 (+392486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.926604 (+   434us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:24.928764 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:24.929744 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3145) took 1119 ms. Trace:
I20260709 12:14:24.930292 12494 rpcz_store.cc:276] 0709 12:14:23.805281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.808044 (+  2763us) service_pool.cc:224] Handling call
0709 12:14:24.924410 (+1116366us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:24.924633 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2464892672}
I20260709 12:14:24.933550 12492 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.564471 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1112) took 1372 ms. Trace:
I20260709 12:14:24.935196 12567 rpcz_store.cc:276] 0709 12:14:23.191751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.282237 (+1090486us) service_pool.cc:224] Handling call
0709 12:14:24.282609 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.425734 (+143125us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.425784 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.425847 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:24.425892 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.564230 (+138338us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:24.564281 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.564410 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328383232}
I20260709 12:14:24.937032 12492 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:24.520309 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1110) took 1435 ms. Trace:
I20260709 12:14:24.971438 12569 rpcz_store.cc:276] 0709 12:14:23.085676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.278042 (+1192366us) service_pool.cc:224] Handling call
0709 12:14:24.278422 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.491350 (+212928us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.491383 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.491437 (+    54us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:24.491480 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.520017 (+ 28537us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:24.520077 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.520240 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489132288}
I20260709 12:14:24.973278 18516 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:24.973683 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.975188 18516 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:24.975659 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.976871 12569 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:24.931931 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.010s	sys 0.009s
W20260709 12:14:24.977967 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.011s	sys 0.010s
I20260709 12:14:24.979718 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2969) took 2034 ms. Trace:
W20260709 12:14:24.931958 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.212s	user 0.024s	sys 0.027s
I20260709 12:14:24.980671 18542 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:24.934177 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3002) took 1663 ms (client timeout 1624 ms). Trace:
W20260709 12:14:24.982002 12566 rpcz_store.cc:269] 0709 12:14:23.271006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.446542 (+175536us) service_pool.cc:224] Handling call
0709 12:14:24.933711 (+1487169us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54003ac354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:24.933961 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1682633088}
I20260709 12:14:24.983160 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:24.938277 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3140) took 1134 ms. Trace:
I20260709 12:14:25.014983 12489 rpcz_store.cc:276] 0709 12:14:23.803702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.810646 (+  6944us) service_pool.cc:224] Handling call
0709 12:14:23.859422 (+ 48776us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:24.881432 (+1022010us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.937894 (+ 56462us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:24.938211 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2346344704}
I20260709 12:14:24.931108 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:24.950856 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.002s	sys 0.025s
W20260709 12:14:24.952035 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.005s	sys 0.044s
W20260709 12:14:24.707795 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:24.953517 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.002s	sys 0.063s
W20260709 12:14:24.955471 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:24.956851 13258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.003s	sys 0.007s
I20260709 12:14:24.887001 12491 rpcz_store.cc:276] 0709 12:14:23.832595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.036066 (+203471us) service_pool.cc:224] Handling call
0709 12:14:24.036470 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.885901 (+849431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.885970 (+    69us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.886052 (+    82us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:24.886091 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.886213 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.886470 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41578,"raft.queue_time_us":63421,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":10885,"thread_start_us":805911,"threads_started":1}
I20260709 12:14:24.956809 12494 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:24.969491 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:24.969343 18515 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:24.970077 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:24.968619 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:24.981066 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.261s	user 0.024s	sys 0.027s
I20260709 12:14:24.563354 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1116) took 1250 ms. Trace:
I20260709 12:14:24.937202 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:24.985996 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:24.980216 12570 rpcz_store.cc:276] 0709 12:14:22.945552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.129407 (+1183855us) service_pool.cc:224] Handling call
0709 12:14:24.130450 (+  1043us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:24.130904 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:24.130933 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:24.131493 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:14:24.978901 (+847408us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:24.979138 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.979290 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:24.979385 (+    95us) raft_consensus.cc:1685] finished
0709 12:14:24.979410 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.979625 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6269,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":124,"thread_start_us":846610,"threads_started":1,"wal-append.queue_time_us":676785}
I20260709 12:14:24.990099 18855 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:24.991696 18946 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.993445 18516 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:24.996170 18752 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:24.999177 18887 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.000021 18792 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.001220 19009 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:24.769692 18959 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:24.989342 18906 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:25.002218 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3009) took 1285 ms. Trace:
I20260709 12:14:24.989283 18760 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:14:25.006094 19003 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:25.007277 19006 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:25.017105 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:25.058915 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.005s	sys 0.044s
W20260709 12:14:25.059016 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.003s	sys 0.025s
W20260709 12:14:25.065865 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.002s	sys 0.064s
I20260709 12:14:24.806130 19026 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:25.068971 13258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.003s	sys 0.008s
I20260709 12:14:25.078504 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.080010 12564 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:25.084234 12563 rpcz_store.cc:276] 0709 12:14:23.716611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.000444 (+1283833us) service_pool.cc:224] Handling call
0709 12:14:25.001213 (+   769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.001720 (+   507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.001757 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:25.001835 (+    78us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:25.001869 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.001950 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.002108 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.088649 19010 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:25.090468 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:25.091518 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:25.091639 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.092342 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 2995) took 2221 ms. Trace:
I20260709 12:14:25.098205 12493 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 14: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:25.089812 12568 rpcz_store.cc:276] 0709 12:14:23.312991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.562368 (+1249377us) service_pool.cc:224] Handling call
0709 12:14:24.562776 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.562992 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.563015 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:24.563063 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:24.563096 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.563153 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:24.563281 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.100498 19165 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:25.101107 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 971641us
I20260709 12:14:25.101809 19174 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.102698 19174 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.103811 12489 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:25.106331 19166 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.106685 19009 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:25.108084 12570 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:25.108914 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3010) took 1392 ms. Trace:
I20260709 12:14:25.101428 19165 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.098663 19010 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:25.113979 18516 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.119904 19127 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:25.108566 19009 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.110824 12570 rpcz_store.cc:276] 0709 12:14:23.716909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.107092 (+1390183us) service_pool.cc:224] Handling call
0709 12:14:25.107649 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.108625 (+   976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.108854 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.131090 18887 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.131299 19171 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:25.132606 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:25.133378 18946 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.136533 18946 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:25.138147 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:25.139653 18959 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.106338 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:25.145105 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:25.146147 12569 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:25.148168 12569 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:25.148917 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2973) took 1270 ms. Trace:
I20260709 12:14:25.120713 19127 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:25.150067 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:25.151612 18811 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.687s	user 0.016s	sys 0.008s
I20260709 12:14:25.151732 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:25.152617 12489 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:25.153256 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.154511 19052 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:25.155287 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.155824 19004 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:25.157467 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:25.125008 19140 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:25.162678 12495 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:14:25.163036 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:25.164263 12568 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:25.113603 19165 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:25.126453 19168 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:25.165228 19168 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.166071 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.166805 12493 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:25.167939 19168 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:25.168077 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.128150 19151 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:25.125114 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1118) took 1364 ms. Trace:
I20260709 12:14:25.130656 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 867) took 2412 ms. Trace:
I20260709 12:14:25.169423 19151 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.169377 12497 rpcz_store.cc:276] 0709 12:14:22.718219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:22.718365 (+   146us) service_pool.cc:224] Handling call
0709 12:14:22.718712 (+   347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:22.729099 (+ 10387us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:14:23.451225 (+722126us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b180033a900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:14:24.251936 (+800711us) log.cc:844] Serialized 14 byte log entry
0709 12:14:24.891253 (+639317us) log.cc:844] Serialized 14 byte log entry
0709 12:14:24.891697 (+   444us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:24.893229 (+  1532us) log.cc:844] Serialized 52 byte log entry
0709 12:14:24.893474 (+   245us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:24.893522 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:24.898105 (+  4583us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:25.130282 (+232177us) raft_consensus.cc:1685] finished
0709 12:14:25.130327 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.130551 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16383,"prepare.queue_time_us":1784907,"prepare.run_cpu_time_us":189,"prepare.run_wall_time_us":6982,"raft.queue_time_us":245583,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":15397,"spinlock_wait_cycles":21522176,"thread_start_us":1388975,"threads_started":2,"wal-append.queue_time_us":520643}
I20260709 12:14:25.173250 19151 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:25.130146 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 883) took 1802 ms. Trace:
I20260709 12:14:25.123539 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.175592 12494 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:25.141319 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:25.099586 12565 rpcz_store.cc:276] 0709 12:14:22.870698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:22.960408 (+ 89710us) service_pool.cc:224] Handling call
0709 12:14:23.448262 (+487854us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:23.718926 (+270664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:23.718962 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:23.719563 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:14:25.090802 (+1371239us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:25.090876 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.091009 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:25.091068 (+    59us) raft_consensus.cc:1685] finished
0709 12:14:25.091079 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.091985 (+   906us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b5400044a50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:25.092259 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":270199,"raft.queue_time_us":270526,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":193,"spinlock_wait_cycles":1121093760,"thread_start_us":1349268,"threads_started":1,"wal-append.queue_time_us":1051721}
I20260709 12:14:25.143925 18792 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.142987 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:25.127475 12567 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:25.178666 18792 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.178920 12567 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:14:25.107312 19166 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.181320 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1119) took 1154 ms. Trace:
I20260709 12:14:25.182076 19166 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.181941 12565 rpcz_store.cc:276] 0709 12:14:24.026641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.179652 (+1153011us) service_pool.cc:224] Handling call
0709 12:14:25.180208 (+   556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.180841 (+   633us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.180874 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:25.180977 (+   103us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:25.181020 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.181088 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.181240 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.151306 19127 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.184830 19127 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.103262 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:25.169234 12563 rpcz_store.cc:276] 0709 12:14:23.760941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.123654 (+1362713us) service_pool.cc:224] Handling call
0709 12:14:25.124198 (+   544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.124659 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.124689 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:25.124776 (+    87us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:25.124822 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.124898 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.125047 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.188611 18887 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.189100 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:25.189420 19010 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.179394 12567 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:14:25.191483 12567 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:25.191948 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:25.192301 19021 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.008s	sys 0.026s
W20260709 12:14:25.192854 19021 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.008s	sys 0.026s
I20260709 12:14:25.194278 19180 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:25.194947 19180 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:25.141237 12570 raft_consensus.cc:1217] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[6.3-15.8]   Dedup: 12.6->[13.7-15.8]
I20260709 12:14:25.173601 19140 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:25.195912 12570 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 6
I20260709 12:14:25.165392 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:25.202706 12570 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 12:14:25.169445 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.191815 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:25.174296 13258 rpcz_store.cc:276] 0709 12:14:23.328204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.328372 (+   168us) service_pool.cc:224] Handling call
0709 12:14:23.362069 (+ 33697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:23.362323 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:24.235846 (+873523us) log.cc:844] Serialized 14 byte log entry
0709 12:14:25.129149 (+893303us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:25.129315 (+   166us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:25.129510 (+   195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.129723 (+   213us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.129993 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1083173,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":2965,"raft.queue_time_us":1214,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":293,"thread_start_us":1733015,"threads_started":2,"wal-append.queue_time_us":938846}
I20260709 12:14:25.149344 12569 rpcz_store.cc:276] 0709 12:14:23.878776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.147097 (+1268321us) service_pool.cc:224] Handling call
0709 12:14:25.147770 (+   673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.148579 (+   809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.148844 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.190379 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:25.203148 12570 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:25.198246 12491 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
W20260709 12:14:25.222159 19163 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.002s	sys 0.023s
W20260709 12:14:25.222882 19163 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.002s	sys 0.024s
I20260709 12:14:25.222582 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:25.222139 12493 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:25.207980 19140 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.195379 19180 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.242125 18811 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:25.245707 19026 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:25.246402 19021 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:25.248108 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.001s	sys 0.013s
W20260709 12:14:25.249078 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.007s	sys 0.007s
W20260709 12:14:25.250312 19189 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.002s	sys 0.000s
I20260709 12:14:25.250545 19010 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:25.251055 19178 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:25.252071 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.246129 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:25.258353 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.299083 12492 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:14:25.306335 12492 pending_rounds.cc:85] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 11
I20260709 12:14:25.306664 12492 pending_rounds.cc:107] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:14:25.307006 12492 raft_consensus.cc:2889] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:25.257866 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.253777 12568 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:14:25.270659 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:25.308456 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.794s	user 0.002s	sys 0.014s
I20260709 12:14:25.255892 19006 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
W20260709 12:14:25.291396 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:25.289944 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:25.277356 12490 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
W20260709 12:14:25.253569 18595 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.534s	user 0.015s	sys 0.007s
I20260709 12:14:25.350678 19026 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.353363 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.354343 12644 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:25.355932 18811 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.356345 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.357350 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:25.358614 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:25.359469 12490 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.362322 18688 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:25.363001 18688 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:25.363446 18688 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:14:25.365806 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135205us
W20260709 12:14:25.266248 13218 connection.cc:256] RPC call timeout handler was delayed by -1.562956720000102s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
I20260709 12:14:25.334594 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2975) took 1429 ms. Trace:
I20260709 12:14:25.291882 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:25.255051 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:25.338361 12565 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.369022 12568 rpcz_store.cc:276] 0709 12:14:23.879259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.164920 (+1285661us) service_pool.cc:224] Handling call
0709 12:14:25.165489 (+   569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.307625 (+142136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.308763 (+  1138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:25.253772 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.338858 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3131) took 1976 ms. Trace:
W20260709 12:14:25.273527 12389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 126) took 18639 ms (client timeout 15000 ms). Trace:
W20260709 12:14:25.339453 18970 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:14:25.286787 19003 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:14:25.283715 19166 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:25.387516 19166 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.389532 12493 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:25.338855 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1114) took 2091 ms. Trace:
I20260709 12:14:25.391439 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 884) took 1154 ms. Trace:
I20260709 12:14:25.392009 13255 rpcz_store.cc:276] 0709 12:14:24.236972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.237173 (+   201us) service_pool.cc:224] Handling call
0709 12:14:25.390987 (+1153814us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:25.391240 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2524580224}
I20260709 12:14:25.392716 19183 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:25.341807 18986 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.836s	user 0.010s	sys 0.016s
I20260709 12:14:25.394069 19178 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.393972 18986 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.394929 19178 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.396731 18986 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.345803 12564 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:25.259428 12491 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:25.390864 12572 rpcz_store.cc:276] 0709 12:14:23.247437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.512410 (+1264973us) service_pool.cc:224] Handling call
0709 12:14:24.512850 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:24.513253 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.513282 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:24.513926 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:14:25.330130 (+816204us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:25.336375 (+  6245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.338366 (+  1991us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:25.338460 (+    94us) raft_consensus.cc:1685] finished
0709 12:14:25.338492 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.338766 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":199,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":127,"thread_start_us":794636,"threads_started":1,"wal-append.queue_time_us":667698}
I20260709 12:14:25.398989 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.400122 12572 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:25.400259 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.330209 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:25.371821 12492 rpcz_store.cc:276] 0709 12:14:23.362290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:23.371706 (+  9416us) service_pool.cc:224] Handling call
0709 12:14:23.372015 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.920602 (+1548587us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001c19a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577106f 00007ffbf57e9f60 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62
0709 12:14:25.338496 (+417894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.338782 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2659815296}
I20260709 12:14:25.402208 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.402760 13260 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:25.403391 19178 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:25.265377 18812 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.506s	user 0.005s	sys 0.015s
I20260709 12:14:25.281985 18761 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:25.401262 19166 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:25.402374 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.403508 12640 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:25.404178 12569 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:25.404426 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:25.402768 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.405277 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.300853 19174 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:25.381407 12563 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:25.402316 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.405239 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.338949 19006 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:25.416639 12643 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:25.417773 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.249123 19153 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:25.419004 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.304971 19021 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.422274 12641 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:25.422618 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.423236 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.423712 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.425740 12493 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.429138 18811 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:25.430689 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:25.433914 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.554s (SENT) [suppressed 3 similar messages]
I20260709 12:14:25.432509 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.432849 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.435439 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.435467 13258 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:25.436285 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:25.437371 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:25.438011 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:25.438970 12568 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:25.439571 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:25.276036 12390 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 2 to 10. New cstate: current_term: 10 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:25.462035 12642 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:14:25.463624 12491 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:25.441102 13261 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:14:25.384147 18516 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.406467 18761 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:25.450138 19006 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:25.455242 12489 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:25.459501 12639 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:25.468037 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:25.468623 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 213831us
I20260709 12:14:25.476176 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.477232 12492 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:25.479097 12642 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:25.450912 19153 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:25.485735 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:25.488818 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:25.482568 12637 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:25.484062 12638 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:25.496055 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.474484 18812 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:25.497434 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.493670 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.472893 18811 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.497718 12645 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:25.468679 19026 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:14:25.506326 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.407598 19183 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.509598 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:25.511729 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.521205 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.522571 12643 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:25.523939 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.524802 12643 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:25.510381 12642 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:25.526584 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.527012 19021 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:25.527889 12639 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:14:25.529026 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.529742 12644 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:25.528712 18761 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:25.535971 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8065269200001239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1617s for a response
I20260709 12:14:25.531683 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:25.541711 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.538338 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.541145 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.570888 12491 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 15. Current term is 16. Ops: [9.8-15.11]
I20260709 12:14:25.581590 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.527393 12642 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
W20260709 12:14:25.568670 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:25.576133 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:25.543063 12493 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:25.563555 12646 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:25.562845 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.544500 12639 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:25.604858 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.011s	sys 0.016s
I20260709 12:14:25.547003 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:25.609649 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:25.617669 12640 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:25.618955 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.619941 12489 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:25.530103 19004 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 12:14:25.621158 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:25.622412 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 322513us
I20260709 12:14:25.571111 12565 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:14:25.636831 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.634505 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:25.646621 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.649355 12489 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:25.641546 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.650338 12645 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:25.621731 18948 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:25.653650 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:25.669386 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55914us
I20260709 12:14:25.661612 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:25.671471 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:14:25.661075 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6775313820000974s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20260709 12:14:25.676818 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6644857030000821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260709 12:14:25.677117 12644 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:25.685009 19187 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20260709 12:14:25.583204 19006 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.648361 18761 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:25.697512 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56229.2us
W20260709 12:14:25.702018 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6162788460001138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260709 12:14:25.704563 12643 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:25.706826 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.634270 18811 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:25.711375 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51765.5us
I20260709 12:14:25.659334 18948 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:25.724386 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5006773480001812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
I20260709 12:14:25.645462 18812 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
W20260709 12:14:25.724992 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3340653860000202s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
I20260709 12:14:25.731962 13262 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:14:25.759753 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53714.4us
W20260709 12:14:25.759795 12536 connection.cc:256] RPC call timeout handler was delayed by -1.314642731000049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2554s for a response
W20260709 12:14:25.761000 12536 connection.cc:256] RPC call timeout handler was delayed by -1.300236993999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17900000000000002s for a response
I20260709 12:14:25.761142 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:25.766124 19172 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:25.767158 19172 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.771727 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:25.769781 19176 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
W20260709 12:14:25.773244 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48484.3us
I20260709 12:14:25.775166 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:25.776252 19003 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:25.776095 19176 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:25.777186 19003 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:25.777835 18969 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:25.778394 19003 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:25.778887 18969 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:25.779327 18969 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:25.780587 18969 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.781910 19176 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:25.782125 18969 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:25.783917 19176 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:25.786270 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3006) took 2070 ms. Trace:
I20260709 12:14:25.786959 19153 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:25.788676 19153 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:25.787451 12571 rpcz_store.cc:276] 0709 12:14:23.715715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.580617 (+864902us) service_pool.cc:224] Handling call
0709 12:14:24.581058 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:24.581324 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:25.358776 (+777452us) log.cc:844] Serialized 14 byte log entry
0709 12:14:25.706990 (+348214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:25.758698 (+ 51708us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:25.785823 (+ 27125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.785952 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:25.786172 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1981,"prepare.queue_time_us":538812,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":88,"raft.queue_time_us":21342,"thread_start_us":1023460,"threads_started":2}
I20260709 12:14:25.793219 19172 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:25.371941 12389 rpcz_store.cc:269] 0709 12:14:06.634196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:06.637951 (+  3755us) service_pool.cc:224] Handling call
0709 12:14:07.703138 (+1065187us) spinlock_profiling.cc:243] Waited 9.71 ms on lock 0x7b5400290090. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cf0b2 00007ffbfb648759 00007ffbfb5e0a76 00007ffbfb5b1d58 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc
0709 12:14:25.273133 (+17569995us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:14:18.851340 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:14:18.943482 (+ 92142us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:14:18.943546 (+    64us) write_op.cc:435] Acquired schema lock
0709 12:14:18.943570 (+    24us) tablet.cc:662] Decoding operations
0709 12:14:18.944943 (+  1373us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:14:18.945054 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 12:14:18.945083 (+    29us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:14:19.230314 (+285231us) tablet.cc:701] Row locks acquired
0709 12:14:19.230392 (+    78us) write_op.cc:260] PREPARE: finished
0709 12:14:19.263395 (+ 33003us) write_op.cc:270] Start()
0709 12:14:19.313972 (+ 50577us) write_op.cc:276] Timestamp: P: 1783599259234355 usec, L: 0
0709 12:14:19.318476 (+  4504us) op_driver.cc:348] REPLICATION: starting
0709 12:14:19.562976 (+244500us) log.cc:844] Serialized 5579 byte log entry
0709 12:14:20.236720 (+673744us) op_driver.cc:464] REPLICATION: finished
0709 12:14:20.524952 (+288232us) write_op.cc:301] APPLY: starting
0709 12:14:20.547205 (+ 22253us) tablet.cc:1367] starting BulkCheckPresence
0709 12:14:20.950953 (+403748us) tablet.cc:1370] finished BulkCheckPresence
0709 12:14:20.951099 (+   146us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:14:21.197342 (+246243us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:14:21.210467 (+ 13125us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:14:21.213947 (+  3480us) write_op.cc:312] APPLY: finished
0709 12:14:21.294526 (+ 80579us) log.cc:844] Serialized 129 byte log entry
0709 12:14:21.985717 (+691191us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:14:22.233019 (+247302us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:22.363521 (+130502us) write_op.cc:454] Released schema lock
0709 12:14:22.376612 (+ 13091us) write_op.cc:341] FINISH: Updating metrics
0709 12:14:22.404675 (+ 28063us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:22.409195 (+  4520us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":75207,"reported_tablets":26,"spinlock_wait_cycles":22337792,"child_traces":[["op",{"apply.queue_time_us":286366,"apply.run_cpu_time_us":69926,"apply.run_wall_time_us":1863708,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62591,"num_ops":14,"prepare.queue_time_us":279628,"prepare.run_cpu_time_us":26395,"prepare.run_wall_time_us":804460,"raft.queue_time_us":186701,"raft.run_cpu_time_us":47236,"raft.run_wall_time_us":168045,"replication_time_us":878667,"spinlock_wait_cycles":63095552,"thread_start_us":1406930,"threads_started":5,"wal-append.queue_time_us":1149386,"wal-append.run_cpu_time_us":20924,"wal-append.run_wall_time_us":2654915}]]}
I20260709 12:14:25.801236 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.803560 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.804352 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:25.805860 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.806058 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.801119 12643 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:14:25.563841 12390 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 7 to 10, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 10 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:25.811940 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.812963 12643 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:25.806856 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:25.687558 18946 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:25.815361 12644 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:25.820359 12642 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:25.823619 18946 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:25.825369 12644 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
W20260709 12:14:25.826673 19192 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.122s	user 0.004s	sys 0.000s
I20260709 12:14:25.828373 18946 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:25.826457 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:25.829831 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:25.829833 12643 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:25.798662 19172 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
W20260709 12:14:25.819350 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84462.3us
I20260709 12:14:25.732790 19179 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:25.686983 19006 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:25.887964 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79053.5us
W20260709 12:14:25.934238 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 173218us
W20260709 12:14:25.849306 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71715us
W20260709 12:14:25.950443 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149902us
W20260709 12:14:25.950742 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:25.936105 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 502197us
W20260709 12:14:26.002743 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:26.019742 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.021098 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:25.959966 18927 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.983s	user 0.005s	sys 0.017s
I20260709 12:14:25.708505 19174 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:26.033072 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3155) took 1140 ms. Trace:
W20260709 12:14:25.992591 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:25.861920 19009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.024s	sys 0.065s
I20260709 12:14:25.931826 19172 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:26.042538 19009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.026s	sys 0.083s
I20260709 12:14:26.024322 18812 consensus_queue.cc:1243] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:25.930086 19021 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:26.052734 12490 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.693s	user 0.014s	sys 0.000s
I20260709 12:14:26.053169 12490 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:26.038522 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:26.044909 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2986) took 1132 ms. Trace:
I20260709 12:14:26.049655 12643 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:26.059873 19168 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:26.053023 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:26.058373 12564 rpcz_store.cc:276] 0709 12:14:24.912198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.420763 (+508565us) service_pool.cc:224] Handling call
0709 12:14:25.421153 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.421577 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:25.483660 (+ 62083us) log.cc:844] Serialized 14 byte log entry
0709 12:14:26.044506 (+560846us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:26.044554 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:26.044605 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.044689 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.044837 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32479,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":8568,"raft.queue_time_us":23731,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"thread_start_us":405618,"threads_started":1}
I20260709 12:14:26.062877 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:26.037745 12488 rpcz_store.cc:276] 0709 12:14:24.892766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.910504 (+ 17738us) service_pool.cc:224] Handling call
0709 12:14:24.911015 (+   511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:24.911655 (+   640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:24.911689 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:24.912347 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:14:25.074282 (+161935us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:25.648037 (+573755us) log.cc:844] Serialized 14 byte log entry
0709 12:14:26.030115 (+382078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.030239 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.032740 (+  2501us) raft_consensus.cc:1685] finished
0709 12:14:26.032760 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.032979 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87285,"prepare.queue_time_us":311453,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":203408,"raft.run_cpu_time_us":600,"raft.run_wall_time_us":41088,"thread_start_us":577238,"threads_started":2,"wal-append.queue_time_us":180196}
I20260709 12:14:26.092737 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:26.094050 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105523us
I20260709 12:14:26.093851 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:26.048586 18969 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:26.097115 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:26.061321 19021 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:26.095137 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:26.103382 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.005s	sys 0.016s
I20260709 12:14:26.061621 19183 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:26.032368 19180 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:26.107592 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.006s	sys 0.017s
I20260709 12:14:26.053495 19179 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.106163 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2979) took 2118 ms. Trace:
I20260709 12:14:26.108877 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:26.114662 12567 rpcz_store.cc:276] 0709 12:14:23.987363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.177885 (+1190522us) service_pool.cc:224] Handling call
0709 12:14:25.178560 (+   675us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:25.192397 (+ 13837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.192432 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:25.193081 (+   649us) log.cc:844] Serialized 26 byte log entry
0709 12:14:25.625242 (+432161us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:25.625281 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.625373 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.105905 (+480532us) raft_consensus.cc:1685] finished
0709 12:14:26.105930 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.106095 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":431586,"threads_started":1,"wal-append.queue_time_us":717713}
I20260709 12:14:26.108866 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:26.100688 18969 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:26.103123 18927 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:26.104720 19021 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:26.116125 18688 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.752s	user 0.014s	sys 0.003s
I20260709 12:14:26.123484 18812 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:26.118940 18792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.022s	sys 0.069s
I20260709 12:14:26.072705 19168 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:26.152192 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103009us
W20260709 12:14:26.161408 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.138819 18811 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:26.128789 19171 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:26.043344 18948 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:26.144081 19183 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:26.176061 18792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.022s	sys 0.069s
I20260709 12:14:26.158666 18927 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:26.159375 18812 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:26.207979 18812 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:26.208763 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 434234us
I20260709 12:14:26.169525 19172 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:14:26.205813 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:26.186779 19183 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:26.162374 18688 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:26.228456 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55066.6us
W20260709 12:14:26.254009 19194 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:14:26.256655 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3007) took 2540 ms. Trace:
W20260709 12:14:26.257984 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167222us
W20260709 12:14:26.234582 19198 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:14:26.229596 19183 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:26.250267 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:26.268818 13257 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:26.238864 19115 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s	user 0.067s	sys 0.094s
I20260709 12:14:26.239910 19009 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:26.270383 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:26.304317 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54362.7us
I20260709 12:14:26.307890 12642 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:26.307549 19179 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:26.310883 19179 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:26.311528 19200 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
W20260709 12:14:26.317853 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 244507us
W20260709 12:14:26.320837 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:26.321357 12638 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.656s	user 0.008s	sys 0.000s
W20260709 12:14:26.322511 19204 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:14:26.324601 19168 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:26.302428 19115 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s	user 0.067s	sys 0.095s
I20260709 12:14:26.333911 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 876) took 1049 ms. Trace:
I20260709 12:14:26.238194 19003 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.257102 12566 rpcz_store.cc:276] 0709 12:14:23.716054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:24.983452 (+1267398us) service_pool.cc:224] Handling call
0709 12:14:24.984088 (+   636us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:24.985417 (+  1329us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:25.070110 (+ 84693us) log.cc:844] Serialized 14 byte log entry
0709 12:14:25.615524 (+545414us) log.cc:844] Serialized 14 byte log entry
0709 12:14:25.619548 (+  4024us) log.cc:844] Serialized 14 byte log entry
0709 12:14:25.619762 (+   214us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:25.620564 (+   802us) log.cc:844] Serialized 26 byte log entry
0709 12:14:25.620711 (+   147us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:25.620754 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:25.620824 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.256412 (+635588us) raft_consensus.cc:1685] finished
0709 12:14:26.256433 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.256586 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":802,"prepare.queue_time_us":135996,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":136,"raft.queue_time_us":13003,"raft.run_cpu_time_us":602,"raft.run_wall_time_us":6112,"spinlock_wait_cycles":48384,"thread_start_us":588924,"threads_started":2,"wal-append.queue_time_us":844715}
W20260709 12:14:26.335522 13218 connection.cc:256] RPC call timeout handler was delayed by -1.2818327840000165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20260709 12:14:26.328933 12638 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:26.302731 19009 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:26.350374 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:26.345350 12491 rpcz_store.cc:276] 0709 12:14:25.284890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.648680 (+363790us) service_pool.cc:224] Handling call
0709 12:14:25.681789 (+ 33109us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:25.764210 (+ 82421us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:14:25.938638 (+174428us) log.cc:844] Serialized 14 byte log entry
0709 12:14:26.333369 (+394731us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:26.333461 (+    92us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:26.333528 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.333655 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.333826 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66601,"prepare.queue_time_us":93108,"prepare.run_cpu_time_us":826,"prepare.run_wall_time_us":5434,"raft.queue_time_us":4032,"thread_start_us":272401,"threads_started":1}
I20260709 12:14:26.248016 18969 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:26.335914 19176 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.540s	user 0.012s	sys 0.003s
I20260709 12:14:26.376579 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:26.381281 12639 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:26.392083 12638 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
W20260709 12:14:26.328842 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 385258us
I20260709 12:14:26.368104 19009 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:26.311549 19179 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:26.378037 19180 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:26.411011 12644 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:26.411943 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:26.412575 12644 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260709 12:14:26.413470 12638 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:26.382218 19178 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:26.395460 19140 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.061s	user 0.012s	sys 0.003s
W20260709 12:14:26.441239 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.395116 18811 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:26.475066 19199 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:14:26.416028 19210 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:26.456405 19178 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:26.389248 19176 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.416625 18946 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:26.466569 18969 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:26.379526 18927 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.528688 18812 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:26.531526 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 281342us
W20260709 12:14:26.533382 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 223853us
I20260709 12:14:26.453948 19140 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.536715 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:26.547305 12646 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:14:26.479588 18970 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:14:26.480321 18811 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:26.500388 19217 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
W20260709 12:14:26.552727 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 282433us
W20260709 12:14:26.548177 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51206.2us
I20260709 12:14:26.534682 18688 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:26.562188 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131600us
W20260709 12:14:26.564733 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:26.551717 19212 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:14:26.571753 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:26.552347 19153 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.761s	user 0.016s	sys 0.000s
W20260709 12:14:26.574994 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166337us
I20260709 12:14:26.512955 19215 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
W20260709 12:14:26.577667 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109022us
I20260709 12:14:26.580631 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:26.465039 12390 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 9 to 11, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 11 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:26.583909 18812 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:26.584375 19178 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:26.586720 19202 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260709 12:14:26.589875 19172 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:26.590371 19172 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:26.590835 19172 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:26.591272 19213 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
W20260709 12:14:26.592320 19205 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:14:26.592481 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3168) took 1102 ms. Trace:
I20260709 12:14:26.593144 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 2972) took 2714 ms. Trace:
I20260709 12:14:26.595522 12570 rpcz_store.cc:276] 0709 12:14:23.878537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.139553 (+1261016us) service_pool.cc:224] Handling call
0709 12:14:25.140427 (+   874us) raft_consensus.cc:1471] Updating replica for 6 ops
0709 12:14:25.249965 (+109538us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:14:25.335171 (+ 85206us) log.cc:844] Serialized 14 byte log entry
0709 12:14:26.419906 (+1084735us) log.cc:844] Serialized 14 byte log entry
0709 12:14:26.459547 (+ 39641us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:26.467152 (+  7605us) log.cc:844] Serialized 52 byte log entry
0709 12:14:26.479764 (+ 12612us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:14:26.479825 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.479927 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.586055 (+106128us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:26.592831 (+  6776us) raft_consensus.cc:1685] finished
0709 12:14:26.592866 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.593065 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86054,"prepare.queue_time_us":499926,"prepare.run_cpu_time_us":182,"prepare.run_wall_time_us":7412,"raft.queue_time_us":160421,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":16072,"spinlock_wait_cycles":34327936,"thread_start_us":915212,"threads_started":2,"wal-append.queue_time_us":805089}
I20260709 12:14:26.593920 12641 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:14:26.539441 18946 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:26.598452 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:26.594116 19207 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:14:26.600127 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:26.600669 19201 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.263s	user 0.001s	sys 0.000s
W20260709 12:14:26.601409 19195 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.559s	user 0.000s	sys 0.002s
I20260709 12:14:26.594916 19176 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:26.592892 12492 rpcz_store.cc:276] 0709 12:14:25.490215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.834700 (+344485us) service_pool.cc:224] Handling call
0709 12:14:25.835295 (+   595us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:25.835818 (+   523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.835844 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:25.836579 (+   735us) log.cc:844] Serialized 26 byte log entry
0709 12:14:26.076721 (+240142us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:26.076769 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.076861 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.592163 (+515302us) raft_consensus.cc:1685] finished
0709 12:14:26.592189 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.592406 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":100,"thread_start_us":239664,"threads_started":1,"wal-append.queue_time_us":432672}
W20260709 12:14:26.603508 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56310us
I20260709 12:14:26.556432 19225 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:26.602264 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3167) took 1228 ms. Trace:
I20260709 12:14:26.606076 12494 rpcz_store.cc:276] 0709 12:14:25.373713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.822128 (+448415us) service_pool.cc:224] Handling call
0709 12:14:25.822561 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:25.822851 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.822883 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:25.823493 (+   610us) log.cc:844] Serialized 26 byte log entry
0709 12:14:26.120752 (+297259us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:26.120814 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.120927 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.602025 (+481098us) raft_consensus.cc:1685] finished
0709 12:14:26.602048 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.602200 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":296722,"threads_started":1,"wal-append.queue_time_us":543692}
I20260709 12:14:26.628604 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:26.638021 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.643041 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:26.602975 19179 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:26.649453 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:26.568511 19115 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:14:26.584910 12390 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 6 to 10, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 10 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:26.636878 18812 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:26.634614 19215 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:26.664417 12570 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:26.639160 19153 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:26.667022 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:26.661803 19003 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:26.670243 19140 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:26.686360 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.687523 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:26.690165 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:26.672928 19215 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:26.695032 12570 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:26.699337 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:26.704732 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:26.704922 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:26.708294 12488 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:26.708382 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:26.712604 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:26.688493 18811 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:26.717284 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:26.717916 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.695214 19153 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:26.723068 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:26.723999 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:26.724658 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:26.725704 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:26.726457 19218 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.178s	user 0.002s	sys 0.000s
W20260709 12:14:26.730504 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95508us
W20260709 12:14:26.735281 19221 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:14:26.741760 12567 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:26.712455 19231 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:26.745694 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:26.748036 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:26.702183 18927 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:26.753604 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:26.732184 13259 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:26.733331 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:26.761030 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:26.766979 12570 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:26.767343 12564 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:26.714274 18946 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:26.773499 19153 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.775250 19241 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:26.773594 18986 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:26.775579 19209 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:14:26.776335 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:26.789201 12565 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:26.789954 19215 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:26.791755 19215 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:26.777740 19222 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:14:26.701149 18812 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:26.790120 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3169) took 1299 ms. Trace:
I20260709 12:14:26.796890 12489 rpcz_store.cc:276] 0709 12:14:25.490617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.844193 (+353576us) service_pool.cc:224] Handling call
0709 12:14:25.856186 (+ 11993us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:25.909680 (+ 53494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:25.909735 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:25.950235 (+ 40500us) log.cc:844] Serialized 26 byte log entry
0709 12:14:26.376191 (+425956us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:26.376245 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:26.376320 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:26.725597 (+349277us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:26.789717 (+ 64120us) raft_consensus.cc:1685] finished
0709 12:14:26.789743 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:26.790033 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38712,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":6923,"spinlock_wait_cycles":4918656,"thread_start_us":313548,"threads_started":1,"wal-append.queue_time_us":467811}
I20260709 12:14:26.802865 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:26.803292 19171 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.613s	user 0.006s	sys 0.007s
I20260709 12:14:26.803974 19171 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:26.830423 19172 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.830726 19163 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:26.840391 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:26.831662 19234 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:26.784705 18948 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:26.791786 19176 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:14:26.834044 19006 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.811s	user 0.010s	sys 0.003s
I20260709 12:14:26.834995 19009 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:26.778491 19010 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.445s	user 0.016s	sys 0.003s
I20260709 12:14:26.805902 12565 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:26.844211 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:26.848865 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:26.850989 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48234us
W20260709 12:14:26.855582 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110112us
I20260709 12:14:26.804796 18986 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:26.772974 19231 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:26.872805 12637 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:14:26.800252 18927 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.848044 19224 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:26.869840 19021 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.003s
I20260709 12:14:26.844405 19234 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:26.886940 12492 raft_consensus.cc:2368] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18.
I20260709 12:14:26.887799 19236 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:26.889081 19235 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:14:26.889521 19244 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:14:26.890331 19230 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:26.890530 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:26.907917 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:26.893937 19234 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:26.904619 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 198185us
I20260709 12:14:26.904896 12495 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
W20260709 12:14:26.910622 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:26.925863 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:26.905777 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 142215us
I20260709 12:14:26.893591 19224 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:26.911299 18761 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.209s	user 0.013s	sys 0.004s
W20260709 12:14:26.994833 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 462380us
I20260709 12:14:27.092146 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:27.000226 19243 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:14:26.894739 19021 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.925808 19236 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:27.074944 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:27.074695 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 165610us
I20260709 12:14:27.074718 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:26.952870 19231 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 17.6, Last appended by leader: 6, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:26.950632 18986 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.075681 18812 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.080971 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.078979 18970 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.081988 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:27.091714 19183 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.789s	user 0.010s	sys 0.013s
W20260709 12:14:27.086485 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196161us
I20260709 12:14:27.085754 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:27.076713 19168 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:14:27.097967 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2530) took 1952 ms (client timeout 2175 ms). Trace:
I20260709 12:14:27.081712 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2851) took 1278 ms. Trace:
I20260709 12:14:27.085573 19167 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:27.088356 19209 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:27.112342 13261 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:27.105574 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.107311 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:27.142995 12493 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:27.134354 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:27.160696 12637 rpcz_store.cc:276] 0709 12:14:25.803131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.806573 (+  3442us) service_pool.cc:224] Handling call
0709 12:14:27.081432 (+1274859us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:27.081594 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2430294400}
I20260709 12:14:27.171315 12489 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:27.115480 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:27.148293 12564 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:27.175570 18969 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.634s	user 0.013s	sys 0.003s
W20260709 12:14:27.163364 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:27.178710 18812 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:14:27.178936 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86876.4us
W20260709 12:14:27.180429 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74938.1us
I20260709 12:14:27.180753 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.181238 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.113961 18970 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.180459 19246 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
W20260709 12:14:27.184476 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69145.2us
I20260709 12:14:27.143752 19224 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.118757 12566 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:27.161514 12495 rpcz_store.cc:269] 0709 12:14:25.145421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:25.165014 (+ 19593us) service_pool.cc:224] Handling call
0709 12:14:27.097740 (+1932726us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:27.097894 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3834763648}
W20260709 12:14:27.167748 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57815.7us
I20260709 12:14:27.184568 19244 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:27.188131 19244 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:27.188666 19244 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:27.192258 12564 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:27.194528 19242 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:27.195145 19242 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:27.113560 12390 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 5 to 8, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 8 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:27.196246 19242 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.143242 19226 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:27.144362 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69934.6us
I20260709 12:14:27.199369 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:27.200630 19172 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:27.201033 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:27.181612 19174 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:27.198107 19226 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.202520 19163 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:27.196720 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:27.203778 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:27.219213 12564 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:27.220291 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:27.167771 19168 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:27.115720 19225 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.116469 19230 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:27.224845 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:27.159793 19167 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:27.116680 19245 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.223394 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.144064 19153 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.204988 19242 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:27.231714 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:27.244896 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64202.2us
I20260709 12:14:27.216787 19246 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:27.266518 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.267230 12564 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:27.225781 19026 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:27.221326 19174 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:27.258036 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.277664 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.264654 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:27.273830 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:27.205979 19234 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.280277 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.284801 12569 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:27.285247 19233 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:27.285820 19233 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:27.286123 19223 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:27.286410 19233 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:27.288368 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.285833 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.290295 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.290904 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:27.253171 19225 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.285853 13257 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:27.288394 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.292865 13259 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:27.293068 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.293905 19246 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:27.283823 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.294405 19234 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:27.295094 19246 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.291847 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.301256 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.300505 12569 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:27.210246 18811 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:14:27.167871 19183 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.285744 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.327940 12567 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:14:27.256939 19004 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:27.324638 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60114.7us
I20260709 12:14:27.331787 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:27.332504 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131574us
I20260709 12:14:27.335994 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.336370 12569 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:27.325783 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:27.311661 19212 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:14:27.347700 19209 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:27.337024 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.293480 19225 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.259560 19215 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:14:27.377177 12610 leader_election.cc:400] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18.
I20260709 12:14:27.379134 12610 leader_election.cc:403] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:14:27.387247 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51333.9us
W20260709 12:14:27.398329 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115804us
I20260709 12:14:27.406000 12571 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:27.471678 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.472462 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
W20260709 12:14:27.411728 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200683us
I20260709 12:14:27.470300 19244 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.475449 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:27.418123 19226 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.280952 19245 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.476440 19244 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.478223 19245 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.318115 19230 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.429870 12568 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:27.375604 19115 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:27.356839 18946 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:27.355260 18792 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:27.398236 19248 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.430908 19225 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:27.481618 18948 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.626s	user 0.015s	sys 0.002s
I20260709 12:14:27.454520 12569 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:27.399770 19209 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.410360 19168 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.400960 19174 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.406490 19227 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:27.457048 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:27.376721 12572 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:27.507081 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:27.460315 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:27.464251 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:27.508158 12641 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:27.509119 12643 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:14:27.466109 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:27.466797 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:27.467051 19183 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.465366 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:27.469549 19176 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.473435 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.481179 12567 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:27.483649 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 251384us
I20260709 12:14:27.334079 19210 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:27.490207 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:27.490885 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:27.481454 19006 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:14:27.504762 12461 leader_election.cc:400] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:27.508087 13258 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:14:27.510571 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:27.509835 12639 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15. Re-sending same reply.
I20260709 12:14:27.512848 12637 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:14:27.541887 12461 leader_election.cc:403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:14:27.502827 19220 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:27.512470 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.517247 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:27.543525 12642 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:27.507082 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:27.496057 19245 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.547328 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.548549 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.554432 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.493222 18946 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.526113 18948 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:27.491277 18792 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.556372 12567 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:27.491586 19225 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.559171 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:27.560859 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85502.1us
I20260709 12:14:27.569317 13262 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:27.538731 19227 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:27.543406 19217 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:27.568257 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 212770us
I20260709 12:14:27.572556 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.572968 19249 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:27.573199 18946 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:27.506812 19248 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.560585 19220 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:27.575273 19220 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.575968 19248 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.580354 18927 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.576067 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.570291 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:27.582571 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:27.552757 19244 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:27.550767 18969 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:27.553593 19242 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:27.585162 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:27.510941 18761 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 12:14:27.593559 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:27.538364 19176 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.559005 19183 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:27.566776 19168 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:27.562599 19262 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:27.663676 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.664710 12564 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 7.
W20260709 12:14:27.606305 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:27.599231 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:27.567094 18948 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:27.554129 19245 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.600643 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.659135 13262 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:27.621462 18761 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:27.550024 19210 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:27.665934 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66830.5us
I20260709 12:14:27.713689 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:27.623674 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67664.8us
I20260709 12:14:27.680109 19227 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.8, Last appended by leader: 8, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.684798 19233 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.685799 19262 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:27.685930 19140 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:27.686688 19242 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:27.690250 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:27.740417 12567 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:27.740806 12640 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 14: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:27.694039 18948 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:27.698860 19230 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.596230 18969 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:27.707722 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.707510 12569 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:27.598570 19249 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.709410 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:27.703454 19183 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:27.721155 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.754026 12564 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:27.712713 18761 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:27.724843 19010 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:27.591234 19217 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:27.764240 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133728us
I20260709 12:14:27.772675 12569 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:14:27.732540 19026 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:27.775314 12569 pending_rounds.cc:85] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 8
I20260709 12:14:27.744597 19242 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:27.775919 12569 pending_rounds.cc:107] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:14:27.739652 19006 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18.
I20260709 12:14:27.773226 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:27.779800 12569 raft_consensus.cc:2889] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:27.751681 18969 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:27.754037 19115 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:27.768859 18761 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:27.782819 12646 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:27.768657 19230 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.793395 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.768862 19140 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:27.773694 19010 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:27.773067 19183 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.781742 19242 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:27.810758 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 141771us
I20260709 12:14:27.801285 12571 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 12:14:27.804476 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 88456.7us
I20260709 12:14:27.780718 19223 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:27.781260 19009 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:27.820724 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45270.5us
I20260709 12:14:27.796516 19006 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:27.798503 19010 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:27.818984 18927 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:27.769603 19249 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:27.836191 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60459.5us
I20260709 12:14:27.828581 19179 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:27.799324 19230 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.842228 19172 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:27.833371 19026 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:27.842711 19244 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:27.848523 19179 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:27.868917 18792 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:27.840778 19006 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.870661 19225 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.846894 18927 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.845151 19231 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
W20260709 12:14:27.898639 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 148305us
I20260709 12:14:27.821874 19140 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.883852 19179 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:27.893083 19265 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.296s	user 0.001s	sys 0.001s
I20260709 12:14:27.829100 19262 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.863241 19272 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:14:27.906214 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:27.914855 19275 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:14:27.916190 19021 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:27.918516 19273 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:14:27.919257 19006 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:27.922258 19274 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:14:27.903225 19245 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.893837 19176 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.868059 19279 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:14:27.932122 19279 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:27.932574 19279 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:27.878645 19026 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.909476 19233 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.935621 19026 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.935631 18948 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.941738 19230 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.942710 18948 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:27.944483 19210 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.946004 19115 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:27.946619 19115 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:27.948379 18927 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:27.947944 19115 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:27.949055 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:27.950591 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:27.951850 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.954370 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:27.955211 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44138.7us
I20260709 12:14:27.956282 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.954627 19244 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:27.957257 19281 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:14:27.958233 19244 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:27.958304 19281 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:27.958715 19244 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:27.959185 19244 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:27.957522 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:27.901966 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.985983 12567 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:27.982012 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80260.7us
I20260709 12:14:27.990221 12460 leader_election.cc:400] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18.
I20260709 12:14:27.991489 12536 leader_election.cc:400] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
I20260709 12:14:27.991998 12536 leader_election.cc:403] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:27.946614 19183 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:27.995872 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:27.959792 19281 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:27.981453 19282 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:27.980744 18986 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:28.000962 19268 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:28.001744 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167452us
I20260709 12:14:28.002161 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:27.988307 18792 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:14:28.003921 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52119.7us
I20260709 12:14:27.990409 19259 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:28.007261 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.007719 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:28.005985 19249 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:27.933478 19279 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.008989 19259 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.022218 18792 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.010421 19231 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:28.036130 19231 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:28.036737 19231 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:28.036507 18792 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:28.038153 19253 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.639s	user 0.012s	sys 0.013s
I20260709 12:14:27.999734 19283 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:28.004493 18986 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:28.028622 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:28.002892 19282 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:28.035044 19174 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:28.041155 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3178) took 1353 ms. Trace:
I20260709 12:14:28.049903 12491 rpcz_store.cc:276] 0709 12:14:26.687422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:26.691514 (+  4092us) service_pool.cc:224] Handling call
0709 12:14:28.040896 (+1349382us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:28.041076 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2538866048}
I20260709 12:14:28.035084 19140 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:14:28.033919 19259 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:14:28.058513 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50855.7us
I20260709 12:14:28.060252 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:28.060961 19279 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:28.062390 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:28.063990 19249 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:28.065097 19249 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.066381 19289 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.066694 18970 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.067076 19289 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.054979 19268 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:28.052660 19282 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:28.065891 19249 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.073992 19290 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:14:28.080613 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69375us
I20260709 12:14:28.049866 18986 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:28.079170 19255 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.680s	user 0.005s	sys 0.000s
I20260709 12:14:28.079315 19249 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:28.069504 19289 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.086261 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.090147 13253 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:28.030176 19010 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.103041 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:28.114845 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107713us
I20260709 12:14:28.044486 18761 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.121222 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:28.126410 19244 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.126981 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.127871 19010 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.129156 19244 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.096771 19227 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.138511 12642 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:28.130317 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.133441 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.139585 12637 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:28.140626 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.129945 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.144557 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.141240 12637 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:14:28.145004 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:28.144737 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.146193 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.146785 12637 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:28.151783 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:28.152038 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:28.152814 13218 connection.cc:256] RPC call timeout handler was delayed by -1.2909811090000858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20260709 12:14:28.153246 13218 connection.cc:256] RPC call timeout handler was delayed by -1.2018811089999417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260709 12:14:28.153674 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1514811090000876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
I20260709 12:14:28.114404 19249 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:28.152486 12644 raft_consensus.cc:2376] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19. Re-sending same reply.
W20260709 12:14:28.154050 13218 connection.cc:256] RPC call timeout handler was delayed by -1.0650811089999479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
I20260709 12:14:28.159225 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:28.159574 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 168298us
I20260709 12:14:28.159889 12642 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:28.160557 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:28.160926 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.161438 19281 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:28.161936 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.162393 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:28.162516 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.162992 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.163215 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.163758 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:28.158115 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.166922 12637 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:28.161881 19281 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.186458 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.189388 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:28.195030 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:14:28.202868 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:28.134291 19179 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:28.224722 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62402.9us
I20260709 12:14:28.228567 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:14:28.230329 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68507.8us
I20260709 12:14:28.231168 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.191704 19281 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:28.232470 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.187091 19289 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:28.232781 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.238082 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:28.238436 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:28.234772 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.236619 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:28.238771 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:28.149744 19227 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:28.232802 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43511.2us
I20260709 12:14:28.238651 12497 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 7.
I20260709 12:14:28.189354 18970 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:28.241850 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78725.6us
I20260709 12:14:28.241839 12496 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:28.244282 12566 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:28.244261 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:28.188896 19153 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:28.248909 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:28.230922 19223 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:28.252355 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:28.228288 19249 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:14:28.253943 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.250672 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:28.250874 12572 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:28.251623 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.258510 12563 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:28.258955 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.259747 12641 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:28.229427 19296 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:28.260835 12489 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:28.262655 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:28.187012 19244 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:14:28.263266 12567 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:28.241619 19289 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.265202 12496 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:14:28.266620 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.277483 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:28.240873 19010 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:28.280339 12497 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:14:28.266047 19268 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:28.285849 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:28.286191 19268 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:28.286480 19291 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:14:28.286744 19268 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:28.287007 19236 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.287356 18927 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:28.287868 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43700us
I20260709 12:14:28.288813 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.290951 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.292521 19260 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:28.293018 19260 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:28.293452 19260 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:28.296499 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.296638 19220 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:14:28.300916 19252 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.817s	user 0.003s	sys 0.008s
I20260709 12:14:28.291585 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:28.262950 19287 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.182s	user 0.002s	sys 0.000s
I20260709 12:14:28.252836 19223 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:28.307214 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.307102 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.314766 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:28.315306 12492 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:28.302796 19172 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:28.264513 19296 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:28.278988 19289 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:28.303910 12497 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:28.322400 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.323945 12496 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 14.
I20260709 12:14:28.279052 19010 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.322748 12644 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:14:28.327646 12644 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 16. Ops: []
I20260709 12:14:28.334060 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.334954 12568 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:28.317009 19223 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:28.338946 19227 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:28.305735 19220 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:28.340025 19220 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:28.342581 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.344080 19282 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.344664 19231 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.323971 19172 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:28.264611 19153 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.302809 19244 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:28.347278 19244 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.347827 19153 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:28.348066 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 962) took 1079 ms. Trace:
I20260709 12:14:28.348460 13255 rpcz_store.cc:276] 0709 12:14:27.268957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:27.333811 (+ 64854us) service_pool.cc:224] Handling call
0709 12:14:27.377798 (+ 43987us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:27.485733 (+107935us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:27.485768 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:27.485814 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:27.490306 (+  4492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:28.347801 (+857495us) spinlock_profiling.cc:243] Waited 95.6 ms on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:28.347860 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:28.348002 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":219859200}
I20260709 12:14:28.350533 19280 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:14:28.351171 19168 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.665s	user 0.009s	sys 0.004s
I20260709 12:14:28.291745 19262 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:28.281188 19242 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.353070 19179 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.350652 18761 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.353869 19179 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:28.329300 19299 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:28.354514 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.357254 12488 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:14:28.354655 12642 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 10
I20260709 12:14:28.357530 12492 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:28.358517 12642 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:14:28.360105 12642 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:28.360540 12642 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:14:28.360841 12642 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:28.362923 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:28.365650 12491 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:28.357513 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3176) took 1671 ms. Trace:
I20260709 12:14:28.366750 12494 rpcz_store.cc:276] 0709 12:14:26.686468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:26.693895 (+  7427us) service_pool.cc:224] Handling call
0709 12:14:26.694416 (+   521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:26.837172 (+142756us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:26.837217 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:26.847688 (+ 10471us) log.cc:844] Serialized 26 byte log entry
0709 12:14:27.077771 (+230083us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:27.077820 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:27.077900 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:28.300031 (+1222131us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:28.354578 (+ 54547us) raft_consensus.cc:1685] finished
0709 12:14:28.354599 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:28.357436 (+  2837us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110204,"raft.run_cpu_time_us":3731,"raft.run_wall_time_us":42536,"spinlock_wait_cycles":3593088,"thread_start_us":351285,"threads_started":2,"wal-append.queue_time_us":492024}
I20260709 12:14:28.348177 19244 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.375756 19282 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:28.376044 19244 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:28.358134 19179 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.378919 19179 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:28.382331 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:28.383240 12496 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:28.356145 18969 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.005s
I20260709 12:14:28.324693 19296 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:28.354274 19010 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:28.390565 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83435.6us
I20260709 12:14:28.398505 12568 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:28.357748 19299 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.351111 19280 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.398800 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.407641 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.389536 19279 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:28.291718 19167 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.402526 19282 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.415817 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:28.421820 19249 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:28.422439 19249 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:28.422889 19249 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:28.424598 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 181461us
I20260709 12:14:28.425908 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:28.427426 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:28.428315 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:28.430083 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.430967 12638 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:28.428977 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.432677 12644 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:28.442945 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.403988 19299 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:28.396608 18969 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.416872 18761 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:14:28.473873 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 180258us
I20260709 12:14:28.482769 12638 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:28.484102 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:28.485277 12638 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:28.460153 19167 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:28.488418 12637 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21.
I20260709 12:14:28.463397 19233 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:28.494611 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:28.497135 12639 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:28.480006 18761 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:28.501322 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.456660 19282 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.502862 12569 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:28.486042 19242 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.501082 19233 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.507035 12644 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:28.508510 19233 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:28.510710 18969 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:28.512454 19172 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.518343 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.519886 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:28.521656 19242 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:28.522197 19242 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.521994 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.523301 19242 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.527168 19174 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:28.529827 19268 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.531613 12563 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:28.532395 19268 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.529023 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.533205 19268 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:28.533396 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.534353 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.536024 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.536363 19260 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.540838 13258 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:28.542035 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.544304 19009 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:28.545157 19009 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.551924 19263 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:28.552924 19263 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:28.553709 19263 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:28.554127 19263 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:28.505960 19305 raft_consensus.cc:2764] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:28.504670 18761 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:28.505209 19231 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:28.525133 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:28.586549 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148655us
I20260709 12:14:28.459201 19280 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:28.598821 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107609us
W20260709 12:14:28.598956 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73888.9us
I20260709 12:14:28.600082 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:28.601814 12638 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:28.578358 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:28.602170 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.603526 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:28.600229 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.614949 13262 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:14:28.620478 13257 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:28.573992 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145749us
I20260709 12:14:28.581437 18812 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:28.494886 19262 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.088s
I20260709 12:14:28.623301 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.627331 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.621912 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.629747 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.640144 13258 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:28.619329 19230 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:28.625309 19174 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:28.623184 18948 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:28.633421 19231 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:28.659785 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:28.656023 19280 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:28.657001 19230 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.653543 19153 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:28.661362 19268 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.670543 19168 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:28.660871 19174 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:28.677682 18986 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.554s	user 0.004s	sys 0.008s
W20260709 12:14:28.686123 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60996.5us
I20260709 12:14:28.643829 19179 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.722395 19299 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.723632 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:28.725015 19299 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.727154 19281 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.701140 19230 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:28.730481 19268 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.729611 19281 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.714015 19174 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:28.679352 19231 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:28.738428 19010 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.738626 19231 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.680153 19220 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.672614 19307 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:28.716717 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.717990 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:28.722869 19179 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:28.716539 19282 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:28.798754 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 182455us
W20260709 12:14:28.761148 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46483.7us
W20260709 12:14:28.828621 19257 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.399s	user 0.003s	sys 0.003s
I20260709 12:14:28.696671 19209 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:28.699949 19153 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:28.847868 12496 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:28.853616 13258 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:14:28.843580 19290 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:28.846971 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:28.852063 13256 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:14:28.861277 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:28.880565 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:28.885248 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 244647us
W20260709 12:14:28.881821 19244 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.503s	user 0.016s	sys 0.004s
I20260709 12:14:28.887566 12494 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
W20260709 12:14:28.802973 19259 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.703s	user 0.007s	sys 0.005s
I20260709 12:14:28.740933 19296 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.796893 19249 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.892127 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:28.893496 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:28.894887 12496 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:28.889302 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.858795 19320 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:28.845386 18812 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:28.852248 18948 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:28.876926 19153 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:28.902076 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:28.926075 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:28.926615 12492 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:28.903604 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5616780370000924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260709 12:14:28.927681 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5022780369999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
I20260709 12:14:28.906818 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:28.909529 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:28.910842 19179 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.891000 19209 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:28.931380 12489 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:28.912236 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:28.913923 19256 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.523s	user 0.008s	sys 0.005s
I20260709 12:14:28.915551 19248 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:28.901093 19320 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:28.919396 13258 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:14:28.919909 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:28.694181 19009 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.864811 19307 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:28.924345 12563 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:28.883441 19299 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.897586 19259 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.741135 19010 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:28.928589 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4932780369999819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
I20260709 12:14:28.713759 19305 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.198s
I20260709 12:14:28.928607 12567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:28.759948 18986 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.930835 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:28.903009 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:28.934834 19319 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:28.946035 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:28.946724 12569 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:28.950982 19311 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:28.952303 19320 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.883867 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:28.895925 18969 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:14:28.953836 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:28.956067 19248 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:29.024327 19289 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.686s	user 0.011s	sys 0.002s
I20260709 12:14:29.024917 19289 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.955237 19296 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:28.960675 19288 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.910s	user 0.000s	sys 0.003s
W20260709 12:14:28.953931 12461 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.733s (SENT)
I20260709 12:14:28.876796 19290 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:28.925978 19260 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:29.002777 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3519780369999808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20260709 12:14:29.058867 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3285780369999429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
W20260709 12:14:29.059319 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2925780370001121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260709 12:14:29.059597 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2700780370000757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
W20260709 12:14:29.059862 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2601780370000597s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1839s for a response
W20260709 12:14:29.060297 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2592780370000582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
W20260709 12:14:29.060655 12536 connection.cc:256] RPC call timeout handler was delayed by -2.18007803699993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
W20260709 12:14:29.061034 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1431780370000979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
I20260709 12:14:29.015328 12496 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:29.016850 12494 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:28.960326 19140 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.902s	user 0.010s	sys 0.003s
W20260709 12:14:28.962081 19308 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.379s	user 0.002s	sys 0.000s
I20260709 12:14:28.957831 18761 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:29.023669 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120741us
I20260709 12:14:29.025363 19307 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.025425 19248 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.961699 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:29.061407 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0828780370000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
W20260709 12:14:29.076035 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0540780369999538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20260709 12:14:29.076413 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9577780370000255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2175s for a response
W20260709 12:14:29.076682 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9568780370000241s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
W20260709 12:14:29.076948 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9163780369999586s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
W20260709 12:14:29.077216 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8911780369999178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20260709 12:14:29.077528 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8524780370000826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2292s for a response
W20260709 12:14:29.077888 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6265780369999447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2543s for a response
I20260709 12:14:28.940932 19179 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:28.956930 19163 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:28.964596 19230 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.049901 12572 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.069818 12490 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:29.068509 19140 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16.
W20260709 12:14:29.078181 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5743780370000877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2601s for a response
W20260709 12:14:29.085573 12536 connection.cc:256] RPC call timeout handler was delayed by -1.527578037000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26530000000000004s for a response
W20260709 12:14:29.086344 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3457780369999455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28550000000000003s for a response
I20260709 12:14:29.084801 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:29.014081 19299 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.079280 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:29.089226 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:29.060237 19320 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:29.091033 12492 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
W20260709 12:14:29.087056 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133162us
I20260709 12:14:29.090852 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:29.018025 19263 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.125353 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:29.083837 19220 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:29.130587 12564 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:14:29.132172 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:29.080139 19324 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:29.141103 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.082376 19163 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:29.143352 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.021255 19319 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.144683 12567 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.141537 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:29.143405 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.147593 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:29.156021 13262 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:29.156327 12496 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:29.156066 12563 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:14:29.140959 19223 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.803s	user 0.007s	sys 0.008s
W20260709 12:14:29.141521 19329 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
W20260709 12:14:29.171247 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82185.7us
I20260709 12:14:29.141773 18969 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.180433 12497 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:29.188691 19332 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.189579 19332 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.193158 19332 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:29.195284 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54042.3us
I20260709 12:14:29.196732 12564 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:29.198030 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.199206 12564 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:29.082114 18948 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.214648 18761 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.154606 19324 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:29.200158 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.203526 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.162562 19010 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.155016 19140 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:29.079470 18812 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.223812 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:29.230336 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:29.231273 12571 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.238044 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:29.241835 12564 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22.
I20260709 12:14:29.130167 19209 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.243743 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.245777 12564 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 15: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.245673 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:29.251418 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.262182 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.262982 12571 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.263556 13256 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:29.264271 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.264931 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.247207 18761 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:29.220224 19249 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:29.255283 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47923.5us
I20260709 12:14:29.259249 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.200167 18969 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.260254 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:29.273356 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:29.260366 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:29.179474 19223 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.265210 12571 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.266378 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:29.237304 19010 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.270184 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.279368 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.280251 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.281102 19321 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.295084 19223 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:29.295611 19336 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:29.286747 19226 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:29.296204 19336 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:29.296648 19336 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:29.296756 19226 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:29.297866 13255 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:29.298274 19263 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.282202 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.279455 18986 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.282241 19335 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.013s
I20260709 12:14:29.312070 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.267123 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.291056 12571 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:29.291019 12640 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 15: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:29.291850 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:29.294374 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.288358 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:29.315455 12645 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:29.295101 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2873) took 1833 ms (client timeout 1733 ms). Trace:
W20260709 12:14:29.302786 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:29.307121 19331 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.050s	user 0.001s	sys 0.001s
I20260709 12:14:29.290026 19259 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.274436 18761 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:29.308413 19333 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:14:29.308966 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.281118 18969 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:29.310046 19226 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.314643 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3012) took 1695 ms. Trace:
I20260709 12:14:29.317909 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.332158 12638 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:14:29.316195 12646 rpcz_store.cc:269] 0709 12:14:27.461576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:27.481974 (+ 20398us) service_pool.cc:224] Handling call
0709 12:14:29.294872 (+1812898us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:29.295013 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553304320}
W20260709 12:14:29.322656 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54008.1us
I20260709 12:14:29.328493 12565 rpcz_store.cc:276] 0709 12:14:27.619376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:27.621997 (+  2621us) service_pool.cc:224] Handling call
0709 12:14:27.667386 (+ 45389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:27.667629 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:27.885771 (+218142us) log.cc:844] Serialized 14 byte log entry
0709 12:14:27.980066 (+ 94295us) log.cc:844] Serialized 14 byte log entry
0709 12:14:27.980360 (+   294us) log.cc:844] Serialized 14 byte log entry
0709 12:14:27.994664 (+ 14304us) log.cc:844] Serialized 14 byte log entry
0709 12:14:28.959744 (+965080us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b18004bbda0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:29.240681 (+280937us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b18004bbda0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:14:29.314208 (+ 73527us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:29.314273 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:29.314338 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:29.314414 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:29.314574 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13522,"prepare.queue_time_us":3204228,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":126,"raft.queue_time_us":23815,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7614,"spinlock_wait_cycles":63724672,"thread_start_us":335697,"threads_started":3,"wal-append.queue_time_us":140476,"wal-append.run_cpu_time_us":2072,"wal-append.run_wall_time_us":1002113}
I20260709 12:14:29.339267 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:29.357839 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66067.8us
I20260709 12:14:29.233145 19163 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.363555 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.364871 12644 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.365752 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.367466 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.368979 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.370319 12638 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:14:29.371708 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.335781 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:29.359779 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:29.289363 19248 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.371982 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.375499 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:29.377206 12638 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:14:29.383864 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.324522 18986 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:14:29.360858 18969 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:29.391767 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.387144 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:29.399098 12497 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:29.378305 19163 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:29.379319 19259 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:29.295215 19321 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.390611 19335 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:29.386933 19299 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.414181 19338 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:29.415686 18761 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:29.397190 18969 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:29.405988 19230 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:29.428459 19319 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.431092 19010 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:29.431605 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:29.386196 19248 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.432868 12494 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:29.413084 19342 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:29.436964 19168 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:29.447962 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:29.462837 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:29.452160 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:29.443651 18969 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:29.447279 19325 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:29.453054 18792 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.465390 18792 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:29.450412 19280 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.753s	user 0.013s	sys 0.000s
I20260709 12:14:29.489408 12645 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:29.456920 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.455057 19281 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:14:29.442726 19174 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.705s	user 0.012s	sys 0.000s
I20260709 12:14:29.446645 19321 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:29.359405 19324 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.459869 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:29.440237 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:29.455538 19153 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.512s	user 0.007s	sys 0.006s
W20260709 12:14:29.461560 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 149563us
W20260709 12:14:29.539829 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 180352us
I20260709 12:14:29.561879 12489 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:29.565191 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:14:29.629753 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 57922.5us
W20260709 12:14:29.630295 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:29.630625 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.483690 19168 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:29.669987 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.671195 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:29.681900 19230 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:29.543720 19174 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.692426 12646 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: []
W20260709 12:14:29.695497 19320 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.000s
I20260709 12:14:29.632089 19153 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:29.697265 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:29.699954 12639 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19.
I20260709 12:14:29.500053 19281 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:29.732311 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:29.735893 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 105374us
I20260709 12:14:29.736336 12496 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:29.736717 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.738122 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:29.738482 12645 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:29.489029 19259 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.167s
I20260709 12:14:29.738971 12494 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:29.750276 12462 leader_election.cc:336] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 2.147s (SENT)
I20260709 12:14:29.735997 19320 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:29.629323 18986 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:14:29.757042 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 297368us
W20260709 12:14:29.767954 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:29.795238 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:29.775926 12645 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:14:29.753181 19263 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:14:29.733366 19321 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:29.782516 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.753690 19010 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.810083 12497 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:29.742794 19153 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:29.790277 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.772153 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:29.812048 12494 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:29.752141 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.642293 19178 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.135s
I20260709 12:14:29.820714 12496 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:14:29.874707 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 122657us
I20260709 12:14:29.877024 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:29.878316 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:29.877381 12645 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 12:14:29.796901 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 415001us
I20260709 12:14:29.873729 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:29.880666 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:29.877079 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2888) took 1755 ms (client timeout 2237 ms). Trace:
I20260709 12:14:29.796872 19248 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:29.897221 12492 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:14:29.894665 12641 rpcz_store.cc:269] 0709 12:14:28.122171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.151402 (+ 29231us) service_pool.cc:224] Handling call
0709 12:14:29.876928 (+1725526us) spinlock_profiling.cc:243] Waited 97 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:29.877026 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223098240}
I20260709 12:14:29.901746 12645 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:14:29.898547 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.814956 19174 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:29.899050 19209 raft_consensus.cc:2764] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:29.919829 19332 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:29.801575 19320 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.923689 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.902570 18761 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:29.901659 18812 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:29.817983 19321 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:29.932950 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:29.885605 18792 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:29.908802 19153 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:29.936776 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:29.944219 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:29.939105 12492 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [19.10-20.11]
I20260709 12:14:29.911675 19299 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:29.831321 19010 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.936522 19332 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:29.807585 19281 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.948128 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:29.949054 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:29.956969 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.950003 13258 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:29.950860 19338 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:29.896166 19009 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:14:29.927983 19226 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:29.976693 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:29.944129 19178 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:29.930725 18761 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:29.959321 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:29.963356 19338 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.996254 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2886) took 1856 ms. Trace:
I20260709 12:14:29.996867 12642 rpcz_store.cc:276] 0709 12:14:28.121477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.243796 (+122319us) service_pool.cc:224] Handling call
0709 12:14:28.262086 (+ 18290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:28.383982 (+121896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:28.384022 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:28.385914 (+  1892us) log.cc:844] Serialized 26 byte log entry
0709 12:14:28.586183 (+200269us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:28.626303 (+ 40120us) log.cc:844] Serialized 14 byte log entry
0709 12:14:28.628945 (+  2642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:29.960550 (+1331605us) spinlock_profiling.cc:243] Waited 92.3 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:29.960613 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:29.960662 (+    49us) raft_consensus.cc:1685] finished
0709 12:14:29.977484 (+ 16822us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:29.977725 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39474,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":2349,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":11565,"spinlock_wait_cycles":214167168,"thread_start_us":172793,"threads_started":1,"wal-append.queue_time_us":304003,"wal-append.run_cpu_time_us":2925,"wal-append.run_wall_time_us":1206158}
I20260709 12:14:29.963721 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:29.963970 12565 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:29.960932 19299 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:29.955231 19153 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:29.957793 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:29.977615 12564 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:29.978338 12571 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:14:29.995756 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117522us
I20260709 12:14:30.000444 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:30.001892 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:30.009742 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50507.7us
I20260709 12:14:30.015671 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.017198 12609 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22.
I20260709 12:14:29.954888 18792 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:30.023350 12564 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:14:30.025517 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.030197 12565 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:30.040244 12565 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:30.008477 18761 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:30.015039 19174 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:30.032473 19230 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:30.032094 19226 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:30.014644 19321 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:30.054088 12565 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:30.035243 19332 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:30.045305 19009 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:14:30.074826 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 279686us
I20260709 12:14:30.036499 19168 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.058658 19226 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:30.078851 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:30.081908 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67609.1us
W20260709 12:14:30.090545 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 340301us
I20260709 12:14:30.055531 18812 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:14:30.093606 12497 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:30.073012 19009 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:30.100353 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.100795 19168 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.087818 19263 consensus_queue.cc:1243] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:14:30.101022 19163 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.000s
I20260709 12:14:30.102689 19338 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:30.103910 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.104887 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.105098 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.106200 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.082445 19226 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:30.109833 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.105815 13255 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:30.114509 19351 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:30.115284 19351 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.117760 19351 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.117882 19168 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:30.119655 19153 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.120554 19320 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.097491 18792 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:14:30.128401 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.130834 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.132547 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.133330 12565 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:30.110450 19246 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.677s	user 0.012s	sys 0.019s
I20260709 12:14:30.135990 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.142208 19009 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:30.143131 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3025) took 1607 ms (client timeout 1656 ms). Trace:
I20260709 12:14:30.142091 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3211) took 1220 ms. Trace:
I20260709 12:14:30.148173 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:30.152379 12572 rpcz_store.cc:269] 0709 12:14:28.535556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.535736 (+   180us) service_pool.cc:224] Handling call
0709 12:14:30.142902 (+1607166us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:30.143056 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1158136064}
I20260709 12:14:30.154026 12494 rpcz_store.cc:276] 0709 12:14:28.921197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.821928 (+900731us) service_pool.cc:224] Handling call
0709 12:14:29.822427 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:29.822920 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:29.822961 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:29.884784 (+ 61823us) log.cc:844] Serialized 26 byte log entry
0709 12:14:29.976836 (+ 92052us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:29.976879 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:29.976958 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:30.141750 (+164792us) raft_consensus.cc:1685] finished
0709 12:14:30.141774 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:30.142005 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":53311,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":12955,"thread_start_us":55138,"threads_started":1,"wal-append.queue_time_us":186628}
I20260709 12:14:30.106704 19311 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:14:30.214126 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 808) took 2651 ms. Trace:
I20260709 12:14:30.189083 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.190338 19010 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.214637 13253 rpcz_store.cc:276] 0709 12:14:27.558331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:27.558487 (+   156us) service_pool.cc:224] Handling call
0709 12:14:27.575690 (+ 17203us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:28.110915 (+535225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:30.209356 (+2098441us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:30.209501 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1170794880}
I20260709 12:14:30.151651 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:30.215853 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64279.8us
I20260709 12:14:30.217015 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:30.150959 19272 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:30.192533 19336 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.895s	user 0.016s	sys 0.000s
I20260709 12:14:30.192517 19172 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.193415 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:30.194289 19168 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:30.094457 19299 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.203732 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3212) took 1282 ms. Trace:
I20260709 12:14:30.112479 19338 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:30.151688 19153 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:30.155916 18812 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:14:30.218142 12565 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:30.222357 19336 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.229447 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.230618 12565 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:30.231015 19336 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:30.231606 12641 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:30.231797 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.223845 12496 rpcz_store.cc:276] 0709 12:14:28.921489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.831229 (+909740us) service_pool.cc:224] Handling call
0709 12:14:29.831684 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:29.831971 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:29.832007 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:29.832580 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:14:30.053184 (+220604us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:30.053239 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:30.053350 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:30.203450 (+150100us) raft_consensus.cc:1685] finished
0709 12:14:30.203472 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:30.203660 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":130988,"threads_started":1,"wal-append.queue_time_us":197225}
W20260709 12:14:30.222075 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117067us
I20260709 12:14:30.193308 19347 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:30.235010 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:30.148181 19246 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.718s	user 0.013s	sys 0.020s
I20260709 12:14:30.226460 19362 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:30.221170 12494 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:30.233489 12565 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:30.233701 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:30.224475 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.236119 12567 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:30.229971 19226 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.222973 19010 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:30.216516 19209 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:30.240075 12572 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:30.238364 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:30.240707 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3218) took 1025 ms. Trace:
I20260709 12:14:30.240844 12496 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:30.243439 12494 rpcz_store.cc:276] 0709 12:14:29.215512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.220713 (+1005201us) service_pool.cc:224] Handling call
0709 12:14:30.221027 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:30.240387 (+ 19360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:30.240642 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:30.244246 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3219) took 1028 ms. Trace:
I20260709 12:14:30.224298 19168 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:30.244899 12496 rpcz_store.cc:276] 0709 12:14:29.215879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.233115 (+1017236us) service_pool.cc:224] Handling call
0709 12:14:30.240663 (+  7548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:30.243645 (+  2982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:30.244124 (+   479us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:30.248974 12571 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:14:30.224447 19338 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:30.235033 19347 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:30.254489 19347 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:30.255542 19226 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.256628 12642 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:30.253080 12571 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:30.242293 19356 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:14:30.258378 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.234660 18792 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:14:30.259986 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:30.260591 18969 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.000s
I20260709 12:14:30.260699 13253 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:30.260964 18969 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.250000 19172 raft_consensus.cc:2764] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:14:30.241199 19355 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:14:30.268716 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:30.269273 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.272411 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.269771 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:30.252691 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:30.272636 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.222471 19272 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.243273 19009 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:30.278354 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:30.283711 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:30.252066 19362 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:30.291277 19321 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:30.291863 19321 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:30.292351 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2895) took 1864 ms (client timeout 1536 ms). Trace:
I20260709 12:14:30.277215 19227 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:30.292769 19321 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:30.293076 12637 rpcz_store.cc:269] 0709 12:14:28.428073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.442542 (+ 14469us) service_pool.cc:224] Handling call
0709 12:14:30.292099 (+1849557us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:30.292253 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53711616}
I20260709 12:14:30.265198 19289 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.302034 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.303033 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:30.296648 18969 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:30.303903 12637 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:30.304167 12644 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:30.304824 18812 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:30.301535 19269 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:30.306398 19172 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:30.306535 19269 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:30.307096 19269 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:30.308025 19266 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.640s	user 0.001s	sys 0.017s
I20260709 12:14:30.301414 19362 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:30.271204 19178 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:30.310472 19227 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:30.299036 19226 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:30.313256 19227 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:30.313839 19227 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:30.311085 18812 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:30.318439 18812 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.319427 19224 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:30.320025 19224 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:30.311734 19362 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.307117 19172 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:30.311344 19359 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:14:30.322865 19172 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.323760 12638 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 5. Preceding OpId from leader: term: 19 index: 7. (index mismatch)
W20260709 12:14:30.331288 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58920us
I20260709 12:14:30.332636 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:30.339323 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.343262 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:30.353384 13217 connection.cc:256] RPC call timeout handler was delayed by -1.4220895299999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
W20260709 12:14:30.364884 19363 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:14:30.352890 19228 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:30.369077 19228 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.297318 19366 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:14:30.335956 19174 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:14:30.371018 19174 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:30.371433 19174 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:30.371474 19272 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.359699 19230 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:30.359856 19323 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:30.373517 19323 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.374963 19290 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.375829 19323 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.352286 19332 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:30.380898 19217 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.382277 19269 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.371624 19228 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.335397 19009 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:30.370158 19366 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:14:30.389786 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.391377 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.343624 19246 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:30.417817 12637 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:30.408418 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:30.419606 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:30.409291 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.415836 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.421208 12567 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:30.426453 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.427367 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.427729 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:30.428151 13253 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:14:30.428452 12644 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:30.427407 12643 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:30.432215 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:30.432320 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:30.359797 19296 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:30.436313 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.433363 13262 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21.
I20260709 12:14:30.435765 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.440191 13259 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:30.436389 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.359508 19010 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:30.341408 19338 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.425529 19360 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:30.420758 19009 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.448318 18761 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.448518 19168 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.451459 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:30.445369 19338 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:30.465987 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:30.451908 19342 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:30.425019 19227 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.454304 19360 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:30.426777 19246 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.458029 19296 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:30.460460 19269 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.480662 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.451301 19299 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:14:30.485896 19342 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:30.482435 19168 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:30.503207 19296 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:30.502454 19209 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:30.483412 19174 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:30.521776 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70383.6us
I20260709 12:14:30.526324 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.527391 12567 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:30.527434 13217 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.024s (SENT) [suppressed 18 similar messages]
I20260709 12:14:30.547436 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.590119 19296 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.531180 19009 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:30.614114 19374 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:14:30.616828 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.617745 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:30.618817 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.537899 19360 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.068s
W20260709 12:14:30.641047 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:30.641366 19375 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:14:30.598260 19342 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.515461 19010 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.657588 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.663470 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.673841 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:30.674547 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.681677 19360 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:30.686571 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.686936 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.687438 12571 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:30.687798 12567 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:30.688611 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.689985 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.690589 12571 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:30.694476 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.695832 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.696692 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:30.651188 19246 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:30.700500 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.662317 18761 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.659595 19321 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:30.703192 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:30.685701 19347 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.716658 19347 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.705426 19228 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:30.717844 19228 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:30.718286 19228 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:30.719890 18986 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:14:30.703544 19360 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:30.724664 19179 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.704656 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.731160 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.714710 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:30.734306 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.738662 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.665288 19172 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:30.741777 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.744506 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:30.746105 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:30.747084 13262 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:30.762430 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.765507 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:30.790513 12496 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:30.766017 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:30.791955 12489 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:30.720463 18986 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:30.798413 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.797384 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:30.799177 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.737779 19179 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:30.799798 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:30.765097 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:30.737305 19168 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:30.760596 19360 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:14:30.803097 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.737430 19320 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:30.794950 19009 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:30.802402 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:30.868675 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106360us
W20260709 12:14:30.869496 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67656.6us
I20260709 12:14:30.870976 12462 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21.
I20260709 12:14:30.737488 18761 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:30.785030 19227 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.810807 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:30.807307 19168 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.878091 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:30.807706 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:30.808317 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:30.824854 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:30.881512 12567 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:30.827813 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.764747 19377 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:30.763401 19172 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:30.858143 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110450us
I20260709 12:14:30.805948 19296 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:30.870782 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.761137 19323 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:30.871886 12462 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:14:30.827133 19228 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.910972 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.739337 19321 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.881808 19299 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:30.858443 18986 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.915570 18986 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:30.890960 12489 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:30.888634 19172 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.897336 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:30.912205 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:30.878403 19360 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:30.926927 19342 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 12:14:30.923125 19354 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:30.913318 19318 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21.
I20260709 12:14:30.957532 19388 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:30.958844 19360 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.960168 19228 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:30.961098 19228 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:30.963063 19388 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:30.964347 19388 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:30.964781 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:30.890830 18761 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.968607 18761 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.913116 19323 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:30.972539 19323 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:30.972589 19115 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:30.930751 19289 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:30.974519 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.975198 12640 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:30.976253 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.976943 12640 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17.
I20260709 12:14:30.965521 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
W20260709 12:14:30.978659 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:30.979166 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:30.980388 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:30.981389 12496 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:14:30.982230 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:30.983470 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3029) took 2092 ms (client timeout 2175 ms). Trace:
I20260709 12:14:30.973191 12564 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:14:30.983946 12566 rpcz_store.cc:269] 0709 12:14:28.891142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.891792 (+   650us) service_pool.cc:224] Handling call
0709 12:14:30.983286 (+2091494us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:30.983409 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2249294208}
I20260709 12:14:30.985445 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:30.986444 12640 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:30.963302 19168 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:30.891690 19009 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:31.000778 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113841us
I20260709 12:14:31.025269 12489 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
W20260709 12:14:31.030869 19344 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.246s	user 0.005s	sys 0.000s
I20260709 12:14:31.024940 19320 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.034730 19320 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.035553 12563 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:14:31.036968 12563 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:31.037497 19280 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.038206 19280 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.039324 12642 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:14:30.896098 19217 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.103s
I20260709 12:14:31.045578 12640 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:31.042531 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:31.049721 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:31.043884 12493 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:31.050617 12495 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:31.051635 18688 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:14:31.058086 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3190) took 2863 ms (client timeout 1741 ms). Trace:
W20260709 12:14:31.058611 12490 rpcz_store.cc:269] 0709 12:14:28.194172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.194353 (+   181us) service_pool.cc:224] Handling call
0709 12:14:31.057796 (+2863443us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.057959 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909014400}
I20260709 12:14:31.059957 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:31.060672 12490 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 14.
I20260709 12:14:31.040103 19323 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
W20260709 12:14:31.072301 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145797us
I20260709 12:14:31.078855 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:31.085320 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:31.086428 13256 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:30.994297 19228 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.067221 19115 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.126753 12489 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:31.134929 19228 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:31.129055 19115 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.145175 19115 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:31.150033 19383 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260709 12:14:31.153519 19009 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:31.168623 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.129360 19390 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:31.175678 19321 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:31.131951 19392 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:31.097918 19389 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:14:31.183287 19389 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:31.184036 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3068) took 1091 ms. Trace:
I20260709 12:14:31.184556 12572 rpcz_store.cc:276] 0709 12:14:30.093003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.253450 (+160447us) service_pool.cc:224] Handling call
0709 12:14:30.289998 (+ 36548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:30.964555 (+674557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:30.964589 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:30.964632 (+    43us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:30.964675 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.183726 (+219051us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:31.183784 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:31.183958 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1550731008}
I20260709 12:14:31.185191 12571 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 16. Ops: []
I20260709 12:14:31.188715 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2579) took 1896 ms. Trace:
I20260709 12:14:31.189219 12646 rpcz_store.cc:276] 0709 12:14:29.293228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.377545 (+ 84317us) service_pool.cc:224] Handling call
0709 12:14:29.394374 (+ 16829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:29.693097 (+298723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.188423 (+1495326us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:31.188638 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":684802432}
W20260709 12:14:31.194577 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.173276 19390 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.206152 12571 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:31.208050 12571 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:31.212054 12572 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
W20260709 12:14:31.206411 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156782us
I20260709 12:14:31.214848 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:31.183706 19389 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:31.218636 12571 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 12:14:31.193241 18688 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.219825 13218 leader_election.cc:400] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17.
I20260709 12:14:31.233944 12496 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:14:31.234098 18688 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.236150 12496 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:31.231866 19323 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260709 12:14:31.240693 19390 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.242915 19390 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:31.254199 19401 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:31.254776 19323 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:31.255362 19323 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.256346 19323 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.260342 19335 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:31.260890 19335 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:31.218150 19217 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:14:31.211892 19392 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:14:31.228937 19320 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:31.270488 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:31.267066 19323 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:31.266827 19389 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:31.283172 19386 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.156s	user 0.002s	sys 0.000s
W20260709 12:14:31.285262 19373 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.778s	user 0.001s	sys 0.001s
I20260709 12:14:31.272727 19320 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:31.286410 19320 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:31.286525 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3214) took 2314 ms. Trace:
I20260709 12:14:31.287024 12497 rpcz_store.cc:276] 0709 12:14:28.972530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.904617 (+932087us) service_pool.cc:224] Handling call
0709 12:14:29.926377 (+ 21760us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:30.094292 (+167915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.286156 (+1191864us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:31.286438 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":384034688}
W20260709 12:14:31.289160 19371 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.646s	user 0.000s	sys 0.003s
I20260709 12:14:31.291254 19217 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:31.291754 19401 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:31.292228 19401 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.292963 12567 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:31.293325 19401 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:31.289165 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3022) took 2882 ms (client timeout 1602 ms). Trace:
W20260709 12:14:31.294392 12568 rpcz_store.cc:269] 0709 12:14:28.406311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.407138 (+   827us) service_pool.cc:224] Handling call
0709 12:14:31.288861 (+2881723us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.289047 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":723867904}
I20260709 12:14:31.291785 19217 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:31.295598 19217 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:31.295807 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:31.292171 19378 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.646s	user 0.003s	sys 0.000s
W20260709 12:14:31.300045 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3189) took 3110 ms (client timeout 2025 ms). Trace:
I20260709 12:14:31.300530 12492 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
W20260709 12:14:31.300527 12488 rpcz_store.cc:269] 0709 12:14:28.189422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.190495 (+  1073us) service_pool.cc:224] Handling call
0709 12:14:31.299735 (+3109240us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540037e454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.299953 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":283504896}
I20260709 12:14:31.301057 12492 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 9
I20260709 12:14:31.301802 12492 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:14:31.302104 12492 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:31.275122 19335 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.310902 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3045) took 1008 ms. Trace:
I20260709 12:14:31.312722 12564 rpcz_store.cc:276] 0709 12:14:30.302007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:31.046191 (+744184us) service_pool.cc:224] Handling call
0709 12:14:31.086033 (+ 39842us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:31.087293 (+  1260us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:31.087332 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:31.093466 (+  6134us) log.cc:844] Serialized 26 byte log entry
0709 12:14:31.217268 (+123802us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:31.217530 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.223517 (+  5987us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:31.310508 (+ 86991us) raft_consensus.cc:1685] finished
0709 12:14:31.310543 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:31.310791 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":739,"raft.queue_time_us":4899,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":32918,"thread_start_us":62435,"threads_started":1,"wal-append.queue_time_us":135957}
I20260709 12:14:31.303442 19335 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:31.330896 19227 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.331660 19227 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.334457 19227 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:31.338529 19183 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.339427 19183 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.341871 19217 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:31.344321 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.347082 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2910) took 1394 ms. Trace:
I20260709 12:14:31.323285 19401 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:31.352871 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:31.353205 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 885) took 3250 ms (client timeout 2123 ms). Trace:
I20260709 12:14:31.360414 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.367333 12496 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:14:31.367019 12495 rpcz_store.cc:269] 0709 12:14:28.102399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.102667 (+   268us) service_pool.cc:224] Handling call
0709 12:14:31.352884 (+3250217us) spinlock_profiling.cc:243] Waited 2.93 s on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.353088 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6730044288}
I20260709 12:14:31.368139 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:31.368692 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:31.364393 12641 rpcz_store.cc:276] 0709 12:14:29.952168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.952349 (+   181us) service_pool.cc:224] Handling call
0709 12:14:31.346835 (+1394486us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.347007 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":531096448}
I20260709 12:14:31.361088 12568 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:31.369940 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.371223 12495 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:31.372439 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.372557 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.371488 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:31.373662 12488 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:31.373821 12489 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:31.374689 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.378328 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:31.379607 12497 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:31.370577 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:31.356035 19336 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:31.385684 19183 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:31.386478 12495 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 12:14:31.425105 13260 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:14:31.387588 19281 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:31.388319 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.430703 12489 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:31.431516 12568 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:31.432930 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:31.434547 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.435849 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.422559 19242 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:31.430464 19351 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:14:31.391578 19280 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:31.396034 19217 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:31.438206 19242 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.396692 19392 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:31.437430 12568 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:31.439347 19392 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.440430 19242 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:31.440346 19392 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.400019 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2557) took 2329 ms. Trace:
I20260709 12:14:31.441864 12492 rpcz_store.cc:276] 0709 12:14:29.070286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.946771 (+876485us) service_pool.cc:224] Handling call
0709 12:14:31.300119 (+1353348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:31.302647 (+  2528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.399748 (+ 97101us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540037e450. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.399935 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3111391488}
I20260709 12:14:31.401106 19228 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:31.443003 19389 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:31.443096 19228 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:31.444222 19228 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:31.405771 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1129) took 4303 ms (client timeout 2024 ms). Trace:
I20260709 12:14:31.445574 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2914) took 1155 ms. Trace:
I20260709 12:14:31.410439 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:31.410948 12572 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:14:31.446385 12572 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 14
I20260709 12:14:31.412194 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:31.424443 19336 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:31.446038 12645 rpcz_store.cc:276] 0709 12:14:30.289829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.289989 (+   160us) service_pool.cc:224] Handling call
0709 12:14:30.290282 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:30.294451 (+  4169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:30.294488 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:30.295090 (+   602us) log.cc:844] Serialized 26 byte log entry
0709 12:14:30.511922 (+216832us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:30.513428 (+  1506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.445155 (+931727us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b54002e5754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:31.445213 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:31.445262 (+    49us) raft_consensus.cc:1685] finished
0709 12:14:31.445277 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:31.445503 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6182,"raft.queue_time_us":6683,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"spinlock_wait_cycles":8603520,"thread_start_us":214865,"threads_started":1,"wal-append.queue_time_us":245197}
I20260709 12:14:31.448598 19336 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.449508 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:31.412909 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.449998 19336 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:31.450567 12493 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:31.451184 12495 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
W20260709 12:14:31.415043 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.448722 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:31.453038 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.446059 19183 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:31.385082 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:31.423820 12564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:31.428763 12491 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:31.429298 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:31.429658 12496 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:31.437413 12488 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:31.474938 12569 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:31.475476 12492 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:14:31.448719 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:31.446731 12572 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:14:31.476831 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:31.453442 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:31.478546 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.446667 13254 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:14:31.429430 13262 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:31.445870 12570 rpcz_store.cc:269] 0709 12:14:27.102077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:27.102781 (+   704us) service_pool.cc:224] Handling call
0709 12:14:31.405286 (+4302505us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.405557 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3426607872}
I20260709 12:14:31.480684 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.455152 12495 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:31.477149 12572 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:31.474444 19245 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.454324 19183 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.482442 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.490008 19245 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.491459 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3079) took 1248 ms. Trace:
I20260709 12:14:31.483960 12645 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:14:31.448544 19242 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:31.509333 12563 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:14:31.463382 19228 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:31.504914 19183 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:14:31.492015 12567 rpcz_store.cc:276] 0709 12:14:30.242940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.884286 (+641346us) service_pool.cc:224] Handling call
0709 12:14:31.292728 (+408442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:31.293399 (+   671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.491119 (+197720us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54001b26d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.491362 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934159616}
W20260709 12:14:31.510520 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61910.7us
I20260709 12:14:31.511173 19242 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:31.511356 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1135) took 1182 ms. Trace:
I20260709 12:14:31.532574 12563 rpcz_store.cc:276] 0709 12:14:30.329321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:31.044732 (+715411us) service_pool.cc:224] Handling call
0709 12:14:31.508820 (+464088us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:31.509055 (+   235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:31.509886 (+   831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.511280 (+  1394us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6450432}
I20260709 12:14:31.512920 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2897) took 2204 ms. Trace:
W20260709 12:14:31.523043 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 252696us
I20260709 12:14:31.516769 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.534736 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:31.535269 12637 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:31.536590 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:31.537992 18688 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:14:31.538722 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.539549 12642 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:31.539587 12567 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:31.538355 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.533951 12638 rpcz_store.cc:276] 0709 12:14:29.308327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.388586 (+ 80259us) service_pool.cc:224] Handling call
0709 12:14:29.391131 (+  2545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:30.324180 (+933049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.512599 (+1188419us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:31.512835 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2143896832}
I20260709 12:14:31.543249 12645 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:31.536655 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.544438 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2920) took 1149 ms. Trace:
I20260709 12:14:31.544816 12637 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:31.547307 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2574) took 2264 ms (client timeout 1801 ms). Trace:
I20260709 12:14:31.485962 19345 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.550877 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.551350 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.552574 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
W20260709 12:14:31.553766 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:31.548380 12639 rpcz_store.cc:269] 0709 12:14:29.266956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:29.267135 (+   179us) service_pool.cc:224] Handling call
0709 12:14:31.522893 (+2255758us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.531026 (+  8133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":941259008}
I20260709 12:14:31.512989 19318 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:31.544880 12643 rpcz_store.cc:276] 0709 12:14:30.395178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.450007 (+ 54829us) service_pool.cc:224] Handling call
0709 12:14:30.476423 (+ 26416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:30.504878 (+ 28455us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:30.618706 (+113828us) log.cc:844] Serialized 14 byte log entry
0709 12:14:30.669775 (+ 51069us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:30.669848 (+    73us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:30.669908 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.544179 (+874271us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:31.544242 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:31.544376 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":175295,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":170,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":64884736,"thread_start_us":158043,"threads_started":2,"wal-append.queue_time_us":51068}
I20260709 12:14:31.551288 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.564390 12640 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:31.565563 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.567500 12644 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:31.537425 12493 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:31.542773 19242 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:31.569592 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.569924 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.571939 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.542593 19324 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:31.547206 19115 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:31.553534 19217 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:31.572592 12566 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:31.580832 12639 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:31.577045 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:31.582130 12637 raft_consensus.cc:1217] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-21.12]   Dedup: 17.10->[19.11-21.12]
I20260709 12:14:31.598834 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.599814 12637 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 10
I20260709 12:14:31.600212 12637 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:14:31.600528 12637 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:31.585876 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:31.602279 12644 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:14:31.598675 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.602038 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.603724 12645 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:31.604801 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.610848 12640 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:31.573283 19402 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:31.625540 12567 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 12:14:31.635883 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.638164 12564 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:31.562785 19345 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.651331 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1144) took 1232 ms. Trace:
I20260709 12:14:31.672896 12572 rpcz_store.cc:276] 0709 12:14:30.418550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:31.227278 (+808728us) service_pool.cc:224] Handling call
0709 12:14:31.410427 (+183149us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:31.484667 (+ 74240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:31.651007 (+166340us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400181d50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.651237 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405475200}
I20260709 12:14:31.657050 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 38) took 2861 ms. Trace:
I20260709 12:14:31.662354 19245 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:31.661755 19224 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.686422 13257 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:31.578536 18688 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:14:31.674463 12491 rpcz_store.cc:276] 0709 12:14:28.807551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:28.841824 (+ 34273us) service_pool.cc:224] Handling call
0709 12:14:31.656802 (+2814978us) spinlock_profiling.cc:243] Waited 2.41 s on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:31.656925 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5548764544}
I20260709 12:14:31.677502 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:31.698613 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.701001 13253 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:31.702731 12567 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:31.705701 12493 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:31.706460 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.708129 12492 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:14:31.597400 19324 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.674466 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:31.704172 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 260509us
I20260709 12:14:31.687937 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:31.704444 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:31.680295 12646 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
W20260709 12:14:31.718482 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.721320 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.722493 13259 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:14:31.722909 13259 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 14
I20260709 12:14:31.723289 13259 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:14:31.723510 13259 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:31.703059 19009 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:31.601266 19115 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.700707 19365 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.681279 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:31.746899 12644 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:31.707096 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:31.733052 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.737558 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.748898 12490 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20. Re-sending same reply.
I20260709 12:14:31.737968 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:31.704730 19245 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.727217 18970 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:31.756073 18970 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:31.749583 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:31.765676 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:31.669966 19402 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.714565 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:31.754488 12639 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:31.716377 19282 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:31.756879 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:31.775625 12493 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:31.775919 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:31.729252 19210 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:14:31.769459 12646 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:31.727378 19224 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:31.772163 12642 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:14:31.795228 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 238585us
I20260709 12:14:31.719741 19209 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:14:31.729027 19263 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:14:31.848459 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:31.769233 18970 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:31.856070 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:31.727046 19352 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:14:31.773687 19279 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.861357 12646 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:14:31.902020 19279 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.745684 19178 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 12:14:31.904570 19279 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:31.906064 19226 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:14:31.913164 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:31.916002 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:31.918114 12563 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:31.870087 19410 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:14:31.754037 19332 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 12:14:31.759369 19365 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.875655 19140 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:31.748093 19009 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.857477 19351 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:14:31.882961 19283 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.883447 19115 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:31.885248 19318 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:31.892158 19408 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.461s	user 0.000s	sys 0.002s
I20260709 12:14:31.899183 19345 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:31.767551 19245 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.783406 19210 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:31.925051 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 200324us
W20260709 12:14:31.927196 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 169395us
I20260709 12:14:31.932807 19389 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:31.873456 19390 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.630s	user 0.006s	sys 0.008s
I20260709 12:14:31.941751 19332 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:31.947232 19209 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:31.949333 19324 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:31.934955 19140 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:31.953387 19402 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:31.955570 19365 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:32.029541 13259 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:31.924569 19412 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:31.956502 19320 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.669s	user 0.010s	sys 0.002s
I20260709 12:14:31.958105 12638 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:31.905282 19279 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.021s
W20260709 12:14:31.958089 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 182246us
I20260709 12:14:32.120836 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:31.925097 18688 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:31.988318 19353 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:14:31.989761 18970 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:31.990855 19174 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:32.175283 19245 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:32.176697 19263 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:32.177309 12640 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:14:32.177810 12640 pending_rounds.cc:85] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
I20260709 12:14:32.178022 12640 pending_rounds.cc:107] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:14:32.178257 12640 raft_consensus.cc:2889] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:31.967087 19401 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.599s	user 0.007s	sys 0.005s
I20260709 12:14:32.024932 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:31.907689 19338 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:32.024482 19332 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:31.998775 19210 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:32.181174 19332 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.9, Last appended by leader: 9, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.183048 19332 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:32.185755 19296 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.152837 19228 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:32.155707 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:32.156514 18688 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.146729 19353 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:32.159638 18970 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:32.160409 19347 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:32.164892 19365 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:14:32.041574 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:31.976835 19318 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.147034 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:32.033493 19389 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:31.964385 19283 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.121412 19402 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.077s
I20260709 12:14:32.149247 19320 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.122215 19412 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:32.187726 19335 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.863s	user 0.011s	sys 0.001s
I20260709 12:14:32.185441 19351 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:32.178701 19263 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:32.032981 19009 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.199275 19338 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:32.203496 19210 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:32.215605 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60005.8us
I20260709 12:14:32.216552 19296 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.225055 19178 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:32.233254 19347 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:32.246192 19353 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:32.250794 12497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.019s	sys 0.038s
W20260709 12:14:32.251101 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 507419us
W20260709 12:14:32.272449 12497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.020s	sys 0.038s
I20260709 12:14:32.253777 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:32.273056 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:32.252710 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 469757us
I20260709 12:14:32.245347 18688 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.239969 19174 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:32.248557 19217 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:32.294382 12563 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:32.294910 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:32.296277 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:32.270654 19242 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.001s
I20260709 12:14:32.320720 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:32.322153 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:32.336678 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:32.302733 19174 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:32.338721 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:32.299408 19365 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:14:32.346408 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1137) took 2016 ms (client timeout 1935 ms). Trace:
W20260709 12:14:32.346966 12565 rpcz_store.cc:269] 0709 12:14:30.330013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:30.618504 (+288491us) service_pool.cc:224] Handling call
0709 12:14:32.346192 (+1727688us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:32.346323 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3102719232}
I20260709 12:14:32.347363 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.316135 19338 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:32.366783 19230 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:32.367172 19338 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:32.367786 19230 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:32.367939 19338 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:32.306054 19217 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:32.351457 12571 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:14:32.352591 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.021s	sys 0.025s
I20260709 12:14:32.329507 19401 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:32.370687 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.021s	sys 0.025s
W20260709 12:14:32.371305 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 538500us
I20260709 12:14:32.356324 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.371605 19174 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.359417 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.372982 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:32.370056 19245 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:32.373792 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:32.372893 19424 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:32.352667 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:32.374253 19248 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:32.374698 18970 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:32.371447 19365 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:32.376081 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:32.371995 19347 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.373421 19009 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:14:32.355212 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.031s	sys 0.022s
W20260709 12:14:32.403422 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.031s	sys 0.023s
I20260709 12:14:32.228399 19281 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.373160 19174 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:32.375593 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:32.374333 19245 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:32.350716 19345 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:32.373960 19426 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.414083 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:32.407120 19424 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:32.416051 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:32.406967 19183 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.024s	sys 0.033s
I20260709 12:14:32.369071 19230 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.376027 19248 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:32.429955 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:32.431319 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:32.432297 12641 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:32.366792 19210 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:32.327958 19242 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.434475 12642 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:32.426983 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:32.415096 19422 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.441895 12639 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:32.370951 19401 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.439352 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:32.470786 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:32.422659 19183 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.025s	sys 0.033s
I20260709 12:14:32.422309 19429 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:14:32.478430 19168 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:32.478677 12495 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:14:32.453212 12646 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:14:32.457785 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:32.371001 19427 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:14:32.419855 19281 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:32.457736 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 425049us
I20260709 12:14:32.472915 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.479238 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.422883 19426 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.468282 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.514060 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:32.481024 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.481423 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:32.524652 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:32.477850 12494 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:32.477763 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.477372 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.527345 12493 raft_consensus.cc:606] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:32.523809 12637 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:32.524605 12495 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:32.527382 12490 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:32.534621 19183 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:32.536522 19168 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:32.537002 19168 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:32.537340 19168 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:32.538043 19283 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:32.426115 19345 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.113s
I20260709 12:14:32.483232 19228 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:14:32.539877 19248 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:32.540478 19248 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:32.540678 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:32.485224 19282 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.683s	user 0.009s	sys 0.000s
I20260709 12:14:32.542716 19282 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:32.434939 18970 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:32.434686 19424 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:32.552599 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:32.550625 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:32.543588 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62270.8us
W20260709 12:14:32.497977 19224 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.661s	user 0.007s	sys 0.003s
W20260709 12:14:32.506141 19423 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:14:32.453739 19324 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:32.478333 19115 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:32.506877 19320 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.480741 19389 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.480340 19412 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:32.479351 19429 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:32.541497 19248 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.546916 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:32.554956 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:32.554533 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:32.460259 19318 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.561558 19428 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:32.509721 19422 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.549813 19365 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 12:14:32.486400 19245 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.556526 19421 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.475831 19402 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:14:32.565277 19401 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:32.569172 19392 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.020s	sys 0.055s
I20260709 12:14:32.552219 19263 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:32.593358 19431 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:32.624513 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 327166us
I20260709 12:14:32.619261 19421 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.585199 18970 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:32.624409 19431 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:32.625737 19283 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:14:32.626319 19431 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:32.556329 19210 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:32.614897 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:32.567616 19281 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.616955 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:32.637900 12642 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:32.621027 12496 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:32.619019 19429 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.622766 19424 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.643355 12637 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:32.645912 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:32.632144 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:32.653204 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:32.629662 19336 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.015s	sys 0.033s
I20260709 12:14:32.613979 19115 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:32.633491 19392 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.020s	sys 0.056s
I20260709 12:14:32.594491 19412 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:14:32.591039 19282 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.063s
W20260709 12:14:32.675000 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60214.8us
I20260709 12:14:32.678884 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:32.664263 19401 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:14:32.671088 19336 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.016s	sys 0.033s
I20260709 12:14:32.664037 19402 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.702415 19242 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:32.703621 19242 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:14:32.704269 19242 consensus_queue.cc:1243] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:32.655105 19332 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:14:32.705825 19332 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:14:32.708148 19421 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.624373 19263 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.670014 19174 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:14:32.680161 19432 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:14:32.690400 19324 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.691480 19115 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.695333 19168 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.700413 19426 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:32.702044 19402 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.665321 19281 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:32.708835 19412 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.718928 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.759344 12569 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:32.594159 19428 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.657613 19210 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.736595 19178 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:32.763036 12570 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.002s	sys 0.029s
W20260709 12:14:32.763916 12570 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.003s	sys 0.029s
I20260709 12:14:32.732690 19321 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.732592 19401 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:32.782166 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:32.782847 12644 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:32.767149 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 449068us
W20260709 12:14:32.795468 12645 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.945s	user 0.014s	sys 0.002s
W20260709 12:14:32.796367 12645 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.015s	sys 0.002s
W20260709 12:14:32.796060 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52472us
I20260709 12:14:32.801064 12565 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:32.747289 19227 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:32.805234 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.814448 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:32.820523 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.820782 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:32.824237 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.017s	sys 0.011s
W20260709 12:14:32.824767 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.018s	sys 0.011s
W20260709 12:14:32.835683 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148628us
I20260709 12:14:32.834017 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:32.744143 19324 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.826921 19283 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:32.814720 19296 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:32.766355 19281 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:32.868402 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 169566us
I20260709 12:14:32.774688 19412 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.858541 19178 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:32.797056 19321 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.853695 19437 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:32.864830 19443 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:32.859519 19228 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:32.911782 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 261409us
I20260709 12:14:32.893594 18970 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:32.858203 19338 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.861348 19430 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.089s
I20260709 12:14:32.861666 19387 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:14:32.827736 19227 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:32.925127 12565 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:14:32.943866 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.943600 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.943502 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:32.933558 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:32.958683 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:32.962018 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:32.901038 19443 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:32.977228 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:32.977936 12494 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:32.978294 12494 raft_consensus.cc:614] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:32.897279 19178 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:32.965479 19457 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:32.967412 19432 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:32.971925 19421 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:32.969995 19168 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:32.975065 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2934) took 1236 ms. Trace:
I20260709 12:14:33.118641 12645 rpcz_store.cc:276] 0709 12:14:31.738787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:31.774060 (+ 35273us) service_pool.cc:224] Handling call
0709 12:14:31.774436 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:31.774713 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:32.797798 (+1023085us) log.cc:844] Serialized 14 byte log entry
0709 12:14:32.974379 (+176581us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:32.974481 (+   102us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:32.974569 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:32.974791 (+   222us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:32.974978 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":647962,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":67298,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":16890,"thread_start_us":1090831,"threads_started":2}
I20260709 12:14:32.954559 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:33.122401 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:33.028463 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:33.069470 19447 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:14:32.980588 19443 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:32.985882 19296 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.053678 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:33.138021 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:33.052004 19457 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.077986 19402 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.923611 19401 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:32.940418 19422 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:32.989852 19178 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:32.939574 19320 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:32.922742 19318 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:14:32.936421 19323 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.025786 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:32.922307 19437 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:33.056699 19387 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.129560 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:33.156953 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121717us
I20260709 12:14:33.206898 12565 raft_consensus.cc:2376] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22. Re-sending same reply.
I20260709 12:14:33.184154 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:33.208083 12645 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:33.208813 12565 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:14:33.185860 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:33.188995 12568 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:33.201524 12638 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:14:33.210901 12638 pending_rounds.cc:85] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 10
I20260709 12:14:33.245019 12638 pending_rounds.cc:107] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:14:33.245412 12638 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:33.272924 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:33.274778 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:33.277421 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:33.175673 19336 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.291051 19336 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.294548 19336 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:33.295460 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:33.297235 12488 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:33.323348 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:33.324288 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:33.141744 19421 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.206534 19210 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:33.335667 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:33.143699 19296 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:33.340312 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:33.354638 12645 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:33.243824 19428 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:33.361418 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:33.281890 19168 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:33.282192 19281 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:33.286980 19324 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:33.327962 19465 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:33.356082 19463 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:14:33.141171 18970 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:33.367977 12638 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:33.272074 19387 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.276436 19318 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:14:33.334481 19412 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:33.209800 19323 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:33.379467 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:33.411716 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:33.442909 12639 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:33.372466 19432 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.354065 19457 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:33.423658 12645 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:14:33.210212 19320 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.425470 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:33.423022 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:33.398792 19324 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.445075 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:33.398350 19424 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:33.424471 19227 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:33.392179 19465 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:33.424078 19217 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.573984 12563 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:33.868886 13261 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:33.550199 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:33.540107 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:33.494230 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:33.502674 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:33.871654 13253 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:33.871785 13257 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:33.495549 13260 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:33.485322 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:33.886238 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 336098us
I20260709 12:14:33.891908 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:33.903661 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:33.867172 19228 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.941s	user 0.011s	sys 0.000s
I20260709 12:14:33.912140 13255 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
W20260709 12:14:33.455133 19462 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.358s	user 0.001s	sys 0.000s
I20260709 12:14:33.437837 19321 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:33.537730 19459 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.401s	user 0.002s	sys 0.000s
W20260709 12:14:33.528985 19478 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260709 12:14:33.501694 19454 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.440s	user 0.000s	sys 0.002s
I20260709 12:14:33.452560 19412 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.844820 19437 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:33.936276 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:33.937167 12645 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18. Re-sending same reply.
I20260709 12:14:33.586105 19281 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:33.924067 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:33.529980 19479 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:33.931541 19217 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:33.945024 19140 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.946704 19140 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:33.947625 19465 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:33.948288 19465 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.948765 19465 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:33.950138 19483 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.495s	user 0.000s	sys 0.001s
W20260709 12:14:33.952095 19475 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20260709 12:14:33.953450 18688 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:33.954105 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 881) took 1526 ms. Trace:
I20260709 12:14:33.954221 18688 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:33.954762 18688 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:33.955363 19345 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:33.956001 19345 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:33.958114 19488 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:33.958778 19488 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.919767 19228 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:14:33.963423 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3244) took 1495 ms (client timeout 1581 ms). Trace:
I20260709 12:14:33.964320 13259 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:33.966250 19345 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:33.971130 19479 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.958104 19421 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:33.974406 19353 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:33.975212 19323 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:33.975961 19296 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:33.953776 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3097) took 1606 ms. Trace:
I20260709 12:14:33.974442 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:33.963987 12496 rpcz_store.cc:269] 0709 12:14:32.467638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.467783 (+   145us) service_pool.cc:224] Handling call
0709 12:14:33.963177 (+1495394us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:33.963340 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199385344}
I20260709 12:14:33.982275 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
W20260709 12:14:33.954632 19467 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.543s	user 0.000s	sys 0.001s
W20260709 12:14:33.995508 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.008s	sys 0.016s
I20260709 12:14:33.954530 12644 rpcz_store.cc:276] 0709 12:14:32.427590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.428503 (+   913us) service_pool.cc:224] Handling call
0709 12:14:32.428845 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:32.783390 (+354545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:33.953881 (+1170491us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:33.954049 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":813514112}
I20260709 12:14:33.995685 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:33.996851 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.009s	sys 0.016s
I20260709 12:14:33.977123 12566 rpcz_store.cc:276] 0709 12:14:32.347221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.348613 (+  1392us) service_pool.cc:224] Handling call
0709 12:14:32.348981 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:32.784836 (+435855us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:32.784874 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:32.829145 (+ 44271us) log.cc:844] Serialized 26 byte log entry
0709 12:14:33.178347 (+349202us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:33.178397 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:33.178481 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:33.953455 (+774974us) raft_consensus.cc:1685] finished
0709 12:14:33.953479 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:33.953704 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9581,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":728345,"threads_started":2,"wal-append.queue_time_us":624689}
W20260709 12:14:33.934947 19473 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.405s	user 0.000s	sys 0.001s
W20260709 12:14:34.002199 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110399us
I20260709 12:14:34.003602 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:33.897984 19324 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:33.976722 19468 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.521s	user 0.000s	sys 0.001s
I20260709 12:14:34.015002 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:34.015164 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.015604 12645 raft_consensus.cc:2376] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17. Re-sending same reply.
I20260709 12:14:33.477109 19432 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.024487 19465 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.028723 19488 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:34.030125 19450 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.889s	user 0.001s	sys 0.002s
I20260709 12:14:34.031685 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 888) took 1487 ms. Trace:
I20260709 12:14:34.032369 12646 rpcz_store.cc:276] 0709 12:14:32.544484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.552466 (+  7982us) service_pool.cc:224] Handling call
0709 12:14:32.561774 (+  9308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:32.562147 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:32.562177 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:32.562679 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:14:32.875189 (+312510us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:32.875234 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:32.875310 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:33.994969 (+1119659us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:34.031142 (+ 36173us) raft_consensus.cc:1685] finished
0709 12:14:34.031185 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:34.031555 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1284,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":56402816,"thread_start_us":312155,"threads_started":1,"wal-append.queue_time_us":492904}
I20260709 12:14:34.035418 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:34.038938 12646 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:34.040622 13261 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:33.899530 19424 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.938680 19485 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.046072 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:34.012001 19401 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:34.010002 19429 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.062625 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3250) took 1476 ms. Trace:
I20260709 12:14:33.943527 19412 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:33.985549 19281 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.055691 19320 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.056752 12494 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:14:34.057686 12493 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:34.075387 12646 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:34.060173 12489 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:14:34.077714 19487 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.086156 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:34.069242 19470 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.007s	sys 0.008s
I20260709 12:14:34.087725 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.062397 19432 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.069026 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.012991 19318 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:34.042685 19242 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.042655 19479 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:34.044945 19178 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.835s	user 0.006s	sys 0.006s
I20260709 12:14:34.061192 12491 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:14:34.086818 12489 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:34.088927 13254 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:34.062992 12490 rpcz_store.cc:276] 0709 12:14:32.586551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.593042 (+  6491us) service_pool.cc:224] Handling call
0709 12:14:32.605395 (+ 12353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:32.614681 (+  9286us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:14:32.892105 (+277424us) log.cc:844] Serialized 14 byte log entry
0709 12:14:33.175443 (+283338us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:33.176049 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 12:14:33.176251 (+   202us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:14:33.176300 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:33.176391 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:34.062382 (+885991us) raft_consensus.cc:1685] finished
0709 12:14:34.062403 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:34.062555 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52014,"prepare.queue_time_us":455056,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":47,"raft.queue_time_us":15239,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"spinlock_wait_cycles":11230464,"thread_start_us":547574,"threads_started":2,"wal-append.queue_time_us":636917}
I20260709 12:14:34.086314 19497 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.081171 19228 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:34.105361 19497 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.106081 19412 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.106940 19281 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.091118 19487 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.094792 13256 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:34.100831 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.103329 13254 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:34.114446 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:34.117609 13254 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.051206 19485 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.073019 19429 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.097311 18688 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:34.213791 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:34.245875 12644 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:34.349241 13254 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:34.132555 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:34.369007 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:34.089474 19470 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.007s	sys 0.008s
I20260709 12:14:34.375209 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.377455 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1143) took 3980 ms. Trace:
I20260709 12:14:34.378425 12571 rpcz_store.cc:276] 0709 12:14:30.396916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:31.223674 (+826758us) service_pool.cc:224] Handling call
0709 12:14:32.348088 (+1124414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:32.370410 (+ 22322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:34.377156 (+2006746us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001a31d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:34.377369 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2584890240}
I20260709 12:14:34.378901 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.107707 19497 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:34.211663 19431 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.584s	user 0.013s	sys 0.003s
W20260709 12:14:34.264926 19443 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.123s	user 0.014s	sys 0.000s
W20260709 12:14:34.246695 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 274263us
I20260709 12:14:34.110924 19365 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:34.387486 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.387734 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.393337 13258 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:34.293730 19507 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.390767 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15.
I20260709 12:14:34.385839 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3070) took 1429 ms. Trace:
I20260709 12:14:34.400377 12568 rpcz_store.cc:276] 0709 12:14:32.956926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.213653 (+256727us) service_pool.cc:224] Handling call
0709 12:14:33.274064 (+ 60411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:33.399413 (+125349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:33.399460 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:33.400062 (+   602us) log.cc:844] Serialized 26 byte log entry
0709 12:14:33.907114 (+507052us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:33.907197 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:34.112109 (+204912us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:34.112187 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:34.112266 (+    79us) raft_consensus.cc:1685] finished
0709 12:14:34.112287 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:34.385764 (+273477us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":9408,"spinlock_wait_cycles":231964544,"thread_start_us":460703,"threads_started":1,"wal-append.queue_time_us":548443}
I20260709 12:14:34.393868 12490 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:14:34.208565 19465 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.233738 19324 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.407347 12489 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:34.264257 19487 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.408064 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.384948 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.409600 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.409835 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.411772 12646 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:34.110692 19318 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.141029 19178 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.411993 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:34.111138 19242 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.399024 19432 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:34.397660 19443 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.393142 19431 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:34.439765 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 133601us
I20260709 12:14:34.415328 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:34.423818 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 152098us
W20260709 12:14:34.534591 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119358us
I20260709 12:14:34.418466 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.415764 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.538115 12490 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:34.442009 12497 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:14:34.445397 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.540930 12643 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:14:34.415253 19494 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.533845 19479 raft_consensus.cc:2764] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:14:34.543476 12641 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:34.401965 19365 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.532277 19487 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.410981 19465 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:34.537132 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132977us
I20260709 12:14:34.537361 12637 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:34.530037 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:34.541868 12646 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:34.405814 19115 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.136s
W20260709 12:14:34.546630 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:34.548861 12490 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:34.550024 12497 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:14:34.534789 19482 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.405992 19507 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.558106 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2605) took 1132 ms. Trace:
I20260709 12:14:34.559723 12646 rpcz_store.cc:276] 0709 12:14:33.425922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.415683 (+989761us) service_pool.cc:224] Handling call
0709 12:14:34.558033 (+142350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:34.559073 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:34.548238 19511 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.532820 19432 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.566100 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:34.552280 19487 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.575660 19318 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:34.570299 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.585490 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:34.590426 19335 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.596657 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.597604 12646 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:34.598863 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.602589 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:34.604427 12496 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:34.577142 19511 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.562215 19482 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:34.608474 19515 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:14:34.609562 19482 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.610268 19507 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.614482 19511 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:34.617731 19503 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:14:34.590602 19282 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:34.589669 19429 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.630290 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.631562 12494 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:34.632999 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.650324 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.651499 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:34.652841 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.600647 18688 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.655268 19431 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:34.595124 19279 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:34.656700 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:34.658093 19242 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:34.707063 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:34.659768 19465 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.661144 19365 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.673887 19422 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:34.679029 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:34.679307 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:34.681027 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.605062 18970 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:14:34.605866 19318 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 12:14:34.723726 12495 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:34.692806 12494 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:34.665527 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3103) took 1948 ms. Trace:
I20260709 12:14:34.544893 19494 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.737709 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.711184 19279 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:34.702178 12490 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:34.704130 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.725834 12564 rpcz_store.cc:276] 0709 12:14:32.717015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.718207 (+  1192us) service_pool.cc:224] Handling call
0709 12:14:32.733276 (+ 15069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:32.803527 (+ 70251us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:33.190196 (+386669us) log.cc:844] Serialized 14 byte log entry
0709 12:14:33.200973 (+ 10777us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:33.201063 (+    90us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:33.201148 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:34.663504 (+1462356us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b1800292de0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419
0709 12:14:34.665142 (+  1638us) spinlock_profiling.cc:243] Waited 69.8 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:34.665209 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:34.665431 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":221212,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":120,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":70867,"spinlock_wait_cycles":171334784,"thread_start_us":364716,"threads_started":2,"wal-append.queue_time_us":171625}
I20260709 12:14:34.705348 13254 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:14:34.746151 13254 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 11
I20260709 12:14:34.590696 19485 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:34.684132 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 823) took 1318 ms. Trace:
W20260709 12:14:34.737129 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 142525us
I20260709 12:14:34.747802 13254 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:14:34.748206 13254 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:34.699877 12462 connection.cc:256] RPC call timeout handler was delayed by -1.03391839599999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:14:34.739593 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
W20260709 12:14:34.690130 12536 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.801s (SENT)
I20260709 12:14:34.750800 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:34.743279 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.604626 19335 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.746842 13260 rpcz_store.cc:276] 0709 12:14:33.365960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.366201 (+   241us) service_pool.cc:224] Handling call
0709 12:14:33.366608 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:33.873269 (+506661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:34.683866 (+810597us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:34.684066 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295490432}
I20260709 12:14:34.713799 19422 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:34.606580 19514 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.142s
I20260709 12:14:34.724050 19365 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:34.771124 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:34.771651 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.724864 19431 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:34.755321 19279 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:34.776655 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.774813 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.776551 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:34.777323 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.782433 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:34.774801 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:34.783794 12491 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:34.783403 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.831310 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.785360 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.858842 12491 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:34.860324 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:34.770565 19347 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 12:14:34.887666 19422 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.775743 19514 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:34.890695 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.892120 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.893419 12494 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:14:34.892295 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:34.896116 12495 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:34.896687 12495 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:34.901050 12490 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:34.908419 12494 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:34.780189 19009 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:34.897779 12495 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:34.914772 12495 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:34.775629 19365 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:34.894292 19178 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:34.782406 19431 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:34.966562 19422 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:34.917462 19318 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:34.947114 19365 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:34.976653 12495 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:14:34.971685 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 197075us
I20260709 12:14:34.983477 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:34.948007 19482 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:14:34.989799 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 213314us
I20260709 12:14:35.202445 19431 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:35.026530 19521 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:14:35.225823 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:35.291908 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66258.3us
I20260709 12:14:34.993116 19510 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.355s
W20260709 12:14:35.304818 19520 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260709 12:14:35.022181 19511 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:35.306037 19511 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.306516 19511 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:35.300593 19318 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:35.058709 19432 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.024885 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:35.372313 13254 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
W20260709 12:14:35.098716 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.372704 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:35.028272 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:35.373850 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74422.4us
W20260709 12:14:35.376886 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 348719us
I20260709 12:14:35.377240 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:34.974192 19508 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.612s
I20260709 12:14:35.378072 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:35.379102 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:35.381387 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:35.392329 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:35.223893 19482 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.020087 19335 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:35.406081 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0934447670001646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20260709 12:14:35.406955 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3068) took 2488 ms. Trace:
W20260709 12:14:35.339193 19529 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:14:35.021458 19494 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.402590 19318 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:35.411141 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:35.406378 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:35.379058 19526 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:35.409883 12565 rpcz_store.cc:276] 0709 12:14:32.919035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.209827 (+290792us) service_pool.cc:224] Handling call
0709 12:14:33.210193 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:33.244184 (+ 33991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:33.275171 (+ 30987us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:33.276096 (+   925us) log.cc:844] Serialized 26 byte log entry
0709 12:14:33.548845 (+272749us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:33.796521 (+247676us) log.cc:844] Serialized 14 byte log entry
0709 12:14:35.375591 (+1579070us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b18004ac920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:14:35.375825 (+   234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.375913 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:35.375955 (+    42us) raft_consensus.cc:1685] finished
0709 12:14:35.406714 (+ 30759us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:35.406884 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31715,"prepare.queue_time_us":257513,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":4891,"raft.queue_time_us":1579181,"raft.run_cpu_time_us":556,"raft.run_wall_time_us":92781,"spinlock_wait_cycles":36095744,"thread_start_us":182655,"threads_started":1,"wal-append.queue_time_us":642286,"wal-append.run_cpu_time_us":2792,"wal-append.run_wall_time_us":1006196}
W20260709 12:14:35.417943 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:35.420099 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.374639 19432 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:14:35.420509 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.417069 12495 raft_consensus.cc:606] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:35.423877 12495 raft_consensus.cc:614] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:35.408424 19431 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:35.045580 19347 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.373s
I20260709 12:14:35.411341 19482 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:35.432263 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2938) took 1988 ms (client timeout 2118 ms). Trace:
I20260709 12:14:35.426805 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:35.411851 19510 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:35.443424 12644 rpcz_store.cc:269] 0709 12:14:33.443799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.068717 (+624918us) service_pool.cc:224] Handling call
0709 12:14:35.431988 (+1363271us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.432177 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":356404992}
I20260709 12:14:35.460222 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:35.461886 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:35.420140 19437 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:35.415856 19528 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:35.463994 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:35.434687 19526 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:35.433382 19431 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.469280 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.437366 19335 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:35.460387 19510 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.435503 19425 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.488993 12490 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:35.435122 19140 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:35.441205 19296 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.462488 19421 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:35.486778 19528 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.473757 19387 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:35.494524 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 804381us
W20260709 12:14:35.516920 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.493398 19431 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:35.518589 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.494740 19335 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.521531 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.521974 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3073) took 2548 ms. Trace:
I20260709 12:14:35.505198 19510 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:35.515663 19528 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:35.506484 18986 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:35.518738 19279 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.627s	user 0.005s	sys 0.009s
I20260709 12:14:35.534022 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.524026 12567 rpcz_store.cc:276] 0709 12:14:32.973522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.947323 (+973801us) service_pool.cc:224] Handling call
0709 12:14:33.947682 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:33.947928 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:33.948312 (+   384us) log.cc:844] Serialized 14 byte log entry
0709 12:14:35.412300 (+1463988us) log.cc:844] Serialized 14 byte log entry
0709 12:14:35.521510 (+109210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:35.521576 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:35.521675 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.521748 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:35.521906 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1393117,"prepare.queue_time_us":616,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":1362208,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":4242,"thread_start_us":203619,"threads_started":2,"wal-append.queue_time_us":293870,"wal-append.run_cpu_time_us":1474,"wal-append.run_wall_time_us":1001498}
I20260709 12:14:35.489689 19437 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.534847 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
I20260709 12:14:35.536092 12564 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
W20260709 12:14:35.526840 19422 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.557s	user 0.008s	sys 0.004s
I20260709 12:14:35.526268 19526 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.526031 19209 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:35.546144 19209 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.547907 19263 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.548547 19209 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.509826 19140 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.550599 19338 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:14:35.551010 19140 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:35.552277 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2600) took 2367 ms (client timeout 2033 ms). Trace:
W20260709 12:14:35.552713 12637 rpcz_store.cc:269] 0709 12:14:33.185382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.185535 (+   153us) service_pool.cc:224] Handling call
0709 12:14:35.552059 (+2366524us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.552203 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3050655488}
I20260709 12:14:35.553915 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.554642 12637 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:35.526005 19335 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:35.556470 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:35.557749 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.557816 19511 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.558549 12640 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:35.559494 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.112s (SENT) [suppressed 22 similar messages]
I20260709 12:14:35.560827 19443 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:35.562314 19511 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:35.523412 19353 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.570696 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:35.565165 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.574769 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:35.575906 12564 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:35.511341 19296 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.575726 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:35.531226 19533 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.579635 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:35.587602 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.509724 19425 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.583640 12568 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:14:35.584491 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.585618 12571 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:35.589519 12571 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:35.589613 12565 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:35.580034 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.591966 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.592610 12566 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:35.588028 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
W20260709 12:14:35.575349 19365 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.572s	user 0.009s	sys 0.004s
I20260709 12:14:35.590725 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 898) took 1320 ms. Trace:
I20260709 12:14:35.595571 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:35.598127 12645 rpcz_store.cc:276] 0709 12:14:34.270513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.577444 (+1306931us) service_pool.cc:224] Handling call
0709 12:14:35.590662 (+ 13218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:35.594244 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:35.599352 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:35.599954 12645 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 15. Re-sending same reply.
I20260709 12:14:35.569903 19525 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.590443 12571 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.603343 12571 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:35.603410 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.606956 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:35.609725 12568 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:35.537757 18986 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.593072 19425 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.540802 19279 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.606865 19365 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.621901 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:35.578523 19353 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.610015 19508 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:35.640049 18986 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.640817 19523 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:35.641530 19523 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.644008 19523 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.649572 19437 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.650138 19353 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:35.653007 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93547.5us
I20260709 12:14:35.655602 12571 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:14:35.656071 12571 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:14:35.656454 12571 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:14:35.656710 12571 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:35.660764 19387 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:35.662318 19387 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:35.640080 19296 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.596969 19533 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.683403 19283 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:35.637921 19443 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:14:35.691460 19283 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:35.692582 19533 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:35.672755 19466 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.390s	user 0.013s	sys 0.010s
W20260709 12:14:35.670696 19461 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.469s	user 0.016s	sys 0.027s
I20260709 12:14:35.665670 19279 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.671713 19365 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.687345 19392 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:14:35.688009 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.696151 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:35.679137 19452 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.607s	user 0.008s	sys 0.004s
I20260709 12:14:35.675637 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 929) took 2732 ms. Trace:
I20260709 12:14:35.687740 19387 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:35.694713 19400 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:14:35.701424 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:35.682126 19528 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.611100 19525 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.698137 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:35.698318 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:35.701365 13262 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:35.699440 19392 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:35.678014 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 997) took 1807 ms. Trace:
I20260709 12:14:35.689059 19430 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.699723 13259 rpcz_store.cc:276] 0709 12:14:32.943744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.943884 (+   140us) service_pool.cc:224] Handling call
0709 12:14:32.961659 (+ 17775us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:33.964820 (+1003161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.675330 (+1710510us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:35.675569 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2305378176}
I20260709 12:14:35.708184 13256 rpcz_store.cc:276] 0709 12:14:33.870684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.870909 (+   225us) service_pool.cc:224] Handling call
0709 12:14:33.908122 (+ 37213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:34.116501 (+208379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.677720 (+1561219us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:35.677940 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":428344704}
I20260709 12:14:35.701978 19390 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:35.695453 19485 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 12:14:35.718107 19387 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.718292 19431 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.719192 19528 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.721845 19431 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:35.722469 19485 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:35.726178 19485 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:35.711318 19430 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.721905 19525 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.732563 19485 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:35.733682 19430 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:35.735387 19510 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.735564 19335 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.740913 19482 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:35.744668 19487 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:35.744928 19482 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:35.746665 19455 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.676s	user 0.005s	sys 0.022s
I20260709 12:14:35.746500 19510 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:35.801723 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3251) took 3214 ms. Trace:
I20260709 12:14:35.802224 12491 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:35.687717 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
W20260709 12:14:35.815133 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.816114 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2612) took 1143 ms. Trace:
I20260709 12:14:35.816962 12640 rpcz_store.cc:276] 0709 12:14:34.671892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.607422 (+935530us) service_pool.cc:224] Handling call
0709 12:14:35.815597 (+208175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:35.818701 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3072) took 2855 ms. Trace:
I20260709 12:14:35.826330 12565 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 14. Current term is 15. Ops: []
I20260709 12:14:35.826488 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.828480 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.830116 12637 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:14:35.803224 12492 rpcz_store.cc:276] 0709 12:14:32.586888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.587069 (+   181us) service_pool.cc:224] Handling call
0709 12:14:32.614516 (+ 27447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:32.634497 (+ 19981us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:32.634534 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:32.635228 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 12:14:32.862551 (+227323us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:32.862622 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:32.862776 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:35.745972 (+2883196us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:35.800248 (+ 54276us) raft_consensus.cc:1685] finished
0709 12:14:35.800352 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:35.801052 (+   700us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17832,"raft.queue_time_us":38922,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":8255,"spinlock_wait_cycles":25208576,"thread_start_us":226814,"threads_started":1,"wal-append.queue_time_us":435379}
W20260709 12:14:35.846886 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 142134us
I20260709 12:14:35.847939 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.851912 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.868507 19335 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:35.870601 12644 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:35.850282 12566 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:35.849339 12640 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:35.853955 19547 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:14:35.878221 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:35.879302 12566 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:35.880020 12572 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:14:35.881203 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:35.854089 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.882345 12565 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:35.883430 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.884308 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2948) took 2380 ms. Trace:
I20260709 12:14:35.884585 12565 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 24 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:35.884894 12640 rpcz_store.cc:276] 0709 12:14:33.498054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.847058 (+2349004us) service_pool.cc:224] Handling call
0709 12:14:35.876968 (+ 29910us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:35.854228 19497 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:35.886955 19497 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.737278 19392 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.889783 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.890280 12572 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:35.891093 12571 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:35.891180 19392 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:14:35.892426 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:35.893302 12571 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:35.894321 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.896085 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.897040 12564 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:35.831835 12563 rpcz_store.cc:276] 0709 12:14:32.962669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.917251 (+954582us) service_pool.cc:224] Handling call
0709 12:14:33.978642 (+ 61391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:35.416573 (+1437931us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:14:35.416614 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:35.427351 (+ 10737us) log.cc:844] Serialized 26 byte log entry
0709 12:14:35.564461 (+137110us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:35.564500 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.564567 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:35.817867 (+253300us) raft_consensus.cc:1685] finished
0709 12:14:35.817915 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:35.818351 (+   436us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1468380,"raft.queue_time_us":1435839,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":3159,"thread_start_us":71416,"threads_started":1,"wal-append.queue_time_us":131060}
I20260709 12:14:35.899380 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.900240 12563 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 15.
W20260709 12:14:35.906827 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.908361 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:35.908917 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2614) took 1220 ms (client timeout 1548 ms). Trace:
W20260709 12:14:35.850330 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 148808us
W20260709 12:14:35.909459 12644 rpcz_store.cc:269] 0709 12:14:34.688262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.468906 (+780644us) service_pool.cc:224] Handling call
0709 12:14:35.908593 (+439687us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.908812 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":839073024}
I20260709 12:14:35.911182 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.912279 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2616) took 1205 ms. Trace:
I20260709 12:14:35.912665 12644 rpcz_store.cc:276] 0709 12:14:34.706946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.910698 (+1203752us) service_pool.cc:224] Handling call
0709 12:14:35.912197 (+  1499us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:35.901377 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.926275 12640 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:35.941005 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:35.941411 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 829) took 2403 ms (client timeout 1974 ms). Trace:
I20260709 12:14:35.913601 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:35.941936 13255 rpcz_store.cc:269] 0709 12:14:33.538128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.539490 (+  1362us) service_pool.cc:224] Handling call
0709 12:14:35.927847 (+2388357us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54004b73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.941037 (+ 13190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93971712}
I20260709 12:14:35.942673 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.944526 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.915776 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.935469 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2606) took 1265 ms. Trace:
I20260709 12:14:35.949329 12640 rpcz_store.cc:276] 0709 12:14:34.669913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.907715 (+1237802us) service_pool.cc:224] Handling call
0709 12:14:35.935392 (+ 27677us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:35.920523 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 899) took 1380 ms (client timeout 1516 ms). Trace:
W20260709 12:14:35.955230 12646 rpcz_store.cc:269] 0709 12:14:34.540478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.600215 (+ 59737us) service_pool.cc:224] Handling call
0709 12:14:35.920098 (+1319883us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.920291 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411312384}
I20260709 12:14:35.957080 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:35.958849 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1000) took 2042 ms (client timeout 1586 ms). Trace:
I20260709 12:14:35.959444 19546 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:35.914376 19547 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:14:35.919776 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.964131 12564 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:14:35.918395 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2601) took 2673 ms (client timeout 1791 ms). Trace:
I20260709 12:14:35.919581 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 890) took 3295 ms. Trace:
I20260709 12:14:35.916390 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:35.964615 13258 rpcz_store.cc:269] 0709 12:14:33.916473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.974098 (+ 57625us) service_pool.cc:224] Handling call
0709 12:14:35.958369 (+1984271us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.958601 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954944896}
I20260709 12:14:35.963454 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:35.977011 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.983167 12563 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:14:35.984385 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:35.985114 12563 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:35.982950 19510 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:35.987613 19510 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:35.990302 12646 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:35.992611 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:35.995436 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 885) took 3460 ms. Trace:
I20260709 12:14:35.974079 12643 rpcz_store.cc:276] 0709 12:14:32.623708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.629292 (+  5584us) service_pool.cc:224] Handling call
0709 12:14:32.633859 (+  4567us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:34.542365 (+1908506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.918861 (+1376496us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:35.919337 (+   476us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4385220864}
I20260709 12:14:35.970587 19546 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:35.998669 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:35.973945 12641 rpcz_store.cc:269] 0709 12:14:33.245329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.274530 (+ 29201us) service_pool.cc:224] Handling call
0709 12:14:35.917936 (+2643406us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.918167 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2915785600}
W20260709 12:14:36.000411 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:35.991358 19510 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.002077 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.005663 12638 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.006083 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:35.748436 19485 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.001039 19546 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:36.007645 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:35.996862 12639 rpcz_store.cc:276] 0709 12:14:32.456636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.456939 (+   303us) service_pool.cc:224] Handling call
0709 12:14:32.479092 (+ 22153us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:33.443483 (+964391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.916948 (+2473465us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54002e5254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:35.917381 (+   433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2189030144}
I20260709 12:14:36.008848 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:36.010072 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.013340 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:36.012820 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.017659 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2950) took 2013 ms. Trace:
I20260709 12:14:36.011292 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.018384 12563 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.018059 12638 rpcz_store.cc:276] 0709 12:14:34.001196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.915875 (+1914679us) service_pool.cc:224] Handling call
0709 12:14:36.014491 (+ 98616us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.016000 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:36.023416 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.024611 12639 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:36.026319 13259 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:14:36.028604 13255 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.027511 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 896) took 1758 ms. Trace:
I20260709 12:14:36.025374 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2618) took 1313 ms. Trace:
I20260709 12:14:36.019111 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2613) took 1343 ms. Trace:
I20260709 12:14:36.029960 12641 rpcz_store.cc:276] 0709 12:14:34.269742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.009542 (+1739800us) service_pool.cc:224] Handling call
0709 12:14:36.027450 (+ 17908us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.030066 12639 rpcz_store.cc:276] 0709 12:14:34.711671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.022891 (+1311220us) service_pool.cc:224] Handling call
0709 12:14:36.025264 (+  2373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.025941 13254 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.030418 12644 rpcz_store.cc:276] 0709 12:14:34.672514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.913438 (+1240924us) service_pool.cc:224] Handling call
0709 12:14:36.016097 (+102659us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:36.038378 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.042704 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 999) took 2037 ms. Trace:
I20260709 12:14:36.043360 13257 rpcz_store.cc:276] 0709 12:14:33.909652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.910492 (+   840us) service_pool.cc:224] Handling call
0709 12:14:35.944625 (+2034133us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:35.945494 (+   869us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:35.945542 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:35.945668 (+   126us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:35.945949 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:35.946059 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:35.947164 (+  1105us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b54004b73d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:35.947382 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4635995904}
W20260709 12:14:36.044610 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:36.025434 19532 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.396s	user 0.001s	sys 0.004s
I20260709 12:14:36.046453 12643 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:36.047510 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2939) took 2603 ms. Trace:
I20260709 12:14:36.048008 12643 rpcz_store.cc:276] 0709 12:14:33.444087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.045281 (+2601194us) service_pool.cc:224] Handling call
0709 12:14:36.045914 (+   633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.047164 (+  1250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.047417 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.048935 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81a9a92616494f46bc97254b53185664"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:36.049376 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.050004 12495 raft_consensus.cc:606] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:36.051453 19497 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.051903 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.052300 19497 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:36.054042 19456 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:36.054525 19456 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:36.055081 19456 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:36.060878 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.061092 19485 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:36.062106 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.050372 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:36.066908 12495 raft_consensus.cc:614] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:36.073026 12572 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:36.027873 19547 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:36.084355 19430 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:14:36.086840 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76059.2us
I20260709 12:14:36.088238 19412 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:36.089278 19412 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.093809 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.094247 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:36.094666 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:36.095090 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3272) took 1438 ms. Trace:
I20260709 12:14:36.095652 12491 rpcz_store.cc:276] 0709 12:14:34.656769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.859930 (+203161us) service_pool.cc:224] Handling call
0709 12:14:36.094768 (+1234838us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:36.094963 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2161143552}
I20260709 12:14:36.096012 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:36.096997 13258 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:36.097261 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.095166 13259 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:36.102239 12638 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:14:36.102330 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50823.3us
W20260709 12:14:36.102762 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.121452 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.124567 12564 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:36.126020 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.126933 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:36.128175 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.129125 12564 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:36.113528 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2617) took 1404 ms. Trace:
I20260709 12:14:36.131357 19228 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:36.132028 19228 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:36.133518 19460 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:36.133966 19551 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.105s	user 0.001s	sys 0.001s
I20260709 12:14:36.134637 19460 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:36.135499 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46026.8us
I20260709 12:14:36.138453 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:36.139874 19242 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:36.140331 19242 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:36.140400 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.140769 19242 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.142844 13262 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:36.149863 19296 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:36.150669 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.151293 18986 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:36.150861 19296 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.151870 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:36.153360 19351 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:36.153882 19351 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:36.153828 19533 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:14:36.154528 19351 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.156456 19437 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:36.156926 19437 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:36.157609 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.158699 12646 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.158318 19437 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.160049 19387 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:36.160624 12646 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:14:36.130934 12638 rpcz_store.cc:276] 0709 12:14:34.708978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.061590 (+1352612us) service_pool.cc:224] Handling call
0709 12:14:36.113456 (+ 51866us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:36.162854 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:36.163529 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.164826 12639 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:36.165473 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 904) took 1784 ms (client timeout 2112 ms). Trace:
I20260709 12:14:36.164921 18986 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:36.165961 12496 rpcz_store.cc:269] 0709 12:14:34.381138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.384467 (+  3329us) service_pool.cc:224] Handling call
0709 12:14:36.165032 (+1780565us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:36.165247 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":440592768}
I20260709 12:14:36.169404 19296 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.172375 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.173578 19497 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:36.165612 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2936) took 2722 ms. Trace:
I20260709 12:14:36.174712 12639 rpcz_store.cc:276] 0709 12:14:33.443198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.068508 (+2625310us) service_pool.cc:224] Handling call
0709 12:14:36.110167 (+ 41659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.165324 (+ 55157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.165532 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.150897 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2575) took 2113 ms. Trace:
I20260709 12:14:36.093362 19412 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:36.133649 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.181087 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.183591 12641 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.222119 12645 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:36.177711 13255 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:14:36.346616 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.588338 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.588330 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:36.233611 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145893us
I20260709 12:14:36.590514 12639 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.590807 12646 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:36.591303 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.291389 13254 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:36.591831 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:36.420380 12566 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.540s	user 0.006s	sys 0.007s
I20260709 12:14:36.596720 12566 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:36.591821 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:36.481980 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 310055us
I20260709 12:14:36.586607 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:36.234695 12462 connection.cc:256] RPC call timeout handler was delayed by -1.0838004229999569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
I20260709 12:14:36.226320 12571 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
W20260709 12:14:36.297078 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61705.4us
W20260709 12:14:36.594877 12563 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.563s	user 0.014s	sys 0.000s
W20260709 12:14:36.220176 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64137us
I20260709 12:14:36.212517 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.599759 12563 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:36.586571 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3285) took 1145 ms. Trace:
W20260709 12:14:36.599015 12462 connection.cc:256] RPC call timeout handler was delayed by -1.0163004230000752s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
I20260709 12:14:36.602284 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2959) took 1106 ms. Trace:
I20260709 12:14:36.602926 12641 rpcz_store.cc:276] 0709 12:14:35.488331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.064994 (+576663us) service_pool.cc:224] Handling call
0709 12:14:36.594115 (+529121us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:36.587188 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 453765us
W20260709 12:14:36.592099 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2332044419999875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3235s for a response
I20260709 12:14:36.601693 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.606436 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.606932 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.598493 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:36.297953 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:36.609203 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.610565 12564 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.600451 12492 rpcz_store.cc:276] 0709 12:14:35.440717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.848604 (+407887us) service_pool.cc:224] Handling call
0709 12:14:35.849366 (+   762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:36.015719 (+166353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:36.015763 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:36.048757 (+ 32994us) log.cc:844] Serialized 26 byte log entry
0709 12:14:36.104243 (+ 55486us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:36.104294 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.104377 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:36.586308 (+481931us) raft_consensus.cc:1685] finished
0709 12:14:36.586333 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.586500 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25578,"raft.queue_time_us":39491,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":203,"thread_start_us":54946,"threads_started":1,"wal-append.queue_time_us":162337}
I20260709 12:14:36.306567 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.297709 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:36.605410 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:36.614508 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.615427 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:36.298499 19283 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.606s	user 0.011s	sys 0.018s
I20260709 12:14:36.617776 12567 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.390704 19497 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:36.482692 19460 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:36.623168 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 405930us
I20260709 12:14:36.420885 18986 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.420518 19353 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:36.625316 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1158) took 3646 ms. Trace:
I20260709 12:14:36.628997 12496 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:14:36.629465 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.629243 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.632836 12497 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.634235 12491 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:36.286017 19456 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:36.635355 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2958) took 1154 ms. Trace:
I20260709 12:14:36.622123 19497 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:36.640863 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.640805 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:36.298118 19547 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.347s
I20260709 12:14:36.637313 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.627043 19485 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:14:36.644029 12565 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:36.177754 12488 rpcz_store.cc:276] 0709 12:14:34.037716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.037888 (+   172us) service_pool.cc:224] Handling call
0709 12:14:34.051979 (+ 14091us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:34.548087 (+496108us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:34.548123 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:34.548694 (+   571us) log.cc:844] Serialized 52 byte log entry
0709 12:14:34.597689 (+ 48995us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:34.622461 (+ 24772us) log.cc:844] Serialized 14 byte log entry
0709 12:14:34.622715 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.150025 (+1527310us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b540020bcd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:36.150104 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:36.150180 (+    76us) raft_consensus.cc:1685] finished
0709 12:14:36.150300 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.150632 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22115,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":62489,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":9533,"spinlock_wait_cycles":1139831680,"thread_start_us":34233,"threads_started":1,"wal-append.queue_time_us":175690,"wal-append.run_cpu_time_us":3378,"wal-append.run_wall_time_us":1124949}
I20260709 12:14:36.652014 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.654951 12490 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:36.658452 12639 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:36.659317 12492 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:36.660321 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:36.661000 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.645694 19497 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:36.664696 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:36.665278 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.640837 18986 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:36.666762 19512 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.080s	user 0.000s	sys 0.008s
I20260709 12:14:36.677098 18986 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.679270 19456 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:36.679781 18986 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.626989 12569 rpcz_store.cc:276] 0709 12:14:32.785436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.802486 (+ 17050us) service_pool.cc:224] Handling call
0709 12:14:32.802906 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:32.819930 (+ 17024us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:14:35.815598 (+2995668us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b1800248b60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:14:35.906241 (+ 90643us) log.cc:844] Serialized 14 byte log entry
0709 12:14:36.021086 (+114845us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:36.323281 (+302195us) log.cc:844] Serialized 26 byte log entry
0709 12:14:36.323545 (+   264us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:36.323606 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.414561 (+ 90955us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:36.414625 (+    64us) raft_consensus.cc:1685] finished
0709 12:14:36.431167 (+ 16542us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.431360 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32144,"prepare.queue_time_us":284156,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":2434,"raft.queue_time_us":69569,"raft.run_cpu_time_us":565,"raft.run_wall_time_us":24556,"spinlock_wait_cycles":8623360,"thread_start_us":120578,"threads_started":2,"wal-append.queue_time_us":121908}
I20260709 12:14:36.668704 12641 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:14:36.686565 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:36.690393 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:36.692247 12488 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:36.690794 18986 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 12:14:36.694106 19456 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:36.693225 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.695960 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.696395 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:36.696897 12569 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22. Re-sending same reply.
I20260709 12:14:36.697046 13255 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:14:36.698232 13255 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:36.687554 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2960) took 1194 ms. Trace:
I20260709 12:14:36.699295 13255 raft_consensus.cc:614] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:36.628069 19283 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:36.700788 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2588) took 1923 ms. Trace:
I20260709 12:14:36.636893 12638 rpcz_store.cc:276] 0709 12:14:35.476735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.162701 (+685966us) service_pool.cc:224] Handling call
0709 12:14:36.163192 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.163722 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:36.164177 (+   455us) log.cc:844] Serialized 14 byte log entry
0709 12:14:36.608204 (+444027us) log.cc:844] Serialized 14 byte log entry
0709 12:14:36.625247 (+ 17043us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:36.625312 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:36.630607 (+  5295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.630684 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.630840 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50605,"prepare.queue_time_us":341,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":103,"raft.queue_time_us":358967,"spinlock_wait_cycles":8890752,"thread_start_us":107124,"threads_started":1}
I20260709 12:14:36.698659 12639 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 11
I20260709 12:14:36.688512 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.700878 12641 rpcz_store.cc:276] 0709 12:14:35.493173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.637409 (+1144236us) service_pool.cc:224] Handling call
0709 12:14:36.667317 (+ 29908us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.686293 (+ 18976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.687060 (+   767us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.661496 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.703449 12639 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:14:36.707767 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:36.635843 19353 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:36.708932 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57377us
I20260709 12:14:36.711046 12496 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.688820 19296 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:36.714272 19296 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:36.709007 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.716609 12566 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:14:36.637473 19460 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:36.692247 19351 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.006s
I20260709 12:14:36.716109 12639 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:36.707026 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.733506 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:36.710180 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:14:36.743114 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:14:36.754488 12638 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:36.701282 12494 rpcz_store.cc:276] 0709 12:14:34.776953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:34.913274 (+136321us) service_pool.cc:224] Handling call
0709 12:14:34.913608 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:34.915478 (+  1870us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:34.915500 (+    22us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:34.946385 (+ 30885us) log.cc:844] Serialized 52 byte log entry
0709 12:14:36.044866 (+1098481us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b18004d6a40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:36.406949 (+362083us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:36.429034 (+ 22085us) log.cc:844] Serialized 14 byte log entry
0709 12:14:36.429333 (+   299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.429453 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:36.700296 (+270843us) raft_consensus.cc:1685] finished
0709 12:14:36.700408 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.700598 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3341,"prepare.queue_time_us":922,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":42197,"raft.queue_time_us":61584,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":35165,"spinlock_wait_cycles":2818816,"thread_start_us":144770,"threads_started":1,"wal-append.queue_time_us":1679410}
I20260709 12:14:36.762429 12641 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:36.764662 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2962) took 1232 ms. Trace:
I20260709 12:14:36.765368 12638 rpcz_store.cc:276] 0709 12:14:35.532723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.703277 (+1170554us) service_pool.cc:224] Handling call
0709 12:14:36.754175 (+ 50898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.764199 (+ 10024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.764559 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.768180 12490 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:36.769285 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2963) took 1217 ms. Trace:
I20260709 12:14:36.769755 12641 rpcz_store.cc:276] 0709 12:14:35.551677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.721901 (+1170224us) service_pool.cc:224] Handling call
0709 12:14:36.754139 (+ 32238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.763576 (+  9437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.769127 (+  5551us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":260992}
W20260709 12:14:36.765949 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76406us
W20260709 12:14:36.766292 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56017.3us
I20260709 12:14:36.772782 19226 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:36.773990 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 148223us
I20260709 12:14:36.775290 19424 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.775759 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:36.776165 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 907) took 2373 ms. Trace:
I20260709 12:14:36.776944 19424 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:36.776729 12493 rpcz_store.cc:276] 0709 12:14:34.402959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.774834 (+2371875us) service_pool.cc:224] Handling call
0709 12:14:36.775349 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.775694 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:36.775717 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:36.775774 (+    57us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:36.775808 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.775860 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.775980 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.777791 19456 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:36.778460 19456 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:36.784428 12489 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:36.732702 19351 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:36.770905 19546 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.749s	user 0.011s	sys 0.005s
I20260709 12:14:36.778894 19456 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:36.790884 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 905) took 2397 ms. Trace:
I20260709 12:14:36.795059 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:36.793217 12489 rpcz_store.cc:276] 0709 12:14:34.389395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.778181 (+2388786us) service_pool.cc:224] Handling call
0709 12:14:36.778643 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.784981 (+  6338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.787007 (+  2026us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.805214 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:36.749719 19460 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.813714 19533 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:36.822199 19547 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:36.823990 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:36.825131 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 193751us
I20260709 12:14:36.785902 19424 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.835282 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:36.823148 19547 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.793411 19546 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.841150 19283 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:36.847255 12641 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.856950 19400 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:36.859663 12565 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 12:14:36.846725 19296 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:36.877159 12646 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:36.860199 12565 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:36.879842 12565 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:14:36.882480 12565 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:36.858517 19228 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.723s	user 0.017s	sys 0.000s
I20260709 12:14:36.888926 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:36.889851 19565 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:36.891062 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:36.890699 19565 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.895272 19283 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:36.895705 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.888245 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3140) took 1011 ms. Trace:
W20260709 12:14:36.897922 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47300.5us
I20260709 12:14:36.897940 12569 rpcz_store.cc:276] 0709 12:14:35.871089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.802869 (+931780us) service_pool.cc:224] Handling call
0709 12:14:36.831401 (+ 28532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.881911 (+ 50510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:36.881959 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:36.882036 (+    77us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:36.882102 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.882202 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.882395 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:36.898325 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:36.883311 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:36.879776 19242 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.738s	user 0.012s	sys 0.005s
I20260709 12:14:36.890043 12646 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:36.909019 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:36.913827 19400 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.922660 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:36.926601 19174 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:36.927253 19174 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:36.928390 19174 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:36.930001 19351 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.934087 19426 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:14:36.938203 19559 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:14:36.939519 19497 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:36.901047 19228 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:36.943609 12496 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:36.947427 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3139) took 1038 ms. Trace:
I20260709 12:14:36.925510 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:36.948232 12565 rpcz_store.cc:276] 0709 12:14:35.870592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.769895 (+899303us) service_pool.cc:224] Handling call
0709 12:14:36.810617 (+ 40722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.908440 (+ 97823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.908773 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:36.956076 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:36.957448 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 894) took 2745 ms. Trace:
I20260709 12:14:36.895735 19283 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:36.967267 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:36.970906 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:36.974326 19424 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:36.966113 19283 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:36.943159 19242 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:36.958637 12643 rpcz_store.cc:276] 0709 12:14:34.205474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.074035 (+1868561us) service_pool.cc:224] Handling call
0709 12:14:36.176917 (+102882us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.247569 (+ 70652us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:14:36.248009 (+   440us) log.cc:844] Serialized 14 byte log entry
0709 12:14:36.950646 (+702637us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:36.950696 (+    50us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:14:36.950733 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.950901 (+   168us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:36.951034 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":201,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":56564,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":104,"thread_start_us":689507,"threads_started":2,"wal-append.queue_time_us":548714}
W20260709 12:14:36.999078 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44560.8us
I20260709 12:14:36.999818 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:37.003176 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:36.971746 19565 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:37.020610 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.021682 12569 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:37.025678 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:37.027575 12567 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: [17.10-18.11]
I20260709 12:14:37.028455 19497 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:37.031388 19563 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260709 12:14:37.031958 19387 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.871s	user 0.014s	sys 0.005s
W20260709 12:14:37.032855 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 80763.6us
W20260709 12:14:37.014770 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2610) took 2343 ms (client timeout 1890 ms). Trace:
I20260709 12:14:37.025068 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3129) took 1160 ms. Trace:
I20260709 12:14:37.032503 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:37.038450 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51628.3us
W20260709 12:14:37.034494 12637 rpcz_store.cc:269] 0709 12:14:34.671387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:35.570281 (+898894us) service_pool.cc:224] Handling call
0709 12:14:36.992496 (+1422215us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:37.014670 (+ 22174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":592679296}
I20260709 12:14:37.043334 12497 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:37.047843 13258 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:37.016043 19460 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:37.050431 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125017us
I20260709 12:14:37.035048 12570 rpcz_store.cc:276] 0709 12:14:35.855538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.738877 (+883339us) service_pool.cc:224] Handling call
0709 12:14:36.758941 (+ 20064us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.780700 (+ 21759us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:36.793075 (+ 12375us) log.cc:844] Serialized 14 byte log entry
0709 12:14:37.015914 (+222839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:37.015966 (+    52us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:37.016021 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.016092 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.016257 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":12140,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":3340,"raft.queue_time_us":109,"thread_start_us":178606,"threads_started":1}
I20260709 12:14:37.057039 19561 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:37.060253 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.067376 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.061484 12563 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:36.999981 19426 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:14:37.070726 19494 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:37.071554 19494 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.073807 19528 raft_consensus.cc:2764] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:14:37.046981 19412 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:37.081691 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:37.085039 12638 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:37.082600 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2961) took 1566 ms. Trace:
I20260709 12:14:37.086382 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.091197 19387 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:37.092010 19387 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:37.092629 19387 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:37.056030 19456 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.096839 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3138) took 1226 ms. Trace:
I20260709 12:14:37.097419 12566 rpcz_store.cc:276] 0709 12:14:35.870171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.747654 (+877483us) service_pool.cc:224] Handling call
0709 12:14:36.786165 (+ 38511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:36.823015 (+ 36850us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:36.823081 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:36.830890 (+  7809us) log.cc:844] Serialized 26 byte log entry
0709 12:14:37.020513 (+189623us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:37.020574 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.020664 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.096460 (+ 75796us) raft_consensus.cc:1685] finished
0709 12:14:37.096500 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.096731 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":964,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":371,"thread_start_us":138948,"threads_started":1,"wal-append.queue_time_us":211576}
I20260709 12:14:37.103657 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:37.104924 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:37.108639 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:37.110947 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:37.086601 12639 rpcz_store.cc:276] 0709 12:14:35.515839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.693927 (+1178088us) service_pool.cc:224] Handling call
0709 12:14:36.698145 (+  4218us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:14:36.773180 (+ 75035us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:36.773209 (+    29us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:14:36.774034 (+   825us) log.cc:844] Serialized 78 byte log entry
0709 12:14:36.991070 (+217036us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:36.991098 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.991150 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.082179 (+ 91029us) raft_consensus.cc:1685] finished
0709 12:14:37.082199 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.082424 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":147,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":157,"thread_start_us":216565,"threads_started":1,"wal-append.queue_time_us":256755}
I20260709 12:14:37.113766 12566 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:37.114180 12637 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:14:37.115135 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:37.115862 12566 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:37.088814 19460 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:37.083972 19565 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:37.135381 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:37.081352 19412 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:37.148403 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44906.4us
I20260709 12:14:37.152455 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:37.107656 19426 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:37.159401 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58798.1us
I20260709 12:14:37.182546 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3079) took 1035 ms. Trace:
I20260709 12:14:37.182984 12570 rpcz_store.cc:276] 0709 12:14:36.147160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.054322 (+907162us) service_pool.cc:224] Handling call
0709 12:14:37.067759 (+ 13437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:37.152047 (+ 84288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:37.152105 (+    58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:37.159362 (+  7257us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:37.182281 (+ 22919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.182350 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.182490 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:37.180656 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:37.188437 13254 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:37.156251 19296 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:37.156822 19412 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:37.137516 19567 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:37.194559 12643 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:37.118219 19460 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:37.198246 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 183005us
I20260709 12:14:37.197702 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.197667 19242 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.178977 19565 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:37.190004 19456 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:37.184945 19494 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:37.189375 19547 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.195037 19296 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:37.195376 19567 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:37.210847 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:37.210824 19296 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:37.229179 19562 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:37.234613 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3141) took 1363 ms. Trace:
I20260709 12:14:37.235201 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.236217 19437 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:37.237269 19578 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.200598 19533 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:14:37.207628 19400 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.239933 19460 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:37.244136 19546 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.246759 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 897) took 2976 ms. Trace:
I20260709 12:14:37.248063 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:37.211073 19567 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:37.250546 19570 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.092s
I20260709 12:14:37.252277 19494 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:37.254566 19574 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:14:37.259119 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
W20260709 12:14:37.259940 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 185335us
I20260709 12:14:37.224884 19283 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.214282 19242 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.309077 12572 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:37.307891 12645 rpcz_store.cc:276] 0709 12:14:34.270176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.069201 (+1799025us) service_pool.cc:224] Handling call
0709 12:14:36.180999 (+111798us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.589536 (+408537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.246426 (+656890us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b5400152f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:37.246647 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102207232}
I20260709 12:14:37.316396 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:37.275123 12568 rpcz_store.cc:276] 0709 12:14:35.871576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.798669 (+927093us) service_pool.cc:224] Handling call
0709 12:14:36.812659 (+ 13990us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:36.841838 (+ 29179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:36.841876 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:36.869082 (+ 27206us) log.cc:844] Serialized 26 byte log entry
0709 12:14:37.048068 (+178986us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:37.048115 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.048204 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.234165 (+185961us) raft_consensus.cc:1685] finished
0709 12:14:37.234189 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.234378 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4994,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":10264,"thread_start_us":120555,"threads_started":1,"wal-append.queue_time_us":191566}
I20260709 12:14:37.325894 13254 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:37.326876 19546 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:37.325417 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:37.336341 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:37.340519 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 271767us
I20260709 12:14:37.318107 19400 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:37.350623 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:14:37.353591 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 173001us
I20260709 12:14:37.354254 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:37.354489 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 294321us
I20260709 12:14:37.358307 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:37.340081 19546 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:37.371450 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64758.4us
I20260709 12:14:37.380995 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.359345 19456 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:37.355170 19585 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:37.367331 19533 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:14:37.391012 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155950us
I20260709 12:14:37.367043 19595 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:37.353240 19578 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.366011 19242 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.396960 19456 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.401883 19584 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:37.362877 19400 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.392266 19494 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.403438 12568 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:14:37.408931 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3148) took 1237 ms. Trace:
I20260709 12:14:37.409303 12566 rpcz_store.cc:276] 0709 12:14:36.171236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.372446 (+1201210us) service_pool.cc:224] Handling call
0709 12:14:37.388489 (+ 16043us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:37.408628 (+ 20139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:37.408653 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:37.408697 (+    44us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:37.408728 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.408786 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.408879 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:37.408797 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3091) took 1243 ms. Trace:
I20260709 12:14:37.398025 19595 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:37.440487 19587 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:14:37.439462 19585 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:37.441387 19577 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.086s	user 0.003s	sys 0.000s
I20260709 12:14:37.440958 19494 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:37.442597 19589 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:14:37.442250 19584 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.400879 19562 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:37.399633 19533 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:37.445976 19533 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.446457 19595 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.451453 19242 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.442278 19580 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:37.455735 19580 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.458191 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3149) took 1286 ms. Trace:
I20260709 12:14:37.458729 12566 rpcz_store.cc:276] 0709 12:14:36.171511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.410126 (+1238615us) service_pool.cc:224] Handling call
0709 12:14:37.457406 (+ 47280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:37.457754 (+   348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:37.457786 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:37.457850 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:37.457897 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.457963 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.458104 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:37.444223 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3078) took 1297 ms. Trace:
I20260709 12:14:37.462604 19283 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.488792 19584 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:37.414270 12568 rpcz_store.cc:276] 0709 12:14:36.165670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.356933 (+1191263us) service_pool.cc:224] Handling call
0709 12:14:37.380807 (+ 23874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:37.408500 (+ 27693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.408725 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:37.491312 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120776us
I20260709 12:14:37.488075 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3082) took 1326 ms. Trace:
I20260709 12:14:37.487166 19494 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.491381 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3080) took 1343 ms. Trace:
W20260709 12:14:37.488523 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101729us
I20260709 12:14:37.492137 12563 rpcz_store.cc:276] 0709 12:14:36.161986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.082468 (+920482us) service_pool.cc:224] Handling call
0709 12:14:37.100314 (+ 17846us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:37.156681 (+ 56367us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:37.156718 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:37.187864 (+ 31146us) log.cc:844] Serialized 26 byte log entry
0709 12:14:37.380709 (+192845us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:37.380760 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.380846 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.487764 (+106918us) raft_consensus.cc:1685] finished
0709 12:14:37.487790 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.487993 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30175,"raft.queue_time_us":38135,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":22865,"thread_start_us":173542,"threads_started":1,"wal-append.queue_time_us":266011}
I20260709 12:14:37.493593 12566 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:14:37.487705 12567 rpcz_store.cc:276] 0709 12:14:36.146839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.028425 (+881586us) service_pool.cc:224] Handling call
0709 12:14:37.028907 (+   482us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:37.029897 (+   990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:37.029946 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:37.108281 (+ 78335us) log.cc:844] Serialized 52 byte log entry
0709 12:14:37.302477 (+194196us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:37.384354 (+ 81877us) log.cc:844] Serialized 14 byte log entry
0709 12:14:37.389693 (+  5339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.389791 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.443894 (+ 54103us) raft_consensus.cc:1685] finished
0709 12:14:37.443927 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.444137 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":68934,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":6028,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":153,"spinlock_wait_cycles":14507776,"thread_start_us":149671,"threads_started":2,"wal-append.queue_time_us":149464}
I20260709 12:14:37.489416 19595 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:37.499420 12571 rpcz_store.cc:276] 0709 12:14:36.147484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.059024 (+911540us) service_pool.cc:224] Handling call
0709 12:14:37.059571 (+   547us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:37.081436 (+ 21865us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:37.081478 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:37.083365 (+  1887us) log.cc:844] Serialized 26 byte log entry
0709 12:14:37.207088 (+123723us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:37.207142 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.207242 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.491118 (+283876us) raft_consensus.cc:1685] finished
0709 12:14:37.491140 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.491320 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7232,"raft.queue_time_us":23378,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":8580,"thread_start_us":25653,"threads_started":1,"wal-append.queue_time_us":154155}
I20260709 12:14:37.522049 12568 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:37.387809 19565 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:37.529721 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56043.8us
I20260709 12:14:37.527786 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1176) took 1488 ms. Trace:
I20260709 12:14:37.536449 19578 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.538748 12570 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:37.543517 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.532533 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:37.531839 12569 rpcz_store.cc:276] 0709 12:14:36.039682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:37.033104 (+993422us) service_pool.cc:224] Handling call
0709 12:14:37.050060 (+ 16956us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:37.073763 (+ 23703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:37.073812 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:37.074450 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:14:37.195789 (+121339us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:14:37.211375 (+ 15586us) log.cc:844] Serialized 14 byte log entry
0709 12:14:37.527132 (+315757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.527233 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.527537 (+   304us) raft_consensus.cc:1685] finished
0709 12:14:37.527554 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.527724 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41448,"prepare.queue_time_us":16310,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":117353,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":23414,"thread_start_us":436228,"threads_started":2,"wal-append.queue_time_us":130528}
I20260709 12:14:37.457082 19580 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.554248 12572 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
W20260709 12:14:37.554927 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60107.8us
I20260709 12:14:37.556963 19588 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:37.557885 19588 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:37.558403 19400 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:37.557176 19604 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:37.560946 19604 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.558516 19588 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:37.565855 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:37.566689 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.574618 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.529791 19412 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.578467 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.523211 19546 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.587836 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:37.582553 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:37.538753 19387 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.583408 19497 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:37.583832 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:37.584005 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50274.7us
I20260709 12:14:37.584659 12645 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:37.561467 19604 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:37.577730 19578 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 12:14:37.586714 19585 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.579195 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:37.582041 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:37.615712 13256 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:14:37.630318 19353 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:37.633106 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58606.3us
I20260709 12:14:37.592247 19494 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:14:37.643272 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80489us
I20260709 12:14:37.644481 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:37.646039 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 891) took 4869 ms. Trace:
I20260709 12:14:37.637487 19584 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:37.592074 19456 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.646471 19585 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:14:37.665591 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93051.1us
I20260709 12:14:37.670795 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:37.671890 12497 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:37.673797 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:37.650189 19578 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.673461 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.663940 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:37.676362 19600 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:37.677032 19600 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:37.678078 19600 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:37.678719 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59737.3us
I20260709 12:14:37.679883 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.652818 19584 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:37.685447 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.686335 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:37.688906 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:37.669212 19585 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:37.695729 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 939) took 1462 ms (client timeout 1616 ms). Trace:
W20260709 12:14:37.696187 13258 rpcz_store.cc:269] 0709 12:14:36.242662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.265320 (+ 22658us) service_pool.cc:224] Handling call
0709 12:14:37.695521 (+1430201us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:37.695655 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1033241344}
I20260709 12:14:37.698313 19534 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:14:37.648454 12642 rpcz_store.cc:276] 0709 12:14:32.704829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:32.704992 (+   163us) service_pool.cc:224] Handling call
0709 12:14:32.705384 (+   392us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:14:32.795978 (+ 90594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:32.796009 (+    31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:14:32.796815 (+   806us) log.cc:844] Serialized 78 byte log entry
0709 12:14:32.982141 (+185326us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:33.046872 (+ 64731us) log.cc:844] Serialized 14 byte log entry
0709 12:14:33.047282 (+   410us) log.cc:844] Serialized 14 byte log entry
0709 12:14:33.952028 (+904746us) spinlock_profiling.cc:243] Waited 7.21 ms on lock 0x7b18004552c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:35.650873 (+1698845us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:35.673520 (+ 22647us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b18004552c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:14:35.820551 (+147031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.564026 (+1743475us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:37.569128 (+  5102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.574296 (+  5168us) raft_consensus.cc:1685] finished
0709 12:14:37.574315 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.574506 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44471,"prepare.queue_time_us":150075,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":19133,"raft.queue_time_us":113277,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":21020,"spinlock_wait_cycles":211393664,"thread_start_us":267579,"threads_started":2,"wal-append.queue_time_us":314745,"wal-append.run_cpu_time_us":28815,"wal-append.run_wall_time_us":3603172}
I20260709 12:14:37.710206 19183 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:37.713174 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2956) took 2281 ms. Trace:
I20260709 12:14:37.714128 12491 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 25 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:14:37.716051 19400 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:37.717808 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:37.742573 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:37.722406 19560 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:14:37.723397 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.723906 19581 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:37.724289 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.701433 19585 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:37.726308 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.727438 19296 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.730132 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:37.726307 19412 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.745007 19400 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:37.763473 19571 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.586s	user 0.004s	sys 0.000s
I20260709 12:14:37.731611 19609 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:37.764166 19609 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:37.764608 19609 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:37.708685 19426 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:37.700098 19533 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:37.732833 19351 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:14:37.771576 19584 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.773320 19584 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:37.734087 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2949) took 4214 ms. Trace:
I20260709 12:14:37.735772 19567 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.721455 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:37.790750 19533 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:14:37.792943 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71598.8us
I20260709 12:14:37.793609 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:37.745282 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57315.1us
I20260709 12:14:37.730139 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.796080 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:37.759243 12489 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:14:37.797129 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:37.798228 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72044.7us
I20260709 12:14:37.799081 19547 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:37.799623 19547 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:37.800020 19547 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:37.800341 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.801990 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.802408 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:37.803122 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79841.1us
I20260709 12:14:37.803486 13258 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:14:37.803894 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.804044 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.803470 19296 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:37.804769 13256 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:37.750578 19581 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.805918 19351 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:37.807018 19494 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:37.808079 19494 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:37.781407 12644 rpcz_store.cc:276] 0709 12:14:33.519465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.074556 (+2555091us) service_pool.cc:224] Handling call
0709 12:14:36.108766 (+ 34210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:36.131699 (+ 22933us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:36.132250 (+   551us) log.cc:844] Serialized 14 byte log entry
0709 12:14:36.946133 (+813883us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:36.946205 (+    72us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:36.946272 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:36.946356 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.733856 (+787500us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b54002e5750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:37.734012 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13574,"prepare.queue_time_us":491,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":68,"raft.queue_time_us":81220,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":116,"spinlock_wait_cycles":14695296,"thread_start_us":794276,"threads_started":2,"wal-append.queue_time_us":534617,"wal-append.run_cpu_time_us":1400,"wal-append.run_wall_time_us":1003720}
I20260709 12:14:37.809675 19283 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:37.749807 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.733569 18986 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:37.694089 19578 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.812884 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.813901 13254 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:37.774061 12495 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:37.780570 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:37.737432 12640 rpcz_store.cc:276] 0709 12:14:35.432241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:36.148603 (+716362us) service_pool.cc:224] Handling call
0709 12:14:36.852796 (+704193us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:36.853060 (+   264us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:36.853657 (+   597us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:14:37.020951 (+167294us) log.cc:844] Serialized 14 byte log entry
0709 12:14:37.052868 (+ 31917us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:37.090173 (+ 37305us) log.cc:844] Serialized 52 byte log entry
0709 12:14:37.090320 (+   147us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:14:37.099774 (+  9454us) log.cc:844] Serialized 14 byte log entry
0709 12:14:37.100109 (+   335us) log.cc:844] Serialized 14 byte log entry
0709 12:14:37.236677 (+136568us) spinlock_profiling.cc:243] Waited 5.31 ms on lock 0x7b1800379960. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:37.237790 (+  1113us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:14:37.712432 (+474642us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b1800379960. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:14:37.712651 (+   219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:37.712764 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:37.712832 (+    68us) raft_consensus.cc:1685] finished
0709 12:14:37.712849 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:37.713078 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48291,"prepare.queue_time_us":241121,"prepare.run_cpu_time_us":188,"prepare.run_wall_time_us":2247,"raft.queue_time_us":122128,"raft.run_cpu_time_us":3053,"raft.run_wall_time_us":205027,"spinlock_wait_cycles":63497984,"thread_start_us":142067,"threads_started":3,"wal-append.queue_time_us":118298}
I20260709 12:14:37.806752 19581 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.763613 19400 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:37.810664 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.810940 12496 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:14:37.811842 12563 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23.
I20260709 12:14:37.805047 19296 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:37.822819 12570 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:37.740873 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:37.824731 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.825606 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:37.826471 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.827270 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:37.825050 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.828751 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:37.827230 13256 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 15.
I20260709 12:14:37.832073 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.709167 19580 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.103s
I20260709 12:14:37.842406 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.830804 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:37.810061 19283 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:37.866485 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:37.816918 19497 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:37.868242 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.869917 13258 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:14:37.876859 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.830040 19183 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.881496 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:37.883275 12497 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:37.867735 19578 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.894042 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:37.896652 19582 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.716s	user 0.002s	sys 0.000s
W20260709 12:14:37.897935 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66010.2us
I20260709 12:14:37.899652 19567 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:37.872641 19497 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:37.873273 18986 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:37.938738 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.942214 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:37.879101 19583 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:14:37.943320 13259 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:37.903741 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.904013 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.904698 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.904465 19183 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.907008 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.908259 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:37.909349 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:37.910100 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104560us
I20260709 12:14:37.888978 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:37.910650 13256 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:37.911561 19387 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.912047 19609 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.889523 19568 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:14:37.889087 19611 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:14:37.892688 19604 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:37.918825 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 178099us
I20260709 12:14:37.966674 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:37.919679 19617 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:37.970103 19617 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.959246 19296 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.971369 19617 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.9, Last appended by leader: 9, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.971482 19296 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:37.976001 19561 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:37.976805 19561 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.977949 19561 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.978581 19228 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.980865 19590 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:14:37.920143 19245 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:37.982019 19245 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:37.982527 19245 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:37.982990 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44355.3us
I20260709 12:14:37.983150 19296 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:37.984030 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.985028 19595 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:37.990805 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:37.991559 19604 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:37.402755 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:37.993938 19609 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:37.995877 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:37.996783 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:37.998705 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90558.8us
I20260709 12:14:37.999387 19568 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:37.999500 19426 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:37.999912 19568 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:38.000267 19426 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:37.999913 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:38.001462 13256 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:14:37.950587 19412 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:37.887168 19565 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:14:38.003089 19578 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:14:37.927577 19603 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20260709 12:14:37.931707 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:38.003695 19424 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:37.934832 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:38.004597 19528 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:37.935247 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:37.936663 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:38.003144 19412 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:38.006450 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69897.4us
I20260709 12:14:38.003777 19565 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:38.007453 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:38.007682 19565 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:38.008665 19565 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:37.938889 19497 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:38.009382 19534 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:38.009994 19534 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:38.010320 19426 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.010984 19534 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:37.901901 19607 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:14:37.946913 12496 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 15.
I20260709 12:14:38.015774 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:37.953231 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:37.924523 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:37.954239 12570 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:38.000262 19168 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:38.003588 19578 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:38.011509 19426 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:37.950690 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71265.1us
I20260709 12:14:38.018290 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:37.960042 13254 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:14:38.027525 12497 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:38.024627 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:38.035684 19591 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:38.036357 19591 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:38.036768 19591 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:37.960659 19437 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:38.041143 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:38.039179 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:38.005157 19424 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:38.047101 12495 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:38.047981 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:38.052170 12497 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:38.048754 12565 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:38.010018 19565 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:37.984054 19296 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:38.000566 19568 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:38.056087 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:38.046761 19494 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:14:38.061305 19590 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:38.069130 19437 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:38.062098 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76525.7us
W20260709 12:14:38.062844 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66151.1us
I20260709 12:14:38.042356 19228 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:38.066540 12493 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:38.076215 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:38.009297 19412 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:38.039832 19578 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:38.047464 19424 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:38.082500 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:38.071280 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 181537us
I20260709 12:14:38.067378 19604 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:38.090965 13255 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:37.989338 19283 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:38.102378 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 165285us
I20260709 12:14:38.106046 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:38.107986 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:38.108208 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:38.080029 19228 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:38.109160 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:38.111529 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70258.8us
I20260709 12:14:38.061160 19245 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.093670 19590 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:38.125070 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:38.125978 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:38.121470 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:38.122869 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:38.123260 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:38.146009 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:38.090512 19581 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:38.127460 12495 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:38.150424 12643 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:38.113518 19228 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:38.157961 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:38.157969 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:38.154104 19226 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:38.122514 19283 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:38.156822 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:38.148784 19604 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:38.174091 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65021.8us
I20260709 12:14:38.168006 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:38.145284 19174 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:38.111636 19528 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.047s
W20260709 12:14:38.165491 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63906us
I20260709 12:14:38.169214 19226 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.170303 19581 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.170856 19424 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.172763 12571 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:14:38.152143 19567 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:14:38.174090 19437 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:38.152254 19412 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:14:38.174605 19604 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:38.217741 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111606us
I20260709 12:14:38.187491 19183 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:38.226562 12640 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:38.188849 19617 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:14:38.224524 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49004.6us
I20260709 12:14:38.224903 19528 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:38.233574 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 127579us
I20260709 12:14:38.187194 19585 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.184346 19590 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.042s
W20260709 12:14:38.232095 19588 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.647s	user 0.009s	sys 0.005s
I20260709 12:14:38.248368 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:38.250702 19588 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.254035 19588 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:38.257668 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:38.188023 19547 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.232375 19604 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:38.225059 19174 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:38.283603 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 171214us
W20260709 12:14:38.284245 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 126899us
W20260709 12:14:38.299913 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73417.3us
W20260709 12:14:38.289080 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49681us
I20260709 12:14:38.236053 19485 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
W20260709 12:14:38.345160 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 150204us
I20260709 12:14:38.348909 12460 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23.
I20260709 12:14:38.349404 12460 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:14:38.336059 19619 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:14:38.360795 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:38.307054 19437 raft_consensus.cc:2764] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
W20260709 12:14:38.366159 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 117721us
W20260709 12:14:38.366811 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65960.5us
I20260709 12:14:38.371723 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:38.373096 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:38.373366 13255 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:14:38.346396 19424 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:38.306634 19183 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:14:38.369462 19628 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:38.324533 19580 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.050s
W20260709 12:14:38.388530 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62423us
I20260709 12:14:38.378136 19245 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:38.396752 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:38.435667 12571 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:14:38.403554 19583 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:38.396901 19609 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:38.452382 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 221722us
I20260709 12:14:38.442729 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:38.457247 19622 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:14:38.457572 19639 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:38.458249 19639 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:38.458751 19639 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:38.460747 19546 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:38.461731 19183 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:38.462198 19226 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:38.462276 19183 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:38.463425 19183 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.464956 19183 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:38.465471 19626 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:14:38.466374 19635 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:14:38.467274 19626 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:38.435017 19174 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:38.485380 19624 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
W20260709 12:14:38.487808 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45172.4us
W20260709 12:14:38.449553 18986 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.510s	user 0.009s	sys 0.004s
I20260709 12:14:38.493201 12495 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:38.448864 19400 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:14:38.408987 19437 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:14:38.428505 19617 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:38.448493 19583 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:38.465510 19168 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23.
I20260709 12:14:38.544802 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:38.528723 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:38.560068 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148595us
I20260709 12:14:38.487733 19494 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:38.565781 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:38.526203 19626 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:38.525058 18986 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:38.591275 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:38.478983 19609 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.485200 19628 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.099s
I20260709 12:14:38.570977 19581 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:38.609172 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100465us
I20260709 12:14:38.610507 12536 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:38.610872 12536 leader_election.cc:403] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:38.612295 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:38.610890 19591 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.000s
I20260709 12:14:38.614277 19591 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.592342 19633 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:38.579950 19585 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:38.604527 19561 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.014s	sys 0.014s
I20260709 12:14:38.596719 19296 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:38.590538 19626 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:38.638090 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:38.643347 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:38.599684 19168 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:38.600404 19547 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:38.648343 19628 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260709 12:14:38.676208 19646 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:38.670357 13255 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:14:38.632104 19633 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:38.632807 19534 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
W20260709 12:14:38.670063 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:38.608368 19494 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.675042 19245 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:38.670943 19296 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:38.737694 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:38.739566 12563 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:14:38.740249 12495 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:38.648186 19561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.014s	sys 0.015s
W20260709 12:14:38.658367 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 296306us
I20260709 12:14:38.677968 19649 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:38.639777 18986 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:38.755399 19296 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.702906 13262 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:14:38.758061 19649 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:38.759521 19649 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.760561 19561 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:38.742877 19245 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.763732 19387 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:38.764238 19387 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:38.764797 19497 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.755s	user 0.011s	sys 0.000s
W20260709 12:14:38.767855 19568 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.692s	user 0.011s	sys 0.001s
I20260709 12:14:38.768376 19568 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:38.772013 19565 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.701s	user 0.009s	sys 0.003s
W20260709 12:14:38.775211 19578 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.683s	user 0.012s	sys 0.000s
I20260709 12:14:38.775770 19578 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.777235 12489 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
W20260709 12:14:38.779040 19283 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.594s	user 0.010s	sys 0.004s
W20260709 12:14:38.779794 19228 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.004s
I20260709 12:14:38.780390 19228 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.781606 19609 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:38.782330 19639 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.784754 19226 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:38.785251 19226 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:38.795019 12490 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:14:38.700060 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 299670us
I20260709 12:14:38.687980 19626 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:38.761564 19426 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.720s	user 0.009s	sys 0.001s
W20260709 12:14:38.737582 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146414us
W20260709 12:14:38.804215 12640 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.572s	user 0.014s	sys 0.000s
I20260709 12:14:38.806107 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 16.
I20260709 12:14:38.749738 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:38.698375 19617 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.800753 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:38.807080 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57518.4us
I20260709 12:14:38.807866 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:38.700347 19584 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.912s	user 0.006s	sys 0.011s
I20260709 12:14:38.809849 19485 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:38.812443 19647 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:38.812978 19647 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:38.741545 19646 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:38.805992 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:38.809408 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:38.698307 19168 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:38.765287 19497 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.818897 13255 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:38.813392 19647 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:38.742998 19653 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.078s
I20260709 12:14:38.830682 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:38.829155 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 263443us
I20260709 12:14:38.801410 19591 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:38.799057 19641 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:14:38.840376 12642 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:38.846455 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:38.811693 19245 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:38.849148 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:38.850471 12496 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:14:38.816830 19604 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.853484 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:38.812482 19485 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:38.828043 19547 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:38.829989 19168 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:38.765288 19387 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:38.908303 12572 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:38.855242 19584 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:38.917323 13256 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
W20260709 12:14:38.922125 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112830us
I20260709 12:14:38.908776 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:38.956459 12568 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:38.838317 19437 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:38.859122 19412 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:38.957217 13257 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
W20260709 12:14:38.955557 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 343333us
W20260709 12:14:38.873206 19590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.009s	sys 0.024s
I20260709 12:14:38.874996 19485 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:38.842849 19226 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:38.858626 19565 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:14:38.909493 19578 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:38.986424 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:38.971069 19590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.012s	sys 0.033s
I20260709 12:14:38.990345 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:38.928597 19609 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:14:38.928609 19245 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:14:38.971717 19387 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:39.006171 13257 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:39.006943 19643 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:39.014851 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:38.974465 19412 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.019014 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:38.992987 19424 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.005s
I20260709 12:14:38.841452 19633 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:39.091370 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72561.3us
I20260709 12:14:38.995024 19568 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:39.095855 12494 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:39.092367 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:39.102339 19174 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:14:39.106307 19625 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.752s	user 0.005s	sys 0.010s
I20260709 12:14:39.109413 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3311) took 1044 ms. Trace:
I20260709 12:14:39.109982 12489 rpcz_store.cc:276] 0709 12:14:38.065356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.068286 (+  2930us) service_pool.cc:224] Handling call
0709 12:14:38.082753 (+ 14467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:38.777675 (+694922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.108965 (+331290us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:39.109302 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1596653952}
I20260709 12:14:39.111724 19412 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:39.117421 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.121568 19437 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:39.121744 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.122931 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.122826 19437 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.123629 12565 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:39.123912 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 906) took 1039 ms. Trace:
I20260709 12:14:39.124348 12640 rpcz_store.cc:276] 0709 12:14:38.084740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.084898 (+   158us) service_pool.cc:224] Handling call
0709 12:14:39.123710 (+1038812us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:39.123845 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":249271040}
W20260709 12:14:39.124946 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:39.126269 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139940us
I20260709 12:14:39.126754 19228 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:39.127210 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:39.127707 19534 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:39.128136 19534 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:39.128504 19534 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:39.129702 19644 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:39.130328 19591 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:39.131258 19661 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:14:39.132371 19497 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:39.133419 19226 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:39.128715 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:39.136967 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:39.137884 19533 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:39.138520 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:14:39.148717 19494 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:39.151795 19534 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.177201 19534 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:39.179487 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
W20260709 12:14:39.180576 19528 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.897s	user 0.000s	sys 0.014s
I20260709 12:14:39.181272 19528 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:38.975013 19646 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.182807 19534 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
W20260709 12:14:39.186313 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47949.6us
I20260709 12:14:39.187254 19642 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:39.188179 19642 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:39.199436 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 321632us
I20260709 12:14:39.202225 19633 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:39.205965 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 245979us
I20260709 12:14:39.207747 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.208391 19658 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:39.208880 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.209002 19658 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.209546 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.210117 12491 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:39.210799 19628 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.211256 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.211330 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.212321 12491 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:39.212561 12493 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:39.212749 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.214702 19658 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:39.213608 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.211359 19628 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.218559 12491 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:39.218658 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:39.219357 12643 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:39.219604 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:39.219899 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.214700 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:39.154075 19643 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:39.159672 19588 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:39.262965 19588 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:39.166829 19583 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:39.020581 19183 consensus_queue.cc:1175] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 20 status { last_received { term: 20 index: 10 } last_committed_idx: 9 last_received_current_leader { term: 20 index: 10 } }
I20260709 12:14:39.271536 12495 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:14:39.220691 12565 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:39.213536 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.216475 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:39.236958 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.273439 12567 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:39.238624 12568 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:39.246189 19643 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.250362 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.275177 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.251461 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.253432 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.256982 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.278213 12640 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:39.166704 19604 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:39.268198 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.280850 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:39.281572 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:39.280849 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188573us
I20260709 12:14:39.283124 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.283815 12565 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:39.284878 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.272908 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:39.279201 12491 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:39.279650 12646 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:39.286705 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.254204 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:39.287225 13218 leader_election.cc:336] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.516s (SENT)
I20260709 12:14:39.287840 12489 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:39.281145 12569 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:39.271622 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.292383 12641 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:14:39.295220 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:39.296792 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.297514 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.299384 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.303448 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.304749 12565 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 16: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:39.309095 12495 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:39.314688 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.315248 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.314550 12493 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:39.323408 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:39.332266 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:39.298656 19600 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:39.335729 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.315622 12565 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:39.338318 12643 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
W20260709 12:14:39.338403 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51222.1us
I20260709 12:14:39.344517 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.353385 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.364880 19628 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:39.289839 19585 consensus_queue.cc:1175] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 20 status { last_received { term: 18 index: 15 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.16-19.16]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:14:39.367215 19642 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:39.368172 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:39.369091 12490 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:39.374681 12638 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:14:39.375082 12644 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:39.376013 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:39.379424 12391 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 8 to 10, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 10 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:14:39.384018 12639 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:39.384285 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3309) took 1319 ms. Trace:
I20260709 12:14:39.384860 12494 rpcz_store.cc:276] 0709 12:14:38.064583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.072826 (+  8243us) service_pool.cc:224] Handling call
0709 12:14:38.100206 (+ 27380us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:14:38.100430 (+   224us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:39.099235 (+998805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.383999 (+284764us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:39.384214 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2292345472}
I20260709 12:14:39.394943 12639 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:39.377014 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.396659 12644 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:39.289345 19588 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.420300 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.418903 19534 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.423689 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:39.423978 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.431953 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:39.432874 12570 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:14:39.418414 19647 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.004s
I20260709 12:14:39.436820 19585 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:39.436702 12391 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:39.451752 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:39.429136 19665 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:14:39.439463 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.442224 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:39.444608 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:39.449059 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:39.462590 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:39.463718 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.449231 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:39.434473 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:39.450222 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:39.465121 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.452661 12644 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:39.470995 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.358752 19485 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.480937 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.481388 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:39.481570 12565 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:39.480937 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.485837 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:39.486712 19426 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:14:39.487318 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.487656 19426 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:39.488159 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:39.489081 12390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 225) took 87460 ms (client timeout 15000 ms). Trace:
I20260709 12:14:39.489748 12492 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:39.490113 12494 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:14:39.494956 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:39.496256 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51730us
I20260709 12:14:39.416093 19168 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.459430 19283 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.492720 12391 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 12, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 12 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:39.498113 12565 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:39.513883 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.475368 19604 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:39.517777 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:39.520591 12572 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:14:39.526824 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.528645 12565 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:14:39.488124 19426 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:39.484655 19588 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:39.445400 19534 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.542166 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:39.543108 12494 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
W20260709 12:14:39.546317 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.008s	sys 0.018s
W20260709 12:14:39.547930 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.008s	sys 0.018s
I20260709 12:14:39.458446 19658 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:14:39.543282 19642 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.506042 19565 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.539016 19494 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:39.572889 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.593612 12644 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:39.594864 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.595293 12563 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:14:39.598558 19528 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:39.527590 19283 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.599184 19528 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:39.600253 19528 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.602263 19283 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:39.603152 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:39.573120 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:39.603426 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:39.633854 12641 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:39.485934 19567 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:14:39.652114 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.454334 19647 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.543151 19585 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.588578 19296 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:39.666107 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.544155 19546 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:14:39.627137 19528 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:39.591099 19642 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.646690 19580 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.646728 19600 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:39.675168 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:39.675659 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:39.649948 19648 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:39.654673 19604 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:39.680223 13258 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:39.688038 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.693279 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:39.580442 19533 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260709 12:14:39.703424 19645 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:39.705279 19626 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.885s	user 0.009s	sys 0.004s
W20260709 12:14:39.696115 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 910) took 1577 ms (client timeout 1584 ms). Trace:
W20260709 12:14:39.707337 12642 rpcz_store.cc:269] 0709 12:14:38.118690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.124538 (+  5848us) service_pool.cc:224] Handling call
0709 12:14:39.695860 (+1571322us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:39.696044 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1558988160}
I20260709 12:14:39.698696 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:39.672473 19663 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:14:39.669589 19590 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:39.647733 19565 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.695923 12391 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 9 to 11, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 11 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:39.715718 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:39.720547 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:39.725021 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:39.706373 19633 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.003s
I20260709 12:14:39.696457 19604 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:39.674109 19641 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:14:39.670264 19585 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.735566 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:39.711460 19456 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:39.739104 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:39.719214 19226 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.000s
I20260709 12:14:39.710978 19590 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:39.753504 12489 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:14:39.711553 19644 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:39.694736 19580 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.678634 19426 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.714354 19645 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.084s
I20260709 12:14:39.750694 12641 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:39.715296 19626 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:39.707796 19648 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.755517 12490 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:14:39.819644 12491 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:14:39.799635 19567 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:14:40.081420 19534 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:40.070261 12570 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.613s	user 0.009s	sys 0.004s
I20260709 12:14:39.933831 12391 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 6 to 9, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 9 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:14:40.107579 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:40.108151 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:39.720412 19632 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:39.898636 19485 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:40.052503 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.002s	sys 0.020s
I20260709 12:14:39.809746 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:40.069363 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 471382us
W20260709 12:14:39.860524 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:40.124589 12572 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:40.145224 12572 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:40.151383 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.007s	sys 0.020s
W20260709 12:14:40.151954 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 609859us
W20260709 12:14:40.055032 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 340353us
I20260709 12:14:40.104391 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3115) took 1608 ms. Trace:
W20260709 12:14:40.111028 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.798977 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:40.156106 12645 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:40.108987 12641 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
W20260709 12:14:39.935388 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:39.970968 13218 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.004s	sys 0.019s
W20260709 12:14:40.157563 13218 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.004s	sys 0.023s
W20260709 12:14:40.157974 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 803174us
W20260709 12:14:40.159559 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:40.187433 12565 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:40.153820 12564 rpcz_store.cc:276] 0709 12:14:38.495690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.495828 (+   138us) service_pool.cc:224] Handling call
0709 12:14:38.527392 (+ 31564us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:38.604023 (+ 76631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:38.604073 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:38.622524 (+ 18451us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.548438 (+925914us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:39.548499 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.548605 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:39.855584 (+306979us) raft_consensus.cc:1685] finished
0709 12:14:39.855607 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:40.104291 (+248684us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14047,"raft.queue_time_us":23183,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6360,"thread_start_us":854758,"threads_started":1,"wal-append.queue_time_us":1052303}
W20260709 12:14:39.850780 19675 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 12:14:40.189266 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:40.193338 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:40.194065 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:40.197885 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:39.864248 19663 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:14:39.860093 19600 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.146634 12572 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.108930 19590 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:39.899530 19649 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.947000 19664 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:39.848923 19296 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:40.315996 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:40.310518 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 151892us
I20260709 12:14:40.308674 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:40.307729 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.11-18.11]
W20260709 12:14:40.352669 13218 connection.cc:256] RPC call timeout handler was delayed by -1.0679816869999286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2335s for a response
W20260709 12:14:40.352588 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:40.354029 12572 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:40.313560 19676 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.362s	user 0.000s	sys 0.001s
I20260709 12:14:40.221383 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.320860 12641 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:14:40.318043 12570 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
W20260709 12:14:40.321439 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:40.327257 12637 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:14:40.329118 12638 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.021s	sys 0.020s
I20260709 12:14:39.856575 19168 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:40.023044 19628 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:40.341517 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188288us
I20260709 12:14:40.298096 12391 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 13, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 13 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:14:40.344542 12563 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.008s	sys 0.020s
W20260709 12:14:40.354199 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:40.352766 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
W20260709 12:14:40.356643 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:39.741210 19604 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:40.314637 19632 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:40.353689 19534 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:40.361596 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:40.353757 19296 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:40.315717 19688 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 12:14:40.164465 19633 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:40.164844 19658 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:40.391531 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:40.438571 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47109.2us
W20260709 12:14:40.391422 12563 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.009s	sys 0.021s
I20260709 12:14:40.391505 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:40.373555 12639 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.744s	user 0.013s	sys 0.000s
I20260709 12:14:40.441659 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:40.355646 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:40.223804 12493 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:40.297200 19648 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:40.407375 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:40.344431 19437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.004s	sys 0.022s
W20260709 12:14:40.445765 12640 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.027s	sys 0.019s
W20260709 12:14:40.446312 12640 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.027s	sys 0.020s
I20260709 12:14:40.396724 19646 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:40.446733 12572 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:14:40.411867 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:40.328723 19647 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:40.355630 19590 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:40.412822 19412 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:40.355718 19664 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:40.431005 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69504.1us
W20260709 12:14:40.394887 12638 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.024s	sys 0.021s
W20260709 12:14:40.403571 19685 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 12:14:40.442292 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3125) took 1016 ms. Trace:
I20260709 12:14:40.464741 12645 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:40.467243 12572 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:14:40.384248 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
W20260709 12:14:40.447014 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91477.7us
W20260709 12:14:40.454949 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 63563.2us
I20260709 12:14:40.447346 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2987) took 1076 ms. Trace:
I20260709 12:14:40.397300 19642 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:40.418957 19534 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:40.480530 12391 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 14, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 14 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:14:40.404726 19565 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:40.144989 19644 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.595961 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2994) took 1223 ms. Trace:
I20260709 12:14:40.599700 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.487224 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 845) took 1587 ms. Trace:
I20260709 12:14:40.489960 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:40.601745 12639 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:40.475899 19485 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:40.413650 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2606) took 1157 ms. Trace:
W20260709 12:14:40.449520 19437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.005s	user 0.008s	sys 0.022s
I20260709 12:14:40.600111 12638 rpcz_store.cc:276] 0709 12:14:39.372907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.377179 (+  4272us) service_pool.cc:224] Handling call
0709 12:14:39.377528 (+   349us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:39.424725 (+ 47197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.424766 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:39.425451 (+   685us) log.cc:844] Serialized 52 byte log entry
0709 12:14:39.657096 (+231645us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:40.503774 (+846678us) log.cc:844] Serialized 14 byte log entry
0709 12:14:40.595641 (+ 91867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.595718 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:40.595762 (+    44us) raft_consensus.cc:1685] finished
0709 12:14:40.595774 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:40.595907 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120991,"prepare.queue_time_us":471776,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":94042,"raft.queue_time_us":68647,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":1056554,"threads_started":2,"wal-append.queue_time_us":431263}
I20260709 12:14:40.398257 19245 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:14:40.346578 19649 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.445060 19633 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.601758 19534 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:40.697736 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:40.699172 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:40.700326 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:40.587658 12640 rpcz_store.cc:276] 0709 12:14:39.370507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.370701 (+   194us) service_pool.cc:224] Handling call
0709 12:14:39.371145 (+   444us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:39.371932 (+   787us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.371966 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:39.372678 (+   712us) log.cc:844] Serialized 52 byte log entry
0709 12:14:39.667577 (+294899us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:40.446664 (+779087us) log.cc:844] Serialized 14 byte log entry
0709 12:14:40.446972 (+   308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.447067 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:40.447123 (+    56us) raft_consensus.cc:1685] finished
0709 12:14:40.447139 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:40.447286 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10205,"prepare.queue_time_us":767420,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":30962,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":6695,"thread_start_us":1019363,"threads_started":2,"wal-append.queue_time_us":484906}
I20260709 12:14:40.451021 19658 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.480934 19695 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:40.484350 19168 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:40.600940 13255 rpcz_store.cc:276] 0709 12:14:38.900107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.900373 (+   266us) service_pool.cc:224] Handling call
0709 12:14:38.900940 (+   567us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:38.919658 (+ 18718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:38.921570 (+  1912us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:38.956829 (+ 35259us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.327029 (+370200us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:39.327102 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.327215 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:40.486883 (+1159668us) raft_consensus.cc:1685] finished
0709 12:14:40.486911 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:40.487157 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":214,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":30354,"thread_start_us":327707,"threads_started":1,"wal-append.queue_time_us":570615}
I20260709 12:14:40.609535 12491 rpcz_store.cc:276] 0709 12:14:39.255926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.301478 (+ 45552us) service_pool.cc:224] Handling call
0709 12:14:39.319669 (+ 18191us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:40.106466 (+786797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.413296 (+306830us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:40.413560 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1148997376}
I20260709 12:14:40.454710 18986 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:40.640686 19701 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:14:40.533743 19688 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:14:40.622282 12646 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [16.8-18.9]
I20260709 12:14:40.589725 12572 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:14:40.719591 12572 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:40.532992 19584 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:40.590710 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:40.666558 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:40.710040 19689 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:14:40.690222 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:40.614796 19642 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:40.615092 19565 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:40.694072 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.019s	sys 0.011s
W20260709 12:14:40.728410 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.997s	user 0.019s	sys 0.012s
I20260709 12:14:40.594790 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:40.728813 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 277525us
I20260709 12:14:40.704078 12638 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:14:40.486531 19646 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.627681 19485 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:40.602316 19591 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.066s
W20260709 12:14:40.662439 19691 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.588s	user 0.000s	sys 0.001s
I20260709 12:14:40.708767 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:40.588451 12563 rpcz_store.cc:276] 0709 12:14:39.426152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.602620 (+176468us) service_pool.cc:224] Handling call
0709 12:14:39.602981 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:39.603218 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:14:39.604694 (+  1476us) log.cc:844] Serialized 14 byte log entry
0709 12:14:39.609777 (+  5083us) log.cc:844] Serialized 14 byte log entry
0709 12:14:40.441140 (+831363us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:40.441223 (+    83us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:40.441304 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.441408 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:40.441619 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4579,"prepare.queue_time_us":21250,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":94,"raft.queue_time_us":41747,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":5192,"thread_start_us":827026,"threads_started":1}
I20260709 12:14:40.430364 19628 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.636889 19647 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:40.666344 19578 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:40.722940 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 132357us
I20260709 12:14:40.689096 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.722640 12391 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 9 to 12, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 12 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:14:40.724126 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 950) took 2029 ms. Trace:
W20260709 12:14:40.712008 19588 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.125s	user 0.009s	sys 0.008s
I20260709 12:14:40.452626 19697 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.162s
I20260709 12:14:40.878263 19709 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:40.729275 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.01358e+06us
I20260709 12:14:40.884903 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:40.731830 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3168) took 1723 ms. Trace:
I20260709 12:14:40.711966 19695 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.740989 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:40.788298 19686 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:40.790205 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81542.4us
I20260709 12:14:40.734452 19584 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:40.754693 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 138363us
W20260709 12:14:40.830282 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:40.741691 19533 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:14:40.837782 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.911443 12638 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:40.788666 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:40.847903 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:40.745476 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:40.744096 19626 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:40.744702 19701 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:40.662178 19632 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.716594 19168 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.737972 19565 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.879000 12644 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:40.728595 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:40.801713 19647 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.839756 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:40.735381 19642 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:40.953018 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.890331 12489 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:40.880079 13254 rpcz_store.cc:276] 0709 12:14:38.695120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.695244 (+   124us) service_pool.cc:224] Handling call
0709 12:14:38.709454 (+ 14210us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:38.742398 (+ 32944us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:38.742445 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:38.751014 (+  8569us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.140795 (+389781us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b18004334e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:39.533049 (+392254us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:39.533109 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.533193 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:40.720328 (+1187135us) raft_consensus.cc:1685] finished
0709 12:14:40.720358 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:40.724041 (+  3683us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8826,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":7724,"spinlock_wait_cycles":60847488,"thread_start_us":391471,"threads_started":1,"wal-append.queue_time_us":978689}
I20260709 12:14:40.954533 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.926730 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:14:40.910763 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:40.792923 19578 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:40.784739 19580 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:40.885893 12572 rpcz_store.cc:276] 0709 12:14:39.008711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.536782 (+528071us) service_pool.cc:224] Handling call
0709 12:14:39.537132 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:40.124428 (+587296us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54002bc620. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:14:40.731482 (+607054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.731771 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364178816}
I20260709 12:14:40.957017 12490 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:40.887928 19588 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:40.928117 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:40.974045 12489 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:14:40.905961 19686 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:40.975371 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:40.981213 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:40.981055 12489 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 10
I20260709 12:14:40.920593 19701 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:40.986176 12489 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:14:40.986598 12489 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:40.982885 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:40.990118 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62099us
I20260709 12:14:40.991813 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:40.923751 18986 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:41.014452 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:41.017601 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:41.017887 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:41.019341 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:40.984915 19682 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.605s	user 0.001s	sys 0.001s
I20260709 12:14:40.959393 19709 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:41.025892 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:40.985069 19686 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:40.977322 19626 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:41.029394 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:40.977290 19699 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.020350 19568 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:40.899547 19485 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:41.023536 19690 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.617s	user 0.000s	sys 0.002s
I20260709 12:14:40.993752 19644 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.014968 19664 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:40.993369 19701 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:41.021914 19580 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:41.022452 18986 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.042287 12489 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:14:41.038442 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 293029us
I20260709 12:14:40.989588 19588 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.030851 19245 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:14:41.031575 19649 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:41.052678 19717 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:14:39.489579 12390 rpcz_store.cc:269] 0709 12:13:12.028961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:12.029228 (+   267us) service_pool.cc:224] Handling call
0709 12:13:12.411281 (+382053us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b5400290090. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cf0b2 00007ffbfb648759 00007ffbfb5e0a76 00007ffbfb5b1d58 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc
0709 12:14:39.488778 (+87077497us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:14:27.782697 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:14:27.923048 (+140351us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:14:27.923106 (+    58us) write_op.cc:435] Acquired schema lock
0709 12:14:27.923127 (+    21us) tablet.cc:662] Decoding operations
0709 12:14:27.924006 (+   879us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:14:27.924095 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:14:27.924123 (+    28us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:14:27.924875 (+   752us) tablet.cc:701] Row locks acquired
0709 12:14:27.924927 (+    52us) write_op.cc:260] PREPARE: finished
0709 12:14:27.925146 (+   219us) write_op.cc:270] Start()
0709 12:14:27.925354 (+   208us) write_op.cc:276] Timestamp: P: 1783599267925064 usec, L: 0
0709 12:14:27.925380 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:14:28.002166 (+ 76786us) log.cc:844] Serialized 2094 byte log entry
0709 12:14:28.695701 (+693535us) op_driver.cc:464] REPLICATION: finished
0709 12:14:29.027585 (+331884us) write_op.cc:301] APPLY: starting
0709 12:14:29.028088 (+   503us) tablet.cc:1367] starting BulkCheckPresence
0709 12:14:29.422705 (+394617us) tablet.cc:1370] finished BulkCheckPresence
0709 12:14:29.422892 (+   187us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:14:29.450728 (+ 27836us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:14:29.451091 (+   363us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 12:14:29.451139 (+    48us) write_op.cc:312] APPLY: finished
0709 12:14:29.451475 (+   336us) log.cc:844] Serialized 57 byte log entry
0709 12:14:29.451710 (+   235us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:29.467040 (+ 15330us) write_op.cc:454] Released schema lock
0709 12:14:29.540374 (+ 73334us) write_op.cc:341] FINISH: Updating metrics
0709 12:14:29.885208 (+344834us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:29.899654 (+ 14446us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":87270,"reported_tablets":51,"spinlock_wait_cycles":11435264,"child_traces":[["op",{"apply.queue_time_us":225185,"apply.run_cpu_time_us":23506,"apply.run_wall_time_us":796157,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25007,"num_ops":5,"prepare.queue_time_us":156073,"prepare.run_cpu_time_us":29841,"prepare.run_wall_time_us":423915,"raft.queue_time_us":295210,"raft.run_cpu_time_us":29760,"raft.run_wall_time_us":244896,"replication_time_us":732229,"spinlock_wait_cycles":4352,"thread_start_us":558435,"threads_started":4,"wal-append.queue_time_us":200853,"wal-append.run_cpu_time_us":42707,"wal-append.run_wall_time_us":2207371}]]}
I20260709 12:14:41.056252 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1198) took 2238 ms. Trace:
W20260709 12:14:41.056537 19528 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.370s	user 0.010s	sys 0.003s
I20260709 12:14:40.993988 19710 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:41.041157 19568 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:41.037513 19633 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.040737 19534 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:41.062252 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156765us
I20260709 12:14:41.057097 19546 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:41.116043 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:41.042433 19712 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:41.041426 19226 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:41.055269 19704 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:14:41.089846 19694 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.430s	user 0.000s	sys 0.002s
I20260709 12:14:41.126093 19718 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:41.127323 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3153) took 2136 ms. Trace:
I20260709 12:14:41.127789 12571 rpcz_store.cc:276] 0709 12:14:38.990728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.399780 (+409052us) service_pool.cc:224] Handling call
0709 12:14:39.400199 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:39.415476 (+ 15277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.415533 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:39.421672 (+  6139us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.726536 (+304864us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:39.726581 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.726653 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:41.127081 (+1400428us) raft_consensus.cc:1685] finished
0709 12:14:41.127103 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.127259 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2352,"raft.queue_time_us":5824,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2436,"thread_start_us":242587,"threads_started":1,"wal-append.queue_time_us":943477}
I20260709 12:14:41.093446 12567 rpcz_store.cc:276] 0709 12:14:38.817532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.344124 (+526592us) service_pool.cc:224] Handling call
0709 12:14:39.344485 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:39.344868 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.344898 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:39.364128 (+ 19230us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.713883 (+349755us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:39.713928 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.714001 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:41.056002 (+1342001us) raft_consensus.cc:1685] finished
0709 12:14:41.056032 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.056179 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2960,"raft.queue_time_us":9022,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":42266,"thread_start_us":349300,"threads_started":1,"wal-append.queue_time_us":625003}
I20260709 12:14:41.050668 19658 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.120755 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:41.112917 19705 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.370s	user 0.000s	sys 0.002s
W20260709 12:14:41.119650 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 229415us
I20260709 12:14:41.127015 19718 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.137326 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.041930 19709 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:41.148288 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 225863us
I20260709 12:14:41.115077 19568 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:41.139489 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.040998 19699 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.118041 19696 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.180493 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2633) took 1100 ms. Trace:
I20260709 12:14:41.188360 12646 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:41.179275 19628 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.118896 19546 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:41.127609 19578 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.123157 19715 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.128238 19633 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:41.193392 12639 rpcz_store.cc:276] 0709 12:14:40.080183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:40.602995 (+522812us) service_pool.cc:224] Handling call
0709 12:14:40.618885 (+ 15890us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:40.619252 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:40.619281 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:40.619775 (+   494us) log.cc:844] Serialized 26 byte log entry
0709 12:14:40.733684 (+113909us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:40.736737 (+  3053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.736891 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:41.180227 (+443336us) raft_consensus.cc:1685] finished
0709 12:14:41.180250 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.180420 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5988,"raft.queue_time_us":1458,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":11058,"thread_start_us":113226,"threads_started":1,"wal-append.queue_time_us":270715}
I20260709 12:14:41.189705 19710 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:41.209331 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72112.8us
I20260709 12:14:41.210724 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:41.213073 19664 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:41.213097 19658 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:41.213889 19664 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:41.214526 19664 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:41.215042 19658 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:41.215598 19658 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:41.218133 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:41.216854 19658 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.220134 19658 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:41.220769 19644 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:41.222044 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:41.225867 19534 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.226694 19534 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:41.227291 19534 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.231949 19588 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:41.240437 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 2986) took 1870 ms. Trace:
W20260709 12:14:41.205185 19703 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.379s	user 0.001s	sys 0.000s
W20260709 12:14:41.243116 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103728us
I20260709 12:14:41.241118 12643 rpcz_store.cc:276] 0709 12:14:39.370075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.406720 (+ 36645us) service_pool.cc:224] Handling call
0709 12:14:39.422583 (+ 15863us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:39.426392 (+  3809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.426428 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:39.427040 (+   612us) log.cc:844] Serialized 52 byte log entry
0709 12:14:39.660893 (+233853us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:39.660980 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:39.661082 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:41.240066 (+1578984us) raft_consensus.cc:1685] finished
0709 12:14:41.240102 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.240341 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3370,"raft.queue_time_us":4603,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":19537,"thread_start_us":233089,"threads_started":1,"wal-append.queue_time_us":557421}
I20260709 12:14:41.175448 19528 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.068s
I20260709 12:14:41.234742 19565 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.236577 19578 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:41.244647 19633 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:41.272781 19707 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.330s	user 0.001s	sys 0.001s
I20260709 12:14:41.274360 19723 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:14:41.274897 19681 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.381s	user 0.022s	sys 0.025s
I20260709 12:14:41.243520 19644 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:41.254302 19649 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:41.279106 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3337) took 1587 ms. Trace:
I20260709 12:14:41.254245 19584 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.257391 19168 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:41.289556 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 260257us
I20260709 12:14:41.256534 19647 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:41.246022 19695 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:41.294111 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93528.4us
W20260709 12:14:41.294816 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:41.258829 19697 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:41.205696 19696 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.289424 19528 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:41.280331 19534 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.296738 12495 rpcz_store.cc:276] 0709 12:14:39.691544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:40.055644 (+364100us) service_pool.cc:224] Handling call
0709 12:14:40.306789 (+251145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:40.722501 (+415712us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:40.722531 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:40.722579 (+    48us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:14:40.722612 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:41.278776 (+556164us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:41.278837 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.279028 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":955465472}
I20260709 12:14:41.289877 19642 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.240111 19715 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.296736 19724 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:41.308550 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.562s (SENT) [suppressed 9 similar messages]
I20260709 12:14:41.292445 18986 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.288280 19565 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:14:41.312846 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:41.328227 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.301376 19644 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:41.322477 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:41.331254 19644 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.336530 19734 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:41.337230 19734 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:41.337726 19734 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:41.306763 19649 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:41.325336 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:41.345175 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.311108 19528 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:41.353286 19721 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:14:41.354965 12495 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:41.355515 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2630) took 1456 ms. Trace:
I20260709 12:14:41.355633 19735 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.356585 19735 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.357422 19744 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:41.359326 19744 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:41.359489 19730 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.360338 19730 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.360267 19744 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.364732 12645 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:14:41.324703 19296 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.870s	user 0.015s	sys 0.000s
I20260709 12:14:41.368930 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3329) took 1882 ms. Trace:
I20260709 12:14:41.370352 12492 rpcz_store.cc:276] 0709 12:14:39.486211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.490888 (+  4677us) service_pool.cc:224] Handling call
0709 12:14:39.491260 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:39.493802 (+  2542us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:14:39.527885 (+ 34083us) log.cc:844] Serialized 14 byte log entry
0709 12:14:40.109515 (+581630us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:40.110020 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:14:40.110154 (+   134us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:40.110183 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.110258 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:41.368656 (+1258398us) raft_consensus.cc:1685] finished
0709 12:14:41.368682 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.368854 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53866,"prepare.queue_time_us":33912,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":38309,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":48446,"thread_start_us":426523,"threads_started":1,"wal-append.queue_time_us":831509}
I20260709 12:14:41.355918 12641 rpcz_store.cc:276] 0709 12:14:39.898987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:40.191270 (+292283us) service_pool.cc:224] Handling call
0709 12:14:40.191755 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:40.393396 (+201641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:41.355235 (+961839us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:41.355452 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74387712}
I20260709 12:14:41.310123 19697 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:41.379518 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134018us
I20260709 12:14:41.389061 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:41.347757 19729 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:41.367048 18986 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:41.376941 19296 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.403445 19741 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:41.403957 19740 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:41.405391 19736 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:41.409773 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 2979) took 2190 ms (client timeout 2040 ms). Trace:
I20260709 12:14:41.410888 19699 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.412552 19697 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.413492 19585 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.413763 19709 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.414698 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 952) took 2053 ms. Trace:
I20260709 12:14:41.456128 13258 rpcz_store.cc:276] 0709 12:14:39.361290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.362099 (+   809us) service_pool.cc:224] Handling call
0709 12:14:39.393570 (+ 31471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:39.696631 (+303061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:41.414340 (+1717709us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:41.414610 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":658568064}
I20260709 12:14:41.416018 13261 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:14:41.417137 19731 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:41.421418 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:41.424212 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101872us
W20260709 12:14:41.432240 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:41.425261 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 355086us
W20260709 12:14:41.440485 12637 rpcz_store.cc:269] 0709 12:14:39.219159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.219325 (+   166us) service_pool.cc:224] Handling call
0709 12:14:41.409479 (+2190154us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:41.409664 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2102246784}
I20260709 12:14:41.440954 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.442345 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.401235 19626 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:41.394301 19643 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:41.324740 19724 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.463063 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:41.460011 12641 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:41.473920 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.464550 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.464869 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.439414 19578 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.417589 19706 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:14:41.452565 19739 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:14:41.438737 19494 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:41.486012 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.493971 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:41.494258 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:41.494762 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 186245us
I20260709 12:14:41.495221 12571 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:41.486954 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:41.496430 12572 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:41.491281 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:41.493513 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:41.500300 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 245656us
I20260709 12:14:41.500558 12563 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: [15.8-15.8]
W20260709 12:14:41.500995 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:41.475512 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.492072 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.493273 12566 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:41.495122 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.479265 19697 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.467126 19626 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:41.506918 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.506927 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:41.491724 13262 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:41.449036 18986 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.463173 19743 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:41.488195 19706 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:41.483359 19730 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.394294 19633 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.547974 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.550685 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:41.543577 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84584.9us
I20260709 12:14:41.572829 12571 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:41.560990 12572 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:41.447047 19741 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.574182 12571 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 8. Preceding OpId from leader: term: 22 index: 10. (index mismatch)
I20260709 12:14:41.575732 19741 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.576646 19741 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:41.511746 19626 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:41.584204 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.585427 12565 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:41.491258 19578 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.601017 19735 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.602885 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:41.599635 19748 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.600136 19747 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:41.598925 19746 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:41.456336 19736 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.608358 12563 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:41.472936 19528 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.473475 19731 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:41.614161 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:41.449177 19740 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.622262 12563 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:41.611485 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.637180 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:41.478145 19649 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.506093 19494 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.546147 19695 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:41.562796 19591 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:41.666083 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:41.667573 13254 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:41.541828 19697 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.671391 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:41.644675 13259 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:41.706583 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:41.711462 19750 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:14:41.713121 19578 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.717522 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.721849 12492 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 12:14:41.647254 19696 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.721750 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1033) took 2730 ms. Trace:
I20260709 12:14:41.665515 19226 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.635785 19497 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.101s
I20260709 12:14:41.664382 19729 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:41.612370 19730 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.097s
I20260709 12:14:41.666002 19644 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:14:41.723278 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 216439us
W20260709 12:14:41.735301 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 228458us
I20260709 12:14:41.719619 19528 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:41.719177 19695 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:41.719866 19168 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:41.759971 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.661609 19735 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.078s
I20260709 12:14:41.664404 19632 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 12:14:41.637977 19747 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.637759 19748 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.731761 13256 rpcz_store.cc:276] 0709 12:14:38.986479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:38.986941 (+   462us) service_pool.cc:224] Handling call
0709 12:14:38.991287 (+  4346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:39.262303 (+271016us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.262337 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:39.262960 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.503980 (+241020us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:39.504045 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:41.665041 (+2160996us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:41.665097 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:41.665149 (+    52us) raft_consensus.cc:1685] finished
0709 12:14:41.665160 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:41.716460 (+ 51300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124544,"raft.queue_time_us":161108,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":9500,"spinlock_wait_cycles":254391168,"thread_start_us":240438,"threads_started":1,"wal-append.queue_time_us":440499,"wal-append.run_cpu_time_us":2048,"wal-append.run_wall_time_us":1002604}
I20260709 12:14:41.804666 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.709357 19591 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:41.806133 19644 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:41.807137 12572 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:41.820530 12572 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:41.808885 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:41.808900 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:41.805377 12571 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:41.812263 19567 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:41.813325 19745 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.151s
W20260709 12:14:41.814635 19749 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
W20260709 12:14:41.838287 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 225965us
W20260709 12:14:41.841640 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 350398us
W20260709 12:14:41.822341 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:41.828858 12571 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:41.829193 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:41.760975 19715 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.773473 19168 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:41.751398 19226 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.872308 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:41.826021 19644 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:41.820799 19296 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.755363 19696 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:14:41.825795 19567 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:41.830344 19657 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.835455 19724 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:41.833321 18986 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:41.759248 19706 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.906858 12641 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:41.887465 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:41.893587 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64499.3us
I20260709 12:14:41.911237 12571 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:41.889580 19168 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:41.914222 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:41.889205 19644 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:41.902557 19715 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:14:41.930022 19604 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.501s	user 0.014s	sys 0.002s
W20260709 12:14:41.962778 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48670.7us
I20260709 12:14:41.966701 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.967474 12567 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:41.931877 19706 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:41.912885 19764 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:14:41.960757 19723 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.000s
I20260709 12:14:41.974526 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.972333 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:41.923159 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105068us
I20260709 12:14:41.963701 19715 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:41.985153 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.986003 12567 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:41.987224 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.971035 19534 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:14:41.977267 19568 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.794s	user 0.007s	sys 0.007s
I20260709 12:14:41.904044 19657 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.914515 19695 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.988481 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:42.098670 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.099176 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.099640 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:42.100567 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.101359 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:42.102254 19761 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.220s	user 0.000s	sys 0.000s
I20260709 12:14:42.102617 19699 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:42.102849 19758 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.344s	user 0.000s	sys 0.002s
I20260709 12:14:42.103190 19699 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.103669 19699 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:42.103786 19755 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:14:41.892382 19729 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.102110 19591 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.106114 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:42.061646 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95010.2us
I20260709 12:14:42.116166 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.117725 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:42.049331 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 175125us
W20260709 12:14:41.992383 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:42.116430 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:42.113492 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.118940 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.116211 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:41.973292 19497 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.120839 12571 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:42.121722 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:41.975445 19604 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.061050 19754 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:41.984647 19715 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:42.124028 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:42.132901 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.123684 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:42.137418 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:41.979226 19736 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.126865 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.138727 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:41.978492 19723 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:42.115116 19226 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:42.151403 19741 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:42.139014 19590 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.605s	user 0.010s	sys 0.007s
I20260709 12:14:42.129796 19534 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.109990 19740 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.148911 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:42.150297 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.180899 12567 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:42.152019 12569 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:42.153393 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.181242 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:42.155409 19777 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:14:42.156304 19747 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.158861 19769 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:42.159487 19578 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:42.145718 19754 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:42.160486 19760 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:14:42.161468 19183 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:14:42.144550 19757 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.348s	user 0.001s	sys 0.000s
I20260709 12:14:42.162359 19748 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.163141 19773 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:14:42.164122 19546 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.924s	user 0.009s	sys 0.004s
I20260709 12:14:42.139861 19710 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:42.213863 19710 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.214630 19710 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:42.213832 19546 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.215160 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.164860 19581 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:42.168689 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3004) took 1384 ms. Trace:
I20260709 12:14:42.217311 12645 rpcz_store.cc:276] 0709 12:14:40.784220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:40.984944 (+200724us) service_pool.cc:224] Handling call
0709 12:14:41.364229 (+379285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:41.365349 (+  1120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.168408 (+803059us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b540018c9d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.168607 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":870397184}
I20260709 12:14:42.169705 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3343) took 1184 ms. Trace:
I20260709 12:14:42.218950 12492 rpcz_store.cc:276] 0709 12:14:40.986395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:41.382331 (+395936us) service_pool.cc:224] Handling call
0709 12:14:41.717764 (+335433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:41.722399 (+  4635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.169479 (+447080us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540034ec50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.169614 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":728249216}
I20260709 12:14:42.216800 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.220793 19583 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.220942 12570 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:42.217007 19546 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:42.222147 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.222666 19183 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:42.223557 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.223842 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3339) took 1258 ms. Trace:
I20260709 12:14:42.217331 19581 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:41.981957 19494 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.173476 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:42.146590 19626 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.007s
I20260709 12:14:42.027225 19770 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:42.154294 19768 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:42.182102 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:42.183265 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:42.164867 19698 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.431s	user 0.005s	sys 0.000s
I20260709 12:14:42.171624 19718 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.185859 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:42.178922 19686 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.137s	user 0.011s	sys 0.003s
I20260709 12:14:42.188166 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.124202 19568 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.125718 19745 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 12:14:42.250185 19745 consensus_queue.cc:1243] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:42.223860 19183 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.248791 19567 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.253857 19567 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.257825 19581 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:42.233219 19768 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:42.258592 19768 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:42.261221 12496 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:42.234139 19626 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.264248 19626 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.265825 19720 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.266798 19720 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.275344 19604 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.276018 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3331) took 2789 ms. Trace:
I20260709 12:14:42.121608 19774 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:14:42.139433 19497 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.285895 19774 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:42.286371 19774 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:42.287508 19497 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:42.201607 19687 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.509s	user 0.001s	sys 0.009s
I20260709 12:14:42.287676 19774 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 19.7, Last appended by leader: 7, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.202560 19781 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:14:42.187088 19585 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:42.289887 19774 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.290375 19585 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:42.291373 19585 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:42.292557 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:42.203357 19701 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.120s	user 0.012s	sys 0.000s
I20260709 12:14:42.293948 19701 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.205655 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.295079 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3151) took 3308 ms. Trace:
I20260709 12:14:42.296257 19701 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.295528 12568 rpcz_store.cc:276] 0709 12:14:38.987023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.363095 (+376072us) service_pool.cc:224] Handling call
0709 12:14:39.415075 (+ 51980us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:39.431629 (+ 16554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:39.431674 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:39.458936 (+ 27262us) log.cc:844] Serialized 26 byte log entry
0709 12:14:39.814513 (+355577us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:40.171140 (+356627us) log.cc:844] Serialized 14 byte log entry
0709 12:14:40.315624 (+144484us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:40.315787 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:42.164295 (+1848508us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:14:42.294769 (+130474us) raft_consensus.cc:1685] finished
0709 12:14:42.294867 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:42.295008 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":320248,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":9305,"raft.queue_time_us":299,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":122166,"spinlock_wait_cycles":2533632,"thread_start_us":244008,"threads_started":1,"wal-append.queue_time_us":1167482}
I20260709 12:14:42.297307 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:42.297497 12563 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.298669 12563 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:42.298856 19583 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:42.299432 19583 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.295425 12564 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:42.207288 19658 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.984s	user 0.012s	sys 0.000s
I20260709 12:14:42.208397 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3000) took 1425 ms. Trace:
I20260709 12:14:42.329200 12646 rpcz_store.cc:276] 0709 12:14:40.782468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:40.782610 (+   142us) service_pool.cc:224] Handling call
0709 12:14:40.984496 (+201886us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:41.206037 (+221541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.207404 (+1001367us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:42.208092 (+   688us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54002c4f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.208303 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":931676800}
W20260709 12:14:42.230706 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57452.6us
I20260709 12:14:42.224275 12488 rpcz_store.cc:276] 0709 12:14:40.965997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:40.974790 (+  8793us) service_pool.cc:224] Handling call
0709 12:14:40.975165 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:41.292174 (+317009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:41.292216 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:41.292874 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:14:41.419018 (+126144us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:41.419080 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.223386 (+804306us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:42.223453 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:42.223503 (+    50us) raft_consensus.cc:1685] finished
0709 12:14:42.223516 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:42.223781 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7037,"raft.queue_time_us":26606,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":8318,"spinlock_wait_cycles":607528704,"thread_start_us":137529,"threads_started":2,"wal-append.queue_time_us":202987}
I20260709 12:14:42.331887 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:42.333914 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:42.235327 12570 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:42.236080 12571 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:42.336088 12571 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:42.217007 19699 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.336282 19626 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
W20260709 12:14:42.242430 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56773.6us
I20260709 12:14:42.231694 19296 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:42.248720 19686 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:42.276233 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.276559 12494 rpcz_store.cc:276] 0709 12:14:39.486700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.492128 (+  5428us) service_pool.cc:224] Handling call
0709 12:14:39.492469 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:39.573493 (+ 81024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.275696 (+2702203us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b540037e454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:42.275931 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115950976}
W20260709 12:14:42.185416 19664 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.969s	user 0.015s	sys 0.000s
I20260709 12:14:42.280304 19568 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.338299 12645 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:14:42.338912 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.339691 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.337062 12571 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.369304 12571 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:42.369033 12640 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:42.284109 19644 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.351290 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.351653 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.373932 12646 raft_consensus.cc:2376] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22. Re-sending same reply.
I20260709 12:14:42.347378 12490 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:42.244032 19494 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:14:42.358925 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209258us
I20260709 12:14:42.299841 19583 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:42.377830 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:42.384037 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.383980 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:42.388104 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.391809 12645 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22.
I20260709 12:14:42.394135 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.350276 19437 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:42.362599 19734 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.024s	user 0.014s	sys 0.000s
I20260709 12:14:42.406673 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:42.399960 12492 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:42.347123 19699 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.350466 19770 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:14:42.366403 19664 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.397805 19658 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:42.381099 19604 consensus_queue.cc:1175] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 19 status { last_received { term: 14 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [16.8-18.9]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:14:42.431813 19658 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.382287 19609 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 12:14:42.439009 12646 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:42.419351 19296 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:42.438294 19658 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.435933 19712 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:42.419425 19644 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.435880 19639 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 16 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.340348 19784 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.086s
I20260709 12:14:42.442572 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.422603 19701 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:42.400902 19646 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:42.398870 19626 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:42.459425 12491 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:14:42.421851 19770 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:42.416953 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.487327 19701 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.496836 19626 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.495977 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.413618 19734 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:42.521524 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 99457.3us
I20260709 12:14:42.499874 12563 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 11. (term mismatch)
I20260709 12:14:42.523298 12563 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:14:42.523561 12563 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 22.11
I20260709 12:14:42.523811 12563 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:42.463608 19528 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.691s	user 0.009s	sys 0.004s
W20260709 12:14:42.525098 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.486450 19721 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:42.418412 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:42.512431 19168 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.001s
W20260709 12:14:42.527071 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108828us
I20260709 12:14:42.507750 19585 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:42.504011 19770 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:42.542722 19437 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:42.544956 19748 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:42.545912 19748 consensus_queue.cc:1243] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:42.547174 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.549919 19748 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:42.550419 19748 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:42.550823 19748 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:14:42.566222 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6583302459998777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
I20260709 12:14:42.569784 12640 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:42.570845 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:42.521875 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142554us
I20260709 12:14:42.573232 12491 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:42.517145 19701 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:42.577616 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.578850 19734 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.579371 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.582578 12563 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:42.582624 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:42.526106 19626 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.581545 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.587360 12567 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.572913 19746 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:42.595860 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.596652 12568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:42.525919 19710 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.609048 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:42.610829 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:42.611228 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 227337us
I20260709 12:14:42.573453 19721 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.573786 19168 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.580972 19609 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260709 12:14:42.593436 19746 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:42.596395 19686 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.615973 12494 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:42.613333 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.589597 19784 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:14:42.626641 12567 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
W20260709 12:14:42.631165 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.631989 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:14:42.632925 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.634761 12568 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16. Re-sending same reply.
I20260709 12:14:42.636153 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.637913 12488 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:42.588950 19770 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:42.647758 19793 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:14:42.633765 19296 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:42.657786 19296 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.659065 19744 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:42.638904 19585 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:42.660905 12564 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:14:42.638276 19664 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.635900 19662 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:14:42.669229 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.669375 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:42.671128 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.671784 12568 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:42.672237 12491 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:42.679397 19296 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.673727 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.680642 12564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:42.672902 19534 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:42.668548 19770 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.692551 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:42.692802 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.695266 12568 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 12:14:42.695384 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2594) took 3488 ms (client timeout 1903 ms). Trace:
I20260709 12:14:42.694610 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.697485 12567 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:42.693267 19426 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:42.681883 19712 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:14:42.700281 19712 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:42.701397 12497 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:14:42.701274 19712 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:42.696112 12496 rpcz_store.cc:269] 0709 12:14:39.207243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.207409 (+   166us) service_pool.cc:224] Handling call
0709 12:14:42.695023 (+3487614us) spinlock_profiling.cc:243] Waited 3.05 s on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.695214 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7020371200}
I20260709 12:14:42.662642 19658 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.687013 19746 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 19.7, Last appended by leader: 7, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:42.765677 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.766208 19709 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.766618 19719 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.714322 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3014) took 1270 ms. Trace:
I20260709 12:14:42.767279 19719 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.768093 12492 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:14:42.753337 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2613) took 3439 ms. Trace:
W20260709 12:14:42.715854 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.772120 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:42.772507 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:42.773257 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:42.716207 19588 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:42.718668 19710 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:42.720392 19715 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.568s	user 0.012s	sys 0.000s
I20260709 12:14:42.721067 19697 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.721652 19721 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.723265 19168 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.723767 19583 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.725262 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.728267 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:42.730298 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1203) took 2998 ms (client timeout 2074 ms). Trace:
I20260709 12:14:42.729135 19628 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:14:42.737105 18986 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:42.737653 19581 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.738682 19748 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.738970 12571 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:14:42.798676 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:42.799984 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.802860 12571 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:42.759366 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3326) took 3274 ms. Trace:
I20260709 12:14:42.760704 19720 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.766944 19709 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.769320 12497 rpcz_store.cc:276] 0709 12:14:39.314124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.314929 (+   805us) service_pool.cc:224] Handling call
0709 12:14:42.700807 (+3385878us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:42.702112 (+  1305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.753048 (+ 50936us) spinlock_profiling.cc:243] Waited 3.36 s on lock 0x7b54000f87d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.753251 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7728506112}
I20260709 12:14:42.777700 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:42.767926 12639 rpcz_store.cc:276] 0709 12:14:41.443560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:41.511513 (+ 67953us) service_pool.cc:224] Handling call
0709 12:14:41.554171 (+ 42658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:41.554522 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:41.751201 (+196679us) log.cc:844] Serialized 14 byte log entry
0709 12:14:42.713640 (+962439us) spinlock_profiling.cc:243] Waited 9.85 ms on lock 0x7b1800452740. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:14:42.713880 (+   240us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:42.713935 (+    55us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:42.713988 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.714078 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:42.714241 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":321802,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":89,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":22658432,"thread_start_us":333917,"threads_started":2,"wal-append.queue_time_us":421157}
I20260709 12:14:42.781682 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:42.808854 13256 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:42.781839 12569 rpcz_store.cc:269] 0709 12:14:39.732104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.732312 (+   208us) service_pool.cc:224] Handling call
0709 12:14:42.730036 (+2997724us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.730186 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4701075200}
W20260709 12:14:42.790762 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:42.782774 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54580.3us
I20260709 12:14:42.791114 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3181) took 1260 ms. Trace:
I20260709 12:14:42.809753 12566 rpcz_store.cc:276] 0709 12:14:41.530768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:41.549583 (+ 18815us) service_pool.cc:224] Handling call
0709 12:14:41.604133 (+ 54550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:42.450094 (+845961us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:42.450132 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:42.450178 (+    46us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:42.450215 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.790785 (+340570us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:42.790837 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:42.791038 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1944864128}
W20260709 12:14:42.797118 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 913) took 3145 ms (client timeout 3237 ms). Trace:
I20260709 12:14:42.810566 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:42.811285 12644 rpcz_store.cc:269] 0709 12:14:39.651921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.652099 (+   178us) service_pool.cc:224] Handling call
0709 12:14:42.796902 (+3144803us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:42.797061 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2787449728}
I20260709 12:14:42.739472 19609 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.803963 12493 rpcz_store.cc:276] 0709 12:14:39.485318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:39.491689 (+  6371us) service_pool.cc:224] Handling call
0709 12:14:39.492009 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:40.443426 (+951417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:42.758981 (+2315555us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:42.759272 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1682285824}
I20260709 12:14:42.812069 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:42.798653 19534 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.806087 19581 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.687824 19768 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.817564 19609 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.819593 19534 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.818018 19628 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:42.822044 19658 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.822782 19719 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.822692 19628 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:42.818367 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:42.824785 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.824383 19628 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.813813 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.825953 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.826247 12491 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:42.819048 19748 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.827178 19626 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.827823 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.828899 12566 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:42.829234 19626 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:42.814229 19725 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:14:42.815846 12569 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:42.792382 18986 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:42.831283 12569 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:14:42.831686 12569 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:14:42.832072 12569 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:42.831369 18986 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.781234 19686 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:14:42.813475 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:42.834260 13261 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:42.834343 19686 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:14:42.835002 19686 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:42.836091 19686 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:42.837391 19686 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:42.837705 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:42.818665 12570 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:42.839222 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:42.824968 19701 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.841266 19701 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.842013 12495 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:42.843647 19720 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:14:42.782261 19715 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.845127 12490 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:42.799608 19168 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:42.846825 19168 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:42.847679 19715 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.847771 19168 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:42.838850 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:42.852555 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.854591 12566 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 12:14:42.855196 12566 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:14:42.856866 12566 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:14:42.857278 12566 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:42.855626 12496 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:14:42.856585 12493 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:14:42.861445 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:42.862247 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50294.2us
I20260709 12:14:42.863350 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:42.863853 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:42.821368 19609 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:42.866977 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:42.867736 12493 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:42.817365 19709 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.865216 19528 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:42.882205 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.898619 12644 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:42.881582 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:42.882987 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.821307 19768 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.834367 19754 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.875077 19658 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:42.887156 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.908453 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:42.909662 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:14:42.833374 18986 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.888384 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.900326 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.887425 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.913424 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.903292 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.903292 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:42.906713 12641 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:42.911954 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.915409 12642 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:42.889227 19456 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:42.913563 12644 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.901021 12639 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:42.913672 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:42.919502 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.921289 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:42.921682 12637 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:42.922458 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.921208 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.906826 19588 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:42.924208 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.925156 12497 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:14:42.882387 19626 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:42.897569 19583 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.887643 19609 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:42.933387 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.896099 19528 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:42.910104 19719 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:42.884109 19534 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:42.960843 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:42.962878 12491 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:42.963872 12491 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:42.937039 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:42.938761 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:42.939284 12641 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:14:42.968111 12494 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:42.968870 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.969686 12641 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:42.944641 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.973547 12640 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:42.951713 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.978139 12646 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:14:42.979576 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.956836 12488 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:42.982569 12488 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:14:42.984341 12488 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:42.986222 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.987834 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.989135 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:42.991724 19245 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:42.993714 19245 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:42.994287 19245 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:42.995455 19245 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.006368 19245 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:43.012760 19651 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.013964 19651 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.033982 12488 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:43.017315 19651 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:42.897760 19744 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:14:43.046646 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:43.047914 12637 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:43.048755 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.920043 19456 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.048900 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:43.070181 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.071388 12643 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:42.924538 19658 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:42.909873 19642 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:14:43.066601 12494 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:42.930364 19701 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:42.935642 19695 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:42.936117 19710 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:43.065801 12646 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:43.094082 12641 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:14:43.097213 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:43.105835 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:43.107218 12566 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:42.956669 19528 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:42.928755 19648 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.149s
I20260709 12:14:43.111346 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.041685 19719 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:43.066071 19664 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:43.126444 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:42.941009 19735 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.164s
I20260709 12:14:43.075876 19585 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.141603 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:43.050189 19781 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.083s
I20260709 12:14:43.137753 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.041244 19609 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.129148 19719 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:43.126698 19664 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:43.178000 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:43.178861 12566 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:43.133381 12639 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:43.121537 19644 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:43.117949 19710 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.183174 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.113081 19651 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:14:43.188302 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.189981 13261 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:43.190237 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:43.191691 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.201959 13259 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:43.144920 19568 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:43.197559 12646 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:43.199975 12637 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:43.206305 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.017s	sys 0.019s
W20260709 12:14:43.208578 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.017s	sys 0.019s
I20260709 12:14:43.210660 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.213189 13258 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:43.211869 13256 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:43.217680 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:43.180976 19664 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:43.223687 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:43.225323 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:43.228601 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:43.229274 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 460416us
I20260709 12:14:43.199209 19642 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:14:43.242532 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:43.204244 19774 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.250368 13254 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:43.147626 19585 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.259209 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.258219 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.263092 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:43.264009 12637 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:43.205755 19568 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.130349 19456 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.276772 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:43.277000 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.281148 13259 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:43.249339 19744 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:43.282822 19744 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:43.283241 19744 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:43.283682 19533 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:43.285584 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.286242 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:43.286840 13253 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:43.286972 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.288012 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:43.289026 13256 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:43.289376 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:43.290155 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:43.292408 13262 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:43.293947 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:43.294837 12493 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:43.295881 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:43.303112 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.304107 13261 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:43.307096 19651 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:43.308748 19657 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:43.310314 19712 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:43.284077 19584 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:14:43.310981 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3341) took 2335 ms (client timeout 1505 ms). Trace:
W20260709 12:14:43.311606 12489 rpcz_store.cc:269] 0709 12:14:40.975331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:41.017534 (+ 42203us) service_pool.cc:224] Handling call
0709 12:14:43.310735 (+2293201us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:43.310885 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52852096}
W20260709 12:14:43.312354 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:43.278815 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.308571 19651 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:43.203815 19644 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.319649 12492 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:43.247740 19642 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.341271 12637 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:43.342633 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:43.326617 19567 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.019s	sys 0.022s
I20260709 12:14:43.203243 19658 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.344285 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:43.281809 12646 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:43.311952 19748 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:43.356204 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:43.329944 19657 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:43.356961 12566 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:43.369901 19657 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:43.370489 19657 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:43.355165 19735 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:43.378997 19735 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.380123 19735 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.324592 19712 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.382658 19712 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.384258 19712 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:43.384698 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:43.385972 19686 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.002s
I20260709 12:14:43.377418 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
W20260709 12:14:43.345567 19567 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.020s	sys 0.022s
I20260709 12:14:43.390771 12566 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:14:43.399376 19735 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:43.324332 19584 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:43.270112 19628 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.083s
I20260709 12:14:43.408711 13253 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:43.413250 13258 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:14:43.269716 19774 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.419389 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.420178 12646 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:43.347659 19456 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:43.279019 19534 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:14:43.413287 19745 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.459879 12564 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:14:43.361706 19748 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:43.437431 19800 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.214s	user 0.002s	sys 0.000s
I20260709 12:14:43.455288 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:43.464334 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:43.497524 12642 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.581s	user 0.012s	sys 0.001s
I20260709 12:14:43.512256 12642 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
W20260709 12:14:43.508533 12644 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.588s	user 0.014s	sys 0.003s
I20260709 12:14:43.738880 12644 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:43.524984 19710 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:43.525846 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:43.529687 19731 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:43.533591 19656 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:43.534188 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:43.748284 12646 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:43.539427 19584 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.547422 19604 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.548795 19644 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:43.552556 19609 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:43.470295 19745 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.555584 19645 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:43.555982 19585 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:43.562613 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3357) took 1108 ms. Trace:
I20260709 12:14:43.757769 12495 rpcz_store.cc:276] 0709 12:14:42.454568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:42.454772 (+   204us) service_pool.cc:224] Handling call
0709 12:14:43.562236 (+1107464us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b540034e754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:43.562502 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":737567616}
I20260709 12:14:43.516507 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.467096 19748 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:14:43.578945 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:43.585937 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3356) took 1154 ms. Trace:
I20260709 12:14:43.762221 12490 rpcz_store.cc:276] 0709 12:14:42.434536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:42.434905 (+   369us) service_pool.cc:224] Handling call
0709 12:14:42.454014 (+ 19109us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:42.845595 (+391581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:43.585442 (+739847us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:43.585824 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898861440}
I20260709 12:14:43.590847 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1205) took 1925 ms. Trace:
I20260709 12:14:43.586903 19600 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:43.654063 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 369088us
I20260709 12:14:43.448665 19642 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 22.10, Last appended by leader: 10, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.764283 19600 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.765180 19642 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:14:43.766287 19808 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:14:43.766923 19600 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:43.767596 19600 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:43.768121 19600 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.676329 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.689073 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:43.770287 19745 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:43.770637 12637 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:43.755967 19645 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.738211 19774 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:43.772665 19645 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.674808 19534 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.700918 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3365) took 1080 ms. Trace:
I20260709 12:14:43.774024 19534 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:43.775585 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:43.776674 19804 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.333s	user 0.000s	sys 0.001s
I20260709 12:14:43.777191 19534 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:43.717865 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:43.786408 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68692.7us
I20260709 12:14:43.788375 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:43.719234 19825 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:43.791312 12569 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
W20260709 12:14:43.721030 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:43.731879 19245 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.725s	user 0.011s	sys 0.002s
W20260709 12:14:43.722522 18986 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.801s	user 0.012s	sys 0.004s
W20260709 12:14:43.734431 19812 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:14:43.740643 12643 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
W20260709 12:14:43.738767 19695 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.628s	user 0.010s	sys 0.005s
I20260709 12:14:43.773927 12497 rpcz_store.cc:276] 0709 12:14:42.620857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:42.929896 (+309039us) service_pool.cc:224] Handling call
0709 12:14:42.930651 (+   755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:42.930935 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:14:43.311064 (+380129us) log.cc:844] Serialized 14 byte log entry
0709 12:14:43.700424 (+389360us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:43.700498 (+    74us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:14:43.700581 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:43.700671 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:43.700843 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7414,"prepare.queue_time_us":746108,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"raft.queue_time_us":165,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":9297,"thread_start_us":768623,"threads_started":2}
I20260709 12:14:43.712312 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3195) took 1082 ms. Trace:
I20260709 12:14:43.805999 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3205) took 1138 ms. Trace:
I20260709 12:14:43.806597 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.807050 19731 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:14:43.770334 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 94133.3us
I20260709 12:14:43.808468 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.808693 19731 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:43.810647 19658 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:43.763731 12565 rpcz_store.cc:276] 0709 12:14:41.665348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:41.665519 (+   171us) service_pool.cc:224] Handling call
0709 12:14:41.719185 (+ 53666us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:41.872399 (+153214us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:14:41.980471 (+108072us) log.cc:844] Serialized 14 byte log entry
0709 12:14:42.125952 (+145481us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:42.126008 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:14:42.126075 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:43.590466 (+1464391us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:43.590529 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:43.590738 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155301,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":4095,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":6233,"spinlock_wait_cycles":351529856,"thread_start_us":240326,"threads_started":2,"wal-append.queue_time_us":186434,"wal-append.run_cpu_time_us":1413,"wal-append.run_wall_time_us":1006379}
I20260709 12:14:43.779709 19533 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:43.806613 12570 rpcz_store.cc:276] 0709 12:14:42.667558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:42.845008 (+177450us) service_pool.cc:224] Handling call
0709 12:14:42.856156 (+ 11148us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:42.856761 (+   605us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:42.856811 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:42.857571 (+   760us) log.cc:844] Serialized 26 byte log entry
0709 12:14:43.253577 (+396006us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:43.253650 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:43.253725 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:43.805717 (+551992us) raft_consensus.cc:1685] finished
0709 12:14:43.805745 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:43.805919 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5262,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9289,"thread_start_us":338097,"threads_started":1,"wal-append.queue_time_us":620572}
I20260709 12:14:43.806603 19710 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:43.792877 19644 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:43.809958 19583 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:43.813588 19583 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:43.754331 19604 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.814591 19583 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.816870 19604 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:43.781025 12563 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:43.795537 19585 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:14:43.811012 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:43.773342 19819 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:43.789685 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:43.822844 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:43.822642 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:43.779362 19664 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.525s	user 0.003s	sys 0.008s
I20260709 12:14:43.774178 19645 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:43.747836 19656 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:43.806021 19695 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:43.806435 12571 rpcz_store.cc:276] 0709 12:14:42.629552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:42.804389 (+174837us) service_pool.cc:224] Handling call
0709 12:14:42.804917 (+   528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:42.805471 (+   554us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:43.318751 (+513280us) log.cc:844] Serialized 14 byte log entry
0709 12:14:43.711830 (+393079us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:43.711916 (+    86us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:43.711983 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:43.712073 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:43.712231 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7112,"prepare.queue_time_us":715334,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"raft.queue_time_us":42934,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":16821,"thread_start_us":827304,"threads_started":2}
I20260709 12:14:43.811985 19533 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.716850 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:43.785454 12637 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
W20260709 12:14:43.796885 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 280526us
W20260709 12:14:43.857556 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:43.823918 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.830010 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [20.9-22.10]
I20260709 12:14:43.830662 12565 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:14:43.808668 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.829380 12563 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:14:43.860188 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:43.802039 19609 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:43.754195 19719 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.567s	user 0.005s	sys 0.010s
I20260709 12:14:43.813496 19644 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:43.865947 12491 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:43.810690 19731 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:43.837906 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:43.862880 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:43.867630 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59047.6us
I20260709 12:14:43.881440 12497 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:43.834884 19709 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:43.834345 18986 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:14:43.885056 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:43.885967 12569 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:14:43.887481 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:43.894327 12565 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:43.895190 12497 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:43.896973 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.866418 19534 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.900982 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:43.897838 13259 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:43.900130 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.905221 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:43.880671 19567 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:43.866811 19585 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:43.879294 19644 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:43.867034 19626 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:43.835752 19664 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.909405 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:43.914404 13253 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:43.926882 13253 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:43.866946 19646 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:43.908329 19653 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:43.882025 19642 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
W20260709 12:14:43.904945 19657 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.534s	user 0.014s	sys 0.000s
I20260709 12:14:43.941534 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:43.945674 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:43.929848 12572 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.907306 19609 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:43.907250 18986 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:43.907020 19709 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:43.957581 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48239.1us
I20260709 12:14:43.910806 19600 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:43.953401 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188868us
I20260709 12:14:43.961417 12572 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:43.962869 12565 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:43.970083 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:43.922523 19827 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:43.971138 12564 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:14:43.887156 19719 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:43.909530 19648 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:14:43.927522 19585 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:43.936537 13253 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:43.966671 19744 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.683s	user 0.011s	sys 0.005s
I20260709 12:14:43.935531 19626 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:43.981022 13253 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:43.911830 19534 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.953186 19609 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:43.992339 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:43.967720 19528 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.839s	user 0.007s	sys 0.009s
I20260709 12:14:43.967067 19686 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:43.996505 12496 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:43.908603 19656 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.994875 19648 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 23.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:14:44.006503 19656 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:43.981952 19245 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:43.956903 19657 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:43.985380 19730 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:44.003803 19823 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:14:43.951331 19646 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.059669 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:44.049607 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57403.3us
I20260709 12:14:44.050740 12572 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:14:44.061754 12572 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:14:44.051052 13253 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:14:44.010308 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3368) took 1105 ms. Trace:
I20260709 12:14:44.054778 19581 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:43.995287 19609 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:44.062166 12572 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:14:44.068905 12572 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:44.067065 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 118448us
I20260709 12:14:43.974777 19818 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:14:44.062466 12488 rpcz_store.cc:276] 0709 12:14:42.904570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:43.035587 (+131017us) service_pool.cc:224] Handling call
0709 12:14:43.036080 (+   493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:43.068453 (+ 32373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:43.068487 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:43.536112 (+467625us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b18003cef80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:14:43.537332 (+  1220us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:43.596660 (+ 59328us) log.cc:844] Serialized 26 byte log entry
0709 12:14:43.899652 (+302992us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:43.899706 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:43.899782 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:44.009988 (+110206us) raft_consensus.cc:1685] finished
0709 12:14:44.010015 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:44.010229 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31762,"raft.queue_time_us":31889,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":28011,"spinlock_wait_cycles":15831680,"thread_start_us":220972,"threads_started":1,"wal-append.queue_time_us":239245}
I20260709 12:14:43.967315 19731 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.075635 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:44.058115 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 98919us
I20260709 12:14:44.062139 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:44.078369 19829 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:14:44.078737 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3133) took 2272 ms. Trace:
I20260709 12:14:44.079264 12572 rpcz_store.cc:276] 0709 12:14:41.806453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:41.806662 (+   209us) service_pool.cc:224] Handling call
0709 12:14:41.806981 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:43.920954 (+2113973us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b4c001e3380. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3a96644 00007ffbf3abdf2d 00007ffbf3abdb93 00007ffbf57ee18b 00007ffbf57eecb5 00007ffbf57e9f30 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416
0709 12:14:44.078466 (+157512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:44.078672 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12100352}
W20260709 12:14:44.080022 19723 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
W20260709 12:14:44.085870 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 166686us
W20260709 12:14:44.091390 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.092003 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.091022 19830 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.085939 19819 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.099516 19581 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:43.989672 19826 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.089s
I20260709 12:14:44.100313 19581 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:44.100735 19646 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.108171 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:43.982229 19827 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.059068 19831 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:44.002172 19528 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.107996 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:43.994274 19744 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.122560 12637 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
W20260709 12:14:44.102496 13218 connection.cc:256] RPC call timeout handler was delayed by -1.8896023380000315s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32370000000000004s for a response
I20260709 12:14:44.127161 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:44.144912 12639 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:14:44.100813 19581 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:44.155436 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3077453889998196s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
W20260709 12:14:44.155925 13218 connection.cc:256] RPC call timeout handler was delayed by -1.246545388999948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
I20260709 12:14:44.149331 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:44.066231 19730 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.127985 19831 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.144721 19626 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.152428 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:44.153838 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91850.4us
I20260709 12:14:44.156625 12638 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:44.134905 12643 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:14:44.158617 12493 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:44.162407 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.105216 19825 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:44.102291 19837 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:44.168800 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 60880.5us
I20260709 12:14:44.171218 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:44.172050 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:44.123059 19646 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:14:44.176213 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:44.171373 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56642.9us
I20260709 12:14:44.124054 19533 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.127638 19840 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:44.177851 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69737.8us
I20260709 12:14:44.099610 19830 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.183025 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:44.188496 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:44.190243 13218 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22.
I20260709 12:14:44.086921 18986 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.191294 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.191942 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:44.193423 19725 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:44.193753 19744 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.194020 19657 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.194669 19834 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:44.173583 19656 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:44.100266 19819 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:44.201870 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.092237 19585 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.205083 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.160699 19686 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:14:44.120355 19609 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.091696 19709 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.219668 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.202868 19533 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:44.238411 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76077.4us
I20260709 12:14:44.239814 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.205107 19664 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.235952 12643 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
W20260709 12:14:44.237366 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.177997 19837 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.207608 19735 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:44.246748 19735 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:44.247236 19735 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:44.251180 19830 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.230671 19835 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:14:44.207573 19646 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:44.267735 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.237673 19656 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:44.211082 19642 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:14:44.208917 19725 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.245517 19834 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:14:44.263386 19840 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:44.242057 19657 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:44.275542 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.258586 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.234745 19841 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:44.272437 19534 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:44.292336 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46394.5us
W20260709 12:14:44.292850 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87887us
I20260709 12:14:44.295980 12638 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
W20260709 12:14:44.298072 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.291527 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.319840 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:14:44.297238 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63328.3us
I20260709 12:14:44.287277 19841 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:44.295614 19533 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:44.323647 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 121392us
W20260709 12:14:44.302196 19748 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.543s	user 0.014s	sys 0.000s
I20260709 12:14:44.287253 19581 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.304558 19609 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:44.301966 19837 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:44.299322 19827 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.301144 19725 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.355234 12643 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:44.336308 19748 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.361567 19581 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:44.292574 19657 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.362433 19748 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.364881 19657 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.366199 19719 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.298528 19646 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.325162 19845 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:44.337968 19826 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:14:44.375473 19826 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:44.375912 19826 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:44.375674 19644 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.326601 19528 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.330580 19841 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:44.376389 19730 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.330324 19831 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.384073 19819 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.336632 19656 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.334312 19840 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:44.379733 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3376) took 1103 ms. Trace:
I20260709 12:14:44.385195 19644 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.328963 19836 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:14:44.385864 19528 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.381805 19847 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 12:14:44.385452 12489 rpcz_store.cc:276] 0709 12:14:43.276095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:43.326013 (+ 49918us) service_pool.cc:224] Handling call
0709 12:14:43.326302 (+   289us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:43.329654 (+  3352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:43.329692 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:44.008356 (+678664us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b18003cf100. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:14:44.009306 (+   950us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:44.051550 (+ 42244us) log.cc:844] Serialized 26 byte log entry
0709 12:14:44.270780 (+219230us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:44.270837 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:44.270962 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:44.379472 (+108510us) raft_consensus.cc:1685] finished
0709 12:14:44.379496 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:44.379659 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2763,"raft.queue_time_us":4451,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":6136,"spinlock_wait_cycles":12439680,"thread_start_us":168551,"threads_started":1,"wal-append.queue_time_us":244637}
I20260709 12:14:44.385278 19840 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:44.389178 19840 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:44.386464 19528 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.391618 19528 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.376930 19826 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.357573 12638 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:14:44.405983 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45957.7us
W20260709 12:14:44.411811 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53812us
I20260709 12:14:44.413722 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.336313 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.415726 12638 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:44.420010 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:44.420388 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:44.420723 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87487.4us
I20260709 12:14:44.415799 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:44.422006 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:44.423955 12642 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:44.425177 12646 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:14:44.425945 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:44.426585 12497 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:44.427390 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.430037 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:44.431041 12489 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:14:44.434525 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.435446 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:44.436259 12642 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:44.436365 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.438352 12639 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:14:44.438905 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:44.454156 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.443020 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:44.451635 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:44.421985 19827 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:14:44.457022 19831 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:44.422995 19819 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:44.423627 19826 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:44.445055 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:44.471171 12645 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:44.474896 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:44.437563 19721 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:44.484609 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:44.489534 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.489179 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:44.488613 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:44.497058 12497 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:14:44.497246 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67269.1us
I20260709 12:14:44.494392 19644 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:14:44.488385 19585 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
W20260709 12:14:44.483906 19846 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:14:44.484320 19730 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:44.503386 19730 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:44.503901 19730 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:44.505672 19731 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:44.475214 19832 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:44.506459 19731 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:44.506954 19832 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:44.507620 19731 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.507984 19832 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.491811 19831 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:44.479317 19709 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:44.510679 19744 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:44.510596 19831 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.494639 19725 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:44.516189 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.517856 19657 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:44.519102 19657 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:44.519740 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:44.520787 12645 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 8.
I20260709 12:14:44.525190 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:44.461469 19686 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:14:44.542807 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.544883 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.494580 19664 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:44.546627 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:44.547483 12641 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:44.510612 19709 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:44.556375 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.556732 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.557524 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:44.521687 19819 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:44.557740 12640 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:44.548343 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.564458 12565 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:44.568569 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.548502 19664 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:44.626825 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:44.627293 19664 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:44.582463 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:44.629179 12567 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:44.565213 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.572294 12569 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:14:44.631683 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:44.579301 12571 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:14:44.490777 19719 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:14:44.639832 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:44.563745 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.646039 19639 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:44.646715 19639 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.648957 19639 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:44.650381 19852 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:14:44.651571 19827 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:14:44.653126 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3370) took 1748 ms. Trace:
I20260709 12:14:44.553465 19644 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:44.636423 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:44.638525 12568 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:14:44.603987 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:44.656636 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.559708 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.632817 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.668665 12640 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:14:44.643551 12565 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:44.653661 12496 rpcz_store.cc:276] 0709 12:14:42.905194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:43.298942 (+393748us) service_pool.cc:224] Handling call
0709 12:14:43.299377 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:43.997109 (+697732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:44.652856 (+655747us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:44.653057 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1602497280}
I20260709 12:14:44.640702 12570 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:44.635692 19721 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:44.635463 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:14:44.660447 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117720us
I20260709 12:14:44.624706 19845 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.658805 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.673430 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:44.678925 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.682226 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.683853 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.671882 12563 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:44.553939 19836 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.112s
I20260709 12:14:44.684794 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.705406 12641 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:14:44.685662 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:44.667560 19644 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:44.668864 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:44.706732 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:44.709771 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:44.672199 19721 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:44.723215 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49877.2us
I20260709 12:14:44.718592 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:44.673485 19819 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:44.733373 12565 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:44.735118 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:44.712159 19644 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:44.709931 19859 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.749877 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:44.670603 19735 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.714612 19835 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:44.751982 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:44.755507 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:44.711444 19534 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:44.759435 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:44.728184 19858 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:44.771056 12640 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:14:44.747928 19528 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:44.775668 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:44.750277 19819 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:44.751435 19864 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.755196 19719 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:44.781493 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:44.782783 19609 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:44.777089 19861 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:44.800100 19609 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.800908 19861 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.802510 19609 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:44.803855 19868 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:44.804575 19868 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:44.767727 19721 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.805502 19868 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:44.807188 19857 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:14:44.782919 19662 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:44.808642 19662 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.760584 19834 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:44.776063 19744 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:44.766764 19534 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:44.814478 19534 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:44.804049 19568 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.818130 19568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:44.820807 19662 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:44.806092 19721 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:44.788501 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:44.789354 19819 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:44.702227 19845 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.785099 12641 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:14:44.728520 19709 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.756070 19859 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.833362 19836 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:14:44.889742 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 121245us
I20260709 12:14:44.798254 19864 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.826602 19664 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.849278 19861 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:44.852185 19748 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:44.832769 19609 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:14:44.883410 19842 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:44.910892 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:44.900574 19866 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:14:44.925993 19864 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:44.926431 19861 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:44.927029 19533 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.589s	user 0.011s	sys 0.001s
I20260709 12:14:44.928007 19859 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.929826 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:44.932790 19841 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20260709 12:14:44.937844 19845 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:44.902037 19731 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:14:44.895421 19836 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:44.940130 19644 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.903519 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:44.851967 19657 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
W20260709 12:14:44.956475 19863 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.121s	user 0.001s	sys 0.001s
I20260709 12:14:44.942404 19609 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:44.899806 19744 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:14:44.883602 19840 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:44.975577 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72110us
I20260709 12:14:44.906028 19748 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:44.983999 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:44.910974 19626 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:14:44.965730 19533 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:44.993019 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:44.998191 12567 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:44.993520 12565 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:14:44.910501 19842 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.004531 13262 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 15. Current term is 16. Ops: []
I20260709 12:14:45.001156 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:44.976877 19609 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:45.011314 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.006786 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.012887 12563 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:14:44.965446 19861 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.009433 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.011632 13262 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:14:45.015686 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.016773 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.017499 12567 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:45.018965 12563 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:44.970880 19841 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.033674 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:44.998327 19735 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.035739 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.038050 12563 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:45.040993 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.041754 13254 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:45.046931 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:45.047645 13255 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:45.058763 19746 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:45.032269 19868 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:45.072362 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.086088 12567 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:45.089205 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.091010 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:45.093616 13255 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:14:45.070072 19730 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.562s	user 0.015s	sys 0.000s
I20260709 12:14:45.069762 19849 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.066017 19746 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:45.109252 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.069087 19868 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:45.082326 19835 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:45.130133 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.104494 19664 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.139449 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:45.144169 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:45.149394 19704 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:45.149518 19662 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:45.150204 19704 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:45.150810 19662 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:45.151211 19864 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:45.151962 19864 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.152887 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:45.115216 19868 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:45.059428 19836 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 19.7, Last appended by leader: 7, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.108425 19730 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.172845 12496 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:45.150837 19704 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:45.177383 12497 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:14:45.112990 19849 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.158321 19664 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:45.160665 19890 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:45.184402 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.167059 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.189158 12497 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:45.188292 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.099151 19609 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.121244 19835 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.197194 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:45.186825 19876 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.192s	user 0.002s	sys 0.000s
I20260709 12:14:45.209559 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.198238 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.210466 12570 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:45.197008 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.201395 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:45.212001 13259 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
W20260709 12:14:45.204123 19885 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:14:45.184347 19644 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:45.206307 19872 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260709 12:14:45.207432 19884 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:14:45.184726 19746 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.219240 19864 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.219816 19730 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.195629 19837 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:45.221680 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:45.244174 13255 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:45.194562 19664 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.187579 19840 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.245762 19837 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:45.246487 19837 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:45.248077 19746 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:45.216226 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.208165 19456 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:45.251503 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:45.252161 12641 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:45.225023 19773 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.258608 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:45.226605 19648 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:45.254204 19456 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:45.258291 19568 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:45.269026 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.196017 19890 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.274505 12568 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:45.275334 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:45.269021 19841 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.282331 12568 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
W20260709 12:14:45.278436 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 290043us
I20260709 12:14:45.258579 19840 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:45.270393 19568 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:45.268620 19773 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.293824 19644 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:45.272198 19648 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:45.293938 19656 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:45.300863 19704 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.320801 12641 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:45.281073 19847 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:45.288772 19456 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.334461 19456 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:45.334968 19835 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:45.335706 19835 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:45.337428 19847 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:45.306115 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:45.339309 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:45.342324 19820 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.457s	user 0.003s	sys 0.004s
I20260709 12:14:45.310223 19744 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:14:45.343434 19886 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:14:45.342850 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:45.305399 12568 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:14:45.348724 12569 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:45.322605 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.313434 19819 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.286036 19730 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:45.318571 19568 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:45.326493 19664 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.326185 19648 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.332767 19837 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.338598 19847 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:45.308003 19725 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.786s	user 0.005s	sys 0.009s
I20260709 12:14:45.305572 19534 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.420792 19725 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:45.422340 12563 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:45.423199 19534 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.363776 19891 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:14:45.366593 12640 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:45.324043 19849 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:45.357357 19721 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.364848 19644 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:45.344543 19744 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 12:14:45.364835 19841 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:45.319464 19840 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.361410 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:45.478539 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:45.437350 19644 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:45.488137 19644 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:45.438910 19721 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:44.638863 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:45.507531 12569 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:45.509362 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2628) took 2738 ms. Trace:
W20260709 12:14:45.510167 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.509941 12491 rpcz_store.cc:276] 0709 12:14:42.770847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:42.964355 (+193508us) service_pool.cc:224] Handling call
0709 12:14:42.964551 (+   196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:43.877941 (+913390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:45.489093 (+1611152us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:45.509269 (+ 20176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2072446208}
W20260709 12:14:45.516284 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.373281 19730 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.524289 12569 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:45.533545 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.535272 12565 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:14:45.535759 19753 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:14:45.533768 19721 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.538022 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.539002 12569 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:45.539904 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.541143 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:45.542400 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:14:45.543119 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.305s	user 0.011s	sys 0.042s
I20260709 12:14:45.544492 19604 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:14:45.544973 19604 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:45.543192 19837 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:45.551281 19604 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:45.552659 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.553006 19604 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:14:45.555133 19730 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.551865 19819 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.551527 19868 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:45.558085 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:45.576452 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 281477us
W20260709 12:14:45.565376 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.577332 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:45.580197 19883 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:45.580773 19883 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:45.581759 12637 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:45.581914 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.519743 19841 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:45.507844 19864 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:45.583420 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.590415 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.591351 12569 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:45.594981 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.472942 19534 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:45.583865 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:45.574175 19906 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.100s	user 0.009s	sys 0.000s
I20260709 12:14:45.584385 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:45.604336 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.367s	user 0.013s	sys 0.043s
W20260709 12:14:45.604862 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 2.3758e+06us
I20260709 12:14:45.614809 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.615835 12569 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:45.619046 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.620321 12569 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:45.589326 19895 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.364s	user 0.001s	sys 0.000s
I20260709 12:14:45.581966 19725 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.622952 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.622983 12490 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:45.627264 12569 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:45.629727 19725 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.602636 19864 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.630236 19890 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.622548 19534 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.624908 19744 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:45.648056 19847 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:45.648463 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.649350 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.649679 19847 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:14:45.649904 19704 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:45.650746 19704 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:45.650848 19847 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:45.651223 19704 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:45.651623 19836 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:45.651748 19645 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:45.652122 19836 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:45.652444 19645 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.653137 19836 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.653687 19731 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.654047 19900 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.654551 19836 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:45.654393 19731 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.655012 19841 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.656962 19731 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.658914 19868 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.659672 19840 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.633359 19664 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:45.630015 19584 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:45.661578 19664 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.653743 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.645758 19883 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.668785 19645 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.670610 19664 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:45.671204 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.670806 19568 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.654714 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 957) took 2435 ms. Trace:
I20260709 12:14:45.654750 19900 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.666999 19744 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.675180 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:45.676060 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.677754 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:45.678836 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:45.674182 13257 rpcz_store.cc:276] 0709 12:14:43.219709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:43.220596 (+   887us) service_pool.cc:224] Handling call
0709 12:14:43.258276 (+ 37680us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:43.274474 (+ 16198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:43.274522 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:43.282152 (+  7630us) log.cc:844] Serialized 52 byte log entry
0709 12:14:45.338391 (+2056239us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b18001e21a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:45.363225 (+ 24834us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:45.363287 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:45.371517 (+  8230us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:45.654335 (+282818us) raft_consensus.cc:1685] finished
0709 12:14:45.654359 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:45.654644 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":185,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":10111,"spinlock_wait_cycles":74731264,"thread_start_us":24558,"threads_started":1,"wal-append.queue_time_us":2151862}
I20260709 12:14:45.680718 12645 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:45.681149 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:45.682019 12497 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 16: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:45.682363 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:45.683358 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:45.651510 19847 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:45.665374 19585 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:45.679512 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:45.677970 12489 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:14:45.691454 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:45.676374 12644 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:45.686517 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.687933 12491 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:45.686373 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.636561 19725 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.674580 19534 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.715452 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.678982 19704 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:45.687011 19664 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.703697 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:45.664322 19864 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:45.709672 19568 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:45.690153 19847 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:45.687463 19744 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.719594 19840 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:45.708107 19900 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:45.703620 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:45.720623 12492 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:45.751019 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:45.751613 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.752890 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:45.720927 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179895us
I20260709 12:14:45.702425 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:45.708065 19644 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:45.726145 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.726434 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:45.741645 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 154119us
I20260709 12:14:45.749572 12644 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:45.765829 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:45.766587 12644 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:45.765602 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:45.690495 19585 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:45.769486 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:45.772908 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.775151 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:45.776048 12640 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:14:45.776420 12489 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:45.781829 12645 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:14:45.782542 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:45.836879 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.797108 12497 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:45.799559 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.739621 19664 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:45.765689 19840 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.831702 12644 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:45.787834 19534 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:14:45.846987 12494 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:14:45.838160 12567 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:45.839152 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.840117 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:45.843261 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:45.844784 12645 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:45.844954 12644 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:14:45.856611 12640 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
W20260709 12:14:45.856812 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.856194 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.862969 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.864692 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.865389 12569 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:45.875532 12569 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:45.876648 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.876914 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:45.889158 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:45.870568 19836 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:45.877177 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:45.883791 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148186us
I20260709 12:14:45.900708 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:45.902733 12570 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:45.903851 12489 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:45.905128 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.905303 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:45.906534 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.907374 12489 raft_consensus.cc:2368] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:45.909427 12565 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: [21.17-21.17]
I20260709 12:14:45.914670 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:45.919377 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:45.921001 12570 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:14:45.921343 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:45.921418 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.923285 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:45.877275 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:45.923952 12567 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:45.923952 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:45.926512 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:45.926841 12644 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:14:45.927139 12569 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:14:45.924134 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:45.928447 12644 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:45.825861 19585 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:45.935331 12536 leader_election.cc:400] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:45.908748 19648 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:45.913183 19859 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:45.919842 19900 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:45.923152 19836 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:45.911976 19723 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:45.954728 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.955585 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:45.957182 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.861340 19912 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:14:45.950728 12567 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:14:45.959909 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:45.980995 13257 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:14:45.986897 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:45.988629 13254 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:45.990458 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 941) took 2808 ms (client timeout 2982 ms). Trace:
W20260709 12:14:45.991082 12493 rpcz_store.cc:269] 0709 12:14:43.182168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:43.295548 (+113380us) service_pool.cc:224] Handling call
0709 12:14:45.990191 (+2694643us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b540020bcd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:45.990379 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1981164800}
W20260709 12:14:46.000391 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:45.955320 19585 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:46.008674 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:46.014751 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2660) took 1492 ms. Trace:
I20260709 12:14:46.015234 12638 rpcz_store.cc:276] 0709 12:14:44.522025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.008418 (+1486393us) service_pool.cc:224] Handling call
0709 12:14:46.014672 (+  6254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.017561 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:46.006783 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:46.019249 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2663) took 1496 ms. Trace:
I20260709 12:14:46.019613 12642 rpcz_store.cc:276] 0709 12:14:44.522829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.006453 (+1483624us) service_pool.cc:224] Handling call
0709 12:14:46.019187 (+ 12734us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.007726 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:45.953969 19919 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:46.020750 19919 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:46.020929 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2662) took 1498 ms (client timeout 1837 ms). Trace:
W20260709 12:14:46.021690 12643 rpcz_store.cc:269] 0709 12:14:44.522579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.007457 (+1484878us) service_pool.cc:224] Handling call
0709 12:14:46.020848 (+ 13391us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:45.951152 19648 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:46.023444 19648 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.024816 19648 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:45.948554 19910 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:45.953148 19754 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:14:46.026305 19910 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:46.027316 19754 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:46.009478 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:46.021198 19919 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:46.027338 19910 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.030282 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:45.961683 19900 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:46.031723 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:46.031855 19841 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:14:46.032212 19900 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:46.033250 19725 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:46.033660 13254 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 21 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:14:46.037942 19841 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:46.037386 19725 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
W20260709 12:14:46.041332 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117586us
I20260709 12:14:46.043085 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:46.046372 12567 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [15.8-16.9]
I20260709 12:14:45.984580 19859 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:46.041608 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:46.001114 19588 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:46.056366 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.056797 13254 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:45.989085 19723 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:46.059950 12537 leader_election.cc:400] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:46.058269 13253 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 12:14:46.053346 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:46.001605 19918 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:46.073558 13253 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 14
W20260709 12:14:46.063301 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 142091us
I20260709 12:14:46.074132 13253 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:14:46.074417 12537 leader_election.cc:403] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:14:46.074576 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:46.014638 19744 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:46.077787 13253 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:46.043437 19534 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.049347 19644 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:46.043347 19920 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:46.062187 19604 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:46.089802 19664 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:46.106205 19604 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:46.106750 19604 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:46.114092 19644 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:14:46.114948 19914 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:14:46.117024 19585 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:46.093961 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:46.118111 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:46.097000 13257 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:14:46.118276 19723 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.118971 13257 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 13
I20260709 12:14:46.119272 13257 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:14:46.119529 13257 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:46.074748 19588 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.122257 19588 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:46.084278 19911 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.265s	user 0.000s	sys 0.002s
I20260709 12:14:46.102715 19840 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:46.103168 19890 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:46.089144 19534 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.090250 19725 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:46.117511 19585 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:46.121143 19836 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.114064 19859 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.182358 19874 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:46.183070 19874 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:46.184212 19874 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:46.176200 19916 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.161s	user 0.001s	sys 0.001s
I20260709 12:14:46.180282 19847 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.203526 19847 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:46.208215 19874 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:46.213611 19842 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:46.215556 19919 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:46.219909 19907 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.635s	user 0.003s	sys 0.000s
I20260709 12:14:46.211969 19861 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:46.144043 19918 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:46.221921 19904 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.438s	user 0.004s	sys 0.000s
I20260709 12:14:46.227121 19918 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.234915 19919 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:46.077786 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:46.278249 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:46.367743 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 289865us
W20260709 12:14:46.369313 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 282695us
I20260709 12:14:46.140918 19585 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:46.369926 19842 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.212401 19900 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.371279 19842 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:46.281570 19704 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.540s	user 0.014s	sys 0.000s
I20260709 12:14:46.094311 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:46.284037 19581 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:46.282812 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:46.334236 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 280795us
W20260709 12:14:46.337759 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:46.346086 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:46.377053 19753 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.012s	sys 0.017s
I20260709 12:14:46.377501 12569 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260709 12:14:46.390264 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.391088 19920 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:14:46.394505 19645 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:46.395568 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 301191us
I20260709 12:14:46.395488 19704 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.397202 19866 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:46.396592 19581 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.397732 19723 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:46.395990 19912 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:46.408622 19866 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:46.437584 19847 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.452015 19866 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:46.452811 19847 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.414320 19534 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:46.414479 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:46.441919 19912 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:46.456734 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77669.7us
I20260709 12:14:46.415284 19861 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:46.457511 19534 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:46.418188 19704 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:46.458622 19861 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:46.443306 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.447824 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3391) took 1294 ms. Trace:
I20260709 12:14:46.462412 19912 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:46.462612 12490 rpcz_store.cc:276] 0709 12:14:45.153293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:45.153800 (+   507us) service_pool.cc:224] Handling call
0709 12:14:45.161899 (+  8099us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:45.627290 (+465391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.447199 (+819909us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:46.447587 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059849088}
I20260709 12:14:46.463912 12564 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:46.449563 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3399) took 1252 ms. Trace:
I20260709 12:14:46.466086 19900 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:46.449878 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.451217 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3394) took 1276 ms. Trace:
I20260709 12:14:46.474738 12495 rpcz_store.cc:276] 0709 12:14:45.175450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:45.175704 (+   254us) service_pool.cc:224] Handling call
0709 12:14:45.187045 (+ 11341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:45.316436 (+129391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:45.316472 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:45.316522 (+    50us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:45.316555 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.450421 (+1133866us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:46.450525 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:46.450924 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296950528}
W20260709 12:14:46.476148 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.464534 12496 rpcz_store.cc:276] 0709 12:14:45.197422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:45.198212 (+   790us) service_pool.cc:224] Handling call
0709 12:14:45.227157 (+ 28945us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:45.271915 (+ 44758us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:45.275255 (+  3340us) log.cc:844] Serialized 14 byte log entry
0709 12:14:45.275525 (+   270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:45.275581 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:45.275637 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.448927 (+1173290us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:46.449014 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:46.449298 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3829,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":7749,"raft.queue_time_us":2446,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":24516,"spinlock_wait_cycles":102010112}
W20260709 12:14:46.407999 19753 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.012s	sys 0.017s
I20260709 12:14:46.451463 19842 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:46.418044 19645 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.491300 19645 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:46.423691 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:46.493243 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 70118.8us
I20260709 12:14:46.494581 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:46.497031 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:46.498673 19864 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:14:46.481884 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75913.9us
I20260709 12:14:46.499624 19864 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:46.426908 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3227) took 1430 ms. Trace:
I20260709 12:14:46.500545 12568 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:14:46.491300 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2664) took 1947 ms. Trace:
I20260709 12:14:46.502276 12637 rpcz_store.cc:276] 0709 12:14:44.544133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:45.521483 (+977350us) service_pool.cc:224] Handling call
0709 12:14:45.529846 (+  8363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:45.582589 (+ 52743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.490742 (+908153us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:46.491076 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118678912}
I20260709 12:14:46.504123 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:46.505255 13253 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:46.435983 12565 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:46.399466 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:46.415679 19581 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:46.500849 12563 rpcz_store.cc:276] 0709 12:14:44.996030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:45.038975 (+ 42945us) service_pool.cc:224] Handling call
0709 12:14:45.039277 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:45.422948 (+383671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.426282 (+1003334us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:46.426682 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880432384}
I20260709 12:14:46.508996 19773 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:46.520383 19754 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:46.521873 19754 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:14:46.522024 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.522291 19754 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:46.523658 19754 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.493348 19847 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:46.536211 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137217us
I20260709 12:14:46.537330 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:46.525362 19754 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:46.590435 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.594389 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:46.602520 12568 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:14:46.576227 19933 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:14:46.610857 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1053) took 2163 ms (client timeout 1911 ms). Trace:
I20260709 12:14:46.595118 19861 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:46.612823 19753 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.624526 19753 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:46.625166 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.626696 12492 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:46.628123 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.628376 19753 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:46.617223 13258 rpcz_store.cc:269] 0709 12:14:44.444513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:44.444715 (+   202us) service_pool.cc:224] Handling call
0709 12:14:46.602028 (+2157313us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:46.608155 (+  6127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":932530432}
W20260709 12:14:46.630330 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93223.6us
I20260709 12:14:46.634598 12646 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:14:46.635600 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3057) took 2092 ms. Trace:
I20260709 12:14:46.633500 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:46.650187 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 153363us
I20260709 12:14:46.651428 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:46.650894 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.640169 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:46.640954 13255 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:46.657351 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:46.658329 12563 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:46.659969 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:46.660866 12563 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 21
W20260709 12:14:46.663625 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.665215 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.639967 19609 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:46.668334 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.668872 19840 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:46.669950 19840 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 21.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:14:46.670650 19840 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:46.670697 19864 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:14:46.668423 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.018s (SENT) [suppressed 12 similar messages]
I20260709 12:14:46.673823 19934 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:46.674685 19934 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:46.591049 19929 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:46.675897 19929 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:14:46.676082 19934 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.678126 19924 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:46.678507 19744 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:46.635964 12646 rpcz_store.cc:276] 0709 12:14:44.543054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.632845 (+2089791us) service_pool.cc:224] Handling call
0709 12:14:46.633755 (+   910us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:46.635211 (+  1456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.635511 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.682052 19628 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:46.683079 19768 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:46.687678 12493 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:46.690351 12493 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: [20.17-20.17]
I20260709 12:14:46.671090 19840 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:46.727963 12564 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:14:46.748987 12493 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:46.749593 19748 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:46.738150 19842 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.774755 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:46.776438 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 125235us
I20260709 12:14:46.776721 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:46.777580 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:46.777926 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65103.5us
I20260709 12:14:46.750191 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:46.749981 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2649) took 1007 ms. Trace:
I20260709 12:14:46.765856 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.781348 12493 rpcz_store.cc:276] 0709 12:14:45.742791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.735329 (+992538us) service_pool.cc:224] Handling call
0709 12:14:46.748716 (+ 13387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:46.749591 (+   875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.749880 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.787361 12489 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:46.754879 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3247) took 1052 ms. Trace:
I20260709 12:14:46.788604 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2645) took 1054 ms. Trace:
I20260709 12:14:46.788990 12570 rpcz_store.cc:276] 0709 12:14:45.660545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.445201 (+784656us) service_pool.cc:224] Handling call
0709 12:14:46.445823 (+   622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:46.446433 (+   610us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:46.448583 (+  2150us) log.cc:844] Serialized 14 byte log entry
0709 12:14:46.712971 (+264388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:46.713028 (+    57us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:46.713076 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.713153 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:46.713279 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5026,"prepare.queue_time_us":1556,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":87,"raft.queue_time_us":23359,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":63261,"thread_start_us":241987,"threads_started":1}
I20260709 12:14:46.791530 19585 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:46.792280 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:46.793182 12570 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:46.794728 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:46.745157 13253 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
W20260709 12:14:46.798839 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53953.7us
I20260709 12:14:46.756337 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.794446 19840 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 20.8, Last appended by leader: 8, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.795166 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:46.766222 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:46.800621 12489 rpcz_store.cc:276] 0709 12:14:45.733760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.688838 (+955078us) service_pool.cc:224] Handling call
0709 12:14:46.689620 (+   782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:46.788142 (+ 98522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.788442 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.766383 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3242) took 1126 ms. Trace:
I20260709 12:14:46.712445 19753 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:46.750666 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:46.811735 12568 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:46.812577 12497 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:46.743583 19744 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:46.764376 19664 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:46.817986 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:46.757787 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:46.815160 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2652) took 1071 ms. Trace:
I20260709 12:14:46.818981 12496 rpcz_store.cc:276] 0709 12:14:45.743815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.765357 (+1021542us) service_pool.cc:224] Handling call
0709 12:14:46.815095 (+ 49738us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.763285 19721 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:46.755123 19920 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.817404 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2642) took 1085 ms. Trace:
I20260709 12:14:46.824831 12490 rpcz_store.cc:276] 0709 12:14:45.732243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.755917 (+1023674us) service_pool.cc:224] Handling call
0709 12:14:46.817334 (+ 61417us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.823426 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.811975 19900 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:46.834213 13254 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:46.835798 12491 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:46.855628 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.774837 19842 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:46.829283 12564 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:46.738283 19924 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:14:46.733822 19628 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.114s
I20260709 12:14:46.829303 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:46.859807 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82484.8us
I20260709 12:14:46.863395 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:46.866464 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.818084 19664 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:46.821431 19721 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:46.866317 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 954) took 1129 ms. Trace:
I20260709 12:14:46.876255 12491 rpcz_store.cc:276] 0709 12:14:45.730728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.757271 (+1026543us) service_pool.cc:224] Handling call
0709 12:14:46.860227 (+102956us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:46.912626 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88822.6us
I20260709 12:14:46.914572 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:46.921957 12639 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:46.805601 12572 rpcz_store.cc:276] 0709 12:14:45.583668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.434695 (+851027us) service_pool.cc:224] Handling call
0709 12:14:46.435408 (+   713us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:46.436057 (+   649us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:46.436566 (+   509us) log.cc:844] Serialized 14 byte log entry
0709 12:14:46.696019 (+259453us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:46.696087 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:46.696138 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.696207 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:46.710143 (+ 13936us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75542,"prepare.queue_time_us":6952,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":97733,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":26634,"spinlock_wait_cycles":14006016,"thread_start_us":176316,"threads_started":1,"wal-append.queue_time_us":337405}
I20260709 12:14:46.933113 12496 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:14:46.929605 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:46.938674 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2659) took 1071 ms. Trace:
I20260709 12:14:46.940918 12496 rpcz_store.cc:276] 0709 12:14:45.867676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.825872 (+958196us) service_pool.cc:224] Handling call
0709 12:14:46.907328 (+ 81456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:46.933796 (+ 26468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.938597 (+  4801us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.936853 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:46.943029 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73475.8us
I20260709 12:14:46.944199 12489 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:46.940233 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:46.958361 12572 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:14:46.856820 19938 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.119s
I20260709 12:14:46.966480 12490 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:46.974303 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2661) took 2416 ms. Trace:
W20260709 12:14:46.975697 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:46.970577 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:46.965849 19874 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.015s
W20260709 12:14:46.976619 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 197961us
I20260709 12:14:46.974786 12639 rpcz_store.cc:276] 0709 12:14:44.522308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.749979 (+2227671us) service_pool.cc:224] Handling call
0709 12:14:46.938252 (+188273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:46.991170 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3245) took 1206 ms. Trace:
I20260709 12:14:46.992350 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.992799 19748 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:46.994463 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:46.994158 19748 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:46.996179 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:46.991765 12571 rpcz_store.cc:276] 0709 12:14:45.659282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.442082 (+782800us) service_pool.cc:224] Handling call
0709 12:14:46.442774 (+   692us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:46.446461 (+  3687us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:46.446498 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:46.484658 (+ 38160us) log.cc:844] Serialized 52 byte log entry
0709 12:14:46.692625 (+207967us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:46.692684 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.692768 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:46.840058 (+147290us) raft_consensus.cc:1685] finished
0709 12:14:46.840079 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:46.865592 (+ 25513us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2790,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"thread_start_us":207066,"threads_started":1,"wal-append.queue_time_us":330468}
W20260709 12:14:47.006227 19604 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.875s	user 0.002s	sys 0.028s
I20260709 12:14:46.906522 19866 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.010545 12565 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
W20260709 12:14:47.012898 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90020.1us
I20260709 12:14:47.007422 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:47.022768 19900 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:47.023728 19748 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:47.025712 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55232.5us
I20260709 12:14:47.027748 19716 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:47.027889 19905 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:47.028650 19861 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:47.028786 19716 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.028930 19905 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.030081 19861 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:47.030583 19861 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:47.031994 19905 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:47.031831 19861 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.031842 19716 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:47.028952 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:47.033422 19861 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:47.035115 19716 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:14:47.035861 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.034853 19859 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:47.036104 19716 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:47.036871 19716 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:47.039036 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:47.041926 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:47.042685 19859 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:47.043336 19859 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:47.043850 19859 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:46.921321 19920 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:46.998109 19840 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:47.057817 12564 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:46.921257 19753 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:46.987057 19934 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:47.071818 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.058720 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:47.068536 12490 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:47.015337 19585 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:47.082271 19905 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:47.089264 12493 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:14:47.090224 12570 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:47.097950 19664 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.092172 19900 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:47.100376 19900 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.104470 19900 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:47.082396 19721 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.109834 19642 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:47.110569 19920 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:47.110818 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62595.4us
I20260709 12:14:47.112354 19866 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:47.113384 19947 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:14:47.150313 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:47.150581 19842 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:47.131956 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:47.114653 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:47.130797 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:47.153074 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2660) took 1284 ms. Trace:
I20260709 12:14:47.153236 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:47.153656 12494 rpcz_store.cc:276] 0709 12:14:45.869022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:47.130025 (+1261003us) service_pool.cc:224] Handling call
0709 12:14:47.152967 (+ 22942us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:47.154316 12489 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:47.155191 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:47.116557 19912 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.653s	user 0.007s	sys 0.009s
I20260709 12:14:47.128747 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2657) took 1383 ms. Trace:
I20260709 12:14:47.146965 19934 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.156975 19912 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.159232 19583 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:47.159662 19912 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:47.159869 12497 rpcz_store.cc:276] 0709 12:14:45.745530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.664594 (+919064us) service_pool.cc:224] Handling call
0709 12:14:47.128368 (+463774us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:47.128616 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305971072}
I20260709 12:14:47.121951 19716 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.151912 12495 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:47.007082 19604 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.161355 19912 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:47.126286 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.162307 12491 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:14:47.121475 19942 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.550s	user 0.004s	sys 0.000s
I20260709 12:14:47.178134 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:47.180644 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 542953us
I20260709 12:14:47.180824 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2658) took 1335 ms. Trace:
I20260709 12:14:47.181376 12495 rpcz_store.cc:276] 0709 12:14:45.858630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:47.131502 (+1272872us) service_pool.cc:224] Handling call
0709 12:14:47.180747 (+ 49245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:47.128180 19753 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:47.145957 19642 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:47.173498 19754 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.604s	user 0.014s	sys 0.000s
I20260709 12:14:47.248291 12566 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:47.085021 19905 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:47.274315 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85737.2us
W20260709 12:14:47.285125 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 213002us
I20260709 12:14:47.294520 12492 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:14:47.295759 12565 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:14:47.252206 19938 consensus_queue.cc:1243] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 22.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:14:47.304291 12563 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.008s
I20260709 12:14:47.308835 12563 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:47.307718 12497 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:47.186298 19842 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.113s
I20260709 12:14:47.249231 19642 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:47.312438 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.249729 19924 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:47.186342 19912 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:47.315526 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 273789us
I20260709 12:14:47.294838 19721 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:47.380743 12565 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:47.389129 12563 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:47.390460 12495 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 12:14:47.390940 12495 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 10
W20260709 12:14:47.408979 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.409173 19604 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:47.409840 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:47.410856 13257 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
W20260709 12:14:47.413789 19953 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.280s	user 0.000s	sys 0.000s
W20260709 12:14:47.422348 19949 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.348s	user 0.001s	sys 0.000s
I20260709 12:14:47.424212 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3204) took 1843 ms. Trace:
I20260709 12:14:47.424875 12569 rpcz_store.cc:276] 0709 12:14:45.580475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.431317 (+850842us) service_pool.cc:224] Handling call
0709 12:14:46.431931 (+   614us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:46.432728 (+   797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:46.432757 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:46.433508 (+   751us) log.cc:844] Serialized 26 byte log entry
0709 12:14:46.815556 (+382048us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:46.815604 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:46.819929 (+  4325us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:47.423786 (+603857us) raft_consensus.cc:1685] finished
0709 12:14:47.423826 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:47.424112 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1445,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":178,"thread_start_us":240100,"threads_started":1,"wal-append.queue_time_us":413768}
I20260709 12:14:47.435897 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:47.445960 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 427823us
I20260709 12:14:47.311261 19748 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:47.466828 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:47.425050 19952 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.339s	user 0.001s	sys 0.001s
W20260709 12:14:47.485133 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.451397 12495 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:14:47.592432 12495 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:47.588606 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:47.427538 19716 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:47.383267 19929 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.210s
I20260709 12:14:47.463236 19604 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:14:47.645139 19960 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.356s	user 0.000s	sys 0.001s
I20260709 12:14:47.650287 19748 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.663877 19716 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:47.675535 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.664953 19716 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.404289 19924 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.703581 19961 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:47.705852 19912 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:47.670717 19604 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:47.721979 19583 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.005s
W20260709 12:14:47.674834 19859 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.630s	user 0.007s	sys 0.007s
W20260709 12:14:47.734608 19861 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.699s	user 0.007s	sys 0.008s
I20260709 12:14:47.744148 19861 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:47.568153 12494 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:14:47.732887 12564 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.654s	user 0.012s	sys 0.000s
I20260709 12:14:47.748322 12564 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:47.685467 19754 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:14:47.754796 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 324324us
W20260709 12:14:47.739395 12489 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.584s	user 0.011s	sys 0.002s
I20260709 12:14:47.745111 19861 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.764451 12489 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:47.738765 19604 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:47.762238 19866 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:47.774017 19866 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.775740 19905 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:47.776820 19866 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:47.771955 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2651) took 2023 ms (client timeout 2665 ms). Trace:
W20260709 12:14:47.778962 12492 rpcz_store.cc:269] 0709 12:14:45.743505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.629393 (+885888us) service_pool.cc:224] Handling call
0709 12:14:47.763594 (+1134201us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:47.766870 (+  3276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1448866176}
I20260709 12:14:47.759712 19754 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:47.785674 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 502595us
I20260709 12:14:47.749434 19859 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:47.799244 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 416521us
W20260709 12:14:47.865586 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.017s	sys 0.023s
W20260709 12:14:47.884109 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.018s	sys 0.029s
W20260709 12:14:47.772313 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.004s	sys 0.030s
W20260709 12:14:47.887533 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.004s	sys 0.031s
W20260709 12:14:47.887969 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 738267us
I20260709 12:14:47.886173 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:47.896539 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.799304 19754 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:47.899631 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:47.804741 19868 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:47.806298 19534 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:47.869937 19748 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:47.907191 19963 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:47.908805 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:47.928124 19534 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.909833 19883 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:47.930003 19883 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:47.930369 19748 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:47.931177 19883 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.931339 19748 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:47.932595 19883 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:47.928905 19963 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:47.936404 19951 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.853s	user 0.000s	sys 0.010s
I20260709 12:14:47.912029 19861 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:47.939378 19840 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.872s	user 0.009s	sys 0.004s
I20260709 12:14:47.936568 19963 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:47.940384 19604 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:47.946276 19609 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.279s	user 0.010s	sys 0.008s
I20260709 12:14:47.913228 19967 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:47.819340 19924 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:47.916712 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:47.792673 19961 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.917532 19905 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:47.919291 13257 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:47.920121 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:47.920981 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:47.927101 19626 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:47.911393 19859 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:47.931376 19534 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:47.947242 19567 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:47.964007 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:48.078729 19567 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.864149 19929 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.090s
I20260709 12:14:47.950752 19967 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:48.058976 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98299.4us
I20260709 12:14:48.058349 19964 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:48.151861 19964 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:48.065976 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:48.066219 19626 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.068405 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3404) took 1200 ms. Trace:
I20260709 12:14:48.068885 19604 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.069558 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3408) took 1107 ms. Trace:
I20260709 12:14:48.160305 12490 rpcz_store.cc:276] 0709 12:14:46.962460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.995792 (+ 33332us) service_pool.cc:224] Handling call
0709 12:14:48.069269 (+1073477us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:48.069478 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164319616}
W20260709 12:14:48.072270 19642 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.000s
W20260709 12:14:48.162115 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103342us
I20260709 12:14:48.161945 19642 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.164103 19924 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:48.164670 19642 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.074939 19961 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.164870 19924 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:47.831936 19716 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.061306 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:48.159709 12491 rpcz_store.cc:276] 0709 12:14:46.867994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.912083 (+ 44089us) service_pool.cc:224] Handling call
0709 12:14:48.068094 (+1156011us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:48.068300 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":396409600}
I20260709 12:14:47.953974 19868 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.155522 19967 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:48.183557 19748 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:48.183971 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 281612us
W20260709 12:14:48.171638 19842 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.046s	sys 0.077s
I20260709 12:14:48.171521 19628 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:48.192149 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:48.191443 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.012s	sys 0.058s
W20260709 12:14:48.193666 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.012s	sys 0.058s
W20260709 12:14:48.188007 19842 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.047s	sys 0.078s
I20260709 12:14:48.200277 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:14:48.208215 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.036s	sys 0.048s
W20260709 12:14:48.208742 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.036s	sys 0.048s
I20260709 12:14:48.189790 19961 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:14:48.225694 19964 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.238122 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.239517 12564 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:14:48.245185 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.249428 12565 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:14:48.250854 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.256409 12566 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:48.238765 19963 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.265872 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:48.265293 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3220) took 1438 ms. Trace:
I20260709 12:14:48.239456 19628 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:48.271656 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 476056us
I20260709 12:14:48.272410 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:48.295329 12564 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:48.269953 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.268759 12571 rpcz_store.cc:276] 0709 12:14:46.827185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:47.173312 (+346127us) service_pool.cc:224] Handling call
0709 12:14:47.252016 (+ 78704us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:47.313227 (+ 61211us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:47.383563 (+ 70336us) log.cc:844] Serialized 14 byte log entry
0709 12:14:48.264760 (+881197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:48.264841 (+    81us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:48.264909 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:48.265017 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:48.265199 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":50647,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":76154,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":800989,"threads_started":2,"wal-append.queue_time_us":803881}
I20260709 12:14:48.287559 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:48.299945 12565 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:48.282323 19962 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.305410 19975 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.308988 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.314989 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3221) took 1435 ms. Trace:
I20260709 12:14:48.320154 12536 leader_election.cc:400] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:48.320992 19818 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.321727 19968 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:48.294027 19961 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:14:48.331723 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:48.282717 19642 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:48.337695 12536 leader_election.cc:403] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:14:48.336110 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69304.5us
I20260709 12:14:48.336387 12572 rpcz_store.cc:276] 0709 12:14:46.879636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:47.181745 (+302109us) service_pool.cc:224] Handling call
0709 12:14:47.249301 (+ 67556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:47.314331 (+ 65030us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:47.314381 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:47.462086 (+147705us) log.cc:844] Serialized 26 byte log entry
0709 12:14:48.209292 (+747206us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:48.209355 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:48.209472 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:48.314638 (+105166us) raft_consensus.cc:1685] finished
0709 12:14:48.314673 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:48.314888 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7672,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":8328,"thread_start_us":530717,"threads_started":1,"wal-append.queue_time_us":666381}
W20260709 12:14:48.345314 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50439.8us
I20260709 12:14:48.351604 19656 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.352006 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.326766 19912 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.326782 19836 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.352432 12637 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:48.283717 19963 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.335778 19628 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.352793 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:48.360867 19970 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:14:48.361183 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.358742 19912 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.362452 12645 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:14:48.363941 19912 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.346352 19961 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:48.364738 19961 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:14:48.376132 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 152274us
I20260709 12:14:48.378511 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:48.379460 12566 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 20
W20260709 12:14:48.381042 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:48.382280 19842 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.382871 19842 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.383195 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:48.383926 12570 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:48.384613 19626 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.384979 19842 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:48.385617 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65837.9us
I20260709 12:14:48.351615 19818 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:48.387969 19959 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.050s	user 0.007s	sys 0.010s
I20260709 12:14:48.388496 19626 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.389125 19626 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.392611 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.393704 12571 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:48.356706 19924 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.340783 19962 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.420691 12570 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:48.423405 12567 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:14:48.425005 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.346190 19975 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:48.427697 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72275.7us
I20260709 12:14:48.375283 19974 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:48.387985 19818 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.352445 19656 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.352962 19968 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.438849 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:48.445703 13257 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:48.443687 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.443812 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.450618 19963 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.452793 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:48.453827 12640 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:48.454205 12571 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: [21.17-21.17]
I20260709 12:14:48.455389 19967 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.457134 19975 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.458038 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:48.398463 19912 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:14:48.431834 19626 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.454028 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.462320 12567 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:48.396065 19642 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.465600 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:48.477216 12572 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:48.482682 12641 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:48.373821 19836 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.478602 13261 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:48.497598 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:48.498894 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104822us
I20260709 12:14:48.509411 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:48.512912 12564 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:48.516727 12640 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:48.516775 12567 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:48.519966 12645 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:14:48.494746 19744 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.495157 19924 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:48.524008 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:48.520748 12645 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 11
I20260709 12:14:48.525542 12645 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:14:48.525898 12645 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:48.480829 19818 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:14:48.528366 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.510994 19967 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.536684 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:48.537292 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:48.534150 12536 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.587s (SENT)
I20260709 12:14:48.539287 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:48.540628 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:48.519583 19626 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.540760 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.476158 19974 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:14:48.556887 12567 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:48.558923 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:48.563057 19943 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.458s	user 0.010s	sys 0.010s
I20260709 12:14:48.564106 19974 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:48.566051 19963 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.566865 19963 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.569298 12643 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:48.569928 19716 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:48.570520 19716 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:48.570976 19716 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:48.518707 19975 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:14:48.574872 19883 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:14:48.575230 19963 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.560793 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:48.597572 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:48.597925 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:48.598434 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:48.573284 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:48.594856 13255 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:48.602367 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.602998 13255 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:48.604354 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:48.636317 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:48.637156 12567 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:48.638664 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.529963 19744 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.533830 19754 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.648634 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:48.653957 12492 raft_consensus.cc:2368] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24.
W20260709 12:14:48.654383 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131462us
I20260709 12:14:48.602149 19748 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:48.658138 12645 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.10-21.10]
W20260709 12:14:48.664700 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 145773us
I20260709 12:14:48.647338 19956 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:48.588712 19924 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.603789 19883 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:14:48.654646 13255 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.594424 19626 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.676098 13257 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:48.663772 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:48.652144 19628 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:48.680897 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:48.643465 19567 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.657056 19642 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.683730 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:48.707528 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 158797us
I20260709 12:14:48.685483 12572 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21. Re-sending same reply.
W20260709 12:14:48.689555 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 177135us
I20260709 12:14:48.711251 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:48.723858 19912 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:48.724645 19912 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:48.725065 19912 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:48.732625 19919 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:48.737274 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.837s (SENT)
I20260709 12:14:48.684746 19962 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.742853 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.742053 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:48.752465 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.754302 19996 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:14:48.755277 19996 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.755789 19996 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.758008 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.760015 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.760900 12563 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:48.760933 19934 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.762511 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.763242 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.764470 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:48.765066 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:48.763342 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:48.766026 12489 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:48.771304 19979 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:14:48.771438 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.772187 19842 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:48.772336 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:48.675575 19748 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.772701 19842 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:48.766677 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.773203 12489 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:48.774714 19656 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.749188 19836 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.766106 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.777225 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.778239 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:48.767577 19905 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.710s	user 0.010s	sys 0.004s
I20260709 12:14:48.770628 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.742280 19628 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.779589 19905 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:14:48.768393 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75427.6us
I20260709 12:14:48.778946 12492 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:48.766805 12497 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:48.765211 12638 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:48.772486 12494 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 17: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:48.781984 12642 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:48.766083 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.772650 12640 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:14:48.783176 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:48.788295 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66805us
I20260709 12:14:48.783123 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:48.684587 19958 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:14:48.792809 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:48.796710 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:48.718596 19968 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:48.827232 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 253995us
I20260709 12:14:48.814735 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.828251 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:48.814189 12642 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:14:48.829385 12492 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:48.815068 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.835799 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:48.836794 12567 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:48.839570 12563 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:48.840557 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:48.792723 19957 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:48.832782 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:48.845152 12565 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:14:48.842789 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:48.847277 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110026us
I20260709 12:14:48.848942 12565 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:48.849550 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:48.843878 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:48.798275 19604 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:48.825922 19958 consensus_queue.cc:1243] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 22.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:14:48.822568 19744 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.854100 12638 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:48.850368 19957 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:48.823378 19754 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.815167 19868 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.878571 19753 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:48.886720 19753 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:48.887146 19753 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:48.849596 19929 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:14:48.745728 19919 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.891966 19919 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.825587 19626 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.819551 19924 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:48.892894 12572 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:48.878247 19744 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 12:14:48.893301 12493 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:48.846390 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:48.894179 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 113095us
I20260709 12:14:48.850251 19968 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.861166 19964 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:48.806600 19716 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.892562 19748 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.846526 19974 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:48.881974 19996 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:48.882664 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98355.3us
I20260709 12:14:48.867067 19628 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:48.864737 19934 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 12:14:48.885880 19754 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:48.899230 12536 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.018s (SENT)
I20260709 12:14:48.853228 12640 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:48.919885 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:48.920740 12640 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:48.916951 12641 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:14:48.921934 12641 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:48.881290 19957 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:48.916667 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.927022 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.927846 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:48.928439 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.928774 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:48.929895 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:48.932109 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.933323 12638 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:48.934548 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:48.885293 19912 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.906261 19964 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:48.869334 19961 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:48.966614 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.881817 12388 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 16 to 19, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 19 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:48.946533 12642 raft_consensus.cc:2368] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
W20260709 12:14:48.941901 12566 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.518s	user 0.004s	sys 0.008s
I20260709 12:14:48.868197 19997 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:14:48.979058 12566 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:48.975443 19999 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:48.988401 12638 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:48.980530 19999 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.914696 19968 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:48.905905 19956 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
I20260709 12:14:48.965801 12641 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:49.009299 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:49.058243 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:49.058768 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.025171 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:48.953008 19986 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:14:49.013311 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114135us
I20260709 12:14:49.061496 12637 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:49.061400 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 192926us
I20260709 12:14:48.975885 19964 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:48.937983 19754 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:49.085386 19754 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.073654 12641 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:49.086207 12640 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:14:48.971596 19716 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:48.918543 19974 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.085206 19753 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.093962 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.091387 19958 raft_consensus.cc:2764] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election decision vote started in defunct term 20: won
I20260709 12:14:49.100421 12565 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:49.101011 12566 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:14:49.101994 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.098693 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.102967 12563 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:49.103607 12567 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:49.100222 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.104586 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.104691 12572 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:49.105535 12637 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:49.106863 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.037s (SENT)
W20260709 12:14:49.108084 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.126s (SENT)
I20260709 12:14:49.108575 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:49.109151 19753 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:49.109464 19748 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:49.110590 19912 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.110774 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:49.112296 19961 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.106640 19999 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.114115 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:49.114995 19974 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:49.115680 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120310us
I20260709 12:14:49.121763 19938 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:49.121912 19912 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:49.131568 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.106630 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.124176 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.133239 12640 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:49.126468 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:49.126778 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.134189 13253 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:49.137018 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.136s (SENT)
I20260709 12:14:49.128072 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:49.149878 13254 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:49.128880 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.129734 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.129878 13261 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:49.151211 12570 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:49.130138 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.130156 12565 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:49.125998 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.153285 12563 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:49.155620 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.127655 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.138144 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.117477 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:49.114912 12567 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:49.157755 12638 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:49.141842 19999 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:49.143359 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.143916 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.131197 19912 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:14:49.146049 13256 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:49.146166 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.146636 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.163763 12640 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:49.161298 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.167526 12638 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:49.149969 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.136s (SENT)
I20260709 12:14:49.075075 19986 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:48.993429 19968 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.167261 19999 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:49.181311 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.182298 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.183869 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.184717 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:49.159183 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.159623 12568 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 7.
W20260709 12:14:49.141000 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:49.186546 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:49.169227 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.169571 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:49.152473 19748 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:49.161919 19818 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:14:49.170461 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:49.220239 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.226661 12638 raft_consensus.cc:2368] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:49.227020 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.227999 12640 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:49.193051 19987 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:49.228842 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.228364 19975 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:49.227301 12565 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.10-21.10]
I20260709 12:14:49.229946 12388 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:49.194422 19860 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:49.231863 19860 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.208513 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.232183 12388 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 13 to 18, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 18 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:49.234321 19664 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:49.234608 12490 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:49.209132 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.220844 13255 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:49.183966 19999 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:49.194643 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 276273us
I20260709 12:14:49.209587 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:49.193445 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3383) took 5071 ms (client timeout 1911 ms). Trace:
I20260709 12:14:49.200104 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.172642 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:49.246583 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177350us
I20260709 12:14:49.236485 12493 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:14:49.200810 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:49.240080 19723 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:49.237582 19748 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.242645 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:49.242836 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92912.5us
I20260709 12:14:49.200655 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.255102 12646 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:14:49.229966 19975 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.235973 12388 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 12 to 20. New cstate: current_term: 20 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:49.262712 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.263878 13254 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:49.247215 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.264904 13256 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 16.
I20260709 12:14:49.058145 19842 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.209128 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.244156 12638 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:49.210482 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:49.214527 19989 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.441s	user 0.001s	sys 0.012s
I20260709 12:14:49.247098 12639 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:49.240483 19818 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:49.261445 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:49.266374 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78090.3us
W20260709 12:14:49.246090 12488 rpcz_store.cc:269] 0709 12:14:44.121554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:44.126791 (+  5237us) service_pool.cc:224] Handling call
0709 12:14:49.193186 (+5066395us) spinlock_profiling.cc:243] Waited 4.03 s on lock 0x7b540012c5d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:49.193337 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9271670144}
I20260709 12:14:49.272109 12492 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:49.291313 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:49.292989 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:49.272842 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.297109 19748 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:49.297995 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.298840 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:49.300377 12646 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:14:49.273350 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.302529 12640 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:49.303552 19975 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:49.274372 12567 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:49.304178 19975 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:49.278986 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.304798 12638 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:49.304733 19975 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.279738 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.306166 12496 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:49.195719 19961 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:49.306996 19961 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:49.307363 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.308045 19961 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.310045 12638 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:14:49.311385 12638 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:49.311832 19964 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.312510 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:49.316054 19987 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:49.316689 19987 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:49.317732 19987 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.319283 12388 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:49.321753 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.225155 19986 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.305955 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.333269 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:49.253894 19604 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:49.254148 19723 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:49.308509 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:49.285773 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.282653 13256 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:14:49.287392 19968 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:14:49.331581 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:49.346551 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 229143us
W20260709 12:14:49.326196 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:49.258937 12641 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:14:49.348584 12641 pending_rounds.cc:85] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 13
I20260709 12:14:49.312717 12638 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:14:49.350054 13259 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:49.351239 12388 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 12 to 19, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 19 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:49.352468 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.354106 12388 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:49.311221 19957 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.336318 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:49.324705 12637 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17. Re-sending same reply.
I20260709 12:14:49.340618 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.283726 19997 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:49.343387 12488 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:49.344252 19967 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 12:14:49.369531 19567 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:49.354547 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.372983 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.374576 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.461887 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:49.367251 19961 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:49.371043 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.476322 12494 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:49.348886 12641 pending_rounds.cc:107] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:14:49.477578 12641 raft_consensus.cc:2889] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:49.355253 19723 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.340498 19999 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 12:14:49.372165 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.480353 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:49.400113 19716 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:49.328366 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:49.488401 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160166us
I20260709 12:14:49.372514 12492 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: [20.17-20.17]
I20260709 12:14:49.395680 19818 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:49.403358 13260 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:49.376767 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:49.448683 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.460613 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:49.465132 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:49.489359 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:49.497033 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:49.467641 13255 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 17. (term mismatch)
I20260709 12:14:49.470378 19967 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:49.471055 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.471817 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:49.500739 13257 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:49.501261 13261 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:49.454455 19609 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:14:49.461757 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.461122 12496 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:49.400583 19744 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
I20260709 12:14:49.462291 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 16.
I20260709 12:14:49.481724 13258 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:49.483114 12490 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:49.370508 19842 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:49.363354 19975 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:49.484597 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 239631us
I20260709 12:14:49.469393 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:49.513350 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:49.485991 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 154519us
I20260709 12:14:49.516853 12488 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:49.490382 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:49.348775 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:49.491547 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.493942 19957 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.450490 19997 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:49.495759 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:49.498360 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:49.498831 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:49.521713 12567 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:49.496440 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.497591 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:49.478816 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.500231 19723 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:49.501662 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124964us
I20260709 12:14:49.499224 13255 pending_rounds.cc:85] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 16
I20260709 12:14:49.527504 13255 pending_rounds.cc:107] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 21.17
I20260709 12:14:49.527812 13255 raft_consensus.cc:2889] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:49.500303 19967 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:49.510676 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:49.460145 19934 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:49.494692 19604 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:49.509526 19868 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:49.526661 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:49.504967 19567 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:49.507328 12645 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:14:49.527241 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178563us
I20260709 12:14:49.544163 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.534968 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 986) took 2719 ms. Trace:
I20260709 12:14:49.533731 19912 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:49.525404 19975 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:49.524993 19997 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:49.545573 13255 rpcz_store.cc:276] 0709 12:14:46.811642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:46.811837 (+   195us) service_pool.cc:224] Handling call
0709 12:14:46.826190 (+ 14353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:48.594659 (+1768469us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400588a20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:14:49.220263 (+625604us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400588a20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5768a1d 00007ffbf574f1a4 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419
0709 12:14:49.530581 (+310318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:49.530862 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1494311552}
I20260709 12:14:49.550848 19912 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:49.551604 19934 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:49.552007 19818 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:49.552165 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51479.4us
I20260709 12:14:49.553146 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.551607 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.554919 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.543578 19604 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:49.556453 19868 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:49.565115 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:49.571928 12567 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:14:49.534147 19920 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:14:49.530851 19744 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:49.610841 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:49.629750 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:49.618455 20009 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 12:14:49.618923 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91591.2us
I20260709 12:14:49.574530 19975 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:49.619977 19964 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.623837 19748 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:49.618091 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:49.613335 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:49.634224 19818 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:49.626430 20013 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:49.663952 12491 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:14:49.659071 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:49.536379 20003 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:14:49.582593 19912 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:49.609082 19604 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:49.580356 19934 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:49.612020 19868 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:49.667483 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56430.2us
I20260709 12:14:49.663610 19567 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:14:49.652237 19957 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:49.664175 19975 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:49.661914 19748 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:49.667038 19818 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:49.675634 19934 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:49.673501 19999 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:14:49.718772 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 223069us
I20260709 12:14:49.720960 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:49.696877 19974 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260709 12:14:49.732816 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96936.6us
I20260709 12:14:49.702363 19957 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:49.710986 19818 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:49.713697 19999 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:49.666819 19967 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:49.717926 19986 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:49.702133 19664 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:14:49.772193 19957 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.779909 19934 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:49.782414 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45143.7us
I20260709 12:14:49.738427 19999 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:49.792145 19934 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:49.795248 19934 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:49.796911 12572 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 12:14:49.798336 20010 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 12:14:49.800146 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3092) took 1206 ms. Trace:
I20260709 12:14:49.800665 12643 rpcz_store.cc:276] 0709 12:14:48.594063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:48.601939 (+  7876us) service_pool.cc:224] Handling call
0709 12:14:48.647988 (+ 46049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:49.161362 (+513374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:49.161398 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:49.170530 (+  9132us) log.cc:844] Serialized 26 byte log entry
0709 12:14:49.399869 (+229339us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:49.399914 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:49.400164 (+   250us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:49.799831 (+399667us) raft_consensus.cc:1685] finished
0709 12:14:49.799857 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:49.800041 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":478918,"raft.queue_time_us":476427,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4783,"thread_start_us":201899,"threads_started":1,"wal-append.queue_time_us":305696}
I20260709 12:14:49.815001 19642 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:14:49.815689 19642 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:49.821101 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 305298us
W20260709 12:14:49.754381 13218 connection.cc:256] RPC call timeout handler was delayed by -1.3024610090001261s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2175s for a response
I20260709 12:14:49.822266 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:49.851707 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 325100us
I20260709 12:14:49.816823 19642 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:49.860730 12536 leader_election.cc:400] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:49.864933 12536 leader_election.cc:403] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:49.878257 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:49.879701 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.014s	sys 0.020s
I20260709 12:14:49.863473 19609 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:49.887262 12567 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:14:49.887223 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:49.888096 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.014s	sys 0.020s
I20260709 12:14:49.889782 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:49.893219 12571 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:49.859861 19567 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:14:49.864406 20013 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:49.893422 12567 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:14:49.876051 19967 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:49.892731 12572 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:49.887372 19723 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:14:49.908677 19860 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:49.914933 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:49.915385 20011 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
I20260709 12:14:49.919777 12566 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:49.895413 19963 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:49.855191 19987 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:14:49.928040 12564 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:14:49.930532 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:49.935370 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 214546us
I20260709 12:14:49.943332 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:49.936061 12609 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.001s	sys 0.040s
W20260709 12:14:49.948833 12609 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.001s	sys 0.041s
W20260709 12:14:49.949380 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 618794us
W20260709 12:14:49.950510 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:49.979462 13261 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [20.15-22.16]
W20260709 12:14:49.980652 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50241us
I20260709 12:14:49.930358 19609 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:49.995909 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:49.994661 19961 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.006s
I20260709 12:14:50.005362 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.002357 19883 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:50.022678 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.008528 20025 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:49.993824 19997 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.002772 19963 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
W20260709 12:14:50.044044 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.040460 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:50.021164 19656 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 12:14:50.045588 12536 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:50.061606 19868 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.024374 19883 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:50.065062 12536 leader_election.cc:403] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:49.973075 20013 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:50.066490 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173658us
I20260709 12:14:50.032634 19567 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:14:50.070581 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 193383us
I20260709 12:14:50.072501 12495 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:50.076133 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:50.040445 20025 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:50.088546 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210418us
W20260709 12:14:50.091310 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45932us
I20260709 12:14:50.092650 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.066897 19860 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:50.032783 20030 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:14:50.065762 20029 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:50.078612 19842 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:50.104379 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 158622us
W20260709 12:14:50.119771 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 176569us
I20260709 12:14:50.126655 20031 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.127552 20031 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:50.130584 13218 leader_election.cc:336] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.982s (SENT)
I20260709 12:14:50.091354 20025 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:50.097084 20015 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:50.104128 19860 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:50.132581 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.078442 19664 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:14:50.131688 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:50.152966 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77019.4us
W20260709 12:14:50.154963 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65416.7us
I20260709 12:14:50.109954 20029 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.122184 19968 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:14:50.170516 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103233us
I20260709 12:14:50.172340 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.173921 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.157668 19860 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:50.152119 20024 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:50.130049 19842 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:50.197762 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77182.1us
W20260709 12:14:50.201977 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.173607 19968 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.130131 20036 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:14:50.178683 19958 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:50.116974 19609 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 22.10, Last appended by leader: 10, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.144026 20033 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 12:14:50.178222 20029 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:50.135066 19883 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.201145 19818 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:50.230880 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 112304us
W20260709 12:14:50.231467 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111787us
I20260709 12:14:50.180017 20015 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:14:50.212086 19968 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:50.195485 20043 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:50.205387 20038 raft_consensus.cc:2764] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:14:50.210060 19965 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:50.241243 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.242125 12490 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:14:50.243741 20015 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:14:50.246001 20042 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:50.274374 12569 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:50.238639 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.274850 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.276718 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.214671 19567 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:14:50.263510 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
W20260709 12:14:50.263303 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 748256us
W20260709 12:14:50.240427 19912 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.552s	user 0.013s	sys 0.000s
I20260709 12:14:50.246133 20047 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:14:50.268317 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:50.267572 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144855us
I20260709 12:14:50.270152 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:50.277984 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77920.1us
I20260709 12:14:50.269275 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:50.237344 19744 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.004s
W20260709 12:14:50.282230 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.268039 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.291651 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:50.275048 19975 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.554s	user 0.013s	sys 0.000s
I20260709 12:14:50.295212 19744 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:50.299738 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:50.247174 19965 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:50.293780 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.292062 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:50.301877 13258 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:50.291015 12571 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:50.293282 13259 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:50.294447 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:50.268046 19997 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.304638 13260 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:50.284217 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.299453 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.248283 20036 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:14:50.274703 20043 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:50.303879 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:50.309116 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:50.293594 20023 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 12:14:50.304426 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:50.306078 12494 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:14:50.301160 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:50.311517 12497 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:50.315666 12567 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:50.287598 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.315752 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.314227 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:50.317742 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:50.320889 19999 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.007s
I20260709 12:14:50.321585 19868 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:50.290169 19912 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.324461 19912 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.321671 20037 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:50.331668 12572 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:50.327069 20018 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.614s	user 0.003s	sys 0.003s
I20260709 12:14:50.326773 20055 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:50.311832 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:50.331894 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:50.320879 19986 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.562s	user 0.013s	sys 0.000s
I20260709 12:14:50.333673 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.338392 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.300681 19744 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:50.328137 19997 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:50.351845 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.359994 12489 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:50.332386 12492 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:14:50.311358 20054 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:50.370522 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:14:50.371122 12492 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 8
I20260709 12:14:50.299672 19975 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.321404 19999 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.373075 12492 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:14:50.376662 12492 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:50.357896 20055 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:14:50.377892 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.327898 20043 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.351573 20037 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:50.382706 20037 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.387223 12640 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:50.391155 20031 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:50.354513 20041 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:14:50.387401 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 887) took 1012 ms. Trace:
I20260709 12:14:50.406446 12489 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:14:50.416478 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 216195us
I20260709 12:14:50.416566 12640 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:14:50.418280 20013 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:14:50.421439 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:50.362982 19997 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.393730 13256 rpcz_store.cc:276] 0709 12:14:49.374510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:49.374680 (+   170us) service_pool.cc:224] Handling call
0709 12:14:49.375066 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:49.375648 (+   582us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:49.375685 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:49.376385 (+   700us) log.cc:844] Serialized 26 byte log entry
0709 12:14:49.497195 (+120810us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:49.497240 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:49.497312 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:50.326393 (+829081us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:50.387023 (+ 60630us) raft_consensus.cc:1685] finished
0709 12:14:50.387048 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:50.387325 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19518,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":43567,"spinlock_wait_cycles":12342656,"thread_start_us":120368,"threads_started":1,"wal-append.queue_time_us":286426}
W20260709 12:14:50.413345 19604 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.730s	user 0.012s	sys 0.003s
I20260709 12:14:50.372440 20054 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:50.375906 19842 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.429551 12492 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:50.429823 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:50.442286 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.444017 12640 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
W20260709 12:14:50.449455 20044 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.001s	sys 0.001s
I20260709 12:14:50.454134 20031 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.424362 19997 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:50.474820 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 275423us
I20260709 12:14:50.434847 19818 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:50.427600 20050 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:14:50.482832 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.482999 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:50.486228 20031 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.496973 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:50.491058 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 259191us
W20260709 12:14:50.494056 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 158884us
I20260709 12:14:50.443238 19744 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:50.496150 20050 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:50.439620 19604 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:50.484503 19967 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.004s
I20260709 12:14:50.502668 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:50.511538 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:50.500269 20060 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:14:50.501205 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:50.554436 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:50.555508 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:50.557806 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56719.5us
I20260709 12:14:50.564245 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.565361 12640 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:50.557219 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.573175 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:50.570040 19842 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 12:14:50.529670 19604 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.636694 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.490185 20064 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:50.505309 20031 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.637648 12640 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
W20260709 12:14:50.581485 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 288933us
I20260709 12:14:50.570010 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:50.424352 19965 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.637521 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:50.580418 20020 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.011s	sys 0.013s
I20260709 12:14:50.682961 12563 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:50.684487 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:50.574726 20043 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:50.691659 12571 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:50.692544 12490 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:50.693121 13256 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:50.647291 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.589922 19818 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:50.701651 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:50.703661 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56361.3us
I20260709 12:14:50.709424 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:50.683252 20020 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.012s	sys 0.014s
I20260709 12:14:50.734362 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:50.737082 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:50.745870 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 324563us
I20260709 12:14:50.746541 12642 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:50.739490 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:50.723235 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3098) took 1835 ms. Trace:
I20260709 12:14:50.748298 12639 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:50.733217 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.733810 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:50.755087 12645 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:50.749347 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 246496us
I20260709 12:14:50.692173 20043 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.508181 19744 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:50.706523 19716 consensus_queue.cc:1175] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 23 status { last_received { term: 22 index: 16 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:14:50.790242 20060 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:50.790952 20060 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.791693 19965 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:50.785981 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.699689 19604 raft_consensus.cc:2764] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:14:50.729908 20048 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:50.794451 20048 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.794821 20048 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:50.795825 20025 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.643s	user 0.012s	sys 0.000s
I20260709 12:14:50.796315 20025 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.699838 19967 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.700726 20064 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.798389 19967 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.732442 19999 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.800546 20064 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:50.801076 19975 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.801323 20020 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.788692 19609 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:50.737766 19842 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.802235 19609 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:50.802785 20031 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.802868 19842 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.509950 20050 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.803362 20031 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.804461 20050 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:50.715008 19818 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.722975 20063 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:50.801936 20020 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.576426 20054 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:50.726801 20062 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:14:50.505590 19860 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:50.812573 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103294us
I20260709 12:14:50.723666 20056 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:50.748409 12644 rpcz_store.cc:276] 0709 12:14:48.755437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:48.757879 (+  2442us) service_pool.cc:224] Handling call
0709 12:14:48.758294 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:49.171848 (+413554us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:14:49.494822 (+322974us) log.cc:844] Serialized 14 byte log entry
0709 12:14:49.865263 (+370441us) log.cc:844] Serialized 14 byte log entry
0709 12:14:49.865523 (+   260us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:49.889746 (+ 24223us) log.cc:844] Serialized 26 byte log entry
0709 12:14:49.890010 (+   264us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:49.890060 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:49.890159 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:50.590390 (+700231us) raft_consensus.cc:1685] finished
0709 12:14:50.590424 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:50.590645 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":544040,"prepare.queue_time_us":144278,"prepare.run_cpu_time_us":871,"prepare.run_wall_time_us":8895,"raft.queue_time_us":422585,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":12006,"thread_start_us":505870,"threads_started":2,"wal-append.queue_time_us":505648}
I20260709 12:14:50.800546 19967 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:50.803485 19609 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.723960 20068 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:50.724095 20059 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.248s
I20260709 12:14:50.827970 20068 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:50.832796 20042 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.000s
I20260709 12:14:50.833288 20042 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:50.805961 19818 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:50.835253 20020 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.837015 20025 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.844825 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:50.852244 12645 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:50.853904 12637 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:14:50.806030 20043 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:50.859251 12639 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:50.859992 12497 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:14:50.805595 19842 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.793480 20060 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:50.860610 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44210.2us
I20260709 12:14:50.870287 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.861785 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3117) took 1419 ms. Trace:
I20260709 12:14:50.864600 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.876905 12497 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:50.875625 12641 rpcz_store.cc:276] 0709 12:14:49.424705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:49.533225 (+108520us) service_pool.cc:224] Handling call
0709 12:14:50.843520 (+1310295us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:50.843667 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2126663936}
I20260709 12:14:50.874624 12488 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:50.871738 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.865329 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.891150 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.891687 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:50.892603 19723 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.955s	user 0.009s	sys 0.013s
I20260709 12:14:50.829493 19997 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.863651 20028 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:50.894243 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.894594 12496 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:50.820793 20056 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:50.885171 20073 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:50.896575 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:14:50.897663 19723 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:14:50.901057 20058 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:50.901324 20025 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:50.918254 20025 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.918861 20074 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:14:50.920467 20025 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:50.922880 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2661) took 2408 ms (client timeout 1790 ms). Trace:
W20260709 12:14:50.923594 12491 rpcz_store.cc:269] 0709 12:14:48.514318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:48.519345 (+  5027us) service_pool.cc:224] Handling call
0709 12:14:50.922484 (+2403139us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:50.922707 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2586829568}
I20260709 12:14:50.897363 19656 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:50.926501 20056 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:50.926597 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.925611 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:50.932809 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:50.933908 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 414949us
I20260709 12:14:50.935524 12642 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:50.935634 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.936888 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:50.940898 12568 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:14:50.945575 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:50.964900 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:50.930905 20073 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:50.902733 20031 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:50.912096 19997 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:50.903599 20028 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:50.980209 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:50.976970 20042 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:50.998489 20042 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:50.999883 20042 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:50.945219 20064 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:50.979847 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.982618 13256 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:14:50.986629 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.988183 13259 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:14:50.994120 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 436996us
I20260709 12:14:51.009150 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:50.985913 19934 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:51.013906 12491 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: [21.14-21.14]
W20260709 12:14:51.015961 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:50.969566 20063 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:14:51.021221 12488 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:50.944099 20074 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.066s
I20260709 12:14:51.025507 12491 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:14:50.989984 19642 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:51.020406 20025 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:50.981712 19963 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:51.011758 19934 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:51.032747 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:50.972791 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:50.991712 20033 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:51.037227 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57491.9us
I20260709 12:14:51.006614 20064 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:50.973183 20073 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:51.072750 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100088us
I20260709 12:14:51.071172 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:51.073652 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:51.048484 20063 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:51.076719 20080 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:51.082785 19934 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.10, Last appended by leader: 10, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.083035 20080 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:51.085297 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.085876 20031 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:51.086581 20028 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:50.983942 19748 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:51.090238 19860 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:51.090616 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:51.119319 12570 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:51.076467 20064 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:51.081267 20068 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:51.081863 20063 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:51.103503 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:51.140128 12563 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:51.072769 19963 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:51.112123 12489 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.016s	sys 0.023s
W20260709 12:14:51.159111 12489 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.017s	sys 0.023s
W20260709 12:14:51.125590 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:51.126595 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55577.5us
I20260709 12:14:51.163192 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:51.120528 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:51.103219 19818 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:51.124876 20029 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.891s	user 0.011s	sys 0.005s
W20260709 12:14:51.143064 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2686) took 1934 ms (client timeout 2467 ms). Trace:
W20260709 12:14:51.172458 12495 rpcz_store.cc:269] 0709 12:14:49.208611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:49.208787 (+   176us) service_pool.cc:224] Handling call
0709 12:14:51.142730 (+1933943us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:51.142964 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1837416320}
I20260709 12:14:51.173162 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:51.116995 20031 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:51.104687 19997 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:14:51.169284 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1242) took 1561 ms (client timeout 1617 ms). Trace:
I20260709 12:14:51.135694 19934 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:51.184108 12569 rpcz_store.cc:269] 0709 12:14:49.579361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:49.579854 (+   493us) service_pool.cc:224] Handling call
0709 12:14:51.140050 (+1560196us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:51.140269 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385910144}
I20260709 12:14:51.193198 12563 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:51.188961 12491 raft_consensus.cc:2376] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17. Re-sending same reply.
I20260709 12:14:51.196776 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:51.157946 20063 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:51.117699 20048 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.202224 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:51.183490 19997 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:51.205047 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:51.207734 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:51.208662 19716 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:51.210201 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47120.2us
I20260709 12:14:51.244841 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:51.241463 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:51.255280 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.258000 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.213706 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:51.215284 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:51.260565 12494 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:51.214927 20073 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:51.216097 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:51.286281 12563 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:51.219067 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:51.287597 12565 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:14:51.184514 20031 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:51.169910 20035 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:51.261543 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:51.289253 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:51.262172 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:51.288468 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:51.295043 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.296223 12645 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:51.296736 12570 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:14:51.295106 12640 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:51.239585 19868 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:51.239313 19997 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:51.295735 12494 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:51.313673 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:51.301898 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.303115 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:51.238521 20080 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:51.314947 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:51.312355 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70981.1us
W20260709 12:14:51.317063 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.289995 20058 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:51.185289 20029 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:51.322069 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:51.327009 13255 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:51.289837 19656 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:51.339564 12563 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.10-21.10]
I20260709 12:14:51.307073 19999 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:51.319854 20028 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:14:51.346336 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.350792 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.356966 12564 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:51.349675 12497 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:51.366086 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:51.367027 13255 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:51.287074 19716 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:51.354820 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:51.373368 13258 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:51.375164 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 478598us
I20260709 12:14:51.377602 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:51.377733 13257 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:51.378748 13254 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:14:51.391626 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:51.393817 12645 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:51.396531 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.358387 20028 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:51.352350 20035 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:14:51.354022 20024 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:14:51.414083 19968 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.148s	user 0.018s	sys 0.000s
I20260709 12:14:51.391603 20090 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:51.359997 19999 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.325835 19963 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.510685 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:51.511727 13258 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:51.498879 12567 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:51.467928 13255 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:51.510759 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:51.474584 12645 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:51.425300 20028 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:51.410326 20080 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:51.443169 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:51.548018 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 242972us
I20260709 12:14:51.433475 20024 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:51.377467 20058 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.462011 20035 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:51.546216 19920 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:51.550056 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.568946 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
W20260709 12:14:51.569900 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4004994129998067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3183s for a response
I20260709 12:14:51.565066 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:51.494098 19818 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:51.592016 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.595129 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:51.574141 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127784us
I20260709 12:14:51.467844 20025 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:51.462316 20079 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:51.428485 20084 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.138s
W20260709 12:14:51.583171 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3549947960000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
I20260709 12:14:51.583271 12645 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:51.561156 19868 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:51.603792 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1607121309998547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2969s for a response
I20260709 12:14:51.573128 19609 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:51.606033 20081 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.312s	user 0.002s	sys 0.000s
I20260709 12:14:51.601114 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:51.573292 20048 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:51.605815 20093 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:51.616643 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.549871 19968 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.411985 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
W20260709 12:14:51.655714 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3240) took 2477 ms (client timeout 2210 ms). Trace:
W20260709 12:14:51.608379 19965 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.816s	user 0.015s	sys 0.000s
W20260709 12:14:51.656215 12566 rpcz_store.cc:269] 0709 12:14:49.158733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:49.158943 (+   210us) service_pool.cc:224] Handling call
0709 12:14:51.635688 (+2476745us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:51.635867 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1583168768}
I20260709 12:14:51.562068 20059 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.035s
W20260709 12:14:51.631356 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 382897us
I20260709 12:14:51.513191 19656 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.575815 20080 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.634179 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.593659 19967 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:14:51.660408 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 189385us
I20260709 12:14:51.601498 20086 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:51.665113 12643 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:51.600334 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:51.549556 20090 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:51.630380 19609 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:51.678041 12567 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:14:51.640748 19818 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:51.677645 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:51.689188 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88959.5us
I20260709 12:14:51.690356 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.603336 19920 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:51.677098 19716 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:51.657408 19974 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:51.694111 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.682811 12645 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
W20260709 12:14:51.702358 20092 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:14:51.702538 19968 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:51.703361 19968 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:51.696132 20035 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.666862 20091 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:14:51.697517 19965 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:51.630920 20025 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.703971 19968 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:51.709784 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.727032 13254 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:14:51.712361 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.779096 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:51.780469 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68159.7us
W20260709 12:14:51.762650 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82419.1us
I20260709 12:14:51.692710 20087 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:51.773167 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.698843 19818 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.659374 20077 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:51.709175 20108 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:51.726068 19965 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:51.782315 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:51.701265 20101 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.247s
I20260709 12:14:51.801183 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:51.788971 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.790874 12566 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:51.735709 19967 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:14:51.764251 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.724591 19997 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:51.862872 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 195378us
W20260709 12:14:51.937150 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150238us
W20260709 12:14:51.937984 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:51.938864 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.791074 20087 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:51.814527 20100 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:14:51.780782 20073 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:51.668807 20093 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:51.911863 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:51.726157 19974 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:51.912046 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.093s (SENT) [suppressed 33 similar messages]
I20260709 12:14:51.797286 20109 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:51.912750 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.803391 19965 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:51.788228 19860 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:51.796904 20108 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:51.921837 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.923583 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:51.773234 19868 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.776674 12388 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 13 to 21, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 21 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:14:51.809793 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:51.967056 12639 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:51.946703 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:51.967528 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:51.964406 12571 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:51.970114 12491 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:51.968148 12644 raft_consensus.cc:2376] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20. Re-sending same reply.
I20260709 12:14:51.972909 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:51.804459 20077 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:51.984637 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:51.988906 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 17.
I20260709 12:14:51.976018 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.991369 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:51.933190 20115 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:51.996091 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:51.954188 20101 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:51.926263 20084 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
W20260709 12:14:52.004673 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 230646us
I20260709 12:14:51.955718 20029 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.005477 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:52.006371 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:51.985075 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 228) took 77198 ms (client timeout 15000 ms). Trace:
W20260709 12:14:51.985328 20106 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.162s	user 0.005s	sys 0.000s
I20260709 12:14:51.969852 19999 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:51.974428 19664 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:14:52.002859 20052 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:14:52.013688 13253 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:52.033445 13253 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:14:52.033937 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 232879us
I20260709 12:14:52.014262 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:51.978861 20097 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.150s	user 0.000s	sys 0.005s
I20260709 12:14:52.022850 12571 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:14:52.030040 13257 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.008s	sys 0.048s
I20260709 12:14:52.024837 12566 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:52.026086 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:52.039939 13257 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.008s	sys 0.048s
I20260709 12:14:52.040612 12645 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:52.024940 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112957us
I20260709 12:14:52.039997 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.034289 13253 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.043447 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.009956 13258 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:14:52.028882 20099 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.367s	user 0.001s	sys 0.001s
I20260709 12:14:52.038276 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:52.008297 19967 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:52.011049 19716 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:52.017061 19968 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.054363 13253 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:52.045176 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:52.046062 20048 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:14:52.045037 12571 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:52.067734 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.029696 19664 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.011502 20115 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.029989 19860 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:51.974509 20109 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.069917 12639 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:52.098635 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54082.8us
I20260709 12:14:52.113127 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:52.043874 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47857.8us
I20260709 12:14:52.096164 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:52.108587 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 186828us
I20260709 12:14:52.115607 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:52.039139 20025 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.111884 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.046233 19609 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:14:52.115893 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:52.127281 12571 raft_consensus.cc:2376] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24. Re-sending same reply.
I20260709 12:14:52.118583 20117 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.092s
I20260709 12:14:52.075059 20077 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.119884 20126 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:52.121325 20128 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:52.051424 20080 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.072368 20029 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:52.122268 20113 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.092s	user 0.000s	sys 0.002s
I20260709 12:14:52.122740 19860 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:52.123122 20056 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.196s	user 0.006s	sys 0.008s
I20260709 12:14:51.985965 20108 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.129594 12645 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:52.131305 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:52.072932 19642 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.035s	user 0.017s	sys 0.000s
I20260709 12:14:52.147593 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3441) took 1850 ms. Trace:
I20260709 12:14:52.121765 20120 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:52.062546 19818 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.077188 20101 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:52.075759 20090 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:52.151886 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1245) took 1078 ms. Trace:
I20260709 12:14:52.155464 20126 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:14:52.155265 20029 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.109575 19968 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:52.176925 19656 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:52.187330 20128 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:52.108078 20093 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:52.180271 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49064.9us
I20260709 12:14:52.182341 12489 rpcz_store.cc:276] 0709 12:14:50.297030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.408105 (+111075us) service_pool.cc:224] Handling call
0709 12:14:50.408684 (+   579us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:50.455022 (+ 46338us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:14:50.986350 (+531328us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b1800474e80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:51.172248 (+185898us) log.cc:844] Serialized 14 byte log entry
0709 12:14:51.428149 (+255901us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:51.429210 (+  1061us) log.cc:844] Serialized 26 byte log entry
0709 12:14:51.429556 (+   346us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:51.429592 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:51.429808 (+   216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:52.125694 (+695886us) raft_consensus.cc:1685] finished
0709 12:14:52.147314 (+ 21620us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.147502 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82783,"prepare.queue_time_us":75238,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":117650,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":61611,"spinlock_wait_cycles":5201664,"thread_start_us":920051,"threads_started":2,"wal-append.queue_time_us":474026}
I20260709 12:14:52.180395 12645 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
W20260709 12:14:52.209419 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3294) took 1910 ms (client timeout 2093 ms). Trace:
W20260709 12:14:52.154189 19934 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.960s	user 0.015s	sys 0.000s
I20260709 12:14:52.157544 12388 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 19, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 19 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:14:52.188162 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.067360 20048 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.086s
W20260709 12:14:52.183794 20028 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.604s	user 0.013s	sys 0.000s
I20260709 12:14:52.196185 12572 rpcz_store.cc:276] 0709 12:14:51.073201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.074000 (+   799us) service_pool.cc:224] Handling call
0709 12:14:51.080171 (+  6171us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:51.103035 (+ 22864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:51.103085 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:51.103838 (+   753us) log.cc:844] Serialized 26 byte log entry
0709 12:14:51.396987 (+293149us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:51.397034 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:51.397114 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:52.151543 (+754429us) raft_consensus.cc:1685] finished
0709 12:14:52.151574 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.151799 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22207,"raft.queue_time_us":37079,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":20010,"thread_start_us":292639,"threads_started":1,"wal-append.queue_time_us":605568}
I20260709 12:14:52.116789 19664 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:14:52.218055 20098 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:14:52.220366 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.222038 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.223337 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.224419 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.192344 20080 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:14:52.226634 20048 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:14:52.227198 20048 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:52.227720 20128 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:52.209022 19656 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.228080 20048 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.228787 20128 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.230202 20115 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.214666 19974 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.231194 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.231416 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:52.201032 20101 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.232381 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.232813 20124 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:14:52.233158 12572 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:14:52.212317 19997 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.233848 19860 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.182308 19883 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:52.235081 12641 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
W20260709 12:14:52.128599 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.160864 20130 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:52.194557 20108 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:52.182289 20056 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.232187 20127 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:52.255453 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3259) took 1702 ms. Trace:
I20260709 12:14:52.234324 20124 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:52.236780 12641 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: [16.9-18.10]
I20260709 12:14:52.183207 20077 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 12:14:52.198884 19818 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:14:52.204517 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82387us
W20260709 12:14:52.204423 20043 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.312s	user 0.013s	sys 0.007s
I20260709 12:14:52.267549 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.237944 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:52.270617 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.256116 12564 rpcz_store.cc:276] 0709 12:14:50.553015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.553152 (+   137us) service_pool.cc:224] Handling call
0709 12:14:52.255104 (+1701952us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.255346 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1800235264}
W20260709 12:14:52.209918 12568 rpcz_store.cc:269] 0709 12:14:50.299294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.299434 (+   140us) service_pool.cc:224] Handling call
0709 12:14:52.209204 (+1909770us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b5400044f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.209349 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1416024832}
I20260709 12:14:52.241043 20130 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.246910 20029 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:52.214980 20064 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.056s	user 0.013s	sys 0.011s
I20260709 12:14:52.229445 20048 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:52.286700 12572 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
W20260709 12:14:52.298393 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60546.2us
I20260709 12:14:52.272389 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.273785 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:52.299567 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:52.299542 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.270188 20124 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.294364 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.315752 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:52.295207 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:52.304121 12568 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:52.307085 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:52.310276 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:52.260004 12489 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:14:52.317852 12568 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:52.318538 20126 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:52.318327 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:14:52.012856 12391 rpcz_store.cc:269] 0709 12:13:34.720595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:35.145954 (+425359us) service_pool.cc:224] Handling call
0709 12:14:51.815327 (+76669373us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:14:41.301810 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:14:41.530291 (+228481us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:14:41.544001 (+ 13710us) write_op.cc:435] Acquired schema lock
0709 12:14:41.553254 (+  9253us) tablet.cc:662] Decoding operations
0709 12:14:41.714044 (+160790us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:14:41.714133 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:14:41.714161 (+    28us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:14:41.715016 (+   855us) tablet.cc:701] Row locks acquired
0709 12:14:41.715070 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:14:41.715299 (+   229us) write_op.cc:270] Start()
0709 12:14:41.715482 (+   183us) write_op.cc:276] Timestamp: P: 1783599281715209 usec, L: 0
0709 12:14:41.715504 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:14:41.807997 (+ 92493us) log.cc:844] Serialized 3248 byte log entry
0709 12:14:42.655564 (+847567us) op_driver.cc:464] REPLICATION: finished
0709 12:14:43.120997 (+465433us) write_op.cc:301] APPLY: starting
0709 12:14:43.139362 (+ 18365us) tablet.cc:1367] starting BulkCheckPresence
0709 12:14:43.765445 (+626083us) tablet.cc:1370] finished BulkCheckPresence
0709 12:14:43.765618 (+   173us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:14:43.914779 (+149161us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:14:43.990658 (+ 75879us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 12:14:44.000354 (+  9696us) write_op.cc:312] APPLY: finished
0709 12:14:44.005080 (+  4726us) log.cc:844] Serialized 81 byte log entry
0709 12:14:44.150443 (+145363us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:44.165375 (+ 14932us) write_op.cc:454] Released schema lock
0709 12:14:44.176679 (+ 11304us) write_op.cc:341] FINISH: Updating metrics
0709 12:14:44.197322 (+ 20643us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:44.197356 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":254618,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":384794,"apply.run_cpu_time_us":35515,"apply.run_wall_time_us":1110743,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37535,"num_ops":8,"prepare.queue_time_us":169204,"prepare.run_cpu_time_us":32797,"prepare.run_wall_time_us":737566,"raft.queue_time_us":225521,"raft.run_cpu_time_us":54050,"raft.run_wall_time_us":166680,"replication_time_us":919274,"thread_start_us":487706,"threads_started":5,"wal-append.queue_time_us":547711,"wal-append.run_cpu_time_us":25524,"wal-append.run_wall_time_us":2342410}]]}
I20260709 12:14:52.273023 19818 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.192701 20120 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.282925 20116 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:52.325177 12568 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 8. Preceding OpId from leader: term: 24 index: 11. (index mismatch)
I20260709 12:14:52.243366 20028 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.295351 19934 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:52.297778 20084 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:52.282732 20130 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:52.329282 12572 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 8. Preceding OpId from leader: term: 21 index: 8. (term mismatch)
I20260709 12:14:52.319324 20126 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:52.352965 12572 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 7
I20260709 12:14:52.334589 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:52.353385 12572 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 20.8
I20260709 12:14:52.312031 20036 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:52.333230 20138 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:52.318126 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2711) took 1183 ms. Trace:
I20260709 12:14:52.335137 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.353834 12572 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:52.320261 19957 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:52.321799 20124 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.324584 19818 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:14:52.326963 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 211136us
I20260709 12:14:52.361920 12489 rpcz_store.cc:276] 0709 12:14:51.135226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.225218 (+1089992us) service_pool.cc:224] Handling call
0709 12:14:52.225517 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.317531 (+ 92014us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.318053 (+   522us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:52.363349 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:52.368415 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:52.298584 20064 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.336246 19999 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:14:52.380901 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45894.3us
I20260709 12:14:52.286610 20043 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.380204 19997 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:52.382723 12536 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20.
I20260709 12:14:52.386080 12565 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:14:52.389137 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.388903 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.401281 13253 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:52.403009 13257 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:52.403512 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.361606 20138 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:52.402307 13258 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:52.408399 12489 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:52.377892 20115 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:52.356067 19934 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.391219 19997 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.420869 12496 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18.
I20260709 12:14:52.422164 20101 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:52.423894 20095 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.736s	user 0.005s	sys 0.000s
I20260709 12:14:52.424059 19754 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:14:52.424726 20133 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:52.425243 20112 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.104s
I20260709 12:14:52.425908 20089 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:52.427044 20063 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.188s	user 0.005s	sys 0.010s
W20260709 12:14:52.427237 20136 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:14:52.428911 20135 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:14:52.386389 19999 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.499902 19999 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:52.500000 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.500815 13261 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:14:52.385270 20080 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:52.502027 20080 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:52.503055 20080 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.503717 20130 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.422904 20115 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.504791 20115 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.411044 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.507405 20077 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:14:52.507617 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96680.2us
I20260709 12:14:52.508626 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:52.509503 20077 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.405359 20109 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:52.438212 19963 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:52.510921 19963 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:52.512419 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 960) took 1665 ms (client timeout 1829 ms). Trace:
W20260709 12:14:52.512897 12496 rpcz_store.cc:269] 0709 12:14:50.847253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.926282 (+ 79029us) service_pool.cc:224] Handling call
0709 12:14:52.512161 (+1585879us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.512347 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3430189824}
I20260709 12:14:52.440640 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2722) took 1177 ms. Trace:
W20260709 12:14:52.438591 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.360994 20084 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.514515 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.516045 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.516438 20084 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.516857 12496 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:52.429548 20031 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.115s	user 0.011s	sys 0.004s
I20260709 12:14:52.437085 19997 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.513978 12639 rpcz_store.cc:276] 0709 12:14:51.263216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.114293 (+851077us) service_pool.cc:224] Handling call
0709 12:14:52.153262 (+ 38969us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:52.153665 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:52.153696 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:52.181941 (+ 28245us) log.cc:844] Serialized 26 byte log entry
0709 12:14:52.295828 (+113887us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:52.295872 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.295949 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:52.440321 (+144372us) raft_consensus.cc:1685] finished
0709 12:14:52.440342 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.440517 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6784,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":9864,"thread_start_us":83844,"threads_started":1,"wal-append.queue_time_us":203444}
I20260709 12:14:52.403849 20127 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:52.453125 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3125) took 1123 ms. Trace:
I20260709 12:14:52.521740 19656 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:52.517441 20031 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.521999 12644 rpcz_store.cc:276] 0709 12:14:51.330070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.120158 (+790088us) service_pool.cc:224] Handling call
0709 12:14:52.120537 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:52.123564 (+  3027us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:52.123604 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:52.124183 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 12:14:52.274476 (+150293us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:52.274519 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.274588 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:52.452818 (+178230us) raft_consensus.cc:1685] finished
0709 12:14:52.452846 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.453043 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2513,"raft.queue_time_us":21587,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":5705,"thread_start_us":149893,"threads_started":1,"wal-append.queue_time_us":229189}
I20260709 12:14:52.523780 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.524178 20031 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:52.525123 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.450376 20088 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:52.525879 20088 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:52.526708 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.526849 20088 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.528137 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.529238 20084 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:52.530001 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.455881 20056 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.459394 12495 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:52.531528 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.495251 12388 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:52.532828 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3444) took 1585 ms. Trace:
I20260709 12:14:52.463390 19965 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.536295 19965 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.471032 20120 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:52.469254 20024 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.880s	user 0.013s	sys 0.001s
I20260709 12:14:52.475908 12493 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:14:52.411135 20138 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:52.537429 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.538241 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 17.
I20260709 12:14:52.497605 20089 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:52.538865 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3417) took 2315 ms (client timeout 1637 ms). Trace:
W20260709 12:14:52.539283 12493 rpcz_store.cc:269] 0709 12:14:50.223974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.224183 (+   209us) service_pool.cc:224] Handling call
0709 12:14:52.538692 (+2314509us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.538803 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5059690240}
I20260709 12:14:52.539721 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 966) took 1067 ms. Trace:
I20260709 12:14:52.540097 12489 rpcz_store.cc:276] 0709 12:14:51.471932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.538868 (+1066936us) service_pool.cc:224] Handling call
0709 12:14:52.539212 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.539387 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:52.539410 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:52.539448 (+    38us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:52.539487 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.539542 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.539642 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:52.480489 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.541836 20111 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:52.542508 20111 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.543398 20087 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.546057 20077 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:52.485275 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 950) took 1668 ms. Trace:
I20260709 12:14:52.494800 12497 raft_consensus.cc:606] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:52.499408 13255 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:14:52.499909 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 959) took 1653 ms. Trace:
I20260709 12:14:52.437718 20126 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.530849 20028 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.521497 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3138) took 1127 ms. Trace:
I20260709 12:14:52.601225 12639 rpcz_store.cc:276] 0709 12:14:51.394079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.519019 (+1124940us) service_pool.cc:224] Handling call
0709 12:14:52.519354 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.519548 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:52.519572 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:52.519613 (+    41us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:52.519644 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.519727 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.521183 (+  1456us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:52.537366 20024 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:52.547238 20026 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.639s	user 0.002s	sys 0.003s
W20260709 12:14:52.548756 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.611056 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.613888 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.480213 20064 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.549417 19818 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.552194 20124 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.545495 20111 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.554698 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.631053 13259 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:52.554872 19748 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:52.555171 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.556569 12489 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:14:52.561952 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 898) took 1160 ms. Trace:
I20260709 12:14:52.456792 19934 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.533246 12495 rpcz_store.cc:276] 0709 12:14:50.947195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.350061 (+402866us) service_pool.cc:224] Handling call
0709 12:14:51.380167 (+ 30106us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.531489 (+1151322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.532556 (+  1067us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:52.532759 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2481323264}
W20260709 12:14:52.481127 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.484272 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3436) took 2239 ms. Trace:
I20260709 12:14:52.646981 12490 rpcz_store.cc:276] 0709 12:14:50.245253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.348019 (+102766us) service_pool.cc:224] Handling call
0709 12:14:50.371094 (+ 23075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:50.693054 (+321960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:51.000737 (+307683us) spinlock_profiling.cc:243] Waited 7.22 ms on lock 0x7b5400295f20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6a02 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:52.482645 (+1481908us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b5400295f20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577041d 00007ffbf57f6a57 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:14:52.483918 (+  1273us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:52.484162 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":862035840}
I20260709 12:14:52.513381 19963 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:52.564090 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55570.2us
I20260709 12:14:52.648991 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:52.533530 12388 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:52.565744 20063 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.642921 13262 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
W20260709 12:14:52.654110 20057 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.009s	sys 0.005s
W20260709 12:14:52.654651 20057 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.009s	sys 0.005s
I20260709 12:14:52.569759 19868 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:14:52.571465 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 947) took 1839 ms (client timeout 1538 ms). Trace:
W20260709 12:14:52.573077 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43572.7us
I20260709 12:14:52.655583 19868 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:52.655684 12638 rpcz_store.cc:269] 0709 12:14:50.732508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.734022 (+  1514us) service_pool.cc:224] Handling call
0709 12:14:52.571253 (+1837231us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.571382 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3319838976}
I20260709 12:14:52.530853 12496 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:52.585516 20089 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.656785 19868 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.591902 13261 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:52.658082 19868 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:14:52.592489 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 237467us
I20260709 12:14:52.595252 12488 rpcz_store.cc:276] 0709 12:14:50.847011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.992388 (+145377us) service_pool.cc:224] Handling call
0709 12:14:50.992926 (+   538us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:51.023537 (+ 30611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.499644 (+1476107us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:52.499849 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64374528}
I20260709 12:14:52.593333 12637 rpcz_store.cc:276] 0709 12:14:50.817112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.817302 (+   190us) service_pool.cc:224] Handling call
0709 12:14:50.853218 (+ 35916us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:50.854350 (+  1132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.485049 (+1630699us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b54002c7250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.485206 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70843136}
I20260709 12:14:52.614763 12570 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.570482 19997 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.663560 19997 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.615772 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.611907 13255 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:52.666936 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.667773 12571 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:52.623669 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3105) took 3866 ms. Trace:
I20260709 12:14:52.626916 20024 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.565249 20138 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.643679 13254 rpcz_store.cc:276] 0709 12:14:51.401159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.403899 (+  2740us) service_pool.cc:224] Handling call
0709 12:14:52.561581 (+1157682us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.561839 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471934976}
I20260709 12:14:52.651373 12564 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:52.668493 12646 rpcz_store.cc:276] 0709 12:14:48.757614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:49.301103 (+543489us) service_pool.cc:224] Handling call
0709 12:14:49.321990 (+ 20887us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:49.335329 (+ 13339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:49.335379 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:49.377856 (+ 42477us) log.cc:844] Serialized 26 byte log entry
0709 12:14:49.897105 (+519249us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:49.980183 (+ 83078us) log.cc:844] Serialized 14 byte log entry
0709 12:14:49.995236 (+ 15053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:49.995342 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:50.907299 (+911957us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b180047ede0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:52.546523 (+1639224us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:52.623366 (+ 76843us) raft_consensus.cc:1685] finished
0709 12:14:52.623394 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.623567 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14725,"prepare.queue_time_us":85477,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":60,"raft.queue_time_us":23426,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":13164,"spinlock_wait_cycles":75911168,"thread_start_us":518717,"threads_started":1,"wal-append.queue_time_us":628527}
I20260709 12:14:52.591543 19664 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.661265 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:52.670678 12568 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:52.627629 12563 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:52.671715 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.672437 12565 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:52.608745 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3133) took 1225 ms. Trace:
I20260709 12:14:52.671437 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:52.676083 20109 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:14:52.676548 20109 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:52.676996 20109 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:52.681795 12570 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:52.681823 20024 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:52.685503 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:52.683148 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.673357 12645 rpcz_store.cc:276] 0709 12:14:51.383611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.209974 (+826363us) service_pool.cc:224] Handling call
0709 12:14:52.242506 (+ 32532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.242821 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:14:52.374578 (+131757us) log.cc:844] Serialized 14 byte log entry
0709 12:14:52.608124 (+233546us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:52.608240 (+   116us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:52.608313 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.608425 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.608649 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56210,"prepare.queue_time_us":215984,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":7959,"raft.queue_time_us":168,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":148,"thread_start_us":321577,"threads_started":2,"wal-append.queue_time_us":232832}
I20260709 12:14:52.689028 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.689565 20138 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.692379 19920 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.694442 19997 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:52.694191 19860 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:52.695053 20086 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:14:52.696043 13254 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:52.696856 19860 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:14:52.697413 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.698096 13261 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:52.684254 19999 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.683014 20077 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:52.704761 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.706743 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:52.707022 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3136) took 1322 ms. Trace:
I20260709 12:14:52.709846 12570 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.710436 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.697301 20101 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:52.738061 20101 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:52.739302 20101 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.740818 20101 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:52.743346 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.744102 12489 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.687215 20124 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:52.748648 19818 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.716507 20130 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.686820 20126 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:52.717216 12388 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:14:52.718703 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.706652 19963 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:52.694273 20063 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.696051 20086 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:14:52.809862 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.810758 13256 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:52.798604 19963 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:52.812335 19963 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.728541 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:52.819603 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 91151.2us
I20260709 12:14:52.728070 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.720804 20115 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.734550 20112 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:14:52.735858 13262 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:14:52.737234 20080 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:52.751699 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.695083 19997 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:52.769389 20126 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.749738 20064 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:14:52.708829 20077 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:52.713331 20138 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:52.791006 12569 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
W20260709 12:14:52.792618 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2703) took 1764 ms (client timeout 2155 ms). Trace:
W20260709 12:14:52.843127 12491 rpcz_store.cc:269] 0709 12:14:51.028560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.213323 (+184763us) service_pool.cc:224] Handling call
0709 12:14:52.792286 (+1578963us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.792538 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1625279232}
I20260709 12:14:52.828788 20084 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:14:52.787184 12570 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:52.829669 19963 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.846928 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.834276 20128 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:52.801014 12565 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:52.850284 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.840446 20126 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:52.808903 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.853350 13255 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:52.843895 20082 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:52.833742 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105743us
W20260709 12:14:52.818423 20123 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.873s	user 0.003s	sys 0.001s
W20260709 12:14:52.838348 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.835587 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:52.844511 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.793577 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:52.862030 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:52.831951 20063 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:14:52.847890 12570 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:52.796393 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 43) took 1462 ms. Trace:
I20260709 12:14:52.851460 12565 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:52.806293 20109 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.688108 20024 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.842718 20077 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:52.878260 12491 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18.
I20260709 12:14:52.862298 13261 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:14:52.875774 12497 rpcz_store.cc:276] 0709 12:14:51.348077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.385407 (+ 37330us) service_pool.cc:224] Handling call
0709 12:14:52.796138 (+1410731us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:52.796275 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1931824384}
I20260709 12:14:52.862426 12489 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:14:52.877372 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:52.885253 12491 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:14:52.886096 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:52.850628 19748 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:14:52.894605 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.895778 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.898033 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.898772 12570 raft_consensus.cc:2376] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21. Re-sending same reply.
I20260709 12:14:52.899471 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:14:52.900447 12489 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:14:52.840482 20080 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.900566 12565 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:52.903276 12489 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:52.904503 12491 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 22 index: 10. (term mismatch)
I20260709 12:14:52.905165 12491 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 9
I20260709 12:14:52.905408 12491 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 21.10
I20260709 12:14:52.905670 12491 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:52.899588 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:52.908532 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:52.896214 20130 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:52.909898 20130 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.910002 20060 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:52.910399 20130 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.910658 20060 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.911202 20060 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:52.912786 20115 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.882313 20082 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:52.919870 20080 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.920712 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3137) took 1535 ms. Trace:
I20260709 12:14:52.921087 12642 rpcz_store.cc:276] 0709 12:14:51.384966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.486194 (+1101228us) service_pool.cc:224] Handling call
0709 12:14:52.486624 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:52.487134 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:52.640970 (+153836us) log.cc:844] Serialized 14 byte log entry
0709 12:14:52.799017 (+158047us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:52.799533 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:14:52.799665 (+   132us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:52.802827 (+  3162us) log.cc:844] Serialized 14 byte log entry
0709 12:14:52.825878 (+ 23051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.826055 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:52.920491 (+ 94436us) raft_consensus.cc:1685] finished
0709 12:14:52.920511 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.920654 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100571,"prepare.queue_time_us":77529,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":110,"raft.queue_time_us":23532,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":32410,"spinlock_wait_cycles":10147712,"thread_start_us":176236,"threads_started":2,"wal-append.queue_time_us":211752}
I20260709 12:14:52.923002 20024 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:52.923718 19997 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.903574 19999 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.929423 20109 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:52.929857 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.929970 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.930938 12497 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:52.931198 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.931933 12491 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:52.932173 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.934172 20080 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:14:52.934648 20080 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:52.935029 20080 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:52.938479 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:14:52.893740 20063 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:52.939585 20063 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:52.940950 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:52.943023 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.943248 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.944041 12491 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:52.852478 20112 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:14:52.946141 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.948782 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.954180 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:52.959295 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:52.900640 19818 raft_consensus.cc:2764] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:14:52.949515 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.963186 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:52.951292 20024 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:14:52.965895 20130 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.968312 20130 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.966414 12570 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:52.979480 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:52.980438 13253 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:52.981642 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:52.985101 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.988806 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:52.989512 13257 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:52.990648 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.981069 20130 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:52.992178 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:52.992400 20130 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:52.992827 12496 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 16.
I20260709 12:14:52.994907 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3269) took 1906 ms. Trace:
I20260709 12:14:52.995205 20130 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:52.995321 12563 rpcz_store.cc:276] 0709 12:14:51.088668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.350196 (+261528us) service_pool.cc:224] Handling call
0709 12:14:51.378450 (+ 28254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.671981 (+1293531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.994674 (+322693us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:52.994846 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2872243072}
I20260709 12:14:52.999739 12497 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:14:53.000676 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:53.003670 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.004510 12489 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:53.005581 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:53.006911 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:53.048952 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85857.8us
I20260709 12:14:53.050181 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.050436 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:53.051338 12489 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:53.067423 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:53.067461 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.069789 12489 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:53.071146 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:53.115958 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44932.4us
I20260709 12:14:53.117141 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:53.119807 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:53.122148 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.123083 12496 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:53.125797 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.126506 12496 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:53.127614 12496 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:53.114454 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.065191 20024 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:53.137084 19883 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.124423 20127 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:53.145879 20024 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:53.138239 20070 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:53.138680 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:53.146485 20024 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:53.146077 19883 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.068533 19818 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:14:53.144711 20077 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.123682 20115 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:53.188156 19883 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.195060 12493 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:14:53.194629 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2710) took 2060 ms. Trace:
I20260709 12:14:53.201411 12570 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:53.130095 20060 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.205757 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3309) took 1536 ms. Trace:
I20260709 12:14:53.209858 12568 rpcz_store.cc:276] 0709 12:14:51.669069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.355263 (+686194us) service_pool.cc:224] Handling call
0709 12:14:52.355572 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.671084 (+315512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:53.205349 (+534265us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:53.205610 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":724257152}
I20260709 12:14:53.191403 19999 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:14:53.207682 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3264) took 2120 ms (client timeout 2042 ms). Trace:
I20260709 12:14:53.198480 12494 rpcz_store.cc:276] 0709 12:14:51.134066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.313354 (+179288us) service_pool.cc:224] Handling call
0709 12:14:53.194401 (+1881047us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:53.194562 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":585465600}
I20260709 12:14:53.212029 12568 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: [21.8-21.8]
W20260709 12:14:53.221154 12569 rpcz_store.cc:269] 0709 12:14:51.087332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.204652 (+117320us) service_pool.cc:224] Handling call
0709 12:14:53.204502 (+1999850us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:53.204716 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3577683840}
I20260709 12:14:53.226682 12568 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:14:53.227061 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88471.7us
I20260709 12:14:53.227968 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:53.202405 20115 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:53.233259 12571 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:14:53.240438 20063 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.241416 19999 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:53.242488 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:53.250159 20074 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:14:53.251120 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:53.254150 12492 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 7. Preceding OpId from leader: term: 21 index: 8. (index mismatch)
I20260709 12:14:53.254916 20033 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:53.255461 20033 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:53.255877 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3448) took 1191 ms. Trace:
I20260709 12:14:53.256388 20033 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.256618 12493 rpcz_store.cc:276] 0709 12:14:52.064095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.540102 (+476007us) service_pool.cc:224] Handling call
0709 12:14:53.194571 (+654469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:53.195611 (+  1040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:53.255534 (+ 59923us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b5400426650. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:53.255770 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1504333568}
I20260709 12:14:53.253455 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:53.262337 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:53.278544 12494 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260709 12:14:53.279196 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:53.280185 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:53.284075 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.285116 12491 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:53.287791 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.281915 20074 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:14:53.291651 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:53.234918 20091 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:14:53.286314 19999 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:53.298147 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:53.299789 12493 raft_consensus.cc:606] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:53.300334 12493 raft_consensus.cc:614] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:53.298591 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:53.304721 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:14:53.307631 19997 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:53.308596 20020 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:53.310753 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:53.312327 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:53.313143 12571 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:53.315698 19999 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:53.317960 20111 raft_consensus.cc:2764] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:14:53.319080 19997 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:53.320261 19997 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.322691 19748 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:14:53.319202 20082 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:53.288662 19934 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 12:14:53.292955 20127 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:53.288972 20030 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:14:53.323446 20020 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:14:53.341184 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 221452us
I20260709 12:14:53.323565 19748 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:14:53.301314 20074 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:53.340895 20130 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:53.320619 20054 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:53.361071 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.342697 20020 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:53.366657 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:53.369563 20104 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.762s	user 0.007s	sys 0.005s
I20260709 12:14:53.374918 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1008) took 2410 ms. Trace:
I20260709 12:14:53.317072 19999 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.375474 13260 rpcz_store.cc:276] 0709 12:14:50.964444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.964638 (+   194us) service_pool.cc:224] Handling call
0709 12:14:50.980617 (+ 15979us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:51.288431 (+307814us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:51.288486 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:51.305337 (+ 16851us) log.cc:844] Serialized 26 byte log entry
0709 12:14:51.519250 (+213913us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:14:51.519311 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:51.519412 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:53.368793 (+1849381us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:53.374442 (+  5649us) raft_consensus.cc:1685] finished
0709 12:14:53.374467 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:53.374820 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":361401,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":25977,"spinlock_wait_cycles":42670336,"thread_start_us":520436,"threads_started":2,"wal-append.queue_time_us":299691}
I20260709 12:14:53.351513 20077 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:53.392375 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:53.395458 12571 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:53.294055 20070 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:53.357587 20074 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:53.397930 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:53.399578 12493 raft_consensus.cc:606] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:53.400049 12493 raft_consensus.cc:614] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:14:53.397843 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52246.1us
I20260709 12:14:53.384196 19754 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:14:53.334522 20115 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.391566 19748 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:53.421303 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.402793 20060 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.430856 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:14:53.434245 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 150506us
I20260709 12:14:53.410383 19754 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:53.437687 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.435242 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:53.441166 12493 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [15.13-17.14]
W20260709 12:14:53.433063 20126 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.554s	user 0.016s	sys 0.006s
I20260709 12:14:53.443938 12571 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:14:53.457711 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:53.456087 20063 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:53.456187 19883 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:53.464811 19883 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.437747 19754 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:53.405921 20127 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.471753 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:53.420701 20020 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.484527 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:53.445469 20174 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:14:53.489568 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 935) took 3132 ms (client timeout 2175 ms). Trace:
W20260709 12:14:53.490097 12643 rpcz_store.cc:269] 0709 12:14:50.355061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:50.360582 (+  5521us) service_pool.cc:224] Handling call
0709 12:14:53.472086 (+3111504us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:53.486027 (+ 13941us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2957814656}
I20260709 12:14:53.453236 20074 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.480406 20033 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:14:53.499296 19883 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.481009 20133 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:53.500084 19748 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.500705 20133 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.502619 20024 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.473950 20091 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.505043 20091 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.515698 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.455154 20126 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.520721 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:53.509156 20033 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:53.524720 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:53.524628 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:53.526535 13257 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:14:53.511721 20179 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:53.511492 20174 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:53.456775 20070 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.533139 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.534523 12494 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:53.527671 19883 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.536024 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:53.537181 12568 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:53.536134 19883 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.527254 20133 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.528708 20179 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:53.544485 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:53.546540 12494 raft_consensus.cc:606] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:53.547019 12494 raft_consensus.cc:614] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:53.549000 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3272) took 2315 ms. Trace:
I20260709 12:14:53.551805 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:53.552441 12493 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:53.553412 19883 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.554811 20140 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.555382 20140 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.556850 12565 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:14:53.556922 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:53.560331 19883 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.561064 19883 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.561407 20174 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.547487 20179 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:53.565577 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:53.566045 19883 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:53.566098 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.567381 20091 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.568316 20033 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.8, Last appended by leader: 8, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:53.571296 20172 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:14:53.532300 19754 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.573158 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:53.573984 12563 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:53.574518 19754 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.576365 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:14:53.577152 12494 raft_consensus.cc:606] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:53.577755 12494 raft_consensus.cc:614] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:14:53.571333 12570 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:53.550048 12567 rpcz_store.cc:276] 0709 12:14:51.233377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.518884 (+285507us) service_pool.cc:224] Handling call
0709 12:14:51.562087 (+ 43203us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:51.686870 (+124783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:53.548704 (+1861834us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:53.548937 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266000768}
I20260709 12:14:53.589264 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [20.9-24.11]
I20260709 12:14:53.598249 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:53.598959 12568 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:14:53.601763 20080 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.666s	user 0.012s	sys 0.002s
W20260709 12:14:53.602305 20185 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:14:53.607156 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.609750 12568 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:14:53.611085 20064 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:53.611930 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:53.616197 12567 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
W20260709 12:14:53.620545 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3305) took 1967 ms (client timeout 1999 ms). Trace:
I20260709 12:14:53.620800 12572 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
W20260709 12:14:53.621194 12566 rpcz_store.cc:269] 0709 12:14:51.653539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.660927 (+  7388us) service_pool.cc:224] Handling call
0709 12:14:53.620270 (+1959343us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:53.620457 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238097152}
I20260709 12:14:53.630755 20133 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:14:53.591511 20091 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:14:53.634889 20091 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:53.635499 20091 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:53.602315 20080 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:53.640919 20080 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:53.644104 20037 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:53.650490 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.659175 12567 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:53.644977 20037 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:53.667995 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.668056 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:37834
I20260709 12:14:53.680348 13258 raft_consensus.cc:606] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:53.681399 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:53.682928 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:53.683791 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.675868 20111 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:14:53.686120 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:53.687615 12494 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:14:53.688069 12644 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:53.693039 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:53.695212 12644 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:14:53.696877 12489 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:53.636775 20091 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.698513 12493 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:14:53.655061 20048 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:53.699451 12642 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:53.699872 12566 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:14:53.695487 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.661187 20087 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.731015 12494 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:14:53.701093 12493 pending_rounds.cc:85] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 11
I20260709 12:14:53.733121 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.733126 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:53.729987 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:53.734473 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:53.733283 12494 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:53.735942 12493 pending_rounds.cc:107] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:14:53.736330 12493 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:53.733604 20086 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:14:53.737927 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:53.737593 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:53.730020 20084 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:53.739223 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:53.740721 12566 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:14:53.742283 12566 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:53.742693 12569 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:14:53.734525 12643 raft_consensus.cc:2376] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25. Re-sending same reply.
I20260709 12:14:53.744947 12489 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:14:53.744947 12566 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 12:14:53.749369 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131147us
I20260709 12:14:53.686743 20111 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:14:53.733930 20188 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.141s	user 0.000s	sys 0.002s
I20260709 12:14:53.700379 20101 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:53.684286 20073 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 12:14:52.708648 12641 rpcz_store.cc:276] 0709 12:14:51.384675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.276825 (+892150us) service_pool.cc:224] Handling call
0709 12:14:52.441523 (+164698us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:52.441826 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:52.633471 (+191645us) log.cc:844] Serialized 14 byte log entry
0709 12:14:52.706439 (+ 72968us) log.cc:844] Serialized 14 byte log entry
0709 12:14:52.706623 (+   184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:52.706666 (+    43us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:52.706720 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:52.706790 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:52.706917 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36778,"prepare.queue_time_us":120093,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":20367,"thread_start_us":147059,"threads_started":2}
I20260709 12:14:53.765949 20128 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:53.769723 20128 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:53.699518 19957 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:14:53.756538 20091 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:53.731076 19965 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
W20260709 12:14:53.798056 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:53.742453 20048 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.759105 19609 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:53.736258 20087 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:53.758672 19656 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:53.986337 20068 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:54.086691 20186 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.331s	user 0.000s	sys 0.001s
I20260709 12:14:53.764472 20101 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.069761 20111 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.094101 19965 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:53.796730 20137 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:54.095788 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 344051us
I20260709 12:14:54.098853 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2716) took 1152 ms. Trace:
I20260709 12:14:54.153805 12497 rpcz_store.cc:276] 0709 12:14:52.946215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.946405 (+   190us) service_pool.cc:224] Handling call
0709 12:14:54.098627 (+1152222us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:54.098782 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105876736}
I20260709 12:14:54.099800 12566 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:14:54.155378 12566 pending_rounds.cc:85] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:14:54.155887 12566 pending_rounds.cc:107] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:14:54.156234 12566 raft_consensus.cc:2889] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:54.098235 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3317) took 1342 ms. Trace:
I20260709 12:14:54.132112 12645 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
W20260709 12:14:54.157758 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55990.5us
I20260709 12:14:54.157209 12565 rpcz_store.cc:276] 0709 12:14:52.756289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.908971 (+152682us) service_pool.cc:224] Handling call
0709 12:14:54.097920 (+1188949us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:54.098151 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434724736}
I20260709 12:14:54.071673 19656 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.118520 19965 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:54.224946 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:54.217558 19656 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:54.220620 20111 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:54.230105 20057 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:54.230615 20057 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:54.231763 20057 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.233206 20057 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:54.236984 20037 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:54.168839 20137 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.237941 20198 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:54.238318 20037 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.239612 20137 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.239264 20037 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.240213 20137 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.240744 20137 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.241254 12565 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:14:54.238392 20198 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:54.242998 20198 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 21.8, Last appended by leader: 8, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.073529 19609 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.218518 20201 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:54.152035 20068 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:54.264464 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:54.265654 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:54.257829 20197 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:54.254712 20179 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.691s	user 0.014s	sys 0.000s
I20260709 12:14:54.282192 20198 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:54.274557 20037 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.298501 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3445) took 3351 ms. Trace:
I20260709 12:14:54.299069 12492 rpcz_store.cc:276] 0709 12:14:50.947525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:51.569380 (+621855us) service_pool.cc:224] Handling call
0709 12:14:51.586763 (+ 17383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:53.254566 (+1667803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:54.298134 (+1043568us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:54.298414 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3823206144}
W20260709 12:14:54.311501 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.003s	sys 0.016s
W20260709 12:14:54.312251 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.003s	sys 0.016s
I20260709 12:14:54.283321 20197 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.279937 20179 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.269840 20201 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:54.308539 20193 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:14:54.344161 12566 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 8. Preceding OpId from leader: term: 25 index: 9. (index mismatch)
W20260709 12:14:54.333105 20128 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.563s	user 0.010s	sys 0.001s
I20260709 12:14:54.352491 20197 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:54.342962 20086 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.033s	sys 0.068s
W20260709 12:14:54.358449 20086 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.033s	sys 0.069s
I20260709 12:14:54.357952 20128 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.361106 20128 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.343847 19868 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:54.362892 20195 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:14:54.363365 19609 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.363780 19963 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:14:54.305565 20037 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:14:54.365927 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3277) took 2297 ms. Trace:
I20260709 12:14:54.410979 12572 rpcz_store.cc:276] 0709 12:14:52.068568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.533617 (+465049us) service_pool.cc:224] Handling call
0709 12:14:53.620505 (+1086888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:53.621577 (+  1072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:54.365654 (+744077us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54002472d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:54.365853 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2470271744}
I20260709 12:14:54.289927 19965 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.368989 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3280) took 1909 ms. Trace:
I20260709 12:14:54.414984 12564 rpcz_store.cc:276] 0709 12:14:52.459872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.480204 (+ 20332us) service_pool.cc:224] Handling call
0709 12:14:54.368732 (+1888528us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:54.368920 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253340928}
I20260709 12:14:54.409582 19868 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.418776 19868 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.420840 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1259) took 1053 ms. Trace:
I20260709 12:14:54.421311 12567 rpcz_store.cc:276] 0709 12:14:53.367395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:53.674951 (+307556us) service_pool.cc:224] Handling call
0709 12:14:53.675483 (+   532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:53.676419 (+   936us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:53.676456 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:53.685009 (+  8553us) log.cc:844] Serialized 26 byte log entry
0709 12:14:54.070380 (+385371us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:54.070451 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:54.070552 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:54.420579 (+350027us) raft_consensus.cc:1685] finished
0709 12:14:54.420604 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:54.420773 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":154,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"thread_start_us":261805,"threads_started":1,"wal-append.queue_time_us":516505}
I20260709 12:14:54.372260 20043 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:54.382839 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:54.383121 20068 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.385236 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:54.425827 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.426808 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.427380 20138 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:14:54.427537 20084 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.688s	user 0.009s	sys 0.003s
I20260709 12:14:54.428166 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:54.428308 20084 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:54.429152 19748 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:14:54.429554 19748 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:54.429804 20029 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:14:54.394734 20137 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.342530 20087 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:54.346798 20048 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:54.431331 20176 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.943s	user 0.006s	sys 0.009s
I20260709 12:14:54.432029 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:54.388180 20201 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:54.404640 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:54.423688 20179 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:54.384352 20086 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.434198 20086 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.429958 19748 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:54.434962 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:54.436053 20091 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:54.436515 13218 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18.
I20260709 12:14:54.436631 20091 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:54.436925 13218 leader_election.cc:403] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:14:54.437670 20086 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.435079 20201 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:54.438925 20200 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 17. Current term is 18.
I20260709 12:14:54.438753 20201 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.434850 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.438051 20111 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:54.428211 20138 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:14:54.443408 20201 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.449832 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.445333 20200 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.450295 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:14:54.455191 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 286272us
I20260709 12:14:54.455227 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:54.457039 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:54.457930 12497 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:54.460636 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:54.459463 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:54.460497 20171 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:14:54.452924 20179 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:54.470455 20179 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:54.470906 20179 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:54.471289 20206 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:54.471838 20206 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.472002 19656 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.461099 20068 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:54.474130 20068 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:54.474479 19656 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:54.474565 20068 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:54.480006 20171 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:54.480506 20171 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:54.480898 20024 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.480743 20070 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:14:54.481400 20171 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.482154 20070 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:54.482595 20070 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:54.481011 20086 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:14:54.483597 20070 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.484104 20086 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:14:54.484570 20086 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:54.452207 20138 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:54.437783 20091 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.487006 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:54.509428 13257 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:54.459523 19868 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:14:54.458385 19957 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 12:14:54.515152 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:54.516219 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:54.517262 13253 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:14:54.517947 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:54.519598 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:54.462879 20029 raft_consensus.cc:2764] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:14:54.523357 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55939.6us
I20260709 12:14:54.484275 20179 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.528436 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.520782 13260 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:14:54.484731 20070 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:54.465346 20201 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:54.455500 20111 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:54.514688 20203 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:14:54.514021 20138 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:14:54.556713 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.538s (SENT)
I20260709 12:14:54.557236 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:54.547487 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93991.6us
I20260709 12:14:54.536186 19868 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:54.522225 20048 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:14:54.555085 20029 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:54.524387 20087 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:14:54.512727 20202 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:14:54.556521 20171 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:54.567472 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98713.9us
I20260709 12:14:54.535444 20194 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:14:54.461593 20200 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.582060 20201 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:54.572678 19957 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:14:54.575563 20206 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:54.583642 20048 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 20.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:14:54.710063 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141321us
I20260709 12:14:54.592125 20137 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:54.558737 20207 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:14:54.573668 20029 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:54.718070 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1257) took 1569 ms. Trace:
I20260709 12:14:54.718472 20029 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:54.707043 20179 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.718681 12563 rpcz_store.cc:276] 0709 12:14:53.148481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:53.574894 (+426413us) service_pool.cc:224] Handling call
0709 12:14:53.577784 (+  2890us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:53.578602 (+   818us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:53.578633 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:53.582674 (+  4041us) log.cc:844] Serialized 26 byte log entry
0709 12:14:54.312909 (+730235us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:54.312981 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:54.313305 (+   324us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:54.717841 (+404536us) raft_consensus.cc:1685] finished
0709 12:14:54.717865 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:54.718004 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":697073,"threads_started":1,"wal-append.queue_time_us":810723}
I20260709 12:14:54.710894 20194 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:14:54.737573 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 153261us
I20260709 12:14:54.723308 19664 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.719672 19957 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:54.730259 20207 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:14:54.724061 20137 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.738925 20194 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:54.743569 20217 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:54.776556 20126 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.709507 20201 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.740468 19748 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.786155 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:54.788743 12497 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:54.787258 20084 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.791488 20084 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:54.785730 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 260754us
I20260709 12:14:54.794303 20084 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.794566 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.795419 20084 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:14:54.795866 20084 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:54.796262 20084 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:14:54.786363 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:54.787070 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 258785us
I20260709 12:14:54.798961 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:54.800091 12497 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:54.802093 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:54.802942 12489 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:54.804257 20171 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:14:54.804855 20171 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:54.805891 20171 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.807245 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:54.808458 12494 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:54.812206 12489 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:14:54.811733 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:54.813155 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:54.811730 20084 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.723960 19868 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.818342 12492 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:14:54.770884 20217 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:54.820510 20217 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.821209 19748 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.763836 20137 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:54.826077 20068 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.772570 20223 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260709 12:14:54.784921 20207 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:54.762418 19957 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:54.784567 20087 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:54.919653 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:54.920375 12488 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:14:54.924363 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121073us
I20260709 12:14:54.829344 20211 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.103s
I20260709 12:14:54.935456 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:54.936445 20068 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.936681 19974 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:54.914726 20137 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:14:54.815203 20218 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.143s
W20260709 12:14:54.927508 20057 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.668s	user 0.013s	sys 0.001s
I20260709 12:14:54.784974 20221 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.148s
I20260709 12:14:54.919826 20217 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:54.925158 20111 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:54.938081 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 141683us
I20260709 12:14:54.918447 20194 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 24.11, Last appended by leader: 11, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.921151 20207 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:54.932564 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 146318us
W20260709 12:14:55.023458 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.049821 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:54.918507 19999 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.422s	user 0.005s	sys 0.054s
I20260709 12:14:55.020618 19748 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:54.981099 20224 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:54.927918 20087 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:54.938997 20225 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:55.058030 19999 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.561s	user 0.016s	sys 0.084s
I20260709 12:14:55.025547 20200 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:54.927214 20222 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.115s
I20260709 12:14:54.960263 20206 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.099s
I20260709 12:14:55.031836 20214 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:54.939366 20198 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.126s
I20260709 12:14:55.073603 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.056622 20084 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:55.063895 20194 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:14:55.069098 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 331082us
I20260709 12:14:55.070919 20231 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:14:55.094280 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44520.8us
I20260709 12:14:55.073787 20171 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.071818 20227 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:55.065326 19748 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:55.087940 20070 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.000s
I20260709 12:14:55.093425 12488 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:14:55.095916 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:55.082468 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.096946 20171 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.097066 20231 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.097254 20233 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:14:55.097977 20227 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.110481 20233 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:55.112068 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.112986 12644 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:14:55.100965 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:55.114104 12488 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:14:55.103580 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.105332 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.105978 20073 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:55.107614 20171 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.109774 20231 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.098323 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.117779 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:55.123641 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 384460us
I20260709 12:14:55.115360 12493 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:55.126155 20087 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:55.128347 19748 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.111325 19999 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:14:55.090586 20214 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:55.145903 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2728) took 2187 ms (client timeout 1618 ms). Trace:
W20260709 12:14:55.151039 12491 rpcz_store.cc:269] 0709 12:14:52.958032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.981300 (+ 23268us) service_pool.cc:224] Handling call
0709 12:14:55.145609 (+2164309us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:55.145806 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":695389056}
I20260709 12:14:55.133826 12563 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:14:55.152992 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 270690us
W20260709 12:14:55.144568 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.155117 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:55.152171 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:55.156466 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.103403 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:14:55.114972 20226 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:14:55.111176 20221 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:14:55.113791 20233 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:14:55.146010 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:14:55.158524 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.164741 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:55.163710 12494 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:14:55.161495 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
W20260709 12:14:55.159083 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.120353 20228 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.116389 20073 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:55.210305 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:55.212213 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:55.227902 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:55.227810 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:55.221355 12642 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:14:55.132412 20088 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:55.079149 20225 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.231302 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:55.248082 12569 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:55.244016 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70895.5us
I20260709 12:14:55.248489 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:55.251964 12568 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:55.243238 12491 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:55.158655 20200 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:55.178656 20138 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.011s
W20260709 12:14:55.183789 20170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.427s	user 0.000s	sys 0.024s
I20260709 12:14:55.243953 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:14:55.258392 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:55.172278 20091 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.265995 13256 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 18: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:55.171340 20219 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:55.129868 19664 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.103s
I20260709 12:14:55.210028 20128 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.159303 20223 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.103s
I20260709 12:14:55.259065 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.270670 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.136636 20029 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.285233 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.285742 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:55.286298 12493 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:14:55.282510 13262 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
W20260709 12:14:55.265746 20170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.513s	user 0.015s	sys 0.034s
I20260709 12:14:55.288550 12491 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:55.282783 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:55.257527 19957 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.261456 20137 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.159031 20235 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
I20260709 12:14:55.280347 20091 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.232041 20228 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.290266 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:55.397980 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.432844 12493 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
W20260709 12:14:55.317317 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.352504 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:55.443044 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:55.437129 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 333762us
I20260709 12:14:55.438263 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.262089 20239 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:14:55.449935 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.448194 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:14:55.448163 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.438321 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.284973 20219 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.457702 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:55.457973 13257 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:14:55.463316 20219 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:55.467568 13257 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:55.470270 12494 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:14:55.469013 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:55.311729 20225 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.270506 20138 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.353156 19748 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.488483 13254 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:55.268328 20200 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.353156 19963 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.485738 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.486357 12644 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:55.487233 12493 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:14:55.447062 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:14:55.451828 20207 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:55.318359 20231 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.493726 12639 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:55.289486 20128 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.353212 20236 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:55.439030 20033 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:55.491685 20222 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:14:55.548714 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:55.479702 20239 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:55.376379 20073 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.550088 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:55.585855 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53847us
I20260709 12:14:55.540287 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:55.472858 20198 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:14:55.578568 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.628263 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:55.445890 20137 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:55.628001 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87840.8us
I20260709 12:14:55.626751 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:55.635304 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:55.641769 12639 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:55.649827 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:55.595278 19974 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.658s	user 0.018s	sys 0.000s
I20260709 12:14:55.578305 20236 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:55.662530 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:55.523849 19963 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.622177 20222 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:55.667145 20236 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:55.699349 20091 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:55.700294 20214 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:55.502146 19748 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.708384 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:55.658178 20073 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:55.716403 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:55.741439 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57696.9us
I20260709 12:14:55.742357 12637 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:55.580476 20231 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:55.646063 19664 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:55.745549 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.751240 12566 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:14:55.758491 12564 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:55.666925 19974 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:14:55.776279 20245 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.198s	user 0.000s	sys 0.002s
I20260709 12:14:55.787493 12644 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:55.701895 20029 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:55.783710 19664 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:55.709832 20222 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:55.802026 20091 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:55.646654 20235 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.106s
W20260709 12:14:55.812273 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.636091 20033 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:55.821452 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.824157 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.702490 20256 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:14:55.826941 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.739481 20115 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:55.848531 12639 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:55.802198 19664 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:55.812486 20255 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:14:55.858572 20138 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.859110 20260 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:14:55.860349 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:55.861346 20200 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:55.716300 20254 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:14:55.924361 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.865252 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:55.986135 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 336382us
I20260709 12:14:55.991456 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:55.868264 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.995137 12644 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:14:55.996464 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:55.996920 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:55.998063 12637 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:14:55.999183 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.999997 12637 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:56.003938 12566 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:55.838253 20048 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:14:56.008239 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.014873 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:55.801476 20257 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:56.016042 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.017189 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:14:56.020941 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.022961 12566 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:55.909296 20088 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.653s	user 0.009s	sys 0.004s
I20260709 12:14:56.027737 13262 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: []
W20260709 12:14:56.034139 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:56.035796 12565 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:56.024544 20257 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:56.037575 20257 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:56.038373 20137 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:14:56.037595 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.044035 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.007730 19974 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:56.046567 19974 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.051720 19974 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.048874 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.054553 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.056867 13259 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:14:55.984633 19963 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:56.015201 20128 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:56.044340 20233 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.796s	user 0.011s	sys 0.005s
I20260709 12:14:55.816671 20223 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 9, Time since last communication: 0.220s
I20260709 12:14:56.063076 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:56.063649 12570 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:55.918126 20228 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:56.010876 20261 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:56.072588 12638 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:14:55.831254 20122 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.243s
I20260709 12:14:56.010098 20138 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:56.079883 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:55.869256 20115 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.083384 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:56.100759 13259 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:56.082650 12566 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24. Re-sending same reply.
I20260709 12:14:56.085172 12563 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:56.013662 20048 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:56.072705 20194 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.978s	user 0.017s	sys 0.000s
I20260709 12:14:56.081895 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.098138 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:56.102367 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.109895 12563 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:55.862103 20256 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.111296 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.109709 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.112335 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:14:56.106254 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:56.104887 20138 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:55.954505 20239 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.151439 20138 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.092626 20089 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:56.127008 20265 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.193s	user 0.001s	sys 0.001s
I20260709 12:14:56.092849 20261 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:56.128754 12640 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 24 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 8.
I20260709 12:14:56.129040 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.129364 12570 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:14:56.129956 12563 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:56.103900 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.155151 12565 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:14:56.085798 20233 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.162376 12644 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:14:56.234925 12637 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 21 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 7.
I20260709 12:14:56.236178 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.174185 12493 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [22.13-24.14]
I20260709 12:14:56.238291 20048 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.303578 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:56.231845 20261 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:56.249935 20033 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.212428 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3466) took 2531 ms. Trace:
I20260709 12:14:56.377194 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:56.379550 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:56.194787 20217 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.143s	user 0.016s	sys 0.000s
I20260709 12:14:56.379122 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.271695 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.390089 12566 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:56.391993 12563 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:14:56.308707 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:56.397790 12495 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:14:56.369575 13259 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:14:56.407138 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.408447 13255 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
W20260709 12:14:56.188931 20244 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.601s	user 0.002s	sys 0.002s
W20260709 12:14:56.271538 20180 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.773s	user 0.007s	sys 0.011s
I20260709 12:14:56.224910 20204 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:56.409005 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.176759 19748 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.319995 19609 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:56.411835 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3492) took 1200 ms. Trace:
I20260709 12:14:56.421924 12642 raft_consensus.cc:2376] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18. Re-sending same reply.
I20260709 12:14:56.442377 12495 rpcz_store.cc:276] 0709 12:14:55.211284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.397105 (+1185821us) service_pool.cc:224] Handling call
0709 12:14:56.397549 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:56.411571 (+ 14022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:56.411781 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:56.443300 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.443885 12495 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:14:56.444042 12642 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:56.444536 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3493) took 1174 ms. Trace:
I20260709 12:14:56.373023 20101 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.274273 20256 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.407738 20033 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:56.155733 19974 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:56.399322 12496 rpcz_store.cc:276] 0709 12:14:53.680695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:53.688572 (+  7877us) service_pool.cc:224] Handling call
0709 12:14:53.689003 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:53.689326 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:53.689745 (+   419us) log.cc:844] Serialized 14 byte log entry
0709 12:14:56.088725 (+2398980us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b1800066a60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:14:56.211008 (+122283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:56.211283 (+   275us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:56.211512 (+   229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:56.211624 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:56.212222 (+   598us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":225,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2355,"raft.queue_time_us":165,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":11305,"spinlock_wait_cycles":14084864,"thread_start_us":120500,"threads_started":1,"wal-append.queue_time_us":2565881}
W20260709 12:14:56.423516 20275 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
W20260709 12:14:56.423966 20277 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:14:56.409081 20122 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:56.424654 20274 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:14:56.425544 20140 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.414666 20115 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:14:56.427188 20273 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.150s	user 0.001s	sys 0.002s
I20260709 12:14:56.163985 20175 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.289s
I20260709 12:14:56.407153 20219 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:56.398152 20089 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:56.465346 19664 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.612s	user 0.016s	sys 0.002s
W20260709 12:14:56.434087 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:56.444768 12495 rpcz_store.cc:276] 0709 12:14:55.270091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.443331 (+1173240us) service_pool.cc:224] Handling call
0709 12:14:56.443660 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:56.444292 (+   632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:56.444484 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:56.444945 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.372427 20128 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:56.237671 20236 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.453776 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:56.424660 20270 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:14:56.431777 20091 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.629s	user 0.008s	sys 0.008s
I20260709 12:14:56.419893 20264 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:56.442191 19609 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:56.486764 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 960) took 2795 ms (client timeout 2080 ms). Trace:
I20260709 12:14:56.429975 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:56.498343 12637 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20. Re-sending same reply.
I20260709 12:14:56.498883 12495 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: [19.11-21.12]
I20260709 12:14:56.514169 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3313) took 1380 ms. Trace:
I20260709 12:14:56.462968 20033 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.429616 20257 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.614768 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.609318 12571 rpcz_store.cc:276] 0709 12:14:55.133347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.134009 (+   662us) service_pool.cc:224] Handling call
0709 12:14:55.134334 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.134733 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.134759 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.135274 (+   515us) log.cc:844] Serialized 26 byte log entry
0709 12:14:55.364203 (+228929us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:55.364272 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:55.364381 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:56.185854 (+821473us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:56.495503 (+309649us) raft_consensus.cc:1685] finished
0709 12:14:56.495551 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:56.514062 (+ 18511us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15857,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":10184,"spinlock_wait_cycles":85382016,"thread_start_us":228296,"threads_started":1,"wal-append.queue_time_us":369990}
I20260709 12:14:56.610178 13258 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:56.472131 20122 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:56.611359 12645 rpcz_store.cc:269] 0709 12:14:53.691718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:53.691903 (+   185us) service_pool.cc:224] Handling call
0709 12:14:56.486394 (+2794491us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:56.486637 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":914710272}
I20260709 12:14:56.612923 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.614048 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3495) took 1221 ms. Trace:
I20260709 12:14:56.626612 12495 rpcz_store.cc:276] 0709 12:14:55.392826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.498229 (+1105403us) service_pool.cc:224] Handling call
0709 12:14:56.498619 (+   390us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:56.609208 (+110589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:56.613981 (+  4773us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:56.615660 12642 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:14:56.579938 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:56.176237 20086 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.493546 20233 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.630281 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:56.630226 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.498762 19664 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:56.560243 20222 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.758s	user 0.012s	sys 0.009s
W20260709 12:14:56.567997 20274 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
W20260709 12:14:56.640321 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60534.9us
I20260709 12:14:56.647421 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.648674 12642 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:56.498445 20219 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:56.649842 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.609154 20223 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:56.657836 20073 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.874s	user 0.018s	sys 0.000s
I20260709 12:14:56.658268 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:56.532207 20128 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.659216 20033 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.453105 20204 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.642618 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.660169 13256 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:56.612288 20140 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:56.647583 19609 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.614944 20264 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.576293 20238 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:56.630620 20091 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:56.663494 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:56.664799 12637 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:56.666540 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.667698 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:56.648044 20256 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:14:56.673481 19609 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:56.690164 20111 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:56.690694 20111 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:56.690785 20128 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.691177 20264 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.664237 20238 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:56.691583 20111 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.699523 20111 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:56.703092 20257 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.657966 20223 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.707701 20223 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.709169 20089 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:56.710117 20020 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:56.711248 20231 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.714619 20221 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:14:56.715405 20221 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:56.660513 20222 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.664633 20140 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.732115 19965 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:56.738035 20219 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.717029 20221 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:56.784271 20222 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.789414 20140 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.742229 20122 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.797513 20221 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:56.802287 20236 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:56.803368 20238 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.809146 20261 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:56.824421 20204 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.617398 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:57.008894 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 392029us
W20260709 12:14:56.900527 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:56.903818 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3324) took 1229 ms. Trace:
I20260709 12:14:57.014964 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:56.906260 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 1.73036e+06us
W20260709 12:14:56.907521 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:57.018985 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:56.915069 12646 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:56.899854 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3321) took 1240 ms. Trace:
I20260709 12:14:57.022955 20239 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.021833 12569 rpcz_store.cc:276] 0709 12:14:55.658449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.658705 (+   256us) service_pool.cc:224] Handling call
0709 12:14:55.659203 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.659756 (+   553us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:14:55.830246 (+170490us) log.cc:844] Serialized 14 byte log entry
0709 12:14:56.089111 (+258865us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:56.100046 (+ 10935us) log.cc:844] Serialized 26 byte log entry
0709 12:14:56.100620 (+   574us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:14:56.100699 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:56.100879 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:56.897969 (+797090us) raft_consensus.cc:1685] finished
0709 12:14:56.898049 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:56.898627 (+   578us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":262274,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":7276,"raft.run_cpu_time_us":705,"raft.run_wall_time_us":59311,"thread_start_us":276244,"threads_started":2,"wal-append.queue_time_us":329017}
I20260709 12:14:56.835669 20261 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:56.912405 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.965992 12566 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
W20260709 12:14:56.924832 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:56.924582 12571 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:14:56.957440 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:56.895720 12494 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:57.010314 20204 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:14:56.960564 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 906) took 1729 ms (client timeout 1865 ms). Trace:
I20260709 12:14:57.036392 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.042305 20264 raft_consensus.cc:2764] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:14:57.041940 12637 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:56.979574 19664 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:57.043098 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 911) took 1758 ms. Trace:
I20260709 12:14:56.927436 19957 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:57.036443 13255 rpcz_store.cc:269] 0709 12:14:55.230667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.230983 (+   316us) service_pool.cc:224] Handling call
0709 12:14:56.959151 (+1728168us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:56.960042 (+   891us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2660516224}
I20260709 12:14:57.054510 13262 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:14:57.055322 13257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 905) took 1828 ms (client timeout 1980 ms). Trace:
I20260709 12:14:57.056228 13259 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:14:57.055924 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.056598 20231 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:14:57.048280 13254 rpcz_store.cc:276] 0709 12:14:55.284824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.285471 (+   647us) service_pool.cc:224] Handling call
0709 12:14:57.042349 (+1756878us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.042864 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122727168}
I20260709 12:14:57.057700 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:57.057560 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3144) took 1572 ms (client timeout 1506 ms). Trace:
I20260709 12:14:57.065157 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:57.056023 13257 rpcz_store.cc:269] 0709 12:14:55.227070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.227643 (+   573us) service_pool.cc:224] Handling call
0709 12:14:57.054714 (+1827071us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.055075 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527303936}
I20260709 12:14:57.065301 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3333) took 1031 ms. Trace:
I20260709 12:14:57.024547 20239 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.075668 12637 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:14:57.063868 12640 rpcz_store.cc:269] 0709 12:14:55.485150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.485294 (+   144us) service_pool.cc:224] Handling call
0709 12:14:57.056822 (+1571528us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.057191 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1444863232}
I20260709 12:14:57.014036 12572 rpcz_store.cc:276] 0709 12:14:55.674747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.674926 (+   179us) service_pool.cc:224] Handling call
0709 12:14:55.741372 (+ 66446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.742102 (+   730us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.742131 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.745548 (+  3417us) log.cc:844] Serialized 26 byte log entry
0709 12:14:56.077756 (+332208us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:14:56.077830 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:56.077943 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:56.902851 (+824908us) raft_consensus.cc:1685] finished
0709 12:14:56.902919 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:56.903575 (+   656us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9239,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":21198,"thread_start_us":330998,"threads_started":1,"wal-append.queue_time_us":476519}
I20260709 12:14:57.069610 12565 rpcz_store.cc:276] 0709 12:14:56.007168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.046179 (+ 39011us) service_pool.cc:224] Handling call
0709 12:14:57.033331 (+987152us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54002477d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.038737 (+  5406us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233491328}
I20260709 12:14:57.084308 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.086529 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:57.071520 20231 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:14:57.090914 20231 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:57.095613 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.098192 12572 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.103370 20227 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:57.097263 20231 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.106678 20133 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:14:57.108633 12569 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:57.090880 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.109900 20126 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:14:57.108556 20133 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:14:57.110495 20126 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:57.112114 20140 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:57.113101 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.114454 12569 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
W20260709 12:14:57.116051 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.122254 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.125131 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.112365 12645 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:14:57.114645 20231 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:57.112322 20126 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.139431 20126 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:57.146981 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.160336 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:57.158715 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 99849.6us
I20260709 12:14:57.160233 19957 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.161073 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3151) took 1670 ms. Trace:
I20260709 12:14:57.182387 12640 rpcz_store.cc:276] 0709 12:14:55.490436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.124436 (+1634000us) service_pool.cc:224] Handling call
0709 12:14:57.160971 (+ 36535us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.172917 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:57.121311 20246 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.531s	user 0.005s	sys 0.000s
I20260709 12:14:57.201030 20239 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:57.207861 20252 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.327s	user 0.004s	sys 0.000s
W20260709 12:14:57.218786 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50708.4us
I20260709 12:14:57.210063 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3494) took 1711 ms. Trace:
I20260709 12:14:57.211346 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:57.218887 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3315) took 2084 ms. Trace:
I20260709 12:14:57.235334 12572 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:57.233172 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.244822 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.247716 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.220892 12496 rpcz_store.cc:276] 0709 12:14:55.352637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.457426 (+1104789us) service_pool.cc:224] Handling call
0709 12:14:56.458425 (+   999us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:56.467512 (+  9087us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:56.467559 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:56.490944 (+ 23385us) log.cc:844] Serialized 52 byte log entry
0709 12:14:57.015468 (+524524us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:57.015801 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.016015 (+   214us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:57.050152 (+ 34137us) raft_consensus.cc:1685] finished
0709 12:14:57.050182 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.064067 (+ 13885us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7781,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":20690,"thread_start_us":396398,"threads_started":1,"wal-append.queue_time_us":412879}
I20260709 12:14:57.249521 12640 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 7.
I20260709 12:14:57.253207 12569 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:57.267935 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:57.271212 20020 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.560s	user 0.011s	sys 0.029s
I20260709 12:14:57.270964 20140 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:14:57.275362 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.224823 12567 rpcz_store.cc:276] 0709 12:14:55.133944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.137010 (+  3066us) service_pool.cc:224] Handling call
0709 12:14:55.137412 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.272713 (+135301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.272755 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.273369 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:14:55.673827 (+400458us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:55.673894 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.144374 (+1470480us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:57.144493 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:57.145227 (+   734us) raft_consensus.cc:1685] finished
0709 12:14:57.145249 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.218667 (+ 73418us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4230,"raft.queue_time_us":936,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5232,"spinlock_wait_cycles":309974272,"thread_start_us":316462,"threads_started":1,"wal-append.queue_time_us":627433}
I20260709 12:14:57.277324 20090 raft_consensus.cc:993] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:14:57.277974 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:57.283593 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72618.7us
I20260709 12:14:57.288391 20261 raft_consensus.cc:1081] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:14:57.289165 12640 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:57.294006 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 963) took 2122 ms. Trace:
I20260709 12:14:57.294687 19868 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.295830 19868 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.298274 20260 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:57.298889 20260 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:57.299320 20260 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:57.299985 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:14:57.294622 12643 rpcz_store.cc:276] 0709 12:14:55.101373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.101532 (+   159us) service_pool.cc:224] Handling call
0709 12:14:55.118956 (+ 17424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.264526 (+145570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.264557 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.265040 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 12:14:55.475200 (+210160us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:55.503575 (+ 28375us) log.cc:844] Serialized 14 byte log entry
0709 12:14:55.503815 (+   240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:55.503890 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:56.700146 (+1196256us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b180048b4a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:57.124063 (+423917us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:57.219556 (+ 95493us) raft_consensus.cc:1685] finished
0709 12:14:57.219587 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.224136 (+  4549us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":136229,"prepare.queue_time_us":30391,"prepare.run_cpu_time_us":1833,"prepare.run_wall_time_us":30299,"raft.queue_time_us":138693,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":8367,"spinlock_wait_cycles":119463168,"thread_start_us":209709,"threads_started":1,"wal-append.queue_time_us":372716}
I20260709 12:14:57.304621 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.305939 20140 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:14:57.306519 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3478) took 2239 ms (client timeout 1762 ms). Trace:
W20260709 12:14:57.307319 12490 rpcz_store.cc:269] 0709 12:14:55.066873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.117410 (+ 50537us) service_pool.cc:224] Handling call
0709 12:14:57.306025 (+2188615us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.306353 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5016457088}
I20260709 12:14:57.308048 20140 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:57.312490 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.319823 20048 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:14:57.282335 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:57.322598 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 102286us
I20260709 12:14:57.323856 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:57.326313 20020 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:57.327028 20020 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:57.327589 20020 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.330557 20112 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:14:57.334789 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3489) took 2216 ms. Trace:
I20260709 12:14:57.314811 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.342715 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.337898 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.341485 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.343667 12637 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:57.344759 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.339946 20112 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:57.346937 20112 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.341423 12493 rpcz_store.cc:276] 0709 12:14:55.118144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.541871 (+423727us) service_pool.cc:224] Handling call
0709 12:14:55.625868 (+ 83997us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:56.237367 (+611499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.333979 (+1096612us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:57.334487 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1258283392}
I20260709 12:14:57.350287 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1028) took 1346 ms. Trace:
W20260709 12:14:57.347631 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.350991 13262 rpcz_store.cc:276] 0709 12:14:56.004007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.102302 (+ 98295us) service_pool.cc:224] Handling call
0709 12:14:57.050828 (+948526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.055206 (+  4378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.349600 (+294394us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b540019d2d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.349872 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2164276608}
I20260709 12:14:57.348752 20020 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:57.353767 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64582.5us
W20260709 12:14:57.354892 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.345566 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3142) took 1862 ms. Trace:
I20260709 12:14:57.357903 12638 rpcz_store.cc:276] 0709 12:14:55.482792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.482960 (+   168us) service_pool.cc:224] Handling call
0709 12:14:55.483394 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:56.089331 (+605937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.344461 (+1255130us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:57.345314 (+   853us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1329835264}
I20260709 12:14:57.359872 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.361171 12638 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:57.371497 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:57.373301 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.243s (SENT) [suppressed 30 similar messages]
I20260709 12:14:57.376230 20112 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:57.377018 12642 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.020s
I20260709 12:14:57.377892 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:14:57.362229 20133 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:14:57.342227 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:57.365921 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.369918 12643 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:57.378504 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:57.381186 12461 connection.cc:256] RPC call timeout handler was delayed by -1.2814867020001657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
I20260709 12:14:57.373335 12638 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 24 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 8.
I20260709 12:14:57.361752 20198 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:57.387972 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:57.393321 20271 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.143s	user 0.003s	sys 0.002s
I20260709 12:14:57.393365 20198 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.395341 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.401546 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.355748 20020 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:57.398263 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3153) took 1897 ms. Trace:
I20260709 12:14:57.402611 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.403496 12638 rpcz_store.cc:276] 0709 12:14:55.501043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.370968 (+1869925us) service_pool.cc:224] Handling call
0709 12:14:57.398177 (+ 27209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.404606 12492 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:14:57.404467 12563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:14:57.408644 20238 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:57.402606 12645 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:14:57.410095 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3490) took 2250 ms. Trace:
I20260709 12:14:57.400429 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:14:57.415004 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 147535us
I20260709 12:14:57.421099 12640 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.413239 12494 rpcz_store.cc:276] 0709 12:14:55.159825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.914314 (+754489us) service_pool.cc:224] Handling call
0709 12:14:55.915245 (+   931us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.052769 (+1137524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.403066 (+350297us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:57.410002 (+  6936us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2252111616}
I20260709 12:14:57.427479 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.429847 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.436491 12639 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
W20260709 12:14:57.445896 12461 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.506s (SENT)
I20260709 12:14:57.448217 13257 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:57.448850 19957 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:57.445899 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.450385 12645 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
W20260709 12:14:57.450277 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 126656us
I20260709 12:14:57.451052 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.451261 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.451762 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.452193 12637 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:14:57.452652 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2732) took 1023 ms. Trace:
I20260709 12:14:57.453236 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.457412 12569 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
W20260709 12:14:57.462157 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51654.3us
I20260709 12:14:57.458053 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 907) took 2203 ms. Trace:
I20260709 12:14:57.453006 12637 rpcz_store.cc:276] 0709 12:14:56.429400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.400123 (+970723us) service_pool.cc:224] Handling call
0709 12:14:57.452597 (+ 52474us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:57.457234 20221 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.646s	user 0.017s	sys 0.014s
W20260709 12:14:57.466254 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.463632 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.455790 20280 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.487118 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:57.493669 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.502358 20280 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.506174 12492 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:57.512940 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.514403 12570 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:57.499749 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67230.9us
W20260709 12:14:57.517544 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.520004 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.472301 13261 rpcz_store.cc:276] 0709 12:14:55.254607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.254826 (+   219us) service_pool.cc:224] Handling call
0709 12:14:55.255295 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.642710 (+387415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.642751 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.643546 (+   795us) log.cc:844] Serialized 26 byte log entry
0709 12:14:55.958667 (+315121us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:14:55.958735 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:55.958962 (+   227us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:57.380519 (+1421557us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:14:57.457431 (+ 76912us) raft_consensus.cc:1685] finished
0709 12:14:57.457465 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.457786 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10866,"raft.queue_time_us":208383,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":11724672,"thread_start_us":512169,"threads_started":2,"wal-append.queue_time_us":434918}
I20260709 12:14:57.527055 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:57.527994 12642 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:14:57.533665 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.536510 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.536893 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.538861 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:57.539304 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.545322 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.548159 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.544749 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:57.552726 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.551476 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.553764 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2737) took 1076 ms. Trace:
I20260709 12:14:57.557796 12643 rpcz_store.cc:276] 0709 12:14:56.477047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.501454 (+1024407us) service_pool.cc:224] Handling call
0709 12:14:57.553668 (+ 52214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:57.565475 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44099.4us
I20260709 12:14:57.544341 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.491422 20020 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:14:57.583254 12638 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:14:57.590044 20048 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.564311 20280 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:57.598433 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.610191 12569 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 17: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:57.624369 12640 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
W20260709 12:14:57.629068 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 180690us
I20260709 12:14:57.628934 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3156) took 2043 ms. Trace:
I20260709 12:14:57.630419 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:57.630025 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0566028570001436s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.36850000000000005s for a response
I20260709 12:14:57.621014 12488 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:14:57.633944 12643 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:14:57.634999 12642 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:14:57.630424 12640 rpcz_store.cc:276] 0709 12:14:55.582766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.554449 (+1971683us) service_pool.cc:224] Handling call
0709 12:14:57.619094 (+ 64645us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.625532 (+  6438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.625864 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.636917 12638 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:14:57.637907 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2730) took 1209 ms. Trace:
I20260709 12:14:57.639966 12642 rpcz_store.cc:276] 0709 12:14:56.428733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.564126 (+1135393us) service_pool.cc:224] Handling call
0709 12:14:57.637831 (+ 73705us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.640895 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:57.637837 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 264601us
I20260709 12:14:57.637897 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3161) took 1996 ms. Trace:
I20260709 12:14:57.648120 12643 rpcz_store.cc:276] 0709 12:14:55.641241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.569207 (+1927966us) service_pool.cc:224] Handling call
0709 12:14:57.618039 (+ 48832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.634406 (+ 16367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.637823 (+  3417us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:57.646595 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7105534329998591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2644s for a response
I20260709 12:14:57.652328 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3162) took 1977 ms. Trace:
W20260709 12:14:57.652694 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:57.653475 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.653102 12638 rpcz_store.cc:276] 0709 12:14:55.665484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.592211 (+1926727us) service_pool.cc:224] Handling call
0709 12:14:57.628352 (+ 36141us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:57.642570 (+ 14218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.642867 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:14:57.651870 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7057733469998766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
I20260709 12:14:57.653838 19965 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:14:57.658164 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5503534329998274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2822s for a response
I20260709 12:14:57.658294 20280 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:57.658213 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:57.659571 20280 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.661369 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.662593 12569 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:57.658831 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:57.665619 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5089534329997605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2868s for a response
W20260709 12:14:57.666170 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3691733469997871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2497s for a response
W20260709 12:14:57.666518 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2407534329997816s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3166s for a response
W20260709 12:14:57.667577 12536 connection.cc:256] RPC call timeout handler was delayed by -1.056253432999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3371s for a response
I20260709 12:14:57.668634 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:57.669781 12489 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:14:57.671494 20088 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.671633 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.672752 12638 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:57.658299 19965 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.676728 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.672356 20088 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.604863 20048 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.678624 12645 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:14:57.678153 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.686223 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.690335 12644 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:14:57.692580 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.692682 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:57.688540 12493 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:14:57.701077 12637 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:14:57.703015 20268 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:14:57.703655 20268 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:57.704036 12638 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:14:57.705086 20088 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:57.700628 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.705854 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:57.706413 12489 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:14:57.711711 12564 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:57.711268 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.714193 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3317) took 2277 ms. Trace:
I20260709 12:14:57.704860 20268 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.714864 12568 rpcz_store.cc:276] 0709 12:14:55.436823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.454410 (+ 17587us) service_pool.cc:224] Handling call
0709 12:14:55.455258 (+   848us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.497856 (+ 42598us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.497885 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.526899 (+ 29014us) log.cc:844] Serialized 26 byte log entry
0709 12:14:57.337472 (+1810573us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b18003f2c20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:14:57.657158 (+319686us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:14:57.678541 (+ 21383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.689851 (+ 11310us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:57.689910 (+    59us) raft_consensus.cc:1685] finished
0709 12:14:57.689932 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.713926 (+ 23994us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5262,"raft.queue_time_us":5426,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":37247,"spinlock_wait_cycles":12818688,"thread_start_us":260752,"threads_started":1,"wal-append.queue_time_us":2130579}
I20260709 12:14:57.714643 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:57.717993 12571 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.716625 20268 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:57.719581 20048 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:57.725381 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:57.726007 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:57.726415 13261 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:14:57.727514 20260 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.729034 20086 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:14:57.731053 20280 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:57.731781 20280 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:57.732375 20280 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.732904 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3482) took 2662 ms. Trace:
I20260709 12:14:57.734876 19965 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:57.736379 12643 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:57.737758 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.739547 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.740289 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:57.741575 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.339717 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:14:57.738541 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:57.739737 12643 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:14:57.725909 12642 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 11. Preceding OpId from leader: term: 25 index: 9. (term mismatch)
W20260709 12:14:57.744546 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 345683us
I20260709 12:14:57.744877 12642 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 8
W20260709 12:14:57.746537 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.740343 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.704921 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3163) took 2008 ms. Trace:
I20260709 12:14:57.755201 12638 rpcz_store.cc:276] 0709 12:14:55.696864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.677509 (+1980645us) service_pool.cc:224] Handling call
0709 12:14:57.703589 (+ 26080us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.704525 (+   936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.704823 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.745267 12643 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
W20260709 12:14:57.762176 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.746093 12642 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 20.9
I20260709 12:14:57.765584 12568 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:57.765702 12643 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:14:57.765938 12642 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:57.768319 12642 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 22.10
I20260709 12:14:57.768596 12642 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:57.768956 12642 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 24.11
I20260709 12:14:57.769212 12642 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:14:57.769922 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.767436 12643 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:57.760481 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3154) took 2212 ms. Trace:
I20260709 12:14:57.772455 12637 rpcz_store.cc:276] 0709 12:14:55.532475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.567164 (+2034689us) service_pool.cc:224] Handling call
0709 12:14:57.632963 (+ 65799us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.701759 (+ 68796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.744342 (+ 42583us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.770696 12493 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 18.
I20260709 12:14:57.776572 12640 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:14:57.777591 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3166) took 2051 ms. Trace:
I20260709 12:14:57.777976 12640 rpcz_store.cc:276] 0709 12:14:55.726907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.697570 (+1970663us) service_pool.cc:224] Handling call
0709 12:14:57.767240 (+ 69670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.777241 (+ 10001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.777519 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.782898 20198 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:57.783813 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.784883 12497 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:14:57.786360 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:57.786649 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3168) took 1996 ms. Trace:
I20260709 12:14:57.783879 20179 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:57.788163 20179 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:57.788575 20179 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:57.790274 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3164) took 2092 ms. Trace:
I20260709 12:14:57.790843 12642 rpcz_store.cc:276] 0709 12:14:55.697465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.684855 (+1987390us) service_pool.cc:224] Handling call
0709 12:14:57.725437 (+ 40582us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.789983 (+ 64546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.790198 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.792764 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:14:57.794567 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:57.795485 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3157) took 2206 ms (client timeout 2446 ms). Trace:
W20260709 12:14:57.795953 12639 rpcz_store.cc:269] 0709 12:14:55.588901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.014525 (+425624us) service_pool.cc:224] Handling call
0709 12:14:57.795275 (+1780750us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.795417 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3239930240}
I20260709 12:14:57.797834 20211 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.798943 20211 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.800211 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:57.801072 12488 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:57.787181 12643 rpcz_store.cc:276] 0709 12:14:55.790377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.738817 (+1948440us) service_pool.cc:224] Handling call
0709 12:14:57.739233 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.786319 (+ 47086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.786578 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.803588 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:57.804411 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3176) took 1692 ms (client timeout 1531 ms). Trace:
W20260709 12:14:57.795485 12609 connection.cc:256] RPC call timeout handler was delayed by -1.5201133499999742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
W20260709 12:14:57.804950 12644 rpcz_store.cc:269] 0709 12:14:56.112150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.235846 (+123696us) service_pool.cc:224] Handling call
0709 12:14:57.804124 (+1568278us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.804326 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2946300800}
I20260709 12:14:57.804463 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:57.810170 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:57.810595 12497 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:57.811152 13262 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:57.812016 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.813112 12567 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:14:57.798029 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3175) took 1711 ms. Trace:
W20260709 12:14:57.809844 12461 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.531s (SENT)
W20260709 12:14:57.813009 12609 connection.cc:256] RPC call timeout handler was delayed by -1.3195268500001021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
W20260709 12:14:57.815244 12609 connection.cc:256] RPC call timeout handler was delayed by -1.1855212780001239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
I20260709 12:14:57.804409 13255 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:57.804816 12643 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
W20260709 12:14:57.815897 12609 connection.cc:256] RPC call timeout handler was delayed by -1.1331133500000306s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20190000000000002s for a response
I20260709 12:14:57.816356 12643 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 14
W20260709 12:14:57.816577 12609 connection.cc:256] RPC call timeout handler was delayed by -1.1062268500002119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260709 12:14:57.816805 12643 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 20.15
W20260709 12:14:57.817090 12609 connection.cc:256] RPC call timeout handler was delayed by -1.0062066039999991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19970000000000002s for a response
I20260709 12:14:57.824481 12638 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:14:57.815986 12642 rpcz_store.cc:276] 0709 12:14:56.086831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.791968 (+1705137us) service_pool.cc:224] Handling call
0709 12:14:57.792538 (+   570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.792847 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:57.792885 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:57.792963 (+    78us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:57.793036 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.793113 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.797938 (+  4825us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.817179 12643 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:57.733415 12491 rpcz_store.cc:276] 0709 12:14:55.070520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.460677 (+390157us) service_pool.cc:224] Handling call
0709 12:14:55.460990 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:55.855552 (+394562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:55.855587 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:55.856105 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:14:56.062089 (+205984us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:56.062143 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.730694 (+1668551us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:57.730778 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:57.730851 (+    73us) raft_consensus.cc:1685] finished
0709 12:14:57.730865 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.732629 (+  1764us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6646,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":20572,"spinlock_wait_cycles":877254016,"thread_start_us":205117,"threads_started":1,"wal-append.queue_time_us":352124,"wal-append.run_cpu_time_us":6963,"wal-append.run_wall_time_us":1386548}
W20260709 12:14:57.822088 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.845203 13261 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:14:57.847205 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3170) took 1990 ms. Trace:
I20260709 12:14:57.862789 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3139) took 2381 ms. Trace:
I20260709 12:14:57.862311 12638 rpcz_store.cc:276] 0709 12:14:55.856874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.756281 (+1899407us) service_pool.cc:224] Handling call
0709 12:14:57.824015 (+ 67734us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.826974 (+  2959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.847098 (+ 20124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.866285 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3327) took 1863 ms. Trace:
I20260709 12:14:57.866174 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.867623 12637 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.863463 12646 rpcz_store.cc:276] 0709 12:14:55.481300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.481537 (+   237us) service_pool.cc:224] Handling call
0709 12:14:56.488103 (+1006566us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.021255 (+533152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.794105 (+772850us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:14:57.862408 (+ 68303us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b5400279050. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.862669 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3257794688}
I20260709 12:14:57.866813 12564 rpcz_store.cc:276] 0709 12:14:56.002517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.002679 (+   162us) service_pool.cc:224] Handling call
0709 12:14:57.711251 (+1708572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.712308 (+  1057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.866078 (+153770us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54003ad750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.866221 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3840602496}
I20260709 12:14:57.875288 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2744) took 1223 ms. Trace:
I20260709 12:14:57.892899 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:57.894766 12493 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:14:57.876020 20211 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:57.896927 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.897953 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3174) took 1865 ms. Trace:
I20260709 12:14:57.898999 12638 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:57.901253 12638 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:57.898058 12646 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:14:57.903523 12646 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:14:57.903717 12643 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:14:57.904289 12643 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 11
I20260709 12:14:57.904603 12643 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:14:57.905048 12643 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:14:57.861533 20260 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:57.905874 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131306us
W20260709 12:14:57.907019 12461 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.714s (SENT)
I20260709 12:14:57.907102 12642 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:14:57.907517 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:14:57.912763 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:57.914595 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.914866 12639 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:14:57.915664 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2729) took 1487 ms. Trace:
I20260709 12:14:57.915961 12639 rpcz_store.cc:276] 0709 12:14:56.428111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.827746 (+1399635us) service_pool.cc:224] Handling call
0709 12:14:57.914669 (+ 86923us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.915430 (+   761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.915610 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.917716 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:57.918361 12564 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:57.921748 20299 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:14:57.922472 20299 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:57.922900 20057 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:14:57.876986 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:14:57.923537 20299 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:57.924330 20235 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:14:57.924759 20235 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:57.893381 12645 rpcz_store.cc:276] 0709 12:14:56.652053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.498242 (+846189us) service_pool.cc:224] Handling call
0709 12:14:57.870218 (+371976us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:57.875189 (+  4971us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":312418176}
I20260709 12:14:57.898514 12640 rpcz_store.cc:276] 0709 12:14:56.032985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.861868 (+1828883us) service_pool.cc:224] Handling call
0709 12:14:57.897238 (+ 35370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.897517 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:57.897564 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:57.897612 (+    48us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:57.897673 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:57.897754 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:57.897883 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:14:57.925900 20235 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 7, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:57.898106 12569 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:14:57.928717 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:57.927929 20235 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.930576 20268 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:14:57.931839 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:57.921195 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:57.942800 12492 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:57.949685 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 353379us
I20260709 12:14:57.951336 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:57.941444 12637 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:57.960072 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:57.955257 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:14:57.964067 12488 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 9.
W20260709 12:14:57.973254 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.973534 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:57.912768 20020 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:14:57.989310 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:57.989938 12645 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:57.991272 12639 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
W20260709 12:14:57.991973 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84999.4us
I20260709 12:14:57.993901 13257 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:14:58.014416 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:58.029310 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:58.029767 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:58.030146 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.030797 12494 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:14:58.033811 20211 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:14:58.034446 20211 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:58.034862 20211 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:58.036139 20247 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:57.999949 20280 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:58.036757 20247 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:58.038736 12641 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:14:58.040942 20247 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:14:58.042433 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3169) took 2186 ms (client timeout 1732 ms). Trace:
I20260709 12:14:58.033882 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:14:58.043141 12641 rpcz_store.cc:269] 0709 12:14:55.856320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:55.860039 (+  3719us) service_pool.cc:224] Handling call
0709 12:14:58.039908 (+2179869us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:58.042316 (+  2408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5006612224}
I20260709 12:14:58.048189 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:58.054088 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.058198 12640 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 25 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:14:58.062759 12497 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:14:58.070418 20290 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:14:58.064055 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 356933us
W20260709 12:14:58.082792 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:58.084362 12488 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:14:58.085942 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:58.089329 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:58.097463 12563 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:14:58.098183 12563 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:14:58.092577 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44562.3us
I20260709 12:14:58.102561 12494 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 17.
I20260709 12:14:58.064208 19868 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:58.100764 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.095094 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:58.099174 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1272) took 1202 ms. Trace:
I20260709 12:14:58.119179 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.121461 12488 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:58.094758 20086 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:14:58.119479 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:58.020690 20048 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
W20260709 12:14:58.145116 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111513us
I20260709 12:14:58.122416 20247 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:14:58.169994 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0696867179999572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2319s for a response
W20260709 12:14:58.172595 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0696867179999572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2319s for a response
W20260709 12:14:58.173316 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0134769510000297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
W20260709 12:14:58.173741 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0134769510000297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
W20260709 12:14:58.165020 20112 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.016s	sys 0.017s
W20260709 12:14:58.174511 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64526.4us
I20260709 12:14:58.175781 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:58.121201 12566 rpcz_store.cc:276] 0709 12:14:56.897333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.112907 (+215574us) service_pool.cc:224] Handling call
0709 12:14:57.115258 (+  2351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.117372 (+  2114us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:14:57.227080 (+109708us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.086022 (+858942us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:58.086113 (+    91us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:58.097256 (+ 11143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.098871 (+  1615us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:58.099087 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":66882,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":27026,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":72,"thread_start_us":794815,"threads_started":3,"wal-append.queue_time_us":516663}
W20260709 12:14:58.192379 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:58.178453 20112 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.016s	sys 0.017s
W20260709 12:14:58.183212 20221 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.004s	sys 0.026s
I20260709 12:14:58.228319 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:58.222987 20238 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.796s	user 0.015s	sys 0.004s
W20260709 12:14:58.254045 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71214.8us
W20260709 12:14:58.259241 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:58.312094 12490 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:14:58.260231 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.320235 13255 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:58.206167 20126 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:14:58.263188 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.505s (SENT)
W20260709 12:14:58.347172 12644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.022s	sys 0.023s
W20260709 12:14:58.517376 12644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.022s	sys 0.023s
W20260709 12:14:58.347043 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.025s	sys 0.035s
W20260709 12:14:58.518023 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.026s	sys 0.035s
I20260709 12:14:58.221038 20020 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:14:58.313235 20221 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.005s	sys 0.027s
W20260709 12:14:58.335244 20140 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.015s	sys 0.033s
I20260709 12:14:58.223732 20247 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:58.342334 20235 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:58.397138 12488 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:14:58.415530 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 437312us
I20260709 12:14:58.522698 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:58.388969 20048 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:14:58.431552 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:58.454078 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62867.1us
W20260709 12:14:58.433265 12611 connection.cc:256] RPC call timeout handler was delayed by -1.1186523909998414s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19530000000000003s for a response
W20260709 12:14:58.457216 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:58.432439 20179 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.643s	user 0.018s	sys 0.000s
W20260709 12:14:58.482739 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:58.478139 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.029s	sys 0.029s
W20260709 12:14:58.552891 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.030s	sys 0.029s
I20260709 12:14:58.490211 12494 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:14:58.395718 20309 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
I20260709 12:14:58.466173 19957 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:58.467904 20086 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
W20260709 12:14:58.470412 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 375381us
W20260709 12:14:58.518167 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58084.2us
W20260709 12:14:58.420761 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:58.537506 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 549738us
W20260709 12:14:58.524894 20140 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.015s	sys 0.033s
I20260709 12:14:58.585508 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.532727 12563 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:58.586246 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
W20260709 12:14:58.586722 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:58.499897 12638 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.020s	sys 0.044s
W20260709 12:14:58.602159 12638 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.020s	sys 0.044s
I20260709 12:14:58.608160 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:58.527801 20126 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:14:58.528015 20020 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:58.639189 12488 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:14:58.645227 12645 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.607s	user 0.014s	sys 0.000s
I20260709 12:14:58.645910 12645 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
W20260709 12:14:58.656399 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58013.5us
I20260709 12:14:58.587405 20179 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:58.599354 20048 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:58.683862 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:58.615823 20300 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:14:58.644290 20260 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:58.693132 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:14:58.700320 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:58.653730 12564 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.014s	sys 0.007s
W20260709 12:14:58.705832 12564 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.015s	sys 0.007s
W20260709 12:14:58.653373 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.006s	sys 0.015s
W20260709 12:14:58.706744 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.007s	sys 0.016s
W20260709 12:14:58.699326 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 557818us
I20260709 12:14:58.702728 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
W20260709 12:14:58.725354 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108883us
W20260709 12:14:58.726918 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:58.607156 20235 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:58.658900 20090 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:58.616565 19957 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:58.769218 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 992) took 1029 ms. Trace:
W20260709 12:14:58.800697 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46815us
W20260709 12:14:58.818076 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62462.6us
I20260709 12:14:58.754532 20300 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:14:58.853292 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3343) took 1067 ms. Trace:
I20260709 12:14:58.815431 12489 rpcz_store.cc:276] 0709 12:14:57.739467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.743931 (+  4464us) service_pool.cc:224] Handling call
0709 12:14:57.766405 (+ 22474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.916873 (+150468us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:14:58.410683 (+493810us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.747047 (+336364us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:58.747346 (+   299us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:14:58.768063 (+ 20717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.768347 (+   284us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:58.768845 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49779,"prepare.queue_time_us":570521,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":8197,"thread_start_us":671640,"threads_started":2}
I20260709 12:14:58.731508 20260 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:58.801853 20261 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:58.808104 20199 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:14:58.928306 20199 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:58.929050 19957 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:58.929589 20199 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:58.931334 20199 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:14:58.790431 20090 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:58.867635 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68032.5us
I20260709 12:14:58.866948 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:14:58.863147 20299 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:14:58.861649 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 260628us
I20260709 12:14:58.901161 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:14:58.850881 20268 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:58.950337 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:58.927019 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2750) took 1948 ms. Trace:
I20260709 12:14:58.958357 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:58.958941 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 316111us
I20260709 12:14:58.912727 12569 rpcz_store.cc:276] 0709 12:14:57.785455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.933291 (+147836us) service_pool.cc:224] Handling call
0709 12:14:57.933709 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.934166 (+   457us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:14:57.934699 (+   533us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.852806 (+918107us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:58.852883 (+    77us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:14:58.852941 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.853022 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:58.853187 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13022,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":276704,"thread_start_us":836310,"threads_started":2,"wal-append.queue_time_us":848385}
I20260709 12:14:58.970248 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:58.971195 13255 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:14:58.974491 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 752320us
I20260709 12:14:58.847203 20126 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:58.935292 20218 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:58.943961 20112 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.000s	sys 0.014s
I20260709 12:14:58.809319 20247 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:58.976882 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:58.952008 12644 rpcz_store.cc:276] 0709 12:14:56.978972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.806522 (+827550us) service_pool.cc:224] Handling call
0709 12:14:57.806880 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:14:57.807182 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:14:58.519928 (+712746us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.926396 (+406468us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:14:58.926471 (+    75us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:14:58.926643 (+   172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.926733 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:58.926938 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":721203,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":23003,"thread_start_us":1005469,"threads_started":2}
I20260709 12:14:59.011310 13257 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:14:58.870594 20020 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:59.002770 20308 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.474s	user 0.001s	sys 0.000s
W20260709 12:14:59.048465 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 181613us
I20260709 12:14:59.041754 13262 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:14:59.042366 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
W20260709 12:14:58.999634 20112 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.000s	sys 0.015s
W20260709 12:14:59.052414 12536 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.527s (SENT)
I20260709 12:14:58.876724 20048 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:59.056289 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.036182 20310 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.421s	user 0.001s	sys 0.002s
I20260709 12:14:59.011279 19965 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:14:58.943925 20261 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:59.082906 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53204us
I20260709 12:14:59.083539 13261 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:59.089063 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 139668us
I20260709 12:14:59.093216 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:14:59.063174 20306 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.406s	user 0.000s	sys 0.002s
I20260709 12:14:59.081786 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:59.004509 20268 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:59.073673 20307 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.457s	user 0.002s	sys 0.000s
W20260709 12:14:59.101272 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143185us
I20260709 12:14:59.069723 19957 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.079389 19965 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:14:59.121673 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 263864us
W20260709 12:14:59.123458 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.164870 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:59.077755 20179 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:59.049400 20086 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.058s
W20260709 12:14:59.193300 20321 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.334s	user 0.000s	sys 0.001s
I20260709 12:14:59.194038 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:14:59.195043 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.866s	user 0.006s	sys 0.025s
W20260709 12:14:59.200635 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.238693 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.101s	user 0.007s	sys 0.025s
I20260709 12:14:59.200486 20333 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:14:59.202950 20330 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.202765 19957 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:14:59.205953 20312 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.478s	user 0.001s	sys 0.001s
I20260709 12:14:59.205818 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1286) took 1272 ms. Trace:
I20260709 12:14:59.208845 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.211086 20198 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:14:59.223500 20090 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:14:59.225538 20342 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:14:59.228160 13262 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:14:59.005069 20218 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.120649 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:14:59.263413 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.119942 20308 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:14:59.273554 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 121948us
W20260709 12:14:59.276729 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.196206 20351 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:14:59.249679 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:59.300089 13261 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:14:59.271463 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 280853us
I20260709 12:14:59.119278 20235 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:59.134349 20238 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:14:59.277331 12564 rpcz_store.cc:276] 0709 12:14:57.933066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:58.105802 (+172736us) service_pool.cc:224] Handling call
0709 12:14:58.106297 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:58.706554 (+600257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:58.706596 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:58.748021 (+ 41425us) log.cc:844] Serialized 26 byte log entry
0709 12:14:58.999431 (+251410us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:14:58.999640 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:59.000083 (+   443us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:59.205395 (+205312us) raft_consensus.cc:1685] finished
0709 12:14:59.205415 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:59.205738 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5677,"raft.queue_time_us":307367,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":5325,"thread_start_us":710901,"threads_started":2,"wal-append.queue_time_us":386321}
I20260709 12:14:59.290179 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.182838 19965 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:14:59.287102 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:59.296836 20310 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:14:59.306346 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:14:59.315205 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:59.332473 20359 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:59.333276 20359 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:59.334343 20359 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:59.306867 20111 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:14:59.312391 20112 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:14:59.339785 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3331) took 1810 ms. Trace:
W20260709 12:14:59.342563 20332 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.205s	user 0.000s	sys 0.000s
I20260709 12:14:59.340404 12570 rpcz_store.cc:276] 0709 12:14:57.529909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.530117 (+   208us) service_pool.cc:224] Handling call
0709 12:14:57.565074 (+ 34957us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:57.606603 (+ 41529us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:14:57.680692 (+ 74089us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.104985 (+424293us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.105274 (+   289us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.591290 (+486016us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:58.685057 (+ 93767us) log.cc:844] Serialized 52 byte log entry
0709 12:14:58.685317 (+   260us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:58.685373 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.685488 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:59.339364 (+653876us) raft_consensus.cc:1685] finished
0709 12:14:59.339391 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:59.339698 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7141,"prepare.queue_time_us":105199,"prepare.run_cpu_time_us":141,"prepare.run_wall_time_us":2801,"raft.queue_time_us":938253,"raft.run_cpu_time_us":485,"raft.run_wall_time_us":19645,"spinlock_wait_cycles":6945152,"thread_start_us":880269,"threads_started":2,"wal-append.queue_time_us":774000}
I20260709 12:14:59.344609 20235 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:14:59.345088 20315 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.432s	user 0.001s	sys 0.001s
W20260709 12:14:59.348284 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 333706us
I20260709 12:14:59.350279 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:14:59.352762 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.304443 20238 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:14:59.099689 20300 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:59.362202 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52706.7us
I20260709 12:14:59.367854 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.374029 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:59.355319 20235 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:59.292600 20330 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:59.407081 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:59.408377 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.366740 20260 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:14:59.412981 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.413899 12571 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:14:59.328452 20333 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.075s
W20260709 12:14:59.416576 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49643.2us
W20260709 12:14:59.416910 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53188us
I20260709 12:14:59.410928 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:14:59.411793 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92226.6us
I20260709 12:14:59.418452 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.418637 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:14:59.419747 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 312306us
I20260709 12:14:59.426295 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1281) took 1943 ms. Trace:
I20260709 12:14:59.405753 20235 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:59.426951 12572 rpcz_store.cc:276] 0709 12:14:57.483052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.483228 (+   176us) service_pool.cc:224] Handling call
0709 12:14:57.496557 (+ 13329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:57.680010 (+183453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:57.680045 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:57.756946 (+ 76901us) log.cc:844] Serialized 26 byte log entry
0709 12:14:58.380699 (+623753us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:58.380736 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.380804 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:59.425943 (+1045139us) raft_consensus.cc:1685] finished
0709 12:14:59.425983 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:59.426204 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13380,"raft.queue_time_us":78497,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":23849,"thread_start_us":621837,"threads_started":2,"wal-append.queue_time_us":673274}
I20260709 12:14:59.433773 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:59.443862 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.449734 12569 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:14:59.450666 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 145436us
I20260709 12:14:59.448745 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:14:59.454032 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:59.456230 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:14:59.457077 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 277739us
I20260709 12:14:59.455196 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:14:59.454041 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.458508 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.459421 12572 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:14:59.462085 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:59.462764 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:14:59.432502 20112 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:14:59.462821 12494 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:59.475135 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:14:59.479267 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.474426 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:14:59.480492 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:14:59.466224 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:59.466785 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:14:59.467439 20357 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.000s	sys 0.002s
I20260709 12:14:59.466811 20360 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:14:59.463374 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.482409 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107764us
I20260709 12:14:59.424305 20238 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:59.506078 12572 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:59.514333 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.430787 20111 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:14:59.518019 12638 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:14:59.519117 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.532764 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.462047 20057 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.533747 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3339) took 1824 ms. Trace:
I20260709 12:14:59.535703 12568 rpcz_store.cc:276] 0709 12:14:57.709543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.783946 (+ 74403us) service_pool.cc:224] Handling call
0709 12:14:57.784494 (+   548us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:14:57.785228 (+   734us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:14:57.785577 (+   349us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.780588 (+995011us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:14:58.817803 (+ 37215us) log.cc:844] Serialized 52 byte log entry
0709 12:14:58.818016 (+   213us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:14:58.818305 (+   289us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.818495 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.818589 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:59.533472 (+714883us) raft_consensus.cc:1685] finished
0709 12:14:59.533495 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:59.533679 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53908,"prepare.queue_time_us":20154,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":113,"raft.queue_time_us":706331,"raft.run_cpu_time_us":723,"raft.run_wall_time_us":55186,"thread_start_us":989400,"threads_started":2,"wal-append.queue_time_us":819110}
I20260709 12:14:59.538561 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.540266 20324 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:14:59.539104 12567 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:14:59.538555 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.535749 12641 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:14:59.525836 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:14:59.538815 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82397.3us
I20260709 12:14:59.452311 20366 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:14:59.539577 20358 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:14:59.557358 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.471792 20086 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:14:59.558426 12644 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:59.568384 12491 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:14:59.491312 20360 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:14:59.473954 20356 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.146s
I20260709 12:14:59.473826 20343 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.164s
I20260709 12:14:59.542013 20048 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:14:59.591820 12639 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
W20260709 12:14:59.734316 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.658257 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:59.572261 12489 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:14:59.571225 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.707840 12643 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:14:59.758107 12645 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:14:59.592056 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 136968us
W20260709 12:14:59.554673 20211 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.519s	user 0.010s	sys 0.011s
W20260709 12:14:59.625461 20362 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:14:59.609133 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:14:59.736995 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:14:59.658090 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94684.3us
W20260709 12:14:59.768458 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:59.596570 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:14:59.716923 20199 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.785s	user 0.014s	sys 0.001s
W20260709 12:14:59.626380 20372 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:14:59.616492 20238 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:14:59.777349 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.773342 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:59.718466 20330 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:59.661140 19957 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:59.624985 20368 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:14:59.637717 20261 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:59.643095 20218 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:59.784307 12563 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:59.790153 20199 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:14:59.794903 20327 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.921s	user 0.006s	sys 0.000s
W20260709 12:14:59.815080 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.820569 20369 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:14:59.823868 12641 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:14:59.823825 20358 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:14:59.832053 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.832964 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.833684 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.834553 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:14:59.836162 20048 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:14:59.836911 20048 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.839967 20048 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:59.582764 20112 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.715806 12392 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 7 to 14. New cstate: current_term: 14 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:14:59.754676 20360 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:14:59.807463 19999 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.718745 19965 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.810168 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:59.811980 20199 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:14:59.813211 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:50142 (request call id 1279) took 2450 ms. Trace:
I20260709 12:14:59.858542 12568 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:14:59.803544 20368 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:14:59.868577 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.910140 12646 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:14:59.862038 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.876026 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:59.862903 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.772233 20211 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:59.843736 20333 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:59.897673 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.618s (SENT)
I20260709 12:14:59.591076 20057 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.861267 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:14:59.916033 12644 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:14:59.899116 20372 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:14:59.900228 12563 rpcz_store.cc:276] 0709 12:14:57.355712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.390176 (+ 34464us) service_pool.cc:224] Handling call
0709 12:14:59.767365 (+2377189us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eaa3d 00007ffbfa7bacd0 00007ffbf3d3f254 00007ffbf3d3f1e2 00007ffbf3d3f111 00007ffbf3d3f08c 00007ffbf3d3e342 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:14:59.806118 (+ 38753us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1576384768}
I20260709 12:14:59.897070 20358 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.926379 12638 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:14:59.904479 20199 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:14:59.900877 20218 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:14:59.942129 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:14:59.942368 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 178716us
I20260709 12:14:59.914422 20368 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:14:59.944446 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 144285us
I20260709 12:14:59.933984 20353 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:14:59.920001 20086 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:14:59.950131 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 866363us
I20260709 12:14:59.897394 20112 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:14:59.955168 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96602.7us
W20260709 12:14:59.960275 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 394741us
I20260709 12:14:59.891607 19999 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.963176 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:14:59.967191 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:14:59.948793 20329 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:14:59.973778 20064 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:14:59.974680 20064 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.971957 20322 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:14:59.976356 20322 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:14:59.977041 20064 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:59.977423 20322 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:59.978802 20211 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:14:59.982929 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:14:59.983692 13255 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:14:59.984124 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 978) took 2049 ms. Trace:
I20260709 12:14:59.973941 19999 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:14:59.937861 20358 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:14:59.984644 12642 rpcz_store.cc:276] 0709 12:14:57.935127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:57.946950 (+ 11823us) service_pool.cc:224] Handling call
0709 12:14:57.976506 (+ 29556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:14:58.031759 (+ 55253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:14:58.031798 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:14:58.032428 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:14:58.708231 (+675803us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:14:58.829170 (+120939us) log.cc:844] Serialized 14 byte log entry
0709 12:14:58.853040 (+ 23870us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:14:58.853215 (+   175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:14:58.931439 (+ 78224us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b180010c300. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:59.982768 (+1051329us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b24003e3618. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:14:59.983811 (+  1043us) raft_consensus.cc:1685] finished
0709 12:14:59.983837 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:14:59.984043 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54513,"prepare.queue_time_us":137592,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":18098,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":38494,"spinlock_wait_cycles":73548800,"thread_start_us":674961,"threads_started":1,"wal-append.queue_time_us":795762}
I20260709 12:14:59.925112 20333 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:14:59.988288 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:14:59.996089 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:14:59.995767 20086 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:00.002405 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.668s (SENT)
W20260709 12:15:00.002969 20377 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
I20260709 12:14:59.985379 19965 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:00.019418 13218 leader_election.cc:336] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.138s (SENT)
I20260709 12:15:00.021412 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.026904 12637 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:14:59.958033 20218 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.025676 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:00.033905 20381 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:15:00.034067 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45852.7us
I20260709 12:15:00.038336 20334 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:00.039765 20247 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:00.040225 20247 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:00.042011 20247 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.047639 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:00.048974 20359 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:15:00.052608 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:00.023607 20235 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.576s	user 0.012s	sys 0.002s
I20260709 12:15:00.053555 20380 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:00.055032 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52679.2us
I20260709 12:15:00.031764 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:00.058768 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91719.4us
W20260709 12:15:00.064550 12536 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.728s (SENT)
W20260709 12:15:00.066123 12536 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.765s (SENT)
I20260709 12:15:00.066645 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.068457 20330 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:00.055927 20309 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:00.070770 20199 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.073768 20333 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.074095 20199 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.074453 20057 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:00.073796 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 126210us
I20260709 12:15:00.055948 20329 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.078120 20322 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:00.079218 20218 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.079775 20329 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.079958 20322 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:00.062127 20235 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:00.080886 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74796.1us
W20260709 12:15:00.082456 12536 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.819s (SENT)
I20260709 12:15:00.086441 20235 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.086997 20112 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.087683 20260 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:00.088317 20358 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.088426 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.089574 20382 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:15:00.087069 12536 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.830s (SENT)
I20260709 12:15:00.090090 20382 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.090303 20268 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.090545 20382 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:00.090323 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.093873 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:00.095870 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:00.084403 19957 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:00.096894 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:00.101676 12567 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:00.103302 20198 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:00.103816 20198 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:00.103945 20360 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.080055 20218 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:15:00.097740 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:00.083704 13254 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:00.098174 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:00.115095 12563 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:15:00.099256 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.099404 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:00.096155 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:00.100873 12569 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:15:00.100793 12570 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:00.107164 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.119243 12568 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:00.124354 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:00.128182 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:00.128938 13259 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:00.129242 13259 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:00.130035 13259 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:00.131737 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.131358 13259 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:00.135715 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:00.136579 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.137717 20359 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:00.138273 20359 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.138706 20359 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:00.141856 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:00.083395 20356 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:00.158226 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.137259 20199 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:15:00.162227 13255 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:15:00.118826 20064 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.181957 20064 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.168841 20218 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:00.144078 20086 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:00.182926 20390 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:15:00.164125 20356 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.184449 12570 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:00.184918 20356 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:00.185101 20064 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.157047 20380 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:00.186172 20380 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.186542 20380 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:00.163676 20333 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:00.189985 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:00.192354 20235 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:00.192909 20235 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.193265 20235 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:00.199582 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:00.200562 13253 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:00.202206 20376 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:00.202757 20376 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:00.203647 20376 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:00.206396 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92378.9us
I20260709 12:15:00.182507 20360 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.166213 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:00.210495 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:00.205296 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:00.105163 20198 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.223078 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:00.204969 20376 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:15:00.226675 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139818us
W20260709 12:15:00.256580 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51389.5us
W20260709 12:15:00.260989 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45237.4us
I20260709 12:15:00.220082 20057 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:15:00.264905 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:00.269668 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.224362 20268 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:00.211393 19965 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.051s
W20260709 12:15:00.280329 12536 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.991s (SENT)
I20260709 12:15:00.220970 20211 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:15:00.214569 19957 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:15:00.263433 19999 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:00.287468 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:00.282029 20362 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:15:00.289804 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.282991 20268 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.329813 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:00.288738 20198 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:00.327622 20343 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:00.380863 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.381315 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:00.325371 20353 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:00.343616 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:00.386821 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:00.333078 20268 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:00.332635 20261 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:00.387825 12565 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:00.390537 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.392475 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:00.395133 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.400750 12490 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:00.395237 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:15:00.398010 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131564us
I20260709 12:15:00.402786 12566 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:00.403546 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.404863 12488 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:00.323081 20020 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:15:00.408026 12492 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:00.380189 20309 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.390610 20057 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:00.390343 20333 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:00.396565 20369 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.008s	sys 0.000s
W20260709 12:15:00.418054 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.394765 20261 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.391665 20199 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.418282 20111 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:00.342154 20336 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.085s
I20260709 12:15:00.418201 20343 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:00.331748 19957 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:15:00.468470 12492 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:15:00.413861 20358 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.433535 20111 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:00.435518 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102910us
I20260709 12:15:00.434666 20261 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:00.436369 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:00.439453 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.080s (SENT)
I20260709 12:15:00.424446 20309 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:00.450336 20399 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:15:00.449855 20199 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.421772 20401 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:15:00.412475 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
W20260709 12:15:00.466403 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.339995 20382 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.396384 20368 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.436195 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.419723 19965 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:15:00.432659 20057 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:00.492801 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:00.492842 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.508271 13255 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22.
W20260709 12:15:00.510249 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:00.510673 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.508898 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.489585 20111 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:00.489316 20358 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.506232 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.511384 12644 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:00.509168 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.507282 12493 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:00.441825 20333 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.521840 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:00.509780 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.522712 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.523399 12640 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:00.523726 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.524363 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:00.525247 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.527604 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:00.527771 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:00.528414 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 192696us
I20260709 12:15:00.529912 12643 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:00.528311 12641 raft_consensus.cc:2376] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20. Re-sending same reply.
W20260709 12:15:00.536553 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48109us
I20260709 12:15:00.507210 20401 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.508723 20368 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.544198 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:00.546169 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85781.5us
I20260709 12:15:00.547549 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.538169 20336 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:00.564692 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:00.552184 20211 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:15:00.579901 20309 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.582253 20358 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.584365 20235 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:00.587756 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148396us
I20260709 12:15:00.611119 20235 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.611629 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:00.547830 20057 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.614905 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:00.615607 12570 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:00.590785 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:00.617210 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:00.622972 12569 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:00.589325 20199 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.604312 20382 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:00.608695 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57309.8us
I20260709 12:15:00.608907 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:00.631400 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.631474 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.630707 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:00.617723 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:00.618546 20380 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.556093 20401 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.637919 20309 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:00.638417 20366 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:00.638648 20401 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.638980 20366 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:00.640053 20366 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.641292 13253 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:00.637976 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:00.594017 20333 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.553673 20300 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:15:00.635231 20403 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:15:00.636158 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:00.549584 20280 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:00.644557 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:00.646595 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:15:00.664325 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108913us
I20260709 12:15:00.646706 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:00.652264 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:00.644807 20359 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.506s	user 0.006s	sys 0.007s
I20260709 12:15:00.644335 20358 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:00.678989 20380 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:00.680852 20247 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:00.616941 20261 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.683310 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.667009 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:00.684783 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.647775 20390 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:00.687016 12489 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:00.695583 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:00.638461 20309 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:00.649907 20366 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:00.545732 20343 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:00.699802 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69249.2us
I20260709 12:15:00.667883 20333 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:15:00.675537 20235 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:00.667037 20300 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:00.681716 20090 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:00.688583 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.648928 20199 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:15:00.649349 20356 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.685329 20359 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.676541 20280 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.709038 20140 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:00.747689 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.747818 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:00.750250 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54751.8us
W20260709 12:15:00.818125 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 129694us
I20260709 12:15:00.693940 19999 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:00.698386 20390 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.732893 13254 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:00.778781 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.779263 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.822057 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:00.751454 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.700462 20309 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:00.701327 20376 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:00.817728 20020 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:00.838989 12492 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:00.715842 19957 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:15:00.779306 20349 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.368s	user 0.003s	sys 0.004s
I20260709 12:15:00.759505 20329 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.746541 20336 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:00.840186 20261 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.928671 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.929525 12495 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:15:00.836805 20368 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:00.822098 20090 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:00.855201 20382 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:00.855855 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.943349 20090 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:00.943611 20382 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:00.857374 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:00.732223 20268 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.945151 20382 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.946491 20268 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:00.946336 20300 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:00.867677 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116329us
I20260709 12:15:00.868458 20199 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:00.868994 20086 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:00.947890 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.948071 20086 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:00.779049 20414 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:00.949126 20198 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.571s	user 0.003s	sys 0.007s
W20260709 12:15:00.950525 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 298324us
I20260709 12:15:00.951500 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:00.708631 20235 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.952692 20235 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:00.953708 20329 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.956132 20329 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.958026 20356 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:15:00.905203 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.829447 20390 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:00.932508 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 184844us
W20260709 12:15:00.966073 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108855us
I20260709 12:15:00.966199 20199 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:00.966611 20358 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:00.967165 20358 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.967231 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:00.845945 20057 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:00.967828 20358 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:00.943728 12492 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:00.856496 20280 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.969201 20280 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:00.969578 20280 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:00.970012 20280 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:00.966993 20199 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:00.971453 20111 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.972209 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.968155 20057 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.974500 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:00.975164 20057 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:00.977181 20381 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:00.978279 20381 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:00.871852 19957 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.980381 19957 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:00.976227 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.949172 20086 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.10, Last appended by leader: 10, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:00.985529 20086 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:00.848022 20376 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.987072 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:00.987604 20376 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:00.988227 20408 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:00.989871 12496 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:00.991711 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:00.993526 20368 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:00.994817 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:00.998370 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:01.010344 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:01.012791 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:00.707979 20333 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:01.038120 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
W20260709 12:15:01.042255 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:01.081032 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:01.122826 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124585us
I20260709 12:15:01.128237 20333 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:15:01.127254 20408 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:01.117647 20199 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:00.974614 20359 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:01.153167 20404 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:01.200474 20408 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:01.223866 20404 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:00.994867 20368 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:01.241768 20404 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:01.248077 20111 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.236289 20408 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:01.263664 20280 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.287405 20280 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.291122 20390 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:01.278954 20414 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:01.294271 20235 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:01.301127 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 333981us
I20260709 12:15:01.300865 20090 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:01.292114 20353 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:01.304255 20112 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:01.306835 20353 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:01.306962 20112 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:01.307565 20112 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:01.311856 19957 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:01.320505 20309 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.323586 20057 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.324221 19957 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.332206 20408 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.323694 20376 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:01.334924 20366 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.000s
I20260709 12:15:01.336179 20358 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.337077 20112 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.344966 20309 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.345343 20376 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:01.361896 20112 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.347718 20353 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:01.348618 20333 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.368434 20333 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.345501 20090 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:01.352398 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 931) took 1154 ms. Trace:
I20260709 12:15:01.369436 13255 rpcz_store.cc:276] 0709 12:15:00.198465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.213845 (+ 15380us) service_pool.cc:224] Handling call
0709 12:15:01.352154 (+1138309us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:01.352323 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":653297920}
I20260709 12:15:01.354750 20404 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:01.372500 20404 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:01.357826 20235 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:01.358206 20390 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:01.358676 20358 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:01.349929 20408 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:01.356308 20057 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:02.646639 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3507) took 1898 ms (client timeout 2029 ms). Trace:
W20260709 12:15:02.647360 12496 rpcz_store.cc:269] 0709 12:15:00.747921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.748572 (+   651us) service_pool.cc:224] Handling call
0709 12:15:02.645446 (+1896874us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:02.646494 (+  1048us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381030144}
I20260709 12:15:02.648909 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:02.650015 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3525) took 1660 ms (client timeout 1777 ms). Trace:
W20260709 12:15:02.650403 12496 rpcz_store.cc:269] 0709 12:15:00.989550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.648568 (+1659018us) service_pool.cc:224] Handling call
0709 12:15:02.649947 (+  1379us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.651515 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:02.652352 12496 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:02.656744 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3523) took 1688 ms (client timeout 2027 ms). Trace:
W20260709 12:15:02.657245 12496 rpcz_store.cc:269] 0709 12:15:00.968219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.651144 (+1682925us) service_pool.cc:224] Handling call
0709 12:15:02.656654 (+  5510us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.658495 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:02.660450 12496 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:02.670336 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:02.671331 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3515) took 1827 ms (client timeout 2193 ms). Trace:
I20260709 12:15:02.671346 12496 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
W20260709 12:15:02.672050 12493 rpcz_store.cc:269] 0709 12:15:00.843488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.669965 (+1826477us) service_pool.cc:224] Handling call
0709 12:15:02.671263 (+  1298us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:02.672415 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3522) took 1820 ms (client timeout 2178 ms). Trace:
I20260709 12:15:02.673254 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:02.672993 12496 rpcz_store.cc:269] 0709 12:15:00.852122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.658092 (+1805970us) service_pool.cc:224] Handling call
0709 12:15:02.672342 (+ 14250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.674058 12493 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:15:02.674667 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3524) took 1685 ms (client timeout 2066 ms). Trace:
I20260709 12:15:02.674909 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:02.676326 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3512) took 1846 ms. Trace:
I20260709 12:15:02.676700 12496 rpcz_store.cc:276] 0709 12:15:00.829970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.674627 (+1844657us) service_pool.cc:224] Handling call
0709 12:15:02.676255 (+  1628us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.680577 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:02.684702 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3518) took 1840 ms. Trace:
I20260709 12:15:02.685223 12496 rpcz_store.cc:276] 0709 12:15:00.844353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.677487 (+1833134us) service_pool.cc:224] Handling call
0709 12:15:02.684609 (+  7122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.686604 12496 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:15:02.687410 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3509) took 1886 ms. Trace:
W20260709 12:15:02.675045 12493 rpcz_store.cc:269] 0709 12:15:00.989114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.672940 (+1683826us) service_pool.cc:224] Handling call
0709 12:15:02.674610 (+  1670us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.687799 12496 rpcz_store.cc:276] 0709 12:15:00.801650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.686021 (+1884371us) service_pool.cc:224] Handling call
0709 12:15:02.686416 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:02.687078 (+   662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:02.687316 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.689527 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:02.689692 12493 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:15:02.690357 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3511) took 1871 ms. Trace:
I20260709 12:15:02.690819 12496 rpcz_store.cc:276] 0709 12:15:00.818996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.688987 (+1869991us) service_pool.cc:224] Handling call
0709 12:15:02.689345 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:02.690014 (+   669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:02.690224 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.691131 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3510) took 1882 ms. Trace:
I20260709 12:15:02.692803 12493 rpcz_store.cc:276] 0709 12:15:00.808446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.688457 (+1880011us) service_pool.cc:224] Handling call
0709 12:15:02.689087 (+   630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:02.690628 (+  1541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:02.691006 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:02.722338 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 1.76852e+06us
I20260709 12:15:02.724341 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:02.750599 20429 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:02.751134 20429 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:02.751528 20429 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:02.763895 20429 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.766798 20429 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.767783 20429 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:02.771005 20431 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:02.772929 20431 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:02.778780 12488 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:02.780444 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3520) took 1933 ms. Trace:
I20260709 12:15:02.781068 12488 rpcz_store.cc:276] 0709 12:15:00.847546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.777770 (+1930224us) service_pool.cc:224] Handling call
0709 12:15:02.778304 (+   534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:02.780003 (+  1699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:02.780354 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.766640 20430 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.018s
W20260709 12:15:02.791749 12569 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 2.148s	user 0.007s	sys 0.020s
I20260709 12:15:02.792838 12569 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
W20260709 12:15:02.793679 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1302) took 2178 ms (client timeout 2742 ms). Trace:
W20260709 12:15:02.794168 12569 rpcz_store.cc:269] 0709 12:15:00.614868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.616866 (+  1998us) service_pool.cc:224] Handling call
0709 12:15:02.793544 (+2176678us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:02.797915 20432 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:02.802287 20048 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:02.803014 20048 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:02.804252 20048 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.818463 20048 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:15:02.822752 13253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.883s	user 0.028s	sys 0.062s
W20260709 12:15:02.823670 13253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.884s	user 0.029s	sys 0.062s
I20260709 12:15:02.824661 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1106) took 1960 ms. Trace:
I20260709 12:15:02.829926 20430 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:02.825030 13253 rpcz_store.cc:276] 0709 12:15:00.864491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.918778 (+ 54287us) service_pool.cc:224] Handling call
0709 12:15:00.938429 (+ 19651us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:00.938869 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:00.938924 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:00.939565 (+   641us) log.cc:844] Serialized 26 byte log entry
0709 12:15:02.824203 (+1884638us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:02.824253 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:02.824335 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:02.824389 (+    54us) raft_consensus.cc:1685] finished
0709 12:15:02.824402 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:02.824578 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8384,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":1884127,"threads_started":1,"wal-append.queue_time_us":347606,"wal-append.run_cpu_time_us":2302,"wal-append.run_wall_time_us":1002318}
W20260709 12:15:02.848585 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50223.3us
I20260709 12:15:02.849402 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:02.857947 20433 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:02.858487 20433 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:02.859542 20433 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:02.863376 20353 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:02.864224 20353 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:02.864727 20353 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:02.870519 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3364) took 2486 ms. Trace:
I20260709 12:15:02.872493 20353 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.871318 12567 rpcz_store.cc:276] 0709 12:15:00.383619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.383765 (+   146us) service_pool.cc:224] Handling call
0709 12:15:00.384164 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:00.384437 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:00.384793 (+   356us) log.cc:844] Serialized 14 byte log entry
0709 12:15:02.816634 (+2431841us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b1800331000. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:15:02.860027 (+ 43393us) log.cc:844] Serialized 14 byte log entry
0709 12:15:02.869922 (+  9895us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:02.869996 (+    74us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:02.870079 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:02.870192 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:02.870410 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6710,"prepare.queue_time_us":5092,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":21480,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":271,"spinlock_wait_cycles":4312832,"thread_start_us":42507,"threads_started":2,"wal-append.queue_time_us":2450739}
I20260709 12:15:02.876904 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:02.878247 20437 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:15:02.878732 20437 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:02.879196 20437 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:15:02.878868 20433 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:02.879873 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.031s	user 0.025s	sys 0.026s
W20260709 12:15:02.881070 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.032s	user 0.025s	sys 0.026s
W20260709 12:15:02.881407 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.05925e+06us
I20260709 12:15:02.881453 20433 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.887373 20433 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:15:02.889138 13218 leader_election.cc:336] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.655s (SENT)
I20260709 12:15:02.890635 20353 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:02.891178 20353 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:02.893560 20353 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.895366 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:02.896386 20353 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:02.897421 20437 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.900166 20437 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.905841 20353 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:02.907212 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:02.909504 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.060s	user 0.039s	sys 0.034s
I20260709 12:15:02.909958 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:02.910826 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.061s	user 0.040s	sys 0.034s
I20260709 12:15:02.911242 20353 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:15:02.911545 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.129s (SENT) [suppressed 37 similar messages]
I20260709 12:15:02.911988 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:02.912822 20429 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:02.915141 20429 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.916234 13253 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:02.917724 20429 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.917909 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:02.913730 20353 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:02.918733 13255 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:02.919242 20353 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:02.919785 20353 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:02.924661 20353 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:02.935366 12536 connection.cc:256] RPC call timeout handler was delayed by -4.5054259730002286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20260709 12:15:02.936421 12536 connection.cc:256] RPC call timeout handler was delayed by -4.4802259730001879s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
W20260709 12:15:02.936995 12536 connection.cc:256] RPC call timeout handler was delayed by -4.4676259730001675s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20260709 12:15:02.940135 12536 connection.cc:256] RPC call timeout handler was delayed by -4.1337259730000824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
W20260709 12:15:02.940614 12536 connection.cc:256] RPC call timeout handler was delayed by -3.9582259730002534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
W20260709 12:15:02.940939 12536 connection.cc:256] RPC call timeout handler was delayed by -3.9168259730001864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2169s for a response
I20260709 12:15:02.941829 20429 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:02.942517 20429 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:02.943518 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 1.62393e+06us
I20260709 12:15:02.945133 20429 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:02.945302 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:02.946254 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3807063940000717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
I20260709 12:15:02.947499 20429 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:02.949517 20429 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:02.946832 12536 connection.cc:256] RPC call timeout handler was delayed by -3.3174259730001268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28350000000000003s for a response
W20260709 12:15:02.954671 12536 connection.cc:256] RPC call timeout handler was delayed by -3.2472259730002406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2913s for a response
I20260709 12:15:02.955590 20429 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:02.956012 12536 connection.cc:256] RPC call timeout handler was delayed by -3.0720063940000273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1916s for a response
W20260709 12:15:02.957408 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3890063940000346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.37860000000000005s for a response
I20260709 12:15:02.958657 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:02.961095 20437 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:02.962380 20437 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:02.966020 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:02.967381 20437 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:15:02.968155 20437 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:02.969400 20437 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.975296 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:02.984345 20429 raft_consensus.cc:2764] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:15:02.984900 20432 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:02.985260 20429 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:02.985773 20432 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.985975 20429 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:02.988826 20429 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.988895 20432 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:02.991444 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:02.991190 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:02.993283 12569 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:02.993826 20433 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:02.994450 20433 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:02.994831 20433 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:03.000514 20434 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:03.001220 20434 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:03.003320 20434 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.005443 20434 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:03.006404 20429 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:03.008600 20429 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:15:03.016080 20440 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:03.016525 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1300) took 2402 ms. Trace:
I20260709 12:15:00.995862 12495 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:03.034024 12570 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
W20260709 12:15:03.035173 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3521) took 2183 ms (client timeout 1718 ms). Trace:
W20260709 12:15:03.041738 12495 rpcz_store.cc:269] 0709 12:15:00.851730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.994436 (+142706us) service_pool.cc:224] Handling call
0709 12:15:03.035090 (+2040654us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:03.031234 20433 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.045346 20433 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.050899 12567 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:15:03.053792 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 141881us
I20260709 12:15:03.054458 20439 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:03.053841 20441 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:15:03.055099 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.056170 12567 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:03.039476 20440 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.057907 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.058646 12565 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:03.058991 20433 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:03.055306 20441 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:03.059911 20441 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:03.059808 20433 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:03.062357 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.074441 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.075215 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.074870 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.077129 20336 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:03.074460 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.055132 20439 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.082103 20441 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.084530 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.084817 20433 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.082702 20439 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:03.082880 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166948us
I20260709 12:15:03.089884 20441 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.091558 20111 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:03.095017 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.096884 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.097661 12565 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.091478 20440 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.119863 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:03.121672 13218 leader_election.cc:336] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.261s (SENT)
I20260709 12:15:03.123894 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:03.125701 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124157us
I20260709 12:15:03.126880 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:03.125995 20440 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:03.135053 20446 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:03.136211 12569 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:03.136364 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:03.136883 20446 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:03.137439 20433 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:03.140340 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 2.12988e+06us
I20260709 12:15:03.137451 20446 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:03.141500 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.144268 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.148653 20433 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:03.149384 20433 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:03.150039 20433 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:03.152200 12489 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:03.155642 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3517) took 2311 ms. Trace:
I20260709 12:15:03.156067 12489 rpcz_store.cc:276] 0709 12:15:00.844071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.714269 (+1870198us) service_pool.cc:224] Handling call
0709 12:15:02.714700 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:03.152737 (+438037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:03.155317 (+  2580us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:03.155554 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1005603200}
I20260709 12:15:03.162011 12569 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:15:03.166237 20260 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:03.167850 20433 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:03.181098 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 235861us
I20260709 12:15:03.185770 12565 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:03.188891 20433 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.198444 12392 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:03.207089 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:03.034952 12563 rpcz_store.cc:276] 0709 12:15:00.614288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.619125 (+  4837us) service_pool.cc:224] Handling call
0709 12:15:00.619422 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:00.619965 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:00.619995 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:00.620532 (+   537us) log.cc:844] Serialized 26 byte log entry
0709 12:15:00.707070 (+ 86538us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:00.707121 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:00.707215 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:03.016227 (+2309012us) raft_consensus.cc:1685] finished
0709 12:15:03.016252 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:03.016440 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":238,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"thread_start_us":85644,"threads_started":1,"wal-append.queue_time_us":525186,"wal-append.run_cpu_time_us":2607,"wal-append.run_wall_time_us":1093749}
I20260709 12:15:03.217530 20446 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.224720 20446 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.225485 20231 raft_consensus.cc:2764] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
W20260709 12:15:03.250339 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800338cc0 after lost signal to thread 11313
W20260709 12:15:03.259213 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 988) took 2754 ms (client timeout 2261 ms). Trace:
W20260709 12:15:03.259737 12638 rpcz_store.cc:269] 0709 12:15:00.504415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.504593 (+   178us) service_pool.cc:224] Handling call
0709 12:15:03.258932 (+2754339us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:03.259112 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1177684736}
W20260709 12:15:03.261044 20406 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.553s	user 0.012s	sys 0.029s
I20260709 12:15:03.265085 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1009) took 2839 ms. Trace:
I20260709 12:15:03.265384 12392 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 5 to 11, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 11 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:15:03.265537 12490 rpcz_store.cc:276] 0709 12:15:00.426084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.426274 (+   190us) service_pool.cc:224] Handling call
0709 12:15:00.437080 (+ 10806us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:00.437530 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:00.437566 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:00.438197 (+   631us) log.cc:844] Serialized 52 byte log entry
0709 12:15:00.505682 (+ 67485us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:00.552885 (+ 47203us) log.cc:844] Serialized 14 byte log entry
0709 12:15:00.815754 (+262869us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b18002bed00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:15:00.815966 (+   212us) log.cc:844] Serialized 14 byte log entry
0709 12:15:00.816190 (+   224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:00.816277 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:01.273042 (+456765us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b18002bed00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:02.845160 (+1572118us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b18002bed00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:03.260488 (+415328us) spinlock_profiling.cc:243] Waited 9.14 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:03.264816 (+  4328us) raft_consensus.cc:1685] finished
0709 12:15:03.264839 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:03.265016 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":173727,"prepare.queue_time_us":319224,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":89,"raft.queue_time_us":3281,"raft.run_cpu_time_us":431,"raft.run_wall_time_us":13986,"spinlock_wait_cycles":52812544,"thread_start_us":67008,"threads_started":1,"wal-append.queue_time_us":214333}
I20260709 12:15:03.267323 12392 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 7 to 16, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 16 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:03.278473 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.279579 12571 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:03.283051 12563 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
W20260709 12:15:03.292374 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1104) took 2702 ms (client timeout 1566 ms). Trace:
W20260709 12:15:03.293042 13254 rpcz_store.cc:269] 0709 12:15:00.590166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:00.590376 (+   210us) service_pool.cc:224] Handling call
0709 12:15:03.291973 (+2701597us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:03.292218 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230962432}
I20260709 12:15:03.299737 20198 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:15:03.300745 20198 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.301323 20198 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.306206 20198 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:03.313154 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6860 after lost signal to thread 12429
W20260709 12:15:03.314677 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800337ae0 after lost signal to thread 12521
W20260709 12:15:03.314988 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800336b20 after lost signal to thread 12522
W20260709 12:15:03.321267 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.654s	user 0.015s	sys 0.068s
W20260709 12:15:03.322024 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.655s	user 0.015s	sys 0.068s
W20260709 12:15:03.322412 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 2.65876e+06us
W20260709 12:15:03.325341 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002e61a0 after lost signal to thread 12596
W20260709 12:15:03.325803 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6160 after lost signal to thread 12597
W20260709 12:15:03.326995 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800389c80 after lost signal to thread 12670
W20260709 12:15:03.328285 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080035a8e0 after lost signal to thread 13286
W20260709 12:15:03.328562 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002b6fc0 after lost signal to thread 13287
W20260709 12:15:03.328802 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08003a4cc0 after lost signal to thread 19957
I20260709 12:15:03.326403 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:03.329526 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08003d4b60 after lost signal to thread 20057
W20260709 12:15:03.329882 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800366cc0 after lost signal to thread 20090
I20260709 12:15:03.329798 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
W20260709 12:15:03.330410 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080037e820 after lost signal to thread 20112
W20260709 12:15:03.330716 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800350980 after lost signal to thread 20140
W20260709 12:15:03.331019 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080036e600 after lost signal to thread 20235
W20260709 12:15:03.331301 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800270560 after lost signal to thread 20268
W20260709 12:15:03.331503 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002289c0 after lost signal to thread 20280
W20260709 12:15:03.332616 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080030b200 after lost signal to thread 20309
W20260709 12:15:03.333006 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08002c33a0 after lost signal to thread 20333
I20260709 12:15:03.336164 20353 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:03.337528 20440 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:03.338044 20440 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:03.338953 20440 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.340408 20440 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:03.333742 20429 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:03.339335 13255 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:03.345482 20429 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.351137 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.352583 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3513) took 2519 ms. Trace:
I20260709 12:15:03.353050 12496 rpcz_store.cc:276] 0709 12:15:00.833364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.693519 (+1860155us) service_pool.cc:224] Handling call
0709 12:15:02.694415 (+   896us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:02.694830 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:02.705738 (+ 10908us) log.cc:844] Serialized 14 byte log entry
0709 12:15:03.352188 (+646450us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:03.352245 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:03.352314 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:03.352392 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:03.352518 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":643939,"prepare.queue_time_us":31130,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"thread_start_us":11626,"threads_started":2,"wal-append.queue_time_us":15764}
W20260709 12:15:03.355245 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080032cbc0 after lost signal to thread 20362
W20260709 12:15:03.355857 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08003d3480 after lost signal to thread 20382
W20260709 12:15:03.356359 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080029e500 after lost signal to thread 20404
I20260709 12:15:03.365562 13253 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
W20260709 12:15:03.366236 12462 connection.cc:256] RPC call timeout handler was delayed by -1.0115394429999469s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
I20260709 12:15:03.366658 13255 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:03.368103 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:03.372114 12569 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:15:03.373858 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:03.374727 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:03.374405 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:03.375597 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.378926 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:03.379668 13253 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:03.379786 12388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 127) took 30478 ms (client timeout 15000 ms). Trace:
I20260709 12:15:03.377588 20381 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.382393 20429 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:03.383253 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 241890us
I20260709 12:15:03.384840 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:03.384943 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.385483 13262 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.382126 20381 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.387034 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:03.390307 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:03.391418 13260 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:15:03.389158 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 204894us
I20260709 12:15:03.358191 20360 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:03.395198 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:03.400019 12611 connection.cc:256] RPC call timeout handler was delayed by -1.063051231000145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
I20260709 12:15:03.402531 13257 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:03.377681 19999 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.393910 20429 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:03.391767 20086 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:03.416724 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.396425 20199 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:03.422850 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.423467 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:03.424624 12637 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:15:03.424968 12637 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 10
I20260709 12:15:03.425181 12637 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:15:03.425390 12637 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:03.416633 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.427342 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3514) took 2584 ms. Trace:
I20260709 12:15:03.419461 12643 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
W20260709 12:15:03.415827 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46480.4us
I20260709 12:15:03.394826 20442 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.419001 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:03.415479 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 208488us
I20260709 12:15:03.398627 20198 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:03.434845 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:03.415683 20358 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.438083 20235 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.419430 20359 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:03.449955 20359 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:03.449535 20235 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.450392 20359 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:03.413132 20238 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:03.451076 20238 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.452286 20235 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.415287 20408 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.432781 20211 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.453960 20408 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.456187 20408 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.394549 20430 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
W20260709 12:15:03.380228 12388 rpcz_store.cc:269] 0709 12:14:32.925780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:33.034619 (+108839us) service_pool.cc:224] Handling call
0709 12:15:03.379516 (+30344897us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:14:53.348874 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:14:53.511338 (+162464us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:14:53.518164 (+  6826us) write_op.cc:435] Acquired schema lock
0709 12:14:53.519819 (+  1655us) tablet.cc:662] Decoding operations
0709 12:14:53.578610 (+ 58791us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:14:53.578730 (+   120us) write_op.cc:641] Partition lock acquired for write op
0709 12:14:53.578765 (+    35us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:14:53.580054 (+  1289us) tablet.cc:701] Row locks acquired
0709 12:14:53.580113 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:14:53.580287 (+   174us) write_op.cc:270] Start()
0709 12:14:53.580459 (+   172us) write_op.cc:276] Timestamp: P: 1783599293580216 usec, L: 0
0709 12:14:53.580481 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:14:53.646503 (+ 66022us) log.cc:844] Serialized 4750 byte log entry
0709 12:14:55.110753 (+1464250us) spinlock_profiling.cc:243] Waited 5.51 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:14:55.435167 (+324414us) op_driver.cc:464] REPLICATION: finished
0709 12:14:55.850432 (+415265us) write_op.cc:301] APPLY: starting
0709 12:14:55.850535 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 12:14:56.630510 (+779975us) tablet.cc:1370] finished BulkCheckPresence
0709 12:14:56.649739 (+ 19229us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:14:56.840690 (+190951us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:14:56.841342 (+   652us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=0,mrs_lookups=0
0709 12:14:56.841789 (+   447us) write_op.cc:312] APPLY: finished
0709 12:14:56.842704 (+   915us) log.cc:844] Serialized 113 byte log entry
0709 12:14:56.843105 (+   401us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:56.843352 (+   247us) write_op.cc:454] Released schema lock
0709 12:14:56.843448 (+    96us) write_op.cc:341] FINISH: Updating metrics
0709 12:14:57.087839 (+244391us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b18001ffde0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:57.088175 (+   336us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:14:57.088247 (+    72us) write_op.cc:454] Released schema lock
0709 12:14:57.105508 (+ 17261us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b18001ffde0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:14:57.108768 (+  3260us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f032a 00007ffbf5775f14 00007ffbf5775ec3 00007ffbf5775e38 00007ffbf5775df3 00007ffbf57750dc 00007ffbf579ee16 00007ffbf5786cbb 00007ffbf577139d 00007ffbf5774d75
Metrics: {"mutex_wait_us":6173328,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":398056,"apply.run_cpu_time_us":41288,"apply.run_wall_time_us":993795,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54239,"num_ops":12,"prepare.queue_time_us":350570,"prepare.run_cpu_time_us":57859,"prepare.run_wall_time_us":2033501,"raft.queue_time_us":10361,"raft.run_cpu_time_us":30901,"raft.run_wall_time_us":1917043,"replication_time_us":1774532,"spinlock_wait_cycles":383708032,"thread_start_us":1961014,"threads_started":4,"wal-append.queue_time_us":1763550,"wal-append.run_cpu_time_us":8254,"wal-append.run_wall_time_us":2018347}]]}
I20260709 12:15:03.457130 20430 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:03.457782 20430 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:03.425997 19999 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.458913 20430 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.459581 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:03.460664 20430 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:03.403679 20356 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.395993 20360 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.433027 12641 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:15:03.394403 20409 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:15:03.418928 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.417900 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:03.432121 12640 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:03.404496 20443 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.475881 20443 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.427783 12493 rpcz_store.cc:276] 0709 12:15:00.843123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:02.696126 (+1853003us) service_pool.cc:224] Handling call
0709 12:15:02.696590 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:02.696861 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:02.704073 (+  7212us) log.cc:844] Serialized 14 byte log entry
0709 12:15:03.412789 (+708716us) log.cc:844] Serialized 14 byte log entry
0709 12:15:03.426998 (+ 14209us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:03.427052 (+    54us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:03.427116 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:03.427181 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:03.427288 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":657440,"prepare.queue_time_us":30604,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":57,"raft.queue_time_us":655277,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":69730,"threads_started":3,"wal-append.queue_time_us":44293}
I20260709 12:15:03.400249 20404 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:15:03.437263 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.435424 20380 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.442000 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.436275 20401 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:03.435734 20407 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.466249 20442 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.443053 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.449168 20358 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.418236 20429 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:03.430141 20086 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.432842 20199 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.430011 20390 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260709 12:15:03.493187 20390 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:03.493593 20390 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:03.494441 20199 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.494513 20390 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.496179 20390 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:03.483269 20442 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.448495 20048 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:03.497540 19999 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.450429 20381 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.497927 20048 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:03.454092 20211 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.498461 20048 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:03.500061 20211 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.422261 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.501497 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.419674 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.462859 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:03.503031 20446 raft_consensus.cc:2764] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:15:03.503479 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.503705 20437 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:15:03.504277 12646 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19.
I20260709 12:15:03.463181 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.505134 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:03.432061 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47181.2us
I20260709 12:15:03.506421 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.483771 12392 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 9 to 12, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 12 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:03.506989 12639 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.484280 20343 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.510111 20343 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.511128 12563 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 17. Current term is 18. Ops: []
I20260709 12:15:03.511289 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.512259 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.512486 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.513098 12570 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:03.422161 20322 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.473519 20409 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:03.513895 20409 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:03.513831 20322 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.514422 13253 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:03.483533 20429 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:03.515264 20409 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.517009 20409 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:03.478348 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.480089 12638 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:03.480693 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.508883 12392 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 7 to 11, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 11 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:03.480767 12642 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:03.480396 20380 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:03.490399 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72586.4us
I20260709 12:15:03.490957 20057 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.477706 20356 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.546116 12392 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 8 to 10, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 10 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:15:03.491411 20086 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.492056 20112 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:03.481600 20407 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.513236 12571 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 26 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 11.
I20260709 12:15:03.475099 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.517738 12646 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.525027 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.503521 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:03.505908 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.529356 12570 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:03.529093 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.530308 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.531719 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:03.470225 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:03.532765 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.538375 13254 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:03.539521 12637 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:03.490986 20376 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:03.603885 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.542265 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.542343 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.556675 12567 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:03.613286 12565 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:03.483801 20358 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.468786 20238 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.544518 20380 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.545276 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.537163 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:03.477581 20360 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.478379 20443 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.542369 12566 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:03.614565 13260 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:03.614411 12563 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
W20260709 12:15:03.615373 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111919us
I20260709 12:15:03.617173 12643 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:15:03.621775 12642 raft_consensus.cc:2376] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20. Re-sending same reply.
W20260709 12:15:03.622155 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84996.2us
W20260709 12:15:03.624862 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 190095us
I20260709 12:15:03.626246 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.626961 12638 raft_consensus.cc:2368] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23.
I20260709 12:15:03.626726 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.633278 12637 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:03.634939 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:03.636878 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.637727 12565 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:03.636410 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.555990 20112 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:03.650056 12569 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:03.651299 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.651299 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.652261 12569 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 26 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:15:03.612412 20381 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.660961 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.654964 12641 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19.
I20260709 12:15:03.661723 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.626792 20446 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:03.664947 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.662885 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:03.665014 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.629925 20366 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:03.670665 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.672158 12571 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:03.664378 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.664629 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.648536 12392 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:03.673643 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.666327 12492 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:03.666549 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:03.663535 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.664076 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.675449 12489 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:03.667071 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.664076 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.668684 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.676373 12569 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:15:03.676800 12637 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:03.653517 20238 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:03.678040 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.678202 20380 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:15:03.678673 12641 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.678882 20199 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:03.679195 20359 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.679944 20199 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:03.679883 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.665450 20446 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:03.681010 20446 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.546116 20057 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.674288 12490 raft_consensus.cc:2376] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24. Re-sending same reply.
I20260709 12:15:03.680555 20199 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.679308 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.675983 12642 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:15:03.685750 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:03.683744 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.689177 12495 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 18: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:03.688318 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.691651 12569 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:15:03.654004 20112 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:03.692247 12392 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 8 to 12. New cstate: current_term: 12 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:15:03.705088 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.706789 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:03.703330 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 266985us
I20260709 12:15:03.625726 20376 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.710371 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.712505 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.714491 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.714747 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.715292 12490 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:03.716858 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.719775 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.721335 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.723948 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.724663 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:03.689707 20238 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:03.730154 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 8 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.731033 12489 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:15:03.730703 20238 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.734103 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.734941 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:03.736143 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.736774 12488 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:03.670063 20381 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.714161 20322 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.689508 20401 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.056s
I20260709 12:15:03.746901 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.748797 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.750291 12495 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:03.752513 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.684239 20359 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.690793 20366 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:03.792702 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.755620 12638 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:03.793699 12565 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:03.795919 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.796602 12565 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21. Re-sending same reply.
I20260709 12:15:03.755975 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:03.756170 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.798671 12642 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:03.798774 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.686810 20358 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 12:15:03.757833 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.802445 12646 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:03.803470 12492 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:03.757409 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.803743 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.804345 12491 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 9.
I20260709 12:15:03.804090 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.761282 20322 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:03.712188 20343 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.762792 20407 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.762655 20459 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:03.709508 20408 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:03.766274 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.761020 20048 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.709523 20086 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:15:03.769881 20443 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:03.754645 20198 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:03.771307 20057 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:03.858072 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.859246 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.859450 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.860095 12492 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:03.771944 20359 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.774020 20458 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:03.775774 19999 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.863590 19999 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.865057 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.741133 20381 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.786057 20456 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:03.708304 20380 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:15:03.788627 20376 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:03.874138 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.875084 13253 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:03.789160 12611 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20260709 12:15:03.790367 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:03.799610 12565 raft_consensus.cc:2376] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21. Re-sending same reply.
I20260709 12:15:03.800529 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.800915 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.760807 12571 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.10-22.10]
I20260709 12:15:03.802503 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.804996 12646 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.877964 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:03.805377 12497 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:03.878952 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.879556 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.880349 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:03.885442 12563 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:03.887179 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.888135 12639 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:15:03.807370 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:03.762136 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.890306 12490 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:03.823753 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.891418 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:03.825297 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.831370 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:03.831372 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.835840 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.839424 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.840147 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.894155 12567 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:03.894459 13262 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:03.840322 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.895318 13255 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:03.850540 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.849586 12489 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:03.693749 20356 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.897221 12640 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:03.898036 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.899055 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:03.900343 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.901955 12569 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: []
I20260709 12:15:03.903272 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.861939 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.851300 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:03.866202 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:03.776203 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:03.867758 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.912762 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.868184 12494 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:15:03.869495 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:03.861610 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:03.915694 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.916677 12565 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:03.916838 12564 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:03.889374 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.919577 12639 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:03.923107 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:03.854292 20408 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:03.928601 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.934338 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:03.935139 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:03.936290 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.939960 12492 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:03.944459 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:03.945343 12642 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:03.945735 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.874795 20430 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:03.952771 12566 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:03.957156 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.957578 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.957731 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:03.958606 12572 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:03.963378 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.964140 12495 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:03.958317 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:03.965273 12488 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:03.966531 12491 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:03.970415 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:03.971153 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48145.4us
I20260709 12:15:03.973179 20048 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:03.974510 20322 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:03.975265 20322 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:03.976261 20322 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:03.974577 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.971310 12569 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:04.000653 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:03.973313 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:03.861960 20359 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.940972 20408 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:04.009500 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:03.866657 20429 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.006378 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:04.015920 12564 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:04.016737 12492 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:04.017197 12565 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:04.019193 12494 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:04.020857 20458 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:15:04.026551 20390 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.530s	user 0.010s	sys 0.004s
I20260709 12:15:03.958976 20440 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:04.032694 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:03.959165 20430 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:04.028128 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:04.031584 12572 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:04.034305 12565 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:04.037909 12642 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:03.897078 20456 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.042620 12569 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:03.875953 12611 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2826)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2803)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2804)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3198)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2805)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3210)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2807)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3197)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3204)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2813)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2814)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2815)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2817)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2819)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2824)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2825)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3212)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3213)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3216)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3218)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3221)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3230)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3232)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3234)
I20260709 12:15:04.045220 12565 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:04.014675 19999 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:04.050027 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.047694 12564 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:15:04.070704 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:04.071138 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:04.065105 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.028759 20433 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:04.072541 12572 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:04.075477 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:04.076800 12488 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:04.077924 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:04.078966 12488 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:04.040675 20430 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:04.081527 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:04.087281 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.087625 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:04.088840 12493 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:04.089179 12488 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:15:04.039577 20440 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.097568 20440 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:04.101289 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.102150 12564 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:04.104532 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:04.105412 20469 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:04.105958 20469 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:04.106575 20469 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:04.106799 20455 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:15:04.107239 20455 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:04.108337 20455 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:03.978703 20366 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:04.109673 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.110378 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:04.111605 12489 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:15:04.111860 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:04.112250 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:04.112883 13254 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:04.105877 20376 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:04.115479 20455 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:15:04.120262 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.121341 12566 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:04.118100 20404 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:04.122294 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.123238 12567 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:04.124208 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.125133 12568 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:15:04.126077 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.121282 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.127259 12563 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:04.113272 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.114143 20440 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:04.122316 20404 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:04.130934 20440 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:04.131534 20440 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:04.132186 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.131951 20404 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.132922 12641 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:04.134366 12564 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:15:04.134773 12491 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:15:04.120316 20376 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.136222 12491 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:15:04.120086 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.136528 20376 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.137428 12645 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.137892 20231 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:04.138531 20231 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.139531 20231 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.140595 20231 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:04.142472 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.143374 12639 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:04.146212 20353 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:04.146940 20353 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:04.118795 20359 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:04.142555 20462 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
W20260709 12:15:04.147672 20466 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:15:04.149304 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.150820 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.151036 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.151388 12492 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: [19.11-22.12]
I20260709 12:15:04.151892 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.092211 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:04.153808 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.154538 12642 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:15:04.113214 12489 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:15:04.160346 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.161100 12644 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:04.167124 12491 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: [15.13-18.15]
I20260709 12:15:04.167937 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:04.169847 12643 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.538s	user 0.007s	sys 0.011s
I20260709 12:15:04.170101 12569 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.170390 12643 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:15:04.153262 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.171586 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.168671 12646 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:04.172428 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
W20260709 12:15:04.172600 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.173977 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.174764 12638 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 19: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:04.176105 12490 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
W20260709 12:15:04.168561 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 307043us
I20260709 12:15:04.179582 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.171656 12641 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:04.180540 12640 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:15:04.138000 20376 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:04.173219 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.177153 12489 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:15:04.182765 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.182839 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.183871 12642 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:04.188295 12495 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:04.188568 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:04.189956 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.195338 12571 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 16. Current term is 17. Ops: []
I20260709 12:15:04.195878 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.197067 12644 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:04.199975 12496 pending_rounds.cc:85] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 13
I20260709 12:15:04.200536 12496 pending_rounds.cc:107] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:15:04.092145 20433 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:04.201151 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:04.201210 12496 raft_consensus.cc:2889] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:04.216074 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.213011 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.217759 12646 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21.
I20260709 12:15:04.190466 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.175474 20211 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:04.166739 20444 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:04.221222 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.214350 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.225385 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.234325 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.230181 12638 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 21.
I20260709 12:15:04.230825 12490 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:15:04.234781 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.235707 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:04.230206 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.237499 20444 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.238772 20444 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.222208 20470 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:04.242022 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.242509 20470 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:04.242542 20359 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:04.243395 12641 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:04.233143 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.237490 20198 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.250864 12639 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:04.256503 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.252787 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:04.224248 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:04.258160 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.212998 20459 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:04.264511 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.263233 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.267172 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.268455 12645 raft_consensus.cc:2368] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23.
I20260709 12:15:04.268797 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.257464 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.273738 12638 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:15:04.270730 12461 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19.
I20260709 12:15:04.274392 12461 leader_election.cc:403] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:15:04.275019 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.223645 20211 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:04.275851 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.280599 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:04.282234 12639 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:15:04.283262 13260 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:04.283625 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:04.282691 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:04.197101 20112 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.287163 13262 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:04.287519 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.245349 20470 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:04.260897 20390 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:04.288828 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:04.292685 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.297237 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:04.287834 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.298561 13262 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:15:04.289688 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.299510 12641 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:04.300432 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.296584 12646 raft_consensus.cc:2376] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24. Re-sending same reply.
I20260709 12:15:04.295234 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.303908 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:04.304363 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.303755 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.306102 12640 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:04.306999 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.223096 19999 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:15:04.316720 12645 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:15:04.288235 20459 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:15:04.331729 20456 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:04.332036 20459 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:04.332588 20459 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.288904 20211 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.305181 20437 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:04.266896 20235 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:15:04.337186 20235 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:04.315430 20198 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:15:04.305423 20359 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:15:04.331182 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73791.5us
I20260709 12:15:04.305557 20390 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:04.338732 20390 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:04.338766 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.340286 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.341885 20359 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.341967 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.342710 20359 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.342958 12644 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:04.344480 12644 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:15:04.333559 20459 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.349273 20459 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:04.338187 20198 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:04.351060 12497 raft_consensus.cc:2368] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23.
I20260709 12:15:04.318929 13261 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:04.335968 20404 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:04.317390 12641 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: [23.12-24.13]
I20260709 12:15:04.351495 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.316982 20260 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:04.285319 20460 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:15:04.370047 12641 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: [18.11-20.12]
I20260709 12:15:04.370925 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.371234 20020 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:15:04.371520 12641 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:15:04.371802 20020 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.372982 12641 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:15:04.372740 20020 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.373960 20465 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:04.374070 20469 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:04.375083 20199 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.665s	user 0.009s	sys 0.002s
I20260709 12:15:04.374996 20380 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:04.375597 20199 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.376022 20380 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:04.376355 20380 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:04.376407 20469 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:04.272257 20057 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:15:04.263528 20381 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:15:04.378906 20381 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:04.379249 20381 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:04.380088 20381 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.381269 20381 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.373138 20463 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:04.327783 20446 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.646s	user 0.019s	sys 0.003s
I20260709 12:15:04.386332 20446 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.386790 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.388206 12565 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:15:04.269470 20443 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:15:04.390710 12638 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.15-21.15]
W20260709 12:15:04.353678 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53963.2us
I20260709 12:15:04.392796 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.390923 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.405104 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:04.406023 13261 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:04.405844 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.401192 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.406898 12641 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:04.404124 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:04.410022 12490 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:15:04.329118 20111 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19.
I20260709 12:15:04.436499 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:04.437291 12494 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.329944 20431 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.411439 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2806) took 1002 ms. Trace:
I20260709 12:15:04.412006 12497 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:04.416468 12642 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:04.437928 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:04.442680 12639 rpcz_store.cc:276] 0709 12:15:03.409304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.256082 (+846778us) service_pool.cc:224] Handling call
0709 12:15:04.411205 (+155123us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:04.411373 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56653184}
W20260709 12:15:04.418473 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 305250us
W20260709 12:15:04.426548 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 140694us
W20260709 12:15:04.427825 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 180007us
I20260709 12:15:04.411598 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.337671 20235 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.415074 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:04.446838 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:04.450927 12490 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:04.363637 20198 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:04.489615 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.402683 20456 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:15:04.495543 20409 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.959s	user 0.012s	sys 0.000s
I20260709 12:15:04.388993 20057 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:04.477917 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:04.438971 20429 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:04.440876 20048 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:15:04.438607 20086 raft_consensus.cc:2764] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:15:04.464852 20433 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.482638 12639 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:04.373394 20430 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:04.486052 20474 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:15:04.488936 20198 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.476310 20020 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:15:04.498961 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108122us
W20260709 12:15:04.499310 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87811.3us
W20260709 12:15:04.500046 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53853us
W20260709 12:15:04.475950 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.500649 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:04.503386 12497 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:04.535704 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.274767 20353 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.353109 20408 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.437507 20199 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:04.520170 12638 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:15:04.523296 12494 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:04.377322 20380 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.505033 20401 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:04.502213 20238 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:15:04.503276 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 369307us
W20260709 12:15:04.538518 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 269774us
I20260709 12:15:04.540953 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.502962 20057 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.533314 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:04.532001 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:04.560977 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.564656 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:04.565263 12572 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:04.541752 12493 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:15:04.533039 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:04.548547 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:04.570179 12564 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:04.570359 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:04.517905 20343 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:04.571672 12637 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:04.510335 20440 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.561357 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:04.574182 12491 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:15:04.571900 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.580204 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.582197 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.582377 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:04.578667 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.586396 13218 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21.
I20260709 12:15:04.596035 13218 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:15:04.557740 20401 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:04.555751 20443 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:04.597110 12638 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:15:04.598690 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:04.614383 13261 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:04.614687 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:04.617969 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55888.6us
W20260709 12:15:04.601526 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 152989us
I20260709 12:15:04.588596 12497 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:04.557031 20446 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:04.609848 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:04.557458 20198 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.585899 20407 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:04.568030 20437 raft_consensus.cc:2764] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:15:04.574638 20048 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:04.595817 20429 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:15:04.586323 20458 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:04.595688 20380 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:04.595505 20456 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:15:04.586220 20463 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:04.646981 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.595804 20469 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:04.646512 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:04.677205 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:04.578926 20260 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:04.611819 20401 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:04.605688 20238 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:04.688428 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2820) took 1241 ms. Trace:
I20260709 12:15:04.542239 20235 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.604789 20199 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 12:15:04.611217 20443 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:04.586270 20441 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.066s
W20260709 12:15:04.703909 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132086us
W20260709 12:15:04.712504 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128519us
I20260709 12:15:04.714601 12637 rpcz_store.cc:276] 0709 12:15:03.446621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:03.684667 (+238046us) service_pool.cc:224] Handling call
0709 12:15:04.687749 (+1003082us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:04.688310 (+   561us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1978609536}
I20260709 12:15:04.596140 20472 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.114s
I20260709 12:15:04.679769 20407 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:04.714520 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 131292us
I20260709 12:15:04.712620 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:04.717711 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71351.2us
I20260709 12:15:04.596246 20481 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.118s
W20260709 12:15:04.719790 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73888us
W20260709 12:15:04.721074 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116639us
W20260709 12:15:04.724094 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.725716 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:04.727011 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.701932 20429 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:04.724833 20443 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:04.714058 20469 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:04.735013 20437 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.733898 20360 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:04.765689 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53152.8us
I20260709 12:15:04.754009 20048 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:04.769062 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.753093 20260 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:04.785876 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:04.788755 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.748157 20407 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:04.774431 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48845.7us
I20260709 12:15:04.778108 20235 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:04.781107 20469 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:04.764664 20473 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21.
I20260709 12:15:04.799227 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:04.778877 20366 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:04.764024 20390 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:04.842036 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54063.2us
W20260709 12:15:04.844099 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46860.4us
I20260709 12:15:04.793897 20465 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.777376 20409 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
W20260709 12:15:04.849588 20488 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:15:04.850695 20484 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:15:04.756994 20057 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.852429 20057 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:04.852722 20483 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:15:04.837972 20048 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 21.8, Last appended by leader: 8, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.778735 20479 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:15:04.854629 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.794162 20446 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:15:04.855504 20366 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:04.855933 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.797554 20359 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:04.800414 20476 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:04.834977 20408 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:04.784694 20475 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:15:04.876199 12493 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.14-20.14]
W20260709 12:15:04.862558 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76754.8us
I20260709 12:15:04.881001 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:04.864316 12536 leader_election.cc:400] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23.
I20260709 12:15:04.865020 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:04.882792 12536 leader_election.cc:403] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:15:04.832589 20430 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:04.869282 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1015) took 1086 ms. Trace:
I20260709 12:15:04.845258 20481 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.861832 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:04.872524 12495 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:04.885242 12496 rpcz_store.cc:276] 0709 12:15:03.782936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.156482 (+373546us) service_pool.cc:224] Handling call
0709 12:15:04.199617 (+ 43135us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:04.232664 (+ 33047us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.232703 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:04.233418 (+   715us) log.cc:844] Serialized 52 byte log entry
0709 12:15:04.294117 (+ 60699us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b1800485560. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:04.528672 (+234555us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:04.711374 (+182702us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.718980 (+  7606us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.731476 (+ 12496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.731610 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:04.868950 (+137340us) raft_consensus.cc:1685] finished
0709 12:15:04.868980 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:04.869201 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34439,"prepare.queue_time_us":402087,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":3590,"raft.queue_time_us":36749,"raft.run_cpu_time_us":571,"raft.run_wall_time_us":22950,"spinlock_wait_cycles":4384512,"thread_start_us":390728,"threads_started":2,"wal-append.queue_time_us":339174}
W20260709 12:15:04.844182 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69742.9us
I20260709 12:15:04.856590 20366 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.800428 20437 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:15:04.871778 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:04.870796 20473 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:04.898554 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:04.896392 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:04.884390 20441 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:04.870400 20476 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:15:04.911262 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63644.2us
I20260709 12:15:04.928929 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:04.895639 20111 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:04.888481 20495 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:04.889901 20440 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:04.923553 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:04.869899 20493 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:15:04.931370 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3212) took 1243 ms. Trace:
I20260709 12:15:04.937043 12564 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:04.936021 12644 rpcz_store.cc:276] 0709 12:15:03.688379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.345608 (+657229us) service_pool.cc:224] Handling call
0709 12:15:04.351618 (+  6010us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.367542 (+ 15924us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.367578 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.402279 (+ 34701us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.583762 (+181483us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:04.796120 (+212358us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.930397 (+134277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.930505 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:04.930555 (+    50us) raft_consensus.cc:1685] finished
0709 12:15:04.930570 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:04.931291 (+   721us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":102739,"prepare.queue_time_us":203791,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":36,"raft.queue_time_us":102069,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":5347,"thread_start_us":318842,"threads_started":3,"wal-append.queue_time_us":277773}
I20260709 12:15:04.937561 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:04.914788 20430 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:04.943734 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:04.919126 20481 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:04.946466 12644 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:04.884559 20429 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.931018 20476 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:04.933740 19999 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:04.943727 20111 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:15:04.962944 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.963380 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:04.959105 20486 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:15:04.959815 20494 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:15:04.941422 20473 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:15:04.963740 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:04.958137 20500 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:15:04.991029 20211 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.001s
I20260709 12:15:04.916494 20260 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:04.993716 20509 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:04.998301 20491 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:15:04.998998 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1022) took 1214 ms. Trace:
I20260709 12:15:05.000384 20390 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.015033 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:05.031203 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2814) took 1586 ms. Trace:
I20260709 12:15:05.052207 12640 rpcz_store.cc:276] 0709 12:15:03.444965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.307078 (+862113us) service_pool.cc:224] Handling call
0709 12:15:04.328345 (+ 21267us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:15:04.329222 (+   877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.329312 (+    90us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:15:04.349663 (+ 20351us) log.cc:844] Serialized 78 byte log entry
0709 12:15:04.509249 (+159586us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:15:05.030322 (+521073us) log.cc:844] Serialized 14 byte log entry
0709 12:15:05.030707 (+   385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.030815 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.030892 (+    77us) raft_consensus.cc:1685] finished
0709 12:15:05.030910 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.031120 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":121021,"prepare.queue_time_us":41657,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":303,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":11302,"thread_start_us":624757,"threads_started":2,"wal-append.queue_time_us":217769}
I20260709 12:15:04.953977 20020 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:05.035108 20404 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.003s
I20260709 12:15:04.978354 19999 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:05.046512 12488 rpcz_store.cc:276] 0709 12:15:03.784849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.163838 (+378989us) service_pool.cc:224] Handling call
0709 12:15:04.165349 (+  1511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.168104 (+  2755us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.168137 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.182606 (+ 14469us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.511133 (+328527us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:04.511176 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.511248 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:04.998684 (+487436us) raft_consensus.cc:1685] finished
0709 12:15:04.998718 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:04.998910 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2172,"raft.queue_time_us":7809,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":4181,"thread_start_us":267463,"threads_started":1,"wal-append.queue_time_us":454922}
W20260709 12:15:05.039938 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 454945us
I20260709 12:15:05.088078 12643 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
W20260709 12:15:05.058327 20471 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.292s	user 0.000s	sys 0.002s
I20260709 12:15:05.050839 20509 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:05.064123 20499 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.236s	user 0.000s	sys 0.001s
I20260709 12:15:05.040046 20473 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:05.067636 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3232) took 1280 ms. Trace:
I20260709 12:15:05.038529 20211 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:04.956146 20481 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.074795 12644 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:05.075078 20112 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.112277 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3233) took 1325 ms. Trace:
I20260709 12:15:05.112818 12644 rpcz_store.cc:276] 0709 12:15:03.787027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.074137 (+1287110us) service_pool.cc:224] Handling call
0709 12:15:05.074568 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:05.111813 (+ 37245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.112161 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:05.115159 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 995) took 1012 ms. Trace:
I20260709 12:15:05.115628 12644 rpcz_store.cc:276] 0709 12:15:04.102273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.113931 (+1011658us) service_pool.cc:224] Handling call
0709 12:15:05.114427 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:05.114702 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:05.114742 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:05.114809 (+    67us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:05.114858 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.114933 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.115081 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:05.088845 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:05.095909 20356 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:05.105253 12640 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:05.034415 20494 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:04.954038 20430 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:05.108330 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145043us
I20260709 12:15:05.118642 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3234) took 1331 ms. Trace:
I20260709 12:15:05.124203 12640 rpcz_store.cc:276] 0709 12:15:03.787305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.075873 (+1288568us) service_pool.cc:224] Handling call
0709 12:15:05.104795 (+ 28922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:05.118236 (+ 13441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.118562 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:05.109261 12638 rpcz_store.cc:276] 0709 12:15:03.786726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.545078 (+758352us) service_pool.cc:224] Handling call
0709 12:15:04.545488 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:04.612626 (+ 67138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.067270 (+454644us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:05.067516 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117722496}
I20260709 12:15:05.126708 12638 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:05.128010 12638 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:05.120630 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 996) took 1018 ms. Trace:
I20260709 12:15:05.055190 20477 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:05.125315 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:05.128756 12644 rpcz_store.cc:276] 0709 12:15:04.102538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.116866 (+1014328us) service_pool.cc:224] Handling call
0709 12:15:05.117284 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:05.120089 (+  2805us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:05.120135 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:05.120209 (+    74us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:05.120269 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.120355 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.120524 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:05.135187 13260 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:15:05.127862 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:05.090876 19999 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:05.136656 12640 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:05.137228 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 368201us
I20260709 12:15:05.138602 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:05.090178 20433 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:05.087548 20446 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23.
I20260709 12:15:05.135720 12644 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:05.141223 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:05.145305 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115544us
W20260709 12:15:05.122623 20484 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:15:05.147379 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:05.144842 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:05.129992 20470 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.743s	user 0.006s	sys 0.007s
W20260709 12:15:05.160204 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:05.163630 12644 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:15:05.164678 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:05.132931 20440 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.181344 12644 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:15:05.148291 20048 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:05.193605 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.145607 20433 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:05.110981 20401 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.134375 20456 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.174291 20444 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:15:05.181481 20520 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:05.191332 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:05.207988 20513 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:15:05.209717 20518 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:15:05.210976 20498 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.211608 20498 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.181887 20112 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:05.199273 20433 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:05.217952 20510 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:05.218775 20502 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:05.220752 20489 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 12:15:05.222851 20456 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.218791 20211 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.213356 20524 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:05.245929 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98714us
W20260709 12:15:05.245886 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81288.3us
I20260709 12:15:05.279165 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:05.280889 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:05.170171 20470 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.188349 20440 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.206490 20048 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.181821 20353 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:15:05.286217 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:05.294476 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135434us
I20260709 12:15:05.212733 20494 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.297219 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:05.282655 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 353662us
I20260709 12:15:05.301080 12572 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:05.301887 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.302564 12644 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:05.237367 20520 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.303871 12461 leader_election.cc:400] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23.
I20260709 12:15:05.307219 12461 leader_election.cc:403] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:15:05.308318 20443 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.524s	user 0.025s	sys 0.018s
I20260709 12:15:05.308904 20443 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.291148 20524 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.310420 20470 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.311359 20443 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.313047 20401 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:05.317675 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:05.318055 12565 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:15:05.287731 20112 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:05.290894 20322 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:05.322235 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.316395 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.320693 12644 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:05.332310 12563 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:05.334333 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.295617 20479 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.296867 20477 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:05.296594 20444 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:05.342001 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3231) took 1555 ms. Trace:
I20260709 12:15:05.342700 12639 rpcz_store.cc:276] 0709 12:15:03.786440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.538290 (+751850us) service_pool.cc:224] Handling call
0709 12:15:04.556541 (+ 18251us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.558785 (+  2244us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:15:04.588249 (+ 29464us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.731901 (+143652us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.755376 (+ 23475us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.755646 (+   270us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:04.755952 (+   306us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.793915 (+ 37963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.794004 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.341374 (+547370us) raft_consensus.cc:1685] finished
0709 12:15:05.341422 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.341892 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46908,"prepare.queue_time_us":205795,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":4421,"raft.queue_time_us":233595,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":34504,"thread_start_us":116754,"threads_started":3,"wal-append.queue_time_us":174602}
I20260709 12:15:05.346570 12536 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19.
I20260709 12:15:05.347021 12536 leader_election.cc:403] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:15:05.318799 20502 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:15:05.394764 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100679us
W20260709 12:15:05.384102 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:05.395763 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:05.330142 20524 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:05.399998 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49374.2us
I20260709 12:15:05.420245 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.268599 20473 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.400271 20521 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 12:15:05.400880 20430 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:05.401789 20533 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:15:05.338335 20211 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23.
I20260709 12:15:05.410490 20444 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:15:05.394349 20380 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.684s	user 0.007s	sys 0.030s
I20260709 12:15:05.409318 20476 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.413123 20020 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:05.415872 20469 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.007s
W20260709 12:15:05.339336 20459 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.971s	user 0.012s	sys 0.006s
W20260709 12:15:05.339040 20198 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.661s	user 0.016s	sys 0.008s
I20260709 12:15:05.337476 20086 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:05.435422 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.336833 20539 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:05.340451 20112 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:05.454515 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148301us
I20260709 12:15:05.460039 12563 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:05.396749 20429 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19.
I20260709 12:15:05.348239 20446 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:05.450114 20524 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:05.517838 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:05.520109 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97391.8us
I20260709 12:15:05.520457 20198 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:05.521112 20198 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.522019 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:05.522842 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88673.1us
I20260709 12:15:05.523806 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.478081 20498 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:05.525854 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:05.446398 20502 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.530930 13260 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:05.522076 20198 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.10, Last appended by leader: 10, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.540915 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3542) took 1151 ms. Trace:
I20260709 12:15:05.493376 20509 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.541939 19999 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:05.543337 20490 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.762s	user 0.003s	sys 0.004s
I20260709 12:15:05.494683 20469 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.541421 12495 rpcz_store.cc:276] 0709 12:15:04.389210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.390666 (+  1456us) service_pool.cc:224] Handling call
0709 12:15:04.439868 (+ 49202us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:04.890176 (+450308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.540597 (+650421us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540020e4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:05.540836 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":994478464}
I20260709 12:15:05.545027 19999 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.496183 20481 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.494486 20539 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:05.349512 20479 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.527195 20431 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:05.566282 13262 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:05.454183 20048 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:05.559175 20381 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.158s	user 0.013s	sys 0.009s
I20260709 12:15:05.536844 20211 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:05.568764 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3220) took 1811 ms. Trace:
I20260709 12:15:05.573693 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:05.576948 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:05.576553 12641 rpcz_store.cc:276] 0709 12:15:03.757364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.407593 (+650229us) service_pool.cc:224] Handling call
0709 12:15:04.407963 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.436004 (+ 28041us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:04.501886 (+ 65882us) log.cc:844] Serialized 14 byte log entry
0709 12:15:05.092238 (+590352us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:05.093508 (+  1270us) log.cc:844] Serialized 26 byte log entry
0709 12:15:05.093915 (+   407us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:05.094849 (+   934us) log.cc:844] Serialized 14 byte log entry
0709 12:15:05.095417 (+   568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.095840 (+   423us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.542722 (+446882us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b24003e36a8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:15:05.568340 (+ 25618us) raft_consensus.cc:1685] finished
0709 12:15:05.568394 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.568660 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42817,"prepare.queue_time_us":70461,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":107,"raft.queue_time_us":145279,"raft.run_cpu_time_us":457,"raft.run_wall_time_us":37153,"spinlock_wait_cycles":93883136,"thread_start_us":552930,"threads_started":2,"wal-append.queue_time_us":190924}
I20260709 12:15:05.563151 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.537397 20446 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:05.581992 19999 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:05.582580 20455 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:05.583165 20455 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.583494 12639 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:05.581604 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2829) took 1283 ms. Trace:
I20260709 12:15:05.586817 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.587672 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.588706 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.585132 12638 rpcz_store.cc:276] 0709 12:15:04.298142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.133090 (+834948us) service_pool.cc:224] Handling call
0709 12:15:05.133418 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:05.133852 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:05.133881 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:05.134346 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:15:05.331484 (+197138us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:05.331545 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.331630 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.581335 (+249705us) raft_consensus.cc:1685] finished
0709 12:15:05.581364 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.581512 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":10659,"thread_start_us":156502,"threads_started":1,"wal-append.queue_time_us":296470}
I20260709 12:15:05.494652 20520 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.571319 20431 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:05.600121 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.602448 12644 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:05.602970 12639 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:15:05.600131 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:05.576172 20211 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:05.603755 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:05.608987 12638 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:15:05.609500 12638 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 13
I20260709 12:15:05.609836 12638 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:15:05.610137 12638 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:05.526752 20477 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.629863 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.604579 20431 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:05.638983 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3208) took 1979 ms. Trace:
I20260709 12:15:05.639554 12643 rpcz_store.cc:276] 0709 12:15:03.660088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.237087 (+576999us) service_pool.cc:224] Handling call
0709 12:15:05.638674 (+1401587us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:05.638895 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1885307648}
I20260709 12:15:05.598420 20440 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.596100 20112 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.614239 20539 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.606412 20538 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:05.634644 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:05.606352 20502 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.607484 20446 raft_consensus.cc:2764] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election decision vote started in defunct term 20: won
I20260709 12:15:05.667392 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:05.565068 20543 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:15:05.670284 20543 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:05.670845 20543 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:05.670547 20480 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:15:05.671819 20543 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.673911 20057 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:05.673821 20543 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:05.682535 20360 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.898s	user 0.016s	sys 0.000s
I20260709 12:15:05.683135 20360 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:05.684289 12641 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 19. (term mismatch)
I20260709 12:15:05.684754 12641 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 18
I20260709 12:15:05.685003 12641 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:15:05.685262 12641 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:05.685782 12491 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:05.610792 20211 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:05.633894 20548 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.708546 20548 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.655612 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:05.710013 12496 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:05.711165 20500 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.711750 20500 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.647297 20539 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:05.713941 12569 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:05.714460 12569 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:05.715034 20359 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:05.715384 12569 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.716224 20359 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:05.658316 12488 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:15:05.705127 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:05.646780 20442 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:05.660010 20458 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:05.735869 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:05.641476 20479 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.661530 20020 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:05.646384 20086 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:05.663211 20538 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:05.738519 20086 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:05.738687 20538 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:05.739729 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.584123 20455 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.663672 20443 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:15:05.743065 20380 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:05.664049 20481 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:05.689134 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:05.719370 20548 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:05.745592 20481 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:05.647585 20433 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.746402 20481 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:05.746838 12490 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 7. Preceding OpId from leader: term: 23 index: 9. (index mismatch)
I20260709 12:15:05.682574 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:05.747622 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2813) took 2302 ms. Trace:
I20260709 12:15:05.654978 12495 raft_consensus.cc:606] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:05.749193 20500 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.698786 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.720686 20531 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.755589 20531 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.758215 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1119) took 1256 ms. Trace:
I20260709 12:15:05.758942 20531 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:05.694944 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.758865 13261 rpcz_store.cc:276] 0709 12:15:04.501805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.502075 (+   270us) service_pool.cc:224] Handling call
0709 12:15:04.502470 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:04.647063 (+144593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.757953 (+1110890us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:05.758142 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":256813312}
W20260709 12:15:05.652673 20487 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.806s	user 0.003s	sys 0.000s
W20260709 12:15:05.649238 20535 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:15:05.676570 20322 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:15:05.762354 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3213) took 2071 ms. Trace:
I20260709 12:15:05.706192 20401 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.763986 12567 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:05.762910 12645 rpcz_store.cc:276] 0709 12:15:03.691080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.347181 (+656101us) service_pool.cc:224] Handling call
0709 12:15:04.347932 (+   751us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.366919 (+ 18987us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.366965 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.387508 (+ 20543us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.572348 (+184840us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:04.696426 (+124078us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.791545 (+ 95119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.791645 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.652093 (+860448us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b24003e3618. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:15:05.761738 (+109645us) raft_consensus.cc:1685] finished
0709 12:15:05.761971 (+   233us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.762272 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21882,"prepare.queue_time_us":205135,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":51,"raft.queue_time_us":148740,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":32223,"spinlock_wait_cycles":28845312,"thread_start_us":363847,"threads_started":3,"wal-append.queue_time_us":375988}
I20260709 12:15:05.736318 20437 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:05.716585 12569 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:05.765200 20437 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:05.735795 20231 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.766163 20231 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.766287 20437 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.768323 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3221) took 2010 ms. Trace:
I20260709 12:15:05.662642 20469 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.673246 20460 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.068s
I20260709 12:15:05.770275 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:05.768872 12642 rpcz_store.cc:276] 0709 12:15:03.757876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.443481 (+685605us) service_pool.cc:224] Handling call
0709 12:15:04.443879 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.505160 (+ 61281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.505201 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.505885 (+   684us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.646197 (+140312us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:15:04.755117 (+108920us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.755396 (+   279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.755503 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.766051 (+1010548us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b24003e3618. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:15:05.767886 (+  1835us) raft_consensus.cc:1685] finished
0709 12:15:05.768077 (+   191us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.768251 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":150471,"prepare.queue_time_us":141583,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":32,"raft.queue_time_us":170667,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":27591,"spinlock_wait_cycles":32424320,"thread_start_us":73510,"threads_started":1,"wal-append.queue_time_us":163121}
I20260709 12:15:05.771051 12645 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:05.687162 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:05.772233 20442 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:05.772440 12489 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:05.773206 20527 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:05.751327 12572 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 12:15:05.752235 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3391) took 1631 ms (client timeout 2072 ms). Trace:
W20260709 12:15:05.779902 12570 rpcz_store.cc:269] 0709 12:15:04.121202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.125825 (+  4623us) service_pool.cc:224] Handling call
0709 12:15:05.751982 (+1626157us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:05.752143 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2124135808}
W20260709 12:15:05.753599 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:05.780771 20231 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.743726 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:05.637384 20407 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.840s	user 0.010s	sys 0.008s
W20260709 12:15:05.786101 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 199323us
I20260709 12:15:05.787073 20407 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.648468 20476 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:05.764447 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3408) took 1180 ms. Trace:
I20260709 12:15:05.771487 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:05.774070 20515 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.611s	user 0.006s	sys 0.000s
I20260709 12:15:05.764354 20401 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.750396 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.806124 12565 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:05.785372 20534 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.565s	user 0.000s	sys 0.005s
I20260709 12:15:05.694375 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.676229 20480 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:05.696396 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:05.850140 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:05.850937 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 154658us
I20260709 12:15:05.851379 12489 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 17.
I20260709 12:15:05.696738 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:05.650161 20475 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:05.787585 12536 leader_election.cc:400] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23.
I20260709 12:15:05.852707 12536 leader_election.cc:403] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:15:05.788424 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:05.853533 12564 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:05.789016 20433 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:05.790637 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:05.791668 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:05.864820 12572 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 7. Preceding OpId from leader: term: 23 index: 9. (index mismatch)
I20260709 12:15:05.866021 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:05.866341 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74750.4us
I20260709 12:15:05.748106 12646 rpcz_store.cc:276] 0709 12:15:03.444726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.301937 (+857211us) service_pool.cc:224] Handling call
0709 12:15:04.365749 (+ 63812us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.389364 (+ 23615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.389438 (+    74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.405131 (+ 15693us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.575149 (+170018us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:04.763537 (+188388us) log.cc:844] Serialized 14 byte log entry
0709 12:15:05.001009 (+237472us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:05.747037 (+746028us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b18004afe00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:15:05.747204 (+   167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.747309 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.747357 (+    48us) raft_consensus.cc:1685] finished
0709 12:15:05.747373 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.747549 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40622,"prepare.queue_time_us":212160,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":109391,"raft.run_cpu_time_us":3081,"raft.run_wall_time_us":168631,"spinlock_wait_cycles":14291072,"thread_start_us":260735,"threads_started":3,"wal-append.queue_time_us":483356}
I20260709 12:15:05.867172 12569 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:15:05.867658 12569 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:05.867918 12569 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:15:05.868171 12569 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:05.792537 12571 rpcz_store.cc:276] 0709 12:15:04.584421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.584641 (+   220us) service_pool.cc:224] Handling call
0709 12:15:04.585006 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.603036 (+ 18030us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:05.055568 (+452532us) log.cc:844] Serialized 14 byte log entry
0709 12:15:05.297159 (+241591us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:05.334049 (+ 36890us) log.cc:844] Serialized 26 byte log entry
0709 12:15:05.334259 (+   210us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:05.334344 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.334437 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.764016 (+429579us) raft_consensus.cc:1685] finished
0709 12:15:05.764045 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.764362 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47877,"prepare.queue_time_us":549674,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":168,"raft.queue_time_us":108248,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":31146,"thread_start_us":555585,"threads_started":2,"wal-append.queue_time_us":314588}
I20260709 12:15:05.691973 20057 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:05.875353 20057 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:05.875496 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:05.796526 20020 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:05.877182 20020 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.806116 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:05.878636 12642 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:05.809091 12496 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:05.716779 20359 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:05.879645 20020 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.880574 20442 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.881960 20048 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:05.882689 20048 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.883817 20431 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.860800 20531 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:05.885803 20322 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:15:05.886217 20531 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.886925 20485 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:05.887841 20495 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:15:05.888082 20485 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:05.888356 20495 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:05.888504 20485 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:05.888784 20495 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:15:05.889223 20531 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:05.849329 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155065us
I20260709 12:15:05.853828 12644 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:05.905267 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:05.768509 20437 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:05.754120 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:05.802634 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:05.911636 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:05.912544 12495 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:05.747650 20481 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.872043 20519 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:05.861541 20475 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:05.860685 20356 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:05.919787 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:05.874167 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1029) took 1014 ms. Trace:
I20260709 12:15:05.880678 20359 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.873090 20407 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:05.892849 12489 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.15-24.15]
W20260709 12:15:05.893812 20409 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260709 12:15:05.894260 20507 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.089s
I20260709 12:15:05.894625 20521 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23.
I20260709 12:15:05.897176 20479 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:05.805294 20443 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.898321 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:05.898761 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:05.885427 20048 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:05.900471 20401 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:05.902274 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:05.894377 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106952us
I20260709 12:15:05.843245 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:05.893402 20538 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.906479 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:05.907073 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:05.907569 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:05.920606 12488 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
W20260709 12:15:05.921546 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:05.862469 20476 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:05.923673 12495 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:15:05.923818 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:05.912547 12494 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:05.898761 20548 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:05.920620 12497 rpcz_store.cc:276] 0709 12:15:04.859756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:04.859971 (+   215us) service_pool.cc:224] Handling call
0709 12:15:04.860423 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:04.893292 (+ 32869us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:04.893330 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:04.901899 (+  8569us) log.cc:844] Serialized 26 byte log entry
0709 12:15:04.969739 (+ 67840us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:04.970269 (+   530us) log.cc:844] Serialized 14 byte log entry
0709 12:15:04.970526 (+   257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:04.970644 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.673259 (+702615us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:05.873751 (+200492us) raft_consensus.cc:1685] finished
0709 12:15:05.873895 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:05.874086 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10423,"prepare.queue_time_us":9317,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":64,"raft.queue_time_us":40175,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":20947,"spinlock_wait_cycles":34997632,"thread_start_us":99469,"threads_started":2,"wal-append.queue_time_us":217235}
I20260709 12:15:05.812013 20404 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.108s
I20260709 12:15:05.951341 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:05.940486 20479 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:05.968539 20020 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:05.904285 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:05.959703 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.889463 20485 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:05.964140 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62005us
I20260709 12:15:05.942157 20359 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:05.936252 20112 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:05.980710 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:05.952144 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.866976 20480 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:05.951270 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:05.985534 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:05.988444 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.960253 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:05.929337 20356 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.928925 20211 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:05.993722 12566 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:05.994428 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:06.027676 12489 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:06.028801 12489 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:06.031212 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.988011 20479 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:06.027792 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47261.1us
I20260709 12:15:05.963373 20443 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.033685 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:05.992689 20521 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:06.033234 20519 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:06.040697 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:06.044623 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.045594 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.046726 13262 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:06.048966 13262 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:06.017449 20444 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.029098 19999 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.056245 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:06.056273 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 153392us
I20260709 12:15:05.981678 20548 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.040277 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:06.029973 20431 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.037016 20430 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:06.038622 20476 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:06.038064 20485 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:06.042569 20539 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:06.015657 20376 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:15:06.048738 20446 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:06.064622 20444 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.067552 19999 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.041785 20519 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:06.091293 13260 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:15:06.092731 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:06.099097 20433 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:15:06.101008 20557 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:15:06.032073 20112 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.102324 20455 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:06.103971 20430 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:06.107087 20538 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.109273 20443 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.112046 20238 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:06.113845 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 209641us
I20260709 12:15:06.114195 20502 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:15:06.116144 20550 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 12:15:06.116765 20551 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.146s	user 0.000s	sys 0.002s
I20260709 12:15:06.042328 20541 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.145s
I20260709 12:15:06.120685 12567 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:15:06.122560 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82428.4us
I20260709 12:15:06.165983 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:06.162957 12496 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:06.134218 20433 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:06.126461 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:06.132720 20446 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:06.017941 20475 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.138255 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.176492 13255 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:15:06.142031 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95996.9us
W20260709 12:15:06.148456 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.129419 20441 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:06.150390 20235 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.138433 20390 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 7, Time since last communication: 0.023s
W20260709 12:15:06.125839 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.169852 12641 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:15:06.128389 20553 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.002s	sys 0.000s
I20260709 12:15:06.171128 12489 raft_consensus.cc:606] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:06.137895 20465 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:06.172067 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:06.210614 12496 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:06.165246 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:06.174643 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:06.228127 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63023.5us
I20260709 12:15:06.215003 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:06.229388 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.143203 20401 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:15:06.199150 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:06.170624 20502 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:06.143203 20048 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:15:06.210363 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:06.148078 20409 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:15:06.239895 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:06.239696 12563 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:06.229856 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:06.136919 20456 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:06.231828 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59813us
W20260709 12:15:06.232249 20544 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:15:06.151198 20020 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:15:06.258934 12489 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:15:06.243602 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:06.246461 12567 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 24 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:15:06.246680 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:06.164050 20238 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.210330 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:06.183045 20198 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:06.236123 20528 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:06.265470 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.163061 20495 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.265179 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:06.240471 20199 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:15:06.272012 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61795.7us
I20260709 12:15:06.279326 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.301230 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:15:06.301831 12536 leader_election.cc:403] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:15:06.301672 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:06.292099 20381 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.282974 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.302402 12492 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:06.225386 20476 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:15:06.254047 20441 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:06.239596 20545 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:15:06.306442 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2830) took 1379 ms. Trace:
I20260709 12:15:06.310781 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:06.305274 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:06.214148 20235 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.262756 20475 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:06.323148 12640 rpcz_store.cc:276] 0709 12:15:04.926760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.143902 (+217142us) service_pool.cc:224] Handling call
0709 12:15:05.144294 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:05.144536 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:05.144567 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:05.145081 (+   514us) log.cc:844] Serialized 26 byte log entry
0709 12:15:05.450007 (+304926us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:05.483359 (+ 33352us) log.cc:844] Serialized 14 byte log entry
0709 12:15:05.483775 (+   416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:05.483875 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:05.683611 (+199736us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b180035f440. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf96c758f 00007ffbf2706571 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd
0709 12:15:06.096808 (+413197us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b180035f440. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:06.306209 (+209401us) raft_consensus.cc:1685] finished
0709 12:15:06.306228 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:06.306381 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1783,"prepare.queue_time_us":32910,"prepare.run_cpu_time_us":1475,"prepare.run_wall_time_us":23339,"raft.queue_time_us":2186,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":25514,"spinlock_wait_cycles":47075968,"thread_start_us":262119,"threads_started":1,"wal-append.queue_time_us":377114}
I20260709 12:15:06.277468 20543 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:06.271109 20198 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:06.284801 20528 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:06.261145 20456 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.243039 20519 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.299927 20473 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.082s
W20260709 12:15:06.357797 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.304040 20444 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.280320 20524 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:06.341006 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:06.357501 20561 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:15:06.323426 20539 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:06.335409 20543 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:06.370810 20198 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:06.311642 20431 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21.
I20260709 12:15:06.383870 20409 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:15:06.381916 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:06.405205 12644 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:06.320726 20211 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.364668 20548 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:06.262475 20430 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.400866 12489 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:06.400576 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:06.345551 12492 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:06.351361 20235 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.355844 20495 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:06.373976 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 118065us
I20260709 12:15:06.384541 20528 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:06.338547 20480 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:06.402748 20543 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:06.320535 20381 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.274238 20433 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.436827 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:06.404666 20356 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:06.483491 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46783.9us
I20260709 12:15:06.439020 12640 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:15:06.401772 20057 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.003s
I20260709 12:15:06.439390 12493 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:15:06.485221 20057 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.391705 20524 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:06.444769 12567 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:15:06.455145 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:06.489545 12489 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:06.457500 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:06.426486 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 182961us
I20260709 12:15:06.469691 12571 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:06.396309 20513 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:06.314538 20446 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.432032 20444 raft_consensus.cc:993] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:15:06.490697 12492 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:15:06.442799 19999 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:06.509361 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60305us
W20260709 12:15:06.478323 20561 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:15:06.515997 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.518417 12491 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:06.467371 20401 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.511408 20524 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:06.403126 20502 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.488184 20465 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:06.557077 12492 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:15:06.507866 20112 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.513927 20476 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:06.507249 20431 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.517290 20475 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:06.567070 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 256374us
I20260709 12:15:06.550793 20356 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.517422 20539 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
W20260709 12:15:06.601387 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70887.7us
W20260709 12:15:06.597234 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76931.2us
I20260709 12:15:06.611009 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:06.580145 13261 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:06.611608 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:06.516914 20480 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
W20260709 12:15:06.601526 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:06.616469 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:06.572492 20086 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.833s	user 0.013s	sys 0.001s
I20260709 12:15:06.605084 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:06.562848 20566 raft_consensus.cc:1081] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
W20260709 12:15:06.634354 20564 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:15:06.607517 20431 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.550539 20401 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.625752 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:06.610157 20475 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.611009 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:06.643924 20581 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:06.604566 20409 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:06.752884 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.608258 20476 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:06.577351 20465 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:06.748037 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137161us
W20260709 12:15:06.754266 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128668us
I20260709 12:15:06.628573 20444 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:06.775226 12567 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:15:06.799332 19999 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:06.800689 20401 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:06.801775 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 180457us
I20260709 12:15:06.802802 12536 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 21.
I20260709 12:15:06.619592 20235 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 9, Time since last communication: 0.133s
I20260709 12:15:06.766973 20441 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:06.612716 20356 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:06.823249 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.824689 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:06.778245 20437 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.839s	user 0.007s	sys 0.011s
W20260709 12:15:06.760919 20485 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.656s	user 0.008s	sys 0.010s
I20260709 12:15:06.768404 20381 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:06.763432 20020 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:06.776916 20480 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:15:06.840055 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:06.751550 20581 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:06.631006 20086 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.848708 12570 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:06.818722 20235 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:15:06.847884 20441 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.846084 20057 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:06.843021 20456 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:06.856122 20480 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:06.879671 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55095us
W20260709 12:15:06.868814 20576 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.106s	user 0.002s	sys 0.000s
I20260709 12:15:06.881398 12536 leader_election.cc:400] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:06.856882 20581 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:06.864638 20020 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:06.859711 20582 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:06.890450 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:06.891237 12492 raft_consensus.cc:606] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:06.891641 12492 raft_consensus.cc:614] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:06.879163 20566 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.884268 20480 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:06.916329 20566 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.898989 20475 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:06.940302 20539 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:06.942405 20587 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:06.943657 13260 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:15:06.879694 20476 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.945318 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.945513 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:06.958103 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:06.947261 20566 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:06.945401 20455 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.800s	user 0.010s	sys 0.007s
W20260709 12:15:06.951220 20479 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.903s	user 0.012s	sys 0.005s
I20260709 12:15:06.888926 20409 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.953145 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:06.971206 12488 raft_consensus.cc:606] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:06.971688 12488 raft_consensus.cc:614] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:06.927362 20020 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:06.958585 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69437.4us
I20260709 12:15:06.886056 20567 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.960294 13257 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:06.966887 12644 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:15:06.987682 12644 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 11
I20260709 12:15:06.988081 12644 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:15:06.988540 12644 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:07.000581 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:06.998842 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:15:06.991796 20543 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.500s	user 0.014s	sys 0.004s
I20260709 12:15:07.006335 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:15:06.955103 20528 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:06.999547 20322 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:06.995977 20458 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:06.978212 20479 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.031425 13255 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:15:07.048853 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:07.017446 20521 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.969s	user 0.013s	sys 0.003s
I20260709 12:15:07.007400 20477 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:07.054527 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61634.8us
I20260709 12:15:07.059194 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:07.022114 20565 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.213s	user 0.000s	sys 0.003s
I20260709 12:15:07.010290 20582 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:07.064097 12489 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:15:07.068965 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:07.074553 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:07.068463 20581 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:07.076488 20556 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.007s	user 0.002s	sys 0.005s
I20260709 12:15:07.011994 20543 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.077680 12494 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:15:07.052012 20587 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:07.081128 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1034) took 1393 ms. Trace:
I20260709 12:15:07.082368 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3556) took 1178 ms. Trace:
I20260709 12:15:07.101478 12491 rpcz_store.cc:276] 0709 12:15:05.687339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.687475 (+   136us) service_pool.cc:224] Handling call
0709 12:15:07.080868 (+1393393us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:07.081048 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391983488}
I20260709 12:15:07.088778 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 947) took 1048 ms. Trace:
I20260709 12:15:07.020922 20524 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:06.995625 20567 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:07.053665 20198 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.620s	user 0.011s	sys 0.003s
I20260709 12:15:07.103006 13261 rpcz_store.cc:276] 0709 12:15:06.040253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.040384 (+   131us) service_pool.cc:224] Handling call
0709 12:15:07.088529 (+1048145us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:07.088694 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":882634624}
I20260709 12:15:07.076983 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.076970 20437 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:07.026870 20356 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.097285 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.097285 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.021759 20458 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.101838 12496 rpcz_store.cc:276] 0709 12:15:05.904659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.904908 (+   249us) service_pool.cc:224] Handling call
0709 12:15:07.082161 (+1177253us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:07.082290 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442530560}
I20260709 12:15:07.088394 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.110023 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:07.103590 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.091293 13262 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:15:07.060120 20521 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:07.114066 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.114944 12565 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27.
I20260709 12:15:07.115870 20437 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:07.073335 20578 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:07.117039 20587 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.118126 20578 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:07.113380 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:07.122756 12563 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:07.101711 20475 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:07.134341 12566 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:07.098130 20086 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.072722 20485 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:15:07.117058 20437 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.155835 20479 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.026204 20322 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.119493 20198 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.097910 20477 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.152356 20475 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:07.119968 20582 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 12:15:07.119168 20578 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.173606 20458 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.175438 20198 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.168680 20587 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.178359 20477 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.179811 20528 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:07.181402 20593 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:15:07.183461 20475 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.175441 20524 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:07.166513 20589 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.170s	user 0.000s	sys 0.002s
I20260709 12:15:07.184471 20481 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:07.186976 20567 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.190512 20581 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:07.223239 20573 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.435s	user 0.004s	sys 0.000s
W20260709 12:15:07.265933 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 167493us
I20260709 12:15:07.268095 12492 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:15:07.270650 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.272704 12494 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:07.274683 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.275635 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:07.277027 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.278530 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033354
I20260709 12:15:07.279745 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.264395 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:07.286882 12493 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:15:07.287617 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.289676 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.289856 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.290709 12494 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:07.292315 12497 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:15:07.299539 12497 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:07.302285 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3256) took 1689 ms. Trace:
I20260709 12:15:07.326620 12572 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:15:07.325073 12637 rpcz_store.cc:276] 0709 12:15:05.612186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.629249 (+ 17063us) service_pool.cc:224] Handling call
0709 12:15:05.645157 (+ 15908us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:05.706900 (+ 61743us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:05.706939 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:05.707519 (+   580us) log.cc:844] Serialized 26 byte log entry
0709 12:15:05.916715 (+209196us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:05.939231 (+ 22516us) log.cc:844] Serialized 14 byte log entry
0709 12:15:06.107338 (+168107us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b1800465ee0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:15:06.107613 (+   275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:06.107724 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:07.075754 (+968030us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:07.301279 (+225525us) raft_consensus.cc:1685] finished
0709 12:15:07.301334 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:07.301826 (+   492us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79141,"prepare.queue_time_us":26859,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":2485,"raft.queue_time_us":235287,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":249,"spinlock_wait_cycles":12600320,"thread_start_us":208669,"threads_started":1,"wal-append.queue_time_us":296567}
I20260709 12:15:07.305364 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3261) took 1065 ms. Trace:
I20260709 12:15:07.292672 12494 raft_consensus.cc:1217] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Deduplicated request from leader. Original: 17.10->[19.11-22.12]   Dedup: 19.11->[22.12-22.12]
I20260709 12:15:07.344419 20528 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.345168 20513 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.346031 20513 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.330031 12643 rpcz_store.cc:276] 0709 12:15:06.239442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.239595 (+   153us) service_pool.cc:224] Handling call
0709 12:15:06.264530 (+ 24935us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:06.292727 (+ 28197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:06.292763 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:06.293445 (+   682us) log.cc:844] Serialized 52 byte log entry
0709 12:15:06.431395 (+137950us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:06.487305 (+ 55910us) log.cc:844] Serialized 14 byte log entry
0709 12:15:06.487756 (+   451us) log.cc:844] Serialized 14 byte log entry
0709 12:15:06.487973 (+   217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:06.488079 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:07.222503 (+734424us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:07.304271 (+ 81768us) raft_consensus.cc:1685] finished
0709 12:15:07.304323 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:07.305094 (+   771us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50538,"prepare.queue_time_us":238745,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":92,"raft.queue_time_us":145178,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":19392,"spinlock_wait_cycles":81201536,"thread_start_us":137053,"threads_started":1,"wal-append.queue_time_us":315559}
I20260709 12:15:07.350915 20543 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.356794 12497 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:07.357805 12494 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: [19.11-22.12]
I20260709 12:15:07.365059 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.366878 20446 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:07.312037 20360 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.306378 20430 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 12:15:07.320511 20322 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.311631 20343 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.377367 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.319654 20437 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:07.386606 20343 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.381238 20528 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.379423 20446 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:07.392104 20446 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.398159 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.399312 12497 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:07.403424 12494 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:07.403677 20528 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.407797 12460 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27.
I20260709 12:15:07.408283 12460 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:07.379330 20513 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.393090 20360 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.394081 12492 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:07.394958 12572 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:15:07.395495 20480 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.396345 20543 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:07.397841 20409 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.415464 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3418) took 1715 ms. Trace:
I20260709 12:15:07.411865 12572 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 12
I20260709 12:15:07.418689 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.420753 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1340) took 1471 ms. Trace:
I20260709 12:15:07.435218 20459 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.435915 20437 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:07.443617 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:07.446830 20430 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.448205 20600 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:15:07.449796 20480 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.451443 20360 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.395565 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:07.469515 20459 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:07.471925 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76770.2us
I20260709 12:15:07.454010 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.455185 20479 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:07.455835 12497 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:15:07.466903 12493 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:07.474985 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:07.462910 12572 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260709 12:15:07.477562 12572 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:07.462466 12568 rpcz_store.cc:276] 0709 12:15:05.700168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.744674 (+ 44506us) service_pool.cc:224] Handling call
0709 12:15:05.775513 (+ 30839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:05.971691 (+196178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:07.414778 (+1443087us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:07.415234 (+   456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293942016}
I20260709 12:15:07.463898 12570 rpcz_store.cc:276] 0709 12:15:05.949984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.951793 (+  1809us) service_pool.cc:224] Handling call
0709 12:15:07.420151 (+1468358us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:07.420486 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884814080}
I20260709 12:15:07.467983 20600 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:07.469774 20578 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.476568 20477 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:07.499143 20437 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.479175 20567 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27.
I20260709 12:15:07.501932 20430 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.502871 20479 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.503223 20485 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:07.506306 20485 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.508347 13253 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:15:07.509549 20479 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.510313 20485 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.510389 12496 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:07.526476 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.529528 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.534355 20409 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.538221 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:07.539206 13253 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:07.540750 20381 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.541792 20381 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.542553 20437 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.543988 20431 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.538494 20600 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:07.546737 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 941577us
W20260709 12:15:07.546137 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 738904us
I20260709 12:15:07.532560 20459 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:07.547271 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.02145e+06us
I20260709 12:15:07.537513 20409 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.576789 20238 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.545603 20381 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.558657 20431 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.547029 20600 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:15:07.549916 20430 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.558838 20480 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:07.580310 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48642.7us
I20260709 12:15:07.547008 20343 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.549983 20048 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:15:07.547701 20198 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.548516 20401 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.598428 20401 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.549389 20456 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:15:07.550526 20086 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:15:07.594826 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:07.594959 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.563705 20437 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.598176 20198 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.603796 20520 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:15:07.583128 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.605999 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.606503 12497 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:07.609735 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.610816 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:07.609911 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:07.611495 20431 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.613940 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.614957 20431 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.602730 12495 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:07.616673 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.617700 12492 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
W20260709 12:15:07.606146 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:07.626036 20481 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:07.627846 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.629988 20198 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.630142 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:07.533854 20578 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.629467 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.644516 20475 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:07.645447 20475 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.646270 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.645617 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.651494 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.652218 20475 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.652890 12497 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:07.655567 20502 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:07.656620 20502 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.657101 12639 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:15:07.658207 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:07.660655 20198 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.661360 20198 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.662477 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:07.663570 12640 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:07.663785 12645 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:07.664160 20198 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.666424 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:07.639222 20543 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:15:07.655488 20381 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:07.668224 20543 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:07.668035 20381 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.668665 20543 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:07.602700 20401 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.653519 12495 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:15:07.635254 20048 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:07.675909 12565 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 12:15:07.676544 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1121) took 1425 ms. Trace:
I20260709 12:15:07.669855 12643 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:07.677373 20048 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.679164 20381 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.677189 13260 rpcz_store.cc:276] 0709 12:15:06.250827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.251027 (+   200us) service_pool.cc:224] Handling call
0709 12:15:06.251443 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:06.922322 (+670879us) spinlock_profiling.cc:243] Waited 86.1 ms on lock 0x7b54004b7ba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:15:06.958606 (+ 36284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:06.963295 (+  4689us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b54004b7ba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6a02 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:15:07.675849 (+712554us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b54004b7ba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577041d 00007ffbf57f6a57 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:15:07.676319 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207013888}
I20260709 12:15:07.682340 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.647380 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:07.663434 20404 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.593735 20480 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.669433 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:07.676486 12565 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 13
I20260709 12:15:07.688315 12565 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:15:07.688735 12565 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:07.683259 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:07.691558 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.670969 20502 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.706058 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.712128 12571 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:07.707480 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.710208 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.716377 13253 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:07.720469 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:07.724871 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.731539 20442 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:07.734001 20613 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:07.727169 20404 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.734992 20613 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:07.736490 20613 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.734094 20366 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:07.738289 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92897.9us
I20260709 12:15:07.740156 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.741097 12497 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:07.742342 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.744941 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.707016 20437 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.747431 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.748260 12496 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:15:07.750540 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81277.9us
W20260709 12:15:07.752272 20611 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:15:07.728996 20543 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.772984 20480 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.752703 12493 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:07.778229 20112 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:07.779422 20527 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:07.780392 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3260) took 1543 ms. Trace:
I20260709 12:15:07.781571 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.785667 20404 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.820844 12565 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 24 index: 11. (index mismatch)
I20260709 12:15:07.786917 12488 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 10. Preceding OpId from leader: term: 24 index: 11. (index mismatch)
I20260709 12:15:07.786963 13262 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 12:15:07.822558 13262 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 13
I20260709 12:15:07.822968 13262 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:15:07.822849 12493 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:07.823339 13262 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:07.794509 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.800125 20112 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:07.810117 20527 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.810447 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.517s	user 0.029s	sys 0.013s Metrics: {"bytes_written":4161,"delete_count":0,"lbm_write_time_us":1131,"lbm_writes_lt_1ms":5,"reinsert_count":0,"update_count":154}
I20260709 12:15:07.812860 13261 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:15:07.816344 20437 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:07.817476 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 174850us
I20260709 12:15:07.809423 20465 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.838146 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.838514 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.839493 20465 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.839808 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.840102 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:07.840708 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:07.805450 12638 rpcz_store.cc:276] 0709 12:15:06.236660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.238379 (+  1719us) service_pool.cc:224] Handling call
0709 12:15:06.238775 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:06.445516 (+206741us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:06.626392 (+180876us) log.cc:844] Serialized 14 byte log entry
0709 12:15:07.688209 (+1061817us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:07.756211 (+ 68002us) log.cc:844] Serialized 26 byte log entry
0709 12:15:07.779055 (+ 22844us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:07.779094 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:07.779900 (+   806us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:07.779950 (+    50us) raft_consensus.cc:1685] finished
0709 12:15:07.779961 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:07.780167 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":768165,"prepare.queue_time_us":167196,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":758336,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":34279,"thread_start_us":432567,"threads_started":4,"wal-append.queue_time_us":280524}
I20260709 12:15:07.841818 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.841312 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:07.842399 20404 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:07.843304 20404 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:07.843999 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:07.844923 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:07.841876 20198 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.846074 20086 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:07.846664 20198 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.846966 20086 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.818735 20578 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.835836 20112 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.840112 20465 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.858238 20390 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:07.859933 12489 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:07.860312 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:07.860594 20465 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:15:07.861304 20545 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.862003 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:07.862334 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.863339 12497 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:15:07.863576 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:07.864287 20235 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.853466 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:07.846990 20476 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:15:07.841312 13253 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:07.857677 20199 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:07.857900 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.179400
I20260709 12:15:07.807426 20477 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:07.873879 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:07.875705 12492 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:15:07.857244 20086 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:07.878450 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.863473 20472 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.013s
I20260709 12:15:07.885653 12493 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:15:07.891841 20381 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:07.892446 20381 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:07.892807 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.865101 20112 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:07.893046 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1070) took 2548 ms (client timeout 1735 ms). Trace:
I20260709 12:15:07.888616 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:07.896947 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.896118 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:07.912256 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:07.895359 13254 rpcz_store.cc:269] 0709 12:15:05.317198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:05.317383 (+   185us) service_pool.cc:224] Handling call
0709 12:15:07.797452 (+2480069us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:07.865272 (+ 67820us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3574093056}
I20260709 12:15:07.915449 12640 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:15:07.892855 20381 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:07.916374 12496 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:07.920622 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.862005 20545 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.898416 20048 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:07.924744 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:07.931452 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:07.898258 20528 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:15:07.938336 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.942845 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:07.924846 20437 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:07.946409 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.955730 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:07.908147 20235 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:07.934748 20475 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:07.944453 20198 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:07.958662 12565 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:07.960623 12495 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:15:07.971757 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:07.960009 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3262) took 1632 ms. Trace:
I20260709 12:15:07.968382 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3257) took 1732 ms. Trace:
I20260709 12:15:07.991259 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:07.996260 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:07.999960 12568 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:15:08.002511 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54208.8us
W20260709 12:15:07.962733 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77311us
I20260709 12:15:07.983881 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1018) took 1549 ms. Trace:
I20260709 12:15:08.022303 12640 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:15:08.009912 12641 rpcz_store.cc:276] 0709 12:15:06.235587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.235807 (+   220us) service_pool.cc:224] Handling call
0709 12:15:06.236170 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:07.780630 (+1544460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:07.780686 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:07.780734 (+    48us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:07.780790 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:07.967995 (+187205us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:07.968036 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:07.968210 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3550713728}
I20260709 12:15:08.016515 12644 rpcz_store.cc:276] 0709 12:15:06.434759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.434915 (+   156us) service_pool.cc:224] Handling call
0709 12:15:06.480277 (+ 45362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:06.989649 (+509372us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:07.983359 (+993710us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:07.983677 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1117762816}
I20260709 12:15:07.950688 20404 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:08.030309 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:07.992409 20456 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:07.991751 20475 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:07.987355 20443 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:08.009557 12642 rpcz_store.cc:276] 0709 12:15:06.239921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:06.240052 (+   131us) service_pool.cc:224] Handling call
0709 12:15:06.240422 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:06.482175 (+241753us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:06.484355 (+  2180us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:07.753048 (+1268693us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:15:07.774159 (+ 21111us) log.cc:844] Serialized 26 byte log entry
0709 12:15:07.806601 (+ 32442us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:07.806644 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:07.850799 (+ 44155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:07.850852 (+    53us) raft_consensus.cc:1685] finished
0709 12:15:07.850872 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:07.872217 (+ 21345us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17880,"raft.queue_time_us":50141,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"spinlock_wait_cycles":21646720,"thread_start_us":242113,"threads_started":2,"wal-append.queue_time_us":58538}
I20260709 12:15:08.047017 20523 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:07.986989 20527 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:08.043583 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:07.980254 20235 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:08.002116 20545 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:07.994755 20444 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:15:07.961380 12569 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:15:07.962229 20048 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.022930 20437 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.051767 12638 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:08.054100 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.054543 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:08.055258 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.069389 12564 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:07.993872 20480 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.062s
I20260709 12:15:08.039188 20456 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:07.989558 20567 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 12:15:08.058089 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:08.058727 20616 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.002s
I20260709 12:15:08.047947 20523 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:08.059558 20620 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:15:08.049100 20443 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:08.060212 20409 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:15:08.060968 20623 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:15:08.042917 20619 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:15:08.103703 20048 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:08.070837 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.083454 20456 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.106804 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.107280 20456 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.107019 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.108141 12572 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:08.108222 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.109084 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:08.109181 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.075667 12563 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:08.083837 12569 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:15:08.178357 12569 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.054307 20475 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:08.095429 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66383.2us
I20260709 12:15:08.098127 12565 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:08.099037 20567 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:08.058082 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:08.149695 20543 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:08.136869 12568 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:08.153203 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:08.105454 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.138577 19999 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:08.128605 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.110498 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:08.145817 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 258193us
I20260709 12:15:08.146275 20502 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:08.131871 20433 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.151367 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:08.193847 12641 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.097267 20437 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.153366 12637 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
W20260709 12:15:08.107177 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.127313 20446 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:08.114121 20523 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:08.146010 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 239349us
I20260709 12:15:08.118104 20409 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:08.147053 20401 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:08.145324 20199 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:08.155004 12644 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:15:08.152076 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.162595 20443 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.141209 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.109948 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.110436 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.177965 12572 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:08.183743 12638 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:08.188755 12640 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.190142 20469 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:08.185374 20567 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:08.203336 20048 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:08.243068 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:08.136998 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.193521 20356 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:08.237448 20442 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.505s	user 0.014s	sys 0.000s
I20260709 12:15:08.225745 20401 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.239533 12489 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:08.191154 19999 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:08.231194 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 173210us
W20260709 12:15:08.243702 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102976us
I20260709 12:15:08.244310 12642 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:08.243767 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:08.204322 20600 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:08.243086 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.247643 12563 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:15:08.247809 12639 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:08.221691 20409 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:08.259322 12567 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
W20260709 12:15:08.260990 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 165929us
W20260709 12:15:08.262115 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.262642 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:08.270702 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:08.271657 12494 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:15:08.272143 12494 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 10
I20260709 12:15:08.273330 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:08.275285 20356 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.275413 20409 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.277878 20020 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:08.278486 20020 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:08.278937 20020 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.249192 20401 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.277196 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:08.250124 20475 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:08.286578 12494 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:15:08.277347 20409 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:08.288403 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:08.290408 12494 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:08.270555 20568 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:08.281903 20443 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:08.305843 12646 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:15:08.302039 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.272962 20437 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:08.282444 20477 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:08.289036 20545 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:08.289942 20475 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.319023 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:08.322515 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.251995 20048 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.249112 20469 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.325270 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.273746 20600 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.339948 20568 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.340864 13260 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:15:08.341391 20376 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:08.355240 20376 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.342550 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.355744 20376 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.356027 12497 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:08.357714 20600 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.350098 19999 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.360309 20485 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:08.361034 19999 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.361771 20485 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:15:08.362593 20485 consensus_queue.cc:1243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 20.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:15:08.363131 20485 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:08.363436 20485 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:08.364403 20485 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:08.351197 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74282.7us
I20260709 12:15:08.351853 12493 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:08.344183 12568 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:08.356236 20494 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:08.358920 20356 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:08.345927 20477 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.369364 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:08.357658 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:08.368302 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:08.374017 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103437us
I20260709 12:15:08.367481 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:08.366052 12637 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:15:08.367832 12568 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:08.382341 13255 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:08.355551 20048 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:08.365388 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:08.386200 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61034.9us
I20260709 12:15:08.381193 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:08.330330 20437 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:08.369045 20494 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:08.331277 20475 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.325111 20545 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.389142 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:08.387871 13262 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:15:08.391162 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:08.406104 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:08.387786 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.544s (SENT) [suppressed 10 similar messages]
I20260709 12:15:08.397568 12568 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.407332 13261 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:08.421142 12565 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:08.382867 20390 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:08.370656 20359 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:08.431357 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.355561 20568 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.392634 20477 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:08.440502 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49404.7us
I20260709 12:15:08.391397 20231 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:08.423460 12569 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.441498 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:08.448395 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:08.449568 20502 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:08.451359 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.452304 20048 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:08.452751 20048 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.453073 20048 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:08.453888 20613 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:08.455171 20469 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.407356 20545 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.396967 20494 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.459408 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:08.482127 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83427.8us
I20260709 12:15:08.457768 20612 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:08.460654 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.461517 20477 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:08.443554 12637 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:08.499069 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.463166 20567 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 25.9, Last appended by leader: 9, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.469401 12570 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:08.470264 12569 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.471949 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.472832 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.473575 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.411342 20472 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:08.484433 12640 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:08.410557 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.409822 20485 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:15:08.505513 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44969.7us
I20260709 12:15:08.438830 20356 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:08.512756 20502 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:08.516142 20356 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:08.516860 20502 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:08.516994 20356 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:08.517146 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.518280 12569 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:15:08.519416 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.522729 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.523392 20199 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:08.525177 20520 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:08.510010 12646 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:08.508854 12642 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:15:08.529985 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105750us
I20260709 12:15:08.447043 20231 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.524502 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:08.532161 12644 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:15:08.499922 20235 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:08.392428 20437 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.564692 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.499491 20612 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.538852 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.544631 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:08.545696 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:08.547674 20359 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:08.537503 12568 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:08.410086 20600 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.107s
W20260709 12:15:08.549016 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 158963us
I20260709 12:15:08.573652 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.571342 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:08.537936 20231 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:08.577814 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:08.584244 12642 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:15:08.588665 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:08.530067 20199 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:08.570819 20443 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:08.574884 20612 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:08.549903 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:08.564472 20235 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.602883 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:08.602113 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.604117 12646 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:08.485812 20390 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.444634 20578 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.138s
I20260709 12:15:08.511153 20626 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.094s
I20260709 12:15:08.564253 20477 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.609764 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:08.616395 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107383us
I20260709 12:15:08.620383 20502 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:08.624442 20629 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:15:08.626080 20566 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:08.626773 20566 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.617717 20235 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:08.628142 20524 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:08.629601 20578 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:08.630162 20578 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:08.631309 20578 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.632875 20578 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.634467 20473 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:08.634963 20473 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.635324 20473 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:08.638530 12496 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:08.639288 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.626106 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.640396 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:08.641755 20582 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:15:08.643702 20390 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.617677 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:08.650400 20469 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:08.650900 20469 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:08.593283 20198 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:15:08.652314 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.652030 20469 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.653211 20469 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.654794 20048 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.654065 20625 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:08.657061 20625 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:08.657996 12638 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:08.658216 20625 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.612989 20431 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:15:08.624596 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.596207 20199 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:08.618808 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.655911 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:08.608520 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58745.2us
I20260709 12:15:08.672683 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:08.606231 20567 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:08.677341 20472 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.677881 20472 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:08.613901 20086 raft_consensus.cc:2764] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:15:08.681706 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.670704 20199 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.683120 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:08.683262 20473 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.673661 20442 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:08.662750 12644 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:15:08.676328 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.674124 20431 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:15:08.686790 20566 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:08.687796 20524 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:08.689015 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:08.737056 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.690356 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:08.666144 20381 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.729s	user 0.013s	sys 0.001s
I20260709 12:15:08.700404 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.755169 12638 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:15:08.582118 20376 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.605861 20443 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:08.715662 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.762115 12489 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:08.716607 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.674356 12567 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
W20260709 12:15:08.763809 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73552us
I20260709 12:15:08.725174 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:08.693197 19999 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:08.767830 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:08.672113 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:08.740126 20446 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.519s	user 0.004s	sys 0.008s
I20260709 12:15:08.711021 20626 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:08.745482 20473 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:08.769946 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97371us
I20260709 12:15:08.700793 20523 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:15:08.775328 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:08.780426 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108437us
I20260709 12:15:08.729390 20442 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:08.788295 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 400503us
I20260709 12:15:08.790266 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:08.790854 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:08.794520 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.755569 20524 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:08.801959 20442 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.795090 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:08.803284 12496 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:15:08.805039 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:08.805397 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:08.806039 12638 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:15:08.779623 20456 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.003s
I20260709 12:15:08.768098 19999 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:08.767774 20443 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.784081 20626 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:08.776257 20390 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:15:08.827064 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:08.831269 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.882912 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:08.890352 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80782.7us
I20260709 12:15:08.787585 20409 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:08.927479 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:08.768993 20381 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:08.808248 20600 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:15:08.942021 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48424.8us
I20260709 12:15:08.813777 20446 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:08.883474 19999 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:08.893724 20626 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:08.882054 20048 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:08.883924 20198 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:08.813261 20444 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.172s
I20260709 12:15:08.808508 20469 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.174s
I20260709 12:15:08.809747 20235 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.049027 12644 raft_consensus.cc:1217] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-25.14]   Dedup: 21.12->[23.13-25.14]
I20260709 12:15:08.982537 20600 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:09.043844 12639 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:09.068574 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.059100 12644 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
I20260709 12:15:09.073256 12644 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:15:09.073642 12644 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:09.026911 20566 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:09.085762 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:09.085088 20548 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.026s	sys 0.079s
I20260709 12:15:09.062451 20433 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.070533 20600 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:09.037242 20409 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
W20260709 12:15:09.086493 20545 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.630s	user 0.013s	sys 0.000s
W20260709 12:15:09.093866 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.028s	sys 0.039s
I20260709 12:15:09.104447 12642 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
W20260709 12:15:09.107194 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.029s	sys 0.039s
W20260709 12:15:09.110749 20635 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:15:09.052479 20198 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:08.983361 20390 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:08.990393 20112 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:09.132943 20528 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.007s	sys 0.030s
W20260709 12:15:09.172021 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.025s	sys 0.019s
W20260709 12:15:09.136358 20502 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.016s	sys 0.021s
W20260709 12:15:09.136086 20356 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.618s	user 0.004s	sys 0.008s
W20260709 12:15:09.176723 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.025s	sys 0.019s
I20260709 12:15:09.095808 20446 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:09.141573 20401 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.824s	user 0.006s	sys 0.006s
W20260709 12:15:09.192209 12493 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.015s	sys 0.034s
W20260709 12:15:09.213390 12493 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.016s	sys 0.034s
I20260709 12:15:09.133090 20477 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:09.174695 20020 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.888s	user 0.012s	sys 0.004s
I20260709 12:15:09.192667 12642 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
W20260709 12:15:09.182957 20528 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.007s	sys 0.030s
I20260709 12:15:09.172006 20581 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:15:09.183010 20502 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.016s	sys 0.021s
I20260709 12:15:09.131021 20545 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.216496 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3455) took 1250 ms. Trace:
I20260709 12:15:09.108402 20231 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.227515 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.242272 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:09.227888 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:09.094915 20566 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.102788 20433 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.213014 20446 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:09.253371 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:09.257159 12566 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.006s	sys 0.024s
W20260709 12:15:09.257867 12566 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.006s	sys 0.024s
I20260709 12:15:09.237198 12564 rpcz_store.cc:276] 0709 12:15:07.966095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.252772 (+286677us) service_pool.cc:224] Handling call
0709 12:15:08.276797 (+ 24025us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:08.277485 (+   688us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:08.277516 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:08.278169 (+   653us) log.cc:844] Serialized 26 byte log entry
0709 12:15:08.566890 (+288721us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:08.567033 (+   143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:08.567297 (+   264us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.216259 (+648962us) raft_consensus.cc:1685] finished
0709 12:15:09.216280 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.216427 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7579,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":12960,"thread_start_us":287888,"threads_started":1,"wal-append.queue_time_us":776907}
W20260709 12:15:09.241286 20612 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.607s	user 0.012s	sys 0.001s
W20260709 12:15:09.255865 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.014s	sys 0.015s
W20260709 12:15:09.264155 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.015s	sys 0.015s
I20260709 12:15:09.235926 20581 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:09.267354 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.004s	sys 0.028s
I20260709 12:15:09.214658 20356 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:09.284044 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.004s	sys 0.028s
I20260709 12:15:09.218166 20112 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:09.260896 20638 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.197s	user 0.001s	sys 0.001s
I20260709 12:15:09.219300 20401 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:09.259946 20437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.030s	sys 0.053s
I20260709 12:15:09.348400 12637 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:09.325214 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:09.256953 20376 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:09.330964 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 446512us
I20260709 12:15:09.230027 20020 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.348253 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:09.309893 20567 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.626s	user 0.013s	sys 0.001s
W20260709 12:15:09.350322 13260 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.011s	sys 0.019s
W20260709 12:15:09.358484 13260 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.011s	sys 0.019s
W20260709 12:15:09.329533 20359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.003s	sys 0.011s
I20260709 12:15:09.299386 20581 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:09.327076 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.365703 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.300781 20381 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:09.299856 20502 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:09.347918 20437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.030s	sys 0.053s
W20260709 12:15:09.392395 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:09.392859 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.282128 20612 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.397094 12492 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:09.402729 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:09.411230 12568 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.838s	user 0.013s	sys 0.002s
I20260709 12:15:09.423396 12640 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:09.426574 12568 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:15:09.368266 20456 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.367004 20235 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:09.443948 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.451085 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:09.458829 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:09.460145 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:09.443429 20475 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.011s	user 0.014s	sys 0.001s
I20260709 12:15:09.469743 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.473577 12568 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
W20260709 12:15:09.458477 20086 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.009s	sys 0.010s
I20260709 12:15:09.488446 12568 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:15:09.490507 12568 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:09.504724 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3458) took 1537 ms. Trace:
I20260709 12:15:09.505419 12568 rpcz_store.cc:276] 0709 12:15:07.966932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.304511 (+337579us) service_pool.cc:224] Handling call
0709 12:15:08.325600 (+ 21089us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:09.504285 (+1178685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.504646 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:09.475847 20472 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.797s	user 0.000s	sys 0.014s
I20260709 12:15:09.507581 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:09.364210 20390 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.398208 20502 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:09.517194 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66233us
I20260709 12:15:09.518465 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:09.512121 20376 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.519809 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.520093 20376 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.520736 12643 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:09.522614 20376 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:09.523308 20538 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
W20260709 12:15:09.490219 20086 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.010s	sys 0.010s
I20260709 12:15:09.525424 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:09.526619 12568 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:09.538035 12571 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.743s	user 0.010s	sys 0.005s
I20260709 12:15:09.543911 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:09.545059 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:09.546156 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:09.511520 20199 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.829s	user 0.014s	sys 0.000s
I20260709 12:15:09.549755 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:09.557991 13257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.008s	sys 0.021s
W20260709 12:15:09.558645 13257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.008s	sys 0.021s
I20260709 12:15:09.572541 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.565367 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:09.490617 20456 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.500263 20475 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.556291 20086 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:09.587319 12643 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:09.549281 20479 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.548250 20381 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:09.580040 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1053) took 1072 ms. Trace:
I20260709 12:15:09.563625 20476 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:09.524317 20472 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.560294 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:09.595440 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 244552us
I20260709 12:15:09.584765 12640 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 12:15:09.597239 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s	user 0.000s	sys 0.023s
W20260709 12:15:09.612360 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.000s	sys 0.024s
I20260709 12:15:09.594682 12493 rpcz_store.cc:276] 0709 12:15:08.507645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.508582 (+   937us) service_pool.cc:224] Handling call
0709 12:15:08.539761 (+ 31179us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:08.590315 (+ 50554us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:09.213877 (+623562us) log.cc:844] Serialized 14 byte log entry
0709 12:15:09.579444 (+365567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:09.579507 (+    63us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:09.579559 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.579850 (+   291us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.579980 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32472,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":110,"thread_start_us":930580,"threads_started":2}
W20260709 12:15:09.585112 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.010s	sys 0.021s
I20260709 12:15:09.607936 12572 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:09.613559 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.010s	sys 0.021s
W20260709 12:15:09.614903 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.548388 20656 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 12:15:09.579682 20568 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:09.610323 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:09.586913 20086 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:09.638440 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78237.1us
I20260709 12:15:09.560735 20657 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:15:09.596508 20433 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:09.655645 13261 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:15:09.646703 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:09.562448 20199 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.573014 20437 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:15:09.609586 20231 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:09.680441 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:09.616629 20356 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:09.697216 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.651103 20086 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:09.700323 12568 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:09.704543 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 958) took 1416 ms. Trace:
I20260709 12:15:09.605962 20658 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:15:09.650519 20640 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:15:09.655930 20636 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.363s	user 0.000s	sys 0.002s
I20260709 12:15:09.698493 12566 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:15:09.703789 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1050) took 1281 ms. Trace:
I20260709 12:15:09.743322 20653 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.653071 20566 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:09.671554 20381 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:09.596329 20479 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:09.756958 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.012s	sys 0.016s
I20260709 12:15:09.760661 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3585) took 1552 ms. Trace:
I20260709 12:15:09.670974 20612 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:09.672816 20112 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:09.748073 13257 rpcz_store.cc:276] 0709 12:15:08.288116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.298604 (+ 10488us) service_pool.cc:224] Handling call
0709 12:15:08.333239 (+ 34635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:08.405040 (+ 71801us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:15:08.946279 (+541239us) log.cc:844] Serialized 14 byte log entry
0709 12:15:09.559138 (+612859us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:09.559212 (+    74us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:09.559281 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.704205 (+144924us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b54005e1354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:09.704278 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.704467 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5558,"prepare.queue_time_us":720315,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":2740,"raft.queue_time_us":18681,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":12975,"spinlock_wait_cycles":148628224,"thread_start_us":947343,"threads_started":2}
I20260709 12:15:09.673491 20401 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:09.776854 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.013s	sys 0.016s
I20260709 12:15:09.744189 20472 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:09.698050 20433 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:15:09.672246 20568 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:09.817348 20652 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 12:15:09.775156 12495 rpcz_store.cc:276] 0709 12:15:08.228963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.261320 (+ 32357us) service_pool.cc:224] Handling call
0709 12:15:08.292852 (+ 31532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:08.358011 (+ 65159us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:15:08.615263 (+257252us) log.cc:844] Serialized 14 byte log entry
0709 12:15:09.656157 (+1040894us) log.cc:844] Serialized 14 byte log entry
0709 12:15:09.705821 (+ 49664us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:09.705905 (+    84us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:15:09.760262 (+ 54357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.760393 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.760576 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52517,"prepare.queue_time_us":271019,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":10018,"raft.queue_time_us":91587,"raft.run_wall_time_us":95563,"spinlock_wait_cycles":42491008,"thread_start_us":1177639,"threads_started":2,"wal-append.queue_time_us":1139759}
I20260709 12:15:09.780891 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:09.785615 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101406us
I20260709 12:15:09.823993 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3457) took 1857 ms. Trace:
I20260709 12:15:09.760612 12490 rpcz_store.cc:276] 0709 12:15:08.196377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.196526 (+   149us) service_pool.cc:224] Handling call
0709 12:15:08.239243 (+ 42717us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:08.252306 (+ 13063us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:08.252347 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:08.310143 (+ 57796us) log.cc:844] Serialized 26 byte log entry
0709 12:15:08.610711 (+300568us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:08.610764 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:08.610872 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.477041 (+866169us) raft_consensus.cc:1685] finished
0709 12:15:09.477072 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.477272 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33107,"raft.queue_time_us":18403,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":18085,"thread_start_us":270799,"threads_started":1,"wal-append.queue_time_us":473454}
I20260709 12:15:09.769559 20198 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:09.770000 20673 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:09.879372 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.880208 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:09.817305 20647 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
W20260709 12:15:09.768029 20649 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.315s	user 0.002s	sys 0.000s
W20260709 12:15:09.810232 20644 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.351s	user 0.001s	sys 0.001s
W20260709 12:15:09.913355 20679 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:15:09.780287 20356 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.134s
I20260709 12:15:09.914255 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3461) took 1790 ms. Trace:
I20260709 12:15:09.913414 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.916258 12643 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:09.917688 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.918566 12643 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
W20260709 12:15:09.918713 20626 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.868s	user 0.010s	sys 0.010s
I20260709 12:15:09.919696 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.912339 20112 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.920496 20112 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.922657 20666 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:09.923162 20112 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:09.919219 20626 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.893414 20502 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:09.925732 20626 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:09.923357 20666 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:09.906997 20690 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:09.930490 20690 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:09.931993 20443 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.030s	user 0.015s	sys 0.000s
I20260709 12:15:09.932511 20443 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.933708 12494 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.12-24.12]
I20260709 12:15:09.935369 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3592) took 1226 ms. Trace:
I20260709 12:15:09.898806 20673 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.935742 12494 rpcz_store.cc:276] 0709 12:15:08.709409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.709662 (+   253us) service_pool.cc:224] Handling call
0709 12:15:08.730130 (+ 20468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:09.934135 (+1204005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.935063 (+   928us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:09.935292 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2767239552}
I20260709 12:15:09.891980 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:09.777434 20669 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:09.914688 12569 rpcz_store.cc:276] 0709 12:15:08.124230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.520310 (+396080us) service_pool.cc:224] Handling call
0709 12:15:08.520668 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:08.528799 (+  8131us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:08.528833 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:08.529446 (+   613us) log.cc:844] Serialized 26 byte log entry
0709 12:15:09.107811 (+578365us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:09.107865 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.107958 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.913963 (+806005us) raft_consensus.cc:1685] finished
0709 12:15:09.913986 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.914179 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7477,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":577840,"threads_started":1,"wal-append.queue_time_us":829468}
W20260709 12:15:09.819655 20659 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:15:09.898801 20541 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:09.901775 12637 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:09.902849 20688 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:09.903882 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:09.774559 20653 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:09.906040 20650 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:15:09.906198 20663 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:09.906646 20648 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.318s	user 0.000s	sys 0.002s
I20260709 12:15:09.907040 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:09.910261 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1091) took 1244 ms. Trace:
W20260709 12:15:09.910539 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.911484 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3595) took 1177 ms. Trace:
W20260709 12:15:09.912264 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.889158 12563 rpcz_store.cc:276] 0709 12:15:07.966661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.292295 (+325634us) service_pool.cc:224] Handling call
0709 12:15:08.305380 (+ 13085us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:08.357093 (+ 51713us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:15:08.357125 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:08.357848 (+   723us) log.cc:844] Serialized 52 byte log entry
0709 12:15:08.710027 (+352179us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:15:08.710090 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:08.715223 (+  5133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.823693 (+1108470us) raft_consensus.cc:1685] finished
0709 12:15:09.823717 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.823916 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24436,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":107,"thread_start_us":351633,"threads_started":1,"wal-append.queue_time_us":664618}
I20260709 12:15:09.882179 20548 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:09.952597 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60780.2us
I20260709 12:15:09.960170 12645 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:09.938553 20669 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:09.977989 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1358) took 1772 ms. Trace:
I20260709 12:15:09.939670 20475 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:09.981793 12565 rpcz_store.cc:276] 0709 12:15:08.205216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.530317 (+325101us) service_pool.cc:224] Handling call
0709 12:15:08.530734 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:08.565667 (+ 34933us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:08.565709 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:08.566336 (+   627us) log.cc:844] Serialized 26 byte log entry
0709 12:15:09.212749 (+646413us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:09.212816 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.212920 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.977771 (+764851us) raft_consensus.cc:1685] finished
0709 12:15:09.977793 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.977929 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1124,"raft.queue_time_us":24678,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":2738,"thread_start_us":571063,"threads_started":1,"wal-append.queue_time_us":945144}
I20260709 12:15:09.951005 20657 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:09.988634 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3465) took 1099 ms. Trace:
I20260709 12:15:09.989269 12565 rpcz_store.cc:276] 0709 12:15:08.889671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.987199 (+1097528us) service_pool.cc:224] Handling call
0709 12:15:09.987623 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:09.987905 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:09.987947 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:09.988012 (+    65us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:09.988059 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.988133 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.988490 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:09.944900 20528 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.008s	sys 0.017s
I20260709 12:15:09.992982 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:09.964453 20199 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:09.960441 20356 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:09.971498 13259 rpcz_store.cc:276] 0709 12:15:08.665873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.666051 (+   178us) service_pool.cc:224] Handling call
0709 12:15:08.680854 (+ 14803us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:08.681293 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:08.681327 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:08.681986 (+   659us) log.cc:844] Serialized 26 byte log entry
0709 12:15:09.166728 (+484742us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:09.166773 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.166846 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.909712 (+742866us) raft_consensus.cc:1685] finished
0709 12:15:09.909740 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.910025 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":105,"thread_start_us":484290,"threads_started":1,"wal-append.queue_time_us":827376}
I20260709 12:15:09.999630 12494 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:09.993283 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:09.993263 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.079537 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2853) took 1528 ms. Trace:
I20260709 12:15:10.080252 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:10.080317 12639 rpcz_store.cc:276] 0709 12:15:08.470008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.471605 (+  1597us) service_pool.cc:224] Handling call
0709 12:15:09.998175 (+1526570us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:09.998411 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1221442944}
I20260709 12:15:10.081411 12642 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:09.967557 20688 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.082681 12639 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:10.083714 20688 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:09.997565 20528 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.009s	sys 0.017s
I20260709 12:15:09.980576 20112 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:10.085156 20112 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.086436 20673 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:10.087105 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 180241us
I20260709 12:15:10.087534 20112 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:09.985512 20502 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:10.088210 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:10.089201 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:09.980690 20663 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.966698 20541 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.098213 12495 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:15:09.985262 20669 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:10.100687 20541 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.101711 20665 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:10.102530 20665 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.104610 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3466) took 1214 ms. Trace:
I20260709 12:15:10.106567 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:09.973296 12496 rpcz_store.cc:276] 0709 12:15:08.734497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.936225 (+201728us) service_pool.cc:224] Handling call
0709 12:15:09.027693 (+ 91468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:09.547175 (+519482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:09.547208 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:09.547839 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 12:15:09.701043 (+153204us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:09.701096 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.701177 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:09.911221 (+210044us) raft_consensus.cc:1685] finished
0709 12:15:09.911244 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:09.911402 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1626,"raft.queue_time_us":170554,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":18411,"thread_start_us":596054,"threads_started":2,"wal-append.queue_time_us":331340}
I20260709 12:15:10.110291 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1363) took 1725 ms. Trace:
I20260709 12:15:10.110809 12570 rpcz_store.cc:276] 0709 12:15:08.385315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.705325 (+320010us) service_pool.cc:224] Handling call
0709 12:15:08.723106 (+ 17781us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:08.723390 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:08.723414 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:08.771664 (+ 48250us) log.cc:844] Serialized 26 byte log entry
0709 12:15:09.369396 (+597732us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:09.369452 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.369550 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:10.109982 (+740432us) raft_consensus.cc:1685] finished
0709 12:15:10.110008 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:10.110214 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":475028,"threads_started":1,"wal-append.queue_time_us":783829}
I20260709 12:15:09.970381 20680 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:10.115792 20680 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:09.992529 20548 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.079732 20694 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.126353 20694 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.129038 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:10.129973 12645 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:10.081156 20666 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.198755 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.199914 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.200244 20660 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:10.200767 12495 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:09.984342 20578 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.351s	user 0.015s	sys 0.005s
I20260709 12:15:10.105232 12563 rpcz_store.cc:276] 0709 12:15:08.889952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.995209 (+1105257us) service_pool.cc:224] Handling call
0709 12:15:10.103845 (+108636us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:10.104130 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:10.104173 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:10.104233 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:10.104281 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.104351 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:10.104521 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:10.200953 20660 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:10.208693 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 187304us
I20260709 12:15:10.212435 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:10.093376 20663 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.228502 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.229435 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.231240 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:15:10.229946 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:10.203181 19999 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.100s	user 0.012s	sys 0.003s
I20260709 12:15:09.969787 20086 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:10.210918 20600 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.046s	user 0.011s	sys 0.004s
I20260709 12:15:10.230643 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:10.237147 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:10.211802 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0890557650000119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
I20260709 12:15:10.238057 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.123155 20548 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:10.167112 20694 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:10.234236 20442 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.418s	user 0.014s	sys 0.002s
I20260709 12:15:10.326052 20663 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:10.326591 20667 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:10.213798 20401 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
W20260709 12:15:10.326256 20524 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.442s	user 0.010s	sys 0.005s
I20260709 12:15:10.327636 20524 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:10.319929 20581 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.918s	user 0.013s	sys 0.000s
I20260709 12:15:10.330026 20581 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:10.318995 20485 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.786s	user 0.012s	sys 0.004s
I20260709 12:15:10.331353 20678 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:10.331955 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:10.324926 12637 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:10.323658 20443 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.241950 20231 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.326188 20442 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.213110 20475 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:10.333333 20437 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.281831 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:10.362248 20475 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.363427 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.364859 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.333663 12568 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:10.326733 20663 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.342067 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.240150 20356 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:15:10.355115 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:10.362530 20437 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:10.332196 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 244100us
I20260709 12:15:10.216792 20673 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:15:10.130546 20657 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.365463 12496 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:10.336345 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:10.345326 12494 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
W20260709 12:15:10.305771 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61459.8us
I20260709 12:15:10.367612 20198 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:10.332327 20581 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.380921 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:10.363714 20653 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:10.246212 20600 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.366253 20697 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.341609 20613 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.241901 19999 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.394387 12490 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:10.394455 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:10.404249 20697 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.407646 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.408670 20697 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.409482 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.394913 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:10.361795 20231 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.388967 20198 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:10.396641 20356 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:10.422281 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:15:10.380229 20520 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:15:10.395056 20665 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.432055 12489 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:10.391673 20524 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.402206 20566 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.409462 20697 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:10.440443 20697 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:10.440888 20697 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:10.441817 20680 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.442203 20231 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:10.437279 20446 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.137s	user 0.015s	sys 0.000s
I20260709 12:15:10.443231 20446 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.444502 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.440456 20566 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.418669 20401 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:10.446221 20198 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.447049 20401 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:10.447639 20198 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:10.448205 20566 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.448021 20401 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.422590 20356 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:10.450074 20356 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:10.449687 20697 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.451932 19999 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.415243 20613 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.456597 20613 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.344198 20667 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.459705 20667 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:10.460878 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:10.445492 20446 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.464155 20600 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.468329 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3427) took 2132 ms. Trace:
I20260709 12:15:10.468918 12572 rpcz_store.cc:276] 0709 12:15:08.336393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.626936 (+290543us) service_pool.cc:224] Handling call
0709 12:15:08.627280 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:09.614973 (+987693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.467955 (+852982us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:10.468227 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2254768896}
I20260709 12:15:10.473309 20666 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:15:10.474082 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87149.1us
I20260709 12:15:10.474546 20472 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:10.474951 20472 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:10.475875 12609 connection.cc:256] RPC call timeout handler was delayed by -1.4526043480000226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1569s for a response
I20260709 12:15:10.476683 20657 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:10.478166 20695 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:15:10.478957 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3416) took 2684 ms (client timeout 1569 ms). Trace:
W20260709 12:15:10.479523 12571 rpcz_store.cc:269] 0709 12:15:07.800920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:07.806464 (+  5544us) service_pool.cc:224] Handling call
0709 12:15:10.478655 (+2672191us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:10.478828 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2129874816}
I20260709 12:15:10.481987 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:10.488677 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157797us
I20260709 12:15:10.503073 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3434) took 1866 ms. Trace:
W20260709 12:15:10.509868 20698 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.002s
I20260709 12:15:10.456661 20697 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.527252 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3437) took 1863 ms. Trace:
I20260709 12:15:10.531468 12569 rpcz_store.cc:276] 0709 12:15:08.663934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.971077 (+1307143us) service_pool.cc:224] Handling call
0709 12:15:09.980703 (+  9626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:09.981032 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:15:10.379789 (+398757us) log.cc:844] Serialized 14 byte log entry
0709 12:15:10.524756 (+144967us) log.cc:844] Serialized 14 byte log entry
0709 12:15:10.526834 (+  2078us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:10.526896 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:15:10.526957 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.527036 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:10.527186 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14310,"prepare.queue_time_us":450284,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":10330,"raft.queue_time_us":5842,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":123,"spinlock_wait_cycles":1088256,"thread_start_us":294642,"threads_started":2}
I20260709 12:15:10.507645 20475 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.507853 12566 rpcz_store.cc:276] 0709 12:15:08.637053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.803899 (+1166846us) service_pool.cc:224] Handling call
0709 12:15:09.831122 (+ 27223us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:09.893673 (+ 62551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:09.893706 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:09.902146 (+  8440us) log.cc:844] Serialized 52 byte log entry
0709 12:15:10.214191 (+312045us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:10.500335 (+286144us) log.cc:844] Serialized 14 byte log entry
0709 12:15:10.502692 (+  2357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.502773 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:10.502831 (+    58us) raft_consensus.cc:1685] finished
0709 12:15:10.502842 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:10.503008 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37660,"raft.queue_time_us":106294,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":27370,"thread_start_us":574678,"threads_started":2,"wal-append.queue_time_us":473542}
I20260709 12:15:10.518695 20520 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.523532 20689 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:10.475965 20472 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.543116 20678 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:10.573324 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:10.563810 20689 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.575278 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3463) took 1686 ms. Trace:
I20260709 12:15:10.575887 12567 rpcz_store.cc:276] 0709 12:15:08.889075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.281875 (+392800us) service_pool.cc:224] Handling call
0709 12:15:10.574880 (+1293005us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:10.575191 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2258558208}
I20260709 12:15:10.583251 20663 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.585891 20442 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.599314 12495 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:10.529398 20613 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:15:10.586370 20669 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.558954 20520 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.587035 20472 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:10.571403 20502 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.573441 20658 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:10.571027 20660 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.622866 20678 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:10.623549 20664 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.805s	user 0.011s	sys 0.008s
I20260709 12:15:10.607911 20086 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:15:10.632073 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1057) took 1845 ms (client timeout 2109 ms). Trace:
W20260709 12:15:10.632650 12489 rpcz_store.cc:269] 0709 12:15:08.786204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.790340 (+  4136us) service_pool.cc:224] Handling call
0709 12:15:10.631720 (+1841380us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:10.631948 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3534094720}
W20260709 12:15:10.637908 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:10.638518 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:10.639258 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:10.646204 12496 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:10.650488 12493 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:15:10.638000 12643 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:10.653710 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.654518 12565 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:10.671452 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:10.673760 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.679157 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.680186 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.681072 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
W20260709 12:15:10.683671 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 159328us
I20260709 12:15:10.655233 20660 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.688690 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.690253 12563 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 21 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:15:10.691501 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.695205 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.700902 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.701327 12569 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:10.675136 20528 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:10.706368 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.646014 20502 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.708854 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.709285 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:10.717300 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.718258 12567 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:10.720814 12567 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:10.712716 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.713718 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.724610 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.725327 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.728214 12565 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:10.731185 13262 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:10.723001 12563 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:10.694197 20713 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.714603 12569 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:10.746603 20520 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:10.748209 20708 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:15:10.750046 20020 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.751398 20112 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.752128 20112 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.755215 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.756078 13262 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:10.757104 13257 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:15:10.762055 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.762897 12497 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:15:10.763695 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:10.765750 12496 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:10.766783 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.712399 20706 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.761212 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.763207 12497 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:10.774631 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.775333 12563 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:10.701078 20660 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.777522 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.772100 12497 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.781281 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.789806 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.789960 12497 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:10.794595 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.806653 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.807658 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.768409 20713 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.886220 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:10.872893 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157845us
I20260709 12:15:10.798063 12489 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:10.881712 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85948.4us
I20260709 12:15:10.882156 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:10.882998 13253 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:10.883538 20472 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:10.843573 20669 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:10.893777 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 195159us
I20260709 12:15:10.898046 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.777894 20706 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.904737 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.912840 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.914848 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.922083 20112 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.897655 20520 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:10.922827 20720 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:15:10.923166 20520 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.924345 20715 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:10.924995 20715 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.927287 20715 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:10.928153 20707 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:15:10.934132 12497 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:10.929994 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.937064 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:10.935362 13255 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:15:10.935586 12496 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 18.
I20260709 12:15:10.936036 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.939476 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.942579 20520 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:10.916572 20669 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:10.943626 20669 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.898480 20678 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:15:10.902762 20689 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.938717 12489 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:10.909601 20356 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.947484 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.947878 20706 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.948508 20713 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:10.949386 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:10.895334 20472 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:10.949594 20502 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:10.950472 20660 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.979295 20669 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:10.980762 20721 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:15:10.982849 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3596) took 2217 ms. Trace:
I20260709 12:15:10.983397 12491 rpcz_store.cc:276] 0709 12:15:08.765571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.216138 (+450567us) service_pool.cc:224] Handling call
0709 12:15:09.239651 (+ 23513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:09.266317 (+ 26666us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:09.332971 (+ 66654us) log.cc:844] Serialized 14 byte log entry
0709 12:15:10.642214 (+1309243us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b1800381a00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:15:10.723538 (+ 81324us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:10.729854 (+  6316us) log.cc:844] Serialized 26 byte log entry
0709 12:15:10.730057 (+   203us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:10.730110 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.730211 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:10.982345 (+252134us) raft_consensus.cc:1685] finished
0709 12:15:10.982538 (+   193us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:10.982761 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39071,"prepare.queue_time_us":83258,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":56,"raft.queue_time_us":207122,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":25886,"spinlock_wait_cycles":4269312,"thread_start_us":81008,"threads_started":1,"wal-append.queue_time_us":1404867}
I20260709 12:15:10.986429 13254 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:15:10.987224 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.988193 12491 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:10.989602 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:10.979563 20676 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.157s	user 0.021s	sys 0.013s
I20260709 12:15:10.985428 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.991530 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.992329 12491 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:10.993037 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:10.993989 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.994268 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:10.995601 20690 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.998765 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:10.999974 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1365) took 2374 ms. Trace:
I20260709 12:15:11.000594 12568 rpcz_store.cc:276] 0709 12:15:08.625232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.762962 (+1137730us) service_pool.cc:224] Handling call
0709 12:15:09.764552 (+  1590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:10.365856 (+601304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.999556 (+633700us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:10.999883 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308339072}
I20260709 12:15:11.001467 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1051) took 2804 ms. Trace:
I20260709 12:15:11.002043 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.001947 12492 rpcz_store.cc:276] 0709 12:15:08.196857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:08.256333 (+ 59476us) service_pool.cc:224] Handling call
0709 12:15:08.279056 (+ 22723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:09.406323 (+1127267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.001202 (+1594879us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:11.001360 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2570709376}
I20260709 12:15:11.003535 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.003834 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.004204 12492 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
W20260709 12:15:11.004360 12537 connection.cc:256] RPC call timeout handler was delayed by -1.5034133229999043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:15:11.004777 12645 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:15:11.005481 13254 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:15:11.006074 12565 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 12. (term mismatch)
I20260709 12:15:11.006479 12565 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:11.006731 12565 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20260709 12:15:11.007097 12565 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:11.007432 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.008169 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.008746 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:10.959383 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:11.010481 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51185.7us
I20260709 12:15:11.010859 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:11.011375 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:11.012115 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.012250 20678 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:11.012607 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:11.008824 12565 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:10.958817 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.022878 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.023413 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.023828 12565 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:10.965191 20472 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:11.025079 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.013391 20437 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:10.961735 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.026226 13255 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:09.713946 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:11.016247 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:10.959559 20713 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:10.963833 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:11.019749 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:11.027840 13262 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:11.029961 20706 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:11.030937 20706 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:11.031940 20568 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:11.032347 20713 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:11.032910 20713 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:11.033546 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3291) took 1134 ms. Trace:
I20260709 12:15:11.034161 20713 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.035913 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:11.033986 12638 rpcz_store.cc:276] 0709 12:15:09.899642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.915345 (+ 15703us) service_pool.cc:224] Handling call
0709 12:15:09.915750 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:11.032695 (+1116945us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:11.032722 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:11.032761 (+    39us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:11.032791 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.033285 (+   494us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:11.033348 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:11.033486 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2568235264}
W20260709 12:15:11.039685 20670 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.214s	user 0.013s	sys 0.000s
I20260709 12:15:11.040405 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:10.959565 20356 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:11.010748 20444 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:11.005144 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:11.013458 20678 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:11.042858 20678 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:10.992792 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.041356 20444 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:11.043357 20678 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:11.043625 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.044741 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:11.045959 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:11.006596 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:11.046856 13254 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:11.046976 20020 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:11.047626 20020 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:11.047822 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.048658 12564 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:11.048570 20020 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:10.957542 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:11.037523 12496 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:15:11.040688 20456 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:11.042248 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.043095 12492 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:15:11.010563 12568 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:15:10.998765 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:10.964540 20714 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:11.015708 13253 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:11.017122 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:11.052011 12611 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:11.054996 12611 leader_election.cc:403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:11.044926 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:10.957731 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.056316 12497 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:11.055651 12638 raft_consensus.cc:2376] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26. Re-sending same reply.
W20260709 12:15:11.043313 20684 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.218s	user 0.007s	sys 0.005s
I20260709 12:15:11.049597 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.070225 12564 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:15:11.059167 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.078451 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.046065 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.088137 12565 raft_consensus.cc:2376] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22. Re-sending same reply.
I20260709 12:15:11.089540 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.090703 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.091105 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.092303 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.061224 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.097470 12645 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24. Re-sending same reply.
I20260709 12:15:11.099519 12491 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:11.099372 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:11.100052 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:11.101133 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.103979 12638 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:11.104880 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.105655 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.107609 12638 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:15:11.108592 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49538.1us
I20260709 12:15:11.109022 12565 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:11.110622 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.108896 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.112032 12568 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:11.106966 20356 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:15:11.059399 20444 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:11.067179 20472 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:11.118134 20472 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:11.118816 20472 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:11.051434 20456 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:11.119791 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.120020 20715 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:11.059167 20673 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:11.121073 20502 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:11.120790 12568 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:11.122414 20689 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:11.122830 20689 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:11.123893 20689 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:11.118021 20727 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:15:11.118059 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.139520 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:11.142058 12497 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:15:11.143920 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.145576 12645 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:15:11.159862 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.150552 20714 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:11.120836 20715 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.010s
I20260709 12:15:11.167917 20494 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:11.168412 20714 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:11.168674 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.169188 20494 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:11.121300 20673 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.169564 12645 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:15:11.169302 12637 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:11.162753 13254 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:11.172219 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:11.173170 12563 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:11.174777 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:11.178778 12567 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:11.134286 20669 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:11.176054 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:11.182451 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:11.186049 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.187674 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:11.188961 13254 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:11.189170 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:11.191802 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:11.169696 20494 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:11.199608 12638 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
W20260709 12:15:11.202054 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:11.201905 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:11.183581 20714 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:11.203768 20714 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:11.204748 20714 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.208701 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.211344 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.211968 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:11.212809 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.214607 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.215042 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.185850 20198 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:11.207394 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.217458 13253 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:11.213589 12497 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:15:11.192324 20666 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:11.225751 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.234273 13254 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:15:11.235754 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.198792 20112 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:11.235643 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:11.238591 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.242271 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:11.243995 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.244640 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:11.245003 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.247409 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.189033 20668 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:15:11.255179 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.222672 20658 raft_consensus.cc:2764] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:15:11.258759 13255 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:11.259755 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:11.260037 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.279917 12497 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:11.251533 20198 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:11.200372 20473 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:15:11.266317 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.266780 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:11.268471 20356 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:11.241195 20666 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:11.257014 20707 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:11.253909 20112 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.294776 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.322320 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1041) took 1336 ms. Trace:
I20260709 12:15:11.326644 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.327713 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:11.293264 20704 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:11.297922 20686 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:11.343755 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.349076 12497 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:11.348966 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.351094 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:11.347957 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 324430us
I20260709 12:15:11.358654 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.349953 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:15:11.366295 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 415762us
I20260709 12:15:11.367282 13257 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:11.361275 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.371346 12645 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:15:11.310428 20666 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:11.372912 20431 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:11.377821 20198 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.379246 20112 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:11.381752 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.384179 20689 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:11.385156 20456 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:11.385658 20545 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:11.403992 12572 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260709 12:15:11.395159 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:11.404500 12572 pending_rounds.cc:85] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:15:11.398841 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:11.404855 12572 pending_rounds.cc:107] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:15:11.405258 12572 raft_consensus.cc:2889] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:11.348450 20686 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:11.364020 20704 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:11.387156 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.388345 20472 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:11.384253 20673 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:11.391070 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1032) took 2121 ms (client timeout 2510 ms). Trace:
I20260709 12:15:11.364493 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:11.402426 12565 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 12:15:11.423321 12565 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:15:11.423597 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1367) took 1219 ms. Trace:
I20260709 12:15:11.423923 12565 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:15:11.424402 12565 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:11.355398 20678 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.428207 20678 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:11.408818 20686 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:11.420156 20538 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:15:11.433002 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.434662 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:11.436066 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:11.409140 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1092) took 2380 ms. Trace:
I20260709 12:15:11.406003 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.438481 20431 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:11.438937 20704 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:11.439435 20706 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:11.440253 20538 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:11.409324 20690 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:15:11.440999 20538 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:11.442165 20690 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:11.442104 20538 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 25.9, Last appended by leader: 9, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.442967 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.443962 12637 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:15:11.444885 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.445425 20713 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:11.422660 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:11.422444 12646 rpcz_store.cc:269] 0709 12:15:09.269906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.270041 (+   135us) service_pool.cc:224] Handling call
0709 12:15:11.390806 (+2120765us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:11.390962 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":537011456}
I20260709 12:15:11.424165 12572 rpcz_store.cc:276] 0709 12:15:10.204313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.392169 (+1187856us) service_pool.cc:224] Handling call
0709 12:15:11.392602 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:11.420007 (+ 27405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.423504 (+  3497us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:11.428227 20700 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.858s	user 0.003s	sys 0.002s
W20260709 12:15:11.447124 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82744.3us
I20260709 12:15:11.437088 13261 rpcz_store.cc:276] 0709 12:15:09.025674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.025888 (+   214us) service_pool.cc:224] Handling call
0709 12:15:09.049676 (+ 23788us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:09.674570 (+624894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.406030 (+1731460us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:11.406331 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1392890112}
I20260709 12:15:11.439600 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.450712 12568 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:11.451092 12568 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:11.442665 20690 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:11.448546 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:11.453660 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:11.438993 20431 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:11.458796 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:11.366427 20356 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:15:11.408842 20112 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:11.460028 13261 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:11.437371 20707 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:11.461081 12568 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.409143 20437 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:15:11.465297 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.467530 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.459662 20545 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:11.469666 20658 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:11.469844 20469 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:11.473073 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:11.458140 20523 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260709 12:15:11.455902 20673 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:11.459460 20690 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:11.537482 12497 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:11.489326 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:11.487386 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:11.457036 20456 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:11.449910 20582 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:11.488690 12494 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.013s	sys 0.026s
I20260709 12:15:11.543530 12568 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:11.465206 20431 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:11.547199 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.486550 20112 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:11.493224 20658 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:11.551384 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:11.535844 20657 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:11.555936 20668 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:11.552554 12645 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:11.463024 20678 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.082s
I20260709 12:15:11.562263 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.556447 20657 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.545313 20456 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:11.571964 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.572894 12495 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:11.555011 20112 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:11.520252 20545 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:11.576025 12383 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.633s	user 0.018s	sys 0.035s
I20260709 12:15:11.558902 20658 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:11.607514 20473 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:11.541493 20660 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.026s
I20260709 12:15:11.627462 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.632566 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:11.632898 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.634478 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:11.634711 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 198708us
W20260709 12:15:11.635635 12492 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.015s	sys 0.019s
I20260709 12:15:11.635291 12645 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:11.645869 12492 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.015s	sys 0.019s
W20260709 12:15:11.634513 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 446718us
I20260709 12:15:11.673710 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.577764 20456 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:11.681582 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.684162 12495 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:11.694531 12568 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
W20260709 12:15:11.679068 20735 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.114s	user 0.001s	sys 0.001s
I20260709 12:15:11.695122 12568 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:11.667043 20443 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:11.697796 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.701205 12568 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 21.12
W20260709 12:15:11.701329 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 213962us
I20260709 12:15:11.785728 12568 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:11.784063 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.793417 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:11.699119 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:11.796795 12645 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:11.797299 12565 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:11.688568 20437 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:15:11.813841 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:11.628773 20473 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:11.819260 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.838317 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.839064 12645 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:11.824693 20444 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.708s	user 0.009s	sys 0.009s
I20260709 12:15:11.848174 12565 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
W20260709 12:15:11.849357 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56086.6us
I20260709 12:15:11.853394 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:11.823362 20714 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:11.852172 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:11.861919 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:11.821477 20758 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:15:11.855105 20472 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:11.854562 20437 raft_consensus.cc:2764] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:15:11.916213 12645 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:11.923623 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:11.924944 12645 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:15:11.925386 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2867) took 1019 ms. Trace:
W20260709 12:15:11.926277 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 257212us
I20260709 12:15:11.917784 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1039) took 1940 ms. Trace:
I20260709 12:15:11.925725 12645 rpcz_store.cc:276] 0709 12:15:10.906076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.922824 (+1016748us) service_pool.cc:224] Handling call
0709 12:15:11.925330 (+  2506us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:11.927459 12642 rpcz_store.cc:276] 0709 12:15:09.977600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.082351 (+104751us) service_pool.cc:224] Handling call
0709 12:15:10.082733 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:10.082999 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:15:11.318965 (+1235966us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.917203 (+598238us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:11.917340 (+   137us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:11.917419 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.917531 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:11.917716 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1248390,"prepare.queue_time_us":1237053,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":154738,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":224,"thread_start_us":581449,"threads_started":2}
I20260709 12:15:11.936828 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:11.939028 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:11.858654 20444 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:11.956485 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.957760 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2874) took 1029 ms. Trace:
I20260709 12:15:11.928978 20401 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:11.975186 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:11.982132 12645 raft_consensus.cc:2376] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26. Re-sending same reply.
I20260709 12:15:11.982831 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3311) took 1027 ms. Trace:
I20260709 12:15:11.917838 20757 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:11.958199 12642 rpcz_store.cc:276] 0709 12:15:10.927875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.938281 (+1010406us) service_pool.cc:224] Handling call
0709 12:15:11.957644 (+ 19363us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:11.976183 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114404us
I20260709 12:15:11.983260 12645 rpcz_store.cc:276] 0709 12:15:10.955532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.940420 (+984888us) service_pool.cc:224] Handling call
0709 12:15:11.982747 (+ 42327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:11.940884 20356 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.012046 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.021107 12489 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:15:11.960227 20657 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.061864 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.063920 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.066057 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.067543 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:12.068230 12564 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:11.935307 20494 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.072221 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2844) took 1263 ms. Trace:
I20260709 12:15:12.072695 12496 rpcz_store.cc:276] 0709 12:15:10.808686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.052139 (+243453us) service_pool.cc:224] Handling call
0709 12:15:11.107540 (+ 55401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:11.131357 (+ 23817us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:11.560942 (+429585us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.847325 (+286383us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:11.847949 (+   624us) log.cc:844] Serialized 26 byte log entry
0709 12:15:11.848107 (+   158us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:11.848140 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.848214 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:12.071990 (+223776us) raft_consensus.cc:1685] finished
0709 12:15:12.072010 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.072151 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18661,"prepare.queue_time_us":45754,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":587,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":17326,"thread_start_us":652762,"threads_started":2,"wal-append.queue_time_us":350530}
I20260709 12:15:12.074093 12563 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:15:12.074925 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:12.076859 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.077899 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:12.078552 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2870) took 1151 ms. Trace:
I20260709 12:15:12.078994 12638 rpcz_store.cc:276] 0709 12:15:10.926771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.076496 (+1149725us) service_pool.cc:224] Handling call
0709 12:15:12.078470 (+  1974us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.080220 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.081261 12638 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:15:12.041522 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.048074 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2843) took 1254 ms. Trace:
W20260709 12:15:12.048755 20755 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
W20260709 12:15:12.049584 20746 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:15:12.050403 20740 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.197s	user 0.000s	sys 0.000s
W20260709 12:15:12.051803 20686 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.001s
I20260709 12:15:12.058012 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.059163 20673 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:12.059394 12565 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:12.060158 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.061398 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.105928 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:15:12.082428 12492 rpcz_store.cc:276] 0709 12:15:10.793714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.051757 (+258043us) service_pool.cc:224] Handling call
0709 12:15:11.052073 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:11.052881 (+   808us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:11.129903 (+ 77022us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.646967 (+517064us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:11.647719 (+   752us) log.cc:844] Serialized 26 byte log entry
0709 12:15:11.647899 (+   180us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:11.647947 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.648053 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:12.047797 (+399744us) raft_consensus.cc:1685] finished
0709 12:15:12.047821 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.047991 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":76809,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":63,"raft.queue_time_us":21097,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":52032,"thread_start_us":516492,"threads_started":1,"wal-append.queue_time_us":763898}
I20260709 12:15:12.089030 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2887) took 1143 ms. Trace:
I20260709 12:15:12.108309 12638 rpcz_store.cc:276] 0709 12:15:10.945171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.079854 (+1134683us) service_pool.cc:224] Handling call
0709 12:15:12.088876 (+  9022us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.109617 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.110419 12638 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:12.111212 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3303) took 1158 ms. Trace:
I20260709 12:15:12.109997 12564 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:15:12.086787 12460 connection.cc:256] RPC call timeout handler was delayed by -1.0292071399999259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
I20260709 12:15:12.065414 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.113499 12568 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:12.111723 12638 rpcz_store.cc:276] 0709 12:15:10.952436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.109277 (+1156841us) service_pool.cc:224] Handling call
0709 12:15:12.111139 (+  1862us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.106213 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 18.
I20260709 12:15:12.117089 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.136261 12492 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:15:12.142818 20401 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:12.143841 20401 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:12.149490 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:12.155246 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3293) took 2250 ms (client timeout 1770 ms). Trace:
W20260709 12:15:12.155733 12643 rpcz_store.cc:269] 0709 12:15:09.904730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.919361 (+ 14631us) service_pool.cc:224] Handling call
0709 12:15:12.154883 (+2235522us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.155156 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1600556800}
I20260709 12:15:12.157232 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.158159 12640 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:12.159544 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3312) took 1203 ms. Trace:
I20260709 12:15:12.163059 12642 rpcz_store.cc:276] 0709 12:15:10.955769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.019871 (+1064102us) service_pool.cc:224] Handling call
0709 12:15:12.159453 (+139582us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.159627 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:12.087574 20356 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:12.167654 20743 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.286s	user 0.001s	sys 0.001s
I20260709 12:15:12.169344 20356 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:12.170938 12638 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25. Re-sending same reply.
I20260709 12:15:12.092772 20545 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:12.152127 20731 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.521s	user 0.002s	sys 0.001s
I20260709 12:15:12.093657 20739 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:12.104641 20673 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:12.105041 20714 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:12.185111 20765 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:15:12.176147 20733 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.359s	user 0.002s	sys 0.000s
I20260709 12:15:12.191870 12492 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:15:12.116775 20686 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:12.180816 20690 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.630s	user 0.011s	sys 0.000s
W20260709 12:15:12.181890 20112 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.001s
I20260709 12:15:12.173666 20770 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.198899 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.170161 20758 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:12.182483 20761 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:12.180109 20473 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:12.206676 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 194780us
I20260709 12:15:12.206916 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3611) took 1016 ms. Trace:
I20260709 12:15:12.215116 12609 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:12.214123 12491 rpcz_store.cc:276] 0709 12:15:11.191026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.191178 (+   152us) service_pool.cc:224] Handling call
0709 12:15:11.191541 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:11.200613 (+  9072us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:11.200646 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:11.235218 (+ 34572us) log.cc:844] Serialized 26 byte log entry
0709 12:15:11.676771 (+441553us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:11.676837 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.676947 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:12.206641 (+529694us) raft_consensus.cc:1685] finished
0709 12:15:12.206664 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.206828 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1563,"raft.queue_time_us":9118,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":11294,"thread_start_us":417592,"threads_started":1,"wal-append.queue_time_us":458202}
I20260709 12:15:12.210577 12492 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
W20260709 12:15:12.210793 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 426830us
I20260709 12:15:12.217283 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3612) took 1025 ms. Trace:
I20260709 12:15:12.217985 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.211172 12642 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:15:12.217883 12496 rpcz_store.cc:276] 0709 12:15:11.191417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.076168 (+884751us) service_pool.cc:224] Handling call
0709 12:15:12.076648 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:12.076885 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:15:12.077250 (+   365us) log.cc:844] Serialized 14 byte log entry
0709 12:15:12.216935 (+139685us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:12.216989 (+    54us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:12.217033 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:12.217098 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.217222 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":197,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":139330,"threads_started":1}
I20260709 12:15:12.211898 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.221396 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.218657 12492 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:15:12.222488 20538 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:12.228363 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.231151 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2889) took 1285 ms. Trace:
I20260709 12:15:12.176676 20658 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.232129 12642 rpcz_store.cc:276] 0709 12:15:10.945717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.182033 (+1236316us) service_pool.cc:224] Handling call
0709 12:15:12.231066 (+ 49033us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.183959 20739 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.189850 20456 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.213769 20442 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:12.219839 20381 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:12.230726 20538 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:12.252044 12646 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24.
I20260709 12:15:12.256160 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.209836 20690 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:12.261337 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44664.7us
I20260709 12:15:12.210412 20112 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.239739 20752 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:12.263993 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.265787 12645 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:12.267479 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3598) took 2287 ms (client timeout 1790 ms). Trace:
I20260709 12:15:12.267755 20657 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:15:12.268071 12490 rpcz_store.cc:269] 0709 12:15:09.974257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.974414 (+   157us) service_pool.cc:224] Handling call
0709 12:15:12.261363 (+2286949us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.261587 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":879197952}
I20260709 12:15:12.270445 20444 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:11.323822 12639 rpcz_store.cc:276] 0709 12:15:09.985575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.083976 (+ 98401us) service_pool.cc:224] Handling call
0709 12:15:10.084380 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:10.212955 (+128575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:10.213001 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:10.227744 (+ 14743us) log.cc:844] Serialized 26 byte log entry
0709 12:15:10.399976 (+172232us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:11.319642 (+919666us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.321814 (+  2172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.321936 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:11.322002 (+    66us) raft_consensus.cc:1685] finished
0709 12:15:11.322016 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:11.322228 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1045050,"raft.queue_time_us":145906,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7187,"thread_start_us":158088,"threads_started":1,"wal-append.queue_time_us":263831}
I20260709 12:15:12.273123 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.273896 20767 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:12.214920 20770 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.226632 20758 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:12.280884 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:12.292798 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.248126 20442 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:12.240475 20653 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:12.287379 20773 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:12.288460 12495 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:12.298576 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3310) took 1343 ms. Trace:
I20260709 12:15:12.307797 12639 rpcz_store.cc:276] 0709 12:15:10.955273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.292440 (+1337167us) service_pool.cc:224] Handling call
0709 12:15:12.298459 (+  6019us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:12.319325 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63343.4us
I20260709 12:15:12.325796 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:12.327699 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 168125us
I20260709 12:15:12.232545 20673 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.329509 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:12.302446 20713 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.856s	user 0.007s	sys 0.008s
I20260709 12:15:12.293161 20356 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:12.261991 20381 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:12.322607 20689 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.938s	user 0.015s	sys 0.000s
I20260709 12:15:12.306422 20442 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:12.333032 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:12.328900 20479 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.335194 12488 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
W20260709 12:15:12.328768 20771 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:15:12.252877 20714 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.339825 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:12.343415 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81027.1us
I20260709 12:15:12.345594 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:12.349373 12637 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:12.334158 20443 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:12.333999 20356 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:12.353688 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.360505 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:12.362656 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:12.354449 12565 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 12:15:12.357883 20782 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:15:12.358572 20520 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:15:12.340554 20713 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:12.293398 20767 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.363657 12565 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:15:12.346185 20673 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:12.339711 20780 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:12.373790 12565 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 12:15:12.378358 12565 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:12.351468 20444 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:12.376823 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:12.324683 20773 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.371050 20752 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:12.390854 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:12.398532 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:12.399367 12565 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:12.380419 20786 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:12.401778 20783 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:15:12.386325 20545 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.402202 20786 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.402567 20545 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.390240 20772 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:15:12.407277 20356 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.408712 20356 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:12.411362 20456 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:12.390045 20713 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:12.413094 20739 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:12.399168 20658 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:12.392553 20780 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:12.416343 20780 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.392241 20686 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.385937 20689 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:15:12.444384 20788 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:15:12.378034 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 4.505s	user 0.106s	sys 0.093s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":24797,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":74375,"lbm_read_time_us":94858,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":19,"lbm_write_time_us":5109,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":9,"peak_mem_usage":253719,"reinsert_count":0,"thread_start_us":432042,"threads_started":3,"update_count":154,"wal-append.queue_time_us":597595}
W20260709 12:15:12.479707 20705 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.864s	user 0.005s	sys 0.009s
I20260709 12:15:12.425971 20690 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:12.507994 20682 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.690s	user 0.004s	sys 0.019s
I20260709 12:15:12.429101 20689 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:12.560089 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:12.570676 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3622) took 1238 ms. Trace:
I20260709 12:15:12.572394 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.571228 12497 rpcz_store.cc:276] 0709 12:15:11.332427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.350806 (+ 18379us) service_pool.cc:224] Handling call
0709 12:15:12.570446 (+1219640us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.570601 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":291408640}
I20260709 12:15:12.573088 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:12.578778 20473 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:12.583630 20473 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:12.584095 20473 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:12.595216 20668 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:12.607462 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.607450 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:12.609287 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:12.610152 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:12.615437 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1033) took 3326 ms. Trace:
I20260709 12:15:12.615964 12641 rpcz_store.cc:276] 0709 12:15:09.270380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.270543 (+   163us) service_pool.cc:224] Handling call
0709 12:15:09.308850 (+ 38307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:09.392496 (+ 83646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:09.392533 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:09.436197 (+ 43664us) log.cc:844] Serialized 26 byte log entry
0709 12:15:09.698639 (+262442us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:09.698748 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:09.698912 (+   164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:12.375797 (+2676885us) spinlock_profiling.cc:243] Waited 9.13 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:12.578974 (+203177us) raft_consensus.cc:1685] finished
0709 12:15:12.579000 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.596621 (+ 17621us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19337,"raft.queue_time_us":114457,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":27583,"spinlock_wait_cycles":20990720,"thread_start_us":197338,"threads_started":1,"wal-append.queue_time_us":349299}
I20260709 12:15:12.618027 13261 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:12.632869 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 287314us
I20260709 12:15:12.637527 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2860) took 1733 ms. Trace:
W20260709 12:15:12.638195 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 270418us
I20260709 12:15:12.640626 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:12.584885 20689 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:12.639842 12640 rpcz_store.cc:276] 0709 12:15:10.904206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.906310 (+  2104us) service_pool.cc:224] Handling call
0709 12:15:12.157949 (+1251639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:12.159892 (+  1943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:12.637264 (+477372us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54002e5c50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.637443 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2832989952}
I20260709 12:15:12.643154 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:12.643839 12497 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:12.639304 12565 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:12.645524 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:12.646155 12497 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
W20260709 12:15:12.648691 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5894892329999948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
I20260709 12:15:12.651526 12497 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 9. Preceding OpId from leader: term: 26 index: 10. (index mismatch)
I20260709 12:15:12.650245 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:12.652371 12496 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:12.651363 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:12.653050 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3606) took 2431 ms. Trace:
W20260709 12:15:12.653087 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45756.5us
I20260709 12:15:12.653900 12490 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:12.652446 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3005892329999824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
W20260709 12:15:12.654410 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2330892330001006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20260709 12:15:12.654740 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1898892330000308s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
I20260709 12:15:12.603381 20198 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:12.655706 13259 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:15:12.653771 12493 rpcz_store.cc:276] 0709 12:15:10.221938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.222152 (+   214us) service_pool.cc:224] Handling call
0709 12:15:10.650148 (+427996us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:10.651082 (+   934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:12.652796 (+2001714us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b54001900d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.652965 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":931686272}
I20260709 12:15:12.649024 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:12.603853 20715 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.651162 20790 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:12.661726 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.662930 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3328) took 1490 ms. Trace:
I20260709 12:15:12.663363 12641 rpcz_store.cc:276] 0709 12:15:11.172459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.636162 (+1463703us) service_pool.cc:224] Handling call
0709 12:15:12.662857 (+ 26695us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.663825 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:12.657709 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.658232 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.658457 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:12.660825 20715 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.671038 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.672904 20739 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:12.674041 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:12.674681 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:12.640825 20668 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.676468 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:12.660764 20112 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:12.677598 20538 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:15:12.678486 20444 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:15:12.679695 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:12.682276 20702 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.052s	user 0.004s	sys 0.004s
I20260709 12:15:12.685947 20657 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:12.660853 20790 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:12.687947 20790 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:12.656976 20690 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.694177 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3468) took 3300 ms. Trace:
I20260709 12:15:12.694615 12570 rpcz_store.cc:276] 0709 12:15:09.394125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.112218 (+718093us) service_pool.cc:224] Handling call
0709 12:15:10.112631 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:10.113085 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:10.113115 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:10.202509 (+ 89394us) log.cc:844] Serialized 26 byte log entry
0709 12:15:10.611237 (+408728us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:10.613697 (+  2460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:10.614017 (+   320us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:12.597514 (+1983497us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:12.693784 (+ 96270us) raft_consensus.cc:1685] finished
0709 12:15:12.693808 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.694077 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1082,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"spinlock_wait_cycles":36008832,"thread_start_us":293008,"threads_started":1,"wal-append.queue_time_us":360740}
I20260709 12:15:12.682264 20545 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:12.697113 20431 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.140s	user 0.013s	sys 0.009s
I20260709 12:15:12.697325 20545 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:12.671694 20479 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.028s
I20260709 12:15:12.665841 20437 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:12.693732 20690 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.671715 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:12.697438 20732 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.064s	user 0.001s	sys 0.005s
I20260709 12:15:12.697569 20431 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.709376 20752 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:12.709697 12566 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:12.698731 20545 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.012s
I20260709 12:15:12.711050 20752 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:12.699576 20715 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:12.711730 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.711963 20715 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:12.677264 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1069) took 1279 ms. Trace:
I20260709 12:15:12.712180 20752 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.701036 20437 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.715140 20437 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.715890 20437 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:12.703091 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.703994 12640 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:15:12.718178 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2890) took 1772 ms. Trace:
I20260709 12:15:12.671083 20409 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:12.718601 12640 rpcz_store.cc:276] 0709 12:15:10.945985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.651478 (+1705493us) service_pool.cc:224] Handling call
0709 12:15:12.718101 (+ 66623us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.719710 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.720530 12640 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:12.720960 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3322) took 1762 ms. Trace:
I20260709 12:15:12.721588 20668 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:12.719374 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48412us
I20260709 12:15:12.705616 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:12.723241 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.705092 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:12.697943 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:12.731928 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:12.728634 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:12.741250 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:12.721294 20401 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.577s	user 0.015s	sys 0.007s
I20260709 12:15:12.760507 20401 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.762003 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.762292 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:12.762801 20401 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:12.763073 12641 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:12.763652 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3315) took 1807 ms. Trace:
I20260709 12:15:12.763975 12641 rpcz_store.cc:276] 0709 12:15:10.956501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.672192 (+1715691us) service_pool.cc:224] Handling call
0709 12:15:12.763579 (+ 91387us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.764691 20715 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:12.765089 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.765959 20767 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:12.766654 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:12.769114 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:12.717478 20770 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:12.706202 20773 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:12.770067 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:12.707321 20784 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:12.703186 20690 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:12.708354 20790 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.730376 13260 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:12.735320 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.774531 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:12.721256 12640 rpcz_store.cc:276] 0709 12:15:10.958847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.719429 (+1760582us) service_pool.cc:224] Handling call
0709 12:15:12.720900 (+  1471us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:12.778790 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:12.779898 20790 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.774461 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:12.672035 20198 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:12.712759 12492 rpcz_store.cc:276] 0709 12:15:11.397982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.231792 (+833810us) service_pool.cc:224] Handling call
0709 12:15:12.241915 (+ 10123us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:15:12.257553 (+ 15638us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:12.257610 (+    57us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:15:12.263347 (+  5737us) log.cc:844] Serialized 104 byte log entry
0709 12:15:12.329440 (+ 66093us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:12.350338 (+ 20898us) log.cc:844] Serialized 14 byte log entry
0709 12:15:12.352522 (+  2184us) log.cc:844] Serialized 14 byte log entry
0709 12:15:12.364010 (+ 11488us) log.cc:844] Serialized 14 byte log entry
0709 12:15:12.638396 (+274386us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:12.638475 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:12.677009 (+ 38534us) raft_consensus.cc:1685] finished
0709 12:15:12.677031 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:12.677200 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12023,"prepare.queue_time_us":27376,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":107212,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":11899,"spinlock_wait_cycles":17939968,"thread_start_us":296653,"threads_started":2,"wal-append.queue_time_us":64606}
I20260709 12:15:12.707072 20381 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:12.788926 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53674.9us
W20260709 12:15:12.805949 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 129499us
W20260709 12:15:12.847939 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73568.1us
I20260709 12:15:12.849895 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:12.851056 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:12.848713 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.853596 12642 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:12.758927 20774 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:12.848806 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:12.855567 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.856686 12640 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:12.862083 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.867238 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:12.847955 20761 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:12.893656 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:12.867105 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.893923 20791 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:12.894428 20791 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:12.895001 20791 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:12.894443 12640 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:12.848146 20784 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:12.897764 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3320) took 1940 ms. Trace:
I20260709 12:15:12.786737 20690 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:12.899441 20690 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:12.899952 20690 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:12.901003 20690 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.901746 20381 raft_consensus.cc:2764] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:15:12.902509 20401 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:12.903916 20401 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:12.904770 20401 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:12.909308 12570 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:15:12.898218 12640 rpcz_store.cc:276] 0709 12:15:10.958256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.863726 (+1905470us) service_pool.cc:224] Handling call
0709 12:15:12.897691 (+ 33965us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.902061 20758 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:12.914240 20198 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.914934 20473 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.914798 20401 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.915843 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.918998 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:12.920229 20401 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:12.921005 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.790967 20538 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:12.922178 20538 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:12.923228 20538 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:12.917433 20473 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.925609 20431 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:12.926448 20431 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:12.926761 20431 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:12.927100 20431 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:12.924037 12642 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:12.855024 20444 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:12.928902 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.929463 20444 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:12.929978 12642 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 19.
I20260709 12:15:12.930519 20444 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:12.931238 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.932367 12642 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:12.918183 13262 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:15:12.933588 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.934516 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:12.935600 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.936633 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:15:12.937477 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70325.9us
I20260709 12:15:12.938025 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.939706 12642 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:12.954891 12642 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:12.956439 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.866117 20662 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:12.959262 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.898182 20784 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:12.934717 20442 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.576s	user 0.001s	sys 0.018s
I20260709 12:15:12.962194 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
W20260709 12:15:12.963874 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3318) took 2006 ms (client timeout 1925 ms). Trace:
I20260709 12:15:12.967619 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:12.972298 12637 rpcz_store.cc:269] 0709 12:15:10.957274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.444584 (+487310us) service_pool.cc:224] Handling call
0709 12:15:12.963530 (+1518946us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.963781 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2078146432}
I20260709 12:15:12.974022 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.975287 12637 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:15:12.976521 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2865) took 2070 ms. Trace:
I20260709 12:15:12.977044 12641 rpcz_store.cc:276] 0709 12:15:10.905619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.764751 (+1859132us) service_pool.cc:224] Handling call
0709 12:15:12.976435 (+211684us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:12.978684 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3324) took 1803 ms (client timeout 1956 ms). Trace:
W20260709 12:15:12.984553 12645 rpcz_store.cc:269] 0709 12:15:11.162853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.019773 (+856920us) service_pool.cc:224] Handling call
0709 12:15:12.965402 (+945629us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:12.965684 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":421163776}
W20260709 12:15:12.978294 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 297546us
I20260709 12:15:12.986255 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:12.986908 12645 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:12.983142 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.991465 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:12.992537 12637 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:12.960474 20662 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:12.995760 12645 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:12.999451 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2864) took 2094 ms. Trace:
I20260709 12:15:12.999852 12645 rpcz_store.cc:276] 0709 12:15:10.905357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.995192 (+2089835us) service_pool.cc:224] Handling call
0709 12:15:12.995550 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:12.999116 (+  3566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:12.999331 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:13.001210 12645 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:15:12.996333 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2872) took 2069 ms. Trace:
I20260709 12:15:13.004290 12637 rpcz_store.cc:276] 0709 12:15:10.927336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.991137 (+2063801us) service_pool.cc:224] Handling call
0709 12:15:12.996238 (+  5101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:12.959015 20401 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:15:13.001866 20780 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:12.993217 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:13.049050 12640 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:13.049244 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3325) took 1843 ms. Trace:
W20260709 12:15:13.052425 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 190401us
I20260709 12:15:13.063102 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:13.052699 12645 rpcz_store.cc:276] 0709 12:15:11.163183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.000689 (+1837506us) service_pool.cc:224] Handling call
0709 12:15:13.001036 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:13.001556 (+   520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.006492 (+  4936us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:13.000861 20662 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:13.063484 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:12.990566 20473 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:13.041906 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 387507us
I20260709 12:15:13.074851 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:12.997288 20356 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:15:12.981400 20442 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.080435 12491 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:13.048645 20780 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:12.994549 20761 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20260709 12:15:13.083741 20780 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:13.093880 20792 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:15:13.095954 20784 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:13.106993 20780 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.049064 20657 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:13.111552 20657 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:13.112886 20780 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:13.117745 20479 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:13.130463 20473 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.143018 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:13.072345 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:13.147748 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75501.3us
W20260709 12:15:13.166401 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 212107us
W20260709 12:15:13.211867 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137145us
I20260709 12:15:13.161708 20805 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:13.223399 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:13.213290 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:13.238050 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:13.224347 12492 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:13.246845 12640 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:13.247885 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3335) took 1818 ms. Trace:
I20260709 12:15:13.248353 12640 rpcz_store.cc:276] 0709 12:15:11.429964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.942902 (+1512938us) service_pool.cc:224] Handling call
0709 12:15:13.247779 (+304877us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:13.173619 20473 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:13.149020 20689 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.290076 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:13.301185 12492 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:13.278119 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 291114us
I20260709 12:15:13.304419 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:13.219833 20805 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:13.303568 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:13.316762 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:13.300432 20806 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:13.264695 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43668.6us
I20260709 12:15:13.215234 20784 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:13.391803 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:13.267835 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:13.350862 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:13.262923 20752 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:13.453928 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:13.291544 20808 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:13.455363 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3329) took 2282 ms. Trace:
W20260709 12:15:13.471853 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 204152us
I20260709 12:15:13.316553 20669 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:15:13.464994 20751 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.524s	user 0.008s	sys 0.009s
I20260709 12:15:13.458629 20780 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:13.310861 20805 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:13.477576 20657 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:15:13.475875 20748 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.261s	user 0.000s	sys 0.002s
I20260709 12:15:13.494668 20780 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.501389 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3332) took 2205 ms. Trace:
I20260709 12:15:13.501955 12643 rpcz_store.cc:276] 0709 12:15:11.296105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.500424 (+2204319us) service_pool.cc:224] Handling call
0709 12:15:13.500799 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.501019 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.501049 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:13.501091 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:13.501132 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.501186 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.501318 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:13.503961 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1044) took 1211 ms. Trace:
I20260709 12:15:13.514003 20780 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:13.521124 20660 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:13.469946 12645 rpcz_store.cc:276] 0709 12:15:11.172739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.064464 (+1891725us) service_pool.cc:224] Handling call
0709 12:15:13.076676 (+ 12212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.076958 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:15:13.290629 (+213671us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.454824 (+164195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:13.454908 (+    84us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:13.454989 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.455105 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.455283 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21745,"raft.queue_time_us":43730,"thread_start_us":328845,"threads_started":2}
I20260709 12:15:13.219481 20690 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:15:13.548256 12492 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:13.475944 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:13.483655 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:13.474663 20431 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.547s	user 0.019s	sys 0.005s
W20260709 12:15:13.504354 20804 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260709 12:15:13.471300 20784 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.554663 20799 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:13.555223 20799 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:13.555634 20799 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:13.556787 20784 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:13.506667 20807 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:15:13.508164 20805 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:13.508795 20741 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.874s	user 0.009s	sys 0.000s
I20260709 12:15:13.406153 20806 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.579962 20805 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:13.580108 20806 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:13.581386 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2836) took 2820 ms. Trace:
I20260709 12:15:13.468791 20808 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:13.581941 12494 rpcz_store.cc:276] 0709 12:15:10.760834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.789897 (+ 29063us) service_pool.cc:224] Handling call
0709 12:15:10.884747 (+ 94850us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:10.887119 (+  2372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:10.887147 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:10.887824 (+   677us) log.cc:844] Serialized 26 byte log entry
0709 12:15:11.544157 (+656333us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:11.544218 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:11.544315 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:13.508098 (+1963783us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:13.581000 (+ 72902us) raft_consensus.cc:1685] finished
0709 12:15:13.581032 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.581300 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74074880,"thread_start_us":622054,"threads_started":1,"wal-append.queue_time_us":737477}
I20260709 12:15:13.523051 20689 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:13.524261 20786 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:13.493923 20752 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:15:13.510119 20657 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:13.529330 20791 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.634s	user 0.012s	sys 0.000s
I20260709 12:15:13.530095 20800 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:13.532143 20442 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:13.534335 20798 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
W20260709 12:15:13.343696 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:13.510543 12643 rpcz_store.cc:276] 0709 12:15:12.293048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.503085 (+1210037us) service_pool.cc:224] Handling call
0709 12:15:13.503408 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.503606 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.503631 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:13.503674 (+    43us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:13.503713 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.503765 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.503895 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:13.540205 12461 connection.cc:256] RPC call timeout handler was delayed by -1.9060297080000055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20260709 12:15:13.543685 13261 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:13.603579 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 415848us
W20260709 12:15:13.606289 12461 connection.cc:256] RPC call timeout handler was delayed by -1.7641128940001636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
W20260709 12:15:13.457787 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113692us
I20260709 12:15:13.456854 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:13.609970 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3487) took 1551 ms. Trace:
W20260709 12:15:13.550572 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67025us
I20260709 12:15:13.610383 12640 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:13.611456 12536 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:13.614197 12536 leader_election.cc:403] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:13.610431 12566 rpcz_store.cc:276] 0709 12:15:12.058210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.060572 (+  2362us) service_pool.cc:224] Handling call
0709 12:15:12.060931 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:12.710268 (+649337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.609717 (+899449us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:13.609910 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1491429248}
I20260709 12:15:13.551841 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:13.551729 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75908.3us
I20260709 12:15:13.557829 20766 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:13.558399 20538 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:13.559126 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3273) took 4537 ms. Trace:
I20260709 12:15:13.560899 20662 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.562326 20669 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:13.557229 20494 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:13.567440 20777 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.298s	user 0.010s	sys 0.000s
I20260709 12:15:13.568394 12645 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:13.542594 20660 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:13.573894 20431 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.703649 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:13.617302 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:13.707366 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90201.9us
I20260709 12:15:13.708266 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:13.626566 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:13.640281 12461 connection.cc:256] RPC call timeout handler was delayed by -1.6370832000000064s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
W20260709 12:15:13.709743 12461 connection.cc:256] RPC call timeout handler was delayed by -1.5884831999999278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260709 12:15:13.710090 12461 connection.cc:256] RPC call timeout handler was delayed by -1.5092832000000271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:15:13.710352 12461 connection.cc:256] RPC call timeout handler was delayed by -1.4957832000000053s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20320000000000002s for a response
W20260709 12:15:13.710659 12461 connection.cc:256] RPC call timeout handler was delayed by -1.4066832000000886s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21309999999999998s for a response
W20260709 12:15:13.710915 12461 connection.cc:256] RPC call timeout handler was delayed by -1.3850832000000537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
W20260709 12:15:13.711261 12461 connection.cc:256] RPC call timeout handler was delayed by -1.3501546340000914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1562s for a response
W20260709 12:15:13.711542 12461 connection.cc:256] RPC call timeout handler was delayed by -1.3193831999999475s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
W20260709 12:15:13.711798 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0234546340000179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1925s for a response
I20260709 12:15:13.713084 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:13.625550 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:13.678354 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:13.682355 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130611us
I20260709 12:15:13.594259 20808 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:13.600809 20657 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:13.703917 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1045) took 1403 ms. Trace:
I20260709 12:15:13.724243 12645 rpcz_store.cc:276] 0709 12:15:12.300354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.543070 (+1242716us) service_pool.cc:224] Handling call
0709 12:15:13.568188 (+ 25118us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.698740 (+130552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.703824 (+  5084us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:13.621106 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:13.727876 20780 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:13.728595 20678 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:13.729251 20678 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:13.728621 20780 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.730692 20431 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:13.732928 20780 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:13.727068 20660 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.734259 20660 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:13.735651 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2841) took 2949 ms. Trace:
I20260709 12:15:13.736140 12489 rpcz_store.cc:276] 0709 12:15:10.786428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.050264 (+263836us) service_pool.cc:224] Handling call
0709 12:15:11.050610 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:12.030060 (+979450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.735372 (+1705312us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:13.735579 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2221068672}
I20260709 12:15:13.691870 20582 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:13.733430 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:13.628224 12644 rpcz_store.cc:276] 0709 12:15:09.022004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:09.022218 (+   214us) service_pool.cc:224] Handling call
0709 12:15:09.048640 (+ 26422us) raft_consensus.cc:1471] Updating replica for 5 ops
0709 12:15:09.094246 (+ 45606us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:09.563121 (+468875us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.014457 (+1451336us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b18000faba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:15:11.282569 (+268112us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b18000faba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf96c758f 00007ffbf2706571 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd
0709 12:15:11.366577 (+ 84008us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.846730 (+480153us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.882751 (+ 36021us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:11.889993 (+  7242us) log.cc:844] Serialized 52 byte log entry
0709 12:15:11.890986 (+   993us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:11.894775 (+  3789us) log.cc:844] Serialized 14 byte log entry
0709 12:15:11.973853 (+ 79078us) log.cc:844] Serialized 14 byte log entry
0709 12:15:12.147017 (+173164us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b18000faba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:12.166025 (+ 19008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:12.166141 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:13.225028 (+1058887us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:13.558809 (+333781us) raft_consensus.cc:1685] finished
0709 12:15:13.558836 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.559012 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":530311,"prepare.queue_time_us":936705,"prepare.run_cpu_time_us":287,"prepare.run_wall_time_us":8449,"raft.queue_time_us":999051,"raft.run_cpu_time_us":1087,"raft.run_wall_time_us":69119,"spinlock_wait_cycles":53149696,"thread_start_us":1322140,"threads_started":6,"wal-append.queue_time_us":844656,"wal-append.run_cpu_time_us":1275,"wal-append.run_wall_time_us":1005517}
I20260709 12:15:13.738602 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:13.739869 12645 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:15:13.740065 20758 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:13.696238 20766 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:13.740674 20758 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.740696 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2894) took 1094 ms. Trace:
I20260709 12:15:13.742414 12645 rpcz_store.cc:276] 0709 12:15:12.646492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.739069 (+1092577us) service_pool.cc:224] Handling call
0709 12:15:13.739538 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:13.740383 (+   845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.740629 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11648}
W20260709 12:15:13.743162 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 122162us
I20260709 12:15:13.722646 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3470) took 1316 ms. Trace:
I20260709 12:15:13.746546 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:13.746769 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:13.744726 12567 rpcz_store.cc:276] 0709 12:15:12.406281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.406484 (+   203us) service_pool.cc:224] Handling call
0709 12:15:12.609484 (+203000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:12.610635 (+  1151us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:12.610663 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:12.622248 (+ 11585us) log.cc:844] Serialized 26 byte log entry
0709 12:15:13.011138 (+388890us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:13.011197 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.011291 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:13.722138 (+710847us) raft_consensus.cc:1685] finished
0709 12:15:13.722314 (+   176us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.722559 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":747,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":19481,"thread_start_us":339961,"threads_started":1,"wal-append.queue_time_us":589005}
I20260709 12:15:13.696297 20713 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:13.744020 20690 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:13.604892 20791 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.697778 20805 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:13.604660 20800 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.747793 20678 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:13.784883 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2888) took 2839 ms (client timeout 2249 ms). Trace:
W20260709 12:15:13.785905 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 2884) took 2841 ms (client timeout 1984 ms). Trace:
W20260709 12:15:13.785393 12638 rpcz_store.cc:269] 0709 12:15:10.945435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.116750 (+1171315us) service_pool.cc:224] Handling call
0709 12:15:13.784629 (+1667879us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:13.784786 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121264000}
W20260709 12:15:13.786464 12646 rpcz_store.cc:269] 0709 12:15:10.944409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.545775 (+601366us) service_pool.cc:224] Handling call
0709 12:15:13.785548 (+2239773us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:13.785825 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582226688}
I20260709 12:15:13.799851 12638 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:13.741456 20766 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.777840 20690 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:13.798880 20779 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:13.825512 20779 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:13.826118 20779 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:13.828339 20668 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:13.799058 20442 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:13.830066 20442 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:13.830758 20442 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:13.831350 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3330) took 2658 ms. Trace:
I20260709 12:15:13.803300 12638 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:15:13.831802 12642 rpcz_store.cc:276] 0709 12:15:11.172991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.066201 (+1893210us) service_pool.cc:224] Handling call
0709 12:15:13.066918 (+   717us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:13.119684 (+ 52766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.119733 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:13.120429 (+   696us) log.cc:844] Serialized 52 byte log entry
0709 12:15:13.592244 (+471815us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:13.696420 (+104176us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.739524 (+ 43104us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.739819 (+   295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.739915 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:13.831106 (+ 91191us) raft_consensus.cc:1685] finished
0709 12:15:13.831128 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.831285 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75491,"prepare.queue_time_us":152843,"prepare.run_cpu_time_us":149,"prepare.run_wall_time_us":25695,"raft.queue_time_us":80833,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":20430,"thread_start_us":471312,"threads_started":1,"wal-append.queue_time_us":702405}
I20260709 12:15:13.813755 20799 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:13.815716 20662 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:13.798368 20805 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:13.836212 20799 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:13.837044 20690 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:13.836807 20805 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:13.848001 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.014s	sys 0.020s
I20260709 12:15:13.821688 20473 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:13.783635 20713 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:13.878361 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1067) took 2481 ms (client timeout 2103 ms). Trace:
W20260709 12:15:13.913396 12495 rpcz_store.cc:269] 0709 12:15:11.397094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.836224 (+439130us) service_pool.cc:224] Handling call
0709 12:15:13.877999 (+2041775us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:13.878237 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":983697664}
I20260709 12:15:13.895746 20657 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.893368 20678 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.905489 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3327) took 2741 ms. Trace:
I20260709 12:15:13.948369 12641 rpcz_store.cc:276] 0709 12:15:11.163750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.015663 (+1851913us) service_pool.cc:224] Handling call
0709 12:15:13.069055 (+ 53392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.069310 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:13.344059 (+274749us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.781531 (+437472us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.802698 (+ 21167us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.839957 (+ 37259us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.874201 (+ 34244us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.878407 (+  4206us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.878872 (+   465us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.904909 (+ 26037us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:13.904996 (+    87us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:13.905094 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.905198 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.905395 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":129299,"prepare.queue_time_us":447424,"prepare.run_cpu_time_us":173,"prepare.run_wall_time_us":7404,"raft.queue_time_us":167819,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":61389,"spinlock_wait_cycles":26218112,"thread_start_us":564958,"threads_started":2,"wal-append.queue_time_us":498816}
W20260709 12:15:13.959226 20736 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:15:13.911092 20689 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:13.968142 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:13.969452 12489 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:13.912082 12644 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:15:13.823695 20669 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.925921 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:13.970552 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44723.9us
I20260709 12:15:13.971372 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:13.927011 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1065) took 2530 ms. Trace:
I20260709 12:15:13.928385 12642 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:15:13.929958 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:13.934926 12387 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 2 to 19, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 19 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:13.936393 20779 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:13.910987 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.014s	sys 0.021s
I20260709 12:15:13.972461 12491 rpcz_store.cc:276] 0709 12:15:11.396529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.216238 (+819709us) service_pool.cc:224] Handling call
0709 12:15:12.216714 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.080936 (+864222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.926744 (+845808us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:13.926946 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1985828096}
I20260709 12:15:13.982000 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:13.984432 12490 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:13.983070 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1054) took 1602 ms. Trace:
I20260709 12:15:13.985883 12644 rpcz_store.cc:276] 0709 12:15:12.380602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.739055 (+1358453us) service_pool.cc:224] Handling call
0709 12:15:13.907758 (+168703us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:13.908007 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.969971 (+ 61964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.982988 (+ 13017us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3656704}
I20260709 12:15:13.988466 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3326) took 2824 ms. Trace:
I20260709 12:15:13.988790 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:13.988950 12637 rpcz_store.cc:276] 0709 12:15:11.163474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.007251 (+1843777us) service_pool.cc:224] Handling call
0709 12:15:13.008321 (+  1070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.008635 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:15:13.343500 (+334865us) log.cc:844] Serialized 14 byte log entry
0709 12:15:13.987987 (+644487us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:13.988045 (+    58us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:13.988090 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:13.988159 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:13.988272 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":430239,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":3703,"thread_start_us":890891,"threads_started":2}
I20260709 12:15:13.992704 20442 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:13.989931 12495 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:13.998580 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 226) took 92772 ms (client timeout 15000 ms). Trace:
I20260709 12:15:13.999266 20678 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.001724 20657 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.008792 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.009699 12491 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:13.984488 20198 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:15:14.009629 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2845) took 3124 ms (client timeout 2075 ms). Trace:
W20260709 12:15:14.029999 12488 rpcz_store.cc:269] 0709 12:15:10.885427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:10.992471 (+107044us) service_pool.cc:224] Handling call
0709 12:15:14.009415 (+3016944us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:14.009550 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2953097600}
I20260709 12:15:14.033581 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:14.038522 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 321264us
I20260709 12:15:14.040046 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.040975 12494 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:14.041556 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 325641us
I20260709 12:15:14.045943 13259 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:15:13.985481 20689 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.050004 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.054780 12571 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:14.049340 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:14.055678 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 430194us
I20260709 12:15:14.028728 20653 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:14.028769 20198 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:14.050952 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:14.056747 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.050956 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1051) took 1689 ms. Trace:
I20260709 12:15:14.052501 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.082405 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:14.082441 12571 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:14.088879 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:14.091902 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.092046 12566 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:15:14.092218 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.093226 12563 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:14.079252 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:14.081796 12643 rpcz_store.cc:276] 0709 12:15:12.361938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.605496 (+1243558us) service_pool.cc:224] Handling call
0709 12:15:13.605906 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:13.606186 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:13.696875 (+ 90689us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.050613 (+353738us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:14.050669 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:14.050723 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.050793 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:14.050900 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90349,"prepare.queue_time_us":94205,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":84,"spinlock_wait_cycles":31265536,"thread_start_us":344785,"threads_started":1}
I20260709 12:15:14.093024 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:14.103071 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.105263 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.113692 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.117812 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.118855 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 18.
I20260709 12:15:14.120237 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:14.122285 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 414055us
I20260709 12:15:14.124778 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:14.125833 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:14.137300 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.138531 12571 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:14.121672 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:14.141297 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.148770 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:14.153126 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:14.154436 12637 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 24 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:15:14.154525 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:14.175211 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:14.175475 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.176287 12568 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:14.178030 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.149905 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:14.034235 20786 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:15:14.189987 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.191875 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.232343 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.237246 12564 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:14.171437 20766 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:14.240350 20766 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:14.193706 12494 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [23.16-25.17]
I20260709 12:15:14.242861 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.096695 20198 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:14.212358 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:14.216231 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:14.236754 12490 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:14.252934 12566 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:14.253141 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:14.057170 20473 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.120s
I20260709 12:15:14.255214 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:14.136204 20653 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:14.265425 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:14.134056 20443 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:14.261191 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.280081 12566 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:15:14.266877 13255 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:14.115927 20791 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.114789 20437 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:14.267335 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4710134479998942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
I20260709 12:15:14.115273 20662 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:14.079037 20784 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.166s
I20260709 12:15:14.241526 20766 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:14.271592 12643 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:14.286731 12570 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:14.134773 20800 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.256186 20786 consensus_queue.cc:1243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:15:14.292920 20443 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.231406 20799 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:14.316524 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.014s	sys 0.050s
W20260709 12:15:14.338866 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.014s	sys 0.051s
I20260709 12:15:14.292488 12566 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:14.321770 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.984s (SENT)
I20260709 12:15:14.340325 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:14.342065 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.327143 12564 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:14.327422 12490 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:15:14.344213 12488 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:14.331967 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:14.339602 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 592914us
I20260709 12:15:14.287561 20472 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:14.290230 20827 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:14.002673 12392 rpcz_store.cc:269] 0709 12:13:41.226073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:41.273722 (+ 47649us) service_pool.cc:224] Handling call
0709 12:15:13.998307 (+92724585us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:15:04.091001 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:15:04.266721 (+175720us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:15:04.285779 (+ 19058us) write_op.cc:435] Acquired schema lock
0709 12:15:04.289718 (+  3939us) tablet.cc:662] Decoding operations
0709 12:15:04.889288 (+599570us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:15:04.899067 (+  9779us) write_op.cc:641] Partition lock acquired for write op
0709 12:15:04.901753 (+  2686us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:15:05.074201 (+172448us) tablet.cc:701] Row locks acquired
0709 12:15:05.074255 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:15:05.074454 (+   199us) write_op.cc:270] Start()
0709 12:15:05.074637 (+   183us) write_op.cc:276] Timestamp: P: 1783599305074378 usec, L: 0
0709 12:15:05.074650 (+    13us) op_driver.cc:348] REPLICATION: starting
0709 12:15:05.086041 (+ 11391us) log.cc:844] Serialized 3636 byte log entry
0709 12:15:05.783282 (+697241us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:06.237874 (+454592us) op_driver.cc:464] REPLICATION: finished
0709 12:15:06.654629 (+416755us) write_op.cc:301] APPLY: starting
0709 12:15:06.798045 (+143416us) tablet.cc:1367] starting BulkCheckPresence
0709 12:15:07.219439 (+421394us) tablet.cc:1370] finished BulkCheckPresence
0709 12:15:07.219503 (+    64us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:15:07.226515 (+  7012us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:15:07.226929 (+   414us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:15:07.226980 (+    51us) write_op.cc:312] APPLY: finished
0709 12:15:07.227456 (+   476us) log.cc:844] Serialized 89 byte log entry
0709 12:15:07.336426 (+108970us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:07.337003 (+   577us) write_op.cc:454] Released schema lock
0709 12:15:07.337177 (+   174us) write_op.cc:341] FINISH: Updating metrics
0709 12:15:07.354412 (+ 17235us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:07.354467 (+    55us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6267546,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":409584,"apply.run_cpu_time_us":35138,"apply.run_wall_time_us":699471,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41711,"num_ops":9,"prepare.queue_time_us":254575,"prepare.run_cpu_time_us":28812,"prepare.run_wall_time_us":1149431,"raft.queue_time_us":251475,"raft.run_cpu_time_us":67236,"raft.run_wall_time_us":429994,"replication_time_us":1094969,"spinlock_wait_cycles":24922368,"thread_start_us":564596,"threads_started":5,"wal-append.queue_time_us":289766,"wal-append.run_cpu_time_us":22370,"wal-append.run_wall_time_us":2859983}]]}
I20260709 12:15:14.305883 20409 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:15:14.347690 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.045s	sys 0.043s
I20260709 12:15:14.345847 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:14.348044 12570 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:14.349277 12494 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:15:14.321058 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:14.325426 20713 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:14.360131 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.046s	sys 0.044s
W20260709 12:15:14.359784 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 970) took 3078 ms (client timeout 1967 ms). Trace:
I20260709 12:15:14.362789 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.368278 12492 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:15:14.372750 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:14.361176 13255 rpcz_store.cc:269] 0709 12:15:11.281445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:11.281752 (+   307us) service_pool.cc:224] Handling call
0709 12:15:14.359542 (+3077790us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:14.359693 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6046862080}
W20260709 12:15:14.374387 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52658.2us
I20260709 12:15:14.374869 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.378929 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.375713 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:14.379747 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.075s (SENT) [suppressed 37 similar messages]
I20260709 12:15:14.380157 13260 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:14.381882 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:14.354383 20822 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:15:14.114638 12387 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 10 to 20. New cstate: current_term: 20 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:14.290874 20653 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.348941 20791 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:14.395572 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.395936 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:14.396616 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:14.343874 20784 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:14.366657 20545 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.004s	sys 0.042s
W20260709 12:15:14.425379 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:14.351792 20409 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:14.334787 20662 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.412582 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.414003 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:14.416311 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:14.417009 12638 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
W20260709 12:15:14.417796 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1341384639999887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
I20260709 12:15:14.419013 12642 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:15:14.432542 12637 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:14.365669 20768 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:14.423328 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.424562 12490 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:14.363196 20479 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:14.429188 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.433677 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:14.385118 20442 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.377005 20713 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:14.428187 20545 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.004s	sys 0.043s
I20260709 12:15:14.355520 20437 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:14.500072 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51132.7us
I20260709 12:15:14.357132 20799 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.466167 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.501272 12497 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:14.466773 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 753668us
I20260709 12:15:14.351868 20472 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.460146 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.502507 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:14.502969 12490 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:15:14.503986 12488 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:14.507562 12492 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:14.508417 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.389458 20805 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.070s
I20260709 12:15:14.356861 20827 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.461117 20829 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:14.550341 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.062s	sys 0.074s
I20260709 12:15:14.408360 20473 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.092s
I20260709 12:15:14.514847 12488 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26. Re-sending same reply.
W20260709 12:15:14.552119 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50697.2us
W20260709 12:15:14.407091 20469 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.014s	sys 0.019s
W20260709 12:15:14.552981 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50579us
W20260709 12:15:14.573176 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.065s	sys 0.074s
I20260709 12:15:14.573477 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:14.422672 20791 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:14.562858 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:14.465093 12387 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 19 to 21, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 21 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:14.590057 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:14.575536 12495 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:14.604413 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:14.520916 20768 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:14.615532 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.524348 20689 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:14.427889 20668 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.150s
I20260709 12:15:14.622745 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:14.641428 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:14.642395 12495 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 26
W20260709 12:15:14.600641 20843 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:15:14.438253 20784 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.634853 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:14.500455 20713 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:14.660580 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 280853us
I20260709 12:15:14.630182 20768 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:14.586371 20773 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:14.646082 20831 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:14.672068 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:14.686975 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
W20260709 12:15:14.689164 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115799us
I20260709 12:15:14.690546 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:14.652549 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.023s	sys 0.044s
I20260709 12:15:14.664515 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:14.704253 12570 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:14.465157 20409 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.626477 20786 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:14.705652 20786 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.705730 20409 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:14.705788 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:14.623525 20796 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:14.708887 20796 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:14.709962 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.023s	sys 0.044s
I20260709 12:15:14.709925 20796 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.710949 20832 raft_consensus.cc:2764] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:15:14.702823 20538 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:15:14.714931 20538 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:14.715536 20538 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:14.718181 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:14.716972 20538 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.720736 20538 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:14.728470 12493 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:14.651224 20778 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:14.713449 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:14.581641 20829 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.769430 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:14.770277 13261 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:14.673064 20768 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:14.671617 20442 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:15:14.761333 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:14.735013 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 128954us
W20260709 12:15:14.734545 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144582us
I20260709 12:15:14.789145 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:14.735144 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 181738us
I20260709 12:15:14.734611 20832 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:14.780855 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:14.613035 12497 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:14.673116 20479 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:14.786590 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:14.732607 20840 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.407s	user 0.002s	sys 0.000s
I20260709 12:15:14.766877 20443 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:14.772315 20786 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:14.802356 20855 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:15:14.802452 20833 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.223s	user 0.000s	sys 0.000s
I20260709 12:15:14.662302 20805 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.114s
I20260709 12:15:14.798137 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:14.813720 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:14.675673 20381 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:14.822000 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131554us
I20260709 12:15:14.842643 12493 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:14.888540 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:14.855369 12495 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
W20260709 12:15:14.889361 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93539.5us
I20260709 12:15:14.750123 20780 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:14.671082 20791 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.728639 20773 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:14.786161 20831 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:14.787110 20442 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:14.854496 20832 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:14.859997 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:14.859524 20689 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:15.001319 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:14.859134 20582 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:15:14.944418 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84501.6us
I20260709 12:15:14.868741 20786 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:14.907840 20834 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.271s	user 0.001s	sys 0.000s
I20260709 12:15:14.869570 20443 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:14.915253 20784 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.930487 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:14.914611 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:14.909107 20845 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
W20260709 12:15:14.908481 20848 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:15:14.930076 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:14.906998 20844 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.302s	user 0.000s	sys 0.000s
I20260709 12:15:15.063889 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:15.109740 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 323231us
W20260709 12:15:14.918955 20826 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.466s	user 0.000s	sys 0.001s
W20260709 12:15:14.915562 20808 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.165s	user 0.009s	sys 0.007s
W20260709 12:15:14.918504 20849 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
W20260709 12:15:14.964128 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:14.906426 20841 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.326s	user 0.000s	sys 0.001s
I20260709 12:15:14.906364 20872 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:14.913602 20869 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:14.900689 20662 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:14.919587 20805 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:14.920665 20859 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:15:15.103919 20690 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.266s	user 0.016s	sys 0.002s
I20260709 12:15:14.910671 20713 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:15.118835 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 204321us
I20260709 12:15:14.952682 20780 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:14.913187 20653 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.085958 20791 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:15.061524 20866 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:15.123878 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:14.908270 20775 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:15:15.126678 12492 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.002s	sys 0.013s
I20260709 12:15:14.914963 20799 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:14.911589 20827 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.001837 20442 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:15.128672 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:15.129309 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65550.4us
I20260709 12:15:14.891983 20838 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.224s
I20260709 12:15:15.010982 20870 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:14.910871 20829 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.115928 20855 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:14.931946 20381 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.125429 20831 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:14.853775 20479 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.184106 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:15.153910 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 347003us
I20260709 12:15:15.105274 20786 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.138321 20871 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:15.143386 20859 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:15.169734 12492 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.003s	sys 0.013s
W20260709 12:15:15.152530 20766 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.812s	user 0.014s	sys 0.000s
W20260709 12:15:15.174296 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50524.7us
W20260709 12:15:15.168674 20198 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.896s	user 0.009s	sys 0.009s
I20260709 12:15:15.218827 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.220844 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:15.181099 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52540.4us
I20260709 12:15:15.178326 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3473) took 2755 ms. Trace:
I20260709 12:15:15.212993 20871 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.182140 20796 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:15.222239 20871 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:15.222664 20796 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.223254 20796 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:15.221881 12569 rpcz_store.cc:276] 0709 12:15:12.423035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.482476 (+ 59441us) service_pool.cc:224] Handling call
0709 12:15:13.610889 (+1128413us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:13.611115 (+   226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:13.644410 (+ 33295us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.644460 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:13.645181 (+   721us) log.cc:844] Serialized 26 byte log entry
0709 12:15:14.049146 (+403965us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:14.049193 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.049296 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:15.173031 (+1123735us) raft_consensus.cc:1685] finished
0709 12:15:15.173056 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.178233 (+  5177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":69397,"spinlock_wait_cycles":282879360,"thread_start_us":357134,"threads_started":1,"wal-append.queue_time_us":672494}
I20260709 12:15:15.227396 20773 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.191663 20381 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:15.180245 20870 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.144717 20872 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.199184 13261 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:15:15.201821 13255 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:15.150341 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:15.150106 20805 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.208209 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:15.169531 20775 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:15.152501 20690 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.210012 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.210292 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:15.210807 20868 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:15:15.254626 12563 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:15.175194 20811 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260709 12:15:15.066659 12387 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 13 to 22, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:15.222327 20657 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:15:15.238922 20834 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:15:15.220956 20758 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:15.211459 20653 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:15.240170 20678 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:15.230142 20870 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:15.221963 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:15.222376 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3471) took 2815 ms. Trace:
W20260709 12:15:15.221951 20538 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.006s
I20260709 12:15:15.246906 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:15.211123 13262 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:15.138145 20808 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.209498 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:15.144474 20832 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:15.269344 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59955.3us
I20260709 12:15:15.263091 12572 rpcz_store.cc:276] 0709 12:15:12.407444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:12.407605 (+   161us) service_pool.cc:224] Handling call
0709 12:15:13.909650 (+1502045us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:13.909894 (+   244us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:13.910343 (+   449us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:15:13.938100 (+ 27757us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.386318 (+448218us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:14.391395 (+  5077us) log.cc:844] Serialized 26 byte log entry
0709 12:15:14.391606 (+   211us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:15:14.391656 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.391751 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:15.222036 (+830285us) raft_consensus.cc:1685] finished
0709 12:15:15.222138 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.222303 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37082,"prepare.queue_time_us":28401,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":58761,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":23677,"spinlock_wait_cycles":455289728,"thread_start_us":410476,"threads_started":1,"wal-append.queue_time_us":648306}
I20260709 12:15:15.273877 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:15.275010 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:15.275354 20808 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.118358 20689 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.261323 13254 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:15.173342 20866 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.262107 20653 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.252210 20800 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:15:15.283157 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133535us
I20260709 12:15:15.278595 20829 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:15.258236 20657 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.230585 20766 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.277901 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:15.281769 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2895) took 2635 ms. Trace:
I20260709 12:15:15.312593 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:15.278529 20758 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:15.280403 20381 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:15:15.277654 20472 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:15.284826 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:15.323935 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:15.285810 20538 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:15:15.345322 20713 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:15.346078 20713 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.346369 20538 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:15.283042 20775 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.347184 20538 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:15.347705 20775 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:15.348539 20713 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:15.286834 12387 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 18. New cstate: current_term: 18 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:15.346622 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2896) took 2699 ms. Trace:
I20260709 12:15:15.350970 12387 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 8 to 19, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 19 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:15.235036 20198 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.280875 20886 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:15.282938 20653 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:15.287626 20838 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:15.291132 20479 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.286621 20799 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
W20260709 12:15:15.297524 20836 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.658s	user 0.000s	sys 0.004s
I20260709 12:15:15.287808 20800 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.298141 20657 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:15.406518 20800 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:15.406814 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:15.238294 20859 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.271597 20863 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:15.299060 20779 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:15.299821 20839 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:15.404196 20870 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.409732 20863 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:15.410930 20863 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.411999 20870 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:15.300524 20866 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.306442 12566 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:15.414484 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.308483 12571 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:15.415146 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2897) took 2767 ms. Trace:
I20260709 12:15:15.276142 20808 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:15.415970 20808 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:15.416524 20808 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:15.231429 20881 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:15:15.261430 20786 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.284742 20829 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.321523 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:15.325484 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47669.1us
I20260709 12:15:15.432300 12568 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:15.327107 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.321556 20869 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:15:15.330047 20431 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:15.322154 20784 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:15:15.331116 20884 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:15:15.331895 20690 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:15.333379 20872 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.336861 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3502) took 1205 ms. Trace:
I20260709 12:15:15.458597 12567 rpcz_store.cc:276] 0709 12:15:14.131782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.180988 (+ 49206us) service_pool.cc:224] Handling call
0709 12:15:14.181394 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:14.181731 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:14.181770 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:14.270313 (+ 88543us) log.cc:844] Serialized 26 byte log entry
0709 12:15:14.826430 (+556117us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:14.826476 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.826588 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:15.336615 (+510027us) raft_consensus.cc:1685] finished
0709 12:15:15.336639 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.336790 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":499893,"threads_started":1,"wal-append.queue_time_us":862236}
W20260709 12:15:15.340161 20778 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.562s	user 0.006s	sys 0.005s
I20260709 12:15:15.461000 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.353235 20768 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.350235 12645 rpcz_store.cc:276] 0709 12:15:12.647048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.743550 (+1096502us) service_pool.cc:224] Handling call
0709 12:15:13.768405 (+ 24855us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:15:13.781021 (+ 12616us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.781061 (+    40us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:15:13.799427 (+ 18366us) log.cc:844] Serialized 78 byte log entry
0709 12:15:14.238604 (+439177us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:14.239081 (+   477us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.256359 (+ 17278us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.268784 (+ 12425us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.268883 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:15.304050 (+1035167us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:15.305234 (+  1184us) spinlock_profiling.cc:243] Waited 8.91 ms on lock 0x7b18004669c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:15.346332 (+ 41098us) raft_consensus.cc:1685] finished
0709 12:15:15.346356 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.346549 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":89549,"prepare.queue_time_us":414,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":7937,"raft.queue_time_us":53069,"raft.run_cpu_time_us":3584,"raft.run_wall_time_us":1066838,"spinlock_wait_cycles":39171712,"thread_start_us":403717,"threads_started":1,"wal-append.queue_time_us":574815}
I20260709 12:15:15.356742 20796 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.358505 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.262691 20831 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:15:15.358632 20381 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:15.359872 20757 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:15.360550 20811 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:15.360692 20689 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:15.361243 20876 consensus_queue.cc:1175] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 26 status { last_received { term: 26 index: 18 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.17-25.17]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:15:15.472201 20757 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:15.478343 20888 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:15:15.478019 20757 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.9, Last appended by leader: 9, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.479382 20757 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:15.364046 20827 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:15.481799 20827 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:15.482656 20827 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.483850 20827 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:15.367583 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3485) took 1250 ms. Trace:
I20260709 12:15:15.485298 12563 rpcz_store.cc:276] 0709 12:15:14.117299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.119462 (+  2163us) service_pool.cc:224] Handling call
0709 12:15:14.119846 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:15.257511 (+1137665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:15.367325 (+109814us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:15.367522 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2609220352}
I20260709 12:15:15.486521 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.370455 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2868) took 1001 ms. Trace:
I20260709 12:15:15.487519 12493 rpcz_store.cc:276] 0709 12:15:14.369575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.369773 (+   198us) service_pool.cc:224] Handling call
0709 12:15:15.370108 (+1000335us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:15.370364 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682009856}
I20260709 12:15:15.372082 20678 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:15.349660 20805 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:15.378396 20766 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:15.489071 20678 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:15.380401 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.381784 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.382666 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3498) took 1328 ms. Trace:
I20260709 12:15:15.490674 12565 rpcz_store.cc:276] 0709 12:15:14.054033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.054209 (+   176us) service_pool.cc:224] Handling call
0709 12:15:14.054613 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:14.054928 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:14.095056 (+ 40128us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.713767 (+618711us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.890712 (+176945us) log.cc:844] Serialized 14 byte log entry
0709 12:15:15.015786 (+125074us) log.cc:844] Serialized 14 byte log entry
0709 12:15:15.017543 (+  1757us) log.cc:844] Serialized 14 byte log entry
0709 12:15:15.306358 (+288815us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b18003aa200. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:15.381309 (+ 74951us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b18003aa200. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf53a9d70 00007ffbf539d293 00007ffbf5392cea 00007ffbf539323a 00007ffbf57fda2a 00007ffbf58065e5 00007ffbf5806568
0709 12:15:15.381777 (+   468us) log.cc:844] Serialized 14 byte log entry
0709 12:15:15.382116 (+   339us) log.cc:844] Serialized 14 byte log entry
0709 12:15:15.382258 (+   142us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:15.382297 (+    39us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:15.382352 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:15.382460 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.382599 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":295232,"prepare.queue_time_us":367033,"prepare.run_cpu_time_us":264,"prepare.run_wall_time_us":16397,"raft.queue_time_us":199144,"raft.run_cpu_time_us":794,"raft.run_wall_time_us":15713,"spinlock_wait_cycles":105692288,"thread_start_us":689036,"threads_started":2,"wal-append.queue_time_us":826461}
I20260709 12:15:15.383353 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.384018 12564 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:15:15.319869 20758 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.283942 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.392025 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.393256 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.392763 20442 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.499142 13254 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:15.467312 20431 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:15.396986 20198 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.501317 13259 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:15.502833 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 218940us
I20260709 12:15:15.504853 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:15.467965 20778 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:15.513770 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2871) took 1143 ms. Trace:
I20260709 12:15:15.409879 20839 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:15.311486 12639 rpcz_store.cc:276] 0709 12:15:12.646752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.739693 (+1092941us) service_pool.cc:224] Handling call
0709 12:15:13.740114 (+   421us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:13.740546 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.740575 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:13.741205 (+   630us) log.cc:844] Serialized 52 byte log entry
0709 12:15:14.044688 (+303483us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:14.052895 (+  8207us) log.cc:844] Serialized 14 byte log entry
0709 12:15:14.053644 (+   749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.053723 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:15.281506 (+1227783us) raft_consensus.cc:1685] finished
0709 12:15:15.281530 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.281706 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5707,"prepare.queue_time_us":2148,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":2411,"raft.queue_time_us":1470,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":43568,"thread_start_us":302389,"threads_started":1,"wal-append.queue_time_us":760975}
I20260709 12:15:15.516865 12572 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:15:15.415696 12640 rpcz_store.cc:276] 0709 12:15:12.647336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:13.750831 (+1103495us) service_pool.cc:224] Handling call
0709 12:15:13.776518 (+ 25687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:13.801820 (+ 25302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:13.801860 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:13.802591 (+   731us) log.cc:844] Serialized 26 byte log entry
0709 12:15:14.058319 (+255728us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:14.058382 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:14.058488 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:15.282682 (+1224194us) spinlock_profiling.cc:243] Waited 7.69 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:15.414786 (+132104us) raft_consensus.cc:1685] finished
0709 12:15:15.414812 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:15.415068 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50806,"raft.queue_time_us":16188,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8019,"spinlock_wait_cycles":17681792,"thread_start_us":253246,"threads_started":1,"wal-append.queue_time_us":712760}
I20260709 12:15:15.419305 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:15.432973 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.517925 12571 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:15.444402 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:15.448943 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121918us
I20260709 12:15:15.519243 12568 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:15:15.520112 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:15.340865 20780 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:15.466511 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:15.526703 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.527760 12567 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:15.442548 20829 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:15.529243 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.348433 20538 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:15.487212 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:15.493175 13255 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:15:15.493993 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 237313us
I20260709 12:15:15.460811 20784 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.394269 20886 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.385428 12387 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 12 to 20, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 20 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:15:15.432142 20819 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:15:15.444080 20473 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
W20260709 12:15:15.507884 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 617742us
I20260709 12:15:15.514475 12497 rpcz_store.cc:276] 0709 12:15:14.370813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.378512 (+  7699us) service_pool.cc:224] Handling call
0709 12:15:15.513486 (+1134974us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:15.513693 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279121792}
I20260709 12:15:15.415192 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:15.556377 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.543646 20780 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:15:15.560899 20538 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:15.531227 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:15.500547 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:15.562367 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61902.9us
I20260709 12:15:15.518564 20838 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:15.533677 12569 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:15.533743 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100848us
I20260709 12:15:15.535023 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.536036 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:15.538623 20444 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:15.538810 20805 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:15:15.539252 20713 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:15.539460 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:15.540126 13253 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:15.489006 20437 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.491989 20381 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:15.524016 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:15.540009 20881 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:15.607244 20881 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:15.502012 20758 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:15.608273 20881 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.574321 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:15.567842 20778 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:15.568104 20779 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:15.576401 20866 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:15.568960 20502 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:15.575744 12387 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 9 to 23. New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:15.577857 20689 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:15.540328 20886 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:15.574040 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:15.563002 20832 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:15:15.570849 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:15.587191 20784 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.590819 20768 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:15.592500 20766 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:15.601969 20839 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.539538 20431 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:15:15.606707 20381 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.613363 20778 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:15.613536 20779 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.628461 20806 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:15.570842 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:15.610304 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86464.7us
I20260709 12:15:15.616302 12639 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:15:15.579545 20657 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.621683 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:15.622753 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51968.7us
I20260709 12:15:15.636179 12569 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
W20260709 12:15:15.628240 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55943us
W20260709 12:15:15.627264 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54200.3us
I20260709 12:15:15.639346 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:15.640144 12637 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23. Re-sending same reply.
I20260709 12:15:15.544603 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:15.644131 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 99626.4us
I20260709 12:15:15.631387 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:15.653044 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82280.4us
I20260709 12:15:15.655023 12638 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:15.658453 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.659507 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:15.660216 12642 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:15.614508 20437 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:15.624656 20431 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:15.614960 12387 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 10 to 18. New cstate: current_term: 18 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:15.673489 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:15.705286 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:15.614118 20502 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.615757 20662 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:15.748484 12570 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:15:15.748477 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:15.623837 20766 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:15.618122 20713 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:15.758781 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:15.672375 12571 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:15.759392 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:15.801698 12641 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:15.702515 20855 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:15.614867 20442 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.629341 20779 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:15.622856 20198 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:15.620497 20832 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:15.627370 20866 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.169s
I20260709 12:15:15.614512 20689 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:15.833688 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:15.629958 20871 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.672643 20690 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.709837 20767 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.116s
I20260709 12:15:15.812105 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:15.624154 20839 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:15.673898 20431 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:15.815639 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:15.674491 20780 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:15.838790 20832 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:15.781730 20502 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:15.700850 20669 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.124s
I20260709 12:15:15.747709 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:15.802960 20828 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:15.796375 20766 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:15.798520 20713 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.848905 12639 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:15.813905 20791 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:15.673061 20437 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:15.673168 20778 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.837033 20198 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.840341 20784 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:15.843819 20653 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.781553 20773 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
W20260709 12:15:15.861312 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63599.4us
W20260709 12:15:15.849649 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173585us
I20260709 12:15:15.780609 20662 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.860921 20780 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:15.873869 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 169252us
I20260709 12:15:15.856221 20872 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:15:15.890868 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.892330 12489 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:15:15.871133 20899 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:15.894018 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.894858 12493 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:15.882863 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.857849 20690 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:15.857547 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 226233us
I20260709 12:15:15.896739 12571 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:15.874698 20198 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:15.900525 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:15.866827 20713 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:15.893324 20662 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:15.874169 20784 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:15.909199 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.909812 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.910492 12497 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:15.900411 20775 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.003s
I20260709 12:15:15.872016 20479 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:15:15.911017 12489 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:15:15.866632 20828 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:15.894315 20780 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:15.895310 20899 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:15.929064 12641 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:15.931442 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:15.931934 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:15.932109 12494 raft_consensus.cc:2376] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27. Re-sending same reply.
I20260709 12:15:15.934913 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.935783 12571 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:15.936523 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.936712 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.937359 12488 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:15.929936 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:15.939872 12489 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:15.942539 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.942847 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:15.916260 20626 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:15.951557 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.952531 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:15.952864 12640 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:15.918496 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:15.927287 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:15.929039 12646 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:15:15.943512 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.908215 20872 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:15.943514 12491 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:15.944347 12494 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:15.985973 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:15.918247 20545 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.903888 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:15.955170 12642 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:15.916477 20806 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:15:15.986923 12641 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:15.945549 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:15.987622 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.988325 12644 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:15:15.988652 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.907505 20805 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.039s
I20260709 12:15:15.991722 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:15.951813 20808 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.013s	sys 0.000s
I20260709 12:15:15.915455 20784 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:16.003062 12495 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:16.003459 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 345068us
I20260709 12:15:16.022514 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:16.011404 12571 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:15.959241 20775 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:16.023837 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:16.024638 12571 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:16.023808 12564 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:15.900231 20832 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:16.033267 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106372us
I20260709 12:15:15.901003 20431 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:15.960012 20657 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:15.866851 12387 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 14 to 24. New cstate: current_term: 24 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:15:16.035123 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.037750 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:16.000345 20800 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.000s
W20260709 12:15:16.023101 20827 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.539s	user 0.013s	sys 0.000s
I20260709 12:15:16.054232 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:15.915863 20766 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.035766 20775 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:15.944427 20437 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.9, Last appended by leader: 9, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.057996 12642 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:16.063855 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:16.087160 12495 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:16.035590 20829 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:16.101069 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116323us
I20260709 12:15:16.003113 20872 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.106563 12571 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:16.100412 12491 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:16.117972 20872 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.116691 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:16.118654 20872 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:16.119392 20872 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.120811 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:16.125036 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:16.127708 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:16.002542 20545 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:16.058780 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 154988us
I20260709 12:15:16.142612 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:16.039789 20808 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.149048 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:16.059871 20431 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:16.151974 12571 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:16.152446 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:16.152114 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:16.150193 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:16.155202 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:16.067518 20689 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.076591 20775 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:16.074870 20653 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:16.204059 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52195.1us
W20260709 12:15:16.205546 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55433.7us
I20260709 12:15:16.209028 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:16.113768 20791 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:16.114125 20766 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:16.113837 20870 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:16.158658 20827 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:15:16.125173 20838 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.004s
I20260709 12:15:16.209493 12565 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:16.214843 12563 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:16.062106 20800 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.082042 20828 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:16.198130 20381 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:16.160259 20832 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:16.239248 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.210794 20855 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:16.218910 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:16.076277 20758 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:16.215615 20678 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.726s	user 0.012s	sys 0.001s
I20260709 12:15:16.103417 20779 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.207559 20757 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:15:16.255621 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:16.257140 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.237834 12566 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:16.239976 12491 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:16.226138 20653 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:16.242848 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.243826 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.261978 12488 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:16.262238 12497 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:15:16.233050 20838 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:16.263568 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:16.258450 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.265467 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.232240 12571 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:16.232188 20870 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.267932 12494 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
W20260709 12:15:16.255823 20893 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.158s	user 0.004s	sys 0.000s
I20260709 12:15:16.236788 20791 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:16.269153 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:16.268098 20895 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.321s	user 0.001s	sys 0.000s
W20260709 12:15:16.267558 20894 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20260709 12:15:16.268085 20875 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.218s
I20260709 12:15:16.304358 20626 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:16.305181 20626 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:16.306295 20626 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.309121 20871 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.309748 20545 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:16.310779 20904 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260709 12:15:16.311065 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3641) took 2067 ms. Trace:
W20260709 12:15:16.311468 20896 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:15:16.311513 12496 rpcz_store.cc:276] 0709 12:15:14.243775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.243938 (+   163us) service_pool.cc:224] Handling call
0709 12:15:14.244348 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:14.272899 (+ 28551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:14.272956 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:14.287431 (+ 14475us) log.cc:844] Serialized 26 byte log entry
0709 12:15:15.525716 (+1238285us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b18003e64a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:15.693100 (+167384us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:15.693171 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:15.693494 (+   323us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:16.310684 (+617190us) raft_consensus.cc:1685] finished
0709 12:15:16.310716 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:16.310979 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27992,"raft.queue_time_us":79297,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":5380,"spinlock_wait_cycles":10558976,"thread_start_us":167034,"threads_started":1,"wal-append.queue_time_us":1505016}
I20260709 12:15:16.290486 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:16.292318 12493 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
W20260709 12:15:16.268069 20892 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.316s	user 0.001s	sys 0.000s
I20260709 12:15:16.255752 20381 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:16.283839 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.316020 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:16.255589 20902 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:16.256090 20915 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:16.294415 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:16.259955 20689 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:16.319873 13259 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:16.295753 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.366012 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.366745 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:16.367654 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:16.267935 20891 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:15:16.267954 20900 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:16.368223 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.398406 12496 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:16.370224 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 218214us
I20260709 12:15:16.310967 20877 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:16.258541 20678 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.317881 20690 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:16.387933 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:16.343581 20898 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.342s	user 0.000s	sys 0.001s
I20260709 12:15:16.274348 20653 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:16.285351 20883 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:16.267982 20901 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:16.369334 20886 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.754s	user 0.013s	sys 0.000s
I20260709 12:15:16.405164 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.413193 12496 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:16.275045 20819 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.105s
I20260709 12:15:16.356251 20902 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:16.267393 20907 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.250s
I20260709 12:15:16.401737 20877 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:16.368244 12497 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
W20260709 12:15:16.417088 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:16.417948 20381 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 25.9, Last appended by leader: 9, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.418829 20779 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.419274 20883 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.420467 20653 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.421121 20886 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.400676 20913 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:16.397578 20689 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:16.268042 20903 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.376s
I20260709 12:15:16.525942 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1160) took 1065 ms. Trace:
I20260709 12:15:16.433673 20829 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:16.534921 12496 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:16.535316 12568 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:16.527237 13256 rpcz_store.cc:276] 0709 12:15:15.460769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.461140 (+   371us) service_pool.cc:224] Handling call
0709 12:15:15.467393 (+  6253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:15.470514 (+  3121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:15.470555 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:15.475075 (+  4520us) log.cc:844] Serialized 26 byte log entry
0709 12:15:15.766388 (+291313us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:15.766451 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:15.766551 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:16.525158 (+758607us) raft_consensus.cc:1685] finished
0709 12:15:16.525188 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:16.525759 (+   571us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3090,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":3546,"thread_start_us":290746,"threads_started":1,"wal-append.queue_time_us":383606}
I20260709 12:15:16.536298 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.510586 20808 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:16.561775 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.566998 12496 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:15:16.432631 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.399559 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:16.431389 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.578205 12489 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:16.318805 20899 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.436206 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:16.439973 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.412624 20859 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:16.237444 12387 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 13 to 23, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 23 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:16.426746 20819 consensus_queue.cc:1243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 25.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:15:16.521111 20872 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.535861 20800 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:16.604794 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 168702us
W20260709 12:15:16.616290 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 216874us
I20260709 12:15:16.615643 12496 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 19. Current term is 20. Ops: []
W20260709 12:15:16.643723 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.560833 20921 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:16.664741 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:16.610483 20768 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:16.616381 20838 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:15:16.617046 20899 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:16.602819 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:16.624298 20758 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.650501 12387 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 13 to 23. New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:16.751417 12488 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:16.749182 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:16.753296 12496 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:16.761409 20886 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:16.650522 20791 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
W20260709 12:15:16.765805 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43969.7us
I20260709 12:15:16.773186 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:16.740731 20901 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:16.745151 20828 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:16.780695 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3525) took 1286 ms. Trace:
I20260709 12:15:16.785070 12566 rpcz_store.cc:276] 0709 12:15:15.494164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.559070 (+ 64906us) service_pool.cc:224] Handling call
0709 12:15:15.559553 (+   483us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:16.261306 (+701753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:16.780401 (+519095us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:16.780614 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1551627136}
I20260709 12:15:16.782956 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1112) took 1582 ms. Trace:
I20260709 12:15:16.766192 20829 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:16.783355 12567 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.009s	sys 0.020s
I20260709 12:15:16.560376 20902 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.757046 20870 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.806129 12387 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 14 to 23. New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:16.790447 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:16.808967 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:16.788551 13253 rpcz_store.cc:276] 0709 12:15:15.200768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.200894 (+   126us) service_pool.cc:224] Handling call
0709 12:15:15.211694 (+ 10800us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:15.576111 (+364417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:16.782705 (+1206594us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:16.782892 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":754805504}
W20260709 12:15:16.793025 12567 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.009s	sys 0.020s
W20260709 12:15:16.813725 20662 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.890s	user 0.008s	sys 0.002s
I20260709 12:15:16.791924 20678 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:16.814610 12566 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:16.778689 20758 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:16.815480 20758 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:16.816030 20502 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.948s	user 0.011s	sys 0.000s
I20260709 12:15:16.816558 20502 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.827298 20502 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:16.836371 12489 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:16.611595 20877 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.799733 20829 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.789520 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:16.808424 12387 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 14 to 16, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 16 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:16.810184 20883 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:16.793417 20828 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.814276 20662 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:16.798458 20832 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.549s	user 0.009s	sys 0.001s
I20260709 12:15:16.730955 20920 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.116s
I20260709 12:15:16.838274 12489 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:16.678920 20921 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:16.853075 20780 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.909s	user 0.006s	sys 0.007s
W20260709 12:15:16.829449 20784 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.780s	user 0.009s	sys 0.000s
I20260709 12:15:16.668612 20689 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.756564 20907 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
W20260709 12:15:16.791540 20925 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:15:16.845129 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 197755us
I20260709 12:15:16.856096 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:16.840030 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
W20260709 12:15:16.878000 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.878659 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:16.862556 20839 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.001s	user 0.014s	sys 0.000s
I20260709 12:15:16.861974 12571 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
W20260709 12:15:16.855611 20713 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.941s	user 0.008s	sys 0.005s
I20260709 12:15:16.786882 20901 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.870414 12491 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:16.874205 12496 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:16.884653 20907 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:16.893229 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:16.895664 12567 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:16.893908 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:16.901886 12496 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:15:16.911966 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:16.880481 20832 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:16.917364 12496 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:15:16.888361 20662 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:16.920058 13253 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:16.868551 20780 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.931252 12496 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:15:16.933807 12567 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:16.906942 20758 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.913946 20907 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:16.870338 20784 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:16.945950 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:16.943974 13255 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:16.923851 20819 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:16.922029 20796 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:16.948367 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:16.943548 20431 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.731s	user 0.015s	sys 0.000s
I20260709 12:15:16.897696 20713 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:16.959071 20901 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:16.929947 20931 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:15:16.964336 12490 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:16.957269 20819 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:16.975678 12489 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:16.939929 20832 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:16.941748 20930 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
W20260709 12:15:17.004854 20928 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.127s	user 0.002s	sys 0.000s
I20260709 12:15:17.012558 12492 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:17.013128 12489 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:17.018103 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1066) took 1095 ms. Trace:
I20260709 12:15:17.012506 20828 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.008176 20758 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:17.009673 20829 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.012115 20910 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:17.017427 20766 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.789s	user 0.014s	sys 0.000s
I20260709 12:15:16.984220 20819 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:17.019438 20198 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.103s	user 0.013s	sys 0.001s
W20260709 12:15:17.027410 20933 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:15:17.025410 20758 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:17.032845 20918 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.727s	user 0.009s	sys 0.000s
I20260709 12:15:16.947542 12387 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 20 to 22, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:17.022342 12639 rpcz_store.cc:276] 0709 12:15:15.922805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.922998 (+   193us) service_pool.cc:224] Handling call
0709 12:15:15.954693 (+ 31695us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:16.074605 (+119912us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:16.074657 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:16.075520 (+   863us) log.cc:844] Serialized 52 byte log entry
0709 12:15:16.968624 (+893104us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b18004d7a00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:16.984137 (+ 15513us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:17.011621 (+ 27484us) log.cc:844] Serialized 14 byte log entry
0709 12:15:17.011934 (+   313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:17.012038 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:17.012081 (+    43us) raft_consensus.cc:1685] finished
0709 12:15:17.017933 (+  5852us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:17.018054 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19780,"raft.queue_time_us":114104,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":40482,"spinlock_wait_cycles":21481088,"thread_start_us":152709,"threads_started":3,"wal-append.queue_time_us":908734}
I20260709 12:15:17.041368 20582 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:17.009693 20907 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.044544 20931 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.051807 20582 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.052088 20931 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.046586 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1404) took 1739 ms. Trace:
I20260709 12:15:17.055800 12563 rpcz_store.cc:276] 0709 12:15:15.307092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.522384 (+215292us) service_pool.cc:224] Handling call
0709 12:15:15.522831 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:16.232628 (+709797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:17.046243 (+813615us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:17.046511 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1590991232}
I20260709 12:15:17.048736 12489 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: [26.10-26.10]
W20260709 12:15:17.061578 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.059840 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3504) took 1069 ms. Trace:
I20260709 12:15:17.043505 20921 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.065230 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.062080 12569 rpcz_store.cc:276] 0709 12:15:15.990035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.061087 (+ 71052us) service_pool.cc:224] Handling call
0709 12:15:16.070174 (+  9087us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:16.070521 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:16.070544 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:16.071211 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:15:16.235369 (+164158us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:16.235447 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:16.235596 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:17.032121 (+796525us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:17.054468 (+ 22347us) raft_consensus.cc:1685] finished
0709 12:15:17.054502 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:17.059771 (+  5269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87596928,"thread_start_us":163685,"threads_started":1,"wal-append.queue_time_us":234502}
I20260709 12:15:17.067723 12563 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:17.069428 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3492) took 1185 ms. Trace:
I20260709 12:15:17.069880 12568 rpcz_store.cc:276] 0709 12:15:15.884235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.884435 (+   200us) service_pool.cc:224] Handling call
0709 12:15:15.900232 (+ 15797us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:16.536124 (+635892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:17.069175 (+533051us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:17.069366 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1459859840}
W20260709 12:15:17.073172 20775 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.857s	user 0.009s	sys 0.005s
W20260709 12:15:17.075837 20887 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.673s	user 0.004s	sys 0.027s
I20260709 12:15:17.077152 20780 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:17.077771 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.078805 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1065) took 2287 ms. Trace:
I20260709 12:15:17.079630 20713 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.062508 20766 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:15:17.081020 20766 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:17.081103 20832 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.045348 20910 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.079259 12643 rpcz_store.cc:276] 0709 12:15:14.791449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.791614 (+   165us) service_pool.cc:224] Handling call
0709 12:15:14.792010 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:15.084162 (+292152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:15.084204 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:15.110323 (+ 26119us) log.cc:844] Serialized 26 byte log entry
0709 12:15:15.205587 (+ 95264us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:15.205806 (+   219us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:15.205951 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:17.022432 (+1816481us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:17.078426 (+ 55994us) raft_consensus.cc:1685] finished
0709 12:15:17.078454 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:17.078733 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":119898,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":67,"spinlock_wait_cycles":39035776,"thread_start_us":312907,"threads_started":2,"wal-append.queue_time_us":142734}
I20260709 12:15:17.083468 20784 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.083652 20775 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.086012 20409 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.086961 20758 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.089949 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:17.091380 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:17.092636 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:17.047547 20198 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.093883 20653 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:17.094398 12387 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 12 to 19. New cstate: current_term: 19 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:15:17.095283 20198 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.096064 20198 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:17.081506 20766 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:17.096379 12387 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 6 to 19, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 19 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:17.097994 20866 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:17.098193 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:17.097544 20766 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.095279 20381 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:17.104350 20653 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:17.100642 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:17.105562 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.105806 20669 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:17.107035 20767 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:17.108681 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.059492 20582 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.096278 20775 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.130249 20832 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:17.130913 20832 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.133392 20713 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.128405 20669 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.134114 20713 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:17.134759 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.136058 20669 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.119436 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.140491 12572 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:17.141585 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:17.142220 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3495) took 1207 ms. Trace:
I20260709 12:15:17.142669 12565 rpcz_store.cc:276] 0709 12:15:15.934781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.943248 (+  8467us) service_pool.cc:224] Handling call
0709 12:15:17.141976 (+1198728us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:17.142151 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":583863040}
I20260709 12:15:17.144395 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.121474 20381 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:17.145951 20381 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:17.149729 20758 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:17.150501 20766 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:17.118430 20866 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:17.158172 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1398) took 1852 ms. Trace:
I20260709 12:15:17.158182 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:17.158676 12570 rpcz_store.cc:276] 0709 12:15:15.305364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.308777 (+  3413us) service_pool.cc:224] Handling call
0709 12:15:15.309177 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:15.781349 (+472172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:17.157848 (+1376499us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54002cfbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:17.158098 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1009645824}
W20260709 12:15:17.122627 20859 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.507s	user 0.006s	sys 0.006s
I20260709 12:15:17.086655 20409 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.137509 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:17.166621 13256 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:17.137831 20831 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:17.168709 20916 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.865s	user 0.003s	sys 0.007s
I20260709 12:15:17.123762 20902 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:17.175511 20913 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:15:17.178824 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.138307 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:17.181602 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:17.139281 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3653) took 1240 ms. Trace:
I20260709 12:15:17.182664 12493 rpcz_store.cc:276] 0709 12:15:15.899084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.909467 (+ 10383us) service_pool.cc:224] Handling call
0709 12:15:17.139105 (+1229638us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:17.139215 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":869949184}
I20260709 12:15:17.183351 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:17.124449 20767 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:17.184495 20779 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.185405 20767 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.186802 20767 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:17.188197 20866 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.159857 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:17.161409 12563 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:15:17.128305 20921 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.192044 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:17.194168 12567 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:17.161798 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:17.195215 13258 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:17.115161 12568 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:17.197767 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3503) took 1242 ms. Trace:
I20260709 12:15:17.139986 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:17.167299 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:17.198423 12564 rpcz_store.cc:276] 0709 12:15:15.955831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.050938 (+ 95107us) service_pool.cc:224] Handling call
0709 12:15:16.051220 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:16.051578 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:16.051609 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:16.052161 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:15:16.206805 (+154644us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:16.206857 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:16.206935 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:17.141227 (+934292us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:17.197306 (+ 56079us) raft_consensus.cc:1685] finished
0709 12:15:17.197345 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:17.197679 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1053,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5152,"spinlock_wait_cycles":89839872,"thread_start_us":139876,"threads_started":1,"wal-append.queue_time_us":247662}
I20260709 12:15:17.167666 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:17.138835 13261 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:17.168220 20901 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:17.139539 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:17.142866 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:17.169209 12387 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 12 to 20, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 20 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:15:17.190335 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53232.5us
I20260709 12:15:17.210880 12565 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:15:17.211046 12564 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
W20260709 12:15:17.167014 20911 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.872s	user 0.000s	sys 0.017s
I20260709 12:15:17.224625 12569 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:17.222003 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:17.185134 20779 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:17.223449 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63648.1us
I20260709 12:15:17.133414 20832 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.225693 20669 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:17.239821 20545 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 12:15:17.215898 20582 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.246623 20582 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.214771 20921 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.251406 20921 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.250730 12387 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 11 to 23, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:15:17.168919 20831 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.254777 20831 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.213894 20883 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:15:17.241531 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:17.263478 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:17.264716 20877 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:17.265020 20758 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:17.265342 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:17.265583 20758 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:17.265971 20758 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:17.267550 20653 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:15:17.267997 20875 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:17.268225 20653 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:17.268682 20875 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:17.269693 20866 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:17.269814 20875 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.205248 20901 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:17.271440 20901 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:17.274626 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.275904 20758 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.278381 20758 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.269366 20653 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.283149 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.283950 12643 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:17.284004 12645 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:17.284986 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.286016 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.287070 20409 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.286887 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.287746 20713 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.290052 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.290445 20778 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:17.291060 12641 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:17.292723 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.294474 12638 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:17.287801 12646 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:17.298827 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.299489 12646 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:17.300179 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.301554 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.302114 12568 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:17.302522 12646 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:17.249327 20582 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.303782 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.308864 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:17.312407 13256 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:17.314427 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.325845 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.326671 12637 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:17.326913 12564 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
W20260709 12:15:17.320573 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.328711 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:17.224177 20911 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.000s	sys 0.018s
I20260709 12:15:17.325006 13257 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:17.339030 12568 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:17.244599 20198 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:17.344380 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:17.351114 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:17.355448 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.356448 12641 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:15:17.364652 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:17.367880 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.369843 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3656) took 1458 ms. Trace:
I20260709 12:15:17.370280 12490 rpcz_store.cc:276] 0709 12:15:15.911442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.911777 (+   335us) service_pool.cc:224] Handling call
0709 12:15:15.922214 (+ 10437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:16.964994 (+1042780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:17.369522 (+404528us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:17.369770 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2396017536}
I20260709 12:15:17.371820 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1087) took 1126 ms. Trace:
I20260709 12:15:17.371958 12496 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:17.372504 12491 rpcz_store.cc:276] 0709 12:15:16.245176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.265210 (+ 20034us) service_pool.cc:224] Handling call
0709 12:15:17.371616 (+1106406us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:17.371758 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1389088640}
I20260709 12:15:17.373202 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.373900 12497 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:17.376967 12565 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:17.373692 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.373649 12637 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:17.378340 12491 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:17.388317 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 166345us
I20260709 12:15:17.395823 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:17.398469 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.459769 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 988) took 1095 ms. Trace:
I20260709 12:15:17.460233 13255 rpcz_store.cc:276] 0709 12:15:16.275950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.276205 (+   255us) service_pool.cc:224] Handling call
0709 12:15:17.371272 (+1095067us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:17.371386 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1252206976}
I20260709 12:15:17.340365 20913 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:17.463649 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:17.465560 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.285005 20931 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:17.376837 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:17.465144 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101862us
I20260709 12:15:17.467335 12488 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:15:17.468392 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.390021 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.469676 12642 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:15:17.469945 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.470608 12491 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:17.471493 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:17.473516 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 109002us
I20260709 12:15:17.474622 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.475593 12490 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:17.473481 20913 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:17.479526 20913 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:17.253306 12387 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 12 to 22, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:17.480875 12387 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 17 to 20, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 20 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:17.459149 20381 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.486040 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.483790 12387 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 14 to 22, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:17.486939 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.487323 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.487643 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:17.487908 12489 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:17.488605 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.489305 12637 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:17.465073 20444 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:17.486052 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.488909 12387 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 11 to 21. New cstate: current_term: 21 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:17.491129 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:17.361315 20198 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.492622 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.491938 12387 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 23, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 23 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:17.493343 12489 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:17.493943 20198 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.494335 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.495553 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.496186 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:17.496243 20831 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:17.496838 20831 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:17.497078 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.465790 20876 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:15:17.357152 20582 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:17.520012 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:17.498885 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.500752 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.521222 12488 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:17.522035 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:17.522257 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.522117 12564 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:15:17.525933 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.465598 13258 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:17.513324 12639 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:17.528509 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:17.531659 12563 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:17.532775 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:17.536329 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:17.538546 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:17.498492 20946 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:15:17.497891 20678 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:17.650555 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:17.664523 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:17.609318 20941 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:15:17.528546 20713 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.537781 20855 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:17.705526 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2876) took 1417 ms. Trace:
I20260709 12:15:17.497972 20831 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.529165 20871 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:17.708004 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:17.591572 12564 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:15:17.531267 20921 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:17.717479 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 189018us
I20260709 12:15:17.735155 12497 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
W20260709 12:15:17.737078 20767 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.550s	user 0.010s	sys 0.000s
I20260709 12:15:17.735764 12488 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:17.809772 12496 rpcz_store.cc:276] 0709 12:15:16.288323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.937770 (+649447us) service_pool.cc:224] Handling call
0709 12:15:17.371748 (+433978us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:17.372613 (+   865us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:17.679679 (+307066us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b5400191750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:17.705383 (+ 25704us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":997198592}
I20260709 12:15:17.839082 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.839417 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:17.809368 12565 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:17.811573 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.829413 20409 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:17.812750 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3650) took 1922 ms (client timeout 1651 ms). Trace:
I20260709 12:15:17.818722 12569 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:17.529511 20582 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:17.843849 12492 rpcz_store.cc:269] 0709 12:15:15.890761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:15.892163 (+  1402us) service_pool.cc:224] Handling call
0709 12:15:17.812517 (+1920354us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:17.812665 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2527133440}
W20260709 12:15:17.713953 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:17.840667 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:17.847419 12488 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:15:17.824294 20942 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.304s	user 0.001s	sys 0.007s
I20260709 12:15:17.841532 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
W20260709 12:15:17.807891 20653 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.012s	sys 0.008s
I20260709 12:15:17.829457 20811 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:17.803810 20444 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:15:17.835026 20198 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.843061 20869 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:17.888562 12643 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:17.851657 20582 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:17.890765 12639 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:15:17.893061 12492 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:17.893122 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:17.842628 20431 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:17.894124 12637 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
W20260709 12:15:17.897265 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:17.894981 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.014s	sys 0.002s
W20260709 12:15:17.899287 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.014s	sys 0.003s
I20260709 12:15:17.820881 20678 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.868940 20831 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:17.879302 20653 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.012s	sys 0.009s
I20260709 12:15:17.871423 20871 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:17.841730 20409 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.825822 20713 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.910784 20784 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:15:17.912771 20902 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.737s	user 0.007s	sys 0.002s
I20260709 12:15:17.915064 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:17.896588 20811 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.920395 20444 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:15:17.921100 20877 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.656s	user 0.004s	sys 0.004s
I20260709 12:15:17.896854 20198 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.842556 20921 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:17.941879 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:17.919243 20901 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.647s	user 0.007s	sys 0.004s
I20260709 12:15:17.899796 20869 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:17.969931 13253 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:15:17.943635 20784 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:17.927443 20431 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:17.945021 20653 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:15:17.961292 20902 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:17.962107 20444 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:17.960997 20775 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:15:17.982669 20901 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.002914 20921 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.003376 20901 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:17.994028 20653 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:15:18.000677 20866 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.730s	user 0.008s	sys 0.004s
I20260709 12:15:18.007647 20678 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:17.996599 20902 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:18.019006 20778 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.002s
I20260709 12:15:17.992141 20871 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:18.023352 20913 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.001s
I20260709 12:15:18.022042 20855 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:18.119338 12491 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:18.118014 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.016s	sys 0.044s
W20260709 12:15:18.122225 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.017s	sys 0.044s
I20260709 12:15:18.124085 12492 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:15:18.126678 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
W20260709 12:15:18.127934 12494 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.006s	sys 0.004s
W20260709 12:15:18.128517 12494 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.007s	sys 0.004s
I20260709 12:15:18.128376 12492 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:18.129106 12492 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:18.132055 12492 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
W20260709 12:15:17.928515 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.053s	sys 0.096s
W20260709 12:15:18.292601 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s	user 0.053s	sys 0.096s
I20260709 12:15:18.298312 12644 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:18.299471 20866 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.300114 20819 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:18.300724 20866 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.195286 20966 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:18.218533 20811 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.224198 20409 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.041255 20913 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:18.230811 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.236001 20713 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.240559 20768 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.242810 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1081) took 2004 ms. Trace:
I20260709 12:15:18.245466 20431 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.032116 20784 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.251559 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1161) took 1115 ms. Trace:
I20260709 12:15:18.018306 20444 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.044844 20855 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.257949 12566 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:15:18.259824 12571 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:18.012110 20653 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:18.269672 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1163) took 1116 ms. Trace:
I20260709 12:15:18.271754 20869 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:18.288419 12645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.009s	sys 0.008s
I20260709 12:15:18.291605 20198 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:18.301158 20819 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:18.194628 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 658157us
W20260709 12:15:18.165601 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 438982us
I20260709 12:15:18.229135 20902 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:18.325449 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.328944 12642 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 16
I20260709 12:15:18.329459 12642 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:15:18.329916 12642 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:18.330623 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3362) took 1014 ms. Trace:
W20260709 12:15:18.330374 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:18.331182 12646 rpcz_store.cc:276] 0709 12:15:17.316409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.323120 (+  6711us) service_pool.cc:224] Handling call
0709 12:15:17.323610 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:17.338789 (+ 15179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:17.338819 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:17.339342 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 12:15:17.899788 (+560446us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:18.318589 (+418801us) log.cc:844] Serialized 14 byte log entry
0709 12:15:18.329765 (+ 11176us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.329923 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:18.330043 (+   120us) raft_consensus.cc:1685] finished
0709 12:15:18.330060 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.330343 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":336702,"prepare.queue_time_us":147022,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":27507,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":16866,"thread_start_us":788114,"threads_started":2,"wal-append.queue_time_us":630631}
W20260709 12:15:18.322417 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 489526us
I20260709 12:15:18.308846 12495 rpcz_store.cc:276] 0709 12:15:16.238426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.238663 (+   237us) service_pool.cc:224] Handling call
0709 12:15:16.239020 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:16.404292 (+165272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:16.404327 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:16.404371 (+    44us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:16.404407 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.242160 (+1837753us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:18.242252 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.242543 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379561856}
W20260709 12:15:18.340775 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.009s	sys 0.008s
W20260709 12:15:18.341372 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.010s	sys 0.008s
I20260709 12:15:18.342897 20768 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.310034 13253 rpcz_store.cc:276] 0709 12:15:17.136016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.136138 (+   122us) service_pool.cc:224] Handling call
0709 12:15:17.136454 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:17.970345 (+833891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.250564 (+280219us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:18.251123 (+   559us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1912423296}
I20260709 12:15:18.345882 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3673) took 1041 ms. Trace:
I20260709 12:15:18.346611 12493 rpcz_store.cc:276] 0709 12:15:17.304544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.310193 (+  5649us) service_pool.cc:224] Handling call
0709 12:15:17.360540 (+ 50347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:17.498051 (+137511us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:17.498084 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:17.528691 (+ 30607us) log.cc:844] Serialized 52 byte log entry
0709 12:15:18.197315 (+668624us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:18.197449 (+   134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.197717 (+   268us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:18.345219 (+147502us) raft_consensus.cc:1685] finished
0709 12:15:18.345246 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.345567 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":188010,"raft.queue_time_us":157543,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":79079,"thread_start_us":388158,"threads_started":1,"wal-append.queue_time_us":531481}
I20260709 12:15:18.348731 12639 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:15:18.352563 12492 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
W20260709 12:15:18.354331 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.329136 20444 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:18.356624 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3528) took 1537 ms. Trace:
W20260709 12:15:18.335604 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.009s	sys 0.009s
I20260709 12:15:18.364180 20819 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.364970 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3360) took 1049 ms. Trace:
I20260709 12:15:18.329160 20913 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:18.365439 12645 rpcz_store.cc:276] 0709 12:15:17.315223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.315386 (+   163us) service_pool.cc:224] Handling call
0709 12:15:17.339732 (+ 24346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:17.378221 (+ 38489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:17.378255 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:17.461401 (+ 83146us) log.cc:844] Serialized 26 byte log entry
0709 12:15:18.364343 (+902942us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:18.364435 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.364588 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:18.364661 (+    73us) raft_consensus.cc:1685] finished
0709 12:15:18.364674 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.364854 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":372528,"raft.queue_time_us":55512,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":11954,"thread_start_us":651430,"threads_started":1,"wal-append.queue_time_us":529177}
I20260709 12:15:18.312783 13259 rpcz_store.cc:276] 0709 12:15:17.136580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.138043 (+  1463us) service_pool.cc:224] Handling call
0709 12:15:18.252968 (+1114925us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54005e1d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:18.253295 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":646761344}
I20260709 12:15:18.323841 20431 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.357110 12571 rpcz_store.cc:276] 0709 12:15:16.783673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.145657 (+361984us) service_pool.cc:224] Handling call
0709 12:15:18.258454 (+1112797us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:18.259146 (+   692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:18.319430 (+ 60284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.320721 (+  1291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50155904}
I20260709 12:15:18.378480 20669 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:18.382120 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:18.390197 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.397605 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.397682 20442 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:18.398756 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:18.405835 12491 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:18.400876 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1095) took 1220 ms. Trace:
I20260709 12:15:18.409543 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.419440 20473 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:18.420080 20473 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:18.426118 20431 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.430436 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:18.432365 12565 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:18.435252 20779 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:18.436043 20786 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:18.436941 20786 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:18.440191 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.006s	sys 0.010s
W20260709 12:15:18.440814 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.007s	sys 0.010s
W20260709 12:15:18.441376 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 973442us
I20260709 12:15:18.407457 20198 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:18.441610 20786 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.443074 20198 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.443074 20811 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:18.449283 20811 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.451953 20811 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.458088 20442 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:18.458865 20442 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.461043 20910 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.462014 20978 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.436257 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:18.463018 20978 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:18.464037 20951 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.448s	user 0.000s	sys 0.002s
I20260709 12:15:18.465286 20866 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.470439 12571 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:15:18.457535 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.011s	sys 0.039s
I20260709 12:15:18.472023 20442 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.472680 20978 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.477883 20409 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.478698 20965 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.479615 20902 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.459219 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:18.492748 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62433.8us
W20260709 12:15:18.499238 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.012s	sys 0.039s
I20260709 12:15:18.480693 20808 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:18.484742 20966 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:18.495792 20409 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.506393 20442 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:18.515974 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3357) took 1206 ms. Trace:
I20260709 12:15:18.528283 20198 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:18.532915 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.506605 20811 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.524628 12644 rpcz_store.cc:276] 0709 12:15:17.301009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.312979 (+ 11970us) service_pool.cc:224] Handling call
0709 12:15:17.357657 (+ 44678us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:18.299057 (+941400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.481149 (+182092us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:18.507202 (+ 26053us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1956178688}
I20260709 12:15:18.519585 20442 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:15:18.543190 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 835568us
I20260709 12:15:18.407016 12494 rpcz_store.cc:276] 0709 12:15:17.180409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.216430 (+ 36021us) service_pool.cc:224] Handling call
0709 12:15:17.216781 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:17.362531 (+145750us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:15:18.129392 (+766861us) log.cc:844] Serialized 14 byte log entry
0709 12:15:18.322432 (+193040us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:18.359559 (+ 37127us) log.cc:844] Serialized 26 byte log entry
0709 12:15:18.359794 (+   235us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:18.359853 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.359998 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:18.368034 (+  8036us) raft_consensus.cc:1685] finished
0709 12:15:18.368061 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.400795 (+ 32734us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57420,"prepare.queue_time_us":513096,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"raft.queue_time_us":156555,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":104338,"spinlock_wait_cycles":61174016,"thread_start_us":843186,"threads_started":2,"wal-append.queue_time_us":193519}
I20260709 12:15:18.552786 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1420) took 1278 ms. Trace:
I20260709 12:15:18.549000 20442 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:18.568095 12637 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:15:18.421535 20473 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.568326 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.559332 20431 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.579159 12565 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:15:18.591233 12646 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:15:18.556538 12572 rpcz_store.cc:276] 0709 12:15:17.274411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.275260 (+   849us) service_pool.cc:224] Handling call
0709 12:15:17.275645 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:17.276053 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:17.276100 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:17.276671 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:15:18.293564 (+1016893us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:18.293666 (+   102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.293965 (+   299us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:18.462973 (+169008us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:18.547241 (+ 84268us) raft_consensus.cc:1685] finished
0709 12:15:18.547269 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.552685 (+  5416us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1344,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"spinlock_wait_cycles":3634048,"thread_start_us":1016104,"threads_started":1,"wal-append.queue_time_us":715623}
I20260709 12:15:18.568821 20978 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:18.559646 20902 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.541661 20966 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.605144 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.602542 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:18.614629 20669 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:18.615856 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:18.624732 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:18.602257 20409 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.625980 20473 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:18.602015 20811 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.629055 20811 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.630020 20381 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.634454 12638 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:18.636222 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.636288 20811 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:18.576028 20808 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.638159 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:18.602941 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:18.626288 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167358us
I20260709 12:15:18.574631 20965 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.659811 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.661412 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.663082 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.663672 20832 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:18.664651 20832 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:18.664444 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.660686 12564 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:18.669205 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.664719 12571 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:18.626255 20409 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:15:18.650621 20811 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:18.599911 20784 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:15:18.655157 20978 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.673240 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.685309 20965 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.685715 20832 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 26.10, Last appended by leader: 10, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.692823 20909 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.693866 20909 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.694291 20713 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:18.695411 20713 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.715655 20713 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.690042 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:18.711725 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:18.693099 20662 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:18.677076 20866 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.715423 20978 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:18.699086 20984 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.000s	sys 0.002s
I20260709 12:15:18.699226 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3683) took 1354 ms. Trace:
I20260709 12:15:18.692229 20832 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:18.743759 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:18.719611 20662 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:18.700488 20902 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.746055 20662 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.746697 12645 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:18.747776 20662 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:18.722646 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.749864 20808 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.676157 20811 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:18.750854 20811 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:18.750923 20786 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:18.751597 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3678) took 1443 ms. Trace:
I20260709 12:15:18.752020 12489 rpcz_store.cc:276] 0709 12:15:17.307875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.870100 (+562225us) service_pool.cc:224] Handling call
0709 12:15:17.872993 (+  2893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:18.286436 (+413443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:18.286505 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:18.287562 (+  1057us) log.cc:844] Serialized 26 byte log entry
0709 12:15:18.322026 (+ 34464us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:18.322068 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:18.333820 (+ 11752us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:18.751104 (+417284us) raft_consensus.cc:1685] finished
0709 12:15:18.751128 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:18.751375 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":46780,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":87,"thread_start_us":446415,"threads_started":2,"wal-append.queue_time_us":217820}
W20260709 12:15:18.726095 20582 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.752s	user 0.006s	sys 0.016s
I20260709 12:15:18.754170 20582 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:18.755117 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130681us
I20260709 12:15:18.756661 20582 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.756963 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:18.688984 20381 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.758899 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:18.759670 20381 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.760694 12643 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:18.706482 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:18.731938 20713 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:18.732944 20866 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.762521 12640 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:15:18.764473 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.764981 20866 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:18.765307 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:18.767642 20381 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.768711 20713 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:18.769182 20713 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:18.774331 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2894) took 1339 ms (client timeout 1689 ms). Trace:
W20260709 12:15:18.774930 12491 rpcz_store.cc:269] 0709 12:15:17.434922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.471283 (+ 36361us) service_pool.cc:224] Handling call
0709 12:15:18.774041 (+1302758us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:18.774226 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1487476608}
I20260709 12:15:18.780740 12572 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:15:18.696758 20909 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.720924 12638 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:18.732197 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:18.791214 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107088us
I20260709 12:15:18.793288 20883 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:18.744189 20978 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:18.799535 20831 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.877s	user 0.015s	sys 0.009s
I20260709 12:15:18.718000 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:18.800071 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.685292 20981 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.092s
I20260709 12:15:18.802870 20981 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:15:18.802994 20768 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.803464 20981 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:18.697206 20980 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:15:18.806588 20980 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:18.807096 20980 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:18.807207 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1410) took 2044 ms (client timeout 2048 ms). Trace:
I20260709 12:15:18.750300 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:15:18.807785 12566 rpcz_store.cc:269] 0709 12:15:16.762653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:16.878327 (+115674us) service_pool.cc:224] Handling call
0709 12:15:18.806890 (+1928563us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:18.807087 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3053042560}
I20260709 12:15:18.803740 20669 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:18.758671 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.740693 12496 rpcz_store.cc:276] 0709 12:15:17.345061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.839027 (+493966us) service_pool.cc:224] Handling call
0709 12:15:18.698747 (+859720us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:18.698953 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1763902208}
I20260709 12:15:18.758101 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.814960 12571 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:15:18.801570 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 279146us
W20260709 12:15:18.818930 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87556.2us
I20260709 12:15:18.821524 20811 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.824492 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:18.824960 20811 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.825363 13259 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:18.826786 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:18.804827 20981 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:18.830076 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.830551 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:18.831704 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:18.834347 20969 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.841715 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:18.842338 20969 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:18.842864 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86032.6us
I20260709 12:15:18.844372 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:18.844955 20969 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.845644 20985 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:18.846165 20985 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:18.846596 20985 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:18.847103 20987 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.003s	sys 0.000s
I20260709 12:15:18.848637 20980 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.830407 20981 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:15:18.859615 13253 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:15:18.854059 20909 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:18.868008 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:18.869176 12490 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:15:18.870738 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3677) took 1563 ms. Trace:
I20260709 12:15:18.871028 20866 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:18.871119 12490 rpcz_store.cc:276] 0709 12:15:17.307507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:17.500467 (+192960us) service_pool.cc:224] Handling call
0709 12:15:18.870486 (+1370019us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540034e754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:18.870645 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3031688960}
I20260709 12:15:18.874593 20678 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.875593 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:18.875510 20678 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.877429 20913 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:15:18.879388 20913 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:18.884263 12569 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 26 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 9.
I20260709 12:15:18.884212 20713 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.876073 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:18.859447 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:18.900888 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.860821 20909 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:18.926739 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58912.3us
I20260709 12:15:18.928581 12645 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:18.885730 20819 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:18.932998 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57522.7us
I20260709 12:15:18.934406 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:18.935395 20909 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:18.937553 20653 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.503s	user 0.008s	sys 0.008s
I20260709 12:15:18.938663 20653 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.941864 12637 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:18.938069 20907 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:18.944841 20907 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:18.945262 20907 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:18.946980 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.947265 20768 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:15:18.949039 20713 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:18.950052 20819 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.947031 20678 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:18.851653 20980 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:18.954486 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:18.956837 20444 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.601s	user 0.015s	sys 0.000s
I20260709 12:15:18.851841 20784 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.984771 20381 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:15:18.996688 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:18.975404 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.980528 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:18.984163 20985 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:18.989956 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:18.976257 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:18.985653 20913 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:18.976531 20442 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:18.955011 20653 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:18.857671 20978 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:18.856702 20811 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:19.009405 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:15:19.012253 12639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.030s	sys 0.104s
I20260709 12:15:19.024889 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:19.027987 12639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.030s	sys 0.104s
I20260709 12:15:19.028267 13262 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:19.010677 20678 raft_consensus.cc:2764] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:15:19.024251 20832 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:19.030593 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.038008 12571 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:19.035419 13259 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:19.041033 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51248.9us
I20260709 12:15:19.043238 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
W20260709 12:15:19.055541 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.057374 20985 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:19.059859 20907 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:19.062259 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77389.7us
I20260709 12:15:19.063614 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:19.027699 20819 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.029798 20811 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:19.034749 20832 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:19.021821 20784 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:19.084414 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 230193us
I20260709 12:15:19.046278 20913 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:19.094512 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:19.095443 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.110349 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.074420 20811 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:19.115801 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.083706 20832 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:19.117071 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.085275 20653 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:19.115394 13253 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:19.121501 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:19.122604 13254 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:19.068202 20978 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.131248 20985 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:19.132135 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:19.135864 20913 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.141259 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.142602 12637 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:19.144548 12645 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 18. Current term is 19. Ops: []
I20260709 12:15:19.144613 20784 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.146905 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:19.148746 20866 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:19.118594 20819 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.150470 20819 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.148561 20758 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:19.142702 13258 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:19.157492 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.182404 12641 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:19.114024 20907 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:19.190541 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.191313 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:19.190604 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:19.192701 12567 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:19.149453 20866 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:19.144445 20669 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.179987 20985 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:19.196165 20768 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:19.196588 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:19.128400 20582 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.197395 12565 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:19.196954 20985 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.197680 20582 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.201279 20582 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.203300 20473 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:19.203944 20473 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.204399 20473 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:19.182264 20431 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.206532 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.206631 20431 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.207650 20909 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:19.208163 20909 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.208503 20909 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:19.194684 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.144915 20786 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.153479 20819 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.158283 20442 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.205889 12567 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:19.155530 20678 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:19.156771 20653 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:15:19.229362 12564 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
W20260709 12:15:19.233513 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 147590us
I20260709 12:15:19.211124 20431 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.219189 20678 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:19.248477 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 168353us
I20260709 12:15:19.235941 20981 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:19.216708 20669 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.250283 20678 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:19.206459 20866 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.268185 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:19.268942 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110113us
W20260709 12:15:19.268772 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.271791 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:19.253589 20786 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.274410 20866 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:19.275347 20786 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:19.284497 20979 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.006s	sys 0.022s
I20260709 12:15:19.284790 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:19.285166 20979 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.006s	sys 0.023s
I20260709 12:15:19.287590 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.289151 20981 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:19.321172 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.322072 12570 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:19.322985 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.323154 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.323617 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:19.324556 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:19.324893 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.325891 12568 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:19.327097 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64627.2us
I20260709 12:15:19.329557 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:19.325214 20832 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.331764 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.308771 20819 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:15:19.341802 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.008s	sys 0.009s
W20260709 12:15:19.344499 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.008s	sys 0.009s
W20260709 12:15:19.353402 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 210183us
I20260709 12:15:19.358178 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.313841 20981 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:19.364673 12563 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:19.366381 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:19.371218 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86632.4us
W20260709 12:15:19.379444 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.382797 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.384650 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:19.392086 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:19.394860 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.346388 20662 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:19.362330 20819 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:19.378289 20442 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.415596 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.416743 12567 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:19.419195 12563 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:15:19.420804 13262 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:19.423902 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.390856 20431 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:19.429769 12568 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:19.404170 20978 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:19.407480 20808 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:19.432299 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:19.439725 12490 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:15:19.442314 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:19.449290 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88447.9us
I20260709 12:15:19.450033 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:19.459607 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:19.426329 20653 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:19.473497 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:19.473791 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.439530 20808 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.416276 20811 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.482335 20871 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:19.444109 20444 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.449717 20786 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:19.422245 20662 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.422645 20442 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:19.485273 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170505us
W20260709 12:15:19.490557 20997 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:15:19.492010 12494 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
W20260709 12:15:19.512457 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.483376 12563 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:19.483542 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:19.515753 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:19.518878 12570 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:19.473477 20978 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:19.494189 20871 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:19.526711 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46183.6us
W20260709 12:15:19.504611 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 138443us
I20260709 12:15:19.522933 12489 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:19.485481 20808 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:19.499564 20669 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:19.499328 20786 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:19.538115 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:19.533118 12567 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:19.533489 12572 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:19.542703 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:19.533860 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.534929 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.535127 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:19.544036 12563 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:19.536031 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:19.548897 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.527642 20832 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.528746 20653 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:19.571372 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.529613 20978 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:19.530285 20871 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:19.564574 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:19.569658 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.551676 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:19.571676 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:19.534600 20768 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:15:19.576859 12491 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:19.545773 20786 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:19.571971 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:19.644479 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:19.577761 20978 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:19.576472 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:19.563576 20713 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:19.581846 20669 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:19.564872 21004 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.101s
W20260709 12:15:19.625746 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 217636us
W20260709 12:15:19.606096 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81501.4us
I20260709 12:15:19.573896 20431 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.520689 20981 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.523706 20444 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:19.648536 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.577912 20985 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:19.650624 12492 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:19.607457 12494 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:19.660833 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.678468 13259 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:19.679558 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:19.581043 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107674us
W20260709 12:15:19.662482 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.630167 20980 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:19.666224 12488 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:19.667840 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:19.670149 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46464.5us
W20260709 12:15:19.670025 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100456us
I20260709 12:15:19.652189 20713 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.563657 20819 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:19.684039 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130975us
I20260709 12:15:19.683969 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:19.658869 20669 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:19.686853 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.664280 20653 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:15:19.700104 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:19.700945 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.692675 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:19.682967 20980 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:19.709543 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:19.676111 20431 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.691991 20713 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:19.719079 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:19.692571 20669 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:19.690397 20985 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:19.738433 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47684.9us
I20260709 12:15:19.705304 20381 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:19.741010 12570 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
W20260709 12:15:19.742168 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58318us
W20260709 12:15:19.743918 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62561.1us
I20260709 12:15:19.713594 20980 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:19.727489 20995 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.313s	user 0.004s	sys 0.000s
I20260709 12:15:19.754204 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.722771 20768 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:15:19.787560 21023 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:19.758847 20784 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:19.757812 20662 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.758664 20811 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:19.796514 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83037.7us
I20260709 12:15:19.788524 21023 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.793964 12462 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
W20260709 12:15:19.799782 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120294us
I20260709 12:15:19.800076 12462 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:15:19.788228 20653 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:19.803387 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54964.2us
W20260709 12:15:19.803016 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67745.3us
I20260709 12:15:19.828434 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.835322 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.830013 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.806156 21021 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:19.782202 20442 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.809778 20582 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:15:19.810411 21009 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:15:19.812507 20431 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.763391 21015 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.066s
I20260709 12:15:19.823729 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:19.807744 20545 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:19.857499 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:19.781776 21028 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:19.837381 12563 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:19.804627 21023 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:19.836607 20662 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:19.793298 20758 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:19.879164 20758 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:19.829282 21013 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
W20260709 12:15:19.834508 20866 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.006s
I20260709 12:15:19.883139 13262 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:15:19.884796 20786 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.886752 20999 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:19.887447 20999 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:19.888471 20999 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.857709 20545 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:19.889950 20786 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:19.890793 20545 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:19.891654 20678 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.000s
W20260709 12:15:19.827561 21005 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
W20260709 12:15:19.828619 21006 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.205s	user 0.000s	sys 0.000s
I20260709 12:15:19.860291 21028 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.894024 21028 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:19.852766 13253 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260709 12:15:19.826390 21007 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:15:19.846614 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:19.828053 20653 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:19.864130 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87139.5us
W20260709 12:15:19.833420 20473 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.628s	user 0.013s	sys 0.000s
I20260709 12:15:19.866329 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:19.855829 20784 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:19.897379 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.874130 20444 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:19.891989 20545 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:19.827198 21002 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:15:19.850534 21004 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:19.813619 20985 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:19.904101 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:19.908025 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65792us
I20260709 12:15:19.912899 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:19.916589 13262 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:19.918488 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:19.898344 21015 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.900141 20678 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.926561 20662 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:19.926839 21024 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:19.927449 21024 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.928790 20678 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.929819 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:19.932217 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:19.933563 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:19.915758 21004 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.939790 20662 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:19.922161 20866 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:19.908742 20978 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.909339 20713 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.934722 20444 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:19.951079 21004 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:19.908442 20473 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.954056 20866 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:19.958834 20768 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:19.976322 20786 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:15:19.958689 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.623s (SENT) [suppressed 7 similar messages]
I20260709 12:15:19.938854 21015 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.922549 20808 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:19.939214 20758 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:19.984923 12495 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:19.989255 12488 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:19.964818 20381 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:15:19.995838 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 160603us
W20260709 12:15:19.996534 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:19.994267 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 296155us
I20260709 12:15:19.998528 12492 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:15:19.999985 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67923.6us
I20260709 12:15:20.000623 20808 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:20.001749 12570 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:19.967196 21024 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:19.940042 20653 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.006405 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:20.017952 13262 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:20.007462 13258 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:20.010650 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:19.957314 20662 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:19.984494 20768 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:20.022698 20768 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:19.959081 20832 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 12:15:19.985759 20866 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:19.987320 20786 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:20.022337 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63696.7us
W20260709 12:15:20.035941 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.707s (SENT)
I20260709 12:15:20.036386 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:20.032380 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:20.037788 12488 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:15:19.980973 20444 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.052775 12493 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:15:20.050602 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:20.050563 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116951us
I20260709 12:15:20.056573 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:20.026225 20758 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:20.016410 21032 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.038795 20786 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:20.035272 20978 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:20.046139 21038 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:20.077340 12489 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:15:20.071796 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 270836us
I20260709 12:15:20.073091 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:20.025470 20669 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:20.074676 13262 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:20.033942 20808 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:15:20.080719 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:20.089990 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:20.064222 21029 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:20.100581 12497 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:15:20.101588 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:20.103695 20980 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.104332 20713 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.104154 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:20.104738 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:20.105406 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:20.079735 21038 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:20.106877 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:20.107674 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:20.139544 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:20.097325 21021 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:15:20.082036 20381 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
I20260709 12:15:20.101866 20758 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:20.151612 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44240.1us
I20260709 12:15:20.105835 20811 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:20.147495 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:20.148070 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52523.6us
I20260709 12:15:20.157158 21034 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:20.110487 21038 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:20.162215 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63864.5us
W20260709 12:15:20.162923 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 127016us
I20260709 12:15:20.076924 21032 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.148681 20713 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:20.170025 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:20.157397 21031 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.086s
I20260709 12:15:20.149613 21029 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:20.158291 20808 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:20.167457 20909 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.958s	user 0.011s	sys 0.008s
I20260709 12:15:20.181535 12496 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:20.183058 12488 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 9. Preceding OpId from leader: term: 27 index: 10. (index mismatch)
I20260709 12:15:20.165552 20662 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.167338 21034 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:20.166391 20811 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.176887 20758 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.200294 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:20.200798 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:20.193560 20473 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.206557 20669 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:20.208671 20779 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:20.238314 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:20.238056 21034 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.239023 12645 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:15:20.239491 21017 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.496s	user 0.002s	sys 0.000s
I20260709 12:15:20.240170 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:20.210548 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:20.211035 20819 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:20.211974 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:20.242990 20980 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.243804 20980 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:20.245115 20980 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:20.246294 20980 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.247051 20811 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.249033 20758 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:20.249675 13257 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:15:20.249884 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:20.251324 20662 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:20.251961 20662 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:20.214354 21041 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:15:20.220883 20381 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:15:20.254086 21041 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:15:20.254459 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:20.254590 21041 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:20.255846 21041 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.11, Last appended by leader: 11, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.257565 12566 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:15:20.258126 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:20.253055 20662 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:20.215751 21039 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 12:15:20.262409 21037 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:15:20.197947 21035 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:20.189502 20909 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:20.263391 21035 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.218060 20444 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.265959 13262 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23.
I20260709 12:15:20.266202 20473 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:20.268054 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:20.268357 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.268781 12571 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:20.269201 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:20.269774 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.233482 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:20.270865 12496 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:20.270954 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.271624 12566 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:20.237249 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:20.275595 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:20.276183 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:20.232791 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:20.190981 21027 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:20.201040 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:20.228253 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:20.191287 20808 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:20.228211 12489 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
W20260709 12:15:20.220613 20871 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.582s	user 0.010s	sys 0.002s
I20260709 12:15:20.236210 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:20.241267 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:20.293890 12493 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 19: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:20.270370 12563 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:20.232144 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:20.254653 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:20.240072 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:20.277704 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:20.278034 12611 leader_election.cc:400] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23.
I20260709 12:15:20.297824 12611 leader_election.cc:403] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
W20260709 12:15:20.267436 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57097.8us
I20260709 12:15:20.275666 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:20.280690 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:15:20.288933 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:20.299079 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107169us
I20260709 12:15:20.265272 20909 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.291162 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:20.311160 12570 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:15:20.292742 12639 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:20.303380 20811 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:20.303678 21046 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:20.232882 21048 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:20.304576 20444 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:20.239526 21034 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:20.305975 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51437.2us
I20260709 12:15:20.295419 12640 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:20.317435 12494 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:20.323613 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 164497us
I20260709 12:15:20.283815 20582 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.336962 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.333236 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.344987 12566 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:20.290123 21021 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:20.347759 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115051us
I20260709 12:15:20.348519 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:20.314342 20811 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:20.349812 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:20.362263 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:20.320667 20713 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:15:20.364578 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:20.322158 21046 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:20.371484 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:20.360296 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:20.376037 20866 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.336503 20758 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:20.403043 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86423.1us
I20260709 12:15:20.406831 20866 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:20.407884 20866 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:15:20.313097 20871 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.399729 12568 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:20.378760 12489 raft_consensus.cc:2376] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22. Re-sending same reply.
I20260709 12:15:20.332055 21049 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:15:20.411422 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46531.3us
I20260709 12:15:20.410266 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.368551 20811 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:20.368255 20978 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:20.399708 20999 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:20.412148 21028 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.518s	user 0.011s	sys 0.002s
I20260709 12:15:20.352185 20582 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.349664 21004 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:20.399439 21046 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:20.472755 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:20.416392 21031 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:15:20.478448 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124648us
I20260709 12:15:20.362941 21038 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:20.490079 12570 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:20.421491 20713 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:20.462149 20978 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:20.409385 20786 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.345278 21048 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.505129 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:20.505939 12570 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:20.506789 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:20.514820 21050 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:15:20.347357 21029 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:20.525097 12637 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:20.531724 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:20.534552 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1105) took 1037 ms. Trace:
I20260709 12:15:20.535077 12495 rpcz_store.cc:276] 0709 12:15:19.497098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:19.497253 (+   155us) service_pool.cc:224] Handling call
0709 12:15:19.512513 (+ 15260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:20.007077 (+494564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:20.534307 (+527230us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:20.534491 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080508800}
W20260709 12:15:20.538609 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58175.5us
I20260709 12:15:20.490401 20662 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.491662 21031 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:20.485275 20999 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:20.570663 21024 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:20.502115 21015 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:20.574802 21052 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.575652 20713 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.576587 21040 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.501353 21049 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.503530 20978 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:20.500555 20678 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:20.588863 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57335.3us
I20260709 12:15:20.576786 20758 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23.
I20260709 12:15:20.588328 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:20.587296 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:20.570730 21051 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260709 12:15:20.579666 20545 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.004s
I20260709 12:15:20.589704 21024 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:20.601722 20859 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:15:20.713045 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116864us
I20260709 12:15:20.535137 20784 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:20.713873 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:20.542459 21032 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:20.585660 21031 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:20.674829 13256 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:15:20.481338 21028 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.464586 20808 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:20.674150 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 175286us
I20260709 12:15:20.528900 20444 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:20.586620 20662 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:20.768579 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:20.774627 12567 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:15:20.785279 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:20.790783 12568 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:20.785857 20859 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:20.533430 21021 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:20.793944 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:20.768834 20907 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:20.800607 20784 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:20.828467 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 228687us
I20260709 12:15:20.784623 21024 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:21.045517 21054 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.046200 21054 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.048564 21054 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:21.049166 21054 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:20.792680 21061 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:21.050211 21061 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:21.050581 21061 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:21.051579 21061 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.052996 21061 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:20.842866 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43966.7us
W20260709 12:15:21.054386 21059 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20260709 12:15:20.854650 12497 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:20.776682 20678 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.060433 20678 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.061019 20678 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:20.749842 21052 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:21.062496 20768 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.039s	user 0.012s	sys 0.002s
I20260709 12:15:21.063031 20768 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.063534 21052 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:20.937213 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 171624us
W20260709 12:15:21.064785 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:21.064332 20859 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.065208 20662 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:20.840015 20871 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:21.065930 21071 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:21.066502 21071 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.067335 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.067443 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.068296 12645 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:21.068714 21071 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.069085 21052 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:21.069591 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.070494 12640 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:15:21.071823 21023 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.162s	user 0.010s	sys 0.004s
I20260709 12:15:21.072301 21023 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.067952 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.074556 12646 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:21.078804 21025 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:21.079408 21025 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:21.080338 21025 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.083257 21048 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:21.019686 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 432659us
W20260709 12:15:20.855029 21008 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.114s	user 0.003s	sys 0.003s
W20260709 12:15:21.022917 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 237725us
I20260709 12:15:21.090170 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:21.068753 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.091313 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:20.798842 20653 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:15:20.768718 21049 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:21.035871 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99644.9us
I20260709 12:15:20.839764 21062 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:21.089300 21046 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.562s	user 0.008s	sys 0.000s
I20260709 12:15:20.855566 20999 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.094691 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.068004 12639 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:15:21.092711 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:21.044504 20784 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:21.222476 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.222568 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.234761 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.234125 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3534) took 1967 ms. Trace:
I20260709 12:15:20.804774 21040 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.238967 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.063683 20786 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:21.238473 12380 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145834us
I20260709 12:15:21.247335 12640 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:21.223476 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.249315 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:21.246069 12638 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:15:21.251099 12646 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:21.257546 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.000s	sys 0.009s
W20260709 12:15:21.258253 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.000s	sys 0.010s
I20260709 12:15:21.258458 13253 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:21.258255 12645 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:21.258967 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 727301us
I20260709 12:15:21.243992 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:21.244243 12565 rpcz_store.cc:276] 0709 12:15:19.266590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:19.266812 (+   222us) service_pool.cc:224] Handling call
0709 12:15:19.278087 (+ 11275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:19.278562 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:19.278592 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:19.279182 (+   590us) log.cc:844] Serialized 26 byte log entry
0709 12:15:19.539687 (+260505us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:19.539734 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:19.539816 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:20.578966 (+1039150us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:21.215437 (+636471us) raft_consensus.cc:1685] finished
0709 12:15:21.215468 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:21.234033 (+ 18565us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1452,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":13855616,"thread_start_us":259933,"threads_started":1,"wal-append.queue_time_us":401383}
W20260709 12:15:21.272444 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 181201us
W20260709 12:15:21.286623 12566 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.001s	sys 0.014s
W20260709 12:15:21.298223 12566 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.001s	sys 0.014s
W20260709 12:15:21.300364 12490 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.007s	sys 0.008s
W20260709 12:15:21.300722 12490 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.007s	sys 0.009s
I20260709 12:15:21.301391 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3705) took 1086 ms. Trace:
I20260709 12:15:21.301790 12490 rpcz_store.cc:276] 0709 12:15:20.214908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.222535 (+  7627us) service_pool.cc:224] Handling call
0709 12:15:20.233615 (+ 11080us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:20.233883 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:20.470489 (+236606us) log.cc:844] Serialized 14 byte log entry
0709 12:15:20.720305 (+249816us) log.cc:844] Serialized 14 byte log entry
0709 12:15:21.301025 (+580720us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:21.301078 (+    53us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:21.301130 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:21.301198 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:21.301333 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11349,"prepare.queue_time_us":733306,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":14849,"raft.queue_time_us":171022,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":9122,"spinlock_wait_cycles":27640448,"thread_start_us":923615,"threads_started":3}
I20260709 12:15:21.257967 20678 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:21.303880 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.306252 20662 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:21.261502 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:21.311020 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3698) took 1228 ms. Trace:
I20260709 12:15:21.223599 21004 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.331620 12497 rpcz_store.cc:276] 0709 12:15:20.083033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.103710 (+ 20677us) service_pool.cc:224] Handling call
0709 12:15:21.310732 (+1207022us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:21.310917 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1694702336}
I20260709 12:15:21.235775 21052 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:21.235971 21025 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:21.399892 12568 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
W20260709 12:15:21.327693 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78447.6us
I20260709 12:15:21.403882 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:21.330961 12639 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:21.314949 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.316707 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:21.334918 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131377us
I20260709 12:15:21.339910 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:21.234103 21038 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.243005 21071 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.321465 20582 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.332633 20662 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:21.354105 20786 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:21.354211 21048 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260709 12:15:21.246447 21046 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.340176 20678 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.102571 20907 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.362365 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.415798 12644 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:21.363502 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.364323 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.416813 12642 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:21.365514 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.370782 12565 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:15:21.418924 12646 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:21.332350 21028 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.377381 12569 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:21.318902 21041 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:21.430025 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.431344 12490 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 25 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:15:21.381021 21061 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:21.381074 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:21.384516 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1092) took 1154 ms. Trace:
W20260709 12:15:21.388671 20978 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.591s	user 0.009s	sys 0.000s
I20260709 12:15:21.390326 21022 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.380892 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:21.415215 20662 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:21.395129 21029 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:21.395524 21024 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.397120 21064 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:15:21.398001 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:21.438501 20980 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.861s	user 0.018s	sys 0.018s
I20260709 12:15:21.203549 20871 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.120s
I20260709 12:15:21.406390 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.408679 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:21.396727 21031 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.550s	user 0.006s	sys 0.003s
I20260709 12:15:21.406848 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:21.313581 20811 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.827s	user 0.006s	sys 0.004s
I20260709 12:15:21.463637 21004 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.435750 12637 rpcz_store.cc:276] 0709 12:15:20.229846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.230066 (+   220us) service_pool.cc:224] Handling call
0709 12:15:21.384158 (+1154092us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:21.384383 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":642060160}
I20260709 12:15:21.442579 13257 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:21.392726 12495 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:15:21.414983 20442 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:15:21.432099 21068 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.076s	user 0.000s	sys 0.007s
W20260709 12:15:21.432297 21075 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.075s	user 0.001s	sys 0.006s
I20260709 12:15:21.398193 21023 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:21.464586 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.472750 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:21.416234 21071 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:21.478682 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70446.5us
I20260709 12:15:21.489079 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:21.494974 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89011.7us
W20260709 12:15:21.476270 20980 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.925s	user 0.018s	sys 0.020s
I20260709 12:15:21.393077 21083 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:21.448309 21061 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.444994 20978 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.490729 20713 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:21.453461 21022 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.514917 12644 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:21.518890 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:21.489173 21028 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.531189 12645 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:21.553140 12645 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:15:21.504967 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 230463us
I20260709 12:15:21.554731 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:21.555704 21021 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:21.558082 21023 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:21.500569 20913 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:21.560626 21028 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.569599 20913 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:21.572631 20913 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:21.534570 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 182441us
I20260709 12:15:21.545359 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3554) took 1199 ms. Trace:
I20260709 12:15:21.582861 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:21.583664 12488 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:21.576630 12572 rpcz_store.cc:276] 0709 12:15:20.345577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.355443 (+  9866us) service_pool.cc:224] Handling call
0709 12:15:20.399808 (+ 44365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:20.485473 (+ 85665us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:20.485507 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:20.534696 (+ 49189us) log.cc:844] Serialized 26 byte log entry
0709 12:15:20.815165 (+280469us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:20.815230 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:20.815341 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:21.545018 (+729677us) raft_consensus.cc:1685] finished
0709 12:15:21.545051 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:21.545267 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":73414,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":46329,"thread_start_us":101092,"threads_started":1,"wal-append.queue_time_us":802819}
I20260709 12:15:21.490852 21031 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:21.603101 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:21.541159 20768 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.504812 20811 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.602324 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.547573 20713 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:21.623607 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:21.616012 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.616991 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.619442 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.626269 12645 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:15:21.629776 12645 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: [26.16-26.16]
W20260709 12:15:21.615393 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63989.6us
I20260709 12:15:21.517886 21004 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.624296 12489 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:21.651202 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.629160 20713 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:21.629261 21049 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:21.631729 21021 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.632480 21062 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:21.595414 20784 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.613029 20980 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:15:21.678869 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.648415 20381 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:21.700519 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47901.1us
I20260709 12:15:21.661480 21073 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.627677 21029 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
W20260709 12:15:21.673089 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95943.3us
I20260709 12:15:21.626446 21064 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:15:21.709528 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:21.703509 20678 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.642095 20582 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:15:21.717478 12488 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:21.719583 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143673us
W20260709 12:15:21.721369 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59755.3us
I20260709 12:15:21.702013 20442 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:21.751657 20442 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:21.726548 21089 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
W20260709 12:15:21.753095 20669 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.001s	user 0.019s	sys 0.065s
I20260709 12:15:21.721254 20907 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.708088 20381 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.759577 20381 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:21.734447 21086 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:15:21.738145 20582 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:21.761300 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:21.761576 20582 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:21.762790 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:21.735625 21076 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:15:21.775375 12490 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:21.743465 20768 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:15:21.744114 20811 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.744256 21024 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.746078 21004 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.703372 21040 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:21.749749 21099 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:21.716197 21016 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:21.759445 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:21.762547 20582 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.763236 21061 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.764398 21071 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.706267 21046 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.765738 20978 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.766739 21031 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.718382 21073 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.808581 21016 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.810571 20582 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:21.770567 12639 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:21.811443 21073 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:21.742143 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:21.812649 12497 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:21.813858 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.814729 12497 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:21.815708 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:21.742923 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:21.816589 21053 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.146s	user 0.016s	sys 0.015s
I20260709 12:15:21.817016 12572 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:21.774514 12489 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:21.819319 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.820015 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:21.820935 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.768256 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
W20260709 12:15:21.783216 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59852.5us
I20260709 12:15:21.772307 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:21.741452 21034 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.390s	user 0.010s	sys 0.001s
I20260709 12:15:21.729285 21052 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:21.788866 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3556) took 1442 ms. Trace:
I20260709 12:15:21.825297 12637 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:21.835114 12642 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:21.768311 20784 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.805119 12494 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:21.821962 21097 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:21.816480 12497 raft_consensus.cc:2368] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:21.824159 12571 rpcz_store.cc:276] 0709 12:15:20.346312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.379768 (+ 33456us) service_pool.cc:224] Handling call
0709 12:15:20.415475 (+ 35707us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:20.497405 (+ 81930us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:20.497449 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:20.509474 (+ 12025us) log.cc:844] Serialized 26 byte log entry
0709 12:15:21.783144 (+1273670us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:21.788121 (+  4977us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b180037c900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1
0709 12:15:21.788422 (+   301us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:21.788487 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:21.788573 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:21.788620 (+    47us) raft_consensus.cc:1685] finished
0709 12:15:21.788632 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:21.788796 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37065,"raft.queue_time_us":41472,"raft.run_cpu_time_us":5915,"raft.run_wall_time_us":1282451,"spinlock_wait_cycles":104860288,"thread_start_us":32657,"threads_started":1,"wal-append.queue_time_us":281734,"wal-append.run_cpu_time_us":2057,"wal-append.run_wall_time_us":1007152}
I20260709 12:15:21.827601 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:21.823204 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50995.3us
I20260709 12:15:21.832405 21022 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:21.707056 21021 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.832980 12489 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
W20260709 12:15:21.753547 20669 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.019s	sys 0.065s
I20260709 12:15:21.803831 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.918078 12488 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:21.919677 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.713714 21062 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.865248 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:21.922726 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57592.2us
I20260709 12:15:21.924003 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:21.924312 21062 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:21.872120 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 162683us
I20260709 12:15:21.802366 21028 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:21.926221 21028 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:21.926949 21028 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:21.927093 20978 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:21.928177 21028 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:21.929840 21064 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:21.868737 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:21.888036 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:21.901506 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.824383 21052 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:21.914243 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1178) took 1772 ms. Trace:
I20260709 12:15:21.915159 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.915795 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:21.954113 12489 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:21.948983 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80311.3us
I20260709 12:15:21.949009 12490 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:22.014540 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.015776 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 20.
I20260709 12:15:21.921273 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.025061 12488 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:21.796025 20662 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:21.951619 13260 rpcz_store.cc:276] 0709 12:15:20.104111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.150241 (+ 46130us) service_pool.cc:224] Handling call
0709 12:15:20.151956 (+  1715us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:20.154018 (+  2062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:20.154049 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:20.157312 (+  3263us) log.cc:844] Serialized 52 byte log entry
0709 12:15:20.357824 (+200512us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:20.357889 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:20.357996 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:21.815831 (+1457835us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:21.845070 (+ 29239us) raft_consensus.cc:1685] finished
0709 12:15:21.845094 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:21.876428 (+ 31334us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4646,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":5402,"spinlock_wait_cycles":106278144,"thread_start_us":199874,"threads_started":1,"wal-append.queue_time_us":362982}
I20260709 12:15:22.099979 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:21.904426 21025 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:22.104568 12637 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:15:22.121438 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.122462 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:22.132158 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.072950 21052 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:22.132936 12642 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:22.134047 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.133694 21052 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.12, Last appended by leader: 12, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.134850 12637 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:22.139292 21021 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:22.158286 21022 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:22.159117 21022 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.075119 21062 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:22.163646 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.167088 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:22.186640 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:22.187076 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 258182us
I20260709 12:15:22.187347 12490 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:22.163434 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:22.188006 12497 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:22.190398 20999 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:22.188284 21022 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:22.198143 21099 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:22.203835 21099 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.198535 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:22.200367 21025 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:22.191769 21061 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:22.193308 21062 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:22.219117 20662 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:22.223229 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:22.233402 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.234659 12571 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:22.232475 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:22.235265 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 304443us
I20260709 12:15:22.213482 21061 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:22.201541 20999 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:22.241088 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1173) took 2202 ms (client timeout 2584 ms). Trace:
W20260709 12:15:22.242935 13253 rpcz_store.cc:269] 0709 12:15:20.035716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.035877 (+   161us) service_pool.cc:224] Handling call
0709 12:15:22.238253 (+2202376us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54004b73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:22.238522 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2729840000}
I20260709 12:15:22.239708 21104 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:22.247776 21099 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.242172 21061 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:22.251214 13258 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:22.226380 21025 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.229837 21031 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:22.255245 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.254990 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1135) took 2002 ms. Trace:
I20260709 12:15:22.242884 21098 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 12:15:22.262102 13258 rpcz_store.cc:276] 0709 12:15:20.252546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.277033 (+ 24487us) service_pool.cc:224] Handling call
0709 12:15:22.239797 (+1962764us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:22.251763 (+ 11966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.253502 (+  1739us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54004b73d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:22.254891 (+  1389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4452066560}
W20260709 12:15:22.254359 21041 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.793s	user 0.012s	sys 0.001s
I20260709 12:15:22.270702 21025 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:22.271863 21064 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:22.272342 21064 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:22.272773 21064 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:22.274284 20713 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.002s
I20260709 12:15:22.274772 20713 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.264479 21031 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.282562 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:22.278694 12536 connection.cc:256] RPC call timeout handler was delayed by -4.2136902439999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20260709 12:15:22.283527 12536 connection.cc:256] RPC call timeout handler was delayed by -4.1065902440000173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260709 12:15:22.284017 12536 connection.cc:256] RPC call timeout handler was delayed by -3.9346902439999667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
I20260709 12:15:22.284193 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:22.284858 12536 connection.cc:256] RPC call timeout handler was delayed by -3.8707902439998634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2255s for a response
W20260709 12:15:22.285387 12536 connection.cc:256] RPC call timeout handler was delayed by -3.5350902440000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26280000000000003s for a response
W20260709 12:15:22.286757 12536 connection.cc:256] RPC call timeout handler was delayed by -3.1741902439998739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3029s for a response
W20260709 12:15:22.287218 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8591102350001165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260709 12:15:22.287683 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8312102350000714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
W20260709 12:15:22.288091 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7736102350002056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
W20260709 12:15:22.288540 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5531102350000765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.205s for a response
W20260709 12:15:22.288926 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4280102350001016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
W20260709 12:15:22.290555 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102787us
I20260709 12:15:22.296119 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.296892 12642 raft_consensus.cc:2376] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23. Re-sending same reply.
I20260709 12:15:22.300623 12637 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:22.292829 21098 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:22.290174 20913 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:22.316535 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.318181 12639 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:22.303175 21098 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:22.322212 12490 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:22.345677 21098 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:22.350236 21031 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.354562 20999 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.294843 21062 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.9, Last appended by leader: 9, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.316710 20713 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:22.305737 20582 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
W20260709 12:15:22.382614 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:22.429152 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:22.430105 12492 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:22.434872 21098 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.435683 21041 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:22.436331 21041 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.437252 21098 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.439208 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:22.384158 20832 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:15:22.497159 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:22.438861 21110 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:15:22.500651 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.504256 12642 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:22.430359 21035 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:22.515964 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.517206 12491 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:22.497953 20832 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:22.442219 20582 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:15:22.554221 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.555838 12489 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:22.557248 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.558426 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.557955 12566 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:22.561977 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.563859 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:22.564630 12489 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:22.567854 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:22.571084 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.571888 12564 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:22.575707 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:22.577154 12563 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:22.589445 12491 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:22.615120 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.618083 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.618377 12567 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:22.616951 21090 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:22.618726 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:22.626569 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.634192 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.634967 12494 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:22.635865 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:22.568267 20582 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:15:22.637753 12564 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:22.579895 20832 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.617151 21028 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:22.623513 21087 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.913s	user 0.008s	sys 0.008s
I20260709 12:15:22.636766 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.637274 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:22.647917 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.622598 21090 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:22.658603 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.680604 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:15:22.663220 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106829us
I20260709 12:15:22.682142 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:15:22.683472 12494 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:22.688735 12563 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:22.639977 21022 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:22.638083 21031 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:22.702757 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:22.703442 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.707305 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.708207 12488 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:22.714999 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.716071 12642 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:22.718541 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:22.719177 21112 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:15:22.657913 21028 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.720780 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:22.721020 20582 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:22.721916 20582 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:22.722864 21028 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.723152 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.723574 21028 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:22.724157 21028 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:22.724253 12564 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:22.724344 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.725600 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.727211 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.729588 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.729930 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.726420 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.730594 12644 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:22.731670 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.732822 12563 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:15:22.733207 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.733728 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.735080 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.735945 12563 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:22.736073 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.737795 20832 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:22.738806 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:22.641716 20913 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:22.738831 12490 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:22.709292 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.744143 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.744822 12488 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:22.748479 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111292us
I20260709 12:15:22.755890 12492 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:22.766897 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.767776 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.768088 12638 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:15:22.739313 21107 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:15:22.772291 12564 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:22.773438 12490 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [21.14-23.15]
I20260709 12:15:22.774552 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3721) took 1184 ms. Trace:
I20260709 12:15:22.774984 12490 rpcz_store.cc:276] 0709 12:15:21.589647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.750590 (+1160943us) service_pool.cc:224] Handling call
0709 12:15:22.773047 (+ 22457us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:22.774155 (+  1108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.774473 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.785032 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:22.785954 12488 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:22.786463 12488 raft_consensus.cc:614] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:22.769469 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:22.792330 12640 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:15:22.788727 12494 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:22.793356 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3725) took 1152 ms. Trace:
I20260709 12:15:22.793957 12494 rpcz_store.cc:276] 0709 12:15:21.640797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.746722 (+1105925us) service_pool.cc:224] Handling call
0709 12:15:22.793261 (+ 46539us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.799592 12644 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:22.790611 12491 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:22.808597 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.752707 20913 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.816403 21090 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.818480 21028 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.821362 12563 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:15:22.829480 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.831581 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:22.834089 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.737802 21061 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.747481 21022 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:22.857331 12564 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:22.863291 12644 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:22.863722 12640 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:22.864506 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.882354 12563 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 19.
I20260709 12:15:22.860750 20444 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:22.884846 21056 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.284s	user 0.013s	sys 0.020s
I20260709 12:15:22.885771 20768 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:22.885100 20444 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.886420 20768 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.888944 20768 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.891737 20913 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:22.892297 21022 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:22.837782 21064 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.565s	user 0.011s	sys 0.003s
I20260709 12:15:22.893841 21064 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:22.896375 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.897156 21064 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:22.840198 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:22.837574 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.878121 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:22.839025 12642 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:15:22.875499 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.915562 12488 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: [24.12-26.13]
I20260709 12:15:22.899358 12646 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:22.899533 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.918290 12563 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23.
I20260709 12:15:22.917711 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3727) took 1162 ms. Trace:
I20260709 12:15:22.913343 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:22.919600 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:22.918601 12488 rpcz_store.cc:276] 0709 12:15:21.754897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.874685 (+1119788us) service_pool.cc:224] Handling call
0709 12:15:22.915315 (+ 40630us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:22.917359 (+  2044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.917651 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.914299 12495 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:15:22.892071 12637 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260709 12:15:22.909886 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69800.7us
W20260709 12:15:22.920372 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82900us
I20260709 12:15:22.897567 12567 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:22.921659 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3729) took 1166 ms. Trace:
W20260709 12:15:22.926400 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 205748us
I20260709 12:15:22.926301 12495 rpcz_store.cc:276] 0709 12:15:21.755595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.892927 (+1137332us) service_pool.cc:224] Handling call
0709 12:15:22.893307 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:22.921321 (+ 28014us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.921573 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.927896 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:22.928072 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.933580 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.936658 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.938632 12495 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:22.938839 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:22.939769 12563 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:22.900161 21061 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:22.930154 21073 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:15:22.944769 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 206647us
I20260709 12:15:22.925917 21090 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:22.931739 20653 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:22.945801 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2905) took 1138 ms. Trace:
I20260709 12:15:22.950090 12495 rpcz_store.cc:276] 0709 12:15:21.803128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.927193 (+1124065us) service_pool.cc:224] Handling call
0709 12:15:22.938360 (+ 11167us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:22.941064 (+  2704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.941388 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.956768 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:22.959350 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1134) took 2841 ms. Trace:
I20260709 12:15:22.958235 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:22.967675 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3732) took 1097 ms. Trace:
I20260709 12:15:22.968433 12495 rpcz_store.cc:276] 0709 12:15:21.870094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.950895 (+1080801us) service_pool.cc:224] Handling call
0709 12:15:22.966879 (+ 15984us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:22.967135 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.967174 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:22.967233 (+    59us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:22.967308 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.967422 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:22.967586 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.959822 13261 rpcz_store.cc:276] 0709 12:15:20.110176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.196954 (+ 86778us) service_pool.cc:224] Handling call
0709 12:15:20.230532 (+ 33578us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:20.254900 (+ 24368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:20.254931 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:20.255555 (+   624us) log.cc:844] Serialized 52 byte log entry
0709 12:15:20.402932 (+147377us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:20.402979 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:20.403047 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:22.884161 (+2481114us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:22.944373 (+ 60212us) raft_consensus.cc:1685] finished
0709 12:15:22.944396 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:22.951521 (+  7125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8334,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":114,"spinlock_wait_cycles":96951040,"thread_start_us":146947,"threads_started":1,"wal-append.queue_time_us":235320}
I20260709 12:15:22.947916 21073 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:23.009889 21073 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:22.992242 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65032.1us
I20260709 12:15:23.010327 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.010957 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:23.012295 12489 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:23.012584 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:22.997449 21061 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:23.013880 21061 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:23.016960 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.018289 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.018432 12563 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:23.008188 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1005) took 2507 ms. Trace:
I20260709 12:15:22.947160 20582 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:22.957656 20978 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:23.013072 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2917) took 1085 ms. Trace:
I20260709 12:15:23.033787 12489 rpcz_store.cc:276] 0709 12:15:21.927358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.011550 (+1084192us) service_pool.cc:224] Handling call
0709 12:15:23.011904 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.012775 (+   871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.012979 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:23.035842 12489 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:23.019547 13254 rpcz_store.cc:276] 0709 12:15:20.500999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:20.501758 (+   759us) service_pool.cc:224] Handling call
0709 12:15:20.515103 (+ 13345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:22.067367 (+1552264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.067400 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:22.067442 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:22.067469 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.007864 (+940395us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:23.007927 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.008102 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3569223936}
I20260709 12:15:23.036762 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2919) took 1100 ms. Trace:
I20260709 12:15:23.037381 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.037235 12489 rpcz_store.cc:276] 0709 12:15:21.936566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.035072 (+1098506us) service_pool.cc:224] Handling call
0709 12:15:23.035564 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.036399 (+   835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.036689 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:23.038383 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:23.039177 12489 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:15:23.035355 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3733) took 1165 ms. Trace:
I20260709 12:15:23.056792 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2922) took 1119 ms. Trace:
I20260709 12:15:23.056805 12495 rpcz_store.cc:276] 0709 12:15:21.870387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.969757 (+1099370us) service_pool.cc:224] Handling call
0709 12:15:23.034610 (+ 64853us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.034863 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:23.034899 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:23.034961 (+    62us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:23.035018 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.035110 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.035277 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:23.057883 12489 rpcz_store.cc:276] 0709 12:15:21.937377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.038511 (+1101134us) service_pool.cc:224] Handling call
0709 12:15:23.038939 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:23.056233 (+ 17294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.056710 (+   477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:22.960261 20653 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:23.067384 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.092439 12563 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:15:23.094040 12495 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:15:23.020018 21031 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:23.096537 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 2926) took 1158 ms. Trace:
I20260709 12:15:23.039832 20784 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:23.039368 20832 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:23.097570 12495 rpcz_store.cc:276] 0709 12:15:21.938586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.058868 (+1120282us) service_pool.cc:224] Handling call
0709 12:15:23.093437 (+ 34569us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:23.094639 (+  1202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.096452 (+  1813us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:23.032682 21122 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:23.045261 21061 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:23.100409 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 172616us
I20260709 12:15:23.091135 21025 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:23.040134 21022 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:15:23.135689 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.089111 20978 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.105099 21122 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:23.103231 20784 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:23.141005 12495 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
W20260709 12:15:23.135636 21023 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.004s	sys 0.017s
I20260709 12:15:23.147218 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:23.144402 21122 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.158386 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.159315 12567 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:23.145642 20784 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:23.172243 12570 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 10. Preceding OpId from leader: term: 27 index: 10. (term mismatch)
I20260709 12:15:23.181237 12570 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:15:23.181618 12570 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 26.10
I20260709 12:15:23.181919 12570 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:23.173889 21041 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:23.186240 12568 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:15:23.187880 12568 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:23.147006 21118 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:23.193188 12570 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:23.142657 20653 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.197813 21118 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.202241 20978 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.204634 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1096) took 1555 ms. Trace:
I20260709 12:15:23.205359 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3724) took 1591 ms. Trace:
W20260709 12:15:23.205948 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57384.9us
I20260709 12:15:23.206290 12490 rpcz_store.cc:276] 0709 12:15:21.613908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.830243 (+1216335us) service_pool.cc:224] Handling call
0709 12:15:22.830823 (+   580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:22.831151 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.831198 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:22.831856 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:15:22.941709 (+109853us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:22.941762 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:22.941849 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:23.205120 (+263271us) raft_consensus.cc:1685] finished
0709 12:15:23.205141 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.205301 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":109273,"threads_started":1,"wal-append.queue_time_us":308382}
I20260709 12:15:23.211097 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.212020 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3734) took 1341 ms. Trace:
I20260709 12:15:23.211797 21073 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.212491 12497 rpcz_store.cc:276] 0709 12:15:21.870736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.736313 (+865577us) service_pool.cc:224] Handling call
0709 12:15:23.210924 (+474611us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:15:23.211960 (+  1036us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28239744}
I20260709 12:15:23.214715 12643 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:23.205219 12645 rpcz_store.cc:276] 0709 12:15:21.649502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:21.658939 (+  9437us) service_pool.cc:224] Handling call
0709 12:15:21.659292 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:21.659624 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:21.659654 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:22.716367 (+1056713us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:15:22.716943 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 12:15:22.951038 (+234095us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:23.031788 (+ 80750us) log.cc:844] Serialized 14 byte log entry
0709 12:15:23.036232 (+  4444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.036336 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:23.204161 (+167825us) raft_consensus.cc:1685] finished
0709 12:15:23.204313 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.204539 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11341,"prepare.queue_time_us":88833,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":2022,"raft.queue_time_us":89678,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":38414,"spinlock_wait_cycles":7558912,"thread_start_us":233667,"threads_started":1,"wal-append.queue_time_us":393719}
I20260709 12:15:23.216012 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.216512 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.218024 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.218992 12644 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:23.143097 21025 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.220504 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.222198 12640 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:23.222517 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:23.222621 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:23.223110 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:23.223868 12640 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:15:23.224551 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:23.224728 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:23.222103 21025 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.225833 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.222529 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.228675 21025 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:23.229662 21137 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:15:23.231190 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3726) took 1569 ms. Trace:
I20260709 12:15:23.229324 21025 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:23.231617 12491 rpcz_store.cc:276] 0709 12:15:21.662098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.858292 (+1196194us) service_pool.cc:224] Handling call
0709 12:15:22.858834 (+   542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:22.868905 (+ 10071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.868953 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:22.883300 (+ 14347us) log.cc:844] Serialized 26 byte log entry
0709 12:15:23.108852 (+225552us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:23.108898 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.108981 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:23.230940 (+121959us) raft_consensus.cc:1685] finished
0709 12:15:23.230968 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.231126 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9257,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":4196,"thread_start_us":167176,"threads_started":1,"wal-append.queue_time_us":248127}
I20260709 12:15:23.233557 20859 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.217445 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.223977 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.219940 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:23.237130 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1183) took 1858 ms (client timeout 2160 ms). Trace:
I20260709 12:15:23.236423 12644 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
W20260709 12:15:23.238124 13262 rpcz_store.cc:269] 0709 12:15:21.378795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:21.380803 (+  2008us) service_pool.cc:224] Handling call
0709 12:15:23.236852 (+1856049us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:23.237035 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18782080}
I20260709 12:15:23.243887 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.244766 12644 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:23.246838 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:23.246937 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.248889 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:23.249763 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.250511 12644 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:23.233497 21025 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.254352 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.254124 21031 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:23.262068 12643 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:23.262599 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:23.263794 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.264954 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.265292 12637 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:23.265478 12645 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.266388 21073 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:23.266945 12638 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:23.268013 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.270035 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.272810 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.269347 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.266449 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:23.275436 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 339283us
I20260709 12:15:23.276362 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:23.279470 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.280630 12640 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:23.280615 12645 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:23.293784 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.296838 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:23.299157 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.295102 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.301836 12644 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:23.302750 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:23.298025 20653 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:23.307852 20653 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:23.308166 21055 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.637s	user 0.016s	sys 0.030s
I20260709 12:15:23.308441 20653 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:23.310039 21073 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:23.305037 21144 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:23.311689 21144 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:23.312105 21144 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:23.312466 21031 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:23.312134 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.319914 21027 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.320998 21027 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.321080 12609 leader_election.cc:400] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23.
I20260709 12:15:23.323841 20784 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.325263 12496 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:23.326772 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:23.319783 20444 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.328106 20444 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:23.328554 20444 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:23.329348 20978 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:23.329485 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.330574 20713 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:15:23.332659 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54086.6us
I20260709 12:15:23.334504 20442 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.344470 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:23.346127 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.341984 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.346810 12637 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24. Re-sending same reply.
I20260709 12:15:23.348788 12639 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:15:23.328958 20444 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.351719 20653 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.351796 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.354530 12640 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:23.355544 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:23.337306 21141 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.140s	user 0.003s	sys 0.000s
I20260709 12:15:23.360356 20653 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:23.364260 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:23.357796 20859 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:23.369717 20859 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:23.370791 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:23.370775 20859 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.371943 21118 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:23.342303 21073 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:23.373024 21073 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.379083 12493 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:23.379920 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3715) took 1792 ms. Trace:
I20260709 12:15:23.380242 12493 rpcz_store.cc:276] 0709 12:15:21.587588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:21.588032 (+   444us) service_pool.cc:224] Handling call
0709 12:15:23.378625 (+1790593us) spinlock_profiling.cc:243] Waited 8.5 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:23.378886 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.379626 (+   740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.379853 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19553024}
I20260709 12:15:23.365595 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.382475 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.384932 12568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:23.375994 21073 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.382908 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.383846 21031 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260709 12:15:23.385807 20999 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:23.393565 20978 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:23.394150 20978 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:23.394510 20978 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:23.394636 21122 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.397171 21122 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.395133 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3730) took 1639 ms. Trace:
I20260709 12:15:23.398949 12488 rpcz_store.cc:276] 0709 12:15:21.755950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.920723 (+1164773us) service_pool.cc:224] Handling call
0709 12:15:22.936675 (+ 15952us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:22.949152 (+ 12477us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.949203 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:22.961322 (+ 12119us) log.cc:844] Serialized 26 byte log entry
0709 12:15:23.098539 (+137217us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:15:23.098599 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.098716 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:23.336487 (+237771us) spinlock_profiling.cc:243] Waited 8.72 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:23.394764 (+ 58277us) raft_consensus.cc:1685] finished
0709 12:15:23.394806 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.395063 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5205,"raft.queue_time_us":5366,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":20108,"spinlock_wait_cycles":20066560,"thread_start_us":92408,"threads_started":1,"wal-append.queue_time_us":190943}
I20260709 12:15:23.405018 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:23.408016 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43883.5us
I20260709 12:15:23.401113 21022 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.414207 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:23.421741 12491 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:23.420091 20473 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:23.426069 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.427431 12495 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:23.434289 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:23.435365 12497 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:23.436367 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.437059 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:23.407931 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.443480 12488 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:23.446913 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:23.447849 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.448531 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:23.452277 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.410113 21022 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.453145 12570 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:23.456241 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:23.458777 21124 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.013s
I20260709 12:15:23.460170 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.471661 12564 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:23.462723 12536 leader_election.cc:400] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:23.460175 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.473213 12536 leader_election.cc:403] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:23.465487 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.475091 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.479139 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.479976 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:23.482451 12568 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:23.483001 12568 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:23.462306 21031 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 12:15:23.484737 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.485169 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.486049 12564 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:15:23.458905 21096 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:23.487073 20832 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:23.449810 21028 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:15:23.449476 20442 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:15:23.469100 20784 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.479661 20999 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:23.506291 12570 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:15:23.486148 12568 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.509395 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:23.509850 12568 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:23.482611 21122 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:23.478246 21124 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:15:23.464641 21144 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.490422 21096 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:23.479869 21025 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:15:23.507085 21088 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.514752 12570 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:23.519273 12564 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:15:23.520850 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:23.544646 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64190.3us
I20260709 12:15:23.551107 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:23.555879 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:23.565337 12570 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:15:23.566890 12570 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:23.567225 12570 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:15:23.567595 12570 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:23.567824 21090 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.568403 21090 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.569872 21161 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:23.570952 21090 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.574734 21163 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:23.568436 20978 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:23.577368 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56659.6us
I20260709 12:15:23.523226 20473 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:23.536921 21096 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:23.580159 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.524569 21022 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.582387 12570 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:15:23.584712 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:23.581795 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:23.586301 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.586613 13262 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:23.583680 12564 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:23.586905 12570 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.586998 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:23.589242 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:23.589141 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:23.590574 13262 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:15:23.592188 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:23.517467 20832 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:15:23.538882 20444 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.610867 13262 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:23.611399 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.613758 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.599752 13258 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:23.599469 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.616062 12564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:23.582448 21144 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:23.597013 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:23.606734 12568 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 13. (term mismatch)
I20260709 12:15:23.563339 21088 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.614723 12641 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26. Re-sending same reply.
I20260709 12:15:23.622702 12568 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 12
I20260709 12:15:23.623013 12568 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 26.13
I20260709 12:15:23.623312 12568 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:23.622233 13254 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:23.623464 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.624222 12637 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.624517 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.598928 20999 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:23.627452 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59227us
I20260709 12:15:23.627805 12641 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [21.11-22.12]
I20260709 12:15:23.630183 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.631148 12644 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:23.589313 21163 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:23.640328 21163 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:23.640861 21163 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:23.641341 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3723) took 2039 ms. Trace:
I20260709 12:15:23.641868 12492 rpcz_store.cc:276] 0709 12:15:21.601881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.771021 (+1169140us) service_pool.cc:224] Handling call
0709 12:15:22.771515 (+   494us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:22.916194 (+144679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.916227 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:22.916725 (+   498us) log.cc:844] Serialized 26 byte log entry
0709 12:15:23.020834 (+104109us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:23.087398 (+ 66564us) log.cc:844] Serialized 14 byte log entry
0709 12:15:23.087725 (+   327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.640891 (+553166us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:23.640966 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:23.641014 (+    48us) raft_consensus.cc:1685] finished
0709 12:15:23.641113 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.641274 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9476,"prepare.queue_time_us":66286,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":38,"raft.queue_time_us":10767,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":18489,"spinlock_wait_cycles":331729792,"thread_start_us":90456,"threads_started":1,"wal-append.queue_time_us":205595}
I20260709 12:15:23.604866 21161 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:23.634370 20999 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.636363 21159 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:23.625218 21031 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:23.639057 21022 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.632725 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:23.636283 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.655514 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.657823 12637 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:23.709698 12568 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:15:23.703981 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:23.711429 12493 raft_consensus.cc:606] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:23.711814 12493 raft_consensus.cc:614] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:23.691658 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:23.700500 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48076.2us
I20260709 12:15:23.713001 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:23.714444 20473 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:15:23.718159 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:23.719579 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.721534 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:23.725983 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1120) took 1786 ms. Trace:
I20260709 12:15:23.727082 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.728173 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:23.651010 21155 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:23.739571 12570 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:23.732118 12568 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:23.741995 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:23.746501 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159573us
I20260709 12:15:23.702368 20832 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 25.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:15:23.750346 12644 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:23.757357 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.759828 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.760001 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:23.729724 21004 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.768298 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:23.773422 12641 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:23.733721 21021 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.772660 12637 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:15:23.729408 21031 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:23.777522 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.778921 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:23.782629 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.784320 21021 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.789806 20859 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:23.791741 13258 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:15:23.792065 21031 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.792717 21088 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.724437 21022 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.800637 13254 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:23.784724 21158 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:23.810671 21100 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:23.811318 21100 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:23.812377 21100 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.816434 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3567) took 1591 ms. Trace:
I20260709 12:15:23.817039 12565 rpcz_store.cc:276] 0709 12:15:22.224655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.226213 (+  1558us) service_pool.cc:224] Handling call
0709 12:15:22.233409 (+  7196us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:22.292175 (+ 58766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:22.292211 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:22.296925 (+  4714us) log.cc:844] Serialized 26 byte log entry
0709 12:15:22.684363 (+387438us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:22.684410 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.815851 (+1131441us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:23.815916 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:23.815985 (+    69us) raft_consensus.cc:1685] finished
0709 12:15:23.816005 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.816350 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4234,"raft.queue_time_us":1201,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":2397,"spinlock_wait_cycles":133966720,"thread_start_us":362994,"threads_started":1,"wal-append.queue_time_us":398592,"wal-append.run_cpu_time_us":1738,"wal-append.run_wall_time_us":1004326}
I20260709 12:15:23.819492 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.821573 20978 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.762136 21159 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:15:23.781467 21004 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.824118 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.825117 21004 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.828289 21167 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:23.829031 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.776414 21155 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:23.829869 12563 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:23.830382 21159 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:23.830852 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.830596 21155 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:23.832010 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:23.832505 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49942.2us
I20260709 12:15:23.833127 21000 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:23.851092 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:23.851855 12492 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:23.852679 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:23.820125 12565 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 21.
I20260709 12:15:23.834370 20444 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.838057 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.854045 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.838188 21164 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:23.854889 21164 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.855517 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.856792 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:23.829499 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:23.839224 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3713) took 2253 ms. Trace:
I20260709 12:15:23.858072 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.858033 12496 rpcz_store.cc:276] 0709 12:15:21.586240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:21.593229 (+  6989us) service_pool.cc:224] Handling call
0709 12:15:22.736979 (+1143750us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:23.171128 (+434149us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cee9c 00007ffbf26cea5b 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:23.171354 (+   226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.325743 (+154389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.838931 (+513188us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:23.839132 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395765760}
I20260709 12:15:23.825140 12646 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:23.854259 12569 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:23.829559 21016 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.848829 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.861063 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.864624 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:23.861192 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:23.859638 21159 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:23.869326 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 329340us
I20260709 12:15:23.872506 13262 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:23.876206 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.848124 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:23.863936 12496 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:23.857329 21164 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:23.859638 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:23.863874 12492 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:23.875258 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:23.877638 12496 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:23.879876 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.901239 12490 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:23.880745 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.902657 12563 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:23.908710 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:23.910032 12639 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:23.912237 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:23.919641 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43532.2us
I20260709 12:15:23.921020 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.839262 21022 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.941962 12461 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:23.942298 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.942775 12461 leader_election.cc:403] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:15:23.943147 12570 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:23.880035 20978 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:23.916150 12496 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.859908 21000 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:23.947559 12496 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:23.948937 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.903795 21159 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:15:23.955824 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.900415 21160 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:23.983381 12497 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:15:23.986357 12563 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 12:15:23.988098 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:23.876576 21124 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 12:15:23.905279 21118 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:15:23.942198 20784 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:23.942413 21122 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:23.981740 21025 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:23.950564 21034 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:23.950584 21000 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:23.988708 21160 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:23.993918 21021 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:24.009589 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:23.992982 20978 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:24.003183 12490 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:23.976239 21022 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:23.991677 21027 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:23.993711 20981 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:23.953943 21164 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:23.950332 21038 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:15:23.992027 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:23.995613 21166 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:24.015341 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:23.992199 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.955257 21096 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:23.992736 21004 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:15:24.004112 21025 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:24.041291 21025 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:24.041522 21163 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.045186 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:23.987525 21082 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:15:24.030916 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:24.004457 20784 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:24.006803 21034 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:24.085108 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3562) took 1523 ms (client timeout 1685 ms). Trace:
W20260709 12:15:24.085539 12566 rpcz_store.cc:269] 0709 12:15:22.561394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:24.044856 (+1483462us) service_pool.cc:224] Handling call
0709 12:15:24.085038 (+ 40182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:24.002048 21159 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:15:24.006239 20444 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:15:24.012837 21160 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:24.022171 20473 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:24.016639 21124 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:15:24.095970 13257 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:24.039149 13261 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
I20260709 12:15:24.024700 20981 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:24.040156 21096 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:24.097726 20981 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.099229 20981 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:24.067453 21022 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:24.101171 21022 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:24.101591 21022 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:24.108490 20582 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:24.109040 20582 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:24.039882 21122 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.109711 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.110149 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.110548 13255 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:24.040828 21004 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:24.111251 21004 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:24.045861 12564 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:24.052693 20913 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:24.023164 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 276518us
W20260709 12:15:24.109787 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:24.113116 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.055374 21025 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:24.062271 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47003us
I20260709 12:15:24.064280 12565 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:24.119398 12565 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:24.069118 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:24.119951 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.074545 12496 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:15:24.076171 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.121471 13262 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:15:24.076390 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.006592 21032 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
W20260709 12:15:24.091878 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:24.122649 12496 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 17
I20260709 12:15:24.123030 12496 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 12:15:24.006913 21162 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.073s
W20260709 12:15:24.117069 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 270653us
I20260709 12:15:24.119197 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:24.123325 12496 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:24.051847 12569 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:24.162273 12572 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:24.084466 21166 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:15:24.165828 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45051.1us
I20260709 12:15:24.168067 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:24.088698 21163 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:24.160385 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:24.173382 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:23.726629 12489 rpcz_store.cc:276] 0709 12:15:21.939280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.062503 (+1123223us) service_pool.cc:224] Handling call
0709 12:15:23.062883 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.063120 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:15:23.639283 (+576163us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b18003a8fa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:15:23.639495 (+   212us) log.cc:844] Serialized 14 byte log entry
0709 12:15:23.725140 (+ 85645us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:23.725277 (+   137us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:23.725358 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:23.725541 (+   183us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:23.725866 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50813,"prepare.queue_time_us":39754,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":23185,"raft.queue_time_us":42471,"spinlock_wait_cycles":3515776,"thread_start_us":105312,"threads_started":2}
W20260709 12:15:24.173787 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60745.6us
I20260709 12:15:24.230901 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:24.247726 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79758.9us
I20260709 12:15:24.193406 12565 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 24.10, Last appended by leader: 10, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:24.193003 21157 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 12:15:24.309692 12565 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:24.322772 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59530.6us
I20260709 12:15:24.339876 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:24.341624 12569 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:15:24.334564 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:24.372678 12646 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.512s	user 0.011s	sys 0.002s
I20260709 12:15:24.373489 12646 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:24.377604 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:24.379089 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:24.394580 21038 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:24.330962 21028 raft_consensus.cc:2764] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:15:24.398801 20442 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:15:24.450623 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.002s	sys 0.030s
W20260709 12:15:24.470912 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.003s	sys 0.030s
W20260709 12:15:24.471799 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 721094us
I20260709 12:15:24.381880 20669 raft_consensus.cc:2764] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:15:24.146209 20913 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:24.489991 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 301652us
I20260709 12:15:24.494674 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:24.486325 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.018s	sys 0.012s
I20260709 12:15:24.402853 21131 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:15:24.499732 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.019s	sys 0.012s
W20260709 12:15:24.500461 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 542396us
I20260709 12:15:24.497004 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:24.249115 20999 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.190s
I20260709 12:15:24.173354 20978 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.202790 20582 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:24.502179 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64384.1us
W20260709 12:15:24.484385 21099 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.025s	sys 0.025s
W20260709 12:15:24.516571 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209900us
I20260709 12:15:24.438843 21038 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:24.523906 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:24.541474 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:24.469025 20442 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:24.544194 12609 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.685s (SENT)
I20260709 12:15:24.536995 13258 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:24.329972 20784 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.550923 21038 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:15:24.598510 21172 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:15:24.519131 21118 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:24.566069 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:24.566905 21180 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:15:24.508318 21168 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:24.620294 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:24.621032 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:24.621996 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:24.622653 12563 raft_consensus.cc:2376] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24. Re-sending same reply.
I20260709 12:15:24.245543 21034 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:24.623833 21034 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:24.383872 20473 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:24.532012 21028 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:15:24.591630 21179 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:24.506181 21131 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:24.592516 21025 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:24.592625 20582 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:24.593252 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:24.636082 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:24.517311 21121 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:15:24.507661 20662 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:24.608690 12565 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 10. Preceding OpId from leader: term: 25 index: 10. (term mismatch)
I20260709 12:15:24.716332 12565 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:15:24.716682 12565 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 24.10
I20260709 12:15:24.716966 12565 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:24.531675 21099 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.026s	sys 0.025s
I20260709 12:15:24.522164 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:24.622087 21004 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.590174 21082 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:24.591496 21194 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:24.628321 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:24.726966 21004 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:24.727052 21194 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:24.631839 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3569) took 2076 ms. Trace:
I20260709 12:15:24.634791 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:24.639947 21131 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:24.641019 12642 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:15:24.728436 12569 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:24.727861 21194 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:24.729332 12572 rpcz_store.cc:276] 0709 12:15:22.555550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:22.556892 (+  1342us) service_pool.cc:224] Handling call
0709 12:15:23.945136 (+1388244us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:24.161990 (+216854us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:24.162863 (+   873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:24.631511 (+468648us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400044f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:24.631742 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503923840}
W20260709 12:15:24.693500 20981 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.000s
I20260709 12:15:24.693559 20713 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:24.687976 20999 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:24.714131 20662 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:24.758172 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125478us
W20260709 12:15:24.759739 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 237628us
W20260709 12:15:24.766894 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:24.741673 21028 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:24.773204 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:24.767534 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.012s	sys 0.008s
I20260709 12:15:24.744822 21027 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:15:24.787384 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.013s	sys 0.013s
W20260709 12:15:24.788409 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 589454us
W20260709 12:15:24.762014 21122 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.005s	sys 0.013s
I20260709 12:15:24.771236 21118 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:24.805533 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:24.813742 12489 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
W20260709 12:15:24.790866 21000 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.776s	user 0.012s	sys 0.000s
I20260709 12:15:24.815099 21000 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:24.817655 21000 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:24.769134 21197 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:24.774688 21185 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:24.837224 21185 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:24.792259 21122 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.005s	sys 0.016s
I20260709 12:15:24.839130 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:24.840003 12491 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:24.757344 20713 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:24.798524 21168 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:24.848611 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:24.764446 20999 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:24.857561 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:24.827405 21199 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:24.841934 21191 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:15:24.866140 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 237960us
I20260709 12:15:24.789829 21028 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:24.856796 21175 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.363s	user 0.000s	sys 0.002s
I20260709 12:15:24.826908 21196 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:24.801328 21027 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:24.834718 21032 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
W20260709 12:15:24.861433 21022 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.759s	user 0.010s	sys 0.003s
I20260709 12:15:24.859890 21168 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:24.884162 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:24.891142 21027 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:24.899670 12645 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:15:24.842819 20662 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:24.907028 21200 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:15:24.912287 21073 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:24.912837 21073 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:24.913652 21191 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:24.916317 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3425) took 1307 ms. Trace:
I20260709 12:15:24.916800 12640 rpcz_store.cc:276] 0709 12:15:23.608816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.608999 (+   183us) service_pool.cc:224] Handling call
0709 12:15:23.632225 (+ 23226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:23.693352 (+ 61127us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:24.051780 (+358428us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b18002b4c20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:15:24.360012 (+308232us) log.cc:844] Serialized 14 byte log entry
0709 12:15:24.759006 (+398994us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:24.759734 (+   728us) log.cc:844] Serialized 26 byte log entry
0709 12:15:24.759949 (+   215us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:24.760006 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:24.760121 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:24.915976 (+155855us) raft_consensus.cc:1685] finished
0709 12:15:24.916102 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:24.916250 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91780,"prepare.queue_time_us":358925,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":89744,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":16087,"spinlock_wait_cycles":3416832,"thread_start_us":602946,"threads_started":2,"wal-append.queue_time_us":428153}
I20260709 12:15:24.919978 21028 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:24.921015 12494 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 9. Preceding OpId from leader: term: 25 index: 10. (index mismatch)
I20260709 12:15:24.927266 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:24.929535 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:24.932190 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3741) took 1081 ms. Trace:
I20260709 12:15:24.936834 12494 rpcz_store.cc:276] 0709 12:15:23.850849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.853755 (+  2906us) service_pool.cc:224] Handling call
0709 12:15:24.920394 (+1066639us) spinlock_profiling.cc:243] Waited 900 us on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:24.920629 (+   235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:24.921442 (+   813us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:24.932126 (+ 10684us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2070912}
I20260709 12:15:24.847851 21197 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:24.955379 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:24.935703 21038 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:24.997952 20713 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:25.001276 12462 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 2.350s (SENT) [suppressed 25 similar messages]
I20260709 12:15:24.867220 21199 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:25.006361 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:25.010618 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:24.885648 21080 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.105s
I20260709 12:15:24.880410 21022 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:25.012725 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 232208us
I20260709 12:15:25.017726 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:25.019394 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.005s	sys 0.007s
W20260709 12:15:25.020591 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.005s	sys 0.007s
W20260709 12:15:25.030002 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 528023us
W20260709 12:15:25.046725 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45412.7us
I20260709 12:15:25.053431 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.016688 21027 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.033782 21032 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:25.073431 12494 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:25.074070 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:25.074708 12491 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:25.077085 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:25.078641 12494 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:15:25.079087 21201 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:15:25.079638 21203 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:25.080063 21203 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:25.080441 21203 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:25.083690 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:25.077492 12491 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:25.087107 20473 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:25.087349 21099 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:15:25.087692 20473 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:25.088639 20473 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.089023 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:25.090283 21099 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:25.090699 21099 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:25.091068 21099 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:25.061378 21096 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:25.068594 20832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.013s	sys 0.010s
I20260709 12:15:25.009105 21073 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.005318 21208 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 12:15:25.044926 21082 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:15:25.120635 21104 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:15:25.127231 21104 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:15:25.128413 21104 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:25.128814 21104 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:25.131389 21208 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.132040 21208 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.071012 21027 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.135290 21040 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:15:25.137051 21027 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:25.117574 20832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.014s	sys 0.010s
I20260709 12:15:25.152740 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:25.159641 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:25.160897 12494 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:25.104862 21032 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.018450 21168 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.146858 21215 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.137902 21199 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.148264 21000 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:15:25.182941 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 134899us
W20260709 12:15:25.175014 21160 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.057s	user 0.010s	sys 0.003s
I20260709 12:15:25.150503 21040 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:15:25.178910 21131 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.178977 21215 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.189762 21131 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.179522 21000 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:25.192349 21215 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.150244 21104 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.195776 21104 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:25.196241 12646 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 29. Ops: []
W20260709 12:15:25.196563 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.197757 21208 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.198040 21199 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.198807 21199 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.148219 21183 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:15:25.164417 21177 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:25.207419 13260 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:15:25.211956 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:25.184592 20999 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.209435 21177 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:25.239315 13256 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:15:25.267645 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55803.4us
I20260709 12:15:25.253243 21177 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.255848 21183 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:15:25.257208 20582 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.563s	user 0.010s	sys 0.000s
I20260709 12:15:25.258136 21199 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.257658 21099 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.198731 21160 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.258821 13258 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:15:25.263249 21100 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:25.264700 21034 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.640s	user 0.011s	sys 0.001s
W20260709 12:15:25.266347 21194 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.536s	user 0.006s	sys 0.008s
I20260709 12:15:25.266848 21022 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.267210 21211 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:25.269331 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:25.288897 13253 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:25.282647 21160 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.237051 21197 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:25.299275 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 231252us
I20260709 12:15:25.300201 12490 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: [18.12-19.13]
I20260709 12:15:25.300557 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.270053 21183 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:25.276549 21131 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.307636 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:25.289469 21177 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:25.326678 12497 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: [24.12-26.13]
I20260709 12:15:25.278132 21226 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:15:25.303829 20669 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:25.349517 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.350924 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:25.334308 20444 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:25.316706 21131 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.343490 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:25.343711 21183 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.288558 21194 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.303894 21082 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:15:25.344657 21000 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.346974 21032 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.303586 21211 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:25.371914 12643 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:15:25.303382 21227 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.075s
I20260709 12:15:25.394668 21211 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.397104 21194 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.386317 21000 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:25.387648 21131 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.388593 21232 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:25.389912 21235 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:15:25.389833 20669 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:25.376633 21203 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.413748 21203 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.414681 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.415112 21099 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:25.419230 20669 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.402277 21215 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:15:25.407032 21232 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.431660 21232 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:25.437104 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3560) took 2876 ms. Trace:
W20260709 12:15:25.365720 21224 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:15:25.353427 20444 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:25.401827 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.404270 12488 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:15:25.440016 12567 rpcz_store.cc:276] 0709 12:15:22.560757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.179823 (+619066us) service_pool.cc:224] Handling call
0709 12:15:23.843291 (+663468us) spinlock_profiling.cc:243] Waited 1.53 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:23.843547 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:23.843814 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:15:23.844204 (+   390us) log.cc:844] Serialized 14 byte log entry
0709 12:15:24.251809 (+407605us) log.cc:844] Serialized 14 byte log entry
0709 12:15:25.436367 (+1184558us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b180030c760. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:15:25.436582 (+   215us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:25.436654 (+    72us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:25.436734 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:25.436826 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:25.436990 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1142928,"prepare.queue_time_us":3499,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":23209,"raft.queue_time_us":128013,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":53544,"spinlock_wait_cycles":49622656,"thread_start_us":338150,"threads_started":1,"wal-append.queue_time_us":522120}
I20260709 12:15:25.407641 12497 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [21.14-23.15]
W20260709 12:15:25.445693 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:25.446343 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.444399 12643 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:25.459266 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:25.460359 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 174347us
I20260709 12:15:25.464162 21211 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.464969 21211 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.467671 21211 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:25.468880 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:25.471685 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:25.472379 12565 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:25.471861 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:25.473946 12567 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:15:25.476150 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.479277 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:25.480288 12643 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:25.469985 21100 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:25.486574 13260 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
W20260709 12:15:25.488852 13261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1010) took 1904 ms (client timeout 2350 ms). Trace:
I20260709 12:15:25.490629 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:25.492712 13261 rpcz_store.cc:269] 0709 12:15:23.583987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.587360 (+  3373us) service_pool.cc:224] Handling call
0709 12:15:25.483854 (+1896494us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:25.488771 (+  4917us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1032727424}
I20260709 12:15:25.493396 12572 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:25.491462 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.493934 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:25.494125 20444 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:25.499248 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.501016 12566 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:15:25.504601 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:25.505540 12563 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:25.451357 21131 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:15:25.470337 21173 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:15:25.509768 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183660us
I20260709 12:15:25.546417 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:25.546321 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:25.548979 12565 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:25.549415 12565 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:25.485606 21031 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.492129 21100 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:25.559149 12565 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.506556 21163 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:25.601120 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.565115 21100 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:25.569360 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.558527 21173 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:25.630158 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.566375 20582 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.567821 12565 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:25.680673 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1467) took 2105 ms. Trace:
I20260709 12:15:25.660207 12572 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:25.681330 12568 rpcz_store.cc:276] 0709 12:15:23.575055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.743756 (+168701us) service_pool.cc:224] Handling call
0709 12:15:23.744113 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:24.016030 (+271917us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:24.036988 (+ 20958us) log.cc:844] Serialized 14 byte log entry
0709 12:15:25.444274 (+1407286us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:25.444988 (+   714us) log.cc:844] Serialized 26 byte log entry
0709 12:15:25.445201 (+   213us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:25.445249 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:25.445337 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:25.680278 (+234941us) raft_consensus.cc:1685] finished
0709 12:15:25.680315 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:25.680575 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1000405,"prepare.queue_time_us":17322,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":1005874,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":74262,"thread_start_us":595654,"threads_started":2,"wal-append.queue_time_us":530156}
I20260709 12:15:25.715301 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.715983 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:25.716957 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.717588 12568 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:25.719199 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.720114 12568 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
W20260709 12:15:25.722047 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183879us
I20260709 12:15:25.724498 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.721750 21188 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:25.730711 21188 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.738130 21131 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:15:25.741981 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.752868 21230 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:25.670132 20582 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.583537 21163 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.763626 21247 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:25.764608 21247 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:25.767194 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.768107 12641 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:15:25.565657 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 1.06862e+06us
I20260709 12:15:25.788949 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.790164 12572 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
W20260709 12:15:25.803242 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78902.7us
I20260709 12:15:25.804796 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:25.786304 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:25.806051 12563 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:25.812842 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:25.813969 12563 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:25.821965 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.564937 21031 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.829540 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.834062 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.836604 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:25.837788 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:25.845585 12563 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:25.845340 12641 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:15:25.866937 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:25.867697 12563 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:25.867835 12572 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:25.875522 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:25.878726 21249 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
W20260709 12:15:25.881341 21243 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:15:25.885890 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:25.888237 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:25.889163 12641 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27. Re-sending same reply.
I20260709 12:15:25.891136 21247 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.891482 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:25.892088 21161 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 12:15:25.892691 12641 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 20: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:25.893999 12563 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:15:25.894634 12563 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 12
I20260709 12:15:25.895361 12563 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:15:25.895720 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.895762 12563 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:25.898334 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:25.843482 21239 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:25.908097 21239 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:25.904229 12460 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24.
I20260709 12:15:25.904942 21173 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.915796 21157 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:25.829049 21246 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:25.920157 21177 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.006s
I20260709 12:15:25.920056 21185 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.899698 21188 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:25.923048 21000 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.517s	user 0.016s	sys 0.000s
I20260709 12:15:25.906527 20582 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.903334 21032 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.006s
W20260709 12:15:25.927249 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.927793 21100 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.930106 21100 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.930680 21100 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.905995 21163 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.931125 21100 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.932837 21031 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:25.933229 21100 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:25.858680 21248 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:25.933970 21248 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:25.934986 21248 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:25.908501 21239 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:25.912501 12645 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:15:25.940574 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3435) took 2032 ms. Trace:
I20260709 12:15:25.912945 21238 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:25.933517 20662 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:25.903784 21203 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:25.951758 21203 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:25.952765 21203 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.904626 21168 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:25.953933 21168 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:25.954478 21168 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:25.954346 21174 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:25.831120 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.958279 21121 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:25.958876 21121 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:25.959335 21121 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:25.962476 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131455us
I20260709 12:15:25.963382 21185 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:25.941254 12644 rpcz_store.cc:276] 0709 12:15:23.907742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.910414 (+  2672us) service_pool.cc:224] Handling call
0709 12:15:23.944048 (+ 33634us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:23.998488 (+ 54440us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:15:24.053278 (+ 54790us) log.cc:844] Serialized 14 byte log entry
0709 12:15:24.494420 (+441142us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:24.515227 (+ 20807us) log.cc:844] Serialized 52 byte log entry
0709 12:15:24.515412 (+   185us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:24.515450 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:24.515545 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:25.903335 (+1387790us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x7b180049d080. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:25.936886 (+ 33551us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b180049d080. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:25.940244 (+  3358us) raft_consensus.cc:1685] finished
0709 12:15:25.940277 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:25.940487 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100868,"prepare.queue_time_us":59756,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":150189,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":11665,"spinlock_wait_cycles":45229056,"thread_start_us":432335,"threads_started":1,"wal-append.queue_time_us":540660}
I20260709 12:15:25.941263 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:25.965451 12645 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:25.967105 13262 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:25.969409 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:25.964078 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.970515 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:25.971500 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:25.972347 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6809556309999607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
I20260709 12:15:25.949218 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:25.974272 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.970130 13258 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:25.957739 20442 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:25.946285 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:25.898293 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:25.973157 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.974974 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:15:25.950033 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.979758 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:25.979828 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:25.981587 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:25.982295 12644 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:25.980940 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:25.984361 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6008556309998312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
W20260709 12:15:25.984922 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5783556309997948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
W20260709 12:15:25.982023 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:25.986254 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:25.988687 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:25.985270 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5756556309997904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1617s for a response
W20260709 12:15:25.996222 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5513556309999785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
W20260709 12:15:25.996675 12536 connection.cc:256] RPC call timeout handler was delayed by -2.54595563099997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260709 12:15:25.997006 12536 connection.cc:256] RPC call timeout handler was delayed by -2.532455630999948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 12:15:25.997293 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5180556309999247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:15:25.997618 12536 connection.cc:256] RPC call timeout handler was delayed by -2.5081556309999087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
W20260709 12:15:25.997941 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4928556309998839s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17090000000000002s for a response
I20260709 12:15:25.992328 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:25.999402 12609 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:15:26.000186 12609 leader_election.cc:403] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:15:25.974203 21188 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.974066 20582 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:25.973919 21161 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:26.062407 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.066385 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:26.067072 12565 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:25.974323 21247 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:26.003790 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.046581 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:26.039117 12643 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:15:26.081467 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
W20260709 12:15:26.082350 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.082818 12643 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [21.11-22.12]
I20260709 12:15:25.988045 21157 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:26.080495 12572 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:15:25.998225 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4757556309998563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260709 12:15:26.085973 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4631556309998359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
W20260709 12:15:26.086696 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4379556309997952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20260709 12:15:26.082891 12645 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:15:26.087255 12536 connection.cc:256] RPC call timeout handler was delayed by -2.367755630999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18480000000000002s for a response
W20260709 12:15:26.088351 12536 connection.cc:256] RPC call timeout handler was delayed by -2.3668556309999076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
W20260709 12:15:26.088765 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2165556309998919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260709 12:15:26.089105 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1202556309999636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
W20260709 12:15:26.089457 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0950556309999229s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21509999999999999s for a response
W20260709 12:15:26.089823 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0761556309998923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21720000000000003s for a response
W20260709 12:15:26.090157 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0545556309998574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
W20260709 12:15:26.090500 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5172556309998981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27930000000000005s for a response
W20260709 12:15:26.090827 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2346556309998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31070000000000003s for a response
I20260709 12:15:26.091778 12536 leader_election.cc:400] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:26.094714 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:26.085275 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:25.987334 20913 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.051323 21000 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:25.996085 21232 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:26.001207 21034 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.051818 20442 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:25.977239 21163 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.088s
I20260709 12:15:26.002978 21124 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:26.108331 12572 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:15:26.002786 21226 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:26.069329 20582 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.125937 21209 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:15:26.126427 21212 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.144124 21212 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.116006 21000 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:26.146932 21000 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:26.109951 20713 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:26.103164 20669 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:15:26.141110 21254 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.138614 21124 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.101478 21157 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.139537 21034 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.142202 21226 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:26.146449 21212 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.110941 21247 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.063383 21188 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.079360 21161 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.150992 12644 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:15:26.150170 20713 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:26.164552 12570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.014s	sys 0.025s
I20260709 12:15:26.166190 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:26.169687 12570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.014s	sys 0.026s
I20260709 12:15:26.153162 21157 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:26.154038 21124 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:26.159947 21230 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:26.163758 21212 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:26.157840 21254 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.170207 20713 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:26.184352 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3599) took 2455 ms. Trace:
I20260709 12:15:26.194111 12564 rpcz_store.cc:276] 0709 12:15:23.728673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:23.784966 (+ 56293us) service_pool.cc:224] Handling call
0709 12:15:23.785388 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:24.112268 (+326880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.183924 (+2071656us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:26.184247 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":598489984}
I20260709 12:15:26.198535 21212 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:26.258266 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173063us
I20260709 12:15:26.262168 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:26.267695 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:26.271432 21254 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.272435 21254 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:26.272799 21254 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.273126 21254 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:26.273898 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.275400 21121 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.216372 21226 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.281944 21121 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:26.284812 21000 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.287509 21000 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.287945 21188 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.216996 21251 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:15:26.311528 21251 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:26.312327 21251 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:26.318005 21247 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:26.319074 21025 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:26.319846 21025 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:26.322796 21034 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.325873 21183 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:26.329407 21162 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:26.334653 21161 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:26.336120 21117 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:26.336738 21117 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:26.341650 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:26.342729 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:26.345204 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:26.358960 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.264113 21164 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.378047 21164 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.370678 20859 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:26.313313 21251 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.370708 21179 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:26.321058 21025 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.337867 21117 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.376432 21038 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:26.300614 21188 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:26.307319 21255 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.004s
I20260709 12:15:26.380815 21164 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:26.355492 21252 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.139s	user 0.004s	sys 0.000s
W20260709 12:15:26.388260 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.040s	sys 0.061s
I20260709 12:15:26.388468 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:26.385655 20859 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:26.391772 21171 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:15:26.402808 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:26.404073 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.398953 20981 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.423920 21179 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:26.424593 21188 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:26.433919 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:26.437481 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.042s	sys 0.061s
W20260709 12:15:26.437232 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48852.7us
I20260709 12:15:26.464406 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:26.464871 21188 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:26.432049 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.466205 13255 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:26.468113 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:26.469220 13261 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:26.469682 21177 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:26.470355 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:26.472254 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:26.472790 21038 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.473323 21038 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.474447 21179 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:26.439136 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 445598us
I20260709 12:15:26.475558 21038 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:26.479290 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:26.481691 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:26.481856 21262 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.482414 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1136) took 1672 ms. Trace:
I20260709 12:15:26.482986 21241 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:26.483871 21241 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:26.482923 12495 rpcz_store.cc:276] 0709 12:15:24.810234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:24.810429 (+   195us) service_pool.cc:224] Handling call
0709 12:15:25.879052 (+1068623us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:25.879355 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:25.879683 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:15:25.880176 (+   493us) log.cc:844] Serialized 14 byte log entry
0709 12:15:26.481934 (+601758us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:26.482010 (+    76us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:26.482079 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.482174 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:26.482334 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18098,"prepare.queue_time_us":929,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"spinlock_wait_cycles":6471552,"thread_start_us":583166,"threads_started":1}
I20260709 12:15:26.484242 21241 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:26.484666 21241 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:26.484899 21132 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:15:26.485780 21000 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:15:26.487288 21168 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.532s	user 0.013s	sys 0.000s
I20260709 12:15:26.482489 21262 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.488536 21160 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.488394 20981 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.489120 21160 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.491057 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:26.495425 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:26.496270 12569 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:26.497756 12563 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:15:26.500684 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:26.504580 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.399396 20662 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.432940 21117 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:26.442329 20859 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:26.492213 21239 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.555s	user 0.009s	sys 0.003s
I20260709 12:15:26.514668 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:26.525418 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:26.431998 21157 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.515362 20662 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.425227 21254 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.451213 21099 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:26.485092 20713 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:26.433718 21181 consensus_queue.cc:1175] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 24 status { last_received { term: 19 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [21.14-23.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:15:26.487913 21168 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:26.542191 21252 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:15:26.516804 21016 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:15:26.388445 21212 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.553259 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:26.557770 12490 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:26.557878 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:26.569557 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.549836 21099 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:26.545897 21215 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.542493 21239 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.576453 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:26.586583 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
W20260709 12:15:26.588706 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.592347 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:26.595206 12490 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:15:26.596247 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 501565us
I20260709 12:15:26.601857 20981 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.602118 21254 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.602909 20981 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:26.603588 20981 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.605397 21121 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.583001 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:26.560889 20442 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:26.607362 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:26.606871 21121 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.605855 21233 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:26.607960 21215 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:26.611585 21259 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:15:26.607906 12536 leader_election.cc:400] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:26.609609 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3605) took 1137 ms. Trace:
I20260709 12:15:26.607805 20442 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.618749 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:26.619747 12564 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:26.616119 12571 rpcz_store.cc:276] 0709 12:15:25.472654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.477299 (+  4645us) service_pool.cc:224] Handling call
0709 12:15:25.492151 (+ 14852us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:25.492543 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:25.559741 (+ 67198us) log.cc:844] Serialized 14 byte log entry
0709 12:15:25.985793 (+426052us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:26.058700 (+ 72907us) log.cc:844] Serialized 26 byte log entry
0709 12:15:26.058837 (+   137us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:26.058876 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.058938 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:26.609351 (+550413us) raft_consensus.cc:1685] finished
0709 12:15:26.609374 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:26.609528 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":79902,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":200674,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":22388,"thread_start_us":413358,"threads_started":1,"wal-append.queue_time_us":599203}
I20260709 12:15:26.614260 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:26.578967 13254 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:26.623795 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42406
I20260709 12:15:26.625324 12646 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:26.625988 12536 leader_election.cc:403] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:15:26.626799 12644 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:26.627419 12644 raft_consensus.cc:614] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:26.628890 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:26.629956 12640 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:26.632175 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:26.622438 21155 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.637s	user 0.013s	sys 0.028s
I20260709 12:15:26.674924 21038 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:26.676011 20662 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.676994 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:26.686980 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.681751 21254 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:26.681123 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3614) took 1183 ms. Trace:
I20260709 12:15:26.693302 12568 rpcz_store.cc:276] 0709 12:15:25.497252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.721541 (+224289us) service_pool.cc:224] Handling call
0709 12:15:25.722070 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:25.794558 (+ 72488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:25.794613 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:25.794682 (+    69us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:25.794727 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.680797 (+886070us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:26.680860 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:26.681044 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":165908864}
I20260709 12:15:26.613046 21215 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:26.682921 21155 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.700s	user 0.013s	sys 0.029s
I20260709 12:15:26.686244 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:26.690294 21038 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.698002 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:26.698664 21260 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:15:26.700076 21038 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:26.699741 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:26.690977 21185 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.727s	user 0.007s	sys 0.007s
I20260709 12:15:26.711110 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:26.632141 12643 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:15:26.608752 21233 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:26.670470 21232 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:26.610244 21121 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:26.716416 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:26.720129 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112366us
I20260709 12:15:26.736595 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:26.730453 20978 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.026s	sys 0.007s
W20260709 12:15:26.729142 21203 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.016s	sys 0.015s
I20260709 12:15:26.731137 21160 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.610091 21099 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:26.744860 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 445194us
I20260709 12:15:26.745333 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:26.767740 12492 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: [27.13-27.13]
I20260709 12:15:26.765230 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:26.756083 12569 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
W20260709 12:15:26.749578 20978 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.027s	sys 0.007s
I20260709 12:15:26.775272 12489 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:26.777432 12564 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
W20260709 12:15:26.754621 21203 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.016s	sys 0.015s
I20260709 12:15:26.797083 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:26.798841 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:26.799721 12494 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.10-25.10]
I20260709 12:15:26.800074 12492 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:26.753821 21254 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:15:26.806361 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.752672 21157 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.735288 21232 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:26.810899 12568 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 20. Current term is 21. Ops: []
W20260709 12:15:26.811364 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66159.7us
I20260709 12:15:26.819103 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:26.801433 20713 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.815570 21160 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:15:26.823364 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:26.834914 12568 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:26.803349 21248 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:15:26.850472 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.810318 21269 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:15:26.844439 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 380087us
W20260709 12:15:26.851361 21264 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:15:26.806216 21254 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:26.862213 21280 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:15:26.850234 21248 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:26.848567 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:26.869561 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:26.884245 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:26.886224 21016 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:26.822484 21155 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:15:26.907627 21232 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:26.909435 21238 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:26.910099 21238 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:26.894011 12492 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:26.914410 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:26.853572 21263 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:26.857759 20442 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:26.875769 21239 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:27.036954 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.038878 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.040467 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.041402 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3606) took 1546 ms. Trace:
I20260709 12:15:26.845592 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:26.902643 21268 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:15:26.895089 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:26.889058 21258 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:15:26.874177 21254 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:26.888681 20978 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.065999 20978 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.068542 20978 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.042027 12570 rpcz_store.cc:276] 0709 12:15:25.494919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.503710 (+  8791us) service_pool.cc:224] Handling call
0709 12:15:25.547266 (+ 43556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:26.186979 (+639713us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:26.187022 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:26.187534 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:15:26.740623 (+553089us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:26.740685 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.740779 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:26.915319 (+174540us) raft_consensus.cc:1685] finished
0709 12:15:26.915348 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:27.041310 (+125962us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13288,"raft.queue_time_us":310683,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"thread_start_us":1089785,"threads_started":2,"wal-append.queue_time_us":636693}
I20260709 12:15:26.898639 21248 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:27.058789 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:27.078099 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:27.079041 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:27.020129 21263 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:27.080499 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:27.078982 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:27.091027 12568 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:27.086658 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 241154us
I20260709 12:15:27.092844 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:27.045280 21117 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.521s	user 0.015s	sys 0.025s
I20260709 12:15:27.097402 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.097268 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:27.105939 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.106808 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:26.898886 20859 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.060181 20832 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:27.123443 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:27.126906 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:27.135064 20713 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:27.137794 20713 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.138300 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1137) took 1924 ms. Trace:
I20260709 12:15:27.138883 13260 rpcz_store.cc:276] 0709 12:15:25.213767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.238048 (+ 24281us) service_pool.cc:224] Handling call
0709 12:15:25.484263 (+246215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:25.492373 (+  8110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.138054 (+1645681us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b540019d2d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:27.138237 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538199424}
I20260709 12:15:27.099260 21163 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:27.141870 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.142545 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:27.090772 21263 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:27.143257 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:27.142406 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1139) took 1910 ms. Trace:
I20260709 12:15:27.144152 13253 rpcz_store.cc:276] 0709 12:15:25.231898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.232057 (+   159us) service_pool.cc:224] Handling call
0709 12:15:25.257950 (+ 25893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:25.302287 (+ 44337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.142063 (+1839776us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:27.142340 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70635264}
I20260709 12:15:27.117584 20978 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.120198 21233 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:27.148175 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.139241 20859 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:27.155081 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110972us
I20260709 12:15:27.159888 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.162715 12568 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:27.163019 12637 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:15:27.166663 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:27.167342 12637 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:27.173890 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:27.174386 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.180630 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.158068 21163 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:27.136386 20832 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.172091 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:27.198874 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:27.199864 12566 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 21 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:15:27.206399 12495 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:27.208753 12637 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:27.214088 12566 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:27.152714 20978 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:27.218824 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121662us
I20260709 12:15:27.212682 21016 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.210588 21117 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 12:15:27.221402 20859 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:27.232137 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:27.214967 21124 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.016s	user 0.017s	sys 0.000s
W20260709 12:15:27.241070 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.223053 21267 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:15:27.226377 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:27.240538 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3457) took 1899 ms (client timeout 1540 ms). Trace:
I20260709 12:15:27.242605 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:27.245138 12642 rpcz_store.cc:269] 0709 12:15:25.340715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.342336 (+  1621us) service_pool.cc:224] Handling call
0709 12:15:27.231878 (+1889542us) spinlock_profiling.cc:243] Waited 7.12 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:15:27.240440 (+  8562us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16384256}
I20260709 12:15:27.245864 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:27.249593 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:27.250175 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:27.251550 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.255548 12570 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:27.256130 12570 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:27.264667 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:27.197291 21233 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
W20260709 12:15:27.236313 21278 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
W20260709 12:15:27.237936 21265 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.359s	user 0.001s	sys 0.000s
W20260709 12:15:27.269100 12462 connection.cc:256] RPC call timeout handler was delayed by -1.0529115920001004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260709 12:15:27.304080 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:27.270371 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:27.264196 21171 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:27.273732 21254 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.243331 20713 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:27.314231 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.224493 21226 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:15:27.252925 21124 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.277140 20978 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.277578 21233 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:27.278908 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1124) took 1904 ms. Trace:
I20260709 12:15:27.287817 21248 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:27.294189 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67893.6us
I20260709 12:15:27.330506 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:27.294648 21272 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.297987 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3459) took 1956 ms. Trace:
I20260709 12:15:27.298686 21163 raft_consensus.cc:2764] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:15:27.236757 21016 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.301115 20832 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.338151 12494 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
W20260709 12:15:27.304375 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 247714us
I20260709 12:15:27.308070 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:27.305935 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3608) took 1810 ms. Trace:
I20260709 12:15:27.339978 12567 rpcz_store.cc:276] 0709 12:15:25.495638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.499528 (+  3890us) service_pool.cc:224] Handling call
0709 12:15:25.499863 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:25.966773 (+466910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:25.966794 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:25.990771 (+ 23977us) log.cc:844] Serialized 26 byte log entry
0709 12:15:26.441448 (+450677us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:26.441493 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.441575 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:27.305607 (+864032us) raft_consensus.cc:1685] finished
0709 12:15:27.305656 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:27.305855 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":169376,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":84012,"thread_start_us":791868,"threads_started":2,"wal-append.queue_time_us":728320}
I20260709 12:15:27.312750 12570 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.327175 12564 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:27.237237 21203 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.083s
I20260709 12:15:27.333135 12646 rpcz_store.cc:276] 0709 12:15:25.341328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.342506 (+  1178us) service_pool.cc:224] Handling call
0709 12:15:26.284896 (+942390us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:26.285144 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:26.626058 (+340914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.297672 (+671614us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:27.297905 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":803195008}
I20260709 12:15:27.323398 12641 rpcz_store.cc:276] 0709 12:15:25.374515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.989713 (+615198us) service_pool.cc:224] Handling call
0709 12:15:25.990816 (+  1103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:26.053878 (+ 63062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:26.053933 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:26.076082 (+ 22149us) log.cc:844] Serialized 26 byte log entry
0709 12:15:26.563104 (+487022us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:26.563162 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.563268 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:27.278612 (+715344us) raft_consensus.cc:1685] finished
0709 12:15:27.278642 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:27.278828 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":447,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":8351,"thread_start_us":417047,"threads_started":1,"wal-append.queue_time_us":767581}
W20260709 12:15:27.333084 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129407us
I20260709 12:15:27.344686 12570 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:15:27.331845 21241 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.847s	user 0.012s	sys 0.007s
I20260709 12:15:27.321156 20713 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:27.351075 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:27.353451 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:27.332243 21156 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:27.357296 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:27.323189 21233 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:27.360613 12568 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:27.375794 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:27.384171 21016 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:27.365029 21292 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:27.396988 13260 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:27.397372 12570 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 19. (term mismatch)
I20260709 12:15:27.371896 20978 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:27.403323 12570 pending_rounds.cc:85] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 18
I20260709 12:15:27.373154 21299 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:27.339098 21272 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.406767 12570 pending_rounds.cc:107] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:15:27.407176 12570 raft_consensus.cc:2889] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:27.433806 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51223.3us
W20260709 12:15:27.436946 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 106564us
I20260709 12:15:27.437922 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:27.442488 12570 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:15:27.443038 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:27.443645 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:27.392596 21295 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:27.448601 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:27.399497 21292 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:27.371119 21241 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.374176 21212 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:15:27.470314 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:27.471967 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.434203 21226 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:27.429855 21299 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:27.474244 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.475378 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.372845 20713 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.479176 21163 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:27.479755 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.480474 20832 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:27.481011 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.481261 20713 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:27.438799 21305 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:27.479183 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:27.471664 21251 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:27.479188 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:27.485244 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.485810 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:27.498859 12640 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:27.486526 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 135435us
I20260709 12:15:27.440795 21298 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:27.452490 21203 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:27.483654 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:27.452342 21295 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:27.449836 21302 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:15:27.501933 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.480789 21232 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:27.502811 12489 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 28. Ops: []
I20260709 12:15:27.505175 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:27.506541 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:27.482684 21226 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:27.480909 20832 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:27.532017 20832 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:27.475211 21292 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:27.510056 21305 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:27.538828 21305 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.511603 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:27.482762 21299 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:27.542119 21291 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 12:15:27.515986 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.543329 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.481271 21293 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:27.515443 21302 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:27.522223 12496 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:27.523187 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.524036 21226 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:27.525146 21241 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:27.512341 21298 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:27.563055 21238 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.652s	user 0.013s	sys 0.000s
I20260709 12:15:27.564704 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.564893 21298 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:27.526703 21203 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:27.566234 21293 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:27.566471 21203 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.528456 21272 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.470518 21233 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 27.13, Last appended by leader: 13, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.530215 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:27.570863 21289 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:15:27.550655 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:27.571383 12488 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:27.571841 12492 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:15:27.572949 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.552827 21302 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:27.553339 12640 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:27.563247 12489 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:15:27.561474 21254 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.561582 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.544538 12493 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:27.486579 21263 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.580799 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:27.579355 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:27.576723 12643 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:27.587337 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.563647 21238 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.515885 21232 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:27.500408 20713 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:15:27.516919 21295 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:27.558758 21163 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:27.545046 21301 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:27.556447 21124 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.612869 12637 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:27.592908 12488 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:27.576021 21215 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:27.618299 12563 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:27.614857 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:27.621547 12493 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:15:27.621856 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.627146 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:27.632112 12492 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:27.634127 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:27.611003 21248 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.653357 21307 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:27.654582 21307 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:27.654995 21307 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:27.623624 21181 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:27.658146 21181 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.658797 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3623) took 1119 ms. Trace:
I20260709 12:15:27.659299 12564 rpcz_store.cc:276] 0709 12:15:26.539258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:26.805600 (+266342us) service_pool.cc:224] Handling call
0709 12:15:26.827539 (+ 21939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:27.344018 (+516479us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.658495 (+314477us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:27.658723 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1148158208}
I20260709 12:15:27.660018 21238 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:27.665458 21262 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:27.666558 12492 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:27.667544 21281 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.212s
W20260709 12:15:27.669062 21304 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
W20260709 12:15:27.665686 21279 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:15:27.665526 12643 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:15:27.674439 21179 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:27.675019 21179 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:27.675200 21287 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.675773 21287 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.676028 21179 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.623193 21239 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.677963 21239 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.680192 21239 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:27.645239 21124 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:27.681963 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.685740 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:27.623333 21163 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:27.677386 21179 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:27.649119 20713 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:27.642499 21215 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:27.693701 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150407us
I20260709 12:15:27.804540 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.755579 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:27.806936 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.674535 21303 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:15:27.687091 21124 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:27.821839 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:27.825805 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 270464us
I20260709 12:15:27.658824 21263 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:27.834931 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.810278 21309 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:15:27.851912 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:27.799678 21021 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.802569 20713 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:27.820878 21248 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:27.756320 21291 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.102s
I20260709 12:15:27.686338 21232 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 25.10, Last appended by leader: 10, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.892431 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.827970 20832 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:27.894868 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67638.8us
I20260709 12:15:27.934362 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3627) took 1096 ms. Trace:
I20260709 12:15:27.965144 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:27.928123 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.920315 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.970480 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.924168 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:42406 (request call id 0) took 1312 ms. Trace:
I20260709 12:15:27.975293 12644 rpcz_store.cc:276] 0709 12:15:26.611946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:26.623641 (+ 11695us) service_pool.cc:224] Handling call
0709 12:15:27.923877 (+1300236us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:27.924037 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5673728}
W20260709 12:15:27.928704 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3759) took 2630 ms (client timeout 1997 ms). Trace:
W20260709 12:15:27.976698 12490 rpcz_store.cc:269] 0709 12:15:25.298619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.307319 (+  8700us) service_pool.cc:224] Handling call
0709 12:15:27.928419 (+2621100us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:27.928577 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2869076736}
I20260709 12:15:27.978483 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:27.934603 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.897575 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.920841 12491 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:15:27.980300 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:27.980814 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.929500 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1479) took 2428 ms. Trace:
I20260709 12:15:27.981813 12565 rpcz_store.cc:276] 0709 12:15:25.500595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.947177 (+446582us) service_pool.cc:224] Handling call
0709 12:15:26.066910 (+119733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:26.067448 (+   538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.929256 (+1861808us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001b21d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:27.929428 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":274519936}
I20260709 12:15:27.855214 21306 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:27.981305 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:27.985129 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:27.869988 20713 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:27.965121 12567 rpcz_store.cc:276] 0709 12:15:26.837599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.342220 (+504621us) service_pool.cc:224] Handling call
0709 12:15:27.342740 (+   520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:27.376973 (+ 34233us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:27.408466 (+ 31493us) log.cc:844] Serialized 14 byte log entry
0709 12:15:27.692374 (+283908us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:27.693107 (+   733us) log.cc:844] Serialized 26 byte log entry
0709 12:15:27.693297 (+   190us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:27.693344 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.693450 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:27.934003 (+240553us) raft_consensus.cc:1685] finished
0709 12:15:27.934036 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:27.934272 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5938,"prepare.queue_time_us":31236,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":17038,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":49282,"thread_start_us":256786,"threads_started":2,"wal-append.queue_time_us":383808}
I20260709 12:15:27.892863 21263 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:27.986450 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:27.993908 12496 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 19.
I20260709 12:15:27.958792 21232 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:27.988209 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:27.991061 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:27.992202 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:27.931879 21181 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:27.998979 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:28.012315 13256 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:28.000185 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:27.869738 21021 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:27.996682 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:28.010310 12494 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:28.007939 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:27.876108 21248 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.985368 21306 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:28.021420 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:28.022209 13253 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:28.023141 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:27.923147 21292 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.985718 21312 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:28.033485 13255 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:28.030391 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:27.889882 21163 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:27.926707 21299 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.036332 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.038233 13261 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:28.036846 12570 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:28.043522 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:15:28.047166 21248 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.049304 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.053668 13259 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:28.048127 21299 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.061097 20832 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.034801 20582 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.061923 21215 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.070081 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.071484 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.074005 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:28.074664 21311 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:15:28.075407 21021 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.076681 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3626) took 1239 ms. Trace:
I20260709 12:15:28.077153 12569 rpcz_store.cc:276] 0709 12:15:26.837239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.322153 (+484914us) service_pool.cc:224] Handling call
0709 12:15:27.322605 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:27.335790 (+ 13185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:27.335823 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:27.347570 (+ 11747us) log.cc:844] Serialized 26 byte log entry
0709 12:15:27.800009 (+452439us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:27.800076 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.800163 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:28.076406 (+276243us) raft_consensus.cc:1685] finished
0709 12:15:28.076438 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:28.076616 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15329,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":24116,"thread_start_us":259560,"threads_started":1,"wal-append.queue_time_us":457781}
I20260709 12:15:28.075233 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.081291 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.083055 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:15:28.089845 21226 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.005s
I20260709 12:15:28.090410 21226 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.027431 21306 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:28.092720 21226 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.096751 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:15:28.097666 21328 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:15:28.098816 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3464) took 1950 ms. Trace:
I20260709 12:15:28.099272 12645 rpcz_store.cc:276] 0709 12:15:26.148055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:26.148466 (+   411us) service_pool.cc:224] Handling call
0709 12:15:28.098621 (+1950155us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.098744 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1006720896}
I20260709 12:15:28.002254 21124 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.11, Last appended by leader: 11, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.101588 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3628) took 1263 ms. Trace:
I20260709 12:15:28.102034 12568 rpcz_store.cc:276] 0709 12:15:26.837929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.173572 (+335643us) service_pool.cc:224] Handling call
0709 12:15:28.101335 (+927763us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.101504 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422463360}
I20260709 12:15:28.052618 21040 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.053822 21254 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:15:28.186858 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.188422 21292 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.191550 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:28.192813 21185 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:28.195180 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:28.196097 21185 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:28.197511 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:28.152304 21040 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.152495 21254 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:28.202143 13253 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: [24.16-26.17]
W20260709 12:15:28.206197 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:28.197568 21302 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.623s	user 0.008s	sys 0.004s
I20260709 12:15:28.222508 12492 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
W20260709 12:15:28.226964 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 219116us
I20260709 12:15:28.204684 21254 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:28.260473 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44225.1us
I20260709 12:15:28.204219 21040 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.249874 21302 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:28.269940 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.212301 21185 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:28.215535 21312 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:28.291525 21251 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.784s	user 0.021s	sys 0.000s
I20260709 12:15:28.292179 20582 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:28.293749 12567 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:28.293571 12569 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.10-25.10]
I20260709 12:15:28.147846 21295 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.213694 21257 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:28.312888 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:28.330014 12495 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:28.313328 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:28.320587 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:28.331370 12494 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:28.331543 12572 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:15:28.333397 12497 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:28.336508 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:28.336647 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.339092 12567 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
W20260709 12:15:28.337576 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.340440 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:28.326138 21269 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.303s	user 0.007s	sys 0.020s
I20260709 12:15:28.341228 12567 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30. Re-sending same reply.
I20260709 12:15:28.351181 12493 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:28.353240 20582 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:28.353940 20582 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:28.355124 20582 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:28.356272 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3635) took 1516 ms (client timeout 1911 ms). Trace:
I20260709 12:15:28.356971 20582 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:28.262178 21239 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 12:15:28.371635 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:28.373067 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.332324 21301 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.375739 12569 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:28.371662 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:28.356747 12563 rpcz_store.cc:269] 0709 12:15:26.839823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.078633 (+238810us) service_pool.cc:224] Handling call
0709 12:15:28.356038 (+1277405us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.356201 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1211865984}
I20260709 12:15:28.375504 12641 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:28.328094 21312 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.353277 21073 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:28.378772 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:28.385159 21329 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:15:28.385861 21185 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:28.385342 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:28.296839 21324 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.112s
I20260709 12:15:28.387634 21324 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:28.376415 21263 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:28.388350 21324 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:28.388643 21263 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.389571 21324 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.390868 21302 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.387888 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3616) took 1861 ms. Trace:
I20260709 12:15:28.392302 21239 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:28.392807 21239 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:28.392858 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:28.394062 12563 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:28.333302 21332 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.113s
I20260709 12:15:28.394150 21239 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:28.395217 21327 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
I20260709 12:15:28.391039 21263 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.396487 21021 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:28.397656 21021 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:28.398058 21021 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:28.347347 21000 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:28.398877 21000 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:28.402697 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.391759 12571 rpcz_store.cc:276] 0709 12:15:26.526264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:26.716364 (+190100us) service_pool.cc:224] Handling call
0709 12:15:26.716673 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:26.727728 (+ 11055us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:27.680896 (+953168us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b1800452080. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:15:27.828159 (+147263us) log.cc:844] Serialized 14 byte log entry
0709 12:15:28.289193 (+461034us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:28.299230 (+ 10037us) log.cc:844] Serialized 26 byte log entry
0709 12:15:28.313249 (+ 14019us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:28.313296 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:28.313378 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:28.387527 (+ 74149us) raft_consensus.cc:1685] finished
0709 12:15:28.387658 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:28.387826 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2077,"prepare.queue_time_us":467289,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":90,"raft.queue_time_us":33216,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":20220,"spinlock_wait_cycles":4148096,"thread_start_us":587228,"threads_started":3,"wal-append.queue_time_us":213789}
I20260709 12:15:28.404843 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.405344 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:28.403475 12565 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:28.395031 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.410296 13256 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:28.411240 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:28.379714 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.421801 21117 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:28.422356 21117 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:28.422758 21117 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:28.424317 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:28.355182 20444 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:28.399048 21021 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:28.351080 21269 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.352s	user 0.007s	sys 0.021s
I20260709 12:15:28.427918 21269 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:28.428344 21021 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:28.428999 21269 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.429096 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.426223 21302 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:28.431213 21302 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:28.432070 21302 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.435379 21212 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:28.436149 21212 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:28.436251 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.439925 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:28.342738 21257 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.443109 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.444119 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:28.444350 21257 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.440371 20981 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.445410 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:28.446303 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.446738 12570 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:28.447510 12565 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:28.387800 21132 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:28.342315 20713 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.412125 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:28.386404 21185 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:28.379256 21248 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:28.460021 21131 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:28.460520 21312 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.450918 21132 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:28.433458 21302 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:28.423964 21263 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:28.453337 21239 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:15:28.484438 21282 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.291s	user 0.005s	sys 0.005s
I20260709 12:15:28.437414 21212 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.407804 21301 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.487087 21185 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:28.486315 21248 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.546053 21298 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.547048 21298 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.547500 21287 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.552735 21295 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.559927 21248 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.562325 21306 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.577353 21038 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:28.578218 21038 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:28.577797 21117 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.579787 21038 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 27.10, Last appended by leader: 10, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.581259 21117 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.582460 21168 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.577214 21185 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:28.584647 21277 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.436s	user 0.003s	sys 0.001s
I20260709 12:15:28.586323 21185 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.587399 21307 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.588974 21307 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:28.592378 21254 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.593689 21132 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.595718 21132 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:28.596840 21301 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.597970 21038 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:28.604030 21233 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:28.532934 21212 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:28.611423 21171 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:28.612138 21171 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:28.614486 21306 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.617839 21254 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.617223 21171 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.618976 21171 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:28.683106 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:15:28.687367 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 308531us
I20260709 12:15:28.421010 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 19.
I20260709 12:15:28.426847 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:28.689550 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.697674 12567 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:15:28.726061 12639 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
W20260709 12:15:28.698529 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.699539 12568 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:28.723364 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.726810 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:28.724720 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3455) took 3384 ms. Trace:
I20260709 12:15:28.699975 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1494) took 2060 ms. Trace:
I20260709 12:15:28.701282 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1148) took 1196 ms. Trace:
I20260709 12:15:28.700887 12569 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:28.710371 12571 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:28.727295 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3489) took 1040 ms. Trace:
I20260709 12:15:28.753448 21124 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.754863 21124 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.694062 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.771669 12637 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:15:28.772219 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1027) took 2263 ms. Trace:
W20260709 12:15:28.772836 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3456) took 3432 ms (client timeout 3280 ms). Trace:
W20260709 12:15:28.773396 12639 rpcz_store.cc:269] 0709 12:15:25.340391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.342050 (+  1659us) service_pool.cc:224] Handling call
0709 12:15:26.582737 (+1240687us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:15:28.772439 (+2189702us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.772677 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4888336512}
I20260709 12:15:28.746821 12489 rpcz_store.cc:276] 0709 12:15:27.505064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.521867 (+ 16803us) service_pool.cc:224] Handling call
0709 12:15:27.553287 (+ 31420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:27.577763 (+ 24476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:28.689351 (+1111588us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b540012b1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.701010 (+ 11659us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71618944}
I20260709 12:15:28.695530 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:28.783571 20713 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.786923 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1029) took 1541 ms. Trace:
I20260709 12:15:28.787643 13260 rpcz_store.cc:276] 0709 12:15:27.245162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.245498 (+   336us) service_pool.cc:224] Handling call
0709 12:15:28.786309 (+1540811us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.786618 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284829440}
I20260709 12:15:28.788589 12565 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:28.782323 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:28.791560 12563 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:28.792057 21298 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.772753 13257 rpcz_store.cc:276] 0709 12:15:26.464053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:26.473739 (+  9686us) service_pool.cc:224] Handling call
0709 12:15:26.474058 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:26.474421 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:26.474453 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:26.474947 (+   494us) log.cc:844] Serialized 26 byte log entry
0709 12:15:26.868211 (+393264us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:26.868284 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:26.868398 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:28.583737 (+1715339us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:28.726570 (+142833us) raft_consensus.cc:1685] finished
0709 12:15:28.726632 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:28.727212 (+   580us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1145,"raft.queue_time_us":28568,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":3843,"spinlock_wait_cycles":7660288,"thread_start_us":392811,"threads_started":1,"wal-append.queue_time_us":643681}
I20260709 12:15:28.777873 21254 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.796594 21254 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.801821 21099 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.803285 21099 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.744244 12638 rpcz_store.cc:276] 0709 12:15:25.340070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:25.341717 (+  1647us) service_pool.cc:224] Handling call
0709 12:15:26.620513 (+1278796us) spinlock_profiling.cc:243] Waited 7.53 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:26.620842 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:26.621185 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:26.621224 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:26.636283 (+ 15059us) log.cc:844] Serialized 26 byte log entry
0709 12:15:27.013883 (+377600us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:27.013963 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:27.014084 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:28.462522 (+1448438us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:28.723784 (+261262us) raft_consensus.cc:1685] finished
0709 12:15:28.723847 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:28.724319 (+   472us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56339200,"thread_start_us":299168,"threads_started":1,"wal-append.queue_time_us":463895}
I20260709 12:15:28.806140 21292 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.804733 21254 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.807433 21292 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:28.747488 12643 rpcz_store.cc:276] 0709 12:15:27.686984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.687216 (+   232us) service_pool.cc:224] Handling call
0709 12:15:27.800432 (+113216us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:27.801065 (+   633us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:28.196282 (+395217us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b18003dcea0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:15:28.272291 (+ 76009us) log.cc:844] Serialized 14 byte log entry
0709 12:15:28.449926 (+177635us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:28.456351 (+  6425us) log.cc:844] Serialized 52 byte log entry
0709 12:15:28.456617 (+   266us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:28.456670 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:28.456785 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:28.726376 (+269591us) raft_consensus.cc:1685] finished
0709 12:15:28.726607 (+   231us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:28.727045 (+   438us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81553,"prepare.queue_time_us":399708,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":77544,"raft.run_cpu_time_us":400,"raft.run_wall_time_us":24862,"spinlock_wait_cycles":15213440,"thread_start_us":176147,"threads_started":2,"wal-append.queue_time_us":215795}
W20260709 12:15:28.775336 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:28.824810 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3613) took 2329 ms (client timeout 2984 ms). Trace:
W20260709 12:15:28.825887 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.690135 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:28.825512 12572 rpcz_store.cc:269] 0709 12:15:26.494937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:26.499102 (+  4165us) service_pool.cc:224] Handling call
0709 12:15:28.824154 (+2325052us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:28.824545 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4207837056}
I20260709 12:15:28.792465 20978 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:15:28.843060 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61754.2us
I20260709 12:15:28.845875 12568 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:28.848928 12568 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 13
I20260709 12:15:28.851387 20978 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:28.852213 12568 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:15:28.874816 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.745234 12566 rpcz_store.cc:276] 0709 12:15:26.639486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.224866 (+585380us) service_pool.cc:224] Handling call
0709 12:15:27.225265 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:27.225683 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:27.225712 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:27.226274 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 12:15:28.104453 (+878179us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b18004b3160. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:28.374645 (+270192us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:28.374753 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:28.374900 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:28.698909 (+324009us) raft_consensus.cc:1685] finished
0709 12:15:28.698971 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:28.699647 (+   676us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10469,"raft.queue_time_us":194,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":9666,"spinlock_wait_cycles":2485632,"thread_start_us":269799,"threads_started":1,"wal-append.queue_time_us":1234943}
I20260709 12:15:28.876022 12568 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:28.874514 21254 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:15:28.888154 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 198545us
I20260709 12:15:28.891376 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:28.892738 21254 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:28.911335 12566 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:28.911533 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:28.921233 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:28.924070 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:28.924739 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:28.926128 12492 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.10-25.10]
I20260709 12:15:28.926604 20978 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:28.928340 20978 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:28.928720 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46739.5us
I20260709 12:15:28.895713 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.931370 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:28.931807 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:28.934262 12572 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:15:28.938172 21099 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:28.928750 21100 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:28.927317 21124 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:28.940202 12572 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:15:28.942909 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.907598 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:28.947906 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:28.948740 12571 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:28.950591 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:28.911386 21292 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.962843 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:28.968077 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64575.9us
I20260709 12:15:28.977051 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:28.957774 21124 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:28.980226 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:28.976150 21163 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:28.979995 20981 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:28.993237 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:29.001384 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 210792us
I20260709 12:15:29.004050 20981 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.009697 20981 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:29.010352 21163 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.013226 21163 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:28.971772 21100 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.022783 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.024847 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.025880 12490 raft_consensus.cc:2376] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29. Re-sending same reply.
I20260709 12:15:29.021155 12492 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:29.029181 12493 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:29.029592 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:29.037572 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 169565us
W20260709 12:15:29.039649 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 108090us
I20260709 12:15:29.040126 13256 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:29.043540 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.044394 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:29.056895 12497 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:29.066139 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.078186 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:29.091312 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:29.110046 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.118544 12572 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:29.099210 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:29.145385 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:29.110913 12611 connection.cc:256] RPC call timeout handler was delayed by -1.0434620900000482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 12:15:29.146979 12611 connection.cc:256] RPC call timeout handler was delayed by -1.0317620900000293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
I20260709 12:15:29.102229 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.148157 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.152827 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:29.158479 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.154450 12492 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 12:15:29.160724 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.157421 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3641) took 1066 ms. Trace:
I20260709 12:15:29.163576 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:29.166730 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82538.2us
W20260709 12:15:29.169101 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66906.5us
W20260709 12:15:29.173547 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104082us
I20260709 12:15:29.177932 21177 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:15:29.178771 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 737950us
I20260709 12:15:29.179071 21177 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:29.178117 21212 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.565s	user 0.020s	sys 0.013s
W20260709 12:15:29.184355 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:29.181957 12570 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.709s	user 0.005s	sys 0.023s
W20260709 12:15:29.186825 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138644us
I20260709 12:15:29.171939 21254 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:29.191989 21305 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:29.163904 12564 rpcz_store.cc:276] 0709 12:15:27.990234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.990381 (+   147us) service_pool.cc:224] Handling call
0709 12:15:28.005484 (+ 15103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:28.005718 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:15:28.021495 (+ 15777us) log.cc:844] Serialized 14 byte log entry
0709 12:15:28.736301 (+714806us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b1800359fe0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:15:29.034159 (+297858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:29.042201 (+  8042us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:29.045337 (+  3136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:29.048449 (+  3112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:29.056662 (+  8213us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15619,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":3073,"raft.queue_time_us":17338,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":7240,"spinlock_wait_cycles":12280704,"thread_start_us":255729,"threads_started":1,"wal-append.queue_time_us":1095984}
I20260709 12:15:29.203544 13258 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:29.205552 20662 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:29.206396 20662 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.206776 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:29.206926 20662 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:29.209022 12564 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:29.194311 21100 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:29.207250 21233 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.014s
I20260709 12:15:29.225867 21171 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:29.221269 20981 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:29.228297 21132 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.581s	user 0.018s	sys 0.004s
W20260709 12:15:29.229156 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 163494us
I20260709 12:15:29.229907 12646 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 27 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 12.
I20260709 12:15:29.231215 21157 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:29.231673 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.231971 21157 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:29.232540 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3476) took 1756 ms (client timeout 1569 ms). Trace:
I20260709 12:15:29.233319 21160 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.233841 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.234196 21160 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.235389 20981 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:29.236210 21157 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:29.239661 13253 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:15:29.240850 21233 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.241730 21233 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.242861 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:29.244328 21233 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:29.244706 13262 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:15:29.245404 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:29.246366 13259 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:29.248448 21344 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.250725 21233 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:29.249409 21344 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:29.255262 21346 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:15:29.198238 12570 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:29.256069 21344 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:29.259057 12570 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260709 12:15:29.259539 12570 pending_rounds.cc:85] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:15:29.259835 12570 pending_rounds.cc:107] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:15:29.260110 12570 raft_consensus.cc:2889] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:29.241065 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:29.181147 21177 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.225080 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:29.189424 21254 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:29.221233 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:29.233171 12646 rpcz_store.cc:269] 0709 12:15:27.476063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:27.478887 (+  2824us) service_pool.cc:224] Handling call
0709 12:15:29.232089 (+1753202us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:29.232391 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4002914560}
I20260709 12:15:29.270167 21254 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:15:29.274852 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53772.7us
I20260709 12:15:29.293503 12566 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:15:29.302313 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.299743 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:29.240990 21131 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:29.291917 21163 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:29.269616 21160 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:29.321362 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.322062 12492 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
W20260709 12:15:29.326915 21348 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.180s	user 0.002s	sys 0.000s
W20260709 12:15:29.327762 21307 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.700s	user 0.012s	sys 0.010s
W20260709 12:15:29.327875 21342 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.332s	user 0.004s	sys 0.013s
W20260709 12:15:29.328387 21000 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.911s	user 0.000s	sys 0.028s
I20260709 12:15:29.288686 21174 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:15:29.330472 12567 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:29.329602 13257 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:15:29.307708 21171 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:29.336385 21038 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:29.340062 20582 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.949s	user 0.015s	sys 0.006s
I20260709 12:15:29.341900 12391 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:29.302050 21100 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:15:29.346480 21168 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:29.349136 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.320549 21163 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.350035 12492 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:29.329036 21000 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.335623 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3632) took 1024 ms. Trace:
I20260709 12:15:29.352660 12569 rpcz_store.cc:276] 0709 12:15:28.315314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:28.439271 (+123957us) service_pool.cc:224] Handling call
0709 12:15:29.313616 (+874345us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:29.335517 (+ 21901us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595439360}
I20260709 12:15:29.353181 21000 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:29.354147 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:29.354463 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54776us
I20260709 12:15:29.354911 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:29.356353 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131507us
W20260709 12:15:29.357975 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.359500 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.360328 12489 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:29.361166 20444 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.905s	user 0.019s	sys 0.002s
I20260709 12:15:29.361958 21274 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.362622 21274 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.363407 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3636) took 1044 ms. Trace:
I20260709 12:15:29.363859 12571 rpcz_store.cc:276] 0709 12:15:28.320387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:28.410820 (+ 90433us) service_pool.cc:224] Handling call
0709 12:15:29.363139 (+952319us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:29.363340 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":657680256}
I20260709 12:15:29.365336 12572 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:15:29.366325 21292 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:29.367023 21292 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.332439 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.335009 21344 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:29.335399 21131 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.335589 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:29.369119 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:29.373806 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:29.376600 21131 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:29.368026 20582 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.378800 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.381171 20662 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:29.383605 21345 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.388s	user 0.004s	sys 0.000s
I20260709 12:15:29.349983 21163 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:29.389472 21344 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.391131 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3646) took 1023 ms. Trace:
W20260709 12:15:29.392187 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83919.4us
W20260709 12:15:29.393782 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:29.426285 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.367659 21274 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:29.395565 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 154758us
W20260709 12:15:29.367818 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55439.4us
I20260709 12:15:29.398141 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.398372 21132 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:15:29.399260 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 133110us
W20260709 12:15:29.399747 21351 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.002s
W20260709 12:15:29.400257 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.400316 12493 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:15:29.369771 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:29.407269 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.424338 12567 rpcz_store.cc:276] 0709 12:15:28.367686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:28.825859 (+458173us) service_pool.cc:224] Handling call
0709 12:15:29.325381 (+499522us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:29.353739 (+ 28358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:29.390788 (+ 37049us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54001d7250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:29.391002 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1143167232}
W20260709 12:15:29.408021 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50256.8us
I20260709 12:15:29.376613 21292 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:29.415932 20582 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.416613 12391 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 23, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 23 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:29.419286 20662 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:29.427359 12489 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:29.431521 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:29.447501 12488 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:29.452889 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:29.466048 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 225) took 35069 ms (client timeout 15000 ms). Trace:
I20260709 12:15:29.469228 20582 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:29.454793 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:29.456550 20859 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:29.439769 12391 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 22, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 22 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:29.457837 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:29.457954 21131 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:29.461208 21124 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.452982 12496 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:29.486261 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44734.5us
I20260709 12:15:29.496901 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:29.507264 21344 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:29.509918 21174 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:29.510507 21174 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.510876 21174 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:29.511782 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3651) took 1102 ms. Trace:
I20260709 12:15:29.512382 12568 rpcz_store.cc:276] 0709 12:15:28.409550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:28.847452 (+437902us) service_pool.cc:224] Handling call
0709 12:15:28.848330 (+   878us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:28.961377 (+113047us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:28.961431 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:29.004592 (+ 43161us) log.cc:844] Serialized 26 byte log entry
0709 12:15:29.162174 (+157582us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:29.162216 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:29.162303 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:29.328930 (+166627us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:29.511195 (+182265us) raft_consensus.cc:1685] finished
0709 12:15:29.511237 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:29.511687 (+   450us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3624832,"thread_start_us":128291,"threads_started":1,"wal-append.queue_time_us":252870}
I20260709 12:15:29.504001 12391 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 16 to 22, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 22 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:29.444031 21171 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:15:29.495402 21132 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:29.524870 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.525252 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.504021 20662 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:29.526933 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:29.504624 21131 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:29.531281 12493 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:15:29.496802 20582 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:15:29.530833 21344 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
W20260709 12:15:29.466876 12387 rpcz_store.cc:269] 0709 12:14:54.375293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:54.375580 (+   287us) service_pool.cc:224] Handling call
0709 12:14:57.382195 (+3006615us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:29.440544 (+32058349us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:15:18.392109 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:15:18.392949 (+   840us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:15:18.393018 (+    69us) write_op.cc:435] Acquired schema lock
0709 12:15:18.393052 (+    34us) tablet.cc:662] Decoding operations
0709 12:15:18.525744 (+132692us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:15:18.543680 (+ 17936us) write_op.cc:641] Partition lock acquired for write op
0709 12:15:18.544881 (+  1201us) tablet.cc:685] Acquiring locks for 24 operations
0709 12:15:18.738915 (+194034us) tablet.cc:701] Row locks acquired
0709 12:15:18.738974 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:15:18.739230 (+   256us) write_op.cc:270] Start()
0709 12:15:18.739438 (+   208us) write_op.cc:276] Timestamp: P: 1783599318739151 usec, L: 0
0709 12:15:18.739492 (+    54us) op_driver.cc:348] REPLICATION: starting
0709 12:15:18.742513 (+  3021us) log.cc:844] Serialized 9432 byte log entry
0709 12:15:20.010020 (+1267507us) op_driver.cc:464] REPLICATION: finished
0709 12:15:20.256336 (+246316us) write_op.cc:301] APPLY: starting
0709 12:15:20.256439 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 12:15:21.671895 (+1415456us) tablet.cc:1370] finished BulkCheckPresence
0709 12:15:21.707886 (+ 35991us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:15:22.316451 (+608565us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:15:22.574895 (+258444us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=0,mrs_lookups=0
0709 12:15:22.617754 (+ 42859us) write_op.cc:312] APPLY: finished
0709 12:15:22.650961 (+ 33207us) log.cc:844] Serialized 212 byte log entry
0709 12:15:22.798958 (+147997us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:22.809385 (+ 10427us) write_op.cc:454] Released schema lock
0709 12:15:22.809504 (+   119us) write_op.cc:341] FINISH: Updating metrics
0709 12:15:22.830164 (+ 20660us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:22.830209 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":505436,"reported_tablets":51,"spinlock_wait_cycles":3038745856,"child_traces":[["op",{"apply.queue_time_us":228160,"apply.run_cpu_time_us":73964,"apply.run_wall_time_us":2573780,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104351,"cfile_init":1,"lbm_read_time_us":492,"lbm_reads_lt_1ms":4,"num_ops":24,"prepare.queue_time_us":276458,"prepare.run_cpu_time_us":41134,"prepare.run_wall_time_us":909336,"raft.queue_time_us":146807,"raft.run_cpu_time_us":46277,"raft.run_wall_time_us":224024,"replication_time_us":1238064,"thread_start_us":1021618,"threads_started":5,"wal-append.queue_time_us":829440,"wal-append.run_cpu_time_us":15350,"wal-append.run_wall_time_us":2526902}]]}
I20260709 12:15:29.537012 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.494859 21124 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.515241 21254 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.532732 20662 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:29.535504 21100 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:15:29.550879 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96169.9us
W20260709 12:15:29.555612 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111286us
I20260709 12:15:29.556684 12490 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:29.560978 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120140us
I20260709 12:15:29.545830 21292 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:29.580049 20859 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:29.565832 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:29.582502 13257 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:29.582248 21274 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:29.583115 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:29.584007 12567 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:29.569957 12391 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:29.586239 21100 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:15:29.587563 21100 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:29.573561 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.574483 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.588686 12493 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:29.575531 21132 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:29.553884 20662 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:29.590005 21132 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:29.590523 21132 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:29.591656 21132 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.593321 21132 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:29.570693 21131 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.583906 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:29.598088 12568 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:29.587981 21100 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:29.588824 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:29.589809 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:29.578198 21254 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:29.575851 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:29.576947 21124 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:29.599653 12493 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:29.587597 12391 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 25, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 25 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:29.611822 21038 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:29.612079 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.612126 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.607612 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3644) took 1240 ms. Trace:
I20260709 12:15:29.620775 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.621779 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
W20260709 12:15:29.624948 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63131.6us
I20260709 12:15:29.619004 12563 rpcz_store.cc:276] 0709 12:15:28.367189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:28.830488 (+463299us) service_pool.cc:224] Handling call
0709 12:15:29.029934 (+199446us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:29.030135 (+   201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:29.030503 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:29.117957 (+ 87454us) log.cc:844] Serialized 14 byte log entry
0709 12:15:29.607177 (+489220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:29.607253 (+    76us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:29.607318 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:29.607400 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:29.607548 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32474,"prepare.queue_time_us":185171,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":75,"spinlock_wait_cycles":3772288,"thread_start_us":475230,"threads_started":3,"wal-append.queue_time_us":402874}
W20260709 12:15:29.638386 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 229563us
I20260709 12:15:29.604974 21292 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:29.638952 12496 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:29.650554 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.642848 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:29.647960 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72225.6us
I20260709 12:15:29.650799 12492 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:29.641686 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:29.647277 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:29.650947 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:29.660140 12391 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 19 to 20, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 20 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:29.662801 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:29.663542 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:29.651389 21292 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:29.665784 21016 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:29.666282 21016 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.666616 21016 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:29.671904 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83254.3us
I20260709 12:15:29.674222 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.677978 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:29.678015 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:29.646863 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 141494us
I20260709 12:15:29.656656 21124 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:29.691403 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65378.7us
I20260709 12:15:29.692732 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:29.685712 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:29.689827 12391 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:15:29.709372 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51161us
I20260709 12:15:29.684072 21099 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:29.671386 21038 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:29.695115 21362 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:29.719959 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:29.671111 21174 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:29.739511 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49289us
I20260709 12:15:29.690719 21344 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.748773 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:29.780963 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.772603 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:29.719648 21177 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:29.721195 21099 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.719395 21124 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:29.809324 13260 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
W20260709 12:15:29.809487 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64156.8us
I20260709 12:15:29.739418 20582 consensus_queue.cc:1243] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 26.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:15:29.818797 12563 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:15:29.739714 21274 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:29.815459 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:29.795234 21307 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:29.821997 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81737.6us
I20260709 12:15:29.815078 21163 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:29.827589 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135155us
I20260709 12:15:29.762017 21212 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:15:29.804785 21099 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:29.826350 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:29.735715 21100 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.846586 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3643) took 1479 ms. Trace:
I20260709 12:15:29.786497 21362 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:29.876264 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44061.3us
I20260709 12:15:29.840054 21274 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:29.826116 21160 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.045s
W20260709 12:15:29.870792 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98339.7us
I20260709 12:15:29.830523 21233 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:29.872149 21307 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:29.861215 21163 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:29.882191 21021 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.887624 21374 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.888626 20662 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.815130 21038 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.894289 21100 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:29.933977 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48427.2us
I20260709 12:15:29.895383 21124 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.9, Last appended by leader: 9, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:29.872660 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46408.9us
I20260709 12:15:29.929317 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:29.928572 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.876001 21131 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:29.894444 21000 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:29.842034 21344 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.897249 21274 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:29.942531 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 251777us
I20260709 12:15:29.924750 21307 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.868640 12565 rpcz_store.cc:276] 0709 12:15:28.366891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:28.830198 (+463307us) service_pool.cc:224] Handling call
0709 12:15:28.873732 (+ 43534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:28.969672 (+ 95940us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:29.163930 (+194258us) log.cc:844] Serialized 14 byte log entry
0709 12:15:29.553376 (+389446us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:29.648830 (+ 95454us) log.cc:844] Serialized 26 byte log entry
0709 12:15:29.649087 (+   257us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:29.649148 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:29.649252 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:29.846154 (+196902us) raft_consensus.cc:1685] finished
0709 12:15:29.846192 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:29.846475 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5013,"prepare.queue_time_us":201674,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":151863,"raft.run_cpu_time_us":656,"raft.run_wall_time_us":44484,"spinlock_wait_cycles":16976768,"thread_start_us":521497,"threads_started":3,"wal-append.queue_time_us":358076}
I20260709 12:15:29.932566 20859 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:15:29.957897 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 269834us
I20260709 12:15:29.921211 20582 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:29.935308 21370 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:15:29.963577 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.943109 21131 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:29.964090 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87019.4us
I20260709 12:15:29.942584 21233 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:29.931403 21359 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:29.972005 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.958971 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:29.973101 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:29.953691 21274 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:29.958249 21307 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:15:29.984328 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:29.984951 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:29.985726 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:29.953130 21177 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:29.973196 21370 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:29.988695 12489 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:29.949152 21000 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.990118 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.974179 21131 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:29.993005 12490 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: [26.16-27.17]
I20260709 12:15:29.992833 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:29.999820 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:30.012095 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82918.1us
I20260709 12:15:30.012463 12497 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 18. (term mismatch)
I20260709 12:15:29.952785 21021 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:29.976308 21368 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:30.011430 21378 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:30.030166 21378 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:30.030650 21378 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:30.033615 21379 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:30.011471 20582 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.034314 20859 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:30.011435 20978 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:30.034967 20859 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:30.034363 21379 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:30.037317 21379 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:30.038545 21038 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.039362 21038 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:30.039743 21038 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:30.040089 21038 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:30.040647 21356 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.369s	user 0.001s	sys 0.003s
I20260709 12:15:30.041795 21365 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:30.042518 21365 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:30.042769 12639 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:15:30.042923 21365 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:30.044978 12638 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:30.012938 12497 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 17
I20260709 12:15:30.055294 12497 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 22.18
I20260709 12:15:30.055680 12497 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:30.025990 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:30.058159 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77205.5us
I20260709 12:15:30.077900 12643 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
W20260709 12:15:30.027066 21367 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:15:30.086416 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:30.015151 21366 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.188s	user 0.001s	sys 0.001s
I20260709 12:15:29.972839 21374 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.034845 20662 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.089131 12462 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:30.013396 21370 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:30.111270 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:30.114286 12572 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:30.116941 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139949us
I20260709 12:15:29.999729 21016 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:29.999231 21383 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.119s
I20260709 12:15:30.010664 21359 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:30.126413 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131861us
I20260709 12:15:30.126454 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.144692 12638 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:30.150509 12643 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:30.099412 21132 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:15:30.153159 21132 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:30.154692 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.155848 12641 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
W20260709 12:15:30.157840 21357 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.267s	user 0.000s	sys 0.002s
I20260709 12:15:30.158877 12645 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 11. Preceding OpId from leader: term: 26 index: 12. (index mismatch)
I20260709 12:15:30.160449 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.160969 21374 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.161770 21387 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:30.157241 21344 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:30.163043 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103497us
I20260709 12:15:30.163826 21132 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.164474 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:30.165246 12494 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:30.166230 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.143575 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:30.168437 21372 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:15:30.168736 12639 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:30.169935 21363 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:30.170612 21363 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:30.067502 21368 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.171904 21021 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:30.171602 21363 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.172763 21233 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.174865 21132 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:15:30.158869 12637 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:30.157254 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.128336 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:30.180011 21368 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.163066 21344 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:30.220929 21344 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:30.222041 21344 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.223464 21344 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:30.224401 21292 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.004s
I20260709 12:15:30.162078 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.228750 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:30.231941 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:30.209921 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.600s (SENT) [suppressed 13 similar messages]
W20260709 12:15:30.234701 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54888.7us
I20260709 12:15:30.240715 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:30.243099 21000 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.245252 21174 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:30.247861 21166 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:30.250152 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.251253 12645 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 27 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 12.
I20260709 12:15:30.252517 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.253161 12645 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:30.258790 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.263880 12643 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:30.265429 12639 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:30.259601 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:30.106515 21177 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.290936 12494 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:30.274786 13258 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:15:30.272675 21132 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:30.308123 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.308370 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.309373 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.310431 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:30.318431 12645 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
W20260709 12:15:30.320255 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 203871us
W20260709 12:15:30.324352 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 179761us
I20260709 12:15:30.329463 12640 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:15:30.231818 21292 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:30.333331 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92704.8us
I20260709 12:15:30.323789 21395 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:30.337661 21370 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:30.335961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85760.3us
I20260709 12:15:30.341418 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:30.342823 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112606us
I20260709 12:15:30.350517 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.208667 21387 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.356217 21000 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:30.357141 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:30.335259 21132 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:30.360107 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.334403 21362 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.306198 21174 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
W20260709 12:15:30.365557 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44866.4us
I20260709 12:15:30.364204 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.369007 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.372148 12639 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:30.372421 12643 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:15:30.376768 12638 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:30.382946 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:30.383783 12496 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:30.383844 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.385398 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.386251 12645 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:30.390017 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:30.391057 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.391979 12639 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:30.392225 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.392982 12645 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:30.393281 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.389994 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:30.398288 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:30.399010 21016 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.399839 21362 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:30.400415 21362 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:30.400852 21362 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:30.388129 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:30.398106 12494 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:15:30.368940 20582 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.412352 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.415139 12643 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:30.360344 21393 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:30.397936 21168 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:30.352718 21395 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.393026 21131 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.426072 12643 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:30.409308 21132 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:30.440798 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.442198 12643 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:30.420656 21168 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:30.393852 20859 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.452000 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.458036 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:30.414238 21233 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:15:30.459513 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.460203 12490 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:30.445931 21100 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:30.464861 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.446970 21168 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:30.467833 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:30.465803 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:30.431715 21021 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:15:30.473047 12492 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:15:30.450919 21395 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:30.479826 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:30.463258 21099 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.005s
I20260709 12:15:30.491326 12493 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.13-24.13]
W20260709 12:15:30.484624 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127652us
I20260709 12:15:30.429816 21393 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.499924 21132 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.502182 21393 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:30.503026 21387 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.465826 21100 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:30.504317 21131 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.505002 12642 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:30.504892 21100 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.505437 21171 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:30.465096 21368 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:30.507831 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.511979 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:30.507952 21368 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:30.478542 21359 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.508597 21000 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:30.509126 21212 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:30.509265 21404 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:15:30.509766 21166 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:30.493789 21099 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.558885 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.499821 21292 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.562147 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.499238 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:30.502347 21403 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:15:30.480669 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:30.565317 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84698.9us
I20260709 12:15:30.567438 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.567929 21292 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:30.568359 12493 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:30.568583 21292 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:30.569654 21292 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:30.501435 21402 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:15:30.551198 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:30.571425 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.572065 21405 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:30.572821 21405 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.573843 21387 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:30.574704 21387 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.575953 21405 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.577330 21387 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.574393 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.549026 12642 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:30.579804 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.484885 21381 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:15:30.553442 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95149.5us
I20260709 12:15:30.548978 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.582984 12496 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:30.583300 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.626752 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:30.628539 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:30.616909 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:30.484170 21038 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.488533 21374 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.142s
I20260709 12:15:30.477380 20582 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.148s
W20260709 12:15:30.754101 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125623us
W20260709 12:15:30.636720 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54403.1us
I20260709 12:15:30.755842 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.583526 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:30.750910 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.637498 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:30.740664 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 241502us
W20260709 12:15:30.681684 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53062.3us
I20260709 12:15:30.637910 13260 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:15:30.761217 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:30.647301 21396 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:15:30.779636 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.781762 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.783396 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:30.647235 21171 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.115s
I20260709 12:15:30.647018 21414 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.116s
I20260709 12:15:30.631711 12494 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:30.587556 21381 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:30.751312 21415 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:15:30.581445 21212 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.550518 21368 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:30.764185 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.650528 21166 raft_consensus.cc:2764] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:15:30.764736 21168 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.586097 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:30.827530 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:30.548115 21363 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:30.775966 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:30.647859 21364 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:30.585553 21016 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:30.629807 21000 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:30.790206 21344 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.566s	user 0.013s	sys 0.001s
I20260709 12:15:30.854548 12492 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:15:30.723534 21177 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:30.814033 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:30.829556 12644 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:30.766958 20859 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:30.585014 21359 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:30.741094 21274 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.742s	user 0.007s	sys 0.007s
I20260709 12:15:30.807893 21381 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:30.649811 21395 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:30.842110 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 497374us
W20260709 12:15:30.849478 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57091us
I20260709 12:15:30.859546 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:30.844008 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:30.637815 21370 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.679987 21387 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:30.808295 21132 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:30.851356 21124 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:15:30.892477 21362 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.843283 21414 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:15:30.849231 21016 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:30.855867 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.555871 21393 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.280s
W20260709 12:15:30.860110 21378 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.829s	user 0.010s	sys 0.004s
I20260709 12:15:30.835382 21364 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:30.899370 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85437.5us
I20260709 12:15:30.921842 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.945312 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:30.900336 12639 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:30.913902 21233 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:30.952987 12493 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:30.920115 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:30.903053 21410 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:15:30.935043 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:30.901952 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:30.949941 21381 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.927191 21100 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:30.918659 21000 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.934337 21393 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:30.900764 21274 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:30.998136 21393 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:30.999035 12497 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:30.934957 21212 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:30.903784 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105875us
I20260709 12:15:30.850458 21374 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:30.927139 12492 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: [25.12-25.12]
I20260709 12:15:30.899999 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:30.900630 21359 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:30.932669 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79550.3us
W20260709 12:15:30.907809 21307 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.915s	user 0.013s	sys 0.001s
I20260709 12:15:31.024821 12567 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:15:31.025449 12567 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:31.024938 21307 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.025831 12567 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:15:31.026170 12567 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:31.026723 21359 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.973618 12637 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:31.005483 21423 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:31.028343 12490 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:31.028148 21423 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.027786 21307 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.915602 21016 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.030664 21016 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:31.030851 21423 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:31.032286 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134526us
I20260709 12:15:31.032903 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:31.033689 12571 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:31.037514 13260 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:31.038405 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:30.975018 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:30.907105 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:30.975059 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.039391 12645 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:30.953507 12643 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
W20260709 12:15:30.959369 21379 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.921s	user 0.015s	sys 0.000s
I20260709 12:15:30.905821 21387 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:30.953307 20582 consensus_queue.cc:1243] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:15:30.961668 21177 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:30.993961 21381 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.994186 21362 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:30.903410 21168 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:30.902129 20913 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.190s
I20260709 12:15:30.908221 21124 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:30.906885 21132 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.060913 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.007331 21131 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:15:31.007769 21365 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.964s	user 0.004s	sys 0.009s
I20260709 12:15:31.008351 21212 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:31.009866 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.010653 21000 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:31.016168 12564 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:15:31.077096 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:31.070890 21421 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:15:31.077961 12564 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:31.016731 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:31.078892 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.080125 12643 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 27 because replica has last-logged OpId of term: 25 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 9.
I20260709 12:15:31.080230 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:31.017446 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:30.965130 21364 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.041092 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:30.980422 21233 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:31.086849 21233 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:31.042650 12637 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:31.044854 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
W20260709 12:15:31.051252 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97340.9us
I20260709 12:15:31.052591 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:31.000715 21099 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:31.054659 12565 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:31.055013 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.089051 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.053323 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:31.021127 21374 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:31.059630 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.258628 12570 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:30.935041 21378 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.065279 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:31.052903 21411 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:31.065994 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:31.259822 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.260628 12645 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:31.068118 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.061221 21124 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:31.070730 21131 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:31.052732 21021 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 12:15:31.070701 21177 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.071707 21398 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:31.071424 21365 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.071903 21132 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:31.066061 21395 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20260709 12:15:31.073964 21387 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:31.074136 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.082124 12568 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:31.079013 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:31.081173 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.276515 12643 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:31.081187 12572 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:31.083514 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 181692us
I20260709 12:15:31.086054 12567 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:31.047241 21379 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:31.093801 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.094878 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.284169 12639 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:31.087432 21038 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:31.127249 21428 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.134s	user 0.002s	sys 0.000s
I20260709 12:15:31.127476 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.128084 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.268805 21131 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:31.179318 21412 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:15:31.181003 12461 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.569s (SENT)
I20260709 12:15:31.181931 21274 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:31.182123 21099 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:31.182766 21406 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.189s	user 0.008s	sys 0.010s
I20260709 12:15:31.275956 21398 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:31.300071 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.276121 21411 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:31.305927 12571 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:31.307013 21378 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:31.087049 21380 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:31.187664 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98709.1us
I20260709 12:15:31.190605 21364 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:31.194495 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.190842 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:31.196024 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3502) took 1035 ms. Trace:
I20260709 12:15:31.199182 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:31.293704 21412 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:31.193985 21370 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:31.220002 21233 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.185271 21212 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:31.192230 21000 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:15:31.230257 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.235528 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:31.237409 12642 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.657s	user 0.007s	sys 0.008s
I20260709 12:15:31.336052 12642 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
I20260709 12:15:31.089592 12569 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:30.999472 21393 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:31.261961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 193907us
I20260709 12:15:31.339108 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.339457 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.339725 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.340495 12493 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:31.340883 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.266328 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.339129 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:31.341989 12645 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:31.342063 13259 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:31.273972 12646 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:31.338722 12497 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:15:31.259696 21381 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:31.266579 21374 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.280531 12564 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:31.285907 12563 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:31.286908 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.269014 21362 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:31.289093 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.291133 12637 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
W20260709 12:15:31.178624 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89063us
I20260709 12:15:31.292936 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:31.295819 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.297689 12641 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:31.354868 12638 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:15:31.355295 12639 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:31.353322 20582 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:31.356079 20582 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:31.356489 20582 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:31.299578 21099 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.286973 21426 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:31.330423 21370 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:31.310490 21380 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.315145 13260 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:31.200062 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:31.319161 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.319840 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.320772 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.328586 13256 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:31.314939 12644 rpcz_store.cc:276] 0709 12:15:30.152464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:30.165799 (+ 13335us) service_pool.cc:224] Handling call
0709 12:15:31.128970 (+963171us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:31.187783 (+ 58813us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1517558016}
I20260709 12:15:31.333721 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:31.301594 21344 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:31.343492 12570 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:31.338501 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.346621 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.350786 12643 raft_consensus.cc:2376] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20. Re-sending same reply.
I20260709 12:15:31.356369 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.182823 21368 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.354125 12565 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:31.359616 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:31.362329 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.302295 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:31.417326 12645 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:31.364553 12567 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:31.367743 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186767us
I20260709 12:15:31.369064 13253 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
W20260709 12:15:31.420264 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.370018 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:31.370776 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:31.371357 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.356808 21379 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:31.359817 21131 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.370054 21393 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:31.428627 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.429311 12645 raft_consensus.cc:2376] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25. Re-sending same reply.
I20260709 12:15:31.429651 21131 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:31.430405 21359 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:31.377055 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3491) took 1415 ms. Trace:
I20260709 12:15:31.431036 21359 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.431468 12646 rpcz_store.cc:276] 0709 12:15:29.971644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:29.976175 (+  4531us) service_pool.cc:224] Handling call
0709 12:15:29.983077 (+  6902us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.351345 (+1368268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.376682 (+ 25337us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:31.376939 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2968192768}
I20260709 12:15:31.379170 21099 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:31.419682 21344 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:31.436523 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:31.425742 21292 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:31.438171 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:31.439760 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106148us
I20260709 12:15:31.440840 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:31.441746 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82206us
I20260709 12:15:31.450191 12568 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:31.455125 12569 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:31.379684 21378 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:15:31.330305 21274 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:15:31.388825 21233 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:31.381558 13259 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:31.389752 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.433390 21359 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:31.379213 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3507) took 1132 ms. Trace:
I20260709 12:15:31.469118 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:31.394009 21411 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.469254 12641 rpcz_store.cc:276] 0709 12:15:30.246349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:30.258456 (+ 12107us) service_pool.cc:224] Handling call
0709 12:15:31.378910 (+1120454us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54002a44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:31.379128 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2383332608}
I20260709 12:15:31.469892 13259 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:31.474440 12490 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:31.475085 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.397737 21363 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:15:31.399158 21365 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:31.351941 21387 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:31.399982 21412 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.400420 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:31.400967 21398 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.363880 21426 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:31.493245 13262 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:31.409003 21374 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:31.416213 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.426723 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.427213 12567 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 16. (term mismatch)
I20260709 12:15:31.435422 21368 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:31.380987 21000 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:31.457365 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:31.501008 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.480633 21038 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:31.479045 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.481112 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.484639 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.485128 13257 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:31.438659 21292 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.481205 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.482921 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:31.484308 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:31.357914 21381 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.494299 12643 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:31.494272 12567 pending_rounds.cc:85] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:15:31.515501 12567 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:15:31.515903 12567 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:31.516332 12567 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:15:31.512715 21100 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:31.516958 21100 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:31.517355 21100 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:31.512981 21425 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:31.525775 12644 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:15:31.526546 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1147) took 1174 ms. Trace:
I20260709 12:15:31.526880 12644 rpcz_store.cc:276] 0709 12:15:30.352171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.524966 (+1172795us) service_pool.cc:224] Handling call
0709 12:15:31.525370 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.526243 (+   873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.526487 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:31.528644 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1149) took 1151 ms. Trace:
I20260709 12:15:31.488811 21395 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:15:31.530400 21395 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:31.530777 21395 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:31.531639 21395 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.532912 21395 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:31.485261 21307 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:15:31.534547 21307 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:31.534927 21307 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:31.535785 21307 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.537233 21307 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:31.488970 21383 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
I20260709 12:15:31.538009 21412 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:31.540714 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.435463 21160 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 12:15:31.542634 21160 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:31.543016 21160 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:31.436559 21344 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.504472 21038 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:31.545615 21344 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:31.546653 12638 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:31.506608 21378 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:31.547473 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 2976) took 1140 ms. Trace:
I20260709 12:15:31.547755 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.547878 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.549371 12494 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: [18.12-20.13]
W20260709 12:15:31.501151 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43941.6us
I20260709 12:15:31.463119 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.547884 12638 rpcz_store.cc:276] 0709 12:15:30.406645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.546118 (+1139473us) service_pool.cc:224] Handling call
0709 12:15:31.546462 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.547202 (+   740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.547409 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:31.466908 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:31.501775 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:31.528993 12644 rpcz_store.cc:276] 0709 12:15:30.377600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.527679 (+1150079us) service_pool.cc:224] Handling call
0709 12:15:31.528041 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.528263 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:31.528294 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:31.528344 (+    50us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:31.528379 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.528433 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:31.528562 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:31.506111 12566 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 26 index: 12. (term mismatch)
I20260709 12:15:31.427551 21212 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.069s
I20260709 12:15:31.481770 20662 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:31.508677 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.509020 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:31.511307 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:31.485064 21166 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:31.516582 12567 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:31.492642 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.566329 12493 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:31.562381 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:31.566169 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57240us
I20260709 12:15:31.544173 21426 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.549975 21292 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:31.551028 12568 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:31.491847 21132 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:31.551673 12495 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:31.500905 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.504828 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.502925 21233 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:15:31.559549 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92742.8us
W20260709 12:15:31.559933 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 265290us
I20260709 12:15:31.555308 12566 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:31.607357 12566 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 25.12
I20260709 12:15:31.560672 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.607702 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:31.608389 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47827.2us
I20260709 12:15:31.535451 21362 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:31.579705 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.597750 12640 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:31.560729 21038 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:31.601850 12644 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:15:31.607707 12566 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:31.609730 12638 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:15:31.619691 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1150) took 1206 ms. Trace:
I20260709 12:15:31.620352 12638 rpcz_store.cc:276] 0709 12:15:30.413495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.554245 (+1140750us) service_pool.cc:224] Handling call
0709 12:15:31.597266 (+ 43021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.619256 (+ 21990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.619598 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:31.565665 21166 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:31.499150 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.632915 12645 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:31.534924 21380 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:31.614493 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:31.612955 12640 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:31.578831 12643 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:31.625244 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1151) took 1192 ms. Trace:
I20260709 12:15:31.638172 12644 rpcz_store.cc:276] 0709 12:15:30.432460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.555850 (+1123390us) service_pool.cc:224] Handling call
0709 12:15:31.595486 (+ 39636us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.617738 (+ 22252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.625171 (+  7433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:31.578311 20913 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:31.578249 20978 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:31.547457 21378 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:31.649951 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.654739 12564 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:31.648473 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.533450 21124 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.665588 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.665470 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.657472 12638 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 10. Preceding OpId from leader: term: 26 index: 10. (term mismatch)
I20260709 12:15:31.606606 21233 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:31.675233 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.675016 21233 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.676575 21233 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:31.615764 21292 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:31.677539 21000 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:31.678107 21000 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:31.678992 21000 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.680222 12566 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:31.680392 21000 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:31.616466 21426 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:31.577929 21383 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.062s
I20260709 12:15:31.616698 21212 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:31.606925 21425 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:15:31.672365 21274 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:31.669211 21160 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:31.678203 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.696851 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.662566 12639 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:31.639780 21166 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:31.667474 12638 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 9
I20260709 12:15:31.720549 12638 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 25.10
I20260709 12:15:31.720947 12638 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:31.629873 21132 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:31.655815 20978 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:31.716593 12640 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:31.719817 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68342.3us
I20260709 12:15:31.663648 21362 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:31.659731 21168 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:15:31.777493 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:31.780495 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.722858 20913 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:15:31.784206 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.784996 12643 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
W20260709 12:15:31.786123 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.722368 21177 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:31.813277 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:31.776204 12671 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (6 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
W20260709 12:15:31.817397 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.801658 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.795869 12495 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:31.796694 12640 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:15:31.719081 21212 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:31.819998 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:31.844784 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67400.4us
W20260709 12:15:31.844954 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 282373us
I20260709 12:15:31.759686 20978 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:31.847100 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.850827 12571 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:15:31.781930 21160 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:31.784034 20913 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:31.859126 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.858197 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.863476 12571 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
W20260709 12:15:31.865952 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.779592 21362 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.861708 21429 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:31.853777 21021 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:15:31.883798 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 321461us
I20260709 12:15:31.859822 12640 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21. Re-sending same reply.
I20260709 12:15:31.928210 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.929451 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:31.930172 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:31.866915 13255 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:31.912932 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:31.930987 12644 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24. Re-sending same reply.
I20260709 12:15:31.929913 21132 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:31.937824 21124 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:31.938359 21307 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:15:31.938623 21124 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:31.938923 21307 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:31.939198 21124 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.939960 21307 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:31.942286 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:31.943326 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:31.946362 21393 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.000s
W20260709 12:15:31.943650 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:31.948680 21131 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.518s	user 0.010s	sys 0.000s
I20260709 12:15:31.951637 12642 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:15:31.815641 21177 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.929688 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.959354 12570 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:31.879923 21363 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:31.966662 12496 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: [23.12-23.12]
W20260709 12:15:31.900679 20582 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.543s	user 0.012s	sys 0.000s
I20260709 12:15:31.979863 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.980607 12489 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:31.989015 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.989305 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:31.990657 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:31.897744 20662 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:31.961651 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.957442 12494 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:15:32.000349 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:32.001183 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.852782 21100 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:32.002936 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:31.993003 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:31.963357 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43665.4us
I20260709 12:15:32.005028 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.005301 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:31.996961 12638 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:32.009938 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.010805 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:32.001067 13260 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.000s
I20260709 12:15:32.011283 12565 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:32.017264 13260 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:32.017619 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.019305 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.020620 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:32.016559 12640 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:31.966420 21381 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:32.031132 12639 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 30 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:15:32.032893 13261 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:32.022140 21131 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.038419 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.038573 21363 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:15:32.039413 12565 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:32.031198 21393 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:32.040398 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.038480 21131 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.041837 21453 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:15:32.042191 21442 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260709 12:15:32.043391 21131 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:32.032505 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:32.042275 21453 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:32.047029 21453 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.048431 21453 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:32.051023 21362 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.051637 12569 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:32.004458 20913 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:31.995535 21021 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.053337 12569 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:15:32.054062 21021 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.055140 21100 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.055874 12569 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [22.12-24.13]
I20260709 12:15:32.032301 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:31.955331 21429 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:32.056728 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.057142 21395 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:32.058646 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.059274 21379 raft_consensus.cc:2764] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:15:32.028122 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:31.992331 21124 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:31.980384 21274 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:32.056384 21021 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:32.072084 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.068928 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:32.117607 12494 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:32.118580 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:32.053256 20913 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:32.115516 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:32.115782 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87755.5us
I20260709 12:15:32.122018 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:15:32.125573 12496 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:32.126549 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:32.127734 12566 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:31.983067 20582 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.129074 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:32.134688 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.158010 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:32.138808 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:32.161820 12497 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:15:32.078662 21131 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:32.191213 12640 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:32.115566 21363 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:32.189033 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:32.200068 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:32.200637 12570 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:32.200946 12638 raft_consensus.cc:2376] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28. Re-sending same reply.
I20260709 12:15:32.079202 21393 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:32.129545 21411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.003s	sys 0.036s
I20260709 12:15:32.190455 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:32.204192 12644 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:32.193508 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:32.164952 21344 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.618s	user 0.012s	sys 0.001s
W20260709 12:15:32.203261 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.208330 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:32.127954 21168 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.153303 21016 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.081s
I20260709 12:15:32.004572 21212 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.114002 21378 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:32.215862 12494 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:15:32.209436 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 339063us
I20260709 12:15:32.225149 13257 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:32.217998 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.212459 12572 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:32.216288 12645 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:15:32.213858 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:32.230154 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.182451 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:32.232371 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.233305 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:32.219061 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:32.234328 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:32.235152 12639 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:32.181336 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:32.233984 12568 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:32.236083 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54836.8us
I20260709 12:15:32.216306 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:32.108681 21099 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.090s
W20260709 12:15:32.210417 21411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.003s	sys 0.039s
I20260709 12:15:32.263625 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:32.253916 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117609us
I20260709 12:15:32.269778 12569 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 20: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:32.066371 21370 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:32.270727 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135986us
I20260709 12:15:32.166200 21398 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:15:32.179853 21450 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.127s
I20260709 12:15:32.282724 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:32.285408 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.287073 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:32.290146 12638 raft_consensus.cc:2376] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23. Re-sending same reply.
W20260709 12:15:32.290899 12493 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.692s	user 0.013s	sys 0.000s
I20260709 12:15:32.291460 12569 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:32.294158 12493 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:32.253252 21344 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:15:32.256969 21412 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.006s
I20260709 12:15:32.300911 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:32.309468 12569 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:32.252717 21447 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:32.224750 21393 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.316280 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:32.321873 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:32.322054 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.243167 21168 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.328850 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3667) took 1271 ms. Trace:
I20260709 12:15:32.329250 12572 rpcz_store.cc:276] 0709 12:15:31.057125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.300841 (+243716us) service_pool.cc:224] Handling call
0709 12:15:32.328616 (+1027775us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:32.328788 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1836990336}
I20260709 12:15:32.330324 12571 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:15:32.334666 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:32.336354 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:32.244122 20662 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:32.340647 21448 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 12:15:32.356117 12568 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:32.337204 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:15:32.308564 21370 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:32.377698 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:32.390640 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 197175us
W20260709 12:15:32.376988 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186432us
W20260709 12:15:32.398764 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82585.4us
I20260709 12:15:32.393966 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1152) took 1942 ms. Trace:
I20260709 12:15:32.213590 21131 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.395748 12639 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:32.325367 21344 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:32.399926 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:32.302309 21445 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.092s
I20260709 12:15:32.442364 21463 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:32.395133 21450 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.489348 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.490209 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.491117 12566 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:15:32.323838 21274 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:32.492563 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.492534 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:32.499974 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100126us
I20260709 12:15:32.504752 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:32.443017 21166 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.689s	user 0.012s	sys 0.000s
I20260709 12:15:32.505486 12639 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:32.509749 12569 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:32.515158 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:32.484414 21177 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.530515 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1530) took 1316 ms. Trace:
I20260709 12:15:32.540117 12571 rpcz_store.cc:276] 0709 12:15:31.213927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.865914 (+651987us) service_pool.cc:224] Handling call
0709 12:15:32.330038 (+464124us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:32.330723 (+   685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:32.530281 (+199558us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b54003ab450. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:32.530442 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1066445696}
I20260709 12:15:32.516742 12536 leader_election.cc:400] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:32.541328 12536 leader_election.cc:403] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:15:32.531000 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:32.518231 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 285923us
I20260709 12:15:32.525018 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:32.545651 12639 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:15:32.546074 12639 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 13
I20260709 12:15:32.546317 12639 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:15:32.546819 12639 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:32.543846 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:32.528468 21166 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.548586 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:32.539523 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.546357 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.515177 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:32.512250 20662 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:32.516047 21425 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:32.585068 21425 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:32.585656 21425 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:32.588235 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:15:32.589066 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.515545 21447 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:15:32.518662 21378 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:32.590803 21378 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:32.591337 21378 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:32.541467 21132 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:32.542208 21021 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:32.549027 21446 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.517s	user 0.002s	sys 0.000s
I20260709 12:15:32.548806 21379 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.542011 21124 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.529036 21463 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:32.570387 21362 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:32.552505 21470 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:32.595722 12639 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:32.596684 12563 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
W20260709 12:15:32.600296 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90634.2us
I20260709 12:15:32.549513 21395 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:32.600942 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52691.5us
I20260709 12:15:32.602990 12488 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:32.555778 12572 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:32.556630 21429 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.511198 21450 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.569499 21099 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:32.594336 20582 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:32.636811 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.594552 21461 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:15:32.610733 21307 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.003s	sys 0.002s
W20260709 12:15:32.630014 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 225285us
I20260709 12:15:32.641819 21470 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:32.637493 21021 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:32.660112 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:32.655297 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.631955 21381 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:15:32.669751 21459 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.362s	user 0.000s	sys 0.002s
I20260709 12:15:32.671145 21131 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:32.680893 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:32.665963 21307 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.007s	sys 0.002s
I20260709 12:15:32.689949 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:32.690986 21455 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:15:32.691432 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.693066 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.594249 21449 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.098s
I20260709 12:15:32.686297 21381 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:32.694927 21381 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:32.687106 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3671) took 1613 ms. Trace:
W20260709 12:15:32.698297 21440 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.214s	user 0.000s	sys 0.000s
I20260709 12:15:32.630785 21474 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:15:32.672580 21470 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:32.700489 21168 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.697937 12567 rpcz_store.cc:276] 0709 12:15:31.074695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.595358 (+520663us) service_pool.cc:224] Handling call
0709 12:15:31.595792 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:31.617328 (+ 21536us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:31.664958 (+ 47630us) log.cc:844] Serialized 14 byte log entry
0709 12:15:32.036584 (+371626us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:32.084392 (+ 47808us) log.cc:844] Serialized 26 byte log entry
0709 12:15:32.126057 (+ 41665us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:32.130008 (+  3951us) log.cc:844] Serialized 14 byte log entry
0709 12:15:32.130428 (+   420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:32.130536 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:32.686818 (+556282us) raft_consensus.cc:1685] finished
0709 12:15:32.686845 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:32.687017 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40590,"prepare.queue_time_us":47001,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":9244,"raft.queue_time_us":18279,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6556,"spinlock_wait_cycles":54758144,"thread_start_us":241515,"threads_started":1,"wal-append.queue_time_us":962960}
W20260709 12:15:32.703856 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43864.9us
I20260709 12:15:32.656948 21177 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.705853 21466 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:32.706030 21393 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:32.706452 20662 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:32.663358 21429 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:32.708959 21474 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:32.709251 21429 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:32.709465 21474 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:32.709807 21429 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:32.710462 21474 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 26.10, Last appended by leader: 10, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.711922 21474 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:32.664280 21461 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:32.714461 21449 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:32.714957 21449 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:32.714413 21461 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.660909 21099 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.715884 21449 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.717104 21449 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:32.717921 21099 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:32.718600 21444 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:15:32.719830 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:32.630034 21124 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.721057 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1178) took 1404 ms. Trace:
I20260709 12:15:32.722014 21475 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:32.723479 21124 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.724476 21307 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:32.721697 12492 rpcz_store.cc:276] 0709 12:15:31.316301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.316536 (+   235us) service_pool.cc:224] Handling call
0709 12:15:31.337804 (+ 21268us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:31.339497 (+  1693us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:31.339533 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:31.359417 (+ 19884us) log.cc:844] Serialized 52 byte log entry
0709 12:15:31.410886 (+ 51469us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b180009e200. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:31.849990 (+439104us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:31.850033 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:31.850116 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:32.720654 (+870538us) raft_consensus.cc:1685] finished
0709 12:15:32.720677 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:32.720989 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1268,"raft.queue_time_us":1432,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":21002,"spinlock_wait_cycles":72327424,"thread_start_us":438874,"threads_started":1,"wal-append.queue_time_us":932084}
W20260709 12:15:32.726321 21465 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.290s	user 0.000s	sys 0.003s
W20260709 12:15:32.727531 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:32.730077 21468 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:15:32.732034 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3829) took 1284 ms. Trace:
I20260709 12:15:32.732244 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:32.732533 12488 rpcz_store.cc:276] 0709 12:15:31.447145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.447343 (+   198us) service_pool.cc:224] Handling call
0709 12:15:32.731723 (+1284380us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:32.731902 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2515712768}
I20260709 12:15:32.639137 21476 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.094s
I20260709 12:15:32.628521 21379 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.734997 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3681) took 1098 ms. Trace:
I20260709 12:15:32.735642 21379 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:32.735440 12564 rpcz_store.cc:276] 0709 12:15:31.636731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.881851 (+245120us) service_pool.cc:224] Handling call
0709 12:15:31.882249 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:31.912897 (+ 30648us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:15:31.923515 (+ 10618us) log.cc:844] Serialized 14 byte log entry
0709 12:15:32.061976 (+138461us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b18002b4800. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:15:32.245596 (+183620us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:32.246243 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 12:15:32.246381 (+   138us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:32.246410 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:32.246474 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:32.416662 (+170188us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b18002b4800. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:32.734756 (+318094us) raft_consensus.cc:1685] finished
0709 12:15:32.734782 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:32.734929 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10543,"prepare.queue_time_us":38983,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":62,"raft.queue_time_us":37925,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":23912,"spinlock_wait_cycles":59864832,"thread_start_us":241202,"threads_started":1,"wal-append.queue_time_us":517668}
I20260709 12:15:32.598021 21462 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.148s
I20260709 12:15:32.746110 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3518) took 1109 ms. Trace:
I20260709 12:15:32.752985 12645 rpcz_store.cc:276] 0709 12:15:31.637172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.637449 (+   277us) service_pool.cc:224] Handling call
0709 12:15:32.745879 (+1108430us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:32.746033 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262602112}
I20260709 12:15:32.746923 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:32.754248 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88085.6us
I20260709 12:15:32.770900 12488 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:32.772490 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:32.752321 12644 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.529s	user 0.011s	sys 0.000s
I20260709 12:15:32.776208 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:32.682747 20582 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:32.708953 21166 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:32.779371 21233 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.102s	user 0.014s	sys 0.003s
I20260709 12:15:32.756971 21379 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.637895 21344 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 27.13, Last appended by leader: 13, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.747355 21387 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:32.706024 21177 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.786985 21124 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:15:32.790201 21464 consensus_queue.cc:1175] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 23 status { last_received { term: 22 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:15:32.747430 21477 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:15:32.747727 21374 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:32.746531 21482 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:15:32.747625 21463 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.794275 21425 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:32.741616 21453 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.693s	user 0.004s	sys 0.007s
W20260709 12:15:32.783967 21000 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.099s	user 0.016s	sys 0.000s
I20260709 12:15:32.747519 21362 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:32.798460 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 227518us
I20260709 12:15:32.747151 21447 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:15:32.814711 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:32.780293 21168 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:32.795961 20582 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:32.852955 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 306671us
I20260709 12:15:32.884146 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.855651 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:32.865573 12490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.003s	sys 0.007s
I20260709 12:15:32.868273 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:32.868507 12643 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:15:32.822376 21426 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.133s	user 0.006s	sys 0.009s
W20260709 12:15:32.841244 20978 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.979s	user 0.014s	sys 0.000s
I20260709 12:15:32.823436 21124 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:32.890470 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:32.891366 12490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.004s	sys 0.008s
I20260709 12:15:32.866406 21482 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 12:15:32.911995 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139607us
I20260709 12:15:32.913486 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:32.914125 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:32.823539 21379 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:32.838495 21378 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.888918 20582 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:32.915098 12646 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:15:32.959192 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:32.939419 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:32.962201 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 269206us
I20260709 12:15:32.921948 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:32.910989 21038 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.267s	user 0.008s	sys 0.007s
I20260709 12:15:32.906244 21099 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:15:32.966848 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44980.3us
W20260709 12:15:32.905486 21160 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.043s	user 0.015s	sys 0.000s
W20260709 12:15:32.921814 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143261us
I20260709 12:15:32.969995 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.895179 21487 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:32.974153 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:32.908955 21124 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:32.917589 21233 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:32.987810 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48505.9us
I20260709 12:15:32.980795 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:32.914987 21471 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:32.981881 12495 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:15:32.975608 21393 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:32.990942 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:32.975662 21475 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:32.986083 13261 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:32.908679 20978 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:33.019533 20913 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.886s	user 0.010s	sys 0.001s
I20260709 12:15:33.021771 20978 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.968809 21426 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:33.022764 21370 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.000s
I20260709 12:15:33.023247 21307 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:33.023725 21307 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:33.024005 13260 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:15:33.024616 21307 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.921686 21482 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:33.027338 21381 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.028033 12493 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:15:33.028371 12493 pending_rounds.cc:85] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 15
I20260709 12:15:33.028586 12493 pending_rounds.cc:107] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:15:33.028800 12493 raft_consensus.cc:2889] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:33.030735 21378 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:33.023409 12492 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:33.038858 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58133.4us
I20260709 12:15:32.906801 21452 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:15:32.990803 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:33.053946 21469 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:33.054915 21374 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:32.993227 21484 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:15:32.409216 12641 rpcz_store.cc:276] 0709 12:15:30.451533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.559751 (+1108218us) service_pool.cc:224] Handling call
0709 12:15:31.577211 (+ 17460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:31.617118 (+ 39907us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:15:32.140601 (+523483us) log.cc:844] Serialized 14 byte log entry
0709 12:15:32.393458 (+252857us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:32.393529 (+    71us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:32.393592 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:32.393748 (+   156us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:32.393897 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15342,"raft.queue_time_us":16085,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":22641,"thread_start_us":638106,"threads_started":2}
I20260709 12:15:33.000353 21379 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.004526 21131 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:32.982282 21099 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:33.012531 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 2977) took 1064 ms. Trace:
I20260709 12:15:33.057679 21131 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:33.058154 21131 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:33.057955 12495 rpcz_store.cc:276] 0709 12:15:31.948481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.951291 (+  2810us) service_pool.cc:224] Handling call
0709 12:15:33.012270 (+1060979us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:33.012462 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2277051776}
I20260709 12:15:33.058741 21099 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:32.985841 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:33.015950 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:32.916296 21450 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:32.990337 21471 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:33.062299 21471 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:33.016752 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:33.046286 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.068208 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:33.069062 12564 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:15:33.069298 12567 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:33.050282 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:33.069986 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.070415 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.051680 12569 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:15:32.991878 21443 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:15:32.972730 21038 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.041826 21393 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.081076 12637 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:33.072247 12570 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:15:33.054874 21469 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:33.071190 12572 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:15:33.054019 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63298.7us
W20260709 12:15:33.076514 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90744.1us
I20260709 12:15:33.084582 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:33.085393 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:33.085868 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.057128 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1531) took 1167 ms. Trace:
I20260709 12:15:32.989861 21447 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:15:33.078799 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:33.082398 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:33.086567 12567 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:33.086552 21378 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:33.087484 13260 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:33.086982 12565 rpcz_store.cc:276] 0709 12:15:31.889851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:32.041425 (+151574us) service_pool.cc:224] Handling call
0709 12:15:32.054234 (+ 12809us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:32.073815 (+ 19581us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:32.073875 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:32.074696 (+   821us) log.cc:844] Serialized 26 byte log entry
0709 12:15:32.529234 (+454538us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b180045fee0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:32.626433 (+ 97199us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:32.626489 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:32.626583 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:33.056822 (+430239us) raft_consensus.cc:1685] finished
0709 12:15:33.056849 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:33.057060 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18963,"raft.queue_time_us":19137,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":58848,"spinlock_wait_cycles":30407424,"thread_start_us":95820,"threads_started":1,"wal-append.queue_time_us":716984}
I20260709 12:15:33.089749 21378 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:33.090806 21378 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.091313 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:33.087926 12564 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:33.093267 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:33.094622 21411 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:33.095103 21132 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:33.095566 21132 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:33.095901 21132 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:33.097822 21395 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:33.098425 21395 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.100976 21395 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:33.103332 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:33.105392 12565 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:15:33.110561 12564 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:33.045861 21000 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:33.121843 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:33.123485 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.071687 21381 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:33.129730 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44998.4us
I20260709 12:15:33.071373 21362 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:33.005551 21470 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.135968 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:33.045135 20913 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:15:33.141152 21471 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.132583 21475 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:33.119726 21307 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:33.115916 21379 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:33.118309 21447 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:33.091890 21021 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:15:33.149590 12568 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:33.150193 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.193429 12570 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:33.152166 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.134472 21411 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:33.162245 21429 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.169039 21038 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:15:33.171175 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.173712 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.174242 21425 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:33.147861 21370 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:33.146203 21449 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.180281 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:33.183008 21497 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:33.183260 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3831) took 1100 ms. Trace:
I20260709 12:15:33.184007 21454 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:33.209721 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:33.185334 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 194484us
I20260709 12:15:33.186098 21498 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:33.187906 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1182) took 1071 ms. Trace:
I20260709 12:15:33.187987 21507 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.197034 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:33.199910 21379 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.202288 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.172960 21475 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.221643 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:33.177736 21461 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:33.231897 21379 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.233037 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3010) took 1980 ms. Trace:
I20260709 12:15:33.240549 21379 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:33.240741 21470 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:33.195248 21469 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.210508 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:33.188059 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61278us
I20260709 12:15:33.246049 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:33.247825 12493 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:15:33.224543 12572 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:33.211416 12490 rpcz_store.cc:276] 0709 12:15:32.116400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:32.116593 (+   193us) service_pool.cc:224] Handling call
0709 12:15:32.153542 (+ 36949us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:32.228768 (+ 75226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:32.228821 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:32.263204 (+ 34383us) log.cc:844] Serialized 26 byte log entry
0709 12:15:32.905844 (+642640us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:32.905897 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:32.905976 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:33.187571 (+281595us) raft_consensus.cc:1685] finished
0709 12:15:33.187603 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:33.187816 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30406,"raft.queue_time_us":83822,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":23205,"thread_start_us":583106,"threads_started":1,"wal-append.queue_time_us":781557}
I20260709 12:15:33.248889 12646 rpcz_store.cc:276] 0709 12:15:31.252185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.515577 (+263392us) service_pool.cc:224] Handling call
0709 12:15:33.232744 (+1717167us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:33.232947 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2997313792}
I20260709 12:15:33.222867 21507 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:33.217983 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86546.1us
I20260709 12:15:33.250392 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:33.248363 12496 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:15:33.249207 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:33.206270 12491 rpcz_store.cc:276] 0709 12:15:32.082845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:32.127195 (+ 44350us) service_pool.cc:224] Handling call
0709 12:15:32.161619 (+ 34424us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:15:32.631335 (+469716us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:32.631376 (+    41us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:15:32.678510 (+ 47134us) log.cc:844] Serialized 78 byte log entry
0709 12:15:32.916895 (+238385us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:32.917500 (+   605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:33.182602 (+265102us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:33.182677 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:33.182758 (+    81us) raft_consensus.cc:1685] finished
0709 12:15:33.182780 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:33.183161 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":229494,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":14966,"spinlock_wait_cycles":564631936,"thread_start_us":402322,"threads_started":2,"wal-append.queue_time_us":386189}
W20260709 12:15:33.260914 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50474.7us
I20260709 12:15:33.280731 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:33.282963 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:33.284168 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3016) took 1992 ms (client timeout 1952 ms). Trace:
I20260709 12:15:33.297585 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:33.297907 12637 rpcz_store.cc:269] 0709 12:15:31.291737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.423620 (+131883us) service_pool.cc:224] Handling call
0709 12:15:33.283929 (+1860309us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540018c4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:33.284092 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3511356160}
I20260709 12:15:33.215045 21450 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:33.287719 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:33.251943 21498 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.289371 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:33.289148 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:33.305491 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:33.306788 12497 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:33.279235 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:33.307366 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:33.207026 21132 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.310101 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:33.208006 21449 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.316968 13257 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
W20260709 12:15:33.320685 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54167.4us
I20260709 12:15:33.321276 12570 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:33.299219 21464 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:33.297127 21370 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:15:33.326546 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45948.5us
I20260709 12:15:33.329733 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:33.330118 21501 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:15:33.331248 21374 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:33.332188 21430 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.332468 21449 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:33.332875 21430 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.333791 21488 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.074s
W20260709 12:15:33.334254 21503 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:15:33.334707 21379 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:33.320744 21447 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.323266 21475 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:33.327039 20913 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:33.337453 21497 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:33.340404 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:33.309156 21362 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:33.341825 21100 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.348343 21426 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:33.302007 21038 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.289816 21474 raft_consensus.cc:2764] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:15:33.353116 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:33.354737 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.322849 21344 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:15:33.320335 21450 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:33.293991 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
W20260709 12:15:33.421182 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97154.2us
I20260709 12:15:33.425758 13260 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:33.327138 21464 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:33.442826 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89813.1us
I20260709 12:15:33.464831 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:33.470242 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.427022 21475 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:33.530524 21132 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.314601 21498 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.411777 21131 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:33.430694 21168 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.000s
I20260709 12:15:33.771546 21472 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:33.771991 21168 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:15:33.438406 21100 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.773212 12491 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:33.773030 21472 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.776127 21496 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.776671 21472 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:33.289129 21411 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.717255 21429 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:33.779796 21498 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:33.714913 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3522) took 1930 ms (client timeout 1516 ms). Trace:
I20260709 12:15:33.718021 21016 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.724563 21450 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.780354 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:33.717785 21482 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.691s	user 0.015s	sys 0.016s
I20260709 12:15:33.780841 21016 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.379669 21378 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.785950 21378 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.786933 21016 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:33.787206 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3671) took 1347 ms. Trace:
W20260709 12:15:33.765511 21513 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.440s	user 0.010s	sys 0.006s
I20260709 12:15:33.788089 21378 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.788686 21378 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.759397 21475 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:33.734797 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 367429us
I20260709 12:15:33.790546 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:33.791220 21166 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.791798 21166 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:33.730041 21387 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.891s	user 0.013s	sys 0.007s
I20260709 12:15:33.409615 21466 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:33.288525 21463 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:15:33.726549 21510 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:33.730041 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.748225 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:33.753311 12564 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:33.835214 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:33.836130 12564 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:33.836972 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.838200 12565 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:33.838454 21488 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:33.839713 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.841688 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:33.756570 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:33.730265 21497 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.736873 21471 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:33.726891 21233 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:15:33.759234 21447 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.727144 21362 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:33.771639 21131 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.466746 21038 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.776739 21496 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.781322 21482 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:15:33.726125 21508 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.193s	user 0.009s	sys 0.004s
I20260709 12:15:33.749643 21370 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:33.748031 21274 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:33.756004 21464 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:33.781651 12643 rpcz_store.cc:269] 0709 12:15:31.784694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:31.801217 (+ 16523us) service_pool.cc:224] Handling call
0709 12:15:33.535586 (+1734369us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:33.714823 (+179237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2296023936}
I20260709 12:15:33.833256 21168 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:33.783062 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:33.725410 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.784232 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:33.784832 21100 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.788781 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:33.791234 12638 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:15:33.754732 21307 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.007s
I20260709 12:15:33.327778 21021 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.114s
W20260709 12:15:33.754251 20582 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.780s	user 0.010s	sys 0.008s
W20260709 12:15:33.754441 21124 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.007s
I20260709 12:15:33.795341 12491 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:33.787823 12563 rpcz_store.cc:276] 0709 12:15:32.439555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:32.494655 (+ 55100us) service_pool.cc:224] Handling call
0709 12:15:32.516423 (+ 21768us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:32.642071 (+125648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:33.786885 (+1144814us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:33.787116 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183901568}
I20260709 12:15:33.811209 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:33.829140 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:33.834097 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:33.844027 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95929.9us
I20260709 12:15:33.845013 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:33.848867 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92426.2us
I20260709 12:15:33.772070 21393 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.820263 21132 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.761070 21344 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 12:15:33.832828 21387 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:33.826330 21474 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:33.826009 21466 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:33.892692 12643 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:15:33.782292 12637 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:33.896193 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:33.851003 21395 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:33.898656 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:33.789193 21378 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.884302 21461 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:33.949532 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:33.846668 21233 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:33.849402 21497 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.910985 12492 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:33.883921 21514 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:33.914062 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:33.914324 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:33.919354 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:33.914645 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:33.849448 21443 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
W20260709 12:15:33.921957 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110924us
I20260709 12:15:33.842991 21510 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:33.875722 21447 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:33.928349 21000 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.797s	user 0.009s	sys 0.005s
I20260709 12:15:33.935390 21166 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:33.924728 21450 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.912611 20582 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.927325 21344 consensus_queue.cc:1243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:15:33.902853 21168 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.954649 12564 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
W20260709 12:15:34.008694 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94114.8us
I20260709 12:15:34.029027 12564 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:15:34.028707 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:34.030081 12638 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:15:33.947970 21461 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:34.001155 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 218155us
I20260709 12:15:33.975921 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:33.961844 12639 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:15:33.952597 12563 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:15:33.968621 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.044063 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:33.934172 21463 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:15:34.007216 21514 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:34.007539 21482 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.026171 21494 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:33.916579 21124 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.957406 21378 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.062976 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:34.026613 21370 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:34.007929 21000 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:34.040164 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:34.074941 12642 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26. Re-sending same reply.
I20260709 12:15:33.945384 21430 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.046078 12567 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:33.937698 21507 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:33.943998 21474 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:33.898631 21274 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.051843 21487 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:34.048229 12641 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:34.051010 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.051796 12565 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:33.942124 21411 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:34.051096 12572 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
W20260709 12:15:34.050896 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136655us
I20260709 12:15:33.957464 21497 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:34.033802 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114512us
I20260709 12:15:33.933947 21132 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:33.979209 21517 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:15:34.070695 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:34.071952 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:34.083312 12490 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:34.047463 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.077049 21395 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:34.105472 12496 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:34.048938 21443 consensus_queue.cc:1243] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:15:34.085950 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.088768 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:34.077979 21021 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:34.045174 21514 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:34.102383 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:33.973968 21362 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.064105 21212 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:34.091948 21411 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:34.087903 21487 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.112207 12638 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:33.962272 21233 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.026079 21450 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:34.119696 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68776.7us
I20260709 12:15:34.087219 21496 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.050241 21482 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.139669 21021 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:34.112465 21430 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:34.104892 21016 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.133337 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3685) took 1091 ms. Trace:
I20260709 12:15:34.128286 21212 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.138489 21038 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.114459 21395 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.149883 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.149798 12568 rpcz_store.cc:276] 0709 12:15:33.041497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.041679 (+   182us) service_pool.cc:224] Handling call
0709 12:15:33.072008 (+ 30329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:33.192868 (+120860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:34.132861 (+939993us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:34.133224 (+   363us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177579776}
W20260709 12:15:34.164644 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118373us
W20260709 12:15:34.165827 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95278.1us
I20260709 12:15:34.166993 12494 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:15:34.167536 12638 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.12-26.12]
I20260709 12:15:34.167989 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:34.140877 21487 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:34.178411 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:34.176319 21021 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.183333 21021 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.184088 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:34.186662 12639 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:15:34.187278 12639 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
I20260709 12:15:34.187505 21168 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.188773 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:34.191644 12563 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:34.176333 21496 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:34.192544 21496 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:34.193135 21496 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:34.193250 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.194245 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:34.195604 12642 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:15:34.197219 21510 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:34.198390 21527 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:15:34.200290 21516 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:34.200762 21516 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:34.202147 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:34.202862 13260 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:34.187536 12639 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260709 12:15:34.203747 12639 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:34.204259 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94736.5us
I20260709 12:15:34.206425 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.207427 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:34.207849 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:34.208933 12638 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:34.161254 21212 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:34.166085 21395 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:34.214545 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:34.182225 21469 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:34.207015 21524 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.280s	user 0.002s	sys 0.000s
I20260709 12:15:34.222345 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:34.225585 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47275.7us
I20260709 12:15:34.182415 21177 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.162493 21016 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.228267 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:34.218727 21100 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:34.247967 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.218362 21429 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:15:34.229194 21381 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:34.240566 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:34.271136 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:34.274072 13260 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:15:34.232646 21510 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.237609 21168 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:34.070648 21494 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.235910 20582 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.228060 21470 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:15:34.324020 21016 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.285466 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.227485 21412 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:15:34.325263 21525 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:15:34.230923 21471 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:15:34.227010 21472 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:15:34.271927 21381 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:34.245085 21507 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:34.282400 21124 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.282537 21000 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.068s
W20260709 12:15:34.351291 21535 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:15:34.326846 21132 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.073428 21466 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.353082 21466 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:34.327111 21482 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.244876 21100 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.357873 21100 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:34.366950 21507 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.204314 21516 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.328166 21464 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.314896 21168 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:34.330652 21534 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:15:34.330967 21537 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:34.331919 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45277.8us
W20260709 12:15:34.402527 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:34.286537 21430 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:34.315603 21510 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.410487 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:34.346436 21463 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:34.325253 21412 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.339322 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.340238 21212 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:34.342510 21522 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:15:34.325949 21532 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:34.274848 21443 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.371842 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:34.210500 21411 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.376261 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:34.459508 12642 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:34.284317 21363 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.286407 21538 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:34.411670 12638 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:34.335021 21536 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:15:34.418757 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:34.441686 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 219462us
I20260709 12:15:34.371408 21474 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.443531 12489 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:34.444456 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.445982 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:34.409293 21527 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
W20260709 12:15:34.449267 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 201386us
I20260709 12:15:34.490645 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.449427 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:34.451144 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:34.372251 21507 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.401724 21168 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:34.415454 21038 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.414669 21132 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.443233 21412 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:34.491741 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.493454 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:34.511459 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.457841 21445 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.506076 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.516104 12563 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:34.516572 12570 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:34.520669 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.521802 12642 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:34.453706 21472 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:34.523515 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:34.524183 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:34.514825 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 286787us
I20260709 12:15:34.481068 21538 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.529163 21538 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:34.526518 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.530709 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:34.531069 12642 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:34.484618 21536 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.488550 21411 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:34.534375 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.539115 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:34.542492 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:34.540813 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50256.1us
W20260709 12:15:34.545601 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:34.542299 12488 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:34.547880 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:34.548007 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.548664 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:34.549938 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:34.530020 21482 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:34.555852 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.523759 20582 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:34.566893 12568 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:34.484867 21363 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.463798 21443 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.575896 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3699) took 1143 ms. Trace:
I20260709 12:15:34.593539 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:34.510249 21000 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:15:34.415949 21537 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.586748 12571 rpcz_store.cc:276] 0709 12:15:33.432247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.909438 (+477191us) service_pool.cc:224] Handling call
0709 12:15:33.909891 (+   453us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:33.926047 (+ 16156us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:33.926105 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:33.946489 (+ 20384us) log.cc:844] Serialized 52 byte log entry
0709 12:15:34.120008 (+173519us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:34.120059 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:34.120139 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:34.575427 (+455288us) raft_consensus.cc:1685] finished
0709 12:15:34.575473 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:34.575782 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5510,"raft.queue_time_us":5712,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":11183,"thread_start_us":68249,"threads_started":1,"wal-append.queue_time_us":198914}
I20260709 12:15:34.508836 21496 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.523110 21132 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.517970 21038 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.549391 21450 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.607968 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:34.532157 21445 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.570123 21429 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:34.558169 21274 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.562068 21425 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:34.543823 21536 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:34.552297 21541 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.584693 21462 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.608810 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.561169 21374 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:15:34.627056 12638 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:34.628247 12567 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:34.630031 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:34.630226 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:34.523463 21381 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 27.10, Last appended by leader: 10, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:34.652117 21475 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.863s	user 0.011s	sys 0.002s
I20260709 12:15:34.652746 21475 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.565806 21482 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:34.654495 21543 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
W20260709 12:15:34.634008 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86137.5us
I20260709 12:15:34.637107 20582 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:34.638088 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:34.639461 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:34.559600 21472 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.668231 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.643914 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.641407 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1165) took 1381 ms. Trace:
I20260709 12:15:34.646284 13258 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:34.671044 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:15:34.628342 21526 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:15:34.661775 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3847) took 1374 ms. Trace:
I20260709 12:15:34.677665 12491 rpcz_store.cc:276] 0709 12:15:33.287579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.287726 (+   147us) service_pool.cc:224] Handling call
0709 12:15:34.661502 (+1373776us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:34.661693 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1063453056}
I20260709 12:15:34.679589 12497 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:15:34.676748 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:34.671058 12646 rpcz_store.cc:276] 0709 12:15:33.259698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.287815 (+ 28117us) service_pool.cc:224] Handling call
0709 12:15:33.294238 (+  6423us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:33.315058 (+ 20820us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:33.315255 (+   197us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:33.344611 (+ 29356us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:33.357558 (+ 12947us) log.cc:844] Serialized 52 byte log entry
0709 12:15:33.890054 (+532496us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:34.038631 (+148577us) log.cc:844] Serialized 14 byte log entry
0709 12:15:34.070346 (+ 31715us) log.cc:844] Serialized 14 byte log entry
0709 12:15:34.070550 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:34.070635 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:34.641027 (+570392us) raft_consensus.cc:1685] finished
0709 12:15:34.641061 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:34.641319 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39065,"prepare.queue_time_us":84342,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":101,"raft.queue_time_us":40660,"raft.run_cpu_time_us":2132,"raft.run_wall_time_us":39766,"spinlock_wait_cycles":33894784,"thread_start_us":506389,"threads_started":2,"wal-append.queue_time_us":682056}
I20260709 12:15:34.683902 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:34.685417 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.630476 21552 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:34.645454 21464 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.664685 21471 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:34.626360 21395 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.703498 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.704586 12491 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:34.708061 21546 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:34.708599 21546 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.697512 21471 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:34.714130 21471 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:34.714224 21482 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.697678 21537 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.718997 21443 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.720463 21374 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:34.721151 21374 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.721861 21550 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:34.723549 21374 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.724829 21475 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.680294 21344 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:34.727244 21510 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.717599 21212 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.703833 13261 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:34.732728 21547 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:34.708881 21100 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.738948 21550 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:34.739595 21557 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.740154 21557 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.740280 21550 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.742910 21557 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:34.738801 21487 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.003s
I20260709 12:15:34.744449 21487 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.708945 21546 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:34.752108 21472 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.646049 21541 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.753707 12488 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:34.701517 21530 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.379s	user 0.001s	sys 0.001s
W20260709 12:15:34.754671 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78926.2us
I20260709 12:15:34.743201 21100 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:34.736604 21514 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.004s
I20260709 12:15:34.662569 21462 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.723578 21132 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.776309 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:34.776388 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 185254us
I20260709 12:15:34.733762 21494 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.756506 21552 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:34.781750 12536 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
W20260709 12:15:34.786369 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 256006us
I20260709 12:15:34.797816 12536 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:15:34.806717 12488 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:34.774899 21487 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.800658 21429 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:34.822597 21562 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260709 12:15:34.823388 21429 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.9, Last appended by leader: 10, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.787581 20582 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.826455 21542 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.011s
I20260709 12:15:34.829540 12491 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:15:34.832463 21541 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.839033 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:34.839780 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58291.4us
W20260709 12:15:34.812914 21370 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.740s	user 0.011s	sys 0.002s
I20260709 12:15:34.854219 21363 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.856546 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.857040 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:34.857359 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:34.857720 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:34.858327 12572 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:34.858518 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:34.859042 12571 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:34.862145 21496 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.863896 21395 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.779311 21514 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.815191 21381 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 21. Current term is 22.
I20260709 12:15:34.861980 21563 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:34.888545 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46407us
I20260709 12:15:34.892887 12571 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:34.897346 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:34.875181 21370 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:34.808939 21494 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.068s
I20260709 12:15:34.929891 12644 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:34.933909 21539 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
W20260709 12:15:34.930172 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:34.968012 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:34.968294 21430 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.546s	user 0.019s	sys 0.004s
I20260709 12:15:34.968803 21430 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:34.932663 12491 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [23.13-23.13]
W20260709 12:15:34.932021 21469 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.711s	user 0.014s	sys 0.000s
I20260709 12:15:34.931954 21498 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:34.937772 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3531) took 1162 ms. Trace:
I20260709 12:15:34.939945 21471 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.875507 21541 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:15:34.946904 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:34.936563 21552 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.945897 21379 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:15:34.950187 21514 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:34.951828 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 56) took 1700 ms. Trace:
I20260709 12:15:34.988348 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:34.958069 21572 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:15:34.958880 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59752.9us
I20260709 12:15:34.988853 12492 rpcz_store.cc:276] 0709 12:15:33.281968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.293769 (+ 11801us) service_pool.cc:224] Handling call
0709 12:15:34.951630 (+1657861us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:34.951726 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369107328}
I20260709 12:15:34.991847 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:34.992515 12492 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:34.971468 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:34.976013 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6332149769998523s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
W20260709 12:15:34.993362 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4334149769997566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
I20260709 12:15:34.959323 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:34.972800 12640 rpcz_store.cc:276] 0709 12:15:33.775199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.784897 (+  9698us) service_pool.cc:224] Handling call
0709 12:15:33.785232 (+   335us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:33.791826 (+  6594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:33.791861 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:33.792556 (+   695us) log.cc:844] Serialized 52 byte log entry
0709 12:15:34.103239 (+310683us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:34.117403 (+ 14164us) log.cc:844] Serialized 14 byte log entry
0709 12:15:34.123079 (+  5676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:34.123171 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:34.937485 (+814314us) raft_consensus.cc:1685] finished
0709 12:15:34.937509 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:34.937704 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33357,"prepare.queue_time_us":15116,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":50103,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":15678,"thread_start_us":310248,"threads_started":1,"wal-append.queue_time_us":420510}
I20260709 12:15:34.981830 21430 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:34.989355 12491 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:34.982543 21379 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:34.993764 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2615149769999334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1967s for a response
I20260709 12:15:34.929463 21466 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 12:15:35.030405 21471 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.931632 21563 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.034212 21563 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:34.972548 21498 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:34.975706 21546 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.035938 21498 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:34.975621 21541 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.037900 21541 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:35.039141 21546 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.043277 21447 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.998919 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.045265 12640 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:34.975515 20913 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.046145 20913 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.048939 12572 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:35.000636 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.050891 12491 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:15:35.051877 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:35.052479 12491 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:35.052861 12491 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:15:35.002534 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2228149769998709s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
I20260709 12:15:35.055233 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:35.056752 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:35.057988 21447 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:35.058797 21447 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:35.058915 21536 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.062157 21536 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.063817 21429 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.037326 21498 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:35.010870 21545 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.012732 12490 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 19. Current term is 20. Ops: []
I20260709 12:15:35.019476 21379 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:34.974877 21168 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.022367 12637 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:15:35.023445 21462 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.023792 12494 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:15:35.024780 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 1548) took 1239 ms. Trace:
I20260709 12:15:35.064515 21429 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.079520 21570 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:35.080268 21570 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.081184 21168 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.066180 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:34.990024 21497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.034s	sys 0.025s
I20260709 12:15:35.067255 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.098851 12572 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:35.099279 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:35.067934 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.050793 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.102437 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:35.103010 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.104202 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.106196 12494 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:35.115518 12572 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:35.070281 12491 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:15:35.071421 12492 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:35.120924 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.077060 12565 rpcz_store.cc:276] 0709 12:15:33.785565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.839343 (+ 53778us) service_pool.cc:224] Handling call
0709 12:15:35.024559 (+1185216us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:35.024686 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":485856512}
I20260709 12:15:35.026165 21412 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.081987 12493 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:35.082077 21429 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:34.975179 21474 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:15:35.083503 21379 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:15:35.083884 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3545) took 1276 ms. Trace:
I20260709 12:15:35.076615 21545 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:35.074085 21538 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.544s	user 0.012s	sys 0.000s
I20260709 12:15:35.092200 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.118084 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.150790 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.105944 21453 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:35.153071 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:35.154386 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.154763 13258 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:35.069854 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 311951us
I20260709 12:15:35.130357 21168 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:35.122638 12571 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:15:35.163745 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.123440 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.164604 12570 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:35.165114 12564 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:15:35.123483 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.130863 21547 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:35.111915 21497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.034s	sys 0.025s
I20260709 12:15:35.168044 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.169169 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.169872 12564 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:35.164242 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:35.111984 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45924us
I20260709 12:15:35.170675 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.136039 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:35.143368 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.143376 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.143432 12641 rpcz_store.cc:276] 0709 12:15:33.807587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.883625 (+ 76038us) service_pool.cc:224] Handling call
0709 12:15:35.083684 (+1200059us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:35.083821 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241469824}
I20260709 12:15:35.146373 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.147015 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:35.074139 21563 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:15:35.151778 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:35.120062 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:35.152616 13255 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:35.152412 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.154132 21453 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.157282 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:35.173362 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53393.8us
I20260709 12:15:35.116932 21462 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:15:35.137954 21307 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:15:35.158710 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:35.155089 13262 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:35.122028 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.162377 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.003539 21430 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:35.161584 21540 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.808s	user 0.000s	sys 0.002s
I20260709 12:15:35.171306 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.171890 12490 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:15:35.172580 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.152778 21429 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.176302 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.185149 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.185802 12566 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:35.186151 12565 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:15:35.177470 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.182462 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:35.182631 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.183180 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.188338 12567 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:15:35.188148 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:35.183501 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.191449 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.192368 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:35.193540 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:35.188493 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.195814 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:35.196352 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.197340 12495 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:15:35.198805 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.206311 12494 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:35.207331 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.179584 21160 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:35.182938 21471 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.208288 12565 raft_consensus.cc:1217] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Deduplicated request from leader. Original: 17.11->[18.12-21.14]   Dedup: 18.12->[19.13-21.14]
I20260709 12:15:35.154417 21538 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.210083 21567 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.210266 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:35.210956 21583 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:15:35.211503 21570 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.211913 21166 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:35.212406 21514 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:35.212947 21544 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.882s	user 0.002s	sys 0.002s
I20260709 12:15:35.228796 21160 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.179049 21462 consensus_queue.cc:1243] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 23.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:15:35.217216 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:35.209136 12490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:35.269142 12490 raft_consensus.cc:606] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:35.269501 12490 raft_consensus.cc:614] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:35.226971 12565 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 12
I20260709 12:15:35.270254 12565 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:15:35.270547 12565 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:35.271462 12564 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:35.272039 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3852) took 1067 ms. Trace:
I20260709 12:15:35.274309 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:35.235152 12570 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:15:35.190491 21547 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:15:35.186981 20913 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.195914 21429 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.200878 21099 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:35.286552 21099 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.287019 21099 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:35.288467 21429 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.289341 12567 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:15:35.291440 21494 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.272433 12496 rpcz_store.cc:276] 0709 12:15:34.204688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.204861 (+   173us) service_pool.cc:224] Handling call
0709 12:15:34.205427 (+   566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:34.205861 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:34.205891 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:34.206527 (+   636us) log.cc:844] Serialized 26 byte log entry
0709 12:15:34.292846 (+ 86319us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:34.292908 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:34.293009 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:35.212287 (+919278us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:35.261452 (+ 49165us) raft_consensus.cc:1685] finished
0709 12:15:35.261483 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:35.271976 (+ 10493us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7104,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":6803,"spinlock_wait_cycles":33474816,"thread_start_us":58673,"threads_started":1,"wal-append.queue_time_us":113377}
I20260709 12:15:35.299914 21545 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.247237 21475 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:35.301011 21475 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:35.301450 21475 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:35.306808 21412 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.243220 21166 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.311292 21166 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:35.315213 21538 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.318933 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.319823 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:35.324429 21541 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:35.241485 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59112.1us
I20260709 12:15:35.244488 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.332546 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:35.220367 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:35.256085 12566 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:35.241339 21567 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.321348 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:35.338675 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 118407us
I20260709 12:15:35.340456 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:35.257992 21453 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.260006 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.263505 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3857) took 1030 ms. Trace:
I20260709 12:15:35.345093 12489 rpcz_store.cc:276] 0709 12:15:34.233487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.285154 (+ 51667us) service_pool.cc:224] Handling call
0709 12:15:35.263278 (+978124us) spinlock_profiling.cc:243] Waited 95.7 ms on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:35.263433 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220053248}
I20260709 12:15:35.228632 21411 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:35.227942 21426 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
W20260709 12:15:35.268640 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.233549 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:35.278198 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.281246 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.195189 21381 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:35.268482 21160 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:35.291342 12569 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:35.321002 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.376919 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.377678 12489 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:35.237326 21471 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.332674 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:35.320333 21099 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.347863 13258 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
W20260709 12:15:35.380024 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47446.2us
I20260709 12:15:35.380128 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.381386 12496 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:15:35.351974 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.381578 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.382616 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:35.355034 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.371385 12572 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:35.347846 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:35.383846 12564 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:15:35.384021 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:35.373041 13261 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:35.290310 21430 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.382983 12494 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:15:35.385190 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.386579 12568 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:35.293108 21494 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.387768 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:35.399180 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.395108 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.395274 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.400897 13259 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:35.395860 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:35.350775 21516 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:35.403172 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.403208 12637 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 12:15:35.409003 12637 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
I20260709 12:15:35.409292 12637 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:15:35.409528 12637 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:35.406106 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58346.9us
I20260709 12:15:35.402114 12491 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:35.410712 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:35.404470 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.412617 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.368072 21514 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:35.348431 21469 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:35.367758 21538 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:35.419090 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.419320 12570 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:35.367589 21547 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:35.429981 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.421299 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:35.426049 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.432928 13261 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:15:35.433393 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.434083 12637 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:35.434340 21463 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:35.435098 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.436326 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.436884 12637 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:35.388816 21545 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:35.439725 12638 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:35.348608 21412 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:15:35.401696 21307 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:35.450469 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.406620 21411 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:35.407655 21370 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:35.453853 13253 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:35.408150 21426 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.426720 21430 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:35.456054 12646 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26.
I20260709 12:15:35.420468 21363 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:35.420222 21476 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:35.421432 21469 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.420758 21514 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.438624 21395 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:35.413404 21572 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:15:35.455415 21463 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.456034 21411 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:35.461555 21545 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:35.504665 12638 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:35.476096 12637 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:35.506531 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.507966 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.465157 21307 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:35.510975 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.471937 21363 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.511090 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.473223 21395 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.543526 12641 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24.
I20260709 12:15:35.470827 21099 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.545063 12638 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:35.473735 21469 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:35.473369 21514 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:35.508173 21539 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:35.466779 21538 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:15:35.474680 21476 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:35.546492 21363 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:35.578333 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.590s (SENT) [suppressed 26 similar messages]
I20260709 12:15:35.509336 21567 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.580781 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.586361 12641 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:35.556731 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.581395 21546 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:35.475054 21426 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.548882 21395 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:35.577785 21539 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:35.509948 21463 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.479038 21461 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:35.619338 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 207685us
I20260709 12:15:35.596550 21546 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.601120 21494 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.603159 21307 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.663458 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3849) took 1494 ms. Trace:
I20260709 12:15:35.664053 12497 rpcz_store.cc:276] 0709 12:15:34.168738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.185692 (+ 16954us) service_pool.cc:224] Handling call
0709 12:15:34.186146 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:34.683873 (+497727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:35.663033 (+979160us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:35.663361 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134121344}
W20260709 12:15:35.604709 21584 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:15:35.666551 21471 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.667613 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.668540 12643 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:35.611647 12639 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:15:35.671043 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1176) took 1266 ms. Trace:
I20260709 12:15:35.671595 12639 rpcz_store.cc:276] 0709 12:15:34.404627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.413490 (+  8863us) service_pool.cc:224] Handling call
0709 12:15:35.670752 (+1257262us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:35.670959 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2574194432}
I20260709 12:15:35.612486 12641 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:35.613169 21016 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:35.616530 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:35.599099 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.583559 21476 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:35.625015 12640 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:35.639578 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:35.631105 21021 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.640530 21450 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:35.615854 21475 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.597182 21099 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:35.658533 21570 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:35.709753 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.659188 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1063) took 1430 ms. Trace:
I20260709 12:15:35.660310 21578 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:35.660802 21370 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:35.676267 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.671687 21546 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:35.684396 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 271044us
I20260709 12:15:35.687686 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.551926 21547 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.589468 21411 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.691471 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3693) took 2602 ms. Trace:
W20260709 12:15:35.697563 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 293136us
I20260709 12:15:35.690589 21494 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:35.707772 12536 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:35.738567 12536 leader_election.cc:403] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:15:35.715803 21516 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:35.660389 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:35.636751 21541 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.750068 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.725654 12572 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 30 because replica has last-logged OpId of term: 27 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 12.
I20260709 12:15:35.685788 21461 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:35.680136 21016 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:35.728127 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:35.685995 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.730370 12491 raft_consensus.cc:606] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:35.727129 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.732509 12569 rpcz_store.cc:276] 0709 12:15:33.077602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.078012 (+   410us) service_pool.cc:224] Handling call
0709 12:15:33.078462 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:35.376831 (+2298369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:35.679439 (+302608us) spinlock_profiling.cc:243] Waited 2.21 s on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:35.679620 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5088541056}
I20260709 12:15:35.734551 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:35.702368 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:35.735639 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.709887 21099 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:35.684484 21572 consensus_queue.cc:1175] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 27 status { last_received { term: 26 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:15:35.620301 21545 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:35.746532 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86264.9us
W20260709 12:15:35.769280 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.726037 21370 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.747255 21578 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:35.713290 13255 rpcz_store.cc:276] 0709 12:15:34.228525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.228687 (+   162us) service_pool.cc:224] Handling call
0709 12:15:35.658926 (+1430239us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:35.659110 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069167872}
W20260709 12:15:35.753686 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46137.5us
I20260709 12:15:35.783279 12536 leader_election.cc:400] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24.
I20260709 12:15:35.783838 12536 leader_election.cc:403] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:15:35.729032 21450 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:35.786301 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:35.770107 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.773599 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.725525 21497 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
W20260709 12:15:35.774458 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.768417 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.777781 12491 raft_consensus.cc:614] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:35.777355 13262 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:35.752729 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:35.793022 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.793367 12572 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:35.788933 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.794907 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.795738 12572 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:35.786206 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59189.6us
I20260709 12:15:35.745348 21430 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:35.799909 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:35.770676 21411 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:35.804538 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:35.754494 21494 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:35.815994 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:35.811753 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.699612 21539 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.12, Last appended by leader: 12, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.734812 21570 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.831980 21430 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:35.832110 21494 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.834419 12568 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:15:35.834834 12568 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 23.12
I20260709 12:15:35.835109 12568 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:35.837339 21016 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.838212 21426 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.838896 21016 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:35.817860 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.791455 21541 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:35.840553 21541 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:35.841003 21541 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:35.796011 21589 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:35.846292 21587 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 12:15:35.847862 21475 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.850342 21463 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:35.851089 21463 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:35.809558 21021 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:15:35.809854 21546 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:35.798983 21429 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:35.828727 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:35.856583 21546 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.856995 13258 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:15:35.797078 21450 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.829610 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:35.859483 12638 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:35.810691 21469 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.799245 21578 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:35.862494 21469 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.759239 21516 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.862720 21578 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.808023 21563 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24.
I20260709 12:15:35.867367 13255 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:15:35.874105 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1172) took 1497 ms. Trace:
I20260709 12:15:35.874696 12642 rpcz_store.cc:276] 0709 12:15:34.377121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.531877 (+154756us) service_pool.cc:224] Handling call
0709 12:15:34.532227 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:34.643985 (+111758us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:34.644043 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:34.670489 (+ 26446us) log.cc:844] Serialized 26 byte log entry
0709 12:15:34.768656 (+ 98167us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:34.891645 (+122989us) log.cc:844] Serialized 14 byte log entry
0709 12:15:34.893234 (+  1589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:35.873528 (+980294us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:35.873608 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:35.873704 (+    96us) raft_consensus.cc:1685] finished
0709 12:15:35.873828 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:35.874023 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1574,"prepare.queue_time_us":225543,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":3357,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":57944,"spinlock_wait_cycles":231220096,"thread_start_us":148789,"threads_started":2,"wal-append.queue_time_us":152177}
I20260709 12:15:35.884873 21516 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.887877 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:35.810413 12572 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:35.899713 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.900741 12572 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:35.906392 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3854) took 1689 ms. Trace:
I20260709 12:15:35.906975 12488 rpcz_store.cc:276] 0709 12:15:34.216980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.549311 (+332331us) service_pool.cc:224] Handling call
0709 12:15:35.906167 (+1356856us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:35.906305 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":467097472}
W20260709 12:15:35.916643 21447 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.857s	user 0.011s	sys 0.004s
I20260709 12:15:35.917303 21447 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:35.921983 12564 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.031s	sys 0.046s
W20260709 12:15:35.922370 12564 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.031s	sys 0.046s
I20260709 12:15:35.860566 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.926501 12569 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:35.928340 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.929466 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:35.923051 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:35.944928 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:35.931406 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:35.946779 21370 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:35.947247 21168 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.778s	user 0.012s	sys 0.000s
I20260709 12:15:35.856860 21429 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:35.978997 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:35.980746 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:35.959184 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.982921 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:35.985087 21166 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.008s
I20260709 12:15:35.985234 21450 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.985584 21166 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:35.986703 21160 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.585s	user 0.007s	sys 0.007s
I20260709 12:15:35.987757 21166 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:35.987403 21160 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:35.946856 21516 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:35.948215 21563 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:15:35.992601 12572 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:35.986891 12494 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:35.950960 21539 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:35.960054 21038 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:36.003221 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:35.952423 21447 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:35.957116 21233 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:35.960266 21177 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:35.995955 21533 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:15:35.950608 21589 consensus_queue.cc:1175] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 26 status { last_received { term: 26 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 26 index: 12 } }
W20260709 12:15:35.991988 21498 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.918s	user 0.005s	sys 0.015s
W20260709 12:15:36.026391 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.023s	sys 0.026s
W20260709 12:15:36.030340 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 452035us
I20260709 12:15:36.012140 12563 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:35.977468 21445 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:35.991819 21168 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:36.071975 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:35.996521 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:36.073199 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:36.074281 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:36.075141 12563 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:36.075981 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:36.076715 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:36.022521 21380 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:36.079144 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:36.095120 12572 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:36.023097 21533 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:36.106582 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:36.111457 21538 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:36.112819 21429 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:36.119098 21380 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:36.069103 21177 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:36.123378 21177 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.110074 21425 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:15:36.132716 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:36.135653 21588 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:15:36.120774 21380 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.139765 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:36.143743 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:15:36.157167 21572 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:36.158535 21160 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:36.183306 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:36.167479 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64127.6us
I20260709 12:15:36.168632 21379 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:36.079540 21445 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:36.105696 21450 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:15:36.203351 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:36.197542 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:36.203779 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3540) took 2407 ms. Trace:
I20260709 12:15:36.204278 12644 rpcz_store.cc:276] 0709 12:15:33.796488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:33.913890 (+117402us) service_pool.cc:224] Handling call
0709 12:15:33.914214 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:34.962923 (+1048709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:36.202686 (+1239763us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:36.203711 (+  1025us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2334100736}
I20260709 12:15:36.196413 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:36.206451 12644 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.12-26.12]
I20260709 12:15:36.207974 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:36.219796 12488 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:36.220784 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:36.218693 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:36.234166 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:36.242182 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:36.254406 12488 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:36.203158 21546 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:36.208771 21450 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:36.190618 21538 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:15:36.270229 12497 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 10. Preceding OpId from leader: term: 27 index: 10. (term mismatch)
I20260709 12:15:36.270881 12497 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 9
I20260709 12:15:36.271231 12497 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 26.10
I20260709 12:15:36.271014 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:36.271688 12497 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:36.217234 21160 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:36.260686 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:36.209779 21592 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:36.217397 21233 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:36.217123 21593 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:36.315353 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:36.260500 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.288798 21038 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:15:36.289306 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:36.202788 21379 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.289992 21168 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:15:36.291174 21594 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:15:36.294438 21589 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:15:36.298840 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 128) took 43382 ms (client timeout 15000 ms). Trace:
W20260709 12:15:36.329504 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.013s	sys 0.020s
I20260709 12:15:36.308162 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:36.310153 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:36.326753 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.013s	sys 0.025s
I20260709 12:15:36.307297 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:36.282006 21450 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:36.343304 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:36.334615 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.013s	sys 0.025s
I20260709 12:15:36.339071 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:36.345461 12488 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:36.231043 21528 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.083s
I20260709 12:15:36.347980 12569 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:15:36.246667 21021 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
W20260709 12:15:36.335232 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.014s	sys 0.020s
I20260709 12:15:36.209153 21445 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.283715 21570 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:36.358685 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.354580 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:36.362879 12645 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
W20260709 12:15:36.351917 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.013s	sys 0.020s
W20260709 12:15:36.384629 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.013s	sys 0.020s
W20260709 12:15:36.385641 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 585802us
I20260709 12:15:36.387809 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.389807 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.390491 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:36.391398 12490 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:36.319518 21593 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:36.403195 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:36.329967 12391 rpcz_store.cc:269] 0709 12:14:52.916670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:52.916876 (+   206us) service_pool.cc:224] Handling call
0709 12:15:36.298726 (+43381850us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:15:30.275909 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:15:30.276682 (+   773us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:15:30.276740 (+    58us) write_op.cc:435] Acquired schema lock
0709 12:15:30.276759 (+    19us) tablet.cc:662] Decoding operations
0709 12:15:30.336527 (+ 59768us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:15:30.336652 (+   125us) write_op.cc:641] Partition lock acquired for write op
0709 12:15:30.336685 (+    33us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:15:30.337603 (+   918us) tablet.cc:701] Row locks acquired
0709 12:15:30.337690 (+    87us) write_op.cc:260] PREPARE: finished
0709 12:15:30.337884 (+   194us) write_op.cc:270] Start()
0709 12:15:30.338085 (+   201us) write_op.cc:276] Timestamp: P: 1783599330337801 usec, L: 0
0709 12:15:30.338137 (+    52us) op_driver.cc:348] REPLICATION: starting
0709 12:15:30.341222 (+  3085us) log.cc:844] Serialized 3218 byte log entry
0709 12:15:31.406281 (+1065059us) op_driver.cc:464] REPLICATION: finished
0709 12:15:31.681824 (+275543us) write_op.cc:301] APPLY: starting
0709 12:15:31.681931 (+   107us) tablet.cc:1367] starting BulkCheckPresence
0709 12:15:32.298093 (+616162us) tablet.cc:1370] finished BulkCheckPresence
0709 12:15:32.341592 (+ 43499us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:15:33.008238 (+666646us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:15:33.008577 (+   339us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 12:15:33.008639 (+    62us) write_op.cc:312] APPLY: finished
0709 12:15:33.009026 (+   387us) log.cc:844] Serialized 81 byte log entry
0709 12:15:33.127804 (+118778us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:33.136510 (+  8706us) write_op.cc:454] Released schema lock
0709 12:15:33.150634 (+ 14124us) write_op.cc:341] FINISH: Updating metrics
0709 12:15:33.250763 (+100129us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:33.260289 (+  9526us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":174245,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":252281,"apply.run_cpu_time_us":52965,"apply.run_wall_time_us":1523046,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37535,"num_ops":8,"prepare.queue_time_us":229192,"prepare.run_cpu_time_us":38485,"prepare.run_wall_time_us":280352,"raft.queue_time_us":47698,"raft.run_cpu_time_us":16559,"raft.run_wall_time_us":94492,"replication_time_us":1067902,"spinlock_wait_cycles":36608,"thread_start_us":392148,"threads_started":5,"wal-append.queue_time_us":767680,"wal-append.run_cpu_time_us":28588,"wal-append.run_wall_time_us":2620912}]]}
W20260709 12:15:36.411103 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 595148us
I20260709 12:15:36.403409 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.326243 21168 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:36.412384 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:36.414559 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:36.349807 21363 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.759s	user 0.016s	sys 0.000s
I20260709 12:15:36.343271 21474 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:36.281025 21541 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.269526 21494 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.100s
I20260709 12:15:36.316437 21160 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:36.348977 21038 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:36.415216 21588 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:15:36.431360 12640 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:36.359766 21233 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:36.432780 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.005s	sys 0.037s
I20260709 12:15:36.319303 21592 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.420230 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:36.431203 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44676.9us
I20260709 12:15:36.410966 21593 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:36.440754 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3563) took 1035 ms. Trace:
I20260709 12:15:36.441295 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:36.418485 21168 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:15:36.445555 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.005s	sys 0.038s
I20260709 12:15:36.442988 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:36.488610 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.028s	sys 0.059s
I20260709 12:15:36.489916 12491 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
W20260709 12:15:36.492422 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.028s	sys 0.060s
I20260709 12:15:36.470825 12646 rpcz_store.cc:276] 0709 12:15:35.404859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.474378 (+ 69519us) service_pool.cc:224] Handling call
0709 12:15:35.505748 (+ 31370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:35.506127 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:35.506151 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:35.513055 (+  6904us) log.cc:844] Serialized 26 byte log entry
0709 12:15:36.048370 (+535315us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:36.048412 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:36.048487 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:36.440491 (+392004us) raft_consensus.cc:1685] finished
0709 12:15:36.440514 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:36.440670 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115853,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":14524,"thread_start_us":534949,"threads_started":1,"wal-append.queue_time_us":667766}
W20260709 12:15:36.449221 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 394000us
W20260709 12:15:36.478621 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58514us
W20260709 12:15:36.527065 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:36.527302 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:36.529165 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:36.494647 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 420346us
I20260709 12:15:36.530426 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:36.530134 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:36.533077 12461 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26.
I20260709 12:15:36.534305 12461 leader_election.cc:403] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:15:36.438915 12488 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:15:36.553587 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66403.6us
I20260709 12:15:36.556010 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:36.555872 21470 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:36.556604 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:36.557891 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:36.560114 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:36.562867 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:36.557926 21561 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:36.425486 21363 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:36.484202 21381 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:36.584561 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52507.1us
I20260709 12:15:36.596029 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.433357 21474 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:36.604676 12490 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:36.549309 21570 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:15:36.563166 21395 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.945s	user 0.014s	sys 0.000s
I20260709 12:15:36.554087 21542 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26.
I20260709 12:15:36.590154 21498 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:36.599442 21021 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:36.620755 21498 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:36.624579 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66865.7us
W20260709 12:15:36.578861 21588 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:15:36.621436 21021 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:36.549536 21599 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:36.632632 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:36.607095 21552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.005s	sys 0.050s
W20260709 12:15:36.623291 21490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.005s	sys 0.014s
W20260709 12:15:36.618691 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63867.9us
I20260709 12:15:36.763269 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:36.765040 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:36.690658 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:36.715736 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 430353us
I20260709 12:15:36.766374 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:36.768265 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:36.719704 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:36.594046 21429 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:36.757162 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3693) took 1635 ms. Trace:
W20260709 12:15:36.772521 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.030s	sys 0.021s
I20260709 12:15:36.712809 21021 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:36.773056 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82574.4us
W20260709 12:15:36.774756 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.030s	sys 0.021s
I20260709 12:15:36.600005 21233 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.154s
I20260709 12:15:36.636292 21387 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:36.777225 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:36.819130 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:36.781996 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:36.770093 12565 rpcz_store.cc:276] 0709 12:15:35.121511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.196923 (+ 75412us) service_pool.cc:224] Handling call
0709 12:15:35.197313 (+   390us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:15:35.343880 (+146567us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:35.381948 (+ 38068us) log.cc:844] Serialized 14 byte log entry
0709 12:15:35.678545 (+296597us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:35.679160 (+   615us) log.cc:844] Serialized 52 byte log entry
0709 12:15:35.679301 (+   141us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:35.679577 (+   276us) log.cc:844] Serialized 14 byte log entry
0709 12:15:35.687207 (+  7630us) log.cc:844] Serialized 14 byte log entry
0709 12:15:35.689197 (+  1990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:35.915891 (+226694us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:35.916105 (+   214us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:36.294678 (+378573us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x7b180038d1c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:36.756880 (+462202us) raft_consensus.cc:1685] finished
0709 12:15:36.756907 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:36.757088 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37710,"prepare.queue_time_us":193198,"prepare.run_cpu_time_us":160,"prepare.run_wall_time_us":11418,"raft.queue_time_us":78258,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":19407,"spinlock_wait_cycles":60798464,"thread_start_us":296303,"threads_started":1,"wal-append.queue_time_us":594240}
I20260709 12:15:36.788374 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:36.746627 21547 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:15:36.836926 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48703.7us
I20260709 12:15:36.843467 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:36.854255 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:36.762702 21490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.005s	sys 0.015s
W20260709 12:15:36.814803 21609 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:15:36.858438 21494 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:36.757249 21599 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:36.875180 21538 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:36.875831 21606 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:36.815280 21601 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:36.825721 21598 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:36.637413 21395 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:36.964979 12641 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.011s	sys 0.072s
I20260709 12:15:36.818076 21381 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:36.970283 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.016s	sys 0.038s
I20260709 12:15:36.780618 21572 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:36.621807 21099 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:36.981567 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:36.818392 21477 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.153s
W20260709 12:15:36.990252 12641 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.012s	sys 0.072s
I20260709 12:15:36.713109 21038 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:36.990259 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.017s	sys 0.039s
W20260709 12:15:36.982551 21380 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.019s	sys 0.040s
W20260709 12:15:36.998039 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:37.011449 12644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.026s	sys 0.069s
W20260709 12:15:37.011950 12644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.026s	sys 0.069s
W20260709 12:15:36.992329 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.020s	sys 0.035s
I20260709 12:15:36.979154 21606 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:37.001649 21599 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:37.012656 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.020s	sys 0.035s
W20260709 12:15:36.993043 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211138us
I20260709 12:15:36.982997 21494 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.024240 12637 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:37.024346 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.030s	sys 0.064s
W20260709 12:15:37.029774 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.031s	sys 0.064s
W20260709 12:15:37.007046 12638 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.016s	sys 0.042s
I20260709 12:15:37.025091 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1209) took 1247 ms. Trace:
I20260709 12:15:37.030956 12492 rpcz_store.cc:276] 0709 12:15:35.777950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.778249 (+   299us) service_pool.cc:224] Handling call
0709 12:15:37.024802 (+1246553us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:37.025012 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1068396800}
W20260709 12:15:37.032269 12638 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.016s	sys 0.042s
W20260709 12:15:37.025229 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.012s	sys 0.048s
W20260709 12:15:37.033166 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.012s	sys 0.048s
I20260709 12:15:36.996644 21603 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:15:37.099467 12462 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.008s	sys 0.022s
W20260709 12:15:37.135492 12462 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.008s	sys 0.022s
W20260709 12:15:37.136085 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 629746us
W20260709 12:15:37.028991 21604 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.268s	user 0.001s	sys 0.001s
I20260709 12:15:37.016691 21099 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:37.011730 21380 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.019s	sys 0.040s
I20260709 12:15:37.022939 21606 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:37.032666 21598 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:37.019376 21445 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.211905 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:37.253305 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:37.127874 21612 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.264918 21363 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:37.262104 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.009s	sys 0.021s
W20260709 12:15:37.270030 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.009s	sys 0.021s
W20260709 12:15:37.270507 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 684677us
W20260709 12:15:37.260442 21466 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.016s	sys 0.028s
W20260709 12:15:37.271606 21466 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.017s	sys 0.029s
I20260709 12:15:37.078662 21494 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.261864 21603 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:37.276337 21494 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:37.277009 21411 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.457s	user 0.010s	sys 0.001s
I20260709 12:15:37.276588 21603 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.278179 12643 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
W20260709 12:15:37.279632 21610 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.204s	user 0.000s	sys 0.000s
I20260709 12:15:37.280149 21603 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.231925 21160 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.249964 21618 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.323662 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:37.342729 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:37.283473 21445 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:37.345870 21559 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:15:37.346753 21600 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.363s	user 0.000s	sys 0.001s
I20260709 12:15:37.348675 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3733) took 1664 ms. Trace:
I20260709 12:15:37.282153 21598 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.349128 12571 rpcz_store.cc:276] 0709 12:15:35.683958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.685982 (+  2024us) service_pool.cc:224] Handling call
0709 12:15:35.686547 (+   565us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:35.687049 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:35.724185 (+ 37136us) log.cc:844] Serialized 14 byte log entry
0709 12:15:36.495082 (+770897us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:36.549985 (+ 54903us) log.cc:844] Serialized 26 byte log entry
0709 12:15:36.550160 (+   175us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:36.550216 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:36.550308 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:37.348417 (+798109us) raft_consensus.cc:1685] finished
0709 12:15:37.348439 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.348607 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":47483,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":496,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":41695,"thread_start_us":703758,"threads_started":1,"wal-append.queue_time_us":1090503}
I20260709 12:15:37.351907 21598 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.291666 21381 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:37.358949 21381 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.364894 21496 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.365689 21496 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.368409 21496 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.369508 21381 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.370200 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1564) took 1596 ms. Trace:
I20260709 12:15:37.370787 12567 rpcz_store.cc:276] 0709 12:15:35.774087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.914418 (+140331us) service_pool.cc:224] Handling call
0709 12:15:35.914756 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:35.964190 (+ 49434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:35.964238 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:35.964302 (+    64us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:35.964350 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.369855 (+1405505us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b54002477d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:37.369929 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.370113 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113065216}
W20260709 12:15:37.373656 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1065) took 2755 ms (client timeout 2048 ms). Trace:
W20260709 12:15:37.374333 13256 rpcz_store.cc:269] 0709 12:15:34.617896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.618077 (+   181us) service_pool.cc:224] Handling call
0709 12:15:37.373245 (+2755168us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:37.373499 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4908056960}
W20260709 12:15:37.373876 21593 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.815s	user 0.010s	sys 0.002s
I20260709 12:15:37.377404 21593 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:37.378535 21614 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:15:37.378877 12646 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: []
I20260709 12:15:37.326532 21411 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.324626 21494 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:37.342366 21488 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:15:37.428609 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.055s	user 0.047s	sys 0.071s
I20260709 12:15:37.392724 21498 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.441558 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3569) took 1953 ms. Trace:
I20260709 12:15:37.340320 21618 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:37.450667 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.077s	user 0.047s	sys 0.071s
W20260709 12:15:37.440399 21617 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:15:37.447229 12642 rpcz_store.cc:276] 0709 12:15:35.488034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.205969 (+717935us) service_pool.cc:224] Handling call
0709 12:15:36.226522 (+ 20553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:36.249021 (+ 22499us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:36.249058 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:36.249824 (+   766us) log.cc:844] Serialized 26 byte log entry
0709 12:15:36.753936 (+504112us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:36.754000 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:36.754100 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:37.441302 (+687202us) raft_consensus.cc:1685] finished
0709 12:15:37.441326 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.441484 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21820,"raft.queue_time_us":54809,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":2664,"thread_start_us":464402,"threads_started":1,"wal-append.queue_time_us":1021170}
I20260709 12:15:37.398502 21411 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:37.451181 12610 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.008s	sys 0.007s
I20260709 12:15:37.451848 21496 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.306214 21612 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.455036 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:37.458225 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.08513e+06us
I20260709 12:15:37.390364 21445 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:37.457867 12610 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.008s	sys 0.007s
W20260709 12:15:37.428267 21536 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.003s	sys 0.011s
I20260709 12:15:37.403961 21601 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:37.452852 21514 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.897s	user 0.017s	sys 0.000s
I20260709 12:15:37.426561 21541 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.401054 21494 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.449705 21381 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:37.473141 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 533645us
I20260709 12:15:37.475226 12642 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:15:37.464491 21476 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.753s	user 0.015s	sys 0.001s
I20260709 12:15:37.492033 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:37.491902 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:37.477588 21538 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:37.463824 21477 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 29.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:15:37.498389 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3565) took 2055 ms. Trace:
I20260709 12:15:37.503578 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3041) took 1810 ms. Trace:
I20260709 12:15:37.504084 12642 rpcz_store.cc:276] 0709 12:15:35.693050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.458047 (+1764997us) service_pool.cc:224] Handling call
0709 12:15:37.474939 (+ 16892us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:37.503265 (+ 28326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.503510 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.506997 12491 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:37.507746 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:37.508765 12639 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 20.
W20260709 12:15:37.484606 21536 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.004s	sys 0.011s
W20260709 12:15:37.509462 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3050) took 1813 ms (client timeout 1862 ms). Trace:
W20260709 12:15:37.513563 12639 rpcz_store.cc:269] 0709 12:15:35.696362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.918584 (+222222us) service_pool.cc:224] Handling call
0709 12:15:37.507526 (+1588942us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:15:37.509369 (+  1843us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94283648}
I20260709 12:15:37.515144 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3574) took 2025 ms. Trace:
I20260709 12:15:37.465556 21496 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.515918 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:37.511660 21381 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.499240 21514 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.495291 21640 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.522578 12642 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:37.519506 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:37.531672 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3713) took 1305 ms. Trace:
I20260709 12:15:37.536276 12564 rpcz_store.cc:276] 0709 12:15:36.226042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.227963 (+  1921us) service_pool.cc:224] Handling call
0709 12:15:36.228460 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:36.249129 (+ 20669us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:36.249159 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:36.260296 (+ 11137us) log.cc:844] Serialized 26 byte log entry
0709 12:15:36.715371 (+455075us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:36.715435 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:36.715539 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:37.531250 (+815711us) raft_consensus.cc:1685] finished
0709 12:15:37.531297 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.531576 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20918,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":13339,"thread_start_us":445977,"threads_started":1,"wal-append.queue_time_us":983574}
I20260709 12:15:37.533211 12639 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:37.550848 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3051) took 1854 ms. Trace:
I20260709 12:15:37.551329 12639 rpcz_store.cc:276] 0709 12:15:35.696642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.514927 (+1818285us) service_pool.cc:224] Handling call
0709 12:15:37.532962 (+ 18035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.550517 (+ 17555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.550778 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.553684 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3049) took 1857 ms. Trace:
I20260709 12:15:37.554381 12642 rpcz_store.cc:276] 0709 12:15:35.696099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.505039 (+1808940us) service_pool.cc:224] Handling call
0709 12:15:37.522366 (+ 17327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.553232 (+ 30866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.553547 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:37.559846 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 479740us
I20260709 12:15:37.565847 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:37.563639 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44228.3us
I20260709 12:15:37.594048 21612 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.594405 21514 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.594831 21641 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.595417 21641 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.600452 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1184) took 1525 ms. Trace:
I20260709 12:15:37.598969 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:37.599481 12639 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 9. Preceding OpId from leader: term: 27 index: 10. (index mismatch)
I20260709 12:15:37.602953 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3052) took 1906 ms. Trace:
I20260709 12:15:37.603384 12639 rpcz_store.cc:276] 0709 12:15:35.696895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.562745 (+1865850us) service_pool.cc:224] Handling call
0709 12:15:37.599095 (+ 36350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.602624 (+  3529us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.602878 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.601012 12642 rpcz_store.cc:276] 0709 12:15:36.075175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.562886 (+1487711us) service_pool.cc:224] Handling call
0709 12:15:37.599829 (+ 36943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.600027 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.600068 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.600171 (+   103us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:37.600212 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.600268 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.600389 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:37.586752 21021 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.773s	user 0.009s	sys 0.004s
I20260709 12:15:37.513597 21476 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:37.596869 21461 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.802s	user 0.016s	sys 0.000s
I20260709 12:15:37.613889 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1187) took 1537 ms. Trace:
I20260709 12:15:37.607985 21466 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:37.596704 21488 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.610756 21514 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 12:15:37.599424 21601 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.621870 21395 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.623867 21641 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.634186 21541 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:37.635123 21411 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:37.638154 21381 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.640239 21496 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:37.643000 21539 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.637s	user 0.008s	sys 0.007s
I20260709 12:15:37.643307 21021 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.644503 21645 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.644218 21538 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:37.710062 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 236995us
I20260709 12:15:37.559707 21640 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.646721 21618 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:37.671087 21572 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.014s	sys 0.017s
I20260709 12:15:37.648664 21476 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.615178 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1185) took 1539 ms. Trace:
I20260709 12:15:37.649369 12642 rpcz_store.cc:276] 0709 12:15:36.076018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.605479 (+1529461us) service_pool.cc:224] Handling call
0709 12:15:37.613296 (+  7817us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.613500 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.613527 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.613576 (+    49us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:37.613653 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.613711 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.613828 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.599048 21616 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:37.596197 21494 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:37.599354 21593 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:37.609915 21567 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:37.656929 21168 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.105s	user 0.014s	sys 0.000s
I20260709 12:15:37.651503 21589 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:37.668556 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 155317us
I20260709 12:15:37.645227 21639 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:37.714361 12494 raft_consensus.cc:606] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:37.747387 12494 raft_consensus.cc:614] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:37.715690 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:37.588263 21636 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:37.754809 12492 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:37.708267 21381 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:15:37.724893 12639 rpcz_store.cc:276] 0709 12:15:36.075493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.604409 (+1528916us) service_pool.cc:224] Handling call
0709 12:15:37.614483 (+ 10074us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.614846 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.614872 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.614916 (+    44us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:37.614956 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.615009 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.615121 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.742537 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3735) took 2057 ms. Trace:
I20260709 12:15:37.658157 21488 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.710091 21536 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:37.718361 21572 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.014s	sys 0.017s
I20260709 12:15:37.749261 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1188) took 1673 ms. Trace:
I20260709 12:15:37.776918 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:37.776779 12642 rpcz_store.cc:276] 0709 12:15:36.076268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.742700 (+1666432us) service_pool.cc:224] Handling call
0709 12:15:37.748722 (+  6022us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.748917 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.748956 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.749001 (+    45us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:37.749039 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.749094 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.749196 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.711985 21496 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:15:37.777758 12492 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:37.744248 21642 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.738636 21616 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:37.779081 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:37.774461 12570 rpcz_store.cc:276] 0709 12:15:35.684685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.685793 (+  1108us) service_pool.cc:224] Handling call
0709 12:15:35.716974 (+ 31181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:35.717267 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:36.220235 (+502968us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.718526 (+1498291us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.742128 (+ 23602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.742197 (+    69us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:37.742251 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.742321 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.742471 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20333,"prepare.queue_time_us":68968,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":209772,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":7196,"thread_start_us":1983594,"threads_started":4,"wal-append.queue_time_us":1109908}
W20260709 12:15:37.666980 21545 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.477s	user 0.015s	sys 0.037s
I20260709 12:15:37.752482 21099 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:37.789806 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:37.790644 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1189) took 1712 ms. Trace:
I20260709 12:15:37.790856 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:37.713652 21645 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.821525 12489 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:37.821144 12642 rpcz_store.cc:276] 0709 12:15:36.078299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.777922 (+1699623us) service_pool.cc:224] Handling call
0709 12:15:37.790070 (+ 12148us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.790284 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.790309 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.790372 (+    63us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:37.790414 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.790471 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.790580 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:37.822983 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:37.828400 12492 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
W20260709 12:15:37.783677 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 159309us
I20260709 12:15:37.831019 12642 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:37.756940 21494 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:37.838495 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122943us
I20260709 12:15:37.839412 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:37.748103 21168 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.779618 21572 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:37.752269 21589 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.746338 21567 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:37.976498 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:37.977226 12564 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:37.980688 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:37.981400 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:37.982859 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:37.810364 21545 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.629s	user 0.015s	sys 0.038s
I20260709 12:15:37.983553 12566 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 20.
I20260709 12:15:37.988337 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:37.989053 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:37.791965 21514 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:37.757474 21636 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:37.985066 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102488us
W20260709 12:15:37.959434 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 317488us
I20260709 12:15:37.791046 21616 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:37.997548 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:37.998572 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:38.040108 12565 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:15:37.991360 21450 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.586s	user 0.016s	sys 0.001s
I20260709 12:15:38.055037 21450 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.057967 12490 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:38.058996 21670 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:38.059805 21670 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.060249 21670 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.061452 21592 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:37.848270 21496 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:37.841533 21639 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.024449 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.066342 21496 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.070081 21660 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:38.070631 21660 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:38.071133 21496 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.030516 21659 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:38.032100 21653 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.192s	user 0.000s	sys 0.000s
W20260709 12:15:38.032369 21644 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20260709 12:15:38.086314 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:38.033444 21648 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:15:38.035311 21542 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:15:37.999745 12497 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:38.040187 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:37.838090 21099 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:37.782052 21536 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.045022 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.045773 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:38.048813 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:38.051000 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:38.053972 21637 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:38.041715 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:38.111812 12489 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23.
I20260709 12:15:38.000049 21654 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.070886 21533 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.063540 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1210) took 2278 ms. Trace:
I20260709 12:15:38.067854 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:38.000690 21606 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.718s	user 0.012s	sys 0.002s
I20260709 12:15:37.882808 21494 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.078104 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:38.080235 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:38.078009 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:38.020344 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:38.147507 12565 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:38.000146 21395 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.000351 21411 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.089669 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.148135 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:38.149178 12564 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:37.794492 21642 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.094728 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:38.156141 13256 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:38.027594 21381 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:38.108899 13254 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:38.066154 21640 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.113276 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.164390 13257 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 32 because replica has last-logged OpId of term: 30 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:15:38.114120 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:38.068106 21592 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.079309 21599 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:38.002574 21514 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:38.109606 21647 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
I20260709 12:15:38.057415 21450 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:38.090488 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51918.2us
I20260709 12:15:38.151539 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:38.156932 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137039us
I20260709 12:15:38.097247 21542 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:38.115170 21663 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:38.159142 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3712) took 1936 ms. Trace:
I20260709 12:15:38.145200 21654 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:38.159407 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118191us
W20260709 12:15:38.166728 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52684.2us
I20260709 12:15:38.212245 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:38.160084 21664 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:38.181020 12491 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:38.130514 21666 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.124617 21618 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.111282 21447 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:15:38.157275 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:38.230847 12565 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 20.
I20260709 12:15:38.231958 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:38.207005 12563 rpcz_store.cc:276] 0709 12:15:36.223297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.223489 (+   192us) service_pool.cc:224] Handling call
0709 12:15:36.224005 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:36.224547 (+   542us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:36.265291 (+ 40744us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.561734 (+1296443us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.878157 (+316423us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:37.878931 (+   774us) log.cc:844] Serialized 26 byte log entry
0709 12:15:37.973756 (+ 94825us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:37.973838 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.973952 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:38.158849 (+184897us) raft_consensus.cc:1685] finished
0709 12:15:38.158883 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.159066 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85683,"prepare.queue_time_us":474026,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":3515,"raft.queue_time_us":446455,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":39446,"thread_start_us":1474166,"threads_started":4,"wal-append.queue_time_us":985933}
I20260709 12:15:38.235705 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:38.236845 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:38.129411 12495 rpcz_store.cc:276] 0709 12:15:35.784990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.785163 (+   173us) service_pool.cc:224] Handling call
0709 12:15:35.789593 (+  4430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:35.922343 (+132750us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:36.444772 (+522429us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.455442 (+1010670us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.455530 (+    88us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:37.455606 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:38.063195 (+607589us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:38.063253 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.063438 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31716,"prepare.queue_time_us":840804,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":252404,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":33759,"spinlock_wait_cycles":304627584,"thread_start_us":1456857,"threads_started":3,"wal-append.queue_time_us":956902}
I20260709 12:15:38.064495 21572 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 27.10, Last appended by leader: 10, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.166687 21636 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.112208 21637 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:38.168085 21648 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:15:38.216043 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:38.155040 21606 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.257979 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3886) took 1333 ms. Trace:
I20260709 12:15:38.258400 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:38.210980 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:38.066872 21639 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.259929 21606 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.261662 21639 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.221930 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:38.262553 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:38.265146 12564 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
W20260709 12:15:38.227154 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 187019us
I20260709 12:15:38.225581 21640 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.159261 21665 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.168952 21641 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:38.071554 21660 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.129436 21168 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.163430 21533 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:38.205063 21649 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.186372 21541 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:38.116972 21567 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.250193 12565 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:15:38.213284 21663 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:38.255416 21637 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:38.258404 12488 rpcz_store.cc:276] 0709 12:15:36.924511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.976754 (+ 52243us) service_pool.cc:224] Handling call
0709 12:15:36.977339 (+   585us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:36.991750 (+ 14411us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:37.028377 (+ 36627us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.558964 (+530587us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:37.559914 (+   950us) log.cc:844] Serialized 52 byte log entry
0709 12:15:37.560076 (+   162us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:37.560114 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.560187 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:38.257711 (+697524us) raft_consensus.cc:1685] finished
0709 12:15:38.257734 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.257909 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37774,"prepare.queue_time_us":39244,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":3181,"raft.queue_time_us":110908,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":10911,"thread_start_us":273244,"threads_started":1,"wal-append.queue_time_us":560559}
I20260709 12:15:38.158970 21496 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.083s
I20260709 12:15:38.308737 21640 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.272322 12563 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:15:38.343756 21640 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.277074 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:38.221225 21664 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:38.092767 21659 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.270270 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:38.159216 21411 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.167081 21395 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.289117 21676 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:38.185581 21671 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.128s
I20260709 12:15:38.289942 21536 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.290901 21642 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.291066 21514 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.291164 21570 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.291420 21665 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.291821 21542 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.408176 21542 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:38.409595 21665 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.292527 21494 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.293256 21645 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.187692 21599 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.412366 21599 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.230616 21661 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:38.413029 21661 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.415263 21661 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.415918 12642 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:38.181587 21592 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.416397 21654 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.318724 21649 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.417954 21592 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.230880 21666 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.418782 21649 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.419939 21666 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:38.269038 21647 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:15:38.288117 21538 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:38.287142 21670 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.248009 21488 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.239825 21618 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.422236 21395 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.422369 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3570) took 2934 ms. Trace:
I20260709 12:15:38.423027 21618 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.296072 21636 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:15:38.298137 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87272.9us
I20260709 12:15:38.424217 21649 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:38.298346 21669 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:38.425046 21649 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:38.425845 21669 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:38.426333 21649 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.422861 12644 rpcz_store.cc:276] 0709 12:15:35.488335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.207786 (+719451us) service_pool.cc:224] Handling call
0709 12:15:36.216339 (+  8553us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:15:36.247110 (+ 30771us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:37.012389 (+765279us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.590713 (+578324us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:15:37.591868 (+  1155us) log.cc:844] Serialized 78 byte log entry
0709 12:15:37.592126 (+   258us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:37.592465 (+   339us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.785415 (+192950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.785506 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:38.422129 (+636623us) raft_consensus.cc:1685] finished
0709 12:15:38.422149 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.422304 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68066,"prepare.queue_time_us":1080579,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":7289,"raft.queue_time_us":120254,"raft.run_cpu_time_us":403,"raft.run_wall_time_us":19762,"thread_start_us":1207246,"threads_started":3,"wal-append.queue_time_us":669030}
I20260709 12:15:38.298759 21563 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:38.427645 21563 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:38.428046 21563 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:38.300314 21363 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:15:38.301384 21668 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.302610 21474 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.305649 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1574) took 1489 ms. Trace:
I20260709 12:15:38.430585 12569 rpcz_store.cc:276] 0709 12:15:36.816643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.817170 (+   527us) service_pool.cc:224] Handling call
0709 12:15:36.841507 (+ 24337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.000524 (+159017us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.000561 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.000616 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:37.000672 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:38.305367 (+1304695us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54002cfbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:38.305412 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.305558 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365103488}
I20260709 12:15:38.306540 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3040) took 2614 ms. Trace:
I20260709 12:15:38.431802 12646 rpcz_store.cc:276] 0709 12:15:35.692510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.550236 (+857726us) service_pool.cc:224] Handling call
0709 12:15:36.555976 (+  5740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:37.379318 (+823342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:38.306311 (+926993us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:38.306483 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1892020096}
I20260709 12:15:38.428053 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.435559 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:38.430003 21363 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:38.337736 21541 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.443617 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.443930 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.294288 21662 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:38.399523 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 187368us
I20260709 12:15:38.355876 21663 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:38.448160 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.448220 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:38.400233 21677 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:38.433359 21570 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:38.375846 21664 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:38.375626 21673 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.420051 21411 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.426748 21669 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.170045 21381 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.444662 12646 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:38.375765 21660 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:38.430519 21676 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:38.479220 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.480868 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:38.482270 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.483526 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:38.486287 21657 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:15:38.491341 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3887) took 1506 ms. Trace:
I20260709 12:15:38.456425 21665 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:38.429996 21655 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.455940 21662 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.494814 21655 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.457666 21099 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.491973 12496 rpcz_store.cc:276] 0709 12:15:36.985047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.985341 (+   294us) service_pool.cc:224] Handling call
0709 12:15:36.985870 (+   529us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:37.402483 (+416613us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.402520 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:37.450360 (+ 47840us) log.cc:844] Serialized 26 byte log entry
0709 12:15:37.714176 (+263816us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:37.714220 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.715460 (+  1240us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:38.490943 (+775483us) raft_consensus.cc:1685] finished
0709 12:15:38.490979 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.491239 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20797,"raft.queue_time_us":41337,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":62980,"thread_start_us":662996,"threads_started":2,"wal-append.queue_time_us":718589}
I20260709 12:15:38.496806 21662 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:38.457062 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 242290us
I20260709 12:15:38.513455 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:38.516469 21542 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:38.524288 21381 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.463047 21673 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:38.532706 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:38.538273 21659 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.430150 21668 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.545364 21363 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.467944 21676 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:38.570788 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3567) took 3083 ms (client timeout 1811 ms). Trace:
W20260709 12:15:38.571535 12637 rpcz_store.cc:269] 0709 12:15:35.487168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.507689 (+ 20521us) service_pool.cc:224] Handling call
0709 12:15:38.570428 (+3062739us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:38.570672 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3463794432}
I20260709 12:15:38.574548 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.549481 21021 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:37.525532 12638 rpcz_store.cc:276] 0709 12:15:35.489553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.420514 (+930961us) service_pool.cc:224] Handling call
0709 12:15:36.420917 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:36.421163 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:37.032777 (+611614us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.514715 (+481938us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:37.514777 (+    62us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:37.514839 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.514911 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.515067 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2497,"prepare.queue_time_us":936429,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":1986,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":149,"thread_start_us":1042999,"threads_started":2}
I20260709 12:15:38.552505 21541 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.492882 21669 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:38.562037 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 303689us
W20260709 12:15:38.563223 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 376424us
W20260709 12:15:38.552852 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3572) took 3063 ms (client timeout 2018 ms). Trace:
I20260709 12:15:38.567371 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:38.582769 12640 rpcz_store.cc:269] 0709 12:15:35.488940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.899713 (+410773us) service_pool.cc:224] Handling call
0709 12:15:36.307964 (+408251us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:15:38.552551 (+2244587us) spinlock_profiling.cc:243] Waited 98.7 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:38.552740 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364599936}
I20260709 12:15:38.537043 21616 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.584650 12646 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:38.585757 21616 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.587558 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:38.599721 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.604190 21572 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:38.605185 21572 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:38.606487 21572 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:38.605937 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:38.607487 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:38.608234 21682 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:38.609014 21687 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:38.611032 21379 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.611114 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:38.613546 21682 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:38.615516 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:38.616915 21605 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.651s	user 0.000s	sys 0.002s
I20260709 12:15:38.618326 21682 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:15:38.618731 21682 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:38.619058 21682 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:38.624521 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.626070 21589 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.627170 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.577289 21676 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:38.631839 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.633229 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3737) took 2947 ms. Trace:
I20260709 12:15:38.633816 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.634701 21668 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.635143 13257 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:38.635989 21682 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.634739 12568 rpcz_store.cc:276] 0709 12:15:35.685450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.819931 (+134481us) service_pool.cc:224] Handling call
0709 12:15:35.834146 (+ 14215us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:35.835931 (+  1785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:35.835967 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:35.836581 (+   614us) log.cc:844] Serialized 52 byte log entry
0709 12:15:36.488455 (+651874us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:36.488521 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:36.520106 (+ 31585us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:38.616266 (+2096160us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:38.630417 (+ 14151us) raft_consensus.cc:1685] finished
0709 12:15:38.630438 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.633139 (+  2701us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30526,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":10175744,"thread_start_us":651260,"threads_started":1,"wal-append.queue_time_us":925919}
I20260709 12:15:38.595791 21616 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:38.638222 13256 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:38.640946 21649 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:38.641556 21616 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:38.642323 21682 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.637526 21673 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:38.643647 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1177) took 4226 ms (client timeout 1671 ms). Trace:
W20260709 12:15:38.644073 12645 rpcz_store.cc:269] 0709 12:15:34.416984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:34.493141 (+ 76157us) service_pool.cc:224] Handling call
0709 12:15:38.643455 (+4150314us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:38.643572 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4256890752}
I20260709 12:15:38.644380 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.644625 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.645148 13254 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:38.645227 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:38.645653 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:38.647007 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:38.649418 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:38.650348 21598 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:38.655205 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.656134 12642 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:38.657015 21598 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:38.657181 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.657433 21598 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:38.657981 21598 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:38.642645 21616 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.658869 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.659530 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:38.660661 21567 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:38.661161 21567 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:38.662081 21567 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 28.11, Last appended by leader: 11, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.662485 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:38.663404 21686 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.663499 21567 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:38.663751 21631 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:38.664294 21631 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:38.663971 21686 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.664687 21631 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:38.666894 21686 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.625242 21570 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.645258 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.674872 12645 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:38.675828 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.675850 13254 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: [30.20-30.20]
I20260709 12:15:38.677045 12645 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:38.678222 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:38.630064 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:38.679032 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:38.679072 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:38.680298 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50299.2us
I20260709 12:15:38.680418 13255 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:38.681746 21450 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:38.682186 21450 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:38.682529 21450 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:38.682822 21563 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.682367 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:38.684999 21563 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.682374 13260 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:15:38.660514 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.686139 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:38.686945 13262 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:15:38.686978 13258 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:38.686228 12642 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:38.691392 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.692116 12640 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:38.693022 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.693372 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:38.694417 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:38.697582 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.698369 12640 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:38.699319 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:38.700742 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.705109 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:38.646870 21572 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.727982 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:38.690567 21637 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.740360 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 60) took 1759 ms. Trace:
I20260709 12:15:38.699885 21663 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.744465 12494 rpcz_store.cc:276] 0709 12:15:36.981181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.981394 (+   213us) service_pool.cc:224] Handling call
0709 12:15:38.740133 (+1758739us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:38.740290 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1651662208}
I20260709 12:15:38.740829 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3751) took 2419 ms. Trace:
I20260709 12:15:38.747440 12644 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:38.746090 21663 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.746206 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:15:38.750268 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:15:38.751165 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:38.726074 21498 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:15:38.755656 21572 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.746631 12572 rpcz_store.cc:276] 0709 12:15:36.322024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.334251 (+ 12227us) service_pool.cc:224] Handling call
0709 12:15:36.334659 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:36.342313 (+  7654us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:36.378267 (+ 35954us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.011212 (+632945us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.017492 (+  6280us) log.cc:844] Serialized 14 byte log entry
0709 12:15:38.262373 (+1244881us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b1800431aa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1
0709 12:15:38.263820 (+  1447us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b1800431aa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:38.602659 (+338839us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b1800431aa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:15:38.740329 (+137670us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:38.740407 (+    78us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:38.740484 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:38.740579 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:38.740761 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5669,"prepare.queue_time_us":1315099,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":122,"raft.queue_time_us":71463,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":143238,"spinlock_wait_cycles":155348352,"thread_start_us":768427,"threads_started":2,"wal-append.queue_time_us":928123,"wal-append.run_cpu_time_us":2161,"wal-append.run_wall_time_us":1040470}
I20260709 12:15:38.757014 21616 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:38.732973 21474 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:15:38.739701 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:38.746752 21661 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:38.764070 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:38.764147 21616 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.774559 21616 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.781359 12571 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:38.779856 21570 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:38.826442 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:38.770495 21661 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.815090 21664 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.890255 21637 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.895104 21676 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.899243 21598 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:38.901167 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 319399us
I20260709 12:15:38.905277 12637 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
W20260709 12:15:38.907703 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.045s	sys 0.029s
W20260709 12:15:38.908248 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.045s	sys 0.029s
I20260709 12:15:38.919596 21661 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:38.908599 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 781658us
I20260709 12:15:38.931540 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:38.923754 21664 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:38.928270 21598 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.936120 13218 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23.
I20260709 12:15:38.941570 13218 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:15:38.946277 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:38.947367 12494 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:15:38.922389 21631 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:38.953392 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:38.951467 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:38.952610 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:38.959813 12491 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:38.969754 12496 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:38.969950 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:38.974318 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:38.974648 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:38.975674 12488 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 31 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:15:38.967816 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:38.980875 12494 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:38.951144 21678 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23.
I20260709 12:15:38.956202 21450 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:38.999389 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:38.987088 21676 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:38.990537 21590 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:38.990952 21528 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:39.011475 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1194) took 1019 ms. Trace:
W20260709 12:15:39.007254 21701 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:15:39.005947 21590 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:39.002748 21593 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:15:39.010973 21676 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:39.012614 21666 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:39.021482 21631 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.010732 21528 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.027941 21528 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.029754 21450 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.030118 21590 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.031615 21590 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:39.031711 21445 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.041982 21655 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.013216 12640 rpcz_store.cc:276] 0709 12:15:37.991737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:38.734525 (+742788us) service_pool.cc:224] Handling call
0709 12:15:38.741461 (+  6936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:38.741771 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:38.992355 (+250584us) log.cc:844] Serialized 14 byte log entry
0709 12:15:39.010396 (+ 18041us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:39.010502 (+   106us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:39.010600 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:39.010932 (+   332us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:39.011361 (+   429us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8140,"prepare.queue_time_us":36015,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":1728,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2770,"thread_start_us":247593,"threads_started":2,"wal-append.queue_time_us":27577}
W20260709 12:15:39.087250 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:39.088274 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:39.090826 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.091703 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.092031 12495 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28. Re-sending same reply.
I20260709 12:15:39.093936 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:39.095628 12640 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23.
I20260709 12:15:39.099472 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.102725 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1193) took 1141 ms. Trace:
I20260709 12:15:39.108758 21445 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.115468 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.117157 12488 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:39.124991 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3883) took 2377 ms. Trace:
I20260709 12:15:39.133806 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.134476 12495 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:39.135880 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.136251 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.133898 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:39.125592 12490 rpcz_store.cc:276] 0709 12:15:36.746242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.762741 (+ 16499us) service_pool.cc:224] Handling call
0709 12:15:36.763213 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:38.058518 (+1295305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:39.123043 (+1064525us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:39.123684 (+   641us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2977023744}
I20260709 12:15:39.160813 21707 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:39.161885 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:39.165797 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:39.165577 12492 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:15:39.166018 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:39.192909 21445 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.196779 21445 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.197971 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.198602 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.170562 21528 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.199569 12488 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:39.200207 21528 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.199579 12638 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:39.202761 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.113525 12642 rpcz_store.cc:276] 0709 12:15:37.985878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:38.712366 (+726488us) service_pool.cc:224] Handling call
0709 12:15:38.712866 (+   500us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:38.730222 (+ 17356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:38.730260 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:38.748953 (+ 18693us) log.cc:844] Serialized 26 byte log entry
0709 12:15:38.831114 (+ 82161us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:38.856653 (+ 25539us) log.cc:844] Serialized 14 byte log entry
0709 12:15:38.856961 (+   308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:38.857040 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:39.102031 (+244991us) raft_consensus.cc:1685] finished
0709 12:15:39.102080 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:39.102442 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1775,"prepare.queue_time_us":26726,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":18607,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":6481,"thread_start_us":74348,"threads_started":1,"wal-append.queue_time_us":170888}
I20260709 12:15:37.503794 12641 rpcz_store.cc:276] 0709 12:15:35.443503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.191663 (+748160us) service_pool.cc:224] Handling call
0709 12:15:36.192160 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:36.213758 (+ 21598us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:36.213816 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:36.253579 (+ 39763us) log.cc:844] Serialized 26 byte log entry
0709 12:15:37.004257 (+750678us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:37.496254 (+491997us) log.cc:844] Serialized 14 byte log entry
0709 12:15:37.497988 (+  1734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:37.498088 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:37.498139 (+    51us) raft_consensus.cc:1685] finished
0709 12:15:37.498153 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:37.498322 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36067,"prepare.queue_time_us":353619,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":17767,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":22254,"thread_start_us":1134915,"threads_started":2,"wal-append.queue_time_us":1020303}
I20260709 12:15:39.208644 21233 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:39.209250 12642 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: [30.20-30.20]
W20260709 12:15:39.210537 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:39.235476 21447 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:39.236304 12645 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:39.238992 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.239703 12572 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:15:39.240110 13257 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:39.240376 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:39.215284 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 788773us
I20260709 12:15:39.245559 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.246915 12567 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:39.216035 13218 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 22 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23.
I20260709 12:15:39.198107 21578 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:39.250074 21578 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:39.250607 21578 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:39.250782 21690 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.651s	user 0.000s	sys 0.007s
W20260709 12:15:39.253026 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:39.254689 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.255888 12490 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:39.256798 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:39.263721 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1213) took 2932 ms (client timeout 1589 ms). Trace:
W20260709 12:15:39.264540 12497 rpcz_store.cc:269] 0709 12:15:36.331403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.336627 (+  5224us) service_pool.cc:224] Handling call
0709 12:15:39.263096 (+2926469us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.263419 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3641022848}
I20260709 12:15:39.266330 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3587) took 1152 ms. Trace:
I20260709 12:15:39.266906 12637 rpcz_store.cc:276] 0709 12:15:38.113321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:38.592767 (+479446us) service_pool.cc:224] Handling call
0709 12:15:39.265763 (+672996us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.266111 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":700070144}
I20260709 12:15:39.273893 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:39.274770 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:39.217897 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3036) took 3611 ms (client timeout 1755 ms). Trace:
I20260709 12:15:39.271873 21632 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.219806 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3582) took 1108 ms. Trace:
I20260709 12:15:39.220151 12496 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:39.277697 12644 rpcz_store.cc:276] 0709 12:15:38.111710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:38.447878 (+336168us) service_pool.cc:224] Handling call
0709 12:15:39.219384 (+771506us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.219631 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":676700416}
I20260709 12:15:39.222532 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3579) took 1111 ms. Trace:
I20260709 12:15:39.225667 12492 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:15:39.226871 21381 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.287371 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.287688 21381 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.289113 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.280228 12646 rpcz_store.cc:276] 0709 12:15:38.110966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:38.483253 (+372287us) service_pool.cc:224] Handling call
0709 12:15:39.222134 (+738881us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.222361 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230511872}
I20260709 12:15:39.294611 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3721) took 1314 ms. Trace:
I20260709 12:15:39.295225 12571 rpcz_store.cc:276] 0709 12:15:37.980226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.980392 (+   166us) service_pool.cc:224] Handling call
0709 12:15:39.294076 (+1313684us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.294318 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1839436160}
I20260709 12:15:39.227073 12488 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:15:39.298482 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:55578 (request call id 10) took 3042 ms. Trace:
I20260709 12:15:39.228710 21429 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:39.234999 21547 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.238572 21447 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.240128 21666 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.219560 21655 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:39.276081 12643 rpcz_store.cc:269] 0709 12:15:35.606511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:35.669440 (+ 62929us) service_pool.cc:224] Handling call
0709 12:15:39.217273 (+3547833us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.217602 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3590204672}
I20260709 12:15:39.214090 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:39.300741 21665 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.227752 21496 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.307536 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.308768 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.278887 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.310331 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3580) took 1199 ms. Trace:
I20260709 12:15:39.298686 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.308908 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.283865 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:39.309108 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.316857 12639 rpcz_store.cc:276] 0709 12:15:36.256538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.778892 (+1522354us) service_pool.cc:224] Handling call
0709 12:15:39.271071 (+1492179us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f9735 00007ffbf57f19fa 00007ffbfa7b94c9 00007ffbf3d35c54 00007ffbf3d35be2 00007ffbf3d35b11 00007ffbf3d35a8c 00007ffbf3d34d42 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:39.298236 (+ 27165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":922707840}
I20260709 12:15:39.334316 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.335531 13260 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:39.336630 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.310593 21632 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.334026 21496 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.337590 12496 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:39.338187 21665 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.354494 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.339735 12497 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:39.326871 12645 rpcz_store.cc:276] 0709 12:15:38.111208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:38.914577 (+803369us) service_pool.cc:224] Handling call
0709 12:15:39.235715 (+321138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:39.236926 (+  1211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:39.310071 (+ 73145us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b5400153e50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:39.310260 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":737245568}
I20260709 12:15:39.338919 21547 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:39.356698 21700 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.424s	user 0.001s	sys 0.004s
I20260709 12:15:39.357158 21665 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.358428 21547 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.341850 21429 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:39.346300 12495 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:39.348515 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.311646 21528 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:39.364612 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:39.366966 21381 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.354070 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:39.380427 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:39.381270 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107547us
I20260709 12:15:39.370878 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.372135 12563 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:39.209263 21233 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:39.342087 21666 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:39.406163 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:39.406970 21233 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.364763 12386 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 19 to 21, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:39.348379 21632 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.392225 21655 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.340977 21447 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.382046 21429 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:39.348722 21496 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.419816 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:39.425756 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:39.428755 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75053.7us
W20260709 12:15:39.447448 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74300.3us
I20260709 12:15:39.414459 12492 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:15:39.422354 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:39.426460 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.434912 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3581) took 1323 ms. Trace:
I20260709 12:15:39.409598 21233 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:39.470669 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:39.413861 12638 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:39.474704 21466 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:39.474931 21545 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.475741 21466 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.475941 21545 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.478437 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.477619 21716 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:39.483940 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61870.4us
I20260709 12:15:39.484197 21716 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.478504 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.487183 21632 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:15:39.487504 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68134.8us
I20260709 12:15:39.487933 21632 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:39.482076 12497 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:15:39.463986 21533 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.474395 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.478825 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.493048 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:39.492689 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:39.484946 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:39.492249 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:39.526284 13258 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:39.459187 12640 rpcz_store.cc:276] 0709 12:15:38.111468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:39.096845 (+985377us) service_pool.cc:224] Handling call
0709 12:15:39.097940 (+  1095us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:39.099086 (+  1146us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:39.099149 (+    63us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:39.100468 (+  1319us) log.cc:844] Serialized 52 byte log entry
0709 12:15:39.156416 (+ 55948us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:39.156490 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:39.160055 (+  3565us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:39.434708 (+274653us) raft_consensus.cc:1685] finished
0709 12:15:39.434729 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:39.434852 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8684,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":338,"thread_start_us":55105,"threads_started":1,"wal-append.queue_time_us":331171}
I20260709 12:15:39.412413 21160 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:39.497427 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.495837 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:39.534355 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:39.428318 21655 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.536962 21160 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:39.538468 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.538854 13260 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:39.539440 12491 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.012s
I20260709 12:15:39.546350 12496 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:39.547698 21716 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:39.460801 12386 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 8 to 19, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 19 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:39.489295 21632 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.574476 12491 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:39.575671 12386 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 19, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 19 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:15:39.494156 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.583756 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:39.558341 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.552410 13258 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:39.558058 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:39.490831 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3888) took 2460 ms. Trace:
I20260709 12:15:39.424698 21707 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:39.489573 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:39.493566 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.570680 21578 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:39.596838 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69145us
I20260709 12:15:39.601270 13254 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:15:39.603533 13253 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:15:39.608906 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124102us
I20260709 12:15:39.611594 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.612691 12570 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:39.586802 21160 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:39.587878 21381 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:39.624130 13262 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:39.628415 21541 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:39.629951 21528 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:39.626928 13257 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:15:39.639325 12638 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:39.639292 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.495364 21533 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.619316 21716 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:39.616323 21655 raft_consensus.cc:2764] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:15:39.624029 21381 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:39.657362 12488 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:39.435783 21429 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.669173 21466 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.590270 12493 rpcz_store.cc:276] 0709 12:15:37.015783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:37.016020 (+   237us) service_pool.cc:224] Handling call
0709 12:15:37.016557 (+   537us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:15:37.078944 (+ 62387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:37.078977 (+    33us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:15:37.079711 (+   734us) log.cc:844] Serialized 78 byte log entry
0709 12:15:37.651522 (+571811us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:37.670588 (+ 19066us) log.cc:844] Serialized 14 byte log entry
0709 12:15:38.547898 (+877310us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b1800463d80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:39.128795 (+580897us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b1800463d80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:15:39.148940 (+ 20145us) log.cc:844] Serialized 14 byte log entry
0709 12:15:39.468863 (+319923us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:39.469395 (+   532us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:39.469758 (+   363us) raft_consensus.cc:1685] finished
0709 12:15:39.469779 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:39.475757 (+  5978us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1457312,"prepare.queue_time_us":1514646,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":101,"raft.queue_time_us":308150,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":9905,"spinlock_wait_cycles":26928384,"thread_start_us":621744,"threads_started":2,"wal-append.queue_time_us":600029,"wal-append.run_cpu_time_us":2627,"wal-append.run_wall_time_us":1153022}
W20260709 12:15:39.670063 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 180626us
W20260709 12:15:39.672349 21720 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.095s	user 0.003s	sys 0.000s
I20260709 12:15:39.684749 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.674461 12386 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 16, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 16 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:39.664278 21381 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:39.676616 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58027us
W20260709 12:15:39.681646 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43731.9us
I20260709 12:15:39.690156 13260 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:39.657689 21716 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:39.705855 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:39.707427 21578 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.710103 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:39.710609 21715 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:15:39.718356 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:39.719578 21233 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:39.718573 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:39.723080 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.724109 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:39.697772 21716 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:39.732156 21160 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.734650 12571 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:15:39.755856 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.753249 12572 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:39.768388 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:39.767879 21498 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:39.774407 21498 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.774863 21498 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:39.775111 21655 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:39.775503 21655 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:39.775885 21655 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:39.777920 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:39.778707 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.779498 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.780841 12645 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 32 because replica has last-logged OpId of term: 30 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:15:39.782131 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.782145 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.778103 12493 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
W20260709 12:15:39.781435 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:39.791234 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92272.9us
I20260709 12:15:39.794560 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:39.786254 12493 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
I20260709 12:15:39.798062 12493 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:15:39.805519 12493 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:39.784260 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.774922 21707 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:39.832372 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111836us
I20260709 12:15:39.820400 12638 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:39.818132 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.837200 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.845114 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.829838 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:39.863766 12640 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:39.870247 21643 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:39.870842 21643 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.871223 21643 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:39.874713 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.875281 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.875780 12567 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:39.876324 12568 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:39.877945 12569 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 12:15:39.878362 12569 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 12
I20260709 12:15:39.878677 12569 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:15:39.879009 12569 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:39.880626 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.881539 12568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:39.884729 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.885569 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:39.891315 21429 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:39.892418 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:39.893901 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45941.8us
I20260709 12:15:39.833496 21665 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:39.912866 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48246.8us
I20260709 12:15:39.913206 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.915705 12567 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:39.903151 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:39.878352 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:39.848145 21547 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:39.896618 12637 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:39.925833 12571 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:15:39.921394 12639 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:39.930287 21547 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:39.921718 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:39.945565 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:39.920068 21466 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:15:39.953691 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 124065us
I20260709 12:15:39.858067 21707 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.969542 21021 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:39.970120 21665 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:39.973865 21590 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:39.974614 21590 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.943451 21547 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:39.982326 21533 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.936271 21541 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.983597 21541 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.987119 21160 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:39.988907 21469 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:39.990010 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.993186 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:39.994031 12569 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:39.995483 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:39.962544 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62442.2us
I20260709 12:15:40.011119 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:40.012428 21021 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:15:40.013149 21665 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:40.016034 21665 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:40.016598 21665 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:40.017038 21665 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:40.018147 21233 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:40.020402 21533 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:40.021522 21533 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:40.022051 21533 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:40.023253 21533 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.9, Last appended by leader: 10, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:39.940711 21496 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:39.964444 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.026361 21496 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.027184 12568 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:39.965272 21528 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:40.029397 21528 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:40.030887 21528 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:39.996273 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:39.995484 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.034785 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.035123 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.038854 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.039916 12569 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:40.041069 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.028684 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.043064 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.043776 21707 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:40.044687 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54880.7us
I20260709 12:15:40.002012 21601 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:39.873416 21498 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:39.960839 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:40.052103 12492 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.556s	user 0.008s	sys 0.006s
I20260709 12:15:40.053891 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.054230 12569 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:15:40.058533 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:40.060729 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:40.074577 12567 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:40.094933 12571 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:40.072075 21545 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.087954 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:40.071167 21716 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:40.091190 21496 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:40.128031 12570 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:40.128912 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.121331 21578 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:40.139096 12571 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:40.142007 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 398998us
I20260709 12:15:40.127305 21496 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:40.136521 21632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.019s	sys 0.017s
I20260709 12:15:40.175647 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:40.177742 12490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.040s	sys 0.043s
W20260709 12:15:40.178335 12490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.040s	sys 0.043s
I20260709 12:15:40.140363 21578 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:40.170866 21632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.019s	sys 0.018s
W20260709 12:15:40.172375 21666 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.762s	user 0.018s	sys 0.000s
I20260709 12:15:40.086565 12386 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 12 to 19, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 19 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:40.206292 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.171958 21541 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.211604 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:15:40.201840 21578 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:40.216533 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.217335 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:40.219669 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 122112us
W20260709 12:15:40.220127 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47634.3us
I20260709 12:15:40.134646 21381 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:40.222743 12570 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 11. Preceding OpId from leader: term: 29 index: 11. (term mismatch)
I20260709 12:15:40.235810 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:40.240572 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:40.240896 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:40.202500 21496 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.271236 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:40.250800 12570 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:15:40.272747 12570 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 28.11
W20260709 12:15:40.272898 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:40.273124 12570 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:40.235874 21498 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.283540 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:40.284650 12567 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:40.210487 21666 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:40.292531 12571 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:40.253305 21716 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.319888 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.324473 12572 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:40.310622 21547 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:40.348153 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107741us
I20260709 12:15:40.364596 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:40.348440 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 107721us
I20260709 12:15:40.359314 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.368568 12570 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:40.370546 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.360186 21496 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:40.361829 21666 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.372898 21666 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:40.373472 21666 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:40.374959 21666 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:40.375751 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1105077700001402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260709 12:15:40.376569 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0618098639999971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
I20260709 12:15:40.378434 21498 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:40.378914 21498 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:40.379097 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:40.379794 21381 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.291543 21601 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:15:40.381734 21601 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:40.381978 21498 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:40.382154 21601 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:40.382566 21601 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:40.382833 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:40.337498 12386 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 20. New cstate: current_term: 20 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:15:40.391160 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:40.296020 21545 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:40.401872 12570 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:40.314677 21590 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.421149 21601 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:40.422116 21643 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.550s	user 0.008s	sys 0.007s
I20260709 12:15:40.422591 21590 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:40.422816 21643 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:40.423754 21590 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:40.397450 21545 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:40.428561 12571 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [23.12-23.12]
W20260709 12:15:40.411375 21655 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.635s	user 0.015s	sys 0.001s
I20260709 12:15:40.433344 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.439229 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.440575 21643 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.440831 21590 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:40.441488 21590 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:40.431330 21545 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:40.434165 21429 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.003s
W20260709 12:15:40.445232 12563 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.003s	sys 0.056s
W20260709 12:15:40.447651 12563 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.003s	sys 0.057s
I20260709 12:15:40.451153 12568 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260709 12:15:40.451153 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.442009 21590 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:40.463131 21665 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.435355 21655 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:40.492389 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 479601us
W20260709 12:15:40.472968 12565 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.006s	sys 0.026s
I20260709 12:15:40.478196 21601 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.444226 21498 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.481184 12386 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 12 to 22, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 22 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:15:40.501494 12565 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.006s	sys 0.026s
W20260709 12:15:40.509948 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:40.510542 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:40.510841 21429 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:40.511606 21429 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.514483 21429 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:40.503315 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 534548us
I20260709 12:15:40.519773 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.509431 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.527657 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.527040 12495 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:40.532258 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.498477 21643 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:40.528887 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:40.534350 12488 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:15:40.535109 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.536791 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.546721 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.547724 12492 raft_consensus.cc:2368] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:40.546686 12570 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:40.548496 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.549212 12493 raft_consensus.cc:2368] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:40.550252 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.552410 12491 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:15:40.549664 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.555686 12497 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:40.558549 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.556268 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:40.567371 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.568562 12488 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:15:40.568781 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.570731 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.572167 12489 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:40.534004 21021 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:15:40.576613 21021 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:40.577051 21021 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:40.571153 21498 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:40.577384 21021 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:40.577457 12567 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:15:40.581753 12570 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:40.579607 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.578245 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:40.583742 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.592912 12497 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:15:40.599788 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.600806 12488 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:15:40.601809 12494 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:40.601996 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.602300 12567 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:40.579188 21707 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:15:40.605991 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85722us
I20260709 12:15:40.607029 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:40.593752 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.040s	sys 0.022s
I20260709 12:15:40.612062 12490 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:40.613811 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.040s	sys 0.023s
W20260709 12:15:40.614338 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 606685us
I20260709 12:15:40.583056 21429 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:40.652704 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.601557 21632 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:40.658191 12611 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:40.658830 12611 leader_election.cc:403] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:15:40.614280 21707 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:40.661082 12567 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23. Re-sending same reply.
I20260709 12:15:40.664533 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:40.664661 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:40.666558 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:40.675635 12489 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:40.679183 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.680023 12567 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:40.659786 21632 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:40.669082 21707 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:40.611927 21643 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:40.700193 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50842us
I20260709 12:15:40.650527 21545 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.703863 12492 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.10-27.10]
W20260709 12:15:40.709553 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 568385us
I20260709 12:15:40.709892 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.718189 12567 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:40.723657 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.011s	sys 0.029s
I20260709 12:15:40.730580 21632 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.692502 21498 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:15:40.724517 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.012s	sys 0.030s
W20260709 12:15:40.745105 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 359775us
I20260709 12:15:40.724498 21590 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.753213 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:40.733093 21632 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:15:40.761085 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 383778us
I20260709 12:15:40.682647 21429 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:40.761310 12641 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.022s	sys 0.025s
W20260709 12:15:40.771189 12641 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.023s	sys 0.026s
I20260709 12:15:40.765976 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:40.754110 12572 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:15:40.775121 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:40.768512 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3777) took 1021 ms. Trace:
W20260709 12:15:40.759115 21578 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.516s	user 0.015s	sys 0.001s
I20260709 12:15:40.779086 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3601) took 2242 ms. Trace:
W20260709 12:15:40.775636 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 341879us
I20260709 12:15:40.785084 12495 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:40.782105 12641 rpcz_store.cc:276] 0709 12:15:38.530478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:39.277046 (+746568us) service_pool.cc:224] Handling call
0709 12:15:39.277555 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:39.278293 (+   738us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:39.312991 (+ 34698us) log.cc:844] Serialized 14 byte log entry
0709 12:15:40.771760 (+1458769us) log.cc:844] Serialized 14 byte log entry
0709 12:15:40.772178 (+   418us) log.cc:844] Serialized 14 byte log entry
0709 12:15:40.772368 (+   190us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:40.772423 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:40.772506 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:40.772606 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:40.772786 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11345,"prepare.queue_time_us":234,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":7269,"raft.queue_time_us":21164,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":6583,"thread_start_us":1308701,"threads_started":2,"wal-append.queue_time_us":817192}
I20260709 12:15:40.750777 21655 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:40.788713 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 222294us
I20260709 12:15:40.755729 21528 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:40.792412 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.792811 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:40.796082 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:40.762125 21665 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:40.797425 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 185807us
I20260709 12:15:40.780282 12563 rpcz_store.cc:276] 0709 12:15:39.747021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:39.755259 (+  8238us) service_pool.cc:224] Handling call
0709 12:15:39.820093 (+ 64834us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:39.886067 (+ 65974us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:40.455971 (+569904us) log.cc:844] Serialized 14 byte log entry
0709 12:15:40.764847 (+308876us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:40.764987 (+   140us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:40.765169 (+   182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:40.765366 (+   197us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:40.768409 (+  3043us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":352653,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":28644,"thread_start_us":809298,"threads_started":2}
W20260709 12:15:40.793677 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.822s (SENT) [suppressed 22 similar messages]
W20260709 12:15:40.800837 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:40.782454 21498 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:40.804683 21601 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:40.805737 21601 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.805012 21666 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:40.819330 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:40.819466 21737 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:40.820369 21737 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:40.820163 21665 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:15:40.821947 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.822227 21733 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:40.822844 21733 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:40.823269 21733 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:40.822572 12563 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28. Re-sending same reply.
I20260709 12:15:40.823580 21498 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:40.823385 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:40.824800 21736 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:15:40.821492 21737 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:40.824818 21498 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.827383 21666 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:40.828228 21666 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:40.823381 21665 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:40.828680 21666 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:40.829172 21665 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:40.830102 21665 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.833719 21666 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.835436 21666 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:40.844203 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78159.4us
W20260709 12:15:40.845307 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51747.1us
I20260709 12:15:40.850656 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:40.857095 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4067016070000591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2831s for a response
I20260709 12:15:40.858048 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:40.807054 21528 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:40.790342 21578 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:40.859791 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.029s	sys 0.055s
W20260709 12:15:40.865732 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52781.3us
W20260709 12:15:40.867687 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.030s	sys 0.055s
I20260709 12:15:40.819545 21734 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:40.802740 21469 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.057s
W20260709 12:15:40.882052 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91071.2us
I20260709 12:15:40.886149 21730 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:40.886750 21730 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:40.887015 21590 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:40.893045 21528 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:40.893438 21547 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:40.895115 21578 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:40.902482 12569 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 12:15:40.902971 12569 pending_rounds.cc:85] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 12
I20260709 12:15:40.903009 12461 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
I20260709 12:15:40.921280 12461 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:15:40.903947 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:40.883946 21742 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:40.923439 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:40.870436 21731 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:40.879570 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.913028 21429 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:40.914067 21545 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:40.915879 21160 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:40.912871 21021 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:40.930347 12570 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:15:40.917959 21747 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.059s	user 0.002s	sys 0.000s
I20260709 12:15:40.887179 21730 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:40.931915 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46388.2us
I20260709 12:15:40.922971 12569 pending_rounds.cc:107] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 12:15:40.929149 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:40.961048 12569 raft_consensus.cc:2889] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:40.961390 12567 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:40.875070 21541 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:40.962472 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.963351 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:40.966524 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:40.967736 13258 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:40.895103 21643 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:40.971324 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111737us
I20260709 12:15:40.974646 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:40.958177 21590 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
I20260709 12:15:40.981178 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:40.982606 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:40.982700 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:40.987049 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:40.989097 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:40.989097 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:40.966964 21541 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:40.990752 13256 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:40.988135 12637 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:40.969053 21528 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:40.995687 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:40.993584 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:40.968809 21533 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:40.997722 13258 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:40.997768 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:40.997248 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.001783 12570 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:15:41.002660 12569 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:15:41.003641 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.004387 12641 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:41.006297 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.003440 12569 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:40.995659 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:41.002974 12639 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:41.014613 12640 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 21: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:41.015607 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.010193 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:41.020711 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:40.994076 21754 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:40.994817 21541 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:41.022948 12646 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:41.023818 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.024761 12567 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:40.994346 21233 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:41.026907 21021 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.026557 12569 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.027694 12569 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:15:41.029744 21750 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.003s	sys 0.001s
W20260709 12:15:41.029877 21746 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.000s	sys 0.002s
I20260709 12:15:41.030825 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:40.932161 21742 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.017241 21601 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.036860 13258 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:41.031594 21733 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.040118 13260 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.20-26.20]
I20260709 12:15:41.043113 21733 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.038213 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3783) took 1162 ms. Trace:
I20260709 12:15:41.049876 12565 rpcz_store.cc:276] 0709 12:15:39.875763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:39.880741 (+  4978us) service_pool.cc:224] Handling call
0709 12:15:39.881311 (+   570us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:39.881922 (+   611us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:15:40.509711 (+627789us) log.cc:844] Serialized 14 byte log entry
0709 12:15:40.789847 (+280136us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:40.805757 (+ 15910us) log.cc:844] Serialized 52 byte log entry
0709 12:15:40.805919 (+   162us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:40.806160 (+   241us) log.cc:844] Serialized 14 byte log entry
0709 12:15:40.806382 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:40.806464 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:40.888084 (+ 81620us) spinlock_profiling.cc:243] Waited 5.66 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:41.037838 (+149754us) raft_consensus.cc:1685] finished
0709 12:15:41.037865 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:41.038092 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":376641,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":96,"raft.queue_time_us":7823,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":14950,"spinlock_wait_cycles":13022464,"thread_start_us":858131,"threads_started":2,"wal-append.queue_time_us":323210}
I20260709 12:15:41.045782 21736 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:41.052704 12569 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:15:41.052614 21736 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:41.032349 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129531us
I20260709 12:15:40.999279 21429 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:41.053920 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58386.3us
I20260709 12:15:41.053115 12569 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 10
I20260709 12:15:41.076148 12569 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:15:41.076541 12569 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:41.057128 12646 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:41.074230 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:41.078428 13258 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:41.032193 12644 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:41.021876 21528 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.210227 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:41.212671 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.104535 12492 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:15:41.193356 12640 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:41.106141 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.197175 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.105664 12494 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260709 12:15:41.192551 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 211664us
I20260709 12:15:41.236352 12639 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:41.092864 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:41.089875 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:41.108805 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.117744 12496 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:15:41.239559 13257 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:41.082152 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.193027 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:41.112537 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:41.241051 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:41.110103 12491 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:15:41.210606 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.237071 12637 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:41.235740 12565 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
W20260709 12:15:41.060868 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.145244 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.188930 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:41.244204 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:15:41.186199 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 205559us
I20260709 12:15:41.108342 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:41.089381 12493 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:15:41.247169 12570 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:41.247051 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:41.239686 12488 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:41.109280 21739 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:15:41.147689 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:41.193691 21755 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:15:41.234407 12641 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:41.267542 21752 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:41.268033 21547 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:41.268373 21752 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.269063 21547 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:41.269660 21547 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:41.271703 21757 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
W20260709 12:15:41.273209 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78503.4us
I20260709 12:15:41.275972 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:41.245896 21742 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.119704 21601 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.145s
I20260709 12:15:41.248790 21643 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:41.284642 21021 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:41.286425 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177739us
I20260709 12:15:41.287209 12638 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:15:41.291344 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:41.109437 21733 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.158s
I20260709 12:15:41.143635 21233 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.295557 21233 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.296533 21233 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:41.296909 21233 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:41.297840 21233 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.282670 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:41.299816 21381 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:41.300277 21381 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:41.301162 21381 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.301832 12496 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:41.303085 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.304548 21447 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:15:41.304869 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 201208us
I20260709 12:15:41.305768 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.306573 21733 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:41.307339 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.307152 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:41.104769 21429 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.054267 21533 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.273310 21466 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:41.109191 21496 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.167s
I20260709 12:15:41.314640 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:41.321621 12645 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:41.327328 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:41.075856 21754 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.270836 21547 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.339854 21758 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.076s
W20260709 12:15:41.340726 21632 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.564s	user 0.013s	sys 0.000s
I20260709 12:15:41.339926 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:41.343684 13260 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:41.316171 21545 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:41.348979 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.358816 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.359860 12567 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:41.370087 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:41.374187 13258 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:41.322407 21655 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:15:41.310730 21643 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.342744 21732 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:41.383826 21601 consensus_queue.cc:1243] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:15:41.385946 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.380971 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.362576 21758 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:41.372270 21160 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:15:41.396853 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93881us
I20260709 12:15:41.392158 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:41.336006 21466 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.382335 21498 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:41.409796 12490 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:41.411580 12488 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:41.413496 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:41.413691 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 140263us
I20260709 12:15:41.414415 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.417729 21772 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:41.420024 21655 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:41.421030 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.388166 21632 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:41.376642 21742 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
W20260709 12:15:41.441820 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.422330 12641 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:41.420490 21655 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:41.428838 21736 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:41.369714 21545 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.436012 21528 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.446228 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
W20260709 12:15:41.461656 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 110181us
I20260709 12:15:41.451889 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.463121 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:41.464008 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156928us
I20260709 12:15:41.460768 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:41.466190 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.466809 12638 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:41.468648 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:41.468865 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89922.1us
I20260709 12:15:41.470106 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.471040 12645 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:15:41.471112 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:41.481114 21160 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:41.422542 21730 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.481724 21160 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:41.481974 21744 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:41.482704 21740 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:15:41.483628 21756 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
I20260709 12:15:41.484894 21528 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:41.485358 21528 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:41.485770 21528 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:41.486089 12566 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:41.487035 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:41.492069 21730 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:41.477572 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104727us
I20260709 12:15:41.495923 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:41.501780 21469 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:41.492794 21496 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:41.408056 21732 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:41.511741 21765 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:15:41.511463 21496 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.469319 21601 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.046s
I20260709 12:15:41.517355 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.513899 12643 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:15:41.441197 21498 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.562744 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.565007 12645 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:41.565279 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48038us
I20260709 12:15:41.414357 21758 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.506688 21655 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.567662 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:41.514413 21590 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:41.528807 21730 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:41.514992 21466 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.513502 21732 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:41.590828 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147496us
I20260709 12:15:41.593431 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:41.594461 12640 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:41.594895 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.569588 21733 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:41.563380 21429 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.570156 21752 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.526118 21734 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.082s
I20260709 12:15:41.612946 21734 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 30.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:15:41.612672 21767 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.083s
I20260709 12:15:41.564152 21632 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:15:41.589762 21496 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.587661 21730 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:41.619563 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:41.622030 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55447.2us
I20260709 12:15:41.626845 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.613773 21590 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:41.634668 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140439us
I20260709 12:15:41.638522 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.644469 12645 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:41.648454 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.649292 12645 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:41.629382 21730 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:41.621421 21533 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.615724 21774 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:41.611485 21732 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:15:41.654079 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:41.655120 12645 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:15:41.646782 21754 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:41.694428 21545 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.688421 21665 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:41.695423 21770 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:41.696112 21770 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:41.696756 21666 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.848s	user 0.009s	sys 0.002s
I20260709 12:15:41.695518 21665 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:41.698956 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46866.4us
I20260709 12:15:41.702178 13257 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:15:41.650945 21496 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:41.719192 12643 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:15:41.758035 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46153.4us
I20260709 12:15:41.656139 21752 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:15:41.758793 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.684960 21778 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:41.664567 21632 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:15:41.685619 21740 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.099s
I20260709 12:15:41.781018 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.690289 21469 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.076s
I20260709 12:15:41.760449 21160 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:41.804417 21541 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.778s	user 0.007s	sys 0.006s
W20260709 12:15:41.817152 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 297459us
I20260709 12:15:41.807155 21496 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:41.802803 21590 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.825908 21771 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.142s
W20260709 12:15:41.831610 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 303370us
I20260709 12:15:41.836069 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.837321 12569 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:41.838086 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.838841 12572 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:41.840803 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.843056 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.843415 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.843984 12568 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:41.845232 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.846138 12564 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:41.840165 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:41.849295 12567 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 28
W20260709 12:15:41.825987 21780 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.243s	user 0.002s	sys 0.000s
I20260709 12:15:41.854005 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.855665 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.861804 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:41.862469 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.871703 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:41.861686 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:41.870761 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111686us
I20260709 12:15:41.877791 21770 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.879371 21665 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.883252 21758 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.883980 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:15:41.884395 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:41.885862 21772 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:41.887804 21772 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:41.888262 21772 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:41.888602 21772 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:41.788498 21778 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:41.831844 21496 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:41.891403 21021 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:15:41.892599 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 321462us
I20260709 12:15:41.893945 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.886281 21160 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:41.917193 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.918275 13257 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:15:41.863950 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.821065 21541 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:41.926648 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.927954 12564 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 25 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:15:41.853929 21498 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.930464 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:41.932852 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:41.925870 21771 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:41.891647 21778 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:41.913677 21496 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:41.945611 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.945952 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:41.921859 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:41.915943 21021 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:41.951921 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 260513us
I20260709 12:15:41.974627 12497 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24.
I20260709 12:15:41.966588 12567 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:15:41.976493 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102003us
I20260709 12:15:41.976394 13256 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:15:41.967780 12568 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:41.977672 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:41.978919 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:41.987079 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:41.989360 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.019s	sys 0.043s
I20260709 12:15:41.947337 21160 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:41.990161 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.020s	sys 0.043s
W20260709 12:15:41.993155 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47619.2us
I20260709 12:15:42.000540 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:42.001543 12564 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:42.004463 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.005329 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.008836 12566 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: [26.20-26.20]
I20260709 12:15:42.013193 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:42.018004 12572 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:42.019884 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:41.986129 21429 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:42.030783 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:42.026230 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48668.7us
W20260709 12:15:42.021495 21528 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.007s
I20260709 12:15:42.012486 21601 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:41.947144 21771 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.014118 21498 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:42.016793 21778 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:42.022090 21742 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:42.025679 21547 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:42.052386 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:42.002172 21160 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.028144 21429 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:42.048295 21759 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.830s	user 0.012s	sys 0.002s
I20260709 12:15:42.048920 21777 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:42.051115 21601 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.061978 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:42.078835 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91848.7us
I20260709 12:15:42.058053 21791 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:15:42.058207 21781 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.219s	user 0.000s	sys 0.002s
I20260709 12:15:42.085433 21160 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.085683 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:42.057766 21798 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:42.086501 21541 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.087142 21788 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:42.088107 21764 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.522s	user 0.005s	sys 0.000s
I20260709 12:15:42.088959 21770 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:42.056617 21547 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:42.062566 21429 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:42.058358 21785 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.206s
I20260709 12:15:42.093905 21547 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:42.089543 21770 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:42.059152 21707 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.320s	user 0.009s	sys 0.008s
I20260709 12:15:42.098987 21770 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.099161 21707 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.102105 13253 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:42.076552 21778 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:42.045614 21021 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.105536 21707 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:42.111896 21021 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:42.114357 21771 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.114903 21777 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.047144 21528 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.058498 21789 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:42.121726 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59809.4us
I20260709 12:15:42.127334 12564 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:42.110867 21778 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:42.130584 21778 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:42.130625 21777 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.12, Last appended by leader: 12, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.142338 21429 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.057770 21796 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:15:42.114414 21744 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.632s	user 0.012s	sys 0.001s
I20260709 12:15:42.158057 12641 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:42.128437 21788 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:42.161454 12564 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:42.165946 12564 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:42.172031 12536 connection.cc:256] RPC call timeout handler was delayed by -1.48440040100013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
W20260709 12:15:42.173002 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4619004010000936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
W20260709 12:15:42.173579 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4484004010000717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 12:15:42.174036 12536 connection.cc:256] RPC call timeout handler was delayed by -1.44120040100006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20260709 12:15:42.174576 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3602004010001565s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
W20260709 12:15:42.175164 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3440004010001303s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 12:15:42.175611 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3134004010000808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260709 12:15:42.176002 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0884004009999444s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
W20260709 12:15:42.176360 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0578004010001223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
I20260709 12:15:42.176932 21528 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:42.177359 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.139008 21547 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.158840 21744 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:42.177352 12564 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 27.10, Last appended by leader: 10, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.163959 21809 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:42.244149 12564 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:42.160210 21778 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.247789 13258 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:15:42.151595 21789 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:42.255551 21806 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:15:42.257272 21807 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
W20260709 12:15:42.258509 21816 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:15:42.261762 21429 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:42.263813 21730 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.600s	user 0.012s	sys 0.001s
I20260709 12:15:42.264228 21789 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.177506 21601 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.246464 21809 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.264339 21730 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.270241 21770 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:42.288681 12564 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 10. Preceding OpId from leader: term: 28 index: 10. (term mismatch)
I20260709 12:15:42.289647 12564 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 9
I20260709 12:15:42.290035 12564 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 27.10
I20260709 12:15:42.290450 12564 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:42.290786 21655 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:42.292316 21547 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.293543 21809 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.294139 21778 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.295087 12563 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:42.297297 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:42.272686 21817 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:42.275300 21429 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:42.285058 21770 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.310151 21770 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.311522 21772 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.311928 21496 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.317660 21768 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:42.318243 21768 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.318995 21783 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:42.319766 21783 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.319437 21768 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.319938 21429 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.321020 21768 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:42.322433 21429 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.323086 21578 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:42.323683 21578 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.325770 21736 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:42.326212 21736 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:42.327190 21736 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.327728 21233 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:42.320829 21783 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.336755 21783 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:42.331920 21736 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:42.341310 21730 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:42.342917 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.343160 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:42.341382 21632 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:42.344033 13254 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
W20260709 12:15:42.347954 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.350066 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.326297 21578 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.350526 21769 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:42.350942 12492 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:42.350651 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.351082 21769 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.352120 21769 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.356170 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1233) took 1028 ms. Trace:
I20260709 12:15:42.354570 21772 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.336581 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:42.365744 21748 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:42.366236 21748 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:42.367231 21748 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.372645 21748 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:42.318179 21496 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.376910 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.378890 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:42.380668 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:42.381709 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:42.383019 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.384514 12495 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 24: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:42.356611 12496 rpcz_store.cc:276] 0709 12:15:41.327710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.327937 (+   227us) service_pool.cc:224] Handling call
0709 12:15:41.328381 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:41.582985 (+254604us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:41.583023 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:41.612914 (+ 29891us) log.cc:844] Serialized 26 byte log entry
0709 12:15:41.807749 (+194835us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:41.808018 (+   269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:42.355611 (+547593us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:42.355708 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:42.355777 (+    69us) raft_consensus.cc:1685] finished
0709 12:15:42.355794 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:42.356102 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28097,"raft.queue_time_us":93208,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":18010,"spinlock_wait_cycles":306127616,"thread_start_us":283642,"threads_started":2,"wal-append.queue_time_us":313206}
I20260709 12:15:42.385294 12491 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:15:42.353351 21769 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:42.385462 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:42.383138 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 466014us
I20260709 12:15:42.384546 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:42.390985 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:42.391306 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 3936) took 1300 ms. Trace:
I20260709 12:15:42.400380 12497 rpcz_store.cc:276] 0709 12:15:41.075763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.075885 (+   122us) service_pool.cc:224] Handling call
0709 12:15:42.376389 (+1300504us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:42.376544 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1674803584}
I20260709 12:15:42.391328 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.402158 12491 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:42.391934 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:42.391934 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.394528 13254 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:42.400779 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:42.390992 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:42.405494 12490 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:42.406276 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.403290 12492 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:42.407343 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.410590 12497 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:42.404173 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.411234 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:42.407908 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.414640 12492 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:42.416556 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:42.421746 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.425452 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.426390 12492 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:42.430584 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.431591 12643 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:42.434446 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.435148 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.436408 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.437004 12639 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:42.437942 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.439843 12637 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
W20260709 12:15:42.467517 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.391129 21786 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.476493 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.399533 21778 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.477989 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.478223 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.476897 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.477342 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.478787 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:42.479377 12645 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:42.404636 21733 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:15:42.477470 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.413307 21789 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.481200 12642 raft_consensus.cc:2368] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:42.481599 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.421096 21785 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:42.424108 21788 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:42.471792 21528 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.494508 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.495388 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.486644 21785 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.498410 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.499235 12497 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:42.502575 12493 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:15:42.504588 21643 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.413764 21665 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:15:42.507277 21730 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:15:42.508255 21730 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:15:42.508600 21730 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:42.508934 21740 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:15:42.509461 21730 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.514278 21730 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:15:42.510468 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 333261us
I20260709 12:15:42.518395 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:42.519450 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.520159 12643 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:42.404847 21021 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:15:42.521931 21021 consensus_queue.cc:1243] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:15:42.497390 21785 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.522912 21785 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:42.522985 21533 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:42.523924 21533 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.525068 21533 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:42.511396 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94985.5us
I20260709 12:15:42.413930 21632 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 12:15:42.531054 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3624) took 1062 ms. Trace:
I20260709 12:15:42.533998 12641 rpcz_store.cc:276] 0709 12:15:41.468490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.468640 (+   150us) service_pool.cc:224] Handling call
0709 12:15:41.481785 (+ 13145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:42.160990 (+679205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:42.530763 (+369773us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:42.530977 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1548070656}
I20260709 12:15:42.514320 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:42.538275 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.518566 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1237) took 1040 ms. Trace:
I20260709 12:15:42.521315 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:42.518395 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:42.529407 12609 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:42.480069 21790 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.486940 21799 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:42.482993 21786 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.574361 12495 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:15:42.584453 12495 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:15:42.584755 12495 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:42.567984 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1186) took 1605 ms. Trace:
I20260709 12:15:42.539726 12488 rpcz_store.cc:276] 0709 12:15:41.478133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.478274 (+   141us) service_pool.cc:224] Handling call
0709 12:15:41.478599 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:41.686211 (+207612us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:41.686243 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:41.686290 (+    47us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:41.686342 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:42.518289 (+831947us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:42.518339 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:42.518492 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476935424}
I20260709 12:15:42.570580 13262 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:15:42.574515 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.586763 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.574872 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:42.576093 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:42.571478 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:42.588105 12569 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:42.589560 12568 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:42.592171 21786 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:42.592864 21786 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.593340 21786 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.594218 21429 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:42.594887 21429 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:42.595373 21429 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:42.594460 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76139.3us
I20260709 12:15:42.606346 12536 leader_election.cc:400] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:42.606757 12536 leader_election.cc:403] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:42.585651 13253 rpcz_store.cc:276] 0709 12:15:40.962855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:40.969967 (+  7112us) service_pool.cc:224] Handling call
0709 12:15:42.567718 (+1597751us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:42.567892 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2588723456}
I20260709 12:15:42.625368 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:42.626113 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:42.608134 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44377.8us
I20260709 12:15:42.636628 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:42.630538 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:42.640369 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.626459 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:42.644253 12643 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:42.647235 21786 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:42.647989 21786 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.648506 21786 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.651737 21811 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:42.652434 21811 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.653417 21811 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.658411 21601 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:15:42.661103 21821 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.182s	user 0.001s	sys 0.001s
I20260709 12:15:42.487756 21778 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.661507 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:42.644402 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.580663 21496 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.578001 21788 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:42.690274 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.489269 21789 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.693668 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.697176 21601 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:42.697760 21496 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.699210 21160 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:42.702080 21021 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:42.702472 13258 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 16
I20260709 12:15:42.722920 13258 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20260709 12:15:42.723295 13258 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:42.707116 12643 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:42.532069 21528 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.730392 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.731328 12490 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:42.673760 21799 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:42.717242 21788 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:42.706023 21823 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:42.582520 21740 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.118s
I20260709 12:15:42.582155 21790 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.713285 21601 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:42.736960 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:42.743849 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137711us
W20260709 12:15:42.744657 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.729406 21733 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:42.748713 21786 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:15:42.754772 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.778906 12642 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:42.755277 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:42.758985 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:42.753189 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81798.1us
I20260709 12:15:42.764106 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.781459 12646 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:42.765918 12490 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:42.687916 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:42.780769 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:42.784716 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:42.762207 21590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.020s	sys 0.050s
I20260709 12:15:42.791200 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:42.800423 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.801664 12610 leader_election.cc:400] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24.
I20260709 12:15:42.802210 12610 leader_election.cc:403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:15:42.803663 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.804890 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:42.780455 21601 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:42.814637 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44689us
W20260709 12:15:42.798713 21590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.020s	sys 0.050s
I20260709 12:15:42.794734 21545 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:42.799592 21772 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:42.843161 12494 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:15:42.835155 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.840030 12646 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:42.806090 21796 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:42.857031 12646 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.10-27.10]
I20260709 12:15:42.865849 21796 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:42.866318 21796 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:42.867206 12646 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:15:42.857954 21788 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.830389 21160 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:42.869952 21160 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:42.870325 21160 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:42.870466 12536 leader_election.cc:400] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:42.871011 12536 leader_election.cc:403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:42.854286 21429 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:42.876354 12646 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 9. Preceding OpId from leader: term: 28 index: 10. (index mismatch)
I20260709 12:15:42.806488 21498 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:42.860736 21469 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.861338 21545 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:42.859289 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 287888us
W20260709 12:15:42.858052 21655 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.005s
I20260709 12:15:42.855062 12489 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:42.879323 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.877161 21665 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:42.881824 21823 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:15:42.818531 21744 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.070s
I20260709 12:15:42.807947 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:42.812150 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:42.812973 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:42.906904 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.807942 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:42.909385 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:42.916169 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108327us
I20260709 12:15:42.891412 21469 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.905805 21799 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.924059 21469 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:42.928067 21736 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.588s	user 0.012s	sys 0.000s
I20260709 12:15:42.893596 21643 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.921461 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:42.892663 21665 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:42.936295 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:42.813257 21233 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24.
I20260709 12:15:42.945737 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:42.930202 21496 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.930527 21748 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:42.954263 21778 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:42.954087 21233 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:42.937924 21665 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:42.960532 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:42.941473 21823 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:42.964614 12642 raft_consensus.cc:2376] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26. Re-sending same reply.
I20260709 12:15:42.959825 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:42.987141 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.948716 21429 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:42.942233 21827 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:15:42.940460 21643 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.007416 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:42.918850 21733 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:42.962831 21778 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:42.949223 21828 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:15:43.036202 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90573.1us
I20260709 12:15:43.037534 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.008314 21789 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.017238 21528 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:43.049670 21752 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:43.087996 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.089785 12643 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:43.056910 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1100) took 1717 ms. Trace:
I20260709 12:15:43.019771 21831 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:43.097090 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:43.098268 13257 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 19.
W20260709 12:15:43.068491 21785 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.000s
W20260709 12:15:43.063469 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:42.955062 21545 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.077361 12568 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:43.112521 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.081317 12642 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:43.113911 12569 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:15:43.083961 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3802) took 1250 ms. Trace:
I20260709 12:15:43.115044 12563 rpcz_store.cc:276] 0709 12:15:41.833545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.833918 (+   373us) service_pool.cc:224] Handling call
0709 12:15:41.854330 (+ 20412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:42.296287 (+441957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:43.083467 (+787180us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:43.083865 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1013089024}
W20260709 12:15:43.060101 21825 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:15:43.022614 21160 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.090936 13262 rpcz_store.cc:276] 0709 12:15:41.339767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.346370 (+  6603us) service_pool.cc:224] Handling call
0709 12:15:42.570294 (+1223924us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:42.587095 (+ 16801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:43.056638 (+469543us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54005875d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:43.056806 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2813739904}
W20260709 12:15:43.062076 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3632) took 1358 ms (client timeout 1509 ms). Trace:
W20260709 12:15:43.123250 12640 rpcz_store.cc:269] 0709 12:15:41.703685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:41.704140 (+   455us) service_pool.cc:224] Handling call
0709 12:15:43.061842 (+1357702us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54002a44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:43.061973 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993048448}
W20260709 12:15:43.110700 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.071547 21466 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:43.086923 21752 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:43.118328 13256 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:43.118422 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:43.131814 13256 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:43.093533 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:43.132663 12572 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:43.086169 21790 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:43.137466 12568 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:15:43.097131 21829 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:43.039392 21778 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.109596 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:43.143772 12571 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:15:43.108656 21748 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:43.166436 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:43.171407 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.187438 12494 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:43.131495 21466 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:43.119829 21831 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:43.132828 13256 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:43.198356 21842 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:15:43.199669 21834 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:15:43.199308 21831 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:43.204226 21768 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.883s	user 0.014s	sys 0.000s
I20260709 12:15:43.205310 21466 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 29.11, Last appended by leader: 11, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:43.189898 21825 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:15:43.206848 21466 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:43.207155 21778 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.152964 21829 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.209322 21829 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:43.205303 12496 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 27 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
W20260709 12:15:43.206264 21769 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.793s	user 0.010s	sys 0.001s
I20260709 12:15:43.198460 21160 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:43.222234 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:43.212673 21796 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:43.222827 21806 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.069s	user 0.000s	sys 0.008s
I20260709 12:15:43.226195 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.236406 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:43.226907 21836 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
W20260709 12:15:43.249694 21841 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:15:43.233785 21429 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:43.250555 21429 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.252955 21429 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.247064 12494 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:43.258441 21838 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:15:43.259232 21665 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.244818 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:43.213521 21752 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.262848 21752 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:43.231616 21021 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:43.266597 12640 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:43.267925 21796 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:43.222016 21730 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.707s	user 0.012s	sys 0.000s
I20260709 12:15:43.269472 21847 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:43.275785 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:43.150437 12564 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:15:43.284927 12567 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 19.
I20260709 12:15:43.286439 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:43.248275 21160 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.287344 21160 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.292440 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:43.268960 21021 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:43.298314 12569 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:15:43.302632 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.292382 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:43.306427 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.309206 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.303519 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47525.5us
I20260709 12:15:43.316114 12572 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:15:43.315026 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:43.317603 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:43.319044 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.319463 12572 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:43.319768 12569 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:15:43.319628 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:43.326000 12491 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:15:43.317891 12494 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:43.295261 21730 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.336728 21730 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.338114 21655 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:43.339004 21796 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:43.340049 21796 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:43.340456 21796 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:43.343233 21665 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:43.296558 21748 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:43.344336 21748 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:43.344837 21748 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:43.301483 21429 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:43.346089 21429 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.332696 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:43.349547 13258 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:43.352519 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:43.353494 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:43.335657 21826 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.394s	user 0.004s	sys 0.000s
I20260709 12:15:43.294983 21601 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.331914 12571 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:43.362844 12493 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:43.371186 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:43.371966 12637 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 16. (term mismatch)
I20260709 12:15:43.372516 12637 pending_rounds.cc:85] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 15
I20260709 12:15:43.372774 12637 pending_rounds.cc:107] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:15:43.373041 12637 raft_consensus.cc:2889] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:43.315865 21021 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:43.367797 21429 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:43.376761 12563 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:43.376837 13254 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260709 12:15:43.394743 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 515477us
W20260709 12:15:43.387444 21788 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.013s	sys 0.010s
I20260709 12:15:43.402657 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.415841 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:43.414489 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3034) took 1036 ms. Trace:
I20260709 12:15:43.418054 12489 rpcz_store.cc:276] 0709 12:15:42.377576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.378035 (+   459us) service_pool.cc:224] Handling call
0709 12:15:43.408018 (+1029983us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:43.414389 (+  6371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":719022208}
I20260709 12:15:43.426565 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.418047 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:43.429615 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.418310 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 380874us
I20260709 12:15:43.446549 21429 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:43.447060 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:43.449503 13258 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:43.462821 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:43.418466 21788 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.013s	sys 0.011s
I20260709 12:15:43.467461 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.472781 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:43.476852 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58907.8us
I20260709 12:15:43.448815 21786 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:43.483048 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134296us
I20260709 12:15:43.485543 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.485692 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.490588 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.467032 21429 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:43.493831 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:43.519302 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:43.523375 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 161318us
W20260709 12:15:43.529122 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43741.3us
I20260709 12:15:43.481909 21655 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:43.533010 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:43.498943 21469 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.496024 21831 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:43.496225 21788 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.520948 21862 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:43.529598 21466 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:43.555675 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:43.557581 21801 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:43.557821 21772 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:43.531955 21853 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
W20260709 12:15:43.558300 21852 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:15:43.529107 21771 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:43.531908 21578 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:15:43.565171 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:43.565182 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:43.563762 21655 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:43.591166 12644 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.11-29.11]
I20260709 12:15:43.590989 21655 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.592826 12644 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:15:43.577898 21466 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:43.594676 21466 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.583441 21752 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:43.574536 21601 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.571290 21730 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.599485 21772 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:43.599807 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:43.541561 21733 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:15:43.568210 21831 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:43.599555 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.602313 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:43.567461 21788 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:43.604626 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64837.1us
I20260709 12:15:43.611066 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:43.486720 21160 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:43.621102 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.667672 21831 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:43.500336 21786 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.674466 12644 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:43.678417 12644 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:43.682451 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:43.684607 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 199484us
I20260709 12:15:43.628613 21849 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.570537 21469 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.699533 21768 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:43.703490 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101314us
I20260709 12:15:43.695664 12644 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.707449 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.711158 21733 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:43.715813 12609 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:43.703063 21768 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.098340 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:43.700052 21752 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:43.722054 21752 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:43.722558 21752 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:43.724390 21858 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:15:43.725069 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3036) took 1346 ms. Trace:
I20260709 12:15:43.728389 21772 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.725538 12496 rpcz_store.cc:276] 0709 12:15:42.378164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.386185 (+  8021us) service_pool.cc:224] Handling call
0709 12:15:43.724767 (+1338582us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:43.724990 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1823820032}
I20260709 12:15:43.742910 21862 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:43.745858 21788 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.746898 21748 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.688265 21865 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:43.700503 21849 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.750624 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:43.705688 21868 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
W20260709 12:15:43.770326 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.704532 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:43.704133 21881 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.782989 21879 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:43.788985 21879 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:43.789415 21879 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:43.792860 21849 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.794003 21748 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:43.780652 21865 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:43.795230 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44711.8us
I20260709 12:15:43.795498 21429 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:43.795964 12638 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:43.796186 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.796612 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3041) took 1218 ms. Trace:
I20260709 12:15:43.782529 21469 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.796999 12488 rpcz_store.cc:276] 0709 12:15:42.577933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.586527 (+  8594us) service_pool.cc:224] Handling call
0709 12:15:43.728015 (+1141488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:43.728344 (+   329us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:15:43.728372 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:43.728418 (+    46us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:43.728457 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:43.728516 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:43.796395 (+ 67879us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54000f87d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:43.796518 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12489,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"spinlock_wait_cycles":2624460160}
W20260709 12:15:43.781349 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76963us
I20260709 12:15:43.797760 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.797606 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.759737 21753 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:43.826833 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:43.806314 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:43.745069 21021 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.810554 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:43.831264 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.831884 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.820102 13256 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:43.843012 12644 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:43.846735 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.841527 21753 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:43.847522 21829 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:43.847981 21829 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:43.848389 21753 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.850021 21528 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:15:43.850200 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.801787 21837 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.565s	user 0.002s	sys 0.000s
I20260709 12:15:43.850989 21528 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:15:43.852341 21233 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:43.850075 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:43.797756 21881 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.854655 21856 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:43.848369 21829 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:43.858382 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.859753 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:43.852872 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1265) took 1180 ms. Trace:
I20260709 12:15:43.860368 12641 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:43.862165 21528 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:43.860533 13253 rpcz_store.cc:276] 0709 12:15:42.672645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.672798 (+   153us) service_pool.cc:224] Handling call
0709 12:15:42.697616 (+ 24818us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:42.698401 (+   785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:42.698437 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:42.699068 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 12:15:42.944175 (+245107us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:42.944222 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:42.944293 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:43.801028 (+856735us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:43.852415 (+ 51387us) raft_consensus.cc:1685] finished
0709 12:15:43.852437 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:43.852804 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"spinlock_wait_cycles":35413760,"thread_start_us":244653,"threads_started":1,"wal-append.queue_time_us":471639}
I20260709 12:15:43.862655 21528 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:43.852958 21233 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:43.864977 21733 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:43.865543 21233 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.863715 21528 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:43.866972 21843 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.659s	user 0.001s	sys 0.002s
I20260709 12:15:43.867450 21233 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:43.798168 21730 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:43.868577 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3951) took 1295 ms. Trace:
I20260709 12:15:43.865680 21733 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.869684 21848 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:43.869068 12495 rpcz_store.cc:276] 0709 12:15:42.572909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.573787 (+   878us) service_pool.cc:224] Handling call
0709 12:15:42.574132 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:42.712261 (+138129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:42.712290 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:42.722282 (+  9992us) log.cc:844] Serialized 52 byte log entry
0709 12:15:43.094894 (+372612us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:43.094952 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:43.095035 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:43.866169 (+771134us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:43.868216 (+  2047us) raft_consensus.cc:1685] finished
0709 12:15:43.868239 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:43.868504 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4613,"raft.queue_time_us":5633,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":11902,"spinlock_wait_cycles":12861824,"thread_start_us":298844,"threads_started":1,"wal-append.queue_time_us":464381}
I20260709 12:15:43.871194 21736 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:43.871594 21736 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:43.872529 21736 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.870659 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3648) took 1272 ms. Trace:
I20260709 12:15:43.873571 12643 rpcz_store.cc:276] 0709 12:15:42.598780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.879867 (+281087us) service_pool.cc:224] Handling call
0709 12:15:42.895615 (+ 15748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:43.090375 (+194760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:43.870398 (+780023us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:43.870593 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":445926656}
I20260709 12:15:43.809219 21768 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:43.875375 21839 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.751s	user 0.004s	sys 0.000s
I20260709 12:15:43.877045 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3950) took 1304 ms. Trace:
I20260709 12:15:43.878562 21528 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:43.878772 12497 rpcz_store.cc:276] 0709 12:15:42.572566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.572722 (+   156us) service_pool.cc:224] Handling call
0709 12:15:42.573118 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:42.611672 (+ 38554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:42.611725 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:42.639779 (+ 28054us) log.cc:844] Serialized 52 byte log entry
0709 12:15:42.949374 (+309595us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:42.949423 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:42.949498 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:43.856248 (+906750us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:43.876732 (+ 20484us) raft_consensus.cc:1685] finished
0709 12:15:43.876753 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:43.876980 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25943,"raft.queue_time_us":26943,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":14442,"spinlock_wait_cycles":12866944,"thread_start_us":236877,"threads_started":1,"wal-append.queue_time_us":413546}
I20260709 12:15:43.874504 13254 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:43.882104 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:43.883030 21733 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:43.883944 21731 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:15:43.883772 21848 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:15:43.884868 21753 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:43.885100 21786 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:43.883795 21768 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 12:15:43.886126 21796 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:43.887322 21876 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:43.887571 21796 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:43.887815 21876 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:43.888566 21829 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.888872 21876 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:43.874363 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.874480 21643 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:43.890869 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.862404 21881 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:43.892526 13256 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
W20260709 12:15:43.908430 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.904929 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 341795us
I20260709 12:15:43.911821 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.913498 21881 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:43.928001 12642 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:43.928310 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130773us
I20260709 12:15:43.934679 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:43.912297 21643 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:15:43.934968 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.911363 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:43.928124 21736 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:43.956430 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:43.957302 12566 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:15:43.932442 21876 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:43.942312 21476 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:15:43.939674 21466 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:43.948207 21831 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:43.949476 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.953761 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:43.966006 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:43.966251 12565 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 21: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:43.928686 21730 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:43.971485 21768 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:43.956631 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.976121 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:43.977285 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.978482 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.978785 21829 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:43.979565 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.958679 21823 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:43.980175 12572 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:43.965619 21466 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:43.980717 21466 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:43.958284 21740 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:43.861011 21879 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:43.962627 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:43.964345 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.939172 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:43.939010 21643 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 12:15:43.967175 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:43.967011 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.969367 21795 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.005s
I20260709 12:15:44.001940 12640 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
W20260709 12:15:44.003361 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68814.8us
I20260709 12:15:44.024044 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:44.024183 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.025053 12571 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:15:44.003230 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:44.026001 21730 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:44.026819 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.027735 12569 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:44.025436 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:44.028514 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.029122 12569 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:44.007302 12564 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 12. Preceding OpId from leader: term: 28 index: 13. (index mismatch)
I20260709 12:15:44.029532 12536 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:44.029872 21429 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:44.030416 12536 leader_election.cc:403] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:44.010640 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.030787 21429 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:44.031319 12495 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:44.031476 21429 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:44.031952 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:44.011390 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:44.032846 21865 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:44.033854 21865 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:44.008244 12570 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:15:44.035517 12644 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:15:43.984609 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.036044 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.981333 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:44.036684 12565 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:44.036823 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:44.037429 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.020197 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.039333 12488 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:44.039503 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 23 index: 9 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.020622 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.040654 12489 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:44.040721 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.021185 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.042313 12566 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 30 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:15:44.042346 12492 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:44.019989 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80919.5us
I20260709 12:15:44.043677 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:44.020984 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.009689 12497 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:44.022126 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.018649 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.026859 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:44.045907 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:44.046070 12496 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:44.047043 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:44.002409 13253 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:44.018054 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:44.021464 12567 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:15:44.048043 13254 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:44.049008 12567 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:44.054088 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.054764 12488 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:44.054802 12497 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:44.056682 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:43.996784 12563 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:44.048902 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:44.064498 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.065544 12571 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:44.033272 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:44.055541 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:44.085613 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.083113 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.086606 12640 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:44.087329 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
W20260709 12:15:44.088750 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55563.5us
I20260709 12:15:44.087798 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.089697 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:44.094401 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.096230 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:44.102469 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58834.1us
I20260709 12:15:44.103572 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.104339 12496 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:15:44.108589 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43668.9us
I20260709 12:15:44.115770 12489 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:44.107807 12566 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:44.124130 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.124809 12566 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:44.126211 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.128559 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.109097 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.130817 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:44.020910 21796 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:44.012858 21643 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:44.140698 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.105720 21847 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.083629 21783 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:15:44.147581 21590 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:44.211565 21590 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:44.212067 21590 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:44.218037 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.218858 12572 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:15:44.224089 12496 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:44.224850 12571 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:15:44.226073 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.228231 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.225263 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.151307 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:44.152464 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84851.8us
I20260709 12:15:44.153105 21160 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:44.155357 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:44.155571 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:44.230978 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75697.3us
I20260709 12:15:44.019430 21466 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.158390 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:44.248819 12571 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:44.249430 12568 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:15:44.162092 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.109606 21528 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:43.999209 21831 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:44.004161 21879 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:44.105515 21772 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260709 12:15:44.166354 12497 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:44.266809 12497 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:15:44.168418 21788 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:44.104643 21881 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:15:44.133849 21857 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:44.110113 21233 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:44.169010 21533 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260709 12:15:44.171216 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.103797 21752 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.175956 21790 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:44.179978 21748 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:44.180416 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.185704 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:44.186103 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 128350us
I20260709 12:15:44.068091 21736 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:44.149958 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.229306 12565 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 26 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:15:44.231259 12569 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:44.252673 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:44.134250 21862 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:44.254925 12571 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:15:44.261399 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
W20260709 12:15:44.369041 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116450us
I20260709 12:15:44.199918 21823 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.201475 21796 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:44.346326 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:44.292699 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:44.299111 12497 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:15:44.259150 21786 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:15:44.359464 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.360759 12572 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:44.400105 12572 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:15:44.401829 12572 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:15:44.397817 21823 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.362690 12492 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:15:44.364799 12564 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:44.301113 21233 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:44.366933 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 226340us
I20260709 12:15:44.404557 21823 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:44.405393 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:44.398121 21796 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:44.404639 21233 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:44.408437 21872 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:44.408994 21872 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:44.409373 21872 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:44.414103 12491 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:44.417857 21768 raft_consensus.cc:2764] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:15:44.419468 21233 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:44.422187 21233 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:44.387837 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 202274us
I20260709 12:15:44.262238 21601 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.269713 21881 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:44.382766 12563 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:44.430027 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.433566 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:44.290082 21857 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:44.437888 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.252616 21643 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:44.368050 21736 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:44.391284 21744 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:44.420840 21796 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.455062 12640 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:15:44.399466 21831 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:44.440104 21786 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:44.479954 21021 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:15:44.480198 21872 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:44.483719 12644 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 11. Preceding OpId from leader: term: 28 index: 13. (index mismatch)
I20260709 12:15:44.485503 12643 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:15:44.486513 12642 pending_rounds.cc:85] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 15
I20260709 12:15:44.549160 12642 pending_rounds.cc:107] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:15:44.549590 12642 raft_consensus.cc:2889] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:44.490388 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60494.7us
W20260709 12:15:44.492961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55185.8us
I20260709 12:15:44.364950 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:44.495879 12645 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:15:44.250274 21847 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:44.524554 12637 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:44.383700 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:44.557830 12488 raft_consensus.cc:606] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:44.558292 12488 raft_consensus.cc:614] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:44.538331 12641 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 13
I20260709 12:15:44.558897 12641 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260709 12:15:44.559216 12641 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:44.559762 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:44.492772 21881 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:44.554105 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.499027 21857 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:44.547791 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 364048us
I20260709 12:15:44.563529 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:44.497258 21730 raft_consensus.cc:2764] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:15:44.569588 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.571776 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.577666 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.578536 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.575434 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:44.580363 13256 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:44.528514 21744 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:44.579236 12638 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:44.556290 21233 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:44.595291 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.601809 13258 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 21. (index mismatch)
I20260709 12:15:44.563365 21799 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.493115 21601 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.606060 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.608767 21847 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:44.601346 21160 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.611320 21160 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.614871 21796 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:44.594204 21744 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:44.621742 21021 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:44.622447 21021 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:44.623260 21872 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:44.623569 21021 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.627352 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1218) took 1676 ms. Trace:
I20260709 12:15:44.627513 21829 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:15:44.628311 12646 rpcz_store.cc:276] 0709 12:15:42.950797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:42.998716 (+ 47919us) service_pool.cc:224] Handling call
0709 12:15:42.999093 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:43.022432 (+ 23339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:43.022486 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:43.036642 (+ 14156us) log.cc:844] Serialized 26 byte log entry
0709 12:15:43.457254 (+420612us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:43.494864 (+ 37610us) log.cc:844] Serialized 14 byte log entry
0709 12:15:43.518202 (+ 23338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:43.518762 (+   560us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:44.625604 (+1106842us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b24003e3618. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:15:44.626876 (+  1272us) raft_consensus.cc:1685] finished
0709 12:15:44.627037 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:44.627261 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67070,"prepare.queue_time_us":39199,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":49070,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":16189,"spinlock_wait_cycles":26333440,"thread_start_us":343751,"threads_started":1,"wal-append.queue_time_us":489779}
I20260709 12:15:44.632592 21733 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:44.634397 21893 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:44.636873 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.606274 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.644207 21466 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:44.633965 21799 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.633332 21733 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:44.657792 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94912.7us
I20260709 12:15:44.673085 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.675081 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:44.660529 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 395556us
I20260709 12:15:44.669492 21856 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.669665 21799 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:44.681918 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:44.682742 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:15:44.682796 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.585975 21476 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.098s
I20260709 12:15:44.666090 21632 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:44.678092 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:44.689529 12495 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 22: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:44.691699 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.692799 12536 leader_election.cc:400] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:44.693354 12536 leader_election.cc:403] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:15:44.698920 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.689108 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:44.710901 21848 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:44.711531 21848 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:44.714393 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.686370 21893 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:44.684794 21823 raft_consensus.cc:2764] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:15:44.685155 21730 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:44.723701 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:44.724071 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:44.727147 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:15:44.732342 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.700052 21752 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:44.737950 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.742395 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.740664 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:44.712873 21848 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:44.721580 21893 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:44.719276 21160 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:44.742862 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:44.689049 21856 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.748776 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.751315 13258 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:15:44.734644 21896 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:44.791612 12646 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:15:44.793136 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:44.794502 12645 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 20. (term mismatch)
I20260709 12:15:44.845503 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:44.845901 12645 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 19
I20260709 12:15:44.846503 12645 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 27.20
I20260709 12:15:44.846789 12645 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:15:44.846009 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.022s	sys 0.022s
W20260709 12:15:44.847395 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.022s	sys 0.022s
I20260709 12:15:44.747522 21872 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:15:44.794926 21900 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.810824 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.817325 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:44.827477 21786 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:44.833226 21429 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.801s	user 0.008s	sys 0.004s
I20260709 12:15:44.745317 21823 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:15:44.851377 12646 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:15:44.847792 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 599091us
I20260709 12:15:44.750281 21730 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:44.855301 12643 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:44.747725 21893 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:44.778030 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:44.876504 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:44.885735 21528 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:15:44.893537 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:44.854048 21872 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:44.894376 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 316829us
W20260709 12:15:44.914729 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 136815us
I20260709 12:15:44.915431 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.779503 21752 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.100s
I20260709 12:15:44.864454 21601 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:44.897660 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 322313us
I20260709 12:15:44.744450 21733 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.929667 12645 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:15:44.925554 21831 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:44.935365 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58953.5us
I20260709 12:15:44.935474 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.870097 21862 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:44.785290 21160 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.105s
I20260709 12:15:44.895571 21528 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:44.966945 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.898396 21643 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:44.849500 21896 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.881217 21429 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.949306 21466 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:15:44.931550 21872 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:44.868139 21900 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:44.952064 21590 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.740s	user 0.015s	sys 0.000s
I20260709 12:15:44.976971 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:44.894459 21730 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:44.941424 21902 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:44.988590 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:44.932484 21799 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:15:44.989682 12643 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:45.001062 12569 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:44.956125 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:44.977713 21903 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:44.966418 21823 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.041s
I20260709 12:15:44.971879 21655 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:15:44.970675 21899 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
W20260709 12:15:45.024945 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48106.7us
I20260709 12:15:45.029556 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:45.027949 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71980.3us
I20260709 12:15:45.025856 21799 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.030974 21902 raft_consensus.cc:2764] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:15:45.003741 21533 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260709 12:15:45.018374 21906 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:45.001343 21590 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:45.060787 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:45.032194 21752 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:44.999402 21643 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
W20260709 12:15:45.066785 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 381632us
I20260709 12:15:45.042857 21827 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:45.042842 21466 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:45.073998 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:45.079684 21744 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.042709 21831 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.076722 21752 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.086057 21827 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:45.087908 21827 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:15:45.089049 21752 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.089648 21730 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.089767 21752 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:45.090273 21730 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.090498 21752 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:15:45.090792 21730 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.077247 21466 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:45.094429 21466 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:45.099874 21744 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.102531 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:45.093364 21730 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.103322 12563 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:45.101451 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:45.107693 12495 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:45.108177 21730 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:45.109833 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:45.110288 12536 leader_election.cc:403] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:45.111523 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:45.111598 21865 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:45.058748 21856 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.026302 21903 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.120067 12492 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:45.031127 21901 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:15:45.124472 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.129572 21896 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.063664 21799 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:45.135454 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.018s	sys 0.053s
W20260709 12:15:45.135980 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.018s	sys 0.053s
I20260709 12:15:45.139031 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.136169 21896 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:45.141214 12495 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:45.126966 21823 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:45.149076 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3795) took 1057 ms. Trace:
I20260709 12:15:45.172688 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:45.188778 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178809us
I20260709 12:15:45.190680 12571 rpcz_store.cc:276] 0709 12:15:44.091502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.381660 (+290158us) service_pool.cc:224] Handling call
0709 12:15:44.390909 (+  9249us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:44.400526 (+  9617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:44.400563 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:44.427711 (+ 27148us) log.cc:844] Serialized 26 byte log entry
0709 12:15:44.849595 (+421884us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:45.147710 (+298115us) log.cc:844] Serialized 14 byte log entry
0709 12:15:45.148419 (+   709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:45.148565 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:45.148730 (+   165us) raft_consensus.cc:1685] finished
0709 12:15:45.148751 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:45.148985 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25392,"raft.queue_time_us":19635,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":18493,"thread_start_us":646109,"threads_started":2,"wal-append.queue_time_us":550032}
I20260709 12:15:45.208978 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:45.176819 21862 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:45.213120 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:15:45.215724 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 224428us
I20260709 12:15:45.217545 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:45.188244 21908 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:45.170131 21879 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.198794 21429 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.223454 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.234395 12637 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:45.222134 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:45.236232 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:45.199101 21856 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:45.237334 21801 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:45.238209 21801 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.241353 21915 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:45.244016 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.245213 12495 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:45.211285 21823 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:45.247310 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:45.274734 12495 raft_consensus.cc:606] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:45.275244 12495 raft_consensus.cc:614] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:45.250705 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:15:45.245977 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 210039us
I20260709 12:15:45.272089 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.242115 21915 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:45.222244 21900 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.197176 21466 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.219381 21913 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:45.287865 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.288527 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.289113 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.285238 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:45.291190 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.291616 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.201133 21906 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:15:45.296895 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.294123 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.289973 12493 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:15:45.301060 12494 pending_rounds.cc:85] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:15:45.302721 12494 pending_rounds.cc:107] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 26.20
I20260709 12:15:45.302942 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.303062 12494 raft_consensus.cc:2889] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:45.297748 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:45.304666 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 273159us
I20260709 12:15:45.306437 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.306532 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:45.233937 21655 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:15:45.244477 21879 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.278208 21914 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:45.306653 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.319414 21913 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:45.293031 21429 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:45.321038 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.323235 12495 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:45.297581 21909 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.328605 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.272727 21856 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.345003 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.345197 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.334226 12496 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:15:45.297886 21865 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:45.331270 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:45.349871 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.293206 21643 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:15:45.297122 21862 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:15:45.326257 21903 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.358774 12572 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:15:45.340929 21823 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:45.348338 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.350090 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.360394 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:45.371658 12489 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:45.371913 12496 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:45.372265 12496 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:45.362187 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:45.232497 21908 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.351521 21896 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:45.329174 21429 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:45.340794 21632 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:45.359731 21910 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.362883 21921 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:45.348495 21909 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.389168 21923 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:15:45.374598 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.311877 21796 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:15:45.344301 21533 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:15:45.351418 21831 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.351683 21730 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:45.370054 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:45.390825 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.392547 21910 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.396785 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:45.421567 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.422581 12572 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:45.308190 21900 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:45.347277 21801 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.399111 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:45.395381 21909 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.407835 21908 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.408568 12644 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:45.434419 12640 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:45.398833 21799 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:45.410580 21857 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.808s	user 0.014s	sys 0.000s
I20260709 12:15:45.390219 21915 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.418885 12568 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:45.379937 21893 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.390460 21429 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:45.398592 21913 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:45.425104 21896 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:45.453817 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.454665 12644 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:45.400080 21865 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:15:45.432777 21900 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:45.468580 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:45.468983 21900 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.472901 21856 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.475055 21736 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.477273 21790 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:15:45.477842 21925 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:15:45.466773 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:45.477766 21790 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:45.467459 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:45.480274 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.481398 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.442800 21909 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:45.482728 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:15:45.483956 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:45.430591 21823 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.029s
I20260709 12:15:45.427565 21733 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:15:45.514653 21466 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.515803 21790 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.517410 21910 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.432955 21924 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260709 12:15:45.518784 21829 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:45.519325 21829 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:45.520226 21829 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.522620 21823 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:45.523239 21823 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.420120 21831 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.487866 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:45.492183 21912 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
I20260709 12:15:45.448570 21857 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.498114 21909 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.488332 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:45.546725 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.547039 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:45.547466 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59220us
I20260709 12:15:45.544414 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:45.512435 21919 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:15:45.514144 12637 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:45.533298 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.550905 12496 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:45.545678 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:45.551895 13256 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:45.547981 13257 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:45.513754 21879 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:45.527149 21881 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.944s	user 0.015s	sys 0.000s
I20260709 12:15:45.559970 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:45.567667 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:45.568588 12568 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:45.506376 21848 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.521536 21829 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:45.566781 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:45.572813 12570 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:15:45.572882 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:45.525601 21823 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.570266 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:45.543682 21847 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:45.547329 21902 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:45.547923 21865 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:45.586335 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.559147 21862 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:45.589493 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.564384 21879 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.593793 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3677) took 1143 ms. Trace:
I20260709 12:15:45.594367 12642 rpcz_store.cc:276] 0709 12:15:44.450773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.450945 (+   172us) service_pool.cc:224] Handling call
0709 12:15:44.486282 (+ 35337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:44.550464 (+ 64182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:44.550520 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:44.564917 (+ 14397us) log.cc:844] Serialized 26 byte log entry
0709 12:15:45.136398 (+571481us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:45.136447 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:45.136532 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:45.593336 (+456804us) raft_consensus.cc:1685] finished
0709 12:15:45.593367 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:45.593593 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":537810,"threads_started":1,"wal-append.queue_time_us":670903}
I20260709 12:15:45.596506 21910 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:15:45.579900 21466 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:45.597915 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.602437 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:45.555647 21796 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:15:45.581367 21748 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:45.586005 21902 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:45.589766 21643 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:15:45.622862 21790 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:45.623159 21429 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.592092 21931 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:45.624656 21931 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:45.624900 21932 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.109s	user 0.002s	sys 0.000s
I20260709 12:15:45.625124 21931 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:45.591106 21862 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:45.625917 21862 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:15:45.627300 21935 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 12:15:45.605103 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.606982 12496 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:45.613215 21748 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:45.631168 21748 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:15:45.631505 21938 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
W20260709 12:15:45.634256 21872 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.635s	user 0.014s	sys 0.000s
I20260709 12:15:45.634819 21872 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:45.635823 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:45.637173 21872 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.569096 21881 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.610886 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:45.600346 21879 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:45.615242 21934 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.146s	user 0.003s	sys 0.000s
I20260709 12:15:45.616528 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3684) took 1162 ms. Trace:
I20260709 12:15:45.616801 21857 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.617426 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.618454 21844 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.681303 21844 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.683831 21844 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.684722 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:45.687582 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.688838 12493 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 12:15:45.619176 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.581812 21848 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.691018 21906 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:45.691617 21906 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:45.692055 21906 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:45.692245 21848 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.590758 21865 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.696141 21865 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.621378 21831 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.585685 21847 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.699651 21847 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.639186 21909 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:45.639721 21466 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:45.701603 21466 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.597692 21915 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:45.706025 21915 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.708988 21915 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:45.642371 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:45.663218 12571 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:45.710933 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.712411 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.679075 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:45.713970 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:45.670452 12641 rpcz_store.cc:276] 0709 12:15:44.488063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.488278 (+   215us) service_pool.cc:224] Handling call
0709 12:15:44.537992 (+ 49714us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:44.813020 (+275028us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:44.859271 (+ 46251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:44.859321 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:44.860160 (+   839us) log.cc:844] Serialized 52 byte log entry
0709 12:15:45.189106 (+328946us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:45.323987 (+134881us) log.cc:844] Serialized 14 byte log entry
0709 12:15:45.332412 (+  8425us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:45.332520 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:45.616176 (+283656us) raft_consensus.cc:1685] finished
0709 12:15:45.616274 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:45.616455 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46325,"prepare.queue_time_us":210610,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":4808,"raft.queue_time_us":31900,"raft.run_cpu_time_us":2828,"raft.run_wall_time_us":238343,"spinlock_wait_cycles":10092288,"thread_start_us":668125,"threads_started":3,"wal-append.queue_time_us":434107}
I20260709 12:15:45.723430 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.725147 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.727739 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:45.729749 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.736145 12571 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 29 because replica has last-logged OpId of term: 26 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 11.
W20260709 12:15:45.733773 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114683us
I20260709 12:15:45.747044 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:45.763713 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:45.770596 13253 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:15:45.762595 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.776814 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:45.641032 21902 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:45.790875 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147352us
I20260709 12:15:45.792538 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.794416 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.779611 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:45.781068 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:45.664651 21801 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:45.783217 12489 raft_consensus.cc:606] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:45.802680 12497 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:15:45.786326 12571 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:45.799244 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.807283 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.808542 12571 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 28 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:15:45.797137 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:45.815035 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:45.816681 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:45.817883 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:45.822109 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:45.824246 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
W20260709 12:15:45.796002 21904 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.823s	user 0.000s	sys 0.003s
I20260709 12:15:45.777770 21643 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:45.784752 21930 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:15:45.822314 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:45.832916 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.839681 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:45.840785 12571 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:45.826268 21909 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:45.829250 21233 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:45.827301 21847 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:45.820971 21733 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:45.852906 21898 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.959s	user 0.001s	sys 0.000s
I20260709 12:15:45.855870 21946 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:15:45.858758 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:45.861135 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68951.6us
I20260709 12:15:45.861605 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:45.819407 21429 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:45.838383 21021 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:45.815750 21801 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:45.904904 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3794) took 1845 ms. Trace:
I20260709 12:15:45.867336 21909 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:45.905608 12566 rpcz_store.cc:276] 0709 12:15:44.059096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.376967 (+317871us) service_pool.cc:224] Handling call
0709 12:15:44.377253 (+   286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:44.413532 (+ 36279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:44.413581 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:44.414278 (+   697us) log.cc:844] Serialized 26 byte log entry
0709 12:15:44.745218 (+330940us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:45.007718 (+262500us) log.cc:844] Serialized 14 byte log entry
0709 12:15:45.008107 (+   389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:45.476774 (+468667us) spinlock_profiling.cc:243] Waited 32.4 ms on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:45.476835 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:45.605321 (+128486us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:45.904624 (+299303us) raft_consensus.cc:1685] finished
0709 12:15:45.904650 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:45.904810 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3197,"prepare.queue_time_us":418134,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":5152,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":10654,"spinlock_wait_cycles":83412096,"thread_start_us":471795,"threads_started":2,"wal-append.queue_time_us":502128}
I20260709 12:15:45.915562 12571 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260709 12:15:45.916347 12571 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 13
I20260709 12:15:45.916716 12571 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:15:45.867647 21847 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:45.916981 12571 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:45.917182 21909 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:45.926470 21847 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:15:45.972404 21949 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:15:45.909391 21021 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:46.088167 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 183561us
I20260709 12:15:45.970554 21643 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:45.900213 21857 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:46.131804 12495 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:15:46.142519 12495 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
W20260709 12:15:46.112872 21829 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.535s	user 0.015s	sys 0.004s
W20260709 12:15:46.110256 21948 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.194s	user 0.004s	sys 0.008s
W20260709 12:15:46.115989 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67652.1us
W20260709 12:15:46.155252 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.031s (SENT) [suppressed 11 similar messages]
W20260709 12:15:46.116734 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:46.119094 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:46.123518 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:46.125200 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:46.128340 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 226826us
W20260709 12:15:46.135013 21934 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.220s	user 0.000s	sys 0.009s
I20260709 12:15:46.115396 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1615) took 2156 ms. Trace:
I20260709 12:15:46.124527 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:46.105585 21879 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.127080 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:46.106401 21922 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:46.165488 12638 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:15:46.160516 12567 rpcz_store.cc:276] 0709 12:15:43.958666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.050228 (+ 91562us) service_pool.cc:224] Handling call
0709 12:15:44.050539 (+   311us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:44.091626 (+ 41087us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:44.091670 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:44.109524 (+ 17854us) log.cc:844] Serialized 26 byte log entry
0709 12:15:44.551585 (+442061us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:44.551633 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:44.551716 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:45.852060 (+1300344us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:46.115042 (+262982us) raft_consensus.cc:1685] finished
0709 12:15:46.115065 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:46.115321 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38120,"raft.queue_time_us":17129,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":20127,"spinlock_wait_cycles":42436352,"thread_start_us":401835,"threads_started":1,"wal-append.queue_time_us":612741}
W20260709 12:15:46.177088 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 262479us
W20260709 12:15:46.180099 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:46.113837 21944 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.527s	user 0.002s	sys 0.006s
I20260709 12:15:46.187359 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:46.116274 21801 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.204063 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:46.201704 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75027.9us
I20260709 12:15:46.121361 21881 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.135438 21847 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.126133 21021 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:46.232291 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:46.196707 21931 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.571s	user 0.015s	sys 0.005s
I20260709 12:15:46.264711 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3793) took 2297 ms. Trace:
I20260709 12:15:46.117121 21788 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.084s
W20260709 12:15:46.234529 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79322.2us
I20260709 12:15:46.274085 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:46.235541 12566 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 22. Ops: []
I20260709 12:15:46.167505 21831 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:46.180703 21922 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:46.145104 21857 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.240748 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.112908 21906 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:46.299852 21950 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:15:46.240513 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:46.211194 21748 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.573s	user 0.015s	sys 0.004s
I20260709 12:15:46.256084 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:46.202880 21957 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:46.202700 21233 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:46.182523 21909 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.266781 12569 rpcz_store.cc:276] 0709 12:15:43.967818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.379855 (+412037us) service_pool.cc:224] Handling call
0709 12:15:44.380263 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:45.001772 (+621509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:46.264462 (+1262690us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:46.264635 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1427198720}
I20260709 12:15:46.202628 21902 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.264586 21772 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:46.271765 21931 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.279125 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.282030 21429 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:15:46.287195 21862 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.661s	user 0.020s	sys 0.002s
I20260709 12:15:46.201658 21927 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 12:15:46.198673 21799 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
W20260709 12:15:46.291818 21790 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.668s	user 0.017s	sys 0.000s
I20260709 12:15:46.292650 21923 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:46.292831 21736 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:46.257025 21856 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:46.309721 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69093us
I20260709 12:15:46.391258 12644 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:46.313917 12640 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:15:46.301451 21906 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:46.330555 12566 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:46.338366 12569 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:46.327132 21772 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:46.322893 21829 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:46.294507 21914 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:46.390882 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94207.9us
I20260709 12:15:46.330127 21910 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:46.298949 21831 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:46.409440 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 107018us
I20260709 12:15:46.419951 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.395988 12642 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:15:46.311439 21748 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.436147 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.426121 12642 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 14
I20260709 12:15:46.437318 12642 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:15:46.437592 12642 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:46.424228 12569 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.15-25.15]
W20260709 12:15:46.407141 21952 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.202s	user 0.001s	sys 0.001s
I20260709 12:15:46.367327 21777 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:15:46.324362 21233 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.450084 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.393901 21923 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:46.453797 12569 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:46.423640 21920 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:15:46.345690 21862 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.321749 21957 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.449777 21777 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:46.473902 21909 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:46.471297 21955 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
W20260709 12:15:46.472069 21956 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:15:46.474072 21857 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:46.475970 12646 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:46.477253 21590 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:46.328306 21922 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.475260 21970 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:46.481863 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:46.488579 12569 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:46.503896 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:46.504278 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.494196 21233 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:46.435030 21914 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.514050 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:46.505110 12569 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:46.447276 21910 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.483851 21931 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:46.531350 21847 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:46.546614 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110542us
I20260709 12:15:46.549616 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:46.547107 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:46.555806 12569 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:15:46.540894 21922 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:15:46.557087 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 129029us
I20260709 12:15:46.559441 12569 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
W20260709 12:15:46.559338 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55508.9us
I20260709 12:15:46.561378 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:46.556007 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.503196 21772 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.511909 21923 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:46.563355 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171753us
I20260709 12:15:46.566138 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:46.587597 21968 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:15:46.574740 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.591197 21862 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.591913 21977 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:46.564522 21857 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:46.541460 21777 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.593601 21914 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.596988 21979 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:46.603523 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 64) took 1147 ms. Trace:
I20260709 12:15:46.594731 21923 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:46.664975 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:46.605070 21957 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.597383 21910 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:46.596536 21978 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:46.596486 21831 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:46.596689 21748 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:46.648604 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3867) took 1255 ms. Trace:
I20260709 12:15:46.682183 12563 rpcz_store.cc:276] 0709 12:15:45.393041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.306881 (+913840us) service_pool.cc:224] Handling call
0709 12:15:46.307370 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:46.307691 (+   321us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:46.307729 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:46.321036 (+ 13307us) log.cc:844] Serialized 26 byte log entry
0709 12:15:46.549794 (+228758us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:46.549834 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:46.549920 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:46.648348 (+ 98428us) raft_consensus.cc:1685] finished
0709 12:15:46.648371 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:46.648535 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":158500,"threads_started":1,"wal-append.queue_time_us":255490}
I20260709 12:15:46.659154 21862 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:15:46.563282 21901 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:46.659513 12489 rpcz_store.cc:276] 0709 12:15:45.455761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.455901 (+   140us) service_pool.cc:224] Handling call
0709 12:15:46.603286 (+1147385us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:46.603407 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":721847552}
I20260709 12:15:46.665789 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:46.698395 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:46.669987 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:46.689261 21982 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:46.711028 12490 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:46.565138 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3864) took 1173 ms. Trace:
I20260709 12:15:46.596061 21021 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.660902 21979 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:46.686579 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:46.618613 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57331us
I20260709 12:15:46.723096 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:46.692790 21881 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:46.695950 21901 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:46.685770 21910 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
W20260709 12:15:46.731287 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 156650us
I20260709 12:15:46.730181 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:46.739532 12489 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:46.714304 12571 rpcz_store.cc:276] 0709 12:15:45.391999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.119627 (+727628us) service_pool.cc:224] Handling call
0709 12:15:46.131380 (+ 11753us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:46.131766 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:15:46.260032 (+128266us) log.cc:844] Serialized 14 byte log entry
0709 12:15:46.562017 (+301985us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:46.562105 (+    88us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:15:46.562201 (+    96us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:46.563516 (+  1315us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:46.564906 (+  1390us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":258603,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4755,"raft.queue_time_us":67729,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":400424,"threads_started":3,"wal-append.queue_time_us":224247}
W20260709 12:15:46.731452 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44972.8us
I20260709 12:15:46.743058 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:46.743470 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.745219 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.693991 21831 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:15:46.681789 21978 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.776491 21978 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:46.769301 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3786) took 2805 ms. Trace:
I20260709 12:15:46.771663 12489 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:15:46.777937 12570 rpcz_store.cc:276] 0709 12:15:43.963492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.263664 (+300172us) service_pool.cc:224] Handling call
0709 12:15:44.264219 (+   555us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:45.574855 (+1310636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:46.768973 (+1194118us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:46.769198 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3008775936}
I20260709 12:15:46.782882 21978 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:46.786144 12570 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:15:46.787971 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.786216 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:46.792483 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:46.749467 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.728996 21979 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:46.737356 21901 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:46.756584 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:46.754861 12392 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 23, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 23 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:46.753151 21931 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260709 12:15:46.735473 21881 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.747531 21601 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:15:46.762192 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:46.898147 12566 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:46.755901 21632 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 12:15:46.737133 21989 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:46.747323 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:46.901540 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154500us
I20260709 12:15:46.902573 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:46.790120 21978 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:46.790797 21972 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:46.793813 21799 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:46.843174 21908 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:15:46.811964 21021 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:46.813973 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:15:46.790927 21911 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:46.814826 21921 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:46.817204 12492 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:15:46.820106 12490 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:46.826752 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:46.931778 12495 raft_consensus.cc:2368] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31.
I20260709 12:15:46.784008 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.830670 12571 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:46.817809 21862 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:46.839871 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:46.788290 21974 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:46.893314 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:46.894584 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:46.946669 12569 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:46.847153 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.864759 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:46.843756 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:46.903949 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:46.907016 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:46.799165 21926 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.479s	user 0.007s	sys 0.000s
I20260709 12:15:46.926065 12488 raft_consensus.cc:606] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:46.948897 13260 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:46.949249 12488 raft_consensus.cc:614] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:15:46.814297 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3808) took 1277 ms. Trace:
I20260709 12:15:46.950547 12565 rpcz_store.cc:276] 0709 12:15:45.536889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.334803 (+797914us) service_pool.cc:224] Handling call
0709 12:15:46.335496 (+   693us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:46.492561 (+157065us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:15:46.492592 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:46.495836 (+  3244us) log.cc:844] Serialized 52 byte log entry
0709 12:15:46.673775 (+177939us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:46.673809 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:46.673875 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:46.813993 (+140118us) raft_consensus.cc:1685] finished
0709 12:15:46.814023 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:46.814219 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":138850,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":255405,"threads_started":2,"wal-append.queue_time_us":192640}
I20260709 12:15:46.952986 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.907497 21989 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:46.935429 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:46.915910 21601 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:46.821242 21910 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.105s
I20260709 12:15:46.954695 12565 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:46.905983 21922 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:46.909811 21831 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:46.959477 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
W20260709 12:15:46.959408 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112418us
I20260709 12:15:46.948359 13262 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:15:46.943495 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159586us
I20260709 12:15:47.043295 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.043979 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:46.958052 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.008610 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.047062 12566 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 12:15:46.925010 21903 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:47.053411 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.941795 21982 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:46.944831 21908 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:47.048269 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:15:47.059535 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:46.957255 21970 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:46.916767 21978 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.054708 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.074656 12566 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:15:47.075668 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:47.032521 21909 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:46.945868 21921 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.091298 12566 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:47.054980 21914 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:47.091279 12565 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:15:47.094333 21590 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.095598 21921 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:46.945312 21911 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:46.957571 21862 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.049789 21922 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:46.924245 21972 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.007289 21974 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.131809 21921 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:47.172101 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3806) took 1636 ms (client timeout 2048 ms). Trace:
I20260709 12:15:47.060755 21982 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:47.174875 21921 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.180274 12570 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:47.180521 21989 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.068351 21799 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.129998 12571 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:47.149758 21922 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.152479 21914 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.153221 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:47.157254 21911 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.158987 21862 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:47.172605 12568 rpcz_store.cc:269] 0709 12:15:45.536001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.617091 (+ 81090us) service_pool.cc:224] Handling call
0709 12:15:47.171892 (+1554801us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:47.172031 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1842373888}
I20260709 12:15:47.188014 21978 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.053316 21786 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.191279 12570 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:15:47.054687 21831 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.222213 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:47.228864 12568 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:47.225653 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.197049 21909 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.138123 21590 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:47.244829 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 342101us
I20260709 12:15:47.257843 21752 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:47.266374 21924 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.267119 21924 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.274248 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1275) took 2842 ms. Trace:
I20260709 12:15:47.274871 13253 rpcz_store.cc:276] 0709 12:15:44.431697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:44.431897 (+   200us) service_pool.cc:224] Handling call
0709 12:15:44.432411 (+   514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:45.773846 (+1341435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.273820 (+1499974us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:47.274119 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3076951936}
I20260709 12:15:47.241765 21978 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.219359 21799 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.287204 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.224783 21914 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.299610 12563 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:47.279982 21903 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:47.252926 21601 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:47.235670 21786 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.218989 21528 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.285293 21533 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:47.282173 21788 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:47.278399 21847 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:15:47.306655 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.298470 21977 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.252108 21909 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.327211 12570 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:47.228751 21908 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:47.345852 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 437602us
I20260709 12:15:47.332145 21528 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:47.373980 21972 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.339617 21921 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.374387 21831 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.299256 21978 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.382647 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:47.398869 21799 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.399875 21799 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.400545 21799 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.331377 21788 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.402045 21601 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.403352 21788 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.405591 21909 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:47.408154 21979 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.509s	user 0.006s	sys 0.004s
I20260709 12:15:47.408385 22010 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.408628 21979 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.408967 22010 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.411001 22010 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.412513 21933 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:47.412915 21933 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:47.329377 21533 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.413741 21933 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.414955 21933 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:47.415455 21533 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.417337 21979 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.417590 12392 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 25, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 25 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.419701 12392 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 18 to 22, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 22 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.371618 22011 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.421762 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:15:47.422345 21974 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.422916 22013 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.422467 21881 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.423489 22013 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.427630 22013 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.429759 12392 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 26. New cstate: current_term: 26 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.432953 22013 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:47.435555 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56279.6us
I20260709 12:15:47.436465 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 3990) took 2109 ms. Trace:
I20260709 12:15:47.437745 21921 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:15:47.433362 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 231) took 51361 ms (client timeout 15000 ms). Trace:
I20260709 12:15:47.438894 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:47.439574 12492 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:47.374538 21590 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.438066 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:47.443090 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178540us
I20260709 12:15:47.447955 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:47.444466 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:47.448242 12392 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 26, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 26 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.453872 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:47.456796 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109770us
I20260709 12:15:47.436990 12497 rpcz_store.cc:276] 0709 12:15:45.327566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.327748 (+   182us) service_pool.cc:224] Handling call
0709 12:15:45.328125 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:45.803352 (+475227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.436051 (+1632699us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:47.436386 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1090261888}
I20260709 12:15:47.366595 21977 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.477577 12392 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.495999 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:47.501226 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:47.387758 21903 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.500739 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.457470 21972 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:15:47.531601 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:47.474417 22019 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:15:47.497310 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60096.8us
W20260709 12:15:47.507225 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:47.443732 22013 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.510419 12392 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:15:47.544798 21923 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.871s	user 0.013s	sys 0.000s
I20260709 12:15:47.489670 21914 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:47.577205 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:47.510967 22016 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:47.588086 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.539965 21233 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:47.568305 22002 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
I20260709 12:15:47.590121 12488 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:47.592870 12572 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:47.534308 21881 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:47.536165 21893 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.532056 21978 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.593706 12392 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.531283 22024 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:15:47.453161 22011 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.546559 22019 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:47.603854 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72348.2us
I20260709 12:15:47.574209 22010 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:47.438742 12386 rpcz_store.cc:269] 0709 12:14:56.072222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:56.159011 (+ 86789us) service_pool.cc:224] Handling call
0709 12:15:47.433224 (+51274213us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:15:41.488902 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:15:41.489652 (+   750us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:15:41.489709 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:15:41.489749 (+    40us) tablet.cc:662] Decoding operations
0709 12:15:41.490802 (+  1053us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:15:41.490937 (+   135us) write_op.cc:641] Partition lock acquired for write op
0709 12:15:41.490972 (+    35us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:15:41.491688 (+   716us) tablet.cc:701] Row locks acquired
0709 12:15:41.491738 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:15:41.491931 (+   193us) write_op.cc:270] Start()
0709 12:15:41.492122 (+   191us) write_op.cc:276] Timestamp: P: 1783599341491857 usec, L: 0
0709 12:15:41.492162 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:15:41.583864 (+ 91702us) log.cc:844] Serialized 2865 byte log entry
0709 12:15:42.176100 (+592236us) op_driver.cc:464] REPLICATION: finished
0709 12:15:42.283106 (+107006us) write_op.cc:301] APPLY: starting
0709 12:15:42.284764 (+  1658us) tablet.cc:1367] starting BulkCheckPresence
0709 12:15:42.316511 (+ 31747us) tablet.cc:1370] finished BulkCheckPresence
0709 12:15:42.316695 (+   184us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:15:42.498960 (+182265us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:15:42.499254 (+   294us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:15:42.499311 (+    57us) write_op.cc:312] APPLY: finished
0709 12:15:42.499654 (+   343us) log.cc:844] Serialized 73 byte log entry
0709 12:15:42.567353 (+ 67699us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:42.567495 (+   142us) write_op.cc:454] Released schema lock
0709 12:15:42.567567 (+    72us) write_op.cc:341] FINISH: Updating metrics
0709 12:15:42.568119 (+   552us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:42.568149 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":311225,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":106498,"apply.run_cpu_time_us":14932,"apply.run_wall_time_us":285050,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33359,"num_ops":7,"prepare.queue_time_us":32191,"prepare.run_cpu_time_us":74326,"prepare.run_wall_time_us":351032,"raft.queue_time_us":132436,"raft.run_cpu_time_us":52503,"raft.run_wall_time_us":68130,"replication_time_us":683717,"thread_start_us":280071,"threads_started":4,"wal-append.queue_time_us":225850,"wal-append.run_cpu_time_us":66484,"wal-append.run_wall_time_us":1670962}]]}
I20260709 12:15:47.631049 12392 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 20 to 25, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 25 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:15:47.633962 21940 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.152s	user 0.001s	sys 0.008s
I20260709 12:15:47.586062 22014 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.621994 21908 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:47.591148 22016 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:47.616405 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.597566 21233 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.618538 21786 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.597373 21848 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:47.649415 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.643476 22016 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:47.650756 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.651526 12572 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:47.653534 21799 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.640766 22010 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.655184 21233 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.647773 22024 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
W20260709 12:15:47.659541 21999 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260709 12:15:47.660434 21901 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.761s	user 0.010s	sys 0.000s
I20260709 12:15:47.607268 22004 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:15:47.633194 12392 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 8 to 21. New cstate: current_term: 21 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:47.670612 22036 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:47.670279 12392 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 19 to 26. New cstate: current_term: 26 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.635617 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.673059 12567 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:15:47.673964 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.674312 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.674893 12572 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:47.674919 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3874) took 1172 ms. Trace:
I20260709 12:15:47.675683 12565 rpcz_store.cc:276] 0709 12:15:46.502849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.119890 (+617041us) service_pool.cc:224] Handling call
0709 12:15:47.120531 (+   641us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:47.313216 (+192685us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:47.313270 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:47.314154 (+   884us) log.cc:844] Serialized 26 byte log entry
0709 12:15:47.567350 (+253196us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:47.599556 (+ 32206us) log.cc:844] Serialized 14 byte log entry
0709 12:15:47.599866 (+   310us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.599978 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:47.666907 (+ 66929us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b180040c540. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:47.674562 (+  7655us) raft_consensus.cc:1685] finished
0709 12:15:47.674713 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:47.674861 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1924,"prepare.queue_time_us":31535,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5192,"raft.queue_time_us":44823,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":6464,"spinlock_wait_cycles":29234560,"thread_start_us":384432,"threads_started":2,"wal-append.queue_time_us":279583}
I20260709 12:15:47.640785 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3707) took 1389 ms. Trace:
I20260709 12:15:47.638799 22014 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.677807 12645 rpcz_store.cc:276] 0709 12:15:46.251436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.410160 (+158724us) service_pool.cc:224] Handling call
0709 12:15:46.450720 (+ 40560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:46.460657 (+  9937us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:46.560453 (+ 99796us) log.cc:844] Serialized 14 byte log entry
0709 12:15:46.560794 (+   341us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:46.560850 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:46.560897 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.640515 (+1079618us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:47.640578 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:47.640718 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":187366,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":88,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":2783,"spinlock_wait_cycles":22269312,"thread_start_us":90358,"threads_started":1}
I20260709 12:15:47.680173 22014 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.680444 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3873) took 1410 ms. Trace:
I20260709 12:15:47.635324 22019 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:47.662142 21977 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:47.662725 21939 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.186s	user 0.002s	sys 0.003s
I20260709 12:15:47.664165 22011 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.666613 22013 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.668150 21978 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.645159 21908 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.674394 22004 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:47.675186 12567 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:47.744005 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.681646 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.744961 12565 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:47.680974 12569 rpcz_store.cc:276] 0709 12:15:46.270505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.066906 (+796401us) service_pool.cc:224] Handling call
0709 12:15:47.067447 (+   541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:47.445121 (+377674us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:47.445161 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:47.447021 (+  1860us) log.cc:844] Serialized 26 byte log entry
0709 12:15:47.589217 (+142196us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:47.589252 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.589330 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:47.680201 (+ 90871us) raft_consensus.cc:1685] finished
0709 12:15:47.680231 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:47.680381 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75489,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":98,"thread_start_us":447082,"threads_started":2,"wal-append.queue_time_us":231239}
I20260709 12:15:47.682485 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:47.747341 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64954.2us
I20260709 12:15:47.748106 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.506130 21982 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.597358 21881 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:47.611230 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:47.688998 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:47.708806 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:47.749373 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138681us
I20260709 12:15:47.731402 12392 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 27, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 27 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:47.753839 12392 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 26, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 26 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:15:47.727514 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 186971us
I20260709 12:15:47.760845 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4010) took 1095 ms. Trace:
I20260709 12:15:47.761291 12490 rpcz_store.cc:276] 0709 12:15:46.665152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.665412 (+   260us) service_pool.cc:224] Handling call
0709 12:15:47.760641 (+1095229us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:47.760756 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31481600}
I20260709 12:15:47.759745 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:47.809034 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49355.4us
I20260709 12:15:47.813846 12392 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 20. New cstate: current_term: 20 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.815026 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:47.812232 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51040.9us
I20260709 12:15:47.728586 21601 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:47.742681 21748 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.764315 21881 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:47.707649 21901 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.763181 21786 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:47.808511 21233 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:47.833169 21983 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:47.833758 21983 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:47.834677 21983 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.14, Last appended by leader: 14, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.836241 21983 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:47.808686 21799 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.840989 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4011) took 1174 ms. Trace:
I20260709 12:15:47.841647 22013 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:47.844288 22013 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.846966 22013 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.849319 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:47.829589 21748 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.853173 21748 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.824239 12392 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:47.828610 21601 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.816910 22028 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:47.875077 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3854) took 2520 ms. Trace:
I20260709 12:15:47.843098 12496 rpcz_store.cc:276] 0709 12:15:46.666922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.667163 (+   241us) service_pool.cc:224] Handling call
0709 12:15:46.680448 (+ 13285us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:46.680866 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:46.680905 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:46.684974 (+  4069us) log.cc:844] Serialized 52 byte log entry
0709 12:15:46.894592 (+209618us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:15:46.927521 (+ 32929us) log.cc:844] Serialized 14 byte log entry
0709 12:15:47.673001 (+745480us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b18004a8a80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:15:47.840511 (+167510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.840605 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:47.840665 (+    60us) raft_consensus.cc:1685] finished
0709 12:15:47.840745 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:47.840918 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31947,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":65,"raft.queue_time_us":169184,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":6031,"spinlock_wait_cycles":18192256,"thread_start_us":299781,"threads_started":2,"wal-append.queue_time_us":208155}
I20260709 12:15:47.753670 21978 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.887058 12564 rpcz_store.cc:276] 0709 12:15:45.354956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.357739 (+  2783us) service_pool.cc:224] Handling call
0709 12:15:45.360005 (+  2266us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:45.360269 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:45.360295 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:45.368142 (+  7847us) log.cc:844] Serialized 26 byte log entry
0709 12:15:45.436356 (+ 68214us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:45.436399 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:45.436498 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:47.662121 (+2225623us) spinlock_profiling.cc:243] Waited 9.93 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:47.854244 (+192123us) raft_consensus.cc:1685] finished
0709 12:15:47.854267 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:47.874982 (+ 20715us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22832896,"thread_start_us":40273,"threads_started":1,"wal-append.queue_time_us":86552}
I20260709 12:15:47.909813 21978 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.857738 21799 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.914466 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.893756 22022 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:47.915532 12564 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:47.903537 22037 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:15:47.904184 22019 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.923470 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:47.926630 22008 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:47.830575 21786 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:47.927946 21786 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:47.927793 22008 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:47.934660 22008 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.903326 21901 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.840021 21233 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:47.882993 22029 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:47.941349 21233 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:47.892491 22016 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.941936 21799 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:47.943845 22016 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.883072 22004 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:47.909327 21908 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:47.915613 22022 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:47.947803 22022 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:47.917299 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.900326 21601 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.953881 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:47.821242 22036 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:47.922679 22019 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:47.880122 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.958871 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:47.959792 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.962903 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:15:47.963371 12564 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
W20260709 12:15:47.963927 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3703) took 2369 ms (client timeout 2294 ms). Trace:
I20260709 12:15:47.962975 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.941502 22029 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:47.964370 12638 rpcz_store.cc:269] 0709 12:15:45.641573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.642016 (+   443us) service_pool.cc:224] Handling call
0709 12:15:47.963749 (+2321733us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:47.963867 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1046864128}
I20260709 12:15:47.965509 12571 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
W20260709 12:15:47.968071 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88624.5us
I20260709 12:15:47.969106 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.976308 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.978976 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:47.981596 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:47.984239 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:47.987265 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:47.988507 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:47.905441 22028 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:47.991822 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1253) took 1843 ms (client timeout 2211 ms). Trace:
W20260709 12:15:47.996310 12644 rpcz_store.cc:269] 0709 12:15:46.148572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.149134 (+   562us) service_pool.cc:224] Handling call
0709 12:15:47.991588 (+1842454us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:47.991741 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":451685248}
I20260709 12:15:47.993572 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3878) took 1036 ms. Trace:
I20260709 12:15:47.997459 12570 rpcz_store.cc:276] 0709 12:15:46.957073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.230120 (+273047us) service_pool.cc:224] Handling call
0709 12:15:47.230413 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:47.338747 (+108334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:47.993262 (+654515us) spinlock_profiling.cc:243] Waited 96.4 ms on lock 0x7b54003ab954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:47.993503 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221668224}
I20260709 12:15:47.998775 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:47.999413 12570 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:48.009145 12563 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:15:47.974251 22024 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:47.966135 22037 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.017277 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.020643 12565 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:47.977270 21983 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:48.018517 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.024099 12564 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:48.007630 22019 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:47.983461 22018 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:48.025949 22042 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:48.026439 22042 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.026786 22042 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:47.970198 22029 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:47.991578 21908 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:48.022395 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103835us
I20260709 12:15:47.982831 21740 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:48.012916 22008 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:48.008018 22028 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:48.013697 21978 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:48.035815 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:48.043722 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:47.974022 22036 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.028704 21983 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:48.046720 12564 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:48.058326 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:48.046310 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:48.059667 12570 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:48.021445 21643 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:15:48.037437 22008 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:48.025333 22038 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:15:48.067812 21233 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.067349 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:48.070246 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:48.074229 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:48.028357 22037 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.034476 21908 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.056677 21983 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:48.111160 21233 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.073909 22008 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:48.077544 21466 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:48.080192 22004 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:48.080823 22024 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:48.054558 21978 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.105249 21901 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.139109 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.142520 12565 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:48.111013 21976 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.118672 21902 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.147152 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.186544 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:48.082926 22022 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.143703 21954 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:48.211974 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:48.214764 12639 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:15:48.215428 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:48.224543 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:48.148124 22034 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:48.141369 21466 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:48.233712 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:48.146173 21799 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.144174 22004 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:48.146631 22024 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:48.214157 21954 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:48.234323 21946 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:48.233707 22034 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:48.182792 21976 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:48.254559 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111792us
I20260709 12:15:48.143772 21901 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.262671 21976 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:48.264488 22044 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:15:48.248070 21954 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:15:48.273464 22048 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
W20260709 12:15:48.274978 22049 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:15:48.276959 12639 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:15:48.278251 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92853.8us
I20260709 12:15:48.275434 21466 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.250524 22034 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:48.244406 22026 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.251410 22051 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.284538 21931 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.300593 22029 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.302207 22037 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.306953 21881 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.279488 21946 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:48.306988 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1248) took 2517 ms (client timeout 1882 ms). Trace:
I20260709 12:15:48.308997 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:48.312345 21931 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.344158 21931 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.344661 22023 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:48.345324 22023 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:48.341171 12646 rpcz_store.cc:269] 0709 12:15:45.789541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.821797 (+ 32256us) service_pool.cc:224] Handling call
0709 12:15:48.306715 (+2484918us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:48.306880 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1501826944}
I20260709 12:15:48.345945 22023 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:48.346273 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.311192 22026 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.348991 22026 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.346295 21632 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:48.312343 21978 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.317283 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:48.352759 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:48.322911 21590 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:48.357908 21590 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:48.359364 21590 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.313190 22004 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.313520 22051 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.364353 22051 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.337052 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.366441 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.361404 21590 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:48.367245 21982 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.368072 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.368288 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.350435 21632 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:48.367959 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.369043 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:15:48.370023 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.313376 22024 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.371335 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.372354 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.361924 21908 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.374177 22036 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.247553 21799 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.370023 21632 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.375422 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4019) took 1532 ms. Trace:
I20260709 12:15:48.375845 12489 rpcz_store.cc:276] 0709 12:15:46.843116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.906817 (+ 63701us) service_pool.cc:224] Handling call
0709 12:15:46.907185 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:47.098815 (+191630us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:47.098862 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:47.098924 (+    62us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:15:47.098972 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:48.375134 (+1276162us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:48.375190 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:48.375350 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":439558528}
I20260709 12:15:48.377656 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:48.378276 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3820) took 1950 ms. Trace:
I20260709 12:15:48.378372 13257 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:48.381470 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:48.381595 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:48.382141 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1260) took 1519 ms (client timeout 1925 ms). Trace:
I20260709 12:15:48.378805 12566 rpcz_store.cc:276] 0709 12:15:46.427587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.092925 (+665338us) service_pool.cc:224] Handling call
0709 12:15:47.962502 (+869577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:47.963442 (+   940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:48.378056 (+414614us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b5400102f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:48.378209 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1999015168}
W20260709 12:15:48.408202 12640 rpcz_store.cc:269] 0709 12:15:46.863129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:46.864273 (+  1144us) service_pool.cc:224] Handling call
0709 12:15:48.381919 (+1517646us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:48.382082 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1476662528}
I20260709 12:15:48.396433 13257 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:48.398272 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.412619 12490 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:48.414043 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:48.415506 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:48.416139 12566 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:48.421860 22013 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:48.393812 21978 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:48.422557 22013 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.422879 21978 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.424923 21978 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.394174 21881 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.396059 21973 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:15:48.421861 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.444888 21545 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.445789 21545 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.392334 21946 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:48.453238 21786 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.524s	user 0.010s	sys 0.003s
I20260709 12:15:48.453933 21786 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.456164 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3831) took 1168 ms. Trace:
I20260709 12:15:48.456718 12563 rpcz_store.cc:276] 0709 12:15:47.287934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.330589 (+ 42655us) service_pool.cc:224] Handling call
0709 12:15:48.008777 (+678188us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:48.009743 (+   966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:48.455825 (+446082us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54003ab950. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:48.456078 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1524422400}
I20260709 12:15:48.458132 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.458690 21923 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:48.458990 12566 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:15:48.459327 21923 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.465019 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4021) took 1312 ms. Trace:
I20260709 12:15:48.465677 12488 rpcz_store.cc:276] 0709 12:15:47.152744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.152941 (+   197us) service_pool.cc:224] Handling call
0709 12:15:48.464762 (+1311821us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:48.464929 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899424000}
W20260709 12:15:48.483122 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:48.484376 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.484678 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:48.485373 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82198us
I20260709 12:15:48.485525 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:48.487844 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:48.489966 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.490639 12571 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:48.490686 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.490984 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.500319 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.491649 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
W20260709 12:15:48.359534 22045 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.152s	user 0.002s	sys 0.000s
I20260709 12:15:48.510082 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.510540 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.511982 12491 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:48.512625 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.511047 13218 leader_election.cc:400] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31.
I20260709 12:15:48.513424 13218 leader_election.cc:403] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:15:48.514147 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:48.515578 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.515748 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.513679 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.519115 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.519796 12490 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:48.521184 22051 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:48.522470 22022 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.527801 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:48.376756 21799 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.399832 22010 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.379061 21983 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.540934 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:48.542021 13257 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
W20260709 12:15:48.505713 22044 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 12:15:48.502926 21908 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.545007 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:48.554144 12639 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:48.556138 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.556901 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:48.559939 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.562400 21545 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.562922 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.563751 12566 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:48.551964 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:48.567536 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:48.568467 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.569516 12645 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:48.571743 13256 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:48.575474 21601 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:48.575759 22052 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:48.576372 21982 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:48.583523 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:48.587210 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:48.593062 12566 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:48.593444 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:48.596477 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.597450 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:48.598176 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.544266 12571 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:48.599038 12569 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:15:48.600075 21233 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.600853 12569 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:15:48.601341 12571 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:15:48.600634 21233 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.610848 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.604871 12639 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:15:48.606796 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:48.608796 12571 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:15:48.597265 22052 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:48.619313 12571 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:48.623188 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.621819 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.625286 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:48.641422 22071 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:48.642249 22071 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.644750 22071 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.648756 21931 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.649772 21931 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.650270 21954 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.564857 21908 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.606036 21930 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.665510 21954 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.665989 21930 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.611260 22026 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:48.668617 22026 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.669225 22026 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:48.653861 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.657888 12571 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:15:48.679571 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:48.681289 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65182.2us
I20260709 12:15:48.628472 22024 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:48.690129 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.637734 22010 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:48.691447 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:48.696228 21233 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.629555 21799 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.702886 12488 raft_consensus.cc:2368] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 24.
I20260709 12:15:48.680970 22001 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:48.705996 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.662174 22052 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:48.720082 21233 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.720311 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.714576 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.716126 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.722901 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.688462 21954 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:48.720701 21233 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.730620 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:48.734148 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.734297 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.735519 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:48.736609 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:48.737871 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.739172 12496 raft_consensus.cc:2368] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:15:48.740167 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.741436 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.743628 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.744135 12491 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:48.744762 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:48.729480 22008 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.001s
I20260709 12:15:48.716545 22001 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:48.757544 22062 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:15:48.759806 21983 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.757622 22001 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:48.762915 22034 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.698634 22037 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.045s
I20260709 12:15:48.752856 21233 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.775874 22023 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.734535 22052 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.780821 21924 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.785117 13262 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
W20260709 12:15:48.761487 22028 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.707s	user 0.010s	sys 0.003s
I20260709 12:15:48.785986 22008 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.803474 21908 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:48.809494 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:48.810251 22008 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.743434 22074 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:48.811307 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:48.788719 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1249) took 2960 ms. Trace:
I20260709 12:15:48.812042 12642 rpcz_store.cc:276] 0709 12:15:45.789826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:45.856997 (+ 67171us) service_pool.cc:224] Handling call
0709 12:15:45.857497 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:46.438126 (+580629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:48.750086 (+2311960us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:48.750283 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1237078912}
I20260709 12:15:48.731889 21906 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:48.716905 22024 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:48.793619 22051 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31.
I20260709 12:15:48.796367 22026 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.714123 21799 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.746337 21752 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
I20260709 12:15:48.799367 21786 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.746816 22029 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.801203 22013 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:48.801349 22042 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.774s	user 0.014s	sys 0.000s
I20260709 12:15:48.802502 22023 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:48.807586 22028 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.812872 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:48.862322 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49566.5us
I20260709 12:15:48.745050 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.814911 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.811203 22074 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.780895 22052 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:48.871191 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 158614us
I20260709 12:15:48.821381 21906 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.872493 12611 leader_election.cc:400] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:48.873062 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:48.874749 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:48.875280 12536 leader_election.cc:400] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 24.
I20260709 12:15:48.875787 12611 leader_election.cc:403] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:15:48.866477 22009 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:48.877398 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.869369 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.814260 22034 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.749541 12496 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:48.877146 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.890333 13257 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:15:48.891513 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:48.888665 12536 leader_election.cc:400] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:15:48.889094 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:48.893121 12536 leader_election.cc:403] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:15:48.895800 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:48.897567 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:48.899811 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.901650 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.922269 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.928045 12490 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:48.898054 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:48.864588 22042 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:48.939545 21857 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:48.904139 21924 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:48.942817 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:48.975448 22026 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:48.975486 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:48.976526 12639 raft_consensus.cc:2368] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:48.946493 22008 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:48.944526 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.945847 22028 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.978325 22008 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:48.950053 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52116.3us
I20260709 12:15:48.978794 22008 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:48.943327 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:48.954787 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:48.980202 12488 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:48.980906 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:48.946732 22075 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:15:48.886531 22024 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:48.886967 21786 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26.
I20260709 12:15:48.909826 22013 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:48.961570 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.962746 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:48.967213 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.950829 22051 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:48.970781 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.970899 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.000754 12641 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 30 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 14.
I20260709 12:15:49.000963 12640 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:15:48.970085 21731 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:15:48.974110 21906 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:48.977483 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:48.940891 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51403.9us
I20260709 12:15:48.896436 22037 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:48.960361 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:48.934250 22011 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:48.998509 12497 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:49.004376 12643 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:49.005867 12646 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:15:49.007378 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.010581 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.998966 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.021340 12637 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:15:48.940421 22076 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:49.013805 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.878890 22023 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.107s
I20260709 12:15:48.977370 22074 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:49.009660 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
W20260709 12:15:48.993006 22072 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.002s	sys 0.000s
I20260709 12:15:49.019382 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.028687 12496 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
W20260709 12:15:49.025978 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:49.026400 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:48.990947 21740 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:48.986186 22009 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:49.031108 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.037849 12639 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:15:49.031867 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.039373 12640 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
W20260709 12:15:49.039649 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93067.4us
I20260709 12:15:49.034634 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:15:49.035540 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.040483 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:49.035730 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.032351 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:49.022719 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 311706us
I20260709 12:15:49.030757 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:49.040735 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:49.043162 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.042407 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.009265 22037 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:49.044509 12639 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:15:49.044514 12637 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:49.041090 12496 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:15:49.051595 22009 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.007769 22001 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.053983 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.054699 12641 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:49.027915 22010 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:15:49.056074 22001 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:49.056918 22010 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:49.057426 22010 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:49.061514 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:49.064095 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.025249 22076 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:49.069068 12644 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:49.074780 12643 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:15:49.015601 21974 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:15:49.082945 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:49.043586 22029 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:49.086493 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:49.089332 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:49.090378 12639 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260709 12:15:49.091035 12639 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 14
I20260709 12:15:49.091337 12639 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:15:49.091663 12639 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:49.059072 22013 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:49.061813 21954 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:49.059883 22037 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:49.092958 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.091912 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:49.085166 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:49.106635 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.107808 12637 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:49.110334 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:49.083379 22052 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:15:49.093034 22080 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.104s	user 0.001s	sys 0.001s
I20260709 12:15:49.084619 21978 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:49.081230 22076 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:49.070574 22016 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:15:49.118851 12569 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:49.124212 12488 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:49.123698 12496 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:49.069262 21740 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:15:49.129513 12644 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:49.109755 21801 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:15:49.105015 22013 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:49.135637 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:49.136112 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95420.4us
I20260709 12:15:49.134109 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:49.139567 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:49.068130 22010 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.117910 22052 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:49.098987 22029 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.129473 21978 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:49.140941 22013 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:49.144825 21752 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:49.157361 21933 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:49.158356 21978 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:49.161476 12640 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:15:49.180231 12646 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:15:49.162562 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:49.165293 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1262) took 1030 ms. Trace:
I20260709 12:15:49.183874 12638 rpcz_store.cc:276] 0709 12:15:48.134628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.135265 (+   637us) service_pool.cc:224] Handling call
0709 12:15:48.135704 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:48.811921 (+676217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.165086 (+353165us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:49.165226 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553165056}
I20260709 12:15:49.171670 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:49.177189 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:49.178318 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:49.186156 12496 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:15:49.174137 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:49.186796 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:49.191705 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:49.152370 22052 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:49.193848 12644 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:15:49.155058 21799 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:49.198859 12637 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:49.170244 22010 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:49.199967 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.193289 21857 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:49.179316 21933 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:49.195466 22036 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:49.205476 12571 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 26. Ops: []
I20260709 12:15:49.196066 21752 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:49.211488 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:49.213024 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.222632 12643 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:15:49.248592 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:49.223048 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.213593 12496 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:49.205600 22086 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:49.200776 22087 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:49.246086 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:49.225147 22081 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.117s	user 0.000s	sys 0.002s
I20260709 12:15:49.225742 22036 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:49.309558 22084 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:15:49.311455 22023 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:49.327525 21923 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:49.327688 22023 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.331594 12488 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:49.223541 21857 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:49.319939 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.324088 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1264) took 1141 ms. Trace:
I20260709 12:15:49.336668 12645 rpcz_store.cc:276] 0709 12:15:48.182622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.182858 (+   236us) service_pool.cc:224] Handling call
0709 12:15:48.219447 (+ 36589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:48.572714 (+353267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.323779 (+751065us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:49.324007 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804566400}
I20260709 12:15:49.230131 21772 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:49.325147 21930 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:49.320814 21752 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:49.341785 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 202313us
I20260709 12:15:49.339787 21857 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:49.225481 21933 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.316728 22087 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.308189 22086 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.356637 21857 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:49.377172 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.378520 12645 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
W20260709 12:15:49.414857 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66754.1us
I20260709 12:15:49.377942 22023 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:49.417116 21933 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:49.373665 22036 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.453357 22086 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:49.449293 22088 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
W20260709 12:15:49.460137 22068 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.692s	user 0.000s	sys 0.003s
I20260709 12:15:49.461166 22011 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:15:49.498627 22087 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:49.501380 22052 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.516017 22076 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.518584 22052 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:49.558055 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1279) took 1046 ms. Trace:
I20260709 12:15:49.562515 12489 rpcz_store.cc:276] 0709 12:15:48.507485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.510787 (+  3302us) service_pool.cc:224] Handling call
0709 12:15:48.511127 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:48.544085 (+ 32958us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:48.544129 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:48.547705 (+  3576us) log.cc:844] Serialized 26 byte log entry
0709 12:15:49.328980 (+781275us) spinlock_profiling.cc:243] Waited 4.02 ms on lock 0x7b18001bd180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:15:49.552009 (+223029us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:49.552218 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.552483 (+   265us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:49.552667 (+   184us) raft_consensus.cc:1685] finished
0709 12:15:49.552710 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:49.553469 (+   759us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53651,"raft.queue_time_us":25160,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":9369,"spinlock_wait_cycles":9250048,"thread_start_us":221900,"threads_started":1,"wal-append.queue_time_us":939996}
I20260709 12:15:49.581009 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1648) took 1619 ms. Trace:
I20260709 12:15:49.581728 12568 rpcz_store.cc:276] 0709 12:15:47.961792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.340470 (+378678us) service_pool.cc:224] Handling call
0709 12:15:48.340872 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:48.491999 (+151127us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:48.492030 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:48.508437 (+ 16407us) log.cc:844] Serialized 26 byte log entry
0709 12:15:48.638189 (+129752us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:48.638254 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:48.638367 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:49.459289 (+820922us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:49.580039 (+120750us) raft_consensus.cc:1685] finished
0709 12:15:49.580099 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:49.580713 (+   614us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":96470,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":9700,"spinlock_wait_cycles":96561024,"thread_start_us":71771,"threads_started":1,"wal-append.queue_time_us":221669}
W20260709 12:15:49.586377 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:49.586548 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.590546 12642 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:49.593453 12643 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:15:49.595197 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.596283 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:49.598779 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:49.600742 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.597492 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.601435 12639 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27. Re-sending same reply.
I20260709 12:15:49.589517 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.621066 12637 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:49.612308 21978 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.637735 22087 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:49.638417 22076 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:49.638743 22087 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:49.649901 12645 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:15:49.658842 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:49.661502 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:49.670341 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:49.672083 12639 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:49.674431 12569 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:49.670620 21978 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:49.724366 22097 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:49.724987 21970 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:49.725914 22097 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.726153 21970 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.722954 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1288) took 1173 ms. Trace:
I20260709 12:15:49.728935 13257 rpcz_store.cc:276] 0709 12:15:48.549846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.551585 (+  1739us) service_pool.cc:224] Handling call
0709 12:15:49.721170 (+1169585us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:49.722743 (+  1573us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":635144576}
W20260709 12:15:49.724403 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1125) took 1357 ms (client timeout 1523 ms). Trace:
W20260709 12:15:49.732388 13262 rpcz_store.cc:269] 0709 12:15:48.366531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.371053 (+  4522us) service_pool.cc:224] Handling call
0709 12:15:49.723028 (+1351975us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:49.724203 (+  1175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":949561728}
I20260709 12:15:49.729826 13256 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:15:49.739609 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:49.740783 13258 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:49.741878 12639 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 26. Ops: []
I20260709 12:15:49.742048 21931 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:49.743621 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:49.750801 22067 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.983s	user 0.000s	sys 0.004s
I20260709 12:15:49.759766 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:49.760686 12571 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:15:49.766844 22092 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.314s	user 0.000s	sys 0.003s
I20260709 12:15:49.773675 21901 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:49.781122 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:49.782346 22094 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.328s	user 0.000s	sys 0.003s
W20260709 12:15:49.782936 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:49.785306 22093 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.332s	user 0.000s	sys 0.002s
I20260709 12:15:49.789045 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1280) took 1281 ms. Trace:
I20260709 12:15:49.791272 21777 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:49.825816 22036 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:49.789492 12492 rpcz_store.cc:276] 0709 12:15:48.507765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.508702 (+   937us) service_pool.cc:224] Handling call
0709 12:15:48.509139 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:48.509560 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:48.509598 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:49.227181 (+717583us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:15:49.227710 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 12:15:49.360587 (+132877us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:49.360628 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.360705 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:49.487378 (+126673us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b4800080058. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:49.784069 (+296691us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:49.788526 (+  4457us) raft_consensus.cc:1685] finished
0709 12:15:49.788555 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:49.788828 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":26744,"raft.run_cpu_time_us":20589,"raft.run_wall_time_us":950460,"spinlock_wait_cycles":21133440,"thread_start_us":132445,"threads_started":1,"wal-append.queue_time_us":225393}
I20260709 12:15:49.817587 12565 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:15:49.839491 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 320460us
W20260709 12:15:49.844635 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60237.7us
I20260709 12:15:49.842388 22042 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:49.862643 13257 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:15:49.865163 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3833) took 2249 ms. Trace:
I20260709 12:15:49.900703 12571 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:15:49.909188 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65436.2us
I20260709 12:15:49.912201 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:49.913655 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:49.928020 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:49.941587 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:49.865919 12567 rpcz_store.cc:276] 0709 12:15:47.610090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.755125 (+145035us) service_pool.cc:224] Handling call
0709 12:15:47.755442 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:47.755690 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:15:47.986447 (+230757us) log.cc:844] Serialized 14 byte log entry
0709 12:15:48.144958 (+158511us) log.cc:844] Serialized 14 byte log entry
0709 12:15:48.345580 (+200622us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b180048c340. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:49.424029 (+1078449us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b180048c340. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:15:49.578605 (+154576us) log.cc:844] Serialized 14 byte log entry
0709 12:15:49.701235 (+122630us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:49.702927 (+  1692us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:15:49.713121 (+ 10194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.723820 (+ 10699us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:49.859290 (+135470us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1477075,"prepare.queue_time_us":1791320,"prepare.run_cpu_time_us":226,"prepare.run_wall_time_us":7122,"raft.queue_time_us":23553,"raft.run_cpu_time_us":581,"raft.run_wall_time_us":45612,"spinlock_wait_cycles":9334016,"thread_start_us":322974,"threads_started":4,"wal-append.queue_time_us":156710,"wal-append.run_cpu_time_us":1942,"wal-append.run_wall_time_us":1160308}
I20260709 12:15:49.952332 12639 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:49.964695 21848 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:15:49.966447 21848 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:49.966835 21848 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:49.968170 12494 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:15:49.970036 22042 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:15:49.971118 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:49.968271 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48039.4us
I20260709 12:15:49.986351 22089 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:49.987279 22089 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.989862 21921 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:49.991173 22087 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:49.968199 21848 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:49.992524 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3903) took 1909 ms. Trace:
I20260709 12:15:49.992944 12566 rpcz_store.cc:276] 0709 12:15:48.083301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.600087 (+516786us) service_pool.cc:224] Handling call
0709 12:15:48.600512 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:48.600766 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:49.736182 (+1135416us) log.cc:844] Serialized 14 byte log entry
0709 12:15:49.931366 (+195184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:49.942987 (+ 11621us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:49.960821 (+ 17834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.978517 (+ 17696us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:49.992326 (+ 13809us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":991976,"prepare.queue_time_us":1072424,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":90,"raft.queue_time_us":3664,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":21071,"spinlock_wait_cycles":6903936,"thread_start_us":280215,"threads_started":2,"wal-append.queue_time_us":212752}
I20260709 12:15:49.997603 22029 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:49.999966 22103 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.003s	sys 0.000s
I20260709 12:15:49.930543 22036 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.002975 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.005046 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:50.011946 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:49.970714 22042 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:50.020560 12571 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:50.022276 22100 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.022939 21847 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:50.023842 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1295) took 1389 ms. Trace:
I20260709 12:15:50.023357 22100 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.024266 13256 rpcz_store.cc:276] 0709 12:15:48.634191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.634369 (+   178us) service_pool.cc:224] Handling call
0709 12:15:49.728845 (+1094476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:49.733952 (+  5107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:50.023582 (+289630us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540058d750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:50.023760 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2503002368}
I20260709 12:15:50.025736 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:49.990561 21921 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:50.048869 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4031) took 1714 ms (client timeout 1859 ms). Trace:
W20260709 12:15:50.055791 22008 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.962s	user 0.022s	sys 0.006s
I20260709 12:15:50.031988 21848 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:50.065574 12637 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:15:50.076354 22008 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:50.082412 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:50.084259 21772 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.697s	user 0.015s	sys 0.016s
I20260709 12:15:50.041352 22036 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:50.085860 21933 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.647s	user 0.017s	sys 0.003s
I20260709 12:15:50.086216 22076 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.086714 12493 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:50.086915 22076 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:50.089388 22113 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:15:50.089834 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.093899 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:50.099851 22115 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:15:50.063483 12494 rpcz_store.cc:269] 0709 12:15:48.334717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.334881 (+   164us) service_pool.cc:224] Handling call
0709 12:15:50.048305 (+1713424us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:50.048678 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3727657856}
W20260709 12:15:50.102622 22104 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.171s	user 0.000s	sys 0.002s
I20260709 12:15:50.103873 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.106305 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1649) took 2144 ms. Trace:
I20260709 12:15:50.085068 21847 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:50.109345 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.076661 22042 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:50.116734 22010 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.861s	user 0.014s	sys 0.013s
W20260709 12:15:50.010824 22024 consensus_queue.cc:1175] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 22 status { last_received { term: 22 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 22 index: 16 } }
I20260709 12:15:50.086673 21933 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:15:50.130211 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.133721 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.136809 21922 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:15:50.136833 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.137589 21922 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:50.137419 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3922) took 1344 ms. Trace:
I20260709 12:15:50.138335 12568 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 34 because replica has last-logged OpId of term: 32 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 19.
I20260709 12:15:50.138685 12565 rpcz_store.cc:276] 0709 12:15:48.792928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.980629 (+187701us) service_pool.cc:224] Handling call
0709 12:15:50.137161 (+1156532us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:50.137340 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1336639360}
I20260709 12:15:50.139266 21922 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.140054 12568 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:15:50.141060 21922 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:50.141969 12565 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:15:50.143699 21533 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.144316 21533 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.164460 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.165481 12643 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:15:50.164559 21857 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.784s	user 0.021s	sys 0.003s
I20260709 12:15:50.170876 21857 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.174935 22097 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:50.181869 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:50.182574 22013 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.000s	user 0.008s	sys 0.017s
I20260709 12:15:50.183346 22013 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:50.186834 22037 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.058s	user 0.010s	sys 0.014s
I20260709 12:15:50.187852 22037 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.189797 13253 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:50.190071 21970 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.190621 21632 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:50.191839 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1214) took 1227 ms. Trace:
I20260709 12:15:50.144968 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 3917) took 1353 ms. Trace:
I20260709 12:15:50.192380 13253 rpcz_store.cc:276] 0709 12:15:48.964903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.965150 (+   247us) service_pool.cc:224] Handling call
0709 12:15:49.006863 (+ 41713us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:50.190430 (+1183567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:50.191411 (+   981us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:50.191752 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2719289088}
I20260709 12:15:50.193063 12569 rpcz_store.cc:276] 0709 12:15:48.791616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:49.135365 (+343749us) service_pool.cc:224] Handling call
0709 12:15:50.144698 (+1009333us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:50.144894 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1233552640}
I20260709 12:15:50.194675 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.195683 22089 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.197082 22100 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:50.199419 22109 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:15:50.201689 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3720) took 1071 ms. Trace:
W20260709 12:15:50.202535 22117 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.181s	user 0.000s	sys 0.002s
I20260709 12:15:50.202275 12642 rpcz_store.cc:276] 0709 12:15:49.130740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:49.131150 (+   410us) service_pool.cc:224] Handling call
0709 12:15:49.143161 (+ 12011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:49.591363 (+448202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:50.201301 (+609938us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:50.201546 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1026988416}
I20260709 12:15:50.203071 22011 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.126117 21978 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.203986 13256 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:15:50.204504 21978 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.207160 21978 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.106863 12572 rpcz_store.cc:276] 0709 12:15:47.962098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.420615 (+458517us) service_pool.cc:224] Handling call
0709 12:15:48.438735 (+ 18120us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:48.510502 (+ 71767us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:49.579037 (+1068535us) log.cc:844] Serialized 14 byte log entry
0709 12:15:49.696599 (+117562us) log.cc:844] Serialized 14 byte log entry
0709 12:15:49.699751 (+  3152us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:49.724355 (+ 24604us) log.cc:844] Serialized 26 byte log entry
0709 12:15:49.724547 (+   192us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:49.724588 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:49.724745 (+   157us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:50.101588 (+376843us) spinlock_profiling.cc:243] Waited 64.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:50.105817 (+  4229us) raft_consensus.cc:1685] finished
0709 12:15:50.105846 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:50.106071 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1193430,"prepare.queue_time_us":1214010,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":82217,"raft.run_cpu_time_us":650,"raft.run_wall_time_us":23958,"spinlock_wait_cycles":149267328,"thread_start_us":62054,"threads_started":1,"wal-append.queue_time_us":279125}
I20260709 12:15:50.199301 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:50.175158 22008 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.212181 12567 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:15:50.173461 21857 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:50.226197 22053 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.368s	user 0.015s	sys 0.005s
W20260709 12:15:50.226701 22053 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.470s	user 0.015s	sys 0.005s
I20260709 12:15:50.229067 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:50.229872 22099 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:50.230243 13253 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:50.231489 22118 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:15:50.231961 12572 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:15:50.204017 22011 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.233161 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.234159 21848 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:50.234927 21848 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:50.235110 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.236068 12566 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:50.236646 22038 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:50.237389 22038 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:50.238510 21970 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.239250 21970 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.239624 22038 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:50.237973 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4026) took 2803 ms (client timeout 2651 ms). Trace:
I20260709 12:15:50.241425 22038 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:50.234454 22011 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.217437 22076 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.229173 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.247839 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:50.256309 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.272207 12571 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24.
I20260709 12:15:50.250484 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:50.255558 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.258088 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.268074 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:50.223031 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:50.261726 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3899) took 2070 ms. Trace:
W20260709 12:15:50.241866 12493 rpcz_store.cc:269] 0709 12:15:47.434022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:47.437752 (+  3730us) service_pool.cc:224] Handling call
0709 12:15:50.237667 (+2799915us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:50.237878 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6069041408}
I20260709 12:15:50.273855 13258 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:50.314363 12569 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:50.259078 22013 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.328590 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.241590 21970 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:50.335059 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:50.249173 21533 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.245105 21848 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.340545 21848 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:50.254249 22097 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:50.348505 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80599us
I20260709 12:15:50.305120 22087 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.367906 21533 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:15:50.367758 22087 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.369019 21971 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.369935 21971 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.370836 22087 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.372406 21971 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.372779 12491 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:15:50.373996 21643 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:50.374714 21643 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.376693 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3843) took 2382 ms. Trace:
I20260709 12:15:50.376900 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:50.377151 21643 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.382000 21978 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:15:50.383168 21978 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:50.383705 21978 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:50.384799 21978 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 26.21, Last appended by leader: 21, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.230631 22099 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.385885 22037 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:50.386678 21921 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.387079 12571 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:50.377998 22119 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:15:50.378149 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:50.378505 12493 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:15:50.378813 21533 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:50.377178 12564 rpcz_store.cc:276] 0709 12:15:47.939114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.210098 (+270984us) service_pool.cc:224] Handling call
0709 12:15:48.210417 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:48.210630 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:15:49.901466 (+1690836us) log.cc:844] Serialized 14 byte log entry
0709 12:15:50.033920 (+132454us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:50.072183 (+ 38263us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:50.201531 (+129348us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:50.214727 (+ 13196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:50.223718 (+  8991us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:50.321152 (+ 97434us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1390991,"prepare.queue_time_us":1484423,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":97,"raft.queue_time_us":1838,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":5986,"spinlock_wait_cycles":11667840,"thread_start_us":368083,"threads_started":2,"wal-append.queue_time_us":104363}
I20260709 12:15:50.302017 12563 rpcz_store.cc:276] 0709 12:15:47.967608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.460510 (+492902us) service_pool.cc:224] Handling call
0709 12:15:48.460826 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:48.461108 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:49.720927 (+1259819us) log.cc:844] Serialized 14 byte log entry
0709 12:15:49.974050 (+253123us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:49.979558 (+  5508us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:50.013666 (+ 34108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:50.027256 (+ 13590us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:50.038166 (+ 10910us) inbound_call.cc:177] Queueing success response
0709 12:15:50.098668 (+ 60502us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
Metrics: {"mutex_wait_us":1118822,"prepare.queue_time_us":1211721,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":71,"raft.queue_time_us":13909,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":41998,"spinlock_wait_cycles":35041152,"thread_start_us":197364,"threads_started":2,"wal-append.queue_time_us":227641}
I20260709 12:15:50.400103 21533 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 29.11, Last appended by leader: 11, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.404693 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.405660 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:50.407034 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:50.407536 22097 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.408174 12492 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:50.410496 22097 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.410953 12495 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:15:50.430572 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.397867 22029 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:50.432775 22029 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.435384 22029 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.435441 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:50.436415 13257 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:50.437019 22099 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.417151 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.437698 22008 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.438547 22008 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.438913 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:50.439724 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:50.441040 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:50.430845 12460 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:50.442184 12460 leader_election.cc:403] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:15:50.453442 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 250822us
I20260709 12:15:50.449419 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:50.476920 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88538.6us
I20260709 12:15:50.454401 21931 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.459853 22008 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:50.484763 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.486939 12494 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
W20260709 12:15:50.481452 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 181007us
I20260709 12:15:50.484723 21978 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:50.505049 12461 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.037s	sys 0.023s
I20260709 12:15:50.504956 21789 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:50.506297 22038 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:15:50.510378 21921 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:50.511240 12462 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
W20260709 12:15:50.512732 12461 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.037s	sys 0.023s
I20260709 12:15:50.516048 12462 leader_election.cc:403] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:15:50.516201 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 674396us
I20260709 12:15:50.492040 22008 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:15:50.520159 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:50.519057 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:50.522091 22008 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:50.523301 22008 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.528904 21789 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.529856 21643 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:50.529992 21789 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.530794 21643 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:50.534536 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 118442us
I20260709 12:15:50.538847 21789 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.484869 21931 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:50.540131 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54632.6us
W20260709 12:15:50.550773 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 385118us
I20260709 12:15:50.537424 22099 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:15:50.558234 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 359018us
I20260709 12:15:50.560045 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:50.564970 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:50.567620 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:50.558588 21632 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.570386 21632 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:50.579288 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 185153us
I20260709 12:15:50.547125 21847 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.561085 22099 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.558822 22029 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:15:50.590324 12536 leader_election.cc:400] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:50.590796 12536 leader_election.cc:403] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:15:50.590597 22038 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:50.592708 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:50.594518 22097 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:15:50.595458 22097 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:50.596487 22036 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:50.598004 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:50.599545 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:50.599829 21740 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:50.600680 21930 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:50.599562 21922 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:50.602653 21971 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:50.602844 21922 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.603404 21922 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:50.606302 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:50.607625 21931 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:50.608376 21881 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:50.608875 21881 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.609218 21881 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:50.609740 21933 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:50.610322 21933 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:50.613785 22127 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:50.617779 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.617910 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.618676 12641 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:50.618939 12642 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:15:50.620055 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:50.620386 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.620997 12643 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:50.622092 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.624296 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.625092 12563 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:50.625399 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.621008 12565 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:50.601269 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:50.640187 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:50.641260 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:50.641875 12642 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
W20260709 12:15:50.643000 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60363.3us
I20260709 12:15:50.601992 22123 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:50.578338 21632 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:50.647097 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.647958 12565 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
I20260709 12:15:50.653088 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.653940 22087 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:50.661643 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83882.1us
I20260709 12:15:50.665185 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.667117 12639 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:50.666096 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:50.670807 13257 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:50.590957 21643 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.673967 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:50.675163 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:50.682873 21856 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:50.683089 12492 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:50.683535 21856 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.684011 21856 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:50.587622 21847 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.681835 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:50.628355 22099 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:50.697925 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:50.700713 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.708401 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.708992 22086 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:50.709878 22086 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.712595 22086 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.715981 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:50.717051 21632 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:50.717702 21632 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:50.718971 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:50.720428 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:50.718902 21632 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.705725 21971 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:50.721616 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:50.723275 21971 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:50.723789 21971 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:50.724020 21772 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:50.724830 21772 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:50.721767 21847 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:50.721928 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:50.727491 21899 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:50.728206 21772 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:50.731498 12639 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:50.735054 21930 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:15:50.722469 21643 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:50.736467 21643 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.735450 22042 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:50.736846 21643 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:50.737179 22042 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:50.737848 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46177us
I20260709 12:15:50.739799 22042 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:50.740374 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:50.740221 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.722169 22097 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:50.741359 12496 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:50.749907 22010 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:50.751101 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:50.752597 22023 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:50.753077 22023 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.753448 22023 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:50.755749 22029 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:15:50.755446 22087 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:50.757047 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 164482us
I20260709 12:15:50.722280 21789 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:50.759615 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.767182 12642 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:50.768100 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.766642 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.772961 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:50.773921 13257 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:50.772568 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:50.778579 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:15:50.766626 21789 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:50.783607 21789 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:50.790118 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:50.764004 22097 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:50.800827 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55977.3us
I20260709 12:15:50.770107 22076 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:50.760917 22123 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:15:50.839919 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:50.835633 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 234535us
I20260709 12:15:50.847199 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 3894) took 2938 ms. Trace:
I20260709 12:15:50.847568 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:50.848819 12567 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:50.850728 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:50.852525 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50959.7us
I20260709 12:15:50.854029 22086 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:50.847815 22125 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.243s	user 0.005s	sys 0.000s
I20260709 12:15:50.885533 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:50.890178 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:50.897233 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 122621us
I20260709 12:15:50.912454 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:50.914263 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 179491us
I20260709 12:15:50.847970 12570 rpcz_store.cc:276] 0709 12:15:47.808131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.075013 (+266882us) service_pool.cc:224] Handling call
0709 12:15:48.078112 (+  3099us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:48.078711 (+   599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:48.078759 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:48.119140 (+ 40381us) log.cc:844] Serialized 52 byte log entry
0709 12:15:48.257175 (+138035us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:49.710160 (+1452985us) log.cc:844] Serialized 14 byte log entry
0709 12:15:50.107518 (+397358us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b1800277420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:15:50.656636 (+549118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:50.674860 (+ 18224us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:50.690206 (+ 15346us) raft_consensus.cc:1685] finished
0709 12:15:50.706588 (+ 16382us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:50.746491 (+ 39903us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1346432,"prepare.queue_time_us":1419883,"prepare.run_cpu_time_us":191,"prepare.run_wall_time_us":21696,"raft.queue_time_us":10478,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":31493,"spinlock_wait_cycles":9846656,"thread_start_us":577444,"threads_started":3,"wal-append.queue_time_us":1173163,"wal-append.run_cpu_time_us":2599,"wal-append.run_wall_time_us":1031838}
I20260709 12:15:50.884539 22029 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:50.897454 21931 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:15:50.903321 22086 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:15:50.906989 22013 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:50.899482 22076 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.913210 22123 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:15:50.925935 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72335us
I20260709 12:15:50.893882 21930 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.029s
I20260709 12:15:50.926625 22086 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:50.936340 12497 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:15:50.942276 12496 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:50.925046 21931 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:50.939709 22086 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:50.952353 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.965162 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:50.975610 22042 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:50.978825 22042 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.981201 22042 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:50.987231 21788 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:50.990572 12568 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:15:50.954193 22123 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:15:50.998404 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.001202 21881 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:51.002589 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78472.2us
I20260709 12:15:51.007985 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.003072 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:51.003590 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:51.011660 22008 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:51.012151 22008 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:51.012037 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:51.012679 22008 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:51.012902 12493 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:51.014091 22129 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:15:51.015528 22076 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:51.019893 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:51.020838 22123 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:15:51.023114 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:51.005807 21971 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.017454 12492 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:15:51.004516 22136 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:15:50.942214 22029 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.052178 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:51.023207 22076 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:51.057884 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:51.058916 12571 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:51.056914 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:51.068383 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:51.069156 13254 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:51.064265 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:51.070667 13258 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:15:51.082011 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:51.082684 21643 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.084071 12496 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:51.085121 21643 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:51.085788 22029 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.086275 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:51.087682 21971 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:51.087937 21931 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.087680 12571 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:51.094799 22099 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.098132 22042 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:50.963763 21922 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:50.969183 21789 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.039888 22123 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:50.950259 22097 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.114850 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:51.115978 21922 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:51.111714 22123 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:51.124950 12566 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: [27.20-27.20]
I20260709 12:15:51.131425 12641 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 21. Current term is 23. Ops: []
W20260709 12:15:51.141443 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.144750 12491 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:51.113216 22011 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:51.120699 21856 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.156659 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:51.157279 21856 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:51.158389 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 193475us
I20260709 12:15:51.166249 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:51.166836 22141 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:51.167171 21789 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:51.167519 22141 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:51.158846 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.170059 22008 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.171277 22023 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.173697 22087 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.167016 12569 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:51.168130 22141 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:51.131322 21931 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:51.125360 22123 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:51.158398 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:51.118386 21933 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.191555 12572 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:51.168155 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:51.194619 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.146751 21922 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:51.203702 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 309478us
I20260709 12:15:51.218019 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.220291 21930 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:51.220356 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.222414 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:51.192746 22008 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.223773 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:51.231324 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 253422us
I20260709 12:15:51.243769 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:51.243974 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:51.244705 12563 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:51.248314 12570 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 11. Preceding OpId from leader: term: 31 index: 12. (index mismatch)
I20260709 12:15:51.204221 22087 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:51.248281 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:51.238965 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:51.260260 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:51.261302 12493 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:51.230805 22010 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:51.251027 21930 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:51.256559 21740 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:51.247620 21881 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:51.238227 21922 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.264966 21921 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:51.316691 21921 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:51.317072 21921 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:51.273988 12643 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 24. Current term is 25. Ops: [24.15-24.15]
W20260709 12:15:51.274029 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106045us
I20260709 12:15:51.299944 12646 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:15:51.300653 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:51.313241 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:51.314291 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94090.2us
I20260709 12:15:51.293584 12637 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:15:51.292068 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 23.
I20260709 12:15:51.297202 12461 leader_election.cc:400] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
I20260709 12:15:51.330529 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:51.331566 12566 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:51.334204 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:51.335182 12495 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:51.339865 12569 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
W20260709 12:15:51.304769 21978 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.784s	user 0.014s	sys 0.000s
I20260709 12:15:51.274150 22010 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:51.291693 22023 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:51.360164 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.274684 21930 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:51.286414 22099 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.293947 21740 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:51.295706 21922 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:51.372656 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:51.373483 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:51.373507 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:51.378005 12490 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:51.378942 12497 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:15:51.379031 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81989.4us
I20260709 12:15:51.381008 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:51.378450 12565 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:51.382831 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:51.383981 12489 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:51.386754 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:51.297395 22158 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
I20260709 12:15:51.394833 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:51.398571 21930 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.403445 21933 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:51.406678 21632 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:15:51.407145 21632 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:51.407989 21632 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.409240 21740 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 26.21, Last appended by leader: 21, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.409842 21922 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:15:51.410637 21922 consensus_queue.cc:1243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:15:51.411180 21922 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:51.411586 21922 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:51.412416 21922 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.378875 22008 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:15:51.414079 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:51.416657 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:51.430696 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.417004 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:51.430970 12493 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:15:51.422106 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77299.3us
I20260709 12:15:51.409471 21632 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:51.413560 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:51.443099 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:51.423116 12566 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:51.478394 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:51.413601 21922 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:51.481572 22155 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:51.487903 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.489816 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.493711 22123 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.496933 21881 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:51.420106 22131 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:15:51.506682 12563 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:51.434731 22158 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:51.427384 21899 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:51.424510 21978 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:15:51.530717 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 150997us
I20260709 12:15:51.485059 21931 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:51.420295 22023 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.094s
I20260709 12:15:51.459232 22076 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:15:51.459468 22086 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.520658 21899 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:51.426764 22141 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:51.627048 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.416843 21788 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.104s
I20260709 12:15:51.627749 12637 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:51.417714 21533 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.103s
W20260709 12:15:51.633103 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:51.523882 22155 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:51.540423 22099 raft_consensus.cc:2764] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:15:51.667564 21978 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:51.668217 21978 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.671062 21978 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:51.675830 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 257282us
I20260709 12:15:51.680665 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:51.545141 22156 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:51.686966 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 208708us
W20260709 12:15:51.704212 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 225599us
W20260709 12:15:51.707413 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 184617us
I20260709 12:15:51.545517 22149 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.169s
W20260709 12:15:51.702767 22126 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:15:51.640518 21931 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.650347 22148 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.070s
I20260709 12:15:51.766038 22148 consensus_queue.cc:1243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 24.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:15:51.770480 22165 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:15:51.775343 22086 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.777455 22123 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.778239 22123 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:51.778800 22123 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.781244 22123 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.705888 22154 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:51.787184 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.758811 22130 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:51.758663 21921 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.760232 22008 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:51.760241 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 338613us
I20260709 12:15:51.762513 22155 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:51.790061 22151 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:15:51.796234 22013 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:51.746481 21643 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:51.802815 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:51.806828 22123 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:51.806119 22154 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:51.805496 22086 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:51.808957 22130 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:51.842451 21643 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:51.843135 22141 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:51.895344 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105876us
I20260709 12:15:51.808625 22008 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.825047 21533 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:15:51.845115 22123 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:51.851085 22154 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:51.916625 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60320us
I20260709 12:15:51.924279 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:51.908331 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 406023us
I20260709 12:15:51.933856 13262 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:15:51.944113 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 454389us
I20260709 12:15:51.945030 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.947520 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.891731 22086 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.931361 21533 consensus_queue.cc:1243] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:15:51.943302 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:51.956475 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:51.957481 12570 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:51.959765 22180 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:51.960556 22180 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:51.960642 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.961884 21930 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:51.962728 21930 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:51.963594 21930 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:15:51.965837 22174 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:15:51.962829 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:51.967996 22162 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:51.970459 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:51.985105 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:51.991122 22013 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:51.991603 22013 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:51.991943 22013 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:51.985733 21931 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.986052 22154 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.979882 22180 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:51.995097 21921 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:51.990788 22099 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
I20260709 12:15:51.985188 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:51.997161 12497 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:52.002990 12563 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: [27.20-27.20]
I20260709 12:15:51.997030 22008 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:52.010013 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:52.005575 22176 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:52.014884 22176 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:51.950630 21643 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.007138 21632 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:15:51.982728 22010 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.608s	user 0.011s	sys 0.002s
I20260709 12:15:52.017599 22193 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:52.021276 12570 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
W20260709 12:15:52.024238 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:52.020289 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49938.5us
I20260709 12:15:52.019217 21643 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:52.046905 22013 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.049532 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:52.050701 22013 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.051638 12489 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 21. Preceding OpId from leader: term: 27 index: 22. (index mismatch)
W20260709 12:15:52.056910 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.057411 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:52.057811 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:52.058235 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:52.059420 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:52.060377 13255 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:15:52.061798 21899 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.063355 21899 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:52.066913 22184 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:15:52.067613 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 125342us
W20260709 12:15:52.069347 22173 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.314s	user 0.001s	sys 0.000s
I20260709 12:15:52.073863 22010 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.076404 22176 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:52.053361 22164 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.368s	user 0.000s	sys 0.001s
I20260709 12:15:52.083767 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:52.085678 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47403.9us
I20260709 12:15:52.086453 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.087895 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.104053 12572 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:52.089164 12564 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 21. Preceding OpId from leader: term: 27 index: 22. (index mismatch)
W20260709 12:15:52.038354 21922 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.001s
I20260709 12:15:52.098102 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:15:52.098910 22187 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260709 12:15:52.093286 22163 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.321s	user 0.000s	sys 0.005s
W20260709 12:15:52.117754 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.120002 12492 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:52.123777 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.133466 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:51.979594 22130 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.125259 22013 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:52.139583 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.086254 22193 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.085685 22123 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.171180 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:52.146749 22008 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:15:52.147981 21931 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:52.136720 22158 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:52.211900 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 172976us
I20260709 12:15:52.212779 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:52.190598 21881 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.024s	sys 0.015s
W20260709 12:15:52.226894 21881 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.024s	sys 0.016s
I20260709 12:15:52.227252 21881 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:52.229135 21881 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.230480 22158 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:52.238853 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.240789 22154 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:52.246484 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:52.208891 22179 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:15:52.259919 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.248965 21933 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:15:52.265749 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167760us
I20260709 12:15:52.265687 21933 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:52.278546 21933 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:52.257766 22086 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.258782 21922 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:15:52.267504 22165 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:15:52.261185 22179 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:52.274565 22188 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:52.302520 22010 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:52.304399 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:52.314530 22197 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260709 12:15:52.328699 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:52.332108 22196 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:15:52.297009 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1140) took 1236 ms. Trace:
W20260709 12:15:52.300940 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 452976us
W20260709 12:15:52.343442 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130736us
W20260709 12:15:52.346915 22199 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:15:52.348474 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.297128 22179 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:15:52.349485 13256 rpcz_store.cc:276] 0709 12:15:51.060470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:51.060956 (+   486us) service_pool.cc:224] Handling call
0709 12:15:51.104242 (+ 43286us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:51.125946 (+ 21704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:51.125980 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:51.161985 (+ 36005us) log.cc:844] Serialized 52 byte log entry
0709 12:15:51.457949 (+295964us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:51.508841 (+ 50892us) log.cc:844] Serialized 14 byte log entry
0709 12:15:51.509123 (+   282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:51.509235 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:51.986983 (+477748us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b1800389e60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:52.076422 (+ 89439us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b24002cfa68. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:15:52.296760 (+220338us) raft_consensus.cc:1685] finished
0709 12:15:52.296784 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:52.296944 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41924,"prepare.queue_time_us":45728,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":17612,"raft.queue_time_us":105309,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":6574,"spinlock_wait_cycles":144796160,"thread_start_us":295412,"threads_started":1,"wal-append.queue_time_us":513262}
I20260709 12:15:52.375074 13253 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:52.373330 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:52.375991 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84208.8us
I20260709 12:15:52.376560 12497 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:52.377910 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.381845 22099 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:52.384362 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.387940 21933 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.390329 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.360894 22086 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:52.391218 12497 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:52.374452 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.394721 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.395553 22010 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:15:52.396293 22010 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:15:52.397723 22010 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:15:52.401126 12497 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:15:52.406167 21856 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:52.406762 21856 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.415683 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:52.416393 12640 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:52.417569 21933 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.419003 21856 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:52.421042 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.422896 22086 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:52.425197 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.426455 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:52.427824 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:52.430796 12567 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:52.431681 22086 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.432273 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:52.434087 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:52.407770 22099 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:52.407975 22162 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:52.411067 22205 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
W20260709 12:15:52.438655 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54395us
I20260709 12:15:52.455168 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:52.423254 21899 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:52.471724 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50732.9us
I20260709 12:15:52.476619 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:52.484062 12609 leader_election.cc:400] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24.
I20260709 12:15:52.482791 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:52.485319 12572 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:15:52.440704 22023 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:52.491827 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.477384 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.493194 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:52.496665 12495 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:15:52.497459 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:52.495484 12646 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:15:52.501098 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:52.452531 22013 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:52.457805 22193 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.509977 22042 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:52.510715 22042 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:52.505122 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:52.457494 22189 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:52.458767 22097 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:15:52.506441 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:52.503217 22013 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.508769 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.512636 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.502058 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.513003 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:52.518146 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.523772 12639 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:52.523504 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:52.529498 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:52.531646 12646 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:52.441526 22038 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.063s
W20260709 12:15:52.537246 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.538846 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.539520 12644 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:15:52.482771 21899 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.542141 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.543382 12645 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
W20260709 12:15:52.543795 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71138us
I20260709 12:15:52.549226 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:52.538704 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0474320029998125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
W20260709 12:15:52.549549 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52342.5us
I20260709 12:15:52.551151 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.551966 12638 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:52.553059 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:52.535929 22097 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:52.540469 21772 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:15:52.561005 12572 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:52.554828 12638 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:52.505239 22023 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.564678 22013 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.565402 22177 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:52.566179 22177 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.571871 22023 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:52.574561 22042 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.575873 22177 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:52.577069 22208 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:15:52.546059 21740 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:52.547986 22155 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:52.578569 22086 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.581573 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.586262 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.585189 22189 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:52.588907 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:52.589184 12645 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:52.628153 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44764.5us
I20260709 12:15:52.637054 12572 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:15:52.635543 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:52.585462 21740 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:52.638732 21740 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:52.639652 22097 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.641258 21971 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:52.641916 21971 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:52.651727 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.658708 21971 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.665659 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:52.666415 21899 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:52.667781 21921 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:52.667470 21899 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:52.668411 21921 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:52.668905 21921 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:52.670523 12645 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:52.671573 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.672221 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
W20260709 12:15:52.673951 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.675400 12461 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:52.675138 22179 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.669297 21899 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:52.733721 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.768826 21899 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:52.769029 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:52.770141 12645 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:52.772377 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97162.9us
I20260709 12:15:52.774333 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:52.774669 12640 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:52.776635 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:52.824841 12641 raft_consensus.cc:2368] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
W20260709 12:15:52.827858 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 246349us
W20260709 12:15:52.828917 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 247081us
I20260709 12:15:52.835114 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.861655 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:52.863171 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.829206 21899 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.864595 21899 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:52.865867 22205 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:52.867651 22179 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:52.874523 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:52.876602 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.877205 12570 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:52.877558 12565 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:52.879514 21921 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.880522 12566 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: [30.17-30.17]
I20260709 12:15:52.881949 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.882752 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:52.883982 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.881949 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:52.886122 12645 raft_consensus.cc:2368] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 24.
I20260709 12:15:52.887313 12645 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:52.888793 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:52.889492 12569 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:52.825378 22177 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:52.894089 22177 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:52.895028 22177 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:52.896741 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.824007 22076 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:52.898317 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.899080 22181 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:52.899530 22181 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:52.899681 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:52.899958 22181 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:52.900439 12572 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 23: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:52.879417 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.902282 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.902513 21921 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:52.902961 12566 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:15:52.904310 12564 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:52.906555 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:52.907614 12566 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:15:52.909762 12564 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:15:52.914472 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:52.917560 12569 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:15:52.947484 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:52.948549 12645 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:52.948889 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.950806 22086 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:52.951476 22086 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:52.984680 12638 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:15:52.986974 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:52.987160 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72821.7us
I20260709 12:15:52.987612 12572 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:15:52.988096 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.993866 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:52.995376 22189 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:52.995838 22189 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:52.996178 22189 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:53.001904 22076 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:53.004599 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.007851 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:53.009327 12640 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:52.990952 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.014253 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.002477 22076 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.018396 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:53.022539 22162 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:53.023541 22011 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:53.024926 22202 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.694s	user 0.011s	sys 0.034s
I20260709 12:15:53.020174 21856 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:53.003983 22179 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:53.029963 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.034404 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:15:53.009302 22013 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:15:53.016397 22154 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.758s	user 0.010s	sys 0.005s
I20260709 12:15:53.015635 22205 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:15:53.070678 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79824us
I20260709 12:15:53.021459 22076 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:53.071218 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.072721 22179 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:53.085970 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:15:53.091872 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 317633us
I20260709 12:15:53.105072 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4085) took 1121 ms. Trace:
I20260709 12:15:53.105576 12493 rpcz_store.cc:276] 0709 12:15:51.983635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:52.002071 (+ 18436us) service_pool.cc:224] Handling call
0709 12:15:52.003003 (+   932us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:52.090164 (+ 87161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:52.090220 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:52.099038 (+  8818us) log.cc:844] Serialized 52 byte log entry
0709 12:15:52.209550 (+110512us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:52.209649 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:52.209900 (+   251us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:53.024026 (+814126us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:53.096281 (+ 72255us) raft_consensus.cc:1685] finished
0709 12:15:53.096306 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:53.104977 (+  8671us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80332,"raft.queue_time_us":86862,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":19060,"spinlock_wait_cycles":23275776,"thread_start_us":109957,"threads_started":1,"wal-append.queue_time_us":183160}
I20260709 12:15:53.087248 22154 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:53.077497 21740 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.096035 22162 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:53.125056 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.088572 22205 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:15:53.136546 22179 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:15:53.142198 22011 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:53.143832 21740 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.128788 22162 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.135748 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:53.163149 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89547us
I20260709 12:15:53.164528 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.153076 22205 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:53.160044 22162 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.158691 22221 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:53.162912 22179 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:15:53.192342 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.193157 12572 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:53.193899 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1305) took 1948 ms. Trace:
I20260709 12:15:53.194083 22155 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:53.194438 12492 rpcz_store.cc:276] 0709 12:15:51.245032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:51.245156 (+   124us) service_pool.cc:224] Handling call
0709 12:15:51.245493 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:52.124285 (+878792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:53.193561 (+1069276us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:53.193816 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2010494848}
I20260709 12:15:53.197554 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:53.198199 22221 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.201022 22221 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:53.202067 22221 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:53.202484 22221 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:53.168972 22205 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:53.202853 22221 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.228018 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.209764 22179 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:53.231518 22097 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:53.233863 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:15:53.233194 22224 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:53.242624 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.249926 12564 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:15:53.246728 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 153747us
I20260709 12:15:53.262313 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:53.265283 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100902us
I20260709 12:15:53.266430 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.264652 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:53.247725 22011 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:53.271521 22013 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:53.299889 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:53.279072 22226 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:53.304415 12571 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:15:53.260972 22205 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.327555 12566 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:53.275053 22097 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.332018 22097 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:53.333210 22123 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.334792 12492 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:53.335708 12496 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:53.344949 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.356668 12564 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:53.264045 22224 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.294538 22177 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:53.314535 22192 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260709 12:15:53.346422 21856 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:15:53.293228 22181 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.317864 22231 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:53.378629 12567 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:15:53.381500 12563 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:53.295786 22076 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.365207 22154 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.374040 22224 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.369186 12570 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:15:53.389207 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71833us
I20260709 12:15:53.367269 22177 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:53.319398 22189 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:53.402668 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.301585 22179 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.415566 22189 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.400902 22177 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:53.423766 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.428761 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.434645 12570 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:53.443565 12563 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:53.392176 22232 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:15:53.446640 22076 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.449198 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.449939 12563 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:53.452790 22086 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.454916 22013 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.455673 22013 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.454921 22181 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:53.458405 22198 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.248s	user 0.019s	sys 0.028s
I20260709 12:15:53.396591 22154 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.485687 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.491811 12568 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:15:53.418689 22234 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.089s
I20260709 12:15:53.453133 22221 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:53.490902 22225 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:15:53.502091 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:53.505379 12568 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:53.505905 22207 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:15:53.525352 22207 consensus_queue.cc:1243] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 27.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:15:53.525619 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1296) took 1526 ms. Trace:
I20260709 12:15:53.502702 22220 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.526657 22220 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.524107 21856 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:53.527899 21856 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.529111 22220 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.526125 12637 rpcz_store.cc:276] 0709 12:15:51.998871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:51.999036 (+   165us) service_pool.cc:224] Handling call
0709 12:15:52.011164 (+ 12128us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:52.013224 (+  2060us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:52.013255 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:52.013782 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:15:52.085535 (+ 71753us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:15:52.105680 (+ 20145us) log.cc:844] Serialized 14 byte log entry
0709 12:15:52.257097 (+151417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:52.257199 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:53.457692 (+1200493us) spinlock_profiling.cc:243] Waited 5.79 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:53.506803 (+ 49111us) raft_consensus.cc:1685] finished
0709 12:15:53.525385 (+ 18582us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:53.525548 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38655,"prepare.queue_time_us":19499,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":50,"raft.queue_time_us":54661,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":11456,"spinlock_wait_cycles":13366784,"thread_start_us":185226,"threads_started":2,"wal-append.queue_time_us":124265}
I20260709 12:15:53.524127 21740 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.524197 22189 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 12:15:53.532692 22154 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:53.530875 21740 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.534160 21856 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:53.541026 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.541888 12569 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:53.542901 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.543665 12569 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:53.546550 22205 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:53.541821 21740 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.556970 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:53.523564 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:53.559433 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52422.6us
I20260709 12:15:53.561286 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.562520 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:53.569818 21740 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:53.571983 22232 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.569819 22013 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.574937 22232 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.574926 21740 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.579784 21740 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.565128 21856 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:15:53.504262 22162 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.565744 21921 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:53.570582 22154 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:15:53.594359 22236 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:53.595119 22236 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:53.596220 22236 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.601115 22162 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.602213 21921 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.604602 21921 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:53.605003 12569 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:15:53.608414 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:53.614441 22233 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:15:53.619861 12460 leader_election.cc:400] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:15:53.620354 12460 leader_election.cc:403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:15:53.621145 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:53.622498 21899 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:53.623070 22232 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.624740 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.625887 22010 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:53.625219 22042 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:53.635931 22013 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:53.636461 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.629801 21899 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:53.637830 21899 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:53.637876 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.638506 21899 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:53.638906 22231 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:53.639389 22231 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.639981 22231 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.631966 22183 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:53.637398 22013 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:53.641979 22013 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.642397 22013 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.647163 22221 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:53.647485 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.650415 21643 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.651309 21643 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.662957 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:53.671136 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:15:53.675096 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.677142 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.677966 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.683640 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:53.684535 12566 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:15:53.684610 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.685060 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:53.685571 12496 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:53.635301 22177 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.687153 13256 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:53.678783 21856 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:53.694145 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:53.695006 13262 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:15:53.695055 21856 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.682000 22154 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:53.696985 22154 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.698479 22013 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.700626 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:53.714880 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.715916 12564 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.722606 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:53.723467 12497 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:53.683030 22097 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:53.726935 22097 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.727362 22097 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:53.700796 21643 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.730890 12566 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: []
I20260709 12:15:53.730700 22086 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:53.740339 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.741281 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
W20260709 12:15:53.742847 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.744333 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:53.752200 12497 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:15:53.751653 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:53.707875 21933 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:53.771337 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.672569 22130 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:15:53.775308 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:53.775857 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.781111 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.767203 13262 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:15:53.783519 12492 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:53.778273 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:53.786985 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.788021 12497 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:53.787488 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.791093 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.792572 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.734338 22086 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.798303 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.721482 22180 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:15:53.761550 22008 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.822964 22008 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.797667 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.770407 22076 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.803515 12644 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:53.799333 12642 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:15:53.731400 21931 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:15:53.788802 21933 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.778899 21533 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:15:53.758028 22013 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.833155 12496 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:15:53.835019 21933 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:53.837797 22235 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.268s	user 0.001s	sys 0.001s
I20260709 12:15:53.838550 22042 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:15:53.819977 22236 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:53.820883 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:53.842693 12495 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:53.843794 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.826707 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.845003 22076 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.846031 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:53.847551 22076 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.802898 12639 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:15:53.740720 22189 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.061s
W20260709 12:15:53.827481 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:53.802191 22086 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.826118 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.845074 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.800732 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.858196 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.803164 22221 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:53.859422 22221 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:53.859824 22221 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:53.864234 21740 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:15:53.865171 21740 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:53.830247 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.866178 22201 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:15:53.866852 22201 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:53.867205 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.867803 22201 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.868585 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.830639 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.869643 22130 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.852792 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.870472 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.870223 22130 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.830320 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.831329 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.771116 22237 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:53.849550 12639 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 14
I20260709 12:15:53.872308 22237 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.872565 12639 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:15:53.873361 12639 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:53.874449 22177 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.874711 22237 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:53.875876 22201 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:15:53.878291 22130 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.882529 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:53.884729 12496 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:15:53.887068 12640 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:53.887228 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:53.759989 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 322598us
I20260709 12:15:53.894076 21899 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.855207 22180 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:53.897092 22180 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:53.897958 12495 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:53.898316 22180 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.899717 22180 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:53.900197 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:53.796458 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:53.903162 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.907998 22008 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:53.911594 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:53.825353 22231 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:53.914991 22176 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:53.916062 12643 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:15:53.916366 22038 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:53.918509 21740 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:53.919551 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:53.920275 21740 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:53.920550 21933 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:53.920967 21933 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:53.921381 21933 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:53.924448 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.926359 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:53.927809 22130 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:53.928392 22130 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:53.930313 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144352us
I20260709 12:15:53.794766 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:53.928809 22130 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:53.946245 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:53.948141 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.951084 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:53.953492 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.004s	sys 0.026s
W20260709 12:15:53.954053 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.004s	sys 0.026s
I20260709 12:15:53.897173 21899 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.914368 22231 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:53.952476 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:53.961513 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:53.980739 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:53.990132 22249 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.005s
I20260709 12:15:54.178817 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:54.036062 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 364600us
I20260709 12:15:54.035183 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.020279 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.181006 12642 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:54.173316 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.175186 12489 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:15:54.176067 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:15:54.030782 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:53.963253 22232 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.095s
I20260709 12:15:54.019250 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.173164 22239 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.184180 12645 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:15:54.182233 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:54.237702 12637 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:54.228185 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:54.225617 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:54.241849 12492 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:15:54.242033 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.242789 12642 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:54.178933 21881 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:54.267891 12644 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:15:54.184219 22179 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.267132 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.182796 22207 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:54.272725 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90582.9us
I20260709 12:15:54.280741 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:54.281383 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:54.278173 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:54.282990 12645 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 10. Preceding OpId from leader: term: 31 index: 12. (index mismatch)
I20260709 12:15:54.283192 12644 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 16
W20260709 12:15:54.244534 22205 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.696s	user 0.008s	sys 0.007s
I20260709 12:15:54.284574 12644 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:15:54.287387 12644 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:54.241374 22237 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.224098 22097 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.235168 22241 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.290253 22179 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.302358 12564 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:15:54.308005 12642 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:54.312145 12641 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:54.313747 12536 leader_election.cc:400] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 24.
I20260709 12:15:54.314864 12638 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:15:54.267127 22261 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.273558 22251 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.339403 12536 leader_election.cc:403] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:15:54.339246 12641 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:15:54.286960 21922 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.306133 22224 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:54.244189 22239 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.292812 22221 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:54.368498 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90454.5us
W20260709 12:15:54.385741 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:54.385728 12569 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:54.397672 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.457991 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:54.456333 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78761.3us
I20260709 12:15:54.453871 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:54.337047 22255 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:54.296267 22207 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.512516 12641 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:54.320835 22237 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:54.513509 12492 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:15:54.457162 22011 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 24.
I20260709 12:15:54.396833 22224 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.519232 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.345098 22241 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.388118 22251 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.507305 22183 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:15:54.498184 22191 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:15:54.498996 22255 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:54.497807 22205 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.065s
W20260709 12:15:54.533802 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 129) took 34311 ms (client timeout 15000 ms). Trace:
W20260709 12:15:54.531395 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 217780us
I20260709 12:15:54.521394 12641 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:15:54.361500 22261 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.527104 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:15:54.611536 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:54.618031 12536 leader_election.cc:400] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
I20260709 12:15:54.618603 12536 leader_election.cc:403] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:54.513262 21922 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:54.525898 22245 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
W20260709 12:15:54.531011 22252 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
W20260709 12:15:54.528981 22264 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
W20260709 12:15:54.555594 22180 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.649s	user 0.011s	sys 0.005s
I20260709 12:15:54.623684 22011 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:54.672003 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:54.553031 22236 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.711s	user 0.012s	sys 0.001s
I20260709 12:15:54.540434 22241 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:54.610631 22201 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.734s	user 0.009s	sys 0.006s
I20260709 12:15:54.634157 12637 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:15:54.633118 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:54.689539 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:54.541903 22224 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:54.531073 22179 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:54.629662 22220 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.546151 22251 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:54.594415 22097 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:54.558466 22255 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:54.549314 22230 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.685281 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:54.790596 12492 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:54.687122 12489 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:54.609552 22191 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:54.696282 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.697047 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.697994 13254 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:54.624964 22042 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:54.794378 12637 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:15:54.700613 13257 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:15:54.552948 22181 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.705377 12640 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:54.700371 12490 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
W20260709 12:15:54.714907 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 486769us
I20260709 12:15:54.775245 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:54.776525 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:54.776355 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.780654 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:54.640575 22177 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:54.795768 12488 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.16-25.16]
W20260709 12:15:54.809293 21933 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.887s	user 0.019s	sys 0.003s
I20260709 12:15:54.810868 12494 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:54.812774 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:15:54.797153 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115010us
I20260709 12:15:54.833979 12645 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:15:54.807355 22251 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:15:54.773610 22179 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:54.799610 12642 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260709 12:15:54.840159 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 386332us
I20260709 12:15:54.866348 12572 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
W20260709 12:15:54.836410 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 218433us
W20260709 12:15:54.854878 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:54.854188 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:54.857363 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.872604 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.835352 22097 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
I20260709 12:15:54.877151 12571 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:54.877899 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:54.794291 22220 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:54.840342 22177 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:54.605309 12392 rpcz_store.cc:269] 0709 12:15:19.975577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:19.975873 (+   296us) service_pool.cc:224] Handling call
0709 12:15:20.412344 (+436471us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:54.279482 (+33867138us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:15:48.291064 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:15:48.329305 (+ 38241us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:15:48.329347 (+    42us) write_op.cc:435] Acquired schema lock
0709 12:15:48.329365 (+    18us) tablet.cc:662] Decoding operations
0709 12:15:48.330704 (+  1339us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:15:48.330761 (+    57us) write_op.cc:641] Partition lock acquired for write op
0709 12:15:48.330790 (+    29us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:15:48.642313 (+311523us) tablet.cc:701] Row locks acquired
0709 12:15:48.642351 (+    38us) write_op.cc:260] PREPARE: finished
0709 12:15:48.642541 (+   190us) write_op.cc:270] Start()
0709 12:15:48.642682 (+   141us) write_op.cc:276] Timestamp: P: 1783599348642477 usec, L: 0
0709 12:15:48.642705 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:15:48.756001 (+113296us) log.cc:844] Serialized 5962 byte log entry
0709 12:15:50.046419 (+1290418us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:15:50.249469 (+203050us) spinlock_profiling.cc:243] Waited 8.86 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eee28 00007ffbf9229581 00007ffbf9228d58 00007ffbf922d9f1 00007ffbf922d9a9 00007ffbf922d939 00007ffbf922d901 00007ffbf922cb2d 0000000000388634 0000000000388479
0709 12:15:51.072078 (+822609us) op_driver.cc:464] REPLICATION: finished
0709 12:15:51.278032 (+205954us) write_op.cc:301] APPLY: starting
0709 12:15:51.278183 (+   151us) tablet.cc:1367] starting BulkCheckPresence
0709 12:15:51.757135 (+478952us) tablet.cc:1370] finished BulkCheckPresence
0709 12:15:51.757205 (+    70us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:15:52.079196 (+321991us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:15:52.079570 (+   374us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=0,mrs_lookups=0
0709 12:15:52.079631 (+    61us) write_op.cc:312] APPLY: finished
0709 12:15:52.080164 (+   533us) log.cc:844] Serialized 139 byte log entry
0709 12:15:52.278145 (+197981us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:52.278631 (+   486us) write_op.cc:454] Released schema lock
0709 12:15:52.278741 (+   110us) write_op.cc:341] FINISH: Updating metrics
0709 12:15:52.279556 (+   815us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:15:52.279600 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1694266,"reported_tablets":26,"spinlock_wait_cycles":412461312,"child_traces":[["op",{"apply.queue_time_us":205487,"apply.run_cpu_time_us":40676,"apply.run_wall_time_us":1001481,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66767,"num_ops":15,"prepare.queue_time_us":207685,"prepare.run_cpu_time_us":31397,"prepare.run_wall_time_us":1968882,"raft.queue_time_us":424744,"raft.run_cpu_time_us":3779,"raft.run_wall_time_us":179151,"replication_time_us":2402617,"spinlock_wait_cycles":128180352,"thread_start_us":2135044,"threads_started":5,"wal-append.queue_time_us":1995956,"wal-append.run_cpu_time_us":58067,"wal-append.run_wall_time_us":2465282}]]}
I20260709 12:15:54.861217 22192 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:54.900045 12642 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:15:54.902987 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:54.927711 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:54.821683 22181 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:54.812705 22230 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:54.954840 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3768) took 1142 ms. Trace:
I20260709 12:15:54.934515 12645 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:54.955382 12638 rpcz_store.cc:276] 0709 12:15:53.790480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.790666 (+   186us) service_pool.cc:224] Handling call
0709 12:15:54.932405 (+1141739us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:54.932550 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1201513344}
I20260709 12:15:54.878377 22251 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:54.899200 22177 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:15:54.903293 22274 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
W20260709 12:15:54.900974 22266 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 12:15:54.957897 22262 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.483s	user 0.000s	sys 0.002s
I20260709 12:15:54.878159 22011 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:54.894963 22286 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.885977 22239 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:54.903453 22201 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.966730 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.966996 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:54.854916 21933 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:54.965927 22184 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:54.911109 12492 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:54.834122 22042 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:54.968770 12461 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.523s (SENT)
I20260709 12:15:54.903436 22280 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:54.956526 22086 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.105s	user 0.016s	sys 0.000s
W20260709 12:15:54.989507 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121787us
I20260709 12:15:54.999257 12494 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:54.902590 22261 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:54.963253 22207 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:54.879372 22179 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:54.898844 21922 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:55.020874 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.009s	sys 0.012s
W20260709 12:15:55.074344 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.009s	sys 0.012s
I20260709 12:15:55.015726 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.012804 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:15:55.023756 12488 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:54.897866 22191 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:55.088922 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120204us
I20260709 12:15:55.087491 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.001327 22011 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:55.089784 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.547s (SENT) [suppressed 13 similar messages]
I20260709 12:15:55.092020 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:55.020617 21899 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.009s	sys 0.010s
W20260709 12:15:55.092092 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 393394us
I20260709 12:15:55.139357 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:54.968931 22183 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.136s
W20260709 12:15:55.151206 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63820.6us
W20260709 12:15:55.018461 22270 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:15:55.149286 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:55.132997 22280 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:55.148614 21899 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.012s	sys 0.013s
I20260709 12:15:55.086983 22261 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:55.087492 22239 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:55.083395 22205 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:55.016052 22201 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.157196 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.116323 21922 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:55.013034 22029 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.141s
I20260709 12:15:55.169312 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.172089 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.168805 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3771) took 1217 ms. Trace:
W20260709 12:15:55.164685 22266 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:15:55.021430 22097 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.153s
I20260709 12:15:55.021476 22184 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.190233 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.014045 22286 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.090983 22086 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.191139 12645 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:15:55.189477 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3109) took 1352 ms. Trace:
I20260709 12:15:55.195806 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1336) took 1019 ms. Trace:
W20260709 12:15:55.197227 22130 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.174s	user 0.008s	sys 0.004s
W20260709 12:15:55.209101 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:55.223658 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:55.228259 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:55.229805 22280 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:55.180400 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.215461 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.232673 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:15:55.233330 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.234068 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:55.235133 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.235847 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.236852 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:55.237795 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:55.238449 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.239995 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:55.220646 12490 rpcz_store.cc:276] 0709 12:15:53.837019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.869566 (+ 32547us) service_pool.cc:224] Handling call
0709 12:15:55.189279 (+1319713us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:55.189406 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1782590336}
I20260709 12:15:55.221381 12495 rpcz_store.cc:276] 0709 12:15:54.176796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.176933 (+   137us) service_pool.cc:224] Handling call
0709 12:15:54.225122 (+ 48189us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:54.610958 (+385836us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:54.611155 (+   197us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:54.616896 (+  5741us) log.cc:844] Serialized 26 byte log entry
0709 12:15:54.814360 (+197464us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:15:54.814401 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.814478 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.195516 (+381038us) raft_consensus.cc:1685] finished
0709 12:15:55.195539 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.195725 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68087,"raft.queue_time_us":150868,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":58894,"thread_start_us":378829,"threads_started":2,"wal-append.queue_time_us":316826}
I20260709 12:15:55.178117 12639 rpcz_store.cc:276] 0709 12:15:53.951574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.951957 (+   383us) service_pool.cc:224] Handling call
0709 12:15:53.952599 (+   642us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:54.181023 (+228424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:54.181061 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:54.223603 (+ 42542us) log.cc:844] Serialized 26 byte log entry
0709 12:15:54.453269 (+229666us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:15:54.453335 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.453451 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.168510 (+715059us) raft_consensus.cc:1685] finished
0709 12:15:55.168534 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.168709 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11812,"raft.queue_time_us":4683,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":2521,"spinlock_wait_cycles":33280,"thread_start_us":173727,"threads_started":1,"wal-append.queue_time_us":373333}
W20260709 12:15:55.195351 22267 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.502s	user 0.002s	sys 0.000s
I20260709 12:15:55.241501 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:55.202334 22273 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.347s	user 0.000s	sys 0.001s
I20260709 12:15:55.171418 22258 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:55.192957 22272 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
W20260709 12:15:55.203559 22265 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.755s	user 0.004s	sys 0.004s
I20260709 12:15:55.254999 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:55.303529 12566 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:55.162665 22261 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.200443 22237 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.296063 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.186645 22181 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:55.185923 22220 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.186990 22259 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.219350 21856 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.101s
I20260709 12:15:55.306365 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3896) took 1452 ms. Trace:
I20260709 12:15:55.307843 22299 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.310142 12638 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:55.315740 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 225948us
I20260709 12:15:55.243970 22298 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:55.322109 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 150259us
I20260709 12:15:55.172469 22239 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.176123 22205 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.185907 22287 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.204977 21922 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:55.365984 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 137829us
I20260709 12:15:55.341867 22207 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:55.331221 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.223249 22130 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.334457 12565 rpcz_store.cc:276] 0709 12:15:53.853981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.855069 (+  1088us) service_pool.cc:224] Handling call
0709 12:15:53.855494 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:53.855947 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:53.855985 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:53.856912 (+   927us) log.cc:844] Serialized 26 byte log entry
0709 12:15:53.988460 (+131548us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:54.234140 (+245680us) log.cc:844] Serialized 14 byte log entry
0709 12:15:54.272668 (+ 38528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.272753 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.306013 (+1033260us) raft_consensus.cc:1685] finished
0709 12:15:55.306047 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.306278 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38224,"prepare.queue_time_us":246361,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5826,"raft.queue_time_us":56480,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"spinlock_wait_cycles":15928192,"thread_start_us":130760,"threads_started":1,"wal-append.queue_time_us":825496}
I20260709 12:15:55.171068 22251 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.14, Last appended by leader: 14, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.370457 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4025) took 1662 ms. Trace:
I20260709 12:15:55.371062 12570 rpcz_store.cc:276] 0709 12:15:53.707787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.737464 (+ 29677us) service_pool.cc:224] Handling call
0709 12:15:53.771496 (+ 34032us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:54.168150 (+396654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:54.168187 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:54.176526 (+  8339us) log.cc:844] Serialized 26 byte log entry
0709 12:15:54.276815 (+100289us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:54.276882 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.276964 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.370105 (+1093141us) raft_consensus.cc:1685] finished
0709 12:15:55.370140 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.370367 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":62460,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":59,"thread_start_us":262488,"threads_started":2,"wal-append.queue_time_us":548396}
I20260709 12:15:55.316918 22283 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:15:55.330408 22176 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.415s	user 0.013s	sys 0.004s
I20260709 12:15:55.350977 21856 consensus_queue.cc:1243] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:15:55.348675 12461 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.653s (SENT)
I20260709 12:15:55.402415 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:55.350391 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.294900 22258 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:55.392158 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188459us
I20260709 12:15:55.404008 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.296960 22011 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.406286 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.410845 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.432535 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:55.399789 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 227587us
I20260709 12:15:55.337184 22259 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:55.408277 22267 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:15:55.438723 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4023) took 1731 ms. Trace:
I20260709 12:15:55.438869 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.454313 12571 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:55.451423 12563 rpcz_store.cc:276] 0709 12:15:53.707178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.709593 (+  2415us) service_pool.cc:224] Handling call
0709 12:15:53.709960 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:53.932652 (+222692us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:53.932688 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:53.994234 (+ 61546us) log.cc:844] Serialized 26 byte log entry
0709 12:15:54.313423 (+319189us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:54.313532 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.313691 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.438382 (+1124691us) raft_consensus.cc:1685] finished
0709 12:15:55.438416 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.438633 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13910,"raft.queue_time_us":123505,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"thread_start_us":295281,"threads_started":2,"wal-append.queue_time_us":663733}
I20260709 12:15:55.444504 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:55.444373 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.459626 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3900) took 1596 ms. Trace:
I20260709 12:15:55.399438 21933 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:55.460207 12567 rpcz_store.cc:276] 0709 12:15:53.863068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.866932 (+  3864us) service_pool.cc:224] Handling call
0709 12:15:53.867377 (+   445us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:53.867849 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:53.867892 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:53.868624 (+   732us) log.cc:844] Serialized 52 byte log entry
0709 12:15:54.074569 (+205945us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:54.074630 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.074738 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.202682 (+1127944us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:55.430772 (+228090us) raft_consensus.cc:1685] finished
0709 12:15:55.430800 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.459536 (+ 28736us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1804,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":130,"spinlock_wait_cycles":102257408,"thread_start_us":205430,"threads_started":1,"wal-append.queue_time_us":403129}
I20260709 12:15:55.464471 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.409610 22176 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:55.470938 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.473335 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.423874 21856 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:15:55.362471 22277 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.090s
I20260709 12:15:55.479296 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:55.481786 12563 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:55.483493 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.490909 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:55.349468 22299 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:55.483884 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135238us
W20260709 12:15:55.494457 12461 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.669s (SENT)
I20260709 12:15:55.495143 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:55.436939 22042 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.354473 22298 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.436652 22280 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:55.449911 22230 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.508555 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3767) took 1723 ms. Trace:
I20260709 12:15:55.489234 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:55.482312 22224 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.708s	user 0.007s	sys 0.007s
I20260709 12:15:55.520557 13254 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:15:55.509212 12646 rpcz_store.cc:276] 0709 12:15:53.785094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.785252 (+   158us) service_pool.cc:224] Handling call
0709 12:15:53.801810 (+ 16558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:53.873545 (+ 71735us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:53.873578 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:53.874133 (+   555us) log.cc:844] Serialized 26 byte log entry
0709 12:15:54.002220 (+128087us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:54.064594 (+ 62374us) log.cc:844] Serialized 14 byte log entry
0709 12:15:54.064911 (+   317us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.065016 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:55.188099 (+1123083us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x7b18004ae4e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:55.194546 (+  6447us) spinlock_profiling.cc:243] Waited 6.22 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:55.508298 (+313752us) raft_consensus.cc:1685] finished
0709 12:15:55.508324 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.508485 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":49092,"prepare.run_cpu_time_us":13639,"prepare.run_wall_time_us":116630,"raft.queue_time_us":1789,"raft.run_cpu_time_us":5290,"raft.run_wall_time_us":88850,"spinlock_wait_cycles":115115136,"thread_start_us":169479,"threads_started":2,"wal-append.queue_time_us":678680}
I20260709 12:15:55.478987 21856 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:55.495745 22299 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:55.502631 21740 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.582s	user 0.008s	sys 0.008s
I20260709 12:15:55.484560 22179 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.476943 22301 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:55.364512 22287 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.516840 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:55.637022 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:55.693313 12496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.001s	sys 0.021s
W20260709 12:15:55.715174 12496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.001s	sys 0.021s
I20260709 12:15:55.704756 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:55.711741 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:55.522823 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.716616 13255 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:15:55.717131 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 194412us
I20260709 12:15:55.505918 22184 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:55.506731 22201 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.724642 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.761049 22320 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.761215 21856 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.761749 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:55.841383 13253 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:55.762579 22307 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:55.763428 22310 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.764209 22306 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.764658 22297 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:55.712414 22281 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.403s	user 0.001s	sys 0.001s
I20260709 12:15:55.765281 22290 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:55.765767 22304 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.757961 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4103) took 1094 ms. Trace:
I20260709 12:15:55.691664 22181 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:55.843225 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:55.861779 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:55.743541 12497 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:15:55.882871 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:55.703604 22291 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:55.832119 22268 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.231s	user 0.000s	sys 0.003s
I20260709 12:15:55.917344 12496 rpcz_store.cc:276] 0709 12:15:54.663266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.691101 (+ 27835us) service_pool.cc:224] Handling call
0709 12:15:54.775757 (+ 84656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:54.776024 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:15:54.967389 (+191365us) log.cc:844] Serialized 14 byte log entry
0709 12:15:55.757521 (+790132us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:55.757591 (+    70us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:55.757674 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:55.757755 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:55.757888 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113436,"prepare.queue_time_us":532113,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":41469,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"thread_start_us":918220,"threads_started":2,"wal-append.queue_time_us":877637}
I20260709 12:15:55.712802 22299 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:55.939172 22300 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.478s	user 0.001s	sys 0.000s
I20260709 12:15:55.939415 22299 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.941042 22259 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.712069 22230 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:55.718804 22207 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:55.713969 22086 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:55.915830 13253 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:15:55.915544 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 421133us
I20260709 12:15:55.602861 22271 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.531s
I20260709 12:15:55.604110 22224 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:56.018105 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.008s	sys 0.012s
W20260709 12:15:56.042191 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.009s	sys 0.013s
W20260709 12:15:56.042688 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 599501us
I20260709 12:15:55.948858 22261 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:55.959573 22130 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:55.942381 22312 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:55.853173 22283 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:56.038648 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:55.952803 22239 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:55.949366 22205 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:55.941571 22299 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:55.958400 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:55.941689 22259 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:56.024142 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.010s	sys 0.012s
W20260709 12:15:55.937552 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:56.062927 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.946182 22230 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.064510 12495 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:15:56.065357 12639 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:15:56.066392 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.092794 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:55.948722 22298 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.078904 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:15:56.062965 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 338396us
W20260709 12:15:56.085772 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.010s	sys 0.012s
W20260709 12:15:56.098455 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 687658us
I20260709 12:15:56.085860 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.086292 12572 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:56.086344 12646 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:15:56.088689 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.100315 12642 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:15:56.102577 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:55.960261 22297 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:55.925065 22307 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:55.853951 22320 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.000015 22290 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.090590 22205 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:55.958240 22291 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.106063 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:55.960158 22310 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.092928 22181 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:56.093132 22299 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:15:56.103731 22177 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.101s	user 0.016s	sys 0.000s
I20260709 12:15:56.000262 22304 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.083112 22302 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.121352 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:56.088734 22192 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.003s	sys 0.029s
I20260709 12:15:55.925235 22306 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:56.124452 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.001s	sys 0.017s
I20260709 12:15:56.129046 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.105134 22298 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:56.138270 12567 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:56.140196 22321 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:56.140583 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.141916 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4033) took 2259 ms. Trace:
I20260709 12:15:56.107283 22261 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:15:56.124935 22328 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:56.144441 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.093565 22259 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.146082 12637 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:15:56.086627 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128374us
I20260709 12:15:56.146064 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4036) took 2205 ms. Trace:
I20260709 12:15:56.150055 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.158804 22302 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.155479 22179 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:56.158330 13258 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:15:56.165059 22277 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:56.165678 22277 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.166365 22310 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.168025 22277 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:56.155565 22192 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.004s	sys 0.029s
I20260709 12:15:56.152946 22177 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.170486 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:56.171536 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:56.172066 22177 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.172329 22220 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:56.173157 22220 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.174360 22239 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:56.174943 22239 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.175311 22239 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:56.175911 22220 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.177873 22010 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:56.179177 22302 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.179596 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.179231 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:56.170519 22261 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 12:15:56.182066 12570 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:56.182571 22304 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.183213 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.184868 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4050) took 1175 ms. Trace:
I20260709 12:15:56.162201 12568 rpcz_store.cc:276] 0709 12:15:53.940923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.228403 (+287480us) service_pool.cc:224] Handling call
0709 12:15:54.229002 (+   599us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:54.272454 (+ 43452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:54.272508 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:54.273372 (+   864us) log.cc:844] Serialized 52 byte log entry
0709 12:15:54.545949 (+272577us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:54.614981 (+ 69032us) log.cc:844] Serialized 14 byte log entry
0709 12:15:54.615243 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.615335 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:56.145749 (+1530414us) raft_consensus.cc:1685] finished
0709 12:15:56.145777 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:56.145984 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50242,"prepare.queue_time_us":69675,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":7816,"raft.queue_time_us":208158,"raft.run_cpu_time_us":517,"raft.run_wall_time_us":54725,"thread_start_us":177367,"threads_started":1,"wal-append.queue_time_us":501241}
I20260709 12:15:56.187463 22258 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.187721 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.189038 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:56.154212 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.002s	sys 0.017s
I20260709 12:15:56.191061 22297 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.191761 22297 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:56.150573 22319 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:56.164422 21856 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:56.192692 22319 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.194150 22307 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.194739 22322 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:56.195075 22319 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.195320 22322 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.196621 22192 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:56.197312 22192 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:56.144297 13253 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
W20260709 12:15:56.197999 22311 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.357s	user 0.000s	sys 0.001s
I20260709 12:15:56.198853 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.158525 12569 rpcz_store.cc:276] 0709 12:15:53.882840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.883004 (+   164us) service_pool.cc:224] Handling call
0709 12:15:53.883318 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:54.474069 (+590751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.141657 (+1667588us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:56.141853 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1116488576}
I20260709 12:15:56.186440 22306 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.192169 22297 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.155933 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:15:56.140703 22293 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.529s	user 0.002s	sys 0.000s
I20260709 12:15:56.205612 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:56.206260 12497 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:56.207267 12569 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:15:56.208225 22310 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:56.208577 12569 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:15:56.208781 22310 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.209240 22310 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:56.185211 12570 rpcz_store.cc:276] 0709 12:15:55.009351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.184076 (+1174725us) service_pool.cc:224] Handling call
0709 12:15:56.184338 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:56.184522 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:56.184553 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:56.184599 (+    46us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:56.184637 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.184694 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:56.184812 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:56.164234 22290 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.200189 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:15:56.192293 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:56.142563 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3780) took 1972 ms. Trace:
W20260709 12:15:56.152033 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3770) took 2190 ms (client timeout 1878 ms). Trace:
I20260709 12:15:56.192129 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:56.203747 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48530.1us
I20260709 12:15:56.204159 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.210745 12568 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:15:56.209508 21533 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.010s
I20260709 12:15:56.219177 12386 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.220548 22320 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.211149 22297 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:56.222250 22286 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.172252 21933 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:56.192056 22010 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:15:56.223958 22317 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.392s	user 0.000s	sys 0.009s
I20260709 12:15:56.131474 22205 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.226116 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:56.229167 12497 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:15:56.232215 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4054) took 1014 ms. Trace:
I20260709 12:15:56.249747 12568 rpcz_store.cc:276] 0709 12:15:55.218288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.189333 (+971045us) service_pool.cc:224] Handling call
0709 12:15:56.189598 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:56.231914 (+ 42316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.232146 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:56.204609 22322 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:56.232167 12640 rpcz_store.cc:269] 0709 12:15:53.951254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.951957 (+   703us) service_pool.cc:224] Handling call
0709 12:15:56.141911 (+2189954us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:56.142031 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1226845568}
I20260709 12:15:56.234972 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.230818 12643 rpcz_store.cc:276] 0709 12:15:54.170143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.280989 (+110846us) service_pool.cc:224] Handling call
0709 12:15:54.297235 (+ 16246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:54.455739 (+158504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:54.455807 (+    68us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:54.455870 (+    63us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:54.455916 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.142278 (+1686362us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:56.142374 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:56.142501 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248196352}
I20260709 12:15:56.199889 22328 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.236985 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.198386 22192 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.245831 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.252230 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:56.157640 22326 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:15:56.253971 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50098.3us
I20260709 12:15:56.162107 22298 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.262439 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:15:56.277258 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.008s	sys 0.004s
I20260709 12:15:56.246892 22290 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:56.300148 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.008s	sys 0.004s
W20260709 12:15:56.280617 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 145206us
I20260709 12:15:56.287916 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.302887 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:56.255918 21933 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:56.233294 22220 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:15:56.254648 22287 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.280308 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.269924 21533 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:15:56.308440 22316 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:15:56.307690 12386 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.288216 22320 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.261287 22315 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:56.272476 22312 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.304482 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:56.262629 22328 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:56.302887 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:56.270177 22324 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:15:56.288925 22224 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.284901 22319 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:56.213380 22283 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:56.377167 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.013s	sys 0.030s
W20260709 12:15:56.382407 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 156346us
W20260709 12:15:56.404418 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.013s	sys 0.032s
I20260709 12:15:56.390937 22301 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:56.328729 22338 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:56.301985 22307 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:15:56.387614 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84780.1us
W20260709 12:15:56.372102 22327 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:15:56.326951 22326 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.300971 22290 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:56.402141 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97794.6us
I20260709 12:15:56.399495 22230 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.466421 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:15:56.403121 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100319us
I20260709 12:15:56.406706 22322 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:56.410347 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.411038 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:56.480602 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69670.4us
I20260709 12:15:56.480774 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.481715 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.413214 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.482915 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.483588 12646 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
W20260709 12:15:56.483712 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70595us
I20260709 12:15:56.484544 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.484740 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.405575 22301 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.419207 12386 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.420169 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.486840 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.487783 12569 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:56.422036 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.413954 22324 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:15:56.422722 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:56.486029 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:56.489558 12380 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105815us
I20260709 12:15:56.425987 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3909) took 1023 ms. Trace:
I20260709 12:15:56.426211 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.426887 22315 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.494415 22324 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.497045 22315 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.497975 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:56.499318 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.499472 22315 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.500120 22180 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:15:56.500516 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.427820 22338 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.430846 22331 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:56.433295 22221 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.433480 22297 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.440184 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1338) took 1364 ms. Trace:
I20260709 12:15:56.518263 12493 rpcz_store.cc:276] 0709 12:15:55.075780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.075914 (+   134us) service_pool.cc:224] Handling call
0709 12:15:55.076312 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:55.292500 (+216188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:55.292552 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:55.329938 (+ 37386us) log.cc:844] Serialized 26 byte log entry
0709 12:15:55.443825 (+113887us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:55.443885 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.223097 (+779212us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:56.439559 (+216462us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:56.439656 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:56.439746 (+    90us) raft_consensus.cc:1685] finished
0709 12:15:56.439773 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:56.440066 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":12363,"spinlock_wait_cycles":526612224,"thread_start_us":77051,"threads_started":1,"wal-append.queue_time_us":127745}
I20260709 12:15:56.442711 21789 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:56.444659 22181 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:56.454205 22298 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.413237 21933 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:56.459126 22325 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.420s	user 0.004s	sys 0.000s
I20260709 12:15:56.394496 22320 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.391378 22179 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:15:56.459623 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4047) took 1553 ms. Trace:
I20260709 12:15:56.557310 12572 rpcz_store.cc:276] 0709 12:15:54.884138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.950127 (+ 65989us) service_pool.cc:224] Handling call
0709 12:15:54.954047 (+  3920us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:56.104669 (+1150622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.436954 (+332285us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:56.437371 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2596770560}
I20260709 12:15:56.459940 22319 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.472880 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.405956 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.475509 22290 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.478497 22239 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.474932 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.486961 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.410352 22330 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:15:56.488654 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.424182 22286 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:56.490358 12570 rpcz_store.cc:276] 0709 12:15:55.402480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.210546 (+808066us) service_pool.cc:224] Handling call
0709 12:15:56.210904 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:56.238857 (+ 27953us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:56.238886 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:56.248061 (+  9175us) log.cc:844] Serialized 26 byte log entry
0709 12:15:56.288311 (+ 40250us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:15:56.288355 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.288432 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:56.425709 (+137277us) raft_consensus.cc:1685] finished
0709 12:15:56.425732 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:56.425905 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45747,"raft.queue_time_us":27620,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":9334,"thread_start_us":39838,"threads_started":1,"wal-append.queue_time_us":79199}
I20260709 12:15:56.491376 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.471192 22326 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.494391 22301 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.553315 22310 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.501116 22230 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:15:56.501472 22180 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:15:56.500965 12386 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.501096 22220 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:15:56.488790 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.612238 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:15:56.612689 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:56.613703 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.427416 22205 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:56.470161 22255 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.650s	user 0.016s	sys 0.000s
W20260709 12:15:56.506310 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4051) took 1318 ms (client timeout 1568 ms). Trace:
I20260709 12:15:56.621462 12639 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:56.508436 12494 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:15:56.384925 22312 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.627144 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.628144 22312 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.465844 22207 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:56.628463 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.629132 12494 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:56.628949 22207 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.631485 22207 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.415359 22343 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:56.525548 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.632205 22343 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.526590 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:56.528371 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.633817 12640 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:56.634205 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.634707 22343 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.528826 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.635124 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.528863 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.635780 12568 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:56.636678 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.637097 12492 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
I20260709 12:15:56.637266 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:15:56.539150 22297 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.540028 21922 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:56.522511 22334 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:15:56.541232 21921 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:56.541399 12496 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:56.639971 21921 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.546173 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3912) took 1101 ms. Trace:
I20260709 12:15:56.546475 13260 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
W20260709 12:15:56.547976 22341 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:15:56.548292 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4049) took 1554 ms. Trace:
I20260709 12:15:56.551112 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.551246 22278 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:15:56.537290 22291 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.551868 22338 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.523260 21789 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:56.644524 21789 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.13, Last appended by leader: 13, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.646010 21789 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:56.646911 21921 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.525291 22181 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:56.648285 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.553699 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:56.564162 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71333.7us
I20260709 12:15:56.578437 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.650792 12572 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:15:56.648315 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.642817 12563 rpcz_store.cc:276] 0709 12:15:54.994050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.820166 (+826116us) service_pool.cc:224] Handling call
0709 12:15:56.546366 (+726200us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:56.546544 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:56.546574 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:56.546616 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:56.546646 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.546696 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:56.548098 (+  1402us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b5400044f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:56.548224 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1668996992}
I20260709 12:15:56.568840 12645 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:56.654382 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.648955 22181 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:56.569406 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:56.580574 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.488700 22322 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.658056 22322 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:56.593060 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.600194 12646 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
W20260709 12:15:56.602663 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141255us
W20260709 12:15:56.603304 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108021us
I20260709 12:15:56.556450 22344 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:56.568671 22008 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:15:56.662457 13262 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
W20260709 12:15:56.616559 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105152us
I20260709 12:15:56.569164 22189 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.641141 12567 rpcz_store.cc:276] 0709 12:15:55.445309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.470522 (+ 25213us) service_pool.cc:224] Handling call
0709 12:15:56.545961 (+1075439us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b5400044f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:56.546099 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1518548352}
I20260709 12:15:56.643074 12488 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:15:56.656391 22181 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:56.573328 22330 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.654382 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
W20260709 12:15:56.620709 12571 rpcz_store.cc:269] 0709 12:15:55.187470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.327887 (+140417us) service_pool.cc:224] Handling call
0709 12:15:56.506072 (+1178185us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:56.506227 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135590784}
I20260709 12:15:56.658098 12564 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:56.604336 12386 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 20 to 26, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 26 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.668974 22278 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:56.670149 12461 leader_election.cc:400] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
W20260709 12:15:56.684245 12536 connection.cc:256] RPC call timeout handler was delayed by -1.125207016000104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
W20260709 12:15:56.684653 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0279526199999509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2209s for a response
I20260709 12:15:56.650317 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:56.673352 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:56.571682 22237 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:15:56.681573 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:56.685406 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:56.695340 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124887us
I20260709 12:15:56.661867 22205 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.696445 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.681190 22330 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:56.703068 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:56.701538 12386 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 14 to 23. New cstate: current_term: 23 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.660401 22255 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.704284 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.683127 22278 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:15:56.709985 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59750.1us
I20260709 12:15:56.696887 21881 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:56.713551 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.720494 12639 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:56.720206 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.679579 22189 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.699120 22291 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:56.723623 12386 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.750264 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.682787 21922 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:15:56.750515 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.754590 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:56.741652 12565 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:15:56.746923 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:56.712747 22343 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:56.698412 22220 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:15:56.719559 22286 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:15:56.720881 22298 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:56.757907 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.721310 21881 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.781464 22278 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.783243 22278 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:56.711894 21533 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:15:56.787817 22011 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:56.702093 22205 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:56.765676 21921 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:15:56.791590 22205 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.766103 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:56.758111 22331 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:15:56.771476 22189 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.772794 22319 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.776068 21899 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:15:56.777906 21856 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:56.777544 22260 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:15:56.723626 22180 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:15:56.778334 22207 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:56.778494 22183 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:56.779505 22255 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:56.780221 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.782083 21881 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:56.781481 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.759352 22348 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:56.865021 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4048) took 1954 ms. Trace:
I20260709 12:15:56.783197 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:56.865942 12643 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:56.792955 13260 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:56.764005 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:56.768975 22326 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.794132 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:56.794106 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.797505 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.798283 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.798584 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.711207 22010 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.054s
W20260709 12:15:56.902767 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108763us
I20260709 12:15:56.865594 12565 rpcz_store.cc:276] 0709 12:15:54.910996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.492236 (+581240us) service_pool.cc:224] Handling call
0709 12:15:55.492632 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:56.755617 (+1262985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:56.864721 (+109104us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:56.864947 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2847818624}
I20260709 12:15:56.819685 12386 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 28. New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:56.820789 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.822569 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.757519 22346 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:56.792444 22354 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:56.794672 22291 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:56.792747 22239 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.852360 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:56.854660 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.861456 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.915132 12645 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:56.864986 22348 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:56.892431 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:56.898334 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134410us
I20260709 12:15:56.899009 22207 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:56.900077 21856 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:56.860255 22183 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:56.891443 12497 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:15:56.846992 22189 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:56.864835 22356 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:15:56.780148 22320 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:56.890996 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:15:56.907217 12643 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:15:56.909610 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.909572 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.906541 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.915745 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.880908 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:56.947940 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:56.949069 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:56.865700 22343 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:56.951700 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45295.1us
I20260709 12:15:56.951851 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:56.919234 12386 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 31, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 31 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:56.870482 22310 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:56.866473 22230 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.918710 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.921511 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:56.921504 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.954991 12492 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:56.955135 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.921784 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:56.935987 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 175811us
I20260709 12:15:56.955935 12495 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:56.937717 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.956688 12497 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:15:56.956770 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.912736 22255 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:56.904769 22010 consensus_queue.cc:1243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 25.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:15:56.950254 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:56.870051 22180 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:15:56.953855 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.911005 22354 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.931747 22207 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:56.911198 22328 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:56.965214 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46566us
I20260709 12:15:56.973930 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.987083 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.988933 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.989192 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.990733 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.991895 12563 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 31 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:15:56.992664 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.915923 22331 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:15:56.923388 22348 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:56.994207 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:56.969995 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:56.987824 12570 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:56.995689 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.996958 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:56.951565 21856 consensus_queue.cc:1243] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 32.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:15:56.997439 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:56.959609 22291 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:15:57.003635 12568 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:57.012956 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:57.001209 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:56.947606 22189 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.009814 22232 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:56.968642 22320 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:56.969769 22290 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:56.963240 22180 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:57.016371 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:56.980604 21922 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:56.995419 22278 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:15:56.981508 22207 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:57.041460 12570 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:57.042682 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:57.018759 22299 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:57.027652 21533 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:56.973349 22354 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:57.030934 12638 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26.
I20260709 12:15:56.982007 22097 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:57.069291 22179 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:57.069700 22097 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:57.069973 22179 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:57.070493 22179 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:57.029521 22189 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:57.073028 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:57.037322 21922 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:56.975353 22230 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.074438 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.074448 21922 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:57.076391 22230 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:57.077414 22180 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.031805 22290 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:57.080016 22290 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:56.969522 22343 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.038997 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:57.014055 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:57.037992 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:15:57.099674 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85768.4us
W20260709 12:15:57.100108 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62228.2us
I20260709 12:15:57.043931 12386 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:57.008553 22220 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:15:57.016057 22239 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:57.088173 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:57.129769 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:57.038163 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:57.088526 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:57.113536 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.126670 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.045491 22278 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:57.070307 22097 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:57.148972 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:57.154191 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.154917 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:57.147521 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 385923us
I20260709 12:15:57.149353 12386 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 19 to 27. New cstate: current_term: 27 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:57.068626 22299 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:57.157618 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.172164 12386 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 19 to 25, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 25 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:57.170682 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.127776 22253 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:15:57.178748 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:57.186146 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:57.112246 22298 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:57.128932 21533 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:57.085448 21881 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.156378 22010 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:57.153192 22189 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:57.161834 22312 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:15:57.196966 22366 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:15:57.148423 22278 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:57.145805 22220 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:57.205049 13260 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:15:57.190469 22319 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:15:57.216157 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.170332 22299 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:57.238057 22029 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:57.232959 22220 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:57.239395 21881 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:57.239992 22029 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:15:57.194844 22362 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:15:57.236644 22338 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:15:57.242780 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:57.243495 12572 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
W20260709 12:15:57.220535 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68249.7us
W20260709 12:15:57.221350 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67276us
W20260709 12:15:57.185376 21789 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.004s
I20260709 12:15:57.171882 22237 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:57.176597 22331 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:57.185909 12386 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 16 to 21, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:15:57.192380 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.200026 21921 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:15:57.232863 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129333us
I20260709 12:15:57.191035 22239 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:57.162048 22289 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:57.235306 22319 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:57.249954 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:57.254797 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:57.215569 22010 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:57.262040 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69708.8us
I20260709 12:15:57.223157 22253 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:57.272280 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3112) took 2170 ms. Trace:
I20260709 12:15:57.308908 12489 rpcz_store.cc:276] 0709 12:15:55.083913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.084086 (+   173us) service_pool.cc:224] Handling call
0709 12:15:55.084420 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:55.097753 (+ 13333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:55.097813 (+    60us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:55.097880 (+    67us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:15:55.097939 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.253519 (+2155580us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:57.253577 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:57.253790 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29847936}
I20260709 12:15:57.215833 22298 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.291079 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:15:57.298938 12386 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 20 to 25, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 25 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:57.250059 22189 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:57.251910 22237 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:57.274117 22230 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:15:57.268951 21921 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:57.352074 21921 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:57.310439 22181 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.630s	user 0.011s	sys 0.002s
I20260709 12:15:57.364447 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:15:57.338483 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:57.369473 12571 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:57.306417 22319 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:15:57.376799 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:57.377197 22322 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.719s	user 0.013s	sys 0.000s
I20260709 12:15:57.308290 22010 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:57.381179 12493 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:57.325053 22179 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.314935 12386 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 13 to 25, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 25 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:15:57.416054 22283 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:57.416586 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:57.400405 12489 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:15:57.418193 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.418615 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.419448 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.419955 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:57.420440 12491 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:15:57.313678 22328 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:57.337466 22237 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:15:57.416762 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3793) took 2721 ms (client timeout 2143 ms). Trace:
W20260709 12:15:57.394567 22373 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:15:57.429561 12491 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
W20260709 12:15:57.433468 12641 rpcz_store.cc:269] 0709 12:15:54.695762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.695975 (+   213us) service_pool.cc:224] Handling call
0709 12:15:57.416535 (+2720560us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:57.416675 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":817114496}
I20260709 12:15:57.437351 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:57.448493 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 409542us
I20260709 12:15:57.453609 12492 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:15:57.410694 22282 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.890s	user 0.001s	sys 0.026s
I20260709 12:15:57.459877 12641 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 25: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:57.461229 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:57.249390 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:57.464298 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:57.495097 12638 raft_consensus.cc:2376] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29. Re-sending same reply.
I20260709 12:15:57.489028 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.477833 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.507081 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:57.508483 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 259295us
I20260709 12:15:57.470188 22221 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:57.404810 22322 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.528872 13253 raft_consensus.cc:2368] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:15:57.472095 22276 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.106s
I20260709 12:15:57.542599 22298 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:57.514170 22230 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:57.543509 22230 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:57.543946 22230 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:57.545491 22343 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:57.511323 22368 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
W20260709 12:15:57.551996 22376 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
W20260709 12:15:57.552670 22232 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.002s
I20260709 12:15:57.553103 22232 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:57.554188 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3172) took 1071 ms. Trace:
I20260709 12:15:57.554584 12646 rpcz_store.cc:276] 0709 12:15:56.482852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.484503 (+  1651us) service_pool.cc:224] Handling call
0709 12:15:57.553987 (+1069484us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:57.554121 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263687424}
I20260709 12:15:57.555258 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.556511 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3122) took 1075 ms. Trace:
I20260709 12:15:57.468566 22275 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:15:57.558126 22275 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.490566 22328 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:15:57.519006 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.416803 22283 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.561347 12641 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 31 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:15:57.562660 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.520637 22253 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.563931 22346 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:15:57.564411 22346 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:57.565725 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3782) took 3386 ms. Trace:
I20260709 12:15:57.518844 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.567261 12638 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:57.557046 12490 rpcz_store.cc:276] 0709 12:15:56.481411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.481950 (+   539us) service_pool.cc:224] Handling call
0709 12:15:57.555018 (+1073068us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:15:57.556426 (+  1408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14536448}
W20260709 12:15:57.526883 22378 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:15:57.518786 22384 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:15:57.527261 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:57.525357 22379 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:57.556660 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1318) took 2069 ms. Trace:
I20260709 12:15:57.555781 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:57.585177 12646 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:57.583985 13257 rpcz_store.cc:276] 0709 12:15:55.487653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:55.487835 (+   182us) service_pool.cc:224] Handling call
0709 12:15:55.488285 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:55.910588 (+422303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:55.910637 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:55.910696 (+    59us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:55.910743 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.556406 (+1645663us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:57.556459 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:57.556600 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":970498304}
I20260709 12:15:57.582983 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:15:57.566171 12644 rpcz_store.cc:276] 0709 12:15:54.179711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.282652 (+102941us) service_pool.cc:224] Handling call
0709 12:15:54.282979 (+   327us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:54.310494 (+ 27515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:54.310542 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:54.311315 (+   773us) log.cc:844] Serialized 52 byte log entry
0709 12:15:54.554379 (+243064us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:54.629781 (+ 75402us) log.cc:844] Serialized 14 byte log entry
0709 12:15:54.630216 (+   435us) log.cc:844] Serialized 14 byte log entry
0709 12:15:54.634298 (+  4082us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:54.634547 (+   249us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:57.373829 (+2739282us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:15:57.565423 (+191594us) raft_consensus.cc:1685] finished
0709 12:15:57.565450 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:57.565618 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20013,"prepare.queue_time_us":237252,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":6475,"raft.queue_time_us":14969,"raft.run_cpu_time_us":423,"raft.run_wall_time_us":55056,"spinlock_wait_cycles":8465536,"thread_start_us":136398,"threads_started":1,"wal-append.queue_time_us":1025422}
I20260709 12:15:57.509552 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:57.589391 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:57.522601 22375 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260709 12:15:57.601341 12571 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:15:57.604826 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:57.494148 21922 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.601847 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.599947 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
W20260709 12:15:57.589715 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80231.2us
W20260709 12:15:57.612185 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86635.9us
I20260709 12:15:57.613024 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:57.605005 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 389665us
I20260709 12:15:57.620632 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:57.598637 22298 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:57.608556 22310 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:15:57.606601 22290 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.003s
I20260709 12:15:57.583745 22384 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:57.608119 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101108us
I20260709 12:15:57.565286 22346 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.632901 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:15:57.643803 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.644610 12646 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:57.644639 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.581596 22221 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.645816 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.628758 22236 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:57.661163 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4035) took 3775 ms. Trace:
I20260709 12:15:57.661799 12566 rpcz_store.cc:276] 0709 12:15:53.886118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:53.886299 (+   181us) service_pool.cc:224] Handling call
0709 12:15:53.886650 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:55.313193 (+1426543us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.660827 (+2347634us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:57.661072 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3257899392}
I20260709 12:15:57.663174 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:57.622184 22255 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:57.666584 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:57.669128 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.679064 12566 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:15:57.627420 22285 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:15:57.680474 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:57.639557 22337 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:15:57.638623 22298 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:57.683894 12566 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:15:57.601388 22379 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.642302 22384 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:15:57.696247 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1348) took 1490 ms. Trace:
I20260709 12:15:57.702241 12496 rpcz_store.cc:276] 0709 12:15:56.213765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.218730 (+  4965us) service_pool.cc:224] Handling call
0709 12:15:57.688921 (+1470191us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:57.696162 (+  7241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":655394176}
I20260709 12:15:57.698184 12566 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:57.700399 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:15:57.709830 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.719846 12495 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:15:57.721925 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.725775 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:57.727049 12495 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:15:57.735764 12491 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:15:57.648201 22290 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.697551 22298 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:57.694837 22310 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:57.765183 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:57.776830 12496 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:15:57.765965 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:57.704100 22344 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.017s	user 0.015s	sys 0.000s
I20260709 12:15:57.686635 22337 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:57.764784 12638 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:15:57.694548 22236 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:15:57.773797 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129257us
W20260709 12:15:57.778258 12642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.023s	sys 0.029s
W20260709 12:15:57.786840 12642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.023s	sys 0.030s
I20260709 12:15:57.787052 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:57.787606 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3176) took 1275 ms. Trace:
I20260709 12:15:57.680346 22220 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.790136 12572 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
W20260709 12:15:57.793470 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:57.765504 22232 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:15:57.762838 22255 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:57.788069 12642 rpcz_store.cc:276] 0709 12:15:56.512049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.657033 (+144984us) service_pool.cc:224] Handling call
0709 12:15:56.657456 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:56.685768 (+ 28312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:56.685800 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:56.693045 (+  7245us) log.cc:844] Serialized 26 byte log entry
0709 12:15:57.098477 (+405432us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:57.221560 (+123083us) log.cc:844] Serialized 14 byte log entry
0709 12:15:57.787245 (+565685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.787332 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:57.787379 (+    47us) raft_consensus.cc:1685] finished
0709 12:15:57.787393 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:57.787544 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79158,"prepare.queue_time_us":125881,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":3704,"raft.queue_time_us":55206,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":10162,"thread_start_us":986132,"threads_started":3,"wal-append.queue_time_us":768653}
I20260709 12:15:57.814726 22389 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:15:57.816180 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:57.819675 12642 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26.
I20260709 12:15:57.698697 22278 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.857528 12494 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:57.789326 22008 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:15:57.789247 22344 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:57.803514 22283 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:57.832168 22377 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:15:57.878135 13261 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: []
W20260709 12:15:57.871380 21921 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.519s	user 0.013s	sys 0.001s
W20260709 12:15:57.895010 22386 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:15:57.895220 21921 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:57.873728 22348 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.854s	user 0.009s	sys 0.004s
I20260709 12:15:57.902105 22348 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.902249 22255 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:57.903725 22255 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:15:57.904155 22255 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:57.904960 22348 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:57.905876 12646 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:57.907028 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:57.905889 22236 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.888895 22372 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:57.890292 22221 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:57.894353 22380 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
W20260709 12:15:57.894681 22390 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 12:15:57.873003 22011 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.085s	user 0.013s	sys 0.004s
W20260709 12:15:57.875192 22097 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.698s	user 0.008s	sys 0.005s
I20260709 12:15:57.925395 12495 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:15:57.884356 22330 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.171s	user 0.009s	sys 0.005s
W20260709 12:15:57.939610 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 239329us
I20260709 12:15:57.942896 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:57.914351 22299 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.650s	user 0.012s	sys 0.001s
I20260709 12:15:57.952239 12569 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:57.942411 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3803) took 3079 ms. Trace:
I20260709 12:15:57.954864 12637 rpcz_store.cc:276] 0709 12:15:54.863111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.884681 (+ 21570us) service_pool.cc:224] Handling call
0709 12:15:56.142080 (+1257399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:56.161645 (+ 19565us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.941998 (+1780353us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001f7cd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:57.942306 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2863808768}
I20260709 12:15:57.915362 22389 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:15:57.967447 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 180503us
I20260709 12:15:57.969409 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:57.905105 22255 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.913158 22344 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:15:57.972780 22396 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:57.962399 22389 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:57.962975 21921 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:57.981166 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1327) took 1128 ms. Trace:
I20260709 12:15:57.981832 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:57.982843 22207 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.884s	user 0.011s	sys 0.003s
I20260709 12:15:58.068050 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:58.071650 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:57.997099 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 3930) took 1009 ms. Trace:
I20260709 12:15:58.005252 12643 rpcz_store.cc:276] 0709 12:15:56.853128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.009127 (+155999us) service_pool.cc:224] Handling call
0709 12:15:57.032876 (+ 23749us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:57.086430 (+ 53554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:57.086469 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:57.087013 (+   544us) log.cc:844] Serialized 26 byte log entry
0709 12:15:57.396355 (+309342us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:57.396410 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.396487 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:57.980901 (+584414us) raft_consensus.cc:1685] finished
0709 12:15:57.980924 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:57.981083 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53327,"raft.queue_time_us":64464,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":32647,"thread_start_us":308846,"threads_started":1,"wal-append.queue_time_us":541878}
I20260709 12:15:58.073148 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:58.073520 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:58.004534 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125779us
I20260709 12:15:57.951773 22097 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.077025 13218 leader_election.cc:400] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26.
I20260709 12:15:57.935892 22372 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:57.957198 22299 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.006762 22389 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:58.074548 12567 rpcz_store.cc:276] 0709 12:15:56.988110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.988255 (+   145us) service_pool.cc:224] Handling call
0709 12:15:56.996166 (+  7911us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:57.005566 (+  9400us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:57.027456 (+ 21890us) log.cc:844] Serialized 14 byte log entry
0709 12:15:57.707129 (+679673us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:57.739878 (+ 32749us) log.cc:844] Serialized 52 byte log entry
0709 12:15:57.740054 (+   176us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:57.747747 (+  7693us) log.cc:844] Serialized 14 byte log entry
0709 12:15:57.943412 (+195665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:57.943554 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:57.996239 (+ 52685us) raft_consensus.cc:1685] finished
0709 12:15:57.996271 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:57.996996 (+   725us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42224,"prepare.queue_time_us":21774,"prepare.run_cpu_time_us":196,"prepare.run_wall_time_us":5869,"raft.queue_time_us":154053,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":27336,"thread_start_us":786101,"threads_started":3,"wal-append.queue_time_us":424882}
I20260709 12:15:58.087215 12643 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:57.956439 22330 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.886289 22337 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:57.984898 22285 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:58.077476 22255 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:58.114023 12568 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
W20260709 12:15:58.114496 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.080206 22253 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:58.116840 12490 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:58.079989 22011 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:15:58.125861 12571 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:58.129663 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:58.132162 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:58.133235 22342 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.135573 12566 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:58.136277 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:58.146100 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 159296us
W20260709 12:15:58.143088 22230 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.003s
I20260709 12:15:58.126534 22290 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.139027 22011 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:58.005805 22396 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.124619 22379 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.080588 22207 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:58.153376 22319 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.740s	user 0.010s	sys 0.003s
I20260709 12:15:58.140722 22403 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:58.147173 22278 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.149960 22220 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.158844 22011 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:15:58.191951 22010 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.759s	user 0.006s	sys 0.009s
I20260709 12:15:58.192556 22010 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.194517 22382 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.092s
I20260709 12:15:58.195443 22400 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:15:58.196063 22400 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:58.197697 22404 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:15:58.157938 22230 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.200569 22230 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.201750 22406 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:58.202453 22398 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:15:58.144536 22342 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.175817 22290 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:58.167037 22403 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:58.210302 22403 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:58.214829 22299 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.177388 22285 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.220994 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:15:58.225647 22412 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:15:58.233687 12637 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: [29.11-31.12]
I20260709 12:15:58.235800 22322 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:58.241115 22237 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.765s	user 0.012s	sys 0.001s
I20260709 12:15:58.242766 22237 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.242978 22372 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.244588 22330 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.202457 22406 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.248054 22406 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.249264 22310 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.250177 22310 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.256034 22207 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:15:58.279422 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:58.285297 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3166) took 1813 ms (client timeout 1825 ms). Trace:
W20260709 12:15:58.285804 12640 rpcz_store.cc:269] 0709 12:15:56.472353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.472537 (+   184us) service_pool.cc:224] Handling call
0709 12:15:58.285043 (+1812506us) spinlock_profiling.cc:243] Waited 66.7 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.285224 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153386240}
I20260709 12:15:58.287952 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3824) took 1019 ms. Trace:
I20260709 12:15:58.288003 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1331) took 1396 ms. Trace:
I20260709 12:15:58.197026 22400 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 31.12, Last appended by leader: 12, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.293121 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:58.293982 22405 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:15:58.288673 12646 rpcz_store.cc:276] 0709 12:15:56.904565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.693234 (+788669us) service_pool.cc:224] Handling call
0709 12:15:57.693603 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:57.906398 (+212795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.287501 (+381103us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:58.287856 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486990080}
I20260709 12:15:58.304826 21899 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.327517 21899 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.303853 22097 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.303853 22259 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.329291 22179 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.253521 22310 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.292963 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1743) took 1159 ms. Trace:
I20260709 12:15:58.343742 12564 rpcz_store.cc:276] 0709 12:15:57.133926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.134136 (+   210us) service_pool.cc:224] Handling call
0709 12:15:57.134551 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:57.380600 (+246049us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:57.380649 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:57.380716 (+    67us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:15:57.380759 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.292724 (+911965us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54002cfbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:58.292770 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.292911 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565183872}
W20260709 12:15:58.296638 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3175) took 1805 ms (client timeout 1505 ms). Trace:
I20260709 12:15:58.344659 22389 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.330933 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:58.297047 12643 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:15:58.288450 12640 rpcz_store.cc:276] 0709 12:15:57.268195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.286695 (+1018500us) service_pool.cc:224] Handling call
0709 12:15:58.287087 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.287306 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.287353 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:58.287410 (+    57us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:15:58.287453 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.287724 (+   271us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.287883 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.327904 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
W20260709 12:15:58.305723 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.336120 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:58.338943 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:58.328733 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1239) took 1919 ms (client timeout 1580 ms). Trace:
I20260709 12:15:58.355898 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3826) took 1013 ms. Trace:
I20260709 12:15:58.356444 12640 rpcz_store.cc:276] 0709 12:15:57.342362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.352297 (+1009935us) service_pool.cc:224] Handling call
0709 12:15:58.355327 (+  3030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.355537 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.355566 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:58.355615 (+    49us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:58.355649 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.355731 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.355842 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.298655 22010 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:58.345474 12639 rpcz_store.cc:269] 0709 12:15:56.491508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.612306 (+120798us) service_pool.cc:224] Handling call
0709 12:15:58.296403 (+1684097us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.296547 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17970048}
I20260709 12:15:58.359753 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3825) took 1057 ms. Trace:
I20260709 12:15:58.330179 21899 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.360373 12646 rpcz_store.cc:276] 0709 12:15:57.302482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.327232 (+1024750us) service_pool.cc:224] Handling call
0709 12:15:58.327669 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.353230 (+ 25561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.359663 (+  6433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.362650 12566 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:58.363099 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3829) took 1019 ms. Trace:
W20260709 12:15:58.208954 22354 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.110s	user 0.013s	sys 0.000s
I20260709 12:15:58.175251 22319 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:58.354965 13253 rpcz_store.cc:269] 0709 12:15:56.409798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.409990 (+   192us) service_pool.cc:224] Handling call
0709 12:15:58.328447 (+1918457us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.328631 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2411560704}
W20260709 12:15:58.292294 22370 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.061s	user 0.000s	sys 0.003s
I20260709 12:15:58.363739 12646 rpcz_store.cc:276] 0709 12:15:57.343351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.361881 (+1018530us) service_pool.cc:224] Handling call
0709 12:15:58.362344 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.362622 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.362664 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:58.362724 (+    60us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:58.362781 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.362869 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.363022 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:58.345738 22407 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.176s	user 0.001s	sys 0.003s
I20260709 12:15:58.349313 22346 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:58.371650 21899 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.372342 21899 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.374742 21899 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.365494 22259 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.381821 22259 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.384654 22259 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.387387 12646 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 12
I20260709 12:15:58.387450 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:58.389771 12646 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 12:15:58.395759 12644 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:15:58.395617 12646 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:15:58.395871 13253 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
W20260709 12:15:58.397239 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 229608us
I20260709 12:15:58.398604 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:15:58.400480 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:58.403132 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:58.299851 12490 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:15:58.406710 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:15:58.406651 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:58.408300 13253 raft_consensus.cc:2368] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:15:58.418704 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:58.429989 22253 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:58.432186 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:58.326478 22183 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 28.10, Last appended by leader: 10, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.435240 22183 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:58.410035 22291 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:58.436184 22291 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:58.436754 22291 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:58.440734 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4134) took 1639 ms. Trace:
I20260709 12:15:58.442364 12566 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:15:58.416086 22416 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:58.402838 22259 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:15:58.443495 22416 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:58.445320 22416 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:15:58.447805 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.449214 22179 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.449771 22179 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.452293 22179 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.454056 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:58.300263 22237 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.455106 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:58.455888 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49267.1us
I20260709 12:15:58.459604 12537 leader_election.cc:400] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:15:58.467579 12537 leader_election.cc:403] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:15:58.441414 12493 rpcz_store.cc:276] 0709 12:15:56.800835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.948765 (+147930us) service_pool.cc:224] Handling call
0709 12:15:58.440481 (+1491716us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b540012c5d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.440645 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":992531200}
W20260709 12:15:58.466921 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3919) took 2046 ms (client timeout 2520 ms). Trace:
W20260709 12:15:58.469292 12569 rpcz_store.cc:269] 0709 12:15:56.420272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.613420 (+193148us) service_pool.cc:224] Handling call
0709 12:15:58.466657 (+1853237us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.466833 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3076422912}
W20260709 12:15:58.422631 22384 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.655s	user 0.009s	sys 0.005s
I20260709 12:15:58.370635 22354 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.487135 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:58.386780 22403 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.486735 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:58.488751 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:58.494699 12571 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:58.488880 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:58.498100 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:58.501319 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:58.461819 22259 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.475574 22097 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:15:58.490994 22086 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:15:58.518070 12641 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:15:58.442953 22011 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.526103 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:58.520560 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:58.493674 22384 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.533941 12571 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:15:58.516482 22008 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:58.517724 22097 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:15:58.544899 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:58.513298 22259 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.534543 22230 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:15:58.548007 22298 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.756s	user 0.009s	sys 0.005s
I20260709 12:15:58.561249 22389 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.569098 12564 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:58.561133 22416 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.549774 22008 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.595078 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:58.598863 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:58.568171 22298 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.600597 22260 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
W20260709 12:15:58.609236 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.602927 21899 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:15:58.642179 21899 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:58.642777 21899 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:58.642936 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:58.647141 22384 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.608191 22297 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:15:58.650578 22297 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:15:58.651105 22297 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:58.652235 22297 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.602200 22181 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:15:58.654323 22181 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:58.655556 22181 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.662755 22259 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.612948 22403 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.672880 22011 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.673169 22403 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:58.673903 22403 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.676569 22403 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.606591 22097 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.617822 22372 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:15:58.618710 22220 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:58.622337 22337 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:15:58.626514 22419 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
W20260709 12:15:58.627632 22422 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:15:58.628835 22421 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:15:58.630879 22424 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:15:58.637585 22416 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.641404 22008 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.682487 12567 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:58.682276 22372 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:58.692708 22297 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:15:58.717036 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3822) took 1449 ms. Trace:
I20260709 12:15:58.717681 12637 rpcz_store.cc:276] 0709 12:15:57.267627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.234824 (+967197us) service_pool.cc:224] Handling call
0709 12:15:58.294510 (+ 59686us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:58.294962 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.294993 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:58.295578 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:15:58.492770 (+197192us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:58.569086 (+ 76316us) log.cc:844] Serialized 14 byte log entry
0709 12:15:58.569693 (+   607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.569806 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:58.716761 (+146955us) raft_consensus.cc:1685] finished
0709 12:15:58.716789 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.716953 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75594,"prepare.queue_time_us":96089,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":2480,"raft.queue_time_us":6856,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":2993,"thread_start_us":257481,"threads_started":2,"wal-append.queue_time_us":237161}
I20260709 12:15:58.720321 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.721208 12637 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:58.725859 22008 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:58.726609 22008 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.729136 22008 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.731431 22298 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.732879 22372 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 31.12, Last appended by leader: 12, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.734426 22372 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:58.747368 22298 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:15:58.753669 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134690us
I20260709 12:15:58.756632 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.759383 22396 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:58.766760 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1332) took 1802 ms. Trace:
I20260709 12:15:58.767174 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3830) took 1423 ms. Trace:
I20260709 12:15:58.772848 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.767288 12642 rpcz_store.cc:276] 0709 12:15:56.964498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.871658 (+907160us) service_pool.cc:224] Handling call
0709 12:15:57.873853 (+  2195us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:58.136944 (+263091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.136996 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:58.137684 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:15:58.367032 (+229348us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:15:58.395887 (+ 28855us) log.cc:844] Serialized 14 byte log entry
0709 12:15:58.401007 (+  5120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.401121 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:58.576705 (+175584us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b18004507c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:15:58.765002 (+188297us) raft_consensus.cc:1685] finished
0709 12:15:58.766500 (+  1498us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.766682 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9617,"prepare.queue_time_us":28121,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":66410,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":9431,"spinlock_wait_cycles":23300096,"thread_start_us":442684,"threads_started":2,"wal-append.queue_time_us":339063}
I20260709 12:15:58.748131 22298 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.767607 12646 rpcz_store.cc:276] 0709 12:15:57.343615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.377075 (+1033460us) service_pool.cc:224] Handling call
0709 12:15:58.377551 (+   476us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:58.404120 (+ 26569us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.404167 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:58.404912 (+   745us) log.cc:844] Serialized 52 byte log entry
0709 12:15:58.549735 (+144823us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:15:58.681880 (+132145us) log.cc:844] Serialized 14 byte log entry
0709 12:15:58.687728 (+  5848us) log.cc:844] Serialized 14 byte log entry
0709 12:15:58.688348 (+   620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.688443 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:58.766950 (+ 78507us) raft_consensus.cc:1685] finished
0709 12:15:58.766975 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.767116 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64515,"prepare.queue_time_us":71007,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":131,"raft.queue_time_us":7445,"raft.run_cpu_time_us":405,"raft.run_wall_time_us":15134,"thread_start_us":242449,"threads_started":2,"wal-append.queue_time_us":196500}
I20260709 12:15:58.779382 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.780186 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:58.780506 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.782737 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:58.785324 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.789031 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.789754 12642 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:58.789991 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:15:58.789674 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.792737 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:58.793303 12637 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:58.793521 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:58.794387 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:58.794644 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.795030 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:58.776245 22346 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.795898 12637 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:58.796952 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:15:58.798265 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:58.756851 22291 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.799437 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:15:58.806703 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3820) took 1553 ms. Trace:
I20260709 12:15:58.807529 12646 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:58.807289 12643 rpcz_store.cc:276] 0709 12:15:57.253044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.102772 (+849728us) service_pool.cc:224] Handling call
0709 12:15:58.296789 (+194017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.352474 (+ 55685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.806394 (+453920us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b5400155250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.806608 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423977856}
I20260709 12:15:58.788301 22010 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:58.799502 22260 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:15:58.799278 22416 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:58.817947 22416 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:58.818409 22416 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:58.808126 22384 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:15:58.821331 22384 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:58.821801 22384 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:15:58.823777 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:58.831462 22291 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:58.832103 22260 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:58.832863 22260 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:58.802397 22346 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:58.838207 22260 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:58.796229 22396 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:15:58.842384 22180 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:15:58.840077 22260 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:58.812070 22010 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:15:58.855250 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3815) took 1618 ms (client timeout 1878 ms). Trace:
I20260709 12:15:58.829344 22298 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:58.855894 12644 rpcz_store.cc:269] 0709 12:15:57.236497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.589083 (+352586us) service_pool.cc:224] Handling call
0709 12:15:58.854906 (+1265823us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:58.855146 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1852498304}
I20260709 12:15:58.865130 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:58.876544 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:15:58.885568 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.854568 22319 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:58.866289 22010 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:58.944478 12637 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
W20260709 12:15:58.949801 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:58.949944 12644 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 27. Ops: []
I20260709 12:15:58.950881 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3180) took 1605 ms. Trace:
I20260709 12:15:58.951229 12644 rpcz_store.cc:276] 0709 12:15:57.345324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.949385 (+1604061us) service_pool.cc:224] Handling call
0709 12:15:58.949748 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.950617 (+   869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.950825 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.959247 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1334) took 1242 ms. Trace:
I20260709 12:15:58.959838 12637 rpcz_store.cc:276] 0709 12:15:57.716898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.958304 (+1241406us) service_pool.cc:224] Handling call
0709 12:15:58.958709 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.958912 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.958935 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:58.958978 (+    43us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:15:58.959006 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.959054 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:58.959182 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.962399 12644 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:15:58.963197 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1335) took 1246 ms. Trace:
I20260709 12:15:58.964268 12644 rpcz_store.cc:276] 0709 12:15:57.717253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.961750 (+1244497us) service_pool.cc:224] Handling call
0709 12:15:58.962074 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.962941 (+   867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.963143 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.971302 12644 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
W20260709 12:15:58.949555 22426 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:15:58.980412 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1744) took 1844 ms. Trace:
I20260709 12:15:58.980974 12568 rpcz_store.cc:276] 0709 12:15:57.136040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.142262 (+  6222us) service_pool.cc:224] Handling call
0709 12:15:57.142595 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.124963 (+982368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.980151 (+855188us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:58.980340 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2227531648}
I20260709 12:15:58.985083 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1336) took 1267 ms. Trace:
I20260709 12:15:58.987140 12644 rpcz_store.cc:276] 0709 12:15:57.717558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.965075 (+1247517us) service_pool.cc:224] Handling call
0709 12:15:58.965378 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.977611 (+ 12233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.984658 (+  7047us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:58.989471 12644 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
I20260709 12:15:58.972751 12637 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:15:58.958618 22220 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.991964 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3186) took 1019 ms. Trace:
I20260709 12:15:58.992465 12637 rpcz_store.cc:276] 0709 12:15:57.972616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.965698 (+993082us) service_pool.cc:224] Handling call
0709 12:15:58.972531 (+  6833us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:58.991667 (+ 19136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.991887 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:15:58.997087 12462 connection.cc:256] RPC call timeout handler was delayed by -1.2969968849999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:15:58.963438 22349 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:15:59.001987 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:15:59.005331 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.001364 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3187) took 1023 ms. Trace:
I20260709 12:15:59.007246 12644 rpcz_store.cc:276] 0709 12:15:57.978320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.988254 (+1009934us) service_pool.cc:224] Handling call
0709 12:15:58.988932 (+   678us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.990133 (+  1201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.990420 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:15:59.000437 22319 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.000226 22176 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.013s
I20260709 12:15:59.013120 22349 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.016245 22349 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.019263 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:59.002132 22220 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.022789 22220 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.023319 22220 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:58.992411 22097 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:59.028236 22097 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.037683 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:59.056435 12644 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 10. Preceding OpId from leader: term: 32 index: 13. (index mismatch)
I20260709 12:15:59.007481 22181 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:58.999557 22313 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
I20260709 12:15:59.003940 21899 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.051074 22400 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:59.104022 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.013211 22319 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.037776 22220 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.029577 22384 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.131170 21899 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.120493 22298 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.132623 22346 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.132768 22298 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.079200 22400 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.135438 22298 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:59.134864 22433 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
W20260709 12:15:59.143072 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 163399us
I20260709 12:15:59.149786 12637 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:15:59.073361 22181 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.152237 22176 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:15:59.152827 22400 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.153071 22181 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.102574 22416 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.155177 22313 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:59.156806 22416 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.157392 12565 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:15:59.182305 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:59.199098 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:59.200277 12563 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:15:59.153450 22176 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:59.188542 22416 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.205809 22181 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:59.210736 22275 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.211103 22181 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.202332 22400 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.203451 22176 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:59.214016 22400 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.215520 22176 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.216977 22400 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.217177 22176 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:59.217952 22384 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.222003 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:59.224814 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4144) took 1512 ms (client timeout 1533 ms). Trace:
I20260709 12:15:59.225577 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:59.226511 12497 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:59.227149 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1364) took 1810 ms. Trace:
I20260709 12:15:59.227572 12494 rpcz_store.cc:276] 0709 12:15:57.417191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.466455 (+ 49264us) service_pool.cc:224] Handling call
0709 12:15:57.466767 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:57.858825 (+392058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:59.226892 (+1368067us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:59.227084 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898164992}
W20260709 12:15:59.230489 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3814) took 1994 ms (client timeout 1832 ms). Trace:
W20260709 12:15:59.231065 12641 rpcz_store.cc:269] 0709 12:15:57.236087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.562258 (+326171us) service_pool.cc:224] Handling call
0709 12:15:59.230198 (+1667940us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:59.230391 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2194941696}
I20260709 12:15:59.227062 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:15:59.225286 12490 rpcz_store.cc:269] 0709 12:15:57.712843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.713133 (+   290us) service_pool.cc:224] Handling call
0709 12:15:59.224619 (+1511486us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:59.224750 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779866496}
I20260709 12:15:59.244225 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1325) took 2391 ms. Trace:
I20260709 12:15:59.247642 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:59.216977 22342 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.244802 12645 rpcz_store.cc:276] 0709 12:15:56.852547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:56.961866 (+109319us) service_pool.cc:224] Handling call
0709 12:15:58.285271 (+1323405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.285494 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.285525 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:15:58.285565 (+    40us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:15:58.285595 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.285690 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:59.243946 (+958256us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400154350. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:59.244130 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3042524544}
I20260709 12:15:59.252636 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:59.255908 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:59.256546 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:59.264432 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:59.252878 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:59.271763 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:59.269248 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:15:59.357587 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4077) took 2099 ms. Trace:
I20260709 12:15:59.327860 13261 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
W20260709 12:15:59.275877 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.270587 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:59.262804 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.359779 12494 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:15:59.219772 22416 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:59.363287 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94159us
I20260709 12:15:59.351790 12571 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:15:59.303162 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.254554 22323 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.111s
I20260709 12:15:59.278301 22310 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.355006 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:59.367493 12496 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:15:59.367270 22310 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.368536 12571 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:15:59.333782 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:59.370409 22310 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.272673 22261 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.267721 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:59.374011 22384 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:59.249759 22435 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.374322 22261 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.375139 22435 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.328013 22437 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:15:59.360123 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:59.377944 22437 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.328593 12490 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 12. Preceding OpId from leader: term: 32 index: 13. (index mismatch)
I20260709 12:15:59.370617 12571 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:59.271468 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:59.378122 22261 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:59.304054 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:59.379338 12495 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:15:59.265133 22181 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.378410 12493 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:59.372486 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:15:59.378463 12641 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:15:59.381206 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:59.372622 22416 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.374528 12488 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:15:59.277041 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:15:59.382599 22261 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:15:59.378496 22437 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:15:59.378780 22310 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:15:59.271664 22205 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.383983 22205 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.366255 12644 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:15:59.359304 12572 rpcz_store.cc:276] 0709 12:15:57.258070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:57.379118 (+121048us) service_pool.cc:224] Handling call
0709 12:15:57.408948 (+ 29830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:57.802549 (+393601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:59.357290 (+1554741us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54003ac354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:59.357511 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":875903744}
I20260709 12:15:59.355592 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:15:59.387212 22440 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:15:59.390127 22435 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.396543 12645 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:15:59.397519 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.398712 22310 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:59.399274 22354 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:59.369585 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:15:59.387260 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:59.404382 12644 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:15:59.405900 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50702.9us
I20260709 12:15:59.208155 22372 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:59.416535 12563 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:15:59.422518 22354 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:15:59.255038 22432 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.387143 22205 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.357287 22343 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.428822 21789 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:59.399315 22310 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:15:59.445417 22184 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:15:59.419904 22319 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:59.447399 22220 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.266309 22313 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:59.374728 22384 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.486654 12563 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:15:59.447643 21922 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:59.527789 22324 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:15:59.529968 22324 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:15:59.488382 22313 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.530356 22324 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:59.487679 22314 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.045s
I20260709 12:15:59.533488 22314 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:15:59.533942 22314 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:15:59.535029 22314 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.536571 22314 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:15:59.538941 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:15:59.540277 21922 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:15:59.540980 22183 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:15:59.544632 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:59.545501 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:59.541543 22183 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:15:59.537997 22324 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:59.566431 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.560869 22183 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.487879 22319 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.443614 22372 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:59.555187 22180 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.002s
I20260709 12:15:59.580420 12496 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:15:59.581394 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3831) took 2237 ms. Trace:
I20260709 12:15:59.584846 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:59.599491 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:15:59.604295 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:59.609318 12644 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:15:59.612082 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.594278 12643 rpcz_store.cc:276] 0709 12:15:57.343907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.850114 (+1506207us) service_pool.cc:224] Handling call
0709 12:15:58.855147 (+  5033us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:15:58.855809 (+   662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.855834 (+    25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:15:58.869424 (+ 13590us) log.cc:844] Serialized 52 byte log entry
0709 12:15:59.197702 (+328278us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:15:59.220743 (+ 23041us) log.cc:844] Serialized 14 byte log entry
0709 12:15:59.250192 (+ 29449us) log.cc:844] Serialized 14 byte log entry
0709 12:15:59.347567 (+ 97375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:59.347706 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:59.581139 (+233433us) raft_consensus.cc:1685] finished
0709 12:15:59.581158 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:15:59.581330 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46643,"prepare.queue_time_us":25034,"prepare.run_wall_time_us":56,"raft.queue_time_us":37636,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":36949,"spinlock_wait_cycles":75082368,"thread_start_us":231004,"threads_started":1,"wal-append.queue_time_us":547265}
I20260709 12:15:59.461369 22432 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:59.593094 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.642423 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:59.642539 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:15:59.561573 22260 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.695s	user 0.013s	sys 0.000s
I20260709 12:15:59.635185 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:59.644002 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:59.643280 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:59.481021 21789 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:59.637840 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50337.8us
I20260709 12:15:59.639814 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.642724 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:15:59.646020 12644 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:15:59.643173 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:15:59.647078 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.466864 22343 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.649905 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:59.648229 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:15:59.656736 12644 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:15:59.591111 22437 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.649031 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:15:59.664567 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.738168 12643 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:15:59.687052 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.593118 22324 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:15:59.755439 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:15:59.634209 22189 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.569360 22278 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:15:59.654129 21922 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.580557 22371 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.161s
I20260709 12:15:59.659157 22342 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.654425 22183 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:59.655160 22180 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:15:59.593322 22319 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.820243 22319 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:15:59.821146 22319 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:59.676543 22205 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.884488 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:15:59.785311 12489 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.017s	sys 0.026s
I20260709 12:15:59.796156 12492 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:15:59.665814 22331 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.797348 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:15:59.799309 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:15:59.743906 22400 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:15:59.801770 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.001s	sys 0.028s
I20260709 12:15:59.742842 22346 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:59.804220 22180 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:15:59.783874 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:15:59.882936 22416 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:15:59.851775 21740 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:15:59.852984 22348 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:15:59.798557 21922 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.860754 22437 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:15:59.862166 22384 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:15:59.875255 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:15:59.804065 22342 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:59.912209 22312 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.086s	user 0.038s	sys 0.057s
I20260709 12:15:59.909828 22180 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:15:59.891189 22205 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.950786 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3194) took 1418 ms. Trace:
I20260709 12:15:59.951419 12637 rpcz_store.cc:276] 0709 12:15:58.532421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:59.003374 (+470953us) service_pool.cc:224] Handling call
0709 12:15:59.005667 (+  2293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:59.150291 (+144624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:59.950337 (+800046us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:15:59.950658 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":327484160}
I20260709 12:15:59.885803 22319 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.891757 12643 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:15:59.916222 22416 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:15:59.904531 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.001s	sys 0.029s
W20260709 12:15:59.894680 22337 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.209s	user 0.009s	sys 0.013s
I20260709 12:15:59.958544 22416 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:15:59.844928 22425 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.247s	user 0.005s	sys 0.014s
I20260709 12:15:59.774977 22189 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.890594 22239 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:15:59.857822 22432 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:15:59.906198 22010 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.944s	user 0.012s	sys 0.016s
W20260709 12:15:59.940219 22261 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.084s	sys 0.162s
I20260709 12:15:59.902145 22331 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.901953 22400 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:15:59.949523 22312 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.138s	user 0.040s	sys 0.059s
I20260709 12:15:59.911845 22346 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.933787 22348 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.931294 21740 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:15:59.960925 22354 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:15:59.971280 22261 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.084s	sys 0.162s
I20260709 12:15:59.978185 22372 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.977937 22313 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.968011 22239 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:15:59.986651 22261 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:15:59.992043 22354 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.031235 22400 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:00.086710 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.008s	sys 0.011s
W20260709 12:16:00.087867 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.009s	sys 0.012s
W20260709 12:16:00.090034 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 730884us
W20260709 12:15:59.912712 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.004s	sys 0.015s
W20260709 12:16:00.094270 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.004s	sys 0.016s
W20260709 12:15:59.890024 12489 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.018s	sys 0.027s
I20260709 12:16:00.102818 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.111815 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:15:59.904651 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3951) took 1416 ms. Trace:
I20260709 12:16:00.118391 12564 rpcz_store.cc:276] 0709 12:15:58.488415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.500873 (+ 12458us) service_pool.cc:224] Handling call
0709 12:15:59.904384 (+1403511us) spinlock_profiling.cc:243] Waited 66.3 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:15:59.904572 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152601984}
I20260709 12:16:00.122466 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:00.122637 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.123641 12491 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:00.125667 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.126453 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:15:59.890815 12568 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.008s	sys 0.081s
I20260709 12:16:00.127683 12497 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:16:00.127918 12568 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.008s	sys 0.081s
W20260709 12:16:00.128439 12567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4090) took 1835 ms (client timeout 1719 ms). Trace:
W20260709 12:16:00.129001 12567 rpcz_store.cc:269] 0709 12:15:58.292648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.292822 (+   174us) service_pool.cc:224] Handling call
0709 12:16:00.127873 (+1835051us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:00.128163 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818498944}
I20260709 12:16:00.129904 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4096) took 1153 ms. Trace:
I20260709 12:16:00.131723 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.133222 12567 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:00.133298 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:00.130364 12565 rpcz_store.cc:276] 0709 12:15:58.976541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.976735 (+   194us) service_pool.cc:224] Handling call
0709 12:15:58.977093 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:59.190628 (+213535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:00.129214 (+938586us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:00.129687 (+   473us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414037760}
I20260709 12:16:00.136173 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.137874 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.139102 12565 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:00.140151 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:15:59.911630 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.012s	sys 0.008s
W20260709 12:15:59.961351 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.015s	sys 0.014s
W20260709 12:15:59.955677 12638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.011s	sys 0.013s
W20260709 12:15:59.962572 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 704884us
I20260709 12:16:00.233731 22458 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:15:59.982273 21789 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:15:59.972703 22010 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.235587 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.237036 22458 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.238148 22323 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:00.239823 22323 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:00.240393 22323 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:00.240525 22010 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.161293 22261 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.176086 22400 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:00.242911 22400 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.176520 22342 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.177186 22331 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:00.170691 22310 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.682s	user 0.016s	sys 0.011s
I20260709 12:16:00.246690 22310 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:00.248256 22176 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.997s	user 0.013s	sys 0.025s
I20260709 12:16:00.141011 12565 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:16:00.250016 12645 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:00.250890 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.252388 12565 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:00.253455 22036 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:16:00.253815 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.013s	sys 0.009s
I20260709 12:16:00.254144 22036 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:00.257002 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:00.196372 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:00.261336 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:00.261884 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.015s	sys 0.014s
I20260709 12:16:00.201972 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:00.264734 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.267319 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3827) took 2924 ms. Trace:
I20260709 12:16:00.267566 12564 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:00.268748 22346 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.204473 22444 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:00.204905 22349 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:00.204473 22260 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:00.206063 22461 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:00.276234 22260 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.278422 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:00.279088 22260 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.210137 22437 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:00.211184 22447 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:00.221336 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.222108 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:00.228524 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:00.234068 12638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.011s	sys 0.014s
I20260709 12:16:00.148291 21740 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:00.118711 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 363245us
W20260709 12:16:00.264071 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 882850us
I20260709 12:16:00.267866 12640 rpcz_store.cc:276] 0709 12:15:57.342734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.357511 (+1014777us) service_pool.cc:224] Handling call
0709 12:15:58.358260 (+   749us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:58.358749 (+   489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.358793 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:58.366124 (+  7331us) log.cc:844] Serialized 26 byte log entry
0709 12:15:59.775294 (+1409170us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b18001859e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:16:00.265310 (+490016us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:00.266024 (+   714us) log.cc:844] Serialized 14 byte log entry
0709 12:16:00.266413 (+   389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:00.266573 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:00.266626 (+    53us) raft_consensus.cc:1685] finished
0709 12:16:00.266735 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:00.266941 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7332,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":8872,"spinlock_wait_cycles":29789440,"thread_start_us":451234,"threads_started":1,"wal-append.queue_time_us":1653491}
I20260709 12:16:00.259148 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.249426 22310 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.271152 22312 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:00.261492 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:00.272437 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.272360 22444 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.275923 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.288301 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:00.290872 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.290786 22312 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.291532 22447 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.291958 22444 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.293444 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3823) took 3025 ms. Trace:
I20260709 12:16:00.293941 12638 rpcz_store.cc:276] 0709 12:15:57.267913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.237498 (+969585us) service_pool.cc:224] Handling call
0709 12:15:58.456523 (+219025us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:58.456889 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:58.457260 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:15:59.424856 (+967596us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b18001f65a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:15:59.425086 (+   230us) log.cc:844] Serialized 14 byte log entry
0709 12:16:00.292686 (+867600us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:00.292837 (+   151us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:00.293010 (+   173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:00.293148 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:00.293376 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3657,"prepare.queue_time_us":104940,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":3533,"raft.queue_time_us":288,"spinlock_wait_cycles":23050880,"thread_start_us":966529,"threads_started":2,"wal-append.queue_time_us":587391}
I20260709 12:16:00.296368 22179 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:00.297817 22432 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:00.298516 22400 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.299005 22432 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.284924 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4099) took 1182 ms. Trace:
I20260709 12:16:00.300612 12568 rpcz_store.cc:276] 0709 12:15:59.102128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:59.102856 (+   728us) service_pool.cc:224] Handling call
0709 12:15:59.103362 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:59.103687 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:16:00.129017 (+1025330us) log.cc:844] Serialized 14 byte log entry
0709 12:16:00.216780 (+ 87763us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:00.220738 (+  3958us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:00.232878 (+ 12140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:00.258514 (+ 25636us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:00.284851 (+ 26337us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11985,"prepare.queue_time_us":884819,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":4888,"raft.queue_time_us":21182,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":18185,"thread_start_us":997559,"threads_started":2,"wal-append.queue_time_us":129265}
I20260709 12:16:00.249526 22348 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.304972 12492 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:16:00.305235 22323 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.285465 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.307665 12495 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:00.286901 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:00.276908 22461 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.309104 12494 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:16:00.296396 22437 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.311386 22458 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.311868 22437 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.276093 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:00.315702 12488 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:00.271572 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3828) took 2915 ms. Trace:
I20260709 12:16:00.319600 22180 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.318464 12639 rpcz_store.cc:276] 0709 12:15:57.343067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.359799 (+1016732us) service_pool.cc:224] Handling call
0709 12:15:58.360336 (+   537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:15:58.366488 (+  6152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:15:58.366583 (+    95us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:15:58.378277 (+ 11694us) log.cc:844] Serialized 26 byte log entry
0709 12:15:58.382124 (+  3847us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:15:58.535630 (+153506us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:15:58.575285 (+ 39655us) log.cc:844] Serialized 14 byte log entry
0709 12:15:58.575699 (+   414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:15:58.575807 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:15:59.843841 (+1268034us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:00.213592 (+369751us) raft_consensus.cc:1685] finished
0709 12:16:00.214294 (+   702us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:00.258359 (+ 44065us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50772,"prepare.queue_time_us":76508,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":7817,"raft.run_cpu_time_us":2421,"raft.run_wall_time_us":12627,"spinlock_wait_cycles":20240000,"thread_start_us":159400,"threads_started":2,"wal-append.queue_time_us":164757}
I20260709 12:16:00.320196 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:00.322878 22180 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.323354 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:00.257807 22036 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.326112 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.326718 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.283006 12567 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:00.327107 12571 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:00.329082 12567 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:16:00.332614 21740 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:00.333374 21740 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.327204 12570 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:00.338201 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:00.339689 13253 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260709 12:16:00.342677 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.347854 22297 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:00.351691 22297 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:00.352578 22310 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:00.352996 22297 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.355018 22461 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.302874 22432 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.359934 22342 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:00.352728 21740 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.345093 22192 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:00.346454 22232 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:00.360106 22324 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.597s	user 0.009s	sys 0.003s
W20260709 12:16:00.376294 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56330.6us
I20260709 12:16:00.343462 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:00.377290 12489 raft_consensus.cc:606] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:00.377406 22342 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.380913 22342 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.381577 13260 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:00.355041 22297 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:00.377471 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:00.344030 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:00.367265 22192 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:00.397665 12497 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:00.398761 22192 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.400612 22192 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:16:00.403649 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.666s (SENT) [suppressed 13 similar messages]
W20260709 12:16:00.369750 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46624.5us
I20260709 12:16:00.353505 22310 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:00.372686 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54288.8us
I20260709 12:16:00.373466 21922 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.373961 22460 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:16:00.373665 22010 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:16:00.373965 22447 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.374962 22461 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:00.372694 22232 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.391125 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4094) took 1600 ms. Trace:
I20260709 12:16:00.394536 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:00.395761 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:00.420682 12494 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:00.396610 12491 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:00.421605 12571 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:00.415624 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:00.417711 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.397601 13255 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:16:00.423569 12488 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:00.427256 13257 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
W20260709 12:16:00.427831 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50608.5us
I20260709 12:16:00.428980 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:00.429760 22435 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:16:00.420356 12566 rpcz_store.cc:276] 0709 12:15:58.790400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:58.790567 (+   167us) service_pool.cc:224] Handling call
0709 12:15:59.409170 (+618603us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:15:59.409458 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:15:59.409791 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:15:59.908363 (+498572us) log.cc:844] Serialized 14 byte log entry
0709 12:16:00.300026 (+391663us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b1800137bc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:00.389529 (+ 89503us) log.cc:844] Serialized 14 byte log entry
0709 12:16:00.390471 (+   942us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:00.390547 (+    76us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:00.390639 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:00.390742 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:00.390945 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":650549,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":79,"raft.queue_time_us":72984,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":230,"spinlock_wait_cycles":78958208,"thread_start_us":799455,"threads_started":2,"wal-append.queue_time_us":331785}
W20260709 12:16:00.434592 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.434051 22310 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.435550 22354 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.426184 22460 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:00.436657 22461 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.438158 22460 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.439213 12565 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:16:00.442060 22232 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.442864 12497 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:00.444334 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:00.457904 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.434118 21922 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:00.456350 22342 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:00.466923 22008 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:00.470419 21922 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:00.482204 21922 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:00.484057 22461 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.466867 22349 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:00.485615 22349 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:00.486188 22349 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:00.490491 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.492014 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.492923 12566 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:16:00.498301 22097 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:00.500018 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71276.1us
I20260709 12:16:00.500900 22319 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.503026 12641 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 26. Current term is 27. Ops: []
W20260709 12:16:00.503327 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44537.2us
I20260709 12:16:00.505235 12640 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:00.480624 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3839) took 1181 ms. Trace:
I20260709 12:16:00.503362 12646 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:00.506542 22342 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:00.507359 22189 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.507916 22239 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.511346 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:00.506481 12645 rpcz_store.cc:276] 0709 12:15:59.299359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:59.884032 (+584673us) service_pool.cc:224] Handling call
0709 12:16:00.462709 (+578677us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:00.480555 (+ 17846us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701836032}
I20260709 12:16:00.517841 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:00.525959 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62670.5us
W20260709 12:16:00.479290 22183 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.673s	user 0.011s	sys 0.011s
I20260709 12:16:00.532899 22180 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 12:16:00.535380 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:00.535939 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.536350 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:00.538625 21789 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:00.539237 21789 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:00.540381 21789 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.13, Last appended by leader: 13, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.541652 22432 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:00.542228 22220 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:00.542779 12637 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:00.542971 22220 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.553067 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.542935 22432 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:16:00.554124 22097 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:16:00.556710 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.570106 12571 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:00.571290 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 227672us
I20260709 12:16:00.564163 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.556999 22432 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:00.574352 22432 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.560282 22312 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.577042 12497 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:00.562983 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.565184 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.581885 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.576453 22432 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:00.554729 22220 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.605597 12570 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:16:00.561693 22205 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.562489 22180 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:00.631587 12461 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:00.615327 22180 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:00.638314 12497 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 17. (term mismatch)
I20260709 12:16:00.638896 12497 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 16
I20260709 12:16:00.639676 12461 leader_election.cc:400] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:16:00.640049 12461 leader_election.cc:403] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:00.635114 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:00.625059 22297 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:00.645344 22471 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:16:00.647348 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1761) took 1065 ms. Trace:
I20260709 12:16:00.639122 12497 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:16:00.649919 12497 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:16:00.655691 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 252104us
I20260709 12:16:00.630430 22342 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:00.663286 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:00.635324 22097 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:16:00.671002 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:00.671032 22342 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.674623 22342 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.678156 22322 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:00.678669 22322 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:00.679045 22322 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:16:00.684458 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138422us
I20260709 12:16:00.692852 12645 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:00.693856 22441 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:16:00.702100 22343 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.703778 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 3957) took 1052 ms. Trace:
W20260709 12:16:00.704645 22469 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:16:00.706329 22323 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:00.706576 12563 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:00.704304 12572 rpcz_store.cc:276] 0709 12:15:59.650716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:59.755098 (+104382us) service_pool.cc:224] Handling call
0709 12:16:00.703331 (+948233us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:00.703545 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1480516992}
I20260709 12:16:00.647807 12564 rpcz_store.cc:276] 0709 12:15:59.582044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:00.268606 (+686562us) service_pool.cc:224] Handling call
0709 12:16:00.269133 (+   527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:00.369160 (+100027us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:00.369209 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:00.382094 (+ 12885us) log.cc:844] Serialized 26 byte log entry
0709 12:16:00.445471 (+ 63377us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:00.445507 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:00.445574 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:00.647061 (+201487us) raft_consensus.cc:1685] finished
0709 12:16:00.647089 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:00.647271 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32211,"raft.queue_time_us":43201,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":150,"thread_start_us":62850,"threads_started":1,"wal-append.queue_time_us":145248}
W20260709 12:16:00.727381 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.645452 22297 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:00.732062 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61674.5us
I20260709 12:16:00.756946 12637 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:00.767850 22372 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:00.768465 22372 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:00.768594 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135422us
I20260709 12:16:00.769891 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:00.771025 22451 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:00.773185 12497 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: [25.20-25.20]
W20260709 12:16:00.789731 13218 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.111s (SENT)
W20260709 12:16:00.793387 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80426.6us
I20260709 12:16:00.794406 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:00.796258 22372 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.797488 22372 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:00.800772 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:00.807274 22297 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:00.814616 22349 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.816880 22349 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:00.818536 22036 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:00.819691 22180 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.826434 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:00.827601 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:00.828227 12645 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 24.
I20260709 12:16:00.821511 22180 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:16:00.827306 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.832095 12488 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:16:00.841849 22343 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:00.842622 22343 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.855897 12497 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:00.856022 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:00.858631 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88921.1us
I20260709 12:16:00.845590 22343 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:00.855051 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.846573 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:00.875039 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85362.4us
I20260709 12:16:00.875200 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:00.880252 12494 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:16:00.888504 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:00.889106 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.896730 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:00.896580 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:00.896890 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:00.899416 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:00.899720 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:00.900177 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:00.903123 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:00.910725 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:00.861979 21922 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:00.924623 22097 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:00.933493 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.923722 21789 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:00.939419 22297 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:16:00.934314 22472 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.268s	user 0.000s	sys 0.006s
I20260709 12:16:00.948189 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:00.948740 22275 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:00.949937 22275 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:00.950528 22275 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:00.947225 22297 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:00.952001 22275 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.950171 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:00.953806 22275 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:00.952188 22297 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:00.943084 21789 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:00.974049 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76211.4us
W20260709 12:16:00.956593 22192 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.555s	user 0.007s	sys 0.009s
I20260709 12:16:00.981288 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:00.989142 22480 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:16:00.990638 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:00.997601 13254 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:01.001603 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.005970 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.017962 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.018958 12497 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:00.973153 22310 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:16:01.041297 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:01.042686 22479 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:16:01.043012 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.043871 12637 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:01.044683 22441 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:01.045583 22324 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:01.041344 13255 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 20. (term mismatch)
I20260709 12:16:01.046362 13255 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 19
I20260709 12:16:01.046867 13255 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 25.20
I20260709 12:16:01.047166 13255 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:01.048477 12489 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
W20260709 12:16:01.053619 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:01.054253 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 181176us
I20260709 12:16:01.042809 12389 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:01.055122 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:01.055519 22232 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:01.069077 21789 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 28.10, Last appended by leader: 10, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.066560 22232 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:01.075241 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:01.076267 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:01.078732 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 130) took 21192 ms (client timeout 15000 ms). Trace:
W20260709 12:16:01.090629 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119274us
I20260709 12:16:01.095183 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.096181 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.098934 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.099061 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.102813 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:01.104780 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120927us
I20260709 12:16:01.105974 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.106508 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:01.107209 12489 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 33 because replica has last-logged OpId of term: 31 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 10.
I20260709 12:16:01.107170 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.108058 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.046320 22324 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.108831 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:01.101924 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.077601 22232 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:01.099731 12496 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:01.111325 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.113960 12492 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:01.115653 12637 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:01.119807 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.100167 21922 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:01.120481 12646 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:01.079706 22322 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.124230 12494 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:01.103387 12389 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:16:01.116560 22477 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.218s	user 0.002s	sys 0.000s
I20260709 12:16:01.147274 12639 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:01.140734 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:01.116828 22342 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:01.158327 13257 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:01.158418 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.161724 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:01.166560 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.903s (SENT)
I20260709 12:16:01.167716 12646 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:01.161325 22232 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.175477 22232 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:01.171132 22460 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:01.182461 22460 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.185178 22460 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.188859 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:01.192387 22342 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:01.196441 22324 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:01.202530 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89333.1us
I20260709 12:16:01.205920 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:01.207746 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.203361 12642 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:01.211771 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.218364 12640 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:01.220008 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49939.2us
I20260709 12:16:01.226187 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:01.232201 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69731.1us
I20260709 12:16:01.160995 22205 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:01.237345 22372 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:01.250242 22314 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:01.251430 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:01.224924 22460 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:01.254045 22205 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.256608 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.258055 22205 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:01.169159 22323 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.070s
I20260709 12:16:01.259326 22323 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:01.259873 22323 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:01.254179 22460 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:01.263105 22460 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.264025 22372 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:01.264510 22372 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.264894 22372 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:01.267002 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:01.270310 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:01.270295 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93026.8us
I20260709 12:16:01.280638 12637 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:01.283049 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:01.279831 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137387us
W20260709 12:16:01.079144 12386 rpcz_store.cc:269] 0709 12:15:39.734049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:48.444755 (+8710706us) service_pool.cc:224] Handling call
0709 12:16:00.923969 (+12479214us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:15:58.215688 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:15:58.216506 (+   818us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:15:58.216554 (+    48us) write_op.cc:435] Acquired schema lock
0709 12:15:58.216570 (+    16us) tablet.cc:662] Decoding operations
0709 12:15:58.222290 (+  5720us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:15:58.222480 (+   190us) write_op.cc:641] Partition lock acquired for write op
0709 12:15:58.222523 (+    43us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:15:58.338098 (+115575us) tablet.cc:701] Row locks acquired
0709 12:15:58.338157 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:15:58.338347 (+   190us) write_op.cc:270] Start()
0709 12:15:58.338527 (+   180us) write_op.cc:276] Timestamp: P: 1783599358338273 usec, L: 0
0709 12:15:58.338547 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:15:58.339384 (+   837us) log.cc:844] Serialized 5962 byte log entry
0709 12:15:59.134088 (+794704us) op_driver.cc:464] REPLICATION: finished
0709 12:15:59.807701 (+673613us) write_op.cc:301] APPLY: starting
0709 12:15:59.807818 (+   117us) tablet.cc:1367] starting BulkCheckPresence
0709 12:15:59.817221 (+  9403us) tablet.cc:1370] finished BulkCheckPresence
0709 12:15:59.817389 (+   168us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:00.037255 (+219866us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b18000af9c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf55320ec 00007ffbf5534af7 00007ffbf5535fb0 00007ffbf924a1bb 00007ffbf92915d4 00007ffbf913d8da 00007ffbf91403cb 00007ffbf913ffd3 00007ffbf923ee34
0709 12:16:00.039824 (+  2569us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:00.040059 (+   235us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=0,mrs_lookups=0
0709 12:16:00.040093 (+    34us) write_op.cc:312] APPLY: finished
0709 12:16:00.040405 (+   312us) log.cc:844] Serialized 139 byte log entry
0709 12:16:00.253970 (+213565us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:00.255015 (+  1045us) write_op.cc:454] Released schema lock
0709 12:16:00.255203 (+   188us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:00.256820 (+  1617us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:00.256857 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":649337,"apply.run_cpu_time_us":29310,"apply.run_wall_time_us":448997,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66767,"num_ops":15,"prepare.queue_time_us":394235,"prepare.run_cpu_time_us":12239,"prepare.run_wall_time_us":320368,"raft.queue_time_us":373575,"raft.run_cpu_time_us":55760,"raft.run_wall_time_us":342207,"replication_time_us":795316,"spinlock_wait_cycles":33037056,"thread_start_us":955186,"threads_started":5,"wal-append.queue_time_us":205573,"wal-append.run_cpu_time_us":31548,"wal-append.run_wall_time_us":2395249}]]}
W20260709 12:16:01.291124 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:01.291246 12640 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.15-31.15]
I20260709 12:16:01.261132 22323 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:01.301259 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:01.302798 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.303812 12645 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:16:01.299082 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:01.301028 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 227) took 17110 ms (client timeout 15000 ms). Trace:
W20260709 12:16:01.312692 12386 rpcz_store.cc:269] 0709 12:15:44.190308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:01.300709 (+17110401us) service_pool.cc:211] Skipping call since client already timed out
0709 12:16:01.300899 (+   190us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:16:01.284946 22312 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:01.314016 12639 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:01.297957 12494 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 32 because replica has last-logged OpId of term: 31 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 14.
I20260709 12:16:01.311066 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:01.326385 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:01.327538 22484 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:16:01.327951 22343 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:01.335008 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:01.338599 22312 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.339411 22487 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:01.339989 22487 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.342834 22487 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.344152 13259 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
W20260709 12:16:01.347112 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:01.365162 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:01.329452 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:01.314556 22180 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:16:01.369128 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90000us
I20260709 12:16:01.366205 13254 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:01.374162 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:01.376415 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.377065 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.377272 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.378026 12563 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:16:01.379930 13255 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
W20260709 12:16:01.382552 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101145us
I20260709 12:16:01.318207 22460 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:16:01.386530 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.389403 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:01.390278 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 80957.6us
I20260709 12:16:01.391484 22205 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:01.391897 22205 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.392252 22205 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:01.400328 22314 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:01.404902 22232 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:01.405362 22232 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:01.382606 22323 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:01.406323 22232 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.400740 22242 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:01.407582 22232 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:01.409998 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 27 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.414345 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:01.318307 22342 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:01.416201 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:01.420042 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.420598 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:01.423204 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.413018 22343 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.003s
I20260709 12:16:01.416553 21922 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.011s
I20260709 12:16:01.388242 22486 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:01.430097 21922 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:16:01.430330 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46221.8us
I20260709 12:16:01.431588 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:01.430332 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113865us
I20260709 12:16:01.433507 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:01.435087 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:01.436452 22312 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:01.436538 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.437265 22312 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:01.437920 22441 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:01.454831 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.454931 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.439092 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:01.439894 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.440045 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.459091 12571 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:01.440624 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:01.442019 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.443424 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.443832 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.461982 12567 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:01.444609 22322 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.453843 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.460667 12566 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:01.454239 22310 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:01.474080 12564 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:01.474925 12572 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:01.477103 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.479559 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.480571 12491 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:01.482159 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:01.483716 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.483887 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.484743 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:01.480887 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:01.489743 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:01.490666 12570 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:01.492264 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.484760 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:01.494570 13254 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:01.493469 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:01.492628 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.497339 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.497762 22349 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:01.499831 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.465965 22441 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:01.500973 22441 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:01.473109 22486 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:16:01.501691 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:01.502017 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.503274 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:01.503325 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:01.504981 22242 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:01.505366 22242 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:01.505795 22242 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:01.506042 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50138.3us
I20260709 12:16:01.456096 22297 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:01.505849 22322 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:16:01.512377 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 121438us
I20260709 12:16:01.516911 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.515980 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.520859 22310 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:01.521536 22310 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.521996 22310 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:01.526072 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.427356 22487 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.067s
I20260709 12:16:01.528398 12492 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:01.455952 21922 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:16:01.539584 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:01.541335 12493 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:01.538739 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.527419 22297 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:01.538486 22432 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:01.568499 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129531us
I20260709 12:16:01.550712 22342 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:01.574463 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:01.551163 22486 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:01.570972 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.540325 22372 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.585695 12563 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
W20260709 12:16:01.588308 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 174106us
W20260709 12:16:01.581607 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95800.6us
W20260709 12:16:01.585330 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76909.9us
I20260709 12:16:01.587946 22277 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:01.600711 12572 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:16:01.600466 22277 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:01.615957 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 93143.3us
I20260709 12:16:01.623404 12495 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:01.574259 22432 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.644620 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:01.648029 12572 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:01.633288 22506 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:01.656415 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:01.665611 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:01.668078 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.679571 22487 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:01.675819 22499 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:01.686367 22499 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.688995 22499 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:01.690693 22180 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:01.695391 22372 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.658957 22297 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.698545 22205 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.701041 22205 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:01.708897 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.712121 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:01.715123 22496 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election decision vote started in defunct term 30: won
W20260709 12:16:01.712492 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45322.8us
W20260709 12:16:01.717522 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94247.1us
I20260709 12:16:01.718727 22501 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:01.719404 22501 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.722949 12496 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:01.727316 12570 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:01.731186 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:01.732100 12570 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:01.732378 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:01.734206 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.737298 22460 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:01.708381 22486 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.738883 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:01.740586 12569 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 13. Preceding OpId from leader: term: 30 index: 14. (index mismatch)
I20260709 12:16:01.702880 22343 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:16:01.742296 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83415.2us
I20260709 12:16:01.742990 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:01.745061 13257 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:01.746933 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.746058 22460 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:01.750399 22460 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:01.750821 22460 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:01.758885 22441 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.764341 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.765826 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.766779 12564 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:01.766559 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.675488 22506 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.768405 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.769378 22312 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:01.769686 22506 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.770123 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.766001 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:01.771471 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.781708 13257 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:01.780004 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:01.781853 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50178.4us
I20260709 12:16:01.788233 12570 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:01.789288 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.791227 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:01.736475 22501 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:01.794535 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49226.2us
I20260709 12:16:01.761430 22441 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:01.800695 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:01.802942 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 141430us
I20260709 12:16:01.802013 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:01.802237 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:01.744503 22343 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:01.806640 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.804557 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.812674 12564 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:01.811161 22205 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:01.812667 13262 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:16:01.787536 22432 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.806627 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:01.782636 22504 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:01.828755 12609 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:16:01.809099 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.827963 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:01.844555 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:01.810917 21922 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:01.810209 22502 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:01.810058 22496 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:01.850276 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:01.832373 22441 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:01.856849 22441 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.857317 22441 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:01.864182 22310 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.876992 12637 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:01.878041 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:01.866009 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62290.5us
I20260709 12:16:01.873425 22242 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:01.831908 22501 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:01.878947 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.879531 22501 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.880030 22501 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:01.880203 12645 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:01.881423 13259 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:01.867218 22277 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:01.866393 22492 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:01.867372 22504 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:01.803884 22180 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:16:01.872323 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:01.883502 12564 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:16:01.874169 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 133204us
I20260709 12:16:01.853081 22297 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:01.885386 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.886030 22297 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.883527 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.879132 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.888173 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.878126 22310 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:01.890153 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:01.865904 22097 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.029s
I20260709 12:16:01.895637 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.895949 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:01.904313 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:01.905040 12637 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:01.851545 22496 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:01.886476 22297 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:01.933298 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:01.935279 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 113622us
I20260709 12:16:01.940431 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:01.903910 22275 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:16:01.948066 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:01.947474 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 138509us
I20260709 12:16:01.956674 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:01.907816 22324 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:01.963274 12640 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:16:01.933431 22180 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:01.963217 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84178.8us
I20260709 12:16:01.962309 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:01.976830 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.933723 22496 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:01.966746 22324 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:01.981114 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131595us
I20260709 12:16:01.982246 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:01.987958 12642 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:01.879040 22343 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:01.993510 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:01.998426 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 113123us
I20260709 12:16:01.974534 22492 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:02.003422 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:01.936761 22310 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:16:01.975245 22275 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:01.986652 22324 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:02.015731 12640 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 10. Preceding OpId from leader: term: 30 index: 11. (index mismatch)
I20260709 12:16:02.016180 12644 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:02.002522 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:01.974179 22487 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:16:02.024688 12637 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:02.018774 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:02.004215 22205 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:02.027573 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:02.030118 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:02.030896 12637 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:02.031442 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:01.996615 22242 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:02.009495 22492 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:01.995831 22518 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:02.003995 22506 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:02.028419 12643 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:02.040727 22492 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:02.043465 22275 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.045822 22343 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:16:02.047742 22507 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:02.048420 22507 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:02.024399 21922 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:02.049450 22507 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.050537 21922 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.023550 21789 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:02.053931 21789 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.037227 22508 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:16:02.055869 22491 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:02.056413 22491 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:02.057065 22491 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:02.057582 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:02.039912 22180 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:02.059072 22314 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:02.059701 22314 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.062091 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:02.062906 12641 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:02.066826 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:02.067823 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:02.068470 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:02.072044 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:02.074262 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:02.074961 12644 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:02.075552 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44188.2us
I20260709 12:16:02.076839 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.077476 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:16:02.082547 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80113.1us
W20260709 12:16:02.082934 22510 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:16:02.083439 22515 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:02.085147 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.074252 22314 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:02.089051 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.090521 12494 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:02.091295 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:02.089685 22097 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:02.041059 22486 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:02.100081 12492 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:16:02.105034 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.106118 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.108651 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:02.109539 12490 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:16:02.128464 22232 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:02.129456 22232 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:02.129999 22232 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:02.111160 21789 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:02.113137 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:02.113935 12563 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 10. Preceding OpId from leader: term: 30 index: 11. (index mismatch)
I20260709 12:16:02.110033 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.134568 12495 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:02.136204 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.120563 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:02.103874 22097 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:02.120772 22520 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:16:02.115501 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:02.106429 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.140056 22097 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:02.105587 22499 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:16:02.140906 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:02.114501 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57023.2us
I20260709 12:16:02.141822 12646 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:02.142020 12637 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:16:02.142729 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:02.142158 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:02.118513 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.134740 22519 raft_consensus.cc:2764] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:16:02.144800 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.145458 12492 raft_consensus.cc:2376] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26. Re-sending same reply.
I20260709 12:16:02.140966 22499 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:02.146445 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.146685 22499 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:02.148042 22499 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:02.149739 12637 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:16:02.152496 12641 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:02.104955 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:02.073410 22506 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:02.160126 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65535.7us
I20260709 12:16:02.161033 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:02.156370 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:02.137287 12490 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:02.163837 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91879.2us
I20260709 12:16:02.164472 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:02.164729 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:02.173055 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.176651 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.103231 22486 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:02.217348 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.191685 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:02.091249 22297 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:02.166651 22504 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:16:02.175442 22513 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:16:02.224128 12490 raft_consensus.cc:2376] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24. Re-sending same reply.
I20260709 12:16:02.169312 22520 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:02.180105 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:02.216102 12492 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:02.213053 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46876.6us
I20260709 12:16:02.232857 22522 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:02.233346 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:02.234272 12492 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:02.233304 22522 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:02.235425 22522 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:02.237215 22520 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:02.209882 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45499.7us
I20260709 12:16:02.231607 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:02.242470 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62512.3us
I20260709 12:16:02.243407 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:02.242493 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:02.247430 12563 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:02.196012 22508 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:02.248921 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:02.227358 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:02.201825 22519 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:02.222901 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:02.255307 13259 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:02.166887 22507 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 12:16:02.214818 22502 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:02.331339 13260 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 22. (term mismatch)
I20260709 12:16:02.331930 13260 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 21
I20260709 12:16:02.332194 13260 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 32.22
I20260709 12:16:02.332475 13260 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:02.226819 22486 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:02.333192 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119160us
I20260709 12:16:02.335664 12494 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:02.342937 22534 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:16:02.343137 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99821.3us
I20260709 12:16:02.363703 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:02.350894 12496 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:02.226394 22515 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:02.197933 22460 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.231418 22504 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:02.249550 22349 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:02.380966 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:02.290203 22314 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:02.226869 22312 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:02.210856 22441 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:02.214980 22518 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.138s
I20260709 12:16:02.360419 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:02.330775 22502 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:02.393690 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:02.222304 22492 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.408475 12567 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:02.354885 22487 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:02.398325 12494 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 28 index: 23. (index mismatch)
I20260709 12:16:02.385807 22504 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:02.399008 12497 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:02.252877 21789 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.119s
I20260709 12:16:02.412230 12495 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:16:02.412743 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58382.4us
I20260709 12:16:02.418416 12491 raft_consensus.cc:2368] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
W20260709 12:16:02.418712 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 191439us
I20260709 12:16:02.421504 12496 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:02.424197 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:02.420555 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:02.420707 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72269.6us
I20260709 12:16:02.407218 22515 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:02.406824 22502 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:02.434244 12566 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
W20260709 12:16:02.439141 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207597us
I20260709 12:16:02.270200 22205 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:02.454057 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:02.414348 22487 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:02.417915 22312 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:02.419716 22519 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:02.458668 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:02.504892 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46359.3us
W20260709 12:16:02.507637 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48832.9us
I20260709 12:16:02.443399 22349 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:02.443058 22323 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:02.502058 22487 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:02.537003 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:02.508240 22519 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:02.514544 22314 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:02.500100 12644 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:02.524958 22507 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:02.523218 22323 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:02.523165 22349 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:02.548038 22343 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.000s
I20260709 12:16:02.544893 22519 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:02.718225 22505 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:16:02.718446 22521 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.360s	user 0.002s	sys 0.000s
W20260709 12:16:02.719844 22527 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
W20260709 12:16:02.915810 22232 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.785s	user 0.020s	sys 0.004s
W20260709 12:16:02.809289 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.009s	sys 0.003s
W20260709 12:16:02.959362 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.010s	sys 0.004s
W20260709 12:16:02.851857 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 470999us
I20260709 12:16:02.522739 22192 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.022s
W20260709 12:16:02.853040 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:02.718209 22537 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:02.853840 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 426303us
W20260709 12:16:02.854401 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 317502us
W20260709 12:16:02.878686 12490 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.007s	sys 0.010s
I20260709 12:16:02.962826 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:02.963330 12490 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.007s	sys 0.010s
I20260709 12:16:02.963658 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:02.908877 21789 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:16:02.947947 22504 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.001s
W20260709 12:16:02.914683 12641 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.762s	user 0.011s	sys 0.007s
I20260709 12:16:02.915922 22242 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:02.524559 22342 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:16:02.911150 12489 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.753s	user 0.010s	sys 0.007s
I20260709 12:16:02.969594 12489 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:02.947077 22349 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:02.947885 22501 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.067s	user 0.029s	sys 0.020s
I20260709 12:16:02.948503 12638 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 22.
W20260709 12:16:02.951578 12495 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.529s	user 0.017s	sys 0.006s
I20260709 12:16:02.991636 12495 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:16:02.628504 22451 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.088s
W20260709 12:16:02.908043 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 450929us
I20260709 12:16:02.906275 22314 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:02.948963 22496 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.947s	user 0.009s	sys 0.012s
I20260709 12:16:02.967686 12641 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:16:02.713327 22514 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:16:02.951078 22522 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.715s	user 0.011s	sys 0.011s
W20260709 12:16:02.949618 22528 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.410s	user 0.004s	sys 0.006s
I20260709 12:16:02.970477 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:02.973064 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:02.919530 22323 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:02.954445 22507 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:02.966807 22232 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.087492 13257 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:03.088042 22514 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:03.089105 22514 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:03.089483 22514 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:03.090399 22514 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 31.15, Last appended by leader: 15, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.091794 22515 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:03.092563 22515 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:02.974195 22275 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.020s	sys 0.020s
W20260709 12:16:03.014012 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 498161us
I20260709 12:16:02.976363 22312 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:02.967336 22460 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:03.017974 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54439.3us
I20260709 12:16:03.108536 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:02.967555 22297 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:03.105108 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132123us
I20260709 12:16:03.119653 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:03.110401 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.666s (SENT)
I20260709 12:16:03.023015 21789 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:03.017123 22546 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:03.152635 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44116.6us
W20260709 12:16:03.122438 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159552us
I20260709 12:16:03.157831 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:03.099200 22275 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.977s	user 0.021s	sys 0.021s
I20260709 12:16:02.993257 22242 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.160974 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4150) took 1275 ms. Trace:
I20260709 12:16:03.087626 22232 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:02.978948 22504 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:02.995155 22501 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.249565 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1177) took 1082 ms. Trace:
I20260709 12:16:03.159408 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1384) took 1055 ms. Trace:
I20260709 12:16:03.091905 22514 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:03.253871 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:03.153426 22192 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:03.015667 22496 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:03.282285 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 162766us
I20260709 12:16:03.063546 22522 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.159288 22492 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:03.256711 13218 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.615s (SENT)
I20260709 12:16:03.095486 22349 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:03.158541 21789 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:03.123987 21922 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:03.286384 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.253086 13254 rpcz_store.cc:276] 0709 12:16:02.099099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:02.099306 (+   207us) service_pool.cc:224] Handling call
0709 12:16:02.105092 (+  5786us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:02.105759 (+   667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:02.105803 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:02.106567 (+   764us) log.cc:844] Serialized 26 byte log entry
0709 12:16:02.256014 (+149447us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:02.256084 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:02.256161 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:03.181479 (+925318us) raft_consensus.cc:1685] finished
0709 12:16:03.181515 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:03.181801 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2252,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":148964,"threads_started":1,"wal-append.queue_time_us":320171}
I20260709 12:16:03.290522 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:03.254931 12493 rpcz_store.cc:276] 0709 12:16:02.077086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:02.077213 (+   127us) service_pool.cc:224] Handling call
0709 12:16:02.086566 (+  9353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:02.086781 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:16:02.125284 (+ 38503us) log.cc:844] Serialized 14 byte log entry
0709 12:16:03.130085 (+1004801us) log.cc:844] Serialized 14 byte log entry
0709 12:16:03.131073 (+   988us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:03.131142 (+    69us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:03.131214 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:03.131318 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:03.132249 (+   931us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":40761,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":685,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":29778,"thread_start_us":1041981,"threads_started":3,"wal-append.queue_time_us":851987}
I20260709 12:16:03.222971 12564 rpcz_store.cc:276] 0709 12:16:01.885229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:01.894121 (+  8892us) service_pool.cc:224] Handling call
0709 12:16:01.894582 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:01.915347 (+ 20765us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:01.915404 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:01.941606 (+ 26202us) log.cc:844] Serialized 26 byte log entry
0709 12:16:02.101516 (+159910us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:02.101564 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:02.101678 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:03.160719 (+1059041us) raft_consensus.cc:1685] finished
0709 12:16:03.160742 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:03.160902 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10918,"raft.queue_time_us":10607,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":5702,"thread_start_us":112828,"threads_started":1,"wal-append.queue_time_us":205911}
I20260709 12:16:03.291899 12645 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:03.006791 22537 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.124235 22441 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.259305 22460 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:03.271633 22343 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:03.266029 22192 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:03.313776 12639 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:03.272514 22432 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:03.254130 22543 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:16:03.316010 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159913us
I20260709 12:16:03.255491 22536 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.285126 22342 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:03.349826 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:03.324910 21922 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:03.328322 22491 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.271s	user 0.001s	sys 0.027s
I20260709 12:16:03.352713 22451 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.356278 22549 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:03.356617 22451 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.357426 22545 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.357966 22545 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.359092 22451 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:03.330940 22502 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.818s	user 0.011s	sys 0.010s
W20260709 12:16:03.359597 22529 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.619s	user 0.000s	sys 0.005s
I20260709 12:16:03.359910 22502 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.362074 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1272) took 1160 ms. Trace:
W20260709 12:16:03.362496 22548 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 12:16:03.300457 22538 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.449s
I20260709 12:16:03.362556 13262 rpcz_store.cc:276] 0709 12:16:02.201789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:02.201966 (+   177us) service_pool.cc:224] Handling call
0709 12:16:02.210457 (+  8491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:02.210900 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:02.210939 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:02.211577 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:16:02.422977 (+211400us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:02.423021 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:02.423094 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:03.084842 (+661748us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:03.361716 (+276874us) raft_consensus.cc:1685] finished
0709 12:16:03.361739 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:03.361990 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3746,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":8151,"spinlock_wait_cycles":14284160,"thread_start_us":210869,"threads_started":1,"wal-append.queue_time_us":245951}
W20260709 12:16:03.364095 22533 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.447s	user 0.000s	sys 0.002s
W20260709 12:16:03.332527 22499 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.184s	user 0.024s	sys 0.004s
I20260709 12:16:03.347488 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:03.331189 22486 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.960s	user 0.017s	sys 0.009s
I20260709 12:16:03.218076 22546 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.365581 22486 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.325726 21789 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:03.354735 22536 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:03.337280 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118281us
I20260709 12:16:03.357038 22549 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:03.370568 22523 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.160s	user 0.002s	sys 0.002s
I20260709 12:16:03.356359 22491 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.371282 22549 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.363756 22312 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:03.366746 22342 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.375898 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3182) took 1216 ms. Trace:
I20260709 12:16:03.376333 12492 rpcz_store.cc:276] 0709 12:16:02.162505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:02.236305 (+ 73800us) service_pool.cc:224] Handling call
0709 12:16:02.242687 (+  6382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:02.528168 (+285481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:02.528207 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:02.619336 (+ 91129us) log.cc:844] Serialized 26 byte log entry
0709 12:16:03.101672 (+482336us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:03.101720 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:03.101814 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:03.375638 (+273824us) raft_consensus.cc:1685] finished
0709 12:16:03.375663 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:03.375829 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7007,"raft.queue_time_us":103340,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":415066,"threads_started":2,"wal-append.queue_time_us":423374}
I20260709 12:16:03.154670 22515 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:03.351269 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65033.2us
W20260709 12:16:03.343012 22324 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.314s	user 0.015s	sys 0.006s
I20260709 12:16:03.356076 22552 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:03.332664 22460 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:03.344242 22487 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.799s	user 0.011s	sys 0.006s
I20260709 12:16:03.395531 12489 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:03.396324 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:03.392709 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45316.3us
I20260709 12:16:03.398069 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.398927 12495 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:03.395758 12497 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:03.407743 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:03.408502 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3856) took 1457 ms. Trace:
I20260709 12:16:03.408820 12564 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:03.342731 22492 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:03.410065 12645 rpcz_store.cc:276] 0709 12:16:01.933619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:01.933763 (+   144us) service_pool.cc:224] Handling call
0709 12:16:03.390856 (+1457093us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:03.391056 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3077187456}
I20260709 12:16:03.411921 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:03.413213 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:03.379513 21922 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:03.422127 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.425382 12492 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:03.425798 12491 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:16:03.411298 22520 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:03.427325 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 170687us
I20260709 12:16:03.434901 12493 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:16:03.435364 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.407450 22460 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:03.441990 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:03.290442 22314 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:03.443861 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67380.3us
I20260709 12:16:03.445956 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.435986 12488 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:03.448249 12497 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 21. Current term is 22. Ops: []
I20260709 12:16:03.449088 12495 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:03.411491 22372 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:03.412016 22275 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:03.450560 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:03.451241 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 335569us
I20260709 12:16:03.461681 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.462836 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:03.447247 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:03.458031 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:03.466025 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.424818 22312 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:03.469146 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:03.397447 22324 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.430918 22518 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:03.472710 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:03.416167 22487 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.474962 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.455178 22515 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:03.481572 12495 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 25 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:16:03.484620 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:03.484916 12497 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:03.485461 22559 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:16:03.489909 22501 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:03.490504 22564 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:16:03.492720 22487 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.494756 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:03.420274 22492 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.496656 12646 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:03.508534 12567 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:03.511600 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:03.518023 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:03.519074 12497 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:16:03.454746 22520 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.496580 22507 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.511435 22504 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:03.443171 22192 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.528968 22507 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.535143 22507 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.529135 22522 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:03.503204 22491 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:03.480686 22312 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.524204 22242 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.534029 22324 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.539975 22312 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:03.540675 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:16:03.540815 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:03.534032 22558 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:16:03.542618 22546 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.542955 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:03.524958 22537 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:03.543630 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3183) took 1380 ms. Trace:
I20260709 12:16:03.543960 12642 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:16:03.547961 22538 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.548883 22487 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:03.549340 22487 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:03.549741 22487 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:03.544236 22536 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.503748 22518 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.553371 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.553674 22518 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:03.554386 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.555043 12496 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:03.555857 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.558024 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.548554 22538 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.559434 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:03.560817 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.559461 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:03.574287 12492 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:03.575044 12488 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:03.561506 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:03.505136 22372 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:03.564523 12496 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:16:03.634423 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:03.755239 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.757200 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.544028 12490 rpcz_store.cc:276] 0709 12:16:02.165532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:02.244937 (+ 79405us) service_pool.cc:224] Handling call
0709 12:16:02.245214 (+   277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:02.968943 (+723729us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:02.968989 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:03.005209 (+ 36220us) log.cc:844] Serialized 26 byte log entry
0709 12:16:03.271776 (+266567us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:16:03.271813 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:03.271907 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:03.543375 (+271468us) raft_consensus.cc:1685] finished
0709 12:16:03.543399 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:03.543560 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11884,"raft.queue_time_us":101074,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":70,"thread_start_us":828245,"threads_started":2,"wal-append.queue_time_us":234300}
I20260709 12:16:03.765312 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.767496 22537 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:03.769361 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.770722 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:03.771788 12495 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:16:03.543309 22520 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:03.566720 22297 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.741101 22349 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.782361 12568 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:16:03.753669 22546 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:03.793491 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:03.798179 12565 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:03.774144 22312 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:03.779433 22522 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:03.769673 22518 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:16:03.832206 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.817462 22537 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:03.795595 22501 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:16:03.821765 22520 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.796437 22297 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.801854 22349 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.819942 22572 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:03.870309 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3849) took 1987 ms (client timeout 2205 ms). Trace:
I20260709 12:16:03.870961 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3863) took 1856 ms. Trace:
W20260709 12:16:03.871021 12638 rpcz_store.cc:269] 0709 12:16:01.883660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:01.887917 (+  4257us) service_pool.cc:224] Handling call
0709 12:16:03.870032 (+1982115us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:03.870207 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2436821376}
I20260709 12:16:03.873320 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.871589 12639 rpcz_store.cc:276] 0709 12:16:02.014180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:02.014443 (+   263us) service_pool.cc:224] Handling call
0709 12:16:02.014829 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:03.341554 (+1326725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:03.870603 (+529049us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:03.870871 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2986668416}
I20260709 12:16:03.830255 22492 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:03.867753 22519 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.899s	user 0.012s	sys 0.001s
W20260709 12:16:03.881116 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 425909us
W20260709 12:16:03.897987 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 437123us
I20260709 12:16:03.828848 22571 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:03.813122 22546 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.852578 22242 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:03.913988 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.867501 22499 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:03.923935 12488 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:16:03.914088 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 463624us
I20260709 12:16:03.928983 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:03.930513 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:03.888345 22522 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:16:03.912789 22323 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.815s	user 0.009s	sys 0.004s
I20260709 12:16:03.890183 22516 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:03.867031 22520 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.955789 12572 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:03.872745 22537 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.938333 22310 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:16:03.971944 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45549.2us
W20260709 12:16:03.976745 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:03.978132 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:03.919164 22519 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:03.962078 21922 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.003s	sys 0.009s
I20260709 12:16:03.880249 22372 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.931372 22499 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:03.935627 22312 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:16:03.929180 22242 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:03.992141 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61563.6us
W20260709 12:16:03.978860 21789 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.611s	user 0.011s	sys 0.001s
I20260709 12:16:03.999511 22492 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:16:04.005893 12570 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:03.996054 22566 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:04.007594 22566 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.008247 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.009485 22499 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.010088 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:04.011826 22312 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:04.012374 22312 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:04.015404 22242 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.018555 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:03.968017 22522 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:16:03.944382 22323 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.021257 12569 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:04.021617 22323 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.022145 22520 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.022490 22542 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:04.023133 22542 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.004684 21789 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.025326 22542 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.026518 22297 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.026808 12494 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:04.025844 22432 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:04.021284 22349 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.028205 22519 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.989873 22538 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.031828 22561 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:04.032691 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:03.988191 21922 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:03.986654 22518 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.032s
W20260709 12:16:04.033075 22515 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.532s	user 0.008s	sys 0.000s
W20260709 12:16:04.034065 22576 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:16:04.013407 22312 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.015440 22545 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:03.879806 22572 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.026448 22583 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:03.993607 22514 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.722s	user 0.011s	sys 0.002s
I20260709 12:16:04.044106 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.045240 12565 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:16:04.040089 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9594389700000647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260709 12:16:03.920184 22571 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.029071 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.052677 12570 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:04.055284 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:04.058171 22432 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:04.026473 21789 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.059712 22349 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:04.061178 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:04.018620 22537 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.002215 22580 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.066705 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:04.078927 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104629us
I20260709 12:16:04.069319 22561 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:03.989653 22579 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
W20260709 12:16:04.082943 22487 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.000s
I20260709 12:16:04.124033 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.148361 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:04.078351 22460 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.003s
W20260709 12:16:04.168689 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107601us
W20260709 12:16:04.168572 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9270389700000123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15860000000000002s for a response
W20260709 12:16:04.181126 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:04.170498 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:04.181375 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8442389700001058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1678s for a response
W20260709 12:16:04.182242 12536 connection.cc:256] RPC call timeout handler was delayed by -1.6831389700000727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
W20260709 12:16:04.183409 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5724389700001211s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
W20260709 12:16:04.183977 12536 connection.cc:256] RPC call timeout handler was delayed by -1.475238969999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2088s for a response
W20260709 12:16:04.184438 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4302389700001186s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20260709 12:16:04.184839 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3654389700000138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
I20260709 12:16:04.190623 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:04.191294 12495 raft_consensus.cc:606] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:04.183643 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:04.151237 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.169080 22580 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.150422 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:04.169261 22583 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.082943 22546 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.193045 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.202975 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:04.199316 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:04.076839 22499 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:16:04.207068 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83201.1us
I20260709 12:16:04.147516 22432 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:04.177527 22537 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.231330 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.222246 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.223248 12646 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:04.232203 12639 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:04.230589 12572 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:04.235361 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:04.236883 22372 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.237458 12566 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:04.237655 22372 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.239214 22571 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.240111 22372 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.240859 22312 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:04.241704 22572 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.084909 22551 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.242383 22530 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:04.243018 22560 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:04.242609 22551 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.243834 22497 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.244525 22497 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.230589 12569 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:04.255817 12643 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:04.224331 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:04.190160 22538 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:04.270826 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:04.242661 22587 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:04.273665 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.288034 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.291096 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:04.287300 12638 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:04.296262 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.299378 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:04.299151 22560 raft_consensus.cc:2764] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:16:04.302639 22497 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.245850 22585 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:04.190199 22487 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.306589 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.198195 22460 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.268361 22522 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.313268 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.355202 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:16:04.355928 12645 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:04.267916 22492 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:04.197292 22501 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.203159 22520 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:16:04.316107 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80882.3us
I20260709 12:16:04.264719 22551 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.361915 22441 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:04.363214 22441 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:04.363392 22536 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:04.221217 22583 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:04.363812 22542 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:04.336294 22499 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:16:04.317744 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44602.3us
I20260709 12:16:04.298748 22372 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.225749 21922 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.324522 22514 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:04.370491 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:04.325711 22586 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:16:04.371623 12646 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:04.319591 22538 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:04.265051 22530 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.323587 22580 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.332432 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.311545 22515 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:04.268193 22537 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:04.332901 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 133674us
I20260709 12:16:04.348440 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.369585 12567 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:04.371704 12563 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:04.364030 12572 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:16:04.375097 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.383067 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:04.391947 12491 raft_consensus.cc:606] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:04.355806 22571 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:04.385505 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:16:04.388157 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97356.8us
I20260709 12:16:04.392390 12491 raft_consensus.cc:614] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:04.364518 22542 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:04.451836 12637 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:16:04.394176 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.454142 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:04.378798 22499 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:04.455287 12488 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:04.392328 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.420674 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.426143 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.370517 22441 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.323297 22561 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.431952 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.333547 22432 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.436403 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.368425 22522 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.438256 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.364200 22536 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.447614 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.472007 12495 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:04.434350 22538 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.372553 22520 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:04.480675 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:04.481199 22520 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.482355 22591 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:04.483364 22546 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:04.484138 22546 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:04.394706 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.456373 12646 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:04.485769 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.486689 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:04.486979 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.487586 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.463616 12491 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:16:04.488085 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.361006 22585 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.488674 22542 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.489262 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.374411 22372 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.467564 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:04.450383 22508 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.478433 22432 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:04.506022 22591 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:04.510031 22508 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.510564 22591 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.510905 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:04.512892 22508 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.512739 22580 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:16:04.513065 22591 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.513846 22542 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:04.514528 22580 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:04.515008 22522 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.515483 12641 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:04.516068 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.483237 22441 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.485816 22536 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.369580 22501 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.520378 22501 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.426826 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.521281 12492 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:04.490161 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.523105 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.523780 12638 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31. Re-sending same reply.
I20260709 12:16:04.524966 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:04.470927 22516 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.004s
I20260709 12:16:04.490360 22585 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.482383 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:04.454862 22596 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:16:04.492789 22460 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.493420 22597 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:04.526827 22597 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.525976 22516 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:04.529083 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:04.453212 22571 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.531383 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.531769 22571 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.485004 22546 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.494758 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:04.533026 12491 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:04.465999 22514 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:16:04.496228 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.497268 22537 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.488094 12645 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 28
W20260709 12:16:04.451619 22598 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:16:04.497838 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1362) took 1002 ms. Trace:
I20260709 12:16:04.536744 12642 rpcz_store.cc:276] 0709 12:16:03.495360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:03.495779 (+   419us) service_pool.cc:224] Handling call
0709 12:16:03.543608 (+ 47829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:03.544590 (+   982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:04.497549 (+952959us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b54002c7250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:04.497761 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79634816}
I20260709 12:16:04.517452 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.520112 22492 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:04.525435 22372 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.407788 22515 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.536217 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1355) took 1110 ms. Trace:
I20260709 12:16:04.538251 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:04.544154 12646 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:04.531430 12638 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 13. Preceding OpId from leader: term: 30 index: 14. (index mismatch)
W20260709 12:16:04.539109 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56827.3us
I20260709 12:16:04.487973 22530 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:04.466524 22487 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.544202 12641 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:04.548210 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.556497 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.559610 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.559957 12646 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:04.542443 12644 rpcz_store.cc:276] 0709 12:16:03.425450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:03.426070 (+   620us) service_pool.cc:224] Handling call
0709 12:16:04.496032 (+1069962us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:16:04.536147 (+ 40115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2772352}
I20260709 12:16:04.564654 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.566852 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.555949 22537 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:04.566494 12568 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:16:04.555588 22591 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:04.564306 22597 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.583360 22499 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.583734 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.586207 22487 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:04.586987 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:04.588232 12568 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:04.579672 22591 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:04.600239 22591 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:04.602072 22515 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.584558 22519 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:04.604904 22519 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:04.605931 22519 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.610309 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.611404 12570 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:04.586872 22487 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:04.613188 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:04.616858 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85527.9us
I20260709 12:16:04.589419 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.618172 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:04.635881 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:04.634482 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:04.655719 12640 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:04.625747 22492 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:04.662474 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.638861 22514 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:04.670185 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.671299 12640 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:04.682513 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.684616 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.683264 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:04.689744 22608 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:04.690403 22608 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:04.690784 22608 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:04.666779 22492 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.694140 22492 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.694226 12568 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:04.696647 22487 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.720669 13262 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:16:04.721683 22504 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:04.722134 22504 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:04.723152 22504 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:04.739514 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:04.741436 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:04.762823 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:04.765264 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:04.765894 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.767203 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:04.713423 22487 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:16:04.770212 12491 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 33 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:04.770668 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.769304 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:04.775321 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90748.9us
I20260709 12:16:04.776603 12536 leader_election.cc:400] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:04.777050 12536 leader_election.cc:403] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:16:04.778074 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:04.778316 22460 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:04.778568 22542 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.779312 13254 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
I20260709 12:16:04.779440 22542 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.780485 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:04.781119 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:04.782055 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:04.783761 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:04.711005 22519 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.059s
I20260709 12:16:04.787175 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:04.782231 22542 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:04.792013 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50753.5us
I20260709 12:16:04.793246 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:04.796003 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:04.799827 12566 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:04.733712 22546 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:04.804239 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:04.807530 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:04.809768 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.749048 22432 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.766893 22538 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:04.808722 22542 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:04.795481 22441 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:04.820946 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:04.828248 22607 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:16:04.822557 22538 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.829689 22349 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.008s
I20260709 12:16:04.819002 22542 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:04.812655 22487 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:04.832573 22542 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:04.833432 22487 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:04.836957 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60557us
I20260709 12:16:04.837821 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.832048 22538 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:04.811542 22536 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:04.814442 22508 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:04.814738 22560 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:04.861784 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:04.823911 22441 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:04.849778 22506 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.021s
W20260709 12:16:04.872303 22612 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:16:04.854274 22536 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:04.882107 22536 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:04.844478 22496 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.857918 22519 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:04.886464 22519 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:04.887538 22519 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.858596 22508 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:04.890595 22508 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:04.816087 22432 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.896168 22432 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:04.897881 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60135.3us
I20260709 12:16:04.899068 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:04.901053 22492 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:16:04.868374 22502 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.812947 22546 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:04.918728 13262 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:04.920181 22546 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:04.941236 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:04.942132 12493 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:04.954509 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:04.956507 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.957559 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.959441 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:04.916911 22608 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.962057 12643 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:04.922065 22441 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:04.971248 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:04.976220 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:04.978484 12639 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:16:04.939963 22502 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:04.853880 22349 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.105s
W20260709 12:16:04.984006 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137509us
I20260709 12:16:04.984917 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:04.949756 22492 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:05.008090 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.010905 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.011624 12644 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:05.010152 22491 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:05.015130 22491 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:05.016103 22491 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:04.981359 22460 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:05.018003 22460 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:05.018456 22460 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:05.019177 22540 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:05.019651 22540 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:05.025729 12493 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:05.006872 22506 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:05.029304 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:05.044834 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:04.977680 22580 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:16:05.041833 22432 raft_consensus.cc:2764] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:16:05.062002 22460 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.062641 22297 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:05.063522 22492 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.065030 22460 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.066002 22372 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:05.066490 22372 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:05.066847 22372 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:05.068758 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.026609 22604 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:05.070132 22604 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.072698 22604 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:05.074436 22608 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:05.041814 22591 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.078148 22591 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.047693 22506 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:05.080109 22506 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:05.051373 22514 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:05.084185 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.026247 22560 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.054s
I20260709 12:16:05.089845 22540 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.091181 12646 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:05.091691 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.092525 12638 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:16:05.093143 12646 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:16:05.095814 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:05.097132 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:05.024803 22487 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:16:05.098857 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4205) took 1544 ms. Trace:
I20260709 12:16:05.099294 12494 rpcz_store.cc:276] 0709 12:16:03.554300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:03.555595 (+  1295us) service_pool.cc:224] Handling call
0709 12:16:05.098597 (+1543002us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:05.098746 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080905600}
I20260709 12:16:05.101413 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4173) took 1073 ms. Trace:
I20260709 12:16:05.101879 12564 rpcz_store.cc:276] 0709 12:16:04.028315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:04.043755 (+ 15440us) service_pool.cc:224] Handling call
0709 12:16:05.101225 (+1057470us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:05.101344 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6288768}
I20260709 12:16:05.008046 22502 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.085487 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.090296 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.108012 12644 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:05.112015 12566 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
W20260709 12:16:05.113886 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137839us
I20260709 12:16:05.122504 12564 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
W20260709 12:16:05.118108 22544 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.854s	user 0.005s	sys 0.023s
I20260709 12:16:05.105541 22502 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:05.113200 22514 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:16:05.134684 22544 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.876s	user 0.005s	sys 0.024s
I20260709 12:16:05.145740 22508 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.148169 22508 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:05.133925 22242 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.100s
I20260709 12:16:05.164606 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4170) took 1137 ms. Trace:
I20260709 12:16:05.178785 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4199) took 1654 ms. Trace:
I20260709 12:16:05.183298 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.183598 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.193672 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4165) took 1759 ms. Trace:
I20260709 12:16:05.226831 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.254782 13262 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 28 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:16:05.229017 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.256004 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:05.229796 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.230473 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.257392 13257 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:05.245666 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.258510 12570 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:05.258877 12570 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:05.199198 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3874) took 1003 ms. Trace:
I20260709 12:16:05.259501 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:05.230150 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.229403 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.257560 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:05.174832 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4001) took 1247 ms. Trace:
I20260709 12:16:05.259699 12641 rpcz_store.cc:276] 0709 12:16:04.195289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:04.287692 (+ 92403us) service_pool.cc:224] Handling call
0709 12:16:05.198910 (+911218us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:05.199106 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":521383296}
I20260709 12:16:05.262775 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:05.246302 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:05.243966 12488 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:05.257769 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:05.264179 12572 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:05.238914 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:05.265080 12495 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:05.252921 12563 rpcz_store.cc:276] 0709 12:16:03.434657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:03.436011 (+  1354us) service_pool.cc:224] Handling call
0709 12:16:03.436374 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:04.406010 (+969636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:05.193201 (+787191us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:05.193526 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2150395776}
I20260709 12:16:05.247170 12569 rpcz_store.cc:276] 0709 12:16:04.027338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:04.028768 (+  1430us) service_pool.cc:224] Handling call
0709 12:16:05.164314 (+1135546us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:05.164496 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188671360}
W20260709 12:16:05.260461 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 217549us
I20260709 12:16:05.268703 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:05.248786 12497 rpcz_store.cc:276] 0709 12:16:03.524592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:03.524774 (+   182us) service_pool.cc:224] Handling call
0709 12:16:03.753874 (+229100us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:03.911631 (+157757us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:03.911666 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:03.924465 (+ 12799us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:03.924526 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:05.178423 (+1253897us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:05.178492 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:05.178701 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1415,"raft.queue_time_us":229,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":10328,"spinlock_wait_cycles":361951104}
I20260709 12:16:05.262041 12571 rpcz_store.cc:276] 0709 12:16:03.927231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:03.927500 (+   269us) service_pool.cc:224] Handling call
0709 12:16:03.935162 (+  7662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:04.198007 (+262845us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:04.198041 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:04.198085 (+    44us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:04.198114 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:05.174427 (+976313us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:05.174507 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:05.174739 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":604011776}
I20260709 12:16:05.270416 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.274427 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.274712 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.275146 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.275264 12492 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:05.139050 22514 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:05.284344 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.284667 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.284933 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.222086 22575 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:05.288746 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:05.259719 12570 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.280193 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.289883 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:05.233934 22192 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:05.292944 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:05.224771 22556 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:05.287122 12497 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:05.262615 22232 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:05.288733 12488 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:16:05.291630 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.273931 22432 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:05.292155 13262 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:05.282734 22451 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:05.290441 12570 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:05.192673 22486 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:05.294148 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.292116 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.290619 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:05.297165 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:05.310416 12566 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:16:05.311560 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.298213 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:16:05.315083 12490 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: [28.23-28.23]
I20260709 12:16:05.288091 22624 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:05.316282 22624 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:05.289312 22575 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:05.317315 22624 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.318514 22575 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.318675 22372 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.319949 22575 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:05.320569 12566 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:05.295518 22323 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:05.321216 22372 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.293802 12492 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27.
I20260709 12:16:05.272679 22620 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:05.303110 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.328423 12491 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:05.297740 22625 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:05.286331 22516 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:05.322731 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.323320 22323 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:05.323720 12493 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:05.342097 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.342851 13262 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:05.343354 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.343927 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.344094 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:05.324759 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.326337 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:05.161842 22506 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.151943 22536 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.330184 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.329547 12496 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:05.330120 13261 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:16:05.340317 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.276532 22310 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:16:05.367592 13259 raft_consensus.cc:2368] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 30.
I20260709 12:16:05.367997 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:05.368088 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.369599 22580 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:05.342221 22323 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:05.365237 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85195.7us
I20260709 12:16:05.367177 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.364056 12570 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260709 12:16:05.370389 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:05.309276 22451 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:05.379096 22451 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:05.373363 22627 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.014s
I20260709 12:16:05.376277 22323 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:05.386364 12536 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:16:05.386763 12536 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:05.386812 12494 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:05.299458 22502 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.070s
I20260709 12:16:05.387861 12495 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:05.376080 12570 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 13
I20260709 12:16:05.341780 22516 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:05.392067 12570 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260709 12:16:05.395637 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:05.396641 12570 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:05.399626 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.368760 22242 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:05.404430 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.410117 22608 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:16:05.367285 22625 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:05.396207 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.374281 22580 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:05.308365 22232 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.414873 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.405424 22497 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:05.376619 22551 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:05.301357 22556 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.419744 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:05.370378 22486 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:16:05.418347 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 389125us
I20260709 12:16:05.387285 22314 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:05.404179 22516 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:16:05.432462 22542 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.005s
I20260709 12:16:05.371217 22441 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.353313 22620 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.415907 22571 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:05.407450 22566 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.441298 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:05.415693 22499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.006s	sys 0.009s
I20260709 12:16:05.404913 22627 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26.
I20260709 12:16:05.432209 22580 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:05.449733 13254 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:05.430100 22522 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:16:05.454357 13257 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:05.447564 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:05.407172 22585 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:16:05.388082 22514 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.480086 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:05.484727 13256 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:05.484431 13260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:05.494779 13260 raft_consensus.cc:606] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:05.495189 13260 raft_consensus.cc:614] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:05.483878 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.500900 13254 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 33 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:05.489039 22232 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:05.477519 22499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.006s	sys 0.009s
W20260709 12:16:05.507791 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88139.3us
I20260709 12:16:05.477069 22441 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:05.508937 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140937us
I20260709 12:16:05.455075 22571 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:05.452246 22542 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.439414 22608 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:05.489826 22502 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:05.537302 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:05.533388 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.514961 22507 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.536366 22499 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:05.540176 22561 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:05.541132 22343 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:16:05.544365 22514 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:05.515803 22372 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:05.484155 22632 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:16:05.535197 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96153.3us
I20260709 12:16:05.454722 22520 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 12:16:05.516842 22536 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.569320 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.564385 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.580927 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.581310 12643 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:16:05.572999 12488 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:16:05.537585 22571 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:05.584234 12642 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:16:05.575915 12639 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:16:05.579680 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:05.580034 22632 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:05.589025 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.590075 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.590106 12638 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:05.589162 22632 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:16:05.597512 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:05.599462 12564 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:16:05.601392 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.603229 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.604058 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:05.606801 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.608408 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:05.596578 12611 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.501s (SENT) [suppressed 20 similar messages]
I20260709 12:16:05.564867 22640 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:05.612996 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.615018 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.615970 12639 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:05.619689 12489 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:16:05.611924 22312 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:05.623288 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.583793 22520 consensus_queue.cc:1243] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:16:05.624097 12642 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:05.624708 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:05.626868 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:05.590169 22608 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:05.571589 22591 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:16:05.576462 22499 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:05.535070 22625 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 27.22, Last appended by leader: 22, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.618252 22314 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:05.582266 22323 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.625278 22634 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:05.707940 12639 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:05.668402 22508 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:05.655984 22608 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:05.711489 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 105075us
I20260709 12:16:05.724586 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:05.723902 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 127076us
I20260709 12:16:05.738034 12639 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:16:05.678774 22507 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:05.743176 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:05.744956 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130186us
I20260709 12:16:05.757870 13258 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:16:05.761381 22451 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.752635 22507 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.767163 22451 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:05.767696 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.735391 22608 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:05.696178 22580 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:05.807633 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115137us
I20260709 12:16:05.725780 22486 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:16:05.812219 12489 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 14
I20260709 12:16:05.813067 12489 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 29.15
I20260709 12:16:05.813452 12489 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:16:05.813549 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 208437us
I20260709 12:16:05.778504 22507 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:05.816359 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:05.782516 22641 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:05.735813 22508 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.793699 22343 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:16:05.825256 12497 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:16:05.827773 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:05.832973 12492 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:05.834726 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.838779 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:05.839526 13261 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:16:05.840329 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:05.818012 22372 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:05.819231 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:05.847223 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:05.833966 22542 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.830924 22650 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:05.831817 22522 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:05.856221 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:05.865953 12490 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:05.831066 22507 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:16:05.859043 22451 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:05.866602 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:05.863222 22646 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:16:05.881476 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:05.882614 12492 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:16:05.883288 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:05.884018 22662 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:05.884471 22662 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:05.884836 22662 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:05.884884 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:05.892693 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:05.895354 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67688.1us
I20260709 12:16:05.855033 22658 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:05.878526 22652 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:05.880152 22343 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:16:05.881335 22192 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.586s	user 0.012s	sys 0.000s
I20260709 12:16:05.892963 22657 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:05.834713 22314 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.029s
I20260709 12:16:05.858731 22372 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.867372 22497 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.068s
I20260709 12:16:05.904119 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:05.932997 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:05.865195 22650 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.941195 13255 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:16:05.867266 22522 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:05.952766 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65357.3us
I20260709 12:16:05.971179 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:05.952101 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.903023 22652 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:05.938270 22323 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:05.916154 13261 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:16:05.963557 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59506.2us
I20260709 12:16:05.974496 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:05.964218 22192 raft_consensus.cc:2764] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:16:05.926627 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:05.963778 22639 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.399s	user 0.001s	sys 0.000s
I20260709 12:16:05.976307 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:05.899960 22451 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.968819 22322 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:05.989832 22322 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.938728 22232 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:16:05.951479 22538 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:05.997416 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:05.994503 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:05.983597 22343 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:06.003881 12492 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:16:05.927480 22658 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:05.980947 22652 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:06.005985 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:06.006088 22575 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.003s
I20260709 12:16:05.999866 22580 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.022418 22508 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:06.006435 22343 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:06.054025 22343 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:06.024134 22632 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.056524 22655 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:16:06.057961 22638 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.541s	user 0.000s	sys 0.005s
I20260709 12:16:06.057054 22659 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.104s
I20260709 12:16:06.054884 22669 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.021498 22623 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:16:06.061085 22312 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:06.027104 22543 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:06.001183 22538 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:06.151693 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 225159us
W20260709 12:16:06.155234 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 160796us
I20260709 12:16:06.150939 22575 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:06.139336 22496 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:16:06.195165 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 221810us
I20260709 12:16:06.202792 22497 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.218948 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:06.227702 13257 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:06.241485 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:06.244860 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.245712 13261 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:06.212355 22575 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:06.245184 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:06.239085 22659 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:06.252003 22659 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:06.252648 12639 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:06.253189 22659 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:06.254153 22668 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:16:06.254853 22659 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:06.256021 22677 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:06.257326 22661 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260709 12:16:06.259687 22671 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:16:06.252033 22674 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:06.269495 22674 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.270205 22681 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.270809 22681 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.272976 22681 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.209695 22669 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.280048 22669 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.282267 22632 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.282502 22650 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:06.283237 22451 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:06.231864 22497 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.283855 22658 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:06.246776 22372 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:06.288034 22636 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.749s	user 0.002s	sys 0.000s
I20260709 12:16:06.230370 22520 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.290649 22520 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:06.294981 22561 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.719s	user 0.014s	sys 0.001s
I20260709 12:16:06.250497 22575 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:16:06.296463 22571 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.699s	user 0.009s	sys 0.004s
I20260709 12:16:06.296942 22571 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.297828 13256 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:16:06.252846 22664 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:16:06.303009 22497 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.306354 22520 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.307753 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1185) took 1536 ms. Trace:
I20260709 12:16:06.273563 22681 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:06.317685 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.309103 13255 rpcz_store.cc:276] 0709 12:16:04.770863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:04.783089 (+ 12226us) service_pool.cc:224] Handling call
0709 12:16:04.783392 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:05.968540 (+1185148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:06.307390 (+338850us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:06.307672 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2654945024}
I20260709 12:16:06.315384 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:06.295032 22644 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.730s	user 0.002s	sys 0.000s
I20260709 12:16:06.251277 22522 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.328341 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.316751 22674 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.246860 22496 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.080s
I20260709 12:16:06.308616 22545 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:06.334283 22545 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.333595 22538 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.328756 22520 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:06.338366 22520 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.338933 22520 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:06.323120 22681 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:06.341257 22628 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.968s	user 0.003s	sys 0.001s
W20260709 12:16:06.342821 22637 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.858s	user 0.001s	sys 0.001s
I20260709 12:16:06.343199 22681 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:06.345652 22630 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.936s	user 0.000s	sys 0.003s
W20260709 12:16:06.346505 22608 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.001s
I20260709 12:16:06.330354 22662 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:06.320513 22502 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.035s	sys 0.035s
W20260709 12:16:06.355626 22502 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.035s	sys 0.037s
I20260709 12:16:06.356057 22502 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:06.357004 22633 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.880s	user 0.003s	sys 0.000s
I20260709 12:16:06.350054 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3239) took 1418 ms. Trace:
I20260709 12:16:06.358078 12640 rpcz_store.cc:276] 0709 12:16:04.931502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:04.931688 (+   186us) service_pool.cc:224] Handling call
0709 12:16:04.932209 (+   521us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:04.932719 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:04.932763 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:04.950417 (+ 17654us) log.cc:844] Serialized 52 byte log entry
0709 12:16:05.126712 (+176295us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:05.345477 (+218765us) log.cc:844] Serialized 14 byte log entry
0709 12:16:05.345810 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:05.345918 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:06.344616 (+998698us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:06.348334 (+  3718us) raft_consensus.cc:1685] finished
0709 12:16:06.349774 (+  1440us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:06.349966 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16628,"prepare.queue_time_us":218008,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":3091,"raft.queue_time_us":874,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":34047,"spinlock_wait_cycles":821945088,"thread_start_us":115198,"threads_started":1,"wal-append.queue_time_us":393157}
W20260709 12:16:06.326781 22516 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.875s	user 0.012s	sys 0.003s
I20260709 12:16:06.360778 22516 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.318908 22506 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.363242 22516 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.364111 12643 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:06.351063 22662 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:06.323184 22514 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.745s	user 0.005s	sys 0.009s
I20260709 12:16:06.350513 22571 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.339960 22343 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.368192 22652 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.369133 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:06.370038 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.370805 13261 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:16:06.365541 22608 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:06.373061 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:06.375555 22604 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:06.390702 22652 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.391513 12642 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:06.392017 22662 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:06.393056 22343 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:06.378575 22575 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:06.383989 22645 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.813s	user 0.003s	sys 0.000s
I20260709 12:16:06.357215 22502 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.400130 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:06.395584 22608 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:06.409466 22501 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.949s	user 0.011s	sys 0.015s
W20260709 12:16:06.413235 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:06.416183 22343 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:06.419952 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4221) took 1175 ms. Trace:
I20260709 12:16:06.420156 22575 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.420526 12494 rpcz_store.cc:276] 0709 12:16:05.244486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.244651 (+   165us) service_pool.cc:224] Handling call
0709 12:16:06.419698 (+1175047us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:06.419880 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254277376}
I20260709 12:16:06.415350 22608 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:06.423913 22608 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.424482 12637 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:16:06.426059 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:06.426367 22608 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:06.427572 22499 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:06.413020 22501 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.012s	sys 0.015s
I20260709 12:16:06.434785 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3201) took 1134 ms. Trace:
I20260709 12:16:06.435981 12488 rpcz_store.cc:276] 0709 12:16:05.300020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.300671 (+   651us) service_pool.cc:224] Handling call
0709 12:16:05.301062 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:05.582095 (+281033us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:06.434463 (+852368us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:06.434711 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":624722304}
I20260709 12:16:06.446080 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.447328 22640 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:06.450706 22520 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.452557 22556 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.453112 22520 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:06.460215 22643 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.896s	user 0.003s	sys 0.000s
I20260709 12:16:06.462674 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:06.463475 13258 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:06.463783 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1820) took 1041 ms. Trace:
I20260709 12:16:06.463560 22504 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:06.464184 12570 rpcz_store.cc:276] 0709 12:16:05.422648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.422834 (+   186us) service_pool.cc:224] Handling call
0709 12:16:05.423233 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:05.433673 (+ 10440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:05.433715 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:05.434343 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 12:16:05.533447 (+ 99104us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:05.533494 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:05.533573 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:06.459319 (+925746us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:06.463442 (+  4123us) raft_consensus.cc:1685] finished
0709 12:16:06.463467 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:06.463714 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9973,"raft.queue_time_us":13374,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":1822,"spinlock_wait_cycles":514556800,"thread_start_us":73782,"threads_started":1,"wal-append.queue_time_us":116591}
I20260709 12:16:06.468575 22322 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.478286 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:06.481500 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:06.482767 22641 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:06.483383 22641 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:06.485415 22486 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:06.493656 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1189) took 1068 ms. Trace:
I20260709 12:16:06.494097 13262 rpcz_store.cc:276] 0709 12:16:05.424912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.425020 (+   108us) service_pool.cc:224] Handling call
0709 12:16:05.547924 (+122904us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:05.572235 (+ 24311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:05.572290 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:06.333141 (+760851us) spinlock_profiling.cc:243] Waited 948 us on lock 0x7b2c00007be8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:06.333648 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:16:06.396653 (+ 63005us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:06.396713 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:06.396820 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:06.423258 (+ 26438us) raft_consensus.cc:1685] finished
0709 12:16:06.423281 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:06.493431 (+ 70150us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54004eeb50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:06.493561 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10104,"raft.queue_time_us":6555,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":4788,"spinlock_wait_cycles":251076224,"thread_start_us":60446,"threads_started":1,"wal-append.queue_time_us":87694}
I20260709 12:16:06.497967 22543 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:06.499534 22543 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.500458 22620 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:06.502255 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 283380us
I20260709 12:16:06.503491 22545 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.504179 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:06.505568 22678 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:06.497895 22297 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:06.511106 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:06.511389 22635 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:06.512359 22635 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:06.516762 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:06.508567 12638 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:06.519361 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:06.530470 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:06.540300 22515 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.000s	sys 0.018s
W20260709 12:16:06.547729 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 327284us
I20260709 12:16:06.548516 22680 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:06.532837 22545 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:06.556361 22675 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:06.556480 22545 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:06.557605 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51054.3us
I20260709 12:16:06.558743 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:06.561002 22451 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:06.557314 22545 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:06.553817 22515 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.162s	user 0.001s	sys 0.018s
I20260709 12:16:06.543298 22608 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:06.581056 22608 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.584316 22608 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:06.585044 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.585716 12495 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:06.586175 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.549301 22520 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:06.586992 12488 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:16:06.589022 22310 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:06.589488 22310 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:06.590104 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.590447 22310 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.592041 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.592391 22310 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:06.592779 12490 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:06.594497 22680 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:06.594954 22680 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:06.595907 22680 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.599440 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.601452 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:06.603190 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 283069us
I20260709 12:16:06.603794 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4032) took 1260 ms. Trace:
I20260709 12:16:06.604027 22372 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:16:06.605535 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:06.606086 22451 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:06.606631 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51655.1us
I20260709 12:16:06.607467 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.607250 22451 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 31.15, Last appended by leader: 15, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.607949 22551 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:06.611827 22675 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:06.604327 12566 rpcz_store.cc:276] 0709 12:16:05.343605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.346487 (+  2882us) service_pool.cc:224] Handling call
0709 12:16:05.346925 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:05.347219 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:16:05.373756 (+ 26537us) log.cc:844] Serialized 14 byte log entry
0709 12:16:05.413836 (+ 40080us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b1800379240. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:16:05.414203 (+   367us) log.cc:844] Serialized 14 byte log entry
0709 12:16:05.434969 (+ 20766us) log.cc:844] Serialized 14 byte log entry
0709 12:16:05.602138 (+167169us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b1800379240. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:06.603251 (+1001113us) spinlock_profiling.cc:243] Waited 6.66 ms on lock 0x7b1800379240. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:16:06.603425 (+   174us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:06.603481 (+    56us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:06.603531 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:06.603606 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:06.603729 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38451,"prepare.queue_time_us":85897,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":147,"raft.queue_time_us":19191,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":11208,"spinlock_wait_cycles":38422656}
I20260709 12:16:06.613090 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.614275 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.615272 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:06.616099 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.616367 22585 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:06.597369 22680 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:06.617761 13258 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:16:06.555191 22635 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.632776 12490 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 22. Preceding OpId from leader: term: 29 index: 23. (index mismatch)
I20260709 12:16:06.632794 12491 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:06.636389 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.637552 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:06.658222 12495 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:06.638537 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.641922 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:06.661190 12488 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:06.662235 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:06.663112 12488 raft_consensus.cc:606] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:06.663556 12488 raft_consensus.cc:614] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:06.663765 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:06.664727 12640 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:06.665763 12496 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:06.667412 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:06.667956 12496 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:16:06.671248 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:06.661605 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:06.673846 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.657049 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:06.680970 22557 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.681187 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:06.682346 12641 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:06.684353 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:06.681951 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:06.689320 12492 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:06.641134 22506 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:16:06.691031 22544 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.675159 22580 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:06.692507 22580 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.693950 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:06.686209 22372 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 12:16:06.696470 22492 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:06.682619 22507 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.015s
I20260709 12:16:06.685706 22571 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.697679 22571 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.685717 22432 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:06.699259 22432 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:06.699666 22432 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:06.701975 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.686041 22514 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.702793 22519 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.703146 22514 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.691694 22675 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.691684 22544 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.681814 22520 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:06.705358 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.721028 12570 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:06.697453 12639 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:06.712440 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:06.723387 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:06.724267 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:06.712680 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:06.696502 22538 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:06.728471 22538 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:06.642599 22585 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:06.696682 22626 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:06.686041 22557 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.719177 22492 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.703323 22519 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.731518 22556 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:16:06.732959 12566 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:06.727952 12492 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:06.799983 12566 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:06.686631 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:06.742055 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:06.703250 22566 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:06.705430 22514 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.737746 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:06.805408 12496 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:06.808000 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:06.808143 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70537.1us
I20260709 12:16:06.803896 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:06.809444 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:06.801110 12566 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 28.23, Last appended by leader: 23, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.814531 12566 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:06.815900 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.852154 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:16:06.744087 22608 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:06.833281 12570 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:06.836910 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.770448 12495 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:16:06.847692 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:06.854837 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:06.865232 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61452.9us
I20260709 12:16:06.856928 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:06.818742 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:06.867038 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.899916 12572 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:06.866443 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.761772 22501 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:06.729456 22538 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.928040 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1276) took 1588 ms. Trace:
I20260709 12:16:06.933314 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.933779 13254 rpcz_store.cc:276] 0709 12:16:05.325490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.326790 (+  1300us) service_pool.cc:224] Handling call
0709 12:16:06.913942 (+1587152us) spinlock_profiling.cc:243] Waited 73.7 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:06.914209 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":169599744}
I20260709 12:16:06.934544 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4031) took 1676 ms. Trace:
W20260709 12:16:06.940824 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84044.4us
I20260709 12:16:06.938879 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.760376 22277 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.128s
I20260709 12:16:06.892284 22675 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:16:06.946297 22660 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:06.972782 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 154141us
W20260709 12:16:06.954684 22688 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:16:06.977870 22530 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:06.983560 22530 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:06.983785 22557 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:06.984038 22530 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:06.984313 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:06.985306 12572 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:06.985822 22499 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:16:06.986295 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:06.986821 22499 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:06.987637 22499 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:06.988667 22499 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.955278 22649 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.212s
I20260709 12:16:06.956015 22687 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:06.955612 22545 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.815608 22566 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.948343 12566 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 23. Preceding OpId from leader: term: 29 index: 23. (term mismatch)
I20260709 12:16:07.003952 12566 pending_rounds.cc:85] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 22
I20260709 12:16:07.004272 12566 pending_rounds.cc:107] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 28.23
I20260709 12:16:07.004511 12566 raft_consensus.cc:2889] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:06.958197 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:06.865636 22608 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:06.963335 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:06.964370 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.006078 12569 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:07.006644 12571 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:07.006780 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43593.2us
I20260709 12:16:06.972508 22562 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:06.964071 22556 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:06.978407 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:07.019410 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:06.765341 22626 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:06.955111 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:16:06.991492 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 229802us
I20260709 12:16:06.987077 12572 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:16:06.957070 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:06.970402 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.009482 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:06.966909 22675 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:06.972198 22660 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:07.072587 22626 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.013137 22557 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:07.098054 22660 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:07.108680 22502 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:07.109571 22502 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:07.114578 22696 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 12:16:07.121482 12637 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:07.122661 22492 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:07.123781 22544 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.124825 22545 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.126793 22566 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.128291 22519 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:07.074640 22690 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:16:07.139190 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:07.075416 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:07.140723 12537 connection.cc:256] RPC call timeout handler was delayed by -1.1778149769997981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
W20260709 12:16:07.141180 12537 connection.cc:256] RPC call timeout handler was delayed by -1.1158791779998865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1902s for a response
I20260709 12:16:07.076421 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:07.070804 22432 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:06.989989 22640 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:07.018915 22499 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:07.013474 22562 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:07.084465 22556 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:07.086315 22642 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:07.084561 22515 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:07.088505 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.089026 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.091790 13255 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:16:06.955155 22653 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.209s
I20260709 12:16:07.162317 12567 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:07.095858 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86340.4us
I20260709 12:16:07.093442 22507 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:07.097505 22700 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:07.097507 12642 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:07.098088 12495 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:07.094204 22675 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:07.068994 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.132368 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:07.167490 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:07.134238 12564 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:06.938895 12565 rpcz_store.cc:276] 0709 12:16:05.258549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.273379 (+ 14830us) service_pool.cc:224] Handling call
0709 12:16:05.746206 (+472827us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:16:05.746443 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:06.934142 (+1187699us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:06.934176 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:06.934221 (+    45us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:06.934267 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:06.934341 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:06.934484 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":909552,"raft.queue_time_us":952624,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":40136960,"thread_start_us":277601,"threads_started":1}
I20260709 12:16:07.129122 22580 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:07.107136 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.198262 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:06.972746 22543 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.046s
I20260709 12:16:07.143138 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:07.148737 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3909) took 1556 ms. Trace:
I20260709 12:16:06.931005 22585 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.153733 13253 raft_consensus.cc:2368] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 30.
I20260709 12:16:07.009251 22608 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:16:07.153401 22641 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.670s	user 0.015s	sys 0.000s
I20260709 12:16:07.063647 22631 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:07.053500 22460 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:07.190562 12491 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
W20260709 12:16:07.175524 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.191990 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.051003 22514 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:07.151091 22519 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:07.191121 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:07.218331 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.165518 22571 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.158491 22515 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:07.230355 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.157393 22640 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:07.229447 22519 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.236090 22519 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.207202 12643 rpcz_store.cc:276] 0709 12:16:05.591897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.593564 (+  1667us) service_pool.cc:224] Handling call
0709 12:16:07.148455 (+1554891us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:07.148647 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1691678080}
I20260709 12:16:07.208002 22620 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:07.237612 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:07.211653 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 68601.8us
I20260709 12:16:07.237952 22556 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:07.239024 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:07.240374 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:07.240584 22702 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:07.179822 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:07.241587 22514 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.190821 22507 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.248857 22502 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:07.249298 22507 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.215523 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:07.225657 12568 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:07.236155 22640 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.217195 22580 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260709 12:16:07.251478 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.206600 22297 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:16:07.137691 12569 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:07.289335 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.290719 12569 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:07.243227 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:07.214663 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.282486 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:07.291751 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.293180 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:07.204408 12640 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.539s	user 0.011s	sys 0.000s
W20260709 12:16:07.263216 12638 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.000s
W20260709 12:16:07.285825 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69164.3us
I20260709 12:16:07.216945 22544 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:16:07.286970 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119565us
I20260709 12:16:07.238992 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.294041 12638 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
W20260709 12:16:07.296854 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.294797 12569 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:07.282738 12563 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:07.293874 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:07.302402 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:07.297570 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.294466 12567 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:07.307313 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:07.310776 12536 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27.
I20260709 12:16:07.310148 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:07.311360 12536 leader_election.cc:403] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:16:07.233155 22515 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:07.281735 22571 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:07.316651 22699 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:07.296412 22640 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:07.317015 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.317212 12537 leader_election.cc:400] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 30.
I20260709 12:16:07.341531 12537 leader_election.cc:403] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:16:07.220242 22641 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.314198 22507 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:07.321928 22701 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:07.333003 22277 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:16:07.327582 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.327726 22649 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:07.232010 22545 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.097s
I20260709 12:16:07.223683 22631 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.306463 22492 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:16:07.247054 22514 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:07.308413 22232 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.008s	sys 0.019s
I20260709 12:16:07.338459 12640 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:07.226837 22460 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.345386 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:07.347532 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:07.350512 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:07.289728 22566 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:07.378077 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:07.277750 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:07.237653 22620 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:07.355306 13262 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:16:07.353417 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:07.362715 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.371447 13257 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:07.369935 22649 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:07.371420 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:07.343560 22699 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:07.340536 22634 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:16:07.348501 22507 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:07.383030 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.356673 22277 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:07.399784 22277 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:07.349548 22543 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:07.389183 22699 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:07.376549 22297 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:16:07.402024 22543 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.402633 22297 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 30.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:16:07.404567 22543 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.386055 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.411590 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.350206 22502 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:07.383966 22649 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:07.371083 22192 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:07.389005 22504 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:07.396566 12572 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:07.343312 22571 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.397151 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:16:07.376341 22232 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.009s	sys 0.020s
I20260709 12:16:07.353353 22626 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27.
I20260709 12:16:07.394567 22492 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:16:07.390324 22491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.019s	user 0.015s	sys 0.022s
I20260709 12:16:07.414410 22657 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:07.383160 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.418845 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.450698 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.423632 22451 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:07.423902 22192 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:07.421187 22486 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:07.426575 22585 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:07.453801 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.458020 22626 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:07.464387 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.470922 22192 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:07.472909 22192 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:07.478667 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:07.441339 22491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.103s	user 0.015s	sys 0.023s
I20260709 12:16:07.444687 22492 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:07.425858 22502 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:07.458859 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75783.9us
I20260709 12:16:07.485945 22641 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:07.423858 22675 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.493850 22675 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:07.495812 22571 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.496518 22491 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:07.497650 22571 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:07.498054 22571 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:07.499090 22571 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.455322 22504 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:07.500738 22571 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:07.501813 22501 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:07.502609 22620 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.503997 22562 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:07.506321 22460 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.507733 22631 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.505385 22562 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:07.482256 22497 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:07.502472 22501 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:07.511549 22497 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:07.512145 22497 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:07.515427 22502 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:07.516417 22634 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:07.487015 12570 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:07.459486 22499 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:07.522528 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:07.523193 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.510917 22540 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:16:07.527384 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63126.9us
I20260709 12:16:07.528321 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:07.545481 22499 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:07.490458 22515 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.574147 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.577479 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:07.647678 12638 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:07.649053 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.581077 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:07.580103 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:07.651254 12638 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
W20260709 12:16:07.662999 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.663854 12492 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260709 12:16:07.669858 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98482.5us
I20260709 12:16:07.673348 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.675451 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.672111 12563 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:07.677009 12572 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: [23.16-25.17]
I20260709 12:16:07.681857 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.683271 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.670785 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:07.696465 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:07.697978 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:07.554570 22647 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:16:07.519523 22634 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:07.728016 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:07.722766 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47387.4us
I20260709 12:16:07.724472 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.732316 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:07.702250 12492 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:07.723996 12641 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:16:07.730283 12491 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:07.735231 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.011s	sys 0.040s
W20260709 12:16:07.737756 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.011s	sys 0.040s
I20260709 12:16:07.732640 12643 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:16:07.717401 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46711.6us
I20260709 12:16:07.744908 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:07.668565 22660 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.560s	user 0.012s	sys 0.000s
W20260709 12:16:07.668866 22530 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.607s	user 0.012s	sys 0.002s
I20260709 12:16:07.809573 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:07.740237 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 515292us
I20260709 12:16:07.873486 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:07.877756 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:07.865363 22310 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.272s	user 0.012s	sys 0.006s
I20260709 12:16:07.814355 12642 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:07.814783 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78413.9us
I20260709 12:16:07.672096 22502 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:07.822480 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:07.814805 22707 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:16:07.824230 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:07.695073 22675 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:07.734941 22514 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:16:07.821480 22680 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.130s	user 0.014s	sys 0.000s
I20260709 12:16:07.758577 22687 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:07.758067 22701 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:07.671149 22491 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.161s
I20260709 12:16:07.821244 22486 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:07.735572 22551 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.113s
I20260709 12:16:07.863843 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:07.946640 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:07.947810 12490 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:07.739797 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:07.959937 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 220206us
I20260709 12:16:07.679401 22608 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.836282 22702 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:07.883811 12492 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:07.883419 12496 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:07.912189 12644 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:16:07.911245 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:07.970414 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:07.975963 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64800us
I20260709 12:16:07.973116 12644 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 18
I20260709 12:16:07.984622 12644 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:16:07.985491 12644 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:16:07.970760 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.032s	sys 0.050s
W20260709 12:16:07.989073 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.032s	sys 0.051s
W20260709 12:16:07.937603 22699 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.005s	sys 0.007s
I20260709 12:16:07.918124 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:07.906551 22506 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:16:07.762655 22492 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:07.926342 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.005885 12643 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:07.881194 22660 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:07.938217 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.012s	sys 0.063s
W20260709 12:16:08.062279 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.012s	sys 0.064s
I20260709 12:16:07.983258 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:07.969152 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:07.949714 22556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.003s	sys 0.021s
I20260709 12:16:07.995504 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:07.997435 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.067334 12492 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:08.068466 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.068861 12492 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:16:07.921607 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:07.812994 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:07.933089 22687 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:08.003619 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:07.970400 22545 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:08.012702 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.078442 13254 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:16:08.083128 12488 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:08.080785 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:07.928408 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:08.086472 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.115s	sys 0.198s
I20260709 12:16:07.929749 22675 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.088882 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.010s	sys 0.045s
W20260709 12:16:08.089357 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.011s	sys 0.045s
I20260709 12:16:07.881121 22530 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:08.033994 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.007s	sys 0.020s
W20260709 12:16:08.122877 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.008s	sys 0.020s
W20260709 12:16:08.033695 22232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.005s	sys 0.022s
I20260709 12:16:08.080763 22557 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:07.991978 22702 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.000178 22699 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.031239 22277 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.004s
I20260709 12:16:08.060895 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3926) took 1384 ms. Trace:
I20260709 12:16:08.064862 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.041908 22634 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:07.995826 22680 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:16:08.086899 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.115s	sys 0.198s
W20260709 12:16:08.124547 22232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.005s	sys 0.023s
I20260709 12:16:08.088443 12536 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:08.018671 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.093223 12489 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:08.066422 22556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.003s	sys 0.022s
W20260709 12:16:08.095495 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:08.160111 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.098510 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:08.157778 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 782836us
I20260709 12:16:08.080322 22544 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:16:07.972213 22486 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.127048 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.030s	sys 0.087s
W20260709 12:16:08.168062 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.031s	sys 0.087s
W20260709 12:16:08.132468 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.077560 22687 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:08.138833 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.088021 22432 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.088896 22520 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.094219 22713 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.196678 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.100033 22580 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:08.148976 12496 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:16:08.158363 12536 leader_election.cc:403] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:16:08.081555 22706 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.256s	user 0.001s	sys 0.001s
W20260709 12:16:08.208102 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119803us
I20260709 12:16:08.162844 22451 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:08.208976 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:08.161561 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.120452 22631 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:08.142601 22571 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.000s
W20260709 12:16:08.202135 22657 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.750s	user 0.008s	sys 0.003s
I20260709 12:16:08.162518 12641 raft_consensus.cc:2368] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:08.102584 22310 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.153674 12646 rpcz_store.cc:276] 0709 12:16:06.676619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:06.676853 (+   234us) service_pool.cc:224] Handling call
0709 12:16:06.725368 (+ 48515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:06.738578 (+ 13210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:06.738626 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:06.778104 (+ 39478us) log.cc:844] Serialized 26 byte log entry
0709 12:16:07.286142 (+508038us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:07.286183 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:07.286267 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.046910 (+760643us) raft_consensus.cc:1685] finished
0709 12:16:08.046931 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.060807 (+ 13876us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22527,"raft.queue_time_us":11889,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":21811,"thread_start_us":368776,"threads_started":1,"wal-append.queue_time_us":850322}
I20260709 12:16:08.175225 13255 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:08.186920 12567 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:08.187124 12640 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:08.095186 22514 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:16:08.197811 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:08.140901 22499 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:16:08.162917 22441 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.163645 22680 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:08.211510 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:08.213802 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4194) took 1257 ms (client timeout 1510 ms). Trace:
I20260709 12:16:08.218475 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:08.219224 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:08.257092 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:08.195935 22716 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:16:08.234261 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.096153 22460 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:08.236976 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.160883 22277 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:08.215059 22642 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.047s	user 0.014s	sys 0.000s
W20260709 12:16:08.259991 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:08.256419 12564 rpcz_store.cc:269] 0709 12:16:06.956190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:06.962570 (+  6380us) service_pool.cc:224] Handling call
0709 12:16:08.213528 (+1250958us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:08.213708 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96178048}
I20260709 12:16:08.258984 12488 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:08.216065 22432 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.262109 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.241719 22556 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.234899 22314 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.268273 12488 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:08.264640 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.216109 22557 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:08.234941 22647 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:08.271116 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.274101 12564 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:16:08.203102 22520 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.241978 22232 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.249781 22657 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:08.280680 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.253677 22571 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:08.281060 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.285123 13253 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:08.265192 22680 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:08.283499 12564 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:08.205618 22713 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.258142 22608 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.264024 22460 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:08.268870 22642 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:08.240770 22631 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.290372 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.292659 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.238266 22451 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.276101 22514 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:08.251655 22310 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.306062 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.309202 12564 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:08.280737 22722 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:08.298997 22715 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.258s	user 0.001s	sys 0.001s
I20260709 12:16:08.312028 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.264878 22441 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.294447 22460 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:08.275862 22432 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.311060 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.308223 22514 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:08.318216 22451 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.320016 12639 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
W20260709 12:16:08.327623 22562 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.818s	user 0.012s	sys 0.001s
W20260709 12:16:08.332281 22540 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.786s	user 0.008s	sys 0.002s
I20260709 12:16:08.284873 22647 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.336766 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.356208 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.357069 12495 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:08.291033 22660 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.298261 12488 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:16:08.340502 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177594us
I20260709 12:16:08.361935 12488 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:16:08.336792 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 127928us
I20260709 12:16:08.344008 12492 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:08.348342 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.282608 22314 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.279286 22556 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.350229 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3930) took 1540 ms. Trace:
I20260709 12:16:08.290899 22232 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.366628 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:08.338941 22372 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:08.359236 22501 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.847s	user 0.010s	sys 0.004s
W20260709 12:16:08.318329 22719 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:16:08.371006 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.368786 12637 rpcz_store.cc:276] 0709 12:16:06.810031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.124314 (+314283us) service_pool.cc:224] Handling call
0709 12:16:07.124709 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:07.125169 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:07.125204 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:07.133421 (+  8217us) log.cc:844] Serialized 26 byte log entry
0709 12:16:07.565259 (+431838us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:07.565302 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:07.565378 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.349995 (+784617us) raft_consensus.cc:1685] finished
0709 12:16:08.350017 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.350168 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14879,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":15651,"thread_start_us":373806,"threads_started":1,"wal-append.queue_time_us":835901}
I20260709 12:16:08.377189 12637 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:08.378955 12564 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:08.373687 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109878us
I20260709 12:16:08.371853 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:08.375041 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.380435 12488 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
W20260709 12:16:08.370348 22725 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:16:08.388338 12639 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:08.380113 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.388697 22441 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.389528 22451 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:08.393549 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.359549 22675 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.393783 22699 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.394388 22702 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.386255 12536 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:08.372375 22492 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:16:08.384883 12637 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: []
I20260709 12:16:08.397455 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:08.398633 12495 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:08.369910 22530 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.396359 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.400782 12564 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:16:08.381771 22501 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.407610 22580 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 30.
I20260709 12:16:08.410104 22432 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.410321 22492 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:08.410707 22492 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:08.410898 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4051) took 1337 ms. Trace:
I20260709 12:16:08.411362 12566 rpcz_store.cc:276] 0709 12:16:07.073330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.238714 (+165384us) service_pool.cc:224] Handling call
0709 12:16:08.410699 (+1171985us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:08.410835 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230627584}
I20260709 12:16:08.412111 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4034) took 1403 ms. Trace:
I20260709 12:16:08.412547 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.412473 12565 rpcz_store.cc:276] 0709 12:16:07.009061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.315516 (+306455us) service_pool.cc:224] Handling call
0709 12:16:08.411036 (+1095520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.411243 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:08.411268 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:08.411305 (+    37us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:08.411339 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.411414 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.411955 (+   541us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400045950. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:08.412053 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2518380416}
I20260709 12:16:08.414808 22631 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.383275 22277 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.414858 22314 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.410960 22556 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.417186 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1422) took 1313 ms. Trace:
I20260709 12:16:08.417609 12493 rpcz_store.cc:276] 0709 12:16:07.103767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.104731 (+   964us) service_pool.cc:224] Handling call
0709 12:16:07.105134 (+   403us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:07.118502 (+ 13368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:07.118544 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:07.119148 (+   604us) log.cc:844] Serialized 52 byte log entry
0709 12:16:07.564473 (+445325us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:07.660759 (+ 96286us) log.cc:844] Serialized 14 byte log entry
0709 12:16:07.672970 (+ 12211us) log.cc:844] Serialized 14 byte log entry
0709 12:16:07.673265 (+   295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:07.673376 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.416934 (+743558us) raft_consensus.cc:1685] finished
0709 12:16:08.416960 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.417115 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17902,"prepare.queue_time_us":511845,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":87,"raft.queue_time_us":31461,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":40216,"spinlock_wait_cycles":92208384,"thread_start_us":444894,"threads_started":1,"wal-append.queue_time_us":871281}
I20260709 12:16:08.420655 22486 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.382179 22733 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:08.423588 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1202) took 1087 ms. Trace:
I20260709 12:16:08.424113 13253 rpcz_store.cc:276] 0709 12:16:07.338141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.338289 (+   148us) service_pool.cc:224] Handling call
0709 12:16:07.338638 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.288934 (+950296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.423276 (+134342us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.423513 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2167109504}
I20260709 12:16:08.427592 22232 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.428694 12566 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 15. Preceding OpId from leader: term: 33 index: 16. (index mismatch)
I20260709 12:16:08.430172 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.403326 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4053) took 1045 ms. Trace:
I20260709 12:16:08.431252 22700 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:08.433032 12564 rpcz_store.cc:276] 0709 12:16:07.357960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.396743 (+1038783us) service_pool.cc:224] Handling call
0709 12:16:08.400448 (+  3705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.403027 (+  2579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.403261 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:08.438578 22700 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.439172 22700 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.440212 12641 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
W20260709 12:16:08.442009 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 458902us
I20260709 12:16:08.444903 22507 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.445873 22297 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.446432 22432 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:08.402124 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.385176 22514 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.430904 12492 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
W20260709 12:16:08.419063 22497 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.906s	user 0.013s	sys 0.000s
I20260709 12:16:08.446493 22507 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:08.448472 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46423.7us
I20260709 12:16:08.461745 22680 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.448946 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52665.8us
I20260709 12:16:08.407720 22501 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.441478 22700 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.339558 22722 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.399863 22585 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.936s	user 0.013s	sys 0.000s
I20260709 12:16:08.395295 22563 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:08.433125 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:08.381019 22372 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.401523 22675 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.482661 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.456096 22492 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:08.488577 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.488708 22492 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:08.489814 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:08.448231 22635 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:08.456799 22515 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.456133 22441 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.492605 22441 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.493098 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4038) took 1477 ms. Trace:
I20260709 12:16:08.494494 22677 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:08.446504 22297 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.478298 22563 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:08.458428 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4035) took 1449 ms. Trace:
I20260709 12:16:08.496775 22563 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.497704 12567 rpcz_store.cc:276] 0709 12:16:07.009291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.334886 (+325595us) service_pool.cc:224] Handling call
0709 12:16:07.335318 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.247575 (+912257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.458145 (+210570us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.458358 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1957945216}
I20260709 12:16:08.498986 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.500172 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.483421 22491 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.500895 22491 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.459132 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4033) took 1450 ms. Trace:
I20260709 12:16:08.502039 12569 rpcz_store.cc:276] 0709 12:16:07.008786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.298512 (+289726us) service_pool.cc:224] Handling call
0709 12:16:07.298890 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:07.306997 (+  8107us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:07.307031 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:07.324849 (+ 17818us) log.cc:844] Serialized 26 byte log entry
0709 12:16:07.520361 (+195512us) spinlock_profiling.cc:243] Waited 7.22 ms on lock 0x7b180032df40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:16:08.089839 (+569478us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:08.089884 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.089955 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.458820 (+368865us) raft_consensus.cc:1685] finished
0709 12:16:08.458847 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.459076 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7695,"raft.queue_time_us":14599,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":4717,"spinlock_wait_cycles":16596352,"thread_start_us":569268,"threads_started":1,"wal-append.queue_time_us":925180}
I20260709 12:16:08.503803 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.505012 12569 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:16:08.448253 22514 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:08.463069 22504 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:16:08.461290 22432 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:08.478520 22660 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:08.463769 12496 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:08.447638 22530 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:08.480584 22372 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.506963 22310 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.508973 22432 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.520965 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3904) took 2949 ms (client timeout 1510 ms). Trace:
I20260709 12:16:08.501302 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.431797 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.465484 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.493530 12572 rpcz_store.cc:276] 0709 12:16:07.015936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.529913 (+513977us) service_pool.cc:224] Handling call
0709 12:16:07.676593 (+146680us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:07.677573 (+   980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.492902 (+815329us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001b17d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:08.493035 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233302272}
W20260709 12:16:08.526182 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60796.5us
I20260709 12:16:08.527357 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.464427 22680 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.487301 22675 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.492081 22635 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:08.521404 12645 rpcz_store.cc:269] 0709 12:16:05.578283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:05.591588 (+ 13305us) service_pool.cc:224] Handling call
0709 12:16:08.520683 (+2929095us) spinlock_profiling.cc:243] Waited 2.74 s on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:08.520822 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6297981824}
W20260709 12:16:08.528126 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96414us
I20260709 12:16:08.537940 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.528785 12492 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 22. Current term is 23. Ops: [22.12-22.12]
W20260709 12:16:08.528920 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44810us
I20260709 12:16:08.533427 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:08.527048 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.542138 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:08.481421 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:08.519083 22649 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.080s	user 0.009s	sys 0.004s
I20260709 12:16:08.470479 22722 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:08.518941 22192 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.001s	user 0.013s	sys 0.001s
I20260709 12:16:08.518009 22538 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:08.559582 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:08.595640 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46711.2us
W20260709 12:16:08.603932 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61894.1us
I20260709 12:16:08.519644 22713 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.518671 22520 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.515947 22441 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:08.537890 22726 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.187s	user 0.000s	sys 0.004s
I20260709 12:16:08.510408 22530 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:08.611285 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.633198 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.614293 12645 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:08.615653 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.634577 12646 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:16:08.617355 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1834) took 1467 ms. Trace:
I20260709 12:16:08.594895 22542 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.534147 22504 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:08.536708 22623 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:16:08.467011 22497 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.620673 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.625998 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:08.631505 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.652611 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:08.652889 12640 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 35 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:16:08.603574 22545 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:08.540791 22566 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:08.603428 22515 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.665936 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.509959 22660 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.595002 22310 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:16:08.670665 12488 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:16:08.616520 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.638854 22702 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:08.603255 22700 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:08.638921 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.647989 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:08.678304 13261 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:08.649551 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.605960 22732 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:08.549355 22585 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.089s
I20260709 12:16:08.655471 22701 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:16:08.655791 22675 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.656656 22530 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.603615 22506 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:16:08.618961 22297 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.661571 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.534273 22372 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.113s
I20260709 12:16:08.606573 22677 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 12:16:08.616621 22632 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:16:08.727478 22632 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:08.727928 22632 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:08.532737 22631 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:16:08.671960 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55575.4us
I20260709 12:16:08.731825 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.702591 22675 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.732695 22297 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:08.636381 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.606735 22649 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.679217 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.683037 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1290) took 1328 ms. Trace:
W20260709 12:16:08.684986 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101091us
I20260709 12:16:08.687683 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.658319 22545 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:08.659415 22504 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:08.751253 22504 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.686947 22701 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:08.746405 22631 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:08.754634 22701 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.756131 22701 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:08.756399 22649 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.757421 13256 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32.
W20260709 12:16:08.758033 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.650115 22542 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.758172 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:08.708602 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4036) took 1699 ms. Trace:
I20260709 12:16:08.759151 12568 rpcz_store.cc:276] 0709 12:16:07.009526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.335026 (+325500us) service_pool.cc:224] Handling call
0709 12:16:07.335471 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:07.450290 (+114819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:07.450321 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:07.456800 (+  6479us) log.cc:844] Serialized 26 byte log entry
0709 12:16:08.123360 (+666560us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:08.123400 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.707920 (+584520us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.707997 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.708076 (+    79us) raft_consensus.cc:1685] finished
0709 12:16:08.708095 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.708481 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1227,"raft.queue_time_us":1885,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":7803,"spinlock_wait_cycles":259977088,"thread_start_us":666211,"threads_started":1,"wal-append.queue_time_us":786426}
I20260709 12:16:08.715261 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.718348 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:08.635160 12570 rpcz_store.cc:276] 0709 12:16:07.150189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.386343 (+1236154us) service_pool.cc:224] Handling call
0709 12:16:08.386633 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.387128 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:16:08.387524 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 12:16:08.617033 (+229509us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:08.617085 (+    52us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:08.617123 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.617182 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.617298 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11426,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":254,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":10643,"thread_start_us":229137,"threads_started":1}
W20260709 12:16:08.719583 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58111.6us
I20260709 12:16:08.744693 13255 rpcz_store.cc:276] 0709 12:16:07.354686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.355187 (+   501us) service_pool.cc:224] Handling call
0709 12:16:07.355490 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.243246 (+887756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.682630 (+439384us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.682943 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884741376}
I20260709 12:16:08.676229 22515 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.732769 22675 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.767748 22542 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.728844 22632 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.768154 22515 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.687335 22626 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:08.739076 12566 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:08.745743 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:08.785387 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.746999 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.733436 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.785863 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:08.786433 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:08.786836 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53475.7us
I20260709 12:16:08.787683 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:08.787993 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.693799 22507 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:08.733381 22297 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:08.800868 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:08.801968 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:08.735046 22277 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:08.685621 22310 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:08.750171 22545 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:08.699756 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1426) took 1181 ms. Trace:
I20260709 12:16:08.773027 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3212) took 1082 ms. Trace:
I20260709 12:16:08.642107 22635 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.744979 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65831.6us
I20260709 12:16:08.789534 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:08.735544 22501 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.813872 12489 rpcz_store.cc:276] 0709 12:16:07.690946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.691166 (+   220us) service_pool.cc:224] Handling call
0709 12:16:07.695998 (+  4832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.162666 (+466668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.772624 (+609958us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.772932 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":912884608}
I20260709 12:16:08.794212 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.783243 22632 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:08.803933 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4203) took 1818 ms. Trace:
I20260709 12:16:08.758402 22520 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.817800 12490 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:08.821372 12571 rpcz_store.cc:276] 0709 12:16:06.985466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.313462 (+327996us) service_pool.cc:224] Handling call
0709 12:16:07.318059 (+  4597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:07.327428 (+  9369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:07.327463 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:07.328037 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 12:16:07.989671 (+661634us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:07.989740 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.790362 (+800622us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.790420 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.790488 (+    68us) raft_consensus.cc:1685] finished
0709 12:16:08.790504 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.803821 (+ 13317us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1353,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":113,"spinlock_wait_cycles":20580480,"thread_start_us":661097,"threads_started":1,"wal-append.queue_time_us":870854}
I20260709 12:16:08.808205 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.686895 22733 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.108s
I20260709 12:16:08.816725 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:08.688364 22700 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.811394 12494 rpcz_store.cc:276] 0709 12:16:07.518494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.520148 (+  1654us) service_pool.cc:224] Handling call
0709 12:16:07.530325 (+ 10177us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:07.530800 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:07.530839 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:07.531497 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:16:08.169385 (+637888us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:08.169447 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.169810 (+   363us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:08.504108 (+334298us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:08.691818 (+187710us) raft_consensus.cc:1685] finished
0709 12:16:08.691847 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:08.699670 (+  7823us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1965,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":8802,"spinlock_wait_cycles":6238976,"thread_start_us":489065,"threads_started":1,"wal-append.queue_time_us":717064}
I20260709 12:16:08.835016 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:08.836573 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.837416 12489 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:08.823536 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.824970 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:08.844596 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55219.9us
I20260709 12:16:08.847476 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.820473 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.701025 22687 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.803010 22713 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:08.855767 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:08.803606 22542 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:08.784003 22626 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:08.818676 22732 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:08.857606 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:08.859571 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1205) took 1514 ms (client timeout 1936 ms). Trace:
I20260709 12:16:08.840405 12493 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:08.806057 22297 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:08.802445 22515 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:08.869172 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.902263 12570 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:08.874469 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.869593 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:08.903308 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45788.2us
I20260709 12:16:08.830456 22514 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.909008 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.861245 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:08.811116 22678 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:08.806926 22486 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.084s
I20260709 12:16:08.858664 22680 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:08.813220 22717 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:08.809048 22606 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.810714 22232 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.084s
I20260709 12:16:08.885828 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:08.797523 22502 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:08.896271 13256 rpcz_store.cc:269] 0709 12:16:07.344884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.345095 (+   211us) service_pool.cc:224] Handling call
0709 12:16:08.841026 (+1495931us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:08.859500 (+ 18474us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3174208896}
I20260709 12:16:08.842056 22324 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:08.851439 22580 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:08.882647 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.866467 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.830626 22507 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:08.836283 22501 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:08.996728 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130363us
I20260709 12:16:08.915169 12497 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 35 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:16:08.859089 22642 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:08.916690 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:08.842028 22675 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.916623 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:08.919534 12492 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25. Ops: [24.15-24.15]
W20260709 12:16:08.940080 22737 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:16:08.894392 22626 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:08.831336 22608 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.082s
I20260709 12:16:08.952628 22660 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:08.843098 22520 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.922868 22678 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:08.970896 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.979164 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:09.058779 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 238373us
I20260709 12:16:09.062156 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.019719 22530 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:08.943554 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57789.2us
W20260709 12:16:09.068301 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:09.072189 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.946556 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3936) took 1838 ms. Trace:
I20260709 12:16:08.997886 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.871213 22623 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:16:09.006438 12640 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:09.008404 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:09.060148 12568 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:09.074702 12638 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:09.008879 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.075265 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.010994 12642 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:16:09.013504 12637 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 33 index: 16. (index mismatch)
I20260709 12:16:09.076946 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.077095 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.077931 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:09.074600 12567 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:09.073944 12644 rpcz_store.cc:276] 0709 12:16:07.108460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.228038 (+119578us) service_pool.cc:224] Handling call
0709 12:16:07.228325 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:07.986893 (+758568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.946279 (+959386us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:08.946488 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1571856256}
I20260709 12:16:09.079231 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.918370 22620 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:09.086552 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:08.889871 22665 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.095s
W20260709 12:16:08.940080 22738 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:16:09.028561 22642 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.956686 22310 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:08.925743 22717 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:09.029948 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125197us
I20260709 12:16:09.037974 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.080735 22497 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:08.956048 22677 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:08.936928 22634 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:09.019850 22745 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:16:08.943760 22372 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:09.042426 12645 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:08.902498 22732 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.043331 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:08.952345 22647 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:08.945158 22491 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.054821 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.063014 12570 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:09.029176 22515 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:16:08.984331 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.923455 22514 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.917773 22566 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:09.079669 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.080017 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.081909 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:09.145119 22591 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:08.975116 22502 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:08.975297 22324 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:09.099052 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101229us
I20260709 12:16:09.235616 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:08.995128 22606 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.112067 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:09.112605 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:09.118532 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:08.980558 22741 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:16:09.065261 22678 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:09.119879 12638 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:08.886773 12490 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
W20260709 12:16:09.110148 22460 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.755s	user 0.009s	sys 0.003s
I20260709 12:16:09.086925 22649 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:09.124639 12639 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
W20260709 12:16:09.249012 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:08.979813 22556 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:09.110737 22504 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:09.138875 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:09.130292 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.086658 22660 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:16:09.151234 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4057) took 1326 ms. Trace:
I20260709 12:16:09.157369 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.159569 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:09.126870 22717 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:09.086310 22530 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.138140 22277 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:09.193886 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.078276 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.239512 22700 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.193724 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.212496 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.195047 12567 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:09.271440 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.215706 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:09.219321 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.223129 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:09.218365 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50833.7us
I20260709 12:16:09.225291 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.226639 12644 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:09.202451 12641 raft_consensus.cc:2368] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:16:09.161526 22732 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:09.201674 22749 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:16:09.254914 22504 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.238397 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:09.227520 22492 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.738s	user 0.006s	sys 0.007s
I20260709 12:16:09.244024 22499 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.028s
I20260709 12:16:09.316958 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:09.239544 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:09.154151 22634 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.258054 12565 rpcz_store.cc:276] 0709 12:16:07.824827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.425797 (+600970us) service_pool.cc:224] Handling call
0709 12:16:08.426139 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:08.426378 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:08.426407 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:08.426881 (+   474us) log.cc:844] Serialized 26 byte log entry
0709 12:16:08.628959 (+202078us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:08.629009 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:08.629102 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:09.150858 (+521756us) raft_consensus.cc:1685] finished
0709 12:16:09.150893 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:09.151139 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":201661,"threads_started":1,"wal-append.queue_time_us":251166}
W20260709 12:16:09.329206 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 210758us
I20260709 12:16:09.329493 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.193768 22566 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:09.251282 22562 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:09.251310 22620 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:09.258621 12564 raft_consensus.cc:2376] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33. Re-sending same reply.
I20260709 12:16:09.340754 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.341706 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:09.343029 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:09.259517 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121018us
I20260709 12:16:09.233963 22687 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.129603 22677 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:09.318050 22536 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:09.315958 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122186us
W20260709 12:16:09.316455 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100925us
I20260709 12:16:09.251636 22657 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:09.316498 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:09.232944 22563 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:16:09.251842 22556 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:09.167797 22515 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:09.340253 22566 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:16:09.268873 22747 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:16:09.324491 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:09.349011 22520 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.325112 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.129941 22507 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:09.269955 22538 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.664s	user 0.012s	sys 0.002s
I20260709 12:16:09.327677 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.327550 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.202019 22591 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:09.466064 22642 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.466372 22591 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.468214 22591 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:09.331627 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92187.8us
I20260709 12:16:09.472270 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:09.395583 22314 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:09.473443 22314 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:09.330380 22760 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:09.474609 22314 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.476631 22536 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:09.240156 22755 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.095s
I20260709 12:16:09.333925 22660 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:09.478210 22660 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.478649 22660 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:09.478693 22502 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.462126 22761 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:09.263230 22519 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:09.477298 22536 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:09.482112 22536 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:16:09.334870 22748 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:16:09.169682 22752 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.173s
I20260709 12:16:09.335127 13256 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
W20260709 12:16:09.335753 22754 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:16:09.484055 22519 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:16:09.484489 22519 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:09.485352 22519 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.486598 22519 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:09.335788 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.403240 22515 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:09.488194 13261 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:09.332334 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:09.336691 22744 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:16:09.252061 22460 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.492316 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1841) took 1258 ms. Trace:
I20260709 12:16:09.338984 12565 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:16:09.494210 22451 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:09.494889 22451 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.495855 12563 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 28. Ops: []
I20260709 12:16:09.348289 12491 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:09.230589 22372 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:16:09.318197 22486 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:09.498309 22486 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:09.326663 22504 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:09.499475 22556 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:09.359174 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117877us
I20260709 12:16:09.365826 12536 leader_election.cc:400] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:09.320269 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:09.500960 12536 leader_election.cc:403] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:09.492754 12569 rpcz_store.cc:276] 0709 12:16:08.233960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.523314 (+289354us) service_pool.cc:224] Handling call
0709 12:16:08.523566 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.333518 (+809952us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:09.333567 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:09.333615 (+    48us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:09.333682 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:09.492063 (+158381us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:09.492124 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:09.492260 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1017,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":1857700736}
I20260709 12:16:09.358677 22620 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:09.401099 22701 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.644s	user 0.007s	sys 0.008s
I20260709 12:16:09.401036 22297 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.503306 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:09.505038 22297 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.405709 22324 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.333954 22606 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.353363 22562 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.503307 22701 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:16:09.415983 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:09.416846 22713 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.000s
I20260709 12:16:09.417858 22631 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:09.509485 22713 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:09.509882 22631 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.324590 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:09.429598 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104533us
I20260709 12:16:09.395172 22492 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.070s
I20260709 12:16:09.461742 22677 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.517274 22700 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:09.517457 22677 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:09.517831 22700 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:09.334543 22634 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.524951 22492 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:09.525730 22492 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.527742 22631 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.528452 22620 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.529920 22460 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.532371 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:09.545722 22562 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.548192 22492 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:09.545940 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 189248us
I20260709 12:16:09.550561 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.552094 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:09.552414 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.552475 12494 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:16:09.552096 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.503122 12569 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 14
I20260709 12:16:09.557469 12490 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:09.546576 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 214362us
I20260709 12:16:09.559430 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:09.557381 12569 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:16:09.565577 12569 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:09.555131 12570 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:09.571154 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:09.571334 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85663.5us
I20260709 12:16:09.575088 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.579669 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.580538 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:09.580595 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:09.573005 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:09.538362 22372 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:09.584869 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:09.586238 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.570199 22530 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.586490 22606 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:09.593832 22733 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:09.571951 22460 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:09.604754 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:09.605945 12496 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:16:09.602563 22297 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:16:09.607175 22562 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:09.610098 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.611640 12644 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:09.609608 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.613080 13262 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:16:09.613723 22718 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.368s	user 0.002s	sys 0.004s
I20260709 12:16:09.615521 22755 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:09.616060 22755 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:09.617100 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1437) took 1005 ms. Trace:
I20260709 12:16:09.619724 12488 rpcz_store.cc:276] 0709 12:16:08.611221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.873341 (+262120us) service_pool.cc:224] Handling call
0709 12:16:08.873870 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.615378 (+741508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:09.615416 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:09.615462 (+    46us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:09.615496 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:09.616822 (+  1326us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:09.616869 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:09.617029 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2486,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":1704369408}
I20260709 12:16:09.621552 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:09.597448 22761 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:09.627486 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.628752 12496 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:09.635042 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.598809 22752 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.028s
I20260709 12:16:09.605739 22585 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:09.604027 22514 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.595082 22372 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.634665 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:09.618743 22538 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:09.651351 22506 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:09.652419 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:09.656342 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.664014 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.664868 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:09.617054 22755 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 30.11, Last appended by leader: 11, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.659821 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.666314 13255 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:09.636560 22759 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:09.665467 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.659822 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.705371 22732 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.707736 22732 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.708918 22536 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.690972 22755 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:09.712361 22759 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:09.712914 22759 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:09.643437 22585 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:09.687920 12646 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:16:09.722031 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:16:09.668979 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:09.718464 22585 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.638661 22631 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:09.664716 13262 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:09.669498 12637 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:16:09.722386 22310 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:09.742410 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:09.742477 22585 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:09.742563 22310 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.748190 22310 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.719605 22506 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:09.752803 22506 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:09.686880 22545 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.846s	user 0.007s	sys 0.012s
I20260709 12:16:09.758078 22545 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.759773 13254 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:09.763266 22545 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.764463 22556 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:09.774709 22660 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:09.770207 22764 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.201s	user 0.000s	sys 0.003s
I20260709 12:16:09.770292 22536 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.772413 22562 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:09.765475 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:09.776580 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:09.699527 22372 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:09.780650 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121428us
W20260709 12:16:09.790194 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124818us
I20260709 12:16:09.808200 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.808964 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.809645 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
W20260709 12:16:09.776821 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124399us
I20260709 12:16:09.813161 12488 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:16:09.815160 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:09.815778 12494 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:09.818218 12462 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32.
I20260709 12:16:09.818634 12462 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:16:09.820672 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.823937 12493 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:16:09.826834 13255 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.21-29.21]
W20260709 12:16:09.785212 22717 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.518s	user 0.006s	sys 0.006s
I20260709 12:16:09.829650 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:09.831537 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:09.740083 22515 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.854210 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:09.796090 22538 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:09.855832 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:09.855221 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:09.789718 22713 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:09.803148 22562 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:09.852952 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:09.801455 22310 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:09.873769 12640 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:09.874397 12494 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:09.866099 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:09.850431 22678 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.598s	user 0.008s	sys 0.006s
I20260709 12:16:09.828255 22732 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:09.837502 22660 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.860244 22192 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:09.861160 22538 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.888144 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:09.864737 22701 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:09.889533 12461 leader_election.cc:400] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:16:09.900198 12461 leader_election.cc:403] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:16:09.898905 13262 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:09.792727 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 198417us
I20260709 12:16:09.832211 22486 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
W20260709 12:16:09.901355 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83407.3us
I20260709 12:16:09.894958 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:09.904882 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:09.911790 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:09.906448 22642 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:09.838135 22504 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.877784 22536 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32.
I20260709 12:16:09.839538 22566 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.928952 22566 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.839635 22759 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.931672 22759 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.932415 12567 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:16:09.916551 22519 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:09.878389 22530 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:09.934466 22530 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.937186 22530 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.908617 13255 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:09.913930 12570 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:16:09.917232 22776 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:16:09.912818 22642 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.948766 22642 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.883966 22678 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.885944 22502 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:09.951134 22502 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.951285 22678 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.951679 22502 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:09.873876 22641 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:16:09.850838 22717 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:09.953480 22626 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.895s	user 0.007s	sys 0.007s
I20260709 12:16:09.954391 22626 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.955330 22717 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.956127 12564 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:09.919375 12494 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:09.957037 22626 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:09.919531 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:09.957388 22675 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:09.957859 22675 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:09.957928 12493 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:09.959018 22675 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.960187 22675 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:09.964406 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:09.921679 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:09.966423 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46680.9us
W20260709 12:16:09.966969 22736 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.176s	user 0.008s	sys 0.005s
I20260709 12:16:09.886045 22497 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:09.968364 22497 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.968820 22497 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:09.913499 22506 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31.
I20260709 12:16:09.921170 22761 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:09.971745 22761 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:09.972817 22761 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:09.909099 22538 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:09.973837 22649 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:09.974807 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1295) took 1238 ms. Trace:
I20260709 12:16:09.976986 13254 rpcz_store.cc:276] 0709 12:16:08.736295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.737058 (+   763us) service_pool.cc:224] Handling call
0709 12:16:08.769466 (+ 32408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.760280 (+990814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:09.974498 (+214218us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:09.974730 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2276784896}
W20260709 12:16:09.980108 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65244.9us
I20260709 12:16:09.980993 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:09.981793 22506 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:09.982189 22506 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:09.982515 22506 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:09.984001 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4052) took 2910 ms. Trace:
I20260709 12:16:09.984458 12563 rpcz_store.cc:276] 0709 12:16:07.073572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:07.677718 (+604146us) service_pool.cc:224] Handling call
0709 12:16:08.214334 (+536616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.496497 (+1282163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:09.982992 (+486495us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:09.983702 (+   710us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54002463d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:09.983880 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4179970304}
I20260709 12:16:09.886818 22732 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:09.915699 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:10.000499 12642 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.011757 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:10.024156 12643 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:10.017453 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:10.026927 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:10.028275 12494 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
W20260709 12:16:10.018426 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.014534 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4276) took 1347 ms. Trace:
I20260709 12:16:10.033414 12489 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:10.032768 12497 rpcz_store.cc:276] 0709 12:16:08.666696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.022530 (+355834us) service_pool.cc:224] Handling call
0709 12:16:09.238929 (+216399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:09.242436 (+  3507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:09.242467 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:09.247177 (+  4710us) log.cc:844] Serialized 26 byte log entry
0709 12:16:09.605860 (+358683us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:16:09.605902 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:09.606051 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:09.691200 (+ 85149us) raft_consensus.cc:1685] finished
0709 12:16:09.691221 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:09.945338 (+254117us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540033e850. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:10.014454 (+ 69116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3126,"raft.queue_time_us":5370,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4805,"spinlock_wait_cycles":429395200,"thread_start_us":308190,"threads_started":1,"wal-append.queue_time_us":390829}
I20260709 12:16:10.039253 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:10.041065 13262 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:16:10.044132 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49966.5us
I20260709 12:16:10.044168 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4265) took 1804 ms. Trace:
I20260709 12:16:10.046085 12494 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:16:10.041827 12497 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:10.048360 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:10.010885 22678 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:16:10.047804 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:10.051585 13254 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:09.989202 22774 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:16:10.044835 12495 rpcz_store.cc:276] 0709 12:16:08.240156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.400157 (+160001us) service_pool.cc:224] Handling call
0709 12:16:08.446001 (+ 45844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:08.470069 (+ 24068us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:16:08.666255 (+196186us) log.cc:844] Serialized 14 byte log entry
0709 12:16:09.042931 (+376676us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:09.043013 (+    82us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:09.043091 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.043894 (+1000803us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b540037e454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:10.043948 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:10.044104 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":252355,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":47,"raft.queue_time_us":108,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":7033,"spinlock_wait_cycles":54702336,"thread_start_us":571960,"threads_started":2,"wal-append.queue_time_us":447014}
I20260709 12:16:10.043131 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:10.061980 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:10.062714 13253 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:09.997699 22515 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:09.999374 22504 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:10.004271 22732 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:09.995429 22520 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:10.081580 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:10.065037 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:10.066287 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:10.008935 22451 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.071931 13257 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:10.080295 12610 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:10.089226 12497 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:10.093544 12610 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
W20260709 12:16:10.106494 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 190870us
I20260709 12:16:10.000269 22717 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.073s
I20260709 12:16:09.993549 22506 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.057974 22606 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:16:10.109866 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.121444 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:10.111454 13254 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:10.095767 22591 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:10.113130 12643 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:10.110893 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.127724 12566 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:10.129236 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.129948 12563 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:10.100764 22515 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:10.136071 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:10.138347 22566 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:10.139254 22566 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:10.139578 22566 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:10.139884 22566 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:10.096112 22520 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.141597 12640 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:10.142902 22501 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:10.142994 22520 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.143666 22641 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:10.144232 22641 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:10.145100 22641 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.146659 22492 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:10.147118 22492 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:10.118214 22776 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:10.117882 22497 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.126456 22499 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:16:10.155920 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:10.156386 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1433) took 1546 ms. Trace:
I20260709 12:16:10.156776 12638 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:10.112586 22701 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.095994 22687 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:10.111660 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.119961 22507 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:10.103705 22519 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:10.133036 13262 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:10.134944 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.157228 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:08.950845 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:10.123871 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.222708 12572 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:10.157006 12491 rpcz_store.cc:276] 0709 12:16:08.609600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.609789 (+   189us) service_pool.cc:224] Handling call
0709 12:16:10.156187 (+1546398us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:10.156326 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1182716800}
I20260709 12:16:10.115077 22451 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.164984 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:10.199051 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:10.226668 13254 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:10.162981 12566 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:10.175181 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:10.123706 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:10.118119 22631 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.233189 12638 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:10.190436 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:10.235116 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.237063 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:10.239377 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:10.136091 22780 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:16:10.242290 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:10.221061 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.227759 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.244009 12566 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:10.233598 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 110012us
I20260709 12:16:10.179410 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.207589 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:10.197973 22678 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:10.178190 22687 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:10.244756 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:10.258919 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:10.239605 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64518.9us
I20260709 12:16:10.260473 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.260964 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:10.245059 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.246902 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:10.254365 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.132215 22538 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.264369 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:10.266078 12490 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:10.268724 22538 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.220446 22519 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:10.233794 22514 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.233474 22507 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:10.286530 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:10.287611 12494 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:10.238332 22620 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.241331 22585 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:10.274308 12638 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.200233 22701 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.251098 22755 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:10.252604 22608 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:10.292116 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.291792 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:10.272151 22687 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:10.258278 22776 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:10.254644 22634 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:10.318908 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.334419 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:10.329583 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.330235 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:10.331100 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.336382 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:10.239964 22631 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.279426 22649 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:10.235613 22451 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.290730 22585 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:10.226702 22492 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.271958 22571 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
W20260709 12:16:10.161887 22680 consensus_queue.cc:1175] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 28 status { last_received { term: 27 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:16:10.400760 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.280100 22491 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:16:10.406184 22623 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:10.272759 22678 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.407757 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1297) took 1005 ms. Trace:
I20260709 12:16:10.408191 13257 rpcz_store.cc:276] 0709 12:16:09.402577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.402732 (+   155us) service_pool.cc:224] Handling call
0709 12:16:10.065464 (+662732us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:10.091786 (+ 26322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.407528 (+315742us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b54004ef7d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:10.407689 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1523159296}
I20260709 12:16:10.371450 22497 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.373337 22701 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.318276 22620 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.366113 22585 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:10.333655 22606 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:10.378296 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:10.386090 22507 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.360651 22634 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:10.426493 22634 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:10.389549 22562 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.004s
I20260709 12:16:10.432960 22562 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.435925 22562 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.397167 22502 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:10.443404 22753 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.183s	user 0.000s	sys 0.002s
I20260709 12:16:10.331005 22776 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:10.403759 22451 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.404139 22649 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:10.444422 22776 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:10.406627 22623 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:10.446394 22623 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.14, Last appended by leader: 14, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.447472 22623 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:10.409114 22678 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.287609 22514 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.450009 22631 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.413460 22642 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:10.354243 22755 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:10.441314 22502 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:10.328842 22608 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.450452 22514 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.451234 22606 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.445255 22649 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.462009 22700 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.479805 22620 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.452378 22642 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:10.494786 22642 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.495239 22687 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.496275 22642 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:10.498162 22755 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.503086 22700 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:10.511660 22608 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:10.518191 22606 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.518339 22687 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:10.560379 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200267us
I20260709 12:16:10.602705 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:10.560930 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 309273us
W20260709 12:16:10.562923 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.404247 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4061) took 1515 ms. Trace:
W20260709 12:16:10.606860 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.606560 12564 rpcz_store.cc:276] 0709 12:16:08.899124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.366939 (+467815us) service_pool.cc:224] Handling call
0709 12:16:09.393418 (+ 26479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.956526 (+563108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.403889 (+447363us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:10.404157 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1292395264}
I20260709 12:16:10.608348 22514 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:10.402058 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.609581 22514 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.565586 12497 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:10.612409 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.613903 22514 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.568248 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1856) took 1042 ms. Trace:
I20260709 12:16:10.614616 22514 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.571342 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:10.361711 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:10.337095 12494 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:10.615067 12565 rpcz_store.cc:276] 0709 12:16:09.526151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.910975 (+384824us) service_pool.cc:224] Handling call
0709 12:16:09.911293 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:09.911536 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:09.911560 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:09.917554 (+  5994us) log.cc:844] Serialized 26 byte log entry
0709 12:16:10.293650 (+376096us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:10.319423 (+ 25773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.319497 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:10.567418 (+247921us) raft_consensus.cc:1685] finished
0709 12:16:10.567463 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:10.567922 (+   459us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":225873,"threads_started":1,"wal-append.queue_time_us":439032}
I20260709 12:16:10.618337 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.618614 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:10.580581 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.622364 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.361301 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.624465 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.625332 12638 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.627619 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.628806 12638 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:10.402740 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:10.584834 12644 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.630568 12641 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:10.634907 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.636126 12640 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:10.627027 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:10.565586 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.590250 22502 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.591777 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:10.485536 22506 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.597963 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:10.619462 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:10.625356 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.615706 22514 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.671936 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:10.672181 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.508646 22566 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:10.685220 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80870.8us
W20260709 12:16:10.690819 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3950) took 1676 ms (client timeout 1892 ms). Trace:
W20260709 12:16:10.691632 12642 rpcz_store.cc:269] 0709 12:16:09.014201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.078006 (+ 63805us) service_pool.cc:224] Handling call
0709 12:16:10.690122 (+1612116us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:10.690528 (+   406us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1684352768}
I20260709 12:16:10.696061 22776 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.702514 12493 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.681582 13257 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:10.731195 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:10.709862 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:10.718250 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:10.674270 22700 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.711016 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.764267 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.732712 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:10.767354 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:10.713498 12489 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:10.714229 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:10.782240 22687 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:10.782929 22687 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:10.783385 22687 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:10.784240 12565 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:10.699033 22566 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.803910 12645 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:10.804067 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:10.807544 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46041.3us
I20260709 12:16:10.797252 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:10.811038 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:10.814910 22519 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.805583 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.808668 22631 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:10.760749 22502 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.818598 12641 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
W20260709 12:16:10.823038 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145454us
W20260709 12:16:10.818694 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.001s	sys 0.018s
I20260709 12:16:10.826954 12564 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:10.826220 22192 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:10.828388 22192 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:10.826416 22631 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.830230 22192 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:10.831156 22631 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.827322 12645 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: []
W20260709 12:16:10.845330 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.001s	sys 0.018s
I20260709 12:16:10.848796 12566 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 31
W20260709 12:16:10.841395 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.892s (SENT) [suppressed 11 similar messages]
I20260709 12:16:10.844542 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:10.855031 12493 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:10.854034 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.861743 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:10.872196 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 138778us
I20260709 12:16:10.874207 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4077) took 1589 ms. Trace:
I20260709 12:16:10.874399 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:10.874778 12570 rpcz_store.cc:276] 0709 12:16:09.284311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.699310 (+414999us) service_pool.cc:224] Handling call
0709 12:16:09.699673 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.938764 (+239091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.873613 (+934849us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:10.874000 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":491780352}
I20260709 12:16:10.876312 12565 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:10.885047 12488 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:16:10.861333 22657 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:10.871366 22519 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:10.857606 22776 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.825522 22700 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.859839 22635 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:10.889371 22657 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:10.908492 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:10.926898 12645 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:10.928843 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.934180 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:10.935220 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1391) took 2106 ms. Trace:
I20260709 12:16:10.939690 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:10.935352 12644 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.950002 22700 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:10.889061 22642 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:10.971946 12488 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:10.973824 12563 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:10.974722 12645 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:10.984181 22713 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:10.985010 22713 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:10.987035 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:10.988062 12644 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:10.999354 12646 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:11.001051 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3283) took 1332 ms. Trace:
W20260709 12:16:11.001668 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 177032us
I20260709 12:16:11.003152 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:11.006130 22297 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:11.006630 22297 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.007105 22297 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:11.010166 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:11.009995 22642 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.013113 22642 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.001428 12646 rpcz_store.cc:276] 0709 12:16:09.668608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.883182 (+214574us) service_pool.cc:224] Handling call
0709 12:16:10.699684 (+816502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:10.999035 (+299351us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b54001bef20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:16:11.000013 (+   978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.000775 (+   762us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54000a82d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:11.000983 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1836898048}
I20260709 12:16:11.019423 22642 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:11.020280 22642 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.026165 12494 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:11.027454 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.028843 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.029472 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:11.029824 12536 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:11.030926 22700 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.031667 22700 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.031910 22642 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:11.034194 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 119411us
I20260709 12:16:10.953109 22566 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:11.042088 22491 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:11.042903 22776 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:16:11.043824 22631 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.044552 22631 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.038022 22566 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:11.046476 22566 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:11.047575 22631 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:10.935987 13258 rpcz_store.cc:276] 0709 12:16:08.828861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:08.829080 (+   219us) service_pool.cc:224] Handling call
0709 12:16:08.884534 (+ 55454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:09.263635 (+379101us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:09.263667 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:09.314463 (+ 50796us) log.cc:844] Serialized 52 byte log entry
0709 12:16:09.575464 (+261001us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:09.649830 (+ 74366us) log.cc:844] Serialized 14 byte log entry
0709 12:16:09.650100 (+   270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:09.769129 (+119029us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:10.934213 (+1165084us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:10.934332 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:10.934408 (+    76us) raft_consensus.cc:1685] finished
0709 12:16:10.934425 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:10.934896 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9283,"prepare.queue_time_us":215822,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":4901,"raft.queue_time_us":81907,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":13782,"spinlock_wait_cycles":854705792,"thread_start_us":300293,"threads_started":2,"wal-append.queue_time_us":241903,"wal-append.run_cpu_time_us":7155,"wal-append.run_wall_time_us":1277022}
I20260709 12:16:11.052009 22502 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:11.052609 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:11.063073 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1446) took 1008 ms. Trace:
I20260709 12:16:11.063517 22514 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.072209 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1417) took 1027 ms. Trace:
I20260709 12:16:11.066946 12495 rpcz_store.cc:276] 0709 12:16:10.054968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.079010 (+ 24042us) service_pool.cc:224] Handling call
0709 12:16:10.106534 (+ 27524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:10.106788 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:10.106813 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:10.107312 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:16:10.903412 (+796100us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:10.957924 (+ 54512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.978701 (+ 20777us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:11.036709 (+ 58008us) raft_consensus.cc:1685] finished
0709 12:16:11.036723 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.062975 (+ 26252us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":696578,"threads_started":1,"wal-append.queue_time_us":681203}
I20260709 12:16:11.080804 12565 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:16:11.081383 12565 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:16:11.081705 12565 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:16:11.081979 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4291) took 1441 ms. Trace:
W20260709 12:16:11.069435 22585 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.618s	user 0.013s	sys 0.007s
I20260709 12:16:11.082638 12496 rpcz_store.cc:276] 0709 12:16:09.640974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.641766 (+   792us) service_pool.cc:224] Handling call
0709 12:16:09.642193 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.663796 (+ 21603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:09.663834 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:09.663883 (+    49us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:09.663919 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.081332 (+1417413us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.081422 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.081726 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49137920}
I20260709 12:16:11.082970 22585 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.085379 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.085496 22734 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.074414 12638 rpcz_store.cc:276] 0709 12:16:10.040121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.812959 (+772838us) service_pool.cc:224] Handling call
0709 12:16:10.887090 (+ 74131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.057929 (+170839us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:11.058204 (+   275us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:11.067424 (+  9220us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:11.067486 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.067578 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.067748 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:11.086136 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:11.082017 12565 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:11.082463 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.072970 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3963) took 1865 ms. Trace:
I20260709 12:16:11.085981 22606 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:16:11.106073 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 279017us
I20260709 12:16:11.105895 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:11.118728 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.129463 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:11.130124 12569 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
W20260709 12:16:11.138088 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.133713 22751 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.013s
I20260709 12:16:11.137058 22492 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:11.103688 22606 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:11.128539 12646 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:16:11.101976 22700 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.086416 22734 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.158133 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:11.158651 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.141268 22631 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:11.170841 22676 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:16:11.172907 22734 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.169258 22752 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:11.173117 22641 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:11.173861 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4086) took 1012 ms. Trace:
I20260709 12:16:11.174367 12569 rpcz_store.cc:276] 0709 12:16:10.161043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.032985 (+871942us) service_pool.cc:224] Handling call
0709 12:16:11.033464 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.138171 (+104707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.173528 (+ 35357us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.173773 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122300288}
I20260709 12:16:11.175367 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4068) took 2269 ms. Trace:
I20260709 12:16:11.176342 12569 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:11.175935 12567 rpcz_store.cc:276] 0709 12:16:08.906045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.465572 (+559527us) service_pool.cc:224] Handling call
0709 12:16:09.466622 (+  1050us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.932866 (+466244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.174668 (+1241802us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.175105 (+   437us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1070739712}
I20260709 12:16:11.177160 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4087) took 1015 ms. Trace:
I20260709 12:16:11.171416 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3236) took 1363 ms. Trace:
I20260709 12:16:11.146217 22606 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:11.190596 12569 rpcz_store.cc:276] 0709 12:16:10.161309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.175619 (+1014310us) service_pool.cc:224] Handling call
0709 12:16:11.176091 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.176893 (+   802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.177104 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:11.191645 12490 rpcz_store.cc:276] 0709 12:16:09.807664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.807828 (+   164us) service_pool.cc:224] Handling call
0709 12:16:11.170901 (+1363073us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:11.171175 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1050534144}
I20260709 12:16:11.197369 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:11.200591 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:11.148778 22502 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:11.229023 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.159308 22700 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:11.087689 12639 rpcz_store.cc:276] 0709 12:16:09.017215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:09.249347 (+232132us) service_pool.cc:224] Handling call
0709 12:16:09.263526 (+ 14179us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:09.263728 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:16:09.319549 (+ 55821us) log.cc:844] Serialized 14 byte log entry
0709 12:16:10.405680 (+1086131us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b180030b380. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:16:10.818138 (+412458us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:10.827542 (+  9404us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:10.842487 (+ 14945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.853152 (+ 10665us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:10.882481 (+ 29329us) inbound_call.cc:177] Queueing success response
0709 12:16:11.024035 (+141554us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b180030b380. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
Metrics: {"mutex_wait_us":4428,"prepare.queue_time_us":55672,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2753,"raft.queue_time_us":8259,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":2415,"spinlock_wait_cycles":37971584,"thread_start_us":355613,"threads_started":1,"wal-append.queue_time_us":1462556}
I20260709 12:16:11.249346 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:11.249650 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.250141 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:11.271446 12488 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:16:11.246603 22642 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.283691 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:11.282428 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 395612us
I20260709 12:16:11.292507 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:11.173877 22752 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.174193 22641 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.286144 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4252) took 1143 ms. Trace:
I20260709 12:16:11.252116 22585 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.303237 12570 rpcz_store.cc:276] 0709 12:16:10.114314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.876225 (+761911us) service_pool.cc:224] Handling call
0709 12:16:10.876796 (+   571us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:10.877343 (+   547us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:16:11.065745 (+188402us) log.cc:844] Serialized 14 byte log entry
0709 12:16:11.256042 (+190297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:11.256207 (+   165us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:11.256311 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.256434 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.256599 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":374487,"thread_start_us":336722,"threads_started":2}
W20260709 12:16:11.366269 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.318943 22580 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:11.347713 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:11.294092 22700 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.352921 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:11.325917 22502 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:11.410102 22515 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.235s	user 0.008s	sys 0.021s
I20260709 12:16:11.410883 22515 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:11.414299 22580 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.416888 22580 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.417820 12568 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:11.419049 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4255) took 1189 ms. Trace:
I20260709 12:16:11.419466 12568 rpcz_store.cc:276] 0709 12:16:10.229902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.323125 (+1093223us) service_pool.cc:224] Handling call
0709 12:16:11.378356 (+ 55231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.418588 (+ 40232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.418932 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:11.421957 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:11.429796 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 221602us
I20260709 12:16:11.431422 22700 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.432264 22700 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.432928 22700 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.434517 22297 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.472739 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:11.435204 22641 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:11.439746 12488 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:16:11.475442 12488 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:11.476871 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.440430 22497 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.445998 22507 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:11.447916 22752 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.451177 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 3984) took 1094 ms. Trace:
I20260709 12:16:11.454497 22635 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.455268 22792 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:11.457855 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 164931us
I20260709 12:16:11.458464 22676 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.458348 22821 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:11.339268 22642 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.461185 22657 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.504103 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.421473 12568 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:11.473764 13257 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:16:11.477738 13258 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:11.480508 12490 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:16:11.379679 22585 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:11.445386 22687 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.003s
I20260709 12:16:11.481480 12570 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:16:11.472767 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.400264 22776 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.485958 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4084) took 1325 ms. Trace:
I20260709 12:16:11.484748 12641 rpcz_store.cc:276] 0709 12:16:10.357162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.055826 (+698664us) service_pool.cc:224] Handling call
0709 12:16:11.077381 (+ 21555us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:11.078238 (+   857us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:11.078277 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:11.079002 (+   725us) log.cc:844] Serialized 26 byte log entry
0709 12:16:11.260277 (+181275us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:11.260332 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.260422 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:11.450873 (+190451us) raft_consensus.cc:1685] finished
0709 12:16:11.450900 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.451096 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":167,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":163,"thread_start_us":180733,"threads_started":1,"wal-append.queue_time_us":301960}
W20260709 12:16:11.492494 12493 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.625s	user 0.005s	sys 0.010s
I20260709 12:16:11.512888 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.484349 22502 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.481292 22192 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:11.522296 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 273066us
I20260709 12:16:11.535209 12493 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:11.519022 12570 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 11
I20260709 12:16:11.541445 12570 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:16:11.542121 12570 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:11.549861 22585 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.552935 22585 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.537778 22687 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.554721 22700 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.555895 22687 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.556613 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:11.557744 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:11.559826 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:11.482621 22497 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.562348 22642 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.562978 22635 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.564266 22497 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:11.565793 22297 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.505831 22792 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.501883 22507 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:11.568542 22657 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:11.568825 22507 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.569828 12643 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:11.570261 22831 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:11.570820 22831 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:11.571151 22831 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:11.571485 22760 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:11.572521 22530 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:11.534475 12564 rpcz_store.cc:276] 0709 12:16:10.160464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.941902 (+781438us) service_pool.cc:224] Handling call
0709 12:16:10.943259 (+  1357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:10.947991 (+  4732us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:16:11.143234 (+195243us) log.cc:844] Serialized 14 byte log entry
0709 12:16:11.485504 (+342270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:11.485577 (+    73us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:11.485663 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.485757 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.485891 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38125,"prepare.queue_time_us":385156,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":13266,"raft.queue_time_us":98446,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"thread_start_us":461062,"threads_started":3,"wal-append.queue_time_us":263669}
W20260709 12:16:11.574488 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 233253us
I20260709 12:16:11.575037 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:11.575953 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:16:11.577229 12564 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:11.565456 22635 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.579291 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.580740 12494 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:11.581691 22687 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:11.585508 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.585994 22776 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.578059 22832 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:11.589005 22832 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.590653 22192 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.591492 22832 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.593219 22832 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.615479 22838 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:11.594978 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49380.3us
I20260709 12:16:11.616509 22832 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.619170 22831 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:11.622602 22566 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.575s	user 0.013s	sys 0.000s
I20260709 12:16:11.604183 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.607625 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1418) took 1432 ms. Trace:
I20260709 12:16:11.577577 12564 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:11.608945 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:11.501511 22821 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:11.610114 22514 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:11.610349 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3242) took 1254 ms. Trace:
I20260709 12:16:11.626015 12642 rpcz_store.cc:276] 0709 12:16:10.174767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.817127 (+642360us) service_pool.cc:224] Handling call
0709 12:16:11.606682 (+789555us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:16:11.606906 (+   224us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.607251 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:11.607278 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:11.607341 (+    63us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:11.607372 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.607426 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.607532 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5591296}
I20260709 12:16:11.594151 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.595438 22700 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:11.617601 22585 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:16:11.636674 12493 rpcz_store.cc:276] 0709 12:16:10.286659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.790504 (+503845us) service_pool.cc:224] Handling call
0709 12:16:11.541200 (+750696us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:11.617827 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:11.622440 22830 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.603044 12496 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:11.635540 22514 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:11.636345 22591 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:11.644450 22591 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.647131 22591 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.647575 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.649370 22832 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.637701 22566 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.651448 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:11.652201 12639 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:11.653064 22566 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.573541 22530 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:16:11.653900 22585 consensus_queue.cc:1243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:16:11.654804 12637 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:11.658735 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:11.672482 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4303) took 1498 ms. Trace:
I20260709 12:16:11.672983 12497 rpcz_store.cc:276] 0709 12:16:10.174369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.177735 (+  3366us) service_pool.cc:224] Handling call
0709 12:16:10.178083 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:10.612079 (+433996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.672038 (+1059959us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.672390 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":889534720}
I20260709 12:16:11.674957 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4248) took 1566 ms. Trace:
I20260709 12:16:11.636556 22687 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:11.676388 22687 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:11.676888 22687 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:11.677551 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1217) took 1618 ms (client timeout 1761 ms). Trace:
W20260709 12:16:11.678042 13262 rpcz_store.cc:269] 0709 12:16:10.058938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.066025 (+  7087us) service_pool.cc:224] Handling call
0709 12:16:11.677113 (+1611088us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:11.677316 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":104749952}
I20260709 12:16:11.625773 22831 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.565186 22676 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.639142 22297 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:11.682832 22297 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.683404 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4095) took 1051 ms. Trace:
I20260709 12:16:11.683905 12566 rpcz_store.cc:276] 0709 12:16:10.632249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.361187 (+728938us) service_pool.cc:224] Handling call
0709 12:16:11.400579 (+ 39392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.467915 (+ 67336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:11.467969 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:11.468043 (+    74us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:11.468088 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.683082 (+214994us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.683152 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:11.683325 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145698176}
I20260709 12:16:11.661972 12567 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.15-29.15]
I20260709 12:16:11.686880 12566 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:11.634909 22776 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.663280 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:11.670004 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:11.675599 22758 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.520929 22734 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:11.640424 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4093) took 1277 ms. Trace:
I20260709 12:16:11.659802 22514 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.675413 12563 rpcz_store.cc:276] 0709 12:16:10.108704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.822866 (+714162us) service_pool.cc:224] Handling call
0709 12:16:10.883507 (+ 60641us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:10.983523 (+100016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.674688 (+691165us) spinlock_profiling.cc:243] Waited 71.2 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.674896 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163673472}
I20260709 12:16:11.661783 22838 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:11.705951 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.705757 12570 rpcz_store.cc:276] 0709 12:16:10.362624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.345004 (+982380us) service_pool.cc:224] Handling call
0709 12:16:11.407879 (+ 62875us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.639986 (+232107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.640332 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:11.635986 22821 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:11.688982 22297 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.635855 22833 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.665362 22635 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:16:11.707126 22830 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:11.753058 12389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 230) took 104641 ms (client timeout 15000 ms). Trace:
I20260709 12:16:11.743285 12568 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:16:11.745942 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.746693 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.757867 22713 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:11.753851 12389 rpcz_store.cc:269] 0709 12:14:27.111963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:27.112251 (+   288us) service_pool.cc:224] Handling call
0709 12:16:11.752767 (+104640516us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:16:01.850005 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:16:02.092240 (+242235us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:16:02.103311 (+ 11071us) write_op.cc:435] Acquired schema lock
0709 12:16:02.122863 (+ 19552us) tablet.cc:662] Decoding operations
0709 12:16:02.372816 (+249953us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:16:02.408328 (+ 35512us) write_op.cc:641] Partition lock acquired for write op
0709 12:16:02.412042 (+  3714us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:16:02.617884 (+205842us) tablet.cc:701] Row locks acquired
0709 12:16:02.663765 (+ 45881us) write_op.cc:260] PREPARE: finished
0709 12:16:02.951966 (+288201us) write_op.cc:270] Start()
0709 12:16:02.992632 (+ 40666us) write_op.cc:276] Timestamp: P: 1783599362922097 usec, L: 0
0709 12:16:03.016887 (+ 24255us) op_driver.cc:348] REPLICATION: starting
0709 12:16:03.293027 (+276140us) log.cc:844] Serialized 929 byte log entry
0709 12:16:04.046814 (+753787us) op_driver.cc:464] REPLICATION: finished
0709 12:16:04.317326 (+270512us) write_op.cc:301] APPLY: starting
0709 12:16:04.326645 (+  9319us) tablet.cc:1367] starting BulkCheckPresence
0709 12:16:04.329369 (+  2724us) tablet.cc:1370] finished BulkCheckPresence
0709 12:16:04.329522 (+   153us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:04.331041 (+  1519us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:04.331339 (+   298us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:16:04.331406 (+    67us) write_op.cc:312] APPLY: finished
0709 12:16:04.331732 (+   326us) log.cc:844] Serialized 33 byte log entry
0709 12:16:04.337824 (+  6092us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:04.353975 (+ 16151us) write_op.cc:454] Released schema lock
0709 12:16:04.358015 (+  4040us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:04.365087 (+  7072us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:04.367904 (+  2817us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":551118,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":237619,"apply.run_cpu_time_us":8255,"apply.run_wall_time_us":50343,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12479,"num_ops":2,"prepare.queue_time_us":130249,"prepare.run_cpu_time_us":33052,"prepare.run_wall_time_us":1574956,"raft.queue_time_us":463824,"raft.run_cpu_time_us":12488,"raft.run_wall_time_us":202231,"replication_time_us":969933,"thread_start_us":641743,"threads_started":4,"wal-append.queue_time_us":154439,"wal-append.run_cpu_time_us":23238,"wal-append.run_wall_time_us":1874145}]]}
I20260709 12:16:11.733094 22297 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:11.761821 22759 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:16:11.762847 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:11.762724 22297 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.763617 13262 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:11.765072 22838 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.765972 22802 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:16:11.743979 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.747947 22676 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.684376 22606 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.748924 13261 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:11.737537 22734 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.750219 22514 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:11.765219 22297 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.731467 22752 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:16:11.740438 22492 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.005s
I20260709 12:16:11.789705 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.790046 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:11.794708 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.795354 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:11.744827 22833 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:11.805998 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 147380us
I20260709 12:16:11.794480 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:11.806814 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:11.795406 12492 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:11.709898 22758 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.800575 22734 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:11.808820 12640 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:11.809954 22545 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:11.810427 22545 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:11.816874 22834 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:11.817814 22834 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.819281 22545 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.824863 22519 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:11.825383 22519 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:11.827167 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.820423 22834 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.819732 12564 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:11.833461 13257 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:11.838483 22519 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.840251 22519 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:11.842207 22837 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.316s	user 0.005s	sys 0.000s
I20260709 12:16:11.842334 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:11.839335 13260 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:11.845779 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:11.846043 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:11.846457 22713 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:11.848678 12496 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:16:11.852381 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 3974) took 1839 ms (client timeout 1564 ms). Trace:
W20260709 12:16:11.852953 12643 rpcz_store.cc:269] 0709 12:16:10.013400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.156947 (+143547us) service_pool.cc:224] Handling call
0709 12:16:11.847424 (+1690477us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000a6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:11.852294 (+  4870us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2786884864}
I20260709 12:16:11.849516 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:11.866461 12564 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:11.866660 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4089) took 1690 ms. Trace:
I20260709 12:16:11.868916 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:11.867347 12572 rpcz_store.cc:276] 0709 12:16:10.170238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.626484 (+456246us) service_pool.cc:224] Handling call
0709 12:16:11.860525 (+1234041us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:11.860716 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2611763072}
I20260709 12:16:11.873723 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4081) took 1714 ms. Trace:
I20260709 12:16:11.874336 22545 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:11.874225 12565 rpcz_store.cc:276] 0709 12:16:10.159648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.878451 (+718803us) service_pool.cc:224] Handling call
0709 12:16:10.897561 (+ 19110us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.090397 (+192836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.873324 (+782927us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54003ab954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.873601 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420214528}
I20260709 12:16:11.875906 12565 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:11.871438 22776 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:16:11.878834 22839 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.347s	user 0.001s	sys 0.002s
I20260709 12:16:11.879266 22833 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:11.868325 22530 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:11.882522 22530 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:11.883095 22530 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:11.884147 22821 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:11.884850 22758 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.867996 22830 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:11.869437 22831 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:11.882580 22606 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:11.852079 22507 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:11.894943 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49050.7us
W20260709 12:16:11.897004 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 251390us
W20260709 12:16:11.898854 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69928.2us
W20260709 12:16:11.911794 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:11.912688 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:11.894958 22831 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:11.956739 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:11.958454 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.946040 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:11.962522 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:11.964753 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3287) took 1805 ms. Trace:
I20260709 12:16:11.970064 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:11.968006 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:11.975338 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4253) took 1746 ms. Trace:
I20260709 12:16:12.007002 12567 rpcz_store.cc:276] 0709 12:16:10.229373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.197744 (+968371us) service_pool.cc:224] Handling call
0709 12:16:11.252254 (+ 54510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:11.686037 (+433783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:11.966719 (+280682us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:11.975258 (+  8539us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899613952}
I20260709 12:16:12.008996 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:11.968042 12640 rpcz_store.cc:276] 0709 12:16:10.159254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.662761 (+1503507us) service_pool.cc:224] Handling call
0709 12:16:11.964524 (+301763us) spinlock_profiling.cc:243] Waited 97 ms on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:11.964683 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223150336}
I20260709 12:16:12.014015 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:12.015378 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:11.962364 22492 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:12.021888 22232 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.059s
I20260709 12:16:12.022644 22823 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:12.023371 22823 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.027925 22792 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.028678 22792 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:12.029094 22792 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:12.029477 22792 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:12.033680 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4254) took 1804 ms. Trace:
I20260709 12:16:12.034644 12387 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 23 to 26, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 26 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.036043 22192 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.034189 12569 rpcz_store.cc:276] 0709 12:16:10.229676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.204938 (+975262us) service_pool.cc:224] Handling call
0709 12:16:11.287317 (+ 82379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.380494 (+ 93177us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:16:11.463448 (+ 82954us) log.cc:844] Serialized 14 byte log entry
0709 12:16:11.727113 (+263665us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:11.727181 (+    68us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:11.727238 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:12.033386 (+306148us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:12.033442 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:12.033590 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":86913,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":132,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":1645,"spinlock_wait_cycles":124724480,"thread_start_us":263148,"threads_started":1,"wal-append.queue_time_us":320555}
I20260709 12:16:11.961985 22507 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.037726 22834 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:16:12.039645 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75109.1us
I20260709 12:16:12.039354 22507 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.040943 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:12.042098 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:12.044097 22642 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:12.044802 22642 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:12.045141 22642 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:12.047322 22811 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.938s	user 0.004s	sys 0.000s
I20260709 12:16:12.045466 22642 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:11.912045 22758 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.141s
I20260709 12:16:12.049391 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.051116 12490 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:16:12.036584 12387 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 25 to 28, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 28 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:12.049460 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4304) took 1874 ms. Trace:
I20260709 12:16:12.058591 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:12.060290 13257 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:12.058518 13262 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:12.063330 13260 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:11.906011 22497 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:12.062711 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:12.065689 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:12.071097 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:12.072047 12566 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
W20260709 12:16:12.074059 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 176300us
W20260709 12:16:12.075491 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 229984us
I20260709 12:16:11.959971 22831 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:12.051558 22838 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:12.081247 12564 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:16:12.079515 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:12.086597 12564 pending_rounds.cc:85] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 16
I20260709 12:16:12.087036 12564 pending_rounds.cc:107] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:16:12.058350 12491 rpcz_store.cc:276] 0709 12:16:10.174720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:10.224414 (+ 49694us) service_pool.cc:224] Handling call
0709 12:16:10.242395 (+ 17981us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:10.242756 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:10.242786 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:10.258518 (+ 15732us) log.cc:844] Serialized 26 byte log entry
0709 12:16:10.834853 (+576335us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b1800448ba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:16:10.925846 (+ 90993us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:16:10.926000 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:10.926149 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:12.046416 (+1120267us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:12.049051 (+  2635us) raft_consensus.cc:1685] finished
0709 12:16:12.049074 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:12.049269 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75073,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":26511,"spinlock_wait_cycles":37954688,"thread_start_us":18645,"threads_started":1,"wal-append.queue_time_us":691223}
I20260709 12:16:12.087790 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:12.088414 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 128487us
I20260709 12:16:12.051304 22758 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:11.910238 22657 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.158s
I20260709 12:16:12.094331 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:12.088466 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:16:12.071134 22515 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:12.039858 22641 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:16:12.087348 12564 raft_consensus.cc:2889] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:11.969887 22606 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.114s
I20260709 12:16:12.094870 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:12.096334 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:12.115653 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4100) took 1266 ms. Trace:
I20260709 12:16:12.084990 22734 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:12.086151 22838 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:12.116413 12564 rpcz_store.cc:276] 0709 12:16:10.854201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.576651 (+722450us) service_pool.cc:224] Handling call
0709 12:16:11.577000 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:11.819123 (+242123us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b54003ef920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577106f 00007ffbf57e9f60 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62
0709 12:16:12.115154 (+296031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:12.115481 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64964864}
I20260709 12:16:12.021919 22492 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.097361 22758 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:12.126814 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:12.129604 12572 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:12.118172 12387 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 12 to 24. New cstate: current_term: 24 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:12.135560 12567 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:12.138742 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:12.140543 13256 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:12.105083 22759 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
W20260709 12:16:12.120929 22585 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 27 current term: 27 is higher.
I20260709 12:16:12.115300 22776 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:12.145917 22680 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:16:12.147450 22843 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:16:12.152456 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.154384 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:12.154682 22838 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.157660 22687 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.128875 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:12.167128 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:12.165485 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
W20260709 12:16:12.166646 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75492.7us
I20260709 12:16:12.134585 22823 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.193063 12568 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:12.194605 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:12.207026 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52836.4us
I20260709 12:16:12.185952 22680 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:12.183698 22641 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:12.218394 22845 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:12.219118 12569 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:16:12.225960 22845 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:12.226428 22580 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.227166 22838 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:12.227182 22580 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.229768 22580 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.229948 22823 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.230747 22823 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.219070 22821 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:12.226097 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104248us
W20260709 12:16:12.243866 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 201810us
W20260709 12:16:12.256700 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.211992 22680 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:12.213918 22635 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:12.216260 22519 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:12.268041 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:12.213004 22641 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:12.275856 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81388.5us
I20260709 12:16:12.203830 22776 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.263630 22497 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:12.242413 22507 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:12.278442 22734 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:12.323711 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56813us
I20260709 12:16:12.294072 22758 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.291496 22676 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
W20260709 12:16:12.331570 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.296298 22515 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:12.227519 22845 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.342054 22641 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.341800 22734 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:12.342955 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 486001us
I20260709 12:16:12.322975 22853 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:12.346042 22856 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:16:12.334350 22752 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.012s	sys 0.002s
I20260709 12:16:12.326973 22852 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.351176 22492 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:12.334942 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55246.4us
I20260709 12:16:12.346071 12387 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 11 to 20. New cstate: current_term: 20 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.317098 22792 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:12.368203 22847 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:16:12.370090 22515 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.370754 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:12.371518 12570 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:12.374601 22852 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.375787 22641 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:12.369333 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:12.394834 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:12.403198 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:12.383596 22855 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:12.381147 22848 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:16:12.425029 13259 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:12.427580 22792 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:12.378891 12387 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:16:12.391396 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53290.4us
W20260709 12:16:12.459540 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:12.480777 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111576us
I20260709 12:16:12.360262 22823 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:12.393951 22713 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.536s	user 0.001s	sys 0.011s
W20260709 12:16:12.391500 22514 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.598s	user 0.012s	sys 0.000s
I20260709 12:16:12.363852 22752 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:16:12.483346 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 154966us
I20260709 12:16:12.392582 22502 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.485170 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:12.391870 22687 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:12.495865 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:12.377467 22852 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:12.491746 12563 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:12.489885 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:12.498755 13259 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:12.494691 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:12.484665 22585 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:12.390067 22497 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.514989 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:12.516052 22507 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:12.516732 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:12.517542 22507 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:12.517961 22507 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:12.518396 22507 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:12.519546 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 277488us
I20260709 12:16:12.521503 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:12.523936 12387 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.525610 12387 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 12 to 23, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 23 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.527150 12387 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 5 to 20, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 20 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.527803 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1865) took 1353 ms. Trace:
I20260709 12:16:12.508734 22758 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:12.488139 22492 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.503109 22680 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.489720 22713 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:12.529587 12387 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 19 to 23, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:16:12.532521 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51035.2us
I20260709 12:16:12.528223 12563 rpcz_store.cc:276] 0709 12:16:11.174024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.707676 (+533652us) service_pool.cc:224] Handling call
0709 12:16:11.708148 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:12.498027 (+789879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:12.527520 (+ 29493us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:12.527738 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1797388160}
W20260709 12:16:12.530014 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.540788 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:12.541725 12643 raft_consensus.cc:2368] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:16:12.542135 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.542832 12644 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:12.496897 22502 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.544440 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.544958 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.545212 22514 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:12.545722 22514 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:12.546080 22514 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:12.546648 12494 raft_consensus.cc:1217] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Deduplicated request from leader. Original: 26.15->[27.16-27.16]   Dedup: 27.16->[]
I20260709 12:16:12.546964 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:12.547260 12494 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:16:12.549315 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.549969 12497 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:12.550858 22492 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.550820 22856 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:12.552050 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.552345 22856 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.551705 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.554304 12491 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:12.554885 22856 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:12.554317 22680 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.534868 22585 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:12.555964 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:12.556304 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.556640 22585 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.550781 22857 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:12.557775 22857 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.559964 22857 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:12.563594 22620 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:16:12.528690 22507 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:12.563980 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.566464 22657 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:12.566605 22519 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:12.566962 22657 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:12.557241 12497 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:12.567250 22519 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.567802 22823 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:12.509418 22530 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.626s	user 0.008s	sys 0.001s
I20260709 12:16:12.573911 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.568655 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:12.541657 22642 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.587324 22776 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.591425 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.593226 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:12.592401 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:12.598430 12493 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:12.575552 22752 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.545014 22502 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:12.614706 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.613065 12563 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:12.613484 13256 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:12.614754 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.578133 22821 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.618266 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:12.616513 12638 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:12.620812 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.621546 12644 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:12.612310 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:12.623369 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.626284 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:12.627806 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.627996 12570 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 28 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:16:12.619458 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.630470 12640 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:12.618816 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.618180 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.632897 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.631242 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:16:12.635959 12491 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:12.575721 22823 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.636937 22752 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.638819 22823 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.639057 22752 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.633726 22821 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.641747 22863 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:12.642014 12564 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:16:12.636943 22833 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:16:12.644374 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58938.4us
I20260709 12:16:12.644901 22514 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.645442 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:12.646221 22657 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.654946 22642 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:12.655558 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.612466 22687 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.567137 22760 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.050s
I20260709 12:16:12.627339 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.680553 12489 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:12.650610 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:12.653455 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 132032us
W20260709 12:16:12.653461 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84889.8us
W20260709 12:16:12.646744 22831 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.541s	user 0.003s	sys 0.009s
I20260709 12:16:12.625699 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:12.655999 12387 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 20 to 21, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 21 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:12.662356 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.695958 12497 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:12.668128 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.579161 22530 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.678156 12643 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 25. Current term is 27. Ops: []
I20260709 12:16:12.678567 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.696697 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.693470 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.707206 12491 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:12.704496 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:12.708329 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:12.715896 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:12.712641 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:12.720448 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.674860 22861 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.727994 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:12.739532 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.704687 22507 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.711431 22845 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:12.739959 12387 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.698338 22192 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:16:12.725687 22657 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:12.730043 22680 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:12.725581 22821 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:12.755097 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46515.6us
I20260709 12:16:12.705098 22831 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.730652 22752 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:12.714982 22687 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.768801 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:12.765039 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.771684 12387 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 19 to 20, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 20 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:16:12.777901 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49992.5us
I20260709 12:16:12.777918 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:12.755194 22833 raft_consensus.cc:2764] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:16:12.757149 22585 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.784961 12641 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:12.786065 12387 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 18 to 21, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 21 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:12.796906 12387 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.770043 22676 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:12.770781 22845 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:12.803079 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.774631 22680 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:12.781152 22519 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.754972 22861 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:12.808427 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:12.792395 22823 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.826428 12387 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 20 to 21, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 21 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:12.770743 22752 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.801013 22642 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:16:12.845813 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45207.7us
W20260709 12:16:12.846881 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56952.4us
I20260709 12:16:12.850020 12387 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 21 to 24, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 24 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.851498 12387 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 11 to 23, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 23 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:12.856104 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:12.794421 22232 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.044s
W20260709 12:16:12.864640 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56899.8us
I20260709 12:16:12.866428 12536 leader_election.cc:400] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:16:12.866801 12536 leader_election.cc:403] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:16:12.867060 12387 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 20 to 22, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 22 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:12.867811 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:12.838830 22642 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:12.852602 22864 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:16:12.808118 22585 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:12.970939 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105397us
I20260709 12:16:12.978606 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:12.975219 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119209us
I20260709 12:16:12.859081 22758 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:12.821884 22676 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:12.860574 22514 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:12.985826 22776 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:12.841200 22823 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:12.966343 22606 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:12.972155 22642 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:12.987394 22869 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 12:16:13.016889 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:12.980460 22519 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:16:13.021893 22881 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.022661 22881 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.024946 22881 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:13.026695 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 203472us
W20260709 12:16:13.043247 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:13.047617 22872 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
W20260709 12:16:13.056545 22870 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
W20260709 12:16:13.059178 22873 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:16:13.062361 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.000797 22530 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:13.082535 22867 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 12:16:13.083213 22833 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:13.006062 22776 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:13.086208 22776 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:13.086547 22752 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.087643 22831 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.089778 22585 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.004228 22845 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.093581 22492 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.095546 22641 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:13.096340 22641 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:13.073669 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1459) took 1023 ms. Trace:
I20260709 12:16:13.096243 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:13.112233 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95504.2us
W20260709 12:16:13.073028 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87575.8us
I20260709 12:16:13.097590 22641 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.128717 12646 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
W20260709 12:16:13.085510 22874 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
W20260709 12:16:13.140188 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.103652 12496 rpcz_store.cc:276] 0709 12:16:12.050172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:12.064686 (+ 14514us) service_pool.cc:224] Handling call
0709 12:16:12.068331 (+  3645us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:12.088339 (+ 20008us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:12.088375 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:12.430395 (+342020us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b180037c4e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:16:12.552774 (+122379us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:12.567963 (+ 15189us) log.cc:844] Serialized 26 byte log entry
0709 12:16:12.748472 (+180509us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:12.748535 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:12.748632 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:13.060914 (+312282us) raft_consensus.cc:1685] finished
0709 12:16:13.060956 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:13.073541 (+ 12585us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88407,"raft.queue_time_us":19939,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":7546,"spinlock_wait_cycles":23042048,"thread_start_us":180020,"threads_started":1,"wal-append.queue_time_us":232324}
I20260709 12:16:13.005180 22758 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:13.156589 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.132323 22676 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:13.164400 22868 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.335s	user 0.000s	sys 0.001s
I20260709 12:16:13.000443 22680 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:13.174799 12490 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.515s	user 0.009s	sys 0.001s
I20260709 12:16:13.175153 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:13.186331 13256 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 35 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:13.191972 22883 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:13.192574 22883 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.193197 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.162938 22687 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.196489 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:13.197387 12494 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:13.199939 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:13.202129 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.203518 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.195199 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:13.204310 22734 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:13.204758 22734 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.205084 22734 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:13.206944 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:13.207441 22515 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:13.207657 13259 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:13.208025 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.152570 22606 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:16:13.208083 22515 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:13.211608 22515 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.209491 22502 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.214218 22502 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.215209 12639 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:16:13.217144 22502 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.218866 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.219698 12637 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:13.223037 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80390.6us
I20260709 12:16:13.223940 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.228968 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.230270 12491 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
W20260709 12:16:13.234665 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 203178us
I20260709 12:16:13.236804 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.237668 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:13.239392 13259 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:13.243848 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:13.248751 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.248937 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.244674 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.252230 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:13.257774 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.251530 12640 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:13.261096 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.258806 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.263432 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.240909 22831 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:13.270597 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.297001 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.270597 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.304843 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:13.306665 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44770.7us
I20260709 12:16:13.306133 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.308696 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.271612 22502 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:13.312304 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.266881 22861 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.315419 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.315622 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.317541 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.343835 12641 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:13.317878 12640 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:13.264420 22687 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:13.321355 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.324905 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.345707 12639 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:13.326673 12646 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:13.301506 22606 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.327658 12637 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:13.348163 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.349535 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.312640 22514 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:13.302034 22886 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:13.351431 12496 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:13.353327 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.271066 22492 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.319060 22585 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:13.368530 12494 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:13.341276 22752 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:13.383010 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.383994 12646 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:13.354317 22821 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:13.388350 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:13.362493 22657 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.599s	user 0.005s	sys 0.010s
I20260709 12:16:13.353428 22502 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.355830 22514 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.392148 12639 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:16:13.393446 12489 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:16:13.393541 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:13.401947 12640 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:13.394217 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.356122 22759 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:13.374344 22861 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:13.341557 22863 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:16:13.356215 22891 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:13.374353 22585 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.373950 22591 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:16:13.416796 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.457399 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:13.459127 12496 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:13.355926 22606 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:13.459697 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110353us
I20260709 12:16:13.399019 22497 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.406667 22514 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:13.389454 22492 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.479880 22492 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.482044 22606 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.490142 22883 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.450204 22891 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.491696 22891 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:13.463801 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.458945 22585 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:13.467762 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:13.451440 22861 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.469997 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.470207 22863 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:16:13.474699 22887 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:16:13.475026 22649 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:13.476730 22889 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:16:13.498678 12646 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:13.503305 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143658us
I20260709 12:16:13.400089 22502 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:13.496776 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.395985 22875 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.100s
I20260709 12:16:13.508805 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.501039 22861 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:13.520407 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.522603 12488 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:13.524610 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.524775 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:13.502065 22492 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:13.537206 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.534256 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:13.528513 12489 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:13.531474 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.501250 22863 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:13.533736 22902 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:13.542995 22863 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.544632 22863 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:13.550678 22900 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:13.503803 22514 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.556784 22514 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.500010 22497 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.558667 22896 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:16:13.558912 22515 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:13.559659 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.560164 22497 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.560843 22497 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:13.560740 22515 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:13.563413 22642 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.535569 22492 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.566282 22492 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:13.561350 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.539155 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.574280 12639 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:13.574887 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.500739 22776 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.579523 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.579530 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.589747 12496 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:13.595301 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.518066 22649 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.576668 22514 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:13.591598 22497 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:13.623979 22497 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.624439 22497 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:13.629654 22734 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.631978 22734 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:13.633864 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 319280us
I20260709 12:16:13.583029 22900 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:13.582525 22875 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:13.616605 22519 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.604662 22902 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:13.619043 22687 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.592969 22515 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:16:13.651067 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:13.643462 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48219.8us
I20260709 12:16:13.598999 12646 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:13.606679 22642 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.659228 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.661250 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.673779 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.674747 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:13.675243 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.676524 22885 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:16:13.651600 22519 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.678666 22752 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:16:13.678947 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.679687 22519 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.646880 22857 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:13.681667 22759 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.658389 22515 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.684180 22515 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:13.682407 22759 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.688084 22759 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.659602 22687 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:13.689807 22497 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.690994 12641 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:13.692840 22497 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.673998 22886 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:13.695164 22886 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.695703 22886 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:13.591758 22606 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:16:13.650725 22514 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:13.700134 22514 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.700802 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.701095 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.701748 12564 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:13.702215 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.702920 12572 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:16:13.702929 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56375.4us
I20260709 12:16:13.705542 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.649664 22902 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.711683 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.712636 12572 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:13.720304 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.728875 12564 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:16:13.727015 22776 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.735872 22585 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.737668 22863 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.738339 22863 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.739440 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.740106 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.743593 22606 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:13.736333 22649 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.744397 12570 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:13.733644 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.757902 12565 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:16:13.761373 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.761778 12572 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:13.762281 12568 raft_consensus.cc:2368] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:13.714197 22902 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:13.767439 12644 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:16:13.768937 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.769369 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.770486 12638 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:13.772738 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.773669 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:13.779975 22492 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.781358 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:13.790282 22585 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:13.791982 22908 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:16:13.792939 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:13.794507 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.798391 22891 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.800346 22823 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.800987 22891 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.744246 22903 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.812265 12565 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:16:13.777014 22515 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.819448 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:13.822760 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.822870 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.793448 22821 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:13.821030 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.826732 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:13.807647 22863 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.764861 22896 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:16:13.823910 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 162784us
I20260709 12:16:13.823992 12565 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:13.823964 12568 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:13.824705 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.830952 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.827910 13256 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:16:13.829322 12644 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:13.757187 22606 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:13.835393 13262 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:13.817749 22903 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.852761 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.855963 12646 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:16:13.840648 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:13.843750 22514 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:13.843317 22606 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:13.843458 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.846264 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:13.858783 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:13.859328 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:13.847054 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.849002 22875 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:13.838416 22861 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.857376 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.855747 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.870268 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.860273 12568 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:13.847503 22502 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.860456 12572 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:13.868090 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.867841 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.877326 12637 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:13.873402 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.875280 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.878324 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.879106 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.879777 12568 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:13.882089 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:13.882833 13258 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:13.878592 22845 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:16:13.861546 22885 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:13.888103 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:13.886533 12461 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:13.890276 12568 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:13.889436 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.867462 22821 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:16:13.892825 13260 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:16:13.887282 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.890895 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:13.894404 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.900456 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:13.879987 22514 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:13.910087 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.881508 22896 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:13.909847 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.906942 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:13.884828 22680 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:16:13.906281 22530 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:13.906002 22641 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:16:13.892393 22884 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.021s
I20260709 12:16:13.895066 22863 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.022s
I20260709 12:16:13.885201 22635 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:13.917387 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.918241 12460 leader_election.cc:400] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:13.910442 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:13.892421 22734 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:13.926554 22492 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:13.927724 22515 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.918773 22833 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:13.930848 22909 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:13.892513 22821 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:13.934191 22823 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.936115 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.935315 22891 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.933809 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:16:13.937122 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46325.4us
I20260709 12:16:13.937552 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:13.950738 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:13.955104 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48232.1us
I20260709 12:16:13.894910 22838 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.044s
I20260709 12:16:13.920794 22875 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:13.952239 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.921254 22857 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:16:13.964046 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.981562 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:13.955921 22514 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:16:13.967872 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:13.893599 22885 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:13.969985 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.947481 22734 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:13.984465 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74093us
I20260709 12:16:13.939169 22635 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.984709 12572 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:13.955559 22915 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:13.975380 22680 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:13.988363 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.958946 22833 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:13.986406 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:13.991302 12568 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:13.948561 22649 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:16:13.982964 22585 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:13.973342 22821 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:13.990101 22514 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:13.995970 22641 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:13.979327 22891 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:14.004415 22910 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260709 12:16:14.005426 22907 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:16:14.006601 22903 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:13.993567 22635 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:13.987670 22591 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:13.977078 22909 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:14.010370 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.010622 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:13.980170 22875 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:14.012764 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:14.013435 22911 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.068s	user 0.000s	sys 0.003s
I20260709 12:16:14.016456 22752 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:13.989243 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.988124 22885 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:13.991668 22776 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:14.022274 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:13.992563 22734 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:13.997442 22915 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:14.017705 22680 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.023718 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.031859 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:14.002718 22833 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:13.983723 22823 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:13.991446 22687 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.033830 22514 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:14.043685 12641 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:14.128151 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148862us
I20260709 12:16:14.042279 22530 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:14.112884 22863 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:14.141976 22886 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.125810 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.041074 22591 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:14.243892 22591 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:14.198433 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.147588 22861 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.114801 22891 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.112871 22875 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:14.118289 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.040088 22641 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:14.118516 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:14.215901 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 184125us
W20260709 12:16:14.206489 22917 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
W20260709 12:16:14.220443 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3269) took 1494 ms (client timeout 1562 ms). Trace:
I20260709 12:16:14.222514 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:14.228390 22680 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.228572 22687 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.231650 22885 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.188457 22909 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.233465 22635 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:14.239148 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:14.239578 22776 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:14.240999 12569 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:14.289450 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.290135 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:14.241995 22902 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.242238 22863 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.245545 12641 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:14.263712 12564 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.265869 12563 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.291723 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.266471 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.254586 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.292850 12568 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:14.293888 22925 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:14.294281 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71892.9us
I20260709 12:16:14.294615 22924 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:14.294803 22925 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.295262 22924 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:14.267935 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:14.296345 22924 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.297003 22925 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:14.214082 22915 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:14.269001 12497 rpcz_store.cc:269] 0709 12:16:12.726637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:12.762538 (+ 35901us) service_pool.cc:224] Handling call
0709 12:16:14.220178 (+1457640us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:14.220367 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1446072704}
I20260709 12:16:14.293140 12492 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:14.267009 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:14.219362 22752 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:14.299017 22752 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:14.299437 22752 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:14.299777 22752 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:14.299047 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:14.272608 22918 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:14.275414 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.304513 12489 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:14.246570 22759 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:14.307549 22759 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:14.308843 22759 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.310499 22759 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:14.311187 22886 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.248107 22530 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.317950 22918 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.320798 22918 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:14.322714 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:14.323144 22821 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.323673 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:14.304682 22776 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.327270 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:14.251271 22896 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:14.267786 22893 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:16:14.280692 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:14.287756 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.292318 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.293982 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.330785 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:14.330883 12641 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:14.239830 22823 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.267220 12643 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:14.306965 22635 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.331107 12568 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:14.302089 12569 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:14.332788 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52356.2us
I20260709 12:16:14.332914 12567 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.334383 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:14.339357 12489 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:14.341813 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.345844 22641 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:14.347391 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59725.6us
I20260709 12:16:14.348325 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1884) took 1242 ms. Trace:
I20260709 12:16:14.340413 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:14.350351 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:14.348898 12566 rpcz_store.cc:276] 0709 12:16:13.105454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:13.105730 (+   276us) service_pool.cc:224] Handling call
0709 12:16:13.118334 (+ 12604us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:13.118925 (+   591us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:16:13.119397 (+   472us) log.cc:844] Serialized 14 byte log entry
0709 12:16:13.794168 (+674771us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18003fb4a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:14.186023 (+391855us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b18003fb4a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:16:14.189799 (+  3776us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.199058 (+  9259us) log.cc:844] Serialized 26 byte log entry
0709 12:16:14.199309 (+   251us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:14.199638 (+   329us) log.cc:844] Serialized 14 byte log entry
0709 12:16:14.207712 (+  8074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:14.207848 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:14.348026 (+140178us) raft_consensus.cc:1685] finished
0709 12:16:14.348053 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:14.348248 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7625,"prepare.queue_time_us":8192,"prepare.run_cpu_time_us":182,"prepare.run_wall_time_us":3501,"raft.queue_time_us":34524,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":26940,"spinlock_wait_cycles":40144512,"thread_start_us":335254,"threads_started":2,"wal-append.queue_time_us":275992}
I20260709 12:16:14.353778 12492 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:14.366436 22823 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:14.367245 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:14.335632 22886 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:14.335110 22585 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:14.376559 12641 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:14.336350 22821 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.376766 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.336908 22918 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:14.340838 22920 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:14.380551 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.398193 12643 raft_consensus.cc:2368] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:14.367350 22492 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:14.399039 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.386869 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.381716 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:14.378008 22925 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:14.393738 22886 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:14.401592 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106143us
I20260709 12:16:14.395202 22902 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.395601 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:14.369782 22896 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:14.403744 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.393946 22776 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:14.404009 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
W20260709 12:16:14.396524 22897 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.780s	user 0.000s	sys 0.005s
I20260709 12:16:14.404167 22896 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:14.407923 22492 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:14.408362 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.409343 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.402137 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:14.410074 12566 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:14.413565 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.414302 12641 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:14.412705 22924 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:14.415396 22492 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.14, Last appended by leader: 14, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.412864 22886 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.400077 22734 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.418403 22886 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.418991 12637 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:14.407333 22896 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.409165 22232 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:14.419978 22232 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:14.420895 22232 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.424469 22502 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.416021 22924 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:14.425901 22924 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:14.428726 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4335) took 1237 ms. Trace:
I20260709 12:16:14.429128 12495 rpcz_store.cc:276] 0709 12:16:13.191556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:13.221784 (+ 30228us) service_pool.cc:224] Handling call
0709 12:16:13.222167 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:13.222548 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:13.222574 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:13.231850 (+  9276us) log.cc:844] Serialized 26 byte log entry
0709 12:16:13.454082 (+222232us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:13.454154 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:13.454270 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:14.395645 (+941375us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:14.409770 (+ 14125us) raft_consensus.cc:1685] finished
0709 12:16:14.409810 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:14.428652 (+ 18842us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":365439,"raft.queue_time_us":10123,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":14322,"spinlock_wait_cycles":40095488,"thread_start_us":221110,"threads_started":1,"wal-append.queue_time_us":309178}
W20260709 12:16:14.432889 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:14.438241 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.445415 22863 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.446625 22915 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.440747 12568 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:16:14.448908 22915 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:14.397164 22514 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.373853 22591 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.470223 22514 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.466421 22920 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:16:14.471596 22591 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.473042 22928 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:14.473497 22928 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:14.459393 22758 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.461880 22925 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:14.465891 22635 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:14.471863 22885 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:14.465090 22905 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.847s	user 0.003s	sys 0.000s
W20260709 12:16:14.490720 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87165.5us
I20260709 12:16:14.493664 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:14.496412 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.500898 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.501497 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:14.505614 22885 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:14.502172 12641 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:14.504577 12566 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:14.517311 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:14.517715 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.518967 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.515136 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:14.523350 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.498104 22902 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:14.535480 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123634us
I20260709 12:16:14.557723 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.563640 12460 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:14.564122 12460 leader_election.cc:403] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:16:14.566452 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.567446 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.571296 22931 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:14.572427 22931 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:14.572984 22931 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:14.474592 22928 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.16, Last appended by leader: 16, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.574815 22928 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:14.521360 22635 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:14.580380 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.532341 22885 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:14.587611 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.590365 12641 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.582592 22891 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:14.563414 22883 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:14.596594 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.597724 12643 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:14.600732 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:14.583590 22635 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:14.582952 22929 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:14.591198 22514 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:14.587124 22885 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:14.636852 22885 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:14.640285 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.598425 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.565909 22902 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.607755 22891 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.606736 22883 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:14.608210 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44731.5us
I20260709 12:16:14.606334 22635 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:14.612299 22752 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.621083 12570 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:14.626845 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4014) took 1093 ms. Trace:
I20260709 12:16:14.630424 22514 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.591076 22933 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:14.641407 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.641597 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.689567 12572 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.634145 22776 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:14.645203 22925 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:14.650848 22924 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.636269 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.700982 12567 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.634346 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.690301 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:14.701804 12645 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:14.701941 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:14.694149 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.703037 12564 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:16:14.646842 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.703763 12564 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:16:14.704099 12564 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:14.704200 12641 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:14.648833 22892 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:14.632440 22734 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:14.652050 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:14.651696 13258 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:16:14.707669 13254 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:14.708320 12641 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:14.708995 13257 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:14.709512 12641 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.21-34.21]
I20260709 12:16:14.695883 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:14.664885 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.699499 22776 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.705861 22892 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:14.706723 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.711835 12565 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:14.676424 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:14.694766 12497 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:16:14.711493 12494 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:14.698066 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.712837 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.676800 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:14.679659 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.682596 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.685191 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:14.685938 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.653401 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.690955 22833 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.645916 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:14.717532 12641 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:16:14.642654 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.822505 12569 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.672204 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:14.659407 12638 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:16:14.664582 12637 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:14.821887 12571 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:14.824819 12570 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:14.681344 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.671679 12639 rpcz_store.cc:276] 0709 12:16:13.533709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:13.533958 (+   249us) service_pool.cc:224] Handling call
0709 12:16:14.626566 (+1092608us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:14.626769 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76485888}
I20260709 12:16:14.822041 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:14.757292 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:14.821220 12563 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
W20260709 12:16:14.845121 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:14.717689 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:14.852802 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 135239us
I20260709 12:16:14.855294 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:14.855487 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3323) took 1017 ms. Trace:
I20260709 12:16:14.856084 12638 rpcz_store.cc:276] 0709 12:16:13.843596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.658672 (+815076us) service_pool.cc:224] Handling call
0709 12:16:14.659027 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:14.825175 (+166148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:14.855409 (+ 30234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:14.856787 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:14.856775 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:14.766993 12488 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:14.858872 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 175807us
I20260709 12:16:14.864053 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:14.864430 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183156us
I20260709 12:16:14.869269 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:14.671228 22752 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.706780 22924 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.875105 12569 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
W20260709 12:16:14.875806 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 139938us
I20260709 12:16:14.673199 22514 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:14.660701 22891 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:14.676548 22507 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:14.657425 22845 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:14.657241 22902 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:14.877921 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179968us
I20260709 12:16:14.852192 22925 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:14.884545 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4264) took 1034 ms. Trace:
W20260709 12:16:14.885551 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 128412us
I20260709 12:16:14.653134 22875 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.912387 12569 rpcz_store.cc:276] 0709 12:16:13.850460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.852163 (+1001703us) service_pool.cc:224] Handling call
0709 12:16:14.870627 (+ 18464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:14.884241 (+ 13614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:14.884483 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:14.886300 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.897274 13254 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:14.865176 22821 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:14.923663 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:14.924482 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4129) took 1098 ms. Trace:
W20260709 12:16:14.900512 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78594.7us
I20260709 12:16:14.903138 12490 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:14.903592 12461 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:14.926643 12461 leader_election.cc:403] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:16:14.906606 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:14.929126 12491 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:14.650997 22931 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.903244 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4136) took 1045 ms. Trace:
I20260709 12:16:14.898654 22492 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:14.677063 22635 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.684254 22885 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.939834 12565 rpcz_store.cc:276] 0709 12:16:13.860770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.881998 (+1021228us) service_pool.cc:224] Handling call
0709 12:16:14.902709 (+ 20711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:14.902903 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.902928 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:14.902976 (+    48us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:14.903007 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:14.903070 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:14.903181 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:14.890625 22232 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:14.925009 12497 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:14.910553 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:14.940817 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 428182us
I20260709 12:16:14.947157 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:14.924988 12568 rpcz_store.cc:276] 0709 12:16:13.826145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.449212 (+623067us) service_pool.cc:224] Handling call
0709 12:16:14.449528 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.486417 (+ 36889us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.486477 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.492758 (+  6281us) log.cc:844] Serialized 26 byte log entry
0709 12:16:14.592720 (+ 99962us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:14.611867 (+ 19147us) log.cc:844] Serialized 14 byte log entry
0709 12:16:14.612222 (+   355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:14.612344 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:14.924188 (+311844us) raft_consensus.cc:1685] finished
0709 12:16:14.924213 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:14.924410 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18841,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":2805,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":4651,"thread_start_us":93307,"threads_started":1,"wal-append.queue_time_us":375599}
I20260709 12:16:14.662617 22883 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.891014 22937 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:14.901289 22922 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:14.930603 22821 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:14.665676 22929 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:14.971578 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59680.4us
I20260709 12:16:14.680891 22933 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.975157 12536 leader_election.cc:400] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:14.923749 22502 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:14.970506 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4265) took 1110 ms. Trace:
I20260709 12:16:15.027335 12569 rpcz_store.cc:276] 0709 12:16:13.860450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.913537 (+1053087us) service_pool.cc:224] Handling call
0709 12:16:14.914156 (+   619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:14.930560 (+ 16404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.930596 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:14.948035 (+ 17439us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:14.970084 (+ 22049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:14.970199 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:14.970405 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:15.018574 22821 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:14.896610 22845 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:14.917143 22530 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:14.935554 12570 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:15.010443 12493 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:15.011116 12491 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:14.971298 13262 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
W20260709 12:16:15.011207 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100813us
I20260709 12:16:14.940021 22545 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:16:15.021143 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49086.7us
I20260709 12:16:14.904058 22507 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:14.897061 22863 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.079s
I20260709 12:16:15.022954 12536 leader_election.cc:403] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
W20260709 12:16:15.052753 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77819.3us
I20260709 12:16:15.053856 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:15.055495 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:15.023351 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132627us
I20260709 12:16:15.025252 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4267) took 1155 ms. Trace:
I20260709 12:16:14.971182 22232 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:15.026710 12488 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:16:14.971709 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45319.1us
I20260709 12:16:15.059856 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:15.060055 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:15.029284 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.037169 12570 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:15.013199 22902 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:14.930069 22925 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:14.951675 22838 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:16:14.899267 22892 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.056453 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1896) took 1146 ms. Trace:
I20260709 12:16:15.056068 12565 rpcz_store.cc:276] 0709 12:16:13.869468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.983984 (+1114516us) service_pool.cc:224] Handling call
0709 12:16:15.024437 (+ 40453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:15.024727 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:15.024769 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:15.024836 (+    67us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:15.024886 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.024964 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.025135 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:15.055670 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:15.022122 22896 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:14.969509 22924 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.063s
I20260709 12:16:15.109598 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.110618 12491 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:15.071429 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:15.074950 12569 rpcz_store.cc:276] 0709 12:16:13.886006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.028543 (+1142537us) service_pool.cc:224] Handling call
0709 12:16:15.031198 (+  2655us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:15.031418 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:15.031445 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:15.031495 (+    50us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:15.031527 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.031592 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.032212 (+   620us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:16:15.069365 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 131776us
I20260709 12:16:15.072054 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1893) took 1243 ms. Trace:
I20260709 12:16:14.980437 22936 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.106s
I20260709 12:16:15.113082 12566 rpcz_store.cc:276] 0709 12:16:13.828808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.697499 (+868691us) service_pool.cc:224] Handling call
0709 12:16:14.697872 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:15.052508 (+354636us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:15.052539 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:15.052577 (+    38us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:15.052606 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.071786 (+ 19180us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:15.071839 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.071986 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":815109888}
I20260709 12:16:15.116361 12566 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:16:14.954649 22492 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.051734 22545 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:15.031858 22606 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:15.132755 12490 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:15.133740 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.134093 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:15.081693 13254 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:16:15.128129 22492 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.129828 22933 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:15.130254 22920 consensus_queue.cc:1175] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 29 status { last_received { term: 29 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 29 index: 15 } }
I20260709 12:16:15.065490 22823 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:15.065744 22232 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:15.060209 22585 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:15.032121 22914 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.083s
I20260709 12:16:15.065196 22530 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:15.074134 22886 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:15.137660 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.139315 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.185002 22492 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:15.201977 12569 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:15.162001 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.202932 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:15.092912 22896 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:15.110401 22734 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:15.213610 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.168696 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:15.111174 22918 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:16:15.183784 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52341.8us
I20260709 12:16:15.132606 22545 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:15.219606 12637 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:16:15.185278 22823 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:15.197935 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:15.200532 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:15.223333 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:15.146276 22507 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.064379 22903 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.092s
I20260709 12:16:15.223475 12641 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:15.205155 12644 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:15.115361 22951 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.129s
I20260709 12:16:15.194221 22937 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:15.204803 22497 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:15.233402 12640 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.003s
I20260709 12:16:15.242427 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
W20260709 12:16:15.234506 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160099us
I20260709 12:16:15.190595 22950 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:15.220798 22886 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:15.250952 22950 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.251610 22886 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.215114 22922 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:15.254045 22922 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:15.254729 22945 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:16:15.215322 22896 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:15.255152 22922 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.256698 22922 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:15.257025 22938 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
I20260709 12:16:15.229346 22823 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:15.258352 22937 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.260986 22937 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:15.246709 22941 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:16:15.250262 22951 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 30.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:15.250061 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.262493 22530 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.263644 12569 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:15.264067 22635 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.221933 22831 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:16:15.257922 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4130) took 1420 ms. Trace:
I20260709 12:16:15.265816 12572 rpcz_store.cc:276] 0709 12:16:13.837803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.705502 (+867699us) service_pool.cc:224] Handling call
0709 12:16:14.711553 (+  6051us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.713354 (+  1801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.713380 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.761401 (+ 48021us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.008725 (+247324us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:16:15.026527 (+ 17802us) log.cc:844] Serialized 14 byte log entry
0709 12:16:15.026754 (+   227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.026831 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:15.257682 (+230851us) raft_consensus.cc:1685] finished
0709 12:16:15.257703 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.257859 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":20174,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":562,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":106039,"spinlock_wait_cycles":5924736,"thread_start_us":130876,"threads_started":1,"wal-append.queue_time_us":195838}
I20260709 12:16:15.267872 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.269040 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.253520 22950 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.270381 22875 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:15.184552 22606 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.273334 22606 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:15.231415 22776 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.274637 22925 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.275005 22776 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:15.275259 22925 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:16:15.247406 22942 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
W20260709 12:16:15.277829 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61404.3us
I20260709 12:16:15.269340 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:15.278060 22635 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:15.278700 22635 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:15.279628 22635 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.280416 22918 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:15.264744 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.281996 12569 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:15.283806 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1434) took 1014 ms. Trace:
I20260709 12:16:15.283982 22885 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.275519 22776 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.285715 22929 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.285223 22918 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.214924 22833 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.275229 12461 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:15.322531 12460 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:15.212119 22687 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:15.337730 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.339870 12572 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:15.300586 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:15.341046 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.284313 12645 rpcz_store.cc:276] 0709 12:16:14.269432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.702448 (+433016us) service_pool.cc:224] Handling call
0709 12:16:14.702839 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.823225 (+120386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.823290 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.869148 (+ 45858us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.080618 (+211470us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:15.080667 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.080750 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:15.283562 (+202812us) raft_consensus.cc:1685] finished
0709 12:16:15.283587 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.283739 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119935,"raft.queue_time_us":172513,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":24599,"thread_start_us":143336,"threads_started":1,"wal-append.queue_time_us":287773}
I20260709 12:16:15.345707 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:15.331142 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 259692us
W20260709 12:16:15.330783 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 218347us
W20260709 12:16:15.334642 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59575.9us
I20260709 12:16:15.277285 22925 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:16:15.335114 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:15.331095 22902 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:15.330286 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1433) took 1061 ms. Trace:
I20260709 12:16:15.369048 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:15.353540 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:15.347095 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 287284us
I20260709 12:16:15.282042 22635 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:15.348867 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:15.354878 12491 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:15.378708 12491 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:15.332407 22606 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:15.360519 22896 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.287112 22776 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.241432 22591 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.121s
I20260709 12:16:15.364449 22502 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:15.371541 12572 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:15.369510 12646 rpcz_store.cc:276] 0709 12:16:14.269198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.678527 (+409329us) service_pool.cc:224] Handling call
0709 12:16:14.681206 (+  2679us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.899858 (+218652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.899895 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.924699 (+ 24804us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.133141 (+208442us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:15.329211 (+196070us) log.cc:844] Serialized 14 byte log entry
0709 12:16:15.329853 (+   642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.329933 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:15.329972 (+    39us) raft_consensus.cc:1685] finished
0709 12:16:15.329985 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.330194 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":252148,"raft.queue_time_us":219044,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":19201,"thread_start_us":256268,"threads_started":2,"wal-append.queue_time_us":282985}
W20260709 12:16:15.336061 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.007s	sys 0.020s
I20260709 12:16:15.373095 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:15.442214 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 73221.3us
W20260709 12:16:15.450814 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.008s	sys 0.021s
I20260709 12:16:15.338032 22507 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:16:15.457666 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:16:15.437736 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59878.8us
I20260709 12:16:15.378140 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:15.446667 12488 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
W20260709 12:16:15.455269 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82251.9us
I20260709 12:16:15.348861 22687 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.367828 22924 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.095s
I20260709 12:16:15.488438 12488 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:16:15.487145 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109063us
I20260709 12:16:15.481341 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:15.490345 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:15.470014 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:15.442045 22838 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:15.503997 12488 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:16:15.504457 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4135) took 1665 ms. Trace:
I20260709 12:16:15.382084 22833 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.097s
W20260709 12:16:15.442895 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89223.2us
W20260709 12:16:15.496933 12571 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.642s	user 0.015s	sys 0.000s
I20260709 12:16:15.468375 22909 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:15.444931 22937 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.057s
I20260709 12:16:15.468570 22967 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:15.517122 12563 rpcz_store.cc:276] 0709 12:16:13.839211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.845314 (+1006103us) service_pool.cc:224] Handling call
0709 12:16:14.845893 (+   579us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.849989 (+  4096us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.850036 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.854698 (+  4662us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.099825 (+245127us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:15.099894 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.100051 (+   157us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:15.504188 (+404137us) raft_consensus.cc:1685] finished
0709 12:16:15.504208 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.504392 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5736,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":6616,"thread_start_us":181183,"threads_started":1,"wal-append.queue_time_us":453054}
I20260709 12:16:15.520359 12571 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:15.529273 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.497509 22680 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:15.537217 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.519876 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:15.525092 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.539814 12493 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
W20260709 12:16:15.525833 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56175.6us
I20260709 12:16:15.531548 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:15.532984 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:15.535755 12497 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.584s	user 0.011s	sys 0.002s
I20260709 12:16:15.544406 12497 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:15.546705 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:15.547153 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.548619 12569 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:15.552386 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.553553 12488 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:15.504846 22891 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:15.449241 22606 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.493738 22914 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:16:15.508431 22958 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:15.517880 22971 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:15.559327 12571 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:15.559294 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:15.563836 12493 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:15.565352 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.554261 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.537801 22680 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:15.567531 12488 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:16:15.540504 22863 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:15.554106 22928 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:15.573503 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.623044 12495 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:15.624182 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.560356 22891 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.581980 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.582252 22776 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:15.582868 22502 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:15.626014 22776 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:15.626093 22502 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.626791 22891 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.555351 22964 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:15.627398 22776 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.628655 22502 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.630657 12566 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:16:15.587592 22606 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:16:15.591671 22960 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:16:15.592342 22976 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:15.592751 22956 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:15.593302 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.594178 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.561544 22902 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.573776 22680 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:15.573117 22958 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:15.561192 22909 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:15.599118 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.599748 22863 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:15.744603 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:15.745617 13260 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:15.599923 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.600559 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.748546 12497 raft_consensus.cc:2368] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:15.601178 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.575095 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:15.749616 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 174609us
I20260709 12:16:15.604825 12571 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:15.606525 22896 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.753867 12494 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:15.607257 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.545961 22886 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:16:15.756971 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:15.611354 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45978.1us
I20260709 12:16:15.758390 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.758991 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:15.574765 22967 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:15.761106 12488 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:16:15.762315 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.763029 22838 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:15.763548 22838 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.765784 22507 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:15.766312 22507 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.613245 22880 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:15.767398 22863 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 30.11, Last appended by leader: 11, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.768172 22880 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:15.769160 22880 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.769964 22882 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:15.770689 22882 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.686192 22956 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:15.611493 22914 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:15.772256 22914 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:15.615020 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52649.1us
I20260709 12:16:15.616331 22232 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.614231 22924 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:15.774091 22924 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 29.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
W20260709 12:16:15.615262 22821 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.558s	user 0.000s	sys 0.009s
I20260709 12:16:15.774516 12490 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:15.589133 22823 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.775008 22821 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.775430 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:15.567373 22885 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:16:15.776978 22823 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.777275 22885 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 25.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:16:15.578481 22687 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:15.778518 22983 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:15.779199 22983 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.783488 22833 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:15.784055 22833 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:15.787700 22958 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.789814 13262 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:15.790930 12565 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:16:15.791589 12491 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:15.737396 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 160230us
I20260709 12:16:15.769325 22680 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.796835 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.737532 22976 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.741779 22606 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 12:16:15.772109 22956 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:15.801491 22606 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:15.801968 22606 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:15.802999 22606 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:15.805307 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61701.2us
I20260709 12:16:15.810316 22497 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:15.814397 12643 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:15.740952 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:15.678074 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 298712us
I20260709 12:16:15.817166 13257 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:15.771546 12568 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [32.18-34.19]
I20260709 12:16:15.825937 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.773329 22914 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.834494 22914 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:15.836361 22924 consensus_queue.cc:1175] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 35 status { last_received { term: 30 index: 17 } last_committed_idx: 17 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [32.18-34.19]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:16:15.785130 22833 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.850617 12495 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:15.780089 22845 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.853108 22680 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.884274 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.889006 22752 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:15.916867 22752 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:15.918166 22752 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.793303 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4140) took 1157 ms. Trace:
I20260709 12:16:15.919173 12565 rpcz_store.cc:276] 0709 12:16:14.636187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.201553 (+565366us) service_pool.cc:224] Handling call
0709 12:16:15.201963 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:15.791441 (+589478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.792977 (+  1536us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:15.793204 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1353822464}
I20260709 12:16:15.857779 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4263) took 2024 ms. Trace:
I20260709 12:16:15.887452 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:15.888507 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:15.860983 12491 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:15.875602 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:15.856756 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.576s (SENT) [suppressed 1 similar messages]
I20260709 12:16:15.905786 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:15.888008 12646 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
W20260709 12:16:15.882853 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129689us
I20260709 12:16:15.890894 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:15.925855 12638 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:15.925714 12564 rpcz_store.cc:276] 0709 12:16:13.833658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.702400 (+868742us) service_pool.cc:224] Handling call
0709 12:16:14.702818 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.704763 (+  1945us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.704811 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.705359 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.453982 (+748623us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:15.454026 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.454107 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:15.786993 (+332886us) raft_consensus.cc:1685] finished
0709 12:16:15.787018 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:15.857680 (+ 70662us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":740512,"threads_started":1,"wal-append.queue_time_us":674994}
I20260709 12:16:15.924664 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:15.910179 12642 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:15.934113 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.938311 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:15.686483 12493 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:16:15.861268 22976 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.941906 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:15.885985 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:15.888896 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:15.918728 22845 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:15.855963 22545 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.615s	user 0.011s	sys 0.000s
I20260709 12:16:15.960528 22545 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:15.960994 22635 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:15.961666 22892 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:15.961717 22635 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.963497 22545 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:15.964185 22635 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.966953 12645 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:16:15.967078 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.967980 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.970373 12569 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:16:15.969363 12645 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 14. Preceding OpId from leader: term: 32 index: 15. (index mismatch)
I20260709 12:16:15.900869 13257 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:16:15.941087 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.017s	sys 0.025s
I20260709 12:16:15.951049 13262 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:15.943171 13260 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:15.979709 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:15.980916 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.981935 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:15.982604 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.018s	sys 0.025s
I20260709 12:16:15.982957 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:15.983032 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 642723us
I20260709 12:16:15.983723 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:15.985100 12646 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:16:15.945734 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:15.889549 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:15.990425 12644 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:15.995097 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:15.997480 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:15.898804 22833 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:16.012156 12491 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 23. (term mismatch)
I20260709 12:16:16.019717 12491 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 22
I20260709 12:16:16.020001 12491 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 32.23
I20260709 12:16:16.020316 12491 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:15.857435 22507 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.027597 12568 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:16:15.877153 22933 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:16:15.999449 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54018.8us
W20260709 12:16:16.000097 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:15.977867 22946 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.066s	user 0.001s	sys 0.006s
I20260709 12:16:15.979280 22969 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260709 12:16:16.035209 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:16.034971 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:16.036155 13260 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:15.871033 22896 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.133s
I20260709 12:16:15.889780 22492 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.116s
I20260709 12:16:16.054991 22969 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:16.055507 22969 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:16.008435 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:15.992815 22232 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.979724 22903 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:16:16.011373 22956 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.012148 12639 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:16:15.979571 22657 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:16:15.999022 22909 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.061340 22909 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.061869 22886 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:16.063625 22909 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.015759 22883 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:16.018154 22821 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:15.860983 22983 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.018586 12493 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 14. Preceding OpId from leader: term: 32 index: 15. (index mismatch)
I20260709 12:16:15.943392 22925 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:16.066072 22925 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:15.910910 22606 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:15.890491 22734 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:16.069334 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:16.069880 22734 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:16.070284 12563 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:16.070498 22956 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:16.071600 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4131) took 2233 ms. Trace:
I20260709 12:16:16.071998 12567 rpcz_store.cc:276] 0709 12:16:13.838108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:14.706741 (+868633us) service_pool.cc:224] Handling call
0709 12:16:14.711620 (+  4879us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:14.712168 (+   548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:14.712201 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:14.851860 (+139659us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.040821 (+188961us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:15.160014 (+119193us) log.cc:844] Serialized 14 byte log entry
0709 12:16:15.347960 (+187946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.348123 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:15.966776 (+618653us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b1800375a60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:16.011171 (+ 44395us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b1800375a60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:16.071374 (+ 60203us) raft_consensus.cc:1685] finished
0709 12:16:16.071409 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:16.071543 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":81440,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":3342,"raft.queue_time_us":46485,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":102930,"spinlock_wait_cycles":51288064,"thread_start_us":383730,"threads_started":3,"wal-append.queue_time_us":330025}
I20260709 12:16:15.912101 22838 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:15.864431 22880 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:16.081135 22978 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.010s
I20260709 12:16:16.030746 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.056381 22969 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.090824 22969 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:16.093030 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:16.100065 22882 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.100497 22734 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:15.991153 22758 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.025s	sys 0.039s
I20260709 12:16:16.002662 22823 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:16:16.036633 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 147822us
I20260709 12:16:16.050491 13262 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
I20260709 12:16:16.051710 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:16.054221 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.057706 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:16.059360 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:16.138954 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79683.4us
I20260709 12:16:16.140336 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:16.060799 22979 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.085s	user 0.010s	sys 0.000s
I20260709 12:16:16.100497 22971 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:16.033929 13257 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:16:16.032876 22918 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:16.032624 22933 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:16.112679 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.127245 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.138075 12564 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:16.139010 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87392.1us
W20260709 12:16:16.142843 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:16.143949 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 287219us
I20260709 12:16:16.145895 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:16.137966 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:16.131659 22758 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.025s	sys 0.039s
I20260709 12:16:16.173977 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:16.173832 22933 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.174396 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:16.175210 22933 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:16.176388 22734 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.177435 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.178380 22680 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:16.178818 22968 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:16.179435 22968 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:16.180464 22968 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.168772 22823 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:16.185592 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.186270 12536 leader_election.cc:400] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:16.182958 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:16.148959 22965 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:16.148746 22776 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:16.147436 22924 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.215080 22924 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.216559 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4032) took 1052 ms. Trace:
I20260709 12:16:16.217236 22924 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:16.197103 22977 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.000s	sys 0.004s
I20260709 12:16:16.200508 12572 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
W20260709 12:16:16.201786 22985 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.080s	user 0.000s	sys 0.007s
I20260709 12:16:16.201965 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:16.190927 22838 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:16:16.149085 22918 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:16.211755 22884 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:16.213387 22880 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.184922 22959 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:16:16.167214 22903 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:16.217033 12639 rpcz_store.cc:276] 0709 12:16:15.163814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.163951 (+   137us) service_pool.cc:224] Handling call
0709 12:16:15.164402 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:16.011938 (+847536us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b5400238fa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:16:16.071018 (+ 59080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:16.216261 (+145243us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:16.216484 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1904353280}
I20260709 12:16:16.256284 22591 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:16:16.193455 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.220240 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.221315 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:16.258059 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64692.3us
I20260709 12:16:16.258661 12496 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:16.258962 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:16.259867 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:16.260443 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.222266 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:16.261678 22929 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:16.262653 22929 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:16.224651 12491 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:16:16.264274 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:16.264880 12491 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:16:16.197163 22967 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:16.265586 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.264806 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.269340 12494 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:16.270714 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:16.253063 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.255512 22880 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.261824 22585 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:16.223218 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.261958 12493 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:16.236016 22976 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:16.243165 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:16.272604 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:16.285696 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:16.271226 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:16.287974 22929 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:16.288784 22929 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:16:16.252164 22918 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:16.263329 22965 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:16.278600 22967 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.239104 22530 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:16.290309 22965 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.290911 22530 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.278391 22903 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:16.293260 22967 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.294502 22925 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.296221 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.251904 22924 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.045s
I20260709 12:16:16.296772 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4141) took 1660 ms. Trace:
I20260709 12:16:16.297360 12497 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:16:16.298578 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:16.299266 22931 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.302320 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:16.303176 12488 raft_consensus.cc:606] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:16.303516 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.303959 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.304207 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:16.304348 12646 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 23. Current term is 24. Ops: []
I20260709 12:16:16.305488 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.297188 12570 rpcz_store.cc:276] 0709 12:16:14.636470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.375418 (+738948us) service_pool.cc:224] Handling call
0709 12:16:15.375802 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:15.437668 (+ 61866us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:15.437711 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:15.458375 (+ 20664us) log.cc:844] Serialized 26 byte log entry
0709 12:16:15.561211 (+102836us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:15.561255 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:15.561331 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:16.296520 (+735189us) raft_consensus.cc:1685] finished
0709 12:16:16.296545 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:16.296706 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3503,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":57803,"thread_start_us":34365,"threads_started":1,"wal-append.queue_time_us":399143}
I20260709 12:16:16.306628 12568 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26.
I20260709 12:16:16.306443 22903 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.307965 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.308701 12642 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:16.250062 22758 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.060s
I20260709 12:16:16.310645 22758 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:16.311067 22758 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:16.311602 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:16.312351 12638 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:16.312127 22758 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.312939 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:16.313318 22987 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20260709 12:16:16.313767 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:16.314054 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.314429 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:16.315151 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49661.7us
I20260709 12:16:16.322821 12536 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:16.323423 12644 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:16:16.315682 22861 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:16.303529 12488 raft_consensus.cc:614] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:16.282140 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:16.318866 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.320538 12460 leader_election.cc:400] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26.
I20260709 12:16:16.329231 12460 leader_election.cc:403] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:16.322069 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.324404 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.325445 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:16.325838 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
W20260709 12:16:16.318125 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47082.7us
I20260709 12:16:16.334824 12489 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:16.319389 12389 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:16.337858 12496 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:16.338862 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.339627 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.331527 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:16.340991 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.342550 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.343430 12640 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:16.351847 12642 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:16.279397 22585 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:16.372922 12639 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:16.297302 22924 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:16.377043 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:16.327778 22232 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:16.378891 12491 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:16:16.386354 22232 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:16.340247 22635 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:16.380785 22585 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:16.387351 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.386826 22232 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:16.384827 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:16.388134 12491 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:16.391001 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:16.394052 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.395709 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:16.396883 12644 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:16:16.340423 22883 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:16.340595 22971 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:16.401192 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.338289 22929 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:16.421811 22929 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.404604 22821 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:16.385452 22924 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:16.406751 12646 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:16:16.383073 22776 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:16.370577 22983 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:16.424360 12637 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 28 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:16:16.353225 22931 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 26
W20260709 12:16:16.466964 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76082.1us
I20260709 12:16:16.469873 12493 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:16.424871 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:16.414578 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:16.477051 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52286.9us
I20260709 12:16:16.477713 12642 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:16.389314 22928 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:16.459347 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.483721 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.469496 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:16.387773 22823 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:16:16.431322 22821 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:16.432251 22959 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:16.431768 22933 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:16.387404 22635 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:16.481722 22995 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:16:16.519125 12642 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:16.498694 22886 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:16.522850 12495 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 33 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:16:16.503834 22909 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:16.558195 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.498315 22983 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:16.560916 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91505us
I20260709 12:16:16.466398 22752 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:16:16.510390 22821 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:16.501827 22928 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.575081 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:16.587101 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:16.527650 22915 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:16.550460 22909 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:16.563884 22955 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:16.565267 22920 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.517712 22964 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:16:16.522882 22933 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.608160 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.597898 22884 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:16.594017 22680 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:16.599747 22909 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:16.627959 12642 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:16.627332 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:16.624078 22833 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:16.634639 23003 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 12:16:16.635411 22999 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:16:16.624972 22929 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:16.637224 22998 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:16:16.639156 22983 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.644771 22585 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.600016 22915 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.527175 22924 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.666386 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:16.615593 22967 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:16:16.676991 12638 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
W20260709 12:16:16.630100 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151843us
I20260709 12:16:16.685025 12488 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:16.685560 22915 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.681063 22955 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:16:16.686456 22680 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:16.687055 22955 consensus_queue.cc:1243] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 33.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:16:16.687331 22680 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:16.687846 22680 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:16.668133 22884 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:16.690375 22884 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:16.690825 22884 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:16.689258 22585 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:16.708894 23005 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:16:16.607985 22920 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.717221 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.718523 22886 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:16.746623 22886 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:16.747179 22886 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:16.748142 22886 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.749243 22886 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:16.722566 12639 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
W20260709 12:16:16.724078 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 289877us
I20260709 12:16:16.726061 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.726532 12644 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:16:16.750443 12642 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:16.751394 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4371) took 1240 ms. Trace:
I20260709 12:16:16.751564 22928 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.751907 12489 rpcz_store.cc:276] 0709 12:16:15.510596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.510765 (+   169us) service_pool.cc:224] Handling call
0709 12:16:15.530470 (+ 19705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:16.335338 (+804868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:16.751136 (+415798us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b540020e4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:16.751327 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1848629504}
I20260709 12:16:16.754139 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:16.726989 23004 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:16:16.728781 22635 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.769073 22933 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.770035 12488 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:16.729384 22884 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:16.730571 23002 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:16:16.730655 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.731182 22967 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.733952 22232 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:16.738545 23001 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:16:16.762568 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:16.762387 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.777860 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.798028 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:16.798552 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:16.784564 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:16.799136 22232 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.785368 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:16.802083 12488 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:16.789438 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.791442 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:16.792557 22967 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.806443 22967 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.811215 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:16.811697 12569 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:16.813234 22933 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:16.813812 22933 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:16.783371 22964 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.769100 22920 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.795554 22884 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.814199 22933 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:16.816150 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:16.788983 23008 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:16:16.721304 22680 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.833971 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.840375 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:16.814670 22964 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.860720 22680 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.842979 22635 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:16.862583 22964 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.839350 22884 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:16.787429 22821 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:16.864208 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:16.828696 22833 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:16:16.865010 22821 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.859997 12492 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:16:16.846457 22914 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:16:16.860787 23009 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.002s	sys 0.000s
I20260709 12:16:16.829248 22918 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.869342 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:16.870865 12645 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:16:16.871786 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.870955 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:16.873759 12639 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:16.873171 12493 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 31 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:16:16.875792 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:16.879642 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.880703 12642 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 31 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:16:16.858665 22931 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26.
I20260709 12:16:16.887679 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:16.887858 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:16.888754 12493 raft_consensus.cc:606] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:16.890638 12493 raft_consensus.cc:614] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:16.890664 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.889103 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:16.898810 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.886516 22680 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:16.907104 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:16.909693 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:16.904290 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150241us
I20260709 12:16:16.933099 12496 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:16.995683 22880 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.994614 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:16.899715 12646 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:16.950141 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:16.997056 22758 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:17.064097 22959 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:17.064819 22959 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:16.972164 22918 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.940161 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:16.984362 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.965153 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:16.968918 12492 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:16:16.964380 22983 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.890429 22920 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:16.951366 22530 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:16.904035 22641 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:16:16.946919 22914 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:17.079355 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 139285us
I20260709 12:16:17.086983 12644 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:16.990912 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:17.087626 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:17.088114 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97309.1us
I20260709 12:16:17.089221 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:16.974961 12639 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:16.934020 22680 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.090869 22752 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:17.092048 22964 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:17.092329 22680 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:16.899200 22821 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:16.952337 23015 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:16:16.954952 22909 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.095202 22585 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:17.095292 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:16.997083 22892 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.056830 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.096978 22530 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:17.097049 12568 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:17.098237 22902 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:17.086431 22880 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:17.068060 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:17.101859 22880 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:17.102021 12645 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:17.102491 22880 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.104933 22880 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:17.071558 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 735015us
I20260709 12:16:17.106235 12490 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 19. (term mismatch)
I20260709 12:16:17.106640 12490 pending_rounds.cc:85] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
I20260709 12:16:17.106865 12490 pending_rounds.cc:107] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 33.19
I20260709 12:16:17.107102 12490 raft_consensus.cc:2889] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:17.077526 12491 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:17.108891 12491 raft_consensus.cc:614] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:16:17.078435 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94184us
I20260709 12:16:17.054368 22497 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.110872 22497 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.111747 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:17.112999 22497 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:17.117208 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:17.117887 12493 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:17.118141 12488 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:16:17.119472 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.120210 12640 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:17.102079 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3299) took 1168 ms. Trace:
I20260709 12:16:17.084887 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.122109 12495 rpcz_store.cc:276] 0709 12:16:15.933166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:16.295901 (+362735us) service_pool.cc:224] Handling call
0709 12:16:17.101837 (+805936us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:17.102017 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":518629632}
I20260709 12:16:17.086424 23013 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:17.087153 22914 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:17.136264 12567 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:16:17.092653 22964 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:17.142239 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:16.971453 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:17.147627 12642 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:17.098099 22530 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:16:17.135677 22929 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.031s
I20260709 12:16:17.160032 22680 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:17.160945 22959 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.160683 22680 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.120414 22920 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.163547 22680 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.101329 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:17.163880 22964 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.165378 22833 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:17.165877 22833 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:17.166286 22833 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:17.122823 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.124235 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:17.124571 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.125550 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:17.132220 12564 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:16:17.124213 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:17.148608 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.154903 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.096490 22909 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:17.176393 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52305.8us
I20260709 12:16:17.096920 22892 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.140406 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:17.164439 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63232.2us
I20260709 12:16:16.940976 23011 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.010s
I20260709 12:16:17.173024 12645 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:17.181041 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:17.066521 22933 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.186650 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.191524 12491 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:17.191944 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:16:17.203771 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63516.7us
I20260709 12:16:17.157954 22845 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:17.218607 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.205024 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:17.231892 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 106448us
W20260709 12:16:17.226446 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49799.8us
I20260709 12:16:17.237870 12642 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:17.146179 22585 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:16:17.119529 22983 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.240931 22928 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:17.242009 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.311422 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:17.269330 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:17.286732 22965 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:16:17.199628 22642 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:17.171172 22880 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 12:16:17.290289 12489 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 19. (term mismatch)
I20260709 12:16:17.331496 12489 pending_rounds.cc:85] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
I20260709 12:16:17.331773 12489 pending_rounds.cc:107] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 34.19
I20260709 12:16:17.332010 12489 raft_consensus.cc:2889] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:17.182523 23013 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:17.293947 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.183712 22959 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:17.199901 22734 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:17.306697 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.226982 22955 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:17.327610 12491 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
W20260709 12:16:17.273818 22997 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:16:17.354605 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58548.8us
I20260709 12:16:17.355605 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:17.330720 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 149381us
I20260709 12:16:17.342361 23013 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.16, Last appended by leader: 16, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.343107 22892 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.344930 12570 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 11. Preceding OpId from leader: term: 32 index: 12. (index mismatch)
I20260709 12:16:17.345444 22924 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:17.291517 22530 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:17.350028 12495 raft_consensus.cc:606] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:17.354157 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46619.9us
I20260709 12:16:17.347434 13262 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:16:17.393386 12495 raft_consensus.cc:614] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:17.393916 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:17.362231 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:17.327288 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93678.3us
I20260709 12:16:17.332358 22933 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.349833 22734 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:17.415050 12572 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:16:17.395077 23015 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:17.427963 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72420.7us
I20260709 12:16:17.309386 23036 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.107s
I20260709 12:16:17.392993 22530 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:17.362766 22752 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 12:16:17.363468 23016 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
W20260709 12:16:17.449411 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55585.8us
I20260709 12:16:17.403416 22955 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:17.411176 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49138us
I20260709 12:16:17.424916 22875 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:17.412911 22920 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.445741 23011 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:17.363618 22641 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.083s
I20260709 12:16:17.469698 12491 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:16:17.453984 22530 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:17.451857 22502 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:17.476378 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:17.483417 23038 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:16:17.484724 23034 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:16:17.485713 23032 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:16:17.446249 23015 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:17.501619 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:17.477483 22875 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:17.507330 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:17.490073 22514 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:17.489773 23038 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:17.521045 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3300) took 1587 ms. Trace:
I20260709 12:16:17.521700 12497 rpcz_store.cc:276] 0709 12:16:15.933406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.933527 (+   121us) service_pool.cc:224] Handling call
0709 12:16:17.520759 (+1587232us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:17.520957 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":832977280}
I20260709 12:16:17.527246 22861 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:17.531874 22875 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:17.533478 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4047) took 1646 ms (client timeout 1926 ms). Trace:
W20260709 12:16:17.534359 12637 rpcz_store.cc:269] 0709 12:16:15.886616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:15.888197 (+  1581us) service_pool.cc:224] Handling call
0709 12:16:17.533133 (+1644936us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:17.533386 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1151942400}
I20260709 12:16:17.464026 23018 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:16:17.553960 23018 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:17.554534 23018 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:17.555552 23018 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.480813 22497 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.025s
I20260709 12:16:17.495950 22882 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:17.559703 23015 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.560570 23015 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.560156 22882 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.563159 22882 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.564796 23038 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.566668 22514 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.569123 22983 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.533692 12638 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:17.482327 22734 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.545167 22955 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.578049 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.576879 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.505045 22918 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.509981 22642 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:17.578723 13257 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:17.591002 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:17.601668 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:17.602870 12488 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:17.620842 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.620685 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:17.569768 22994 consensus_queue.cc:1175] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 27 status { last_received { term: 27 index: 17 } last_committed_idx: 17 last_received_current_leader { term: 27 index: 17 } }
I20260709 12:16:17.623450 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:17.623585 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.624775 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:17.557353 23018 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:17.630404 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.652992 12563 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:17.669355 22976 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:17.619611 22920 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:17.588966 22657 consensus_queue.cc:1175] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 28 status { last_received { term: 28 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 28 index: 16 } }
I20260709 12:16:17.670506 22920 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.671190 23036 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.671468 22734 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:17.671890 23036 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.561214 23015 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.509287 23006 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.109s
I20260709 12:16:17.578860 22885 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.576906 22514 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.698951 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.594745 22883 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.517448 22902 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:17.706915 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:17.714802 23015 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.715392 12567 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:17.725518 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:17.673002 22971 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:17.729949 12488 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 36 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:17.734810 12571 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:17.649984 23043 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:17.629410 22918 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.625615 22833 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.745963 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:17.709177 22920 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.673167 22909 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 12:16:17.766011 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:17.737020 22823 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:17.721946 22885 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.759451 22845 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.767242 22880 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.792470 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.759104 23043 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:17.793695 12645 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:17.787151 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 308441us
I20260709 12:16:17.760485 22971 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:17.732065 22883 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.737509 23041 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
I20260709 12:16:17.800776 12571 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:17.779989 22823 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:17.807185 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 331727us
I20260709 12:16:17.820219 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:17.821694 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.822578 12642 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:17.824936 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:17.826669 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:17.825870 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:17.802945 23043 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:17.816934 22931 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:17.855077 23016 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:17.859086 22902 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:17.845090 22823 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.822450 22920 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:17.803063 22845 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.869964 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:17.826982 23036 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.818465 22497 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:16:17.910643 22920 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:17.913301 22920 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:17.807948 22880 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.819661 23011 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:17.892066 22823 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:17.893034 22885 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.887408 22902 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.900656 22883 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.901132 22845 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.901852 22833 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:17.902197 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.902803 23051 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:17.903241 22530 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:17.878561 23032 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:16:17.917273 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4061) took 1201 ms. Trace:
I20260709 12:16:17.919687 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.920593 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.921334 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:17.978623 12645 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:17.925889 22924 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.004s
I20260709 12:16:17.926667 23016 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:17.934939 12389 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:17.948055 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.955781 12567 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
W20260709 12:16:17.956341 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 330250us
I20260709 12:16:17.957726 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:17.967439 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.985561 13254 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:17.976461 12638 rpcz_store.cc:276] 0709 12:16:16.716206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:16.720672 (+  4466us) service_pool.cc:224] Handling call
0709 12:16:17.533309 (+812637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:17.570593 (+ 37284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:17.916937 (+346344us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54001f68d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:17.917182 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867911040}
I20260709 12:16:17.987390 13254 raft_consensus.cc:606] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:17.987872 13254 raft_consensus.cc:614] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:17.970829 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:17.973297 12644 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:16:17.991247 12491 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 34 because replica has last-logged OpId of term: 32 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:16:17.883261 22931 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:17.996105 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:17.998170 23036 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:18.008086 12639 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:17.977330 23051 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:18.023968 12638 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:17.998739 22545 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:16:18.027056 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:16:18.022219 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 196457us
W20260709 12:16:18.039351 23054 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:16:18.036872 22924 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.041118 22924 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.041759 22902 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.027967 23050 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.024s
I20260709 12:16:18.048158 23016 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.022372 22918 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:18.011143 23036 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:18.052165 23036 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:18.052681 22914 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.884s	user 0.008s	sys 0.004s
I20260709 12:16:18.067026 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:18.070255 12389 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:18.072590 12389 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 26. New cstate: current_term: 26 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:18.053181 22914 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.029161 23051 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:18.084007 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.086272 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:18.084962 12569 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:17.963583 23043 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.040362 22530 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:18.094724 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.064877 22959 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:18.107668 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:18.111975 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:18.118683 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.090185 23059 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.046s
I20260709 12:16:18.124486 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.125177 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.125221 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:16:18.126206 12568 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:18.127257 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:18.127733 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.128459 12565 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:18.127931 12564 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:18.132221 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.142275 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.143316 12496 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:18.074184 23048 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.086673 23018 raft_consensus.cc:2764] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
W20260709 12:16:18.149425 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51419.8us
I20260709 12:16:18.151026 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:17.977726 22971 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.154511 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.156001 12570 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:18.157544 13262 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:18.163259 12564 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:18.140847 22680 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:18.149026 22929 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:18.203312 22924 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:18.204012 22880 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.146005 23011 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:18.206197 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111550us
I20260709 12:16:18.207494 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.181177 23058 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:18.215490 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.146770 22959 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.217336 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:18.185770 22497 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.194005 22680 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:18.152108 23057 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:16:18.217604 23058 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:18.152976 23048 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:18.237668 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 253792us
I20260709 12:16:18.217267 22861 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:18.220199 23055 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.230962 22918 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:18.248692 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.250200 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.255643 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:18.240645 23058 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:18.230062 22497 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.260783 22931 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:18.267100 22821 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.174s	user 0.019s	sys 0.006s
I20260709 12:16:18.291109 22497 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:18.281044 22902 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:18.282846 23063 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:16:18.299484 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.302587 22959 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.303202 22914 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:18.303817 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 531935us
I20260709 12:16:18.307008 23036 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.336900 23036 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:18.338302 12489 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:16:18.338524 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.339196 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.339351 12494 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:18.342166 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.311777 23048 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:18.313839 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79682.2us
I20260709 12:16:18.257864 23055 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.277484 22967 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:18.343328 23036 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:18.276614 22680 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.343912 23036 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:18.344203 22680 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:18.345412 23055 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:18.321000 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:18.344988 23036 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.349062 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.349880 12564 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:18.350615 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.352743 22959 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:18.353161 12646 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:16:18.354444 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 11 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.356565 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.360766 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:18.363533 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:18.364276 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:18.366798 22585 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:16:18.368140 22964 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:18.368523 22964 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:18.369454 22964 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 32.13, Last appended by leader: 13, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.371110 22964 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:18.370997 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:18.371992 22968 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:18.372949 22968 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:18.373355 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:18.374173 22968 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:18.374632 22968 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:18.375953 22968 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.378207 22968 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:18.381853 12497 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:18.383209 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.383514 12494 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 11. Preceding OpId from leader: term: 32 index: 12. (index mismatch)
I20260709 12:16:18.383939 12488 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:16:18.385283 12493 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 20. (term mismatch)
I20260709 12:16:18.385749 12493 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:16:18.385967 12493 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 32.20
I20260709 12:16:18.386219 12493 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:18.387671 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:18.396122 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:18.407681 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:18.408705 12497 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:18.408502 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.410372 12494 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:18.401809 12493 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: [31.16-31.16]
I20260709 12:16:18.418845 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:18.421468 12497 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:16:18.423588 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.425783 12497 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:18.426492 23076 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:18.427201 23076 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.429569 23076 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.433604 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.436568 13261 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
W20260709 12:16:18.442539 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:18.439064 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86670us
W20260709 12:16:18.448580 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87988.5us
I20260709 12:16:18.448614 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.450657 23059 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:18.451182 23059 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:18.330539 22821 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.237s	user 0.019s	sys 0.006s
I20260709 12:16:18.451579 23059 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:18.451864 22821 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:18.452625 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.453377 12641 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:18.454077 22886 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:18.454636 22886 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.453024 22821 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 31.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.456816 22886 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:18.437561 23077 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:16:18.460381 23011 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:18.460834 22914 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.461061 22886 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:18.461584 22914 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.461078 23011 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.465895 12564 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:18.468482 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:18.444255 23071 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.122s	user 0.002s	sys 0.000s
I20260709 12:16:18.482218 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.471887 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.483843 12646 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:18.479768 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:18.473729 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.486024 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.488376 23051 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.491274 23051 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.494915 13254 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:18.500461 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.502188 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.502048 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.507542 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.509001 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.331377 23057 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.153s
I20260709 12:16:18.481297 22964 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:18.463259 22886 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:18.516252 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.514976 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.516846 22959 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:18.511159 22902 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:18.536749 23048 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:18.553434 12643 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:16:18.516060 22920 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:18.524926 23062 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.499781 23013 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.048s
I20260709 12:16:18.567206 23013 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:18.567663 23013 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:18.569306 22964 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:18.569845 22964 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:18.523270 23018 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:16:18.538332 22641 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:16:18.564814 23048 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.544035 22959 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:18.573938 22959 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:18.580051 23062 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.593669 22920 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.604763 22914 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:18.541903 22880 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:18.625283 12490 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
W20260709 12:16:18.619807 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104893us
I20260709 12:16:18.608567 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:18.641639 13260 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:18.645395 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.645552 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:18.629909 23066 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:18.652686 22838 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:18.653234 22838 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:18.653693 22838 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:18.609480 22971 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:18.670547 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:18.673599 12563 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:18.675765 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.686165 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:18.609791 22880 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.675681 22902 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:18.689204 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.644157 23016 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.035s
I20260709 12:16:18.668928 22734 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:18.655135 23057 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
W20260709 12:16:18.699244 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 182947us
W20260709 12:16:18.700616 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:18.706138 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:18.686697 23011 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:18.671013 22758 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.028s
I20260709 12:16:18.687455 22530 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:18.696776 22994 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:18.717675 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:18.712278 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:18.712743 22971 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:18.725394 22920 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.727675 23048 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.734979 22892 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:18.683643 22641 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:16:18.684880 23058 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.753005 23058 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:18.694404 23018 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.029s
I20260709 12:16:18.735991 22606 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260709 12:16:18.748698 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:18.715062 23049 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:18.677065 22886 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.783762 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:18.698208 22902 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.791415 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.798396 12488 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
W20260709 12:16:18.817859 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119169us
I20260709 12:16:18.817567 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.786455 22680 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:18.821455 23062 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.795904 22502 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:18.840770 22502 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:18.823553 12646 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:18.824620 22752 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:18.831176 22606 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:18.800040 23048 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:18.809115 23036 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:16:18.811578 23058 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:18.809372 22994 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:18.768133 22971 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.821738 22955 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:18.836027 23043 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:18.807102 22959 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.844476 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.845384 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:18.843058 23095 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:16:18.854274 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70687.2us
I20260709 12:16:18.885871 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:18.882969 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61029.2us
I20260709 12:16:18.850207 22606 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:18.893854 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:18.897228 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.871083 23058 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:18.870761 22994 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:18.907630 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:18.912256 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:18.899765 22823 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:18.918812 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:18.919783 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:18.919688 22914 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:18.920626 22914 consensus_queue.cc:1243] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:18.920796 22892 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:18.925592 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:18.929584 12572 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:16:18.901468 22955 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:18.938627 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:18.940296 12564 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:18.905682 22920 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:18.921468 22892 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:18.940086 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:18.913894 22680 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:18.912693 23049 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:18.948256 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:18.949506 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:18.923331 22823 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:18.950532 22823 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:18.954447 22902 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.955194 23058 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:18.956691 23058 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:18.946940 22606 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.17, Last appended by leader: 17, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.958640 22823 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.959069 22606 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:18.922619 22833 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:18.965868 22833 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:18.966323 22833 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:18.967315 22967 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:18.967885 22967 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:18.969494 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:18.970414 12493 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:18.970449 22971 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.970721 23038 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:18.973016 22823 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.978464 22880 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.983114 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:18.986472 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:18.883093 22502 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 16, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.988061 22994 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:18.989264 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:18.990124 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:18.991005 12570 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:18.994563 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3317) took 1816 ms. Trace:
I20260709 12:16:18.995029 12496 rpcz_store.cc:276] 0709 12:16:17.179965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:17.181659 (+  1694us) service_pool.cc:224] Handling call
0709 12:16:17.182041 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:18.143781 (+961740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:18.994258 (+850477us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:18.994481 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2210184960}
I20260709 12:16:18.947191 22892 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:18.998540 22497 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:18.999320 22959 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:18.999155 22497 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.000414 22955 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.004442 22924 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:19.004913 22924 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:18.944324 22931 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:19.014971 22931 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:19.015414 22931 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.018474 23109 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:18.939450 23048 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:16:19.025852 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:19.026650 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86636us
I20260709 12:16:19.027304 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:19.027961 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:19.027305 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:19.031213 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.032950 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.033721 12645 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:19.044567 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:19.066346 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.067224 12642 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:19.071020 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84653.6us
I20260709 12:16:19.071892 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:19.076237 12488 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:19.091863 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:19.051110 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112946us
I20260709 12:16:19.095161 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:19.085065 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:19.025257 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:19.087571 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.117977 13258 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
W20260709 12:16:19.087038 22964 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.010s
I20260709 12:16:19.120690 12567 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:19.121313 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:19.066481 22920 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:19.131152 23109 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:19.124888 12563 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:19.087872 22880 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:16:19.132798 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104951us
I20260709 12:16:19.131654 22232 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.146792 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:16:19.149696 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.150534 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.128790 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:19.140762 12642 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:19.087035 22959 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:19.151266 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.157363 12645 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:19.152602 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:19.155277 12567 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:16:19.155663 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128765us
I20260709 12:16:19.160334 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.159443 12645 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: [26.18-28.19]
I20260709 12:16:19.097700 23050 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:19.098117 23051 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:19.167093 13254 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:19.118567 22968 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:19.097344 22642 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.060s
I20260709 12:16:19.111351 22838 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.177850 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.222532 12641 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:19.173102 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44420.4us
I20260709 12:16:19.198165 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:19.176091 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.158438 23036 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:19.141525 22880 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:19.223002 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:19.220873 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67359.1us
I20260709 12:16:19.226222 12638 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29.
I20260709 12:16:19.156519 23049 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:19.271368 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47089.4us
I20260709 12:16:19.241355 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:19.246737 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48682.2us
W20260709 12:16:19.247401 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53050.7us
W20260709 12:16:19.202962 23072 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:16:19.250835 12566 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:19.123771 22964 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.249076 22821 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:19.096701 22924 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.174999 22635 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:19.280709 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79403.4us
I20260709 12:16:19.220312 22882 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:19.298208 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55412.4us
I20260709 12:16:19.234728 23048 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:19.169376 23056 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.115s
I20260709 12:16:19.232990 22929 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:19.234834 23036 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:19.269830 22880 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.275614 23093 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:19.171427 22959 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.300940 22983 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:19.220183 22968 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.313571 22635 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:19.269256 23067 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.091s
I20260709 12:16:19.314118 22497 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:19.274816 22886 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:16:19.328656 23036 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.286573 23049 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:19.367013 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52188.6us
I20260709 12:16:19.340106 23106 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:19.308573 22821 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.354521 22635 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:19.346133 22823 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:19.324921 23112 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:16:19.321453 23048 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.329234 22929 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:19.391763 23108 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260709 12:16:19.392544 23111 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:16:19.392117 23013 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:19.394701 23107 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:16:19.339788 22918 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
W20260709 12:16:19.380703 23059 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.929s	user 0.008s	sys 0.020s
I20260709 12:16:19.381704 12641 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:19.316998 22920 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.414549 23013 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:19.307322 23109 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.379408 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:19.439944 12638 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:19.446384 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4401) took 1022 ms. Trace:
I20260709 12:16:19.447360 22964 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.448487 22838 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:19.446924 12492 rpcz_store.cc:276] 0709 12:16:18.423462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:18.423644 (+   182us) service_pool.cc:224] Handling call
0709 12:16:18.424178 (+   534us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:18.424729 (+   551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:18.424770 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:18.425677 (+   907us) log.cc:844] Serialized 52 byte log entry
0709 12:16:18.507900 (+ 82223us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:18.543794 (+ 35894us) log.cc:844] Serialized 14 byte log entry
0709 12:16:18.544153 (+   359us) log.cc:844] Serialized 14 byte log entry
0709 12:16:18.544323 (+   170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:18.544393 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:18.652986 (+108593us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b180042cdc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:19.309980 (+656994us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b2400121568. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:16:19.446116 (+136136us) raft_consensus.cc:1685] finished
0709 12:16:19.446142 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:19.446317 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3956,"prepare.queue_time_us":102059,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":94,"raft.queue_time_us":12131,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":11762,"spinlock_wait_cycles":53683712,"thread_start_us":59795,"threads_started":1,"wal-append.queue_time_us":140320}
I20260709 12:16:19.454712 22968 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.456645 22929 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.456370 22502 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:19.444273 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102905us
I20260709 12:16:19.403730 22823 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.456584 23049 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:19.475986 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96662.8us
I20260709 12:16:19.480489 22959 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.481319 22821 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.494993 22892 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:19.485591 23038 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.514s	user 0.013s	sys 0.000s
I20260709 12:16:19.497403 22892 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.478046 23103 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:19.530212 12563 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:19.442699 23059 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.526635 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4179) took 2342 ms. Trace:
I20260709 12:16:19.534291 12566 rpcz_store.cc:276] 0709 12:16:17.169354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:17.169958 (+   604us) service_pool.cc:224] Handling call
0709 12:16:17.193216 (+ 23258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:19.287083 (+2093867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:19.510998 (+223915us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:19.511384 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4731539840}
W20260709 12:16:19.506156 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 411069us
W20260709 12:16:19.538678 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:19.479264 23116 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:16:19.527441 23076 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:19.547231 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 307915us
W20260709 12:16:19.552280 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59900.4us
W20260709 12:16:19.553293 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 247298us
I20260709 12:16:19.564661 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.571174 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:19.541604 22823 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.562058 23116 consensus_queue.cc:1243] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:19.555402 23103 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:19.590950 23114 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 12:16:19.556936 12461 leader_election.cc:400] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29.
W20260709 12:16:19.625015 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66408.2us
W20260709 12:16:19.639422 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63840.4us
W20260709 12:16:19.625615 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:19.577204 23038 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:19.589754 22903 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:19.648504 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:19.666483 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109710us
I20260709 12:16:19.667094 13258 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:19.642484 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:19.666172 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:19.648717 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:19.560954 23076 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.667420 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:19.581290 23055 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:16:19.748744 23133 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:19.628996 22956 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:19.749962 22956 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.751788 22635 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.753706 23093 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:19.754139 23093 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:19.754237 22635 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:19.697178 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52712.6us
W20260709 12:16:19.699401 23134 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:16:19.749373 23133 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:19.755278 23093 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:19.700253 23121 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:16:19.756608 23133 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:19.704789 22833 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.738s	user 0.009s	sys 0.003s
W20260709 12:16:19.712770 23058 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.756s	user 0.013s	sys 0.000s
W20260709 12:16:19.714175 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73831.1us
I20260709 12:16:19.696290 22821 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:19.718014 23126 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:19.719702 23059 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:19.642722 23103 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:19.720517 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54514.3us
I20260709 12:16:19.725965 13258 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:19.734110 23131 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:19.792294 23131 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:19.793519 23103 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:19.793254 23131 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.16, Last appended by leader: 16, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.794838 23119 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:19.689967 23048 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.795418 23119 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.669211 23038 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.797881 23038 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:19.728710 22680 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.767s	user 0.008s	sys 0.004s
W20260709 12:16:19.748147 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80801.7us
I20260709 12:16:19.766942 22892 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:19.799158 22892 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:19.752251 22956 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:19.629257 23016 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:19.692672 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:19.757459 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:19.804252 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.807590 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:19.776443 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:19.783874 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:19.810125 12570 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.784266 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:19.757818 22833 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.759824 23058 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:19.761499 22931 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.746s	user 0.012s	sys 0.002s
I20260709 12:16:19.758971 12639 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:16:19.810487 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.829991 12638 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:19.812103 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.813337 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:19.832674 13258 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:19.810050 12572 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:19.835783 13254 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:19.781450 22903 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:19.773830 22967 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.765s	user 0.012s	sys 0.001s
I20260709 12:16:19.844693 22967 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.847782 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.819783 22680 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:19.758574 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:19.850824 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92339.6us
I20260709 12:16:19.851784 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:19.852171 22994 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:19.839113 12646 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:16:19.838414 23058 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.756713 23093 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.759264 22821 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.741056 23036 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.824718 22635 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:19.856357 22635 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:19.856500 22821 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.857308 22821 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:19.772305 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:19.869323 13262 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.852100 22964 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:16:19.859578 22931 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.855542 23058 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:19.883850 22931 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.884766 12488 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:19.765988 22880 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.890990 22880 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.820016 23038 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:19.857659 22635 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 29.23, Last appended by leader: 23, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.802302 23016 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:19.858337 12491 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:19.901829 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.895309 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:19.845098 23119 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.837219 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:19.864089 12638 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:19.865224 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:19.769268 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120653us
I20260709 12:16:19.867245 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.841461 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:19.767031 22232 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.611s	user 0.012s	sys 0.000s
I20260709 12:16:19.858292 22585 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:19.770879 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:19.869618 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.908434 12566 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:19.908753 12492 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:19.875023 12490 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:19.877853 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:19.879251 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:19.910456 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:19.880373 22892 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.912114 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:19.913249 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 76181us
I20260709 12:16:19.915557 12643 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:19.916622 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.917129 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.909894 12572 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:19.880921 22903 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.853850 12646 pending_rounds.cc:85] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 17
I20260709 12:16:19.857429 22833 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.847883 22967 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:19.892252 23013 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:19.857921 23036 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:19.897619 13262 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:19.934919 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:19.898769 12570 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:19.903880 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.904381 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
W20260709 12:16:19.893445 23132 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:16:19.874200 12493 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:19.919167 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.921260 22821 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:19.918380 12645 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:19.917320 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:19.923240 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.924290 12646 pending_rounds.cc:107] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:16:19.953392 12646 raft_consensus.cc:2889] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:19.905010 22823 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:19.952714 12643 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:19.932248 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.976588 12496 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:19.976764 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:19.977294 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:19.978224 12495 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:19.937089 12497 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:19.895474 22606 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:16:19.943046 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:19.929311 23103 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:19.981290 12494 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:19.949450 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:19.912170 22232 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:19.894251 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:19.948906 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:19.982450 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.991271 12641 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:19.930773 22752 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:19.927013 22585 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:19.902117 23038 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.993789 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:19.995245 12536 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:19.934340 22892 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:19.929064 23062 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:20.018103 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:19.996218 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.007359 12536 leader_election.cc:403] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:20.007442 12641 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:20.013052 22919 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:16:20.013783 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119668us
I20260709 12:16:19.981768 23067 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:19.984362 23058 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:19.934809 23013 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:19.986642 23103 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:20.023646 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:19.995026 22752 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:20.027298 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:20.038417 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.028617 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.040314 12569 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:20.019234 23049 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:20.064801 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4405) took 1084 ms. Trace:
I20260709 12:16:20.070592 12489 rpcz_store.cc:276] 0709 12:16:18.980601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:18.980737 (+   136us) service_pool.cc:224] Handling call
0709 12:16:18.981187 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:19.163775 (+182588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:19.163808 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:19.164377 (+   569us) log.cc:844] Serialized 26 byte log entry
0709 12:16:19.641806 (+477429us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:16:19.641985 (+   179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:19.642109 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:19.892754 (+250645us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:20.064473 (+171719us) raft_consensus.cc:1685] finished
0709 12:16:20.064497 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:20.064729 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":112568,"raft.queue_time_us":149662,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":58052,"spinlock_wait_cycles":42736512,"thread_start_us":341066,"threads_started":1,"wal-append.queue_time_us":527624}
I20260709 12:16:20.067205 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:20.075752 12640 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 36 because replica has last-logged OpId of term: 34 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 16.
I20260709 12:16:19.995306 22959 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:16:19.984234 22823 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.035604 22886 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:16:20.099139 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.069052 23036 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:20.039484 23103 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.109412 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:20.118036 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.055616 22861 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.032864 23067 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:20.112903 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:20.051472 22919 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:20.126566 12495 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:16:20.118489 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.130527 12569 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:20.039875 22971 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.077s
I20260709 12:16:20.137552 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.039096 23058 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.140684 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:20.024803 23062 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.112346 23036 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:20.113401 23126 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.164975 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.128109 22919 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.174558 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.176127 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.176839 12569 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:20.177989 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.101953 23049 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.128470 22880 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:20.187378 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:20.248219 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:20.167738 23036 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:20.158615 23056 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:20.263263 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80159.9us
I20260709 12:16:20.179966 23126 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.135183 22861 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.163847 23124 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.152112 23018 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:16:20.105331 22821 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.262437 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:20.267127 22833 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:20.267562 22956 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.133018 22752 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.24, Last appended by leader: 24, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.258805 23016 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:20.269925 23138 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:16:20.119146 22585 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:20.255947 23122 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:16:20.286980 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:20.294765 23126 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:20.268630 23140 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:16:20.297937 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:20.346638 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48793.2us
I20260709 12:16:20.347677 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:20.355578 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:20.313571 22680 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:20.299396 23124 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.281832 23056 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.273424 23067 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.351977 23109 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:16:20.405541 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130613us
I20260709 12:16:20.364805 22920 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:20.360548 23038 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:20.323941 22956 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.371569 22833 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:20.378504 22752 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:20.371685 23013 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.406906 23076 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:20.446295 23136 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.299s	user 0.000s	sys 0.002s
I20260709 12:16:20.450197 23109 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:20.457574 22823 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:20.500756 23141 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:16:20.497746 23058 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.525012 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3379) took 1607 ms. Trace:
I20260709 12:16:20.525494 12646 rpcz_store.cc:276] 0709 12:16:18.917383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:18.917574 (+   191us) service_pool.cc:224] Handling call
0709 12:16:18.917949 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:19.954037 (+1036088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:20.524627 (+570590us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:20.524914 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2083078016}
I20260709 12:16:20.531566 22823 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:20.540234 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:20.545941 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.547616 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.548985 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:20.566540 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:20.567988 12488 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:20.555388 23093 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.701s	user 0.011s	sys 0.004s
W20260709 12:16:20.571285 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 215799us
I20260709 12:16:20.571974 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:20.551959 23142 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
I20260709 12:16:20.575593 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.576735 12564 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:16:20.557068 22924 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:20.556882 23058 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.555871 22823 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.580403 12572 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:20.587268 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.634753 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:20.591845 23124 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.594348 22956 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.569087 22861 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.580139 23150 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:16:20.601006 22994 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.748s	user 0.009s	sys 0.004s
I20260709 12:16:20.577793 22635 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:20.603118 23016 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:20.602731 23058 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.590425 22734 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:20.590256 23118 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:20.606457 23056 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.606794 22919 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.610392 22687 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:16:20.614709 23125 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.922s	user 0.002s	sys 0.000s
I20260709 12:16:20.651865 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.568557 23049 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:20.568928 23109 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:20.617970 23120 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.921s	user 0.003s	sys 0.000s
W20260709 12:16:20.619907 22886 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.004s
W20260709 12:16:20.624594 23144 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.378s	user 0.000s	sys 0.006s
I20260709 12:16:20.642727 22497 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:20.626430 22892 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.605s	user 0.010s	sys 0.003s
I20260709 12:16:20.627058 23103 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.644404 22734 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:20.633515 12570 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 31 because replica has last-logged OpId of term: 30 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 23.
I20260709 12:16:20.647097 12642 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 33 because replica has last-logged OpId of term: 31 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
W20260709 12:16:20.644680 23145 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260709 12:16:20.654938 22861 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.685911 22861 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.688446 22861 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.690797 23059 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.691426 23059 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.670678 22886 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:20.589407 23062 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.641283 22635 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:20.641527 23150 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:20.641958 23016 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:20.665117 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.669528 12565 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:20.675171 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.644145 23118 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:20.679499 23148 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260709 12:16:20.648311 22956 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.655412 22687 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:20.679795 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:20.672626 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.695355 23059 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.678855 22530 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.703033 22823 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:20.723644 22956 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:20.726500 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:20.703178 22919 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.704275 23016 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.728037 23016 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:20.695653 22635 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.707331 23103 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:20.738049 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58331.5us
I20260709 12:16:20.677399 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.739081 13262 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:20.715696 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3384) took 1680 ms. Trace:
I20260709 12:16:20.679757 22734 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.724035 22530 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.702674 23036 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.695955 23150 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:20.743830 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:20.749004 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:20.749969 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:20.750579 12640 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:20.750713 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3343) took 1390 ms. Trace:
I20260709 12:16:20.751200 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.751444 12490 rpcz_store.cc:276] 0709 12:16:19.360429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.447928 (+ 87499us) service_pool.cc:224] Handling call
0709 12:16:19.448241 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:19.909328 (+461087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:20.750334 (+841006us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:20.750615 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944712448}
I20260709 12:16:20.751935 12645 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:20.757246 23059 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:20.757892 23059 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:20.758322 23059 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:20.759320 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:20.760533 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:20.761381 12645 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:20.762223 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 75) took 1607 ms. Trace:
I20260709 12:16:20.762928 12496 rpcz_store.cc:276] 0709 12:16:19.146467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.146643 (+   176us) service_pool.cc:224] Handling call
0709 12:16:20.753971 (+1607328us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:20.754131 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1906129664}
I20260709 12:16:20.771749 12638 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:16:20.758385 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:20.740216 12644 rpcz_store.cc:276] 0709 12:16:19.035107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.036321 (+  1214us) service_pool.cc:224] Handling call
0709 12:16:19.036680 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:19.073480 (+ 36800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:19.073513 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:19.097227 (+ 23714us) log.cc:844] Serialized 26 byte log entry
0709 12:16:19.538153 (+440926us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:19.647905 (+109752us) log.cc:844] Serialized 14 byte log entry
0709 12:16:20.002220 (+354315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:20.002356 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:20.617263 (+614907us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:20.715217 (+ 97954us) raft_consensus.cc:1685] finished
0709 12:16:20.715357 (+   140us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:20.715599 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72703,"prepare.queue_time_us":247184,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":91384,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":17468,"spinlock_wait_cycles":9131392,"thread_start_us":764739,"threads_started":3,"wal-append.queue_time_us":459364}
I20260709 12:16:20.677901 22497 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.726570 22956 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.786069 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:20.786828 13262 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.722842 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:20.758329 22687 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 32.12, Last appended by leader: 12, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.795991 22687 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:20.796286 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:20.797785 12637 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:16:20.798702 23156 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.799237 23156 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.799907 13262 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:20.801151 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:20.742520 22919 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:20.801467 23156 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.802212 23156 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:20.802681 23156 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:20.803273 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.803918 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:20.782205 22959 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.804711 22959 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.805198 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:20.803546 23156 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.806053 23016 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:20.806493 23016 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:20.806851 23016 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.807276 22959 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.782559 22591 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:16:20.811889 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4407) took 1830 ms (client timeout 2268 ms). Trace:
W20260709 12:16:20.812508 12491 rpcz_store.cc:269] 0709 12:16:18.981203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:18.982853 (+  1650us) service_pool.cc:224] Handling call
0709 12:16:20.811651 (+1828798us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:20.811811 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2010346368}
W20260709 12:16:20.819631 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4098) took 1668 ms (client timeout 2221 ms). Trace:
I20260709 12:16:20.820166 22920 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:20.820784 22920 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.821786 22880 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:20.822283 12641 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:20.822400 22880 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.824846 23062 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.825655 23062 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:20.827287 23146 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.656s	user 0.002s	sys 0.001s
I20260709 12:16:20.829191 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1525) took 1472 ms. Trace:
I20260709 12:16:20.812036 22919 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.829552 22886 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.832891 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:20.833835 12567 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:20.834403 12565 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:16:20.834275 22919 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.833261 22886 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.838327 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.802680 23036 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:20.841310 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:20.820186 12642 rpcz_store.cc:269] 0709 12:16:19.150886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.152547 (+  1661us) service_pool.cc:224] Handling call
0709 12:16:20.819374 (+1666827us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:20.819558 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3416791424}
I20260709 12:16:20.848569 12493 rpcz_store.cc:276] 0709 12:16:19.356531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.356816 (+   285us) service_pool.cc:224] Handling call
0709 12:16:19.357131 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:19.951891 (+594760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:20.828843 (+876952us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:20.829101 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1153904384}
W20260709 12:16:20.853397 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 439715us
I20260709 12:16:20.824980 22880 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:20.829381 22833 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:20.873170 22833 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.874316 23016 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.876240 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.888550 23016 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.892284 22530 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:20.893507 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:20.885198 22918 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.014s
W20260709 12:16:20.900893 23155 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:16:20.862928 22920 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.903647 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:20.904422 13258 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:20.914170 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47045.2us
I20260709 12:16:20.858224 12638 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:16:20.863550 13257 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:16:20.866884 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:20.918507 13258 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:20.924312 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:20.862946 22886 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.926196 22833 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.927656 22530 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:20.928401 22530 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:20.915014 22734 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.934264 22497 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.867359 22919 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:20.935238 12644 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:20.929538 22530 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.938396 22734 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.783211 23118 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.878036 23058 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:20.946168 23059 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:20.948527 23118 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:20.954075 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:20.909353 22635 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:20.964679 23150 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:21.002691 23102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.292s	user 0.007s	sys 0.014s
W20260709 12:16:21.011842 23102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.312s	user 0.007s	sys 0.014s
I20260709 12:16:21.010231 23059 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.017680 23150 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:21.104403 12646 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:21.120355 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:21.122131 13254 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:21.128834 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.146735 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.148074 12493 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:21.153672 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.154932 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:21.160921 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1464) took 2210 ms. Trace:
I20260709 12:16:21.187302 12639 rpcz_store.cc:276] 0709 12:16:18.950648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:18.996902 (+ 46254us) service_pool.cc:224] Handling call
0709 12:16:18.997298 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:19.829386 (+832088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:21.160103 (+1330717us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:21.160601 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1750841728}
I20260709 12:16:21.137137 22635 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:21.192839 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.164551 12568 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:21.167927 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.194520 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.197964 22919 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:21.198769 12572 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:21.162397 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.206470 13254 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:21.216344 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.222846 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4346) took 1304 ms. Trace:
I20260709 12:16:21.226837 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.209468 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4365) took 1033 ms. Trace:
I20260709 12:16:21.241812 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.213907 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.227404 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.269558 12568 rpcz_store.cc:276] 0709 12:16:19.918563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.918798 (+   235us) service_pool.cc:224] Handling call
0709 12:16:19.919158 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:21.200758 (+1281600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:21.222319 (+ 21561us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:21.222738 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2861616896}
I20260709 12:16:21.238538 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:21.295463 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.201611 22497 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:21.223217 22502 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:21.278081 12569 rpcz_store.cc:276] 0709 12:16:20.176371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:20.177739 (+  1368us) service_pool.cc:224] Handling call
0709 12:16:21.209027 (+1031288us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:21.209260 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185183488}
I20260709 12:16:21.223647 22606 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:21.237672 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50593.8us
W20260709 12:16:21.222005 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56453.5us
I20260709 12:16:21.338718 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:21.352577 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76985.3us
W20260709 12:16:21.359009 22956 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.012s	sys 0.027s
W20260709 12:16:21.371469 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.010s	sys 0.014s
I20260709 12:16:21.365816 12567 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:21.368537 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:21.371896 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:21.382102 22585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.021s	sys 0.037s
I20260709 12:16:21.382493 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.382798 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.383632 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:21.383880 12563 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
W20260709 12:16:21.373756 22956 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.013s	sys 0.027s
I20260709 12:16:21.380621 12564 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
W20260709 12:16:21.384382 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.011s	sys 0.015s
I20260709 12:16:21.385182 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:21.388875 12572 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:21.372390 22875 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:21.214270 12389 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 26 to 32. New cstate: current_term: 32 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:21.398183 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.399137 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.405069 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:21.406713 23139 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.233s	user 0.003s	sys 0.000s
I20260709 12:16:21.409547 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4217) took 1637 ms. Trace:
I20260709 12:16:21.410327 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:21.410049 12571 rpcz_store.cc:276] 0709 12:16:19.772412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.772536 (+   124us) service_pool.cc:224] Handling call
0709 12:16:19.772827 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:19.789929 (+ 17102us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:19.789972 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:19.790600 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 12:16:19.997020 (+206420us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:19.997144 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:19.997233 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:21.405423 (+1408190us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:21.408863 (+  3440us) raft_consensus.cc:1685] finished
0709 12:16:21.408889 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:21.409323 (+   434us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":6226,"spinlock_wait_cycles":107938560,"thread_start_us":205928,"threads_started":1,"wal-append.queue_time_us":327060}
I20260709 12:16:21.413341 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.415488 23062 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.419117 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.203701 22919 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.424571 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.430805 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:21.430122 13254 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 12:16:21.419516 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 508431us
W20260709 12:16:21.431402 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:21.443264 12493 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:21.419780 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.438938 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:21.445587 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:21.450783 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:21.453889 23102 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:21.455291 23102 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:21.448522 12570 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.020s	sys 0.048s
I20260709 12:16:21.467763 12638 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:21.471370 12570 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.021s	sys 0.049s
I20260709 12:16:21.475559 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:21.479106 23048 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:21.482158 12489 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:21.484160 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:21.433979 22606 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.489104 23115 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:21.489774 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.491151 12569 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:16:21.493160 23170 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:21.494290 23170 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:21.496691 22687 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.700s	user 0.016s	sys 0.009s
I20260709 12:16:21.427017 22502 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.460513 22497 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:21.477448 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50571.8us
I20260709 12:16:21.465411 22919 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:21.470762 22875 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.505857 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:21.506045 12568 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:16:21.490751 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:16:21.505362 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:21.511337 12565 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:21.519091 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4348) took 1587 ms. Trace:
I20260709 12:16:21.528540 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:21.534045 12564 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:21.501613 22502 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:21.521898 12563 rpcz_store.cc:276] 0709 12:16:19.928181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:19.928402 (+   221us) service_pool.cc:224] Handling call
0709 12:16:19.928820 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:21.424360 (+1495540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:21.503167 (+ 78807us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:21.515027 (+ 11860us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3345047936}
W20260709 12:16:21.540637 23164 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.005s
I20260709 12:16:21.540935 12640 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:21.545730 23153 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:21.546049 22606 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:21.534138 23062 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:21.550390 23062 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.554986 23102 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:21.557399 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52383.6us
I20260709 12:16:21.539350 12567 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:21.568356 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:21.569176 22502 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:21.517488 22497 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.570065 22502 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:21.573973 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48969.4us
I20260709 12:16:21.574713 23131 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:21.573988 23133 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:21.576969 23018 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:21.577845 23133 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:21.578632 23133 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:21.573882 23109 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 12:16:21.578104 22497 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:21.582170 22959 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:21.582964 22959 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.583364 22585 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:16:21.583715 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:21.584220 22585 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 27.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:16:21.585116 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:21.531288 23170 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:21.586026 23036 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:21.594904 23036 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:21.595425 23036 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:21.586107 22959 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.573020 22502 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.553375 23062 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.574088 23165 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.035s
I20260709 12:16:21.610098 23109 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:21.611007 23109 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.611989 22875 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:21.588165 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:21.578250 23018 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:21.589736 23171 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260709 12:16:21.593606 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:21.606696 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47291.5us
I20260709 12:16:21.616465 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
W20260709 12:16:21.620724 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132601us
I20260709 12:16:21.624044 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
W20260709 12:16:21.624778 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:21.615661 22585 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:21.628688 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43771.3us
I20260709 12:16:21.630436 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:21.632874 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:21.635106 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:21.619709 23109 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:21.637089 22959 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:21.652283 23171 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:21.640882 23049 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:21.644809 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.646768 13258 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:16:21.646546 12389 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:21.647858 23165 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:21.649423 13257 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:16:21.650694 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:21.655521 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67582.9us
I20260709 12:16:21.655591 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:21.634661 22585 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:21.661191 22585 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:21.661962 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.663280 23048 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:21.664988 12563 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:21.656698 23171 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:21.666051 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:21.667383 23171 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.668145 23170 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:21.670187 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.671622 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:21.636102 22956 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:21.673482 12566 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:21.658761 23165 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:21.669008 23170 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:16:21.681005 22886 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:16:21.682777 22886 consensus_queue.cc:1243] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:16:21.682910 23177 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.014s
I20260709 12:16:21.688124 23018 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.691165 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.692260 12488 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:21.692629 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.694600 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.690007 23165 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.14, Last appended by leader: 14, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.703261 12389 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:21.704964 22530 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:21.705305 23170 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:21.706176 23170 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.706838 22530 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:16:21.709344 23170 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.711357 22875 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:21.712100 23170 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:21.714609 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.719733 23049 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:21.719823 23109 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:21.720886 23049 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.721001 23109 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.675814 22956 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:21.693866 12489 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 34 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
I20260709 12:16:21.724411 23109 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:21.725802 23115 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:21.726241 23115 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:21.727447 23115 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.729111 23115 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:16:21.746407 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 232) took 61515 ms (client timeout 15000 ms). Trace:
I20260709 12:16:21.751704 12568 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 19. (term mismatch)
I20260709 12:16:21.752640 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.752178 12568 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 18
I20260709 12:16:21.754346 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.756203 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.756956 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.768644 12572 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 13
I20260709 12:16:21.754397 12568 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:16:21.781275 12568 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:21.785514 12564 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 16. Preceding OpId from leader: term: 35 index: 17. (index mismatch)
W20260709 12:16:21.788271 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:21.788456 12565 pending_rounds.cc:85] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 23
I20260709 12:16:21.788897 12565 pending_rounds.cc:107] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 30.24
I20260709 12:16:21.789149 12565 raft_consensus.cc:2889] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:21.792480 23062 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:21.791172 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.793301 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.793669 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.794679 12493 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 30 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:16:21.794900 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.793363 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.796218 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.746937 22956 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.797617 22956 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:21.798103 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:21.794593 12491 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:21.800191 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:21.801478 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:21.802688 23109 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:21.770246 12572 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260709 12:16:21.804517 12572 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:21.805572 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.807116 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.797338 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.705545 12389 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 23 to 25, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 25 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:21.821244 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:21.822594 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.726261 23049 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:21.808740 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47204.3us
I20260709 12:16:21.828907 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.717129 23170 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:21.835206 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 1958) took 1061 ms. Trace:
I20260709 12:16:21.839430 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.795498 23048 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:16:21.867723 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:21.839437 12570 rpcz_store.cc:276] 0709 12:16:20.784189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:20.821194 (+ 37005us) service_pool.cc:224] Handling call
0709 12:16:20.842230 (+ 21036us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:20.842571 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:16:21.532711 (+690140us) log.cc:844] Serialized 14 byte log entry
0709 12:16:21.676190 (+143479us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:21.679930 (+  3740us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:21.760642 (+ 80712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:21.787134 (+ 26492us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:21.829467 (+ 42333us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":483,"prepare.queue_time_us":625323,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":4911,"raft.queue_time_us":92,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":2522,"thread_start_us":730273,"threads_started":2,"wal-append.queue_time_us":176453}
I20260709 12:16:21.844812 23049 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:21.870658 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:21.870942 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85464.9us
I20260709 12:16:21.878327 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:21.878527 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.853595 23109 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:16:21.885399 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:21.886814 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:21.889016 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:21.891040 23048 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:21.891638 23048 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:21.892377 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:16:21.893373 12564 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:16:21.905074 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:21.907927 12563 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:21.911170 12642 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:16:21.903030 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.930694 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:21.943647 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:21.903597 23152 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:21.940328 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.950007 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:21.953639 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115122us
I20260709 12:16:21.955556 12638 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:16:21.931715 23049 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:21.958493 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:21.956480 12639 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:21.943322 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96116.4us
W20260709 12:16:21.933014 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 260286us
I20260709 12:16:21.941169 12389 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 28 to 34. New cstate: current_term: 34 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:21.892717 23048 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:21.903918 23153 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:16:21.979799 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:21.982753 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.990312 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:21.974174 23102 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:21.995007 12646 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:16:21.999951 22920 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.108s
I20260709 12:16:21.999570 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:16:21.748862 12387 rpcz_store.cc:269] 0709 12:15:20.254435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:29.571646 (+9317211us) service_pool.cc:224] Handling call
0709 12:16:21.745784 (+52174138us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:16:13.612300 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:16:13.613076 (+   776us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:16:13.613147 (+    71us) write_op.cc:435] Acquired schema lock
0709 12:16:13.613169 (+    22us) tablet.cc:662] Decoding operations
0709 12:16:13.616932 (+  3763us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:16:13.617051 (+   119us) write_op.cc:641] Partition lock acquired for write op
0709 12:16:13.617079 (+    28us) tablet.cc:685] Acquiring locks for 18 operations
0709 12:16:13.674934 (+ 57855us) tablet.cc:701] Row locks acquired
0709 12:16:13.674989 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:16:13.675155 (+   166us) write_op.cc:270] Start()
0709 12:16:13.675324 (+   169us) write_op.cc:276] Timestamp: P: 1783599373675089 usec, L: 0
0709 12:16:13.675368 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:16:13.676170 (+   802us) log.cc:844] Serialized 7125 byte log entry
0709 12:16:14.334883 (+658713us) op_driver.cc:464] REPLICATION: finished
0709 12:16:14.665858 (+330975us) write_op.cc:301] APPLY: starting
0709 12:16:14.665981 (+   123us) tablet.cc:1367] starting BulkCheckPresence
0709 12:16:15.266000 (+600019us) tablet.cc:1370] finished BulkCheckPresence
0709 12:16:15.266119 (+   119us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:15.629669 (+363550us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:15.630062 (+   393us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=0,mrs_lookups=0
0709 12:16:15.630117 (+    55us) write_op.cc:312] APPLY: finished
0709 12:16:15.630576 (+   459us) log.cc:844] Serialized 164 byte log entry
0709 12:16:15.681140 (+ 50564us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:15.681390 (+   250us) write_op.cc:454] Released schema lock
0709 12:16:15.681494 (+   104us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:15.682033 (+   539us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:15.682062 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3852,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":259776,"apply.run_cpu_time_us":43873,"apply.run_wall_time_us":1016184,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79295,"num_ops":18,"prepare.queue_time_us":294373,"prepare.run_cpu_time_us":54860,"prepare.run_wall_time_us":209504,"raft.queue_time_us":105665,"raft.run_cpu_time_us":14825,"raft.run_wall_time_us":278551,"replication_time_us":614458,"thread_start_us":573876,"threads_started":5,"wal-append.queue_time_us":250743,"wal-append.run_cpu_time_us":32107,"wal-append.run_wall_time_us":2348015}]]}
I20260709 12:16:21.989820 12389 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 18 to 27, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 27 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:21.986863 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:22.031951 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:22.013219 23180 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:16:22.035609 23048 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:21.971887 23036 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.042781 12493 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
W20260709 12:16:22.044754 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 234) took 20021 ms (client timeout 15000 ms). Trace:
W20260709 12:16:22.059270 12387 rpcz_store.cc:269] 0709 12:16:02.042913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:22.044467 (+20001554us) service_pool.cc:211] Skipping call since client already timed out
0709 12:16:22.044657 (+   190us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:16:22.056991 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 160023us
I20260709 12:16:22.018635 23102 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:22.069923 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:21.984447 23049 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.024012 23131 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:22.079241 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:22.078329 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:22.070761 23102 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:22.104264 23170 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:22.094105 12389 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:22.108165 12389 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 31 to 33, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 33 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:22.112074 23160 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:22.083505 23153 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:22.054540 23133 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.112890 23160 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.094551 22886 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:22.094898 23152 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:22.091735 23131 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:22.171586 23131 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.174108 12389 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:22.152873 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:22.156288 22821 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:22.162717 22585 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.500s	user 0.009s	sys 0.006s
I20260709 12:16:22.163327 23160 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:22.163862 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:22.164975 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:22.166285 12495 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: [30.16-30.16]
I20260709 12:16:22.166136 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:22.166870 12490 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:22.167259 23133 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.146219 23152 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:22.192018 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:22.192577 23152 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:22.193090 12495 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 17.
W20260709 12:16:22.176977 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:22.178818 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:22.199226 23118 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.078s
W20260709 12:16:22.202008 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:22.174167 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:22.228557 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:22.181897 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:22.190200 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 223635us
I20260709 12:16:22.191366 23196 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:22.187300 12493 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:22.234516 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:22.233465 23196 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.237035 23018 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:22.237370 23196 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.237831 23018 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:22.238641 23018 consensus_queue.cc:1243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 30.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:22.241665 23049 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.249857 22821 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:22.252740 22821 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:22.259837 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:22.260510 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:22.262058 23185 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:16:22.261824 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:22.266615 23036 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.267364 23036 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:22.267879 23036 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.269809 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:22.259827 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:22.282701 12389 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:22.205706 22585 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:22.291564 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:22.291949 23184 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.473s	user 0.002s	sys 0.000s
I20260709 12:16:22.297999 12563 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:22.301005 23186 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.470s	user 0.003s	sys 0.000s
I20260709 12:16:22.303079 22964 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:22.237421 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:22.304885 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67584.4us
I20260709 12:16:22.306488 12493 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:22.307667 23165 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:22.303745 22964 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.311683 22964 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.312552 22502 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:22.312996 22502 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:22.313316 22502 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:22.284818 12389 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 27 to 30. New cstate: current_term: 30 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:22.320118 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:22.321038 12564 raft_consensus.cc:2368] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34.
I20260709 12:16:22.328843 13257 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
W20260709 12:16:22.356521 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125314us
I20260709 12:16:22.357743 12389 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 28. New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:22.368891 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:22.367055 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:22.318497 22585 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:22.397980 12460 leader_election.cc:400] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34.
I20260709 12:16:22.398464 12460 leader_election.cc:403] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:16:22.397986 12389 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 27 to 32, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 32 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:16:22.408594 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:22.403005 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:22.417917 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60563.2us
I20260709 12:16:22.217010 23153 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:22.442898 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:22.445544 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:22.450896 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 123676us
I20260709 12:16:22.452132 12491 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:22.457325 12496 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
W20260709 12:16:22.463289 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106512us
I20260709 12:16:22.426222 23177 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:22.388751 22687 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.088s
I20260709 12:16:22.438781 22964 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:22.484447 12492 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 31 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:16:22.441565 23036 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.487968 12389 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 26 to 30. New cstate: current_term: 30 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:22.502674 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:22.499495 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:22.534816 12389 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 28, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:16:22.544241 23193 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:16:22.547369 23200 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:22.552002 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69820.8us
I20260709 12:16:22.496377 22956 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 12:16:22.550374 23109 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:22.572969 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:22.555503 23131 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:22.582579 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99532us
W20260709 12:16:22.575071 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88914.1us
I20260709 12:16:22.552134 12389 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 27, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 27 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:16:22.607004 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:22.609279 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 148620us
I20260709 12:16:22.548080 23200 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.615707 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:22.632812 22734 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:22.637756 22919 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:22.610857 23191 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:16:22.601406 23131 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:22.644461 23105 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:22.654445 23153 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.654825 22964 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:22.657524 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88217us
I20260709 12:16:22.632332 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:22.667366 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69418.8us
I20260709 12:16:22.672596 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:22.673563 12493 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:16:22.673939 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43610.7us
I20260709 12:16:22.675001 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:22.667825 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 304502us
I20260709 12:16:22.677320 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:22.687417 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:22.691249 23036 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:22.695245 23036 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:22.695729 23036 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:22.672062 23105 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:22.680335 22687 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:22.683744 23196 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:22.681950 22919 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:22.714851 12489 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:22.691537 22734 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:16:22.716316 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 258506us
I20260709 12:16:22.703316 23049 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:22.729936 23049 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:22.730413 23049 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:22.729952 22821 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:22.735651 22821 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.738878 23131 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:22.708833 23200 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:22.712795 23202 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:16:22.707468 23018 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:22.703984 23105 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:22.737429 23102 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.603s	user 0.006s	sys 0.003s
W20260709 12:16:22.760617 22886 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.612s	user 0.007s	sys 0.002s
I20260709 12:16:22.761219 22886 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:22.773937 22964 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:22.766502 23204 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:16:22.767940 23171 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:22.740475 22919 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:22.770447 23183 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.558s	user 0.001s	sys 0.002s
I20260709 12:16:22.770853 23187 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:22.768111 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:22.802127 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:22.791096 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118600us
I20260709 12:16:22.704728 23197 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:16:22.804783 22606 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:22.780539 23102 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.805574 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:22.806599 12563 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:22.807987 12563 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 17
I20260709 12:16:22.808265 12563 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 30.18
I20260709 12:16:22.808514 12563 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:22.750152 23018 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:22.809912 22821 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.810063 23018 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:22.815734 22502 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.008s	sys 0.004s
I20260709 12:16:22.816269 22502 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.820523 12571 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:16:22.821798 23187 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.823799 23109 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34.
I20260709 12:16:22.824159 23187 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.825132 12572 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:16:22.821431 23049 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.798667 22497 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
W20260709 12:16:22.826308 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71723.8us
I20260709 12:16:22.827196 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:22.828191 23049 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:22.828750 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:22.829747 23062 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:22.830588 23062 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:22.831037 23062 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:22.831450 23062 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:22.799379 23171 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:22.834463 23171 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.835942 23171 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:22.820683 23018 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.804858 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:22.833403 23205 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.183s	user 0.004s	sys 0.000s
I20260709 12:16:22.845221 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4382) took 1128 ms. Trace:
I20260709 12:16:22.836411 23105 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.856724 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:22.791807 23131 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:22.867173 12566 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:22.849154 12569 rpcz_store.cc:276] 0709 12:16:21.716738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:21.716898 (+   160us) service_pool.cc:224] Handling call
0709 12:16:21.717320 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:21.717937 (+   617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:21.717976 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:21.718750 (+   774us) log.cc:844] Serialized 52 byte log entry
0709 12:16:21.900058 (+181308us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:21.954784 (+ 54726us) log.cc:844] Serialized 14 byte log entry
0709 12:16:21.958603 (+  3819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:21.958683 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:22.647072 (+688389us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:22.838499 (+191427us) raft_consensus.cc:1685] finished
0709 12:16:22.845011 (+  6512us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:22.845162 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3506,"prepare.queue_time_us":45272,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":6252,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":9757,"spinlock_wait_cycles":8322816,"thread_start_us":179333,"threads_started":1,"wal-append.queue_time_us":309095}
I20260709 12:16:22.878249 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:22.880918 13254 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:22.883106 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:22.885221 22734 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:22.887796 23211 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:16:22.888294 22734 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.891096 23137 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:22.891822 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110291us
W20260709 12:16:22.892047 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87626.2us
I20260709 12:16:22.893371 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:22.893679 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:22.894795 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:22.903473 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:22.906221 12643 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:22.805617 22606 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:22.934504 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59126.7us
W20260709 12:16:22.935606 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107025us
W20260709 12:16:22.942662 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59676.1us
I20260709 12:16:22.900656 23048 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:22.891855 23137 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.947078 23221 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:16:22.950062 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55355.6us
I20260709 12:16:22.951489 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:22.952301 22606 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:22.952883 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:22.953882 23048 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:22.954294 23196 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:22.956229 23217 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:22.956750 23048 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:22.907990 23170 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:16:22.957441 23058 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:22.907747 22734 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:22.918948 23018 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:22.922415 22502 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:22.911096 23218 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:22.953043 22606 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:22.908900 23049 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:22.878331 23131 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:22.906441 23103 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:16:22.972503 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:22.949365 23137 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:22.974828 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:22.978312 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:22.917523 23062 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.945335 22886 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:22.953920 23197 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:22.989200 23219 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:22.990428 23131 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:22.990834 22886 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.968616 22606 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:22.980201 22920 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:22.992388 23152 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.799s	user 0.010s	sys 0.005s
I20260709 12:16:22.950484 23036 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:22.969038 23218 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:23.003649 23105 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:23.030351 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4120) took 1084 ms. Trace:
I20260709 12:16:23.030881 12643 rpcz_store.cc:276] 0709 12:16:21.949028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:21.954889 (+  5861us) service_pool.cc:224] Handling call
0709 12:16:21.981916 (+ 27027us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:22.931331 (+949415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:23.029988 (+ 98657us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:23.030244 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1969225600}
I20260709 12:16:22.991395 23062 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:23.014446 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4440) took 1217 ms. Trace:
W20260709 12:16:23.034597 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90916.4us
I20260709 12:16:23.035494 12489 rpcz_store.cc:276] 0709 12:16:21.791952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:21.799822 (+  7870us) service_pool.cc:224] Handling call
0709 12:16:23.008913 (+1209091us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:23.009113 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2061946624}
I20260709 12:16:23.014853 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.015040 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.004274 23216 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:23.038362 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:23.037662 12569 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:23.017661 22606 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:23.060389 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82928.1us
I20260709 12:16:23.061746 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:23.066123 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45433.8us
I20260709 12:16:23.020318 23152 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:23.100137 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 217853us
I20260709 12:16:23.106068 12638 raft_consensus.cc:2368] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34.
I20260709 12:16:23.086410 23103 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:23.104270 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:23.108222 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:23.129369 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:23.132079 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:23.131963 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.109035 23200 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:23.103590 23062 raft_consensus.cc:2764] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:16:23.148177 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:23.148901 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.104753 23049 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:23.149981 12565 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:23.167652 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:23.171360 12570 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:23.174209 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.181816 23197 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:23.180845 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:23.191744 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 298143us
I20260709 12:16:23.193229 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:23.144586 23105 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:23.147512 23200 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:23.241209 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:23.257820 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77840.4us
I20260709 12:16:23.259907 22734 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:23.260434 22734 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:23.260793 22734 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:23.124585 23218 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:23.263688 23215 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:23.264427 23215 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:23.264843 23215 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:16:23.265481 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:23.265580 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84750.2us
I20260709 12:16:23.266590 23105 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:23.269109 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207544us
I20260709 12:16:23.269718 23160 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:23.270300 23160 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.270689 23170 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:23.271301 23170 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:23.273689 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:23.274417 12638 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:23.276726 23102 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:23.282321 23153 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:23.283198 23153 consensus_queue.cc:1243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 34.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:16:23.283787 23153 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:23.284102 23153 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:23.281325 12461 leader_election.cc:400] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34.
I20260709 12:16:23.286375 23058 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:23.287868 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95245.7us
I20260709 12:16:23.289152 23218 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:23.289521 12643 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:23.269228 12566 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:23.270319 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:23.289547 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:23.295886 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:23.311957 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:23.313838 12638 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 30 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:16:23.326232 12642 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: [31.16-31.16]
I20260709 12:16:23.143816 23103 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.239447 23200 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:23.189280 23131 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.127s
I20260709 12:16:23.343497 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:23.346484 12567 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:16:23.344695 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.343643 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:23.346621 12646 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:23.335148 23225 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:16:23.299382 23170 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:23.354760 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 127732us
I20260709 12:16:23.351873 12638 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:16:23.354224 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:23.197690 23049 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:23.364725 12570 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:23.199903 23228 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.307s
I20260709 12:16:23.345399 22886 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:23.351213 23062 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:23.374866 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.356289 23036 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:23.327533 23153 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.17, Last appended by leader: 17, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.368301 23048 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
W20260709 12:16:23.393283 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123103us
I20260709 12:16:23.419667 12567 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:16:23.431972 23049 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:23.437932 23152 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:23.439483 22606 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.441409 23103 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:23.443116 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:23.445139 22606 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:23.454694 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 163128us
I20260709 12:16:23.469988 23197 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:23.470729 23197 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.453655 23105 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:23.464102 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:23.372455 23165 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:16:23.475415 23165 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:23.446421 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:23.453434 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:23.479091 23036 raft_consensus.cc:2764] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election decision vote started in defunct term 29: won
I20260709 12:16:23.469802 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.480350 23036 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:23.481707 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.473333 23197 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:23.475142 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:23.491161 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:23.492769 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:23.501809 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.505211 23048 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:23.505950 23048 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:23.506516 23197 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:23.506002 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:23.508476 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:23.475905 23165 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:23.506844 23048 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:23.513595 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:23.522029 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:23.525266 12569 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:23.522915 12640 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:16:23.532145 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:23.528185 23165 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:23.541870 23170 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:23.538765 23219 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:23.513144 22606 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:23.552352 22606 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:23.554425 23217 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:23.554979 23217 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:23.555403 23217 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:23.555882 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:23.556713 23062 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:23.558686 23234 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:23.559334 23234 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:23.559788 23234 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:23.560175 23215 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:23.513803 23160 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:23.445611 22734 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.584050 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:23.584820 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:23.583994 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46273.8us
I20260709 12:16:23.590140 12567 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:16:23.532797 23133 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:16:23.580456 23171 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.744s	user 0.011s	sys 0.006s
W20260709 12:16:23.655082 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 186200us
I20260709 12:16:23.579138 23221 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:23.590619 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:23.534523 23105 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:23.762617 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:23.766435 12643 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:23.770437 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:23.762125 23197 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:23.814675 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:23.778153 23219 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:23.838732 23218 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.527s	user 0.013s	sys 0.004s
I20260709 12:16:23.839119 23115 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:23.849249 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97892.7us
I20260709 12:16:23.769729 23105 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.811997 23171 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:23.776365 23221 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.823626 23133 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:23.829708 23152 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:23.822896 23197 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:23.896935 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 306390us
I20260709 12:16:23.898346 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:23.832593 23219 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:23.902491 23246 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:16:23.876410 23109 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:23.892544 23133 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:23.907886 23248 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:16:23.862913 23115 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:23.872113 23171 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:23.942907 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 171904us
I20260709 12:16:23.912164 23109 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.914136 23133 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:23.944341 23200 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.583s	user 0.012s	sys 0.006s
I20260709 12:16:23.966974 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:23.943687 23218 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:23.898741 23234 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.982981 23218 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:23.987141 23253 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:16:23.988910 23257 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:23.989737 23257 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:23.990914 23257 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 32.12, Last appended by leader: 12, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.992633 23165 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.996388 23006 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:23.997272 23218 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:23.998718 23234 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:23.983906 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.001251 23229 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:23.977018 23200 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.002452 23131 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:24.002988 23131 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:24.010877 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:23.997011 23006 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.004771 22734 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:24.020100 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76596.8us
I20260709 12:16:24.004148 23131 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:23.977922 23058 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:16:24.023870 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.044262 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.045190 12638 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:24.041972 22502 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:24.058182 23228 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:24.059026 23228 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.062317 23228 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.055387 23219 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.055094 23131 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:24.067170 23006 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.072716 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.076169 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.078239 23197 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.078557 22585 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:24.079704 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:24.078837 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 521154us
I20260709 12:16:24.085444 23115 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.089524 23105 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:24.086833 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.109931 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:24.090559 23048 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.003s
I20260709 12:16:24.146009 23219 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.146936 23219 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:24.148136 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93048.8us
I20260709 12:16:24.088419 23109 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:24.155545 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107198us
I20260709 12:16:24.157511 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.162894 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.163673 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
W20260709 12:16:24.163708 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77000.6us
I20260709 12:16:24.055809 23217 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.153707 23036 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:24.180476 23219 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.181142 22821 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:24.181846 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.184278 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.191776 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.151436 23048 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.168861 23103 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:16:24.185745 23036 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:24.201409 23233 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:24.202008 22956 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:24.204598 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:24.208702 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.190395 23234 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:24.217319 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.218120 12641 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:24.218880 23197 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:24.219436 23197 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.208052 23152 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:24.222487 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.223699 23237 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:24.206300 23036 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:24.199282 23109 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:24.239853 23109 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.240522 23272 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.241123 23272 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.241849 23262 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:24.242167 23232 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.976s	user 0.003s	sys 0.002s
I20260709 12:16:24.242560 23262 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.215483 23256 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:24.248445 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4273) took 1364 ms. Trace:
I20260709 12:16:24.229382 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.205125 23269 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:16:24.230914 23196 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:24.232487 23237 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:24.234036 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.235045 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:24.235793 23273 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:24.215945 23234 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:24.239346 23247 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.283s	user 0.001s	sys 0.002s
I20260709 12:16:24.247336 23256 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.270289 23237 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.253515 23196 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:24.270516 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65477.7us
I20260709 12:16:24.269870 23273 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.278046 23196 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.279224 23196 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:24.279925 23273 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.237520 22956 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:24.250948 12572 rpcz_store.cc:276] 0709 12:16:22.884287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:22.884418 (+   131us) service_pool.cc:224] Handling call
0709 12:16:22.901223 (+ 16805us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:22.936542 (+ 35319us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:22.936575 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:22.965236 (+ 28661us) log.cc:844] Serialized 26 byte log entry
0709 12:16:23.051546 (+ 86310us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:23.091046 (+ 39500us) log.cc:844] Serialized 14 byte log entry
0709 12:16:23.091351 (+   305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:23.091443 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:23.278801 (+187358us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:16:23.459613 (+180812us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b1800158860. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:24.241447 (+781834us) spinlock_profiling.cc:243] Waited 88.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:24.248228 (+  6781us) raft_consensus.cc:1685] finished
0709 12:16:24.248250 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:24.248386 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39425,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":39624,"raft.run_cpu_time_us":11519,"raft.run_wall_time_us":348485,"spinlock_wait_cycles":294165888,"thread_start_us":32590,"threads_started":1,"wal-append.queue_time_us":180267}
W20260709 12:16:24.252532 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68355.9us
I20260709 12:16:24.272883 23269 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:16:24.247030 23231 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.955s	user 0.008s	sys 0.000s
I20260709 12:16:24.287601 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.248862 23267 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:24.288978 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.277467 23256 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.289460 22956 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.289247 12492 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:24.289407 23267 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.294160 23109 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.255950 23260 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:24.302707 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.246743 23217 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.248657 22734 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:24.328500 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.329485 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.312546 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.245523 23048 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.245519 23262 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.330428 12489 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:24.299064 23269 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:24.333150 23267 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.275733 23234 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:24.334046 12495 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 32 because replica has last-logged OpId of term: 31 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:16:24.337553 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.337420 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.338290 12491 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:24.281050 23197 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.343443 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.344383 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1517) took 1502 ms. Trace:
I20260709 12:16:24.344977 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:24.380178 12646 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:24.351332 22606 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:24.374244 23236 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.007s	user 0.000s	sys 0.007s
I20260709 12:16:24.352459 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.353293 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.353466 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.354063 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.354158 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.354187 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:24.355417 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:24.358950 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.363231 23255 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:24.367223 23250 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.536s	user 0.002s	sys 0.000s
I20260709 12:16:24.370841 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.371519 23221 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.372265 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.459990 12572 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:16:24.372958 23200 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.373420 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.297724 23260 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.339171 23269 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:24.408141 12639 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:24.418471 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.464946 12497 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:24.389607 22734 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:24.377998 13258 rpcz_store.cc:276] 0709 12:16:22.842381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:22.842549 (+   168us) service_pool.cc:224] Handling call
0709 12:16:22.860367 (+ 17818us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:22.895856 (+ 35489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:22.895890 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:22.943399 (+ 47509us) log.cc:844] Serialized 26 byte log entry
0709 12:16:23.049647 (+106248us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:23.049701 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:23.049794 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:24.246238 (+1196444us) spinlock_profiling.cc:243] Waited 39 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:24.333810 (+ 87572us) raft_consensus.cc:1685] finished
0709 12:16:24.333848 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:24.344274 (+ 10426us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9672,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":37984,"spinlock_wait_cycles":89772288,"thread_start_us":68533,"threads_started":1,"wal-append.queue_time_us":172380}
I20260709 12:16:24.444191 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.444293 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.446981 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4135) took 1153 ms. Trace:
I20260709 12:16:24.476697 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.390643 12490 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:24.387846 12640 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:24.476605 12643 rpcz_store.cc:276] 0709 12:16:23.293315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:23.316089 (+ 22774us) service_pool.cc:224] Handling call
0709 12:16:23.343574 (+ 27485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:23.774606 (+431032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:24.418850 (+644244us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:24.446919 (+ 28069us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":972176768}
W20260709 12:16:24.452206 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93340.6us
I20260709 12:16:24.477423 12491 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:24.478982 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.479592 12572 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:24.456382 23261 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:24.483659 23261 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.484097 23261 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:24.485055 23189 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:24.477970 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.481237 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.479996 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:24.495007 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.496001 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.497097 23133 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.498133 22734 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.459396 23263 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:24.474802 23006 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:24.499850 23263 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.500139 23006 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.501322 23215 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.459232 23255 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.502992 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.505708 23189 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.508061 23189 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.510306 23255 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.497375 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.515801 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:24.516758 12637 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:16:24.516958 23170 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:24.517087 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.519141 23263 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.522523 23006 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:24.522441 23260 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.524521 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:24.529152 12571 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:24.529968 12497 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:24.531121 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.533115 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.533843 12567 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:24.535161 12571 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:24.535708 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.535825 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.536279 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.537264 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.537355 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.540068 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.517402 23170 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.541736 23170 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:24.542743 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:24.543543 12638 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:24.544695 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.546018 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.545064 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.548146 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.548840 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.549870 22585 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:24.548885 12495 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:24.555038 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.556788 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.557971 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.558533 12496 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:24.559267 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.560940 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.561530 12494 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:24.563023 23165 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.565110 23153 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:24.565759 23171 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.567607 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.568738 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.569517 12488 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:16:24.559926 12496 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:24.472576 12489 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:24.572356 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.573972 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.573138 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.575946 12489 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:24.577710 12497 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:24.579353 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.582552 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.583427 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:24.577203 12493 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:24.584468 23264 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:24.586136 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72395.5us
I20260709 12:16:24.580291 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:24.592056 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.590207 12494 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:24.593253 12497 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:16:24.594763 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:24.598417 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.601959 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.593235 12495 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 13
I20260709 12:16:24.603150 12495 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260709 12:16:24.603470 12495 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:24.599778 23062 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:16:24.618269 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46016.3us
I20260709 12:16:24.619867 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.620169 22606 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:24.620682 23197 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:24.620882 22606 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.620277 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:24.622313 13258 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:24.623185 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.610679 23230 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:24.613965 22585 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:24.615469 12489 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:24.627549 12493 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: [28.15-30.16]
I20260709 12:16:24.635736 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:24.637352 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:24.574810 23272 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.640118 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:24.639338 12490 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:24.649183 23255 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.649986 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.621518 22606 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:24.638025 23160 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:24.649996 23255 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.652937 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.653916 23229 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:24.654434 23255 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.655134 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.655663 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.656457 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.654453 23229 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:24.657395 23229 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:24.657764 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.659819 12565 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 12. Preceding OpId from leader: term: 33 index: 13. (index mismatch)
I20260709 12:16:24.663559 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.664068 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:24.664429 12497 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:16:24.666528 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:24.667616 23197 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:24.668919 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:24.669391 22606 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.669948 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:24.668308 23197 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.673020 23197 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.669436 12637 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:24.624833 23269 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.674913 12639 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:24.675367 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.676107 23269 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:24.575376 23036 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.676795 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.678552 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.678898 23036 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.681255 12493 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:24.681488 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.682600 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.682767 12638 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:16:24.683300 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:24.684019 12641 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:24.694814 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:24.685123 12565 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25. Ops: []
I20260709 12:16:24.677500 12497 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:24.695632 12493 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:24.696350 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.698586 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.702380 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:24.698078 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.709733 12637 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: [28.15-30.16]
I20260709 12:16:24.710264 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:24.711513 12490 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:24.722137 12571 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:24.722260 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.722790 12490 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:24.728662 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.727759 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:24.753654 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55695.3us
I20260709 12:16:24.721873 22502 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:24.751245 23137 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:24.717087 23152 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:24.623083 23264 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.750447 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.728813 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:24.748837 22956 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:24.749173 22606 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:24.782119 23268 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:24.753084 23049 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:16:24.752959 23160 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:16:24.724006 23269 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:24.747432 23272 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:24.789517 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60793.1us
I20260709 12:16:24.797147 23137 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:24.765306 23217 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:24.794987 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44663.7us
I20260709 12:16:24.793883 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:24.770910 23058 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:24.765431 23165 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:16:24.795414 23221 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:24.809999 12492 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 35 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 17.
I20260709 12:16:24.810911 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.783357 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.816447 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:24.797984 22956 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:24.797362 23261 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.823873 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.820514 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.826203 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.826206 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:24.839517 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.840484 23058 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:24.858362 23264 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.854341 23221 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:24.859071 23229 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.856873 23058 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 32.13, Last appended by leader: 13, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.860026 23221 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.850476 23272 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.864002 23272 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.820647 23269 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:24.865504 23269 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.866335 23170 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.868535 23170 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.814915 23234 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.868693 12642 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:24.870774 12491 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:24.871127 23234 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:24.872623 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:24.876824 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:24.878078 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.878463 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:24.879576 23255 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:24.880029 23255 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:24.880380 23255 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:24.881141 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:24.829691 23006 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:24.886883 23006 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:24.887315 23006 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:16:24.848907 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65605.6us
I20260709 12:16:24.888029 22821 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:24.888689 22821 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:24.890676 12489 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:24.890980 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.891708 23261 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.891451 22821 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.892916 22821 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:24.887154 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.901736 22956 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.900763 12492 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:24.910025 12564 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:24.912331 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.855103 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:24.913292 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58314.5us
I20260709 12:16:24.914186 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:24.916594 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:24.917234 13258 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:24.920488 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:24.921605 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 291822us
I20260709 12:16:24.921959 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:24.893435 23261 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:24.855500 23197 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:24.854419 23200 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:24.924223 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.861521 23229 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:24.922535 23189 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:24.937376 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:24.944749 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.941026 23189 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:24.952839 23189 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:24.953781 22232 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:24.954550 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.955536 12564 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:24.938308 23049 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:24.958341 23049 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:24.959306 23263 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:16:24.961583 23115 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:24.962093 23115 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:24.928242 23165 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:24.963075 23115 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:24.926788 23261 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:24.964581 23115 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:24.932641 23272 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:24.970187 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:24.975420 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:24.923712 23170 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.044s
I20260709 12:16:24.971122 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:25.054594 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:25.055279 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:24.941915 23197 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:24.967724 23133 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:16:25.064214 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93350.9us
I20260709 12:16:25.065356 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:24.968657 23049 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:24.971060 23261 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:25.058517 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:25.071954 12492 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:16:25.087431 12491 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:25.058393 23234 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:25.082703 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:25.050880 23272 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:25.104494 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.070108 23268 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:25.060571 22821 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:25.105767 23234 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.138546 23229 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:25.079555 23215 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:25.082090 23217 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:25.183794 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79384.2us
I20260709 12:16:25.185298 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:25.186184 23275 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:25.186691 23275 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:25.187076 23275 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:25.187350 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:25.187520 23197 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.183884 23217 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.190585 23217 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.184003 23291 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:25.191431 23272 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.192842 22585 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:25.193358 22585 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:25.193815 22585 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:25.200711 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:25.186934 23285 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.205s	user 0.002s	sys 0.000s
I20260709 12:16:25.205585 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:25.183269 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75313.1us
I20260709 12:16:25.106617 23118 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:25.216326 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:25.108002 23268 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:25.223542 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:25.226842 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.150945 23293 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:25.217388 23291 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:25.215073 22886 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:25.230005 23062 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:25.231040 22502 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:25.214196 23264 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:25.213966 23196 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:25.180675 23049 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.054s
W20260709 12:16:25.243753 12644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.002s	sys 0.027s
W20260709 12:16:25.255957 12644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.003s	sys 0.028s
I20260709 12:16:25.244446 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.206116 22821 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.205044 23229 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:25.249146 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 371769us
I20260709 12:16:25.232865 23230 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:25.253571 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:25.268766 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.274034 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:25.274629 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.275278 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.231753 23165 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:16:25.259181 12563 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:25.255795 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 432483us
I20260709 12:16:25.261022 22886 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:25.282786 12565 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:25.250516 23062 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:25.286055 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 350053us
I20260709 12:16:25.285429 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:25.232434 23103 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.011s	sys 0.040s
I20260709 12:16:25.293134 23293 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 12:16:25.307017 23229 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:25.281872 23233 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:16:25.310770 23103 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.011s	sys 0.040s
I20260709 12:16:25.317240 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:25.289675 23260 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:25.255710 23264 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.331483 23160 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:25.284299 23036 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:16:25.341166 23293 consensus_queue.cc:1243] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:16:25.335947 23260 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:25.408573 23260 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.325685 23255 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.364616 23006 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.231383 23268 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.411859 23006 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.412384 23268 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:25.413772 22886 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:25.414320 22886 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:25.415390 22886 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.416932 22886 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:25.418593 23255 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:25.419354 23305 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
W20260709 12:16:25.420759 23297 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.113s	user 0.002s	sys 0.000s
I20260709 12:16:25.427188 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:25.411864 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4485) took 1089 ms. Trace:
W20260709 12:16:25.430950 23292 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.124s	user 0.001s	sys 0.001s
W20260709 12:16:25.335125 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43913.2us
I20260709 12:16:25.391976 23036 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:25.290341 23171 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:16:25.394403 22821 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.395303 23234 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.395750 23189 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.397022 23264 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:25.346366 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:25.400800 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:25.403882 23296 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:16:25.406061 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4484) took 1084 ms. Trace:
I20260709 12:16:25.349788 23062 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.433676 12570 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:25.444435 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130893us
I20260709 12:16:25.446151 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.448347 23293 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:25.449833 12567 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:25.450866 23260 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:25.450551 22585 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.430151 12496 rpcz_store.cc:276] 0709 12:16:24.322207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:24.599517 (+277310us) service_pool.cc:224] Handling call
0709 12:16:24.610269 (+ 10752us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:24.799741 (+189472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:24.799840 (+    99us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:24.806676 (+  6836us) log.cc:844] Serialized 26 byte log entry
0709 12:16:25.287656 (+480980us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:25.305288 (+ 17632us) log.cc:844] Serialized 14 byte log entry
0709 12:16:25.312896 (+  7608us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:25.411410 (+ 98514us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:25.411475 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:25.411530 (+    55us) raft_consensus.cc:1685] finished
0709 12:16:25.411640 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:25.411795 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16175,"prepare.queue_time_us":17699,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":8702,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":26067,"spinlock_wait_cycles":434610048,"thread_start_us":441152,"threads_started":1,"wal-append.queue_time_us":580095}
I20260709 12:16:25.441331 23160 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:25.453022 12495 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:16:25.443783 23036 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:25.517495 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:25.452649 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:25.448283 23133 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:25.501199 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:25.501991 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:25.526473 12492 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:25.506992 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:25.459764 12488 rpcz_store.cc:276] 0709 12:16:24.321895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:24.599449 (+277554us) service_pool.cc:224] Handling call
0709 12:16:24.609676 (+ 10227us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:24.624980 (+ 15304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:24.625011 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:24.641754 (+ 16743us) log.cc:844] Serialized 26 byte log entry
0709 12:16:25.108909 (+467155us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:25.181059 (+ 72150us) log.cc:844] Serialized 14 byte log entry
0709 12:16:25.208119 (+ 27060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:25.208274 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:25.405716 (+197442us) raft_consensus.cc:1685] finished
0709 12:16:25.405756 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:25.405963 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41529,"prepare.queue_time_us":186,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":55029,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":13451,"thread_start_us":438034,"threads_started":1,"wal-append.queue_time_us":614701}
W20260709 12:16:25.520568 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72524.3us
I20260709 12:16:25.444063 23261 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.512923 23272 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:25.593056 12492 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:25.536986 23264 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:25.543021 23196 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:25.602648 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:25.602762 23314 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.107s	user 0.002s	sys 0.000s
I20260709 12:16:25.511555 23234 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.081s
I20260709 12:16:25.578253 23293 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:16:25.607012 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 260737us
I20260709 12:16:25.543430 23291 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.063s
I20260709 12:16:25.610935 23228 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:25.618148 23160 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.619102 12489 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:25.620306 23275 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.621814 23036 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.623301 23261 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.623526 23036 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:25.597484 23264 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:25.624526 23264 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:25.624399 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:25.627893 22585 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.634032 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.636515 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:25.639243 12567 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
W20260709 12:16:25.633055 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92943.1us
I20260709 12:16:25.645056 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.645308 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.598235 23272 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.658288 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:25.657084 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:25.659466 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:25.664322 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46543.2us
I20260709 12:16:25.665603 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.666620 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.668733 23103 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:25.668823 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.670396 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.671710 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.672890 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.641224 23304 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:25.673918 23233 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:25.675572 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.675626 23272 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.677608 23237 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:25.679545 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.649246 23230 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:25.684362 23322 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:25.684916 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.608661 23196 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.685820 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.691964 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:25.694965 12638 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 12. Preceding OpId from leader: term: 33 index: 13. (index mismatch)
I20260709 12:16:25.695128 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.708084 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.708801 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:25.703751 12641 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:25.651764 23058 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:25.715603 23058 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.717571 23118 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:16:25.718106 23118 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:25.719573 23118 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.720532 12638 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:16:25.722635 23058 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:25.713011 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.718140 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.690726 23319 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:25.714223 12645 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:25.714840 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.725119 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:25.706418 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:25.748872 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46738us
W20260709 12:16:25.756224 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49874us
I20260709 12:16:25.757900 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.758780 12641 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:25.763969 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.685799 23317 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:25.771353 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:25.773126 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:25.779489 12564 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 29 index: 15. (term mismatch)
I20260709 12:16:25.786194 12564 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 14
I20260709 12:16:25.786509 12564 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 28.15
I20260709 12:16:25.786834 12564 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:25.789044 12564 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 30.16
I20260709 12:16:25.789353 12564 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:25.767902 23006 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:25.819581 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105558us
I20260709 12:16:25.820757 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:25.821139 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:25.786816 23215 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:25.772727 23118 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:16:25.843115 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:25.823096 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62596.7us
I20260709 12:16:25.813920 22886 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.020s
W20260709 12:16:25.834185 23313 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:16:25.841095 23272 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:25.782936 23189 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.859934 23196 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:25.788007 23318 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:25.825788 23215 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:25.862046 23215 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:25.867372 12566 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:25.867966 12566 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:25.787607 23230 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.852303 23275 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.855166 23264 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.773350 23301 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
W20260709 12:16:25.859519 23300 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.554s	user 0.000s	sys 0.013s
I20260709 12:16:25.868927 12566 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.898188 12566 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:25.886523 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65967.1us
I20260709 12:16:25.893502 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:25.908139 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.861171 23318 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.912458 12489 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:25.914312 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.888628 23291 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:25.916502 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:25.918813 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:25.921144 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:25.881314 23272 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.931007 23006 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:16:25.934902 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125625us
I20260709 12:16:25.931535 23304 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:25.944535 23291 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:25.945953 23291 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:25.945986 23160 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:25.946586 23291 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:25.946785 23160 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.948212 23318 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.947568 23291 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.15, Last appended by leader: 15, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:25.950249 23291 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:25.950814 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:25.951557 12491 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:25.952827 23160 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:25.920349 23324 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:25.961931 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:25.963135 12571 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:25.980911 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.957973 23006 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:25.982372 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.982676 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.983158 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.983686 12641 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:25.986109 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.991773 12646 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:25.993031 12638 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:25.986809 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:25.987439 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:25.988404 23321 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:25.975176 23272 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:25.994474 12639 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:26.013792 23006 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:25.996541 23321 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:26.016124 23321 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.014513 23264 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:26.032887 23293 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:26.033504 23293 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:26.034730 23221 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:26.035246 23221 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:26.036394 23221 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.042075 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.043107 23103 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:26.042578 23221 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:16:26.044472 23325 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.111s	user 0.000s	sys 0.003s
I20260709 12:16:26.053089 23275 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.053812 23275 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.043942 23215 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.025364 23321 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:26.045562 23058 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:26.068562 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:26.072103 13259 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:26.074182 12566 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
I20260709 12:16:26.079198 12566 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:16:26.079535 12566 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 29.16
I20260709 12:16:26.079799 12566 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:26.091954 23058 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:26.092649 23058 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.097879 22821 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:26.100850 22821 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:26.102108 23215 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:26.085793 23322 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 30 current term: 30 is higher.
I20260709 12:16:26.128988 23058 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:26.099130 23330 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:16:26.134792 23330 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:26.135195 23330 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:26.136183 23330 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:26.144140 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:26.144613 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.128232 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:26.147949 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.152010 23293 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.152661 23318 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.153321 23318 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:26.143491 23105 consensus_queue.cc:1175] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 30 status { last_received { term: 30 index: 19 } last_committed_idx: 19 last_received_current_leader { term: 30 index: 19 } }
I20260709 12:16:26.154803 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.155931 23318 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:26.156540 23318 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.158051 12566 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:16:26.159112 23318 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.161900 23318 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:26.164340 22821 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.170065 23103 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:26.172573 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:26.173308 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 37
W20260709 12:16:26.174192 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46062.5us
I20260709 12:16:26.177392 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:26.177795 23058 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:26.178180 13261 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:26.179742 12564 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:16:26.179916 23058 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:16:26.182889 23339 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:26.183377 23339 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:26.184198 23275 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:26.184401 23339 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.186122 23338 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:26.186564 12569 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:16:26.186772 23338 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.189393 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:26.190992 23261 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:26.191844 23171 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:26.192276 23171 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:26.192615 23171 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:26.189383 23338 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:26.194934 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.195593 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:26.196338 12571 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:16:26.197013 12640 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
W20260709 12:16:26.198509 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.199612 23229 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:26.199808 23318 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:26.200148 23229 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:26.200459 23318 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.201242 23229 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.143544 23330 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:26.203763 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:26.204519 12644 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:26.205770 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:26.206112 12639 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:16:26.207602 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:26.208279 12646 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:26.212019 23105 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:26.212790 23105 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.216652 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:26.218847 13258 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:16:25.820940 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:26.226943 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:26.229943 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:26.230919 12646 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:26.235036 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:26.246745 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51915.1us
I20260709 12:16:26.201025 23317 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:26.250128 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.253077 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:26.254562 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.255465 12492 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
W20260709 12:16:26.262133 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62766.9us
I20260709 12:16:26.153328 23293 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:26.264312 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:26.267895 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.165561 22821 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:26.273368 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.277463 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.249136 23229 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:26.286455 23221 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:26.291679 23308 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.246349 23264 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:16:26.275058 23196 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:26.275373 23272 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:26.276870 23133 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:26.320520 12646 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:26.311945 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136316us
I20260709 12:16:26.317003 12566 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:16:26.292671 23308 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.309794 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:16:26.363886 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90606.4us
I20260709 12:16:26.365058 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.370695 23233 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:26.371394 23233 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.372988 23307 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:16:26.373471 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:26.376231 23338 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:26.376370 12492 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:26.376791 23338 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:26.377311 23338 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:26.379519 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.381385 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:26.379421 23233 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:26.381970 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.401031 12496 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 13. Preceding OpId from leader: term: 34 index: 14. (index mismatch)
I20260709 12:16:26.402700 12489 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:26.388087 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.389573 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.405665 12493 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
W20260709 12:16:26.406318 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.415165 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:26.418454 12495 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:26.419695 22606 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:26.420717 22606 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:26.421077 22606 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:26.326769 23269 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
I20260709 12:16:26.422055 22606 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.363361 23304 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.322188 23196 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:26.327536 23118 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.081s
I20260709 12:16:26.383562 23272 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:26.435663 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:26.437258 12389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 131) took 27330 ms (client timeout 15000 ms). Trace:
I20260709 12:16:26.438635 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:26.363807 23260 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.070s
I20260709 12:16:26.404389 23306 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:26.434342 23133 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:26.423576 23301 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:26.441229 23165 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:26.478118 23105 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:26.480635 23308 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:26.457051 22606 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:26.403169 23307 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:16:26.500784 12571 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:26.463166 12389 rpcz_store.cc:269] 0709 12:15:59.077309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:11.762742 (+12685433us) service_pool.cc:224] Handling call
0709 12:16:26.407933 (+14645191us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:16:22.986457 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:16:22.992359 (+  5902us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:16:22.992416 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:16:22.992435 (+    19us) tablet.cc:662] Decoding operations
0709 12:16:22.994218 (+  1783us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:16:22.994337 (+   119us) write_op.cc:641] Partition lock acquired for write op
0709 12:16:22.994371 (+    34us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:16:23.030238 (+ 35867us) tablet.cc:701] Row locks acquired
0709 12:16:23.030310 (+    72us) write_op.cc:260] PREPARE: finished
0709 12:16:23.030515 (+   205us) write_op.cc:270] Start()
0709 12:16:23.030677 (+   162us) write_op.cc:276] Timestamp: P: 1783599383030443 usec, L: 0
0709 12:16:23.030718 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:16:23.031527 (+   809us) log.cc:844] Serialized 7877 byte log entry
0709 12:16:23.526501 (+494974us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b1800451fc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:23.847582 (+321081us) op_driver.cc:464] REPLICATION: finished
0709 12:16:24.086100 (+238518us) write_op.cc:301] APPLY: starting
0709 12:16:24.086285 (+   185us) tablet.cc:1367] starting BulkCheckPresence
0709 12:16:24.367390 (+281105us) tablet.cc:1370] finished BulkCheckPresence
0709 12:16:24.367441 (+    51us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:25.200154 (+832713us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:25.314899 (+114745us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=0,mrs_lookups=0
0709 12:16:25.340158 (+ 25259us) write_op.cc:312] APPLY: finished
0709 12:16:25.362217 (+ 22059us) log.cc:844] Serialized 180 byte log entry
0709 12:16:25.402026 (+ 39809us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:25.402282 (+   256us) write_op.cc:454] Released schema lock
0709 12:16:25.402371 (+    89us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:25.407017 (+  4646us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:25.407061 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5110316,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":196019,"apply.run_cpu_time_us":38664,"apply.run_wall_time_us":1350112,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87647,"num_ops":20,"prepare.queue_time_us":157008,"prepare.run_cpu_time_us":14938,"prepare.run_wall_time_us":406125,"raft.queue_time_us":299844,"raft.run_cpu_time_us":13391,"raft.run_wall_time_us":88575,"replication_time_us":816603,"spinlock_wait_cycles":13590400,"thread_start_us":696457,"threads_started":5,"wal-append.queue_time_us":343401,"wal-append.run_cpu_time_us":33438,"wal-append.run_wall_time_us":2286794}]]}
I20260709 12:16:26.514961 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:26.457904 23345 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:16:26.540678 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.446327 23304 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.508914 23261 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:26.552366 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.557415 12495 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:26.535554 23221 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:26.512779 23306 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.571626 23339 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:26.572701 23306 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:26.503914 23301 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.581923 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:26.552227 23318 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:26.585058 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 307695us
I20260709 12:16:26.588841 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.589164 23230 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:16:26.589496 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.536860 23233 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:16:26.541513 23105 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.033s
W20260709 12:16:26.596858 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 355329us
I20260709 12:16:26.599310 13258 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:26.601300 23105 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.601382 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.602098 23105 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.604400 23105 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:26.604980 12563 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:26.604422 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.607381 12640 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:26.611783 12643 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:26.588522 23221 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:26.625592 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:26.626362 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:26.627398 12638 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:26.630565 12567 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 13. Preceding OpId from leader: term: 34 index: 14. (index mismatch)
I20260709 12:16:26.558130 23196 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.17, Last appended by leader: 17, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.652211 12536 leader_election.cc:400] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:26.654726 12536 leader_election.cc:403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:26.653124 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:26.656237 12641 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:26.614653 23318 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:16:26.608305 23230 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:26.670307 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.635938 23189 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.620337 23324 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:26.660578 23165 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.661958 22886 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:26.683105 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.669597 23275 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:26.664526 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.681432 23189 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:26.672704 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 237128us
W20260709 12:16:26.700320 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.705281 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.681558 23324 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:26.677462 23230 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:26.783964 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.720288 12643 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
W20260709 12:16:26.761327 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:26.708654 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44244.2us
I20260709 12:16:26.828451 12572 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:16:26.830076 12536 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:26.831485 12536 leader_election.cc:403] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:26.753947 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.751850 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.768805 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.849735 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:26.850389 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145270us
I20260709 12:16:26.851053 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.846621 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:26.707610 23171 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.515s	user 0.014s	sys 0.000s
I20260709 12:16:26.860291 12489 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 26. Current term is 27. Ops: []
W20260709 12:16:26.730543 23347 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:16:26.863827 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.822281 22956 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.870687 12568 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:26.824023 23264 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.741365 23301 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:26.822005 23261 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:26.748620 23189 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:26.871809 23293 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.000s
I20260709 12:16:26.743534 23338 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.848948 23324 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:26.881148 12569 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:26.743536 23304 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:26.849980 23036 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:26.852048 22821 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:26.900054 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:26.903131 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:26.684423 23165 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.688143 22886 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.713888 23291 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:26.865903 23349 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:26.869560 23322 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:26.706488 23221 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.936153 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.872718 23171 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.937853 12565 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
W20260709 12:16:26.938802 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58742.9us
I20260709 12:16:26.959167 12495 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
W20260709 12:16:26.964691 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102173us
I20260709 12:16:26.927388 23275 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.972972 12569 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:26.898134 23264 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:26.888408 22956 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:26.981611 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:26.981848 12494 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:26.927764 23353 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:26.982478 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:26.978840 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:26.943910 23345 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:26.940577 23322 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:26.973053 23196 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:26.999773 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:27.008206 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.009509 12567 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:16:26.989657 23322 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:26.991863 23293 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:27.037534 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.041240 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:27.042290 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:27.046378 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:27.047511 13258 raft_consensus.cc:606] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:16:27.047942 13258 raft_consensus.cc:614] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:27.052129 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:27.053062 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:27.059089 22956 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:26.989796 23261 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:26.982960 23275 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.085784 13258 raft_consensus.cc:606] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:27.086577 13258 raft_consensus.cc:614] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:27.085191 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:27.038759 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 138812us
I20260709 12:16:27.117516 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:26.986876 23291 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:16:27.005051 23330 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:26.989606 23345 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:27.166667 22606 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.675s	user 0.014s	sys 0.001s
I20260709 12:16:27.060518 23165 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:27.198885 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 190418us
I20260709 12:16:27.167928 23268 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:27.029076 23301 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.171s
W20260709 12:16:27.269800 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 152405us
I20260709 12:16:27.262895 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:27.040659 23105 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.216s
I20260709 12:16:27.191228 23196 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.223266 23349 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:27.223498 23269 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:27.222771 23365 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:27.268860 23339 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.697s	user 0.012s	sys 0.001s
I20260709 12:16:27.265046 23268 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:27.145043 23361 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.233s
I20260709 12:16:27.262027 23171 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:27.277571 23351 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 12:16:27.271428 23293 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.268867 23165 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.328333 12646 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
W20260709 12:16:27.346920 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81060.3us
I20260709 12:16:27.348553 23261 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:27.350028 23261 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:16:27.354368 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67798.3us
I20260709 12:16:27.355383 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:27.274132 23367 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.274314 22606 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:27.367262 23354 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.306s	user 0.000s	sys 0.001s
W20260709 12:16:27.368191 23360 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.328s	user 0.000s	sys 0.001s
I20260709 12:16:27.368963 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:27.369841 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:27.370976 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:27.371600 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:16:27.372848 12488 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:27.374243 23275 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.376236 23036 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:27.376842 23036 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:27.377112 12488 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:16:27.325908 23372 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:27.357980 22956 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.379479 22956 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.310113 23268 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:27.327924 23230 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.341085 23338 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:27.404348 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:27.407465 23382 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:27.411885 13261 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:27.412851 13261 raft_consensus.cc:606] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:27.413277 13261 raft_consensus.cc:614] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:27.422003 22886 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.432806 23264 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.300678 23349 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.435761 23349 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.359808 23165 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.439415 23165 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.359356 23293 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.442811 23293 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.383383 23367 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.450338 23367 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:27.457106 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52838.7us
I20260709 12:16:27.458369 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:27.462212 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.009s	sys 0.048s
I20260709 12:16:27.376283 23345 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.461388 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:27.294991 23196 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.294803 23269 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.490720 23377 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.493199 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:27.397595 23137 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:27.494555 13261 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:16:27.499123 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 192295us
I20260709 12:16:27.500012 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:27.502454 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:27.461846 22956 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:27.464249 23301 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:27.302201 23365 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.384385 22606 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.463189 23318 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.481215 23293 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.482064 23382 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:27.491685 23377 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.516366 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:27.516718 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:27.518113 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:27.511826 23367 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:27.519129 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:27.505663 23374 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:27.537297 13260 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:27.533696 13258 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:27.536005 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:27.540774 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:27.514098 23301 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:27.543303 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.545027 23301 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:27.545490 12639 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:27.544561 23322 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.505s	user 0.012s	sys 0.000s
I20260709 12:16:27.550280 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.555135 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:27.512830 23345 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:16:27.562544 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.562191 23359 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:27.578284 23359 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:27.578887 23359 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:27.579666 23349 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:27.580183 23349 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:27.580534 23349 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:27.580753 23377 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:27.578205 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115325us
I20260709 12:16:27.583236 12644 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 17. Preceding OpId from leader: term: 37 index: 18. (index mismatch)
I20260709 12:16:27.583482 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.584424 12645 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:27.535441 23293 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.533895 23318 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.587903 12638 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:16:27.598059 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 263287us
I20260709 12:16:27.599884 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.600551 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.601066 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.601306 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.601408 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:27.601848 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:27.600638 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:27.606845 23293 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.607705 23268 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.615017 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4340) took 1016 ms. Trace:
I20260709 12:16:27.615572 12564 rpcz_store.cc:276] 0709 12:16:26.598711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.603180 (+  4469us) service_pool.cc:224] Handling call
0709 12:16:26.603530 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:26.902993 (+299463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:26.903023 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:26.913455 (+ 10432us) log.cc:844] Serialized 26 byte log entry
0709 12:16:27.462943 (+549488us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:27.463003 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:27.463285 (+   282us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:27.614718 (+151433us) raft_consensus.cc:1685] finished
0709 12:16:27.614747 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:27.614936 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9497,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":16397,"thread_start_us":815563,"threads_started":2,"wal-append.queue_time_us":592984}
I20260709 12:16:27.616686 23383 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.617249 23383 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.619954 23393 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.621706 23036 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.566562 23322 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.627588 23322 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.625677 23036 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:27.629412 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:27.635025 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:27.640434 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:27.642740 23196 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.643081 23215 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:27.643488 23215 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:27.643905 23215 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:27.644199 23365 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.647825 23269 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:27.648957 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:27.649886 12564 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:16:27.650702 22606 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:27.656368 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1505) took 1059 ms. Trace:
I20260709 12:16:27.620601 23393 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.656931 12646 rpcz_store.cc:276] 0709 12:16:26.597267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.597822 (+   555us) service_pool.cc:224] Handling call
0709 12:16:26.602064 (+  4242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:27.334294 (+732230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:27.656050 (+321756us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:27.656282 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1668807936}
I20260709 12:16:27.642581 23367 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:27.659284 23367 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:27.659727 23367 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:27.659778 23393 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.634305 23322 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.664721 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4519) took 1283 ms. Trace:
I20260709 12:16:27.669456 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:27.666551 12494 rpcz_store.cc:276] 0709 12:16:26.380811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.380953 (+   142us) service_pool.cc:224] Handling call
0709 12:16:26.381287 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:26.985305 (+604018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:27.664351 (+679046us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:27.664609 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1380563840}
I20260709 12:16:27.671258 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:27.673450 12563 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:27.673744 23322 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.674914 12566 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:27.674860 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.676397 23322 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.619592 23383 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.680012 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4518) took 1301 ms. Trace:
I20260709 12:16:27.686203 12492 rpcz_store.cc:276] 0709 12:16:26.378961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.379112 (+   151us) service_pool.cc:224] Handling call
0709 12:16:26.379458 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:26.379853 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:26.379880 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:27.380794 (+1000914us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:27.381536 (+   742us) log.cc:844] Serialized 26 byte log entry
0709 12:16:27.514416 (+132880us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:27.514454 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:27.514529 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:27.677716 (+163187us) raft_consensus.cc:1685] finished
0709 12:16:27.677736 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:27.679906 (+  2170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":13510016,"thread_start_us":132421,"threads_started":1,"wal-append.queue_time_us":214045}
I20260709 12:16:27.690320 23196 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.013s
I20260709 12:16:27.710786 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4523) took 1320 ms. Trace:
I20260709 12:16:27.711381 12493 rpcz_store.cc:276] 0709 12:16:26.389962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.390092 (+   130us) service_pool.cc:224] Handling call
0709 12:16:26.390325 (+   233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:26.406060 (+ 15735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:27.710438 (+1304378us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:27.710697 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34440576}
I20260709 12:16:27.690790 23322 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.712878 23322 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.715121 23322 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.717846 12496 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:27.718045 23322 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:27.718581 23322 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:27.711897 23324 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.777s	user 0.013s	sys 0.000s
I20260709 12:16:27.722829 23324 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.723564 23322 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.726006 23324 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.727434 23237 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:27.727974 23237 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:27.751753 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.753041 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:27.753782 12640 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:27.756748 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:27.759063 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.759855 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:27.766105 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.769335 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.774315 12639 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:27.770426 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.774487 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.775350 12641 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:27.691867 23393 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:27.776582 12569 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 17. Preceding OpId from leader: term: 37 index: 18. (index mismatch)
I20260709 12:16:27.784729 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.789544 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:27.695047 23036 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:27.795805 22821 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.003s
I20260709 12:16:27.787211 23393 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:27.807034 12564 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:27.765723 23397 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.808436 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:27.765030 23324 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:27.777047 23233 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.815832 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:27.810930 12640 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:27.816818 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:27.821736 12643 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:27.814990 23397 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.823145 23352 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:27.823589 23352 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:27.823938 23352 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:27.825946 23390 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:27.826603 23390 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.787871 23268 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.782408 23322 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.831061 23322 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.833621 23322 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.833711 23275 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:27.834625 23275 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:27.728957 23237 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.835319 23275 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:27.835891 23237 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:27.800402 23293 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:16:27.800760 22886 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:27.838541 22886 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.839218 23352 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.820189 23171 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:27.840934 22886 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.841384 23352 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.840548 23171 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.843514 23368 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.023s
I20260709 12:16:27.844295 23171 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.800989 23230 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:27.845188 23230 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.846074 23233 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.847426 23230 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.848721 23233 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.771733 23301 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:27.851286 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:27.851711 23301 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:27.857861 23324 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:27.859148 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69757.5us
I20260709 12:16:27.860456 23324 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:27.861156 23363 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:27.816354 23393 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:27.865113 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:27.815130 23036 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.868110 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:27.860540 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:27.799260 23345 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:27.864092 23363 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.874002 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:27.876314 12494 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:27.857296 23359 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.879148 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.898259 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.884573 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:27.898553 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.901068 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:27.908845 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:27.909019 12645 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:27.908845 12489 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:27.910221 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:27.912312 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.915184 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:27.916283 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96485.2us
I20260709 12:16:27.928737 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:27.892367 23347 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.001s	user 0.004s	sys 0.005s
W20260709 12:16:28.016372 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 155900us
I20260709 12:16:28.027532 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:28.028915 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:27.870400 23390 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.018610 12643 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:27.870854 23397 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:27.875355 23268 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:27.887677 23291 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:27.898733 23036 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:27.870455 23196 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.053898 23409 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:28.055490 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:27.917847 23301 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:27.918604 23105 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:28.016271 23230 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:28.070360 23301 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.071450 23105 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.082316 23397 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:28.082630 22232 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.063593 23268 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.885480 23345 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.091912 23133 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:28.120139 22232 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.093864 23398 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:28.095284 23385 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:27.885332 23363 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:27.897768 23215 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.097867 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4344) took 1498 ms. Trace:
I20260709 12:16:28.043109 23353 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:16:28.103250 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.102363 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.154045 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:28.154582 12564 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:27.886686 23367 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.115281 12645 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:28.058915 23264 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:28.087121 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:28.034932 23171 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:27.875711 23349 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.129344 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.061928 23368 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.137173 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.138805 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.135790 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.068653 23291 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:28.147909 12572 rpcz_store.cc:276] 0709 12:16:26.599893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.609770 (+  9877us) service_pool.cc:224] Handling call
0709 12:16:26.636683 (+ 26913us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:26.829439 (+192756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:28.097493 (+1268054us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:28.097791 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264864512}
W20260709 12:16:28.149857 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:28.099295 23230 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:28.155265 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.160471 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.216260 12564 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:28.162730 13259 raft_consensus.cc:606] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:16:28.144232 23385 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:28.133306 23133 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:28.068466 23196 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:28.222988 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:28.145356 23036 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:28.210863 22232 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.158336 23353 consensus_queue.cc:1243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 34.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:16:28.163677 23361 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.246976 23393 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.213442 23368 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.227197 23275 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.248179 23291 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.249577 23275 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.150857 23398 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.162351 23264 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.250910 23264 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:28.252624 23398 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.218683 22886 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:28.230327 23371 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.185s	user 0.004s	sys 0.012s
I20260709 12:16:28.231000 23105 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.231531 23367 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.193975 23171 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.240620 23215 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.242260 23268 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.243216 23301 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.244661 23349 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.245972 23359 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:28.246862 23361 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.209220 12570 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:16:28.254961 23260 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.257046 23409 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:28.257985 23345 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.259994 23363 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.223913 23385 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:28.263563 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.265342 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.312212 12566 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:28.266057 23322 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:28.266619 23293 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:28.272316 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 193472us
I20260709 12:16:28.227739 22606 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.218616 23230 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:28.209722 23352 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.280686 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:28.214845 23324 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.285408 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.285853 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.287195 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.301738 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.384066 12495 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:28.307026 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:28.302480 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 178278us
I20260709 12:16:28.385303 12488 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:28.388701 23330 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:28.277794 22886 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.292608 23398 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.288264 23374 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:28.366197 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209819us
I20260709 12:16:28.358178 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:28.288295 23367 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.356805 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.277015 23036 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.376725 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
W20260709 12:16:28.376850 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96322.3us
I20260709 12:16:28.392784 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:28.249570 23368 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.389696 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3404) took 1631 ms. Trace:
I20260709 12:16:28.428369 12496 rpcz_store.cc:276] 0709 12:16:26.758233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.758563 (+   330us) service_pool.cc:224] Handling call
0709 12:16:27.717543 (+958980us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:27.718461 (+   918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:28.389461 (+671000us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54000f55d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:28.389610 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2198527232}
I20260709 12:16:28.397149 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.389616 23330 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.387686 23397 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.420941 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.370061 23275 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.421846 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:28.380209 23371 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:16:28.307384 23260 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.441331 12488 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:28.376113 22734 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.444968 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.446306 12489 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:28.391474 23409 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:28.363824 23322 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:28.483239 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55803.6us
I20260709 12:16:28.393149 23359 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.075s
W20260709 12:16:28.451399 23408 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:16:28.395654 22606 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:28.491041 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96631.1us
I20260709 12:16:28.412901 23352 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.317250 23133 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.379256 23293 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.502277 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:28.445942 23415 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:28.484382 22734 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.456231 23397 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.587371 22734 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:28.590323 23264 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.581141 23417 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.115s
I20260709 12:16:28.584445 23345 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.625905 23345 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.508265 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.466331 23275 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.505919 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.487939 23406 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:28.582659 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.587610 23397 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:28.630084 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.695135 13258 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:28.592190 22886 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.502022 22821 raft_consensus.cc:2764] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:16:28.595021 23321 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.633020 23260 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.699932 23321 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.701144 23406 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.701747 23406 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:28.704248 22821 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.704912 22821 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.412870 23324 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:28.597936 23370 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.555s	user 0.004s	sys 0.003s
I20260709 12:16:28.602393 23215 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.439373 23374 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.608101 23261 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.609031 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.782394 12489 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:28.609958 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.610555 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.613221 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.617877 23393 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:28.619752 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.446933 23367 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.621044 23412 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:28.447055 23398 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.508337 23409 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:28.621786 23394 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.655820 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.664876 23221 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.596170 23196 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.652770 23237 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:28.595150 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:28.597510 23416 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.710340 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.711314 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.711405 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.712924 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:28.712920 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.714368 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.714010 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.714066 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.715548 23345 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.714629 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.716831 23133 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:28.717275 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.717512 23318 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.646804 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.843381 12569 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:16:28.601429 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.733700 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:28.747735 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 243694us
I20260709 12:16:28.755913 13258 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:16:28.771685 13259 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 37: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:28.765913 23415 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.698948 22886 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:28.778684 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 359137us
I20260709 12:16:28.784946 12494 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:28.847262 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:28.795372 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 402649us
I20260709 12:16:28.795452 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.802210 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.805029 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.848642 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
I20260709 12:16:28.809321 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.809988 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.833760 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.849622 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.829810 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:16:28.850360 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:28.851570 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:28.790885 23118 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.789590 23275 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.796756 23307 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:28.791744 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.835107 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.872079 12571 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:28.786805 23215 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.858156 12494 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:16:28.849849 12490 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:16:28.858806 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.860734 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.802147 23036 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:28.767123 23105 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
W20260709 12:16:28.856472 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:28.880461 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.880856 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:28.827265 23237 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:28.883503 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.843140 23428 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:28.884968 12572 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:28.901404 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.789989 23261 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.905213 12567 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
W20260709 12:16:28.904863 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59109us
I20260709 12:16:28.908001 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:28.826879 23221 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.885869 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4360) took 1942 ms. Trace:
I20260709 12:16:28.893158 12493 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:28.871229 23307 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:28.867506 23318 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:28.814880 23394 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.919402 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:28.876253 23322 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.811082 23412 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.904243 12388 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 15 to 18, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 18 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:28.916579 12565 rpcz_store.cc:276] 0709 12:16:26.943358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.945499 (+  2141us) service_pool.cc:224] Handling call
0709 12:16:26.945931 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:26.984580 (+ 38649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:26.984631 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:26.990341 (+  5710us) log.cc:844] Serialized 26 byte log entry
0709 12:16:27.046397 (+ 56056us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:27.046447 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:27.046530 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:28.431495 (+1384965us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:28.878252 (+446757us) raft_consensus.cc:1685] finished
0709 12:16:28.878276 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:28.885802 (+  7526us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56774,"raft.queue_time_us":33027,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":13554,"spinlock_wait_cycles":41708928,"thread_start_us":55542,"threads_started":1,"wal-append.queue_time_us":52254}
W20260709 12:16:28.957089 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109870us
I20260709 12:16:28.872510 23118 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:28.866961 22886 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.964148 13256 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:28.964818 13256 raft_consensus.cc:606] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:28.965200 13256 raft_consensus.cc:614] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:28.967963 12564 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:28.930987 12571 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:28.962940 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:28.859423 23416 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:28.920202 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:28.974731 12609 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:16:28.975112 12609 leader_election.cc:403] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:16:28.902217 23345 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:28.971413 13262 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:28.959270 23361 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.959471 23429 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:28.978125 12495 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:28.995540 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:16:28.919935 23269 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.068s
I20260709 12:16:28.924813 23171 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:28.988209 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:28.980731 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72783.1us
I20260709 12:16:28.980994 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:28.993988 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.004869 12567 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:29.005139 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:29.006002 12493 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:29.005915 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:28.998273 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:28.982336 12564 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:28.996717 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.006405 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.007225 12491 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:29.007253 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:29.002177 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:29.007067 12490 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:29.014037 12491 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:16:29.019831 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:29.020744 12493 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:29.017226 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:28.914433 23428 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:29.026038 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45110.4us
I20260709 12:16:29.023947 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:28.964035 23318 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.030213 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:29.011381 23230 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.001s
I20260709 12:16:29.032265 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:29.043020 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:29.043128 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:29.033154 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:29.039546 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:29.004263 23393 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:29.011513 22821 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.040243 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:28.907830 23409 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.026391 23352 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.054720 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:29.048467 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:29.055974 22886 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.052386 23165 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:29.056391 12566 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:29.034423 23324 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.058842 23318 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.081727 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.059393 23349 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:29.059819 23427 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:29.060256 23419 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:16:29.036564 23152 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:29.065418 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:29.034770 22606 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.057507 23261 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.066893 23430 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:29.070300 23362 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:29.070808 23333 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:29.071236 23366 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:29.080379 23268 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:29.040841 23196 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:28.994509 23345 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.075063 12569 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:29.067447 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:29.046942 23417 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:29.079567 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.049577 23304 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:29.126699 23304 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.126994 23412 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.127315 23304 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:29.102151 23152 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:29.064078 23425 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:29.082759 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.129129 23152 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.131007 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.131785 12491 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:29.083552 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.008104 23429 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.133040 12489 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:29.061456 23390 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:29.133878 23390 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.134423 23390 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:29.134680 23429 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.101905 23411 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:29.135633 23411 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.108925 23196 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:29.138696 23196 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:29.090452 22502 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:29.139755 22502 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.068737 23353 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:29.145097 23353 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.145706 23353 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:29.110806 23417 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:29.152474 23417 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:29.112129 23366 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:29.156571 23366 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:29.162904 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48789.9us
I20260709 12:16:29.095768 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:29.004966 23307 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.096701 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:29.068193 23165 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:29.099438 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:29.069227 23367 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:29.101684 23424 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260709 12:16:29.050961 23230 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:29.072075 23422 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 12:16:29.050685 23393 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.111195 12495 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 23.
I20260709 12:16:29.088953 23427 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.086812 23374 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:29.114997 23421 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:16:29.104977 23362 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:29.115724 23432 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:16:29.119072 23398 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.120623 23337 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:29.121549 23394 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.104776 23333 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.088388 23215 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.167550 12388 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:29.163918 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:29.185420 22606 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:29.174728 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:29.175014 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:29.178020 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82308us
I20260709 12:16:29.181236 22502 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:29.263543 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:29.265417 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101586us
W20260709 12:16:29.181833 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82407.5us
I20260709 12:16:29.102619 23376 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:16:29.134130 23415 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:29.227397 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4369) took 1087 ms. Trace:
W20260709 12:16:29.207834 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:29.096508 23395 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.104s
I20260709 12:16:29.161885 23349 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:29.270336 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:29.227069 23436 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:16:29.180289 23165 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:29.247344 12563 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:29.317963 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.176929 23321 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.187088 23380 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:29.124622 23345 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.241673 23367 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:29.074240 23368 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:16:29.354291 23368 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:29.354647 23368 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:29.355573 23368 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 33.13, Last appended by leader: 13, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.356825 23368 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:29.361713 23345 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:29.362815 23345 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:29.250998 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.576s (SENT) [suppressed 5 similar messages]
I20260709 12:16:29.269965 12570 rpcz_store.cc:276] 0709 12:16:28.139642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:28.289147 (+149505us) service_pool.cc:224] Handling call
0709 12:16:28.289590 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:28.290871 (+  1281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:28.290921 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:28.303297 (+ 12376us) log.cc:844] Serialized 26 byte log entry
0709 12:16:28.438797 (+135500us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:28.438847 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:28.438923 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:29.227067 (+788144us) raft_consensus.cc:1685] finished
0709 12:16:29.227099 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:29.227313 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10386,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":2252,"thread_start_us":79722,"threads_started":1,"wal-append.queue_time_us":190435}
I20260709 12:16:29.366621 23345 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.371309 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.304032 23337 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.372236 23337 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:29.373471 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.374413 12492 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:29.377537 23363 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:29.378607 23264 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.379189 23363 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:29.384757 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4373) took 1244 ms (client timeout 1576 ms). Trace:
I20260709 12:16:29.384981 12565 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:29.388972 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.314515 23394 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:29.397912 12492 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:29.398620 12492 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:29.399781 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.239703 23307 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:29.277514 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4367) took 1139 ms. Trace:
I20260709 12:16:29.279551 23393 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:29.283250 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.412066 12495 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:29.412209 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.413408 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.175113 23237 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.289104 23349 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:29.297901 23230 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.238739 23427 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:29.311100 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:29.447438 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136471us
I20260709 12:16:29.307919 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:29.321902 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.323099 23221 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.319037 23215 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20260709 12:16:29.318504 23105 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:29.304900 12388 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:29.175179 23268 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.083s
I20260709 12:16:29.301059 23362 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:29.328511 23411 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.329553 23364 raft_consensus.cc:993] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: : Instructing follower 6d4ece942fd5414998215d26edf91001 to start an election
I20260709 12:16:29.332633 23376 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:29.336030 23293 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:29.402113 12563 rpcz_store.cc:269] 0709 12:16:28.140743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:28.154968 (+ 14225us) service_pool.cc:224] Handling call
0709 12:16:29.384466 (+1229498us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:29.384653 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2367133952}
I20260709 12:16:29.460675 22734 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:29.404624 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:29.405165 12639 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 16
I20260709 12:16:29.499109 12639 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 34.17
I20260709 12:16:29.499540 12639 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:29.407678 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.409063 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.414217 12495 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:29.284289 23398 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:29.510099 23398 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.510617 23398 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:29.256376 23118 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.399935 12492 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.515347 12492 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:29.418061 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.095s	sys 0.160s
W20260709 12:16:29.418697 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:29.516525 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.095s	sys 0.160s
W20260709 12:16:29.517041 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 675944us
I20260709 12:16:29.518270 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:29.420234 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:29.352670 23367 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.453058 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.319015 23416 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:29.455319 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 147527us
I20260709 12:16:29.456702 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.460376 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.461938 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.487164 23428 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.479039 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.480890 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.484017 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.536212 12644 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 21.
W20260709 12:16:29.459334 23385 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.096s	user 0.000s	sys 0.018s
I20260709 12:16:29.436622 23330 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.489945 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.446193 23437 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:29.401321 12568 rpcz_store.cc:276] 0709 12:16:28.137973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:28.138117 (+   144us) service_pool.cc:224] Handling call
0709 12:16:28.164813 (+ 26696us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:28.192371 (+ 27558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:28.192424 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:28.214580 (+ 22156us) log.cc:844] Serialized 26 byte log entry
0709 12:16:28.418284 (+203704us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:28.418330 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:28.418410 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:29.277208 (+858798us) raft_consensus.cc:1685] finished
0709 12:16:29.277233 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:29.277432 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69898,"raft.queue_time_us":27348,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":14917,"thread_start_us":116034,"threads_started":1,"wal-append.queue_time_us":595040}
I20260709 12:16:29.501350 12488 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:29.474961 23171 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:29.523409 12643 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:16:29.555573 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.482069 23321 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:29.527392 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:29.498211 23362 raft_consensus.cc:1081] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Signalling peer 6d4ece942fd5414998215d26edf91001 to start an election
W20260709 12:16:29.476362 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:29.570868 12536 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:29.571267 12536 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:29.571550 23439 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:29.536517 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:29.537350 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.573308 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.541481 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:29.573995 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.559031 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.574753 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 24.
I20260709 12:16:29.515065 22734 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.558249 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.564723 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.565838 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.572125 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:29.567171 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.562498 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.583000 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:29.583927 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.584903 12568 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:29.568688 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.576164 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.539750 23293 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:29.586992 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:29.589607 12637 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:29.579070 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.589974 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.588465 12645 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:29.495801 23196 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.588936 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.599678 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.600077 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.601130 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:29.574542 23133 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.716s	user 0.009s	sys 0.005s
I20260709 12:16:29.594471 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.631857 12563 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:29.587368 23268 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.018s
I20260709 12:16:29.636291 12641 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:29.602003 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.574136 23417 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.602711 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.574901 23437 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.583288 23433 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:16:29.578264 23321 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.606698 23171 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:29.589526 23293 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:29.608718 23196 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.609131 23304 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.656993 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.612149 23165 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.588392 23429 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:29.616782 23237 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:29.621982 23428 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:29.557498 23385 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.513761 23376 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:29.624259 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53536us
I20260709 12:16:29.625514 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.592259 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:29.632431 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:29.559697 23349 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.15, Last appended by leader: 15, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.635144 12492 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:29.585672 22734 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:29.580637 23363 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:29.642143 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.638258 12490 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:29.691378 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.692480 12566 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:29.648272 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 397343us
I20260709 12:16:29.584045 23411 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:29.653321 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 208736us
I20260709 12:16:29.697750 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.698581 12572 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:29.654536 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:29.610586 23333 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:29.657774 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:29.623214 23366 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:29.663234 12565 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:29.652854 23293 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:16:29.703474 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:29.713603 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.601135 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:29.674965 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.677670 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.678061 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:29.708453 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:29.715798 12570 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:29.669481 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:29.685796 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:29.686520 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54227.3us
I20260709 12:16:29.688633 12493 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:29.689571 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97409.3us
I20260709 12:16:29.660634 23439 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:16:29.700557 23353 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.547s	user 0.013s	sys 0.000s
I20260709 12:16:29.692996 12496 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:29.697750 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:29.665735 23393 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:29.668437 23358 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:29.721050 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:29.721876 12644 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:29.647665 12497 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:29.726234 23439 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:29.762341 23443 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:16:29.703099 23233 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.116142 23233 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:29.711006 23411 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:30.013677 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:30.118335 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:30.059557 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 353034us
W20260709 12:16:30.057097 23409 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.010s	sys 0.014s
I20260709 12:16:30.084139 23410 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:30.085291 23363 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.087275 23437 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.088085 23366 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:29.763702 23171 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.088918 13260 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:29.693115 23268 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:16:30.090885 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 421483us
I20260709 12:16:29.718559 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:29.695719 23433 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:16:30.091976 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 383586us
I20260709 12:16:30.100389 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.098419 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:30.093804 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:30.120795 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 402778us
I20260709 12:16:30.111375 13253 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:30.140019 12488 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:30.139942 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3430) took 1058 ms. Trace:
I20260709 12:16:30.223665 12493 rpcz_store.cc:276] 0709 12:16:29.081148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.081351 (+   203us) service_pool.cc:224] Handling call
0709 12:16:30.139681 (+1058330us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:30.139874 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393372928}
I20260709 12:16:30.116896 23358 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:30.122404 23307 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.704s	user 0.000s	sys 0.021s
I20260709 12:16:30.137365 23410 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:30.129745 23409 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.010s	sys 0.014s
I20260709 12:16:30.179289 23233 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:30.179828 23427 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.719s	user 0.004s	sys 0.015s
I20260709 12:16:30.257534 23427 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:30.181797 23337 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.809s	user 0.012s	sys 0.009s
I20260709 12:16:30.258232 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:30.099773 23321 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.260927 23321 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:30.098788 23304 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.098649 23345 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:30.149142 12570 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:30.184011 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.159246 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:30.100446 23364 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:30.114779 23221 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.192308 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3412) took 1586 ms. Trace:
I20260709 12:16:30.299320 12496 rpcz_store.cc:276] 0709 12:16:28.605940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:28.610154 (+  4214us) service_pool.cc:224] Handling call
0709 12:16:30.192001 (+1581847us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:30.192205 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2088852224}
I20260709 12:16:30.300064 12640 raft_consensus.cc:2368] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34.
W20260709 12:16:30.194582 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3415) took 1485 ms (client timeout 1543 ms). Trace:
I20260709 12:16:30.093344 23428 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:30.132524 23368 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.775s	user 0.023s	sys 0.001s
W20260709 12:16:30.232625 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:30.182619 23390 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.048s	user 0.012s	sys 0.012s
I20260709 12:16:30.245834 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:30.120004 23353 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:30.253684 22734 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.002s
W20260709 12:16:30.256448 23293 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.509s	user 0.009s	sys 0.003s
I20260709 12:16:30.258637 23337 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.258900 23165 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.098901 23264 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:30.182539 23133 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:30.263161 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.238304 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:30.432147 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 193987us
W20260709 12:16:30.273823 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50885.8us
I20260709 12:16:30.276705 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.179289 23411 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.282775 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.121990 23393 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.233739 23367 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.384675 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:30.448839 13256 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
W20260709 12:16:30.300537 12497 rpcz_store.cc:269] 0709 12:16:28.709536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:28.709967 (+   431us) service_pool.cc:224] Handling call
0709 12:16:30.194342 (+1484375us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:30.194508 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1856339200}
I20260709 12:16:30.262874 23410 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:30.385463 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 139756us
I20260709 12:16:30.402130 23268 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:30.403110 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:30.268363 12488 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:54234
I20260709 12:16:30.456429 12488 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:30.457033 12488 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:30.287601 23237 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.623s	user 0.021s	sys 0.014s
I20260709 12:16:30.183382 23363 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.414036 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.438519 23433 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:30.415422 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.279639 23345 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:30.490074 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.490645 23345 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.491663 23345 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:30.270429 23321 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.492689 23358 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.117224 23439 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.442970 12489 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:30.413693 23304 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:30.495059 23304 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:30.495498 23304 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:30.401171 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:30.414613 23374 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:30.457756 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:30.498077 23374 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:30.498661 23374 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:30.404729 23221 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.501391 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.502775 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.504191 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:30.504809 23394 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.112s	user 0.013s	sys 0.013s
I20260709 12:16:30.423254 23368 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:30.443284 23406 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.741s	user 0.020s	sys 0.011s
I20260709 12:16:30.463526 12462 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:30.482659 13262 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:30.415199 12496 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:30.487242 12643 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:30.430732 23133 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:30.431313 23293 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.399487 23390 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.399096 23428 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.414333 22734 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.505398 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.507810 12644 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
W20260709 12:16:30.510139 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:30.525460 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1312) took 1498 ms. Trace:
I20260709 12:16:30.547283 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:30.544902 12462 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:16:30.498756 23395 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:30.514325 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:30.527465 23105 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.047s	user 0.008s	sys 0.012s
I20260709 12:16:30.438126 23264 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.520207 23353 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:30.549564 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86246.2us
W20260709 12:16:30.526643 23398 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.016s	user 0.008s	sys 0.011s
I20260709 12:16:30.483587 23376 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:30.567193 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 848664us
I20260709 12:16:30.561035 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:30.569607 12646 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:30.570534 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:30.571246 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:30.560943 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.547430 13260 rpcz_store.cc:276] 0709 12:16:29.027027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.027201 (+   174us) service_pool.cc:224] Handling call
0709 12:16:30.525166 (+1497965us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:30.525346 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1659429248}
I20260709 12:16:30.572257 12644 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:30.551508 23304 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.575315 23304 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.505476 23394 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:30.578771 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:30.568313 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:30.489094 23417 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:30.508419 23321 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.521307 23385 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:30.519076 23427 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:30.529542 23337 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:30.484381 23268 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.596055 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:30.549472 12488 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.583895 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:30.587976 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.561409 23165 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:30.590377 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:30.598927 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:30.603206 13261 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:30.599176 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:30.565918 23105 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:30.550971 23368 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.618131 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:30.620388 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 899364us
I20260709 12:16:30.492746 23363 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.589298 23196 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:30.626681 12488 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:30.563237 23406 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.629143 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:30.598207 23321 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:30.580540 23398 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.631227 23196 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:30.629726 12637 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:30.675017 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4220) took 1091 ms. Trace:
I20260709 12:16:30.673012 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:30.675513 12643 rpcz_store.cc:276] 0709 12:16:29.550535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.555186 (+  4651us) service_pool.cc:224] Handling call
0709 12:16:30.641685 (+1086499us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:30.641882 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2140576640}
I20260709 12:16:30.676255 13260 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:30.605233 23409 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:30.623329 23385 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:30.676626 13260 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:30.626024 23428 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.689352 13255 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:30.694355 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.702605 12490 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.20-35.20]
I20260709 12:16:30.706347 23428 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.707034 13260 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.683562 23385 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:30.710237 23385 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:30.711416 23293 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:30.619400 23133 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.713238 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.713689 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.713938 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.713199 23133 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:30.715858 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:30.716652 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:16:30.713126 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1519) took 1137 ms. Trace:
I20260709 12:16:30.717923 23398 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.707211 23428 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.679392 23196 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:30.720595 23428 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.721263 23428 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:30.722044 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.723249 23268 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.717334 12645 rpcz_store.cc:276] 0709 12:16:29.575898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.610520 (+ 34622us) service_pool.cc:224] Handling call
0709 12:16:29.611238 (+   718us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:29.622034 (+ 10796us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:29.622065 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:29.622583 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:16:29.985553 (+362970us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:16:29.985763 (+   210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:29.986364 (+   601us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:30.712845 (+726481us) raft_consensus.cc:1685] finished
0709 12:16:30.712877 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:30.713056 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1141,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":362235,"threads_started":1,"wal-append.queue_time_us":1001704}
I20260709 12:16:30.728341 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4393) took 1029 ms. Trace:
I20260709 12:16:30.728802 12569 rpcz_store.cc:276] 0709 12:16:29.699438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.699948 (+   510us) service_pool.cc:224] Handling call
0709 12:16:29.700331 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:29.700758 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:29.700802 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:29.701497 (+   695us) log.cc:844] Serialized 26 byte log entry
0709 12:16:30.425161 (+723664us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:30.425228 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:30.425514 (+   286us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:30.728020 (+302506us) raft_consensus.cc:1685] finished
0709 12:16:30.728048 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:30.728271 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1017,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":15244,"thread_start_us":281108,"threads_started":1,"wal-append.queue_time_us":916985}
I20260709 12:16:30.714072 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:30.726305 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1606) took 1409 ms. Trace:
I20260709 12:16:30.737494 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.734870 12491 rpcz_store.cc:276] 0709 12:16:29.317192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.317369 (+   177us) service_pool.cc:224] Handling call
0709 12:16:29.319406 (+  2037us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:29.319859 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:29.319891 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:29.391505 (+ 71614us) log.cc:844] Serialized 26 byte log entry
0709 12:16:29.648966 (+257461us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:16:29.649016 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:29.649095 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:30.725938 (+1076843us) raft_consensus.cc:1685] finished
0709 12:16:30.725987 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:30.726212 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":743,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":256922,"threads_started":1,"wal-append.queue_time_us":1297552}
I20260709 12:16:30.641606 23321 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:30.740216 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 305991us
I20260709 12:16:30.747114 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:30.750846 13253 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:30.750612 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.634014 23458 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.110s
I20260709 12:16:30.741842 23428 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:30.772694 12494 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.054s	sys 0.069s
W20260709 12:16:30.773320 12494 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.054s	sys 0.070s
I20260709 12:16:30.762856 23368 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.766078 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:30.742393 23321 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:30.770967 13260 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:30.781359 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:30.771004 23406 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.799679 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.802466 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4564) took 1167 ms. Trace:
I20260709 12:16:30.809188 12494 rpcz_store.cc:276] 0709 12:16:29.634702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.700621 (+ 65919us) service_pool.cc:224] Handling call
0709 12:16:29.701010 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:29.701269 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:16:30.236393 (+535124us) log.cc:844] Serialized 14 byte log entry
0709 12:16:30.801962 (+565569us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:30.802050 (+    88us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:30.802113 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:30.802210 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:30.802387 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28005,"prepare.queue_time_us":807904,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":681703,"threads_started":2}
I20260709 12:16:30.809894 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.789175 23393 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:30.819687 12639 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.001s	sys 0.042s
W20260709 12:16:30.820263 12639 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.001s	sys 0.042s
I20260709 12:16:30.815464 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:30.807910 23368 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.797845 23467 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:16:30.821870 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 376746us
I20260709 12:16:30.858350 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:30.850349 12638 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.019s	sys 0.024s
I20260709 12:16:30.821378 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4206) took 1417 ms. Trace:
W20260709 12:16:30.850644 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.011s	sys 0.024s
W20260709 12:16:30.868404 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.012s	sys 0.024s
W20260709 12:16:30.868196 12638 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.019s	sys 0.024s
I20260709 12:16:30.871376 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4209) took 1436 ms. Trace:
I20260709 12:16:30.875895 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.875761 12565 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:30.876865 12570 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:16:30.877435 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.878093 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:30.878239 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:30.879077 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:30.826284 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.010s	sys 0.025s
I20260709 12:16:30.879904 12564 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 14.
W20260709 12:16:30.880295 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.010s	sys 0.025s
I20260709 12:16:30.882290 12570 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:16:30.880849 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:30.883519 12564 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:30.880038 13258 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:30.886433 12563 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:30.868240 12639 rpcz_store.cc:276] 0709 12:16:29.404363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.404504 (+   141us) service_pool.cc:224] Handling call
0709 12:16:29.404915 (+   411us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:29.525078 (+120163us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:29.525128 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:29.525977 (+   849us) log.cc:844] Serialized 52 byte log entry
0709 12:16:29.988225 (+462248us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:30.240658 (+252433us) log.cc:844] Serialized 14 byte log entry
0709 12:16:30.820742 (+580084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:30.820866 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:30.820936 (+    70us) raft_consensus.cc:1685] finished
0709 12:16:30.820966 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:30.821293 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":232062,"prepare.queue_time_us":265542,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":173185,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":30255,"thread_start_us":945172,"threads_started":2,"wal-append.queue_time_us":916877}
W20260709 12:16:30.884013 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 660151us
I20260709 12:16:30.893235 12610 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:30.892362 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1363) took 2167 ms (client timeout 2160 ms). Trace:
I20260709 12:16:30.871872 12638 rpcz_store.cc:276] 0709 12:16:29.434803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.434968 (+   165us) service_pool.cc:224] Handling call
0709 12:16:29.451755 (+ 16787us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:29.475135 (+ 23380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:29.475172 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:29.546714 (+ 71542us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:29.561032 (+ 14318us) log.cc:844] Serialized 52 byte log entry
0709 12:16:29.841788 (+280756us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:30.144486 (+302698us) log.cc:844] Serialized 14 byte log entry
0709 12:16:30.261662 (+117176us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b1800480f40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:16:30.262075 (+   413us) log.cc:844] Serialized 14 byte log entry
0709 12:16:30.530149 (+268074us) spinlock_profiling.cc:243] Waited 6.36 ms on lock 0x7b1800480f40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:30.870870 (+340721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:30.870992 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:30.871060 (+    68us) raft_consensus.cc:1685] finished
0709 12:16:30.871081 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:30.871298 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":174359,"prepare.queue_time_us":310949,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":110,"raft.queue_time_us":162899,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":9339,"spinlock_wait_cycles":163887360,"thread_start_us":846739,"threads_started":2,"wal-append.queue_time_us":1123367}
W20260709 12:16:30.894840 13255 rpcz_store.cc:269] 0709 12:16:28.712792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:28.713611 (+   819us) service_pool.cc:224] Handling call
0709 12:16:30.879628 (+2166017us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:30.879755 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4265016064}
I20260709 12:16:30.895309 23466 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.014s
I20260709 12:16:30.896073 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:30.824276 22734 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:30.898634 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:30.903347 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:30.898882 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.905062 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:30.902206 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.906488 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:30.898996 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.916316 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4568) took 1275 ms. Trace:
I20260709 12:16:30.916875 12490 rpcz_store.cc:276] 0709 12:16:29.641175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.729679 (+ 88504us) service_pool.cc:224] Handling call
0709 12:16:30.090563 (+360884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:30.703533 (+612970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:30.915922 (+212389us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:30.916215 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1388847488}
I20260709 12:16:30.922720 23437 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:30.868166 23390 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.927366 22606 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:30.927861 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:30.881307 23339 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:30.894048 23467 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:30.892447 23301 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:30.930913 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:30.909185 23411 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:30.906898 23264 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:30.967742 23165 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:30.852334 23368 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:30.971213 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:30.973843 22606 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:16:30.968686 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64964.8us
I20260709 12:16:30.986552 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.987613 12564 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:30.988409 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.938868 23301 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:30.989249 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:30.993326 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:30.975351 23411 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:31.029752 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 395268us
I20260709 12:16:31.031805 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:30.999462 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:31.033504 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:31.035748 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.036688 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:30.996336 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67345.2us
I20260709 12:16:31.033838 12565 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:31.039633 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:31.014288 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.047070 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.047461 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:31.048295 12563 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:30.935993 23437 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.048854 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:31.052340 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:30.993170 23237 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:30.995379 23301 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:31.058271 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67494.7us
W20260709 12:16:31.058326 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60025.7us
I20260709 12:16:31.063301 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:30.972848 23339 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:30.997053 23477 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.001782 23462 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:30.967876 23196 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.035631 23411 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:30.967245 23467 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.071259 13261 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:30.992408 23385 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.083745 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:31.070895 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:31.049345 23474 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:31.108734 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4388) took 1433 ms. Trace:
I20260709 12:16:31.115192 12571 rpcz_store.cc:276] 0709 12:16:29.675307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.675432 (+   125us) service_pool.cc:224] Handling call
0709 12:16:29.702074 (+ 26642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:29.702252 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:16:29.758028 (+ 55776us) log.cc:844] Serialized 14 byte log entry
0709 12:16:30.914672 (+1156644us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b180035ff20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:16:31.108225 (+193553us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:31.108307 (+    82us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:31.108377 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:31.108484 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:31.108658 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19437,"prepare.queue_time_us":55797,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"spinlock_wait_cycles":51670912,"thread_start_us":173757,"threads_started":1,"wal-append.queue_time_us":1020677}
I20260709 12:16:31.067508 23291 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:16:31.117699 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46901.2us
W20260709 12:16:31.125113 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61923.8us
I20260709 12:16:31.120040 13260 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:31.126184 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:31.127810 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:31.128515 22734 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:31.129263 23165 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:31.129858 23165 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:31.130276 23471 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:16:31.130295 23165 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:31.110620 23478 raft_consensus.cc:2764] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:16:31.139096 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3439) took 1737 ms. Trace:
I20260709 12:16:31.139611 12489 rpcz_store.cc:276] 0709 12:16:29.401244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.480457 (+ 79213us) service_pool.cc:224] Handling call
0709 12:16:31.138807 (+1658350us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:31.139015 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2074476928}
W20260709 12:16:31.141520 23395 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.579s	user 0.009s	sys 0.000s
I20260709 12:16:31.177012 23395 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:31.073225 23394 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:31.178861 23394 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:31.158795 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4573) took 1430 ms. Trace:
I20260709 12:16:31.143821 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:31.153438 23439 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:31.175745 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:31.182451 23367 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:31.183163 23367 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:31.184180 23367 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.110875 23479 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:16:31.187428 23479 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:31.187981 23479 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:31.068413 23237 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:31.191489 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4205) took 1787 ms (client timeout 2036 ms). Trace:
I20260709 12:16:31.191744 23237 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:31.192019 12640 rpcz_store.cc:269] 0709 12:16:29.403994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.404231 (+   237us) service_pool.cc:224] Handling call
0709 12:16:31.191265 (+1787034us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:31.191415 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1842185600}
I20260709 12:16:31.170473 23345 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:31.197997 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:31.070382 23307 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.133s
W20260709 12:16:31.121765 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58370us
I20260709 12:16:31.198896 13260 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:16:31.147873 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.179545 23394 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:31.180935 12492 rpcz_store.cc:276] 0709 12:16:29.728056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:30.456261 (+728205us) service_pool.cc:224] Handling call
0709 12:16:30.456639 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:30.671449 (+214810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:30.671490 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:30.671542 (+    52us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:30.671581 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:31.158432 (+486851us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:31.158504 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:31.158704 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493378944}
I20260709 12:16:31.198478 12565 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
W20260709 12:16:31.192430 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49093.9us
I20260709 12:16:31.207829 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.208812 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:31.210979 23476 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:31.083544 23428 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.079371 23462 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:31.205466 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79409.3us
I20260709 12:16:30.997364 23465 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.129s
I20260709 12:16:31.216514 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:31.181042 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:31.237857 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:31.185412 23233 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:31.244158 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63217.1us
I20260709 12:16:31.147922 23221 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:31.125380 23291 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:31.214267 23384 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:31.243845 23473 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.002s
I20260709 12:16:31.246089 23472 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:31.244558 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:31.075594 23477 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.213544 23476 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:31.266790 23476 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:31.242362 23307 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:31.250260 23105 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:16:31.258453 23469 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
I20260709 12:16:31.245275 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:31.148526 23165 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.243223 23393 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:31.275702 23393 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:31.261606 23291 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:31.279812 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:31.280646 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:31.280988 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:31.285420 12641 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:31.286341 22606 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.287024 22606 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.290608 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:31.270524 23307 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:31.296636 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:31.298391 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:31.278554 23338 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:31.250765 23233 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.283840 23384 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:31.338833 22606 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:31.277312 23393 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.344107 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:31.350454 23374 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.851s	user 0.017s	sys 0.000s
I20260709 12:16:31.351239 23374 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.282470 23417 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:31.322726 23480 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:16:31.304420 23221 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:31.324165 23233 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:31.326819 23391 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:31.331863 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:31.333156 23433 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:31.337788 23339 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:31.325423 13256 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:31.356330 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:31.295868 23133 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.578s	user 0.009s	sys 0.005s
W20260709 12:16:31.357697 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4569) took 1708 ms (client timeout 2041 ms). Trace:
I20260709 12:16:31.360783 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.362242 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:31.320030 12641 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:31.365820 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:31.372795 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:31.357568 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:31.375385 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.378067 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:31.295228 23333 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:31.378669 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:31.379732 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.381541 13260 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:31.399338 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:31.399996 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:31.403190 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1509) took 1950 ms (client timeout 1890 ms). Trace:
W20260709 12:16:31.403913 12646 rpcz_store.cc:269] 0709 12:16:29.452346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.452478 (+   132us) service_pool.cc:224] Handling call
0709 12:16:31.402884 (+1950406us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:31.403082 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2390941568}
W20260709 12:16:31.403646 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:31.412744 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.294629 23476 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.426944 23476 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:31.427763 22606 consensus_queue.cc:1175] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 30 status { last_received { term: 30 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 30 index: 16 } }
W20260709 12:16:31.403952 12495 rpcz_store.cc:269] 0709 12:16:29.649114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:29.651787 (+  2673us) service_pool.cc:224] Handling call
0709 12:16:31.357444 (+1705657us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:31.357594 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612065664}
I20260709 12:16:31.431690 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:31.433743 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76262.8us
I20260709 12:16:31.401050 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
W20260709 12:16:31.382983 23410 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.895s	user 0.014s	sys 0.002s
I20260709 12:16:31.436407 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:31.454149 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:31.454901 12492 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:31.397464 23433 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:31.459795 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:31.380604 23391 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:31.470631 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 433966us
I20260709 12:16:31.474565 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:31.449146 23473 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:31.455873 23133 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:31.373256 23417 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.456151 23333 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:31.462483 23465 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:31.500838 12565 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:16:31.382277 23221 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.489423 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:31.531000 12492 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:31.493423 23472 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:31.462057 23410 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.505712 22734 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:31.505616 23385 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:31.624925 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 314755us
I20260709 12:16:31.450728 23307 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.555238 23473 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:31.588642 23465 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.592811 23472 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:31.637326 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 147979us
I20260709 12:16:31.632903 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:31.640064 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:31.588264 23133 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:31.597548 23362 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.641162 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:31.638176 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 259606us
I20260709 12:16:31.642058 13254 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
W20260709 12:16:31.636334 23321 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.845s	user 0.016s	sys 0.000s
W20260709 12:16:31.645013 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.649926 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:31.625727 22734 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:31.659168 23345 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:31.638600 23462 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.642601 23465 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:31.645738 23264 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:31.648092 23483 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:31.643029 23472 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:31.628705 23394 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.660596 22734 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:31.693866 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:31.665978 23477 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.657799 23428 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:31.681301 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:31.688354 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 326355us
I20260709 12:16:31.696791 13262 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:31.715181 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:31.656576 23321 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.670567 23307 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:31.638432 23391 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.655314 23362 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.688156 23466 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:31.680158 23483 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:31.710848 23374 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:31.725812 23488 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:16:31.712843 23428 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:31.744375 23362 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:31.770670 23428 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:31.746394 23417 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:31.756563 23337 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:31.774151 23337 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:31.776305 23474 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.000s
I20260709 12:16:31.787994 23474 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:16:31.765254 23410 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.788130 23472 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.766736 23394 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.727075 23497 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:31.738073 23483 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:31.768261 23439 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.002s
I20260709 12:16:31.847972 12563 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:32.028420 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:31.688216 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.873158 12638 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:31.876746 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "472e786b46074d868d00df67a0985afd"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:31.882254 23133 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:31.893196 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.895239 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:31.896360 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:31.778154 23301 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.702s	user 0.010s	sys 0.003s
I20260709 12:16:31.677537 23333 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.042348 23301 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.775746 23337 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.045866 23301 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:31.900434 23264 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.046794 23337 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:32.047765 12492 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:32.047256 23264 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.901644 23367 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:31.901762 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:31.903228 23362 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:31.905181 23293 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:31.907203 13287 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (10 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:16:31.920307 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1370) took 1483 ms. Trace:
W20260709 12:16:31.932366 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:32.053331 23362 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.054522 23293 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:32.055867 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.939136 23368 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:31.947382 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:31.949643 23165 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.951753 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.965976 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.059665 12569 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:31.968850 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:31.973650 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:31.975057 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:31.981791 23221 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.983933 23321 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:31.985050 23196 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:31.988484 23391 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:31.992560 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:31.994733 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:31.995497 12488 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
W20260709 12:16:31.763006 23411 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.655s	user 0.011s	sys 0.001s
I20260709 12:16:31.786963 23433 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.081740 23437 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:31.732338 23466 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:32.086771 23466 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:32.087831 23411 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:31.784366 23465 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.090276 12490 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:32.090808 22734 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:32.091629 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:32.092235 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:32.093086 23465 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.094691 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:31.780336 23479 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.592s	user 0.007s	sys 0.005s
I20260709 12:16:32.037854 13258 raft_consensus.cc:606] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:16:32.114244 13258 raft_consensus.cc:614] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:32.114614 23479 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.104883 23362 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.107369 22734 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.033761 23477 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.118649 23477 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.119197 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4405) took 1110 ms. Trace:
I20260709 12:16:32.122318 23433 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:32.122712 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 481471us
I20260709 12:16:32.125268 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:32.125928 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 447222us
I20260709 12:16:32.126688 12497 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:16:32.127923 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 477932us
I20260709 12:16:32.053990 13254 rpcz_store.cc:276] 0709 12:16:30.436226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:30.436744 (+   518us) service_pool.cc:224] Handling call
0709 12:16:30.437055 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:31.644013 (+1206958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:31.919465 (+275452us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:31.919982 (+   517us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2770713344}
I20260709 12:16:32.074767 13262 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:32.131332 23467 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.079698 12496 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:32.090139 12571 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:31.804473 23472 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.137542 12488 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
W20260709 12:16:32.139158 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65496.1us
I20260709 12:16:31.722306 23473 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.15, Last appended by leader: 15, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.140372 12489 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:32.097043 23478 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.142448 23478 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.144479 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.146291 12567 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:16:32.103560 12491 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:32.050323 23367 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.047369 23502 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:16:32.108124 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.109468 23411 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.111332 23293 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.136107 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.119635 12563 rpcz_store.cc:276] 0709 12:16:31.008950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:31.112411 (+103461us) service_pool.cc:224] Handling call
0709 12:16:31.112781 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:32.028158 (+915377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:32.118783 (+ 90625us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:32.119085 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1685558144}
I20260709 12:16:32.163003 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:32.175819 23293 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:32.176508 12490 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.20-35.20]
I20260709 12:16:32.177193 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.178153 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.181994 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.183094 12491 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:32.184048 23479 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.185288 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.187937 12564 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:16:32.188791 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.178156 12489 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:32.190929 12565 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:32.193732 12568 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:32.185194 12496 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
W20260709 12:16:32.188741 23105 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.917s	user 0.010s	sys 0.020s
I20260709 12:16:32.202427 12492 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:32.205053 23345 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:32.210314 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.216691 13262 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:32.218075 12494 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:32.221858 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:32.167675 23467 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.056s
W20260709 12:16:32.224001 23515 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:16:32.224795 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.159597 23506 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:16:32.226459 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.228215 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.224822 23467 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:32.230872 23467 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:32.232123 23467 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.217367 23105 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:32.234099 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.235869 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.237576 12490 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:16:32.240788 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.241782 12488 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:32.242126 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:32.244735 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:32.246069 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.247151 12491 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:32.247706 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.248734 12571 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:32.249341 12490 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:32.221745 23293 consensus_queue.cc:1243] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 30.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:16:32.233583 23467 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:32.253990 23477 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:32.250526 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72590.2us
I20260709 12:16:32.262387 23264 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.261814 12493 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:32.266359 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:32.267961 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51420.3us
I20260709 12:16:32.268390 12490 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:32.270136 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:32.270864 23495 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:32.274163 23495 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:32.274333 12563 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:32.274602 23495 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:32.279515 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:32.281989 23171 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:32.285567 23465 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.007s
I20260709 12:16:32.285887 23477 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:32.332006 12609 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:32.291368 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:32.333104 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.336977 12609 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:32.291594 23367 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.291956 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.292232 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:32.292640 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.298792 23478 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:32.300895 23291 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.012s	user 0.010s	sys 0.019s
I20260709 12:16:32.301476 23502 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:32.308565 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:32.319619 23462 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:32.344959 23462 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:32.346297 23462 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:32.322247 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52231.8us
I20260709 12:16:32.350637 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:32.351676 23391 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:32.352551 23391 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:32.353672 23391 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.357187 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.357681 23466 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.360445 23466 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.327296 12491 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:32.328758 23105 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:32.314028 12565 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:32.367296 23105 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:32.367779 23105 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:32.332990 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.370168 12490 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:32.332602 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.289482 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.341081 13262 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:32.338483 13258 raft_consensus.cc:606] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:32.371241 13258 raft_consensus.cc:614] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:32.342283 13254 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:32.342262 23502 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.343122 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.372670 12566 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:16:32.343278 23291 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.344069 12569 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:32.326328 12489 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:32.365144 23477 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:32.365468 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:32.353427 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.382930 12564 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:32.261428 23428 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.386135 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.368839 23105 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.395936 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:16:32.397604 12570 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:32.394205 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:32.399473 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68496.1us
I20260709 12:16:32.387074 23465 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:32.406133 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:32.410399 23514 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:32.411638 23477 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.440407 23333 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:32.441572 23333 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:32.442170 23333 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:32.441669 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:32.443876 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:32.445963 12536 connection.cc:256] RPC call timeout handler was delayed by -2.8545911590001651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260709 12:16:32.446161 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:32.446516 12536 connection.cc:256] RPC call timeout handler was delayed by -2.78618606200007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
W20260709 12:16:32.446928 12536 connection.cc:256] RPC call timeout handler was delayed by -2.71689115900017s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1839s for a response
W20260709 12:16:32.447422 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4605678820000776s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20260709 12:16:32.447927 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0696134610002446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
W20260709 12:16:32.448274 12536 connection.cc:256] RPC call timeout handler was delayed by -2.031968445000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260709 12:16:32.454193 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8870684450000681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20260709 12:16:32.398330 23479 raft_consensus.cc:2764] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election decision vote started in defunct term 38: won
W20260709 12:16:32.457274 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8663684450000346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 12:16:32.457731 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8219233090001126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
W20260709 12:16:32.458022 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8150684449999517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
W20260709 12:16:32.458308 12536 connection.cc:256] RPC call timeout handler was delayed by -1.7034684449999986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
W20260709 12:16:32.458581 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4513882890000787s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
W20260709 12:16:32.458855 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3289882890001081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1714s for a response
W20260709 12:16:32.459117 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3244882890001008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 12:16:32.459447 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1660882890000721s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20260709 12:16:32.459720 12536 connection.cc:256] RPC call timeout handler was delayed by -1.13998828900003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260709 12:16:32.459970 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0994882890001918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
W20260709 12:16:32.460245 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0301882890000797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20260709 12:16:32.422891 23411 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:32.424116 23105 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:32.463270 23411 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:32.463758 23411 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:32.426339 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:32.426482 23291 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.375988 23171 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.472358 23171 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.428499 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:32.433495 23495 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.436285 23345 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:32.476380 13258 raft_consensus.cc:606] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:32.476838 13258 raft_consensus.cc:614] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:32.477015 23495 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.478019 13261 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:32.474037 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.440058 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:32.419471 13260 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:32.479784 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.449355 23333 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:32.449774 23483 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.606s	user 0.018s	sys 0.000s
I20260709 12:16:32.483855 23291 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:32.483963 23483 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.484567 23291 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:32.485100 23291 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:32.485442 23479 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:32.485931 12572 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:32.486470 13256 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:32.449000 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.488703 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.451920 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:32.452005 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.451798 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2051) took 1450 ms. Trace:
I20260709 12:16:32.490319 12570 rpcz_store.cc:276] 0709 12:16:31.001655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:31.070709 (+ 69054us) service_pool.cc:224] Handling call
0709 12:16:31.071131 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:32.423099 (+1351968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:32.451493 (+ 28394us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:32.451699 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3016260992}
I20260709 12:16:32.491027 13256 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:32.491712 13256 raft_consensus.cc:606] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.492100 13256 raft_consensus.cc:614] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:32.449892 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.452445 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.492254 12570 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:16:32.492908 12497 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:32.492872 23385 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:16:32.494149 23385 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:32.494544 23385 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:32.454839 12569 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:32.495445 23385 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.496538 12569 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:16:32.497200 23385 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:32.416288 23340 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 27
W20260709 12:16:32.497740 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:32.406636 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.415445 23502 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.498502 12566 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:32.499544 23483 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.482686 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:32.484352 23333 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:32.500958 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:32.449779 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.480238 23509 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:32.450110 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 158326us
I20260709 12:16:32.502220 12572 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:32.503077 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.492381 23345 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:32.504881 23345 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:32.505302 23345 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:32.505908 23483 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:32.506448 23483 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:32.506808 23483 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:32.507150 23514 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:32.507751 23514 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:32.508754 23514 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.509981 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.509814 23362 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:32.511104 23458 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:32.511507 23458 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:32.512247 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.513447 23264 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:32.513862 23465 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:32.514473 23264 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:32.452041 23428 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.515822 23362 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:16:32.517083 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.517367 12565 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:32.512542 23458 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.524706 23458 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:32.530292 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:32.533126 13261 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:32.530165 23428 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:32.533339 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:32.534775 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51206.2us
I20260709 12:16:32.534430 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:32.535833 23502 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:32.536167 13258 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:32.537529 23509 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:32.538030 23509 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:32.537999 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 184685us
I20260709 12:16:32.497428 13254 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:16:32.459120 12564 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:16:32.420239 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:32.564164 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
W20260709 12:16:32.571002 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58880.9us
W20260709 12:16:32.576964 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156831us
I20260709 12:16:32.582548 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:32.583401 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.584257 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:32.573549 23340 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:32.601846 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:32.602823 23391 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:32.604043 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.478099 23411 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.605978 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:16:32.606437 23495 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:32.610659 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:32.611565 23391 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:16:32.611565 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 68580.7us
I20260709 12:16:32.613922 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.614771 23333 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:32.616214 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:32.606623 23411 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.620642 12572 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:16:32.622008 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:32.612277 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:32.590840 23368 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:16:32.640303 13256 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:32.642192 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:32.652849 13261 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:32.623495 23428 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election decision vote started in defunct term 32: won
I20260709 12:16:32.624926 23479 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:32.567950 23509 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.671705 13254 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:32.631479 23165 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.679937 23340 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:32.684445 23427 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.684288 23340 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.685304 23427 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.685878 23470 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.020s
I20260709 12:16:32.663224 22734 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:32.689375 23509 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:32.689955 22734 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.692162 22734 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.692203 23105 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.693017 23105 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.693518 12610 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:32.655092 23368 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:32.637809 23345 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.666529 23133 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:32.701510 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:32.687139 23340 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.689388 23333 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:32.710491 23333 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.711004 23427 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.712096 23368 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.713102 23333 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.714078 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:32.714843 12564 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:32.718082 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.726079 23105 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:32.731262 23495 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.732129 23495 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:32.743827 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130026us
I20260709 12:16:32.745138 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:32.746299 12495 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:32.748245 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1621) took 1295 ms. Trace:
I20260709 12:16:32.748713 12495 rpcz_store.cc:276] 0709 12:16:31.453071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:31.453380 (+   309us) service_pool.cc:224] Handling call
0709 12:16:31.453936 (+   556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:32.746946 (+1293010us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:32.747870 (+   924us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540012c5d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:32.748141 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2618365696}
I20260709 12:16:32.662297 23339 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:16:32.698530 23458 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:16:32.753078 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48603.2us
W20260709 12:16:32.777886 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 215236us
I20260709 12:16:32.777026 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.704221 23171 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:32.803579 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148884us
W20260709 12:16:32.788021 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:32.814617 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:32.784328 23458 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:32.772830 23427 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.039s
I20260709 12:16:32.751339 23165 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.807442 23465 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:32.809300 23293 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.834726 23411 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.834651 23293 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.835462 23411 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.837134 23293 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.836971 23291 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.830662 23340 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:32.839138 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:32.839593 23291 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.831969 23427 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:32.840433 23427 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:32.840616 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:32.841521 23427 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.843073 23427 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:32.847349 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:32.851737 23458 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.854352 13258 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:32.855706 23514 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:32.856426 23514 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.832985 23495 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.837524 23411 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.832068 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:32.832279 23165 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:32.816504 23133 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:32.870332 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 161226us
I20260709 12:16:32.913311 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:32.915287 12491 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 38 because replica has last-logged OpId of term: 37 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 22.
I20260709 12:16:32.913283 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.916550 12497 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:32.876175 12564 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:32.839710 23340 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:32.886055 13258 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:16:32.920868 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.819142 23171 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:32.890944 23345 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:32.880965 23339 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:16:32.892660 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.894025 23483 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:32.896683 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:32.898113 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:32.898494 12490 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:16:32.899492 12493 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:32.902858 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:32.874342 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45762.5us
I20260709 12:16:32.931814 12494 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 38 because replica has last-logged OpId of term: 37 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 22.
I20260709 12:16:32.933718 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:32.937914 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:32.928118 23428 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:32.946401 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:32.940181 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:32.959314 12569 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:32.960939 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:32.972282 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:33.003460 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 361346us
I20260709 12:16:32.957511 23105 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:32.935817 23293 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.028s
I20260709 12:16:33.031083 13258 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:16:32.966341 23339 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:33.010299 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:33.033851 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 201956us
I20260709 12:16:33.058120 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:33.058470 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48279.4us
I20260709 12:16:33.058842 13254 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:33.063364 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:33.062794 23340 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.069883 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 324906us
I20260709 12:16:33.063961 13260 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:33.037039 23339 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:33.030826 23345 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:33.054466 23367 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:33.071141 23165 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.008s
I20260709 12:16:33.080780 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.082998 23339 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:33.061502 23105 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.090967 23333 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:33.091575 23333 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.091171 23105 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.098241 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:33.097429 23533 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:16:33.144903 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:33.146764 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 185943us
I20260709 12:16:33.155891 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:33.157829 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:33.143719 23428 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:33.116636 23345 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:33.168433 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93902.1us
I20260709 12:16:33.157210 23495 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:33.170017 23264 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:33.181030 23264 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:33.181722 23473 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:33.182224 23264 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.183359 23538 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:33.183933 23538 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:33.184290 23538 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:33.185726 23525 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.002s	sys 0.000s
I20260709 12:16:33.186909 23498 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:33.168049 23534 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:33.162246 23533 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.192346 23541 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:33.193151 23541 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.193020 23539 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:33.195788 23541 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.195519 23539 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.197958 23171 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:33.198741 23539 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.199671 23473 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.201031 23473 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.205549 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109797us
I20260709 12:16:33.148013 23367 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:33.208935 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 278285us
I20260709 12:16:33.212148 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.223245 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:33.223733 23345 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.226468 23534 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.228740 23534 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:33.230445 23509 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.540s	user 0.013s	sys 0.000s
W20260709 12:16:33.231607 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53698.4us
I20260709 12:16:33.232913 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.233954 23291 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:33.235117 23293 raft_consensus.cc:2764] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:16:33.234514 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:33.238090 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.236119 23293 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:33.243749 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.244572 23477 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:33.245467 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.266901 12564 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:33.245127 23362 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Committing config change with OpId 21.14: config changed from index 7 to 14, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:33.247654 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.211539 23498 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.272076 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.249545 23367 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.251713 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:33.234949 23483 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.257287 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.260080 12566 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:33.240312 23293 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:33.281126 12567 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:33.240676 23514 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:33.280324 23291 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:33.291679 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.275457 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:33.276818 12571 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:33.278863 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.298455 23339 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.302484 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.303660 12638 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:33.306341 12570 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 15. Preceding OpId from leader: term: 34 index: 16. (index mismatch)
I20260709 12:16:33.296448 23264 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:33.307580 23264 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:33.307945 23264 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:33.314682 23293 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.316048 23293 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:33.312768 23333 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.332746 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:33.339402 13256 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:33.342959 23339 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.343823 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:33.349856 12565 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:33.353098 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:33.355562 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:33.357121 13261 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:33.358204 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:33.362180 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.363209 12569 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:33.364514 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:33.366741 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123114us
W20260709 12:16:33.371273 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119676us
I20260709 12:16:33.369460 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:33.301208 23514 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.051s
I20260709 12:16:33.375275 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.375465 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:33.376346 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.347136 23540 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:33.354583 23368 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:33.375761 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:33.367478 23509 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:33.370750 23345 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:33.373893 23543 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:33.374959 23411 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:16:33.391656 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124699us
I20260709 12:16:33.403386 12570 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:33.394384 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:33.396909 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.397477 23506 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:33.401885 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:33.398229 23549 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:16:33.385354 23368 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:33.399155 23543 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:33.415946 23472 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:33.416553 23472 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.406286 23411 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:33.417532 23472 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.418188 23411 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:33.419193 23472 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:33.414449 23333 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:33.416038 23543 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:33.424746 23467 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:16:33.425280 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:33.425318 22734 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:33.426079 22734 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.426921 22734 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.428017 22734 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:33.436801 23337 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:16:33.438071 23340 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:33.438567 23340 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:33.439440 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52917.3us
I20260709 12:16:33.439956 12572 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:16:33.439710 23340 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.440482 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.441267 23340 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:33.442337 12565 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Committing config change with OpId 21.14: config changed from index 7 to 14, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:33.443675 23337 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:33.412317 23427 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:33.412084 23470 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:16:33.389415 23483 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.060s
I20260709 12:16:33.469941 23483 consensus_queue.cc:1243] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:33.387699 23493 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:33.471931 23493 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.463331 23291 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:33.366654 23533 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.475050 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:33.453035 12495 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 23. (term mismatch)
I20260709 12:16:33.475366 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.475824 12495 pending_rounds.cc:85] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 22
I20260709 12:16:33.476084 12495 pending_rounds.cc:107] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 37.23
I20260709 12:16:33.476414 12495 raft_consensus.cc:2889] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:33.477022 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.479831 23502 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:16:33.480348 23502 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.407408 23345 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:33.481946 23467 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:16:33.482378 23345 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.453886 23514 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:33.456364 23368 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:33.465032 23427 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:33.485980 23427 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:33.487557 23467 raft_consensus.cc:2764] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:16:33.488099 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:33.460093 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50547.6us
I20260709 12:16:33.469205 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.413440 23509 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.468864 23337 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.490087 12638 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:33.491393 23509 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:33.399381 23540 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.493886 23540 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:33.466596 12639 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:33.498147 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.474618 23493 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.451719 23498 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:33.488837 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:33.494611 23391 raft_consensus.cc:993] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:33.494527 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.515198 12568 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:33.514993 12563 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
W20260709 12:16:33.519857 23544 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260709 12:16:33.523767 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:33.525954 23467 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:33.526711 23467 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:33.527226 23467 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:33.528239 23467 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.530333 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70611.8us
I20260709 12:16:33.530624 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.531251 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:33.532514 23391 raft_consensus.cc:1081] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:33.542066 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:33.551415 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:33.494997 23428 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:33.559298 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:33.561255 12490 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:33.559347 12496 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:33.572883 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:33.481223 23502 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.577257 23545 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:33.578055 12493 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:16:33.582683 12572 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:33.587217 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:33.583846 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.506222 23458 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.065s
W20260709 12:16:33.590767 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 215355us
I20260709 12:16:33.546212 12563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:16:33.592003 12563 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:33.592517 12563 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:33.598155 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:33.600553 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:33.601403 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.601964 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.602411 12641 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:16:33.603374 23540 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:33.603950 23540 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:33.607107 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.612118 12489 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:33.603799 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.616648 12490 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:33.601999 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:33.595503 12495 pending_rounds.cc:85] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:16:33.618397 12495 pending_rounds.cc:107] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:16:33.618806 12495 raft_consensus.cc:2889] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:33.613341 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:33.620124 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:33.621899 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:33.622532 12493 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:33.583408 23493 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:33.622710 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:33.631193 12490 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:33.638816 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:33.641333 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110180us
I20260709 12:16:33.609153 23264 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.646988 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52233.9us
I20260709 12:16:33.644601 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:33.611441 23477 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:33.655536 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:33.605094 23540 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.662367 22734 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:33.636938 23473 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:33.665710 23473 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:33.630740 23493 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:33.672430 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:33.653797 23535 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.010s
I20260709 12:16:33.677688 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.686766 12639 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:33.660760 23428 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:33.695868 12563 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.699364 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.695008 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.702405 23411 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.675853 23493 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:33.705338 23495 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.002s
I20260709 12:16:33.662843 23545 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:33.709259 12572 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:33.708868 23545 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.711443 23545 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:33.711287 23470 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:16:33.712523 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.682742 23514 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:33.713644 23514 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:33.714001 23514 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:33.718250 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.718961 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:33.719893 23362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54574 (request call id 0) took 2397 ms. Trace:
I20260709 12:16:33.694726 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.729126 12642 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:33.666204 23473 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:33.747594 12493 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:33.751820 12563 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.720337 23362 rpcz_store.cc:276] 0709 12:16:31.322827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:31.323006 (+   179us) service_pool.cc:224] Handling call
0709 12:16:31.492761 (+169755us) log.cc:844] Serialized 501 byte log entry
0709 12:16:33.719753 (+2226992us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":205,"raft.queue_time_us":179725,"raft.run_cpu_time_us":2333,"raft.run_wall_time_us":2332,"thread_start_us":285272,"threads_started":1,"wal-append.queue_time_us":239319,"wal-append.run_cpu_time_us":3041,"wal-append.run_wall_time_us":1059249}
W20260709 12:16:33.757707 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 133055us
I20260709 12:16:33.675518 23534 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:33.757383 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93269.7us
I20260709 12:16:33.678344 23337 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.767024 23534 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.765278 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:33.771934 13260 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:33.772320 13260 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:33.717511 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49363.8us
I20260709 12:16:33.773093 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.751475 23291 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:33.722177 23428 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:33.755120 23411 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:33.794642 12642 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
W20260709 12:16:33.808461 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:33.809343 12638 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:33.804464 13260 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.752282 23543 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.823673 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:33.790221 23428 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:33.756441 23477 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.831782 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:33.838490 12642 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
W20260709 12:16:33.841004 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72143us
W20260709 12:16:33.834887 23538 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.646s	user 0.010s	sys 0.004s
I20260709 12:16:33.849658 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.850476 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:33.842250 23293 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:33.820825 23368 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.887722 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.886128 23291 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:33.918239 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
W20260709 12:16:33.915489 12461 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.907s (SENT)
I20260709 12:16:33.920977 23493 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:33.922674 23472 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.001s
I20260709 12:16:33.931807 12646 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:16:33.896891 23264 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:33.916435 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:33.936070 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:33.937681 12496 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:33.937891 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:33.896415 23556 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:16:33.939353 13261 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:16:33.918597 23345 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:33.849524 23538 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.943758 23534 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:33.946499 23477 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:33.947113 23543 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:16:33.947990 23560 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260709 12:16:33.940927 23559 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260709 12:16:33.979943 23333 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.557s	user 0.004s	sys 0.007s
I20260709 12:16:33.983021 23514 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:33.896157 12381 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 13)
W20260709 12:16:33.985496 12381 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89433.6us
I20260709 12:16:33.986562 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:33.987529 23493 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:33.937361 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:33.969820 12493 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:33.948895 23427 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.975955 23345 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:33.976279 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:33.976857 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61420us
I20260709 12:16:34.000265 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:33.977720 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:33.978226 12497 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:34.001137 12639 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:33.977752 12643 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:16:33.978605 23538 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:33.979127 23368 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:34.000998 12646 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
W20260709 12:16:34.003613 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66329.7us
I20260709 12:16:34.006662 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:34.008816 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:34.011684 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:34.021806 12637 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:34.023574 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.007929 23477 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:34.020299 23293 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:34.018844 23291 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:34.036607 12493 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:34.037079 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.038633 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.040153 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.040833 12643 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:16:34.041923 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.025408 23467 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:34.050004 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.048978 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:34.005601 23473 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.051443 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.053434 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.034375 23291 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:34.057471 12639 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:34.059120 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.068462 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.070005 12646 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:16:34.070363 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.071348 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:34.066532 12639 pending_rounds.cc:85] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 17
I20260709 12:16:34.066027 23333 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.060931 23467 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:34.061077 23291 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:34.074633 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.060459 23535 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:34.078069 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.100267 12489 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:34.114821 12488 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:16:34.090245 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:34.088217 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.116009 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:34.086778 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62454.5us
I20260709 12:16:34.116613 12489 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:34.073709 12639 pending_rounds.cc:107] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 27.18
I20260709 12:16:34.089562 12493 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:16:34.117199 12639 raft_consensus.cc:2889] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:34.087872 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:34.117830 12491 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:34.124501 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.129904 12493 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:34.118403 12643 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:16:34.124173 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:34.182286 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94119.1us
I20260709 12:16:34.135124 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:34.096292 23574 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:16:34.210776 23477 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:34.212276 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:34.069330 23540 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.221122 12493 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:34.222175 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97438.1us
I20260709 12:16:34.211690 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.135720 23133 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.076521 23427 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:34.083285 23514 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:34.075233 23333 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.239044 23133 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.216248 23458 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:16:34.240340 23333 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:34.077554 23345 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:34.191766 23564 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.317s	user 0.000s	sys 0.004s
I20260709 12:16:34.121404 23532 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:34.089949 23543 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:34.225699 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.123100 23293 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:34.226336 23339 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:34.235765 23428 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.095741 23573 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.125s
I20260709 12:16:34.223105 12637 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:34.242184 23133 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:34.119618 23534 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.246907 12489 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:34.202863 23498 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:34.122581 23473 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:34.252979 12488 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:34.251252 23532 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:34.261111 23543 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.265812 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:34.266242 23293 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.228893 23540 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.352420 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:34.344928 12493 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:34.355571 12492 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:34.224577 23548 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:16:34.278340 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:34.280143 23534 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:34.287426 23578 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.164s	user 0.000s	sys 0.003s
I20260709 12:16:34.291771 23428 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:34.293094 23579 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.293902 12568 raft_consensus.cc:2368] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:34.276480 23498 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:34.269987 23477 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.295044 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.295943 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:34.297267 12567 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:34.302327 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.371153 12570 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:34.302491 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:34.304392 23133 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:16:34.380439 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78194.2us
I20260709 12:16:34.307082 23535 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:34.311602 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1637) took 1088 ms. Trace:
I20260709 12:16:34.313810 22734 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.384680 22734 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.315637 23291 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.388448 23291 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:34.389812 23537 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:34.390821 23537 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.394188 23540 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:34.396020 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100164us
I20260709 12:16:34.396090 23543 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:34.397080 23543 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:34.398684 23293 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:34.399363 23539 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:34.400215 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:34.401805 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:34.322969 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.383909 12497 rpcz_store.cc:276] 0709 12:16:33.222751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:33.222938 (+   187us) service_pool.cc:224] Handling call
0709 12:16:34.311301 (+1088363us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:34.311487 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1698534656}
I20260709 12:16:34.075834 23467 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.351020 23532 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:34.352506 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160291us
W20260709 12:16:34.407913 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:34.356590 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:34.357281 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:34.358768 12493 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:16:34.409017 12637 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:34.409123 13262 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:34.362807 13261 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:16:34.410507 23532 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:16:34.407840 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.505s (SENT) [suppressed 20 similar messages]
I20260709 12:16:34.371199 23498 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:34.374357 12490 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:34.371470 23579 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.397377 23534 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:34.407037 12495 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:34.419987 23579 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:34.264770 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45149.2us
I20260709 12:16:34.407605 23535 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.431170 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:34.433272 12611 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:34.437157 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:34.406093 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.452813 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:34.431684 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:34.484948 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53337us
W20260709 12:16:34.487727 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81752.1us
I20260709 12:16:34.464053 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.489953 12565 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:16:34.464182 12566 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 21. (term mismatch)
I20260709 12:16:34.491284 12566 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:16:34.491518 12566 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 31.21
I20260709 12:16:34.491791 12566 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:34.464386 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.444278 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:34.494226 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.495172 12571 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:34.470429 12569 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:16:34.466419 12563 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:34.427723 13260 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:34.468007 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.472929 13258 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:34.461520 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:34.510514 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.513373 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.514837 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.517077 12570 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:16:34.494474 13261 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:34.518687 12491 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:34.521761 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:34.533188 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47355.1us
I20260709 12:16:34.539937 13262 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:34.421612 23105 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:16:34.535939 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47183.8us
I20260709 12:16:34.546132 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.446695 23502 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:34.449678 23535 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:34.524262 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.557920 12563 raft_consensus.cc:2368] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:16:34.550542 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:34.558843 12490 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:34.553586 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:34.551272 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:34.566278 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:34.495533 23493 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:34.547595 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.519714 23362 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:16:34.568358 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 135283us
I20260709 12:16:34.578662 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.569128 12570 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:34.579620 12492 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:34.572345 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:34.578689 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:34.580279 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:34.581074 12491 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:34.520493 23385 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:34.590876 23473 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:34.591104 23385 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.592952 23548 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:34.544727 23537 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:34.545276 23105 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:34.597689 23339 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:34.598029 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.598330 12609 leader_election.cc:400] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:16:34.598682 12609 leader_election.cc:403] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:16:34.598949 23539 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:34.599265 23333 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:34.599843 23571 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:34.600397 23571 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:34.601374 23571 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.603631 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.604832 23534 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:16:34.608914 23534 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:34.609328 23333 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:16:34.610082 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.610965 12567 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:34.612186 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.613651 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:34.615301 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:34.582674 23540 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:34.616240 23537 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:34.616889 23537 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:34.617254 23552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.034s	sys 0.050s
W20260709 12:16:34.617872 23552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.034s	sys 0.050s
I20260709 12:16:34.618091 23537 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.541723 23367 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:16:34.623449 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.624464 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.599303 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.627710 23367 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.630601 23367 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.631117 23385 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:34.599436 23539 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:34.632123 23539 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:34.633011 23367 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:34.635501 12567 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 23. Current term is 24. Ops: []
I20260709 12:16:34.581390 23345 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:34.588011 23362 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:34.638157 23362 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:34.628036 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:34.641963 12563 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:16:34.644994 22734 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:34.662657 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.656729 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:34.664470 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:34.646106 23534 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:34.639714 23345 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:34.670725 23339 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:34.663587 22734 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:34.675902 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.678084 12490 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:34.678520 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.695166 12639 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:34.652714 23495 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:34.658947 23493 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:34.685830 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:34.649253 13260 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
W20260709 12:16:34.687505 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111825us
I20260709 12:16:34.698807 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.676808 23477 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:34.699506 12639 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:34.698911 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:34.649070 23337 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:16:34.702536 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.702924 12564 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:34.702854 22734 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:34.703459 22734 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:34.703536 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.703823 22734 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:34.704361 12638 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:34.704847 12460 leader_election.cc:400] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:34.705101 12460 leader_election.cc:403] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
W20260709 12:16:34.701793 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 150614us
I20260709 12:16:34.707422 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.708729 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.686924 23470 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:34.746881 12640 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:34.741684 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.745684 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.745797 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:34.748883 13254 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:16:34.745270 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80897.2us
I20260709 12:16:34.701042 23477 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:34.709609 23427 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:34.815053 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.796550 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.790849 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.708488 23534 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:34.799481 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91665us
W20260709 12:16:34.820566 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115799us
I20260709 12:16:34.844182 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.844936 12645 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:34.711123 23587 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33.
I20260709 12:16:34.811225 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.760290 23473 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.081s
I20260709 12:16:34.853829 12497 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 21.
I20260709 12:16:34.862568 12639 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:34.823529 22734 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.798373 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.823895 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.822674 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:34.762537 23458 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:34.827756 12641 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:16:34.794956 12496 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:34.835383 12642 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:34.837424 12489 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:34.866478 12489 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:34.792964 23477 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:34.871366 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.873167 12489 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 33.16, Last appended by leader: 16, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:34.874161 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.879855 23595 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:34.869444 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:34.896632 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.872999 23458 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.867388 23470 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:34.923367 22734 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:34.924003 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:34.925822 12639 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:34.860662 23367 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.058s
I20260709 12:16:34.928824 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.940222 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.846028 23345 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:34.945686 12641 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:34.918160 23470 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:34.951028 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108002us
I20260709 12:16:34.967281 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:34.969121 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:34.970642 23548 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:34.972332 23495 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:34.976148 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:34.978693 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:35.009198 23596 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:16:34.978585 23427 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:35.015715 23427 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.018582 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.031857 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.053260 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.058732 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.066864 12640 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:34.976941 23534 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:35.072458 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 219612us
I20260709 12:16:35.086618 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.093732 12639 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:35.074059 12646 raft_consensus.cc:2955] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Committing config change with OpId 21.14: config changed from index 7 to 14, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:16:35.099491 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.097589 12640 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260709 12:16:35.068737 23483 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.003s	sys 0.024s
I20260709 12:16:35.057400 23362 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.103186 12641 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:35.096746 23345 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:35.160871 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:35.153015 23483 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.003s	sys 0.025s
I20260709 12:16:35.083395 23477 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.166182 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.168785 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:35.183779 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.126331 23611 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.078s
I20260709 12:16:35.170084 23605 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:35.200953 12641 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:35.202651 23539 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.570s	user 0.015s	sys 0.000s
I20260709 12:16:35.098649 23552 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.091s
I20260709 12:16:35.133558 23105 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.230257 12640 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:35.238801 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:35.247192 12495 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:35.293959 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:35.322312 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:35.322824 12640 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:35.315955 23470 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.239607 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.330454 12641 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:16:35.299551 23339 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.598s	user 0.013s	sys 0.000s
W20260709 12:16:35.239674 23498 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.824s	user 0.014s	sys 0.010s
W20260709 12:16:35.251050 23608 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
W20260709 12:16:35.251801 23616 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 12:16:35.292294 23620 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 12:16:35.290201 23614 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
W20260709 12:16:35.342295 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:35.244701 23604 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.235s	user 0.003s	sys 0.000s
I20260709 12:16:35.303109 23105 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:35.350852 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.298421 23493 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:35.246347 23362 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.352734 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:35.354916 12461 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.858s (SENT)
I20260709 12:16:35.357185 12640 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28.
I20260709 12:16:35.245224 23605 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.361728 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.363323 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.364120 12640 raft_consensus.cc:2376] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26. Re-sending same reply.
I20260709 12:16:35.363421 23458 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.245154 23539 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.366333 12639 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33. Re-sending same reply.
I20260709 12:16:35.366691 23539 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:35.365734 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.367378 12637 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37.
I20260709 12:16:35.351470 23339 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:35.363508 23605 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:35.375999 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:35.358316 23477 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:35.378572 23599 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.364s	user 0.004s	sys 0.008s
I20260709 12:16:35.352574 23498 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:35.407970 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:35.444083 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:35.476698 23619 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.235s	user 0.009s	sys 0.014s
I20260709 12:16:35.486858 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.499478 12638 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:35.358150 23493 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:35.494220 12639 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35. Re-sending same reply.
W20260709 12:16:35.502848 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 150255us
I20260709 12:16:35.503598 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.498413 12495 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:35.505769 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
W20260709 12:16:35.512084 12461 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.900s (SENT)
I20260709 12:16:35.512578 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:35.512888 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.514655 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.516074 13260 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:35.516110 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.519057 12641 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35. Re-sending same reply.
I20260709 12:16:35.516469 23495 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:35.520568 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.520287 23495 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.517563 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.516033 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.521910 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.527150 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.510147 23534 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:35.543413 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.538923 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.549727 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.550575 12641 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35. Re-sending same reply.
I20260709 12:16:35.553118 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.559523 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.543608 23545 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:35.568683 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:35.553845 23502 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.986s	user 0.017s	sys 0.000s
I20260709 12:16:35.575187 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.575842 12641 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:35.589372 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.558893 23587 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:35.589505 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:35.572314 23618 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.343s	user 0.001s	sys 0.005s
I20260709 12:16:35.597708 13256 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:35.620570 12489 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:35.550410 23534 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.635938 23539 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:35.636592 23539 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.632169 23561 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:35.638270 23561 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.647349 23539 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:35.640508 23561 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.656041 12489 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 16. Preceding OpId from leader: term: 34 index: 16. (term mismatch)
I20260709 12:16:35.656781 12489 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 15
I20260709 12:16:35.657150 12489 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 33.16
I20260709 12:16:35.657502 12489 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:35.590366 23545 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:35.661283 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.669631 12637 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30. Re-sending same reply.
W20260709 12:16:35.664115 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157963us
I20260709 12:16:35.667183 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.678696 13256 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:35.680157 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 167679us
W20260709 12:16:35.682605 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.205s	user 0.032s	sys 0.009s
W20260709 12:16:35.683045 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.205s	user 0.032s	sys 0.009s
W20260709 12:16:35.683336 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.25216e+06us
W20260709 12:16:35.684756 12461 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.904s (SENT)
I20260709 12:16:35.683197 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.687346 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:35.688989 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.689728 12641 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:35.690029 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:35.584100 23493 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.698271 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.701428 12641 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:35.706277 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4232) took 1026 ms. Trace:
I20260709 12:16:35.706745 12642 rpcz_store.cc:276] 0709 12:16:34.679584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:34.684922 (+  5338us) service_pool.cc:224] Handling call
0709 12:16:35.706092 (+1021170us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:35.706212 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86037248}
I20260709 12:16:35.709802 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.710747 12646 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:35.711594 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.731478 12644 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:16:35.695226 23498 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.707042 23587 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:35.744951 23534 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:35.747777 12646 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:16:35.748589 12637 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:35.750224 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:35.751299 23630 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:35.751929 23630 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:35.752846 23630 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.757458 12646 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
W20260709 12:16:35.762043 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90312.5us
I20260709 12:16:35.762454 23545 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:35.763222 12461 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.924s (SENT)
I20260709 12:16:35.763809 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.767154 23611 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:35.770717 23495 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:35.772078 23470 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:35.772797 23470 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:35.773252 23470 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.732509 23337 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:35.776124 23337 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.738903 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.782255 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.741590 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.736790 12641 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:35.787180 13261 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:35.792934 13261 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:16:35.796028 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54624us
I20260709 12:16:35.797219 23534 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:35.794562 23548 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:35.801997 23548 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:35.804378 23548 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.797272 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.814998 12642 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:35.815632 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.816656 12641 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:35.818327 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.824047 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.828711 12643 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:16:35.831265 12461 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.036s (SENT)
I20260709 12:16:35.833603 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.835388 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.837569 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4475) took 1210 ms. Trace:
I20260709 12:16:35.838297 12570 rpcz_store.cc:276] 0709 12:16:34.627122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:34.629060 (+  1938us) service_pool.cc:224] Handling call
0709 12:16:34.629377 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:34.629597 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:16:34.650595 (+ 20998us) log.cc:844] Serialized 14 byte log entry
0709 12:16:34.695454 (+ 44859us) log.cc:844] Serialized 14 byte log entry
0709 12:16:35.837002 (+1141548us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b1800204820. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:16:35.837181 (+   179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:35.837241 (+    60us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:35.837296 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:35.837378 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:35.837503 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40432,"prepare.queue_time_us":22669,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":1226712,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":44139,"spinlock_wait_cycles":12810112,"thread_start_us":42908,"threads_started":1}
I20260709 12:16:35.837476 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:35.840983 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:35.841531 23133 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:35.842597 23543 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:35.819591 23502 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:35.830214 23427 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:35.847312 12489 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:16:35.838578 23539 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:35.848737 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:35.835007 23605 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:35.868160 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44886.3us
I20260709 12:16:35.869760 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:35.873765 23615 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.700s	user 0.005s	sys 0.000s
I20260709 12:16:35.876030 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1551) took 1036 ms. Trace:
I20260709 12:16:35.876463 12645 rpcz_store.cc:276] 0709 12:16:34.839276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:34.851054 (+ 11778us) service_pool.cc:224] Handling call
0709 12:16:34.864358 (+ 13304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:34.890047 (+ 25689us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:34.890084 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:34.903332 (+ 13248us) log.cc:844] Serialized 26 byte log entry
0709 12:16:35.055600 (+152268us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:35.171325 (+115725us) log.cc:844] Serialized 14 byte log entry
0709 12:16:35.175110 (+  3785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:35.175203 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:35.873068 (+697865us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:35.875672 (+  2604us) raft_consensus.cc:1685] finished
0709 12:16:35.875791 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:35.875959 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27103,"prepare.queue_time_us":195463,"prepare.run_cpu_time_us":6920,"prepare.run_wall_time_us":42382,"raft.queue_time_us":38984,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":13210,"spinlock_wait_cycles":13400448,"thread_start_us":207585,"threads_started":2,"wal-append.queue_time_us":207252}
W20260709 12:16:35.879731 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3491) took 2278 ms (client timeout 1913 ms). Trace:
W20260709 12:16:35.880280 23105 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.522s	user 0.018s	sys 0.005s
W20260709 12:16:35.880187 12644 rpcz_store.cc:269] 0709 12:16:33.601470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:33.606657 (+  5187us) service_pool.cc:224] Handling call
0709 12:16:35.879518 (+2272861us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:35.879635 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4570565504}
I20260709 12:16:35.843916 23601 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:35.885377 23612 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:35.886144 23612 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.843029 23133 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.891494 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:35.892237 12495 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:35.894474 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:35.895290 12489 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:35.897230 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:35.904202 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:35.905078 12497 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:16:35.897944 12496 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:35.918252 23133 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.847831 23337 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:35.867518 23333 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:16:35.923120 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 238763us
I20260709 12:16:35.913328 12489 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
W20260709 12:16:35.935596 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98211.9us
I20260709 12:16:35.947737 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:35.949156 12640 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37.
I20260709 12:16:35.951227 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:35.961194 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:35.932907 23333 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:35.963896 23333 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:35.923082 23477 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:35.966635 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:35.965385 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:35.954026 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 113123us
I20260709 12:16:35.888607 23612 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:35.973647 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:35.978772 13261 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 31 because replica has last-logged OpId of term: 30 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 18.
I20260709 12:16:35.966136 23293 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:35.986959 23472 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:35.987241 23293 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:35.986959 13256 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:16:35.998497 12572 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:16:35.966451 23592 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.003729 12567 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:16:36.005138 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.006532 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.007596 12645 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:36.009101 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.003170 23635 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 12:16:36.010142 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.003080 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.011667 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.012537 12641 raft_consensus.cc:2376] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33. Re-sending same reply.
I20260709 12:16:35.976709 23579 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:36.015749 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:35.564656 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:36.031601 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.032677 12646 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:16:35.990839 23612 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:36.034363 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.039073 12566 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:36.024206 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.040207 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:35.971045 23477 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.041349 12645 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:35.989568 23293 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:36.031062 23609 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.933s	user 0.009s	sys 0.000s
I20260709 12:16:36.044584 12641 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
W20260709 12:16:36.048993 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46015.6us
W20260709 12:16:36.050501 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83963.6us
I20260709 12:16:36.052866 12536 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:36.016001 23473 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:16:36.051486 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:35.995564 23428 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:16:36.037348 23498 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:36.053236 12536 leader_election.cc:403] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:36.066573 12570 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:36.070215 12564 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
W20260709 12:16:36.057062 23632 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:16:36.057004 23533 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:36.076385 23483 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:36.082257 23533 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:36.077014 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.082981 23533 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:36.083566 23483 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:36.084223 23483 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:36.085321 23483 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 33.13, Last appended by leader: 13, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.068959 23498 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.044235 23579 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.015221 23592 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.056172 23612 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.055794 23473 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:36.118342 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1346) took 1410 ms. Trace:
I20260709 12:16:36.113121 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:36.118996 13260 rpcz_store.cc:276] 0709 12:16:34.707534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:34.707738 (+   204us) service_pool.cc:224] Handling call
0709 12:16:34.708018 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:35.518757 (+810739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:36.117951 (+599194us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:36.118250 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1857828224}
I20260709 12:16:36.104426 23635 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:36.108618 23473 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:16:36.134100 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82701.1us
I20260709 12:16:36.134977 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.116148 23548 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.142676 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:36.146466 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:36.148062 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4635) took 1320 ms. Trace:
I20260709 12:16:36.151257 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.150240 12490 rpcz_store.cc:276] 0709 12:16:34.827315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:34.843854 (+ 16539us) service_pool.cc:224] Handling call
0709 12:16:34.844228 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:34.857877 (+ 13649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:34.857915 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:34.858529 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:16:34.958717 (+100188us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:34.958791 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:34.965814 (+  7023us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:35.985541 (+1019727us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b2400121718. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:16:36.139266 (+153725us) raft_consensus.cc:1685] finished
0709 12:16:36.139316 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:36.147952 (+  8636us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8026,"raft.queue_time_us":18964,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":16716,"spinlock_wait_cycles":70777088,"thread_start_us":99686,"threads_started":1,"wal-append.queue_time_us":196518}
I20260709 12:16:36.109308 23612 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.169215 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:36.170894 12566 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:16:36.172546 12461 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.533s (SENT)
I20260709 12:16:36.173264 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:36.178835 23611 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:36.181478 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.184013 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.184885 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:36.187506 23586 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.188683 23586 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.191895 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:36.192566 23586 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.195032 12495 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:36.198329 23165 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.199329 23165 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.203401 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.209869 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.205372 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.213573 12567 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
W20260709 12:16:36.177392 23637 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
W20260709 12:16:36.216518 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107763us
I20260709 12:16:36.218509 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:16:36.144833 23635 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:36.225242 12566 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:16:36.227875 23483 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.228072 23586 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.228327 12566 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:16:36.228601 23483 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.228861 23586 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.231443 23483 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.232060 23586 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.236757 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:36.233777 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1768958450002174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
I20260709 12:16:36.154721 23514 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:36.237967 23514 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:36.238188 12567 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 24: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:36.238444 23514 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:36.155081 23498 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.243520 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:36.286384 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151483us
W20260709 12:16:36.292160 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48775.1us
I20260709 12:16:36.295125 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:36.158109 23573 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.089s
I20260709 12:16:36.153846 23548 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.297456 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:36.313419 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115083us
I20260709 12:16:36.287231 23586 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.316148 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:36.316956 12567 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:36.291638 23630 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.302021 23428 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:16:36.322670 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109989us
I20260709 12:16:36.330443 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.328423 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.333695 12641 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:36.328896 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.334777 12645 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:36.333472 12566 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:16:36.329198 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.337301 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.306344 23649 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:16:36.339893 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167370us
W20260709 12:16:36.340695 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:36.334995 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.343503 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.345257 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:36.346516 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.345463 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:36.348846 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.367739 12490 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:16:36.327025 23472 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:36.364305 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48202.9us
I20260709 12:16:36.371862 13256 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:16:36.326344 23293 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.372877 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:36.378010 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:36.379714 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.383100 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:36.383685 12536 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:36.363817 23635 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.342264 23165 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.385855 23133 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:16:36.386055 23640 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:16:36.387076 23647 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.058s
I20260709 12:16:36.327992 23630 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.321997 23586 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.390939 23630 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.391920 23586 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.392421 23612 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:36.392776 12388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 229) took 82044 ms (client timeout 15000 ms). Trace:
I20260709 12:16:36.364410 23477 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.386207 23641 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:16:36.389524 23651 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:16:36.384812 23638 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:16:36.335767 23428 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:36.407701 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83246.7us
I20260709 12:16:36.408324 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.348023 23495 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:36.426049 23642 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:16:36.425168 23428 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:36.385751 23643 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:16:36.390981 23650 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:16:36.374871 23498 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.444450 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:36.370627 23579 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.370577 23592 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.448825 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:36.448980 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84022.5us
I20260709 12:16:36.409133 23483 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:36.432163 23612 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:36.427297 23586 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:36.461251 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:36.482581 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99758.2us
I20260709 12:16:36.485489 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.488910 23592 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:36.490669 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:36.495229 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171577us
I20260709 12:16:36.400012 23472 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.386061 23652 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.078s
I20260709 12:16:36.458483 23653 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.414487 23293 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.474261 23587 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:36.498931 23649 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:36.524695 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:36.436684 23495 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.494459 23483 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:36.539446 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.554836 12642 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:16:36.538292 23641 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.544301 23662 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:36.545032 23477 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:36.463903 23165 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:16:36.545549 23640 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.547535 23472 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:36.547902 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57394.2us
I20260709 12:16:36.548172 23655 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:16:36.540427 23579 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.526803 23595 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 35.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.025s
I20260709 12:16:36.524646 23644 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.581847 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.582904 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:16:36.462175 23498 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.524466 23587 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:36.566002 13258 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
W20260709 12:16:36.567881 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109932us
I20260709 12:16:36.531582 23638 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.550345 23611 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:36.593381 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 197052us
W20260709 12:16:36.565850 23333 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.601s	user 0.012s	sys 0.003s
I20260709 12:16:36.563899 23477 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:36.591030 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.592021 12563 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:36.578711 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.563100 23533 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:36.393308 12388 rpcz_store.cc:269] 0709 12:15:14.427610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:14.519333 (+ 91723us) service_pool.cc:224] Handling call
0709 12:16:36.370336 (+81851003us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:16:30.443221 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:16:30.523176 (+ 79955us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:16:30.523227 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:16:30.523253 (+    26us) tablet.cc:662] Decoding operations
0709 12:16:30.524104 (+   851us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:16:30.524185 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 12:16:30.524226 (+    41us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:16:30.524619 (+   393us) tablet.cc:701] Row locks acquired
0709 12:16:30.524645 (+    26us) write_op.cc:260] PREPARE: finished
0709 12:16:30.524852 (+   207us) write_op.cc:270] Start()
0709 12:16:30.525005 (+   153us) write_op.cc:276] Timestamp: P: 1783599390524778 usec, L: 0
0709 12:16:30.525033 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:16:30.679033 (+154000us) log.cc:844] Serialized 1317 byte log entry
0709 12:16:31.150616 (+471583us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eee28 00007ffbf9229581 00007ffbf9228d58 00007ffbf922d9f1 00007ffbf922d9a9 00007ffbf922d939 00007ffbf922d901 00007ffbf922cb2d 0000000000388634 0000000000388479
0709 12:16:31.747497 (+596881us) op_driver.cc:464] REPLICATION: finished
0709 12:16:32.091101 (+343604us) write_op.cc:301] APPLY: starting
0709 12:16:32.091262 (+   161us) tablet.cc:1367] starting BulkCheckPresence
0709 12:16:32.231707 (+140445us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b18003b1b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf55320ec 00007ffbf5534af7 00007ffbf5535fb0 00007ffbf924a1bb 00007ffbf924a532 00007ffbf9291b93 00007ffbf913ede3 00007ffbf913e6ab 00007ffbf913fe77
0709 12:16:32.232958 (+  1251us) tablet.cc:1370] finished BulkCheckPresence
0709 12:16:32.233012 (+    54us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:32.297698 (+ 64686us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b18003b1b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:32.326784 (+ 29086us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:32.327233 (+   449us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 12:16:32.327350 (+   117us) write_op.cc:312] APPLY: finished
0709 12:16:32.328083 (+   733us) log.cc:844] Serialized 41 byte log entry
0709 12:16:32.330343 (+  2260us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:32.330703 (+   360us) write_op.cc:454] Released schema lock
0709 12:16:32.330838 (+   135us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:32.541154 (+210316us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:32.546586 (+  5432us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20776,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":343109,"apply.run_cpu_time_us":14233,"apply.run_wall_time_us":300760,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16655,"num_ops":3,"prepare.queue_time_us":715759,"prepare.run_cpu_time_us":71169,"prepare.run_wall_time_us":738554,"raft.queue_time_us":259261,"raft.run_cpu_time_us":52233,"raft.run_wall_time_us":335379,"replication_time_us":1222221,"spinlock_wait_cycles":178493952,"thread_start_us":1100997,"threads_started":4,"wal-append.queue_time_us":531442,"wal-append.run_cpu_time_us":28173,"wal-append.run_wall_time_us":2096774}]]}
I20260709 12:16:36.563299 23514 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:36.596800 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 95377.2us
I20260709 12:16:36.609331 12461 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:36.610373 23495 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.611691 23514 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.612596 13260 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:36.591104 23644 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.613788 23649 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.615159 23467 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:36.615566 23467 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:36.528579 23653 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.595062 23611 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:36.619063 23611 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:36.619190 23533 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.619294 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:36.620234 13256 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
I20260709 12:16:36.574394 23640 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.621897 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:36.621971 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.622174 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4262) took 1418 ms. Trace:
I20260709 12:16:36.597213 23638 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.598398 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:36.565653 23641 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.643535 23293 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.644364 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.645093 23641 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.608012 23483 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.635491 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:36.635118 23595 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
W20260709 12:16:36.642499 12388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 132) took 19740 ms (client timeout 15000 ms). Trace:
W20260709 12:16:36.647743 12388 rpcz_store.cc:269] 0709 12:16:16.902166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.642174 (+19740008us) service_pool.cc:211] Skipping call since client already timed out
0709 12:16:36.642381 (+   207us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:16:36.643172 23587 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.588249 23579 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.589931 23659 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:16:36.639225 12567 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:36.598763 23477 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:36.615576 23644 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.648792 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:36.645967 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.650245 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.632064 12637 rpcz_store.cc:276] 0709 12:16:35.203416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:35.740245 (+536829us) service_pool.cc:224] Handling call
0709 12:16:35.748409 (+  8164us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:35.749013 (+   604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:36.621960 (+872947us) spinlock_profiling.cc:243] Waited 7.79 ms on lock 0x7b54002c6d50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:36.622112 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17914112}
I20260709 12:16:36.601127 23657 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:36.635651 13255 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:36.598948 23333 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.636140 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4283) took 1103 ms. Trace:
I20260709 12:16:36.671372 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.672933 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.649979 23498 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.671027 12638 rpcz_store.cc:276] 0709 12:16:35.523684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:35.555669 (+ 31985us) service_pool.cc:224] Handling call
0709 12:16:36.621802 (+1066133us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:16:36.627160 (+  5358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11923328}
I20260709 12:16:36.677778 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.680163 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.678316 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:36.678506 12638 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:36.679462 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.684523 12637 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:36.678880 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:36.674347 23587 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:36.688313 12643 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:36.651557 23533 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:16:36.664193 23657 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:36.690826 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.691847 23666 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:36.695729 23666 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.698259 23657 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:36.699008 23666 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.700404 23333 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.695732 23670 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:36.702422 23670 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.702950 23670 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:36.704174 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:36.704968 13255 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:36.706424 23640 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.678005 23502 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:36.708192 23502 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.710583 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.670832 23653 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:36.723668 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.716495 12489 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:36.717895 12490 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:36.718264 23502 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.718473 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.724617 12492 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:36.729456 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:36.730345 12638 raft_consensus.cc:2376] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30. Re-sending same reply.
I20260709 12:16:36.731065 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:36.684819 23472 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:36.736198 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.736502 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:36.737190 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57116.8us
I20260709 12:16:36.737886 23674 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:36.739122 23662 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:36.720741 23473 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.572s	user 0.013s	sys 0.000s
I20260709 12:16:36.737340 12489 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:36.741067 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:36.751860 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.752898 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.757160 13255 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:36.773186 12637 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:36.783492 12489 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:16:36.783517 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43793us
I20260709 12:16:36.785758 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.740649 23663 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:36.784821 12638 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 12:16:36.797421 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139484us
I20260709 12:16:36.805730 12645 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:16:36.804669 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.750264 23649 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:16:36.762913 23473 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.797456 23658 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.819092 12572 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:16:36.821254 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:36.790851 23105 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.019s
W20260709 12:16:36.835289 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:36.837811 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:36.837359 12642 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:16:36.801309 23472 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 12:16:36.814035 23533 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:36.840291 23658 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:36.850571 23676 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:16:36.808300 23647 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:16:36.852387 23658 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:36.867055 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49574.7us
I20260709 12:16:36.869057 23470 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.869310 23473 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:36.862740 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.843842 23533 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:36.887929 12644 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:36.876125 23579 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:36.886786 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:36.884111 23539 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:36.899930 12390 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 11 to 18, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 18 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:36.865324 23502 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
W20260709 12:16:36.912168 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53807.1us
I20260709 12:16:36.930797 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.923538 12390 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 15 to 22, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 22 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:16:36.937299 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:36.939400 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:36.939910 12490 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:16:36.912644 23612 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:36.935201 12536 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28.
I20260709 12:16:36.938123 12497 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:36.943277 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.889519 23533 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:36.946249 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.949134 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.956557 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.947881 23105 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.966979 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.941377 23669 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.934459 23539 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:36.986459 23539 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:36.988669 23670 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:36.935703 23483 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:36.971539 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:36.958604 23339 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:16:36.990181 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57116.6us
I20260709 12:16:36.990496 23339 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:36.991214 23339 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:36.991313 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.991793 12645 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 13. Preceding OpId from leader: term: 35 index: 14. (index mismatch)
I20260709 12:16:36.992901 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:36.993784 23540 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:36.995244 23483 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:36.993834 23612 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:16:36.998016 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120799us
I20260709 12:16:36.999646 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.002835 23670 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.002727 23533 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.006510 23339 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.007596 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:37.008543 23612 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:36.946064 23659 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:36.978292 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.014580 12637 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 33 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 16.
W20260709 12:16:36.963207 23636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.010s	sys 0.014s
I20260709 12:16:36.979950 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:36.946516 23635 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:36.934037 23586 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:16:37.021719 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:37.024641 12490 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:37.005412 23533 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.027851 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.028426 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:37.030057 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:37.031279 12493 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:37.018543 23636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.961s	user 0.010s	sys 0.015s
I20260709 12:16:37.035089 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:37.022292 23681 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.145s	user 0.001s	sys 0.001s
I20260709 12:16:37.036934 23339 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.018239 23470 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.042048 23470 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.038329 12490 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:36.958729 23105 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.014250 23669 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.059060 12493 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:37.056562 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.060222 12495 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:37.065616 12490 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:37.062388 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:37.072636 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:37.035759 23534 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:16:37.048689 23428 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:37.076539 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:37.077965 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56397.8us
I20260709 12:16:37.032287 23657 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.049835 23659 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:37.079006 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:37.068655 23586 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:37.046986 23470 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.026165 23635 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.050318 23687 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:37.081754 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53999.5us
I20260709 12:16:37.030870 23477 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.052371 23611 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.080848 23534 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:37.105213 12644 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:37.090544 23612 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:37.103487 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:37.077872 23428 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:37.080860 23636 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:37.080619 23663 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:37.092708 23473 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:37.121395 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.124977 12645 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 26.
I20260709 12:16:37.126312 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.095418 23687 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:37.128543 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:37.125576 23663 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:37.136185 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:37.133785 12496 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:37.139633 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.093303 23639 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:16:37.136519 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4612) took 1081 ms. Trace:
I20260709 12:16:37.119933 23612 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:37.159662 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.154127 12564 rpcz_store.cc:276] 0709 12:16:36.055262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.071347 (+ 16085us) service_pool.cc:224] Handling call
0709 12:16:36.086093 (+ 14746us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:36.120838 (+ 34745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:36.120877 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:36.127488 (+  6611us) log.cc:844] Serialized 26 byte log entry
0709 12:16:36.219352 (+ 91864us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:36.219392 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:37.136134 (+916742us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:37.136185 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:37.136230 (+    45us) raft_consensus.cc:1685] finished
0709 12:16:37.136245 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:37.136459 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":604,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":7657,"spinlock_wait_cycles":67485056,"thread_start_us":62931,"threads_started":1,"wal-append.queue_time_us":148193}
I20260709 12:16:37.128839 23636 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:37.139076 23663 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:37.171000 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4605) took 1210 ms. Trace:
I20260709 12:16:37.141703 23659 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:37.142190 23473 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:37.136816 23687 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:37.174659 12569 rpcz_store.cc:276] 0709 12:16:35.960168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:35.965324 (+  5156us) service_pool.cc:224] Handling call
0709 12:16:35.965608 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:35.989897 (+ 24289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:35.989936 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:35.989986 (+    50us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:35.990026 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:37.170632 (+1180606us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:37.170696 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:37.170911 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52222336}
I20260709 12:16:37.163224 23612 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:37.179140 12637 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:16:37.224200 23473 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.223089 23636 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.233068 23493 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:37.233174 23663 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.238632 23498 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.239410 23663 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.239984 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1350) took 1048 ms. Trace:
I20260709 12:16:37.240350 13260 rpcz_store.cc:276] 0709 12:16:36.191261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.193493 (+  2232us) service_pool.cc:224] Handling call
0709 12:16:36.193940 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:36.613084 (+419144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:37.239742 (+626658us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:37.239922 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":962231552}
I20260709 12:16:37.248453 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:37.249584 23502 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:37.250212 23502 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:37.250670 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.251466 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:37.251142 23502 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.252780 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.265131 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:37.273011 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.274008 23105 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.267853 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:37.275785 23669 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.276463 23669 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:37.276926 23669 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.277922 23669 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:37.274312 23694 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:16:37.272856 23686 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:37.239413 23498 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:37.323377 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:37.185274 23534 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.309082 23498 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.306877 23686 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:37.308018 23611 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.304950 23697 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:37.358117 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 341818us
I20260709 12:16:37.342751 23657 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.373126 23647 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:37.373871 23647 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:37.371295 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211729us
I20260709 12:16:37.379981 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:37.384523 12640 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:37.385529 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.386225 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.387624 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.390061 12495 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:37.394160 12490 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:37.378464 23498 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:37.372633 23697 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:37.400204 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.397078 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.398020 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.399050 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.405226 12488 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:37.397872 23498 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:37.410449 23498 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:37.413590 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:37.402400 23697 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:37.415416 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.428987 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.414402 23333 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:37.432991 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 165240us
W20260709 12:16:37.439981 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60133.9us
I20260709 12:16:37.442016 23493 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:37.442893 23493 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.445530 23548 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.433845 23333 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.432777 23669 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:16:37.462471 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 192521us
I20260709 12:16:37.466051 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.466712 12488 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:37.463124 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:37.471657 23696 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:16:37.475333 23587 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.004s
I20260709 12:16:37.468539 23689 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.010s
I20260709 12:16:37.478518 23689 consensus_queue.cc:1243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 33.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:37.479050 23689 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:37.479374 23689 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.480211 23689 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:37.471827 23699 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:16:37.482007 23573 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:37.482736 23493 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.485860 23657 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:37.486750 23657 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.489074 23635 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.489995 12488 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:37.491101 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.491685 12488 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:16:37.492827 23428 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.493511 23709 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:37.494364 23428 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:37.482570 23573 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:37.496075 23573 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:37.497411 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:37.468992 23669 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.491101 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:37.500102 23639 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:37.498585 23669 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.500470 23483 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:37.501099 23483 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.503234 23483 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.502092 23659 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:37.505798 23635 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.506317 23635 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.507227 23687 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.508417 23635 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.508708 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3512) took 1173 ms. Trace:
I20260709 12:16:37.509181 12642 rpcz_store.cc:276] 0709 12:16:36.334844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.645606 (+310762us) service_pool.cc:224] Handling call
0709 12:16:37.508473 (+862867us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:37.508636 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411716352}
W20260709 12:16:37.516795 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46256.9us
I20260709 12:16:37.481536 23689 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:37.525718 23612 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.497242 23498 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.504787 23697 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.534348 23502 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:37.535285 23635 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.509560 23687 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:37.542583 23693 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.446s	user 0.004s	sys 0.000s
I20260709 12:16:37.489130 23657 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.546408 23502 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.554740 23635 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.517016 23639 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:37.571095 23702 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:16:37.575098 23498 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.526760 23659 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.049s
I20260709 12:16:37.575728 23467 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.576799 23467 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:37.576962 23587 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:37.577368 23587 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.578832 23587 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.580242 23587 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:37.545665 23473 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:37.571399 23639 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.587262 23473 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:37.594432 23653 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.000s	sys 0.020s
W20260709 12:16:37.595093 23653 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.000s	sys 0.020s
I20260709 12:16:37.655805 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.607635 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.659226 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:37.599093 23703 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.160s	user 0.002s	sys 0.000s
I20260709 12:16:37.602859 23697 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.600288 23540 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.621034 23477 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.595568 23473 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.597234 23612 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.607178 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:37.625408 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.664049 13260 raft_consensus.cc:2376] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29. Re-sending same reply.
W20260709 12:16:37.648736 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132279us
I20260709 12:16:37.608692 23646 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:16:37.664911 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.660534 12565 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:37.665910 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:37.665925 12566 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:37.668668 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.669994 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:37.671255 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200382us
I20260709 12:16:37.676070 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.676317 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.624646 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:37.681345 13261 raft_consensus.cc:2368] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:37.683574 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.672425 12461 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37.
I20260709 12:16:37.687449 12461 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
I20260709 12:16:37.687161 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.689603 12570 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:16:37.690516 12611 leader_election.cc:400] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:37.690892 12611 leader_election.cc:403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:16:37.691807 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.697382 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.698412 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:37.607597 23706 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:16:37.702464 12570 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:37.699055 12572 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:16:37.656150 23649 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:37.701876 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.701150 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.698202 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:37.704211 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.705404 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.706142 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.611950 23264 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:37.710012 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.622517 23502 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.706455 12564 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:37.711284 12570 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 27 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:16:37.713361 13262 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:37.715508 13262 raft_consensus.cc:606] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:37.718817 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:37.719673 12565 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:37.713434 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:37.697897 23612 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.691066 23514 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:37.702145 23534 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:37.663640 23540 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.702746 23635 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.710593 23264 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.747493 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:37.750869 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78633us
I20260709 12:16:37.752082 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.748504 12568 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:37.726558 23514 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:37.758961 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1388) took 1330 ms (client timeout 1751 ms). Trace:
I20260709 12:16:37.721434 23717 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37.
I20260709 12:16:37.758929 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:37.723198 23697 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:16:37.709290 23649 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:37.759786 13258 rpcz_store.cc:269] 0709 12:16:36.420319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.420828 (+   509us) service_pool.cc:224] Handling call
0709 12:16:37.750774 (+1329946us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:37.750941 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252266880}
I20260709 12:16:37.747637 23493 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.762069 23514 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:37.773335 12572 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:16:37.777952 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:37.767803 23709 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:37.780671 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.780920 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.783659 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:37.785159 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.729795 23612 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.764267 23473 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.789531 12565 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:37.794308 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.765560 23713 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:37.798324 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:37.769833 23635 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:37.800698 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:37.801386 12572 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:37.807123 12572 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:37.756835 23573 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.779958 23605 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:37.826385 12566 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:37.793236 23709 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:37.792837 23717 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:37.796830 23653 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:37.808334 12572 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.840427 13254 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.22-32.22]
I20260709 12:16:37.840688 12572 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:37.797142 23689 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:37.799103 23669 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:37.829437 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.782507 23716 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.046s
I20260709 12:16:37.839862 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:37.792702 23165 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:16:37.828297 23605 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:37.840802 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.809501 23714 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:37.848006 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:37.835311 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51739.6us
I20260709 12:16:37.842150 12565 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:37.834569 23709 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:37.855842 12572 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 19. (term mismatch)
I20260709 12:16:37.857024 12570 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:16:37.834034 23661 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:37.782514 23493 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.843473 23717 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.857419 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:37.873811 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:37.798592 23473 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.830996 23721 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:37.863848 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:37.884919 23711 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:16:37.885203 23612 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:37.837735 23502 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:37.902915 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.903837 12645 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27. Re-sending same reply.
I20260709 12:16:37.905189 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:37.905478 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:37.867133 12572 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 18
I20260709 12:16:37.909302 12572 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 30.19
I20260709 12:16:37.909612 12572 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:37.911151 23657 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:37.875658 12494 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:16:37.912128 23540 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:37.860563 12492 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:37.828085 23264 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.916661 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.878688 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:37.826169 23635 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.892773 23573 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:37.866245 23659 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:37.862378 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:37.895833 23717 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.887795 23605 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:37.903517 23502 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:37.911809 23657 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:37.914031 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:37.919239 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:37.876540 23661 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:37.884692 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:37.917380 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:16:37.988539 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.019963 12563 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:16:38.024326 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:38.028191 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87504.5us
I20260709 12:16:38.035048 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.035584 12643 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32. Re-sending same reply.
I20260709 12:16:38.036360 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.036808 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:16:37.927881 23720 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.091s	user 0.001s	sys 0.001s
I20260709 12:16:38.037549 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.031318 12642 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:37.926035 23649 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:38.047561 23661 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.052944 12638 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
W20260709 12:16:38.055073 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:38.060201 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 346824us
I20260709 12:16:38.061439 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:38.038239 23605 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:38.034231 23428 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.066993 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:38.090447 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66237.7us
I20260709 12:16:38.051544 23573 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:38.018692 23700 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.065s
I20260709 12:16:38.091915 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:38.059675 23473 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:38.097995 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:38.090080 23428 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.068967 23467 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:38.084503 23687 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:38.090090 23612 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:16:38.040879 23721 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:16:38.101063 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.128521 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.138787 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:38.139747 12642 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:38.133127 12495 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:16:38.137794 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.102937 23534 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27.
I20260709 12:16:38.119732 23687 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:38.119138 23635 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.116770 23428 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.099716 23659 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:38.122090 23657 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.118928 23502 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.107983 23647 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.151041 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:38.169070 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:38.176038 23705 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.718s	user 0.004s	sys 0.000s
I20260709 12:16:38.179545 23638 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.180438 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4495) took 1815 ms. Trace:
I20260709 12:16:38.095906 23333 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.189545 12563 rpcz_store.cc:276] 0709 12:16:36.365135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.659163 (+294028us) service_pool.cc:224] Handling call
0709 12:16:36.666203 (+  7040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:38.033931 (+1367728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:38.180095 (+146164us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:38.180348 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3112171776}
I20260709 12:16:38.115959 23697 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:16:38.162360 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:38.199359 12488 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:38.151041 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.180969 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.197098 12646 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:38.201953 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.137787 23473 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.149264 23704 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:16:38.163301 23733 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:16:38.151041 23579 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:38.117456 23467 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.162179 23652 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:16:38.161792 23498 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:38.160497 23659 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:38.187449 23483 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:38.248957 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97956.3us
I20260709 12:16:38.250165 12642 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:38.125193 23573 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.163547 23726 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.087s
I20260709 12:16:38.252548 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.159580 23612 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.073s
W20260709 12:16:38.258042 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 191205us
I20260709 12:16:38.263175 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.266311 12642 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:38.266655 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.267342 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.267967 12642 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:38.218786 23697 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:38.217530 23704 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:38.268829 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:38.268836 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:38.158830 23687 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.272812 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.252990 23659 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:16:38.295622 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.239162 23498 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:38.236114 23639 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:38.297914 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:38.300788 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48399us
I20260709 12:16:38.307736 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.310705 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4666) took 1374 ms. Trace:
I20260709 12:16:38.232841 23473 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.317720 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.319824 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.321417 23502 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:38.322254 23502 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.310429 23730 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:38.324944 23730 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.328521 23730 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:38.310818 23514 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.519s	user 0.016s	sys 0.002s
I20260709 12:16:38.330500 23514 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.332653 23493 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.315507 12491 rpcz_store.cc:276] 0709 12:16:36.936302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.938422 (+  2120us) service_pool.cc:224] Handling call
0709 12:16:36.950360 (+ 11938us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:37.022851 (+ 72491us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:37.022900 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:37.036673 (+ 13773us) log.cc:844] Serialized 26 byte log entry
0709 12:16:37.393596 (+356923us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:37.393759 (+   163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:37.393837 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:38.175069 (+781232us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:38.298825 (+123756us) raft_consensus.cc:1685] finished
0709 12:16:38.298865 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:38.310593 (+ 11728us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29031,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":8040832,"thread_start_us":390022,"threads_started":2,"wal-append.queue_time_us":374141}
I20260709 12:16:38.287760 23704 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:38.349879 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:38.352825 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.368530 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.340914 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1569) took 1221 ms. Trace:
I20260709 12:16:38.366207 13257 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c56dde469374f4da7440f134140670c"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:38.257509 23483 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.337584 12645 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:16:38.339394 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.339712 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:38.339730 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.298439 23638 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:38.406996 13257 raft_consensus.cc:606] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.339980 12644 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:38.409273 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89587.5us
I20260709 12:16:38.399433 23428 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.378548 12640 rpcz_store.cc:276] 0709 12:16:37.119173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:37.126445 (+  7272us) service_pool.cc:224] Handling call
0709 12:16:37.139442 (+ 12997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:37.384939 (+245497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:38.340471 (+955532us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:38.340772 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":563167616}
I20260709 12:16:38.382504 12641 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:16:38.378726 23579 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.337893 23714 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:38.409993 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.350334 23514 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.352659 23165 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:38.397697 23493 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:38.345273 23635 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.077s
I20260709 12:16:38.428069 23579 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:38.434113 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1562) took 3064 ms (client timeout 2764 ms). Trace:
W20260709 12:16:38.430881 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 284485us
I20260709 12:16:38.389154 23477 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:38.433223 12494 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:38.407286 23502 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:38.435089 12536 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:16:38.414582 23428 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:38.441479 12639 rpcz_store.cc:269] 0709 12:16:35.369905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:35.538523 (+168618us) service_pool.cc:224] Handling call
0709 12:16:38.433884 (+2895361us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:38.434022 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4287012096}
W20260709 12:16:38.478381 23709 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.606s	user 0.014s	sys 0.000s
I20260709 12:16:38.478857 23709 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.479053 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.479108 23428 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.479970 12644 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:38.480036 13256 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:16:38.422161 23697 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.421838 23498 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.482556 23498 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:38.442866 23165 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.487645 23165 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.489319 12493 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:38.490423 23701 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.490592 12493 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:38.490974 23701 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.439751 23573 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.424944 23714 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:38.439193 23467 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.450721 13257 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:38.495817 23714 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:38.496183 13257 raft_consensus.cc:606] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:38.496536 13257 raft_consensus.cc:614] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:38.452616 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:38.497709 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.498263 12489 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:16:38.499010 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.499948 23548 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:38.500437 12489 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:16:38.502210 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4654) took 1786 ms. Trace:
I20260709 12:16:38.502668 12496 rpcz_store.cc:276] 0709 12:16:36.715864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.718385 (+  2521us) service_pool.cc:224] Handling call
0709 12:16:36.718735 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:37.138050 (+419315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:38.501967 (+1363917us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:38.502150 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":941658368}
I20260709 12:16:38.511694 23579 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:38.500494 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.514432 23548 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:38.516656 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:38.517056 12565 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:38.370190 23657 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:16:38.518178 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.448431 23483 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.457954 23687 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.458487 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.519677 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:38.471385 12536 leader_election.cc:403] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:16:38.434069 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:38.521406 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86460.9us
I20260709 12:16:38.522706 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.524456 23467 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 24. Current term is 25.
I20260709 12:16:38.524578 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.525259 23467 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:16:38.525573 23428 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:38.526629 23467 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:38.474718 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.527791 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.473907 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:38.528820 12640 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:38.529536 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.478209 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.481349 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.455566 23473 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.531898 12494 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 24.
I20260709 12:16:38.532032 12566 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:38.532967 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.505376 12496 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:16:38.549211 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.549475 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.548797 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:38.552289 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.559079 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:38.560038 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 70410us
W20260709 12:16:38.562451 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.563489 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.561700 12566 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:16:38.566835 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.569191 12490 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:38.570209 12645 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:16:38.570459 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:38.575146 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.577924 23579 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:38.561679 23467 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:38.579205 23687 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:38.580221 23687 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:38.580554 23687 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:38.582608 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.583281 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.584056 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.587020 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.587607 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.592700 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.595016 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.597929 12566 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 18. (term mismatch)
I20260709 12:16:38.598579 12566 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 17
I20260709 12:16:38.598899 12566 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 31.18
I20260709 12:16:38.599228 12566 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:38.604493 12640 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:38.609272 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.613421 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:38.616073 23605 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 12:16:38.619791 23704 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.622464 23704 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.585239 23483 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:38.623679 12644 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:38.616184 12571 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:38.624583 23721 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:38.627429 13257 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:38.629108 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:38.585551 23649 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:38.629798 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.631052 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:38.631363 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72389us
I20260709 12:16:38.629982 13258 raft_consensus.cc:606] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:38.635224 13258 raft_consensus.cc:614] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:38.630172 23649 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:38.655498 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106361us
I20260709 12:16:38.666286 23473 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:38.655534 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46353.8us
I20260709 12:16:38.674856 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.611689 23428 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:38.683485 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49106.8us
I20260709 12:16:38.703241 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.706292 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.718559 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.726495 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.728183 12644 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: [29.18-29.18]
W20260709 12:16:38.666922 23716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.005s	sys 0.021s
I20260709 12:16:38.662542 23467 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:38.671644 23657 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.672864 23649 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:38.671309 23666 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:38.670163 23579 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:38.678520 23477 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:16:38.743587 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 216641us
I20260709 12:16:38.662206 23587 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:38.699730 23705 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:16:38.736775 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:38.581377 23687 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.683316 23611 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:16:38.748872 12488 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.000s
W20260709 12:16:38.759874 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:38.774478 12488 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:38.683194 23704 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:38.742461 23716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.005s	sys 0.021s
I20260709 12:16:38.777663 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:16:38.677594 23473 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:38.798573 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80119.2us
I20260709 12:16:38.766983 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:38.783924 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.759722 23540 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:38.776510 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.787364 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.811439 12643 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:16:38.721657 23333 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 12:16:38.721940 23638 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260709 12:16:38.773999 23611 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:38.751552 23427 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:38.797885 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.798386 12570 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:38.778074 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:38.803049 23687 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:38.806229 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.803507 23701 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:38.804080 23534 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:38.763188 23477 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:38.808681 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.834520 23534 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:38.809476 12566 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:38.835354 23534 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:38.836361 12566 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:16:38.796701 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.817708 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.814688 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.823974 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.826188 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:38.830560 13258 raft_consensus.cc:606] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:38.841290 13258 raft_consensus.cc:614] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:38.843043 13258 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:16:38.844061 23751 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:38.845459 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:16:38.846964 23701 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:38.847513 23701 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:38.848151 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:38.848492 23701 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.849243 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:38.831486 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.851354 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.852810 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.765991 23579 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.832094 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:38.800447 23666 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:38.758208 23657 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.833820 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:38.807467 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.834759 23477 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:38.844411 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.906175 13261 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:38.906888 13258 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:38.785746 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.855387 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.907670 13255 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:38.857040 12496 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:16:38.861339 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:38.910429 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49198.4us
I20260709 12:16:38.912443 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.819294 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:38.862195 13260 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:38.934988 13255 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:38.828001 23674 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:38.774266 23587 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:38.799125 23704 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:38.856628 23611 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:16:38.941582 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122410us
I20260709 12:16:38.942157 13257 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:16:38.881942 23666 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:38.942677 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 242765us
I20260709 12:16:38.960332 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:38.960009 12568 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:38.940726 23427 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:16:38.970769 23715 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.122s	user 0.000s	sys 0.003s
I20260709 12:16:38.946499 23493 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:16:38.975286 23744 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260709 12:16:38.991816 13262 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
W20260709 12:16:39.012013 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 382279us
I20260709 12:16:38.933174 23333 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:39.031873 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 345058us
I20260709 12:16:39.038115 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:39.042590 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:39.046401 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:38.991331 23666 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:39.070811 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.990233 23709 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:39.074828 12494 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:39.082669 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.983206 23743 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:39.084218 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:39.044829 23714 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.548s	user 0.007s	sys 0.004s
I20260709 12:16:39.091343 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:38.958189 23502 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.124s
W20260709 12:16:39.102564 23742 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
I20260709 12:16:39.103509 23587 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:39.105152 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4303) took 1222 ms. Trace:
I20260709 12:16:39.105810 12642 rpcz_store.cc:276] 0709 12:16:37.885869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:38.306332 (+420463us) service_pool.cc:224] Handling call
0709 12:16:38.306719 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:38.308711 (+  1992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:38.308769 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:38.321337 (+ 12568us) log.cc:844] Serialized 26 byte log entry
0709 12:16:38.663682 (+342345us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:38.743446 (+ 79764us) log.cc:844] Serialized 14 byte log entry
0709 12:16:38.749469 (+  6023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:38.749556 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:39.104805 (+355249us) raft_consensus.cc:1685] finished
0709 12:16:39.104838 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:39.105061 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21040,"prepare.queue_time_us":69054,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":7008,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":51554,"thread_start_us":341841,"threads_started":1,"wal-append.queue_time_us":491199}
I20260709 12:16:39.110584 23579 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.112404 23579 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:39.112943 23579 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.113193 23264 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:39.073722 23427 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:16:39.071597 23647 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:39.116933 23756 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:39.118836 23483 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:39.073118 23717 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:39.135668 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:39.070019 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 333334us
W20260709 12:16:39.136943 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.002s	sys 0.017s
I20260709 12:16:39.178330 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:39.181344 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.002s	sys 0.018s
I20260709 12:16:39.184762 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:39.183826 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.105933 23704 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:39.187325 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.181106 23647 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:39.126214 23657 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.110651 23473 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:39.189841 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 198576us
W20260709 12:16:39.190699 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 203693us
W20260709 12:16:39.183218 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 369747us
I20260709 12:16:39.203938 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.180260 23612 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:39.188669 23659 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.880s	user 0.013s	sys 0.000s
I20260709 12:16:39.242998 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:39.235926 23657 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:39.238840 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.007s	sys 0.015s
W20260709 12:16:39.244024 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.008s	sys 0.015s
I20260709 12:16:39.136734 23714 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.244148 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:39.256397 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 176234us
I20260709 12:16:39.261085 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:39.262442 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:39.263849 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:39.265185 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:39.231540 23647 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:39.269870 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.184106 23493 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:16:39.184638 23717 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:39.275087 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.277875 12497 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:16:39.278149 13256 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:39.277983 12637 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:39.202200 23483 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.068s
I20260709 12:16:39.254865 23473 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:39.242785 23704 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.038s
I20260709 12:16:39.253505 23657 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.306808 23657 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.306988 23548 raft_consensus.cc:993] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:39.307956 23548 raft_consensus.cc:1081] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:39.309903 12497 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:39.311198 23769 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.311957 23769 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.312641 23701 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:39.313560 23579 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.314697 23579 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:39.315089 23579 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:39.317528 23748 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:16:39.311538 23483 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.318768 23483 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.321393 23483 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.324334 23534 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.325378 12563 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:39.326020 12563 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:39.326817 12563 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.327220 23717 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.327953 12563 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:39.328805 23714 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:39.268658 23758 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.078s
I20260709 12:16:39.331483 23758 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:39.331851 23758 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:39.298261 23765 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.327450 23534 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:39.334331 23765 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.336082 23133 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:39.336908 23765 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.339694 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:39.338375 12494 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:39.253058 23659 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.270684 23333 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:16:39.251767 23753 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.086s
I20260709 12:16:39.272044 23638 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.066s
I20260709 12:16:39.297200 23467 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:39.358423 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.007s	sys 0.012s
W20260709 12:16:39.360041 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.007s	sys 0.013s
I20260709 12:16:39.332791 23758 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.356981 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:39.337126 23133 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.011s
I20260709 12:16:39.345844 23686 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:39.350656 23687 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.513s	user 0.008s	sys 0.004s
I20260709 12:16:39.371981 12490 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:39.376004 12563 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 20. (term mismatch)
I20260709 12:16:39.407207 12563 pending_rounds.cc:85] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:16:39.407526 12563 pending_rounds.cc:107] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:16:39.407826 12563 raft_consensus.cc:2889] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:39.349923 23717 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:39.350029 23704 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:39.421052 12496 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:16:39.427938 23758 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:39.429294 23428 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.694s	user 0.008s	sys 0.004s
I20260709 12:16:39.430130 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1575) took 1569 ms. Trace:
I20260709 12:16:39.459746 12563 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:16:39.432540 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:39.339051 23579 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.434335 23659 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.407783 23743 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:39.468616 23743 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:39.471033 23649 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.001s
I20260709 12:16:39.471285 23743 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:39.436055 23717 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.471696 23649 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.473798 23717 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:39.473994 12643 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:39.448419 23467 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:39.474812 23579 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:39.475989 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:39.460187 12638 rpcz_store.cc:276] 0709 12:16:37.860363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:37.860627 (+   264us) service_pool.cc:224] Handling call
0709 12:16:37.861096 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:38.054006 (+192910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:39.429846 (+1375840us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:39.430056 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":439899904}
I20260709 12:16:39.474874 23467 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:39.437417 23736 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:16:39.410832 23708 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:39.478497 23736 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:39.478650 23708 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:39.479120 23736 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:39.479856 23708 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.478181 23467 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.462908 23540 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.020s
I20260709 12:16:39.471696 23769 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:39.465485 23428 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:39.491879 12495 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:16:39.492116 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1579) took 1618 ms (client timeout 1500 ms). Trace:
W20260709 12:16:39.492676 12646 rpcz_store.cc:269] 0709 12:16:37.873308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:37.873481 (+   173us) service_pool.cc:224] Handling call
0709 12:16:39.491900 (+1618419us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:39.492034 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":626890624}
I20260709 12:16:39.493218 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4691) took 1271 ms. Trace:
I20260709 12:16:39.494076 12495 rpcz_store.cc:276] 0709 12:16:38.223286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:38.477915 (+254629us) service_pool.cc:224] Handling call
0709 12:16:39.493003 (+1015088us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b540034f154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:39.493140 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2208958720}
I20260709 12:16:39.480290 23736 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:39.423791 23639 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 35 current term: 35 is higher.
I20260709 12:16:39.501339 12563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:16:39.501969 23649 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:39.502233 12563 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:39.502728 12563 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:39.503409 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:39.507440 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45897.5us
W20260709 12:16:39.509433 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.003s	sys 0.011s
W20260709 12:16:39.511806 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.004s	sys 0.011s
I20260709 12:16:39.514914 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.496464 23467 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:39.505470 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 272930us
W20260709 12:16:39.516392 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 246621us
I20260709 12:16:39.523008 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:39.523391 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.533996 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:39.534942 12488 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:39.535118 12563 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.501230 23659 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.014s
W20260709 12:16:39.540316 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.500s (SENT) [suppressed 28 similar messages]
I20260709 12:16:39.539273 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:39.541703 12565 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:39.526911 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:39.520248 23736 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:39.419230 23686 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:39.568755 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45830.5us
I20260709 12:16:39.533078 23649 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:39.533293 23428 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:39.579427 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:39.580269 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:39.546197 23659 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:39.548619 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.581528 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1588) took 2958 ms. Trace:
I20260709 12:16:39.586079 12563 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:39.586017 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:39.598713 13254 rpcz_store.cc:276] 0709 12:16:36.617738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.617960 (+   222us) service_pool.cc:224] Handling call
0709 12:16:37.750944 (+1132984us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:37.840940 (+ 89996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:39.429475 (+1588535us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:39.575783 (+146308us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400578ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:39.576035 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2809497216}
I20260709 12:16:39.586655 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:39.600126 12572 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:39.600761 12565 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:16:39.601788 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53294.2us
I20260709 12:16:39.582346 12566 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:16:39.602321 12565 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
W20260709 12:16:39.603451 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 416361us
I20260709 12:16:39.631373 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:39.633932 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:39.606626 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:39.604848 12566 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:16:39.634882 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:39.616030 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:39.565811 23644 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.081s
I20260709 12:16:39.564672 23767 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:39.638334 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.617401 12644 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
W20260709 12:16:39.650470 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43941.2us
W20260709 12:16:39.707598 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69492.1us
W20260709 12:16:39.654299 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.638303 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:39.642120 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:39.661381 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57357.1us
I20260709 12:16:39.604785 23635 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
W20260709 12:16:39.604382 23738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.007s	sys 0.014s
I20260709 12:16:39.576406 23649 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:39.735785 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131842us
W20260709 12:16:39.749205 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.749498 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:39.760280 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 219960us
I20260709 12:16:39.675231 23573 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.098s
I20260709 12:16:39.762569 23427 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:16:39.677811 23647 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.783242 23649 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:39.673947 23668 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.787803 23668 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.744093 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:39.734055 23652 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:16:39.789302 23727 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.696s	user 0.008s	sys 0.003s
I20260709 12:16:39.583904 23717 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:39.637229 23428 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:39.788183 23647 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:39.789054 23473 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:39.676578 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.790583 12640 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:39.592727 23659 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:39.791354 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.660365 23753 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.821262 23753 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:39.793242 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49282.6us
I20260709 12:16:39.676493 23775 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:39.796113 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4507) took 2117 ms. Trace:
W20260709 12:16:39.800635 23477 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.856s	user 0.007s	sys 0.003s
I20260709 12:16:39.836205 23690 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.002s
I20260709 12:16:39.836522 23477 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.753356 23767 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:39.803256 23668 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.803367 23579 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:39.839273 23579 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:39.839833 23579 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:39.838846 23767 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.803709 23652 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:39.843734 23652 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:39.844249 23652 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:39.808480 23611 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.805s	user 0.014s	sys 0.000s
I20260709 12:16:39.848351 23611 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.850418 23686 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:39.814513 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138043us
I20260709 12:16:39.815161 23514 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:39.797703 23428 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:39.638602 23534 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.161s
W20260709 12:16:39.816499 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 212244us
I20260709 12:16:39.818286 23473 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:39.819715 23659 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:39.820792 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.863554 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:39.792438 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:39.772984 23738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.008s	sys 0.031s
I20260709 12:16:39.824968 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.825326 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:39.734416 12461 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.826259 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:39.838810 23477 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:39.843888 23785 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:16:39.807891 23666 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.717s	user 0.010s	sys 0.000s
I20260709 12:16:39.833418 12567 rpcz_store.cc:276] 0709 12:16:37.673821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:37.673993 (+   172us) service_pool.cc:224] Handling call
0709 12:16:37.674469 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:37.677852 (+  3383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:37.677889 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:37.699936 (+ 22047us) log.cc:844] Serialized 26 byte log entry
0709 12:16:37.847588 (+147652us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:37.847638 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:37.847721 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:38.295971 (+448250us) spinlock_profiling.cc:243] Waited 7.28 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:16:39.584666 (+1288695us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:39.791125 (+206459us) raft_consensus.cc:1685] finished
0709 12:16:39.791150 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:39.791336 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2901,"raft.queue_time_us":3578,"raft.run_cpu_time_us":3168,"raft.run_wall_time_us":608878,"spinlock_wait_cycles":64531456,"thread_start_us":102352,"threads_started":1,"wal-append.queue_time_us":303715}
I20260709 12:16:39.851624 12492 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:39.663450 23772 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.233s
I20260709 12:16:39.859154 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4705) took 1007 ms. Trace:
I20260709 12:16:39.821666 23663 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:39.823168 23674 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:16:39.869877 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.762540 23543 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.101s
W20260709 12:16:39.873037 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 158828us
I20260709 12:16:39.873178 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.734272 23783 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:39.875263 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50362us
I20260709 12:16:39.897179 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.833158 23775 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.898105 23635 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:39.898499 23775 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:39.898156 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:39.851725 23539 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:16:39.901027 23647 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:39.834126 23644 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:16:39.874806 12493 rpcz_store.cc:276] 0709 12:16:38.851658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:38.851838 (+   180us) service_pool.cc:224] Handling call
0709 12:16:38.852257 (+   419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:38.852647 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:38.852672 (+    25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:38.853424 (+   752us) log.cc:844] Serialized 52 byte log entry
0709 12:16:39.360458 (+507034us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:39.425942 (+ 65484us) log.cc:844] Serialized 14 byte log entry
0709 12:16:39.426230 (+   288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:39.426319 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:39.839626 (+413307us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b180037b640. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:39.852730 (+ 13104us) raft_consensus.cc:1685] finished
0709 12:16:39.858880 (+  6150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:39.859078 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33051,"prepare.queue_time_us":59308,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":182877,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":18418,"spinlock_wait_cycles":39227264,"thread_start_us":506540,"threads_started":1,"wal-append.queue_time_us":614291}
I20260709 12:16:39.901937 23534 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:16:39.902796 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.866631 23428 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:39.898732 23635 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:39.903853 23635 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:16:39.884512 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59256.5us
I20260709 12:16:39.906531 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.907825 23427 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:16:39.908211 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:39.881829 23666 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.909647 23783 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.911867 23666 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:39.873878 23753 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.861573 23514 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:39.912887 23783 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:39.913461 23514 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.916145 23709 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:39.871450 23639 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:39.876785 23467 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:16:39.919888 23755 raft_consensus.cc:993] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: : Instructing follower 32889a507a6b4c74aec421ffbfd1e25d to start an election
I20260709 12:16:39.920326 23467 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:39.920579 23755 raft_consensus.cc:1081] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Signalling peer 32889a507a6b4c74aec421ffbfd1e25d to start an election
I20260709 12:16:39.921895 23477 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:39.876313 23697 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:39.922695 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:39.923121 23697 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:39.923519 23697 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:39.916298 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:39.926510 12493 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:16:39.863095 23473 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:39.892861 23758 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:16:39.933605 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80585us
I20260709 12:16:39.901577 23647 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:39.884541 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:39.934809 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50412.6us
I20260709 12:16:39.881973 23659 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:39.935638 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.969904 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:39.986881 12570 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:39.960446 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:39.987668 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:39.988271 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:39.988513 12563 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:16:39.945989 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:39.990592 12490 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:39.994838 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:39.920760 23467 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:39.931746 23473 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.998551 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:40.005039 12571 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:39.999325 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:39.943543 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:40.002444 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.007578 12563 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:40.011051 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67595.5us
I20260709 12:16:40.013931 12644 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:40.015141 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:39.941461 23772 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:40.004058 23686 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:40.019871 23686 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.004294 23751 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:39.943810 23663 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.079s
I20260709 12:16:40.022722 23751 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.024715 23644 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.025141 23751 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.025439 23644 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.025803 23781 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:40.010936 23645 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:39.940598 23477 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:40.026831 23645 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:39.944815 23543 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:40.028146 23543 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:40.028501 23543 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:40.029083 23645 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:39.937680 23573 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.092s
I20260709 12:16:40.032003 23769 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:40.027827 23644 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:40.035022 23647 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.035709 23587 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.036254 23587 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.038450 23587 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.025902 23668 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.011245 23775 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:39.982980 23635 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.050824 12644 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:16:40.054111 12490 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:39.983328 23674 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.105s
I20260709 12:16:40.066779 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:16:40.059441 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:40.064191 23502 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:40.054059 23775 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.084352 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.087025 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.088989 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:40.096961 12572 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:40.097126 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:40.098541 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.098716 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:40.098721 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:40.100512 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:40.101549 12644 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27. Re-sending same reply.
I20260709 12:16:39.874289 23717 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:40.092198 12563 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:40.139230 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:40.139704 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.061798 23668 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.110275 23644 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:40.110757 23333 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:40.111716 23467 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.112718 23753 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.116209 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.195662 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:40.102767 23502 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.056612 23635 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:40.061971 23778 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.103067 23605 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:40.188418 23775 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.025s
I20260709 12:16:40.128712 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:40.125334 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:40.216922 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:40.136650 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.216969 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:40.218329 12570 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:40.130895 23579 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.141300 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.176851 12644 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c56dde469374f4da7440f134140670c"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:35404
I20260709 12:16:40.169140 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:40.181123 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:40.121292 23428 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.171286 23674 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:40.128000 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:40.233305 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52361us
I20260709 12:16:40.235409 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:40.189421 23333 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:40.142704 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:40.247084 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104458us
I20260709 12:16:40.220675 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:40.226049 13256 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:40.225704 12563 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:40.226861 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:40.192406 23652 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.190192 23768 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:16:40.228474 23498 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:40.256524 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:40.218523 23467 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:40.243633 23428 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.206918 23644 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:40.223228 23686 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.200425 23649 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.257149 23751 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:40.256040 23534 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:40.259994 23540 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:40.250893 23333 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:40.271555 12496 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:16:40.204787 23753 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.280292 12637 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 31. Current term is 32. Ops: []
W20260709 12:16:40.258590 23736 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.688s	user 0.011s	sys 0.002s
I20260709 12:16:40.250710 23543 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.231983 23755 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:16:40.261610 23661 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:40.219815 23778 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.255453 23697 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.273118 23751 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:40.267714 23635 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:40.303014 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:16:40.306010 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:40.289151 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 300960us
I20260709 12:16:40.280205 23534 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:40.317914 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:40.325892 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:40.299654 23760 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:40.309580 23751 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:40.308744 23645 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:40.308809 23635 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:40.340794 23781 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:40.345909 23502 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:40.347434 23796 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:16:40.354054 23772 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.354898 23540 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:40.358076 23611 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:40.360734 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4320) took 1100 ms. Trace:
I20260709 12:16:40.389406 12640 rpcz_store.cc:276] 0709 12:16:39.260235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:39.266685 (+  6450us) service_pool.cc:224] Handling call
0709 12:16:39.267021 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:39.797832 (+530811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:40.360301 (+562469us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:40.360628 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1203401088}
I20260709 12:16:40.318938 23579 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:40.370877 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45101.8us
I20260709 12:16:40.349221 23543 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:40.381304 23264 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.332005 23802 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:16:40.371552 23797 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:16:40.403955 23264 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.392417 23540 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.407102 23264 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:40.370069 23635 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:40.410349 13253 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:40.410836 13253 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:40.421259 23498 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:40.425096 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.425750 23534 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.428843 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.398527 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:40.402099 23652 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.403913 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:40.434307 13253 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.441808 13253 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:40.404109 23802 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:40.429593 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:40.432715 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:40.435870 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.379335 23645 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:40.447440 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:40.454975 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56550.1us
I20260709 12:16:40.456184 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:40.463843 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:40.469715 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:40.327921 23644 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.477785 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.481451 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:40.489908 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:40.471629 23798 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.225s	user 0.002s	sys 0.000s
I20260709 12:16:40.379750 23781 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.461910 23579 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:40.389283 23772 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.495988 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:40.498903 23645 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.446406 23802 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:40.512106 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.516180 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.519658 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.521024 12643 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:40.522423 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.483289 23639 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:16:40.539729 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:40.521085 13253 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:40.546860 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.543215 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:40.527937 23649 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:40.537186 23428 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:40.578461 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.569823 23639 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:40.574153 23428 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.573204 23697 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:40.594223 12644 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:40.594754 12644 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:40.578644 23753 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.586218 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.598284 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:40.603606 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:40.611955 13256 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:40.619511 12643 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:40.620801 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:40.592075 23428 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:40.590859 23778 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.633365 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:40.634439 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:40.624987 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:40.593655 23639 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:40.637766 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:40.639348 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:40.616379 23767 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:40.647501 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.649896 12492 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:40.646445 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.651363 12495 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:16:40.652009 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.647153 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.658042 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:40.618451 12644 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.659552 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.621587 12643 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:16:40.665030 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60369.7us
I20260709 12:16:40.664875 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.641430 23756 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.672995 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.674819 12642 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [32.18-32.18]
W20260709 12:16:40.644412 23807 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.243s	user 0.000s	sys 0.002s
I20260709 12:16:40.673091 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.629904 23690 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.698783 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:40.699638 12493 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:16:40.664938 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 180245us
I20260709 12:16:40.713943 13254 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.22-32.22]
I20260709 12:16:40.664247 23587 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.664444 23652 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.676734 23649 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:40.649421 23686 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:40.695894 23811 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:16:40.753926 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:40.720031 23812 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:16:40.787120 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:40.787881 13253 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:40.786154 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:40.789109 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122992us
I20260709 12:16:40.767714 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:16:40.775710 23539 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:16:40.777421 23817 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:16:40.777841 23639 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.665971 23822 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:16:40.708611 23801 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:16:40.783825 23717 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:40.813511 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 134495us
I20260709 12:16:40.715682 23690 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:40.817373 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 353557us
I20260709 12:16:40.791244 23787 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:40.780438 23781 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:40.703632 23756 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.785872 23772 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:40.811033 23649 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:40.762524 23587 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.764027 23652 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.785352 23686 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:40.869998 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159210us
I20260709 12:16:40.873220 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:40.852805 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:40.840157 23787 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:40.881052 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:40.885918 23778 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:40.887331 12569 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: [35.19-35.19]
I20260709 12:16:40.887522 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:40.824143 23611 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:16:40.900691 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:40.841615 23717 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:40.912298 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:40.927677 12569 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.18-33.18]
W20260709 12:16:40.918085 23818 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
I20260709 12:16:40.940732 12569 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:40.927860 23787 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:40.970026 23826 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:16:40.970503 23787 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:40.970736 23814 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:16:40.976030 12569 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:40.980043 23839 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:40.980732 23839 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:40.981129 23839 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:40.987257 23652 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:40.989640 23785 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:16:41.006855 23829 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:16:40.904801 23778 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:40.914984 23768 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:40.927349 23810 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.268s	user 0.000s	sys 0.002s
W20260709 12:16:40.927268 23816 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
W20260709 12:16:40.958274 23776 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:16:41.033787 23700 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.035161 23760 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:40.910920 23635 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:41.041494 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 160498us
I20260709 12:16:40.926894 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:40.932644 23674 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:41.055621 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:40.960486 23838 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:16:41.057559 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130752us
I20260709 12:16:41.066780 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:41.069861 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.017729 23736 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:41.073607 12493 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:16:41.067538 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.094126 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.111636 12642 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:41.114774 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.055994 23760 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:41.117249 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:41.061250 23769 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:41.124723 23832 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:16:41.126515 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4556) took 1050 ms. Trace:
I20260709 12:16:41.126924 12568 rpcz_store.cc:276] 0709 12:16:40.076228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.471576 (+395348us) service_pool.cc:224] Handling call
0709 12:16:40.493008 (+ 21432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:40.596257 (+103249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:40.596298 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:40.596943 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 12:16:40.787462 (+190519us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:40.787497 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:40.787566 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:41.126254 (+338688us) raft_consensus.cc:1685] finished
0709 12:16:41.126274 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.126449 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":42427,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":7309,"thread_start_us":292815,"threads_started":2,"wal-append.queue_time_us":428300}
I20260709 12:16:41.128739 23540 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:41.119372 23668 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.129995 23165 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:41.130496 23165 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:41.131615 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2136) took 1194 ms. Trace:
I20260709 12:16:41.131842 23165 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.132164 12565 rpcz_store.cc:276] 0709 12:16:39.936829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:39.938929 (+  2100us) service_pool.cc:224] Handling call
0709 12:16:39.943259 (+  4330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:40.028203 (+ 84944us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:40.028237 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:40.028280 (+    43us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:40.028312 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.131355 (+1103043us) spinlock_profiling.cc:243] Waited 84.7 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:41.131400 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.131552 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194834304}
I20260709 12:16:41.133234 23165 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:41.133716 13262 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:16:41.134228 23690 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.110546 23827 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:41.136677 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4551) took 1145 ms. Trace:
I20260709 12:16:41.136942 23827 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:41.146770 23827 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:40.924959 23649 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.122154 23635 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.115144 23652 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:41.153050 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:41.153957 23819 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.483s	user 0.004s	sys 0.000s
I20260709 12:16:41.123235 23717 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.058547 23674 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:41.109854 23840 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.060452 23700 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:41.142570 23333 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.858s	user 0.014s	sys 0.000s
I20260709 12:16:41.080178 23767 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:41.079731 23768 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:41.110880 23736 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:41.172533 23751 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.827s	user 0.007s	sys 0.007s
W20260709 12:16:41.163143 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.113420 23264 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:41.168013 12494 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:16:41.168381 12497 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:41.145704 23775 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:41.192982 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:41.193207 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.177773 12644 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.173454 23652 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:16:41.184293 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:41.177237 23467 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.910s	user 0.016s	sys 0.000s
I20260709 12:16:41.190907 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:41.210722 13257 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:41.181917 23840 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.211805 23778 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:41.176095 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64283.8us
I20260709 12:16:41.213183 23840 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.213464 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:41.168186 23769 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:41.214128 23548 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:41.214860 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.214550 23769 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.215692 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.216099 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:41.173908 23645 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:41.216744 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:41.217164 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:41.217311 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.218056 12565 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:41.218904 23700 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.219580 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4540) took 1264 ms. Trace:
I20260709 12:16:41.221345 12568 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:41.185362 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.222790 12568 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:16:41.205587 23674 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.223719 23674 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:41.188933 23767 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:41.225869 23767 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:41.227012 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1601) took 1026 ms. Trace:
I20260709 12:16:41.227538 12639 rpcz_store.cc:276] 0709 12:16:40.201050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.201264 (+   214us) service_pool.cc:224] Handling call
0709 12:16:40.201753 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:40.221311 (+ 19558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:40.221353 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:40.221409 (+    56us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:40.221455 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.226722 (+1005267us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:41.226778 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.226941 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44363520}
W20260709 12:16:41.228886 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.214661 23548 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.229857 23768 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:41.230288 23768 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:41.230682 23768 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:41.230988 23736 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.231539 23548 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:41.232175 23802 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.002s
I20260709 12:16:41.217068 23645 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.233544 23264 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:41.234977 23645 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:41.236297 23264 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:41.236372 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:41.237085 23264 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:41.137215 12571 rpcz_store.cc:276] 0709 12:16:39.990808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.046453 (+ 55645us) service_pool.cc:224] Handling call
0709 12:16:40.046951 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:40.329866 (+282915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:40.329901 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:40.330465 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:16:40.529828 (+199363us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:40.529874 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.136126 (+606252us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:41.136197 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:41.136257 (+    60us) raft_consensus.cc:1685] finished
0709 12:16:41.136274 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.136603 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3250,"raft.queue_time_us":4921,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":8165,"spinlock_wait_cycles":641857920,"thread_start_us":198991,"threads_started":1,"wal-append.queue_time_us":268814}
I20260709 12:16:41.238889 12641 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:41.219987 12567 rpcz_store.cc:276] 0709 12:16:39.955078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:39.955303 (+   225us) service_pool.cc:224] Handling call
0709 12:16:39.955771 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:39.956294 (+   523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:39.956335 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:39.993495 (+ 37160us) log.cc:844] Serialized 26 byte log entry
0709 12:16:40.278398 (+284903us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:41.209589 (+931191us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b18004eebc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:16:41.218869 (+  9280us) log.cc:844] Serialized 14 byte log entry
0709 12:16:41.219142 (+   273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.219249 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:41.219292 (+    43us) raft_consensus.cc:1685] finished
0709 12:16:41.219387 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.219517 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":99237,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":4151,"raft.queue_time_us":26560,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":153,"spinlock_wait_cycles":35370368,"thread_start_us":362100,"threads_started":2,"wal-append.queue_time_us":300869}
W20260709 12:16:41.238719 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 413274us
I20260709 12:16:41.246737 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.193737 13253 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:41.247927 13253 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:41.248998 13253 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.250118 13253 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:41.198589 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.253222 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:41.192943 23775 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:41.195431 23811 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.141s	user 0.000s	sys 0.003s
I20260709 12:16:41.232662 23802 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.258575 12493 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:41.219350 23477 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:41.263422 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:35404 (request call id 1599) took 1059 ms. Trace:
I20260709 12:16:41.270563 12644 rpcz_store.cc:276] 0709 12:16:40.174739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.176052 (+  1313us) service_pool.cc:224] Handling call
0709 12:16:41.124556 (+948504us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eaa3d 00007ffbfa7bacd0 00007ffbf3d3f254 00007ffbf3d3f1e2 00007ffbf3d3f111 00007ffbf3d3f08c 00007ffbf3d3e342 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:41.197728 (+ 73172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":849457408}
I20260709 12:16:41.264750 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:41.260463 12490 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:16:41.272998 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155811us
I20260709 12:16:41.273813 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:41.275539 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.219434 23700 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.191413 23605 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.182639 23333 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.216759 23769 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.280822 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.281916 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4555) took 1288 ms. Trace:
I20260709 12:16:41.282819 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:41.285208 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:41.268492 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.007s	sys 0.007s
I20260709 12:16:41.237845 23652 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.315698 23652 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.278189 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:41.278496 23467 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:41.289425 12638 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:16:41.275637 23428 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.637s	user 0.013s	sys 0.001s
I20260709 12:16:41.292862 13253 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:41.262362 23778 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:41.278299 23840 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:41.305267 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:41.274922 23645 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:41.195734 23751 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:41.318876 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.007s	sys 0.007s
I20260709 12:16:41.294004 12570 rpcz_store.cc:276] 0709 12:16:39.993216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.311830 (+318614us) service_pool.cc:224] Handling call
0709 12:16:40.315070 (+  3240us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:40.324794 (+  9724us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:40.324921 (+   127us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:40.331390 (+  6469us) log.cc:844] Serialized 26 byte log entry
0709 12:16:40.487734 (+156344us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:40.487794 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:40.487886 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:41.121255 (+633369us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b24001522b8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:16:41.277194 (+155939us) raft_consensus.cc:1685] finished
0709 12:16:41.277217 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.281843 (+  4626us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8874,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":23442176,"thread_start_us":142716,"threads_started":1,"wal-append.queue_time_us":242850}
I20260709 12:16:41.331219 12646 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:41.333205 12491 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:41.330668 23661 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.334242 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:41.318310 23264 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:41.340920 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.358551 12490 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:41.341915 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.291220 23605 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.359544 12570 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:41.332267 23428 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.360517 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.344022 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.345048 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.346028 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.308467 23756 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.361234 23605 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:41.335116 23778 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.368495 12565 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:41.328405 23775 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.370220 23778 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.370865 23775 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:41.374680 23428 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.373306 23473 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:41.366102 12496 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:41.382678 23813 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.799s	user 0.005s	sys 0.000s
I20260709 12:16:41.389586 23772 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:41.393121 23772 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:41.337447 23840 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.394858 23668 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:16:41.395272 23668 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:41.396463 23840 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.396219 23668 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:41.398433 23838 consensus_queue.cc:1175] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 32 status { last_received { term: 32 index: 18 } last_committed_idx: 18 last_received_current_leader { term: 32 index: 18 } }
I20260709 12:16:41.398991 23333 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:41.353684 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.283509 23839 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.399966 23587 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.401327 23839 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.335400 23847 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.402436 23847 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.352521 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.403818 12497 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:41.404743 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.398638 23264 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:41.405802 23686 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.364976 23751 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.286991 23700 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.413007 23700 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.292021 23579 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:41.414621 23579 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.417963 23661 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.421028 23661 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.421986 23652 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.422711 23652 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.423245 23652 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.427661 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:41.428151 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:41.428480 13255 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:41.429208 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:41.430915 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:41.431318 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:41.437999 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:41.452478 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:41.453792 13253 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:41.457803 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.472549 23751 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:41.474205 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1401) took 1999 ms. Trace:
I20260709 12:16:41.447482 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:41.481007 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:41.405548 23847 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:41.484210 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 201485us
W20260709 12:16:41.460084 23788 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.014s	sys 0.022s
I20260709 12:16:41.476362 13262 rpcz_store.cc:276] 0709 12:16:39.475059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:39.489337 (+ 14278us) service_pool.cc:224] Handling call
0709 12:16:39.489798 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:41.134181 (+1644383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.469981 (+335800us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:41.474112 (+  4131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3779909888}
I20260709 12:16:41.493649 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:41.500133 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:41.502023 12568 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:41.503336 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:41.504691 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:41.506489 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.473322 23751 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:41.511940 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.531816 23428 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.512120 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:41.492177 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44801.5us
W20260709 12:16:41.474622 23831 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.633s	user 0.000s	sys 0.005s
I20260709 12:16:41.417284 23579 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.526872 13255 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:41.450229 23477 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.507921 23473 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:41.558995 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47502.5us
I20260709 12:16:41.512008 23778 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:41.519637 23788 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.015s	sys 0.023s
I20260709 12:16:41.511869 23686 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.512069 23769 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:41.561568 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:41.555186 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.572942 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.534482 23652 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.483125 23760 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.579006 13255 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:16:41.586212 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4557) took 1509 ms. Trace:
I20260709 12:16:41.589286 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:41.561920 23639 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.588900 12572 rpcz_store.cc:276] 0709 12:16:40.077113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.859699 (+782586us) service_pool.cc:224] Handling call
0709 12:16:40.860061 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:40.860338 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:16:40.885618 (+ 25280us) log.cc:844] Serialized 14 byte log entry
0709 12:16:41.585800 (+700182us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:41.585865 (+    65us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:41.585918 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.585997 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.586145 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5613,"prepare.queue_time_us":25031,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":68,"raft.queue_time_us":33477,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":11781,"thread_start_us":684002,"threads_started":2,"wal-append.queue_time_us":682430}
W20260709 12:16:41.570473 23770 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.004s	sys 0.029s
I20260709 12:16:41.568459 23645 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:41.532721 23428 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.572544 23333 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.627648 12492 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:41.576840 23830 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:41.632704 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71221.4us
I20260709 12:16:41.639145 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.642999 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.643988 12646 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: [35.19-35.19]
W20260709 12:16:41.625610 23770 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.004s	sys 0.030s
I20260709 12:16:41.609114 23579 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.646338 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.572842 23778 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.571274 23477 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.582203 23686 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.608752 23827 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.606920 23644 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:16:41.610595 23788 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.083s
I20260709 12:16:41.612186 23639 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:41.696355 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89011.8us
W20260709 12:16:41.745824 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.023s	sys 0.019s
I20260709 12:16:41.651238 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:41.754040 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.024s	sys 0.020s
W20260709 12:16:41.652839 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.676725 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.651526 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.757591 12642 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:41.758184 12645 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:41.699290 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:41.651893 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.756235 12646 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
I20260709 12:16:41.649308 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:41.806257 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.022s	sys 0.049s
W20260709 12:16:41.823092 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.025s	sys 0.057s
W20260709 12:16:41.823784 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 549991us
I20260709 12:16:41.824571 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:41.788830 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.819494 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.820096 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.820689 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.826627 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:41.814327 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58884.4us
I20260709 12:16:41.827270 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:16:41.825402 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 176205us
I20260709 12:16:41.803552 12643 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:41.828999 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:41.693186 23781 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:41.855198 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:41.700452 23802 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:41.857936 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102143us
I20260709 12:16:41.675997 23652 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:41.801929 23579 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.805964 23828 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:41.862145 12566 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:16:41.857573 23548 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.021s
I20260709 12:16:41.805299 23853 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.626320 23768 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.858090 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.648617 23767 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:41.654341 23830 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.816157 23663 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:41.648375 23333 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.888927 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:41.895752 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.896909 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:41.897385 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.897810 23849 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:41.899309 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:41.874490 23851 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.220s	user 0.000s	sys 0.000s
I20260709 12:16:41.900810 12488 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
W20260709 12:16:41.875123 23854 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:16:41.866719 23686 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.876422 23751 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:41.875641 23856 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.292s	user 0.000s	sys 0.001s
I20260709 12:16:41.902083 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4718) took 1957 ms. Trace:
W20260709 12:16:41.873204 23857 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260709 12:16:41.902962 12489 rpcz_store.cc:276] 0709 12:16:39.945029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:39.945203 (+   174us) service_pool.cc:224] Handling call
0709 12:16:39.945659 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:39.946086 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:39.946127 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:39.976827 (+ 30700us) log.cc:844] Serialized 26 byte log entry
0709 12:16:40.970418 (+993591us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b18004876c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:16:41.324132 (+353714us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:16:41.324194 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.324307 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:41.901707 (+577400us) raft_consensus.cc:1685] finished
0709 12:16:41.901739 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:41.901991 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40011,"raft.queue_time_us":6477,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":12042,"spinlock_wait_cycles":47314688,"thread_start_us":353446,"threads_started":1,"wal-append.queue_time_us":1607020}
W20260709 12:16:41.904656 23165 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.729s	user 0.008s	sys 0.002s
I20260709 12:16:41.877662 23760 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:41.876317 23858 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 12:16:41.875912 23548 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:41.867249 23639 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.905845 23548 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:41.871912 23579 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.874147 23828 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:41.906735 23828 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:41.873594 23852 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:16:41.876412 23855 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:41.871389 23840 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:41.892601 23663 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.872392 23428 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.908427 23721 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:16:41.922716 23760 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:41.909739 12642 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:41.924036 23760 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.924911 23781 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:41.909569 12571 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:41.869760 23477 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.878912 23778 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.864900 23827 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.926055 23781 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.915297 12645 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:41.931810 12494 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:41.931532 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.932586 12489 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:41.892287 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:41.936777 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47958.5us
I20260709 12:16:41.939333 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:41.879232 23674 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.655s	user 0.007s	sys 0.005s
I20260709 12:16:41.937747 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:41.939333 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.942045 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:41.916428 23663 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:41.944950 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81976.8us
I20260709 12:16:41.940331 12494 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:16:41.906874 23767 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.863194 23867 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:41.970993 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:41.971354 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:41.974203 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:41.975345 12488 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:41.977763 12488 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: [35.14-35.14]
I20260709 12:16:41.912508 23333 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.906924 23548 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.980074 23768 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:41.945614 23165 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:41.982777 23165 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:41.983188 23165 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:41.911518 23855 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:41.986440 23855 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.950313 23686 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:41.876857 23853 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:41.996376 23853 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:41.947989 23751 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:41.997570 23751 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:41.997987 23751 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:41.876469 23834 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.336s
I20260709 12:16:41.935133 23788 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 12:16:41.920097 23830 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.005035 23760 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.005791 23668 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:16:42.006173 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:41.943365 23721 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:42.007588 23721 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:41.863219 23847 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:42.008546 23721 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:42.010558 23264 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.605s	user 0.012s	sys 0.000s
I20260709 12:16:41.980108 23548 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:42.011828 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.012717 12571 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:42.014566 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:42.016141 23663 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:42.017096 23635 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:42.018787 23649 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:42.013533 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.022771 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:42.023909 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.024588 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:42.025489 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:42.026611 12571 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:42.028272 23652 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:42.029608 23467 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:42.030597 23427 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:16:42.030884 23775 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.659s	user 0.008s	sys 0.004s
W20260709 12:16:42.033737 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86940.5us
I20260709 12:16:42.097671 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.034987 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.038859 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.042598 23751 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:42.043256 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71965.7us
I20260709 12:16:42.063277 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.062152 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:42.050774 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.056912 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.048662 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.100209 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:42.045543 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.128456 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.101500 12566 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:42.046865 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:42.129772 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:42.129954 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.130393 12566 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:42.130656 12569 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:42.131587 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.052248 12495 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:42.132468 12495 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:42.102334 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:16:42.092376 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76560.6us
I20260709 12:16:42.114151 23788 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.116505 23865 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.117476 23827 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.120038 23778 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:42.047323 23836 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:42.086550 23467 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:42.095428 23767 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.134959 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:42.145219 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.149763 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:42.150368 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.124559 23514 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:42.132761 12643 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:42.096642 23427 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:42.139257 23847 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:16:42.179075 23836 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:42.213533 23514 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:42.175994 23827 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.216041 23867 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:42.176996 23865 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.169497 23788 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.232052 23427 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.211159 23767 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.239383 23514 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:42.256840 23649 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:42.275061 23427 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:42.243254 23467 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 35.14, Last appended by leader: 14, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.294762 23467 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:42.265676 23836 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.304351 23635 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:42.304868 23865 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.308764 23767 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.294523 23828 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.302979 23751 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.313009 23788 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.313755 23827 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.147985 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:42.124078 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:42.352087 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 456176us
I20260709 12:16:42.435155 23605 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.135573 12495 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.438824 12495 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:42.123050 12497 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:42.051816 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:42.442848 12567 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:42.143726 12563 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:16:42.360445 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:42.361699 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:42.362807 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:42.206904 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72073.9us
I20260709 12:16:42.364022 12489 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:42.454473 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:42.121542 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:42.456476 12490 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:42.189384 12571 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:42.458307 12496 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:42.456533 12488 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:42.374603 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:42.462568 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 341295us
I20260709 12:16:42.456573 23767 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:42.426890 23751 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.464064 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:42.360321 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 25.
I20260709 12:16:42.386368 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.315651 23828 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:42.315251 23836 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:42.466702 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:42.472749 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.473052 12571 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:16:42.402249 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:42.408202 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 307943us
W20260709 12:16:42.402284 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.397610 23756 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:42.464792 23751 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.389665 23644 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:42.421536 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:42.436605 23605 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.484709 23775 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.442751 23709 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:42.486085 23709 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.358865 12568 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:16:42.306214 23635 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:42.489590 12568 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:16:42.479454 23865 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:42.480443 23867 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:16:42.503646 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.484743 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:42.513990 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:42.515386 23877 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.211s	user 0.000s	sys 0.002s
W20260709 12:16:42.515517 23861 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.818s	user 0.007s	sys 0.002s
I20260709 12:16:42.520051 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:42.521379 12496 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:16:42.521600 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4546) took 2551 ms. Trace:
I20260709 12:16:42.522243 12564 rpcz_store.cc:276] 0709 12:16:39.969687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:40.010650 (+ 40963us) service_pool.cc:224] Handling call
0709 12:16:40.011106 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:40.016491 (+  5385us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:16:40.290464 (+273973us) log.cc:844] Serialized 14 byte log entry
0709 12:16:41.139996 (+849532us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b18003f5bc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:16:41.484127 (+344131us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:41.510420 (+ 26293us) log.cc:844] Serialized 26 byte log entry
0709 12:16:41.510615 (+   195us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:41.510655 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:41.510752 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:42.514062 (+1003310us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:42.521121 (+  7059us) raft_consensus.cc:1685] finished
0709 12:16:42.521147 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.521420 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10117,"prepare.queue_time_us":284418,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":44,"raft.queue_time_us":16756,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":27162,"spinlock_wait_cycles":21317760,"thread_start_us":593346,"threads_started":2,"wal-append.queue_time_us":1277845}
I20260709 12:16:42.491772 23635 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.527449 23635 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:16:42.529891 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.530622 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:42.531621 12492 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:42.490156 23875 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.049s
I20260709 12:16:42.539719 23875 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:42.540246 23875 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:42.541127 23697 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:42.541913 23697 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.542778 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:42.544590 23697 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.544867 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:42.548465 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:42.550006 23264 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:42.550558 23264 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:42.551108 23264 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:42.550025 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80494.2us
I20260709 12:16:42.552620 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:42.555179 23708 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.549501 23853 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:42.550331 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64636.7us
I20260709 12:16:42.569618 23827 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.570730 23827 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.578254 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:42.578636 23708 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:42.585204 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.587755 13254 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:16:42.588320 23875 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.591090 23709 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:42.604873 23668 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.598s	user 0.013s	sys 0.006s
I20260709 12:16:42.561784 23775 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.606072 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2142) took 1010 ms. Trace:
I20260709 12:16:42.608840 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.610462 23751 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.622037 23865 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:42.615872 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4679) took 1109 ms. Trace:
I20260709 12:16:42.622581 23605 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.624257 23853 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:42.622658 23652 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:42.625581 12563 rpcz_store.cc:276] 0709 12:16:41.595234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.469128 (+873894us) service_pool.cc:224] Handling call
0709 12:16:42.603552 (+134424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.603913 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.603947 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.604021 (+    74us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:42.604061 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.604137 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.605645 (+  1508us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54002cf1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:42.605815 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307660672}
W20260709 12:16:42.634688 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86567.9us
I20260709 12:16:42.630628 12572 rpcz_store.cc:276] 0709 12:16:41.506192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.050486 (+544294us) service_pool.cc:224] Handling call
0709 12:16:42.611105 (+560619us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:42.615752 (+  4647us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":572130688}
I20260709 12:16:42.633881 23652 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:42.645525 23778 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:40.903228 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:42.656221 23778 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.647019 23767 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.658427 23767 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.658984 23867 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.659734 23867 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.660676 12565 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:16:42.661296 23767 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.651760 12461 leader_election.cc:400] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
I20260709 12:16:42.663549 12461 leader_election.cc:403] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:42.663905 23827 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:42.668522 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 158427us
I20260709 12:16:42.672703 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:42.673597 13255 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:42.650429 23853 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:42.673581 23605 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:42.675997 23709 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:16:42.701083 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:42.702026 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.711578 12495 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:16:42.712258 12495 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 17
I20260709 12:16:42.712620 12495 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:16:42.712976 12495 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:42.714843 23775 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.677806 23708 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:42.728755 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 244199us
I20260709 12:16:42.692857 23875 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:42.675760 23165 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.698949 23605 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:42.745465 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 193167us
I20260709 12:16:42.746320 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4575) took 1143 ms. Trace:
I20260709 12:16:42.746820 12565 rpcz_store.cc:276] 0709 12:16:41.602598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.652513 (+1049915us) service_pool.cc:224] Handling call
0709 12:16:42.655848 (+  3335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.661187 (+  5339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.746032 (+ 84845us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:42.746263 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9567616}
I20260709 12:16:42.748819 12565 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:42.749033 23827 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.749989 23827 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.751652 23165 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.752650 23827 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.647279 23652 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.753957 23652 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:42.754063 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 188414us
W20260709 12:16:42.756260 23659 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 34 current term: 34 is higher.
W20260709 12:16:42.756922 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.756328 23775 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:42.758273 23775 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.760350 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4681) took 1157 ms. Trace:
I20260709 12:16:42.760908 23775 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.731736 23867 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.762266 12564 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:16:42.762831 12564 pending_rounds.cc:85] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 17
I20260709 12:16:42.762876 23778 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.763000 23847 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.762174 23751 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.764722 23847 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.719127 23853 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.761838 23772 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.767930 23772 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.769837 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4583) took 1159 ms. Trace:
I20260709 12:16:42.763142 12564 pending_rounds.cc:107] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:16:42.770632 12564 raft_consensus.cc:2889] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:42.770380 12565 rpcz_store.cc:276] 0709 12:16:41.610141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.747893 (+1137752us) service_pool.cc:224] Handling call
0709 12:16:42.748478 (+   585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.749206 (+   728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.769761 (+ 20555us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.699772 23709 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:42.753301 23827 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.760802 12568 rpcz_store.cc:276] 0709 12:16:41.602722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.743313 (+1140591us) service_pool.cc:224] Handling call
0709 12:16:42.759595 (+ 16282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.759945 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.759971 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.760034 (+    63us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:42.760065 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.760121 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.760236 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.746356 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:42.785529 12496 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:42.786728 23778 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:42.787169 23778 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.787575 23778 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:42.784111 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4740) took 1522 ms. Trace:
I20260709 12:16:42.780611 12461 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:42.796337 23847 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.797199 23775 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.797611 23659 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:42.797958 23775 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.798247 23659 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.802223 23645 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:42.800446 23659 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.780491 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1606) took 1009 ms. Trace:
W20260709 12:16:42.805930 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:42.789212 12495 rpcz_store.cc:276] 0709 12:16:41.257165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:41.268240 (+ 11075us) service_pool.cc:224] Handling call
0709 12:16:41.268703 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.052130 (+783427us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b54001c4ba0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:16:42.726978 (+674848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.779342 (+ 52364us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141962496}
I20260709 12:16:42.803082 23645 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.802524 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4576) took 1199 ms. Trace:
I20260709 12:16:42.814292 12564 rpcz_store.cc:276] 0709 12:16:41.602881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.738596 (+1135715us) service_pool.cc:224] Handling call
0709 12:16:42.761569 (+ 22973us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.802169 (+ 40600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.802448 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.809689 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4582) took 1200 ms. Trace:
I20260709 12:16:42.815460 12572 rpcz_store.cc:276] 0709 12:16:41.608796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.696432 (+1087636us) service_pool.cc:224] Handling call
0709 12:16:42.770140 (+ 73708us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.792822 (+ 22682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.792859 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.809240 (+ 16381us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:42.809302 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.809402 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.809577 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.812806 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:42.784441 23709 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:42.800431 23775 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.843830 12643 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:16:42.780431 23333 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:42.808856 23668 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30.
W20260709 12:16:42.851017 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:42.852859 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 225849us
I20260709 12:16:42.846035 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2146) took 1247 ms. Trace:
I20260709 12:16:42.855688 12571 rpcz_store.cc:276] 0709 12:16:41.598468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.634676 (+1036208us) service_pool.cc:224] Handling call
0709 12:16:42.700101 (+ 65425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.829780 (+129679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.829820 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.845748 (+ 15928us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:42.845792 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.845860 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.845975 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.804292 12639 rpcz_store.cc:276] 0709 12:16:41.606152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:41.606266 (+   114us) service_pool.cc:224] Handling call
0709 12:16:41.606548 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:41.803441 (+196893us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:41.803476 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:42.053731 (+250255us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:42.054259 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:16:42.434441 (+380182us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:42.434629 (+   188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.438296 (+  3667us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:42.532826 (+ 94530us) raft_consensus.cc:1685] finished
0709 12:16:42.532849 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.615093 (+ 82244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":217866,"raft.queue_time_us":195544,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":15214,"spinlock_wait_cycles":3119488,"thread_start_us":377675,"threads_started":1,"wal-append.queue_time_us":461295}
I20260709 12:16:42.841069 23165 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.841863 23867 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:42.854271 23756 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:16:42.867122 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156653us
I20260709 12:16:42.841217 23751 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.889752 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:42.897379 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:42.899302 23788 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.900206 23788 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.861085 23827 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.887553 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:42.895007 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:42.894075 12566 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.736s	user 0.015s	sys 0.004s
I20260709 12:16:42.911114 23867 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.893730 23473 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:42.912976 23473 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:42.893986 23605 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:42.915701 23473 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:42.918980 23788 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:42.919888 12571 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:42.920763 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4585) took 1058 ms. Trace:
I20260709 12:16:42.921105 12571 rpcz_store.cc:276] 0709 12:16:41.862508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.870450 (+1007942us) service_pool.cc:224] Handling call
0709 12:16:42.888827 (+ 18377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.920404 (+ 31577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.920670 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.923288 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4685) took 1048 ms. Trace:
I20260709 12:16:42.879231 23756 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:42.925792 12571 rpcz_store.cc:276] 0709 12:16:41.875267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.922020 (+1046753us) service_pool.cc:224] Handling call
0709 12:16:42.922535 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.922901 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.922941 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.922992 (+    51us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:42.923030 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.923093 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.923232 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.925045 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:42.939023 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:42.959504 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:42.962018 12569 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:42.961900 23865 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.964620 23767 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:42.966063 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:42.966356 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4586) took 1036 ms. Trace:
I20260709 12:16:42.967332 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:42.967062 12572 rpcz_store.cc:276] 0709 12:16:41.929603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.922506 (+992903us) service_pool.cc:224] Handling call
0709 12:16:42.923005 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.923394 (+   389us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:16:42.923800 (+   406us) log.cc:844] Serialized 14 byte log entry
0709 12:16:42.965902 (+ 42102us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.965973 (+    71us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:42.966038 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.966114 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.966278 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17631,"prepare.queue_time_us":666,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"thread_start_us":23833,"threads_started":1}
I20260709 12:16:42.971117 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4686) took 1091 ms. Trace:
I20260709 12:16:42.966993 12639 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:42.971741 12566 rpcz_store.cc:276] 0709 12:16:41.879354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.129506 (+250152us) service_pool.cc:224] Handling call
0709 12:16:42.951575 (+822069us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.954630 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2145) took 1358 ms. Trace:
I20260709 12:16:42.980960 23874 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:42.966482 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:42.981670 23847 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:42.982229 23847 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:42.982585 12642 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 34 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:16:42.982700 23847 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:42.981282 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:42.986140 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:42.986797 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:42.989580 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:42.991108 23473 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:42.991669 23473 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:42.992071 23473 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:42.993966 23635 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:42.994700 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4584) took 1304 ms. Trace:
I20260709 12:16:42.995044 23855 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:42.995198 12568 rpcz_store.cc:276] 0709 12:16:41.690767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.826614 (+1135847us) service_pool.cc:224] Handling call
0709 12:16:42.888014 (+ 61400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.971193 (+ 83179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.971233 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.994298 (+ 23065us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:42.994376 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:42.994455 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:42.994627 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.986959 12645 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:42.997437 12568 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
W20260709 12:16:43.032629 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 158372us
I20260709 12:16:42.937088 23756 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:43.007326 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:42.979930 12567 rpcz_store.cc:276] 0709 12:16:41.595934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.051564 (+455630us) service_pool.cc:224] Handling call
0709 12:16:42.941599 (+890035us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:16:43.007025 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60384.2us
I20260709 12:16:43.008670 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4682) took 1266 ms. Trace:
I20260709 12:16:43.042366 12564 rpcz_store.cc:276] 0709 12:16:41.742663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.847711 (+1105048us) service_pool.cc:224] Handling call
0709 12:16:42.889165 (+ 41454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.937581 (+ 48416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:42.937653 (+    72us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:42.951005 (+ 13352us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:42.951051 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.008352 (+ 57301us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:43.008426 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:43.008586 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83218304}
I20260709 12:16:42.882812 23333 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.045222 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4591) took 1021 ms. Trace:
I20260709 12:16:42.994454 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:42.997341 23635 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:16:42.981266 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:42.894614 23165 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.049496 12568 rpcz_store.cc:276] 0709 12:16:42.023791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.996665 (+972874us) service_pool.cc:224] Handling call
0709 12:16:42.997164 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:43.041988 (+ 44824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.045129 (+  3141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:42.963385 23865 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:43.053696 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 273305us
W20260709 12:16:43.050235 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108521us
I20260709 12:16:43.006443 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:43.046687 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:42.950156 23788 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:43.067072 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:43.044456 23772 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.075148 12490 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:16:43.076633 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:43.073367 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79017.8us
I20260709 12:16:43.051661 23652 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:43.079628 12495 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:43.078562 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:43.081314 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121991us
I20260709 12:16:42.981794 23874 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.085505 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.091526 23874 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:43.092298 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.093106 23540 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:43.093560 23540 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:43.093958 23540 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:43.095031 23165 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.097541 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:43.098045 12640 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:43.098459 12643 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:43.099624 12566 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:16:43.085824 23887 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:43.085541 23467 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.788s	user 0.010s	sys 0.011s
I20260709 12:16:43.101944 12496 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:43.086771 23644 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.608s	user 0.011s	sys 0.011s
I20260709 12:16:43.102748 23644 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:43.102317 23887 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.104934 12644 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:43.105257 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.080262 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:43.083225 12489 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:43.077371 23867 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.126391 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.106689 23887 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.122066 12639 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:43.123131 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56424.8us
I20260709 12:16:43.133615 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.127816 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.142616 12536 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:43.143128 12536 leader_election.cc:403] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:43.150353 12646 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:43.080876 23788 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.157546 12489 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:16:43.157995 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:43.154614 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90265.4us
I20260709 12:16:43.122931 23827 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.163407 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.166909 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.167716 12536 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:43.168041 12536 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:16:43.169865 23887 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 31
W20260709 12:16:43.171631 23514 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.832s	user 0.004s	sys 0.019s
I20260709 12:16:43.173724 23514 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.178664 13262 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37.
W20260709 12:16:43.178874 23427 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.890s	user 0.008s	sys 0.012s
I20260709 12:16:43.179781 23514 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.179735 23888 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.112s
I20260709 12:16:43.172889 12490 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:43.173044 23865 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.172425 12642 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:43.175024 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.131842 23668 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:43.182443 23853 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:43.189867 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:43.159459 23333 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:43.198364 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4580) took 1590 ms. Trace:
I20260709 12:16:43.198915 12569 rpcz_store.cc:276] 0709 12:16:41.608245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.647285 (+1039040us) service_pool.cc:224] Handling call
0709 12:16:42.937582 (+290297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.962590 (+ 25008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.198056 (+235466us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54003ac350. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:43.198252 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665919232}
I20260709 12:16:43.125937 23652 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.224988 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:43.207155 12640 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:43.245082 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.221828 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4579) took 1613 ms. Trace:
I20260709 12:16:43.189504 23865 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:43.155929 23867 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.064s
I20260709 12:16:43.192482 23645 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:43.257333 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:43.257345 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:43.261296 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 150805us
I20260709 12:16:43.259068 23775 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:43.280642 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 123005us
I20260709 12:16:43.276911 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:43.276793 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102528us
I20260709 12:16:43.251075 12570 rpcz_store.cc:276] 0709 12:16:41.607973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.645487 (+1037514us) service_pool.cc:224] Handling call
0709 12:16:42.646113 (+   626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.646666 (+   553us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:16:42.855723 (+209057us) log.cc:844] Serialized 14 byte log entry
0709 12:16:43.221362 (+365639us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:43.221428 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:16:43.221499 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.221588 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:43.221757 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26901,"prepare.queue_time_us":220942,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":6673,"raft.queue_time_us":52174,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":13224,"thread_start_us":500091,"threads_started":3,"wal-append.queue_time_us":283653}
I20260709 12:16:43.267479 23751 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.289685 12641 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:43.291241 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:43.293277 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.270798 23668 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:43.298949 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:43.306000 23775 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:43.287853 23264 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.736s	user 0.017s	sys 0.000s
I20260709 12:16:43.259857 23778 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.316540 12490 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.17-32.17]
W20260709 12:16:43.320916 12488 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.859s	user 0.011s	sys 0.006s
I20260709 12:16:43.327446 12488 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
W20260709 12:16:43.322330 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72079.5us
I20260709 12:16:43.312620 12645 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:43.288954 23659 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:43.267799 23853 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.338400 12643 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:43.338636 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.310303 23668 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:43.346724 23473 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:43.340019 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3525) took 1242 ms (client timeout 1529 ms). Trace:
I20260709 12:16:43.348882 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:43.351480 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4589) took 1329 ms. Trace:
I20260709 12:16:43.352481 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:43.355475 12646 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:16:43.359897 12571 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:43.283604 23540 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.368316 12490 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
W20260709 12:16:43.365917 12488 rpcz_store.cc:269] 0709 12:16:42.094228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.147537 (+ 53309us) service_pool.cc:224] Handling call
0709 12:16:43.328299 (+1180762us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:43.374963 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.375525 12639 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: [27.16-28.17]
I20260709 12:16:43.329294 23264 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.366631 12563 rpcz_store.cc:276] 0709 12:16:42.022014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:43.007954 (+985940us) service_pool.cc:224] Handling call
0709 12:16:43.008545 (+   591us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:43.068397 (+ 59852us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:43.068439 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:43.096628 (+ 28189us) log.cc:844] Serialized 26 byte log entry
0709 12:16:43.258455 (+161827us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:43.258504 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.258585 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:43.351187 (+ 92602us) raft_consensus.cc:1685] finished
0709 12:16:43.351214 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:43.351386 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36852,"raft.queue_time_us":54247,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":4129,"thread_start_us":161387,"threads_started":1,"wal-append.queue_time_us":213114}
I20260709 12:16:43.380074 12494 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:43.380836 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:43.386119 12645 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:43.379658 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:43.393970 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.394802 12644 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 21.
I20260709 12:16:43.378894 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.394665 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.366532 23668 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:43.403730 23540 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.370467 23659 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:43.407769 23899 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:43.407853 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.423082 23778 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.425288 23473 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:43.426635 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.380769 23264 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.435998 12641 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:43.378506 23865 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:43.446647 12639 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:43.382897 23751 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:16:43.433133 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.440354 12642 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:43.381100 23772 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.444734 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.444115 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:43.451910 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 99523us
I20260709 12:16:43.454198 13253 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:16:43.454208 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 174300us
I20260709 12:16:43.459168 12643 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:43.440986 23867 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:43.454080 23709 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.597s	user 0.012s	sys 0.004s
W20260709 12:16:43.465701 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 203080us
I20260709 12:16:43.474036 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.474084 23772 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.475749 13255 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:16:43.474083 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:43.478405 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.479346 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.451421 23751 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:43.482779 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.474584 23548 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:43.485001 12490 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:43.486389 23605 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:43.486614 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:43.486984 23605 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:43.487499 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:43.488291 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:43.488216 23605 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.491536 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4587) took 1561 ms. Trace:
I20260709 12:16:43.491976 12571 rpcz_store.cc:276] 0709 12:16:41.930359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.935056 (+1004697us) service_pool.cc:224] Handling call
0709 12:16:42.972576 (+ 37520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:43.373243 (+400667us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.491275 (+118032us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:43.491477 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":890234752}
I20260709 12:16:43.493664 23467 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:43.494246 23467 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:43.498003 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.498275 23827 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.504267 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2148) took 1878 ms. Trace:
I20260709 12:16:43.467777 23788 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:43.518518 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.517104 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:43.521611 12491 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:43.477448 23772 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.548188 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:43.529258 12495 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:16:43.527760 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:43.529448 12641 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:16:43.502920 23751 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:43.557186 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.557186 12489 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:43.558105 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:43.566383 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:43.514504 23652 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.528472 23644 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:43.504869 12565 rpcz_store.cc:276] 0709 12:16:41.625562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.783626 (+1158064us) service_pool.cc:224] Handling call
0709 12:16:42.906360 (+122734us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:42.921982 (+ 15622us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:16:42.922347 (+   365us) log.cc:844] Serialized 14 byte log entry
0709 12:16:43.285840 (+363493us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:43.285906 (+    66us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:43.285958 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:43.286742 (+   784us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:43.504074 (+217332us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b54002477d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:43.504203 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16710,"prepare.queue_time_us":1682,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":79,"raft.queue_time_us":45182,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"spinlock_wait_cycles":121409920,"thread_start_us":338109,"threads_started":2,"wal-append.queue_time_us":281040}
I20260709 12:16:43.483901 23709 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:43.580366 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111500us
W20260709 12:16:43.585320 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63103.2us
I20260709 12:16:43.588430 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:43.589128 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:43.590157 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.529605 23548 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:43.558738 23899 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:43.516232 23827 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.596652 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:43.513528 23867 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.603952 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:43.649317 12643 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 37
W20260709 12:16:43.595059 23875 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.833s	user 0.009s	sys 0.009s
W20260709 12:16:43.656736 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65647.2us
I20260709 12:16:43.572511 23540 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:43.666764 23775 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.668001 23467 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:43.669224 23893 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260709 12:16:43.656224 23548 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.675418 12391 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 22, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:43.677703 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:43.596693 23874 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.071s
I20260709 12:16:43.669057 23721 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.601028 23788 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.695904 23540 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.588119 23645 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:43.697916 12491 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:43.598260 23899 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:43.703424 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100724us
I20260709 12:16:43.704113 12611 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37.
I20260709 12:16:43.723742 12611 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:16:43.705597 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48402.8us
I20260709 12:16:43.702637 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:43.725243 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122072us
I20260709 12:16:43.592296 23855 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:43.709832 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.712038 12566 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:16:43.713897 12490 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:16:43.592857 23644 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.682272 23853 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.669077 23895 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:43.722321 12646 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:43.587893 23659 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:43.723963 23887 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.554s	user 0.015s	sys 0.000s
I20260709 12:16:43.743410 23887 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:43.743760 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.744596 13256 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:43.746311 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.747382 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:43.755234 23874 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.756079 23874 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.740049 23853 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.741333 23548 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.733063 23788 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.731433 23778 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:43.758244 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:43.758912 23644 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:43.760252 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:43.735654 23855 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.760993 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:43.755844 23865 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:16:43.764289 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.791426 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:43.766856 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121105us
I20260709 12:16:43.792297 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:43.733721 23900 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.136s	user 0.001s	sys 0.001s
I20260709 12:16:43.727005 23899 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:43.769284 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:43.757340 12570 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:16:43.802384 12570 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 29.20
I20260709 12:16:43.802757 12570 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:16:43.771916 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87140.3us
I20260709 12:16:43.768326 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.783905 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:43.784917 12572 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:16:43.734004 23885 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:43.737370 23895 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:43.789176 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:43.805302 12565 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:43.789726 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:43.791693 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:43.807559 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69057.4us
I20260709 12:16:43.714457 23467 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:43.800696 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:43.808619 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:43.809501 23894 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:43.809794 23903 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.811883 13259 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:16:43.812338 23772 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:43.813107 23772 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:43.814041 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44839.5us
I20260709 12:16:43.803560 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:43.784886 23756 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.718s	user 0.008s	sys 0.006s
I20260709 12:16:43.859557 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:43.842015 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.842998 12536 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:43.846758 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:43.866048 12536 leader_election.cc:403] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:43.847558 23899 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.850868 23885 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.816838 23895 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:43.826924 23548 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:43.810343 23903 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.856489 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:43.791196 23865 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:43.815804 23902 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:43.776757 23778 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:43.733155 23721 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.924062 23867 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.924854 23867 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:16:43.926290 23721 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.736485 23645 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.928762 23645 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:43.929003 23847 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.946s	user 0.004s	sys 0.010s
I20260709 12:16:43.930583 23540 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.810429 23894 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:43.931367 23894 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:43.931805 23894 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:43.874449 23548 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:43.933857 23548 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:43.874871 13254 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:43.875174 23605 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:43.875291 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.877856 23756 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.886121 23887 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:43.886595 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.861169 23333 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37.
I20260709 12:16:43.950409 13262 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:43.887739 23788 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.888010 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:43.894095 23709 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.896484 23874 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.899011 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:43.814275 23772 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.900315 23865 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:43.901912 23165 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:43.905839 23855 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:43.906360 12637 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:43.972849 12490 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 19. (term mismatch)
I20260709 12:16:43.973316 12490 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
I20260709 12:16:43.973577 12490 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 35.19
I20260709 12:16:43.973819 12490 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:43.974125 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4365) took 1102 ms. Trace:
I20260709 12:16:43.909993 12496 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:43.968777 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:43.910701 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.871567 23264 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:43.906195 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:43.915581 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:43.872231 23853 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:43.920666 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:43.921548 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:43.977514 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101584us
I20260709 12:16:43.979454 13254 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:16:43.922691 23903 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.785763 23473 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260709 12:16:43.981024 23427 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.992126 12496 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:16:43.922806 12391 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 18 to 19, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 19 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:16:43.923913 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55614.4us
I20260709 12:16:43.953846 23652 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:43.994313 23652 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.994731 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:43.997144 23652 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.924240 23697 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.002s
I20260709 12:16:43.873201 12567 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:43.944473 13253 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:43.923295 23827 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.999594 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:43.924180 23778 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:43.965583 13259 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:43.967212 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:43.939942 23885 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:43.970973 23788 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:43.974615 12637 rpcz_store.cc:276] 0709 12:16:42.871559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:42.890465 (+ 18906us) service_pool.cc:224] Handling call
0709 12:16:43.973892 (+1083427us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:43.974028 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1600623360}
I20260709 12:16:43.976997 12492 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:43.981297 12497 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:43.980108 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:43.929469 23847 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:43.913219 23775 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.096s
I20260709 12:16:43.987565 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.011428 23894 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.012394 12493 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:43.991755 23427 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.012437 23847 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:43.923362 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64943.5us
I20260709 12:16:44.013664 12488 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:44.014365 23894 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:43.947737 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:44.004492 13260 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:44.015419 23427 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:44.014176 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:43.999117 12495 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
W20260709 12:16:44.017230 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118304us
I20260709 12:16:44.004985 12391 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 22, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:43.978278 23264 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:44.022683 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55598.1us
I20260709 12:16:44.022591 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.024704 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.017082 23788 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:44.033691 23645 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:44.034345 23645 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:44.034538 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54571.6us
I20260709 12:16:44.034757 23645 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:44.039466 23756 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.017302 23751 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.028396 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:16:44.043754 12497 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:44.044721 12494 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:44.023883 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.018491 23827 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:44.019655 23659 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:44.021239 23721 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:44.041317 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.060990 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:16:44.049445 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:44.004786 23540 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:16:44.064754 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.041709 23264 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:44.062387 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:44.066553 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.067198 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.069353 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:44.074451 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.077840 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.078743 23899 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.060488 23427 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:44.079452 12489 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:44.080765 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.088645 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:44.089790 12640 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:16:44.091619 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69110.9us
I20260709 12:16:44.056309 23885 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:44.091979 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.092638 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.106424 12639 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:44.091733 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.092808 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.093106 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.084146 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:44.094600 23918 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:16:44.094815 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:44.097110 23899 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:44.098773 23788 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.098951 23895 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.130434 23895 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.102124 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:44.131327 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:16:44.131474 23751 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.132381 12569 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:44.102883 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:44.127660 23867 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:44.134186 23867 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:44.134696 23867 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:44.135699 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.136497 12572 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:44.138293 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:44.122721 23825 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:16:44.139436 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46898.7us
I20260709 12:16:44.139737 23825 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:44.140410 23825 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:44.140420 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.110961 12567 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:44.113785 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:44.117000 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54675.3us
I20260709 12:16:44.151396 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:44.122905 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73562.9us
I20260709 12:16:44.149441 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.153658 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:44.126622 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:44.092859 23756 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:44.072013 23827 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:44.091624 23659 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:44.118770 23668 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.676s	user 0.004s	sys 0.010s
I20260709 12:16:44.097538 23915 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:44.164649 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.071954 23875 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.076s
I20260709 12:16:44.146556 23895 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.145618 23644 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:44.193146 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:44.191546 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.145978 23874 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:44.156803 23903 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.209367 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.211057 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:44.190755 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:44.169792 23899 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:16:44.217417 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54833.5us
I20260709 12:16:44.189869 23915 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:44.190286 23721 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:44.197098 23751 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:44.152683 23885 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.050s
I20260709 12:16:44.196856 23644 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:44.229046 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:44.232623 12536 leader_election.cc:403] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:44.200348 23874 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:44.234395 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.236995 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.202023 23756 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:44.178774 23548 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.241922 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.242255 23897 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:44.242888 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.243739 23770 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:44.255882 23928 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:16:44.257827 23467 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:44.258810 23827 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.231223 23697 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:44.260811 23697 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:44.234596 23644 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:16:44.262904 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124736us
I20260709 12:16:44.221010 23788 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:44.218254 23899 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:44.251078 23908 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.337s	user 0.005s	sys 0.000s
I20260709 12:16:44.226068 23894 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:44.201850 23925 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:16:44.254251 23897 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:44.229480 23915 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:44.295426 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.191181 23668 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.146582 23825 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:44.294103 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141737us
I20260709 12:16:44.295102 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.303054 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:44.215929 23895 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.303646 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.239357 23874 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:44.295135 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.306529 12572 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:16:44.302373 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.223410 23865 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.304687 12494 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:44.303717 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.280131 23899 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:44.312747 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.310361 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.263866 23751 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:44.316334 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:44.312012 12497 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:44.323128 12491 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:44.312901 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.239661 23903 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.305794 23887 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:44.328157 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.328951 12570 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:44.313053 23894 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:44.336344 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.313720 23467 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.340039 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:44.337693 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 187729us
I20260709 12:16:44.338181 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.339771 12490 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:44.354121 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.357371 23721 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.355201 12492 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:44.251088 23659 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.337438 23825 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:44.343173 23925 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:44.311439 23645 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.376788 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:44.385399 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 238343us
I20260709 12:16:44.386716 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:44.391728 12569 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:44.393106 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:44.393895 12492 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:16:44.396184 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79955.6us
I20260709 12:16:44.401057 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.404595 23645 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.405748 23751 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.410313 12391 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:44.414783 23903 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.422515 23668 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:44.423522 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107484us
I20260709 12:16:44.423794 23652 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:44.424242 23652 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:44.424590 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:44.421468 23853 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.428030 23887 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:44.430526 23895 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.431932 23897 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.433579 23921 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.434305 23921 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.436758 23921 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:44.430450 12390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 228) took 118632 ms (client timeout 15000 ms). Trace:
I20260709 12:16:44.437796 23929 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:44.438340 23929 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:44.439646 23929 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.440061 12391 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:44.440915 23929 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:44.312340 23697 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.448210 12391 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:16:44.485940 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:44.496449 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:44.497371 12488 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:44.380002 23467 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:44.503718 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.507920 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.519310 23917 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:44.520766 12381 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet a30ed0986fa641bd8f88912b4b9ee320 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 14
I20260709 12:16:44.523087 12536 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:44.524235 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:44.525166 23917 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:44.525895 23917 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:44.526818 23917 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.529035 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.530771 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:44.531873 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.532420 23915 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:44.533175 23931 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.000s	sys 0.002s
I20260709 12:16:44.425210 23652 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:44.438133 12390 rpcz_store.cc:269] 0709 12:14:45.828380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:45.842198 (+ 13818us) service_pool.cc:224] Handling call
0709 12:14:47.123270 (+1281072us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b5400290090. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cf640 00007ffbfb5a6846 00007ffbfb6b090e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:44.430129 (+117306859us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:16:37.450399 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:16:37.451274 (+   875us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:16:37.451345 (+    71us) write_op.cc:435] Acquired schema lock
0709 12:16:37.451367 (+    22us) tablet.cc:662] Decoding operations
0709 12:16:37.451928 (+   561us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:16:37.452024 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:16:37.452055 (+    31us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:16:37.452522 (+   467us) tablet.cc:701] Row locks acquired
0709 12:16:37.452572 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:16:37.452767 (+   195us) write_op.cc:270] Start()
0709 12:16:37.452924 (+   157us) write_op.cc:276] Timestamp: P: 1783599397452679 usec, L: 0
0709 12:16:37.452944 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:16:37.475588 (+ 22644us) log.cc:844] Serialized 929 byte log entry
0709 12:16:38.027462 (+551874us) op_driver.cc:464] REPLICATION: finished
0709 12:16:38.459164 (+431702us) write_op.cc:301] APPLY: starting
0709 12:16:38.459267 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 12:16:38.461208 (+  1941us) tablet.cc:1370] finished BulkCheckPresence
0709 12:16:38.461356 (+   148us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:38.827764 (+366408us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:38.913859 (+ 86095us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:16:38.944131 (+ 30272us) write_op.cc:312] APPLY: finished
0709 12:16:39.033360 (+ 89229us) log.cc:844] Serialized 33 byte log entry
0709 12:16:39.883516 (+850156us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:39.920866 (+ 37350us) write_op.cc:454] Released schema lock
0709 12:16:39.920939 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:39.921462 (+   523us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:39.921497 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":154687,"reported_tablets":51,"spinlock_wait_cycles":6140800,"child_traces":[["op",{"apply.queue_time_us":430155,"apply.run_cpu_time_us":48852,"apply.run_wall_time_us":1462267,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12479,"mutex_wait_us":8526,"num_ops":2,"prepare.queue_time_us":101204,"prepare.run_cpu_time_us":29641,"prepare.run_wall_time_us":110218,"raft.queue_time_us":101846,"raft.run_cpu_time_us":20028,"raft.run_wall_time_us":373910,"replication_time_us":574333,"thread_start_us":1303695,"threads_started":5,"wal-append.queue_time_us":1007354,"wal-append.run_cpu_time_us":91821,"wal-append.run_wall_time_us":2343632}]]}
I20260709 12:16:44.545993 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.485103 23920 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.543469 23930 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:44.541501 23925 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.550467 23853 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:44.557816 12571 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:44.380239 23756 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.568099 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:44.569312 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:44.569849 12569 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:44.571202 23865 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.572973 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.580494 23925 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.583027 23925 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.584049 23548 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.565946 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.566547 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.586485 12570 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 37 because replica has last-logged OpId of term: 35 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 13.
W20260709 12:16:44.586957 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.565148 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.587471 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:44.587837 12563 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:44.588899 12570 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
W20260709 12:16:44.590167 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82333.6us
I20260709 12:16:44.591217 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:44.567481 23652 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:44.593067 23659 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:44.593508 23659 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:44.593907 23659 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:44.498063 23894 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.452029 23778 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.584214 23865 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.591356 23909 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:44.606765 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:44.540235 23934 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.117s	user 0.000s	sys 0.002s
W20260709 12:16:44.604095 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78562.5us
I20260709 12:16:44.604414 12488 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:44.606755 12565 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:16:44.609270 12570 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:16:44.613365 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:44.610818 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 294791us
I20260709 12:16:44.558259 12391 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 23 to 24, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 24 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:16:44.623726 12569 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:44.607282 23909 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:44.628336 23909 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.629148 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.630213 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.630383 12492 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
W20260709 12:16:44.631911 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.636261 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:44.637655 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.639066 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:44.656282 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 269639us
W20260709 12:16:44.662367 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47130.2us
I20260709 12:16:44.634302 23772 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:44.635797 23921 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:44.673977 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65653.5us
I20260709 12:16:44.599615 23867 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:44.687150 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 309674us
I20260709 12:16:44.686658 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.686872 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.645579 23903 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:16:44.710657 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.646404 23847 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:44.688992 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:44.691779 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:44.702553 23467 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.702247 23927 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:16:44.707927 23778 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.731101 13262 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:16:44.596047 23920 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.732788 23917 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:16:44.612824 23473 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.057s
I20260709 12:16:44.734213 13262 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
W20260709 12:16:44.736944 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50171.7us
I20260709 12:16:44.684916 23772 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:44.655423 23775 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:44.655894 23895 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.068s
I20260709 12:16:44.752477 23867 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.775049 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.775468 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.733999 23920 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.740700 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.738574 23875 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:44.742035 23922 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:44.733952 23917 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:44.794514 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.749822 23772 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:44.721029 23847 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:44.763710 23909 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:44.765777 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:44.766672 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81169.3us
I20260709 12:16:44.767570 23770 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:44.770594 23897 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:44.681304 23865 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.789567 23875 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:44.807889 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.680557 23644 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.773108 23938 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.817862 23938 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.818742 23659 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.819808 23644 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:44.824944 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.826120 12645 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:44.795986 23853 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:44.774195 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 33 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.774420 23947 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:44.837356 23947 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.839116 23865 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.775004 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.775434 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.839964 12568 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:16:44.840111 12642 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 26.
I20260709 12:16:44.840893 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.841706 12637 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:44.775779 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.754665 23467 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.022s
I20260709 12:16:44.776553 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.779150 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.791368 13262 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:44.723320 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:44.847083 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 123899us
I20260709 12:16:44.801278 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.801926 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.772342 12567 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260709 12:16:44.810196 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:44.794247 23264 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.678s	user 0.013s	sys 0.000s
I20260709 12:16:44.807912 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:44.848666 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:44.773051 23874 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:44.828822 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63133.9us
I20260709 12:16:44.850912 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:44.828681 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.798666 23917 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:16:44.825999 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.832087 23936 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:16:44.833084 12638 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 37 because replica has last-logged OpId of term: 35 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 13.
I20260709 12:16:44.842926 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:44.800998 23847 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:44.846055 12639 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 34 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:16:44.815611 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:44.845938 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66885.2us
I20260709 12:16:44.854588 23938 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:44.854971 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.858268 23264 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:44.880030 23874 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.881879 23865 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:44.885332 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:44.886417 23927 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:16:44.826644 23915 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.795163 23825 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:16:44.815708 23875 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:44.899401 23922 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:44.914201 23922 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.811480 23867 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:16:44.842712 23778 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.918015 23933 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:44.918452 23778 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.918897 13260 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:44.919418 23933 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:44.919852 23933 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:44.921047 23778 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:44.886271 12497 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:44.880156 23947 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.878898 23540 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:44.794505 23855 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:16:44.878136 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:44.894866 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.881599 23887 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:44.899442 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91620.8us
I20260709 12:16:44.924171 23946 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:44.924628 12609 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:44.925652 12563 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:44.927192 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.821493 23659 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:44.907567 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:44.907791 13256 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:44.904825 23945 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:44.908335 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.908634 23920 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:44.905722 12642 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:44.910171 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:44.911046 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.920764 23933 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.922029 23540 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:44.928231 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:44.928709 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.929999 13260 raft_consensus.cc:606] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:44.941732 13260 raft_consensus.cc:614] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:44.932859 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:44.895678 12494 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:44.934959 12495 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:44.937428 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.943750 12639 raft_consensus.cc:2368] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:44.943570 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.939833 12490 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:44.945405 12570 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:44.951079 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.954869 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:44.940822 23921 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:44.961148 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.942041 23825 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:44.966336 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:44.959882 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:44.963874 12569 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:16:45.017393 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.018663 12569 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 34 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:44.957787 23645 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:44.990979 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105790us
I20260709 12:16:44.948277 23775 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.018s
I20260709 12:16:45.024722 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:44.963140 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:45.030205 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:45.030849 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67768.8us
W20260709 12:16:44.980752 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.948762 23945 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:45.036204 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:44.970239 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:45.011698 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.039150 12638 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 33 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:16:45.012562 12641 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:44.938508 23756 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.038s
I20260709 12:16:44.948549 23920 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:44.966593 23825 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:45.052438 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 105453us
I20260709 12:16:44.963025 23855 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:16:45.034766 12642 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 32 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 19.
I20260709 12:16:45.040220 12566 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.040999 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:16:45.043205 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.008895 23264 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:45.044037 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:44.991758 23952 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:45.008080 23659 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:16:45.051131 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.018702 23778 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:44.992962 23933 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:45.059062 12489 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:16:45.060863 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:45.061755 23959 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:16:45.062860 23945 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.063890 23949 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:45.064630 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.110848 12643 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:16:45.066419 23899 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.739s	user 0.000s	sys 0.014s
I20260709 12:16:44.992111 23938 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.056s
I20260709 12:16:45.068177 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.109723 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.069849 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:45.078678 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53498.9us
W20260709 12:16:45.080376 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120527us
I20260709 12:16:45.042142 23333 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:45.119907 12490 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 34 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:45.107295 12641 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:45.109242 12638 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:16:45.067620 23825 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:45.124554 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.127820 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.130920 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.131793 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:45.133780 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.135665 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.112936 23949 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:45.140928 23949 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:45.146406 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.138236 23333 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:45.105561 23751 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.033s
I20260709 12:16:45.151407 23920 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.152444 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.120532 23899 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.156569 23899 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.157481 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.158128 13259 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:45.022084 23540 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.158159 12493 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:45.164072 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.161026 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.165555 12568 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:16:45.166481 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:45.146678 23917 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.176636 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:45.178010 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.182896 23875 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.184295 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:16:45.185724 23875 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:45.123323 23659 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:45.187842 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.194602 12566 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:45.195159 12494 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:45.196226 23902 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:16:45.196954 23902 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:45.153137 23920 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.139101 23947 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.171183 23952 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.146467 23955 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:45.163796 23775 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:45.201777 12565 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 21. Current term is 22. Ops: []
I20260709 12:16:45.163897 23865 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:45.149137 23333 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.204114 12640 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 27. Current term is 28. Ops: []
I20260709 12:16:45.190857 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:45.207173 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4689) took 1112 ms. Trace:
I20260709 12:16:45.218101 12564 rpcz_store.cc:276] 0709 12:16:44.088128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:44.088370 (+   242us) service_pool.cc:224] Handling call
0709 12:16:45.200762 (+1112392us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:45.200908 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":614698240}
I20260709 12:16:45.218871 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:45.116075 23778 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.205590 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:45.228483 12571 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:45.225981 23865 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.231765 23659 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.198479 23887 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:45.232986 23887 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:45.233479 23887 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.206213 23955 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.234884 23955 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.206339 23947 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.231465 23867 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:45.238583 23867 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.239060 23867 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.240121 12564 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:45.243487 12494 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:45.221186 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:45.262079 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4693) took 1093 ms. Trace:
I20260709 12:16:45.262581 12567 rpcz_store.cc:276] 0709 12:16:44.168775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:44.192879 (+ 24104us) service_pool.cc:224] Handling call
0709 12:16:45.261855 (+1068976us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:45.262015 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326850816}
W20260709 12:16:45.268772 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47667.9us
I20260709 12:16:45.270263 12536 leader_election.cc:400] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:45.270705 12536 leader_election.cc:403] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
W20260709 12:16:45.273974 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.282076 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:45.281436 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 171824us
I20260709 12:16:45.282799 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:45.283679 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.284358 12643 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:45.284758 12566 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:16:45.281916 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.285302 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.285961 12494 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:45.286857 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.287299 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.288367 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.288618 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.286805 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.289595 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.290460 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.292018 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.291339 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:45.293388 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.292421 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.294221 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.295210 12494 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:45.296190 23954 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:45.294795 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.296657 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.297685 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.295269 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.300756 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.199375 23847 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.314626 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.315163 12564 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:16:45.309909 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.297142 23865 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.299396 23968 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.239676 23915 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:16:45.269392 23934 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:16:45.318167 23961 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:45.311733 23875 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.010s
I20260709 12:16:45.325996 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.297545 23467 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:45.207577 23920 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.327795 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.342164 12488 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:16:45.349141 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:45.354723 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.329305 23644 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:16:45.357350 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.353195 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.334951 23915 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:45.367480 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:45.369788 23965 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:45.337819 23949 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.374394 23966 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:45.374908 23971 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:45.375277 23427 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:45.363404 23264 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:45.377233 23847 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.378048 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.419525 12494 raft_consensus.cc:2368] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:45.342162 23467 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.257848 23902 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.382772 23973 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:45.425964 23467 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:45.426522 23902 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.383628 23917 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:45.385756 23825 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.408025 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:45.383332 23920 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:45.413810 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46480us
W20260709 12:16:45.393486 23772 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.000s
I20260709 12:16:45.359992 23938 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:45.421779 23981 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:45.342010 23968 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.431656 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.398890 23915 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:45.434363 12639 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:45.414685 23659 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:16:45.435593 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73053.4us
W20260709 12:16:45.436642 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 136029us
I20260709 12:16:45.476828 23659 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.477509 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:45.479485 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.480408 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.481837 23961 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:45.482272 23961 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:45.482661 23961 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:45.484797 23920 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:45.441793 12494 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:45.447651 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.485574 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.486547 12488 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:45.448011 23825 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:45.488133 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.488336 12641 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 20. (term mismatch)
I20260709 12:16:45.488775 12641 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 19
I20260709 12:16:45.489022 12641 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 31.20
I20260709 12:16:45.489256 12641 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:45.490269 23984 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:45.491166 23984 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:45.477406 23659 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:45.492029 23917 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:45.452786 23947 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:45.433346 23427 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:45.495823 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.459453 23778 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.461143 23968 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:45.462314 23974 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:16:45.498132 23467 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.501398 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.502099 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.502890 12637 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:45.504391 23968 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:45.504988 23968 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.505369 23968 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:45.508250 23954 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:45.432965 23788 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31.
I20260709 12:16:45.440188 23938 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.471575 23949 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:45.473445 23751 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:16:45.474341 23697 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:16:45.439597 12643 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:16:45.496160 23427 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.510237 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.515913 12639 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:45.515705 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.518534 12637 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:16:45.485354 23920 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.525554 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.534157 12568 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:16:45.547202 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.552332 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.552671 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.557334 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:45.558732 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.508889 23954 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.524739 23778 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.413918 23955 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.539263 23938 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:45.560388 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.538306 23697 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:45.551103 23984 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:45.575363 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.555837 23853 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:45.558997 23981 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:45.569478 23939 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.558368 23917 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:45.537909 23751 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:45.549652 23920 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.561334 23467 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.569025 23988 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:16:45.565447 23958 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.498s	user 0.001s	sys 0.000s
I20260709 12:16:45.446373 23772 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.566130 23865 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:16:45.582024 23966 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:45.585732 23847 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:45.587021 23697 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.629519 12641 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:16:45.588083 23939 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.573896 23778 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.575537 23903 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:16:45.592914 23887 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.572219 23954 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.600143 12493 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:45.601809 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.609930 23853 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:45.611312 23751 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.615468 23955 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.622176 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:45.661669 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59919.9us
I20260709 12:16:45.623982 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.588217 23922 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.637053 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.641033 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.598260 23988 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.646903 23921 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:45.651180 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3545) took 1329 ms. Trace:
I20260709 12:16:45.651974 23939 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:45.652225 23778 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:45.671599 23952 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:16:45.658607 12637 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:16:45.659539 12492 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:45.664165 12645 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:16:45.675631 23887 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.676497 23697 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:16:45.681361 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.682502 23751 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:45.682948 23867 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.684670 12610 leader_election.cc:400] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:45.685073 12610 leader_election.cc:403] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:16:45.685544 23867 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.685990 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.686695 12642 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:45.673414 23921 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.686795 23874 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:45.686868 23947 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:45.665668 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.688313 23929 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:16:45.630555 23467 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:45.689924 23467 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.690448 23467 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:45.667419 23865 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:16:45.658682 23984 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.012s
I20260709 12:16:45.669770 23988 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:45.689464 23929 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:45.665241 23853 consensus_queue.cc:1243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 33.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:16:45.669394 12495 rpcz_store.cc:276] 0709 12:16:44.322327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:44.324306 (+  1979us) service_pool.cc:224] Handling call
0709 12:16:44.334242 (+  9936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:44.948168 (+613926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:45.650905 (+702737us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:45.651112 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1380878208}
I20260709 12:16:45.663138 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.695066 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.700917 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.701498 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.703503 12488 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:45.704828 12639 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:45.704741 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.703585 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.706915 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.707535 23965 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:45.707901 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.708461 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.708961 23772 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:45.709312 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.710103 12644 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.710464 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.707564 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:45.707559 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.715123 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.715884 12494 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:45.715167 12490 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:45.718331 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:45.667083 23917 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:45.719316 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.720968 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:45.721666 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.707948 23965 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.726327 23965 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.726682 12646 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 17
I20260709 12:16:45.727067 12646 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:16:45.697418 23955 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:45.727481 12646 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:45.700064 23922 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:45.700755 23929 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:45.711613 23867 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:45.741101 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.714308 23865 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.704523 23847 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:16:45.727101 23925 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:45.744729 23925 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:45.745910 23925 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.717150 23961 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.749076 23961 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.732105 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.736389 23938 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.739640 12610 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:45.746387 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:45.756762 12495 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.728734 12639 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:45.757767 12610 leader_election.cc:403] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:16:45.758195 12490 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
W20260709 12:16:45.767663 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.711006 23902 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.031s
I20260709 12:16:45.727998 23788 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:45.770598 23788 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.771093 23788 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.771768 23938 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:45.772722 12645 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:45.781024 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61790.5us
I20260709 12:16:45.785115 12494 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:45.786672 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.788470 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:45.789398 12571 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:45.792232 12571 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:45.797013 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.801167 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:45.742998 23867 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.808432 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.814975 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.809343 23888 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:45.817773 23540 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:45.818222 23888 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.821070 23888 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.778128 23929 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.823276 23929 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.823820 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.814551 23949 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33.
I20260709 12:16:45.825865 23936 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:16:45.827596 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:45.827685 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:45.808982 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77720.9us
I20260709 12:16:45.809314 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1725) took 1175 ms. Trace:
I20260709 12:16:45.828586 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.807143 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:45.828500 13258 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:45.817158 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.805336 23895 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:45.794804 23961 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:45.828964 12492 rpcz_store.cc:276] 0709 12:16:44.633994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:44.634177 (+   183us) service_pool.cc:224] Handling call
0709 12:16:44.634545 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:45.672670 (+1038125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:45.809048 (+136378us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:45.809246 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2356587904}
I20260709 12:16:45.772742 23909 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.844982 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:45.845711 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.858489 12563 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:45.739673 23955 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.749449 23968 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.759950 23915 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.830976 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:45.848579 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:45.862576 12566 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:45.859571 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:45.862869 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:45.777576 23467 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.816138 23867 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:45.814602 23887 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:45.818260 23894 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:45.741978 23922 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.854537 23952 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:45.779680 23865 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.777576 23659 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.889864 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:45.891539 23659 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.894024 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:45.894691 12641 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:45.896265 23915 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:45.897323 12570 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:45.870863 23917 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:45.902233 12488 raft_consensus.cc:2368] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:45.871879 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.836654 12495 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:45.851296 23888 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:45.881815 23427 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:45.865983 23751 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:45.902554 12641 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
W20260709 12:16:45.907938 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.930032 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.930919 12567 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:45.911938 23902 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:45.912623 23975 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:16:45.913316 12492 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 25. Current term is 26. Ops: []
I20260709 12:16:45.913406 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:45.916029 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.917282 13262 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
W20260709 12:16:45.917650 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:45.902560 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:45.882189 23928 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:45.922827 12644 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:16:45.903158 23887 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:45.870533 23827 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:45.918469 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:45.936666 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:45.937263 12495 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:45.942863 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.956626 12571 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:45.944765 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:45.949488 12567 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:45.859393 23998 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
W20260709 12:16:45.925348 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53569.7us
I20260709 12:16:45.925743 23427 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:45.989562 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159696us
I20260709 12:16:45.867370 23909 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:45.996065 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.960697 13259 raft_consensus.cc:2368] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:45.982103 12563 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:16:45.986205 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:45.959355 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56922.8us
I20260709 12:16:45.961179 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:45.959910 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43985.7us
I20260709 12:16:45.987028 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:46.012224 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.988044 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:45.988086 12568 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 17. (term mismatch)
I20260709 12:16:46.014190 12568 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 16
I20260709 12:16:46.014554 12568 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 28.17
I20260709 12:16:46.014859 12568 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:45.930086 23915 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:45.995435 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:45.998013 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173463us
I20260709 12:16:46.023927 12571 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:16:46.019763 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:45.958719 23887 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:45.902544 23965 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.029439 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:46.034072 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:45.918216 23770 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.070s
I20260709 12:16:45.981202 23825 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:46.035133 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:45.988313 23751 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:46.055380 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:46.056584 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:46.057328 13259 raft_consensus.cc:606] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:46.057848 13259 raft_consensus.cc:614] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:46.056449 23645 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:16:46.060825 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:46.022387 23772 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:46.021899 23827 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:45.987934 23473 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:46.125417 12491 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:46.038457 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:46.124540 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 138443us
I20260709 12:16:46.054003 23888 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:16:46.037607 23659 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:46.035835 23947 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:45.958482 23928 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.131486 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:46.025908 23847 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
W20260709 12:16:46.142864 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 104494us
W20260709 12:16:46.243637 12638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.003s	sys 0.017s
W20260709 12:16:46.244175 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112777us
W20260709 12:16:46.244134 12638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.003s	sys 0.017s
W20260709 12:16:46.148808 23985 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:16:46.131843 23473 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:46.149637 23999 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:16:46.069375 23975 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.088s
I20260709 12:16:46.023363 23961 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.112s
I20260709 12:16:46.132887 23952 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:46.060045 23645 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:46.146782 23902 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:16:46.144152 23895 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:46.250044 23921 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.562s	user 0.013s	sys 0.002s
I20260709 12:16:46.151438 23751 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.162806 23827 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.150888 23825 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:46.150013 24002 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:46.035858 23697 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.107s
I20260709 12:16:46.026690 23897 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.045s
W20260709 12:16:46.148419 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:46.386053 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:46.245127 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:46.235564 23903 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:16:46.236209 23988 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.516s	user 0.016s	sys 0.000s
I20260709 12:16:46.260149 12640 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:16:46.125808 23984 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:46.239920 23514 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:16:46.396988 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151972us
I20260709 12:16:46.399413 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:46.243270 23853 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:46.250306 23473 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:46.237094 23888 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:46.275086 23467 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.270118 23968 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.270341 23955 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.261611 23922 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:46.320376 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:46.422647 24003 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:16:46.423909 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:46.424480 23928 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.424732 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:46.256659 23915 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:46.139798 23772 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.427575 23772 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.429692 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1424) took 1489 ms. Trace:
I20260709 12:16:46.430202 13257 rpcz_store.cc:276] 0709 12:16:44.940420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:44.940604 (+   184us) service_pool.cc:224] Handling call
0709 12:16:44.954949 (+ 14345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:45.006730 (+ 51781us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:45.006772 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:45.656970 (+650198us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b1800382ae0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:16:45.657368 (+   398us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b2c00007be8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:45.657983 (+   615us) log.cc:844] Serialized 26 byte log entry
0709 12:16:46.054692 (+396709us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:46.054733 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:46.054804 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:46.429328 (+374524us) raft_consensus.cc:1685] finished
0709 12:16:46.429352 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:46.429556 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51334,"raft.queue_time_us":51615,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":9316,"spinlock_wait_cycles":18498048,"thread_start_us":396166,"threads_started":1,"wal-append.queue_time_us":619733}
I20260709 12:16:46.424742 12492 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:46.433707 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:46.434015 23888 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.434942 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:46.435072 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:46.435694 23874 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:46.436110 23874 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:46.437134 23874 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.438613 23874 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:46.446138 23697 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:16:46.446717 23697 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:46.446945 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3609) took 1157 ms. Trace:
I20260709 12:16:46.447326 12640 rpcz_store.cc:276] 0709 12:16:45.289748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.326280 (+ 36532us) service_pool.cc:224] Handling call
0709 12:16:45.347345 (+ 21065us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:46.387730 (+1040385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:46.446632 (+ 58902us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:46.446847 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2094318464}
I20260709 12:16:46.413102 23894 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:46.452764 12566 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:46.266515 23895 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.264904 23972 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.120s
I20260709 12:16:46.402202 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:46.418068 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:46.394636 23920 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.792s	user 0.006s	sys 0.023s
I20260709 12:16:46.419036 12490 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:16:46.454828 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:46.273451 23921 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.459292 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:46.467439 12641 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:46.381923 24002 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:46.379639 23952 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:46.401201 23721 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.074s
I20260709 12:16:46.473145 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:46.471912 23930 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.003s
I20260709 12:16:46.476922 12638 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:16:46.478195 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:46.481545 12611 leader_election.cc:400] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:46.396415 23988 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.461563 23467 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:46.462460 23965 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.463651 23865 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.497522 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:46.463145 23909 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.465411 23427 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.505800 13260 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:46.480011 24002 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:46.416474 23645 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.508525 23645 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:16:46.503161 23467 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.514868 12565 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:16:46.465552 23887 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:46.497704 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 461448us
W20260709 12:16:46.517563 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 452913us
I20260709 12:16:46.467465 23920 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.496636 23961 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:46.516731 23929 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.693s	user 0.016s	sys 0.003s
I20260709 12:16:46.541524 12571 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:16:46.447599 23697 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:46.542758 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:46.550405 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:46.539786 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 504683us
I20260709 12:16:46.553005 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:46.491984 23952 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:46.539399 23788 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.768s	user 0.021s	sys 0.000s
I20260709 12:16:46.536523 23909 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:46.589457 12497 raft_consensus.cc:2368] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:16:46.597738 23895 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.544870 23467 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.614037 12641 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
W20260709 12:16:46.590095 23867 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.004s
I20260709 12:16:46.621402 12495 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:16:46.627072 23920 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.589740 23909 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:46.638269 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:46.593708 23929 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:46.640974 12568 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
W20260709 12:16:46.643337 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 404464us
I20260709 12:16:46.644201 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:46.651206 12494 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:46.671413 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:46.617427 23659 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:46.631148 23965 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:46.590000 23788 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.632076 23770 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:46.644064 23917 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.012s	sys 0.028s
W20260709 12:16:46.678833 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57668us
I20260709 12:16:46.587788 23961 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.633425 23697 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:16:46.672611 23920 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20260709 12:16:46.688856 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:46.718405 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:46.627816 23473 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.646899 23514 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:46.716771 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:46.727782 12644 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:46.690806 12494 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:16:46.632652 23867 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.726037 12564 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:46.656136 23915 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:46.726061 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:46.655825 23938 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.062s
W20260709 12:16:46.694124 23899 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.017s	sys 0.006s
W20260709 12:16:46.735937 23899 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.018s	sys 0.006s
W20260709 12:16:46.737494 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81745.5us
I20260709 12:16:46.738801 12488 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:46.678532 23968 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:46.712256 23917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.012s	sys 0.028s
I20260709 12:16:46.730605 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:46.741261 12490 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:46.731439 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:46.734169 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:46.675743 23772 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:16:46.750619 12572 raft_consensus.cc:2368] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:16:46.680681 23965 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.730906 24017 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:46.757395 24009 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:16:46.765596 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:46.712092 23770 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:46.773974 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:46.785724 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:46.786585 12568 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:46.752492 23921 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.796571 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:46.761535 23988 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.754144 23903 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:46.763139 23333 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:46.768767 23894 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:46.748847 23968 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.789582 23902 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:46.801412 23936 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.975s	user 0.005s	sys 0.023s
I20260709 12:16:46.783335 23929 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:46.792712 23659 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:46.738106 23909 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.790561 23925 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:46.822600 23514 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.797484 23887 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.839797 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4411) took 1082 ms. Trace:
I20260709 12:16:46.840427 12639 rpcz_store.cc:276] 0709 12:16:45.757482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.758723 (+  1241us) service_pool.cc:224] Handling call
0709 12:16:45.782235 (+ 23512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:45.810818 (+ 28583us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:45.810850 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:45.830139 (+ 19289us) log.cc:844] Serialized 26 byte log entry
0709 12:16:45.908666 (+ 78527us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b18003c2500. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:16:46.347182 (+438516us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:46.347238 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:46.347318 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:46.835025 (+487707us) raft_consensus.cc:1685] finished
0709 12:16:46.835049 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:46.839689 (+  4640us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52431,"raft.queue_time_us":62036,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":3888,"spinlock_wait_cycles":7611136,"thread_start_us":413679,"threads_started":1,"wal-append.queue_time_us":744740}
W20260709 12:16:46.825453 23936 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.999s	user 0.005s	sys 0.024s
I20260709 12:16:46.835218 23929 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.848515 23929 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.825611 23952 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.782586 24017 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.851711 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:46.852579 12639 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:46.863408 12536 leader_election.cc:400] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29.
I20260709 12:16:46.866881 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:46.868307 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:46.868914 12564 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:46.874511 24034 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:46.875607 24006 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.483s	user 0.000s	sys 0.003s
I20260709 12:16:46.876255 23894 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.878410 23855 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:46.878880 23855 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:46.879823 23855 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.849408 23659 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:46.881079 23855 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:46.881906 23659 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.884207 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:46.885586 24003 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:46.886057 24003 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:46.886404 24003 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:46.890092 23903 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.890827 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:46.829919 23902 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.894887 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:46.898852 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:46.875211 24034 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.873497 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:46.819120 23333 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.902830 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4410) took 1171 ms. Trace:
I20260709 12:16:46.915287 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:46.917412 12639 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:46.918233 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:46.859104 23929 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:46.931265 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:46.896093 23473 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.935940 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:46.937397 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76885us
I20260709 12:16:46.921052 12642 rpcz_store.cc:276] 0709 12:16:45.731168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.731373 (+   205us) service_pool.cc:224] Handling call
0709 12:16:45.755771 (+ 24398us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:45.808188 (+ 52417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:45.808224 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:45.830148 (+ 21924us) log.cc:844] Serialized 52 byte log entry
0709 12:16:46.043031 (+212883us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:46.592699 (+549668us) log.cc:844] Serialized 14 byte log entry
0709 12:16:46.592967 (+   268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:46.593046 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:46.874965 (+281919us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:46.885425 (+ 10460us) raft_consensus.cc:1685] finished
0709 12:16:46.885530 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:46.902739 (+ 17209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58066,"prepare.queue_time_us":82672,"prepare.run_cpu_time_us":3882,"prepare.run_wall_time_us":62913,"raft.queue_time_us":54467,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":28947,"spinlock_wait_cycles":35596544,"thread_start_us":621206,"threads_started":2,"wal-append.queue_time_us":395930}
I20260709 12:16:46.902376 23952 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:46.826841 23915 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 35.14, Last appended by leader: 14, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.902307 23988 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:46.915392 23998 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:16:46.920372 23540 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:46.955855 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:46.929078 24026 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:46.928987 23867 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.920604 24035 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:46.949556 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178880us
I20260709 12:16:46.957595 13259 raft_consensus.cc:606] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:46.984156 13259 raft_consensus.cc:614] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:46.956943 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:46.984956 13262 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:46.988188 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4637) took 1201 ms. Trace:
I20260709 12:16:46.988631 12572 rpcz_store.cc:276] 0709 12:16:45.787292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.800785 (+ 13493us) service_pool.cc:224] Handling call
0709 12:16:46.987922 (+1187137us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54001d6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:46.988111 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2181158144}
I20260709 12:16:46.941144 12639 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:46.991662 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:46.992379 12639 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:46.992772 23984 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:46.993458 23984 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:46.994560 23984 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.18, Last appended by leader: 18, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:46.995910 23984 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:46.993458 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:47.003594 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.009474 24002 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.012126 24002 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.935407 24016 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:16:46.960516 23917 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.015s
W20260709 12:16:47.018077 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4795) took 1334 ms (client timeout 1618 ms). Trace:
W20260709 12:16:47.018982 12497 rpcz_store.cc:269] 0709 12:16:45.683145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.694798 (+ 11653us) service_pool.cc:224] Handling call
0709 12:16:47.017740 (+1322942us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:47.017976 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1980984704}
W20260709 12:16:47.020743 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:47.024278 24031 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:16:47.024602 24020 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:16:47.032792 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:46.971268 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:47.058523 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:47.058887 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:47.059278 12645 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
W20260709 12:16:46.971962 23874 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.003s
I20260709 12:16:47.060962 12642 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 37 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:16:47.062208 12492 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:47.063038 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:47.063786 23721 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:16:47.064277 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:47.020900 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.068109 12497 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:47.069453 23920 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:47.069864 23772 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:47.071738 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:46.973366 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:46.960191 23952 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.076833 23909 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:46.966106 23988 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:46.958437 23961 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.965759 24028 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.078329 24028 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.079079 12643 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:47.079396 23988 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.079926 23920 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:47.077389 23952 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.081394 23825 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:46.976545 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:46.932570 24027 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:46.967743 24026 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:47.087538 24026 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:46.978236 12638 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:16:46.979120 24017 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.980330 23770 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:46.932889 23652 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:16:47.090803 23652 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:47.091289 23652 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:47.091647 13257 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:47.092286 23652 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.093039 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.093827 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4640) took 1305 ms. Trace:
I20260709 12:16:47.094349 12565 rpcz_store.cc:276] 0709 12:16:45.788351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.788594 (+   243us) service_pool.cc:224] Handling call
0709 12:16:45.792991 (+  4397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:46.517563 (+724572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:47.093496 (+575933us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:47.093729 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1659415296}
I20260709 12:16:47.095896 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.096405 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.097258 13257 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:47.093917 23652 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:47.099025 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:47.100646 24028 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.044732 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:47.046612 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:47.103843 24001 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:47.104422 24001 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:47.105275 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:47.106495 23770 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:16:47.108265 23770 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:47.110080 24001 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.048107 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:46.938207 23467 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.111727 23427 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:47.111764 24001 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:47.041074 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.049433 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:47.113286 12490 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:47.053293 12493 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:16:47.112254 23427 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:47.051733 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:47.096619 12565 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:47.037177 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.039168 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:47.120527 12640 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:47.042937 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:47.035099 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:47.101505 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.122439 13258 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
W20260709 12:16:47.122745 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71084.3us
I20260709 12:16:47.123430 23874 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:16:47.123831 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.123970 23874 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:47.124641 23874 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:16:47.124998 24021 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:47.125515 24021 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.125921 24021 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:47.128124 12491 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:47.101994 12496 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:16:47.129976 12610 leader_election.cc:400] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:16:47.131202 24028 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:16:47.112246 23467 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.130268 12610 leader_election.cc:403] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:16:47.135285 23947 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:16:47.135645 23467 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.136440 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:47.137864 23972 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:47.143699 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.053864 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.189706 12569 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:47.117492 23427 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 34.14, Last appended by leader: 14, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.192278 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:47.181499 23874 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.200763 23984 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:47.201508 23984 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:47.201975 23825 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:47.202813 23984 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.150616 23947 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:16:47.195324 23427 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:47.195905 23770 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:16:47.207369 23770 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:47.207813 23770 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:16:47.211103 12637 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:16:47.164089 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56042.3us
I20260709 12:16:47.215219 12609 leader_election.cc:400] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28.
I20260709 12:16:47.219587 23972 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.220414 24017 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:47.220189 23972 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.220938 24017 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.222874 23972 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.203150 23874 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.225312 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.223138 24017 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:47.235277 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.236100 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:16:47.235966 23972 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:16:47.226723 23770 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.240250 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.241334 23909 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:16:47.241921 23909 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:47.196285 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52652.2us
W20260709 12:16:47.243873 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63103.3us
I20260709 12:16:47.244714 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:47.245096 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:47.245452 23949 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:16:47.248366 12643 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:16:47.251034 13256 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:46.938300 23788 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.256995 12637 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:47.257057 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:47.188508 24026 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.255596 24035 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.264597 24035 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:46.942000 23929 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.299455 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:47.300169 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:47.240432 23770 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.306419 13262 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:47.307032 13258 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:47.299553 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.308688 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:47.309509 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.259686 23961 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:47.321049 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 347735us
I20260709 12:16:47.322763 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:47.293810 23827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.006s	sys 0.021s
I20260709 12:16:47.340135 12566 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:47.272569 23909 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:47.339725 23938 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.350039 23853 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.350683 23853 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:47.350237 23938 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.351173 23788 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:16:47.351986 23788 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:47.353034 23938 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:47.353984 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:46.967904 23540 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.290724 23887 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.360286 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:47.301229 23847 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:47.294966 23920 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.390492 12639 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:47.308413 23927 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.254031 23865 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.053s
I20260709 12:16:47.360134 23659 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.296175 23902 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.426020 23659 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.303901 24017 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.428553 23659 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.428786 24017 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:47.422900 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62773.2us
I20260709 12:16:47.423892 23887 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:47.444139 24030 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.506s	user 0.000s	sys 0.000s
I20260709 12:16:47.419034 23770 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.304819 24021 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.448407 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4721) took 1520 ms. Trace:
W20260709 12:16:47.462270 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139609us
I20260709 12:16:47.475441 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:47.469524 12570 rpcz_store.cc:276] 0709 12:16:45.928087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.928525 (+   438us) service_pool.cc:224] Handling call
0709 12:16:45.949441 (+ 20916us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:45.959313 (+  9872us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:16:46.478090 (+518777us) log.cc:844] Serialized 14 byte log entry
0709 12:16:46.794664 (+316574us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:46.813872 (+ 19208us) log.cc:844] Serialized 26 byte log entry
0709 12:16:46.814115 (+   243us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:46.824456 (+ 10341us) log.cc:844] Serialized 14 byte log entry
0709 12:16:46.824695 (+   239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:46.824776 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:47.448130 (+623354us) raft_consensus.cc:1685] finished
0709 12:16:47.448160 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:47.448337 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108065,"prepare.queue_time_us":690765,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":8669,"raft.queue_time_us":25793,"raft.run_cpu_time_us":593,"raft.run_wall_time_us":31004,"thread_start_us":738341,"threads_started":2,"wal-append.queue_time_us":360252}
W20260709 12:16:47.401214 12568 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.608s	user 0.013s	sys 0.002s
W20260709 12:16:47.340905 23827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.006s	sys 0.022s
I20260709 12:16:47.301059 23921 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.419586 23847 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:47.294469 23825 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.115s
I20260709 12:16:47.431267 23540 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.339669 23965 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.447299 23902 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.324678 23947 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:47.391456 23961 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.478420 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:47.501935 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:47.463150 23894 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:47.450518 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 22.
I20260709 12:16:47.481426 24017 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:47.490630 23968 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.478363 23927 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.478261 23920 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.480587 12568 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:47.445595 23887 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.447053 23899 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:47.448853 23548 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:16:47.436252 24003 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.549s	user 0.004s	sys 0.006s
I20260709 12:16:47.450739 23929 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.492663 23952 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:47.455911 24021 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.508603 23847 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:47.508343 24029 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.012s	sys 0.015s
W20260709 12:16:47.551266 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72953.1us
I20260709 12:16:47.567171 12572 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:47.592645 12492 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:16:47.451282 23770 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.466125 23853 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.527580 23751 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.508800 23897 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.060s
I20260709 12:16:47.588320 12570 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:16:47.587534 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4733) took 1631 ms. Trace:
I20260709 12:16:47.510623 23921 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.572394 23827 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:47.592425 24029 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.012s	sys 0.016s
I20260709 12:16:47.594331 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:47.584074 23548 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:47.590632 23952 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:47.602360 23333 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.536114 23902 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.553256 23947 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.596678 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.586808 24003 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.630642 12567 rpcz_store.cc:276] 0709 12:16:45.956026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:45.995135 (+ 39109us) service_pool.cc:224] Handling call
0709 12:16:47.587309 (+1592174us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:47.587465 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3421824896}
I20260709 12:16:47.632501 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.602500 24034 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.638338 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
W20260709 12:16:47.623633 23928 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.006s	sys 0.013s
W20260709 12:16:47.655910 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62155.6us
I20260709 12:16:47.608132 23751 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:47.651636 23928 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.006s	sys 0.013s
I20260709 12:16:47.668696 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.672996 12489 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:47.636837 24026 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.674505 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.679625 12570 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:47.698967 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:47.700237 12497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.062s	sys 0.054s
W20260709 12:16:47.708022 12497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.064s	sys 0.055s
I20260709 12:16:47.699143 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:47.700592 12570 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
W20260709 12:16:47.708187 24037 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.202s	user 0.001s	sys 0.001s
I20260709 12:16:47.650254 23548 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 33.25, Last appended by leader: 25, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.696825 24060 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.715445 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:47.719142 24029 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:47.717800 23887 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.720153 23966 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:47.742172 23548 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:47.719810 24025 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:47.743367 23920 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:47.773581 24049 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260709 12:16:47.774158 24048 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:16:47.710666 23909 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.757472 23887 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:47.775604 24051 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:16:47.776093 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4742) took 1040 ms. Trace:
W20260709 12:16:47.776688 24056 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:16:47.773703 24025 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.778906 23909 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:47.780248 24025 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:47.782147 23652 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.001s
I20260709 12:16:47.783113 12495 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 37 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
W20260709 12:16:47.789714 24037 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:16:47.756631 24064 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.791329 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4734) took 1657 ms. Trace:
I20260709 12:16:47.791793 24064 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.791865 12563 rpcz_store.cc:276] 0709 12:16:46.134046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:46.162030 (+ 27984us) service_pool.cc:224] Handling call
0709 12:16:46.162748 (+   718us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:46.241560 (+ 78812us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:16:46.712796 (+471236us) log.cc:844] Serialized 14 byte log entry
0709 12:16:46.931994 (+219198us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:46.965303 (+ 33309us) log.cc:844] Serialized 26 byte log entry
0709 12:16:46.965486 (+   183us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:46.965520 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:46.965596 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:47.791014 (+825418us) raft_consensus.cc:1685] finished
0709 12:16:47.791046 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:47.791219 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6872,"prepare.queue_time_us":442102,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":7668,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":16154,"thread_start_us":630661,"threads_started":2,"wal-append.queue_time_us":571968}
I20260709 12:16:47.776618 12564 rpcz_store.cc:276] 0709 12:16:46.735666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:46.869477 (+133811us) service_pool.cc:224] Handling call
0709 12:16:46.906121 (+ 36644us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:46.906638 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:46.906682 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:46.922597 (+ 15915us) log.cc:844] Serialized 26 byte log entry
0709 12:16:47.419565 (+496968us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:47.419633 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:47.419715 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:47.775817 (+356102us) raft_consensus.cc:1685] finished
0709 12:16:47.775843 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:47.776019 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":165,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":16271,"thread_start_us":459675,"threads_started":1,"wal-append.queue_time_us":657445}
I20260709 12:16:47.794719 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.795454 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:47.795689 23853 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.796379 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:47.796981 12563 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:47.795670 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:47.799209 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.800316 12564 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:47.777341 23887 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.691867 23952 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.802774 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.803478 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:47.803706 24061 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.804225 24061 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.804718 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:47.805470 23897 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.805888 24017 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:47.806349 24017 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:47.806080 23897 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.806890 24017 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:47.807317 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:47.807672 24059 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:16:47.808926 23827 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:47.811339 23909 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:47.813189 24035 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:47.813726 24035 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:47.738206 24060 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.814265 24035 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:47.817821 23827 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.818589 23827 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.821352 24066 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:47.822130 24066 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.743057 23961 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:47.762862 24042 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.167s	user 0.002s	sys 0.000s
W20260709 12:16:47.766742 24050 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:16:47.801405 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:47.830847 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:47.832424 12563 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:47.832739 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.755764 23770 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:47.772181 24052 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:16:47.765123 23966 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:47.845386 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:47.757795 24029 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:47.842875 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.003s	sys 0.009s
W20260709 12:16:47.845726 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:47.849455 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.004s	sys 0.009s
I20260709 12:16:47.846046 23961 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.841372 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:47.855077 12563 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:47.840579 23902 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.855999 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.856132 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.870472 13259 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:16:47.849776 23887 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:47.848212 23897 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.857571 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.858035 23947 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:47.858765 24066 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.860534 23847 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.862385 24060 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.848121 24021 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.018s
I20260709 12:16:47.866094 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:47.844187 24025 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 12:16:47.852658 23894 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:16:47.846529 24034 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:47.844324 24003 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.871640 12488 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:16:47.873370 13260 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:47.844534 23921 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.878868 12564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:47.871608 12563 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:16:47.881047 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:47.890792 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:47.891360 12563 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:47.892204 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.897055 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.897790 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:47.886940 23897 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:47.904595 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:47.901387 23897 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:47.910395 24057 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:16:47.912218 23847 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.912003 24029 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.25, Last appended by leader: 25, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.915874 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.913544 23697 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:16:47.917227 23897 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:47.917606 23697 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:47.918124 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:47.919963 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.921231 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.922040 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:47.923775 23659 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:47.924293 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.925400 12497 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30.
I20260709 12:16:47.924625 23659 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:47.929347 23899 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:47.943212 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3576) took 1119 ms. Trace:
I20260709 12:16:47.943486 23899 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:47.943701 12495 rpcz_store.cc:276] 0709 12:16:46.823364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:47.046225 (+222861us) service_pool.cc:224] Handling call
0709 12:16:47.942936 (+896711us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:47.943133 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1561468288}
I20260709 12:16:47.925709 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:47.927891 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:47.929180 24067 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:16:47.923278 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.885667 23772 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:47.905475 24046 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.241s	user 0.000s	sys 0.002s
I20260709 12:16:47.935680 23721 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:47.944483 23659 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:47.946281 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:47.946488 23984 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:47.954147 23984 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:47.954551 23984 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:47.959123 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:47.959877 12566 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:16:47.947106 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:47.962461 12570 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:47.964298 12570 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:47.964669 12570 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:47.873312 23961 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:47.937189 24062 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:16:47.931428 23966 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.939656 24034 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:47.966981 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:47.969009 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.983765 12497 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
W20260709 12:16:47.899248 24038 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.286s	user 0.004s	sys 0.000s
I20260709 12:16:47.971365 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.985250 12495 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:47.965957 12570 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:47.979442 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:47.982877 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:47.989537 12570 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:47.988446 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:47.918121 23697 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:48.008858 12489 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:48.011790 12495 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:16:48.014542 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.016258 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.009922 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.019114 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.948820 23721 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:48.028402 23961 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.028002 23721 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.030061 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.031567 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.032289 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:16:48.014397 23855 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.033197 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:47.986689 24034 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:48.072588 23770 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.072870 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:48.073328 23770 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.075719 23770 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.076330 23770 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:48.076738 23770 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:48.077109 23770 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:48.018726 23925 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:48.010290 23847 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:48.036814 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.009428 24064 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:47.944689 23899 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.028007 23825 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.044572 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:48.047555 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2180) took 1163 ms. Trace:
I20260709 12:16:48.055336 23855 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.059561 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:48.060236 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:48.096016 13260 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:48.062839 23927 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:48.063066 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:48.104082 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:48.063575 23514 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.064465 24016 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:48.061517 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:48.035136 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:48.078828 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4749) took 1066 ms. Trace:
W20260709 12:16:48.131906 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154966us
I20260709 12:16:48.133422 12565 rpcz_store.cc:276] 0709 12:16:47.012441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:47.118740 (+106299us) service_pool.cc:224] Handling call
0709 12:16:47.119163 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:47.271608 (+152445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:47.271647 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:47.305588 (+ 33941us) log.cc:844] Serialized 26 byte log entry
0709 12:16:47.629550 (+323962us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:16:47.630324 (+   774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:47.630441 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:48.078485 (+448044us) raft_consensus.cc:1685] finished
0709 12:16:48.078517 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:48.078735 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":273042,"raft.queue_time_us":181428,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":17286,"thread_start_us":323506,"threads_started":1,"wal-append.queue_time_us":420491}
I20260709 12:16:48.134547 23947 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:16:48.135775 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.036628 24063 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:48.138566 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:48.133236 12569 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:48.139791 24015 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:48.140318 24015 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:48.140753 24015 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:48.140905 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.142530 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.143357 12569 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:48.144255 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:48.147565 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86135.5us
I20260709 12:16:48.148049 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.148926 12639 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:47.998857 23659 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:48.079289 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:48.079282 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.151379 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:48.099541 23825 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.100271 23855 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:48.101979 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:48.084636 24077 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:16:48.137264 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.104156 12572 rpcz_store.cc:276] 0709 12:16:46.884561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:47.065025 (+180464us) service_pool.cc:224] Handling call
0709 12:16:47.566811 (+501786us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:47.592791 (+ 25980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:48.047244 (+454453us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54001a22d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:48.047463 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1153154816}
I20260709 12:16:48.113157 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:48.158576 12646 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:48.113730 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.151777 12493 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:48.086177 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4745) took 1241 ms. Trace:
I20260709 12:16:48.159552 12641 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:48.157591 12565 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:16:48.163503 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.164484 12644 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:16:48.148375 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:48.091179 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:48.123010 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:48.124605 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:48.159875 12571 rpcz_store.cc:276] 0709 12:16:46.844750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:46.917032 (+ 72282us) service_pool.cc:224] Handling call
0709 12:16:46.917485 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:47.265940 (+348455us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:16:47.271050 (+  5110us) log.cc:844] Serialized 14 byte log entry
0709 12:16:47.455932 (+184882us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b18003aaec0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:16:47.461349 (+  5417us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b18003aaec0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf96c758f 00007ffbf2706571 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd
0709 12:16:47.683133 (+221784us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:47.688645 (+  5512us) log.cc:844] Serialized 26 byte log entry
0709 12:16:47.688827 (+   182us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:47.688885 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:48.085678 (+396793us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:48.085758 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:48.085835 (+    77us) raft_consensus.cc:1685] finished
0709 12:16:48.085849 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:48.086071 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58612,"prepare.queue_time_us":24576,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":8793,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":38208,"spinlock_wait_cycles":879390976,"thread_start_us":203422,"threads_started":1,"wal-append.queue_time_us":543892}
I20260709 12:16:48.145394 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.175921 12569 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:48.172703 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81641.5us
I20260709 12:16:48.177162 12536 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30.
I20260709 12:16:48.177759 12536 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:48.180038 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:48.182061 24026 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:48.182654 24026 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:48.183013 24026 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:48.186079 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:48.186466 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.187126 12497 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:48.188199 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.146073 23514 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.191154 23514 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.084393 23847 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.192700 23847 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:48.195327 12646 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:48.132810 23467 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.216591 12564 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:16:48.216619 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:48.217315 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66029.4us
I20260709 12:16:48.188925 12497 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:16:48.195142 13260 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:48.220818 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:48.222520 13258 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:16:48.157517 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:48.222179 23874 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:48.226109 12637 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:48.226857 12640 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:16:48.227452 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.131850 24016 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:48.156924 23659 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:48.155879 24063 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:48.229341 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.126483 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:48.179131 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:48.229288 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.249142 12646 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:48.153825 23333 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.229398 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:48.214620 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.167980 24075 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:48.214493 23894 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:16:48.102820 23984 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.250159 24066 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30.
I20260709 12:16:48.268532 24060 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:48.270202 24063 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.270341 23333 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.303340 23333 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.271446 23952 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:48.303983 23927 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:48.304306 23952 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.305064 23952 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:48.306321 23825 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.237396 23659 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:48.228916 23467 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.274221 23827 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.275941 24061 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.277808 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:48.340360 13260 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:16:48.280122 24043 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:16:48.280380 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.238951 23514 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:16:48.250474 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71470.7us
I20260709 12:16:48.222342 23853 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:48.294584 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.323856 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:48.334774 23548 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:16:48.338574 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:48.296036 23921 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:48.343070 12570 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 18. (term mismatch)
I20260709 12:16:48.344830 12637 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:48.346369 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:48.347867 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:48.329368 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:48.373543 24048 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:16:48.350189 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:48.269150 23697 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.314002 24016 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.290796 24075 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:48.304723 23927 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:48.378060 12491 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:48.379213 12495 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:48.322400 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:48.382268 12492 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:16:48.384966 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:48.359405 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 140929us
I20260709 12:16:48.315534 24035 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.385800 12570 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 17
I20260709 12:16:48.420408 12570 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:16:48.420804 12570 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:48.387210 12646 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:48.390479 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.387486 12493 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
W20260709 12:16:48.386276 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47785.5us
I20260709 12:16:48.338114 24060 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:48.339450 24034 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.402652 12645 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:48.374580 23514 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:48.409335 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.377410 23925 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:48.415032 23927 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:48.390033 24080 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:16:48.390273 23961 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:16:48.415962 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:48.449620 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:48.416549 23917 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:48.412111 24075 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:48.426299 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.401580 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:16:48.444624 12645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:60230
I20260709 12:16:48.460601 12645 raft_consensus.cc:606] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:48.460979 12645 raft_consensus.cc:614] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:48.417495 24016 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:48.458204 12567 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:48.385711 23853 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.450798 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:48.468458 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146154us
I20260709 12:16:48.427546 23902 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:48.442166 23514 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:48.405735 23903 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:16:48.454401 23540 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.454391 24061 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:48.498443 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.492045 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:48.457050 23827 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:48.406574 23921 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.454703 23917 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.431173 23825 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:16:48.497061 12641 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
W20260709 12:16:48.487493 24017 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.680s	user 0.018s	sys 0.005s
I20260709 12:16:48.501022 12646 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:16:48.463779 24080 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:48.427037 24039 consensus_queue.cc:1175] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 33 status { last_received { term: 33 index: 24 } last_committed_idx: 24 last_received_current_leader { term: 33 index: 24 } }
I20260709 12:16:48.506222 12492 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:16:48.484395 23915 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:48.510154 12493 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 13. Preceding OpId from leader: term: 35 index: 14. (index mismatch)
W20260709 12:16:48.502271 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 273657us
I20260709 12:16:48.526036 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:48.526700 12570 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:48.530759 23770 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.501837 23827 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.533771 23827 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:48.533963 12639 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:48.525554 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.512629 23917 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:48.540555 12645 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:48.543072 12489 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:48.543437 12572 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:48.488415 23902 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.514653 24061 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.523737 23915 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:48.553848 12570 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:48.557597 23921 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.560300 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.489550 24060 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.523584 24094 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:48.503970 23540 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.544339 23770 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:48.513669 24017 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.521708 23874 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:16:48.584774 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:48.588250 13258 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:48.563259 23915 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:48.562585 23984 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.594731 12645 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:16:48.522343 23925 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.575124 23903 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.596990 13262 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:48.575497 24094 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:48.601536 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80266us
I20260709 12:16:48.562945 24061 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.604382 23548 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:16:48.604548 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.605247 12570 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:16:48.605566 24061 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:48.606248 24061 consensus_queue.cc:1175] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 34 status { last_received { term: 34 index: 17 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.17-33.17]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:16:48.585258 23721 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:16:48.615756 23721 raft_consensus.cc:993] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:48.616248 23721 raft_consensus.cc:1081] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:48.598054 23751 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.626294 23751 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.626873 23751 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.587330 24060 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:48.630611 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.633833 24015 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.637125 24015 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.637806 23540 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:48.638980 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:16:48.605765 12570 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:48.646205 12644 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:48.642211 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.674418 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.592490 23467 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:48.683864 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 152313us
I20260709 12:16:48.692932 12565 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:16:48.693797 12565 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:48.694249 12565 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:48.701989 24034 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.703444 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1648) took 1357 ms. Trace:
I20260709 12:16:48.703931 12643 rpcz_store.cc:276] 0709 12:16:47.359319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:47.372964 (+ 13645us) service_pool.cc:224] Handling call
0709 12:16:47.407081 (+ 34117us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:48.378604 (+971523us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:16:48.378642 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:48.378686 (+    44us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:48.378724 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:48.703158 (+324434us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:48.703221 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:48.703377 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":535446,"raft.queue_time_us":590094,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":80150912,"thread_start_us":400620,"threads_started":1}
I20260709 12:16:48.585222 23921 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:16:48.709232 23853 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.709456 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:48.710482 12570 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:16:48.714177 23909 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:16:48.715420 24017 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.716285 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:48.716305 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:48.689498 24026 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.506s	user 0.005s	sys 0.007s
I20260709 12:16:48.641010 23902 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:48.690845 24096 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.002s	sys 0.002s
W20260709 12:16:48.691007 24087 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.000s	sys 0.010s
I20260709 12:16:48.685081 23751 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:48.690433 24079 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.113s	user 0.004s	sys 0.000s
W20260709 12:16:48.723483 24092 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.001s	sys 0.003s
I20260709 12:16:48.686556 24094 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:48.758438 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.759532 24001 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:48.753170 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.690413 24035 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.754015 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:48.757068 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:48.749452 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 175956us
I20260709 12:16:48.729203 23772 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:48.776208 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.684489 23903 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.780041 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.777107 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:48.780828 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:48.782692 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:48.783485 12643 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:48.782269 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:48.760039 24001 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:48.774144 23847 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.581s	user 0.013s	sys 0.000s
I20260709 12:16:48.641533 24080 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.748162 23659 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.747447 24026 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.775290 23751 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:16:48.788990 24035 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.789858 24063 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.828382 12497 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:48.816045 12565 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.820379 24084 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.820644 24089 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:16:48.823493 23903 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.844002 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.842677 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.786443 23772 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.852358 12565 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:48.855122 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:48.820695 24101 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.856827 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.806136 23847 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.862428 12491 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:48.833767 23921 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:48.840243 23902 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:48.875051 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:48.828182 24035 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.854022 23751 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:48.834070 23920 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:16:48.845116 24084 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.858927 24089 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:48.872396 23921 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.896282 12491 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:48.835839 24063 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:48.880009 23952 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.562s	user 0.008s	sys 0.006s
I20260709 12:16:48.905913 24103 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.895500 24089 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.906693 24103 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.907418 23772 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.909219 24103 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.908989 23751 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.910063 24091 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.910866 24091 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.912807 24107 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.913486 24107 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.915376 24035 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:48.919328 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4758) took 1027 ms. Trace:
I20260709 12:16:48.876430 23902 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.919837 12568 rpcz_store.cc:276] 0709 12:16:47.892295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:47.964351 (+ 72056us) service_pool.cc:224] Handling call
0709 12:16:47.964750 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:47.970317 (+  5567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:47.970345 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:47.980415 (+ 10070us) log.cc:844] Serialized 26 byte log entry
0709 12:16:48.331517 (+351102us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:48.331578 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:48.331694 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:48.919042 (+587348us) raft_consensus.cc:1685] finished
0709 12:16:48.919070 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:48.919248 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":304494,"threads_started":1,"wal-append.queue_time_us":819499}
I20260709 12:16:48.922173 23902 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.929877 12492 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:16:48.868355 24101 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:48.932848 24095 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:16:48.935743 12488 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
W20260709 12:16:48.898327 24090 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:16:48.939343 24107 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:48.949239 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 176107us
I20260709 12:16:48.886054 24109 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:48.874456 23917 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:48.856827 24001 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:48.976951 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:48.933319 24101 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.988432 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:48.995990 24026 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:48.996722 24105 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:16:48.903538 23952 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.000094 23952 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.003573 24101 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:49.008369 23847 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.009793 23659 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.010802 23921 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:48.919899 24075 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.011605 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.001296 12565 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 34 index: 26. (index mismatch)
I20260709 12:16:49.019649 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.036357 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.037918 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.038681 12563 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:49.038559 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:49.018810 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:16:49.040405 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.041225 12572 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:16:49.012445 23927 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.001s
I20260709 12:16:49.047112 12490 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:49.050563 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.050318 23927 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.051597 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.052397 12565 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:49.052726 24084 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.053462 23697 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:49.001394 24091 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.054642 23847 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:49.055200 23847 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:49.036463 23514 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.003s
I20260709 12:16:49.053534 24101 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:49.057610 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.061877 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.063890 23847 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:49.066970 24113 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:16:49.053231 23927 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:49.075987 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.078130 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.079001 12569 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:16:48.986658 24109 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.083635 24109 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:49.084921 24097 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.483s	user 0.003s	sys 0.000s
I20260709 12:16:49.086958 24060 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:49.087641 24060 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.089437 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.090721 23917 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.090975 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.091909 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:49.095191 12563 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:49.097484 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.097666 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:16:49.100251 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.101186 13258 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:16:49.091867 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.102506 12491 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 39 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 24.
I20260709 12:16:49.103726 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.105568 13258 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 22: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:49.101470 12563 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:49.107762 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.109429 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.110512 12563 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:49.128348 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.128957 12572 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:49.133168 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.134256 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:49.140545 13256 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:49.134636 24001 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:49.138525 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.135756 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3589) took 1126 ms. Trace:
I20260709 12:16:49.142840 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.071964 23514 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.144434 12494 rpcz_store.cc:276] 0709 12:16:48.009149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:48.009603 (+   454us) service_pool.cc:224] Handling call
0709 12:16:49.135481 (+1125878us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:49.135687 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1694521728}
I20260709 12:16:49.144682 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:16:49.106518 23827 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.572s	user 0.015s	sys 0.000s
I20260709 12:16:49.137848 24109 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:49.103427 23847 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:49.102382 23917 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:49.142169 12563 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:49.158022 23847 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:49.164201 23847 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:49.166883 13258 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 34 index: 26. (index mismatch)
I20260709 12:16:49.151374 24060 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.161027 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.209112 12563 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:49.210649 12563 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:49.213177 12572 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:16:49.214685 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:49.215164 12569 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:49.216477 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:49.156718 24109 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:49.224972 12569 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:16:49.154121 23827 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.204586 24088 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.234985 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.239972 12569 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:16:49.219658 23972 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.222424 23927 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:16:49.224493 24109 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:49.231895 24028 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:49.168699 23917 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.217314 23925 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.026s
I20260709 12:16:49.235731 24063 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.167285 24101 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:49.282542 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:49.346016 23514 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.250073 23927 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:49.243503 24088 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.354432 13256 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:49.332607 24075 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:49.247357 23972 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.335920 24028 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:49.332031 24094 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:49.382323 24124 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260709 12:16:49.403934 24082 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.785s	user 0.002s	sys 0.009s
W20260709 12:16:49.382373 24122 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:16:49.414615 24089 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:49.415283 24089 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:49.416857 24016 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.937s	user 0.010s	sys 0.004s
W20260709 12:16:49.417984 23915 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.818s	user 0.008s	sys 0.007s
I20260709 12:16:49.418736 23917 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.418493 23915 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:49.438568 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51229.8us
I20260709 12:16:49.442137 12564 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:49.443638 24089 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:49.444620 24102 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.653s	user 0.006s	sys 0.002s
I20260709 12:16:49.449787 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1434) took 1633 ms. Trace:
I20260709 12:16:49.450060 23827 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.450490 13262 rpcz_store.cc:276] 0709 12:16:47.815986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:47.824441 (+  8455us) service_pool.cc:224] Handling call
0709 12:16:47.824852 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:48.601178 (+776326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:49.449425 (+848247us) spinlock_profiling.cc:243] Waited 772 ms on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:49.449697 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1775193472}
I20260709 12:16:49.405330 23927 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.456535 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:16:49.457963 24028 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.460140 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.461272 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.462497 24089 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:49.462636 23915 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.464030 23827 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:16:49.465016 24088 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:49.460347 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3628) took 1243 ms (client timeout 1620 ms). Trace:
I20260709 12:16:49.466092 12536 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:49.466605 12536 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:49.467267 23972 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.468580 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2194) took 1432 ms. Trace:
I20260709 12:16:49.469003 12571 rpcz_store.cc:276] 0709 12:16:48.036734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:48.210108 (+173374us) service_pool.cc:224] Handling call
0709 12:16:48.210673 (+   565us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:49.217342 (+1006669us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:49.217391 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:49.217453 (+    62us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:49.217499 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:49.468351 (+250852us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:49.468394 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:49.468519 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2314380544}
I20260709 12:16:49.470130 24094 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:49.471974 24119 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.233s	user 0.002s	sys 0.000s
W20260709 12:16:49.466157 12644 rpcz_store.cc:269] 0709 12:16:48.216887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:48.228870 (+ 11983us) service_pool.cc:224] Handling call
0709 12:16:49.460153 (+1231283us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540018ced4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:49.460290 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":915533184}
I20260709 12:16:49.474308 23909 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:49.474953 23909 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.476670 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1658) took 1352 ms. Trace:
I20260709 12:16:49.477234 23909 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.477087 13259 rpcz_store.cc:276] 0709 12:16:48.123999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:48.124208 (+   209us) service_pool.cc:224] Handling call
0709 12:16:49.476418 (+1352210us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:49.476561 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2151268096}
I20260709 12:16:49.481879 23903 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:49.482424 23903 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:49.482756 23903 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:49.484375 24003 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:49.484934 24003 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.486487 23899 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:49.487447 24003 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.488101 24003 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.488560 24003 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.494481 24063 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.495182 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:49.496217 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:49.496992 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.497797 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.497812 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:16:49.498481 12496 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:49.492430 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.500003 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1656) took 1021 ms. Trace:
I20260709 12:16:49.500350 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.502323 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:49.380666 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:49.500411 12637 rpcz_store.cc:276] 0709 12:16:48.479036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:48.479264 (+   228us) service_pool.cc:224] Handling call
0709 12:16:48.479755 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:48.480047 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:16:48.504548 (+ 24501us) log.cc:844] Serialized 14 byte log entry
0709 12:16:49.499406 (+994858us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b180022f3c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:16:49.499595 (+   189us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:49.499656 (+    61us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:49.499709 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:49.499783 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:49.499927 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5219,"prepare.queue_time_us":26783,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"spinlock_wait_cycles":7068032}
I20260709 12:16:49.505916 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.506464 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.498096 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.505786 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.507761 12489 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:49.514789 12494 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:49.515703 12491 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:49.510097 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 121048us
I20260709 12:16:49.514235 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.519531 12571 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:16:49.461148 23920 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:49.520634 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:49.495493 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.528s (SENT) [suppressed 1 similar messages]
I20260709 12:16:49.521687 13259 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:16:49.520466 23652 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:49.529126 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.529078 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.529009 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:49.494469 24060 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:49.534829 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:49.536202 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:49.532977 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.541674 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.552227 12570 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:49.552278 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 22.
I20260709 12:16:49.554716 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:49.508327 23894 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:49.508677 23827 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.537058 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:49.547356 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:49.566162 12569 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:49.554949 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:49.568229 12568 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:49.569360 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.556311 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.570008 12568 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:49.538419 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:49.521692 24075 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:49.571609 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.569912 12460 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:16:49.568454 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.575700 12460 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20260709 12:16:49.567418 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.578452 12569 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:49.560076 23473 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.583433 12566 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:16:49.583076 23473 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.555655 23929 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:49.585515 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:49.586189 23929 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.589926 23920 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:49.591308 23920 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.546149 23652 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:49.594633 23652 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:49.597841 24075 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:49.599334 24075 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.606071 24085 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:49.607148 24085 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.610029 23473 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:49.611116 23540 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:49.611912 23540 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:49.615425 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 818619us
I20260709 12:16:49.616282 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:49.570609 23894 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:49.617115 23894 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:49.617813 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:49.619309 13260 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:49.619992 23874 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:16:49.620694 23874 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:49.621809 23874 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.624547 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.625765 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.626057 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.626328 23855 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:49.627022 23855 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.627393 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:49.626650 12497 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:49.628559 24080 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.629941 23825 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:49.491663 23903 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.630520 23825 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.629590 24080 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:49.633030 23825 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:49.634999 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:49.635797 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 97440.2us
I20260709 12:16:49.545389 23972 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:16:49.546854 24081 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.495187 24063 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.589247 23929 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:49.636976 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:49.643512 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.671766 12564 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30.
I20260709 12:16:49.648754 12496 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:16:49.641363 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:49.649281 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79749.4us
W20260709 12:16:49.650741 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65376.2us
I20260709 12:16:49.663216 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:49.676522 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:49.575532 24084 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:49.678848 12567 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:49.694232 13256 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:49.698411 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:49.699232 12645 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:49.700846 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:49.694881 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:49.702111 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:49.611828 23770 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.697103 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.705452 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.706312 12492 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:49.712971 12496 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
W20260709 12:16:49.642540 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67214.8us
I20260709 12:16:49.705790 12566 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:16:49.730268 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:49.727275 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:49.715507 12493 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:16:49.649076 24113 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:49.747273 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:49.755609 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:49.755043 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:49.746224 24094 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:49.758275 24094 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:49.571650 24104 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.128s
I20260709 12:16:49.758716 24094 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:49.715927 24110 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.043s
I20260709 12:16:49.670404 24040 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:49.634239 24080 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:49.749297 23770 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.756073 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:49.699177 24060 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:16:49.581645 23827 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.735433 12570 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 34 index: 17. (term mismatch)
I20260709 12:16:49.833853 12570 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 16
W20260709 12:16:49.762476 23847 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.598s	user 0.013s	sys 0.000s
I20260709 12:16:49.670326 24103 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:16:49.843672 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:49.855461 12641 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:49.856547 12642 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:16:49.857275 24085 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:49.857349 24003 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.698957 24091 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:49.864332 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:16:49.832824 23825 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:49.806017 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50033us
I20260709 12:16:49.882385 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:49.685736 24081 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:49.845854 24040 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:49.786561 12645 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:49.746711 24029 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:49.834175 12570 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 33.17
I20260709 12:16:49.580488 23966 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.157s
I20260709 12:16:49.834342 12646 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:16:49.835493 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.947201 12638 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:49.837641 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:49.839083 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:49.746778 23855 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:49.862295 24091 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:49.844511 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:49.982831 13256 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30.
I20260709 12:16:49.984035 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:49.803579 23887 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.077s
I20260709 12:16:49.713079 24112 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.135s
W20260709 12:16:49.832643 24083 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:16:49.801681 24060 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:49.849123 23952 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:49.851347 23827 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:49.873445 13259 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:49.845176 24080 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:49.767443 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:49.911990 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:49.920732 12570 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:49.888715 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.007061 24091 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:49.694469 23929 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:49.959321 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:49.962967 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 467482us
W20260709 12:16:50.047777 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 280134us
I20260709 12:16:49.912793 13262 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:49.980296 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:49.847270 12492 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:50.050917 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:49.859190 23847 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:50.007810 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.011s	sys 0.016s
I20260709 12:16:50.015977 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:50.016935 12640 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:49.921295 24085 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:50.064498 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.012s	sys 0.016s
I20260709 12:16:50.065002 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:49.947844 24029 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:49.945058 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.024456 24060 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:16:50.030192 23540 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.011873 24016 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.057s
I20260709 12:16:50.050887 23927 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:50.053761 12644 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:50.055143 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.073378 24107 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:49.885533 23825 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.083768 24039 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:50.079308 24029 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:50.090005 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:50.120563 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 22.
I20260709 12:16:50.091289 12566 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:16:50.075595 23909 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.027340 23952 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.079630 24103 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:16:50.129997 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 488667us
W20260709 12:16:50.101426 23652 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.004s
I20260709 12:16:50.102303 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.134393 12645 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:50.134297 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:50.104437 24109 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.852s	user 0.012s	sys 0.000s
I20260709 12:16:49.981393 24040 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.115149 23927 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:50.046689 23827 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:50.153720 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:50.113320 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.154450 12492 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:50.156445 23770 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.151489 23652 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.133041 23909 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.163229 24081 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.164016 23909 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.133889 24080 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:50.165938 24080 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:50.166849 24080 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.168011 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.155443 23927 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:50.170771 23825 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.170560 23827 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.172151 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:50.082617 23899 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.596s	user 0.013s	sys 0.000s
I20260709 12:16:50.091317 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:50.174568 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:50.123101 24015 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.009s	sys 0.019s
I20260709 12:16:50.078814 24085 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:50.126693 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4787) took 1032 ms. Trace:
I20260709 12:16:50.100072 13259 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:16:50.135494 12644 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:50.138608 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:50.151170 23874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.007s	sys 0.025s
I20260709 12:16:50.080331 24003 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.053s
I20260709 12:16:50.116261 12488 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
W20260709 12:16:50.175633 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84420.2us
I20260709 12:16:50.068873 23929 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.190053 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:50.176403 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:50.179314 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.040201 23972 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:16:50.182536 13258 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:16:50.168048 24080 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:50.181584 12563 rpcz_store.cc:276] 0709 12:16:49.093913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.219233 (+125320us) service_pool.cc:224] Handling call
0709 12:16:49.220966 (+  1733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:49.221136 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:16:49.648842 (+427706us) log.cc:844] Serialized 14 byte log entry
0709 12:16:50.126300 (+477458us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:50.126362 (+    62us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:50.126414 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.126486 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:50.126626 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49600,"prepare.queue_time_us":121503,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":64769,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":17615,"thread_start_us":736886,"threads_started":2}
I20260709 12:16:50.191668 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.161403 23652 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.173451 23827 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:50.193186 23917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.012s	sys 0.013s
W20260709 12:16:50.185829 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 442757us
I20260709 12:16:50.181262 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:50.189796 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:50.214211 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:50.180207 24015 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.009s	sys 0.019s
I20260709 12:16:50.197710 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.189204 23909 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:16:50.188915 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
W20260709 12:16:50.196991 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:50.216926 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:50.175047 24063 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.178330 23825 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.178414 23770 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:16:50.184688 23874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.007s	sys 0.026s
I20260709 12:16:50.183147 24103 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:50.194303 12611 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30.
I20260709 12:16:50.182245 24085 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:50.154366 24109 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.220461 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.198210 24028 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.186527 23966 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:16:50.221966 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:50.187443 24104 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.030s
I20260709 12:16:50.230103 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.214628 23652 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.220913 23972 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:16:50.275182 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.020s	sys 0.016s
I20260709 12:16:50.191381 24112 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
W20260709 12:16:50.284890 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.022s	sys 0.017s
W20260709 12:16:50.285507 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63618.1us
I20260709 12:16:50.282886 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:50.231896 12611 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:16:50.237725 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.290696 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.291386 12645 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:50.229267 23984 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:50.221061 23605 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:50.217024 23855 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:16:50.217469 23909 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:50.248805 23659 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:16:50.250787 24133 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260709 12:16:50.254644 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:50.258836 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.312911 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:50.228746 23917 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:16:50.197909 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 522193us
I20260709 12:16:50.229118 24063 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:50.293653 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.019s	sys 0.023s
W20260709 12:16:50.322289 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.020s	sys 0.023s
I20260709 12:16:50.231897 23847 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.295811 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:50.301745 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.292342 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:50.233974 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.008s	sys 0.020s
W20260709 12:16:50.294121 24094 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.535s	user 0.013s	sys 0.000s
I20260709 12:16:50.232920 23825 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.240864 24028 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.329980 12645 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:50.403754 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.331197 12491 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:16:50.292373 23952 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.299507 23605 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:50.384032 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.008s	sys 0.028s
I20260709 12:16:50.335728 12537 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:16:50.314340 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.239523 24103 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.220949 24091 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.308519 23909 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:50.277268 23652 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.420810 12537 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:16:50.420311 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:50.420737 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 744272us
W20260709 12:16:50.487490 23894 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.870s	user 0.006s	sys 0.009s
I20260709 12:16:50.424795 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.490160 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.490439 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:50.430231 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 134528us
I20260709 12:16:50.479156 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.491891 12641 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:16:50.280722 24029 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.308198 23659 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:50.499330 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 305175us
I20260709 12:16:50.433507 23605 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:50.383716 23947 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:50.484187 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.467933 24153 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:50.490914 12645 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:50.491809 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:50.426719 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4839) took 1447 ms. Trace:
I20260709 12:16:50.495857 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:50.415859 24016 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30.
I20260709 12:16:50.331506 23917 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:50.504905 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.536751 12641 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:50.493335 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 176398us
I20260709 12:16:50.502703 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:50.520833 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:50.518463 12488 rpcz_store.cc:276] 0709 12:16:48.979501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.093052 (+113551us) service_pool.cc:224] Handling call
0709 12:16:49.136596 (+ 43544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:50.191687 (+1055091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.426316 (+234629us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:50.426625 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2252282112}
I20260709 12:16:50.520556 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:50.539073 12640 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:50.535035 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:50.404279 24094 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.540894 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:50.548070 12495 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:50.541162 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:50.535730 12645 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:16:50.411335 24063 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.550307 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.555660 12496 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:16:50.487998 23894 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.564621 12645 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:50.502763 24138 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.509423 24089 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.518693 24040 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.611887 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.296761 23874 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:50.522063 23514 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:50.648515 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:50.467654 24145 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.128s
I20260709 12:16:50.533360 23929 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.536674 23855 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:50.678435 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 137059us
W20260709 12:16:50.723057 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:50.750427 13260 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:50.756354 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:50.760006 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:50.759763 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:50.760921 12565 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:50.683259 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.763499 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:50.564698 23966 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:50.554667 24153 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:50.678732 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 137619us
I20260709 12:16:50.769352 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:50.769670 12641 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:50.771159 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:50.546422 23770 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 24, Time since last communication: 0.208s
I20260709 12:16:50.658253 23874 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:50.812206 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.657604 24066 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:16:50.696542 24132 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.453s	user 0.001s	sys 0.000s
W20260709 12:16:50.818791 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49516.1us
I20260709 12:16:50.748944 24112 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:50.662396 23903 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.823932 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:50.825977 24066 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:16:50.763303 24152 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
W20260709 12:16:50.831066 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.083s	sys 0.121s
I20260709 12:16:50.839766 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:50.657775 24138 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.674504 24040 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:50.869598 24153 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:50.871418 12646 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:50.825512 23917 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.887101 23917 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:50.825387 23874 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:50.890410 23855 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:50.891017 23855 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:50.848613 23966 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:50.897003 24029 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:50.849447 23825 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.858347 13262 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
W20260709 12:16:50.863432 24151 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.381s	user 0.001s	sys 0.000s
I20260709 12:16:50.903381 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1660) took 1265 ms. Trace:
I20260709 12:16:50.903756 12637 rpcz_store.cc:276] 0709 12:16:49.638146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.638585 (+   439us) service_pool.cc:224] Handling call
0709 12:16:49.638903 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:49.640643 (+  1740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:49.640673 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:49.677518 (+ 36845us) log.cc:844] Serialized 26 byte log entry
0709 12:16:50.193531 (+516013us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:16:50.193573 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.193675 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:50.903169 (+709494us) raft_consensus.cc:1685] finished
0709 12:16:50.903197 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:50.903323 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74805,"raft.queue_time_us":502,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":29469,"thread_start_us":390308,"threads_started":1,"wal-append.queue_time_us":553830}
W20260709 12:16:50.864382 24155 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.206s	user 0.000s	sys 0.000s
I20260709 12:16:50.906915 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4796) took 1180 ms. Trace:
I20260709 12:16:50.907387 12571 rpcz_store.cc:276] 0709 12:16:49.726478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.730695 (+  4217us) service_pool.cc:224] Handling call
0709 12:16:49.731130 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:49.747130 (+ 16000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:49.747184 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:49.747885 (+   701us) log.cc:844] Serialized 26 byte log entry
0709 12:16:50.323657 (+575772us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:50.323724 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.323831 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:50.906668 (+582837us) raft_consensus.cc:1685] finished
0709 12:16:50.906689 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:50.906848 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117405,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":87,"thread_start_us":575302,"threads_started":1,"wal-append.queue_time_us":762428}
I20260709 12:16:50.820618 24003 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:50.910248 24003 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.912752 24003 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:50.814248 24104 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.052s
I20260709 12:16:50.871510 24066 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:16:50.921319 24160 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:16:50.874284 23984 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:50.875368 24040 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:50.875644 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.083s	sys 0.121s
I20260709 12:16:50.674922 24089 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.927134 24162 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:16:50.879009 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:50.928830 12638 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:50.874928 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:50.876124 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:50.865979 24144 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:16:50.772715 24109 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:50.867980 24060 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.756s	user 0.009s	sys 0.002s
I20260709 12:16:50.677461 23659 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:50.658476 23972 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.181s
I20260709 12:16:50.773197 24001 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:16:50.882267 23540 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:50.952683 23770 consensus_queue.cc:1243] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 35.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:16:50.844213 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:50.925225 24040 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:50.865176 24142 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.430s	user 0.000s	sys 0.001s
W20260709 12:16:50.951624 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76880.4us
I20260709 12:16:50.875958 24138 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:51.006590 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 467172us
W20260709 12:16:50.976681 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 641123us
I20260709 12:16:50.998663 23659 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:51.028081 24066 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.039330 12567 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:50.977272 12566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.004s	sys 0.030s
I20260709 12:16:50.928134 24162 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:16:50.967839 24135 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.681s	user 0.001s	sys 0.001s
I20260709 12:16:50.925230 23947 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:50.935256 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1664) took 1187 ms. Trace:
W20260709 12:16:50.968117 24136 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.691s	user 0.000s	sys 0.002s
I20260709 12:16:50.950980 23903 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:51.005198 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:50.964174 12571 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
W20260709 12:16:50.969645 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.028s	sys 0.120s
W20260709 12:16:51.073913 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.029s	sys 0.120s
I20260709 12:16:50.992882 23540 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:51.054147 12566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.004s	sys 0.031s
I20260709 12:16:50.986254 24112 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:51.010201 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:51.060304 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50921.3us
I20260709 12:16:51.055282 13260 rpcz_store.cc:276] 0709 12:16:49.748163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.761093 (+ 12930us) service_pool.cc:224] Handling call
0709 12:16:50.934988 (+1173895us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:50.935173 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1846279936}
W20260709 12:16:51.053287 23605 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.000s
I20260709 12:16:51.004689 24001 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:50.981881 24060 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:51.087553 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77479us
I20260709 12:16:51.085001 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:51.056252 24164 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:51.083334 24040 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:51.094138 24040 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.094676 24089 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.095335 24089 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:16:51.096396 24040 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:51.080582 23770 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:51.097069 24143 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:16:51.097680 24143 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:51.097327 23770 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:51.098001 24163 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:16:51.099440 24094 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.099958 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1760) took 1606 ms. Trace:
I20260709 12:16:51.100205 23770 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.101087 23966 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:51.103544 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4668) took 1537 ms. Trace:
W20260709 12:16:51.103930 24085 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.828s	user 0.009s	sys 0.002s
I20260709 12:16:51.104522 24085 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.080369 23652 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.079500 23947 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.106442 12642 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:51.107533 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4440) took 1265 ms. Trace:
I20260709 12:16:51.107811 23947 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:51.107863 23514 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:51.108485 23514 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:51.108654 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3611) took 1403 ms. Trace:
I20260709 12:16:51.108883 23514 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:51.080974 23903 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.098075 24143 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:51.110852 12494 rpcz_store.cc:276] 0709 12:16:49.705395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.705541 (+   146us) service_pool.cc:224] Handling call
0709 12:16:49.705915 (+   374us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:50.185227 (+479312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:50.185263 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:50.221276 (+ 36013us) log.cc:844] Serialized 52 byte log entry
0709 12:16:50.540700 (+319424us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:50.586616 (+ 45916us) log.cc:844] Serialized 14 byte log entry
0709 12:16:50.692395 (+105779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.692527 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:51.108398 (+415871us) raft_consensus.cc:1685] finished
0709 12:16:51.108422 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:51.108576 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":160695,"prepare.queue_time_us":86301,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":67,"raft.queue_time_us":256240,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":14576,"thread_start_us":755308,"threads_started":2,"wal-append.queue_time_us":778433}
I20260709 12:16:51.072378 24066 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:51.114360 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109244us
I20260709 12:16:51.091192 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:51.118669 12638 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:16:51.093261 23540 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:51.110294 23909 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.612s	user 0.007s	sys 0.004s
I20260709 12:16:51.085529 23899 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.008s
I20260709 12:16:51.126187 12565 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:16:51.125792 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:51.125792 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.109208 24085 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.107898 12642 rpcz_store.cc:276] 0709 12:16:49.886583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.888434 (+  1851us) service_pool.cc:224] Handling call
0709 12:16:51.107320 (+1218886us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:51.107467 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2473453440}
I20260709 12:16:51.103989 12569 rpcz_store.cc:276] 0709 12:16:49.566514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.582987 (+ 16473us) service_pool.cc:224] Handling call
0709 12:16:49.583275 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:49.633312 (+ 50037us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:49.633344 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:49.633900 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 12:16:50.123199 (+489299us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:50.156211 (+ 33012us) log.cc:844] Serialized 14 byte log entry
0709 12:16:50.503555 (+347344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.503663 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:51.103298 (+599635us) raft_consensus.cc:1685] finished
0709 12:16:51.103322 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:51.103473 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81146,"prepare.queue_time_us":32724,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":203425,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":58712,"thread_start_us":816715,"threads_started":2,"wal-append.queue_time_us":665276}
I20260709 12:16:51.100347 12490 rpcz_store.cc:276] 0709 12:16:49.493897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.494055 (+   158us) service_pool.cc:224] Handling call
0709 12:16:49.494382 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:49.494722 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:49.494749 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:49.495239 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:16:49.981146 (+485907us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:49.981194 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:49.981262 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:51.099752 (+1118490us) raft_consensus.cc:1685] finished
0709 12:16:51.099775 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:51.099899 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1046,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":16321,"thread_start_us":485444,"threads_started":1,"wal-append.queue_time_us":717368}
I20260709 12:16:51.114223 24146 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:51.131899 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:51.132305 12570 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: [40.22-40.22]
I20260709 12:16:51.113430 23903 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.134141 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:51.134737 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4466) took 1170 ms. Trace:
I20260709 12:16:51.141992 12565 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:16:51.093246 24164 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.168421 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.167950 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:51.176859 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:51.182262 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.177744 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:51.178802 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:51.105084 23874 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.184613 12565 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:51.190019 12569 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:51.192078 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.194224 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.174528 12638 rpcz_store.cc:276] 0709 12:16:49.964412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.133532 (+1169120us) service_pool.cc:224] Handling call
0709 12:16:51.133953 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.134236 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.134279 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:51.134346 (+    67us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:51.134409 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:51.134481 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:51.134653 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:16:51.143656 23697 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.005s	sys 0.020s
W20260709 12:16:51.201454 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.012s	sys 0.033s
I20260709 12:16:51.131367 23917 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:51.194264 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.202222 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:51.204548 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.013s	sys 0.033s
W20260709 12:16:51.211876 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 709256us
I20260709 12:16:51.212246 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.205734 12609 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30.
I20260709 12:16:51.212788 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.208998 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:51.130852 24146 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:51.131346 24091 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:51.280941 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54741.5us
I20260709 12:16:51.176009 23966 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:51.298954 12645 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:16:51.299858 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4462) took 1336 ms. Trace:
W20260709 12:16:51.210552 23697 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.005s	sys 0.024s
W20260709 12:16:51.322980 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.197398 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:51.115581 23605 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.224071 12564 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:51.121721 23984 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260709 12:16:51.176262 24066 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:16:51.316385 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103669us
I20260709 12:16:51.346762 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.300256 12645 rpcz_store.cc:276] 0709 12:16:49.963213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.298023 (+1334810us) service_pool.cc:224] Handling call
0709 12:16:51.298480 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.299501 (+  1021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:51.299775 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:51.349574 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:51.350611 12645 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:51.320641 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.212347 13262 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:51.197817 24103 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:51.325420 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128111us
W20260709 12:16:51.325821 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.195482 23947 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:16:51.202394 24085 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:51.329057 12640 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:16:51.202215 23894 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.208519 24063 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.206663 23652 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:51.200773 24170 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.161s
I20260709 12:16:51.331151 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.333779 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4449) took 1444 ms. Trace:
I20260709 12:16:51.335155 12565 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:51.355233 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:51.219563 23903 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:51.380535 12646 rpcz_store.cc:276] 0709 12:16:49.889031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.937320 (+1048289us) service_pool.cc:224] Handling call
0709 12:16:50.937678 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:50.937877 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:16:50.976576 (+ 38699us) log.cc:844] Serialized 14 byte log entry
0709 12:16:51.333253 (+356677us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:51.333334 (+    81us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:51.333402 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:51.333492 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:51.333695 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1438,"prepare.queue_time_us":42699,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":12529,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":11532,"thread_start_us":355786,"threads_started":1}
I20260709 12:16:51.291610 24167 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:51.292609 23917 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:51.356688 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.327869 23952 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:51.227178 24146 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:51.317874 23966 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:51.366560 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.128232 23909 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.370278 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:51.331827 24091 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:51.207552 23770 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:51.327673 24171 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:51.379627 12645 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:51.436245 24063 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:51.436872 24063 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.447451 23952 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:51.450133 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.379951 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.226982 24015 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.135s
I20260709 12:16:51.315006 24168 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.077s
W20260709 12:16:51.466941 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.395083 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.397580 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:51.470099 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:16:51.471223 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:51.358251 24154 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:16:51.405761 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49185.5us
I20260709 12:16:51.472724 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.363514 24175 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:51.409652 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:51.474092 12641 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38.
I20260709 12:16:51.475291 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:51.476114 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:51.477154 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:51.481046 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 110903us
I20260709 12:16:51.481784 12641 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:51.372255 24172 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.062s
I20260709 12:16:51.426214 23827 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:51.401320 24167 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:51.485342 13258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.009s	sys 0.018s
W20260709 12:16:51.491741 13258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.009s	sys 0.018s
I20260709 12:16:51.462630 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.399643 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:51.368651 24085 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:51.411842 23966 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:51.344292 24001 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.478495 12645 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:16:51.480029 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.013s	sys 0.013s
I20260709 12:16:51.480705 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:51.505578 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.013s	sys 0.013s
I20260709 12:16:51.365370 24066 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:51.512528 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.483738 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.452473 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.515547 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:51.515705 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.516901 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:51.454337 23855 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.563s	user 0.008s	sys 0.004s
I20260709 12:16:51.529587 13260 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:16:51.530086 12564 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:51.491063 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 370760us
I20260709 12:16:51.507488 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:51.537325 13262 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38.
I20260709 12:16:51.517882 12565 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:51.489295 24175 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.510288 23984 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.030s
I20260709 12:16:51.497085 24167 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:51.489207 24149 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:16:51.499219 24174 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:51.551231 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.552052 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.508965 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:51.452880 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.068s	user 0.021s	sys 0.065s
I20260709 12:16:51.500752 24085 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:51.520434 24176 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.117s
I20260709 12:16:51.494452 23827 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.528151 24091 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.111s
I20260709 12:16:51.531381 24075 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:51.534884 24060 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.571786 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:51.642918 24060 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:51.643059 13257 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:51.534861 24025 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:16:51.644196 23514 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.001s
I20260709 12:16:51.637809 24149 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:51.644699 23514 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.606463 24103 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:51.645913 24149 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.571707 24110 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.076s
I20260709 12:16:51.646610 24103 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.634407 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:51.649839 12488 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:51.650336 24103 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.634003 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.642035 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2218) took 1337 ms. Trace:
I20260709 12:16:51.654307 12566 rpcz_store.cc:276] 0709 12:16:50.304574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.304782 (+   208us) service_pool.cc:224] Handling call
0709 12:16:50.384215 (+ 79433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:51.077095 (+692880us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.077145 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:51.077830 (+   685us) log.cc:844] Serialized 26 byte log entry
0709 12:16:51.411586 (+333756us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:51.411644 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:51.411748 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:51.641716 (+229968us) raft_consensus.cc:1685] finished
0709 12:16:51.641746 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:51.641943 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30491,"raft.queue_time_us":181629,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":59178,"thread_start_us":994648,"threads_started":2,"wal-append.queue_time_us":454018}
I20260709 12:16:51.633483 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.658231 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.659229 12569 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:16:51.570139 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:51.659889 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89914us
I20260709 12:16:51.543118 12645 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:16:51.511015 24015 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:51.546124 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:51.532653 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:51.554877 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.021s	sys 0.066s
I20260709 12:16:51.534612 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:51.564150 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:51.666111 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.3737e+06us
W20260709 12:16:51.660717 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94809us
I20260709 12:16:51.405392 23903 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.671032 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.672858 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:16:51.662402 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:51.674729 13257 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:51.541332 24154 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:16:51.664391 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 118321us
I20260709 12:16:51.678674 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:51.680064 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:51.681589 12641 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:51.680943 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:51.682188 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.701143 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.702325 12496 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:51.610828 23927 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.440s	user 0.010s	sys 0.007s
I20260709 12:16:51.703197 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:51.711910 13254 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
W20260709 12:16:51.575381 24080 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.374s	user 0.014s	sys 0.002s
W20260709 12:16:51.635279 24181 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:16:51.703948 12569 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:51.622879 23605 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.705207 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.707252 12565 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:51.709223 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.540839 24175 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.681831 24168 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:51.741122 24168 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:51.741648 24168 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:51.690577 24015 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.743736 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:51.638593 24182 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:16:51.653118 24075 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.713686 13255 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:51.533425 23855 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.725586 12644 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:51.575104 24063 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.757922 12642 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:51.703627 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.623574 23909 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.689553 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:51.704381 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:51.722107 12461 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38.
I20260709 12:16:51.763816 12461 leader_election.cc:403] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
W20260709 12:16:51.764341 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49749.6us
I20260709 12:16:51.765057 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:51.609292 23952 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.624364 24028 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.629861 23874 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:51.723929 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:16:51.766383 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:51.762712 24153 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.028s	sys 0.053s
I20260709 12:16:51.728062 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:51.643486 24060 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.728647 23514 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:51.708325 24146 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.550390 23917 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.719965 23927 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.730863 24101 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:51.773716 24175 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:51.699811 23984 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
W20260709 12:16:51.720353 24143 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.608s	user 0.007s	sys 0.003s
I20260709 12:16:51.726863 24179 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:51.737061 23652 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:51.687341 24103 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:16:51.623315 24164 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.739348 23903 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.754920 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:51.756304 12645 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: [33.21-33.21]
W20260709 12:16:51.763185 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73713.3us
I20260709 12:16:51.767675 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:51.725260 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.610116 24066 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.747361 24075 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:51.747051 23770 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:51.749056 23855 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:51.778265 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:51.734414 24176 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:51.745564 24015 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:51.707504 24150 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 12:16:51.945869 24075 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:51.768500 24153 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.028s	sys 0.053s
I20260709 12:16:51.875473 13256 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:51.771862 24063 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:51.882891 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.013s	sys 0.009s
W20260709 12:16:51.887387 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 428383us
I20260709 12:16:51.822790 23853 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:16:51.917128 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:51.776984 24167 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.823407 23966 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:51.768885 24080 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.107s
I20260709 12:16:51.822669 24101 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:51.926087 12489 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:51.827848 23874 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:51.999895 23874 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.865053 24175 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:52.004326 24175 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.772819 24183 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.165s
I20260709 12:16:52.006022 24183 consensus_queue.cc:1243] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 32.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:16:51.932855 23770 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:52.007537 23770 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.017930 24101 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:51.902247 24179 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:51.960026 24063 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:52.038295 24063 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:51.958182 23947 consensus_queue.cc:765] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 24 is ahead of the local log (next sequential op: 24). Destination peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.24, Next index: 25, Last known committed idx: 21, Time since last communication: 0.847s
I20260709 12:16:51.943353 24103 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:52.043579 24103 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.052942 24167 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.060781 12493 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:51.944036 23605 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:51.944231 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166110us
I20260709 12:16:51.770135 23659 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38.
I20260709 12:16:51.917831 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:16:51.979866 23853 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:52.064720 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.065560 12641 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:16:52.066538 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.067169 12641 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:16:52.068084 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:51.943289 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.070624 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:51.950644 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3658) took 1784 ms (client timeout 1739 ms). Trace:
W20260709 12:16:52.072080 12644 rpcz_store.cc:269] 0709 12:16:50.166004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.181428 (+ 15424us) service_pool.cc:224] Handling call
0709 12:16:51.950350 (+1768922us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:51.950536 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3511051648}
I20260709 12:16:52.073347 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.074193 12644 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:52.075212 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:51.953868 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.014s	sys 0.009s
I20260709 12:16:52.077121 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4803) took 1319 ms. Trace:
I20260709 12:16:52.077497 12572 rpcz_store.cc:276] 0709 12:16:50.757524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.757670 (+   146us) service_pool.cc:224] Handling call
0709 12:16:50.812307 (+ 54637us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:51.135293 (+322986us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.135332 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:51.200553 (+ 65221us) log.cc:844] Serialized 26 byte log entry
0709 12:16:52.076641 (+876088us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:52.076703 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.076806 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:52.076860 (+    54us) raft_consensus.cc:1685] finished
0709 12:16:52.076877 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.077061 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3271,"raft.queue_time_us":189859,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":63726,"thread_start_us":1063492,"threads_started":2,"wal-append.queue_time_us":717905}
I20260709 12:16:51.959631 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:51.967618 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.960724 12643 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:52.080739 12643 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:51.961087 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:52.081914 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:51.922161 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.083833 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.084443 12645 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:16:51.986964 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 454381us
I20260709 12:16:52.085882 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:52.063002 23605 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.090274 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.069363 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.111927 12641 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:52.114647 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:52.117727 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:52.081565 12643 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.120383 12643 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:52.126539 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.127954 12491 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:16:52.132496 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:51.897778 24143 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.134895 12644 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:52.136482 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:51.930919 23697 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.152s
I20260709 12:16:51.936648 23909 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.179s
I20260709 12:16:52.100989 23894 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:52.143115 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.156116 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.169559 12641 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30.
I20260709 12:16:52.101413 24110 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:52.167853 12491 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:52.184875 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.141491 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.166669 12643 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:16:52.171319 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.186983 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:16:52.172837 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.004s	sys 0.012s
I20260709 12:16:52.122591 24153 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:52.122402 24164 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:52.133035 23952 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:16:52.187860 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.004s	sys 0.012s
I20260709 12:16:52.100001 23853 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:52.187592 12638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.021s	sys 0.033s
I20260709 12:16:52.198949 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:52.156431 24150 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:52.188237 12641 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:52.152125 23984 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:16:52.200371 12644 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:52.166136 23874 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.212962 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2219) took 1693 ms. Trace:
I20260709 12:16:52.183257 23909 consensus_queue.cc:1243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 32.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:16:52.184633 23966 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.217669 12563 rpcz_store.cc:276] 0709 12:16:50.519569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.519793 (+   224us) service_pool.cc:224] Handling call
0709 12:16:50.553981 (+ 34188us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:51.123760 (+569779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.123798 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:51.124302 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:16:51.653877 (+529575us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:16:51.653925 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:51.654002 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:52.212578 (+558576us) raft_consensus.cc:1685] finished
0709 12:16:52.212609 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.212871 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91872,"raft.queue_time_us":397473,"spinlock_wait_cycles":14952192,"thread_start_us":929630,"threads_started":2,"wal-append.queue_time_us":748649}
I20260709 12:16:52.220909 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.224999 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:52.227655 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.028s	sys 0.033s
I20260709 12:16:52.227977 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:52.230589 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:52.226604 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:52.230572 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.028s	sys 0.033s
W20260709 12:16:52.229700 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:52.228605 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.234560 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:52.234894 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:52.234987 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:52.235922 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:52.207599 23952 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:52.242060 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.242624 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:52.245321 12497 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:52.245524 12644 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:52.247143 12491 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:52.247967 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:52.251559 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.252203 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:52.279034 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.280800 12644 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:16:52.278225 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:52.282903 12563 raft_consensus.cc:2368] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:16:52.285455 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.287211 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.288527 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:52.291749 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:52.231506 24150 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:52.197088 24110 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.233850 24179 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.248708 23514 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:52.250063 23952 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:52.251468 24146 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:52.371584 24203 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:16:52.372174 23984 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:52.373245 24080 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:52.283291 23927 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.375922 24167 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:52.357960 23514 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:52.340812 23874 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:52.259074 24202 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:52.376645 24150 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 34.14, Last appended by leader: 14, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.373256 12495 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:52.248593 24060 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.379248 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:52.243973 24176 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:52.371330 23909 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:16:52.382676 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.484735 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.485697 12491 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:16:52.281579 24001 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:52.274143 12641 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:16:52.412096 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.010s	sys 0.037s
W20260709 12:16:52.509356 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.011s	sys 0.038s
I20260709 12:16:52.445765 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:52.403542 24179 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:52.506608 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.043s	sys 0.087s
W20260709 12:16:52.511536 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.043s	sys 0.088s
I20260709 12:16:52.510260 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:52.515129 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.513295 12495 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:52.518536 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.520805 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.523169 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:16:52.473768 23514 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:52.526085 12491 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:52.554349 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:52.539224 24179 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.508579 24001 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:52.473966 23984 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.572090 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.473315 24167 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.580919 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:52.559926 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.584564 12491 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
W20260709 12:16:52.589823 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 347287us
I20260709 12:16:52.569176 23966 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:52.615912 23966 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:52.617257 23966 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.622367 23605 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:52.624285 24198 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260709 12:16:52.626348 24194 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:16:52.631755 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4820) took 1410 ms. Trace:
I20260709 12:16:52.632344 12494 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.17-33.17]
I20260709 12:16:52.654398 24110 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.573881 24001 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:52.610329 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:52.612300 24179 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.614655 23984 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.655936 13253 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:52.632296 12567 rpcz_store.cc:276] 0709 12:16:51.221535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.678826 (+457291us) service_pool.cc:224] Handling call
0709 12:16:51.679135 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:51.701991 (+ 22856us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.702032 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:51.702615 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:16:52.511969 (+809354us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:52.512010 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.512077 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:52.631448 (+119371us) raft_consensus.cc:1685] finished
0709 12:16:52.631479 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.631675 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":39045,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":36791,"thread_start_us":809014,"threads_started":1,"wal-append.queue_time_us":926838}
I20260709 12:16:52.655993 24179 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:52.639259 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1659) took 3008 ms. Trace:
I20260709 12:16:52.657027 24179 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.657163 12643 rpcz_store.cc:276] 0709 12:16:49.630535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.636173 (+  5638us) service_pool.cc:224] Handling call
0709 12:16:49.636543 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.960590 (+2324047us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54002bd020. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:16:52.188611 (+228021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.638853 (+450242us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.639129 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3134128256}
I20260709 12:16:52.513543 24202 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.658543 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.659188 24179 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.659720 24179 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:52.645701 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1435) took 1764 ms. Trace:
I20260709 12:16:52.660290 24179 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.660619 13259 rpcz_store.cc:276] 0709 12:16:50.881075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.881205 (+   130us) service_pool.cc:224] Handling call
0709 12:16:50.881539 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:50.933122 (+ 51583us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:50.933169 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:50.933229 (+    60us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:16:50.933275 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.645337 (+1712062us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.645406 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.645604 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118007680}
I20260709 12:16:52.662771 24179 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.662950 24167 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.582617 24143 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.665210 24143 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:52.649335 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4676) took 1461 ms. Trace:
I20260709 12:16:52.650770 23827 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.651568 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3645) took 3886 ms. Trace:
I20260709 12:16:52.654778 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:52.659657 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.640267 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2213) took 2584 ms. Trace:
I20260709 12:16:52.513341 24176 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.575179 24200 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:52.648470 23853 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:52.584932 24149 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.013s	sys 0.017s
I20260709 12:16:52.679123 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4473) took 1479 ms. Trace:
W20260709 12:16:52.682569 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72393.5us
I20260709 12:16:52.702116 12643 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:52.708050 12646 rpcz_store.cc:276] 0709 12:16:51.199949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.487989 (+288040us) service_pool.cc:224] Handling call
0709 12:16:51.488345 (+   356us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:16:51.497979 (+  9634us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.498032 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:16:51.521310 (+ 23278us) log.cc:844] Serialized 52 byte log entry
0709 12:16:52.201538 (+680228us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:52.596277 (+394739us) log.cc:844] Serialized 14 byte log entry
0709 12:16:52.596754 (+   477us) log.cc:844] Serialized 14 byte log entry
0709 12:16:52.678581 (+ 81827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.678711 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:52.678775 (+    64us) raft_consensus.cc:1685] finished
0709 12:16:52.678796 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.679018 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90639,"raft.queue_time_us":174556,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":29117,"thread_start_us":959059,"threads_started":2,"wal-append.queue_time_us":657229}
I20260709 12:16:52.710182 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:52.710561 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 476081us
I20260709 12:16:52.854313 12637 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 34 index: 17. (index mismatch)
I20260709 12:16:52.855680 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:52.827610 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.000s	sys 0.013s
I20260709 12:16:52.702454 12639 rpcz_store.cc:276] 0709 12:16:48.765208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:48.778243 (+ 13035us) service_pool.cc:224] Handling call
0709 12:16:48.778659 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:49.617886 (+839227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:49.617920 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:49.618430 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:16:50.118180 (+499750us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:16:50.382644 (+264464us) log.cc:844] Serialized 14 byte log entry
0709 12:16:50.382986 (+   342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:50.383147 (+   161us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:50.901271 (+518124us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b180037fea0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:52.626730 (+1725459us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b180037fea0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:52.651293 (+ 24563us) raft_consensus.cc:1685] finished
0709 12:16:52.651325 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.651493 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":862697,"prepare.queue_time_us":301410,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":46,"raft.queue_time_us":972895,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":223,"spinlock_wait_cycles":79508992,"thread_start_us":741048,"threads_started":2,"wal-append.queue_time_us":877002}
W20260709 12:16:52.858035 12638 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.011s	sys 0.017s
W20260709 12:16:52.859575 12638 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.012s	sys 0.017s
I20260709 12:16:52.698621 12565 rpcz_store.cc:276] 0709 12:16:51.187729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.553615 (+365886us) service_pool.cc:224] Handling call
0709 12:16:51.554040 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.723866 (+169826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.649046 (+925180us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.649264 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":351798016}
W20260709 12:16:52.716012 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:52.860343 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:52.869619 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:52.857985 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.000s	sys 0.015s
I20260709 12:16:52.871301 12639 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:52.700003 12570 rpcz_store.cc:276] 0709 12:16:50.056210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.056436 (+   226us) service_pool.cc:224] Handling call
0709 12:16:50.056920 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:51.132851 (+1075931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.639950 (+1507099us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.640191 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2472456064}
I20260709 12:16:52.856972 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4460) took 2894 ms. Trace:
I20260709 12:16:52.718117 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.874119 12637 rpcz_store.cc:276] 0709 12:16:49.962627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.851534 (+2888907us) service_pool.cc:224] Handling call
0709 12:16:52.852873 (+  1339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:52.856361 (+  3488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.856790 (+   429us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:52.873111 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.716802 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:52.892578 23605 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:52.714883 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 466622us
I20260709 12:16:52.910542 24214 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:16:52.757268 24075 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.811s	user 0.011s	sys 0.004s
W20260709 12:16:52.677453 24085 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.977s	user 0.005s	sys 0.005s
I20260709 12:16:52.716336 24143 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.912602 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1779) took 1254 ms. Trace:
I20260709 12:16:52.913050 24075 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:52.913487 24085 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:52.717154 24149 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.014s	sys 0.018s
W20260709 12:16:52.767113 24168 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.025s	user 0.003s	sys 0.012s
I20260709 12:16:52.916821 24143 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.918309 12640 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:16:52.729667 24176 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.923219 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:52.924008 12495 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:52.660919 24202 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.889165 12637 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:52.926123 12637 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:52.764225 24101 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.745s	user 0.012s	sys 0.003s
W20260709 12:16:52.891431 24175 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.008s	sys 0.012s
I20260709 12:16:52.901249 23605 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.904138 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:52.906214 23874 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:52.916352 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4850) took 1246 ms. Trace:
I20260709 12:16:52.916167 24216 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:52.892606 23925 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:52.919013 12489 rpcz_store.cc:276] 0709 12:16:51.657956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.658140 (+   184us) service_pool.cc:224] Handling call
0709 12:16:51.664212 (+  6072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.994461 (+330249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.911761 (+917300us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.912274 (+   513us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":601772288}
I20260709 12:16:52.928131 12637 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.950119 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4679) took 1761 ms. Trace:
I20260709 12:16:52.950819 12564 rpcz_store.cc:276] 0709 12:16:51.188908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.580374 (+391466us) service_pool.cc:224] Handling call
0709 12:16:51.580930 (+   556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.666372 (+ 85442us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.666405 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:51.666469 (+    64us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:51.666500 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.949296 (+1282796us) spinlock_profiling.cc:243] Waited 85.2 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.949378 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.949803 (+   425us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195949824}
I20260709 12:16:52.950860 24101 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:52.953943 12488 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:52.953430 23925 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:52.943224 12493 rpcz_store.cc:276] 0709 12:16:51.669787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.670654 (+   867us) service_pool.cc:224] Handling call
0709 12:16:52.915764 (+1245110us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:52.916121 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":893577600}
I20260709 12:16:52.957170 23925 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.959458 12489 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:52.971791 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1434) took 2114 ms. Trace:
I20260709 12:16:52.974362 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:53.025676 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1442) took 1347 ms (client timeout 1598 ms). Trace:
W20260709 12:16:53.026788 13262 rpcz_store.cc:269] 0709 12:16:51.677934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.973736 (+1295802us) service_pool.cc:224] Handling call
0709 12:16:53.025543 (+ 51807us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:53.026562 24101 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:52.950163 12637 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:52.957537 24150 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:53.030354 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121995us
I20260709 12:16:53.031525 24150 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:52.964670 24063 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.925s	user 0.016s	sys 0.000s
I20260709 12:16:53.033114 24063 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.034880 24150 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.969319 23605 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:52.962301 24085 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:53.037999 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 328380us
I20260709 12:16:52.966046 24168 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.035744 24063 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:53.021816 13258 rpcz_store.cc:276] 0709 12:16:50.856910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:50.858386 (+  1476us) service_pool.cc:224] Handling call
0709 12:16:50.858663 (+   277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:50.927420 (+ 68757us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:50.927453 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:50.928012 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:16:51.492176 (+564164us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:51.492240 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:52.970513 (+1478273us) spinlock_profiling.cc:243] Waited 68.4 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:52.970611 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:52.970782 (+   171us) raft_consensus.cc:1685] finished
0709 12:16:52.970801 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:52.971383 (+   582us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2425,"raft.queue_time_us":1328,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":15594,"spinlock_wait_cycles":157208448,"thread_start_us":563691,"threads_started":1,"wal-append.queue_time_us":610473,"wal-append.run_cpu_time_us":1771,"wal-append.run_wall_time_us":1001786}
I20260709 12:16:53.045274 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4827) took 1327 ms. Trace:
I20260709 12:16:53.045799 12563 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:16:53.046082 12569 rpcz_store.cc:276] 0709 12:16:51.717410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.558410 (+841000us) service_pool.cc:224] Handling call
0709 12:16:52.558952 (+   542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:52.577041 (+ 18089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:52.577087 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:52.577709 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:16:53.043906 (+466197us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:53.044032 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.044170 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:53.044243 (+    73us) raft_consensus.cc:1685] finished
0709 12:16:53.044265 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.044690 (+   425us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9270,"raft.queue_time_us":104102,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":72983,"threads_started":1,"wal-append.queue_time_us":370777}
W20260709 12:16:53.050714 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178073us
I20260709 12:16:53.056836 24039 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:53.059033 12641 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:16:52.985216 12536 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30.
I20260709 12:16:53.076411 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:52.970806 23874 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:53.081360 13254 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:53.057593 24039 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:53.086536 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54374.4us
I20260709 12:16:53.058934 24081 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:53.083096 23874 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:53.090929 24081 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:53.106319 12645 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:53.108629 23770 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:53.111054 24039 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.112690 23929 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:53.118093 23929 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.119469 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.106202 24081 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.122082 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.122598 24081 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:53.123100 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:53.123814 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139486us
I20260709 12:16:53.123340 12564 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:16:53.126078 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 270907us
I20260709 12:16:53.128067 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.128953 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:53.136341 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:53.132465 23874 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.22, Last appended by leader: 22, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.138162 24233 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:16:53.129567 23514 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.554s	user 0.017s	sys 0.012s
I20260709 12:16:53.145376 24003 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:16:53.146234 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55445.8us
W20260709 12:16:53.146924 23952 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.699s	user 0.008s	sys 0.016s
I20260709 12:16:53.149906 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:53.174870 12637 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 17. (term mismatch)
I20260709 12:16:53.143417 24231 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:53.133987 23855 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:53.157233 24168 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:53.157997 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:53.158968 12610 leader_election.cc:400] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:53.163130 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:53.150930 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.182593 12610 leader_election.cc:403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:53.176934 12637 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 16
I20260709 12:16:53.178961 23514 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.134193 23929 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:53.181507 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4856) took 1504 ms (client timeout 1585 ms). Trace:
I20260709 12:16:53.193740 12494 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:53.194240 24028 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:16:53.194849 24028 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:53.195636 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4801) took 3465 ms (client timeout 3050 ms). Trace:
W20260709 12:16:53.196148 12568 rpcz_store.cc:269] 0709 12:16:49.729695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:49.729881 (+   186us) service_pool.cc:224] Handling call
0709 12:16:53.195027 (+3465146us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:53.195335 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2341954944}
I20260709 12:16:53.196316 24028 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.200455 23929 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 38. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:53.202225 24231 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:16:53.193766 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44050.7us
I20260709 12:16:53.206143 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:53.181586 24003 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:53.188206 23952 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:53.190225 12488 rpcz_store.cc:269] 0709 12:16:51.676649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.703312 (+ 26663us) service_pool.cc:224] Handling call
0709 12:16:53.180335 (+1477023us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:53.181422 (+  1087us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2741492480}
I20260709 12:16:53.213835 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:53.214320 12496 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:53.216601 23915 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.207158 12565 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:53.205614 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.218241 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4485) took 1691 ms. Trace:
I20260709 12:16:53.183909 12637 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 31.17
I20260709 12:16:53.223661 12564 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:53.213779 23874 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:53.226574 12637 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:53.223249 24003 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.227674 23915 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.225948 23929 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:53.227610 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.234696 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.237445 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:53.238629 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.240077 13259 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
W20260709 12:16:53.241395 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94703.9us
I20260709 12:16:53.234869 24201 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:53.242707 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.242981 24201 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.244204 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:53.245061 23952 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.246009 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:53.247056 13258 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:53.248083 24187 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:53.248795 24187 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:53.250378 24187 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.252279 24187 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:53.245421 24201 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.258132 23947 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:16:53.248221 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.238479 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.264966 23929 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:16:53.260792 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 147134us
I20260709 12:16:53.270767 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:53.272718 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:53.277179 23915 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.277213 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.307372 13258 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:53.278556 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1680) took 1189 ms. Trace:
I20260709 12:16:53.308871 12645 rpcz_store.cc:276] 0709 12:16:52.088887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.089925 (+  1038us) service_pool.cc:224] Handling call
0709 12:16:53.278252 (+1188327us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:53.278437 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2256397056}
W20260709 12:16:53.318084 24178 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.591s	user 0.012s	sys 0.010s
W20260709 12:16:53.318718 24178 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.695s	user 0.012s	sys 0.010s
I20260709 12:16:53.318847 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:53.319896 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:16:53.274294 24085 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:16:53.282714 24183 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:53.322729 23947 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:53.322968 24183 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:53.323408 23947 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:53.286162 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3675) took 1123 ms. Trace:
I20260709 12:16:53.325804 12643 rpcz_store.cc:276] 0709 12:16:52.162300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.045827 (+883527us) service_pool.cc:224] Handling call
0709 12:16:53.046566 (+   739us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:53.088084 (+ 41518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:53.088121 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:53.134589 (+ 46468us) log.cc:844] Serialized 26 byte log entry
0709 12:16:53.177352 (+ 42763us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:53.231305 (+ 53953us) log.cc:844] Serialized 14 byte log entry
0709 12:16:53.260433 (+ 29128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.260514 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:53.285718 (+ 25204us) raft_consensus.cc:1685] finished
0709 12:16:53.285744 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.285966 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61384,"prepare.queue_time_us":55055,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":6383,"raft.queue_time_us":79088,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":51726,"thread_start_us":30000,"threads_started":1,"wal-append.queue_time_us":71297}
I20260709 12:16:53.328243 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:53.287256 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76098.7us
I20260709 12:16:53.331864 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4868) took 1596 ms. Trace:
I20260709 12:16:53.282456 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:53.287719 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:53.294838 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4873) took 1557 ms (client timeout 1522 ms). Trace:
W20260709 12:16:53.333793 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:53.298671 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:53.245131 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:53.304853 24039 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:53.315073 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4698) took 1356 ms. Trace:
I20260709 12:16:53.323504 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3622) took 1113 ms. Trace:
W20260709 12:16:53.288372 24184 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.664s	user 0.006s	sys 0.012s
I20260709 12:16:53.267725 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.321367 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:53.233821 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:53.333751 12492 rpcz_store.cc:269] 0709 12:16:51.737696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.743431 (+  5735us) service_pool.cc:224] Handling call
0709 12:16:53.293026 (+1549595us) spinlock_profiling.cc:243] Waited 86.2 ms on lock 0x7b54000ecf20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6e18 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:53.294310 (+  1284us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:53.294557 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2719942272}
I20260709 12:16:53.332422 12495 rpcz_store.cc:276] 0709 12:16:51.735984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.564753 (+828769us) service_pool.cc:224] Handling call
0709 12:16:52.565032 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:52.924736 (+359704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.331387 (+406651us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:53.331764 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":813852032}
I20260709 12:16:53.348860 12563 rpcz_store.cc:276] 0709 12:16:51.959121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.570350 (+611229us) service_pool.cc:224] Handling call
0709 12:16:53.044767 (+474417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:53.046552 (+  1785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.314676 (+268124us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54002463d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:53.314943 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059322752}
I20260709 12:16:53.352033 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:53.349416 12494 rpcz_store.cc:276] 0709 12:16:52.210155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.653187 (+443032us) service_pool.cc:224] Handling call
0709 12:16:53.193300 (+540113us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:53.194407 (+  1107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.323237 (+128830us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b5400367750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:53.323420 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1212111104}
I20260709 12:16:53.323681 24183 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:53.358408 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.362998 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4487) took 1205 ms. Trace:
I20260709 12:16:53.363495 12637 rpcz_store.cc:276] 0709 12:16:52.153209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.875386 (+722177us) service_pool.cc:224] Handling call
0709 12:16:52.888701 (+ 13315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:53.357943 (+469242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.358213 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:16:53.220939 12638 rpcz_store.cc:276] 0709 12:16:51.497539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.659844 (+162305us) service_pool.cc:224] Handling call
0709 12:16:51.660233 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:51.664651 (+  4418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:51.664684 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:51.665163 (+   479us) log.cc:844] Serialized 26 byte log entry
0709 12:16:52.207618 (+542455us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:16:52.860423 (+652805us) log.cc:844] Serialized 14 byte log entry
0709 12:16:53.096084 (+235661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.102276 (+  6192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:53.173872 (+ 71596us) raft_consensus.cc:1685] finished
0709 12:16:53.173901 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.189205 (+ 15304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16532,"prepare.queue_time_us":545614,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":59242,"raft.run_cpu_time_us":483,"raft.run_wall_time_us":24061,"thread_start_us":1350948,"threads_started":3,"wal-append.queue_time_us":686723}
I20260709 12:16:53.366597 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:53.325016 23947 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.371475 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:53.375895 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:53.379328 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:53.386312 12490 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 38 because replica has last-logged OpId of term: 37 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 14.
I20260709 12:16:53.388908 13259 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:16:53.390175 13253 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:53.370443 24231 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:53.392498 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 244846us
I20260709 12:16:53.392957 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:53.391927 24138 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:53.394537 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:51.452277 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:16:53.388586 23915 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:53.366189 23855 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:53.403887 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 212441us
I20260709 12:16:53.413929 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:53.404759 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 137195us
W20260709 12:16:53.405326 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 187139us
I20260709 12:16:53.404264 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:53.398031 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:53.420850 24171 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.753s	user 0.002s	sys 0.035s
W20260709 12:16:53.425220 24171 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.817s	user 0.002s	sys 0.036s
W20260709 12:16:53.416545 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 4875) took 1654 ms (client timeout 2086 ms). Trace:
I20260709 12:16:53.434387 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3618) took 1223 ms. Trace:
W20260709 12:16:53.435855 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 226) took 90084 ms (client timeout 15000 ms). Trace:
W20260709 12:16:53.431658 12490 rpcz_store.cc:269] 0709 12:16:51.762308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.079964 (+317656us) service_pool.cc:224] Handling call
0709 12:16:53.298090 (+1218126us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:16:53.416471 (+118381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46668032}
W20260709 12:16:53.414708 24236 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.192s	user 0.002s	sys 0.000s
W20260709 12:16:53.396899 24173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.764s	user 0.011s	sys 0.026s
W20260709 12:16:53.416067 24001 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.760s	user 0.007s	sys 0.023s
W20260709 12:16:53.445722 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 229342us
I20260709 12:16:53.414530 23947 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:53.419198 24085 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:53.437461 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:53.435037 12491 rpcz_store.cc:276] 0709 12:16:52.164596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.595242 (+430646us) service_pool.cc:224] Handling call
0709 12:16:52.595706 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:52.595996 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:16:52.842944 (+246948us) log.cc:844] Serialized 14 byte log entry
0709 12:16:53.098034 (+255090us) log.cc:844] Serialized 14 byte log entry
0709 12:16:53.313334 (+215300us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b18003d92a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:16:53.350642 (+ 37308us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:53.359188 (+  8546us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:16:53.366902 (+  7714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.378206 (+ 11304us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.387285 (+  9079us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":350656,"prepare.queue_time_us":313066,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":155,"raft.queue_time_us":255249,"raft.run_cpu_time_us":539,"raft.run_wall_time_us":28797,"spinlock_wait_cycles":15109504,"thread_start_us":232283,"threads_started":2,"wal-append.queue_time_us":261085}
I20260709 12:16:53.404409 23853 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
W20260709 12:16:53.399343 24170 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 32 current term: 32 is higher.
W20260709 12:16:53.450737 24173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.835s	user 0.012s	sys 0.026s
W20260709 12:16:53.399267 24180 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.748s	user 0.000s	sys 0.014s
I20260709 12:16:53.399402 24080 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:53.468765 12637 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:53.473397 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:53.486682 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1781) took 1525 ms. Trace:
I20260709 12:16:53.466965 24001 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.491496 24001 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.462724 24085 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.494045 24085 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:53.498672 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47585.4us
W20260709 12:16:53.471212 24180 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.860s	user 0.001s	sys 0.014s
I20260709 12:16:53.487397 12496 rpcz_store.cc:276] 0709 12:16:51.961379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.082784 (+121405us) service_pool.cc:224] Handling call
0709 12:16:53.162598 (+1079814us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:16:53.486546 (+323948us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41905920}
W20260709 12:16:53.491497 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54188.4us
W20260709 12:16:53.436501 12391 rpcz_store.cc:269] 0709 12:15:23.351748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:36.428351 (+13076603us) service_pool.cc:224] Handling call
0709 12:16:53.435494 (+77007143us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:16:45.007135 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:16:45.058333 (+ 51198us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:16:45.058401 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:16:45.058413 (+    12us) tablet.cc:662] Decoding operations
0709 12:16:45.059436 (+  1023us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:16:45.059535 (+    99us) write_op.cc:641] Partition lock acquired for write op
0709 12:16:45.059562 (+    27us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:16:45.060339 (+   777us) tablet.cc:701] Row locks acquired
0709 12:16:45.060398 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:16:45.060608 (+   210us) write_op.cc:270] Start()
0709 12:16:45.060782 (+   174us) write_op.cc:276] Timestamp: P: 1783599405060526 usec, L: 0
0709 12:16:45.060823 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:16:45.068538 (+  7715us) log.cc:844] Serialized 2861 byte log entry
0709 12:16:45.594404 (+525866us) op_driver.cc:464] REPLICATION: finished
0709 12:16:45.632478 (+ 38074us) write_op.cc:301] APPLY: starting
0709 12:16:45.632567 (+    89us) tablet.cc:1367] starting BulkCheckPresence
0709 12:16:45.679773 (+ 47206us) tablet.cc:1370] finished BulkCheckPresence
0709 12:16:45.679892 (+   119us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:16:46.064418 (+384526us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:16:46.150172 (+ 85754us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:16:46.150245 (+    73us) write_op.cc:312] APPLY: finished
0709 12:16:46.150623 (+   378us) log.cc:844] Serialized 73 byte log entry
0709 12:16:46.150817 (+   194us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:46.238473 (+ 87656us) write_op.cc:454] Released schema lock
0709 12:16:46.255263 (+ 16790us) write_op.cc:341] FINISH: Updating metrics
0709 12:16:46.274500 (+ 19237us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:16:46.274540 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3438222,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":37755,"apply.run_cpu_time_us":20362,"apply.run_wall_time_us":637571,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33359,"num_ops":7,"prepare.queue_time_us":216894,"prepare.run_cpu_time_us":57837,"prepare.run_wall_time_us":393743,"raft.queue_time_us":137351,"raft.run_cpu_time_us":9981,"raft.run_wall_time_us":38173,"replication_time_us":533356,"thread_start_us":344499,"threads_started":4,"wal-append.queue_time_us":331189,"wal-append.run_cpu_time_us":44079,"wal-append.run_wall_time_us":1725717}]]}
W20260709 12:16:53.513016 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 133) took 15958 ms (client timeout 15000 ms). Trace:
W20260709 12:16:53.517263 12391 rpcz_store.cc:269] 0709 12:16:37.554393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.512676 (+15958283us) service_pool.cc:211] Skipping call since client already timed out
0709 12:16:53.512898 (+   222us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:16:53.509307 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:53.508283 13253 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:53.417935 23697 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.091s
I20260709 12:16:53.554533 12644 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:16:53.551527 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.562646 12566 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:53.563768 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:53.566243 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.567369 12566 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.513401 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4497) took 1134 ms. Trace:
I20260709 12:16:53.517824 12536 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:16:53.500429 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.510238 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:53.485285 23855 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.544692 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.572695 12566 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:53.574040 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:53.574600 12568 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:16:53.575460 23855 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:53.576260 24094 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:53.581763 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4811) took 2303 ms. Trace:
I20260709 12:16:53.452154 24231 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.546569 24171 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:53.625067 24171 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.625752 24171 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:16:53.547502 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:53.632033 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.634055 23514 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.548517 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:53.642495 12572 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:53.549597 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:53.508589 24080 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.545804 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.647652 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4474) took 2447 ms. Trace:
I20260709 12:16:53.648319 12640 rpcz_store.cc:276] 0709 12:16:51.200275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.493562 (+293287us) service_pool.cc:224] Handling call
0709 12:16:51.513334 (+ 19772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:52.920093 (+1406759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.647269 (+727176us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:53.647572 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3229572992}
I20260709 12:16:53.570431 12641 rpcz_store.cc:276] 0709 12:16:52.379206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.083277 (+704071us) service_pool.cc:224] Handling call
0709 12:16:53.186238 (+102961us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:53.198023 (+ 11785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:53.198093 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:53.198962 (+   869us) log.cc:844] Serialized 26 byte log entry
0709 12:16:53.229971 (+ 31009us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:16:53.230029 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.230130 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:53.513105 (+282975us) raft_consensus.cc:1685] finished
0709 12:16:53.513133 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.513323 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":523,"raft.queue_time_us":8725,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":237,"thread_start_us":23619,"threads_started":1,"wal-append.queue_time_us":253156}
I20260709 12:16:53.651445 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.652320 24060 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:53.652964 24060 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.653386 24060 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:53.653775 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4491) took 1497 ms. Trace:
I20260709 12:16:53.582263 12571 rpcz_store.cc:276] 0709 12:16:51.124266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.124421 (+   155us) service_pool.cc:224] Handling call
0709 12:16:51.124789 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.223687 (+ 98898us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:16:51.675605 (+451918us) log.cc:844] Serialized 14 byte log entry
0709 12:16:52.122418 (+446813us) log.cc:844] Serialized 14 byte log entry
0709 12:16:52.122608 (+   190us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:52.122660 (+    52us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:16:52.122739 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.148678 (+1025939us) spinlock_profiling.cc:243] Waited 9.05 ms on lock 0x7b18003e8f60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:16:53.386506 (+237828us) spinlock_profiling.cc:243] Waited 98.6 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:53.386570 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.427906 (+ 41336us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":710346,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":9161,"raft.queue_time_us":202927,"raft.run_cpu_time_us":490,"raft.run_wall_time_us":12862,"spinlock_wait_cycles":247987072,"thread_start_us":797225,"threads_started":2,"wal-append.queue_time_us":588849}
I20260709 12:16:53.655781 23514 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:16:53.497907 24177 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.142s
I20260709 12:16:53.614890 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3652) took 3649 ms. Trace:
I20260709 12:16:53.615577 24164 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:53.657063 24164 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.657701 24170 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:53.658319 24177 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:53.658761 12642 rpcz_store.cc:276] 0709 12:16:49.965429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:51.132887 (+1167458us) service_pool.cc:224] Handling call
0709 12:16:51.133264 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:51.758497 (+625233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.614293 (+1855796us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:53.614669 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1419060480}
I20260709 12:16:53.659323 23540 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:53.618618 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118798us
I20260709 12:16:53.660732 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.590106 12638 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:16:53.654310 12639 rpcz_store.cc:276] 0709 12:16:52.155893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.938551 (+782658us) service_pool.cc:224] Handling call
0709 12:16:52.986551 (+ 48000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:53.016902 (+ 30351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:53.016972 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:53.018103 (+  1131us) log.cc:844] Serialized 26 byte log entry
0709 12:16:53.178106 (+160003us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:16:53.178153 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:53.178234 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:53.653354 (+475120us) raft_consensus.cc:1685] finished
0709 12:16:53.653390 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:53.653655 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3999,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":20851,"thread_start_us":159287,"threads_started":1,"wal-append.queue_time_us":203509}
I20260709 12:16:53.661600 23915 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:16:53.651321 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 141238us
I20260709 12:16:53.656486 24080 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:53.697161 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 232755us
I20260709 12:16:53.697103 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:16:53.699815 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:53.707463 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46860.7us
I20260709 12:16:53.709985 24209 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:16:53.711339 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:53.715734 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 262476us
W20260709 12:16:53.730646 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 214874us
W20260709 12:16:53.734483 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80329.5us
I20260709 12:16:53.727012 24080 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:53.729003 23915 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:53.750089 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:53.725108 23925 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.745119 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:16:53.754981 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 670289us
I20260709 12:16:53.740188 24080 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:53.745589 24164 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.762682 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:53.754326 24231 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:53.760078 24080 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:53.775555 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64396.3us
I20260709 12:16:53.770231 23917 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:16:53.780946 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:53.781562 24215 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:53.764839 24063 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:53.789106 24147 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:53.790616 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:53.800923 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.773948 23915 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.803752 12496 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:16:53.806118 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:53.805333 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:53.814079 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:53.841868 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77772.2us
I20260709 12:16:53.819901 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:53.778105 23925 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.825150 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:53.810832 24177 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:53.826335 13258 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:53.826774 24149 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:53.829149 24153 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:16:53.843009 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104829us
W20260709 12:16:53.843487 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64644.5us
I20260709 12:16:53.766230 24214 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260709 12:16:53.853736 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:53.858897 24231 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:53.881927 23972 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:16:53.872267 12639 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:53.880726 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:53.885015 23917 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:53.887493 12646 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:53.851954 24153 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:53.877923 24149 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.851545 24177 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:53.893558 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92745.7us
I20260709 12:16:53.913130 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.913916 12462 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38.
I20260709 12:16:53.915310 23929 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:53.916095 23929 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.917193 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:53.918159 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:53.919067 23605 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:53.919842 23605 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:53.920778 23605 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:53.921250 23605 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:53.921795 23855 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:16:53.922226 23855 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:53.923192 23855 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.924515 23855 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:53.896790 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:16:53.928025 23929 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:53.915373 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:53.910144 12642 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:53.902201 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48636.4us
W20260709 12:16:53.906441 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73446.9us
I20260709 12:16:53.928426 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.905184 12494 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:53.941036 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:53.945180 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:53.903951 24207 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:53.950344 23659 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:53.949983 24207 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.950976 23659 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:53.951499 23659 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:53.946516 24164 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.952663 24164 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.888489 23915 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:53.954588 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:53.955644 23972 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:16:53.956204 12462 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:53.929823 24179 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:53.907743 24103 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:53.958567 13253 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:53.958036 24179 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.906952 24149 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:53.959430 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:53.958725 23853 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:53.901517 23697 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:16:53.811007 24183 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:53.877956 24063 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.930200 24205 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:53.900408 23917 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.18, Last appended by leader: 18, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:53.943485 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:53.846305 24147 consensus_queue.cc:1243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 32.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
W20260709 12:16:54.010442 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49109.4us
W20260709 12:16:53.998446 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83163.8us
W20260709 12:16:54.017879 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53307.6us
I20260709 12:16:53.993355 23972 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:53.902868 24153 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:53.906140 24177 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:53.996500 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:54.018798 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.070272 12609 leader_election.cc:400] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:16:54.071116 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.044771 12643 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:54.079891 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.073933 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:16:54.079017 12609 leader_election.cc:403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
W20260709 12:16:54.083184 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:54.084885 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:53.958037 24103 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:53.992796 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:54.088429 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 151324us
I20260709 12:16:54.093269 12645 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:54.036839 23853 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:53.993707 24149 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.102s
I20260709 12:16:54.096300 24147 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:54.096642 24150 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:54.096931 24172 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:53.998850 23947 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:54.002568 24179 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.096968 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:54.069933 23915 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:54.030853 24175 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:16:53.995738 24164 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:54.095059 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102426us
I20260709 12:16:54.097313 24091 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:54.102808 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:54.111011 13253 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:16:54.111814 13253 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 23: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:54.112416 12461 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:54.112816 12461 leader_election.cc:403] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:16:53.910004 24215 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.042421 23972 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:54.119635 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:54.113773 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:54.092566 24103 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:54.099886 23853 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:53.988775 24162 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.165s
I20260709 12:16:54.101884 24080 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.004299 24171 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.127303 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.110927 24214 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.104765 23929 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:16:53.964329 24085 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.082032 24205 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.110376 24091 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:16:54.136780 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 97344.9us
I20260709 12:16:54.110965 24147 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:54.111358 24175 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:54.140210 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173111us
I20260709 12:16:54.124439 23972 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:54.161197 12564 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:54.014817 23605 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.134078 24170 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:54.132920 24003 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.120899 23915 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:54.180859 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109970us
I20260709 12:16:54.158218 24091 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:16:54.184283 24060 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.530s	user 0.008s	sys 0.006s
I20260709 12:16:54.186161 12572 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: [40.22-40.22]
I20260709 12:16:54.186388 24173 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:54.154881 24150 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35.
I20260709 12:16:54.143529 23697 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:54.194135 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.193724 24173 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:54.201012 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104128us
I20260709 12:16:54.187613 24162 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.212013 12642 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:16:54.213137 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 144728us
I20260709 12:16:54.175069 24214 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:54.251269 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.214864 23770 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.196413 24060 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:54.300799 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 173544us
I20260709 12:16:54.208527 23697 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.210237 24003 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:54.314365 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58940.1us
I20260709 12:16:54.218322 24162 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.288445 24215 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.315773 24183 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.298888 24080 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.345839 13258 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:54.265575 24138 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
W20260709 12:16:54.351280 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3668) took 2220 ms (client timeout 2111 ms). Trace:
W20260709 12:16:54.351958 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178558us
I20260709 12:16:54.291323 24113 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
W20260709 12:16:54.353475 12646 rpcz_store.cc:269] 0709 12:16:52.130358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:52.709827 (+579469us) service_pool.cc:224] Handling call
0709 12:16:54.338833 (+1629006us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:54.351190 (+ 12357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2705632000}
I20260709 12:16:54.266013 23770 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.341118 24063 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.287216 23659 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:54.362064 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 171504us
I20260709 12:16:54.345595 24164 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.349880 24171 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:54.371907 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85505.8us
I20260709 12:16:54.266180 24147 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.361982 24085 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.284484 24175 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:54.390312 24258 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:16:54.394035 23915 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.398535 24060 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.399160 23697 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.400089 12568 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:54.379689 24164 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.400867 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.380014 23903 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:54.402616 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.431542 12563 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:54.403332 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.404042 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.404903 24214 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.406124 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:54.409008 24138 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:54.447880 24138 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.449457 24025 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:54.433562 24150 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.017s
I20260709 12:16:54.450294 24025 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.409140 23659 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.451366 24138 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.452733 24025 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:54.386626 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 197459us
I20260709 12:16:54.456887 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.457161 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.457883 12571 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 37 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:16:54.458230 23903 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:16:54.361445 24173 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.458998 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.459327 24173 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:54.459782 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:16:54.460793 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.462039 23853 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.464566 23853 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:54.465437 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59434.7us
I20260709 12:16:54.466351 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.466818 23915 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:16:54.467859 24091 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.468957 23915 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:16:54.469455 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:54.469830 24171 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:16:54.470098 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:54.386091 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:54.429003 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43739.1us
W20260709 12:16:54.417932 24255 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.230s	user 0.002s	sys 0.000s
I20260709 12:16:54.453333 24025 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:54.433879 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.433530 23855 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.446667 12572 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:54.446975 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.425498 23514 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.029s
I20260709 12:16:54.451033 24003 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.482497 13254 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:54.458352 24162 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.482738 24060 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:16:54.485086 24164 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.486070 23853 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.496829 12563 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 30 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
W20260709 12:16:54.514468 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128446us
I20260709 12:16:54.524864 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.525911 24162 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:54.519767 24025 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.526559 24162 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:54.528103 24207 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.527845 24162 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.528437 24025 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.529553 12495 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:16:54.530885 24091 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.531991 23915 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:54.532413 23915 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:54.524910 23853 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.533411 23915 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.534757 23915 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:54.535298 23853 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.537932 24107 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.538919 24107 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.541893 24101 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.542573 24101 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.544797 24101 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.545593 24109 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:16:54.547503 24172 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:54.547950 24172 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:54.548240 24153 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.550197 24215 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:54.550567 24215 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:54.551569 24103 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.557991 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:54.559121 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:54.563553 24003 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:16:54.564122 24003 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:54.565850 24001 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:54.566398 24001 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:54.567474 12494 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:54.567286 24001 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.496258 23855 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.571748 23855 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.572283 23855 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.572719 23855 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.579522 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:54.583334 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50277.4us
I20260709 12:16:54.584435 24271 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:16:54.587476 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.587879 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.588779 12563 raft_consensus.cc:2368] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:54.589243 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.589682 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.590106 12571 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:54.592502 24272 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.592562 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.593465 24272 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.594442 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.595353 12568 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:54.565171 24003 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.612272 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1683) took 1412 ms. Trace:
I20260709 12:16:54.613000 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1686) took 1409 ms. Trace:
I20260709 12:16:54.613394 12645 rpcz_store.cc:276] 0709 12:16:53.203466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.364959 (+161493us) service_pool.cc:224] Handling call
0709 12:16:54.612794 (+1247835us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:54.612943 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1641341824}
W20260709 12:16:54.614096 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.615525 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.616079 24205 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.616761 24205 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:54.617072 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:54.618762 24272 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.612756 12643 rpcz_store.cc:276] 0709 12:16:53.199981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.327858 (+127877us) service_pool.cc:224] Handling call
0709 12:16:54.612082 (+1284224us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:54.612201 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1329777408}
I20260709 12:16:54.590055 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.591919 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.548960 24172 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.621788 12572 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:54.625769 24272 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:16:54.628820 12641 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:16:54.635113 12571 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
W20260709 12:16:54.641654 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 173211us
I20260709 12:16:54.642050 12567 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 22. Current term is 23. Ops: []
I20260709 12:16:54.643479 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:54.648916 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:54.648896 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:54.651206 12495 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:54.653493 12571 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
W20260709 12:16:54.663964 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.661958 24107 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.801477 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.802001 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:54.808841 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 172976us
W20260709 12:16:54.810211 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.810489 23770 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.712478 12567 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:54.713984 12572 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 14. Preceding OpId from leader: term: 36 index: 15. (index mismatch)
I20260709 12:16:54.817435 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.716526 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.718818 24175 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:16:54.723465 23605 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:54.644073 24103 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.694650 24153 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:54.659098 24172 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:54.722919 23903 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:16:54.798583 12568 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 34 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 16.
I20260709 12:16:54.686862 23855 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.689869 24205 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:54.691061 12491 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:54.839069 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:54.840428 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:54.840977 12491 raft_consensus.cc:2376] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35. Re-sending same reply.
I20260709 12:16:54.648944 24177 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.843923 24177 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:54.732029 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.844758 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:54.687877 23972 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.001s
I20260709 12:16:54.846076 12566 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:54.846405 23972 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.848675 23972 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:54.810683 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.716252 23925 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:54.817261 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.818168 12569 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:54.852205 12495 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:16:54.832307 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.831140 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:54.853924 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.821838 24170 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:54.856135 12493 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:54.864323 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.864672 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.870263 12497 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:54.873368 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.865334 12491 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
W20260709 12:16:54.883095 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0720582689998537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260709 12:16:54.889429 12488 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
W20260709 12:16:54.892221 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0675582689998464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1755s for a response
I20260709 12:16:54.890064 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:54.894832 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.887570 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.904011 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.906036 12569 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:16:54.916651 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.916957 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.916712 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:54.865650 23915 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.923043 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.923635 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.918537 13253 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:54.911926 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:54.921348 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.925302 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:54.928364 12497 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:54.921944 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.916476 24028 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.017s
I20260709 12:16:54.933874 12497 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:54.933164 12494 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:54.935429 12497 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:54.936559 24028 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 34.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:16:54.937206 24028 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:54.937498 24272 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:54.937873 24028 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:54.938452 24272 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.939153 24028 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:54.940521 24028 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:54.941514 24272 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:54.945076 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.947136 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.947875 12570 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:54.918160 23605 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:54.952003 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.900393 24283 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:54.879364 24170 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.917878 24171 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:54.958403 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:54.937318 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:54.955531 12570 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:16:54.913594 24177 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.042s
I20260709 12:16:54.966065 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:54.963456 13261 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:54.950150 24153 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:54.953999 23540 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:54.954819 24172 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:54.984082 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127858us
I20260709 12:16:54.985146 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:54.979920 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:54.942621 23915 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:54.980859 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:54.993847 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:54.996068 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.996940 12493 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:54.998031 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.001199 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1682) took 1801 ms. Trace:
I20260709 12:16:55.001757 12637 rpcz_store.cc:276] 0709 12:16:53.199472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.486645 (+287173us) service_pool.cc:224] Handling call
0709 12:16:53.522320 (+ 35675us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:53.663003 (+140683us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:16:53.663379 (+   376us) log.cc:844] Serialized 14 byte log entry
0709 12:16:54.125182 (+461803us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:54.125246 (+    64us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:16:54.131621 (+  6375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:55.000890 (+869269us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:55.000944 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:55.001125 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6842,"prepare.queue_time_us":5836,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":8544,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":7458,"spinlock_wait_cycles":322861952,"thread_start_us":454375,"threads_started":1,"wal-append.queue_time_us":528375}
I20260709 12:16:55.035252 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.036144 12496 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:55.042670 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:55.061064 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.064515 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.064872 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4831) took 1522 ms. Trace:
W20260709 12:16:55.066754 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81699.4us
I20260709 12:16:54.974261 24153 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:54.980775 23540 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:55.071942 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:55.065488 12564 rpcz_store.cc:276] 0709 12:16:53.542153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.542285 (+   132us) service_pool.cc:224] Handling call
0709 12:16:53.542702 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:54.169314 (+626612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:55.064567 (+895253us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:55.064792 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421250432}
I20260709 12:16:54.998992 24215 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:55.092756 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110702us
I20260709 12:16:55.093971 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:54.979913 24094 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:54.979480 23605 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:16:55.101564 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120802us
I20260709 12:16:55.103722 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:55.107304 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.108255 12568 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:55.109512 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:55.109537 12568 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:55.101523 24215 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:55.039551 24138 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:54.978971 24171 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.039178 23855 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.069033 23929 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:55.125007 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.100126 24025 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.126405 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.099261 24153 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:55.129035 12494 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:55.034403 24172 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.104712 23917 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:55.138913 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:55.064049 24283 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.136243 24025 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.143374 24170 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:55.126866 24173 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:55.144773 12489 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 18. Preceding OpId from leader: term: 39 index: 19. (index mismatch)
I20260709 12:16:55.144636 12462 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:55.149154 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.153731 12462 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:16:55.134003 23697 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:55.128847 23770 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.165308 24272 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:55.170390 24025 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:55.174325 24290 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:16:55.178951 12566 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:16:55.191609 24172 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.192353 23540 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.137387 23855 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.156481 12609 leader_election.cc:400] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:55.213109 12609 leader_election.cc:403] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:16:55.214020 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57772.7us
W20260709 12:16:55.204581 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.197299 24171 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.122697 23605 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.216961 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:55.235399 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97451.1us
I20260709 12:16:55.232990 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.158280 24170 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:55.130868 24138 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.236723 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:55.243659 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.245481 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.191713 24162 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:55.210381 24272 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:55.239653 24205 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:55.254194 23605 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.254913 23855 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.252357 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:55.243090 24081 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:16:55.233942 24271 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:16:55.251402 23540 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:55.280535 23855 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:16:55.245208 23770 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.301739 24272 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.253489 24094 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.307260 12491 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:55.269860 24205 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.256978 24283 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:55.268566 24170 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:16:55.306247 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.924s (SENT) [suppressed 20 similar messages]
I20260709 12:16:55.315640 12571 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:16:55.317104 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69272.5us
I20260709 12:16:55.305079 24171 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:55.321879 12571 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:55.329363 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83979.9us
I20260709 12:16:55.318609 24094 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:55.321218 23929 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:55.335388 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.316319 23915 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.320905 24205 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:55.320427 24170 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.344587 23697 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.345449 23697 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.350445 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.351616 12570 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:55.349818 23697 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.326725 23855 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:16:55.393344 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55538.8us
I20260709 12:16:55.396859 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.339491 12571 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.400744 12571 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:16:55.413961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72694.5us
W20260709 12:16:55.417695 24292 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:16:55.420176 24299 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:55.420837 24299 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.421213 24299 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:55.422236 23855 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.393296 24170 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:55.416994 23929 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.427383 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:55.429618 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.443665 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:55.455394 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.455638 23770 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.456420 23770 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:16:55.464744 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65624.9us
I20260709 12:16:55.452934 23697 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:16:55.473491 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.470463 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:55.471943 12571 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 21. (term mismatch)
I20260709 12:16:55.468326 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.474712 12571 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:16:55.475409 12571 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 33.21
I20260709 12:16:55.475673 12571 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:55.478663 24177 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:55.478969 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.479676 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:16:55.483311 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.484094 12570 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
W20260709 12:16:55.484815 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57541.4us
I20260709 12:16:55.485570 24272 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.485956 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:55.486495 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.477912 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.490903 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:55.492116 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:55.493248 24028 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.550s	user 0.014s	sys 0.001s
I20260709 12:16:55.501847 24171 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:16:55.506798 23605 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:55.487486 24268 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:16:55.517530 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.521608 24173 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:55.544916 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.522358 12641 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 22. Preceding OpId from leader: term: 29 index: 22. (term mismatch)
I20260709 12:16:55.545645 12641 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 21
I20260709 12:16:55.545996 12641 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 28.22
I20260709 12:16:55.546274 12641 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:55.524153 12637 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:16:55.528425 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:55.550220 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:55.551208 24277 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:55.552124 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:55.529853 24304 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:16:55.552682 24272 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:55.524398 23605 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.553098 24272 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.553345 23605 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:55.553505 24272 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:55.555094 24177 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:16:55.537295 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.538117 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:55.525167 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.533743 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1456) took 1537 ms. Trace:
I20260709 12:16:55.556807 13258 rpcz_store.cc:276] 0709 12:16:53.995950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.996142 (+   192us) service_pool.cc:224] Handling call
0709 12:16:55.533456 (+1537314us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:55.533620 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438072192}
I20260709 12:16:55.535293 24299 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.504128 24172 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:55.558607 24172 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.559033 24172 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:55.560003 24299 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.517668 24081 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 38. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39.
I20260709 12:16:55.518352 24306 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:16:55.560971 24299 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.544952 24295 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:55.561364 24306 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.552018 24277 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:55.536218 24138 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.504781 24300 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:55.563778 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.572073 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:55.563666 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:55.575845 12571 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:55.556376 23540 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.561884 24283 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:55.583979 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.585279 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.589044 12571 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:16:55.590354 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.582674 24091 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:55.599227 24091 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.599659 24091 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:55.569145 24205 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.602599 12564 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:16:55.603922 24205 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:55.604454 12564 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 38 index: 19. (index mismatch)
I20260709 12:16:55.562410 23929 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:55.569932 24171 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:16:55.605822 24200 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.010s
I20260709 12:16:55.606354 24171 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:55.606846 24171 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:55.607817 24171 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.609208 24171 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:55.609283 23540 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.616711 12571 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:55.561883 24295 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.562021 24306 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:55.633145 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 141899us
I20260709 12:16:55.584553 24277 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:55.561565 24299 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.632396 24268 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.607070 24200 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:55.588085 24094 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.658812 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:55.665817 12641 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:55.568454 24172 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:55.649433 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91829.6us
I20260709 12:16:55.656721 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:55.651811 23929 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.617862 23540 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:55.663105 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74170.4us
I20260709 12:16:55.609963 24029 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.050s
I20260709 12:16:55.670099 12643 raft_consensus.cc:2376] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34. Re-sending same reply.
I20260709 12:16:55.645032 24295 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:55.669430 12494 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:16:55.670051 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:55.618557 24271 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:16:55.586182 24300 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.662606 24277 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.679108 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:55.664345 24200 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:55.688755 12645 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:55.689826 24081 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.693032 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.695190 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:55.696165 12566 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:55.698330 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.704417 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.662277 24153 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:55.708799 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.672902 24311 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.677361 23855 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.693415 24277 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:55.717120 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.679785 24138 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.709450 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.694342 23540 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.667914 24268 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:55.742326 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72365.7us
I20260709 12:16:55.700734 24312 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:55.743441 12570 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:55.746433 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:55.747849 12497 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:55.722851 24025 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.753052 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.692488 23929 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:55.765262 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63324us
I20260709 12:16:55.793830 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:55.700723 24309 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.060s
I20260709 12:16:55.688683 24272 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.799912 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:55.773941 24175 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.930s	user 0.013s	sys 0.002s
I20260709 12:16:55.797325 12494 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:55.732942 24138 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.710639 24081 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.801720 24170 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.722563 24298 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.073s
I20260709 12:16:55.689980 24200 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.734649 23855 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.799306 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:55.820399 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:55.831115 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:55.835665 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85521.1us
I20260709 12:16:55.840842 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:55.840544 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:55.751282 23540 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.771742 24313 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:55.796415 24321 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.740664 24311 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.755427 24091 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:55.866803 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.804008 24283 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:16:55.794920 24025 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.869737 24172 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.871133 24025 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.874167 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.885231 12571 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:16:55.888316 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1450) took 1420 ms. Trace:
I20260709 12:16:55.889904 24094 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.888844 13262 rpcz_store.cc:276] 0709 12:16:54.467764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:54.467919 (+   155us) service_pool.cc:224] Handling call
0709 12:16:55.533963 (+1066044us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:55.536766 (+  2803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:55.536799 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:55.538931 (+  2132us) log.cc:844] Serialized 26 byte log entry
0709 12:16:55.692764 (+153833us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:55.692810 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:55.692879 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:55.886912 (+194033us) raft_consensus.cc:1685] finished
0709 12:16:55.886941 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:55.887958 (+  1017us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400113fd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:55.888212 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2303,"raft.queue_time_us":7210,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":2450694016,"thread_start_us":140199,"threads_started":1,"wal-append.queue_time_us":256180}
I20260709 12:16:55.891901 24299 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:55.892642 24299 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.893780 23927 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.894452 23927 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.896764 23927 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.898207 24153 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.898413 23929 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.900075 23929 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.900724 23929 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:55.902523 24268 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.903000 23855 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.903602 24317 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:55.904076 23855 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:55.904229 24317 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.904793 23855 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.848256 24312 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:55.877864 24324 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:16:55.882433 24314 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:16:55.884186 24271 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:55.922501 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.875082 23540 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:55.913884 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.934473 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:16:55.849725 23915 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:55.915086 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.893257 24299 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.899984 24300 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:55.950075 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:55.914932 24312 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.929251 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.803215 24297 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.057s
I20260709 12:16:55.923738 24153 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.941071 24025 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:16:55.928740 24291 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.767s	user 0.002s	sys 0.002s
I20260709 12:16:55.921653 24172 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:16:55.902938 24138 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:55.937594 23915 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.916582 24028 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.048s
I20260709 12:16:55.918567 24313 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
I20260709 12:16:55.927320 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:55.964576 23540 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:16:55.964102 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114796us
W20260709 12:16:55.965256 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.966744 12566 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:16:55.916548 24324 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:55.924129 24271 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:55.966156 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:55.927019 12641 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:16:55.951018 12488 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:16:55.990996 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:55.856732 24321 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.956259 24283 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:55.965492 24312 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:55.938271 23605 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:56.005404 24305 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:16:55.986364 24172 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:56.096439 24321 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.097271 24321 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:56.096583 24172 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 37.15, Last appended by leader: 15, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.991492 23915 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:56.017692 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:56.018599 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52539us
W20260709 12:16:55.988270 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 194516us
I20260709 12:16:56.100312 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.028578 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.103407 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.001331 23540 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.031205 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:56.104216 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:55.971083 24153 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:55.993265 24317 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.035197 24334 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:56.036697 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:56.108217 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71609.9us
W20260709 12:16:56.037485 12536 leader_election.cc:336] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.111s (SENT)
I20260709 12:16:56.038373 24138 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:56.038223 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.039350 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.039992 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.040766 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.041051 12638 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:56.041499 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.046845 23929 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:56.047659 24091 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.130954 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.048167 24326 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.091s
I20260709 12:16:55.999524 24025 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.060501 24268 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:56.148401 24268 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.022480 24283 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.113996 23659 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:16:56.149858 24138 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.151337 24283 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:56.062449 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.152354 24138 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:56.152552 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90218us
I20260709 12:16:56.154508 24025 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:56.063676 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.155181 24025 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:56.155596 24025 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:56.155988 24025 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:56.064656 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.157200 13253 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:16:56.063464 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.148413 12637 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:56.160143 13258 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:16:56.065824 24308 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 12:16:56.107518 23540 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:56.067222 24324 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.17, Last appended by leader: 17, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:56.074689 24286 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.233s	user 0.002s	sys 0.008s
I20260709 12:16:56.081482 24299 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.166181 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:56.083978 23605 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:56.032212 24307 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.312s	user 0.002s	sys 0.000s
W20260709 12:16:56.168530 24285 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.201s	user 0.001s	sys 0.003s
I20260709 12:16:56.169723 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:56.170600 24282 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.340s	user 0.002s	sys 0.007s
I20260709 12:16:56.170811 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:16:56.172142 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1785) took 1649 ms. Trace:
I20260709 12:16:56.172782 24326 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:56.173273 24326 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:56.174430 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.175169 12564 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:56.176188 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.176819 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:56.090355 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.179627 12495 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:16:56.179859 24324 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:56.090658 23855 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.091936 24311 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.182989 24268 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.092469 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4860) took 1299 ms. Trace:
W20260709 12:16:56.183651 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.010993 24271 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:56.186311 24326 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.16, Last appended by leader: 16, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.105975 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.102209 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:56.190156 12639 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
W20260709 12:16:56.190230 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88111.9us
I20260709 12:16:56.192761 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:56.112502 13259 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:16:56.103801 12489 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:56.194618 24311 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:56.195084 24311 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.086297 24323 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:56.195814 24323 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:56.098536 24172 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:56.120662 12646 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:56.116729 12488 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:56.122323 24205 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:16:56.126708 24170 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.127358 24081 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.128868 24272 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.131778 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:56.136003 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:56.137219 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99802.7us
W20260709 12:16:56.216248 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.217402 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.217950 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.138818 24295 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.148514 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.061323 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:56.221889 13256 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:56.061602 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.155865 24313 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:56.159307 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.109274 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.224138 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:56.193450 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72787.9us
I20260709 12:16:56.192800 24200 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:56.195477 24311 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:56.117534 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:56.183907 12568 rpcz_store.cc:276] 0709 12:16:54.792702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:55.113414 (+320712us) service_pool.cc:224] Handling call
0709 12:16:55.113788 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:55.161561 (+ 47773us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:55.161614 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:55.161706 (+    92us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:16:55.161757 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:56.092185 (+930428us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:56.092236 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:56.092401 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109092608}
I20260709 12:16:56.118845 24334 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.210078 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:56.193429 24138 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:16:56.097906 24321 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.172559 12492 rpcz_store.cc:276] 0709 12:16:54.522937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:54.523114 (+   177us) service_pool.cc:224] Handling call
0709 12:16:54.525163 (+  2049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:54.525436 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:54.525478 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:54.543749 (+ 18271us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:16:54.544252 (+   503us) log.cc:844] Serialized 26 byte log entry
0709 12:16:54.763911 (+219659us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:54.763967 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:54.764060 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:56.167572 (+1403512us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:56.171886 (+  4314us) raft_consensus.cc:1685] finished
0709 12:16:56.171911 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:56.172076 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48639616,"thread_start_us":219154,"threads_started":1,"wal-append.queue_time_us":356390}
I20260709 12:16:56.139647 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.213008 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:56.188678 24326 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:56.259588 24313 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.287017 24318 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:56.288280 24313 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:56.288833 24171 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.656s	user 0.011s	sys 0.000s
I20260709 12:16:56.289816 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:56.207145 24283 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:56.246150 12563 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:56.253999 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.258244 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:56.258934 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 149760us
I20260709 12:16:56.300590 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:56.259950 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49982.8us
I20260709 12:16:56.260661 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:56.301918 12567 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:56.259094 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.266263 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:56.266448 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:56.304885 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.308071 12637 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:56.275820 24153 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.276198 13255 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:56.277209 24060 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:16:56.290539 13258 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:56.299365 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 23.
I20260709 12:16:56.321166 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:56.291453 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.264751 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:56.254142 23929 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:56.338011 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.298244 13261 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:56.211136 24317 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:56.305935 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:56.319339 12563 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.15-36.15]
I20260709 12:16:56.260578 24334 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:56.275862 24321 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.217412 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:56.298885 24138 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:56.343653 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.343654 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.290904 24170 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:16:56.374883 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:56.347685 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83077.9us
I20260709 12:16:56.384758 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:56.327705 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:56.367641 23659 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.019s
W20260709 12:16:56.388420 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.389810 24153 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:56.390326 24153 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.390728 24153 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:56.357016 24060 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:56.394258 24060 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.394666 24060 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:56.395771 12496 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.402117 24107 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:16:56.404764 24272 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:56.405349 24272 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:56.346773 24171 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:56.406142 24171 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:56.406345 24272 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 36.15, Last appended by leader: 15, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.407963 24272 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:56.407424 24171 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.410094 12646 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:56.379951 13257 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:56.411453 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:56.412437 13255 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:56.350653 12488 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:56.345396 23770 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:16:56.319298 24283 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.353619 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.354058 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.356143 24295 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:56.357573 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:56.433173 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:56.433714 13262 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:56.362504 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.329432 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:56.352689 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46899us
W20260709 12:16:56.437470 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108190us
I20260709 12:16:56.420149 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:56.434993 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 23.
I20260709 12:16:56.421010 12639 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:56.440508 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3634) took 1567 ms. Trace:
I20260709 12:16:56.441056 12491 rpcz_store.cc:276] 0709 12:16:54.873547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:54.911660 (+ 38113us) service_pool.cc:224] Handling call
0709 12:16:56.440291 (+1528631us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:56.440442 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":879714048}
I20260709 12:16:56.424952 24297 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:56.444926 24297 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.447661 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.447806 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.475147 12644 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:16:56.453898 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.477824 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:16:56.404438 24107 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:56.476972 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
W20260709 12:16:56.486331 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 195447us
I20260709 12:16:56.424857 13256 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:56.480798 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.122911 24322 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.336s
I20260709 12:16:56.457644 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:56.460090 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.491273 13253 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:56.435074 23770 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:56.492236 12495 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:16:56.459551 24171 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:56.491888 13259 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:16:56.491506 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.474910 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.523595 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.361228 24317 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.513038 23770 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:56.197098 24323 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:56.537520 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.526422 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.527293 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.539155 12572 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:56.540675 13254 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:56.288275 24318 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:56.530053 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129404us
I20260709 12:16:56.545433 12644 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 39 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
I20260709 12:16:56.556244 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.558931 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.564043 12641 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:56.559118 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:56.564762 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:56.535605 24295 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:56.570432 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.548089 24338 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:16:56.522751 24107 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.583798 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.572117 24333 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:16:56.589998 12644 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:56.574379 24313 raft_consensus.cc:2764] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:16:56.610009 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.569701 24322 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:56.574882 24329 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:16:56.615805 23915 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.000s
I20260709 12:16:56.627595 23915 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.629464 12564 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:56.562373 24330 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.054s
I20260709 12:16:56.602625 24333 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.599512 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:56.570250 23917 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:16:56.603915 24312 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.519s	user 0.013s	sys 0.000s
I20260709 12:16:56.624264 13258 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:16:56.655923 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56612.4us
I20260709 12:16:56.692358 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:56.697969 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 132530us
I20260709 12:16:56.694912 24295 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:16:56.707664 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.572572 24138 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 34.14, Last appended by leader: 14, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.642941 24170 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:56.693784 24333 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:56.698837 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:56.697306 24312 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:56.717207 24342 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:16:56.719312 24312 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.719754 23929 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:56.721168 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:56.722177 24341 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:16:56.723616 24283 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:56.731344 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.699415 23917 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:56.733944 23917 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.707994 24295 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.738080 24295 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:56.738797 24318 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:56.710312 24313 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:16:56.742321 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.743268 12639 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:56.744318 24297 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.744575 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.745321 12639 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:56.717360 24107 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.746408 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.746675 23659 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:56.747217 23659 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:16:56.751499 24277 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.960s	user 0.011s	sys 0.000s
I20260709 12:16:56.752089 24277 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.753664 12490 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:56.758704 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.759488 12643 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:56.760540 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.758836 12640 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 18. Preceding OpId from leader: term: 39 index: 19. (index mismatch)
I20260709 12:16:56.762938 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.766937 12640 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:56.769826 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.764925 24215 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:16:56.770249 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.772569 13262 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:16:56.776456 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.777951 13262 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 35 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:16:56.778716 12641 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:16:56.783931 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.784235 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.780097 23929 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:56.786226 23929 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.787012 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.771883 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:56.786244 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.788218 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.789402 12646 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:16:56.789760 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:56.712951 24340 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.350s	user 0.002s	sys 0.000s
I20260709 12:16:56.796711 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.690280 24060 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:56.780596 24172 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.583s	user 0.008s	sys 0.004s
I20260709 12:16:56.644234 24322 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:56.790993 23540 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.003s
W20260709 12:16:56.805528 12638 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.583s	user 0.007s	sys 0.000s
I20260709 12:16:56.821538 12638 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:56.799453 12643 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:16:56.793309 24311 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.005s
W20260709 12:16:56.794840 24200 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.002s
I20260709 12:16:56.827189 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.779556 24215 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:56.830186 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.831107 12638 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:56.791129 23929 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:56.838680 12567 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.009s
I20260709 12:16:56.845191 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:16:56.799263 24283 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:56.850572 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.850667 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:56.818416 23915 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:56.853960 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82269.8us
I20260709 12:16:56.854328 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.855207 12640 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:56.858024 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.834707 24312 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:56.829022 24081 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:16:56.862478 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:56.866127 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.867642 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.875772 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:56.870169 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:56.869451 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72702.1us
I20260709 12:16:56.871326 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:56.879580 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.851181 24283 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.850927 24317 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:56.883980 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:56.887259 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 194978us
I20260709 12:16:56.834448 24311 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.828429 23540 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.891849 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.890061 24283 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:56.911653 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.913514 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.914417 12638 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:56.912390 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:56.915510 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.920004 24295 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.874877 23915 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
W20260709 12:16:56.922179 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:56.899084 24277 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.925974 24060 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.886955 24215 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.927984 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.941143 12642 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:56.940994 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:56.930213 24025 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.749s	user 0.010s	sys 0.002s
I20260709 12:16:56.931324 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.909219 24138 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:16:56.932094 24332 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:16:56.934471 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46142.5us
I20260709 12:16:56.887140 24153 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:56.938653 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.948275 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:56.939512 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55701.4us
I20260709 12:16:56.942497 12646 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:16:56.942492 12638 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:56.942489 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:56.953389 12642 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:56.929030 23770 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.955319 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:56.935360 24326 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.657s	user 0.012s	sys 0.002s
I20260709 12:16:56.953532 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:56.953085 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.958830 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.947809 24332 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:56.961949 24268 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:56.962298 24338 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:56.969231 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:56.950021 24025 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:56.952787 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:56.970041 12646 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:56.934041 24362 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:56.972806 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:56.963099 24153 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:56.970036 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.945304 24295 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:56.947685 24001 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:56.978963 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88348.8us
I20260709 12:16:56.987061 12641 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:56.989395 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.989516 12642 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:56.992421 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.993327 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:56.995478 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:56.991163 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.997251 12568 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:56.975080 24060 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:16:56.998486 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:56.970973 23915 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:56.999909 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.002875 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.003127 12642 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:16:56.978312 24362 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:56.978531 24297 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:16:57.006773 12638 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:16:56.995592 24324 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.815s	user 0.011s	sys 0.009s
I20260709 12:16:56.983964 24326 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:57.033843 24170 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:16:56.990932 24354 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:56.990645 24003 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:57.035910 12642 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.17-34.17]
I20260709 12:16:57.036305 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.049299 12640 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:16:57.050395 12637 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.051880 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:57.037375 12638 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:57.009109 23770 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.038060 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:57.002122 24312 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:16:57.005486 23917 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
W20260709 12:16:57.040695 24272 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.632s	user 0.008s	sys 0.003s
W20260709 12:16:57.006816 24271 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.822s	user 0.003s	sys 0.011s
I20260709 12:16:57.044315 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:57.065021 13258 raft_consensus.cc:606] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.064752 24271 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:57.006363 24306 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.344s	user 0.007s	sys 0.009s
I20260709 12:16:57.065457 13258 raft_consensus.cc:614] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:57.065814 24306 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.067000 24329 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:16:57.068511 24306 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:57.005857 24364 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:57.049391 12497 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:57.025499 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:57.069558 24364 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:57.023963 24297 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.021958 24362 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:57.070276 24297 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:57.071113 12644 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:57.048704 24354 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:57.072715 24354 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:57.041581 24029 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.073557 23917 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:16:57.074105 23917 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.074043 24029 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:57.058286 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:57.075692 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:57.076226 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:57.076802 24029 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.028705 24295 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:57.069871 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:16:57.078852 13258 raft_consensus.cc:606] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:57.079253 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.080165 12492 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:57.005422 23915 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:57.083177 23915 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:57.011134 24311 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:57.060768 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:57.042801 24334 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.688s	user 0.009s	sys 0.009s
I20260709 12:16:57.028293 24153 raft_consensus.cc:2764] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20260709 12:16:57.061908 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.090271 24334 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.044039 24322 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:57.070310 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44861.3us
I20260709 12:16:57.060634 24312 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:57.092576 24312 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:57.094962 12645 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.096720 23540 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.093263 24334 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:16:57.118952 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54944.4us
I20260709 12:16:57.121532 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.056043 12642 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
W20260709 12:16:57.053982 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:57.124120 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:57.124548 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:57.124825 12642 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:16:57.017189 24081 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.126904 24081 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:57.127848 23697 raft_consensus.cc:993] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:57.129050 24268 raft_consensus.cc:1081] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:57.044587 24025 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.096366 24362 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.131685 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.099184 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:57.100059 24354 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.047902 24003 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.016669 24332 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 34.26, Last appended by leader: 26, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.141217 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:57.103056 24162 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:16:57.105615 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.223528 24162 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:56.998993 24001 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.224473 13255 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:16:57.132709 12646 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.134080 24025 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:16:57.135573 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.136193 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.136768 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.137218 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:57.226609 12568 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:16:57.137840 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.138751 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:57.142258 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:57.143121 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:57.116165 24356 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.146780 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:57.160135 24295 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:57.161041 24362 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.164004 24358 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:57.165436 24365 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:57.166421 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:57.166520 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.070724 24364 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.170358 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.170444 12564 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 14. Preceding OpId from leader: term: 37 index: 15. (index mismatch)
I20260709 12:16:57.160145 24003 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:57.140545 12642 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.106058 24170 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:16:57.191286 24271 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:57.194099 24171 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.672s	user 0.010s	sys 0.009s
I20260709 12:16:57.103863 24312 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.202106 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:57.210801 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.290205 12640 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.211653 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.214955 24323 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:57.222393 12563 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:16:57.293191 12644 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:57.223012 24153 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:57.224442 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.295099 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.295810 12645 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:57.296146 12643 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.074810 23917 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:16:57.148314 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:57.230726 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 112974us
I20260709 12:16:57.226500 13260 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39. Re-sending same reply.
W20260709 12:16:57.233515 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98041.6us
W20260709 12:16:57.236219 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95099.2us
I20260709 12:16:57.299011 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:57.299503 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:57.252578 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105937us
I20260709 12:16:57.253273 12492 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:57.264688 13258 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:16:57.221230 24332 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:16:57.300668 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 176243us
I20260709 12:16:57.237947 24209 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:16:57.291868 12571 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.251284 24355 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:57.222525 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3652) took 1203 ms. Trace:
I20260709 12:16:57.298148 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:57.284963 24295 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:16:57.260170 24354 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.309409 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.221201 24333 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.319489 12568 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 24.
I20260709 12:16:57.305665 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.310750 12497 rpcz_store.cc:276] 0709 12:16:56.019208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:56.113689 (+ 94481us) service_pool.cc:224] Handling call
0709 12:16:57.222273 (+1108584us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:57.222441 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1916837120}
I20260709 12:16:57.315196 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:57.237712 24356 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.326911 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.328750 24091 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:16:57.329262 24091 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.329615 24091 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:57.320822 24322 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:57.331606 23659 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.332943 23659 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:57.279220 24365 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.311477 24272 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:57.291721 24362 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:16:57.279443 24358 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.311631 24001 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:57.317852 24295 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.342084 24295 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:57.324216 24355 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:57.346807 24355 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.347397 24283 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.348664 12570 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:16:57.351364 24283 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.352180 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:57.361411 24162 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.326764 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:16:57.375914 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49261.1us
I20260709 12:16:57.376498 12565 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:57.377337 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:57.365856 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.368844 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:57.367769 23659 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:16:57.378952 23659 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:57.368254 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.379201 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:16:57.379884 12568 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:16:57.368460 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62910.7us
I20260709 12:16:57.369903 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.381196 12639 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:57.370208 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.371302 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:57.383340 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.384059 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:57.388517 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.388818 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:57.316351 24171 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:16:57.316198 24029 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:57.390421 12565 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:16:57.393728 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.403409 12494 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:16:57.394938 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.404716 12642 raft_consensus.cc:2376] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36. Re-sending same reply.
I20260709 12:16:57.398970 12567 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:16:57.399395 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.406056 12495 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:57.396224 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.407661 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.408037 13255 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:16:57.401185 24324 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:57.409920 24060 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:16:57.410573 24060 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:57.411621 24060 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.417827 12642 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.419271 12642 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:16:57.420526 12642 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 28. Current term is 29. Ops: []
I20260709 12:16:57.421481 24356 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.422188 24318 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:57.422859 12642 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:16:57.422760 24318 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.423875 12495 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:16:57.425972 24215 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:16:57.425019 24318 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.427115 24318 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.427703 24318 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.430505 24318 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.430887 12640 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:16:57.431303 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:16:57.443699 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.375970 24209 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:57.472959 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.391163 24354 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:16:57.437080 23659 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.380661 24306 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:16:57.485148 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.524354 12495 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:57.486698 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:57.487421 24367 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:16:57.488976 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.396507 24172 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:16:57.489614 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.545435 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.506489 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.507180 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.489594 24162 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:16:57.401669 24323 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.508260 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:57.561409 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:57.510033 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.562376 12646 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.511247 12645 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:60230
I20260709 12:16:57.512286 23915 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.400379 24171 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:57.513278 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.589752 12641 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:16:57.499887 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105250us
I20260709 12:16:57.518798 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.519778 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:57.521960 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:57.522575 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 378913us
I20260709 12:16:57.522436 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.522329 24328 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.382807 23540 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.107s
I20260709 12:16:57.595717 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:57.596112 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.596849 12496 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:16:57.528672 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.530545 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.531176 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.401808 24025 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:57.547801 12494 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.557809 12491 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:57.488242 24365 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.563248 12645 raft_consensus.cc:606] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:57.603014 12645 raft_consensus.cc:614] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:16:57.494526 12563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:16:57.603986 12563 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:16:57.604472 12563 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:57.599938 24171 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:57.588997 12643 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: [35.25-35.25]
I20260709 12:16:57.547571 24354 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:57.591718 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.612462 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:57.591795 12565 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:16:57.514536 24029 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:57.527547 12642 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:16:57.602469 12495 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:57.604065 12489 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:57.572988 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.567946 24271 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.568799 24306 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:57.622949 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:57.607895 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85588.7us
I20260709 12:16:57.562773 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:57.628734 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.630082 12461 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:57.630599 12461 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:16:57.587872 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:57.521903 24001 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.075s
I20260709 12:16:57.618820 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.633584 12490 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:57.631860 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:57.613687 24350 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:16:57.642460 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79766.8us
I20260709 12:16:57.611209 24366 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:57.645872 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:57.573828 24323 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.615898 24025 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:57.659292 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.660588 12643 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:16:57.593464 24355 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:16:57.693934 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:16:57.680505 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 135312us
I20260709 12:16:57.689678 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.607726 24091 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.641705 24365 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:16:57.654398 12563 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:57.684253 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:57.722607 12639 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
W20260709 12:16:57.691197 24367 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:16:57.663094 24317 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:16:57.702360 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:57.661969 24209 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
W20260709 12:16:57.702725 24373 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:16:57.734715 12563 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.734995 12492 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:57.735641 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.736406 12639 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39. Re-sending same reply.
I20260709 12:16:57.703238 24375 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.737844 24375 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.704236 24306 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.706601 24323 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.706858 24283 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.708436 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.745452 12642 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:57.713341 24358 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:57.690263 23972 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:16:57.715073 23915 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.716434 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.722168 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.610142 24328 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.749473 12646 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.666355 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.721369 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.683499 24325 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:57.733163 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.733850 24091 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.737304 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.762405 12639 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:57.762518 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.763196 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.763408 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.763820 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.764397 24025 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.16, Last appended by leader: 16, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:57.700119 24081 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.573s	user 0.006s	sys 0.005s
I20260709 12:16:57.766259 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:57.757581 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91315.3us
I20260709 12:16:57.722003 24365 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:16:57.760424 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.763902 12639 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:57.767738 12461 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39.
I20260709 12:16:57.768721 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.629509 24271 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.730407 24317 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:57.663261 24312 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.696979 23697 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:16:57.782997 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:57.784396 12645 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.795903 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:57.796825 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:16:57.765770 24297 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.695s	user 0.011s	sys 0.001s
I20260709 12:16:57.789356 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:57.783905 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:57.794065 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.846087 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.825901 12645 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:57.849059 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:57.851883 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57941.8us
I20260709 12:16:57.792378 24325 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:57.747758 24283 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.759649 24295 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.870822 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.874817 12646 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:16:57.880671 12646 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:16:57.791708 24358 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:57.858726 12642 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:57.857522 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:16:57.868044 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:57.869076 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:57.803864 23915 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:16:57.783532 24172 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:57.804332 24209 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:57.792260 23697 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:57.791817 23972 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:16:57.737851 24029 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:57.907111 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 294712us
I20260709 12:16:57.935958 12641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:60230
I20260709 12:16:57.938983 12641 raft_consensus.cc:606] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:16:57.866959 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 180020us
W20260709 12:16:57.952068 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84149.9us
I20260709 12:16:57.788831 24091 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:57.954594 12496 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:57.958192 23659 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:16:57.749989 24354 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:57.788967 24317 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:16:57.963662 23659 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:57.858867 23540 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:58.026700 12646 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: [39.19-39.19]
I20260709 12:16:57.845806 24297 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:57.951411 24209 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:58.016369 23659 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:57.991510 24003 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.702s	user 0.009s	sys 0.004s
I20260709 12:16:57.936117 23915 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:57.960803 24366 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:16:58.041474 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77450.2us
I20260709 12:16:57.950623 23972 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:57.955025 24375 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:57.950393 23697 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:16:57.783964 24081 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.134891 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:58.153229 12493 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:16:58.155349 12488 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:16:58.166926 13256 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 24: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:16:57.935365 24365 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.135103 23540 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.137207 24209 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:58.148018 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:57.868542 24325 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.140196 23915 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:16:58.204329 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56492.3us
W20260709 12:16:58.206306 12644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.016s	sys 0.029s
W20260709 12:16:58.207036 12644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.016s	sys 0.029s
I20260709 12:16:58.207325 12563 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:16:58.207894 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:58.211686 12563 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:16:58.256865 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:16:58.257483 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 390289us
W20260709 12:16:58.338128 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130286us
I20260709 12:16:58.286883 12490 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 15. Preceding OpId from leader: term: 37 index: 15. (term mismatch)
I20260709 12:16:58.339228 12490 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 14
I20260709 12:16:58.337090 13255 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:58.339773 12490 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 36.15
I20260709 12:16:58.340247 12490 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:16:58.339263 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:16:58.262625 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 327065us
I20260709 12:16:58.340108 13255 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:16:58.273276 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 331227us
I20260709 12:16:58.344195 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:58.350263 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:58.290208 24328 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.348582 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:16:58.312960 24378 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.516s	user 0.000s	sys 0.007s
W20260709 12:16:58.213119 24162 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.003s
W20260709 12:16:58.339547 24306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.006s	sys 0.017s
W20260709 12:16:58.339330 24364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.004s	sys 0.020s
W20260709 12:16:58.211902 24331 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.007s	sys 0.017s
I20260709 12:16:58.179267 24365 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:58.362576 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:16:58.322430 24382 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:58.182931 24295 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:58.169731 24383 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:58.319177 24271 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:58.290138 24375 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:58.149873 24003 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.320966 24397 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:58.376377 13255 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.449011 13255 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:58.298195 24318 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260709 12:16:58.174067 23972 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.213131 24215 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.188s
I20260709 12:16:58.406706 24162 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:58.384526 24297 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:58.413968 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51501.6us
I20260709 12:16:58.454730 13259 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]
W20260709 12:16:58.392066 24306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.007s	sys 0.018s
W20260709 12:16:58.395555 24364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.005s	sys 0.022s
I20260709 12:16:58.415858 24283 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:58.435817 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45710.8us
I20260709 12:16:58.393596 24391 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:58.401744 24384 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:58.245779 24172 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.454770 24398 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:16:58.396070 24331 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.011s	sys 0.019s
I20260709 12:16:58.468770 24215 consensus_queue.cc:1243] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 36.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:16:58.445951 24375 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.501423 24081 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:58.275110 24366 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.164s
I20260709 12:16:58.434379 24382 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.445544 24383 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.489336 24283 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:16:58.512939 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 168855us
I20260709 12:16:58.513590 23659 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:58.493477 24325 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:58.515439 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166948us
I20260709 12:16:58.506327 24398 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:58.530952 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:58.506029 24384 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:58.508031 24381 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:16:58.535930 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:58.562860 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:58.486624 24162 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:58.512198 23917 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.204s	user 0.015s	sys 0.000s
I20260709 12:16:58.569815 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:58.527630 23659 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.455765 24397 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:58.514465 24322 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.796s	user 0.003s	sys 0.011s
I20260709 12:16:58.571005 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:58.571745 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:58.570020 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:58.514474 24215 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:16:58.576149 12492 raft_consensus.cc:2368] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:16:58.576303 12497 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:16:58.579304 12489 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:58.489533 24391 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.568944 24283 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:58.561043 24398 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:16:58.577858 24322 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.003s	sys 0.012s
I20260709 12:16:58.574462 24407 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:58.584642 24171 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.965s	user 0.008s	sys 0.005s
I20260709 12:16:58.597607 12493 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
W20260709 12:16:58.590195 24332 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.279s	user 0.011s	sys 0.003s
W20260709 12:16:58.588874 24388 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:16:58.581297 24384 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:16:58.606312 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118308us
W20260709 12:16:58.630492 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:58.620755 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:58.632241 12536 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:58.632766 12536 leader_election.cc:403] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:16:58.577149 23917 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.599174 24405 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:58.639676 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:58.604264 24407 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.707540 13255 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:16:58.711540 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90948.6us
I20260709 12:16:58.709867 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:16:58.718489 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78949.3us
I20260709 12:16:58.719954 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:58.742425 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:58.658223 24396 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:16:58.751930 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:58.661230 24408 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 12:16:58.659312 24385 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:16:58.746306 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:58.760787 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:58.747020 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:58.762086 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
W20260709 12:16:58.648574 24394 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
W20260709 12:16:58.743199 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104202us
I20260709 12:16:58.742161 24322 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:16:58.652664 24377 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.969s	user 0.005s	sys 0.007s
I20260709 12:16:58.642786 24354 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:58.658529 24393 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:16:58.604283 24171 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.654146 24215 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:16:58.648192 24411 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 33
W20260709 12:16:58.660037 24386 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
I20260709 12:16:58.661808 24003 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.657204 23917 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.654975 24391 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:58.653520 24333 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:58.718883 24364 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:58.652963 24382 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.772359 24419 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:16:58.718310 24366 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:58.750401 24332 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:16:58.771275 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:58.771658 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:58.751157 24060 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:16:58.765672 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 285882us
I20260709 12:16:58.615649 24410 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.152s
I20260709 12:16:58.772415 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:58.853111 12564 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:58.754913 24355 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:58.787989 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:58.757783 24356 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:58.792168 12568 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:16:58.795977 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:58.784192 24392 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:16:58.653842 24405 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.809437 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:58.808202 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:16:58.795822 24091 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.768s	user 0.006s	sys 0.007s
I20260709 12:16:58.770011 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:58.780464 24399 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:16:58.794353 24025 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.006s	sys 0.028s
I20260709 12:16:58.877435 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:58.758533 24283 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.831440 24402 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:58.842998 24419 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:58.788260 24354 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.834986 24375 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.835268 24407 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.842018 24391 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:16:58.799695 24172 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:58.838718 24383 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:58.838856 24153 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.541s	user 0.015s	sys 0.000s
I20260709 12:16:58.840039 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1706) took 1348 ms. Trace:
I20260709 12:16:58.879400 24091 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:16:58.846582 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:16:58.851819 13218 connection.cc:256] RPC call timeout handler was delayed by -1.6881322239999008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
I20260709 12:16:58.865190 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:58.876494 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:58.663313 24384 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.830791 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2271) took 1111 ms. Trace:
W20260709 12:16:58.916775 13218 connection.cc:256] RPC call timeout handler was delayed by -1.6035322239999914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
I20260709 12:16:58.893096 24332 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:16:58.878436 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56246.4us
I20260709 12:16:58.883598 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:58.884892 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:58.885903 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76570us
I20260709 12:16:58.889518 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4766) took 1372 ms. Trace:
I20260709 12:16:58.834390 24412 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:58.890691 12637 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 33 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:16:58.894207 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.062057 24365 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:58.894632 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.065317 24365 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.067569 24365 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.034396 24171 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:58.862552 24330 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:59.086726 12646 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:59.086978 12564 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:16:58.878062 24025 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.006s	sys 0.030s
I20260709 12:16:58.878973 24162 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:58.846167 12565 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:58.905941 12640 rpcz_store.cc:276] 0709 12:16:57.491224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.584302 (+ 93078us) service_pool.cc:224] Handling call
0709 12:16:57.584830 (+   528us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:57.585166 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:57.585204 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:57.608680 (+ 23476us) log.cc:844] Serialized 26 byte log entry
0709 12:16:57.866760 (+258080us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:16:57.866805 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:57.866892 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:58.839784 (+972892us) raft_consensus.cc:1685] finished
0709 12:16:58.839808 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:58.839972 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":251533,"threads_started":1,"wal-append.queue_time_us":864183}
I20260709 12:16:58.911397 12493 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 32 because replica has last-logged OpId of term: 30 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:16:58.919458 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:59.093256 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:58.890694 24060 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:58.920877 12563 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:16:58.859305 24366 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.888710 24399 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:58.905576 24397 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:58.893711 23917 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:58.915035 12572 rpcz_store.cc:276] 0709 12:16:57.718867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.719091 (+   224us) service_pool.cc:224] Handling call
0709 12:16:57.719401 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:57.719757 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:57.719795 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:57.727804 (+  8009us) log.cc:844] Serialized 26 byte log entry
0709 12:16:58.192030 (+464226us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:16:58.192098 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:58.192201 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:58.830457 (+638256us) raft_consensus.cc:1685] finished
0709 12:16:58.830484 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:58.830696 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":273098,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":115554,"thread_start_us":457428,"threads_started":1,"wal-append.queue_time_us":719549}
W20260709 12:16:58.998076 24428 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.151s	user 0.001s	sys 0.001s
W20260709 12:16:58.981619 24423 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:16:58.919305 12566 rpcz_store.cc:276] 0709 12:16:57.517496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.517672 (+   176us) service_pool.cc:224] Handling call
0709 12:16:57.518070 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:57.523330 (+  5260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:57.523377 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:57.561872 (+ 38495us) log.cc:844] Serialized 26 byte log entry
0709 12:16:57.978631 (+416759us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:16:57.978702 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:57.978828 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:58.889246 (+910418us) raft_consensus.cc:1685] finished
0709 12:16:58.889269 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:58.889447 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4736,"raft.queue_time_us":7048,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":25196,"thread_start_us":355152,"threads_started":1,"wal-append.queue_time_us":780594}
I20260709 12:16:58.978440 24382 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:58.999363 24419 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.19, Last appended by leader: 19, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:58.920341 24153 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:59.046840 12567 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:16:59.120060 23917 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:59.136271 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:59.136744 23917 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.138062 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.141147 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3733) took 1375 ms. Trace:
I20260709 12:16:59.142277 12637 rpcz_store.cc:276] 0709 12:16:57.765765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.793876 (+ 28111us) service_pool.cc:224] Handling call
0709 12:16:59.140924 (+1347048us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:59.141074 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2387077376}
I20260709 12:16:59.143831 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.144448 12637 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:59.152051 12643 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:16:59.153295 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:59.158397 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:59.167414 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1711) took 1657 ms. Trace:
I20260709 12:16:59.175686 12645 rpcz_store.cc:276] 0709 12:16:57.509921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.870693 (+360772us) service_pool.cc:224] Handling call
0709 12:16:57.873929 (+  3236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:58.182568 (+308639us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:58.182614 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:58.182662 (+    48us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:58.182704 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:59.166997 (+984293us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:59.167085 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:59.167316 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":709172992}
I20260709 12:16:59.180495 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:16:59.058480 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79706.5us
W20260709 12:16:58.900652 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92753.8us
I20260709 12:16:59.183347 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:58.839960 24364 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.056850 24172 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:59.091850 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.186461 24172 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.187068 12646 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:59.187881 24364 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:16:59.188967 24172 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.189716 24333 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.190387 24333 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.192911 24333 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.093991 12640 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:16:59.098234 24060 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.098481 24330 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.100418 24323 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:16:59.117291 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4930) took 1245 ms. Trace:
I20260709 12:16:59.216737 12490 rpcz_store.cc:276] 0709 12:16:57.871810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.872002 (+   192us) service_pool.cc:224] Handling call
0709 12:16:57.873373 (+  1371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:58.355849 (+482476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:59.106975 (+751126us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:59.117187 (+ 10212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917530496}
W20260709 12:16:59.097272 23697 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.912s	user 0.007s	sys 0.007s
I20260709 12:16:59.185225 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:59.116560 24420 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:16:59.196830 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.207581 12570 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
W20260709 12:16:59.116382 24431 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:16:59.111471 23915 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.849s	user 0.014s	sys 0.010s
I20260709 12:16:59.194336 24383 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.170817 24354 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:59.202296 24209 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.019s	user 0.015s	sys 0.000s
I20260709 12:16:59.243379 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:16:59.215332 24323 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:59.197853 24329 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.150131 24382 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.276513 24358 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.296784 24358 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.299912 24358 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:59.302726 24398 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.699s	user 0.012s	sys 0.000s
I20260709 12:16:59.303143 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.303544 24329 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.289697 24209 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:59.308632 24317 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.283s	user 0.012s	sys 0.003s
I20260709 12:16:59.309221 24317 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.311772 24317 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.318672 24399 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.319645 12488 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:59.320395 12494 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:16:59.324872 23972 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:16:59.330222 24325 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:16:59.330904 24325 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:59.332145 24325 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.335076 24153 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:59.335747 24153 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:59.342192 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1697) took 2237 ms (client timeout 1542 ms). Trace:
W20260709 12:16:59.342715 13256 rpcz_store.cc:269] 0709 12:16:57.105112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.105290 (+   178us) service_pool.cc:224] Handling call
0709 12:16:59.341921 (+2236631us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b54004b73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:59.342085 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2158397696}
I20260709 12:16:59.347278 13260 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:16:59.349112 24389 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.990s	user 0.001s	sys 0.002s
W20260709 12:16:59.349913 12393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 227) took 167708 ms (client timeout 15000 ms). Trace:
W20260709 12:16:59.350597 12393 rpcz_store.cc:269] 0709 12:14:11.641384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:11.675708 (+ 34324us) service_pool.cc:224] Handling call
0709 12:16:59.349699 (+167673991us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2123786,"reported_tablets":51}
I20260709 12:16:59.358183 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:59.359181 12563 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:16:59.273275 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88233.8us
I20260709 12:16:59.090528 24412 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.208854 24312 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:16:59.286136 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89440.6us
I20260709 12:16:59.366974 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:16:59.362147 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.035s	sys 0.028s
I20260709 12:16:59.367517 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:16:59.367848 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.035s	sys 0.028s
I20260709 12:16:59.296105 24381 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:16:59.282110 24433 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:16:59.270078 23697 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:16:59.398041 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:59.382149 24268 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.012s
I20260709 12:16:59.395488 24400 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:16:59.394065 24001 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:16:59.392789 23540 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.390985 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.373294 24325 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:16:59.285068 23915 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.288089 24383 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.303288 24398 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.389333 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.415484 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1703) took 1925 ms. Trace:
I20260709 12:16:59.416057 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.381402 24312 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:59.413117 12567 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 24: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:59.386497 24370 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:59.422909 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:16:59.425813 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.429593 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.434312 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.415899 12644 rpcz_store.cc:276] 0709 12:16:57.490503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.567576 (+ 77073us) service_pool.cc:224] Handling call
0709 12:16:57.602356 (+ 34780us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:57.611673 (+  9317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:57.611702 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:57.615999 (+  4297us) log.cc:844] Serialized 26 byte log entry
0709 12:16:58.207645 (+591646us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:16:58.440636 (+232991us) log.cc:844] Serialized 14 byte log entry
0709 12:16:58.441117 (+   481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:58.441207 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:59.348429 (+907222us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:16:59.403935 (+ 55506us) raft_consensus.cc:1685] finished
0709 12:16:59.415226 (+ 11291us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:59.415411 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32598,"prepare.queue_time_us":373669,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5754,"raft.queue_time_us":12916,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":22389,"spinlock_wait_cycles":43890432,"thread_start_us":698815,"threads_started":2,"wal-append.queue_time_us":662205}
I20260709 12:16:59.406503 24400 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:59.412377 24329 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.440481 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.450242 12640 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:16:59.412971 24429 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:16:59.446121 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.446767 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.456362 12637 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:59.432003 24354 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:16:59.426011 24172 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election decision vote started in defunct term 36: won
I20260709 12:16:59.498339 12563 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:16:59.452847 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.438843 24209 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:59.501322 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 581915us
I20260709 12:16:59.502759 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:16:59.453751 24153 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.411228 24380 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.045s
I20260709 12:16:59.403532 24283 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 37.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:16:59.430809 23917 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:16:59.464337 24437 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:16:59.526521 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:59.526548 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:16:59.465315 24439 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:16:59.525924 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.467159 23697 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.473595 24405 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.480530 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1695) took 2454 ms. Trace:
I20260709 12:16:59.481145 23659 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.481798 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:16:59.483755 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4894) took 2114 ms (client timeout 1772 ms). Trace:
I20260709 12:16:59.491070 24091 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:16:59.493288 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.493906 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.494176 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.494809 24081 raft_consensus.cc:993] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:59.494638 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.495559 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.439007 24025 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:16:59.441845 24434 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:16:59.499840 12646 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:59.460160 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:16:59.449594 24411 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.645s	user 0.008s	sys 0.006s
I20260709 12:16:59.507735 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.609879 12563 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:16:59.510273 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.609947 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.512530 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.611325 12637 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:16:59.512521 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:59.437413 24384 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 12:16:59.516220 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.517096 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:59.455669 24429 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:59.462548 24354 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:59.627350 12566 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:16:59.519120 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.628623 12490 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:16:59.522467 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:16:59.630437 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108116us
I20260709 12:16:59.630743 12564 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:16:59.631420 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:16:59.460877 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57354.4us
I20260709 12:16:59.502322 24391 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:16:59.524772 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.646512 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.525148 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.525719 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.530153 13257 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:16:59.506325 24317 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.540709 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:16:59.609102 12567 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:16:59.609169 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.514356 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.610643 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:16:59.518697 24355 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:59.694401 24355 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.696801 24355 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.698240 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1471) took 1036 ms. Trace:
I20260709 12:16:59.699427 24323 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.700178 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.701260 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:16:59.702330 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.702492 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:16:59.703617 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:16:59.705255 12637 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:16:59.706809 24398 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.712589 24215 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:16:59.713093 24215 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:59.714812 24419 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:59.722573 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4892) took 2355 ms. Trace:
I20260709 12:16:59.723026 12570 rpcz_store.cc:276] 0709 12:16:57.367555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.367966 (+   411us) service_pool.cc:224] Handling call
0709 12:16:57.397366 (+ 29400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:59.278462 (+1881096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:59.722328 (+443866us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:59.722505 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4153889152}
I20260709 12:16:59.724311 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:16:59.730352 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1704) took 2239 ms (client timeout 2234 ms). Trace:
W20260709 12:16:59.730844 12638 rpcz_store.cc:269] 0709 12:16:57.490755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.500884 (+ 10129us) service_pool.cc:224] Handling call
0709 12:16:59.730051 (+2229167us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:59.730235 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2155781376}
I20260709 12:16:59.734201 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.687021 24441 raft_consensus.cc:1081] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:16:59.738478 24382 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.739624 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.740109 24412 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:16:59.741894 24443 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:16:59.742071 24370 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:16:59.742635 24370 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:16:59.743508 24370 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.744835 24370 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:59.686321 12642 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:16:59.751359 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1713) took 2240 ms. Trace:
I20260709 12:16:59.690531 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.752848 13255 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:16:59.698935 13258 rpcz_store.cc:276] 0709 12:16:58.662387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:58.727270 (+ 64883us) service_pool.cc:224] Handling call
0709 12:16:58.756641 (+ 29371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:58.863309 (+106668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:16:58.863345 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:16:58.876044 (+ 12699us) log.cc:844] Serialized 26 byte log entry
0709 12:16:58.882406 (+  6362us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b48000f00d8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:16:59.086095 (+203689us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:16:59.086153 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:59.086233 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:16:59.697750 (+611517us) raft_consensus.cc:1685] finished
0709 12:16:59.697786 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:16:59.698156 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":392,"raft.queue_time_us":36868,"raft.run_cpu_time_us":2630,"raft.run_wall_time_us":91195,"spinlock_wait_cycles":6198016,"thread_start_us":315636,"threads_started":2,"wal-append.queue_time_us":345260}
W20260709 12:16:59.689874 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:59.751711 12642 rpcz_store.cc:276] 0709 12:16:57.510723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.887436 (+376713us) service_pool.cc:224] Handling call
0709 12:16:59.685477 (+1798041us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:16:59.685796 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:59.751056 (+ 65260us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:59.751297 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3593472}
I20260709 12:16:59.531955 24329 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.607765 24434 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:59.629077 24411 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:16:59.685070 24358 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:16:59.703763 13262 raft_consensus.cc:2368] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:16:59.528817 24365 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.725497 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:16:59.639360 12571 rpcz_store.cc:269] 0709 12:16:57.368980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.369859 (+   879us) service_pool.cc:224] Handling call
0709 12:16:59.483501 (+2113642us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:59.483673 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3914412416}
I20260709 12:16:59.656282 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.688776 12492 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:59.779286 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:16:59.639490 13259 rpcz_store.cc:276] 0709 12:16:57.026511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:57.026790 (+   279us) service_pool.cc:224] Handling call
0709 12:16:57.027207 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:16:58.455474 (+1428267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:16:59.480198 (+1024724us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:16:59.480455 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3282200320}
I20260709 12:16:59.613829 24172 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:59.690073 12496 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:16:59.672997 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:16:59.648456 12493 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:16:59.638262 24354 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:16:59.776031 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:59.637776 24375 raft_consensus.cc:2764] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Leader pre-election decision vote started in defunct term 35: won
I20260709 12:16:59.633318 24442 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:16:59.779986 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 123788us
I20260709 12:16:59.528506 23770 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:59.807693 23770 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.810081 23770 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.693711 24333 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:16:59.794505 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:16:59.778793 24323 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:16:59.783761 24441 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:59.614789 24366 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.606910 24171 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:16:59.640949 23540 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:16:59.803644 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.628372 24330 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.782382 24172 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:59.806329 24442 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.842568 24333 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:59.843582 24333 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:16:59.843967 24442 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.772028 24091 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.844178 24333 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.846038 24091 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.846988 24172 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:59.848122 12567 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:16:59.811900 12644 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:16:59.775695 24434 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:16:59.853259 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:59.641990 24025 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:59.764190 24317 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:16:59.826279 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 299817us
I20260709 12:16:59.822731 12563 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:16:59.826892 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:59.527127 24400 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.837616 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.836483 24171 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:16:59.825491 23915 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.864161 12536 leader_election.cc:400] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
W20260709 12:16:59.878158 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60848.9us
I20260709 12:16:59.881031 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1820) took 1027 ms. Trace:
I20260709 12:16:59.864668 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:16:59.817659 24001 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:16:59.881512 12494 rpcz_store.cc:276] 0709 12:16:58.853613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:58.864910 (+ 11297us) service_pool.cc:224] Handling call
0709 12:16:59.880787 (+1015877us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:16:59.880936 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":941758720}
I20260709 12:16:59.866235 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.774935 24329 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.877079 12491 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 16.
I20260709 12:16:59.883752 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:16:59.884889 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.889431 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:16:59.893011 12637 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:16:59.800715 24365 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.907707 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:16:59.869277 24317 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:16:59.914228 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 411529us
I20260709 12:16:59.916039 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:16:59.849140 24429 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:16:59.871065 24366 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:16:59.915963 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.824757 24215 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.923274 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:59.923800 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.901005 23770 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:16:59.952484 24215 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:16:59.953117 23770 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:16:59.954046 23770 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:16:59.912075 24358 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:16:59.955855 24365 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.924204 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.874185 23540 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:16:59.927057 12646 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 34 because replica has last-logged OpId of term: 33 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:16:59.957968 24412 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:16:59.957788 23540 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.958586 24412 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.960505 23540 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.960891 24412 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:16:59.933316 12645 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31. Re-sending same reply.
I20260709 12:16:59.887542 24441 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:59.934185 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:16:59.934756 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.963155 12488 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:16:59.964206 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:16:59.966567 12490 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
W20260709 12:16:59.968878 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:16:59.969388 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:59.932824 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.923897 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.938261 24171 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.19, Last appended by leader: 19, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:16:59.944087 24153 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:16:59.920871 24405 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:16:59.941397 24391 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:59.950970 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:59.951311 24329 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:16:59.951725 24324 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:16:59.986073 23659 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:16:59.898659 24375 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.054s
I20260709 12:16:59.979394 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:16:59.976202 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:16:59.985006 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:59.986629 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:16:59.986649 23659 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:16:59.994987 23659 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:16:59.925570 24271 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:16:59.998240 23915 consensus_queue.cc:1175] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" responder_term: 39 status { last_received { term: 39 index: 19 } last_committed_idx: 19 last_received_current_leader { term: 39 index: 19 } }
I20260709 12:16:59.999982 24333 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:16:59.999185 24391 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 37.15, Last appended by leader: 15, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.009614 24171 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:16:59.994395 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.056017 12495 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
W20260709 12:17:00.013464 24402 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.113s	user 0.017s	sys 0.000s
I20260709 12:16:59.990610 24405 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:00.017567 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:16:59.984699 24162 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:00.023286 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.026896 24383 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:00.027576 24435 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:00.028223 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:16:59.992461 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:00.038657 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.039254 23659 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.072857 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1824) took 1204 ms. Trace:
W20260709 12:16:59.987571 24325 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.000s
I20260709 12:16:59.969838 24413 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:00.022547 24271 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:16:59.983067 24153 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.062026 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.072451 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:00.087177 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:00.031620 24411 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:00.073457 12489 rpcz_store.cc:276] 0709 12:16:58.868228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:58.868426 (+   198us) service_pool.cc:224] Handling call
0709 12:16:58.888330 (+ 19904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:16:59.381850 (+493520us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:16:59.391832 (+  9982us) log.cc:844] Serialized 14 byte log entry
0709 12:16:59.899779 (+507947us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b1800272a40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:16:59.899987 (+   208us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:16:59.900044 (+    57us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:16:59.900095 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:00.072547 (+172452us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:00.072612 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:00.072794 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":179367,"prepare.queue_time_us":10320,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":457569,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":11939,"spinlock_wait_cycles":1163585536,"thread_start_us":58110,"threads_started":1,"wal-append.queue_time_us":235198}
I20260709 12:17:00.068562 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:00.072510 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80111.3us
I20260709 12:17:00.093660 12493 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
W20260709 12:17:00.084097 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:00.071637 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:00.032778 24447 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:17:00.088377 12492 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
W20260709 12:17:00.088299 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 196965us
I20260709 12:16:59.980162 24404 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.089s
I20260709 12:17:00.090219 12496 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:17:00.096701 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:00.097093 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.103844 12493 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:17:00.108532 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.109937 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.108753 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.105692 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:00.111138 12637 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:00.111424 13258 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:00.112020 12492 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:17:00.112934 23659 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.112064 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:00.108084 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.115334 12646 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:00.117496 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.118257 12569 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:17:00.118377 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46851.6us
W20260709 12:17:00.119699 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 230343us
I20260709 12:17:00.067241 24402 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:00.120607 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.066956 24162 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:00.125162 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:00.125761 12492 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:00.125209 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.127367 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:00.128341 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.128837 12639 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: [38.19-38.19]
I20260709 12:17:00.129170 12640 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:00.130123 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.131335 12646 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [33.20-34.21]
I20260709 12:17:00.132398 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.089521 24333 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.135792 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.091427 24324 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:00.089815 24312 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:00.142802 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:00.146323 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.148113 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.148985 12639 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:00.149842 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.089964 24153 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:00.096724 24417 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:00.134276 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.137872 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.126312 13254 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:00.202711 12565 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:00.122236 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:00.203781 12641 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
W20260709 12:17:00.204176 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82039.7us
I20260709 12:17:00.153993 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.165370 12637 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:00.205927 12639 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:00.202711 12646 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:00.207839 12637 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:00.205331 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:00.133399 24162 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:00.105846 24383 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:00.209578 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.144611 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:16:59.989431 24025 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.102047 24435 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:00.214159 12646 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: [39.19-39.19]
I20260709 12:17:00.215114 12638 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:17:00.215680 12565 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32884
I20260709 12:17:00.216308 23540 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:00.213306 12637 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:17:00.107347 24411 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:00.133781 24325 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:00.136804 24317 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:00.123370 23972 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:00.236831 12644 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 27.
I20260709 12:17:00.240435 12641 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:00.240448 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.245038 12637 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:17:00.142894 24312 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:00.102895 24382 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.111s
I20260709 12:17:00.143150 24324 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.244400 24325 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:00.243618 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.202793 24333 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:00.300500 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56983.6us
I20260709 12:17:00.212774 24329 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:00.212963 24441 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:00.090022 24413 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.135275 24375 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.114s
I20260709 12:17:00.212246 24451 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:00.279989 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:00.136940 24399 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.110s
I20260709 12:17:00.254881 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:00.259687 24419 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.544s	user 0.011s	sys 0.001s
W20260709 12:17:00.270298 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125748us
I20260709 12:17:00.241317 24383 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:00.201822 24417 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:00.348610 12494 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:00.354593 12493 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:00.278247 24449 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.011s
I20260709 12:17:00.289705 24435 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.290269 23915 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.272099 24271 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.247495 24317 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:00.295269 24448 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.457s	user 0.001s	sys 0.010s
I20260709 12:17:00.297214 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.380867 12563 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:00.299616 12640 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
I20260709 12:17:00.301316 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:00.261653 24366 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:17:00.325290 12492 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 16. Preceding OpId from leader: term: 35 index: 17. (index mismatch)
I20260709 12:17:00.387671 23659 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:00.257051 24312 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:00.303407 24324 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:00.318341 24418 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:00.399423 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:00.352473 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97671us
I20260709 12:17:00.279031 23972 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:00.356238 24370 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.611s	user 0.006s	sys 0.008s
I20260709 12:17:00.374156 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:00.374907 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.524s (SENT) [suppressed 23 similar messages]
W20260709 12:17:00.388319 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 87080.2us
W20260709 12:17:00.363628 24172 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.516s	user 0.011s	sys 0.002s
I20260709 12:17:00.390877 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:00.356382 24329 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:00.350431 12639 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
W20260709 12:17:00.395198 24323 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.001s
I20260709 12:17:00.404331 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:00.417124 12492 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:00.419121 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:00.419524 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.420236 12563 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:00.417838 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:00.402408 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.423735 12491 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:00.381145 24317 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:00.427091 12565 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:00.427736 12565 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:00.422164 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48089us
I20260709 12:17:00.423060 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.444288 24413 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.444628 23972 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.374270 23915 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.446955 23915 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.447140 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.414873 23540 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.039s
I20260709 12:17:00.455272 23540 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:00.455705 23540 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.456018 23540 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:00.459573 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:00.457387 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:00.463086 12565 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.425773 24172 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.466886 24333 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:00.467823 24333 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:00.468693 24172 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.399466 24418 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:00.380165 24411 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.17, Last appended by leader: 17, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.430681 24375 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:00.426998 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:00.472029 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45103.1us
I20260709 12:17:00.382367 24325 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.468514 24417 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:00.473338 12564 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:17:00.407897 24399 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:00.469764 24418 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:00.474092 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55045us
I20260709 12:17:00.492172 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.476297 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.485198 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.520006 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:00.490792 24387 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:00.488605 24404 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:00.430634 24405 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.570667 24417 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:00.598059 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:00.585309 24354 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.774s	user 0.014s	sys 0.003s
I20260709 12:17:00.602181 24354 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.605372 12566 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 30: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:00.606129 24354 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.511848 23915 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.607352 23915 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:00.484511 24452 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:00.610049 24452 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:00.611142 24452 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:00.519315 24434 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.657s	user 0.003s	sys 0.009s
I20260709 12:17:00.613516 24434 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.516985 12497 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:00.425104 24383 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:00.596561 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111987us
I20260709 12:17:00.488080 24366 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.103s
W20260709 12:17:00.597163 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120955us
I20260709 12:17:00.487771 24358 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.100s
I20260709 12:17:00.624083 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:00.626061 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:00.596697 24387 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:00.594775 24404 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:00.607771 24459 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:17:00.593256 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:00.586308 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211416us
W20260709 12:17:00.615763 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 95834.4us
W20260709 12:17:00.616709 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:00.622260 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:00.637671 12462 leader_election.cc:400] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:17:00.643990 12462 leader_election.cc:403] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
W20260709 12:17:00.640014 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:00.634783 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.633s (SENT)
W20260709 12:17:00.638322 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 191278us
I20260709 12:17:00.648600 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.651615 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.658241 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:00.658499 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:00.669525 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 246524us
I20260709 12:17:00.623585 24323 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:00.662232 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:00.662492 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:00.623854 24331 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:00.671531 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:00.633482 24404 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:00.634639 24091 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:00.679041 12563 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:00.674531 13254 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:00.674835 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.635792 24399 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:00.675875 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:00.689327 13255 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:00.689797 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.690407 12563 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:17:00.691253 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.658854 24370 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.691920 12563 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:00.664012 24429 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:00.662407 24215 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:17:00.700075 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.701140 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:00.673236 24391 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:00.679277 24323 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.709692 24323 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:00.672799 24365 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:17:00.718281 24461 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:17:00.677675 23915 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:00.722186 24391 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:00.722852 24391 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:00.677878 24405 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:17:00.739897 24312 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:00.750557 24462 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:17:00.706746 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:00.697942 24457 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.019s
I20260709 12:17:00.751464 12567 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:00.752077 12563 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:17:00.677481 23972 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.032s
I20260709 12:17:00.689122 24358 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.041s
W20260709 12:17:00.730544 24153 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.001s
I20260709 12:17:00.725015 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.754881 24153 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.731122 24365 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:00.733852 24466 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:00.757136 24466 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:00.757504 12571 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:00.757660 24466 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:00.759013 24332 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:00.759521 24332 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:00.759893 24332 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:00.735159 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:00.764305 23540 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:00.745785 24460 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:17:00.684840 24413 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.072s
I20260709 12:17:00.708583 24434 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.753979 24317 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:00.764846 24081 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.015s	sys 0.031s
I20260709 12:17:00.744645 24325 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:00.768294 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:00.791679 12638 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:17:00.770200 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.771514 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.771152 24435 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.663825 24162 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.773550 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.773969 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.793704 12645 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:00.765407 24153 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:00.772966 24413 consensus_queue.cc:1243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 32.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:17:00.777309 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:00.781494 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.781831 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.805686 12643 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:00.684682 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.786583 12644 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:17:00.724929 24399 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:00.788658 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 112876us
I20260709 12:17:00.774287 24358 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:17:00.782933 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96609us
I20260709 12:17:00.799208 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.844341 12497 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:00.800366 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:00.806833 12641 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:00.807101 12489 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:00.808794 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124199us
I20260709 12:17:00.844465 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:00.851878 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.849818 23659 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.003s
I20260709 12:17:00.852757 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.845593 12491 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 30: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:00.850248 12563 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:00.849121 23540 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:00.702275 24370 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:00.802826 24398 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:00.851878 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.856426 12496 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:00.856531 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.856885 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:00.857570 23659 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:17:00.794450 24081 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.016s	sys 0.031s
I20260709 12:17:00.858223 23659 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:00.786012 23972 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:00.817193 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130468us
I20260709 12:17:00.858098 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:00.865731 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:00.892974 12489 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:00.860674 12496 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:00.890133 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.893684 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:00.894166 12644 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:00.891361 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:00.858285 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.901204 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:00.915458 23540 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:17:00.952590 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94375.6us
I20260709 12:17:00.953541 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:00.952656 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51596.2us
I20260709 12:17:00.791224 24382 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.086s
I20260709 12:17:00.845963 24153 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:00.843021 24366 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:00.888612 24434 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:00.804838 24324 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.843469 24358 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.905208 12646 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: [34.24-34.24]
I20260709 12:17:00.902422 12492 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:17:00.849291 24387 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:00.810998 24091 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.141s
I20260709 12:17:00.803650 24435 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.966279 24003 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:17:01.002424 23659 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.903328 24465 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:01.010126 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:00.961915 24417 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:00.961968 24411 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:00.951313 24354 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:01.024166 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:01.012606 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:00.947507 23972 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:00.951555 24405 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:17:01.033728 12572 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:00.862213 24398 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:00.999876 24434 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:01.034960 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141368us
I20260709 12:17:00.973109 24153 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:01.006022 24366 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:01.015136 24358 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:00.963899 24323 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:01.036754 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177464us
I20260709 12:17:01.058439 12569 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:17:01.042529 24003 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.075944 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:01.078969 12644 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [34.22-36.23]
I20260709 12:17:01.079221 24370 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.022035 24449 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:17:01.011237 24375 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.039s
I20260709 12:17:01.032021 24312 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.065492 24465 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.043766 24402 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:17:01.058040 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:01.045426 24354 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:01.083268 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:01.083915 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119847us
I20260709 12:17:01.025187 24387 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.087424 12496 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:01.059530 24434 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:01.055045 24398 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:01.035179 23697 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.039s
I20260709 12:17:01.068235 24366 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:01.112032 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 4921) took 1377 ms. Trace:
I20260709 12:17:01.052431 24468 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:17:01.145219 24324 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:01.145727 24473 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:17:01.125490 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:01.128822 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70882us
I20260709 12:17:01.151335 24323 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.097766 24003 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.152879 12563 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 26. Current term is 27. Ops: []
W20260709 12:17:01.136359 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:01.153950 24003 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:01.154204 24471 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:17:01.155148 24391 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:01.156601 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:01.157804 24391 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.059060 24404 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.142083 12566 rpcz_store.cc:276] 0709 12:16:59.710543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:59.739370 (+ 28827us) service_pool.cc:224] Handling call
0709 12:17:01.087301 (+1347931us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:01.087455 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1988818176}
I20260709 12:17:01.130079 24354 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:01.161484 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:01.161772 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:01.122002 24402 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:01.137457 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98813.7us
I20260709 12:17:01.168279 13258 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:01.158126 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:01.198881 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73450.3us
I20260709 12:17:01.122367 24370 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:01.130511 24452 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:01.202497 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:01.207484 24477 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:17:01.208317 24482 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:17:01.210538 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52506.9us
I20260709 12:17:01.222173 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:01.164676 24162 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.125428 24474 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.051s
I20260709 12:17:01.217388 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:01.222468 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:01.276230 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59014.3us
I20260709 12:17:01.278940 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:01.279711 12644 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:01.225903 24469 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
I20260709 12:17:01.281888 23659 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.287395 13258 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:17:01.228329 24383 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.013s	sys 0.013s
I20260709 12:17:01.217048 23697 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:01.219007 24402 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:17:01.291114 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 129721us
I20260709 12:17:01.227252 24312 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:01.295328 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:01.298538 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76167.7us
I20260709 12:17:01.287040 23659 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:01.273439 24484 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.308043 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:01.215994 24332 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:01.295683 24383 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.013s	sys 0.014s
I20260709 12:17:01.285194 24417 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:01.332831 12489 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:01.285754 24375 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:01.294533 24488 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.336464 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:01.320037 24449 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:01.298605 24215 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:17:01.333730 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:01.220798 24370 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.310056 24312 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:01.379722 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:01.436415 12489 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:01.437767 12489 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:17:01.342595 12644 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:17:01.381872 24488 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.329141 24490 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:01.402174 24404 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.311470 23697 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:01.322259 24081 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:01.396870 24482 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:17:01.383618 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 161539us
I20260709 12:17:01.411576 24435 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.352463 24323 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:01.324501 24484 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:01.319540 24452 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.103s
I20260709 12:17:01.417027 24449 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.535009 24449 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:01.536193 24484 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.496368 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:01.538918 24488 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.539634 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:01.541031 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:01.501895 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:01.352258 24483 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.115s
I20260709 12:17:01.445468 23972 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.547055 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:01.548470 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:01.549532 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:01.551061 12644 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:01.552263 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:01.552779 12644 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:01.557008 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4569) took 1431 ms. Trace:
I20260709 12:17:01.557682 12640 rpcz_store.cc:276] 0709 12:17:00.125611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.154143 (+ 28532us) service_pool.cc:224] Handling call
0709 12:17:00.299409 (+145266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:00.382403 (+ 82994us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:01.556708 (+1174305us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54003b06d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:01.556910 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333239040}
I20260709 12:17:01.563355 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4929) took 1170 ms. Trace:
I20260709 12:17:01.564047 12570 rpcz_store.cc:276] 0709 12:17:00.393002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.859740 (+466738us) service_pool.cc:224] Handling call
0709 12:17:00.860179 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:01.086749 (+226570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:01.086786 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:01.086833 (+    47us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:01.086867 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:01.563069 (+476202us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54003ac354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:01.563134 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:01.563281 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":520494848}
I20260709 12:17:01.445082 24312 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:01.507961 24491 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:17:01.508636 24476 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.381s	user 0.001s	sys 0.000s
I20260709 12:17:01.510491 23697 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:01.512095 24493 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:17:01.526470 24465 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.528151 24332 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.497049 24153 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.441920 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:01.673703 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:01.673374 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:01.354631 24417 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:01.456599 24470 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.377s	user 0.001s	sys 0.001s
W20260709 12:17:01.567602 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:01.598445 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:01.549355 24499 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:01.619522 24325 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.823s	user 0.009s	sys 0.004s
I20260709 12:17:01.560909 24449 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:01.492374 24081 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.691558 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4774) took 2176 ms. Trace:
W20260709 12:17:01.692406 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.010s	sys 0.007s
W20260709 12:17:01.692803 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.010s	sys 0.007s
I20260709 12:17:01.692586 12564 rpcz_store.cc:276] 0709 12:16:59.515162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:59.674905 (+159743us) service_pool.cc:224] Handling call
0709 12:16:59.675478 (+   573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:00.495984 (+820506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:01.691103 (+1195119us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:01.691461 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1833992448}
I20260709 12:17:01.682761 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:01.549979 24507 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:01.696786 12492 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:01.698386 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:01.699102 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.014s	sys 0.005s
W20260709 12:17:01.703848 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.014s	sys 0.006s
W20260709 12:17:01.693625 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 539811us
W20260709 12:17:01.707391 12645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.006s	sys 0.014s
W20260709 12:17:01.707906 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.007s	sys 0.014s
I20260709 12:17:01.662799 24387 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:01.689850 24333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.011s	sys 0.005s
I20260709 12:17:01.717979 13253 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:01.718532 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:01.697201 24081 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:01.734859 13254 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
I20260709 12:17:01.738488 13257 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
W20260709 12:17:01.719895 24333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.011s	sys 0.006s
W20260709 12:17:01.715232 24478 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.487s	user 0.000s	sys 0.001s
I20260709 12:17:01.764084 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:01.765322 13259 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:17:01.714951 24328 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:01.713861 24153 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:01.726810 24400 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:01.782752 24489 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:01.792240 24417 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:01.792537 24162 raft_consensus.cc:476] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Not starting pre-election -- already a leader
W20260709 12:17:01.798310 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4782) took 1884 ms (client timeout 1823 ms). Trace:
I20260709 12:17:01.798600 24468 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.690127 24499 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.732371 24003 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:01.798976 12571 rpcz_store.cc:269] 0709 12:16:59.913886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:59.924657 (+ 10771us) service_pool.cc:224] Handling call
0709 12:17:01.798056 (+1873399us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:01.798218 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1827188992}
W20260709 12:17:01.798385 24492 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 12:17:01.826360 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:01.833164 12565 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:01.842005 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:32884 (request call id 4787) took 1631 ms. Trace:
I20260709 12:17:01.842618 12565 rpcz_store.cc:276] 0709 12:17:00.203382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.205192 (+  1810us) service_pool.cc:224] Handling call
0709 12:17:01.805213 (+1600021us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eaa3d 00007ffbfa7bacd0 00007ffbf3d3f254 00007ffbf3d3f1e2 00007ffbf3d3f111 00007ffbf3d3f08c 00007ffbf3d3e342 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:01.834238 (+ 29025us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461377280}
I20260709 12:17:01.793344 24162 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:01.821484 24494 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:17:01.710621 24507 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:01.858934 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 256052us
I20260709 12:17:01.859869 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:01.865177 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3761) took 1097 ms. Trace:
I20260709 12:17:01.865736 12642 rpcz_store.cc:276] 0709 12:17:00.767778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.767936 (+   158us) service_pool.cc:224] Handling call
0709 12:17:00.768344 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:00.776226 (+  7882us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:00.776276 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:00.776928 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 12:17:01.024673 (+247745us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:01.024720 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:01.024816 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:01.864933 (+840117us) raft_consensus.cc:1685] finished
0709 12:17:01.864956 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:01.865111 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7394,"raft.queue_time_us":17424,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9104,"thread_start_us":215973,"threads_started":1,"wal-append.queue_time_us":347641}
I20260709 12:17:01.799412 24468 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:01.848520 24505 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.063s	user 0.003s	sys 0.000s
I20260709 12:17:01.801872 24328 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.892585 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:01.911759 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:01.907550 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:01.913951 13257 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 24.
I20260709 12:17:01.914469 24523 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:01.915107 24523 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.915808 13257 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:17:01.838107 24419 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:17:01.917187 24419 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:01.918421 24519 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:17:01.918290 24419 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:01.919349 24519 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:01.919620 24419 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:17:01.920619 24519 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:01.926959 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67187.4us
I20260709 12:17:01.927980 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:01.929059 24504 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:01.929533 24504 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.930066 24331 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:01.930411 24504 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:01.930820 24331 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:01.932049 24317 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.940516 24504 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:01.890223 24354 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.680s	user 0.012s	sys 0.000s
W20260709 12:17:01.942639 24510 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:17:01.942339 24354 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.944804 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:01.945082 24354 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:01.806352 24153 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.853328 24521 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.852295 24457 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:01.822422 24003 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:01.950749 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:01.941448 24418 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.420s	user 0.012s	sys 0.001s
W20260709 12:17:01.913223 24429 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.207s	user 0.012s	sys 0.003s
W20260709 12:17:01.993238 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 310773us
I20260709 12:17:01.997036 12565 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
W20260709 12:17:02.012403 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89596.7us
I20260709 12:17:01.913959 24370 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:01.858764 24162 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.023360 13256 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:02.009398 24329 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:01.999653 24523 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.042064 24520 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:17:01.911903 24511 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.152s
I20260709 12:17:02.033876 24383 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.053763 24499 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.054873 24507 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.051667 24328 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.098711 24153 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.090988 24003 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.092803 24162 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.100503 24468 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.097304 24383 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.103200 24524 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:02.105882 23659 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:02.108965 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4784) took 2078 ms (client timeout 1894 ms). Trace:
W20260709 12:17:02.109730 12568 rpcz_store.cc:269] 0709 12:17:00.030581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.030736 (+   155us) service_pool.cc:224] Handling call
0709 12:17:01.767006 (+1736270us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b54002b0320. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6e18 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.108649 (+341643us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.108858 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2030017792}
W20260709 12:17:02.102167 24485 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.726s	user 0.004s	sys 0.002s
W20260709 12:17:02.052901 24434 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.893s	user 0.012s	sys 0.000s
I20260709 12:17:02.117342 24434 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.120784 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2288) took 2001 ms. Trace:
I20260709 12:17:02.121208 12567 rpcz_store.cc:276] 0709 12:17:00.119690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.121323 (+  1633us) service_pool.cc:224] Handling call
0709 12:17:00.121688 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:00.751928 (+630240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.120488 (+1368560us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54002477d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.120715 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1447831936}
I20260709 12:17:02.122953 12564 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:02.055330 24449 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:02.044719 24479 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.509s	user 0.001s	sys 0.001s
I20260709 12:17:02.122494 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.130306 12567 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:02.094595 24399 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.246s	user 0.010s	sys 0.005s
W20260709 12:17:02.062026 24466 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.304s	user 0.014s	sys 0.003s
W20260709 12:17:02.096649 24366 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.926s	user 0.009s	sys 0.000s
W20260709 12:17:02.089205 24402 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.001s
I20260709 12:17:02.111181 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.127974 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.145041 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.140914 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.119740 24434 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.142962 12565 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:02.147861 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.148866 12565 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:17:01.998149 24521 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.163378 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.043861 24329 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:02.164374 12567 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:02.142518 24003 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:17:02.167448 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.168170 24003 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:17:02.142630 24466 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.017105 24418 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.143353 24399 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:02.156504 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3766) took 1377 ms. Trace:
I20260709 12:17:02.021859 24457 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.163015 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:02.139890 24358 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.044s	user 0.007s	sys 0.007s
I20260709 12:17:02.154176 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:02.138271 24504 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.169304 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:02.151026 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 206286us
I20260709 12:17:02.153504 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.174352 12639 rpcz_store.cc:276] 0709 12:17:00.779378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.779535 (+   157us) service_pool.cc:224] Handling call
0709 12:17:00.779953 (+   418us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:00.786380 (+  6427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:00.786427 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:00.787111 (+   684us) log.cc:844] Serialized 52 byte log entry
0709 12:17:01.034489 (+247378us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:17:01.034532 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:01.034601 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:02.156132 (+1121531us) raft_consensus.cc:1685] finished
0709 12:17:02.156195 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:02.156415 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26663,"raft.queue_time_us":18322,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":43725,"thread_start_us":186548,"threads_started":1,"wal-append.queue_time_us":523840}
I20260709 12:17:02.184821 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.185737 12565 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:02.186419 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:02.138696 24312 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.188760 24521 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.189015 12495 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:02.195264 24323 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.196448 24323 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.165918 24381 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:02.198436 24381 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:02.199222 24398 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.052s	user 0.011s	sys 0.002s
I20260709 12:17:02.199712 24398 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.188813 24418 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.202208 13260 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:17:02.182297 24328 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:02.202865 24328 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:02.203258 24328 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:17:02.203145 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 761251us
W20260709 12:17:02.204421 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1473) took 1548 ms (client timeout 1709 ms). Trace:
W20260709 12:17:02.204892 13256 rpcz_store.cc:269] 0709 12:17:00.655945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.656089 (+   144us) service_pool.cc:224] Handling call
0709 12:17:02.204190 (+1548101us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.204347 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3114852224}
I20260709 12:17:02.206552 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:02.182286 24358 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.207448 24520 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:17:02.208232 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 4949) took 1344 ms. Trace:
I20260709 12:17:02.209189 24358 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.209951 12643 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35.
I20260709 12:17:02.211210 24329 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.157763 24402 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.213928 24317 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.214747 24317 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.216439 12638 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:02.150991 24366 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.204263 24325 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:17:02.225291 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:02.193367 24518 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.185s	user 0.003s	sys 0.000s
I20260709 12:17:02.208736 12491 rpcz_store.cc:276] 0709 12:17:00.863432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.863778 (+   346us) service_pool.cc:224] Handling call
0709 12:17:00.951732 (+ 87954us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:01.026241 (+ 74509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:01.026383 (+   142us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:01.027820 (+  1437us) log.cc:844] Serialized 26 byte log entry
0709 12:17:01.318906 (+291086us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:17:01.720528 (+401622us) log.cc:844] Serialized 14 byte log entry
0709 12:17:02.207840 (+487312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.207935 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:02.207982 (+    47us) raft_consensus.cc:1685] finished
0709 12:17:02.207997 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:02.208160 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":502180,"prepare.queue_time_us":223527,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":118240,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":8339,"thread_start_us":679024,"threads_started":3,"wal-append.queue_time_us":500795}
I20260709 12:17:02.168705 24003 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:02.225685 24215 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.231956 24215 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.233299 24317 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.234068 24398 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:02.234392 24215 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.229643 24490 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.235347 24490 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.235497 24383 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.202256 24483 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:02.236953 24483 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.237807 24404 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:02.238288 24483 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.240785 24483 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:02.241094 24404 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.175350 24331 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.241729 24325 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:02.242923 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1729) took 1231 ms (client timeout 1524 ms). Trace:
I20260709 12:17:02.244179 24081 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.245163 24466 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.245968 24399 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.246667 12567 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:17:02.242981 24457 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.247804 24366 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:02.189327 24312 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:02.253583 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67238.1us
I20260709 12:17:02.258463 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:02.263160 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:02.226354 24329 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:02.267277 24490 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.269328 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.291780 13255 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:02.226598 24450 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.272819 24484 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:02.271195 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.298004 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:17:02.273342 12643 rpcz_store.cc:269] 0709 12:17:01.012069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.012227 (+   158us) service_pool.cc:224] Handling call
0709 12:17:02.242676 (+1230449us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.242841 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2676943744}
W20260709 12:17:02.275425 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3684) took 2284 ms (client timeout 2163 ms). Trace:
W20260709 12:17:02.275542 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45516.8us
W20260709 12:17:02.299469 12490 rpcz_store.cc:269] 0709 12:16:59.986184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:59.986389 (+   205us) service_pool.cc:224] Handling call
0709 12:17:02.270531 (+2284142us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.270847 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2243812608}
I20260709 12:17:02.280081 13258 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:02.279978 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.265129 24475 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:02.306368 24475 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.306747 12493 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:02.307443 24475 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.284895 24399 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.309293 24475 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:02.309710 12488 raft_consensus.cc:606] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:02.310132 12488 raft_consensus.cc:614] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:02.310582 24391 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.311069 24391 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.300827 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.315308 24399 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.318251 12567 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:02.319972 12567 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:02.286377 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:02.273615 24504 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:17:02.320372 13255 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:02.285377 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.328554 12563 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:02.330217 24457 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:02.331066 24457 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:02.332297 24457 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 37.15, Last appended by leader: 15, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.333158 24081 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:02.339015 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 80521.7us
I20260709 12:17:02.339860 24419 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:02.340456 24419 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:02.340876 24419 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:02.342128 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.343737 24399 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:17:02.344310 24399 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.345752 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.346655 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.347208 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.347858 13256 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:02.348109 24091 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:02.348515 24091 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:02.348888 24091 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:02.348821 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.349843 24325 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.342360 12491 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:02.273346 24404 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.345417 24399 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.354596 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.362812 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.363255 12494 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:02.357558 24504 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.357180 12496 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 17. Preceding OpId from leader: term: 36 index: 18. (index mismatch)
I20260709 12:17:02.384656 12563 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:02.359964 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.323617 24328 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.359105 12567 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:17:02.325235 24402 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.371726 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.347867 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.371897 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:02.369269 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 76) took 2311 ms. Trace:
W20260709 12:17:02.375144 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:02.394829 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:02.394531 12488 rpcz_store.cc:276] 0709 12:17:00.064714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.066590 (+  1876us) service_pool.cc:224] Handling call
0709 12:17:02.368940 (+2302350us) spinlock_profiling.cc:243] Waited 2.21 s on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.369123 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5075292416}
I20260709 12:17:02.395442 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:02.396291 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:02.376609 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1828) took 2054 ms. Trace:
I20260709 12:17:02.386276 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:02.370555 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.397095 12495 rpcz_store.cc:276] 0709 12:17:00.322409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.322644 (+   235us) service_pool.cc:224] Handling call
0709 12:17:00.323222 (+   578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:02.189727 (+1866505us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.376338 (+186611us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.376543 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4290421632}
I20260709 12:17:02.398574 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.400732 12646 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:02.404524 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.405643 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:17:02.405787 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2281) took 2930 ms. Trace:
I20260709 12:17:02.407298 12572 rpcz_store.cc:276] 0709 12:16:59.475266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:59.477882 (+  2616us) service_pool.cc:224] Handling call
0709 12:16:59.478208 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:01.049777 (+1571569us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.405503 (+1355726us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54001b2bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.405725 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3576745728}
I20260709 12:17:02.409382 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.410456 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4580) took 1334 ms. Trace:
I20260709 12:17:02.411028 12646 rpcz_store.cc:276] 0709 12:17:01.075814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.077745 (+  1931us) service_pool.cc:224] Handling call
0709 12:17:02.370383 (+1292638us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:02.410369 (+ 39986us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4657024}
I20260709 12:17:02.412104 24509 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:02.335286 24484 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.416988 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:02.379155 24411 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.427211 24332 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:02.442930 24411 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.443086 24332 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.403717 24312 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:02.434576 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.415412 24383 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:02.446681 24383 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:02.447783 24383 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 34.26, Last appended by leader: 26, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.449667 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.449867 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.450765 12567 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:02.335749 23697 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.073s
I20260709 12:17:02.436883 24328 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.391987 24504 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.339049 24003 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.441109 24387 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:02.442044 24532 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:17:02.328436 24450 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.468698 12491 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:02.455892 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4931) took 2057 ms. Trace:
I20260709 12:17:02.470238 12569 rpcz_store.cc:276] 0709 12:17:00.398381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.861262 (+462881us) service_pool.cc:224] Handling call
0709 12:17:00.952585 (+ 91323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:01.072938 (+120353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.455668 (+1382730us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.455834 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":242546944}
I20260709 12:17:02.456128 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.460676 12494 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:02.462435 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.463603 12572 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:17:02.454905 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:02.474781 12495 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:02.478428 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.479419 12495 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:17:02.480459 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.481226 12495 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:02.481828 24387 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.482364 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.483716 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.484481 24387 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.476411 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:02.485924 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.487097 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.497969 12495 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:02.416536 24402 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:17:02.421591 24375 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.982s	user 0.010s	sys 0.010s
I20260709 12:17:02.484427 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.445493 24411 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.505100 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.506760 12563 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:02.522533 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.524061 12494 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:02.527195 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.538184 12489 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:17:02.540140 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.541569 12567 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:17:02.544217 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:02.503460 24375 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.069s	user 0.011s	sys 0.010s
I20260709 12:17:02.548637 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.549871 12572 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:02.550737 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.551926 12497 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:17:02.509161 24418 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:02.573894 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:17:02.554798 24375 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:02.527349 24387 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.539527 24509 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:02.539151 24003 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:02.562618 24398 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:02.554927 24332 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.580350 24418 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.592058 24387 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.614354 24418 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:02.615108 24091 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.616411 24387 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.582895 24509 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.617488 24509 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:02.617568 24091 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.618969 24507 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:02.619472 24507 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:02.620528 24507 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.604183 24375 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.626231 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3709) took 1290 ms. Trace:
I20260709 12:17:02.626930 12497 rpcz_store.cc:276] 0709 12:17:01.335745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.026326 (+690581us) service_pool.cc:224] Handling call
0709 12:17:02.026753 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:02.571152 (+544399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.625814 (+ 54662us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b540020bcd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.626154 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1206966528}
I20260709 12:17:02.629688 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.632800 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3703) took 1298 ms. Trace:
I20260709 12:17:02.633311 12492 rpcz_store.cc:276] 0709 12:17:01.334220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.408593 (+ 74373us) service_pool.cc:224] Handling call
0709 12:17:01.408898 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:01.703208 (+294310us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.632513 (+929305us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.632730 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":660977920}
I20260709 12:17:02.636791 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.611807 12495 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:02.645576 24509 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.674809 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.676003 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.680976 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.698026 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:02.698483 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.698969 12565 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:02.699628 12494 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:02.700816 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.701663 12565 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:02.596628 24498 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:02.698136 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.695158 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:02.710379 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.702836 24325 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:02.715837 24450 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.711776 24332 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
I20260709 12:17:02.716989 24271 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:02.717360 24484 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:02.644851 24381 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.719455 24323 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:02.720692 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.695798 24091 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:02.611441 24398 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:02.710088 24498 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:02.692229 24331 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:02.740657 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52123.4us
I20260709 12:17:02.613314 12488 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:02.724876 24475 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:02.725355 24325 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.747601 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1726) took 1849 ms. Trace:
I20260709 12:17:02.750512 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.751637 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.723229 24507 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:02.755937 12646 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:02.759423 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.757031 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:02.762466 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:02.763315 12488 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:02.763522 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:02.737738 24091 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.736582 24490 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:02.754730 12645 rpcz_store.cc:276] 0709 12:17:00.897951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.948236 (+ 50285us) service_pool.cc:224] Handling call
0709 12:17:00.948647 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:00.948887 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:17:00.974156 (+ 25269us) log.cc:844] Serialized 14 byte log entry
0709 12:17:02.192594 (+1218438us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:02.746970 (+554376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:02.747074 (+   104us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:02.747172 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.747293 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:02.747503 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1038231,"prepare.queue_time_us":19789,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"spinlock_wait_cycles":12175744,"thread_start_us":733774,"threads_started":1,"wal-append.queue_time_us":905164}
I20260709 12:17:02.737515 24404 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.768985 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.766754 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:02.774312 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.775421 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.776343 12641 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:02.777961 13218 leader_election.cc:336] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.524s (SENT)
I20260709 12:17:02.779277 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.779157 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.780751 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.781082 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.781508 12640 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:02.780198 13255 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:02.782681 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.778622 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:02.783612 12644 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:02.785006 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.758812 24332 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:17:02.785939 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.786969 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.754861 24400 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:02.754390 24475 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.754194 24215 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:02.794602 24516 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.795094 24215 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.786581 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:02.796280 24441 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.796753 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.758736 24381 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:02.770352 24411 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:17:02.768080 24312 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.030s
I20260709 12:17:02.768617 24271 consensus_queue.cc:1243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 32.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:17:02.779749 24317 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.023s
I20260709 12:17:02.787460 12646 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:02.770792 24091 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:02.793109 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:02.795202 24516 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.790611 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:02.797607 12645 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:02.804004 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.788935 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.813145 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:17:02.814092 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.815141 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.815624 12644 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:02.815898 12644 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:02.804916 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:02.801725 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:02.801772 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 785980us
I20260709 12:17:02.808667 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:02.810072 13259 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:02.797418 24452 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.770572 24490 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.820426 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:02.825960 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 333184us
I20260709 12:17:02.775336 12565 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:17:02.822974 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:02.809165 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.822393 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.816588 12644 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 37.15, Last appended by leader: 15, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.820216 24483 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:02.781876 24539 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:02.831408 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.769948 24382 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.056s
I20260709 12:17:02.830667 12644 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:02.828064 12646 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:02.846189 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:02.847276 12646 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:02.842286 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.848896 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.834290 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.850961 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:02.852263 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.791443 24450 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:17:02.798471 24541 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:17:02.764907 24328 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.044s
I20260709 12:17:02.802625 24475 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:17:02.894337 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65962.2us
I20260709 12:17:02.903139 13257 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:02.790566 24400 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.801391 24481 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:02.840986 12641 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:02.839658 24539 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.801594 24513 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.802332 24451 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.798074 24411 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:02.935184 12644 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:02.935618 24411 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:17:02.936652 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.797596 24215 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.854280 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.939776 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.940712 12646 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:02.943284 12491 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
W20260709 12:17:02.904381 24419 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.003s
I20260709 12:17:02.801877 24333 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:02.862553 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1727) took 1963 ms. Trace:
W20260709 12:17:02.864198 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3696) took 2090 ms (client timeout 1504 ms). Trace:
I20260709 12:17:02.864739 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1478) took 1262 ms. Trace:
I20260709 12:17:02.865497 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:02.866827 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2289) took 1005 ms. Trace:
W20260709 12:17:02.959057 12493 rpcz_store.cc:269] 0709 12:17:00.773504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.773684 (+   180us) service_pool.cc:224] Handling call
0709 12:17:02.864003 (+2090319us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.864121 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3473763712}
I20260709 12:17:02.905084 24405 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:02.881105 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:02.855479 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:02.891474 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1476) took 1174 ms. Trace:
I20260709 12:17:02.892649 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:02.837342 24452 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:02.979077 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86551.3us
I20260709 12:17:02.899533 12492 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:17:02.903668 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:02.837380 24483 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:02.854063 24519 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.909487 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:03.000895 12641 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:02.800091 24504 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:02.910245 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.848446 24331 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:17:02.912238 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.912117 24481 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:02.857003 24317 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:02.901772 24500 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:02.924808 24521 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:17:02.925781 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:02.796816 24441 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.934574 13257 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:02.934444 24451 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.849965 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:02.937408 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:02.986100 24435 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.039811 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:02.874743 24271 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:02.959749 13258 rpcz_store.cc:276] 0709 12:17:01.602839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.603026 (+   187us) service_pool.cc:224] Handling call
0709 12:17:01.603386 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:02.301460 (+698074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.864449 (+562989us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.864648 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1555800448}
I20260709 12:17:03.002421 24483 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:02.971163 12496 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:02.962152 12571 rpcz_store.cc:276] 0709 12:17:01.861093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.861228 (+   135us) service_pool.cc:224] Handling call
0709 12:17:01.911250 (+ 50022us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:02.013182 (+101932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:02.013238 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:02.013441 (+   203us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:02.013493 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.866592 (+853099us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.866641 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:02.866764 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233694592}
I20260709 12:17:02.802098 24498 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.908183 24400 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:02.983812 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:17:02.978217 13260 rpcz_store.cc:276] 0709 12:17:01.716895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.717757 (+   862us) service_pool.cc:224] Handling call
0709 12:17:02.862010 (+1144253us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:02.891366 (+ 29356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1005419264}
W20260709 12:17:02.990697 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109785us
I20260709 12:17:03.009099 24081 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:02.908277 24325 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.19, Last appended by leader: 19, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:02.997102 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.009864 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.998437 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:02.970686 24419 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:02.921569 24539 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:03.019721 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.109113 12564 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:03.022119 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109955us
I20260709 12:17:02.974124 24333 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:03.027478 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 123930us
I20260709 12:17:03.029788 24457 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:03.030046 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:03.026835 24317 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:03.031190 12567 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:02.919884 24025 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.089s
I20260709 12:17:02.997126 24474 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:02.958063 12638 rpcz_store.cc:276] 0709 12:17:00.899174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:00.899298 (+   124us) service_pool.cc:224] Handling call
0709 12:17:00.899669 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:02.225149 (+1325480us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.862299 (+637150us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:02.862485 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3027769216}
I20260709 12:17:02.956007 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:03.208148 12646 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:03.043835 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:03.209407 12563 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:03.041937 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 192038us
I20260709 12:17:03.212451 12569 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:03.015173 24328 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:02.871910 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:02.991374 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:03.013938 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:03.218504 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:02.952142 24513 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.099808 12643 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:03.098992 23540 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.002s
I20260709 12:17:03.220257 23540 consensus_queue.cc:1243] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 31.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:17:03.024000 24519 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.229254 12494 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:02.837442 24398 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.110502 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:03.044438 24271 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:03.032454 24365 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:02.947906 24490 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 38.16, Last appended by leader: 16, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.003477 24405 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.208513 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:03.113869 24543 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.336s	user 0.002s	sys 0.000s
I20260709 12:17:03.112622 12570 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:03.057614 24453 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:03.230290 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 239064us
I20260709 12:17:03.057183 24332 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:03.240167 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:03.038452 24366 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:17:03.252146 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.105928 24482 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:03.209585 24544 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.407s	user 0.002s	sys 0.000s
I20260709 12:17:03.114394 24504 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.259205 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:03.295204 12638 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:03.038843 24500 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.105669 24358 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:03.106518 24466 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.265148 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:03.302147 12642 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:03.114068 24496 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.169s
I20260709 12:17:03.267230 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:03.488255 12645 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
W20260709 12:17:03.445438 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:03.277298 12568 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:03.250006 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.529937 12638 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:03.266485 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:03.532917 12638 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 20
W20260709 12:17:03.468267 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 357701us
I20260709 12:17:03.258062 12640 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:03.291648 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:03.060350 24480 consensus_queue.cc:1175] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 34 status { last_received { term: 32 index: 22 } last_committed_idx: 22 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.23-33.23]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:17:03.534848 12536 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:17:03.535429 12536 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:17:03.535530 12568 raft_consensus.cc:1217] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Deduplicated request from leader. Original: 27.17->[30.18-33.19]   Dedup: 30.18->[33.19-33.19]
I20260709 12:17:03.536484 12568 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [30.18-33.19]
I20260709 12:17:03.484179 12642 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:03.538017 24387 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.269148 12644 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:03.294133 12646 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:03.539398 24387 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:03.302186 13218 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.925s (SENT)
I20260709 12:17:03.542723 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:03.542907 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:03.544071 13254 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:03.206563 23770 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:03.545912 24521 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:03.545763 23770 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.547089 24521 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.295413 24025 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:03.549345 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:03.549523 24025 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:03.551748 24333 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.099201 24081 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.249903 24271 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:03.207463 24317 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:03.291867 24332 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:03.590386 24561 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:17:03.591372 24332 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.257927 24365 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.601011 24365 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.296510 24366 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:03.105768 24383 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.170s
W20260709 12:17:03.602301 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 319387us
I20260709 12:17:03.603714 24383 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:03.603641 24366 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.604501 24383 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:03.351717 24419 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.606602 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:03.587219 24500 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.347009 24405 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.059196 24435 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.613224 12641 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:03.615931 12641 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 36 index: 18. (index mismatch)
I20260709 12:17:03.281534 24453 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:03.617861 24507 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.853s	user 0.012s	sys 0.004s
I20260709 12:17:03.617528 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:03.621080 12571 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 22. (term mismatch)
I20260709 12:17:03.314901 24466 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.258129 24474 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:03.629957 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1481) took 1898 ms. Trace:
I20260709 12:17:03.630652 13261 rpcz_store.cc:276] 0709 12:17:01.690598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.704347 (+ 13749us) service_pool.cc:224] Handling call
0709 12:17:01.713447 (+  9100us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:01.820238 (+106791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:01.820274 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:01.826864 (+  6590us) log.cc:844] Serialized 26 byte log entry
0709 12:17:01.918470 (+ 91606us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b48000f00d8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:17:02.741826 (+823356us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b1800169060. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:17:03.046116 (+304290us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:03.046164 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:03.046244 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:03.570389 (+524145us) raft_consensus.cc:1685] finished
0709 12:17:03.574191 (+  3802us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:03.588398 (+ 14207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86374,"raft.queue_time_us":92104,"raft.run_cpu_time_us":1199,"raft.run_wall_time_us":107054,"spinlock_wait_cycles":16753792,"thread_start_us":396698,"threads_started":2,"wal-append.queue_time_us":1432721}
I20260709 12:17:03.632910 24466 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.349360 24483 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.634369 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:03.636675 24483 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.290179 24482 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.304874 24504 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.642287 24569 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:03.643075 24569 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:03.643668 24569 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:03.644122 24504 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.395658 24513 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.318257 24516 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.393713 24519 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:03.355383 24540 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
W20260709 12:17:03.350457 24547 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:17:03.276846 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70776.5us
W20260709 12:17:03.691416 24411 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.755s	user 0.022s	sys 0.003s
I20260709 12:17:03.692203 24411 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.701469 23915 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:03.702072 23915 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:03.564291 24452 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.551918 24025 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 37.15, Last appended by leader: 15, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.555615 24496 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.007s
I20260709 12:17:03.554467 12638 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 32.21
I20260709 12:17:03.711416 12638 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:03.317729 24358 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:03.593762 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 291604us
I20260709 12:17:03.744984 24358 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.595100 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:03.746733 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.748313 23915 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.469245 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:03.750763 23915 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:03.345363 24441 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.207687 24457 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:03.754415 24457 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:03.623642 23770 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.627238 24435 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.627085 24405 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:17:03.625039 12571 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:17:03.762962 12571 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 34.22
I20260709 12:17:03.763360 12571 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:03.621372 24453 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.664207 24387 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:03.673565 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56306.2us
W20260709 12:17:03.677568 24475 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.765s	user 0.013s	sys 0.013s
I20260709 12:17:03.678073 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:03.680327 12646 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:17:03.681846 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:03.555765 24568 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36.
I20260709 12:17:03.694036 24517 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.694991 24325 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:03.697886 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:03.698884 24474 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:03.700224 24539 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.709733 13254 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:03.608079 24383 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 38.27, Last appended by leader: 27, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.719014 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:03.727608 24519 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.734519 24482 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.735589 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:03.736172 24391 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:03.736557 24381 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.921s	user 0.006s	sys 0.009s
I20260709 12:17:03.745146 24450 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:03.594714 24551 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:03.733188 24328 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:03.786258 12645 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
W20260709 12:17:03.802636 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 124852us
I20260709 12:17:03.779408 24474 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:03.800298 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3774) took 1005 ms. Trace:
I20260709 12:17:03.807756 12645 rpcz_store.cc:276] 0709 12:17:02.794836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.651302 (+856466us) service_pool.cc:224] Handling call
0709 12:17:03.719618 (+ 68316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:03.799983 (+ 80365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:03.800216 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:03.783654 24475 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.814884 24475 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.820052 24413 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.821030 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:03.822670 12497 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:03.823947 24517 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.827170 24517 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.827428 24539 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.794242 24450 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:03.821039 24413 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.831035 24450 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:03.833034 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:03.832918 24413 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.792022 24435 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.799172 24328 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:03.833169 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1732) took 1060 ms. Trace:
W20260709 12:17:03.843010 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143758us
I20260709 12:17:03.798523 24453 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.850441 24325 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.851922 24453 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.854628 24453 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:03.856091 24383 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:03.857857 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:03.861699 24003 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.129s	user 0.016s	sys 0.008s
I20260709 12:17:03.863211 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:03.864949 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:03.865180 24405 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:03.810746 24474 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:03.865859 12572 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 26. Preceding OpId from leader: term: 36 index: 27. (index mismatch)
W20260709 12:17:03.851217 24091 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.025s	user 0.009s	sys 0.020s
I20260709 12:17:03.856237 24517 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:03.832929 24450 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:03.880410 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78035us
I20260709 12:17:03.758189 24481 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.843889 12639 rpcz_store.cc:276] 0709 12:17:02.749829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.752255 (+  2426us) service_pool.cc:224] Handling call
0709 12:17:02.752737 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:02.842541 (+ 89804us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:02.842577 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:02.859597 (+ 17020us) log.cc:844] Serialized 26 byte log entry
0709 12:17:03.300190 (+440593us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:03.660028 (+359838us) log.cc:844] Serialized 14 byte log entry
0709 12:17:03.780717 (+120689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:03.794566 (+ 13849us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:03.794645 (+    79us) raft_consensus.cc:1685] finished
0709 12:17:03.794681 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:03.810578 (+ 15897us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":133658,"prepare.queue_time_us":300948,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":110,"raft.queue_time_us":144437,"raft.run_cpu_time_us":632,"raft.run_wall_time_us":44754,"thread_start_us":424220,"threads_started":2,"wal-append.queue_time_us":471438}
W20260709 12:17:03.883695 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:03.889020 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1739) took 1097 ms. Trace:
I20260709 12:17:03.884868 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:03.916229 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:03.942886 12644 rpcz_store.cc:276] 0709 12:17:02.791957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.039470 (+247513us) service_pool.cc:224] Handling call
0709 12:17:03.888589 (+849119us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:03.888804 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":525336320}
I20260709 12:17:03.857980 24539 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.948134 12497 raft_consensus.cc:2368] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36.
W20260709 12:17:03.948669 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 159811us
I20260709 12:17:03.853510 24325 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.895378 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.856082 24381 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.868400 24411 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:03.884282 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:03.894121 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1489) took 1082 ms. Trace:
W20260709 12:17:03.946305 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50328.5us
I20260709 12:17:03.881175 24568 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:17:03.876698 24405 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:03.884361 23697 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:03.901157 24521 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:03.884229 24563 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:03.843768 24519 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:03.918429 24515 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:04.005779 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:03.939195 24517 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:04.005079 24391 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:03.911688 24271 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.894498 24435 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.884235 24508 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.124s
I20260709 12:17:03.956434 24475 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:17:04.013245 12567 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:04.068996 24475 consensus_queue.cc:1243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 32.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:17:04.069921 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.059966 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4610) took 1275 ms. Trace:
I20260709 12:17:04.071736 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.071954 12640 rpcz_store.cc:276] 0709 12:17:02.784078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.851856 (+ 67778us) service_pool.cc:224] Handling call
0709 12:17:04.059535 (+1207679us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:04.059745 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":931373952}
W20260709 12:17:04.024988 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.073472 12639 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:17:04.074858 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3792) took 1167 ms. Trace:
I20260709 12:17:04.075385 12567 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 19. Preceding OpId from leader: term: 40 index: 20. (index mismatch)
I20260709 12:17:04.005115 24568 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:04.027797 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.029227 24515 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.078292 24568 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.890620 24328 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.080435 24328 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:04.028224 24391 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.087976 24391 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.030818 12563 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:04.088553 24509 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.033937 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.034520 23540 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:04.036001 24451 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:04.092249 23540 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.095093 23540 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:04.097831 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.030232 24519 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.048746 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4609) took 1264 ms. Trace:
I20260709 12:17:04.012244 24517 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:03.985793 13262 rpcz_store.cc:276] 0709 12:17:02.811216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.811419 (+   203us) service_pool.cc:224] Handling call
0709 12:17:02.823183 (+ 11764us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:03.010119 (+186936us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:03.010154 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:03.019857 (+  9703us) log.cc:844] Serialized 26 byte log entry
0709 12:17:03.260804 (+240947us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:03.260864 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:03.260992 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:03.541793 (+280801us) raft_consensus.cc:1685] finished
0709 12:17:03.541835 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:03.893419 (+351584us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5400578ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:03.893830 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5323,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":8385,"spinlock_wait_cycles":26223872,"thread_start_us":417113,"threads_started":2,"wal-append.queue_time_us":342810}
I20260709 12:17:04.057438 24398 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:17:04.058013 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.059914 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:04.063588 24498 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:04.065639 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1841) took 1438 ms. Trace:
I20260709 12:17:04.075418 12639 rpcz_store.cc:276] 0709 12:17:02.907637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.953028 (+1045391us) service_pool.cc:224] Handling call
0709 12:17:04.072911 (+119883us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:04.074367 (+  1456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:04.074778 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:04.036230 24435 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.126078 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:04.127956 12646 rpcz_store.cc:276] 0709 12:17:02.783847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.541125 (+757278us) service_pool.cc:224] Handling call
0709 12:17:03.541898 (+   773us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:03.767849 (+225951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:04.048109 (+280260us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:04.048601 (+   492us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":245188992}
I20260709 12:17:04.054126 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:03.957846 24539 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:03.949615 24091 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.024341 24563 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:03.983784 24381 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.081504 24515 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:04.137784 12488 rpcz_store.cc:276] 0709 12:17:02.626808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.763879 (+137071us) service_pool.cc:224] Handling call
0709 12:17:02.866127 (+102248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:02.866353 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:02.866377 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:02.866426 (+    49us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:02.866463 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:02.866522 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:04.065073 (+1198551us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540012b1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:04.065333 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234327808}
I20260709 12:17:04.008136 23697 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.151655 12572 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:04.168082 12571 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:04.171146 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3785) took 1366 ms. Trace:
W20260709 12:17:04.171203 24567 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.425s	user 0.001s	sys 0.005s
I20260709 12:17:04.175810 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.176301 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:04.176433 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.177466 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:04.178079 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.180584 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4603) took 1398 ms. Trace:
W20260709 12:17:04.180775 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 121007us
W20260709 12:17:04.182125 24400 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.115s	user 0.007s	sys 0.011s
I20260709 12:17:04.228258 24400 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.218292 12638 rpcz_store.cc:276] 0709 12:17:02.782375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.531577 (+749202us) service_pool.cc:224] Handling call
0709 12:17:03.532315 (+   738us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:03.832358 (+300043us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:03.832398 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:03.833194 (+   796us) log.cc:844] Serialized 52 byte log entry
0709 12:17:03.941653 (+108459us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:03.941700 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:03.941796 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:04.180163 (+238367us) raft_consensus.cc:1685] finished
0709 12:17:04.180188 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:04.180459 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5876,"raft.queue_time_us":31116,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":185,"thread_start_us":107701,"threads_started":1,"wal-append.queue_time_us":321809}
I20260709 12:17:04.229296 24271 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.230010 12567 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
W20260709 12:17:04.182915 24418 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.568s	user 0.012s	sys 0.012s
I20260709 12:17:04.230885 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.231554 24418 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.232353 12569 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:17:04.190266 24405 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.236732 24381 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.203547 24539 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:04.204881 24577 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:17:04.143875 24457 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.205772 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4608) took 1422 ms. Trace:
W20260709 12:17:04.210081 24576 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:17:04.247061 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:04.210990 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85250.2us
I20260709 12:17:04.211750 24515 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:17:04.213294 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 270353us
I20260709 12:17:04.213524 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.228747 12563 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
I20260709 12:17:04.198206 24511 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:04.211771 13257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1713) took 1433 ms (client timeout 1703 ms). Trace:
I20260709 12:17:04.278574 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.216919 12641 rpcz_store.cc:276] 0709 12:17:02.804318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.677989 (+873671us) service_pool.cc:224] Handling call
0709 12:17:03.678688 (+   699us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:03.701292 (+ 22604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:03.701344 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:03.702128 (+   784us) log.cc:844] Serialized 26 byte log entry
0709 12:17:03.849214 (+147086us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:03.849268 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:03.849360 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:04.170789 (+321429us) raft_consensus.cc:1685] finished
0709 12:17:04.170836 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:04.171060 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62245,"raft.queue_time_us":23605,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":22930,"thread_start_us":146179,"threads_started":1,"wal-append.queue_time_us":318686}
I20260709 12:17:04.289992 12490 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38.
I20260709 12:17:04.290136 24435 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:04.291513 24441 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:04.292140 24317 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.543s	user 0.010s	sys 0.007s
I20260709 12:17:04.292965 24317 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.293488 13260 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:04.230976 24400 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.298202 13258 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:04.309700 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:04.265724 24511 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:04.275862 13257 rpcz_store.cc:269] 0709 12:17:02.778290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.778441 (+   151us) service_pool.cc:224] Handling call
0709 12:17:04.211313 (+1432872us) spinlock_profiling.cc:243] Waited 93.5 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:04.211523 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215107968}
W20260709 12:17:04.275265 24569 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.006s
I20260709 12:17:04.292304 24441 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.325641 12572 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:04.323886 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:04.323081 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.335870 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.332070 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.337296 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.336608 12569 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:04.244037 12642 rpcz_store.cc:276] 0709 12:17:02.783616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.538672 (+755056us) service_pool.cc:224] Handling call
0709 12:17:03.539387 (+   715us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:03.583334 (+ 43947us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:17:03.721302 (+137968us) log.cc:844] Serialized 14 byte log entry
0709 12:17:03.869841 (+148539us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:03.869894 (+    53us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:03.869941 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:04.205364 (+335423us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:04.205442 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:04.205691 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3838,"prepare.queue_time_us":18173,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":40243,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":113,"spinlock_wait_cycles":34944384,"thread_start_us":186325,"threads_started":2,"wal-append.queue_time_us":233224}
I20260709 12:17:04.322489 24511 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:04.299450 24539 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:04.296073 24317 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:04.351020 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.347563 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:04.359522 23540 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:17:04.360492 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3787) took 1539 ms. Trace:
I20260709 12:17:04.361181 23540 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:04.359556 12609 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:04.362440 23540 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:04.342975 24332 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.012s	sys 0.018s
I20260709 12:17:04.334941 24562 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:04.371703 13254 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:17:04.375598 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.322481 24509 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:17:04.331476 24569 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:04.389251 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:04.401436 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 376174us
I20260709 12:17:04.402498 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.381280 23540 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.361235 12643 rpcz_store.cc:276] 0709 12:17:02.821205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.780170 (+958965us) service_pool.cc:224] Handling call
0709 12:17:03.849963 (+ 69793us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:04.006408 (+156445us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:17:04.134745 (+128337us) log.cc:844] Serialized 14 byte log entry
0709 12:17:04.359755 (+225010us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:04.359843 (+    88us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:04.359913 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:04.360018 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:04.360205 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6516,"prepare.queue_time_us":128092,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":9373,"raft.queue_time_us":22269,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"thread_start_us":185859,"threads_started":1}
I20260709 12:17:04.405670 12571 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:17:04.383708 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.415132 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:04.403314 24332 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.012s	sys 0.018s
I20260709 12:17:04.382891 24418 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.426649 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:04.298796 24563 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:04.431687 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72282.7us
I20260709 12:17:04.430212 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:04.440860 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 162503us
I20260709 12:17:04.446457 12568 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:04.387749 24539 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.399804 24474 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.466174 23540 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:04.468360 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.439673 24317 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:04.485018 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.491633 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.460024 24413 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:17:04.495042 24484 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:04.495694 24484 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:04.496099 24484 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:04.455127 24515 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:04.487066 12564 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:04.470681 24450 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:04.510828 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.512684 12645 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [36.25-36.25]
I20260709 12:17:04.514532 12571 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 39 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 18.
W20260709 12:17:04.520407 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:04.520748 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50564.2us
I20260709 12:17:04.521951 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.522182 24457 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:04.528729 24587 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 12:17:04.532651 24383 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.676s	user 0.008s	sys 0.004s
I20260709 12:17:04.533543 24509 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:04.534409 24509 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.535226 24539 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.535605 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:04.536130 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.537086 12564 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:04.537174 24509 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.538314 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:04.538600 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.542495 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.543445 13258 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:04.543977 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.545459 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.547008 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.555182 23659 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:04.555915 23659 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:04.556330 23659 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:04.561096 24560 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:04.562496 24441 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:04.463034 24562 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.519475 24317 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.511822 24517 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.574503 24517 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.519291 24091 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.518867 24081 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:04.578296 24474 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.539614 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.585297 12564 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:04.590703 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:04.593113 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.514853 24450 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:04.595271 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:04.601432 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 146228us
I20260709 12:17:04.618964 12569 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:04.620514 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.622324 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:04.622399 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81504.2us
I20260709 12:17:04.623848 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.623961 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:04.626010 24475 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:04.534431 24569 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.630086 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:04.630455 13258 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:04.631106 12571 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 25: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:04.632305 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.603080 24450 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:04.641757 24562 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.642745 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:04.644577 24474 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:04.645356 24474 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:04.646968 24474 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.650286 13262 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20260709 12:17:04.612408 24517 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:04.653467 24517 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:04.654135 24517 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:04.614108 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:04.679049 13254 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:17:04.677958 24498 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.515s	user 0.006s	sys 0.008s
W20260709 12:17:04.695386 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.694408 23540 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.007s
W20260709 12:17:04.692446 24580 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.775s	user 0.000s	sys 0.006s
I20260709 12:17:04.691295 24081 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260709 12:17:04.709472 24490 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:04.711032 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.722280 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:04.730981 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:04.734300 24499 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:04.735289 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.736863 12638 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 27.
I20260709 12:17:04.738412 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:04.736450 24499 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:04.747733 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.749588 24003 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:04.752460 24511 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.752818 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:04.754083 12638 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:04.756853 24081 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:04.757175 24564 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:04.757619 24081 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.758121 12571 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:04.760361 24081 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.762341 24498 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:04.762934 24498 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:04.763329 24498 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:04.725229 24091 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:04.766049 24091 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.768572 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.769347 12644 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:04.769763 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:04.773909 12640 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:17:04.786327 12638 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:04.735353 24317 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:04.794711 12643 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:04.758247 24564 consensus_queue.cc:1243] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 31.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:17:04.789230 24511 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:04.802872 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119417us
I20260709 12:17:04.807185 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.808143 12644 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:04.799521 24383 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:04.839578 12637 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 35 because replica has last-logged OpId of term: 33 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:17:04.842180 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:04.845953 24317 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:04.846487 24317 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:04.846884 24317 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:04.855476 12644 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 18. Preceding OpId from leader: term: 40 index: 20. (index mismatch)
I20260709 12:17:04.857331 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.858349 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:04.866783 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:04.838433 23659 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:04.828770 24405 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.013s	sys 0.032s
I20260709 12:17:04.878561 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:04.880474 23659 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.887328 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:04.876593 24405 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.013s	sys 0.032s
I20260709 12:17:04.897933 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:04.871407 24383 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.919766 12641 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:04.911247 12644 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:04.897022 24450 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.894901 24091 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.922508 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:04.928272 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:04.948623 12569 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:04.951310 13262 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:17:04.952684 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107944us
I20260709 12:17:04.954195 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:04.955442 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1843) took 2328 ms (client timeout 2216 ms). Trace:
I20260709 12:17:04.956068 24383 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:04.955964 12490 rpcz_store.cc:269] 0709 12:17:02.627303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.636431 (+  9128us) service_pool.cc:224] Handling call
0709 12:17:04.955120 (+2318689us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:04.955328 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3578227072}
I20260709 12:17:04.961050 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.965881 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:04.962321 13261 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:04.969692 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:04.967967 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:04.976298 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:04.977536 12488 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 38 because replica has last-logged OpId of term: 36 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:17:04.974063 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:04.981345 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110480us
I20260709 12:17:04.982295 24323 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:04.982450 13253 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:04.982997 24323 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:04.951951 24091 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:04.984236 12489 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:04.985024 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:17:04.988719 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:04.989892 12495 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20260709 12:17:04.989472 24091 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:04.992441 24091 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:04.995380 24481 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:04.996558 13257 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:05.017189 12569 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:17:05.019410 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.015225 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.023110 12492 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:05.023716 12566 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:05.024832 24564 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.025557 24564 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.023252 24474 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.026844 24474 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.028136 24564 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.029152 24474 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:05.029963 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.033403 24323 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.033922 24564 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:05.034749 24564 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:05.035627 24564 consensus_queue.cc:1243] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 33.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:17:05.031064 12644 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:05.031734 12490 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:05.045373 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:05.046379 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 448871us
I20260709 12:17:05.046671 12644 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 15. Preceding OpId from leader: term: 39 index: 16. (index mismatch)
W20260709 12:17:05.048334 24597 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.277s	user 0.002s	sys 0.000s
I20260709 12:17:05.050485 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:05.031718 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3790) took 2187 ms. Trace:
I20260709 12:17:05.052342 12645 rpcz_store.cc:276] 0709 12:17:02.844885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.862534 (+1017649us) service_pool.cc:224] Handling call
0709 12:17:03.863023 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:04.515649 (+652626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:05.031426 (+515777us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:05.031646 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1483253632}
W20260709 12:17:05.014068 24484 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.507s	user 0.012s	sys 0.003s
I20260709 12:17:05.050345 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:05.056932 13258 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:05.061066 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:05.065500 12646 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:05.065542 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46432.4us
I20260709 12:17:05.066429 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:05.066908 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.073405 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:05.078334 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.074237 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:05.081524 12641 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
W20260709 12:17:05.086287 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.056212 23697 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.063264 24329 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:17:05.087984 12490 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:05.088305 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.088591 12642 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:17:05.089610 13257 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 36 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 24.
I20260709 12:17:05.090912 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.096232 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:05.087585 24568 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:05.099619 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:05.101353 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.065191 24484 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.112808 24484 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.093094 24329 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:05.114837 24329 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.116729 24329 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:17:05.118067 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.120947 12567 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:17:05.122784 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:05.131831 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 80875.5us
I20260709 12:17:05.133422 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:05.134238 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4611) took 2349 ms (client timeout 2024 ms). Trace:
I20260709 12:17:05.134264 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:05.098790 24323 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:05.138809 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.145612 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:05.129276 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.097723 24413 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:17:05.143667 12637 rpcz_store.cc:269] 0709 12:17:02.784393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:02.808900 (+ 24507us) service_pool.cc:224] Handling call
0709 12:17:05.133840 (+2324940us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:05.134116 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4345864064}
I20260709 12:17:05.155102 12569 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:17:05.143850 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.154999 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.160823 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.100946 24475 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.051s
W20260709 12:17:05.152956 24603 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.268s	user 0.001s	sys 0.004s
I20260709 12:17:05.098423 24091 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:17:05.142254 24508 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:17:05.156584 24517 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.007s
I20260709 12:17:05.107371 24568 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.172272 12567 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:17:05.175407 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:05.178133 12564 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.513s	user 0.013s	sys 0.002s
I20260709 12:17:05.179255 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:05.182570 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.170769 24508 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:05.188097 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44380.9us
I20260709 12:17:05.189240 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.186493 12565 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:05.195261 24511 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:05.195843 24511 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:05.196233 24511 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:05.186131 24563 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:05.198091 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.198293 24563 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.199512 24490 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.200112 24490 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.200989 24563 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.201380 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.201733 23659 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.184136 24381 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:05.202368 23659 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.202876 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4796) took 3707 ms. Trace:
I20260709 12:17:05.202523 24381 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.203298 12566 rpcz_store.cc:276] 0709 12:17:01.495421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:01.554143 (+ 58722us) service_pool.cc:224] Handling call
0709 12:17:01.554430 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:05.016128 (+3461698us) spinlock_profiling.cc:243] Waited 945 us on lock 0x7b54001007a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:17:05.024310 (+  8182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:05.202419 (+178109us) spinlock_profiling.cc:243] Waited 2.78 s on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:05.202724 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6397276928}
I20260709 12:17:05.208416 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.209441 12490 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
I20260709 12:17:05.209718 23540 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:05.210327 23540 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.210573 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.212704 23540 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:05.216120 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.217197 24568 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.186920 24508 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:05.219688 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2302) took 1360 ms. Trace:
W20260709 12:17:05.222190 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82766.2us
I20260709 12:17:05.224886 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.225607 12567 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:17:05.225939 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:05.226874 12638 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 33 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:17:05.228583 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4811) took 1015 ms. Trace:
I20260709 12:17:05.229055 12572 rpcz_store.cc:276] 0709 12:17:04.213478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:04.230526 (+ 17048us) service_pool.cc:224] Handling call
0709 12:17:05.228363 (+997837us) spinlock_profiling.cc:243] Waited 88.3 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:05.228519 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":203067264}
I20260709 12:17:05.192348 24498 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.230907 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:05.231880 12496 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:05.232322 24474 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.232849 12639 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:05.233022 24474 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.232647 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.208268 24317 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.238916 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:05.240046 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50943.6us
I20260709 12:17:05.220230 12570 rpcz_store.cc:276] 0709 12:17:03.859117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:03.859344 (+   227us) service_pool.cc:224] Handling call
0709 12:17:04.090440 (+231096us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:04.150885 (+ 60445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:04.150941 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:04.235780 (+ 84839us) log.cc:844] Serialized 52 byte log entry
0709 12:17:04.480882 (+245102us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:04.514297 (+ 33415us) log.cc:844] Serialized 14 byte log entry
0709 12:17:05.037259 (+522962us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b18004a0b60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:17:05.037480 (+   221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:05.037566 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:05.047769 (+ 10203us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b24001523d8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:17:05.108391 (+ 60622us) raft_consensus.cc:1685] finished
0709 12:17:05.108425 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:05.219439 (+111014us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b5400045450. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:05.219568 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110529,"prepare.queue_time_us":32989,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":74,"raft.queue_time_us":590459,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":17919,"spinlock_wait_cycles":542750592,"thread_start_us":244491,"threads_started":1,"wal-append.queue_time_us":498923}
I20260709 12:17:05.242564 23659 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:05.241235 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141762us
I20260709 12:17:05.245214 12488 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:05.247484 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.174495 24517 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.253443 12490 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:17:05.254034 12496 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:05.257836 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.258540 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.241685 24081 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:05.261297 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:05.238930 24413 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.266320 12492 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:17:05.269112 13254 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:17:05.259055 24271 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.012s
I20260709 12:17:05.272233 23540 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:05.272608 24498 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:05.265698 24312 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:05.274499 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.275266 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:05.197185 24499 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.278316 24499 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:05.269389 24569 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:17:05.284668 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:05.285538 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.287325 13254 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:05.279067 24517 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.262552 24081 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:05.305866 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.308051 12490 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:05.318148 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.305874 24552 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:05.303664 24312 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:05.303952 24317 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.308144 24081 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:17:05.324980 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.326500 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.329010 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.308259 24569 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:05.343482 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.344377 24569 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.344975 24496 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:05.347064 24003 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:05.322579 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.332134 12497 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:05.336778 24312 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.314967 24474 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:05.347453 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.348100 12491 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:05.308714 24568 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:17:05.349045 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83986us
I20260709 12:17:05.357477 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.329998 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.312566 24323 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.353595 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.320533 24490 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.358486 12494 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:05.348687 24498 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:05.351718 24003 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:05.378897 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93487.4us
I20260709 12:17:05.375912 12491 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 26. Preceding OpId from leader: term: 36 index: 27. (index mismatch)
I20260709 12:17:05.375538 24474 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:05.381877 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.384881 12490 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:05.387526 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.390563 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.390901 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.356196 23697 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:17:05.415974 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.412788 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:05.417851 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 179308us
I20260709 12:17:05.435367 12610 leader_election.cc:400] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36.
I20260709 12:17:05.435750 12610 leader_election.cc:403] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:17:05.414242 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.418794 12488 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 23. (term mismatch)
I20260709 12:17:05.417984 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.436959 12488 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 22
I20260709 12:17:05.430701 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.438848 12488 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 36.23
I20260709 12:17:05.434316 12567 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:05.430967 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.439169 12488 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:05.439155 12490 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:17:05.406798 12494 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:17:05.441062 12564 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:05.434881 12497 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:05.452296 24382 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:05.453013 24382 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:05.417562 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:05.450691 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96499.9us
I20260709 12:17:05.455928 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.431209 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:05.458637 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.438555 12491 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:05.460909 12645 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:17:05.408022 24490 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.453454 24382 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:05.470366 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.421139 24450 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.436950 23697 consensus_queue.cc:1243] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 31.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:17:05.473619 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.470381 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.387001 24511 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.501783 12564 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:05.502589 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67380.9us
I20260709 12:17:05.503363 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:05.465022 24613 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:05.460114 24474 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36.
I20260709 12:17:05.393047 24498 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.505357 12494 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:17:05.510814 24474 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.508669 12495 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:05.511588 24474 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.514724 24474 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:05.422726 24508 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.516537 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.386796 24413 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.522490 12569 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:17:05.503453 24271 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:05.502930 24499 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.410162 24003 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:05.526500 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 109038us
I20260709 12:17:05.526185 24312 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
W20260709 12:17:05.526266 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149331us
I20260709 12:17:05.501379 24615 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:05.535970 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.506028 24613 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:05.539503 12572 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:05.545557 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.518611 24450 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:05.545749 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.553860 23540 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20260709 12:17:05.548997 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44462.1us
I20260709 12:17:05.548866 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:05.561586 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.563189 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.568095 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:05.554347 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.571451 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:05.572803 12570 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:05.573063 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.544073 24271 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.572903 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.575613 23540 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:05.568641 24317 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:05.524564 24490 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.599159 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.600102 12564 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:05.527166 24081 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:05.600708 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.547025 24613 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:05.542160 24618 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:05.560735 24568 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:05.597478 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.606909 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.506580 24323 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.611167 23540 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:17:05.612993 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:05.614059 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58900.9us
I20260709 12:17:05.615484 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.544770 24499 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:05.592365 24611 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.219s	user 0.001s	sys 0.001s
I20260709 12:17:05.622504 12570 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 24. Current term is 25. Ops: []
W20260709 12:17:05.619000 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116054us
I20260709 12:17:05.627251 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.577008 24474 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:05.592952 24511 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.629575 12564 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:17:05.595463 24003 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:05.591845 24317 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.594432 24508 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.612411 24618 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:05.641191 12570 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 17. (term mismatch)
I20260709 12:17:05.573957 24450 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.641883 23540 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:05.625147 24091 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:05.577432 24615 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.059s
W20260709 12:17:05.647292 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.647351 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.643788 12570 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 16
I20260709 12:17:05.650352 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.650280 12570 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 31.17
I20260709 12:17:05.636564 24612 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:05.651532 12570 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:05.652428 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:05.644068 24617 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:17:05.641804 24475 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:17:05.672101 24450 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.674527 24317 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.676729 24081 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.642445 24618 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:05.659346 24474 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:17:05.678740 24499 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.682221 24481 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:17:05.685946 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 88602.1us
I20260709 12:17:05.686269 24613 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.687178 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.687479 12494 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:05.648213 24091 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:05.688308 23540 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:05.689914 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:05.698429 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:05.710335 12564 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
W20260709 12:17:05.714856 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53399.2us
I20260709 12:17:05.728101 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:05.660578 24610 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:05.730870 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:05.735309 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88057.6us
I20260709 12:17:05.736819 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:05.682336 24620 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:17:05.711005 24091 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:05.710457 24511 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:17:05.744460 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 224) took 165477 ms (client timeout 15000 ms). Trace:
I20260709 12:17:05.744916 24399 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:17:05.658838 24391 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.071s
I20260709 12:17:05.726951 24475 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.729583 24498 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.726658 24626 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:17:05.753619 23540 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:05.763454 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4993) took 1173 ms. Trace:
W20260709 12:17:05.753734 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66626.4us
W20260709 12:17:05.756312 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 114876us
I20260709 12:17:05.733799 24312 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.734189 24610 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:05.768558 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:05.770076 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72993.1us
W20260709 12:17:05.747277 24329 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.630s	user 0.014s	sys 0.004s
W20260709 12:17:05.750360 12394 rpcz_store.cc:269] 0709 12:14:20.084505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:14:20.084699 (+   194us) service_pool.cc:224] Handling call
0709 12:17:05.552502 (+165467803us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2440265,"reported_tablets":51}
I20260709 12:17:05.774905 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.731931 24612 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:17:05.753201 24399 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:05.755674 24405 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:05.765442 12568 rpcz_store.cc:276] 0709 12:17:04.590595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:04.695924 (+105329us) service_pool.cc:224] Handling call
0709 12:17:04.700391 (+  4467us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:05.072086 (+371695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:05.072120 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:05.107689 (+ 35569us) log.cc:844] Serialized 527 byte log entry
0709 12:17:05.183427 (+ 75738us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:05.183474 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:05.183554 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:05.763201 (+579647us) raft_consensus.cc:1685] finished
0709 12:17:05.763225 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:05.763388 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":407105,"raft.queue_time_us":367515,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":2424,"thread_start_us":57377,"threads_started":1,"wal-append.queue_time_us":203827}
I20260709 12:17:05.749387 24624 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:05.786216 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 314508us
I20260709 12:17:05.751693 24081 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:17:05.796408 24413 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.010s
I20260709 12:17:05.796689 24630 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:05.802695 24630 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:05.803221 24630 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:05.806073 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:05.808054 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.808969 24511 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:05.809567 24511 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:05.797199 24312 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.810619 24511 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.812528 24312 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:05.809681 24600 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.855s	user 0.011s	sys 0.000s
W20260709 12:17:05.813740 24627 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:17:05.791115 24405 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:05.818936 24405 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.823158 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:05.828254 24613 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.773352 24610 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:05.732764 24271 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.786623 24435 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:05.785159 24399 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:17:05.764329 24474 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.035s
I20260709 12:17:05.791842 24498 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:05.842079 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.844240 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.831496 23697 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.846086 24632 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.008s
I20260709 12:17:05.846347 23697 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.769086 24618 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.829228 24615 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:05.772286 24382 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.840219 24081 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:05.859163 24625 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:05.850778 24329 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:05.878852 24081 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:05.882834 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:05.857543 23697 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.889129 13254 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:05.839365 24508 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:17:05.874768 24435 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.879607 24615 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:05.898905 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:05.901928 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 172638us
I20260709 12:17:05.904201 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:05.894814 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.889474 24081 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:05.876938 24475 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:05.922544 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.923607 24638 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:05.926690 12564 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:17:05.944209 12564 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 31.21
I20260709 12:17:05.944572 12564 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:05.929474 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.937036 24511 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:05.892999 24312 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:05.941859 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:05.952742 13258 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:05.858220 24630 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:05.946027 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51296.4us
I20260709 12:17:05.946668 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.947474 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.968755 13260 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:05.947528 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.903250 24490 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:05.977666 12493 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.23-42.23]
I20260709 12:17:05.970649 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.980887 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:05.979699 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.983160 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:05.983983 13262 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:05.984364 23540 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:05.984848 23540 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:05.985167 23540 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:05.985332 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:05.968050 13253 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:05.988535 13259 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:05.989507 24519 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:17:05.990825 24382 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:05.990648 24615 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.991837 24382 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:05.990828 13255 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:17:05.950945 24474 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:05.998010 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:05.963387 24612 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:06.008986 13258 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:05.992412 24382 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.986148 24625 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:06.011982 24382 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:06.019321 24618 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:06.020704 24628 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.175s	user 0.002s	sys 0.000s
I20260709 12:17:06.017488 13262 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:05.941526 24498 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:17:05.938935 24091 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.025990 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:05.996127 24564 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:05.997193 24490 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:06.042220 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:05.980497 24312 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:06.034231 24612 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:06.071697 23540 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:05.985867 24329 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:17:06.080950 24499 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:06.068317 24490 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:06.099470 24499 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:06.032666 24435 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.115887 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:06.126048 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83935.5us
I20260709 12:17:06.127422 12610 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:17:06.127993 12610 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:06.123720 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:06.140993 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1747) took 1085 ms. Trace:
I20260709 12:17:06.141584 12645 rpcz_store.cc:276] 0709 12:17:05.056112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:05.221784 (+165672us) service_pool.cc:224] Handling call
0709 12:17:05.222180 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:05.461413 (+239233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:06.138753 (+677340us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:06.140916 (+  2163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547872640}
I20260709 12:17:06.157552 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:06.152796 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:06.159936 12569 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:06.161772 12569 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
W20260709 12:17:06.118086 24633 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
W20260709 12:17:06.159507 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 133621us
I20260709 12:17:06.157316 24381 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:06.224948 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:06.225787 13262 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:06.141762 24450 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:06.122646 24564 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:06.105418 24612 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.139186 24635 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:06.179801 23540 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:06.179123 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 176154us
W20260709 12:17:06.236907 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59468.3us
W20260709 12:17:06.256518 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:06.270805 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 333397us
W20260709 12:17:06.272073 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 503543us
I20260709 12:17:06.144917 24329 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:06.303757 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:06.280751 24648 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:06.288058 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:06.285313 24641 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.183s	user 0.002s	sys 0.000s
I20260709 12:17:06.287586 23659 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Committing config change with OpId 30.19: config changed from index 7 to 19, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:06.101583 24490 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.19, Last appended by leader: 19, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.145468 24517 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:06.158532 24630 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:06.339147 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:06.330410 24517 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:06.306597 24450 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:06.350118 24496 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.005s
I20260709 12:17:06.300052 24564 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:06.309854 24648 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:06.351714 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 224516us
W20260709 12:17:06.357733 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51978.7us
I20260709 12:17:06.355655 13253 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:17:06.359685 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:06.365785 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:06.368477 23540 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:17:06.373484 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.374128 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:06.375496 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:06.377884 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2321) took 1034 ms. Trace:
I20260709 12:17:06.378588 12572 rpcz_store.cc:276] 0709 12:17:05.343207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:05.347062 (+  3855us) service_pool.cc:224] Handling call
0709 12:17:06.377547 (+1030485us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001b2bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:06.377791 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423758080}
I20260709 12:17:06.364894 24323 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.017s
I20260709 12:17:06.380306 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.380726 24323 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:06.381186 12492 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:06.360987 24636 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.365s	user 0.000s	sys 0.002s
I20260709 12:17:06.382844 12490 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38.
I20260709 12:17:06.403491 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:06.355700 24517 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:06.401182 12460 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:17:06.413666 12460 leader_election.cc:403] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:06.355165 24632 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:06.381276 24323 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:06.350984 24635 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:06.351446 24640 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:06.416029 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:06.416152 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.361326 24413 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:06.410012 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44362.6us
I20260709 12:17:06.419041 24655 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:06.415591 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.363299 24496 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 34.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:17:06.366156 24564 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:06.497736 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81813.9us
I20260709 12:17:06.421180 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:06.314908 24329 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.367103 24648 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:06.407732 24091 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:06.473500 24623 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.004s
I20260709 12:17:06.508687 23540 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:17:06.429795 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:06.426828 24652 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:17:06.489110 24626 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.707s	user 0.007s	sys 0.003s
I20260709 12:17:06.427136 24651 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.063s
I20260709 12:17:06.491638 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:06.433912 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:06.352324 24618 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.064s
W20260709 12:17:06.529377 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95575.4us
I20260709 12:17:06.530615 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:06.532336 23540 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:17:06.532939 23540 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:06.533341 23540 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:17:06.500531 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 212588us
I20260709 12:17:06.444222 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 4994) took 1853 ms. Trace:
I20260709 12:17:06.503662 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:06.435389 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:06.429273 13218 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38.
I20260709 12:17:06.536774 13218 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:06.420742 24632 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:06.489424 24657 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:17:06.516206 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:06.507437 24508 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:17:06.455651 24653 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:06.420572 24383 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.088s
I20260709 12:17:06.450692 24312 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:06.459121 24450 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:06.556910 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:06.465512 24081 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:06.442806 24650 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 35.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.204s
I20260709 12:17:06.501194 24613 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.058s
I20260709 12:17:06.534890 12563 rpcz_store.cc:276] 0709 12:17:04.590944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:04.935108 (+344164us) service_pool.cc:224] Handling call
0709 12:17:04.935557 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:05.094221 (+158664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:05.094276 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:05.119439 (+ 25163us) log.cc:844] Serialized 26 byte log entry
0709 12:17:05.750834 (+631395us) spinlock_profiling.cc:243] Waited 6.94 ms on lock 0x7b18003c96a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:17:05.853022 (+102188us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:05.853062 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:05.853148 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:06.443774 (+590626us) raft_consensus.cc:1685] finished
0709 12:17:06.443800 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:06.444125 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":188551,"raft.queue_time_us":161948,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":3127,"spinlock_wait_cycles":15956224,"thread_start_us":100764,"threads_started":1,"wal-append.queue_time_us":762518}
W20260709 12:17:06.575103 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.012s	sys 0.015s
W20260709 12:17:06.579968 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.012s	sys 0.016s
I20260709 12:17:06.576819 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.582659 12643 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:06.510156 24655 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:06.589857 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 160828us
I20260709 12:17:06.593931 12569 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:17:06.549281 24653 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:06.595409 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.595760 24653 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:06.534104 24626 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:17:06.559944 24399 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.703s	user 0.015s	sys 0.000s
I20260709 12:17:06.518529 24435 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:06.547164 24632 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:06.597898 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:06.558089 24091 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 23 FOLLOWER]: Advancing to term 25
W20260709 12:17:06.652390 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54578us
I20260709 12:17:06.650821 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.598724 12645 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 15. Preceding OpId from leader: term: 38 index: 16. (index mismatch)
I20260709 12:17:06.597321 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.500721 24640 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:06.610206 24616 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.036s	user 0.000s	sys 0.004s
I20260709 12:17:06.601711 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.675544 12638 raft_consensus.cc:2376] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30. Re-sending same reply.
I20260709 12:17:06.570145 24624 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38.
I20260709 12:17:06.617614 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.682220 12637 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:06.654795 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.605026 24081 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:06.686093 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:06.686965 23659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59760 (request call id 0) took 11444 ms. Trace:
W20260709 12:17:06.589093 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 153787us
I20260709 12:17:06.690229 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:06.593164 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:06.568674 24383 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:17:06.654764 24475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.022s	sys 0.047s
I20260709 12:17:06.644210 24003 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:06.571508 24649 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:17:06.599994 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8234424769998441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
I20260709 12:17:06.451133 24413 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:06.614084 24610 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.775s	user 0.005s	sys 0.015s
I20260709 12:17:06.708791 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:06.662582 23540 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.672945 12640 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:06.596683 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.677973 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.711622 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:06.678619 24312 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:06.680809 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:06.562546 24625 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.552s	user 0.009s	sys 0.003s
I20260709 12:17:06.618448 24613 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:17:06.687683 23659 rpcz_store.cc:276] 0709 12:16:55.234639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:55.234775 (+   136us) service_pool.cc:224] Handling call
0709 12:16:56.056383 (+821608us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b54001c37a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5770520 00007ffbf57f9b42 00007ffbf57f8800 00007ffbf57f19fa 00007ffbfa7b94c9 00007ffbf3d35c54 00007ffbf3d35be2 00007ffbf3d35b11 00007ffbf3d35a8c 00007ffbf3d34d42
0709 12:16:56.099448 (+ 43065us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b54001c37a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57704b1 00007ffbf57f8863 00007ffbf57f19fa 00007ffbfa7b94c9 00007ffbf3d35c54 00007ffbf3d35be2 00007ffbf3d35b11 00007ffbf3d35a8c 00007ffbf3d34d42 00007ffbf3b2e48c
0709 12:16:56.394876 (+295428us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b580098e1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57ee70a 00007ffbf57ef76a 00007ffbf57ee91b 00007ffbf57fa03c 00007ffbf57f96b3 00007ffbf57f19fa 00007ffbfa7b94c9 00007ffbf3d35c54
0709 12:16:56.763028 (+368152us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b4c0014ac00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3a96644 00007ffbf3abdf2d 00007ffbf3abdb93 00007ffbf574c1af 00007ffbf574bf3e 00007ffbf57d1888 00007ffbf57ee75c 00007ffbf57ef76a 00007ffbf57ee91b
0709 12:16:56.764561 (+  1533us) log.cc:844] Serialized 501 byte log entry
0709 12:17:06.655171 (+9890610us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":285867,"raft.queue_time_us":286952,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":278,"spinlock_wait_cycles":314708864}
I20260709 12:17:06.616384 24517 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.572407 24658 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.119s
I20260709 12:17:06.604552 24498 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:06.705945 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 330069us
W20260709 12:17:06.695122 24475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.022s	sys 0.047s
I20260709 12:17:06.720870 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.734769 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.711514 12639 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:17:06.727928 24625 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:06.723747 24481 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:06.725574 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132495us
W20260709 12:17:06.729403 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 172608us
I20260709 12:17:06.650149 24399 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.742807 12640 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:17:06.743933 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.744668 12638 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:06.691947 24383 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:06.734372 23540 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:06.703893 24649 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:06.745458 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:06.746654 12637 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:06.744737 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:06.684885 24381 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:17:06.743443 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.753988 12569 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:06.755954 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.719671 24610 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.761644 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.609448 24655 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 38.27, Last appended by leader: 27, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:06.766330 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105604us
I20260709 12:17:06.763499 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:06.733989 24624 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:06.707551 24003 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:06.777056 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:17:06.777441 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:06.762681 12564 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Committing config change with OpId 30.19: config changed from index 7 to 19, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:06.811257 12490 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:06.811925 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.721714 24081 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.039s
I20260709 12:17:06.606144 24635 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.769942 23540 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:06.827776 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:06.755178 24649 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:06.828893 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3743) took 1569 ms. Trace:
I20260709 12:17:06.718118 24435 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:06.831161 12380 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68172.2us
I20260709 12:17:06.837953 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:06.838529 12494 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 15. Preceding OpId from leader: term: 38 index: 16. (index mismatch)
I20260709 12:17:06.839912 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:06.842859 12493 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:06.844233 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.846696 24413 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:06.822304 24669 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:06.847666 24669 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.848832 24383 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:06.845427 12493 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:06.850370 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.844350 24640 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:06.854205 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.873157 23540 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:06.855571 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:06.856995 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:06.864295 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:06.872828 12495 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:06.876348 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:06.877270 13259 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:06.834175 12496 rpcz_store.cc:276] 0709 12:17:05.259722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:05.259842 (+   120us) service_pool.cc:224] Handling call
0709 12:17:05.267290 (+  7448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:05.321748 (+ 54458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:05.321783 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:05.326315 (+  4532us) log.cc:844] Serialized 26 byte log entry
0709 12:17:05.387487 (+ 61172us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:05.387542 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:05.387615 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:06.507744 (+1120129us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:06.805501 (+297757us) raft_consensus.cc:1685] finished
0709 12:17:06.805537 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:06.828772 (+ 23235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27001,"raft.queue_time_us":11737,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":4926,"spinlock_wait_cycles":43244288,"thread_start_us":60644,"threads_started":1,"wal-append.queue_time_us":189299}
I20260709 12:17:06.805925 24624 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:06.839735 24637 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:17:06.899262 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.900060 12497 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:06.910244 24081 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:06.911360 12496 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:17:06.911926 12494 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:17:06.916733 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:06.904842 24668 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:17:06.919164 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:06.919831 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.883096 12646 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:06.926324 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:06.929209 12639 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:06.930101 12639 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:17:06.899952 24624 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:06.934413 12566 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:17:06.932245 12488 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:06.943347 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:06.917614 24329 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:06.943389 24383 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:06.928050 24081 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.983141 12488 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 21
I20260709 12:17:06.983527 12488 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 34.22
I20260709 12:17:06.983775 12488 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:06.987205 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:06.991598 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:06.997772 23659 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.000444 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:07.001176 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:06.998706 23659 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:06.999316 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:06.991127 24669 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:07.009927 24659 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
W20260709 12:17:07.034296 24660 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:17:07.029340 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:07.040998 23659 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.037992 24679 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.009826 24517 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.018327 24678 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:07.030043 24640 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:07.053076 24662 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.217s	user 0.002s	sys 0.000s
I20260709 12:17:07.047917 24610 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.057973 24653 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.055225 24632 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.052416 24003 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.054855 24669 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:17:07.086743 24673 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:17:07.090353 24675 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:17:07.092574 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:07.094535 24517 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:07.098874 24647 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.810s	user 0.006s	sys 0.000s
W20260709 12:17:07.100015 24671 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
W20260709 12:17:07.100899 24665 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:17:07.101861 24623 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:07.104964 24081 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.106472 13259 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:17:07.106422 24091 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:17:07.106880 24435 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:07.107914 24667 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:17:07.111619 24405 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:07.113999 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:07.101488 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:07.119918 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:17:07.319209 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 205318us
I20260709 12:17:07.220592 24323 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.209674 23540 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:07.215644 24648 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.693s	user 0.005s	sys 0.005s
W20260709 12:17:07.218715 24662 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:17:07.122407 24669 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:07.204941 24564 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.683s	user 0.006s	sys 0.004s
I20260709 12:17:07.076823 24678 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:07.216428 24686 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:17:07.208670 24632 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.316794 24081 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:07.066817 24640 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.206637 24653 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.216974 24693 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:07.219282 24672 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.058246 24679 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:07.209123 24642 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.105s	user 0.004s	sys 0.000s
I20260709 12:17:07.218770 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.243480 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:07.188479 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1727) took 1256 ms. Trace:
I20260709 12:17:07.215896 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.351946 13257 rpcz_store.cc:276] 0709 12:17:05.932369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:05.946602 (+ 14233us) service_pool.cc:224] Handling call
0709 12:17:05.947030 (+   428us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:05.947491 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:05.947537 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:05.948261 (+   724us) log.cc:844] Serialized 52 byte log entry
0709 12:17:06.034501 (+ 86240us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:06.038501 (+  4000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:06.038626 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:07.088242 (+1049616us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:07.188114 (+ 99872us) raft_consensus.cc:1685] finished
0709 12:17:07.188136 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:07.188399 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5810,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":17157,"spinlock_wait_cycles":410967424,"thread_start_us":85687,"threads_started":1,"wal-append.queue_time_us":151311}
W20260709 12:17:07.142992 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 305330us
I20260709 12:17:07.218918 24399 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.201200 24312 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:07.214113 23697 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:07.333981 24669 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:07.205333 13262 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:07.357431 13262 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:17:07.212189 24381 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.267693 24329 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.184237 24517 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:07.310668 24490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.014s	sys 0.006s
W20260709 12:17:07.323405 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 221963us
I20260709 12:17:07.316920 24623 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:07.327536 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 610727us
I20260709 12:17:07.304497 24649 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.337190 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:07.303781 24624 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:07.167251 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74802.4us
I20260709 12:17:07.326625 24271 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:07.326874 24626 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:07.332572 24498 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.001s
I20260709 12:17:07.348635 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.354965 23697 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:07.211750 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:07.335160 24323 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:07.353462 12489 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:07.324558 23540 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:07.366384 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.370085 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:07.445425 12497 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:07.339709 24564 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.331899 24648 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:07.364463 24490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.015s	sys 0.006s
I20260709 12:17:07.446722 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.359470 24693 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:07.448457 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:07.471820 12567 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 33 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:17:07.372105 12572 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:17:07.373620 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:07.372560 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:07.368148 12645 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:07.452369 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:07.373023 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:07.459760 12644 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:07.459651 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:07.477445 12568 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:07.351083 24081 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.422120 12643 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:07.368474 24312 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.464305 12493 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:07.473202 12572 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:07.465917 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145934us
I20260709 12:17:07.489020 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:07.469354 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44002.5us
I20260709 12:17:07.490931 12492 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:07.368847 24672 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.494313 24640 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.495749 24081 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.496905 12564 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:07.497527 12564 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:07.498687 12564 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.20, Last appended by leader: 20, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.499938 12564 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:07.361045 24381 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.503050 24381 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.504215 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:07.505038 12643 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:07.505970 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1504) took 1092 ms. Trace:
I20260709 12:17:07.507093 12640 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:07.476743 23540 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.363979 24329 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.482963 12570 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:07.508428 12569 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:17:07.449159 24498 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:07.509804 12646 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:07.487351 12572 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.539531 12572 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:17:07.516312 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.507s (SENT) [suppressed 14 similar messages]
I20260709 12:17:07.506418 13254 rpcz_store.cc:276] 0709 12:17:06.413234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.421899 (+  8665us) service_pool.cc:224] Handling call
0709 12:17:06.504526 (+ 82627us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:06.505187 (+   661us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:17:06.518462 (+ 13275us) log.cc:844] Serialized 14 byte log entry
0709 12:17:06.711853 (+193391us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:06.734444 (+ 22591us) log.cc:844] Serialized 527 byte log entry
0709 12:17:06.734625 (+   181us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:06.734662 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:06.734754 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:07.505675 (+770921us) raft_consensus.cc:1685] finished
0709 12:17:07.505700 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:07.505893 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22607,"prepare.queue_time_us":18392,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":28581,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":17429,"thread_start_us":119498,"threads_started":1,"wal-append.queue_time_us":203119}
I20260709 12:17:07.517067 24474 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:07.517763 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:07.520545 24626 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.511991 24678 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.418555 24496 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.072s
I20260709 12:17:07.458124 23697 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.529196 12643 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:07.530200 12564 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:17:07.565618 12536 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
I20260709 12:17:07.535377 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.566030 12536 leader_election.cc:403] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:07.566442 12489 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:07.546455 12567 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:17:07.508806 24638 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:07.558169 12645 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:17:07.559402 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70547.2us
I20260709 12:17:07.571040 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.550232 24474 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:07.555392 24381 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:07.570178 24638 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:07.555121 24612 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:17:07.575098 24081 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:07.565285 24569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.446s	user 0.028s	sys 0.047s
W20260709 12:17:07.585500 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:07.580139 24568 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
W20260709 12:17:07.601388 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49819.8us
I20260709 12:17:07.602038 12568 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:07.564222 24626 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.627154 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:07.602634 24569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.483s	user 0.028s	sys 0.047s
I20260709 12:17:07.600384 24638 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:07.641759 24698 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
W20260709 12:17:07.654469 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83535.2us
I20260709 12:17:07.655413 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.659881 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:07.626890 24381 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.667266 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:07.661885 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:07.675870 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48815.7us
I20260709 12:17:07.635213 24682 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:07.675984 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.603689 24081 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.693280 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:07.649375 24697 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:07.696228 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.697083 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.701048 24499 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:07.719033 24702 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:17:07.719710 24623 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.674988 24381 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:07.766511 24499 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:07.691948 24564 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:07.721910 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65207.7us
I20260709 12:17:07.768887 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.770193 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:07.771000 24091 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:07.771472 24091 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:07.771826 24091 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:07.722517 24687 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:07.724511 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.772975 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:07.773147 24687 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:07.773533 12491 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 36 because replica has last-logged OpId of term: 35 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:17:07.773710 24687 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:07.774164 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.774930 24652 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:17:07.775496 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.775620 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.776211 12492 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:07.776453 12491 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 28: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:07.777136 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:07.724920 24329 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:07.731807 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.733250 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.589452 24693 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.736433 24626 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.737166 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.737332 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2322) took 1837 ms. Trace:
I20260709 12:17:07.737250 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:07.740913 24383 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.762s	user 0.011s	sys 0.002s
I20260709 12:17:07.743793 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1503) took 1359 ms. Trace:
I20260709 12:17:07.750150 24648 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.752617 24081 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:07.699577 24701 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:07.699339 24474 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:07.756335 24685 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.677s	user 0.002s	sys 0.000s
I20260709 12:17:07.757732 24382 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:07.774132 24687 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:07.758006 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.758495 24624 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:07.761219 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:07.763896 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:07.668908 24655 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:07.777254 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.793171 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4647) took 1046 ms. Trace:
I20260709 12:17:07.711935 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.829223 12495 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:17:07.781687 12570 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:17:07.803784 12571 rpcz_store.cc:276] 0709 12:17:05.900413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:05.901060 (+   647us) service_pool.cc:224] Handling call
0709 12:17:05.943146 (+ 42086us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:06.139579 (+196433us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:17:06.233655 (+ 94076us) log.cc:844] Serialized 14 byte log entry
0709 12:17:06.302568 (+ 68913us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:06.302626 (+    58us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:06.302674 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:07.737055 (+1434381us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:07.737109 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:07.737261 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6940,"prepare.queue_time_us":98008,"prepare.run_cpu_time_us":219,"prepare.run_wall_time_us":588,"raft.queue_time_us":7261,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":42944,"spinlock_wait_cycles":530914560,"thread_start_us":61491,"threads_started":1,"wal-append.queue_time_us":192954}
I20260709 12:17:07.829849 12570 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 39.20
I20260709 12:17:07.780767 24323 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:17:07.694379 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 178095us
I20260709 12:17:07.805598 12493 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:07.783352 23540 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:07.807567 12572 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
I20260709 12:17:07.834707 12572 pending_rounds.cc:85] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:17:07.835008 12572 pending_rounds.cc:107] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 29.16
W20260709 12:17:07.806356 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109419us
I20260709 12:17:07.836133 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:07.809317 12645 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:17:07.803192 13262 rpcz_store.cc:276] 0709 12:17:06.384922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.385108 (+   186us) service_pool.cc:224] Handling call
0709 12:17:07.743518 (+1358410us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54004b73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:07.743717 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1622912896}
I20260709 12:17:07.806826 12489 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:07.783474 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:07.811596 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:07.690078 24682 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:07.811427 24701 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:07.692857 24649 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:07.822089 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:07.848148 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.849062 12490 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 33 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:17:07.780836 24666 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.057s
I20260709 12:17:07.830314 12570 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:07.855037 12461 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
I20260709 12:17:07.777383 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.855800 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.829649 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:07.827981 12638 rpcz_store.cc:276] 0709 12:17:06.747086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.747267 (+   181us) service_pool.cc:224] Handling call
0709 12:17:06.766126 (+ 18859us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:06.770594 (+  4468us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:06.770647 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:06.823527 (+ 52880us) log.cc:844] Serialized 26 byte log entry
0709 12:17:07.010060 (+186533us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:07.304731 (+294671us) log.cc:844] Serialized 14 byte log entry
0709 12:17:07.792620 (+487889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:07.792786 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:07.792858 (+    72us) raft_consensus.cc:1685] finished
0709 12:17:07.792876 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:07.793079 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19682,"prepare.queue_time_us":329736,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":169383,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":7217,"thread_start_us":848809,"threads_started":3,"wal-append.queue_time_us":292438}
I20260709 12:17:07.811765 24663 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:07.835263 12572 raft_consensus.cc:2889] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:07.843874 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:07.802997 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:07.780328 24091 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.845356 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.862840 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.863129 12495 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:07.828061 12492 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:07.863886 12490 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:07.846585 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:07.839282 24323 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:07.852108 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.858393 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.847198 24511 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:07.872263 24672 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:07.872435 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.873678 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:07.875258 12489 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:17:07.862732 24490 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:07.878357 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:07.869668 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:07.879199 12638 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:07.871539 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:07.879279 12491 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:07.861586 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:07.850445 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67072.3us
I20260709 12:17:07.863066 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.882277 13259 raft_consensus.cc:2368] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:07.882486 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:07.842532 24635 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:07.868474 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:07.850284 24474 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:07.884366 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:07.888533 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.892372 12491 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.23-42.23]
I20260709 12:17:07.884142 13259 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:07.827075 24382 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:07.860419 24271 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:07.901808 12489 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:07.859562 24652 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:07.877120 24405 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:07.908838 13259 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:07.899258 12491 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:07.888245 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:07.901943 12611 leader_election.cc:400] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:07.913998 12611 leader_election.cc:403] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:07.847203 24701 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:07.906922 12563 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:17:07.880771 24323 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:07.915246 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.886003 24626 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:07.920531 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:07.909826 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4649) took 1146 ms. Trace:
I20260709 12:17:07.926133 12645 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:17:07.859764 24697 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:17:07.933370 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:07.938081 12638 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:07.942539 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56006.2us
I20260709 12:17:07.952556 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:07.930984 12637 rpcz_store.cc:276] 0709 12:17:06.763300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.763538 (+   238us) service_pool.cc:224] Handling call
0709 12:17:06.807632 (+ 44094us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:06.905199 (+ 97567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:06.905241 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:06.944489 (+ 39248us) log.cc:844] Serialized 26 byte log entry
0709 12:17:06.997016 (+ 52527us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:07.333614 (+336598us) log.cc:844] Serialized 14 byte log entry
0709 12:17:07.333868 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:07.333966 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:07.755595 (+421629us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:07.898319 (+142724us) raft_consensus.cc:1685] finished
0709 12:17:07.909584 (+ 11265us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:07.909756 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14424,"prepare.queue_time_us":324638,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":151505,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":13958,"spinlock_wait_cycles":5723392,"thread_start_us":454039,"threads_started":3,"wal-append.queue_time_us":100379}
W20260709 12:17:07.956004 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54226us
I20260709 12:17:07.907996 24405 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:07.960889 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:07.929877 24626 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:07.916551 24271 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:07.932688 24649 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:07.963675 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.905072 24508 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.048s
I20260709 12:17:07.906075 24490 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:17:07.957262 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:07.900266 24569 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.052s
I20260709 12:17:07.951938 24383 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:07.969008 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101880us
I20260709 12:17:07.973239 12645 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:07.981011 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.062211 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.969141 24626 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:07.940508 24312 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:17:08.072587 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:07.967070 24709 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:17:08.063040 12646 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:08.070318 12638 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
W20260709 12:17:08.082978 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.072137 12637 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:07.963683 24405 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:08.086280 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.090317 12645 raft_consensus.cc:2376] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37. Re-sending same reply.
I20260709 12:17:07.909188 24682 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:08.105038 24711 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:17:08.075593 24383 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.111899 24383 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.112402 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4657) took 1121 ms. Trace:
I20260709 12:17:08.112794 12639 rpcz_store.cc:276] 0709 12:17:06.991207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.996952 (+  5745us) service_pool.cc:224] Handling call
0709 12:17:07.012248 (+ 15296us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:07.012761 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:07.012800 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:07.017011 (+  4211us) log.cc:844] Serialized 52 byte log entry
0709 12:17:07.740812 (+723801us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b18003a3000. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:17:07.869832 (+129020us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:07.904061 (+ 34229us) log.cc:844] Serialized 14 byte log entry
0709 12:17:07.924791 (+ 20730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:07.924881 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:08.112051 (+187170us) raft_consensus.cc:1685] finished
0709 12:17:08.112185 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:08.112338 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22557,"prepare.queue_time_us":34576,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":20680,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":16061,"spinlock_wait_cycles":54075136,"thread_start_us":128728,"threads_started":1,"wal-append.queue_time_us":934474}
I20260709 12:17:08.114997 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.115607 12639 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:08.115661 24382 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:08.111968 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:08.088667 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 187353us
I20260709 12:17:08.119469 12490 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:08.116650 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.126715 12639 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 28: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:07.990235 24649 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.087617 24652 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:08.131829 12493 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
W20260709 12:17:08.112460 24703 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.434s	user 0.001s	sys 0.001s
I20260709 12:17:08.101434 24674 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:08.145035 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 158867us
I20260709 12:17:08.146595 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.148645 24091 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.150130 24383 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:08.126799 24716 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:17:08.157943 24687 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.161095 24687 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.144302 24678 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:17:08.170300 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:08.174291 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50687.2us
I20260709 12:17:08.176628 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:08.180038 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:08.177790 24569 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:08.180989 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:08.133203 24329 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:08.136051 24625 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.183645 24649 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.135610 24564 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:08.185065 24713 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:17:08.143849 24612 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:08.162463 24674 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.188428 24312 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:17:08.191005 24669 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.769s	user 0.014s	sys 0.000s
I20260709 12:17:08.191715 24718 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:08.192981 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:08.211064 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.171s (SENT)
I20260709 12:17:08.212247 12639 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:17:08.197232 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:08.212606 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:08.197552 12492 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:08.168494 24697 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:08.197146 12495 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:08.202692 24652 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:08.206416 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.157378 24663 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:08.207356 24564 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:08.198459 24638 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.531s	user 0.011s	sys 0.000s
I20260709 12:17:08.191864 24728 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.015s
I20260709 12:17:08.233976 24638 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.208987 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:08.213353 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:08.236476 24638 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.200912 24383 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:08.237578 24383 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:08.238747 24383 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.200119 24569 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:08.210098 24511 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:08.213280 24669 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.231649 24674 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.206558 24329 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.255414 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:08.247313 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:08.206797 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 303477us
I20260709 12:17:08.211722 24508 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:08.275436 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:08.207580 24625 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.237412 24623 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:08.284373 12642 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 34 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:17:08.281821 12570 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:17:08.278043 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69168.4us
I20260709 12:17:08.292583 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:08.249270 24312 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:17:08.301141 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58424.7us
I20260709 12:17:08.259493 24508 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:08.307885 12646 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:08.281463 24728 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:08.320715 24656 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.864s	user 0.004s	sys 0.001s
I20260709 12:17:08.322279 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:08.323269 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:08.324891 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:08.258720 24271 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.331729 24612 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:08.332298 24612 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:08.338436 24564 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.316577 24626 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.289464 24689 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.342047 24689 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.343145 24626 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.347059 24669 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.306989 24312 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:08.348469 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56027.2us
I20260709 12:17:08.348668 24312 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.349293 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3807) took 1318 ms. Trace:
W20260709 12:17:08.360006 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4642) took 1644 ms (client timeout 1623 ms). Trace:
I20260709 12:17:08.370882 12646 rpcz_store.cc:276] 0709 12:17:07.030934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.130908 (+1099974us) service_pool.cc:224] Handling call
0709 12:17:08.349082 (+218174us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:08.349232 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328681856}
W20260709 12:17:08.371323 12644 rpcz_store.cc:269] 0709 12:17:06.715710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.715868 (+   158us) service_pool.cc:224] Handling call
0709 12:17:08.359793 (+1643925us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:08.359924 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1660054400}
W20260709 12:17:08.354285 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.386184 12496 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:08.387782 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.390849 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:08.349903 24697 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:08.397488 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:08.350261 24312 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:08.350987 24718 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.404403 12495 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:17:08.353667 24508 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:08.394797 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 183762us
I20260709 12:17:08.410106 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.462559 12646 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:08.374589 24689 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.432940 12495 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: []
W20260709 12:17:08.494140 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:08.484313 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:08.392249 24726 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.101s	user 0.000s	sys 0.002s
I20260709 12:17:08.386523 24719 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:08.497475 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.464592 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:08.401517 24626 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.502288 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:08.359409 24701 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.515496 24709 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:08.517745 24716 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:08.518466 24716 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.521277 24716 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.521358 24729 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:08.521687 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1864) took 1547 ms. Trace:
I20260709 12:17:08.373548 24569 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:08.498029 24731 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:17:08.398136 24484 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.109s
I20260709 12:17:08.493873 24499 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:08.506175 24474 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.610s	user 0.013s	sys 0.000s
W20260709 12:17:08.504081 24323 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.566s	user 0.012s	sys 0.000s
I20260709 12:17:08.506659 24626 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.526178 24625 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.493950 24697 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:08.529981 24329 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.534348 24517 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.536919 24381 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.539577 12644 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:17:08.593276 13257 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:08.543907 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:08.547139 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.551081 12462 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 24. Current term is 25.
W20260709 12:17:08.594734 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43851.7us
I20260709 12:17:08.553745 24623 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.546396 24612 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.596290 24623 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.598803 24623 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.599052 24612 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.555683 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:08.556980 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54817.7us
I20260709 12:17:08.566705 13259 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:17:08.503564 24689 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:08.623553 12492 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: [39.16-39.16]
W20260709 12:17:08.658423 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4631) took 2115 ms (client timeout 2038 ms). Trace:
W20260709 12:17:08.659204 12642 rpcz_store.cc:269] 0709 12:17:06.542743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.542893 (+   150us) service_pool.cc:224] Handling call
0709 12:17:08.658016 (+2115123us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:08.658290 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3744846720}
I20260709 12:17:08.665009 12496 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:08.665086 12493 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 22. (term mismatch)
I20260709 12:17:08.666735 12493 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 21
I20260709 12:17:08.667014 12493 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 35.22
I20260709 12:17:08.667294 12493 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:08.578307 12639 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:08.581817 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:08.676913 12495 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:08.567864 12488 rpcz_store.cc:276] 0709 12:17:06.974405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.974518 (+   113us) service_pool.cc:224] Handling call
0709 12:17:06.982937 (+  8419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:06.984564 (+  1627us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:06.984611 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:08.110181 (+1125570us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:08.110903 (+   722us) log.cc:844] Serialized 52 byte log entry
0709 12:17:08.137045 (+ 26142us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:08.137127 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.137242 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:08.521211 (+383969us) raft_consensus.cc:1685] finished
0709 12:17:08.521244 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:08.521567 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10519,"raft.queue_time_us":59888,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":9354,"spinlock_wait_cycles":7284480,"thread_start_us":25438,"threads_started":1,"wal-append.queue_time_us":104857}
I20260709 12:17:08.576538 24729 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:08.586473 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:08.517546 24701 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.590339 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:08.497350 24718 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.584730 24323 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.600109 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:08.553426 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:08.602916 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:08.608968 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 322894us
I20260709 12:17:08.702924 12571 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:08.612679 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4651) took 1766 ms. Trace:
I20260709 12:17:08.618690 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:08.507467 24719 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.580309 24405 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.678906 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4663) took 1467 ms. Trace:
I20260709 12:17:08.599747 24569 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:08.584506 24499 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.506645 24435 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.080s
I20260709 12:17:08.677388 13255 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:08.591740 24732 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:08.705988 12494 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: [38.21-38.21]
I20260709 12:17:08.701715 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:08.713089 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.676366 24689 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.719400 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:08.719442 12639 rpcz_store.cc:276] 0709 12:17:07.211049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.140576 (+929527us) service_pool.cc:224] Handling call
0709 12:17:08.678833 (+538257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.709218 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:08.699793 24729 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:08.706851 24728 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:08.777508 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.703508 12641 rpcz_store.cc:276] 0709 12:17:06.856425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.879375 (+ 22950us) service_pool.cc:224] Handling call
0709 12:17:06.899355 (+ 19980us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:06.900087 (+   732us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:06.900121 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:06.910793 (+ 10672us) log.cc:844] Serialized 52 byte log entry
0709 12:17:06.991648 (+ 80855us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:07.310169 (+318521us) log.cc:844] Serialized 14 byte log entry
0709 12:17:07.726644 (+416475us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18003250c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:08.361128 (+634484us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b18003250c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:17:08.608408 (+247280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.608533 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:08.608586 (+    53us) raft_consensus.cc:1685] finished
0709 12:17:08.608598 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:08.612582 (+  3984us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2832,"prepare.queue_time_us":196073,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":102418,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":12158,"spinlock_wait_cycles":11271808,"thread_start_us":414580,"threads_started":3,"wal-append.queue_time_us":124592,"wal-append.run_cpu_time_us":9852,"wal-append.run_wall_time_us":1279885}
I20260709 12:17:08.620007 24669 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:08.756011 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154014us
I20260709 12:17:08.756083 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.758229 12492 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: [38.16-38.16]
I20260709 12:17:08.761752 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:08.784848 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.785941 12639 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:08.774134 24732 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:08.775758 12642 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:08.792428 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83360.2us
I20260709 12:17:08.796452 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.797411 12641 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:08.800839 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4648) took 2049 ms. Trace:
I20260709 12:17:08.801478 12640 rpcz_store.cc:276] 0709 12:17:06.750957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:06.751112 (+   155us) service_pool.cc:224] Handling call
0709 12:17:06.763636 (+ 12524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:07.509731 (+746095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.800542 (+1290811us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:08.800765 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1709279104}
I20260709 12:17:08.804173 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:08.793792 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 213845us
I20260709 12:17:08.805153 12639 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:08.811472 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.812371 12643 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:08.781805 24701 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.839087 24679 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:08.838965 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:08.842846 12640 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 34 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:17:08.854151 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.854897 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:17:08.811723 24663 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:17:08.808622 24329 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:08.789011 24728 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.877756 12641 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:17:08.863502 24742 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:17:08.895958 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.840289 24625 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:08.789275 24669 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.871994 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.876598 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:08.882977 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:08.883617 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:08.922504 13255 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:17:08.887367 12497 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:08.923628 12497 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:08.924297 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47773.6us
I20260709 12:17:08.888016 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:08.889796 24737 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 12:17:08.891440 24739 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
W20260709 12:17:08.879429 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.894702 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.930393 12641 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:08.920706 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4664) took 1709 ms. Trace:
I20260709 12:17:08.930991 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1763) took 1049 ms. Trace:
I20260709 12:17:08.931356 12641 rpcz_store.cc:276] 0709 12:17:07.881678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.894049 (+1012371us) service_pool.cc:224] Handling call
0709 12:17:08.930927 (+ 36878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.811851 24474 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:08.898716 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:08.900545 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:08.922806 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:08.933094 12493 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:08.872308 12639 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:08.903292 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:08.924829 12497 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.26, Last appended by leader: 26, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.905011 12640 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 36 because replica has last-logged OpId of term: 33 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:17:08.935329 12497 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:08.935660 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:08.937619 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:08.907897 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.943827 12640 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:17:08.944597 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3809) took 1913 ms. Trace:
I20260709 12:17:08.944945 12640 rpcz_store.cc:276] 0709 12:17:07.031574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.942892 (+1911318us) service_pool.cc:224] Handling call
0709 12:17:08.943413 (+   521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:08.944322 (+   909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.944536 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:08.946318 24738 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:17:08.947077 12638 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:08.947840 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3810) took 1916 ms. Trace:
I20260709 12:17:08.948546 24405 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.948974 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5003) took 1134 ms. Trace:
I20260709 12:17:08.950534 24323 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.949440 12491 rpcz_store.cc:276] 0709 12:17:07.814300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:07.918400 (+104100us) service_pool.cc:224] Handling call
0709 12:17:07.918793 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:08.200055 (+281262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:08.200099 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:08.211527 (+ 11428us) log.cc:844] Serialized 26 byte log entry
0709 12:17:08.556659 (+345132us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:08.582853 (+ 26194us) log.cc:844] Serialized 14 byte log entry
0709 12:17:08.583122 (+   269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.583204 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:08.948705 (+365501us) raft_consensus.cc:1685] finished
0709 12:17:08.948732 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:08.948905 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9077,"prepare.queue_time_us":26523,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":137922,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":9215,"spinlock_wait_cycles":9069440,"thread_start_us":571627,"threads_started":2,"wal-append.queue_time_us":405787}
I20260709 12:17:08.951495 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:08.948488 24735 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 12:17:08.812574 24725 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 12:17:08.862541 24383 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:08.927855 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:08.931133 12637 rpcz_store.cc:276] 0709 12:17:07.211391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.871658 (+1660267us) service_pool.cc:224] Handling call
0709 12:17:08.920621 (+ 48963us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.952438 13261 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 38 because replica has last-logged OpId of term: 36 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 24.
I20260709 12:17:08.952852 12492 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:17:08.957016 12564 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:17:08.957978 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:08.948233 12638 rpcz_store.cc:276] 0709 12:17:07.031914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.946442 (+1914528us) service_pool.cc:224] Handling call
0709 12:17:08.946851 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:08.947527 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.947776 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.940448 12642 raft_consensus.cc:2376] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37. Re-sending same reply.
I20260709 12:17:08.963361 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:08.972066 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44312.7us
I20260709 12:17:08.972334 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:08.972848 12641 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 35 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:17:08.973047 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:08.974736 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:08.975695 23540 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:17:08.809825 24732 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:08.859269 24701 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:08.968042 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:08.973848 12566 raft_consensus.cc:2368] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:17:08.973083 12639 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:17:08.915385 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:08.965565 12570 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:08.977459 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1759) took 1341 ms. Trace:
I20260709 12:17:08.978153 12642 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [34.17-36.18]
I20260709 12:17:08.978590 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:08.978988 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:08.979080 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3815) took 1931 ms. Trace:
I20260709 12:17:08.979717 12642 rpcz_store.cc:276] 0709 12:17:07.048132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.971073 (+1922941us) service_pool.cc:224] Handling call
0709 12:17:08.971666 (+   593us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:08.978653 (+  6987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:08.978998 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.980677 12610 leader_election.cc:400] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:17:08.981078 12610 leader_election.cc:403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:08.981607 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:08.977888 12639 rpcz_store.cc:276] 0709 12:17:07.646295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.952858 (+1306563us) service_pool.cc:224] Handling call
0709 12:17:08.977390 (+ 24532us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.991267 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:08.991672 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:08.993145 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:08.993744 13259 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:17:08.952140 24740 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:08.792410 24697 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.014663 12640 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: [33.20-35.22]
I20260709 12:17:09.000825 13258 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 42 index: 26. (index mismatch)
W20260709 12:17:09.016062 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 212709us
I20260709 12:17:09.003903 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4659) took 1826 ms. Trace:
I20260709 12:17:09.005129 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.018640 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 25.
I20260709 12:17:09.019418 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.003152 12497 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 26. Preceding OpId from leader: term: 42 index: 26. (term mismatch)
I20260709 12:17:08.897433 24329 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.009049 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:09.008289 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.002218 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:09.017674 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.027971 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.029387 12567 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:08.959779 24626 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:09.032692 24626 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:09.033061 24626 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.033399 24626 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:09.034832 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:09.020640 12497 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 25
I20260709 12:17:09.014928 12639 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:09.037019 12497 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 41.26
I20260709 12:17:09.037333 12497 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:09.037411 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4658) took 1900 ms. Trace:
I20260709 12:17:09.037837 12639 rpcz_store.cc:276] 0709 12:17:07.137573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.986058 (+1848485us) service_pool.cc:224] Handling call
0709 12:17:09.014742 (+ 28684us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.037119 (+ 22377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.037352 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.039363 12639 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 23. (term mismatch)
I20260709 12:17:09.039695 12639 pending_rounds.cc:85] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 22
I20260709 12:17:09.039938 12639 pending_rounds.cc:107] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 33.23
I20260709 12:17:09.040174 12639 raft_consensus.cc:2889] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:09.040971 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4661) took 1830 ms. Trace:
I20260709 12:17:09.041790 12639 rpcz_store.cc:276] 0709 12:17:07.210342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.038604 (+1828262us) service_pool.cc:224] Handling call
0709 12:17:09.039005 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.040705 (+  1700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.040914 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.020063 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.014349 12641 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:09.043565 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3818) took 1970 ms. Trace:
I20260709 12:17:09.009089 24732 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:08.997726 24679 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:09.046025 24679 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:09.046800 12639 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:09.048329 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4662) took 1837 ms. Trace:
W20260709 12:17:09.049240 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.049240 12639 rpcz_store.cc:276] 0709 12:17:07.210771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.045607 (+1834836us) service_pool.cc:224] Handling call
0709 12:17:09.046315 (+   708us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.047892 (+  1577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.048218 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.053336 12639 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: [35.22-37.23]
I20260709 12:17:09.053455 24569 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:09.055179 24405 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:09.055634 24405 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.056546 24405 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.17, Last appended by leader: 17, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.057220 12644 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:09.057984 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1755) took 1481 ms. Trace:
I20260709 12:17:09.058485 24719 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:09.046583 24626 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.011735 24564 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:09.029748 24632 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 29.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:09.062399 24632 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:17:09.062805 24632 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:09.063969 24632 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.064726 24569 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:17:09.030628 24745 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:09.031481 12570 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 38 because replica has last-logged OpId of term: 36 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 24.
I20260709 12:17:09.065702 24663 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:17:09.066012 24745 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:09.066367 24663 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.067198 24745 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.068583 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1757) took 1481 ms. Trace:
I20260709 12:17:09.068670 24569 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:17:09.043936 12641 rpcz_store.cc:276] 0709 12:17:07.074976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.984069 (+1909093us) service_pool.cc:224] Handling call
0709 12:17:09.014147 (+ 30078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.043320 (+ 29173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.043513 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.070199 24381 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:09.070652 24381 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.071027 12641 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:09.071907 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1758) took 1468 ms. Trace:
I20260709 12:17:09.071581 24381 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.18, Last appended by leader: 18, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.072373 12641 rpcz_store.cc:276] 0709 12:17:07.603795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.070367 (+1466572us) service_pool.cc:224] Handling call
0709 12:17:09.070791 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.071536 (+   745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.071834 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.073030 24381 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:09.067445 24663 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.074324 24663 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:17:09.057991 24405 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:09.075822 24569 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:17:09.076920 24569 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 38.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:17:09.077387 24568 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:09.065435 24632 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:09.077795 24568 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.078892 24568 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 39.16, Last appended by leader: 16, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.058321 12644 rpcz_store.cc:276] 0709 12:17:07.576426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.056657 (+1480231us) service_pool.cc:224] Handling call
0709 12:17:09.057031 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.057660 (+   629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.057908 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.080011 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5034) took 1585 ms. Trace:
I20260709 12:17:09.080569 12563 rpcz_store.cc:276] 0709 12:17:07.494369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:07.511032 (+ 16663us) service_pool.cc:224] Handling call
0709 12:17:07.537863 (+ 26831us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:07.919846 (+381983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.079600 (+1159754us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:09.079908 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":847795072}
I20260709 12:17:09.081800 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1761) took 1236 ms. Trace:
I20260709 12:17:09.082250 12644 rpcz_store.cc:276] 0709 12:17:07.845436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.080642 (+1235206us) service_pool.cc:224] Handling call
0709 12:17:09.081057 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.081398 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.081427 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:09.081476 (+    49us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:09.081523 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.081588 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:09.081742 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.084144 12644 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:17:09.084753 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4667) took 1021 ms. Trace:
I20260709 12:17:09.085085 12644 rpcz_store.cc:276] 0709 12:17:08.063619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.083461 (+1019842us) service_pool.cc:224] Handling call
0709 12:17:09.083804 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.084475 (+   671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.084699 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.086568 12644 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 19. (term mismatch)
I20260709 12:17:09.086875 12644 pending_rounds.cc:85] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 18
I20260709 12:17:09.087102 12644 pending_rounds.cc:107] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 36.19
I20260709 12:17:09.087359 12644 raft_consensus.cc:2889] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:09.089713 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1753) took 1513 ms. Trace:
I20260709 12:17:09.091260 12639 rpcz_store.cc:276] 0709 12:17:07.575929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.051458 (+1475529us) service_pool.cc:224] Handling call
0709 12:17:09.052448 (+   990us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:09.087917 (+ 35469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.089203 (+  1286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:08.930810 24625 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.016814 12646 rpcz_store.cc:276] 0709 12:17:07.177363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.002785 (+1825422us) service_pool.cc:224] Handling call
0709 12:17:09.003115 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.003413 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.003453 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:09.003528 (+    75us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:09.003595 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.003672 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:09.003821 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.021960 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3817) took 1957 ms. Trace:
I20260709 12:17:09.103951 12640 rpcz_store.cc:276] 0709 12:17:07.064966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.983640 (+1918674us) service_pool.cc:224] Handling call
0709 12:17:09.014359 (+ 30719us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:09.016083 (+  1724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.021866 (+  5783us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.092341 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.011422 24499 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.952579 24474 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.133335 24329 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:08.981056 24383 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.135430 24669 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.135645 24663 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:09.002480 24672 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:17:09.013998 24740 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:09.136924 24740 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:09.138379 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.139726 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.140691 24672 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:09.140849 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.141244 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.141222 24672 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.141752 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:09.142942 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.143571 24672 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.111354 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.048221 24718 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:09.046721 24679 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:09.119952 24312 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.627s	user 0.009s	sys 0.005s
I20260709 12:17:09.062829 24626 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:09.108035 24519 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:17:09.127362 24689 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:09.127806 24735 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:17:09.129545 24625 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:09.130097 24564 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:17:09.135937 24383 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.106601 12639 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:17:09.190258 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4670) took 1102 ms. Trace:
I20260709 12:17:09.190686 12639 rpcz_store.cc:276] 0709 12:17:08.088059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.094163 (+1006104us) service_pool.cc:224] Handling call
0709 12:17:09.094662 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.189928 (+ 95266us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.190187 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.191881 24728 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:09.191957 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:09.112463 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:09.143143 24508 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.641s	user 0.010s	sys 0.002s
I20260709 12:17:09.145171 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:09.145856 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.147725 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.043630 24701 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.140923 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.069552 12645 rpcz_store.cc:276] 0709 12:17:07.587005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.067559 (+1480554us) service_pool.cc:224] Handling call
0709 12:17:09.067927 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.068145 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.068174 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:09.068225 (+    51us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:09.068269 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.068337 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:09.068494 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.142832 12637 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:17:09.150754 12646 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:17:09.196681 12565 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:09.204241 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4671) took 1103 ms. Trace:
I20260709 12:17:09.204872 12646 rpcz_store.cc:276] 0709 12:17:08.101140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.103209 (+1002069us) service_pool.cc:224] Handling call
0709 12:17:09.150364 (+ 47155us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.203863 (+ 53499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.204149 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:09.106166 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 292713us
I20260709 12:17:09.156826 12640 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
W20260709 12:17:09.158576 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.158790 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4668) took 1089 ms. Trace:
I20260709 12:17:09.135468 24474 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.189131 24740 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.168149 24729 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.224305 24740 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.224563 24729 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:09.169557 24679 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.170220 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.225816 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4673) took 1066 ms. Trace:
I20260709 12:17:09.157800 24672 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.025s
I20260709 12:17:09.232774 13260 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:09.156497 24669 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:09.184670 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.185438 24519 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:09.159397 24329 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.033s
I20260709 12:17:09.144400 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.192888 12639 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37. Re-sending same reply.
I20260709 12:17:09.207204 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4672) took 1073 ms. Trace:
I20260709 12:17:09.241328 12564 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:09.241369 12640 rpcz_store.cc:276] 0709 12:17:08.134042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.113608 (+979566us) service_pool.cc:224] Handling call
0709 12:17:09.156385 (+ 42777us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.206876 (+ 50491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.207138 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.207072 12644 rpcz_store.cc:276] 0709 12:17:08.069391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.085949 (+1016558us) service_pool.cc:224] Handling call
0709 12:17:09.086253 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.158477 (+ 72224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.158717 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:09.212527 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.218899 24382 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
W20260709 12:17:09.219499 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:09.244717 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 60185.7us
W20260709 12:17:09.220153 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.174114 24564 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.226238 12637 rpcz_store.cc:276] 0709 12:17:08.159716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.142143 (+982427us) service_pool.cc:224] Handling call
0709 12:17:09.142491 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.203385 (+ 60894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.225744 (+ 22359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:09.253213 12644 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:17:09.227159 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.254824 12644 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: []
W20260709 12:17:09.227415 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.256498 12644 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: [39.16-39.16]
I20260709 12:17:09.229382 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.257393 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:09.258430 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.259075 12572 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 25.
I20260709 12:17:09.228356 24508 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.260969 24329 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:09.230762 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.261847 24508 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:09.261605 24329 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.262697 12569 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:09.231693 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.233680 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:09.184538 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.238504 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:09.264655 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.244993 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:09.239754 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.262233 12567 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:09.234315 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.264747 12645 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:09.271231 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.263808 12565 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:09.276858 12637 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:09.278324 13262 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:17:09.264966 12563 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:09.283658 24725 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:09.284250 24725 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.284724 12609 leader_election.cc:400] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35.
I20260709 12:17:09.284621 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:09.285295 24725 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.287055 24725 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:09.287423 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.287691 23697 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:09.290259 24568 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:09.291239 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:09.291988 13254 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 25.
I20260709 12:17:09.292073 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.293501 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.294734 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.299587 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.294337 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.303717 12638 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:09.306104 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.311518 12643 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:17:09.310621 12644 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:09.310536 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.303653 13254 raft_consensus.cc:2955] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Committing config change with OpId 30.19: config changed from index 7 to 19, VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:09.317251 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:09.317399 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72510.4us
I20260709 12:17:09.323275 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.318276 12638 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 20. (term mismatch)
I20260709 12:17:09.324198 12638 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 19
I20260709 12:17:09.324532 12638 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 33.20
I20260709 12:17:09.324827 12638 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:09.344300 13253 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:17:09.345158 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.244061 24382 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.264302 24564 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:17:09.346719 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.344676 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.356829 12572 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:17:09.350852 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:09.359478 12570 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:17:09.361100 12570 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:17:09.362694 12570 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
W20260709 12:17:09.349344 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 426633us
I20260709 12:17:09.364140 12570 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:17:09.277354 24728 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:09.371999 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:09.382457 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 373463us
I20260709 12:17:09.384019 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.377863 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:09.345315 24519 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:17:09.391774 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.352687 24638 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:09.350132 24745 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:17:09.350322 24701 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:17:09.408764 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45396.1us
I20260709 12:17:09.409499 23540 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:09.406286 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.410089 23540 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:09.357825 24490 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:09.368940 24740 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:09.397079 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66491.5us
I20260709 12:17:09.389782 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.293419 24312 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.077s
I20260709 12:17:09.408345 24499 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:09.439363 24499 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.439836 24499 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:09.442770 24498 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:09.343915 24732 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:09.429997 24329 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.445971 24732 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.418383 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:09.448593 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.450007 23540 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.17, Last appended by leader: 17, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.402204 12639 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:17:09.351030 24474 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:09.452414 24474 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.453255 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3819) took 1070 ms. Trace:
I20260709 12:17:09.405985 24519 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.453730 12639 rpcz_store.cc:276] 0709 12:17:08.383025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.241728 (+858703us) service_pool.cc:224] Handling call
0709 12:17:09.242030 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.452277 (+210247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.452928 (+   651us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:09.453181 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360430336}
I20260709 12:17:09.344776 24751 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:17:09.394347 13259 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:09.344530 24753 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:09.370880 24625 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:09.407902 24508 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:09.412254 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.462833 24508 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.463650 24682 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:09.465955 24701 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:09.344671 24681 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.122s
I20260709 12:17:09.466467 24701 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.466977 24701 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:09.426435 24729 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:09.468446 24729 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:09.469872 24672 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:09.470455 24672 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:09.469739 24729 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.471772 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:09.407713 24718 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:09.427711 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.429105 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:09.430073 24382 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.390154 24689 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:17:09.354727 24669 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:17:09.375121 24728 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.412068 24638 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.498217 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.404438 24323 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:09.454301 24519 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:17:09.412012 24745 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:09.473563 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61444.3us
I20260709 12:17:09.477761 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.479324 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:09.483270 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.011s	sys 0.016s
W20260709 12:17:09.512065 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.011s	sys 0.017s
I20260709 12:17:09.484424 12492 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:09.487428 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:09.489111 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:09.513767 12497 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:09.488175 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.494329 12490 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
W20260709 12:17:09.494737 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67102.3us
I20260709 12:17:09.492923 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:09.462368 24753 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.515694 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.501410 12494 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:09.470881 24672 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:09.525434 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:09.502815 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88142.2us
I20260709 12:17:09.510231 24382 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:09.522701 24508 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:09.491070 24329 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:09.527868 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:09.482730 24653 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:09.504374 24498 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:09.482969 24718 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.570369 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:09.505769 24728 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:09.565651 24623 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:17:09.581925 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66318.2us
I20260709 12:17:09.500867 24681 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:09.510232 24746 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:17:09.585582 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 484743us
W20260709 12:17:09.586272 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58497.1us
W20260709 12:17:09.587750 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108528us
I20260709 12:17:09.608983 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:09.585958 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:09.611078 24484 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:09.612035 24740 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:09.612491 24740 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:09.612896 24740 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:09.583818 13261 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40. Re-sending same reply.
I20260709 12:17:09.618331 24746 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.618829 24746 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:17:09.621307 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.523936 24753 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:09.622236 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:09.622767 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.623756 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.625020 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.538412 13259 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.28-40.28]
I20260709 12:17:09.628872 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:09.586423 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:09.631486 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:09.504223 24638 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:09.582106 24653 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:09.598070 12394 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 27 to 34. New cstate: current_term: 34 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:09.597272 12497 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:09.531061 24745 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:17:09.588363 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:09.635186 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46951.7us
I20260709 12:17:09.540534 24717 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:09.629988 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.502954 24323 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.540119 24475 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:17:09.630842 12488 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
W20260709 12:17:09.665567 12641 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.022s	sys 0.020s
I20260709 12:17:09.638407 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:09.666172 12641 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.023s	sys 0.020s
W20260709 12:17:09.588186 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60877.8us
I20260709 12:17:09.643133 24498 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:17:09.669201 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.017s	sys 0.024s
I20260709 12:17:09.643388 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.670791 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:09.644140 24623 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:17:09.679589 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.012s	sys 0.031s
I20260709 12:17:09.644910 24754 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:09.603715 24615 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:09.684185 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.013s	sys 0.031s
I20260709 12:17:09.649828 24669 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:09.641986 24508 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.011s
I20260709 12:17:09.586715 24689 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:09.685729 24689 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:09.686857 24689 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.688728 24689 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:09.581538 24625 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:09.689113 24740 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.689968 24625 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.692462 24625 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.653723 24732 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.628222 24718 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.640829 24474 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.641083 24653 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:09.661602 24519 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.565574 24697 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:09.695245 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.695184 24697 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.667555 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.697170 24740 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.666445 12567 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:17:09.664129 24499 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.641340 24630 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:09.697600 24697 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:09.698240 12570 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:17:09.700450 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:09.700748 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.709568 12567 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:17:09.671991 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.018s	sys 0.024s
I20260709 12:17:09.523748 24683 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.144s
I20260709 12:17:09.701038 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:09.715437 12394 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:17:09.677884 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49119.5us
I20260709 12:17:09.639577 24481 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:09.644245 24405 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:17:09.523300 24312 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.117s
I20260709 12:17:09.677520 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:09.717183 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:09.718144 12394 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 32 to 35. New cstate: current_term: 35 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:09.706627 24653 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.667882 24323 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:09.640235 24717 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:09.733553 13218 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.537s (SENT)
I20260709 12:17:09.712227 24383 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:09.711968 24003 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:09.669412 24435 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.116s
W20260709 12:17:09.787344 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86392.4us
W20260709 12:17:09.784500 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117161us
I20260709 12:17:09.807452 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.726498 24719 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:09.792086 12645 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
W20260709 12:17:09.797003 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79921.1us
I20260709 12:17:09.819010 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:09.780292 24669 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:09.719389 24484 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.081s
I20260709 12:17:09.791653 24717 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:09.714960 24749 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.096s
I20260709 12:17:09.780586 24630 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:09.722748 24684 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.098s
I20260709 12:17:09.797811 24383 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.851189 24383 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:09.851600 24630 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 38.16, Last appended by leader: 15, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.832581 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3825) took 1303 ms. Trace:
I20260709 12:17:09.853948 12645 rpcz_store.cc:276] 0709 12:17:08.529135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.272683 (+743548us) service_pool.cc:224] Handling call
0709 12:17:09.273067 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.818589 (+545522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.832222 (+ 13633us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:09.832486 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1193069440}
I20260709 12:17:09.792914 24740 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:09.789063 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:09.721393 24751 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.083s
I20260709 12:17:09.819130 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:09.842626 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.843148 12567 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:09.863507 12571 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:09.856222 24312 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:09.811261 24003 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:09.870157 24312 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.871196 24003 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 38.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.871466 24312 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 38.16, Last appended by leader: 16, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.873351 24312 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:09.865039 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:09.876900 24740 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:17:09.877489 24740 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:09.878790 24740 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:09.879803 24760 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:17:09.879913 24383 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.882104 24625 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:09.882313 24751 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:17:09.883324 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64328.1us
I20260709 12:17:09.883245 24475 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:09.883647 24751 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:17:09.885908 24751 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:09.886622 24484 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:09.887288 24484 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:09.888197 24484 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.669746 24498 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:09.797151 24474 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:09.702916 24499 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:09.782109 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:09.889719 24382 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:09.897325 24490 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:09.932849 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:09.821717 12394 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:17:09.902740 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:09.903605 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 170093us
I20260709 12:17:09.904436 24672 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:09.905140 24624 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:17:09.906850 24519 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:09.923339 24498 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:09.910703 24564 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.000s
I20260709 12:17:09.916004 24701 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:09.916709 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:09.844434 24271 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:17:09.831332 24679 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:09.832348 24677 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:09.794942 24759 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.195s
W20260709 12:17:09.945127 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.011s	sys 0.010s
W20260709 12:17:10.026549 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.011s	sys 0.010s
I20260709 12:17:09.962963 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:09.963156 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:09.965019 12638 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:17:09.998292 12394 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 22 to 29. New cstate: current_term: 29 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.005273 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:09.933437 24382 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:09.933566 24490 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:09.907512 24653 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:10.013172 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:09.944847 24317 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.011s	sys 0.010s
W20260709 12:17:10.018573 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.008s	sys 0.016s
W20260709 12:17:10.046875 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.009s	sys 0.016s
W20260709 12:17:10.018711 13254 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.017s	sys 0.009s
W20260709 12:17:10.048123 13254 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.018s	sys 0.009s
W20260709 12:17:10.019122 13255 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.005s	sys 0.019s
W20260709 12:17:10.048781 13255 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.006s	sys 0.020s
I20260709 12:17:09.923290 24474 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:09.962971 24745 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.099097 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:10.101518 23540 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.008s	sys 0.022s
I20260709 12:17:09.897687 24383 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:10.034868 12394 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 28 to 35. New cstate: current_term: 35 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.038959 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.107044 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.079536 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:10.079169 24317 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.011s	sys 0.010s
I20260709 12:17:10.111281 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:10.111506 24382 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.101980 23540 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.008s	sys 0.022s
I20260709 12:17:10.025347 24484 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:10.039227 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.112112 12492 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41.
I20260709 12:17:10.113027 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:10.116127 12497 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:10.030078 24271 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:10.015697 24499 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.069s
W20260709 12:17:10.120711 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:10.047608 24719 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.034866 24677 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:10.080732 24435 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:10.117334 24381 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.018s	sys 0.013s
I20260709 12:17:10.098959 24474 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:10.112046 12394 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 34, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 34 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.137143 13261 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:10.022179 24564 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.140045 23697 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.020s	sys 0.027s
W20260709 12:17:10.162388 13260 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.014s	sys 0.014s
W20260709 12:17:10.140403 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:10.136535 24382 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:10.160462 12394 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 27 to 28, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:17:10.163261 13260 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.014s	sys 0.015s
I20260709 12:17:10.170881 12394 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 33 to 40. New cstate: current_term: 40 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:17:10.162487 23697 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.020s	sys 0.027s
I20260709 12:17:10.176079 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.047302 24623 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.088s
I20260709 12:17:10.177093 12497 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:10.022490 12644 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:60230
I20260709 12:17:10.185391 12644 raft_consensus.cc:606] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:10.185942 12644 raft_consensus.cc:614] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:10.100780 24625 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.059s
W20260709 12:17:10.158020 24746 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.520s	user 0.012s	sys 0.000s
W20260709 12:17:10.214516 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.215095 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.215492 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:10.215369 12571 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
W20260709 12:17:10.160113 24381 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.018s	sys 0.013s
I20260709 12:17:10.196487 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.198941 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.203195 24323 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:10.207295 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 229) took 192225 ms (client timeout 15000 ms). Trace:
I20260709 12:17:10.136734 24490 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.221988 12395 rpcz_store.cc:269] 0709 12:13:57.910183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:13:57.922635 (+ 12452us) service_pool.cc:224] Handling call
0709 12:13:59.266864 (+1344229us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:10.123798 (+190856934us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51265,"reported_tablets":51,"spinlock_wait_cycles":729745152}
I20260709 12:17:10.223291 24490 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:10.209952 24745 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:10.211455 24677 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.179354 12394 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 30 to 35. New cstate: current_term: 35 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.224692 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5056) took 1028 ms. Trace:
W20260709 12:17:10.231933 12563 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.006s	sys 0.019s
W20260709 12:17:10.232311 12563 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.006s	sys 0.019s
W20260709 12:17:10.232755 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.012s	sys 0.009s
W20260709 12:17:10.233285 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.012s	sys 0.009s
I20260709 12:17:10.229738 12564 rpcz_store.cc:276] 0709 12:17:09.196464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.242266 (+ 45802us) service_pool.cc:224] Handling call
0709 12:17:09.242647 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.246004 (+  3357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.246037 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.264056 (+ 18019us) log.cc:844] Serialized 26 byte log entry
0709 12:17:10.027179 (+763123us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:10.027261 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.027370 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:10.224390 (+197020us) raft_consensus.cc:1685] finished
0709 12:17:10.224414 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.224607 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2882,"raft.queue_time_us":29940,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":15938,"thread_start_us":736473,"threads_started":1,"wal-append.queue_time_us":931582}
W20260709 12:17:10.233587 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 591962us
I20260709 12:17:10.099453 24669 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.235716 12394 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 30 to 36, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:17:10.235010 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.005s	sys 0.016s
W20260709 12:17:10.241498 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.005s	sys 0.016s
W20260709 12:17:10.242009 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 578680us
I20260709 12:17:10.240803 12394 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 32 to 39. New cstate: current_term: 39 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.157405 24719 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.249336 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:10.254899 13253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.033s	sys 0.030s
W20260709 12:17:10.255482 13253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.034s	sys 0.030s
I20260709 12:17:10.265103 12564 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
W20260709 12:17:10.231750 24674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.011s	sys 0.010s
I20260709 12:17:10.282588 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:10.178887 24435 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.283087 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:10.249454 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.023s	sys 0.013s
I20260709 12:17:10.268469 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.286504 12492 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:10.289320 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.023s	sys 0.013s
W20260709 12:17:10.287952 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.020s	sys 0.022s
W20260709 12:17:10.290139 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.020s	sys 0.022s
W20260709 12:17:10.290990 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.291505 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.292214 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:10.294791 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:10.296032 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:10.296872 12564 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:17:10.240420 24508 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.008s	sys 0.012s
W20260709 12:17:10.298952 12566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.020s	sys 0.024s
W20260709 12:17:10.299326 12566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.020s	sys 0.024s
W20260709 12:17:10.269348 24753 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.647s	user 0.012s	sys 0.000s
I20260709 12:17:10.301251 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:10.315371 12490 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:10.318091 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4862) took 1204 ms. Trace:
I20260709 12:17:10.219066 24746 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.290035 24674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.011s	sys 0.011s
I20260709 12:17:10.310652 12567 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:17:10.326476 12571 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:10.330572 12493 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:17:10.343910 12567 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:17:10.339056 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:10.269117 24312 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.302968 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:10.250316 12394 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 7 to 14, term changed from 21 to 23, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to <none>, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 23 committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:17:10.293114 24729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.004s	sys 0.019s
W20260709 12:17:10.334255 13262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.006s	sys 0.020s
W20260709 12:17:10.357204 13262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.006s	sys 0.020s
I20260709 12:17:10.360526 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:10.311192 24508 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.009s	sys 0.013s
I20260709 12:17:10.364323 12564 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
W20260709 12:17:10.365592 12641 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.000s	sys 0.031s
W20260709 12:17:10.365985 12641 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.000s	sys 0.031s
I20260709 12:17:10.364399 12490 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
W20260709 12:17:10.348125 13257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.011s	sys 0.014s
W20260709 12:17:10.348634 12646 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.008s	sys 0.011s
W20260709 12:17:10.350426 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.012s	sys 0.015s
I20260709 12:17:10.353312 12572 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:10.370792 12394 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 30 to 39, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 39 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.416342 12394 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 28 to 32. New cstate: current_term: 32 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.417900 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:10.415191 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.014s	sys 0.017s
W20260709 12:17:10.376389 13257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.011s	sys 0.014s
I20260709 12:17:10.422026 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:10.413865 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:10.429721 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 774679us
I20260709 12:17:10.429745 12394 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 24 to 38, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 38 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:10.438752 12491 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
W20260709 12:17:10.468364 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:10.322746 24756 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.076s
W20260709 12:17:10.368992 24766 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
W20260709 12:17:10.374651 24615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.015s	sys 0.017s
W20260709 12:17:10.483198 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.007s	sys 0.019s
W20260709 12:17:10.568418 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.007s	sys 0.020s
W20260709 12:17:10.568881 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 706275us
I20260709 12:17:10.488065 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
W20260709 12:17:10.362795 24729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.004s	sys 0.020s
I20260709 12:17:10.350601 24672 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:10.365437 24718 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:10.494344 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:10.592232 12496 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:17:10.355978 24481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.005s	sys 0.018s
W20260709 12:17:10.424126 12646 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.008s	sys 0.018s
I20260709 12:17:10.255412 24271 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.557655 12488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.025s	sys 0.024s
I20260709 12:17:10.426319 24745 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:10.367453 24381 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:10.420245 24769 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.339s	user 0.001s	sys 0.000s
W20260709 12:17:10.418548 24740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.008s	sys 0.024s
I20260709 12:17:10.329576 24753 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.567668 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1394300210001802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30800000000000005s for a response
W20260709 12:17:10.609406 12488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.027s	sys 0.024s
W20260709 12:17:10.611177 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.028s	sys 0.024s
I20260709 12:17:10.376271 24679 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:10.567338 24630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.028s	sys 0.038s
W20260709 12:17:10.616005 24761 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.457s	user 0.001s	sys 0.000s
W20260709 12:17:10.615530 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.654193 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.014s	sys 0.024s
I20260709 12:17:10.490303 24632 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:10.624886 13259 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:10.600276 24508 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:10.598596 24764 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.480s	user 0.000s	sys 0.002s
W20260709 12:17:10.611429 24763 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.503s	user 0.000s	sys 0.001s
I20260709 12:17:10.639209 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:10.610244 24481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.005s	sys 0.019s
W20260709 12:17:10.615844 24681 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.007s	user 0.013s	sys 0.000s
I20260709 12:17:10.647373 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:10.427472 24624 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.199s
W20260709 12:17:10.521699 24568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.008s	sys 0.037s
W20260709 12:17:10.654438 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.589901 24615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.017s	sys 0.017s
W20260709 12:17:10.708864 13254 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.066s	sys 0.064s
W20260709 12:17:10.709961 13254 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.066s	sys 0.065s
W20260709 12:17:10.658221 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.014s	sys 0.024s
W20260709 12:17:10.642601 24740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.008s	sys 0.025s
I20260709 12:17:10.366639 24312 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.643079 24381 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:10.660964 24317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.042s	sys 0.066s
W20260709 12:17:10.644117 24630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.029s	sys 0.039s
W20260709 12:17:10.697273 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.020s	sys 0.024s
W20260709 12:17:10.698354 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59334.3us
I20260709 12:17:10.702044 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
W20260709 12:17:10.705538 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:10.629130 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.029s	sys 0.025s
I20260709 12:17:10.748090 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:10.749370 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.020s	sys 0.024s
W20260709 12:17:10.749959 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 726286us
I20260709 12:17:10.643275 24745 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:10.713322 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1514) took 1410 ms. Trace:
W20260709 12:17:10.696094 24771 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:17:10.721306 24323 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:10.786814 24740 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.767030 13254 rpcz_store.cc:276] 0709 12:17:09.302755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.302915 (+   160us) service_pool.cc:224] Handling call
0709 12:17:09.303234 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.303500 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:09.387529 (+ 84029us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.110330 (+722801us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.712754 (+602424us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:10.712848 (+    94us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:10.712931 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.713039 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.713241 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33196,"prepare.queue_time_us":317391,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":9781,"raft.queue_time_us":326030,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":40607,"spinlock_wait_cycles":40324224,"thread_start_us":1262778,"threads_started":2,"wal-append.queue_time_us":984577}
W20260709 12:17:10.757855 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:10.760571 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:10.691866 24481 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:10.710541 24681 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:10.771539 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:10.773823 24775 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:17:10.776633 24683 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:10.809850 13260 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.006s	sys 0.046s
I20260709 12:17:10.777961 24630 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:10.780627 24435 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:10.783489 24746 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:10.785135 24756 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:10.790390 12564 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:10.793802 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1883) took 1110 ms. Trace:
I20260709 12:17:10.821136 12488 rpcz_store.cc:276] 0709 12:17:09.683539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.683790 (+   251us) service_pool.cc:224] Handling call
0709 12:17:09.684353 (+   563us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.726172 (+ 41819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.726223 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.782696 (+ 56473us) log.cc:844] Serialized 26 byte log entry
0709 12:17:10.628858 (+846162us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:10.628921 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.629019 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:10.793452 (+164433us) raft_consensus.cc:1685] finished
0709 12:17:10.793483 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.793719 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41176,"raft.queue_time_us":103672,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":7079,"thread_start_us":816663,"threads_started":1,"wal-append.queue_time_us":992125}
W20260709 12:17:10.718863 24568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.013s	sys 0.038s
I20260709 12:17:10.778065 24383 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:10.810513 13260 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.006s	sys 0.046s
W20260709 12:17:10.755733 24317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.042s	sys 0.067s
W20260709 12:17:10.814710 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.012s	sys 0.011s
I20260709 12:17:10.754814 24754 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.728775 24781 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:10.655902 24679 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.830943 12489 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:10.812964 24759 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:10.840216 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.013s	sys 0.012s
W20260709 12:17:10.813035 24761 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:17:10.813225 24638 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.148s	user 0.013s	sys 0.000s
I20260709 12:17:10.857105 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5045) took 2608 ms. Trace:
W20260709 12:17:10.846807 24728 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.221s	user 0.011s	sys 0.003s
W20260709 12:17:10.856946 24766 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:17:10.840812 24719 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:10.684337 24508 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:10.858896 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.017s	sys 0.013s
W20260709 12:17:10.895399 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.017s	sys 0.013s
I20260709 12:17:10.831611 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1518) took 1354 ms. Trace:
I20260709 12:17:10.770469 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:10.899744 23540 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.900578 23540 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.904285 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:10.820920 24683 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.896466 13260 rpcz_store.cc:276] 0709 12:17:09.477048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.477762 (+   714us) service_pool.cc:224] Handling call
0709 12:17:09.483000 (+  5238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.483246 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:17:10.178579 (+695333us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.831185 (+652606us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:10.831244 (+    59us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:10.831301 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.831369 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.831550 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":394624,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":25709,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":10025,"thread_start_us":1321844,"threads_started":2}
W20260709 12:17:10.861797 13255 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.013s	sys 0.019s
I20260709 12:17:10.805534 24740 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.914038 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:10.867386 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1760) took 3028 ms. Trace:
W20260709 12:17:10.845969 24003 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s	user 0.012s	sys 0.041s
I20260709 12:17:10.947790 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1746) took 1428 ms. Trace:
I20260709 12:17:10.896251 24759 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:10.846808 23697 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.011s	sys 0.039s
I20260709 12:17:10.985773 23540 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:10.986074 23697 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.012s	sys 0.040s
W20260709 12:17:10.909268 13255 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.014s	sys 0.020s
I20260709 12:17:10.819674 24499 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.912910 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:10.988472 12640 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:10.915213 12642 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:10.916314 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:10.893661 24689 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.204s	user 0.014s	sys 0.003s
I20260709 12:17:10.916826 12639 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: [38.16-38.16]
I20260709 12:17:10.918452 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:10.919238 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:10.994432 12637 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:10.995566 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:10.846095 24383 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.947603 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:11.004503 12638 raft_consensus.cc:2376] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36. Re-sending same reply.
I20260709 12:17:10.950148 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:10.860212 24754 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.978315 13257 rpcz_store.cc:276] 0709 12:17:09.519116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.519319 (+   203us) service_pool.cc:224] Handling call
0709 12:17:09.535116 (+ 15797us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.535659 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.535707 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.565283 (+ 29576us) log.cc:844] Serialized 26 byte log entry
0709 12:17:10.433161 (+867878us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:10.433230 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.433349 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:10.947432 (+514083us) raft_consensus.cc:1685] finished
0709 12:17:10.947468 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.947692 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5249,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":6767,"thread_start_us":765262,"threads_started":1,"wal-append.queue_time_us":1241135}
I20260709 12:17:10.873453 12568 rpcz_store.cc:276] 0709 12:17:08.248370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.248509 (+   139us) service_pool.cc:224] Handling call
0709 12:17:08.248875 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:08.293873 (+ 44998us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:08.293908 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:08.307541 (+ 13633us) log.cc:844] Serialized 26 byte log entry
0709 12:17:09.143842 (+836301us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b18003a32a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:17:09.712243 (+568401us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:09.712282 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:09.712351 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:10.840229 (+1127878us) raft_consensus.cc:1685] finished
0709 12:17:10.840253 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.857016 (+ 16763us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34971,"raft.queue_time_us":45658,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":12194,"spinlock_wait_cycles":103502080,"thread_start_us":568177,"threads_started":1,"wal-append.queue_time_us":1686000}
I20260709 12:17:11.011324 12645 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
I20260709 12:17:10.787606 24672 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:10.996384 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 29.
I20260709 12:17:11.012765 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:11.014086 12640 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:11.011158 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:10.976882 24003 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.085s	user 0.013s	sys 0.042s
W20260709 12:17:10.982602 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.009s	sys 0.001s
W20260709 12:17:11.021672 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.010s	sys 0.001s
I20260709 12:17:10.922430 24481 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.947014 24508 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.016141 12642 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:11.050151 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:10.893716 24638 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:10.893677 24728 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.018903 24672 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:10.981938 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211515us
I20260709 12:17:10.863960 24781 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:10.985117 24490 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.761s	user 0.013s	sys 0.000s
I20260709 12:17:10.911164 24568 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:11.028712 24802 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:17:10.993100 24624 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:11.030155 24740 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.030211 24312 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.031759 24683 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.016168 24784 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
W20260709 12:17:11.032765 24798 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:17:11.035020 24383 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:11.035741 24773 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.355s	user 0.001s	sys 0.000s
W20260709 12:17:11.010666 24674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.000s	sys 0.040s
I20260709 12:17:11.019819 24801 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:11.037065 24632 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:11.037768 24679 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.038197 24753 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.038437 24799 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.015671 24499 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:11.040498 24474 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.860s	user 0.010s	sys 0.000s
I20260709 12:17:11.040618 24754 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:11.158510 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:11.159250 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:11.078058 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 430780us
W20260709 12:17:11.078652 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.006s	sys 0.005s
W20260709 12:17:11.161808 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.006s	sys 0.005s
I20260709 12:17:11.079293 23540 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:11.163087 23540 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.080083 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4674) took 2880 ms. Trace:
I20260709 12:17:11.012429 24759 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:10.974502 12641 rpcz_store.cc:276] 0709 12:17:07.838679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.073236 (+1234557us) service_pool.cc:224] Handling call
0709 12:17:09.083522 (+ 10286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.084353 (+   831us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.084416 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.086872 (+  2456us) log.cc:844] Serialized 26 byte log entry
0709 12:17:09.666708 (+579836us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:10.380871 (+714163us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.867019 (+486148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.867103 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:10.867148 (+    45us) raft_consensus.cc:1685] finished
0709 12:17:10.867162 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.867317 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1060742,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":22395,"raft.queue_time_us":405479,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":179,"thread_start_us":1764038,"threads_started":3,"wal-append.queue_time_us":934351}
W20260709 12:17:11.014840 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 445205us
I20260709 12:17:11.173547 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:11.174574 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5062) took 1935 ms. Trace:
I20260709 12:17:11.175905 24728 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.175034 12569 rpcz_store.cc:276] 0709 12:17:09.239206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.356074 (+116868us) service_pool.cc:224] Handling call
0709 12:17:09.356431 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.356815 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.356845 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.357350 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:17:10.047615 (+690265us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:10.047681 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.047787 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:11.174325 (+1126538us) raft_consensus.cc:1685] finished
0709 12:17:11.174353 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.174495 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1262,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":20431,"spinlock_wait_cycles":85120,"thread_start_us":625512,"threads_started":1,"wal-append.queue_time_us":1085365}
I20260709 12:17:11.139824 24684 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:17:11.076828 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:11.074966 24498 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.045s	user 0.012s	sys 0.001s
I20260709 12:17:11.133530 24801 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.178011 24498 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:11.178364 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101650us
I20260709 12:17:11.180059 24801 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.155593 24799 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.180729 24801 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:11.149075 24679 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:11.132105 24632 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:11.182308 24799 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.182127 24679 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.161595 24474 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.164397 12646 rpcz_store.cc:276] 0709 12:17:08.199601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.206447 (+1006846us) service_pool.cc:224] Handling call
0709 12:17:09.241157 (+ 34710us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.241609 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.241670 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.251609 (+  9939us) log.cc:844] Serialized 26 byte log entry
0709 12:17:09.722030 (+470421us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:10.604584 (+882554us) log.cc:844] Serialized 14 byte log entry
0709 12:17:11.079108 (+474524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:11.079714 (+   606us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:11.079794 (+    80us) raft_consensus.cc:1685] finished
0709 12:17:11.079821 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.080013 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1050654,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":182466,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":19639,"thread_start_us":1730434,"threads_started":3,"wal-append.queue_time_us":874509}
I20260709 12:17:11.184273 24679 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.184850 24474 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.185019 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.185417 24719 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:17:11.082125 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:11.186455 12638 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:11.068555 24689 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.097677 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:11.187148 24753 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:11.187659 24783 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:17:11.187284 24689 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.188396 24753 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.191318 24753 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:11.192067 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.018s	sys 0.035s
W20260709 12:17:11.192543 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.018s	sys 0.035s
I20260709 12:17:11.192723 24383 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:10.908185 24745 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.104700 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.193863 24756 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:11.193279 24383 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.102718 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.197340 12564 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:17:11.197875 24804 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:17:11.198428 24383 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.199890 24701 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.121730 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:17:11.200977 24484 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.078s	user 0.014s	sys 0.002s
I20260709 12:17:11.100085 24740 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.109712 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:11.161280 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.202064 24740 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.203311 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.203433 24653 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:11.203995 24474 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.204824 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.204720 24474 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.207397 12610 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41.
W20260709 12:17:11.207665 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112805us
I20260709 12:17:11.209009 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:11.144665 24329 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.554s	user 0.015s	sys 0.001s
W20260709 12:17:11.178407 24788 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.456s	user 0.003s	sys 0.002s
I20260709 12:17:11.098037 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.084477 24003 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:11.194135 12572 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.092417 24381 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.100044 24312 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.174991 24638 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.204042 24653 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:11.209189 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:11.105744 24568 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:11.177166 24382 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.994s	user 0.011s	sys 0.000s
W20260709 12:17:11.080380 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.258549 12572 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:11.203737 24799 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:11.222916 24474 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.224411 24809 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:11.260883 24329 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:17:11.262228 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88822us
I20260709 12:17:11.262356 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:11.263260 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 498256us
W20260709 12:17:11.228019 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70144.8us
I20260709 12:17:11.210980 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:11.264005 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54916.9us
I20260709 12:17:11.302909 12493 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:11.262219 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5073) took 1901 ms. Trace:
W20260709 12:17:11.302560 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:11.211817 24717 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.358s	user 0.012s	sys 0.003s
I20260709 12:17:11.324571 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:11.279668 24697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.026s	sys 0.036s
W20260709 12:17:11.303491 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94555us
I20260709 12:17:11.223671 24625 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:11.319075 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.266412 12492 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:11.226311 24825 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:11.327280 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.000s	sys 0.011s
I20260709 12:17:11.226779 24817 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:11.318637 12563 rpcz_store.cc:276] 0709 12:17:09.360607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.432826 (+ 72219us) service_pool.cc:224] Handling call
0709 12:17:09.433161 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.433406 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:10.253359 (+819953us) log.cc:844] Serialized 14 byte log entry
0709 12:17:11.261772 (+1008413us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:11.261843 (+    71us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:11.261908 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:11.261995 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.262147 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39204,"prepare.queue_time_us":132530,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":18735,"raft.queue_time_us":330676,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":139,"spinlock_wait_cycles":19971968,"thread_start_us":1739752,"threads_started":3,"wal-append.queue_time_us":770178}
W20260709 12:17:11.336752 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 176909us
I20260709 12:17:11.292199 24381 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:11.214267 24672 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:11.332938 24697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.028s	sys 0.036s
W20260709 12:17:11.319614 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.333323 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:11.217223 24499 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.291776 24317 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:17:11.338792 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.000s	sys 0.013s
I20260709 12:17:11.258024 24003 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:11.180059 24498 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.332149 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.191987 24781 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:11.347050 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.349192 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:11.334582 12493 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:17:11.312062 24568 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.312695 24786 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:11.327334 24615 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:11.357775 12494 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 34 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:17:11.268193 24312 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.324056 24779 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:11.368559 13257 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:11.402665 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89785.3us
W20260709 12:17:11.343326 24749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.052s	sys 0.091s
I20260709 12:17:11.343194 24819 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:17:11.371639 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.372027 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.383189 24329 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:11.384744 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52676.7us
I20260709 12:17:11.351842 24817 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:11.328593 24717 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.403111 12572 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:11.321688 24759 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.354099 24381 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.386445 24615 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:11.347290 24653 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.28, Last appended by leader: 28, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.346647 24825 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.406960 12571 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:17:11.570086 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 213669us
I20260709 12:17:11.490547 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:11.660604 12496 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:11.663336 12572 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:11.658356 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5078) took 1349 ms. Trace:
I20260709 12:17:11.479027 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:11.709153 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:11.710381 12488 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:11.711939 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:11.404740 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89618.1us
I20260709 12:17:11.442951 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.453953 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:11.450161 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45858.2us
I20260709 12:17:11.550303 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:11.749169 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.755538 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:11.493188 24816 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:17:11.717994 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:11.668226 12565 rpcz_store.cc:276] 0709 12:17:10.308858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:10.313728 (+  4870us) service_pool.cc:224] Handling call
0709 12:17:10.314149 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:10.314619 (+   470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:10.314658 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:10.326732 (+ 12074us) log.cc:844] Serialized 26 byte log entry
0709 12:17:11.162495 (+835763us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:11.162544 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:11.162640 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:11.658033 (+495393us) raft_consensus.cc:1685] finished
0709 12:17:11.658060 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.658282 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2620,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":12739,"thread_start_us":796276,"threads_started":1,"wal-append.queue_time_us":1104159}
W20260709 12:17:11.735605 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 185404us
W20260709 12:17:11.754319 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95071.3us
I20260709 12:17:11.789814 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.648903 24701 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.649242 24745 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.806039 12488 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:17:11.790634 12571 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:11.479809 24797 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.766501 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:11.817058 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:11.450196 24831 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:11.439440 24782 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:17:11.751464 24682 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.441100 24719 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:11.436096 24317 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:11.793900 24653 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:11.793720 24435 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:11.794435 24754 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.795163 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.801308 24381 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.566250 24821 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.351s
I20260709 12:17:11.802986 24717 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.584749 24564 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.767222 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:11.781945 12492 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:17:10.318742 12566 rpcz_store.cc:276] 0709 12:17:09.113914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.114115 (+   201us) service_pool.cc:224] Handling call
0709 12:17:09.135905 (+ 21790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.136137 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:09.136478 (+   341us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.317170 (+1180692us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.317534 (+   364us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:10.317609 (+    75us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:10.317728 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.317829 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:10.318003 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1845,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":101,"raft.queue_time_us":131725,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9240,"spinlock_wait_cycles":15628032,"thread_start_us":1162765,"threads_started":2,"wal-append.queue_time_us":910769}
I20260709 12:17:11.432628 24812 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:11.564291 24785 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.771566 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:11.413056 24749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.052s	sys 0.092s
I20260709 12:17:11.450641 24638 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.314s
W20260709 12:17:11.490423 24778 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.152s	user 0.001s	sys 0.001s
W20260709 12:17:11.438560 24806 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:17:11.878964 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5080) took 1569 ms. Trace:
I20260709 12:17:11.813521 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.770248 12495 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:11.480078 24481 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:11.564675 24718 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.954s	user 0.009s	sys 0.005s
I20260709 12:17:11.441004 24003 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.492985 24674 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.827744 13257 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: [33.24-33.24]
I20260709 12:17:11.866214 24812 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:11.828675 24615 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:11.829694 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.492353 24817 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.830359 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:11.937304 13260 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
W20260709 12:17:11.833606 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5021) took 1581 ms (client timeout 1781 ms). Trace:
I20260709 12:17:11.835335 12571 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:11.430076 24781 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:11.937940 12497 rpcz_store.cc:269] 0709 12:17:10.226933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:10.227323 (+   390us) service_pool.cc:224] Handling call
0709 12:17:11.807792 (+1580469us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:11.807937 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96424704}
I20260709 12:17:11.406002 24683 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.384s
I20260709 12:17:11.492933 24689 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.563588 24789 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:11.520892 24568 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:11.841136 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74737.9us
I20260709 12:17:11.491977 24312 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.845088 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.840248 24682 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.947006 24329 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.957896 24329 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.959003 24624 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:11.959554 24624 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:11.959904 24624 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:11.961953 24689 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.954460 24789 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.962632 24689 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.963615 24729 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:11.964190 24729 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:11.964737 24789 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:11.435467 24697 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:11.967133 24329 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.944238 24781 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.861339 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.865577 12496 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:11.843547 24632 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.661s	user 0.007s	sys 0.004s
I20260709 12:17:11.970785 24781 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.844463 24838 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.027s
W20260709 12:17:11.876307 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 158448us
I20260709 12:17:11.970932 24632 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:11.864180 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97047.4us
I20260709 12:17:11.971906 24838 consensus_queue.cc:1243] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 36.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:11.972025 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1881) took 2431 ms. Trace:
I20260709 12:17:11.762130 24759 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:11.949730 24819 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:11.881412 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68029.3us
I20260709 12:17:11.972867 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:11.789937 24828 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.100s
I20260709 12:17:11.879513 12567 rpcz_store.cc:276] 0709 12:17:10.309531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:10.344775 (+ 35244us) service_pool.cc:224] Handling call
0709 12:17:10.372228 (+ 27453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:10.372673 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:10.372715 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:10.373335 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:17:11.017504 (+644169us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:11.017553 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:11.017655 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:11.878678 (+861023us) raft_consensus.cc:1685] finished
0709 12:17:11.878703 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.878894 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":46599,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":11727,"thread_start_us":431951,"threads_started":1,"wal-append.queue_time_us":869103}
I20260709 12:17:11.830135 24797 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.873380 24785 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.915213 24674 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:11.919021 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.842295 24701 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.906545 24317 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.905764 24790 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:11.887408 24564 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:11.836946 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:11.939219 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1519) took 2440 ms. Trace:
I20260709 12:17:11.982613 13262 rpcz_store.cc:276] 0709 12:17:09.477549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.477867 (+   318us) service_pool.cc:224] Handling call
0709 12:17:09.493556 (+ 15689us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.522174 (+ 28618us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.522217 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.522801 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:17:10.357700 (+834899us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:10.357743 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.357811 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:11.917525 (+1559714us) raft_consensus.cc:1685] finished
0709 12:17:11.917551 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.917772 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29750,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":24660,"thread_start_us":834424,"threads_started":1,"wal-append.queue_time_us":1624316}
W20260709 12:17:11.940558 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.940724 13260 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:11.846215 24719 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.989269 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.942817 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.992125 12571 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:11.993454 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.994310 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:17:11.839131 24745 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.945026 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1519) took 2470 ms. Trace:
I20260709 12:17:11.868788 24754 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:11.891278 24826 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:11.842190 24831 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:11.964901 24689 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:11.854585 12565 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 40 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:17:11.959084 24682 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:11.846503 24813 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:11.973031 24819 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.028128 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.972473 12489 rpcz_store.cc:276] 0709 12:17:09.541065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.541238 (+   173us) service_pool.cc:224] Handling call
0709 12:17:09.541617 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:10.859777 (+1318160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:11.971738 (+1111961us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:11.971931 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2955800832}
I20260709 12:17:11.974957 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:11.973562 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:11.889782 24481 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.032492 12568 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 38 because replica has last-logged OpId of term: 36 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 16.
I20260709 12:17:11.955955 24811 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:12.043875 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:12.006124 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 182920us
W20260709 12:17:12.047698 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:11.970695 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:12.008234 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:11.979106 24615 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:11.940462 24779 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:12.017107 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 180250us
I20260709 12:17:12.003279 24785 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.024312 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:12.076251 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:12.033125 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59654.1us
I20260709 12:17:12.077445 12566 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:12.056237 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:11.995644 13253 rpcz_store.cc:276] 0709 12:17:09.475094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.475246 (+   152us) service_pool.cc:224] Handling call
0709 12:17:09.502259 (+ 27013us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:09.540885 (+ 38626us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:09.540915 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:09.581344 (+ 40429us) log.cc:844] Serialized 26 byte log entry
0709 12:17:10.258952 (+677608us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:10.258996 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:10.259068 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:11.944798 (+1685730us) raft_consensus.cc:1685] finished
0709 12:17:11.944820 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:11.944953 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43625,"raft.queue_time_us":69988,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":8395,"thread_start_us":636823,"threads_started":1,"wal-append.queue_time_us":1509366}
I20260709 12:17:12.037933 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.025991 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:12.050371 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:12.082772 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.083354 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:12.084456 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.084578 12565 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:12.085089 12565 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:12.085237 13254 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:12.086374 12567 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:12.087366 12462 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:12.088676 13260 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.28-40.28]
I20260709 12:17:12.090021 12569 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 39 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 15.
I20260709 12:17:12.092993 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:12.093952 12489 raft_consensus.cc:606] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:12.096487 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:11.965229 24729 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 36.27, Last appended by leader: 27, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.034924 24813 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:12.033895 24822 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:12.086719 24828 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:12.005996 24638 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.088s
I20260709 12:17:12.109282 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:12.091230 12565 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.25, Last appended by leader: 25, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.115401 13260 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:12.075987 24689 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:12.132932 12565 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:12.094609 24821 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:12.090497 24717 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:12.132725 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.089262 24825 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:12.101133 24684 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:17:12.116452 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:12.163790 13262 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 26: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:12.010399 24826 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.104218 24759 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:12.115762 24813 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:12.111707 24781 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:12.090453 24811 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:12.158804 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:12.187184 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.110605 12571 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:12.153314 24821 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:12.088727 24564 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260709 12:17:12.094300 24484 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.050s
I20260709 12:17:12.127522 24729 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:12.145046 24840 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 12:17:12.118178 24822 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:12.001506 24812 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.187865 24825 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:12.224946 12643 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: [35.22-37.23]
I20260709 12:17:12.226755 12643 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
W20260709 12:17:12.179276 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70098.4us
I20260709 12:17:12.227795 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:12.135018 24732 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:12.095676 24779 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.188835 24329 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:12.192852 24841 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
W20260709 12:17:12.195365 24845 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:17:12.181092 24842 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:12.197679 24701 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:12.200747 24811 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:12.167675 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.153808 24828 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:12.222744 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:12.229641 24745 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.200770 24797 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.228564 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.234064 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:12.234941 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 262170us
I20260709 12:17:12.265261 12645 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:12.214360 24781 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:12.260325 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.309813 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.311323 12646 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:12.261078 12640 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:12.254339 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:12.261596 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.315200 24855 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:12.315905 24674 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.235644 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:12.187225 24759 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.319020 24759 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:12.319981 24825 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:17:12.321036 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85523.2us
I20260709 12:17:12.321862 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:12.322707 24825 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:12.323298 24825 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:12.322799 24781 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.324294 24825 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.255661 24838 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:12.324679 24632 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.331077 24740 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:12.305409 24812 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:12.233771 24717 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:12.344841 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.226389 24822 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:12.329986 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:17:12.333041 24828 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.316020 24855 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.331015 24819 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:12.254050 24842 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:12.233289 24732 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:12.342286 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.309924 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:12.345779 13262 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:17:12.209168 24821 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:12.350850 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.365206 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.258960 24329 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:12.338315 24768 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.190s	user 0.005s	sys 0.009s
I20260709 12:17:12.373726 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.345314 24674 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:12.332942 24003 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:12.366158 13253 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 34 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
I20260709 12:17:12.367444 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.367929 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.378160 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:12.380007 12493 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:12.338775 24781 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.364476 12489 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:17:12.331396 24779 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:12.375151 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65297.9us
I20260709 12:17:12.344513 24717 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:12.343775 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:12.386478 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.363112 24842 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:12.387893 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:12.390708 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.391628 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:12.370219 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 277662us
I20260709 12:17:12.363813 24732 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:12.370570 24632 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:12.352914 12645 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:12.418648 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:12.421010 12645 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:12.426880 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:12.427727 12640 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:12.434883 12637 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
W20260709 12:17:12.443274 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48418.7us
W20260709 12:17:12.465735 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.475163 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:12.482088 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:12.482482 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.483377 12640 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:12.484577 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.486387 12640 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:12.488790 12641 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:12.392666 24819 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:12.496476 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:12.492831 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 232448us
I20260709 12:17:12.499648 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.500619 12496 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:12.503437 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:12.504521 24847 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:17:12.508622 24855 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:12.384104 24674 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.392887 24745 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.119s
I20260709 12:17:12.511987 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.512843 24674 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.513288 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.383872 24624 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.514688 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:12.515467 12496 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:12.516170 13254 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:12.516335 12646 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:17:12.518590 12640 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:12.419780 24852 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:12.434020 24781 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:12.434556 24564 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:17:12.475318 24740 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:12.397241 24779 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.124s
I20260709 12:17:12.483180 24838 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:12.558447 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3831) took 1647 ms (client timeout 1880 ms). Trace:
I20260709 12:17:12.558755 12641 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
W20260709 12:17:12.558961 12644 rpcz_store.cc:269] 0709 12:17:10.911512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:10.911688 (+   176us) service_pool.cc:224] Handling call
0709 12:17:12.558202 (+1646514us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:12.558358 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60767104}
I20260709 12:17:12.560546 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.561811 12644 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:17:12.563947 12644 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 15. Preceding OpId from leader: term: 40 index: 17. (index mismatch)
I20260709 12:17:12.557327 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1522) took 2844 ms. Trace:
I20260709 12:17:12.559653 12641 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 26: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:12.526595 12565 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 25. Preceding OpId from leader: term: 42 index: 25. (term mismatch)
I20260709 12:17:12.532536 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:12.573421 12646 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:12.576020 12644 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:17:12.576628 24329 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:12.569008 24689 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:12.577705 24684 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:17:12.578661 24689 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.523133 24717 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.515050 24819 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:12.552559 12492 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.040s	sys 0.036s
W20260709 12:17:12.593649 12492 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.040s	sys 0.036s
I20260709 12:17:12.540704 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:12.594732 12640 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:12.490329 24785 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:17:12.526764 24852 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:12.568960 13259 rpcz_store.cc:276] 0709 12:17:09.712849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:09.713059 (+   210us) service_pool.cc:224] Handling call
0709 12:17:09.713433 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:10.660925 (+947492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:12.556829 (+1895904us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:12.557222 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2095601408}
I20260709 12:17:12.581907 12565 pending_rounds.cc:85] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 24
I20260709 12:17:12.596722 12565 pending_rounds.cc:107] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.25
I20260709 12:17:12.597119 12565 raft_consensus.cc:2889] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:12.555265 24740 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:12.596864 24819 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:12.540223 24781 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.570495 24813 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:12.612821 24740 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:12.599903 24852 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:12.628619 24717 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:12.637429 24817 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:12.678200 24850 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 12:17:12.687731 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.689520 12640 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:12.693924 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:12.711767 13259 raft_consensus.cc:606] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:12.712366 13259 raft_consensus.cc:614] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:17:12.697003 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.690125 24828 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:12.683526 24624 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:12.701165 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 221734us
I20260709 12:17:12.694594 24729 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:12.713712 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.702713 24785 consensus_queue.cc:1243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:12.728156 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:12.715301 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.681371 24817 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.733500 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:12.732498 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:12.734470 13253 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:12.751000 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:12.752583 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 277519us
I20260709 12:17:12.761037 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:12.744542 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:12.770378 24808 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.737s	user 0.007s	sys 0.001s
I20260709 12:17:12.773797 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:12.774919 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:12.727730 24828 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:12.735723 24632 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:17:12.730629 24729 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.729508 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:12.748239 24317 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:17:12.776356 24856 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.182s	user 0.001s	sys 0.001s
I20260709 12:17:12.799010 12568 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:12.730074 24747 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:17:12.796494 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:12.724291 24783 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260709 12:17:12.767256 24615 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:12.744436 24638 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.053s
I20260709 12:17:12.807312 24825 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:12.795751 24785 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:12.807554 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.800503 24623 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:17:12.815961 24857 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:17:12.816247 24684 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:12.817188 24638 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:12.827451 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.838075 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:12.810292 24317 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:12.798564 24797 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.025s
I20260709 12:17:12.816042 24632 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:12.863042 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.896826 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:12.822600 24878 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:12.827564 24684 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:12.820873 24825 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:12.838078 24813 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:12.899539 12497 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
W20260709 12:17:12.905361 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145183us
I20260709 12:17:12.839732 24638 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:12.901098 12640 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:17:12.855449 24781 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:12.855151 24623 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:12.864817 24632 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:13.025841 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.718s (SENT) [suppressed 18 similar messages]
I20260709 12:17:12.950448 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2354) took 1139 ms. Trace:
I20260709 12:17:13.006275 24869 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.015s
I20260709 12:17:12.909652 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:12.994611 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:12.984069 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:12.992841 24861 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.272s	user 0.002s	sys 0.000s
W20260709 12:17:12.995551 24862 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.282s	user 0.000s	sys 0.002s
I20260709 12:17:13.033872 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:12.570636 24838 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:12.911880 24828 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.043272 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:13.058740 24749 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:17:13.059082 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:13.059284 24749 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:13.043483 12496 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:13.060639 24749 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.014806 24825 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:13.044014 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 134480us
I20260709 12:17:13.007138 24884 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:17:13.041448 12566 rpcz_store.cc:276] 0709 12:17:11.810573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.078175 (+267602us) service_pool.cc:224] Handling call
0709 12:17:12.106132 (+ 27957us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:12.106564 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:12.106597 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:12.120409 (+ 13812us) log.cc:844] Serialized 26 byte log entry
0709 12:17:12.520907 (+400498us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:12.520962 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:12.521056 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:12.949872 (+428816us) raft_consensus.cc:1685] finished
0709 12:17:12.949902 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:12.950325 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":7384,"thread_start_us":373592,"threads_started":1,"wal-append.queue_time_us":538531}
I20260709 12:17:13.052971 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:13.041481 24632 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:17:13.048435 24653 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.184s	user 0.012s	sys 0.005s
I20260709 12:17:13.049876 24883 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.011s
I20260709 12:17:13.039989 24729 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:13.057179 24838 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.056341 24623 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.067418 24879 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:17:13.064881 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3784) took 1751 ms. Trace:
I20260709 12:17:13.071619 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.004359 24317 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.073093 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.035848 24615 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:17:13.065320 24638 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.20, Last appended by leader: 20, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.101428 24719 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.101555 24638 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:13.047848 24858 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:13.104944 24858 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:13.105330 24875 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:17:13.106757 24815 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:13.107214 24815 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:13.107687 24872 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:17:13.037724 24813 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 12:17:13.086154 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1740) took 4147 ms. Trace:
I20260709 12:17:13.108235 24815 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.108788 13255 rpcz_store.cc:276] 0709 12:17:08.939205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:08.939395 (+   190us) service_pool.cc:224] Handling call
0709 12:17:08.944554 (+  5159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:09.003813 (+ 59259us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:09.367634 (+363821us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.101394 (+733760us) log.cc:844] Serialized 14 byte log entry
0709 12:17:10.251259 (+149865us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b18003a20a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:17:10.987235 (+735976us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:10.987321 (+    86us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:10.987407 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:11.871008 (+883601us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b18003a20a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:17:12.682586 (+811578us) spinlock_profiling.cc:243] Waited 9.28 ms on lock 0x7b18003a20a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:13.085824 (+403238us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:13.085881 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:13.086075 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":152072,"prepare.queue_time_us":790857,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":126,"raft.queue_time_us":217363,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":22122,"spinlock_wait_cycles":177096960,"thread_start_us":1757756,"threads_started":3,"wal-append.queue_time_us":997662,"wal-append.run_cpu_time_us":3323,"wal-append.run_wall_time_us":1421375}
W20260709 12:17:13.077512 24876 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:17:13.069608 24740 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.095942 24870 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.270s	user 0.002s	sys 0.000s
I20260709 12:17:13.090876 12639 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 41 index: 28. (term mismatch)
I20260709 12:17:13.047318 24693 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.039s
I20260709 12:17:13.113312 24815 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:13.128072 24880 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.136s	user 0.002s	sys 0.000s
I20260709 12:17:13.128683 24740 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.095338 13253 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:13.093200 12495 rpcz_store.cc:276] 0709 12:17:11.313182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:11.313336 (+   154us) service_pool.cc:224] Handling call
0709 12:17:11.313712 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:11.884929 (+571217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.064483 (+1179554us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b540012bbd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:13.064771 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1049307392}
I20260709 12:17:13.096836 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
W20260709 12:17:13.132995 24873 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 12:17:13.133250 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.133796 24831 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:13.100378 24780 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.064s	user 0.007s	sys 0.010s
I20260709 12:17:13.105432 24858 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:17:13.140874 24732 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.665s	user 0.013s	sys 0.002s
I20260709 12:17:13.141187 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:13.141837 24732 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.144516 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:13.145110 24817 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.145546 12643 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:13.151446 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.152161 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:13.153249 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:13.154042 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4891) took 1782 ms (client timeout 1586 ms). Trace:
I20260709 12:17:13.154338 12571 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 16. Preceding OpId from leader: term: 40 index: 17. (index mismatch)
I20260709 12:17:13.153496 12493 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:13.156080 12643 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:13.157255 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:13.157958 24881 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:17:13.158785 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.159735 12643 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:13.160964 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.164603 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:17:13.168298 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1894) took 1639 ms. Trace:
I20260709 12:17:13.170082 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:13.171021 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:13.171612 24821 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.799s	user 0.013s	sys 0.000s
I20260709 12:17:13.149622 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:13.172291 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:13.172228 24821 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.173158 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:13.173497 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:13.095600 24825 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:13.134397 12638 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:13.142343 12568 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:17:13.094595 12640 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:13.167840 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:13.168859 12491 rpcz_store.cc:276] 0709 12:17:11.528563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:11.814386 (+285823us) service_pool.cc:224] Handling call
0709 12:17:11.814916 (+   530us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:12.033957 (+219041us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:12.033992 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:12.075144 (+ 41152us) log.cc:844] Serialized 26 byte log entry
0709 12:17:12.387678 (+312534us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:12.387737 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.167879 (+780142us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:13.167932 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:13.167981 (+    49us) raft_consensus.cc:1685] finished
0709 12:17:13.167994 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:13.168227 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":535,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":51141,"spinlock_wait_cycles":502618880,"thread_start_us":266818,"threads_started":1,"wal-append.queue_time_us":438306}
I20260709 12:17:13.182719 13257 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:13.078186 24770 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.479s	user 0.000s	sys 0.015s
I20260709 12:17:13.171478 12642 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:60230
I20260709 12:17:13.167754 23540 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:13.118106 12639 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 27
I20260709 12:17:13.146564 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:13.184486 12639 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 40.28
I20260709 12:17:13.184844 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:13.184988 12639 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:13.184283 12642 raft_consensus.cc:606] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:13.185937 12642 raft_consensus.cc:614] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:13.186995 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:13.085041 12496 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:13.188551 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:13.154552 12569 rpcz_store.cc:269] 0709 12:17:11.371502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:11.371664 (+   162us) service_pool.cc:224] Handling call
0709 12:17:13.153723 (+1782059us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54003e0154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:13.153900 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1503407360}
I20260709 12:17:13.196764 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:13.197005 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.198575 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:13.201745 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.192713 23540 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:13.192922 12568 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 28.
I20260709 12:17:13.204974 12490 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:13.206024 12643 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:13.054332 24684 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.205788 12491 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 17. Preceding OpId from leader: term: 36 index: 18. (index mismatch)
I20260709 12:17:13.192565 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.215822 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.218143 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.211135 12642 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [32.19-34.20]
I20260709 12:17:13.221416 12642 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
W20260709 12:17:13.205927 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:13.216068 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.224141 12490 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:13.208681 12643 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: [35.18-35.18]
I20260709 12:17:13.219419 12491 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:13.226770 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:13.228108 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.228533 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.231076 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.232430 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.233536 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.233256 12490 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:17:13.186224 24329 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.241335 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:17:13.213229 24822 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.847s	user 0.012s	sys 0.006s
I20260709 12:17:13.191442 24693 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.258334 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.256214 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.204465 24813 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.213737 24484 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:13.229820 24887 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:17:13.263386 13257 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:13.267292 12494 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
W20260709 12:17:13.251767 24717 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.005s
I20260709 12:17:13.283382 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.283041 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.225487 24885 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:13.337867 12497 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:13.338608 13257 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:13.344765 13257 raft_consensus.cc:606] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:13.266057 24842 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.832s	user 0.015s	sys 0.000s
W20260709 12:17:13.282248 24812 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.915s	user 0.011s	sys 0.005s
W20260709 12:17:13.271018 24003 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.889s	user 0.013s	sys 0.002s
I20260709 12:17:13.379223 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:17:13.379439 12495 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:17:13.388376 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.389757 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:13.330504 24826 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:13.358997 24885 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.400311 24886 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:17:13.400652 24821 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:13.403306 24859 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.759s	user 0.004s	sys 0.002s
W20260709 12:17:13.405745 24874 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.590s	user 0.003s	sys 0.000s
I20260709 12:17:13.373378 24745 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:13.409056 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.410444 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:13.266875 24329 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.281826 24813 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.359164 24811 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.422698 12643 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:13.320170 23540 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:13.265820 24822 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:13.283650 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:13.281320 24770 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:17:13.427171 12490 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.28-40.28]
W20260709 12:17:13.440702 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 414838us
W20260709 12:17:13.441392 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.012s	sys 0.024s
I20260709 12:17:13.443778 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:13.284384 24817 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:13.407889 24819 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.759s	user 0.014s	sys 0.001s
I20260709 12:17:13.469964 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:13.507665 12644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.012s	sys 0.004s
W20260709 12:17:13.472697 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 215184us
W20260709 12:17:13.507984 12644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.012s	sys 0.004s
I20260709 12:17:13.261445 24803 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:13.485468 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:13.482506 23540 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.396552 24853 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.268508 24693 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.492219 13257 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
W20260709 12:17:13.500103 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.013s	sys 0.025s
I20260709 12:17:13.332118 24484 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.500325 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56624.4us
I20260709 12:17:13.379099 24842 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.380422 24858 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.452939 24885 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:13.509972 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:13.506558 12637 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.006s	sys 0.010s
W20260709 12:17:13.567975 12637 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.006s	sys 0.010s
I20260709 12:17:13.480006 24623 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.516260 12488 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:13.523940 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5097) took 1562 ms. Trace:
I20260709 12:17:13.523742 12563 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:13.525921 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:13.503568 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.010s	sys 0.010s
I20260709 12:17:13.507121 24789 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:13.528587 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:13.534305 12646 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.008s	sys 0.013s
W20260709 12:17:13.573891 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.010s	sys 0.011s
W20260709 12:17:13.581136 12646 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.008s	sys 0.014s
W20260709 12:17:13.581162 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 745232us
I20260709 12:17:13.584021 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:13.554275 24853 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.591883 23540 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:13.590965 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:13.593000 13259 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:13.552970 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:13.565929 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:13.522656 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:13.451380 24745 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.604581 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76109.9us
W20260709 12:17:13.605988 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83455.2us
I20260709 12:17:13.594688 13253 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:13.609871 13253 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:13.516285 24868 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:13.585978 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:13.593286 23540 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:13.612831 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4706) took 1012 ms. Trace:
I20260709 12:17:13.531090 24312 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:13.496325 24811 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.615082 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49252us
I20260709 12:17:13.622640 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:13.619187 12646 rpcz_store.cc:276] 0709 12:17:12.600673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.600812 (+   139us) service_pool.cc:224] Handling call
0709 12:17:12.630541 (+ 29729us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:12.631001 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:12.631029 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:12.637206 (+  6177us) log.cc:844] Serialized 26 byte log entry
0709 12:17:12.799685 (+162479us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:13.127354 (+327669us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:13.581915 (+454561us) log.cc:844] Serialized 14 byte log entry
0709 12:17:13.591624 (+  9709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.591720 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:13.591773 (+    53us) raft_consensus.cc:1685] finished
0709 12:17:13.612525 (+ 20752us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:13.612751 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8992,"prepare.queue_time_us":236961,"prepare.run_cpu_time_us":3828,"prepare.run_wall_time_us":4597,"raft.queue_time_us":2471,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":5869,"spinlock_wait_cycles":7812992,"thread_start_us":879884,"threads_started":2,"wal-append.queue_time_us":193740}
W20260709 12:17:13.582299 24864 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.016s	sys 0.023s
I20260709 12:17:13.606959 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:13.626787 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:13.616209 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:13.629787 23540 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.661157 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:13.590853 24789 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:13.661376 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:13.724303 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63040.8us
I20260709 12:17:13.725458 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:13.615420 24732 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:13.667827 12572 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.739439 12572 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:13.515710 24817 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:13.526856 24819 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:13.675275 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68442.8us
W20260709 12:17:13.676187 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 60126.7us
I20260709 12:17:13.676187 23540 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.625907 24677 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:13.604439 24828 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:13.723349 12644 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:13.625214 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:13.727159 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4697) took 1294 ms. Trace:
I20260709 12:17:13.617233 24868 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:13.733644 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111194us
I20260709 12:17:13.612848 24902 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:13.661089 24884 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:13.532809 24803 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.590570 24623 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.571442 12565 rpcz_store.cc:276] 0709 12:17:11.961223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.598460 (+637237us) service_pool.cc:224] Handling call
0709 12:17:12.598934 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:12.600192 (+  1258us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:12.802836 (+202644us) log.cc:844] Serialized 14 byte log entry
0709 12:17:13.520844 (+718008us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:13.520936 (+    92us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:13.521047 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.521156 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:13.523862 (+  2706us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":102747,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":7440,"raft.queue_time_us":139,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":23315,"thread_start_us":890934,"threads_started":2,"wal-append.queue_time_us":703114}
W20260709 12:17:13.982865 12638 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.030s	sys 0.051s
I20260709 12:17:13.746747 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:14.046474 12572 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
W20260709 12:17:14.193459 24405 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 36 current term: 36 is higher.
I20260709 12:17:13.873447 24813 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:13.975508 13257 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:17:13.869185 24811 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.869468 24842 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:13.873059 24484 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.197512 24811 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:14.198770 23540 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.923981 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:14.201687 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 277741us
I20260709 12:17:14.193511 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4888) took 2827 ms. Trace:
I20260709 12:17:14.204962 12572 rpcz_store.cc:276] 0709 12:17:11.365935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:11.366135 (+   200us) service_pool.cc:224] Handling call
0709 12:17:11.398930 (+ 32795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:13.667345 (+2268415us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b4c003e0e80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf3abfa20 00007ffbf3ad0a29 00007ffbf3a96644 00007ffbf3abdf2d 00007ffbf3abdb93 00007ffbf57ee18b 00007ffbf57eecb5 00007ffbf57e9f30 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416
0709 12:17:14.193172 (+525827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:14.193447 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234454272}
W20260709 12:17:13.746431 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 114384us
W20260709 12:17:14.029461 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:14.047458 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:13.873890 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5095) took 1920 ms. Trace:
I20260709 12:17:13.881553 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:14.208170 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:13.902921 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5094) took 1949 ms. Trace:
I20260709 12:17:14.211987 12572 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 15.
W20260709 12:17:13.872800 12609 connection.cc:256] RPC call timeout handler was delayed by -1.1288923199999772s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15860000000000002s for a response
W20260709 12:17:14.187883 12637 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.014s	sys 0.025s
I20260709 12:17:13.886390 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1756) took 1761 ms. Trace:
I20260709 12:17:13.627611 24312 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:14.216347 12637 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.015s	sys 0.026s
I20260709 12:17:14.203954 23540 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.002s
I20260709 12:17:13.974531 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:14.211818 12563 rpcz_store.cc:276] 0709 12:17:11.953180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.159102 (+205922us) service_pool.cc:224] Handling call
0709 12:17:12.159438 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:13.579603 (+1420165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.902532 (+322929us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:13.902819 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3121888128}
W20260709 12:17:13.917124 24852 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.246s	user 0.013s	sys 0.000s
I20260709 12:17:14.209488 12571 rpcz_store.cc:276] 0709 12:17:11.953453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.211368 (+257915us) service_pool.cc:224] Handling call
0709 12:17:13.153910 (+942542us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:13.154976 (+  1066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.873584 (+718608us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54003e0150. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:13.873808 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2166596992}
W20260709 12:17:14.216737 12609 connection.cc:256] RPC call timeout handler was delayed by -1.1144923199999539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20260709 12:17:14.070372 13258 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:14.212219 12641 rpcz_store.cc:276] 0709 12:17:12.432284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.490080 (+ 57796us) service_pool.cc:224] Handling call
0709 12:17:13.726781 (+1236701us) spinlock_profiling.cc:243] Waited 54.4 ms on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:13.727043 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125114752}
I20260709 12:17:13.892663 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:14.251335 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:14.250389 13254 rpcz_store.cc:276] 0709 12:17:12.124907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.125098 (+   191us) service_pool.cc:224] Handling call
0709 12:17:12.125452 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:12.516618 (+391166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.885998 (+1369380us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:13.886294 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898017024}
I20260709 12:17:14.261687 12565 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:17:13.666798 24653 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:14.251344 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:13.745941 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:14.226531 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:14.214601 24864 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.412s	user 0.019s	sys 0.028s
W20260709 12:17:13.733651 24880 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:17:13.627415 24838 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:14.252035 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4700) took 1819 ms. Trace:
I20260709 12:17:14.268680 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:14.194424 24677 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:14.118678 13261 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:14.266866 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:14.102432 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:13.972072 24815 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.848s	user 0.016s	sys 0.000s
I20260709 12:17:14.119644 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:13.917692 24871 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.111s	user 0.004s	sys 0.000s
W20260709 12:17:13.680845 24875 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:17:13.723685 24812 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:17:14.317118 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:14.136511 24684 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:13.915155 24821 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:14.362504 24821 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:14.363010 24821 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:13.807914 24826 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:14.366691 24826 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:14.367331 24826 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:13.908010 24745 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:14.272598 24003 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.045s
I20260709 12:17:14.306911 24852 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.373102 24852 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.139947 24689 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:13.738528 24789 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:14.374437 24819 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.374771 24689 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:14.375365 24689 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:14.337095 24838 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:14.319068 24312 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:14.222892 24825 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.047s	user 0.015s	sys 0.000s
I20260709 12:17:14.378541 24825 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:14.378907 24838 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.379185 24825 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.321568 23697 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:14.381973 23697 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.378911 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:14.383515 13257 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:13.877005 24701 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:14.384914 24701 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:14.386663 24701 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.26, Last appended by leader: 26, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.389118 24701 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:13.728322 24740 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:14.390162 24740 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:14.390410 24854 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.752s	user 0.001s	sys 0.005s
I20260709 12:17:14.391558 24740 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.392851 24789 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:13.882273 24890 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.222s	user 0.000s	sys 0.000s
W20260709 12:17:14.052376 24632 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.891s	user 0.006s	sys 0.007s
W20260709 12:17:13.882750 24888 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.291s	user 0.000s	sys 0.000s
W20260709 12:17:13.878185 24899 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
W20260709 12:17:13.883755 24893 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
W20260709 12:17:13.876818 24891 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.307s	user 0.000s	sys 0.000s
W20260709 12:17:14.319983 24878 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.353s	user 0.006s	sys 0.011s
I20260709 12:17:13.972568 24781 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:13.914935 24853 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:13.875012 24693 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:13.915943 24858 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.223268 24867 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:17:14.425344 24781 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:13.911445 24822 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.318261 24817 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:14.361079 24892 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.307s	user 0.000s	sys 0.000s
I20260709 12:17:14.381228 24754 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.128468 24801 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.218s
I20260709 12:17:14.370416 24745 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:14.394699 24812 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:14.402657 24885 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.008s	sys 0.043s
I20260709 12:17:14.259516 24828 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.260619 24884 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.427770 24878 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:14.400573 24868 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:14.438961 24885 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.008s	sys 0.044s
I20260709 12:17:14.448843 24868 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:14.498173 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:13.948318 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:14.506248 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.002s	sys 0.007s
I20260709 12:17:14.506719 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:14.507261 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.002s	sys 0.007s
W20260709 12:17:14.509526 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.016s	sys 0.033s
W20260709 12:17:14.510397 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.016s	sys 0.034s
I20260709 12:17:13.872602 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:14.252693 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:14.523527 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:13.930234 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:14.264312 12568 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 35 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 18.
I20260709 12:17:14.261528 12569 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:14.260051 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:14.322137 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:14.322494 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 576586us
I20260709 12:17:14.308782 12637 rpcz_store.cc:276] 0709 12:17:12.433130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.496764 (+ 63634us) service_pool.cc:224] Handling call
0709 12:17:12.497171 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:12.517403 (+ 20232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:12.517437 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:12.522884 (+  5447us) log.cc:844] Serialized 26 byte log entry
0709 12:17:12.764622 (+241738us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:13.568443 (+803821us) log.cc:844] Serialized 14 byte log entry
0709 12:17:14.251654 (+683211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:14.251744 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:14.251794 (+    50us) raft_consensus.cc:1685] finished
0709 12:17:14.251828 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.251977 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19718,"prepare.queue_time_us":257440,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":3397,"raft.queue_time_us":83835,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":37284,"thread_start_us":1623215,"threads_started":3,"wal-append.queue_time_us":390073}
I20260709 12:17:14.700596 24904 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.003s
I20260709 12:17:14.702665 24904 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:14.703380 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:14.703229 24904 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:14.546841 12494 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:14.546746 12566 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:17:14.700405 24822 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:14.706624 24858 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:14.707741 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:14.707695 24858 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:14.709177 12566 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 20.
W20260709 12:17:14.548194 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 240719us
I20260709 12:17:14.711083 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:14.712373 12569 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:14.549335 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4717) took 1110 ms. Trace:
I20260709 12:17:14.550518 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:14.713884 12642 rpcz_store.cc:276] 0709 12:17:13.438747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.438881 (+   134us) service_pool.cc:224] Handling call
0709 12:17:13.439264 (+   383us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:13.439748 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:17:13.498627 (+ 58879us) log.cc:844] Serialized 14 byte log entry
0709 12:17:13.833667 (+335040us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:13.858209 (+ 24542us) log.cc:844] Serialized 52 byte log entry
0709 12:17:13.858472 (+   263us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:17:13.858514 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.858602 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:14.548588 (+689986us) raft_consensus.cc:1685] finished
0709 12:17:14.548642 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.549156 (+   514us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59934,"prepare.queue_time_us":83472,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":54,"raft.queue_time_us":366085,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":29641,"thread_start_us":265296,"threads_started":1,"wal-append.queue_time_us":847493}
I20260709 12:17:14.712625 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:14.552472 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4708) took 1910 ms. Trace:
I20260709 12:17:14.553030 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4718) took 1113 ms. Trace:
I20260709 12:17:14.505009 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:14.719583 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1897) took 3155 ms. Trace:
I20260709 12:17:14.719432 24822 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:14.555359 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:14.721124 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:14.544317 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4900) took 3107 ms. Trace:
I20260709 12:17:14.531345 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:14.332052 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63464us
W20260709 12:17:14.567512 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.015s	sys 0.034s
I20260709 12:17:14.341787 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:14.352937 23540 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:17:14.384310 23697 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:14.187305 24317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.014s	sys 0.026s
I20260709 12:17:13.910027 24329 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:13.916466 24481 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.435839 24812 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:14.373142 24623 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:14.579715 24674 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:14.337813 24677 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:14.748399 24907 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:14.418202 24632 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.751662 13255 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:14.753144 24632 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.583189 24745 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:14.753774 24717 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.021s	sys 0.016s
W20260709 12:17:14.754653 24717 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.021s	sys 0.017s
I20260709 12:17:14.586934 24693 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:14.590507 24797 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:14.595902 24803 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.544550 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4709) took 1882 ms. Trace:
I20260709 12:17:14.724961 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:14.604668 24828 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.756875 24693 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:14.673247 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128688us
I20260709 12:17:14.700276 12489 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:14.717896 12645 rpcz_store.cc:276] 0709 12:17:12.641945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.175796 (+533851us) service_pool.cc:224] Handling call
0709 12:17:13.176220 (+   424us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:13.176847 (+   627us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:13.176927 (+    80us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:13.181016 (+  4089us) log.cc:844] Serialized 52 byte log entry
0709 12:17:13.691058 (+510042us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:13.838759 (+147701us) log.cc:844] Serialized 14 byte log entry
0709 12:17:13.839050 (+   291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.839169 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:14.550709 (+711540us) raft_consensus.cc:1685] finished
0709 12:17:14.550751 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.552217 (+  1466us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":147006,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":364516,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":251,"thread_start_us":489494,"threads_started":1,"wal-append.queue_time_us":1200874}
W20260709 12:17:14.726003 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.015s	sys 0.035s
W20260709 12:17:14.730147 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:14.716158 24858 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:14.731865 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:14.718667 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:14.598712 24815 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.002s
I20260709 12:17:14.747304 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:14.746230 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 163152us
I20260709 12:17:14.588886 24789 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.776768 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:14.761539 24329 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:14.725062 12643 rpcz_store.cc:276] 0709 12:17:13.439111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.481231 (+ 42120us) service_pool.cc:224] Handling call
0709 12:17:13.481679 (+   448us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:13.628101 (+146422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:13.628154 (+    53us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:17:13.660927 (+ 32773us) log.cc:844] Serialized 78 byte log entry
0709 12:17:13.863935 (+203008us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:13.864004 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:13.864141 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:14.551710 (+687569us) raft_consensus.cc:1685] finished
0709 12:17:14.551853 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.552582 (+   729us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110702,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":177038,"threads_started":2,"wal-append.queue_time_us":656505}
I20260709 12:17:14.768471 24828 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:14.758112 24674 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.755604 24745 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.771616 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:14.771948 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5039) took 1574 ms (client timeout 1602 ms). Trace:
W20260709 12:17:14.791025 12489 rpcz_store.cc:269] 0709 12:17:13.198020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.198206 (+   186us) service_pool.cc:224] Handling call
0709 12:17:14.771594 (+1573388us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:14.771833 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2978440448}
I20260709 12:17:14.774071 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:14.774389 24317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.212s	user 0.015s	sys 0.027s
I20260709 12:17:14.778656 24815 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:14.780771 24825 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.727049 23540 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:14.775727 24693 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.763860 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:14.789348 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77159us
I20260709 12:17:14.806423 24405 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:14.720101 12492 rpcz_store.cc:276] 0709 12:17:11.564362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:11.874574 (+310212us) service_pool.cc:224] Handling call
0709 12:17:11.875086 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:11.965806 (+ 90720us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:17:12.594172 (+628366us) log.cc:844] Serialized 14 byte log entry
0709 12:17:12.753583 (+159411us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:12.753694 (+   111us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:12.753772 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:14.718816 (+1965044us) spinlock_profiling.cc:243] Waited 90.4 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:14.718950 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.719328 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":232886,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":14366,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":10557,"spinlock_wait_cycles":207887744,"thread_start_us":771157,"threads_started":2,"wal-append.queue_time_us":321576,"wal-append.run_cpu_time_us":2928,"wal-append.run_wall_time_us":1201056}
I20260709 12:17:14.808571 12642 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:14.705878 24904 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.809616 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:14.810081 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64907.2us
I20260709 12:17:14.812130 24904 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:14.816170 24329 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.818768 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:14.819535 12645 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:17:14.820479 13261 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:14.823506 24674 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:14.825138 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89382.3us
I20260709 12:17:14.822049 12637 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:14.804054 23540 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:14.809448 13257 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:14.829139 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:14.807921 24405 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:14.842697 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:14.844542 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:14.819938 24329 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.794004 24815 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:14.850756 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:14.853586 24405 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:14.724202 12570 rpcz_store.cc:276] 0709 12:17:11.437057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.031438 (+594381us) service_pool.cc:224] Handling call
0709 12:17:12.040617 (+  9179us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:12.041472 (+   855us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:12.041515 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:12.042660 (+  1145us) log.cc:844] Serialized 26 byte log entry
0709 12:17:12.418743 (+376083us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:12.418780 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:12.418871 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:14.389434 (+1970563us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:14.529088 (+139654us) raft_consensus.cc:1685] finished
0709 12:17:14.529163 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.543904 (+ 14741us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12749184,"thread_start_us":306402,"threads_started":1,"wal-append.queue_time_us":502150}
I20260709 12:17:14.861953 24817 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:14.862802 24817 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:14.863507 24817 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:14.851174 24815 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.23, Last appended by leader: 23, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.871687 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:14.873492 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43600.6us
I20260709 12:17:14.832559 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:14.858304 24828 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:14.834534 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1793) took 1295 ms. Trace:
I20260709 12:17:14.876173 12642 rpcz_store.cc:276] 0709 12:17:13.513892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.717412 (+1203520us) service_pool.cc:224] Handling call
0709 12:17:14.794466 (+ 77054us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:14.809290 (+ 14824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:14.809557 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:14.834018 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 780401us
I20260709 12:17:14.884541 12566 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:14.886193 12566 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 33 because replica has last-logged OpId of term: 32 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
W20260709 12:17:14.887295 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102453us
I20260709 12:17:14.758492 12638 rpcz_store.cc:276] 0709 12:17:12.662227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.179439 (+517212us) service_pool.cc:224] Handling call
0709 12:17:13.179993 (+   554us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:13.180562 (+   569us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:13.180593 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:13.181274 (+   681us) log.cc:844] Serialized 52 byte log entry
0709 12:17:13.435738 (+254464us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:13.479397 (+ 43659us) log.cc:844] Serialized 14 byte log entry
0709 12:17:14.204440 (+725043us) log.cc:844] Serialized 14 byte log entry
0709 12:17:14.204701 (+   261us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:14.204817 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:14.543569 (+338752us) raft_consensus.cc:1685] finished
0709 12:17:14.543630 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:14.544202 (+   572us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26643,"prepare.queue_time_us":29905,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":102,"raft.queue_time_us":211816,"raft.run_cpu_time_us":423,"raft.run_wall_time_us":45892,"thread_start_us":964390,"threads_started":2,"wal-append.queue_time_us":350435}
I20260709 12:17:14.901852 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:14.899134 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:14.899111 24819 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:14.916080 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5034) took 1722 ms. Trace:
I20260709 12:17:14.916570 12495 rpcz_store.cc:276] 0709 12:17:13.193157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.194391 (+  1234us) service_pool.cc:224] Handling call
0709 12:17:13.194964 (+   573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:13.387150 (+192186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:14.915383 (+1528233us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:14.915796 (+   413us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423276800}
W20260709 12:17:14.923805 24701 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.007s
I20260709 12:17:14.933064 24919 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:14.934286 24919 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:14.936810 24929 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:14.938189 24928 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:17:14.939195 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67748.1us
I20260709 12:17:14.939565 24329 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:14.940753 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90722.8us
I20260709 12:17:14.942167 24906 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:14.943027 24906 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:14.943593 24906 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:14.941005 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:14.960741 24405 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:14.945907 24927 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:14.947672 24925 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:14.953517 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54637.9us
I20260709 12:17:14.940841 24898 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:14.958582 24701 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:14.979969 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:14.974471 24828 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:14.994204 24869 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:14.972530 24819 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:14.962033 24405 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:14.998912 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:14.979207 24701 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:15.019065 24812 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.019474 24819 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.021840 24719 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.022179 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.023612 12566 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:17:15.023114 24719 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.026309 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.026731 24405 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.027705 13256 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 26.
W20260709 12:17:15.019523 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78846.4us
I20260709 12:17:15.028963 24902 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.029798 24902 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.029954 24869 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.032665 24869 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.032841 24615 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.033517 24615 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.013989 24828 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:15.034904 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:15.035351 24828 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.036128 12489 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:15.037988 24858 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:15.038738 24858 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:15.032668 24902 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.039215 24858 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:17:15.039458 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45123.3us
W20260709 12:17:15.044210 24689 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.668s	user 0.011s	sys 0.016s
W20260709 12:17:15.046838 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 57333.3us
I20260709 12:17:15.003961 24919 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.012858 24701 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:15.050715 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:15.073545 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:15.081085 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1527) took 1489 ms. Trace:
I20260709 12:17:15.085358 13255 rpcz_store.cc:276] 0709 12:17:13.585717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.585893 (+   176us) service_pool.cc:224] Handling call
0709 12:17:15.053061 (+1467168us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:15.075507 (+ 22446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2473960832}
I20260709 12:17:15.052564 24405 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:17:15.013870 24927 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.013702 24925 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.115761 24719 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.119596 24925 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.062317 24689 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.122946 24689 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:15.125272 12489 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:15.126523 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131269us
I20260709 12:17:15.127878 24740 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:15.128787 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:15.105732 24615 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:15.115562 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:15.130311 12566 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:17:15.138149 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.051120 24898 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.113678 23540 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.129577 12570 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:15.093451 24919 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:15.141875 24821 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.778s	user 0.014s	sys 0.025s
I20260709 12:17:15.142968 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:15.131976 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.158710 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.138060 24828 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.186868 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.144829 24689 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:15.140544 24405 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:15.149988 24869 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:15.191169 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:15.149466 24719 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:15.194911 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.205377 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:15.143512 24329 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:15.213595 23540 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:15.215505 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72784.7us
I20260709 12:17:15.193826 24689 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:15.196811 24928 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:15.209002 24812 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.197989 24405 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:15.229312 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:15.233691 24916 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:15.232429 24898 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:15.228220 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4722) took 1628 ms. Trace:
I20260709 12:17:15.236672 24869 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.234241 24819 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:15.239439 24677 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.221455 24689 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:15.244415 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5052) took 1908 ms. Trace:
I20260709 12:17:15.204221 24719 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.246330 24927 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.193212 24821 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.248901 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.273407 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:15.249022 24405 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.255043 12637 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:17:15.256844 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:15.224303 24919 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.279395 12570 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:15.280052 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:15.257134 12572 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
I20260709 12:17:15.227608 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:15.258157 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.226230 24481 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:15.261466 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:15.261996 24329 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:15.265694 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 137249us
W20260709 12:17:15.265928 24932 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:17:15.198860 24817 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.270074 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.314284 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:15.316123 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.318277 12571 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:15.291970 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.267522 12496 rpcz_store.cc:276] 0709 12:17:13.263752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.264100 (+   348us) service_pool.cc:224] Handling call
0709 12:17:13.352090 (+ 87990us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:13.352897 (+   807us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:13.352933 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:14.919556 (+1566623us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:14.920367 (+   811us) log.cc:844] Serialized 26 byte log entry
0709 12:17:15.073290 (+152923us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:15.073346 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:15.082601 (+  9255us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:15.147077 (+ 64476us) raft_consensus.cc:1685] finished
0709 12:17:15.147108 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:15.171921 (+ 24813us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14465,"raft.queue_time_us":792,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":17139,"spinlock_wait_cycles":6658688,"thread_start_us":152350,"threads_started":1,"wal-append.queue_time_us":209345}
I20260709 12:17:15.299495 24719 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:15.303054 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:15.324429 12572 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:17:15.281976 24931 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.169s	user 0.002s	sys 0.000s
I20260709 12:17:15.332682 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1785) took 3096 ms. Trace:
I20260709 12:17:15.333323 12639 rpcz_store.cc:276] 0709 12:17:12.237785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.238268 (+   483us) service_pool.cc:224] Handling call
0709 12:17:12.238624 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:13.185787 (+947163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:15.331908 (+2146121us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:15.332390 (+   482us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1958910848}
W20260709 12:17:15.334941 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52427us
I20260709 12:17:15.336331 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:15.343115 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:15.300978 24884 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:15.279577 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:15.316210 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5044) took 2108 ms. Trace:
I20260709 12:17:15.317898 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:15.292722 24869 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.296046 24677 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.343537 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:15.359175 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:15.322398 12568 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 35 because replica has last-logged OpId of term: 33 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 23.
I20260709 12:17:15.265007 12643 rpcz_store.cc:276] 0709 12:17:13.595804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.806715 (+1210911us) service_pool.cc:224] Handling call
0709 12:17:14.807589 (+   874us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:14.881310 (+ 73721us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:14.881366 (+    56us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:17:14.914456 (+ 33090us) log.cc:844] Serialized 78 byte log entry
0709 12:17:15.084287 (+169831us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:15.184680 (+100393us) log.cc:844] Serialized 14 byte log entry
0709 12:17:15.223567 (+ 38887us) log.cc:844] Serialized 14 byte log entry
0709 12:17:15.223776 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:15.223856 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:15.223893 (+    37us) raft_consensus.cc:1685] finished
0709 12:17:15.223903 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:15.224047 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60607,"prepare.queue_time_us":145195,"raft.queue_time_us":120622,"raft.run_cpu_time_us":743,"raft.run_wall_time_us":40059,"thread_start_us":239055,"threads_started":3,"wal-append.queue_time_us":209279}
I20260709 12:17:15.319989 12569 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
W20260709 12:17:15.354111 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92840.3us
I20260709 12:17:15.325985 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:15.329665 24329 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:15.330770 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:15.358479 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143352us
I20260709 12:17:15.346438 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:15.369510 24937 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:17:15.369927 24898 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.370493 24745 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:15.371977 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:15.352939 12490 rpcz_store.cc:276] 0709 12:17:13.207627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.236151 (+ 28524us) service_pool.cc:224] Handling call
0709 12:17:13.426917 (+190766us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:13.494291 (+ 67374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:15.315479 (+1821188us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540012c5d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:15.315929 (+   450us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437734144}
I20260709 12:17:15.358574 24405 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:15.449839 24923 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.673s	user 0.012s	sys 0.004s
W20260709 12:17:15.450202 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78484.9us
I20260709 12:17:15.450762 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.451665 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:15.380800 12639 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:15.453964 24869 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:15.373250 24916 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.457329 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:15.459349 24677 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:15.461114 24916 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.392109 24884 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.393553 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:15.398047 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:15.399235 24947 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:15.389453 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:17:15.342023 24481 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.338037 24927 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:15.451800 24878 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:15.493588 23540 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:15.457355 12639 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:17:15.460747 24677 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.386278 24919 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:15.382361 24935 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:17:15.489511 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 91558.6us
I20260709 12:17:15.488001 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:17:15.505414 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.718s (SENT)
I20260709 12:17:15.496483 24878 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:15.508242 24878 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:15.497056 24740 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:15.497994 24884 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.461967 24898 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:15.519743 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:15.476140 12637 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:15.514536 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:15.527683 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 344899us
I20260709 12:17:15.337476 24929 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:15.488909 24927 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:15.530977 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:15.511252 24329 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:17:15.464373 24745 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:15.559507 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:15.559921 13256 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:17:15.562639 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.547648 24615 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.552181 24869 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:15.588001 24615 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.588673 24869 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:15.589149 24869 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:15.591209 24615 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.598517 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:15.600737 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 283142us
I20260709 12:17:15.583696 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2362) took 3654 ms. Trace:
I20260709 12:17:15.602696 12567 rpcz_store.cc:276] 0709 12:17:11.883435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:12.095340 (+211905us) service_pool.cc:224] Handling call
0709 12:17:12.124361 (+ 29021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:12.204796 (+ 80435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:12.204844 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:12.204905 (+    61us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:12.204967 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:15.417598 (+3212631us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:15.447714 (+ 30116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:15.537351 (+ 89637us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184295808}
I20260709 12:17:15.401515 24826 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:15.555909 24927 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:15.539826 24929 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:15.583690 24789 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:15.553145 24740 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:15.623793 24740 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:15.585448 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:15.586369 24817 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:15.563503 24329 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:15.587340 13259 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:15.609464 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:15.568825 24745 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:15.615152 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 324971us
I20260709 12:17:15.619532 12572 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
W20260709 12:17:17.960537 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2376) took 2703 ms (client timeout 1849 ms). Trace:
W20260709 12:17:17.961064 12572 rpcz_store.cc:269] 0709 12:17:15.257258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.302583 (+ 45325us) service_pool.cc:224] Handling call
0709 12:17:17.960385 (+2657802us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:15.584223 12490 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 35 because replica has last-logged OpId of term: 33 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 23.
I20260709 12:17:15.704885 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:17.966753 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
W20260709 12:17:17.967389 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4741) took 2316 ms (client timeout 1714 ms). Trace:
W20260709 12:17:17.967706 12638 rpcz_store.cc:269] 0709 12:17:15.650908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.693923 (+ 43015us) service_pool.cc:224] Handling call
0709 12:17:17.967276 (+2273353us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:17.969129 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4742) took 2318 ms. Trace:
I20260709 12:17:17.969419 12638 rpcz_store.cc:276] 0709 12:17:15.651224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:17.968387 (+2317163us) service_pool.cc:224] Handling call
0709 12:17:17.968672 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:17.968887 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:17.968902 (+    15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:17.968931 (+    29us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:17.968957 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:17.968993 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:17.969071 (+    78us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:15.676122 13255 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:15.825868 24906 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.866s	user 0.016s	sys 0.003s
I20260709 12:17:15.654018 24821 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:15.615849 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.79191e+06us
I20260709 12:17:15.666692 24817 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:15.621963 24789 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.188599 24789 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:18.189258 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.190044 12568 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:18.191479 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1527) took 4218 ms. Trace:
I20260709 12:17:18.193390 12572 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 33 index: 19. (term mismatch)
I20260709 12:17:18.193913 12572 pending_rounds.cc:85] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 18
I20260709 12:17:18.194190 12572 pending_rounds.cc:107] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 32.19
I20260709 12:17:18.194490 12572 raft_consensus.cc:2889] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:18.194640 24481 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.646361 23540 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:15.802735 24947 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:15.899112 24929 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.199410 23540 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.199849 24929 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:18.200013 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.200951 12565 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:18.200186 24947 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:15.916240 24959 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:18.203161 24815 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:15.924170 24904 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.111s	user 0.014s	sys 0.009s
I20260709 12:17:15.933772 24927 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:18.206051 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.604s	user 0.021s	sys 0.090s
W20260709 12:17:18.206641 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.605s	user 0.022s	sys 0.090s
I20260709 12:17:18.206796 24927 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:16.029039 24745 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:16.903422 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080034e640 after lost signal to thread 11313
W20260709 12:17:18.209069 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080034e460 after lost signal to thread 12429
W20260709 12:17:17.419552 12611 connection.cc:256] RPC call timeout handler was delayed by -1.1172180759999719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20260709 12:17:18.209297 24745 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:17.435945 24965 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.210428 24965 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:17.436497 12566 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.842s	user 0.006s	sys 0.021s
W20260709 12:17:17.496220 24878 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.987s	user 0.020s	sys 0.009s
I20260709 12:17:18.212216 12566 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:18.212347 24878 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.213038 24965 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:18.213410 24904 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.214084 24904 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.214694 24878 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:17.501255 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:18.215513 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:17.865386 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.25672e+06us
I20260709 12:17:18.216722 24904 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.217235 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:18.218392 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:18.219485 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:17.944926 12563 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.442s	user 0.024s	sys 0.046s
W20260709 12:17:18.226277 12563 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.724s	user 0.025s	sys 0.047s
I20260709 12:17:18.227358 24878 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:17.948614 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 2.34233e+06us
W20260709 12:17:17.965282 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1912) took 2457 ms (client timeout 2242 ms). Trace:
W20260709 12:17:18.229871 12490 rpcz_store.cc:269] 0709 12:17:15.507708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.514092 (+  6384us) service_pool.cc:224] Handling call
0709 12:17:17.965153 (+2451061us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:17.971374 24906 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:17.971375 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1536) took 2400 ms (client timeout 1714 ms). Trace:
I20260709 12:17:18.226212 24927 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:17:18.003947 24869 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 2.414s	user 0.009s	sys 0.021s
I20260709 12:17:18.005860 24632 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 29.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:17:18.030651 12639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.470s	user 0.022s	sys 0.033s
I20260709 12:17:18.033887 24942 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:18.039606 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.350s	user 0.010s	sys 0.013s
W20260709 12:17:18.041826 24689 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 2.762s	user 0.042s	sys 0.015s
I20260709 12:17:18.243643 24689 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.244095 24906 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.045781 24967 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.248929 24967 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.241678 24869 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.251452 24967 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:18.051054 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2378) took 2761 ms. Trace:
I20260709 12:17:18.051672 24838 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.253146 24869 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:18.053666 12537 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.818s (SENT) [suppressed 3 similar messages]
I20260709 12:17:18.054149 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:18.065549 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.481s	user 0.025s	sys 0.054s
I20260709 12:17:18.072647 24953 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:15.881219 24884 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:15.827503 24677 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:18.080084 24740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.353s	user 0.014s	sys 0.024s
W20260709 12:17:18.091010 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.474s	user 0.021s	sys 0.029s
W20260709 12:17:18.107985 24868 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.310s	user 0.010s	sys 0.013s
W20260709 12:17:18.111617 24701 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 3.027s	user 0.027s	sys 0.022s
W20260709 12:17:18.119886 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.452s	user 0.018s	sys 0.014s
W20260709 12:17:18.283885 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.616s	user 0.018s	sys 0.015s
W20260709 12:17:18.284365 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 2.6184e+06us
I20260709 12:17:18.130342 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:18.285506 12643 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:18.286760 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2369) took 3032 ms (client timeout 1554 ms). Trace:
W20260709 12:17:18.287369 12566 rpcz_store.cc:269] 0709 12:17:15.254369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.267606 (+ 13237us) service_pool.cc:224] Handling call
0709 12:17:18.286475 (+3018869us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:18.286634 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":719804800}
W20260709 12:17:18.131404 24819 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 2.853s	user 0.018s	sys 0.018s
I20260709 12:17:18.289149 24819 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:18.290033 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.671s	user 0.021s	sys 0.030s
I20260709 12:17:18.155200 24312 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:18.159890 24822 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:15.616582 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57772.4us
I20260709 12:17:18.045135 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:18.171033 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:18.174222 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4914) took 3958 ms (client timeout 2792 ms). Trace:
I20260709 12:17:15.440042 24781 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.194868 12572 pending_rounds.cc:107] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:17:18.337971 12572 raft_consensus.cc:2889] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:18.338136 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.340022 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.340561 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.342654 12492 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 20. (term mismatch)
I20260709 12:17:18.343128 12492 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:17:18.343369 12492 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:17:18.343664 12492 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:15.640878 24329 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:17:15.853029 24898 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.331116 24813 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.350210 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.207616 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4721) took 4612 ms. Trace:
I20260709 12:17:18.351619 12646 rpcz_store.cc:276] 0709 12:17:13.595453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.723110 (+1127657us) service_pool.cc:224] Handling call
0709 12:17:15.599571 (+876461us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:17:15.599859 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:15.600560 (+   701us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:15.600595 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:15.601320 (+   725us) log.cc:844] Serialized 26 byte log entry
0709 12:17:18.207119 (+2605799us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:18.207165 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.207256 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:18.207304 (+    48us) raft_consensus.cc:1685] finished
0709 12:17:18.207315 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.207554 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1518,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88,"spinlock_wait_cycles":29694720,"thread_start_us":2605254,"threads_started":1,"wal-append.queue_time_us":289220,"wal-append.run_cpu_time_us":2565,"wal-append.run_wall_time_us":1014095}
W20260709 12:17:18.215830 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:18.354601 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:18.355098 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:17.521411 11315 debug-util.cc:398] Leaking SignalData structure 0x7b080000fe40 after lost signal to thread 12536
I20260709 12:17:17.932627 24801 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:18.206091 24964 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.358508 24964 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.359368 24822 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:18.223829 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.550s	user 0.020s	sys 0.020s
I20260709 12:17:18.221133 24821 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.221735 24825 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:17:17.945986 24858 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 2.859s	user 0.029s	sys 0.020s
I20260709 12:17:18.229393 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:18.231261 13255 rpcz_store.cc:269] 0709 12:17:15.571318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.578185 (+  6867us) service_pool.cc:224] Handling call
0709 12:17:17.971255 (+2393070us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:18.401808 24906 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:17.983134 24797 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:18.403036 24797 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:18.241374 12639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.680s	user 0.023s	sys 0.033s
I20260709 12:17:18.397933 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.406629 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4738) took 3147 ms. Trace:
I20260709 12:17:18.242885 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.408573 24968 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:18.242666 24942 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.411666 24405 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.411996 24942 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:18.245095 23540 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:18.412406 24405 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.412619 24968 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:17:18.246994 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.560s	user 0.012s	sys 0.015s
I20260709 12:17:18.247145 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:18.248202 24689 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:18.415382 24405 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:18.253746 24838 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.417770 24838 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:18.053530 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1791) took 4541 ms (client timeout 2863 ms). Trace:
W20260709 12:17:18.418467 12644 rpcz_store.cc:269] 0709 12:17:13.511853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.552504 (+ 40651us) service_pool.cc:224] Handling call
0709 12:17:18.053069 (+4500565us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:18.053289 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295164672}
I20260709 12:17:18.420032 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:18.239481 24904 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.016s
I20260709 12:17:18.254662 12641 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:18.239624 24927 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.252548 12569 rpcz_store.cc:276] 0709 12:17:15.306872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.516626 (+209754us) service_pool.cc:224] Handling call
0709 12:17:15.577400 (+ 60774us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:17.117225 (+1539825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:17.117265 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:17.118152 (+   887us) log.cc:844] Serialized 26 byte log entry
0709 12:17:17.390891 (+272739us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:17.436961 (+ 46070us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.050649 (+613688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.050739 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:18.050788 (+    49us) raft_consensus.cc:1685] finished
0709 12:17:18.050798 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.050983 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1546007,"prepare.queue_time_us":46908,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":2148421,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":315,"thread_start_us":281015,"threads_started":2,"wal-append.queue_time_us":302548}
I20260709 12:17:18.423252 24927 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:18.424226 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:18.424674 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3858) took 4215 ms (client timeout 2121 ms). Trace:
I20260709 12:17:18.425863 24825 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.425983 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.255795 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.231410 24878 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:18.261469 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.677s	user 0.025s	sys 0.055s
W20260709 12:17:18.278888 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 73689.2us
I20260709 12:17:18.427358 24878 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:18.428074 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1914) took 2875 ms. Trace:
I20260709 12:17:18.428140 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:18.292114 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08002c8d40 after lost signal to thread 12521
I20260709 12:17:18.292115 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.430727 24825 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:18.430905 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.290302 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:18.432439 12568 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:17:18.432583 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:18.430913 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08000678c0 after lost signal to thread 12596
W20260709 12:17:18.298931 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 244430us
W20260709 12:17:18.284222 24740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.558s	user 0.015s	sys 0.025s
I20260709 12:17:18.297923 13256 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 26.
I20260709 12:17:18.300710 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.302207 13259 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:18.298414 12571 rpcz_store.cc:269] 0709 12:17:14.215542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.259659 (+ 44117us) service_pool.cc:224] Handling call
0709 12:17:18.173891 (+3914232us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:18.174128 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1384156416}
I20260709 12:17:18.307174 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2377) took 3049 ms. Trace:
I20260709 12:17:18.312090 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:18.318908 24965 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:17:18.269364 24953 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.296816 24701 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.324062 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.303973 24677 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.011s
I20260709 12:17:18.309227 24907 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:17:18.330374 12460 connection.cc:256] RPC call timeout handler was delayed by -3.3974670810000589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
W20260709 12:17:18.450911 12460 connection.cc:256] RPC call timeout handler was delayed by -3.3902670810000473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
I20260709 12:17:18.330157 24684 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.331094 24719 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:18.451450 12460 connection.cc:256] RPC call timeout handler was delayed by -3.2066670809999778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
I20260709 12:17:18.331477 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.332355 24869 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:17:18.333284 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 2.84254e+06us
W20260709 12:17:18.462643 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1768) took 4156 ms (client timeout 2118 ms). Trace:
W20260709 12:17:18.463341 13258 rpcz_store.cc:269] 0709 12:17:14.306446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.308575 (+  2129us) service_pool.cc:224] Handling call
0709 12:17:18.448097 (+4139522us) spinlock_profiling.cc:243] Waited 3.93 s on lock 0x7b54005e1d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:18.462520 (+ 14423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9041613184}
I20260709 12:17:18.348047 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.348913 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.464849 12488 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:18.350679 24813 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.389068 24964 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.360194 24822 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.360549 24481 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.362718 24789 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.366472 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.340791 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:18.370591 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.367046 24898 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.371870 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.372424 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.373935 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.376371 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.378525 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:18.379213 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4712) took 5672 ms. Trace:
I20260709 12:17:18.379571 12643 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:18.381388 24959 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:18.487347 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4740) took 2836 ms (client timeout 2031 ms). Trace:
I20260709 12:17:18.487556 12494 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:18.487957 12643 rpcz_store.cc:269] 0709 12:17:15.650537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:18.129900 (+2479363us) service_pool.cc:224] Handling call
0709 12:17:18.487235 (+357335us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:18.385989 24947 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:18.386513 24817 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:18.386925 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.388451 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.388984 12637 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
W20260709 12:17:18.389710 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:18.390945 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.395977 24821 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:18.398717 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 169400us
W20260709 12:17:18.436918 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.763s	user 0.020s	sys 0.024s
I20260709 12:17:18.407096 12639 rpcz_store.cc:276] 0709 12:17:15.260981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.508696 (+247715us) service_pool.cc:224] Handling call
0709 12:17:15.509450 (+   754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:15.559441 (+ 49991us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:17:15.559862 (+   421us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.406141 (+2846279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:18.406215 (+    74us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:17:18.406290 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.406378 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.406538 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3053,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2566,"raft.queue_time_us":1827,"thread_start_us":2844918,"threads_started":1,"wal-append.queue_time_us":326748,"wal-append.run_cpu_time_us":1436,"wal-append.run_wall_time_us":1030357}
I20260709 12:17:18.437906 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.426975 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:18.426939 12564 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
W20260709 12:17:18.440950 12426 debug-util.cc:398] Leaking SignalData structure 0x7b0800288780 after lost signal to thread 12670
I20260709 12:17:18.440836 12563 rpcz_store.cc:276] 0709 12:17:15.257592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.500910 (+243318us) service_pool.cc:224] Handling call
0709 12:17:15.501433 (+   523us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:15.502184 (+   751us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:17:18.227062 (+2724878us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.306820 (+ 79758us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:18.306877 (+    57us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:18.306935 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.307006 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.307121 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":231375,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":1737,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":2003,"thread_start_us":2799374,"threads_started":2,"wal-append.queue_time_us":106650}
W20260709 12:17:18.444228 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143601us
W20260709 12:17:18.425218 12641 rpcz_store.cc:269] 0709 12:17:14.209017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.716418 (+507401us) service_pool.cc:224] Handling call
0709 12:17:18.215284 (+3498866us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:18.424401 (+209117us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:18.424568 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99966080}
I20260709 12:17:18.428664 12495 rpcz_store.cc:276] 0709 12:17:15.552693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.562242 (+  9549us) service_pool.cc:224] Handling call
0709 12:17:15.581428 (+ 19186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:15.583884 (+  2456us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:17:15.584347 (+   463us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.427637 (+2843290us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:18.427703 (+    66us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:17:18.427772 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.427860 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.428002 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":9490,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"thread_start_us":2842604,"threads_started":1,"wal-append.queue_time_us":335864,"wal-append.run_cpu_time_us":1331,"wal-append.run_wall_time_us":1001355}
I20260709 12:17:18.397202 24858 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:18.453377 12460 connection.cc:256] RPC call timeout handler was delayed by -3.109467081000048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2147s for a response
I20260709 12:17:18.453315 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.447443 24674 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:18.441859 24965 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:18.456947 24953 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:18.459699 24701 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.460420 24719 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.191916 13260 rpcz_store.cc:276] 0709 12:17:13.973282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.975846 (+  2564us) service_pool.cc:224] Handling call
0709 12:17:13.976485 (+   639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:14.334901 (+358416us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:17:15.375036 (+1040135us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b1800370c60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:17:15.375350 (+   314us) log.cc:844] Serialized 14 byte log entry
0709 12:17:15.729749 (+354399us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:15.763098 (+ 33349us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:17:17.415622 (+1652524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:17.420219 (+  4597us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b1800370c60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:18.191013 (+770794us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:18.191064 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.191258 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13349,"prepare.queue_time_us":360214,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":157,"raft.queue_time_us":197480,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"spinlock_wait_cycles":915799552,"thread_start_us":664718,"threads_started":2,"wal-append.queue_time_us":332736,"wal-append.run_cpu_time_us":3220,"wal-append.run_wall_time_us":1003210}
I20260709 12:17:18.477939 12571 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:18.476613 24684 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.462625 24869 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:18.485088 12567 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:18.576817 12639 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:18.545529 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:18.499775 12638 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 39 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 23.
I20260709 12:17:18.541736 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.540196 24789 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:18.540936 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 209609us
I20260709 12:17:18.481974 24481 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:18.550693 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 2.90766e+06us
W20260709 12:17:18.554805 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08003feba0 after lost signal to thread 13286
I20260709 12:17:18.511916 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:18.550705 13253 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
W20260709 12:17:18.587265 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08003bcb80 after lost signal to thread 24484
I20260709 12:17:18.556628 12644 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:18.587682 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08001da9c0 after lost signal to thread 24693
I20260709 12:17:18.572008 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4739) took 2922 ms. Trace:
I20260709 12:17:18.495381 24797 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.574817 12570 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:18.499109 24838 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:17:18.494736 12491 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:18.547549 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:18.577777 12643 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:18.577884 13259 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:18.486943 12640 rpcz_store.cc:276] 0709 12:17:12.696322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:13.182201 (+485879us) service_pool.cc:224] Handling call
0709 12:17:13.182655 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:13.191689 (+  9034us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:17:13.226003 (+ 34314us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.254843 (+5028840us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b18002f1b20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:17:18.255048 (+   205us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:18.255696 (+   648us) log.cc:844] Serialized 26 byte log entry
0709 12:17:18.306084 (+ 50388us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:18.306128 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.309142 (+  3014us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:18.364593 (+ 55451us) raft_consensus.cc:1685] finished
0709 12:17:18.368592 (+  3999us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.368753 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8497,"prepare.queue_time_us":38927,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":63388,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":25182,"spinlock_wait_cycles":48893696,"thread_start_us":399637,"threads_started":3,"wal-append.queue_time_us":1264060,"wal-append.run_cpu_time_us":1475,"wal-append.run_wall_time_us":1001440}
I20260709 12:17:18.581241 24781 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:18.540115 24822 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.582901 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.583225 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.589072 12568 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:18.589663 12645 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:18.591732 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.497663 24898 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:18.599485 24985 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:17:18.601610 24481 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.631256 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:18.602357 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:18.603659 24969 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:18.603670 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:18.647892 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149129us
I20260709 12:17:18.605819 24965 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.497215 24959 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.608355 24684 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:18.672813 13218 connection.cc:256] RPC call timeout handler was delayed by -2.3757582920000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2863s for a response
I20260709 12:17:18.561885 24817 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.576407 24869 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:18.612669 12493 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 31 index: 23. (term mismatch)
I20260709 12:17:18.618036 12496 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:18.634460 12495 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:18.638248 12492 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:18.629483 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:18.638348 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.674152 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:18.639643 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:18.642196 12646 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:18.683456 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54084.2us
W20260709 12:17:18.645756 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43557us
I20260709 12:17:18.605274 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4913) took 4390 ms. Trace:
I20260709 12:17:18.548379 24947 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:18.686326 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48079.2us
W20260709 12:17:18.673337 13218 connection.cc:256] RPC call timeout handler was delayed by -2.1793989580000925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
W20260709 12:17:18.687465 13218 connection.cc:256] RPC call timeout handler was delayed by -2.1793989580000925s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
W20260709 12:17:18.687920 13218 connection.cc:256] RPC call timeout handler was delayed by -2.0533989580001162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260709 12:17:18.685318 12564 rpcz_store.cc:276] 0709 12:17:14.215319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:14.752590 (+537271us) service_pool.cc:224] Handling call
0709 12:17:14.753466 (+   876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:18.547703 (+3794237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.592064 (+ 44361us) spinlock_profiling.cc:243] Waited 3.67 s on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:18.605198 (+ 13134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8446838144}
W20260709 12:17:18.688282 13218 connection.cc:256] RPC call timeout handler was delayed by -2.0533989580001162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
W20260709 12:17:18.688671 13218 connection.cc:256] RPC call timeout handler was delayed by -1.9867989580002359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20260709 12:17:18.682293 12493 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 22
W20260709 12:17:18.689000 13218 connection.cc:256] RPC call timeout handler was delayed by -1.6330989580001187s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2242s for a response
I20260709 12:17:18.689177 12493 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 30.23
W20260709 12:17:18.689369 13218 connection.cc:256] RPC call timeout handler was delayed by -1.4629989580002984s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2431s for a response
I20260709 12:17:18.689489 12493 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:18.608881 12497 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:18.676815 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.684242 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:18.684473 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:18.653906 12644 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 41 index: 24. (term mismatch)
I20260709 12:17:18.650003 24822 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:18.651373 24868 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:18.690960 12644 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 23
I20260709 12:17:18.696072 12644 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 40.24
I20260709 12:17:18.696465 12644 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:18.696103 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:18.700551 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96996us
W20260709 12:17:18.701759 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119598us
I20260709 12:17:18.650851 24789 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:17:18.705188 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135661us
I20260709 12:17:18.711375 24869 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.712702 24869 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:18.713728 24312 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.010s
I20260709 12:17:18.714922 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:18.715585 24884 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:18.716243 24884 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:18.687542 24953 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:18.716077 24745 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:17:18.725206 24898 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.717450 24884 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.692104 24907 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:18.687654 24838 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:17:18.730844 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:18.738514 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:18.717543 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.736581 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.644145 24797 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:17:18.724570 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.750011 12496 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:18.694134 24815 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:18.743722 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:18.698232 24822 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:18.688118 24929 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:17:18.719161 24988 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:17:18.699442 24925 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:18.722129 24969 raft_consensus.cc:2764] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election decision vote started in defunct term 40: won
I20260709 12:17:18.808657 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.720142 24813 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.809518 13259 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:18.753540 24329 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:18.811826 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:18.761708 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44283us
I20260709 12:17:18.812562 12495 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:18.706223 24789 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.765167 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:18.806069 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.761417 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:18.758105 12488 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:18.773203 24942 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:18.766302 24481 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:17:18.807987 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.757164 24815 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:18.802831 24929 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.723699 24719 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:18.811658 12461 connection.cc:256] RPC call timeout handler was delayed by -1.9995962069999678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
I20260709 12:17:18.822276 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.762249 24677 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:17:18.823362 12644 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:18.826143 24813 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.825747 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.831094 12637 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:18.809826 24821 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:18.831646 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:18.806105 24822 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:18.838790 24929 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:18.761986 24898 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:18.751012 24838 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.738744 24927 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.811275 24925 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:18.822889 24789 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:18.837704 24815 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:18.840492 24965 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:18.814633 24878 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.853443 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.766000 24717 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260709 12:17:18.810388 24953 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:17:18.827970 24969 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:18.837230 24942 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:18.809710 24825 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.883617 24825 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:18.884253 24947 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:18.885352 12566 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:18.840669 24677 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:18.852495 13254 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:18.856767 24925 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:18.867774 24969 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:18.906663 24677 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:18.906855 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:18.907819 13254 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:17:18.906833 24969 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:18.910328 24674 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:18.910789 24674 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:18.861186 24684 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:18.913810 24701 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.588326 12642 rpcz_store.cc:276] 0709 12:17:15.650119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.650303 (+   184us) service_pool.cc:224] Handling call
0709 12:17:15.674729 (+ 24426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:15.685904 (+ 11175us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:17:15.686498 (+   594us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.442967 (+2756469us) log.cc:844] Serialized 14 byte log entry
0709 12:17:18.571471 (+128504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:18.571553 (+    82us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:18.571619 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:18.571711 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:18.571898 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29233,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":91,"raft.queue_time_us":5144,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":8112,"spinlock_wait_cycles":23680,"thread_start_us":2883432,"threads_started":3,"wal-append.queue_time_us":381314,"wal-append.run_cpu_time_us":1492,"wal-append.run_wall_time_us":1005938}
I20260709 12:17:18.914613 24701 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.915647 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.839412 24813 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:18.917721 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.919607 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:18.921443 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:18.863713 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:18.856084 24838 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:17:18.866756 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:18.927655 23540 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:18.928424 23540 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:18.929092 12496 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:17:18.866791 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136149us
W20260709 12:17:18.870096 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 155290us
W20260709 12:17:18.871055 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105994us
I20260709 12:17:18.839581 24481 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 39.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:17:18.835049 24745 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.029s
I20260709 12:17:18.875396 24858 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:18.878684 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:18.886257 24817 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:18.886996 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.894845 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.896003 24942 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:18.898108 24329 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:18.901507 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4726) took 3691 ms (client timeout 2128 ms). Trace:
I20260709 12:17:18.864904 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.930982 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.935038 12496 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:17:18.946774 12495 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:17:18.935716 12489 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:18.936058 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:18.936606 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:18.929334 23540 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:18.946826 12645 rpcz_store.cc:269] 0709 12:17:15.210653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:15.216037 (+  5384us) service_pool.cc:224] Handling call
0709 12:17:18.901270 (+3685233us) spinlock_profiling.cc:243] Waited 3.34 s on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:18.901421 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7686687616}
I20260709 12:17:18.951638 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.952600 12643 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:18.960631 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:18.963052 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:18.967947 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:18.973186 12637 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:18.973976 12637 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:18.975764 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:18.978832 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:18.984431 24329 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:17:18.992218 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157800us
I20260709 12:17:18.990275 13254 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 27.
I20260709 12:17:18.946169 24947 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:17:19.006860 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144152us
I20260709 12:17:18.964864 24992 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:18.970578 24819 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:19.016932 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.031689 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:18.998862 24959 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:19.034760 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83979.4us
I20260709 12:17:19.010968 24994 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:19.023047 24819 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:18.986416 24801 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.054s
I20260709 12:17:19.025808 24701 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:19.052740 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.064543 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:19.067780 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 134) took 21903 ms (client timeout 15000 ms). Trace:
I20260709 12:17:19.052003 24819 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:19.042232 24994 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:19.075210 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68718.6us
I20260709 12:17:19.029188 24632 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:17:19.028882 24992 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.070381 24317 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:19.073346 24825 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:17:19.114171 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74349.3us
W20260709 12:17:19.114502 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 198162us
I20260709 12:17:19.089140 24927 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.087594 24994 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:19.109244 24878 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:19.130205 24996 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:17:19.133275 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.135329 24992 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.110507 24329 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:19.135239 24815 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.112743 24317 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:19.137564 24826 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:19.163138 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.140239 25003 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:19.153043 24992 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:19.155480 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:19.156324 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.157047 24317 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:19.157992 24813 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.161073 24999 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:17:19.068248 12394 rpcz_store.cc:269] 0709 12:16:57.014230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:05.817907 (+8803677us) service_pool.cc:224] Handling call
0709 12:17:18.869096 (+13051189us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:17:11.437767 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:17:11.438662 (+   895us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:17:11.438730 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:17:11.438767 (+    37us) tablet.cc:662] Decoding operations
0709 12:17:11.445563 (+  6796us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:17:11.445685 (+   122us) write_op.cc:641] Partition lock acquired for write op
0709 12:17:11.445730 (+    45us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:17:11.591942 (+146212us) tablet.cc:701] Row locks acquired
0709 12:17:11.592013 (+    71us) write_op.cc:260] PREPARE: finished
0709 12:17:11.592281 (+   268us) write_op.cc:270] Start()
0709 12:17:11.592440 (+   159us) write_op.cc:276] Timestamp: P: 1783599431592178 usec, L: 0
0709 12:17:11.592480 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:17:11.593499 (+  1019us) log.cc:844] Serialized 6248 byte log entry
0709 12:17:12.731395 (+1137896us) op_driver.cc:464] REPLICATION: finished
0709 12:17:14.425656 (+1694261us) write_op.cc:301] APPLY: starting
0709 12:17:14.432583 (+  6927us) tablet.cc:1367] starting BulkCheckPresence
0709 12:17:14.458260 (+ 25677us) tablet.cc:1370] finished BulkCheckPresence
0709 12:17:14.685974 (+227714us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:17:15.198340 (+512366us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:17:15.254181 (+ 55841us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:17:15.254305 (+   124us) write_op.cc:312] APPLY: finished
0709 12:17:15.255310 (+  1005us) log.cc:844] Serialized 148 byte log entry
0709 12:17:15.584430 (+329120us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:15.584938 (+   508us) write_op.cc:454] Released schema lock
0709 12:17:15.585035 (+    97us) write_op.cc:341] FINISH: Updating metrics
0709 12:17:15.585872 (+   837us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:15.585909 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":163777,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":1655664,"apply.run_cpu_time_us":206399,"apply.run_wall_time_us":1176541,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70943,"num_ops":16,"prepare.queue_time_us":452316,"prepare.run_cpu_time_us":16416,"prepare.run_wall_time_us":398544,"raft.queue_time_us":369576,"raft.run_cpu_time_us":49694,"raft.run_wall_time_us":1617574,"replication_time_us":1128723,"thread_start_us":2411349,"threads_started":5,"wal-append.queue_time_us":885296,"wal-append.run_cpu_time_us":50201,"wal-append.run_wall_time_us":2845579}]]}
W20260709 12:17:19.174515 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131945us
I20260709 12:17:19.176843 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.180697 13258 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:17:19.183431 24677 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.162593 24826 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.187644 12565 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:17:19.185793 24969 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.191596 24969 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.200533 24815 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:19.206343 24997 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:17:19.207098 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.169234 24868 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:19.208570 24869 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:19.175863 24927 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:19.171729 24992 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:19.218518 24992 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:19.219475 24674 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.220945 24674 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:19.213094 24927 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:19.222322 24927 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:17:19.223757 25006 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:17:19.228014 24822 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.236186 24317 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.212913 24677 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.239032 24317 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.242038 24329 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.204838 24838 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.256235 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.179525 25003 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:19.262758 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 200175us
W20260709 12:17:19.263293 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86574.5us
I20260709 12:17:19.264797 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:19.287760 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.290534 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.295936 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:19.267481 24677 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:19.307195 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:19.312721 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63319.2us
I20260709 12:17:19.284970 24967 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:19.296945 24884 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:19.312093 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 220107us
I20260709 12:17:19.272253 24959 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260709 12:17:19.297529 24813 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:19.309202 24481 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:19.320076 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:19.315630 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 301482us
I20260709 12:17:19.321321 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:19.321813 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 154569us
I20260709 12:17:19.268111 24789 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.324761 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:19.276549 24929 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.327144 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.331092 24745 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:19.331254 24965 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:19.332856 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.333508 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.333731 24745 raft_consensus.cc:2764] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election decision vote started in defunct term 36: won
I20260709 12:17:19.334120 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.334511 12566 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:17:19.335096 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.335453 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.335770 12567 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:19.336575 12570 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:19.329500 24822 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.339124 24789 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.314018 25009 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:19.339993 25009 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.311818 24797 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:17:19.342379 25009 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.342561 24826 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.343320 24929 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.344245 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.344439 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:19.344998 13258 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:19.345885 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.347384 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.289520 24819 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.349922 24819 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.343405 24858 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:19.283950 25001 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:17:19.351483 24838 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.326025 24481 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:19.352871 24481 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:19.336475 24677 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:19.353777 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 207137us
W20260709 12:17:19.346222 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74024.6us
I20260709 12:17:19.358671 24947 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:17:19.359391 24947 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:19.359452 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:19.361725 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:19.353363 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:19.363029 13254 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:19.350359 25003 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.364461 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.357370 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.369098 12497 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:19.374387 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:19.376379 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49455.5us
I20260709 12:17:19.379438 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.380585 12572 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 37 because replica has last-logged OpId of term: 36 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 17.
I20260709 12:17:19.379427 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.381286 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.381742 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:19.381965 12566 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.382510 12566 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:19.382890 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.383479 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.384240 12571 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:19.385417 12566 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.23-36.23]
I20260709 12:17:19.385686 12570 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [38.16-40.17]
I20260709 12:17:19.386255 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.386358 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.386797 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.387142 12571 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:19.387631 12568 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:19.387898 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.388698 12566 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:19.390103 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.390326 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:19.390151 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 273560us
I20260709 12:17:19.390955 12489 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:19.360733 24947 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.319904 24995 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:17:19.346554 25011 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:17:19.368556 25013 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:17:19.311686 24994 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.402732 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.393065 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.404147 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.405009 13260 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:19.367925 24826 raft_consensus.cc:2764] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:17:19.393584 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:19.406858 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.400310 24689 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:19.400002 24701 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.008s
I20260709 12:17:19.410334 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.481341 12569 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:19.410989 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.412529 24481 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:19.418062 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70795.6us
I20260709 12:17:19.389475 24719 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:17:19.420455 12567 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:19.491817 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.346307 24953 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.044s
I20260709 12:17:19.368198 24869 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:19.373447 24821 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:19.514267 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.425506 12571 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:17:19.461606 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:19.466626 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:19.466861 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 73351.5us
I20260709 12:17:19.467216 12637 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:19.518021 12645 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:19.467845 12644 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:19.475450 24995 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:19.527360 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:19.476337 12386 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 28. New cstate: current_term: 28 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:19.482035 12572 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:17:19.486158 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128886us
I20260709 12:17:19.487061 24801 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:19.492574 12567 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:19.489770 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99757.7us
I20260709 12:17:19.481161 24822 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:19.503223 24964 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:19.539836 24817 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.009s
W20260709 12:17:19.508490 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85092.5us
I20260709 12:17:19.508646 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.510030 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.511162 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:19.511737 24927 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.526763 13257 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:19.492787 24925 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.003s
I20260709 12:17:19.512724 25008 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:19.535459 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:19.537448 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.536962 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.492578 25009 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:19.608981 25009 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:19.610183 25009 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.20, Last appended by leader: 20, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.610622 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:19.466246 25012 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:19.611657 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.612257 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.616072 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:19.617295 13262 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:19.545311 24964 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:19.618270 24964 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:19.619400 24964 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.531957 24801 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:19.558813 13254 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:19.622215 12493 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:19.621351 24964 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:19.561102 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.626876 12639 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:17:19.563822 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:19.628399 13253 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:19.564719 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 27 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.629413 13261 raft_consensus.cc:1813] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Handling vote request from an unknown peer 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.630007 13261 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 27: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:19.566320 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.570350 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.631449 13259 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:19.563824 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:19.631937 24797 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:19.632956 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69226.1us
I20260709 12:17:19.575456 12489 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:17:19.580608 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.546964 24822 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:19.530853 12641 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:19.473187 24992 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.562711 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.637253 12497 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:19.577358 12386 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:17:19.638500 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75965.4us
I20260709 12:17:19.637241 24822 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:19.639597 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:19.642211 25009 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:19.643687 24992 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.632962 24329 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:19.644701 24329 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:19.646005 24953 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:19.646876 24953 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:19.650465 24329 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:19.653447 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.508358 24481 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.531627 24689 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:19.662987 12565 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40.
I20260709 12:17:19.512184 24815 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:19.656565 12491 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:19.658752 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.666191 12643 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:19.584659 25008 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:19.639604 12386 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:17:19.655165 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 149514us
I20260709 12:17:19.662096 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.662187 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.655040 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:19.659121 12567 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:19.678676 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141883us
I20260709 12:17:19.679236 12493 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
W20260709 12:17:19.671092 25028 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260709 12:17:19.680276 25022 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.098s	user 0.008s	sys 0.005s
W20260709 12:17:19.686271 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 136903us
I20260709 12:17:19.667191 24821 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:17:19.686578 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.650522 24817 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:17:19.699836 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.700639 12643 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:19.608541 24925 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.707592 12460 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.699204 12386 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 20 to 22, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 22 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:19.684844 25008 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:19.674027 24815 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.741372 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet a30ed0986fa641bd8f88912b4b9ee320 with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:59760
W20260709 12:17:19.742118 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57117.8us
I20260709 12:17:19.802992 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.751909 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:19.752628 24789 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:19.754803 24999 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:17:19.755954 12646 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:19.758231 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:19.760404 24927 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.762316 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:19.767524 25019 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:17:19.768265 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97381.1us
I20260709 12:17:19.770738 12443 placement_policy_util.cc:248] tablet a30ed0986fa641bd8f88912b4b9ee320: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20260709 12:17:19.827898 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.830065 12645 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:19.773881 24689 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 37.15, Last appended by leader: 15, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:19.662868 24801 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.780995 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.776602 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.780498 12492 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:19.778972 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.788661 12569 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:19.791056 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.779525 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.851918 12493 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:19.713541 24717 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:19.774248 24884 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:19.858135 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:19.862107 12643 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:19.863299 12386 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 25, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 25 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:19.805312 12637 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:17:19.807531 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55734.9us
I20260709 12:17:19.809443 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:19.869128 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.711724 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:19.813895 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.815987 13256 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:19.730981 24813 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:17:19.771896 24942 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:17:19.841233 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.842535 12642 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:19.774886 24994 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:19.779408 24925 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.830027 25009 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:17:19.884769 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 173059us
I20260709 12:17:19.932123 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.932837 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.933815 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.934818 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:19.945657 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.946851 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.947741 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.779606 24822 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.955065 12564 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:19.847748 25028 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:17:19.905081 24868 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:19.911475 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:19.904431 24674 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.683s	user 0.008s	sys 0.005s
I20260709 12:17:19.733984 24992 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.124s
I20260709 12:17:19.978130 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.978754 24992 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:17:19.979080 12639 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:19.979167 24992 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:19.980268 24992 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.976933 24801 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:17:19.879354 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70035.2us
I20260709 12:17:19.982388 13217 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40.
I20260709 12:17:19.982753 13217 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20260709 12:17:19.984009 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:19.984575 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:19.859555 25016 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:17:19.985517 12641 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:19.857810 24999 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 39.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
W20260709 12:17:19.953990 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49432us
I20260709 12:17:19.986804 25012 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:19.987474 25012 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:19.857270 25010 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:19.875123 24717 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:20.006775 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.007201 24717 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:20.007869 12565 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:19.969913 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.010422 12567 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:19.949429 24967 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:19.904889 24745 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:19.893003 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:19.847167 24701 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.107s
W20260709 12:17:19.967579 24990 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:17:19.976641 12638 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:19.977056 24815 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:19.988976 24953 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:20.043458 24745 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:20.044090 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:19.985323 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:19.991998 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.044888 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:17:20.044641 24745 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 36.27, Last appended by leader: 27, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.045456 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:19.992130 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:19.993088 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.046658 12568 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:19.988588 25012 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:19.997782 25010 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:20.011234 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.066311 12566 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:19.718556 24995 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.016402 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.020792 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.031490 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.068631 12572 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:20.068778 12571 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:17:20.070123 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.071197 12572 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:20.023293 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:19.997020 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:20.047427 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154578us
I20260709 12:17:20.049616 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:20.052037 25027 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.283s	user 0.002s	sys 0.001s
I20260709 12:17:20.051714 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.073292 12567 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:20.032414 24967 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.052858 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:20.052872 25035 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.307s	user 0.005s	sys 0.000s
I20260709 12:17:20.053489 24815 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:17:19.998287 24868 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:20.073258 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:20.075992 12568 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:20.079015 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.081482 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:20.083655 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:20.084852 24822 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:20.085619 12563 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:19.903069 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:20.085398 25023 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.331s	user 0.003s	sys 0.000s
I20260709 12:17:20.116168 13256 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:20.091050 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.041765 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:20.090960 12386 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 25, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 25 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:20.093760 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.089823 24684 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:20.132285 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.101815 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.084851 24967 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 21, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.079425 24992 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:20.107206 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.059679 24632 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.046s
I20260709 12:17:20.083393 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:20.085542 24868 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:20.128489 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:20.091924 24719 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:17:20.161551 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.041s	sys 0.033s
W20260709 12:17:20.162036 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.041s	sys 0.033s
W20260709 12:17:20.162393 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 583319us
W20260709 12:17:20.152467 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69168.3us
I20260709 12:17:20.152760 12490 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:20.155781 24481 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:20.158811 12386 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:20.090270 24815 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.164700 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:20.171414 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.182180 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.183377 12564 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:20.161761 24719 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:20.186127 24719 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:20.186445 24967 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:20.130880 24701 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:20.190887 25034 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:20.191411 25034 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.192112 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:20.192322 25034 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.194000 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.193859 24317 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:20.130867 24994 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40.
I20260709 12:17:20.197952 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.198751 12570 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:20.199734 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.190645 24701 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:20.201799 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.162503 24684 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.215797 12571 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:20.219676 12568 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:20.219892 12571 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:20.223232 12640 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.029s	sys 0.046s
W20260709 12:17:20.225047 12640 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.029s	sys 0.047s
I20260709 12:17:20.173338 25033 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:20.223170 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.166268 24481 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.229230 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.230713 12565 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
W20260709 12:17:20.236203 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196895us
I20260709 12:17:20.238834 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:20.198004 24789 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:20.242808 12572 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 27: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:20.245822 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:20.243932 13258 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.038s	sys 0.094s
W20260709 12:17:20.247114 13258 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.038s	sys 0.095s
I20260709 12:17:20.247087 12564 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:20.246460 12386 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.211553 24994 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:20.258394 12570 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:20.259673 12567 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:17:20.223130 25034 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:20.269353 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.272511 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.272845 12386 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 26, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 26 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.232839 25033 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:20.268712 12571 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.276252 12572 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:20.279546 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.287393 12489 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:17:20.279212 12569 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 40 index: 21. (term mismatch)
I20260709 12:17:20.292582 12569 pending_rounds.cc:85] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:17:20.292858 12569 pending_rounds.cc:107] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 38.21
I20260709 12:17:20.293154 12569 raft_consensus.cc:2889] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:20.297971 12571 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:20.310025 12386 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 29. New cstate: current_term: 29 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.272099 25003 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:20.271227 24789 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.201392 25010 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:20.275166 24329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.036s	sys 0.072s
I20260709 12:17:20.332888 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.290154 25033 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:20.336895 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:20.254446 25017 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.061s
I20260709 12:17:20.338740 12640 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:20.338403 24329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.036s	sys 0.073s
I20260709 12:17:20.347260 12571 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 23. (term mismatch)
W20260709 12:17:20.404789 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.409890 12494 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
W20260709 12:17:20.409479 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.407941 12386 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 21 to 24, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 24 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.409561 12571 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 22
I20260709 12:17:20.413414 12569 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:20.413827 12571 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 38.23
I20260709 12:17:20.414383 12571 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:20.413992 12569 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:20.349215 25003 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:20.335916 24317 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:20.423214 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:20.419716 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 224204us
I20260709 12:17:20.429840 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.428966 12386 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.420634 12569 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.449312 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:20.450398 13259 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:20.462453 12569 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:20.464278 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:20.421598 25010 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:20.422127 25001 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:20.439647 12386 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 14 to 7, term changed from 23 to 21, leader changed from <none> to 9485b0afaf144396adf854fff857414e (127.11.11.132), VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) added. New cstate: current_term: 21 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:20.494805 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.488528 12386 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 28. New cstate: current_term: 28 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.403527 24684 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.507684 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.464323 25017 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:20.403281 24815 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.484553 12494 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:20.484629 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.511745 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:20.518198 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:20.515415 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.403849 24481 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 40.17, Last appended by leader: 17, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:20.528980 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.532439 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:20.531598 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.530580 12491 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:17:20.487593 25001 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:20.536448 25001 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:20.538605 25043 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
W20260709 12:17:20.541312 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.542501 12637 raft_consensus.cc:1217] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Deduplicated request from leader. Original: 32.17->[35.18-35.18]   Dedup: 35.18->[]
I20260709 12:17:20.543454 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.544497 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.544771 12643 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [34.24-36.25]
I20260709 12:17:20.549885 24317 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:20.551586 24813 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.659s	user 0.013s	sys 0.000s
I20260709 12:17:20.665335 13259 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:20.568706 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:20.570330 25048 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:17:20.665798 24813 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:17:20.571417 25044 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:17:20.668303 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1787) took 1040 ms. Trace:
I20260709 12:17:20.572281 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:20.668702 13258 rpcz_store.cc:276] 0709 12:17:19.627793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:19.650723 (+ 22930us) service_pool.cc:224] Handling call
0709 12:17:19.652596 (+  1873us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:19.654529 (+  1933us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:19.655592 (+  1063us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:19.663842 (+  8250us) log.cc:844] Serialized 26 byte log entry
0709 12:17:20.247894 (+584052us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:20.247935 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:20.247994 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:20.668045 (+420051us) raft_consensus.cc:1685] finished
0709 12:17:20.668068 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:20.668235 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8827,"raft.queue_time_us":6484,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":4723,"thread_start_us":554439,"threads_started":1,"wal-append.queue_time_us":732930}
W20260709 12:17:20.576663 24801 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.595s	user 0.009s	sys 0.000s
I20260709 12:17:20.510401 24789 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.20, Last appended by leader: 20, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.670482 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.670941 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:20.671705 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.672217 12646 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:17:20.672485 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:20.575470 24701 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:20.589962 12491 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:17:20.591759 24481 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:20.595638 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.534782 24329 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:17:20.606001 25008 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.758s	user 0.011s	sys 0.006s
I20260709 12:17:20.607959 24817 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:20.618340 12569 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:17:20.702783 12569 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 17
I20260709 12:17:20.703306 12569 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:17:20.703629 12569 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:20.483234 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.504957 12386 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 16 to 21. New cstate: current_term: 21 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:20.652832 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
W20260709 12:17:20.655007 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 122691us
I20260709 12:17:20.707222 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:20.551381 25003 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.669447 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.706094 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:20.714422 12641 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:20.565263 25051 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
I20260709 12:17:20.658716 24884 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:20.673962 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.584273 25029 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:20.676505 12638 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:20.676954 12637 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: [35.18-35.18]
I20260709 12:17:20.672502 12642 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:20.670168 12645 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:20.519105 24717 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.511s	user 0.013s	sys 0.001s
I20260709 12:17:20.691128 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.701571 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:20.520507 25017 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.717051 12639 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:20.674702 24927 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:20.724985 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52582.4us
I20260709 12:17:20.690478 24925 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:20.711642 24817 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:20.704164 24822 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:20.711871 25008 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.695358 24701 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:20.783164 24701 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:20.739313 24719 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.553s	user 0.013s	sys 0.000s
I20260709 12:17:20.785918 24719 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.788057 24719 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:20.788426 25033 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.718374 24947 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:20.790593 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:20.724267 25001 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:20.792299 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 356784us
I20260709 12:17:20.762737 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:20.701552 24329 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:20.802342 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.802978 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.803527 12641 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 37 because replica has last-logged OpId of term: 36 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 22.
I20260709 12:17:20.769050 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:20.752882 25017 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:20.780231 25008 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:20.711278 24953 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:20.791493 12497 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:20.808943 12496 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:20.751749 13256 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:17:20.791636 12639 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
W20260709 12:17:20.749557 24868 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.548s	user 0.014s	sys 0.000s
I20260709 12:17:20.726867 25029 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:20.802259 24822 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:20.766085 24817 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:20.774745 24689 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:20.790441 24947 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.815819 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.822415 12496 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
W20260709 12:17:20.821321 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57946.8us
W20260709 12:17:20.793936 24992 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.631s	user 0.015s	sys 0.000s
I20260709 12:17:20.802610 24329 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:20.840356 24689 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:20.841109 24689 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:20.843961 24953 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:20.844732 24953 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.834263 25052 tablet_replica.cc:333] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:17:20.750733 24717 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.753664 24927 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:20.821312 24822 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:20.755384 24884 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:20.880872 12643 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:20.886554 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:20.889457 12489 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:20.889663 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:20.907699 12496 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 29: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:20.920425 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96036.3us
I20260709 12:17:20.923771 12572 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:17:20.925451 12493 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
W20260709 12:17:20.925467 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 219811us
W20260709 12:17:20.929086 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81557.7us
I20260709 12:17:20.821583 24868 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:20.916594 25037 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:17:20.932400 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:20.926961 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:20.936031 12569 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:17:20.941169 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:20.949159 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:20.915328 24964 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.009s	sys 0.017s
W20260709 12:17:20.915040 24995 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.006s	sys 0.016s
I20260709 12:17:20.907814 24992 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:20.960698 12490 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:20.947611 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:20.917021 25017 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:20.966084 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 258943us
W20260709 12:17:20.951934 24964 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.009s	sys 0.017s
W20260709 12:17:20.963411 24995 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.007s	sys 0.017s
I20260709 12:17:20.977662 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:20.966617 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:20.964425 24925 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:20.981590 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.997035 12566 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:20.970063 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:20.985973 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:20.998411 12571 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 35
W20260709 12:17:20.999058 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66781.5us
I20260709 12:17:20.988590 12643 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 16.
I20260709 12:17:20.994966 12496 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:21.009752 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72440.1us
W20260709 12:17:21.011528 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.030761 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:21.032114 12490 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 37 because replica has last-logged OpId of term: 36 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 22.
I20260709 12:17:21.039733 12571 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:21.027110 24789 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:21.058710 24701 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:21.063854 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:21.075474 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:21.065399 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:21.077067 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:20.926330 25029 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.002471 25031 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.060s
I20260709 12:17:21.090250 12489 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 16.
I20260709 12:17:20.979801 24992 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.054082 24999 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.095085 13259 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:21.077905 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:21.084758 24868 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.085443 24947 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:21.048918 25033 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:20.990042 25017 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.085897 24953 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:20.982918 24822 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.074359 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:21.129194 12493 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:17:21.107563 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:21.130666 12490 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:21.044553 25069 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
W20260709 12:17:21.136024 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 158461us
W20260709 12:17:21.070174 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 122603us
I20260709 12:17:21.137167 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:21.086068 24789 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:21.127737 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:21.111565 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131652us
I20260709 12:17:21.144882 12563 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 27. Preceding OpId from leader: term: 38 index: 28. (index mismatch)
I20260709 12:17:21.092232 24701 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.106069 24481 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:17:21.212769 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64987.4us
W20260709 12:17:21.213008 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75975.9us
I20260709 12:17:21.215727 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.153190 24925 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:21.222592 12490 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:21.111724 24999 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.225406 24927 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:21.282580 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:21.231212 25076 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
W20260709 12:17:21.231990 25075 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
W20260709 12:17:21.232398 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.304617 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:21.239765 25072 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:17:21.244706 25080 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:21.218492 25081 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:21.250142 24819 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:17:21.261649 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:21.263773 25034 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.984s	user 0.014s	sys 0.000s
I20260709 12:17:21.159088 24717 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.201560 24868 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:21.157413 24815 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.073s
I20260709 12:17:21.329619 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:21.273193 25079 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:17:21.274562 25017 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.201177 25084 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.120s
I20260709 12:17:21.205116 24947 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:21.230378 25073 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:17:21.283878 12497 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:21.278753 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45703.7us
I20260709 12:17:21.344129 12497 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:21.296975 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 285812us
I20260709 12:17:21.301895 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:21.305290 12643 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:21.219072 24317 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.309512 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:21.225535 25028 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:17:21.305323 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:21.350196 12570 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
W20260709 12:17:21.350592 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45382.4us
I20260709 12:17:21.306733 24965 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.157919 25066 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.138s
I20260709 12:17:21.278908 25008 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:17:21.300212 25029 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:21.382357 25095 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:21.253473 24884 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.282493 24481 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:21.377022 24317 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.354620 25090 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:21.386605 24481 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.387470 25085 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.387799 24317 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.388332 25088 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:21.326695 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.390045 12571 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:17:21.143960 25082 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.097s
I20260709 12:17:21.330636 12639 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:21.391196 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:21.331805 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116156us
I20260709 12:17:21.394261 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:21.395169 25001 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.584s	user 0.005s	sys 0.008s
I20260709 12:17:21.395718 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.395923 25001 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:21.396821 25001 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:21.393401 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:21.222309 24632 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.095s
I20260709 12:17:21.308373 24927 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.338229 12491 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 38 because replica has last-logged OpId of term: 37 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 18.
I20260709 12:17:21.310031 24684 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:21.348496 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:21.348789 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.351430 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:21.404848 12643 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:17:21.404992 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53662.6us
I20260709 12:17:21.357815 25003 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:21.323065 25081 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.334410 24868 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.362187 24964 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:21.360608 24925 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.355000 25086 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.079s
I20260709 12:17:21.356446 24789 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.26, Last appended by leader: 26, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:21.379554 25077 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.244s	user 0.000s	sys 0.000s
I20260709 12:17:21.381373 24717 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:21.210366 24701 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.397364 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:21.401165 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55177.9us
I20260709 12:17:21.426852 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:21.407032 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58701.2us
I20260709 12:17:21.431854 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:21.433918 24717 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:21.434974 25009 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:17:21.435103 24717 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:21.397642 25010 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.880s	user 0.014s	sys 0.001s
I20260709 12:17:21.415729 12643 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:60230
I20260709 12:17:21.386423 24884 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.438915 12643 raft_consensus.cc:606] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:21.439332 12643 raft_consensus.cc:614] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:21.439069 24884 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.410481 24317 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:21.441072 24317 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:21.413353 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:21.410511 25085 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.346733 24329 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.401693 24965 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.408326 24927 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.402187 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:21.411412 24817 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.004s
I20260709 12:17:21.463202 12496 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
W20260709 12:17:21.444505 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47233.3us
W20260709 12:17:21.401716 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 263638us
I20260709 12:17:21.435665 25009 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:21.467211 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.468372 12566 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:21.410704 24994 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:21.469820 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.413305 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:17:21.475278 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:21.478511 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.470623 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:21.412660 24964 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:21.484591 13259 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:21.487927 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:21.485785 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:21.415287 25080 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.486207 24817 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:21.497157 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95095.6us
I20260709 12:17:21.497318 12641 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:17:21.498884 12644 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:21.500526 12644 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:21.499061 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.501335 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:21.502338 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:21.503109 12646 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:17:21.499758 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:21.508069 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:21.510903 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.512101 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:21.512812 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.513867 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:21.513149 13254 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:21.515542 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.518450 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:21.519143 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.519868 12644 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:21.520279 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.521003 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.521136 12641 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:21.523667 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:21.444749 25090 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:21.483505 24689 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.602s	user 0.015s	sys 0.000s
I20260709 12:17:21.521966 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:21.441547 24884 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.444175 24999 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.449805 25008 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.525933 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:21.443486 24701 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:21.419850 24684 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.509567 25086 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.518656 25010 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:21.531248 24947 raft_consensus.cc:2764] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election decision vote started in defunct term 40: won
I20260709 12:17:21.533567 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:21.484257 24994 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:21.443526 25081 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:21.541657 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.569890 12643 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:17:21.525064 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:21.486960 24801 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:21.532246 25084 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:21.540585 25087 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:21.557060 25008 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.573114 12645 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 42 index: 21. (index mismatch)
I20260709 12:17:21.585254 12643 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:21.573323 25100 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:21.573748 24994 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:21.571530 13260 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:21.541100 25092 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:21.541451 24819 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:21.573866 25097 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:21.507201 25080 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.530431 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.629896 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:21.564245 24884 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:17:21.589221 12639 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:17:21.557158 25096 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.632788 12646 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
W20260709 12:17:21.593621 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.442081 24317 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.618844 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:21.635998 13254 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:21.619879 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:21.626425 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:21.552452 24689 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.592444 24701 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:21.557495 25093 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.064s
I20260709 12:17:21.591910 25100 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:21.570343 25086 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.595481 25084 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:21.583096 25095 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:21.635129 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:21.593286 25001 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:17:21.583461 24801 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.619608 24868 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.666508 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:21.668442 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:21.669287 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157964us
W20260709 12:17:21.680707 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45706.5us
I20260709 12:17:21.597770 25010 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:17:21.646237 25097 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:21.618582 24719 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.054s
W20260709 12:17:21.682332 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106365us
W20260709 12:17:21.689769 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.650941 24817 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:21.703673 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.592183 24964 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.679582 25093 consensus_queue.cc:1243] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 37.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
W20260709 12:17:21.715517 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.664582 25100 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:21.708371 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:21.709559 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:21.710673 12570 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:21.720047 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.711153 13254 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:21.711458 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.712174 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:21.621464 25009 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.717434 12563 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:17:21.717832 12569 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:17:21.697764 25031 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:21.729383 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.710235 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:21.721256 12567 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:21.704260 25097 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:21.708300 25081 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:21.643453 25096 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.765003 25086 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:21.728339 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:21.792551 12564 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:21.793856 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:21.793229 12564 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:21.729319 25066 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:21.770968 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 256464us
I20260709 12:17:21.798218 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:21.774085 12563 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 16. Preceding OpId from leader: term: 40 index: 16. (term mismatch)
W20260709 12:17:21.778000 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67857.7us
I20260709 12:17:21.813586 12563 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:17:21.814091 12563 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 38.16
I20260709 12:17:21.768573 12568 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:21.795882 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:21.807574 12488 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:21.815544 13254 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:21.811550 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:21.814577 12563 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:21.766832 24965 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:21.819818 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 178500us
I20260709 12:17:21.788018 24964 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:21.814294 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:21.782740 24329 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:21.783775 25031 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:21.790330 25001 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.792400 24947 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:21.830667 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111384us
I20260709 12:17:21.780535 25097 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:21.784032 24927 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.855259 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:21.850414 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.811118 25066 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.812618 25085 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.823092 25086 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:21.859431 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:21.861584 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47379.5us
I20260709 12:17:21.819027 25010 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:21.879122 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.838317 24999 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:21.859174 24684 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.851155 25093 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:21.886507 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:21.889014 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:21.880426 24481 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:21.911990 25096 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.850873 25001 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.897660 24999 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:21.914127 25083 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:21.914777 25091 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:21.915959 25104 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:17:21.901122 25085 raft_consensus.cc:2764] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
W20260709 12:17:21.916961 25108 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:17:21.905787 24689 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:21.918749 25107 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:17:21.919832 25078 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:21.797915 25095 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.899499 25086 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:21.884819 25010 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:21.923411 25029 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.000s
W20260709 12:17:21.928798 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69476.7us
I20260709 12:17:21.885026 24927 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:21.931257 25098 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:21.837193 25103 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.278s
I20260709 12:17:21.863471 24947 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.885154 25092 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.939956 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:21.916734 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:21.943122 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90300.7us
I20260709 12:17:21.947597 12638 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:21.918092 25120 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:21.900269 25121 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:21.929894 24481 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:21.936739 24999 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:21.977128 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109574us
I20260709 12:17:21.938391 24884 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:21.980137 24801 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:21.937109 25083 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:21.908788 25093 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.985775 25098 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:21.996254 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:21.997673 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57949.9us
I20260709 12:17:22.020010 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:21.999261 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:22.000840 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:21.957248 25078 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:22.004395 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.003963 25096 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:22.008757 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:22.009035 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:21.805831 25084 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:22.026839 25071 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.897s	user 0.006s	sys 0.001s
I20260709 12:17:22.027185 25111 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:22.028136 24947 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:22.028486 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033398
I20260709 12:17:22.030512 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:22.034444 25087 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:22.009254 24481 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:21.979722 24927 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:22.038674 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:21.989442 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:22.004962 12638 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:17:21.951154 25091 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.027880 25111 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:21.953689 25031 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.26, Last appended by leader: 26, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:21.939603 24717 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.045s
I20260709 12:17:22.037318 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:22.040378 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.018396 25083 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:22.047312 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.020805 25098 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:22.024960 25086 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.20, Last appended by leader: 20, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.054875 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.041256 24927 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:17:22.072402 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83062.8us
I20260709 12:17:22.073242 12488 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 27. Preceding OpId from leader: term: 38 index: 28. (index mismatch)
I20260709 12:17:22.041061 25085 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:22.004837 25092 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.082232 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:22.083456 25120 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:22.084573 25092 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:22.088080 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:22.094904 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47696.3us
I20260709 12:17:22.095902 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.100378 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:22.101908 25114 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:22.109902 25126 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:17:22.110577 25119 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:17:22.111097 25010 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:22.113943 25111 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.114799 25111 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:22.115577 25111 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:22.117116 25092 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:22.118160 25125 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:22.118875 25125 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:22.111464 25123 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:17:22.059477 24819 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:22.059026 25096 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:22.138885 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.010726 25121 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.048708 24817 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:17:22.121578 25125 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:22.055583 24947 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:17:22.142777 25126 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:17:22.165073 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.165845 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.166990 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:22.167747 25130 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:22.168507 25130 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.171120 25130 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:22.173200 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.172456 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:22.131609 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:22.179857 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48364.1us
I20260709 12:17:22.145113 25114 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:22.145896 24789 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:22.086874 24994 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.186518 25111 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:22.187882 25034 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:22.185571 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.172529 25066 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.191206 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:22.219376 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.221127 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:22.180112 25103 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:17:22.235750 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3822) took 1508 ms (client timeout 1567 ms). Trace:
I20260709 12:17:22.185031 25092 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:22.196080 24819 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:22.239635 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.185144 25001 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:22.240731 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49633.5us
W20260709 12:17:22.236439 12491 rpcz_store.cc:269] 0709 12:17:20.727334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:20.762338 (+ 35004us) service_pool.cc:224] Handling call
0709 12:17:22.235376 (+1473038us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:22.235645 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1237919616}
I20260709 12:17:22.254703 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:22.245999 25134 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:17:22.236752 25085 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:22.268019 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:22.270411 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:22.271394 13254 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:22.244369 25096 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:22.274528 25101 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.356s	user 0.002s	sys 0.000s
I20260709 12:17:22.275980 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1789) took 1563 ms. Trace:
I20260709 12:17:22.276140 25067 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:22.276547 13258 rpcz_store.cc:276] 0709 12:17:20.712170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:20.712325 (+   155us) service_pool.cc:224] Handling call
0709 12:17:20.737081 (+ 24756us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:22.274721 (+1537640us) spinlock_profiling.cc:243] Waited 5.4 ms on lock 0x7b54000d19a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576e858 00007ffbf57f2e9d 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c
0709 12:17:22.274815 (+    94us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:22.274855 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:22.274929 (+    74us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:22.274998 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:22.275640 (+   642us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:22.275686 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:22.275863 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5509,"raft.queue_time_us":5875,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9089,"spinlock_wait_cycles":2883771520}
I20260709 12:17:22.282002 13259 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:22.201964 24789 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.244809 24717 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:22.307602 13259 raft_consensus.cc:614] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:22.308429 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:22.310014 25136 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.071s
I20260709 12:17:22.244072 24819 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:22.319023 25136 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:22.319193 24819 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:22.319535 25136 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:22.230051 25034 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.320662 25136 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.322333 25091 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:22.150187 25078 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.25, Last appended by leader: 25, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.313673 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.266556 25103 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:22.341920 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:22.308162 25100 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.004s
I20260709 12:17:22.268026 24925 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:22.307971 25076 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.002s	sys 0.000s
I20260709 12:17:22.344274 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:22.346340 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.281325 25097 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.314713 25114 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:22.351845 24822 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.121s	user 0.006s	sys 0.014s
I20260709 12:17:22.345096 25085 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:22.371896 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.379135 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.359436 25093 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.364192 25136 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:22.374226 24822 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.143s	user 0.007s	sys 0.014s
I20260709 12:17:22.392833 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.350409 25100 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.383147 24925 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:22.384168 25078 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:22.392637 25096 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:22.404285 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:22.391017 24717 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:22.370643 25114 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:22.423681 25081 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.632s	user 0.014s	sys 0.000s
I20260709 12:17:22.430883 24927 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:22.436866 25138 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:17:22.437415 25083 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.438879 25008 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:22.436744 24999 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.446472 24789 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.449729 25100 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.450702 25097 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:22.452710 25145 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:17:22.463018 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63036.3us
I20260709 12:17:22.594574 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:22.508392 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:22.596181 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:22.541770 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:22.598758 13258 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:22.547809 12488 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:22.528029 25103 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:22.555786 25081 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:22.624873 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:22.539446 25144 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:17:22.536855 25096 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:22.627398 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.574131 24994 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.534808 25093 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:22.635389 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.745780 12568 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:22.527469 24925 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:22.524117 24717 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.646528 25096 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:22.629910 24822 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:22.638413 25121 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:22.641806 25085 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:22.649441 25097 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:17:22.740983 24964 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.890s	user 0.014s	sys 0.000s
I20260709 12:17:22.780232 13254 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:17:22.756384 25096 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:22.793795 25113 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:17:22.762760 25137 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:17:22.797510 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:22.753341 24925 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.800735 12639 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 29.
I20260709 12:17:22.801976 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:22.797271 25113 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:22.803560 12639 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:22.802778 24925 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.763487 25085 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:22.810708 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 183421us
I20260709 12:17:22.812391 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:22.812582 13254 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
W20260709 12:17:22.802877 25106 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.018s	user 0.003s	sys 0.000s
I20260709 12:17:22.763887 24822 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.813735 25085 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:22.847911 12639 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:17:22.857795 12491 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 15. Preceding OpId from leader: term: 40 index: 16. (index mismatch)
I20260709 12:17:22.869228 25113 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.869762 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:22.870492 25113 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:17:22.855393 24481 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.820s	user 0.013s	sys 0.002s
I20260709 12:17:22.876788 13258 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:22.878141 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:22.880555 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:22.863907 24927 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.869131 24822 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:22.883409 25147 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.001s	sys 0.001s
I20260709 12:17:22.876996 25083 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:22.887876 25098 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.800s	user 0.014s	sys 0.000s
I20260709 12:17:22.900908 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:22.902694 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:22.887539 24999 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.906622 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:22.913671 12571 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:17:22.880290 24481 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:22.917479 25156 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:17:22.918255 25154 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260709 12:17:22.905295 24819 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.585s	user 0.013s	sys 0.001s
I20260709 12:17:22.919654 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.925945 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:22.927600 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 95252.7us
I20260709 12:17:22.928704 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.929804 13254 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:17:22.930083 12462 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:22.930485 12462 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:22.932261 24789 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:22.903321 25098 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:22.933683 25080 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:22.935040 12565 raft_consensus.cc:2368] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:22.919458 25059 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:22.929738 12571 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:17:22.943485 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:22.935211 25155 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.061s	user 0.000s	sys 0.002s
I20260709 12:17:22.944631 12571 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:22.924069 24717 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:22.958546 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1819) took 1654 ms. Trace:
I20260709 12:17:22.959087 12642 rpcz_store.cc:276] 0709 12:17:21.304500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.305681 (+  1181us) service_pool.cc:224] Handling call
0709 12:17:21.306101 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:21.429005 (+122904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:21.429038 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:21.429082 (+    44us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:17:21.429122 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:22.958239 (+1529117us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:22.958291 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:22.958471 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":282145152}
I20260709 12:17:22.946503 25059 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:22.965757 12642 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
W20260709 12:17:22.967383 25110 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.184s	user 0.001s	sys 0.010s
I20260709 12:17:22.972718 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:22.965293 25059 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:23.088812 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.806s (SENT) [suppressed 9 similar messages]
I20260709 12:17:22.927212 24819 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:23.097995 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:23.100121 12571 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:23.170046 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:23.116446 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:23.172772 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:22.942847 24789 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:17:23.179077 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:22.944731 25125 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.139s
I20260709 12:17:22.944386 25033 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.139s
I20260709 12:17:23.083458 25090 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:23.192198 12642 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:23.206351 25131 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:23.206703 24953 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.193179 24964 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:23.207441 24953 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.207679 24964 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.206713 25033 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 44.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:17:23.191238 25083 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:23.215456 25083 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.216094 25058 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:23.216615 25058 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:23.222314 25052 raft_consensus.cc:2243] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:17:23.222920 24481 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:23.191092 24994 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.193694 25090 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:23.207173 25131 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.223405 25052 raft_consensus.cc:2272] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:17:23.234766 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:23.223130 25096 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.217470 25058 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.27, Last appended by leader: 27, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:23.230248 25078 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.771s	user 0.003s	sys 0.011s
I20260709 12:17:23.243736 25096 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:23.243762 25131 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:23.245716 24953 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.245875 25083 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.249338 25052 ts_tablet_manager.cc:1916] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:17:23.229521 25098 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:23.232858 25090 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:23.234311 24992 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:23.259557 12568 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 37 index: 25. (term mismatch)
I20260709 12:17:23.261602 12568 pending_rounds.cc:85] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 24
I20260709 12:17:23.261978 12568 pending_rounds.cc:107] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 36.25
I20260709 12:17:23.262262 12568 raft_consensus.cc:2889] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:23.224268 24994 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:23.259146 25064 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.049s	user 0.006s	sys 0.012s
W20260709 12:17:23.269239 12383 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.866s	user 0.000s	sys 0.000s
I20260709 12:17:23.272257 24994 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:23.278029 25153 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.419s	user 0.007s	sys 0.010s
I20260709 12:17:23.280019 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4795) took 1275 ms. Trace:
I20260709 12:17:23.280516 12640 rpcz_store.cc:276] 0709 12:17:22.004405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.050039 (+ 45634us) service_pool.cc:224] Handling call
0709 12:17:22.077668 (+ 27629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:22.085654 (+  7986us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:22.085709 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:22.085777 (+    68us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:22.085827 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.279684 (+1193857us) spinlock_profiling.cc:243] Waited 7.64 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:23.279737 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.279923 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17561472}
I20260709 12:17:23.282832 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4955) took 1283 ms. Trace:
W20260709 12:17:23.283303 25105 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.559s	user 0.019s	sys 0.005s
W20260709 12:17:23.284770 25115 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.367s	user 0.004s	sys 0.013s
I20260709 12:17:23.286873 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5114) took 1880 ms. Trace:
I20260709 12:17:23.287259 12493 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:17:23.288959 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5113) took 1899 ms. Trace:
I20260709 12:17:23.289449 12490 rpcz_store.cc:276] 0709 12:17:21.389319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.392415 (+  3096us) service_pool.cc:224] Handling call
0709 12:17:21.392789 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:21.393040 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:21.393070 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:21.393618 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:17:21.598118 (+204500us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:21.598161 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:21.598231 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:23.282581 (+1684350us) spinlock_profiling.cc:243] Waited 80.9 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:23.286115 (+  3534us) raft_consensus.cc:1685] finished
0709 12:17:23.286148 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.288880 (+  2732us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186046208,"thread_start_us":203962,"threads_started":1,"wal-append.queue_time_us":276157}
I20260709 12:17:23.287315 12497 rpcz_store.cc:276] 0709 12:17:21.406102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.406505 (+   403us) service_pool.cc:224] Handling call
0709 12:17:21.408591 (+  2086us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:21.430018 (+ 21427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:21.430070 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:21.468462 (+ 38392us) log.cc:844] Serialized 26 byte log entry
0709 12:17:21.708769 (+240307us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:21.708817 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:21.708896 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:23.283705 (+1574809us) spinlock_profiling.cc:243] Waited 8.37 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:23.286483 (+  2778us) raft_consensus.cc:1685] finished
0709 12:17:23.286509 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.286779 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8262,"raft.queue_time_us":12954,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"spinlock_wait_cycles":19257728,"thread_start_us":154131,"threads_started":1,"wal-append.queue_time_us":358462}
I20260709 12:17:23.283366 12569 rpcz_store.cc:276] 0709 12:17:21.999531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.999737 (+   206us) service_pool.cc:224] Handling call
0709 12:17:22.000309 (+   572us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:22.062899 (+ 62590us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:17:22.256856 (+193957us) log.cc:844] Serialized 14 byte log entry
0709 12:17:22.353089 (+ 96233us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:22.358674 (+  5585us) log.cc:844] Serialized 26 byte log entry
0709 12:17:22.358872 (+   198us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:22.358911 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.282217 (+923306us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:23.282289 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:23.282357 (+    68us) raft_consensus.cc:1685] finished
0709 12:17:23.282489 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.282705 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":162771,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":3010,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":18303,"spinlock_wait_cycles":142439296,"thread_start_us":188188,"threads_started":2,"wal-append.queue_time_us":589989}
W20260709 12:17:23.269846 25136 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.875s	user 0.012s	sys 0.003s
I20260709 12:17:23.311324 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:23.321982 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.323351 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:23.333947 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.334992 12640 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:23.343019 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:23.346716 25074 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.084s	user 0.016s	sys 0.022s
I20260709 12:17:23.348177 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.349772 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:23.350390 12644 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:17:23.355871 24942 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:23.356427 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:23.356556 24953 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.357053 13261 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:23.357188 24953 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.357829 25125 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:17:23.359799 24953 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:23.360409 24953 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.360311 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:23.360954 24953 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.361289 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:23.361781 12637 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:17:23.361931 25033 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:23.362438 25033 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:23.362838 25033 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:23.366159 13257 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 23
I20260709 12:17:23.366531 13257 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 41.24
I20260709 12:17:23.366772 13257 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:23.367041 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4960) took 1202 ms. Trace:
I20260709 12:17:23.367486 12565 rpcz_store.cc:276] 0709 12:17:22.164372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.164622 (+   250us) service_pool.cc:224] Handling call
0709 12:17:23.366789 (+1202167us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:23.366946 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1768366976}
I20260709 12:17:23.368496 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.368943 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4791) took 1860 ms. Trace:
I20260709 12:17:23.369168 13255 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 30.
I20260709 12:17:23.369385 12643 rpcz_store.cc:276] 0709 12:17:21.508923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.622674 (+113751us) service_pool.cc:224] Handling call
0709 12:17:21.640343 (+ 17669us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:21.656674 (+ 16331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:21.656708 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:21.656762 (+    54us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:21.656812 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.368685 (+1711873us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:23.368737 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.368876 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37072128}
I20260709 12:17:23.362739 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.370219 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:23.371666 25034 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:23.371990 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.372519 25034 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:17:23.371569 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:23.373344 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.359808 24819 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.374747 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.374562 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.375092 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.375092 25125 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:23.376395 25125 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:23.377125 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4789) took 1872 ms. Trace:
I20260709 12:17:23.377290 25125 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.377605 12645 rpcz_store.cc:276] 0709 12:17:21.504728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.586280 (+ 81552us) service_pool.cc:224] Handling call
0709 12:17:23.376370 (+1790090us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:17:23.376562 (+   192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:23.376781 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:23.376810 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:23.376861 (+    51us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:23.376905 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.376962 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.377061 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5648384}
I20260709 12:17:23.364071 12637 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:17:23.381836 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.382877 12496 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:23.374631 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.384658 12640 raft_consensus.cc:2376] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37. Re-sending same reply.
I20260709 12:17:23.386055 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:23.358628 24942 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:23.381240 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:23.390394 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:23.389163 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:23.394116 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:23.404157 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1750364050001281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
I20260709 12:17:23.411733 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.412770 13255 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:23.412879 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.362902 25131 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.420930 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.387650 24964 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.392880 25112 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.400259 25082 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:17:23.400574 25069 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:23.407758 25031 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:23.413375 24819 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:17:23.392460 24992 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.038s
I20260709 12:17:23.446251 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4961) took 1281 ms. Trace:
I20260709 12:17:23.443208 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5109) took 2120 ms. Trace:
I20260709 12:17:23.458613 12489 rpcz_store.cc:276] 0709 12:17:21.322440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.322786 (+   346us) service_pool.cc:224] Handling call
0709 12:17:21.323121 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:21.582269 (+259148us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:21.582323 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:21.582390 (+    67us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:21.582452 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.442782 (+1860330us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:23.442865 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.443106 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595328896}
I20260709 12:17:23.412884 24815 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:17:23.455790 12566 rpcz_store.cc:276] 0709 12:17:22.164773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.168857 (+  4084us) service_pool.cc:224] Handling call
0709 12:17:22.169183 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:22.175988 (+  6805us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:22.176040 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:22.184689 (+  8649us) log.cc:844] Serialized 26 byte log entry
0709 12:17:22.375139 (+190450us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:22.559471 (+184332us) log.cc:844] Serialized 14 byte log entry
0709 12:17:22.559776 (+   305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:22.559864 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:23.276299 (+716435us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:17:23.287283 (+ 10984us) spinlock_profiling.cc:243] Waited 8.72 ms on lock 0x7b18001ba900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:17:23.289095 (+  1812us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b18001ba900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:23.445874 (+156779us) raft_consensus.cc:1685] finished
0709 12:17:23.445913 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.446149 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7397,"prepare.queue_time_us":181418,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":27053,"raft.run_cpu_time_us":20907,"raft.run_wall_time_us":1094813,"spinlock_wait_cycles":126417280,"thread_start_us":180803,"threads_started":1,"wal-append.queue_time_us":385904}
I20260709 12:17:23.463284 13261 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:23.431481 25069 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:23.434276 24942 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:23.474944 24953 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.423846 25081 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:17:23.478157 24942 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:23.432408 25112 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.409427 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.380s	user 0.035s	sys 0.005s Metrics: {"bytes_written":3846,"delete_count":0,"lbm_write_time_us":3380,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":137}
I20260709 12:17:23.488754 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:23.489776 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:23.498296 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:23.430032 25131 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:23.507017 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:23.472520 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57099.4us
I20260709 12:17:23.481995 25069 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:23.461790 25033 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.517148 24953 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:23.538508 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.167210
W20260709 12:17:23.532549 25165 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:17:23.537443 25164 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:17:23.533746 24942 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:23.525605 25085 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.664s	user 0.008s	sys 0.008s
W20260709 12:17:23.552771 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 173773us
I20260709 12:17:23.532727 25112 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:23.524902 25173 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:23.536469 24481 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:23.541376 24953 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:23.563063 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130197us
I20260709 12:17:23.564539 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:23.566447 24953 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:23.567823 24942 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:23.563661 24481 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.521452 25171 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.038s
W20260709 12:17:23.573896 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52064.3us
I20260709 12:17:23.577661 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:23.578259 25120 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:17:23.579300 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143543us
I20260709 12:17:23.559316 25089 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.590032 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.573237 25174 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.557876 25085 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.589557 25089 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.594615 25174 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.605266 25180 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:23.580188 25178 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:23.590476 25173 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:23.591557 24481 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:23.624118 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46566.7us
I20260709 12:17:23.625260 12571 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:23.625547 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:23.628167 12494 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:23.630282 25114 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:23.623623 25169 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.159s	user 0.000s	sys 0.002s
I20260709 12:17:23.597293 25129 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:17:23.636752 25131 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:23.644871 25159 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.408s	user 0.002s	sys 0.000s
W20260709 12:17:23.646860 25161 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.409s	user 0.002s	sys 0.000s
I20260709 12:17:23.620425 25180 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:23.622136 25178 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:23.673034 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4802) took 1273 ms. Trace:
I20260709 12:17:23.688529 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.688063 25031 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:23.692613 25031 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.691176 12639 rpcz_store.cc:276] 0709 12:17:22.399894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.875159 (+475265us) service_pool.cc:224] Handling call
0709 12:17:22.886585 (+ 11426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:22.887047 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:22.887070 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:22.898715 (+ 11645us) log.cc:844] Serialized 26 byte log entry
0709 12:17:23.106013 (+207298us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:23.106076 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.106182 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:23.646107 (+539925us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:23.650503 (+  4396us) raft_consensus.cc:1685] finished
0709 12:17:23.650528 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.672932 (+ 22404us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3492,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":1998,"spinlock_wait_cycles":4633344,"thread_start_us":190729,"threads_started":1,"wal-append.queue_time_us":332053}
W20260709 12:17:23.697284 25177 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:17:23.703737 25180 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.655965 25090 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.707010 25089 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:23.708155 25090 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.709501 25085 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:23.698333 25178 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:23.690142 24964 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:23.700021 25131 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:23.711071 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:23.712787 25174 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:23.730166 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 104720us
I20260709 12:17:23.731346 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:23.714552 25033 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:23.730286 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4788) took 2225 ms. Trace:
I20260709 12:17:23.734624 12646 rpcz_store.cc:276] 0709 12:17:21.504433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.573617 (+ 69184us) service_pool.cc:224] Handling call
0709 12:17:21.573976 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:21.633350 (+ 59374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.729975 (+2096625us) spinlock_profiling.cc:243] Waited 58.4 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:23.730206 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134390656}
I20260709 12:17:23.732420 24964 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:23.738265 25125 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:23.740772 25184 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:23.745576 25184 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.722333 25180 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:23.754318 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1824) took 2167 ms. Trace:
I20260709 12:17:23.754987 12641 rpcz_store.cc:276] 0709 12:17:21.587004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:21.640701 (+ 53697us) service_pool.cc:224] Handling call
0709 12:17:21.641051 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:22.765203 (+1124152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:22.765256 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:22.765299 (+    43us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:17:22.765344 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.754018 (+988674us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:23.754082 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.754234 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2584660864}
I20260709 12:17:23.758229 25173 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.759020 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:23.765745 25181 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:17:23.767524 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:23.768366 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:23.766685 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1928) took 1778 ms. Trace:
I20260709 12:17:23.777642 12493 rpcz_store.cc:276] 0709 12:17:21.983073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.035696 (+ 52623us) service_pool.cc:224] Handling call
0709 12:17:23.286960 (+1251264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:23.288167 (+  1207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.761302 (+473135us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540037da50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:23.761565 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2875886336}
I20260709 12:17:23.771286 25080 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.771235 25033 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.796978 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.797420 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:23.754937 24953 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.811990 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.812088 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.812922 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:23.814944 12646 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:23.811990 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.816108 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:23.792819 25081 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.820137 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:23.798327 25114 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:23.822032 25114 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.765133 25069 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.782838 25080 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.798216 25181 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:23.819955 25081 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:23.818120 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:23.848079 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:50116 (request call id 25) took 1802 ms. Trace:
I20260709 12:17:23.825398 13256 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:23.848722 13259 rpcz_store.cc:276] 0709 12:17:22.030133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.030333 (+   200us) service_pool.cc:224] Handling call
0709 12:17:23.832100 (+1801767us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:23.832260 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":576546560}
W20260709 12:17:23.855018 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:23.854514 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.857091 12640 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:17:23.808847 25184 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:23.857262 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.831881 25182 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.805356 25033 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:23.892526 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4964) took 1581 ms. Trace:
W20260709 12:17:23.895948 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.08353e+06us
I20260709 12:17:23.900552 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:23.893208 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 305377us
I20260709 12:17:23.903882 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:23.904506 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:23.909843 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 1.03167e+06us
I20260709 12:17:23.916620 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:23.917747 13259 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:23.914060 12462 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d3b77689c31445cbba405c2b232495e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:23.886808 24701 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:23.893595 25184 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.937268 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:23.903215 25181 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:23.904129 25131 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.944597 13258 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.25-41.25]
W20260709 12:17:23.928959 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46084.7us
I20260709 12:17:23.951102 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.953487 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.960960 12639 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:23.905085 25114 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:23.965015 12638 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:23.962595 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:23.968003 12643 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:23.975751 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:23.991616 12566 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:17:23.926962 24927 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:23.926222 24822 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:23.998435 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.922008 25136 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:23.921826 25158 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:24.001904 12566 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:17:24.002565 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.004185 12570 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 42 index: 21. (index mismatch)
I20260709 12:17:24.005155 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:23.965095 25115 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:17:24.007153 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.939962 25173 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:17:24.007715 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107449us
I20260709 12:17:24.031030 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:24.010067 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:24.034577 25096 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:17:24.024271 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:24.013964 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.014703 25093 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:24.004397 25114 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:24.037315 12567 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:24.026180 13259 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:17:24.042721 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:24.022938 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.026491 12565 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 40 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 15.
I20260709 12:17:24.054561 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:24.029855 12568 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:17:23.950393 24481 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.081s
I20260709 12:17:23.973213 25052 ts_tablet_manager.cc:1929] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 21.13
I20260709 12:17:24.025525 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:24.068706 12563 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:24.058662 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:23.903188 25182 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.070012 12643 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:24.071127 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:24.071750 12643 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:24.071962 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:24.072973 13258 raft_consensus.cc:606] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:24.073460 13258 raft_consensus.cc:614] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:23.897243 12572 rpcz_store.cc:276] 0709 12:17:22.311371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.311596 (+   225us) service_pool.cc:224] Handling call
0709 12:17:22.312112 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:22.312671 (+   559us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:17:22.350040 (+ 37369us) log.cc:844] Serialized 14 byte log entry
0709 12:17:22.761722 (+411682us) log.cc:844] Serialized 14 byte log entry
0709 12:17:22.761981 (+   259us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:22.772642 (+ 10661us) log.cc:844] Serialized 26 byte log entry
0709 12:17:22.772856 (+   214us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:22.774920 (+  2064us) log.cc:844] Serialized 14 byte log entry
0709 12:17:22.969060 (+194140us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b18001491a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:23.891631 (+922571us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b18001491a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:17:23.891893 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:23.892004 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:23.892193 (+   189us) raft_consensus.cc:1685] finished
0709 12:17:23.892219 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:23.892443 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":188903,"prepare.queue_time_us":84101,"prepare.run_cpu_time_us":160,"prepare.run_wall_time_us":4865,"raft.queue_time_us":1136511,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":23565,"spinlock_wait_cycles":20073216,"thread_start_us":364140,"threads_started":1,"wal-append.queue_time_us":439974}
I20260709 12:17:24.074541 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.066779 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:23.991621 24317 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 12:17:24.022074 24925 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:24.086340 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:23.951186 24701 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.076296 12565 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 30 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:17:24.040892 25086 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:24.091590 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:24.092361 12637 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:24.067164 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:24.092624 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179213us
I20260709 12:17:24.049363 24927 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:24.078857 12566 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:17:23.976989 25184 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:24.106664 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 152091us
I20260709 12:17:24.048336 25093 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:24.109229 25194 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:17:24.090724 24925 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.112535 25033 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:24.113989 12568 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:24.127770 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:24.038358 25186 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.082s
I20260709 12:17:23.980572 25181 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.081822 12646 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:24.103190 25052 log.cc:1199] T a30ed0986fa641bd8f88912b4b9ee320 P 6d4ece942fd5414998215d26edf91001: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/wals/a30ed0986fa641bd8f88912b4b9ee320
I20260709 12:17:23.994441 25131 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.077445 12571 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
W20260709 12:17:24.091375 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 182370us
I20260709 12:17:24.021348 25172 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.059s
I20260709 12:17:24.063417 25114 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.154847 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:24.156069 25080 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:24.156507 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:24.129601 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:24.164831 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.165836 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:24.167070 12644 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:24.022349 24822 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.134857 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:24.082144 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.157198 25184 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:24.135536 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.078418 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:24.026852 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.259718 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:24.260054 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.116671 24953 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:24.093948 12609 leader_election.cc:400] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:24.143040 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.141837 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:24.145027 25203 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:24.151424 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
W20260709 12:17:24.300415 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 158789us
I20260709 12:17:24.300757 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:24.152465 25187 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.293s	user 0.001s	sys 0.001s
I20260709 12:17:24.070828 25088 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:24.081100 25111 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:24.157601 24927 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.160594 25182 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:24.185784 25080 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:24.153427 25191 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
W20260709 12:17:24.137392 25188 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.320s	user 0.002s	sys 0.000s
I20260709 12:17:24.231685 12646 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:24.135493 12639 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:24.262549 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:24.261349 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:24.267678 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 173931us
I20260709 12:17:24.273590 12644 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:17:24.274185 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:24.286803 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.299070 12567 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:24.301733 12563 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:17:24.302831 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 195280us
W20260709 12:17:24.308071 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 229827us
I20260709 12:17:24.320688 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.170397 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:24.223588 24884 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:24.335609 12572 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:17:24.341683 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.038s	sys 0.054s
I20260709 12:17:24.352128 12570 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:17:24.356871 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55139.9us
I20260709 12:17:24.299868 25203 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:24.369962 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.390060 12488 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:24.391409 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:24.370869 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.373819 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.363147 12644 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.25-41.25]
I20260709 12:17:24.394805 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.395218 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:24.395773 12570 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:24.359313 12571 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:24.371189 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.374735 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.381407 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:24.381050 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 378306us
I20260709 12:17:24.397099 13257 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:24.391511 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:24.398694 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.377701 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.399717 12567 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:24.400180 12637 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:24.400467 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:24.401515 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.364795 12565 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 39 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 22.
I20260709 12:17:24.403307 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:24.366870 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.038s	sys 0.054s
I20260709 12:17:24.156121 25114 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:24.404981 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 822603us
I20260709 12:17:24.327384 25198 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:24.406795 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:24.417651 12566 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 38 because replica has last-logged OpId of term: 34 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
I20260709 12:17:24.423981 12565 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:17:24.339815 25173 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:24.407912 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:24.408711 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.403409 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:24.437103 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:24.427384 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.395432 12497 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:24.430317 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.439703 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.440287 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.440539 12567 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:24.435410 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:17:24.442961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51548.6us
I20260709 12:17:24.433405 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:24.450045 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:24.446326 12497 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:24.459754 12497 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 41 index: 21. (index mismatch)
I20260709 12:17:24.464152 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:24.335166 25111 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.351352 25199 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:24.465384 12492 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:24.464233 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:24.501443 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1926) took 2523 ms. Trace:
I20260709 12:17:24.494545 12570 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 30 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:17:24.501560 12567 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:24.342252 25174 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.010s	sys 0.007s
I20260709 12:17:24.514016 12497 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:24.502007 12494 rpcz_store.cc:276] 0709 12:17:21.978242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.036817 (+ 58575us) service_pool.cc:224] Handling call
0709 12:17:22.060179 (+ 23362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:23.628852 (+1568673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:24.501177 (+872325us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:24.501380 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3605404928}
I20260709 12:17:24.493710 12571 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:24.512064 25103 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:24.502950 25198 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:24.448657 25172 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:24.439594 25136 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:24.386219 24884 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.460896 25158 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:24.345737 25080 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.500725 25199 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:24.516669 25174 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.011s	sys 0.007s
W20260709 12:17:24.538929 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:24.533367 12492 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:24.517289 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.573731 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.556739 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2406) took 1638 ms. Trace:
I20260709 12:17:24.563944 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.639724 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:24.643573 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.449072 25173 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.636312 12564 rpcz_store.cc:276] 0709 12:17:22.917906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.918558 (+   652us) service_pool.cc:224] Handling call
0709 12:17:22.918907 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:23.920321 (+1001414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:23.920355 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:23.940684 (+ 20329us) log.cc:844] Serialized 26 byte log entry
0709 12:17:24.307515 (+366831us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:24.307568 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:24.307652 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:24.556404 (+248752us) raft_consensus.cc:1685] finished
0709 12:17:24.556439 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:24.556643 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1065496,"raft.queue_time_us":1001288,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":15318,"thread_start_us":291002,"threads_started":1,"wal-append.queue_time_us":494731}
I20260709 12:17:24.538177 25127 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:17:24.527751 25196 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.385s	user 0.001s	sys 0.000s
I20260709 12:17:24.571795 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:24.525790 25211 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:24.528888 25131 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:24.539409 25052 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59760 (request call id 1) took 5033 ms. Trace:
I20260709 12:17:24.567518 25111 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:24.549052 24884 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:24.644918 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:24.547297 25158 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:24.556139 25199 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:24.896935 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:24.541417 25103 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:24.664237 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92609us
I20260709 12:17:24.649351 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:24.647751 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:24.678385 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:25.037345 12642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.070s	sys 0.099s
W20260709 12:17:25.037997 12642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.071s	sys 0.099s
W20260709 12:17:25.039892 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.005s	sys 0.013s
W20260709 12:17:25.047920 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.008s	sys 0.004s
W20260709 12:17:25.048514 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.008s	sys 0.004s
W20260709 12:17:25.049326 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.036s	sys 0.081s
W20260709 12:17:24.908179 25125 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.169s	user 0.012s	sys 0.005s
W20260709 12:17:25.046046 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.005s	sys 0.013s
I20260709 12:17:25.050002 12564 raft_consensus.cc:2368] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
W20260709 12:17:25.049849 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.036s	sys 0.081s
W20260709 12:17:25.055502 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 604541us
I20260709 12:17:24.654278 25098 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:24.656939 25127 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:24.546845 25136 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.645918 25080 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:24.813587 25202 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:24.545819 25198 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:24.884794 25172 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:25.081611 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.028s	sys 0.040s
I20260709 12:17:25.065084 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:25.038337 24822 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:24.912199 25158 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:25.066812 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:25.065752 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 640929us
W20260709 12:17:25.102885 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:25.100562 12569 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.986s	user 0.008s	sys 0.007s
I20260709 12:17:25.103557 12571 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:17:25.118130 12571 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:25.056675 25210 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:25.098160 25201 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.000s	sys 0.013s
W20260709 12:17:25.112746 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.036s	sys 0.040s
W20260709 12:17:25.126883 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 689173us
W20260709 12:17:25.107205 25033 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.013s	sys 0.006s
I20260709 12:17:25.128032 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:25.129768 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63071.8us
I20260709 12:17:25.131019 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:25.128943 12570 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
W20260709 12:17:25.114145 13257 thread.cc:641] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.004s	sys 0.012s
I20260709 12:17:25.119917 12571 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
W20260709 12:17:25.060168 25178 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.350s	user 0.018s	sys 0.003s
I20260709 12:17:25.099996 25127 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:25.123998 12569 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.010s	user 0.008s	sys 0.007s
I20260709 12:17:25.101734 24717 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:25.109043 25196 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:17:24.817302 25211 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:25.138451 13257 thread.cc:608] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.004s	sys 0.012s
I20260709 12:17:25.110466 25202 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:25.132596 25033 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.014s	sys 0.007s
I20260709 12:17:25.154107 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4817) took 1196 ms. Trace:
I20260709 12:17:25.160066 12640 rpcz_store.cc:276] 0709 12:17:23.949471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.949693 (+   222us) service_pool.cc:224] Handling call
0709 12:17:25.144604 (+1194911us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:25.145903 (+  1299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":605408512}
I20260709 12:17:25.162825 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3902) took 1104 ms. Trace:
W20260709 12:17:25.139106 24927 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.002s	sys 0.020s
I20260709 12:17:25.166335 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5141) took 1371 ms. Trace:
I20260709 12:17:25.163322 12639 rpcz_store.cc:276] 0709 12:17:24.058563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.066769 (+  8206us) service_pool.cc:224] Handling call
0709 12:17:25.162547 (+1095778us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:25.162751 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96697088}
I20260709 12:17:25.155867 25121 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:25.170054 25121 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:25.168715 12495 rpcz_store.cc:276] 0709 12:17:23.795059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.795247 (+   188us) service_pool.cc:224] Handling call
0709 12:17:23.795571 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:23.803931 (+  8360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:23.803956 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:23.809838 (+  5882us) log.cc:844] Serialized 26 byte log entry
0709 12:17:23.994155 (+184317us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:17:24.036831 (+ 42676us) log.cc:844] Serialized 14 byte log entry
0709 12:17:24.429901 (+393070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:24.430042 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:25.166042 (+736000us) raft_consensus.cc:1685] finished
0709 12:17:25.166070 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:25.166256 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":158995,"prepare.queue_time_us":43563,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":9202,"raft.queue_time_us":65856,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":7230,"thread_start_us":485944,"threads_started":2,"wal-append.queue_time_us":241081}
I20260709 12:17:25.170513 25121 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:25.120738 25172 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:25.158950 25131 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:17:25.166168 24927 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.002s	sys 0.020s
I20260709 12:17:25.316700 25173 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:25.148171 25178 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.159797 25202 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:25.120103 25210 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:25.230865 25131 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:25.163378 25221 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.331s
I20260709 12:17:25.108328 25098 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:25.126735 25201 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.000s	sys 0.014s
I20260709 12:17:25.136443 25203 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:24.665385 25052 rpcz_store.cc:276] 0709 12:17:19.505988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:19.530562 (+ 24574us) service_pool.cc:224] Handling call
0709 12:17:20.144790 (+614228us) ts_tablet_manager.cc:1151] Deleting tablet a30ed0986fa641bd8f88912b4b9ee320
0709 12:17:20.144896 (+   106us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:17:20.834035 (+689139us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54000f69d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57fbff9 00007ffbfa809482 00007ffbfa83a2d0 00007ffbfa82083d 00007ffbfa8207e9 00007ffbfa820779 00007ffbfa820741 00007ffbfa81f96d 0000000000388634 0000000000388479
0709 12:17:23.736550 (+2902515us) tablet_metadata.cc:665] Metadata flushed
0709 12:17:23.951487 (+214937us) tablet_metadata.cc:665] Metadata flushed
0709 12:17:24.539259 (+587772us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2149159,"spinlock_wait_cycles":1216574336,"tablet-delete.queue_time_us":317481,"thread_start_us":115747,"threads_started":1}
I20260709 12:17:25.483054 25229 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:25.487865 25232 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:25.499814 25131 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:25.567425 25098 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:25.571079 25224 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 12:17:25.529918 25233 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:25.584426 25103 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:25.560920 25226 ts_tablet_manager.cc:933] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:25.626468 25211 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:25.586709 25172 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:25.587402 25178 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:25.674978 25230 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:17:25.562620 25229 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.602420 25233 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:25.758376 25229 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:25.760711 25233 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:25.761435 25233 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:25.770929 25233 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:25.773434 25233 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:25.786232 25210 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:25.798364 25232 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:25.156420 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:25.799682 25121 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.006s
W20260709 12:17:25.918183 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.019s	sys 0.023s
W20260709 12:17:25.918099 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.007s	sys 0.020s
W20260709 12:17:25.873440 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.004s	sys 0.000s
W20260709 12:17:25.894030 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.001s	sys 0.013s
W20260709 12:17:25.902568 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.000s	sys 0.008s
W20260709 12:17:25.902590 12571 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.002s	sys 0.010s
W20260709 12:17:25.901795 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.002s	sys 0.010s
I20260709 12:17:25.971948 25103 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:25.973889 25085 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:25.975333 25085 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:25.976401 25098 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:25.977689 25098 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.977954 25241 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:25.927973 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4822) took 1972 ms. Trace:
W20260709 12:17:25.979457 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.001s	sys 0.013s
I20260709 12:17:25.979315 25241 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:25.918696 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:25.982077 25033 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:25.979038 12645 rpcz_store.cc:276] 0709 12:17:23.955961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.960837 (+  4876us) service_pool.cc:224] Handling call
0709 12:17:23.961353 (+   516us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:24.116025 (+154672us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:24.116060 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:24.116134 (+    74us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:24.116191 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:25.927231 (+1811040us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:25.927346 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:25.927700 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":354733952}
I20260709 12:17:25.158288 24717 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.985962 25229 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:25.986202 25226 tablet_copy_client.cc:323] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:17:25.987478 25229 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.988765 24717 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:25.973945 25103 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.990096 25178 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:17:25.991087 25178 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:25.990784 25202 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:25.972546 25236 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:25.983035 25033 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:25.997325 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.002s	sys 0.011s
I20260709 12:17:25.997418 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1803) took 2602 ms. Trace:
I20260709 12:17:25.982128 24927 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:25.998829 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.001s	sys 0.009s
W20260709 12:17:25.999235 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.005s	sys 0.000s
I20260709 12:17:25.998399 13256 rpcz_store.cc:276] 0709 12:17:23.394400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.394535 (+   135us) service_pool.cc:224] Handling call
0709 12:17:23.418021 (+ 23486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:23.850423 (+432402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:25.996432 (+2146009us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:25.997004 (+   572us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":936352512}
W20260709 12:17:25.999770 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 881483us
W20260709 12:17:25.879281 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 722710us
W20260709 12:17:26.001812 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.003s	sys 0.010s
W20260709 12:17:25.904359 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5198) took 1819 ms (client timeout 1877 ms). Trace:
W20260709 12:17:25.986054 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.065s	sys 0.133s
W20260709 12:17:25.840554 25199 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.794s	user 0.014s	sys 0.004s
I20260709 12:17:25.996783 25249 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:26.003648 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.020s	sys 0.024s
W20260709 12:17:26.008203 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.007s	sys 0.021s
I20260709 12:17:25.917021 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:25.973976 25221 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:26.017409 25221 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.022027 25249 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.023661 25033 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.017251 12495 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:26.025260 25199 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.025548 25249 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:26.014379 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.066s	sys 0.133s
I20260709 12:17:26.028874 25249 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.033969 25249 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:26.004130 12568 rpcz_store.cc:269] 0709 12:17:24.084393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.086016 (+  1623us) service_pool.cc:224] Handling call
0709 12:17:25.903573 (+1817557us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b54003e0b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:25.904058 (+   485us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61503360}
I20260709 12:17:26.038645 25249 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.047164 25249 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.047561 25229 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:25.998983 24927 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.047909 25249 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:26.051728 12612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.000s	sys 0.010s
W20260709 12:17:26.052225 12612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.000s	sys 0.011s
I20260709 12:17:26.053567 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5203) took 1895 ms. Trace:
I20260709 12:17:26.054232 12566 rpcz_store.cc:276] 0709 12:17:24.157976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.418689 (+260713us) service_pool.cc:224] Handling call
0709 12:17:24.419155 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:24.524295 (+105140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:24.524347 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:24.524414 (+    67us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:24.524476 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:26.031722 (+1507246us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:26.039942 (+  8220us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:26.053331 (+ 13389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241049472}
I20260709 12:17:26.056234 24701 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:26.057057 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 940998us
I20260709 12:17:26.051226 25249 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.059185 25093 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.061396 25236 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.065062 25236 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.066251 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4983) took 1545 ms. Trace:
I20260709 12:17:26.066619 12570 rpcz_store.cc:276] 0709 12:17:24.521088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:25.134006 (+612918us) service_pool.cc:224] Handling call
0709 12:17:25.134415 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:25.134693 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:17:25.153968 (+ 19275us) log.cc:844] Serialized 14 byte log entry
0709 12:17:26.028897 (+874929us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:26.029036 (+   139us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:17:26.029188 (+   152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:26.065930 (+ 36742us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:26.066165 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28973,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":47110,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"thread_start_us":855929,"threads_started":2,"wal-append.queue_time_us":442655}
W20260709 12:17:26.080963 25246 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.195s	user 0.003s	sys 0.000s
I20260709 12:17:26.084408 25178 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.086921 25221 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:26.089763 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 957314us
I20260709 12:17:26.094214 12495 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:26.089872 25103 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.096230 25202 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.108285 25192 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:26.109462 24789 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:26.109550 25192 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.111817 24789 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:26.112437 25192 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:25.997864 25121 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.116060 25121 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.131945 25136 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:26.133214 25136 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.136583 25136 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.141414 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:26.145519 25236 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:26.137336 25136 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.147436 25136 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.150844 25136 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.113173 25192 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:26.155916 25199 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:26.159355 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.004s	sys 0.008s
I20260709 12:17:26.209857 25236 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:26.034086 25241 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:26.233807 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.004s	sys 0.008s
I20260709 12:17:26.032119 25098 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.051023 24927 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.214843 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.216509 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:26.219517 13261 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:26.221776 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.223206 25085 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:26.251914 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:26.231086 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61872.3us
I20260709 12:17:26.239532 13262 raft_consensus.cc:2410] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 36 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20260709 12:17:26.241648 13256 raft_consensus.cc:2368] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:26.241917 25250 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.242486 25217 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.243031 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.243326 25219 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.243573 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.245139 13257 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:26.245016 25184 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.266314 25184 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.268316 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:26.270027 24819 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.271317 24819 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.276520 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:26.278777 25217 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.283416 25219 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:26.286127 25131 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.702s	user 0.011s	sys 0.016s
I20260709 12:17:26.287029 25131 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.287885 25083 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.290053 25089 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:26.294668 24927 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:26.301275 25202 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:26.301991 24927 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.303589 25083 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.302073 24789 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:26.307387 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81308.7us
I20260709 12:17:26.301936 25233 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:26.309998 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 249299us
I20260709 12:17:26.309494 25250 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.294740 25089 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:26.314990 25089 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:26.318939 24925 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.319742 25251 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.320677 25251 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.332490 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.333880 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:26.335227 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:26.335985 12495 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:26.338332 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:26.339478 12492 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:26.340008 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:26.308646 24927 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.343972 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:26.340065 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.346774 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.384286 25083 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.345160 25242 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:17:26.354803 25095 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.390132 24925 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.392535 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:26.416606 12566 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:26.394505 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.394860 12563 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:26.400606 25217 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:26.417177 12563 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:26.401131 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
W20260709 12:17:26.403004 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59336us
I20260709 12:17:26.404143 25235 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:26.418500 25085 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.405239 24819 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.406939 25184 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.419193 25085 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:26.408893 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 132638us
I20260709 12:17:26.418901 25235 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:26.421022 25235 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:26.421000 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:26.422981 25056 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:26.423563 25056 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:26.424045 25056 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:26.418526 12563 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.407797 25199 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:26.425868 25129 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:26.427824 25129 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:26.428282 25129 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:26.430246 25129 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:26.435149 25129 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:26.432756 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:26.442236 24995 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:26.443033 24995 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:26.445297 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.453918 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.379734 25081 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.459180 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.466986 25095 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.458366 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.470593 25250 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.447062 25131 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.485797 24925 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.486356 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.487248 12491 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:26.481565 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:26.489899 25158 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.361s	user 0.020s	sys 0.013s
W20260709 12:17:26.490856 25255 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.002s	sys 0.000s
I20260709 12:17:26.490587 25158 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.491513 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 4991) took 1945 ms. Trace:
W20260709 12:17:26.496317 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 253099us
I20260709 12:17:26.498241 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.499388 25081 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:26.500557 25081 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:26.503695 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:26.503813 25184 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:26.506106 12565 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:17:26.507047 24925 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.508183 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.444576 24995 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.508703 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.513763 13258 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:26.525604 12566 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
W20260709 12:17:26.502866 13218 connection.cc:256] RPC call timeout handler was delayed by -1.4313799000001382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
W20260709 12:17:26.526685 13218 connection.cc:256] RPC call timeout handler was delayed by -1.270279900000105s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
I20260709 12:17:26.515424 25158 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.516520 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.472323 25199 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:26.492033 12571 rpcz_store.cc:276] 0709 12:17:24.546601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:25.137143 (+590542us) service_pool.cc:224] Handling call
0709 12:17:25.137418 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:25.137679 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:17:26.002809 (+865130us) log.cc:844] Serialized 14 byte log entry
0709 12:17:26.409521 (+406712us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:26.446023 (+ 36502us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:26.458601 (+ 12578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:26.478524 (+ 19923us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:26.491437 (+ 12913us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45681,"prepare.queue_time_us":265152,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":79765,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":217,"thread_start_us":1143777,"threads_started":2,"wal-append.queue_time_us":404635}
W20260709 12:17:26.527190 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1559799000001476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
W20260709 12:17:26.541016 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1001799000000574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
I20260709 12:17:26.522651 25184 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:26.543262 12492 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:26.525967 13258 raft_consensus.cc:606] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:26.545677 13261 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:26.539323 25089 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.549219 25089 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.547041 13260 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:26.551963 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:17:26.543015 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.559053 13257 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:17:26.574445 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.539641 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:26.582515 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50496.4us
I20260709 12:17:26.553352 25235 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:26.591404 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93386.8us
I20260709 12:17:26.592514 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.593683 25089 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:26.594146 25089 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.594559 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4816) took 2652 ms. Trace:
I20260709 12:17:26.594594 25089 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:26.594993 12642 rpcz_store.cc:276] 0709 12:17:23.941692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.941920 (+   228us) service_pool.cc:224] Handling call
0709 12:17:23.942447 (+   527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:24.032085 (+ 89638us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:24.032628 (+   543us) log.cc:844] Serialized 14 byte log entry
0709 12:17:25.038498 (+1005870us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:25.039131 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:17:25.039307 (+   176us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:25.039342 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:25.039425 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:26.358537 (+1319112us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b18004ca920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:26.575527 (+216990us) raft_consensus.cc:1685] finished
0709 12:17:26.580200 (+  4673us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:26.594368 (+ 14168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68688,"prepare.queue_time_us":263,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":244839,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":18744,"spinlock_wait_cycles":96685952,"thread_start_us":1024908,"threads_started":3,"wal-append.queue_time_us":485454}
I20260709 12:17:26.597239 13255 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:17:26.598660 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:26.604209 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:26.606021 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:26.589368 25219 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.573982 25229 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:26.608464 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:26.608698 25229 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.610421 25095 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.611649 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:26.611936 25229 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.557825 25254 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:26.612803 12566 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 27.
I20260709 12:17:26.613000 25254 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.617415 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.578338 25251 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.618297 12564 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:17:26.618892 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 4985) took 2075 ms (client timeout 2689 ms). Trace:
W20260709 12:17:26.587652 25257 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.107s	user 0.002s	sys 0.000s
W20260709 12:17:26.619297 12564 rpcz_store.cc:269] 0709 12:17:24.543755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:25.057681 (+513926us) service_pool.cc:224] Handling call
0709 12:17:26.617029 (+1559348us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:26.618831 (+  1802us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16918784}
I20260709 12:17:26.545739 25184 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.615854 25254 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.623178 25184 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:26.623346 25089 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.625200 25180 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:17:26.621877 25127 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.464s	user 0.017s	sys 0.017s
I20260709 12:17:26.621651 25181 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.625787 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:26.626260 25089 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.628357 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.588218 25199 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:26.628839 25227 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.074s	user 0.010s	sys 0.007s
I20260709 12:17:26.640791 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:26.643887 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88753us
I20260709 12:17:26.638511 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:26.642717 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:26.636397 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46129.1us
I20260709 12:17:26.639655 25180 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:26.636729 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:26.613603 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1565) took 2954 ms. Trace:
I20260709 12:17:26.633963 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.640473 25081 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:26.650157 13254 rpcz_store.cc:276] 0709 12:17:23.659276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.691687 (+ 32411us) service_pool.cc:224] Handling call
0709 12:17:23.692100 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:24.089777 (+397677us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:24.089835 (+    58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:24.089914 (+    79us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:24.089970 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:26.613055 (+2523085us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:26.613144 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:26.613397 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8008,"raft.queue_time_us":2885,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":12342,"spinlock_wait_cycles":908373888}
I20260709 12:17:26.646085 25219 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:26.644322 25186 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:17:26.672721 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:26.668836 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.666301 25201 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
W20260709 12:17:26.674629 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46465.4us
I20260709 12:17:26.675048 12567 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:26.675804 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:26.640537 24815 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:17:26.676765 25189 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:26.675938 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:26.678452 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:26.643064 25127 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.678030 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.680807 12489 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:26.677985 24815 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:26.670560 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:26.631923 25034 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.108s
I20260709 12:17:26.667237 25081 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:26.689216 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.686693 25189 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:26.694890 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.683688 25089 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:26.697088 12490 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:26.688788 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:26.693495 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.687068 24815 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:26.699370 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.701785 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:26.703815 24815 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:26.704156 24819 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:26.705217 25031 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:26.707229 25172 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:17:26.707683 25172 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:26.708108 25172 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:17:26.710390 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.710500 12567 raft_consensus.cc:2410] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 30 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:17:26.711163 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.697811 25186 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:26.714762 25127 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.714987 25186 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.22, Last appended by leader: 22, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.716104 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.716933 25199 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.716892 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:26.718000 25180 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.716137 25059 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.129s
I20260709 12:17:26.719705 25059 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:26.719790 25180 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:26.692524 25081 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:26.726799 12495 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:26.692483 25219 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:26.732079 24815 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.732930 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 4d3b77689c31445cbba405c2b232495e from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:17:26.733423 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:26.733935 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:26.734031 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 4d3b77689c31445cbba405c2b232495e from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-4d3b77689c31445cbba405c2b232495e
I20260709 12:17:26.735157 25158 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:26.736620 24815 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.736994 25229 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:17:26.737450 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100901us
I20260709 12:17:26.739140 24925 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:26.739558 25111 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:26.735397 25253 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:26.739611 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:26.742502 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 89174.6us
I20260709 12:17:26.740276 24925 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.721712 25235 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.720747 25059 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:26.756944 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:26.759785 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:26.760820 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:26.755936 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:26.740415 25086 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.023s
I20260709 12:17:26.751520 25089 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:26.766625 25069 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.767021 24925 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.767657 25241 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.768505 25241 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.760026 25253 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:26.771129 25241 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.770548 25253 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.722045 25056 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.772630 25131 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:26.772678 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61712.7us
I20260709 12:17:26.746385 24819 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:26.778016 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:26.779225 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.780687 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.785450 25190 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.785912 12492 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:17:26.786080 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.781581 25229 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.005s
I20260709 12:17:26.785986 12568 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: [29.22-29.22]
I20260709 12:17:26.789063 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.786396 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.790525 12570 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:26.789947 12567 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:26.792224 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:26.796675 25085 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.797695 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.798385 25056 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.798959 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:26.799208 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5204) took 2640 ms (client timeout 1747 ms). Trace:
I20260709 12:17:26.799846 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 30.
I20260709 12:17:26.801195 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:26.802130 12460 connection.cc:256] RPC call timeout handler was delayed by -1.4739822449998883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
I20260709 12:17:26.802021 25235 raft_consensus.cc:2764] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:17:26.803251 25059 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.804574 25056 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.804960 25059 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:26.805287 25056 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:26.806316 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66936.4us
I20260709 12:17:26.806645 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.805004 12563 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:26.802146 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:17:26.809645 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53771.2us
W20260709 12:17:26.799589 12572 rpcz_store.cc:269] 0709 12:17:24.158239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.259641 (+101402us) service_pool.cc:224] Handling call
0709 12:17:26.798803 (+2539162us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:26.798966 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170838272}
I20260709 12:17:26.821533 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:26.824052 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:26.825145 12566 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:26.828501 12494 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:26.802316 25241 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:26.765249 25089 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.820839 24701 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:26.854262 25212 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:17:26.858281 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77728.2us
I20260709 12:17:26.858999 25172 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.781663 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 3.120s	user 0.101s	sys 0.065s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":24289,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":46408,"dirs.run_cpu_time_us":744,"dirs.run_wall_time_us":6258,"lbm_read_time_us":28429,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":25,"lbm_write_time_us":3368,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":9,"peak_mem_usage":115873,"reinsert_count":0,"thread_start_us":158031,"threads_started":2,"update_count":137}
I20260709 12:17:26.873759 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.778035 24819 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.861510 25190 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:26.865490 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53989.3us
I20260709 12:17:26.859907 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:26.873349 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43841.4us
I20260709 12:17:26.858436 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.766253 25189 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.839090 25131 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:26.880882 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:26.883405 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:26.891955 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56609.5us
I20260709 12:17:26.894805 12565 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:17:26.900447 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.874375 24701 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:26.904420 12570 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:26.895869 12572 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
W20260709 12:17:26.890790 25268 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:17:26.850191 25241 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:26.844081 25181 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.878001 25056 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.883469 25185 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:26.905936 25204 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.776s	user 0.045s	sys 0.046s
I20260709 12:17:26.907426 24701 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:26.903556 25253 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:26.939126 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:26.941116 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61889us
I20260709 12:17:26.944772 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.939227 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:26.946206 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:26.936389 25204 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.812s	user 0.045s	sys 0.047s
I20260709 12:17:26.947562 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.952594 13255 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:17:26.933456 25172 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:26.958065 25253 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.893301 25219 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 40.17, Last appended by leader: 17, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.960844 25181 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:26.961917 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:26.964202 25181 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.966423 25241 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:26.969276 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 122940us
I20260709 12:17:26.970600 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:26.971452 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:26.928906 25083 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.980742 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:26.977581 24994 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:26.979744 25086 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:26.952571 25069 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:26.983741 25069 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:26.984189 25069 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:26.985963 25083 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.986331 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:26.988660 25083 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.989346 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.990422 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.990715 25190 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:26.991156 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.991345 12492 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:26.991752 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.991192 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:26.983294 12645 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:17:26.995479 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:26.997823 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138122us
I20260709 12:17:26.955333 25081 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.999406 24819 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:27.001082 12638 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:27.001540 25081 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:26.999449 12641 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [32.19-32.19]
W20260709 12:17:27.003465 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97747us
W20260709 12:17:26.996472 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43676us
I20260709 12:17:27.005025 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.014447 12565 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 28.
I20260709 12:17:27.015952 12566 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:27.016441 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:27.017050 25235 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:27.018217 25212 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:27.018635 25212 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:27.019059 25212 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:27.019328 25235 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:17:27.019891 25235 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:27.020877 25235 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:26.960953 25253 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:27.037992 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95705.7us
I20260709 12:17:27.040058 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:27.041381 25172 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:27.041990 25172 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:27.043269 25172 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:27.047191 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76805.4us
W20260709 12:17:27.024643 25274 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:17:27.051010 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.051973 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:27.054250 25081 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:27.054937 25081 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:27.055917 25081 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.057377 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.058032 13259 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:27.058511 12641 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
W20260709 12:17:27.059641 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1566) took 2696 ms (client timeout 1881 ms). Trace:
I20260709 12:17:27.059942 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.060964 12637 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:27.049361 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:27.053537 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48622.2us
W20260709 12:17:27.038024 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44050us
I20260709 12:17:27.070598 12563 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 22. (term mismatch)
W20260709 12:17:27.062727 13259 rpcz_store.cc:269] 0709 12:17:24.363614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.364922 (+  1308us) service_pool.cc:224] Handling call
0709 12:17:26.806283 (+2441361us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:27.059409 (+253126us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:27.059532 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":605802752}
I20260709 12:17:27.071216 12563 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:17:27.079536 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.081792 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:27.053596 12507 tablet_copy_source_session.cc:215] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:17:27.093243 25056 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.004s
I20260709 12:17:27.079598 12563 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 35.22
I20260709 12:17:27.096057 12563 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:27.099781 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:27.100517 12639 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:27.101420 25056 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:17:27.104403 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.106331 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:27.108845 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59629.5us
I20260709 12:17:27.109923 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.111503 12644 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 23. (term mismatch)
I20260709 12:17:27.111984 12644 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 22
I20260709 12:17:27.112226 12644 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 34.23
I20260709 12:17:27.112571 12644 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:27.113030 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.114682 24701 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.124207 12609 leader_election.cc:400] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:27.124686 12609 leader_election.cc:403] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:17:27.127013 12637 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:17:27.128288 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:27.128762 25058 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:27.129092 12637 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:27.130753 12643 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:27.131088 12646 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:17:27.132186 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:27.141842 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.142851 12640 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:27.124269 25112 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:27.142608 25078 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:27.132205 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.144879 12639 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:27.145357 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:27.141875 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.132328 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:27.117417 24701 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:27.103462 25086 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:27.156816 12646 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:27.107686 25056 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:27.171979 12645 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:27.178051 25031 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:17:27.203308 12646 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 43. Ops: []
I20260709 12:17:27.200008 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:27.178215 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.147948 25112 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:27.200533 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.178236 12644 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
W20260709 12:17:27.204469 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91546.5us
I20260709 12:17:27.213318 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:27.165320 25033 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:27.225939 24953 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:27.163235 25229 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:27.213102 25112 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:27.241488 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.234274 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:17:27.246873 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44431.7us
I20260709 12:17:26.706527 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet a30ed0986fa641bd8f88912b4b9ee320 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:17:27.249881 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2416) took 3277 ms. Trace:
I20260709 12:17:27.250851 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:27.238718 24701 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:27.216792 25033 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:27.216998 25069 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.260409 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:27.260691 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94901.2us
I20260709 12:17:27.261096 13259 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:27.261859 25069 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:27.179888 25204 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.066s
I20260709 12:17:27.212877 25212 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:27.266525 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 560156us
I20260709 12:17:27.266709 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:27.273679 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:27.275444 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.275746 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.277070 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.277433 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.278131 25056 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.250530 12569 rpcz_store.cc:276] 0709 12:17:23.972450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.973050 (+   600us) service_pool.cc:224] Handling call
0709 12:17:24.014179 (+ 41129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:24.113788 (+ 99609us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:17:25.156745 (+1042957us) log.cc:844] Serialized 14 byte log entry
0709 12:17:26.229718 (+1072973us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:26.229947 (+   229us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:26.230211 (+   264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:27.234507 (+1004296us) spinlock_profiling.cc:243] Waited 99.2 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:27.234577 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:27.249771 (+ 15194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":965640,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":48005,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":261,"spinlock_wait_cycles":228266368,"thread_start_us":2105031,"threads_started":3,"wal-append.queue_time_us":472201,"wal-append.run_cpu_time_us":1316,"wal-append.run_wall_time_us":1001310}
I20260709 12:17:27.279515 25056 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:27.275988 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.266530 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.284595 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:27.289090 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.259434 25033 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:27.292167 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:27.292889 12497 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:27.293462 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:27.326680 25281 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:17:27.329862 12496 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 44 index: 27. (term mismatch)
I20260709 12:17:27.403100 12496 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 26
I20260709 12:17:27.403443 12496 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 43.27
I20260709 12:17:27.403689 12496 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:27.329754 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.330466 13256 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 29. Current term is 30. Ops: []
I20260709 12:17:27.405277 13255 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:27.406178 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:27.406610 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.331158 25204 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:27.333153 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.335712 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
W20260709 12:17:27.339891 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104870us
I20260709 12:17:27.340648 25229 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:27.342064 13259 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:17:27.343055 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:27.344403 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.345124 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.345903 12645 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:27.346347 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.347332 12639 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:17:27.347855 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86093.2us
I20260709 12:17:27.347906 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.352970 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.354211 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.345427 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:27.355329 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:27.358350 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:27.376009 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 91450.7us
I20260709 12:17:27.413980 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.386004 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.386889 25092 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:27.377439 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.390678 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:27.415117 12641 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:27.392907 25098 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:27.371132 25128 raft_consensus.cc:2764] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:17:27.254518 25235 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.029s
I20260709 12:17:27.409178 13256 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:17:27.416173 12638 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 38 because replica has last-logged OpId of term: 36 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 18.
I20260709 12:17:27.410235 13261 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:27.394942 25086 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.007s
I20260709 12:17:27.417397 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.421075 12643 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:27.427470 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:27.429855 12637 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:27.418195 25098 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:27.415617 25092 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:27.435463 25078 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.002s
I20260709 12:17:27.424855 25204 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:27.452296 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 216433us
I20260709 12:17:27.455291 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.457677 25114 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:27.458446 25114 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.460553 25092 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.454358 13254 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 20
I20260709 12:17:27.439975 25081 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.014s
I20260709 12:17:27.463994 13254 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 33.21
I20260709 12:17:27.465881 13254 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:27.472570 13255 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:27.480822 25092 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:27.479749 25086 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:17:27.492129 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:27.498862 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56033.2us
I20260709 12:17:27.500125 12611 leader_election.cc:400] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 30 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 30. Current term is 31.
I20260709 12:17:27.490609 24481 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:27.500968 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:27.502115 12489 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:27.503939 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:27.554266 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 109359us
I20260709 12:17:27.494699 25086 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:27.642635 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:27.645186 12494 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:27.650882 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:27.651978 12493 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:27.654644 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.655483 25033 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.658622 25033 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:27.658242 25086 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.661016 25031 raft_consensus.cc:993] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:17:27.661757 25031 raft_consensus.cc:1081] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
W20260709 12:17:27.662928 25269 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.834s	user 0.005s	sys 0.000s
W20260709 12:17:27.665095 25287 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.340s	user 0.003s	sys 0.000s
I20260709 12:17:27.679108 25098 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.25, Last appended by leader: 25, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.686467 25098 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:27.687359 25229 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
W20260709 12:17:27.691104 25283 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.342s	user 0.002s	sys 0.000s
I20260709 12:17:27.701084 25229 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:27.711989 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:27.694315 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.698279 25113 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:27.718124 25113 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.698453 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:27.700019 25204 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:27.720332 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:27.714776 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.700383 25212 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:27.718365 25229 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.721150 12495 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:27.715342 12609 connection.cc:256] RPC call timeout handler was delayed by -1.2407451919998493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26890000000000003s for a response
I20260709 12:17:27.716926 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:27.719205 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1935) took 4335 ms. Trace:
I20260709 12:17:27.732812 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:27.724200 12496 rpcz_store.cc:276] 0709 12:17:23.384209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:23.384378 (+   169us) service_pool.cc:224] Handling call
0709 12:17:23.384747 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:27.404254 (+4019507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:27.718892 (+314638us) spinlock_profiling.cc:243] Waited 3.94 s on lock 0x7b54001450d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:27.719105 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9071269760}
I20260709 12:17:27.733942 12489 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:27.734032 12493 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:27.738754 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.740698 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:27.741837 12565 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:27.743944 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:27.760725 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:27.766520 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52155.7us
W20260709 12:17:27.771843 25296 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
W20260709 12:17:27.774089 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131835us
I20260709 12:17:27.775396 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.744335 25250 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.034s
I20260709 12:17:27.778077 25241 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:27.779388 25277 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.448s	user 0.002s	sys 0.000s
I20260709 12:17:27.777339 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:27.785787 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 136637us
I20260709 12:17:27.787793 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:27.789238 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50604.1us
I20260709 12:17:27.790974 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.720131 25218 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.049s
I20260709 12:17:27.754585 25092 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:27.758786 25212 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:27.792912 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:27.794816 25189 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:27.787214 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:27.804747 12564 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:27.795599 25229 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:27.799189 25089 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.003s	sys 0.019s
I20260709 12:17:27.800700 25113 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:27.801544 12569 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:27.800101 25212 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:27.811561 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:27.810225 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:27.801642 25250 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:17:27.812511 25089 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.003s	sys 0.019s
I20260709 12:17:27.804006 25241 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:27.806046 25114 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:27.824864 12495 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:17:27.813969 25272 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.021s	sys 0.014s
I20260709 12:17:27.822703 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.828150 12564 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
W20260709 12:17:27.833724 25272 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.021s	sys 0.015s
I20260709 12:17:27.815310 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.835489 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:27.842219 12640 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:17:27.843199 25289 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:27.843654 25289 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:27.843986 25289 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:27.844614 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67465.1us
I20260709 12:17:27.845723 25113 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:27.846125 25113 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:27.846455 25113 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:27.847008 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.847313 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.847968 12564 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:27.848169 12568 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:27.849409 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:27.836823 25112 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.564s	user 0.016s	sys 0.003s
I20260709 12:17:27.850063 12570 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:27.850347 25112 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:27.850793 12637 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:27.851524 12569 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:27.845474 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.853912 25241 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:27.855283 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.855700 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.856146 12564 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:27.856660 12563 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:27.856843 25009 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:27.857360 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:27.857831 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:27.858264 12572 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:27.858328 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.855298 25180 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:27.859627 25180 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:27.859747 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.860306 13257 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:17:27.860172 25180 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:27.857388 25009 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:27.857378 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.861332 25009 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:27.863168 25112 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:27.863592 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:27.814750 25212 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:27.874717 12490 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:27.876302 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:27.876479 25241 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:27.877805 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:27.876864 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:27.878827 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:27.879048 25078 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:27.879813 24884 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:27.880025 25078 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:27.880328 24884 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:27.880640 25078 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:27.880941 24884 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:27.881105 25078 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:27.878839 12568 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:27.882704 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:27.883430 12570 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:27.883666 12569 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:27.880333 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:27.887281 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77179.2us
I20260709 12:17:27.886439 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.889135 13256 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:17:27.890187 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:27.886976 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.852275 25198 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:27.897488 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:27.900236 25114 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:27.900703 25086 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:27.907289 25203 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:27.908475 25009 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.910729 25085 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:27.911804 25069 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:27.912074 25009 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:27.912415 25069 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:27.913818 25066 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.017s
I20260709 12:17:27.915236 25066 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:27.915607 25066 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:27.916597 25066 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.924079 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:27.926280 25058 raft_consensus.cc:2764] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:17:27.930356 24701 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:27.930900 24701 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:27.931933 24701 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:27.934576 25086 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:27.889937 25241 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:27.940114 25241 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:27.940682 25241 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:27.970279 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:27.976881 25066 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:27.945066 25098 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:17:27.983566 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 202220us
W20260709 12:17:27.980173 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53517.1us
I20260709 12:17:27.985961 12564 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 23. (index mismatch)
I20260709 12:17:27.991400 12566 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 25. Preceding OpId from leader: term: 42 index: 25. (term mismatch)
I20260709 12:17:27.972724 24701 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:27.994449 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97891.7us
I20260709 12:17:27.991974 12566 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 24
I20260709 12:17:27.999924 12566 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.25
I20260709 12:17:28.000293 12566 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:28.002143 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:28.008735 12496 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 17. Preceding OpId from leader: term: 42 index: 18. (index mismatch)
I20260709 12:17:28.018514 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:27.991504 12563 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50142
I20260709 12:17:28.029038 12563 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:28.029528 12563 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:28.031906 13261 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:28.034176 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:28.040586 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:28.045151 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:28.043366 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.048395 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:28.050707 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:28.060488 12567 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:27.966220 25069 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.036909 25092 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:28.036125 24994 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:17:28.078621 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64707.8us
I20260709 12:17:28.081003 12570 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
W20260709 12:17:28.076386 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 189463us
I20260709 12:17:28.086683 12563 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.092347 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.036463 25112 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:17:28.062845 25228 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:28.108556 12563 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:28.111274 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:28.079094 25092 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:28.090385 25009 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:17:28.114774 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.116926 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:28.112776 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.120289 12496 raft_consensus.cc:2368] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:17:28.122051 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.119004 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.115535 25229 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.007s
I20260709 12:17:28.122890 12645 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 31: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:28.123178 25229 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:28.123621 25229 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:28.124248 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.124672 25180 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.125077 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.125523 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.124599 25229 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 41.21, Last appended by leader: 21, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.127351 25078 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.115798 24481 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
W20260709 12:17:28.128630 25301 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:17:28.130044 25009 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:28.131672 25113 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.134454 25114 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:28.135924 25081 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:17:28.115940 25228 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.021s
I20260709 12:17:28.115777 25092 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:28.128750 24884 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.139954 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: true dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:28.140219 25198 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:17:28.142637 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.143707 25289 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.151352 13259 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:17:28.156080 25069 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 43
I20260709 12:17:28.094924 25272 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.063s
I20260709 12:17:28.167035 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.207180 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.208021 12638 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:28.167680 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.169123 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.173241 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.173424 12637 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:28.178298 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59605.1us
I20260709 12:17:28.211114 12640 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
W20260709 12:17:28.180455 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142393us
W20260709 12:17:28.182204 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.185521 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:28.191488 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.191834 12492 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.21-41.21]
I20260709 12:17:28.192193 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.193477 13255 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
W20260709 12:17:28.196956 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122138us
I20260709 12:17:28.206168 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.206243 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:28.217375 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.120913 25128 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.078s
I20260709 12:17:28.218832 12641 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:28.216620 13255 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:28.219049 12490 raft_consensus.cc:2368] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
I20260709 12:17:28.218856 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.177767 25066 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.218953 12564 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:28.221972 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:28.187299 25081 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:28.228585 13257 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: [38.23-38.23]
I20260709 12:17:28.230684 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:28.215909 12460 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.526s (SENT) [suppressed 12 similar messages]
I20260709 12:17:28.187175 25114 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:28.226631 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.236406 12637 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:28.238268 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.236398 12488 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:28.239390 12489 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:28.240747 24994 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:28.245779 12641 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.25-41.25]
I20260709 12:17:28.246867 25310 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:28.247634 25310 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:28.247309 25114 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.248993 25114 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:28.223115 25305 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.254850 12572 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:28.254369 25305 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.256457 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:28.257180 12572 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:28.260381 24884 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.261732 12643 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:17:28.262477 12488 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 30 index: 23. (index mismatch)
I20260709 12:17:28.232170 13255 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.248122 25310 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:28.221817 25229 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:28.299392 12567 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:28.227169 25098 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.254559 25226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d3b77689c31445cbba405c2b232495e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:17:28.311218 12564 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:28.323056 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:28.324448 12492 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 43. Ops: []
W20260709 12:17:28.329295 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113437us
I20260709 12:17:28.299548 25305 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:28.324978 12567 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39. Re-sending same reply.
I20260709 12:17:28.339227 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:28.347298 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 247169us
I20260709 12:17:28.347410 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:28.352058 12567 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 40 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 22.
W20260709 12:17:28.353717 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117081us
I20260709 12:17:28.243170 25066 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:17:28.360373 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.371062 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:28.378539 12564 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:28.373020 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:28.382114 12567 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
W20260709 12:17:28.385032 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138953us
I20260709 12:17:28.323843 25241 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.346978 24994 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.339695 25098 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.393186 25098 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.348124 25212 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.334365 25009 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 37.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.410207 12570 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:28.411586 25315 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.413410 25113 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.364780 25250 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:17:28.416859 25241 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.416587 25228 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:28.408546 25078 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.418912 25066 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:28.419886 25311 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.183s	user 0.002s	sys 0.000s
I20260709 12:17:28.428597 12637 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:28.425829 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:28.433656 12568 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.545s	user 0.013s	sys 0.002s
I20260709 12:17:28.446337 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:28.447481 12563 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [31.19-32.20]
W20260709 12:17:28.436105 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46012.8us
I20260709 12:17:28.356134 25081 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.453859 12567 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:28.454473 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56200.1us
I20260709 12:17:28.455389 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:28.456955 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.437281 25180 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.464151 12564 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:17:28.470650 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97722.7us
I20260709 12:17:28.476548 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:28.477721 12564 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:17:28.483731 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:28.484421 12568 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 32
W20260709 12:17:28.473819 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166654us
I20260709 12:17:28.459451 25289 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:28.475741 25203 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.568s	user 0.014s	sys 0.000s
W20260709 12:17:28.501823 25298 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.616s	user 0.008s	sys 0.000s
I20260709 12:17:28.482316 25081 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:17:28.483398 25250 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:28.510303 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:28.504841 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1583) took 1091 ms. Trace:
I20260709 12:17:28.414094 25312 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.093s
I20260709 12:17:28.511543 13254 rpcz_store.cc:276] 0709 12:17:27.413485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:27.429726 (+ 16241us) service_pool.cc:224] Handling call
0709 12:17:27.438086 (+  8360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:27.480336 (+ 42250us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:27.480370 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:27.484280 (+  3910us) log.cc:844] Serialized 26 byte log entry
0709 12:17:27.784577 (+300297us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:27.784611 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:27.784685 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:28.439848 (+655163us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:28.504494 (+ 64646us) raft_consensus.cc:1685] finished
0709 12:17:28.504514 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:28.504748 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6203136,"thread_start_us":142865,"threads_started":1,"wal-append.queue_time_us":328323}
I20260709 12:17:28.520658 12568 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:17:28.521740 25308 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:17:28.510917 25127 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.526238 25127 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.529976 25326 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:28.534091 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:28.540243 25058 raft_consensus.cc:2764] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election decision vote started in defunct term 41: won
W20260709 12:17:28.542486 25316 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:17:28.543358 24701 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.537s	user 0.012s	sys 0.000s
W20260709 12:17:28.543603 25307 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.299s	user 0.003s	sys 0.000s
I20260709 12:17:28.544500 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:28.545738 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:28.546350 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5232) took 1618 ms. Trace:
I20260709 12:17:28.547636 25066 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:28.548161 25066 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:28.548596 25066 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:28.551192 25092 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.546761 12571 rpcz_store.cc:276] 0709 12:17:26.928190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:27.917882 (+989692us) service_pool.cc:224] Handling call
0709 12:17:27.918481 (+   599us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:27.919345 (+   864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:27.919398 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:27.920260 (+   862us) log.cc:844] Serialized 26 byte log entry
0709 12:17:28.095270 (+175010us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:28.095309 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:28.095386 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:28.543343 (+447957us) spinlock_profiling.cc:243] Waited 64.4 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:17:28.546056 (+  2713us) raft_consensus.cc:1685] finished
0709 12:17:28.546079 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:28.546289 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7962,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":99,"spinlock_wait_cycles":148104960,"thread_start_us":174479,"threads_started":1,"wal-append.queue_time_us":274470}
I20260709 12:17:28.498618 25323 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:28.554049 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:28.555114 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43617.4us
I20260709 12:17:28.545835 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:28.536164 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:28.567016 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.486773 25272 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.026s
I20260709 12:17:28.512166 25250 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:28.580796 12492 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:28.576448 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:28.579447 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:28.575368 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:28.575949 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120644us
I20260709 12:17:28.584290 12493 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:28.593688 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.594969 25128 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:17:28.580505 25058 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:28.597904 25232 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:28.599363 25212 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:28.600087 25212 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:28.601071 25202 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:17:28.601294 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:28.602273 25323 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:28.602651 25202 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:28.604528 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.592252 25078 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.606194 25078 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.607062 25318 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:28.568014 25180 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.039s
I20260709 12:17:28.608261 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.537771 25203 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:28.506029 25309 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.069s
I20260709 12:17:28.621263 12564 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
W20260709 12:17:28.583724 25320 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.057s	user 0.000s	sys 0.002s
I20260709 12:17:28.638774 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.641342 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.568249 25324 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:28.644754 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.652324 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.603236 25202 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:28.584884 25250 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:28.654587 12491 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 21. (term mismatch)
I20260709 12:17:28.657945 25078 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.644204 25236 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:28.657449 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.655897 12646 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:28.709739 12491 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 20
I20260709 12:17:28.719328 12491 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 36.21
I20260709 12:17:28.719664 12491 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:28.721131 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.722074 12640 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:17:28.652992 25312 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:28.732702 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.647518 25324 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:28.734589 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 149077us
I20260709 12:17:28.744747 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.713907 25189 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:28.764731 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:28.765005 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 168339us
I20260709 12:17:28.756637 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:28.762717 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.765550 12643 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:28.768911 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:28.769827 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128594us
I20260709 12:17:28.770932 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.772590 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.763747 12572 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:28.640481 25327 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.119s
I20260709 12:17:28.765074 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.775966 12640 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:28.718624 25236 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:28.778308 12641 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:28.779642 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.787961 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:28.749179 25312 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:28.802867 12640 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:28.795161 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.801440 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:28.774838 25189 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:28.805193 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:28.776939 25330 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.787600 24701 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:28.780665 25272 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:28.850000 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.850495 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:28.795701 25318 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:28.805431 25312 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:28.857059 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:28.858780 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63736.9us
I20260709 12:17:28.897138 12639 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
W20260709 12:17:28.897089 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 130971us
I20260709 12:17:28.899492 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:28.902629 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45673.3us
I20260709 12:17:28.815847 25189 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:28.794271 25066 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.914485 12638 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:17:28.848621 24701 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:28.814517 25098 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:28.814173 25092 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:28.896862 25327 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:28.933298 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:28.863791 25313 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:28.814817 25325 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.115s
I20260709 12:17:28.910703 25180 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:17:28.862021 25318 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:28.914788 25309 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:17:28.953555 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 152171us
I20260709 12:17:28.928709 24701 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:28.969066 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:28.979971 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.933804 25327 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:28.973371 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:28.969612 25322 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.403s	user 0.002s	sys 0.000s
I20260709 12:17:28.983929 12490 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:28.943007 25232 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:28.942776 25338 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:17:28.967166 25318 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:28.966811 25309 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:28.997094 25309 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:28.997947 25180 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:28.997992 25127 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:29.002242 25180 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:17:29.002818 25180 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:29.004456 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1576) took 1139 ms. Trace:
I20260709 12:17:29.004940 13257 rpcz_store.cc:276] 0709 12:17:27.864951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:27.865096 (+   145us) service_pool.cc:224] Handling call
0709 12:17:27.865471 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:28.237601 (+372130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:29.004109 (+766508us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:29.004381 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":834405120}
I20260709 12:17:29.006364 25127 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:29.007102 25127 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:29.007584 25127 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:29.008838 25113 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:29.006345 25212 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:28.929359 25236 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.009343 25113 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:29.019470 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:29.030388 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:28.903772 25202 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.990559 25098 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:29.024864 25127 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:28.994868 25241 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:29.036016 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102821us
I20260709 12:17:29.061697 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:28.989959 25335 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:17:29.021721 25331 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260709 12:17:29.046952 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:29.054250 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 203873us
W20260709 12:17:29.055153 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.069312 12490 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:29.073329 12490 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
W20260709 12:17:29.078323 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88959.8us
I20260709 12:17:29.079428 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:29.087996 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 308417us
I20260709 12:17:29.090860 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.038894 25089 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.013476 25112 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.074s
I20260709 12:17:29.103814 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.041875 25098 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:29.106918 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1845) took 1103 ms. Trace:
W20260709 12:17:29.106411 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 194227us
I20260709 12:17:29.108645 12642 rpcz_store.cc:276] 0709 12:17:28.003760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.004634 (+   874us) service_pool.cc:224] Handling call
0709 12:17:28.030129 (+ 25495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:28.117337 (+ 87208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:28.117372 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:28.230869 (+113497us) log.cc:844] Serialized 26 byte log entry
0709 12:17:28.327031 (+ 96162us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:28.327078 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:28.327150 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:29.106639 (+779489us) raft_consensus.cc:1685] finished
0709 12:17:29.106663 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:29.106846 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12243,"raft.queue_time_us":71259,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7408,"thread_start_us":125930,"threads_started":2,"wal-append.queue_time_us":200325}
I20260709 12:17:29.106050 25203 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:17:29.108592 25098 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:29.112239 25236 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:29.135344 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:29.077760 25241 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:29.132649 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 112833us
I20260709 12:17:29.095978 25235 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.032s
I20260709 12:17:29.123485 25339 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:29.134419 25113 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.147225 25354 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:29.106305 25089 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.146886 12389 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 34. New cstate: current_term: 34 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:17:29.149838 25341 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:17:29.151189 25345 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:17:29.153473 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:29.172423 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.226363 25326 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:29.227035 25326 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:29.229038 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:29.221168 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75559.3us
I20260709 12:17:29.235967 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:29.238276 13256 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:29.208628 25345 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:29.273852 12645 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 43. Ops: []
I20260709 12:17:29.231460 25066 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.280632 12641 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
W20260709 12:17:29.280915 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70116.6us
I20260709 12:17:29.289789 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:29.293941 12572 raft_consensus.cc:2376] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35. Re-sending same reply.
I20260709 12:17:29.291719 12563 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 23. (term mismatch)
I20260709 12:17:29.296324 12563 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 22
I20260709 12:17:29.296655 12563 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 40.23
I20260709 12:17:29.296972 12563 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:29.298730 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:29.307216 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:29.267102 25338 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:29.308079 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:17:29.308648 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.309669 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:29.274776 25128 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.019s	sys 0.018s
I20260709 12:17:29.311502 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:29.314579 12566 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39. Re-sending same reply.
I20260709 12:17:29.311685 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:29.315798 12569 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:29.334734 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:29.336047 12572 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:29.314366 25128 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.019s	sys 0.019s
I20260709 12:17:29.342674 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:29.319636 25336 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.327s	user 0.002s	sys 0.000s
I20260709 12:17:29.347498 12566 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 18. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
W20260709 12:17:29.337533 25229 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.999s	user 0.016s	sys 0.000s
I20260709 12:17:29.299744 25372 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:29.345825 25355 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
W20260709 12:17:29.346486 25348 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
W20260709 12:17:29.367816 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 306296us
I20260709 12:17:29.382750 12568 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 41 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 20.
I20260709 12:17:28.997825 25232 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.285009 25346 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.215s
I20260709 12:17:29.314759 25339 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.055s
W20260709 12:17:29.350193 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 110957us
I20260709 12:17:28.437846 25315 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:28.896528 25330 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.382524 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.372331 25353 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:29.385128 25359 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.386716 25294 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:29.378032 25368 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.348240 25309 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.312315 25326 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.430845 13258 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:17:29.345556 25352 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.180s
I20260709 12:17:29.363202 25128 consensus_queue.cc:1243] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:17:28.862260 25324 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:29.409587 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200094us
I20260709 12:17:29.001994 25327 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.413259 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.440578 12491 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:29.422668 25363 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:29.422134 25359 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.348582 25127 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:28.603291 25323 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:29.427605 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 336802us
I20260709 12:17:29.400650 25089 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:29.364252 25318 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.466694 25318 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:29.425384 25368 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.472625 25368 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.436867 25351 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:17:29.437330 25362 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:17:29.412602 25375 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:29.455051 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:29.451978 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.457955 25373 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:29.459501 25232 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:29.460004 25078 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:17:29.460855 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:29.488510 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.460251 25359 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.463332 25356 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:29.476604 12643 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:17:29.478628 25241 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.451480 25380 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:29.494356 25031 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.490195 25381 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:29.567170 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112175us
W20260709 12:17:29.571872 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 158688us
I20260709 12:17:29.572211 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.495069 25364 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.390712 25345 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.494843 25356 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:29.573109 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.583968 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.486714 25375 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.595057 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.587405 25241 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:29.587435 25338 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:29.591475 25318 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:29.615177 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.594091 25352 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:29.594597 25382 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:29.599224 25351 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:29.618227 25349 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:29.629241 25383 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:29.629937 25383 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.632428 25383 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:29.633051 25361 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:29.635986 12536 leader_election.cc:400] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:17:29.636401 12536 leader_election.cc:403] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:29.637691 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.626370 25352 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.638757 13255 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:29.639806 25388 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:17:29.640671 25352 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.640869 25353 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32.
I20260709 12:17:29.641705 25353 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:17:29.648351 25358 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:17:29.648947 25330 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.581928 25381 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.624320 25389 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:17:29.633534 25361 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:29.651013 25361 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:29.651908 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5191) took 1003 ms. Trace:
I20260709 12:17:29.652040 12536 leader_election.cc:400] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
W20260709 12:17:29.652208 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.576962 25031 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.652331 12493 rpcz_store.cc:276] 0709 12:17:28.648883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.649221 (+   338us) service_pool.cc:224] Handling call
0709 12:17:28.710235 (+ 61014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:28.710542 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:28.710582 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:28.720678 (+ 10096us) log.cc:844] Serialized 26 byte log entry
0709 12:17:29.088949 (+368271us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:29.089006 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:29.089098 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:29.651655 (+562557us) raft_consensus.cc:1685] finished
0709 12:17:29.651678 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:29.651846 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":288332,"threads_started":1,"wal-append.queue_time_us":709057}
I20260709 12:17:29.652662 12536 leader_election.cc:403] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:17:29.655408 25343 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.523s	user 0.002s	sys 0.000s
I20260709 12:17:29.655920 25078 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34.
I20260709 12:17:29.660615 25382 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:29.661046 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:29.662109 25392 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:29.661870 25382 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.662722 25392 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:29.663609 25392 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.615226 25356 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.664958 25356 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:29.654743 25351 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:29.668161 25333 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.079s	user 0.011s	sys 0.000s
I20260709 12:17:29.668947 25359 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:29.668814 25351 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.654670 25338 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.670024 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2442) took 1217 ms. Trace:
I20260709 12:17:29.669520 25359 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:29.670701 25328 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:29.671172 25328 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:29.670593 12567 rpcz_store.cc:276] 0709 12:17:28.452655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.458737 (+  6082us) service_pool.cc:224] Handling call
0709 12:17:28.461697 (+  2960us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:28.475954 (+ 14257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:28.475987 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:28.483878 (+  7891us) log.cc:844] Serialized 52 byte log entry
0709 12:17:28.582342 (+ 98464us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:28.582385 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:28.582455 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:29.667476 (+1085021us) spinlock_profiling.cc:243] Waited 3.27 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:29.669684 (+  2208us) raft_consensus.cc:1685] finished
0709 12:17:29.669709 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:29.669964 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6266,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":3840,"spinlock_wait_cycles":7512832,"thread_start_us":66357,"threads_started":1,"wal-append.queue_time_us":83307}
I20260709 12:17:29.671384 25359 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.672216 25328 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:29.622534 25081 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.110s	user 0.015s	sys 0.001s
I20260709 12:17:29.674970 12642 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:17:29.690286 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:29.691885 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.696875 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:29.697912 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:29.701467 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.693296 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:29.708772 12490 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
W20260709 12:17:29.707964 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 110174us
I20260709 12:17:29.715291 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.718078 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:29.722908 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:29.718971 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:29.722342 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:29.726311 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:29.727357 12492 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 38 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:17:29.729022 25385 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.728873 25241 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:29.691013 25328 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:29.730479 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:29.714573 25335 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:29.753773 25335 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:29.754207 25335 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:29.755324 12389 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 18 to 23, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 23 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:29.721746 25009 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:29.734084 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:29.737555 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:29.731379 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:29.764724 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:29.595093 25364 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.733618 25078 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:29.745103 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:29.726222 25228 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:29.746678 25310 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.422s	user 0.012s	sys 0.007s
W20260709 12:17:29.747298 25312 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.818s	user 0.012s	sys 0.000s
I20260709 12:17:29.748081 25375 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:29.748919 25347 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:29.749133 25390 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.721613 25368 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:29.752756 25385 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.732348 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:29.760952 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.757333 12389 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 23 to 29. New cstate: current_term: 29 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:29.625511 25318 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.778698 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:29.783645 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:29.763684 25309 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:29.836588 12497 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:17:29.763376 25349 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:29.775622 25236 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.601s	user 0.013s	sys 0.002s
W20260709 12:17:29.793125 25098 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.004s
I20260709 12:17:29.762692 25009 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:29.819082 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:29.827385 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95085.6us
I20260709 12:17:29.838189 25386 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.104s
W20260709 12:17:29.840447 25226 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.182s	user 0.011s	sys 0.008s
I20260709 12:17:29.845872 25321 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.112s
W20260709 12:17:29.847025 24701 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.869s	user 0.006s	sys 0.008s
W20260709 12:17:29.847450 25189 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.902s	user 0.013s	sys 0.001s
I20260709 12:17:29.854292 25363 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:29.795178 25078 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.854738 25385 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:29.815112 25390 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.793991 25312 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.777150 25325 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.045s
I20260709 12:17:29.814851 25323 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:29.759385 25241 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.863641 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:29.866001 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105124us
I20260709 12:17:29.819901 25375 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:29.873040 12566 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 16. Preceding OpId from leader: term: 42 index: 18. (index mismatch)
I20260709 12:17:29.845381 25113 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:29.875259 13255 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:29.881300 12572 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 27. Current term is 28. Ops: []
I20260709 12:17:29.884142 25031 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:29.884876 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:29.885694 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:29.949831 12568 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:29.952127 12566 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:29.997509 13258 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 35 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:17:29.790024 25310 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.785184 25228 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:29.926129 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.008080 12637 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:29.875106 25009 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:29.929931 12564 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:17:29.875861 25380 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.869585 25305 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:29.875367 25353 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:29.875736 25378 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:29.941560 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:29.890053 24701 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.995965 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:29.881915 25381 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.008867 25309 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.006796 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1580) took 1857 ms. Trace:
I20260709 12:17:29.916894 25272 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:29.815135 25347 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:29.874969 25098 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.014994 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:29.935747 25386 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.088s
I20260709 12:17:29.938874 25349 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:29.938069 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:29.918144 25363 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:30.024948 12639 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:30.019591 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:29.947299 25375 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:30.064543 12640 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:30.028316 12570 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:17:30.028239 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.066100 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:30.029742 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:17:30.016750 25250 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.270s	user 0.013s	sys 0.003s
I20260709 12:17:30.006527 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:29.920383 25189 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:30.012995 25313 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.068s	user 0.013s	sys 0.001s
I20260709 12:17:30.036967 13255 rpcz_store.cc:276] 0709 12:17:28.139341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.139551 (+   210us) service_pool.cc:224] Handling call
0709 12:17:29.638331 (+1498780us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5800500fd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c
0709 12:17:29.996122 (+357791us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2586643328}
W20260709 12:17:30.057476 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119519us
I20260709 12:17:30.069373 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:30.048674 25323 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:30.036900 25229 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:30.081072 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:30.048708 25078 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.069646 25375 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:30.023952 25361 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.095502 12491 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:30.069489 25232 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:30.110057 12644 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: [37.20-37.20]
W20260709 12:17:30.118458 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177008us
I20260709 12:17:30.121361 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.080116 25313 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:30.026228 25380 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.125458 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.049840 25309 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.109771 25345 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:30.038159 25353 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.038121 25378 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.133613 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.078696 25250 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.119060 25232 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:30.057384 25272 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.064524 25349 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.147287 25189 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:30.168612 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:30.147958 25353 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:30.149117 13254 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:30.149988 12646 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:30.140275 25363 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.150851 25180 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.152562 24701 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.153249 25241 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.149225 25250 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.131114 25305 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:30.135682 25078 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:30.162696 25349 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.164089 13257 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:30.165330 25272 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.129580 25390 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.171236 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:30.223029 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51876.5us
I20260709 12:17:30.224920 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:30.227800 25364 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:30.154189 25089 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:30.142249 25232 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:30.155094 25226 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50712 (request call id 1569) took 5681 ms. Trace:
I20260709 12:17:30.159243 25312 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.166531 25098 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.207255 25180 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:30.208212 25400 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.366s	user 0.003s	sys 0.001s
I20260709 12:17:30.208361 25380 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.209766 25378 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.211156 25361 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:30.231174 25363 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:30.212694 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.214257 25228 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:30.198844 25402 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.305s	user 0.002s	sys 0.000s
I20260709 12:17:30.223163 25241 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:17:30.199148 25405 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:17:30.187978 25318 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:30.233559 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:30.252684 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:30.259173 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:30.260857 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.261260 25378 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:17:30.199383 25404 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:17:30.181666 12493 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
W20260709 12:17:30.233878 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:30.262364 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:30.207070 25408 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:30.230031 25078 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.014s
I20260709 12:17:30.263266 25408 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.243582 25305 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:30.265168 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:30.236778 25226 rpcz_store.cc:276] 0709 12:17:24.458206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.458462 (+   256us) service_pool.cc:224] Handling call
0709 12:17:25.624314 (+1165852us) ts_tablet_manager.cc:934] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
0709 12:17:28.009684 (+2385370us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:17:28.637547 (+627863us) tablet_metadata.cc:665] Metadata flushed
0709 12:17:28.651565 (+ 14018us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:17:29.931691 (+1280126us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:17:30.034100 (+102409us) tablet_replica.cc:206] Creating consensus instance
0709 12:17:30.100042 (+ 65942us) ts_tablet_manager.cc:989] Replica registered.
0709 12:17:30.139468 (+ 39426us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":798,"tablet-copy.queue_time_us":734653,"thread_start_us":850417,"threads_started":2}
I20260709 12:17:30.187247 25323 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.266182 25189 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:17:30.266513 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:30.267002 25189 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 39.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:17:30.248999 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.264911 25305 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.272112 25364 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:30.273115 25305 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.266194 25226 tablet_copy_client.cc:806] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:17:30.274559 25226 tablet_copy_client.cc:670] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:17:30.265879 25408 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.201808 25353 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:30.276183 25406 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:17:30.275866 25353 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.267959 25356 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.277349 25356 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.278295 25353 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:30.279136 25318 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:30.279760 25356 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:30.287770 25232 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:30.305907 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 190909us
I20260709 12:17:30.313221 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.255337 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.267959 12496 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:30.270761 25078 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:30.291363 25364 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:30.316764 12639 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:30.318341 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.199606 25411 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:30.237735 25390 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:30.199157 25417 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:30.268208 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:30.355957 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87843.3us
W20260709 12:17:30.294269 25328 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.002s
W20260709 12:17:30.294822 25425 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:17:30.295480 25309 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:30.296546 25310 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.300096 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:30.303364 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:30.247905 25347 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.261763 25378 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:30.315487 25356 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.368347 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.374349 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.375272 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.379567 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.379323 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:30.241859 25089 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.390231 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.386782 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.391052 12493 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:30.370138 25390 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:30.392154 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:30.393744 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:30.394554 12492 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:30.395278 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:30.364163 25426 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:30.401937 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:30.406150 25078 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:30.410292 25390 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:30.411505 25326 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:30.398187 25417 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:30.412467 25417 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:30.412977 25417 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:30.387607 25378 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:30.417021 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.417825 12639 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:30.418802 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.420006 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:30.421080 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.422582 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:30.423148 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:30.423383 25386 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:17:30.423251 25375 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.424440 25127 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.425055 25127 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.427354 25127 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.429682 25359 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:30.433938 12642 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:30.436311 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.436668 25058 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:30.437104 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:30.437175 25058 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:30.439028 25335 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.684s	user 0.013s	sys 0.000s
I20260709 12:17:30.439388 25426 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:30.440428 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:30.441661 25407 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.311s	user 0.002s	sys 0.000s
I20260709 12:17:30.487283 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.489466 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.490216 12490 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:30.381518 25114 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:17:30.446578 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4885) took 1215 ms. Trace:
I20260709 12:17:30.452564 12638 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:30.457167 25009 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.458125 12389 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 28, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 28 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:30.459666 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.461258 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.462388 25327 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:30.464174 25315 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.360451 25411 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.475689 25232 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.477178 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1590) took 1508 ms. Trace:
I20260709 12:17:30.396315 25356 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.491101 12496 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:30.495425 12640 rpcz_store.cc:276] 0709 12:17:29.231561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:29.635975 (+404414us) service_pool.cc:224] Handling call
0709 12:17:29.636265 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:30.068740 (+432475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:30.446241 (+377501us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:30.446488 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984518912}
I20260709 12:17:30.500401 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.501287 12493 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:17:30.512528 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.514668 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:30.514912 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:30.521219 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 162691us
I20260709 12:17:30.521575 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:30.544234 12570 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:30.544656 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:30.522080 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.545518 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.546530 25388 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:30.546757 12569 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:30.547147 25388 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:30.548015 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:30.548683 12569 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:30.550012 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:30.551235 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.526100 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:30.552456 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:30.553267 25228 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:30.553380 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:30.536446 13254 rpcz_store.cc:276] 0709 12:17:28.969143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.969310 (+   167us) service_pool.cc:224] Handling call
0709 12:17:28.969663 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:30.169980 (+1200317us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:30.429536 (+259556us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:30.477114 (+ 47578us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2701152000}
I20260709 12:17:30.545949 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:30.554788 12564 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:30.555119 12570 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:30.555325 25078 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:30.487423 25426 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:30.555910 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.557178 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.519531 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:30.533150 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.560218 25294 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:30.560631 25294 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:30.560990 25294 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:30.555950 25078 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:30.563427 25078 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:30.563555 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:30.544845 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.565881 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:30.566722 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.567185 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:30.568312 25385 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:30.549324 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:30.570319 12565 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:30.572518 12563 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:17:30.553893 25228 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:30.574162 25228 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:30.548182 25388 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.581431 12490 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:30.581971 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:30.583423 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:30.584223 12493 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:30.584527 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.571344 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:30.587049 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:30.520911 25272 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:30.591320 12564 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:30.520706 25089 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.588332 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.597733 13261 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:30.591812 12642 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:30.592502 12492 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:30.595034 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.600338 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:30.580106 25407 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:17:30.635890 12490 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.21-37.21]
I20260709 12:17:30.600991 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:30.630026 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.599614 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:30.601428 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:30.639709 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45043.4us
I20260709 12:17:30.640040 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:30.495613 25335 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:30.593729 25351 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:30.660279 12489 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:30.625763 25229 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:30.601490 12492 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:17:30.664791 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.632311 25189 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:30.654734 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:30.688472 12495 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
W20260709 12:17:30.647912 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46585.6us
I20260709 12:17:30.675763 13262 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:30.645988 25098 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:17:30.636319 25232 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:30.580523 25058 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.663591 25385 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:30.658341 25375 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:30.676980 25229 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:30.598973 25218 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.072s
I20260709 12:17:30.668260 25351 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:30.715425 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4873) took 2096 ms. Trace:
I20260709 12:17:30.659704 25377 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:30.716851 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3879) took 1019 ms. Trace:
I20260709 12:17:31.021816 12491 rpcz_store.cc:276] 0709 12:17:29.697457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:29.697582 (+   125us) service_pool.cc:224] Handling call
0709 12:17:30.716562 (+1018980us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:30.716758 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360354048}
I20260709 12:17:30.709760 12567 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 15
I20260709 12:17:31.023062 25351 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:30.693410 25379 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.024083 25379 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.625404 25272 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.020962 12643 rpcz_store.cc:276] 0709 12:17:28.618579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.794342 (+175763us) service_pool.cc:224] Handling call
0709 12:17:28.794837 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:29.536606 (+741769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:30.715099 (+1178493us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b540018c4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:30.715353 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1566941056}
I20260709 12:17:30.671716 25204 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.062s
I20260709 12:17:31.026564 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:31.027171 25204 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:31.027616 25204 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:31.027874 25272 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:30.701193 25411 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:30.872220 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 285245us
I20260709 12:17:31.028599 25204 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.029543 25377 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:31.029714 25236 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:31.030212 25236 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:31.030516 25377 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:31.031333 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:31.031046 25236 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:30.826903 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:31.023411 12567 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 39.16
I20260709 12:17:30.687391 25189 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:30.641841 25089 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:17:30.880664 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.691907 25430 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:30.968879 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:30.961977 13261 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:30.969048 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.718325 25397 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.179s
I20260709 12:17:30.706178 25078 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:30.799026 25333 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:17:31.021435 25058 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:30.745388 25338 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:30.880034 25180 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:31.019865 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:30.727401 25356 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.027906 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.036338 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:31.032958 12567 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:30.971442 25229 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:30.969606 25375 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:30.908802 25232 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:30.881978 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:31.052112 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:31.052621 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67249.3us
I20260709 12:17:31.032255 25236 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:31.019933 25365 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.096946 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:31.045121 25189 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:31.090369 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.101701 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:31.102964 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.022s	sys 0.047s
W20260709 12:17:31.100813 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49675.5us
I20260709 12:17:31.104441 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:31.066833 25009 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.103765 12643 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:17:31.071673 25202 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.116472 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:31.117010 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:31.100785 25294 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.539s	user 0.012s	sys 0.000s
I20260709 12:17:31.103854 12640 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:17:31.105888 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:31.111768 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.070111 25344 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:31.086992 25204 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:31.107805 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.022s	sys 0.048s
W20260709 12:17:31.107409 25390 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.001s	sys 0.039s
I20260709 12:17:31.128126 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:31.129849 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:31.086843 25310 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:31.136063 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:31.102502 25386 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:31.094151 25180 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.102011 25440 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.096s
I20260709 12:17:31.154031 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:31.187865 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58140us
I20260709 12:17:31.197053 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:31.140486 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.096060 25379 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:31.236100 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:31.114444 25089 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.229173 13261 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 38 index: 27. (index mismatch)
I20260709 12:17:31.224362 12640 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:31.230640 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:31.250694 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3952) took 1017 ms. Trace:
I20260709 12:17:31.251803 12640 rpcz_store.cc:276] 0709 12:17:30.233119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.140161 (+907042us) service_pool.cc:224] Handling call
0709 12:17:31.250614 (+110453us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:31.188292 25327 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.654s	user 0.008s	sys 0.016s
W20260709 12:17:31.179805 25417 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.766s	user 0.010s	sys 0.001s
I20260709 12:17:31.134049 25218 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:31.105885 25375 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:31.238596 25363 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.984s	user 0.015s	sys 0.000s
I20260709 12:17:31.188796 25203 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:31.187278 25437 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.285391 25437 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.252162 25379 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:31.287720 25379 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:31.288154 25379 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:31.244848 25272 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.044s
I20260709 12:17:31.288486 25379 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:31.273216 25417 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.291736 25417 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.139837 25294 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.292972 13256 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:31.294370 25226 tablet_copy_client.cc:538] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:17:31.258005 25089 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.296823 25089 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:31.145241 25335 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.109711 25365 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.302121 25365 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:31.302759 25365 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:17:31.170899 25078 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.172711 25232 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.304535 25203 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:31.305101 25203 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:31.307619 25351 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.308111 25189 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.311703 25180 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.186249 25441 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.173302 25397 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:31.186874 25009 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:31.274044 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.188762 25356 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:31.243542 25310 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.263942 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.319623 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1863) took 1067 ms. Trace:
I20260709 12:17:31.320461 12643 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:31.321545 12640 rpcz_store.cc:276] 0709 12:17:30.252202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.273746 (+1021544us) service_pool.cc:224] Handling call
0709 12:17:31.319551 (+ 45805us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.122000 25202 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.324820 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3947) took 1603 ms. Trace:
I20260709 12:17:31.297467 25089 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:31.326959 12643 rpcz_store.cc:276] 0709 12:17:29.721003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.263649 (+1542646us) service_pool.cc:224] Handling call
0709 12:17:31.324748 (+ 61099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.328341 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.279364 25352 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.330960 12640 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:31.334357 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3957) took 1099 ms. Trace:
I20260709 12:17:31.334882 12640 rpcz_store.cc:276] 0709 12:17:30.234661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.327897 (+1093236us) service_pool.cc:224] Handling call
0709 12:17:31.334270 (+  6373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.145007 25344 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.313977 25327 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:17:31.320735 25426 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.764s	user 0.026s	sys 0.003s
I20260709 12:17:31.346652 12643 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:31.294436 25294 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.324295 25356 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:31.347988 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 309252us
I20260709 12:17:31.332162 25218 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:31.351043 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 469283us
W20260709 12:17:31.330137 25228 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.755s	user 0.010s	sys 0.003s
I20260709 12:17:31.329764 25397 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:31.287750 25437 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:31.324221 25441 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:31.354791 25378 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.940s	user 0.014s	sys 0.002s
I20260709 12:17:31.319096 25009 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.463497 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:31.467911 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:31.468894 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:31.357586 12640 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:17:31.361428 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1859) took 1653 ms. Trace:
I20260709 12:17:31.471416 12643 rpcz_store.cc:276] 0709 12:17:29.707540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.339570 (+1632030us) service_pool.cc:224] Handling call
0709 12:17:31.340066 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.361002 (+ 20936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.361347 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.473009 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.358572 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:31.474107 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115678us
I20260709 12:17:31.411322 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:31.475853 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64721.1us
I20260709 12:17:31.414296 12491 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:31.411281 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:31.420509 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:31.346262 25352 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.352990 25327 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:31.451495 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:31.479732 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:17:31.358021 25356 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:31.493791 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82631us
I20260709 12:17:31.371536 25078 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.363256 25363 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.483352 12492 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:17:31.491122 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:31.481750 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:31.494516 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:31.408062 25417 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.499287 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4886) took 1780 ms. Trace:
I20260709 12:17:31.526017 12640 rpcz_store.cc:276] 0709 12:17:29.719110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.351357 (+1632247us) service_pool.cc:224] Handling call
0709 12:17:31.357101 (+  5744us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:31.498818 (+141717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.499197 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:31.526602 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45000.1us
I20260709 12:17:31.527590 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:31.503863 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:31.456050 25388 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.356279 25426 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.533385 12640 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:17:31.540647 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4887) took 1821 ms. Trace:
W20260709 12:17:31.545285 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:31.405658 25228 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.491591 25382 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:31.552937 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3885) took 1298 ms. Trace:
I20260709 12:17:31.541157 12640 rpcz_store.cc:276] 0709 12:17:29.719410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.532632 (+1813222us) service_pool.cc:224] Handling call
0709 12:17:31.533007 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.540253 (+  7246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.540565 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.555226 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:31.555887 12640 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:17:31.506275 25327 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:31.502348 25385 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:31.503017 25365 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:31.405571 25218 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.553567 12495 rpcz_store.cc:276] 0709 12:17:30.240872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.241018 (+   146us) service_pool.cc:224] Handling call
0709 12:17:30.241402 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:30.689139 (+447737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.539278 (+850139us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:31.539578 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1027502464}
I20260709 12:17:31.613829 12488 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:31.637097 12488 raft_consensus.cc:606] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:31.557654 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3949) took 1836 ms. Trace:
I20260709 12:17:31.466758 25378 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.637537 12488 raft_consensus.cc:614] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:31.637734 12640 rpcz_store.cc:276] 0709 12:17:29.721300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.554905 (+1833605us) service_pool.cc:224] Handling call
0709 12:17:31.555378 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.557240 (+  1862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.557558 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.611814 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:31.646046 25450 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.560846 25180 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:31.552575 25272 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:31.646806 25454 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.550657 25363 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.647632 12563 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:31.525724 25078 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.648008 25382 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:31.654675 25441 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.656616 25379 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.657245 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1831) took 1256 ms. Trace:
I20260709 12:17:31.723831 13256 rpcz_store.cc:276] 0709 12:17:30.400972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.401185 (+   213us) service_pool.cc:224] Handling call
0709 12:17:30.708732 (+307547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.293697 (+584965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.656213 (+362516us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b54005c4c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:31.657015 (+   802us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54005c4c50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:31.657180 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2020352512}
I20260709 12:17:31.520875 25338 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:31.666061 25409 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:31.451498 25397 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:31.669824 25444 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.511s	user 0.000s	sys 0.009s
I20260709 12:17:31.671697 25449 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:31.636051 25453 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.113s
I20260709 12:17:31.672634 25189 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:31.609315 25448 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:17:31.609786 25451 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.067s
I20260709 12:17:31.609323 25383 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:31.678789 25228 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.679418 13255 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:31.736510 12643 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:31.680496 25310 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:31.681113 25375 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:31.682785 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4876) took 2976 ms (client timeout 1691 ms). Trace:
I20260709 12:17:31.683178 25009 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:31.677433 25385 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.709527 25450 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:31.552428 25442 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:17:31.551926 25417 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.553689 25390 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:31.690677 25344 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:31.692005 25058 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.617s	user 0.013s	sys 0.000s
I20260709 12:17:31.692114 12495 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:31.684682 25356 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:31.692808 25326 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.281s	user 0.016s	sys 0.000s
I20260709 12:17:31.694923 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4866) took 3080 ms. Trace:
W20260709 12:17:31.669683 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3946) took 1948 ms (client timeout 2166 ms). Trace:
W20260709 12:17:31.768266 12642 rpcz_store.cc:269] 0709 12:17:29.720855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.266250 (+545395us) service_pool.cc:224] Handling call
0709 12:17:31.669341 (+1403091us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:31.669530 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":383351040}
I20260709 12:17:31.767524 12637 rpcz_store.cc:276] 0709 12:17:28.614728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.614889 (+   161us) service_pool.cc:224] Handling call
0709 12:17:28.653418 (+ 38529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:30.008612 (+1355194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.694589 (+1685977us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:31.694836 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3114936704}
I20260709 12:17:31.643436 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:31.778481 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135159us
I20260709 12:17:31.613824 25324 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:31.710755 13259 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:31.694923 25426 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:31.716717 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4870) took 3100 ms. Trace:
W20260709 12:17:31.708382 25452 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:17:31.560544 25388 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.710134 25202 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:31.726295 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:31.727524 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1855) took 2151 ms. Trace:
I20260709 12:17:31.615108 25440 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.157s
I20260709 12:17:31.708369 25272 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:31.727934 25236 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.623s	user 0.012s	sys 0.000s
I20260709 12:17:31.734148 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:31.713966 25454 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:31.737558 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.015s	sys 0.014s
W20260709 12:17:31.843394 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.015s	sys 0.015s
W20260709 12:17:31.738387 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.000s	sys 0.030s
W20260709 12:17:31.740069 12646 rpcz_store.cc:269] 0709 12:17:28.707013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.720645 (+ 13632us) service_pool.cc:224] Handling call
0709 12:17:31.682367 (+2961722us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54001f72d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:31.682647 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3283811200}
W20260709 12:17:31.844504 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.000s	sys 0.031s
I20260709 12:17:31.549355 25437 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:31.733419 25377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.043s	sys 0.060s
I20260709 12:17:31.730923 25398 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.728992 25449 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:31.746976 25229 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.637s	user 0.014s	sys 0.000s
I20260709 12:17:31.766912 12489 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:31.811653 12644 rpcz_store.cc:276] 0709 12:17:28.615997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:28.754005 (+138008us) service_pool.cc:224] Handling call
0709 12:17:28.776663 (+ 22658us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:30.115366 (+1338703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.716249 (+1600883us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:31.716616 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3054745472}
I20260709 12:17:31.823782 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1865) took 1552 ms. Trace:
W20260709 12:17:31.845198 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 648167us
I20260709 12:17:31.860648 12637 rpcz_store.cc:276] 0709 12:17:30.271125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.773242 (+1502117us) service_pool.cc:224] Handling call
0709 12:17:31.823226 (+ 49984us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.823434 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.823463 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:31.823501 (+    38us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:31.823543 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.823599 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:31.823721 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.811827 25409 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:31.874672 12568 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 30. Current term is 31. Ops: [30.15-30.15]
I20260709 12:17:31.806023 25323 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.814004 25383 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.886667 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:31.859198 25377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.043s	sys 0.060s
I20260709 12:17:31.890733 13254 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:31.825807 12645 rpcz_store.cc:276] 0709 12:17:29.576037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:29.650647 (+ 74610us) service_pool.cc:224] Handling call
0709 12:17:29.651083 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:29.998307 (+347224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:29.998407 (+   100us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:30.015406 (+ 16999us) log.cc:844] Serialized 26 byte log entry
0709 12:17:30.115788 (+100382us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:30.115848 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.714393 (+1598545us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:31.714475 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:31.714543 (+    68us) raft_consensus.cc:1685] finished
0709 12:17:31.714561 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:31.727388 (+ 12827us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22927,"raft.queue_time_us":62609,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7020,"spinlock_wait_cycles":378859776,"thread_start_us":255305,"threads_started":2,"wal-append.queue_time_us":168127,"wal-append.run_cpu_time_us":2858,"wal-append.run_wall_time_us":1047102}
I20260709 12:17:31.863379 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:31.748968 25385 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.840907 25272 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:31.836088 25426 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.880218 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:31.884965 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:31.858122 12488 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:31.887291 12568 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:31.889243 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:31.902495 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4892) took 1498 ms. Trace:
I20260709 12:17:31.903054 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:31.902999 12637 rpcz_store.cc:276] 0709 12:17:30.403881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.881785 (+1477904us) service_pool.cc:224] Handling call
0709 12:17:31.901870 (+ 20085us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.902112 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.902155 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:31.902203 (+    48us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:31.902253 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.902315 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:31.902430 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.904419 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:17:31.890416 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:31.905409 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:31.877938 25009 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:17:31.910233 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:31.911145 12569 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:31.906231 12564 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:31.867880 25409 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:31.900655 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4896) took 1380 ms. Trace:
I20260709 12:17:31.916077 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:31.917586 12646 rpcz_store.cc:276] 0709 12:17:30.520566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.899614 (+1379048us) service_pool.cc:224] Handling call
0709 12:17:31.900044 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.900268 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.900298 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:31.900345 (+    47us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:31.900400 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.900467 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:31.900590 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.921695 12572 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:31.923851 12566 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:31.924432 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1866) took 1467 ms. Trace:
I20260709 12:17:31.915217 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:31.925832 12637 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:31.926641 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4898) took 1391 ms. Trace:
I20260709 12:17:31.927014 12637 rpcz_store.cc:276] 0709 12:17:30.536051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.904500 (+1368449us) service_pool.cc:224] Handling call
0709 12:17:31.925542 (+ 21042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.926354 (+   812us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.926579 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.928382 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:31.928840 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 318340us
I20260709 12:17:31.913277 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2467) took 1365 ms. Trace:
I20260709 12:17:31.924911 12645 rpcz_store.cc:276] 0709 12:17:30.456919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.906730 (+1449811us) service_pool.cc:224] Handling call
0709 12:17:31.923800 (+ 17070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.924045 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.924075 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:31.924126 (+    51us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:31.924166 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.924226 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:31.924355 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:31.818718 25382 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.892692 25236 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.020s
I20260709 12:17:31.864907 25398 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.874547 25229 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.817963 25338 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.935171 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:31.937258 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.891664 25440 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.041s
I20260709 12:17:31.865080 25437 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.907236 25413 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.948556 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:31.914292 25378 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:31.881232 25375 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:17:31.938984 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.950922 12492 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:31.923144 25409 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:31.890882 25323 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.957432 25009 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:31.971762 25385 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.972813 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.973650 12645 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:31.973937 25454 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:31.974897 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:31.976014 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2468) took 1427 ms. Trace:
I20260709 12:17:31.932377 25363 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.893252 25383 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:31.967621 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:31.969566 25434 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.855s	user 0.000s	sys 0.005s
I20260709 12:17:31.970521 25388 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:31.971927 25009 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:31.976281 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:31.983263 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:31.919499 25426 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.992265 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:31.987088 12563 rpcz_store.cc:276] 0709 12:17:30.548842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.634523 (+ 85681us) service_pool.cc:224] Handling call
0709 12:17:30.660689 (+ 26166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.714137 (+1053448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.975752 (+261615us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:31.975950 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2251818368}
I20260709 12:17:31.930768 12570 rpcz_store.cc:276] 0709 12:17:30.548366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.612321 (+ 63955us) service_pool.cc:224] Handling call
0709 12:17:30.612777 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:30.613065 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:30.658718 (+ 45653us) log.cc:844] Serialized 14 byte log entry
0709 12:17:30.663550 (+  4832us) log.cc:844] Serialized 14 byte log entry
0709 12:17:31.649823 (+986273us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b18003be1e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf53a9d70 00007ffbf539d293 00007ffbf5392cea 00007ffbf539323a 00007ffbf57fda2a 00007ffbf580d0bf 00007ffbf580d038
0709 12:17:31.912899 (+263076us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:31.912962 (+    63us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:31.913023 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.913108 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:31.913221 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2807,"prepare.queue_time_us":82340,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":110,"raft.queue_time_us":3256,"spinlock_wait_cycles":19787776,"thread_start_us":262719,"threads_started":1}
I20260709 12:17:32.018748 12564 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:17:31.979229 25401 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.098s	user 0.003s	sys 0.012s
I20260709 12:17:31.955845 25218 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:31.977473 25440 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
W20260709 12:17:32.007953 25403 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.121s	user 0.003s	sys 0.006s
I20260709 12:17:32.043843 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:31.937086 25327 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.007385 25378 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:31.964143 25397 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.054468 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:32.055588 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.056124 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:32.057493 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.057976 25466 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.058578 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.058826 25466 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:31.972810 25385 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:31.920928 25449 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.991166 25356 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.062320 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:31.963176 25204 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:31.943785 25377 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.104s
I20260709 12:17:32.008689 25363 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.059365 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:32.060899 25385 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.984130 25078 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.065347 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:32.054494 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:31.965207 25413 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:31.976528 25326 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.070s
I20260709 12:17:32.084635 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:32.035511 25388 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.037398 25229 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.053282 25465 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:32.092814 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:32.136387 12645 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:32.045802 25467 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.142247 25467 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.035846 25440 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:32.094309 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:32.078557 25455 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:17:32.082479 25463 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:32.156518 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:32.151940 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.157698 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:32.152134 12490 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:17:32.085130 25450 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.096402 25417 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.165243 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3894) took 1763 ms. Trace:
I20260709 12:17:32.153122 12646 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:32.104554 25078 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:32.080789 25464 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:32.104897 25375 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.084957 25363 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:32.152045 25458 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:17:32.146827 25466 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:32.157054 25401 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:17:32.078788 25378 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.165912 12497 rpcz_store.cc:276] 0709 12:17:30.401484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.414801 (+ 13317us) service_pool.cc:224] Handling call
0709 12:17:30.415140 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:30.415512 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:30.415546 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:30.416094 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:17:30.657593 (+241499us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:31.067331 (+409738us) log.cc:844] Serialized 14 byte log entry
0709 12:17:31.098086 (+ 30755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.098207 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:31.968797 (+870590us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:32.157750 (+188953us) raft_consensus.cc:1685] finished
0709 12:17:32.164874 (+  7124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.165150 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70149,"prepare.queue_time_us":43702,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":53602,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":12788,"spinlock_wait_cycles":146056064,"thread_start_us":617026,"threads_started":2,"wal-append.queue_time_us":630115}
I20260709 12:17:32.179291 25212 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.164645 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.193863 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.131817 25366 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.187937 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:32.189366 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124134us
I20260709 12:17:32.197840 12490 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:32.198993 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:32.200526 12645 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:32.189289 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:32.164134 25463 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.202486 25440 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.203945 25440 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:32.205019 25463 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.206290 25383 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:32.207088 25398 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:32.132521 25388 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.185462 25375 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.207561 12572 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:17:32.157090 25465 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:32.210561 25469 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:17:32.159940 25009 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.211020 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:32.209385 25388 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.212743 12492 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:32.171633 25474 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:32.183071 25459 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.275s	user 0.001s	sys 0.001s
I20260709 12:17:32.233325 12497 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:17:32.242205 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:32.242882 12645 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:32.207016 25363 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.247356 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.250324 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.254452 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:32.255717 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4895) took 1743 ms (client timeout 1691 ms). Trace:
I20260709 12:17:32.266183 12566 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
W20260709 12:17:32.266746 12638 rpcz_store.cc:269] 0709 12:17:30.511992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.255496 (+1743504us) service_pool.cc:211] Skipping call since client already timed out
0709 12:17:32.255643 (+   147us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:17:32.267861 12564 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
W20260709 12:17:32.257196 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116486us
I20260709 12:17:32.278672 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:32.247131 25467 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.255470 25009 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:32.217195 25212 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.219626 25366 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:32.296777 25309 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.198s	user 0.014s	sys 0.019s
W20260709 12:17:32.364848 25475 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
W20260709 12:17:32.365654 25468 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
W20260709 12:17:32.366305 25309 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.281s	user 0.015s	sys 0.019s
W20260709 12:17:32.356036 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74400.3us
I20260709 12:17:32.296802 25437 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.303042 25462 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.330904 25226 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:17:32.370674 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1589) took 1333 ms. Trace:
I20260709 12:17:32.371296 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:32.303850 25472 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:17:32.348153 12646 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:32.363446 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:32.373746 13258 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:32.366809 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:32.302529 12638 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:17:32.376129 12645 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:32.376600 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4901) took 1327 ms. Trace:
I20260709 12:17:32.377152 12638 rpcz_store.cc:276] 0709 12:17:31.049487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.295592 (+1246105us) service_pool.cc:224] Handling call
0709 12:17:32.302195 (+  6603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:32.376325 (+ 74130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.376548 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:32.379379 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4903) took 1329 ms. Trace:
I20260709 12:17:32.301829 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:32.366761 25212 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.300791 25397 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:32.383226 25397 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:32.385281 25397 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.381393 25437 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:32.368677 25462 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.391306 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1861) took 2140 ms. Trace:
I20260709 12:17:32.371156 13262 rpcz_store.cc:276] 0709 12:17:31.037251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.250640 (+213389us) service_pool.cc:224] Handling call
0709 12:17:31.283617 (+ 32977us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:31.718708 (+435091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.718754 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:31.723329 (+  4575us) log.cc:844] Serialized 26 byte log entry
0709 12:17:31.952820 (+229491us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:31.952880 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:31.953002 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:32.370432 (+417430us) raft_consensus.cc:1685] finished
0709 12:17:32.370458 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.370605 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2806,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":642403,"threads_started":2,"wal-append.queue_time_us":318500}
I20260709 12:17:32.392666 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:32.380497 12638 rpcz_store.cc:276] 0709 12:17:31.050098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.378268 (+1328170us) service_pool.cc:224] Handling call
0709 12:17:32.378697 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:32.378960 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:32.379000 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:32.379058 (+    58us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:32.379113 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.379184 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.379312 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:32.298892 25461 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.381878 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:32.368393 25477 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:17:32.367551 25478 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.395082 25378 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:32.395717 25413 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:32.407621 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:32.413219 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4902) took 1363 ms. Trace:
I20260709 12:17:32.413597 12645 rpcz_store.cc:276] 0709 12:17:31.049806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.377307 (+1327501us) service_pool.cc:224] Handling call
0709 12:17:32.411989 (+ 34682us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:32.412994 (+  1005us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:32.413010 (+    16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:32.413039 (+    29us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:32.413063 (+    24us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.413098 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.413174 (+    76us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:32.402469 12642 rpcz_store.cc:276] 0709 12:17:30.251436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.769278 (+1517842us) service_pool.cc:224] Handling call
0709 12:17:31.769740 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:31.774605 (+  4865us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.774660 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:31.810775 (+ 36115us) log.cc:844] Serialized 26 byte log entry
0709 12:17:32.047167 (+236392us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:32.186584 (+139417us) log.cc:844] Serialized 14 byte log entry
0709 12:17:32.186829 (+   245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.186910 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:32.391060 (+204150us) raft_consensus.cc:1685] finished
0709 12:17:32.391083 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.391245 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5369,"prepare.queue_time_us":309318,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":36406,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":4875,"spinlock_wait_cycles":7794560,"thread_start_us":282596,"threads_started":2,"wal-append.queue_time_us":323855}
I20260709 12:17:32.416181 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:32.419853 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:32.421424 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4904) took 1362 ms. Trace:
I20260709 12:17:32.421931 12638 rpcz_store.cc:276] 0709 12:17:31.050363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.406118 (+1355755us) service_pool.cc:224] Handling call
0709 12:17:32.412444 (+  6326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:32.412721 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:32.412761 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:32.412821 (+    60us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:32.412882 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.412958 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.413117 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:32.335606 25479 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.353437 25465 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.330062 25426 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.427397 25485 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:32.342163 25375 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.430307 25492 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:32.431018 25378 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:32.398638 25327 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.433576 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:32.435426 12642 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:32.438176 25386 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:17:32.438572 25456 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:32.438954 25456 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:32.440016 25386 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:32.439708 25456 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.440747 25456 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:32.442433 13262 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:32.389530 25397 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:32.424705 25203 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.107s	user 0.017s	sys 0.000s
I20260709 12:17:32.443149 13262 raft_consensus.cc:606] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:32.410259 25437 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:32.444190 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87738.9us
I20260709 12:17:32.448097 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:32.455849 25203 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.431818 25478 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.434463 25474 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:32.462437 12490 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:17:32.436239 25485 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.471462 13258 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:32.484119 12645 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:32.487206 13258 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: [38.27-38.27]
I20260709 12:17:32.487836 12645 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:32.461014 25375 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:32.494995 12496 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:32.460548 25492 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.471064 25437 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:32.497573 12568 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:32.503223 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.504256 12565 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:32.420215 25461 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.490331 25212 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:32.490402 25485 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:32.506352 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:32.511809 25272 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.594s	user 0.004s	sys 0.009s
I20260709 12:17:32.512358 25272 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.514997 25272 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.515741 25272 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:32.516150 25272 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:32.518069 25366 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.519119 25203 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:32.508877 25212 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.519776 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4891) took 2495 ms. Trace:
I20260709 12:17:32.524109 25212 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:32.524422 25476 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:17:32.526513 25462 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.528429 25478 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:32.532132 25324 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.715s	user 0.010s	sys 0.005s
I20260709 12:17:32.532641 25411 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.532943 25323 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:32.533174 25411 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:32.535820 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3953) took 2302 ms (client timeout 2161 ms). Trace:
I20260709 12:17:32.536444 25411 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:32.536193 12639 rpcz_store.cc:269] 0709 12:17:30.233394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.535586 (+2302192us) service_pool.cc:211] Skipping call since client already timed out
0709 12:17:32.535716 (+   130us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:17:32.538834 25461 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:32.445984 25479 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.542308 25479 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.542539 12389 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 26 to 29, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 29 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:32.544414 12389 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 24 to 27. New cstate: current_term: 27 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:32.501928 25492 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:32.508177 25375 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:32.549152 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.545604 12389 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 23 to 29. New cstate: current_term: 29 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:17:32.549758 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3959) took 2163 ms (client timeout 2027 ms). Trace:
I20260709 12:17:32.550671 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:32.548516 12490 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
W20260709 12:17:32.559755 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:32.560384 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:32.561527 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:32.550311 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:32.556685 12641 rpcz_store.cc:269] 0709 12:17:30.386619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.549506 (+2162887us) service_pool.cc:211] Skipping call since client already timed out
0709 12:17:32.549680 (+   174us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:17:32.563911 12495 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:32.547683 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:32.518133 25359 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:32.568140 25359 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:32.569159 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3958) took 2183 ms. Trace:
I20260709 12:17:32.569056 25359 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 40.16, Last appended by leader: 16, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.570191 25359 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:32.553063 25375 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.572291 25204 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:32.573056 25204 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:17:32.576982 25481 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260709 12:17:32.554584 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:32.569571 12644 rpcz_store.cc:276] 0709 12:17:30.386219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.860398 (+1474179us) service_pool.cc:224] Handling call
0709 12:17:31.881150 (+ 20752us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:31.905752 (+ 24602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.905796 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:31.915201 (+  9405us) log.cc:844] Serialized 52 byte log entry
0709 12:17:32.071743 (+156542us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:32.270263 (+198520us) log.cc:844] Serialized 14 byte log entry
0709 12:17:32.287923 (+ 17660us) log.cc:844] Serialized 14 byte log entry
0709 12:17:32.288266 (+   343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.288375 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:32.298281 (+  9906us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b180037b640. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:32.568906 (+270625us) raft_consensus.cc:1685] finished
0709 12:17:32.568932 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.569098 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58062,"prepare.queue_time_us":449653,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":22114,"raft.run_cpu_time_us":512,"raft.run_wall_time_us":28191,"spinlock_wait_cycles":5184896,"thread_start_us":206912,"threads_started":2,"wal-append.queue_time_us":260624}
I20260709 12:17:32.523027 12640 rpcz_store.cc:276] 0709 12:17:30.024474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.645612 (+1621138us) service_pool.cc:224] Handling call
0709 12:17:31.686090 (+ 40478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.878604 (+192514us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:17:32.079578 (+200974us) log.cc:844] Serialized 14 byte log entry
0709 12:17:32.175648 (+ 96070us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:32.175725 (+    77us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:17:32.176172 (+   447us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.519455 (+343283us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54003b06d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:32.519527 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.519703 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40351,"prepare.queue_time_us":399748,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":7641,"raft.queue_time_us":23034,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":7560,"spinlock_wait_cycles":528819968,"thread_start_us":254387,"threads_started":2,"wal-append.queue_time_us":215392}
I20260709 12:17:32.517057 25272 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.635457 25386 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.636088 25386 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.639504 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.640345 25212 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.641312 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.642655 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:32.644181 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89743.8us
I20260709 12:17:32.623409 25203 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:32.660739 25495 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:32.660926 25203 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:32.661432 25495 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:32.661548 25203 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:32.643474 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.662750 12497 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:32.663956 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:32.668751 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:32.670832 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5041) took 2121 ms. Trace:
I20260709 12:17:32.592602 25440 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.675806 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.676556 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:32.661942 25495 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:32.678296 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:32.679236 12566 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:17:32.679291 13253 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:32.679239 12490 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:32.688957 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.690090 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:32.630694 12495 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:32.694829 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:32.663018 25483 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
I20260709 12:17:32.700893 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:32.696005 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:32.671325 12567 rpcz_store.cc:276] 0709 12:17:30.549209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.664876 (+115667us) service_pool.cc:224] Handling call
0709 12:17:30.709489 (+ 44613us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:31.106144 (+396655us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:31.106176 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:17:31.116715 (+ 10539us) log.cc:844] Serialized 78 byte log entry
0709 12:17:31.843904 (+727189us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:17:32.047836 (+203932us) log.cc:844] Serialized 14 byte log entry
0709 12:17:32.163324 (+115488us) log.cc:844] Serialized 14 byte log entry
0709 12:17:32.300156 (+136832us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b180032b8a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:32.666477 (+366321us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b180032b8a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:17:32.670524 (+  4047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.670600 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:32.670639 (+    39us) raft_consensus.cc:1685] finished
0709 12:17:32.670650 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:32.670782 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19817,"prepare.queue_time_us":266507,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":16432,"raft.queue_time_us":116583,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":21974,"spinlock_wait_cycles":21747840,"thread_start_us":956891,"threads_started":3,"wal-append.queue_time_us":753916}
I20260709 12:17:32.713683 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:32.714395 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.715281 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.623235 25497 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:32.724519 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 525578us
I20260709 12:17:32.728498 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
W20260709 12:17:32.712507 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 234424us
W20260709 12:17:32.727308 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55492.7us
W20260709 12:17:32.720316 25396 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.835s	user 0.021s	sys 0.016s
I20260709 12:17:32.732009 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:32.710481 25462 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:32.734493 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1835) took 2047 ms. Trace:
I20260709 12:17:32.744897 13255 rpcz_store.cc:276] 0709 12:17:30.687010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.698521 (+ 11511us) service_pool.cc:224] Handling call
0709 12:17:30.699005 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.736809 (+1037804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:32.734188 (+997379us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:32.734427 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2254148992}
I20260709 12:17:32.745728 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:32.664611 25437 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.741346 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.668970 25450 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:32.712378 25272 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:32.751461 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:32.743747 25396 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.863s	user 0.022s	sys 0.016s
I20260709 12:17:32.753031 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:32.739948 25409 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.748s	user 0.015s	sys 0.001s
I20260709 12:17:32.763661 13253 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:32.731629 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:32.768302 13257 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:32.766803 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:32.732514 25480 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:32.708859 25440 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.761538 25375 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:32.794054 25375 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.795425 25440 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.797740 25375 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.803982 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.805044 12567 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 40 because replica has last-logged OpId of term: 38 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 22.
I20260709 12:17:32.805451 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:32.806016 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.806655 12567 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:17:32.807410 25482 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:32.808111 25482 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.808910 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:32.810668 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84178.4us
I20260709 12:17:32.754582 25228 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:32.776119 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:32.776489 13255 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:32.775178 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:32.778062 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 271745us
I20260709 12:17:32.753716 25386 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:32.789768 25498 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:32.790745 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:32.791684 25480 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:32.792744 25504 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:32.813608 13261 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:32.812808 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:32.812390 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:32.822271 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:32.953212 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:32.770932 25462 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:32.993423 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:32.950294 25456 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.509s	user 0.013s	sys 0.000s
I20260709 12:17:33.011123 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.011399 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.004649 12566 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:17:33.002363 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:32.815773 25229 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:32.816524 24701 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:32.734054 25497 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:32.817827 25228 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:33.055938 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53701.9us
I20260709 12:17:32.770007 25450 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.016991 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:32.970814 25366 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:32.949169 25498 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.043058 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:33.044106 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 231810us
I20260709 12:17:33.066865 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:33.027308 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.000970 25430 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:33.002969 25363 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:33.000685 25496 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:32.783221 25409 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:33.058172 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:33.017791 25467 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:33.041230 25464 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:33.043149 25377 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.025729 25212 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.078794 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:33.064465 25366 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.078125 13256 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:33.060813 25388 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:33.063977 25456 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:33.083904 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:33.061098 25228 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:33.074353 25496 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.085461 25484 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.277s
I20260709 12:17:33.085517 25467 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:33.088821 25492 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.000s
I20260709 12:17:33.015800 25375 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
W20260709 12:17:33.090714 25204 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.000s
I20260709 12:17:33.090811 25480 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.091980 25377 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.092967 25450 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.043571 24701 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:33.077991 25430 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.015993 25485 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.068778 25498 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:33.074487 25390 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
W20260709 12:17:33.138207 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 140187us
I20260709 12:17:33.102597 25411 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:33.104257 25366 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:33.417191 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.417433 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.418200 13261 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:33.421346 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.422852 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:33.425349 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.435496 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.436607 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:33.436707 13253 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:33.438151 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:33.408771 25113 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:33.122346 25456 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.095199 25482 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:33.177172 25501 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.380s	user 0.000s	sys 0.001s
I20260709 12:17:33.173825 25496 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:33.464634 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3948) took 3743 ms. Trace:
I20260709 12:17:33.078440 25363 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.467777 25363 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:33.469003 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1587) took 2432 ms (client timeout 2023 ms). Trace:
W20260709 12:17:33.471911 25203 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.771s	user 0.013s	sys 0.001s
W20260709 12:17:33.472612 25495 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.794s	user 0.013s	sys 0.000s
I20260709 12:17:33.472397 25203 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.474025 12491 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 40 because replica has last-logged OpId of term: 38 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 22.
W20260709 12:17:33.469405 13254 rpcz_store.cc:269] 0709 12:17:31.036137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.096640 (+ 60503us) service_pool.cc:224] Handling call
0709 12:17:33.468817 (+2372177us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:33.468930 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":975108480}
I20260709 12:17:33.476177 25485 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.450299 25456 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:33.477876 25203 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.305106 25486 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:33.306010 25365 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.290897 25480 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:33.306442 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107339us
I20260709 12:17:33.439062 25390 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:33.490367 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 473437us
I20260709 12:17:33.309975 25464 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 40.16, Last appended by leader: 16, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:33.315078 25505 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:17:33.316617 25352 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:33.321431 25499 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:17:33.330133 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.331305 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.331521 12490 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.21-37.21]
I20260709 12:17:33.473145 25495 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.337883 25506 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.344050 25467 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.345250 24701 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.186023 25492 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.085466 25462 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.193586 25204 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.367379 25437 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:33.369098 25359 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.798s	user 0.012s	sys 0.000s
W20260709 12:17:33.520257 25491 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.162s	user 0.010s	sys 0.007s
W20260709 12:17:33.371214 25272 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.001s
W20260709 12:17:33.298350 25475 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:17:33.359354 25497 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.388484 25379 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.375444 25377 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.172637 25212 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:33.416625 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.417235 25411 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:33.440548 25397 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.960s	user 0.012s	sys 0.001s
I20260709 12:17:33.445264 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.448105 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.442169 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:33.454421 25228 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.465155 12643 rpcz_store.cc:276] 0709 12:17:29.721148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.546253 (+1825105us) service_pool.cc:224] Handling call
0709 12:17:31.546704 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.741095 (+194391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:33.464275 (+1723180us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:33.464526 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":431584512}
I20260709 12:17:33.511871 25480 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.477490 12389 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 23 to 24, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 24 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:33.480384 25229 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:33.186091 25351 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.441766 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.553093 12564 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
W20260709 12:17:33.484553 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46731.7us
I20260709 12:17:33.301805 25409 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.444558 25113 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:33.191720 12461 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.691s (SENT)
I20260709 12:17:33.538209 25518 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:33.491458 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:33.558545 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 366878us
W20260709 12:17:33.558681 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67298.2us
I20260709 12:17:33.559516 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.508251 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.510031 25474 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:33.514928 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.560979 12495 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
W20260709 12:17:33.335820 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:33.561398 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.563702 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:33.564244 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1585) took 3196 ms. Trace:
I20260709 12:17:33.493939 25363 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:17:33.568511 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60355.7us
I20260709 12:17:33.547161 25411 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:33.516148 25486 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 32.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:33.539489 13255 raft_consensus.cc:2368] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:17:33.532619 25506 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.542906 12494 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
W20260709 12:17:33.542150 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.124s	sys 0.160s
I20260709 12:17:33.579173 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:33.579628 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.124s	sys 0.160s
W20260709 12:17:33.580037 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 804885us
I20260709 12:17:33.580724 13260 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:33.543867 13258 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
W20260709 12:17:33.545918 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.073s	sys 0.103s
I20260709 12:17:33.511714 25398 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:17:33.505034 25365 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.548121 13253 raft_consensus.cc:2376] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39. Re-sending same reply.
I20260709 12:17:33.476683 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:33.499269 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:33.517518 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.564978 13259 rpcz_store.cc:276] 0709 12:17:30.368180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:30.368344 (+   164us) service_pool.cc:224] Handling call
0709 12:17:30.383299 (+ 14955us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.813767 (+1430468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:33.563875 (+1750108us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:33.564164 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3052260096}
I20260709 12:17:33.564320 12643 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:33.564435 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:33.565135 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:33.174803 25388 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:33.571544 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.027s (SENT) [suppressed 7 similar messages]
I20260709 12:17:33.545207 25212 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.575650 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:33.541651 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:33.581264 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.073s	sys 0.103s
I20260709 12:17:33.569262 25379 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:33.592155 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 846445us
I20260709 12:17:33.592386 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.574803 25397 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:33.578537 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:33.574097 25482 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:33.587003 12640 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:33.576467 25351 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:33.568778 25363 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.581589 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:33.601259 12495 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:33.601963 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:33.602761 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:33.602854 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.561398 25345 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:33.573529 25113 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.594503 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:33.604676 25352 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.005s
I20260709 12:17:33.602018 25379 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.617204 25517 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:33.648392 25517 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.606935 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:33.649785 12495 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:33.614903 25430 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.594663 25397 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:33.650902 25397 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:33.590595 25359 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:33.651358 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.651871 25359 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:33.569680 25467 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:17:33.652622 25359 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:33.653093 25467 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:33.653509 25467 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:33.653750 25351 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:33.549471 25390 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 41.21, Last appended by leader: 21, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.615990 25363 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:33.654507 25467 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.655889 25467 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:33.621351 12570 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:17:33.624897 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:33.625355 25250 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:33.619446 25366 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.629119 25508 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:33.676692 25508 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.678073 25366 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.633824 25437 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:33.634541 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56099.8us
I20260709 12:17:33.577607 25518 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.636029 25352 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:17:33.596496 25212 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:33.637135 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5210) took 2169 ms. Trace:
I20260709 12:17:33.649197 25379 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.698026 12489 rpcz_store.cc:276] 0709 12:17:31.467467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.467709 (+   242us) service_pool.cc:224] Handling call
0709 12:17:31.503669 (+ 35960us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:31.859905 (+356236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:33.636897 (+1776992us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:33.637053 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":604886400}
I20260709 12:17:33.637724 25228 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.637539 25112 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.626682 25463 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:33.644620 25113 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:33.646420 25204 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.647089 25365 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.649825 12637 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:33.652078 12495 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:33.636085 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5300) took 1148 ms. Trace:
I20260709 12:17:33.626645 25506 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.663569 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:33.616947 12491 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:33.667749 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:33.669059 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:17:33.670027 25359 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.18, Last appended by leader: 18, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.670715 25388 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.672472 25495 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.673477 25492 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.677104 13259 raft_consensus.cc:2368] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46.
I20260709 12:17:33.632736 25411 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.684350 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.636261 25338 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:33.597787 25515 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:33.693336 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:33.694170 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91430.6us
I20260709 12:17:33.668083 25250 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:33.710438 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 118392us
I20260709 12:17:33.746408 12566 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 46
I20260709 12:17:33.715004 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.747076 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.717283 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.646503 25345 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.720273 25377 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.751173 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.723703 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.677482 25496 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.691988 25437 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:33.734014 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114922us
I20260709 12:17:33.709195 25112 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.734439 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:33.707175 25463 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:33.737457 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53261.6us
I20260709 12:17:33.766762 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:33.767200 25112 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.767407 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:33.740165 25503 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:33.747252 12489 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:33.716002 12568 rpcz_store.cc:276] 0709 12:17:32.487346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.487618 (+   272us) service_pool.cc:224] Handling call
0709 12:17:32.487905 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:32.498324 (+ 10419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:33.635825 (+1137501us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:33.636021 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21640960}
I20260709 12:17:33.749274 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:33.752681 12495 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:33.752948 25365 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:33.758445 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:33.786667 25504 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:33.681591 25456 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.787302 25504 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:33.788259 12490 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:33.717793 25517 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.756776 25411 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:33.788342 25504 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.753180 25359 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:33.762449 25437 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.810756 25058 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:33.811362 25525 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:17:33.812000 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4908) took 1903 ms (client timeout 1639 ms). Trace:
W20260709 12:17:33.812551 12646 rpcz_store.cc:269] 0709 12:17:31.908266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.928107 (+ 19841us) service_pool.cc:224] Handling call
0709 12:17:33.811713 (+1883606us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:33.811894 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515162496}
I20260709 12:17:33.739102 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:33.772078 25497 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:17:33.817021 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.776450 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:33.818063 25089 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:33.778154 13257 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:33.818257 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.818806 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.819031 13257 raft_consensus.cc:606] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:33.819428 12489 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:33.761215 25496 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.819728 25089 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:33.820288 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.805059 12566 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:17:33.805009 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46670.5us
I20260709 12:17:33.785408 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:33.753607 25492 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:33.822218 25379 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:33.809046 25345 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.822253 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:33.823951 25411 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:33.824609 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:33.825497 25411 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:33.789764 25508 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.758111 25377 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.796073 25250 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.827371 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:33.827850 25250 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:33.818356 12570 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:17:33.819577 13257 raft_consensus.cc:614] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:33.747850 25515 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:33.810281 25365 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:33.831097 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.829560 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:33.830317 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.832186 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:33.811383 25058 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:33.845813 25492 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:33.845541 25058 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.846288 25492 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:33.839960 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54635.4us
I20260709 12:17:33.846652 25058 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:33.847268 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:33.850503 25351 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:33.851018 25351 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.851368 25351 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:17:33.852914 12522 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (10 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:17:33.853996 25345 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:33.854476 25345 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.854867 25345 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:33.855556 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:33.783689 25503 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.854974 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:33.862174 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86361.4us
W20260709 12:17:33.858244 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1841) took 1995 ms (client timeout 1836 ms). Trace:
W20260709 12:17:33.862892 13256 rpcz_store.cc:269] 0709 12:17:31.862741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:31.862919 (+   178us) service_pool.cc:224] Handling call
0709 12:17:33.857969 (+1995050us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540056a9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:33.858152 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2703054208}
W20260709 12:17:33.864744 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:33.859390 12489 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:33.790944 25112 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:33.860442 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.754560 25498 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.870123 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:33.871803 25411 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:33.872248 25411 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.872587 25411 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:33.874279 25523 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:33.846671 25492 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:33.839954 25089 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:33.844545 25235 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:33.766790 25463 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.871316 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:33.840926 25515 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:33.860661 25525 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:33.863583 25503 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:33.863314 25480 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.889765 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.882247 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.897531 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:33.897838 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:33.883100 25089 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:33.885510 25235 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:33.849334 25397 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.901166 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:33.906235 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.907382 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:33.892874 25525 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:33.882653 25363 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.899289 25229 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:33.921515 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125363us
I20260709 12:17:33.922535 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:33.903404 25089 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:33.941581 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86663.2us
I20260709 12:17:33.956960 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:33.978791 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:33.917721 25456 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:33.924935 25078 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.213s	user 0.058s	sys 0.068s
I20260709 12:17:33.963338 25272 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:33.921108 25390 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:33.917462 25525 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:33.965116 25363 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:33.921233 25480 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.071s
I20260709 12:17:33.992731 25498 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:33.994043 25359 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:33.995375 25534 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:33.925921 25229 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:33.988448 25078 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.277s	user 0.059s	sys 0.069s
I20260709 12:17:33.993103 25212 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:33.975502 25379 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:17:34.012166 25212 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:34.005839 25078 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:34.009397 12493 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
W20260709 12:17:34.001446 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 145961us
I20260709 12:17:34.007149 25229 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:34.024158 25078 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:34.028906 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:34.029804 12566 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
W20260709 12:17:34.034052 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52770us
I20260709 12:17:34.034086 12570 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:34.034840 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:34.037230 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:34.037608 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:34.038460 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:33.979647 25365 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.043s
I20260709 12:17:34.040995 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:34.042706 12568 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 31.
I20260709 12:17:34.032600 25467 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:34.055317 25397 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:34.036912 25250 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.000694 25272 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:33.974195 25235 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.041726 25359 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:34.077095 25467 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:34.048328 25390 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:34.120613 25397 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:34.132483 25390 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.113770 25359 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:34.133669 25503 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.149178 25518 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:34.156936 25467 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.159475 25467 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:34.163010 25411 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:34.163910 25272 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:34.139297 25492 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.113649 25363 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.028s
I20260709 12:17:34.166584 25363 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:34.167239 25363 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:34.168831 25492 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:34.120549 25536 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.170804 12495 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:34.112716 25250 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.121753 25538 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.173724 12491 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33.
I20260709 12:17:34.139067 25089 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:34.184022 25480 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:34.142194 25359 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:34.198881 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:34.199668 12564 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:34.184930 25480 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.193075 25345 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.168753 25363 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:34.209975 25539 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:17:34.180697 25351 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.186952 25538 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:34.170149 25536 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:34.456363 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:34.460908 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:34.466727 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:34.468014 13262 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:34.466848 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1982) took 1820 ms (client timeout 1534 ms). Trace:
W20260709 12:17:34.471493 12492 rpcz_store.cc:269] 0709 12:17:32.639761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.643253 (+  3492us) service_pool.cc:224] Handling call
0709 12:17:34.460350 (+1817097us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:34.460500 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":939803008}
I20260709 12:17:34.473059 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:34.474186 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:34.474567 25212 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.482645 25212 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:34.475133 25397 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.475548 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:34.486006 25411 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:34.486785 13257 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:34.486104 25397 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:34.453073 25363 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:34.490793 25397 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:34.487927 25250 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:34.458449 25503 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:34.500115 25089 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:34.542893 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:34.544566 12492 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:34.546756 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:34.547883 12565 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:34.552471 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5061) took 1141 ms. Trace:
I20260709 12:17:34.554256 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:34.556775 12488 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:17:34.560518 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.012s	sys 0.001s
W20260709 12:17:34.562294 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.012s	sys 0.002s
W20260709 12:17:34.543936 25515 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.005s
W20260709 12:17:34.563045 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 525860us
W20260709 12:17:34.563264 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:34.563902 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:34.553782 12565 rpcz_store.cc:276] 0709 12:17:33.411180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.413957 (+  2777us) service_pool.cc:224] Handling call
0709 12:17:34.546563 (+1132606us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:34.552386 (+  5823us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13798144}
I20260709 12:17:34.560195 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:34.565650 12492 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:34.569005 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:34.572700 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:34.575665 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:34.573794 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:34.578697 12492 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:34.583849 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:34.573601 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:34.644979 12488 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:34.612170 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:34.611579 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:34.699573 12497 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:34.593118 12564 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
W20260709 12:17:34.746976 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 134938us
I20260709 12:17:34.692725 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5236) took 1355 ms. Trace:
I20260709 12:17:34.641219 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:34.615734 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46806.8us
W20260709 12:17:34.672744 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.001s	sys 0.018s
W20260709 12:17:34.750800 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.005s	sys 0.031s
W20260709 12:17:34.698985 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1973) took 2575 ms (client timeout 2146 ms). Trace:
I20260709 12:17:34.701751 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1593) took 1285 ms. Trace:
I20260709 12:17:34.751446 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:34.752224 12496 rpcz_store.cc:269] 0709 12:17:32.124022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.124293 (+   271us) service_pool.cc:224] Handling call
0709 12:17:34.698530 (+2574237us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400190d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:34.698818 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":776348032}
I20260709 12:17:34.749121 12490 rpcz_store.cc:276] 0709 12:17:33.337196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.515080 (+177884us) service_pool.cc:224] Handling call
0709 12:17:34.692420 (+1177340us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:34.692633 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464909056}
I20260709 12:17:34.755283 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:34.655730 25398 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.007s	sys 0.011s
I20260709 12:17:34.754635 12646 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:34.588153 25397 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:34.756270 12488 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:34.780166 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:34.780929 12488 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:34.588284 25366 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:34.783569 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:34.767083 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:34.767235 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:34.752440 13260 rpcz_store.cc:276] 0709 12:17:33.416154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.416313 (+   159us) service_pool.cc:224] Handling call
0709 12:17:34.701463 (+1285150us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:34.701662 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68974464}
I20260709 12:17:34.767721 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:34.575047 25515 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:34.591002 25411 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:34.628316 25379 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:34.772710 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:34.771283 25398 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.016s	sys 0.017s
I20260709 12:17:34.888161 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:34.907032 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:34.756011 12492 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:34.593595 25325 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:34.775053 12646 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:17:34.611953 25189 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.643218 25430 raft_consensus.cc:2764] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election decision vote started in defunct term 35: won
W20260709 12:17:34.778434 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 204893us
I20260709 12:17:34.665657 25480 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:34.778887 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 211288us
W20260709 12:17:34.767995 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 310317us
I20260709 12:17:34.631745 25492 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:34.769474 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.002425 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:35.002594 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.003854 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.004853 12641 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:34.870288 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.006326 12496 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:34.872941 25542 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.422s	user 0.007s	sys 0.000s
I20260709 12:17:34.590618 25250 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.177s
I20260709 12:17:34.873979 25552 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.874378 25551 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:34.874570 25366 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:17:34.875129 25359 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.669s	user 0.009s	sys 0.004s
I20260709 12:17:34.875413 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.039259 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3903) took 1407 ms. Trace:
I20260709 12:17:34.872951 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:34.755003 25365 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:34.908913 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987) took 1039 ms. Trace:
I20260709 12:17:34.771747 25536 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:34.927440 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:34.929144 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.045595 12492 rpcz_store.cc:276] 0709 12:17:33.869847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.573331 (+703484us) service_pool.cc:224] Handling call
0709 12:17:34.908829 (+335498us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.046587 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:17:34.899830 25226 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 12:17:34.947347 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:34.972175 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.050576 12564 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:35.050791 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:35.051225 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:34.877152 25351 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.055385 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:34.873471 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.007706 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:34.758924 25309 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.146s
I20260709 12:17:35.010866 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991) took 1014 ms. Trace:
I20260709 12:17:35.011966 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.057224 12496 rpcz_store.cc:276] 0709 12:17:33.996947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.773509 (+776562us) service_pool.cc:224] Handling call
0709 12:17:35.010782 (+237273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.056685 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.059702 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3907) took 1427 ms. Trace:
I20260709 12:17:35.060151 12492 rpcz_store.cc:276] 0709 12:17:33.632585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.056378 (+1423793us) service_pool.cc:224] Handling call
0709 12:17:35.059611 (+  3233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.022670 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:35.061446 13257 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:35.016137 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:35.027938 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.062865 12565 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:35.028915 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:34.891080 25411 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.039778 12488 rpcz_store.cc:276] 0709 12:17:33.631613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.875085 (+1243472us) service_pool.cc:224] Handling call
0709 12:17:35.039167 (+164082us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:35.046698 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119355us
I20260709 12:17:34.878137 25397 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.006088 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:35.056876 12490 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:35.076808 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:35.078388 12565 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 34 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:17:35.027561 25366 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.087051 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.087587 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3901) took 1456 ms. Trace:
I20260709 12:17:35.088217 12572 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:35.088291 12490 rpcz_store.cc:276] 0709 12:17:33.631164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.873153 (+1241989us) service_pool.cc:224] Handling call
0709 12:17:35.087482 (+214329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.063544 12568 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:35.057701 12641 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34. Re-sending same reply.
I20260709 12:17:35.084255 12570 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:17:35.080852 25365 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:34.933693 25189 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.092171 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:35.097971 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.098397 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:35.094897 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 347125us
I20260709 12:17:35.100172 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.102586 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.104466 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:35.019982 25492 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.105782 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:35.109277 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.111109 12644 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:35.112206 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.113029 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.094069 25366 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:35.117910 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.057139 25430 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:35.114530 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.115541 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.029794 25552 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.100194 25390 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:35.117343 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.037s	sys 0.043s
W20260709 12:17:35.119830 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 336389us
W20260709 12:17:35.140002 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.052599 25359 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.114344 25482 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:35.114826 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.045040 25310 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.054s
I20260709 12:17:35.125217 12641 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33.
I20260709 12:17:35.125572 12570 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:35.138242 25363 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.637s	user 0.014s	sys 0.000s
I20260709 12:17:35.042032 25551 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:35.146041 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.037s	sys 0.043s
I20260709 12:17:35.120446 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:35.114780 25549 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.144212 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:35.162937 12643 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.118292 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:35.125869 25430 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:35.162909 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.165688 12572 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:35.166265 12641 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.163007 12568 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:35.156081 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.167302 12644 raft_consensus.cc:2376] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34. Re-sending same reply.
I20260709 12:17:35.167917 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:35.168677 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.170396 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.173662 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.164824 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.151458 25482 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:35.175632 25189 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:35.176949 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5070) took 1146 ms. Trace:
I20260709 12:17:35.137996 25536 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:17:35.150143 25550 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:17:35.172012 12568 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:35.169428 12644 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:17:35.171437 25430 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:35.169528 25363 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:35.187723 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.189114 12566 rpcz_store.cc:276] 0709 12:17:34.030909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.031105 (+   196us) service_pool.cc:224] Handling call
0709 12:17:34.031503 (+   398us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:34.039519 (+  8016us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:34.039545 (+    26us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:17:34.054028 (+ 14483us) log.cc:844] Serialized 78 byte log entry
0709 12:17:34.131712 (+ 77684us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:34.132289 (+   577us) log.cc:844] Serialized 14 byte log entry
0709 12:17:34.133045 (+   756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:34.133142 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:35.176683 (+1043541us) raft_consensus.cc:1685] finished
0709 12:17:35.176707 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:35.176880 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3834,"prepare.queue_time_us":402,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":4157,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":12114,"spinlock_wait_cycles":181376,"thread_start_us":9492,"threads_started":1,"wal-append.queue_time_us":81064}
W20260709 12:17:35.164304 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46099.6us
I20260709 12:17:35.170100 25351 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:17:35.211556 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105833us
I20260709 12:17:35.184770 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.218272 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:35.212651 13257 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:35.218868 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:35.219944 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52139.2us
I20260709 12:17:35.220355 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.187975 25482 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:35.181504 25558 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:35.227572 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.228838 12566 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:17:35.189862 25550 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:35.223088 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.222330 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.228755 12644 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:35.227561 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:35.231112 12570 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:35.218995 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:35.231557 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.232704 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.232923 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.236443 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:35.168855 25549 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.253959 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3909) took 1620 ms. Trace:
I20260709 12:17:35.246533 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:35.247921 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1595) took 1831 ms (client timeout 1548 ms). Trace:
W20260709 12:17:35.255585 13255 rpcz_store.cc:269] 0709 12:17:33.416784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.421002 (+  4218us) service_pool.cc:224] Handling call
0709 12:17:35.247611 (+1826609us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.247821 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269380864}
I20260709 12:17:35.222239 25351 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.254684 12496 rpcz_store.cc:276] 0709 12:17:33.633153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.211048 (+1577895us) service_pool.cc:224] Handling call
0709 12:17:35.253847 (+ 42799us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.258486 12644 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.261186 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.224788 25363 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:35.226702 25515 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:35.238103 25536 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:35.260689 25351 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:35.272970 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.275154 25552 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:35.180584 25365 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.262110 25490 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:17:35.276547 25553 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:17:35.282253 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.284701 12460 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:35.249619 25565 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:35.238204 25550 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:35.248880 25484 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:35.287596 12496 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:35.291594 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.300144 12460 leader_election.cc:403] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20260709 12:17:35.303838 12644 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.314502 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5076) took 1266 ms. Trace:
I20260709 12:17:35.288941 25536 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.315022 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5226) took 1991 ms. Trace:
I20260709 12:17:35.316398 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:35.295791 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:35.316628 12567 rpcz_store.cc:276] 0709 12:17:34.048325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.608492 (+560167us) service_pool.cc:224] Handling call
0709 12:17:34.608963 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:34.609225 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:17:34.609566 (+   341us) log.cc:844] Serialized 14 byte log entry
0709 12:17:35.314016 (+704450us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:35.314090 (+    74us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:35.314143 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.314234 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:35.314411 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":180,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":18769,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":703929,"threads_started":2,"wal-append.queue_time_us":571506}
I20260709 12:17:35.322856 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.323887 12566 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:35.288233 25397 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.320966 12494 rpcz_store.cc:276] 0709 12:17:33.323499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.324653 (+  1154us) service_pool.cc:224] Handling call
0709 12:17:33.324919 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:33.579589 (+254670us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.314743 (+1735154us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:35.314927 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":500873984}
I20260709 12:17:35.330533 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.331144 12494 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:35.305052 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.326673 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.332073 12644 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.337141 12570 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:35.307523 25359 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.338017 12566 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
W20260709 12:17:35.333827 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67829.7us
I20260709 12:17:35.341006 25551 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:35.344496 25430 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.314301 25493 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:35.347236 25366 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.348806 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.352281 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:35.352942 25554 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.187s	user 0.001s	sys 0.001s
I20260709 12:17:35.315129 25490 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:35.354534 25236 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:35.354545 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:35.355144 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5307) took 2534 ms (client timeout 1735 ms). Trace:
W20260709 12:17:35.415899 12571 rpcz_store.cc:269] 0709 12:17:32.820774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.971383 (+150609us) service_pool.cc:224] Handling call
0709 12:17:35.354759 (+2383376us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.355046 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606673792}
I20260709 12:17:35.356725 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5229) took 2032 ms. Trace:
I20260709 12:17:35.358819 12643 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:35.319150 25492 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.325230 25480 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:35.393121 12496 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:35.359364 25363 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.305974 25484 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:35.360550 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.339748 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:35.364605 12494 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
W20260709 12:17:35.366559 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.327769 25536 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:35.371536 12570 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:35.414327 25127 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.378235 25379 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:35.383741 25351 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.384481 25463 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:35.385512 25250 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:35.386646 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1989) took 1492 ms. Trace:
I20260709 12:17:35.369618 24701 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:35.390658 12566 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:35.448055 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.431322 25484 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:35.394512 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:35.417321 12497 rpcz_store.cc:276] 0709 12:17:33.324499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.325119 (+   620us) service_pool.cc:224] Handling call
0709 12:17:34.699271 (+1374152us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:34.748825 (+ 49554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.356520 (+607695us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400190d50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.356663 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3159443840}
I20260709 12:17:35.341017 12641 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:35.390667 25495 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.428278 12571 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
W20260709 12:17:35.432166 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.432950 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:35.375293 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1985) took 1744 ms (client timeout 1856 ms). Trace:
W20260709 12:17:35.436199 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96543.7us
I20260709 12:17:35.423893 25558 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:35.436465 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.393195 25496 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.434201 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:35.437390 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143122us
I20260709 12:17:35.436486 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.423014 25377 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.444717 12495 rpcz_store.cc:276] 0709 12:17:33.894496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.937571 (+ 43075us) service_pool.cc:224] Handling call
0709 12:17:35.386369 (+1448798us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.386564 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":459381632}
I20260709 12:17:35.433035 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.444828 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.439637 25492 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:35.579361 25492 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:35.579943 25492 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:35.582867 25366 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:35.444410 25250 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:35.585081 25250 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.29, Last appended by leader: 29, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.586659 25250 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:35.442463 25127 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.591954 25127 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:35.616909 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:35.647562 25430 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:35.647248 25482 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.442658 25379 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:35.654956 25463 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.655090 25379 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.27, Last appended by leader: 27, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.655787 25463 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.657039 25379 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:35.658080 25490 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.665091 25482 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.672952 12567 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:17:35.674090 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:35.675087 12491 rpcz_store.cc:269] 0709 12:17:33.630676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.766424 (+135748us) service_pool.cc:224] Handling call
0709 12:17:35.374944 (+1608520us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.375196 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933773568}
I20260709 12:17:35.676558 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.677132 12491 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:35.677453 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.680177 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.681118 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.681991 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:35.682852 25525 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.659s	user 0.015s	sys 0.000s
W20260709 12:17:35.683828 25568 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 12:17:35.683543 25525 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.684860 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.685276 12639 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.685490 12495 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:35.686722 25525 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:35.686882 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.688408 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:17:35.688437 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.681854 25562 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:35.674417 25467 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.514s	user 0.016s	sys 0.000s
I20260709 12:17:35.449563 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.689477 25562 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.691211 12566 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 28.
I20260709 12:17:35.693918 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:17:35.636351 25554 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
I20260709 12:17:35.694723 25567 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
W20260709 12:17:35.695977 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 301529us
I20260709 12:17:35.686993 25351 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:35.701928 12497 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:17:35.696062 25567 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:35.678319 12488 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 38 because replica has last-logged OpId of term: 37 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 18.
I20260709 12:17:35.704650 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3910) took 2071 ms. Trace:
I20260709 12:17:35.704416 25567 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.683439 25496 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.709348 25496 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:35.711742 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.713189 12644 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:35.723618 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.724825 12571 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:35.725602 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:35.727566 25226 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.134s	user 0.004s	sys 0.000s
I20260709 12:17:35.730268 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:35.689409 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.732664 25558 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.520990 25564 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:35.735296 25127 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.735905 25127 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.735879 25564 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.674775 12567 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:35.705226 12488 rpcz_store.cc:276] 0709 12:17:33.633389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.433796 (+1800407us) service_pool.cc:224] Handling call
0709 12:17:35.704574 (+270778us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:35.742583 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 125836us
I20260709 12:17:35.689401 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:35.724103 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.682811 12641 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
W20260709 12:17:35.748091 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.746307 12646 raft_consensus.cc:2376] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32. Re-sending same reply.
I20260709 12:17:35.746524 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1872) took 2183 ms. Trace:
I20260709 12:17:35.746348 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:17:35.749892 12637 rpcz_store.cc:276] 0709 12:17:33.563213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.563346 (+   133us) service_pool.cc:224] Handling call
0709 12:17:35.745780 (+2182434us) spinlock_profiling.cc:243] Waited 67.7 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.746465 (+   685us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155627776}
I20260709 12:17:35.751631 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.753055 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.753810 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5237) took 2416 ms. Trace:
I20260709 12:17:35.750172 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.754328 12497 rpcz_store.cc:276] 0709 12:17:33.337505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.700299 (+2362794us) service_pool.cc:224] Handling call
0709 12:17:35.701260 (+   961us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:35.703308 (+  2048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.753730 (+ 50422us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.755558 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:35.756807 12536 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.027s (SENT)
W20260709 12:17:35.757108 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58527.1us
I20260709 12:17:35.757427 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:35.758679 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983) took 2128 ms. Trace:
I20260709 12:17:35.759279 12488 rpcz_store.cc:276] 0709 12:17:33.630099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.757346 (+2127247us) service_pool.cc:224] Handling call
0709 12:17:35.758082 (+   736us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.758278 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:35.758300 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:35.758361 (+    61us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:35.758393 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.758448 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:35.758603 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.761027 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.765257 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.695240 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:35.774435 12495 raft_consensus.cc:606] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:35.774923 12495 raft_consensus.cc:614] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:35.774578 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:35.775429 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 81) took 1861 ms. Trace:
I20260709 12:17:35.777518 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:35.778738 13255 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:35.775756 12495 rpcz_store.cc:276] 0709 12:17:33.913910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.695096 (+1781186us) service_pool.cc:224] Handling call
0709 12:17:35.775369 (+ 80273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.780479 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.781239 12641 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:35.777448 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.782750 12497 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:35.783293 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.786233 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:35.787158 13260 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:35.786962 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:35.788038 13253 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:35.788331 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:35.789237 13256 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:35.784648 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:35.792826 12538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.004s	sys 0.016s
W20260709 12:17:35.794646 12538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.004s	sys 0.016s
I20260709 12:17:35.793238 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:35.795210 12488 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:35.796407 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:35.796407 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.797291 12488 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:17:35.795500 13257 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:35.802335 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.803290 12495 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:35.796242 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:35.756352 25430 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:35.813824 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:35.814051 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:35.819294 12563 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:35.806105 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:35.807299 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.820209 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.804663 12488 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:35.808370 13258 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
W20260709 12:17:35.808069 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:35.778440 25390 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.634s	user 0.011s	sys 0.001s
I20260709 12:17:35.797446 12637 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 28.
I20260709 12:17:35.810652 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:35.780367 25351 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:35.838892 12491 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:35.840062 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992) took 1842 ms. Trace:
I20260709 12:17:35.840643 12491 rpcz_store.cc:276] 0709 12:17:33.997803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.676255 (+1678452us) service_pool.cc:224] Handling call
0709 12:17:35.839954 (+163699us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.840952 12462 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:35.821014 12641 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:35.841456 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:35.803500 13253 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:35.834589 25482 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:35.823060 12488 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 44 index: 29. (index mismatch)
I20260709 12:17:35.849852 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1993) took 1791 ms. Trace:
W20260709 12:17:35.819429 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5310) took 2997 ms (client timeout 1626 ms). Trace:
I20260709 12:17:35.841598 12462 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:35.831050 12644 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:35.844264 13257 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I20260709 12:17:35.831627 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.851869 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.852142 12637 raft_consensus.cc:2376] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35. Re-sending same reply.
I20260709 12:17:35.853024 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:35.850472 12569 rpcz_store.cc:269] 0709 12:17:32.823029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:32.948850 (+125821us) service_pool.cc:224] Handling call
0709 12:17:35.811996 (+2863146us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:35.819325 (+  7329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1562472320}
I20260709 12:17:35.850315 12488 rpcz_store.cc:276] 0709 12:17:34.058340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.822531 (+1764191us) service_pool.cc:224] Handling call
0709 12:17:35.822768 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.849524 (+ 26756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.849794 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.853020 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.836529 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:17:35.837587 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.855446 12637 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:35.855887 12461 connection.cc:256] RPC call timeout handler was delayed by -1.2091584779998357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260709 12:17:35.856313 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1974584779998168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
W20260709 12:17:35.856838 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1335584779999408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260709 12:17:35.857321 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1110584779999044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
I20260709 12:17:35.837937 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.858403 12565 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:35.842746 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:35.824154 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:35.860150 12641 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.858700 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.793825 25467 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:35.693938 25495 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.886703 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:35.911782 12491 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:35.861078 12495 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:35.869786 12497 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: [38.24-38.24]
I20260709 12:17:35.822088 25552 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:35.830799 25356 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:17:35.861583 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:35.922916 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61438.6us
I20260709 12:17:35.847196 25212 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:35.894510 13218 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.542s (SENT)
I20260709 12:17:35.952606 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:35.859728 25236 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:35.782408 25550 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.840247 25564 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:35.914701 25525 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:35.844115 25127 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:35.949451 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:35.975543 12569 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
I20260709 12:17:35.949579 12491 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:17:35.976701 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5246) took 1043 ms. Trace:
I20260709 12:17:35.976955 12491 rpcz_store.cc:276] 0709 12:17:34.933111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.913517 (+980406us) service_pool.cc:224] Handling call
0709 12:17:35.949264 (+ 35747us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:35.976474 (+ 27210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.976661 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:35.847661 25482 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:35.849879 25390 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:35.848752 25480 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:35.955713 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61265.2us
I20260709 12:17:35.908675 25457 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:17:35.865828 25365 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:35.957355 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:35.865970 25504 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:35.923693 25203 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.055s
I20260709 12:17:35.868703 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:17:35.963125 12609 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:17:35.970595 12490 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.29-43.29]
I20260709 12:17:36.022385 12609 leader_election.cc:403] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:35.924280 25430 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:17:35.973403 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:35.865944 25009 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.101s
I20260709 12:17:35.860373 25351 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:35.987210 12564 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:35.948904 25359 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:35.966264 25496 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:35.860832 25574 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:35.961670 25463 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:36.018299 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 176995us
I20260709 12:17:35.922376 25112 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.057s
I20260709 12:17:35.991398 12569 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:36.075114 12642 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:17:36.026124 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1603) took 2408 ms (client timeout 2227 ms). Trace:
I20260709 12:17:36.025272 12641 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:35.979298 25482 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:35.983523 25127 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:36.051513 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5251) took 1013 ms. Trace:
I20260709 12:17:36.082101 12491 rpcz_store.cc:276] 0709 12:17:35.044208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.993221 (+949013us) service_pool.cc:224] Handling call
0709 12:17:36.050647 (+ 57426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.050953 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:36.050995 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:36.051061 (+    66us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:36.051127 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.051214 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.051413 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.084419 25009 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:36.084697 12491 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:17:36.086468 25351 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:36.055372 25203 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 39.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
W20260709 12:17:36.091434 25580 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:17:36.091656 25127 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
W20260709 12:17:36.092085 25492 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.512s	user 0.007s	sys 0.003s
I20260709 12:17:36.092180 25127 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.092765 25492 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:36.067214 12610 connection.cc:256] RPC call timeout handler was delayed by -1.1196601460001148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
I20260709 12:17:36.070117 12490 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 16. Preceding OpId from leader: term: 41 index: 17. (index mismatch)
I20260709 12:17:36.024712 25430 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:36.023211 25525 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:36.076035 13259 rpcz_store.cc:269] 0709 12:17:33.617677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.617920 (+   243us) service_pool.cc:224] Handling call
0709 12:17:36.018614 (+2400694us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b54006e5354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.026037 (+  7423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37097344}
W20260709 12:17:36.047256 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84294.9us
W20260709 12:17:36.101014 12536 leader_election.cc:336] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.161s (SENT)
I20260709 12:17:36.121969 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:36.048430 25504 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:36.118628 12610 connection.cc:256] RPC call timeout handler was delayed by -1.0206601459999547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
I20260709 12:17:36.095709 25359 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:36.093465 25127 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.128974 12569 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:36.129251 25359 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.129944 25127 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:36.126505 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.007586 25365 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:36.093479 25536 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.000s
W20260709 12:17:36.135643 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.094213 25463 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.073072 25189 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:17:36.118258 25009 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.095299 25550 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.102300 25578 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.129812 25390 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:36.146358 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.152060 12641 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:36.127583 25430 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:36.138585 25356 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.128865 25496 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.012s
W20260709 12:17:36.154474 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.142957 25379 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.096367 25574 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:35.987622 25480 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:36.140247 25536 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.144623 25463 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:36.185259 25581 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:17:36.175405 25379 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.168008 25189 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:36.378439 25189 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.379755 25365 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.381145 25379 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:36.382513 25235 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:36.386341 25385 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.386333 25294 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.002s
I20260709 12:17:36.282322 12491 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:36.390427 25078 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:36.390712 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3927) took 1005 ms. Trace:
I20260709 12:17:36.391428 12491 rpcz_store.cc:276] 0709 12:17:35.385169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.118933 (+733764us) service_pool.cc:224] Handling call
0709 12:17:36.281520 (+162587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.390316 (+108796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.390630 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.391543 25078 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:36.394626 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:36.290879 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 163248us
I20260709 12:17:36.292204 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5337) took 1009 ms. Trace:
I20260709 12:17:36.302544 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1984) took 2672 ms. Trace:
I20260709 12:17:36.296298 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1990) took 2380 ms. Trace:
I20260709 12:17:36.305657 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5326) took 1127 ms. Trace:
W20260709 12:17:36.313795 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5231) took 2984 ms (client timeout 1801 ms). Trace:
I20260709 12:17:36.319923 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5060) took 2908 ms. Trace:
I20260709 12:17:36.325541 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5314) took 1267 ms. Trace:
I20260709 12:17:36.332459 12492 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 43 index: 22. (index mismatch)
I20260709 12:17:36.123726 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:36.243095 25536 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:36.346527 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.345697 12389 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 24 to 33. New cstate: current_term: 33 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.350005 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:36.351187 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.398833 12563 rpcz_store.cc:276] 0709 12:17:33.410804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.413805 (+  3001us) service_pool.cc:224] Handling call
0709 12:17:35.812198 (+2398393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.819985 (+  7787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.319009 (+499024us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b54001b26d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.319552 (+   543us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5490263424}
I20260709 12:17:36.352106 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:36.353971 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.354785 12646 raft_consensus.cc:2368] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 32. Current term is 33.
I20260709 12:17:36.357019 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.359177 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4923) took 1588 ms. Trace:
W20260709 12:17:36.362092 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4920) took 1595 ms (client timeout 1943 ms). Trace:
W20260709 12:17:36.195539 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.686s (SENT)
I20260709 12:17:36.367993 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:36.372907 25250 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.786s	user 0.014s	sys 0.001s
I20260709 12:17:36.178511 25009 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.179863 25351 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.375692 25359 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.397429 12494 rpcz_store.cc:276] 0709 12:17:33.630418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.758200 (+2127782us) service_pool.cc:224] Handling call
0709 12:17:35.758699 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.758993 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:35.791622 (+ 32629us) log.cc:844] Serialized 14 byte log entry
0709 12:17:36.300398 (+508776us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b1800161da0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:17:36.301122 (+   724us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:36.301251 (+   129us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:36.301444 (+   193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.301596 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.302314 (+   718us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":34551,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":94081,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":310,"spinlock_wait_cycles":16679168,"thread_start_us":61659,"threads_started":1,"wal-append.queue_time_us":178468}
I20260709 12:17:36.395426 25325 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:36.386847 25356 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.437574 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.312111 12565 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:36.438529 25009 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:36.458417 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.460516 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:36.236449 25492 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.462605 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.463177 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.398978 12572 rpcz_store.cc:276] 0709 12:17:35.058453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.115489 (+ 57036us) service_pool.cc:224] Handling call
0709 12:17:35.161790 (+ 46301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.166227 (+  4437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.324416 (+1158189us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54003e1050. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.325019 (+   603us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105531264}
I20260709 12:17:36.183043 25504 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.465068 13253 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:36.465188 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:36.430341 12639 rpcz_store.cc:276] 0709 12:17:34.770618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.770839 (+   221us) service_pool.cc:224] Handling call
0709 12:17:36.358477 (+1587638us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.358857 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1482134912}
I20260709 12:17:36.467849 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.468238 13256 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:17:36.468809 12639 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 43 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 28.
W20260709 12:17:36.470096 12645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4930) took 1559 ms (client timeout 1623 ms). Trace:
W20260709 12:17:36.470487 12645 rpcz_store.cc:269] 0709 12:17:34.914356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.351372 (+1437016us) service_pool.cc:224] Handling call
0709 12:17:36.470013 (+118641us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.191046 25525 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.471603 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.398139 12568 rpcz_store.cc:276] 0709 12:17:35.178031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.230699 (+ 52668us) service_pool.cc:224] Handling call
0709 12:17:36.304940 (+1074241us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.305300 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993279232}
W20260709 12:17:36.235846 25583 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:17:36.155396 25578 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:36.222620 25577 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:17:36.343532 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.477967 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3919) took 2761 ms. Trace:
I20260709 12:17:36.478469 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.438617 25236 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:36.479701 12637 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:36.480271 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3970) took 1104 ms. Trace:
I20260709 12:17:36.480351 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.480695 12637 rpcz_store.cc:276] 0709 12:17:35.375475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.478118 (+1102643us) service_pool.cc:224] Handling call
0709 12:17:36.480177 (+  2059us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:36.426564 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:36.233131 25484 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.571s	user 0.009s	sys 0.007s
I20260709 12:17:36.435667 25189 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.375696 25112 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
W20260709 12:17:36.398219 12493 rpcz_store.cc:269] 0709 12:17:33.328694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.329754 (+  1060us) service_pool.cc:224] Handling call
0709 12:17:36.313154 (+2983400us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.313487 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139454336}
I20260709 12:17:36.473332 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.477900 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.482106 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:36.440307 25359 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.423700 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:36.483121 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:36.440917 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73523.6us
I20260709 12:17:36.430235 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:36.415706 12570 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.008s	sys 0.008s
I20260709 12:17:36.397840 12495 rpcz_store.cc:276] 0709 12:17:33.915751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.832991 (+1917240us) service_pool.cc:224] Handling call
0709 12:17:35.860903 (+ 27912us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.913460 (+ 52557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.295654 (+382194us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b540012b1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.295978 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63277312}
I20260709 12:17:36.451653 12642 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:36.398824 12564 rpcz_store.cc:276] 0709 12:17:35.282688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.685416 (+402728us) service_pool.cc:224] Handling call
0709 12:17:35.688885 (+  3469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.042797 (+353912us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.291207 (+248410us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:36.291836 (+   629us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":685626240}
I20260709 12:17:36.441367 25250 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:36.496110 25250 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.496009 25484 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.497022 13257 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:17:36.498801 13256 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:36.454078 25325 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:36.499191 25484 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.499755 25449 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:36.500945 13253 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:36.500648 25325 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.23, Last appended by leader: 23, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.460928 25463 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:36.442059 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 246491us
I20260709 12:17:36.452656 12643 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:36.452135 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
W20260709 12:17:36.450824 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 327062us
I20260709 12:17:36.454872 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:36.493031 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:36.430414 12640 rpcz_store.cc:269] 0709 12:17:34.766869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.767017 (+   148us) service_pool.cc:224] Handling call
0709 12:17:36.361350 (+1594333us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.361785 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2100329856}
I20260709 12:17:36.440146 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.496718 25250 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:36.492636 25112 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.475507 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.457458 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:36.414875 12566 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.000s	sys 0.019s
I20260709 12:17:36.506008 13255 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:36.455300 25078 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:36.509284 12570 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.009s	sys 0.009s
I20260709 12:17:36.451088 25490 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:17:36.502465 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:36.519783 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.000s	sys 0.015s
W20260709 12:17:36.520435 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.000s	sys 0.016s
I20260709 12:17:36.439445 25351 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:36.531008 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91155.9us
I20260709 12:17:36.532825 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:36.533823 25359 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:36.535442 25112 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:36.537824 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.004s	sys 0.011s
W20260709 12:17:36.538710 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.005s	sys 0.012s
I20260709 12:17:36.539122 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3969) took 1164 ms. Trace:
I20260709 12:17:36.539592 12639 rpcz_store.cc:276] 0709 12:17:35.375169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.469619 (+1094450us) service_pool.cc:224] Handling call
0709 12:17:36.539045 (+ 69426us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:36.541458 12566 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.000s	sys 0.020s
I20260709 12:17:36.515208 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:36.523918 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:36.497550 25578 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.541355 12641 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:36.538897 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:36.545572 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3972) took 1169 ms. Trace:
I20260709 12:17:36.550683 12645 rpcz_store.cc:276] 0709 12:17:35.376089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.475025 (+1098936us) service_pool.cc:224] Handling call
0709 12:17:36.545506 (+ 70481us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:36.552623 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.552587 13260 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:36.518697 25490 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:36.478587 12496 rpcz_store.cc:276] 0709 12:17:33.716798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.767669 (+2050871us) service_pool.cc:224] Handling call
0709 12:17:35.768115 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:35.768536 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:35.768565 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:35.779115 (+ 10550us) log.cc:844] Serialized 26 byte log entry
0709 12:17:35.886937 (+107822us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:35.887003 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:35.887111 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:36.477413 (+590302us) raft_consensus.cc:1685] finished
0709 12:17:36.477459 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.477765 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":144,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":14192,"thread_start_us":47932,"threads_started":1,"wal-append.queue_time_us":265889}
I20260709 12:17:36.544225 25112 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:36.558288 13260 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:36.558501 25363 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:36.564280 25363 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:36.564873 25363 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:36.566108 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.566298 12494 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:17:36.568732 25359 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:36.569845 12567 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:36.559283 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:17:36.575048 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.575531 12495 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:36.580179 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:36.566144 12488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.003s	sys 0.012s
W20260709 12:17:36.583021 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59619.6us
I20260709 12:17:36.588546 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.590333 25325 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:36.590961 25496 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.592181 25496 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.588594 13255 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 33 because replica has last-logged OpId of term: 31 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 22.
I20260709 12:17:36.588828 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3971) took 1213 ms. Trace:
W20260709 12:17:36.596135 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.000s	sys 0.018s
W20260709 12:17:36.596666 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.000s	sys 0.019s
W20260709 12:17:36.597111 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 748338us
I20260709 12:17:36.597553 25356 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.599346 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:36.598920 25356 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.591236 25484 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:36.601804 25496 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.602463 25356 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.562944 25490 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:17:36.609270 12488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.003s	sys 0.012s
I20260709 12:17:36.610980 25114 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:36.611924 25114 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.595829 12637 rpcz_store.cc:276] 0709 12:17:35.375746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.518909 (+1143163us) service_pool.cc:224] Handling call
0709 12:17:36.588749 (+ 69840us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.615445 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1884) took 1680 ms. Trace:
I20260709 12:17:36.658704 12637 rpcz_store.cc:276] 0709 12:17:34.934691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.614176 (+1679485us) service_pool.cc:224] Handling call
0709 12:17:36.614696 (+   520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.614994 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:36.615017 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:36.615086 (+    69us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:36.615133 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.615191 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.615333 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.590999 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.660429 25549 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:36.626386 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 785532us
I20260709 12:17:36.630345 25351 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.631632 25189 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:17:36.678264 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.007s	sys 0.016s
W20260709 12:17:36.678884 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.008s	sys 0.016s
I20260709 12:17:36.644029 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:36.679559 12462 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:17:36.680490 25463 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:36.644940 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.649513 25538 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:36.655265 12638 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: [41.22-41.22]
W20260709 12:17:36.657980 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58850.3us
I20260709 12:17:36.661151 25078 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:17:36.661793 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71004.6us
I20260709 12:17:36.602829 25484 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:36.664386 25562 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.665678 25356 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:36.696775 25356 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.697564 25356 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:36.697909 25484 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.678687 25189 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:36.689464 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45709.8us
I20260709 12:17:36.585395 25359 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:36.726087 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.702209 25189 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:36.730765 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.743153 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.742831 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.780869 13260 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:36.782327 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:36.783308 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:36.784777 25411 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:36.786037 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 108666us
W20260709 12:17:36.789836 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50420.1us
I20260709 12:17:36.791016 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.791350 25411 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:36.795992 25492 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:36.780408 25359 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.791970 25536 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:17:36.805224 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:36.798926 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 222109us
W20260709 12:17:36.798223 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53989.7us
I20260709 12:17:36.814777 12637 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 44 index: 29. (term mismatch)
I20260709 12:17:36.823977 12637 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 28
I20260709 12:17:36.825376 25388 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:36.827026 25114 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.809448 25441 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:17:36.828500 25441 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:36.828785 12564 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 24. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:17:36.828910 25441 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.797861 25236 raft_consensus.cc:2764] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
I20260709 12:17:36.797572 25492 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
W20260709 12:17:36.831240 25480 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.630s	user 0.011s	sys 0.007s
I20260709 12:17:36.832186 25492 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:36.831790 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:36.832693 25492 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.831929 25480 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.799778 25345 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.834606 12643 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
I20260709 12:17:36.835294 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4950) took 1014 ms. Trace:
I20260709 12:17:36.835729 12643 rpcz_store.cc:276] 0709 12:17:35.821174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.699816 (+878642us) service_pool.cc:224] Handling call
0709 12:17:36.835219 (+135403us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.835940 25538 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:36.803860 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:36.840356 25461 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:36.840745 25366 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
W20260709 12:17:36.841872 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.053s (SENT)
I20260709 12:17:36.842684 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.843874 25551 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:36.844720 25551 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.846086 25337 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:17:36.808516 25562 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.038s
I20260709 12:17:36.846819 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4918) took 2087 ms. Trace:
I20260709 12:17:36.847108 25567 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:36.778105 25549 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:17:36.832792 25493 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:36.849126 25493 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:36.850647 25493 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.852041 25493 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:36.852505 12642 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:17:36.854921 25495 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:36.799719 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5093) took 1355 ms. Trace:
I20260709 12:17:36.855765 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:36.856678 12565 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:36.855943 12570 rpcz_store.cc:276] 0709 12:17:35.444574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.727095 (+282521us) service_pool.cc:224] Handling call
0709 12:17:35.727622 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.728044 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:36.562522 (+834478us) log.cc:844] Serialized 14 byte log entry
0709 12:17:36.772227 (+209705us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:36.799164 (+ 26937us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:36.799300 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.799445 (+   145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.799645 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1951,"prepare.queue_time_us":300629,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":71,"raft.queue_time_us":31633,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":74259,"thread_start_us":878130,"threads_started":2,"wal-append.queue_time_us":288512}
I20260709 12:17:36.853188 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:36.858795 12568 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:36.861685 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:36.863468 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3988) took 1282 ms. Trace:
I20260709 12:17:36.864089 12641 rpcz_store.cc:276] 0709 12:17:35.580688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.456900 (+876212us) service_pool.cc:224] Handling call
0709 12:17:36.863145 (+406245us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:36.863371 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66817792}
I20260709 12:17:36.824363 12637 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 43.29
I20260709 12:17:36.865387 12637 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:36.866389 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1885) took 1854 ms. Trace:
I20260709 12:17:36.866286 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:36.867202 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 4947) took 1046 ms. Trace:
I20260709 12:17:36.867733 25345 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.867837 12637 rpcz_store.cc:276] 0709 12:17:35.012103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.813679 (+1801576us) service_pool.cc:224] Handling call
0709 12:17:36.814219 (+   540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.866107 (+ 51888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.866310 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.868628 25345 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.847386 12638 rpcz_store.cc:276] 0709 12:17:34.758982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:34.778735 (+ 19753us) service_pool.cc:224] Handling call
0709 12:17:34.850944 (+ 72209us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:36.737553 (+1886609us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.846404 (+108851us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:36.846723 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4147511168}
I20260709 12:17:36.869575 12637 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:17:36.870342 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4938) took 1774 ms. Trace:
W20260709 12:17:36.870587 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.867514 12644 rpcz_store.cc:276] 0709 12:17:35.820416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.865914 (+1045498us) service_pool.cc:224] Handling call
0709 12:17:36.867149 (+  1235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.874815 12637 rpcz_store.cc:276] 0709 12:17:35.096038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.868841 (+1772803us) service_pool.cc:224] Handling call
0709 12:17:36.869255 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.870065 (+   810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.870279 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.871557 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1881) took 1972 ms. Trace:
I20260709 12:17:36.875612 12639 rpcz_store.cc:276] 0709 12:17:34.899336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.870271 (+1970935us) service_pool.cc:224] Handling call
0709 12:17:36.870629 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.870922 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:36.871106 (+   184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:36.871193 (+    87us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:36.871256 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.871333 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.871483 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.877869 12639 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:17:36.877233 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:36.872852 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:36.811725 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.861349 12645 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:36.814435 13255 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:36.853371 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4936) took 1775 ms. Trace:
I20260709 12:17:36.881052 12642 rpcz_store.cc:276] 0709 12:17:35.077600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.851513 (+1773913us) service_pool.cc:224] Handling call
0709 12:17:36.852207 (+   694us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:36.853073 (+   866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.853310 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.884342 13260 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:36.879978 12641 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
W20260709 12:17:36.813580 25430 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.646s	user 0.013s	sys 0.005s
W20260709 12:17:36.888303 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 106099us
I20260709 12:17:36.888245 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3975) took 1506 ms. Trace:
I20260709 12:17:36.891521 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4931) took 1972 ms. Trace:
I20260709 12:17:36.891692 12639 rpcz_store.cc:276] 0709 12:17:35.376952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.876802 (+1499850us) service_pool.cc:224] Handling call
0709 12:17:36.877295 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.878469 (+  1174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.883506 (+  5037us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.892086 12645 rpcz_store.cc:276] 0709 12:17:34.914754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.860402 (+1945648us) service_pool.cc:224] Handling call
0709 12:17:36.860994 (+   592us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.886542 (+ 25548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.886807 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.816133 25235 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.869458 25571 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:36.827026 25359 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:36.901361 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69772.6us
I20260709 12:17:36.893370 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4937) took 1805 ms. Trace:
I20260709 12:17:36.902474 12641 rpcz_store.cc:276] 0709 12:17:35.082944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.865019 (+1782075us) service_pool.cc:224] Handling call
0709 12:17:36.865463 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.887867 (+ 22404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.888090 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.908406 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:36.914518 12638 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:36.811419 25597 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.097s
I20260709 12:17:36.895798 25571 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:36.930868 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:36.941393 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.798199 25589 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.132s
I20260709 12:17:36.947327 25551 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.948324 25571 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.949857 25441 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.951788 25441 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:17:36.952174 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 230) took 105877 ms (client timeout 15000 ms). Trace:
I20260709 12:17:36.954373 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5338) took 1671 ms. Trace:
I20260709 12:17:36.954653 12641 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:17:36.476862 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:36.955080 12566 rpcz_store.cc:276] 0709 12:17:35.282978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.692835 (+409857us) service_pool.cc:224] Handling call
0709 12:17:35.747502 (+ 54667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:35.747761 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:17:36.556036 (+808275us) log.cc:844] Serialized 14 byte log entry
0709 12:17:36.917869 (+361833us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:36.927379 (+  9510us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:36.942661 (+ 15282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.953973 (+ 11312us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:36.954220 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40098,"prepare.queue_time_us":342216,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":16497,"spinlock_wait_cycles":10549120,"thread_start_us":935074,"threads_started":2}
I20260709 12:17:36.956698 12644 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:36.956362 12389 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 29 to 33, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 33 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:36.957522 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4943) took 1784 ms. Trace:
I20260709 12:17:36.957243 25463 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:36.957943 12644 rpcz_store.cc:276] 0709 12:17:35.172656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.883211 (+1710555us) service_pool.cc:224] Handling call
0709 12:17:36.929365 (+ 46154us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.957213 (+ 27848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.957448 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.959802 25584 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:36.960345 25508 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.960532 25584 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:36.961002 25508 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:36.963056 25345 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.965076 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:36.898042 25430 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.966068 12640 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:36.967744 25430 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:36.968977 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:36.970175 25525 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:36.970808 25525 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:36.970988 12610 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:36.970832 12569 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:17:36.967854 12640 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:36.937759 12565 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:36.959993 12644 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 27. Preceding OpId from leader: term: 39 index: 27. (term mismatch)
I20260709 12:17:36.959156 12389 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 26 to 31, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 31 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.970146 12489 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:17:36.944072 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.930190 25359 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:36.956240 12570 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:36.973702 12644 pending_rounds.cc:85] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 26
I20260709 12:17:36.981477 12644 pending_rounds.cc:107] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 38.27
I20260709 12:17:36.981904 12644 raft_consensus.cc:2889] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:36.975116 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3990) took 1393 ms. Trace:
I20260709 12:17:36.982530 12641 rpcz_store.cc:276] 0709 12:17:35.581229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.918553 (+1337324us) service_pool.cc:224] Handling call
0709 12:17:36.954086 (+ 35533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.972173 (+ 18087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.975042 (+  2869us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.978641 12639 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:17:36.985144 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4949) took 1155 ms. Trace:
I20260709 12:17:36.940012 25480 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:36.907032 25235 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.985580 12640 rpcz_store.cc:276] 0709 12:17:35.820934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.967072 (+1146138us) service_pool.cc:224] Handling call
0709 12:17:36.967526 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.976515 (+  8989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.976790 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.990722 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3986) took 1410 ms. Trace:
I20260709 12:17:36.992901 12639 rpcz_store.cc:276] 0709 12:17:35.580094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.895704 (+1315610us) service_pool.cc:224] Handling call
0709 12:17:36.976163 (+ 80459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:36.987911 (+ 11748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.990633 (+  2722us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:36.974220 25525 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:36.998926 12563 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
W20260709 12:17:37.004968 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.009100 25500 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.011847 12642 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:17:37.008477 12637 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
W20260709 12:17:37.016429 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:36.994999 25571 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:17:37.066651 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 255075us
W20260709 12:17:37.067951 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3918) took 3358 ms (client timeout 2054 ms). Trace:
I20260709 12:17:37.068463 12497 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:17:37.068799 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:37.070262 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 3974) took 1693 ms (client timeout 2142 ms). Trace:
W20260709 12:17:37.070713 12646 rpcz_store.cc:269] 0709 12:17:35.376669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.755276 (+378607us) service_pool.cc:224] Handling call
0709 12:17:37.069987 (+1314711us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:37.070193 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1373539200}
I20260709 12:17:37.071125 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4941) took 1929 ms. Trace:
I20260709 12:17:37.071579 12638 rpcz_store.cc:276] 0709 12:17:35.142137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.870091 (+1727954us) service_pool.cc:224] Handling call
0709 12:17:36.870599 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.931585 (+ 60986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.070838 (+139253us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:37.071056 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72707840}
W20260709 12:17:37.074093 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60496.9us
I20260709 12:17:37.077229 25411 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:37.078402 25411 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:37.078892 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:36.952620 12386 rpcz_store.cc:269] 0709 12:15:51.075224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:01.369109 (+10293885us) service_pool.cc:224] Handling call
0709 12:16:01.552827 (+183718us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:36.951850 (+95399023us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:17:21.715053 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:17:21.894767 (+179714us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:17:21.894840 (+    73us) write_op.cc:435] Acquired schema lock
0709 12:17:21.894862 (+    22us) tablet.cc:662] Decoding operations
0709 12:17:21.988116 (+ 93254us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:17:22.008831 (+ 20715us) write_op.cc:641] Partition lock acquired for write op
0709 12:17:22.012900 (+  4069us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:17:22.309263 (+296363us) tablet.cc:701] Row locks acquired
0709 12:17:22.309314 (+    51us) write_op.cc:260] PREPARE: finished
0709 12:17:22.309539 (+   225us) write_op.cc:270] Start()
0709 12:17:22.309766 (+   227us) write_op.cc:276] Timestamp: P: 1783599442309460 usec, L: 0
0709 12:17:22.309784 (+    18us) op_driver.cc:348] REPLICATION: starting
0709 12:17:22.340936 (+ 31152us) log.cc:844] Serialized 6028 byte log entry
0709 12:17:23.387366 (+1046430us) op_driver.cc:464] REPLICATION: finished
0709 12:17:23.588111 (+200745us) write_op.cc:301] APPLY: starting
0709 12:17:23.596844 (+  8733us) tablet.cc:1367] starting BulkCheckPresence
0709 12:17:23.756999 (+160155us) tablet.cc:1370] finished BulkCheckPresence
0709 12:17:23.757104 (+   105us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:17:25.760254 (+2003150us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:17:25.760579 (+   325us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 12:17:25.760640 (+    61us) write_op.cc:312] APPLY: finished
0709 12:17:25.761037 (+   397us) log.cc:844] Serialized 139 byte log entry
0709 12:17:25.823776 (+ 62739us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:25.824000 (+   224us) write_op.cc:454] Released schema lock
0709 12:17:25.824071 (+    71us) write_op.cc:341] FINISH: Updating metrics
0709 12:17:25.824654 (+   583us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:25.824696 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1050883,"reported_tablets":51,"spinlock_wait_cycles":2317568,"child_traces":[["op",{"apply.queue_time_us":179988,"apply.run_cpu_time_us":42935,"apply.run_wall_time_us":2244513,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66767,"num_ops":15,"prepare.queue_time_us":568108,"prepare.run_cpu_time_us":134331,"prepare.run_wall_time_us":1067017,"raft.queue_time_us":114567,"raft.run_cpu_time_us":62625,"raft.run_wall_time_us":189303,"replication_time_us":1027897,"thread_start_us":991972,"threads_started":5,"wal-append.queue_time_us":572080,"wal-append.run_cpu_time_us":23955,"wal-append.run_wall_time_us":2586293}]]}
I20260709 12:17:37.079353 25411 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:37.079897 25411 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:37.080132 25558 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:37.080704 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4946) took 1261 ms. Trace:
I20260709 12:17:37.081327 12644 rpcz_store.cc:276] 0709 12:17:35.819883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.958981 (+1139098us) service_pool.cc:224] Handling call
0709 12:17:36.959467 (+   486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.015543 (+ 56076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.080633 (+ 65090us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:37.018108 25500 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.018393 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3985) took 1435 ms. Trace:
I20260709 12:17:37.019178 25363 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:37.068733 12489 rpcz_store.cc:269] 0709 12:17:33.709601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:33.898999 (+189398us) service_pool.cc:224] Handling call
0709 12:17:37.067531 (+3168532us) spinlock_profiling.cc:243] Waited 3.06 s on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:37.067822 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7035798784}
I20260709 12:17:37.100828 12568 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:37.020864 25235 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.022451 12389 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 26, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 26 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:36.999264 25492 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.023947 25589 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:37.024719 25584 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 39.29, Last appended by leader: 29, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:36.984881 25356 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.026005 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3968) took 1639 ms. Trace:
W20260709 12:17:37.026192 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128536us
I20260709 12:17:37.010201 25250 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.027031 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:37.028705 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1888) took 1035 ms. Trace:
W20260709 12:17:37.029124 25482 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.952s	user 0.005s	sys 0.020s
W20260709 12:17:37.113570 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 135) took 15679 ms (client timeout 15000 ms). Trace:
I20260709 12:17:37.030021 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.031505 12640 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 43 index: 22. (index mismatch)
W20260709 12:17:37.032424 25598 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.501s	user 0.005s	sys 0.000s
W20260709 12:17:37.018131 25127 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.779s	user 0.000s	sys 0.025s
I20260709 12:17:36.985975 25590 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.052s
I20260709 12:17:37.042718 25490 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.131186 12643 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 28. Current term is 29. Ops: []
I20260709 12:17:37.132434 25490 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.052491 25538 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:37.010493 25359 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.135458 25359 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.007413 25536 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:17:37.058028 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72493.4us
I20260709 12:17:37.080751 25558 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.136559 25558 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:37.136662 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1886) took 2104 ms. Trace:
I20260709 12:17:37.137296 12643 rpcz_store.cc:276] 0709 12:17:35.032843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.837212 (+1804369us) service_pool.cc:224] Handling call
0709 12:17:36.879504 (+ 42292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.131630 (+252126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.136316 (+  4686us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:37.136556 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":578249984}
I20260709 12:17:37.137923 25363 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.138470 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.139496 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.083899 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2499) took 1065 ms. Trace:
I20260709 12:17:37.082980 12642 rpcz_store.cc:276] 0709 12:17:35.579784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.891246 (+1311462us) service_pool.cc:224] Handling call
0709 12:17:36.971914 (+ 80668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.012466 (+ 40552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.015021 (+  2555us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:37.111955 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85021.9us
I20260709 12:17:37.140137 12569 rpcz_store.cc:276] 0709 12:17:36.018911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.330341 (+311430us) service_pool.cc:224] Handling call
0709 12:17:36.331072 (+   731us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:36.976546 (+645474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.077455 (+100909us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:37.083801 (+  6346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1470458240}
I20260709 12:17:37.018846 25390 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:37.141484 25536 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:37.141999 25536 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.142390 25536 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:37.114084 12386 rpcz_store.cc:269] 0709 12:17:21.430985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:37.109697 (+15678712us) service_pool.cc:211] Skipping call since client already timed out
0709 12:17:37.109901 (+   204us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:17:37.142781 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:37.143522 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 237) took 15511 ms (client timeout 15000 ms). Trace:
W20260709 12:17:37.143971 12386 rpcz_store.cc:269] 0709 12:17:21.631751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:37.143261 (+15511510us) service_pool.cc:211] Skipping call since client already timed out
0709 12:17:37.143447 (+   186us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:17:37.145195 25449 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:37.117722 12565 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 37 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 19.
I20260709 12:17:37.113903 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.146485 25449 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:37.146718 12563 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:37.113890 25112 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:37.147931 25112 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.147768 25449 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 41.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.107053 25589 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.125353 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5247) took 2119 ms. Trace:
I20260709 12:17:37.148656 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.126327 25492 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:37.111490 12641 rpcz_store.cc:276] 0709 12:17:35.992976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:37.003389 (+1010413us) service_pool.cc:224] Handling call
0709 12:17:37.027830 (+ 24441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.028199 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:37.028235 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:37.028331 (+    96us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:37.028378 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.028455 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:37.028600 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:37.129879 25388 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:37.148420 25112 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.148952 25589 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:37.155725 25359 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:37.156247 25359 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.156606 25359 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.158188 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5243) took 2239 ms. Trace:
I20260709 12:17:37.158627 12488 rpcz_store.cc:276] 0709 12:17:34.932077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.865672 (+933595us) service_pool.cc:224] Handling call
0709 12:17:35.866304 (+   632us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:35.866668 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:35.866713 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:35.867432 (+   719us) log.cc:844] Serialized 26 byte log entry
0709 12:17:36.610359 (+742927us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:36.610447 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:36.610598 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:37.031515 (+420917us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:37.157725 (+126210us) raft_consensus.cc:1685] finished
0709 12:17:37.157751 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:37.158091 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20574464,"thread_start_us":742195,"threads_started":1,"wal-append.queue_time_us":663333}
I20260709 12:17:37.160140 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:37.161327 25597 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:37.161779 25597 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:37.162186 25597 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.152985 25235 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:17:37.116305 25385 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:37.166155 25235 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:17:37.166689 25235 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:37.110531 12637 rpcz_store.cc:276] 0709 12:17:35.374861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.887134 (+1512273us) service_pool.cc:224] Handling call
0709 12:17:36.943292 (+ 56158us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.013920 (+ 70628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.014124 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:37.130174 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61462.5us
W20260709 12:17:37.167363 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63978.3us
W20260709 12:17:37.165781 25325 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.575s	user 0.013s	sys 0.003s
I20260709 12:17:37.139573 25574 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.170090 12645 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:37.148984 12497 rpcz_store.cc:276] 0709 12:17:35.001305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:35.944286 (+942981us) service_pool.cc:224] Handling call
0709 12:17:37.067867 (+1123581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.068980 (+  1113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.120009 (+ 51029us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540037df50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:37.120222 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2581908224}
I20260709 12:17:37.170331 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.112879 25508 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.082020 25411 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.188789 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:37.135299 25584 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:37.187009 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:37.183562 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.193748 12490 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 21. (term mismatch)
I20260709 12:17:37.194226 12490 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 20
I20260709 12:17:37.194643 12490 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 38.21
I20260709 12:17:37.194897 12490 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:37.181589 12389 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 26 to 28, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 28 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:37.220124 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 3987) took 1639 ms. Trace:
I20260709 12:17:37.220775 12645 rpcz_store.cc:276] 0709 12:17:35.580389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.903897 (+1323508us) service_pool.cc:224] Handling call
0709 12:17:36.974758 (+ 70861us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.171895 (+197137us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.213614 (+ 41719us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:37.220017 (+  6403us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448628096}
I20260709 12:17:37.227294 12610 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 31. Current term is 32.
I20260709 12:17:37.191372 25325 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:37.167896 25482 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.240769 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:37.185392 25189 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.229745 25508 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:37.227546 25574 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:37.232523 25325 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.245616 12389 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 22 to 32, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 32 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:37.256703 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.258046 12389 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 21 to 24. New cstate: current_term: 24 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:17:37.259627 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72710.8us
I20260709 12:17:37.250746 25508 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.253780 25325 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:37.238749 25525 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:37.275574 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51661.4us
W20260709 12:17:37.282939 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.283174 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.285046 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.286684 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:37.288939 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 119802us
I20260709 12:17:37.265215 25495 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:37.287809 12563 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:37.291849 12389 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 28 to 30, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 30 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:37.294744 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:37.278981 25508 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:37.281070 25411 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:37.277000 25356 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.302172 12565 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:37.249163 25235 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.294389 25490 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.314653 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:37.318322 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.328145 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:37.333925 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109342us
I20260709 12:17:37.334420 25482 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.335929 25189 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.339263 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:37.340147 13253 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:37.341877 25572 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:37.316005 25500 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:37.347752 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.352766 25589 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.353741 12571 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:37.356948 12565 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:17:37.358400 25235 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:17:37.360718 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:37.361748 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:37.362027 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:37.329692 25250 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.362900 25525 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.352846 25490 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.320508 12389 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 26 to 28, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 28 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:37.364377 12568 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:17:37.310382 25495 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:37.366799 25495 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.367362 25127 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:17:37.368175 25127 consensus_queue.cc:1243] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:17:37.366477 12389 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 22 to 26. New cstate: current_term: 26 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:17:37.378073 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:37.378144 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59929.7us
W20260709 12:17:37.386296 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97614.2us
I20260709 12:17:37.381145 25500 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:37.396699 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.397007 25500 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.397583 25500 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.404913 25482 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:37.406152 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95054.1us
I20260709 12:17:37.406670 12497 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 24. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:17:37.407105 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:37.395581 25441 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:37.408172 25589 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.407867 25536 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.411234 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:37.411741 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:37.412460 25359 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.414994 25359 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.407828 25482 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.416587 25112 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.412899 25536 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.406029 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:37.419185 13253 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:37.365520 25490 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.386189 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.418293 25482 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.420706 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.423954 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:37.424821 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 117083us
I20260709 12:17:37.433970 12488 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:37.433598 12489 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:37.442687 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:37.443390 13256 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:37.449286 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:37.457082 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:37.446875 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:37.352772 25558 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.463999 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.463483 25127 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.470175 25127 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:37.473007 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.448645 25597 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.475910 25597 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.476056 25127 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.478549 25574 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:37.451646 25359 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:37.421305 25493 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.042s
I20260709 12:17:37.489053 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:37.494413 13256 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:37.497978 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3926) took 2113 ms. Trace:
I20260709 12:17:37.499881 12490 rpcz_store.cc:276] 0709 12:17:35.384845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.118520 (+733675us) service_pool.cc:224] Handling call
0709 12:17:36.287771 (+169251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.230432 (+942661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.492373 (+261941us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:37.497910 (+  5537us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2033905408}
I20260709 12:17:37.466859 25356 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:37.467800 25441 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:37.499706 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.457448 25536 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:37.510214 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:37.510623 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 99493.5us
I20260709 12:17:37.497483 25525 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:17:37.515175 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85780.2us
I20260709 12:17:37.510533 25572 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:37.516106 25612 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:17:37.521374 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119390us
W20260709 12:17:37.523132 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:37.524557 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.511853 12389 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 24 to 30, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 30 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:37.507068 25359 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:37.519872 25619 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:17:37.508949 25441 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:37.514237 25356 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:37.523350 25616 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:17:37.517004 25525 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:37.535650 25571 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.522089 25551 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:37.536052 25359 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.522465 25572 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:37.537806 25356 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.534786 25536 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.541302 25309 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:17:37.542017 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:37.542317 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.012s	sys 0.007s
I20260709 12:17:37.543105 12389 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:17:37.553093 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.012s	sys 0.007s
I20260709 12:17:37.543931 25558 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.522596 25612 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:37.558770 25612 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.558449 25498 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:17:37.545068 25112 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.544777 25500 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.542089 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:37.546792 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139801us
W20260709 12:17:37.551962 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.534590 25441 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:37.553450 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 640909us
I20260709 12:17:37.569320 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.570263 12570 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:37.561132 25345 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:37.577361 13257 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:37.575663 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2498) took 1594 ms. Trace:
I20260709 12:17:37.564038 25574 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:37.594036 25482 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:37.567840 25250 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:37.598124 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56170.1us
I20260709 12:17:37.599656 25366 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:37.594058 25500 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.601699 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:37.602859 25525 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.604759 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:37.605065 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.609031 12568 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:37.621822 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.618046 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:37.576263 25490 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:37.549817 25571 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.629614 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:37.611631 12572 pending_rounds.cc:85] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 14
I20260709 12:17:37.632553 12572 pending_rounds.cc:107] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:17:37.633040 12572 raft_consensus.cc:2889] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:37.625495 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.628953 25449 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:37.588819 25345 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:37.590368 12571 rpcz_store.cc:276] 0709 12:17:35.981390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.322329 (+340939us) service_pool.cc:224] Handling call
0709 12:17:36.323048 (+   719us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:37.354318 (+1031270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.575390 (+221072us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:37.575607 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2132983680}
I20260709 12:17:37.640491 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.614336 25574 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:37.643784 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:37.643417 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.637024 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:37.651233 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.652246 13256 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:37.568046 25551 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.659791 25449 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:37.660022 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.660770 25390 raft_consensus.cc:2764] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election decision vote started in defunct term 35: won
I20260709 12:17:37.653680 25493 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.008s
I20260709 12:17:37.662654 25508 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.656176 12490 raft_consensus.cc:2376] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40. Re-sending same reply.
I20260709 12:17:37.669382 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:37.575042 25325 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:37.674808 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:37.667514 25571 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.675720 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:37.677570 13258 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:37.652123 25574 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.683115 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:37.671895 25309 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:37.685437 12493 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 41 because replica has last-logged OpId of term: 38 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 22.
I20260709 12:17:37.636327 25250 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.684331 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:37.673497 25525 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:37.671707 25112 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:37.684484 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124357us
W20260709 12:17:37.706804 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70523.4us
W20260709 12:17:37.709604 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132408us
I20260709 12:17:37.716882 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:37.718255 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.677820 25495 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.024s
I20260709 12:17:37.718950 13262 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:37.726343 13254 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:17:37.677405 25572 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.731689 25545 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:37.732319 25545 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:37.736338 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55470us
I20260709 12:17:37.737017 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.737526 13260 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 29.
I20260709 12:17:37.737883 12568 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:17:37.767109 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:37.770313 13218 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.539s (SENT)
I20260709 12:17:37.777987 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:37.770116 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.788802 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:37.790175 25449 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.719053 25390 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:37.770537 25366 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:37.709280 25385 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.075s
I20260709 12:17:37.787518 25112 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:37.808575 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:37.803885 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131934us
I20260709 12:17:37.768980 25441 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.799314 25536 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.791903 25590 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.020s
I20260709 12:17:37.788451 25500 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.021s
I20260709 12:17:37.803469 25495 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:37.733546 25545 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.808732 25112 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:37.810555 25309 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:37.846692 12497 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:37.849174 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.811177 25584 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:37.865509 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95230.3us
I20260709 12:17:37.843832 25482 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:37.869135 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71509.8us
I20260709 12:17:37.875710 12571 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 35 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:17:37.847975 25112 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:37.881330 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:37.809253 25390 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.887471 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
W20260709 12:17:37.889102 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:37.889940 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104845us
W20260709 12:17:37.892520 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125138us
I20260709 12:17:37.899274 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.916790 12568 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:17:37.931185 25572 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.932098 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.932621 25549 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.933101 25549 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.934653 25631 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:37.935317 25631 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.936829 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.939453 12568 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:37.931895 25572 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.926110 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:37.940935 12569 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 33 because replica has last-logged OpId of term: 31 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 22.
I20260709 12:17:37.942241 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.933475 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.941282 13257 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:17:37.898994 25508 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:37.905359 25551 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.886777 25584 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:37.912177 25545 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:37.953819 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:37.948074 12567 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:37.953802 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:37.918497 25325 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.953203 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:37.957393 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:37.970636 12571 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:37.968439 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:37.947969 25631 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:37.961494 25627 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.111s	user 0.000s	sys 0.005s
I20260709 12:17:37.908306 25482 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:37.984494 25626 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:17:37.985463 25482 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.986222 25482 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:37.986788 25482 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:37.989140 25482 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:37.958873 25250 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:37.973570 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:37.967111 25463 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.014s
I20260709 12:17:37.976711 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:37.990849 25630 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.004s	sys 0.001s
W20260709 12:17:37.994046 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52674.1us
I20260709 12:17:37.995196 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:37.985798 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:37.990003 25226 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.427s	user 0.005s	sys 0.007s
I20260709 12:17:38.005069 12567 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:37.978513 25536 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:17:37.978821 25309 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:38.007436 25597 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:17:38.030186 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106337us
I20260709 12:17:38.006633 25356 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:38.031558 12567 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:17:37.974296 25634 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.054s
I20260709 12:17:38.040541 12570 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:38.001291 25574 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.023744 25551 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:38.022054 25536 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:38.015151 25366 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:38.044461 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.051635 12567 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 29: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:38.052276 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.037101 25642 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:17:38.061978 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.856s (SENT)
I20260709 12:17:38.046517 25356 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:38.065033 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:38.054999 25611 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.758s	user 0.008s	sys 0.000s
I20260709 12:17:38.057909 25642 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:38.067251 25629 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:17:38.066830 25536 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:38.069907 25612 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.511s	user 0.011s	sys 0.003s
W20260709 12:17:38.067271 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72230.1us
I20260709 12:17:38.066907 25646 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.004s
I20260709 12:17:38.041179 25250 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:17:38.070892 25366 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.078804 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.079473 12495 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:38.080214 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.129232 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:38.080336 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.082059 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.130476 12494 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:38.085604 25572 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:38.087539 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.135236 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.089751 25463 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:17:38.090139 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.090803 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.090966 25226 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=18 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:17:38.155019 12566 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:38.094035 25549 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:38.095290 25390 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:38.097154 25574 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:38.099893 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:38.086539 25345 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:38.110060 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.160125 25640 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:38.160580 25640 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:38.122277 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:38.122367 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:38.123114 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:38.126947 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.080986 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.129141 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.130398 12491 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:38.106969 25356 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:38.138926 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5346) took 1287 ms. Trace:
I20260709 12:17:38.139290 25482 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:38.143507 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.144649 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.146159 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.146425 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.129685 25612 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.147388 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.148317 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.149206 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.150028 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.135938 12567 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:38.125949 25235 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:38.153829 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.155901 12569 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:38.158282 25226 tablet_bootstrap.cc:492] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:17:38.184379 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.185007 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.185617 12646 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.186018 12495 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.188704 13257 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.160926 25551 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.191854 25640 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.192507 25551 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.194712 25441 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:38.195829 25551 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:38.161159 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99250.4us
W20260709 12:17:38.152370 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.176111 12568 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:38.178727 12637 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 42 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:17:38.179023 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:38.177752 12642 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:38.180667 12641 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.190500 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.191399 12563 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:38.191591 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.197894 25590 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.180486 13258 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:38.198967 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.199498 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.188633 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.201280 25226 ts_tablet_manager.cc:1403] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 5.897s	user 0.101s	sys 0.052s
I20260709 12:17:38.201730 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:38.158226 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.195457 25441 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.202404 25642 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.203410 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:38.203488 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:38.187099 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.184571 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.225733 12570 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43.
I20260709 12:17:38.208424 25584 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:38.209427 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:38.211171 25635 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.282s	user 0.003s	sys 0.000s
W20260709 12:17:38.211905 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.229102 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.188272 25390 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:38.180984 25482 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:38.185693 25639 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.258s	user 0.000s	sys 0.004s
I20260709 12:17:38.177403 12564 rpcz_store.cc:276] 0709 12:17:36.851046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:36.860359 (+  9313us) service_pool.cc:224] Handling call
0709 12:17:36.860915 (+   556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:36.887779 (+ 26864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:36.887815 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:36.895893 (+  8078us) log.cc:844] Serialized 26 byte log entry
0709 12:17:37.104663 (+208770us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:37.104718 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.109515 (+  4797us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:37.962276 (+852761us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:38.138550 (+176274us) raft_consensus.cc:1685] finished
0709 12:17:38.138576 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:38.138850 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8098,"raft.queue_time_us":14302,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":31701,"spinlock_wait_cycles":3461888,"thread_start_us":191879,"threads_started":1,"wal-append.queue_time_us":340026}
I20260709 12:17:38.110401 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:38.186985 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.193354 25640 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:38.188549 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.214557 12644 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.187404 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.184379 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:38.187342 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58125.5us
I20260709 12:17:38.229899 12640 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:38.238129 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:38.217760 12494 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:38.221494 12492 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 19.
I20260709 12:17:38.187050 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.225313 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.222904 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:38.204854 25632 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.277s	user 0.002s	sys 0.000s
I20260709 12:17:38.228076 12460 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43.
I20260709 12:17:38.236382 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:38.225790 12639 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:38.242735 12460 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Cancelling election due to peer responding with higher term
I20260709 12:17:38.230867 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:38.241535 12490 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.245141 12569 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 31: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:38.245369 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84272us
I20260709 12:17:38.247103 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.236296 25482 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:38.248028 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.248992 12570 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.236037 25390 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:38.249835 25625 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:38.250594 25625 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:38.250309 25390 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 41.17, Last appended by leader: 17, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.251734 25493 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:38.235715 25641 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:38.252745 25493 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:38.254051 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.254863 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:38.255426 25250 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:38.256090 25250 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:38.256178 12571 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 36. Ops: []
I20260709 12:17:38.245431 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.251621 25625 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.262458 12568 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:38.244443 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:38.255096 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.264020 12566 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 19.
I20260709 12:17:38.273615 25482 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.264106 25508 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 42. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43.
I20260709 12:17:38.279564 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.230710 25189 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.283195 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.213503 25112 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.283960 12640 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:38.243016 25642 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:38.285027 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:38.252018 25390 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:38.290838 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.295347 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.258991 25571 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:38.297317 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.259256 25463 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:38.257265 25250 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.301028 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:38.305087 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:38.306319 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:38.308460 13258 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:38.306588 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.238564 25441 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.310091 12568 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 29. Preceding OpId from leader: term: 40 index: 29. (term mismatch)
I20260709 12:17:38.334424 12568 pending_rounds.cc:85] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 28
I20260709 12:17:38.334802 12568 pending_rounds.cc:107] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 39.29
I20260709 12:17:38.335114 12568 raft_consensus.cc:2889] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:38.258510 25590 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.297899 25490 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:38.312621 13256 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:38.309863 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.315407 25536 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.316354 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.318989 25545 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:38.320963 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.322336 25309 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:38.323098 25572 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:17:38.361564 12640 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:38.323375 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.362677 12644 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:38.323951 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.324597 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.324729 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.364244 12490 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.325362 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.365104 12497 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.326097 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.289549 25495 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:38.367219 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:38.352284 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125479us
I20260709 12:17:38.276676 25482 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:38.346309 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45227.1us
I20260709 12:17:38.396183 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.397073 12492 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:38.397410 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.397444 25508 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.302822 25551 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:38.399096 25551 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.399605 25551 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:38.302518 25463 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.403056 25463 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:38.302778 25645 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.428443 12565 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:38.360152 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:38.398325 25508 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.406733 12638 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33.
I20260709 12:17:38.408546 12491 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
I20260709 12:17:38.408812 12489 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:38.412665 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:38.413481 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.418401 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.435626 12640 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.302909 25498 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.059s
I20260709 12:17:38.347985 25571 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:38.310374 25189 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:38.439348 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123081us
W20260709 12:17:38.443171 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83087.4us
I20260709 12:17:38.445192 12568 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
W20260709 12:17:38.432600 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64073.8us
I20260709 12:17:38.447341 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.366981 25525 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:38.446671 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.455567 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.456913 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:38.462940 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.407217 25359 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:17:38.409413 25449 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:38.469187 12488 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:38.465494 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.435391 25584 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:38.406217 25536 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:38.484951 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.488189 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.442425 25495 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:38.491617 12490 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:38.490716 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.497839 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:38.499505 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 139233us
I20260709 12:17:38.500409 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:38.441711 25545 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:38.409051 25641 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.062s
I20260709 12:17:38.449462 25490 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:38.502709 12645 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:38.505379 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:38.505618 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.506507 12638 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.513250 12492 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:17:38.515852 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.517750 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.517807 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.482218 25112 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:38.521014 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:38.479882 25508 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:38.479907 25572 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:38.509541 25490 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.530471 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.534968 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.444633 25645 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:38.526796 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56910.2us
I20260709 12:17:38.538650 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.536705 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.479103 25493 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:17:38.542169 12492 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:38.485280 25500 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.043s
I20260709 12:17:38.539357 12637 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:38.547537 12642 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: [37.21-37.21]
I20260709 12:17:38.551836 12497 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:38.552059 12640 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:38.502575 25584 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.554934 12638 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:38.557224 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.558611 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.560107 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:38.560962 12490 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:17:38.561735 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:38.562508 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.563443 12489 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 37 because replica has last-logged OpId of term: 35 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:17:38.564715 12489 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:17:38.566571 12489 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:17:38.467638 25571 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.587149 25385 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:38.587481 12495 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:38.587836 25385 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:38.588806 25385 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.590631 25441 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:38.591321 12496 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: [37.26-37.26]
I20260709 12:17:38.591730 25495 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.545701 25359 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:17:38.567080 25656 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260709 12:17:38.571063 25654 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:17:38.572085 25662 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:38.572427 25612 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:38.573580 25189 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:38.573999 25650 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:38.574815 25634 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:38.579959 12645 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:17:38.579733 25226 raft_consensus.cc:359] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 33 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:38.582594 25651 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
W20260709 12:17:38.593376 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81905.8us
I20260709 12:17:38.607141 13258 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:38.633574 12495 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 28. Preceding OpId from leader: term: 40 index: 29. (index mismatch)
I20260709 12:17:38.540284 25449 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:38.539007 25490 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:38.522420 25545 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.652707 12492 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:17:38.640429 12496 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:38.593458 25500 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:17:38.641405 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139801us
I20260709 12:17:38.642810 25646 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:38.549196 25309 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:17:38.572111 25653 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:17:38.649853 12497 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 19. (term mismatch)
I20260709 12:17:38.636787 25662 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:38.714952 12568 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:17:38.630982 25359 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:38.711465 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.721079 12497 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
W20260709 12:17:38.655390 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155067us
I20260709 12:17:38.723790 12497 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 35.19
I20260709 12:17:38.724103 12497 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:38.724274 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:38.643884 25441 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:38.727362 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 288083us
I20260709 12:17:38.739880 12640 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:38.741729 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.742511 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:38.739961 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:38.745905 12563 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:17:38.746572 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:38.747433 12645 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:38.749511 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.750219 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 32.
I20260709 12:17:38.655061 25500 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:38.757539 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:38.712966 25634 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:38.776521 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:38.793526 12640 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:38.713866 25574 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.065s
I20260709 12:17:38.801113 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:38.806473 25670 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:38.806960 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:38.754808 25536 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:38.809898 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.810590 25309 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:38.811638 25500 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:38.814143 25645 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:38.815908 25390 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.513s	user 0.005s	sys 0.006s
I20260709 12:17:38.751839 25441 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.812526 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:38.739081 25662 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:38.834803 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:38.894273 12494 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
W20260709 12:17:38.902540 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90096.5us
I20260709 12:17:38.886804 25549 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
W20260709 12:17:38.912874 25671 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.144s	user 0.002s	sys 0.000s
W20260709 12:17:38.900480 25632 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:17:38.933614 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:38.939592 12643 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.001s
I20260709 12:17:38.940179 12643 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:38.941161 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.886204 25449 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:17:38.946431 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:38.948110 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.913681 25549 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:38.944268 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:38.950065 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:38.950212 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:38.914382 25676 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:17:38.952891 25463 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.002s
W20260709 12:17:38.943511 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 186058us
I20260709 12:17:38.961329 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:38.967877 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:38.975575 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:38.978340 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69741.2us
I20260709 12:17:38.979492 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:38.942153 25226 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 33 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:17:38.981169 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:38.985996 12568 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:38.946079 25525 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:38.920979 25309 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.046s
I20260709 12:17:38.897400 25359 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:38.954164 25385 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:38.953797 25664 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:38.999222 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:39.001884 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200874us
I20260709 12:17:39.010880 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:38.984848 25536 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:38.978600 25676 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:38.991845 25584 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:39.042717 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84486.4us
I20260709 12:17:39.044523 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.035171 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:39.005435 25385 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:38.975056 25463 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.029616 25545 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:39.056347 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 95165us
I20260709 12:17:38.948966 25634 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:39.131762 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:39.144537 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100109us
I20260709 12:17:39.146102 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.031095 25668 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:39.147039 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96565.4us
I20260709 12:17:39.037875 25682 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:39.050633 25385 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:39.147624 25545 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.018267 25664 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.055536 25584 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:39.132874 25536 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:39.029057 25549 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 42.18, Last appended by leader: 18, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.150482 25668 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.179628 25678 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.201s
W20260709 12:17:39.179978 25637 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.125s	user 0.005s	sys 0.005s
I20260709 12:17:39.180052 25669 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:39.185310 25658 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:39.186039 25665 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
W20260709 12:17:39.186754 25679 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
W20260709 12:17:39.309475 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:39.050546 25226 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:17:39.188786 25681 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:17:39.212185 25684 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:39.189554 25690 raft_consensus.cc:2764] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:17:39.178157 25677 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:17:39.203150 25545 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.218281 25441 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:39.156982 25640 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.915s	user 0.015s	sys 0.000s
I20260709 12:17:39.129851 25675 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:39.208623 25676 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.356535 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4026) took 1148 ms. Trace:
I20260709 12:17:39.357131 12642 rpcz_store.cc:276] 0709 12:17:38.207739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.569971 (+362232us) service_pool.cc:224] Handling call
0709 12:17:38.638597 (+ 68626us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:38.639035 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:38.639077 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:38.639754 (+   677us) log.cc:844] Serialized 26 byte log entry
0709 12:17:38.893175 (+253421us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:38.893222 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:38.893311 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:39.356218 (+462907us) raft_consensus.cc:1685] finished
0709 12:17:39.356247 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:39.356426 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34821,"raft.queue_time_us":10703,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":15602,"thread_start_us":154750,"threads_started":1,"wal-append.queue_time_us":283431}
I20260709 12:17:39.334334 25658 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:39.367388 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4022) took 1160 ms. Trace:
W20260709 12:17:39.206185 25642 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.903s	user 0.015s	sys 0.000s
W20260709 12:17:39.226435 25490 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.511s	user 0.013s	sys 0.000s
I20260709 12:17:39.226228 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.228773 25463 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:39.155794 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:39.269829 25385 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:39.294747 12645 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.003s
W20260709 12:17:39.301824 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 266720us
I20260709 12:17:39.188767 25664 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:39.312525 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:39.234580 25584 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:39.290373 25668 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:39.331630 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5362) took 1730 ms. Trace:
I20260709 12:17:39.332262 25549 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:39.370777 12637 rpcz_store.cc:276] 0709 12:17:38.206448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.569041 (+362593us) service_pool.cc:224] Handling call
0709 12:17:38.636861 (+ 67820us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:38.947786 (+310925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:38.947827 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:38.948548 (+   721us) log.cc:844] Serialized 26 byte log entry
0709 12:17:39.306660 (+358112us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:39.306708 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:39.306814 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:39.367056 (+ 60242us) raft_consensus.cc:1685] finished
0709 12:17:39.367084 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:39.367305 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68858,"raft.queue_time_us":154133,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":139,"thread_start_us":472403,"threads_started":2,"wal-append.queue_time_us":361762}
I20260709 12:17:39.394090 12645 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:39.321163 25662 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:39.396974 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 241267us
W20260709 12:17:39.398861 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 172797us
I20260709 12:17:39.401208 12572 rpcz_store.cc:276] 0709 12:17:37.601713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:37.610953 (+  9240us) service_pool.cc:224] Handling call
0709 12:17:37.611371 (+   418us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:37.679702 (+ 68331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:37.679731 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:37.692075 (+ 12344us) log.cc:844] Serialized 52 byte log entry
0709 12:17:37.809752 (+117677us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:17:37.809827 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:37.809954 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:38.881869 (+1071915us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:39.318995 (+437126us) raft_consensus.cc:1685] finished
0709 12:17:39.319037 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:39.331547 (+ 12510us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2740,"raft.queue_time_us":8502,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":1858,"spinlock_wait_cycles":28393856,"thread_start_us":87345,"threads_started":1,"wal-append.queue_time_us":157130}
W20260709 12:17:39.405099 25551 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.005s	user 0.016s	sys 0.000s
I20260709 12:17:39.393061 25490 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.415197 25551 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.417814 25551 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.418601 25551 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:39.419625 25359 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:39.388067 25658 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.423719 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:39.424872 25658 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:39.426065 25683 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:39.426651 25683 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:39.426998 25683 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:39.427124 25385 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.428375 25672 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.429205 25672 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.401149 25584 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:39.431461 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.431955 25672 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.434275 25463 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.435006 25650 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.434863 25463 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.435501 25662 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.435680 25650 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.436450 12638 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [41.21-43.22]
I20260709 12:17:39.438740 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4023) took 1232 ms. Trace:
I20260709 12:17:39.439034 13254 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:17:39.439109 12638 rpcz_store.cc:276] 0709 12:17:38.206728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.569673 (+362945us) service_pool.cc:224] Handling call
0709 12:17:38.646919 (+ 77246us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:39.436929 (+790010us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:39.438452 (+  1523us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:39.438680 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1815015168}
I20260709 12:17:39.441591 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:39.447682 13259 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:17:39.446656 25345 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:39.450789 25545 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.458503 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:39.459578 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:39.442698 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:39.460615 13260 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:39.443243 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:39.443354 12565 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:39.472174 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.474579 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:39.477859 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:39.476907 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:39.482797 13257 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 39 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 26.
I20260709 12:17:39.440898 25634 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:39.483968 13254 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:17:39.482612 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:39.495949 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:39.497450 13218 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.527s (SENT)
I20260709 12:17:39.498126 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:39.501011 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:39.502221 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 394913us
I20260709 12:17:39.522403 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.440660 25658 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
W20260709 12:17:39.537335 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:39.495666 25672 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:39.500829 25675 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.487540 25634 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:39.524186 25250 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.125s	user 0.020s	sys 0.016s
I20260709 12:17:39.500591 25693 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:39.524013 25677 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:39.554852 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109607us
I20260709 12:17:39.555966 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5281) took 1395 ms. Trace:
I20260709 12:17:39.487082 25669 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.062s
W20260709 12:17:39.552333 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129422us
I20260709 12:17:39.486785 25664 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.067s
I20260709 12:17:39.563664 12494 rpcz_store.cc:276] 0709 12:17:38.160568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.573259 (+412691us) service_pool.cc:224] Handling call
0709 12:17:38.894015 (+320756us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:38.894868 (+   853us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:39.555716 (+660848us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54000c44d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:39.555891 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":736771328}
I20260709 12:17:39.528187 25490 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:39.559602 25250 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.160s	user 0.021s	sys 0.016s
I20260709 12:17:39.548887 25678 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.548709 25682 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.540005 25676 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:17:39.579631 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209566us
W20260709 12:17:39.584108 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:39.560683 25634 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:39.558681 25693 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:39.588652 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66368.8us
I20260709 12:17:39.575860 25640 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:39.593024 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.579854 25463 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.600874 12638 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:39.576740 25597 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.018s
I20260709 12:17:39.587013 25676 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:39.604710 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103806us
I20260709 12:17:39.607255 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.561045 25675 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.592468 25693 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:39.593020 25650 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.582298 25356 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.023s
I20260709 12:17:39.575449 25677 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.608639 25676 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:39.599170 25698 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:39.592451 25689 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.021s
I20260709 12:17:39.623792 25226 ts_tablet_manager.cc:1434] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 1.397s	user 0.024s	sys 0.011s
I20260709 12:17:39.609381 25700 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:39.588514 25682 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.618492 25463 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:39.600557 25640 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.643329 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:39.595410 25678 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.647754 25683 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.627694 25698 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:39.657141 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 175326us
I20260709 12:17:39.654151 12643 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:39.652136 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.659070 12638 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:39.633021 25658 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:39.630008 25695 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:39.627418 25226 ts_tablet_manager.cc:1467] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Tablet startup took 1403ms
I20260709 12:17:39.662129 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.664125 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.662344 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:39.658205 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.668s (SENT) [suppressed 42 similar messages]
I20260709 12:17:39.682142 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.683542 12641 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:39.684337 12644 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:17:39.686064 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:39.686868 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 91582.6us
I20260709 12:17:39.686223 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.689875 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:39.689977 12643 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:39.690061 12637 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 37 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 19.
I20260709 12:17:39.692330 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:39.694712 12639 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
W20260709 12:17:39.695508 25707 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:17:39.660808 25698 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:39.699456 25678 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.700350 25640 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.700700 25675 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.658402 25463 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:39.701968 25463 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.23, Last appended by leader: 23, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.702425 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:39.703712 25682 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:39.704303 25696 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:17:39.708114 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.707396 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:39.710145 12644 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:39.713253 12462 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:17:39.715867 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.716593 12639 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:39.718026 25714 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:39.718971 25714 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:39.719462 25714 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:39.720505 25714 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 42.18, Last appended by leader: 18, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:39.722451 25702 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:17:39.722931 25698 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.713761 25650 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.015s
I20260709 12:17:39.743782 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:39.744532 12496 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:39.714643 25712 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:17:39.709904 25672 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:39.745466 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:39.685655 25708 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.066s
I20260709 12:17:39.753750 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:39.750842 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138608us
I20260709 12:17:39.687383 25695 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.759359 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:39.763484 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:39.760449 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.763558 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:39.766340 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.766726 12488 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:17:39.764341 12491 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:39.740700 25683 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.762725 12495 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:39.764698 12489 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:17:39.768338 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:39.770486 12639 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:39.726493 25698 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.770599 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:39.771826 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113669us
I20260709 12:17:39.773753 12637 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:39.780197 25677 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:39.785670 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78323.2us
I20260709 12:17:39.788583 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.789369 12644 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:39.768368 25682 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.795328 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.798877 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:39.801838 12639 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: [37.19-37.19]
I20260709 12:17:39.802033 12565 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:39.767789 25634 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.790177 25640 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.805227 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:39.808112 25640 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.811390 12491 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:17:39.821177 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:39.804117 25718 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:39.809717 25634 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:39.826375 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.812956 25683 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:39.829332 12646 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:39.801375 25682 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.820632 25189 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.835587 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:39.837555 12638 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:39.838613 12645 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:17:39.841231 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.836855 25640 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.845046 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.866144 12638 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:39.862483 25675 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.846997 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:39.867419 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.842113 25189 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:39.869899 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:39.870534 12646 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:39.871569 25189 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.852624 12493 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:39.862155 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.872738 12637 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:39.872786 12496 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:39.867434 12571 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:39.873824 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:39.865386 25695 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.867208 25675 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.850298 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:39.876161 12491 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:39.877197 25675 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:39.877328 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:39.879849 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:39.877916 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:39.884760 12565 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:17:39.887993 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47774.6us
I20260709 12:17:39.888435 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:39.889484 12496 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:39.893227 12644 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:39.897869 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
W20260709 12:17:39.906540 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:39.845597 25706 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.082s
I20260709 12:17:39.907888 25584 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.911513 25682 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:39.900852 25551 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:39.912851 25551 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.922356 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:39.926723 12565 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:39.901408 25597 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:17:39.935338 25666 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.124s	user 0.005s	sys 0.000s
W20260709 12:17:39.664237 25226 ts_tablet_manager.cc:1469] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Trace:
0709 12:17:24.458206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:24.458462 (+   256us) service_pool.cc:224] Handling call
0709 12:17:25.624314 (+1165852us) ts_tablet_manager.cc:934] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
0709 12:17:28.009684 (+2385370us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:17:28.637547 (+627863us) tablet_metadata.cc:665] Metadata flushed
0709 12:17:28.651565 (+ 14018us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:17:29.931691 (+1280126us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:17:30.034100 (+102409us) tablet_replica.cc:206] Creating consensus instance
0709 12:17:30.100042 (+ 65942us) ts_tablet_manager.cc:989] Replica registered.
0709 12:17:30.139468 (+ 39426us) inbound_call.cc:177] Queueing success response
0709 12:17:32.281792 (+2142324us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:17:38.242426 (+5960634us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:17:38.482545 (+240119us) tablet_replica.cc:261] Starting instrumentation
0709 12:17:38.483426 (+   881us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":798,"mutex_wait_us":4001,"raft.queue_time_us":12745,"spinlock_wait_cycles":9978240,"tablet-copy.queue_time_us":734653,"tablet-download-pool.queue_time_us":287,"tablet-download-pool.run_cpu_time_us":1629,"tablet-download-pool.run_wall_time_us":1018874,"thread_start_us":1014277,"threads_started":3}
I20260709 12:17:39.945987 12565 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:39.911028 25584 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:39.961607 25668 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.521s	user 0.007s	sys 0.005s
I20260709 12:17:39.962280 25668 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:39.968752 12637 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:17:39.971352 12641 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: [32.17-32.17]
I20260709 12:17:39.974574 12637 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
W20260709 12:17:40.001694 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 107676us
I20260709 12:17:39.953394 25676 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:40.010492 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96588us
W20260709 12:17:40.012887 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.018991 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4028) took 1373 ms. Trace:
I20260709 12:17:40.019562 12640 rpcz_store.cc:276] 0709 12:17:38.645355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.645502 (+   147us) service_pool.cc:224] Handling call
0709 12:17:40.018727 (+1373225us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:40.018899 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37471616}
W20260709 12:17:40.027752 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 229908us
I20260709 12:17:40.029075 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:40.031347 25730 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:40.031885 25730 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.032222 25730 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:40.033164 25189 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.034117 25189 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.041416 25525 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.041654 25189 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.050225 25500 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:40.056448 25658 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:40.057149 25658 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.058298 25441 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:40.058882 25441 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.059799 25441 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.061268 25441 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.062259 25676 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.066706 25693 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.063773 25658 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.071624 25658 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:40.073733 25689 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:40.073930 25525 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.074291 25689 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.080116 25672 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:17:40.083739 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54771.1us
I20260709 12:17:40.087131 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1620) took 1971 ms. Trace:
I20260709 12:17:40.087311 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:40.088223 13256 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:40.090548 12646 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:40.091077 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:40.091650 12638 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:40.095302 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.087684 13253 rpcz_store.cc:276] 0709 12:17:38.111606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.111782 (+   176us) service_pool.cc:224] Handling call
0709 12:17:38.112174 (+   392us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:38.112556 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:38.112586 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:38.126075 (+ 13489us) log.cc:844] Serialized 52 byte log entry
0709 12:17:38.221680 (+ 95605us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:38.221753 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:38.221846 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:38.571464 (+349618us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:40.080657 (+1509193us) spinlock_profiling.cc:243] Waited 6.92 ms on lock 0x7b24002cf9d8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:17:40.082896 (+  2239us) raft_consensus.cc:1685] finished
0709 12:17:40.082927 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:40.083147 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22351,"raft.queue_time_us":1792,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":7460,"spinlock_wait_cycles":102509696,"thread_start_us":87375,"threads_started":1,"wal-append.queue_time_us":197411}
W20260709 12:17:40.102675 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 71858.1us
I20260709 12:17:40.107132 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:40.112015 13256 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:17:40.114336 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.016s (SENT)
I20260709 12:17:40.114733 12637 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:40.114979 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.091054 25642 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.115521 12646 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:17:40.116642 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.124960 25597 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:40.126595 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-4d3b77689c31445cbba405c2b232495e received from {username='slave'} at 127.0.0.1:54234
I20260709 12:17:40.127303 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-4d3b77689c31445cbba405c2b232495e on tablet 4d3b77689c31445cbba405c2b232495e with peer 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.120476 25697 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:40.127913 12491 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: [40.29-40.29]
I20260709 12:17:40.129446 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.129333 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:40.130155 12495 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:40.104887 25112 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:40.131047 13259 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:40.131644 25112 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:40.139918 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.142942 25682 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:40.118341 25641 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.098400 25572 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.055s
I20260709 12:17:40.148586 12646 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:17:40.113407 25706 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.145699 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:40.166191 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:40.168228 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:40.154326 25646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.349s	user 0.026s	sys 0.033s
I20260709 12:17:40.102692 25558 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.053s
W20260709 12:17:40.170295 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55979.4us
W20260709 12:17:40.178535 13218 leader_election.cc:336] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.140s (SENT)
I20260709 12:17:40.179144 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.148816 25693 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:40.183159 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:40.173048 25646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.368s	user 0.027s	sys 0.033s
I20260709 12:17:40.185824 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.147876 25641 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.144354 25642 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.196732 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.197844 25713 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.198473 25713 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.199416 25642 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.200835 25713 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.202224 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.202976 12497 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:40.203477 25730 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.204058 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.207847 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:40.208873 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:40.209697 12568 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:40.210430 12563 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:17:40.210952 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:40.211621 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 29.
I20260709 12:17:40.196974 25112 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:40.204706 12497 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:40.180439 25235 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.011s
I20260709 12:17:40.163650 25706 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.215756 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:40.147277 25682 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.219273 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.184937 25500 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:17:40.223651 25625 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.416s	user 0.009s	sys 0.028s
I20260709 12:17:40.218230 25697 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:40.229449 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:40.232064 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:40.230110 25625 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.419s	user 0.009s	sys 0.028s
I20260709 12:17:40.225173 25645 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.237601 12497 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:40.227468 25500 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.233769 25640 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.023s
I20260709 12:17:40.231729 25713 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:40.242504 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.256580 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.269174 12492 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:40.269598 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.258911 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.256136 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.274600 12495 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:40.253880 25500 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.253991 25558 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:17:40.277042 25558 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.277454 25558 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:40.262468 25590 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:40.255501 25655 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.859s	user 0.009s	sys 0.005s
I20260709 12:17:40.261960 25713 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.283272 25713 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:40.287328 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.290366 25689 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.291555 25689 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:40.296026 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:40.243611 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.267535 12494 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:40.300081 12491 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:17:40.309384 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.241088 25730 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.248147 25682 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.249361 25641 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.322415 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.323975 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.324645 12492 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:17:40.326606 25500 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.329188 25500 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.334370 25645 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:40.338416 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.342870 25463 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:40.343443 25463 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:40.344507 25463 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.325563 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:40.351209 12497 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
W20260709 12:17:40.347604 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115884us
I20260709 12:17:40.355067 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:40.354269 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124866us
I20260709 12:17:40.359270 13258 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 33 LEARNER]: Advancing to term 34
I20260709 12:17:40.315022 25723 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:40.357813 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.358511 25463 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:40.367015 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.366962 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.371887 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.347180 25525 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.375203 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:40.357389 25682 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.012s
I20260709 12:17:40.358565 25545 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.012s
I20260709 12:17:40.366153 25645 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:40.358760 25359 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:17:40.369215 25723 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.360956 25536 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.012s
I20260709 12:17:40.384690 12488 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:17:40.377689 25641 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:40.375355 25572 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:40.374759 25693 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:40.387953 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:40.385810 25697 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.009s
I20260709 12:17:40.385560 25642 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.018s
I20260709 12:17:40.437315 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:40.429709 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:40.391959 25664 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:17:40.447827 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59991.1us
I20260709 12:17:40.448441 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:40.415630 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:40.452231 12488 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.439960 25640 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.410600 25545 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.455998 25640 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.456875 25545 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.460881 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:40.462705 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.462787 25694 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:17:40.464057 25694 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:40.465925 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.474736 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:17:40.478060 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.483690 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.492086 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.493743 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.495188 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:40.495973 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:17:40.402253 25682 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:40.497408 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:40.451531 25664 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.417120 25558 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.499115 25690 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:40.500422 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.500965 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:40.501183 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.451756 25359 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:40.503436 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:40.504052 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:40.464442 25694 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.489399 25590 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:40.506932 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.495034 25493 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:40.492646 25693 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:40.520476 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:40.518889 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.518055 12488 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:40.500053 25508 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:40.524801 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:40.509763 25716 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:40.510536 25664 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.492208 25112 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:17:40.530472 12491 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 40 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 23.
I20260709 12:17:40.514537 25711 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:40.538710 25545 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.526691 25441 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:40.544616 25750 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:17:40.544919 25441 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.515143 25359 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.517349 25669 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:40.415098 25723 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.533375 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:40.542979 25658 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:40.564893 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:40.541677 25689 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:17:40.561676 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.546622 25441 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:40.569175 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:40.545028 13262 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:40.573307 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:40.548300 25359 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:40.577755 25690 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:40.510409 25682 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.531718 25693 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.594394 12491 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 24. (term mismatch)
I20260709 12:17:40.595059 12491 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 23
I20260709 12:17:40.595439 12491 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 38.24
I20260709 12:17:40.580869 25711 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.002s
I20260709 12:17:40.581959 25508 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:17:40.565158 25545 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.565047 25536 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:40.577966 25572 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:40.583338 25699 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:17:40.583577 25721 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.754s	user 0.004s	sys 0.001s
I20260709 12:17:40.601686 25545 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.583447 25713 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.603741 13260 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:17:40.604135 25545 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.604660 25713 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.570814 25689 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.585671 25668 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:40.586329 25112 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:17:40.605511 25536 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.607976 25536 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:40.609256 25572 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:40.587281 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.610311 25558 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.610770 25716 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:40.580305 25690 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.613296 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:40.547107 25525 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.549526 25669 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.613706 25690 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.614490 25669 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:40.595966 12491 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:40.598794 13262 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:17:40.605372 25689 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:40.619875 25722 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:40.621023 25693 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:40.621433 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 801164us
I20260709 12:17:40.621730 25112 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:40.622303 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.623656 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.624768 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.600809 25699 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.626868 25699 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.628180 25699 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:40.628825 25650 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:40.629364 25650 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:40.641491 25683 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:40.652096 25728 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:40.655828 12568 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:40.635350 25634 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:40.665534 12567 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:17:40.663429 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:40.668226 12568 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
W20260709 12:17:40.671521 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46814us
I20260709 12:17:40.675377 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:40.685420 12572 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:40.657656 25640 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.688993 12564 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:40.544621 25694 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.21, Last appended by leader: 21, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.691895 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:40.702271 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:40.703361 12570 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.705727 25684 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.710826 12565 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:40.610312 25572 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.681551 25631 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:40.684284 25672 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.717356 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:40.674304 25549 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:17:40.707394 25536 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.732297 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.710417 25711 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.737983 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:40.706451 25684 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:40.760921 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.708182 25650 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.771814 25682 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:40.750211 25668 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:17:40.778074 25359 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.728964 25631 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.783036 25631 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.741914 25536 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.781419 25551 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:40.790907 12572 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:40.791185 12644 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:40.791772 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:40.792541 12564 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:40.793387 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:40.794178 12563 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:40.795302 25684 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.796128 25690 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:40.747794 25753 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:17:40.796727 25690 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:40.796886 25753 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:40.747646 25590 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:40.798491 25590 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:40.798022 25753 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.798877 25590 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:40.802531 25668 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:40.803668 25359 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:40.731134 25672 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.806396 25672 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:40.808887 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.811540 25189 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:40.812211 25189 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:40.802942 25684 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:40.737200 25112 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:17:40.811668 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:40.802449 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:40.799537 25753 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:17:40.825872 12496 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:40.843762 12563 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:40.845269 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:40.859678 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:40.873168 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70776.6us
I20260709 12:17:40.848389 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:40.822997 25359 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:40.917042 25359 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.918004 25658 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:40.893445 25641 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:40.976464 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 30.
I20260709 12:17:40.894992 12569 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
W20260709 12:17:40.952071 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:40.746973 25711 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.813314 25189 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:40.978998 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:40.950702 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76273.6us
I20260709 12:17:40.888154 25112 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:40.962236 12496 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:40.831116 25689 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.012053 25689 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:40.987366 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:41.013367 25642 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:17:41.014068 25642 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.015287 25642 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.008927 25495 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:41.028019 25642 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:41.035565 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.037320 12496 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:41.042287 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.043507 12569 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:40.919327 25640 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.127s
I20260709 12:17:40.947602 25664 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:40.986405 25641 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.071014 25641 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:41.075814 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.050462 25682 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:41.083458 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.084224 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.091491 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.067569 25441 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.094065 12496 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:41.115549 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:41.118288 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2012) took 3249 ms (client timeout 1641 ms). Trace:
I20260709 12:17:41.120652 12496 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:41.120704 12493 rpcz_store.cc:269] 0709 12:17:37.868601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:37.868895 (+   294us) service_pool.cc:224] Handling call
0709 12:17:41.118050 (+3249155us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:41.118194 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4531772160}
I20260709 12:17:41.118818 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:41.124306 12565 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
W20260709 12:17:41.125126 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1623) took 2820 ms (client timeout 1660 ms). Trace:
W20260709 12:17:41.125788 13255 rpcz_store.cc:269] 0709 12:17:38.304505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:38.304734 (+   229us) service_pool.cc:224] Handling call
0709 12:17:41.124730 (+2819996us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:41.124973 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4591969664}
I20260709 12:17:41.132602 12496 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
W20260709 12:17:41.116825 25669 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.001s
I20260709 12:17:41.118279 25495 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:41.121029 25761 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
W20260709 12:17:41.154930 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134192us
I20260709 12:17:41.158783 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 4965) took 1476 ms. Trace:
I20260709 12:17:41.159448 12645 rpcz_store.cc:276] 0709 12:17:39.682476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:39.706412 (+ 23936us) service_pool.cc:224] Handling call
0709 12:17:39.838247 (+131835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:39.838990 (+   743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:41.158490 (+1319500us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b5400153950. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:41.158701 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":299988352}
I20260709 12:17:41.148593 12497 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: [40.25-40.25]
I20260709 12:17:41.092355 25682 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.172345 25711 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:41.174639 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.175518 12565 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
W20260709 12:17:41.177790 25762 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:17:41.179629 25441 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:41.180490 25765 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:17:41.185657 12568 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [32.19-34.20]
W20260709 12:17:41.192713 25634 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.004s
W20260709 12:17:41.193222 25728 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.540s	user 0.008s	sys 0.003s
I20260709 12:17:41.149258 25669 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.196795 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:41.202705 25747 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:41.210608 25664 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:41.211319 25495 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.215950 25536 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:41.229576 25723 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:41.229996 25597 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.006s
I20260709 12:17:41.236152 25597 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:41.235617 25723 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:41.229701 25763 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:17:41.225214 25728 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:41.217136 25359 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:41.218937 25634 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:41.221032 25709 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:41.225807 25747 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:41.246675 25767 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:17:41.256491 12565 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:41.242507 25723 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:41.257156 12565 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:41.251765 25590 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.261721 12490 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:41.261181 12565 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.262681 12565 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:41.269893 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.270493 12569 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:41.271643 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:41.266943 25112 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:41.280606 12565 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 21. (term mismatch)
I20260709 12:17:41.281016 12565 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:17:41.281219 12565 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 38.21
I20260709 12:17:41.281438 12565 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:41.282413 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.267400 25706 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:41.283932 25716 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:41.255402 25709 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:41.298733 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.299099 25682 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.301924 25684 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:41.302595 25684 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.304828 25669 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.305044 25684 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:41.307726 25709 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:41.308454 25723 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.310004 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.312800 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.277128 25641 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:41.277339 25747 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 40.29, Last appended by leader: 29, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.323730 25590 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.324748 25747 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:41.325898 25723 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:41.328997 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.329747 12565 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:41.332643 13256 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:41.334800 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:41.335585 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:41.339373 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:41.323022 25641 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:41.340054 12638 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:41.343577 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:41.344841 12644 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: [32.19-34.20]
I20260709 12:17:41.344995 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:41.344614 25690 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:41.343410 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:41.355592 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:41.346447 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:41.357715 12645 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:41.353420 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:41.358445 13255 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:41.353049 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:41.336233 25669 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.362525 25669 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.362885 25699 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:41.331039 25694 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:41.360864 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:41.357295 25572 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:41.368633 25753 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:41.369087 25493 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:41.369109 25753 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:41.371124 25753 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:41.340646 25684 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.043s
I20260709 12:17:41.383682 25493 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.386054 12565 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:41.378029 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:41.390801 13255 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:17:41.396366 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:41.359158 25641 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:41.392241 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:41.369026 25572 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:41.400550 25774 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:17:41.400971 25716 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:41.407371 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:41.366528 25699 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:41.387547 25441 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.596238 13255 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:41.413278 25723 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:41.501057 25737 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:41.597021 25737 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.597110 25723 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:41.598148 25723 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.398471 25493 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.600152 12638 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:17:41.505219 25641 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.607753 25641 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:41.475674 25775 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:17:41.409549 25664 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:41.609826 25664 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:41.610877 25664 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.620592 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:41.514525 25572 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.516261 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:41.518157 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:41.521032 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1879) took 2071 ms (client timeout 2226 ms). Trace:
I20260709 12:17:41.522413 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1883) took 1063 ms. Trace:
I20260709 12:17:41.499167 25711 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:41.529235 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.627902 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:41.628981 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:41.529678 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 854327us
I20260709 12:17:41.533501 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.630880 25536 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:41.631425 12488 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:41.631582 25536 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:41.632164 25536 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:41.632782 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.533690 12571 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:41.634155 25714 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:41.634963 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.635147 25714 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:41.636587 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.637055 25714 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:41.534407 25463 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:41.535094 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:41.633524 12488 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:41.638164 12491 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:41.535326 12565 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:41.535923 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.536505 25676 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:41.537247 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:41.640247 12494 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:41.537705 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.641403 12564 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:41.538043 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.538616 25677 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.538753 12570 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:17:41.539386 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.540573 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.540697 25753 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:41.541203 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.544281 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.545974 25189 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:41.658284 12495 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 34 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:17:41.644579 12568 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:41.548358 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.552662 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2019) took 1445 ms. Trace:
I20260709 12:17:41.660585 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:41.550556 25773 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.337s	user 0.003s	sys 0.000s
I20260709 12:17:41.563833 25112 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:41.571152 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2028) took 1190 ms. Trace:
I20260709 12:17:41.666090 12497 rpcz_store.cc:276] 0709 12:17:40.380989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:40.628158 (+247169us) service_pool.cc:224] Handling call
0709 12:17:40.628498 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:41.164444 (+535946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:41.570714 (+406270us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:41.571057 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164425088}
W20260709 12:17:41.666676 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43513.2us
I20260709 12:17:41.575783 25690 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:41.577641 25766 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.382s	user 0.004s	sys 0.000s
I20260709 12:17:41.669431 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.578032 25645 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:41.582523 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:41.587037 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:41.587494 25684 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.587718 25689 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.588405 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:41.590054 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:41.595908 25441 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.679582 25441 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.405841 25669 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:41.598104 25737 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.680689 25737 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:41.681787 25790 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:41.682472 25790 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:41.683477 25790 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.412074 25716 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:41.685938 25716 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:41.601406 25493 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.503116 25699 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.690814 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.692214 12569 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 34 because replica has last-logged OpId of term: 32 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 16.
I20260709 12:17:41.601619 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:41.625075 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.630820 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:41.646199 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.627314 13260 rpcz_store.cc:276] 0709 12:17:40.458742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:40.458903 (+   161us) service_pool.cc:224] Handling call
0709 12:17:40.459241 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:40.604205 (+144964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:41.522061 (+917856us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:41.522342 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":331630464}
W20260709 12:17:41.639328 13259 rpcz_store.cc:269] 0709 12:17:39.449396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:39.458667 (+  9271us) service_pool.cc:224] Handling call
0709 12:17:41.520756 (+2062089us) spinlock_profiling.cc:243] Waited 643 ms on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:41.520928 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1478067456}
I20260709 12:17:41.639523 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:41.559805 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.662365 12490 rpcz_store.cc:276] 0709 12:17:40.107222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:40.108530 (+  1308us) service_pool.cc:224] Handling call
0709 12:17:41.118759 (+1010229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:41.262233 (+143474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:41.551430 (+289197us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:41.552435 (+  1005us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54000f78d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:41.552598 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2638649088}
I20260709 12:17:41.664404 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:41.694444 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55058.9us
I20260709 12:17:41.694818 12570 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:41.694981 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:41.695343 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:41.664630 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.696386 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.697235 12563 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:41.697438 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.665045 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:41.700205 12571 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:41.658350 25572 raft_consensus.cc:2764] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election decision vote started in defunct term 35: won
I20260709 12:17:41.698542 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
W20260709 12:17:41.667022 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50357.8us
I20260709 12:17:41.668896 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.637473 25714 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:41.672526 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90111.5us
I20260709 12:17:41.677307 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.705947 12565 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:41.707401 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:41.701856 25441 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:41.707764 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.708930 12563 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:41.709043 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.667423 25112 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:41.710062 12567 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:41.710047 12494 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:17:41.714085 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49130.7us
I20260709 12:17:41.717869 25675 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:17:41.720168 25714 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.720819 25753 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:41.724622 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:41.714174 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.727106 25682 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:41.727329 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:41.727444 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96698.9us
I20260709 12:17:41.728073 12565 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:41.728777 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:41.729229 25646 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:41.730451 25646 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:41.731189 25646 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:41.731630 25646 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:41.707733 25359 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:41.732306 25359 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:41.732712 25359 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:41.732586 25646 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.725749 25714 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:41.738579 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:41.729235 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:41.740203 12568 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:41.740984 12565 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 16.
I20260709 12:17:41.748811 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:41.751071 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.759953 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.758627 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:41.761059 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:41.691294 25495 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:41.764321 12570 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 29. Current term is 30. Ops: []
W20260709 12:17:41.767644 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103647us
I20260709 12:17:41.767932 12571 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:17:41.663511 25676 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.768500 12568 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:41.700088 25669 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:41.766419 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:41.665355 25677 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:41.779376 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.668081 25711 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.780627 12568 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:17:41.784579 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:41.784993 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:41.790657 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:41.791481 12563 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:41.759776 25748 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:41.687892 25645 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.799959 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:41.813055 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:41.815656 12570 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:41.804275 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:41.798709 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:41.833402 13253 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:41.806210 25676 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:41.687656 25684 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.692430 25689 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:41.848986 25558 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:17:41.768494 25235 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.770817 25189 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.061s
I20260709 12:17:41.802593 12568 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:17:41.753891 25759 raft_consensus.cc:2764] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election decision vote started in defunct term 36: won
I20260709 12:17:41.810721 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:41.834381 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:41.836510 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45912.8us
I20260709 12:17:41.782059 25675 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:41.840832 12566 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:41.854281 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:41.774642 25631 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.065s
I20260709 12:17:41.863612 12493 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 18.
W20260709 12:17:41.846522 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47964.5us
I20260709 12:17:41.837915 25572 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.018s
I20260709 12:17:41.848035 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:41.830289 25597 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:41.795254 25385 raft_consensus.cc:2764] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:17:41.852198 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5308) took 1371 ms. Trace:
I20260709 12:17:41.816751 12564 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:41.846304 25728 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:41.855795 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:41.856324 13253 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:17:41.764385 25441 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:41.842128 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70236.5us
I20260709 12:17:41.837234 25795 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.025s
I20260709 12:17:41.781323 25646 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:41.796083 25495 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.858803 25709 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:41.810329 25757 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:41.818605 25493 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:41.869269 25694 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:41.926365 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:41.867331 12494 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:41.927316 13253 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 40 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:17:41.796216 25650 raft_consensus.cc:2764] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election decision vote started in defunct term 38: won
I20260709 12:17:41.869858 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:17:41.868423 12492 rpcz_store.cc:276] 0709 12:17:40.480396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:40.777222 (+296826us) service_pool.cc:224] Handling call
0709 12:17:40.777617 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:40.778043 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:40.778072 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:40.778612 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:17:41.120047 (+341435us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:41.159693 (+ 39646us) log.cc:844] Serialized 14 byte log entry
0709 12:17:41.167098 (+  7405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:41.167212 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:41.419283 (+252071us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b18003ec920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:41.576672 (+157389us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:41.851840 (+275168us) raft_consensus.cc:1685] finished
0709 12:17:41.851880 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:41.852095 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":38469,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":16008,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":3323,"spinlock_wait_cycles":54241280,"thread_start_us":341052,"threads_started":1,"wal-append.queue_time_us":392723}
I20260709 12:17:41.867483 25748 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:41.863937 25235 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.870612 25463 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:41.857903 25675 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:41.896368 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 129908us
I20260709 12:17:41.843938 25747 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.029s
W20260709 12:17:41.932646 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57201.8us
W20260709 12:17:41.940619 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 278359us
I20260709 12:17:41.949705 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:41.941491 25806 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:17:41.958714 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:41.955440 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:41.926025 25641 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:17:41.964601 25747 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:41.965111 25747 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.897502 25757 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:41.962709 12494 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:41.984522 12568 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I20260709 12:17:41.934644 25748 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.892958 25797 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.075s
W20260709 12:17:42.000278 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50961.2us
I20260709 12:17:41.950304 25675 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:42.003073 25495 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:41.995419 25757 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:41.958326 25463 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.006266 25677 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.005692 25748 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.007207 25645 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:42.010433 25235 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.012761 25536 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:42.015769 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161584us
I20260709 12:17:42.023669 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.046463 12565 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:41.997049 25709 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:41.983425 25802 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:17:42.041479 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 228092us
I20260709 12:17:42.036626 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:42.057066 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.021101 25801 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:42.060460 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.056349 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:42.064589 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:42.067629 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65071.7us
W20260709 12:17:42.074575 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 218920us
I20260709 12:17:42.079162 12497 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I20260709 12:17:42.082086 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.083469 12494 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:42.058687 25806 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.021s
I20260709 12:17:42.085362 12491 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:42.063675 25801 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:42.071611 25668 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:42.077054 25711 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.104681 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:42.112015 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.113452 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:42.112560 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.114378 13253 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 23.
I20260709 12:17:42.114734 12568 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 40 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:17:42.092015 25815 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:42.120260 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.104122 25805 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.030s
I20260709 12:17:42.141675 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.112406 25690 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.151322 25690 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.116245 25668 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.156096 25668 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.118966 25815 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:42.108384 25706 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.050s
I20260709 12:17:42.112138 25536 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:42.157696 25813 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.000s	sys 0.002s
I20260709 12:17:42.158160 25815 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:42.168136 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.168512 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.119189 25631 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:42.178900 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:42.167404 25441 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:42.188694 25737 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.004s
I20260709 12:17:42.206092 25463 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.206596 25789 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:42.208173 25789 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.158322 25804 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.116706 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:42.214754 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:42.215378 12495 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:42.216001 12495 raft_consensus.cc:614] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:42.127529 25493 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.070s
W20260709 12:17:42.221879 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 161493us
I20260709 12:17:42.225669 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:42.226797 25631 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:42.227247 25631 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:42.200778 25737 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.197921 25690 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.235976 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.227720 25631 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:42.219977 25802 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:42.127703 25709 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.236227 25359 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.238862 25737 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.242156 25634 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.007s
I20260709 12:17:42.248091 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:42.132427 25801 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 43.22, Last appended by leader: 22, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.254257 12565 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 43 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 25.
I20260709 12:17:42.264650 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.265661 25634 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:42.266130 25634 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:42.266475 25634 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:42.266958 25714 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.532s	user 0.007s	sys 0.011s
W20260709 12:17:42.259346 25716 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.573s	user 0.008s	sys 0.006s
I20260709 12:17:42.267797 25716 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:42.269366 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.272718 25737 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.273617 13253 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:42.239929 25677 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.021s
W20260709 12:17:42.261718 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62755.9us
I20260709 12:17:42.222942 25804 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.219717 25789 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.281914 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:42.255343 25811 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:42.255532 25823 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:42.257464 25493 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.274606 25802 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.286773 25801 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:42.294446 25716 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.302820 25753 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:42.303610 25753 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:42.293553 25112 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.528s	user 0.011s	sys 0.004s
I20260709 12:17:42.312001 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:42.312966 12495 raft_consensus.cc:606] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:42.313472 12495 raft_consensus.cc:614] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:42.293287 25757 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.300069 25493 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.298792 25677 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:42.341166 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.304921 25753 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.342061 12497 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:42.296144 25811 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.343686 12496 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:42.346410 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.347589 12488 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:42.323874 25714 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.320990 25112 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.366911 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.373811 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:42.397244 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.397969 12495 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:42.406095 12495 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 18. Preceding OpId from leader: term: 43 index: 19. (index mismatch)
W20260709 12:17:42.406781 25827 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:17:42.407347 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.408834 13259 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c56dde469374f4da7440f134140670c"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:42.408119 12495 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:42.409710 13259 raft_consensus.cc:606] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:42.410269 13259 raft_consensus.cc:614] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:42.410772 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.411826 12495 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:17:42.409390 25757 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.416133 13260 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:17:42.422091 12488 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:42.397425 25753 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:42.446699 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.425659 12571 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:42.459785 12488 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: [35.20-35.20]
I20260709 12:17:42.361202 25714 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.464349 25815 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.474464 25804 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:42.455479 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.467644 25795 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:42.481585 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:42.482664 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.472915 25359 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:42.488511 25789 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:42.490953 25825 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:17:42.460580 25820 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.026s
I20260709 12:17:42.476470 25634 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.495539 25684 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.496640 25634 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.498585 25748 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:42.472244 25714 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.502404 25699 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:42.502735 12488 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:17:42.506690 25811 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.485996 25441 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.508059 25441 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.506942 25815 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:42.511549 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.514947 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.515712 12488 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:42.517477 25689 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.518599 25493 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.521299 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.522408 12488 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:42.524652 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.527031 25807 raft_consensus.cc:2764] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:17:42.527786 25807 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:42.528553 25675 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:42.531009 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 581439us
I20260709 12:17:42.531565 25631 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.531908 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:42.532955 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.533460 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.534403 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:42.534449 25759 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:42.535471 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 580093us
I20260709 12:17:42.535274 25759 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.536293 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:42.537480 25631 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.537904 25759 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.538162 25753 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:42.531536 12488 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:42.535300 25807 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:42.539502 25807 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.539692 25642 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.539919 25807 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:42.541168 25642 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.541667 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.542284 25796 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:42.542552 25675 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:42.542778 25796 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.543460 25675 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:42.543665 25642 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.544414 25675 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.545711 25818 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.546337 25818 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.546520 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.548007 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.548770 25818 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.549865 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.550697 12645 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 23.
I20260709 12:17:42.551558 12640 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:17:42.552346 25359 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:42.552663 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.552913 25359 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:42.553305 12639 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:42.553896 25359 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.542547 12488 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:42.557593 12497 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:17:42.521445 12492 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: [40.25-40.25]
I20260709 12:17:42.566653 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:42.525493 12493 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:42.571943 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.543327 25796 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:42.572853 12488 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:42.545008 25675 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.577123 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.538782 25753 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:42.581856 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.541891 25800 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:42.585023 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:42.587917 13259 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:42.585187 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.589715 12644 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:42.586208 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.590458 12639 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:42.585296 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:42.598006 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:42.597913 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.599177 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:17:42.599340 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:42.574631 25664 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.602357 12638 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
W20260709 12:17:42.605037 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.603807 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:42.576655 25572 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.581102 25493 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:42.616765 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.627616 25675 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:42.628516 25675 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:42.632078 25753 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 37.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.634939 25818 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:42.635819 25818 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:42.636384 25818 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:42.642467 25808 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.680s	user 0.003s	sys 0.000s
I20260709 12:17:42.605964 25812 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:42.644243 25812 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.644721 25812 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:42.646109 25676 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:17:42.648181 25819 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.475s	user 0.000s	sys 0.001s
I20260709 12:17:42.618806 25572 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.602952 25590 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:42.606171 25536 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:42.648931 25631 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:42.687603 12493 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:17:42.641744 25493 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.692324 25493 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.652362 25676 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:42.657433 25590 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:42.614271 25664 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:42.714581 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 149996us
W20260709 12:17:42.702499 25790 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.971s	user 0.004s	sys 0.011s
I20260709 12:17:42.688231 25631 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.637531 25818 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.723778 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:42.706564 25805 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.710191 25493 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:42.730072 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132140us
I20260709 12:17:42.736218 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:42.721159 25790 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s	user 0.005s	sys 0.011s
I20260709 12:17:42.739073 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.722275 25631 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:42.759009 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.762694 25690 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.003s
I20260709 12:17:42.768432 12643 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:17:42.769892 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.735661 25590 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.772348 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.772943 25590 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:42.773391 12637 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
W20260709 12:17:42.732698 25682 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.995s	user 0.006s	sys 0.012s
W20260709 12:17:42.736794 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 137522us
I20260709 12:17:42.761765 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.783958 12640 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:17:42.762024 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.762801 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:42.774614 25572 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.760737 25711 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:42.744388 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:42.760486 25359 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:42.781941 12639 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:42.785301 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:42.738703 25805 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:42.795442 25837 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
W20260709 12:17:42.783382 25682 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.055s	user 0.006s	sys 0.012s
I20260709 12:17:42.741189 25814 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.731591 25684 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:42.798347 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.802021 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:42.803148 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58871.4us
I20260709 12:17:42.803483 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.803678 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.806360 25572 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:42.793134 25845 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:42.809379 25843 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:17:42.741928 25493 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:17:42.800460 25842 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:17:42.812000 12637 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:17:42.815500 25841 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:17:42.820525 12644 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
W20260709 12:17:42.822156 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.823843 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:42.824038 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.829649 25805 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.825179 25845 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.841424 12493 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:42.833138 25536 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:42.834517 25441 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.835011 25664 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.842945 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.807211 25829 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:42.845150 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.838883 25493 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.846025 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:42.847175 25829 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.847636 25829 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:42.854068 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.856688 25807 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.858628 12645 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:42.860092 25748 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:42.860667 25847 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:17:42.860980 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.872133 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.822798 25359 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:42.874924 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.871346 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:42.846168 25845 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:42.884796 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:42.887938 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:42.888211 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.888860 12639 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:17:42.889046 12644 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:42.819375 25814 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.891836 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:42.866374 25684 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:42.869863 25650 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.891189 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.893774 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:42.894120 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:42.895637 12536 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:17:42.896333 12536 leader_election.cc:403] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20260709 12:17:42.893064 13260 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:42.884575 25493 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.860878 25748 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.900825 25798 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:42.896034 25845 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:42.899397 25796 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.896060 25684 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.889699 25823 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.025s
I20260709 12:17:42.902988 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:42.907454 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:42.924189 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.924858 12639 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:42.908630 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:42.904685 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:42.927012 25801 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:42.927440 25801 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.927579 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:42.927871 25801 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:42.928639 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:42.874622 25675 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:17:42.918079 25849 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:42.930081 25849 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.932726 25849 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:42.933504 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5143) took 1401 ms (client timeout 1655 ms). Trace:
W20260709 12:17:42.934202 12566 rpcz_store.cc:269] 0709 12:17:41.532523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:41.540320 (+  7797us) service_pool.cc:224] Handling call
0709 12:17:42.933254 (+1392934us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:42.933418 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455561088}
W20260709 12:17:42.936224 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5148) took 1395 ms (client timeout 1767 ms). Trace:
I20260709 12:17:42.936897 25823 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:42.887604 25807 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:42.899853 25650 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.915092 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:42.897439 25699 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:17:42.936756 12572 rpcz_store.cc:269] 0709 12:17:41.540660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:41.548023 (+  7363us) service_pool.cc:224] Handling call
0709 12:17:42.935995 (+1387972us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:42.936140 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344830336}
I20260709 12:17:42.944249 12641 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
W20260709 12:17:42.939877 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2033) took 1351 ms (client timeout 1762 ms). Trace:
W20260709 12:17:42.950783 12489 rpcz_store.cc:269] 0709 12:17:41.588030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:41.588152 (+   122us) service_pool.cc:224] Handling call
0709 12:17:42.939617 (+1351465us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54000f87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:42.939777 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":446125824}
I20260709 12:17:42.952353 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:42.952704 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.953430 12491 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:42.943135 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:17:42.942075 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.943352 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:42.954833 12488 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:42.955108 12493 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:42.944365 12495 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:17:42.918207 25846 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.953189 12489 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:42.961372 25748 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.942224 25805 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:42.965175 25699 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:42.965197 25805 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.967177 25189 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:42.912245 25493 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.967869 25189 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:42.968341 12488 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:42.964280 25631 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.970194 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:42.971050 12565 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:42.966681 25699 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.974529 25846 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.975821 25493 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:42.977909 25747 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.993374 25747 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:42.995916 25675 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:42.996474 25675 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.996073 25829 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.997323 25845 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:42.997795 25845 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:42.997802 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51015.4us
I20260709 12:17:42.998811 25845 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.000355 25845 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:17:42.983089 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74502.8us
I20260709 12:17:42.922465 25684 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:43.003849 25676 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:43.003096 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.937876 25646 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:42.986586 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:42.914414 25812 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:42.996899 25675 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:42.940151 25650 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.002065 25805 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.003096 25829 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.015025 25646 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:42.977331 25189 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:42.982334 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.000407 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:43.011683 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:42.937700 25823 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:43.015326 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:43.005393 25676 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:43.022336 12572 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 26. Preceding OpId from leader: term: 44 index: 26. (term mismatch)
I20260709 12:17:43.035398 12572 pending_rounds.cc:85] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 25
I20260709 12:17:42.991926 25748 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.030042 12639 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:17:43.035804 12572 pending_rounds.cc:107] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.26
I20260709 12:17:43.031054 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.037243 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.039391 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.040014 12572 raft_consensus.cc:2889] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:43.039948 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:43.044713 12491 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:17:43.045850 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.049117 12643 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:43.050370 12493 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:43.052124 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.021927 25646 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:43.056016 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:43.035538 25728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.014s	sys 0.016s
I20260709 12:17:43.063067 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:43.064816 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:43.072638 12491 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:43.037985 25682 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:43.080919 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.086086 25816 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:43.091358 12493 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:43.022888 25493 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.037s
I20260709 12:17:43.099095 25846 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:43.106884 25112 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.113833 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.114854 12571 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:17:43.115770 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.116366 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.116943 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.117331 12572 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:43.055326 25748 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.123351 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.124349 12568 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:43.124466 25748 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.125105 25748 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:43.122078 25112 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.057519 25806 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:43.059415 25189 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:43.075133 25737 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:43.078568 12490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:43.122256 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.073710 12565 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
W20260709 12:17:43.071981 25728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.127s	user 0.015s	sys 0.017s
I20260709 12:17:43.162850 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.126221 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.125618 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.130048 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.169061 12491 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:43.170734 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:43.171569 12491 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:43.135725 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.125943 25642 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:43.173116 25642 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.173529 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.174367 12493 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:43.128324 25796 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:43.177011 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47091.2us
I20260709 12:17:43.175239 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.178283 12493 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 20.
I20260709 12:17:43.178074 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.173075 12643 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:43.151580 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:43.180464 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.154178 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5003) took 1303 ms. Trace:
I20260709 12:17:43.139750 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.183138 12637 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:43.154800 12494 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:43.155603 12570 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:43.137671 12572 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:43.187494 12567 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:43.181672 12646 rpcz_store.cc:276] 0709 12:17:41.850508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:41.850706 (+   198us) service_pool.cc:224] Handling call
0709 12:17:41.851206 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:41.851558 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:41.866308 (+ 14750us) log.cc:844] Serialized 14 byte log entry
0709 12:17:41.866825 (+   517us) log.cc:844] Serialized 14 byte log entry
0709 12:17:42.522743 (+655918us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b180019c360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf96c758f 00007ffbf2706571 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd
0709 12:17:42.809197 (+286454us) spinlock_profiling.cc:243] Waited 2.91 ms on lock 0x7b180019c360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:43.153653 (+344456us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b180019c360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:17:43.153776 (+   123us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:43.153830 (+    54us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:43.153884 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:43.153958 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:43.154106 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16812,"prepare.queue_time_us":957954,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":2132,"raft.queue_time_us":660869,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":11630,"spinlock_wait_cycles":104673792,"thread_start_us":40425,"threads_started":1}
I20260709 12:17:43.149463 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.146719 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.154642 12639 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:43.190843 12493 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:17:43.153954 13260 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:43.181116 12568 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
W20260709 12:17:43.198036 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.166447 12571 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:43.201603 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.202721 12563 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:43.208766 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.208794 12572 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:43.210779 12572 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:43.214326 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:43.218564 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.219054 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.220168 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:43.225878 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:43.226331 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.234637 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.236255 12568 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:43.126590 25857 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:17:43.117558 25812 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:43.177274 25728 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:43.273047 25728 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.211781 12572 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.275987 12572 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:43.128134 25493 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:43.277262 25493 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.279564 25493 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.190770 25809 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.280441 25809 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.282734 25809 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:43.175609 25642 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.134744 25631 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.128891 25112 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.134546 25684 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.287081 25112 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.287899 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:43.288545 12494 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:43.289866 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.291188 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.194943 25805 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:43.292243 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.292798 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.294010 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.294574 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:17:43.297749 12572 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 20. (term mismatch)
I20260709 12:17:43.298199 12572 pending_rounds.cc:85] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:17:43.298408 12572 pending_rounds.cc:107] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 35.20
I20260709 12:17:43.134009 25806 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.302157 25806 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:43.139112 25858 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:17:43.122361 25855 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:17:43.247359 12638 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:43.255089 12491 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:43.227061 12571 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39. Re-sending same reply.
I20260709 12:17:43.256479 25831 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:43.129112 25798 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:17:43.258780 12489 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:43.311654 12571 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:17:43.258553 25801 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.259553 25495 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.312764 25495 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.315002 25801 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.313138 12563 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:43.314218 12570 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:43.315256 25495 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:43.134423 25814 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.128582 25823 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.125598 25748 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.319607 25748 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.320240 25790 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:43.320936 25631 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:43.321303 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119802us
I20260709 12:17:43.191433 25829 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:43.321801 25631 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.322286 25829 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.323494 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:43.324252 25831 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election decision vote started in defunct term 40: won
I20260709 12:17:43.324489 25829 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.325196 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:43.325488 25801 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.326234 25801 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.326916 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:43.244155 25572 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:43.327847 25829 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:43.327795 25572 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.328331 25829 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.328702 25829 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:43.328989 25801 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.332432 12566 raft_consensus.cc:1217] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Deduplicated request from leader. Original: 33.22->[35.23-35.23]   Dedup: 35.23->[]
W20260709 12:17:43.154117 25800 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.554s	user 0.014s	sys 0.000s
I20260709 12:17:43.335583 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:43.353219 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:43.333436 12566 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:17:43.355876 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.357545 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.359089 12566 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:43.343379 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.360260 12568 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:43.298629 12572 raft_consensus.cc:2889] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:43.339389 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.361595 12494 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:43.363572 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.324626 25631 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.147881 25737 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.368242 12490 raft_consensus.cc:606] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.437038 12490 raft_consensus.cc:614] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:43.368311 12639 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:43.375690 12496 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:43.379310 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5145) took 1846 ms. Trace:
I20260709 12:17:43.401051 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.404493 12646 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:43.373463 25801 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:43.326247 25493 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:43.410864 12640 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: [35.20-35.20]
I20260709 12:17:43.410873 12641 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [41.21-43.22]
I20260709 12:17:43.351923 25849 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:43.351979 25668 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.374178 25800 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:43.415191 12637 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
W20260709 12:17:43.454767 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 392856us
I20260709 12:17:43.448745 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.459161 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.460158 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.461385 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:43.461824 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.462394 12491 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:43.463425 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:43.464555 12494 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:43.402945 25805 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:43.421950 25572 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:43.426942 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:43.475144 12563 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 41
I20260709 12:17:43.433615 25682 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:43.403023 25112 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:43.424847 25646 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.429564 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:43.441177 12571 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:43.444715 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:43.439004 12567 rpcz_store.cc:276] 0709 12:17:41.533048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:41.762134 (+229086us) service_pool.cc:224] Handling call
0709 12:17:43.136133 (+1373999us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:17:43.187305 (+ 51172us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:43.187968 (+   663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:43.379018 (+191050us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b5400045950. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:43.379209 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":120471296}
I20260709 12:17:43.496395 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:43.447964 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.451618 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3966) took 1099 ms. Trace:
I20260709 12:17:43.435252 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:43.498080 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.498334 12638 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:43.497859 12496 rpcz_store.cc:276] 0709 12:17:42.351964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:42.352222 (+   258us) service_pool.cc:224] Handling call
0709 12:17:42.352787 (+   565us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:43.375450 (+1022663us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54001c1ea0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:17:43.438539 (+ 63089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:43.451532 (+ 12993us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30300416}
I20260709 12:17:43.454339 12565 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 23.
I20260709 12:17:43.454048 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.470214 12497 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:43.471452 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.501716 25676 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.497200 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.503475 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:17:43.503715 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:43.504777 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.504792 12637 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:43.507365 12567 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 20.
I20260709 12:17:43.510468 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.471462 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:43.471758 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.445466 25737 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:43.477234 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:43.515120 12644 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 23.
I20260709 12:17:43.481444 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:43.487962 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.436726 25669 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.042s
I20260709 12:17:43.517940 12495 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:43.500208 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:43.501224 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.513801 12646 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:43.478133 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:43.522747 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.479094 25112 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.530104 25112 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:43.532208 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 193860us
I20260709 12:17:43.467975 25493 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:43.534597 25493 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.453018 25801 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.544703 25646 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.434005 25675 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.547426 25675 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.544778 25814 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.557538 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:43.558265 12492 raft_consensus.cc:606] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:43.477918 25805 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:43.548089 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 477194us
I20260709 12:17:43.558616 12492 raft_consensus.cc:614] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:17:43.561378 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61284.9us
I20260709 12:17:43.516000 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 31.
I20260709 12:17:43.565438 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:43.570806 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:43.575788 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 146274us
I20260709 12:17:43.578625 12646 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 19. (term mismatch)
I20260709 12:17:43.579276 12646 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 18
I20260709 12:17:43.579591 12646 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 29.19
I20260709 12:17:43.579989 12646 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:43.586597 12646 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 23. (term mismatch)
I20260709 12:17:43.587291 12646 pending_rounds.cc:85] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 22
I20260709 12:17:43.587648 12646 pending_rounds.cc:107] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 35.23
I20260709 12:17:43.587996 12646 raft_consensus.cc:2889] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:43.466045 25849 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.590585 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.592974 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:43.470530 25668 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.597297 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:43.600468 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.602190 12572 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:43.603967 12646 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:43.570756 25805 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:43.570267 25646 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:43.495117 25682 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.600347 25642 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:43.589608 25800 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:17:43.606158 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:43.575413 25590 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:43.618403 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:43.619045 12492 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:43.619385 12492 raft_consensus.cc:614] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:43.619575 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:43.620630 12571 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 16.
I20260709 12:17:43.620810 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:43.563090 25597 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260709 12:17:43.621932 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.627714 12571 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:43.623311 12491 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:43.623136 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:43.558992 25676 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.597424 25859 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:17:43.598373 25802 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.027s
I20260709 12:17:43.635077 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:43.633348 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:43.570614 25814 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.635016 12571 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:43.614967 25747 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
W20260709 12:17:43.636933 13220 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.041s	sys 0.062s
W20260709 12:17:43.645291 13220 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.041s	sys 0.062s
I20260709 12:17:43.640255 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:17:43.637588 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.623111 25803 raft_consensus.cc:2764] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
I20260709 12:17:43.614017 25675 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.022s
I20260709 12:17:43.626281 25711 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:43.623771 25669 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:43.675034 12571 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:17:43.615842 25625 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.034s
I20260709 12:17:43.716918 25846 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.636773 25800 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:17:43.660332 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.688086 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:43.672976 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:43.717829 25846 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.630785 25829 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.630044 25590 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.728659 25864 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:17:43.725306 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52500.6us
I20260709 12:17:43.650949 25747 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.676918 25694 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.773227 25694 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.774884 25814 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.775449 25694 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:43.777974 25845 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.777s	user 0.011s	sys 0.001s
I20260709 12:17:43.778824 25747 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:43.677591 25728 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:43.689754 25711 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.788368 25711 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:43.770226 25792 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.794236 25792 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.796927 25792 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.768281 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:43.809123 12637 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:43.821095 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:43.643684 25189 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.079s
I20260709 12:17:43.824862 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:43.826440 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58270.4us
W20260709 12:17:43.827253 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.812206 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.831492 25189 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:43.832373 25189 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:43.831300 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:43.835539 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:43.845922 12572 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 37 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 20.
I20260709 12:17:43.833658 25849 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.696321 25806 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:43.846411 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:43.869159 12571 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 21.
I20260709 12:17:43.827587 25572 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:43.799145 25684 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.871260 13257 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:17:43.862649 25682 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:43.828611 25864 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:43.834870 25748 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:43.797595 25792 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:43.865253 25829 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:43.866555 25642 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.004s
I20260709 12:17:43.833841 25597 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:43.866428 25676 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:43.934351 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:43.938526 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 345653us
I20260709 12:17:43.833767 25189 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:43.955740 25812 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:43.959743 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:43.960050 25823 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:43.960171 25811 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:43.893929 25748 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:43.962391 12572 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 36 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 20.
I20260709 12:17:43.963788 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:17:44.018939 12536 leader_election.cc:403] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:43.965343 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:44.020071 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54825.1us
W20260709 12:17:44.022550 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62944.2us
I20260709 12:17:43.933207 25865 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:44.024274 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60644.9us
I20260709 12:17:44.025954 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:43.961181 25867 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:17:44.021265 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:44.035024 12563 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:17:44.036568 25676 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:17:44.045951 25876 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:44.034216 25748 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:44.048303 25801 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:44.049683 25865 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:44.051008 25868 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:17:44.037070 25803 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:44.052129 25865 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.061585 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:44.064738 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:44.065742 25876 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:44.017066 25812 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:44.018524 25823 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:44.052650 25803 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:44.033257 25682 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:44.026537 25646 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:44.079051 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:44.067185 25189 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:44.015226 25798 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:17:44.047506 25668 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.070262 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:44.067683 25631 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:44.075376 25112 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.007s	sys 0.006s
I20260709 12:17:44.081545 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1918) took 1321 ms. Trace:
I20260709 12:17:44.077440 25849 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:44.080945 25876 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.033s
I20260709 12:17:44.091392 12643 rpcz_store.cc:276] 0709 12:17:42.759670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.295146 (+535476us) service_pool.cc:224] Handling call
0709 12:17:43.295504 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:43.295736 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:43.295759 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:43.296267 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:17:43.644695 (+348428us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:17:44.061288 (+416593us) log.cc:844] Serialized 14 byte log entry
0709 12:17:44.081154 (+ 19866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.081248 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:44.081299 (+    51us) raft_consensus.cc:1685] finished
0709 12:17:44.081315 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.081478 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13408,"raft.queue_time_us":16067,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":3062,"thread_start_us":732364,"threads_started":2,"wal-append.queue_time_us":574224}
I20260709 12:17:44.088058 12571 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:43.936939 25792 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.078262 25803 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:44.034060 25811 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.149986 13253 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
W20260709 12:17:44.091449 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 252407us
I20260709 12:17:44.152244 25811 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:44.068660 25801 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:44.077663 25572 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:44.154348 25572 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.156965 25572 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:44.157472 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.159476 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:44.071266 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50091.3us
W20260709 12:17:44.138809 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.092478 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:44.081382 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 244277us
W20260709 12:17:44.132097 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.004s	sys 0.008s
I20260709 12:17:44.086548 25682 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:44.147974 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.086746 25747 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.202910 12490 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:17:44.176837 12494 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
W20260709 12:17:44.186620 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.161135 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.161080 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.161531 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.160714 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:44.201153 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.006s	sys 0.010s
I20260709 12:17:44.208421 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.078608 25864 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.217700 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:44.117936 25805 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:44.114285 25590 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:44.207605 25849 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:44.102900 25112 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.093662 25806 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.230998 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.236549 13255 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:44.233095 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:44.239624 13262 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:17:44.212925 12492 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:44.240871 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:44.213018 12493 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:44.229545 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:44.240222 12643 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:44.241613 12494 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:44.212257 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:44.085788 25872 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.152s
I20260709 12:17:44.247014 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:44.203744 25801 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:44.251165 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.242849 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.251246 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.252666 12488 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:44.206424 25874 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:44.255849 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:44.256470 25597 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:44.260319 12490 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:17:44.264369 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.243593 25798 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:44.274683 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.094432 25631 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.231248 25790 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:44.270841 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.260588 25590 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:44.207037 25646 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.254335 25642 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:44.106317 25812 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.263368 25882 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:44.284053 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:44.307035 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5022) took 1301 ms. Trace:
I20260709 12:17:44.249699 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:44.116573 25823 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.267963 25597 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:44.235106 25669 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:17:44.287575 25798 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:44.290614 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1928) took 1452 ms. Trace:
I20260709 12:17:44.309952 12637 rpcz_store.cc:276] 0709 12:17:43.007126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.823682 (+816556us) service_pool.cc:224] Handling call
0709 12:17:43.846573 (+ 22891us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:43.846820 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:17:44.082592 (+235772us) log.cc:844] Serialized 14 byte log entry
0709 12:17:44.306572 (+223980us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:44.306627 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:44.306678 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.306790 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.306962 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2752,"prepare.queue_time_us":376780,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"thread_start_us":431524,"threads_started":2}
I20260709 12:17:44.298805 12494 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
W20260709 12:17:44.320585 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71024.4us
W20260709 12:17:44.307327 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89886.8us
W20260709 12:17:44.303453 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47678.6us
I20260709 12:17:44.304697 25675 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:44.336216 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:44.308395 25493 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:44.339341 12646 rpcz_store.cc:276] 0709 12:17:42.874949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.619423 (+744474us) service_pool.cc:224] Handling call
0709 12:17:43.657347 (+ 37924us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:43.657612 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:17:44.085235 (+427623us) log.cc:844] Serialized 14 byte log entry
0709 12:17:44.290093 (+204858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:44.290151 (+    58us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:44.290231 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.290366 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.290527 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20195,"prepare.queue_time_us":565737,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":566337,"threads_started":2}
I20260709 12:17:44.331266 25597 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:44.351665 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:44.396133 25699 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.292s	user 0.009s	sys 0.013s
I20260709 12:17:44.397179 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5025) took 1389 ms. Trace:
W20260709 12:17:44.398532 25699 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.295s	user 0.009s	sys 0.013s
I20260709 12:17:44.352522 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:44.401046 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:44.348819 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 256385us
W20260709 12:17:44.403288 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98844us
W20260709 12:17:44.363847 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117270us
I20260709 12:17:44.283157 25682 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.300684 25790 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.349481 25668 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:44.411369 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59774.4us
I20260709 12:17:44.397904 12646 rpcz_store.cc:276] 0709 12:17:43.008223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.356793 (+1348570us) service_pool.cc:224] Handling call
0709 12:17:44.396139 (+ 39346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.396573 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:44.396622 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:44.396699 (+    77us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:44.396754 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.396854 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.397065 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:44.353243 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5024) took 1345 ms. Trace:
I20260709 12:17:44.321067 25845 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.045s
W20260709 12:17:44.394929 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 139322us
I20260709 12:17:44.359774 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5023) took 1352 ms. Trace:
I20260709 12:17:44.343546 25812 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:44.408454 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.306300 25642 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.411463 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:44.409744 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:44.423695 12492 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.19-43.19]
I20260709 12:17:44.425936 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:44.424015 12637 rpcz_store.cc:276] 0709 12:17:43.007597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.341429 (+1333832us) service_pool.cc:224] Handling call
0709 12:17:44.358929 (+ 17500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.359229 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:44.359272 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:44.359334 (+    62us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:44.359402 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.359483 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.359643 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:44.427891 12493 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:17:44.423583 12643 rpcz_store.cc:276] 0709 12:17:43.007925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.352232 (+1344307us) service_pool.cc:224] Handling call
0709 12:17:44.352580 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.352802 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:44.352838 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:44.352904 (+    66us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:44.352962 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.353043 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.353175 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:44.412866 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:44.320694 25849 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.435314 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:44.436292 12491 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:44.437968 12492 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: [38.20-38.20]
W20260709 12:17:44.431284 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73522.7us
I20260709 12:17:44.439589 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:44.337172 25876 consensus_queue.cc:1175] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 39 status { last_received { term: 39 index: 27 } last_committed_idx: 27 last_received_current_leader { term: 39 index: 27 } }
I20260709 12:17:44.413198 25872 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:44.450450 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:44.458720 12637 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:44.410764 25879 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:17:44.445190 25737 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.875s	user 0.014s	sys 0.000s
I20260709 12:17:44.470006 12643 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:17:44.470573 12643 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 17
I20260709 12:17:44.470832 12643 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:17:44.471072 12643 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:44.471891 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5026) took 1463 ms. Trace:
I20260709 12:17:44.472206 12643 rpcz_store.cc:276] 0709 12:17:43.008488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.432233 (+1423745us) service_pool.cc:224] Handling call
0709 12:17:44.455474 (+ 23241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.471544 (+ 16070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.471816 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:44.466878 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:44.346657 25801 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.444597 25747 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.412814 25889 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.060s
I20260709 12:17:44.424750 25668 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.491356 13259 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:44.423841 25495 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:17:44.480253 25737 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:44.493992 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84364us
W20260709 12:17:44.494118 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86723.9us
I20260709 12:17:44.502157 25747 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:17:44.502549 25884 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 12:17:44.504061 25883 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.221s	user 0.001s	sys 0.001s
I20260709 12:17:44.504443 25900 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:44.514788 12637 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:44.580380 12637 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:17:44.496029 25872 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:44.536535 25493 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:44.581228 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4061) took 1134 ms. Trace:
I20260709 12:17:44.515512 25881 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:44.495780 25805 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:44.513972 25887 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:17:44.584687 12497 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:44.588014 12637 rpcz_store.cc:276] 0709 12:17:43.446400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.579427 (+1133027us) service_pool.cc:224] Handling call
0709 12:17:44.579919 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.580902 (+   983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.581158 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:44.496140 25112 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:44.582734 25888 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
W20260709 12:17:44.581986 25885 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 12:17:44.460325 25798 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.21, Last appended by leader: 21, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.604306 25890 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:17:44.599231 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 165628us
W20260709 12:17:44.610617 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:44.611882 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.589562 25872 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:44.624552 12497 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:44.629762 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5036) took 1152 ms. Trace:
I20260709 12:17:44.630272 12637 rpcz_store.cc:276] 0709 12:17:43.477102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.602241 (+1125139us) service_pool.cc:224] Handling call
0709 12:17:44.629085 (+ 26844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.629330 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:44.629356 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:44.629412 (+    56us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:44.629462 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.629523 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.629686 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:44.592808 25895 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:44.585275 25495 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.031s
I20260709 12:17:44.599211 25811 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:44.613756 25747 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:17:44.642442 25897 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:17:44.651397 25902 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
W20260709 12:17:44.655874 25189 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.000s
W20260709 12:17:44.659463 25803 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.000s
I20260709 12:17:44.615085 25631 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.661866 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:44.662117 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:44.632899 25872 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:44.690938 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5019) took 1805 ms. Trace:
I20260709 12:17:44.685406 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:44.596175 25493 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.689304 12496 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:44.691413 12638 rpcz_store.cc:276] 0709 12:17:42.885416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.730565 (+845149us) service_pool.cc:224] Handling call
0709 12:17:43.731050 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:43.731502 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:43.731537 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:43.732205 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:17:44.230322 (+498117us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:44.230367 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.230436 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:44.690661 (+460225us) raft_consensus.cc:1685] finished
0709 12:17:44.690686 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.690866 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":150,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":151,"thread_start_us":497673,"threads_started":1,"wal-append.queue_time_us":600069}
I20260709 12:17:44.598338 25881 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:44.678860 25711 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.890s	user 0.015s	sys 0.000s
I20260709 12:17:44.700011 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:44.661144 25747 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:44.661193 25895 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:44.705823 12638 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
I20260709 12:17:44.633574 25905 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:17:44.683071 25902 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:44.635617 25914 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.080s
W20260709 12:17:44.707942 25676 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.668s	user 0.008s	sys 0.006s
W20260709 12:17:44.717260 25810 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.028s	user 0.007s	sys 0.021s
W20260709 12:17:44.725221 25810 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.007s	sys 0.021s
W20260709 12:17:44.721517 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 262564us
W20260709 12:17:44.731812 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.733794 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:44.736064 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5030) took 1608 ms. Trace:
I20260709 12:17:44.736595 12646 rpcz_store.cc:276] 0709 12:17:43.127594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.473998 (+1346404us) service_pool.cc:224] Handling call
0709 12:17:44.496007 (+ 22009us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:44.516734 (+ 20727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:44.516777 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:44.517680 (+   903us) log.cc:844] Serialized 52 byte log entry
0709 12:17:44.519333 (+  1653us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:44.519756 (+   423us) log.cc:844] Serialized 14 byte log entry
0709 12:17:44.521762 (+  2006us) log.cc:844] Serialized 14 byte log entry
0709 12:17:44.523381 (+  1619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.523503 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:44.735792 (+212289us) raft_consensus.cc:1685] finished
0709 12:17:44.735813 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.735992 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":305,"prepare.queue_time_us":446,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":361139,"raft.run_cpu_time_us":527,"raft.run_wall_time_us":19075,"thread_start_us":23811,"threads_started":4,"wal-append.queue_time_us":93432}
I20260709 12:17:44.738543 25597 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.739542 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:44.741163 12494 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.24-31.24]
I20260709 12:17:44.710484 25747 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:44.714936 25642 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.745285 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 3983) took 1306 ms. Trace:
I20260709 12:17:44.752512 12488 rpcz_store.cc:276] 0709 12:17:43.438639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.438973 (+   334us) service_pool.cc:224] Handling call
0709 12:17:43.458063 (+ 19090us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.253164 (+795101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.745000 (+491836us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:44.745215 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1826942848}
W20260709 12:17:44.751906 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 285092us
I20260709 12:17:44.703964 25631 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.040s
I20260709 12:17:44.808562 25631 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:44.809214 25631 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:44.806267 25895 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.707792 25711 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.811409 25711 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.811465 25590 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:44.694414 25803 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.812230 25590 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.813692 25803 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:44.814591 12563 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:17:44.814713 25590 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.815649 25882 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:17:44.783349 25892 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
W20260709 12:17:44.827600 25871 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.716s	user 0.003s	sys 0.000s
I20260709 12:17:44.784191 25493 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.771837 25905 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:44.828883 25905 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:44.829125 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:44.787827 25748 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.733s	user 0.008s	sys 0.006s
I20260709 12:17:44.806133 25668 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.806566 25845 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:44.743131 25676 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:17:44.817894 25631 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 40.29, Last appended by leader: 29, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.818831 25879 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:44.821848 25846 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:44.822535 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4060) took 1810 ms. Trace:
I20260709 12:17:44.825100 25737 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:44.879632 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:44.880319 12572 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:44.877888 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:44.880993 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:44.772505 25910 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:44.826205 25881 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:44.829995 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.832028 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1905) took 1922 ms. Trace:
I20260709 12:17:44.887579 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:44.888749 13254 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:17:44.832206 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.839447 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:44.840037 12570 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:44.715999 25646 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:44.841013 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:44.842629 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5029) took 1831 ms (client timeout 1907 ms). Trace:
W20260709 12:17:44.891389 12639 rpcz_store.cc:269] 0709 12:17:43.010888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.040900 (+ 30012us) service_pool.cc:224] Handling call
0709 12:17:44.842388 (+1801488us) spinlock_profiling.cc:243] Waited 82.4 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:44.842560 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189577984}
I20260709 12:17:44.842943 12642 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:44.830271 25905 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 43.19, Last appended by leader: 19, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:44.876924 25869 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.855s	user 0.007s	sys 0.001s
I20260709 12:17:44.878750 12643 rpcz_store.cc:276] 0709 12:17:43.012263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.472996 (+1460733us) service_pool.cc:224] Handling call
0709 12:17:44.489025 (+ 16029us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:44.489520 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:44.489577 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:44.495706 (+  6129us) log.cc:844] Serialized 52 byte log entry
0709 12:17:44.695230 (+199524us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:44.718213 (+ 22983us) log.cc:844] Serialized 14 byte log entry
0709 12:17:44.718699 (+   486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.718802 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:44.822265 (+103463us) raft_consensus.cc:1685] finished
0709 12:17:44.822289 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:44.822461 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24109,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":73,"raft.queue_time_us":106283,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":17961,"spinlock_wait_cycles":15059840,"thread_start_us":115080,"threads_started":1,"wal-append.queue_time_us":310306}
I20260709 12:17:44.888012 13257 rpcz_store.cc:276] 0709 12:17:42.909595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:42.909869 (+   274us) service_pool.cc:224] Handling call
0709 12:17:43.152940 (+243071us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:43.882057 (+729117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:44.830845 (+948788us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:44.831818 (+   973us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400537850. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:44.831964 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2186893952}
I20260709 12:17:44.885165 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:44.890574 12570 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36.
I20260709 12:17:44.897423 13256 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:44.906740 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:44.952656 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46066.4us
I20260709 12:17:44.896744 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:44.953105 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:44.954339 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:44.955808 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:44.956516 12567 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:44.963848 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:44.964682 12609 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36.
I20260709 12:17:44.965292 12609 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:17:44.966495 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81414.5us
I20260709 12:17:44.968032 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:44.968746 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:44.877732 25879 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:44.971596 12639 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:17:44.896143 25846 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:44.846381 25748 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.928052 25905 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:44.989185 12645 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:44.877785 25845 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.894902 25812 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:44.927708 25920 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:44.975176 25682 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 36
I20260709 12:17:44.984787 12567 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
W20260709 12:17:44.992744 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5027) took 1983 ms (client timeout 2134 ms). Trace:
W20260709 12:17:44.997412 12644 rpcz_store.cc:269] 0709 12:17:43.009081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.146355 (+137274us) service_pool.cc:224] Handling call
0709 12:17:44.988689 (+1842334us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:44.992680 (+  3991us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748714368}
I20260709 12:17:44.990878 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:44.998057 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:44.978371 25849 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:44.999753 12567 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:45.000739 13253 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38.
I20260709 12:17:44.999037 25493 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.006788 25913 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.011489 25845 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.012138 25849 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.930100 25646 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:44.970640 25917 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:45.029326 25646 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.030157 25646 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:45.030946 25646 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:45.031970 25646 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:45.035517 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65955us
I20260709 12:17:44.895198 25910 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.016114 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:45.040599 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:45.043521 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:45.018971 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:44.986181 25914 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:44.985944 25803 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:45.024457 25913 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:44.996076 25846 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:45.014644 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:44.890206 25902 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.29, Last appended by leader: 29, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.035961 25748 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.075881 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:45.076195 12537 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.888s (SENT) [suppressed 14 similar messages]
I20260709 12:17:45.081204 25801 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:45.037156 25917 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:45.081985 25927 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:45.081980 12568 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 32.
W20260709 12:17:45.083932 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93124us
I20260709 12:17:45.086663 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:45.022836 25493 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.084992 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:45.050515 25831 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:45.053225 25590 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
W20260709 12:17:45.136574 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:45.020895 25920 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:45.137307 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50724.1us
I20260709 12:17:45.083868 25631 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:45.142679 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87995us
I20260709 12:17:45.086544 25907 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.085439 25917 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.049s
I20260709 12:17:45.154110 12569 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
W20260709 12:17:45.148816 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109165us
I20260709 12:17:45.135960 25914 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.136379 25895 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.169509 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.146101 25668 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:45.170519 12644 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:45.149063 25902 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:45.176663 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5442) took 2048 ms (client timeout 2134 ms). Trace:
I20260709 12:17:45.144143 25803 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:45.163491 25631 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:45.177186 12565 rpcz_store.cc:269] 0709 12:17:43.128472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.161236 (+ 32764us) service_pool.cc:224] Handling call
0709 12:17:45.176454 (+2015218us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:45.176595 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":662429056}
I20260709 12:17:45.182789 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:45.162744 25590 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:45.136165 25747 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:45.202426 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55698.1us
I20260709 12:17:45.175264 25914 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.223068 13260 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.25-45.25]
I20260709 12:17:45.198657 25668 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:45.164602 25907 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.183243 25933 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.053s
W20260709 12:17:45.232577 25930 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:17:45.232234 25590 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 43.22, Last appended by leader: 22, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.234278 25936 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.090s
I20260709 12:17:45.235436 25914 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:45.235723 25919 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
W20260709 12:17:45.236563 25929 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:17:45.220350 25845 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:45.187748 25803 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:45.255470 25928 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 12:17:45.256577 25913 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:45.232365 25937 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
W20260709 12:17:45.235507 25932 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:17:45.179415 25895 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:45.278972 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63559us
I20260709 12:17:45.172565 25846 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.246941 25798 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:45.216702 25831 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:17:45.289165 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:45.282881 25812 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:45.299044 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.300213 12644 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:45.293367 25910 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.310129 25946 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:45.302052 25931 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:17:45.303399 25668 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.326594 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:45.323675 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:45.333894 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:45.334735 25950 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:17:45.336791 25926 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:17:45.337252 25936 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:45.338685 25676 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:45.336863 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:45.341817 25948 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:45.342489 25948 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:45.343457 25948 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.346328 25493 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.346736 25112 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.347224 25112 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.347843 25936 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:45.348961 25790 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:45.348606 25936 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.349541 25790 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.347838 25493 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.350243 25790 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.353426 25790 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.318460 25879 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.354164 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.354882 13262 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:45.356005 25879 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.356324 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.356441 25936 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.356897 12494 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.357460 12494 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 37.
I20260709 12:17:45.357715 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.358762 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.359561 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.339258 25676 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:45.350435 25493 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.351313 25947 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.366875 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.359468 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:17:45.349350 25112 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.372763 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.374094 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.374236 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:45.377503 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:45.382444 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.384021 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.391189 12494 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
W20260709 12:17:45.368598 25872 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.660s	user 0.013s	sys 0.002s
I20260709 12:17:45.376858 25947 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.395609 25895 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:45.325863 25946 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.398121 25944 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.026s
I20260709 12:17:45.399524 25944 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:45.399987 25944 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:45.400439 25946 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.400584 25831 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.401415 25831 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.402534 25803 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.403891 25831 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.406221 25631 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.408221 25675 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:45.409003 25675 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:45.411981 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:45.415829 25823 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:45.335044 25802 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:45.373140 25676 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:45.424857 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.435351 12387 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 19 to 26. New cstate: current_term: 26 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.437760 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:45.438939 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:45.441951 12493 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:17:45.448511 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:45.449563 12640 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:45.418371 25682 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36.
I20260709 12:17:45.408212 25747 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.386417 25920 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.418550 25947 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.462713 12387 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 28 to 31, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 31 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.471274 12387 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 25 to 29, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 29 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.477270 12610 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 37.
I20260709 12:17:45.369580 25812 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 42.18, Last appended by leader: 18, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.428622 25802 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.474035 25914 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:45.493747 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:45.497954 12640 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
W20260709 12:17:45.466337 25669 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.117s	user 0.004s	sys 0.017s
I20260709 12:17:45.401023 25944 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.463231 25845 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:45.418200 25872 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.489821 12387 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 26 to 27, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 27 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.480504 25907 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:45.501955 25669 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.156s	user 0.004s	sys 0.018s
I20260709 12:17:45.686043 25802 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.556957 12387 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 27, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 27 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.730470 12387 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:45.688349 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
W20260709 12:17:45.557399 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:45.684370 25872 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:45.687490 25966 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:45.722855 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:45.503895 25948 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:45.725064 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:45.725642 25944 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:45.727329 25493 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.722059 25920 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.637961 25189 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.684552 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:45.739648 25966 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.741025 25845 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 40.16, Last appended by leader: 16, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.744645 12387 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 28, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 28 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.765926 25920 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.766291 25189 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.767158 25966 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.768453 25920 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:45.725883 12389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 228) took 89083 ms (client timeout 15000 ms). Trace:
I20260709 12:17:45.754029 12489 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:45.759159 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 3989) took 1352 ms. Trace:
I20260709 12:17:45.741046 25792 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:45.761605 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.772575 12493 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:45.773111 12641 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
W20260709 12:17:45.765111 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 80780.2us
I20260709 12:17:45.765120 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4048) took 2984 ms. Trace:
I20260709 12:17:45.765466 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.768599 25189 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.771313 12497 rpcz_store.cc:276] 0709 12:17:44.406611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.411037 (+  4426us) service_pool.cc:224] Handling call
0709 12:17:45.758931 (+1347894us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:45.759082 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363685120}
I20260709 12:17:45.788312 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.790050 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.791718 12497 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:45.778707 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:45.794813 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.795619 12638 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:45.774111 12645 rpcz_store.cc:276] 0709 12:17:42.776208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.381953 (+605745us) service_pool.cc:224] Handling call
0709 12:17:44.988932 (+1606979us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.989932 (+  1000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:45.760452 (+770520us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54001f68d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:45.760741 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3624833408}
W20260709 12:17:45.796346 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1638) took 1639 ms (client timeout 1888 ms). Trace:
I20260709 12:17:45.797406 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:45.797081 13253 rpcz_store.cc:269] 0709 12:17:44.156942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.157122 (+   180us) service_pool.cc:224] Handling call
0709 12:17:45.796061 (+1638939us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:45.796261 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1937014144}
I20260709 12:17:45.796451 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:45.799510 12638 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:45.779774 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:45.780423 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:45.769146 12389 rpcz_store.cc:269] 0709 12:16:16.642116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:26.590482 (+9948366us) service_pool.cc:224] Handling call
0709 12:17:45.507508 (+78917026us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:17:38.158586 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:17:38.159342 (+   756us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:17:38.159408 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:17:38.159428 (+    20us) tablet.cc:662] Decoding operations
0709 12:17:38.277201 (+117773us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:17:38.277310 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 12:17:38.277344 (+    34us) tablet.cc:685] Acquiring locks for 20 operations
0709 12:17:38.326250 (+ 48906us) tablet.cc:701] Row locks acquired
0709 12:17:38.326305 (+    55us) write_op.cc:260] PREPARE: finished
0709 12:17:38.326490 (+   185us) write_op.cc:270] Start()
0709 12:17:38.326678 (+   188us) write_op.cc:276] Timestamp: P: 1783599458326413 usec, L: 0
0709 12:17:38.326721 (+    43us) op_driver.cc:348] REPLICATION: starting
0709 12:17:38.327536 (+   815us) log.cc:844] Serialized 7894 byte log entry
0709 12:17:38.898615 (+571079us) op_driver.cc:464] REPLICATION: finished
0709 12:17:39.264670 (+366055us) write_op.cc:301] APPLY: starting
0709 12:17:39.264807 (+   137us) tablet.cc:1367] starting BulkCheckPresence
0709 12:17:39.281702 (+ 16895us) tablet.cc:1370] finished BulkCheckPresence
0709 12:17:39.281901 (+   199us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:17:39.723500 (+441599us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:17:39.724165 (+   665us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=40,key_file_lookups=40,delta_file_lookups=0,mrs_lookups=0
0709 12:17:39.724225 (+    60us) write_op.cc:312] APPLY: finished
0709 12:17:39.724763 (+   538us) log.cc:844] Serialized 180 byte log entry
0709 12:17:39.725017 (+   254us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:39.743999 (+ 18982us) write_op.cc:454] Released schema lock
0709 12:17:39.766296 (+ 22297us) write_op.cc:341] FINISH: Updating metrics
0709 12:17:39.795946 (+ 29650us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:39.797897 (+  1951us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8170151,"reported_tablets":51,"spinlock_wait_cycles":33152,"child_traces":[["op",{"apply.queue_time_us":349330,"apply.run_cpu_time_us":34444,"apply.run_wall_time_us":508083,"cfile_cache_hit":82,"cfile_cache_hit_bytes":87647,"cfile_init":1,"lbm_read_time_us":624,"lbm_reads_lt_1ms":4,"num_ops":20,"prepare.queue_time_us":162845,"prepare.run_cpu_time_us":48699,"prepare.run_wall_time_us":320295,"raft.queue_time_us":202750,"raft.run_cpu_time_us":53087,"raft.run_wall_time_us":270551,"replication_time_us":462697,"thread_start_us":430597,"threads_started":4,"wal-append.queue_time_us":217321,"wal-append.run_cpu_time_us":11792,"wal-append.run_wall_time_us":2182793}]]}
I20260709 12:17:45.780841 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.784909 25894 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:45.774073 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:45.745590 25676 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.803602 12489 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:45.774451 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:45.805156 12489 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:17:45.764269 25948 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:45.763439 12461 connection.cc:256] RPC call timeout handler was delayed by -1.0243896019999283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
I20260709 12:17:45.800071 12639 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 36
I20260709 12:17:45.725817 25675 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.769162 12387 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:45.808409 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.813707 12641 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:45.814465 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.814720 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.815408 12641 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:45.809602 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.811203 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:45.816978 13256 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 40 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 24.
I20260709 12:17:45.812551 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:45.821139 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:45.822366 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.826835 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.802927 25894 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:45.806581 25493 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:45.806526 25792 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:45.841944 25894 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.845911 25676 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:45.846563 25675 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:45.834079 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.836023 12645 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 40 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 24.
I20260709 12:17:45.838027 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:45.830667 25881 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:45.824105 25590 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:17:45.838598 25920 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:45.851722 25920 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:45.852366 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.853192 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:45.850981 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:45.854652 25920 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:45.855135 12646 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:17:45.830255 25668 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:45.863503 12639 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:45.873838 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1910) took 1668 ms. Trace:
I20260709 12:17:45.874421 13258 rpcz_store.cc:276] 0709 12:17:44.205584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.872766 (+1667182us) service_pool.cc:224] Handling call
0709 12:17:45.873193 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:45.873410 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:45.873440 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:45.873492 (+    52us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:45.873530 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:45.873592 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:45.873767 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:45.875476 25959 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.424s	user 0.004s	sys 0.010s
I20260709 12:17:45.868265 12638 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:45.889578 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:45.902801 13256 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:17:45.879930 25189 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:45.908228 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69507.4us
W20260709 12:17:45.912647 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:45.917546 25189 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:45.920897 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:45.922550 25189 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.923318 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:45.884865 25881 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:17:45.924254 25684 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:45.924821 25684 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:45.925858 25684 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:45.927630 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5197) took 1904 ms. Trace:
I20260709 12:17:45.928175 12572 rpcz_store.cc:276] 0709 12:17:44.023432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.027634 (+  4202us) service_pool.cc:224] Handling call
0709 12:17:44.049501 (+ 21867us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.881545 (+832044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:45.927201 (+1045656us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:45.927519 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909870464}
I20260709 12:17:45.932204 25849 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:45.932559 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.935871 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.885844 25954 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:45.906912 25675 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:45.940737 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.946125 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:45.948303 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:45.948660 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:45.949609 13256 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38.
I20260709 12:17:45.975930 12565 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:17:45.976455 12567 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:17:45.956633 13253 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
I20260709 12:17:46.000792 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.001895 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.002432 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.003482 13259 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 29. Preceding OpId from leader: term: 41 index: 30. (index mismatch)
I20260709 12:17:46.003480 12567 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:45.990788 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:45.949704 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:46.008905 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:46.010272 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:46.010982 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.013795 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:45.989079 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:46.016042 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:45.937937 25711 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:45.977206 25112 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:45.947461 25676 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:45.950282 25831 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:46.023586 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:45.946385 25801 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.024238 25944 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:45.950892 25823 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:46.013257 25845 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:45.986683 25942 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.049s
I20260709 12:17:45.949221 25890 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.042759 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5199) took 2004 ms. Trace:
I20260709 12:17:46.047084 12563 rpcz_store.cc:276] 0709 12:17:44.038123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.038259 (+   136us) service_pool.cc:224] Handling call
0709 12:17:44.069384 (+ 31125us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:44.815054 (+745670us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:46.042323 (+1227269us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:46.042658 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1713219456}
W20260709 12:17:46.031448 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81842.2us
I20260709 12:17:46.033221 12565 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:46.050349 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:46.019497 12567 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:17:46.037662 12569 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:17:45.945367 25811 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260709 12:17:45.944970 25441 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:46.042742 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4053) took 3242 ms. Trace:
I20260709 12:17:45.982733 25675 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.052547 12537 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38.
I20260709 12:17:45.950393 25954 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.036312 25711 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:46.052778 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:46.055760 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:46.066900 12537 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:46.057801 12642 rpcz_store.cc:276] 0709 12:17:42.800530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.399598 (+599068us) service_pool.cc:224] Handling call
0709 12:17:44.842676 (+1443078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:44.892943 (+ 50267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:46.042515 (+1149572us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54002a5dd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:46.042676 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3291777408}
I20260709 12:17:46.068428 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:46.013475 25849 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.035s
I20260709 12:17:46.042784 25927 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.054847 25831 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:46.081794 25831 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:46.082247 25831 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:46.082753 25831 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:46.047637 25845 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.075626 25669 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:46.045684 25944 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:46.078687 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:46.055223 25823 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:46.087570 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:46.038676 25676 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.116489 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:46.117586 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.040314 25801 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.075366 25948 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:46.057260 25911 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.061s
I20260709 12:17:46.140430 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:46.141601 12571 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:46.141695 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.056809 25890 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.062244 25947 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.064s
I20260709 12:17:46.112591 25669 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:46.126228 25441 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:46.164846 12572 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:46.140079 25694 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:46.169505 25694 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:46.171139 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.088726 25927 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:46.173250 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100617us
I20260709 12:17:46.165812 25684 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:17:46.187395 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116051us
I20260709 12:17:46.118705 25823 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.162300 25441 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:46.162163 25669 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:46.173048 25694 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:46.210618 12563 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:17:46.206197 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:46.164039 25711 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.037s
I20260709 12:17:46.207875 25441 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:46.216394 25859 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.538s	user 0.010s	sys 0.015s
W20260709 12:17:46.244557 25859 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.568s	user 0.011s	sys 0.015s
W20260709 12:17:46.248548 25973 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:17:46.249722 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:46.250514 25728 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:46.209458 25684 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:46.252945 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74244.2us
W20260709 12:17:46.285502 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:46.218561 25944 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.265704 25982 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:17:46.280666 25625 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:46.282063 25954 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.295686 25977 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.338809 25989 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:46.340323 25801 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.263891 25859 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.055s
W20260709 12:17:46.320181 25974 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:17:46.354295 25986 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.324496 25978 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.073s
W20260709 12:17:46.391597 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 185572us
I20260709 12:17:46.370191 25676 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:46.395865 25985 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260709 12:17:46.396828 25983 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 12:17:46.381497 25845 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:46.398061 25976 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
I20260709 12:17:46.389482 25991 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:46.417920 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:46.388163 25989 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:46.392074 25890 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.344435 25684 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.417062 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:46.380713 25625 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.423209 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1936) took 1061 ms. Trace:
I20260709 12:17:46.425428 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:46.427009 12637 rpcz_store.cc:276] 0709 12:17:45.361930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.362068 (+   138us) service_pool.cc:224] Handling call
0709 12:17:45.381170 (+ 19102us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:45.381707 (+   537us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:45.381743 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:45.861813 (+480070us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:45.862440 (+   627us) log.cc:844] Serialized 26 byte log entry
0709 12:17:46.087048 (+224608us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:46.203112 (+116064us) log.cc:844] Serialized 14 byte log entry
0709 12:17:46.350115 (+147003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:46.350252 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:46.422676 (+ 72424us) raft_consensus.cc:1685] finished
0709 12:17:46.422827 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:46.423096 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13871,"prepare.queue_time_us":167917,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":54210,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":11124,"spinlock_wait_cycles":38295296,"thread_start_us":414728,"threads_started":3,"wal-append.queue_time_us":329461}
I20260709 12:17:46.427464 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:46.388607 25977 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.454027 25977 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.420375 25996 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.042s
I20260709 12:17:46.456298 25890 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 30.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:46.457535 25890 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:46.457978 25890 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:46.458355 25890 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:46.460942 25625 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:46.440811 25831 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.423022 25991 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.425305 25989 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:46.445967 25911 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:46.446039 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 165840us
I20260709 12:17:46.446647 25849 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:17:46.419988 25676 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:17:46.494385 25831 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.495265 25597 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.502516 25978 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:46.507364 25978 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:46.507953 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:46.510279 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:46.511070 25823 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.511385 12489 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:46.504663 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:46.432392 25927 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.502916 25676 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 34.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:46.481976 25911 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:46.510075 12568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5207) took 1497 ms (client timeout 1903 ms). Trace:
W20260709 12:17:46.516211 12568 rpcz_store.cc:269] 0709 12:17:45.012469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.043103 (+ 30634us) service_pool.cc:224] Handling call
0709 12:17:46.509899 (+1466796us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:46.510015 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":85915520}
I20260709 12:17:46.505499 25996 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:46.517612 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:46.445259 25668 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.589s	user 0.012s	sys 0.000s
I20260709 12:17:46.467514 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.399405 25986 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.476213 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:46.506799 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5203) took 1709 ms. Trace:
I20260709 12:17:46.524791 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.511328 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1913) took 1671 ms. Trace:
I20260709 12:17:46.497177 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:46.463850 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:46.526654 13256 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:46.526018 13260 rpcz_store.cc:276] 0709 12:17:44.839750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.840112 (+   362us) service_pool.cc:224] Handling call
0709 12:17:44.840470 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:45.227048 (+386578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:46.511070 (+1284022us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:46.511270 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":878961024}
I20260709 12:17:46.525522 12566 rpcz_store.cc:276] 0709 12:17:44.797064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.821878 (+ 24814us) service_pool.cc:224] Handling call
0709 12:17:44.822290 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:45.737260 (+914970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:46.506553 (+769293us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:46.506742 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1697096064}
I20260709 12:17:46.498507 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:46.529439 12638 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:46.529088 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.499609 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.508857 12491 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 32: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:46.518954 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:46.533491 12568 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:46.536478 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.537577 12566 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 41 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 16.
I20260709 12:17:46.536626 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:46.538578 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:46.540130 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76385.5us
I20260709 12:17:46.541832 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:46.542708 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:46.543311 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.543970 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:17:46.545140 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.550302 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:46.550331 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:46.556730 12637 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:46.526319 25979 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.573418 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:46.574765 12642 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 37 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 23.
I20260709 12:17:46.510136 25845 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.032s
I20260709 12:17:46.554790 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.518502 25998 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:46.529053 25668 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:46.501708 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 37 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:46.525776 25917 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.050s
I20260709 12:17:46.508756 25978 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.534765 25441 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.517539 25996 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.540409 25682 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.055s
I20260709 12:17:46.638540 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:46.642149 12570 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:46.646302 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:46.556479 25905 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:46.649457 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.649874 13256 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 32 because replica has last-logged OpId of term: 31 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 23.
I20260709 12:17:46.638479 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:46.655270 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:46.655555 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:17:46.591758 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
W20260709 12:17:46.644273 12380 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142804us
I20260709 12:17:46.657666 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:46.659178 12568 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:17:46.660876 12568 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:17:46.663400 25728 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:46.663667 25711 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:46.669661 12566 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 19.
W20260709 12:17:46.682713 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177313us
W20260709 12:17:46.709896 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:46.676637 12565 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:17:46.716427 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1304868940001143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
W20260709 12:17:46.717387 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:46.699337 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:46.669373 25978 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:46.688570 25966 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:46.668694 25831 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:17:46.736037 25991 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.579761 25979 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.716715 25728 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.661393 25998 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.768824 25890 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.780982 25986 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.782869 25944 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:46.806092 25441 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.806847 26011 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:46.807519 26011 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.808195 25728 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.810194 25675 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:46.812108 25999 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:17:46.813084 25669 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.594s	user 0.010s	sys 0.000s
I20260709 12:17:46.816834 25989 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.820462 25989 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.816834 25669 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.823156 25669 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:46.827476 25979 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:46.834056 25995 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 12:17:46.841498 25996 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:46.843816 26007 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.063s	user 0.001s	sys 0.001s
I20260709 12:17:46.797286 25991 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:46.846510 25991 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:46.847030 25991 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:46.850194 25944 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.859781 25890 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.801841 25910 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:46.861331 25910 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.871074 25910 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.877174 25798 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:46.878048 25798 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:46.837483 25998 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.896689 25859 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:46.897461 25859 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:46.898600 25859 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:46.899971 25859 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:46.901854 26011 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:46.903414 25944 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:46.904040 25894 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:46.909654 25972 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.822s	user 0.001s	sys 0.000s
W20260709 12:17:46.910858 25982 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.003s
I20260709 12:17:46.911600 25982 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38.
I20260709 12:17:46.889241 25993 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:17:46.912760 25993 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:46.914228 25993 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.916421 25993 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:46.917014 25911 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:46.920168 26009 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.144s	user 0.002s	sys 0.000s
W20260709 12:17:46.922472 26003 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:17:46.924314 25876 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:46.924800 25876 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:46.925758 25876 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:46.926828 25876 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:46.993899 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2060) took 1487 ms. Trace:
I20260709 12:17:46.995059 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5354) took 1640 ms. Trace:
I20260709 12:17:46.998135 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.005956 12491 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:46.995884 12490 rpcz_store.cc:276] 0709 12:17:45.354816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.357853 (+  3037us) service_pool.cc:224] Handling call
0709 12:17:45.358176 (+   323us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:45.358567 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:45.358596 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:45.359174 (+   578us) log.cc:844] Serialized 52 byte log entry
0709 12:17:45.798795 (+439621us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:45.816130 (+ 17335us) log.cc:844] Serialized 14 byte log entry
0709 12:17:45.816449 (+   319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:45.816679 (+   230us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:46.872874 (+1056195us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b18004c3c60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:46.993966 (+121092us) raft_consensus.cc:1685] finished
0709 12:17:46.994223 (+   257us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:46.994751 (+   528us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5531,"prepare.queue_time_us":17171,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":4124,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":19317,"spinlock_wait_cycles":27945344,"thread_start_us":439204,"threads_started":1,"wal-append.queue_time_us":487396}
I20260709 12:17:47.004124 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.000255 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:47.007450 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.009498 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.011904 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:46.995450 12497 rpcz_store.cc:276] 0709 12:17:45.506471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.800770 (+294299us) service_pool.cc:224] Handling call
0709 12:17:45.801081 (+   311us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:45.801487 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:45.801519 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:45.807909 (+  6390us) log.cc:844] Serialized 26 byte log entry
0709 12:17:45.939084 (+131175us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:45.939211 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:45.939418 (+   207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:46.908748 (+969330us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:46.992257 (+ 83509us) raft_consensus.cc:1685] finished
0709 12:17:46.992448 (+   191us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:46.993446 (+   998us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":157,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":4282,"spinlock_wait_cycles":62032640,"thread_start_us":83620,"threads_started":1,"wal-append.queue_time_us":219837}
I20260709 12:17:47.015058 12568 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:47.032599 12637 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:17:47.032830 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:47.033548 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.034686 12638 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: [37.22-37.22]
I20260709 12:17:47.035704 25572 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.038520 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.079277 25572 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.080570 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:47.040102 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.083467 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:47.084178 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.042243 12643 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:47.043553 25859 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:47.086542 25859 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.043740 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1938) took 1666 ms. Trace:
I20260709 12:17:47.054627 12495 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:47.055717 25977 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:47.055928 26008 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:47.052155 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.057164 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:47.044028 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.041273 25112 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.063169 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.067296 12493 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 21. Preceding OpId from leader: term: 44 index: 22. (index mismatch)
I20260709 12:17:47.067981 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.068533 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:47.071951 26000 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.072928 12491 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:47.075440 12489 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:47.076138 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:47.078406 12646 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:47.085278 12641 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:47.056170 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.092075 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.089603 12644 rpcz_store.cc:276] 0709 12:17:45.376806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.380064 (+  3258us) service_pool.cc:224] Handling call
0709 12:17:45.388733 (+  8669us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:45.391068 (+  2335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:45.391122 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:46.550631 (+1159509us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:46.551024 (+   393us) log.cc:844] Serialized 26 byte log entry
0709 12:17:46.668612 (+117588us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:46.685377 (+ 16765us) log.cc:844] Serialized 14 byte log entry
0709 12:17:46.685759 (+   382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:46.685891 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:46.921827 (+235936us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:47.042976 (+121149us) raft_consensus.cc:1685] finished
0709 12:17:47.043035 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.043418 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4613,"prepare.queue_time_us":16152,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":7416,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":4477,"spinlock_wait_cycles":48415744,"thread_start_us":117238,"threads_started":1,"wal-append.queue_time_us":230131}
I20260709 12:17:47.101465 12564 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:47.135056 12564 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:47.136322 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5208) took 1187 ms. Trace:
I20260709 12:17:47.102018 25904 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:17:47.138752 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.102345 26008 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.142023 25993 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.102795 25977 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.145342 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.102589 26000 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.137171 12572 rpcz_store.cc:276] 0709 12:17:45.948255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.949933 (+  1678us) service_pool.cc:224] Handling call
0709 12:17:47.135565 (+1185632us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:47.135908 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":372912512}
I20260709 12:17:47.103184 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.150601 12643 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 30.
I20260709 12:17:47.151080 12638 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:17:47.112615 25963 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:47.152081 25963 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:47.152560 25963 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:47.141597 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.106072 25572 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.157169 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1646) took 2312 ms. Trace:
W20260709 12:17:47.162674 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.154963 12488 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:47.126128 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:47.131129 12495 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.24-31.24]
I20260709 12:17:47.131929 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.126495 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.184244 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5046) took 1408 ms. Trace:
I20260709 12:17:47.128577 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:47.160185 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.184864 12645 rpcz_store.cc:276] 0709 12:17:45.775901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.797123 (+ 21222us) service_pool.cc:224] Handling call
0709 12:17:47.183613 (+1386490us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:47.183957 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86262016}
I20260709 12:17:47.187582 13253 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:47.128839 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5355) took 1773 ms. Trace:
I20260709 12:17:47.129922 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.122278 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.205210 12494 rpcz_store.cc:276] 0709 12:17:45.355063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.358097 (+  3034us) service_pool.cc:224] Handling call
0709 12:17:45.358340 (+   243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:45.391851 (+ 33511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.128014 (+1736163us) spinlock_profiling.cc:243] Waited 32.4 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:47.128526 (+   512us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74417408}
I20260709 12:17:47.178910 13262 rpcz_store.cc:276] 0709 12:17:44.844130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:44.876617 (+ 32487us) service_pool.cc:224] Handling call
0709 12:17:44.876998 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:45.355269 (+478271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.156426 (+1801157us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:47.156887 (+   461us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1098486144}
I20260709 12:17:47.125550 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.216717 12565 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:47.222241 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.222606 12566 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 32. Current term is 33. Ops: []
I20260709 12:17:47.223474 25859 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.122843 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5353) took 1743 ms. Trace:
W20260709 12:17:47.232077 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.226255 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:47.225420 25694 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.004s	user 0.010s	sys 0.014s
I20260709 12:17:47.143101 26008 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.132766 25189 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:47.240667 12646 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:47.190050 25572 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:47.245986 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:47.136710 12564 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.226318 26000 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.245968 25694 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.250437 25572 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.259603 25572 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.261616 12492 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:47.261917 25682 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:47.263037 25682 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.267091 25694 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.273191 12644 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 45
I20260709 12:17:47.271257 25625 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.003s
I20260709 12:17:47.274981 25625 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:47.276729 12642 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 18. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:17:47.277451 25812 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:47.278483 25812 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.280910 12639 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 21. Preceding OpId from leader: term: 44 index: 22. (index mismatch)
I20260709 12:17:47.282562 25189 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:47.113546 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83895.6us
W20260709 12:17:47.129093 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 399595us
I20260709 12:17:47.305536 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.314450 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.315768 25684 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:47.303810 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50977.4us
I20260709 12:17:47.319350 12491 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:47.275733 25625 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:47.328195 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.114730 25112 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.331562 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.334514 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.336336 25694 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:47.341589 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.227694 12496 rpcz_store.cc:276] 0709 12:17:45.354539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:45.359742 (+  5203us) service_pool.cc:224] Handling call
0709 12:17:45.360037 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:45.369186 (+  9149us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:46.712272 (+1343086us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b18004ba9c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:17:46.995292 (+283020us) log.cc:844] Serialized 14 byte log entry
0709 12:17:47.014007 (+ 18715us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:47.015610 (+  1603us) log.cc:844] Serialized 26 byte log entry
0709 12:17:47.017425 (+  1815us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:47.017540 (+   115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.017782 (+   242us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:47.046111 (+ 28329us) raft_consensus.cc:1685] finished
0709 12:17:47.048748 (+  2637us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.097779 (+ 49031us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9582,"prepare.queue_time_us":77394,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":37211,"raft.run_cpu_time_us":496,"raft.run_wall_time_us":7871,"spinlock_wait_cycles":13977472,"thread_start_us":300196,"threads_started":2,"wal-append.queue_time_us":18671}
I20260709 12:17:47.143170 25993 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.347489 12644 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:47.338712 12637 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:47.358224 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.347182 12490 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:47.362380 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.366135 26008 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:47.367722 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.369980 25590 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.370796 25572 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:47.371399 25572 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:47.371865 25572 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:47.373183 25694 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.381175 25872 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:47.381883 25872 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:47.383665 25872 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.381500 25977 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.386302 25872 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:47.371047 25590 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.378053 12564 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:47.396384 12571 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:47.323401 25684 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.405108 12646 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:17:47.411365 25993 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.411952 25812 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:47.412500 12492 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
W20260709 12:17:47.413216 26027 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.125s	user 0.000s	sys 0.002s
I20260709 12:17:47.413714 25189 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:47.414914 25894 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.416375 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:47.418231 13253 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:47.438299 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.439257 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2596) took 1069 ms. Trace:
I20260709 12:17:47.439615 12571 rpcz_store.cc:276] 0709 12:17:46.364473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:46.464238 (+ 99765us) service_pool.cc:224] Handling call
0709 12:17:47.419197 (+954959us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:47.433907 (+ 14710us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1135387392}
I20260709 12:17:47.436646 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:47.446887 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68539.9us
I20260709 12:17:47.448225 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:47.448573 25689 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:47.449096 25811 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:47.455884 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 164124us
I20260709 12:17:47.474368 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2594) took 1087 ms. Trace:
I20260709 12:17:47.475085 25441 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:47.476306 25894 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.471237 13256 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:47.459326 25812 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election decision vote started in defunct term 43: won
I20260709 12:17:47.456208 25682 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:47.486950 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:47.470535 25811 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.486356 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:47.473063 25694 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.495961 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.449615 25689 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:47.499711 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.482779 25684 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.480432 25590 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.502818 25811 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.508016 25894 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:47.510105 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49266.1us
I20260709 12:17:47.512722 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:47.515964 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:47.519891 25689 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.510473 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:47.479441 12563 rpcz_store.cc:276] 0709 12:17:46.363748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:46.363918 (+   170us) service_pool.cc:224] Handling call
0709 12:17:46.380496 (+ 16578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:46.380741 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:17:46.391780 (+ 11039us) log.cc:844] Serialized 14 byte log entry
0709 12:17:47.080508 (+688728us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b1800472360. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:17:47.307893 (+227385us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:47.340473 (+ 32580us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:17:47.424202 (+ 83729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.432235 (+  8033us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.450549 (+ 18314us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10881,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":67,"raft.queue_time_us":63565,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":216,"spinlock_wait_cycles":4259584,"thread_start_us":158752,"threads_started":1,"wal-append.queue_time_us":1061820}
W20260709 12:17:47.527853 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:47.525548 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109556us
W20260709 12:17:47.535468 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49480.8us
I20260709 12:17:47.543744 12565 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:47.553752 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.539039 25689 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:47.520777 25625 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.564108 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.564751 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.564966 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.565878 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:47.569569 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.570297 12637 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:47.555660 25812 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:47.575784 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.571981 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.570762 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.574954 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.581198 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:47.583076 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:47.585990 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.599026 25572 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:47.601784 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.591871 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:47.606138 12641 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:47.578598 25812 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:47.602455 12638 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:47.609160 12640 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:47.622375 12642 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:47.592358 25112 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.601511 25572 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:47.621374 25812 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:47.603878 25963 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.632055 25689 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:47.644253 25689 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:47.644491 25676 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:47.645730 25676 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.648824 25676 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:47.653172 25977 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.011s
I20260709 12:17:47.654491 25977 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:47.654989 25977 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:47.656705 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 168024us
I20260709 12:17:47.659737 25590 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:17:47.635018 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66449.5us
I20260709 12:17:47.637234 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:47.676433 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:47.680315 12563 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 17. Preceding OpId from leader: term: 43 index: 17. (term mismatch)
I20260709 12:17:47.680737 12563 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 16
I20260709 12:17:47.677426 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.685262 25572 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:17:47.689203 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 122362us
I20260709 12:17:47.681041 12563 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.17
I20260709 12:17:47.690497 12563 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:47.703280 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.706931 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.679495 25590 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:17:47.711764 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:47.722091 12489 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:47.723048 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5451) took 3910 ms. Trace:
I20260709 12:17:47.682133 25625 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:47.717588 25590 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:47.732710 26029 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.083s	user 0.000s	sys 0.005s
I20260709 12:17:47.733043 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:47.734078 25572 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:17:47.734387 26031 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.195s	user 0.000s	sys 0.002s
W20260709 12:17:47.734997 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69286.7us
I20260709 12:17:47.735208 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5232) took 1097 ms. Trace:
I20260709 12:17:47.735783 13258 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:17:47.736438 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5226) took 1321 ms. Trace:
W20260709 12:17:47.736948 25814 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 40 current term: 40 is higher.
I20260709 12:17:47.740603 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5228) took 1325 ms. Trace:
I20260709 12:17:47.733784 25590 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.741248 25790 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:47.681262 25682 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.049s
I20260709 12:17:47.741027 12567 rpcz_store.cc:276] 0709 12:17:46.415336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.130618 (+715282us) service_pool.cc:224] Handling call
0709 12:17:47.131418 (+   800us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.238614 (+107196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.238657 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:47.272120 (+ 33463us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:47.272772 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 12:17:47.332579 (+ 59807us) spinlock_profiling.cc:243] Waited 36.2 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:17:47.428282 (+ 95703us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:47.428353 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.428514 (+   161us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:47.740281 (+311767us) raft_consensus.cc:1685] finished
0709 12:17:47.740302 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.740437 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93805,"raft.queue_time_us":111668,"raft.run_cpu_time_us":3767,"raft.run_wall_time_us":84450,"spinlock_wait_cycles":133575680,"thread_start_us":154794,"threads_started":1,"wal-append.queue_time_us":404051}
I20260709 12:17:47.736909 12570 rpcz_store.cc:276] 0709 12:17:46.414812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.105461 (+690649us) service_pool.cc:224] Handling call
0709 12:17:47.204230 (+ 98769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:47.284022 (+ 79792us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:17:47.284500 (+   478us) log.cc:844] Serialized 14 byte log entry
0709 12:17:47.425801 (+141301us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:47.425862 (+    61us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:47.425912 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.425977 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.712831 (+286854us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b1800285760. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:17:47.736112 (+ 23281us) spinlock_profiling.cc:243] Waited 69.8 ms on lock 0x7b5400044f50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:47.736255 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78938,"prepare.queue_time_us":856,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":77,"raft.queue_time_us":85996,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":4318,"spinlock_wait_cycles":173533824,"thread_start_us":42878,"threads_started":1,"wal-append.queue_time_us":170263}
W20260709 12:17:47.747033 26026 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.217s	user 0.002s	sys 0.000s
I20260709 12:17:47.736737 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:47.749209 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.750300 25905 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:47.750752 25905 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:47.751245 25905 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:47.751435 26017 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:47.752489 26017 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:47.754020 25876 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:47.754462 25876 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:47.755012 25904 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:47.755748 25876 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.755764 25904 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:47.758388 25904 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.761402 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.767155 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.733865 13261 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:47.772006 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.735849 12563 rpcz_store.cc:276] 0709 12:17:46.637721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.590546 (+952825us) service_pool.cc:224] Handling call
0709 12:17:47.626781 (+ 36235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:47.734839 (+108058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.735131 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:17:47.786018 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 163793us
W20260709 12:17:47.786507 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:47.787005 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 237971us
I20260709 12:17:47.700723 25894 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:47.735387 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5229) took 1319 ms. Trace:
I20260709 12:17:47.756383 12566 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 20. (term mismatch)
I20260709 12:17:47.798075 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.727049 25977 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.720613 25689 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.742044 25790 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:47.797201 25441 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:47.794399 12566 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:17:47.794209 25894 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:47.742708 25682 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:47.788983 12569 rpcz_store.cc:276] 0709 12:17:46.415564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.133082 (+717518us) service_pool.cc:224] Handling call
0709 12:17:47.133858 (+   776us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.238315 (+104457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.238380 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:47.254584 (+ 16204us) log.cc:844] Serialized 26 byte log entry
0709 12:17:47.357806 (+103222us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:47.357858 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.357944 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:47.720904 (+362960us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:47.734892 (+ 13988us) raft_consensus.cc:1685] finished
0709 12:17:47.734918 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.735192 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":135299,"raft.queue_time_us":109465,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":45217,"spinlock_wait_cycles":29922560,"thread_start_us":102479,"threads_started":1,"wal-append.queue_time_us":369658}
I20260709 12:17:47.803457 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:47.813382 12566 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 38.20
I20260709 12:17:47.835347 12566 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:47.825210 25682 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:47.838076 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.838830 25572 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:47.838642 25901 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:47.839154 25682 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.839738 25572 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:47.821676 25894 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:47.840921 25682 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:47.802816 25977 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:47.805825 25590 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:47.831391 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.833339 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.825130 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.849906 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.850680 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:47.833575 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:47.853058 12495 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 28. Preceding OpId from leader: term: 41 index: 30. (index mismatch)
I20260709 12:17:47.859030 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:47.837580 26033 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:17:47.801383 12567 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:47.866477 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.868851 12569 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 43 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 18.
W20260709 12:17:47.829792 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 159114us
I20260709 12:17:47.862251 12496 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:47.801141 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:47.873133 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72125.4us
W20260709 12:17:47.882990 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:47.890884 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:47.897969 12489 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:47.902567 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5478) took 1398 ms. Trace:
I20260709 12:17:47.903100 12572 rpcz_store.cc:276] 0709 12:17:46.519783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.282336 (+762553us) service_pool.cc:224] Handling call
0709 12:17:47.282942 (+   606us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:47.314932 (+ 31990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.315011 (+    79us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:47.315963 (+   952us) log.cc:844] Serialized 52 byte log entry
0709 12:17:47.620575 (+304612us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:47.620662 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.620805 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:47.902296 (+281491us) raft_consensus.cc:1685] finished
0709 12:17:47.902316 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.902499 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30835,"raft.queue_time_us":32593,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":177,"thread_start_us":303884,"threads_started":1,"wal-append.queue_time_us":413696}
I20260709 12:17:47.906136 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:47.909312 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:47.912142 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:47.915650 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148660us
I20260709 12:17:47.915946 12491 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:47.918737 25872 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.531s	user 0.016s	sys 0.005s
I20260709 12:17:47.919843 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:47.808699 25790 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:47.926057 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139767us
I20260709 12:17:47.927402 25963 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:47.928689 25798 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.050s	user 0.011s	sys 0.017s
I20260709 12:17:47.929327 25798 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.931794 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.935290 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.932565 12488 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:47.936352 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:47.946427 12568 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:47.948414 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.885031 25993 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:47.952319 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:47.915658 25991 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.068s	user 0.009s	sys 0.015s
W20260709 12:17:47.956453 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74147.2us
I20260709 12:17:47.962217 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:47.969005 25676 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:47.971184 25668 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.972061 25668 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:47.964483 25597 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:47.946045 25963 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:47.926359 25812 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.969213 25493 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:47.968134 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:17:47.830087 25905 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:47.955904 25993 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.013492 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:48.022219 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57179.4us
W20260709 12:17:48.042739 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.055792 12491 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:48.058493 25668 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.058862 25493 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.059950 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:48.060619 12642 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:48.061791 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.062711 25876 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:48.066638 12566 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 22. Preceding OpId from leader: term: 45 index: 23. (index mismatch)
W20260709 12:17:48.068890 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111018us
I20260709 12:17:48.071964 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.981047 25963 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.013288 25872 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:17:48.096300 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 121972us
I20260709 12:17:48.103932 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:17:48.110256 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:47.982122 25798 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.123032 12489 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:48.058326 26016 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.008s	sys 0.011s
I20260709 12:17:48.130301 12491 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:17:48.055158 25993 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:48.138849 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.140218 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.146667 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.148484 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:48.143851 12641 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:17:48.061533 25493 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:48.134661 26016 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.008s	sys 0.012s
I20260709 12:17:48.117843 25872 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.169166 12644 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 45 index: 23. (index mismatch)
I20260709 12:17:47.978341 25991 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.131428 25668 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:48.187719 12640 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:17:48.197531 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.202945 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:48.198767 12641 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:17:48.221936 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.184592 25872 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:17:48.190189 26027 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:17:47.724355 12564 rpcz_store.cc:276] 0709 12:17:43.810482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:43.812461 (+  1979us) service_pool.cc:224] Handling call
0709 12:17:43.812910 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:47.101103 (+3288193us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b54004f0720. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:17:47.701596 (+600493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.701667 (+    71us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:47.701737 (+    70us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:47.701802 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:47.701941 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:47.720439 (+ 18498us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1058,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":35769,"spinlock_wait_cycles":5605389056}
I20260709 12:17:48.176819 25493 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.227423 12495 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:48.222116 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 365859us
I20260709 12:17:48.204064 25689 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:48.167084 25676 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.067s
W20260709 12:17:48.241030 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 402750us
I20260709 12:17:48.248080 12491 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
W20260709 12:17:48.266657 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.175765 25814 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.064s
I20260709 12:17:48.218602 25894 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.283432 26034 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.139s
I20260709 12:17:48.298257 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:48.315423 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 244225us
I20260709 12:17:48.322906 12491 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:48.246289 25493 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:48.328339 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 210772us
I20260709 12:17:48.325673 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:48.356956 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:48.370577 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 128328us
I20260709 12:17:48.336417 25905 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.347312 26050 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:48.385093 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.306206 25689 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.385102 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.362547 25812 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:48.395951 12491 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:17:48.396639 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5481) took 1854 ms. Trace:
W20260709 12:17:48.397883 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 454009us
W20260709 12:17:48.394928 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99830.3us
I20260709 12:17:48.379245 25963 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.397029 12568 rpcz_store.cc:276] 0709 12:17:46.541927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.379811 (+837884us) service_pool.cc:224] Handling call
0709 12:17:47.380370 (+   559us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:47.951766 (+571396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.396348 (+444582us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:48.396572 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1267600256}
W20260709 12:17:48.386377 12610 connection.cc:256] RPC call timeout handler was delayed by -1.079349147999892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
I20260709 12:17:48.389335 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:48.388842 26050 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:48.416365 26045 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:17:48.417827 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48633.2us
I20260709 12:17:48.418392 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:48.428715 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.434676 25493 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:48.435727 25977 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.594s	user 0.017s	sys 0.000s
W20260709 12:17:48.437734 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.437944 25963 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:48.438740 25963 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.441262 25963 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.442013 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5234) took 1089 ms. Trace:
I20260709 12:17:48.443300 25872 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.443817 12570 rpcz_store.cc:276] 0709 12:17:47.353001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.746883 (+393882us) service_pool.cc:224] Handling call
0709 12:17:47.747458 (+   575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.748554 (+  1096us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.748596 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:47.749313 (+   717us) log.cc:844] Serialized 26 byte log entry
0709 12:17:48.127557 (+378244us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:48.127599 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.127674 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:48.441721 (+314047us) raft_consensus.cc:1685] finished
0709 12:17:48.441749 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.441939 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":150,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":376382,"threads_started":1,"wal-append.queue_time_us":525212}
I20260709 12:17:48.445953 25872 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.447072 25689 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.454095 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2605) took 1301 ms. Trace:
I20260709 12:17:48.455132 25112 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.455813 25112 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:48.456848 26048 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:17:48.454767 12565 rpcz_store.cc:276] 0709 12:17:47.153174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.740983 (+587809us) service_pool.cc:224] Handling call
0709 12:17:47.741412 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.743573 (+  2161us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:17:47.744415 (+   842us) log.cc:844] Serialized 14 byte log entry
0709 12:17:48.171405 (+426990us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:48.208988 (+ 37583us) log.cc:844] Serialized 26 byte log entry
0709 12:17:48.209240 (+   252us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:48.209303 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.209407 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:48.453767 (+244360us) raft_consensus.cc:1685] finished
0709 12:17:48.453797 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.454007 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":186,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":1149,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":175,"thread_start_us":426582,"threads_started":1,"wal-append.queue_time_us":665159}
I20260709 12:17:48.460220 12495 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:17:48.426720 25905 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:48.461040 25112 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.461228 25905 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:48.417068 26050 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:48.461858 25905 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:48.462648 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4093) took 1130 ms. Trace:
I20260709 12:17:48.462896 25689 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.464110 25689 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.463109 12639 rpcz_store.cc:276] 0709 12:17:47.332418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.345341 (+ 12923us) service_pool.cc:224] Handling call
0709 12:17:47.371884 (+ 26543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:47.372254 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:47.627484 (+255230us) log.cc:844] Serialized 14 byte log entry
0709 12:17:48.342069 (+714585us) log.cc:844] Serialized 14 byte log entry
0709 12:17:48.462057 (+119988us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b18002eac80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:17:48.462228 (+   171us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:48.462285 (+    57us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:48.462334 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.462407 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.462574 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7690,"prepare.queue_time_us":385786,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":249761,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":18092,"spinlock_wait_cycles":59266176,"thread_start_us":889717,"threads_started":3}
I20260709 12:17:48.466650 26065 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:48.466912 25689 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.467406 26065 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.463238 25112 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.437443 26052 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:48.471711 25112 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:48.472424 26052 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:48.473753 26052 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:48.475986 25112 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.476815 26034 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.478449 25894 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.477516 26034 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:48.477350 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:48.480997 26034 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.482064 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.484992 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:48.485857 12496 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:48.487331 26043 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:17:48.488740 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5231) took 1851 ms. Trace:
I20260709 12:17:48.489585 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:48.505865 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75559.4us
I20260709 12:17:48.489176 12571 rpcz_store.cc:276] 0709 12:17:46.637380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.589966 (+952586us) service_pool.cc:224] Handling call
0709 12:17:47.638268 (+ 48302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.688684 (+ 50416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.688728 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:47.753782 (+ 65054us) log.cc:844] Serialized 26 byte log entry
0709 12:17:48.240610 (+486828us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:48.240681 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.240799 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:48.488486 (+247687us) raft_consensus.cc:1685] finished
0709 12:17:48.488519 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.488674 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":679,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":15777,"thread_start_us":486260,"threads_started":1,"wal-append.queue_time_us":616710}
I20260709 12:17:48.508567 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:48.509496 26017 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:48.510304 26017 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:48.493878 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:48.510820 26017 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:48.494071 26056 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:17:48.517138 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5237) took 1058 ms. Trace:
W20260709 12:17:48.496198 12645 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.005s	sys 0.023s
W20260709 12:17:48.519096 12645 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.231s	user 0.005s	sys 0.024s
I20260709 12:17:48.519150 26062 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:48.519932 26062 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:48.500659 26037 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.452s	user 0.001s	sys 0.002s
I20260709 12:17:48.500846 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.501341 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.517745 12569 rpcz_store.cc:276] 0709 12:17:47.458606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.910733 (+452127us) service_pool.cc:224] Handling call
0709 12:17:47.911278 (+   545us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.944445 (+ 33167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.944677 (+   232us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:47.958486 (+ 13809us) log.cc:844] Serialized 26 byte log entry
0709 12:17:48.302904 (+344418us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:48.302959 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.303054 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:48.516816 (+213762us) raft_consensus.cc:1685] finished
0709 12:17:48.516850 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.517048 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20313,"raft.queue_time_us":37973,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":2314,"thread_start_us":338961,"threads_started":1,"wal-append.queue_time_us":432225}
I20260709 12:17:48.527338 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.528389 12496 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:48.520394 26062 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:48.417469 25991 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.535342 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.536741 12488 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:48.540940 12489 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 16. Preceding OpId from leader: term: 43 index: 17. (index mismatch)
I20260709 12:17:48.544993 12490 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:17:48.546362 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.543063 25905 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.548995 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:48.552317 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:48.553241 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:48.553658 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.554100 26059 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.554258 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.554754 26059 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.555569 25991 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.555846 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:48.556316 25991 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.557824 25977 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.558418 12489 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:48.559073 25991 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.558473 25977 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.560263 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:48.561187 13253 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:48.562222 25977 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.518507 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.564502 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.565862 25814 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:48.566473 25814 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.566978 25814 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:48.570120 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5236) took 1119 ms. Trace:
I20260709 12:17:48.571678 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.573814 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:48.574682 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.575480 26059 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.575860 12488 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
W20260709 12:17:48.565285 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46892.2us
I20260709 12:17:48.578408 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.579514 25905 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.575392 26052 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.583215 26052 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.583599 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:48.584966 25811 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:48.585126 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:48.585736 25811 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:48.577513 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.586390 25112 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:48.495247 26065 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:48.546276 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:48.589560 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.590906 13253 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:48.590030 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.593047 13256 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:48.594625 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:48.595682 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:48.598141 26059 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:48.598750 26059 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.599362 26059 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:48.502856 25590 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.660s	user 0.014s	sys 0.000s
I20260709 12:17:48.601959 25977 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:17:48.602818 25977 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:48.605895 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:48.605942 25977 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:48.626356 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.606954 25905 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:48.586375 25811 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:48.570921 12563 rpcz_store.cc:276] 0709 12:17:47.450957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.907764 (+456807us) service_pool.cc:224] Handling call
0709 12:17:47.908302 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:47.954709 (+ 46407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:47.954744 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:48.072143 (+117399us) log.cc:844] Serialized 26 byte log entry
0709 12:17:48.378126 (+305983us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:48.378193 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.378318 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:48.569731 (+191413us) raft_consensus.cc:1685] finished
0709 12:17:48.569765 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.570021 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22170,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":216,"thread_start_us":222040,"threads_started":1,"wal-append.queue_time_us":297599}
I20260709 12:17:48.611618 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.613878 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.614497 12491 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:17:48.614287 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:48.617530 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.626567 12568 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
W20260709 12:17:48.609732 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 176761us
I20260709 12:17:48.633466 12570 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 37 because replica has last-logged OpId of term: 35 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:17:48.639145 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:48.639472 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.640448 12570 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:48.640972 12568 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:17:48.633845 12572 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:48.642400 12568 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:48.642730 26052 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:17:48.646235 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.535208 26017 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.636920 12569 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:48.656337 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.604261 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:48.669350 13256 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:48.649848 26065 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:48.669960 25112 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:48.670382 26065 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:48.671154 26065 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:48.678196 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.634528 25905 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:48.670499 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 240508us
I20260709 12:17:48.657392 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:48.696797 12497 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:48.687978 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.685590 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:48.703608 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.703891 12571 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:48.685141 25905 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:48.706921 25993 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.503s	user 0.008s	sys 0.005s
I20260709 12:17:48.707602 25993 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.710407 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.712896 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:48.713589 13256 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:17:48.705473 25876 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.642s	user 0.012s	sys 0.006s
I20260709 12:17:48.686684 25112 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:48.739887 25994 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:48.740759 25994 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.726534 25441 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:48.744041 25441 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.744516 25441 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:48.744570 26062 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:48.746794 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142605us
I20260709 12:17:48.748605 26070 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.756907 26070 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.758078 25811 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:48.760629 26070 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.749814 25590 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:48.719565 25189 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:48.750401 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.714601 25991 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:48.732156 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:48.731716 26060 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:17:48.762465 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.025s	sys 0.036s
W20260709 12:17:48.769762 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.025s	sys 0.037s
I20260709 12:17:48.770205 12492 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:48.770524 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.681567 26050 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:48.764031 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53702.6us
W20260709 12:17:48.763959 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 234150us
W20260709 12:17:48.776113 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:48.776919 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:48.781356 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49867.8us
I20260709 12:17:48.785467 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:48.784468 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:48.763324 25811 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.767961 25189 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:48.762872 25994 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.770614 25991 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:48.750150 25814 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.794840 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:48.795109 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:48.793895 26060 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:17:48.783469 25894 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.769836 25625 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.022s
I20260709 12:17:48.792804 26070 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:48.769183 25689 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.023s
I20260709 12:17:48.808772 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5067) took 1574 ms. Trace:
I20260709 12:17:48.790138 25684 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:48.812319 12566 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:48.815653 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:48.794853 25189 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:48.794951 25991 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:48.794380 26034 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:48.819581 26072 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:48.797394 26055 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:48.829094 26068 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.053s
I20260709 12:17:48.807816 26025 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:48.834697 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:48.823825 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60136.7us
I20260709 12:17:48.823849 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:48.822839 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:48.837096 13256 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:48.837685 12642 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:48.806540 25811 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:48.824611 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.813472 26070 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:48.848598 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:48.815379 25993 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.821934 25684 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:48.841584 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.841599 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:48.843637 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.843953 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:48.844354 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.846066 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.812029 25963 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 20, Time since last communication: 0.026s
I20260709 12:17:48.827631 25590 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:17:48.830215 26062 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.850324 12639 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:48.853303 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:48.882143 12567 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:48.887465 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.865062 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:48.863754 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77312.9us
I20260709 12:17:48.898380 12642 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:48.832319 25790 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:48.899236 12572 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:17:48.908366 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:48.851732 12641 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:48.911518 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:48.916731 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:48.921116 12568 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:17:48.915210 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:48.912457 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:48.849259 26070 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:48.851403 26017 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:48.897553 25684 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:17:48.967605 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66182.2us
I20260709 12:17:48.851868 25994 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:48.862859 25814 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.835191 25894 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:48.870818 26050 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:48.977358 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.979452 25590 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:48.980947 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:48.983654 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 156326us
W20260709 12:17:48.986999 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76175.6us
W20260709 12:17:48.987864 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 146351us
I20260709 12:17:48.848399 25112 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.120s
I20260709 12:17:48.917390 25876 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:49.004834 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:48.916082 25790 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:48.915729 26034 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:49.023442 12571 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:17:49.001785 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:48.861194 25572 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.121s
I20260709 12:17:48.973850 26072 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:48.833475 26055 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.035759 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:48.866829 25441 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.033038 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:17:48.975435 25994 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:48.910295 25811 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.088s
W20260709 12:17:49.048830 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 216316us
I20260709 12:17:49.002697 25590 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:17:49.054853 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.056847 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:49.064416 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:49.069752 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.055114 25994 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:49.056349 26017 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:49.088817 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:49.073083 25676 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:49.076002 26059 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.104018 26083 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:49.075398 25901 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:17:49.105176 12564 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:49.083604 26017 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:49.120985 13256 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:17:49.142346 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98003.8us
W20260709 12:17:49.145670 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56978.8us
W20260709 12:17:49.125247 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.096832 25676 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:49.124306 26017 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:17:49.156736 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96784.9us
I20260709 12:17:49.166684 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.170295 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.142114 25814 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:49.158295 26042 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:17:49.192730 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.199265 26059 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.159962 25872 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:49.209995 12386 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 20 to 34. New cstate: current_term: 34 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:49.150537 25811 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:49.147363 26061 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:17:49.157013 25676 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:49.207829 26077 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.152s	user 0.003s	sys 0.000s
I20260709 12:17:49.160300 25876 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:49.164295 26079 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:17:49.236862 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90164.1us
I20260709 12:17:49.221666 25814 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:49.250286 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80089.2us
I20260709 12:17:49.123790 25790 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.233577 26061 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:49.159775 25590 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.293500 13254 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:17:49.291188 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.231092 25811 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:49.247066 26079 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:49.305881 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:49.284982 25977 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.640s	user 0.013s	sys 0.001s
W20260709 12:17:49.306563 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46463.8us
I20260709 12:17:49.216683 26087 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.119s
I20260709 12:17:49.306672 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:48.816645 12645 rpcz_store.cc:276] 0709 12:17:47.234933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:47.240331 (+  5398us) service_pool.cc:224] Handling call
0709 12:17:47.285610 (+ 45279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:47.286707 (+  1097us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:48.519571 (+1232864us) log.cc:844] Serialized 14 byte log entry
0709 12:17:48.807477 (+287906us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:48.807546 (+    69us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:48.807607 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:48.807683 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:48.808686 (+  1003us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20631,"prepare.queue_time_us":460829,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":35000,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":7524,"thread_start_us":1484081,"threads_started":2}
W20260709 12:17:49.318888 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.322350 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:49.310287 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67093.6us
I20260709 12:17:49.232502 25872 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.104779 26083 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:49.353834 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48195.4us
W20260709 12:17:49.316342 26077 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:17:49.354832 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:49.246205 25876 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.249723 26059 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.096s
I20260709 12:17:49.316947 26034 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:49.305294 26088 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:49.316998 25811 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:49.144757 26072 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:49.355221 26082 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.272s	user 0.002s	sys 0.000s
W20260709 12:17:49.358459 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.297904 26068 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.059s
I20260709 12:17:49.356973 25441 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.324555 25901 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.045s
W20260709 12:17:49.385403 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60702.4us
I20260709 12:17:49.348593 25977 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.400826 12571 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 30. Current term is 31. Ops: []
I20260709 12:17:49.398754 26055 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:49.392545 26108 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:49.383153 26034 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.417709 26099 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:17:49.444574 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68163.3us
I20260709 12:17:49.374569 26061 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:49.476955 26089 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:17:49.475059 26097 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:49.465816 26084 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:49.473709 26101 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.120s
I20260709 12:17:49.491240 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:49.500501 26061 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:49.406757 25441 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.034s
I20260709 12:17:49.436794 26025 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:17:49.515355 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 277685us
I20260709 12:17:49.437714 26098 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:49.438115 26108 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:49.540750 26102 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.019s
W20260709 12:17:49.543761 26104 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:17:49.546264 25872 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.547124 26092 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:49.451256 26109 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:17:49.532778 26100 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:17:49.520619 26084 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:49.550026 26084 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:17:49.552673 26093 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:17:49.552850 26087 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:49.558553 26083 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.538230 26088 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:49.562150 25814 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.563463 26098 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.565765 26108 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:49.569659 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 395092us
I20260709 12:17:49.561507 25876 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:49.591466 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209163us
W20260709 12:17:49.593921 25991 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.762s	user 0.015s	sys 0.007s
I20260709 12:17:49.529803 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:49.619072 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:17:49.602154 26065 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.930s	user 0.021s	sys 0.007s
W20260709 12:17:49.526623 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114665us
I20260709 12:17:49.624133 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:49.627633 12567 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:49.628129 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.649385 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:49.650470 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 120850us
I20260709 12:17:49.651804 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:49.625231 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:49.656152 12563 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:49.653280 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:49.663197 26112 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:49.620534 25991 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.668812 25991 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:49.595114 25572 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.031s
I20260709 12:17:49.669734 26101 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:49.673525 26025 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:49.674039 26025 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:49.675132 25905 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.969s	user 0.024s	sys 0.008s
I20260709 12:17:49.675041 26025 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.675591 25905 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.676424 26025 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:49.565537 26088 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:49.683468 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55414.1us
I20260709 12:17:49.684468 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:49.669941 25977 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.648947 26065 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.694149 25876 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:49.695116 25876 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:49.695566 25876 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:49.695899 26065 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:49.699913 25790 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:49.706046 26117 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:49.710944 26098 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:49.712633 26034 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.623724 26120 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.099s
I20260709 12:17:49.655647 26114 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.061s
I20260709 12:17:49.716966 26114 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:49.717514 26114 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:49.718941 26114 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:49.681408 25994 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.593s	user 0.010s	sys 0.010s
I20260709 12:17:49.692315 25905 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.685092 26101 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:49.728753 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75568.3us
W20260709 12:17:49.738219 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 388108us
I20260709 12:17:49.747385 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:49.754302 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:49.773205 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88818.9us
I20260709 12:17:49.776401 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:49.734424 26112 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:17:49.781934 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 487873us
I20260709 12:17:49.782631 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:49.782888 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:49.726866 26114 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:49.806488 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:49.807247 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59943us
I20260709 12:17:49.810292 12566 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:49.733567 25994 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.751957 26101 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:49.851991 26079 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.814131 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:49.812954 13260 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:49.819947 25994 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:49.821354 26091 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.591s	user 0.002s	sys 0.004s
W20260709 12:17:49.821866 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.748236 25811 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.826176 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.776309 25977 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:49.884474 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:49.833263 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 40 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:49.836217 25189 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.003s	user 0.011s	sys 0.010s
I20260709 12:17:49.888681 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:49.813664 25963 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:49.846441 12386 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 31 to 35. New cstate: current_term: 35 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:49.813318 25814 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:49.849018 25590 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:17:49.849269 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 240289us
I20260709 12:17:49.855276 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:49.813870 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:49.912014 12569 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:49.912292 13261 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:49.860011 12567 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:49.814905 13253 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:49.862577 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:49.866405 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.914511 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:49.915359 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:49.870728 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61010.7us
I20260709 12:17:49.878868 13256 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:17:49.874819 26070 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.841s	user 0.015s	sys 0.015s
W20260709 12:17:49.825362 25684 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.829s	user 0.015s	sys 0.015s
W20260709 12:17:49.838330 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:49.906486 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:49.916412 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:49.871542 26084 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.908387 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:49.922497 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 297323us
I20260709 12:17:49.927935 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:49.930428 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67965us
I20260709 12:17:49.907656 25189 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.886579 26120 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
W20260709 12:17:49.928748 25676 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.017s
I20260709 12:17:49.932865 25994 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:49.985332 25994 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:49.933207 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:50.010967 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77860.1us
I20260709 12:17:49.936237 12566 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:49.937187 12571 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:49.941417 12639 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:49.942763 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:50.016425 12565 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:50.017362 13256 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:49.934188 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:49.936971 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 33 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:49.981889 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5518) took 1094 ms. Trace:
W20260709 12:17:49.974985 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 148925us
I20260709 12:17:50.009487 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:49.988840 25994 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:49.901743 25977 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:49.943149 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:49.943742 12386 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 29 to 34. New cstate: current_term: 34 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:50.026938 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:49.982018 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:50.027823 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5095) took 1186 ms. Trace:
I20260709 12:17:50.028292 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:50.022627 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5519) took 1118 ms. Trace:
I20260709 12:17:50.041949 12569 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:49.936550 25812 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:49.947728 26124 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:49.933370 26122 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:49.937849 26125 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:50.037364 12563 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:50.022109 12572 rpcz_store.cc:276] 0709 12:17:48.888215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:48.919855 (+ 31640us) service_pool.cc:224] Handling call
0709 12:17:48.920248 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:49.358680 (+438432us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:49.358728 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:49.359346 (+   618us) log.cc:844] Serialized 26 byte log entry
0709 12:17:49.629169 (+269823us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:49.629227 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:49.629348 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:49.965428 (+336080us) raft_consensus.cc:1685] finished
0709 12:17:49.965460 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:49.981760 (+ 16300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13802,"raft.queue_time_us":29848,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":23774,"thread_start_us":608503,"threads_started":2,"wal-append.queue_time_us":443120}
W20260709 12:17:49.951915 26118 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.002s	sys 0.000s
I20260709 12:17:50.037573 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:50.037997 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 103928us
I20260709 12:17:50.065244 12566 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:50.065323 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:49.875264 26108 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.041323 12567 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:49.943482 25493 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:49.976500 26111 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:49.937430 25112 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:17:50.043409 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 127057us
I20260709 12:17:50.041982 12565 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:49.989154 25894 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:50.040932 12639 rpcz_store.cc:276] 0709 12:17:48.841010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:49.051205 (+210195us) service_pool.cc:224] Handling call
0709 12:17:49.069975 (+ 18770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:50.021596 (+951621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:50.027571 (+  5975us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:50.027760 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2003486976}
I20260709 12:17:50.025141 26083 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:50.045380 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:50.045989 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:49.942941 25963 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:50.048378 13261 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:49.941143 25189 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:50.042178 12570 rpcz_store.cc:276] 0709 12:17:48.904003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:48.972369 (+ 68366us) service_pool.cc:224] Handling call
0709 12:17:48.972753 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:48.972993 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:48.973017 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:48.992982 (+ 19965us) log.cc:844] Serialized 26 byte log entry
0709 12:17:49.100062 (+107080us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:49.235649 (+135587us) log.cc:844] Serialized 14 byte log entry
0709 12:17:49.352779 (+117130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:49.352881 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:49.776086 (+423205us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:50.018371 (+242285us) raft_consensus.cc:1685] finished
0709 12:17:50.022398 (+  4027us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:50.022557 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36490,"prepare.queue_time_us":115858,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":1961,"raft.queue_time_us":170984,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":24564,"spinlock_wait_cycles":12020608,"thread_start_us":275134,"threads_started":3,"wal-append.queue_time_us":125544}
I20260709 12:17:49.930100 26070 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.020763 26060 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:50.051417 12386 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 24 to 31. New cstate: current_term: 31 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:50.036621 13253 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:17:50.060976 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:50.049751 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:50.114560 12493 raft_consensus.cc:606] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:50.022729 25814 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:50.076285 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:50.068782 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:50.035820 25572 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:50.047562 26079 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:50.088562 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:49.944224 25676 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.088936 12642 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:49.934420 25684 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.094209 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:50.094210 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:50.100449 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 231) took 93705 ms (client timeout 15000 ms). Trace:
I20260709 12:17:50.049369 25905 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:50.062507 26122 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:50.111531 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:50.095372 25493 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:50.122543 12386 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 26 to 36. New cstate: current_term: 36 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:50.092026 25112 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:50.138447 12563 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:50.101802 26124 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:50.047816 26055 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.058s
I20260709 12:17:50.090023 26111 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:50.117240 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:50.116645 25811 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:50.084123 25682 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:17:50.129215 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:50.175160 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:50.181011 12639 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:50.114324 26017 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.900s	user 0.009s	sys 0.018s
I20260709 12:17:50.132491 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:50.106688 25694 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:50.143211 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93606.1us
I20260709 12:17:50.128491 26122 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:50.145887 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:50.128612 25905 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:50.076442 26125 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.151398 25684 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:50.151218 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57139.1us
I20260709 12:17:50.231496 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:50.152168 12572 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:50.091002 26121 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:50.162483 12491 raft_consensus.cc:2368] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
W20260709 12:17:50.152886 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124330us
I20260709 12:17:50.105027 26114 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.035s
I20260709 12:17:50.173933 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:50.161844 12386 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 27 to 35, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 35 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:50.262054 12386 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 37. New cstate: current_term: 37 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:50.179736 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:50.065593 25876 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:50.156227 26129 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:17:50.162010 25112 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:50.383060 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:50.384050 12497 raft_consensus.cc:606] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:50.126329 25572 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.171825 26034 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:17:50.152417 26084 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.118221 26083 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.229524 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:50.156674 26098 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:50.153775 26131 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:50.153456 26088 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:50.154713 26070 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:50.233461 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 164773us
I20260709 12:17:50.227708 25694 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.153662 26095 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:50.153240 25676 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:50.251021 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5520) took 1015 ms. Trace:
I20260709 12:17:50.363011 26121 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:50.361971 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:50.230746 25905 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:50.366034 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5392) took 1494 ms (client timeout 1615 ms). Trace:
W20260709 12:17:50.234599 26025 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.558s	user 0.016s	sys 0.000s
I20260709 12:17:50.276602 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:50.448861 12572 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:50.206745 25991 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:50.141456 25994 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.086s
I20260709 12:17:50.370127 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:50.188905 26060 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:50.378145 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 197343us
I20260709 12:17:50.206946 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:50.385613 12386 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 38. New cstate: current_term: 38 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:50.397836 12646 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:17:50.192194 26017 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.157920 25493 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.457994 13257 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:17:50.227885 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89589.5us
I20260709 12:17:50.385018 25790 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:50.459124 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:50.377405 25901 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.044s
I20260709 12:17:50.387120 26107 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:50.422497 13254 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:50.384438 25684 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:50.447602 12491 rpcz_store.cc:269] 0709 12:17:48.804367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:48.804526 (+   159us) service_pool.cc:224] Handling call
0709 12:17:50.265997 (+1461471us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540012c5d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:50.298663 (+ 32666us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3094530304}
I20260709 12:17:50.420938 12564 rpcz_store.cc:276] 0709 12:17:49.235436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:49.235643 (+   207us) service_pool.cc:224] Handling call
0709 12:17:49.248891 (+ 13248us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:49.293838 (+ 44947us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:49.349283 (+ 55445us) log.cc:844] Serialized 14 byte log entry
0709 12:17:49.724747 (+375464us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:49.784152 (+ 59405us) log.cc:844] Serialized 26 byte log entry
0709 12:17:49.810714 (+ 26562us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:49.810761 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:49.810851 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:49.951468 (+140617us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b24001522b8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:17:50.206795 (+255327us) raft_consensus.cc:1685] finished
0709 12:17:50.250644 (+ 43849us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:50.250920 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6620,"prepare.queue_time_us":58184,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":59,"raft.queue_time_us":163540,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":4848,"spinlock_wait_cycles":31399808,"thread_start_us":350236,"threads_started":1,"wal-append.queue_time_us":499874}
I20260709 12:17:50.380528 26127 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:50.411881 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:50.410225 25894 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:50.420529 26084 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:17:50.491307 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79528us
W20260709 12:17:50.462185 26100 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:17:50.420135 12563 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:50.407152 26111 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.458901 25977 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.441150 26088 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:50.503710 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:50.518169 12563 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 18
I20260709 12:17:50.518627 12563 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.19
I20260709 12:17:50.518909 12563 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:50.522238 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80544.5us
I20260709 12:17:50.504736 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:50.503961 12570 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.17-43.17]
I20260709 12:17:50.364290 25963 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.21, Last appended by leader: 21, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:50.127184 12387 rpcz_store.cc:269] 0709 12:16:16.305976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:22.122713 (+5816737us) service_pool.cc:224] Handling call
0709 12:17:49.936466 (+87813753us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:17:46.109785 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:17:46.246589 (+136804us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:17:46.252636 (+  6047us) write_op.cc:435] Acquired schema lock
0709 12:17:46.256183 (+  3547us) tablet.cc:662] Decoding operations
0709 12:17:46.403454 (+147271us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:17:46.410646 (+  7192us) write_op.cc:641] Partition lock acquired for write op
0709 12:17:46.413135 (+  2489us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:17:46.462351 (+ 49216us) tablet.cc:701] Row locks acquired
0709 12:17:46.462370 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:17:46.462596 (+   226us) write_op.cc:270] Start()
0709 12:17:46.462733 (+   137us) write_op.cc:276] Timestamp: P: 1783599466462522 usec, L: 0
0709 12:17:46.462754 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:17:46.542125 (+ 79371us) log.cc:844] Serialized 3253 byte log entry
0709 12:17:46.950153 (+408028us) op_driver.cc:464] REPLICATION: finished
0709 12:17:46.951444 (+  1291us) write_op.cc:301] APPLY: starting
0709 12:17:46.951547 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 12:17:47.136050 (+184503us) tablet.cc:1370] finished BulkCheckPresence
0709 12:17:47.136298 (+   248us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:17:47.267680 (+131382us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:17:47.268245 (+   565us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 12:17:47.268355 (+   110us) write_op.cc:312] APPLY: finished
0709 12:17:47.269123 (+   768us) log.cc:844] Serialized 81 byte log entry
0709 12:17:47.325351 (+ 56228us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:47.387345 (+ 61994us) write_op.cc:454] Released schema lock
0709 12:17:47.387566 (+   221us) write_op.cc:341] FINISH: Updating metrics
0709 12:17:47.448872 (+ 61306us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:47.448907 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1395294,"reported_tablets":51,"spinlock_wait_cycles":22784,"child_traces":[["op",{"apply.queue_time_us":1006,"apply.run_cpu_time_us":21238,"apply.run_wall_time_us":487594,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37535,"num_ops":8,"prepare.queue_time_us":208127,"prepare.run_cpu_time_us":14885,"prepare.run_wall_time_us":631014,"raft.queue_time_us":104515,"raft.run_cpu_time_us":866,"raft.run_wall_time_us":2469,"replication_time_us":487261,"thread_start_us":366409,"threads_started":4,"wal-append.queue_time_us":174654,"wal-append.run_cpu_time_us":16746,"wal-append.run_wall_time_us":1548906}]]}
I20260709 12:17:50.514726 12386 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 40, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 40 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:50.486176 26107 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:50.499228 26070 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:50.491497 25991 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:50.496656 26050 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:17:50.493080 25894 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:50.476439 25790 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.501241 25684 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.550414 25991 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.551748 12570 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:50.554184 26025 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
W20260709 12:17:50.554471 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 323049us
I20260709 12:17:50.559324 12566 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:50.561951 12386 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 28 to 34, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 34 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:50.514708 26108 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.564476 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:17:50.530509 26062 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:50.512346 26084 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.539671 26107 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:50.416126 26131 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.465312 26095 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.579221 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:50.571172 25876 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.591217 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:50.594040 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48537.1us
I20260709 12:17:50.533491 26127 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:50.542152 26050 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:50.599990 12386 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 20 to 32, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 32 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:50.577147 26111 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:50.538115 26055 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.063s
I20260709 12:17:50.622253 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:50.452968 26124 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.633554 12639 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:50.590626 26025 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:50.637554 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:50.637763 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:50.597222 26062 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:50.649549 12566 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:17:50.647971 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:50.648973 26135 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
I20260709 12:17:50.623963 26127 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:50.649101 26145 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
W20260709 12:17:50.636121 26133 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.376s	user 0.001s	sys 0.001s
I20260709 12:17:50.622463 26050 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:50.591176 26065 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.046s
I20260709 12:17:50.651499 12570 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:50.656876 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:50.642868 12386 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 39 to 42, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 42 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:17:50.659272 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68205.2us
I20260709 12:17:50.668195 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:50.668624 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 439164us
I20260709 12:17:50.659188 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:50.616560 26083 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.657900 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:50.669947 13254 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
W20260709 12:17:50.656019 26137 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260709 12:17:50.644873 26070 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:50.650049 26117 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:17:50.644356 25682 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:17:50.666337 25590 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.741s	user 0.013s	sys 0.000s
I20260709 12:17:50.678668 12386 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 28 to 37, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 37 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:50.689530 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:50.690985 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:50.690702 12386 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 31. New cstate: current_term: 31 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:50.692749 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:50.676469 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:50.649772 26114 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:50.650470 26126 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:17:50.681206 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102072us
I20260709 12:17:50.738515 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:50.664826 26132 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.400s	user 0.000s	sys 0.002s
I20260709 12:17:50.684592 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:50.685288 13254 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: [41.26-41.26]
I20260709 12:17:50.685669 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:50.635824 26080 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.119s
W20260709 12:17:50.741753 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44634us
I20260709 12:17:50.688390 12570 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 23. (term mismatch)
I20260709 12:17:50.742550 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:50.692617 12386 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 32 to 40, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 40 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:50.658179 26127 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:50.742527 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49898.5us
I20260709 12:17:50.747871 12570 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 22
I20260709 12:17:50.750576 12570 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 42.23
I20260709 12:17:50.750986 12570 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:50.749203 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:50.753074 12568 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:50.649667 26128 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 21, Time since last communication: 0.220s
I20260709 12:17:50.736425 25876 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:50.755645 25994 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.003s
I20260709 12:17:50.756551 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:50.682098 26055 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.070s
I20260709 12:17:50.625375 25894 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.26, Last appended by leader: 26, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.689103 26068 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:50.613760 26121 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.696821 26150 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:50.743036 25676 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:50.619846 26060 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.29, Last appended by leader: 29, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.743986 26126 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:50.772426 12386 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 30 to 38, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 38 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:50.749372 25590 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:50.650429 26088 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.688886 26070 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:50.753597 26147 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:50.762465 26017 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:50.822388 26142 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:17:50.822958 26128 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:50.868507 26128 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:50.869108 26128 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:50.823570 26148 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:17:50.823684 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67233.7us
W20260709 12:17:50.824674 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167919us
I20260709 12:17:50.870507 12386 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 24 to 40, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:50.802779 26150 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:50.749331 26025 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.25, Last appended by leader: 25, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:50.829471 26139 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.331s	user 0.001s	sys 0.002s
W20260709 12:17:50.840130 25814 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.717s	user 0.012s	sys 0.001s
I20260709 12:17:50.814960 25694 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.842293 26084 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:50.843216 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 192320us
W20260709 12:17:50.817366 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78972.9us
I20260709 12:17:50.852725 26126 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:50.932341 12386 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 36, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:17:50.853997 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 169498us
I20260709 12:17:50.802148 25977 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:50.831173 25811 raft_consensus.cc:993] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:17:50.826329 25684 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.837553 26065 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:17:50.926406 26142 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.002s	sys 0.001s
I20260709 12:17:50.818555 26112 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:50.849432 25894 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:50.849393 25682 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:50.896828 26155 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:50.816768 26062 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:50.905313 26122 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.648s	user 0.001s	sys 0.010s
I20260709 12:17:50.950600 26025 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:50.909905 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5529) took 1193 ms. Trace:
I20260709 12:17:50.978034 12565 rpcz_store.cc:276] 0709 12:17:49.716622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:50.101801 (+385179us) service_pool.cc:224] Handling call
0709 12:17:50.117241 (+ 15440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:50.139363 (+ 22122us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:50.139402 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:50.142038 (+  2636us) log.cc:844] Serialized 26 byte log entry
0709 12:17:50.527022 (+384984us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:17:50.527067 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:50.527150 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:50.909609 (+382459us) raft_consensus.cc:1685] finished
0709 12:17:50.909673 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:50.909837 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14216,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":272436,"threads_started":1,"wal-append.queue_time_us":492480}
W20260709 12:17:50.850338 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107861us
I20260709 12:17:50.919050 25790 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.858793 26147 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:50.983965 26147 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:50.985651 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:50.972770 25977 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:50.989409 25977 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:50.966095 25682 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:50.991999 25682 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:50.940496 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:50.942962 26095 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:50.946717 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:50.803175 25676 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:50.998755 25676 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:50.998567 26050 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.000887 26050 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:50.951989 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:50.857132 25590 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:51.002206 26149 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:17:50.955842 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:50.891520 26150 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:50.963160 12386 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 29 to 31, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 31 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:17:50.930262 25814 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:50.957821 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:50.945779 26065 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:50.906706 26079 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.780s	user 0.006s	sys 0.007s
I20260709 12:17:50.975759 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:50.975841 26131 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:51.017442 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74932.1us
I20260709 12:17:50.972862 25684 raft_consensus.cc:1081] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:17:50.965420 25901 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.038718 25901 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:51.040401 25905 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.579s	user 0.007s	sys 0.004s
I20260709 12:17:51.041153 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.041893 26165 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:51.043092 26061 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:17:51.042883 26165 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.043474 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.044240 25901 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.044296 12571 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:17:51.045135 26152 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:51.045362 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.045742 26152 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:51.046511 12565 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:51.046877 26152 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.047446 26065 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.047930 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:51.010978 12643 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: [42.23-42.23]
I20260709 12:17:51.048717 26065 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:51.050119 26125 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:51.052310 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5530) took 1332 ms. Trace:
I20260709 12:17:51.040928 25905 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.018636 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:51.052767 12567 rpcz_store.cc:276] 0709 12:17:49.719994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:50.103767 (+383773us) service_pool.cc:224] Handling call
0709 12:17:50.104109 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:50.125683 (+ 21574us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:50.125725 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:50.180006 (+ 54281us) log.cc:844] Serialized 26 byte log entry
0709 12:17:50.603710 (+423704us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:50.603770 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:50.603874 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:51.052067 (+448193us) raft_consensus.cc:1685] finished
0709 12:17:51.052092 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:51.052244 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24883,"raft.queue_time_us":10761,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":3168,"thread_start_us":343794,"threads_started":1,"wal-append.queue_time_us":554525}
I20260709 12:17:50.957077 26127 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.061s
I20260709 12:17:50.962214 25189 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.053s
I20260709 12:17:51.029796 26162 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.034988 25590 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.037695 25812 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:51.041990 12566 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:50.977994 26122 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.048276 26152 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:50.979920 25112 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.576s	user 0.008s	sys 0.004s
I20260709 12:17:51.053179 12569 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:51.019079 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.097436 12645 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:51.053221 26083 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:51.008064 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.011704 12386 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 29 to 37, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 37 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:51.006350 26050 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.109928 26050 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.112179 25894 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:51.112406 26050 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.112761 25894 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:51.070884 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.082304 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:51.086899 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.087076 12637 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.22-44.22]
I20260709 12:17:51.087199 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.090756 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.090982 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.114884 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:51.096696 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78191.7us
W20260709 12:17:51.100144 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166969us
I20260709 12:17:50.941457 26080 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:17:51.093842 25812 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:51.055473 26165 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:50.966483 26112 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.107633 26134 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:51.112883 25676 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.115254 26088 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:17:51.123975 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.160312 12638 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:17:51.129078 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.134661 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.136380 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.137356 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.099483 26117 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:17:51.134763 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:51.145838 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.121733 12386 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 33 to 40. New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:51.166312 12639 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:51.166472 12386 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 24 to 32, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 32 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:51.158526 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:51.163252 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45923.3us
I20260709 12:17:51.161895 12646 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:51.168251 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:51.168903 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.171065 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.171339 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:51.142586 26101 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.290s	user 0.016s	sys 0.000s
W20260709 12:17:51.188702 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54062.9us
I20260709 12:17:51.202615 12571 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:51.182060 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.145942 26111 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.188887 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.094309 26162 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.142010 26168 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:51.222043 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.222446 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.213523 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.207223 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5257) took 1141 ms. Trace:
I20260709 12:17:51.106045 25112 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.223280 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.223656 12569 rpcz_store.cc:276] 0709 12:17:50.066098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:50.464361 (+398263us) service_pool.cc:224] Handling call
0709 12:17:50.504282 (+ 39921us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:51.096889 (+592607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:51.206836 (+109947us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:51.207133 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261829888}
I20260709 12:17:51.224319 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.225318 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.216037 12494 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.22-44.22]
I20260709 12:17:51.152642 26083 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:51.178346 12386 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 38 to 41. New cstate: current_term: 41 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:17:51.261122 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 320684us
I20260709 12:17:51.260008 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.187808 26088 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:51.173046 26113 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.273885 12386 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 30 to 36, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 36 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:51.165364 25590 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.264132 12495 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:51.265347 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.266101 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.302034 12490 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:51.266275 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.264853 12488 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:51.146607 25441 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.070s
I20260709 12:17:51.165212 26061 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:51.209232 26084 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.165026 26117 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:51.339196 12386 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 28 to 36, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:51.343212 12497 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:51.293982 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.221251 26111 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:51.307657 26164 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:17:51.307873 25812 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.309327 26072 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:51.308959 26084 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.311375 25112 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.312865 26062 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:51.312733 25682 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.146967 25894 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.315557 26162 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.316462 26083 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.320168 26177 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:17:51.321318 26112 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.323109 25676 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.324082 26095 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.414341 26095 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.416985 26095 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.329156 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:51.326086 26087 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.342057 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.418835 26087 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.419905 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.420569 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.421955 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.422245 25682 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:51.422408 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.209177 25694 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:51.423142 12497 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:51.423383 25694 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:51.424706 25694 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.338449 25590 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.427678 25590 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.350749 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.432499 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:51.433257 26107 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.814s	user 0.009s	sys 0.002s
I20260709 12:17:51.260046 26168 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:51.357002 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.442833 12490 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:51.358155 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64285.3us
I20260709 12:17:51.270288 26147 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.376886 26072 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:51.271395 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.336949 26117 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:51.372079 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 31.
I20260709 12:17:51.214989 26101 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.378221 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:51.395874 26163 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.368s	user 0.001s	sys 0.000s
I20260709 12:17:51.358564 25905 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.356719 25991 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:51.389881 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.396624 26172 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.366348 26134 raft_consensus.cc:2764] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election decision vote started in defunct term 44: won
W20260709 12:17:51.408599 26161 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.241s	user 0.000s	sys 0.000s
I20260709 12:17:51.397540 25572 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:51.420725 12494 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:51.266500 26070 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:51.431036 26173 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.214s	user 0.006s	sys 0.000s
I20260709 12:17:51.433403 12491 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:51.434248 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:51.496096 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 275083us
I20260709 12:17:51.499181 12571 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:51.433995 26088 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.29, Last appended by leader: 29, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.435055 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.435829 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.436611 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.354276 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.436368 26061 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.437906 26122 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.439432 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.440483 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.506477 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.443037 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:51.506974 12568 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:51.504371 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.507237 13257 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:51.444051 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:51.445173 12641 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:51.364347 12386 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 28 to 35. New cstate: current_term: 35 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:51.450904 26168 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:51.427809 26034 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.023s	user 0.012s	sys 0.001s
I20260709 12:17:51.329679 26113 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:51.484019 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.486447 12645 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:51.487082 12495 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:17:51.491050 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101252us
I20260709 12:17:51.523757 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.524567 12489 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.18-34.18]
I20260709 12:17:51.524667 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:51.496675 12496 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:51.502075 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:51.450913 25901 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.457118 12497 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:51.535952 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:51.536314 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.536631 12495 raft_consensus.cc:606] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:51.537038 12495 raft_consensus.cc:614] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:17:51.511915 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67975.9us
I20260709 12:17:51.513727 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:51.517280 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:51.496680 26176 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.179s	user 0.000s	sys 0.002s
I20260709 12:17:51.533179 13253 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:51.543949 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.545115 12386 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 22 to 32, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 32 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:17:51.546685 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.537379 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.543217 13254 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:17:51.549597 13254 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:51.554512 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.453648 26107 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.543345 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.557300 12491 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:51.557163 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.505462 26087 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.568102 12497 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:51.559234 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.561543 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.566404 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.566931 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.523562 25572 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:51.585368 13257 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:17:51.579337 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.563957 13254 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.580796 12490 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:51.530819 26034 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:51.532989 25684 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:51.595947 12645 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:17:51.596439 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.591179 12637 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47.
I20260709 12:17:51.593717 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.590732 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:51.580392 26128 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.711s	user 0.014s	sys 0.000s
I20260709 12:17:51.600850 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:51.601490 12572 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:51.526192 25814 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.059s
I20260709 12:17:51.506911 26172 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.612435 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.499339 25991 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.584262 26098 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:51.623541 12495 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:51.609172 12386 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 7 to 14, term changed from 21 to 27, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 27 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:17:51.625118 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:51.605229 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.614589 13261 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:17:51.614562 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:51.615650 12497 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:51.565771 25676 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:51.609582 12492 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:17:51.553660 25901 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.600795 25905 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:51.626624 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.635689 12638 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:51.602105 26111 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:51.665818 26025 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.663s	user 0.012s	sys 0.001s
I20260709 12:17:51.620666 13254 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:51.608565 26095 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.064s
I20260709 12:17:51.636822 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.605576 26087 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:17:51.610363 25590 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:17:51.674357 26150 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.654s	user 0.013s	sys 0.000s
I20260709 12:17:51.639341 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.623042 25572 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:17:51.682760 25572 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:51.683750 25572 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.684491 26101 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.684846 13254 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:17:51.608723 26177 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:51.651299 26084 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:51.651744 26126 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.695s	user 0.011s	sys 0.000s
I20260709 12:17:51.653327 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:51.653573 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.686471 13261 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:51.686388 26126 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.688170 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.688772 26126 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:51.655108 12386 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 26 to 32, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 32 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:17:51.689894 12640 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
W20260709 12:17:51.677538 26065 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.628s	user 0.004s	sys 0.009s
I20260709 12:17:51.691993 12565 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:17:51.692232 12640 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:51.692672 12565 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:51.693078 12565 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:51.657334 12461 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47.
I20260709 12:17:51.693845 12461 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Cancelling election due to peer responding with higher term
I20260709 12:17:51.645123 25991 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.665755 26134 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:51.631860 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.670921 12493 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:51.682251 26150 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.676693 25901 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.678290 12638 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:51.617383 26187 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.050s
I20260709 12:17:51.643990 26113 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.702805 12494 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:17:51.631943 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72845.9us
I20260709 12:17:51.729590 12639 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:51.739032 12494 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:51.618211 25977 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.093s
W20260709 12:17:51.742913 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89456.5us
I20260709 12:17:51.743372 26152 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:51.744009 26150 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:51.744784 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:51.750156 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:51.750712 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.751402 12567 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:51.752259 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:51.753093 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95959.8us
I20260709 12:17:51.754142 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:51.739770 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.755065 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:17:51.755268 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 57406us
I20260709 12:17:51.756160 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:51.757504 12564 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
W20260709 12:17:51.757714 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143241us
I20260709 12:17:51.759816 12494 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: [40.24-41.25]
I20260709 12:17:51.737334 12569 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:51.753299 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:51.761992 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:51.764343 12568 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:51.612931 26128 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.675637 26060 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:17:51.724931 26189 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:17:51.766937 12567 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:51.767285 12570 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:51.771483 12610 leader_election.cc:400] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:17:51.675328 26107 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:51.796569 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.773944 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.777041 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.797560 12571 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:51.797817 12568 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 23.
W20260709 12:17:51.782670 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.769903 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:51.608485 26072 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 43.17, Last appended by leader: 17, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.798738 12571 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:51.763248 25590 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:51.689987 26155 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.082s
I20260709 12:17:51.766621 26150 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:17:51.803478 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.765738 25901 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.803098 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.812824 12567 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:51.806823 12642 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:51.781899 26062 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:51.769322 25493 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:51.811173 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:51.813843 12567 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.17-43.17]
I20260709 12:17:51.832576 12569 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:51.833015 12638 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:17:51.820705 12568 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:51.788473 25977 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:51.832819 12565 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:51.838013 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.838617 12570 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:17:51.851325 12571 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:17:51.851743 12571 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:51.777809 26095 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.054s
I20260709 12:17:51.856413 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.838532 25901 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.853510 25977 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:51.863387 12565 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.820516 26107 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:17:51.864140 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93351.7us
I20260709 12:17:51.839397 26072 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:51.831596 25493 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:51.873693 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.803089 26170 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.064s
I20260709 12:17:51.888267 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.889153 26084 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:51.889581 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.890399 26084 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:51.890820 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.891582 26084 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:51.892004 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.892740 26084 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:51.874568 12571 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 44.22, Last appended by leader: 22, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.896399 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:51.900229 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:51.905066 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:51.903707 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:51.884457 26062 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:51.937960 26127 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:51.938763 26175 raft_consensus.cc:993] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:17:51.939340 26175 raft_consensus.cc:1081] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
W20260709 12:17:51.940758 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.875723 26107 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:51.950956 26134 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:51.952306 26134 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.899207 26147 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.898904 26025 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.057s
I20260709 12:17:51.956590 26025 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 45.28 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 27
I20260709 12:17:51.914038 26088 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:51.919736 26101 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:51.957921 26101 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:51.958285 26101 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:51.960134 26187 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:51.909198 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:51.961800 26187 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
W20260709 12:17:51.962069 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52939.9us
I20260709 12:17:51.962446 26187 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:51.963096 26088 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:51.963619 26088 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:51.963986 26088 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:51.894867 26079 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:51.894057 25493 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:51.920785 25977 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:51.923316 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:51.928702 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.930743 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.931653 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:51.932313 12490 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:17:51.936957 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.938575 26172 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:51.885792 26169 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:51.912473 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.916596 12571 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:51.982961 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:51.980999 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2626) took 1097 ms. Trace:
I20260709 12:17:51.996750 12643 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:51.957370 26180 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:17:51.965080 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.999779 12489 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:51.926681 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.000685 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.002319 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:51.998628 26117 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.001s
I20260709 12:17:51.996634 12564 rpcz_store.cc:276] 0709 12:17:50.883498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:50.890012 (+  6514us) service_pool.cc:224] Handling call
0709 12:17:50.921876 (+ 31864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:51.757991 (+836115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:51.980749 (+222758us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:51.980937 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1921203456}
I20260709 12:17:52.004912 12497 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:51.997962 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:51.942277 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:51.946594 26177 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:52.001446 12492 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:51.987004 26206 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:17:52.012761 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70560us
I20260709 12:17:52.017663 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:51.951116 26107 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:52.028137 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.030282 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.031450 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 31.
I20260709 12:17:51.972635 25590 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 46. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47.
I20260709 12:17:52.027690 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:51.973483 26165 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:52.101576 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.102932 12638 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:52.104415 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.025077 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:52.019109 26202 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:52.119751 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:52.018065 26206 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:52.138581 12564 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 33.
I20260709 12:17:51.991135 26147 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:52.011842 26025 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:52.126600 12568 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:17:51.999269 26180 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:51.978374 26134 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:52.012862 26150 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.134377 12491 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:52.150951 12497 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:52.153136 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:51.982476 26168 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.021811 26095 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.131s
I20260709 12:17:52.017715 26117 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.012652 26169 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:51.967633 26187 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 44.19, Last appended by leader: 19, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.280699 12640 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
W20260709 12:17:52.231400 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.278908 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.217777 12571 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 22. Preceding OpId from leader: term: 45 index: 22. (term mismatch)
I20260709 12:17:52.288388 12568 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:17:52.292104 12571 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:17:52.292748 12571 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 44.22
I20260709 12:17:52.293146 12571 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:52.218767 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:52.293953 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75312.3us
W20260709 12:17:52.207238 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 182287us
I20260709 12:17:52.298768 12492 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:52.125658 26127 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:52.307994 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.316488 12491 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:17:52.308841 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:52.306484 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 278851us
I20260709 12:17:52.311501 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:52.319015 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:52.319432 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:52.154049 12638 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.22-44.22]
I20260709 12:17:52.323741 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:52.324677 12568 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:17:52.209527 26149 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:17:52.123476 26165 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:52.345104 12640 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:52.346125 12641 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 42 because replica has last-logged OpId of term: 41 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 25.
I20260709 12:17:52.344841 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:52.347126 12496 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:52.202567 26177 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:52.355760 25189 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:52.228557 26128 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:52.206029 26205 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:52.316484 26120 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:52.353176 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:52.248644 26202 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:52.151145 26206 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:52.210755 26147 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.351666 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.354354 26187 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:52.353588 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:52.238196 26204 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:17:52.221823 26080 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:52.221720 26175 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:52.369822 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.374179 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:52.372475 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53163.4us
I20260709 12:17:52.387811 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:52.404255 12640 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:52.362951 12497 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:52.406472 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:52.383056 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:52.364465 26050 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:52.424592 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.424808 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:52.426399 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.377451 26120 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:52.489455 26120 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:52.431061 12492 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42.
W20260709 12:17:52.433373 26213 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:17:52.393934 26202 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.437785 26050 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.438658 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.441511 25812 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:52.444015 26197 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:17:52.446137 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 127349us
W20260709 12:17:52.490717 26083 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.009s	sys 0.054s
I20260709 12:17:52.427460 26062 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:17:52.447242 26080 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.449919 26169 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:52.452510 26175 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.453984 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.553067 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.454545 26168 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:52.368839 25790 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.086s
I20260709 12:17:52.456408 26147 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.460479 25905 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:52.462390 26111 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:52.464596 26201 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.606s	user 0.000s	sys 0.010s
I20260709 12:17:52.470701 25894 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:52.470886 26117 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.471614 26127 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:52.472213 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4030) took 1265 ms (client timeout 1649 ms). Trace:
I20260709 12:17:52.479988 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:52.591251 12490 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:52.534818 26034 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:52.352813 26180 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.320814 26025 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.485169 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5256) took 2419 ms. Trace:
I20260709 12:17:52.598251 12572 rpcz_store.cc:276] 0709 12:17:50.065818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:50.456032 (+390214us) service_pool.cc:224] Handling call
0709 12:17:50.456559 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:51.612169 (+1155610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:52.484930 (+872761us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:52.485111 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624498048}
I20260709 12:17:52.487103 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:52.600261 12494 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:52.600694 12494 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:52.488480 12645 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:52.419391 26205 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:52.321130 26150 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.498229 13254 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:52.502790 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:52.503042 12643 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.501s	user 0.010s	sys 0.002s
I20260709 12:17:52.607973 12643 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:52.503847 12489 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:17:52.505302 12496 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:17:52.609848 26205 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.610024 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.611166 26205 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.611676 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.504226 12568 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.610172 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.613183 12496 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:52.613979 12568 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.614122 26050 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:52.614179 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.615340 26180 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:52.615738 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:52.616488 12496 raft_consensus.cc:606] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:52.616828 12496 raft_consensus.cc:614] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:52.562789 25905 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:52.533972 26101 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.000s
I20260709 12:17:52.557859 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:52.557883 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:52.532226 25493 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.003s
I20260709 12:17:52.494412 26177 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:52.572505 12638 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 23.
I20260709 12:17:52.644253 26025 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:52.648138 26151 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:52.648710 26107 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.000s
I20260709 12:17:52.649261 26175 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.648775 26151 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.660439 26080 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:52.663899 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 257556us
W20260709 12:17:52.484436 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:52.593170 26211 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:17:52.593420 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:52.590240 12493 rpcz_store.cc:269] 0709 12:17:51.206927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.207053 (+   126us) service_pool.cc:224] Handling call
0709 12:17:52.472012 (+1264959us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:52.472142 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":342859648}
I20260709 12:17:52.601604 12494 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.563616 26197 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:52.508982 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:52.689554 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.534723 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:52.651576 26088 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.687s	user 0.012s	sys 0.000s
I20260709 12:17:52.619777 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:17:52.622803 13261 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:17:52.623276 12496 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:52.649222 26107 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:52.626226 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75415.1us
I20260709 12:17:52.628183 26101 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.607301 26034 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.629925 26217 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:52.596421 26165 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.668246 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:52.672923 12637 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.644s	user 0.012s	sys 0.000s
W20260709 12:17:52.673708 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.588863 26111 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.624372 25905 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:52.618786 12571 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:52.695178 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2072) took 1165 ms. Trace:
I20260709 12:17:52.625907 26202 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:52.685895 26070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.951s	user 0.001s	sys 0.045s
I20260709 12:17:52.697808 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:52.650470 25493 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.686188 26177 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.697374 26088 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.706029 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.734129 12496 rpcz_store.cc:276] 0709 12:17:51.529256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.619283 (+1090027us) service_pool.cc:224] Handling call
0709 12:17:52.619692 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:52.694811 (+ 75119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:52.695096 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:17:52.746807 12493 raft_consensus.cc:2368] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41.
I20260709 12:17:52.707255 26151 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.711175 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.754099 12497 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
W20260709 12:17:52.543409 26083 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.010s	sys 0.055s
I20260709 12:17:52.723282 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:52.727640 12491 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
W20260709 12:17:52.738438 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.692341 26025 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:52.730252 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62176.7us
I20260709 12:17:52.734364 12572 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:52.737710 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:52.739835 13261 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:52.743664 12490 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:17:52.757547 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:52.728559 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 180457us
I20260709 12:17:52.758841 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:52.757917 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:52.766036 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:52.766067 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:52.742616 26165 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:52.736061 26216 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:17:52.737033 26070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.001s	sys 0.046s
I20260709 12:17:52.779177 12492 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
W20260709 12:17:52.775691 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.746101 26117 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:52.780794 12496 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:17:52.783258 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.783936 12490 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:17:52.781455 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.795970 13257 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 46 index: 30. (index mismatch)
I20260709 12:17:52.798460 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:52.806185 26225 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:52.805807 26224 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:52.806902 26225 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.808089 26225 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.810225 26061 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:52.811268 26061 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.805864 13261 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:17:52.814965 13261 raft_consensus.cc:606] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:52.815390 13261 raft_consensus.cc:614] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:17:52.816553 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:52.810268 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.818560 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.819386 12493 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:52.770012 26025 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.843237 12496 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: [37.21-39.22]
I20260709 12:17:52.770020 26151 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:52.777871 25977 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:52.782419 26117 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:52.878702 26080 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:52.882305 26080 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:52.883561 26080 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.864477 26222 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:52.777593 26217 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:52.801716 26216 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.894253 13254 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:17:52.833211 26025 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:52.849426 12490 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [38.24-40.25]
W20260709 12:17:52.876336 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:52.899050 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:52.879894 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114697us
I20260709 12:17:52.786065 26127 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:52.855055 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56720.8us
I20260709 12:17:52.912209 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:52.861563 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45150.4us
I20260709 12:17:52.864498 26205 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:52.896823 12643 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:17:52.903726 12496 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.17-43.17]
I20260709 12:17:52.885922 26117 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:52.915849 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:52.888787 25977 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.919195 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.778364 26197 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.916582 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:52.929226 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.931541 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:52.924414 25991 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.008s
I20260709 12:17:52.922808 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:52.934603 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:52.941073 12496 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: [44.19-44.19]
I20260709 12:17:52.922783 26215 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:17:52.954523 26226 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:17:52.954974 26215 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.956084 26215 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.946529 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:52.959908 12490 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:17:52.935359 26230 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:52.937747 26187 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:52.923095 26025 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.027s
I20260709 12:17:52.962378 26101 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.962791 26025 consensus_queue.cc:1243] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 43.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:17:52.933485 25812 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:52.919579 26217 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.033s
I20260709 12:17:52.963587 26061 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.964246 26061 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:52.957919 26034 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:52.964612 26061 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:52.954536 12496 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:52.965801 26107 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.966692 25977 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.967480 12496 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:52.969151 26088 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.889901 26222 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:52.972268 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.973038 26222 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:52.974090 26111 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:52.977346 26216 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:52.978996 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:52.980190 25493 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:53.001354 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:52.961472 12490 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:53.039754 12496 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:52.914671 26127 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:52.938822 26205 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.080797 12645 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 21. Preceding OpId from leader: term: 45 index: 22. (index mismatch)
I20260709 12:17:52.982013 26202 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:17:53.061820 26233 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:17:53.082676 12645 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:53.056756 26187 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:53.065649 26228 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.226s	user 0.001s	sys 0.003s
I20260709 12:17:53.094122 12490 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:53.095181 13257 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
W20260709 12:17:53.097949 26098 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 45 current term: 45 is higher.
I20260709 12:17:53.056627 25901 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:53.057005 25812 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:53.080461 26229 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:53.102171 26206 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.685s	user 0.007s	sys 0.007s
I20260709 12:17:53.092394 26184 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:53.124518 12571 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:53.104111 26202 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:52.965576 26061 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:53.124354 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 201630us
W20260709 12:17:53.133314 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:53.134915 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 200414us
I20260709 12:17:53.137696 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:53.143167 12567 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
W20260709 12:17:53.144117 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 233326us
I20260709 12:17:53.154733 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.147121 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:53.140930 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:53.156577 12572 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:53.156781 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:53.125594 26098 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.158042 12568 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:53.125810 26123 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.165872 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.167976 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.169170 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.170228 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.162523 25694 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:17:53.128810 26204 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:53.174599 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.139583 26060 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.191875 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5136) took 1640 ms. Trace:
I20260709 12:17:53.192540 12644 rpcz_store.cc:276] 0709 12:17:51.551391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.789470 (+238079us) service_pool.cc:224] Handling call
0709 12:17:51.807057 (+ 17587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:51.821088 (+ 14031us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:51.821147 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:51.843025 (+ 21878us) log.cc:844] Serialized 26 byte log entry
0709 12:17:52.639445 (+796420us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b18000651a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:17:52.768161 (+128716us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:52.768215 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:52.768289 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:53.174956 (+406667us) raft_consensus.cc:1685] finished
0709 12:17:53.174991 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:53.191769 (+ 16778us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86944,"raft.queue_time_us":14350,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":22089,"spinlock_wait_cycles":12887808,"thread_start_us":128468,"threads_started":1,"wal-append.queue_time_us":1048134}
I20260709 12:17:53.119535 26229 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.154404 26080 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:53.121775 26206 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.203549 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:53.204502 12572 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:53.270314 12640 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
W20260709 12:17:53.272642 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 115957us
I20260709 12:17:53.273502 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:53.230084 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.222883 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5411) took 1249 ms. Trace:
I20260709 12:17:53.276258 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:53.157729 25901 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.107s
I20260709 12:17:53.140534 26187 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.278858 12563 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:53.175873 26180 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.093s
I20260709 12:17:53.147360 26230 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.118s
I20260709 12:17:53.160399 26107 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.103s
I20260709 12:17:53.277869 12644 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
W20260709 12:17:53.269351 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131787us
I20260709 12:17:53.299506 12644 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:53.300034 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:53.193970 26061 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:53.205367 26184 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:53.271550 26234 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.346s	user 0.000s	sys 0.002s
I20260709 12:17:53.275774 12493 rpcz_store.cc:276] 0709 12:17:51.974020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.820178 (+846158us) service_pool.cc:224] Handling call
0709 12:17:52.820670 (+   492us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:52.892504 (+ 71834us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:52.892558 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:52.904679 (+ 12121us) log.cc:844] Serialized 52 byte log entry
0709 12:17:53.090640 (+185961us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:53.197909 (+107269us) log.cc:844] Serialized 14 byte log entry
0709 12:17:53.221585 (+ 23676us) log.cc:844] Serialized 14 byte log entry
0709 12:17:53.222171 (+   586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:53.222303 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:53.222367 (+    64us) raft_consensus.cc:1685] finished
0709 12:17:53.222388 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:53.222803 (+   415us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71070,"raft.queue_time_us":72716,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":72028,"spinlock_wait_cycles":9728256,"thread_start_us":227465,"threads_started":2,"wal-append.queue_time_us":173502}
I20260709 12:17:53.189836 26123 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:53.302928 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:53.322129 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.195963 26060 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.183207 26098 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.196228 26204 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.267990 26080 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:53.309039 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.309561 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:53.309999 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4126) took 2128 ms (client timeout 1659 ms). Trace:
W20260709 12:17:53.333324 12639 rpcz_store.cc:269] 0709 12:17:51.176764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.213094 (+ 36330us) service_pool.cc:224] Handling call
0709 12:17:53.304863 (+2091769us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54000a87d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.304990 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":989835136}
I20260709 12:17:53.171736 26205 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.281361 26244 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
W20260709 12:17:53.298833 26238 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
W20260709 12:17:53.303855 26120 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.814s	user 0.010s	sys 0.004s
I20260709 12:17:53.319585 12644 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.30, Last appended by leader: 30, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.344095 12644 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:53.189404 25812 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.340034 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:53.312687 26107 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:17:53.378166 26060 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:53.379168 26204 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:53.380556 26200 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.647s	user 0.007s	sys 0.008s
I20260709 12:17:53.349188 26120 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.384028 26120 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:53.386719 12564 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:53.394434 12496 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:53.346642 26241 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:53.280432 26202 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.402005 12644 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 46 index: 30. (term mismatch)
I20260709 12:17:53.349398 26252 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.319233 26184 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.405153 12644 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:17:53.405530 12644 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 45.30
I20260709 12:17:53.386096 26127 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:53.414691 26248 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
W20260709 12:17:53.400296 26243 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:17:53.418728 26117 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.420161 26080 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.422613 26123 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:53.395267 26256 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:53.424036 26256 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.424958 26256 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:53.427902 26240 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:17:53.428812 26206 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:53.429394 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5141) took 1133 ms. Trace:
I20260709 12:17:53.402570 26241 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:53.429895 12638 rpcz_store.cc:276] 0709 12:17:52.295971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.438229 (+142258us) service_pool.cc:224] Handling call
0709 12:17:53.429147 (+990918us) spinlock_profiling.cc:243] Waited 50.1 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.429320 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115163904}
I20260709 12:17:53.430665 26241 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.25, Last appended by leader: 25, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.432096 26241 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:53.432629 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5145) took 1135 ms. Trace:
I20260709 12:17:53.437237 26229 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:53.411895 26165 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.003s
I20260709 12:17:53.400091 26250 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:53.401376 26060 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:53.443910 26060 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:53.412343 26186 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.741s	user 0.011s	sys 0.004s
I20260709 12:17:53.396673 26107 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.446203 26060 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:53.433051 12643 rpcz_store.cc:276] 0709 12:17:52.297529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.608743 (+311214us) service_pool.cc:224] Handling call
0709 12:17:52.609065 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:52.917568 (+308503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:53.432358 (+514790us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:53.432562 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":660562816}
I20260709 12:17:53.446732 26252 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.447906 26107 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.449120 26252 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:53.451591 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.410012 12644 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:53.453513 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:17:53.457217 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4118) took 2365 ms (client timeout 2086 ms). Trace:
W20260709 12:17:53.458313 12642 rpcz_store.cc:269] 0709 12:17:51.092164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.168977 (+ 76813us) service_pool.cc:224] Handling call
0709 12:17:53.456988 (+2288011us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.457136 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385372928}
W20260709 12:17:53.459949 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159991us
I20260709 12:17:53.464250 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:53.404264 26215 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.078s
I20260709 12:17:53.464460 26147 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:17:53.430716 26184 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:53.531952 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1958) took 1026 ms. Trace:
I20260709 12:17:53.510030 26060 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:53.525756 25901 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:53.538859 25905 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.796s	user 0.011s	sys 0.003s
I20260709 12:17:53.542029 13256 rpcz_store.cc:276] 0709 12:17:52.501324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.502388 (+  1064us) service_pool.cc:224] Handling call
0709 12:17:53.510465 (+1008077us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.527151 (+ 16686us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1250386688}
I20260709 12:17:53.461946 26222 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.073s
W20260709 12:17:53.554633 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93605.9us
I20260709 12:17:53.535022 26147 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.562127 12610 leader_election.cc:400] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42.
I20260709 12:17:53.559688 26222 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:53.567050 26222 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:53.568172 26222 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.547888 25901 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.572381 26121 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:17:53.582785 26121 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:53.583771 26121 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.584934 26121 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:53.585122 26197 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:53.586184 26197 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:53.586654 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:53.573495 26147 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.587766 26147 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:53.587545 26197 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.589154 26197 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:53.572971 26098 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:53.583808 25901 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:53.590883 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.602310 13253 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:17:53.609795 12494 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:17:53.544778 26216 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.007s	sys 0.007s
I20260709 12:17:53.617555 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:53.565886 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 259955us
W20260709 12:17:53.632051 12495 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.001s	sys 0.013s
W20260709 12:17:53.622143 26216 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.008s	sys 0.008s
I20260709 12:17:53.569602 26222 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:53.593485 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.674s (SENT) [suppressed 14 similar messages]
W20260709 12:17:53.617587 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.672824 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:17:53.678557 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116671us
I20260709 12:17:53.679697 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:53.681182 12572 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:17:53.681352 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:53.635396 12495 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.001s	sys 0.013s
W20260709 12:17:53.634588 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.682796 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.682886 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.683370 12494 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:53.683609 12570 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:17:53.684007 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4052) took 1801 ms (client timeout 2108 ms). Trace:
I20260709 12:17:53.684922 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.685278 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:53.684624 12494 rpcz_store.cc:269] 0709 12:17:51.882853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.486819 (+603966us) service_pool.cc:224] Handling call
0709 12:17:53.428512 (+941693us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b1800052e00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf582e646 00007ffbf576858e 00007ffbf57e9f9e 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c
0709 12:17:53.605478 (+176966us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5800984bd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c
0709 12:17:53.683938 (+ 78460us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2101493376}
I20260709 12:17:53.686096 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.686862 12566 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
W20260709 12:17:53.688614 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 343949us
I20260709 12:17:53.687589 12570 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:17:53.689483 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:53.683728 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:17:53.694883 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:53.696923 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.565985 25905 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.689013 26252 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:53.723079 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5413) took 1748 ms (client timeout 2095 ms). Trace:
W20260709 12:17:53.724217 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4128) took 2172 ms (client timeout 2085 ms). Trace:
I20260709 12:17:53.722855 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:53.731158 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51587.7us
W20260709 12:17:53.731745 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:53.732635 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.730713 26169 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.666416 26184 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.735280 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:53.727919 12492 rpcz_store.cc:269] 0709 12:17:51.974700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.508867 (+534167us) service_pool.cc:224] Handling call
0709 12:17:53.722830 (+1213963us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.723011 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516971392}
I20260709 12:17:53.731279 26250 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:17:53.736812 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:53.743506 12567 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
W20260709 12:17:53.739776 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.732447 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:53.728751 12637 rpcz_store.cc:269] 0709 12:17:51.551851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.612175 (+ 60324us) service_pool.cc:224] Handling call
0709 12:17:53.723994 (+2111819us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.724135 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":955090816}
I20260709 12:17:53.630048 26065 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.112s
I20260709 12:17:53.769405 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:53.770118 12637 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:53.727483 26216 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:53.666182 26120 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.099s
I20260709 12:17:53.743760 12644 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 41
W20260709 12:17:53.740576 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49557.6us
W20260709 12:17:53.739858 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50493.4us
I20260709 12:17:53.730578 26117 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:53.777735 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:53.781807 12643 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:17:53.781348 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:53.791224 26177 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:53.797149 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:53.776103 26165 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:53.801107 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.808445 13253 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:17:53.809320 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:53.810246 12572 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
W20260709 12:17:53.814181 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82053.1us
I20260709 12:17:53.736905 26252 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.813491 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1961) took 1047 ms. Trace:
I20260709 12:17:53.743614 26169 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.776433 26188 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:53.800251 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:53.804400 12644 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:17:53.816059 13254 rpcz_store.cc:276] 0709 12:17:52.765611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.765836 (+   225us) service_pool.cc:224] Handling call
0709 12:17:52.787231 (+ 21395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:52.894692 (+107461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:53.813239 (+918547us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:53.813429 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":245246208}
I20260709 12:17:53.824168 13253 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
W20260709 12:17:53.855803 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1948) took 1900 ms (client timeout 1674 ms). Trace:
I20260709 12:17:53.824527 12637 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:53.811762 25572 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:53.847726 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:53.870879 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70756.8us
I20260709 12:17:53.871137 12564 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:53.847748 12643 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:53.785565 26216 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.850325 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:53.871977 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:53.853093 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4122) took 2706 ms (client timeout 1639 ms). Trace:
I20260709 12:17:53.855719 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:53.745963 26184 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:53.875875 12646 rpcz_store.cc:269] 0709 12:17:51.146779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.168625 (+ 21846us) service_pool.cc:224] Handling call
0709 12:17:53.852856 (+2684231us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.853000 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4072151808}
W20260709 12:17:53.858330 12641 rpcz_store.cc:269] 0709 12:17:51.954926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.997679 (+ 42753us) service_pool.cc:224] Handling call
0709 12:17:53.819062 (+1821383us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:53.855715 (+ 36653us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":772131712}
I20260709 12:17:53.825717 25977 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:53.851464 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:53.869493 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:53.885689 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:53.879033 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:53.890376 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:53.877146 26244 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.892813 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4043) took 2550 ms. Trace:
W20260709 12:17:53.897106 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45755.7us
I20260709 12:17:53.806494 26123 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.072s
I20260709 12:17:53.898183 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:53.898525 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.875298 26065 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:53.904018 26060 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:53.904506 26244 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 44.19, Last appended by leader: 19, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.904948 26187 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:53.905563 26187 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:53.896836 12488 rpcz_store.cc:276] 0709 12:17:51.337568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.538744 (+201176us) service_pool.cc:224] Handling call
0709 12:17:51.567474 (+ 28730us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:51.609202 (+ 41728us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:51.609232 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:51.633783 (+ 24551us) log.cc:844] Serialized 52 byte log entry
0709 12:17:51.793982 (+160199us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:17:51.818327 (+ 24345us) log.cc:844] Serialized 14 byte log entry
0709 12:17:51.837199 (+ 18872us) log.cc:844] Serialized 14 byte log entry
0709 12:17:51.837406 (+   207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:51.837481 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:52.266340 (+428859us) spinlock_profiling.cc:243] Waited 9.16 ms on lock 0x7b1800393160. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:53.880331 (+1613991us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b1800393160. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:53.887575 (+  7244us) raft_consensus.cc:1685] finished
0709 12:17:53.887607 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:53.887826 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65426,"prepare.queue_time_us":26946,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":104,"raft.queue_time_us":79891,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":35425,"spinlock_wait_cycles":27245696,"thread_start_us":131816,"threads_started":1,"wal-append.queue_time_us":202302}
I20260709 12:17:53.906044 26187 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:53.906574 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.909564 12610 leader_election.cc:400] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41.
I20260709 12:17:53.877774 26117 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:53.897871 26197 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:53.909864 25812 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:53.910602 26117 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.910658 26197 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:17:53.914955 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59340.1us
I20260709 12:17:53.870358 25572 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:53.897673 26061 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:53.919283 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:53.909989 12610 leader_election.cc:403] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:53.883774 26177 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:53.819020 26165 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.871030 26253 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.048s
I20260709 12:17:53.926254 12564 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
W20260709 12:17:53.927834 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 239288us
I20260709 12:17:53.931862 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:53.931257 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:17:53.927165 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.939203 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:53.877894 26263 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.050s
I20260709 12:17:53.946365 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:53.947113 12646 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:53.923898 25572 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:53.923002 26197 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:53.944008 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:53.931031 26177 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:53.921142 26117 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:53.944527 12641 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 34. Current term is 35. Ops: []
I20260709 12:17:53.950870 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:53.937016 26147 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:53.924474 26061 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:53.955431 12638 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: [44.26-44.26]
I20260709 12:17:53.956353 12643 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:17:53.925374 26265 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:53.945770 12565 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:53.955297 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:53.957707 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:53.958698 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:53.976912 12646 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 32.
I20260709 12:17:53.927881 26269 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:53.959965 12637 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:53.949854 26123 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:53.936451 26065 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 40. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41.
I20260709 12:17:53.962963 12564 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:53.978471 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 384986us
I20260709 12:17:53.979082 13261 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:53.978879 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:53.991344 12571 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:17:53.992065 12494 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:53.992656 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:53.993240 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:53.993713 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:53.947197 26241 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:53.984964 12497 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:17:53.995051 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:53.936131 25812 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.027s
I20260709 12:17:53.995481 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:53.996016 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.000669 12493 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:17:53.961547 26177 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:53.998468 12489 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:17:54.003458 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.006984 12488 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:17:53.954128 26124 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.045s
I20260709 12:17:53.954443 26120 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.041s
I20260709 12:17:54.014174 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:53.980651 26061 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:53.980858 25814 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:54.011029 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:54.046129 25814 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:53.996112 25905 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.047225 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:54.049067 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:54.018342 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.018069 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:53.986047 26123 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:54.047295 25814 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.051213 26123 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:53.995573 26250 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.057s
I20260709 12:17:54.052496 26117 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:54.052986 26117 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:54.053248 26250 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:54.026119 26241 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:54.053977 26250 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:54.054872 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:54.030961 26262 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:17:54.055145 26250 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.056777 26250 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:54.008852 26269 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:54.059795 26269 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:53.990623 26258 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.040030 26252 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.017707 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.053496 26117 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:54.026239 26113 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:54.054657 26241 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:53.990007 26265 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.038890 26065 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:17:53.953439 26216 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:54.053268 26239 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.759s	user 0.001s	sys 0.003s
I20260709 12:17:54.084075 12564 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
W20260709 12:17:54.076671 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.087651 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:54.067152 26121 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:17:54.094149 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:54.122813 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:54.130144 26169 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.130028 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.003057 26230 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.085s
I20260709 12:17:54.061678 26147 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:17:54.149755 12564 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:17:54.122902 26065 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:54.096414 26121 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:54.080281 26258 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:54.191236 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.199751 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:54.207891 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.187489 25901 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:17:54.193513 26121 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:17:54.218195 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 204188us
W20260709 12:17:54.222118 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:54.222277 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 267087us
I20260709 12:17:54.224642 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:54.251911 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.267532 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:54.301815 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:54.306809 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.211084 25901 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.213963 26088 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.309930 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1688) took 2548 ms. Trace:
I20260709 12:17:54.220314 26279 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:54.318892 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 196168us
W20260709 12:17:54.312994 12495 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.002s	sys 0.010s
I20260709 12:17:54.225049 26165 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.321190 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:54.321477 12495 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.002s	sys 0.010s
I20260709 12:17:54.313211 13258 rpcz_store.cc:276] 0709 12:17:51.761080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:51.761564 (+   484us) service_pool.cc:224] Handling call
0709 12:17:51.767454 (+  5890us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:51.876465 (+109011us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:52.644282 (+767817us) spinlock_profiling.cc:243] Waited 8.13 ms on lock 0x7b1800221e00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:17:52.767797 (+123515us) log.cc:844] Serialized 14 byte log entry
0709 12:17:53.066076 (+298279us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:53.066782 (+   706us) log.cc:844] Serialized 26 byte log entry
0709 12:17:53.066982 (+   200us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:53.067029 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:54.022517 (+955488us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:54.022585 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:54.052614 (+ 30029us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:54.309358 (+256744us) raft_consensus.cc:1685] finished
0709 12:17:54.309397 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:54.309818 (+   421us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7938,"prepare.queue_time_us":1050400,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":15084,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":25372,"spinlock_wait_cycles":275481984,"thread_start_us":380502,"threads_started":2,"wal-append.queue_time_us":404944}
I20260709 12:17:54.330586 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5418) took 1668 ms. Trace:
I20260709 12:17:54.311861 25901 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:54.331009 12495 rpcz_store.cc:276] 0709 12:17:52.662770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.967003 (+304233us) service_pool.cc:224] Handling call
0709 12:17:52.967466 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:52.967937 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:52.967975 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:52.968651 (+   676us) log.cc:844] Serialized 26 byte log entry
0709 12:17:53.091145 (+122494us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:53.689254 (+598109us) log.cc:844] Serialized 14 byte log entry
0709 12:17:54.330102 (+640848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:54.330287 (+   185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:54.330363 (+    76us) raft_consensus.cc:1685] finished
0709 12:17:54.330374 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:54.330523 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45384,"prepare.queue_time_us":257491,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":57004,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":7905,"thread_start_us":1294845,"threads_started":3,"wal-append.queue_time_us":194582}
I20260709 12:17:54.337994 25493 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:17:54.338202 26095 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20260709 12:17:54.352079 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5423) took 1687 ms. Trace:
I20260709 12:17:54.356621 12496 rpcz_store.cc:276] 0709 12:17:52.664309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.061348 (+397039us) service_pool.cc:224] Handling call
0709 12:17:53.061725 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:53.395009 (+333284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:54.351780 (+956771us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:54.352000 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":745359616}
W20260709 12:17:54.359728 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 147887us
I20260709 12:17:54.318747 26088 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:54.348059 26222 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.669s	user 0.014s	sys 0.001s
W20260709 12:17:54.369027 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101643us
W20260709 12:17:54.386302 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.354470 26288 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:54.393699 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:54.399196 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 175759us
I20260709 12:17:54.408509 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:54.409246 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:54.409644 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:54.413443 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:54.414494 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:54.414882 12643 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:54.417750 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:54.418577 12643 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:54.443295 12639 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:17:54.444645 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:54.448594 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:54.449672 12641 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:17:54.451088 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:54.445490 12639 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:54.394976 26222 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.343904 26065 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.491153 12640 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:17:54.501272 12642 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:17:54.472671 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:54.393227 26265 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:54.408311 26288 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:54.502583 12637 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:54.503604 12642 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:17:54.492669 26165 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:54.515938 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:54.424536 25814 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.456624 26168 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:17:54.441880 26205 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.523952 12640 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: []
I20260709 12:17:54.446877 26098 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.467278 26122 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.506636 26288 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:54.488999 26273 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:54.574841 26098 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.565795 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:54.570441 26303 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:17:54.494894 26222 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.398675 26187 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.511013 26120 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.596865 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:54.527997 26168 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:17:54.617076 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.612203 13258 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 46 because replica has last-logged OpId of term: 44 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 25.
W20260709 12:17:54.656931 26247 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.206s	user 0.004s	sys 0.007s
I20260709 12:17:54.657461 25493 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:54.637351 26121 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.658696 26209 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:17:54.656576 26127 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:17:54.660732 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:54.661389 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:54.662129 12488 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:54.664233 12490 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:54.664386 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:54.664997 26255 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.209s	user 0.000s	sys 0.003s
I20260709 12:17:54.662640 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.666198 12491 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.18-34.18]
W20260709 12:17:54.668334 26197 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.000s
W20260709 12:17:54.669147 26283 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.335s	user 0.000s	sys 0.001s
I20260709 12:17:54.668879 26197 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:54.670137 26290 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:17:54.671209 26197 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.671808 26197 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.672236 26197 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.564229 26205 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.674816 26222 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.675357 26205 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.675856 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:54.676509 12497 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:54.677038 26187 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.327883 26279 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.631791 26303 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:54.678699 26303 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.644999 26298 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:54.680963 26088 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.679175 26279 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.682404 13261 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:17:54.675209 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:54.605480 26281 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.377s	user 0.002s	sys 0.000s
I20260709 12:17:54.683521 12496 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:54.586699 26258 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:54.691640 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 283222us
I20260709 12:17:54.694602 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:54.725169 12496 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
W20260709 12:17:54.726707 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 331738us
I20260709 12:17:54.731315 12488 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:17:54.733155 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:54.687136 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.538218 25814 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.741592 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:54.668334 26285 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.279s	user 0.000s	sys 0.000s
I20260709 12:17:54.660111 26121 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.587833 26122 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.629098 26120 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.756356 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1702) took 1855 ms. Trace:
I20260709 12:17:54.586171 26287 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.160s
W20260709 12:17:54.755465 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.763913 13257 rpcz_store.cc:276] 0709 12:17:52.886411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.900531 (+ 14120us) service_pool.cc:224] Handling call
0709 12:17:52.906231 (+  5700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:53.100602 (+194371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:53.435705 (+335103us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b54003bb8a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6a02 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:54.741341 (+1305636us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54005e1354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:54.741513 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437762816}
I20260709 12:17:54.767782 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:54.768385 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.768482 12488 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:54.736924 26113 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:54.757136 26283 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:17:54.770598 12491 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
W20260709 12:17:54.774303 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79812.2us
I20260709 12:17:54.775331 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:54.743615 26187 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.748843 26098 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:54.778672 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50908.7us
I20260709 12:17:54.783807 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:54.789896 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.799887 12610 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:17:54.800275 12610 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:17:54.806370 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2640) took 2067 ms (client timeout 2145 ms). Trace:
W20260709 12:17:54.806980 12569 rpcz_store.cc:269] 0709 12:17:52.738581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:52.738749 (+   168us) service_pool.cc:224] Handling call
0709 12:17:54.806105 (+2067356us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:54.806294 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2114105216}
I20260709 12:17:54.800521 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:54.778765 26121 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.780757 26258 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:54.812690 12497 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
W20260709 12:17:54.798810 26250 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.742s	user 0.012s	sys 0.004s
I20260709 12:17:54.810954 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:54.787921 26298 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:17:54.784101 26256 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.356s	user 0.014s	sys 0.009s
I20260709 12:17:54.834370 12566 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:54.835939 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:54.836647 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:54.784998 26127 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.837442 26265 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:17:54.837956 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:54.838519 25991 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:17:54.834493 26256 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.407s	user 0.014s	sys 0.009s
I20260709 12:17:54.834467 12571 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:54.839265 25991 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:54.831110 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:54.840855 25991 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.838817 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:54.824255 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:54.843519 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.012s	sys 0.051s
I20260709 12:17:54.843379 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:54.844552 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.012s	sys 0.051s
W20260709 12:17:54.845252 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 524130us
I20260709 12:17:54.845971 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:54.846638 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46268.9us
I20260709 12:17:54.845525 13258 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:17:54.811563 26197 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.812690 26289 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.857689 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:54.842645 25991 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:54.831874 26202 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.400s	user 0.002s	sys 0.016s
I20260709 12:17:54.778565 26113 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.818629 26258 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:54.813796 26304 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:17:54.860872 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:54.797581 26187 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.827843 26244 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.939514 26244 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.940719 26187 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.842852 26265 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:17:54.915264 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.941742 26265 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:54.942207 26265 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:54.915629 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:54.916618 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92458.5us
W20260709 12:17:54.917143 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111610us
W20260709 12:17:54.888100 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:54.945492 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:54.920110 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.023s	sys 0.008s
I20260709 12:17:54.923028 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:54.946774 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.023s	sys 0.008s
I20260709 12:17:54.947129 12493 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:54.924891 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:54.827486 25676 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:54.936712 12571 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:17:54.948581 12491 raft_consensus.cc:606] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:54.905145 26202 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.485s	user 0.002s	sys 0.017s
I20260709 12:17:54.948683 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:54.948751 25676 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.949690 12572 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:17:54.949950 26241 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:54.902352 26242 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.595s	user 0.005s	sys 0.000s
I20260709 12:17:54.950364 26241 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:54.950810 26241 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:54.951485 25676 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:54.947208 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 801433us
I20260709 12:17:54.954056 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:17:54.950829 12641 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:54.821812 26121 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:54.902943 26216 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:54.956534 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:17:54.916982 26197 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.862084 26297 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:17:54.860153 26222 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:54.861701 26298 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:54.922617 25905 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:54.960456 26117 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.872s	user 0.011s	sys 0.004s
I20260709 12:17:54.855821 26127 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.961191 26279 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:54.997884 26300 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:17:54.999321 26300 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:17:55.000613 26301 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:55.001585 26301 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:55.002074 26301 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:55.003258 26301 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.005259 26291 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.006003 26291 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.006780 26312 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:17:55.007596 26312 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:55.008153 26312 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:55.010052 26308 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:17:54.929925 26250 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.053s
I20260709 12:17:54.864146 26305 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.125s
I20260709 12:17:54.860307 26256 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.119s
I20260709 12:17:55.008621 26291 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:55.017120 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5295) took 1336 ms (client timeout 1671 ms). Trace:
I20260709 12:17:55.013954 26302 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:55.062265 26302 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:55.062872 26302 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:55.065554 26120 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.066720 26122 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.971062 26216 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:55.067914 25814 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.068270 26216 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:54.958266 26289 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.070057 26127 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.983390 26304 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:55.071532 26304 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:55.072180 26254 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.514s	user 0.000s	sys 0.012s
I20260709 12:17:55.072634 26289 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:55.077134 26177 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.016s	user 0.009s	sys 0.012s
I20260709 12:17:55.077646 26177 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.013612 26310 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.078837 26310 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.080097 26177 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:54.985517 26197 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.015749 26288 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.986128 26222 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:55.060256 25572 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.098s	user 0.017s	sys 0.000s
W20260709 12:17:55.074740 12563 rpcz_store.cc:269] 0709 12:17:53.680884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.681023 (+   139us) service_pool.cc:224] Handling call
0709 12:17:55.016929 (+1335906us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54001d6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:55.017057 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478424320}
I20260709 12:17:55.089026 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5150) took 2730 ms. Trace:
I20260709 12:17:55.089691 12645 rpcz_store.cc:276] 0709 12:17:52.359103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.093433 (+734330us) service_pool.cc:224] Handling call
0709 12:17:53.093774 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:53.115008 (+ 21234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:53.115064 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:53.115829 (+   765us) log.cc:844] Serialized 26 byte log entry
0709 12:17:53.350133 (+234304us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:53.350197 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:53.350302 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:55.012053 (+1661751us) spinlock_profiling.cc:243] Waited 3.98 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:55.076147 (+ 64094us) raft_consensus.cc:1685] finished
0709 12:17:55.076171 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:55.088926 (+ 12755us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8041,"raft.queue_time_us":13608,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":5532,"spinlock_wait_cycles":9150720,"thread_start_us":190174,"threads_started":1,"wal-append.queue_time_us":324507}
I20260709 12:17:55.014814 26305 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
W20260709 12:17:55.074914 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.094731 26222 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:55.098809 26061 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.056s	user 0.014s	sys 0.000s
I20260709 12:17:55.099375 26061 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.100701 13260 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:55.094862 26305 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:55.108278 26305 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.109443 26121 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.110168 26305 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:55.113322 26123 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.062s	user 0.008s	sys 0.012s
I20260709 12:17:55.103134 25572 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.110443 26197 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.119128 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5304) took 1209 ms. Trace:
I20260709 12:17:55.134853 12572 rpcz_store.cc:276] 0709 12:17:53.909413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.909590 (+   177us) service_pool.cc:224] Handling call
0709 12:17:53.910041 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:54.950174 (+1040133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.118670 (+168496us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.119029 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2390430592}
I20260709 12:17:55.137821 26079 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.138484 26079 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.134833 26310 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.142674 26197 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:55.142951 26264 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.409s	user 0.007s	sys 0.006s
I20260709 12:17:55.144012 12646 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:55.144793 25572 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:55.146240 26282 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.781s	user 0.005s	sys 0.000s
I20260709 12:17:55.141422 26079 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.147786 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5303) took 1247 ms. Trace:
I20260709 12:17:55.143384 26197 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.148784 26288 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:55.148430 12570 rpcz_store.cc:276] 0709 12:17:53.900422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.900751 (+   329us) service_pool.cc:224] Handling call
0709 12:17:53.901271 (+   520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:53.918307 (+ 17036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:53.918339 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:53.925183 (+  6844us) log.cc:844] Serialized 26 byte log entry
0709 12:17:54.151017 (+225834us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:17:54.151063 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:54.151132 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:55.121263 (+970131us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:55.147389 (+ 26126us) raft_consensus.cc:1685] finished
0709 12:17:55.147417 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:55.147707 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16496,"raft.queue_time_us":19825,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":1412,"spinlock_wait_cycles":5524864,"thread_start_us":196029,"threads_started":1,"wal-append.queue_time_us":379840}
W20260709 12:17:55.150437 26316 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:17:55.150844 26197 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.129098 12638 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:55.120153 26216 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.156649 26216 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.021242 26244 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:54.985311 26205 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.111s
I20260709 12:17:55.146077 26088 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:55.126293 26241 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.158694 26088 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.122331 26265 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.160578 26184 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:55.151986 26320 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.065s	user 0.001s	sys 0.001s
I20260709 12:17:55.163285 26265 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.163712 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1707) took 1160 ms. Trace:
I20260709 12:17:55.164448 26205 consensus_queue.cc:1243] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 44.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:17:55.164356 13261 rpcz_store.cc:276] 0709 12:17:54.002859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:54.003028 (+   169us) service_pool.cc:224] Handling call
0709 12:17:54.003345 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:54.685551 (+682206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.163323 (+477772us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.163623 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1561189760}
I20260709 12:17:55.157444 26168 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:54.981204 26187 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.105s
I20260709 12:17:55.181924 26187 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 34.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:55.019861 26117 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:55.177541 26245 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.763s	user 0.005s	sys 0.017s
I20260709 12:17:55.178617 26168 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:55.179324 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2645) took 1371 ms. Trace:
I20260709 12:17:55.198024 12565 rpcz_store.cc:276] 0709 12:17:53.807791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.808994 (+  1203us) service_pool.cc:224] Handling call
0709 12:17:53.856499 (+ 47505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:53.974443 (+117944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.178974 (+1204531us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.179245 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204709248}
W20260709 12:17:55.180828 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1693) took 1508 ms (client timeout 1947 ms). Trace:
W20260709 12:17:55.132827 25901 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.781s	user 0.014s	sys 0.000s
I20260709 12:17:55.200109 25901 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.018942 25905 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.022351 26279 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.205677 12492 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 42 index: 26. (index mismatch)
I20260709 12:17:55.206696 13262 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:55.184195 26252 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:17:55.016505 26314 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.120s
I20260709 12:17:55.135756 26123 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.193224 26061 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.182394 12495 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:17:55.199744 13256 rpcz_store.cc:269] 0709 12:17:53.672309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.672463 (+   154us) service_pool.cc:224] Handling call
0709 12:17:55.180567 (+1508104us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:55.180715 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640258944}
I20260709 12:17:55.218744 12495 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:55.191942 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:17:55.220275 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 359461us
W20260709 12:17:55.213692 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.208560 26117 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:55.217936 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 355541us
I20260709 12:17:55.016013 26258 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.208694 26113 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.160737 26241 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:55.259413 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:55.304363 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45074.5us
I20260709 12:17:55.247916 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:55.164170 26088 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.270771 12644 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:17:55.268239 12495 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:55.305809 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58011.8us
I20260709 12:17:55.217316 26197 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.216938 26252 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:55.354552 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.007s	sys 0.025s
W20260709 12:17:55.362066 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.007s	sys 0.025s
I20260709 12:17:55.211108 26222 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.366631 12641 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:17:55.368126 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.368623 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.369688 12639 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 43 index: 26. (index mismatch)
W20260709 12:17:55.320896 26303 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.006s	sys 0.015s
I20260709 12:17:55.370671 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:55.374862 12637 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:17:55.375932 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.376788 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:17:55.321965 26215 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:55.349911 26117 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.263506 26253 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.101s
I20260709 12:17:55.322466 25676 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:17:55.395762 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 221221us
W20260709 12:17:55.398967 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.367522 25814 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:55.271664 26265 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.108s
I20260709 12:17:55.367065 26124 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:55.381285 26303 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.007s	sys 0.015s
I20260709 12:17:55.394528 25901 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:55.409973 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.361755 26197 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.401475 26215 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:55.422142 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.430694 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:55.414582 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:55.403702 26314 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.418145 25676 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:55.401750 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 421048us
I20260709 12:17:55.393898 26312 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:55.444941 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 468535us
I20260709 12:17:55.445593 12497 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:17:55.449492 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.354714 26298 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.402493 26117 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.425907 25905 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.461534 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.451699 12496 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:17:55.462690 12639 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:17:55.433903 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:55.364868 26107 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:55.450409 12494 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:55.407763 25814 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.443116 26287 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:55.460220 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45755.5us
I20260709 12:17:55.466277 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.415717 26124 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.469717 12493 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:17:55.403306 26252 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.472942 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.443478 25901 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:55.474840 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.477458 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:55.478519 13262 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:55.479179 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.490703 12639 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:55.479806 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.492134 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.469098 26265 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:55.460289 12497 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:17:55.508533 26214 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:55.446871 26065 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:55.471570 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.007s	user 0.012s	sys 0.032s
W20260709 12:17:55.509352 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.045s	user 0.012s	sys 0.032s
I20260709 12:17:55.508970 26225 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:55.486073 26287 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.510636 26080 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.511248 26080 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.512362 26287 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.510659 26244 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.004s
I20260709 12:17:55.506999 26331 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.516242 26331 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.475888 26123 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:55.518065 26323 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:17:55.518229 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.519420 26331 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.519698 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.469612 26297 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:55.520876 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.521802 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.521498 26297 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.523190 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.524475 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.525065 12493 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:55.525871 26253 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:55.521279 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.482775 25905 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:55.527392 26124 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.527837 25905 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:55.528143 26252 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:17:55.528314 25905 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:55.529328 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:55.495877 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.535004 13262 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 38 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 21.
I20260709 12:17:55.534173 26287 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.27, Next index: 28, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:55.480914 12496 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:17:55.536463 25493 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:17:55.509459 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.538671 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.539877 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.540454 12497 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:55.543428 26225 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:55.537118 25493 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:55.472579 26107 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.497488 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.499240 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:55.569370 12643 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:55.503109 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:55.497115 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.481920 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.571406 13261 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
W20260709 12:17:55.510123 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.11647e+06us
I20260709 12:17:55.572497 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:55.574244 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:55.488497 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:55.503664 25901 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.513738 26325 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:55.446918 26314 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.565860 26060 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:17:55.502589 25991 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.003s
I20260709 12:17:55.465993 26268 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.295s
W20260709 12:17:55.557432 26317 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:17:55.576589 12494 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:55.580101 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.592411 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.593026 13262 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 33. Current term is 34. Ops: []
I20260709 12:17:55.503440 12639 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:55.601779 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1968) took 1647 ms. Trace:
I20260709 12:17:55.604377 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:55.605839 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44796.2us
I20260709 12:17:55.509042 26214 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:55.505718 26279 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.621896 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:55.558566 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.623690 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:17:55.623819 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.513530 26080 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:55.513837 26326 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:55.567070 26222 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:55.626876 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.627897 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.653164 12490 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:55.622296 13260 rpcz_store.cc:276] 0709 12:17:53.954684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.954866 (+   182us) service_pool.cc:224] Handling call
0709 12:17:53.970006 (+ 15140us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:55.101290 (+1131284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.601364 (+500074us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400537854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.601685 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2599613824}
I20260709 12:17:55.621941 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:55.630440 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.654443 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.656219 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:55.546658 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.579525 26225 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:55.642292 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.642766 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.638226 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:55.604492 26113 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:17:55.647100 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.775910 13260 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:55.776557 12639 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:55.774816 12491 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:17:55.776475 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 138339us
I20260709 12:17:55.778277 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.778738 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.779135 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:55.649528 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:55.589123 26330 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:55.645850 26095 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:17:55.558300 26065 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.620190 26325 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.709977 12641 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:55.791692 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.709245 12643 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:55.628895 25694 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.629031 26265 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:55.602834 26060 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:55.640039 26241 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:55.770991 26334 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.752633 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.550647 26215 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.621803 12493 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:17:55.552994 25676 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.724706 26204 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:17:55.725247 26279 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:55.628535 26150 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.108s
I20260709 12:17:55.559464 25493 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.770025 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.782323 12490 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:55.784502 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:55.786036 26117 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:55.787175 26168 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.586s	user 0.010s	sys 0.003s
I20260709 12:17:55.759368 26268 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:55.789669 12497 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 18. Preceding OpId from leader: term: 45 index: 20. (index mismatch)
I20260709 12:17:55.792748 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.792131 25814 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.795315 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.840924 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:55.842208 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:55.842972 12640 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:55.843634 26065 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:55.844062 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:55.791400 26305 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.25, Next index: 26, Last known committed idx: 23, Time since last communication: 0.053s
I20260709 12:17:55.845228 26305 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:55.845654 26305 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:55.846465 25493 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:55.762022 25991 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.035s
W20260709 12:17:55.847956 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.803637 26312 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.850504 26265 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.852466 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.803138 26328 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:55.807577 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:55.854177 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4138) took 1442 ms. Trace:
I20260709 12:17:55.854632 12638 rpcz_store.cc:276] 0709 12:17:54.411317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:54.411501 (+   184us) service_pool.cc:224] Handling call
0709 12:17:54.411931 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:55.153012 (+741081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.853906 (+700894us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.854110 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648323456}
I20260709 12:17:55.814208 12639 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:55.855887 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.807812 26279 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.856670 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.857569 12639 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:55.858546 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:55.858923 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5575) took 1892 ms. Trace:
I20260709 12:17:55.859280 12567 rpcz_store.cc:276] 0709 12:17:53.966498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.966686 (+   188us) service_pool.cc:224] Handling call
0709 12:17:53.967145 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:54.122577 (+155432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:54.122608 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:55.081534 (+958926us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:17:55.082062 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:17:55.454844 (+372782us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:55.454892 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.858603 (+403711us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.858657 (+    54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:55.858705 (+    48us) raft_consensus.cc:1685] finished
0709 12:17:55.858718 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:55.858854 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":483390208,"thread_start_us":372355,"threads_started":1,"wal-append.queue_time_us":517906}
I20260709 12:17:55.793730 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.862949 12493 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
W20260709 12:17:55.863114 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 917605us
I20260709 12:17:55.863778 26325 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.796890 26197 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.864701 26225 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.867655 26258 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:55.867934 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:55.869393 26209 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:55.841202 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:55.872495 26187 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:55.872964 26187 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:55.872689 26300 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:55.873524 26187 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:55.880038 13257 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 31. Current term is 32. Ops: [31.20-31.20]
W20260709 12:17:55.880514 26322 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.473s	user 0.000s	sys 0.003s
I20260709 12:17:55.881856 12495 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:55.884353 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4060) took 1221 ms. Trace:
I20260709 12:17:55.885365 26060 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.884788 12489 rpcz_store.cc:276] 0709 12:17:54.663251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:54.663484 (+   233us) service_pool.cc:224] Handling call
0709 12:17:54.667845 (+  4361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:17:54.689499 (+ 21654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:54.689551 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:17:54.690284 (+   733us) log.cc:844] Serialized 52 byte log entry
0709 12:17:54.938405 (+248121us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:55.404373 (+465968us) log.cc:844] Serialized 14 byte log entry
0709 12:17:55.404605 (+   232us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.404693 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:55.879866 (+475173us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b1800146e00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db
0709 12:17:55.884006 (+  4140us) raft_consensus.cc:1685] finished
0709 12:17:55.884112 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:55.884285 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62953,"prepare.queue_time_us":163393,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":76277,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":25851,"spinlock_wait_cycles":5897984,"thread_start_us":552580,"threads_started":2,"wal-append.queue_time_us":607723}
I20260709 12:17:55.887148 26060 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:55.887359 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1700) took 1054 ms. Trace:
I20260709 12:17:55.888141 26125 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:55.890295 26107 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.800220 26113 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:17:55.891099 26113 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:55.891495 26113 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:55.799721 25694 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.895901 25694 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.803385 26334 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.899158 26334 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.888113 13257 rpcz_store.cc:276] 0709 12:17:54.832828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:54.832994 (+   166us) service_pool.cc:224] Handling call
0709 12:17:54.833437 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:55.880521 (+1047084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.887075 (+  6554us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004b73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.887298 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2406233344}
I20260709 12:17:55.766260 26214 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.26, Last appended by leader: 26, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:55.901460 26214 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:55.878576 12642 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: [38.20-38.20]
I20260709 12:17:55.916565 13256 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:55.822424 12491 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:17:55.829522 26314 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.830193 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1959) took 1884 ms. Trace:
I20260709 12:17:55.830585 26327 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:55.831422 25901 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:55.799124 26241 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:55.832479 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.835556 12496 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:55.836227 26301 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:55.838284 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2083) took 2013 ms. Trace:
I20260709 12:17:55.937800 12492 rpcz_store.cc:276] 0709 12:17:53.824809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.824996 (+   187us) service_pool.cc:224] Handling call
0709 12:17:55.083560 (+1258564us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:17:55.083832 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:55.206155 (+122323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.837986 (+631831us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.838209 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281919488}
I20260709 12:17:55.806478 12490 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [33.19-36.21]
I20260709 12:17:55.916723 12494 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:55.918637 12644 rpcz_store.cc:276] 0709 12:17:53.951643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:53.952093 (+   450us) service_pool.cc:224] Handling call
0709 12:17:53.953401 (+  1308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:55.310714 (+1357313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.829872 (+519158us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.830107 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3029162752}
I20260709 12:17:55.947486 13256 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:17:55.864743 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:55.953208 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88559.5us
I20260709 12:17:55.888800 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:55.932766 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.935801 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.927562 26327 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:55.924124 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:55.949590 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.955752 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.956907 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:55.957221 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:55.858556 26279 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:55.964358 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.967700 12495 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:17:55.975911 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:55.976999 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.962908 12646 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:55.968102 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:55.978562 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:17:55.962096 12641 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:17:55.971839 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47826.1us
W20260709 12:17:55.961844 26302 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.898s	user 0.014s	sys 0.001s
I20260709 12:17:55.956183 26065 raft_consensus.cc:2764] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
W20260709 12:17:55.974417 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85703.3us
I20260709 12:17:55.976197 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4055) took 1316 ms. Trace:
I20260709 12:17:55.968659 26080 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:55.980340 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:55.997519 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:55.963310 26315 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:17:55.984923 12642 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: [46.30-46.30]
I20260709 12:17:55.928479 25905 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:55.965860 26269 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.022s	sys 0.045s
I20260709 12:17:55.937508 26209 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.038s
I20260709 12:17:56.000346 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:56.016448 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.017155 13256 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:56.018079 13256 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 39.
I20260709 12:17:55.995008 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:17:55.948764 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 307267us
I20260709 12:17:56.017463 12567 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:17:56.014463 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:55.993748 12495 rpcz_store.cc:276] 0709 12:17:54.659460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:54.660409 (+   949us) service_pool.cc:224] Handling call
0709 12:17:55.182210 (+521801us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400296920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6e18 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:55.881104 (+698894us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b5800a031d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f727c 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c
0709 12:17:55.976130 (+ 95026us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1630673024}
I20260709 12:17:56.021008 12611 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 39.
I20260709 12:17:56.021493 12611 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:17:55.974184 26216 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:55.999962 26080 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:56.026714 26080 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:56.026902 26124 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:56.027711 26124 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 39
I20260709 12:17:56.015071 26165 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.015s
I20260709 12:17:56.027631 25905 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:55.962261 26334 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.041s
I20260709 12:17:56.038221 26298 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:17:56.041793 26124 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 37. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 39.
W20260709 12:17:56.044349 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:17:56.048692 26286 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.049443 26286 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.052093 26286 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.063794 26080 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:55.986044 26168 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:17:56.070215 26168 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:56.071439 26168 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 43.17, Last appended by leader: 17, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.073844 26327 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:56.079799 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.083190 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:56.021836 26269 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.022s	sys 0.045s
W20260709 12:17:56.015596 26309 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 34 current term: 34 is higher.
W20260709 12:17:56.003855 26120 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.008s	sys 0.026s
I20260709 12:17:56.000283 26284 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:17:55.996119 26303 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:56.014451 26314 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:56.003669 26315 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:56.077848 26123 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.145160 12564 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:55.996584 26302 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:56.143913 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60878.2us
W20260709 12:17:56.142653 26120 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.008s	sys 0.026s
I20260709 12:17:56.168910 26123 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.181160 26315 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 33.25, Last appended by leader: 25, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:56.086338 26216 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:56.198263 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.207068 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.209544 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.210310 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:56.173516 26298 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:56.210943 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.211234 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.216449 12645 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:17:56.216737 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:17:56.208043 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:17:56.217768 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.225021 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.226786 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.232254 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.237398 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.219362 25905 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.177460 26314 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:56.177093 26303 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:56.220363 26298 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:56.263206 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:56.268939 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 489860us
W20260709 12:17:56.269851 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.020s	sys 0.057s
W20260709 12:17:56.271137 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.020s	sys 0.057s
W20260709 12:17:56.272805 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 647754us
I20260709 12:17:56.264391 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.258939 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.274938 12570 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:56.275197 12565 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:17:56.277958 13261 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:17:56.278469 13256 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 25
I20260709 12:17:56.278852 13256 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 44.26
W20260709 12:17:56.264961 26252 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.002s
I20260709 12:17:56.238024 26328 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
W20260709 12:17:56.278898 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.036s	sys 0.032s
I20260709 12:17:56.279340 13256 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:56.290638 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.036s	sys 0.032s
W20260709 12:17:56.303447 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 681617us
I20260709 12:17:56.326673 13261 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:56.280197 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:56.329100 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:17:56.327196 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:56.276926 26298 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:56.342319 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.344564 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:56.348980 13253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.021s	sys 0.064s
I20260709 12:17:56.350287 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:56.352347 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.350358 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.353331 12567 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 36 because replica has last-logged OpId of term: 34 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
W20260709 12:17:56.302917 26122 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.023s	sys 0.038s
I20260709 12:17:56.360653 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.361675 12569 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:56.280712 26286 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:56.367998 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:56.358445 13253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.021s	sys 0.064s
I20260709 12:17:56.266429 26187 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.266825 25905 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:56.498042 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:56.368925 12638 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.020s	sys 0.019s
I20260709 12:17:56.456765 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5588) took 2269 ms. Trace:
W20260709 12:17:56.448136 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.525326 12564 rpcz_store.cc:276] 0709 12:17:54.188057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:54.188405 (+   348us) service_pool.cc:224] Handling call
0709 12:17:54.202548 (+ 14143us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:56.169488 (+1966940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:56.456407 (+286919us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:56.456667 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4466853632}
I20260709 12:17:56.497120 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:56.357766 26297 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.028s	sys 0.023s
W20260709 12:17:56.428696 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120980us
W20260709 12:17:56.531581 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 251448us
W20260709 12:17:56.529366 12638 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.030s	sys 0.038s
W20260709 12:17:56.511988 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143218us
I20260709 12:17:56.494809 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.494300 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:56.361037 26122 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.024s	sys 0.039s
W20260709 12:17:56.521992 26301 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.566s	user 0.008s	sys 0.004s
W20260709 12:17:56.465834 26336 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:17:56.457746 26205 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 21, Time since last communication: 0.095s
I20260709 12:17:56.350839 26080 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:56.364193 26241 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.553010 26309 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.632705 26309 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.558261 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4159) took 1151 ms. Trace:
I20260709 12:17:56.559062 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.634785 12564 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 43 because replica has last-logged OpId of term: 42 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 25.
I20260709 12:17:56.634037 12637 rpcz_store.cc:276] 0709 12:17:55.454139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.464885 (+ 10746us) service_pool.cc:224] Handling call
0709 12:17:55.486886 (+ 22001us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:55.495228 (+  8342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:55.495265 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:55.526499 (+ 31234us) log.cc:844] Serialized 26 byte log entry
0709 12:17:55.970833 (+444334us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:55.970877 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:55.970948 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:56.558008 (+587060us) raft_consensus.cc:1685] finished
0709 12:17:56.558032 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:56.558188 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56313,"raft.queue_time_us":17643,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":113,"thread_start_us":443878,"threads_started":1,"wal-append.queue_time_us":840509}
I20260709 12:17:56.555758 26302 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:56.530360 26297 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.963s	user 0.030s	sys 0.029s
I20260709 12:17:56.565595 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:56.567030 26305 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:56.571620 26348 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.108s	user 0.003s	sys 0.000s
I20260709 12:17:56.574749 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.345543 26263 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:17:56.539423 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.535810 26286 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.579283 26187 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:56.457984 26330 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:17:56.580328 12641 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.002s	sys 0.010s
W20260709 12:17:56.643462 12641 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.002s	sys 0.010s
W20260709 12:17:56.580776 25676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.007s	sys 0.010s
I20260709 12:17:56.581216 25905 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.581266 26327 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:56.646039 26305 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:56.582144 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.584031 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.585211 13262 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: []
I20260709 12:17:56.585937 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:56.654341 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70436.6us
I20260709 12:17:56.655396 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.585907 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:56.446696 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:56.659149 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.660455 12570 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 43 because replica has last-logged OpId of term: 42 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 25.
W20260709 12:17:56.586975 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 145879us
I20260709 12:17:56.665930 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: [36.24-38.25]
I20260709 12:17:56.587575 26214 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:17:56.588392 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56964.8us
I20260709 12:17:56.554754 26123 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:56.641742 26241 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:56.590397 26122 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
W20260709 12:17:56.590651 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.591761 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.598107 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.686215 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:56.581300 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.010s	sys 0.002s
I20260709 12:17:56.601766 26315 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:17:56.612401 12566 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:17:56.613507 26314 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:56.614431 26303 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.615105 26216 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.620743 26298 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.621580 26095 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.623086 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:56.535291 26328 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:56.628863 26113 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.737s	user 0.011s	sys 0.000s
I20260709 12:17:56.435472 26287 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
W20260709 12:17:56.636518 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71193.4us
W20260709 12:17:56.638626 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63969.7us
W20260709 12:17:56.696439 25493 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.848s	user 0.012s	sys 0.002s
I20260709 12:17:56.706228 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:56.707453 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.642899 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:56.633980 26168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.011s	sys 0.044s
I20260709 12:17:56.661540 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.708470 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:56.710831 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.712251 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.642024 26279 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:17:56.645848 25676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.007s	sys 0.010s
I20260709 12:17:56.474949 26120 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:56.682504 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:56.730394 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48021.9us
I20260709 12:17:56.730952 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.731351 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.732159 26309 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.732339 26123 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:56.732852 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:56.687565 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.010s	sys 0.002s
W20260709 12:17:56.734088 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 739136us
I20260709 12:17:56.684957 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:56.688642 13259 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:56.676435 25901 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:17:56.700133 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:56.736016 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51126.2us
I20260709 12:17:56.644687 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:56.694885 26258 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.826s	user 0.009s	sys 0.007s
I20260709 12:17:56.687325 26122 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:56.644061 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.737017 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:56.737836 12494 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:17:56.739494 12643 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:17:56.723210 26352 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:56.715523 26298 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:56.709471 26168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.011s	sys 0.044s
I20260709 12:17:56.706930 26328 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:56.704816 26095 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:17:56.719964 26286 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.720956 26314 raft_consensus.cc:993] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:17:56.693122 26341 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.021s
I20260709 12:17:56.707508 26113 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:56.722527 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:56.707888 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:56.693800 26265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.057s	sys 0.041s
I20260709 12:17:56.702898 26303 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:17:56.778597 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70770.5us
I20260709 12:17:56.779112 26303 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:17:56.779371 26113 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:56.679141 25905 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.019s
I20260709 12:17:56.659149 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:56.780445 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:56.781237 12569 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:56.779199 26095 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:56.735153 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:56.784240 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49163.1us
W20260709 12:17:56.785773 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.786115 26168 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:17:56.786619 26168 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:56.786962 26168 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:56.778596 26265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.058s	sys 0.041s
I20260709 12:17:56.743449 26098 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:56.704387 26287 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:56.790951 26287 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.792162 26287 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:56.792847 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51397.9us
I20260709 12:17:56.724294 26279 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:56.793911 26279 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:56.749181 12566 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:17:56.742370 13257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.010s	sys 0.019s
I20260709 12:17:56.759311 12495 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:56.760602 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:56.758747 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5176) took 1194 ms. Trace:
I20260709 12:17:56.800400 12490 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:56.748971 26353 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.023s
I20260709 12:17:56.765699 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:56.729430 26120 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:56.745018 26258 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:56.800103 12643 rpcz_store.cc:276] 0709 12:17:55.564178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.793422 (+229244us) service_pool.cc:224] Handling call
0709 12:17:56.758497 (+965075us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:56.758681 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2018158848}
I20260709 12:17:56.803617 26120 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.805097 26120 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:17:56.805306 13257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.010s	sys 0.019s
I20260709 12:17:56.744431 26352 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:56.776058 26328 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:56.778460 12497 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:56.739574 26302 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:56.789681 12572 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:17:56.805943 26352 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:56.757764 26204 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.052s
W20260709 12:17:56.805611 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.818147 25901 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:17:56.818846 25901 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:56.797328 26168 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:56.819800 25901 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:56.820283 12572 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
W20260709 12:17:56.822054 13256 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.014s	sys 0.029s
W20260709 12:17:56.822434 13256 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.014s	sys 0.029s
I20260709 12:17:56.822759 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:56.757284 26215 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.013s	sys 0.025s
I20260709 12:17:56.825074 12572 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:17:56.837824 12572 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
W20260709 12:17:56.844374 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.851979 12567 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
W20260709 12:17:56.877502 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155057us
I20260709 12:17:56.878751 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:56.881345 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:56.882863 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:56.886219 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:56.891201 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68529.4us
W20260709 12:17:56.841640 26215 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.013s	sys 0.025s
I20260709 12:17:56.892099 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.833050 26314 raft_consensus.cc:1081] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:17:56.915365 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:56.900036 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:56.905400 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.916360 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:17:56.908810 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.918763 12490 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 20. (term mismatch)
I20260709 12:17:56.919302 12490 pending_rounds.cc:85] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:17:56.919680 12490 pending_rounds.cc:107] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:17:56.920037 12490 raft_consensus.cc:2889] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:56.921917 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:56.927997 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:56.928575 12643 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:17:56.929924 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:56.931661 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:56.933073 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:56.935053 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:56.935601 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.928735 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:56.935631 12492 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:17:56.936494 12642 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:56.936784 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:56.935055 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:56.933007 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:56.941424 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.015s	sys 0.038s
W20260709 12:17:56.942049 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.015s	sys 0.038s
I20260709 12:17:56.930287 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:56.880370 26098 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:56.946983 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:56.890807 26113 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:56.947456 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:56.935662 26263 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:56.927568 26356 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:56.903280 26204 consensus_queue.cc:1243] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 41.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:17:56.934763 26286 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.934516 26187 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:56.906131 26258 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:56.940706 26284 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:56.953704 12489 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:56.953955 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:56.974431 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.949s	user 0.023s	sys 0.014s
W20260709 12:17:56.980826 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.955s	user 0.024s	sys 0.014s
I20260709 12:17:56.905395 26364 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.947525 26252 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:56.951674 26301 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:17:56.980791 12637 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:56.977984 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:56.959489 26113 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:56.964974 26309 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:56.999630 12644 raft_consensus.cc:2376] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40. Re-sending same reply.
W20260709 12:17:57.004105 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:56.971611 26080 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:17:56.981606 26225 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:56.974866 26168 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:56.983172 26263 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:56.960126 26123 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.046s
I20260709 12:17:57.001837 26120 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.003752 26113 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:17:57.035389 26362 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 12:17:57.037524 26359 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:17:57.037860 26098 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.044018 26122 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.009294 26353 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:56.981351 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 960474us
I20260709 12:17:56.983762 26286 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.004690 26252 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.022465 26080 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:57.000064 26364 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.094848 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1978) took 1179 ms. Trace:
I20260709 12:17:57.095561 13257 rpcz_store.cc:276] 0709 12:17:55.915616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.915847 (+   231us) service_pool.cc:224] Handling call
0709 12:17:55.916351 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:55.916877 (+   526us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:55.916920 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:55.917526 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 12:17:56.806061 (+888535us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:56.806129 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:56.806234 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.094521 (+288287us) raft_consensus.cc:1685] finished
0709 12:17:57.094555 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.094760 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":11597,"thread_start_us":839385,"threads_started":1,"wal-append.queue_time_us":1104702}
I20260709 12:17:57.020387 26225 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.100410 26225 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:57.101853 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.018s	sys 0.026s
W20260709 12:17:57.103737 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.019s	sys 0.027s
I20260709 12:17:57.110518 12489 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:17:57.119428 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.128129 12572 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:57.128823 12572 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:57.136790 12563 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 19. (term mismatch)
I20260709 12:17:57.137583 12563 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 18
I20260709 12:17:57.049350 26120 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.139197 12563 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 35.19
I20260709 12:17:57.143671 12563 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:57.141165 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:17:57.140722 12638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.013s	sys 0.033s
W20260709 12:17:57.146996 12638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.014s	sys 0.034s
W20260709 12:17:57.147562 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.148653 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5167) took 1587 ms. Trace:
I20260709 12:17:57.069448 26353 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.111719 26330 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:17:57.150923 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:57.115271 26187 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.116837 26168 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.071434 26297 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.067s
I20260709 12:17:57.149261 12638 rpcz_store.cc:276] 0709 12:17:55.561540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.856907 (+295367us) service_pool.cc:224] Handling call
0709 12:17:55.862321 (+  5414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:55.862566 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:56.533205 (+670639us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.148209 (+615004us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:57.148283 (+    74us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:57.148350 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.148433 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.148580 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":970073,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":8751,"raft.queue_time_us":111,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":2482,"thread_start_us":1284186,"threads_started":2}
W20260709 12:17:57.162712 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.168967 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:57.102049 26155 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:17:57.169487 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:57.177172 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:57.178272 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:17:57.178896 12637 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
W20260709 12:17:57.174842 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 394388us
I20260709 12:17:57.179651 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:57.130110 12572 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:57.159372 25694 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.012s	sys 0.018s
I20260709 12:17:57.184262 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.182418 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:57.195554 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:57.174793 26309 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:57.197221 26369 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:17:57.197059 26309 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:57.199601 26367 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:17:57.199936 26309 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:57.200853 26360 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:17:57.165266 26187 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.204149 12640 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.26-41.26]
I20260709 12:17:57.205569 26364 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.163538 26284 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.206492 26284 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.207495 26252 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.146426 26080 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.208973 26284 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.163816 26305 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.210124 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5168) took 1648 ms. Trace:
I20260709 12:17:57.209976 26305 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.210556 12639 rpcz_store.cc:276] 0709 12:17:55.561860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.858588 (+296728us) service_pool.cc:224] Handling call
0709 12:17:55.858962 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:55.859823 (+   861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:55.859850 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:55.860423 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:17:56.343198 (+482775us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:17:56.343292 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:56.343492 (+   200us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.209845 (+866353us) raft_consensus.cc:1685] finished
0709 12:17:57.209874 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.210048 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":133,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":122,"thread_start_us":481907,"threads_started":1,"wal-append.queue_time_us":1066196}
I20260709 12:17:57.212337 26305 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.170644 26168 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.218176 12643 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:17:57.175524 26123 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:57.219095 26123 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.219937 26168 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:17:57.221532 26286 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.223242 26120 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:57.225136 26353 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:57.231745 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.233433 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.236277 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:57.237828 12645 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:17:57.203675 26187 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.251194 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.254218 12495 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:17:57.246160 26356 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.010s
I20260709 12:17:57.145359 26263 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.257256 26279 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.258971 12497 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
W20260709 12:17:57.259145 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:57.262740 12641 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.012s	sys 0.024s
W20260709 12:17:57.263195 12641 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.012s	sys 0.024s
W20260709 12:17:57.184262 25694 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.014s	sys 0.018s
W20260709 12:17:57.184401 26088 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.009s	sys 0.023s
I20260709 12:17:57.263998 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5187) took 1341 ms. Trace:
I20260709 12:17:57.272821 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:57.282842 12640 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:57.284029 12641 rpcz_store.cc:276] 0709 12:17:55.922550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.979658 (+ 57108us) service_pool.cc:224] Handling call
0709 12:17:55.986385 (+  6727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:55.986624 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:56.644051 (+657427us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.263642 (+619591us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:57.263708 (+    66us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:57.263753 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.263828 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.263934 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":927888,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":15918,"thread_start_us":1244277,"threads_started":2}
W20260709 12:17:57.283787 26088 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.009s	sys 0.023s
W20260709 12:17:57.292174 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.005s	sys 0.029s
W20260709 12:17:57.297977 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.300784 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:57.300472 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.005s	sys 0.029s
W20260709 12:17:57.300031 13253 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.012s	sys 0.016s
I20260709 12:17:57.301968 12637 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:57.302356 13253 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.012s	sys 0.016s
I20260709 12:17:57.284577 26123 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.305980 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:57.302434 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 647123us
I20260709 12:17:57.307111 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:57.310880 13254 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.021s	sys 0.014s
W20260709 12:17:57.311237 13254 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.021s	sys 0.014s
I20260709 12:17:57.311539 12644 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:17:57.309988 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:57.317655 12640 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:57.289238 26352 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:57.326704 13220 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.011s	sys 0.037s
W20260709 12:17:57.328009 13220 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.011s	sys 0.037s
I20260709 12:17:57.317132 26279 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.317155 26241 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:17:57.326735 26315 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.631s	user 0.013s	sys 0.001s
W20260709 12:17:57.345775 26303 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.014s	sys 0.021s
W20260709 12:17:57.370486 26303 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.015s	sys 0.023s
I20260709 12:17:57.339432 26241 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.370970 26303 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:17:57.372606 26371 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:17:57.372480 26303 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.28, Last appended by leader: 28, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:57.326519 26205 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.017s	sys 0.023s
W20260709 12:17:57.382251 26205 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.018s	sys 0.023s
I20260709 12:17:57.405231 26379 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:17:57.405937 26379 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:57.406540 26379 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:57.414603 26378 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:57.415477 26378 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:57.415978 26378 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:57.384027 26385 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.022s
I20260709 12:17:57.420704 26385 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:57.422076 26385 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:57.422601 26385 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:57.423105 26385 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:57.467617 12491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.011s	sys 0.009s
W20260709 12:17:57.468930 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.011s	sys 0.009s
W20260709 12:17:57.471906 12494 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.000s	sys 0.012s
W20260709 12:17:57.472505 12494 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.000s	sys 0.012s
I20260709 12:17:57.474555 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.475505 12495 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.22-35.22]
W20260709 12:17:57.476363 12496 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.011s	sys 0.033s
W20260709 12:17:57.488937 12496 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.011s	sys 0.034s
W20260709 12:17:57.476583 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.019s	sys 0.017s
W20260709 12:17:57.490770 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.019s	sys 0.017s
W20260709 12:17:57.491750 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2287367250000898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260709 12:17:57.476676 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:57.494825 12488 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 35: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:57.478269 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:57.496515 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.498421 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:17:57.481576 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.013s	sys 0.012s
W20260709 12:17:57.510145 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.014s	sys 0.012s
I20260709 12:17:57.482297 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5602) took 1235 ms. Trace:
I20260709 12:17:57.512277 12571 rpcz_store.cc:276] 0709 12:17:56.246678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.246874 (+   196us) service_pool.cc:224] Handling call
0709 12:17:56.247266 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:56.251985 (+  4719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:56.252016 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:56.268794 (+ 16778us) log.cc:844] Serialized 26 byte log entry
0709 12:17:56.942806 (+674012us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:56.942868 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:56.942969 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.481466 (+538497us) raft_consensus.cc:1685] finished
0709 12:17:57.481515 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.481918 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4299,"raft.queue_time_us":4957,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4937,"thread_start_us":596839,"threads_started":1,"wal-append.queue_time_us":1009307}
I20260709 12:17:57.483042 12567 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:57.485555 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:57.485931 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.001s	sys 0.009s
W20260709 12:17:57.482306 12566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.000s	sys 0.025s
W20260709 12:17:57.486670 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.008s	sys 0.001s
W20260709 12:17:57.479926 12490 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.001s	sys 0.000s
I20260709 12:17:57.488332 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:57.320027 12641 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: [38.20-38.20]
I20260709 12:17:57.507679 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:57.565929 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:57.565466 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.009s	sys 0.002s
W20260709 12:17:57.566524 12566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.000s	sys 0.025s
W20260709 12:17:57.593086 12490 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.001s	sys 0.001s
W20260709 12:17:57.593160 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.002s	sys 0.009s
I20260709 12:17:57.611179 12488 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:57.626626 12640 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:17:57.632959 26123 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:57.634579 26123 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.637864 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:17:57.640765 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.643087 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2661) took 1155 ms. Trace:
I20260709 12:17:57.645783 26298 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:57.646934 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.647290 26366 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:57.648135 12571 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:17:57.648514 26366 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.650295 26168 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:57.651573 26168 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.643841 12566 rpcz_store.cc:276] 0709 12:17:56.487251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.799241 (+311990us) service_pool.cc:224] Handling call
0709 12:17:56.799685 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:57.642137 (+842452us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:17:57.642211 (+    74us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:57.642323 (+   112us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:57.642382 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.642584 (+   202us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.642923 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":108080,"raft.queue_time_us":113967,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8987,"thread_start_us":715746,"threads_started":1}
I20260709 12:17:57.657483 12491 raft_consensus.cc:1813] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Handling vote request from an unknown peer 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:17:57.645901 26098 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.016s	sys 0.028s
I20260709 12:17:57.516623 25676 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.139s
W20260709 12:17:57.665593 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.667733 26327 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:57.667745 12640 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 41 index: 21. (index mismatch)
I20260709 12:17:57.668913 26327 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.672497 26187 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.673512 26187 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.677464 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5319) took 1333 ms. Trace:
I20260709 12:17:57.673758 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.678341 26388 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:17:57.694694 26388 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.692983 12571 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:57.698503 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:57.704751 12491 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 39.
I20260709 12:17:57.689415 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:57.712639 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.684232 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:57.677687 12492 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.016s	sys 0.028s
I20260709 12:17:57.706189 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:57.692137 26123 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.693250 26284 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.725718 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.725930 12566 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:17:57.727667 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.689370 25694 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
W20260709 12:17:57.729882 25901 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.000s	sys 0.025s
I20260709 12:17:57.678050 12565 rpcz_store.cc:276] 0709 12:17:56.344000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.692476 (+348476us) service_pool.cc:224] Handling call
0709 12:17:56.702673 (+ 10197us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:56.703086 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:56.703122 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:56.710807 (+  7685us) log.cc:844] Serialized 26 byte log entry
0709 12:17:57.511423 (+800616us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:57.511603 (+   180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.511804 (+   201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.677168 (+165364us) raft_consensus.cc:1685] finished
0709 12:17:57.677192 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.677391 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":662,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":3590,"thread_start_us":764083,"threads_started":1,"wal-append.queue_time_us":939951}
I20260709 12:17:57.730549 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.673672 26352 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:57.741678 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43708.4us
W20260709 12:17:57.742552 12569 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.012s	sys 0.004s
W20260709 12:17:57.743067 12569 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.012s	sys 0.005s
I20260709 12:17:57.744112 26123 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:57.744925 26123 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:57.731192 12492 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.018s	sys 0.030s
I20260709 12:17:57.745450 26123 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:57.737900 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.748441 12570 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7069a8d3584b466aab95934840523b8d"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:17:57.738597 26187 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:57.749809 25901 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.000s	sys 0.026s
I20260709 12:17:57.686589 26305 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.751796 26305 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.740381 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.737782 26284 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:57.754442 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.755461 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.756567 26284 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.732515 25694 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:57.715874 26404 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:57.749490 12570 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 12:17:57.697551 26113 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.641s	user 0.014s	sys 0.003s
I20260709 12:17:57.700742 26155 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:57.762305 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 226249us
I20260709 12:17:57.781260 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:17:57.759230 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.699810 26388 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:57.757570 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.679705 25676 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:57.760313 12570 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:57.701125 26279 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.733785 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.759958 26404 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.806430 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.808074 12495 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:57.809478 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5329) took 1371 ms. Trace:
I20260709 12:17:57.810467 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.810380 12568 rpcz_store.cc:276] 0709 12:17:56.438552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.772115 (+333563us) service_pool.cc:224] Handling call
0709 12:17:56.772623 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:56.834423 (+ 61800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:56.834475 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:56.876742 (+ 42267us) log.cc:844] Serialized 26 byte log entry
0709 12:17:57.646185 (+769443us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:17:57.646312 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.646470 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.808987 (+162517us) raft_consensus.cc:1685] finished
0709 12:17:57.809020 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.809347 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110749,"raft.queue_time_us":101549,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":4009,"thread_start_us":642177,"threads_started":1,"wal-append.queue_time_us":901779}
I20260709 12:17:57.812582 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.812696 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4087) took 1054 ms. Trace:
I20260709 12:17:57.814967 26404 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:57.816200 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:57.814643 12494 rpcz_store.cc:276] 0709 12:17:56.758242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.759506 (+  1264us) service_pool.cc:224] Handling call
0709 12:17:56.759987 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:56.767657 (+  7670us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:56.767697 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:56.833689 (+ 65992us) log.cc:844] Serialized 26 byte log entry
0709 12:17:57.473380 (+639691us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:57.473498 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.473677 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.812307 (+338630us) raft_consensus.cc:1685] finished
0709 12:17:57.812338 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.812580 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69499,"raft.queue_time_us":815,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":44807,"thread_start_us":558951,"threads_started":1,"wal-append.queue_time_us":928730}
I20260709 12:17:57.819350 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.819711 26305 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:57.793238 26113 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.825513 26113 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.809456 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.830555 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:17:57.825781 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 630104us
I20260709 12:17:57.827508 12565 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:57.826759 26155 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.833560 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:57.837991 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 530679us
I20260709 12:17:57.826064 12497 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: [38.21-38.21]
I20260709 12:17:57.792451 25676 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:57.864909 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.870505 12488 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:17:57.759459 25694 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 45.22, Last appended by leader: 22, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.850163 26187 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.873507 12570 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:57.880061 26395 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.131s	user 0.002s	sys 0.000s
I20260709 12:17:57.907615 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1973) took 2114 ms. Trace:
I20260709 12:17:57.907851 13259 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:57.910693 26080 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:57.911119 25676 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.915217 25676 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:57.917135 26155 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.920405 12489 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 16. Preceding OpId from leader: term: 43 index: 17. (index mismatch)
I20260709 12:17:57.872771 26279 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:57.926396 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:57.927278 26187 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.899788 26390 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:17:57.932017 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.901074 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:57.942268 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:57.950874 26155 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:57.952474 26155 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:57.956023 12637 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.653s	user 0.008s	sys 0.019s
I20260709 12:17:57.952689 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:57.956576 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:57.957444 26405 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:17:57.958073 26080 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.961128 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:57.958199 26405 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:57.962167 26404 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:17:57.963516 26405 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.30, Last appended by leader: 30, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:57.965796 26305 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:57.968817 26398 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:57.971976 26168 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:57.973312 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76992.5us
I20260709 12:17:57.975929 26284 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:17:57.976477 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 114271us
I20260709 12:17:57.976477 26284 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:57.977464 26284 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:57.978464 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:57.980474 26386 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:57.981316 26386 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:57.980712 26388 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:17:57.981328 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:57.980768 13262 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
W20260709 12:17:57.984311 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151207us
I20260709 12:17:57.988098 26279 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.988839 12494 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:17:57.954106 26380 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:57.967836 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:57.991974 12570 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:57.939421 25694 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:57.997552 26187 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:57.998260 12565 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:17:57.998829 12565 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 37.20
I20260709 12:17:58.012059 12565 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:58.018234 26417 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.019065 26417 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.021180 26417 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:58.027686 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50888.6us
I20260709 12:17:58.028232 26415 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.029251 26415 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:58.032405 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.032809 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4100) took 1109 ms. Trace:
W20260709 12:17:58.036284 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:57.908317 13253 rpcz_store.cc:276] 0709 12:17:55.766563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:55.766783 (+   220us) service_pool.cc:224] Handling call
0709 12:17:55.767151 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:55.767537 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:55.767568 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:55.771715 (+  4147us) log.cc:844] Serialized 26 byte log entry
0709 12:17:56.527849 (+756134us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:17:57.305375 (+777526us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.865206 (+559831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.869952 (+  4746us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.875716 (+  5764us) raft_consensus.cc:1685] finished
0709 12:17:57.875739 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:57.881367 (+  5628us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":876968,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":115994,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":2704,"thread_start_us":1846551,"threads_started":3,"wal-append.queue_time_us":1152254}
I20260709 12:17:58.036931 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4102) took 1100 ms. Trace:
I20260709 12:17:58.046576 26416 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:17:57.992815 26366 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.051939 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.054044 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4091) took 1291 ms. Trace:
W20260709 12:17:58.055438 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:58.061457 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92304.4us
I20260709 12:17:58.065466 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:58.021730 26417 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:58.052742 26284 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.070196 26417 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.072042 26284 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.072690 26417 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:58.073467 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95206.9us
I20260709 12:17:58.075425 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:58.076323 26378 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.558s	user 0.017s	sys 0.010s
I20260709 12:17:58.076624 26204 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:17:58.076972 26378 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.077831 26388 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:17:58.078847 26388 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:58.080052 26378 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.081079 13259 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 35. Current term is 36. Ops: []
I20260709 12:17:58.075559 26401 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:17:58.023172 26123 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.054450 26415 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.034955 12492 rpcz_store.cc:276] 0709 12:17:56.923176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.941159 (+ 17983us) service_pool.cc:224] Handling call
0709 12:17:56.964719 (+ 23560us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:56.965229 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:17:57.746204 (+780975us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.921051 (+174847us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.921257 (+   206us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:17:57.922316 (+  1059us) log.cc:844] Serialized 78 byte log entry
0709 12:17:57.922471 (+   155us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:57.922731 (+   260us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.922928 (+   197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.923073 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:57.991475 (+ 68402us) raft_consensus.cc:1685] finished
0709 12:17:57.991508 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:58.032722 (+ 41214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18096,"prepare.queue_time_us":440622,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":129,"raft.queue_time_us":13331,"raft.run_cpu_time_us":611,"raft.run_wall_time_us":21412,"thread_start_us":895218,"threads_started":2,"wal-append.queue_time_us":183400}
I20260709 12:17:57.998533 26390 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:58.111639 26412 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.069s	user 0.001s	sys 0.001s
W20260709 12:17:58.112143 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 144571us
W20260709 12:17:58.114967 26385 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.663s	user 0.018s	sys 0.004s
I20260709 12:17:58.115473 26123 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.051198 26380 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 41.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:17:58.116568 26330 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:17:58.044502 12490 rpcz_store.cc:276] 0709 12:17:56.936696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.953562 (+ 16866us) service_pool.cc:224] Handling call
0709 12:17:56.953913 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:56.954109 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:17:57.693810 (+739701us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.882588 (+188778us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:17:57.894513 (+ 11925us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:17:57.940379 (+ 45866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.991975 (+ 51596us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:58.036862 (+ 44887us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":420644,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":8475,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":14175,"thread_start_us":816514,"threads_started":2,"wal-append.queue_time_us":211174}
W20260709 12:17:58.125540 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78389.8us
I20260709 12:17:58.062127 12496 rpcz_store.cc:276] 0709 12:17:56.762670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.763066 (+   396us) service_pool.cc:224] Handling call
0709 12:17:56.763515 (+   449us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:17:56.840322 (+ 76807us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:17:57.489758 (+649436us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.664222 (+174464us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:17:57.902565 (+238343us) log.cc:844] Serialized 78 byte log entry
0709 12:17:57.905400 (+  2835us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:17:57.948555 (+ 43155us) log.cc:844] Serialized 14 byte log entry
0709 12:17:57.948896 (+   341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:58.033007 (+ 84111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:58.033056 (+    49us) raft_consensus.cc:1685] finished
0709 12:17:58.033072 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:58.053970 (+ 20898us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91711,"prepare.queue_time_us":573086,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":88,"raft.queue_time_us":134647,"raft.run_cpu_time_us":703,"raft.run_wall_time_us":32588,"spinlock_wait_cycles":41357056,"thread_start_us":710124,"threads_started":2,"wal-append.queue_time_us":144956}
I20260709 12:17:58.133008 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.148433 12568 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:58.150007 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:58.134820 26330 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:58.150516 26155 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.125905 26366 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.030s
W20260709 12:17:58.163427 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 133192us
I20260709 12:17:58.152276 26386 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:17:58.172892 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49249.3us
I20260709 12:17:58.139114 26385 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.174857 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.187182 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:58.188680 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95291.3us
I20260709 12:17:58.191913 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5332) took 1752 ms. Trace:
I20260709 12:17:58.192430 12569 rpcz_store.cc:276] 0709 12:17:56.439788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.782335 (+342547us) service_pool.cc:224] Handling call
0709 12:17:56.782768 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:57.743928 (+961160us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:57.744006 (+    78us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:57.745147 (+  1141us) log.cc:844] Serialized 26 byte log entry
0709 12:17:57.892158 (+147011us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:17:57.892211 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.898565 (+  6354us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:58.169222 (+270657us) raft_consensus.cc:1685] finished
0709 12:17:58.169246 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:58.191834 (+ 22588us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":162486,"raft.queue_time_us":166943,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":10606,"thread_start_us":975686,"threads_started":2,"wal-append.queue_time_us":276422}
I20260709 12:17:58.163654 26380 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:58.198547 26380 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:58.199152 26380 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:58.198693 26330 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.166800 26352 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.200865 26352 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.203853 26352 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:58.171895 26386 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.205003 26386 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:58.205188 26327 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:17:58.211056 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.216439 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:58.215437 12572 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:58.217597 26405 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:58.218262 26405 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:58.219369 26405 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.200423 26330 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:17:58.223728 26414 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:17:58.227689 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.228767 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5189) took 1055 ms. Trace:
W20260709 12:17:58.229967 26379 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.823s	user 0.009s	sys 0.018s
I20260709 12:17:58.230222 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.230729 26379 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.223572 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.233045 12566 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:17:58.233843 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.231956 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:58.234735 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.229135 12642 rpcz_store.cc:276] 0709 12:17:57.173488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.227919 (+ 54431us) service_pool.cc:224] Handling call
0709 12:17:58.227270 (+999351us) spinlock_profiling.cc:243] Waited 9.79 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:58.228704 (+  1434us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22526336}
I20260709 12:17:58.236603 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4168) took 1067 ms. Trace:
I20260709 12:17:58.237491 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:58.244083 26385 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.250360 12570 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:17:58.235433 26080 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:58.250774 12643 rpcz_store.cc:276] 0709 12:17:57.169054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.169185 (+   131us) service_pool.cc:224] Handling call
0709 12:17:58.236265 (+1067080us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:58.236421 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97503488}
I20260709 12:17:58.254832 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:58.261765 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.259940 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.267302 26384 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:58.252938 26204 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:58.266352 26123 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.268821 13256 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:17:58.269248 26123 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.268281 26204 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.271297 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.271757 26204 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:58.272061 26123 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.272296 26303 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:17:58.272810 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:58.275377 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.247982 26122 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:58.278434 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.266093 26187 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:58.272766 26303 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:58.284158 26303 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.26, Last appended by leader: 26, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.286509 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:58.287489 12566 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:58.290242 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:58.291939 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 141814us
I20260709 12:17:58.290050 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4173) took 1116 ms. Trace:
I20260709 12:17:58.290550 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.292891 12638 rpcz_store.cc:276] 0709 12:17:57.171786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.176764 (+  4978us) service_pool.cc:224] Handling call
0709 12:17:58.237151 (+1060387us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:58.288622 (+ 51471us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23928704}
I20260709 12:17:58.318229 12572 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 21. (term mismatch)
W20260709 12:17:58.319614 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103366us
I20260709 12:17:58.277415 26379 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.318794 12572 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:17:58.324669 12572 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 40.21
I20260709 12:17:58.325146 12572 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:58.327443 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140461us
I20260709 12:17:58.328936 12610 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 39.
I20260709 12:17:58.317799 26303 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:17:58.339282 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 149684us
I20260709 12:17:58.346210 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.351048 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:58.364370 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:58.372637 12570 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 16. Preceding OpId from leader: term: 45 index: 18. (index mismatch)
I20260709 12:17:58.381767 26204 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:58.385901 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62643.1us
I20260709 12:17:58.386904 12493 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:58.388837 26424 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:17:58.362489 26123 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.391625 26301 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
W20260709 12:17:58.393965 26325 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 41 is higher.
W20260709 12:17:58.395335 26400 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.575s	user 0.004s	sys 0.000s
I20260709 12:17:58.403976 26241 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.406335 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5614) took 1285 ms. Trace:
I20260709 12:17:58.432854 13261 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
W20260709 12:17:58.409847 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81325.4us
I20260709 12:17:58.425304 12493 raft_consensus.cc:614] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:17:58.427277 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.429942 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.432847 12572 rpcz_store.cc:276] 0709 12:17:57.121240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.123762 (+  2522us) service_pool.cc:224] Handling call
0709 12:17:57.127934 (+  4172us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:58.213053 (+1085119us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b54002b0320. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5768a1d 00007ffbf574f1a4 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419
0709 12:17:58.373734 (+160681us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:58.406233 (+ 32499us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134627584}
I20260709 12:17:58.439946 12569 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:58.443120 26385 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:17:58.443917 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66963us
I20260709 12:17:58.445025 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:58.445168 26204 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.428341 26301 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:58.446935 26301 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.448751 26301 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:17:58.448880 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.439764 26123 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.451684 12491 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:17:58.438687 26386 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.459200 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.451607 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.461310 26386 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.483636 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.487690 26384 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:17:58.488754 26384 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:17:58.492791 12566 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:58.477854 26379 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:17:58.505189 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78875us
I20260709 12:17:58.506356 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:58.506848 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.508412 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.500142 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:58.503634 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44565.3us
I20260709 12:17:58.498952 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5626) took 1237 ms. Trace:
I20260709 12:17:58.512885 12565 rpcz_store.cc:276] 0709 12:17:57.260981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.997283 (+736302us) service_pool.cc:224] Handling call
0709 12:17:57.997844 (+   561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:58.022460 (+ 24616us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:58.022500 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:58.035870 (+ 13370us) log.cc:844] Serialized 26 byte log entry
0709 12:17:58.163868 (+127998us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:17:58.163945 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:58.164047 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:58.498558 (+334511us) raft_consensus.cc:1685] finished
0709 12:17:58.498587 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:58.498810 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":100081,"threads_started":1,"wal-append.queue_time_us":222012}
I20260709 12:17:58.506969 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 40 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.521209 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:58.490239 26384 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.530652 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.530789 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.532793 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4175) took 1360 ms. Trace:
I20260709 12:17:58.533748 12639 rpcz_store.cc:276] 0709 12:17:57.172553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.212288 (+ 39735us) service_pool.cc:224] Handling call
0709 12:17:58.506622 (+1294334us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:17:58.532674 (+ 26052us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58241280}
I20260709 12:17:58.509868 25676 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:17:58.538641 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:58.544381 26216 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:17:58.540432 12571 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
W20260709 12:17:58.557677 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.551932 26168 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:58.571296 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.576032 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5464) took 1095 ms. Trace:
I20260709 12:17:58.601192 12489 rpcz_store.cc:276] 0709 12:17:57.480286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.991434 (+511148us) service_pool.cc:224] Handling call
0709 12:17:58.041990 (+ 50556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:58.130592 (+ 88602us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:17:58.130630 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:58.171877 (+ 41247us) log.cc:844] Serialized 26 byte log entry
0709 12:17:58.349689 (+177812us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:17:58.349756 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:58.349875 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:58.575715 (+225840us) raft_consensus.cc:1685] finished
0709 12:17:58.575745 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:58.575943 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":111138,"threads_started":1,"wal-append.queue_time_us":225264}
W20260709 12:17:58.582500 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112866us
I20260709 12:17:58.604106 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.606945 26433 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:58.607594 26405 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.608498 26405 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.610075 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5618) took 1487 ms. Trace:
I20260709 12:17:58.610589 12567 rpcz_store.cc:276] 0709 12:17:57.122927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.164246 (+ 41319us) service_pool.cc:224] Handling call
0709 12:17:58.609726 (+1445480us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:58.609960 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682650880}
I20260709 12:17:58.611678 26405 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.614462 12567 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
W20260709 12:17:58.614825 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.617934 26380 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.620880 26380 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.614848 12567 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 31.20
I20260709 12:17:58.583302 26123 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:58.590492 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.591325 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.594291 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 34 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.622560 12567 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:17:58.617338 12462 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.547s (SENT) [suppressed 3 similar messages]
I20260709 12:17:58.624312 13258 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:17:58.553872 26417 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:17:58.593246 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:17:58.613457 12461 connection.cc:256] RPC call timeout handler was delayed by -1.5222331620000205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24820000000000003s for a response
I20260709 12:17:58.630771 13262 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 42 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 26.
I20260709 12:17:58.632720 12570 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 43 index: 26. (term mismatch)
I20260709 12:17:58.633190 12570 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 25
I20260709 12:17:58.633433 12570 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 42.26
I20260709 12:17:58.633359 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:17:58.634384 12570 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:58.642793 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.642445 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:58.642411 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.645116 13257 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 34: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:17:58.644744 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:58.646227 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:58.646149 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.648087 12637 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:17:58.646327 12642 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:17:58.627604 26417 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:58.654088 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.654196 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:58.652781 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:58.652787 12638 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:17:58.658068 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:17:58.660281 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.639609 26303 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:58.664404 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.667145 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:58.665350 12645 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:17:58.645844 26380 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:58.675424 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90012us
I20260709 12:17:58.675359 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:58.661777 26303 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.679276 12565 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:17:58.681852 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:58.684567 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.684985 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:58.692906 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75627.1us
I20260709 12:17:58.691660 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:58.695289 26401 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:17:58.695616 26330 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.695909 26401 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:58.696627 26401 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:58.696527 26330 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.677402 26168 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:17:58.697757 26433 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:17:58.683566 26303 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:58.678963 26080 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.016s
I20260709 12:17:58.690018 26123 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:58.702554 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:58.699885 26415 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:17:58.767309 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61887.7us
I20260709 12:17:58.772495 26216 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.773804 26352 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.773451 26216 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.774673 26352 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.776722 26216 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.709300 26123 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:58.777871 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.779958 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.780781 12565 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:58.783061 12571 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
W20260709 12:17:58.784552 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 148853us
I20260709 12:17:58.785701 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:17:58.786473 26435 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:17:58.787504 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.788395 26433 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:17:58.789116 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:58.780827 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:17:58.693609 26405 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.067s
I20260709 12:17:58.701501 26366 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.068s
W20260709 12:17:58.799819 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:58.701588 26380 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.777429 26352 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.713426 26421 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:58.792631 26123 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:58.708645 26122 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.094s
I20260709 12:17:58.795367 26168 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:17:58.795898 26379 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:17:58.815429 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:58.816751 12570 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:17:58.818203 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105893us
I20260709 12:17:58.780992 26314 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.109s
W20260709 12:17:58.828919 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:58.842965 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53931.9us
I20260709 12:17:58.820684 26433 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:58.823028 26080 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:17:58.806329 26216 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:17:58.826721 26421 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:58.840924 26415 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:58.830689 26284 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.840248 26380 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.858052 26284 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.860428 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:58.862766 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:58.862974 12489 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [40.25-42.26]
I20260709 12:17:58.865537 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:58.866156 26449 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.866595 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:17:58.865659 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:58.867173 26449 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.850214 26330 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.871819 26330 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:58.872757 26314 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:17:58.872550 26330 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.873214 26314 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:58.874383 26314 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 46.30, Last appended by leader: 30, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.875362 26330 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.876196 26314 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:58.848224 26080 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:58.878278 26080 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:58.878630 26421 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.872365 26415 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.887066 26216 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:17:58.888149 26216 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:58.843497 26241 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.890137 26241 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.890885 26305 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:58.867945 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:58.889215 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5474) took 1141 ms. Trace:
I20260709 12:17:58.870710 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:58.870658 26442 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:17:58.843678 26168 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.906337 12491 rpcz_store.cc:276] 0709 12:17:57.747964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.757108 (+  9144us) service_pool.cc:224] Handling call
0709 12:17:58.888554 (+1131446us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:58.888854 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1523745536}
W20260709 12:17:58.900904 26412 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:17:58.927454 26433 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 44.19, Last appended by leader: 19, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.928767 26168 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.943809 12570 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 41 index: 29. (index mismatch)
I20260709 12:17:58.860942 26284 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.932232 26386 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
W20260709 12:17:58.935568 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:58.937542 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67007.1us
W20260709 12:17:58.925904 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64008.3us
I20260709 12:17:58.932402 12492 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:17:58.929430 26390 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:58.964977 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 1983) took 1127 ms. Trace:
I20260709 12:17:58.965427 13256 rpcz_store.cc:276] 0709 12:17:57.834593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.834914 (+   321us) service_pool.cc:224] Handling call
0709 12:17:57.835609 (+   695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:58.269371 (+433762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:58.960185 (+690814us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:58.961927 (+  1742us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":948747648}
I20260709 12:17:58.948959 26386 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:17:58.967227 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:58.963352 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132208us
I20260709 12:17:58.970952 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:58.965121 26204 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:58.976415 12644 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:17:58.962352 26216 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 45.20, Last appended by leader: 20, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.983443 26447 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:17:58.986302 26216 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:58.987648 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 92) took 2218 ms. Trace:
I20260709 12:17:58.988149 12493 rpcz_store.cc:276] 0709 12:17:56.769306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.824181 (+ 54875us) service_pool.cc:224] Handling call
0709 12:17:58.987282 (+2163101us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:58.987475 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3372807424}
I20260709 12:17:58.990041 26303 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.991011 26241 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:58.992537 26391 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.169s	user 0.004s	sys 0.005s
I20260709 12:17:58.992790 26168 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.993595 26168 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:58.994285 26168 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:58.996907 26168 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:58.999415 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.000398 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:17:58.960101 26384 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.016s
I20260709 12:17:59.006321 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.003898 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:58.960847 26305 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:58.978132 26204 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:59.012918 12563 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:17:59.013259 26204 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:59.015980 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:59.017223 26241 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:59.017532 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5622) took 1879 ms. Trace:
I20260709 12:17:59.019932 12568 rpcz_store.cc:276] 0709 12:17:57.137736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.821346 (+683610us) service_pool.cc:224] Handling call
0709 12:17:57.822070 (+   724us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:58.149223 (+327153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:59.017220 (+867997us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:59.017455 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":729852160}
W20260709 12:17:59.023452 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 157917us
I20260709 12:17:59.024852 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5325) took 2591 ms. Trace:
I20260709 12:17:59.025800 26303 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.026444 26366 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:59.017660 26241 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.027248 26241 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:59.027089 26366 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.029847 26353 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.008s
I20260709 12:17:59.035548 26453 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.036023 26449 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.036873 26449 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.037544 26449 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.040002 26449 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.035957 26305 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:59.041590 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:59.044894 26301 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:59.045403 26301 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.045966 26301 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:59.036481 26453 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.045078 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:17:59.048002 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66445.7us
I20260709 12:17:59.052119 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:59.022292 26447 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:59.053022 26447 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.053426 26447 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:59.055464 26453 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:59.052238 26451 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:17:59.058364 13261 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:59.066767 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.029505 26366 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:17:59.057662 26450 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.003s
I20260709 12:17:59.077062 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:59.072196 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:59.025305 12564 rpcz_store.cc:276] 0709 12:17:56.433670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:56.761307 (+327637us) service_pool.cc:224] Handling call
0709 12:17:56.768707 (+  7400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:56.832940 (+ 64233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:56.832982 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:56.833747 (+   765us) log.cc:844] Serialized 26 byte log entry
0709 12:17:57.646925 (+813178us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:17:57.647069 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:57.647220 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:58.991821 (+1344601us) spinlock_profiling.cc:243] Waited 37.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:59.018023 (+ 26202us) raft_consensus.cc:1685] finished
0709 12:17:59.018061 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:59.024748 (+  6687us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113141,"raft.queue_time_us":64906,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":35701,"spinlock_wait_cycles":85865856,"thread_start_us":686138,"threads_started":1,"wal-append.queue_time_us":941581}
W20260709 12:17:59.096285 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51316.1us
I20260709 12:17:59.086653 26301 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:59.104393 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 136221us
I20260709 12:17:59.092702 26168 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:17:59.126334 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 76178.2us
I20260709 12:17:59.073909 26453 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.045s
I20260709 12:17:59.139096 12571 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 39 index: 27. (index mismatch)
I20260709 12:17:59.140100 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.052160 26386 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.146056 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:59.144768 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.093051 26449 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:17:59.167683 26384 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.003s
I20260709 12:17:59.172344 26415 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:17:59.173308 26415 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:59.173714 26415 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:59.174707 26415 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.176330 26415 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:59.178150 26263 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:17:59.180186 26241 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.182535 26401 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.185252 26401 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.186786 26301 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.136145 26305 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.052s
I20260709 12:17:59.188364 26433 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:17:59.166693 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.194906 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.138475 25676 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:17:59.197153 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53823.8us
I20260709 12:17:59.200248 12571 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:59.199157 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.188752 26433 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:59.206781 26385 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.211266 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.210701 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.214732 12564 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:59.135545 26187 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.067s
I20260709 12:17:59.217046 26204 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.219054 26390 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:17:59.229053 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:17:59.199748 25676 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:17:59.214512 26385 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:59.253268 12571 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:17:59.259109 26204 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:17:59.260972 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64390.2us
I20260709 12:17:59.243070 26284 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:59.263911 26284 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.264330 26284 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:59.236475 26123 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.241154 26080 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.268416 26451 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:17:59.275887 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.238372 25676 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:17:59.278020 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.281399 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.283807 12571 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:59.279486 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:59.282140 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:17:59.289027 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 179535us
I20260709 12:17:59.291659 26123 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:59.299932 12563 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:17:59.293277 26451 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:17:59.303854 26451 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.304312 26451 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:17:59.305500 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.310163 26241 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.311185 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.298313 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.311686 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.301855 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.318981 12565 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.319475 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:59.321400 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:59.322376 26366 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:17:59.322978 26366 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:59.323182 26301 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:59.323792 26301 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.324255 26301 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.325798 26366 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.329337 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.330302 26352 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:17:59.324203 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.331568 26305 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:59.331921 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5352) took 1106 ms. Trace:
I20260709 12:17:59.332448 26352 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:17:59.333117 26352 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:59.334614 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:59.336284 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:17:59.337821 26461 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:17:59.339241 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:17:59.340240 26382 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.882s	user 0.008s	sys 0.005s
I20260709 12:17:59.332491 12569 rpcz_store.cc:276] 0709 12:17:58.225777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:58.253758 (+ 27981us) service_pool.cc:224] Handling call
0709 12:17:59.331658 (+1077900us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:59.331843 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":401219968}
I20260709 12:17:59.343523 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.343837 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.344439 12493 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:59.344846 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.345243 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.346063 12491 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:17:59.346407 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:59.331977 26305 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.347237 26305 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.347357 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.348232 12497 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:17:59.350128 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.351027 12490 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 46 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 29.
W20260709 12:17:59.353931 26440 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.690s	user 0.003s	sys 0.001s
I20260709 12:17:59.354219 26447 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.354678 12492 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:17:59.355212 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:59.355827 12639 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.356758 26447 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.357901 12571 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 19.
I20260709 12:17:59.358104 12642 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:17:59.358498 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.359421 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:59.361287 12637 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: [40.23-40.23]
W20260709 12:17:59.372957 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.374470 12491 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:17:59.374284 26284 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.374832 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.375826 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.376362 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.376799 26284 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.376688 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.377135 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.376209 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.384868 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.386091 12488 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 19.
I20260709 12:17:59.350145 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.388649 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:59.389315 12646 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:59.390297 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:59.388425 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.394913 12571 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:17:59.396636 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.395236 12494 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:17:59.394520 12565 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:17:59.395159 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.398772 12569 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:59.393929 12495 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:17:59.398011 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
I20260709 12:17:59.404006 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:59.404831 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:59.406032 12642 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:17:59.404997 12643 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:59.406773 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.409830 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.410553 12491 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:17:59.418572 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84072.4us
I20260709 12:17:59.395773 26314 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:17:59.419472 26284 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.419948 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:59.420219 26284 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.421834 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.422578 12641 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:17:59.422740 26284 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.423899 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.403479 26123 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:17:59.425323 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:59.438048 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 101890us
I20260709 12:17:59.428876 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:59.446704 12491 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:17:59.449110 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:17:59.421506 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.432686 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:59.452801 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:59.454900 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.462598 26284 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:59.463423 26284 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.463886 26284 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:59.466333 26464 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:17:59.467195 26464 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:59.469969 12497 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:17:59.437757 26298 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:17:59.480604 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:17:59.438220 26462 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.437934 26123 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.434311 26445 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:17:59.484463 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1199069539998163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1579s for a response
I20260709 12:17:59.429692 26416 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.483945 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:59.490154 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0110069539998676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
I20260709 12:17:59.509202 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
W20260709 12:17:59.507194 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85751.6us
I20260709 12:17:59.492841 12641 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:17:59.505616 12639 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:59.479581 26452 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:59.510993 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:59.515547 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60826.1us
I20260709 12:17:59.519212 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.490280 26123 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:17:59.486040 26241 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.486675 26366 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:17:59.492512 26445 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.430033 26352 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.522504 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:59.510989 26421 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:59.527520 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.528376 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:17:59.535245 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.535718 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.536535 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:17:59.536767 26401 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:59.537106 12490 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [44.19-45.20]
I20260709 12:17:59.537325 26401 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.539482 26401 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.539654 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.540179 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.541179 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:17:59.541124 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.542130 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.542488 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.542830 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.543653 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.543942 12488 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.25-44.25]
I20260709 12:17:59.544126 12497 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:17:59.544308 12495 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:17:59.544795 12495 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:59.544981 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.545595 12495 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.546828 12495 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.546427 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.546586 12488 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 43 index: 26. (index mismatch)
I20260709 12:17:59.550072 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.550902 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.542968 12493 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:17:59.551987 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.552349 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.552786 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.553208 12497 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:17:59.555027 12493 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 41 index: 29. (index mismatch)
I20260709 12:17:59.556319 12493 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 30. Preceding OpId from leader: term: 47 index: 31. (index mismatch)
I20260709 12:17:59.559756 12495 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:17:59.497273 25676 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.525733 26447 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:17:59.572211 26385 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.568428 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:59.573900 26467 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:17:59.572870 26447 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.488168 26301 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.574821 25676 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:59.575932 12497 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:17:59.497956 26416 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.562151 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:17:59.579856 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60591.9us
I20260709 12:17:59.521364 25694 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:17:59.522262 26451 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.530045 26445 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:59.525663 26366 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:17:59.577527 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50111.7us
W20260709 12:17:59.624040 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55859.6us
I20260709 12:17:59.505822 26462 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.631429 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.622530 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:17:59.586320 26447 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:59.527990 26241 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.605182 26452 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:17:59.605808 26446 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:17:59.644712 12488 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:17:59.626051 26391 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.095s	user 0.000s	sys 0.004s
I20260709 12:17:59.656392 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:59.611073 25694 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.617210 26453 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:17:59.661449 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:59.665474 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 232876us
I20260709 12:17:59.667529 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.668763 12643 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 34.
I20260709 12:17:59.627686 26366 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:17:59.673298 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:17:59.686370 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 134477us
I20260709 12:17:59.630489 26415 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:17:59.690320 26485 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:59.667595 25694 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:59.686520 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:17:59.704582 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:17:59.601581 26464 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.685163 26327 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:17:59.712720 12643 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:17:59.711863 26446 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:17:59.715040 26482 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.051s
I20260709 12:17:59.715852 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5202) took 2441 ms. Trace:
I20260709 12:17:59.716612 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 1989) took 1441 ms. Trace:
I20260709 12:17:59.728410 13262 rpcz_store.cc:276] 0709 12:17:58.275452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:58.275589 (+   137us) service_pool.cc:224] Handling call
0709 12:17:59.716375 (+1440786us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:17:59.716523 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":777884416}
I20260709 12:17:59.696141 26453 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:17:59.724239 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:59.697165 26384 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:17:59.702056 26405 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.029s
I20260709 12:17:59.728032 12644 rpcz_store.cc:276] 0709 12:17:57.274309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.325651 (+ 51342us) service_pool.cc:224] Handling call
0709 12:17:57.326109 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:59.022511 (+1696402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:59.715472 (+692961us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:59.715753 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3736077696}
I20260709 12:17:59.722708 26446 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.754745 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:17:59.761603 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49862.7us
I20260709 12:17:59.731110 26485 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:17:59.750842 26327 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:17:59.725181 26401 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:17:59.778313 26479 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:17:59.785768 26447 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.778345 26448 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:17:59.787822 26448 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:59.787916 26462 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:59.789542 26447 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.787309 12639 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:17:59.789077 26448 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.794412 26384 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:17:59.763352 26446 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:17:59.795397 26431 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.357s	user 0.012s	sys 0.008s
I20260709 12:17:59.792975 26433 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.795214 26448 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:17:59.771143 26385 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:17:59.797102 26433 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:17:59.798115 26455 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:17:59.798622 26455 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:17:59.785153 26123 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.799523 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:17:59.800284 12497 raft_consensus.cc:606] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.802291 26123 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.805954 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:17:59.797060 26436 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.131s	user 0.003s	sys 0.000s
I20260709 12:17:59.805877 26366 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.812137 26366 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.813455 26417 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.816854 26417 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:17:59.818357 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.821678 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:17:59.774001 26327 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:17:59.841794 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.860849 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:17:59.830541 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:17:59.870640 12639 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:17:59.874331 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1714) took 1943 ms. Trace:
I20260709 12:17:59.889783 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:17:59.890616 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:17:59.892294 13259 rpcz_store.cc:276] 0709 12:17:57.931497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:57.931715 (+   218us) service_pool.cc:224] Handling call
0709 12:17:57.939280 (+  7565us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:17:58.081690 (+142410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:59.874021 (+1792331us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:17:59.874254 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":325431168}
W20260709 12:17:59.848479 26305 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.501s	user 0.014s	sys 0.000s
I20260709 12:17:59.823885 26301 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.896265 12563 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
W20260709 12:17:59.859490 26438 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.193s	user 0.000s	sys 0.004s
I20260709 12:17:59.832670 26386 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:17:59.835968 26330 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:17:59.886236 26327 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:17:59.826481 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 131269us
W20260709 12:17:59.899997 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 226780us
I20260709 12:17:59.842584 26416 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:59.839754 26241 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.854912 26401 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.784986 26284 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.870288 26080 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:17:59.826647 25694 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.861011 26449 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:17:59.914911 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:17:59.837433 26451 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:59.928288 12565 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 20. Preceding OpId from leader: term: 46 index: 20. (term mismatch)
I20260709 12:17:59.928475 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.929864 12564 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:17:59.929157 12565 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:17:59.930397 12565 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 45.20
I20260709 12:17:59.930693 12565 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:17:59.906123 26330 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:17:59.861202 26462 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.057s
I20260709 12:17:59.918260 26452 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:17:59.933620 26330 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.941191 26452 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:17:59.945756 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.947109 26496 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.947759 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:17:59.948133 26451 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:17:59.948755 26401 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:17:59.948935 26451 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.951604 26284 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:17:59.947865 26496 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:17:59.955992 12495 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:17:59.962944 26385 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:17:59.937423 26205 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:17:59.964061 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:17:59.963707 26385 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.901981 26305 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:17:59.966398 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:17:59.971139 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2667) took 1492 ms. Trace:
I20260709 12:17:59.971765 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.929895 26449 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:17:59.974112 12566 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:17:59.972659 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:17:59.971844 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:59.934768 26462 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 32.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:17:59.972333 12572 rpcz_store.cc:276] 0709 12:17:58.480350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:58.485770 (+  5420us) service_pool.cc:224] Handling call
0709 12:17:58.486383 (+   613us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:58.486734 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:58.486766 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:58.487470 (+   704us) log.cc:844] Serialized 26 byte log entry
0709 12:17:58.623690 (+136220us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:17:58.623736 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:17:58.623825 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:17:59.858668 (+1234843us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:17:59.966972 (+108304us) raft_consensus.cc:1685] finished
0709 12:17:59.967010 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:17:59.971036 (+  4026us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18540288,"thread_start_us":135663,"threads_started":1,"wal-append.queue_time_us":178497}
I20260709 12:17:59.978788 12644 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 47 index: 31. (index mismatch)
I20260709 12:17:59.976223 26416 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.005s
I20260709 12:17:59.988620 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.990756 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.994534 26353 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:17:59.998385 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:17:59.999578 12570 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:17:59.954492 26301 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:17:59.965984 26080 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:18:00.003630 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48140.8us
I20260709 12:18:00.006011 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:00.004740 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:00.008967 26490 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:00.011548 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:17:59.983770 26385 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.026577 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.015287 26496 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.006811 26080 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:00.041822 12490 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.23-41.23]
I20260709 12:18:00.042649 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:18:00.046455 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:00.042579 13259 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:00.020995 26415 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:00.058851 12645 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
W20260709 12:18:00.059242 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47814.9us
I20260709 12:18:00.062728 12566 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:00.113838 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:00.026520 26301 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.123867 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.119393 26453 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:00.125067 26453 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:00.125483 26453 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:00.125696 26301 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:00.126499 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.128006 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.128612 12569 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:00.058750 26080 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:18:00.130551 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99924.1us
I20260709 12:18:00.114529 26490 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:18:00.135370 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.054172 26385 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:00.136094 26353 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:00.136608 26385 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:00.121560 26496 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:00.138548 25694 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.139392 26385 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.140442 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:00.141198 13257 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:00.055214 26491 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:18:00.148168 26401 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:18:00.148339 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102031us
I20260709 12:18:00.149428 26491 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:00.150611 26491 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.25, Last appended by leader: 25, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:00.152817 13262 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:00.125505 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.156574 12566 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 47
I20260709 12:18:00.155505 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:00.163163 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:00.163695 12570 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
W20260709 12:18:00.165030 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:00.165263 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:00.167068 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124680us
I20260709 12:18:00.054584 26187 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:00.178334 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:00.175506 26401 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:18:00.180846 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 208244us
I20260709 12:18:00.189726 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:00.191859 12488 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:00.194990 12494 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 18. Preceding OpId from leader: term: 46 index: 20. (index mismatch)
W20260709 12:18:00.171514 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178203us
I20260709 12:18:00.196724 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:00.193571 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:00.204289 13259 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:00.204272 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74333.6us
I20260709 12:18:00.209111 12497 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:18:00.058914 26352 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:00.213145 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62058.3us
I20260709 12:18:00.224589 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:18:00.231353 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:00.218904 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:00.162631 26385 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:00.242481 26385 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:18:00.245981 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:00.248510 12644 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:00.249260 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:00.250237 12639 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:00.167420 26501 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:00.252209 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:00.253592 26352 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.243146 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.255198 12567 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:00.212826 26449 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:00.256835 26449 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:00.190043 25694 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:00.257225 26301 consensus_queue.cc:1175] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 38 status { last_received { term: 38 index: 23 } last_committed_idx: 23 last_received_current_leader { term: 38 index: 23 } }
I20260709 12:18:00.244951 26415 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:00.240290 12497 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:00.261931 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:00.259737 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:00.269593 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:00.269901 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:00.242208 12488 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:00.176481 26496 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:00.264293 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:00.180817 26401 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:00.018093 26455 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.26, Last appended by leader: 26, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.306253 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:00.309583 12569 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: [36.24-39.26]
I20260709 12:18:00.310592 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:00.307185 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:18:00.291057 26482 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:00.317955 12493 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:00.291896 26448 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:00.258145 25694 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.298290 26401 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:00.306437 26494 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:00.269968 26205 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:00.341547 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:00.316913 26464 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:00.363019 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:00.363941 12492 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:00.367862 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:00.368747 12493 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:00.371775 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:00.327343 26482 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:00.298013 26187 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.406814 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:00.408800 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:00.410005 12497 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 24.
I20260709 12:18:00.356691 26448 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:00.331431 26123 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.048s
W20260709 12:18:00.438885 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 305404us
I20260709 12:18:00.324472 26496 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:00.444020 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:00.379135 26482 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:00.347558 26284 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.069s
I20260709 12:18:00.461856 13256 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:00.467680 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 269269us
I20260709 12:18:00.474130 12488 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:18:00.452178 26433 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.654s	user 0.007s	sys 0.010s
W20260709 12:18:00.451815 26386 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.543s	user 0.013s	sys 0.000s
I20260709 12:18:00.492661 12488 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:18:00.377558 26464 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.369692 26205 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.401448 26330 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:00.487429 26472 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:18:00.483655 26384 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:00.536561 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:00.543468 13256 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
W20260709 12:18:00.499989 26493 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.479s	user 0.002s	sys 0.000s
I20260709 12:18:00.496527 26352 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:00.560117 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 349331us
I20260709 12:18:00.572451 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:00.463277 26415 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:00.564267 26446 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.768s	user 0.010s	sys 0.003s
I20260709 12:18:00.594717 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4106) took 1056 ms. Trace:
I20260709 12:18:00.597726 12490 rpcz_store.cc:276] 0709 12:17:59.538654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.538787 (+   133us) service_pool.cc:224] Handling call
0709 12:17:59.539156 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:00.053912 (+514756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:00.591502 (+537590us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:00.594629 (+  3127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147067648}
I20260709 12:18:00.650141 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:00.679698 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78067.8us
W20260709 12:18:00.615665 26493 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:18:00.683625 26452 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:00.684427 26508 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:00.686019 26500 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
W20260709 12:18:00.687172 26510 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:18:00.690907 12490 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:00.691435 26464 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:00.693253 26187 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:00.738664 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:00.595893 26501 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.115s
I20260709 12:18:00.705585 12494 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
W20260709 12:18:00.707741 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171255us
W20260709 12:18:00.668296 26499 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.360s	user 0.002s	sys 0.000s
W20260709 12:18:00.710359 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:00.674110 26509 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:18:00.562670 26384 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:00.731349 12490 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:00.671731 26466 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:00.677399 26241 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:00.738009 25694 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.562566 26448 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.787431 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:00.788112 12569 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:00.699919 12493 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
W20260709 12:18:00.741284 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 272537us
I20260709 12:18:00.694804 26496 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:00.731292 26498 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.389s	user 0.000s	sys 0.002s
W20260709 12:18:00.793601 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.010s	sys 0.014s
W20260709 12:18:00.794384 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.011s	sys 0.014s
W20260709 12:18:00.794862 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 599405us
I20260709 12:18:00.797472 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:00.760632 13259 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.002s
I20260709 12:18:00.802619 13259 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 42 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 26.
W20260709 12:18:00.763646 12567 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.508s	user 0.017s	sys 0.001s
W20260709 12:18:00.763667 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 199504us
W20260709 12:18:00.764781 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:00.804483 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:18:00.616154 26446 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.757067 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:00.665588 25676 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.073s
I20260709 12:18:00.780128 12491 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:18:00.784507 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:18:00.806012 12491 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:00.806216 12536 leader_election.cc:403] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
W20260709 12:18:00.806668 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49757.2us
W20260709 12:18:00.807258 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:00.785368 12570 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [36.19-36.19]
W20260709 12:18:00.763127 26327 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.839s	user 0.014s	sys 0.001s
I20260709 12:18:00.782590 12490 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.808558 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:00.788335 26241 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:00.830006 12638 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:00.828979 12637 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:00.827353 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.798058 26466 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.905433 26517 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:00.922869 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 1985) took 1497 ms (client timeout 1845 ms). Trace:
W20260709 12:18:00.923399 12642 rpcz_store.cc:269] 0709 12:17:59.425116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.480306 (+ 55190us) service_pool.cc:224] Handling call
0709 12:18:00.922715 (+1442409us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:00.922810 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2344979328}
I20260709 12:18:00.924969 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:00.834271 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:00.845678 12569 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:18:00.926011 26514 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.375s	user 0.001s	sys 0.003s
I20260709 12:18:00.930727 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:00.892416 26205 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.926883 26527 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:00.826946 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:00.904224 26504 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
W20260709 12:18:00.876767 26417 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.059s	user 0.013s	sys 0.004s
I20260709 12:18:00.923285 26517 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.871675 26263 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:00.815689 26327 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:00.823392 12490 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:00.935604 12639 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:00.930962 12644 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:00.891767 26384 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.935477 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:00.967010 12642 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:00.968788 12642 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:00.943985 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.873843 25676 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 34.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
W20260709 12:18:00.975759 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167316us
I20260709 12:18:00.946888 12572 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:00.841603 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:00.932858 26080 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.803s	user 0.012s	sys 0.003s
W20260709 12:18:00.833591 26453 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.707s	user 0.010s	sys 0.000s
I20260709 12:18:00.855669 12491 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:00.962903 12492 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.565s	user 0.012s	sys 0.000s
I20260709 12:18:00.962687 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:00.986081 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:00.853152 26330 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.986668 12638 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:00.965044 26416 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:00.815788 12567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2679) took 1714 ms (client timeout 1957 ms). Trace:
W20260709 12:18:00.987974 12567 rpcz_store.cc:269] 0709 12:17:59.090309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:00.212625 (+1122316us) service_pool.cc:224] Handling call
0709 12:18:00.804975 (+592350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:00.987782 26416 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:00.989646 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:00.921429 26513 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:00.989825 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.994164 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:00.855306 26446 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:00.942852 26449 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.685s	user 0.008s	sys 0.008s
I20260709 12:18:00.809614 26405 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.138s
I20260709 12:18:01.051504 26521 raft_consensus.cc:2764] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election decision vote started in defunct term 43: won
I20260709 12:18:00.975085 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:01.053180 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.054934 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.975598 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:00.981087 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 197561us
I20260709 12:18:01.055823 26241 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:01.056069 12637 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:01.056880 12536 leader_election.cc:400] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:18:01.057255 12536 leader_election.cc:403] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:00.812282 26491 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.150s
I20260709 12:18:01.058805 26491 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:01.059206 26491 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:01.060351 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5502) took 1243 ms. Trace:
I20260709 12:18:01.060218 26491 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:00.958134 26445 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.314s	user 0.016s	sys 0.000s
I20260709 12:18:01.060971 12495 rpcz_store.cc:276] 0709 12:17:59.816854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.821236 (+  4382us) service_pool.cc:224] Handling call
0709 12:18:01.060071 (+1238835us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:01.060267 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305003392}
I20260709 12:18:01.061463 26445 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.063365 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.064096 26445 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:00.998883 26522 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:18:00.998751 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5223) took 1351 ms. Trace:
I20260709 12:18:00.999258 26532 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:01.065444 12640 rpcz_store.cc:276] 0709 12:17:59.647555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.873130 (+225575us) service_pool.cc:224] Handling call
0709 12:17:59.873529 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:17:59.888898 (+ 15369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:17:59.889042 (+   144us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:17:59.918847 (+ 29805us) log.cc:844] Serialized 26 byte log entry
0709 12:18:00.222508 (+303661us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:00.222554 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:00.222660 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:00.998453 (+775793us) raft_consensus.cc:1685] finished
0709 12:18:00.998481 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:00.998661 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":204,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":3732,"thread_start_us":256009,"threads_started":1,"wal-append.queue_time_us":653501}
I20260709 12:18:01.069116 26491 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:00.997153 26518 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:18:00.922343 26524 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 44.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.214s
W20260709 12:18:00.999923 26511 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:18:01.000357 26525 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:18:01.002405 12639 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:18:01.007033 26327 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.009121 26466 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:00.960676 26527 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.015746 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.993275 26080 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.019187 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:00.983179 26352 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:01.019770 26517 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:00.946202 26464 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:01.022729 26401 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.656s	user 0.014s	sys 0.000s
I20260709 12:18:01.024358 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1982) took 1600 ms. Trace:
I20260709 12:18:01.081789 12643 rpcz_store.cc:276] 0709 12:17:59.424202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.767208 (+343006us) service_pool.cc:224] Handling call
0709 12:17:59.981813 (+214605us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:00.026345 (+ 44532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:00.026379 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:00.055833 (+ 29454us) log.cc:844] Serialized 26 byte log entry
0709 12:18:00.345159 (+289326us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:00.345220 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:00.345342 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:00.925012 (+579670us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:01.023820 (+ 98808us) raft_consensus.cc:1685] finished
0709 12:18:01.023853 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:01.024276 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88697,"raft.queue_time_us":44552,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":29672,"spinlock_wait_cycles":9974656,"thread_start_us":147246,"threads_started":1,"wal-append.queue_time_us":299902}
I20260709 12:18:00.830377 26497 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.213s
I20260709 12:18:01.025991 12571 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:18:01.029933 26314 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:18:01.031311 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:01.087894 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:01.088038 12566 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:01.032550 26538 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:18:01.032974 26516 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:18:01.034415 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:00.974628 26187 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:01.039242 26512 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:18:01.041512 12492 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:00.922382 26519 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:01.040522 26453 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.057978 26462 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35.
I20260709 12:18:00.995651 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.058113 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:00.972936 26263 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.102074 12567 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:18:01.079283 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:01.103266 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.104516 12567 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 24.
I20260709 12:18:01.093847 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:01.098604 26516 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:01.107367 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49341.2us
I20260709 12:18:01.078322 26449 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:01.125684 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 310176us
I20260709 12:18:01.127938 12568 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:01.130450 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:01.127125 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:01.094820 26538 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:01.118803 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 300811us
I20260709 12:18:01.099198 26519 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:01.139207 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 330001us
I20260709 12:18:01.141016 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:01.141965 12492 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 25.
I20260709 12:18:01.144407 12639 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:18:01.147378 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:01.101531 26187 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.150386 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:01.114301 26314 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 37.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:18:01.121434 26327 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:01.149292 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:01.087131 26352 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.161491 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:01.172209 12494 raft_consensus.cc:2368] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43.
I20260709 12:18:01.162451 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:01.163206 12566 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 39 because replica has last-logged OpId of term: 38 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 20.
I20260709 12:18:01.163115 12640 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:18:01.125192 26421 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.178857 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:01.179721 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:01.095160 26401 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.157544 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:01.173105 12646 raft_consensus.cc:2368] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43.
I20260709 12:18:01.175928 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.156149 26462 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:01.186208 26462 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.188951 26462 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.113590 26464 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.189597 26330 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:01.189845 25694 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:18:01.190335 26330 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:01.190750 26330 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:01.164561 26417 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:01.191684 26527 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.192894 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:01.194257 12643 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:01.194362 26352 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.195264 26352 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:01.197487 26445 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:01.203300 12641 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:01.140250 26538 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:18:01.106823 26531 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.089s
I20260709 12:18:01.227008 12490 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 23. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
I20260709 12:18:01.227083 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.146214 26519 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:01.125878 26517 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.044s
I20260709 12:18:01.125434 26449 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.160990 26524 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.235881 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:01.153581 26420 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:01.240078 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:01.240125 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 254108us
I20260709 12:18:01.238428 12637 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:01.237694 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:01.267926 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:01.276840 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:01.277356 13256 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:01.279913 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:01.280493 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:01.286077 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.169922 26446 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.073s
I20260709 12:18:01.289273 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:01.295939 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:01.229985 26327 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.242053 26448 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
W20260709 12:18:01.318768 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144515us
I20260709 12:18:01.320519 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:01.322680 12490 raft_consensus.cc:2368] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:18:01.226665 26421 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.306896 13253 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:01.308214 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:01.325760 13259 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:01.324958 26417 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:01.307459 13256 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:01.257507 26455 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:01.329469 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:01.330492 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:01.307837 26385 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:01.318291 26386 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.355628 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:01.362963 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77020.3us
I20260709 12:18:01.304836 26420 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:01.308562 26542 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.319547 26541 raft_consensus.cc:2764] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:18:01.351055 26517 consensus_queue.cc:1243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 42.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:18:01.350479 26521 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:01.374509 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:01.372205 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 222984us
I20260709 12:18:01.286733 26524 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.331422 26515 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:18:01.369418 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:01.343369 26448 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:01.387827 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:01.362473 26490 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:01.390961 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:01.393518 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2111) took 1207 ms. Trace:
I20260709 12:18:01.367858 26385 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:01.386793 26453 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:01.308511 26516 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.417979 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:18:01.395918 26482 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.903s	user 0.017s	sys 0.000s
I20260709 12:18:01.418160 12565 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:01.407284 26385 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:01.424269 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 76428.5us
I20260709 12:18:01.396539 12496 rpcz_store.cc:276] 0709 12:18:00.186123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:00.186338 (+   215us) service_pool.cc:224] Handling call
0709 12:18:00.218833 (+ 32495us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:00.268468 (+ 49635us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:00.268503 (+    35us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:00.330767 (+ 62264us) log.cc:844] Serialized 78 byte log entry
0709 12:18:00.818670 (+487903us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:01.301095 (+482425us) log.cc:844] Serialized 14 byte log entry
0709 12:18:01.318508 (+ 17413us) log.cc:844] Serialized 14 byte log entry
0709 12:18:01.393146 (+ 74638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:01.393236 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:01.393278 (+    42us) raft_consensus.cc:1685] finished
0709 12:18:01.393288 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:01.393457 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31626,"prepare.queue_time_us":479273,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":221006,"raft.run_cpu_time_us":664,"raft.run_wall_time_us":63669,"spinlock_wait_cycles":12249856,"thread_start_us":988502,"threads_started":3,"wal-append.queue_time_us":507277}
I20260709 12:18:01.397488 26080 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:01.438347 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:01.439695 12490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:01.442977 12490 raft_consensus.cc:606] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:01.379290 26386 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.367580 26543 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.069s
I20260709 12:18:01.398702 26521 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:01.451630 12496 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:18:01.395465 26517 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.044s
I20260709 12:18:01.465174 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.469034 26547 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:01.423794 26327 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:01.454157 26546 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:18:01.383625 26542 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.434530 26549 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:18:01.493304 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55077.2us
I20260709 12:18:01.498741 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.406134 26490 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.502036 26515 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:18:01.504120 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 361180us
I20260709 12:18:01.505542 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:01.505156 12566 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:01.506142 26543 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:18:01.506549 26543 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:01.506956 26543 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:18:01.507293 13253 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:01.507477 12566 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:01.506568 13256 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:01.508844 12566 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:01.509389 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.510111 12566 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:01.511359 12566 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:01.514380 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 36. Current term is 37.
I20260709 12:18:01.424875 26527 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.059s
I20260709 12:18:01.520304 12496 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: [33.19-37.21]
I20260709 12:18:01.520437 12566 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 35 index: 26. (index mismatch)
I20260709 12:18:01.515821 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.523674 12572 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:01.436245 26482 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:01.529588 26545 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:18:01.483952 26453 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.039s
I20260709 12:18:01.444407 26417 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.26, Last appended by leader: 26, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.553687 12566 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:01.499449 26330 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.514580 26517 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.029s
I20260709 12:18:01.563246 12563 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:18:01.571305 12496 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: [36.22-39.24]
I20260709 12:18:01.539429 26449 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.540097 26527 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:01.575947 26490 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:01.578657 26451 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.592420 26524 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.582079 26562 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:01.583140 13262 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:18:01.583461 26554 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:18:01.585059 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:01.604739 12639 raft_consensus.cc:2376] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41. Re-sending same reply.
I20260709 12:18:01.581804 26563 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.553977 26466 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.602018 13262 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:01.521997 26448 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 45.22, Last appended by leader: 22, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.497426 26420 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.618326 26576 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:01.618986 26576 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.603983 26401 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.587361 26527 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.623095 26527 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.623957 26330 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.624401 26527 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:01.626652 26575 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:01.627151 26575 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.628118 26575 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.630334 26575 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:01.606762 26562 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.614837 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:01.614979 26542 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.615736 13262 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 39.26, Last appended by leader: 26, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.598112 26517 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:01.617007 26566 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:18:01.604975 26491 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.533s	user 0.013s	sys 0.000s
W20260709 12:18:01.656581 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:01.630872 26482 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.632718 26386 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.613391 12488 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:01.660919 26542 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:01.648510 13262 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:01.495278 26521 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.671083 12488 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: [40.22-40.22]
I20260709 12:18:01.620074 26576 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.24, Last appended by leader: 24, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.649609 26549 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:01.673091 26506 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:01.678031 26538 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.650022 26517 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.653779 26566 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:01.686981 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 171239us
I20260709 12:18:01.639490 26466 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.700043 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.613814 26563 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.698990 26566 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:01.708384 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:01.709362 26517 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.720513 26517 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:01.722247 26526 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.851s	user 0.000s	sys 0.003s
I20260709 12:18:01.704149 26506 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.711506 26563 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:01.718528 12569 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 25.
I20260709 12:18:01.727212 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:01.719295 26462 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:01.728614 26462 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.730005 26562 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.729707 26462 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.705289 26524 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:01.684022 26385 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.669190 26542 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:01.755451 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46972us
I20260709 12:18:01.731318 26462 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:01.732890 26327 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:01.725406 26506 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.727705 26549 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.769037 13259 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:01.769834 12569 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:18:01.771580 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:01.776409 26466 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.766585 26448 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.777472 26448 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.780097 26542 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.780620 26448 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:01.787240 12569 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:01.717235 26543 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.789551 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.791965 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.797886 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.800490 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.801653 26575 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:01.807387 13262 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 40 index: 26. (term mismatch)
I20260709 12:18:01.807853 26562 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:01.808907 26562 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:01.809703 26581 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:18:01.807861 13262 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 25
I20260709 12:18:01.812640 13262 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 39.26
I20260709 12:18:01.812988 13262 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:01.814783 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.815914 12569 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:01.817420 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:01.817744 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:01.821705 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:01.824296 25676 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:01.824846 25676 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.825193 26543 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.825883 25676 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.827507 26566 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.829367 26417 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:01.792695 26327 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.816180 26420 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:18:01.823171 13262 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:01.820588 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.794605 26448 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:01.841922 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44342us
I20260709 12:18:01.843779 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:01.844792 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:01.843627 12536 leader_election.cc:400] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43.
I20260709 12:18:01.845886 12536 leader_election.cc:403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Cancelling election due to peer responding with higher term
I20260709 12:18:01.833300 26417 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:01.843056 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:01.847357 26575 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:01.768713 26560 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.073s
I20260709 12:18:01.825098 26580 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:01.833207 26562 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.850920 26579 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:01.811645 26466 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.039s
I20260709 12:18:01.846002 26455 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.869866 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:01.872551 12536 leader_election.cc:400] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43.
I20260709 12:18:01.873912 12536 leader_election.cc:400] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:18:01.874203 12536 leader_election.cc:403] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:01.867663 26301 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:01.875842 26455 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.877350 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:01.887502 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:01.869011 26543 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:01.890853 26543 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:01.891366 26543 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:01.878389 26417 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.895078 26464 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.886759 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:01.906925 26560 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:18:01.906956 26464 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:01.889215 26580 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:01.941458 26464 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:01.936718 26543 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.874732 26517 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election lost for term 42. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43.
W20260709 12:18:01.920635 26519 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.593s	user 0.015s	sys 0.000s
I20260709 12:18:01.943271 26519 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.944811 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 1989) took 2518 ms. Trace:
I20260709 12:18:01.945104 26543 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.945303 12645 rpcz_store.cc:276] 0709 12:17:59.426575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.783114 (+356539us) service_pool.cc:224] Handling call
0709 12:17:59.783446 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:00.121270 (+337824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:01.944492 (+1823222us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:01.944737 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":619496192}
I20260709 12:18:01.906414 26527 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:01.946596 26527 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:01.926787 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:01.951061 26519 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.923072 26566 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.947593 26527 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.931176 26455 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.954094 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:01.957221 26455 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.927615 26538 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:01.932322 26263 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.897010 26549 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 23, Time since last communication: 0.012s
I20260709 12:18:01.935127 26205 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:18:01.869541 26385 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:01.942081 26464 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:01.945961 26543 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:01.868595 26420 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:18:01.852427 26448 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.949162 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:01.914300 26327 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.990820 13259 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:01.979681 26566 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:01.985960 26519 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:01.991328 12570 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:01.982620 26527 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:01.993201 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:01.988879 26508 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:01.960877 26455 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:01.979375 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:01.998991 26462 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:01.996928 26566 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:01.986956 26538 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.019s
I20260709 12:18:02.010203 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.011059 26448 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.034750 26549 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:02.035256 26549 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.006817 26508 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.049660 26549 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:02.049254 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2112) took 1862 ms. Trace:
I20260709 12:18:01.993100 26543 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.052202 26566 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:01.998481 26519 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.053730 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.054338 12570 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:02.072510 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:02.040630 26205 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:02.077528 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.078301 12643 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:02.078824 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.073736 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.079998 12645 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:18:02.069149 26544 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.882s	user 0.000s	sys 0.005s
I20260709 12:18:02.067683 12637 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:02.070986 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.052845 26560 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:18:02.070083 12489 rpcz_store.cc:276] 0709 12:18:00.186741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:00.186895 (+   154us) service_pool.cc:224] Handling call
0709 12:18:00.261294 (+ 74399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:00.403695 (+142401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:00.403746 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:00.454277 (+ 50531us) log.cc:844] Serialized 52 byte log entry
0709 12:18:01.731245 (+1276968us) spinlock_profiling.cc:243] Waited 959 us on lock 0x7b1800408520. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:18:01.861857 (+130612us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:01.861889 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:01.861946 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:02.048701 (+186755us) raft_consensus.cc:1685] finished
0709 12:18:02.048740 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:02.049133 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53165,"raft.queue_time_us":226887,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":7456,"spinlock_wait_cycles":2205568,"thread_start_us":218771,"threads_started":2,"wal-append.queue_time_us":1495216}
I20260709 12:18:02.073998 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:02.086701 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.074560 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:02.087915 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.062778 26455 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.037595 26462 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:02.083703 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.084826 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:02.084046 26530 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.022s	sys 0.027s
I20260709 12:18:02.083734 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.087486 13253 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:02.091511 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.085238 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.093163 12644 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:02.094489 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:02.085114 26513 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.093s	user 0.000s	sys 0.012s
I20260709 12:18:02.087284 26566 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:18:02.091049 26353 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:02.133265 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:02.114521 26530 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.023s	sys 0.027s
I20260709 12:18:02.138310 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.140646 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:02.091499 26580 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:02.136349 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:02.123853 26420 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:02.130662 26575 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:18:02.129582 26513 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.136s	user 0.000s	sys 0.013s
I20260709 12:18:02.084918 26205 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.133211 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.122664 26455 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:02.141872 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.135932 26453 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.157058 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:02.157126 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.138034 26462 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.27, Last appended by leader: 27, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.135761 26353 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.134028 26560 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.021s
I20260709 12:18:02.154996 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.152674 26513 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:02.142856 26305 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.158795 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.161201 12643 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:02.142251 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.132668 26263 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.167300 12644 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:02.163204 26353 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:02.154377 26420 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:02.174135 13257 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:02.169085 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.170629 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.172678 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.175784 12646 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:02.176170 12643 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:02.174877 26263 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.136058 26508 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.185042 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.185835 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:02.151163 26580 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.184062 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:02.162257 26453 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.179286 26560 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 38.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:18:02.190769 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:02.195106 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:02.195677 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.198832 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:02.200955 26543 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.201694 26566 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:18:02.202107 26580 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.202384 26263 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.202484 26566 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:02.204205 26566 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.204581 26566 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:02.211270 12645 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:18:02.212190 12638 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:18:02.189733 26565 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.190759 26305 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.182690 26485 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 23, Time since last communication: 0.028s
I20260709 12:18:02.191973 26595 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.192076 26455 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:02.217708 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.214619 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.213040 26543 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:02.210136 26508 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:02.226789 12645 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:18:02.219228 26305 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.235919 26305 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.240290 26525 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:02.240481 26560 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.219630 26565 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.241397 26560 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.241561 26525 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.242378 26539 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:02.242980 26539 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.243907 26539 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.244283 26525 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.244426 26542 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.245229 26542 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.247215 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.247759 26542 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.251407 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.252635 26560 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.248123 13257 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:02.248605 26557 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.253829 26557 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.256696 26557 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.249022 26453 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.225050 26543 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.262701 26543 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:02.250264 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.251180 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:02.265540 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.265568 12637 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:18:02.258745 26515 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:02.258523 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:02.259600 26579 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:02.223575 26420 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.252575 12646 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:02.275665 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:02.224406 26455 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.245148 26539 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:02.286775 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.290266 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.291539 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.280527 26515 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:02.279175 26305 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:18:02.306901 12645 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:02.292380 26575 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.018s
W20260709 12:18:02.314895 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5643) took 1484 ms (client timeout 1946 ms). Trace:
W20260709 12:18:02.315474 12572 rpcz_store.cc:269] 0709 12:18:00.830434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.015353 (+184919us) service_pool.cc:224] Handling call
0709 12:18:02.314593 (+1299240us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:02.314807 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1026162560}
I20260709 12:18:02.317062 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:02.318306 26517 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:02.318833 26517 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:02.319967 26517 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 41.30, Last appended by leader: 30, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.285528 26453 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:02.321858 26453 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.322296 26453 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:02.327003 26581 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:02.327595 26581 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:02.328585 26581 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.329133 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5630) took 1551 ms. Trace:
I20260709 12:18:02.329766 12565 rpcz_store.cc:276] 0709 12:18:00.777304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.279667 (+502363us) service_pool.cc:224] Handling call
0709 12:18:01.327590 (+ 47923us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:01.426428 (+ 98838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.328822 (+902394us) spinlock_profiling.cc:243] Waited 90.2 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:02.329056 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207409408}
I20260709 12:18:02.331211 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.331862 12565 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:02.332598 12566 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 23. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
I20260709 12:18:02.332923 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.334995 26581 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:02.339612 26416 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.340631 13262 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:02.225893 26513 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:02.342034 26241 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 40 current term: 40 is higher.
I20260709 12:18:02.343952 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.227084 26595 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.345474 26205 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.348054 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:02.352582 26168 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:02.353060 26168 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:02.305920 26525 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.040s
I20260709 12:18:02.355157 26516 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:02.357125 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.358661 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:02.359818 26241 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:02.360549 26241 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:02.361832 26491 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:02.362466 26491 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:02.363518 25676 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:02.363466 26491 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.298806 26455 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:02.367537 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5399) took 1410 ms. Trace:
I20260709 12:18:02.368093 12566 rpcz_store.cc:276] 0709 12:18:00.957423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.636789 (+679366us) service_pool.cc:224] Handling call
0709 12:18:02.332138 (+695349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:02.333287 (+  1149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.367268 (+ 33981us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54001b30d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:02.367453 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1567964928}
I20260709 12:18:02.345319 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.371501 12572 raft_consensus.cc:2376] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39. Re-sending same reply.
I20260709 12:18:02.371554 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.372535 13262 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:02.372575 12565 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:02.374150 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.375200 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:18:02.379930 12639 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:02.380330 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:02.381882 12569 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 22.
I20260709 12:18:02.353457 26168 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:02.385710 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.386245 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.386801 12646 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 25.
I20260709 12:18:02.388554 26353 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.389810 26517 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:02.392167 26464 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.394193 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.395350 12572 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 38 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 19.
I20260709 12:18:02.397060 12572 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:18:02.390115 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.400964 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:02.402671 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.347931 26595 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.405908 12570 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43.
I20260709 12:18:02.348219 26421 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.413857 12566 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:02.401150 26401 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.402341 26305 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:02.383994 26416 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.030s
I20260709 12:18:02.411484 26485 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:02.435353 26464 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:02.439567 26353 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.425777 26305 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:02.448890 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 1.06642e+06us
I20260709 12:18:02.449118 26352 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:18:02.453418 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:02.372452 26241 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.467608 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:02.460798 12564 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:18:02.425806 26401 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.459167 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5521) took 1351 ms. Trace:
I20260709 12:18:02.483238 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4127) took 1348 ms. Trace:
I20260709 12:18:02.444852 26549 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.483378 12497 rpcz_store.cc:276] 0709 12:18:01.108199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.108799 (+   600us) service_pool.cc:224] Handling call
0709 12:18:01.109228 (+   429us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:01.143450 (+ 34222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:01.143483 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:01.203449 (+ 59966us) log.cc:844] Serialized 52 byte log entry
0709 12:18:01.390309 (+186860us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:18:01.447813 (+ 57504us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.458691 (+1010878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.458775 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:02.458917 (+   142us) raft_consensus.cc:1685] finished
0709 12:18:02.458931 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:02.459098 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1044248,"prepare.queue_time_us":66178,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2292,"raft.queue_time_us":1046447,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":22962,"thread_start_us":139752,"threads_started":2,"wal-append.queue_time_us":313639}
I20260709 12:18:02.450361 26485 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.495808 12646 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:02.495842 12643 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:02.507815 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:02.455319 26551 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.421480 26421 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.059s
I20260709 12:18:02.478369 26305 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:02.521868 12645 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:02.522262 12640 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: [37.23-37.23]
I20260709 12:18:02.477748 26451 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:02.523243 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5386) took 2635 ms. Trace:
I20260709 12:18:02.488742 26330 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:02.528857 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:02.534909 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70407.9us
I20260709 12:18:02.509408 26485 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:02.529809 12571 rpcz_store.cc:276] 0709 12:17:59.888010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:59.891745 (+  3735us) service_pool.cc:224] Handling call
0709 12:17:59.982461 (+ 90716us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:01.093366 (+1110905us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.522933 (+1429567us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001d6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:02.523167 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2399111552}
I20260709 12:18:02.501072 26445 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:02.496160 26353 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.495914 26492 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:02.530332 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:02.543800 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:02.548243 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.549333 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.549521 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.504657 26446 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.034s
I20260709 12:18:02.504310 26566 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.555510 12494 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:02.554224 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:02.557798 12571 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:02.560235 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52531.2us
I20260709 12:18:02.560626 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5638) took 1759 ms. Trace:
I20260709 12:18:02.561028 12568 rpcz_store.cc:276] 0709 12:18:00.801034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.062970 (+261936us) service_pool.cc:224] Handling call
0709 12:18:02.560387 (+1497417us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x7b540017ff54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:02.560559 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128919424}
I20260709 12:18:02.562296 12494 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:02.563588 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.568465 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.569139 12490 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:02.574131 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.562417 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:02.574980 12493 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:02.543466 26595 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:02.568750 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.580288 12494 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:02.523067 26551 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.583367 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.583896 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:02.582037 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.585070 12492 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 21.
I20260709 12:18:02.584024 12497 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:02.587332 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.588045 12489 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:02.544466 26445 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.557193 26314 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:18:02.538304 26451 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.539621 26330 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.587777 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5395) took 1716 ms. Trace:
W20260709 12:18:02.607502 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.577944 26416 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.019s
W20260709 12:18:02.608712 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.598030 26401 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.576754 26579 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:02.603935 26543 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:02.609470 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:02.613729 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65972.3us
I20260709 12:18:02.579708 26595 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.552089 26353 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.616468 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:02.619167 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.607570 12563 rpcz_store.cc:276] 0709 12:18:00.871447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.571285 (+699838us) service_pool.cc:224] Handling call
0709 12:18:02.318059 (+746774us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:02.318592 (+   533us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:02.318629 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:02.319267 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:18:02.407772 (+ 88505us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:02.407836 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.407940 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:02.565760 (+157820us) raft_consensus.cc:1685] finished
0709 12:18:02.565785 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:02.566729 (+   944us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b54001b35d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:02.587699 (+ 20970us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4542,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":1715976576,"thread_start_us":87961,"threads_started":1,"wal-append.queue_time_us":150115}
I20260709 12:18:02.619992 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:02.613062 26446 consensus_queue.cc:1243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 44.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
W20260709 12:18:02.621007 26599 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:18:02.595580 26421 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.638629 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.639580 12492 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:02.638958 26421 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.627058 26551 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.622081 26451 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.623080 26314 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:02.626552 26566 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.627239 26298 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:02.626955 26496 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:02.654330 26298 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:02.653713 12387 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 34 to 38, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 38 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:02.654891 26298 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:02.655179 26496 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:02.655649 26496 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.656252 12387 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 34 to 46, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:02.628991 26330 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.657814 26565 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.658141 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.658463 26549 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.658490 26610 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:02.659972 26610 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.614670 26579 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.629194 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4122) took 1504 ms. Trace:
I20260709 12:18:02.666671 12642 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: [40.22-40.22]
I20260709 12:18:02.631790 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.667994 12493 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:02.555310 26492 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.630473 26595 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:02.640269 26445 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.24, Last appended by leader: 24, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.641064 26353 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.619956 12494 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
W20260709 12:18:02.682556 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66173.4us
I20260709 12:18:02.646189 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.646245 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:18:02.648242 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.685545 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.650784 26576 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:02.652066 12571 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:02.652630 26543 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.656677 26496 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.618590 26352 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.661396 12570 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 22. Preceding OpId from leader: term: 46 index: 23. (index mismatch)
I20260709 12:18:02.670090 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:02.658058 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43517.8us
I20260709 12:18:02.631464 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:02.693068 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.703778 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:02.664767 12496 rpcz_store.cc:276] 0709 12:18:01.133341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.573183 (+439842us) service_pool.cc:224] Handling call
0709 12:18:01.599531 (+ 26348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:01.606221 (+  6690us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:01.639170 (+ 32949us) log.cc:844] Serialized 14 byte log entry
0709 12:18:01.811007 (+171837us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b18003100c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:02.563305 (+752298us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.628641 (+ 65336us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:02.628786 (+   145us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:02.628859 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.628962 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:02.629108 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":710746,"prepare.queue_time_us":53961,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":115,"raft.queue_time_us":705059,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":15177,"spinlock_wait_cycles":13139200,"thread_start_us":111810,"threads_started":2,"wal-append.queue_time_us":332135}
I20260709 12:18:02.678893 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.678552 26453 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.709156 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.672202 26530 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:02.680092 26485 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.680505 26579 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.644137 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.717615 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.719200 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.661058 26610 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.721979 26508 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:18:02.722733 26508 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.707463 12570 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:18:02.705199 13262 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:02.723857 26508 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.712013 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.727950 12494 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:18:02.715344 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69200.1us
I20260709 12:18:02.733289 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.714012 12571 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:18:02.742719 26314 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:02.745502 26451 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:02.746995 26330 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:02.747525 26330 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.748925 26330 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.751291 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:02.752790 26553 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:02.753523 26553 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.755450 26553 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.763921 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.764487 12637 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:02.731931 12570 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 44 index: 25. (term mismatch)
I20260709 12:18:02.765717 12570 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 24
I20260709 12:18:02.764616 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:18:02.766101 12570 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.25
I20260709 12:18:02.766587 12570 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:02.765504 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:02.765048 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61077.1us
I20260709 12:18:02.662524 26305 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.689719 26539 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:02.690886 26421 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.767827 12496 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:02.774132 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.692444 26445 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:02.769311 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.779122 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.769644 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.696465 26576 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:02.776681 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.783768 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:02.693189 26492 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.787386 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:02.769938 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 213331us
W20260709 12:18:02.792517 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:02.794673 26614 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:18:02.795508 12493 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:02.796744 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.796631 26576 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.765018 26352 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.798625 26576 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:02.800150 26352 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.800985 12497 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:02.803828 12491 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:02.768748 26611 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:02.817597 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.821504 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:02.820667 26543 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.828743 26543 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.829500 26305 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.829617 12491 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 19. (term mismatch)
I20260709 12:18:02.830183 12491 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
I20260709 12:18:02.830399 12491 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 36.19
I20260709 12:18:02.830614 12491 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:02.831234 26543 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.831326 26556 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:02.832140 26556 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:02.834951 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:02.823987 26539 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.012s
I20260709 12:18:02.837610 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.837739 26352 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:02.838912 12643 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:02.839628 26352 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:02.840629 26352 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:02.841979 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.843254 26515 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:02.483749 12488 rpcz_store.cc:276] 0709 12:18:01.135017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.675700 (+540683us) service_pool.cc:224] Handling call
0709 12:18:01.685209 (+  9509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:01.685493 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:01.685963 (+   470us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.482728 (+796765us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.482890 (+   162us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:02.482932 (+    42us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:02.482977 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.483051 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:02.483179 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":689891,"prepare.queue_time_us":3865,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":5172,"thread_start_us":106234,"threads_started":1,"wal-append.queue_time_us":189456}
I20260709 12:18:02.844823 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.846920 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5519) took 1742 ms. Trace:
I20260709 12:18:02.847414 12495 rpcz_store.cc:276] 0709 12:18:01.104887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.106079 (+  1192us) service_pool.cc:224] Handling call
0709 12:18:01.121230 (+ 15151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:01.200710 (+ 79480us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:18:01.502970 (+302260us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.522747 (+1019777us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.576077 (+ 53330us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:02.576134 (+    57us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:02.576178 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.846643 (+270465us) spinlock_profiling.cc:243] Waited 79.2 ms on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:02.846690 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:02.846844 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":979062,"prepare.queue_time_us":194118,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":1242,"raft.queue_time_us":1032110,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":13604,"spinlock_wait_cycles":182629888,"thread_start_us":421529,"threads_started":3,"wal-append.queue_time_us":162195}
I20260709 12:18:02.849105 26485 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:02.850692 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.851426 26455 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.852059 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:02.852756 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:18:02.764585 26530 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.815666 26496 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:02.837858 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:02.854647 12570 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:18:02.855572 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.859594 12572 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:02.860293 12495 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:18:02.863098 26298 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.864760 12569 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:02.858903 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.841775 26352 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.875893 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:02.886642 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65247.4us
I20260709 12:18:02.887132 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.875778 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:02.889254 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51647.5us
I20260709 12:18:02.890908 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.891439 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.887933 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:02.867331 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5517) took 2534 ms. Trace:
I20260709 12:18:02.825249 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.892906 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:02.858122 26551 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:02.894474 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.904819 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.898486 12491 rpcz_store.cc:276] 0709 12:18:00.333064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:00.779398 (+446334us) service_pool.cc:224] Handling call
0709 12:18:00.779930 (+   532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:01.630912 (+850982us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b58003d79d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:18:02.780990 (+1150078us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b5800882bd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:18:02.831205 (+ 50215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.867161 (+ 35956us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060444800}
I20260709 12:18:02.908042 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:02.903381 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:02.901707 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:02.914215 12495 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:02.895195 26314 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:02.892906 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:02.917508 12460 connection.cc:256] RPC call timeout handler was delayed by -2.1366685370001051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31560000000000005s for a response
I20260709 12:18:02.898394 26415 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:02.904227 26305 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:02.890270 26549 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:02.891523 26538 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:02.919694 12488 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:18:02.897698 26416 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:02.914623 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.925348 13259 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
W20260709 12:18:02.915244 26168 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.510s	user 0.002s	sys 0.010s
I20260709 12:18:02.898376 26515 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:02.891741 26616 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:02.924712 26455 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:02.929101 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:02.911809 26551 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:02.913425 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:02.909485 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:03.006580 26611 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:03.004376 26530 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.015322 26549 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:02.962410 12495 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:03.016474 26549 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:03.017962 26549 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.019665 12495 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:03.021272 12493 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
W20260709 12:18:03.024238 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95269.5us
I20260709 12:18:02.909699 25694 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:18:03.035485 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 126113us
I20260709 12:18:03.038870 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.039831 12642 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:02.922276 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:03.043040 12491 raft_consensus.cc:606] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:03.044478 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.045080 12491 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
W20260709 12:18:03.033317 26620 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:18:02.913470 26539 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:02.922857 26314 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:02.939260 26352 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:03.007321 26551 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.048298 13217 leader_election.cc:400] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43.
W20260709 12:18:03.048816 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:03.050282 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135774us
I20260709 12:18:03.020694 26538 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:03.013855 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100546us
I20260709 12:18:03.064409 12567 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:18:03.050520 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:02.973325 26298 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.060571 12495 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 25. Preceding OpId from leader: term: 35 index: 26. (index mismatch)
I20260709 12:18:03.061299 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:03.060772 26314 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:03.066836 12495 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:18:03.066929 26521 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:03.047531 12491 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:18:03.051151 26611 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:02.938405 26305 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:03.034459 26416 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:03.070353 12491 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:03.065042 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:03.048839 26401 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:03.073017 12488 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:03.074014 12488 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:03.054419 26616 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:18:03.081131 12643 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:03.083259 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:03.085783 12495 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:18:03.048357 26168 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.093511 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:03.074191 26305 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:03.051083 26241 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.032s
I20260709 12:18:03.076112 26464 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:03.087546 26576 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.087747 12488 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.074196 26416 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:03.060105 26455 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.029s
I20260709 12:18:03.090436 26565 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:18:03.087673 26515 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.088445 26401 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:03.114341 26580 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:03.115640 26330 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:03.115859 26576 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.119902 12494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:03.113757 26263 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:03.130642 12494 raft_consensus.cc:606] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:03.131197 12494 raft_consensus.cc:614] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:18:03.127810 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65249.8us
I20260709 12:18:03.122522 12387 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 38. New cstate: current_term: 38 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:03.123826 12495 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:18:03.124234 12488 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:03.073210 26538 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.125944 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5420) took 1035 ms. Trace:
I20260709 12:18:03.139763 12563 rpcz_store.cc:276] 0709 12:18:02.090870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.632763 (+541893us) service_pool.cc:224] Handling call
0709 12:18:02.632985 (+   222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:02.633288 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:02.633305 (+    17us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:02.662109 (+ 28804us) log.cc:844] Serialized 26 byte log entry
0709 12:18:02.786544 (+124435us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:02.786590 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.786653 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:03.125563 (+338910us) raft_consensus.cc:1685] finished
0709 12:18:03.125605 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:03.125852 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2672,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"thread_start_us":107861,"threads_started":1,"wal-append.queue_time_us":247416}
I20260709 12:18:03.144855 12488 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:18:03.132195 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:03.101660 26513 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 21, Time since last communication: 0.040s
W20260709 12:18:03.195611 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63541.6us
I20260709 12:18:03.197327 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:03.151711 26263 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:03.200316 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.202195 12494 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:03.203601 12494 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:18:03.204015 12494 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:03.203713 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.128620 26515 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.126847 26495 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 18, Time since last communication: 0.070s
I20260709 12:18:03.127717 26595 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.216054 12495 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
W20260709 12:18:03.217231 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.228863 26515 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:03.230254 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 156479us
W20260709 12:18:03.228971 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.209882 26611 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:03.209514 26579 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:03.222493 12494 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.214669 26241 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:18:03.245519 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.215929 26495 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 32.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:18:03.232985 12643 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:03.247634 12494 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:18:03.249027 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47278.7us
W20260709 12:18:03.258450 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46971.9us
I20260709 12:18:03.223863 26421 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.031s
I20260709 12:18:03.261505 12638 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:03.262609 26631 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:03.258082 26538 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.222452 26464 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.031s
I20260709 12:18:03.260294 26263 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.270560 26633 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:03.271284 26595 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:03.246176 26401 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.250818 26611 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:03.274722 26627 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
W20260709 12:18:03.275200 26523 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 44 current term: 44 is higher.
I20260709 12:18:03.264125 26630 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:18:03.286795 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84731.6us
I20260709 12:18:03.300999 12494 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:18:03.297195 12492 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
W20260709 12:18:03.324890 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69765.6us
W20260709 12:18:03.327843 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96693.7us
I20260709 12:18:03.275205 26168 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:03.346040 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.352937 12570 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:03.356099 12495 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:03.356835 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:03.356860 12571 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:03.357714 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:03.359184 12495 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:03.359360 12572 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:03.356292 12563 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:03.324599 26633 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:03.373934 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 40 committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:18:03.375228 12380 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48109.6us
I20260709 12:18:03.338011 26638 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.049s
I20260709 12:18:03.378481 26633 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:03.380797 12495 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:18:03.383567 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5405) took 1620 ms (client timeout 1601 ms). Trace:
W20260709 12:18:03.384223 12564 rpcz_store.cc:269] 0709 12:18:01.763033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:01.763245 (+   212us) service_pool.cc:224] Handling call
0709 12:18:03.383284 (+1620039us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54001a31d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:03.383452 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1541402496}
I20260709 12:18:03.278839 26305 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.389581 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5240) took 1317 ms. Trace:
I20260709 12:18:03.390275 12640 rpcz_store.cc:276] 0709 12:18:02.071914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.072069 (+   155us) service_pool.cc:224] Handling call
0709 12:18:02.087228 (+ 15159us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:02.526077 (+438849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:03.385176 (+859099us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:03.389512 (+  4336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993430912}
W20260709 12:18:03.398646 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.391960 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:03.401149 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.399377 12571 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:18:03.335583 26549 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.061s
I20260709 12:18:03.401444 12564 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:18:03.307886 26634 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.085s
W20260709 12:18:03.452000 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86025.6us
I20260709 12:18:03.463817 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:03.464670 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:03.466868 12640 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:18:03.468734 26611 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:03.363729 26631 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.049s
I20260709 12:18:03.471266 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:03.471928 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.470839 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.474654 13262 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:03.472429 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:03.479440 12571 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 42 because replica has last-logged OpId of term: 40 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:18:03.400713 26576 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:03.361872 26595 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.089s
I20260709 12:18:03.390290 26421 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.069s
I20260709 12:18:03.496155 12570 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:03.497988 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.501228 12571 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:03.470085 26241 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:03.482702 26455 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.511641 12564 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:03.455991 26314 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.513535 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.514487 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:03.515720 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.516860 12564 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:03.518124 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.519052 12571 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:18:03.469707 26563 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.044s
I20260709 12:18:03.522279 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.526216 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:03.508713 26648 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.509930 26464 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:03.513785 26241 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.547389 26421 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:18:03.560050 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87674us
I20260709 12:18:03.522948 26630 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.035s
I20260709 12:18:03.561260 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.528903 26637 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:03.582762 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:03.589262 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:03.553267 26263 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:03.523145 26455 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.591998 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.573526 26644 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.598868 13253 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:18:03.602620 13253 raft_consensus.cc:606] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:03.603138 13253 raft_consensus.cc:614] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:03.596105 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.582566 26464 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:03.606096 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:03.607347 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86128.9us
I20260709 12:18:03.596897 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.608783 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:03.608884 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.607666 12570 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:03.594135 26263 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:03.610235 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.613231 13257 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:18:03.596408 26637 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:03.615787 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.617130 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.576004 26648 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.620358 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:18:03.620244 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:03.621251 13262 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:03.621805 12564 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:03.622840 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.624305 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.615410 26634 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:03.624940 26634 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:03.625728 12564 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:18:03.615011 26263 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:03.628429 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:03.630504 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:03.617147 26305 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:03.632917 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.634498 12571 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:03.634394 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.636018 12570 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:03.625388 26634 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:03.638679 26633 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.639889 26314 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:03.625725 26645 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 12:18:03.647063 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.602082 26644 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:03.633127 26330 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.002s
I20260709 12:18:03.658891 12642 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:18:03.653786 12645 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:18:03.663868 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:03.664662 12642 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:03.652537 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.666100 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:03.668146 12645 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:03.674134 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.675552 12638 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:03.677767 26464 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.668721 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:03.683604 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.684358 26241 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.685864 26648 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.687695 26644 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:03.662168 26642 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.099s	user 0.000s	sys 0.002s
I20260709 12:18:03.690686 26455 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:03.693848 12564 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:03.695132 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.696786 26556 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.659096 26576 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.698190 26576 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.700737 26576 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.693226 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:03.701468 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.702317 12572 raft_consensus.cc:2368] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43.
I20260709 12:18:03.676930 26637 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:03.703302 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.703927 12572 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
W20260709 12:18:03.704995 26352 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.642s	user 0.012s	sys 0.001s
I20260709 12:18:03.706008 12489 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:03.708899 25694 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:03.709475 25694 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:03.709906 26495 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:03.710132 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5417) took 1700 ms. Trace:
I20260709 12:18:03.710384 26495 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:03.711481 26495 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 46.23, Last appended by leader: 23, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.710744 12566 rpcz_store.cc:276] 0709 12:18:02.010344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.421432 (+411088us) service_pool.cc:224] Handling call
0709 12:18:02.425828 (+  4396us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:02.435137 (+  9309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:02.435180 (+    43us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:02.436019 (+   839us) log.cc:844] Serialized 78 byte log entry
0709 12:18:02.518478 (+ 82459us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:02.900036 (+381558us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.908210 (+  8174us) log.cc:844] Serialized 14 byte log entry
0709 12:18:02.921752 (+ 13542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.921898 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:03.701926 (+780028us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b1800339760. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:03.709799 (+  7873us) raft_consensus.cc:1685] finished
0709 12:18:03.709825 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:03.710036 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7425,"prepare.queue_time_us":100539,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":4457,"raft.queue_time_us":29045,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":16355,"spinlock_wait_cycles":36482304,"thread_start_us":435761,"threads_started":2,"wal-append.queue_time_us":178854}
I20260709 12:18:03.713063 26495 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:03.713444 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.714354 12566 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:03.719739 12490 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:03.722306 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:03.722676 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:03.723078 12496 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:03.726078 26513 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:18:03.726183 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.727064 12637 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:03.727687 26513 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:03.728204 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.728042 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.728996 12637 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:03.729519 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:03.730690 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.730965 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5415) took 1729 ms. Trace:
I20260709 12:18:03.732051 12646 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:18:03.731429 12565 rpcz_store.cc:276] 0709 12:18:02.001938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.403601 (+401663us) service_pool.cc:224] Handling call
0709 12:18:02.561857 (+158256us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:02.574450 (+ 12593us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:18:02.574485 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:02.587734 (+ 13249us) log.cc:844] Serialized 26 byte log entry
0709 12:18:02.784969 (+197235us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:18:02.785015 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:02.785090 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:02.855569 (+ 70479us) raft_consensus.cc:1685] finished
0709 12:18:02.855589 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:03.730678 (+875089us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540017ff50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:03.730892 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12061,"raft.queue_time_us":16148,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":10048,"spinlock_wait_cycles":343182720,"thread_start_us":177792,"threads_started":1,"wal-append.queue_time_us":226695}
W20260709 12:18:03.730378 26416 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.000s
I20260709 12:18:03.729867 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:03.734118 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.734941 26633 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:03.737178 26385 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:03.729969 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:03.738550 26565 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:18:03.753829 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61680.2us
I20260709 12:18:03.741423 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.762351 12641 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:03.762789 12571 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:18:03.741796 26649 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:03.742787 26506 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20260709 12:18:03.744112 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:03.771842 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:03.772279 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.729823 26595 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:03.744669 26187 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:03.747148 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.704849 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:03.749189 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:03.750312 26515 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:03.751715 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 137253us
W20260709 12:18:03.752470 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 124107us
W20260709 12:18:03.790681 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.790580 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:03.751312 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.755398 12644 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:03.760161 12564 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:03.792739 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:03.755059 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:03.768395 12642 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:03.770411 12565 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:18:03.744577 26523 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:03.777459 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:03.779234 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 315480us
I20260709 12:18:03.802443 12646 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.29-42.29]
I20260709 12:18:03.802985 12638 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:03.751202 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:03.792934 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:03.804814 12564 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:03.806322 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:03.760619 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:03.808157 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53244.4us
I20260709 12:18:03.762204 26565 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:03.806438 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:03.801246 12497 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:03.774055 26556 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:03.825481 26556 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:03.827164 26556 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:03.828800 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:03.777326 26576 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.811436 12572 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:03.851424 12572 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: [41.21-41.21]
I20260709 12:18:03.811671 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:03.806337 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.810353 26506 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:03.810674 26523 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.766461 26416 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.814225 26187 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:03.814079 26565 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.815968 26644 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:18:04.003088 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:04.004012 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.810717 26611 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:03.817329 26580 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:18:03.818600 12496 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:18:03.818684 12565 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:03.821969 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:18:03.823077 12571 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
W20260709 12:18:03.830472 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79442.5us
W20260709 12:18:03.834189 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:03.809545 12564 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:18:04.009720 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.010788 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:04.012001 12564 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:18:03.835776 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:03.843351 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:03.783653 26445 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:03.777406 26595 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:03.849480 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:03.853600 12572 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:03.778864 26263 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:03.786641 26531 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.022526 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:04.030625 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 181289us
I20260709 12:18:04.033762 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:04.036808 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:04.038552 12460 leader_election.cc:400] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43.
I20260709 12:18:04.039259 12460 leader_election.cc:403] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:04.040838 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:04.047176 12637 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:04.071797 12644 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:03.937850 26506 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:04.095559 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:04.043656 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 208004us
I20260709 12:18:03.938853 26579 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:04.057997 12565 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:18:03.951704 26330 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:04.128067 12496 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:03.807313 26515 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.131661 12565 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:04.148552 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:04.154413 12496 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:18:04.001194 26644 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:04.153210 12643 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:04.054570 26551 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:04.054821 26508 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:04.043022 26298 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 43
I20260709 12:18:04.046001 26384 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:04.178624 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:04.045096 26352 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:18:04.123767 26314 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:03.950891 26576 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.167317 26641 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:04.170636 26644 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:04.196007 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:04.043098 26445 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.178509 26565 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:04.200278 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 159502us
I20260709 12:18:04.096012 26531 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.188534 12565 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.058308 26539 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.131s
I20260709 12:18:04.203756 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:04.218384 12565 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:04.220588 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:04.221148 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:04.189103 26330 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:04.189177 26643 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:04.219798 26508 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:04.182526 26305 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.035s
I20260709 12:18:04.206882 26187 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:04.192461 26663 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.232630 12570 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 42 index: 23. (term mismatch)
I20260709 12:18:04.236656 12460 leader_election.cc:400] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
I20260709 12:18:04.240983 12460 leader_election.cc:403] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:04.198731 26641 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:04.203015 26654 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:04.240841 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.258585 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:04.240813 12570 pending_rounds.cc:85] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 22
I20260709 12:18:04.264345 12570 pending_rounds.cc:107] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.23
I20260709 12:18:04.264842 12570 raft_consensus.cc:2889] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:04.225747 26330 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:04.272714 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.202096 26314 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:04.278477 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.252003 26549 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:04.289997 26668 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.290819 26668 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.295432 26576 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.296725 26576 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.298647 26263 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:04.299858 26508 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.306107 26666 raft_consensus.cc:476] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Not starting pre-election -- already a leader
I20260709 12:18:04.307108 26666 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:04.307799 26666 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:04.308879 26495 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.595s	user 0.011s	sys 0.000s
I20260709 12:18:04.232102 26506 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 42.29, Last appended by leader: 29, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.318225 26506 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:04.320322 26496 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.320937 26496 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.310813 26445 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.317380 26540 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:18:04.322842 26519 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:04.324596 26660 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.002s
W20260709 12:18:04.327342 26656 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:18:04.250654 26663 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.191092 26595 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.271873 26330 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:04.334554 12565 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 20. (term mismatch)
I20260709 12:18:04.334945 26515 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.339200 26523 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:04.312951 26642 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:18:04.310914 26539 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:18:04.273593 26634 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.615s	user 0.013s	sys 0.004s
I20260709 12:18:04.314185 26671 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:04.360283 25694 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.650s	user 0.016s	sys 0.001s
I20260709 12:18:04.232515 26643 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.364383 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:04.360819 26579 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:04.350245 26549 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:04.369043 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:04.253362 26641 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:04.364463 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:04.402580 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.407949 12637 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
W20260709 12:18:04.408892 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.368484 12387 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 37 to 39. New cstate: current_term: 39 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:04.386834 12496 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:04.334951 12565 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:18:04.413230 12565 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:18:04.413549 12565 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:04.418223 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.363412 26654 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:04.364715 26523 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.428467 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:04.368712 26496 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.410051 26611 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:04.407061 26519 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.414547 25694 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.410444 26579 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:04.432307 26666 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.297443 26576 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.432621 26539 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:04.437837 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.381055 26668 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:04.442993 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5544) took 1625 ms (client timeout 2053 ms). Trace:
W20260709 12:18:04.519116 12489 rpcz_store.cc:269] 0709 12:18:02.817202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.821553 (+  4351us) service_pool.cc:224] Handling call
0709 12:18:04.442694 (+1621141us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:04.442892 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1985003776}
I20260709 12:18:04.444173 12495 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:18:04.466908 26644 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.446062 26549 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.406431 26671 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.369496 26616 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:04.451562 26453 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:04.452451 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5429) took 2069 ms (client timeout 2054 ms). Trace:
I20260709 12:18:04.455000 26654 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:04.457731 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:04.458829 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:04.458936 26298 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 42. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43.
W20260709 12:18:04.529945 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.460180 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:04.460600 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5533) took 1780 ms (client timeout 1756 ms). Trace:
I20260709 12:18:04.462586 13257 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:18:04.463434 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:04.464502 26513 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.736s	user 0.011s	sys 0.004s
I20260709 12:18:04.468150 26531 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.369644 26634 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.479360 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:04.489821 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:04.538831 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:04.373181 26556 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.000s
I20260709 12:18:04.540000 13256 raft_consensus.cc:2368] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47.
I20260709 12:18:04.540725 13253 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:04.518945 26637 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.350435 26515 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:04.369436 26416 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:04.510783 12570 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:04.411651 26330 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.414578 26551 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40.
W20260709 12:18:04.525074 12569 rpcz_store.cc:269] 0709 12:18:02.382888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.385968 (+  3080us) service_pool.cc:224] Handling call
0709 12:18:04.452179 (+2066211us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:04.452363 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1073432832}
W20260709 12:18:04.529170 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71569.9us
W20260709 12:18:04.531196 12492 rpcz_store.cc:269] 0709 12:18:02.680629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.684330 (+  3701us) service_pool.cc:224] Handling call
0709 12:18:04.460332 (+1776002us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:04.460492 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1309259520}
I20260709 12:18:04.631100 12639 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: []
W20260709 12:18:04.415841 26657 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.291s	user 0.001s	sys 0.003s
W20260709 12:18:04.548770 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.548830 12565 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.29-42.29]
I20260709 12:18:04.541500 26513 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:04.504344 26523 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.525571 26420 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:18:04.563702 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:04.657070 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:04.659338 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.565454 26676 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:04.568166 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 309669us
W20260709 12:18:04.568799 26638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.043s	user 0.031s	sys 0.035s
I20260709 12:18:04.504825 26519 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.617367 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2133) took 1144 ms. Trace:
I20260709 12:18:04.618754 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:04.673218 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.524673 26616 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:04.570325 26633 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.091s
I20260709 12:18:04.625732 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5435) took 1249 ms. Trace:
I20260709 12:18:04.533960 26496 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:04.632064 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:04.632987 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:04.680704 12494 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:04.633947 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:04.683488 12492 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
W20260709 12:18:04.630455 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 410543us
I20260709 12:18:04.425235 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:04.684854 12496 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:04.521889 26539 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.684398 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:04.653405 12493 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:04.620755 26531 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.672245 12491 rpcz_store.cc:276] 0709 12:18:03.472586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.472793 (+   207us) service_pool.cc:224] Handling call
0709 12:18:03.473253 (+   460us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:18:03.741824 (+268571us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:03.741859 (+    35us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:18:03.742790 (+   931us) log.cc:844] Serialized 104 byte log entry
0709 12:18:04.147532 (+404742us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:04.550568 (+403036us) log.cc:844] Serialized 14 byte log entry
0709 12:18:04.551869 (+  1301us) log.cc:844] Serialized 14 byte log entry
0709 12:18:04.616816 (+ 64947us) log.cc:844] Serialized 14 byte log entry
0709 12:18:04.617068 (+   252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.617142 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:04.617183 (+    41us) raft_consensus.cc:1685] finished
0709 12:18:04.617192 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:04.617302 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":328509,"prepare.queue_time_us":433348,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":119,"raft.queue_time_us":335079,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":73653,"spinlock_wait_cycles":31616,"thread_start_us":769574,"threads_started":2,"wal-append.queue_time_us":608124}
I20260709 12:18:04.629781 26515 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.555877 26556 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.677525 12566 rpcz_store.cc:276] 0709 12:18:03.376740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.715453 (+338713us) service_pool.cc:224] Handling call
0709 12:18:03.715827 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:04.042866 (+327039us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:04.042900 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:04.062510 (+ 19610us) log.cc:844] Serialized 26 byte log entry
0709 12:18:04.613157 (+550647us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:04.613219 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.614463 (+  1244us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:04.625317 (+ 10854us) raft_consensus.cc:1685] finished
0709 12:18:04.625342 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:04.625646 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47215,"raft.queue_time_us":47339,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7694,"thread_start_us":702205,"threads_started":2,"wal-append.queue_time_us":521829}
W20260709 12:18:04.687167 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55173.6us
I20260709 12:18:04.702622 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:04.626793 26637 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.681394 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:04.656493 26420 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:04.504499 26641 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.663506 26668 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:04.670209 26638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.148s	user 0.032s	sys 0.036s
I20260709 12:18:04.690164 26455 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:04.679800 26496 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:04.707285 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:04.697750 26670 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.708674 12494 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:04.703884 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:04.690052 26543 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.630389 26611 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 45.22, Last appended by leader: 22, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.697034 26508 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:04.726886 12497 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:04.677800 26298 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.029s
I20260709 12:18:04.728201 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:04.805750 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77543.9us
I20260709 12:18:04.707261 12536 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:04.708534 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:04.671618 26678 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.038s
I20260709 12:18:04.731436 13259 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:04.703106 26305 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:18:04.777012 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73228.6us
I20260709 12:18:04.778257 12564 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:04.754828 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:04.733136 12492 pending_rounds.cc:85] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 18
I20260709 12:18:04.735775 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.741457 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.672928 26676 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.775880 26501 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:18:04.820533 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5421) took 2681 ms. Trace:
I20260709 12:18:04.821023 12567 rpcz_store.cc:276] 0709 12:18:02.138950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.651621 (+512671us) service_pool.cc:224] Handling call
0709 12:18:02.652047 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:03.064886 (+412839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.820246 (+1755360us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:04.820464 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":947411840}
I20260709 12:18:04.781379 26420 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.782727 26523 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.711758 26451 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.099s
I20260709 12:18:04.784126 12565 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:04.832203 12565 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:04.784861 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.701174 26531 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.690312 26654 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.793462 26330 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.732693 26455 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:04.829203 26187 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.000s
I20260709 12:18:04.809669 26669 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:18:04.807893 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 100713us
I20260709 12:18:04.713100 26643 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.730908 26672 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:18:04.817121 12572 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:04.866062 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:04.816931 12570 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:04.871133 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:04.824674 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4147) took 1035 ms. Trace:
W20260709 12:18:04.783159 26689 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:18:04.834626 12571 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 23.
I20260709 12:18:04.833391 12565 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.794911 26641 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.799798 26634 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.871579 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:04.788789 26610 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.730346 26496 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:04.878476 26610 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.798835 26556 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:04.883481 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.798198 26644 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.786551 26543 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.788177 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.809553 25694 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.815409 12492 pending_rounds.cc:107] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 32.19
I20260709 12:18:04.712218 26666 raft_consensus.cc:2764] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:18:04.787876 26683 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:04.911542 26683 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:04.809757 26664 raft_consensus.cc:2764] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:18:04.807143 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4152) took 1007 ms. Trace:
I20260709 12:18:04.810958 26671 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.730819 26668 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.918093 26455 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:04.918756 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:04.919124 26668 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.875977 26637 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:04.854172 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 217687us
I20260709 12:18:04.777604 26670 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.930395 26670 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.838464 26305 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:04.932433 26677 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.020s
I20260709 12:18:04.873875 12495 rpcz_store.cc:276] 0709 12:18:03.789431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.789673 (+   242us) service_pool.cc:224] Handling call
0709 12:18:04.443597 (+653924us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:04.520313 (+ 76716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.824409 (+304096us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b540037df50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:04.824606 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1502981376}
I20260709 12:18:04.876051 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2705) took 2807 ms. Trace:
I20260709 12:18:04.848426 26187 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:04.882385 26610 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.890810 26694 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:04.778905 26616 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.898391 26515 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.944940 12563 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:04.898725 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:04.945606 26538 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:18:04.946099 26538 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:04.947139 26538 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:04.836580 26168 consensus_queue.cc:1175] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 36 status { last_received { term: 35 index: 19 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:18:04.899538 26579 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:04.838014 26675 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.031s
I20260709 12:18:04.900847 26676 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:04.901610 26672 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:18:04.903357 26690 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:18:04.903427 26531 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:04.907014 12566 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 37 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 18.
I20260709 12:18:04.851075 26580 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 18, Time since last communication: 0.053s
I20260709 12:18:04.913244 12492 raft_consensus.cc:2917] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Aborting config change with OpId 32.19: Aborted: Op aborted by new leader
I20260709 12:18:04.964113 12492 raft_consensus.cc:2889] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:04.914548 12496 rpcz_store.cc:276] 0709 12:18:03.799670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.424770 (+625100us) service_pool.cc:224] Handling call
0709 12:18:04.807055 (+382285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:04.916782 12564 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:18:04.849743 13259 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:04.922798 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:04.926467 26453 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.927474 26693 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.932135 26305 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.967902 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.968928 12570 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:04.863276 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:04.970700 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161426us
I20260709 12:18:04.970850 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.936987 12568 rpcz_store.cc:276] 0709 12:18:02.068499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.606118 (+537619us) service_pool.cc:224] Handling call
0709 12:18:02.606477 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:02.641910 (+ 35433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:02.641956 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:02.650095 (+  8139us) log.cc:844] Serialized 26 byte log entry
0709 12:18:02.789674 (+139579us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:02.789730 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.864703 (+2074973us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:04.864794 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:04.864882 (+    88us) raft_consensus.cc:1685] finished
0709 12:18:04.864901 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:04.875920 (+ 11019us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3718,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"spinlock_wait_cycles":80346496,"thread_start_us":138964,"threads_started":1,"wal-append.queue_time_us":211894,"wal-append.run_cpu_time_us":2046,"wal-append.run_wall_time_us":1002319}
I20260709 12:18:04.970983 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:04.973480 12569 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:04.963271 26420 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:04.913862 26556 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:04.942822 26694 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:04.972458 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:04.948395 26697 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:04.950413 12637 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:04.947933 12572 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:18:04.998572 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 132646us
I20260709 12:18:05.003284 12536 leader_election.cc:400] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:18:05.003644 12536 leader_election.cc:403] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:04.946107 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:05.004930 12563 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:05.005030 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:18:05.005990 12567 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:04.948354 26538 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:18:04.974485 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:04.991556 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155974us
I20260709 12:18:05.007871 12569 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:04.943886 26491 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:18:05.008880 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:05.010306 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:04.972290 26580 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 37.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:18:05.014928 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:04.967530 26693 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.002923 26556 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:05.002480 26314 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:05.024104 26579 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:05.014374 26531 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40.
I20260709 12:18:05.001926 26697 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:05.042789 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60516 (request call id 0) took 42590 ms. Trace:
I20260709 12:18:05.044689 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5563) took 1251 ms. Trace:
I20260709 12:18:05.045799 12491 rpcz_store.cc:276] 0709 12:18:03.793236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.696497 (+903261us) service_pool.cc:224] Handling call
0709 12:18:04.696813 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:04.697226 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:04.697247 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:04.699202 (+  1955us) log.cc:844] Serialized 26 byte log entry
0709 12:18:04.991028 (+291826us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:04.991097 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.991219 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:05.044393 (+ 53174us) raft_consensus.cc:1685] finished
0709 12:18:05.044418 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.044586 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15322,"raft.queue_time_us":121,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":7758,"thread_start_us":260070,"threads_started":1,"wal-append.queue_time_us":256826}
I20260709 12:18:05.004698 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:05.051170 26603 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.397s	user 0.004s	sys 0.011s
I20260709 12:18:05.026636 26187 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:18:04.999469 26453 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:05.064715 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:05.039779 26556 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:05.053301 12492 rpcz_store.cc:276] 0709 12:17:22.447106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:22.447320 (+   214us) service_pool.cc:224] Handling call
0709 12:17:22.758613 (+311293us) log.cc:844] Serialized 506 byte log entry
0709 12:17:22.934403 (+175790us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b18001478e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:23.630112 (+695709us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b18001478e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:17:23.644012 (+ 13900us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:05.026141 (+41382129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1907,"raft.queue_time_us":2733,"raft.run_cpu_time_us":1759,"raft.run_wall_time_us":29336,"spinlock_wait_cycles":22998400,"thread_start_us":125069,"threads_started":1,"wal-append.queue_time_us":434532,"wal-append.run_cpu_time_us":7024,"wal-append.run_wall_time_us":1519341}
I20260709 12:18:05.086324 26697 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.091498 26697 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:05.010478 26702 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.076s
W20260709 12:18:05.093779 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83679.5us
I20260709 12:18:05.094796 12569 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:05.107141 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:05.050082 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45456.4us
I20260709 12:18:05.109421 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:05.057192 26515 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:05.118005 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:05.000069 26683 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.132164 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:05.078636 26187 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:05.140552 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:05.141659 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:05.152374 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:05.154943 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:05.156240 12638 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:05.151839 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:05.128404 26531 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:18:05.164601 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:05.193562 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:05.198290 12640 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:05.128454 26515 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:05.204649 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5565) took 1410 ms. Trace:
I20260709 12:18:05.205255 12494 rpcz_store.cc:276] 0709 12:18:03.793799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.785413 (+991614us) service_pool.cc:224] Handling call
0709 12:18:04.785869 (+   456us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:05.022085 (+236216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.022117 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:05.028799 (+  6682us) log.cc:844] Serialized 78 byte log entry
0709 12:18:05.124266 (+ 95467us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:05.203125 (+ 78859us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.204144 (+  1019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.204240 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:05.204280 (+    40us) raft_consensus.cc:1685] finished
0709 12:18:05.204290 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.204571 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46478,"raft.queue_time_us":75614,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":12334,"thread_start_us":324812,"threads_started":3,"wal-append.queue_time_us":81390}
W20260709 12:18:05.247979 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:05.154788 26672 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:05.128697 26610 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.073s
I20260709 12:18:05.161705 26543 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:05.148154 26694 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:05.189502 26495 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:05.135468 26676 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.070s
I20260709 12:18:05.202545 26515 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:05.279112 26701 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.564s
I20260709 12:18:05.451840 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5566) took 1657 ms. Trace:
W20260709 12:18:05.547271 26508 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.739s	user 0.012s	sys 0.003s
I20260709 12:18:05.580928 26508 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:18:05.537418 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:05.278904 26698 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:05.447882 26453 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:05.426476 26693 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:05.279906 26704 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:18:05.453331 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4150) took 1654 ms (client timeout 1951 ms). Trace:
I20260709 12:18:05.585757 26704 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:05.586481 26704 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:05.586349 12493 rpcz_store.cc:269] 0709 12:18:03.798976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.351981 (+553005us) service_pool.cc:224] Handling call
0709 12:18:05.453081 (+1101100us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:05.453248 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295488896}
W20260709 12:18:05.279671 26706 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 12:18:05.588909 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5567) took 1794 ms. Trace:
I20260709 12:18:05.589399 12495 rpcz_store.cc:276] 0709 12:18:03.794306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.934308 (+1140002us) service_pool.cc:224] Handling call
0709 12:18:04.934874 (+   566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:04.935412 (+   538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:04.935457 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:04.936108 (+   651us) log.cc:844] Serialized 26 byte log entry
0709 12:18:05.007928 (+ 71820us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:05.007970 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.008039 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:05.588702 (+580663us) raft_consensus.cc:1685] finished
0709 12:18:05.588722 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.588843 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5559,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":9997,"thread_start_us":71074,"threads_started":1,"wal-append.queue_time_us":100783}
I20260709 12:18:05.590122 26672 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.588909 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5572) took 1793 ms. Trace:
I20260709 12:18:05.437676 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2013) took 1690 ms. Trace:
I20260709 12:18:05.241714 26670 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:05.610044 26670 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:05.609606 12646 rpcz_store.cc:276] 0709 12:18:03.747733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.748059 (+   326us) service_pool.cc:224] Handling call
0709 12:18:03.794952 (+ 46893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:03.807358 (+ 12406us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.437256 (+1629898us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b54003c0350. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:05.437546 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74659200}
I20260709 12:18:05.596410 26611 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:05.597594 26669 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:05.599908 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5568) took 1805 ms. Trace:
I20260709 12:18:05.599828 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:05.656780 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57089.7us
W20260709 12:18:05.322067 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:05.583979 26698 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:05.572911 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.000s	sys 0.010s
W20260709 12:18:05.667327 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.000s	sys 0.011s
I20260709 12:18:05.599926 26543 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.013s
I20260709 12:18:05.419056 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:05.580315 12488 rpcz_store.cc:276] 0709 12:18:03.794072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.828839 (+1034767us) service_pool.cc:224] Handling call
0709 12:18:04.829255 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:04.972446 (+143191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:04.972478 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:04.972968 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:18:05.144045 (+171077us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:05.144108 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.148301 (+  4193us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:05.451384 (+303083us) raft_consensus.cc:1685] finished
0709 12:18:05.451414 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.451743 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7453,"raft.queue_time_us":59602,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":252647,"threads_started":2,"wal-append.queue_time_us":438733}
I20260709 12:18:05.668726 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:05.199923 26664 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:18:05.672564 12571 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.016s	sys 0.015s
W20260709 12:18:05.673087 12571 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.016s	sys 0.015s
I20260709 12:18:05.675977 12488 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
W20260709 12:18:05.649688 26696 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.598s	user 0.002s	sys 0.000s
I20260709 12:18:05.683238 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:05.635077 26672 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:05.648257 26420 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.674s	user 0.009s	sys 0.002s
I20260709 12:18:05.611204 26670 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.26, Last appended by leader: 26, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.655803 12496 rpcz_store.cc:276] 0709 12:18:03.794546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.964972 (+1170426us) service_pool.cc:224] Handling call
0709 12:18:04.965437 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:04.965748 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:05.104485 (+138737us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.599365 (+494880us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:05.599468 (+   103us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:05.599548 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.599671 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.599831 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38026,"prepare.queue_time_us":230565,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":32686,"raft.queue_time_us":39344,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":545813,"threads_started":2}
I20260709 12:18:05.696076 12496 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:05.696983 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2137) took 1866 ms. Trace:
I20260709 12:18:05.600304 26495 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.677112 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2136) took 1868 ms. Trace:
I20260709 12:18:05.700539 12488 rpcz_store.cc:276] 0709 12:18:03.808290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.675308 (+1867018us) service_pool.cc:224] Handling call
0709 12:18:05.675771 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.676798 (+  1027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.677045 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:05.702522 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2138) took 1764 ms. Trace:
W20260709 12:18:05.679029 12566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.003s	sys 0.024s
I20260709 12:18:05.702661 12568 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:18:05.703922 12488 rpcz_store.cc:276] 0709 12:18:03.938468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.701523 (+1763055us) service_pool.cc:224] Handling call
0709 12:18:05.701913 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.702126 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.702162 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:05.702210 (+    48us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:05.702245 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.702306 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.702460 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:05.680761 12493 rpcz_store.cc:276] 0709 12:18:03.795664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.587973 (+1792309us) service_pool.cc:224] Handling call
0709 12:18:05.588379 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.588578 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.588609 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:05.588656 (+    47us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:05.588688 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.588736 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.588843 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:05.697304 12496 rpcz_store.cc:276] 0709 12:18:03.830636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.695434 (+1864798us) service_pool.cc:224] Handling call
0709 12:18:05.695864 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.696651 (+   787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.696922 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:05.707074 12488 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20260709 12:18:05.708283 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2141) took 1694 ms. Trace:
I20260709 12:18:05.708837 12488 rpcz_store.cc:276] 0709 12:18:04.014225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.705226 (+1691001us) service_pool.cc:224] Handling call
0709 12:18:05.706413 (+  1187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.707916 (+  1503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.708208 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:05.703512 12566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.004s	sys 0.025s
I20260709 12:18:05.612864 12494 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:05.711347 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5571) took 1916 ms. Trace:
I20260709 12:18:05.711951 12494 rpcz_store.cc:276] 0709 12:18:03.795343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.251972 (+1456629us) service_pool.cc:224] Handling call
0709 12:18:05.612617 (+360645us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.710980 (+ 98363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.711265 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:05.671214 26521 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:05.714854 26521 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.675504 26187 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.718580 26521 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:05.670773 26668 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:05.675670 26453 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:05.720414 26668 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:05.720197 26453 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.723758 26453 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:05.682608 12491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.023s	sys 0.042s
W20260709 12:18:05.724812 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.023s	sys 0.042s
I20260709 12:18:05.699954 26464 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:05.726476 26464 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.694536 26580 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:05.738763 26580 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.747761 26495 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:05.751681 26538 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.729s	user 0.012s	sys 0.000s
I20260709 12:18:05.671407 26669 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.759830 26669 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:05.707192 26671 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:05.768064 26698 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:05.700255 26496 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.822s	user 0.012s	sys 0.002s
I20260709 12:18:05.697773 26611 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.782424 12568 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:18:05.796790 26352 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:18:05.787943 26496 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.800182 26580 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:05.800402 26464 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:05.835157 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5549) took 2990 ms (client timeout 2009 ms). Trace:
W20260709 12:18:05.835697 12490 rpcz_store.cc:269] 0709 12:18:02.838924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:02.892313 (+ 53389us) service_pool.cc:224] Handling call
0709 12:18:05.829175 (+2936862us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:05.829319 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1698255744}
I20260709 12:18:05.817592 26305 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:05.826368 26523 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:05.825605 26549 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:05.829478 26715 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.252s	user 0.002s	sys 0.000s
I20260709 12:18:05.855134 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5448) took 1031 ms. Trace:
I20260709 12:18:05.826918 26352 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:05.857507 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 174332us
I20260709 12:18:05.855468 26305 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.856916 12571 rpcz_store.cc:276] 0709 12:18:04.823232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.007038 (+183806us) service_pool.cc:224] Handling call
0709 12:18:05.026839 (+ 19801us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.027059 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:05.673563 (+646504us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.854487 (+180924us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:05.854566 (+    79us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:05.854748 (+   182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.854842 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.855034 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15626,"prepare.queue_time_us":69925,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":7261,"raft.queue_time_us":238,"thread_start_us":802354,"threads_started":2}
I20260709 12:18:05.867564 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:05.875129 12563 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 32. Current term is 33. Ops: []
I20260709 12:18:05.886176 12563 raft_consensus.cc:1217] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Deduplicated request from leader. Original: 32.18->[33.19-38.22]   Dedup: 35.20->[36.21-38.22]
I20260709 12:18:05.890516 12563 pending_rounds.cc:85] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:18:05.890949 12563 pending_rounds.cc:107] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 37.21
I20260709 12:18:05.891340 12563 raft_consensus.cc:2889] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:05.888650 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:05.856362 26523 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.894902 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:05.899334 26676 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:05.883054 26670 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:05.900612 26670 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:05.901878 26670 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:05.909891 26352 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.878592 26664 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:05.864625 26704 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.857988 26549 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.923946 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:05.928239 26704 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:05.930837 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:05.931967 26676 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:05.933738 26730 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:05.946812 26453 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:05.948215 26676 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:05.951251 26549 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:05.951520 26698 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:05.953202 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5457) took 1077 ms. Trace:
I20260709 12:18:05.951333 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:05.936206 26531 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:05.961764 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2729) took 1015 ms. Trace:
W20260709 12:18:05.954748 26697 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.837s	user 0.013s	sys 0.002s
I20260709 12:18:05.965494 12572 rpcz_store.cc:276] 0709 12:18:04.875726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.614244 (+738518us) service_pool.cc:224] Handling call
0709 12:18:05.614677 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.614922 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:18:05.777249 (+162327us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.948008 (+170759us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.952794 (+  4786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:05.952856 (+    62us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:05.952905 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.952995 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.953133 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13424,"prepare.queue_time_us":222345,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":87,"raft.queue_time_us":16360,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":250,"thread_start_us":319999,"threads_started":2}
I20260709 12:18:05.949802 26733 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:05.973487 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:05.964123 26676 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:05.964587 26611 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:05.963974 26453 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:05.963647 26730 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:05.977257 12536 connection.cc:256] RPC call timeout handler was delayed by -2.19919714699995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
I20260709 12:18:06.019940 26305 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.038386 26670 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.031491 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5268) took 1548 ms. Trace:
I20260709 12:18:05.996625 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.018376 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.018504 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2135) took 2216 ms. Trace:
W20260709 12:18:05.986507 26726 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:18:06.011289 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2020) took 1555 ms (client timeout 1553 ms). Trace:
I20260709 12:18:05.999022 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5446) took 1176 ms. Trace:
I20260709 12:18:06.006139 26576 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:06.020658 26732 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:18:05.969998 26521 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:06.019160 26496 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:05.984689 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5683) took 1091 ms. Trace:
I20260709 12:18:05.975036 26733 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.013515 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5445) took 1191 ms. Trace:
I20260709 12:18:06.003641 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:06.020781 12611 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.553s (SENT) [suppressed 4 similar messages]
I20260709 12:18:05.985659 26735 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:05.978488 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.048333 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5562) took 2255 ms. Trace:
I20260709 12:18:06.071887 12489 rpcz_store.cc:276] 0709 12:18:03.792976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.696649 (+903673us) service_pool.cc:224] Handling call
0709 12:18:04.706163 (+  9514us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:04.729297 (+ 23134us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:04.729339 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:04.734642 (+  5303us) log.cc:844] Serialized 52 byte log entry
0709 12:18:04.850796 (+116154us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:04.877028 (+ 26232us) log.cc:844] Serialized 14 byte log entry
0709 12:18:04.877240 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:04.877322 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:06.043104 (+1165782us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18004f67e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:06.047858 (+  4754us) raft_consensus.cc:1685] finished
0709 12:18:06.047995 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:06.048251 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9356,"prepare.queue_time_us":25898,"prepare.run_wall_time_us":61,"raft.queue_time_us":43768,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":1633,"spinlock_wait_cycles":47652864,"thread_start_us":69357,"threads_started":1,"wal-append.queue_time_us":96630}
I20260709 12:18:06.060952 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.059857 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.067432 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5564) took 2274 ms. Trace:
I20260709 12:18:05.979188 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:06.063619 13259 rpcz_store.cc:269] 0709 12:18:04.455561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.458517 (+  2956us) service_pool.cc:224] Handling call
0709 12:18:06.011046 (+1552529us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54005c3fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:06.011193 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471706880}
I20260709 12:18:06.081949 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.099099 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 32.
W20260709 12:18:06.066987 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:06.070885 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50146us
I20260709 12:18:06.082005 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:06.056197 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1541971470001045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
W20260709 12:18:06.055606 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 2.01408e+06us
I20260709 12:18:06.082369 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.067170 12569 rpcz_store.cc:276] 0709 12:18:04.893660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.669177 (+775517us) service_pool.cc:224] Handling call
0709 12:18:05.700404 (+ 31227us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:05.887527 (+187123us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.887574 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:05.895636 (+  8062us) log.cc:844] Serialized 26 byte log entry
0709 12:18:05.963250 (+ 67614us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:18:05.963273 (+    23us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.963316 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:05.984328 (+ 21012us) raft_consensus.cc:1685] finished
0709 12:18:05.984360 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.984593 (+   233us) inbound_call.cc:177] Queueing success response
0709 12:18:06.029001 (+ 44408us) spinlock_profiling.cc:243] Waited 62.4 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
Metrics: {"raft.queue_time_us":13546,"raft.run_cpu_time_us":3710,"raft.run_wall_time_us":314770,"spinlock_wait_cycles":143414144,"thread_start_us":238160,"threads_started":2,"wal-append.queue_time_us":68449}
I20260709 12:18:06.102509 12490 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:06.083250 12488 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
W20260709 12:18:06.107931 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:06.105446 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.104106 12494 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:06.104723 12493 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:06.111137 12490 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:06.060485 12495 rpcz_store.cc:276] 0709 12:18:03.802642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.591793 (+1789151us) service_pool.cc:224] Handling call
0709 12:18:05.592158 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:05.790152 (+197994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.790191 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:05.799013 (+  8822us) log.cc:844] Serialized 26 byte log entry
0709 12:18:05.915957 (+116944us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:05.916007 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.916103 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:06.018261 (+102158us) raft_consensus.cc:1685] finished
0709 12:18:06.018282 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:06.018439 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8107,"raft.queue_time_us":8779,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":143,"thread_start_us":268375,"threads_started":2,"wal-append.queue_time_us":132884}
I20260709 12:18:05.967736 26531 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:06.069860 12564 rpcz_store.cc:276] 0709 12:18:04.822238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.826739 (+  4501us) service_pool.cc:224] Handling call
0709 12:18:04.827248 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:04.966554 (+139306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:06.013159 (+1046605us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:06.013431 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204588288}
I20260709 12:18:06.064248 12567 rpcz_store.cc:276] 0709 12:18:04.822586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.822808 (+   222us) service_pool.cc:224] Handling call
0709 12:18:05.005720 (+182912us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.006894 (+  1174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.998692 (+991798us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54001b35d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:05.998927 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":419479936}
I20260709 12:18:06.109534 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.057142 12637 rpcz_store.cc:276] 0709 12:18:04.483755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.484110 (+   355us) service_pool.cc:224] Handling call
0709 12:18:04.484554 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.002120 (+517566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:06.031049 (+1028929us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:06.031376 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1070508288}
W20260709 12:18:06.102100 12536 connection.cc:256] RPC call timeout handler was delayed by -2.0668971469999633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
I20260709 12:18:06.114429 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.059219 12640 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:18:06.118409 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.119766 12642 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:18:06.119048 12536 connection.cc:256] RPC call timeout handler was delayed by -2.04169714700015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260709 12:18:06.017468 26640 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.557s	user 0.008s	sys 0.007s
W20260709 12:18:06.120863 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9147971469999447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19510000000000002s for a response
W20260709 12:18:06.128108 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8706971470001008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
W20260709 12:18:06.128324 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:06.120668 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:06.047142 26679 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.264s	user 0.000s	sys 0.012s
W20260709 12:18:05.990211 26718 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:18:06.026558 26556 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.931s	user 0.005s	sys 0.009s
I20260709 12:18:06.076432 12492 rpcz_store.cc:276] 0709 12:18:03.793498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.732431 (+938933us) service_pool.cc:224] Handling call
0709 12:18:04.732883 (+   452us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:18:05.596262 (+863379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.596316 (+    54us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:18:05.597409 (+  1093us) log.cc:844] Serialized 104 byte log entry
0709 12:18:05.795119 (+197710us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:05.821027 (+ 25908us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.825809 (+  4782us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.826165 (+   356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.826707 (+   542us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:06.067108 (+240401us) raft_consensus.cc:1685] finished
0709 12:18:06.067135 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:06.067353 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3903,"prepare.queue_time_us":55522,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":110,"raft.queue_time_us":254878,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":7839,"thread_start_us":189246,"threads_started":1,"wal-append.queue_time_us":286553}
I20260709 12:18:06.077503 26735 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:06.041591 26719 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.211s	user 0.000s	sys 0.002s
I20260709 12:18:06.125546 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:05.992779 26539 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.133315 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.134330 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.115854 26579 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:04.432194 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:06.009101 26616 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:06.154322 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.134941 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:06.155766 13253 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:06.136286 12493 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:06.136246 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.138497 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:06.139015 12488 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:18:06.062310 26523 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.065840 26521 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.158311 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.059285 26453 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.159011 13254 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:06.126673 12489 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:18:06.159439 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.152609 26540 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.160418 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.160315 26540 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.152630 26734 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.019s
I20260709 12:18:06.163364 26540 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.146039 26496 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.122283 26694 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:06.164295 26496 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.164788 26694 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.160296 12495 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:06.167501 26694 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.168195 26579 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.147194 26735 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:05.968657 26664 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.121750 26733 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:06.173345 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:06.174161 26521 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.174993 26496 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.140552 26576 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:18:06.175722 26737 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:06.176535 26737 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:06.112432 26611 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.146311 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.179102 26611 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.134449 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:06.180739 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.180936 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.151134 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:06.151551 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 4d3b77689c31445cbba405c2b232495e: Aborted: Op aborted by new leader
I20260709 12:18:06.157783 26445 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.178431 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:06.179361 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.152096 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.150197 26556 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.181574 26611 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.198257 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5452) took 1356 ms. Trace:
I20260709 12:18:06.198771 12566 rpcz_store.cc:276] 0709 12:18:04.842149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.026176 (+184027us) service_pool.cc:224] Handling call
0709 12:18:05.037277 (+ 11101us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:05.042835 (+  5558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.042886 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:05.057051 (+ 14165us) log.cc:844] Serialized 26 byte log entry
0709 12:18:05.710458 (+653407us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:05.710523 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.710626 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:06.197911 (+487285us) raft_consensus.cc:1685] finished
0709 12:18:06.197949 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:06.198177 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":812,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":4352,"thread_start_us":593501,"threads_started":1,"wal-append.queue_time_us":741118}
I20260709 12:18:05.997000 26515 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.184612 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5569) took 2389 ms. Trace:
I20260709 12:18:06.206811 12491 rpcz_store.cc:276] 0709 12:18:03.794796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.069424 (+1274628us) service_pool.cc:224] Handling call
0709 12:18:05.069998 (+   574us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:05.093111 (+ 23113us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:18:05.093153 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:05.129309 (+ 36156us) log.cc:844] Serialized 26 byte log entry
0709 12:18:05.725406 (+596097us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:05.725466 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.725567 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:06.040922 (+315355us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:06.184310 (+143388us) raft_consensus.cc:1685] finished
0709 12:18:06.184333 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:06.184547 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5958,"raft.queue_time_us":25417,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":19272,"spinlock_wait_cycles":10338048,"thread_start_us":595694,"threads_started":1,"wal-append.queue_time_us":666377}
I20260709 12:18:06.184971 12492 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:18:06.192255 12387 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 31 to 33, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 33 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:06.185317 12490 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:06.184243 12644 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:06.183003 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.212232 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.185381 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.214026 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.183554 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.216116 12495 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 24.
I20260709 12:18:06.217199 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.220373 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.224844 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.226975 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.181655 12493 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:06.231089 12387 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 29 to 36. New cstate: current_term: 36 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.184523 26543 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:06.239631 12387 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 39. New cstate: current_term: 39 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:06.241145 12387 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 34 to 36, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 36 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:06.206691 26384 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.235872 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.185175 26556 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.219123 26670 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.237398 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.218907 26298 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.243134 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.258361 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:06.242648 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:06.230003 26698 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:06.261123 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:06.239531 26743 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.078s	user 0.002s	sys 0.000s
I20260709 12:18:06.266162 12495 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 44 because replica has last-logged OpId of term: 43 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 24.
I20260709 12:18:06.182452 26445 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.185003 26737 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.067s
W20260709 12:18:06.277312 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 208299us
I20260709 12:18:06.274478 12387 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 25 to 37. New cstate: current_term: 37 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.272878 26298 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.250666 26496 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:06.283859 12610 connection.cc:256] RPC call timeout handler was delayed by -1.8532262450000871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
I20260709 12:18:06.287894 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:06.288774 12610 connection.cc:256] RPC call timeout handler was delayed by -1.7908957640001972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
W20260709 12:18:06.289690 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:06.193163 26531 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.26, Last appended by leader: 26, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.289206 12491 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:18:06.262189 26543 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:06.304448 26263 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:06.303074 12387 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 30 to 40, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 40 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:06.277523 26694 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:06.279722 26663 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.280158 26698 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:06.310730 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5461) took 1155 ms. Trace:
I20260709 12:18:06.311188 12563 rpcz_store.cc:276] 0709 12:18:05.156097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.883313 (+727216us) service_pool.cc:224] Handling call
0709 12:18:05.885844 (+  2531us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:18:05.898340 (+ 12496us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:18:05.906852 (+  8512us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.959637 (+ 52785us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.959769 (+   132us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:05.962776 (+  3007us) log.cc:844] Serialized 52 byte log entry
0709 12:18:05.962943 (+   167us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:05.962976 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.963052 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:06.310441 (+347389us) raft_consensus.cc:1685] finished
0709 12:18:06.310467 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:06.310660 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11690,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":118,"raft.queue_time_us":19901,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":3168,"thread_start_us":46450,"threads_started":1,"wal-append.queue_time_us":236236}
W20260709 12:18:06.309355 26746 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:18:06.259775 26384 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.319136 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.321432 26330 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:06.322005 26330 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:06.322962 26330 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.21, Last appended by leader: 21, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:05.965515 12568 rpcz_store.cc:276] 0709 12:18:04.946157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.796885 (+850728us) service_pool.cc:224] Handling call
0709 12:18:05.799527 (+  2642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:05.799820 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:05.935405 (+135585us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.961095 (+ 25690us) log.cc:844] Serialized 14 byte log entry
0709 12:18:05.961340 (+   245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:05.961396 (+    56us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:05.961455 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:05.961532 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:05.961698 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30031,"prepare.queue_time_us":54554,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":1425,"raft.queue_time_us":4655,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":4760,"spinlock_wait_cycles":338304,"thread_start_us":115827,"threads_started":2}
I20260709 12:18:06.324220 26330 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:06.324957 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.327651 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.328317 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:06.299342 26515 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.335958 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.337103 12564 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:06.338310 12563 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 21. (index mismatch)
I20260709 12:18:06.338557 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.340520 12567 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:06.343111 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.344184 26519 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.344563 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.346004 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.344277 12564 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:18:06.349387 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.350260 12563 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 33: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:06.351101 26298 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.351291 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.353575 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.354257 12569 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:06.355947 12569 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:18:06.360772 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.361537 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:18:06.363597 12638 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
W20260709 12:18:06.363932 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4153) took 2564 ms (client timeout 1635 ms). Trace:
W20260709 12:18:06.364321 12497 rpcz_store.cc:269] 0709 12:18:03.800005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.842702 (+ 42697us) service_pool.cc:224] Handling call
0709 12:18:06.363739 (+2521037us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b540020e4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:06.363856 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640596736}
I20260709 12:18:06.365958 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:06.365958 12497 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 23. (term mismatch)
I20260709 12:18:06.367486 12497 pending_rounds.cc:85] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 22
I20260709 12:18:06.367735 12497 pending_rounds.cc:107] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 37.23
I20260709 12:18:06.367987 12497 raft_consensus.cc:2889] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:06.369328 26579 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:06.370477 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53471.6us
I20260709 12:18:06.372844 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:06.345094 12387 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 7 to 19, term changed from 25 to 36, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) evicted. New cstate: current_term: 36 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.377771 12568 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:18:06.307812 26694 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:06.384424 13254 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:06.310385 26730 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:06.390244 12566 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:18:06.301024 26496 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:06.388906 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 118544us
I20260709 12:18:06.391862 12387 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 38 to 39, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 39 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:06.398739 12387 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 43, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 43 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.404008 12387 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 38 to 40, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 40 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:06.406570 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 234) took 42342 ms (client timeout 15000 ms). Trace:
I20260709 12:18:06.397169 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.310865 26263 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:06.412710 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74038.4us
I20260709 12:18:06.389926 26694 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:06.416906 12387 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 39 to 44. New cstate: current_term: 44 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.419718 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:06.418422 26698 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.439023 26384 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:06.439697 26748 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:18:06.440419 12496 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:18:06.387650 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet a30ed0986fa641bd8f88912b4b9ee320 with delete_type TABLET_DATA_TOMBSTONED (TS 6d4ece942fd5414998215d26edf91001 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:59760
I20260709 12:18:06.441895 26263 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.421274 26515 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:06.452452 12387 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 41 to 44. New cstate: current_term: 44 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.457062 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:06.462450 26747 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:18:06.465581 12387 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 29 to 40. New cstate: current_term: 40 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:06.461835 26595 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:06.470134 26735 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.452414 26737 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:06.458582 26515 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:06.511220 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 114246us
I20260709 12:18:06.432982 26187 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.033s
I20260709 12:18:06.482376 12496 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.25-46.25]
I20260709 12:18:06.483952 12492 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: [40.21-40.21]
I20260709 12:18:06.494444 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.496438 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.498708 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.522051 12567 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:18:06.523018 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:06.491613 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.523440 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:06.421952 26676 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.505039 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:06.461872 26664 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:06.476150 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:06.524060 12567 raft_consensus.cc:2376] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41. Re-sending same reply.
I20260709 12:18:06.524626 12564 raft_consensus.cc:2376] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42. Re-sending same reply.
I20260709 12:18:06.529891 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.538443 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.539640 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.539801 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:06.540630 26540 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:06.541050 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.541397 26540 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:06.542920 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.538462 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.545493 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.546238 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:06.547307 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:06.547963 12564 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:06.550550 26556 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:06.551147 26556 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:06.551628 26556 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:06.556746 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 80679.1us
I20260709 12:18:06.562045 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.562952 12492 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:06.503196 26519 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.567646 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:06.565932 12568 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:06.591078 12569 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
W20260709 12:18:06.598209 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:06.595082 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.599663 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.603366 12569 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 35 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 21.
I20260709 12:18:06.596515 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.607673 12572 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:06.570850 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:06.610540 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.588411 26415 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
W20260709 12:18:06.620669 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49944.1us
I20260709 12:18:06.522063 26515 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:06.623054 12567 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:06.565071 26540 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.618671 26415 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:06.622325 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:06.593010 26519 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.553930 26701 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:06.628883 12569 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:18:06.531211 26298 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:06.588590 26664 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:06.627732 26539 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:18:06.637984 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2024) took 2174 ms (client timeout 2160 ms). Trace:
W20260709 12:18:06.651561 13256 rpcz_store.cc:269] 0709 12:18:04.456972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.463136 (+  6164us) service_pool.cc:224] Handling call
0709 12:18:06.631595 (+2168459us) spinlock_profiling.cc:243] Waited 7.32 ms on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:06.631769 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16826752}
I20260709 12:18:06.642284 12492 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:06.648316 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:06.630718 12564 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:06.625710 26737 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:06.407117 12386 rpcz_store.cc:269] 0709 12:17:24.040924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:37.144759 (+13103835us) service_pool.cc:224] Handling call
0709 12:18:06.381841 (+29237082us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:17:51.961835 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:17:51.962561 (+   726us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:17:51.962609 (+    48us) write_op.cc:435] Acquired schema lock
0709 12:17:51.962625 (+    16us) tablet.cc:662] Decoding operations
0709 12:17:51.964817 (+  2192us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:17:51.964924 (+   107us) write_op.cc:641] Partition lock acquired for write op
0709 12:17:51.964953 (+    29us) tablet.cc:685] Acquiring locks for 29 operations
0709 12:17:52.283367 (+318414us) tablet.cc:701] Row locks acquired
0709 12:17:52.283439 (+    72us) write_op.cc:260] PREPARE: finished
0709 12:17:52.283669 (+   230us) write_op.cc:270] Start()
0709 12:17:52.283860 (+   191us) write_op.cc:276] Timestamp: P: 1783599472283579 usec, L: 0
0709 12:17:52.283889 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 12:17:52.340350 (+ 56461us) log.cc:844] Serialized 11343 byte log entry
0709 12:17:53.564327 (+1223977us) op_driver.cc:464] REPLICATION: finished
0709 12:17:53.946075 (+381748us) write_op.cc:301] APPLY: starting
0709 12:17:53.946227 (+   152us) tablet.cc:1367] starting BulkCheckPresence
0709 12:17:55.418450 (+1472223us) tablet.cc:1370] finished BulkCheckPresence
0709 12:17:55.434899 (+ 16449us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:17:57.882167 (+2447268us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:17:57.907521 (+ 25354us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=58,key_file_lookups=58,delta_file_lookups=0,mrs_lookups=0
0709 12:17:57.907615 (+    94us) write_op.cc:312] APPLY: finished
0709 12:17:57.908457 (+   842us) log.cc:844] Serialized 252 byte log entry
0709 12:17:58.045270 (+136813us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:58.046154 (+   884us) write_op.cc:454] Released schema lock
0709 12:17:58.046248 (+    94us) write_op.cc:341] FINISH: Updating metrics
0709 12:17:58.051789 (+  5541us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:17:58.053989 (+  2200us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":949975,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":374662,"apply.run_cpu_time_us":117780,"apply.run_wall_time_us":4103011,"cfile_cache_hit":118,"cfile_cache_hit_bytes":125231,"num_ops":29,"prepare.queue_time_us":194427,"prepare.run_cpu_time_us":59683,"prepare.run_wall_time_us":740372,"raft.queue_time_us":246882,"raft.run_cpu_time_us":16177,"raft.run_wall_time_us":270321,"replication_time_us":1270843,"thread_start_us":793806,"threads_started":5,"wal-append.queue_time_us":622992,"wal-append.run_cpu_time_us":28219,"wal-append.run_wall_time_us":2758981}]]}
I20260709 12:18:06.628962 26668 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.650285 26298 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.661557 26298 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.662473 26298 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:06.663126 26298 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:06.664376 26298 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:06.650352 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 254896us
I20260709 12:18:06.682943 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:06.637070 26540 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.689145 26540 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.695989 12492 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:18:06.645169 26664 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:06.707600 26415 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.714881 26415 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:06.718191 26579 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:06.698247 26668 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.718825 26579 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:06.719331 26579 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:06.720574 26638 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:18:06.724850 26616 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.000s
I20260709 12:18:06.694607 26701 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:06.726457 13262 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:06.651185 26539 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:06.710538 26664 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:06.770124 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.773205 12492 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:06.774346 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.775136 12492 raft_consensus.cc:2376] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40. Re-sending same reply.
I20260709 12:18:06.725545 26701 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:06.791162 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2009) took 3129 ms. Trace:
I20260709 12:18:06.694643 26737 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.778004 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.799830 12496 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:18:06.780093 26758 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
W20260709 12:18:06.786654 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 238) took 19445 ms (client timeout 15000 ms). Trace:
W20260709 12:18:06.806197 12386 rpcz_store.cc:269] 0709 12:17:47.341489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.786362 (+19444873us) service_pool.cc:211] Skipping call since client already timed out
0709 12:18:06.786552 (+   190us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:18:06.702134 26730 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260709 12:18:06.776249 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.794503 12639 rpcz_store.cc:276] 0709 12:18:03.661848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.662030 (+   182us) service_pool.cc:224] Handling call
0709 12:18:03.662420 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:04.631945 (+969525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:06.790801 (+2158856us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:06.791080 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2227057024}
I20260709 12:18:06.791447 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:06.773294 26539 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:06.822672 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:06.825423 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4223) took 3178 ms. Trace:
I20260709 12:18:06.825953 12643 rpcz_store.cc:276] 0709 12:18:03.647104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:03.647341 (+   237us) service_pool.cc:224] Handling call
0709 12:18:03.647671 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:04.171171 (+523500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:06.825115 (+2653944us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:06.825336 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125687552}
I20260709 12:18:06.831488 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.793888 26668 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:06.835937 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:06.798648 26515 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.825165 26417 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:06.852362 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:06.857985 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:06.858744 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:06.855356 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 207155us
W20260709 12:18:06.856166 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 235) took 18785 ms (client timeout 15000 ms). Trace:
I20260709 12:18:06.858927 12496 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:06.861239 12386 rpcz_store.cc:269] 0709 12:17:48.059973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.833396 (+18773423us) service_pool.cc:211] Skipping call since client already timed out
0709 12:18:06.845262 (+ 11866us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:18:06.852754 26416 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:06.829126 26762 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:18:06.862020 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 160267us
I20260709 12:18:06.862135 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:06.871261 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:06.858024 26676 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.880522 26523 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:06.881963 12460 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:18:06.885689 26416 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:06.888065 12460 leader_election.cc:403] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:18:06.899708 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:06.867652 26417 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:06.891846 26523 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:06.911703 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:06.905646 26416 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:06.916939 12496 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:06.920459 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.922282 26611 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.922585 26416 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 46.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.922842 26611 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:06.921756 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:06.925733 26538 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.925894 26298 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:06.926445 26538 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.921193 26540 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:06.862452 26760 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.048s
W20260709 12:18:06.937166 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:06.938187 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:06.904160 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:06.956782 26415 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:06.957573 26415 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:06.958114 26415 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:06.958750 26519 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.893517 26730 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:18:06.960680 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56719.6us
I20260709 12:18:06.965806 26595 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:06.966452 26595 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:06.967084 26556 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:06.968374 26768 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:18:06.968953 26595 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:06.919956 26330 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.595s	user 0.016s	sys 0.001s
I20260709 12:18:06.969379 26556 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.916103 26523 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:06.921093 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:06.976758 12565 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:06.981823 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.983808 13253 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:06.980898 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:06.914281 26762 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:06.982223 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:06.980130 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59251.6us
I20260709 12:18:06.989606 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.990806 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:06.996546 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:06.992738 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.921501 26770 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:06.994640 13253 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 33.
I20260709 12:18:06.999217 13261 raft_consensus.cc:2410] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 26.
I20260709 12:18:07.000025 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:07.000476 12564 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 22. Preceding OpId from leader: term: 47 index: 23. (index mismatch)
I20260709 12:18:07.003432 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:07.010783 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:06.970561 26694 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.536s	user 0.008s	sys 0.004s
I20260709 12:18:07.013759 13260 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 42 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 28.
W20260709 12:18:07.017944 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 159309us
W20260709 12:18:07.023849 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:06.863010 26701 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 41.30, Last appended by leader: 30, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.032177 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.974807 26515 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:06.995916 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet a30ed0986fa641bd8f88912b4b9ee320 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:18:06.969589 26595 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.035578 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.990420 26556 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:07.041040 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.042773 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:06.906699 26654 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.092s
I20260709 12:18:06.991923 26762 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.047669 12563 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:06.963454 26730 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.048450 12565 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 41 index: 22. (term mismatch)
I20260709 12:18:06.937227 26760 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.088s
W20260709 12:18:07.049469 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53583.7us
I20260709 12:18:07.047742 13261 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:18:07.068439 12565 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:18:07.070124 12565 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 40.22
I20260709 12:18:07.042630 26734 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.071815 12565 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:07.087121 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:07.087852 13261 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 33.
I20260709 12:18:07.024916 26540 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.055s
I20260709 12:18:07.051249 26417 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:07.034138 26187 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.046s
I20260709 12:18:07.023334 26675 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.059s
I20260709 12:18:07.092531 13261 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 43 index: 29. (term mismatch)
I20260709 12:18:07.103211 13261 pending_rounds.cc:85] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 28
I20260709 12:18:07.103538 13261 pending_rounds.cc:107] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 42.29
I20260709 12:18:07.103835 13261 raft_consensus.cc:2889] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:07.075417 26730 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:07.064045 26556 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:07.063035 26762 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:07.051298 26595 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.123013 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:07.080226 26416 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.031s
I20260709 12:18:07.129228 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:07.130329 26774 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:18:07.130962 26774 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.131384 26774 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:18:07.131650 26671 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:07.090876 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5673) took 2356 ms. Trace:
I20260709 12:18:07.038192 26694 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.102185 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:07.106352 26417 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.114893 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:07.134718 13261 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:07.135017 26694 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.135934 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:07.136842 26671 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:18:07.137351 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:07.138195 26671 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:07.138599 26671 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.138953 26671 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:07.136381 12565 rpcz_store.cc:276] 0709 12:18:04.734206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:04.734430 (+   224us) service_pool.cc:224] Handling call
0709 12:18:04.783959 (+ 49529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:06.975921 (+2191962us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b58009ec0d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:18:07.090079 (+114158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:07.090752 (+   673us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":278083456}
I20260709 12:18:07.149120 13262 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:07.147821 26415 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.153980 26779 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:18:07.155114 26778 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:18:07.089098 26734 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.154276 26415 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:07.175480 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130597us
W20260709 12:18:07.248916 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63050.9us
W20260709 12:18:07.253018 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 75896.2us
I20260709 12:18:07.276763 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:07.279665 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:07.190433 26187 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.095s
W20260709 12:18:07.283818 26743 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:18:07.252493 26769 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.293147 26655 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:07.298427 26694 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:07.329681 26694 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.271625 26654 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.052s
I20260709 12:18:07.338037 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.340662 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:18:07.351399 26781 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260709 12:18:07.353179 26782 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:18:07.354708 26595 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.201092 26556 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.316208 26787 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:07.361307 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:07.367429 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1754) took 1126 ms. Trace:
I20260709 12:18:07.368923 26314 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:07.296789 26539 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.367825 13254 rpcz_store.cc:276] 0709 12:18:06.241340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.241522 (+   182us) service_pool.cc:224] Handling call
0709 12:18:07.367244 (+1125722us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:07.367368 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279734528}
W20260709 12:18:07.373805 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63954.3us
I20260709 12:18:07.340312 26694 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.340695 26655 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:07.327898 26769 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.371938 26314 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:07.396521 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.397619 12496 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:07.397753 13218 connection.cc:256] RPC call timeout handler was delayed by -1.9491639200000463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
W20260709 12:18:07.401149 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:07.401921 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:07.402415 13218 connection.cc:256] RPC call timeout handler was delayed by -1.7772639199999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
I20260709 12:18:07.403198 26595 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:07.403904 26595 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.405877 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:07.406658 12646 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 33.
I20260709 12:18:07.406680 26595 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.407620 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:07.407930 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:07.408015 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:07.409039 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:07.411773 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:07.414741 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.418404 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.419348 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:07.420650 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.394807 26314 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:07.380025 26774 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.430253 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:07.435974 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:07.436964 12564 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:07.428071 26734 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.426469 26523 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:07.459872 26579 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.740s	user 0.005s	sys 0.012s
I20260709 12:18:07.459986 26523 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.460628 26579 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.464443 26579 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:07.465382 26523 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.466140 26523 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.466893 26774 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:07.468765 26795 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:18:07.469461 26523 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.470103 26539 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.471189 26770 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:18:07.479583 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:07.485862 12563 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:07.487918 12564 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:18:07.488421 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.489384 12572 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:07.491535 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:07.492432 26769 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.493139 26769 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:18:07.493659 26769 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:07.494617 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.494604 26769 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.26, Last appended by leader: 26, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:07.498775 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:07.499541 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:07.514966 12493 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:07.521534 12493 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:07.522989 12493 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:07.444418 26655 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.517050 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:07.526000 12496 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40.
W20260709 12:18:07.526494 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96335.7us
I20260709 12:18:07.533401 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:07.515045 26664 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.722s	user 0.011s	sys 0.003s
I20260709 12:18:07.526265 26644 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:07.479664 26671 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:07.598210 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:07.625603 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:07.546409 26644 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.544165 26809 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:07.713414 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4157) took 1565 ms (client timeout 2037 ms). Trace:
W20260709 12:18:07.714107 12490 rpcz_store.cc:269] 0709 12:18:06.148181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.180584 (+ 32403us) service_pool.cc:224] Handling call
0709 12:18:07.713046 (+1532462us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:07.713318 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7578496}
I20260709 12:18:07.718743 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.718743 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:07.719861 12490 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:07.720175 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5580) took 1639 ms. Trace:
I20260709 12:18:07.720911 12494 rpcz_store.cc:276] 0709 12:18:06.080706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.130433 (+ 49727us) service_pool.cc:224] Handling call
0709 12:18:06.130963 (+   530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:06.166928 (+ 35965us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:06.167001 (+    73us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:06.167053 (+    52us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:06.167111 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:07.719818 (+1552707us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:07.719899 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:07.720086 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82095488}
I20260709 12:18:07.719885 12566 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 25.
W20260709 12:18:07.715067 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:07.711428 26644 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:07.734675 26770 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.735610 26770 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.746389 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:07.747746 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4160) took 1598 ms (client timeout 1720 ms). Trace:
W20260709 12:18:07.748279 12488 rpcz_store.cc:269] 0709 12:18:06.149459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.168665 (+ 19206us) service_pool.cc:224] Handling call
0709 12:18:07.746114 (+1577449us) spinlock_profiling.cc:243] Waited 8.5 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:18:07.747653 (+  1539us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19540224}
W20260709 12:18:07.751636 26798 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
I20260709 12:18:07.754240 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.755291 12565 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:07.716781 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:07.758822 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:07.764909 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.765766 12489 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
W20260709 12:18:07.766352 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4164) took 1615 ms (client timeout 1641 ms). Trace:
W20260709 12:18:07.766723 12489 rpcz_store.cc:269] 0709 12:18:06.150614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.161704 (+ 11090us) service_pool.cc:224] Handling call
0709 12:18:07.764737 (+1603033us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:18:07.766292 (+  1555us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15070464}
I20260709 12:18:07.767886 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:07.772578 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55903.4us
W20260709 12:18:07.702185 26762 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.576s	user 0.013s	sys 0.000s
I20260709 12:18:07.541858 26664 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:07.765483 26766 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.968s	user 0.003s	sys 0.000s
I20260709 12:18:07.704981 26576 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.778134 12610 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40.
I20260709 12:18:07.773355 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:07.705271 26774 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.797310 26701 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:07.802917 12490 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:07.806519 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.806802 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:07.808122 12494 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:07.810456 12490 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: [39.21-39.21]
W20260709 12:18:07.813339 26754 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.277s	user 0.001s	sys 0.000s
I20260709 12:18:07.813721 12497 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:07.819968 12494 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 26. Preceding OpId from leader: term: 43 index: 27. (index mismatch)
I20260709 12:18:07.822077 12494 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:18:07.809285 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:07.824061 26775 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:07.824725 26775 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:07.825819 26775 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.832895 26187 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:18:07.833916 26187 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:07.834955 26187 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:07.835335 26187 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:07.795463 26523 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:07.842250 12494 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 42 index: 26. (index mismatch)
I20260709 12:18:07.857558 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:07.795069 26700 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.068s
I20260709 12:18:07.837551 26519 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:07.860078 26700 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:07.859884 26519 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.860550 26700 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:07.862462 26811 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:18:07.801251 26733 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:07.866096 26733 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.803193 26701 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.867604 26701 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:07.867830 26579 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:18:07.868923 26733 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:07.791422 26759 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.056s
I20260709 12:18:07.870069 26671 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.872481 26664 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.836419 26187 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.21, Last appended by leader: 21, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:07.853767 12380 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 40 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:07.815806 26677 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:07.881732 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:07.879205 12494 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:18:07.905408 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:07.805035 26774 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.844410 26654 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:07.855427 26523 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:07.909503 12494 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:07.903002 26539 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:07.912663 26774 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.802491 26658 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.109s
W20260709 12:18:07.916872 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:07.910089 26677 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.802918 26576 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.922474 26519 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.923225 26187 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:07.903427 26700 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.925228 26644 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.791270 26762 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:07.929688 26314 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:07.934281 26737 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:07.935249 26770 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:07.942126 12566 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 30. Preceding OpId from leader: term: 42 index: 31. (index mismatch)
W20260709 12:18:07.943568 26730 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.819s	user 0.014s	sys 0.000s
W20260709 12:18:07.943545 26763 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.147s	user 0.020s	sys 0.000s
I20260709 12:18:07.943482 26523 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:07.939487 26654 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:07.967907 26556 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:07.971017 26579 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.020s
I20260709 12:18:07.985399 26658 consensus_queue.cc:1243] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 38.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
W20260709 12:18:07.999747 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 471709us
W20260709 12:18:08.173367 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 302102us
I20260709 12:18:07.967008 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:08.371984 12494 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 37: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:08.372252 26496 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:08.179661 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:08.203334 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.213974 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:08.375313 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:08.032899 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 303928us
I20260709 12:18:08.379419 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:08.069917 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:08.178221 12491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.008s	sys 0.009s
I20260709 12:18:07.969524 26539 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.264132 26314 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.003983 26556 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.302383 26576 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.000587 26700 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:07.996086 26730 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:08.410542 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:08.414702 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 93e029db6a8e4a7db72ad88760a6b7d3: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 39 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:18:08.334054 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 123233us
I20260709 12:18:08.344002 26644 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.354466 26701 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.329294 26737 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:08.358695 26677 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.358006 26654 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.363842 26797 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:08.363739 26762 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:08.366644 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4234) took 2395 ms. Trace:
I20260709 12:18:08.382170 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:08.386366 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.010s	sys 0.018s
I20260709 12:18:08.387519 26496 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:18:08.386298 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 458254us
I20260709 12:18:08.397681 26819 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:08.400297 26556 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.402381 26638 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:18:08.417796 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.423331 26496 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:08.425117 26819 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:08.423015 26797 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:08.441977 26654 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:08.443732 26701 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:08.444506 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62682.1us
I20260709 12:18:08.445083 26823 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:08.425782 26638 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:08.447644 26616 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:08.448383 26797 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.422497 12640 rpcz_store.cc:276] 0709 12:18:05.971458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.972252 (+   794us) service_pool.cc:224] Handling call
0709 12:18:05.979692 (+  7440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:06.119751 (+140059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.365824 (+2246073us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:08.366308 (+   484us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":181919104}
I20260709 12:18:08.449448 26823 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:08.434893 26496 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:08.449792 26810 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.002s
I20260709 12:18:08.434501 26737 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.457762 26823 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.462536 26638 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:08.463908 26725 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.528s	user 0.006s	sys 0.000s
I20260709 12:18:08.464416 26638 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:08.462670 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.466452 26539 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.437077 26819 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:08.467236 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:08.468320 12488 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:08.470727 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.473970 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:08.479198 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:08.461565 26737 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.498340 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:08.500190 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74087.8us
I20260709 12:18:08.506417 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5686) took 2010 ms. Trace:
I20260709 12:18:08.498459 26810 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.31, Next index: 32, Last known committed idx: 29, Time since last communication: 0.013s
I20260709 12:18:08.507529 12570 rpcz_store.cc:276] 0709 12:18:06.495411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.495593 (+   182us) service_pool.cc:224] Handling call
0709 12:18:06.512181 (+ 16588us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:06.673031 (+160850us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:18:06.673079 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:06.699071 (+ 25992us) log.cc:844] Serialized 26 byte log entry
0709 12:18:06.868490 (+169419us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:18:06.868534 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.502762 (+1634228us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:08.502878 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:08.502998 (+   120us) raft_consensus.cc:1685] finished
0709 12:18:08.503018 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.506102 (+  3084us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8993,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":5777,"spinlock_wait_cycles":368471936,"thread_start_us":146714,"threads_started":1,"wal-append.queue_time_us":268186,"wal-append.run_cpu_time_us":2184,"wal-append.run_wall_time_us":1004440}
W20260709 12:18:08.526585 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.528537 26737 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:08.530049 26737 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.540508 26810 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:08.542181 26352 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:18:08.542760 26644 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:08.543556 26644 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:08.543351 26818 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:08.545485 26644 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 41.29, Last appended by leader: 29, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.548238 26644 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:08.551537 26538 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.551779 26836 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:08.554315 26836 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:08.563627 26445 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.551604 26737 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:08.569291 26827 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.223s	user 0.004s	sys 0.000s
I20260709 12:18:08.577733 26538 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:08.578836 12494 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.21-37.21]
W20260709 12:18:08.579430 26822 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.231s	user 0.002s	sys 0.000s
I20260709 12:18:08.580170 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:08.582018 26445 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:18:08.584513 26821 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
W20260709 12:18:08.586444 26834 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.204s	user 0.000s	sys 0.002s
W20260709 12:18:08.588299 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48550.2us
I20260709 12:18:08.561211 26496 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:08.581444 26828 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.237s	user 0.001s	sys 0.002s
I20260709 12:18:08.637183 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:08.638548 12566 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:08.638684 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:08.639925 12572 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:08.619525 26641 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:08.641852 26641 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.621758 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:08.625813 26698 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 45.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:08.646474 12563 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:08.646766 26698 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:08.647397 26698 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:08.647840 26735 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.649539 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:08.627640 26496 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.630497 13261 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
W20260709 12:18:08.631587 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51713.5us
I20260709 12:18:08.631790 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5604) took 1112 ms. Trace:
I20260709 12:18:08.634195 26737 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:08.635532 26549 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:08.616407 12570 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 26. Preceding OpId from leader: term: 36 index: 27. (index mismatch)
I20260709 12:18:08.648813 26698 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.25, Last appended by leader: 25, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.657842 26539 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:08.668748 26800 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.212s	user 0.000s	sys 0.003s
I20260709 12:18:08.662019 26549 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.669473 26539 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.674762 26549 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:08.675263 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5459) took 3554 ms. Trace:
I20260709 12:18:08.676810 12563 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:18:08.677269 12565 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:18:08.675957 12571 rpcz_store.cc:276] 0709 12:18:05.120650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:05.882969 (+762319us) service_pool.cc:224] Handling call
0709 12:18:05.913851 (+ 30882us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:05.925219 (+ 11368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:05.925267 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:05.929383 (+  4116us) log.cc:844] Serialized 26 byte log entry
0709 12:18:07.290717 (+1361334us) spinlock_profiling.cc:243] Waited 952 us on lock 0x7b18004dfaa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:18:07.387095 (+ 96378us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:07.387136 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:07.387209 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:08.666301 (+1279092us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:08.674459 (+  8158us) raft_consensus.cc:1685] finished
0709 12:18:08.674495 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.674896 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20226,"raft.queue_time_us":10977,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":2984,"spinlock_wait_cycles":30823040,"thread_start_us":96143,"threads_started":1,"wal-append.queue_time_us":1502007}
I20260709 12:18:08.679659 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2142) took 2441 ms. Trace:
I20260709 12:18:08.680253 12497 rpcz_store.cc:276] 0709 12:18:06.238146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.378863 (+140717us) service_pool.cc:224] Handling call
0709 12:18:07.715062 (+1336199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:07.814150 (+ 99088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.498197 (+684047us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:08.679022 (+180825us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540034dd50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:08.679335 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3298289280}
I20260709 12:18:08.681114 12566 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: [39.26-39.26]
W20260709 12:18:08.664263 26793 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.277s	user 0.000s	sys 0.001s
I20260709 12:18:08.666203 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5607) took 1130 ms. Trace:
I20260709 12:18:08.645427 26641 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:08.667382 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:08.654320 12496 rpcz_store.cc:276] 0709 12:18:07.518861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:07.526703 (+  7842us) service_pool.cc:224] Handling call
0709 12:18:07.527105 (+   402us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:07.527498 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:07.527530 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:07.540028 (+ 12498us) log.cc:844] Serialized 52 byte log entry
0709 12:18:07.973353 (+433325us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:08.178982 (+205629us) log.cc:844] Serialized 14 byte log entry
0709 12:18:08.205923 (+ 26941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.206139 (+   216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:08.583334 (+377195us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:08.631169 (+ 47835us) raft_consensus.cc:1685] finished
0709 12:18:08.631279 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.631562 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11314,"prepare.run_cpu_time_us":181,"prepare.run_wall_time_us":420,"raft.queue_time_us":101146,"raft.run_cpu_time_us":767,"raft.run_wall_time_us":28116,"spinlock_wait_cycles":116919040,"thread_start_us":429756,"threads_started":1,"wal-append.queue_time_us":803342}
I20260709 12:18:08.691885 13256 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:18:08.691325 26836 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.686661 12489 rpcz_store.cc:276] 0709 12:18:07.535801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:07.770166 (+234365us) service_pool.cc:224] Handling call
0709 12:18:07.770515 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:07.771721 (+  1206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:07.771752 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:07.772566 (+   814us) log.cc:844] Serialized 26 byte log entry
0709 12:18:08.205256 (+432690us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:08.205483 (+   227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.205799 (+   316us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:08.553067 (+347268us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:08.661423 (+108356us) raft_consensus.cc:1685] finished
0709 12:18:08.661459 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.665976 (+  4517us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":631,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"spinlock_wait_cycles":42575232,"thread_start_us":405096,"threads_started":1,"wal-append.queue_time_us":572664}
I20260709 12:18:08.689384 12572 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:08.696213 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5706) took 1697 ms. Trace:
I20260709 12:18:08.696774 12567 rpcz_store.cc:276] 0709 12:18:06.998962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:07.001836 (+  2874us) service_pool.cc:224] Handling call
0709 12:18:07.002143 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:07.002498 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:07.002525 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:07.038783 (+ 36258us) log.cc:844] Serialized 26 byte log entry
0709 12:18:07.252814 (+214031us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:07.252870 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:07.252955 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:08.663179 (+1410224us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:08.695613 (+ 32434us) raft_consensus.cc:1685] finished
0709 12:18:08.695640 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:08.695970 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":719,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":40393,"spinlock_wait_cycles":2531712,"thread_start_us":172551,"threads_started":1,"wal-append.queue_time_us":284861}
W20260709 12:18:08.698652 26825 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.351s	user 0.000s	sys 0.002s
I20260709 12:18:08.701305 26737 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:08.702028 26737 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:08.702991 26735 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:08.704149 26735 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.703680 26737 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.691633 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.706933 13254 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 34: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:08.707345 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2028) took 1741 ms (client timeout 1640 ms). Trace:
W20260709 12:18:08.707870 13262 rpcz_store.cc:269] 0709 12:18:06.979534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.990481 (+ 10947us) service_pool.cc:224] Handling call
0709 12:18:08.706826 (+1716345us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:08.707124 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":354336128}
I20260709 12:18:08.709776 26735 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:08.711113 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85743.5us
I20260709 12:18:08.712244 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:08.713491 26797 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:08.692145 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:08.720304 13256 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 33. Current term is 34. Ops: []
I20260709 12:18:08.721975 26823 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:08.722551 26823 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:08.722796 26730 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:08.722973 26823 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:08.723766 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.725878 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.760665 13262 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:08.768299 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.763279 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.772145 13254 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:08.779462 26838 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260709 12:18:08.781291 26677 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:08.782989 26677 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:08.782718 26676 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:08.783490 26677 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:08.784811 26677 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.785532 13261 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
W20260709 12:18:08.728024 26523 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.724s	user 0.009s	sys 0.021s
W20260709 12:18:08.790149 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.729063 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.730890 26539 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.793922 26677 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:08.796895 13262 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:18:08.732154 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:08.733469 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.733915 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:08.800760 12496 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 38 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 24.
I20260709 12:18:08.734344 26737 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:08.789098 26523 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:08.802357 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89526.5us
I20260709 12:18:08.802632 12493 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:08.803743 26523 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.804066 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:08.734614 26826 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.390s	user 0.002s	sys 0.000s
I20260709 12:18:08.735062 12488 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:08.735723 12491 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:08.736601 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:08.787379 26698 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:08.808838 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72445.4us
I20260709 12:18:08.809942 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:08.811975 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:08.741297 26820 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.383s	user 0.001s	sys 0.001s
I20260709 12:18:08.741865 26616 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:08.742398 26818 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260709 12:18:08.814499 13257 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39. Re-sending same reply.
I20260709 12:18:08.814642 26616 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:08.814561 26824 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:18:08.754333 26809 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:08.758679 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:08.691432 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.816380 26809 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:08.743300 26832 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:08.817853 26832 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:08.818360 26832 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:08.817991 26809 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.819904 26809 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:08.819793 26832 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.26, Last appended by leader: 26, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.820740 26611 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:08.820899 26824 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:08.821684 26824 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:08.821475 26832 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:08.825371 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:08.826572 12491 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 40 because replica has last-logged OpId of term: 39 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:08.828649 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:08.829674 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102733us
I20260709 12:18:08.815162 26595 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:18:08.843638 12572 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
W20260709 12:18:08.783568 12611 connection.cc:256] RPC call timeout handler was delayed by -2.2592290379998303s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
W20260709 12:18:08.846961 12611 connection.cc:256] RPC call timeout handler was delayed by -2.1562751129999924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2136s for a response
W20260709 12:18:08.848446 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44547.3us
I20260709 12:18:08.851977 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:08.852492 26676 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:08.852872 12491 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48. Re-sending same reply.
I20260709 12:18:08.853135 26676 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:08.854310 26676 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:08.857319 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.858611 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:08.866894 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108343us
I20260709 12:18:08.868944 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:08.870014 26539 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:08.870545 26539 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:08.870899 26539 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:08.871611 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.872419 13256 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:08.873100 13257 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:18:08.873718 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.873958 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.874481 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:08.865461 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53630.5us
I20260709 12:18:08.864902 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:08.893383 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.893040 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:08.895679 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:08.908354 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.903422 13261 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:08.909581 13262 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:18:08.883088 26611 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:08.918874 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.918972 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:08.874585 26551 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.033s
I20260709 12:18:08.922516 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.927503 13257 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:08.896235 26595 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.26, Next index: 27, Last known committed idx: 24, Time since last communication: 0.025s
I20260709 12:18:08.931006 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:08.939226 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:08.943763 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:08.940577 12566 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:08.923039 26330 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.040s
W20260709 12:18:08.952664 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87920.9us
I20260709 12:18:08.914299 26818 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.030s
I20260709 12:18:08.958269 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:08.959321 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:08.919935 26819 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:08.964540 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:08.964445 12611 connection.cc:256] RPC call timeout handler was delayed by -2.1346751129999575s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20260709 12:18:08.969591 12611 connection.cc:256] RPC call timeout handler was delayed by -2.13017511299995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21650000000000003s for a response
I20260709 12:18:08.981037 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:08.990672 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:08.992347 26698 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:08.992975 12611 connection.cc:256] RPC call timeout handler was delayed by -1.9525628349999806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19710000000000003s for a response
I20260709 12:18:08.993225 26698 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:08.994125 26305 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:08.995816 26698 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:08.995980 26305 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:08.996016 26760 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:08.992597 26730 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:08.996528 13260 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:08.997970 26417 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:08.942168 26611 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:08.999574 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.948896 26737 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:08.968298 26644 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:18:09.002357 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:08.992564 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:09.008354 13260 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:08.998260 26421 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:08.985394 26770 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:08.992406 26641 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:09.014544 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:09.035148 13255 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:18:09.035744 13255 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:09.035498 26737 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.036806 13255 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 35.26, Last appended by leader: 26, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.038987 26421 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.041911 26730 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election won for term 48
W20260709 12:18:09.042505 26841 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.177s	user 0.003s	sys 0.000s
I20260709 12:18:09.014132 26644 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:09.044878 26611 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:09.044997 26644 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:09.045722 26417 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.048470 26417 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:09.041718 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:09.057745 13254 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:09.064720 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 63336.4us
W20260709 12:18:09.069721 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50525.8us
I20260709 12:18:09.059727 13257 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:09.071537 13255 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:18:09.075865 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68336.2us
I20260709 12:18:09.079270 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:09.080715 13254 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:09.028148 26305 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.084728 26305 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:09.084829 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 86989us
I20260709 12:18:09.061220 13260 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.26, Last appended by leader: 26, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.086190 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.086777 13260 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:09.087612 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64354.4us
I20260709 12:18:09.085542 26421 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:09.029762 26770 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:09.096772 13258 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:09.016139 26654 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.060s
W20260709 12:18:09.097609 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73529us
I20260709 12:18:09.061645 26616 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:09.065438 26730 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:09.049821 26836 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:09.118803 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:09.071205 26641 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:09.077214 26818 raft_consensus.cc:2764] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election decision vote started in defunct term 40: won
I20260709 12:18:09.055126 26760 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.109738 26735 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:09.130247 26694 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:09.119771 26654 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:09.168449 12567 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:18:09.138883 26737 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:09.164736 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 76037.6us
I20260709 12:18:09.171028 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:09.177250 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66534.9us
I20260709 12:18:09.158529 26694 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:09.195166 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:09.206563 26770 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.21, Last appended by leader: 21, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.207844 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:09.210243 26843 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:18:09.211858 26819 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:09.213397 26851 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
W20260709 12:18:09.218315 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 117851us
W20260709 12:18:09.217531 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:09.182027 26187 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.095s	user 0.008s	sys 0.029s
W20260709 12:18:09.221272 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47532.1us
W20260709 12:18:09.224380 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.226385 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:09.250896 13255 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 26. Preceding OpId from leader: term: 36 index: 27. (index mismatch)
W20260709 12:18:09.180271 26352 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.637s	user 0.013s	sys 0.013s
I20260709 12:18:09.215426 26694 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:09.267047 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46873.8us
I20260709 12:18:09.153295 26836 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.268621 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.270792 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:09.281201 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:09.268160 26810 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 36 current term: 36 is higher.
W20260709 12:18:09.262023 26848 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:18:09.281044 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:09.278220 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.211169 26854 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.111s
W20260709 12:18:09.321920 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 203502us
W20260709 12:18:09.331261 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 201856us
W20260709 12:18:09.333462 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64959.9us
I20260709 12:18:09.334595 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.344295 12497 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 41 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 25.
I20260709 12:18:09.347007 26760 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:09.348737 26760 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.350858 26352 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:09.358563 12645 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:09.359055 12645 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:09.360139 12645 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.28, Last appended by leader: 28, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.361224 12645 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:09.362133 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.260464 26676 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.060s
W20260709 12:18:09.365825 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52235.5us
I20260709 12:18:09.387018 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
W20260709 12:18:09.390421 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57862.4us
W20260709 12:18:09.393931 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.345502 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:09.404075 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.408872 26668 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:09.409570 26668 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:09.410072 26668 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:09.412104 12492 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 34.
I20260709 12:18:09.413190 26773 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.414041 26773 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.398818 26698 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:18:09.419834 26860 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:18:09.421821 26853 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:18:09.428247 26858 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:18:09.429926 12611 leader_election.cc:400] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47.
I20260709 12:18:09.413125 26496 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20260709 12:18:09.449831 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 115336us
I20260709 12:18:09.419569 26868 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:09.455513 26698 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:09.457266 13260 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 40 index: 26. (term mismatch)
W20260709 12:18:09.459005 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52257.7us
I20260709 12:18:09.324105 26641 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:09.441797 26838 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260709 12:18:09.473119 13256 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.011s
I20260709 12:18:09.477466 13256 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:09.462342 13260 pending_rounds.cc:85] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 25
I20260709 12:18:09.483064 26769 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.483914 26769 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.473955 26556 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:09.349479 26760 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.518532 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.492606 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.485580 26773 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:09.522557 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:09.523733 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59123.4us
I20260709 12:18:09.504216 26644 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.505847 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:09.487694 26556 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:09.529779 13262 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.611s	user 0.015s	sys 0.000s
I20260709 12:18:09.478427 13260 pending_rounds.cc:107] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 39.26
I20260709 12:18:09.530407 13262 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
W20260709 12:18:09.512413 26809 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.002s
I20260709 12:18:09.297818 26730 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.512694 26668 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:09.526198 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.532140 13260 raft_consensus.cc:2889] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:09.545184 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:09.549064 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:09.551811 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:09.559391 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.555348 26523 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.561508 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:09.565130 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.555838 12495 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:18:09.569067 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 63404.5us
I20260709 12:18:09.569319 26733 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:09.570065 26733 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:09.570393 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.577340 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:09.582113 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:09.559787 26595 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 40 current term: 40 is higher.
I20260709 12:18:09.576226 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:09.540977 26654 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.584796 13259 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:09.557746 26539 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.686s	user 0.008s	sys 0.008s
W20260709 12:18:09.579494 26677 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.785s	user 0.008s	sys 0.007s
I20260709 12:18:09.598177 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:09.603447 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0081527570000617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
I20260709 12:18:09.570618 26733 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:09.610637 12572 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:09.611629 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:09.570842 26780 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.616218 26773 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:09.617004 26698 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 41.29, Last appended by leader: 29, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.619086 26330 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:09.579492 26868 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.033s
I20260709 12:18:09.620651 26868 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:09.620785 26420 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:09.621122 26868 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:09.621228 26420 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:09.622658 12568 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:09.622287 26868 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.27, Last appended by leader: 27, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.623401 26496 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:09.624692 26868 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:09.606535 26417 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:09.629189 12645 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 47 index: 28. (term mismatch)
I20260709 12:18:09.630947 26556 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:09.632619 26668 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:09.634305 26760 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:18:09.635267 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.638816 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:09.569186 26523 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.640776 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.594436 26654 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.622287 26420 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.28, Last appended by leader: 28, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.650522 12497 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
W20260709 12:18:09.655074 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43553us
I20260709 12:18:09.656062 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.605050 26836 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:09.660586 26823 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.937s	user 0.012s	sys 0.006s
I20260709 12:18:09.660549 12572 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:09.643543 26417 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:18:09.662046 26420 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:09.657979 12645 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 27
I20260709 12:18:09.662972 12497 raft_consensus.cc:1217] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Deduplicated request from leader. Original: 40.25->[41.26-41.26]   Dedup: 41.26->[]
I20260709 12:18:09.670686 12497 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: [41.26-41.26]
I20260709 12:18:09.669672 12645 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 46.28
I20260709 12:18:09.671558 12645 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:09.643772 26731 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:09.595925 26539 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.643813 26654 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:09.641556 26523 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:09.667577 26330 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:09.663460 26496 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:09.688336 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.691649 26873 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.649799 26780 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:09.703617 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5594) took 2933 ms (client timeout 1905 ms). Trace:
W20260709 12:18:09.704187 12492 rpcz_store.cc:269] 0709 12:18:06.770565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:06.775908 (+  5343us) service_pool.cc:224] Handling call
0709 12:18:09.703298 (+2927390us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:09.703505 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5641352448}
I20260709 12:18:09.668110 26417 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:09.651701 26694 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:09.648783 26733 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.713147 26417 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:18:09.716485 26780 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:09.717131 26875 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:09.719777 26760 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:18:09.716470 26832 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.894s	user 0.015s	sys 0.000s
W20260709 12:18:09.723263 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.692497 26873 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.669507 26823 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.742807 13261 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 25.
I20260709 12:18:09.744158 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.745090 12645 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:09.734187 12497 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
W20260709 12:18:09.737176 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.738528 12490 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:09.709758 26330 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:09.752171 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84646.8us
I20260709 12:18:09.748574 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.728854 26644 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:09.756579 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85364.2us
W20260709 12:18:09.756754 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100789us
I20260709 12:18:09.757241 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.756024 12492 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 42 because replica has last-logged OpId of term: 40 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 22.
I20260709 12:18:09.726600 26773 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:09.760211 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:09.763254 13253 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:09.766453 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:09.763715 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.772783 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.769727 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:09.773654 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:09.782110 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet c126dc4ef9114297874fa66b1a0c56b0: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 46 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:09.763041 26352 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:09.781850 26875 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:09.783373 26733 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:09.786646 26823 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:09.787608 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.794265 26780 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:18:09.772367 26644 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:09.772475 26654 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.031s
I20260709 12:18:09.804131 26330 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:09.804248 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:09.806162 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet a1d35de928f546349f88cbea7818041c: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:18:09.791126 26824 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.969s	user 0.013s	sys 0.002s
I20260709 12:18:09.808954 26824 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.815083 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:09.802780 26539 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:09.818787 13254 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:09.817514 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.821985 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.761812 26496 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 39.27, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.825944 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:09.823971 26773 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.829360 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:09.831332 26824 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:09.831688 26793 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.549s	user 0.006s	sys 0.000s
I20260709 12:18:09.841410 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:09.841540 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.843243 12495 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:09.835701 26644 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:09.847117 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:09.847605 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.858628 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.859833 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:09.839330 26417 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.851159 26694 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:09.852994 26644 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:09.872119 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.874415 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:09.873605 12640 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:09.882050 12491 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.28-40.28]
I20260709 12:18:09.853328 26733 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.019s
I20260709 12:18:09.891294 12639 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 43 index: 27. (index mismatch)
I20260709 12:18:09.890604 26644 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 40.27, Last appended by leader: 27, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:09.895942 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:09.896975 26694 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:09.898344 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:09.898994 12570 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:09.899755 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:09.891387 26877 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.186s	user 0.003s	sys 0.000s
I20260709 12:18:09.912017 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1766) took 1039 ms. Trace:
I20260709 12:18:09.912176 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.914363 12643 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:18:09.916472 12641 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:18:09.942660 12639 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
W20260709 12:18:09.953853 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67574.9us
I20260709 12:18:09.912662 13259 rpcz_store.cc:276] 0709 12:18:08.872173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:08.903078 (+ 30905us) service_pool.cc:224] Handling call
0709 12:18:09.911771 (+1008693us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:09.911950 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1192671744}
W20260709 12:18:09.956022 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:09.969254 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:09.962869 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:09.973407 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:09.974732 12570 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:09.980796 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:09.883606 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c56dde469374f4da7440f134140670c"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:10.018744 12497 raft_consensus.cc:606] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:10.021865 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:09.969869 26595 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.050s
I20260709 12:18:09.976068 26824 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:09.986390 26818 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:10.005831 26875 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:10.014036 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:09.986730 26868 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:09.982609 26523 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:18:10.037145 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.044305 12637 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:18:10.055491 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.039391 26818 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:10.070127 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.051671 26735 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:10.041123 26868 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:10.098099 12640 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:10.048319 26824 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.099572 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:10.101953 26885 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.123s	user 0.001s	sys 0.001s
I20260709 12:18:10.098845 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.106753 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.107177 12642 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:10.107771 12641 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:10.110009 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.111869 12640 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:10.112037 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134628us
W20260709 12:18:10.116149 26847 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.036s	user 0.000s	sys 0.006s
I20260709 12:18:10.116418 26868 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:10.117945 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "472e786b46074d868d00df67a0985afd"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:10.118662 12489 raft_consensus.cc:606] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:10.119071 12489 raft_consensus.cc:614] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:10.119771 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2740) took 1456 ms. Trace:
I20260709 12:18:10.124881 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1768) took 1175 ms. Trace:
W20260709 12:18:10.129243 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.042s	sys 0.068s
W20260709 12:18:10.129657 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.042s	sys 0.068s
I20260709 12:18:10.129540 26668 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.001s
W20260709 12:18:10.130475 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 904139us
I20260709 12:18:10.131915 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.132349 26868 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:10.132748 12570 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:10.132877 12572 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: [39.21-39.21]
I20260709 12:18:10.120177 12569 rpcz_store.cc:276] 0709 12:18:08.662938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:08.666265 (+  3327us) service_pool.cc:224] Handling call
0709 12:18:08.666817 (+   552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:08.701206 (+ 34389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:08.701249 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:08.702244 (+   995us) log.cc:844] Serialized 26 byte log entry
0709 12:18:08.942078 (+239834us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:08.942150 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:08.942269 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:10.115775 (+1173506us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:18:10.119283 (+  3508us) raft_consensus.cc:1685] finished
0709 12:18:10.119307 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:10.119570 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32056,"raft.queue_time_us":33925,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"spinlock_wait_cycles":24791808,"thread_start_us":238246,"threads_started":1,"wal-append.queue_time_us":280108}
I20260709 12:18:10.125356 13261 rpcz_store.cc:276] 0709 12:18:08.949859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.002532 (+ 52673us) service_pool.cc:224] Handling call
0709 12:18:10.124582 (+1122050us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:10.124804 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1643076480}
I20260709 12:18:10.144042 12571 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:10.144747 26824 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:10.146683 12566 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.24-40.24]
I20260709 12:18:10.148746 12567 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: [39.25-39.25]
I20260709 12:18:10.151072 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.155048 12567 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
W20260709 12:18:10.161847 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.158852 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5643) took 1030 ms. Trace:
I20260709 12:18:10.162469 12490 rpcz_store.cc:276] 0709 12:18:09.128068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.128226 (+   158us) service_pool.cc:224] Handling call
0709 12:18:09.145911 (+ 17685us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:09.750094 (+604183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:10.158445 (+408351us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:10.158766 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1354465024}
I20260709 12:18:10.164397 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1765) took 1292 ms. Trace:
I20260709 12:18:10.164819 13253 rpcz_store.cc:276] 0709 12:18:08.871827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:08.873571 (+  1744us) service_pool.cc:224] Handling call
0709 12:18:10.164182 (+1290611us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:10.164336 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2031221120}
I20260709 12:18:10.167531 13262 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:10.150053 26868 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:10.175011 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.175305 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:10.176007 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.175876 12571 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:10.175850 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:10.186400 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:18:10.191361 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:10.157421 26891 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:18:10.196169 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.203043 12569 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:10.207355 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.208333 12564 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:10.207506 26701 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:10.208923 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.217373 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.225675 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5640) took 1119 ms. Trace:
I20260709 12:18:10.225827 12492 raft_consensus.cc:1217] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Deduplicated request from leader. Original: 45.24->[46.25-46.25]   Dedup: 46.25->[]
I20260709 12:18:10.225548 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.228783 12565 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:18:10.228632 12492 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.25-46.25]
I20260709 12:18:10.134646 26818 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.221493 26780 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:18:10.240767 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:10.250772 26856 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.092s	user 0.005s	sys 0.000s
I20260709 12:18:10.251274 12646 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:10.263417 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.264158 12563 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:18:10.264184 12570 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:18:10.270080 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.276007 12646 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:18:10.276979 12563 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:10.281891 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.282958 12639 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:18:10.277931 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260709 12:18:10.298965 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117790us
I20260709 12:18:10.299147 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.300200 12640 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 40 because replica has last-logged OpId of term: 39 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 21.
I20260709 12:18:10.302281 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:10.303181 12497 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:10.306118 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.308981 12563 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:10.311525 12567 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:10.312688 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5651) took 1057 ms. Trace:
I20260709 12:18:10.313423 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.316151 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.320799 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.300601 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:10.321977 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.322916 12640 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:10.322657 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.323303 26832 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:18:10.313172 12491 rpcz_store.cc:276] 0709 12:18:09.255837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.256027 (+   190us) service_pool.cc:224] Handling call
0709 12:18:09.872282 (+616255us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:18:09.872509 (+   227us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:09.883131 (+ 10622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:10.312351 (+429220us) spinlock_profiling.cc:243] Waited 9.24 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:10.312597 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24942208}
I20260709 12:18:10.323957 26832 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:10.325074 26873 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:10.301411 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.328122 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:10.328699 12570 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:18:10.329440 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47152.8us
W20260709 12:18:10.330286 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:10.330652 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.331950 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:10.333297 26668 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:10.335098 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5644) took 1190 ms. Trace:
I20260709 12:18:10.335963 26616 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:10.335887 12496 rpcz_store.cc:276] 0709 12:18:09.144301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.153000 (+  8699us) service_pool.cc:224] Handling call
0709 12:18:09.168819 (+ 15819us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:09.310316 (+141497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:09.310367 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:09.310431 (+    64us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:09.310489 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:10.334813 (+1024324us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:10.334870 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:10.335027 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":324694784}
I20260709 12:18:10.336828 26616 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.337590 26668 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:10.328568 26417 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:10.340451 26668 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:10.338593 26868 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:10.339497 26616 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:10.325491 26832 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.354019 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.355134 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.355858 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.356184 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.358038 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.358048 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.364019 26760 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:10.364511 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:10.364843 26760 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:10.365267 13262 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:18:10.367707 26760 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:10.313009 12567 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:10.370396 13261 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.27-44.27]
I20260709 12:18:10.372429 12645 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:18:10.373519 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:10.375669 26897 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.074s	user 0.007s	sys 0.010s
I20260709 12:18:10.376719 13261 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:10.378635 26892 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.104s	user 0.000s	sys 0.003s
W20260709 12:18:10.380277 26893 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.138s	user 0.001s	sys 0.001s
W20260709 12:18:10.375677 26879 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.664s	user 0.002s	sys 0.005s
I20260709 12:18:10.372526 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:10.384598 26898 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.057s	user 0.000s	sys 0.002s
I20260709 12:18:10.384555 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:10.375653 12564 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:10.387611 12566 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:10.388870 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.389900 12645 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:10.391067 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.392082 12572 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:10.333385 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:10.399837 26873 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:18:10.400173 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66873.9us
I20260709 12:18:10.402168 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:10.403239 26873 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:10.406653 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.407460 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.408175 12646 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:10.408396 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.408773 26769 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:10.409407 12640 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 40 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:10.411738 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.412794 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.413115 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.414731 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.415922 12642 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:10.416098 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.414324 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.418483 12638 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:10.420188 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.421934 12643 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 21. (term mismatch)
I20260709 12:18:10.422498 12643 pending_rounds.cc:85] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 20
I20260709 12:18:10.423501 12643 pending_rounds.cc:107] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 37.21
I20260709 12:18:10.424216 12643 raft_consensus.cc:2889] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:10.426393 26802 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.005s
I20260709 12:18:10.433185 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:10.433899 26802 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:10.434826 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:10.435272 26802 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:10.405823 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.449875 12645 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42. Re-sending same reply.
I20260709 12:18:10.452045 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.452366 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.453370 12564 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:10.452926 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.437155 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.452736 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:10.449747 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45949.4us
I20260709 12:18:10.453557 12639 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34.
I20260709 12:18:10.452054 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.453109 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:10.452113 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.459240 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.460563 12645 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 27.
I20260709 12:18:10.467394 12567 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42.
I20260709 12:18:10.472645 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.474982 12645 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:10.473665 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:10.481344 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:10.470001 26735 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:10.483137 12566 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:10.483635 12570 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34.
I20260709 12:18:10.484011 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:10.491468 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.496088 26802 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:10.501850 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:10.497345 12536 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.539s (SENT)
W20260709 12:18:10.509524 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.513319 12565 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:18:10.515928 12567 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:10.516520 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:10.482359 26780 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:10.522001 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:10.521615 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 292116us
I20260709 12:18:10.525336 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:10.528048 12564 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:18:10.527537 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:10.531713 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.504503 26802 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:10.537551 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:10.519490 26873 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:10.544044 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.502238 26735 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:10.544704 12644 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:10.551688 12645 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:10.540056 26769 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:10.543570 26701 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:18:10.562095 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.545046 26903 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:18:10.527242 26780 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:10.567442 13261 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:18:10.568584 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71287.1us
I20260709 12:18:10.568866 26698 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:18:10.226410 12488 rpcz_store.cc:276] 0709 12:18:09.106741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:09.106939 (+   198us) service_pool.cc:224] Handling call
0709 12:18:09.107384 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:09.107648 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:10.120286 (+1012638us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b18000ea340. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:18:10.120448 (+   162us) log.cc:844] Serialized 14 byte log entry
0709 12:18:10.225018 (+104570us) log.cc:844] Serialized 14 byte log entry
0709 12:18:10.225267 (+   249us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:10.225323 (+    56us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:10.225384 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:10.225453 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:10.225587 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10647,"prepare.queue_time_us":68238,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":7910,"raft.queue_time_us":187,"spinlock_wait_cycles":74946944,"thread_start_us":185680,"threads_started":2}
I20260709 12:18:10.556113 26769 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:10.611517 12645 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
W20260709 12:18:10.638155 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.562502 26701 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:10.641585 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 229903us
I20260709 12:18:10.642540 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:10.658030 12461 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.658s (SENT)
W20260709 12:18:10.672628 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 381099us
I20260709 12:18:10.697486 26770 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:18:10.706226 26694 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:10.706863 26694 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:10.708181 26694 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 42.31, Last appended by leader: 31, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.712579 12488 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:10.658675 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:10.637281 26769 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:10.732371 26330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.004s	sys 0.010s
I20260709 12:18:10.737692 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:10.739394 12494 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:18:10.740407 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.750232 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:10.655579 26701 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:10.640905 26698 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:10.752621 12489 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
W20260709 12:18:10.761336 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.773777 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.775331 12488 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 48 index: 32. (index mismatch)
I20260709 12:18:10.732875 26873 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:10.782704 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 124108us
I20260709 12:18:10.784180 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:10.777482 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 264352us
I20260709 12:18:10.787729 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:10.789207 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:10.785773 26873 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.795425 26903 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:10.741895 26330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.004s	sys 0.010s
I20260709 12:18:10.798292 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:10.813898 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:10.819305 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161322us
I20260709 12:18:10.837438 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:10.844540 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:10.844828 12461 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.788s (SENT)
I20260709 12:18:10.845278 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:10.848809 12645 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:10.797976 26882 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:10.862802 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:10.863045 12495 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:10.866220 12644 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:10.882342 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:10.883911 12645 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:10.817800 26883 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:10.851497 26892 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:18:10.895007 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:10.851370 26932 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:10.913403 26932 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:10.917009 26932 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:10.925995 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:10.913228 26330 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:10.934953 26903 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:10.960300 13262 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:18:10.970939 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:10.952450 26884 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.009s	sys 0.019s
W20260709 12:18:10.994788 26884 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.010s	sys 0.020s
I20260709 12:18:11.003629 13259 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 41 index: 28. (index mismatch)
I20260709 12:18:10.906229 26883 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:11.019023 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.020061 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:10.997896 26903 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:18:11.030275 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135376us
W20260709 12:18:11.032610 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 248508us
I20260709 12:18:11.034997 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:11.035953 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
W20260709 12:18:11.024925 26908 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.176s	user 0.007s	sys 0.000s
W20260709 12:18:11.069717 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:11.119650 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.023031 26780 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:11.121096 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79118.6us
I20260709 12:18:11.027261 26903 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:11.122263 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:11.026855 26735 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:11.134634 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:11.140985 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:10.982528 26698 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:11.173655 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:11.177789 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.179318 12646 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 27.
I20260709 12:18:11.147051 26903 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:11.186225 12568 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 43 index: 22. (index mismatch)
I20260709 12:18:11.187821 12572 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 42 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 20.
W20260709 12:18:11.189967 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49164.9us
I20260709 12:18:11.159940 26873 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:11.207525 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:11.219238 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.197801 26770 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:11.204284 26924 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:11.173118 26780 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.044s
I20260709 12:18:11.165587 26453 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.196s
W20260709 12:18:11.231419 26668 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.886s	user 0.014s	sys 0.000s
W20260709 12:18:11.236192 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 204310us
I20260709 12:18:11.145054 26934 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.186s
I20260709 12:18:11.223120 26873 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:11.235615 26913 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.224s	user 0.001s	sys 0.001s
I20260709 12:18:11.252054 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.233016 26884 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:11.286567 26884 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:11.286697 26911 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.265s	user 0.001s	sys 0.001s
I20260709 12:18:11.287618 26884 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:11.286589 26873 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:11.250280 26939 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:11.247851 26668 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:11.225804 26906 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.090s
I20260709 12:18:11.255944 26730 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:18:11.332070 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107228us
I20260709 12:18:11.293377 26733 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
W20260709 12:18:11.379346 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 244910us
W20260709 12:18:11.381040 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:11.382606 12563 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:11.382149 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:11.381465 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135185us
I20260709 12:18:11.301945 26780 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.082s
I20260709 12:18:11.394774 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.399448 12646 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:11.359095 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.302601 26417 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.081s
W20260709 12:18:11.389753 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135818us
I20260709 12:18:11.328588 26730 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:11.430296 12493 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 29. Preceding OpId from leader: term: 42 index: 30. (index mismatch)
I20260709 12:18:11.356633 26733 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:11.427466 26770 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:11.433240 26770 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:11.436944 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:11.442361 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:11.444187 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.008s	sys 0.016s
W20260709 12:18:11.444617 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.008s	sys 0.016s
I20260709 12:18:11.437789 12568 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:11.437397 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:11.451227 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:11.455643 12572 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:18:11.387791 26882 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:18:11.383914 26735 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.057s
I20260709 12:18:11.458968 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:11.490778 13253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.005s	sys 0.020s
I20260709 12:18:11.494714 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:11.329407 26939 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:11.503080 13253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.006s	sys 0.021s
I20260709 12:18:11.505216 12491 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 48 because replica has last-logged OpId of term: 47 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 25.
I20260709 12:18:11.505966 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.506911 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:11.486953 26538 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:11.459682 26868 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:11.522504 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 677639us
I20260709 12:18:11.538872 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.540306 12641 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:11.532580 26868 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:11.523576 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.800s (SENT) [suppressed 60 similar messages]
W20260709 12:18:11.548381 12461 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.596s (SENT)
I20260709 12:18:11.524968 26538 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:11.518510 26883 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:11.586953 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:11.524973 26949 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:11.541594 26770 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:11.723126 13259 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.000s	sys 0.006s
I20260709 12:18:11.738375 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:11.743729 13259 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.000s	sys 0.007s
I20260709 12:18:11.620966 26538 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:11.745072 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1770) took 1353 ms. Trace:
W20260709 12:18:11.752944 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:11.754902 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 206577us
I20260709 12:18:11.634794 26523 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:11.764878 13261 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:18:11.742048 26733 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 46.24, Last appended by leader: 24, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:11.767894 26943 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.511s	user 0.000s	sys 0.001s
I20260709 12:18:11.634251 26916 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:11.739205 26780 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:11.783615 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:11.786706 26945 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.461s	user 0.002s	sys 0.000s
I20260709 12:18:11.792078 26964 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:11.786842 26939 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:11.794687 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.795655 12644 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:11.796700 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.798062 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.798969 12642 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:11.799934 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.800823 12643 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 27.
I20260709 12:18:11.801973 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.804723 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.805974 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.786978 26668 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:11.807112 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.585242 12646 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:11.807950 12638 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 40 because replica has last-logged OpId of term: 39 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:11.808065 26733 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:11.808588 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.808854 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.550813 26730 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:11.810703 13260 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
I20260709 12:18:11.811877 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.771023 26523 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:11.813391 26523 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:11.819080 12643 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 21. Preceding OpId from leader: term: 40 index: 21. (term mismatch)
I20260709 12:18:11.819577 12643 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 20
I20260709 12:18:11.819938 12643 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 39.21
I20260709 12:18:11.820194 12643 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:11.822670 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.823377 12642 raft_consensus.cc:2376] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40. Re-sending same reply.
I20260709 12:18:11.825043 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.825946 12644 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:11.770967 26884 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:11.827322 12638 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43. Re-sending same reply.
I20260709 12:18:11.751889 26944 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:11.792985 26964 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:11.769147 26918 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:11.880568 13260 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 29. Preceding OpId from leader: term: 42 index: 30. (index mismatch)
I20260709 12:18:11.880718 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:11.889322 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.787905 26780 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:11.890616 12642 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40. Re-sending same reply.
W20260709 12:18:11.874811 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 415928us
I20260709 12:18:11.786873 26954 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:11.751782 26940 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 46.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.503s
I20260709 12:18:11.833340 26953 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:11.764848 13253 rpcz_store.cc:276] 0709 12:18:10.364943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.366171 (+  1228us) service_pool.cc:224] Handling call
0709 12:18:10.366536 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:10.366823 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:18:10.713986 (+347163us) log.cc:844] Serialized 14 byte log entry
0709 12:18:11.718054 (+1004068us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:11.718152 (+    98us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:11.718238 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:11.718346 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:11.718554 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14914,"prepare.queue_time_us":670584,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"thread_start_us":1258414,"threads_started":3,"wal-append.queue_time_us":864454}
W20260709 12:18:11.913174 12461 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.575s (SENT)
I20260709 12:18:11.927300 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:11.914059 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:11.943868 26951 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:11.944521 26951 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:11.944950 26951 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:11.951731 12568 pending_rounds.cc:85] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 26
I20260709 12:18:11.952267 12568 pending_rounds.cc:107] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 44.27
I20260709 12:18:11.952576 12568 raft_consensus.cc:2889] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:11.961059 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:11.909457 26780 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:11.905687 26899 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.073s
I20260709 12:18:11.972239 12642 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 38 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 20.
I20260709 12:18:11.889274 26947 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.122s
I20260709 12:18:11.894788 26944 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.009225 26964 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:11.881326 26916 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.004197 26818 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.010800 26818 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.011957 26916 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.014675 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:12.043388 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.043938 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.045776 12639 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 21. Preceding OpId from leader: term: 43 index: 22. (index mismatch)
I20260709 12:18:12.045956 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.047472 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.048435 12637 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:12.043514 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:11.890128 26873 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.074s
I20260709 12:18:12.050155 12643 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:12.055070 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:12.066349 12643 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:11.980997 26959 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.087918 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.096830 26694 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:12.097297 26903 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.887s	user 0.009s	sys 0.003s
I20260709 12:18:12.097806 26903 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.099442 26644 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:11.981562 26925 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.143s	user 0.003s	sys 0.005s
I20260709 12:18:12.100281 26959 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.125735 26644 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.127488 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.141399 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.142624 12642 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:12.143216 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2753) took 1343 ms. Trace:
I20260709 12:18:12.143862 12563 rpcz_store.cc:276] 0709 12:18:10.799648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.799866 (+   218us) service_pool.cc:224] Handling call
0709 12:18:10.830705 (+ 30839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:11.383125 (+552420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.142801 (+759676us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:12.143116 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1268748160}
I20260709 12:18:12.053449 26944 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.167236 26644 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.167438 26773 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:12.169993 12638 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
W20260709 12:18:12.128875 26701 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.339s	user 0.011s	sys 0.005s
I20260709 12:18:12.172427 26701 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.175573 26701 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:12.175277 26802 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.639s	user 0.012s	sys 0.004s
I20260709 12:18:12.176542 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:12.177326 12641 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:12.179276 26903 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.181568 12563 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:12.182847 26523 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.192904 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.245024 12642 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42. Re-sending same reply.
I20260709 12:18:12.196031 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.197553 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.247696 12644 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 41 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 27.
I20260709 12:18:12.092150 26818 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.205551 26959 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:12.167665 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 254514us
W20260709 12:18:12.214779 26948 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.778s	user 0.000s	sys 0.018s
W20260709 12:18:12.218575 26769 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.436s	user 0.013s	sys 0.004s
I20260709 12:18:11.953866 26953 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:12.226161 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:12.287555 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.248271 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:12.246522 12638 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43. Re-sending same reply.
W20260709 12:18:12.252398 12461 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.616s (SENT)
I20260709 12:18:12.263999 26944 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:12.264837 26903 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:12.279745 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.030s	sys 0.062s
W20260709 12:18:12.305006 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53001us
W20260709 12:18:12.307456 12461 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.683s (SENT)
I20260709 12:18:12.308003 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:12.308796 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.030s	sys 0.063s
I20260709 12:18:12.290235 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:12.316084 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.026s	sys 0.075s
W20260709 12:18:12.330909 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.026s	sys 0.076s
W20260709 12:18:12.331650 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 880516us
W20260709 12:18:12.332094 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.052s	sys 0.075s
W20260709 12:18:12.335105 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.053s	sys 0.075s
I20260709 12:18:12.303393 26769 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:12.335139 26698 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.038s	sys 0.084s
W20260709 12:18:12.352636 26928 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.516s	user 0.000s	sys 0.018s
I20260709 12:18:12.319742 26903 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.317934 26944 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:12.354302 26733 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.544s	user 0.014s	sys 0.000s
W20260709 12:18:12.369424 26538 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.584s	user 0.014s	sys 0.001s
I20260709 12:18:12.380956 26944 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.373432 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1782) took 1321 ms. Trace:
I20260709 12:18:12.377072 12640 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:12.383972 13259 rpcz_store.cc:276] 0709 12:18:11.045077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.045690 (+   613us) service_pool.cc:224] Handling call
0709 12:18:11.144615 (+ 98925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:11.144881 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:11.744943 (+600062us) log.cc:844] Serialized 14 byte log entry
0709 12:18:12.365523 (+620580us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:12.365598 (+    75us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:12.365681 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.365767 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:12.365933 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60456,"prepare.queue_time_us":97838,"prepare.run_cpu_time_us":2434,"prepare.run_wall_time_us":7639,"raft.queue_time_us":152440,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":1122157,"threads_started":3,"wal-append.queue_time_us":569524}
I20260709 12:18:12.388391 26884 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:12.370615 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:12.368765 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.032s	sys 0.071s
W20260709 12:18:12.389967 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.032s	sys 0.071s
W20260709 12:18:12.390533 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 588698us
I20260709 12:18:12.390985 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.392447 26733 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:12.374326 26934 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.048s
I20260709 12:18:12.441874 26523 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:12.454344 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.709s (SENT)
I20260709 12:18:12.454845 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.444500 13259 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:18:12.456099 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:12.429742 26818 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.446614 13253 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 26. Preceding OpId from leader: term: 42 index: 26. (term mismatch)
I20260709 12:18:12.459519 13253 pending_rounds.cc:85] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 25
I20260709 12:18:12.460060 13253 pending_rounds.cc:107] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 41.26
I20260709 12:18:12.460434 13253 raft_consensus.cc:2889] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:12.385044 26953 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.394008 26698 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:12.429334 26832 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.456781 13255 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:12.445501 13254 raft_consensus.cc:1217] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Deduplicated request from leader. Original: 38.24->[40.25-44.27]   Dedup: 40.25->[41.26-44.27]
I20260709 12:18:12.458004 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:12.466805 13254 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: [40.25-44.27]
I20260709 12:18:12.434209 26972 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:12.486773 26940 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:12.487600 26940 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.489355 26832 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.492645 26832 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.495036 26903 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.464413 26944 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:12.462359 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:12.496368 26698 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.497828 26733 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:12.498498 26733 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.499590 26769 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.500895 26733 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:12.501667 26818 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.503516 26944 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.504122 26944 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.504745 26954 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:12.451375 26951 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.003s
I20260709 12:18:12.505604 26954 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.506093 26951 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.509739 26780 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.510213 26940 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.511474 12645 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:12.410629 26538 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.473560 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:12.518110 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:12.547796 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.559505 13258 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: [39.25-39.25]
I20260709 12:18:12.566655 13256 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:12.501543 26733 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:12.573753 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 182852us
W20260709 12:18:12.546923 26950 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.995s	user 0.002s	sys 0.002s
W20260709 12:18:12.576920 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.579391 13256 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.24-40.24]
I20260709 12:18:12.541528 26944 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.544008 26956 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:12.586553 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.582748 26956 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:12.582698 26972 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.004s
I20260709 12:18:12.588471 26956 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.589550 26972 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:12.590188 26698 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.591053 26954 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.591729 26818 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:12.594587 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 142913us
I20260709 12:18:12.580690 26934 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:12.606318 12536 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.720s (SENT)
I20260709 12:18:12.613022 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.610212 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.615280 12643 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 34.
I20260709 12:18:12.580710 26733 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.610832 12638 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:18:12.581931 26951 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.618628 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.619519 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.620066 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.617667 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:12.618306 12536 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.740s (SENT)
I20260709 12:18:12.623698 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.620333 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:12.627945 26954 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:12.625609 26698 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:12.641824 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.642827 12639 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 42 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 27.
I20260709 12:18:12.636564 12643 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 42 index: 28. (index mismatch)
I20260709 12:18:12.637835 12646 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:12.643857 12646 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:12.637437 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:12.597996 26972 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:12.646692 13256 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34.
I20260709 12:18:12.648824 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.659087 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:12.679394 13260 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:12.645175 12646 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.659670 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.623142 26956 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:12.680141 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.685434 12646 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:12.656036 26698 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:12.659009 26780 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:12.626346 26934 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:12.691219 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69424.6us
I20260709 12:18:12.696280 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:12.697091 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78827.7us
W20260709 12:18:12.697788 12536 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.762s (SENT)
I20260709 12:18:12.661957 26995 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:12.681720 26953 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:12.698407 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80855.8us
I20260709 12:18:12.662395 26654 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 29, Time since last communication: 0.033s
I20260709 12:18:12.702477 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.702225 26972 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 46.20, Last appended by leader: 20, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.703353 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.703231 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:12.715371 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.705474 26972 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:12.718554 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.719414 12645 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:12.699649 26996 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:12.726061 13260 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:18:12.727448 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.731494 12570 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:12.732412 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5753) took 1590 ms. Trace:
I20260709 12:18:12.732754 12570 rpcz_store.cc:276] 0709 12:18:11.142277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.142497 (+   220us) service_pool.cc:224] Handling call
0709 12:18:12.731079 (+1588582us) spinlock_profiling.cc:243] Waited 9.67 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:18:12.731321 (+   242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:12.732119 (+   798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.732348 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22230912}
I20260709 12:18:12.734117 12643 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 44 because replica has last-logged OpId of term: 41 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 20.
I20260709 12:18:12.702947 26953 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:12.741600 26994 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:18:12.702503 26995 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.741814 26453 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:12.747694 26453 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:12.748699 26453 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.749397 26934 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.749898 26453 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:12.723547 26538 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.742332 27001 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:12.758690 27001 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.759146 27001 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:12.763293 26995 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:12.751608 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5309) took 1187 ms. Trace:
I20260709 12:18:12.764057 12641 rpcz_store.cc:276] 0709 12:18:11.556327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.620385 (+ 64058us) service_pool.cc:224] Handling call
0709 12:18:12.743411 (+1123026us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54002c6354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:12.743536 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1003422464}
I20260709 12:18:12.765395 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.765564 26961 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:12.766053 26961 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:12.767048 26961 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.25, Last appended by leader: 25, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.768337 26961 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:12.769793 26733 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:12.770532 26918 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:12.771217 26918 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:12.779731 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2163) took 1327 ms. Trace:
W20260709 12:18:12.781831 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 491703us
I20260709 12:18:12.785950 26976 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:12.786509 26953 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.786674 26976 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:12.787612 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 480162us
I20260709 12:18:12.787834 26976 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.791708 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:12.792239 26352 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:12.792460 12563 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:12.792810 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:12.794027 12567 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 48
I20260709 12:18:12.794690 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:12.795002 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.795434 12572 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:12.795708 12640 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:18:12.796540 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44843us
I20260709 12:18:12.798477 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4207) took 1289 ms. Trace:
I20260709 12:18:12.798925 12497 rpcz_store.cc:276] 0709 12:18:11.508912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.509162 (+   250us) service_pool.cc:224] Handling call
0709 12:18:11.509655 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:11.509987 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:11.934878 (+424891us) log.cc:844] Serialized 14 byte log entry
0709 12:18:12.798051 (+863173us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:12.798119 (+    68us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:12.798186 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.798269 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:12.798410 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":465712,"prepare.queue_time_us":27985,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":65696,"thread_start_us":798886,"threads_started":2,"wal-append.queue_time_us":518122}
I20260709 12:18:12.803117 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.803730 12495 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:18:12.805539 26523 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:12.806066 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.806507 26523 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.807495 26884 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:12.807677 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.808130 26884 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.809830 26965 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:12.810086 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.810484 26965 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.811851 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.812891 12488 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 44 because replica has last-logged OpId of term: 42 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 30.
I20260709 12:18:12.812968 26965 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.813199 26730 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:12.797479 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:12.825183 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97211.9us
I20260709 12:18:12.830166 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:12.831562 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:12.814779 26951 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:12.815821 26977 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 41 is higher.
I20260709 12:18:12.816679 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:12.846684 12488 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:12.817819 26906 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.818894 26868 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.820194 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.820040 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:12.847965 12568 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:12.820328 26769 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:12.821691 26496 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.821780 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:12.822100 12571 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:12.825800 26417 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:12.796695 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.826753 26924 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.805619 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:12.758036 26962 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.846s	user 0.013s	sys 0.001s
I20260709 12:18:12.752297 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.809296 26523 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:12.833096 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.834332 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:12.840296 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:18:12.890055 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55803.2us
I20260709 12:18:12.850332 26951 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.892266 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:12.851240 26941 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:12.851586 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:12.852639 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.895366 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.878397 12565 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:18:12.884498 12646 raft_consensus.cc:2368] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
W20260709 12:18:12.885342 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 182131us
I20260709 12:18:12.780355 12493 rpcz_store.cc:276] 0709 12:18:11.452352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.452577 (+   225us) service_pool.cc:224] Handling call
0709 12:18:11.453046 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:11.453308 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:11.457848 (+  4540us) log.cc:844] Serialized 14 byte log entry
0709 12:18:12.309759 (+851911us) log.cc:844] Serialized 14 byte log entry
0709 12:18:12.779134 (+469375us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b1800504760. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:18:12.779297 (+   163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:12.779374 (+    77us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:12.779425 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.779508 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:12.779665 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16954,"prepare.queue_time_us":5710,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":6295,"raft.queue_time_us":1287,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":5566,"spinlock_wait_cycles":22079488,"thread_start_us":851082,"threads_started":1,"wal-append.queue_time_us":923926}
I20260709 12:18:12.892632 12637 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:12.894542 12643 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:12.897501 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.853814 26903 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:12.851244 26884 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:12.882901 26417 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.919349 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:12.926929 12646 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 34. Re-sending same reply.
I20260709 12:18:12.922016 12493 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:12.922358 12572 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:12.907629 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.885433 26924 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:12.924142 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:12.936883 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:12.851244 26953 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.056s
I20260709 12:18:12.947191 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.947993 12495 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:12.926129 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:12.940657 12497 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:12.898686 26538 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:18:12.881588 26769 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.923501 26969 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.016s
I20260709 12:18:12.961964 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.945808 12640 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:12.955511 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166721us
I20260709 12:18:12.964478 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:12.880406 26906 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:12.946271 26938 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:12.931167 26932 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:12.951808 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:12.974457 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.975441 12493 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 35.
I20260709 12:18:12.977185 26932 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.977998 26932 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:12.929284 12567 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:12.901345 26959 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.061s
I20260709 12:18:12.965946 12488 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:12.934733 26417 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:12.964824 12461 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.726s (SENT)
I20260709 12:18:12.934857 26868 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.012s
I20260709 12:18:12.882452 26496 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.985944 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:12.964749 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:12.963315 26884 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:12.955323 26453 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:12.959380 26538 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:12.931532 26903 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:12.967355 26977 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:12.991595 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:13.026924 26903 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:13.027168 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:13.028460 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:13.029464 26611 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:13.029896 26611 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:13.030289 26611 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:12.988451 26965 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:12.971741 26938 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:13.039397 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131859us
W20260709 12:18:13.016587 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
W20260709 12:18:13.022158 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.023866 26538 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:12.958941 26956 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.033s
W20260709 12:18:13.005322 12461 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.825s (SENT)
I20260709 12:18:13.043601 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:13.015929 27005 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:18:13.042505 12609 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42.
I20260709 12:18:13.053246 12609 leader_election.cc:403] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:18:13.046886 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:13.050601 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.055382 12495 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:18:13.046274 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5763) took 1095 ms. Trace:
I20260709 12:18:12.970769 26330 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.062s
I20260709 12:18:13.003859 26884 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:13.064994 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.060679 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.053838 26901 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:13.056231 12569 rpcz_store.cc:276] 0709 12:18:11.950702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.950914 (+   212us) service_pool.cc:224] Handling call
0709 12:18:11.960809 (+  9895us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:12.326411 (+365602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:12.326442 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:12.327262 (+   820us) log.cc:844] Serialized 52 byte log entry
0709 12:18:12.529297 (+202035us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:12.533420 (+  4123us) log.cc:844] Serialized 14 byte log entry
0709 12:18:12.533712 (+   292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:12.760588 (+226876us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b18003d14a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:13.045704 (+285116us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b5400180e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:13.045800 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:13.045877 (+    77us) raft_consensus.cc:1685] finished
0709 12:18:13.045900 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:13.046177 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8925,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":2447,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":285,"spinlock_wait_cycles":882035328,"thread_start_us":201494,"threads_started":1,"wal-append.queue_time_us":262299}
I20260709 12:18:13.042927 26938 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:13.067924 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:13.065924 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.087121 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:13.089697 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49126.1us
I20260709 12:18:13.090873 12609 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34.
I20260709 12:18:13.091262 12609 leader_election.cc:403] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:18:13.093997 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.060357 26883 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:13.099403 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.069784 26884 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:13.064095 27003 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.102658 12611 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.034359 27001 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.057431 26868 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 23, Time since last communication: 0.041s
I20260709 12:18:13.109532 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:13.087268 26903 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:13.113184 26616 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
W20260709 12:18:13.110148 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81760.3us
I20260709 12:18:13.134610 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.138722 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.100950 26907 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:18:13.138819 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147676us
I20260709 12:18:13.098470 26780 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42.
I20260709 12:18:13.104292 26733 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:18:13.144582 12461 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.930s (SENT)
I20260709 12:18:13.153435 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:13.103775 26995 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34.
I20260709 12:18:13.104089 26996 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:13.164341 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.111059 26539 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.047s
W20260709 12:18:13.170737 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 136) took 25892 ms (client timeout 15000 ms). Trace:
I20260709 12:18:13.153151 26733 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.171612 26733 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:13.138499 26883 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:13.101714 27004 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.061s
I20260709 12:18:13.109509 27000 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.065s
I20260709 12:18:13.107887 26989 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.077s
I20260709 12:18:13.157430 26453 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:13.152434 26616 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:13.208427 12495 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 48. Current term is 49. Ops: [48.32-48.32]
I20260709 12:18:13.209754 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.163966 26951 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:13.111173 27003 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.179277 26901 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:13.189604 27004 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 34.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:13.225180 26946 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:13.159687 26977 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.058s
W20260709 12:18:13.224639 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80078us
I20260709 12:18:13.227532 12496 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:13.230451 12641 raft_consensus.cc:2410] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 40 because replica has last-logged OpId of term: 39 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 24.
I20260709 12:18:13.224503 26883 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.25, Last appended by leader: 25, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:13.264478 27007 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:18:13.305166 26901 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.271351 26931 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:13.309314 27014 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260709 12:18:13.295014 12461 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.999s (SENT)
I20260709 12:18:13.314545 27003 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:13.316512 26611 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.229786 26694 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:18:13.171350 12387 rpcz_store.cc:269] 0709 12:17:47.278383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:50.670272 (+3391889us) service_pool.cc:224] Handling call
0709 12:18:13.170375 (+22500103us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:07.197807 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:07.358624 (+160817us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:07.358687 (+    63us) write_op.cc:435] Acquired schema lock
0709 12:18:07.358709 (+    22us) tablet.cc:662] Decoding operations
0709 12:18:07.489746 (+131037us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:07.537044 (+ 47298us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:07.539735 (+  2691us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:18:07.938183 (+398448us) tablet.cc:701] Row locks acquired
0709 12:18:07.938242 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:18:07.938447 (+   205us) write_op.cc:270] Start()
0709 12:18:07.938616 (+   169us) write_op.cc:276] Timestamp: P: 1783599487938378 usec, L: 0
0709 12:18:07.938642 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:18:07.978542 (+ 39900us) log.cc:844] Serialized 6665 byte log entry
0709 12:18:08.516922 (+538380us) op_driver.cc:464] REPLICATION: finished
0709 12:18:08.558554 (+ 41632us) write_op.cc:301] APPLY: starting
0709 12:18:08.558667 (+   113us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:09.436292 (+877625us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:09.436497 (+   205us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:09.906018 (+469521us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:09.906442 (+   424us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0709 12:18:09.906515 (+    73us) write_op.cc:312] APPLY: finished
0709 12:18:09.907141 (+   626us) log.cc:844] Serialized 156 byte log entry
0709 12:18:10.042264 (+135123us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:10.166596 (+124332us) write_op.cc:454] Released schema lock
0709 12:18:10.166691 (+    95us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:10.167301 (+   610us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:10.167337 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4159452,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":22199,"apply.run_cpu_time_us":77647,"apply.run_wall_time_us":1608750,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75119,"num_ops":17,"prepare.queue_time_us":232801,"prepare.run_cpu_time_us":37179,"prepare.run_wall_time_us":1185021,"raft.queue_time_us":86745,"raft.run_cpu_time_us":3750,"raft.run_wall_time_us":76762,"replication_time_us":553725,"thread_start_us":403194,"threads_started":5,"wal-append.queue_time_us":508386,"wal-append.run_cpu_time_us":25402,"wal-append.run_wall_time_us":2149202}]]}
I20260709 12:18:13.335901 27001 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:13.336081 26938 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.338663 26611 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.313047 26616 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:13.340513 26918 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.569s	user 0.008s	sys 0.004s
I20260709 12:18:13.341674 26538 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.325603 26931 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.325587 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:13.273680 26496 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:13.332378 26972 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.610s	user 0.008s	sys 0.005s
W20260709 12:18:13.296115 27002 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 12:18:13.352794 26951 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:18:13.358158 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 290353us
W20260709 12:18:13.359130 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.011s	sys 0.002s
W20260709 12:18:13.364097 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.012s	sys 0.002s
I20260709 12:18:13.367300 26932 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.368192 26931 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:13.368826 26961 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.004s
I20260709 12:18:13.341218 26918 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.369588 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2751) took 3049 ms. Trace:
I20260709 12:18:13.370023 12566 rpcz_store.cc:276] 0709 12:18:10.320290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:10.500288 (+179998us) service_pool.cc:224] Handling call
0709 12:18:10.520951 (+ 20663us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:10.530239 (+  9288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:10.530272 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:10.548672 (+ 18400us) log.cc:844] Serialized 26 byte log entry
0709 12:18:10.735523 (+186851us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:10.735571 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:10.735660 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:11.862224 (+1126564us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:12.220892 (+358668us) raft_consensus.cc:1685] finished
0709 12:18:12.220917 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:13.369408 (+1148491us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b5400181d50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:13.369529 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13882,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":6414,"spinlock_wait_cycles":75272960,"thread_start_us":127136,"threads_started":1,"wal-append.queue_time_us":266681,"wal-append.run_cpu_time_us":15768,"wal-append.run_wall_time_us":2384298}
I20260709 12:18:13.371415 26918 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.371309 26924 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.374732 26417 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.376567 26924 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.376817 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5754) took 2234 ms. Trace:
W20260709 12:18:13.377660 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82701.9us
W20260709 12:18:13.378381 12461 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.013s (SENT)
I20260709 12:18:13.377153 12564 rpcz_store.cc:276] 0709 12:18:11.142658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:11.142877 (+   219us) service_pool.cc:224] Handling call
0709 12:18:12.773889 (+1631012us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:18:12.774111 (+   222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:12.774546 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:18:12.774886 (+   340us) log.cc:844] Serialized 14 byte log entry
0709 12:18:13.052402 (+277516us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:13.083347 (+ 30945us) log.cc:844] Serialized 26 byte log entry
0709 12:18:13.087644 (+  4297us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:13.087710 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:13.087826 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:13.376464 (+288638us) raft_consensus.cc:1685] finished
0709 12:18:13.376592 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:13.376735 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3056,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":90110,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":9397,"spinlock_wait_cycles":24401792,"thread_start_us":276969,"threads_started":2,"wal-append.queue_time_us":234680}
W20260709 12:18:13.389827 12491 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.010s	sys 0.004s
W20260709 12:18:13.391494 12491 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.010s	sys 0.004s
I20260709 12:18:13.392686 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:13.420603 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.426616 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:13.430400 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52035.1us
W20260709 12:18:13.431385 12461 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.039s (SENT)
I20260709 12:18:13.352248 26884 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.435357 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.418308 26918 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:13.440855 12461 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.129s (SENT)
I20260709 12:18:13.441687 26417 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:13.451921 26918 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:13.453178 12461 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.183s (SENT)
I20260709 12:18:13.453886 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:13.418880 27016 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.054s
I20260709 12:18:13.455072 26918 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.422341 26938 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.445911 26733 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.459741 26733 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:13.449000 26884 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.450091 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.470803 12569 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:13.451015 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 486328us
I20260709 12:18:13.471647 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.437132 12564 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 42 index: 28. (index mismatch)
I20260709 12:18:13.476181 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:13.474959 27027 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:13.480309 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.481439 27027 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:13.482231 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:13.482865 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:13.484006 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:13.485318 26884 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:13.490338 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:13.485836 12461 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.200s (SENT)
I20260709 12:18:13.492596 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.439008 26538 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.492602 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.506954 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:13.515105 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.516148 12564 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 44 because replica has last-logged OpId of term: 42 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 30.
W20260709 12:18:13.529196 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.513209 26965 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:13.541934 12570 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:13.548318 12566 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 48. Ops: []
I20260709 12:18:13.554513 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:13.555899 12461 connection.cc:256] RPC call timeout handler was delayed by -1.6458394710002722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2456s for a response
I20260709 12:18:13.531945 26951 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:13.557807 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:13.556457 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:13.531599 27029 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:18:13.559902 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67406.3us
I20260709 12:18:13.559535 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:13.563329 12461 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.278s (SENT)
I20260709 12:18:13.564127 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:13.562004 26616 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.010s
I20260709 12:18:13.560281 26951 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.571455 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.573746 26951 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:13.575601 26616 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:13.557350 27024 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:13.580533 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.581738 26965 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.582646 26918 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:18:13.584762 27027 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:18:13.585865 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.582171 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.589279 27027 raft_consensus.cc:2764] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election decision vote started in defunct term 42: won
I20260709 12:18:13.598457 26927 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:13.599128 26927 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:13.600369 26927 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:13.606608 27022 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.390s	user 0.004s	sys 0.000s
I20260709 12:18:13.611148 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:13.619558 13260 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:18:13.618496 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:13.612104 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:13.615600 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44305us
I20260709 12:18:13.616441 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:13.616465 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.621652 12492 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 46 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 26.
I20260709 12:18:13.585619 27026 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:13.623559 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.629528 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:13.630267 13257 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:18:13.629060 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65799.1us
I20260709 12:18:13.639814 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:13.640024 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4291) took 1001 ms. Trace:
W20260709 12:18:13.640769 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 472387us
I20260709 12:18:13.642670 13253 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 34. Current term is 35. Ops: []
I20260709 12:18:13.649125 12611 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 33 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 33. Current term is 34.
I20260709 12:18:13.627261 27027 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:13.628733 26927 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:13.636209 26933 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:13.640558 26924 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:13.656239 26968 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:13.673038 13260 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:13.674293 13254 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [40.25-40.25]
I20260709 12:18:13.674638 26932 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.667709 13261 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [35.25-36.26]
W20260709 12:18:13.685477 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.690662 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:13.684784 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.695935 26906 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.700605 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.676083 26933 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:13.702819 26933 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:13.704365 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.706502 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:13.707674 13258 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:13.708483 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90529us
I20260709 12:18:13.709594 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.711676 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.713218 26924 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:18:13.714088 26903 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:18:13.723189 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:13.725414 26417 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.728035 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:13.733731 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.734908 12570 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:18:13.663028 26694 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:13.737885 26694 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:13.686036 26968 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:13.739620 26968 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:13.680438 26884 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:13.742869 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:13.743299 27027 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:13.743863 12639 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:13.746361 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57192.5us
I20260709 12:18:13.746942 12568 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:13.708920 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:13.748477 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:13.748469 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:13.663208 27000 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:13.643127 12637 rpcz_store.cc:276] 0709 12:18:12.644101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.932421 (+288320us) service_pool.cc:224] Handling call
0709 12:18:12.933096 (+   675us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:12.939919 (+  6823us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:12.939954 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:12.960661 (+ 20707us) log.cc:844] Serialized 52 byte log entry
0709 12:18:13.071218 (+110557us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:13.115659 (+ 44441us) log.cc:844] Serialized 14 byte log entry
0709 12:18:13.161844 (+ 46185us) log.cc:844] Serialized 14 byte log entry
0709 12:18:13.581435 (+419591us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b180036bf80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:18:13.582389 (+   954us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b180036bf80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:13.633294 (+ 50905us) spinlock_profiling.cc:243] Waited 7.48 ms on lock 0x7b180036bf80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:18:13.639521 (+  6227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:13.639638 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:13.639683 (+    45us) raft_consensus.cc:1685] finished
0709 12:18:13.639696 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:13.639927 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":514735,"prepare.queue_time_us":53835,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":99,"raft.queue_time_us":538110,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":21634,"spinlock_wait_cycles":48494592,"thread_start_us":61903,"threads_started":1,"wal-append.queue_time_us":176806}
I20260709 12:18:13.705497 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:13.739506 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:13.758690 12489 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:13.759522 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 656557us
I20260709 12:18:13.754087 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:13.769922 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:13.770028 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77281us
W20260709 12:18:13.777197 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92508.8us
I20260709 12:18:13.781454 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:13.782217 12644 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42. Re-sending same reply.
I20260709 12:18:13.782581 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:13.783344 12640 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:13.785465 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:13.726557 26417 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.789533 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:13.741240 26694 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:13.814003 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:13.817205 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:13.818368 12566 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:13.820731 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:13.772713 26903 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:13.689620 26539 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.119s
W20260709 12:18:13.824019 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47653.9us
I20260709 12:18:13.788836 26330 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:13.745067 27027 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.843168 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:13.844019 12569 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:13.845280 12637 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:13.846903 12646 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 41 index: 28. (term mismatch)
I20260709 12:18:13.847424 12646 pending_rounds.cc:85] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 27
I20260709 12:18:13.847679 12646 pending_rounds.cc:107] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 40.28
I20260709 12:18:13.847947 12646 raft_consensus.cc:2889] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:13.814082 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.850562 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:13.854975 12641 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 35: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:13.855254 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:13.858228 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:13.774191 26944 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.066s
I20260709 12:18:13.773941 26733 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.065s
I20260709 12:18:13.816337 27001 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.036s
I20260709 12:18:13.827032 26802 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.036s
W20260709 12:18:13.863870 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49926.9us
I20260709 12:18:13.872550 13257 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:13.833750 26903 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.826310 27016 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.037s
W20260709 12:18:13.886943 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 175343us
I20260709 12:18:13.892267 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.868460 12639 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:13.874946 27004 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:13.901855 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:13.896998 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.904779 12568 raft_consensus.cc:2376] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39. Re-sending same reply.
I20260709 12:18:13.913816 13257 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:13.914495 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:13.917490 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:13.918169 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.918907 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:13.925299 12536 leader_election.cc:400] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
W20260709 12:18:13.899528 27036 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.230s	user 0.000s	sys 0.002s
I20260709 12:18:13.890918 26868 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:18:13.894771 26903 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:13.927235 12565 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:18:13.932691 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:13.933046 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:13.946127 26802 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:13.884493 26453 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.046s
I20260709 12:18:13.946687 26935 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:13.904510 27004 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:13.965200 26951 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:13.969677 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:13.890692 26780 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.039s
I20260709 12:18:13.924405 26944 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:13.985764 12566 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:13.990851 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.879621 26901 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.042s
I20260709 12:18:13.993741 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:13.971498 27013 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.782s	user 0.002s	sys 0.000s
I20260709 12:18:13.998948 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:14.007871 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:13.980466 26868 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:14.018836 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.015425 26453 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 34.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:14.024099 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4294) took 1378 ms. Trace:
I20260709 12:18:14.025148 26951 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.024770 12645 rpcz_store.cc:276] 0709 12:18:12.645700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:12.727183 (+ 81483us) service_pool.cc:224] Handling call
0709 12:18:14.023818 (+1296635us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:14.024009 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1452169600}
I20260709 12:18:14.028730 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.035154 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
W20260709 12:18:14.045043 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:13.995224 26935 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:14.002839 26330 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:18:14.059543 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68985us
I20260709 12:18:14.048820 12569 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:14.001995 27004 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:14.053498 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:14.066857 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.055399 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:14.022010 26934 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.022768 26698 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.086624 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:14.062335 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.081382 26769 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:14.021358 26901 consensus_queue.cc:1243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 46.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:18:14.076781 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.097047 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:14.057597 26330 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:14.115576 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:14.090795 26934 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.090687 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:14.050066 26780 consensus_queue.cc:1243] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 49.32 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 31
I20260709 12:18:14.049698 26956 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 43.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.048s
I20260709 12:18:14.058007 26935 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:14.051761 26906 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:14.087703 26453 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:14.083591 26730 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:14.149278 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:14.048125 26884 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:14.270390 12565 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:18:14.105415 26951 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:14.277107 12569 raft_consensus.cc:1217] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Deduplicated request from leader. Original: 38.23->[40.24-40.24]   Dedup: 40.24->[]
I20260709 12:18:14.290932 12569 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.24-40.24]
I20260709 12:18:14.278837 12568 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:14.280120 12489 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.521s	user 0.013s	sys 0.000s
I20260709 12:18:14.292464 12489 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:14.136930 26330 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:14.105959 26868 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:14.300779 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 292950us
W20260709 12:18:14.418843 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.000s	sys 0.029s
I20260709 12:18:14.251273 26901 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:14.105343 27028 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.200s
I20260709 12:18:14.247635 26932 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.301697 27001 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:14.266826 26906 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:14.093869 26698 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.451246 27001 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:14.363796 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 446341us
I20260709 12:18:14.272264 27045 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:14.362010 26802 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:14.374243 26934 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:14.394586 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:14.278574 26730 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:14.267182 26453 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:14.295274 27029 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:14.252331 26977 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:14.126684 26417 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:14.421475 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.000s	sys 0.029s
W20260709 12:18:14.422410 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 343812us
I20260709 12:18:14.484871 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:14.422549 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 432447us
W20260709 12:18:14.426484 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.451562 26906 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:14.424906 26933 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.001s
W20260709 12:18:14.474669 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 426432us
I20260709 12:18:14.453729 26698 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:14.479739 26453 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:14.490386 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:14.467221 26802 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:14.497551 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 421971us
I20260709 12:18:14.477659 27053 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:14.449509 26932 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.503309 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:14.498400 27057 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:14.512564 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:14.485222 26934 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:14.521318 27023 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.305s	user 0.001s	sys 0.014s
I20260709 12:18:14.524205 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:14.525755 27000 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:18:14.526999 27052 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:18:14.536358 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:14.489564 26977 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.514330 27063 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:14.515012 27057 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:14.548612 27043 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
W20260709 12:18:14.557605 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56635.8us
I20260709 12:18:14.531728 26352 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:14.559111 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44424.4us
I20260709 12:18:14.513481 26769 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.039s
I20260709 12:18:14.562080 27028 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:14.563225 27051 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:18:14.562745 27028 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.562603 27057 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.564249 27057 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:14.551501 27000 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:14.566617 27000 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.567111 27028 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.561204 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:14.568173 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55744us
I20260709 12:18:14.568902 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:14.567898 27061 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:14.557745 27001 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.574335 27001 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:18:14.575229 27001 consensus_queue.cc:1243] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 45.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:18:14.575817 27001 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:14.576232 27001 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:14.576640 27001 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:14.551805 26884 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:14.578145 26884 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.580917 26884 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:14.567831 27000 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:14.582861 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.563231 26769 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:18:14.563865 27041 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:18:14.585618 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.586181 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.586825 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:14.494407 26933 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.516929 26951 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:18:14.551192 26955 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.035s
I20260709 12:18:14.584903 27062 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.004s
I20260709 12:18:14.620077 26932 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:14.621088 26352 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:14.622624 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:14.527767 26730 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.626626 13254 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: []
W20260709 12:18:14.623471 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 133167us
I20260709 12:18:14.628484 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:14.518098 27053 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.581568 26884 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:14.594564 27028 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:14.554148 26934 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.615051 26769 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:14.640588 26933 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:14.636475 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:14.612696 27027 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:14.622378 26955 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:14.646910 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:14.661096 12489 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:14.649864 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:14.573441 27004 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.554086 26802 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 48.32, Last appended by leader: 32, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.656917 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:14.596664 27045 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.040s
W20260709 12:18:14.664487 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94345.5us
W20260709 12:18:14.667857 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96040.1us
I20260709 12:18:14.645231 27028 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:14.671777 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:14.690467 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:14.693203 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.698230 13255 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:14.698787 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:14.653777 27059 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:18:14.664008 26884 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:14.632742 27065 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.063s
I20260709 12:18:14.664285 27071 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:14.660647 26955 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:14.707811 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:14.721136 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.710207 13253 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 34 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:18:14.710537 12489 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:14.691581 27028 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:14.666934 27027 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:14.688759 27069 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:14.724678 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:14.725126 12572 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 48 index: 26. (index mismatch)
I20260709 12:18:14.670181 26780 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.31, Next index: 32, Last known committed idx: 29, Time since last communication: 0.047s
I20260709 12:18:14.737881 26933 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election won for term 43
W20260709 12:18:14.716951 26944 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.695s	user 0.011s	sys 0.004s
I20260709 12:18:14.699740 27064 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:18:14.726852 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51467.4us
W20260709 12:18:14.730489 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58800us
I20260709 12:18:14.749579 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:18:14.733510 26968 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.993s	user 0.017s	sys 0.001s
I20260709 12:18:14.755335 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.763430 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:14.717653 27071 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:14.768560 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132853us
I20260709 12:18:14.744145 27061 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:14.776481 27056 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:14.777102 27056 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:14.777554 27056 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:14.779642 27077 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:14.772740 26951 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.010s
W20260709 12:18:14.783385 26903 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.787s	user 0.011s	sys 0.002s
I20260709 12:18:14.783877 26903 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.786198 27073 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:14.788252 26694 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:14.788878 26694 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:14.789397 27053 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.756424 26968 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.790091 26694 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 47.23, Last appended by leader: 23, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.791482 26694 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:14.792213 26968 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.793000 26968 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:18:14.793808 27076 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:18:14.794051 12571 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:14.794687 12567 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:14.753033 26417 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:14.795744 26417 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.798761 26934 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:14.799564 26884 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 45.18, Last appended by leader: 18, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.804317 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:14.805966 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:14.806118 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:14.806650 26972 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:18:14.807106 26972 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:14.809175 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:14.775620 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:14.811978 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:14.812851 13257 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:14.814477 12645 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: [35.25-35.25]
I20260709 12:18:14.816987 12497 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 49 index: 32. (index mismatch)
I20260709 12:18:14.738036 27004 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.741529 26933 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:14.751904 27064 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:14.754701 26944 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:14.824483 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:14.832721 12645 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:14.829974 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66599.5us
I20260709 12:18:14.834231 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:14.834926 12641 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:14.830323 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:14.838146 13255 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [35.25-36.26]
I20260709 12:18:14.843278 12637 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:14.846256 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.847460 12643 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:14.736300 27069 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.851081 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:14.851389 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:14.852473 12638 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:18:14.853271 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.807080 26903 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.862468 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.824704 26968 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 34.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:14.864820 12569 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:14.822592 26938 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:14.886447 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51049.3us
W20260709 12:18:14.866120 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 175828us
I20260709 12:18:14.729743 26769 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:14.841431 27024 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:18:14.887656 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 133205us
W20260709 12:18:14.899747 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.900987 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.837257 27073 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:14.912125 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:14.808282 26972 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.26, Last appended by leader: 26, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.930398 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:14.921571 12496 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:14.926493 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:14.943385 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:14.866412 13257 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
W20260709 12:18:14.948292 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47363.8us
I20260709 12:18:14.852728 26977 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:14.834070 27064 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:14.843557 27079 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:14.866714 12645 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:14.949286 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.950461 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.951143 27076 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:18:14.866591 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.952922 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.891448 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:14.953181 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.952347 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.957286 12495 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: [37.20-37.20]
I20260709 12:18:14.808780 26955 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.991767 12497 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c56dde469374f4da7440f134140670c"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:14.994063 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5528) took 1083 ms. Trace:
I20260709 12:18:14.951392 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 27144fedf0014de09f007ac6cc3f7275 with delete_type TABLET_DATA_TOMBSTONED (TS 32889a507a6b4c74aec421ffbfd1e25d not found in new config with opid_index 19) from {username='slave'} at 127.0.0.1:47032
I20260709 12:18:14.996174 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:14.991487 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.999176 12497 raft_consensus.cc:606] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:15.000591 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:14.999019 12566 rpcz_store.cc:276] 0709 12:18:13.911101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.085605 (+174504us) service_pool.cc:224] Handling call
0709 12:18:14.103647 (+ 18042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:14.527580 (+423933us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:14.527626 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:14.537403 (+  9777us) log.cc:844] Serialized 26 byte log entry
0709 12:18:14.614429 (+ 77026us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:14.900285 (+285856us) log.cc:844] Serialized 14 byte log entry
0709 12:18:14.993596 (+ 93311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:14.993752 (+   156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:14.993810 (+    58us) raft_consensus.cc:1685] finished
0709 12:18:14.993822 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:14.993979 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":102103,"raft.queue_time_us":471413,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":12868,"thread_start_us":727871,"threads_started":3,"wal-append.queue_time_us":93092}
I20260709 12:18:15.000711 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:14.991751 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:15.019119 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:15.021196 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129819us
I20260709 12:18:15.022568 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:15.025123 12572 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:14.919252 27074 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:15.031531 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:14.953795 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:15.039537 12572 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:14.896477 26938 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.977612 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.991487 26934 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:14.915382 27078 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:15.042766 12572 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.26-36.26]
I20260709 12:18:15.001646 26698 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:18:15.057139 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 100099us
W20260709 12:18:15.049345 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:15.022843 27076 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.059723 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:14.958936 27060 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.097s
I20260709 12:18:15.035174 27045 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:15.030896 27079 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:15.034884 26956 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:15.063546 27067 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.109s
W20260709 12:18:15.065990 27072 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:18:15.062142 26654 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:18:15.066422 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.066677 26802 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:14.995548 26933 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.30, Last appended by leader: 30, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:14.895067 27027 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 46.20, Last appended by leader: 20, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.042109 26417 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:15.096216 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 271259us
I20260709 12:18:15.059743 26968 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:14.903051 27071 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:15.115548 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.099612 26944 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.101177 27060 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 34.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:15.125265 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:15.148104 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:15.152205 13262 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:15.127164 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:15.130841 27097 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:15.131772 27085 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:18:15.132308 27066 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:15.105957 26956 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:15.133685 27094 tablet_replica.cc:333] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:18:15.134047 27079 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.127164 27073 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.135977 27069 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:15.130900 27068 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:15.136979 26330 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.716s	user 0.011s	sys 0.004s
I20260709 12:18:15.110899 27077 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:15.148447 13255 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:15.194628 26956 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:15.211412 27001 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.634s	user 0.013s	sys 0.001s
I20260709 12:18:15.211920 27001 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.188987 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:15.213990 27026 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:15.214259 27001 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:15.214615 27026 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:15.215167 27001 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:15.215598 27001 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:15.216131 27001 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:15.215556 27026 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.217140 27026 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:15.189996 13259 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:15.186851 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 347686us
I20260709 12:18:15.214737 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:15.194103 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84261.4us
W20260709 12:18:15.194556 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67489.7us
W20260709 12:18:15.190598 26453 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.672s	user 0.010s	sys 0.005s
I20260709 12:18:15.129112 27074 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:15.147789 26968 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:15.090431 27076 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.221352 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:15.223906 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:15.232046 12488 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:15.236068 13255 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 48 index: 26. (index mismatch)
I20260709 12:18:15.185254 26944 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.238631 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:15.222630 26330 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.247767 12489 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:15.245100 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:15.234575 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:15.257119 12638 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 35.
I20260709 12:18:15.259812 12641 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:15.224864 26733 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:15.230379 27003 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.267017 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:15.268638 27104 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:15.276147 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:15.240446 26954 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.269723 26654 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:18:15.281801 27093 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:18:15.232703 27074 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.284039 12641 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:15.286944 27100 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:15.287664 27080 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.288290 27095 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.088s
I20260709 12:18:15.289966 26968 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 42.31, Last appended by leader: 31, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.293967 26938 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:15.298302 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:15.344065 27098 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:18:15.410279 27103 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.215790 12494 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: [42.30-42.30]
I20260709 12:18:16.216004 27103 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.441671 27107 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:15.498467 27100 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:15.548542 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:15.629942 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:16.223021 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:16.223891 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:16.225142 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.673261 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:15.340590 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.493845 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5532) took 1581 ms. Trace:
I20260709 12:18:15.342549 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2768) took 1485 ms. Trace:
I20260709 12:18:16.302687 12565 rpcz_store.cc:276] 0709 12:18:13.912315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.291830 (+379515us) service_pool.cc:224] Handling call
0709 12:18:14.698501 (+406671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:14.699554 (+  1053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.493498 (+793944us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400245ed0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:15.493749 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748892416}
I20260709 12:18:15.521890 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5792) took 1563 ms. Trace:
I20260709 12:18:16.303397 12570 rpcz_store.cc:276] 0709 12:18:13.857391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.859508 (+  2117us) service_pool.cc:224] Handling call
0709 12:18:13.859948 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:13.860315 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:13.880720 (+ 20405us) log.cc:844] Serialized 14 byte log entry
0709 12:18:15.295321 (+1414601us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b1800323560. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:15.342205 (+ 46884us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:15.342277 (+    72us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:15.342325 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.342379 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:15.342495 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18216,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":125,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":7508,"spinlock_wait_cycles":5958528,"thread_start_us":46564,"threads_started":1,"wal-append.queue_time_us":1506526}
I20260709 12:18:15.567245 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 5801) took 1067 ms. Trace:
I20260709 12:18:16.307766 12571 rpcz_store.cc:276] 0709 12:18:14.499470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.523797 (+ 24327us) service_pool.cc:224] Handling call
0709 12:18:15.566876 (+1043079us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:15.567110 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560149760}
W20260709 12:18:15.662056 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:15.540735 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:15.320181 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:15.576349 12642 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:16.384794 27131 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.385389 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5363) took 1469 ms. Trace:
I20260709 12:18:16.385686 27129 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.385918 12642 rpcz_store.cc:276] 0709 12:18:14.916076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.575675 (+659599us) service_pool.cc:224] Handling call
0709 12:18:15.576096 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.385006 (+808910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.385296 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.386421 27129 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.387892 12642 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [40.21-40.21]
I20260709 12:18:15.407474 12641 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 48 index: 32. Preceding OpId from leader: term: 49 index: 32. (term mismatch)
I20260709 12:18:16.392442 12641 pending_rounds.cc:85] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 31
I20260709 12:18:16.392817 12641 pending_rounds.cc:107] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 48.32
I20260709 12:18:16.393149 12641 raft_consensus.cc:2889] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:15.302328 12644 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
W20260709 12:18:15.329865 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110414us
I20260709 12:18:15.694787 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1792) took 1976 ms. Trace:
I20260709 12:18:15.533310 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:15.559300 26496 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:15.358290 26330 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:15.225180 26453 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.636433 26730 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:15.297572 26868 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:18:15.538555 26883 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:16.422698 26883 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:16.423481 26453 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.424041 26883 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:15.352658 26906 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.852s	user 0.015s	sys 0.000s
I20260709 12:18:16.427875 26906 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.603343 26931 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:16.431429 26931 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.367187 26944 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:15.311408 26954 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.552575 26955 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:16.439666 27129 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.439474 26883 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:15.453430 26956 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.442402 12493 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:16.444000 26956 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.444092 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5707) took 1535 ms. Trace:
W20260709 12:18:15.328889 26935 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.039s	user 0.012s	sys 0.004s
I20260709 12:18:16.437397 26954 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:15.419108 27000 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.836s	user 0.008s	sys 0.008s
I20260709 12:18:15.493086 27001 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.297901 27003 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:15.470934 27028 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.729s	user 0.015s	sys 0.000s
I20260709 12:18:16.476359 27028 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:15.471192 27056 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.004s
W20260709 12:18:15.404831 27063 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.814s	user 0.009s	sys 0.006s
I20260709 12:18:16.481588 27063 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:15.452775 27064 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.194613 27066 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.485909 12564 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:16.487999 27066 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:15.228534 27068 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.489248 12572 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:16.486032 12637 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 46 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 26.
I20260709 12:18:16.490980 27068 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:15.343850 27074 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:15.403553 27076 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:15.338888 27080 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:15.345834 27087 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
W20260709 12:18:15.669129 27088 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.602s	user 0.002s	sys 0.000s
I20260709 12:18:15.202059 27097 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:15.781857 27090 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:16.506559 27090 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.508559 27097 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.509234 27090 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:15.789697 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.012s	sys 0.013s
I20260709 12:18:15.791584 26968 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:15.811040 12496 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:16.509845 27080 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.512984 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5713) took 1567 ms. Trace:
I20260709 12:18:15.827178 27024 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.514117 27024 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.516791 27024 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:15.829005 27029 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.517601 27029 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.520056 27029 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:15.829222 27027 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.520828 27027 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:15.831352 27059 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.521931 27059 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.524452 27059 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:15.832050 27062 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:15.852618 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5798) took 1891 ms. Trace:
I20260709 12:18:16.526127 12567 rpcz_store.cc:276] 0709 12:18:13.960756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.678389 (+717633us) service_pool.cc:224] Handling call
0709 12:18:14.678824 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:14.795168 (+116344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.852268 (+1057100us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:15.852534 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266009984}
I20260709 12:18:16.527567 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.529057 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:15.881145 26698 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.530058 26698 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.532779 26698 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.142505 12380 catalog_manager.cc:5244] ChangeConfig:REMOVE_PEER RPC for tablet 27144fedf0014de09f007ac6cc3f7275 with cas_config_opid_index 7: aborting the task: latest config opid_index 19; task opid_index 7
I20260709 12:18:16.190356 27120 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.534754 27120 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.192631 26924 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.215195 27121 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.538045 26924 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.538292 27121 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.541076 27121 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.226874 26352 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:16.226429 27107 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.228698 27100 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.222295 27103 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.232359 26977 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.226512 12489 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:16.233711 12497 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:16.236016 27122 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.246737 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5362) took 1331 ms. Trace:
W20260709 12:18:16.260334 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:16.260733 26733 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:16.547480 26733 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:16.549003 27027 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.273535 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.274641 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.550745 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.551318 26977 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.552299 12494 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 20. Preceding OpId from leader: term: 47 index: 21. (index mismatch)
I20260709 12:18:16.553841 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4222) took 1328 ms. Trace:
I20260709 12:18:16.281860 13258 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:16.554222 12489 rpcz_store.cc:276] 0709 12:18:15.225545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.672958 (+447413us) service_pool.cc:224] Handling call
0709 12:18:16.553786 (+880828us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.283928 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:16.285049 13260 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [40.25-40.25]
I20260709 12:18:16.555197 26924 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.287199 27094 raft_consensus.cc:2243] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:18:16.556576 27094 pending_rounds.cc:70] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Trying to abort 1 pending ops.
I20260709 12:18:16.304229 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.305869 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:16.304842 12569 rpcz_store.cc:276] 0709 12:18:13.958648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.400801 (+442153us) service_pool.cc:224] Handling call
0709 12:18:14.412394 (+ 11593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:14.865377 (+452983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.521466 (+656089us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b5400246dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:15.521796 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1039937280}
I20260709 12:18:16.556839 27122 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.557983 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.558319 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.558600 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:16.559221 12570 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:16.559607 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:16.311071 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5711) took 1369 ms. Trace:
I20260709 12:18:16.560174 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:16.311810 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:16.314002 27126 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:16.316380 12495 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:16.319605 27128 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:16.341571 27078 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.561360 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:16.345579 27071 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:16.561902 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.562286 27078 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.541922 26352 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.560213 12569 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 41
I20260709 12:18:16.361248 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5716) took 1219 ms. Trace:
I20260709 12:18:16.375294 13259 pending_rounds.cc:85] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 24
I20260709 12:18:16.379698 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.379855 26417 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.002s
W20260709 12:18:16.383963 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.06379e+06us
I20260709 12:18:16.565044 27078 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.385480 27131 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.566301 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5714) took 1425 ms. Trace:
I20260709 12:18:16.566699 12495 rpcz_store.cc:276] 0709 12:18:15.140922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.142106 (+  1184us) service_pool.cc:224] Handling call
0709 12:18:15.187744 (+ 45638us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.562589 (+1374845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.566081 (+  3492us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.566250 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2595060608}
I20260709 12:18:16.567142 27078 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:18:16.567689 27131 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.390753 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5367) took 1472 ms. Trace:
I20260709 12:18:16.395541 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2056) took 1492 ms. Trace:
I20260709 12:18:16.397332 27133 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:16.402364 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5357) took 1501 ms (client timeout 1918 ms). Trace:
I20260709 12:18:16.391935 27132 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.569097 27133 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.569504 27132 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.570546 26417 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:16.570940 26417 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:16.571576 27133 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:16.407855 13218 connection.cc:256] RPC call timeout handler was delayed by -1.5336472139999842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20260709 12:18:16.565814 13261 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
W20260709 12:18:16.572304 13218 connection.cc:256] RPC call timeout handler was delayed by -1.4310472139998183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
I20260709 12:18:16.411588 26496 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:16.572841 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1882032480000362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1998s for a response
I20260709 12:18:16.421855 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:16.401568 27134 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:16.555238 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 271446us
I20260709 12:18:16.573753 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:16.428687 12395 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 29 to 35. New cstate: current_term: 35 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:16.574388 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5359) took 1672 ms. Trace:
I20260709 12:18:16.571861 26417 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.574935 12645 rpcz_store.cc:276] 0709 12:18:14.901809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.000340 (+ 98531us) service_pool.cc:224] Handling call
0709 12:18:16.574213 (+1573873us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54000a6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:16.574332 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3243359488}
I20260709 12:18:16.575541 12395 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 31 to 38. New cstate: current_term: 38 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:18:16.569044 12644 rpcz_store.cc:269] 0709 12:18:14.901171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.902599 (+  1428us) service_pool.cc:224] Handling call
0709 12:18:16.402258 (+1499659us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:15.494462 26933 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.588230 26933 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.444761 12493 rpcz_store.cc:276] 0709 12:18:14.908405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.913840 (+  5435us) service_pool.cc:224] Handling call
0709 12:18:14.914149 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.442955 (+1528806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.443755 (+   800us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.443996 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3513973632}
I20260709 12:18:16.445807 26935 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.590834 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2069) took 1057 ms. Trace:
I20260709 12:18:16.591898 26935 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.560901 27107 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.450536 27001 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:16.452880 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.548864 26733 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 45.18, Last appended by leader: 18, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.450724 27130 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.577400 12395 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 26, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 26 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:16.593456 27130 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.593936 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.594875 12565 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:16.594712 12395 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 28 to 36, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:16.453171 27129 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.464564 12640 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: [47.23-47.23]
I20260709 12:18:16.471861 27003 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.472476 26931 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.596262 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.480124 27056 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.596295 27129 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.553435 26977 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.559293 27122 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.599417 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5366) took 1682 ms. Trace:
I20260709 12:18:16.565032 26352 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.599803 12640 rpcz_store.cc:276] 0709 12:18:14.916923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.666150 (+749227us) service_pool.cc:224] Handling call
0709 12:18:15.666684 (+   534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.598528 (+931844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.599141 (+   613us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.599361 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1834141952}
I20260709 12:18:16.556947 27094 pending_rounds.cc:77] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting op as it isn't in flight: id { term: 30 index: 18 } timestamp: 7305623178907586560 op_type: NO_OP noop_request { }
I20260709 12:18:16.601254 27094 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:18:16.601904 27094 raft_consensus.cc:2272] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:18:16.488489 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5800) took 1997 ms. Trace:
I20260709 12:18:16.604120 12564 rpcz_store.cc:276] 0709 12:18:14.491254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.731619 (+240365us) service_pool.cc:224] Handling call
0709 12:18:14.731990 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.486673 (+1754683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.488134 (+  1461us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.488404 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4033007360}
I20260709 12:18:16.605572 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.606986 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.607689 12564 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:16.599434 27056 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.490859 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2770) took 1692 ms. Trace:
W20260709 12:18:16.492483 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2055) took 1637 ms (client timeout 1816 ms). Trace:
I20260709 12:18:16.492893 27082 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.490712 27057 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.495990 13253 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 30. Preceding OpId from leader: term: 44 index: 32. (index mismatch)
I20260709 12:18:16.496508 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:16.500955 26884 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.609678 12572 rpcz_store.cc:276] 0709 12:18:14.798191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.060849 (+262658us) service_pool.cc:224] Handling call
0709 12:18:15.061204 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.489790 (+1428586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.490552 (+   762us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.490778 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3283471104}
I20260709 12:18:16.610710 26884 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.611694 27082 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.611439 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.503798 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2047) took 1691 ms. Trace:
W20260709 12:18:16.544701 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.277s	user 0.013s	sys 0.013s
I20260709 12:18:16.545549 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.613395 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.613992 13256 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:16.614466 12572 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
W20260709 12:18:16.609798 12637 rpcz_store.cc:269] 0709 12:18:14.855162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.862070 (+  6908us) service_pool.cc:224] Handling call
0709 12:18:16.492245 (+1630175us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:16.492415 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3705392000}
I20260709 12:18:16.615497 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.616701 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.554459 12494 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 31. Preceding OpId from leader: term: 44 index: 32. (index mismatch)
I20260709 12:18:16.613251 12646 rpcz_store.cc:276] 0709 12:18:14.812558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.812719 (+   161us) service_pool.cc:224] Handling call
0709 12:18:14.813084 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:14.813321 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:14.813349 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:14.831845 (+ 18496us) log.cc:844] Serialized 26 byte log entry
0709 12:18:15.035630 (+203785us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:15.035684 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.035788 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:16.503471 (+1467683us) raft_consensus.cc:1685] finished
0709 12:18:16.503506 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.503709 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":145741,"threads_started":1,"wal-append.queue_time_us":250347}
I20260709 12:18:16.277431 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2049) took 1464 ms. Trace:
I20260709 12:18:16.619685 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:16.618567 12639 rpcz_store.cc:276] 0709 12:18:14.813191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.813329 (+   138us) service_pool.cc:224] Handling call
0709 12:18:14.844849 (+ 31520us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:14.919533 (+ 74684us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:14.919574 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:14.990978 (+ 71404us) log.cc:844] Serialized 26 byte log entry
0709 12:18:15.146616 (+155638us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:15.146664 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.146748 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:16.277101 (+1130353us) raft_consensus.cc:1685] finished
0709 12:18:16.277132 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.277315 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60841,"raft.queue_time_us":74185,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":78284,"thread_start_us":97628,"threads_started":1,"wal-append.queue_time_us":186925}
I20260709 12:18:16.282423 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:16.284070 27065 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:16.568392 27078 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:16.621832 27078 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:16.622010 26330 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:16.622805 27078 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.406982 13254 rpcz_store.cc:276] 0709 12:18:13.717531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.717787 (+   256us) service_pool.cc:224] Handling call
0709 12:18:13.718253 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:14.627360 (+909107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:15.693421 (+1066061us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:15.693768 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2088324864}
I20260709 12:18:16.550470 27062 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.004s
I20260709 12:18:16.624269 27078 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:16.624717 27062 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:18:16.625134 27062 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:16.573331 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1559991389999595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
I20260709 12:18:15.524753 26972 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:16.591414 13262 rpcz_store.cc:276] 0709 12:18:15.533365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.533523 (+   158us) service_pool.cc:224] Handling call
0709 12:18:15.533905 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.589900 (+1055995us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.589932 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:16.589972 (+    40us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:16.590016 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.590537 (+   521us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.590592 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.590759 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2428074368}
I20260709 12:18:16.569744 13259 pending_rounds.cc:107] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 34.25
I20260709 12:18:16.628988 13259 raft_consensus.cc:2889] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:16.620961 27065 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.633066 27065 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.444892 12566 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:16.626680 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:16.635299 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5543) took 2123 ms. Trace:
I20260709 12:18:16.513427 12496 rpcz_store.cc:276] 0709 12:18:14.945211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.945825 (+   614us) service_pool.cc:224] Handling call
0709 12:18:14.947107 (+  1282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.510743 (+1563636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.512674 (+  1931us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.512910 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1986218368}
I20260709 12:18:16.635807 12566 rpcz_store.cc:276] 0709 12:18:14.511688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.034681 (+522993us) service_pool.cc:224] Handling call
0709 12:18:15.051525 (+ 16844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.633994 (+1582469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.635023 (+  1029us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.635229 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3203911936}
I20260709 12:18:16.561432 27126 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.560388 12491 rpcz_store.cc:276] 0709 12:18:14.941434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.941731 (+   297us) service_pool.cc:224] Handling call
0709 12:18:14.942156 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:15.054111 (+111955us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:15.054166 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:15.054234 (+    68us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:15.054293 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.310663 (+1256370us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.310739 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.310928 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":256654208}
I20260709 12:18:16.568749 12641 rpcz_store.cc:276] 0709 12:18:14.902864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.316715 (+413851us) service_pool.cc:224] Handling call
0709 12:18:15.338511 (+ 21796us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.393825 (+1055314us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.394819 (+   994us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.395442 (+   623us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":157564800}
I20260709 12:18:16.568256 12642 rpcz_store.cc:276] 0709 12:18:14.917192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.387163 (+1469971us) service_pool.cc:224] Handling call
0709 12:18:16.387635 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.389406 (+  1771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.389682 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.562358 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4223) took 1328 ms. Trace:
I20260709 12:18:16.580818 12571 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:16.638774 12497 rpcz_store.cc:276] 0709 12:18:15.233971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.548183 (+314212us) service_pool.cc:224] Handling call
0709 12:18:16.562299 (+1014116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:16.640110 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:16.564855 12492 rpcz_store.cc:276] 0709 12:18:15.141508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.141743 (+   235us) service_pool.cc:224] Handling call
0709 12:18:15.142236 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.246341 (+1104105us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.246382 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:16.246435 (+    53us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:16.246480 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.360920 (+114440us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.360980 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.361169 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2538549120}
I20260709 12:18:16.638756 27107 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:16.642087 27107 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:16.582192 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.571861 27132 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.637575 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:16.643220 13261 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:16.643400 12566 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:16.644337 27103 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.645032 27103 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.647032 27097 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:16.647676 27097 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.642484 27107 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:16.648106 27097 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:16.651100 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:16.651482 27126 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.651480 26933 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.652570 27129 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.652562 27082 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.654412 27130 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.656150 26884 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:16.658367 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84728.6us
W20260709 12:18:16.661221 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 379186us
I20260709 12:18:16.661464 27133 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:16.661971 27133 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.662326 27133 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:16.662432 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:16.664182 27132 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:16.664611 27132 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.664978 27132 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:16.665954 26935 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:16.668386 12642 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: [41.23-41.23]
I20260709 12:18:16.670300 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:16.670928 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.671504 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:16.671679 12642 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:16.672361 27131 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 41.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20260709 12:18:16.671686 26935 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:18:16.673332 27131 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:16.673777 27131 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.674120 27131 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:16.674691 26935 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20260709 12:18:16.675635 27121 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:16.676038 27121 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.676402 27121 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:16.677882 27103 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.678550 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:16.678787 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.680781 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.683943 27122 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:16.684432 27122 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:16.482479 27028 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.544742 12638 rpcz_store.cc:276] 0709 12:18:14.915766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.431079 (+515313us) service_pool.cc:224] Handling call
0709 12:18:15.431670 (+   591us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:15.534915 (+103245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:15.534959 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:15.535593 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:18:15.568168 (+ 32575us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:15.572128 (+  3960us) log.cc:844] Serialized 14 byte log entry
0709 12:18:15.572351 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.246101 (+673750us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.246172 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:16.246241 (+    69us) raft_consensus.cc:1685] finished
0709 12:18:16.246370 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.246622 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2475,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":5369,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":1353,"spinlock_wait_cycles":235776896,"thread_start_us":32063,"threads_started":1,"wal-append.queue_time_us":39078}
W20260709 12:18:16.686936 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.585095 26955 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:16.617565 12572 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:16.687677 26955 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:16.686576 12570 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:16.688099 26955 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:16.585440 27074 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:18:16.573342 26496 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.28, Last appended by leader: 28, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.690585 26496 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:16.586578 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.694038 27121 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.510306 26968 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.587180 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.696677 27132 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.697677 26968 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:16.451284 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60803.8us
I20260709 12:18:16.699074 27132 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.700217 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:16.701293 27130 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:16.478928 12490 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:16.593401 26733 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:18:16.703534 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4218) took 1480 ms (client timeout 1835 ms). Trace:
W20260709 12:18:16.703944 12490 rpcz_store.cc:269] 0709 12:18:15.222901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.223105 (+   204us) service_pool.cc:224] Handling call
0709 12:18:16.703263 (+1480158us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:16.703426 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2833045760}
I20260709 12:18:16.705608 27069 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.706548 12569 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:16.491631 27064 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.613968 27057 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.709647 27057 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:16.687791 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68185us
I20260709 12:18:16.556604 27100 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.576050 26417 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:16.583597 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.626636 26931 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 33.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:16.562095 27128 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.625998 27062 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 36.27, Last appended by leader: 27, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.592749 12568 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:16.706280 27069 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.463479 26906 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.537395 27120 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.608669 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.713791 27107 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.732103 27094 ts_tablet_manager.cc:1916] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:18:16.720165 27078 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.794292 27078 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.796955 27078 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.722738 27062 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:16.733283 12565 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:16.733530 27107 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.712950 27121 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:16.734827 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800357180 after lost signal to thread 11313
W20260709 12:18:16.799485 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800354060 after lost signal to thread 12429
W20260709 12:18:16.801456 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8480 after lost signal to thread 12521
W20260709 12:18:16.801816 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce4e0 after lost signal to thread 12522
W20260709 12:18:16.803246 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08000e36c0 after lost signal to thread 12596
W20260709 12:18:16.804843 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08002eff80 after lost signal to thread 12670
W20260709 12:18:16.805159 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08003acfa0 after lost signal to thread 12671
I20260709 12:18:16.710657 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:16.736347 26883 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.809587 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.810179 26883 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.812894 26883 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.816551 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:16.818473 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107901us
I20260709 12:18:16.819096 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.819830 13259 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:16.820609 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:16.822214 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.736413 27024 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.823213 27024 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.824165 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.699157 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:16.737007 27028 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:16.826128 27024 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.826588 27028 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:16.827953 12518 debug-util.cc:398] Leaking SignalData structure 0x7b08003c8920 after lost signal to thread 13286
W20260709 12:18:16.828389 12518 debug-util.cc:398] Leaking SignalData structure 0x7b0800340400 after lost signal to thread 13287
I20260709 12:18:16.828475 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.829216 27028 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.738945 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.723131 27069 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:16.726408 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.830984 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.720121 27128 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.833271 26733 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.833907 26733 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.671504 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:16.835672 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 164240us
I20260709 12:18:16.722097 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5793) took 2763 ms. Trace:
I20260709 12:18:16.837971 12568 rpcz_store.cc:276] 0709 12:18:13.958957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.478318 (+519361us) service_pool.cc:224] Handling call
0709 12:18:14.492023 (+ 13705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.720578 (+2228555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.721828 (+  1250us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.722028 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4830681984}
I20260709 12:18:16.839432 27121 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:16.846048 27069 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:16.846267 27128 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:18:16.846985 27069 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.850459 27069 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.853839 27065 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:16.854703 26883 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:16.743160 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5370) took 1825 ms. Trace:
I20260709 12:18:16.854470 26733 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.743440 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.857055 12494 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:18:16.855491 12645 rpcz_store.cc:276] 0709 12:18:14.918039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.575999 (+1657960us) service_pool.cc:224] Handling call
0709 12:18:16.576317 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.576523 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:16.674007 (+ 97484us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b18001a7fa0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:18:16.674144 (+   137us) log.cc:844] Serialized 14 byte log entry
0709 12:18:16.742422 (+ 68278us) log.cc:844] Serialized 14 byte log entry
0709 12:18:16.742653 (+   231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:16.742714 (+    61us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:16.742798 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.742892 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:16.743079 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":81763,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":100,"raft.queue_time_us":2400,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"spinlock_wait_cycles":8048384,"thread_start_us":152659,"threads_started":2,"wal-append.queue_time_us":36124}
I20260709 12:18:16.857928 26883 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.858333 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.859232 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:16.859761 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.860741 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.861372 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.861402 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:16.862802 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.863489 12645 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:16.864378 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.745855 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.867331 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.868981 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.870740 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.872350 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.873908 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.874672 12490 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:16.746584 12567 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:16.747242 26906 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:16.880981 26906 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.883750 26906 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.748598 27090 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.886654 27090 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:16.888233 12609 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.590s (SENT) [suppressed 16 similar messages]
I20260709 12:18:16.749754 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:16.750337 26352 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.889721 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.890168 26352 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.751279 26932 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:16.890681 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:16.891770 27107 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:16.892131 12489 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:16.892505 26417 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:18:16.752694 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101705us
I20260709 12:18:16.893137 12394 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 29 to 36. New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:16.753324 26949 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:18:16.752471 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:16.756400 12643 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:18:16.896808 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5355) took 1996 ms. Trace:
I20260709 12:18:16.756829 27089 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:18:16.897150 12643 rpcz_store.cc:276] 0709 12:18:14.900557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.902427 (+  1870us) service_pool.cc:224] Handling call
0709 12:18:14.992157 (+ 89730us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:16.895948 (+1903791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:16.896543 (+   595us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:16.896748 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4056540928}
I20260709 12:18:16.897884 27089 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:16.750764 26884 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.898470 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.898831 27089 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.899022 26884 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.763422 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.900002 27004 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.900203 27089 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:16.723726 12571 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:16.770618 27074 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.900688 12488 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:16.900570 27004 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.735793 26955 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.901046 27074 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.757369 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.902149 12491 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:16.776783 26972 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 39.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:16.781258 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.903648 27130 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:16.904211 27130 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:16.781685 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.785938 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:16.905072 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.738112 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.905206 27130 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:16.900295 12497 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:16.906627 27130 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:16.906976 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.784049 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.908473 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.837515 27056 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:18:16.854503 27065 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.685451 27122 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 42.30, Last appended by leader: 30, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.772095 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.910429 27065 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:16.910812 26955 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:16.889397 26733 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:16.890517 12496 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:16.912374 27004 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:16.858484 26883 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:16.894598 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:16.913439 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.892184 27107 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:16.914302 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:16.914870 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.915164 26932 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:16.899605 13259 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:16.903760 12493 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 46. Ops: []
I20260709 12:18:16.915755 26932 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:16.916759 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.829788 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.917970 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:16.904597 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:16.914770 27107 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.905813 12495 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:16.915982 26933 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:18:16.781081 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:16.919157 27107 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:16.916949 26932 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.29, Last appended by leader: 29, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.919585 26933 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:16.920619 26933 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.899331 12643 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:16.919239 12492 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 29. Preceding OpId from leader: term: 43 index: 31. (index mismatch)
I20260709 12:18:16.915525 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:16.922381 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.923286 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:16.745201 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 73766.2us
I20260709 12:18:16.924027 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.919612 12646 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:16.912423 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.925738 27100 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:16.926321 27100 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:16.927320 27100 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.28, Last appended by leader: 28, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:16.930142 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:16.930835 12643 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 41 index: 24. (term mismatch)
I20260709 12:18:16.931348 12643 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 23
I20260709 12:18:16.931581 12643 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 40.24
I20260709 12:18:16.931806 12643 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:16.932022 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:16.751971 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101796us
I20260709 12:18:16.930187 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:16.925076 12494 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 49 because replica has last-logged OpId of term: 47 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 23.
I20260709 12:18:16.932916 12638 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:16.910431 27074 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.935988 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.936075 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:16.937801 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117285us
I20260709 12:18:16.938614 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:16.939309 13261 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:16.939764 12497 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:16.941138 12637 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.29-43.29]
I20260709 12:18:16.940810 12642 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
W20260709 12:18:16.941680 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 192051us
I20260709 12:18:16.942049 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:16.958539 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66441.5us
I20260709 12:18:16.959666 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:16.942149 12488 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:16.760408 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.936741 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
W20260709 12:18:16.962349 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:16.941274 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.945415 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.943539 26931 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:16.947466 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.948561 27074 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:16.949615 27122 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:16.954761 12646 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:16.956565 12639 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:16.910480 26352 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:16.978019 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89839.6us
I20260709 12:18:16.961690 12491 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:16.963899 12492 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:16.944940 12642 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 25. (term mismatch)
I20260709 12:18:16.979985 12642 pending_rounds.cc:85] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 24
I20260709 12:18:16.979861 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:16.980485 12642 pending_rounds.cc:107] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 39.25
I20260709 12:18:16.980964 12642 raft_consensus.cc:2889] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:16.966650 12570 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:16.984231 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:16.890715 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 233) took 77544 ms (client timeout 15000 ms). Trace:
I20260709 12:18:16.889410 27090 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:16.970989 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.971856 12489 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:16.972286 12645 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 37 index: 26. (term mismatch)
I20260709 12:18:16.972699 13262 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:16.973024 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:16.975083 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:16.982354 27128 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:16.987079 12645 pending_rounds.cc:85] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 25
I20260709 12:18:16.977980 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.987877 12494 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:16.995879 12645 pending_rounds.cc:107] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 36.26
I20260709 12:18:16.997026 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.986222 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:16.990821 13261 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:16.997331 12645 raft_consensus.cc:2889] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:16.998634 12566 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 49 because replica has last-logged OpId of term: 47 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 23.
W20260709 12:18:16.993613 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:16.996716 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:17.000403 12491 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:16.997121 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:17.001441 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.521s (SENT)
I20260709 12:18:16.984938 13260 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:17.002967 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:17.003410 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43811.4us
I20260709 12:18:17.004638 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:17.004539 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.007532 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.008097 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:17.009404 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.009822 12571 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
W20260709 12:18:17.010512 12536 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.548s (SENT)
I20260709 12:18:17.010993 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.012567 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.996060 12394 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 27 to 36, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:17.013351 12566 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:17.012106 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:17.013175 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:16.977396 27071 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.014278 27080 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:17.019735 27071 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.019898 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:16.969204 27074 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:17.018936 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.021075 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:16.974187 27078 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.021857 12568 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:17.023532 12565 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:16.937033 26733 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.027987 12645 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 20. Preceding OpId from leader: term: 47 index: 21. (index mismatch)
I20260709 12:18:17.044067 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.031574 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.034617 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.041069 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.045439 12646 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:18:17.046766 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.035282 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.047477 12570 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 22.
I20260709 12:18:17.022452 26977 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:17.039213 26884 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:17.015631 26954 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:17.007222 27128 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.040257 27071 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:17.040932 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:17.072983 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62490.1us
I20260709 12:18:17.013805 26972 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:17.047931 12639 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:17.035588 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:17.079905 12644 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.042s	sys 0.077s
I20260709 12:18:17.080446 12642 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:17.067649 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.042145 12566 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:17.082751 12645 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:16.987445 12395 rpcz_store.cc:269] 0709 12:16:59.346451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:10.223525 (+10877074us) service_pool.cc:224] Handling call
0709 12:18:16.890590 (+66667065us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:16.667512 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:16.668131 (+   619us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:16.668176 (+    45us) write_op.cc:435] Acquired schema lock
0709 12:18:16.668197 (+    21us) tablet.cc:662] Decoding operations
0709 12:18:16.668839 (+   642us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:16.668954 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:16.668979 (+    25us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:18:16.669448 (+   469us) tablet.cc:701] Row locks acquired
0709 12:18:16.669498 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:18:16.677048 (+  7550us) write_op.cc:270] Start()
0709 12:18:16.677224 (+   176us) write_op.cc:276] Timestamp: P: 1783599496669593 usec, L: 0
0709 12:18:16.677252 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:18:16.677953 (+   701us) log.cc:844] Serialized 1706 byte log entry
0709 12:18:16.787249 (+109296us) op_driver.cc:464] REPLICATION: finished
0709 12:18:16.806951 (+ 19702us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f032a 00007ffbf5775f14 00007ffbf5775ec3 00007ffbf5775e38 00007ffbf5775df3 00007ffbf57750dc 00007ffbf579ee16 00007ffbf5786cbb 00007ffbf577139d 00007ffbf5774d75
0709 12:18:16.807215 (+   264us) write_op.cc:301] APPLY: starting
0709 12:18:16.807316 (+   101us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:16.810412 (+  3096us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:16.810445 (+    33us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:16.812965 (+  2520us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:16.816579 (+  3614us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=0,mrs_lookups=0
0709 12:18:16.816653 (+    74us) write_op.cc:312] APPLY: finished
0709 12:18:16.816927 (+   274us) log.cc:844] Serialized 49 byte log entry
0709 12:18:16.817102 (+   175us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:16.817205 (+   103us) write_op.cc:454] Released schema lock
0709 12:18:16.817242 (+    37us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:16.854349 (+ 37107us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:16.854381 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":16711,"apply.run_cpu_time_us":6872,"apply.run_wall_time_us":50029,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20831,"num_ops":4,"prepare.queue_time_us":68461,"prepare.run_cpu_time_us":4505,"prepare.run_wall_time_us":119627,"raft.queue_time_us":10557,"raft.run_cpu_time_us":13619,"raft.run_wall_time_us":66095,"replication_time_us":109810,"spinlock_wait_cycles":108767744,"thread_start_us":211940,"threads_started":4,"wal-append.queue_time_us":49080}]]}
W20260709 12:18:17.080492 12644 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.042s	sys 0.077s
I20260709 12:18:16.984769 26496 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.057s
I20260709 12:18:17.018414 27001 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
W20260709 12:18:17.078605 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.040s	sys 0.106s
W20260709 12:18:17.088781 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.040s	sys 0.106s
I20260709 12:18:17.075115 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.089929 27071 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:17.090685 27071 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.088856 27001 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:17.097347 27001 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.112149 27071 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:17.122048 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.003674 26931 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.058s
W20260709 12:18:17.130172 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55165.7us
I20260709 12:18:17.045864 27074 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:17.022778 27003 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:17.136224 12570 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 42 index: 21. (term mismatch)
I20260709 12:18:17.135931 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.136721 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:17.145570 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:17.180934 12570 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:18:17.174294 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.057149 26977 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:17.181362 12570 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 40.21
I20260709 12:18:17.135722 27056 raft_consensus.cc:2764] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election decision vote started in defunct term 45: won
I20260709 12:18:17.184443 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:17.184883 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.190001 12639 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:17.185227 12570 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:17.191735 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.146008 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:17.060166 26954 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:17.177049 12394 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 33 to 40. New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:17.065937 27028 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:17.127857 27120 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:18:17.212536 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76760.6us
I20260709 12:18:17.213719 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:17.195271 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.061523 26330 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.118s
I20260709 12:18:17.063941 27078 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.151160 27003 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:17.230633 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84730.5us
I20260709 12:18:17.182215 27004 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:17.079200 26972 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:17.215462 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:18:17.212219 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 209188us
I20260709 12:18:17.188781 26977 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:17.200457 26954 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:17.173473 27057 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.052s
I20260709 12:18:17.221977 26935 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:17.333738 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:17.297663 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82307.2us
I20260709 12:18:17.456396 12489 raft_consensus.cc:2410] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 42 because replica has last-logged OpId of term: 40 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 24.
I20260709 12:18:17.456970 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.457320 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.231802 27003 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:17.222430 27129 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:17.219751 27065 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.286775 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:17.215364 27028 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:17.307492 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.311897 27056 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:17.246918 27004 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.237540 27027 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
W20260709 12:18:17.239753 27131 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.565s	user 0.010s	sys 0.000s
I20260709 12:18:17.362833 12492 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:17.498956 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:17.354097 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.305039 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.465123 27121 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:18:17.380834 27155 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 12:18:17.244661 26972 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:17.373474 27128 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:17.397647 12641 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:17.318398 12394 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 24 to 30. New cstate: current_term: 30 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:18:17.522442 27158 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
I20260709 12:18:17.412910 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.526412 12566 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:17.418171 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.525061 12394 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 25 to 35, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 35 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:17.537923 27082 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:17.538726 27082 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.541397 27082 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:17.466144 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.542405 26733 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:17.546209 12489 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 42 index: 21. (index mismatch)
W20260709 12:18:17.476984 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 190328us
W20260709 12:18:17.463523 27089 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.563s	user 0.012s	sys 0.001s
I20260709 12:18:17.487488 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.549088 27066 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:18:17.550187 27066 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:17.461882 26935 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:17.550529 27066 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:17.550793 26935 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:17.551497 27066 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.552810 27066 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:17.550834 26972 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.556020 27078 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:17.492789 12494 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:17.511997 12497 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:17.502295 12492 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 22.
I20260709 12:18:17.209057 12490 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:17.465145 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:17.548151 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:17.486052 27028 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:17.461772 27097 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.813s	user 0.010s	sys 0.007s
I20260709 12:18:17.562245 12496 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:17.483564 27090 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:17.572221 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:17.587051 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.590651 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:17.594187 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:17.612670 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.620726 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:17.622802 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.005s	sys 0.009s
W20260709 12:18:17.623200 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.005s	sys 0.009s
W20260709 12:18:17.625208 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 613139us
W20260709 12:18:17.582219 27062 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.785s	user 0.012s	sys 0.000s
W20260709 12:18:17.585945 27133 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.910s	user 0.009s	sys 0.005s
I20260709 12:18:17.642606 12567 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:17.621124 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.652580 12569 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:18:17.643790 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.659518 12566 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:17.561380 27027 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:17.491926 27129 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.125s
I20260709 12:18:17.672010 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.682708 12568 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:17.618980 27161 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.575794 26972 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:17.711197 12491 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.012s	sys 0.042s
W20260709 12:18:17.711676 12491 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.012s	sys 0.042s
I20260709 12:18:17.672785 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.621749 26330 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:18:17.653566 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 416610us
W20260709 12:18:17.654975 27107 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.735s	user 0.007s	sys 0.007s
I20260709 12:18:17.728500 13255 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:18:17.731117 27164 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260709 12:18:17.733557 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:17.734603 27160 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:18:17.735405 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.736289 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:18:17.737926 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.738446 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:17.698027 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.008s	sys 0.012s
W20260709 12:18:17.739063 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.009s	sys 0.012s
I20260709 12:18:17.739256 12640 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:17.739395 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 763651us
I20260709 12:18:17.740053 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.741115 27089 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:18:17.741827 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:17.742410 26932 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:17.742861 26932 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:17.743785 26932 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.744462 12610 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:18:17.744940 12610 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Cancelling election due to peer responding with higher term
I20260709 12:18:17.740741 12640 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:17.746629 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.747685 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.748230 12640 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:17.713840 12572 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:17.750397 27078 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 45. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:18:17.751682 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:17.742409 12645 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:17.752683 26932 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:17.737314 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.753991 27024 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:17.754484 27024 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:17.754778 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.755458 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:17.755505 27024 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.756430 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.756764 27024 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:17.757277 12642 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 39 because replica has last-logged OpId of term: 37 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 19.
I20260709 12:18:17.758052 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:17.758626 13260 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:17.762395 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:17.763085 13261 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:18:17.763597 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.765597 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:17.766127 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.767287 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5802) took 1976 ms. Trace:
I20260709 12:18:17.768090 27165 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:17.641880 27162 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.150s
I20260709 12:18:17.770282 13256 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:17.771119 27162 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:17.767688 12563 rpcz_store.cc:276] 0709 12:18:15.791117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:15.993133 (+202016us) service_pool.cc:224] Handling call
0709 12:18:16.011715 (+ 18582us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.499020 (+487305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:16.499067 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:16.499955 (+   888us) log.cc:844] Serialized 26 byte log entry
0709 12:18:17.089343 (+589388us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:17.089407 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.766892 (+677485us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:17.766941 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:17.766990 (+    49us) raft_consensus.cc:1685] finished
0709 12:18:17.767004 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:17.767229 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1065,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11195,"spinlock_wait_cycles":1118456064,"thread_start_us":587946,"threads_started":1,"wal-append.queue_time_us":799910}
I20260709 12:18:17.719942 26330 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:17.722584 26883 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.809s	user 0.013s	sys 0.000s
W20260709 12:18:17.726704 27163 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:18:17.738548 12643 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:17.802250 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2189) took 1037 ms. Trace:
W20260709 12:18:17.669293 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 437018us
I20260709 12:18:17.802840 12496 rpcz_store.cc:276] 0709 12:18:16.764769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.917783 (+153014us) service_pool.cc:224] Handling call
0709 12:18:17.558109 (+640326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:17.581760 (+ 23651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.801563 (+219803us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b540012b1d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:17.802166 (+   603us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1471747840}
I20260709 12:18:17.789943 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:17.775972 13260 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:18:17.665328 27100 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.003s	sys 0.012s
I20260709 12:18:17.779035 12645 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:17.783102 12563 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:18:17.774125 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:17.789283 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5371) took 2870 ms. Trace:
I20260709 12:18:17.810926 12644 rpcz_store.cc:276] 0709 12:18:14.918371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.576754 (+1658383us) service_pool.cc:224] Handling call
0709 12:18:16.577143 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.577647 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:17.086704 (+509057us) log.cc:844] Serialized 14 byte log entry
0709 12:18:17.563185 (+476481us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:17.617218 (+ 54033us) log.cc:844] Serialized 26 byte log entry
0709 12:18:17.617502 (+   284us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:17.620745 (+  3243us) log.cc:844] Serialized 14 byte log entry
0709 12:18:17.621025 (+   280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.621119 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:17.788930 (+167811us) raft_consensus.cc:1685] finished
0709 12:18:17.788967 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:17.789185 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":87446,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":3402,"raft.queue_time_us":94401,"raft.run_cpu_time_us":400,"raft.run_wall_time_us":11048,"thread_start_us":984443,"threads_started":3,"wal-append.queue_time_us":379817}
I20260709 12:18:17.771723 27162 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.795892 26330 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.797570 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:17.799021 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:17.800168 27154 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.484s	user 0.000s	sys 0.030s
I20260709 12:18:17.821794 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:17.805024 26883 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.617158 27097 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.515033 27131 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:17.813380 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:17.795246 12568 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 21. (term mismatch)
I20260709 12:18:17.834939 12568 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:18:17.675639 27133 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:17.821446 13256 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:17.826681 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.872s	user 0.017s	sys 0.017s
I20260709 12:18:17.828217 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:17.825421 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:17.813913 27100 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.003s	sys 0.012s
I20260709 12:18:17.831818 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.833209 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:17.835347 12568 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 34.21
W20260709 12:18:17.837553 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.017s	sys 0.017s
I20260709 12:18:17.842970 12568 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:17.843335 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 898535us
I20260709 12:18:17.848752 27133 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:17.849400 12564 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.26-48.26]
I20260709 12:18:17.837028 26972 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:17.858096 26972 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:17.858501 26972 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:17.858613 27004 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:17.861619 27089 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:17.717594 27161 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.857091 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2188) took 1092 ms. Trace:
I20260709 12:18:17.865258 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:17.868724 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:17.869082 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43768.2us
I20260709 12:18:17.873761 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.876065 13257 raft_consensus.cc:2368] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:18:17.872967 12493 rpcz_store.cc:276] 0709 12:18:16.764503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.917006 (+152503us) service_pool.cc:224] Handling call
0709 12:18:16.917394 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:16.917778 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 31
0709 12:18:16.988069 (+ 70291us) log.cc:844] Serialized 14 byte log entry
0709 12:18:17.217654 (+229585us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:17.222556 (+  4902us) log.cc:844] Serialized 26 byte log entry
0709 12:18:17.222728 (+   172us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:18:17.222771 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:17.222839 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:17.856771 (+633932us) raft_consensus.cc:1685] finished
0709 12:18:17.856805 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:17.856991 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":216587,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":8219,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":7763,"thread_start_us":222313,"threads_started":2,"wal-append.queue_time_us":431611}
I20260709 12:18:17.871048 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.883499 12570 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:17.875346 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:17.829203 27166 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:17.880649 27069 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:17.889410 27078 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:17.892666 27062 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:18:17.836208 27090 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:17.885389 27167 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:17.900508 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.904942 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:17.944639 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.893076 27071 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:17.947521 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:17.948110 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.906s	user 0.005s	sys 0.043s
W20260709 12:18:17.948442 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.907s	user 0.006s	sys 0.043s
W20260709 12:18:17.917869 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 459686us
I20260709 12:18:17.950598 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.906929 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:17.953109 12644 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:17.953814 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.906270 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:17.956338 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:17.966552 27089 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:17.835279 26935 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:17.906378 27134 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:18:17.969787 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 343160us
W20260709 12:18:17.999291 13257 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1807) took 1413 ms (client timeout 1659 ms). Trace:
I20260709 12:18:17.969602 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:17.973836 12494 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
W20260709 12:18:17.999329 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93186.3us
I20260709 12:18:17.897356 27166 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.002369 12496 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 42 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
W20260709 12:18:18.014783 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45308us
I20260709 12:18:18.015713 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:18.003824 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:18.003545 13257 rpcz_store.cc:269] 0709 12:18:16.585813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.586901 (+  1088us) service_pool.cc:224] Handling call
0709 12:18:17.999046 (+1412145us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:17.999212 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2712859520}
I20260709 12:18:17.903779 27069 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:17.944515 27078 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:18.014385 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44328.2us
I20260709 12:18:18.024048 12571 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:18.020406 12488 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:17.968503 27173 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:17.968848 27133 raft_consensus.cc:2764] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:18:18.033509 12642 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 44 index: 30. (index mismatch)
I20260709 12:18:18.035496 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:18.031721 27089 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:17.838886 27027 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.038156 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:18.044981 12644 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:18:18.010737 27166 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:18.120545 27097 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:18.122474 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.123586 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:18.124343 12570 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:18.018293 27004 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:18.126065 27004 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:18.126528 27004 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:18.004359 27121 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.043s
I20260709 12:18:18.129837 27065 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.101563 27071 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.136730 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5822) took 1582 ms. Trace:
I20260709 12:18:18.137531 12564 rpcz_store.cc:276] 0709 12:18:16.554488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.730410 (+175922us) service_pool.cc:224] Handling call
0709 12:18:16.730766 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:17.850318 (+1119552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.133573 (+283255us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:18.136654 (+  3081us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2563596672}
I20260709 12:18:18.139420 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4255) took 1113 ms. Trace:
W20260709 12:18:18.048707 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46135.3us
I20260709 12:18:18.032367 27024 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:18.031735 27156 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.038s
W20260709 12:18:18.102281 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64256.3us
W20260709 12:18:18.103340 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68009.8us
I20260709 12:18:18.140035 12491 rpcz_store.cc:276] 0709 12:18:17.025666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.026102 (+   436us) service_pool.cc:224] Handling call
0709 12:18:17.026578 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:17.712121 (+685543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:17.712169 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:17.728151 (+ 15982us) log.cc:844] Serialized 26 byte log entry
0709 12:18:18.013319 (+285168us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:18.013377 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.013511 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:18.139066 (+125555us) raft_consensus.cc:1685] finished
0709 12:18:18.139091 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:18.139337 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20271,"raft.queue_time_us":39653,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":71080,"thread_start_us":949254,"threads_started":2,"wal-append.queue_time_us":404957}
I20260709 12:18:18.149238 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:18.051039 26733 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.089s
W20260709 12:18:18.107206 26954 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.794s	user 0.015s	sys 0.000s
W20260709 12:18:18.107491 27170 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:18:18.099332 27089 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:18.108868 27162 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.109953 27167 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:18.102587 26883 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.112049 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5377) took 1331 ms. Trace:
I20260709 12:18:18.199769 12641 rpcz_store.cc:276] 0709 12:18:16.780211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.785616 (+  5405us) service_pool.cc:224] Handling call
0709 12:18:18.111797 (+1326181us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54002c5e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:18.111971 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":985780992}
I20260709 12:18:18.113206 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1812) took 1138 ms. Trace:
I20260709 12:18:18.114440 12572 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:18.145946 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:18.147627 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4337) took 1110 ms. Trace:
I20260709 12:18:18.202204 12646 rpcz_store.cc:276] 0709 12:18:17.036752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.046419 (+  9667us) service_pool.cc:224] Handling call
0709 12:18:18.147387 (+1100968us) spinlock_profiling.cc:243] Waited 963 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:18.147548 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2214937856}
W20260709 12:18:18.096331 26977 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.780s	user 0.012s	sys 0.002s
I20260709 12:18:18.201367 13255 rpcz_store.cc:276] 0709 12:18:16.974697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.974820 (+   123us) service_pool.cc:224] Handling call
0709 12:18:18.112997 (+1138177us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:18.113129 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1703752576}
W20260709 12:18:18.152118 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148402us
I20260709 12:18:18.208128 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:18.152582 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 313452us
I20260709 12:18:18.131722 27071 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.209823 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:18.091421 27069 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:18.096163 27173 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:18.169421 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 169769us
I20260709 12:18:18.172034 27103 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:18.183207 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:18.185344 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 377880us
W20260709 12:18:18.110680 27146 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.050s	user 0.003s	sys 0.000s
I20260709 12:18:18.149837 27175 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:18:18.150017 27024 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:18.168848 27122 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.166s	user 0.016s	sys 0.000s
I20260709 12:18:18.209841 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
W20260709 12:18:18.215946 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:18.217168 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71366.8us
I20260709 12:18:18.169068 27121 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:18.215459 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:18.092324 27120 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.094s
W20260709 12:18:18.088866 27057 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.013s	sys 0.030s
I20260709 12:18:18.172930 27131 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.187901 27089 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:18:18.208213 27003 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.726s	user 0.012s	sys 0.004s
I20260709 12:18:18.235659 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:18.234387 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:18.237234 12564 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 18. Preceding OpId from leader: term: 46 index: 19. (index mismatch)
I20260709 12:18:18.091668 27090 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.254884 12609 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:18:18.277782 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:18.269381 12569 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 30. Preceding OpId from leader: term: 43 index: 31. (index mismatch)
I20260709 12:18:18.179888 26954 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.275959 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:18.230288 27065 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:18.198885 26417 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.055s
I20260709 12:18:18.230604 27103 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:18.185547 27177 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.081s
I20260709 12:18:18.230713 27173 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:18.207091 27174 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.056s
W20260709 12:18:18.284960 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69591.3us
I20260709 12:18:18.234010 27186 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:18.254554 27165 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:18.216421 27097 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:18:18.353276 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:18.277263 26932 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.002s
W20260709 12:18:18.264335 27057 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.014s	sys 0.030s
I20260709 12:18:18.253074 27024 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:18.283864 27122 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:18.343041 27065 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:18.366866 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 158854us
I20260709 12:18:18.355320 27103 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:18.257659 27121 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:18.221988 26977 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:18.377159 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.351507 27188 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:18.390980 12564 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:18.366170 12462 leader_election.cc:400] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:18:18.393743 12639 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:18.416890 12462 leader_election.cc:403] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:18.341605 27129 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:18.394538 13254 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:18.283098 27128 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:18.395462 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 212392us
I20260709 12:18:18.236460 26884 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.133s
W20260709 12:18:18.397545 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 133671us
W20260709 12:18:18.430394 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64446.6us
I20260709 12:18:18.431308 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:18.428737 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:18.399490 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:18.255481 27001 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
W20260709 12:18:18.384812 27074 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.204s	user 0.017s	sys 0.002s
I20260709 12:18:18.239683 27078 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 46.24, Last appended by leader: 24, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:18.354349 27154 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:18:18.410400 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:18.369927 26935 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.380606 27168 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:18.287128 27003 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.421393 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:18.401324 27161 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.400779 27065 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:18.413571 27165 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:18.408668 26417 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:18.470031 27165 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:18.471145 27165 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.472486 27165 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:18.474606 27168 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.477327 27168 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:18.457870 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:18.390544 27186 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.464701 27003 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.436619 27128 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.465915 27156 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:18.425777 27194 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.051s
I20260709 12:18:18.475072 26954 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:18.414407 27121 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:18.434499 27057 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:18.419493 27162 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.065s
I20260709 12:18:18.412335 26955 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:18:18.480862 27166 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.436847 27183 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 42
I20260709 12:18:18.409229 27122 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.459774 27131 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:18.445706 27188 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:18.418191 26330 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.043s
I20260709 12:18:18.568403 27183 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 41. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42.
I20260709 12:18:18.644652 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4321) took 1859 ms. Trace:
W20260709 12:18:18.614439 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 180235us
I20260709 12:18:18.503813 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:18.524516 26733 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:18.674163 26330 consensus_queue.cc:1243] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:18:18.507830 26932 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:18.686946 26932 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:18.495111 26935 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:18.546307 26977 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.528936 27057 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.691774 27196 raft_consensus.cc:2764] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election decision vote started in defunct term 47: won
I20260709 12:18:18.455374 27074 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.500475 27004 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.628512 27024 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.510215 27094 ts_tablet_manager.cc:1929] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 30.18
I20260709 12:18:18.548024 27122 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.696312 27094 log.cc:1199] T 27144fedf0014de09f007ac6cc3f7275 P 32889a507a6b4c74aec421ffbfd1e25d: Deleting WAL directory at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/wals/27144fedf0014de09f007ac6cc3f7275
I20260709 12:18:18.691282 27182 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:18.699070 27094 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47032 (request call id 0) took 3748 ms. Trace:
I20260709 12:18:18.693252 27196 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:18.543540 27131 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:18.700556 27196 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:18.701311 27131 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:18.702530 13261 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:18.449834 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50450.2us
I20260709 12:18:18.707163 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:18.542970 27186 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.522979 27174 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.024s
W20260709 12:18:18.542141 27192 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:18:18.711139 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4335) took 1674 ms (client timeout 1760 ms). Trace:
W20260709 12:18:18.711947 12639 rpcz_store.cc:269] 0709 12:18:17.036110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.191397 (+155287us) service_pool.cc:224] Handling call
0709 12:18:18.710521 (+1519124us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:18.710857 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2761930624}
W20260709 12:18:18.569290 27203 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:18:18.575117 27188 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.509660 27206 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:18.646157 27166 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:18.501849 27207 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:18.720137 26496 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:18.721717 27134 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:18.725888 27134 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:18.710970 27196 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.699164 27182 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:18.727743 27182 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:18.690018 26932 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:18.724226 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.724762 27186 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:18.691243 26935 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.699966 27094 rpcz_store.cc:276] 0709 12:18:14.950787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:14.951026 (+   239us) service_pool.cc:224] Handling call
0709 12:18:15.133000 (+181974us) ts_tablet_manager.cc:1151] Deleting tablet 27144fedf0014de09f007ac6cc3f7275
0709 12:18:15.133132 (+   132us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:18:17.478253 (+2345121us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:18.490336 (+1012083us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:18.698833 (+208497us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1061879,"tablet-delete.queue_time_us":96039,"thread_start_us":46022,"threads_started":1}
I20260709 12:18:18.729439 26496 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:18.749804 26496 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.720783 27206 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:18.760182 27202 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:18:18.729326 27024 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.776544 27057 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.777743 27074 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.765698 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:18.787293 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 358528us
I20260709 12:18:18.790087 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:18.791005 26496 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:18.794068 27183 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:18:18.801813 27080 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:18.802438 27080 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:18.804157 27080 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.812160 27128 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.818971 26931 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:18.819768 26931 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:18.785266 27206 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:18.821288 26931 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.30, Last appended by leader: 30, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.826135 26931 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:18.826938 26935 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.830571 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:18.830781 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:18.831552 12496 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:18.840871 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:18.802322 27028 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.185s	user 0.012s	sys 0.015s
I20260709 12:18:18.842710 27028 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.846388 27028 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:18.846482 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:18.847224 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:18.849941 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:18.645547 12637 rpcz_store.cc:276] 0709 12:18:16.784936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:16.942092 (+157156us) service_pool.cc:224] Handling call
0709 12:18:16.942404 (+   312us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:16.970539 (+ 28135us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:16.997605 (+ 27066us) log.cc:844] Serialized 14 byte log entry
0709 12:18:17.952197 (+954592us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:17.969244 (+ 17047us) log.cc:844] Serialized 52 byte log entry
0709 12:18:17.969398 (+   154us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:18.187607 (+218209us) log.cc:844] Serialized 14 byte log entry
0709 12:18:18.197925 (+ 10318us) log.cc:844] Serialized 14 byte log entry
0709 12:18:18.411404 (+213479us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b1800340000. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:18:18.507403 (+ 95999us) spinlock_profiling.cc:243] Waited 9.28 ms on lock 0x7b1800340000. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:18.643324 (+135921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.643572 (+   248us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:18.643688 (+   116us) raft_consensus.cc:1685] finished
0709 12:18:18.643708 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:18.644255 (+   547us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81981,"prepare.queue_time_us":116714,"prepare.run_cpu_time_us":229,"prepare.run_wall_time_us":231,"raft.queue_time_us":54761,"raft.run_cpu_time_us":712,"raft.run_wall_time_us":56076,"spinlock_wait_cycles":48648320,"thread_start_us":1308136,"threads_started":3,"wal-append.queue_time_us":1092600}
W20260709 12:18:18.855149 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 315832us
W20260709 12:18:18.885905 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 179197us
I20260709 12:18:18.728186 27134 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:18.819940 12645 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
W20260709 12:18:18.869599 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.011s	sys 0.007s
W20260709 12:18:18.856884 26972 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.998s	user 0.019s	sys 0.011s
I20260709 12:18:18.885659 27196 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:18.874323 26955 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:18.924751 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:18.927737 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:18.946782 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:18.954871 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 162282us
I20260709 12:18:18.963002 27134 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:18.957965 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:18.934717 26955 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:18.975598 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 464583us
I20260709 12:18:18.976605 27071 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:18.981818 26906 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:18.983063 26906 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.985857 27080 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:18.990208 27089 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.536s	user 0.009s	sys 0.014s
I20260709 12:18:18.990839 26955 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:18.994752 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 584494us
I20260709 12:18:18.995426 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:18.996203 27184 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:18.996958 27184 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:18.975580 27187 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.496s	user 0.005s	sys 0.002s
I20260709 12:18:18.998258 27184 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 48.26, Last appended by leader: 26, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.999599 27184 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:19.000151 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5835) took 1215 ms. Trace:
I20260709 12:18:19.000619 12567 rpcz_store.cc:276] 0709 12:18:17.785102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.785351 (+   249us) service_pool.cc:224] Handling call
0709 12:18:17.785848 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:17.786380 (+   532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:17.786427 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:17.787085 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:18:18.169433 (+382348us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:18.169481 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:18.169554 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:18.756611 (+587057us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:18.999610 (+242999us) raft_consensus.cc:1685] finished
0709 12:18:18.999634 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:18.999957 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":11495,"spinlock_wait_cycles":13892480,"thread_start_us":347544,"threads_started":1,"wal-append.queue_time_us":567154}
I20260709 12:18:18.934285 26972 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.002202 27222 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.055s
I20260709 12:18:19.003958 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 27144fedf0014de09f007ac6cc3f7275 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
W20260709 12:18:19.004420 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.004957 26972 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.005832 27165 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:19.006837 27165 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:19.007390 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:18.968191 26330 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.008431 26330 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.009812 27078 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:19.010454 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.010630 27078 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.011747 26906 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:18.990773 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 64574.4us
I20260709 12:18:18.991034 27089 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:18.982098 12488 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20260709 12:18:18.996277 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.002056 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.011953 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:19.031929 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:19.032325 27133 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:19.039583 27089 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.040326 27133 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.041162 27078 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.043304 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.044739 13256 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:19.045034 13259 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 46 index: 28. (index mismatch)
I20260709 12:18:19.035476 27128 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.051482 13254 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 37 because replica has last-logged OpId of term: 35 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20260709 12:18:19.044245 13253 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:19.056758 27174 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.057785 27174 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.058871 13258 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:19.017692 27071 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:19.046409 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:19.059823 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:19.056583 27128 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:19.043076 27121 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.514s	user 0.011s	sys 0.011s
I20260709 12:18:19.018082 27131 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.069305 27175 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:19.070277 27180 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:19.070668 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:19.059957 27028 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:19.074020 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 99556us
I20260709 12:18:19.075831 27128 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.043085 27133 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.076586 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.078423 12488 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:19.079434 26906 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:19.080109 27228 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:19.062075 13261 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:19.049778 27165 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.095245 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.121419 13253 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:19.073897 27071 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.091722 26330 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.126510 12565 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
W20260709 12:18:19.128485 27065 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.663s	user 0.012s	sys 0.007s
I20260709 12:18:19.128328 27182 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.129065 27065 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.129817 27133 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:19.104948 26977 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 47.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.039s
I20260709 12:18:19.122534 27028 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.130718 12496 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:19.116757 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.120752 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:19.090968 26906 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:19.084331 27180 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:19.155467 12564 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:19.155843 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.149530 27175 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.157502 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:19.158519 12488 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:19.160538 27175 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.165591 26883 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.165980 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.166499 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.166992 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:19.167546 12490 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:18:19.168419 12495 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:19.168387 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.169013 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.169836 12492 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:19.170933 27131 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.172816 27196 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.166517 26883 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.176071 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.176992 12492 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:19.179045 26883 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.131258 27128 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.069s
I20260709 12:18:19.089340 27121 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.195591 12563 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 35. Current term is 36. Ops: []
W20260709 12:18:19.197742 27103 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.651s	user 0.013s	sys 0.011s
I20260709 12:18:19.198550 27103 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.200333 13255 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:19.146864 27180 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:19.130605 27133 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:19.208711 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:19.209319 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:19.210649 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140213us
I20260709 12:18:19.213016 12566 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.26-48.26]
I20260709 12:18:19.208331 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:19.214891 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 218849us
W20260709 12:18:19.210769 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55138.4us
I20260709 12:18:19.214476 13254 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:18:19.210625 27078 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:19.228704 27078 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:19.229893 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77036.1us
I20260709 12:18:19.230206 27078 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.231884 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.233359 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:19.235042 27135 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:18:19.235277 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:19.215281 27133 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:19.238389 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 48 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.211457 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:19.240443 27056 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:19.240916 27056 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:19.241314 27056 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:19.201577 27103 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.207165 27065 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.245589 27066 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:19.246181 27028 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.145850 26906 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:19.246670 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:19.214293 27173 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.825s	user 0.012s	sys 0.009s
I20260709 12:18:19.246356 27066 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.249353 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:19.269757 27173 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.272750 27173 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.273723 27228 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:19.274308 27228 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:19.274682 27228 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:19.277022 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:19.211966 27090 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:19.209969 27165 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
W20260709 12:18:19.211952 27069 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.970s	user 0.019s	sys 0.005s
I20260709 12:18:19.285341 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:19.285480 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.289904 13254 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:19.289934 12644 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:19.290612 26496 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:19.301258 26496 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.304572 26496 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:19.297415 27231 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.402s	user 0.001s	sys 0.001s
I20260709 12:18:19.294205 27069 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.310803 27069 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.289356 27090 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.316161 27090 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.297724 27227 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.326315 26906 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.327620 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:19.286763 27165 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:19.330860 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.331279 27165 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.332181 27241 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:19.332666 27241 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:19.332720 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.333194 27241 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:19.335505 26931 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:19.338105 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.338958 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:19.340991 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.341898 26931 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:19.343555 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.250007 27196 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.288584 27128 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:19.287058 27121 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.373313 12566 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:19.374039 12637 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:19.365298 12491 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:19.393509 27120 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:19.394372 27120 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.403384 27165 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:19.415797 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.421818 27215 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.422662 27215 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.425140 27215 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.435186 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.436132 12571 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:19.436247 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.436715 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.437429 12569 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:19.436309 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.437968 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:18:19.438567 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:19.438709 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:19.440111 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.443360 27196 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.321852 26977 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.054s
I20260709 12:18:19.353304 12492 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:19.359804 27128 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:19.472349 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93993.2us
I20260709 12:18:19.374656 26931 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:19.474112 12493 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:19.476989 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:19.483613 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:19.351480 27241 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.473587 12488 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:19.479199 27056 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.475162 27128 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:19.504788 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:19.365998 27069 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.363895 27206 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:19.483991 27004 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.519189 27001 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.513149 27183 raft_consensus.cc:2764] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election decision vote started in defunct term 43: won
I20260709 12:18:19.527716 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.524848 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:19.516991 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.475503 27103 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:19.527912 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:19.476492 26906 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.363391 27227 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.525511 27182 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.553839 27216 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:18:19.561517 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 462344us
I20260709 12:18:19.536267 27131 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:18:19.555166 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70086.8us
I20260709 12:18:19.490917 27121 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:19.483821 27196 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.057s
W20260709 12:18:19.567044 27080 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.577s	user 0.011s	sys 0.005s
W20260709 12:18:19.593411 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 341958us
I20260709 12:18:19.564927 27215 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:19.602077 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:19.575762 27216 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:19.612113 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.612538 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:19.605291 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.626965 27173 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:19.630568 27173 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.596486 27120 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.634498 26977 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:19.637477 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:19.638583 12491 raft_consensus.cc:606] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:19.638957 12491 raft_consensus.cc:614] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:19.644289 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.576180 27103 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.584925 27121 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.561101 27001 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.584367 27069 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:19.687770 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101728us
I20260709 12:18:19.690241 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.691320 12644 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:19.696290 12645 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:19.654060 27228 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.715479 27056 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.716560 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:18:19.720446 27128 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.723703 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 48 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.725171 27227 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.726563 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.791647 12564 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:19.739730 27001 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.794067 12637 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:19.745730 27121 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.750028 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:19.694136 27175 raft_consensus.cc:2764] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election decision vote started in defunct term 45: won
I20260709 12:18:19.768160 27071 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.773070 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:19.810110 27071 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:19.799602 26932 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.004s	sys 0.014s
I20260709 12:18:19.817250 27069 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:19.774616 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 269901us
W20260709 12:18:19.776059 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.693032 27180 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.776465 27241 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:19.799106 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 293591us
I20260709 12:18:19.803929 12566 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:19.804435 12639 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:19.807271 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:18:19.839339 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 227674us
W20260709 12:18:19.815166 26932 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.005s	sys 0.014s
I20260709 12:18:19.789908 27216 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:19.848848 12637 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:18:19.847438 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:19.848718 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138823us
I20260709 12:18:19.856699 12493 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:19.849573 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.858466 12488 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:19.858718 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:19.853185 27133 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.616s	user 0.013s	sys 0.002s
I20260709 12:18:19.844455 27080 raft_consensus.cc:2764] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election decision vote started in defunct term 48: won
I20260709 12:18:19.865085 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:19.873116 12644 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 18. Preceding OpId from leader: term: 46 index: 19. (index mismatch)
I20260709 12:18:19.850032 27228 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.874591 27056 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:19.882534 27180 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.871695 27133 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.884464 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5846) took 1037 ms. Trace:
I20260709 12:18:19.885529 12567 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 35. Current term is 36. Ops: []
I20260709 12:18:19.877393 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.875568 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:19.886672 13258 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:19.884913 12565 rpcz_store.cc:276] 0709 12:18:18.847600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.847788 (+   188us) service_pool.cc:224] Handling call
0709 12:18:18.848306 (+   518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.127054 (+278748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:19.884152 (+757098us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:19.884392 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":588902144}
I20260709 12:18:19.890816 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:19.904160 27066 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.906286 27128 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:19.875397 27103 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:19.885779 27056 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:19.914052 27056 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:19.917115 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:19.920804 27173 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.922895 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.924491 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.924937 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.926363 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1825) took 1439 ms. Trace:
I20260709 12:18:19.926499 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.926843 13255 rpcz_store.cc:276] 0709 12:18:18.486972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.487386 (+   414us) service_pool.cc:224] Handling call
0709 12:18:18.487751 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.200907 (+713156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:19.926033 (+725126us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b54005e1d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:19.926276 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1637677440}
I20260709 12:18:19.928112 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.928841 12571 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:19.929833 27174 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:19.925325 12565 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:19.930528 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.931134 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.931365 12569 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:19.931764 12565 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:19.934128 27173 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:19.934538 27173 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:19.934862 27173 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:19.938772 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:19.941478 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:19.928004 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.945158 13258 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:19.939008 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:19.947556 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:19.953295 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62598.5us
I20260709 12:18:19.953693 13257 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:19.960548 12491 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:18:19.969017 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:19.979368 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
W20260709 12:18:19.980286 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:19.957952 27133 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:20.002450 12643 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:20.003789 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.004608 12643 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:20.005861 27245 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.278s	user 0.004s	sys 0.001s
I20260709 12:18:20.006217 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:20.010068 27165 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.606s	user 0.015s	sys 0.000s
I20260709 12:18:19.991431 27097 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:20.011781 27097 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:20.013236 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:19.994539 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:20.030686 13261 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 48: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:19.996843 13255 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 46 index: 28. (index mismatch)
I20260709 12:18:19.997036 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:19.997583 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:19.998593 13262 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:19.999392 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:19.991501 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:20.010893 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46910.6us
I20260709 12:18:19.990501 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:20.011046 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5843) took 1763 ms. Trace:
I20260709 12:18:19.991614 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:20.034723 13258 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:20.020195 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:19.979416 27228 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:20.012899 27097 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:19.958724 27196 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:20.045600 12565 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:20.019685 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:20.005380 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.035465 12563 rpcz_store.cc:276] 0709 12:18:18.248072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.267588 (+ 19516us) service_pool.cc:224] Handling call
0709 12:18:18.267876 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.196365 (+928489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:20.010664 (+814299us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b54001d6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:20.010977 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2131800832}
I20260709 12:18:20.045360 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:20.048287 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.048980 12640 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:20.050490 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:20.020269 27167 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.022939 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:20.032196 13259 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:18:20.020506 27071 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:20.027591 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.061685 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:20.042138 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:20.035490 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48582.1us
I20260709 12:18:20.044138 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5755) took 1168 ms. Trace:
I20260709 12:18:19.965994 27066 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.052s
I20260709 12:18:20.036914 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.047892 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:20.053027 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:20.033776 27078 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.066798 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.062815 12496 rpcz_store.cc:276] 0709 12:18:18.867666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:18.867839 (+   173us) service_pool.cc:224] Handling call
0709 12:18:18.925620 (+ 57781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.138160 (+212540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:20.035599 (+897439us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b540037da54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:20.035903 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":470873472}
I20260709 12:18:20.066063 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:20.068025 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:20.047899 27196 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.079015 13261 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:20.067807 12564 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:20.082329 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.085572 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.088135 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.089310 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:20.090063 12568 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 44
I20260709 12:18:20.090183 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.056936 27080 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:20.064505 27242 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:20.093833 12571 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:20.095147 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.089195 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:20.084542 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:20.099851 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.101152 12493 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 42 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:18:20.110546 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.111334 12492 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:20.112326 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:20.037945 27056 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.110546 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.086328 27174 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:20.081643 27196 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:20.092638 27128 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:20.121229 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.123425 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:20.055573 27228 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.119424 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:20.145637 12495 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:20.095471 27242 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:20.149560 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:20.171499 27242 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:20.092308 26955 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:20.173026 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:20.173188 26955 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:20.173565 27204 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:20.173933 12497 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:20.131301 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.123495 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.093070 27080 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:20.131836 12564 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:20.147383 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:20.092391 27071 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:20.148877 12566 raft_consensus.cc:2368] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38.
I20260709 12:18:20.130957 26931 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 49.32, Next index: 33, Last known committed idx: 31, Time since last communication: 0.019s
I20260709 12:18:20.149453 27251 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:20.118880 27183 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:20.151532 12638 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
W20260709 12:18:20.152268 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 129538us
I20260709 12:18:20.124105 27174 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.157509 12569 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:20.162287 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:20.163709 27024 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:20.165021 27228 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:20.098582 27078 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.147372 27121 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:20.170507 27128 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:20.171629 12491 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:20.208179 27080 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:20.230666 13217 leader_election.cc:400] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38.
I20260709 12:18:20.194103 13257 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:20.194164 12637 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:20.202005 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
W20260709 12:18:20.156688 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72248.2us
I20260709 12:18:20.129786 27229 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.214672 27174 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:20.239912 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:20.219695 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108242us
I20260709 12:18:20.217890 12644 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:20.223533 12570 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:20.151429 27222 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.102066 27167 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.175451 12493 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 42 index: 25. (term mismatch)
I20260709 12:18:20.275369 12493 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 24
I20260709 12:18:20.275765 12493 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 40.25
I20260709 12:18:20.275815 12642 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:18:20.276162 12493 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:20.151428 27056 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:20.228317 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66156.8us
I20260709 12:18:20.169811 27188 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.099s
I20260709 12:18:20.225875 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.212538 27069 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.221273 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.244007 12643 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:18:20.244036 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:20.242362 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:20.245297 12637 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:20.283192 12564 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
W20260709 12:18:20.285480 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:20.301967 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59720.9us
I20260709 12:18:20.236409 27227 raft_consensus.cc:2764] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election decision vote started in defunct term 37: won
I20260709 12:18:20.278259 27229 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:20.291863 27262 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:18:20.174379 26955 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 47.23, Last appended by leader: 23, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.278602 27184 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 24, Time since last communication: 0.015s
I20260709 12:18:20.287459 27256 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:20.239519 27180 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:20.272475 27252 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:20.310312 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:20.313058 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69103.1us
W20260709 12:18:20.308957 27265 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:18:20.316985 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:20.321913 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:20.336884 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.280836 27121 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.341843 27121 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:20.342141 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:20.346127 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:20.346341 27260 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:18:20.347134 12645 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:20.319442 27256 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:20.348090 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.316454 27184 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:20.342572 27121 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:20.348711 12645 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:20.280732 27222 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.304800 27069 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.353634 12570 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 24. Preceding OpId from leader: term: 48 index: 25. (index mismatch)
I20260709 12:18:20.354911 27173 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.331353 26330 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:20.320466 27180 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.384855 27128 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.386111 27181 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.057s
I20260709 12:18:20.388737 27229 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:20.411252 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 196150us
I20260709 12:18:20.399513 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5383) took 1073 ms. Trace:
I20260709 12:18:20.355276 27078 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:20.386615 12564 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: [42.21-42.21]
W20260709 12:18:20.401572 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79772.8us
W20260709 12:18:20.406236 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64238.3us
I20260709 12:18:20.383711 27066 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:20.417981 12639 rpcz_store.cc:276] 0709 12:18:19.325716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.332309 (+  6593us) service_pool.cc:224] Handling call
0709 12:18:20.399231 (+1066922us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54001f68d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:20.399429 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1068647680}
I20260709 12:18:20.423828 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:20.413683 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:20.389539 12644 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:18:20.413422 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.406401 27024 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:20.429801 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:20.411211 27188 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:20.442195 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.417313 27180 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:20.411461 27121 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:20.444543 12638 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 36: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:20.468703 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.348469 27256 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 49.32, Last appended by leader: 32, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.478199 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:20.491833 12495 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:20.438313 27024 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:20.467666 27078 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:20.471089 27097 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:20.506721 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83012.1us
I20260709 12:18:20.521107 27080 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.523969 27080 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:20.526273 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.529920 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.530812 12644 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:20.499563 27024 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:20.509428 27204 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:20.478717 27283 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.069s
I20260709 12:18:20.494592 27242 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.556915 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.535499 26955 raft_consensus.cc:2764] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election decision vote started in defunct term 46: won
I20260709 12:18:20.526729 26883 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
W20260709 12:18:20.572969 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 139343us
I20260709 12:18:20.575173 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:20.548427 27078 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:20.559463 27241 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:20.568843 27282 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:20.590495 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 491456us
I20260709 12:18:20.541586 27097 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.637084 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.577831 27275 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:20.661305 27156 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
W20260709 12:18:20.661724 27272 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:18:20.662942 12646 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 46 index: 28. (index mismatch)
I20260709 12:18:20.617554 27204 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.661896 27156 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:18:20.665843 27273 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:18:20.668104 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.670698 27279 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:20.671236 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.672101 12572 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:20.657882 27173 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:20.695528 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90469.7us
I20260709 12:18:20.696405 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:20.702207 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:20.712042 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179411us
I20260709 12:18:20.722733 12645 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:20.723032 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:20.683337 27276 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:18:20.653846 27275 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:20.686897 27242 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:20.566284 27121 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 36.27, Last appended by leader: 27, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.711277 27228 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:20.702436 27188 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:20.725212 26955 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.677456 27196 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.733446 27222 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:20.756626 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.659408 27282 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.739596 27275 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:20.741369 27071 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:20.744700 27229 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:18:20.756798 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60488.8us
W20260709 12:18:20.753427 27284 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.195s	user 0.000s	sys 0.002s
I20260709 12:18:20.760975 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.765955 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.769243 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.768939 27229 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:20.770216 27228 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:20.772334 27069 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:20.773777 27270 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260709 12:18:20.758551 27156 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.777228 27229 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.782196 27228 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:20.755873 26955 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.783804 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:20.790542 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.773428 27183 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:20.795207 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.781456 27174 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.802951 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:20.801519 12638 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:18:20.769335 27071 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.804179 12643 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:20.805454 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:20.807260 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 383241us
I20260709 12:18:20.791823 27184 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:20.808561 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.809285 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:20.810983 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.812368 27071 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:20.797676 27228 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:20.796386 27183 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:20.814674 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.814580 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.804823 27135 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:20.817767 12638 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:20.820317 13261 raft_consensus.cc:1217] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Deduplicated request from leader. Original: 38.24->[40.25-46.28]   Dedup: 40.25->[41.26-46.28]
I20260709 12:18:20.820878 13261 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 25
I20260709 12:18:20.821122 13261 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 42.26
I20260709 12:18:20.821365 13261 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:20.808799 27069 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:20.809731 27078 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:20.825531 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.826812 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.827173 12496 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:20.831756 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:20.831975 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.833024 12497 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 43 because replica has last-logged OpId of term: 42 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 24.
I20260709 12:18:20.833433 27066 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:20.834342 27066 consensus_queue.cc:1243] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 37.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:18:20.834668 12493 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:20.835134 27066 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:20.835580 27066 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:20.836397 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:20.836596 27066 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.837508 12491 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 49 because replica has last-logged OpId of term: 48 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 25.
I20260709 12:18:20.839320 27024 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.840204 26955 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:20.840637 12494 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:18:20.842912 27174 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:20.844954 27066 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:20.846504 27056 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:20.847045 27056 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:20.847977 27056 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.849617 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:20.850052 26496 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:20.850409 26496 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:20.851301 26496 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.849452 27056 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:20.852552 26496 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:20.853055 27241 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:20.832942 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:20.816425 27228 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:20.828001 27078 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:20.866973 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:20.870296 27001 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:20.853598 27241 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:20.845008 27024 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:20.886559 13260 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:18:20.878768 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:20.885855 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:20.861263 27228 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:20.804358 27188 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 42.21, Last appended by leader: 21, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.912338 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:20.913749 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:20.917675 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:20.922020 12645 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 45 because replica has last-logged OpId of term: 44 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 29.
I20260709 12:18:20.923163 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:20.923650 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57069.5us
I20260709 12:18:20.923985 12496 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:20.925354 12646 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.24-41.24]
I20260709 12:18:20.931588 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:20.932114 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:20.932315 12497 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:20.829593 27184 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:20.879644 27078 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:20.938720 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105889us
I20260709 12:18:20.939743 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:20.940773 27298 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:18:20.941754 27302 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:18:20.917018 27304 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:20.944238 27184 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:20.949668 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5867) took 1023 ms. Trace:
I20260709 12:18:20.958120 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:20.852475 27183 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:20.941848 27282 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:20.972735 12569 rpcz_store.cc:276] 0709 12:18:19.925990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.933433 (+  7443us) service_pool.cc:224] Handling call
0709 12:18:19.939358 (+  5925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:20.214692 (+275334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:20.949388 (+734696us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:20.949573 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476008832}
I20260709 12:18:20.976837 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.977003 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.974730 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:20.957731 12536 connection.cc:256] RPC call timeout handler was delayed by -2.4330930429998716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
W20260709 12:18:20.979604 12536 connection.cc:256] RPC call timeout handler was delayed by -2.384493042999793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
W20260709 12:18:20.979921 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2251930429997628s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
W20260709 12:18:20.980228 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9947930429998451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
W20260709 12:18:20.980435 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9821930429998247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20470000000000002s for a response
W20260709 12:18:20.980599 12536 connection.cc:256] RPC call timeout handler was delayed by -1.9632930429997941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20260709 12:18:20.980755 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8714930429998731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20260709 12:18:20.980909 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8588930429998527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
W20260709 12:18:20.981168 12536 connection.cc:256] RPC call timeout handler was delayed by -1.8129930429997785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2235s for a response
W20260709 12:18:20.981396 12536 connection.cc:256] RPC call timeout handler was delayed by -1.802193042999761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2247s for a response
W20260709 12:18:20.981557 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3593930429999546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2739s for a response
W20260709 12:18:20.981945 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0956930429997556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3032s for a response
W20260709 12:18:20.982170 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0155930429998534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31210000000000004s for a response
W20260709 12:18:20.982405 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0002930429998287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3138s for a response
I20260709 12:18:20.979207 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:20.991828 12642 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: [47.21-47.21]
I20260709 12:18:20.998229 27097 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.007274 12643 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:21.005843 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.014218 12489 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:21.006260 26496 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:21.016155 27167 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:21.026003 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.026907 12492 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:21.027124 27183 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:21.027802 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.028981 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:20.971347 27135 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 39.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
W20260709 12:18:21.030879 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76872.4us
I20260709 12:18:21.032214 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:21.034070 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.017228 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.026652 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.043481 13260 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:18:21.046810 12642 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:21.047077 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:21.051666 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.052702 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.053601 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:18:21.043699 27181 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:21.043097 27167 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:21.060189 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:21.062541 27090 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:18:21.063867 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:21.065055 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:19.990885 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:21.068609 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:21.073163 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:21.070022 12489 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:18:21.071893 12569 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:18:21.083601 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.074432 12496 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:21.087462 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.079659 12497 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:21.072297 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.077315 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:21.071147 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.092475 12642 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:21.088550 12566 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:21.112812 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.116032 27228 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.075697 27184 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.042s
I20260709 12:18:21.054834 27056 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:21.120265 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.122172 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.055982 12638 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:21.124202 12492 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
W20260709 12:18:21.114777 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 190055us
W20260709 12:18:21.078240 27180 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.579s	user 0.013s	sys 0.000s
I20260709 12:18:21.127266 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:21.132249 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:21.133852 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.135756 27188 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:21.069199 27181 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:18:21.138175 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60869.7us
I20260709 12:18:21.118747 27241 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.127339 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.140197 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.065582 27256 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:21.140697 12572 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:21.141083 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.141415 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.158332 12496 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:21.080430 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.066790 27167 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:21.143556 12571 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
W20260709 12:18:21.155139 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86632.8us
I20260709 12:18:21.151875 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.161289 12489 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:21.067696 27253 raft_consensus.cc:2764] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
I20260709 12:18:21.067451 27090 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:21.159262 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78976.1us
I20260709 12:18:21.119833 27121 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:21.073931 26496 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:18:21.206650 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:21.150431 27229 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:18:21.207734 27311 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:18:21.151187 27180 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.217976 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:21.230800 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66929.7us
I20260709 12:18:21.227399 12496 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:21.220571 12572 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:21.083918 27275 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:21.212265 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60462us
I20260709 12:18:21.236308 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.163224 27188 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:18:21.222118 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.163480 27174 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.214349 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.240020 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:21.235548 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.244874 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.258790 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.260032 27175 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:21.260692 27175 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.244154 26496 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:21.263118 27275 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:21.263769 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:21.264864 27184 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:21.266980 27184 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:21.267590 27184 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.262951 27175 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.245728 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.268589 27184 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.27, Last appended by leader: 27, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.211789 27317 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:21.270138 27317 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.270480 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 48 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.270659 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:21.271836 27299 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:21.269263 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.272497 27317 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:21.272809 27196 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.216384 27241 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:21.274245 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.275460 12492 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:21.222668 27090 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.277722 12643 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:21.277731 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:21.280431 12490 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 46 index: 28. (index mismatch)
I20260709 12:18:21.259886 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.247339 27188 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.220949 27167 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.21, Last appended by leader: 21, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.283442 27167 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:21.283222 27188 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.247848 27279 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:21.248765 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.251267 27180 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.251928 26932 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:21.288311 26932 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:21.162909 27103 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.078s
I20260709 12:18:21.289933 26932 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.216747 27229 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.292764 27229 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:21.220448 27069 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:21.254769 12488 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:21.288297 12495 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:21.298601 26330 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:21.299250 26330 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.300259 26330 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.304106 27299 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.307161 27204 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:21.308102 27204 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.311502 27204 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.311561 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.312479 12642 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:21.314085 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:21.314666 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:21.257218 27242 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:21.281762 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2798) took 1550 ms. Trace:
I20260709 12:18:21.315794 27242 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:21.305668 27299 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.317193 12572 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:21.317011 27242 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 44.32, Last appended by leader: 32, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.315723 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:21.319043 27304 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:18:21.320012 27304 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:21.320361 27304 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.320446 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:21.315130 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93103.6us
I20260709 12:18:21.243630 27174 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.323251 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.256412 27228 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:21.324093 27174 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.325019 27133 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:18:21.325789 27090 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:21.321394 27304 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 46.19, Last appended by leader: 19, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.327404 27304 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:21.314814 12638 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:21.331436 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71677.1us
I20260709 12:18:21.333019 27227 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:21.333485 27227 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.333830 27227 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:21.334421 27066 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.335150 27066 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.269158 27078 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.336536 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.338371 12495 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:21.338814 27078 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:21.315708 12567 rpcz_store.cc:276] 0709 12:18:19.731738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:19.731952 (+   214us) service_pool.cc:224] Handling call
0709 12:18:19.732357 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:19.891418 (+159061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:21.281499 (+1390081us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:21.281703 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":351784832}
W20260709 12:18:21.339998 27313 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.109s	user 0.000s	sys 0.003s
I20260709 12:18:21.309264 27156 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:21.343773 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.349071 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.350096 12572 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 38 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 20.
I20260709 12:18:21.301533 26330 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:21.353082 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.355096 12572 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:18:21.356509 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.254396 27056 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:21.362428 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.317844 26496 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:21.369973 12637 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:21.233876 27282 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:21.377826 12563 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:21.372074 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.377568 12386 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 38 to 40. New cstate: current_term: 40 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:21.379416 12566 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:21.394022 27066 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:21.394991 26496 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.401827 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.423086 12564 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
W20260709 12:18:21.423629 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 188187us
W20260709 12:18:21.401816 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131240us
I20260709 12:18:21.403028 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.404105 12568 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:21.404255 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:21.425168 13260 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
W20260709 12:18:21.425464 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.405211 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.397609 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.406980 27165 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:21.428917 27241 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:18:21.429484 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57539.3us
I20260709 12:18:21.429840 27317 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:21.430567 27271 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:18:21.431037 27271 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:21.431150 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:21.431749 12568 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:21.408185 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.372555 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:21.403038 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79870.6us
I20260709 12:18:21.416639 12489 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
W20260709 12:18:21.435294 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:21.439546 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67086.7us
I20260709 12:18:21.419509 12638 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:21.419872 13256 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:21.421466 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.427229 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:21.418730 12569 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:21.441032 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.444330 12566 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:21.459043 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:21.462141 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54067.4us
I20260709 12:18:21.467546 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.432780 26496 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:21.463703 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:21.474777 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 48 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.475867 12568 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:21.463093 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.473393 12572 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 23. Preceding OpId from leader: term: 49 index: 24. (index mismatch)
I20260709 12:18:21.447786 27282 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.476539 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:21.439106 27256 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:21.486563 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.445796 27174 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:21.491999 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.493139 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.445447 27317 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:21.468036 27103 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:21.474779 27165 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:21.489981 27183 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:21.518422 27183 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.519682 27183 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.490442 27256 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.460500 27310 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.060s
I20260709 12:18:21.522012 27256 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:21.497123 27275 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:21.522979 27275 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:21.523422 27275 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:21.509078 13257 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 27. Preceding OpId from leader: term: 38 index: 28. (index mismatch)
I20260709 12:18:21.482108 27135 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:21.526175 27135 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:21.527218 27135 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.497778 27071 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:21.499020 27174 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.531226 27174 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:21.512177 27069 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:21.512370 27103 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:21.432019 27271 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.500540 27204 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:21.550714 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.482345 27228 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:21.529978 27317 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:21.510164 27315 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.033s
I20260709 12:18:21.511583 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.538940 12386 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 34 to 40. New cstate: current_term: 40 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:21.541388 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.541582 26330 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:21.563347 12568 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:21.547763 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.564558 12638 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:21.548949 12566 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:21.481029 27282 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:21.551645 12564 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:18:21.547776 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:21.473289 27229 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.576560 12638 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:18:21.578533 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.579480 12638 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 47 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 26.
I20260709 12:18:21.580055 26330 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:21.550280 27071 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:21.590555 27229 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:21.544422 27066 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.037s
I20260709 12:18:21.591526 27066 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:21.550158 27167 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:21.591931 27066 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:21.590432 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48477.3us
I20260709 12:18:21.592281 27103 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 48.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.593070 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:21.594393 12570 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: [38.20-38.20]
I20260709 12:18:21.596104 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.596873 27317 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.598237 27121 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:21.598824 27121 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:21.545933 27216 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.041s
I20260709 12:18:21.599786 27121 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.601061 27165 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:21.601567 27165 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:21.603000 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.605019 27271 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:21.544207 27056 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.039s
I20260709 12:18:21.581868 27090 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:21.574595 27304 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:21.621984 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63343us
I20260709 12:18:21.589897 12386 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 37 to 41. New cstate: current_term: 41 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:21.609057 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.616104 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:21.624650 27071 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:18:21.627764 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 202897us
I20260709 12:18:21.592924 27066 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.633291 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:21.628731 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.625264 27071 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:21.620270 27090 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:21.618768 27167 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.563271 27227 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.638970 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:21.713793 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85123.3us
I20260709 12:18:21.666617 12386 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 37 to 39. New cstate: current_term: 39 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:21.694386 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98366.9us
W20260709 12:18:21.718241 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.669423 26496 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.725999 27227 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.677237 27282 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.731212 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:21.733265 27282 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.734447 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:21.735325 12568 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:21.666497 27066 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:21.762923 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:21.636724 27181 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:21.734704 27024 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:21.628052 27304 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.707726 27090 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:21.771059 26330 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:21.773983 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.630668 27216 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.134s
I20260709 12:18:21.807389 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:21.806788 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:21.721552 27196 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:21.639451 27180 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.128s
I20260709 12:18:21.817922 13258 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:21.764420 27310 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.821933 26330 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:21.848825 12386 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 28 to 35. New cstate: current_term: 35 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:21.852348 12493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.036s	sys 0.022s
I20260709 12:18:21.677469 27165 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 48.25, Last appended by leader: 25, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:21.855975 12493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.037s	sys 0.022s
I20260709 12:18:21.889942 12386 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 42 to 47. New cstate: current_term: 47 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:21.914203 26330 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:21.903214 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:21.917770 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.818360 27315 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:21.923782 26330 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:18:21.923964 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 161804us
I20260709 12:18:21.926949 27107 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:21.927155 27315 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.927595 27107 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.930214 27181 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:21.930792 27181 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:21.931876 27181 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:21.936308 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 230) took 63504 ms (client timeout 15000 ms). Trace:
I20260709 12:18:21.937849 26496 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:21.938730 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:21.939232 27181 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:21.938732 26496 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:18:21.942263 27107 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:21.943565 27167 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:21.850445 27128 raft_consensus.cc:2764] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election decision vote started in defunct term 48: won
I20260709 12:18:21.953570 12568 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:21.957948 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:21.956746 12386 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 47. New cstate: current_term: 47 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:21.945106 27174 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:21.993494 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:21.960282 27241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.005s	sys 0.012s
I20260709 12:18:22.005218 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.006807 12637 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:21.957854 27069 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:21.854372 27310 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:22.009033 27241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.006s	sys 0.012s
I20260709 12:18:22.005056 27175 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:21.996376 12386 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 40. New cstate: current_term: 40 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:22.038404 12569 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.017s	sys 0.014s
W20260709 12:18:22.029258 12565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.002s	sys 0.026s
I20260709 12:18:22.062950 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:21.857672 27071 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.19, Last appended by leader: 19, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.063735 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:22.067212 12386 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 39 to 43. New cstate: current_term: 43 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:18:22.063107 12569 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.017s	sys 0.014s
W20260709 12:18:22.065091 12565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.002s	sys 0.026s
W20260709 12:18:22.030537 27184 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.006s	sys 0.024s
W20260709 12:18:22.049350 26932 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.009s	sys 0.016s
I20260709 12:18:22.084882 12637 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
W20260709 12:18:22.088702 12443 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver 0a29b5d9f9d44298b02ffcd460811eef not available for placement
I20260709 12:18:22.089025 26330 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:21.998936 27256 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:22.056892 27299 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.014s	sys 0.025s
W20260709 12:18:22.068356 27275 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.004s
I20260709 12:18:22.096257 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
W20260709 12:18:22.102610 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.122022 12492 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 32. Preceding OpId from leader: term: 45 index: 33. (index mismatch)
I20260709 12:18:22.123673 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:22.088654 26932 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.009s	sys 0.016s
I20260709 12:18:22.114004 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:22.092527 12386 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 43 to 46. New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:18:22.087458 27184 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.007s	sys 0.025s
W20260709 12:18:22.095824 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 322250us
W20260709 12:18:22.119364 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 480540us
I20260709 12:18:22.140863 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:22.098354 27299 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.014s	sys 0.025s
I20260709 12:18:22.078604 27175 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:18:21.936879 12394 rpcz_store.cc:269] 0709 12:17:18.431828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:19.234826 (+802998us) service_pool.cc:224] Handling call
0709 12:18:21.936054 (+62701228us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:18.135342 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:18.135952 (+   610us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:18.135988 (+    36us) write_op.cc:435] Acquired schema lock
0709 12:18:18.136007 (+    19us) tablet.cc:662] Decoding operations
0709 12:18:18.365401 (+229394us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:18.369723 (+  4322us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:18.381006 (+ 11283us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:18:18.508778 (+127772us) tablet.cc:701] Row locks acquired
0709 12:18:18.513219 (+  4441us) write_op.cc:260] PREPARE: finished
0709 12:18:18.527835 (+ 14616us) write_op.cc:270] Start()
0709 12:18:18.536948 (+  9113us) write_op.cc:276] Timestamp: P: 1783599498527729 usec, L: 0
0709 12:18:18.557701 (+ 20753us) op_driver.cc:348] REPLICATION: starting
0709 12:18:18.558508 (+   807us) log.cc:844] Serialized 2094 byte log entry
0709 12:18:18.928996 (+370488us) op_driver.cc:464] REPLICATION: finished
0709 12:18:19.237877 (+308881us) write_op.cc:301] APPLY: starting
0709 12:18:19.238037 (+   160us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:19.245751 (+  7714us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:19.245908 (+   157us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:19.292544 (+ 46636us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:19.292915 (+   371us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 12:18:19.292983 (+    68us) write_op.cc:312] APPLY: finished
0709 12:18:19.293448 (+   465us) log.cc:844] Serialized 57 byte log entry
0709 12:18:19.295920 (+  2472us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:19.296185 (+   265us) write_op.cc:454] Released schema lock
0709 12:18:19.296254 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:19.296935 (+   681us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:19.296974 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7273,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":306818,"apply.run_cpu_time_us":11116,"apply.run_wall_time_us":60605,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25007,"num_ops":5,"prepare.queue_time_us":590383,"prepare.run_cpu_time_us":5957,"prepare.run_wall_time_us":423512,"replication_time_us":370905,"thread_start_us":439540,"threads_started":2}]]}
I20260709 12:18:22.162061 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:22.162783 12569 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:22.065332 27069 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.158309 27107 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:22.172493 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:22.166832 12386 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 44 to 45, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 45 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:18:22.188237 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.188773 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:22.193024 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:22.196237 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.149645 27307 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.218163 12570 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:22.219089 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:22.212605 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:22.231698 12386 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 44 to 47. New cstate: current_term: 47 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:22.187650 27024 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:22.137444 27275 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.286365 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:22.190975 27107 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.297046 12496 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:22.179703 27167 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:22.191572 27304 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:22.147270 27056 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 44.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.096s
W20260709 12:18:22.169705 27188 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.005s	sys 0.009s
I20260709 12:18:22.212476 27097 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:22.300987 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:22.163165 27241 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 30, Time since last communication: 0.074s
I20260709 12:18:22.365123 12490 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:18:22.366115 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:22.306890 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:22.369123 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:22.369289 12638 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:22.307834 12386 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 32 to 41. New cstate: current_term: 41 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:22.312038 12569 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:22.312685 12488 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:22.312697 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:22.342271 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.384620 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:22.386096 12496 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:22.190310 27103 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.110s
I20260709 12:18:22.300354 27107 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:22.363034 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:22.297900 27174 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:22.300612 26496 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:18:22.286635 27317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.010s	sys 0.025s
W20260709 12:18:22.416683 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 391323us
W20260709 12:18:22.409257 27328 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.102s	user 0.001s	sys 0.001s
W20260709 12:18:22.361447 27188 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.006s	sys 0.010s
I20260709 12:18:22.391080 27056 consensus_queue.cc:1243] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 45.33 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 32
I20260709 12:18:22.381971 27216 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:22.393520 27121 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.443212 12568 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:22.392962 27296 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.437000 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:22.414937 27282 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:22.318674 27167 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:22.421607 27317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.010s	sys 0.026s
I20260709 12:18:22.391957 12386 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 32 to 34, now has a pending config: VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129), VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 34 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } } pending_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:18:22.420786 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 415178us
I20260709 12:18:22.474921 12486 tablet_service.cc:1558] Processing DeleteTablet for tablet a30ed0986fa641bd8f88912b4b9ee320 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 14)) from {username='slave'} at 127.0.0.1:59760
I20260709 12:18:22.455559 27228 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.020s
I20260709 12:18:22.476948 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2816) took 1041 ms. Trace:
I20260709 12:18:22.362098 27307 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.503614 12565 rpcz_store.cc:276] 0709 12:18:21.435337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.435775 (+   438us) service_pool.cc:224] Handling call
0709 12:18:21.448793 (+ 13018us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:21.468109 (+ 19316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:21.468147 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.498523 (+ 30376us) log.cc:844] Serialized 26 byte log entry
0709 12:18:22.090096 (+591573us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:22.090230 (+   134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:22.090421 (+   191us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:22.476631 (+386210us) raft_consensus.cc:1685] finished
0709 12:18:22.476668 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:22.476881 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":15240,"spinlock_wait_cycles":1590528,"thread_start_us":566309,"threads_started":1,"wal-append.queue_time_us":650682}
W20260709 12:18:22.481599 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44684us
W20260709 12:18:22.468307 27271 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.849s	user 0.014s	sys 0.000s
W20260709 12:18:22.476835 27327 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
W20260709 12:18:22.511911 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 338416us
I20260709 12:18:22.379235 27175 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.487531 27097 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:22.530882 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 336515us
I20260709 12:18:22.506134 27188 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:22.537524 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:22.540949 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 172979us
I20260709 12:18:22.507076 27216 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:22.543471 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62477.4us
W20260709 12:18:22.530453 27320 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.443s	user 0.000s	sys 0.001s
I20260709 12:18:22.546906 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:22.548056 13257 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:22.545493 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:22.514262 27310 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:22.490481 27296 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.532691 27317 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.561546 27317 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.556895 26330 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:22.490226 27121 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.564236 27317 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:22.564615 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:22.565253 27121 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:22.543499 27324 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.233s	user 0.002s	sys 0.000s
I20260709 12:18:22.557936 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.566805 27317 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.566740 27296 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:22.569295 27331 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
W20260709 12:18:22.569505 27325 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.339s	user 0.001s	sys 0.000s
I20260709 12:18:22.513033 27024 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:22.583454 27335 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:18:22.584088 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5877) took 1232 ms. Trace:
W20260709 12:18:22.570232 27336 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
W20260709 12:18:22.586211 27329 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:18:22.587342 27275 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:22.551153 27069 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:22.554286 27216 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:22.593772 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.598403 27216 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:22.604457 27090 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.767s	user 0.006s	sys 0.006s
I20260709 12:18:22.605053 27167 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:22.607242 27307 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:22.598438 27330 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.365s	user 0.000s	sys 0.001s
I20260709 12:18:22.539047 27188 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:22.574054 27333 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
I20260709 12:18:22.573856 26496 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:22.553486 27258 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:22.575032 27304 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:18:22.570106 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.622124 12640 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:22.575711 26330 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:22.533893 27256 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:22.585220 27338 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:18:22.608687 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1830) took 1794 ms. Trace:
I20260709 12:18:22.568014 27317 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.633764 27317 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:22.573112 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:22.634769 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2096) took 1154 ms. Trace:
I20260709 12:18:22.539973 27097 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.615574 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:22.623726 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:22.630465 27307 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.630542 26330 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.584578 12567 rpcz_store.cc:276] 0709 12:18:21.351818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.352924 (+  1106us) service_pool.cc:224] Handling call
0709 12:18:21.364344 (+ 11420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:21.364806 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:21.364843 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.408389 (+ 43546us) log.cc:844] Serialized 26 byte log entry
0709 12:18:21.902674 (+494285us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:21.902724 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:21.902811 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:22.583808 (+680997us) raft_consensus.cc:1685] finished
0709 12:18:22.583837 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:22.584017 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":564,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":16271,"thread_start_us":493325,"threads_started":1,"wal-append.queue_time_us":677174}
I20260709 12:18:22.643442 12640 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:22.635288 13254 rpcz_store.cc:276] 0709 12:18:21.480843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.481062 (+   219us) service_pool.cc:224] Handling call
0709 12:18:21.481524 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:21.502323 (+ 20799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:21.502377 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.542289 (+ 39912us) log.cc:844] Serialized 26 byte log entry
0709 12:18:22.057786 (+515497us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:22.057844 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:22.057951 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:22.634409 (+576458us) raft_consensus.cc:1685] finished
0709 12:18:22.634456 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:22.634684 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6259,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":459319,"threads_started":1,"wal-append.queue_time_us":568598}
I20260709 12:18:22.619417 27188 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:22.626820 13261 rpcz_store.cc:276] 0709 12:18:20.814597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:20.815040 (+   443us) service_pool.cc:224] Handling call
0709 12:18:20.820061 (+  5021us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:18:20.834004 (+ 13943us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:18:20.956030 (+122026us) log.cc:844] Serialized 14 byte log entry
0709 12:18:21.248788 (+292758us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b180018ba40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:21.619973 (+371185us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:21.620807 (+   834us) log.cc:844] Serialized 78 byte log entry
0709 12:18:21.621016 (+   209us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:21.627338 (+  6322us) log.cc:844] Serialized 14 byte log entry
0709 12:18:21.630307 (+  2969us) log.cc:844] Serialized 14 byte log entry
0709 12:18:21.632278 (+  1971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:21.665817 (+ 33539us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:22.608399 (+942582us) raft_consensus.cc:1685] finished
0709 12:18:22.608427 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:22.608612 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24315,"prepare.queue_time_us":168523,"prepare.run_cpu_time_us":182,"prepare.run_wall_time_us":3159,"raft.queue_time_us":13850,"raft.run_cpu_time_us":430,"raft.run_wall_time_us":6820,"spinlock_wait_cycles":25017472,"thread_start_us":491336,"threads_started":2,"wal-append.queue_time_us":1034326}
W20260709 12:18:22.651942 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78882.1us
I20260709 12:18:22.651566 27307 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.652693 27344 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:18:22.655272 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.663172 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:22.649505 27334 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:18:22.674996 13257 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:22.605091 27090 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.636341 27258 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:22.716261 27258 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.734813 27216 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.735571 27216 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:22.710749 27347 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:18:22.712898 27097 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:22.713866 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:22.751214 12489 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
W20260709 12:18:22.714241 27066 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.943s	user 0.014s	sys 0.001s
I20260709 12:18:22.714534 27342 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.048s
W20260709 12:18:22.735713 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 68998.9us
I20260709 12:18:22.710372 12491 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 48 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:22.753029 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:22.751832 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:22.757421 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:22.758513 13257 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
I20260709 12:18:22.769618 27307 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:22.769853 27352 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:22.777212 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:22.777806 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.777994 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:22.778725 12643 raft_consensus.cc:2368] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37.
I20260709 12:18:22.778499 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:22.738234 27183 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:22.781569 12638 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: [46.19-46.19]
I20260709 12:18:22.769038 27107 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.784581 27107 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:22.785208 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:22.750020 27069 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:22.786397 27069 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.788234 27216 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:22.785394 27358 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.010s
I20260709 12:18:22.790544 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5623) took 1025 ms. Trace:
I20260709 12:18:22.760784 27296 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.794929 27296 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.800850 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:22.801894 12567 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:18:22.772841 27097 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:22.804329 27097 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:22.804843 27097 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:22.812003 12638 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:22.725344 27315 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:22.817168 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.817703 12644 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:22.818002 12643 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:22.819221 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:22.826076 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74313.2us
I20260709 12:18:22.825487 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:22.794186 12563 rpcz_store.cc:276] 0709 12:18:21.765085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.765282 (+   197us) service_pool.cc:224] Handling call
0709 12:18:21.812528 (+ 47246us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:21.813014 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:21.813055 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.890153 (+ 77098us) log.cc:844] Serialized 26 byte log entry
0709 12:18:22.297745 (+407592us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:22.298198 (+   453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:22.298284 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:22.790293 (+492009us) raft_consensus.cc:1685] finished
0709 12:18:22.790320 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:22.790474 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3551,"raft.queue_time_us":622,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":19930,"thread_start_us":363702,"threads_started":1,"wal-append.queue_time_us":625517}
I20260709 12:18:22.837543 12644 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:18:22.837934 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:22.839299 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.840004 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:22.841029 12567 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:22.792713 27069 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:22.863941 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:22.860041 12642 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 21. Preceding OpId from leader: term: 43 index: 22. (index mismatch)
I20260709 12:18:22.861064 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.865272 12645 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:22.865135 12644 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:22.810350 27304 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:22.872531 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45453.2us
I20260709 12:18:22.873528 12497 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:22.826321 27296 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:22.816650 27183 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:22.880046 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54706.1us
I20260709 12:18:22.815977 27307 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.883699 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:22.889907 12489 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:22.861672 27358 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:18:22.886829 12646 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 44 index: 28. (index mismatch)
I20260709 12:18:22.838578 27343 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:22.885843 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:22.875591 27167 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.896379 26330 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:22.897150 26330 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.872954 27346 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:22.901437 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:22.903333 13257 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:22.883996 27196 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.905228 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:22.905339 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:22.910481 12489 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:22.911394 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5895) took 1321 ms. Trace:
I20260709 12:18:22.914495 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:22.920625 12645 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:22.913381 12564 rpcz_store.cc:276] 0709 12:18:21.590480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.597021 (+  6541us) service_pool.cc:224] Handling call
0709 12:18:21.597426 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:21.607408 (+  9982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:21.607443 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.618030 (+ 10587us) log.cc:844] Serialized 26 byte log entry
0709 12:18:22.082389 (+464359us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:22.082434 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:22.082513 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:22.911087 (+828574us) raft_consensus.cc:1685] finished
0709 12:18:22.911118 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:22.911310 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9487,"raft.queue_time_us":10178,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":5566,"thread_start_us":416120,"threads_started":1,"wal-append.queue_time_us":795947}
I20260709 12:18:22.918426 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:22.894369 27317 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:22.922783 27354 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:22.930330 27354 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:22.930812 27354 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:22.936008 12643 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:22.881029 27304 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.902743 27069 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:22.922230 27181 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.981s	user 0.011s	sys 0.006s
I20260709 12:18:22.956506 27304 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:22.958777 27353 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:22.966379 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 542039us
I20260709 12:18:22.924580 27352 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:22.969221 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:22.987692 12570 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:18:22.969954 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 446003us
I20260709 12:18:22.970973 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:22.970315 27183 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.972055 12566 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:22.894114 27307 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:22.975332 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.610s	user 0.038s	sys 0.071s
I20260709 12:18:22.948304 27373 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:22.986248 12495 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 39 because replica has last-logged OpId of term: 38 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 19.
W20260709 12:18:22.992823 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.628s	user 0.038s	sys 0.071s
I20260709 12:18:23.001319 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5614) took 1653 ms. Trace:
I20260709 12:18:22.993916 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:22.995543 12643 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: [46.27-46.27]
I20260709 12:18:22.972939 27358 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:22.903649 27167 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.968629 12563 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:23.008684 12563 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:22.914364 27346 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:22.916709 27196 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.001786 12571 rpcz_store.cc:276] 0709 12:18:21.347452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.347578 (+   126us) service_pool.cc:224] Handling call
0709 12:18:21.347979 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:21.348437 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:21.348469 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.364663 (+ 16194us) log.cc:844] Serialized 26 byte log entry
0709 12:18:23.000911 (+1636248us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:23.000968 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:23.001076 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:23.001121 (+    45us) raft_consensus.cc:1685] finished
0709 12:18:23.001136 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:23.001263 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5467,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":331,"thread_start_us":1635835,"threads_started":1,"wal-append.queue_time_us":775284}
I20260709 12:18:23.019002 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.017078 12645 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 23. Preceding OpId from leader: term: 49 index: 24. (index mismatch)
I20260709 12:18:23.008136 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:23.011719 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:22.911396 27343 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:22.983829 27317 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:23.016052 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:22.991138 27228 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.016343 27103 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.007s
I20260709 12:18:23.003234 27204 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:23.033239 12570 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:23.014879 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.034618 12568 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:23.026320 12571 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:23.034814 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:23.025686 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
I20260709 12:18:23.021071 12642 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:23.027469 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.003540 27056 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:23.007150 27121 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:23.017822 12563 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 44.32, Last appended by leader: 32, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.002991 27253 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:23.008462 27358 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:23.031476 12461 leader_election.cc:400] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37.
I20260709 12:18:23.047326 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.047326 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:23.049007 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.050621 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.051380 12571 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:23.017182 27352 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:23.052280 27056 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.052632 27348 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:23.054330 12563 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:23.054733 27056 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:22.994457 27353 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.055335 27056 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:18:23.057313 27353 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:23.046268 27282 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:23.060104 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:23.052626 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:22.982717 27069 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.066758 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.068015 12570 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:23.053287 27352 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.068461 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5892) took 1563 ms. Trace:
I20260709 12:18:23.062032 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.072517 12571 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 42 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:23.025559 27304 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.070477 12572 rpcz_store.cc:276] 0709 12:18:21.504642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.518506 (+ 13864us) service_pool.cc:224] Handling call
0709 12:18:21.518879 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:21.574671 (+ 55792us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:22.015756 (+441085us) log.cc:844] Serialized 14 byte log entry
0709 12:18:22.410578 (+394822us) log.cc:844] Serialized 14 byte log entry
0709 12:18:22.729059 (+318481us) spinlock_profiling.cc:243] Waited 3.75 ms on lock 0x7b1800156dc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:22.965790 (+236731us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b1800156dc0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:18:22.966019 (+   229us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:22.966077 (+    58us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:22.966123 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:23.068173 (+102050us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:23.068228 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:23.068398 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":571850,"prepare.queue_time_us":1067244,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":114,"raft.queue_time_us":561659,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":243,"spinlock_wait_cycles":147073024,"thread_start_us":819813,"threads_started":2,"wal-append.queue_time_us":503929}
I20260709 12:18:23.032918 27299 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:23.078217 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:23.079506 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.080163 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.086185 12571 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:23.059968 27358 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:23.047811 27343 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:23.092278 13260 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:23.006520 27373 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.066636 27282 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:23.103197 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.078143 27227 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:23.070227 27253 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.135033 27253 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.087781 27296 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:23.037832 27228 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.139889 12563 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 32. Preceding OpId from leader: term: 45 index: 33. (index mismatch)
I20260709 12:18:23.093454 27024 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:23.059657 27121 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.078590 27181 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.060s
I20260709 12:18:23.086527 27304 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:23.170389 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.087323 27184 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.059s
I20260709 12:18:23.176955 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:23.177901 12571 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:23.175038 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.180212 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.150409 27024 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:23.152287 27227 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:23.131361 27343 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.037s
W20260709 12:18:23.177193 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 129951us
I20260709 12:18:23.130911 27097 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.193931 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:23.191797 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:23.198136 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.204712 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.183318 27024 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:23.223686 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.229539 27379 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:23.230191 27379 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:23.230786 27379 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:23.231057 27376 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:18:23.232373 27349 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:23.199046 27315 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:23.233568 27279 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 45 current term: 45 is higher.
I20260709 12:18:23.199311 27181 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.234194 27227 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.234827 27181 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.238462 27165 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:18:23.247774 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161730us
I20260709 12:18:23.253981 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:23.270120 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.270977 12563 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
W20260709 12:18:23.274111 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49246.4us
I20260709 12:18:23.205854 27299 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:23.286517 27196 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.194245 27377 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.215s
W20260709 12:18:23.290374 27367 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
W20260709 12:18:23.290802 27372 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
I20260709 12:18:23.166805 27317 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 49.32, Last appended by leader: 32, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.292296 27097 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:23.308596 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128505us
W20260709 12:18:23.286588 27378 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:18:23.279453 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.295215 27307 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:23.335788 27381 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.106s	user 0.003s	sys 0.000s
I20260709 12:18:23.338598 27228 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:23.339562 27069 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:23.340224 27299 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:23.204942 27282 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.30, Last appended by leader: 30, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.341361 13256 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:23.345224 26330 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:23.384756 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 128713us
I20260709 12:18:23.293978 27349 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
W20260709 12:18:23.348487 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69102.7us
I20260709 12:18:23.355545 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.365298 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.340718 27377 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:23.233481 27315 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.354480 27346 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.358292 27253 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:23.396060 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:23.346661 27184 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.061s
I20260709 12:18:23.420341 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.423177 26330 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:23.429172 26330 consensus_queue.cc:1243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 46.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:18:23.395720 27228 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:23.431268 27304 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.431144 27385 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.008s
I20260709 12:18:23.433212 27279 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20260709 12:18:23.433792 27383 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.111s	user 0.002s	sys 0.000s
W20260709 12:18:23.434676 27375 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.238s	user 0.004s	sys 0.000s
I20260709 12:18:23.435282 27380 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.237s
I20260709 12:18:23.438266 27299 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.394838 27377 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:23.426573 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:23.414569 27181 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.415338 27069 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:23.461665 26330 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:23.483835 26330 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:23.484336 26330 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:23.442646 27373 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.458689 27121 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:23.424593 27135 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.039s
I20260709 12:18:23.463609 27229 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:23.494635 27229 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:23.416692 27253 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:23.467290 27315 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:23.467799 27385 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
W20260709 12:18:23.481066 27188 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.811s	user 0.013s	sys 0.000s
I20260709 12:18:23.462195 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:23.506582 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44503.5us
W20260709 12:18:23.492617 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101466us
I20260709 12:18:23.502810 26330 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.513163 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.515857 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.517009 12644 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:23.518429 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.521044 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.526823 12644 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 22
I20260709 12:18:23.527451 12644 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 46.23
I20260709 12:18:23.528384 12644 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:23.528749 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.501886 27229 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:23.525673 27188 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.504882 27377 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.495095 27358 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.543058 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.502151 27253 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:23.548537 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:23.570735 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61667.2us
I20260709 12:18:23.572252 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:23.526166 27196 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:23.510521 27121 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:23.504264 27385 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:23.559788 27379 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:23.558111 27354 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.627s	user 0.013s	sys 0.001s
I20260709 12:18:23.575237 26330 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.571478 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:23.576236 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:23.628379 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56991us
I20260709 12:18:23.553536 27188 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:23.640326 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 327211us
I20260709 12:18:23.580834 27373 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.663316 27024 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.657212 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:23.702471 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.704003 27056 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:23.704507 27056 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:23.634836 27315 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:18:23.636831 12536 connection.cc:256] RPC call timeout handler was delayed by -1.50083411300011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260709 12:18:23.699295 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:23.701603 12640 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 49 because replica has last-logged OpId of term: 48 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 24.
W20260709 12:18:23.711808 12536 connection.cc:256] RPC call timeout handler was delayed by -1.489134113000091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260709 12:18:23.627115 27349 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:23.687335 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:23.656051 27395 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:23.593441 27121 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:23.712566 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4423341130000153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
W20260709 12:18:23.702385 27333 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:18:23.698099 27228 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:23.724056 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4351341130000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1627s for a response
I20260709 12:18:23.697465 27373 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.634693 27354 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:23.769383 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2090) took 1930 ms (client timeout 2240 ms). Trace:
W20260709 12:18:23.770155 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4252341129999877s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
W20260709 12:18:23.770639 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3883341130001554s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16790000000000002s for a response
I20260709 12:18:23.718704 27395 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:23.771023 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3865341130001525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:18:23.772790 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3685341130001234s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20260709 12:18:23.773339 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3577341130001059s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
I20260709 12:18:23.750560 27181 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:18:23.770079 12639 rpcz_store.cc:269] 0709 12:18:21.839495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:21.853780 (+ 14285us) service_pool.cc:224] Handling call
0709 12:18:23.769180 (+1915400us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:23.769316 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413497088}
I20260709 12:18:23.713881 27216 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:23.773845 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3280341130000579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
I20260709 12:18:23.705475 27056 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.27, Last appended by leader: 27, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:23.770556 27323 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.679s	user 0.007s	sys 0.006s
I20260709 12:18:23.792306 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.793272 12641 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:23.774035 27395 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:23.794373 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:23.795161 13261 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:23.750193 27307 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:23.797940 27307 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.799729 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:23.799531 27229 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.800485 13261 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:23.800616 27307 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:23.801070 12637 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:23.782025 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3217341130000477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
W20260709 12:18:23.805824 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1138341130001663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
W20260709 12:18:23.806273 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0580341130000761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20260709 12:18:23.782125 12645 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:23.807031 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:23.808373 27340 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.809167 27340 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.788287 27024 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.792454 27358 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:23.811887 27056 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:23.812239 27229 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:23.809428 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.812929 12566 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:23.813400 27348 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:23.815229 27348 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:23.782776 27181 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:23.816040 27348 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:18:23.816938 27348 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:18:23.818292 27348 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:23.818351 27354 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:23.818789 27348 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:23.819701 27348 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 47.21, Last appended by leader: 21, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.820258 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.816157 27181 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:23.821108 27348 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:23.822096 27349 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.26, Last appended by leader: 26, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.786996 27216 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.826696 27090 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.816121 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.828804 27097 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:23.829283 27321 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.708s	user 0.001s	sys 0.010s
I20260709 12:18:23.829566 27097 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:23.830246 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.830924 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2211) took 2946 ms. Trace:
I20260709 12:18:23.813364 12566 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:23.774385 27322 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.683s	user 0.014s	sys 0.001s
I20260709 12:18:23.834113 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.820489 27121 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:23.628599 27196 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:23.831382 12493 rpcz_store.cc:276] 0709 12:18:20.884812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:20.884988 (+   176us) service_pool.cc:224] Handling call
0709 12:18:20.916085 (+ 31097us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:20.916427 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:20.916476 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:21.264731 (+348255us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:21.265177 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:18:21.910536 (+645359us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:21.910577 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:21.910656 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:23.705939 (+1795283us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:23.830643 (+124704us) raft_consensus.cc:1685] finished
0709 12:18:23.830680 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:23.830829 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43999872,"thread_start_us":644902,"threads_started":1,"wal-append.queue_time_us":755121}
I20260709 12:18:23.821527 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1836) took 1274 ms. Trace:
I20260709 12:18:23.831801 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.844266 12566 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.847353 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:23.850994 12642 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:18:23.848932 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:23.852182 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.840632 13260 rpcz_store.cc:276] 0709 12:18:22.547637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.554045 (+  6408us) service_pool.cc:224] Handling call
0709 12:18:22.570042 (+ 15997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:23.097203 (+527161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:23.821204 (+724001us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b540056a9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:23.821446 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1200393984}
I20260709 12:18:23.842525 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:23.856241 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.857316 12566 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:23.861243 12641 raft_consensus.cc:2368] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 48. Current term is 49.
I20260709 12:18:23.861711 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.867749 13261 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:23.861791 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:23.869016 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:23.839075 27354 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:23.714619 27385 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.27, Last appended by leader: 27, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:23.870114 12563 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 49
I20260709 12:18:23.915583 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:23.850507 27396 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:23.828837 27379 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.908815 12642 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:18:24.013775 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:23.966933 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:24.015662 12567 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:23.989499 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:23.991223 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:24.004812 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 299433us
I20260709 12:18:23.907325 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:24.006757 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:24.003331 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:24.016979 12569 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 36: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:24.008104 12641 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:24.009579 12566 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
I20260709 12:18:23.855561 27377 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:23.855684 27121 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:18:23.929067 27322 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:18:24.018832 12568 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 49 because replica has last-logged OpId of term: 47 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 20.
I20260709 12:18:23.950395 27388 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:23.855214 27204 raft_consensus.cc:2764] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
I20260709 12:18:23.870185 27056 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:23.850248 27397 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.168s
I20260709 12:18:23.949839 27403 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:24.009938 27396 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:23.889798 27097 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:23.920248 27340 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.028803 12638 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: [35.25-36.26]
I20260709 12:18:24.031991 12380 catalog_manager.cc:5028] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): tablet a30ed0986fa641bd8f88912b4b9ee320 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:18:24.014638 27196 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.042153 27388 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.064545 27373 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.045246 12567 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:24.046461 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:24.071390 12644 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:24.049410 13260 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 36.
W20260709 12:18:24.029099 27380 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 41 is higher.
I20260709 12:18:24.054565 12639 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:23.946020 27181 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.078353 27395 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.071609 12642 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:24.080158 27396 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.066360 27121 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:18:24.018088 27354 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.101327 12644 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:18:24.102675 12641 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:24.067493 27397 consensus_queue.cc:1243] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 48.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:24.064592 27388 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.110001 27204 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:24.115527 27396 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:24.120394 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5451) took 1119 ms. Trace:
I20260709 12:18:24.120992 12643 rpcz_store.cc:276] 0709 12:18:23.000531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:23.004698 (+  4167us) service_pool.cc:224] Handling call
0709 12:18:23.005040 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:23.088913 (+ 83873us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:23.088969 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:23.089043 (+    74us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:23.089091 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.120088 (+1030997us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b54001f7cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:24.120146 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:24.120316 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192200576}
I20260709 12:18:24.126909 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.127995 12490 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:24.102211 27373 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:24.130013 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.130854 12491 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:24.130666 27373 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.131976 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:24.132791 12496 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:24.133773 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.110638 27204 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:24.148118 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.151723 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.155380 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.156790 12489 raft_consensus.cc:2368] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37.
I20260709 12:18:24.158419 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.170101 12494 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: [44.24-44.24]
W20260709 12:18:24.170769 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:24.171360 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:24.174170 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5790) took 1935 ms (client timeout 1915 ms). Trace:
W20260709 12:18:24.174729 12495 rpcz_store.cc:269] 0709 12:18:22.238930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.296824 (+ 57894us) service_pool.cc:224] Handling call
0709 12:18:24.173880 (+1877056us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:24.174033 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333527296}
I20260709 12:18:24.175868 12638 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: [43.22-43.22]
I20260709 12:18:24.176811 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.177454 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.178422 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.178972 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.179628 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.180068 12566 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:24.180512 12566 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:24.181488 12566 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.183328 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.178927 12564 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:24.187220 12564 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:24.187672 12571 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 44 index: 28. (index mismatch)
I20260709 12:18:24.189505 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:24.190325 12643 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:24.139904 27121 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:24.194370 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:24.198997 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:24.194294 12566 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:24.202040 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:24.203008 12565 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:24.205017 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:24.202286 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:24.205844 12571 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:18:24.212100 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.216379 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.170104 27069 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:24.186640 27377 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.222579 27354 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.223506 27181 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.230194 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.231364 12570 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:24.232800 12569 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:24.233387 12569 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:24.220281 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 358586us
I20260709 12:18:24.205397 27229 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:18:24.240130 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 247067us
I20260709 12:18:24.242035 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.244995 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.238185 12569 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.246394 12569 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:24.252416 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:24.256603 12569 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 22. (term mismatch)
I20260709 12:18:24.262136 12569 pending_rounds.cc:85] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:18:24.263095 12569 pending_rounds.cc:107] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 39.22
I20260709 12:18:24.263826 12569 raft_consensus.cc:2889] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:24.242620 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:24.264784 27349 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:18:24.275451 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.270419 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:24.221060 27228 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:24.271622 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:24.227689 27204 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.27, Last appended by leader: 27, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.307225 27204 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:24.235227 27377 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.208009 27121 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.310408 12495 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
I20260709 12:18:24.280447 27354 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.032s
I20260709 12:18:24.281075 27228 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:24.273406 12566 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:18:24.220257 27135 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:24.277980 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:24.314977 12610 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.915s (SENT) [suppressed 1 similar messages]
I20260709 12:18:24.278977 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.221382 27069 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:24.314915 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.314051 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.322598 12496 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 39 index: 20. (term mismatch)
I20260709 12:18:24.316987 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.320384 12571 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:24.321676 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50138.2us
I20260709 12:18:24.323103 12496 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 19
I20260709 12:18:24.324594 12496 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 38.20
I20260709 12:18:24.276072 27408 raft_consensus.cc:2764] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election decision vote started in defunct term 47: won
I20260709 12:18:24.324945 12496 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:24.326717 12491 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 49 because replica has last-logged OpId of term: 47 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 20.
I20260709 12:18:24.320138 27228 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:24.335457 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2098) took 1814 ms. Trace:
I20260709 12:18:24.337457 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:24.332233 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.364903 12565 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
W20260709 12:18:24.362748 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.376842 26932 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.377883 27056 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.377537 26932 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.372305 12496 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 32. Preceding OpId from leader: term: 50 index: 33. (index mismatch)
I20260709 12:18:24.378479 27056 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.371163 13256 rpcz_store.cc:276] 0709 12:18:22.520698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.521044 (+   346us) service_pool.cc:224] Handling call
0709 12:18:22.536687 (+ 15643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:23.390129 (+853442us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:24.335168 (+945039us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:24.335388 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1849955072}
W20260709 12:18:24.324447 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44013.4us
W20260709 12:18:24.367468 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97155.2us
I20260709 12:18:24.367497 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:24.329278 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 126407us
I20260709 12:18:24.384220 12564 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:24.384358 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.366468 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:24.383103 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:24.385766 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:24.386705 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.387171 27024 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:24.387604 27416 raft_consensus.cc:2764] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election decision vote started in defunct term 43: won
I20260709 12:18:24.388160 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.386051 27415 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:24.389369 27416 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:24.381209 27056 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.390010 27416 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:24.391311 27416 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 46.19, Last appended by leader: 19, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:24.391929 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.385412 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:24.398694 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.400614 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:24.402205 12496 raft_consensus.cc:606] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:24.402542 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:24.403976 27354 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:24.403980 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:24.404913 27354 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:24.405524 27135 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:24.406091 27354 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.21, Last appended by leader: 21, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.407281 27024 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:24.407788 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:24.409409 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.371657 27409 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.412940 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:24.413321 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.413756 12493 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:24.417661 27409 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.418081 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.418860 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:24.419906 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:24.420173 27409 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.335538 27121 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.422726 27282 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:24.423234 27282 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:24.423637 27282 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:24.413908 27253 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.821s	user 0.019s	sys 0.000s
I20260709 12:18:24.426265 27253 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.427235 12569 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: [41.29-41.29]
I20260709 12:18:24.428416 12569 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:24.428469 27253 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.327946 27377 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.332310 27069 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.29, Last appended by leader: 29, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.432156 26330 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:18:24.432458 12563 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:24.325047 27422 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:24.432830 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:24.433907 12569 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:24.445823 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:24.444428 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:24.415680 27275 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.175s	user 0.008s	sys 0.034s
I20260709 12:18:24.449590 27377 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:24.450290 27377 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:24.450408 27275 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.210s	user 0.009s	sys 0.035s
I20260709 12:18:24.450743 27377 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:24.458666 27409 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:24.462657 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:24.464740 27409 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:24.465330 27409 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:24.465278 27184 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.007s
I20260709 12:18:24.471922 27380 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:24.472776 27380 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.473466 27409 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.475709 27380 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.374447 27349 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:24.333618 27395 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:24.483145 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 168200us
I20260709 12:18:24.484916 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:24.487166 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51845.9us
I20260709 12:18:24.487972 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:24.495479 12646 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
I20260709 12:18:24.432929 27024 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.066s
I20260709 12:18:24.505571 12572 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:24.507215 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.406106 27135 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:24.514343 12610 leader_election.cc:400] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37.
I20260709 12:18:24.444308 27069 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:24.516472 12493 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:24.519773 12610 leader_election.cc:403] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:24.522444 13261 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:24.523774 12493 raft_consensus.cc:606] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:24.522933 27253 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:24.482627 27349 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:24.500285 27374 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.510039 27001 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:24.529829 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.522099 27181 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:24.594769 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.522225 27315 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:24.547780 27349 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:24.546568 27001 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:24.763159 27377 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.773926 27165 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.774607 27165 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.774718 27135 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.548024 27253 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.613332 27181 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.613241 27380 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.201s
W20260709 12:18:24.804003 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:24.819072 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.758935 27184 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.757938 27001 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:18:24.828078 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.787012 27348 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:24.797178 27229 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.001s
I20260709 12:18:24.554117 27374 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.831807 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:24.791437 27128 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:24.791777 27121 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.792023 27279 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:24.841655 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.667769 27420 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:24.835474 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:24.836628 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 322528us
I20260709 12:18:24.828704 27395 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.835255 27379 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:24.839088 27424 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:24.795919 27167 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:24.796806 26932 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.799086 27253 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.841595 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:24.847509 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.795883 27315 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.044s
I20260709 12:18:24.838279 12492 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:24.851358 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.829985 27184 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.841634 27229 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:24.854135 27379 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:24.862749 12496 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:24.857352 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 330167us
I20260709 12:18:24.852237 27377 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.840018 27348 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:24.917330 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66777.2us
I20260709 12:18:24.918503 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:24.885921 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.885453 12492 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:24.849519 27282 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.890481 27181 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.844626 27128 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.847640 27279 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.901343 13261 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.27-41.27]
I20260709 12:18:24.860481 27395 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.903754 27374 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:24.846180 27444 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37.
I20260709 12:18:24.920886 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.924340 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.856514 27420 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.921429 27342 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:24.930404 12488 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:18:24.903280 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:24.944597 12567 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:24.845041 27121 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:24.945452 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.948367 12492 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 36 index: 27. Preceding OpId from leader: term: 38 index: 28. (index mismatch)
I20260709 12:18:24.930330 27167 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:24.884503 27424 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.952929 27395 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.952999 27424 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:24.951272 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.932595 27445 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:24.968712 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:18:24.973651 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:24.937265 27229 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.031s
I20260709 12:18:24.977898 12644 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 29. Preceding OpId from leader: term: 42 index: 29. (term mismatch)
I20260709 12:18:24.978407 12644 pending_rounds.cc:85] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 28
I20260709 12:18:24.978813 12644 pending_rounds.cc:107] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 41.29
I20260709 12:18:24.979061 12644 raft_consensus.cc:2889] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:24.979312 12497 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:24.980507 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.962478 27438 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:24.981181 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:24.950475 27315 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.027s
I20260709 12:18:24.986626 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:24.972232 27167 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:24.988086 12490 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 41. Ops: []
I20260709 12:18:24.970113 27443 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:24.991712 12496 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:24.950138 27342 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:24.997969 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:24.999423 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:24.980507 27069 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:24.990111 27374 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:25.001473 12567 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:25.002656 12492 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:24.951162 27379 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.27, Last appended by leader: 27, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.005203 27379 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:25.005277 27184 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.006156 27420 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.008237 27128 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.010247 27279 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:25.009728 27253 consensus_queue.cc:1175] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 42 status { last_received { term: 40 index: 26 } last_committed_idx: 26 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.27-41.27]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:18:25.016326 12497 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:25.021236 27121 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.992517 27229 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.029s
I20260709 12:18:25.019232 27348 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:24.976223 27445 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:18:24.996333 27438 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:25.028767 27228 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.616s	user 0.001s	sys 0.012s
I20260709 12:18:25.028982 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5470) took 1301 ms. Trace:
I20260709 12:18:25.029647 27342 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.029917 27438 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.003358 27374 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:25.029449 12638 rpcz_store.cc:276] 0709 12:18:23.728107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.063276 (+335169us) service_pool.cc:224] Handling call
0709 12:18:24.175526 (+112250us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.186652 (+ 11126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.028736 (+842084us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54002a5dd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:25.028909 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":257206016}
I20260709 12:18:24.994670 27427 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.034s
I20260709 12:18:25.036813 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.019994 27282 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.029379 27228 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.038416 12490 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:25.039613 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:25.040460 27228 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:25.040694 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.042824 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5434) took 2228 ms. Trace:
I20260709 12:18:25.043268 12640 rpcz_store.cc:276] 0709 12:18:22.814573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.814740 (+   167us) service_pool.cc:224] Handling call
0709 12:18:25.042595 (+2227855us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54002a4ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:25.042729 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1980051328}
W20260709 12:18:25.046644 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.047104 12496 raft_consensus.cc:2410] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 50 because replica has last-logged OpId of term: 49 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 28.
I20260709 12:18:25.048228 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.025287 12497 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:25.053432 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:25.062999 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.064431 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.064580 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.066529 12567 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:25.066740 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.067577 12495 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:25.067802 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.068766 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:25.071837 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.072621 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:25.076723 27427 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:25.077409 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:25.080778 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:25.078801 27427 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:18:25.081494 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 158723us
I20260709 12:18:25.085939 27282 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:25.086997 27349 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.091704 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:25.092481 12565 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: [42.21-43.22]
I20260709 12:18:25.079028 27348 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.015s
I20260709 12:18:25.117089 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:25.055732 27001 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.098289 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.098639 27024 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:25.101182 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.104070 12566 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:18:25.096339 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.122521 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.149163 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.154258 12563 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 46 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 26.
I20260709 12:18:25.153052 12491 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:25.160149 12491 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:25.161468 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.163964 12563 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:25.165388 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:25.167155 12563 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 42 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 26.
I20260709 12:18:25.175750 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.115307 27427 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:25.186378 12495 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:18:25.188238 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:25.189375 12566 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:25.118018 27242 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:25.120481 27349 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:25.196244 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.205076 12496 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:25.150451 27379 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:25.109613 27282 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.198220 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:25.151885 27024 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:25.207943 26330 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:25.220634 12565 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:25.224056 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:25.192058 27343 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.006s	sys 0.016s
I20260709 12:18:25.227442 27001 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:25.229302 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.208796 27275 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.233770 27275 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.236423 27275 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:25.238113 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.238885 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.239490 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.240000 12572 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 49 because replica has last-logged OpId of term: 47 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 26.
I20260709 12:18:25.240988 27184 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:25.241487 27184 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:25.241768 27343 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.007s	sys 0.016s
I20260709 12:18:25.242007 27184 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:25.242578 27343 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:18:25.245493 27345 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:18:25.247078 27343 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:25.249114 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:25.249871 12493 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:25.250916 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5647) took 1432 ms. Trace:
I20260709 12:18:25.251338 12568 rpcz_store.cc:276] 0709 12:18:23.818875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:23.819933 (+  1058us) service_pool.cc:224] Handling call
0709 12:18:25.250616 (+1430683us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54003e0154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:25.250793 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453695872}
I20260709 12:18:25.254142 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:25.254829 12646 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:25.253113 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.249933 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:25.258081 12495 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:25.260203 27374 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.241326 12572 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 46 index: 31. (index mismatch)
I20260709 12:18:25.262980 12566 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
W20260709 12:18:25.267074 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:25.273324 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:25.269127 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48366.4us
I20260709 12:18:25.277814 12564 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:18:25.280424 12609 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:25.280819 12609 leader_election.cc:403] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Cancelling election due to peer responding with higher term
W20260709 12:18:25.284412 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 181969us
I20260709 12:18:25.280905 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.262370 27444 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:25.287024 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.302206 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:25.303429 12646 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:18:25.305025 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:25.297751 27296 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:18:25.298147 26496 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:25.280761 27342 raft_consensus.cc:993] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
W20260709 12:18:25.305364 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60118.9us
I20260709 12:18:25.275240 27349 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:25.211454 27242 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.307880 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:25.326139 12564 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 50
I20260709 12:18:25.309044 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.292636 12493 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
W20260709 12:18:25.316155 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:25.335114 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48193.7us
W20260709 12:18:25.284847 27315 consensus_queue.cc:1175] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 44 status { last_received { term: 44 index: 24 } last_committed_idx: 24 last_received_current_leader { term: 44 index: 24 } }
I20260709 12:18:25.248431 27343 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.301287 27444 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:25.384968 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:25.395519 12566 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 22.
I20260709 12:18:25.306820 27229 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:25.310211 27296 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:25.396865 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84871.4us
I20260709 12:18:25.401846 12610 leader_election.cc:400] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:25.308439 27445 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.075s
I20260709 12:18:25.432319 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5812) took 1234 ms. Trace:
W20260709 12:18:25.433368 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.398263 27444 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:25.338013 27349 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.456090 27349 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.457989 27282 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.332211 27346 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.130s
W20260709 12:18:25.466171 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88011.1us
I20260709 12:18:25.467250 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.377759 27307 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.468529 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.463066 27346 consensus_queue.cc:1243] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 37.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:18:25.378057 27459 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:25.393896 27377 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:25.414520 27181 raft_consensus.cc:1081] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
W20260709 12:18:25.415140 27453 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
W20260709 12:18:25.415239 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.333196 27024 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 43.22, Last appended by leader: 22, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.334913 26496 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:25.323943 27427 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.405567 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.516222 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.432821 12494 rpcz_store.cc:276] 0709 12:18:24.198375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.198567 (+   192us) service_pool.cc:224] Handling call
0709 12:18:24.215986 (+ 17419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.216407 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.216441 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.216946 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:18:25.089872 (+872926us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b180041ba80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:18:25.207638 (+117766us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:25.207707 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.209504 (+  1797us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.431974 (+222470us) raft_consensus.cc:1685] finished
0709 12:18:25.431995 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.432249 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1298,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":12872,"spinlock_wait_cycles":6823296,"thread_start_us":117480,"threads_started":1,"wal-append.queue_time_us":1211188}
W20260709 12:18:25.516626 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111168us
I20260709 12:18:25.517249 13260 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 47 because replica has last-logged OpId of term: 46 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 27.
I20260709 12:18:25.471524 27456 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:25.490661 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 370539us
I20260709 12:18:25.512017 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 36 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.522284 13254 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:25.523311 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.488749 27377 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:25.544358 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:25.545606 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5938) took 1028 ms. Trace:
I20260709 12:18:25.556761 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:25.557016 12567 rpcz_store.cc:276] 0709 12:18:24.517496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.076402 (+558906us) service_pool.cc:224] Handling call
0709 12:18:25.079661 (+  3259us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.080033 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.080458 (+   425us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:25.082031 (+  1573us) log.cc:844] Serialized 26 byte log entry
0709 12:18:25.533336 (+451305us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:25.533381 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.533462 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.545299 (+ 11837us) raft_consensus.cc:1685] finished
0709 12:18:25.545330 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.545530 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2136,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":14166,"thread_start_us":357453,"threads_started":1,"wal-append.queue_time_us":387657}
W20260709 12:18:25.545893 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77459.2us
I20260709 12:18:25.548385 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.554347 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.566175 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:25.567297 13261 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:25.477170 27307 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.520673 27275 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.577021 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:25.541163 27342 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:25.542790 27459 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:25.536278 27024 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:25.582255 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5937) took 1073 ms. Trace:
I20260709 12:18:25.590903 12571 rpcz_store.cc:276] 0709 12:18:24.508394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.903094 (+394700us) service_pool.cc:224] Handling call
0709 12:18:24.903703 (+   609us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.904218 (+   515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.904276 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:24.904975 (+   699us) log.cc:844] Serialized 26 byte log entry
0709 12:18:25.070609 (+165634us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:25.070677 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.070792 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:25.581970 (+511178us) raft_consensus.cc:1685] finished
0709 12:18:25.581997 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.582160 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1668,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":10173,"thread_start_us":165199,"threads_started":1,"wal-append.queue_time_us":293785}
W20260709 12:18:25.583173 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 342428us
I20260709 12:18:25.536883 27416 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:25.558786 27377 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:25.587168 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:25.596614 12646 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:25.570004 27097 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.023s
W20260709 12:18:25.614249 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.542011 27456 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.584448 27358 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.624711 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:25.613406 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:25.627493 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:25.630892 27462 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:18:25.643167 27307 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:25.647523 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.647819 27377 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.648923 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.650067 27377 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.651080 27377 raft_consensus.cc:993] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:18:25.651528 27377 raft_consensus.cc:1081] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:18:25.654702 27296 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.26, Last appended by leader: 26, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.656386 27296 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:25.599048 27459 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:25.657948 27459 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 48.25, Last appended by leader: 25, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.664467 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.587426 27275 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.613927 27167 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.594627 27349 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:25.622471 27379 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:25.595350 27121 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:25.677184 12536 leader_election.cc:400] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 48. Current term is 49.
I20260709 12:18:25.683386 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:25.660967 27466 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.674849 27349 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:25.680070 27167 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.649861 27242 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.679670 27379 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.672020 27135 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.622148 27348 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260709 12:18:25.692692 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:25.698028 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.598366 27416 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.679081 27279 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.597645 26496 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.628549 27397 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.058s
I20260709 12:18:25.664757 27438 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.058s
I20260709 12:18:25.665119 27358 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.707854 27167 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:25.716205 27135 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.714154 27379 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:25.724627 13261 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:25.692795 27445 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.728541 27279 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.773521 27279 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.705739 27229 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election lost for term 44. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
W20260709 12:18:25.728152 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44845us
W20260709 12:18:25.775974 27473 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:18:25.779241 27069 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:25.779842 27069 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:25.692194 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.746693 27346 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:25.782760 27167 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:25.783332 27167 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:25.742774 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166482us
W20260709 12:18:25.746075 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.704826 27466 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.731065 27397 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:25.782020 27429 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:25.791538 27429 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.749025 27348 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:25.795193 27107 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:25.795953 27135 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.800133 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:25.768400 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:25.800604 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 234755us
I20260709 12:18:25.705736 27349 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:25.807765 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:25.807601 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:25.811057 12491 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:25.808604 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.812176 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:25.815181 12639 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:18:25.780846 27069 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.820506 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:25.823581 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.820544 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:25.830080 12639 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:25.837298 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:25.824092 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:25.769289 27097 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:25.870240 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.871712 12645 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:25.875839 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2100) took 1706 ms. Trace:
I20260709 12:18:25.878316 27385 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:25.882637 12644 rpcz_store.cc:276] 0709 12:18:24.169063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.187341 (+ 18278us) service_pool.cc:224] Handling call
0709 12:18:24.187667 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.979662 (+791995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.875386 (+895724us) spinlock_profiling.cc:243] Waited 790 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:25.875729 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1816373504}
I20260709 12:18:25.885524 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.878783 27066 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:25.814157 27443 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.822778 27424 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:25.821713 27373 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.795202 27348 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:25.871263 27473 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:18:25.893131 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.891353 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:25.908618 12645 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:25.800640 27445 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.898897 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:25.899883 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.814476 27107 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.081s
W20260709 12:18:25.899256 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89374.8us
I20260709 12:18:25.814976 27444 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:25.870272 27167 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.889590 27229 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.899366 27395 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:25.906288 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5813) took 1694 ms. Trace:
I20260709 12:18:25.889050 27069 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:25.900695 27184 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.658s	user 0.013s	sys 0.002s
I20260709 12:18:25.921000 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:25.914458 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.952798 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:25.954391 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:25.940603 12489 rpcz_store.cc:276] 0709 12:18:24.213162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.213733 (+   571us) service_pool.cc:224] Handling call
0709 12:18:24.214371 (+   638us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.214724 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:18:24.239687 (+ 24963us) log.cc:844] Serialized 14 byte log entry
0709 12:18:25.789712 (+1550025us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b18002ae080. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:25.905435 (+115723us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:25.905686 (+   251us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:25.905866 (+   180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.905996 (+   130us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:25.906199 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24222,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"raft.queue_time_us":30438,"spinlock_wait_cycles":3335936,"thread_start_us":104106,"threads_started":1}
I20260709 12:18:25.924430 27135 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:18:25.966624 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45723.7us
I20260709 12:18:25.911171 27424 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:25.970094 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:25.887220 27097 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:25.916460 27466 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.981010 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:25.921840 27456 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:25.983062 12644 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:25.986761 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 162739us
W20260709 12:18:25.989045 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 151816us
I20260709 12:18:25.899483 27343 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.056s
I20260709 12:18:25.924772 27484 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:25.940183 27279 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:25.947618 27253 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:25.953420 27482 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:25.989418 27489 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:25.989128 27167 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:18:26.000939 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:25.998878 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 154752us
I20260709 12:18:25.949252 27395 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.007272 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.006210 27253 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.019105 27444 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.015918 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:26.023342 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:25.950932 27229 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:26.068434 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:25.996184 27397 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:26.024690 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:25.981096 27184 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:26.075690 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52513.9us
I20260709 12:18:26.006805 27489 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.004096 27482 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:26.020776 27395 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:26.090437 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:26.083900 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.088177 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.091346 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:26.080991 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92187.6us
I20260709 12:18:26.091373 12571 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:26.092568 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.096266 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:26.092180 12563 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 36.
I20260709 12:18:25.992358 27348 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 47.20, Last appended by leader: 20, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.068145 27167 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:26.109580 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:26.113783 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89162.4us
I20260709 12:18:26.110532 12497 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:26.112195 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:26.019536 27408 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.072s
I20260709 12:18:26.004878 27279 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.132941 27279 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.081944 27253 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:25.904567 27443 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.078687 27397 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:26.133986 27397 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:26.134029 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 1033139a62f44d368f8443d665a2cf22: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 39 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:26.135373 27443 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.115578 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:26.136849 12642 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:26.081365 27343 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.138111 27343 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.138607 27229 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.140561 27343 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:25.951381 27478 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.155s
I20260709 12:18:26.141219 27478 consensus_queue.cc:1243] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:18:26.143842 27097 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.144796 27097 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:26.145218 27097 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.088964 27489 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:26.145713 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:26.121714 27487 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 12:18:26.150278 27167 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.151350 27184 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.122480 27445 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.154826 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:26.122797 27485 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:18:26.123392 27482 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 42.28, Last appended by leader: 28, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.126029 12492 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 37 index: 26. (term mismatch)
I20260709 12:18:26.166837 12492 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 25
I20260709 12:18:26.167223 12492 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 36.26
W20260709 12:18:26.167304 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 92ef671b99df423ea5602f8b7ccff307: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:18:26.167559 12492 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:26.167883 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55785.8us
I20260709 12:18:26.112360 27486 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.130183 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.169826 12645 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
W20260709 12:18:26.121714 27483 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:18:26.130915 27348 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:25.994812 27484 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.174366 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5662) took 1762 ms. Trace:
I20260709 12:18:26.173435 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.185673 27379 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.165942 27097 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:26.186936 27484 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.188237 27379 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:26.183096 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68202.8us
I20260709 12:18:26.183681 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.171519 27475 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:26.185303 12569 rpcz_store.cc:276] 0709 12:18:24.411826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.436916 (+ 25090us) service_pool.cc:224] Handling call
0709 12:18:24.437221 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:24.437575 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:24.437610 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:25.659109 (+1221499us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b180048d4e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:18:25.660127 (+  1018us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:25.673169 (+ 13042us) log.cc:844] Serialized 26 byte log entry
0709 12:18:25.845335 (+172166us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:25.845403 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:25.845509 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.174015 (+328506us) raft_consensus.cc:1685] finished
0709 12:18:26.174048 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.174277 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":33336,"spinlock_wait_cycles":100576768,"thread_start_us":126203,"threads_started":1,"wal-append.queue_time_us":208114}
I20260709 12:18:26.170879 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:26.176626 27493 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:18:26.175102 27279 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:26.178936 27253 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.227296 27253 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.161940 27373 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.021s
I20260709 12:18:26.230547 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.233424 12571 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 42 index: 22. (index mismatch)
I20260709 12:18:26.189775 27482 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:26.255069 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84834.8us
W20260709 12:18:26.264182 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:26.265789 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 342566us
I20260709 12:18:26.284299 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.226101 12569 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:18:26.286715 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.285614 12569 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:26.289232 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.288172 12569 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:26.176898 27486 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.342365 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:26.288404 27498 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:26.388347 27253 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.398015 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:26.399338 12571 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:18:26.399896 12571 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:26.400269 12571 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:26.402354 27496 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
W20260709 12:18:26.402721 27499 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:18:26.350826 27216 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.400305 27342 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.408330 27342 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:26.426172 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:26.427857 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 335372us
I20260709 12:18:26.429121 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.408406 27500 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:26.433944 12565 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:26.451922 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:26.453472 12645 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:26.286648 27478 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
W20260709 12:18:26.306416 27024 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.001s
I20260709 12:18:26.456938 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.471362 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.475598 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.476511 12569 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.468536 12571 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:26.481067 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 191916us
I20260709 12:18:26.486006 12638 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:26.492626 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:26.494571 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.496057 12645 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45. Re-sending same reply.
I20260709 12:18:26.498706 12569 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.501732 12565 raft_consensus.cc:1217] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Deduplicated request from leader. Original: 39.22->[40.23-44.25]   Dedup: 40.23->[42.24-44.25]
I20260709 12:18:26.520954 12565 pending_rounds.cc:85] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 23
I20260709 12:18:26.521275 12565 pending_rounds.cc:107] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.24
I20260709 12:18:26.506018 12571 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.470080 27504 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:26.458652 27500 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:26.474416 27478 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:26.510941 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:26.529470 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:26.530807 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:26.477012 27379 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:26.478451 27484 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:26.521543 12565 raft_consensus.cc:2889] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:26.519776 27349 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.684s	user 0.009s	sys 0.013s
I20260709 12:18:26.520144 27501 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:26.521229 27505 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:26.524207 27504 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:26.551896 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.552771 12645 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:26.532421 27408 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:26.557886 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.520244 27397 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.537238 27484 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:26.551949 27508 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:18:26.569187 12645 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:26.583546 27489 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.587432 27517 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:26.567524 27408 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:26.590134 27408 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:26.593859 27299 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.594658 27299 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.598946 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:26.616230 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58461.3us
W20260709 12:18:26.616796 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:26.558779 27504 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:26.562886 27379 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:26.568866 27484 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:26.627035 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.544582 27349 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:26.607410 27506 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
W20260709 12:18:26.600096 27385 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.699s	user 0.012s	sys 0.000s
W20260709 12:18:26.620081 27395 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.512s	user 0.013s	sys 0.004s
W20260709 12:18:26.636279 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105579us
I20260709 12:18:26.640276 12496 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 22. Preceding OpId from leader: term: 44 index: 22. (term mismatch)
I20260709 12:18:26.597599 27299 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.646077 12496 pending_rounds.cc:85] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 21
I20260709 12:18:26.645579 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:26.646377 12496 pending_rounds.cc:107] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 43.22
I20260709 12:18:26.646673 12496 raft_consensus.cc:2889] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:26.646857 27379 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:26.648528 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.650452 12490 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 46 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 26.
I20260709 12:18:26.618980 27397 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.659376 12570 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
W20260709 12:18:26.640851 27507 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:18:26.669574 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.642980 27253 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:26.673794 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.681218 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.684541 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.667513 27424 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.672456 27397 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.600310 27500 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.666406 27486 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.661248 27229 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:18:26.623386 27478 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 46.19, Last appended by leader: 19, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.692085 27494 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.692931 27489 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:26.693146 27490 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.624s	user 0.002s	sys 0.000s
I20260709 12:18:26.694038 27397 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.738247 27397 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.694869 27519 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:26.743453 27097 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.556s	user 0.012s	sys 0.009s
I20260709 12:18:26.744076 27097 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.698987 27429 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:26.701865 27512 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.192s	user 0.002s	sys 0.000s
I20260709 12:18:26.703392 27349 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:26.704469 27348 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.517s	user 0.021s	sys 0.001s
I20260709 12:18:26.678563 27501 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.756891 27501 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.654173 27395 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.705518 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.710395 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.682245 27475 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.713379 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:26.764060 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50792.5us
I20260709 12:18:26.766799 27348 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:26.770841 27395 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.711434 27424 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.772012 27358 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.731582 27494 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.775636 27494 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.776074 27408 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.778689 27408 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.779767 26496 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:26.780267 26496 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:26.719301 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:26.723928 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5838) took 1203 ms. Trace:
I20260709 12:18:26.783613 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.781100 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:26.784761 13254 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:26.783006 12494 rpcz_store.cc:276] 0709 12:18:25.519927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.574806 (+ 54879us) service_pool.cc:224] Handling call
0709 12:18:25.575096 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.851176 (+276080us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.851211 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:25.889350 (+ 38139us) log.cc:844] Serialized 26 byte log entry
0709 12:18:26.243463 (+354113us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:26.243520 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.243626 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:26.723454 (+479828us) raft_consensus.cc:1685] finished
0709 12:18:26.723487 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:26.723720 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75291,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":593150,"threads_started":2,"wal-append.queue_time_us":438992}
I20260709 12:18:26.749169 27097 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.760931 12645 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:26.786041 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.786397 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.750144 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:26.789383 12642 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:26.791519 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:26.781081 26496 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.28, Last appended by leader: 28, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.793695 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:26.795568 12496 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:26.795857 12496 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:26.794829 12494 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:26.803490 26496 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:26.803723 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.804420 12640 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:26.806005 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.807981 12496 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 49.24, Last appended by leader: 24, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.815232 12496 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:26.816542 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.788604 27503 raft_consensus.cc:2764] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:18:26.799010 27505 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:26.804271 27501 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.817497 12494 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:26.820291 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.788743 27475 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.823114 12646 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:26.824002 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:26.830258 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:26.830369 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.831982 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.792936 27494 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.832069 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:26.824880 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:26.833494 12494 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:26.834514 12494 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:26.814174 27374 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:26.836205 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.837699 12494 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.851090 12494 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:26.851758 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:26.851266 12642 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:26.853122 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.851595 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.813949 27484 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.855690 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:26.857208 13258 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:26.859828 12496 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:26.862820 12646 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:26.863931 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.865185 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.866195 12496 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:26.867345 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:26.868080 12640 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:26.870329 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:26.871167 12642 raft_consensus.cc:2368] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:26.876219 27484 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.837085 27494 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.879057 27475 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.880589 27494 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.869401 27165 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.885769 27518 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:26.886405 27518 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:26.886937 27518 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:26.891166 27275 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.893183 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.895851 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.896607 12496 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:26.899376 12494 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:26.900368 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.902777 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.865331 27504 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.904042 27121 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:26.904623 27121 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:26.906138 27504 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.906873 27121 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.910688 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:26.912894 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.913864 12564 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 44 index: 22. (index mismatch)
I20260709 12:18:26.914364 12568 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:26.916579 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.916960 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.917809 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:26.918561 27165 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:18:26.919799 27342 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.001s
I20260709 12:18:26.920164 12641 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:26.920435 27342 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.920903 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5478) took 1077 ms. Trace:
I20260709 12:18:26.922338 12565 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 39 index: 20. (index mismatch)
I20260709 12:18:26.924528 27097 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:18:26.925899 27500 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:26.911765 27495 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.015s
I20260709 12:18:26.927815 27346 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:26.928495 27346 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:26.929092 27165 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:26.929821 27165 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:26.932186 27165 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:26.935099 27354 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:26.934859 27342 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:26.936203 12568 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:26.938505 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2238) took 2101 ms. Trace:
I20260709 12:18:26.938807 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.938959 12488 rpcz_store.cc:276] 0709 12:18:24.836950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:24.838303 (+  1353us) service_pool.cc:224] Handling call
0709 12:18:24.838725 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:24.944087 (+105362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.938272 (+1994185us) spinlock_profiling.cc:243] Waited 89.4 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:26.938440 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205642368}
I20260709 12:18:26.940264 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.941603 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.943157 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.943351 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:26.943256 27090 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:18:26.944593 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:26.921247 12641 rpcz_store.cc:276] 0709 12:18:25.843407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.919442 (+1076035us) service_pool.cc:224] Handling call
0709 12:18:26.919798 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.920629 (+   831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.920842 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:26.941416 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:26.957458 12642 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:26.974864 12564 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 47 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 18.
I20260709 12:18:26.945307 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:26.975471 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5476) took 1132 ms. Trace:
I20260709 12:18:26.961491 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4378) took 1155 ms. Trace:
I20260709 12:18:26.964730 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:26.954751 12571 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:26.956785 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:26.949451 12567 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:26.976064 12642 rpcz_store.cc:276] 0709 12:18:25.842792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.872507 (+1029715us) service_pool.cc:224] Handling call
0709 12:18:26.957176 (+ 84669us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.975124 (+ 17948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:26.975408 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:26.978408 12572 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:26.952255 12565 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:26.959555 12569 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:26.975817 12488 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:26.979389 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:26.980083 12567 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:26.980118 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:26.976806 12638 rpcz_store.cc:276] 0709 12:18:25.806055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.807520 (+  1465us) service_pool.cc:224] Handling call
0709 12:18:26.961262 (+1153742us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:26.961423 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":981112192}
I20260709 12:18:26.944597 12646 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
W20260709 12:18:26.981473 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123164us
I20260709 12:18:26.981587 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:26.975231 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:26.995712 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.006879 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2843) took 1208 ms. Trace:
I20260709 12:18:27.006817 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.008549 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4392) took 1167 ms. Trace:
I20260709 12:18:26.982831 12496 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:27.006652 27494 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:18:27.009586 12646 rpcz_store.cc:276] 0709 12:18:25.840846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.872029 (+1031183us) service_pool.cc:224] Handling call
0709 12:18:26.944401 (+ 72372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.008248 (+ 63847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.008487 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:27.011896 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 03d0d1b7d2b24046b4dd2cdb4cf7c0af: Illegal state: Replica 6d4ece942fd5414998215d26edf91001 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 47 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:27.007802 12570 rpcz_store.cc:276] 0709 12:18:25.797977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.798111 (+   134us) service_pool.cc:224] Handling call
0709 12:18:25.798472 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.661976 (+863504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.006575 (+344599us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:27.006813 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1972735872}
I20260709 12:18:27.006817 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.017283 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.027174 12564 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:27.032428 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:27.036480 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:26.947166 27505 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.945410 27090 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:26.949319 27504 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:27.038952 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:27.054883 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.059082 12565 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:27.040776 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.060261 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:27.044070 12567 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:26.911559 27216 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:26.952594 27484 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:26.982460 27374 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.048866 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.066776 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5475) took 1224 ms. Trace:
I20260709 12:18:26.948606 27416 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:26.946144 27428 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.112s
I20260709 12:18:26.976462 27354 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:27.060920 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:27.064342 12489 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:27.059549 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:27.062273 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:27.049018 12496 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:27.069541 12643 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:27.045845 26330 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.047847 27121 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:27.023085 27373 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:18:27.070230 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.075297 12642 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:27.056202 27503 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:27.082988 12536 leader_election.cc:400] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:27.083348 12536 leader_election.cc:403] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Cancelling election due to peer responding with higher term
I20260709 12:18:27.076723 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.078428 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.084671 12489 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:27.085162 12494 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:27.085685 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:27.080173 12565 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:27.076681 26330 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.077374 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.062801 27501 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:27.054457 27196 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.037s
I20260709 12:18:27.098521 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.110064 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:27.110747 12494 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:27.111076 12565 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
I20260709 12:18:27.114889 12565 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:18:27.098901 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.191406 12565 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:27.192322 12489 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:27.048034 27184 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:18:27.073271 27374 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.117450 27501 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:27.202525 27475 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:27.203159 27475 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:27.202363 27501 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.204360 27475 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.058544 27505 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.119719 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:27.208499 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88933us
I20260709 12:18:27.063519 27504 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.069478 27443 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.210561 27443 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.108739 27275 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.122529 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.109313 27503 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:27.094388 27416 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:27.196301 27340 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:18:27.219785 27416 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:27.219887 27503 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 49.24, Last appended by leader: 24, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.221411 27416 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.221966 27443 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.125340 27228 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:27.128314 27354 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.140281 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2247) took 1543 ms. Trace:
I20260709 12:18:27.141041 12572 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: [42.22-42.22]
I20260709 12:18:27.108628 27358 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:18:27.078174 27121 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:27.094151 27408 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election lost for term 46. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:27.146832 12495 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:27.149282 27216 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.072039 27484 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.155617 27069 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.164556 26330 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.091730 27090 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:27.170083 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:27.173419 12569 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:27.175231 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.262795 12637 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:27.186971 27488 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.107s	user 0.003s	sys 0.000s
I20260709 12:18:27.196481 27184 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:27.198127 27494 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
W20260709 12:18:27.200348 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5827) took 2152 ms (client timeout 2560 ms). Trace:
I20260709 12:18:27.200712 27374 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.212926 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:27.213821 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:27.094787 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:27.224064 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:27.224080 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 182462us
I20260709 12:18:27.230017 12497 rpcz_store.cc:276] 0709 12:18:25.596544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.704508 (+107964us) service_pool.cc:224] Handling call
0709 12:18:25.704864 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.117302 (+412438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.139944 (+1022642us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:27.140207 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":919332608}
I20260709 12:18:27.236259 12642 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:27.237551 12494 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:27.243778 12495 raft_consensus.cc:606] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:27.223723 27167 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:27.226733 27275 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:27.285204 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.323400 12639 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 38 because replica has last-logged OpId of term: 37 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 25.
I20260709 12:18:27.285251 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.256529 27475 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:27.245707 27121 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:27.325326 12644 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:27.288226 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:27.329304 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 234629us
I20260709 12:18:27.329753 12494 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
W20260709 12:18:27.270252 12490 rpcz_store.cc:269] 0709 12:18:25.048263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.062723 (+ 14460us) service_pool.cc:224] Handling call
0709 12:18:27.199984 (+2137261us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:27.200209 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3646875904}
I20260709 12:18:27.324034 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:27.335111 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 121399us
W20260709 12:18:27.339313 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:27.340150 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.340081 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:27.341513 12639 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:27.255757 27340 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.082s
W20260709 12:18:27.327018 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117599us
I20260709 12:18:27.348212 12570 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 20. Preceding OpId from leader: term: 48 index: 20. (term mismatch)
I20260709 12:18:27.348655 12570 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 19
I20260709 12:18:27.348763 26330 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.349732 12637 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:18:27.349933 26330 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.348894 12570 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 47.20
I20260709 12:18:27.354584 12570 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:27.354571 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.357831 26330 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.336037 27374 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:27.361614 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:27.274421 27216 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.269786 27184 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:27.276302 27484 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.279904 27514 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:27.264432 27479 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:27.335750 27442 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.013s	sys 0.018s
I20260709 12:18:27.437027 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:27.437866 12570 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:27.268939 27069 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:27.446463 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.446704 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.447438 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:27.449963 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:27.450453 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.366410 12494 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 24. Preceding OpId from leader: term: 50 index: 26. (index mismatch)
I20260709 12:18:27.365358 26330 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:18:27.276710 27090 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:27.328233 27342 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.284706 27346 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.396833 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:27.395165 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46239.8us
I20260709 12:18:27.338143 27520 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:27.260298 27518 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.337474 27438 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:18:27.402964 27196 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:27.406313 27184 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.407675 27216 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.410451 27424 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.419658 12638 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:27.429039 12639 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:27.529265 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5488) took 1637 ms. Trace:
W20260709 12:18:27.479159 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44001.8us
I20260709 12:18:27.529683 12638 rpcz_store.cc:276] 0709 12:18:25.892189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.418566 (+1526377us) service_pool.cc:224] Handling call
0709 12:18:27.419077 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.529005 (+109928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.529214 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:27.504338 26330 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:27.530961 12488 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:27.504606 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 107956us
I20260709 12:18:27.531679 12644 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:27.505326 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.511447 12571 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 27. Preceding OpId from leader: term: 48 index: 27. (term mismatch)
I20260709 12:18:27.532516 12571 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 26
I20260709 12:18:27.532814 12571 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 47.27
I20260709 12:18:27.533010 12571 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:27.533990 13260 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:27.495568 27478 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:18:27.480041 27442 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.014s	sys 0.019s
I20260709 12:18:27.515923 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:27.496695 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:27.537901 13256 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:27.506618 12646 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:18:27.539011 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5489) took 1640 ms. Trace:
I20260709 12:18:27.539477 12646 rpcz_store.cc:276] 0709 12:18:25.898497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.428573 (+1530076us) service_pool.cc:224] Handling call
0709 12:18:27.506345 (+ 77772us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.538600 (+ 32255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.538928 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:27.505769 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.531751 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:27.533484 27196 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:27.544413 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:27.545419 26330 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:27.495504 27165 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.542934 27228 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:27.397779 27500 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.120s
I20260709 12:18:27.564232 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:27.526873 27506 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
W20260709 12:18:27.589746 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:27.576498 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 302631us
I20260709 12:18:27.546729 27478 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:27.505275 27342 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.496624 27479 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.584229 12646 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 30. Preceding OpId from leader: term: 46 index: 31. (index mismatch)
I20260709 12:18:27.605190 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:27.481299 27514 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.511197 27438 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:27.614741 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.576046 27373 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.622057 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:27.563117 27424 raft_consensus.cc:2764] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election decision vote started in defunct term 48: won
I20260709 12:18:27.574112 27494 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:27.606840 27478 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:27.629252 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:27.567422 27165 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.567183 27196 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.610052 27184 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:27.652144 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:27.654448 27279 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:27.658378 27279 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:27.659526 27279 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:27.660539 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:27.661785 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:27.662097 27541 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 12:18:27.645607 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:18:27.622459 27416 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:27.568266 27216 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.068s
I20260709 12:18:27.663987 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.664244 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.604862 27497 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.042s
I20260709 12:18:27.666421 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:27.671787 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:27.672070 27165 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.632364 27346 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.673189 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5480) took 1829 ms. Trace:
I20260709 12:18:27.679428 12637 rpcz_store.cc:276] 0709 12:18:25.844077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.337746 (+1493669us) service_pool.cc:224] Handling call
0709 12:18:27.338120 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.350284 (+ 12164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.672941 (+322657us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:27.673115 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12447104}
I20260709 12:18:27.681020 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:27.681126 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.630048 27438 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:27.675679 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.657441 27424 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:27.682427 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 37.
I20260709 12:18:27.639991 27514 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.608367 27090 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 39.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.640358 27342 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.710273 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.642109 27495 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.044s
I20260709 12:18:27.717594 13254 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:27.687471 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:27.732173 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:27.708802 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.744418 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.751245 12637 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:27.741333 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:27.758606 12570 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:27.687918 27484 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:27.664768 27494 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.759593 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:27.679185 27416 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:27.712467 27545 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:18:27.652598 27373 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.780818 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.687186 27181 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:27.784785 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:27.710592 27408 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:27.800531 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.800836 27408 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:27.801295 27408 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.712298 27165 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.709829 27454 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:27.684352 27379 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.801867 27518 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:27.709036 27188 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:27.802904 27069 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:27.802991 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.802361 27379 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.804656 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.805006 27069 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.797858 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.802161 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.801939 27165 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.802821 27188 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.784587 27427 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 45.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.028s
I20260709 12:18:27.805604 27069 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.820093 12637 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:18:27.820873 27542 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.550s	user 0.003s	sys 0.000s
I20260709 12:18:27.821070 27069 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:27.805830 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.822726 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5951) took 1630 ms. Trace:
I20260709 12:18:27.806393 27479 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:27.811240 27188 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.823150 12566 rpcz_store.cc:276] 0709 12:18:26.192316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.193930 (+  1614us) service_pool.cc:224] Handling call
0709 12:18:26.225569 (+ 31639us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:26.236099 (+ 10530us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:26.777595 (+541496us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:18:26.958476 (+180881us) spinlock_profiling.cc:243] Waited 9.65 ms on lock 0x7b18003194e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:18:27.054170 (+ 95694us) log.cc:844] Serialized 14 byte log entry
0709 12:18:27.126272 (+ 72102us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.126845 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.126985 (+   140us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:27.127019 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.127098 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:27.820314 (+693216us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:27.822509 (+  2195us) raft_consensus.cc:1685] finished
0709 12:18:27.822529 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.822662 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32007,"prepare.queue_time_us":726332,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":41656,"raft.run_cpu_time_us":3162,"raft.run_wall_time_us":516687,"spinlock_wait_cycles":64489344,"thread_start_us":111850,"threads_started":2,"wal-append.queue_time_us":180777}
I20260709 12:18:27.812242 27443 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:27.826063 27443 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:27.828655 27443 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.825241 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:27.815207 13257 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:27.815783 13256 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:27.832273 27504 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.816138 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.816455 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.834263 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2245) took 2529 ms. Trace:
I20260709 12:18:27.817165 27121 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.836892 27121 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.814580 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.838522 12564 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:27.834903 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5485) took 1954 ms. Trace:
I20260709 12:18:27.831885 12637 raft_consensus.cc:2376] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38. Re-sending same reply.
I20260709 12:18:27.839840 12644 rpcz_store.cc:276] 0709 12:18:25.880180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.347361 (+1467181us) service_pool.cc:224] Handling call
0709 12:18:27.531491 (+184130us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.533496 (+  2005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.834580 (+301084us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b540018c4d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:27.834809 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422706432}
I20260709 12:18:27.840777 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:27.817755 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.841596 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4406) took 1125 ms. Trace:
I20260709 12:18:27.842152 12644 rpcz_store.cc:276] 0709 12:18:26.716969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.840762 (+1123793us) service_pool.cc:224] Handling call
0709 12:18:27.841058 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.841272 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.841300 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.841347 (+    47us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:27.841377 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.841428 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.841539 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:27.841526 12637 raft_consensus.cc:2376] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45. Re-sending same reply.
I20260709 12:18:27.834600 12492 rpcz_store.cc:276] 0709 12:18:25.305209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.393442 (+ 88233us) service_pool.cc:224] Handling call
0709 12:18:25.393822 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.168153 (+774331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.834044 (+1665891us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:27.834210 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1683021056}
I20260709 12:18:27.713382 27431 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 49.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.082s
I20260709 12:18:27.818490 27494 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.867959 27373 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.805721 27379 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.817067 27416 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 47.21, Last appended by leader: 21, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.787546 27342 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:27.856382 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2126) took 1109 ms. Trace:
I20260709 12:18:27.872207 12637 rpcz_store.cc:276] 0709 12:18:26.746705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.849700 (+1102995us) service_pool.cc:224] Handling call
0709 12:18:27.855673 (+  5973us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.855926 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.855956 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.856030 (+    74us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:27.856065 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.856128 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.856305 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:27.811448 27408 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:27.817341 27495 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:27.808136 27165 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:27.812388 27181 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:27.811687 27454 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.880808 27342 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.906570 27342 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:27.067613 12640 rpcz_store.cc:276] 0709 12:18:25.842488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.872677 (+1030189us) service_pool.cc:224] Handling call
0709 12:18:26.944920 (+ 72243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:26.945149 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:26.945849 (+   700us) log.cc:844] Serialized 14 byte log entry
0709 12:18:27.066377 (+120528us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.066438 (+    61us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:27.066491 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.066568 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:27.066717 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":182,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":93,"thread_start_us":120152,"threads_started":1}
I20260709 12:18:27.913457 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4314) took 1200 ms. Trace:
I20260709 12:18:27.917042 12488 rpcz_store.cc:276] 0709 12:18:26.713092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.038216 (+325124us) service_pool.cc:224] Handling call
0709 12:18:27.913167 (+874951us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54000f78d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:27.913369 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080424832}
I20260709 12:18:27.888831 27495 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:27.938345 12640 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
W20260709 12:18:27.940457 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 251452us
I20260709 12:18:27.943112 27520 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:27.951231 12640 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:27.904000 27181 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:27.959992 12640 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:27.947244 27340 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
W20260709 12:18:27.967533 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 417422us
I20260709 12:18:27.949947 27228 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:28.032370 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.036170 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:27.975428 27482 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.037611 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:27.975317 27340 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:28.039489 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:28.069128 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:28.035125 27342 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:28.047067 27373 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:28.089738 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 437683us
I20260709 12:18:28.039343 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:18:28.095379 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:27.976895 27495 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 45.33, Last appended by leader: 33, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:28.071951 27279 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.102085 27228 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:28.103677 27554 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:18:28.049947 27482 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:28.115064 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75811.6us
W20260709 12:18:28.096665 27549 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:18:28.125372 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5850) took 1219 ms. Trace:
I20260709 12:18:28.107976 27373 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:18:28.143249 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:28.120966 27538 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.848s	user 0.012s	sys 0.000s
I20260709 12:18:28.144709 12496 rpcz_store.cc:276] 0709 12:18:26.905485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.218313 (+312828us) service_pool.cc:224] Handling call
0709 12:18:27.239288 (+ 20975us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.239577 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:27.688858 (+449281us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.124581 (+435723us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.124720 (+   139us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:28.124848 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.125033 (+   185us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.125280 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5718,"prepare.queue_time_us":68157,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":107,"raft.queue_time_us":23379,"thread_start_us":834215,"threads_started":2}
I20260709 12:18:28.092272 27342 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:28.104691 27340 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:28.168411 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.115603 27279 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:28.170343 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2254) took 1400 ms. Trace:
I20260709 12:18:28.173920 27475 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:18:28.175107 27346 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:28.176610 26330 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:18:28.178689 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5686) took 1148 ms (client timeout 1512 ms). Trace:
W20260709 12:18:28.231182 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.179893 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5961) took 1231 ms. Trace:
I20260709 12:18:28.232757 12568 rpcz_store.cc:276] 0709 12:18:26.948034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.978467 (+ 30433us) service_pool.cc:224] Handling call
0709 12:18:26.978856 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.211935 (+233079us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:27.212333 (+   398us) log.cc:844] Serialized 14 byte log entry
0709 12:18:27.425873 (+213540us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:27.425932 (+    59us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:27.425980 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.179477 (+753497us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400045954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:28.179532 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.179823 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27432,"prepare.queue_time_us":5447,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":35986,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":107,"spinlock_wait_cycles":535398784,"thread_start_us":185463,"threads_started":1,"wal-append.queue_time_us":458084}
I20260709 12:18:28.236092 27296 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.180786 12567 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:28.182374 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
W20260709 12:18:28.182936 27552 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:18:28.183406 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.187182 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.188391 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.188673 12645 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:18:28.189615 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.251518 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2129) took 1087 ms. Trace:
I20260709 12:18:28.252470 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.182503 27514 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election won for term 50
W20260709 12:18:28.256068 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.203351 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.210757 27181 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:28.257302 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74005.8us
I20260709 12:18:28.257332 12494 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:18:28.210052 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.225731 12566 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:18:28.260401 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.225463 12489 rpcz_store.cc:276] 0709 12:18:26.770119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.224900 (+454781us) service_pool.cc:224] Handling call
0709 12:18:27.225554 (+   654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.225939 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:27.720077 (+494138us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.169335 (+449258us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.169524 (+   189us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:28.169691 (+   167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.169871 (+   180us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.170196 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":69966,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":710,"raft.queue_time_us":26163,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":245,"thread_start_us":900975,"threads_started":2}
W20260709 12:18:28.230969 12564 rpcz_store.cc:269] 0709 12:18:27.030574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.040467 (+  9893us) service_pool.cc:224] Handling call
0709 12:18:28.178039 (+1137572us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54003ac354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:28.178460 (+   421us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1787762560}
I20260709 12:18:28.261322 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:28.254522 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.215960 27495 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:28.213528 27505 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:28.252043 12645 rpcz_store.cc:276] 0709 12:18:27.164484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.187883 (+1023399us) service_pool.cc:224] Handling call
0709 12:18:28.188300 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.251133 (+ 62833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.251433 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:28.269951 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2130) took 1074 ms. Trace:
I20260709 12:18:28.270525 12645 rpcz_store.cc:276] 0709 12:18:27.195121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.268717 (+1073596us) service_pool.cc:224] Handling call
0709 12:18:28.269186 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.269441 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.269478 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.269543 (+    65us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:28.269592 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.269694 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.269871 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:28.273293 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5978) took 1048 ms. Trace:
I20260709 12:18:28.273921 12572 rpcz_store.cc:276] 0709 12:18:27.224724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.230758 (+  6034us) service_pool.cc:224] Handling call
0709 12:18:27.231328 (+   570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.254733 (+ 23405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.254792 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.260050 (+  5258us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.762475 (+502425us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:27.762519 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.762600 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:28.272964 (+510364us) raft_consensus.cc:1685] finished
0709 12:18:28.273002 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.273210 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35784,"raft.queue_time_us":2081,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":18385,"thread_start_us":455556,"threads_started":1,"wal-append.queue_time_us":645695}
I20260709 12:18:28.276438 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.280812 12492 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:18:28.241888 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.236932 27296 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:28.224157 27479 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:28.262699 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.292227 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.291872 12645 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:18:28.238262 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:28.294147 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55980us
I20260709 12:18:28.297384 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.264592 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.255862 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5975) took 1175 ms. Trace:
I20260709 12:18:28.293399 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.298914 12565 rpcz_store.cc:276] 0709 12:18:27.080436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.192472 (+112036us) service_pool.cc:224] Handling call
0709 12:18:27.192838 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.193676 (+   838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.193722 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.194413 (+   691us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.736049 (+541636us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:27.736118 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:27.736239 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:28.255575 (+519336us) raft_consensus.cc:1685] finished
0709 12:18:28.255608 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.255789 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6517,"raft.queue_time_us":372,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":22348,"thread_start_us":498831,"threads_started":1,"wal-append.queue_time_us":766591}
I20260709 12:18:28.296060 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:28.263749 27275 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:28.256321 27340 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:28.280577 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:28.229136 27475 consensus_queue.cc:1243] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 37.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:18:28.282809 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.282667 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.284823 12496 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:28.285965 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:28.226655 27503 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.006s	sys 0.015s
I20260709 12:18:28.315053 12568 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
W20260709 12:18:28.316324 12611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.003s	sys 0.019s
W20260709 12:18:28.316732 12611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.003s	sys 0.019s
I20260709 12:18:28.317031 12568 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: [39.20-39.20]
I20260709 12:18:28.315063 12496 raft_consensus.cc:614] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:28.319334 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:28.261096 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:28.297876 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.301294 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.302582 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:28.306327 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.015s	sys 0.012s
I20260709 12:18:28.308621 12489 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:28.308938 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.310293 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.290714 27568 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:28.320215 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.016s	sys 0.028s
W20260709 12:18:28.346395 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.016s	sys 0.028s
W20260709 12:18:28.347108 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 587608us
I20260709 12:18:28.347885 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:28.268400 27484 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:28.334674 12572 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:28.324584 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.015s	sys 0.012s
I20260709 12:18:28.338688 12565 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 46 because replica has last-logged OpId of term: 44 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 23.
I20260709 12:18:28.290689 27479 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:18:28.317984 27503 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.007s	sys 0.015s
I20260709 12:18:28.354282 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.266594 27565 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:18:28.291554 27514 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:28.304181 27373 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:28.352880 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.379220 12565 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: [49.29-49.29]
W20260709 12:18:28.380179 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119237us
I20260709 12:18:28.313390 27520 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:28.384445 27296 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:28.385128 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.317556 27181 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:28.334102 27377 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:28.385885 27358 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:28.388317 12492 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:28.399107 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.389453 13261 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
W20260709 12:18:28.462527 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.390622 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:28.453437 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:28.404336 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 364948us
I20260709 12:18:28.465548 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.451041 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:28.453945 12570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.007s	sys 0.015s
I20260709 12:18:28.452013 27504 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:28.397979 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:28.467571 12570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.008s	sys 0.016s
I20260709 12:18:28.476725 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.477303 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:28.480677 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90132.9us
I20260709 12:18:28.482262 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:28.487056 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.471755 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:28.365268 27443 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.094s
I20260709 12:18:28.489913 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.496179 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:28.500586 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5479) took 2656 ms. Trace:
I20260709 12:18:28.501879 12643 rpcz_store.cc:276] 0709 12:18:25.843769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:26.926129 (+1082360us) service_pool.cc:224] Handling call
0709 12:18:26.926539 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.076232 (+149693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.500228 (+1423996us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:28.500502 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":327772928}
I20260709 12:18:28.406524 27373 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:28.501343 12492 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:18:28.509816 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:28.387028 27500 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.008s	sys 0.020s
W20260709 12:18:28.513211 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Illegal state: Replica 32889a507a6b4c74aec421ffbfd1e25d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 38 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:28.513185 27448 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:28.525519 27482 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:28.562593 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:28.514932 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.515080 27279 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:28.516883 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.475682 27121 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 46.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.060s
I20260709 12:18:28.376664 27568 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:28.460628 27181 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:28.520124 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:28.547886 27497 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:28.516544 27358 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.486320 27229 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:18:28.537253 27408 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.653s	user 0.010s	sys 0.003s
I20260709 12:18:28.538295 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:28.542900 27563 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:18:28.543604 27570 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:18:28.544514 27520 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:28.518213 27572 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.538877 27484 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:28.618276 27090 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.009s	sys 0.019s
I20260709 12:18:28.670907 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.514300 27373 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:28.549173 27514 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:28.553980 27500 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.952s	user 0.009s	sys 0.020s
I20260709 12:18:28.498733 27315 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:28.554797 27448 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:28.567514 27430 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.008s	sys 0.016s
W20260709 12:18:28.694142 27242 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.016s	sys 0.009s
I20260709 12:18:28.744491 26496 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:28.705745 27438 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.992s	user 0.012s	sys 0.002s
I20260709 12:18:28.604887 27121 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:28.609480 12489 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
W20260709 12:18:28.602701 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.015s	sys 0.032s
W20260709 12:18:28.752317 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.016s	sys 0.032s
I20260709 12:18:28.753265 12489 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
W20260709 12:18:28.602550 27416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.003s	sys 0.025s
I20260709 12:18:28.596478 27358 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:28.742925 27430 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.008s	sys 0.017s
W20260709 12:18:28.614051 12569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.016s	sys 0.016s
W20260709 12:18:28.776544 12569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.887s	user 0.016s	sys 0.016s
W20260709 12:18:28.748926 27242 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.017s	sys 0.010s
W20260709 12:18:28.548504 12646 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.016s	sys 0.112s
W20260709 12:18:28.789841 12646 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.016s	sys 0.113s
I20260709 12:18:28.589216 27181 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:28.752007 27121 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:28.548365 12639 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.003s	sys 0.019s
I20260709 12:18:28.681809 27577 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.100s
I20260709 12:18:28.685068 12493 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
W20260709 12:18:28.635802 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53984.8us
I20260709 12:18:28.598469 27560 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 50.33, Next index: 34, Last known committed idx: 32, Time since last communication: 0.090s
I20260709 12:18:28.600690 27497 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:28.607163 27408 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:28.689366 27563 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:18:28.503962 27479 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:28.580041 27275 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:28.731806 27571 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.202s	user 0.001s	sys 0.001s
I20260709 12:18:28.750437 12643 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 18. Preceding OpId from leader: term: 48 index: 20. (index mismatch)
I20260709 12:18:28.702754 27090 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:28.577273 27342 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:28.615547 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:28.622570 27572 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:28.743707 27448 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:28.617156 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:28.802690 12639 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.004s	sys 0.020s
I20260709 12:18:28.614261 27520 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:28.803128 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 455288us
W20260709 12:18:28.810254 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 454644us
W20260709 12:18:28.791373 27569 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.273s	user 0.001s	sys 0.000s
I20260709 12:18:28.894599 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:28.806586 27478 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.137s	user 0.017s	sys 0.000s
I20260709 12:18:28.754892 27216 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 37.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.056s
I20260709 12:18:28.816854 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:28.842833 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.789333 27500 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.054s
I20260709 12:18:28.764585 27438 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:28.848242 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:28.883448 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2860) took 1349 ms. Trace:
W20260709 12:18:28.889812 27570 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:18:28.963771 12572 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
W20260709 12:18:28.883572 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 538395us
I20260709 12:18:28.965283 12568 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:28.846884 27567 consensus_queue.cc:1175] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 49 status { last_received { term: 49 index: 29 } last_committed_idx: 29 last_received_current_leader { term: 49 index: 29 } }
I20260709 12:18:28.845374 27275 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:28.798149 27374 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.048s
I20260709 12:18:28.818325 27505 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:28.757668 27482 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:28.846580 27479 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:28.901911 27520 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:28.948503 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48054.2us
W20260709 12:18:28.843972 27578 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:18:28.958456 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5853) took 2047 ms. Trace:
W20260709 12:18:28.895551 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 278499us
I20260709 12:18:28.892948 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.994040 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:28.844396 27562 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.893927 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5490) took 2466 ms. Trace:
W20260709 12:18:28.979202 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 219954us
I20260709 12:18:28.984987 13259 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: []
I20260709 12:18:28.898312 27165 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:28.949491 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.020298 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5851) took 2110 ms. Trace:
W20260709 12:18:28.988720 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 501746us
I20260709 12:18:29.022576 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:29.005111 12639 rpcz_store.cc:276] 0709 12:18:26.427284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.529384 (+1102100us) service_pool.cc:224] Handling call
0709 12:18:27.529752 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.529984 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:27.952500 (+422516us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.893444 (+940944us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.893520 (+    76us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:28.893592 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.893713 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.893849 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3154,"prepare.queue_time_us":608600,"prepare.run_cpu_time_us":179,"prepare.run_wall_time_us":5916,"thread_start_us":1346749,"threads_started":2,"wal-append.queue_time_us":850312}
I20260709 12:18:28.902752 27090 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:29.011230 27275 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:29.012349 27358 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:28.955754 27478 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.014523 27408 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.017546 27216 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:28.985738 27586 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:29.007648 27581 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
I20260709 12:18:29.021292 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:28.993552 12497 rpcz_store.cc:276] 0709 12:18:26.910616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.333155 (+422539us) service_pool.cc:224] Handling call
0709 12:18:27.333717 (+   562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.626970 (+293253us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:27.655946 (+ 28976us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.086446 (+430500us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.087201 (+   755us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.087390 (+   189us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:28.087438 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.169810 (+ 82372us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:28.169868 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:28.958008 (+788140us) raft_consensus.cc:1685] finished
0709 12:18:28.958148 (+   140us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.958357 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29295,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":1786,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":6266,"spinlock_wait_cycles":673346816,"thread_start_us":428368,"threads_started":1,"wal-append.queue_time_us":679954}
I20260709 12:18:29.021018 12493 rpcz_store.cc:276] 0709 12:18:26.910050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.260708 (+350658us) service_pool.cc:224] Handling call
0709 12:18:27.261137 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.807121 (+1545984us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.020002 (+212881us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400193054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:29.020233 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3273936128}
I20260709 12:18:28.888685 27416 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:28.992493 27576 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.376s	user 0.000s	sys 0.002s
I20260709 12:18:29.027840 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.026461 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:29.029183 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5984) took 1557 ms. Trace:
I20260709 12:18:28.981714 27503 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:29.033262 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:29.020161 27574 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.401s	user 0.000s	sys 0.001s
I20260709 12:18:28.896270 27379 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:29.019294 27593 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:18:28.985484 27568 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:28.896135 27589 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.130s
W20260709 12:18:29.075654 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.028530 27374 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
W20260709 12:18:29.038295 27424 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.323s	user 0.009s	sys 0.007s
I20260709 12:18:29.140664 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5985) took 1623 ms. Trace:
I20260709 12:18:29.141320 12570 rpcz_store.cc:276] 0709 12:18:27.517402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.784036 (+266634us) service_pool.cc:224] Handling call
0709 12:18:27.785352 (+  1316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:28.476574 (+691222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.476609 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.477179 (+   570us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.895866 (+418687us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:28.895911 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.895991 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.140284 (+244293us) raft_consensus.cc:1685] finished
0709 12:18:29.140318 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.140573 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14838,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":96,"thread_start_us":1077858,"threads_started":2,"wal-append.queue_time_us":541402}
W20260709 12:18:29.045845 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 237121us
I20260709 12:18:29.079633 12569 rpcz_store.cc:276] 0709 12:18:27.471485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.471707 (+   222us) service_pool.cc:224] Handling call
0709 12:18:27.472183 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.478040 (+  5857us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:18:27.511970 (+ 33930us) log.cc:844] Serialized 14 byte log entry
0709 12:18:27.889274 (+377304us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b1800378700. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:28.777186 (+887912us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.777961 (+   775us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.778173 (+   212us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:18:28.778220 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.778337 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.020990 (+242653us) raft_consensus.cc:1685] finished
0709 12:18:29.028974 (+  7984us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.029122 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8474,"prepare.queue_time_us":33632,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":5637,"raft.queue_time_us":19938,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":11332,"spinlock_wait_cycles":14684544,"thread_start_us":887588,"threads_started":1,"wal-append.queue_time_us":1377775}
W20260709 12:18:29.091373 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65000.3us
I20260709 12:18:28.986239 27505 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.076321 27610 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:28.994096 27482 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:29.078015 27340 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.764s	user 0.008s	sys 0.007s
I20260709 12:18:29.114468 27503 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.056664 27560 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:29.148967 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.172511 12568 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:29.026788 27562 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:29.092273 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 64545.2us
W20260709 12:18:29.151351 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 392406us
I20260709 12:18:29.173873 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.174170 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:29.155609 12570 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:29.167207 12569 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 25. Preceding OpId from leader: term: 50 index: 26. (index mismatch)
W20260709 12:18:29.138154 27583 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.396s	user 0.000s	sys 0.002s
W20260709 12:18:29.166723 27184 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.492s	user 0.012s	sys 0.005s
I20260709 12:18:29.114522 27408 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:29.152225 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2128) took 2209 ms. Trace:
W20260709 12:18:29.152995 27585 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.364s	user 0.000s	sys 0.001s
I20260709 12:18:29.135744 27374 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:29.132998 27588 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:29.157140 27424 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:29.172245 27560 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:29.055781 27416 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.192083 12637 rpcz_store.cc:276] 0709 12:18:26.942432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.883076 (+940644us) service_pool.cc:224] Handling call
0709 12:18:27.883609 (+   533us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.884021 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.884049 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.884647 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.291711 (+407064us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:28.291761 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.291836 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.151950 (+860114us) raft_consensus.cc:1685] finished
0709 12:18:29.151974 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.152149 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":212,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":406700,"threads_started":1,"wal-append.queue_time_us":713481}
I20260709 12:18:29.146633 27619 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:29.052904 27586 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.128669 27379 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.138458 27342 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.046s
I20260709 12:18:29.213976 27602 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.214244 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4397) took 2563 ms. Trace:
I20260709 12:18:29.214515 27613 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:29.209997 27568 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:29.215641 27560 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.29, Last appended by leader: 29, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.243634 27560 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:29.217118 27438 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.206365 27408 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:29.246639 27611 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:18:29.247776 27408 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.218513 27478 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.219045 27482 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:29.199344 27495 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.916s	user 0.011s	sys 0.002s
W20260709 12:18:29.220093 27601 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:18:29.249414 27599 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:18:29.249678 13258 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:29.170387 27610 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.252948 27610 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.253376 27377 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:29.253605 27586 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.254099 27377 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:29.254882 27497 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.256587 27505 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:29.220628 27584 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.374s	user 0.000s	sys 0.001s
I20260709 12:18:29.193254 27430 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:18:29.219147 27582 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.367s	user 0.000s	sys 0.001s
I20260709 12:18:29.247821 27572 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.220254 27597 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:29.231412 27598 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:18:29.257486 27594 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.342s	user 0.002s	sys 0.000s
I20260709 12:18:29.254302 27242 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:29.248543 27596 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
I20260709 12:18:29.234206 27619 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.259370 27613 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.302731 27619 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.305088 27613 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:29.257849 27591 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.341s	user 0.001s	sys 0.000s
I20260709 12:18:29.305338 27619 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:29.263021 27430 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:29.297806 27597 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.323750 27562 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:29.324887 27597 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.291589 27577 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.047s
I20260709 12:18:29.240810 27602 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:29.345729 27501 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.075s	user 0.018s	sys 0.027s
I20260709 12:18:29.365929 27602 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.374905 27568 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:29.417567 12488 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:29.420498 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:29.422348 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 434536us
W20260709 12:18:29.422555 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 529524us
I20260709 12:18:29.422974 12492 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 43 because replica has last-logged OpId of term: 42 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:29.420667 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4304) took 3642 ms. Trace:
I20260709 12:18:29.424760 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.424446 12491 rpcz_store.cc:276] 0709 12:18:25.778114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:25.822544 (+ 44430us) service_pool.cc:224] Handling call
0709 12:18:25.822908 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:25.823266 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:25.823296 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.705054 (+1881758us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:27.718450 (+ 13396us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.284603 (+566153us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:28.284656 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.285222 (+   566us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.419328 (+1134106us) raft_consensus.cc:1685] finished
0709 12:18:29.419425 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.420274 (+   849us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5853,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":107,"spinlock_wait_cycles":28834176,"thread_start_us":493672,"threads_started":1,"wal-append.queue_time_us":918962}
I20260709 12:18:29.429337 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.439020 12564 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: [42.22-42.22]
I20260709 12:18:29.188222 12569 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.27-42.27]
I20260709 12:18:29.441404 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5693) took 1875 ms. Trace:
I20260709 12:18:29.441941 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2863) took 1058 ms. Trace:
I20260709 12:18:29.442672 12569 rpcz_store.cc:276] 0709 12:18:28.383162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.187581 (+804419us) service_pool.cc:224] Handling call
0709 12:18:29.187920 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.440848 (+252928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.441693 (+   845us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.444458 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5998) took 1106 ms. Trace:
I20260709 12:18:29.445101 12568 rpcz_store.cc:276] 0709 12:18:28.337834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.180729 (+842895us) service_pool.cc:224] Handling call
0709 12:18:29.443085 (+262356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.443783 (+   698us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.443817 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.443905 (+    88us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:29.443955 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.444017 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.444199 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:29.446910 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2866) took 1062 ms. Trace:
I20260709 12:18:29.447717 12569 rpcz_store.cc:276] 0709 12:18:28.383960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.445151 (+1061191us) service_pool.cc:224] Handling call
0709 12:18:29.445848 (+   697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.446362 (+   514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.446404 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.446498 (+    94us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:29.446558 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.446631 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.446802 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:29.451020 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 282743us
I20260709 12:18:29.442020 12566 rpcz_store.cc:276] 0709 12:18:27.565265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.303684 (+738419us) service_pool.cc:224] Handling call
0709 12:18:28.305018 (+  1334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:28.305562 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.305605 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:28.318198 (+ 12593us) log.cc:844] Serialized 52 byte log entry
0709 12:18:28.833029 (+514831us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:28.891194 (+ 58165us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.891629 (+   435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.891744 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.440513 (+548769us) raft_consensus.cc:1685] finished
0709 12:18:29.440574 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.441058 (+   484us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":184942,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":8363,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":19774,"spinlock_wait_cycles":19808128,"thread_start_us":363315,"threads_started":1,"wal-append.queue_time_us":654145}
I20260709 12:18:29.452893 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.456904 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:29.458626 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:29.234426 12638 rpcz_store.cc:276] 0709 12:18:26.650845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.548635 (+897790us) service_pool.cc:224] Handling call
0709 12:18:27.552365 (+  3730us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.956726 (+404361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.956772 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.957528 (+   756us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.454850 (+497322us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:28.454919 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.455009 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.213993 (+758984us) raft_consensus.cc:1685] finished
0709 12:18:29.214016 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.214182 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5792,"raft.queue_time_us":6077,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":12520,"thread_start_us":880925,"threads_started":2,"wal-append.queue_time_us":718980}
W20260709 12:18:29.463024 12644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.013s	sys 0.010s
W20260709 12:18:29.463711 12644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.013s	sys 0.010s
I20260709 12:18:29.463900 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5498) took 2414 ms. Trace:
I20260709 12:18:29.465536 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2125) took 2736 ms. Trace:
I20260709 12:18:29.263312 27165 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:29.472028 27165 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:29.472590 26330 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:29.461189 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 254442us
I20260709 12:18:29.473876 26330 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.476686 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.464537 12640 rpcz_store.cc:276] 0709 12:18:27.048739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.031760 (+983021us) service_pool.cc:224] Handling call
0709 12:18:28.069886 (+ 38126us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:28.070448 (+   562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.070486 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.081038 (+ 10552us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.605595 (+524557us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:28.605671 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.605941 (+   270us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.463022 (+857081us) raft_consensus.cc:1685] finished
0709 12:18:29.463065 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.463466 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":10194,"thread_start_us":487403,"threads_started":1,"wal-append.queue_time_us":711975}
I20260709 12:18:29.470331 12644 rpcz_store.cc:276] 0709 12:18:26.729703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.843774 (+1114071us) service_pool.cc:224] Handling call
0709 12:18:27.881684 (+ 37910us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.882002 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:28.368584 (+486582us) log.cc:844] Serialized 14 byte log entry
0709 12:18:29.464530 (+1095946us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.464716 (+   186us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:29.464884 (+   168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.465102 (+   218us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.465397 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":335683,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":179644,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":1581083,"threads_started":3,"wal-append.queue_time_us":656135}
I20260709 12:18:29.473217 27165 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:29.478646 27196 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:29.480331 27165 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:29.483736 27242 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.485868 27275 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:29.484944 27165 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:29.489786 27296 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.490796 27296 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.491760 27090 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.481463 26330 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.493002 27090 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.493727 27296 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.494625 27296 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:29.496606 27342 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.496927 27296 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.497644 27342 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.284729 27374 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 39.20, Last appended by leader: 20, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.457199 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:29.500236 27296 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.500700 27374 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:29.501214 27408 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:29.502399 27408 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.504047 27478 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:29.504966 27478 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:29.505685 27478 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:29.495539 27340 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:29.506808 27340 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.507512 27416 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.327405 27430 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:29.510094 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.503511 27408 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:29.513720 27501 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.242s	user 0.018s	sys 0.028s
I20260709 12:18:29.513942 27495 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:29.515683 27495 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:29.516618 27501 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 46.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:29.519230 27482 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:29.519851 27482 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:29.520056 27501 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.515764 12563 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:29.522643 27503 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.523592 27408 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.495435 27196 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.493669 26330 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.520982 27501 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:29.563396 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.007s	sys 0.005s
W20260709 12:18:29.564015 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.007s	sys 0.005s
W20260709 12:18:29.575670 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117346us
I20260709 12:18:29.500731 27342 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.599649 26330 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.606483 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.615386 27090 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.616066 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.616611 12491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:29.617401 12495 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:29.617733 12491 raft_consensus.cc:606] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:29.619119 26330 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.585861 27416 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:29.521557 27482 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.27, Last appended by leader: 27, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.587141 27340 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:29.626120 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.006s	sys 0.005s
W20260709 12:18:29.626648 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.006s	sys 0.005s
I20260709 12:18:29.629961 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2132) took 1369 ms. Trace:
I20260709 12:18:29.632290 27165 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:29.634791 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.639765 27501 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.630620 12645 rpcz_store.cc:276] 0709 12:18:28.260545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.310026 (+ 49481us) service_pool.cc:224] Handling call
0709 12:18:28.310578 (+   552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:28.311026 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:18:28.311329 (+   303us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.745101 (+433772us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.756559 (+ 11458us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.756790 (+   231us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:18:28.789855 (+ 33065us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.791968 (+  2113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.792083 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.623161 (+831078us) raft_consensus.cc:1685] finished
0709 12:18:29.626892 (+  3731us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.629707 (+  2815us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44909,"prepare.queue_time_us":149,"prepare.run_wall_time_us":43,"raft.queue_time_us":96447,"raft.run_cpu_time_us":475,"raft.run_wall_time_us":41110,"spinlock_wait_cycles":98422912,"thread_start_us":433161,"threads_started":1,"wal-append.queue_time_us":681468}
W20260709 12:18:29.642344 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.637392 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.649348 12497 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:29.649874 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:29.649992 27342 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:29.651094 12569 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:29.653066 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:18:29.586941 27196 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.657886 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.657235 27196 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.665246 27340 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:29.666085 27482 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:29.669914 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.673142 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5866) took 1474 ms. Trace:
I20260709 12:18:29.673549 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:29.684250 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61022.3us
W20260709 12:18:29.674911 27181 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.865s	user 0.017s	sys 0.011s
I20260709 12:18:29.679226 12645 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:18:29.688431 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2868) took 1189 ms. Trace:
I20260709 12:18:29.688597 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5995) took 1464 ms. Trace:
I20260709 12:18:29.669273 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5868) took 1463 ms. Trace:
I20260709 12:18:29.680110 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5970) took 2719 ms. Trace:
W20260709 12:18:29.695623 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.006s	sys 0.021s
W20260709 12:18:29.696228 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.007s	sys 0.021s
I20260709 12:18:29.695847 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:29.690073 12490 rpcz_store.cc:276] 0709 12:18:28.198726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.198945 (+   219us) service_pool.cc:224] Handling call
0709 12:18:28.199444 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.239049 (+ 39605us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.239112 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.239174 (+    62us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:28.239229 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.672423 (+1433194us) spinlock_profiling.cc:243] Waited 39.3 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:29.672508 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.672931 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90465152}
I20260709 12:18:29.679219 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.691965 12564 rpcz_store.cc:276] 0709 12:18:28.213528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.366988 (+153460us) service_pool.cc:224] Handling call
0709 12:18:28.367440 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.439845 (+1072405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.677193 (+237348us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:29.677566 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2462074624}
I20260709 12:18:29.676573 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2257) took 1397 ms. Trace:
I20260709 12:18:29.683607 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:29.694622 12488 rpcz_store.cc:276] 0709 12:18:28.208835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.254119 (+ 45284us) service_pool.cc:224] Handling call
0709 12:18:29.668782 (+1414663us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:29.669095 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1872412416}
I20260709 12:18:29.705569 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:29.707981 27478 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.691411 12568 rpcz_store.cc:276] 0709 12:18:28.499326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.448236 (+948910us) service_pool.cc:224] Handling call
0709 12:18:29.448838 (+   602us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.449997 (+  1159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.450031 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:29.451297 (+  1266us) log.cc:844] Serialized 26 byte log entry
0709 12:18:29.578223 (+126926us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:29.578312 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.587759 (+  9447us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.688128 (+100369us) raft_consensus.cc:1685] finished
0709 12:18:29.688161 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.688360 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":70032,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":311,"thread_start_us":126070,"threads_started":1,"wal-append.queue_time_us":212543}
I20260709 12:18:29.714140 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.715127 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:29.715423 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:29.716305 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58547.6us
I20260709 12:18:29.723485 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:29.727700 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.701543 27181 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.740388 27121 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 46.28, Last appended by leader: 28, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.730365 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.731545 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:29.738734 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55287.1us
I20260709 12:18:29.742206 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:29.750087 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.747695 12640 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:29.750087 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.754801 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5510) took 1140 ms. Trace:
I20260709 12:18:29.756526 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2122) took 3103 ms. Trace:
I20260709 12:18:29.759982 12645 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 20. Preceding OpId from leader: term: 47 index: 21. (index mismatch)
I20260709 12:18:29.694674 12567 rpcz_store.cc:276] 0709 12:18:26.960474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.061603 (+101129us) service_pool.cc:224] Handling call
0709 12:18:28.180525 (+1118922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.237722 (+ 57197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.679652 (+1441930us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003ac350. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:29.679896 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2559553920}
W20260709 12:18:29.766202 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:29.766321 13260 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
W20260709 12:18:29.766561 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45017.9us
I20260709 12:18:29.768152 27196 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.766063 27623 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:29.770598 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.728302 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.786465 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.729336 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.781025 13262 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:29.780874 12643 rpcz_store.cc:276] 0709 12:18:28.614805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.615011 (+   206us) service_pool.cc:224] Handling call
0709 12:18:28.615427 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:28.841098 (+225671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.754265 (+913167us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:29.754629 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309291904}
I20260709 12:18:29.797901 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.781654 27181 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.700753 12494 rpcz_store.cc:276] 0709 12:18:28.279437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.288783 (+  9346us) service_pool.cc:224] Handling call
0709 12:18:28.289318 (+   535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:28.319517 (+ 30199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:28.319566 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:28.346137 (+ 26571us) log.cc:844] Serialized 26 byte log entry
0709 12:18:28.797618 (+451481us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:28.797690 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.675707 (+878017us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:29.675796 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.675887 (+    91us) raft_consensus.cc:1685] finished
0709 12:18:29.675903 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.676320 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5587,"raft.queue_time_us":60049,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":61355904,"thread_start_us":336165,"threads_started":1,"wal-append.queue_time_us":568658}
I20260709 12:18:29.802145 27642 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:29.804672 27642 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.807256 27642 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.781438 12646 rpcz_store.cc:276] 0709 12:18:26.653049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.629425 (+976376us) service_pool.cc:224] Handling call
0709 12:18:27.666288 (+ 36863us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:27.684416 (+ 18128us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:27.684449 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:27.685001 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:18:27.981419 (+296418us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:28.796876 (+815457us) log.cc:844] Serialized 14 byte log entry
0709 12:18:29.049154 (+252278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.755856 (+706702us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:29.755933 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:29.756023 (+    90us) raft_consensus.cc:1685] finished
0709 12:18:29.756134 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.756350 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12190,"prepare.queue_time_us":236482,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":107173,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":79868,"spinlock_wait_cycles":14737792,"thread_start_us":1279043,"threads_started":3,"wal-append.queue_time_us":514804}
I20260709 12:18:29.799304 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.810436 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:29.811113 27520 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.788s	user 0.008s	sys 0.016s
I20260709 12:18:29.810065 27196 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.810640 12564 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:29.814013 27196 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.814849 27196 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.811802 27520 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.808820 27430 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.817879 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.818250 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.819804 12643 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
W20260709 12:18:29.818123 27448 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.862s	user 0.015s	sys 0.010s
I20260709 12:18:29.821303 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.810122 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.820865 27520 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.828812 27430 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:29.830253 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88199.8us
I20260709 12:18:29.829325 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:29.832720 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:29.834616 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45314.7us
I20260709 12:18:29.838893 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.840138 12644 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:29.841282 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.843039 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:29.843653 27377 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.589s	user 0.011s	sys 0.011s
I20260709 12:18:29.843982 12567 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:29.844498 27377 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.841311 27454 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:29.847703 13254 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:29.851370 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.852550 12637 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:29.854477 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:29.854763 27504 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:29.837193 27428 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:29.856225 27428 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:29.855901 27504 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.856616 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:29.858081 27427 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:29.858711 27427 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:29.859011 27504 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.816131 27196 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.858855 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.859921 27427 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 46.28, Last appended by leader: 28, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.860744 12637 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:29.862090 27196 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.855866 27430 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:29.821007 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:29.869510 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.869870 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:29.828824 27373 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.085s	user 0.016s	sys 0.008s
I20260709 12:18:29.870958 12637 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:29.878842 12644 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
W20260709 12:18:29.878098 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.438s (SENT) [suppressed 13 similar messages]
I20260709 12:18:29.826984 27354 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:29.882912 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.887566 12643 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:29.888248 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5999) took 1549 ms. Trace:
I20260709 12:18:29.890908 27279 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:29.891968 27279 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.861928 27428 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.26, Last appended by leader: 26, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.895135 27428 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:29.895577 27504 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:29.896359 27504 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.870339 27430 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:29.898402 27430 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:29.899381 27504 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.901852 27642 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:29.902748 27642 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.903596 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.908951 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:29.888446 27431 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.913321 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:29.888736 12570 rpcz_store.cc:276] 0709 12:18:28.338223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.180828 (+842605us) service_pool.cc:224] Handling call
0709 12:18:29.181359 (+   531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.181732 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:29.189643 (+  7911us) log.cc:844] Serialized 14 byte log entry
0709 12:18:29.682866 (+493223us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b18004de540. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:29.887393 (+204527us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:29.887543 (+   150us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:29.887655 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:29.887804 (+   149us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:29.888045 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1603,"prepare.queue_time_us":131,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":3316,"spinlock_wait_cycles":3709824,"thread_start_us":113413,"threads_started":1}
I20260709 12:18:29.934693 27431 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:29.935669 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.911765 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:29.914625 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.915139 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:29.915398 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:29.938416 27431 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.941404 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:29.918604 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:29.919730 27642 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:29.919993 12640 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:29.943528 12493 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:29.916088 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.865025 27448 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.945322 12641 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:29.945205 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:29.920938 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64532.1us
I20260709 12:18:29.946244 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:29.947204 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5513) took 1330 ms. Trace:
I20260709 12:18:29.947669 12641 rpcz_store.cc:276] 0709 12:18:28.616434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.949146 (+332712us) service_pool.cc:224] Handling call
0709 12:18:29.946916 (+997770us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:29.947120 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2125416832}
I20260709 12:18:29.915735 27373 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:29.949107 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.922942 27597 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:29.926177 27600 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.905431 12637 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:29.927012 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:29.927140 27196 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:29.931288 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:29.934096 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:29.909219 27279 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.940495 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:29.941560 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:29.943933 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.925120 27408 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:29.921849 27504 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:29.950821 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.938550 27377 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:29.975106 27340 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:29.976086 12640 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
W20260709 12:18:29.976985 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45787.9us
I20260709 12:18:29.978519 12643 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:29.979992 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:29.986694 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:29.995952 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:29.995333 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:29.997584 12565 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:29.995368 27196 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.010840 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.011686 27373 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.012110 27377 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:30.012893 27377 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:30.014281 27448 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:29.984912 27340 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.016095 27340 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:30.012837 27279 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:30.018481 27279 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:30.019212 27279 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:30.019939 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:30.020920 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94061.9us
I20260709 12:18:30.022089 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:30.024061 27586 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:30.024843 27586 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:30.024613 27504 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.026247 27586 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.027405 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:29.993873 27430 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.029028 27494 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:30.029228 27597 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.030424 27494 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:30.030685 27430 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.030934 27494 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.031376 27494 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:30.031611 27602 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:30.031934 27597 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.032162 27602 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:30.027429 27504 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.041877 27602 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.043574 27602 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:30.033926 12567 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:30.037724 12490 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:30.036264 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.033209 27377 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.27, Last appended by leader: 27, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:30.037829 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 70077.9us
I20260709 12:18:30.038324 12643 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:30.038527 27454 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:30.046521 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44092.5us
I20260709 12:18:29.986506 27374 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:30.005198 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:30.061758 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.033097 27165 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.072330 12641 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:30.073486 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.074993 12567 raft_consensus.cc:2410] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 44 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:18:29.995045 27600 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.081329 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:30.083834 27597 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:30.084414 27597 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.084820 27597 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:30.086150 12566 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 51 because replica has last-logged OpId of term: 48 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 24.
I20260709 12:18:30.082026 27443 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:30.087522 27443 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:30.089910 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.089967 27520 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:30.091707 27454 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:30.092338 27454 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.092731 27454 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:30.094992 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.096882 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:30.097910 27448 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:30.098376 27448 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.098780 27448 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:30.048247 27408 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.102152 27443 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.102669 12568 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:30.102916 27443 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:30.103865 27408 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.034041 27196 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:30.118186 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 240199us
I20260709 12:18:30.085054 27600 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.134171 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.138036 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:30.070910 27165 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.138132 27373 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.148535 27373 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.151306 27373 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.157882 27427 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:30.175462 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:30.177136 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43764.2us
I20260709 12:18:30.180743 27561 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:30.183616 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:30.162392 27196 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:18:30.190196 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 103772us
I20260709 12:18:30.195284 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:30.199047 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:30.144292 27279 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.198405 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:30.202689 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47343.7us
I20260709 12:18:30.210031 12564 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 19. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:18:30.213090 27577 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:30.213380 27443 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:30.213907 27443 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:30.214318 27443 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:18:30.215210 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4319) took 1934 ms (client timeout 1787 ms). Trace:
W20260709 12:18:30.215884 12495 rpcz_store.cc:269] 0709 12:18:28.281079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.318948 (+ 37869us) service_pool.cc:224] Handling call
0709 12:18:30.213566 (+1894618us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:30.215111 (+  1545us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2544155392}
I20260709 12:18:30.191494 27561 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:30.218624 27561 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 50.33, Last appended by leader: 33, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.220762 27561 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:30.224931 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.226773 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.231369 12569 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:30.234494 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.205430 27196 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:30.237004 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46466.3us
I20260709 12:18:30.242833 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.229516 27416 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:30.248358 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52567.8us
I20260709 12:18:30.190056 27340 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.204581 27296 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:18:30.253820 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:30.260865 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61912.5us
I20260709 12:18:30.270155 12564 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:30.275846 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:30.278155 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.257264 27623 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:30.288616 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:30.285379 27188 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:18:30.295197 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.295756 27216 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:30.296196 27216 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.296615 27216 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:30.297744 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 175463us
I20260709 12:18:30.301900 27600 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.302868 27600 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.303802 27296 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:30.304432 27296 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.304812 27296 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:18:30.306001 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 209264us
I20260709 12:18:30.307102 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:30.302150 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55448.7us
I20260709 12:18:30.310151 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:30.312573 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 183437us
I20260709 12:18:30.320596 12569 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
W20260709 12:18:30.324406 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53873.9us
I20260709 12:18:30.331532 27167 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:30.294068 27607 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:30.329092 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.337956 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:30.339951 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90554.3us
I20260709 12:18:30.294152 27196 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.345296 27579 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:30.345577 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.323928 27431 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:30.346622 12569 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:30.353693 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:30.354624 27478 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.353811 27579 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:30.356573 27579 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.334379 27279 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:30.361243 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54288.3us
W20260709 12:18:30.358532 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 267363us
I20260709 12:18:30.339932 27448 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.370903 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:30.371394 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61352.8us
I20260709 12:18:30.359344 27431 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.373881 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:30.357901 27167 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:30.391345 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:30.376817 27431 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:30.395130 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:30.382922 27408 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.014s
I20260709 12:18:30.378021 27315 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.031s
I20260709 12:18:30.339951 27623 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:30.399385 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2869) took 1768 ms (client timeout 1549 ms). Trace:
I20260709 12:18:30.407160 12491 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:30.375344 27416 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.022s
W20260709 12:18:30.403668 12572 rpcz_store.cc:269] 0709 12:18:28.521410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.619932 (+ 98522us) service_pool.cc:224] Handling call
0709 12:18:30.268966 (+1649034us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:30.290002 (+ 21036us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584136960}
I20260709 12:18:30.412967 12569 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:30.412704 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:30.394393 27374 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:30.417685 27475 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:30.421597 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:30.409296 27279 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:30.426651 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.428939 12569 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 51
I20260709 12:18:30.411557 27340 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.432937 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.398043 27494 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:30.433825 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60119.6us
I20260709 12:18:30.436733 13254 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:30.435943 12572 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:30.437608 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:30.434702 27475 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:30.439064 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:30.439999 12638 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:30.440150 27475 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 44.22, Last appended by leader: 22, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.442497 27165 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.444041 27242 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.445569 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:30.445773 27630 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:30.447943 12643 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 50 index: 29. (term mismatch)
I20260709 12:18:30.448467 12643 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 28
I20260709 12:18:30.448807 12643 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 49.29
I20260709 12:18:30.449142 12643 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:30.452626 27494 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.454355 27408 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:18:30.455111 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 88155.9us
I20260709 12:18:30.455713 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2123) took 2654 ms. Trace:
I20260709 12:18:30.456228 13259 rpcz_store.cc:276] 0709 12:18:27.800903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.806520 (+  5617us) service_pool.cc:224] Handling call
0709 12:18:27.806866 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.006250 (+1199384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.455081 (+1448831us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54004b73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:30.455451 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2708974464}
I20260709 12:18:30.457983 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.459354 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.431610 27279 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.432042 27427 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:30.480345 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:28.955557 12571 rpcz_store.cc:276] 0709 12:18:27.534327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.942847 (+408520us) service_pool.cc:224] Handling call
0709 12:18:27.943178 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:27.943402 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 31
0709 12:18:27.957495 (+ 14093us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.844202 (+886707us) log.cc:844] Serialized 14 byte log entry
0709 12:18:28.883106 (+ 38904us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:28.883161 (+    55us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:18:28.883204 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:28.883270 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:28.883388 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57597,"prepare.queue_time_us":37523,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":98,"raft.queue_time_us":167838,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":73943,"thread_start_us":880839,"threads_started":2,"wal-append.queue_time_us":771432}
I20260709 12:18:30.467638 27494 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:18:30.485365 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47795.2us
I20260709 12:18:30.478157 27279 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:30.499894 27395 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.550798 27395 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.553126 27395 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.502381 27216 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.505853 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.554383 12571 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 47
I20260709 12:18:30.507196 27374 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:30.558516 27374 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.507786 27630 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:30.562592 27630 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:30.479799 27427 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:30.563374 27427 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:30.563700 27630 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.24, Last appended by leader: 24, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.509958 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:30.566555 12495 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:18:30.510862 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51586us
I20260709 12:18:30.488848 27494 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.567853 27623 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.568150 27494 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:30.561053 27374 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.520190 27661 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:30.567912 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.570402 27661 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:30.570869 27661 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:30.571100 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:30.572209 27661 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.521282 12640 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 52 because replica has last-logged OpId of term: 50 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 32.
I20260709 12:18:30.573843 27373 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:30.574769 27373 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.575318 27373 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:30.521992 27448 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.575368 27216 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.576960 27379 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.537358 27558 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.040s
W20260709 12:18:30.572235 27651 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
W20260709 12:18:30.576217 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4318) took 2295 ms (client timeout 2921 ms). Trace:
W20260709 12:18:30.584764 12492 rpcz_store.cc:269] 0709 12:18:28.280788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.519798 (+239010us) service_pool.cc:224] Handling call
0709 12:18:30.575949 (+2056151us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:30.576145 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1937034496}
W20260709 12:18:30.548455 27656 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:18:30.483904 27340 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:30.553660 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:18:30.561586 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:30.588068 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:30.583966 27661 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:30.533946 27296 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.596252 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:30.600051 12496 raft_consensus.cc:2368] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:30.595160 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.601470 12569 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:30.598315 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:30.608560 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117070us
I20260709 12:18:30.603504 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.607488 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:30.610865 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.612193 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.616446 13256 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:30.618938 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:30.631090 12492 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:30.636449 12491 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:30.595451 27340 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:30.609277 27597 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.524s	user 0.018s	sys 0.000s
I20260709 12:18:30.529167 27454 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:30.673194 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:30.675386 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:30.617998 27478 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:30.677172 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.677764 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:30.678624 12492 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:30.682358 12640 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:30.638113 27181 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:30.632272 27602 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:30.679088 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:30.635615 27448 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.640380 27395 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:30.688022 12569 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:30.683777 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:30.690311 12643 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:18:30.691732 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81153.7us
I20260709 12:18:30.693367 12488 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:30.660403 27340 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:30.703926 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:30.697572 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 276021us
I20260709 12:18:30.706948 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.706079 12496 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:30.709177 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:30.704054 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:30.708103 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6011) took 1057 ms. Trace:
I20260709 12:18:30.714568 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:30.719069 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.694398 27602 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.691670 27478 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:30.671154 27597 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.711480 27501 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:30.742295 27623 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:30.742769 27623 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.743129 27623 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:30.714733 27561 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:30.745568 27561 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:30.746050 27561 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:30.730686 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.739168 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:30.747555 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:30.732755 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:30.705765 27448 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.739427 27188 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:30.740358 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:30.741088 27296 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.742017 27501 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.739389 27454 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.746258 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:30.767342 13255 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 42 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 26.
I20260709 12:18:30.770046 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.723625 27565 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:30.779790 12571 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44.
I20260709 12:18:30.771039 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:30.714181 12565 rpcz_store.cc:276] 0709 12:18:29.650788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.126983 (+476195us) service_pool.cc:224] Handling call
0709 12:18:30.127510 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:30.166966 (+ 39456us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:18:30.434815 (+267849us) log.cc:844] Serialized 14 byte log entry
0709 12:18:30.707646 (+272831us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:30.707724 (+    78us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:30.707800 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.707891 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.708039 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6209,"prepare.queue_time_us":44900,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":17711,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":449816,"threads_started":2}
I20260709 12:18:30.729281 27602 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:30.781786 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5514) took 2152 ms. Trace:
I20260709 12:18:30.797326 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.788127 27279 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.791170 27597 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:30.791359 27560 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:30.793874 27443 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.579s	user 0.009s	sys 0.007s
I20260709 12:18:30.794644 27478 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 45.33, Last appended by leader: 33, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:30.799474 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 123004us
I20260709 12:18:30.799863 12569 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:30.768424 27630 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.032s
W20260709 12:18:30.782038 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58784.3us
I20260709 12:18:30.793385 12493 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:30.801793 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.807950 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:30.819878 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140924us
I20260709 12:18:30.825071 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:30.825325 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.833376 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:30.798216 12639 rpcz_store.cc:276] 0709 12:18:28.616741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.024175 (+407434us) service_pool.cc:224] Handling call
0709 12:18:29.024574 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:29.162087 (+137513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:29.162133 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:29.752687 (+590554us) spinlock_profiling.cc:243] Waited 9.05 ms on lock 0x7b2c00147008. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:29.753383 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:18:29.969481 (+216098us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:30.744927 (+775446us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b1800338c20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:18:30.745251 (+   324us) log.cc:844] Serialized 14 byte log entry
0709 12:18:30.746988 (+  1737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.769355 (+ 22367us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:30.769401 (+    46us) raft_consensus.cc:1685] finished
0709 12:18:30.769416 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:30.769530 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40255,"prepare.queue_time_us":763412,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":122510,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":17778,"spinlock_wait_cycles":153028096,"thread_start_us":315442,"threads_started":2,"wal-append.queue_time_us":235008}
I20260709 12:18:30.839674 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:30.846149 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:30.847038 13259 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:30.847134 27501 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.813874 27296 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.869554 12496 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:30.846149 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:30.852100 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:30.822621 27630 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:18:30.873157 13255 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:30.875236 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:30.826449 27657 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.398s	user 0.003s	sys 0.000s
I20260709 12:18:30.819242 27181 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:30.876542 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:30.893111 12645 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:30.885272 12495 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:30.816277 27443 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.900441 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:30.905918 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.906895 12646 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:30.907572 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.908221 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:30.909379 12639 pending_rounds.cc:85] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 19
I20260709 12:18:30.943173 12639 pending_rounds.cc:107] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 37.20
I20260709 12:18:30.943575 12639 raft_consensus.cc:2889] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:30.909725 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:30.944998 13253 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:30.851263 27565 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:30.906003 27427 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.874993 27669 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:30.964361 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.884135 27520 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:30.878265 27630 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:30.922310 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82732.9us
I20260709 12:18:30.849795 27296 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:30.891151 27428 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:30.917359 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:30.975550 12489 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:30.968225 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:30.989518 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:30.995714 27675 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:30.996006 27443 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.996496 27675 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:30.997237 27184 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:30.997817 27184 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:30.998245 27184 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:30.999020 27675 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:30.999145 27427 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:30.964352 27669 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:30.969399 27520 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:31.016361 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:30.997342 27181 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.29, Last appended by leader: 28, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.019970 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:30.978356 27428 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:31.022454 27377 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:31.023146 27377 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:31.024210 27377 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.029536 27685 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:31.030575 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:31.032646 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:31.035493 27296 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.036456 27296 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:18:31.038426 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:31.039006 27672 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:18:31.039839 27167 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:31.040444 27167 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:31.039117 27670 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:18:30.980674 27373 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:31.044507 27658 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.743s	user 0.002s	sys 0.000s
I20260709 12:18:31.044837 27373 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.045567 27669 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:30.724542 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:31.050308 12565 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:31.052065 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.072232 13253 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
W20260709 12:18:31.017942 27431 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.613s	user 0.009s	sys 0.006s
I20260709 12:18:31.048002 27520 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:31.091601 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70266.5us
I20260709 12:18:31.100711 12489 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:30.974654 27565 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.104635 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:31.073267 27448 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.115710 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:31.116433 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:31.119086 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:31.119541 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.117341 13255 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:18:31.120654 12638 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:31.127944 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:31.133921 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.023172 27428 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.139879 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.116112 27443 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.142799 12495 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:31.147959 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:31.094513 27431 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.153055 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.150031 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.041579 27167 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 44.22, Last appended by leader: 22, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.126662 27279 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.158788 12644 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:31.128299 27684 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:31.125270 27686 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:31.208127 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.155113 27443 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:31.213960 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47317us
I20260709 12:18:31.214527 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.215114 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:31.216511 27431 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:31.216775 13256 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44.
I20260709 12:18:31.218883 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.220940 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.165439 27454 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:31.206935 27686 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.143322 27602 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.227728 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.149361 27623 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.255653 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:31.223474 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.259855 12645 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:31.231837 27602 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.226562 27454 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.253224 27561 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.228928 27686 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:31.137784 27630 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:31.271168 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140474us
I20260709 12:18:31.227499 27565 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.280084 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.245836 27443 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:31.268079 27340 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.008s	sys 0.005s
I20260709 12:18:31.229336 27167 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:31.221035 27685 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:18:31.284664 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136622us
I20260709 12:18:31.296962 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.281430 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:31.278319 27661 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.678s	user 0.015s	sys 0.006s
I20260709 12:18:31.262869 27568 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:31.278015 27431 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20260709 12:18:31.291564 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3218240150001748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29209999999999997s for a response
I20260709 12:18:31.302706 27676 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election won for term 39
W20260709 12:18:31.306041 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5500) took 4256 ms (client timeout 2438 ms). Trace:
I20260709 12:18:31.309777 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:31.293555 27340 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.310132 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.283977 27561 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:31.314429 27494 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.746s	user 0.014s	sys 0.000s
I20260709 12:18:31.314530 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6024) took 1426 ms. Trace:
I20260709 12:18:31.331534 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.331692 12568 rpcz_store.cc:276] 0709 12:18:29.888456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:29.935321 (+ 46865us) service_pool.cc:224] Handling call
0709 12:18:31.314294 (+1378973us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.314460 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381991168}
I20260709 12:18:31.316885 27184 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.319531 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.320137 12493 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.29-42.29]
I20260709 12:18:31.321733 12496 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:31.322750 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.323841 27630 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:18:31.305873 27682 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:18:31.326370 12642 rpcz_store.cc:269] 0709 12:18:27.049324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.323709 (+274385us) service_pool.cc:224] Handling call
0709 12:18:31.305706 (+3981997us) spinlock_profiling.cc:243] Waited 2.83 s on lock 0x7b54002c6854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.305916 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6516648832}
I20260709 12:18:31.345351 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.346896 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.347975 12645 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:31.331120 27216 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
W20260709 12:18:31.354737 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.357074 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.357942 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.358521 12490 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:31.359946 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.361070 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:31.361202 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:31.360661 12645 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:31.336995 27676 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:31.369372 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:31.371963 12495 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:31.343330 27568 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.379431 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:31.391651 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.393599 12638 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:31.402511 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:31.394430 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.394623 12496 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:31.395107 27568 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.398157 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.407908 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 5977) took 4223 ms. Trace:
I20260709 12:18:31.407708 27669 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.409003 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.414865 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:31.392853 12643 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:31.398846 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:31.425074 12489 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:31.422741 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:31.400022 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:31.356863 27216 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.405009 27279 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.423112 12641 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:31.336740 27494 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.397748 12497 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 47 index: 29. (index mismatch)
I20260709 12:18:31.402036 12495 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:31.423653 12563 rpcz_store.cc:276] 0709 12:18:27.184620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:27.218146 (+ 33526us) service_pool.cc:224] Handling call
0709 12:18:27.239536 (+ 21390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:29.522451 (+2282915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:31.407606 (+1885155us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:31.407851 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5233216256}
I20260709 12:18:31.429694 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:31.438304 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.453199 12645 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:31.398522 27623 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:31.440961 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:31.442258 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:31.440675 27568 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:31.436621 27216 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:31.485674 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:31.489992 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 60478.4us
I20260709 12:18:31.491017 12489 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:18:31.493937 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.495913 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.497655 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.497850 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.503307 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.504092 12645 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:31.439226 27661 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:31.507573 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.508569 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:31.510962 12496 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:31.492358 27229 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:31.510370 26496 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:31.502573 27431 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 41.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.011s
W20260709 12:18:31.513269 27689 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:18:31.536254 12644 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 37 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:18:31.518389 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.519214 27494 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:31.537034 12646 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:31.522681 26496 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:31.526530 12496 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:31.521706 27676 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.519265 27340 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:31.532660 27520 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.534431 27669 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:31.500726 27623 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:31.537251 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135233us
I20260709 12:18:31.519917 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.551824 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.481194 27377 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 48.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.059s
I20260709 12:18:31.479703 27279 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:31.587042 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103943us
I20260709 12:18:31.606041 26496 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 49.24, Last appended by leader: 24, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.536747 27568 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.610868 27568 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.615240 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.615777 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:31.622411 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:31.622920 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.624765 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:31.625892 27373 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:31.626538 27373 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:31.627815 27373 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.621717 27568 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:31.598050 27229 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:31.624377 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.620544 27279 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.639565 12494 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:31.639787 27520 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.640295 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.640946 12536 leader_election.cc:400] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:31.618671 27184 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:31.641103 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.645525 12645 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:31.650586 12497 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 51 because replica has last-logged OpId of term: 50 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 25.
I20260709 12:18:31.650794 12488 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:31.648762 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.653147 12639 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:31.650288 12536 leader_election.cc:403] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:31.654460 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:31.655354 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.655920 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:31.658016 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.655953 12638 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:31.658027 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.651418 27229 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.657382 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.653553 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:31.665591 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.667587 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.669836 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:31.670817 12494 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46.
W20260709 12:18:31.671104 27702 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:18:31.672735 27184 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:31.672746 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:31.674898 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:31.678498 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:31.681782 27520 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 46. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:18:31.683401 27600 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.683310 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:31.686115 12563 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:31.687048 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46372.6us
I20260709 12:18:31.688347 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:31.690670 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5718) took 1779 ms. Trace:
I20260709 12:18:31.691160 12572 rpcz_store.cc:276] 0709 12:18:29.939281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.454109 (+514828us) service_pool.cc:224] Handling call
0709 12:18:30.798018 (+343909us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:30.810654 (+ 12636us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:30.810710 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:30.820022 (+  9312us) log.cc:844] Serialized 26 byte log entry
0709 12:18:31.415922 (+595900us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b18004afe00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:18:31.524294 (+108372us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:31.524336 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:31.524408 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:31.688877 (+164469us) raft_consensus.cc:1685] finished
0709 12:18:31.688898 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:31.690484 (+  1586us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b54001b30d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.690608 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21324,"raft.queue_time_us":5119,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":2139,"spinlock_wait_cycles":799422464,"thread_start_us":108166,"threads_started":1,"wal-append.queue_time_us":816829}
I20260709 12:18:31.695860 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:31.695966 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.624293 27623 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.696553 12572 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:31.698654 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:31.710453 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:31.706317 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:31.713730 27184 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:31.714419 27184 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:31.714846 27184 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:31.716492 27315 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:31.721227 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:31.718001 27504 raft_consensus.cc:2764] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election decision vote started in defunct term 39: won
I20260709 12:18:31.723078 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:31.651598 27568 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.699858 27494 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:18:31.700189 27662 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:31.731103 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:31.744172 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:31.741132 12645 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:31.747104 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:31.710844 27558 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:31.754403 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:31.755710 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:31.755224 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
W20260709 12:18:31.763526 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79665.8us
W20260709 12:18:31.763885 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.765838 12645 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:31.770578 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:31.729722 27188 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:31.774039 12645 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:31.769944 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:31.777997 27572 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:31.779596 12572 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:31.760281 27600 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.782105 27600 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.782523 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.782811 27623 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.784577 27600 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.765760 27504 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.020s
I20260709 12:18:31.785925 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.786844 13254 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:31.787065 27504 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:31.787523 27504 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:31.788084 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:31.788502 27504 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.789209 27572 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:31.789732 27572 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:31.790113 27572 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:31.756420 27558 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:31.760125 27377 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:31.759848 27494 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:31.728678 27686 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.757704 27662 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.802700 12495 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: [42.29-42.29]
I20260709 12:18:31.765501 27478 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:18:31.731505 27660 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.044s
I20260709 12:18:31.807200 27662 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:31.801976 27504 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:31.809950 27121 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:31.811259 27568 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:31.815713 27184 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:31.804291 27494 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:31.818104 27229 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:31.818439 27454 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:31.810002 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:31.795409 27431 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:31.796830 27188 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:31.820600 27600 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:31.822394 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:31.824916 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5725) took 1333 ms (client timeout 1688 ms). Trace:
I20260709 12:18:31.830238 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:31.841925 27558 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:31.850559 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.836655 27714 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.109s
I20260709 12:18:31.861606 12493 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 52: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:31.861429 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5721) took 1802 ms. Trace:
I20260709 12:18:31.867458 12488 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
W20260709 12:18:31.870827 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101005us
I20260709 12:18:31.873687 27494 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 47.21, Last appended by leader: 21, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:31.861918 27660 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:31.916081 12639 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:31.981601 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2148) took 1017 ms. Trace:
I20260709 12:18:31.975678 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:31.883302 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.980718 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:31.884039 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:31.978982 13256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2134) took 1279 ms (client timeout 1688 ms). Trace:
W20260709 12:18:31.943441 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5534) took 1321 ms (client timeout 1719 ms). Trace:
W20260709 12:18:31.987074 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:31.897383 12565 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 47 index: 29. (index mismatch)
W20260709 12:18:31.917477 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1882) took 3756 ms (client timeout 1860 ms). Trace:
I20260709 12:18:32.075424 12645 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: [39.22-39.22]
I20260709 12:18:32.076798 12568 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:32.077831 12644 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:18:32.078572 12643 rpcz_store.cc:269] 0709 12:18:30.621544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.713784 (+ 92240us) service_pool.cc:224] Handling call
0709 12:18:31.943169 (+1229385us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.943323 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333892352}
I20260709 12:18:32.058188 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:32.056722 12571 rpcz_store.cc:269] 0709 12:18:30.491881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.588934 (+ 97053us) service_pool.cc:224] Handling call
0709 12:18:31.824631 (+1235697us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.824835 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":386199424}
I20260709 12:18:32.037385 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2874) took 1049 ms. Trace:
I20260709 12:18:32.083504 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:32.085070 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109449us
I20260709 12:18:32.088408 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:32.091197 12495 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:32.092082 12639 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
W20260709 12:18:31.944948 27674 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.112s	user 0.000s	sys 0.003s
I20260709 12:18:32.096879 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:32.074141 13258 rpcz_store.cc:269] 0709 12:18:28.161208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:28.161480 (+   272us) service_pool.cc:224] Handling call
0709 12:18:31.917020 (+3755540us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54006e46d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.917210 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2359037184}
I20260709 12:18:32.077464 12638 rpcz_store.cc:276] 0709 12:18:30.964377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.723964 (+759587us) service_pool.cc:224] Handling call
0709 12:18:31.729914 (+  5950us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:31.765529 (+ 35615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:31.765568 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:31.765655 (+    87us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:31.765699 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:31.981262 (+215563us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b54002e5254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:31.981326 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:31.981524 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":67791616}
I20260709 12:18:31.976039 27567 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:31.928500 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4336) took 1196 ms. Trace:
I20260709 12:18:32.059608 27600 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:31.940814 27681 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.976s	user 0.004s	sys 0.001s
I20260709 12:18:31.974884 27493 raft_consensus.cc:993] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:18:31.977996 27340 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:32.104949 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:31.933588 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:32.106388 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48348.6us
I20260709 12:18:32.200569 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:31.878002 27184 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.071959 27612 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:32.052579 27275 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:31.974759 27395 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:31.939461 27629 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:31.881052 27686 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.171415 12492 rpcz_store.cc:276] 0709 12:18:30.732429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.769090 (+ 36661us) service_pool.cc:224] Handling call
0709 12:18:31.928241 (+1159151us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b540012b1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.928422 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1214052096}
I20260709 12:18:31.924819 27572 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.035244 12566 rpcz_store.cc:276] 0709 12:18:30.058762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.787254 (+728492us) service_pool.cc:224] Handling call
0709 12:18:31.315144 (+527890us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:31.315559 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:31.315588 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:31.315637 (+    49us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:31.315676 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:31.315737 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:31.830620 (+514883us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54003e0650. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.861293 (+ 30673us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":475,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":1213085056}
I20260709 12:18:32.198717 27431 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:32.094373 27558 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:32.170362 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 423367us
I20260709 12:18:32.191252 12495 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:18:32.194996 12497 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:32.096387 27597 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.201660 27340 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:31.910248 27216 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.271346 27216 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:32.102833 27296 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:32.272249 27296 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:32.273309 27296 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.107411 12563 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 50 index: 29. (index mismatch)
I20260709 12:18:32.203329 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:32.209802 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 277426us
W20260709 12:18:32.208040 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 137) took 24712 ms (client timeout 15000 ms). Trace:
I20260709 12:18:32.174015 12567 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:32.279565 27597 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.191037 27454 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:18:32.215142 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:32.097376 27660 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.217986 27679 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:32.218706 27617 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:32.219519 27275 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.223042 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6031) took 1971 ms. Trace:
I20260709 12:18:32.213207 27395 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:32.210144 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105303us
I20260709 12:18:32.193516 27567 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:32.081044 13256 rpcz_store.cc:269] 0709 12:18:30.699632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.699813 (+   181us) service_pool.cc:224] Handling call
0709 12:18:31.978644 (+1278831us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54004ef7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:31.978815 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141565824}
I20260709 12:18:32.239356 12638 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:32.244513 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4327) took 1688 ms. Trace:
I20260709 12:18:32.260442 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.098295 27676 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:32.266011 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:32.210487 27493 raft_consensus.cc:1081] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:18:32.268198 12642 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 48 index: 28. (index mismatch)
W20260709 12:18:32.284471 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81227.4us
I20260709 12:18:32.323997 12570 rpcz_store.cc:276] 0709 12:18:30.251357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.794216 (+542859us) service_pool.cc:224] Handling call
0709 12:18:30.794530 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:30.794752 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:30.794780 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:30.795307 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:18:30.860582 (+ 65275us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:30.860639 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:30.860725 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:31.939878 (+1079153us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:32.222653 (+282775us) raft_consensus.cc:1685] finished
0709 12:18:32.222685 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:32.222965 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9182976,"thread_start_us":64733,"threads_started":1,"wal-append.queue_time_us":107584}
I20260709 12:18:32.327044 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.228071 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2880) took 1211 ms. Trace:
I20260709 12:18:32.340200 12565 rpcz_store.cc:276] 0709 12:18:31.016407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.269821 (+253414us) service_pool.cc:224] Handling call
0709 12:18:31.285669 (+ 15848us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.095985 (+810316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.227779 (+131794us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:32.227991 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1405860096}
I20260709 12:18:32.327288 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:32.261991 27568 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.329326 12489 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:32.329702 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.343974 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.333573 12490 rpcz_store.cc:276] 0709 12:18:30.556750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.580832 (+ 24082us) service_pool.cc:224] Handling call
0709 12:18:30.581234 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:31.361941 (+780707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.243941 (+882000us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:32.244421 (+   480us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1781547904}
W20260709 12:18:32.282768 27475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.838s	user 0.005s	sys 0.016s
W20260709 12:18:32.336881 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76533.8us
W20260709 12:18:32.345460 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85414.4us
I20260709 12:18:32.346378 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.348809 26496 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.226022 27629 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.355314 12495 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:32.290452 27679 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:32.262122 27229 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.074s
I20260709 12:18:32.212040 27612 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.377734 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 48 index: 28. (index mismatch)
I20260709 12:18:32.360585 26496 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:32.351485 27475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.906s	user 0.006s	sys 0.017s
I20260709 12:18:32.198391 27188 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.372378 12642 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
W20260709 12:18:32.372788 27504 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.564s	user 0.011s	sys 0.002s
I20260709 12:18:32.378829 12644 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:32.379786 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:32.350279 27216 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:32.422313 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76020.8us
I20260709 12:18:32.354816 27560 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:32.426232 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:32.423823 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:32.354622 27676 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:32.378227 27617 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.009s
I20260709 12:18:32.464764 12642 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:32.370041 27568 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.385216 12490 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:32.385605 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:32.417259 27443 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:32.431428 12491 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
W20260709 12:18:32.374238 27642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.009s	sys 0.030s
I20260709 12:18:32.374972 27679 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:18:32.467038 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43598.4us
I20260709 12:18:32.494168 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:32.494594 12565 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 33. Preceding OpId from leader: term: 47 index: 34. (index mismatch)
I20260709 12:18:32.496678 12570 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 51 index: 30. (index mismatch)
I20260709 12:18:32.464246 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:32.490065 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2270) took 1167 ms (client timeout 1511 ms). Trace:
I20260709 12:18:32.491209 12488 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
W20260709 12:18:32.498318 12493 rpcz_store.cc:269] 0709 12:18:31.322726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.343342 (+ 20616us) service_pool.cc:224] Handling call
0709 12:18:32.489841 (+1146499us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b540012c5d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:32.489990 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1107651328}
I20260709 12:18:32.508808 27676 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.429325 27717 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:32.540218 27567 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:32.429023 27560 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:32.541917 27560 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:32.323654 27600 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.546775 27617 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:32.547622 27617 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.490828 27242 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 48.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.063s
I20260709 12:18:32.554734 27242 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:32.555280 27242 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:32.556563 27242 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 48.27, Last appended by leader: 27, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.559434 27275 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.566962 27660 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:32.567114 12571 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 24. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:18:32.569536 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.570565 12571 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:32.575539 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81457.9us
I20260709 12:18:32.578037 26496 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.554981 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:32.582638 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4442) took 1433 ms. Trace:
I20260709 12:18:32.583169 12645 rpcz_store.cc:276] 0709 12:18:31.149512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.779612 (+630100us) service_pool.cc:224] Handling call
0709 12:18:32.075192 (+295580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:32.076469 (+  1277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.582369 (+505900us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54001f72d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:32.582566 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":678911744}
I20260709 12:18:32.541105 27567 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:32.490065 27443 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:32.598635 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.610548 12571 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:32.610903 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.611966 12566 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
W20260709 12:18:32.507689 27642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.009s	sys 0.031s
W20260709 12:18:32.593063 27623 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.021s	sys 0.032s
W20260709 12:18:32.615231 27623 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.021s	sys 0.032s
I20260709 12:18:32.615809 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:32.616338 27714 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:32.616845 12489 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:32.618461 12571 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:32.624776 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:32.083878 12569 rpcz_store.cc:276] 0709 12:18:30.988171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.226407 (+238236us) service_pool.cc:224] Handling call
0709 12:18:31.226649 (+   242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:31.226878 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:31.330996 (+104118us) log.cc:844] Serialized 14 byte log entry
0709 12:18:32.036573 (+705577us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b18004276c0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:18:32.036872 (+   299us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:32.036955 (+    83us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:32.037020 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.037108 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:32.037302 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":248292,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":6825,"raft.queue_time_us":16069,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":18038,"spinlock_wait_cycles":3287296,"thread_start_us":275176,"threads_started":2,"wal-append.queue_time_us":224604}
I20260709 12:18:32.626394 27602 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:18:32.626608 12569 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32896
I20260709 12:18:32.627027 27602 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:32.627337 12569 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:32.627409 26330 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:32.628099 27602 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 50.33, Last appended by leader: 33, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.629961 12646 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: [47.21-49.22]
I20260709 12:18:32.586189 27669 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:32.625435 27685 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:32.382969 27340 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 48.27, Last appended by leader: 27, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.631446 27669 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:32.630769 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4444) took 1354 ms. Trace:
I20260709 12:18:32.652290 12646 rpcz_store.cc:276] 0709 12:18:31.276055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.629199 (+1353144us) service_pool.cc:224] Handling call
0709 12:18:32.629670 (+   471us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:32.630439 (+   769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.630709 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:32.629540 26330 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 25, Time since last communication: 0.006s
W20260709 12:18:32.278082 12386 rpcz_store.cc:269] 0709 12:18:07.458637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:07.460139 (+  1502us) service_pool.cc:224] Handling call
0709 12:18:32.107325 (+24647186us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:22.983917 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:23.022855 (+ 38938us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:23.022918 (+    63us) write_op.cc:435] Acquired schema lock
0709 12:18:23.022940 (+    22us) tablet.cc:662] Decoding operations
0709 12:18:23.024459 (+  1519us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:23.024568 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:23.024598 (+    30us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:18:23.286826 (+262228us) tablet.cc:701] Row locks acquired
0709 12:18:23.286883 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:18:23.287089 (+   206us) write_op.cc:270] Start()
0709 12:18:23.287348 (+   259us) write_op.cc:276] Timestamp: P: 1783599503287002 usec, L: 0
0709 12:18:23.287383 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:18:23.288246 (+   863us) log.cc:844] Serialized 5772 byte log entry
0709 12:18:23.693270 (+405024us) op_driver.cc:464] REPLICATION: finished
0709 12:18:25.269816 (+1576546us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b1800194da0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:25.286561 (+ 16745us) write_op.cc:301] APPLY: starting
0709 12:18:25.299929 (+ 13368us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:26.344436 (+1044507us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:26.348130 (+  3694us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:26.782705 (+434575us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:26.783001 (+   296us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:18:26.783048 (+    47us) write_op.cc:312] APPLY: finished
0709 12:18:26.783382 (+   334us) log.cc:844] Serialized 129 byte log entry
0709 12:18:26.850197 (+ 66815us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:26.856699 (+  6502us) write_op.cc:454] Released schema lock
0709 12:18:26.864867 (+  8168us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:26.878047 (+ 13180us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:26.878095 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":167997,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":243958,"apply.run_cpu_time_us":41492,"apply.run_wall_time_us":1591460,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62591,"num_ops":14,"prepare.queue_time_us":310052,"prepare.run_cpu_time_us":51128,"prepare.run_wall_time_us":509411,"raft.queue_time_us":126338,"raft.run_cpu_time_us":6400,"raft.run_wall_time_us":235559,"replication_time_us":405674,"spinlock_wait_cycles":151850240,"thread_start_us":648617,"threads_started":5,"wal-append.queue_time_us":337124,"wal-append.run_cpu_time_us":10711,"wal-append.run_wall_time_us":2143826}]]}
I20260709 12:18:32.581100 27600 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:32.631330 27714 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:32.648097 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56220.2us
I20260709 12:18:32.649827 12640 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:18:32.579833 27717 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:32.627779 12569 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:32.675361 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4443) took 1507 ms. Trace:
I20260709 12:18:32.675747 12640 rpcz_store.cc:276] 0709 12:18:31.167764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.649109 (+1481345us) service_pool.cc:224] Handling call
0709 12:18:32.649514 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:32.674979 (+ 25465us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.675284 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:32.677022 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.656985 26496 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 48.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:32.590389 27275 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:32.666966 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.632812 27443 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:18:32.649511 27579 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.199s	user 0.051s	sys 0.049s
I20260709 12:18:32.706511 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46.
I20260709 12:18:32.707329 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Cancelling election due to peer responding with higher term
I20260709 12:18:32.663285 27690 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:32.672245 27242 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:32.692648 27597 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.691900 27275 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:32.665760 27454 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.062s
I20260709 12:18:32.713955 27669 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.743995 27275 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.744140 27717 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.27, Last appended by leader: 26, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:32.697726 27579 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.247s	user 0.052s	sys 0.049s
I20260709 12:18:32.745798 27275 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:32.630005 27373 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.070s
I20260709 12:18:32.699472 27705 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:18:32.692816 27660 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:32.747762 27714 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:32.709367 27395 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:32.754468 27579 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election lost for term 45. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46.
I20260709 12:18:32.716763 27379 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:32.716536 27690 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:32.803941 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.709924 27520 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:32.755188 27629 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.783174 27568 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:32.838284 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 494344us
W20260709 12:18:32.800151 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46035.7us
W20260709 12:18:32.813043 27577 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.032s	sys 0.031s
I20260709 12:18:32.813917 27379 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:32.792062 27660 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:32.853924 27577 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.032s	sys 0.031s
I20260709 12:18:32.864212 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:32.845511 27612 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:32.861304 27379 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:32.839890 27395 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:32.885413 12609 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44.
I20260709 12:18:32.887847 12609 leader_election.cc:403] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Cancelling election due to peer responding with higher term
I20260709 12:18:32.901957 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:32.903672 27612 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 43. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44.
I20260709 12:18:32.855273 27520 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:32.886068 27579 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:32.930377 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.018s	sys 0.026s
W20260709 12:18:32.931895 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.019s	sys 0.027s
I20260709 12:18:32.904861 27620 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:18:32.940506 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 597671us
W20260709 12:18:32.929306 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.948309 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.948561 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:32.913408 27520 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:32.938490 12496 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.040s	sys 0.041s
W20260709 12:18:32.960618 12496 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.040s	sys 0.042s
W20260709 12:18:32.938392 13219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.007s	sys 0.039s
W20260709 12:18:32.962297 13219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.007s	sys 0.039s
W20260709 12:18:32.962443 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:32.954496 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2150) took 1542 ms. Trace:
I20260709 12:18:32.956035 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:32.963641 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:32.963416 12643 rpcz_store.cc:276] 0709 12:18:31.412427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.080377 (+667950us) service_pool.cc:224] Handling call
0709 12:18:32.080718 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.385976 (+305258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:32.386022 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:32.386080 (+    58us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:32.386124 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.954077 (+567953us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:32.954158 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:32.954396 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701510016}
I20260709 12:18:32.866618 27690 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.25, Last appended by leader: 25, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:32.978015 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 141932us
I20260709 12:18:32.988160 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
W20260709 12:18:33.011515 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.014703 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:33.015618 12567 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:32.960582 27705 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:33.023319 13256 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:33.026567 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:33.029187 12494 raft_consensus.cc:2368] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:33.029429 12490 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:33.033385 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:33.033866 27373 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.035517 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:33.034240 13254 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:33.035035 27373 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:33.019651 27229 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.005s	sys 0.009s
I20260709 12:18:32.934875 27579 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.053332 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:33.059540 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 394161us
I20260709 12:18:33.069135 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.070215 12494 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:18:33.075421 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3922396279997429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22440000000000004s for a response
W20260709 12:18:33.050717 27229 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.005s	sys 0.009s
I20260709 12:18:33.106616 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.107223 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.008140 12569 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.113327 12490 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:33.112460 12494 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:33.120888 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.123270 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4352) took 1484 ms. Trace:
I20260709 12:18:33.123708 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.123816 12497 rpcz_store.cc:276] 0709 12:18:31.638497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:31.669038 (+ 30541us) service_pool.cc:224] Handling call
0709 12:18:31.669370 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.281503 (+612133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.122961 (+841458us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b540020dfd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:33.123190 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1203254016}
I20260709 12:18:33.126338 27660 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:33.129161 27728 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:18:33.110719 27431 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.847s	user 0.011s	sys 0.002s
I20260709 12:18:33.125432 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.140960 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:33.136482 12495 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:33.114128 27597 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.111218 27690 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:33.146653 12569 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:33.149046 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:33.176380 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:33.176324 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.177438 12495 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:33.178580 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:33.179227 13259 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:33.180757 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:33.185276 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:33.139956 27669 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.179082 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:33.189802 13256 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:33.192721 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:33.200631 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:33.149359 27431 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:33.150341 27520 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:33.147387 27504 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:33.209455 13258 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 51 index: 30. (index mismatch)
I20260709 12:18:33.211277 12497 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:33.176219 27612 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:33.209729 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:33.227270 13257 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: []
W20260709 12:18:33.202979 27725 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.296s	user 0.004s	sys 0.001s
I20260709 12:18:33.138754 27676 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.065s
I20260709 12:18:33.234205 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5572) took 1290 ms. Trace:
I20260709 12:18:33.259603 12644 rpcz_store.cc:276] 0709 12:18:31.943371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.430865 (+487494us) service_pool.cc:224] Handling call
0709 12:18:32.476973 (+ 46108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.477190 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:33.011829 (+534639us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.233773 (+221944us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:33.233847 (+    74us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:33.233917 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.233993 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:33.234135 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29335,"prepare.queue_time_us":557706,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"thread_start_us":641121,"threads_started":2}
W20260709 12:18:33.217015 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 228528us
I20260709 12:18:33.234278 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:33.240810 27577 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.209309 27607 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:33.245257 12497 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:18:33.196714 27427 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:33.265033 27427 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:33.266250 27427 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:33.246822 27560 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.704s	user 0.008s	sys 0.007s
I20260709 12:18:33.266968 12497 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: [49.22-49.22]
I20260709 12:18:33.267546 27560 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.247743 27373 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:33.151298 27629 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.058s
I20260709 12:18:33.248975 12563 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:33.265311 27577 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.270321 12563 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:33.271149 12563 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.27, Last appended by leader: 27, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.271832 27577 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:33.272032 12563 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:33.188632 27600 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:33.274753 27427 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:33.276845 27560 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:33.205686 27660 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:33.310056 27756 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:33.310786 27756 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.313169 27756 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:33.315158 27720 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:18:33.175779 27597 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.316018 27579 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:33.317675 27597 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:33.209987 27737 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:33.318962 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84788.3us
I20260709 12:18:33.318790 27737 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.319697 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:33.247828 27454 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.324069 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.249929 27642 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:33.230058 27612 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:33.337949 12495 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
W20260709 12:18:33.339088 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 138580us
I20260709 12:18:33.260071 27731 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:33.342985 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.343796 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:33.329732 27242 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.004s
I20260709 12:18:33.205534 27669 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.350666 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:33.325722 27660 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:33.365330 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.332659 27629 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:18:33.368045 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.230335 27504 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:33.369807 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 165110us
I20260709 12:18:33.375784 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:33.370591 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5564) took 1728 ms. Trace:
I20260709 12:18:33.379441 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:33.374985 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 194356us
I20260709 12:18:33.379132 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:33.341256 27676 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:33.376464 12563 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:33.379709 12638 rpcz_store.cc:276] 0709 12:18:31.642589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.334112 (+691523us) service_pool.cc:224] Handling call
0709 12:18:32.334551 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.334882 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:32.832269 (+497387us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.370056 (+537787us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:33.370153 (+    97us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:33.370232 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.370325 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:33.370513 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":630908,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":193067,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"thread_start_us":1031556,"threads_started":3}
I20260709 12:18:33.315177 27730 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.047s
I20260709 12:18:33.386145 12644 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 38 index: 27. (index mismatch)
I20260709 12:18:33.347637 27612 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:33.341562 27711 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:33.367595 27443 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.659s	user 0.011s	sys 0.004s
I20260709 12:18:33.400074 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.391939 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:33.392910 27560 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:33.436666 13256 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:33.612002 13259 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:33.393713 27629 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:33.340423 27379 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.471182 27730 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.472162 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:33.376745 27660 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:33.334205 27454 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:33.435523 27740 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:18:33.478544 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5550) took 2049 ms. Trace:
W20260709 12:18:33.441417 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 62396.8us
I20260709 12:18:33.376907 27600 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:33.494576 12572 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Stepping down as leader of term 38
W20260709 12:18:33.502930 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 111118us
W20260709 12:18:33.659518 27558 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.354s	user 0.018s	sys 0.000s
W20260709 12:18:33.606319 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.611337 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.402551 27642 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:33.391796 27676 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:33.615907 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:33.617324 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:33.363538 27731 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:33.701845 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:33.622305 27717 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:33.611949 27560 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:33.731294 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:33.495311 27737 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:33.711683 27679 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.144s	user 0.008s	sys 0.008s
W20260709 12:18:33.741266 13254 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.702s	user 0.011s	sys 0.002s
I20260709 12:18:33.729614 12572 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:33.441430 27443 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.775805 27572 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:33.777064 27443 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:33.739066 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 379123us
I20260709 12:18:33.733989 12639 rpcz_store.cc:276] 0709 12:18:31.429127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.141588 (+712461us) service_pool.cc:224] Handling call
0709 12:18:32.198912 (+ 57324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.199250 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:18:32.671041 (+471791us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.422132 (+751091us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:33.477737 (+ 55605us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:33.477824 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.478355 (+   531us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:33.478489 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":686640,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":189326,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":150,"thread_start_us":1160624,"threads_started":3}
I20260709 12:18:33.742286 13262 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:33.765321 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:33.770711 13254 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:33.738194 12569 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:33.778896 27739 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 12:18:33.780109 27751 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.305s	user 0.000s	sys 0.000s
W20260709 12:18:33.781209 27750 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 12:18:33.769452 12572 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.742347 27676 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:33.724037 27766 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:33.474303 27711 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:33.797906 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70257.2us
W20260709 12:18:33.746364 27745 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
W20260709 12:18:33.840008 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45312.5us
I20260709 12:18:33.725926 27764 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:33.816112 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51165.8us
I20260709 12:18:33.765544 27558 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:33.776984 27572 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:33.840327 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5917) took 1316 ms. Trace:
I20260709 12:18:33.842008 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:33.773480 27560 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:33.770203 27600 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.858969 12572 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:33.718220 27730 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.888996 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:33.894277 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:33.897193 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55285.5us
I20260709 12:18:33.771411 27679 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.884477 27642 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:33.895665 27558 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:33.924544 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:33.933221 12492 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:33.896620 12488 rpcz_store.cc:276] 0709 12:18:32.523836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.575020 (+ 51184us) service_pool.cc:224] Handling call
0709 12:18:32.603483 (+ 28463us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:32.623275 (+ 19792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:32.623310 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:32.624032 (+   722us) log.cc:844] Serialized 52 byte log entry
0709 12:18:33.050365 (+426333us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:33.111366 (+ 61001us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.111797 (+   431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:33.111895 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:33.798261 (+686366us) raft_consensus.cc:1685] finished
0709 12:18:33.798289 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:33.840222 (+ 41933us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40279,"prepare.queue_time_us":63577,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":19842,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":6246,"thread_start_us":425857,"threads_started":1,"wal-append.queue_time_us":625406}
W20260709 12:18:33.953392 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:33.957288 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72283.1us
I20260709 12:18:33.921067 27684 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.916738 27642 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:33.866456 27629 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:33.924554 27504 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:33.928277 27572 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:18:33.977743 27747 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
W20260709 12:18:33.978708 27757 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:18:33.904649 27764 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:33.982084 27431 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:33.982841 27773 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:33.983738 27761 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:33.984694 27427 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.709s	user 0.011s	sys 0.003s
I20260709 12:18:33.985018 27568 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:33.986157 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:33.911015 27600 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.033479 27427 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:34.035612 27600 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:34.036854 27723 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.345s	user 0.000s	sys 0.010s
I20260709 12:18:33.953449 27669 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.040686 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.003789 27684 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.041980 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:33.940395 27762 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:33.981447 27730 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:33.932036 27765 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:33.988550 27642 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:33.985004 27686 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.034s
I20260709 12:18:34.002120 27504 raft_consensus.cc:2764] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election decision vote started in defunct term 46: won
I20260709 12:18:34.009538 27558 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.020123 27764 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:34.020888 27690 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.822s	user 0.005s	sys 0.007s
I20260709 12:18:34.020838 12495 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:34.010208 27379 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.021733 27568 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:34.022454 27761 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:34.026351 27748 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:18:34.026824 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.079803 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:34.030182 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:34.030196 27742 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
W20260709 12:18:34.031742 27736 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:18:34.031858 27766 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:34.033366 27731 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.036721 27454 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.042819 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:34.042972 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:33.997936 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74016.2us
I20260709 12:18:34.051949 12572 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 24. Preceding OpId from leader: term: 39 index: 24. (term mismatch)
I20260709 12:18:34.093233 12572 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 23
I20260709 12:18:34.093604 12572 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 38.24
I20260709 12:18:34.093469 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.094072 12572 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:18:34.051409 27721 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.141s	user 0.004s	sys 0.009s
W20260709 12:18:34.042260 27724 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.350s	user 0.008s	sys 0.000s
I20260709 12:18:34.062639 26496 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:34.011564 27431 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.084028 27558 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:34.064783 27767 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 46.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.013s
I20260709 12:18:34.106388 27767 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:34.106928 27767 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:34.107511 27340 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:34.107843 27767 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 46.28, Last appended by leader: 28, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.099984 27504 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:34.110212 27504 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:34.071996 27684 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.111037 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5889) took 3269 ms. Trace:
I20260709 12:18:34.111682 12491 rpcz_store.cc:276] 0709 12:18:30.841361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.846370 (+  5009us) service_pool.cc:224] Handling call
0709 12:18:30.846699 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.480559 (+1633860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.110631 (+1630072us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:34.110941 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3644011264}
I20260709 12:18:34.109397 27767 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:34.118379 27454 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:34.119102 27454 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.043919 27679 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.127874 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:34.133426 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:34.134328 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.084795 27560 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.085024 27568 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:34.089964 27520 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.874s	user 0.006s	sys 0.010s
I20260709 12:18:34.042927 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:34.097530 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:34.099030 26496 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:34.069558 27765 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:34.056298 27746 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.059746 27762 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:34.150125 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:34.153031 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55611.5us
I20260709 12:18:34.154543 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
I20260709 12:18:34.156734 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.160181 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.161047 12568 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:34.162600 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:34.162534 26496 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.26, Last appended by leader: 26, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.166227 27765 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.26, Last appended by leader: 26, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.152472 27731 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.167747 27731 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.156697 27746 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.171696 27431 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.171327 27454 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.172297 27746 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.156947 27379 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:34.173858 12570 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:18:34.174396 27679 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.175137 27679 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.175849 27560 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.181064 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:34.191576 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.192678 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.152220 27520 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:34.200724 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2151) took 2788 ms. Trace:
I20260709 12:18:34.213217 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.177529 27679 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.205971 12641 rpcz_store.cc:276] 0709 12:18:31.412739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.080906 (+668167us) service_pool.cc:224] Handling call
0709 12:18:32.175431 (+ 94525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.351567 (+176136us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:18:32.841445 (+489878us) log.cc:844] Serialized 14 byte log entry
0709 12:18:33.239832 (+398387us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:33.239934 (+   102us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:33.240017 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.200420 (+960403us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:34.200485 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.200649 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39573,"prepare.queue_time_us":683008,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":100444,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"spinlock_wait_cycles":404371840,"thread_start_us":843972,"threads_started":2,"wal-append.queue_time_us":1091999}
I20260709 12:18:34.194445 27779 raft_consensus.cc:2764] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election decision vote started in defunct term 44: won
I20260709 12:18:34.192216 27690 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.165129 27642 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.229882 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:34.229974 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 180802us
W20260709 12:18:34.233207 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 345216us
I20260709 12:18:34.200258 27669 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:18:34.186776 27579 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:18:34.194718 27568 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:34.111109 27504 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 42.28, Last appended by leader: 28, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:34.224148 27602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.306s	user 0.039s	sys 0.065s
I20260709 12:18:34.194905 27379 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.040s
I20260709 12:18:34.256479 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5908) took 2159 ms. Trace:
I20260709 12:18:34.252065 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.243898 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.243947 27520 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:34.243163 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.269062 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:34.260700 12645 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
W20260709 12:18:34.269824 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 283720us
I20260709 12:18:34.216385 27340 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:34.261550 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.303488 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:34.304360 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.208302 27600 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.039s
I20260709 12:18:34.260054 12496 rpcz_store.cc:276] 0709 12:18:32.097371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.328099 (+230728us) service_pool.cc:224] Handling call
0709 12:18:32.328504 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:32.328766 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:32.961375 (+632609us) log.cc:844] Serialized 14 byte log entry
0709 12:18:34.044458 (+1083083us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b18001c1fe0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:18:34.044781 (+   323us) log.cc:844] Serialized 14 byte log entry
0709 12:18:34.255934 (+211153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:34.256002 (+    68us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:34.256065 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.256171 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.256384 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60819,"prepare.queue_time_us":1401717,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":10717,"raft.queue_time_us":50678,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":10842,"spinlock_wait_cycles":30160512,"thread_start_us":973749,"threads_started":3,"wal-append.queue_time_us":473471}
I20260709 12:18:34.212299 27623 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.276759 12493 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:34.270999 27769 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:34.278290 13257 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 33. Preceding OpId from leader: term: 47 index: 34. (index mismatch)
I20260709 12:18:34.279513 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.281466 27684 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:34.287811 13255 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 32. Preceding OpId from leader: term: 50 index: 33. (index mismatch)
I20260709 12:18:34.287747 27188 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:34.275159 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.236720 27690 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.275202 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.307878 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:34.308189 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.308935 12640 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
W20260709 12:18:34.259598 27602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.350s	user 0.044s	sys 0.070s
I20260709 12:18:34.215770 27597 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 38.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:18:34.331485 12637 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:34.332571 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:34.272217 27778 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.000s	sys 0.002s
I20260709 12:18:34.292367 27734 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.346855 12638 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:34.305961 27660 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.013s	sys 0.003s
I20260709 12:18:34.323621 13260 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:34.339370 13254 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 42 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 26.
I20260709 12:18:34.276405 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.340322 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:34.238840 27714 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:18:34.300590 27560 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:34.343997 12644 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:34.335736 27779 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:34.342387 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:34.315209 27568 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:34.280531 27764 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.070s
I20260709 12:18:34.353127 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:34.315464 27761 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.346349 27188 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.344964 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.359279 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6053) took 2013 ms. Trace:
I20260709 12:18:34.335502 27776 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:34.346388 27565 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 48.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.050s
I20260709 12:18:34.362573 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:34.291438 27679 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.060s
I20260709 12:18:34.394649 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.351377 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.427147 12639 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
W20260709 12:18:34.425985 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63534.7us
W20260709 12:18:34.397220 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44053.3us
I20260709 12:18:34.406584 12563 rpcz_store.cc:276] 0709 12:18:32.345900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.418309 (+ 72409us) service_pool.cc:224] Handling call
0709 12:18:32.418702 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:33.386408 (+967706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.358965 (+972557us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:34.359207 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1834291584}
I20260709 12:18:34.286324 27494 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.068s
I20260709 12:18:34.419526 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.360105 27731 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.433187 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.433703 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:34.354209 27579 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:34.426080 26496 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:34.393062 27763 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:34.368610 27779 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:34.334481 27623 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:34.459760 12597 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (15 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:18:34.339190 27676 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.393556 27568 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:34.436407 27600 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
W20260709 12:18:34.449760 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 205857us
I20260709 12:18:34.404947 27617 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.451555 12644 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
W20260709 12:18:34.461603 27733 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.251s	user 0.000s	sys 0.012s
I20260709 12:18:34.436537 27755 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:34.457973 27779 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 43.22, Last appended by leader: 22, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.355387 27660 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.338840 27769 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.461512 27623 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.354837 27734 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.552403 27769 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.552613 27734 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.405102 27602 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.059s
W20260709 12:18:34.554080 27787 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:18:34.487953 12644 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 42 index: 22. (index mismatch)
I20260709 12:18:34.345057 27684 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.558208 27755 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.560071 27684 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.488330 27642 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.562225 27642 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:34.563309 27642 consensus_queue.cc:1243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 51.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:34.566587 27660 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.569948 12569 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:34.488554 27783 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:34.489822 27754 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.780s	user 0.002s	sys 0.002s
I20260709 12:18:34.582123 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5914) took 2224 ms. Trace:
I20260709 12:18:34.582533 12492 rpcz_store.cc:276] 0709 12:18:32.357500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.420913 (+ 63413us) service_pool.cc:224] Handling call
0709 12:18:32.421189 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:33.947425 (+1526236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.581857 (+634432us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540012c0d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:34.582061 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3467640064}
I20260709 12:18:34.584494 27669 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:18:34.587178 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.588022 12566 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:34.588830 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.590277 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2156) took 1262 ms. Trace:
I20260709 12:18:34.590687 12642 rpcz_store.cc:276] 0709 12:18:33.329647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:33.334906 (+  5259us) service_pool.cc:224] Handling call
0709 12:18:33.335226 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:33.840548 (+505322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:33.840587 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:33.840637 (+    50us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:33.840667 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.590008 (+749341us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:34.590058 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.590189 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1104064384}
I20260709 12:18:34.592576 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5578) took 1933 ms. Trace:
I20260709 12:18:34.592989 12646 rpcz_store.cc:276] 0709 12:18:32.659260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.030789 (+1371529us) service_pool.cc:224] Handling call
0709 12:18:34.031238 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:34.031518 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:34.031541 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:34.032051 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:18:34.196363 (+164312us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:34.196406 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.196488 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.592333 (+395845us) raft_consensus.cc:1685] finished
0709 12:18:34.592357 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:34.592511 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13568,"thread_start_us":163940,"threads_started":1,"wal-append.queue_time_us":416999}
I20260709 12:18:34.595009 12646 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:34.594676 12644 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 40 index: 22. (index mismatch)
I20260709 12:18:34.600212 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:34.589561 12566 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:34.601058 12568 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:34.601413 13256 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:34.601920 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.602545 12566 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:34.490871 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:34.491425 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.492770 12572 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:34.493386 27763 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:34.603636 13254 raft_consensus.cc:2368] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44.
I20260709 12:18:34.492914 12536 leader_election.cc:400] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:34.603612 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.482977 27753 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.262s
I20260709 12:18:34.452301 27340 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.075s
I20260709 12:18:34.483031 27568 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:34.490862 27188 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.527542 27690 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.483618 27762 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.484014 27600 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:34.450531 27579 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:34.534205 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.534274 27617 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.536559 27676 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:34.675217 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.513409 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.676940 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:34.542444 12609 connection.cc:256] RPC call timeout handler was delayed by -1.010332070000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
W20260709 12:18:34.679308 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.446663 26496 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:34.568414 27760 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.504s	user 0.006s	sys 0.000s
I20260709 12:18:34.681977 26496 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:34.574059 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.682821 12496 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
W20260709 12:18:34.577093 27612 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.126s	user 0.011s	sys 0.004s
I20260709 12:18:34.684005 27762 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:34.683911 27612 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.684580 27762 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:34.579106 13262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2142) took 1555 ms (client timeout 2046 ms). Trace:
W20260709 12:18:34.685555 13262 rpcz_store.cc:269] 0709 12:18:33.023938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:33.024170 (+   232us) service_pool.cc:224] Handling call
0709 12:18:34.578810 (+1554640us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:34.578968 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1623524608}
I20260709 12:18:34.685995 27275 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:34.686592 27275 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.687132 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.580575 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.687806 27731 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.603615 12572 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:34.688400 27731 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.688902 12572 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.22, Last appended by leader: 22, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.514982 27782 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:34.689944 27782 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.690402 27676 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.690948 27676 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.690656 27600 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.692399 27782 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.606300 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.692718 27600 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:34.677333 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:34.693507 12566 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 24.
I20260709 12:18:34.694777 12566 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:34.694464 27579 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 50.25, Last appended by leader: 25, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.696237 12566 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:18:34.696431 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.697245 12571 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:34.608418 12611 leader_election.cc:400] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44.
I20260709 12:18:34.694821 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:34.698137 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 89975.3us
I20260709 12:18:34.561254 27755 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.623588 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.627596 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.577111 27779 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:34.702244 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 209489us
I20260709 12:18:34.628255 27734 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.706766 27734 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.707904 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:34.687824 13262 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:34.707986 12572 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:34.665863 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 02f3cd3c4cc54eb9adb6f7c50cea457e: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:18:34.668313 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.670722 27728 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.537309 12641 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:34.699024 27730 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:34.696638 27579 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:34.688138 12567 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:34.718490 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:34.697773 12640 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:34.721818 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:34.724272 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:34.726810 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 296290us
I20260709 12:18:34.728653 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:18:34.725221 12567 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:34.712400 27617 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:34.731762 27770 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.876s	user 0.002s	sys 0.000s
I20260709 12:18:34.732527 27617 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.734201 27774 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:34.734541 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.735234 27669 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:18:34.736187 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.741272 27660 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:34.741137 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.742043 27660 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.742699 13253 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:34.743719 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.744563 13253 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 42 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 28.
I20260709 12:18:34.748267 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:34.752344 27782 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.753005 27782 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.667513 27560 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:34.755043 27560 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:34.755489 27560 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:34.755882 27560 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:34.734776 27774 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:34.758103 27734 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.758741 27774 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.760040 27774 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:34.760746 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4356) took 1734 ms. Trace:
I20260709 12:18:34.736142 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.764366 12644 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 20. Preceding OpId from leader: term: 50 index: 22. (index mismatch)
I20260709 12:18:34.765614 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:34.766898 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.744383 27660 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.767613 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.761168 12493 rpcz_store.cc:276] 0709 12:18:33.026258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:33.026964 (+   706us) service_pool.cc:224] Handling call
0709 12:18:33.035410 (+  8446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:34.315794 (+1280384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:34.760472 (+444678us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540033e854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:34.760678 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2854291328}
I20260709 12:18:34.769770 12567 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 38 index: 27. (index mismatch)
I20260709 12:18:34.769834 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.770939 13253 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:18:34.726505 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.774796 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.776916 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:34.777673 12494 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:34.778014 12494 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:34.779865 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:34.780767 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:18:34.781610 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.783087 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:34.773870 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:34.784330 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:34.780788 12492 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:34.789883 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:34.780205 12494 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.792627 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.816181 12494 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:34.816987 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.793383 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.794392 27567 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:34.795395 27431 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:34.797055 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.820794 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.821906 12488 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:34.799661 27782 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.800032 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.826545 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:34.829202 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.830331 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.800473 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:34.801209 27660 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:34.782232 27791 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:18:34.803588 27597 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 40 current term: 40 is higher.
I20260709 12:18:34.604339 27783 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.630347 27558 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.138s
I20260709 12:18:34.780313 27340 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.810531 13262 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 43 index: 27. (index mismatch)
I20260709 12:18:34.810316 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.879487 13259 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:34.811677 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:34.784340 26496 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.817716 12572 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:34.717382 27761 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:34.830641 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:34.834067 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.800231 27711 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.835178 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:34.885128 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73582.9us
I20260709 12:18:34.768581 27728 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.886617 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:34.887428 12644 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:34.888777 27216 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:34.889231 27216 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:34.843672 27791 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:34.805548 27788 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:18:34.838621 12643 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:34.839481 12494 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:34.840708 12570 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 50. Current term is 51. Ops: [50.29-50.29]
I20260709 12:18:34.729244 27730 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:34.899461 12563 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 22. Preceding OpId from leader: term: 42 index: 22. (term mismatch)
I20260709 12:18:34.900028 12563 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:18:34.900334 12563 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.22
I20260709 12:18:34.900651 12563 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:34.901660 12489 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
W20260709 12:18:34.868374 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.844106 27660 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:34.883731 12491 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:34.895273 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:34.902935 27660 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:34.903352 12640 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:34.836220 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:34.891755 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:34.910473 27711 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.910645 27504 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.911021 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:34.911641 27504 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.912868 27617 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:34.913429 27617 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:34.913947 27617 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:34.916987 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:34.918130 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:34.911057 27711 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.920826 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:34.912338 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:34.922389 27642 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:34.923138 27642 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:34.923723 27642 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:34.923329 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:34.930205 27504 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:34.921515 27711 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:34.934208 27711 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.935549 27711 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:34.933212 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.943421 27504 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:18:34.946686 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49370.9us
I20260709 12:18:34.950740 12640 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:34.952847 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:34.957144 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:34.901791 27705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.838s	user 0.009s	sys 0.023s
W20260709 12:18:35.022511 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 308655us
I20260709 12:18:35.027709 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:35.029603 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.964457 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:35.036140 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71853.8us
I20260709 12:18:34.899163 27690 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:34.898221 27602 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:18:35.049721 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70872.9us
I20260709 12:18:35.039422 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:34.944902 27730 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:35.041271 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.042695 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:34.898675 27791 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:18:35.073274 27795 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20260709 12:18:35.075446 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:35.076133 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:35.077100 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:35.078275 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.060230 12644 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:35.080237 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:35.080282 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 46 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:35.030426 27705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.978s	user 0.009s	sys 0.024s
I20260709 12:18:34.886785 27340 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:34.966923 27779 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:35.069216 27728 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.032342 27504 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:35.071394 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.025166 27734 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:18:34.967959 27761 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:18:35.084267 12639 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:35.063475 27763 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election won for term 49
W20260709 12:18:35.090327 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:35.115830 12638 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:35.118299 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:34.948293 27565 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.114s
I20260709 12:18:35.125309 12488 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:35.126243 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.130084 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:35.027420 27188 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.092s
I20260709 12:18:35.107142 27705 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:35.136200 27275 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:35.084797 27690 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:35.138859 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2283) took 1753 ms (client timeout 2056 ms). Trace:
W20260709 12:18:35.139835 27721 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.037s	user 0.000s	sys 0.004s
I20260709 12:18:35.139967 12490 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:18:35.142417 27805 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.142611 12643 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 24. Preceding OpId from leader: term: 51 index: 25. (index mismatch)
I20260709 12:18:35.141299 27804 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:35.146297 12646 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:35.147540 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.124872 27734 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:35.128790 27761 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:35.116047 27568 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.151372 27808 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:35.152917 27493 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:35.153898 27379 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:35.158522 27676 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:35.160995 27781 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.794s	user 0.000s	sys 0.002s
I20260709 12:18:35.162990 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.166750 27791 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.167189 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6072) took 1028 ms. Trace:
I20260709 12:18:35.184655 12639 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:35.240432 12563 rpcz_store.cc:276] 0709 12:18:34.138374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.430307 (+291933us) service_pool.cc:224] Handling call
0709 12:18:34.430629 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:34.901276 (+470647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:35.166863 (+265587us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b5400044054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:35.167113 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1077036800}
I20260709 12:18:35.241056 27275 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:35.241616 27275 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:35.089869 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64645us
I20260709 12:18:35.242944 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:35.243676 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.207690 27731 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:35.202944 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:35.148309 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.199936 27763 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:35.250718 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:35.107499 27216 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.26, Last appended by leader: 26, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.221046 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:35.258327 27799 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:18:35.259413 27568 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:35.225858 27805 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.262216 27805 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:35.236409 27379 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:35.265084 27379 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.266701 27379 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:35.266247 27763 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 48.20, Last appended by leader: 20, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.272388 27783 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.272969 12637 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:35.274279 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.275357 12497 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 41 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 24.
I20260709 12:18:35.276499 27791 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:35.280094 27579 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.002s
I20260709 12:18:35.281903 27705 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.282681 27660 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.286772 12569 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: [47.29-47.29]
I20260709 12:18:35.076251 27623 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.109s
I20260709 12:18:35.232569 12638 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:35.235481 12492 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
W20260709 12:18:35.200881 12495 rpcz_store.cc:269] 0709 12:18:33.384940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:33.610931 (+225991us) service_pool.cc:224] Handling call
0709 12:18:35.138542 (+1527611us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:35.138763 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":664544128}
W20260709 12:18:35.294555 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50997.6us
I20260709 12:18:35.295326 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.225750 27734 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:35.216114 27804 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:35.247373 13257 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 27. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:18:35.249082 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:35.222767 27711 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:35.227799 27761 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:35.258719 27617 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.235805 27493 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:35.128010 27504 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.290900 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:35.292229 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:35.292970 12643 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 44 index: 23. (index mismatch)
I20260709 12:18:35.238777 27669 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:18:35.296902 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 171833us
I20260709 12:18:35.297772 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5710) took 5382 ms. Trace:
I20260709 12:18:35.326486 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.311179 27783 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.293169 27660 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:35.331374 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:35.327286 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.347188 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:35.312563 27804 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:35.346778 12563 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
W20260709 12:18:35.349084 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 307919us
I20260709 12:18:35.342002 12493 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:35.351172 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:35.336253 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.354310 27805 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:35.355010 27431 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:35.355770 27805 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:35.355935 27431 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.355270 27427 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:35.310089 27728 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:35.357151 27728 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.356905 27805 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 46.31, Last appended by leader: 31, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.360653 27728 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:35.355140 27731 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.007s
I20260709 12:18:35.328325 12564 rpcz_store.cc:276] 0709 12:18:29.906557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:30.347547 (+440990us) service_pool.cc:224] Handling call
0709 12:18:30.365517 (+ 17970us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:30.385374 (+ 19857us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:30.403035 (+ 17661us) log.cc:844] Serialized 14 byte log entry
0709 12:18:31.890413 (+1487378us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b180044c800. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:32.324935 (+434522us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:32.350703 (+ 25768us) log.cc:844] Serialized 26 byte log entry
0709 12:18:32.350892 (+   189us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:32.350931 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:32.350998 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:34.046101 (+1695103us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:35.139476 (+1093375us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:18:35.288106 (+148630us) raft_consensus.cc:1685] finished
0709 12:18:35.288141 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:35.288361 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6093,"prepare.queue_time_us":17922,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":3166,"raft.queue_time_us":8661,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":17550,"spinlock_wait_cycles":423804800,"thread_start_us":434272,"threads_started":1,"wal-append.queue_time_us":2438317}
I20260709 12:18:35.310880 27568 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:35.356388 27431 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:35.362152 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:35.368108 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:35.316613 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210377us
I20260709 12:18:35.371122 27731 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:35.371805 27731 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:35.372190 27731 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:35.379649 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.323225 27504 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:35.339607 27783 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.383620 12570 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: []
W20260709 12:18:35.339181 27560 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.583s	user 0.013s	sys 0.000s
I20260709 12:18:35.385376 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.381986 13256 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:35.393469 27753 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:35.393657 27612 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:35.397043 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49333.4us
W20260709 12:18:35.421561 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 70463.6us
I20260709 12:18:35.397972 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.384446 27565 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 48.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:18:35.412226 27340 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.340533 27579 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:35.414525 27642 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.410159 27520 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.006s
I20260709 12:18:35.410529 27753 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:35.338595 27629 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:35.415961 27783 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.338043 27676 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:35.422727 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.249818 27275 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.430590 13256 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:18:35.421793 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:35.442476 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.444221 12563 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:35.408039 27568 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:18:35.416725 12493 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 42 index: 22. (index mismatch)
W20260709 12:18:35.432296 27600 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.739s	user 0.013s	sys 0.000s
I20260709 12:18:35.419292 27756 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:18:35.451839 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:35.433498 27597 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:35.446085 27612 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:35.463182 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:35.464296 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:35.432502 27814 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:35.482211 12495 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:35.449436 27520 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:35.419989 27560 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.496169 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:35.495857 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.501013 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:35.438688 27802 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.043s
I20260709 12:18:35.508108 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:35.543619 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47920.6us
I20260709 12:18:35.538923 27597 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:35.551373 27617 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:35.551666 27597 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.28, Last appended by leader: 28, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:35.552862 27817 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:18:35.448899 27454 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.048s
I20260709 12:18:35.503762 27814 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.561866 27802 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:35.562327 27802 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:35.563160 27802 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 43.31, Last appended by leader: 31, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.542801 27791 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.026s
I20260709 12:18:35.565521 27690 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.566254 27753 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 46.31, Last appended by leader: 31, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.568356 27242 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20260709 12:18:35.569283 27814 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:35.569727 27821 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:18:35.469867 27756 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.450073 27579 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:35.577275 12460 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 3.070s (SENT) [suppressed 12 similar messages]
I20260709 12:18:35.457139 27676 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:35.582443 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 140050us
I20260709 12:18:35.465574 27711 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:35.590193 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.539659 27600 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:35.592265 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:35.592985 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.481249 27805 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:35.602393 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:35.602871 12566 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 22. (term mismatch)
I20260709 12:18:35.603538 12566 pending_rounds.cc:85] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:18:35.603837 12566 pending_rounds.cc:107] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.22
I20260709 12:18:35.604141 12566 raft_consensus.cc:2889] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:35.606593 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.607811 12645 raft_consensus.cc:2410] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 42 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 28.
I20260709 12:18:35.573989 27660 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:35.609876 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:35.470337 27779 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.124s
I20260709 12:18:35.588266 27756 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:35.613504 12566 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:35.619483 12564 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 25. Preceding OpId from leader: term: 51 index: 25. (term mismatch)
I20260709 12:18:35.619881 12564 pending_rounds.cc:85] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 24
W20260709 12:18:35.459013 27769 consensus_queue.cc:1175] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 47 status { last_received { term: 47 index: 26 } last_committed_idx: 26 last_received_current_leader { term: 47 index: 26 } }
I20260709 12:18:35.594724 27811 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:35.702260 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:35.703223 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:35.704965 12566 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: [47.34-47.34]
I20260709 12:18:35.573280 27804 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.620129 12564 pending_rounds.cc:107] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 50.25
I20260709 12:18:35.710876 12564 raft_consensus.cc:2889] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:35.592124 27676 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:35.604861 27805 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.611254 27809 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:18:35.719779 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 101213us
I20260709 12:18:35.618148 27791 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:35.608458 27711 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:35.617537 27454 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:35.629918 27660 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:35.594834 27730 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.702836 27811 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.609457 27431 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.610028 27731 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.742321 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:35.718909 27805 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:35.753377 27811 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:35.734133 27791 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:35.766299 27791 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:35.776068 13254 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:35.786664 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 194481us
I20260709 12:18:35.787703 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.789011 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.790602 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.741290 27809 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:35.795867 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.759219 27772 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 LEADER]: Stepping down as leader of term 50
W20260709 12:18:35.803007 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:35.803143 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.809288 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:35.810544 12497 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:35.781792 27730 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:35.826380 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 249137us
I20260709 12:18:35.797147 27763 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:35.802165 27772 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:18:35.882810 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:35.830092 27520 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:35.844632 27600 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:35.921008 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 117954us
W20260709 12:18:35.924176 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:35.895354 27825 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:35.928001 27504 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.511s	user 0.014s	sys 0.000s
I20260709 12:18:35.909860 27684 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.019s
I20260709 12:18:35.928634 27504 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.928004 27825 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.930675 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4370) took 1153 ms. Trace:
I20260709 12:18:35.930428 27600 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.931123 12490 rpcz_store.cc:276] 0709 12:18:34.776873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.777054 (+   181us) service_pool.cc:224] Handling call
0709 12:18:35.139469 (+362415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.204164 (+ 64695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:35.930433 (+726269us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b540034dd50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:35.930606 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":832497920}
I20260709 12:18:35.925047 27833 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:35.906354 27560 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:35.850181 27794 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.074s
W20260709 12:18:35.922672 27714 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.481s	user 0.012s	sys 0.009s
I20260709 12:18:35.942384 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:35.913772 27602 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.026s
I20260709 12:18:35.947857 27837 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.948635 27837 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.951112 27837 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.939867 27779 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:18:35.951992 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1892) took 1426 ms (client timeout 1718 ms). Trace:
I20260709 12:18:35.952153 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:35.953194 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:35.953320 27504 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.953823 27623 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:35.962272 27602 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:35.962886 27602 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:35.954324 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.955150 13262 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:35.924232 27579 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:35.832957 27711 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 48.27, Last appended by leader: 27, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.971375 27804 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:35.928102 27753 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:35.939767 27772 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 50.29, Last appended by leader: 29, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.973187 27772 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:35.973280 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:35.939565 27454 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.974861 27783 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:35.975124 27756 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.975425 27783 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:35.975497 27804 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:35.975982 27783 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:18:35.952561 13254 rpcz_store.cc:269] 0709 12:18:34.525887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.627231 (+101344us) service_pool.cc:224] Handling call
0709 12:18:35.951718 (+1324487us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54005c4c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:35.951907 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2469697408}
I20260709 12:18:35.969547 26496 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:35.976891 27761 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.654s	user 0.010s	sys 0.002s
I20260709 12:18:35.977181 26496 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
I20260709 12:18:35.974656 27454 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:35.963280 27602 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:35.968786 12637 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:35.974656 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:35.992340 27803 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:35.993064 27803 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.996868 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:35.943081 27714 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.505s	user 0.013s	sys 0.009s
I20260709 12:18:35.873315 27660 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:35.939584 27520 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:35.976367 27804 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:35.965719 27832 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:18:35.942093 27836 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:36.015196 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.016767 12564 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:35.999627 27746 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:36.023092 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:36.006645 27165 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:36.026360 12564 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:35.977376 27761 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.031250 12641 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
I20260709 12:18:36.003012 27684 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.022s
W20260709 12:18:36.032056 27669 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.700s	user 0.015s	sys 0.000s
I20260709 12:18:36.036446 27669 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:35.938047 27809 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 50.29, Last appended by leader: 29, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.012187 27792 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.027s
W20260709 12:18:36.030215 27831 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260709 12:18:36.063719 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:36.064240 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.023834 27431 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.066427 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.068609 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.077314 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.078017 12644 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:36.079527 12638 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:18:36.081784 12644 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:36.081566 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.082942 27825 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.085319 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.084642 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.083060 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:36.083199 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.079095 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:36.084156 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.088716 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:36.091476 12638 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [45.27-45.27]
I20260709 12:18:36.095647 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.098301 12642 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:36.098342 13261 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:36.099925 27844 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:36.100449 27844 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:36.101271 27844 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.34, Last appended by leader: 34, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.102552 27602 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.104800 27602 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.035933 27560 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.063s
I20260709 12:18:36.106148 27560 consensus_queue.cc:1243] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 43.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:36.108865 27762 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.111519 13258 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:36.113893 27833 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:36.114538 27833 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:36.074440 27504 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.115124 27843 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:36.115471 27833 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.115685 27504 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.115803 27843 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.086339 27827 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.117447 27827 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.119271 27843 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.109185 27805 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.120102 27827 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:36.120817 27827 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.120129 27493 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 48.20, Last appended by leader: 20, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.122313 27805 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.123415 27493 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:36.126596 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2162) took 1449 ms. Trace:
I20260709 12:18:36.127132 12646 rpcz_store.cc:276] 0709 12:18:34.676866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.677754 (+   888us) service_pool.cc:224] Handling call
0709 12:18:35.146061 (+468307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.215279 (+ 69218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.126268 (+910989us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b54002e5250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:36.126520 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1076190592}
I20260709 12:18:36.134332 12644 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:36.135079 13255 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:36.066421 27792 consensus_queue.cc:1243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 50.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
W20260709 12:18:36.111860 27734 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.789s	user 0.012s	sys 0.000s
I20260709 12:18:36.139979 13259 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:36.087832 27669 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:36.145969 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:36.154278 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122121us
I20260709 12:18:36.102608 27844 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:36.139518 27714 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.138670 27783 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.121317 27827 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.197959 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2890) took 1431 ms. Trace:
I20260709 12:18:36.198621 12569 rpcz_store.cc:276] 0709 12:18:34.766494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.768841 (+  2347us) service_pool.cc:224] Handling call
0709 12:18:34.769151 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:35.287480 (+518329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.197515 (+910035us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:36.197861 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1189525888}
I20260709 12:18:36.200135 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.111639 27811 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.205132 27504 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.207376 12568 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:36.207837 12568 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:36.208626 12568 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 49.22, Last appended by leader: 22, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.172219 27714 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.212749 27714 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.149195 27734 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.215358 27734 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.216550 12568 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:36.219328 27763 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:36.219890 27763 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:36.220253 27827 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.220829 27783 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.224952 27756 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.201560 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.228164 27676 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.229269 12569 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:36.230058 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 1891) took 1766 ms. Trace:
I20260709 12:18:36.233264 27275 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:36.233603 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.233052 13260 rpcz_store.cc:276] 0709 12:18:34.463293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.491138 (+ 27845us) service_pool.cc:224] Handling call
0709 12:18:36.217487 (+1726349us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:36.229941 (+ 12454us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1951967488}
I20260709 12:18:36.237790 27676 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.238823 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.240168 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.240760 12563 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:36.232919 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:36.242447 13258 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:36.243302 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5598) took 1475 ms. Trace:
I20260709 12:18:36.238329 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:36.230271 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.244956 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
W20260709 12:18:36.245734 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.247339 12496 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:36.248070 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.248739 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.249440 12488 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:36.243743 12645 rpcz_store.cc:276] 0709 12:18:34.768484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.799737 (+ 31253us) service_pool.cc:224] Handling call
0709 12:18:36.243075 (+1443338us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54002e6154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:36.243226 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1797278080}
I20260709 12:18:36.251611 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.252592 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.253227 12496 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:36.251542 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.254863 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.241564 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.258811 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:36.258085 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.261586 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:36.266728 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1330224430000726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
I20260709 12:18:36.270709 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.270998 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.272532 12563 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:36.272800 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.274102 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.274528 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.275370 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.275701 12497 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:36.276929 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.270120 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.278375 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:36.279932 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.280862 12495 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:36.281418 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.282161 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.283106 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.279844 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.285663 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.287074 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:36.288493 12495 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:36.289021 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.282347 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:36.293186 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:36.296696 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.298286 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:36.298754 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.287865 27431 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.012s
I20260709 12:18:36.313258 27431 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:36.313728 27431 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:36.295521 27784 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:36.301355 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.301764 27756 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:36.315321 27794 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:36.300482 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.290100 27714 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.274153 27772 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.301617 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.287098 27753 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:36.312366 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.314249 12497 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:36.315358 27756 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:36.270247 27493 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:36.299525 12490 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:36.324587 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:36.327698 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.314821 27784 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:36.331857 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.302979 27783 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:36.335758 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:36.316210 27794 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.288054 27676 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 50.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.035s
I20260709 12:18:36.337285 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:36.330025 12568 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 22. Preceding OpId from leader: term: 50 index: 22. (term mismatch)
I20260709 12:18:36.351856 12568 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:18:36.352165 12568 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 49.22
I20260709 12:18:36.352458 12568 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:36.335139 27756 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:36.354666 12569 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:36.362174 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:36.367241 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.368022 13259 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:36.361434 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:36.345893 27784 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:36.405368 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.348227 27808 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:36.346974 27783 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.435175 13259 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
W20260709 12:18:36.435849 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74690.4us
I20260709 12:18:36.350890 27794 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:36.448747 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.333086 27504 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.035s
I20260709 12:18:36.423295 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.421727 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:36.354095 27617 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:36.450134 13258 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:36.450300 27617 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.450726 27617 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:36.449805 12490 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:36.456665 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.458290 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:36.458570 13258 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:36.458897 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.330716 27843 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.025s
I20260709 12:18:36.314973 27431 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 47.29, Last appended by leader: 29, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.437767 27494 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:36.438894 27684 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:36.416141 27833 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.024s
I20260709 12:18:36.440404 27808 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:36.335412 27753 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.443849 12494 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 48 because replica has last-logged OpId of term: 47 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 33.
I20260709 12:18:36.443069 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:36.504208 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62033.4us
I20260709 12:18:36.444033 27783 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:36.445163 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.327070 27714 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.462008 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.331532 27772 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.464922 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:36.368059 27734 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:36.589464 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2157) took 1569 ms. Trace:
I20260709 12:18:36.452226 12395 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 23 to 44, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 44 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:36.487510 12564 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:36.590005 13262 rpcz_store.cc:276] 0709 12:18:34.940359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.940636 (+   277us) service_pool.cc:224] Handling call
0709 12:18:35.954918 (+1014282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.971183 (+ 16265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.509415 (+538232us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54005c4c50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:36.509592 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2331252096}
W20260709 12:18:36.590669 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.342260 27493 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.415537 27660 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:36.491624 12497 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:18:36.462280 27825 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:36.484335 27579 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:18:36.492831 27791 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.725s	user 0.017s	sys 0.001s
I20260709 12:18:36.491943 27845 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:36.494779 27684 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:36.488826 27560 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:36.608361 12395 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 34 to 47, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 47 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:36.558502 27846 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:18:36.593555 27815 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:36.616518 27493 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.608762 27579 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:36.625552 13262 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:36.498342 27717 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.115s
I20260709 12:18:36.631481 12644 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:36.598270 27660 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:36.635725 12395 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 26 to 38. New cstate: current_term: 38 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:36.506131 27504 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.113s
I20260709 12:18:36.613168 27734 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:36.627921 27684 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:36.661092 12395 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 39. New cstate: current_term: 39 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:36.634498 27845 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:36.639211 27705 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.608609 27825 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.666498 12644 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:36.671731 12644 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:36.671952 27825 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.672757 27705 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.639000 27815 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:36.537369 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:36.653798 27660 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:36.682744 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 145492us
I20260709 12:18:36.648488 27579 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:36.667949 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.620630 27791 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.684689 12395 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 29 to 36, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 36 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:36.637122 27560 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:36.693671 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207495us
I20260709 12:18:36.701143 12395 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 31 to 37, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 37 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:36.700016 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:36.708667 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:36.709492 27852 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.106s	user 0.001s	sys 0.001s
I20260709 12:18:36.717515 12566 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:36.710518 27856 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:18:36.720235 12395 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 41 to 42, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 42 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:36.680011 27843 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 51.30, Next index: 31, Last known committed idx: 28, Time since last communication: 0.038s
I20260709 12:18:36.726374 27714 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.703882 27612 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.024s
I20260709 12:18:36.728686 27612 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:36.729151 27612 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:36.730266 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6083) took 1119 ms. Trace:
I20260709 12:18:36.731299 12563 rpcz_store.cc:276] 0709 12:18:35.610900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.273576 (+662676us) service_pool.cc:224] Handling call
0709 12:18:36.294429 (+ 20853us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:36.294686 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:36.294711 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:36.295187 (+   476us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.496627 (+201440us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:36.496684 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.496793 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.729815 (+233022us) raft_consensus.cc:1685] finished
0709 12:18:36.729856 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.730165 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":162301,"threads_started":1,"wal-append.queue_time_us":369686}
W20260709 12:18:36.733031 27835 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.813s	user 0.017s	sys 0.000s
I20260709 12:18:36.688768 27808 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.710547 27833 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:36.734339 27845 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.740442 27772 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.741035 27734 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.743417 27791 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.744648 12570 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:36.744578 27815 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.29, Last appended by leader: 29, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.746112 27815 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:36.750895 27705 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.756863 27454 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.745564 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6080) took 1141 ms. Trace:
I20260709 12:18:36.758245 12570 rpcz_store.cc:276] 0709 12:18:35.603930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.743782 (+1139852us) service_pool.cc:224] Handling call
0709 12:18:36.744399 (+   617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.745247 (+   848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.745497 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:36.758787 27802 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:36.762005 27860 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
W20260709 12:18:36.764340 27765 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 50 current term: 50 is higher.
I20260709 12:18:36.764618 27617 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:36.766213 27842 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.609s	user 0.000s	sys 0.011s
W20260709 12:18:36.774516 12488 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.523s	user 0.014s	sys 0.000s
I20260709 12:18:36.775050 12488 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
W20260709 12:18:36.714902 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 233282us
I20260709 12:18:36.769097 27765 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:36.769949 27612 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.771644 12495 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:18:36.768486 27454 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:36.691493 27805 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:36.777083 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:36.777765 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:36.713052 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:36.792295 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79337.9us
I20260709 12:18:36.793267 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.794195 27791 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:18:36.772987 27275 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.539s	user 0.014s	sys 0.000s
I20260709 12:18:36.777778 27714 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:36.801975 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.797955 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:36.785436 27854 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:18:36.778189 27717 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:36.785430 27838 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:36.799578 27851 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:18:36.795804 27779 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:18:36.847506 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48009.6us
I20260709 12:18:36.801972 27805 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:36.819692 27763 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.001s
I20260709 12:18:36.851898 12395 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 47. New cstate: current_term: 47 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:36.850802 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.796123 27454 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.809607 27734 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:36.789173 27765 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.817346 27791 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:36.858414 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6084) took 1246 ms. Trace:
W20260709 12:18:36.863844 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44133.9us
I20260709 12:18:36.869889 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.863734 12568 rpcz_store.cc:276] 0709 12:18:35.611811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.364050 (+752239us) service_pool.cc:224] Handling call
0709 12:18:36.364460 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:36.414725 (+ 50265us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:36.414763 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:36.415305 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.652885 (+237580us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:36.652952 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.653063 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.858168 (+205105us) raft_consensus.cc:1685] finished
0709 12:18:36.858189 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.858354 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":283,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":54530,"thread_start_us":206785,"threads_started":1,"wal-append.queue_time_us":271158}
W20260709 12:18:36.872643 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:36.859520 27868 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:18:36.817361 27861 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:36.867996 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.875877 27494 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:36.876965 27869 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:18:36.842777 27860 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.035s
I20260709 12:18:36.879941 27765 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:36.880710 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6087) took 1265 ms. Trace:
I20260709 12:18:36.883320 27833 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:36.932317 12564 rpcz_store.cc:276] 0709 12:18:35.615197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.592545 (+977348us) service_pool.cc:224] Handling call
0709 12:18:36.592972 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:36.602883 (+  9911us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:36.602923 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:36.611528 (+  8605us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.781588 (+170060us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:36.781669 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.781780 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:36.880433 (+ 98653us) raft_consensus.cc:1685] finished
0709 12:18:36.880461 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:36.880627 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11477,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":8191,"thread_start_us":165884,"threads_started":1,"wal-append.queue_time_us":201870}
I20260709 12:18:36.816589 27794 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.883584 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.807475 27714 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.863914 12395 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 36 to 41. New cstate: current_term: 41 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:36.873891 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:36.950106 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82215.9us
I20260709 12:18:36.820019 27717 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.859195 27803 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:36.850087 27275 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.037s
I20260709 12:18:36.946255 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.883080 27784 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.872661 27612 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:36.951449 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.890000 27734 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:36.953047 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:36.954794 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:36.956313 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.847301 27838 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:36.973285 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.967787 12395 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 41, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 41 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:18:36.956833 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 132393us
I20260709 12:18:36.993710 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:36.953738 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.970786 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:36.911844 27861 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:36.873425 27779 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.091s
I20260709 12:18:36.961822 12490 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:36.946846 27791 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:36.959380 27863 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.114s	user 0.001s	sys 0.001s
I20260709 12:18:36.973429 13258 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:36.997774 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:36.999116 12490 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.003154 13256 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [41.27-43.28]
I20260709 12:18:36.999989 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.003337 12495 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:36.861384 27763 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.957360 27765 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:36.958289 27617 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.961948 27454 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:36.865247 27783 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:36.957981 27714 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.009666 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:37.042945 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:36.971719 27870 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:37.050179 13262 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:37.039137 12395 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 31 to 35. New cstate: current_term: 35 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:36.996272 27861 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:37.063814 12395 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 45. New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.004490 27872 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:37.055078 12495 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:37.008894 12490 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.28, Last appended by leader: 28, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.072221 12497 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:37.070145 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:37.072928 12395 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 30 to 37, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 37 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.051661 27877 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:37.008919 27791 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.077279 12490 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:36.999042 27805 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 42.28, Last appended by leader: 28, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.109104 13262 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 31. Preceding OpId from leader: term: 47 index: 32. (index mismatch)
I20260709 12:18:37.106734 12568 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 44 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 28.
I20260709 12:18:37.122958 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5946) took 2049 ms. Trace:
I20260709 12:18:37.130379 12493 rpcz_store.cc:276] 0709 12:18:35.073344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.384955 (+311611us) service_pool.cc:224] Handling call
0709 12:18:35.414035 (+ 29080us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.457190 (+ 43155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.122597 (+1665407us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:37.122871 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5364608}
I20260709 12:18:37.087817 12395 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 44, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 44 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:37.118606 12642 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:37.122218 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1920) took 1196 ms. Trace:
I20260709 12:18:37.136642 13256 rpcz_store.cc:276] 0709 12:18:35.925784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.926263 (+   479us) service_pool.cc:224] Handling call
0709 12:18:36.872261 (+945998us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:18:36.872441 (+   180us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:37.003628 (+131187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.121794 (+118166us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:37.122119 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":303945216}
W20260709 12:18:37.107270 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.078768 27756 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.664s	user 0.010s	sys 0.005s
I20260709 12:18:37.082430 27454 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.078408 27617 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:37.143906 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:37.129745 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:37.140652 12460 connection.cc:256] RPC call timeout handler was delayed by -1.0440114680002353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18130000000000002s for a response
I20260709 12:18:37.066287 27705 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.019s
I20260709 12:18:37.148269 12495 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:37.148205 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.245113 12488 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:37.245966 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.246568 12488 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: []
W20260709 12:18:37.170454 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.247936 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 294989us
I20260709 12:18:37.249032 27882 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.072741 27872 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.094949 27792 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.092128 27520 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.250228 12570 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:37.241385 13254 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:37.109591 27763 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:37.249680 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 94873.4us
W20260709 12:18:37.255002 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.072559 27870 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260709 12:18:37.103904 27753 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.240871 27684 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.106427 27717 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.088081 27660 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.268554 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5930) took 2342 ms. Trace:
I20260709 12:18:37.301901 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5614) took 1164 ms. Trace:
W20260709 12:18:37.268806 27579 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.001s
I20260709 12:18:37.093403 27827 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:37.270499 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5928) took 2370 ms. Trace:
I20260709 12:18:37.315263 12492 rpcz_store.cc:276] 0709 12:18:34.899681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.899813 (+   132us) service_pool.cc:224] Handling call
0709 12:18:34.900217 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.294554 (+394337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.270216 (+1975662us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:37.270434 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":770112768}
I20260709 12:18:37.240397 27642 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.272920 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.273685 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6081) took 1669 ms. Trace:
I20260709 12:18:37.249883 27881 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.025s
I20260709 12:18:37.275311 13262 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 47 index: 30. (index mismatch)
I20260709 12:18:37.241451 27756 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.276654 27731 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:37.091627 27866 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:18:37.258391 12564 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 37.
W20260709 12:18:37.250262 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.266525 27520 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:37.256669 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 123461us
I20260709 12:18:37.382643 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:37.254208 27873 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:18:37.112188 27784 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.092787 27838 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.287392 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:37.298305 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5609) took 1201 ms. Trace:
I20260709 12:18:37.111196 27794 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.301350 12489 rpcz_store.cc:276] 0709 12:18:34.926690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:34.928397 (+  1707us) service_pool.cc:224] Handling call
0709 12:18:34.928846 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:35.605705 (+676859us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:35.605740 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:35.605781 (+    41us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:35.605811 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.268124 (+1662313us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:37.268192 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.268479 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1556051200}
I20260709 12:18:37.311862 13254 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 53 because replica has last-logged OpId of term: 52 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 33.
W20260709 12:18:37.378484 27863 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:18:37.108865 27761 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.116186 27379 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:37.302363 12638 rpcz_store.cc:276] 0709 12:18:36.138076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.489372 (+351296us) service_pool.cc:224] Handling call
0709 12:18:36.489934 (+   562us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:36.501875 (+ 11941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:36.502808 (+   933us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:36.604019 (+101211us) log.cc:844] Serialized 52 byte log entry
0709 12:18:36.688039 (+ 84020us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:37.271086 (+583047us) log.cc:844] Serialized 14 byte log entry
0709 12:18:37.301549 (+ 30463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.301659 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:37.301701 (+    42us) raft_consensus.cc:1685] finished
0709 12:18:37.301711 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.301846 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":612787,"prepare.queue_time_us":583299,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":98,"spinlock_wait_cycles":32840448,"thread_start_us":47460,"threads_started":1,"wal-append.queue_time_us":156226}
I20260709 12:18:37.258023 27454 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.411502 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.317731 12569 rpcz_store.cc:276] 0709 12:18:35.604213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.269278 (+665065us) service_pool.cc:224] Handling call
0709 12:18:36.269875 (+   597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.356720 (+ 86845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.273373 (+916653us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:37.273594 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194334464}
I20260709 12:18:37.280777 27808 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:37.332130 12609 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.608s (SENT)
I20260709 12:18:37.296675 27874 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.109s
I20260709 12:18:37.374519 12491 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:37.259418 27803 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:37.281174 27782 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:37.255779 27791 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.430652 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.381206 12563 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:37.296432 27617 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:37.320713 27597 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.400s	user 0.006s	sys 0.015s
I20260709 12:18:37.251775 12395 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 46, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 46 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.383410 13259 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:37.443769 27520 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.444845 27617 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 48.20, Last appended by leader: 20, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.297233 27762 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.446478 27762 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.384449 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.251772 27872 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.253476 27792 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.087846 27877 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.393325 12645 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:37.385145 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:37.398986 13258 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
I20260709 12:18:37.091509 27783 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.399767 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:37.451267 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51661.6us
I20260709 12:18:37.453918 13259 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:37.390664 12643 rpcz_store.cc:276] 0709 12:18:36.097439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.097710 (+   271us) service_pool.cc:224] Handling call
0709 12:18:36.120613 (+ 22903us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.120901 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:36.328472 (+207571us) log.cc:844] Serialized 14 byte log entry
0709 12:18:37.269444 (+940972us) log.cc:844] Serialized 14 byte log entry
0709 12:18:37.297929 (+ 28485us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.297983 (+    54us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:37.298031 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.298101 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.298239 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1456535,"prepare.queue_time_us":1156724,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":101,"raft.queue_time_us":32865,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":14227,"spinlock_wait_cycles":21967232,"thread_start_us":367943,"threads_started":2,"wal-append.queue_time_us":349115}
W20260709 12:18:37.400837 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113586us
I20260709 12:18:37.402299 27805 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:37.404022 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:37.315119 27827 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:37.409420 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.412909 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:37.282596 27600 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:37.249617 27882 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.411854 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:37.462811 13262 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:37.401036 27717 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.049s
W20260709 12:18:37.418017 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85980.5us
W20260709 12:18:37.464490 12609 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.672s (SENT)
I20260709 12:18:37.465116 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:37.088728 27734 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 48.27, Last appended by leader: 27, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.371254 27275 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.082s
I20260709 12:18:37.384927 27797 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:37.432124 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.466681 27845 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:37.467053 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.468389 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.316447 27579 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.410789 27188 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
W20260709 12:18:37.488628 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6088) took 1781 ms (client timeout 1728 ms). Trace:
W20260709 12:18:37.434574 27886 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:18:37.416819 27379 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:37.490664 27379 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.491811 27379 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:37.437480 27782 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.423522 27808 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.434993 27803 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:37.500152 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.502803 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:37.503445 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6090) took 1774 ms. Trace:
I20260709 12:18:37.503674 12489 raft_consensus.cc:606] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:37.504020 12570 rpcz_store.cc:276] 0709 12:18:35.728684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.764622 (+1035938us) service_pool.cc:224] Handling call
0709 12:18:36.765080 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.262295 (+497215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.502520 (+240225us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5400140120. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6a02 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:37.503146 (+   626us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b5400044a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:37.503366 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1149302272}
I20260709 12:18:37.469033 27872 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:37.512766 27872 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.513176 27872 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:37.518030 27792 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.440364 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:37.459089 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:37.456230 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:37.441174 27802 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.006s
I20260709 12:18:37.549072 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:37.461052 27804 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
W20260709 12:18:37.441563 27597 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.528s	user 0.006s	sys 0.016s
I20260709 12:18:37.469408 12569 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:37.477012 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2165) took 2428 ms. Trace:
I20260709 12:18:37.568202 12639 rpcz_store.cc:276] 0709 12:18:35.048575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.242099 (+193524us) service_pool.cc:224] Handling call
0709 12:18:35.242424 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:35.245772 (+  3348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:35.245819 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:35.246464 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.816875 (+1570411us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b1800436c00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:18:36.949053 (+132178us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:36.949111 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.949192 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:37.464641 (+515449us) raft_consensus.cc:1685] finished
0709 12:18:37.464661 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.476872 (+ 12211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24868,"raft.queue_time_us":4506,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":2454,"spinlock_wait_cycles":3397888,"thread_start_us":131914,"threads_started":1,"wal-append.queue_time_us":1791403}
I20260709 12:18:37.570112 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.477331 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.571293 12643 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:37.572078 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:37.437487 27684 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.470203 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.573580 12572 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:37.574156 27808 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.574546 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:37.480486 27660 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.481853 12395 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 44, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 44 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:37.578276 27494 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 37.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:37.579880 27794 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:37.580417 27794 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:37.580814 27794 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:37.582625 27877 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.583163 27825 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:37.584408 27782 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.585316 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:37.585795 27782 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:18:37.586416 27494 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:37.586901 27494 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.587863 12643 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:37.588042 27803 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.591271 27882 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.464028 27600 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.450616 27861 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.483814 27762 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.484820 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.605233 12638 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
W20260709 12:18:37.605299 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120493us
W20260709 12:18:37.487036 27814 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 45 current term: 45 is higher.
I20260709 12:18:37.488821 27797 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:18:37.489130 12566 rpcz_store.cc:269] 0709 12:18:35.707391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.714884 (+  7493us) service_pool.cc:224] Handling call
0709 12:18:37.488383 (+1773499us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:37.488557 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2208054656}
I20260709 12:18:37.567080 27188 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
W20260709 12:18:37.527089 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 70974.6us
I20260709 12:18:37.612310 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:37.461755 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76690.6us
I20260709 12:18:37.566622 13260 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:37.566599 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:37.602140 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143106us
I20260709 12:18:37.614982 12570 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:37.601888 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:37.467134 27756 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:37.579134 12395 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 34 to 38, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), no longer has a pending config: VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129), VOTER 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130), VOTER 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 38 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:18:37.607005 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 316349us
I20260709 12:18:37.607021 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:37.526800 13254 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:37.485441 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:37.615118 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:37.619192 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:37.624964 12490 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:37.625437 12490 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.626430 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.587868 27494 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.627976 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.628209 27494 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:37.629124 27684 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:37.567813 27783 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.048s
I20260709 12:18:37.614077 27881 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:37.615608 27454 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.622947 27765 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.010s
I20260709 12:18:37.607200 27579 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.629518 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:37.623610 12395 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 48, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 48 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:37.639556 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:37.639649 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.639986 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.637253 13254 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:37.640722 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:37.634012 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.634835 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:37.645706 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:37.644366 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.645709 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:37.643800 27660 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:37.647532 27825 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:37.617570 27188 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.624725 27814 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:37.649219 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:37.676438 12572 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:37.652175 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:37.665498 12570 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:37.630956 27600 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.631125 27881 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.654621 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4381) took 1397 ms. Trace:
I20260709 12:18:37.672380 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:37.683566 12494 rpcz_store.cc:276] 0709 12:18:36.257378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.288757 (+ 31379us) service_pool.cc:224] Handling call
0709 12:18:37.654409 (+1365652us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540034d854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:37.654561 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":354039040}
I20260709 12:18:37.655014 12638 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:37.655580 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2167) took 1389 ms. Trace:
I20260709 12:18:37.633337 27765 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:37.656973 27560 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.676901 27783 raft_consensus.cc:2764] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election decision vote started in defunct term 49: won
I20260709 12:18:37.688994 27765 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:37.689455 27811 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.642213 27684 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.635334 27861 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.648936 12395 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 40. New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:37.647279 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:37.671880 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:37.672556 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:37.673480 12566 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 39 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 21.
I20260709 12:18:37.665757 12497 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:37.673869 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:37.692768 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:37.695344 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:37.696370 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:37.697404 12570 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 44 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 28.
I20260709 12:18:37.670298 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:37.655866 27756 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:18:37.700255 27684 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.662420 27899 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:37.701475 27899 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.685511 12645 rpcz_store.cc:276] 0709 12:18:36.265911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.312062 (+ 46151us) service_pool.cc:224] Handling call
0709 12:18:37.655429 (+1343367us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:37.655526 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2449474304}
I20260709 12:18:37.705490 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.706069 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.708004 12570 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:37.709246 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:37.686936 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.684885 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:37.675873 27825 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:37.712059 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.715261 27825 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:37.704087 27814 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 52.34, Next index: 35, Last known committed idx: 32, Time since last communication: 0.012s
I20260709 12:18:37.716315 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:37.717811 27782 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:18:37.720000 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:37.720458 27783 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:37.721235 27783 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.721577 27783 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:37.724532 27897 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:37.725224 27897 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.725942 27877 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:37.724864 27825 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.726969 27897 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:37.729025 27825 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.729802 27825 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:37.730387 27825 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.731343 27825 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.26, Last appended by leader: 26, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.715703 12568 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 40 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 20.
I20260709 12:18:37.733562 12395 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 32 to 37, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 37 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.690836 27623 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:37.737908 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:37.704223 27690 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:37.711146 27494 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:37.690053 27765 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 49.21, Last appended by leader: 21, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.747241 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6091) took 1997 ms. Trace:
I20260709 12:18:37.747102 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:37.748124 12568 rpcz_store.cc:276] 0709 12:18:35.749795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.738847 (+1989052us) service_pool.cc:224] Handling call
0709 12:18:37.746181 (+  7334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.746497 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.746536 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.746600 (+    64us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:37.746903 (+   303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.746979 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.747147 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.748777 12395 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 38 to 43, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 43 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.753473 12566 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 50. Current term is 51. Ops: [50.29-50.29]
I20260709 12:18:37.754375 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2900) took 1991 ms. Trace:
I20260709 12:18:37.754768 12566 rpcz_store.cc:276] 0709 12:18:35.763235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.746522 (+1983287us) service_pool.cc:224] Handling call
0709 12:18:37.753209 (+  6687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:37.754091 (+   882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.754317 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.755450 27783 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.755638 27188 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 48.27, Last appended by leader: 27, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:37.713897 27861 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:18:37.759147 12568 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:37.756623 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6094) took 1951 ms. Trace:
I20260709 12:18:37.760461 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6092) took 1967 ms. Trace:
I20260709 12:18:37.760929 12568 rpcz_store.cc:276] 0709 12:18:35.793003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.758374 (+1965371us) service_pool.cc:224] Handling call
0709 12:18:37.758827 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.760162 (+  1335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.760403 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.760802 12566 rpcz_store.cc:276] 0709 12:18:35.805006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.755747 (+1950741us) service_pool.cc:224] Handling call
0709 12:18:37.756098 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.756290 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.756316 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.756364 (+    48us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:37.756398 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.756464 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.756569 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.713898 27899 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:37.763939 27911 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.007s
I20260709 12:18:37.778784 27911 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:37.779227 27911 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.766222 27765 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:18:37.767051 27900 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:18:37.737473 27623 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:37.767714 12395 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 36 to 40. New cstate: current_term: 40 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:18:37.755306 27872 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.774811 27885 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.042s
I20260709 12:18:37.776139 27794 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:37.784839 12572 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:37.757419 27784 raft_consensus.cc:2764] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election decision vote started in defunct term 47: won
I20260709 12:18:37.802750 12570 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:37.818321 12395 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 27 to 39, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 39 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:18:37.808051 12497 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
W20260709 12:18:37.810431 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78716us
I20260709 12:18:37.775648 27859 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:37.820804 12566 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:37.780225 27911 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:37.834705 27889 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.335s	user 0.003s	sys 0.000s
I20260709 12:18:37.839818 27911 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:37.841769 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:37.843429 12642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5612) took 1706 ms (client timeout 1865 ms). Trace:
W20260709 12:18:37.843916 12642 rpcz_store.cc:269] 0709 12:18:36.136712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.136867 (+   155us) service_pool.cc:224] Handling call
0709 12:18:37.843218 (+1706351us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:37.843361 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2225224448}
I20260709 12:18:37.845316 27912 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:37.846619 27912 consensus_queue.cc:1243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 46.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:18:37.852872 12644 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:37.853839 12570 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:37.810967 27623 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:37.855682 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2902) took 1087 ms. Trace:
I20260709 12:18:37.862483 12572 rpcz_store.cc:276] 0709 12:18:36.768818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.817779 (+1048961us) service_pool.cc:224] Handling call
0709 12:18:37.855210 (+ 37431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.855387 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.855409 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:37.855460 (+    51us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:37.855488 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.855533 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:37.855624 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:37.855334 12395 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 26 to 35, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 35 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.865415 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:37.867125 12395 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 32 to 37. New cstate: current_term: 37 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:18:37.869959 27876 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:37.873102 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.872524 27876 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:37.923594 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.924188 12488 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: []
W20260709 12:18:37.889318 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:37.869581 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:37.949030 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 79576.5us
I20260709 12:18:37.894750 27216 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:37.898362 27882 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:37.875217 27794 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.900079 27872 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:37.876081 27690 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:37.926642 27913 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:37.972834 27914 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:37.967777 27882 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.025301 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5771) took 1096 ms. Trace:
I20260709 12:18:38.027293 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2293) took 1587 ms. Trace:
I20260709 12:18:38.027848 12491 rpcz_store.cc:276] 0709 12:18:36.439933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.444761 (+  4828us) service_pool.cc:224] Handling call
0709 12:18:38.026986 (+1582225us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54001900d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:38.027204 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1978562944}
I20260709 12:18:38.026067 12564 rpcz_store.cc:276] 0709 12:18:36.928799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.945909 (+ 17110us) service_pool.cc:224] Handling call
0709 12:18:38.025001 (+1079092us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:38.025193 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":670551808}
I20260709 12:18:38.019418 27783 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.055845 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:38.045902 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:38.056499 12491 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:38.047662 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6103) took 2090 ms. Trace:
I20260709 12:18:38.058318 12568 rpcz_store.cc:276] 0709 12:18:35.957574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.803319 (+1845745us) service_pool.cc:224] Handling call
0709 12:18:37.803819 (+   500us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:37.804386 (+   567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.804429 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:37.805302 (+   873us) log.cc:844] Serialized 52 byte log entry
0709 12:18:38.004467 (+199165us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:38.004557 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.004675 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:38.047413 (+ 42738us) raft_consensus.cc:1685] finished
0709 12:18:38.047437 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.047592 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1561,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":7336,"thread_start_us":151818,"threads_started":1,"wal-append.queue_time_us":229167}
I20260709 12:18:38.059996 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5780) took 1127 ms. Trace:
I20260709 12:18:38.060446 12565 rpcz_store.cc:276] 0709 12:18:36.932695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.924201 (+991506us) service_pool.cc:224] Handling call
0709 12:18:37.949373 (+ 25172us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.949697 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:18:37.963450 (+ 13753us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.036828 (+ 73378us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.036975 (+   147us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:38.037073 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.037199 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.059921 (+ 22722us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16607,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":763,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":71788,"threads_started":1}
I20260709 12:18:38.049252 12493 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: [39.29-39.29]
I20260709 12:18:38.050079 12490 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:38.054289 27915 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:18:38.040427 27878 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:18:38.065236 12568 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:38.063509 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5789) took 1116 ms. Trace:
I20260709 12:18:38.066071 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5784) took 1132 ms. Trace:
W20260709 12:18:38.063563 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 113682us
I20260709 12:18:38.066975 12568 rpcz_store.cc:276] 0709 12:18:36.933916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.061560 (+1127644us) service_pool.cc:224] Handling call
0709 12:18:38.063915 (+  2355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.065790 (+  1875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.066017 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.068586 27494 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:18:38.068742 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:38.066234 12565 rpcz_store.cc:276] 0709 12:18:36.949512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.061960 (+1112448us) service_pool.cc:224] Handling call
0709 12:18:38.063023 (+  1063us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.063261 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.063286 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.063330 (+    44us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:38.063351 (+    21us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.063387 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.063463 (+    76us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.069684 27914 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:38.069756 27494 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:38.070495 27914 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:38.071242 12489 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.27-44.27]
I20260709 12:18:38.071470 12565 pending_rounds.cc:85] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:18:38.071852 12565 pending_rounds.cc:107] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 39.22
I20260709 12:18:38.072109 12565 raft_consensus.cc:2889] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:38.070103 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5790) took 1117 ms. Trace:
I20260709 12:18:38.072526 12490 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:38.072909 12568 rpcz_store.cc:276] 0709 12:18:36.952861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.069245 (+1116384us) service_pool.cc:224] Handling call
0709 12:18:38.069559 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.069774 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.069802 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:38.069848 (+    46us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:38.069883 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.069929 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.070052 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:38.069826 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:38.084580 12637 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:38.031584 27897 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.086546 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5593) took 3322 ms. Trace:
I20260709 12:18:38.087394 12641 rpcz_store.cc:276] 0709 12:18:34.764429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.138124 (+373695us) service_pool.cc:224] Handling call
0709 12:18:35.138571 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.032651 (+894080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.086316 (+2053665us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:38.086482 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2052371712}
I20260709 12:18:38.065917 27802 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:38.090474 12637 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:38.072225 27878 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:38.020790 27690 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 46.28, Last appended by leader: 28, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.094128 12490 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:18:38.094846 12491 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:38.083841 27898 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.013s
I20260709 12:18:38.098049 27897 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.101786 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5779) took 1169 ms. Trace:
I20260709 12:18:38.102325 12572 rpcz_store.cc:276] 0709 12:18:36.932374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.863729 (+931355us) service_pool.cc:224] Handling call
0709 12:18:37.889640 (+ 25911us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:37.891081 (+  1441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.891131 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:37.891818 (+   687us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.009442 (+117624us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:38.020364 (+ 10922us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.020654 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.020746 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:38.101497 (+ 80751us) raft_consensus.cc:1685] finished
0709 12:18:38.101521 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.101711 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":582,"prepare.queue_time_us":10627,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":2545,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":144,"spinlock_wait_cycles":16776448,"thread_start_us":117207,"threads_started":1,"wal-append.queue_time_us":162746}
W20260709 12:18:38.104094 27917 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:18:38.104557 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:38.105690 27919 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:18:38.087234 27494 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:38.107983 27912 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 46 is higher.
I20260709 12:18:38.107689 27494 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 50.26, Last appended by leader: 26, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.109124 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.109647 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.112314 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.112972 12637 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:38.118700 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.082854 27783 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:38.135782 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.109870 27802 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:38.130223 27623 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:38.158490 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78030.7us
I20260709 12:18:38.106711 27878 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.139668 27783 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:38.153664 27876 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:38.177184 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.179884 12640 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 43 index: 27. (index mismatch)
I20260709 12:18:38.182530 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.183341 12642 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 41 index: 23. (index mismatch)
I20260709 12:18:38.184731 13260 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 42 index: 28. (index mismatch)
I20260709 12:18:38.186638 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.187568 12645 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:18:38.190295 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.191094 12645 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:38.190949 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.192718 27660 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:38.193150 27660 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:38.193526 27660 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:38.193661 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.194993 27779 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:38.197610 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:38.199831 13255 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48.
I20260709 12:18:38.199739 12638 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:38.201854 12642 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [45.27-45.27]
I20260709 12:18:38.200826 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.204619 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.205431 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.180603 27746 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.185076 27711 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:38.207695 12642 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:38.186261 27878 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:38.210933 12641 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:38.185253 27792 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.133294 27914 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 43.22, Last appended by leader: 22, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.211959 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.212805 27876 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:38.218072 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.219460 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.219684 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:38.220233 12643 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:38.221117 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.222522 13254 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 43 index: 30. (index mismatch)
I20260709 12:18:38.223548 12611 leader_election.cc:400] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48.
I20260709 12:18:38.223879 12611 leader_election.cc:403] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Cancelling election due to peer responding with higher term
I20260709 12:18:38.224052 13261 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 28. Preceding OpId from leader: term: 45 index: 29. (index mismatch)
I20260709 12:18:38.224429 27216 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:38.225180 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:38.225988 13255 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:38.216842 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.227535 12638 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:38.230135 27898 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.230719 27898 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.231192 27925 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.231917 27925 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.233014 27898 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:38.233399 27216 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 47. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48.
I20260709 12:18:38.234447 12643 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:38.234891 27925 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:38.234668 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.237425 12642 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:38.237946 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.238569 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.240235 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.243044 12645 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:38.236229 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.210929 27802 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.27, Last appended by leader: 27, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.212805 27746 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.258261 13256 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: [47.30-47.30]
I20260709 12:18:38.261253 12643 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:38.240870 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:38.258014 27882 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.263536 27811 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.264300 12638 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:38.264535 27756 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.212637 27711 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.271237 27914 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:38.265405 27756 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.262326 27216 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.017s
W20260709 12:18:38.277784 12611 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f402dfd68ae4c8697765a76d9f1c4e6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:38.261602 27597 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:38.261513 27560 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.299425 27560 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:38.305692 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.308054 12642 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:18:38.311280 27893 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.828s	user 0.001s	sys 0.007s
I20260709 12:18:38.313490 27753 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:38.314280 27753 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:38.314824 27711 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.267423 27623 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.264348 27811 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.315865 27753 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 39.29, Last appended by leader: 29, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:38.320936 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 621524us
W20260709 12:18:38.327176 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.289091 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4393) took 1823 ms. Trace:
I20260709 12:18:38.328440 27881 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:38.329192 27560 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:38.267742 27783 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.258558 27792 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.332621 27925 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:38.333809 27684 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:38.334247 27684 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:38.334839 27792 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.335505 27705 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:38.335191 27684 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.336027 27705 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:38.339946 27684 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:38.328130 12496 rpcz_store.cc:276] 0709 12:18:36.465324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.466271 (+   947us) service_pool.cc:224] Handling call
0709 12:18:36.466679 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:36.671002 (+204323us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:36.790626 (+119624us) log.cc:844] Serialized 14 byte log entry
0709 12:18:36.796638 (+  6012us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:36.796697 (+    59us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:36.796749 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.288748 (+1491999us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b540020dad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:38.288813 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.289010 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":126668,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":1968,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":5198,"spinlock_wait_cycles":469051776,"thread_start_us":119315,"threads_started":2,"wal-append.queue_time_us":3727,"wal-append.run_cpu_time_us":2185,"wal-append.run_wall_time_us":1157582}
I20260709 12:18:38.296066 27734 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:38.287076 27769 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:38.323561 27827 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 50.26, Last appended by leader: 26, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:37.239981 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 269752us
I20260709 12:18:38.292874 27809 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:38.345469 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5601) took 3542 ms. Trace:
I20260709 12:18:38.373936 12640 rpcz_store.cc:276] 0709 12:18:34.803121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.189880 (+386759us) service_pool.cc:224] Handling call
0709 12:18:35.218018 (+ 28138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.180399 (+2962381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.345177 (+164778us) spinlock_profiling.cc:243] Waited 2.96 s on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:38.345389 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6810679040}
I20260709 12:18:38.345592 12637 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:38.376062 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:38.357717 13259 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:38.376441 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:38.377534 13253 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 52: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:38.378158 12645 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
W20260709 12:18:38.343392 27894 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.823s	user 0.003s	sys 0.006s
I20260709 12:18:38.328184 27756 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:38.346822 27753 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:38.354696 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145988us
I20260709 12:18:38.357882 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:38.380690 13262 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:38.364881 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:38.355255 27931 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:18:38.340248 27913 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:38.370504 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.398793 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.385730 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:38.385730 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:38.400774 13255 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:38.387959 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:38.372387 27891 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.853s	user 0.000s	sys 0.004s
I20260709 12:18:38.393257 12496 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:38.337137 27705 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.402630 12496 raft_consensus.cc:606] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:38.414485 13261 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 38: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:38.408361 27711 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:38.409099 27833 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:38.412020 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6107) took 1277 ms. Trace:
I20260709 12:18:38.386101 27827 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:18:38.423590 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 138744us
I20260709 12:18:38.385984 27560 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:38.424546 12571 rpcz_store.cc:276] 0709 12:18:37.134344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.147655 (+1013311us) service_pool.cc:224] Handling call
0709 12:18:38.148246 (+   591us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.149125 (+   879us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.149161 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:38.149553 (+   392us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.258359 (+108806us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:38.258405 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.258512 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:38.411703 (+153191us) raft_consensus.cc:1685] finished
0709 12:18:38.411726 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.411927 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1778,"raft.queue_time_us":133,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":4408,"thread_start_us":95661,"threads_started":1,"wal-append.queue_time_us":125459}
I20260709 12:18:38.427145 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:38.427603 12496 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:38.428833 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:38.429919 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:38.433849 12492 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: [42.22-42.22]
I20260709 12:18:38.443343 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.443853 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.444134 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:38.445663 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.447125 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.448691 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.449443 12642 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:38.434556 27560 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:38.434818 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:38.437523 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:38.455122 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.413040 27756 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:38.458379 12491 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 50 index: 28. (index mismatch)
I20260709 12:18:38.458194 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:38.458664 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:38.459295 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:38.413656 27216 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:38.464219 12494 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.24-40.24]
I20260709 12:18:38.464982 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5970) took 1311 ms. Trace:
I20260709 12:18:38.478328 12495 rpcz_store.cc:276] 0709 12:18:37.154029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.247093 (+ 93064us) service_pool.cc:224] Handling call
0709 12:18:37.247691 (+   598us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:37.248067 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:37.248102 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:37.258676 (+ 10574us) log.cc:844] Serialized 506 byte log entry
0709 12:18:37.381213 (+122537us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:37.381259 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:37.381338 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:38.308665 (+927327us) spinlock_profiling.cc:243] Waited 45.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:38.432625 (+123960us) raft_consensus.cc:1685] finished
0709 12:18:38.432644 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:38.464881 (+ 32237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103951744,"thread_start_us":122220,"threads_started":1,"wal-append.queue_time_us":202212}
I20260709 12:18:38.465253 12489 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:38.468928 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.487301 12571 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:38.488921 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.490058 12571 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:38.491237 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.492722 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.470201 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:38.392076 27494 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.044s
I20260709 12:18:38.494534 12638 raft_consensus.cc:2410] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 51 because replica has last-logged OpId of term: 50 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 21.
I20260709 12:18:38.494616 27494 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:38.493428 12571 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:38.437809 27705 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:38.512240 12642 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
W20260709 12:18:38.514989 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.507368 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2901) took 2678 ms. Trace:
I20260709 12:18:38.517555 12566 rpcz_store.cc:276] 0709 12:18:35.829321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.778145 (+1948824us) service_pool.cc:224] Handling call
0709 12:18:37.820554 (+ 42409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.831905 (+ 11351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.507151 (+675246us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54001a36d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:38.507305 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96496000}
I20260709 12:18:38.518975 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:38.522090 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105138us
W20260709 12:18:38.524338 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.527331 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:38.460997 27188 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:38.540592 13255 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
W20260709 12:18:38.531944 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110968us
I20260709 12:18:38.545262 12571 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:38.500869 27779 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.579959 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.579865 27779 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.581277 12571 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 50 because replica has last-logged OpId of term: 49 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 20.
I20260709 12:18:38.582422 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.583062 27779 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:38.583758 27859 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:38.572810 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.587458 27216 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:38.601063 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.510937 27431 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:38.437877 27711 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.506752 27882 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:38.607688 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:38.751436 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.507143 27811 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:38.524694 27792 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:38.510371 27520 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:38.454960 27833 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.631181 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.500648 27872 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.099s
W20260709 12:18:38.730083 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 275064us
W20260709 12:18:38.739184 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:38.756600 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:38.722926 27660 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 12:18:38.772753 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.794086 12571 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:38.775624 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 252365us
W20260709 12:18:38.787115 27937 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20260709 12:18:38.748333 27216 consensus_queue.cc:1243] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 48.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:18:38.757376 27792 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:38.803764 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 276523us
W20260709 12:18:38.804152 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 247785us
I20260709 12:18:38.813704 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:38.806667 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.815104 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.815917 12566 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 50 because replica has last-logged OpId of term: 49 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 20.
I20260709 12:18:38.610512 27940 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.189s
I20260709 12:18:38.821553 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:38.822285 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:38.823652 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.831220 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.833338 12566 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:38.827099 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.836059 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.839061 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:38.840137 12566 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 44 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 28.
I20260709 12:18:38.844623 12571 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:38.851056 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:38.852362 12566 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:18:38.856886 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:38.803056 27934 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:38.730443 27859 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.777366 27623 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.061s
I20260709 12:18:38.764261 27882 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.818663 27792 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:38.769682 27520 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.832863 27779 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:38.850373 27878 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.611s	user 0.006s	sys 0.006s
I20260709 12:18:38.792939 27660 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.848915 27379 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:38.790824 27945 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.102s
W20260709 12:18:38.896602 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50960.4us
I20260709 12:18:38.915182 12537 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:38.894141 27944 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:38.850742 27911 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.055s
I20260709 12:18:38.920218 27865 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:38.920990 27865 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.923447 27865 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:38.923532 27934 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:38.924276 27934 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:38.924386 27959 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.927546 27943 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.928282 27943 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.930018 27811 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:38.930748 27811 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:38.931203 27942 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:18:38.932803 27859 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.933593 27911 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:38.934341 27833 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:38.935536 27935 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.321s	user 0.001s	sys 0.000s
I20260709 12:18:38.925092 27959 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.939044 27959 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:38.939294 27914 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.667s	user 0.010s	sys 0.000s
I20260709 12:18:38.939831 27811 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:38.940618 27941 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:18:38.940889 27943 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:38.942389 27520 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.907634 27872 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:38.944916 27872 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:38.945957 27872 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:38.946866 27926 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.770s	user 0.000s	sys 0.011s
I20260709 12:18:38.950407 27882 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.951905 27945 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:38.952425 27945 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:38.961404 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 46373.3us
I20260709 12:18:38.908263 27612 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.066s
I20260709 12:18:38.974054 27882 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:18:38.979372 27753 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.001s
I20260709 12:18:38.981809 27494 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:38.986927 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 173362us
I20260709 12:18:38.987217 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:38.994720 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:38.996671 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.000507 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:39.003618 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:39.009063 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:38.983243 27882 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:38.983258 27881 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.016485 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:39.019114 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:39.034687 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.017205 27952 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:39.006188 27943 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:38.999626 27833 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.041s
W20260709 12:18:39.043253 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:39.069185 27853 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 2.404s	user 0.009s	sys 0.000s
I20260709 12:18:39.009219 27865 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:39.017444 27882 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:39.069903 27865 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:39.070403 27865 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:39.070843 27782 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:18:39.071288 27782 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:39.072175 27782 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 52.34, Last appended by leader: 34, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:38.953347 27945 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.048908 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.073661 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5610) took 2974 ms. Trace:
I20260709 12:18:39.074002 12643 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:39.074055 12646 rpcz_store.cc:276] 0709 12:18:36.099358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.128470 (+ 29112us) service_pool.cc:224] Handling call
0709 12:18:36.128896 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:36.129326 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:36.129354 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:36.149963 (+ 20609us) log.cc:844] Serialized 26 byte log entry
0709 12:18:36.417688 (+267725us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:36.417758 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:36.417867 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.068309 (+2650442us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:39.073229 (+  4920us) raft_consensus.cc:1685] finished
0709 12:18:39.073256 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.073561 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":133,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":141,"spinlock_wait_cycles":28993280,"thread_start_us":206336,"threads_started":1,"wal-append.queue_time_us":437386}
W20260709 12:18:39.039252 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 203271us
I20260709 12:18:39.000607 27520 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.039s
I20260709 12:18:39.046020 27952 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:39.047614 27948 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.016s
I20260709 12:18:39.060253 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.081578 12642 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:39.062553 27797 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:38.925464 27934 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 43.22, Last appended by leader: 22, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:39.086740 27705 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.553s	user 0.012s	sys 0.000s
I20260709 12:18:39.043939 27711 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.088083 27711 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:39.088462 27711 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:39.088522 27948 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.088874 27711 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:39.089257 27948 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.065691 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.009478 27949 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:39.077395 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:39.096077 12642 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 48 index: 29. (index mismatch)
I20260709 12:18:39.105788 12642 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:18:39.108022 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.039042 27881 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:39.139678 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.142746 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.144316 12642 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:39.121840 27684 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.781s	user 0.012s	sys 0.005s
I20260709 12:18:39.150722 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:39.181684 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.185097 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:39.193945 12496 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:39.194981 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:39.197665 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:39.199775 13217 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:39.207613 12495 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 20. Preceding OpId from leader: term: 50 index: 21. (index mismatch)
I20260709 12:18:39.209940 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:39.216887 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:39.215324 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:39.218729 12496 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
W20260709 12:18:39.219573 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.149654 27705 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.220319 12493 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:39.186769 27753 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.183430 27914 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.207195 27858 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:39.250175 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:39.251129 12493 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 47 because replica has last-logged OpId of term: 45 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 26.
I20260709 12:18:39.252445 12493 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: [42.22-42.22]
I20260709 12:18:39.256176 27865 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.208056 27797 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:39.257769 27797 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:39.258457 27797 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:39.257851 27914 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.261207 27779 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:39.231662 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:39.267854 12495 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:39.268592 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.269359 12642 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:18:39.218158 27859 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:39.234417 27881 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.235168 27494 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:39.237854 27948 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.237999 27963 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:39.370453 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:39.239459 27431 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.510s	user 0.010s	sys 0.000s
I20260709 12:18:39.230698 27660 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.212054 27961 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:39.241765 12566 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:39.242597 27952 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:39.457780 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5632) took 1011 ms. Trace:
I20260709 12:18:39.458278 12641 rpcz_store.cc:276] 0709 12:18:38.445914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.446115 (+   201us) service_pool.cc:224] Handling call
0709 12:18:38.499688 (+ 53573us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.811091 (+311403us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:38.862508 (+ 51417us) log.cc:844] Serialized 14 byte log entry
0709 12:18:39.112134 (+249626us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:39.140638 (+ 28504us) log.cc:844] Serialized 26 byte log entry
0709 12:18:39.140955 (+   317us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:39.141027 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.141182 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.457496 (+316314us) raft_consensus.cc:1685] finished
0709 12:18:39.457520 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.457711 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33803,"prepare.queue_time_us":50741,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":5241,"raft.queue_time_us":86146,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":20324,"thread_start_us":513688,"threads_started":2,"wal-append.queue_time_us":318667}
I20260709 12:18:39.460711 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.461833 27948 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.462446 27948 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.464776 27948 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.465323 27948 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.244773 27953 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:39.473986 27953 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:39.475188 27953 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.476660 27953 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:39.245996 27756 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.746s	user 0.012s	sys 0.001s
I20260709 12:18:39.477934 27756 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:39.247257 27792 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.478173 27963 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.231236 27705 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.480549 27963 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:39.248386 27958 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 12:18:39.483047 27865 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:39.255981 27882 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.484207 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5631) took 1038 ms. Trace:
I20260709 12:18:39.484579 27847 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:18:39.485167 27847 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:39.486105 27882 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:39.261889 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.207545 27878 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.487818 27878 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.484834 12640 rpcz_store.cc:276] 0709 12:18:38.445464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.445604 (+   140us) service_pool.cc:224] Handling call
0709 12:18:38.446011 (+   407us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:38.547594 (+101583us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:18:38.774658 (+227064us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.907595 (+132937us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:38.908730 (+  1135us) log.cc:844] Serialized 78 byte log entry
0709 12:18:38.908952 (+   222us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:38.958379 (+ 49427us) log.cc:844] Serialized 14 byte log entry
0709 12:18:38.958656 (+   277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.960041 (+  1385us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b540018c4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:38.960111 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:39.248029 (+287918us) spinlock_profiling.cc:243] Waited 7.83 ms on lock 0x7b24003e36a8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:18:39.483810 (+235781us) raft_consensus.cc:1685] finished
0709 12:18:39.483849 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:39.484117 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109895,"prepare.queue_time_us":155767,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":103453,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":49756,"spinlock_wait_cycles":214502272,"thread_start_us":257746,"threads_started":2,"wal-append.queue_time_us":160541}
I20260709 12:18:39.223100 12494 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 28. Preceding OpId from leader: term: 43 index: 30. (index mismatch)
I20260709 12:18:39.230280 27753 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.494164 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5792) took 1094 ms. Trace:
I20260709 12:18:39.496166 12566 rpcz_store.cc:276] 0709 12:18:38.399932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.850691 (+450759us) service_pool.cc:224] Handling call
0709 12:18:39.494097 (+643406us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.498658 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:39.499481 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:39.500128 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5796) took 1082 ms. Trace:
I20260709 12:18:39.500553 12566 rpcz_store.cc:276] 0709 12:18:38.418186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.498375 (+1080189us) service_pool.cc:224] Handling call
0709 12:18:39.500059 (+  1684us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.502564 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:39.504812 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:39.505692 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:39.506294 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5810) took 1005 ms. Trace:
I20260709 12:18:39.506659 12566 rpcz_store.cc:276] 0709 12:18:38.501040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.504461 (+1003421us) service_pool.cc:224] Handling call
0709 12:18:39.506239 (+  1778us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.480466 27792 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.401690 27494 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.422497 27779 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:18:39.453444 27379 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.000s
W20260709 12:18:39.243578 27827 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.807s	user 0.013s	sys 0.001s
I20260709 12:18:39.455919 27872 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:39.451721 27188 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.907s	user 0.012s	sys 0.000s
W20260709 12:18:39.275884 27936 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.508s	user 0.004s	sys 0.001s
I20260709 12:18:39.485780 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:39.491295 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.271469 27914 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.507040 27961 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:39.492727 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.508603 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:39.495285 27878 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.473209 27797 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.550596 27872 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.562788 27797 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.564152 27872 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.550321 27188 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:39.565582 27188 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:39.566198 27188 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.566789 27961 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.550078 27731 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.568699 27731 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.570122 12492 raft_consensus.cc:606] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:39.570744 12492 raft_consensus.cc:614] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:39.572094 12497 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:39.270500 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.423146 27954 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.421145 27520 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:39.594040 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.494722 27948 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.486845 27847 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 51.30, Last appended by leader: 30, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.263010 12493 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: [47.30-47.30]
I20260709 12:18:39.530791 27379 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:39.554791 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2909) took 1092 ms. Trace:
I20260709 12:18:39.556423 27957 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.551966 27623 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.552140 27911 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.558421 27860 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:39.558907 27753 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.559720 27494 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.559800 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4415) took 1885 ms. Trace:
I20260709 12:18:39.531369 27779 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:39.588258 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.507927 27772 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.549181 27792 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.529958 27660 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.551884 27431 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.601931 12566 rpcz_store.cc:276] 0709 12:18:38.462964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.508308 (+1045344us) service_pool.cc:224] Handling call
0709 12:18:39.554720 (+ 46412us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:39.603197 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.013s	sys 0.017s
W20260709 12:18:39.623473 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.013s	sys 0.017s
I20260709 12:18:39.610718 12488 rpcz_store.cc:276] 0709 12:18:37.674302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.681987 (+  7685us) service_pool.cc:224] Handling call
0709 12:18:37.682439 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.924684 (+242245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.559488 (+1634804us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:39.559723 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":555455872}
W20260709 12:18:39.606858 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 384500us
W20260709 12:18:39.612738 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:39.608184 27711 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.519s	user 0.014s	sys 0.000s
I20260709 12:18:39.605238 27847 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:18:39.629338 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 205516us
I20260709 12:18:39.602911 27705 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:39.604730 27973 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.610116 27860 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:39.626422 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:39.630256 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:39.614384 27779 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:39.633873 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:39.631376 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.634320 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:39.636066 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 419142us
I20260709 12:18:39.638867 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5806) took 1204 ms. Trace:
I20260709 12:18:39.597661 27954 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.642066 12566 rpcz_store.cc:276] 0709 12:18:38.437040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.629905 (+1192865us) service_pool.cc:224] Handling call
0709 12:18:39.638792 (+  8887us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:39.645357 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.646064 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:39.640141 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:39.623126 27188 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.649161 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:39.653488 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.655503 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:39.608826 27623 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:39.666544 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.628687 27494 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:39.671669 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.611570 27911 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.672248 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.607311 27957 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.670651 27753 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:39.717746 12640 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: [52.34-52.34]
I20260709 12:18:39.670786 27859 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.732769 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:39.675611 27494 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.676715 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.675844 27860 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.28, Last appended by leader: 28, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.689477 27948 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.635689 27379 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.629423 27431 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.620489 27772 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.699293 27978 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.668483 27976 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.622171 27792 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.634986 27711 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.717746 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:39.641219 27705 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:39.763829 27705 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:39.625730 27660 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.766237 27792 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.767596 27954 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.767405 12566 raft_consensus.cc:2368] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43.
I20260709 12:18:39.738933 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:39.741700 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.672747 27827 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:39.749008 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.707370 27952 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:39.718600 27753 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:39.774564 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.763972 27976 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.764468 27911 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.639833 27973 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.775297 12646 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:39.768227 27711 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.767609 27978 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.781330 12642 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:39.763880 27431 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.772567 27860 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:39.790119 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:39.779752 27753 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:39.746462 27859 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.792430 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.759266 27379 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:39.803946 27623 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.804605 27494 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.812700 27976 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.816167 27973 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.820066 27859 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:39.820287 27975 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.181s	user 0.003s	sys 0.000s
I20260709 12:18:39.820716 27957 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.821612 27978 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:39.824360 27846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.157s	user 0.011s	sys 0.003s
I20260709 12:18:39.952085 27859 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.799906 27660 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:39.896741 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:39.874143 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.919422 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.915480 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.917915 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:39.908339 12609 leader_election.cc:336] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.970s (SENT)
I20260709 12:18:39.962287 27957 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:39.964260 27957 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:39.950910 27494 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:39.798686 27705 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.972244 27957 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:39.971647 27846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.309s	user 0.011s	sys 0.005s
I20260709 12:18:39.974802 27846 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:39.975672 27859 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.971664 27494 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:39.979288 27859 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:39.980232 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:39.980834 27494 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:39.981870 27379 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:39.987347 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 108) took 1529 ms. Trace:
I20260709 12:18:39.996680 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6089) took 4286 ms. Trace:
I20260709 12:18:39.997485 12563 rpcz_store.cc:276] 0709 12:18:35.710039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.733218 (+1023179us) service_pool.cc:224] Handling call
0709 12:18:36.774140 (+ 40922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.441914 (+667774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:39.995747 (+2553833us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:39.996317 (+   570us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1388144000}
I20260709 12:18:39.999231 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5813) took 1462 ms. Trace:
I20260709 12:18:40.000257 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:39.988003 12492 rpcz_store.cc:276] 0709 12:18:38.458201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.459097 (+   896us) service_pool.cc:224] Handling call
0709 12:18:39.986820 (+1527723us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:39.987059 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2543089408}
I20260709 12:18:40.001677 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:40.003654 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5788) took 3060 ms (client timeout 1797 ms). Trace:
W20260709 12:18:40.001600 12567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6075) took 4399 ms (client timeout 1813 ms). Trace:
I20260709 12:18:39.999998 12566 rpcz_store.cc:276] 0709 12:18:38.536601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.645727 (+1109126us) service_pool.cc:224] Handling call
0709 12:18:39.998470 (+352743us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:39.998862 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277205888}
W20260709 12:18:40.008582 12569 rpcz_store.cc:269] 0709 12:18:36.942473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.412618 (+470145us) service_pool.cc:224] Handling call
0709 12:18:40.002302 (+2589684us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.002671 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23224704}
W20260709 12:18:40.010824 12567 rpcz_store.cc:269] 0709 12:18:35.601954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.602092 (+   138us) service_pool.cc:224] Handling call
0709 12:18:40.000903 (+4398811us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.001218 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5814576000}
I20260709 12:18:40.016110 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.019021 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.015911 12568 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: [47.28-47.28]
I20260709 12:18:40.019929 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5817) took 1254 ms. Trace:
I20260709 12:18:40.020409 12566 rpcz_store.cc:276] 0709 12:18:38.765677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.015599 (+1249922us) service_pool.cc:224] Handling call
0709 12:18:40.019820 (+  4221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:39.977079 27846 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.017803 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.021466 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.023113 12569 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 50
I20260709 12:18:40.023962 27379 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:40.026654 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:40.036674 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.040337 12646 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:40.023152 12567 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38. Re-sending same reply.
I20260709 12:18:40.025219 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.041963 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.026911 27379 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.031216 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.042842 27216 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:40.045612 27216 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:40.048461 27216 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.052412 27746 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.052790 12537 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:40.053996 27978 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.056064 12566 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:40.057085 27216 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:40.045957 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5814) took 1431 ms. Trace:
I20260709 12:18:40.061924 12567 rpcz_store.cc:276] 0709 12:18:38.612619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.021054 (+1408435us) service_pool.cc:224] Handling call
0709 12:18:40.043944 (+ 22890us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.064376 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5809) took 1599 ms. Trace:
I20260709 12:18:40.069613 27597 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:40.068744 12566 rpcz_store.cc:276] 0709 12:18:38.462535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.024702 (+1562167us) service_pool.cc:224] Handling call
0709 12:18:40.061456 (+ 36754us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.067660 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.072897 13258 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:40.063179 27978 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.077353 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:40.080192 27772 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:40.080775 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.082266 13255 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 52 because replica has last-logged OpId of term: 50 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 28.
I20260709 12:18:40.082020 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:40.082587 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.083724 13256 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:18:40.088430 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.091913 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.091820 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.092857 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.090348 27731 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:40.094869 12566 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 43 because replica has last-logged OpId of term: 42 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:40.095733 27978 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.098362 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5815) took 1375 ms. Trace:
I20260709 12:18:40.098747 12566 rpcz_store.cc:276] 0709 12:18:38.723554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.078464 (+1354910us) service_pool.cc:224] Handling call
0709 12:18:40.098302 (+ 19838us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.099603 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.100682 13260 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:40.096812 12567 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:40.107265 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.106176 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.076741 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 168863us
I20260709 12:18:40.105477 13258 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.26-43.26]
I20260709 12:18:40.110801 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:40.095851 12492 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:40.113026 27925 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:40.137020 27765 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:18:40.137856 27765 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:40.139408 27765 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 45.33, Last appended by leader: 33, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.127142 27925 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:40.142251 27925 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:40.145026 27597 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:40.146236 27597 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:40.146656 27597 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:40.147399 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5795) took 1729 ms (client timeout 1731 ms). Trace:
I20260709 12:18:40.147794 27597 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:40.148232 12571 rpcz_store.cc:269] 0709 12:18:38.417685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.630835 (+213150us) service_pool.cc:224] Handling call
0709 12:18:40.146890 (+1516055us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54001b2bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.147141 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93975296}
I20260709 12:18:40.149322 12567 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:40.149931 27597 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:40.151571 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.152660 27753 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.154917 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2168) took 3885 ms. Trace:
I20260709 12:18:40.173839 12644 rpcz_store.cc:276] 0709 12:18:36.269606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:36.673495 (+403889us) service_pool.cc:224] Handling call
0709 12:18:36.677383 (+  3888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.854667 (+1177284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.154047 (+2299380us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:40.154561 (+   514us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2700891904}
I20260709 12:18:40.150071 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5824) took 1152 ms. Trace:
I20260709 12:18:40.148720 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5774) took 3219 ms. Trace:
I20260709 12:18:40.160097 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.153507 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.164197 27973 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:40.131831 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.139768 12609 leader_election.cc:336] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 1.797s (SENT)
W20260709 12:18:40.170524 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.172726 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.176708 12567 rpcz_store.cc:276] 0709 12:18:38.998110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.104134 (+1106024us) service_pool.cc:224] Handling call
0709 12:18:40.150002 (+ 45868us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.175712 12570 rpcz_store.cc:276] 0709 12:18:36.929612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:37.832597 (+902985us) service_pool.cc:224] Handling call
0709 12:18:37.833155 (+   558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:37.854510 (+ 21355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.148122 (+2293612us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:40.148512 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46780672}
I20260709 12:18:40.180071 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:40.184195 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.184329 12492 raft_consensus.cc:606] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:40.185047 12492 raft_consensus.cc:614] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:18:40.187141 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 109654us
I20260709 12:18:40.165031 27560 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 47.30, Last appended by leader: 30, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.190717 27560 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:40.191771 27833 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:40.192409 27833 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:40.196679 27910 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:40.198218 13260 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:40.199385 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:40.180043 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5816) took 1430 ms. Trace:
I20260709 12:18:40.201288 12563 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:40.201920 12566 rpcz_store.cc:276] 0709 12:18:38.749699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.105732 (+1356033us) service_pool.cc:224] Handling call
0709 12:18:40.179890 (+ 74158us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.204916 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5791) took 1802 ms (client timeout 2146 ms). Trace:
W20260709 12:18:40.205540 12563 rpcz_store.cc:269] 0709 12:18:38.399600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.000903 (+1601303us) service_pool.cc:224] Handling call
0709 12:18:40.202141 (+201238us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.204598 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5802) took 1779 ms (client timeout 2145 ms). Trace:
I20260709 12:18:40.207093 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.211841 27847 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.019s
W20260709 12:18:40.213745 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.207221 12572 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:40.226552 12567 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:40.220427 13256 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [41.27-43.28]
I20260709 12:18:40.220310 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:40.224485 13258 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 49 because replica has last-logged OpId of term: 47 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 28.
I20260709 12:18:40.204378 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:40.226022 12569 rpcz_store.cc:269] 0709 12:18:38.421464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.018477 (+1597013us) service_pool.cc:224] Handling call
0709 12:18:40.200244 (+181767us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.225715 27379 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.264395 12537 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:40.265733 27973 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:40.266031 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.266265 27973 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.266749 27973 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:40.267110 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5820) took 1404 ms. Trace:
I20260709 12:18:40.267590 12570 rpcz_store.cc:276] 0709 12:18:38.862599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.243114 (+1380515us) service_pool.cc:224] Handling call
0709 12:18:40.267020 (+ 23906us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.267319 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.268906 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:40.269088 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.269757 27802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.984s	user 0.019s	sys 0.018s
I20260709 12:18:40.270267 27833 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.28, Last appended by leader: 28, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.271814 27833 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:40.271965 13256 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:40.273725 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.274260 13256 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 48 index: 29. (index mismatch)
I20260709 12:18:40.274712 13254 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:40.275988 13256 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 29. Preceding OpId from leader: term: 40 index: 29. (term mismatch)
I20260709 12:18:40.276448 13256 pending_rounds.cc:85] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 28
I20260709 12:18:40.280951 13256 pending_rounds.cc:107] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 39.29
I20260709 12:18:40.281613 13256 raft_consensus.cc:2889] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:40.283205 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:40.283661 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.284847 27860 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:40.285305 27860 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.285753 27860 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:40.286734 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:40.247103 27847 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.288062 27847 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.289032 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:40.278435 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 138824us
W20260709 12:18:40.270257 27802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.022s	user 0.019s	sys 0.018s
I20260709 12:18:40.290823 27847 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.279913 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:40.280298 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.291224 27792 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:40.291756 13258 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:40.292043 13254 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:40.292268 27792 consensus_queue.cc:1243] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 44.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 25
I20260709 12:18:40.293262 13257 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:40.293309 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.294616 13254 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:40.296123 13254 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 48 index: 30. (index mismatch)
I20260709 12:18:40.280853 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 48 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.285559 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.290189 27711 raft_consensus.cc:2764] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
W20260709 12:18:40.293458 27779 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.619s	user 0.004s	sys 0.023s
I20260709 12:18:40.298707 12566 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 51 because replica has last-logged OpId of term: 49 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 20.
I20260709 12:18:40.289198 26496 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:40.299149 27779 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.300122 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.300925 12566 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:40.300674 27925 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:40.301599 27779 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.303516 12571 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:40.305800 27901 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.306495 27901 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.308936 27901 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.310045 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:18:40.313601 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:40.314383 12462 connection.cc:256] RPC call timeout handler was delayed by -1.4276571769999009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260709 12:18:40.290465 12609 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.007s (SENT)
I20260709 12:18:40.318632 26496 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.319403 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:40.277614 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.276504 12643 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 30. Preceding OpId from leader: term: 48 index: 30. (term mismatch)
I20260709 12:18:40.319599 27753 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.319765 26496 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:40.328128 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.294265 27734 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:40.321687 12643 pending_rounds.cc:85] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:18:40.330291 12643 pending_rounds.cc:107] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 47.30
I20260709 12:18:40.330665 12643 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:40.329770 26496 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.336688 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.337942 13254 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:40.341253 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.230458 27910 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.342981 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:40.342597 26496 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:18:40.345993 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68611.5us
I20260709 12:18:40.346069 13260 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:40.335191 27560 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:18:40.362161 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.363333 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:40.372270 13258 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:40.373888 26496 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.387588 26496 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.372006 27847 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:40.390612 27847 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:40.391142 27847 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:40.389271 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.396422 27690 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:40.397318 27690 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.400309 27690 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.401995 13262 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: [42.28-42.28]
I20260709 12:18:40.410339 27901 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:40.423792 26496 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.424921 12566 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 46 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 26.
I20260709 12:18:40.431788 27910 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.361850 27860 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:40.436467 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147636us
I20260709 12:18:40.437820 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.439272 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.440433 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.440620 27782 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:40.423867 27901 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.445618 27912 raft_consensus.cc:2764] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election decision vote started in defunct term 39: won
I20260709 12:18:40.447028 27901 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.447871 26496 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:40.448341 26496 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.448760 26496 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:40.449496 27928 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.234s	user 0.012s	sys 0.017s
I20260709 12:18:40.451357 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:40.452566 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:40.454339 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:40.456471 12492 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 54 because replica has last-logged OpId of term: 52 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 33.
I20260709 12:18:40.456787 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5783) took 3523 ms. Trace:
I20260709 12:18:40.457175 12564 rpcz_store.cc:276] 0709 12:18:36.933583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.030962 (+1097379us) service_pool.cc:224] Handling call
0709 12:18:38.031524 (+   562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.035159 (+  3635us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.035210 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:38.072481 (+ 37271us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.186417 (+113936us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:38.186465 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.186547 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:40.448458 (+2261911us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:40.456305 (+  7847us) raft_consensus.cc:1685] finished
0709 12:18:40.456326 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.456605 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":914,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":3064,"spinlock_wait_cycles":97264000,"thread_start_us":93770,"threads_started":1,"wal-append.queue_time_us":126338}
I20260709 12:18:40.459143 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:40.459343 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.460353 12564 raft_consensus.cc:2376] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43. Re-sending same reply.
W20260709 12:18:40.462296 27932 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.246s	user 0.012s	sys 0.011s
I20260709 12:18:40.467214 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.480202 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.471781 27925 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.473819 27597 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:40.477114 27945 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:40.468698 27940 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
W20260709 12:18:40.482566 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.482913 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:40.493309 27910 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:40.493870 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6105) took 3359 ms. Trace:
I20260709 12:18:40.500665 27860 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.471930 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5821) took 1584 ms. Trace:
I20260709 12:18:40.522470 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.524544 12571 rpcz_store.cc:276] 0709 12:18:38.887187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.149546 (+1262359us) service_pool.cc:224] Handling call
0709 12:18:40.471620 (+322074us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54003ab954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.471828 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279615872}
I20260709 12:18:40.524153 12564 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:18:40.528798 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.509083 27940 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:40.539299 12569 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 47 because replica has last-logged OpId of term: 45 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 26.
I20260709 12:18:40.511504 27910 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:40.542148 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5826) took 1543 ms. Trace:
W20260709 12:18:40.542021 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59412us
I20260709 12:18:40.544495 12569 rpcz_store.cc:276] 0709 12:18:38.998804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.479734 (+1480930us) service_pool.cc:224] Handling call
0709 12:18:40.542057 (+ 62323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.520020 12565 rpcz_store.cc:276] 0709 12:18:37.133837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.070449 (+936612us) service_pool.cc:224] Handling call
0709 12:18:38.071247 (+   798us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:38.076111 (+  4864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:38.076152 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:38.081039 (+  4887us) log.cc:844] Serialized 26 byte log entry
0709 12:18:38.211589 (+130550us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:38.211640 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:38.211727 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:40.461065 (+2249338us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:40.492989 (+ 31924us) raft_consensus.cc:1685] finished
0709 12:18:40.493058 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.493620 (+   562us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64658304,"thread_start_us":102564,"threads_started":1,"wal-append.queue_time_us":129267}
I20260709 12:18:40.550987 27684 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.551234 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:40.552032 27684 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.556574 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.556943 27684 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.557780 27216 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.558135 27825 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:40.558741 27825 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:40.559342 27825 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:40.553005 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.558317 12609 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.094s (SENT)
I20260709 12:18:40.545831 27910 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:40.568167 27765 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:40.568910 27765 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:40.569489 27765 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:40.561909 27560 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:40.571259 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4501) took 2353 ms. Trace:
I20260709 12:18:40.578173 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6106) took 3444 ms. Trace:
I20260709 12:18:40.578770 12568 rpcz_store.cc:276] 0709 12:18:37.134091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.074067 (+939976us) service_pool.cc:224] Handling call
0709 12:18:40.008434 (+1934367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:40.020429 (+ 11995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.577870 (+557441us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001d7250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.578081 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4435341056}
W20260709 12:18:40.580868 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.579890 12565 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:40.572482 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:40.584247 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.560694 27940 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.572737 12571 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:40.572422 12645 rpcz_store.cc:276] 0709 12:18:38.217985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.243910 (+ 25925us) service_pool.cc:224] Handling call
0709 12:18:38.377826 (+133916us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:38.378637 (+   811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.570818 (+2192181us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54002e5250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.571029 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307170176}
I20260709 12:18:40.594486 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2907) took 3153 ms. Trace:
I20260709 12:18:40.595316 12565 rpcz_store.cc:276] 0709 12:18:37.436748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.567190 (+3130442us) service_pool.cc:224] Handling call
0709 12:18:40.576955 (+  9765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.589801 (+ 12846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.590476 (+   675us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.593721 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.202s (SENT) [suppressed 27 similar messages]
I20260709 12:18:40.558741 27216 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.596071 27858 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.599169 27858 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.599803 27734 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:40.600857 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4490) took 2418 ms (client timeout 2374 ms). Trace:
I20260709 12:18:40.605489 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 5988) took 1398 ms. Trace:
I20260709 12:18:40.605960 12569 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:40.606537 27560 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.606146 12497 rpcz_store.cc:276] 0709 12:18:39.207245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.209551 (+  2306us) service_pool.cc:224] Handling call
0709 12:18:40.604997 (+1395446us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.605229 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830736768}
W20260709 12:18:40.605273 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.617205 26496 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.620918 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2905) took 3352 ms. Trace:
I20260709 12:18:40.615974 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:40.621670 12569 rpcz_store.cc:276] 0709 12:18:37.261767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.563565 (+3301798us) service_pool.cc:224] Handling call
0709 12:18:40.595132 (+ 31567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.613532 (+ 18400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.613780 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.593467 27684 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:18:40.613157 12639 rpcz_store.cc:269] 0709 12:18:38.181805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.182026 (+   221us) service_pool.cc:224] Handling call
0709 12:18:40.600218 (+2418192us) spinlock_profiling.cc:243] Waited 2.37 s on lock 0x7b5400153454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:40.600513 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5441902336}
I20260709 12:18:40.619581 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.594637 27846 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:40.634270 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83330.1us
I20260709 12:18:40.635108 26496 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:40.636477 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.611295 27560 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:40.637573 12564 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: [51.25-51.25]
I20260709 12:18:40.637923 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.639513 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:18:40.643337 13261 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: [47.30-47.30]
I20260709 12:18:40.600983 27734 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:40.646395 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88189.9us
I20260709 12:18:40.597506 27847 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.651305 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.659271 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.662266 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6109) took 3527 ms. Trace:
I20260709 12:18:40.657226 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.637856 28003 ts_tablet_manager.cc:933] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.663007 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5822) took 1756 ms. Trace:
W20260709 12:18:40.647691 12609 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Timed out: RequestConsensusVote RPC to 127.11.11.130:32927 timed out after 2.099s (SENT)
I20260709 12:18:40.663105 12564 rpcz_store.cc:276] 0709 12:18:37.134812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.562315 (+3427503us) service_pool.cc:224] Handling call
0709 12:18:40.612735 (+ 50420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:40.661917 (+ 49182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.662183 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.650288 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.654619 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 5983) took 1463 ms. Trace:
I20260709 12:18:40.649492 27216 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:40.668901 27792 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:40.671258 27965 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:40.672001 27965 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:40.673221 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6130) took 1179 ms. Trace:
I20260709 12:18:40.675165 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.684443 12567 rpcz_store.cc:276] 0709 12:18:39.493853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.649811 (+1155958us) service_pool.cc:224] Handling call
0709 12:18:40.673052 (+ 23241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:40.673241 27965 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.666635 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:40.674643 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.675105 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.676393 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5641) took 1669 ms. Trace:
I20260709 12:18:40.675196 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.656378 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5645) took 1540 ms. Trace:
I20260709 12:18:40.680051 27955 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:40.680341 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:40.666347 27940 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:40.655081 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:40.625236 27684 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.684751 12564 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:40.666308 12572 rpcz_store.cc:276] 0709 12:18:38.906413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.650874 (+1744461us) service_pool.cc:224] Handling call
0709 12:18:40.662935 (+ 12061us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:40.686049 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.674374 27858 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.685979 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:40.667574 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58195.8us
I20260709 12:18:40.700661 12643 rpcz_store.cc:276] 0709 12:18:39.007426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.076022 (+ 68596us) service_pool.cc:224] Handling call
0709 12:18:39.076354 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.361248 (+1284894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.676061 (+314813us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000a73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:40.676311 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2758283776}
I20260709 12:18:40.700595 12640 rpcz_store.cc:276] 0709 12:18:39.115517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.492647 (+377130us) service_pool.cc:224] Handling call
0709 12:18:39.493208 (+   561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:39.719511 (+226303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.655766 (+936255us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:40.656128 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515548032}
I20260709 12:18:40.714804 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.700881 12570 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:40.703118 27887 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.705590 27597 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.706483 27847 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.706915 27792 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:40.668365 12490 rpcz_store.cc:276] 0709 12:18:39.190529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.190805 (+   276us) service_pool.cc:224] Handling call
0709 12:18:39.191694 (+   889us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.262078 (+ 70384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:39.262123 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:39.262184 (+    61us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:39.262231 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.653991 (+1391760us) spinlock_profiling.cc:243] Waited 70 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:40.654070 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.654346 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161113472}
I20260709 12:18:40.647679 27734 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:40.713809 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
W20260709 12:18:40.731256 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47487.8us
W20260709 12:18:40.738391 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 83453.5us
I20260709 12:18:40.742165 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.743685 13262 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 54 because replica has last-logged OpId of term: 52 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 33.
W20260709 12:18:40.745532 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65280us
W20260709 12:18:40.750167 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.752393 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.749739 27957 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.714450 27684 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:40.755700 12569 raft_consensus.cc:2376] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38. Re-sending same reply.
W20260709 12:18:40.756173 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:40.759519 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.665405 27973 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.714814 27833 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.764954 27833 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.776167 12564 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:40.779579 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.782033 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.761029 27379 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:40.785257 12566 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:40.784163 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.787276 27959 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:40.787403 27494 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.805467 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.790006 12490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:40.790185 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:40.790927 27597 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.794538 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1930) took 2419 ms. Trace:
I20260709 12:18:40.819885 12490 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:40.820438 12490 raft_consensus.cc:614] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:40.795277 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:40.798053 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 150446us
I20260709 12:18:40.799841 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:40.760965 27216 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:40.749540 27887 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.802204 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:40.804508 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.804745 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.806315 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.808386 27623 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:40.809331 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:40.825522 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.820101 13257 rpcz_store.cc:276] 0709 12:18:38.375519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:38.375739 (+   220us) service_pool.cc:224] Handling call
0709 12:18:38.391046 (+ 15307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.294054 (+1903008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.794065 (+500011us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:40.794421 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4373540224}
I20260709 12:18:40.813500 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
W20260709 12:18:40.788328 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51168.7us
I20260709 12:18:40.822540 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:40.818753 12643 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:40.763733 27957 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.828339 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:40.809803 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.809746 27660 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:40.828995 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53056.4us
I20260709 12:18:40.836486 12567 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:40.825008 27965 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:40.845925 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.825865 27860 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:40.846897 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.825970 27858 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.025s
I20260709 12:18:40.841312 27494 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.856103 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:40.849850 27887 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.850682 12646 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:18:40.854031 12563 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:40.854382 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:40.838924 12639 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:40.856547 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:40.857439 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.863536 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:40.858814 12564 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50. Re-sending same reply.
I20260709 12:18:40.867635 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:40.872818 12640 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:40.873656 12638 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:18:40.873914 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:40.874642 12564 raft_consensus.cc:2376] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41. Re-sending same reply.
I20260709 12:18:40.879706 12644 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:40.882836 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53131.3us
I20260709 12:18:40.883149 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.887537 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.895807 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.899528 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:40.895860 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:40.900945 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101284us
I20260709 12:18:40.897013 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:40.900964 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:40.796424 27792 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.907298 12497 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:40.879524 27711 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:40.911520 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:40.912509 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:40.910213 13259 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:40.841049 28003 tablet_copy_client.cc:323] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:18:40.856060 27825 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.898705 27690 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:40.905840 27753 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:40.934556 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.935642 12563 raft_consensus.cc:2410] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 39 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:18:40.871114 27860 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:40.937460 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72398.4us
I20260709 12:18:40.910213 27946 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:40.934195 27753 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.945112 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6118) took 2859 ms. Trace:
I20260709 12:18:40.943907 27753 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:40.924324 27858 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
W20260709 12:18:40.958515 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47798.8us
I20260709 12:18:40.879302 27660 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.959460 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:40.961097 27746 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:40.986749 27746 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:40.987334 27746 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:40.962874 12564 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 51 because replica has last-logged OpId of term: 49 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 27.
I20260709 12:18:40.963439 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.991773 12567 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:40.993713 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 42 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:40.994949 12567 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:40.935384 27690 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.996196 27690 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:40.902287 27765 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:40.996433 27597 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.997125 12568 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:40.910071 27968 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.049s
I20260709 12:18:40.998762 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:40.999402 27765 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.000377 27765 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:40.972613 27910 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:40.974982 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:40.936003 27945 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.039s
W20260709 12:18:40.975566 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140173us
I20260709 12:18:41.022365 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:41.023391 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:41.023957 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:40.975893 27833 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:40.934036 27797 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.071s
I20260709 12:18:40.977931 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:40.980072 27660 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.980692 12569 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:40.981709 27887 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.020s
I20260709 12:18:40.982923 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4515) took 1332 ms. Trace:
I20260709 12:18:40.977973 27825 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:40.952040 27734 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:18:40.958807 27858 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:40.995252 27494 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:40.972266 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61895.4us
W20260709 12:18:41.002532 28007 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:18:40.971438 27860 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:41.009889 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56648.1us
I20260709 12:18:40.973016 27973 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:41.009470 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60655.1us
I20260709 12:18:40.961117 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.011780 27940 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:18:41.066493 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82674.9us
I20260709 12:18:41.012259 12644 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:41.009383 12570 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:41.072877 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6116) took 2993 ms. Trace:
I20260709 12:18:41.074527 12570 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:41.073681 12571 rpcz_store.cc:276] 0709 12:18:38.079644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.589392 (+2509748us) service_pool.cc:224] Handling call
0709 12:18:40.620656 (+ 31264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:40.630014 (+  9358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:40.630040 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:40.650869 (+ 20829us) log.cc:844] Serialized 26 byte log entry
0709 12:18:40.786287 (+135418us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:40.786345 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.816287 (+ 29942us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:41.072494 (+256207us) raft_consensus.cc:1685] finished
0709 12:18:41.072526 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:41.072761 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4559,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":5266,"thread_start_us":134817,"threads_started":1,"wal-append.queue_time_us":279213}
I20260709 12:18:41.075289 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6122) took 2914 ms. Trace:
I20260709 12:18:41.075649 12570 rpcz_store.cc:276] 0709 12:18:38.160938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.071578 (+2910640us) service_pool.cc:224] Handling call
0709 12:18:41.074281 (+  2703us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.074991 (+   710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.075201 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.076784 12571 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 50 index: 29. (index mismatch)
W20260709 12:18:41.077586 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:41.022312 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.077754 27753 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.028049 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:18:41.033403 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55606.9us
I20260709 12:18:41.061559 27951 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.018s
I20260709 12:18:41.079672 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.040078 12566 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 44 because replica has last-logged OpId of term: 43 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 25.
I20260709 12:18:41.026083 27833 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.016s
I20260709 12:18:41.080972 27753 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:41.050976 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 96756us
W20260709 12:18:41.051054 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 155366us
I20260709 12:18:41.025543 27910 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:41.060446 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:18:41.015326 27765 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.086107 27765 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:41.077062 12570 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:41.088325 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5800) took 2667 ms. Trace:
I20260709 12:18:41.068599 28016 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
W20260709 12:18:41.089690 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61768.6us
I20260709 12:18:41.089946 28016 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.090565 28016 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:41.092036 27746 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.062492 27858 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:41.093864 27860 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:18:41.064095 27660 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:41.097968 27660 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:41.077504 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5797) took 2659 ms. Trace:
I20260709 12:18:41.030766 27946 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:41.098996 12571 rpcz_store.cc:276] 0709 12:18:38.418763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.076002 (+2657239us) service_pool.cc:224] Handling call
0709 12:18:41.076402 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.077191 (+   789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.077420 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.042961 12637 rpcz_store.cc:276] 0709 12:18:39.652909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:39.653282 (+   373us) service_pool.cc:224] Handling call
0709 12:18:39.653933 (+   651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:39.654177 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:39.718073 (+ 63896us) log.cc:844] Serialized 14 byte log entry
0709 12:18:40.791319 (+1073246us) spinlock_profiling.cc:243] Waited 906 us on lock 0x7b180020e420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:40.923607 (+132288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:40.935170 (+ 11563us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:40.941683 (+  6513us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.960214 (+ 18531us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.982722 (+ 22508us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":58115,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":7226,"raft.queue_time_us":9024,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":26712,"spinlock_wait_cycles":2084224,"thread_start_us":87867,"threads_started":1,"wal-append.queue_time_us":1284882}
I20260709 12:18:41.100246 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.056880 27797 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:41.110491 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.107435 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.088697 12570 rpcz_store.cc:276] 0709 12:18:38.420662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.076511 (+2655849us) service_pool.cc:224] Handling call
0709 12:18:41.076872 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.087980 (+ 11108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.088233 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.020743 27690 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.112811 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:41.113793 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:41.112851 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.115718 13253 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:41.116546 27968 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:41.116969 27734 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:41.118188 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:41.118944 13262 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:41.120244 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:41.120563 12563 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:41.121094 13259 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:41.121866 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:41.122982 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:18:41.123190 28019 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:41.123734 28019 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.124289 28019 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:41.124625 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:41.124089 27560 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Committing config change with OpId 39.25: config changed from index -1 to 25, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:18:41.126914 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6114) took 3047 ms. Trace:
I20260709 12:18:41.127422 12568 rpcz_store.cc:276] 0709 12:18:38.079034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.580048 (+2501014us) service_pool.cc:224] Handling call
0709 12:18:40.580606 (+   558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.997612 (+417006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.126583 (+128971us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:41.126817 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":957364992}
I20260709 12:18:41.129400 27809 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:41.130192 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.131445 27809 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:41.131955 27809 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.132404 27809 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:41.133677 27940 raft_consensus.cc:2764] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election decision vote started in defunct term 43: won
I20260709 12:18:41.104408 27825 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:41.136569 12569 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 21. Preceding OpId from leader: term: 50 index: 21. (term mismatch)
I20260709 12:18:41.137028 12569 pending_rounds.cc:85] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 20
I20260709 12:18:41.137276 12569 pending_rounds.cc:107] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 49.21
I20260709 12:18:41.138917 27957 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.139712 12566 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 50 index: 28. (index mismatch)
I20260709 12:18:41.139853 27959 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.140394 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5823) took 2225 ms. Trace:
I20260709 12:18:41.140481 27959 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:41.141513 27846 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:41.141986 27901 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:41.142700 27901 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:41.143280 27216 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:41.143766 27216 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:41.144289 27901 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 38.27, Last appended by leader: 27, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:40.946713 12565 rpcz_store.cc:276] 0709 12:18:38.080195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.596733 (+2516538us) service_pool.cc:224] Handling call
0709 12:18:40.704672 (+107939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:40.718342 (+ 13670us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:40.761531 (+ 43189us) log.cc:844] Serialized 14 byte log entry
0709 12:18:40.933869 (+172338us) log.cc:844] Serialized 14 byte log entry
0709 12:18:40.938636 (+  4767us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:40.938721 (+    85us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:40.938807 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:40.938929 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:40.939107 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":42959,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":1890,"raft.queue_time_us":15020,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":13927,"thread_start_us":94172,"threads_started":1}
I20260709 12:18:41.101716 27946 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.146175 27946 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:41.121397 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5804) took 2699 ms. Trace:
I20260709 12:18:41.146819 12563 rpcz_store.cc:276] 0709 12:18:38.422184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.119907 (+2697723us) service_pool.cc:224] Handling call
0709 12:18:41.120358 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.121122 (+   764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.121337 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.144919 27216 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.114837 27684 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.151538 27690 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:41.129419 12568 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:41.152206 27684 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:41.152781 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5811) took 2645 ms. Trace:
I20260709 12:18:41.152930 27216 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:41.153209 12568 rpcz_store.cc:276] 0709 12:18:38.507248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.128616 (+2621368us) service_pool.cc:224] Handling call
0709 12:18:41.129134 (+   518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.152447 (+ 23313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.152669 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.140769 12566 rpcz_store.cc:276] 0709 12:18:38.914765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.138970 (+2224205us) service_pool.cc:224] Handling call
0709 12:18:41.139376 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.140139 (+   763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.140335 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.137508 12569 raft_consensus.cc:2889] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:41.149531 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5825) took 2151 ms. Trace:
I20260709 12:18:41.161329 12567 rpcz_store.cc:276] 0709 12:18:38.998489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.148282 (+2149793us) service_pool.cc:224] Handling call
0709 12:18:41.148807 (+   525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.149138 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:41.149165 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:41.149214 (+    49us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:41.149264 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.149326 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:41.149462 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:41.164239 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49438.1us
W20260709 12:18:41.164645 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.163182 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:41.168602 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.171075 27825 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:41.167142 27734 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.172210 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:41.171223 27973 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:41.174220 27965 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:41.174276 27973 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.174767 27734 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:41.175379 27973 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:41.175710 27734 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:41.174943 27965 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.176635 27965 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:41.176438 27734 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.172575 27825 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:41.177801 27797 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:41.181072 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5819) took 2354 ms. Trace:
I20260709 12:18:41.181756 12569 rpcz_store.cc:276] 0709 12:18:38.826697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.135741 (+2309044us) service_pool.cc:224] Handling call
0709 12:18:41.136139 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.180746 (+ 44607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.180992 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:41.182251 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69508.2us
I20260709 12:18:41.183694 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.184652 27494 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:41.185071 27494 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.185410 27494 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:41.185871 27684 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:41.189875 27684 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:41.190311 27684 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:41.193993 27955 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:18:41.194728 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 84332.4us
I20260709 12:18:41.195362 27955 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:41.195792 27955 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:41.198396 27957 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 48.28, Next index: 29, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:18:41.199986 27967 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:41.200434 27967 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:41.154637 27753 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.201465 27753 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.201435 27967 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 48.28, Last appended by leader: 28, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.202765 27967 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:41.142055 27846 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:41.204087 27753 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:41.204732 27753 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:41.204499 27846 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.195715 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:41.169571 27940 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:41.206302 27940 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:41.206825 27940 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:41.207015 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:41.207778 27940 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.209163 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.210307 12571 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:41.216370 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:41.170508 13253 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:41.220553 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:41.222025 12570 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:41.220633 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5801) took 2799 ms. Trace:
I20260709 12:18:41.222939 12571 rpcz_store.cc:276] 0709 12:18:38.421002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.155054 (+2734052us) service_pool.cc:224] Handling call
0709 12:18:41.210096 (+ 55042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.220349 (+ 10253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.220569 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:41.227239 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97149.7us
I20260709 12:18:41.227665 28026 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:41.228400 28025 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:41.253252 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:41.253453 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:41.254281 12571 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:41.254506 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:41.255072 12567 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:41.153426 27660 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 48.34, Last appended by leader: 34, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.233826 27690 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.235091 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:41.258436 12563 raft_consensus.cc:2368] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45.
I20260709 12:18:41.197026 27765 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.256585 27660 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:41.210099 27782 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:18:41.160797 27968 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:41.162119 27951 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.073s
W20260709 12:18:41.256181 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 90492.5us
I20260709 12:18:41.256546 12565 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 26. Preceding OpId from leader: term: 51 index: 27. (index mismatch)
I20260709 12:18:41.196987 27955 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 44.23, Last appended by leader: 23, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.205323 27753 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.268352 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6121) took 3181 ms. Trace:
I20260709 12:18:41.263706 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:41.267820 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6127) took 2130 ms. Trace:
I20260709 12:18:41.272187 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:41.279248 12565 rpcz_store.cc:276] 0709 12:18:39.137775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.216614 (+2078839us) service_pool.cc:224] Handling call
0709 12:18:41.256218 (+ 39604us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.267520 (+ 11302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.267763 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:41.216730 27901 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:41.274914 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:41.260140 27957 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:41.282444 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.283952 12564 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:41.288007 12646 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:41.269189 27623 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:41.266232 27782 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:41.293547 12460 leader_election.cc:400] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45.
I20260709 12:18:41.293397 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:41.294581 12460 leader_election.cc:403] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Cancelling election due to peer responding with higher term
I20260709 12:18:41.279544 12639 raft_consensus.cc:2376] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43. Re-sending same reply.
I20260709 12:18:41.299090 12563 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:41.292210 27957 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.264276 27690 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.301999 12537 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:41.314675 12490 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: [39.29-39.29]
I20260709 12:18:41.306175 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.308231 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.310662 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.282228 27746 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:41.327484 27746 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:41.324364 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.323118 12489 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 37. Current term is 38. Ops: []
I20260709 12:18:41.323011 27814 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:41.329535 27782 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 51.25, Last appended by leader: 25, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.330999 27623 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:41.329612 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.349804 27782 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:41.351356 27734 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:41.353557 12637 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 42 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:41.356333 27623 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.34, Last appended by leader: 34, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.339203 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5827) took 2068 ms. Trace:
I20260709 12:18:41.370005 12568 rpcz_store.cc:276] 0709 12:18:39.270966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.303064 (+2032098us) service_pool.cc:224] Handling call
0709 12:18:41.338445 (+ 35381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.338790 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:41.338817 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:41.338865 (+    48us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:41.338934 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.338990 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:41.339127 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:41.360666 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 149470us
I20260709 12:18:41.361845 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:41.298520 27858 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.348222 27955 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:41.316414 28020 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:41.387069 28020 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:41.387637 28020 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:41.390334 12566 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:41.391148 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5829) took 1680 ms. Trace:
I20260709 12:18:41.368511 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5818) took 2573 ms. Trace:
I20260709 12:18:41.280972 12572 rpcz_store.cc:276] 0709 12:18:38.080958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.024318 (+2943360us) service_pool.cc:224] Handling call
0709 12:18:41.024823 (+   505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.025175 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:41.068199 (+ 43024us) log.cc:844] Serialized 14 byte log entry
0709 12:18:41.262224 (+194025us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:41.262320 (+    96us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:41.262410 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.262527 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:41.262699 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":43058,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":6640,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":5190,"thread_start_us":212829,"threads_started":2,"wal-append.queue_time_us":228562}
I20260709 12:18:41.371587 12565 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:41.365497 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:41.392212 12564 rpcz_store.cc:276] 0709 12:18:38.795165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.135290 (+2340125us) service_pool.cc:224] Handling call
0709 12:18:41.135736 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.288572 (+152836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.368235 (+ 79663us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:41.368441 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340339584}
I20260709 12:18:41.393077 12638 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:41.381999 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:41.345149 27968 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.306324 27846 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:18:41.347716 28018 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:18:41.394170 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.296691 28026 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:41.391530 12566 rpcz_store.cc:276] 0709 12:18:39.710710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.315315 (+1604605us) service_pool.cc:224] Handling call
0709 12:18:41.390123 (+ 74808us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.390838 (+   715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.391077 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:41.397996 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 188926us
I20260709 12:18:41.403416 12571 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:41.404654 12568 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:41.400475 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6128) took 2136 ms. Trace:
I20260709 12:18:41.356997 28016 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.406219 12569 rpcz_store.cc:276] 0709 12:18:39.263782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.294540 (+2030758us) service_pool.cc:224] Handling call
0709 12:18:41.358199 (+ 63659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.358517 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:18:41.358980 (+   463us) log.cc:844] Serialized 14 byte log entry
0709 12:18:41.400074 (+ 41094us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:41.400132 (+    58us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:41.400202 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.400275 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:41.400411 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":259,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":168,"thread_start_us":40522,"threads_started":1}
I20260709 12:18:41.411157 12639 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:41.413268 12571 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 26. Preceding OpId from leader: term: 44 index: 26. (term mismatch)
I20260709 12:18:41.413908 12571 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 25
I20260709 12:18:41.414294 12571 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.26
I20260709 12:18:41.414703 12571 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:41.344821 27765 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.379025 27792 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:41.332178 27957 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.419322 12572 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:41.419888 12490 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:41.423775 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:41.442274 12489 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:18:41.443972 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:41.450489 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.453596 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.457527 12492 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:41.454519 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.459326 12496 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:41.416688 27858 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:41.417269 27797 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.044s
I20260709 12:18:41.458420 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.445760 28019 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.462239 12639 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:41.463869 12639 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:41.464129 12568 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:41.467566 12570 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:41.469854 27973 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.472610 27973 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:41.473867 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:41.475162 27858 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:18:41.475586 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63213.8us
I20260709 12:18:41.477666 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:41.478489 12641 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:41.482441 12642 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 43 index: 26. (index mismatch)
I20260709 12:18:41.485877 12563 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:41.455173 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.489885 12493 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:18:41.491079 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.455745 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.492471 27858 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:41.492630 12495 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 38.
I20260709 12:18:41.458487 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.494550 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.495024 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.495652 12492 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:18:41.495747 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.496611 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.460451 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.454128 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.427034 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.477524 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:41.506449 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:41.502135 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.497366 12492 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:41.510842 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:41.510942 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83988.7us
I20260709 12:18:41.507880 12495 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 35.
I20260709 12:18:41.460422 27946 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.496953 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.526557 12568 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: [51.25-51.25]
I20260709 12:18:41.530366 12493 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:41.530615 12563 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 25. Preceding OpId from leader: term: 43 index: 26. (index mismatch)
I20260709 12:18:41.456327 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.532217 27753 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.532092 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.536338 27690 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.521057 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:41.522114 27797 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:41.542532 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:41.538213 28026 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.511164 27965 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:41.546401 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.547704 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:41.546233 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 317602us
I20260709 12:18:41.551926 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:41.554503 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.516202 27792 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:41.587548 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.500097 28019 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.556867 12642 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:41.560431 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:41.564507 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.593302 12638 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
W20260709 12:18:41.594632 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.588642 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.608220 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:41.635423 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71869.1us
I20260709 12:18:41.568009 12489 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
W20260709 12:18:41.636981 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 193091us
I20260709 12:18:41.556349 27797 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.658272 12643 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:41.645123 12564 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:41.646659 12490 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:18:41.455021 27684 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:41.652565 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.652810 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:41.653149 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:41.661919 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.654057 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.656100 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.658408 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 47 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.561753 27825 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
W20260709 12:18:41.646441 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.616s (SENT)
I20260709 12:18:41.669793 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:41.661926 12488 raft_consensus.cc:2376] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43. Re-sending same reply.
I20260709 12:18:41.679289 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.681034 12494 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:41.525007 27959 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.685643 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.677775 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:41.665159 27833 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.704424 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.666281 27797 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:41.696144 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:41.708009 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.590693 27494 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.710412 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.720363 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:41.722004 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 77810.7us
I20260709 12:18:41.690407 27825 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:41.716447 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 218848us
I20260709 12:18:41.692520 27560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:57108 (request call id 3) took 6618 ms. Trace:
W20260709 12:18:41.726334 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.726215 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.732430 12495 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:41.727052 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.715715 12490 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:41.666571 27690 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.046s
W20260709 12:18:41.744923 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 197959us
I20260709 12:18:41.690837 27957 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.034s
I20260709 12:18:41.747044 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:41.729027 27967 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.006s
I20260709 12:18:41.749012 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.719838 28025 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:41.727469 27940 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:41.755264 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.755797 12488 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:18:41.754899 12492 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:41.735926 27734 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:41.725069 27910 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.032s
I20260709 12:18:41.760480 27340 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.085s
I20260709 12:18:41.729488 27560 rpcz_store.cc:276] 0709 12:18:35.050058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:35.274043 (+223985us) service_pool.cc:224] Handling call
0709 12:18:35.967734 (+693691us) spinlock_profiling.cc:243] Waited 927 us on lock 0x7b54002bd520. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576aec4 00007ffbf57ee8d7 00007ffbf57fa03c 00007ffbf57f96b3 00007ffbfa7b9d29 00007ffbf3d381d4 00007ffbf3d38162 00007ffbf3d38091 00007ffbf3d3800c 00007ffbf3d372c2
0709 12:18:36.019743 (+ 52009us) log.cc:844] Serialized 506 byte log entry
0709 12:18:37.849675 (+1829932us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f9735 00007ffbfa7b9d29 00007ffbf3d381d4 00007ffbf3d38162 00007ffbf3d38091 00007ffbf3d3800c 00007ffbf3d372c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:41.642684 (+3793009us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252264960}
I20260709 12:18:41.762320 28016 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.763315 27959 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.763767 27963 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.058s
I20260709 12:18:41.770767 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:41.786197 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 63191.1us
W20260709 12:18:41.773483 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.715513 27833 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.772841 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.781366 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:41.793764 12571 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:41.794272 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 147885us
I20260709 12:18:41.794523 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:41.782302 27660 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.514s	user 0.013s	sys 0.000s
W20260709 12:18:41.771848 28031 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:18:41.767985 27734 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:41.781555 12639 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:41.801008 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:41.809998 12565 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:41.823189 12571 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
W20260709 12:18:41.823113 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 167152us
I20260709 12:18:41.825413 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:41.822144 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84150.1us
I20260709 12:18:41.832027 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.833169 12488 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:41.783814 28034 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:41.784175 28030 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:41.800714 27860 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:41.844907 28017 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:41.845530 28028 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.783918 27858 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.051s
W20260709 12:18:41.847613 28032 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:18:41.748312 27792 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 43.31, Last appended by leader: 31, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.835932 28019 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:41.870146 28019 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:41.871189 28019 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:41.830003 27809 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.697s	user 0.015s	sys 0.001s
I20260709 12:18:41.814921 28016 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.803104 27814 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 44. Reason: Vote denied by peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927) with higher term. Message: Invalid argument: T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45.
W20260709 12:18:41.852994 12567 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.002s
I20260709 12:18:41.896891 12567 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:41.897846 12566 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
W20260709 12:18:41.898777 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44560.3us
I20260709 12:18:41.898751 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:41.796455 27940 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.904093 13253 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:41.783831 28024 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:41.829440 27734 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:41.862759 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.864903 27833 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.875147 12493 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:18:41.878392 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:41.879060 12536 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.648s (SENT)
W20260709 12:18:41.888501 13218 leader_election.cc:336] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.618s (SENT)
I20260709 12:18:41.851708 28034 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:41.872354 27809 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.877813 27660 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.920591 13261 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:41.920579 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:41.875483 27946 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:41.927242 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.930205 12488 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:41.927580 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.932024 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.903141 27965 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:41.940464 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:41.942044 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115216us
I20260709 12:18:41.943099 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.813881 27825 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.945531 13253 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:18:41.965967 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.966153 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:41.969548 12493 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:18:41.972476 27809 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:41.975840 27216 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.822s	user 0.014s	sys 0.001s
I20260709 12:18:41.926736 27684 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:41.978309 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.979782 12493 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:41.902074 28016 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:41.981087 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 102076us
W20260709 12:18:41.981731 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93312us
I20260709 12:18:41.984120 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:41.985988 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 175139us
I20260709 12:18:41.986982 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2921) took 1198 ms. Trace:
I20260709 12:18:41.898043 28028 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:41.903045 27860 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:41.931942 28034 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:41.990743 12566 rpcz_store.cc:276] 0709 12:18:40.788415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.407728 (+619313us) service_pool.cc:224] Handling call
0709 12:18:41.897618 (+489890us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:41.898236 (+   618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:41.986777 (+ 88541us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b5400045450. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:41.986921 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1113833728}
I20260709 12:18:41.985392 12537 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:41.996372 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:41.997560 12494 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:18:41.999676 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59291.7us
I20260709 12:18:41.956053 27833 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:41.956956 27973 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.002754 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:41.981381 27858 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:42.006457 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:42.007639 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:42.017424 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74403.5us
I20260709 12:18:42.018363 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:41.996995 27753 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:41.946501 27660 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.005345 27965 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:42.032188 27860 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:42.034464 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:42.035609 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.037081 27858 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
W20260709 12:18:42.039163 28043 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 12:18:42.043403 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.044298 28044 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:42.044824 28044 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.045245 28044 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:42.056277 13261 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:42.058301 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:42.060714 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:42.005339 27746 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:18:42.073426 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48282.9us
W20260709 12:18:42.075323 12536 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.991s (SENT)
I20260709 12:18:42.075870 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:42.076761 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.078887 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:42.082715 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:42.083968 12493 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:42.084830 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:42.085489 28052 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:18:42.085799 12490 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:42.094041 28046 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:42.094722 28046 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.095170 28046 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:42.095836 28050 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:42.096657 28050 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:42.102291 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:42.094930 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51606.3us
I20260709 12:18:42.111063 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.112288 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.113119 12563 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:42.086364 27965 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:42.099242 12655 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:18:42.070331 28024 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.004787 27797 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.117130 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.119709 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:42.016906 27973 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.144599 27973 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:42.145882 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:42.127302 28044 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.128890 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.128942 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.150422 12644 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:42.151072 28016 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:42.130177 27792 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:42.137872 12655 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 5f402dfd68ae4c8697765a76d9f1c4e6 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-5f402dfd68ae4c8697765a76d9f1c4e6
I20260709 12:18:42.151800 28016 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Stepping down as leader of term 49
W20260709 12:18:42.158129 27846 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.745s	user 0.012s	sys 0.003s
I20260709 12:18:42.138206 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:42.159596 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.152519 28016 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:42.132513 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:42.170521 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 150943us
I20260709 12:18:42.103744 27940 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:42.179592 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55016.6us
I20260709 12:18:42.028178 27753 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.153712 28028 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:42.157409 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:42.160086 12563 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:42.152056 28044 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:42.180011 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:42.181525 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 147238us
I20260709 12:18:42.183879 12490 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
W20260709 12:18:42.186931 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.131466 28024 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:42.191473 12490 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:42.188681 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:42.179257 27901 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.890s	user 0.006s	sys 0.008s
I20260709 12:18:42.192477 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:42.195827 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120535us
W20260709 12:18:42.197903 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65493.7us
I20260709 12:18:42.199558 27809 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:42.200091 27809 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:42.200433 27967 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:42.181561 28061 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:42.201112 27967 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:42.202013 28055 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.018s
I20260709 12:18:42.204066 27901 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:42.204473 27965 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.28, Last appended by leader: 28, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.200491 27809 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:42.207233 27753 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:42.204973 13253 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:42.212644 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:42.156123 27660 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:42.207612 27967 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.29, Last appended by leader: 29, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.225842 27216 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:42.226155 27967 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:42.218237 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:42.215044 28016 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.215795 13259 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
W20260709 12:18:42.224222 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:42.200783 27782 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.849s	user 0.008s	sys 0.004s
I20260709 12:18:42.232060 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:42.234498 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.235893 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:42.197178 28020 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.809s	user 0.011s	sys 0.005s
I20260709 12:18:42.204670 27901 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:42.246886 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56866.7us
I20260709 12:18:42.246366 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:42.251865 12490 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:42.252315 12494 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:42.252547 12496 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 52 because replica has last-logged OpId of term: 51 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 24.
I20260709 12:18:42.250490 27901 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:42.179651 27746 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 41.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.048s
I20260709 12:18:42.226369 27797 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:42.264237 12536 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.991s (SENT)
W20260709 12:18:42.270272 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81684.1us
I20260709 12:18:42.271163 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.227478 28046 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.263536 28016 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:42.263485 28055 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:42.279007 12490 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:42.283083 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:42.189690 28050 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.257807 28028 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.019s
I20260709 12:18:42.289127 12497 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:42.300926 27876 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:42.306914 12566 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:42.301971 28030 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:42.268023 27660 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.017s
I20260709 12:18:42.252048 28020 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.304595 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:42.321285 12497 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 48. Current term is 49. Ops: []
W20260709 12:18:42.308102 28058 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:18:42.323701 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.325290 12640 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
W20260709 12:18:42.323130 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58918.7us
I20260709 12:18:42.326125 28030 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:42.326896 28030 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:42.332056 12497 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:42.336676 27660 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.338163 28020 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:42.337278 27660 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.340852 27660 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:42.343146 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.343376 28046 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:42.379971 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.344054 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.345660 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.346998 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.348129 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.355448 28054 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:42.389160 12643 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:42.383298 12642 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:42.356915 28065 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:18:42.334936 27734 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.360473 27846 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:42.361718 28051 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:42.362965 27746 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:42.364722 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.303399 28057 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.079s
I20260709 12:18:42.366142 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:42.367483 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.368688 12497 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:42.371196 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.384541 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.387625 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.412657 12567 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:42.412840 12572 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:42.413618 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.413887 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.415294 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.379858 28019 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:42.416030 28019 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:42.417037 28019 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.416637 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.419499 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.420254 12571 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:42.393388 12563 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:18:42.422212 12572 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:18:42.422610 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.423949 12572 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 51. Current term is 52. Ops: [50.29-51.30]
I20260709 12:18:42.424573 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:42.395112 12641 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 22. Preceding OpId from leader: term: 44 index: 23. (index mismatch)
I20260709 12:18:42.395645 28034 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.428508 28034 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:42.396709 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.397118 28061 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:42.397890 12645 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
W20260709 12:18:42.398908 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 94344.8us
I20260709 12:18:42.430785 28061 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:18:42.403050 12644 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.26-46.26]
I20260709 12:18:42.431759 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:42.432641 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.433476 12644 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:42.403163 12494 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
W20260709 12:18:42.436383 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.873s (SENT)
I20260709 12:18:42.436980 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.405349 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:42.438485 12570 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 41 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 24.
I20260709 12:18:42.439844 12490 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.26-46.26]
I20260709 12:18:42.440053 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:18:42.441346 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.408396 12566 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:42.408406 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.443693 12642 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:42.315905 28064 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:42.447427 28064 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:42.447994 28064 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:42.448454 28064 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:42.430786 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.451002 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:42.452292 28061 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:42.454001 28061 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.30, Last appended by leader: 30, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:42.476305 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.502096 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.503702 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:42.516047 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.518083 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.523149 12639 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:42.523581 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:18:42.524273 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87925us
I20260709 12:18:42.526774 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:42.408152 27846 consensus_queue.cc:1243] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 44.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 28
I20260709 12:18:42.527652 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.533087 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.534139 12564 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:42.511366 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:42.525836 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:42.535813 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.410398 28051 consensus_queue.cc:1243] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 43.28 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 27
I20260709 12:18:42.541994 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:18:42.548842 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146300us
I20260709 12:18:42.521322 27660 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:42.550288 12461 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.743s (SENT)
I20260709 12:18:42.551040 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.533100 28050 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:18:42.329077 28030 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:42.566164 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.549250 28020 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:42.567625 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:42.568869 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144378us
I20260709 12:18:42.557094 27746 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:42.575459 12564 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:42.580473 13257 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:18:42.582564 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.583209 12640 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45. Re-sending same reply.
I20260709 12:18:42.569348 27846 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.015s
I20260709 12:18:42.584877 27846 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:18:42.585254 27846 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:42.586283 27846 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 51.30, Last appended by leader: 30, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.587803 27846 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:42.588768 28030 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:42.563481 28051 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:18:42.592514 28054 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:18:42.599028 28066 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:18:42.606124 27494 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:42.576447 28028 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:42.616463 27734 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:42.618023 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.618633 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4442) took 1014 ms. Trace:
I20260709 12:18:42.619097 12489 rpcz_store.cc:276] 0709 12:18:41.637350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.647498 (+ 10148us) service_pool.cc:224] Handling call
0709 12:18:42.618385 (+970887us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:42.618574 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1323054976}
I20260709 12:18:42.620697 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4430) took 1137 ms. Trace:
I20260709 12:18:42.621143 12492 rpcz_store.cc:276] 0709 12:18:41.482893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.525363 (+ 42470us) service_pool.cc:224] Handling call
0709 12:18:42.620504 (+1095141us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:42.620634 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269016960}
I20260709 12:18:42.627928 12563 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [40.29-40.29]
I20260709 12:18:42.628772 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.629464 12563 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:42.627871 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:42.630126 27746 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:42.631105 12566 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:42.631621 13262 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 31. Preceding OpId from leader: term: 45 index: 32. (index mismatch)
I20260709 12:18:42.630035 12570 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 34. Preceding OpId from leader: term: 49 index: 35. (index mismatch)
I20260709 12:18:42.627307 12567 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.27-48.27]
I20260709 12:18:42.629616 12572 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:42.638125 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.646050 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:42.648083 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:42.670270 12536 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.503s (SENT)
I20260709 12:18:42.672894 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.630218 27340 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:42.681851 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:42.684669 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.558636 27660 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.627187 28054 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:42.685461 12640 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:18:42.686419 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.630471 27957 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:42.688587 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:42.678056 28039 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.816s	user 0.002s	sys 0.002s
I20260709 12:18:42.693482 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:42.702697 12638 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:42.695736 12567 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:42.677450 27797 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.687543 27876 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.680502 27340 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:42.751130 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:42.736125 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.714282 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4444) took 1049 ms. Trace:
I20260709 12:18:42.745148 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.698056 27825 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
W20260709 12:18:42.737411 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 187146us
W20260709 12:18:42.748893 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85856.6us
I20260709 12:18:42.775177 12643 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:42.772330 12638 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:42.690678 27746 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:42.721560 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92552.4us
W20260709 12:18:42.773442 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 195033us
I20260709 12:18:42.776468 12640 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:42.690338 27957 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:42.779868 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69553.4us
I20260709 12:18:42.705353 27765 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:42.784934 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:42.715487 27782 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:42.749891 27792 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.788173 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1948) took 1117 ms. Trace:
I20260709 12:18:42.786213 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:42.789815 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:42.797263 12646 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:18:42.778719 27951 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:42.804373 12642 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:42.803894 13261 rpcz_store.cc:276] 0709 12:18:41.662976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.673150 (+ 10174us) service_pool.cc:224] Handling call
0709 12:18:42.780065 (+1106915us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:42.780186 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489696128}
W20260709 12:18:42.812958 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.810599 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:42.819528 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:42.822192 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:42.831650 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:42.844336 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:42.846098 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:42.848440 27560 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:42.859979 13262 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:42.862022 13258 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:42.864539 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:42.866948 27940 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:42.867913 27940 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:42.868476 27940 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.871976 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:42.873706 27560 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:42.874985 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:42.876070 13258 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:42.879204 27940 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:42.736301 27901 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.771376 27684 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.032s
W20260709 12:18:42.947365 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146997us
I20260709 12:18:42.950340 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:42.789216 27825 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:42.778628 27797 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:42.958153 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51990.8us
I20260709 12:18:42.813817 27746 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:42.977836 27746 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:42.979053 27746 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.898481 27494 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:42.980268 27494 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:42.959870 28060 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:42.968217 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.975895 27797 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:42.899139 27782 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:42.969664 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 159105us
I20260709 12:18:42.748370 28064 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:42.990316 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:42.773339 27876 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:42.808488 27951 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:43.010313 28029 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.286s	user 0.005s	sys 0.000s
W20260709 12:18:43.015296 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 188391us
I20260709 12:18:42.981247 27494 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.009794 27782 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.025802 12572 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 52 index: 31. (index mismatch)
I20260709 12:18:43.048921 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:43.049409 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:43.053261 12565 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.29-45.29]
W20260709 12:18:43.053577 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.055547 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:43.058086 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 62763.9us
W20260709 12:18:43.058277 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.814832 27792 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.066780 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:43.061429 28017 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.160s	user 0.008s	sys 0.012s
I20260709 12:18:43.051046 27957 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.076761 28017 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:43.077085 27957 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:43.064280 28091 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:43.077965 28091 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.078368 28091 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:43.018337 27797 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:18:43.059262 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72297.3us
I20260709 12:18:43.079286 12569 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:18:43.090238 28099 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:18:43.079376 27967 consensus_queue.cc:1175] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 41 status { last_received { term: 40 index: 21 } last_committed_idx: 21 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:18:43.092252 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:43.093111 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:43.028057 27660 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:43.072549 28024 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.858s	user 0.015s	sys 0.000s
W20260709 12:18:43.104614 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.101639 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:42.897858 27340 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.122591 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:43.128424 13258 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:43.101785 28050 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:43.139207 13261 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
W20260709 12:18:43.150418 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94989.1us
I20260709 12:18:43.095978 28089 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:43.169461 13258 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:43.079793 28044 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.051s
I20260709 12:18:43.206156 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:43.208045 13258 raft_consensus.cc:2368] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45.
W20260709 12:18:43.209266 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.203058 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 143930us
W20260709 12:18:43.211313 28098 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:18:43.170509 27340 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:42.982401 28060 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.138700 27825 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.29, Last appended by leader: 29, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.125867 28024 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.202766 28050 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:43.234800 13254 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.30-48.30]
W20260709 12:18:43.217514 28083 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 12:18:43.219476 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:43.254192 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 223739us
I20260709 12:18:43.201288 27660 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:43.178018 28089 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.215222 27684 consensus_queue.cc:1243] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 49.28 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 27
W20260709 12:18:43.231588 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.261968 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.134713 27951 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 48.27, Last appended by leader: 27, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.242605 27876 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:43.246449 27901 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:43.206142 28017 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.041s
I20260709 12:18:43.246806 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:43.247437 27792 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:43.248679 27809 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.043s	user 0.014s	sys 0.004s
I20260709 12:18:43.246493 13260 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.29-48.29]
I20260709 12:18:43.254140 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:43.258736 13258 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 42. Ops: [40.29-40.29]
W20260709 12:18:43.225454 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 126594us
I20260709 12:18:43.248724 28064 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:43.263711 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:43.246498 28095 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 12:18:43.280252 13262 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:43.270121 27797 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:43.274608 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.274339 13261 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:43.260850 27660 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:43.280593 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 225641us
W20260709 12:18:43.283313 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 333090us
I20260709 12:18:43.283553 13260 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 33. Preceding OpId from leader: term: 49 index: 35. (index mismatch)
W20260709 12:18:43.255825 27846 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.668s	user 0.013s	sys 0.004s
W20260709 12:18:43.276451 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.286748 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:43.284162 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:43.290729 12571 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 40 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:43.289340 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:43.291344 13254 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:43.294528 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.295630 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.296470 12488 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 41 because replica has last-logged OpId of term: 40 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 29.
I20260709 12:18:43.263204 28089 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:43.298817 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.298934 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:43.303194 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.303534 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:43.304410 27765 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:43.305032 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:43.304881 27765 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:43.305330 27876 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:43.299568 27782 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.299086 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.306187 27765 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:43.306200 27876 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:43.307349 27876 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:43.307451 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.299412 27797 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.28, Last appended by leader: 28, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.306371 27782 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.308759 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.310254 27782 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:43.311123 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:43.304823 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.338253 12489 raft_consensus.cc:2376] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39. Re-sending same reply.
I20260709 12:18:43.311920 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:43.313081 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.339488 12571 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:43.309798 12492 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:43.298313 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:43.340756 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:43.290743 27660 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:43.343437 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.347307 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.335952 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.298686 27792 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:43.278151 27494 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.057s
I20260709 12:18:43.282475 27809 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.382025 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.384028 27809 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:43.385852 28050 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.385316 27876 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.367029 28106 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.089s
I20260709 12:18:43.391568 28106 consensus_queue.cc:1243] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 48.34 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 33
W20260709 12:18:43.368561 28104 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:18:43.354774 27792 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.395136 27792 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:43.396175 12564 pending_rounds.cc:85] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 33
I20260709 12:18:43.396543 12564 pending_rounds.cc:107] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 48.34
I20260709 12:18:43.396783 12564 raft_consensus.cc:2889] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:43.396848 27779 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:43.397804 27684 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:43.398290 27684 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:43.398705 27684 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:43.400715 27814 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:18:43.401134 27814 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:42.770506 12491 rpcz_store.cc:276] 0709 12:18:41.654443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.275729 (+621286us) service_pool.cc:224] Handling call
0709 12:18:42.285536 (+  9807us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:42.285833 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:18:42.408610 (+122777us) log.cc:844] Serialized 14 byte log entry
0709 12:18:42.703014 (+294404us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:42.703096 (+    82us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:42.703154 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.703232 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:42.703389 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19501,"prepare.queue_time_us":247153,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":4185,"raft.queue_time_us":14540,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":30961,"thread_start_us":248958,"threads_started":2}
I20260709 12:18:43.358543 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.407172 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.380878 28105 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.118s
I20260709 12:18:43.384809 12637 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:43.390219 28116 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 50.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:43.400532 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.359421 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:43.377779 28090 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:18:43.378577 28060 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:43.408027 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.379901 28097 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.110s
I20260709 12:18:43.407018 12495 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:18:43.401475 27814 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:43.452754 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2201) took 1298 ms. Trace:
I20260709 12:18:43.456203 12571 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:18:43.486292 12639 rpcz_store.cc:276] 0709 12:18:42.108730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.108956 (+   226us) service_pool.cc:224] Handling call
0709 12:18:43.407087 (+1298131us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:43.407245 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804056064}
W20260709 12:18:43.377286 28030 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.788s	user 0.010s	sys 0.003s
W20260709 12:18:43.479756 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 168757us
I20260709 12:18:43.369586 28113 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:43.497118 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:43.491655 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.497861 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.397535 27779 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.498821 12489 raft_consensus.cc:2376] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45. Re-sending same reply.
I20260709 12:18:43.505256 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.558189 12491 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 39: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:43.507462 27876 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:43.510110 28119 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:43.513067 28093 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:43.515116 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.518307 28118 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:43.520331 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:43.520507 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6167) took 1412 ms. Trace:
I20260709 12:18:43.521526 28125 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:43.524171 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.530181 28113 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:43.532866 12492 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:43.533561 28055 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:43.534871 27814 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.532578 12488 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:43.453413 28108 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.565831 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.565145 12568 rpcz_store.cc:276] 0709 12:18:42.107755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.632358 (+524603us) service_pool.cc:224] Handling call
0709 12:18:42.632898 (+   540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:42.707385 (+ 74487us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:42.707441 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:42.725239 (+ 17798us) log.cc:844] Serialized 26 byte log entry
0709 12:18:42.995594 (+270355us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:42.995658 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.995734 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:43.520153 (+524419us) raft_consensus.cc:1685] finished
0709 12:18:43.520181 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:43.520419 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39491,"raft.queue_time_us":34693,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":10574,"thread_start_us":209548,"threads_started":1,"wal-append.queue_time_us":382547}
I20260709 12:18:43.566347 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 42 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.567945 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:43.568096 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47834.1us
I20260709 12:18:43.565353 27779 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:43.586936 12491 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:18:43.590448 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 292175us
I20260709 12:18:43.563828 28093 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:43.597259 13258 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 51 index: 29. (index mismatch)
I20260709 12:18:43.600800 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:43.600800 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:43.603211 12489 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:43.608692 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.606276 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:43.613499 27951 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:43.613814 12571 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:43.614547 12568 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:43.622138 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:43.626439 28026 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.627326 28026 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:43.634457 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4433) took 2150 ms (client timeout 1833 ms). Trace:
W20260709 12:18:43.635210 12496 rpcz_store.cc:269] 0709 12:18:41.483750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:41.546250 (+ 62500us) service_pool.cc:224] Handling call
0709 12:18:43.633988 (+2087738us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:43.634272 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1526332672}
I20260709 12:18:43.637041 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.638339 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4447) took 1982 ms. Trace:
I20260709 12:18:43.639248 12490 rpcz_store.cc:276] 0709 12:18:41.655471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.286011 (+630540us) service_pool.cc:224] Handling call
0709 12:18:42.439480 (+153469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:42.440410 (+   930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.638008 (+1197598us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540020e4d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:43.638248 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297598848}
I20260709 12:18:43.641093 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.642297 12493 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:18:43.553748 27765 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.574110 28105 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:43.625247 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4530) took 1301 ms. Trace:
I20260709 12:18:43.655589 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:43.664853 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:43.656762 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:43.665956 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.656957 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:43.646551 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:43.574569 28119 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:43.665270 12639 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:43.654418 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:43.577108 28017 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.673357 12638 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:43.646986 12566 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: [42.28-42.28]
I20260709 12:18:43.673789 12638 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:43.661072 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:43.572831 28055 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:43.605681 28091 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.527s	user 0.002s	sys 0.011s
I20260709 12:18:43.663707 12491 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 31. Preceding OpId from leader: term: 45 index: 32. (index mismatch)
I20260709 12:18:43.664508 12637 rpcz_store.cc:276] 0709 12:18:42.323844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.324265 (+   421us) service_pool.cc:224] Handling call
0709 12:18:42.366844 (+ 42579us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:43.477679 (+1110835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.624880 (+147201us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:43.625151 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2340497664}
I20260709 12:18:43.672489 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:43.680460 12642 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:43.680644 12567 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: [47.32-47.32]
W20260709 12:18:43.675067 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.573671 27684 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.675982 12495 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:43.676491 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:43.677139 13256 raft_consensus.cc:2410] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 49 because replica has last-logged OpId of term: 47 index: 32, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 31.
I20260709 12:18:43.678574 12489 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:43.682715 12638 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 42.28, Last appended by leader: 28, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:43.672669 28097 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:43.594228 28108 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:43.694911 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 402405us
I20260709 12:18:43.696074 12638 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:43.675805 28105 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:43.781505 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:43.873943 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 175488us
I20260709 12:18:43.875113 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:43.880347 12642 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:43.679452 28060 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.026s
I20260709 12:18:43.698118 28102 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.679113 28017 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:43.694506 28030 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.695003 28097 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:43.830783 27901 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:43.683974 28091 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:43.970928 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.000876 12610 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:43.914304 28127 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:44.023002 28127 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.003499 27901 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:44.024549 28130 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.327s	user 0.000s	sys 0.001s
I20260709 12:18:44.025324 28131 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.002650 28030 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.025982 27901 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.023361 27684 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:43.915100 28026 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:43.915333 28118 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.032263 12638 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 28. Preceding OpId from leader: term: 43 index: 28. (term mismatch)
I20260709 12:18:43.954717 28097 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:44.044981 28114 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:44.047108 28114 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:44.026021 28131 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.012547 28119 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 38.28, Last appended by leader: 28, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.038257 12638 pending_rounds.cc:85] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 27
I20260709 12:18:44.054853 12638 pending_rounds.cc:107] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 42.28
I20260709 12:18:44.055312 12638 raft_consensus.cc:2889] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:43.688205 28113 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.048014 28114 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:44.048424 28026 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.040371 28127 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:44.072227 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 383579us
I20260709 12:18:43.934165 28102 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.076792 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:44.035485 27901 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.036s
W20260709 12:18:44.071738 27660 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.692s	user 0.009s	sys 0.004s
I20260709 12:18:44.075220 27814 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.079890 28026 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:44.089130 28085 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.137s	user 0.000s	sys 0.002s
I20260709 12:18:43.923240 28055 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 47.32, Last appended by leader: 32, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.081453 28127 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:44.052380 28118 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:44.110101 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.076507 28114 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:44.119540 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:44.112519 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.018s	sys 0.025s
W20260709 12:18:44.121143 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.018s	sys 0.025s
W20260709 12:18:44.121665 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 519498us
W20260709 12:18:44.122422 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.202s (SENT)
W20260709 12:18:44.113010 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 180149us
W20260709 12:18:44.108917 28050 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.004s	sys 0.020s
I20260709 12:18:44.068220 28105 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.098805 28127 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.132812 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:44.094857 27660 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.152868 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:44.151306 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68978.2us
I20260709 12:18:44.118600 27765 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:44.137370 28050 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.005s	sys 0.021s
I20260709 12:18:44.139597 28055 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:18:44.170223 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 169457us
I20260709 12:18:44.224469 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.162112 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:44.151561 28060 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:44.166155 27779 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.524s	user 0.010s	sys 0.003s
I20260709 12:18:44.191804 12489 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "256d044fd2e244d4be827104688867ee"
dest_uuid: "6d4ece942fd5414998215d26edf91001"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:53272
I20260709 12:18:44.192930 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:44.225134 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210730us
I20260709 12:18:44.255460 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.168025 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.256835 12642 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 40 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
I20260709 12:18:44.256765 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:44.232715 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73548us
I20260709 12:18:44.146176 28132 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.135s
I20260709 12:18:44.161521 28135 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:44.255414 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:44.267030 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:44.202394 28089 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.858s	user 0.011s	sys 0.003s
I20260709 12:18:44.256449 12646 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49. Re-sending same reply.
I20260709 12:18:44.258067 12637 raft_consensus.cc:2376] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39. Re-sending same reply.
I20260709 12:18:44.161237 28108 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.215667 28030 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:44.234200 27951 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.620s	user 0.007s	sys 0.007s
I20260709 12:18:44.274549 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.189512 27876 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:44.221899 28050 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.162096 28115 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.286293 12642 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:44.280809 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.314306 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
W20260709 12:18:44.307427 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.305320 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.318282 12637 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:44.286293 12638 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:44.312011 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:44.305119 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:44.322105 13262 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:44.241065 28144 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.311540 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:44.325524 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:44.278694 28132 consensus_queue.cc:1243] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 44.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 28
I20260709 12:18:44.314419 27797 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:44.292963 28091 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:44.340888 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116487us
I20260709 12:18:44.292096 28136 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:44.289714 28135 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:44.346112 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:44.339430 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.304831 28131 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.332727 28119 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:44.357914 28119 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.354225 28127 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:44.358351 28119 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:44.358762 28161 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:44.359577 28161 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:44.360018 28161 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:44.356842 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95388.1us
I20260709 12:18:44.355316 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.277463 27779 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.335073 28026 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.341022 28102 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.325091 28115 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.278085 28060 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.280107 28089 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.332829 28144 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.333312 28140 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
W20260709 12:18:44.363334 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 98487.1us
I20260709 12:18:44.355860 28118 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.365213 12639 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:44.326005 27876 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.343668 27797 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.324736 28050 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.350868 28136 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.386294 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:44.367626 28142 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:18:44.371030 12564 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
I20260709 12:18:44.389060 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:44.413110 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71960.7us
I20260709 12:18:44.414999 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.403787 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.405897 12572 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
W20260709 12:18:44.416249 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 254297us
I20260709 12:18:44.409188 12568 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:44.363695 28132 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
W20260709 12:18:44.427775 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116368us
I20260709 12:18:44.438789 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.439842 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:18:44.442894 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.445739 28151 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:44.447732 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.453869 12641 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 51 index: 30. Preceding OpId from leader: term: 52 index: 31. (index mismatch)
I20260709 12:18:44.454651 28131 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:44.455332 28144 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.455689 28050 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:44.475301 28131 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.384536 28162 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:44.456415 28091 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:44.483606 12642 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: [44.26-44.26]
I20260709 12:18:44.484541 28091 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:44.484961 28091 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:44.485347 12644 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:18:44.485936 28091 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.487819 28135 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:44.489925 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.457384 28145 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:18:44.493121 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.495637 28088 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:44.495903 12639 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
W20260709 12:18:44.498126 27216 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 43 current term: 43 is higher.
I20260709 12:18:44.459051 12564 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:44.499688 28089 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.499920 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.500633 28060 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.459438 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.501017 28139 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:44.411566 28136 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:44.496140 28088 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:44.482944 27660 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.506978 27779 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.483678 28157 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.049s
I20260709 12:18:44.463825 12565 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 50 index: 28. (index mismatch)
I20260709 12:18:44.411142 27797 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:44.509048 28138 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260709 12:18:44.460872 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:44.464534 12571 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:44.465905 12572 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:44.479043 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:44.482941 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:44.501566 12566 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:44.403074 28127 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:18:44.514752 12568 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:18:44.515117 12637 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:44.515341 12495 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:44.512639 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:44.501410 12563 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
I20260709 12:18:44.443244 12462 leader_election.cc:400] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45.
W20260709 12:18:44.520905 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79864.4us
I20260709 12:18:44.525097 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:44.444298 28114 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.465113 28132 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:44.545872 13262 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
W20260709 12:18:44.552083 12462 connection.cc:256] RPC call timeout handler was delayed by -1.234206942000128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20260709 12:18:44.487361 28091 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:44.523269 27792 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.128s	user 0.001s	sys 0.014s
I20260709 12:18:44.560745 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:44.562336 13262 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:44.563405 28165 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:18:44.563886 13262 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 27. Preceding OpId from leader: term: 44 index: 29. (index mismatch)
I20260709 12:18:44.564574 28165 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:18:44.564973 28165 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.565351 28165 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:44.563565 28025 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:44.566872 28108 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.574411 28131 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.576864 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:44.551136 28144 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:44.584122 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.595546 12638 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
W20260709 12:18:44.603991 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 174813us
I20260709 12:18:44.544837 28151 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.042s
I20260709 12:18:44.604738 28089 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.011s
I20260709 12:18:44.547253 28137 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 46.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.049s
W20260709 12:18:44.608723 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 80426.4us
I20260709 12:18:44.610651 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.610674 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.610944 12491 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:44.613687 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.615794 12638 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:44.578248 28025 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:44.593724 28153 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.606014 28114 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.562119 27792 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.628329 12638 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:44.625066 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.597213 27779 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.030s
I20260709 12:18:44.634536 28160 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.635192 28160 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.632613 28114 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.628530 28118 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.010s
I20260709 12:18:44.638911 28178 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:18:44.628404 28060 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:44.632731 28144 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.011s
I20260709 12:18:44.630170 28153 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.642477 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.643036 28114 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.643697 28115 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:44.644272 28097 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.602s	user 0.014s	sys 0.001s
I20260709 12:18:44.644794 28097 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.645856 27765 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:44.645996 28160 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.596143 28108 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.646919 28025 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.29, Last appended by leader: 29, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.643961 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.648339 28025 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:44.635111 28141 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.025s
I20260709 12:18:44.641101 28125 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260709 12:18:44.650907 28153 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.651379 28125 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:44.645237 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:44.653223 28119 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.656152 28119 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.656503 13258 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:44.648569 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:44.658764 13262 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:44.657769 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:44.657611 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.669704 13258 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:44.673765 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.674878 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:18:44.683110 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 58120.6us
I20260709 12:18:44.684078 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:44.684718 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:44.685019 28178 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:18:44.685403 28178 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.685807 28178 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:44.688535 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:44.689647 27965 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:44.689939 28147 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.690402 27965 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:44.690618 28147 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.691457 27965 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 44.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.692394 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.693215 12564 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:18:44.693344 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.694156 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 38.
I20260709 12:18:44.643097 28144 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:44.694646 28144 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:44.695443 28144 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.22, Last appended by leader: 22, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.692628 28055 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:44.696473 28055 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:44.696820 12572 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.29-45.29]
I20260709 12:18:44.697122 28055 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:44.700119 28147 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.670400 28097 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.671614 28151 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.030s
I20260709 12:18:44.703939 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.648569 28108 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:44.705410 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.706176 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:44.711632 28178 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.715554 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.696646 28144 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:44.695662 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:44.680982 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:44.721149 13258 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:44.723111 28108 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:44.723636 28108 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.724041 28149 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:44.724053 28108 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:44.659413 28131 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:44.725024 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53539.8us
I20260709 12:18:44.725507 28131 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.726378 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:44.720356 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:44.735185 28153 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.26, Next index: 27, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:44.735956 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.736720 12572 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:18:44.736912 28139 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:44.734702 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.753669 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:44.754447 13261 raft_consensus.cc:2376] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50. Re-sending same reply.
I20260709 12:18:44.754858 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:44.753608 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:44.761148 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57288.3us
I20260709 12:18:44.762156 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:44.722328 28113 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:44.773131 12572 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:44.744367 28151 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election won for term 53
W20260709 12:18:44.776051 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.744155 28051 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:18:44.745157 28125 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:44.783866 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.785120 12637 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:44.786623 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.787585 12639 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49. Re-sending same reply.
I20260709 12:18:44.782606 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.789147 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:18:44.781154 28170 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:44.795441 28170 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:44.796005 28097 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:44.791664 28051 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:44.796530 28170 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.797328 28051 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:44.791651 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.793568 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.803526 12638 raft_consensus.cc:2376] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45. Re-sending same reply.
I20260709 12:18:44.796336 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.807600 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.804738 12645 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:18:44.816998 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:44.827219 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:44.846798 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.793078 28139 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:44.847580 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:44.848471 13255 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 29. Preceding OpId from leader: term: 49 index: 30. (index mismatch)
I20260709 12:18:44.849414 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.852190 13258 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:44.854092 12537 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:44.855284 28105 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:44.856042 28105 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:44.848240 28099 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.756s	user 0.005s	sys 0.024s
W20260709 12:18:44.859014 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 96948.4us
W20260709 12:18:44.859928 28099 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.768s	user 0.005s	sys 0.025s
I20260709 12:18:44.833266 28153 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.843272 28151 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:44.861598 28170 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:44.814157 28113 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.048s
I20260709 12:18:44.868541 28105 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.28, Last appended by leader: 28, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.879743 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.880738 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.880901 28187 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.881657 28187 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.882136 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:44.883213 12642 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:18:44.884709 28147 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:44.885113 28187 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.885525 28147 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.887651 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:44.889052 28187 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:44.889678 28187 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:44.890231 28187 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:44.891040 12637 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:44.896041 28147 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:44.900388 28100 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.663s	user 0.000s	sys 0.012s
I20260709 12:18:44.816220 28136 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.908062 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.916004 13262 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:44.842514 28165 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:44.918738 12571 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:44.918533 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:44.920264 12645 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:44.918038 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5855) took 1957 ms. Trace:
I20260709 12:18:44.873028 28131 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.923008 12570 rpcz_store.cc:276] 0709 12:18:42.960959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.961148 (+   189us) service_pool.cc:224] Handling call
0709 12:18:42.974485 (+ 13337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:42.974959 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:18:42.974989 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:42.975576 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:18:43.070816 (+ 95240us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:43.070861 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.070955 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:44.899874 (+1828919us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:18:44.903809 (+  3935us) raft_consensus.cc:1685] finished
0709 12:18:44.903835 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:44.917863 (+ 14028us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7965,"raft.queue_time_us":159126,"raft.run_cpu_time_us":9119,"raft.run_wall_time_us":66251,"spinlock_wait_cycles":59025024,"thread_start_us":94788,"threads_started":1,"wal-append.queue_time_us":189981}
I20260709 12:18:44.921072 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:44.928740 12637 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
I20260709 12:18:44.929138 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6017) took 1333 ms. Trace:
I20260709 12:18:44.929718 28151 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 51.25, Last appended by leader: 25, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.931502 12496 rpcz_store.cc:276] 0709 12:18:43.595844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.667697 (+ 71853us) service_pool.cc:224] Handling call
0709 12:18:43.679471 (+ 11774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:43.679756 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:43.879072 (+199316us) log.cc:844] Serialized 14 byte log entry
0709 12:18:44.802648 (+923576us) spinlock_profiling.cc:243] Waited 46.3 ms on lock 0x7b180046dc80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:44.928452 (+125804us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:44.928543 (+    91us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:44.928635 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:44.928838 (+   203us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:44.929042 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":119777,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":7921,"raft.queue_time_us":185,"spinlock_wait_cycles":106451328,"thread_start_us":125416,"threads_started":1}
I20260709 12:18:44.919723 28178 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:44.875260 28153 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:44.956225 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:44.949608 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:44.959028 12571 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:44.959187 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:44.952215 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4477) took 1658 ms (client timeout 2034 ms). Trace:
I20260709 12:18:44.928809 27825 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:44.960644 12488 rpcz_store.cc:269] 0709 12:18:43.293498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.360103 (+ 66605us) service_pool.cc:224] Handling call
0709 12:18:44.951906 (+1591803us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:44.952106 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":322402560}
I20260709 12:18:45.012964 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.014127 12488 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 41 because replica has last-logged OpId of term: 40 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 28.
I20260709 12:18:44.936679 28091 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:44.964843 27792 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:45.023303 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:45.027899 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.003238 28184 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:45.031872 28091 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:45.016880 27825 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.049175 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.015646 28137 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:45.050472 12563 raft_consensus.cc:1217] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Deduplicated request from leader. Original: 48.27->[50.28-50.28]   Dedup: 50.28->[]
I20260709 12:18:45.051187 28091 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 49.35, Last appended by leader: 35, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.032142 28149 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:45.093245 12563 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: [50.28-50.28]
I20260709 12:18:45.142573 12645 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 49 because replica has last-logged OpId of term: 48 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 28.
I20260709 12:18:45.021119 28178 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.043s
I20260709 12:18:45.146993 28136 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.072414 28137 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:45.159382 27797 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.650s	user 0.009s	sys 0.006s
I20260709 12:18:45.021066 28193 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.164s
I20260709 12:18:45.161434 28135 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:45.195115 28137 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:45.226182 28185 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 12:18:45.229050 28189 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:18:45.198966 28194 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.037s
I20260709 12:18:45.226779 28150 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
W20260709 12:18:45.244292 28144 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.500s	user 0.012s	sys 0.000s
I20260709 12:18:45.247635 28187 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.253502 28153 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.239277 28165 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:45.257756 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.260902 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.267287 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.199162 27797 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.236752 28150 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.272677 12489 raft_consensus.cc:606] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:45.223361 28135 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.262181 28193 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.268258 28034 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:45.288789 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:45.290053 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 369103us
I20260709 12:18:45.270347 28178 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:45.271832 28150 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:45.291638 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:45.299351 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:45.286062 28034 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:45.303304 28055 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.605s	user 0.015s	sys 0.000s
I20260709 12:18:45.303777 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:45.303927 28055 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.304659 12637 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
W20260709 12:18:45.304813 28081 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.502s	user 0.010s	sys 0.000s
I20260709 12:18:45.305889 12639 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:45.306985 28055 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:45.309355 28190 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:18:45.311719 28187 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.317719 28017 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.292904 28178 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:45.320963 12642 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:18:45.321360 28178 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.322588 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:45.322896 28178 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:45.323305 12642 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:45.323647 28205 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:45.323567 12637 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:18:45.324259 28205 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.324663 28205 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:45.325156 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:45.331043 27876 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.301769 28065 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.335181 28065 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.292579 28099 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.340534 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:45.340728 28204 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:45.342234 28034 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.25, Last appended by leader: 25, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.343623 28099 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.344694 27876 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:45.345310 27876 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.337097 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:45.350534 28099 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.351245 27876 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:45.351858 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:45.352914 28204 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:45.351505 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:45.353961 28204 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:18:45.333156 28126 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.319s	user 0.002s	sys 0.000s
I20260709 12:18:45.336088 28135 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:45.337618 28025 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.689s	user 0.018s	sys 0.000s
I20260709 12:18:45.365675 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:45.338490 28187 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:18:45.371815 13257 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:45.373061 12645 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:45.375865 27797 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:45.376895 28051 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.578s	user 0.011s	sys 0.002s
W20260709 12:18:45.375921 28108 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.651s	user 0.014s	sys 0.000s
I20260709 12:18:45.380885 28108 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.381769 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.382972 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.383699 12491 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:45.384420 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.384917 12491 raft_consensus.cc:2376] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39. Re-sending same reply.
I20260709 12:18:45.385603 28108 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.385902 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.386405 28187 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:45.387393 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.388245 12491 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:45.388520 28024 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.389212 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.391297 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.391515 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.392232 28117 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:45.392452 12491 raft_consensus.cc:2410] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 50 because replica has last-logged OpId of term: 48 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 26.
I20260709 12:18:45.392953 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.393975 13262 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:45.394543 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.395540 12491 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:45.397787 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.392524 13256 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:45.370461 28099 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:18:45.401439 13257 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:45.375759 12655 tablet_copy_source_session.cc:215] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:45.398700 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:45.292044 28193 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.405257 28108 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:45.421841 28108 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.405362 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6006) took 3395 ms. Trace:
I20260709 12:18:45.422509 28108 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:45.418987 12491 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:45.422612 12494 rpcz_store.cc:276] 0709 12:18:42.010342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.435043 (+424701us) service_pool.cc:224] Handling call
0709 12:18:42.435609 (+   566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:42.436176 (+   567us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:42.436221 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:42.437038 (+   817us) log.cc:844] Serialized 26 byte log entry
0709 12:18:42.508052 (+ 71014us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:18:42.508147 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.508335 (+   188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:45.304145 (+2795810us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:45.402051 (+ 97906us) raft_consensus.cc:1685] finished
0709 12:18:45.402076 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:45.405259 (+  3183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6340,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":128,"spinlock_wait_cycles":474852096,"thread_start_us":70177,"threads_started":1,"wal-append.queue_time_us":302105}
I20260709 12:18:45.424913 28193 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.410161 12655 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:35404 (request call id 2200) took 3398 ms. Trace:
I20260709 12:18:45.425784 28099 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:18:45.411273 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45774.8us
I20260709 12:18:45.425987 12655 rpcz_store.cc:276] 0709 12:18:42.098578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.098848 (+   270us) service_pool.cc:224] Handling call
0709 12:18:45.375246 (+3276398us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57fd3f7 00007ffbfa793036 00007ffbfa79ca3e 00007ffbf206027b 00007ffbfa79c9ba 00007ffbfa797acc 00007ffbfa792c7b 00007ffbfa7878d1 00007ffbf983b4b4 00007ffbf983b442
0709 12:18:45.410016 (+ 34770us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2074821504}
I20260709 12:18:45.424410 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.427587 12491 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:45.429178 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.377470 28051 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.427829 28175 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.430974 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.434088 12639 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 52. Current term is 53. Ops: [52.31-52.31]
I20260709 12:18:45.434401 12491 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:45.433082 27825 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.436653 28025 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:45.437788 28025 consensus_queue.cc:1243] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 44.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:18:45.431568 28175 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.402462 28117 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.440611 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6186) took 1060 ms. Trace:
I20260709 12:18:45.440294 28117 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 50.22, Last appended by leader: 22, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.442981 28003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f402dfd68ae4c8697765a76d9f1c4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:45.443377 12646 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:45.443608 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.431167 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.399816 28187 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 52.34, Last appended by leader: 34, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.418866 28065 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.441021 12566 rpcz_store.cc:276] 0709 12:18:44.381865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.382255 (+   390us) service_pool.cc:224] Handling call
0709 12:18:44.408664 (+ 26409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:44.512567 (+103903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.440309 (+927742us) spinlock_profiling.cc:243] Waited 92.7 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:45.440549 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213155200}
I20260709 12:18:45.425335 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 50 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.437345 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.448460 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:45.449765 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:45.443822 12646 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:45.452576 12494 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 54 because replica has last-logged OpId of term: 52 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 33.
I20260709 12:18:45.454777 13262 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 27. Preceding OpId from leader: term: 45 index: 29. (index mismatch)
I20260709 12:18:45.454866 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:45.455883 27957 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:45.456787 27957 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.456985 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.459236 28137 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.463567 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:18:45.444959 28125 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.642s	user 0.007s	sys 0.006s
I20260709 12:18:45.470620 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 34 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.471681 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:45.472715 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:45.473464 12494 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:45.477035 28079 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.478982 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:45.467298 28046 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:45.481946 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:45.479149 28114 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:45.483901 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:45.433117 28051 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:45.487793 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:45.490005 28088 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 50.28, Last appended by leader: 28, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.490548 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:45.492038 28088 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:45.468431 12646 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 43.30, Last appended by leader: 30, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.494138 12646 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:45.494407 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.445219 28175 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.498510 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.498865 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53272 (request call id 111) took 1307 ms. Trace:
I20260709 12:18:45.501950 12489 rpcz_store.cc:276] 0709 12:18:44.191291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.191620 (+   329us) service_pool.cc:224] Handling call
0709 12:18:45.498683 (+1307063us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540034e754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:45.498786 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2342999936}
I20260709 12:18:45.492638 28106 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.011s
I20260709 12:18:45.503857 12489 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
W20260709 12:18:45.492116 28200 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.253s	user 0.003s	sys 0.000s
I20260709 12:18:45.468189 27963 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.042s
I20260709 12:18:45.509886 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:45.510353 28087 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.414s	user 0.007s	sys 0.009s
I20260709 12:18:45.510653 27963 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:18:45.499938 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:45.511229 27963 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.511538 12566 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:45.467983 27779 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 26, Time since last communication: 0.046s
I20260709 12:18:45.514415 27779 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:45.510782 12567 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:18:45.518626 13262 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.30-43.30]
I20260709 12:18:45.519476 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:45.501143 12488 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:45.515123 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6011) took 3422 ms. Trace:
I20260709 12:18:45.521950 28106 consensus_queue.cc:1243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 43.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:45.520664 12497 rpcz_store.cc:276] 0709 12:18:42.092834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.554787 (+461953us) service_pool.cc:224] Handling call
0709 12:18:42.555225 (+   438us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:42.555677 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:42.555711 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:42.563066 (+  7355us) log.cc:844] Serialized 52 byte log entry
0709 12:18:42.772592 (+209526us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:42.789487 (+ 16895us) log.cc:844] Serialized 14 byte log entry
0709 12:18:42.802715 (+ 13228us) log.cc:844] Serialized 14 byte log entry
0709 12:18:43.283911 (+481196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:43.283997 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:43.398937 (+114940us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b18004d5180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:43.538895 (+139958us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x7b18004d5180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:18:43.539478 (+   583us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b18004d5180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:44.143061 (+603583us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b18004d5180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:45.509318 (+1366257us) spinlock_profiling.cc:243] Waited 2.38 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:45.514817 (+  5499us) raft_consensus.cc:1685] finished
0709 12:18:45.514857 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:45.515043 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":146170,"prepare.queue_time_us":154119,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":7038,"raft.queue_time_us":194272,"raft.run_cpu_time_us":445,"raft.run_wall_time_us":93181,"spinlock_wait_cycles":244184576,"thread_start_us":435307,"threads_started":2,"wal-append.queue_time_us":378519}
I20260709 12:18:45.522616 28116 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:18:45.522992 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:45.520282 12489 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 49 index: 27. (term mismatch)
I20260709 12:18:45.501703 12494 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:18:45.466750 28135 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:45.459594 27957 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 48.29, Last appended by leader: 29, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.509682 28124 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:45.486711 27825 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:45.525446 12497 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:45.526721 12568 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:45.527567 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:45.523427 27340 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:18:45.524622 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:45.528647 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.523871 12489 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 26
I20260709 12:18:45.533417 12489 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 48.27
I20260709 12:18:45.533893 12489 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:45.355207 28204 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.28, Last appended by leader: 28, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.512187 27963 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.534135 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:45.537294 12566 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:45.540987 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:45.522993 28116 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.539536 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:45.530234 28106 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:45.487322 28079 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.550159 28094 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.529971 27825 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:45.532233 27340 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.541311 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:45.539906 27957 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:45.558597 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:18:45.486814 28125 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:45.529325 28124 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:45.549320 12566 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:45.550971 27963 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:45.576543 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:45.538801 28114 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:45.570268 12646 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 30. Preceding OpId from leader: term: 44 index: 30. (term mismatch)
I20260709 12:18:45.543627 28099 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:45.541669 28157 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:45.579761 12646 pending_rounds.cc:85] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:18:45.561058 28106 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.578723 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:45.579519 28114 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.587927 28106 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:45.567343 27825 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:45.588315 28156 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:45.588826 28137 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.582810 12646 pending_rounds.cc:107] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 43.30
I20260709 12:18:45.588294 28114 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:45.580325 27734 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.578228 28065 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.580685 28151 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 26, Time since last communication: 0.013s
I20260709 12:18:45.589175 28156 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.600113 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:45.578023 28124 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:45.567157 28111 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:45.594419 12646 raft_consensus.cc:2889] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:45.594594 12567 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:45.585232 28025 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:18:45.575489 28175 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:45.578020 28046 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.601094 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:45.609395 12566 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:45.585178 28157 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.602932 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2209) took 1152 ms. Trace:
I20260709 12:18:45.610706 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:45.589708 12568 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Committing config change with OpId 39.25: config changed from index -1 to 25, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:18:45.567787 28094 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.613505 28046 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.582584 28099 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:45.610834 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:45.613003 28145 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 44 current term: 44 is higher.
I20260709 12:18:45.615345 28025 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:45.642407 28175 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:45.616025 12643 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:45.611459 28111 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.593017 27494 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 40.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.022s
I20260709 12:18:45.618274 28094 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:45.603235 28156 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:45.602571 27734 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.614689 28079 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.629910 13253 pending_rounds.cc:85] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 26
I20260709 12:18:45.603577 28065 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.676385 28161 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.568266 28205 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.611227 12646 rpcz_store.cc:276] 0709 12:18:44.450385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.453525 (+  3140us) service_pool.cc:224] Handling call
0709 12:18:44.453978 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:45.436158 (+982180us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b54001bcc20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:18:45.602575 (+166417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.602864 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127538944}
W20260709 12:18:45.638971 28191 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.448s	user 0.003s	sys 0.000s
I20260709 12:18:45.646525 13262 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 50 index: 28. (index mismatch)
I20260709 12:18:45.657143 12644 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:45.650745 13256 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.30-48.30]
I20260709 12:18:45.653070 12570 raft_consensus.cc:1275] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 49 index: 30. (index mismatch)
I20260709 12:18:45.654567 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:45.656185 28099 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:45.644057 28025 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 51.30, Last appended by leader: 30, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.670084 13253 pending_rounds.cc:107] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 42.27
I20260709 12:18:45.688060 13253 raft_consensus.cc:2889] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:45.670547 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.678597 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:45.680132 12491 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 50 index: 28. Preceding OpId from leader: term: 51 index: 29. (index mismatch)
W20260709 12:18:45.646514 28082 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 2.638s	user 0.007s	sys 0.004s
I20260709 12:18:45.610325 28157 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:45.649988 27494 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:45.689479 12489 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:45.688591 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:45.739830 13256 raft_consensus.cc:1217] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Deduplicated request from leader. Original: 48.27->[50.28-50.28]   Dedup: 50.28->[]
I20260709 12:18:45.668881 28046 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.676410 28065 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:45.599118 27340 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 51.27, Last appended by leader: 27, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.748651 13258 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:45.789804 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.810142 12489 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:45.801782 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.797070 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108545us
I20260709 12:18:45.814219 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:45.830014 13256 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: [50.28-50.28]
I20260709 12:18:45.739912 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.832710 13262 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:45.762449 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:45.837052 12497 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:45.880852 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 40.
W20260709 12:18:45.882283 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68223us
I20260709 12:18:45.747486 27734 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:45.926409 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6196) took 1192 ms. Trace:
I20260709 12:18:45.935578 12563 rpcz_store.cc:276] 0709 12:18:44.733795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.735168 (+  1373us) service_pool.cc:224] Handling call
0709 12:18:44.735563 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:45.128917 (+393354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:45.925936 (+797019us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:45.926294 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":723779840}
I20260709 12:18:45.912926 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:45.928783 28088 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.645061 28175 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 42.22, Last appended by leader: 22, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.746965 28149 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:45.596028 28116 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.649323 28111 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 43.22, Last appended by leader: 22, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.784162 28115 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.934712 28024 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
W20260709 12:18:45.935176 28191 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:18:45.963742 28114 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.949908 28088 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:45.968096 28088 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:45.932053 28161 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.969291 28161 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:45.972280 28161 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:45.954449 28116 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:45.948210 27825 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:45.947312 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.975855 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.977301 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:45.978273 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:45.979775 12563 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:45.981740 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:45.790546 27494 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:45.839604 28204 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:45.690546 28079 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:45.998652 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.000618 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:45.932331 28055 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:46.003088 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.021108 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:46.022269 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:46.023792 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:45.956267 28051 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:46.028434 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:45.987677 28065 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:46.030927 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:46.032577 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 2926) took 4258 ms (client timeout 2487 ms). Trace:
W20260709 12:18:46.039866 12569 rpcz_store.cc:269] 0709 12:18:41.774274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.101887 (+327613us) service_pool.cc:224] Handling call
0709 12:18:46.032343 (+3930456us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.032482 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244856576}
I20260709 12:18:46.021522 28092 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:45.954921 28115 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.022886 28204 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:45.937143 28108 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.066246 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.021390 28161 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.023s
I20260709 12:18:46.070977 12569 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:46.077099 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:46.026670 28055 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.083758 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.028010 27814 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 43.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.036s
I20260709 12:18:46.037573 28065 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.085800 28205 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.044202 27494 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.035513 28160 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 26, Time since last communication: 0.042s
I20260709 12:18:46.065604 28092 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.091521 28115 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.093740 27825 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:46.094621 12489 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 42 because replica has last-logged OpId of term: 40 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 28.
W20260709 12:18:46.102759 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.072793 28150 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.770s	user 0.015s	sys 0.001s
I20260709 12:18:46.164346 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.105813 28054 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:46.107389 27963 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.000s
I20260709 12:18:46.111399 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2343) took 2824 ms. Trace:
I20260709 12:18:46.115409 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.170168 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:18:46.172205 12566 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:46.116766 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6209) took 1176 ms. Trace:
I20260709 12:18:46.174054 12570 rpcz_store.cc:276] 0709 12:18:44.940778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.342880 (+402102us) service_pool.cc:224] Handling call
0709 12:18:45.343199 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:45.682720 (+339521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.116454 (+433734us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:46.116695 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":711033088}
I20260709 12:18:46.121486 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6197) took 1378 ms. Trace:
I20260709 12:18:46.123791 28108 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.124053 12637 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:46.125517 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:46.126246 28147 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:18:46.127322 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:46.127489 27957 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.550s	user 0.007s	sys 0.003s
I20260709 12:18:46.128553 28114 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.105337 28134 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.024s
I20260709 12:18:46.130236 12495 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:46.136379 12497 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:18:46.039083 28051 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.149019 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.075940 28046 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:46.213430 28055 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.165038 12569 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:18:46.167623 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.175853 12572 rpcz_store.cc:276] 0709 12:18:44.743414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.743654 (+   240us) service_pool.cc:224] Handling call
0709 12:18:44.766826 (+ 23172us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:44.779107 (+ 12281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.121151 (+1342044us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b54001b30d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:46.121407 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4764416}
I20260709 12:18:46.178264 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:46.168280 12493 rpcz_store.cc:276] 0709 12:18:43.287154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.568984 (+281830us) service_pool.cc:224] Handling call
0709 12:18:43.606378 (+ 37394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:43.660459 (+ 54081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.111173 (+2450714us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5400190d50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.111324 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34055936}
W20260709 12:18:46.181942 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56497.6us
I20260709 12:18:46.184123 12570 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:18:46.155848 28160 consensus_queue.cc:1243] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:46.181391 28150 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.201594 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.225237 12639 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:46.203105 28003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50708 (request call id 2186) took 5878 ms. Trace:
W20260709 12:18:46.204803 28156 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.000s
I20260709 12:18:46.205839 28088 raft_consensus.cc:2764] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election decision vote started in defunct term 52: won
W20260709 12:18:46.207015 28124 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.000s
W20260709 12:18:46.230964 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52839.9us
I20260709 12:18:46.206832 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:46.207505 28106 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.619s	user 0.011s	sys 0.002s
W20260709 12:18:46.210912 28135 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.678s	user 0.010s	sys 0.002s
W20260709 12:18:46.211113 28099 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.002s
I20260709 12:18:46.198990 28108 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:46.162866 28092 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.213155 28051 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.225440 12572 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 44 index: 30. (index mismatch)
I20260709 12:18:46.168497 27797 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:46.231019 12569 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 52. Current term is 53. Ops: [50.29-52.31]
I20260709 12:18:46.213595 28204 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.238204 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.239116 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:46.183868 28054 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.260416 12570 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:46.252220 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 205146us
I20260709 12:18:46.237599 28135 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:46.237383 28108 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.226662 28055 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:18:46.237844 28201 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:46.213809 28046 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.265244 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:46.273350 12572 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:46.268050 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.276438 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.279469 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.234593 28156 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.240309 28099 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.253868 28025 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.020s
W20260709 12:18:46.272194 28187 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.012s	sys 0.009s
W20260709 12:18:46.295794 28187 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.012s	sys 0.009s
I20260709 12:18:46.270728 28108 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:46.292474 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4482) took 1736 ms. Trace:
I20260709 12:18:46.236974 28124 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.301290 12491 rpcz_store.cc:276] 0709 12:18:44.555614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.502401 (+946787us) service_pool.cc:224] Handling call
0709 12:18:45.532410 (+ 30009us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:45.690058 (+157648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.292196 (+602138us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:46.292403 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338518912}
I20260709 12:18:46.306461 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6210) took 1079 ms. Trace:
I20260709 12:18:46.237350 28106 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.307200 12568 rpcz_store.cc:276] 0709 12:18:45.226676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.570703 (+344027us) service_pool.cc:224] Handling call
0709 12:18:45.587754 (+ 17051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:45.589565 (+  1811us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:18:45.991671 (+402106us) log.cc:844] Serialized 14 byte log entry
0709 12:18:46.303369 (+311698us) log.cc:844] Serialized 14 byte log entry
0709 12:18:46.305955 (+  2586us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:46.306015 (+    60us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:46.306089 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.306196 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:46.306376 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1416,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":15160,"thread_start_us":281193,"threads_started":2,"wal-append.queue_time_us":298318}
W20260709 12:18:46.310070 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.296257 28156 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.320919 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:46.322110 12568 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 52
W20260709 12:18:46.323330 28073 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 3.572s	user 0.013s	sys 0.009s
I20260709 12:18:46.325114 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 1955) took 3974 ms. Trace:
I20260709 12:18:46.326189 13261 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 44. Ops: [42.28-42.28]
I20260709 12:18:46.325500 13259 rpcz_store.cc:276] 0709 12:18:42.350328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:42.350858 (+   530us) service_pool.cc:224] Handling call
0709 12:18:42.351445 (+   587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:42.352049 (+   604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:42.352098 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:42.352912 (+   814us) log.cc:844] Serialized 26 byte log entry
0709 12:18:42.398686 (+ 45774us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:42.398733 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:42.398815 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:46.309352 (+3910537us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:46.324778 (+ 15426us) raft_consensus.cc:1685] finished
0709 12:18:46.324798 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:46.325042 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17373,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":20150,"spinlock_wait_cycles":544434560,"thread_start_us":45206,"threads_started":1,"wal-append.queue_time_us":327704}
I20260709 12:18:46.226943 28003 rpcz_store.cc:276] 0709 12:18:40.335425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.339993 (+  4568us) service_pool.cc:224] Handling call
0709 12:18:40.687814 (+347821us) ts_tablet_manager.cc:934] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
0709 12:18:45.441051 (+4753237us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:45.964801 (+523750us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:45.964933 (+   132us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:46.185303 (+220370us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:46.202498 (+ 17195us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:46.202920 (+   422us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:46.202997 (+    77us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":12801,"tablet-copy.queue_time_us":250098,"thread_start_us":278204,"threads_started":2}
I20260709 12:18:46.330679 28003 tablet_copy_client.cc:806] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:18:46.331468 28003 tablet_copy_client.cc:670] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:18:46.339775 12568 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:18:46.343478 12572 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:46.347507 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:46.350633 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:46.351590 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.340674 28065 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.318367 27957 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.035s
W20260709 12:18:46.359023 28219 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:18:46.356442 28071 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:18:46.385723 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.387089 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.387106 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.385882 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.358764 27216 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:46.391855 12496 raft_consensus.cc:2368] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:18:46.392959 12491 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:46.361970 28187 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.036s
I20260709 12:18:46.395994 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.396209 28071 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:46.397163 12646 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:46.395898 27216 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.397505 27963 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.397737 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.398342 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.408581 12641 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:46.399403 12640 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.26-46.26]
I20260709 12:18:46.400301 28046 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:46.361835 28205 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.031s
I20260709 12:18:46.419353 12644 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.23-44.23]
I20260709 12:18:46.406004 28054 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.392961 28150 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.413484 27963 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.425711 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.427099 27963 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.427335 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.428053 12646 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:46.432825 28225 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:18:46.436087 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.438473 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.439314 28106 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.439152 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.439435 12644 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:46.440313 12493 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:46.443712 28124 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.447367 12646 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:46.440060 28099 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:46.438472 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.967s (SENT) [suppressed 43 similar messages]
I20260709 12:18:46.450603 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.451614 12646 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:46.452672 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.453250 12646 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:46.457201 27940 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.459254 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:46.449393 28157 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.514s	user 0.008s	sys 0.006s
I20260709 12:18:46.434156 28071 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.30, Last appended by leader: 30, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.467733 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.456913 28034 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:46.462347 27940 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.474313 12497 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:46.480716 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.485155 27940 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:46.486714 28220 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.192s	user 0.003s	sys 0.000s
W20260709 12:18:46.487377 28224 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:18:46.491672 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.501003 28122 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:46.501540 28122 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:46.502496 28122 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.27, Last appended by leader: 27, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.503674 28122 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:46.506615 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2202) took 1136 ms. Trace:
I20260709 12:18:46.507076 13261 rpcz_store.cc:276] 0709 12:18:45.370266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.370482 (+   216us) service_pool.cc:224] Handling call
0709 12:18:46.325921 (+955439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:46.326593 (+   672us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.506307 (+179714us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b5400537850. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.506495 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196081024}
I20260709 12:18:46.490885 28158 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:46.514528 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.513141 28158 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.497298 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.518564 12644 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:46.520126 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.521473 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.523391 12644 raft_consensus.cc:2368] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52.
I20260709 12:18:46.524400 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.522625 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.527040 12638 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:18:46.528012 12644 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:18:46.530591 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4474) took 3238 ms (client timeout 1967 ms). Trace:
W20260709 12:18:46.531433 12492 rpcz_store.cc:269] 0709 12:18:43.292454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:43.355028 (+ 62574us) service_pool.cc:224] Handling call
0709 12:18:46.530366 (+3175338us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b5400144bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.530503 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135750016}
I20260709 12:18:46.532845 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.537166 13255 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.29-48.29]
I20260709 12:18:46.538563 12641 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:46.539207 12644 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:46.540494 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 1972) took 1092 ms. Trace:
I20260709 12:18:46.540910 13258 rpcz_store.cc:276] 0709 12:18:45.447867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.449291 (+  1424us) service_pool.cc:224] Handling call
0709 12:18:46.540291 (+1091000us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.540430 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":679429376}
I20260709 12:18:46.549270 12495 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:46.550158 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.550426 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.551314 12495 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:46.552027 12642 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:18:46.529340 12494 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.22, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:18:46.552791 12462 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f402dfd68ae4c8697765a76d9f1c4e6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:46.553591 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.554268 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:46.554554 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:46.555117 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116694us
W20260709 12:18:46.555989 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.034s (SENT)
I20260709 12:18:46.556075 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.556705 12638 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:46.557199 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:46.552314 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.554666 12497 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:46.558086 12495 raft_consensus.cc:2376] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43. Re-sending same reply.
I20260709 12:18:46.555503 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:46.559319 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.560308 12495 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:46.559010 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:46.561329 12497 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:46.561733 12461 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:46.563458 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:46.556721 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3991681479997169s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20260709 12:18:46.560479 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.570117 13255 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: [47.34-49.35]
I20260709 12:18:46.590883 28150 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:46.594540 28150 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:46.594966 28150 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:46.595549 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.595849 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5883) took 1166 ms. Trace:
I20260709 12:18:46.596285 12644 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 46
I20260709 12:18:46.600418 28093 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.26, Last appended by leader: 26, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.604517 28150 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.476548 28157 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.567945 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:46.592157 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:46.573061 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:46.596451 12567 rpcz_store.cc:276] 0709 12:18:45.429182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.519227 (+ 90045us) service_pool.cc:224] Handling call
0709 12:18:46.595606 (+1076379us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b54003acd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.595770 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171102080}
I20260709 12:18:46.589380 12646 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:46.613547 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:46.609901 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.614033 12497 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:18:46.606318 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3631681479998861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
I20260709 12:18:46.612040 12641 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:46.615826 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3505681479998657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260709 12:18:46.616276 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.570650 13261 raft_consensus.cc:1217] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Deduplicated request from leader. Original: 40.26->[41.27-44.29]   Dedup: 43.28->[44.29-44.29]
W20260709 12:18:46.554353 28220 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:18:46.566429 28153 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:18:46.618952 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:46.617473 13261 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: [41.27-44.29]
I20260709 12:18:46.617717 12495 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.22-41.22]
I20260709 12:18:46.564292 28016 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:46.625870 12638 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:46.627370 12495 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:18:46.564889 28124 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:46.490513 27951 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Committing config change with OpId 39.25: config changed from index -1 to 25, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:18:46.630451 12639 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:46.568411 28205 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:46.619246 28233 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 12:18:46.642673 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.642166 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:46.620893 28061 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:18:46.649075 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.649099 12492 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:46.645705 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.623538 28093 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:46.650372 12641 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 50
I20260709 12:18:46.642536 28160 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:18:46.624181 28108 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.674252 12571 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:46.631197 27963 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.032s
I20260709 12:18:46.633407 28016 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.648144 28134 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 28, Time since last communication: 0.028s
I20260709 12:18:46.681102 12641 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:46.647717 28205 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.682914 12536 leader_election.cc:400] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:18:46.683259 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:46.685009 12536 leader_election.cc:403] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Cancelling election due to peer responding with higher term
I20260709 12:18:46.685719 12641 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:46.692585 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:46.693933 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:46.677739 28169 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.709221 12641 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:18:46.703466 28060 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:46.689955 28091 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 26, Time since last communication: 0.026s
I20260709 12:18:46.713775 27779 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:46.735672 28242 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:18:46.738394 28242 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:46.723109 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:46.710906 28169 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.738829 28242 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:46.741354 28169 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.742846 28016 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.744870 12492 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:46.726637 28108 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:46.726938 28178 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:46.731406 28150 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.733189 28091 consensus_queue.cc:1243] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 50.28 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:18:46.720319 28157 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.752764 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:46.751230 28178 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:46.753957 27779 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:46.793828 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5716) took 1195 ms. Trace:
I20260709 12:18:46.794631 28205 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.776391 28244 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:46.802537 28244 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:46.763128 28178 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.786216 27825 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:46.794485 12642 rpcz_store.cc:276] 0709 12:18:45.598434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.598620 (+   186us) service_pool.cc:224] Handling call
0709 12:18:45.611482 (+ 12862us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.552621 (+941139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:46.793416 (+240795us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b540018d3d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:46.793718 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2162217728}
I20260709 12:18:46.778654 28125 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:46.804454 12497 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
W20260709 12:18:46.805233 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52594.8us
I20260709 12:18:46.813763 27779 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:18:46.816162 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:46.802986 28244 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:46.816560 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.823755 28125 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:46.824652 27779 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:46.817339 27684 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:46.825337 27779 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:46.814487 28091 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:46.825484 27684 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.832397 28244 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.29, Last appended by leader: 29, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.834367 28242 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.811214 28134 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.810312 27825 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:46.845795 28244 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:46.818817 28178 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:46.861565 28248 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:46.835842 28136 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.881065 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2217) took 1578 ms. Trace:
I20260709 12:18:46.881814 12645 rpcz_store.cc:276] 0709 12:18:45.302618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.339001 (+ 36383us) service_pool.cc:224] Handling call
0709 12:18:46.880744 (+1541743us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:46.880962 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72323968}
I20260709 12:18:46.888017 12495 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [44.23-44.23]
I20260709 12:18:46.826193 27765 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 43.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.048s
I20260709 12:18:46.891825 28136 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.905022 28136 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:46.906688 28242 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:46.854730 28134 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.910244 27684 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:46.912259 28134 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:46.921983 27779 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.26, Last appended by leader: 26, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.938623 28016 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:46.939308 28016 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.941802 28016 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.943827 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:46.944928 12497 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:46.884006 27825 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 49.35, Last appended by leader: 35, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.946993 27825 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:46.950251 28158 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:46.955894 27660 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:46.956518 27660 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:46.956750 28209 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.287s	user 0.006s	sys 0.000s
I20260709 12:18:46.957485 27660 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 44.23, Last appended by leader: 23, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.855175 28125 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:46.959565 27660 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:46.885679 28124 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:46.870167 28248 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:46.963275 28248 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:46.963994 27216 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:46.973328 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 157281us
I20260709 12:18:46.974874 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:46.976351 13253 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:46.976774 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:46.976197 12489 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:46.981276 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:46.982106 13255 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:46.980501 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:46.984819 12536 leader_election.cc:400] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52.
I20260709 12:18:46.984719 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:46.985828 12536 leader_election.cc:403] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:46.984866 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:46.986768 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:46.986370 13257 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:18:46.989859 12462 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3eadf951b63a43fd993cea2c96d5214f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:18:46.980084 28242 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:46.980050 28248 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:46.985512 28113 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:46.988222 28091 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:46.995057 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:46.991521 28205 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.005s
I20260709 12:18:46.990513 27765 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 51. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52.
I20260709 12:18:46.989451 28124 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:18:47.030128 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:46.989895 28024 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:46.999291 28242 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.036s
I20260709 12:18:47.037880 27660 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:47.047420 27660 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:47.038605 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:47.032217 27684 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:47.056213 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.048928 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:47.050015 28024 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:47.049099 28124 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:47.066639 12489 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:47.067415 12491 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:47.069697 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.070640 12489 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:47.072057 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:47.069905 12491 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:47.074013 12489 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:47.057669 28060 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 49. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:18:47.077046 12497 raft_consensus.cc:1217] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Deduplicated request from leader. Original: 46.26->[48.27-51.29]   Dedup: 48.27->[49.28-51.29]
I20260709 12:18:47.078043 12497 pending_rounds.cc:85] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 27
I20260709 12:18:47.078365 12497 pending_rounds.cc:107] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 50.28
I20260709 12:18:47.078657 12497 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:47.060326 28242 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:47.056120 27684 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.075716 28093 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:18:47.095012 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:47.092053 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.076453 28024 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:47.020423 28113 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.075721 28124 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:47.097091 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.089777 28242 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:47.101456 28242 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.105028 12489 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 51 index: 30. Preceding OpId from leader: term: 53 index: 31. (index mismatch)
I20260709 12:18:47.095341 28117 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:47.112939 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:47.098354 28093 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:47.110812 28060 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:47.114768 27779 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:47.123742 12564 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 48 because replica has last-logged OpId of term: 46 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 25.
I20260709 12:18:47.123661 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:47.122375 28251 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:18:47.139793 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.136226 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:47.144826 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.136703 28252 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:47.091109 27660 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.154417 27340 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:47.157512 28258 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:47.158219 28258 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:47.160684 28258 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.146651 28252 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:47.162057 28252 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:47.150691 28244 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:47.163180 28244 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:47.163666 28244 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:47.133572 28060 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.111385 28071 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.040s
I20260709 12:18:47.169961 28060 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.139448 28091 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:47.171002 28134 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.171800 28134 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.171981 28091 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:47.174500 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5715) took 1576 ms (client timeout 1842 ms). Trace:
W20260709 12:18:47.152089 28249 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.159s	user 0.001s	sys 0.001s
I20260709 12:18:47.178185 12643 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 45. Ops: []
I20260709 12:18:47.180204 27684 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:47.184650 28256 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:18:47.193740 28134 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:47.194175 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:47.197870 28248 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:47.198174 28183 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:47.198776 28183 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:47.199874 28183 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.27, Last appended by leader: 27, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:47.175028 12637 rpcz_store.cc:269] 0709 12:18:45.597727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.597883 (+   156us) service_pool.cc:224] Handling call
0709 12:18:47.174258 (+1576375us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54000a7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:47.174431 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185550080}
W20260709 12:18:47.210134 12488 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6026) took 2703 ms (client timeout 1925 ms). Trace:
I20260709 12:18:47.201375 28183 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:47.215116 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:47.211047 12610 connection.cc:256] RPC call timeout handler was delayed by -1.070305560999941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
W20260709 12:18:47.210861 12488 rpcz_store.cc:269] 0709 12:18:44.507045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.029237 (+522192us) service_pool.cc:224] Handling call
0709 12:18:47.209795 (+2180558us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400368154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:47.210019 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036497280}
I20260709 12:18:47.216289 12491 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:47.216998 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.217255 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.218420 12488 raft_consensus.cc:2368] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
W20260709 12:18:47.219712 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.219909 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.221052 12488 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:47.226626 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.223618 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.228451 12491 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:47.229559 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:47.227491 12496 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:47.243852 12496 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:18:47.237787 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:47.253544 12494 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:47.202561 28178 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 51.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.033s
I20260709 12:18:47.259833 28252 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.262931 27494 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:47.266791 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6211) took 2005 ms (client timeout 1512 ms). Trace:
I20260709 12:18:47.267026 12565 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 47 index: 26. (term mismatch)
I20260709 12:18:47.267367 12565 pending_rounds.cc:85] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 25
I20260709 12:18:47.267702 12565 pending_rounds.cc:107] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 46.26
I20260709 12:18:47.267935 12565 raft_consensus.cc:2889] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:47.274859 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:47.280968 12491 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
W20260709 12:18:47.267266 12564 rpcz_store.cc:269] 0709 12:18:45.261231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.290652 (+ 29421us) service_pool.cc:224] Handling call
0709 12:18:47.266519 (+1975867us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54002472d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:47.266714 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4193476992}
I20260709 12:18:47.287178 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:47.288535 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:47.211588 28016 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:47.218578 28205 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.314831 28205 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.316576 13261 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:47.316566 28124 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.317781 28205 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.318526 28205 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:18:47.318917 28205 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:47.225495 27779 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:47.256520 12488 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:18:47.136886 28093 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 38.27, Last appended by leader: 27, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:47.290778 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 134731us
I20260709 12:18:47.323282 12492 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:47.236083 27765 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:47.329891 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.331414 12569 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 44 index: 23. (index mismatch)
I20260709 12:18:47.326157 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5877) took 1930 ms. Trace:
I20260709 12:18:47.320609 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:47.333972 12565 rpcz_store.cc:276] 0709 12:18:45.396130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.639962 (+243832us) service_pool.cc:224] Handling call
0709 12:18:47.266664 (+1626702us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:47.268401 (+  1737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.325916 (+ 57515us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54002472d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:47.326087 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3740154112}
W20260709 12:18:47.336072 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49009.6us
W20260709 12:18:47.352317 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52386.1us
I20260709 12:18:47.352851 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:47.319850 28205 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 52.31, Last appended by leader: 31, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.353886 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:47.354825 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 53 index: 31. (index mismatch)
I20260709 12:18:47.329578 12492 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:47.279587 28071 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.364301 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.356477 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:47.357290 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:47.369140 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.355609 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:47.370020 12488 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:47.300554 27951 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:47.214123 28125 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.100s
I20260709 12:18:47.371764 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.372925 12494 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 53 because replica has last-logged OpId of term: 51 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 26.
I20260709 12:18:47.362265 12492 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.29, Last appended by leader: 29, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.388339 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.386011 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:47.351917 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.394171 12569 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:47.394403 12492 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:47.375136 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.321728 28016 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:47.397176 28213 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.721s	user 0.014s	sys 0.000s
I20260709 12:18:47.398029 12386 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 42, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 42 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:47.399144 28016 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:47.402964 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.402434 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:47.355273 28218 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:47.366847 27340 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:47.366338 28111 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:47.390232 28028 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.426949 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.439237 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.421711 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:47.401544 28242 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.388218 27963 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:47.419615 28205 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:47.443495 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:47.425581 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.452091 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.524121 12564 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:47.524369 12568 raft_consensus.cc:2410] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 53 because replica has last-logged OpId of term: 51 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 26.
I20260709 12:18:47.337881 28244 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:47.525013 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65114.8us
I20260709 12:18:47.464788 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.386976 27951 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:47.529260 27951 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:47.357218 27765 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.428275 27216 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:18:47.534667 12567 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:47.472893 12488 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:47.444262 28248 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:18:47.356273 27779 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.388506 28071 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.538087 28071 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:47.440383 28153 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.099s
I20260709 12:18:47.534492 28158 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:47.517156 28028 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.425801 28016 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:47.542887 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.527112 12569 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:47.543083 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6207) took 2665 ms. Trace:
I20260709 12:18:47.549604 12571 rpcz_store.cc:276] 0709 12:18:44.877971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:44.960149 (+ 82178us) service_pool.cc:224] Handling call
0709 12:18:45.020142 (+ 59993us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:46.676200 (+1656058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.542720 (+866520us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:47.542997 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3791507328}
W20260709 12:18:47.553797 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 348781us
W20260709 12:18:47.558738 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 335964us
I20260709 12:18:47.561081 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:47.532866 27765 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.448720 28218 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:47.548477 27825 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.600s	user 0.010s	sys 0.005s
I20260709 12:18:47.526731 28242 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.580415 27765 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.555595 28153 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.561908 27809 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:47.558871 28158 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:47.594944 28158 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.597738 28153 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.600787 28153 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.602555 28218 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.603384 27765 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.606020 27809 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.587143 28252 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.610376 28124 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:47.556620 28016 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.613112 28016 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:47.613931 27765 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.615537 28028 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:47.616353 28028 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.613454 12565 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:47.585004 27660 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.007s
W20260709 12:18:47.613104 28024 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.505s	user 0.012s	sys 0.000s
I20260709 12:18:47.616997 28028 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:47.654083 12492 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 45 index: 29. (term mismatch)
I20260709 12:18:47.654661 12492 pending_rounds.cc:85] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 28
I20260709 12:18:47.654979 12492 pending_rounds.cc:107] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 44.29
I20260709 12:18:47.655314 12492 raft_consensus.cc:2889] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:47.656183 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4507) took 1233 ms. Trace:
I20260709 12:18:47.648308 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.651364 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.633702 28016 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:47.669971 12569 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:47.671037 28016 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:47.671638 28016 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:47.672039 28016 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:47.665123 28117 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.553s	user 0.006s	sys 0.008s
I20260709 12:18:47.674616 12490 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:18:47.675158 28122 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.675886 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6037) took 2223 ms. Trace:
I20260709 12:18:47.675752 28122 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.676328 12490 rpcz_store.cc:276] 0709 12:18:45.452186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.494116 (+ 41930us) service_pool.cc:224] Handling call
0709 12:18:47.675718 (+2181602us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:47.675824 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5011939200}
I20260709 12:18:47.677081 28178 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:47.643936 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.684230 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.685956 12492 rpcz_store.cc:276] 0709 12:18:46.422823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.745940 (+323117us) service_pool.cc:224] Handling call
0709 12:18:46.746315 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:47.323176 (+576861us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b540029aa20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:18:47.655865 (+332689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:47.656124 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34518912}
I20260709 12:18:47.690435 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.692173 12569 raft_consensus.cc:2393] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 40: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:47.664911 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 335366us
I20260709 12:18:47.637441 27660 consensus_queue.cc:1243] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 40.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 28
I20260709 12:18:47.710259 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.681340 27779 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:47.653246 28024 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.737030 12563 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 24. Preceding OpId from leader: term: 53 index: 26. (index mismatch)
I20260709 12:18:47.734278 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 43 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.711982 27660 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.682564 28178 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.744915 28122 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.746507 28242 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.791904 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.809293 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:47.809365 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.802237 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.743201 27809 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:47.752445 28244 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:47.825980 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.828171 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.830315 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 52. Ops: []
I20260709 12:18:47.834118 28242 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 41.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.008s
I20260709 12:18:47.812072 28252 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:47.757860 28267 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:47.841053 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.844336 12569 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
I20260709 12:18:47.852339 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:47.853174 12569 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 35.
W20260709 12:18:47.853552 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.856940 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:47.859899 12569 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 23. Preceding OpId from leader: term: 45 index: 23. (term mismatch)
I20260709 12:18:47.860646 12569 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 22
I20260709 12:18:47.861243 12569 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 44.23
I20260709 12:18:47.861697 12569 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:47.861864 12572 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
W20260709 12:18:47.866039 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 442638us
I20260709 12:18:47.763077 27660 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:47.878474 28262 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:18:47.840579 28252 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:47.885229 28252 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:47.899597 28264 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:18:47.902222 28270 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:47.903185 28270 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:47.904994 28263 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.193s	user 0.002s	sys 0.000s
I20260709 12:18:47.904688 28270 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:47.882153 28122 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:18:47.940762 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 511708us
I20260709 12:18:47.853199 28060 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.089s
W20260709 12:18:47.991523 28106 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 45 current term: 45 is higher.
W20260709 12:18:47.909404 28183 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.650s	user 0.006s	sys 0.007s
W20260709 12:18:47.909062 27494 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.646s	user 0.010s	sys 0.004s
I20260709 12:18:48.012599 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:47.906090 28028 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:47.923275 27833 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:48.027878 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.030s	sys 0.033s
I20260709 12:18:48.022866 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:47.869065 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:47.852108 28267 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.029759 12493 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:47.859045 28244 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.164s
I20260709 12:18:48.010166 28111 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:48.037793 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.018532 27797 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.000240 27951 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.053194 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:48.052619 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 503516us
I20260709 12:18:48.015259 28106 consensus_queue.cc:1048] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.025s
W20260709 12:18:48.039305 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.007s	sys 0.024s
W20260709 12:18:48.062747 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.009s	sys 0.025s
W20260709 12:18:48.063617 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 709827us
I20260709 12:18:48.063835 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.064967 13217 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:48.055521 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 495436us
W20260709 12:18:48.054399 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.031s	sys 0.033s
W20260709 12:18:48.054345 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.059015 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:48.071681 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:48.054453 28114 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.014s	sys 0.048s
I20260709 12:18:48.036301 28117 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.030066 28270 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:48.031476 27963 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:48.038658 27494 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:48.077699 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.079710 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.082401 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.090117 12567 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:48.091899 12638 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.28-43.28]
I20260709 12:18:48.052106 27216 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:48.095184 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.095880 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.057956 28099 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:48.038487 27340 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:48.065804 28258 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.006s	sys 0.027s
I20260709 12:18:48.101303 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.103736 12644 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:48.076009 28024 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:48.086359 28244 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.086376 27825 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:48.104981 28258 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.006s	sys 0.027s
W20260709 12:18:48.088778 28184 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.014s	sys 0.032s
I20260709 12:18:48.086758 28218 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.080524 28183 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:18:48.112052 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:48.162678 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.163880 12638 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:48.108901 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:48.164585 12644 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:18:48.164973 12643 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:18:48.075192 28091 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.009s	sys 0.025s
I20260709 12:18:48.113665 12639 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:18:48.165382 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:48.163050 27494 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:48.081429 28265 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.068s
W20260709 12:18:48.099715 28071 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.023s	sys 0.042s
I20260709 12:18:48.142874 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.176610 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.144471 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.037392 27833 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.093034 28124 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:48.086292 28122 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
W20260709 12:18:48.074501 28114 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.014s	sys 0.049s
I20260709 12:18:48.154343 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:48.154343 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.158560 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.187167 12637 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 54 because replica has last-logged OpId of term: 51 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 24.
W20260709 12:18:48.161481 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47699.9us
I20260709 12:18:48.188349 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:48.102660 28178 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.087978 28117 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.172367 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.172418 12642 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:18:48.175124 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.176759 12640 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:48.055219 27797 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.193121 12638 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:48.199755 27797 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.200424 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:48.200666 27216 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 50.22, Last appended by leader: 22, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.200464 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:48.173516 28218 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.203685 28218 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:48.204303 27660 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.205101 28287 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:18:48.206691 28287 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:18:48.207329 28287 consensus_queue.cc:1243] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:18:48.208459 28287 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:18:48.208741 12643 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:48.156186 28111 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.173099 28153 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.041s
I20260709 12:18:48.171178 28183 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.218338 27765 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:48.218168 28183 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:48.169807 28091 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.010s	sys 0.027s
I20260709 12:18:48.194967 28269 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:48.190184 28061 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.009s	sys 0.030s
I20260709 12:18:48.218801 27765 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:48.220741 28183 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:48.221174 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4502) took 1801 ms (client timeout 1751 ms). Trace:
I20260709 12:18:48.220875 27765 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.29, Last appended by leader: 29, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.226755 27340 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 44.23, Last appended by leader: 23, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.140652 28276 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:48.229076 12569 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 51 index: 30. Preceding OpId from leader: term: 53 index: 31. (index mismatch)
I20260709 12:18:48.156083 27825 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.230536 27833 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:48.230731 28267 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.210168 12386 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 38, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 38 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:48.233563 27951 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:48.164382 28184 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.014s	sys 0.032s
I20260709 12:18:48.236475 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.351123 28276 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:48.241999 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.177450 28071 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.023s	sys 0.042s
I20260709 12:18:48.246891 12564 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:18:48.210328 28113 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.236637 28158 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:48.248304 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:48.248179 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.382922 13253 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:48.154430 28244 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.282068 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:48.278616 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.020s	sys 0.011s
I20260709 12:18:48.284948 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: []
W20260709 12:18:48.384004 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.020s	sys 0.011s
I20260709 12:18:48.385130 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:48.254458 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:48.386158 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.386402 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.286377 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:48.293985 28279 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.294167 27765 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:48.302731 12641 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 25. Preceding OpId from leader: term: 53 index: 26. (index mismatch)
I20260709 12:18:48.251834 28283 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:48.338538 28117 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.334271 28092 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:48.340405 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.342118 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.433972 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:48.189767 27963 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 48.28, Last appended by leader: 28, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:48.347729 28205 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.803s	user 0.011s	sys 0.004s
I20260709 12:18:48.236855 12643 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
W20260709 12:18:48.275017 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.003s	sys 0.017s
W20260709 12:18:48.441895 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.004s	sys 0.018s
I20260709 12:18:48.363118 28273 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:48.368579 28183 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.369269 28061 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:48.370889 27825 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:48.374184 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 301638us
I20260709 12:18:48.446230 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.352090 28114 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:48.237375 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49042.5us
I20260709 12:18:48.440984 12386 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 46 to 49. New cstate: current_term: 49 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:48.450155 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.382848 28158 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:48.385367 28244 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.386013 28285 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:48.386612 28286 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.287694 12637 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:48.387264 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:48.460155 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72986.2us
I20260709 12:18:48.461673 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:48.387584 12640 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:48.463598 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:48.464465 12492 raft_consensus.cc:2368] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:48.465484 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.466352 12488 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:48.384029 13259 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:48.385424 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:48.263569 12493 rpcz_store.cc:269] 0709 12:18:46.419653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:46.442778 (+ 23125us) service_pool.cc:224] Handling call
0709 12:18:48.220962 (+1778184us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:48.221085 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3606394624}
W20260709 12:18:48.279425 13220 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.005s	sys 0.014s
W20260709 12:18:48.467974 13220 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.908s	user 0.005s	sys 0.014s
W20260709 12:18:48.468617 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83271.2us
I20260709 12:18:48.474438 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:48.409889 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
W20260709 12:18:48.430169 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 351022us
I20260709 12:18:48.435189 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.478258 12642 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:48.436396 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:48.478535 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.444693 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.479193 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:48.397815 28055 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.479931 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.397169 28024 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.480631 13260 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:48.481089 28024 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:48.481515 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.396870 28278 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.455958 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.457702 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.483927 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:48.479142 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.433380 28092 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:48.451447 28242 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:48.487428 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.507587 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:48.508229 12490 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:18:48.508371 12637 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:48.492429 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.509377 13253 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:18:48.399117 28276 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.494379 13260 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:48.518013 12571 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:48.450948 28248 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.440276 28272 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.506155 12566 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
W20260709 12:18:48.490062 28292 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:18:48.488495 12639 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:48.480343 28055 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.511464 13254 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:48.511934 27340 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:18:48.512590 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.514735 13255 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:18:48.531358 12642 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:48.533100 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.535671 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:48.514536 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.482976 28283 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20260709 12:18:48.507239 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51369.7us
I20260709 12:18:48.522028 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.522255 12566 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:48.519317 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:48.515189 28284 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:18:48.549571 13260 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:48.454103 28114 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.564875 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.528138 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.538200 13253 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 50 because replica has last-logged OpId of term: 49 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 28.
I20260709 12:18:48.454878 28183 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.570590 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.521282 12386 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 39 to 42. New cstate: current_term: 42 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:48.487762 28117 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:48.555989 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:48.571910 12536 leader_election.cc:403] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:18:48.560319 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.487370 28296 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:48.554174 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:48.577553 12493 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:48.562925 12640 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:48.549505 12489 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:48.524613 13261 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 33. Preceding OpId from leader: term: 50 index: 36. (index mismatch)
I20260709 12:18:48.425669 28279 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:48.567759 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93446.1us
I20260709 12:18:48.569538 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.567102 12461 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:48.558459 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 49 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.565613 13262 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:48.576131 13254 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 52 index: 31. Preceding OpId from leader: term: 53 index: 31. (term mismatch)
I20260709 12:18:48.584322 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:48.585215 13254 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 30
I20260709 12:18:48.585549 13254 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 52.31
I20260709 12:18:48.586253 13254 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:48.586315 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.580925 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:48.587214 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.587728 13261 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:48.577971 12637 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:48.544787 28171 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:48.586797 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.592579 12489 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:18:48.584878 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:48.595111 12492 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.29-41.29]
I20260709 12:18:48.584874 12461 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:48.559844 27809 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:48.603643 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:48.604144 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 45 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.604457 12494 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:48.587857 12491 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:48.606384 13257 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:48.488371 28285 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.610272 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:48.574961 27833 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:48.576223 27779 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.574481 28117 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:48.623551 12642 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:48.504752 28278 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.486855 28286 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.639360 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:48.552304 28252 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.051s
I20260709 12:18:48.615422 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.631317 27833 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:48.605444 28153 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:18:48.535182 28248 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.644308 27809 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:48.652783 27809 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:48.657037 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72789.3us
W20260709 12:18:48.658074 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.609715 28028 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.622267 28093 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:48.632697 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.680037 12489 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:48.583069 28113 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.065s
W20260709 12:18:48.606585 28016 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.934s	user 0.017s	sys 0.000s
W20260709 12:18:48.623626 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63400.7us
I20260709 12:18:48.535570 28272 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.584669 28273 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:48.648406 12491 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:48.723686 12537 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:48.550341 28111 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.053s
I20260709 12:18:48.574836 28158 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.569007 27951 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.056s
I20260709 12:18:48.632102 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.650022 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.662227 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:48.664422 13261 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:48.637552 12493 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:48.755713 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.756584 12493 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:48.759595 12493 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:18:48.639716 12496 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:48.631517 28024 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.611948 28281 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:48.636360 28117 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:48.622748 28296 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:48.681468 13259 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:48.684504 12494 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:48.640424 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:48.594137 27660 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.057s
I20260709 12:18:48.657671 27833 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:48.794376 28296 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:48.795271 28305 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 12:18:48.797106 28296 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:48.657904 28184 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:48.632045 27340 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:48.747756 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.798818 12491 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:18:48.801322 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6218) took 1101 ms. Trace:
I20260709 12:18:48.801769 12571 rpcz_store.cc:276] 0709 12:18:47.700456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.702840 (+  2384us) service_pool.cc:224] Handling call
0709 12:18:47.745668 (+ 42828us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:48.517719 (+772051us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b5400280120. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f51c0 00007ffbf57f6231 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c
0709 12:18:48.522985 (+  5266us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.801079 (+278094us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:48.801259 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1635169408}
I20260709 12:18:48.807126 28117 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.811977 12492 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 50: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:48.754522 13254 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 46 because replica has last-logged OpId of term: 44 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 25.
I20260709 12:18:48.611075 28283 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:48.664893 28258 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:48.580992 28092 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 45.24, Last appended by leader: 24, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:48.767551 28306 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.001s	sys 0.001s
I20260709 12:18:48.768052 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:48.679483 28093 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:48.615753 28274 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.114s
W20260709 12:18:48.768915 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 106773us
I20260709 12:18:48.784721 13255 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:48.647259 27779 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:48.790894 28303 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
W20260709 12:18:48.792249 28310 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:18:48.658583 28153 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:48.750032 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.813441 12493 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
W20260709 12:18:48.816519 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 231704us
I20260709 12:18:48.892259 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:48.831418 28308 ts_tablet_manager.cc:933] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:18:48.784006 28312 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.133s	user 0.001s	sys 0.001s
I20260709 12:18:48.769147 28281 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:48.683393 28028 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.881685 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.880168 28093 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:48.883301 12637 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:48.885043 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:48.883235 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.742115 28171 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
W20260709 12:18:48.888038 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120078us
I20260709 12:18:48.894030 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:48.819727 13256 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:48.819988 28273 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:48.859017 27809 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:48.904670 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:48.729497 28016 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:48.906983 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.909214 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:48.882189 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 241577us
I20260709 12:18:48.926846 28122 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:48.891392 28092 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:48.890461 28111 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:48.890026 28276 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:48.896412 27967 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:18:48.912429 28281 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:48.895784 28153 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:48.941332 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:48.943356 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:48.905706 28252 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:48.874822 28258 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:48.882153 27825 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.046s
I20260709 12:18:48.955770 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.946303 28125 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:48.958757 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.987874 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.996572 12565 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:48.959712 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:48.961416 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:48.966570 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57474.6us
I20260709 12:18:48.936885 12642 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:18:48.998199 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:48.929414 28273 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:48.943611 28271 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:18:48.988438 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.000470 12564 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:48.964126 28055 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:48.957939 12489 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [45.24-45.24]
I20260709 12:18:48.920758 28296 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.059s
I20260709 12:18:48.979908 28171 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:48.991276 28205 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:48.996658 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:48.963356 27660 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:49.004205 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:48.859685 28283 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:48.965517 27951 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:48.972731 27967 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:49.009011 12572 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:49.001397 12496 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 43 because replica has last-logged OpId of term: 40 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 28.
I20260709 12:18:48.981998 28252 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:49.051165 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.020s	sys 0.025s
I20260709 12:18:48.983901 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:49.053359 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.020s	sys 0.025s
I20260709 12:18:49.027024 12610 leader_election.cc:400] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:48.995560 28114 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:48.994437 28183 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:48.975651 27340 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.033s
I20260709 12:18:49.005522 28273 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:49.029726 28248 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.010437 28279 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.087745 28258 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.089562 28293 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:49.090111 28293 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.090610 28293 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:49.091668 27779 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.093025 28016 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:49.094551 28321 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
W20260709 12:18:49.099633 28270 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.007s	user 0.010s	sys 0.002s
I20260709 12:18:49.003845 28125 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.102495 28125 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.028587 28205 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:49.103580 28205 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:49.054399 27660 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.108186 27660 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.055081 27967 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:49.055925 28252 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:49.049548 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.115075 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.116008 12564 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:49.050447 12565 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:48.993571 28304 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:18:48.970052 28318 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.064s
I20260709 12:18:49.054101 12610 leader_election.cc:403] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:18:49.120074 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93227.3us
I20260709 12:18:49.120901 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:49.121870 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.122509 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:49.123404 28171 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:49.124117 28171 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:49.125897 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:49.121978 28318 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 44. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:49.127007 28286 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.055748 27951 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.129863 28315 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:49.131250 28315 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:49.131855 28315 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:49.130059 28278 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.051637 28287 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:49.134402 28287 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:49.134668 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:49.135099 28287 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:49.140826 12490 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:49.141916 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:49.206460 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64684.9us
W20260709 12:18:49.053866 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 574700us
I20260709 12:18:49.218065 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.219285 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:49.219797 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:18:49.205247 28311 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.444s	user 0.000s	sys 0.001s
W20260709 12:18:49.205340 28313 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.084s	user 0.005s	sys 0.004s
I20260709 12:18:49.241139 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4577) took 1149 ms. Trace:
I20260709 12:18:49.241906 12646 rpcz_store.cc:276] 0709 12:18:48.092698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.093538 (+   840us) service_pool.cc:224] Handling call
0709 12:18:48.147149 (+ 53611us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:48.155136 (+  7987us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.155175 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:48.162582 (+  7407us) log.cc:844] Serialized 26 byte log entry
0709 12:18:48.449016 (+286434us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:48.449066 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.449136 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.240755 (+791619us) raft_consensus.cc:1685] finished
0709 12:18:49.240791 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.241041 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":83749,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"thread_start_us":253659,"threads_started":1,"wal-append.queue_time_us":451176}
I20260709 12:18:49.221199 28171 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.025s
I20260709 12:18:49.248203 28293 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.251385 27216 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:18:49.076894 28271 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:49.253075 28283 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.204914 28286 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:18:49.220682 28329 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.101s	user 0.005s	sys 0.004s
W20260709 12:18:49.217842 28314 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.335s	user 0.000s	sys 0.007s
I20260709 12:18:49.237355 28028 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:49.277580 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:49.218247 28324 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.170s	user 0.002s	sys 0.005s
I20260709 12:18:49.211069 28296 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:49.218125 28316 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.223s	user 0.000s	sys 0.007s
I20260709 12:18:49.283133 28271 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.283941 12642 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 50 index: 33. Preceding OpId from leader: term: 54 index: 35. (index mismatch)
I20260709 12:18:49.284734 27951 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.294380 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:49.350916 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56657.2us
I20260709 12:18:49.351994 12462 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:49.229478 28322 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.048s
I20260709 12:18:49.303156 12641 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:49.219720 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:49.356431 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 136787us
I20260709 12:18:49.221127 28318 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.083s
I20260709 12:18:49.257330 27967 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.277225 28286 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.324611 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.362161 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:49.324813 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.363492 13260 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 52: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:49.326236 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.365814 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:49.260056 28307 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.508s	user 0.000s	sys 0.007s
I20260709 12:18:49.347110 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:49.347815 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 462529us
W20260709 12:18:49.353922 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.019s	sys 0.025s
I20260709 12:18:49.357497 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:49.378393 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:49.309549 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:49.377046 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.019s	sys 0.025s
W20260709 12:18:49.380291 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 656665us
I20260709 12:18:49.380973 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:49.358153 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.384390 12536 leader_election.cc:400] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:49.385460 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:49.387209 12642 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 44 index: 29. (index mismatch)
I20260709 12:18:49.351192 28183 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:49.356169 28134 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:49.363771 28028 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:49.359782 27825 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:49.392870 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.365182 28286 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:49.407066 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:49.400966 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:49.409085 12490 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:18:49.415305 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.416972 13260 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:18:49.418223 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:49.419427 13253 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:18:49.398420 27765 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.973s	user 0.012s	sys 0.003s
I20260709 12:18:49.388664 27963 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:49.386562 28320 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.387004 28296 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:49.388064 28122 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.395674 28270 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:18:49.445542 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.424738 28323 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:49.419050 28274 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:49.413228 27660 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 51.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.061s
I20260709 12:18:49.419220 28279 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 33, Time since last communication: 0.055s
I20260709 12:18:49.425675 28028 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:49.429684 27216 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.048s
I20260709 12:18:49.428380 27963 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:49.440241 12637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:18:49.431174 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.431345 12567 raft_consensus.cc:2410] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 29.
I20260709 12:18:49.397557 28134 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:49.426461 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:49.434805 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.515947 12571 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:49.534319 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.516151 13253 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:49.427670 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:49.435546 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.436343 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.400530 28293 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.425807 27825 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:49.444681 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5744) took 1103 ms. Trace:
I20260709 12:18:49.556011 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:49.549887 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:49.557945 12493 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:49.558547 12644 rpcz_store.cc:276] 0709 12:18:48.340813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.434532 (+ 93719us) service_pool.cc:224] Handling call
0709 12:18:48.457432 (+ 22900us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:48.457877 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.457912 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:48.458570 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:18:48.563765 (+105195us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:48.594436 (+ 30671us) log.cc:844] Serialized 14 byte log entry
0709 12:18:48.594694 (+   258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.594780 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.444286 (+849506us) raft_consensus.cc:1685] finished
0709 12:18:49.444324 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.444588 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17234,"prepare.queue_time_us":19632,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":5324,"raft.queue_time_us":125833,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":21663,"spinlock_wait_cycles":38893952,"thread_start_us":103749,"threads_started":1,"wal-append.queue_time_us":289216}
I20260709 12:18:49.479485 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.546193 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:49.562901 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:49.547358 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:49.551542 12646 raft_consensus.cc:2410] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 50 because replica has last-logged OpId of term: 48 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 29.
I20260709 12:18:49.552006 12494 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.25-45.25]
W20260709 12:18:49.571069 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 143460us
I20260709 12:18:49.411690 28125 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 43.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.080s
I20260709 12:18:49.411962 27779 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.103s
I20260709 12:18:49.399849 28183 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:49.582229 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:49.461557 28295 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.952s	user 0.003s	sys 0.000s
W20260709 12:18:49.535269 27833 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.723s	user 0.005s	sys 0.009s
I20260709 12:18:49.476982 28272 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.477824 27963 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:49.431742 28055 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:49.582304 12495 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.25-43.25]
I20260709 12:18:49.540067 28113 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.439452 27765 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.471843 28117 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.442138 28320 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.597134 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:49.539188 28244 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:49.437909 28296 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:49.463824 28323 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:49.469537 28273 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.607323 27825 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 43.30, Last appended by leader: 30, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.608517 28315 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.610878 28183 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:49.515228 28122 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.707551 12497 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:49.708869 28122 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.601598 27833 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.612389 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 39 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:49.636353 28113 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.710166 12644 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:49.712203 28113 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.713912 12564 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:49.714984 28273 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.693109 28272 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.023s
W20260709 12:18:49.616241 28281 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.000s
I20260709 12:18:49.720835 12569 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
W20260709 12:18:49.619642 28153 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.628s	user 0.013s	sys 0.001s
I20260709 12:18:49.477109 28028 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 47.32, Last appended by leader: 32, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.574719 28124 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:49.557740 28134 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:49.557458 28270 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:49.472406 28274 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:49.632774 12495 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:18:49.489759 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:49.581501 28279 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:18:49.635313 12639 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:49.736294 13255 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:49.582576 28308 tablet_copy_client.cc:323] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.129:38859
I20260709 12:18:49.429414 27340 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:18:49.626495 28158 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.030s	sys 0.037s
W20260709 12:18:49.743130 28158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.030s	sys 0.037s
W20260709 12:18:49.744511 28289 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.329s	user 0.003s	sys 0.000s
I20260709 12:18:49.719708 28281 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.737908 28279 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:49.747684 28315 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:49.747970 28281 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.729077 28153 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.750705 28285 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:49.752328 28093 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.810s	user 0.014s	sys 0.000s
I20260709 12:18:49.752871 28093 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:18:49.650652 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.033s	sys 0.042s
I20260709 12:18:49.473565 28205 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.606431 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:49.627163 28125 consensus_queue.cc:1243] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 44.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 27
W20260709 12:18:49.762552 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 361632us
I20260709 12:18:49.669682 12571 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
W20260709 12:18:49.672014 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.019s	sys 0.066s
W20260709 12:18:49.645589 28252 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.015s	sys 0.000s
I20260709 12:18:49.666472 28323 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:49.738366 28273 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:49.699744 28286 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.702597 28320 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.651535 28244 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:49.726277 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:49.735519 12645 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:49.591115 27967 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.635778 27765 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.740407 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5902) took 2412 ms. Trace:
I20260709 12:18:49.861580 12570 rpcz_store.cc:276] 0709 12:18:47.328317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:47.328466 (+   149us) service_pool.cc:224] Handling call
0709 12:18:47.328854 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:47.724355 (+395501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:47.724409 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:47.753234 (+ 28825us) log.cc:844] Serialized 26 byte log entry
0709 12:18:48.442436 (+689202us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:48.442496 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:48.442600 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:49.460209 (+1017609us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:49.735344 (+275135us) raft_consensus.cc:1685] finished
0709 12:18:49.735368 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:49.740328 (+  4960us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":207759,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":5053,"spinlock_wait_cycles":14995712,"thread_start_us":1050176,"threads_started":2,"wal-append.queue_time_us":670822}
W20260709 12:18:49.627195 28111 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.039s	sys 0.027s
I20260709 12:18:49.754735 12637 raft_consensus.cc:606] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:49.757974 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.033s	sys 0.042s
I20260709 12:18:49.761785 12640 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 40: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:49.792346 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.019s	sys 0.067s
W20260709 12:18:49.871490 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 887622us
I20260709 12:18:49.778720 12496 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 44 index: 23. (index mismatch)
I20260709 12:18:49.794796 12565 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:49.738381 28274 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:49.700070 28092 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.750s	user 0.015s	sys 0.000s
I20260709 12:18:49.738011 28270 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:49.667210 28296 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:49.805787 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:49.750705 28153 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:49.816157 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 51349us
W20260709 12:18:49.766803 28287 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.007s
I20260709 12:18:49.719877 27833 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.801752 28024 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.886330 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.857864 28244 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:18:49.899521 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93946.1us
I20260709 12:18:49.912259 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:49.900552 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:49.895663 28111 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.040s	sys 0.028s
I20260709 12:18:49.904825 13253 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:49.897894 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.907889 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.914748 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:49.905778 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:49.925257 13261 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:49.926062 12640 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:18:49.809643 28252 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.934610 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.930136 12642 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:49.922576 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:49.931121 13257 raft_consensus.cc:2368] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:49.908437 28016 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:49.825743 28273 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.937086 13254 raft_consensus.cc:2376] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49. Re-sending same reply.
I20260709 12:18:49.944000 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:49.938454 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.941401 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:49.912962 27216 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:49.948562 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:49.949994 13262 raft_consensus.cc:2368] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:49.949935 12571 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:49.956008 12570 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:49.957474 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.958343 12571 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:49.956700 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:49.962496 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:49.972877 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:49.973369 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72935.3us
I20260709 12:18:49.960338 28016 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:18:49.978163 28354 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:18:49.989316 28286 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:49.958865 28333 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:49.995952 28252 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:49.909082 28287 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:49.923529 28111 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:49.960052 28315 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 43.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.014s
I20260709 12:18:49.933880 28171 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.000589 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:50.013154 13255 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:50.013767 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:50.056890 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56392.2us
I20260709 12:18:50.057181 12646 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 47 because replica has last-logged OpId of term: 45 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 29.
I20260709 12:18:50.015461 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:50.057823 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:50.058229 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94156us
W20260709 12:18:50.015781 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59158.9us
I20260709 12:18:50.059301 12570 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:18:50.060895 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:50.021221 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109063us
I20260709 12:18:49.944665 28092 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:50.023427 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.913690 28024 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:49.963313 28342 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:50.064057 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:50.022636 28341 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
W20260709 12:18:50.022384 28348 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
W20260709 12:18:50.022521 28347 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.116s	user 0.001s	sys 0.001s
I20260709 12:18:49.964639 27216 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.050112 28273 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.029107 28333 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:50.024451 28322 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:50.061425 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:49.956964 28113 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.075s
I20260709 12:18:50.068567 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:50.071843 27660 raft_consensus.cc:2764] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election decision vote started in defunct term 52: won
I20260709 12:18:50.067612 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:50.038144 28328 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:50.038499 28158 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:50.106660 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:50.072419 28252 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:50.142138 27765 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.089639 27967 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:50.143700 27967 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:50.148394 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6238) took 1193 ms. Trace:
I20260709 12:18:50.149037 12568 rpcz_store.cc:276] 0709 12:18:48.954499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.957040 (+  2541us) service_pool.cc:224] Handling call
0709 12:18:48.987068 (+ 30028us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:49.388710 (+401642us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.388764 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:49.394695 (+  5931us) log.cc:844] Serialized 52 byte log entry
0709 12:18:49.720655 (+325960us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:50.104091 (+383436us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.104745 (+   654us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.104958 (+   213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.105063 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:50.148074 (+ 43011us) raft_consensus.cc:1685] finished
0709 12:18:50.148109 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.148318 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7406,"raft.queue_time_us":144911,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":8478,"spinlock_wait_cycles":5258496,"thread_start_us":1057383,"threads_started":3,"wal-append.queue_time_us":390732}
I20260709 12:18:50.070354 28342 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.092927 28286 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:50.072202 27825 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:50.122704 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:50.124866 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56430.7us
I20260709 12:18:50.089944 28338 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.093664 28315 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:50.100152 27340 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:50.129338 28111 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 48.29, Last appended by leader: 29, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.130851 27216 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.131151 28024 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.131805 28124 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:50.132692 28158 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.070132 28016 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.29, Last appended by leader: 29, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.133924 28328 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.135919 28183 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:50.136655 28092 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:50.113049 28205 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.115475 28322 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.071242 28171 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.110425 28333 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:50.132001 28349 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:50.144701 27967 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.150701 28293 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:18:50.174706 28358 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 12:18:50.164083 28342 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:50.161986 28286 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:50.171520 28242 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:50.177263 28244 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.267690 28092 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.269905 28322 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:50.170607 28270 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.178951 27340 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.295241 27340 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:50.297003 28134 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.000s
I20260709 12:18:50.176738 28338 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.299321 28338 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.292538 28287 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.293190 28328 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.297494 28134 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.302758 28134 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.304159 28270 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:50.306238 28361 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:18:50.307487 28322 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.312664 28322 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.315296 28158 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.271651 28183 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.288691 28242 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:50.287350 28092 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:50.328431 28137 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:18:50.337055 28171 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:50.341059 28323 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.516s	user 0.013s	sys 0.000s
I20260709 12:18:50.332660 28286 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 43.30, Last appended by leader: 30, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.337949 28244 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.343879 28296 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.347594 28286 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:50.349246 28296 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.349920 28323 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.351271 28242 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.337679 28183 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.353590 28242 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.349854 28315 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:50.353406 28323 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.060593 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:50.410673 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 349929us
I20260709 12:18:50.427170 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:50.429001 12488 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:50.443483 12492 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 22. Preceding OpId from leader: term: 52 index: 23. (index mismatch)
I20260709 12:18:50.443550 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:50.446105 12507 tablet_copy_service.cc:140] P 6d4ece942fd5414998215d26edf91001: Received BeginTabletCopySession request for tablet 3eadf951b63a43fd993cea2c96d5214f from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:54234)
I20260709 12:18:50.447183 12507 tablet_copy_service.cc:161] P 6d4ece942fd5414998215d26edf91001: Beginning new tablet copy session on tablet 3eadf951b63a43fd993cea2c96d5214f from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:54234: session id = 9485b0afaf144396adf854fff857414e-3eadf951b63a43fd993cea2c96d5214f
I20260709 12:18:50.446116 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:50.134955 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67446.2us
I20260709 12:18:50.475063 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:50.479482 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:50.482435 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:50.486338 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6241) took 1521 ms. Trace:
I20260709 12:18:50.487193 12563 rpcz_store.cc:276] 0709 12:18:48.965231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.965440 (+   209us) service_pool.cc:224] Handling call
0709 12:18:48.986503 (+ 21063us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:48.986909 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.986937 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:48.987553 (+   616us) log.cc:844] Serialized 52 byte log entry
0709 12:18:49.871400 (+883847us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:50.037588 (+166188us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.037904 (+   316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.038009 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:50.485321 (+447312us) raft_consensus.cc:1685] finished
0709 12:18:50.485382 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.485965 (+   583us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":310241,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":139504,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":29575,"thread_start_us":968640,"threads_started":2,"wal-append.queue_time_us":790657}
I20260709 12:18:50.490947 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.491839 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.491925 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.492882 12568 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:18:50.493067 12563 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:50.493319 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.494779 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.495760 12568 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:50.115913 12570 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:50.498582 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:50.499240 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 51 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.500708 12570 raft_consensus.cc:2410] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 54 because replica has last-logged OpId of term: 53 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 25.
I20260709 12:18:50.505019 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:50.510352 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:50.510824 12644 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 27. Preceding OpId from leader: term: 40 index: 28. (index mismatch)
I20260709 12:18:50.518970 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2229) took 1147 ms. Trace:
I20260709 12:18:50.519100 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:50.519672 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.519719 12641 rpcz_store.cc:276] 0709 12:18:49.361947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.365027 (+  3080us) service_pool.cc:224] Handling call
0709 12:18:50.266924 (+901897us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:50.509516 (+242592us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1049556224}
I20260709 12:18:50.508735 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.524327 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.540230 12570 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:50.541464 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.542972 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.534679 27340 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:18:50.543876 27340 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:50.545426 27340 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 51.27, Last appended by leader: 27, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.535565 27779 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.546608 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:50.547572 27779 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.535604 27660 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.550374 27660 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.537258 27825 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:50.554490 27825 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.559033 27779 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.559701 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:50.553488 27660 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.497241 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.559963 27779 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:50.318575 27963 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.684s	user 0.011s	sys 0.004s
I20260709 12:18:50.537843 27951 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:50.557402 27825 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.646843 12641 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:50.662559 12563 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:18:50.660539 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:50.671243 12488 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:50.674726 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.686158 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:50.339303 28137 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:50.690692 12568 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:50.684295 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:50.696082 28369 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:50.697896 28349 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:50.698892 28092 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.700213 27963 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.699299 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:50.696841 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 108501us
I20260709 12:18:50.711018 28171 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:50.711895 28171 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.711581 28328 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:50.712463 28171 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:50.714123 28242 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:50.714694 28113 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:50.715324 28113 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.715869 28113 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:50.720679 28315 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:18:50.723021 28323 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:50.722580 28322 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:18:50.724756 28287 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:50.725672 28287 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:50.726116 28287 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:50.727667 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.730381 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:50.731990 28369 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:50.717988 28137 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:50.732743 28369 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:50.734025 27216 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:50.732967 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:50.739197 28326 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.739743 28322 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:50.740475 28322 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:50.353888 28274 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.702761 28092 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.740422 28326 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.735345 27216 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.763371 28281 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:18:50.747709 28322 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.764832 28281 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:50.766397 28319 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.735800 28369 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.766669 28281 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.767197 28319 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.767997 28369 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:50.768321 28274 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.769905 12571 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:50.773434 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2962) took 1478 ms. Trace:
I20260709 12:18:50.774027 12568 rpcz_store.cc:276] 0709 12:18:49.294701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.743504 (+1448803us) service_pool.cc:224] Handling call
0709 12:18:50.772468 (+ 28964us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.772926 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:50.772958 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:50.773037 (+    79us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:50.773088 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.773165 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.773320 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.775477 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:50.776186 28319 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:50.762269 28242 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:50.779414 27963 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:50.780582 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.000s	sys 0.020s
W20260709 12:18:50.780948 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.001s	sys 0.021s
I20260709 12:18:50.780381 28242 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.782589 28252 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:50.783073 28252 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:50.784523 28252 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.27, Last appended by leader: 27, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.722266 28270 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.035s
I20260709 12:18:50.790478 28270 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:18:50.791091 28270 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:50.792301 28270 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.722380 27951 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.777405 12568 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 35. Current term is 36. Ops: []
I20260709 12:18:50.801748 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5913) took 1384 ms. Trace:
I20260709 12:18:50.757295 28349 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.803359 28349 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.806994 28349 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.802239 12568 rpcz_store.cc:276] 0709 12:18:49.415732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.776798 (+1361066us) service_pool.cc:224] Handling call
0709 12:18:50.777180 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.799825 (+ 22645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.800002 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.810030 27951 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:50.814234 28326 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.814548 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6059) took 1269 ms. Trace:
I20260709 12:18:50.816776 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6242) took 1850 ms. Trace:
I20260709 12:18:50.820240 12566 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.28, Last appended by leader: 28, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.822407 27216 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.824268 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6240) took 1868 ms. Trace:
I20260709 12:18:50.825947 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6237) took 1871 ms. Trace:
W20260709 12:18:50.833284 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.832793 12569 rpcz_store.cc:276] 0709 12:18:48.954122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.956474 (+  2352us) service_pool.cc:224] Handling call
0709 12:18:48.956755 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.721500 (+764745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.825360 (+1103860us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54001d7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:50.825755 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1756593024}
I20260709 12:18:50.831842 12567 rpcz_store.cc:276] 0709 12:18:48.955279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.961066 (+  5787us) service_pool.cc:224] Handling call
0709 12:18:50.823257 (+1862191us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:50.823811 (+   554us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":996084096}
I20260709 12:18:50.830917 12564 rpcz_store.cc:276] 0709 12:18:48.966139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.116668 (+150529us) service_pool.cc:224] Handling call
0709 12:18:49.117271 (+   603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.714478 (+597207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.816033 (+1101555us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:50.816519 (+   486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1371477248}
I20260709 12:18:50.845250 12569 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: [47.26-47.26]
I20260709 12:18:50.846426 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5919) took 1423 ms. Trace:
I20260709 12:18:50.846746 12569 rpcz_store.cc:276] 0709 12:18:49.423444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.839494 (+1416050us) service_pool.cc:224] Handling call
0709 12:18:50.844850 (+  5356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:50.846117 (+  1267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.846334 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.848518 28252 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:50.856204 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:42064 (request call id 0) took 1432 ms. Trace:
I20260709 12:18:50.857790 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2235) took 1424 ms. Trace:
I20260709 12:18:50.867893 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:50.830385 12496 rpcz_store.cc:276] 0709 12:18:49.544729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.547858 (+  3129us) service_pool.cc:224] Handling call
0709 12:18:49.548201 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.872985 (+324784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.813847 (+940862us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:50.814273 (+   426us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":528810368}
I20260709 12:18:50.859998 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2237) took 1381 ms. Trace:
W20260709 12:18:50.863099 28286 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.005s
I20260709 12:18:50.869189 12645 rpcz_store.cc:276] 0709 12:18:49.478618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.479565 (+   947us) service_pool.cc:224] Handling call
0709 12:18:50.859582 (+1380017us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54001f6dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:50.859764 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447351552}
I20260709 12:18:50.868261 12639 rpcz_store.cc:276] 0709 12:18:49.433686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.433829 (+   143us) service_pool.cc:224] Handling call
0709 12:18:50.857313 (+1423484us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002e5254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:50.857565 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405843200}
I20260709 12:18:50.873770 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:50.867750 12637 rpcz_store.cc:276] 0709 12:18:49.423856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.424119 (+   263us) service_pool.cc:224] Handling call
0709 12:18:50.855671 (+1431552us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57ed853 00007ffbfa7bb60a 00007ffbf3d417d4 00007ffbf3d41762 00007ffbf3d41691 00007ffbf3d4160c 00007ffbf3d408c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:50.855889 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547654016}
W20260709 12:18:50.882552 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91194.7us
W20260709 12:18:50.898044 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90134.8us
I20260709 12:18:50.874508 28287 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.908118 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5914) took 1491 ms. Trace:
W20260709 12:18:50.913882 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.917778 28111 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:18:50.920751 28111 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:50.927753 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 1968) took 1332 ms. Trace:
I20260709 12:18:50.929901 27340 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:50.930416 27340 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:50.930809 27340 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:50.914489 12568 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
I20260709 12:18:50.933816 28137 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:18:50.934432 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5918) took 1516 ms. Trace:
I20260709 12:18:50.934860 12568 rpcz_store.cc:276] 0709 12:18:49.417880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.842097 (+1424217us) service_pool.cc:224] Handling call
0709 12:18:50.899981 (+ 57884us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.934127 (+ 34146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.934364 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:50.936353 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:50.942858 12637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:18:50.943866 12637 raft_consensus.cc:606] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:18:50.944341 12637 raft_consensus.cc:614] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:50.944907 28124 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:50.946262 28281 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 50.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:18:50.947875 27951 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:50.948352 27951 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:50.949689 27951 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 50.28, Last appended by leader: 28, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:50.950594 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 461119us
I20260709 12:18:50.921376 28111 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:50.928256 13260 rpcz_store.cc:276] 0709 12:18:49.594937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.595089 (+   152us) service_pool.cc:224] Handling call
0709 12:18:49.595495 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.623655 (+ 28160us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:49.623688 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:49.623736 (+    48us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:49.623780 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.927114 (+1303334us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:50.927186 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.927488 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64167168}
I20260709 12:18:50.955294 28382 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.956115 28382 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.958600 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:50.959453 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:50.959493 12564 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:50.960467 27951 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:50.961109 28183 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:50.968159 12637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:18:50.971095 12637 raft_consensus.cc:606] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:50.912051 12570 rpcz_store.cc:276] 0709 12:18:49.416083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.829814 (+1413731us) service_pool.cc:224] Handling call
0709 12:18:50.850462 (+ 20648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.851038 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:50.851837 (+   799us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.907249 (+ 55412us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:50.907349 (+   100us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:50.907433 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:50.907540 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:50.907894 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":446,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":6756,"thread_start_us":49445,"threads_started":1}
I20260709 12:18:50.965293 28113 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.978319 28183 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.990449 28349 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:50.986248 28279 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:50.992363 28349 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:50.993431 28183 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.979300 28125 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:50.983461 12637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:18:50.984863 28061 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.001s
I20260709 12:18:50.985144 28382 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:50.986577 28003 tablet_copy_client.cc:538] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:50.987385 28016 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:50.988669 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:50.998111 28319 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:50.999867 28349 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:50.999967 28171 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.007522 12569 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: [42.22-44.23]
W20260709 12:18:51.010387 28302 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.498s	user 0.004s	sys 0.002s
I20260709 12:18:50.995970 28125 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:51.013502 28287 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:51.013407 28171 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:51.019480 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2965) took 1347 ms. Trace:
I20260709 12:18:51.019029 28125 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.29, Last appended by leader: 29, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.026100 12563 rpcz_store.cc:276] 0709 12:18:49.672520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.018224 (+1345704us) service_pool.cc:224] Handling call
0709 12:18:51.018713 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.019094 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.019129 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:51.019194 (+    65us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:51.019233 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.019288 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.019403 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.021183 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2372) took 2429 ms. Trace:
I20260709 12:18:51.021778 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.024004 13259 raft_consensus.cc:1217] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Deduplicated request from leader. Original: 41.27->[43.28-43.28]   Dedup: 43.28->[]
I20260709 12:18:51.023833 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.024400 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:51.027580 13254 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:51.037065 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4597) took 1133 ms. Trace:
I20260709 12:18:51.027007 12490 rpcz_store.cc:276] 0709 12:18:48.591566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.609917 (+ 18351us) service_pool.cc:224] Handling call
0709 12:18:51.020730 (+2410813us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:51.020922 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1204432640}
I20260709 12:18:51.037721 28171 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:18:51.030251 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6235) took 2076 ms (client timeout 2100 ms). Trace:
I20260709 12:18:51.037935 28184 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.036784 28287 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:18:51.045158 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:51.032951 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.039302 13259 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.28-43.28]
I20260709 12:18:51.032248 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6257) took 1271 ms. Trace:
W20260709 12:18:51.038729 12572 rpcz_store.cc:269] 0709 12:18:48.953288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.955526 (+  2238us) service_pool.cc:224] Handling call
0709 12:18:51.029833 (+2074307us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:51.030080 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39634688}
I20260709 12:18:51.042474 13255 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:51.037603 12644 rpcz_store.cc:276] 0709 12:18:49.903271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.903960 (+   689us) service_pool.cc:224] Handling call
0709 12:18:49.927754 (+ 23794us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.513880 (+586126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.036592 (+522712us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:51.036922 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1338349440}
I20260709 12:18:51.041942 28315 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:51.047281 12571 rpcz_store.cc:276] 0709 12:18:49.760478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.492826 (+732348us) service_pool.cc:224] Handling call
0709 12:18:51.031951 (+539125us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54002463d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:51.032102 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":632448256}
I20260709 12:18:51.030969 28319 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.031711 13256 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:51.036294 28125 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:51.047606 12637 raft_consensus.cc:606] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:51.046509 28296 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:18:51.057039 28296 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:51.057245 28016 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:51.058254 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5928) took 1415 ms. Trace:
I20260709 12:18:51.058727 12568 rpcz_store.cc:276] 0709 12:18:49.642787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.006484 (+1363697us) service_pool.cc:224] Handling call
0709 12:18:51.057247 (+ 50763us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.057798 (+   551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.057826 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:51.057893 (+    67us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:51.057947 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.058014 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.058150 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.058562 28296 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.059413 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.060560 12563 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:51.061692 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.062228 28113 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.061883 28184 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:51.063087 28344 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.345s	user 0.004s	sys 0.000s
I20260709 12:18:51.064858 28184 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.060372 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.062762 12563 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:51.067333 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5920) took 1637 ms. Trace:
I20260709 12:18:51.067477 28153 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.068222 28153 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.068914 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4542) took 2113 ms. Trace:
I20260709 12:18:51.069442 28319 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:51.067812 12569 rpcz_store.cc:276] 0709 12:18:49.429062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.847549 (+1418487us) service_pool.cc:224] Handling call
0709 12:18:50.848066 (+   517us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:51.008150 (+160084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.066120 (+ 57970us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54003abe54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:51.066370 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364677376}
W20260709 12:18:51.070719 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82267.7us
I20260709 12:18:51.071467 12569 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:51.071849 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:51.069437 12491 rpcz_store.cc:276] 0709 12:18:48.955103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.955252 (+   149us) service_pool.cc:224] Handling call
0709 12:18:48.955680 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:48.956114 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:48.956146 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:48.956709 (+   563us) log.cc:844] Serialized 26 byte log entry
0709 12:18:49.423318 (+466609us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:49.765246 (+341928us) log.cc:844] Serialized 14 byte log entry
0709 12:18:49.765559 (+   313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:49.765678 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:51.062073 (+1296395us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:51.066582 (+  4509us) raft_consensus.cc:1685] finished
0709 12:18:51.068374 (+  1792us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.068615 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":273003,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":1138,"raft.run_cpu_time_us":536,"raft.run_wall_time_us":42091,"spinlock_wait_cycles":8641280,"thread_start_us":731322,"threads_started":2,"wal-append.queue_time_us":715856}
W20260709 12:18:51.072945 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6055) took 1529 ms (client timeout 1958 ms). Trace:
I20260709 12:18:51.073583 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:51.073527 12497 rpcz_store.cc:269] 0709 12:18:49.543474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.550771 (+  7297us) service_pool.cc:224] Handling call
0709 12:18:51.072483 (+1521712us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400367c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:51.072700 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346662272}
I20260709 12:18:51.074242 28061 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:51.076119 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:51.077586 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.073061 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6245) took 2105 ms. Trace:
I20260709 12:18:51.078866 28287 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:51.079466 13256 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 30. Preceding OpId from leader: term: 44 index: 31. (index mismatch)
I20260709 12:18:51.080052 12507 tablet_copy_source_session.cc:215] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:18:51.081055 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.082579 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.060729 28296 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:18:51.083570 28281 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:51.084206 28281 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:51.084856 13257 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 39 index: 27. Preceding OpId from leader: term: 40 index: 27. (term mismatch)
I20260709 12:18:51.085278 13257 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 26
I20260709 12:18:51.085976 13257 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 39.27
I20260709 12:18:51.086328 13257 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:51.083858 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5929) took 1344 ms. Trace:
I20260709 12:18:51.090304 12569 rpcz_store.cc:276] 0709 12:18:49.740771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.070702 (+1329931us) service_pool.cc:224] Handling call
0709 12:18:51.071189 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.071913 (+   724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.083778 (+ 11865us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.092353 12569 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: [47.26-47.26]
I20260709 12:18:51.092943 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6249) took 1344 ms. Trace:
I20260709 12:18:51.089427 28124 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:51.094168 28124 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:51.094707 28124 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:51.095584 27963 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:51.083563 13253 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:18:51.097090 28349 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.096302 27963 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.097476 27825 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:51.098248 27825 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.099121 28349 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.100550 27809 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.100912 28134 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:51.101266 27809 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.077304 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:51.093271 12569 rpcz_store.cc:276] 0709 12:18:49.748629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.091643 (+1343014us) service_pool.cc:224] Handling call
0709 12:18:51.092062 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:51.092708 (+   646us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.092886 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.089583 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:51.025918 28270 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.064s
I20260709 12:18:51.078627 12565 rpcz_store.cc:276] 0709 12:18:48.967286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.118349 (+151063us) service_pool.cc:224] Handling call
0709 12:18:49.118654 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.873114 (+754460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.072409 (+1199295us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:51.072829 (+   420us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1554418048}
I20260709 12:18:51.066143 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:51.099723 27963 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:51.147531 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97725.7us
I20260709 12:18:51.162485 13254 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:51.101512 28349 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:51.100590 27825 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:51.079538 28287 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:51.124142 28315 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:51.125355 28137 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:51.187633 28342 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.861s	user 0.014s	sys 0.007s
I20260709 12:18:51.188360 28342 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.191313 28342 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.192994 28339 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:51.194413 28319 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.195266 28319 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.198459 28369 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:18:51.198994 28369 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:51.199290 12569 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 46 index: 24. (term mismatch)
I20260709 12:18:51.199738 12569 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 23
I20260709 12:18:51.200083 12569 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 45.24
I20260709 12:18:51.200338 12569 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:51.200165 28369 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.201336 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6250) took 1452 ms. Trace:
I20260709 12:18:51.203334 12571 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:51.208806 12569 rpcz_store.cc:276] 0709 12:18:49.748883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.112119 (+1363236us) service_pool.cc:224] Handling call
0709 12:18:51.198711 (+ 86592us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.200944 (+  2233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.201265 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.209204 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5932) took 1138 ms. Trace:
I20260709 12:18:51.203612 28153 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:51.204777 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.206244 13254 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 38. Current term is 39. Ops: []
W20260709 12:18:51.211158 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.085259 28281 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.209611 12571 rpcz_store.cc:276] 0709 12:18:50.071172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.202366 (+1131194us) service_pool.cc:224] Handling call
0709 12:18:51.202922 (+   556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.208911 (+  5989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.209148 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.218691 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.182116 27963 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.221299 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:51.218875 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145383us
I20260709 12:18:51.212991 28369 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:18:51.221524 12637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:18:51.222508 12568 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 41. Current term is 42. Ops: []
W20260709 12:18:51.223192 28333 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.945s	user 0.009s	sys 0.013s
I20260709 12:18:51.223659 13257 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 32. Preceding OpId from leader: term: 49 index: 33. (index mismatch)
I20260709 12:18:51.224321 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6255) took 1464 ms. Trace:
I20260709 12:18:51.234961 12565 rpcz_store.cc:276] 0709 12:18:49.759770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.143904 (+1384134us) service_pool.cc:224] Handling call
0709 12:18:51.189594 (+ 45690us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.223865 (+ 34271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.223900 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:51.223950 (+    50us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:51.223985 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.224072 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.224234 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.226367 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 38 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.228499 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5922) took 1787 ms. Trace:
I20260709 12:18:51.230338 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5926) took 1646 ms. Trace:
I20260709 12:18:51.101567 28134 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.237160 12570 rpcz_store.cc:276] 0709 12:18:49.583570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.994165 (+1410595us) service_pool.cc:224] Handling call
0709 12:18:51.102442 (+108277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.229806 (+127364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.229838 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:51.229901 (+    63us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:51.229978 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.230144 (+   166us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.230266 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.238682 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5931) took 1176 ms. Trace:
I20260709 12:18:51.239300 12568 rpcz_store.cc:276] 0709 12:18:50.062615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.185009 (+1122394us) service_pool.cc:224] Handling call
0709 12:18:51.222141 (+ 37132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.238284 (+ 16143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.238597 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.245971 12490 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:51.249081 28315 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:51.215090 27825 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:18:51.249199 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 151522us
W20260709 12:18:51.246802 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177646us
I20260709 12:18:51.257450 13253 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 41 because replica has last-logged OpId of term: 40 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 27.
I20260709 12:18:51.241992 28153 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:51.257086 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.265792 12569 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:18:51.265043 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:51.268256 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5934) took 1104 ms. Trace:
I20260709 12:18:51.272078 12497 raft_consensus.cc:2368] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:51.272219 12569 rpcz_store.cc:276] 0709 12:18:50.163880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.218261 (+1054381us) service_pool.cc:224] Handling call
0709 12:18:51.261121 (+ 42860us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:51.267968 (+  6847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.268199 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:51.242776 28342 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:51.236686 12567 rpcz_store.cc:276] 0709 12:18:49.441812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.889609 (+1447797us) service_pool.cc:224] Handling call
0709 12:18:50.910528 (+ 20919us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:18:50.955211 (+ 44683us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:50.955248 (+    37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:18:50.956156 (+   908us) log.cc:844] Serialized 78 byte log entry
0709 12:18:50.996212 (+ 40056us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:51.052271 (+ 56059us) log.cc:844] Serialized 14 byte log entry
0709 12:18:51.056500 (+  4229us) log.cc:844] Serialized 14 byte log entry
0709 12:18:51.067092 (+ 10592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.107696 (+ 40604us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:51.228217 (+120521us) raft_consensus.cc:1685] finished
0709 12:18:51.228241 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.228415 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39484,"prepare.queue_time_us":54846,"prepare.run_wall_time_us":78,"raft.queue_time_us":107578,"spinlock_wait_cycles":9148544,"thread_start_us":39574,"threads_started":1,"wal-append.queue_time_us":168353}
I20260709 12:18:51.273746 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:51.276718 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 187265us
I20260709 12:18:51.276996 12490 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:51.249881 28137 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:51.281927 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64761.6us
I20260709 12:18:51.282945 12565 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 27. Preceding OpId from leader: term: 53 index: 28. (index mismatch)
I20260709 12:18:51.286953 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:51.250943 28333 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.295689 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.296499 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.317454 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:51.318141 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73096.9us
I20260709 12:18:51.297570 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.298133 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.319622 12646 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 23. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
I20260709 12:18:51.320560 12488 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:51.321020 28134 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.296885 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.323109 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.324222 12495 raft_consensus.cc:2393] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:51.299080 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.304347 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6251) took 1555 ms. Trace:
I20260709 12:18:51.306100 28398 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.319689 12642 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 43 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 24.
I20260709 12:18:51.326534 28398 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.326152 12563 rpcz_store.cc:276] 0709 12:18:49.749152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.129526 (+1380374us) service_pool.cc:224] Handling call
0709 12:18:51.181807 (+ 52281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:51.187446 (+  5639us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:51.187498 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:51.188352 (+   854us) log.cc:844] Serialized 26 byte log entry
0709 12:18:51.233593 (+ 45241us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:51.233680 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.233785 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:51.304076 (+ 70291us) raft_consensus.cc:1685] finished
0709 12:18:51.304100 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.304274 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8363,"thread_start_us":44440,"threads_started":1,"wal-append.queue_time_us":91337}
W20260709 12:18:51.329237 28386 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.369s	user 0.004s	sys 0.001s
I20260709 12:18:51.285162 28184 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:51.339501 28184 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.340101 28184 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:51.309592 12645 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:18:51.311019 12490 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 44 because replica has last-logged OpId of term: 43 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 23.
I20260709 12:18:51.242906 28339 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.317067 12386 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 33 to 37, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 37 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.319963 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:51.306959 28396 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:18:51.331934 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2222) took 1426 ms. Trace:
I20260709 12:18:51.335172 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.318393 12497 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:51.299469 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:51.343295 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:51.288385 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1102322900001127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
I20260709 12:18:51.341140 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.329416 28398 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.348488 27951 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:51.348953 12491 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:18:51.361097 12488 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:51.372367 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.373242 12644 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:18:51.374675 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54827.4us
I20260709 12:18:51.373761 12386 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 40, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 40 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:51.376034 28024 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:18:51.377085 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.271502 28287 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.378342 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5921) took 1940 ms. Trace:
I20260709 12:18:51.376382 12386 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 39 to 42, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 42 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.353570 13258 rpcz_store.cc:276] 0709 12:18:49.905444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.906060 (+   616us) service_pool.cc:224] Handling call
0709 12:18:49.906555 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:50.107122 (+200567us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:18:50.512417 (+405295us) log.cc:844] Serialized 14 byte log entry
0709 12:18:50.688452 (+176035us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:50.778078 (+ 89626us) log.cc:844] Serialized 26 byte log entry
0709 12:18:50.778262 (+   184us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:50.778320 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.331199 (+552879us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:51.331253 (+    54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:51.331319 (+    66us) raft_consensus.cc:1685] finished
0709 12:18:51.331427 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:51.331722 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":206319,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":5952,"raft.run_cpu_time_us":623,"raft.run_wall_time_us":48830,"spinlock_wait_cycles":459728256,"thread_start_us":552773,"threads_started":2,"wal-append.queue_time_us":257585}
I20260709 12:18:51.385433 12639 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:51.282867 27963 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.313833 28342 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.382576 12564 rpcz_store.cc:276] 0709 12:18:49.438171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:50.866544 (+1428373us) service_pool.cc:224] Handling call
0709 12:18:50.867201 (+   657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.968632 (+101431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.378081 (+409449us) spinlock_profiling.cc:243] Waited 91.4 ms on lock 0x7b54003e1054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:51.378272 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":210314624}
I20260709 12:18:51.379089 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 27144fedf0014de09f007ac6cc3f7275 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 19)) from {username='slave'} at 127.0.0.1:47032
I20260709 12:18:51.288874 28153 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.390753 12496 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:51.339919 28111 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.394447 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.399577 12610 leader_election.cc:400] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:51.399994 12610 leader_election.cc:403] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Cancelling election due to peer responding with higher term
I20260709 12:18:51.400086 12386 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 42, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 42 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.401965 28318 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:51.403513 28392 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.234s	user 0.002s	sys 0.000s
I20260709 12:18:51.403311 12386 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 47 to 50. New cstate: current_term: 50 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.406522 28137 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.405769 12386 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 47, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 47 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.407995 12386 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 47 to 49, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 49 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.409945 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.411168 12386 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 43 to 46, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.412142 28287 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:18:51.414889 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 236) took 37481 ms (client timeout 15000 ms). Trace:
I20260709 12:18:51.413370 12386 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 46 to 48. New cstate: current_term: 48 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.416483 28333 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:51.416788 12386 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 44 to 46, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 46 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:18:51.421119 28153 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:51.428889 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46597.7us
I20260709 12:18:51.421711 28339 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.422678 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.401420 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:51.430657 12496 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:51.388394 12495 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:51.421211 12386 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 45 to 46, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.378983 28365 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:51.423623 27963 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.434432 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.418802 28342 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:51.434795 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:51.440375 12639 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 22. Preceding OpId from leader: term: 45 index: 23. (index mismatch)
W20260709 12:18:51.440984 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 110342us
W20260709 12:18:51.446035 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 182865us
W20260709 12:18:51.458804 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 69967.3us
W20260709 12:18:51.461417 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 174597us
I20260709 12:18:51.465242 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:51.465230 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:51.466722 12637 raft_consensus.cc:606] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:51.467247 12637 raft_consensus.cc:614] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:18:51.476578 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:51.475634 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97978.9us
I20260709 12:18:51.481900 12644 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:51.484220 12386 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 41 to 43. New cstate: current_term: 43 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:18:51.485675 28395 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
W20260709 12:18:51.486596 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85217.2us
I20260709 12:18:51.480798 28124 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.492879 28124 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:51.415463 12395 rpcz_store.cc:269] 0709 12:18:13.932721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:17.084585 (+3151864us) service_pool.cc:224] Handling call
0709 12:18:51.414488 (+34329903us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:38.241736 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:38.250088 (+  8352us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:38.250169 (+    81us) write_op.cc:435] Acquired schema lock
0709 12:18:38.250200 (+    31us) tablet.cc:662] Decoding operations
0709 12:18:38.252978 (+  2778us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:38.253149 (+   171us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:38.253197 (+    48us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:18:38.959001 (+705804us) tablet.cc:701] Row locks acquired
0709 12:18:38.959058 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:18:38.959290 (+   232us) write_op.cc:270] Start()
0709 12:18:38.967139 (+  7849us) write_op.cc:276] Timestamp: P: 1783599518959199 usec, L: 0
0709 12:18:38.988496 (+ 21357us) op_driver.cc:348] REPLICATION: starting
0709 12:18:39.090140 (+101644us) log.cc:844] Serialized 9815 byte log entry
0709 12:18:39.823863 (+733723us) op_driver.cc:464] REPLICATION: finished
0709 12:18:39.826934 (+  3071us) write_op.cc:301] APPLY: starting
0709 12:18:39.827100 (+   166us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:40.185993 (+358893us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:40.186217 (+   224us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:41.176009 (+989792us) spinlock_profiling.cc:243] Waited 9.56 ms on lock 0x7b18003d43e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf55320ec 00007ffbf557ec0a 00007ffbf557e4f7 00007ffbf557e490 00007ffbf5535f48 00007ffbf924a1bb 00007ffbf92915d4 00007ffbf913d8da 00007ffbf91403cb
0709 12:18:41.476436 (+300427us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:41.476968 (+   532us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0709 12:18:41.518834 (+ 41866us) write_op.cc:312] APPLY: finished
0709 12:18:41.777536 (+258702us) log.cc:844] Serialized 220 byte log entry
0709 12:18:42.002050 (+224514us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:42.031053 (+ 29003us) write_op.cc:454] Released schema lock
0709 12:18:42.031152 (+    99us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:42.031894 (+   742us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:42.031927 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":62533,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":2668,"apply.run_cpu_time_us":78412,"apply.run_wall_time_us":2204926,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108527,"num_ops":25,"prepare.queue_time_us":72390,"prepare.run_cpu_time_us":30585,"prepare.run_wall_time_us":1241840,"raft.queue_time_us":1386,"raft.run_cpu_time_us":2857,"raft.run_wall_time_us":2856,"replication_time_us":835162,"spinlock_wait_cycles":21976832,"thread_start_us":341496,"threads_started":5,"wal-append.queue_time_us":660354,"wal-append.run_cpu_time_us":9994,"wal-append.run_wall_time_us":2442718}]]}
I20260709 12:18:51.494521 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:51.496084 28319 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:51.496892 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:51.476199 27340 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.004s
I20260709 12:18:51.487895 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:51.474043 27963 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:51.505143 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:51.506759 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.507691 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:51.507956 28279 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:51.509878 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:51.510322 27765 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:18:51.511240 27765 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:51.511786 28339 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:51.466065 28153 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.016s
I20260709 12:18:51.478757 28396 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:18:51.515586 28111 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:51.517391 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.517994 27963 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:51.519232 27963 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.521483 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.521677 27951 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.018s
I20260709 12:18:51.522935 27951 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:18:51.523360 27951 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:51.524307 27951 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.525954 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:51.526180 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.468071 27825 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.032s
I20260709 12:18:51.504689 27340 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.529469 28342 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:51.529981 28342 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:51.440618 28365 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.533191 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:51.533494 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.534108 13255 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:51.536490 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:51.530397 28342 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:51.540230 12494 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:51.540367 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:51.544215 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.544839 12563 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 40 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 23.
W20260709 12:18:51.549168 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.549678 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.550660 12571 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 46 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 26.
I20260709 12:18:51.547465 12644 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 36.
I20260709 12:18:51.530777 28395 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:51.506759 28273 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:51.558251 12639 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 39. Current term is 40. Ops: [39.23-39.23]
I20260709 12:18:51.529461 28281 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:51.559954 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.559401 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.564134 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.563872 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.564800 12493 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:51.542841 27951 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:51.569300 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.573122 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.574187 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:51.579068 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.579948 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.580192 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.581000 12645 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:51.580955 12641 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:18:51.584033 12644 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:51.587594 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.589298 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:51.589700 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84684us
I20260709 12:18:51.560992 28395 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.558748 28318 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election lost for term 44. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:51.561236 27340 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.599805 12641 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:51.601302 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.602829 12641 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:51.603986 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.606011 28134 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:51.607863 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.608660 12639 raft_consensus.cc:2376] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45. Re-sending same reply.
I20260709 12:18:51.586805 28333 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
W20260709 12:18:51.608944 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.609581 28333 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:51.592749 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.610762 28333 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.26, Last appended by leader: 26, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.612833 12563 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 46 index: 31. Preceding OpId from leader: term: 49 index: 33. (index mismatch)
I20260709 12:18:51.512930 27765 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.582193 28153 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 40.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.048s
I20260709 12:18:51.626166 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.625597 12611 leader_election.cc:400] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
I20260709 12:18:51.591378 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:51.627068 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.636456 12569 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:51.595104 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.598704 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.598968 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.599349 12638 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 29. Preceding OpId from leader: term: 50 index: 30. (index mismatch)
I20260709 12:18:51.641068 12646 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:18:51.641848 28111 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.642479 12642 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:51.642921 28184 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.643708 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.644536 13258 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [40.27-40.27]
I20260709 12:18:51.645555 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.642518 28111 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:51.532321 28339 consensus_queue.cc:1175] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "32889a507a6b4c74aec421ffbfd1e25d" responder_term: 47 status { last_received { term: 47 index: 26 } last_committed_idx: 25 last_received_current_leader { term: 47 index: 26 } }
I20260709 12:18:51.649461 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:51.618589 28365 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:51.619386 28093 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.619656 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 27144fedf0014de09f007ac6cc3f7275 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:18:51.623855 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:51.593513 12570 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:51.653070 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:51.625324 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.595546 28395 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:51.563090 28273 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.628199 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.627236 12565 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:51.628469 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:51.650907 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.613022 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.654183 12497 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:51.660008 12646 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:51.662948 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43500.5us
I20260709 12:18:51.667995 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:51.678006 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.678251 12644 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:51.630719 28153 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:51.687278 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:51.689783 28405 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:18:51.692598 12564 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:51.696292 12571 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 47 index: 30. (index mismatch)
I20260709 12:18:51.667492 28134 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:51.699819 12639 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 30. Preceding OpId from leader: term: 44 index: 31. (index mismatch)
I20260709 12:18:51.704422 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.683843 28273 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:51.708534 12641 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:18:51.717561 12488 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:51.692287 28153 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:51.723577 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:18:51.727888 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:18:51.722463 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 97185.2us
W20260709 12:18:51.730775 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 134511us
I20260709 12:18:51.731587 12644 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:51.728977 28342 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.685314 27825 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.035s
I20260709 12:18:51.703920 28134 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:51.710894 28318 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:51.737074 12569 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:18:51.710358 28061 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:51.720408 28365 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:51.745874 12642 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:18:51.730518 28111 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.748025 12638 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 26. Preceding OpId from leader: term: 53 index: 28. (index mismatch)
I20260709 12:18:51.753088 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.754297 12488 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:51.749282 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120976us
I20260709 12:18:51.755721 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:51.740172 28134 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:51.756932 12611 leader_election.cc:400] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45.
W20260709 12:18:51.761860 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 109693us
I20260709 12:18:51.762981 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:51.763456 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:51.764984 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:18:51.765532 28271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.814s	user 0.020s	sys 0.021s
W20260709 12:18:51.766009 28271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.852s	user 0.020s	sys 0.021s
I20260709 12:18:51.770182 28024 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:51.788131 12568 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:18:51.789494 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:51.785151 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6069) took 2210 ms. Trace:
I20260709 12:18:51.814559 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.814819 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.817528 12488 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:18:51.805213 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.806770 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.800331 12492 rpcz_store.cc:276] 0709 12:18:49.574875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:49.690394 (+115519us) service_pool.cc:224] Handling call
0709 12:18:49.733432 (+ 43038us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:50.444252 (+710820us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:51.744991 (+1300739us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54000c44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:51.785033 (+ 40042us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1630497152}
I20260709 12:18:51.825691 12565 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:51.846748 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.850437 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:51.789920 28296 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:51.852991 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:51.856312 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:51.770985 28153 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:51.804854 28333 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.054s
I20260709 12:18:51.858352 28342 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.859179 28184 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:51.859274 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:51.805930 28287 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.054s
I20260709 12:18:51.860240 12565 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:51.861538 28365 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election won for term 52
W20260709 12:18:51.862346 28419 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:18:51.863276 12572 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:18:51.864393 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.864928 12572 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:18:51.866006 28061 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:51.866211 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.866768 28061 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:51.867031 12492 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:51.868302 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.868103 28061 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.29, Last appended by leader: 29, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.869038 12492 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:51.852682 28318 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:51.862275 28365 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:51.846411 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:51.877976 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:51.879307 12565 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:18:51.879769 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 126537us
I20260709 12:18:51.880574 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:51.884981 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:51.887964 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 98568.2us
I20260709 12:18:51.886685 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:51.887993 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.813282 28396 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:18:51.898593 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:51.878940 28024 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:51.902910 28024 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.27, Last appended by leader: 27, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.883108 27963 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 26, Time since last communication: 0.003s
I20260709 12:18:51.892606 28333 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:51.892812 28287 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:51.930497 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:51.904008 28318 raft_consensus.cc:993] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: : Instructing follower 0a29b5d9f9d44298b02ffcd460811eef to start an election
I20260709 12:18:51.914006 28333 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.919764 28296 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:51.906162 27825 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.909258 27963 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:51.914855 28287 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.908886 28003 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
W20260709 12:18:51.952867 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 188021us
I20260709 12:18:51.949115 27963 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:51.936256 28395 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.913064 28055 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:18:51.946236 28333 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:51.942965 28296 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:51.950565 28287 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:51.971436 27963 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 50.29, Last appended by leader: 29, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:51.979305 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:51.916997 28134 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:51.992578 12492 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:18:52.000705 28308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eadf951b63a43fd993cea2c96d5214f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:18:51.995234 28296 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:52.003546 27340 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.004504 27967 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:51.951941 27825 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:52.016124 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 109842us
I20260709 12:18:52.003012 28184 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:52.044697 12497 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:52.047315 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:52.055496 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4651422599999933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
I20260709 12:18:52.059056 12496 raft_consensus.cc:2393] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:52.069409 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:52.066999 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 169554us
W20260709 12:18:52.072888 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 174370us
I20260709 12:18:52.076051 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:52.111723 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.111157 12645 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.517s	user 0.016s	sys 0.000s
I20260709 12:18:52.115602 12645 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:52.067791 27963 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:52.065616 28184 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:52.130291 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:52.130908 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55371.6us
I20260709 12:18:52.128022 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:51.996958 28365 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.134537 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:52.067309 27809 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:52.137900 12493 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:18:52.146662 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:52.148041 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.149240 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:52.150337 12492 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:52.151531 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.058398 27340 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:18:52.187073 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.140633 28184 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:52.188334 12493 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:52.199894 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:52.201511 12566 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:52.133163 28318 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 50.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.058s
I20260709 12:18:52.231630 12493 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:52.235832 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.234973 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.127372 27967 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.200695 28055 consensus_queue.cc:1243] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:18:52.252539 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.259445 28440 raft_consensus.cc:1081] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Signalling peer 0a29b5d9f9d44298b02ffcd460811eef to start an election
W20260709 12:18:52.260255 28422 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:18:52.260136 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 186685us
W20260709 12:18:52.261404 28428 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:18:52.261806 28429 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20260709 12:18:52.263010 28427 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:18:52.263893 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:52.278491 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 205072us
I20260709 12:18:52.280844 12493 raft_consensus.cc:2410] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 48 because replica has last-logged OpId of term: 47 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 25.
W20260709 12:18:52.275804 28281 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.692s	user 0.007s	sys 0.008s
W20260709 12:18:52.291417 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 167048us
W20260709 12:18:52.289331 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 152160us
I20260709 12:18:52.296759 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.279983 28055 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
W20260709 12:18:52.290993 28279 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.782s	user 0.015s	sys 0.001s
I20260709 12:18:52.306807 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:52.298543 28055 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:52.309909 12493 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:18:52.310673 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:52.312085 12571 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:52.312933 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:52.310725 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:52.293771 28395 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:52.316861 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.319125 12567 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 42: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:52.289953 28318 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.023s
I20260709 12:18:52.327836 12493 raft_consensus.cc:2368] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51.
I20260709 12:18:52.308434 28134 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:52.335819 28028 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.336899 27967 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:52.337621 27967 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:52.338718 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:52.339133 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.339874 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:52.340227 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.340801 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:52.341321 28444 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:52.341804 28444 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:52.342774 28444 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 50.28, Last appended by leader: 28, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.344499 28444 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:52.344975 28061 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:52.345613 28061 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:52.345811 12566 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 28. Preceding OpId from leader: term: 44 index: 29. (index mismatch)
W20260709 12:18:52.346254 28431 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:18:52.346715 27833 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.346647 28061 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.347523 27833 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.341442 12494 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:52.355491 28055 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 44.23, Last appended by leader: 23, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.358583 27340 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:52.288801 28440 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.036s
I20260709 12:18:52.368995 28440 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.315066 28336 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:52.370122 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62408.8us
I20260709 12:18:52.336585 28028 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.379272 12494 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:52.382659 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:52.384976 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75673.6us
I20260709 12:18:52.382361 27967 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.005s
I20260709 12:18:52.370131 27825 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:52.390638 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.011s	sys 0.007s
W20260709 12:18:52.391067 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.011s	sys 0.007s
W20260709 12:18:52.391776 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 636076us
I20260709 12:18:52.392874 28430 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:52.393748 28430 consensus_queue.cc:1243] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 44.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 28
I20260709 12:18:52.321310 28281 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.065s
I20260709 12:18:52.384965 28440 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.383186 28336 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.414095 12572 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 46 index: 30. (index mismatch)
I20260709 12:18:52.415848 13255 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 28. Preceding OpId from leader: term: 50 index: 30. (index mismatch)
I20260709 12:18:52.416847 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:52.442724 28281 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:52.443290 13258 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:52.443419 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.444783 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:52.417560 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:52.410802 28395 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.039s
I20260709 12:18:52.448598 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2950) took 4370 ms. Trace:
I20260709 12:18:52.449556 28158 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:18:52.450019 28158 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:52.451061 28158 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.419862 12570 pending_rounds.cc:85] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 28
I20260709 12:18:52.452917 12570 pending_rounds.cc:107] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 44.29
I20260709 12:18:52.453190 12570 raft_consensus.cc:2889] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:52.426606 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:52.454144 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71620.6us
I20260709 12:18:52.429085 12642 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.28-43.28]
I20260709 12:18:52.428254 12638 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.24-43.24]
I20260709 12:18:52.425587 12489 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:52.432055 12491 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.28-43.28]
I20260709 12:18:52.426942 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.444901 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.428956 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:52.405611 12490 raft_consensus.cc:1240] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 53. Current term is 54. Ops: []
I20260709 12:18:52.426163 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.432866 12494 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:52.435603 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:52.449115 12566 rpcz_store.cc:276] 0709 12:18:48.078166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.078334 (+   168us) service_pool.cc:224] Handling call
0709 12:18:48.078654 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:49.469053 (+1390399us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54002b1c20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf577106f 00007ffbf57e9f60 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62
0709 12:18:50.818997 (+1349944us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54002b1c20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf5770520 00007ffbf57e9f70 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62
0709 12:18:52.190692 (+1371695us) spinlock_profiling.cc:243] Waited 759 ms on lock 0x7b58008f09d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:18:52.346432 (+155740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:52.448186 (+101754us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b5400245ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:52.448488 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3665666688}
I20260709 12:18:52.426146 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.493882 12644 raft_consensus.cc:2393] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:52.531920 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.532735 12644 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:18:52.536088 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73277.9us
I20260709 12:18:52.544039 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 50 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.494959 28028 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:52.423054 27340 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.107s
I20260709 12:18:52.499104 28281 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.041s
W20260709 12:18:52.572504 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148150us
I20260709 12:18:52.507472 28296 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.531159 27967 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.041s
I20260709 12:18:52.576785 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.585145 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 46 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.586133 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.534714 28024 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.041s
W20260709 12:18:52.573889 28273 consensus_meta.cc:287] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.835s	user 0.017s	sys 0.000s
I20260709 12:18:52.590150 12643 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 51 because replica has last-logged OpId of term: 48 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 28.
I20260709 12:18:52.556609 28430 consensus_queue.cc:1059] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 53.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.036s
I20260709 12:18:52.592773 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:52.593732 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56490.6us
I20260709 12:18:52.597764 12641 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:18:52.598146 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:52.593156 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:52.590001 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 269161us
I20260709 12:18:52.599900 12644 raft_consensus.cc:2393] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:52.576216 28448 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.579139 28028 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.558020 28425 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.029s
I20260709 12:18:52.581355 28322 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:52.580871 28339 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:52.614318 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.619745 12642 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:52.578941 28440 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:52.620596 12638 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:52.628592 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.634048 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.634941 12637 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:52.629537 12643 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:18:52.637686 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:52.620973 28287 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.605s	user 0.009s	sys 0.005s
I20260709 12:18:52.643396 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:52.617082 27340 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:18:52.653960 12642 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 47 index: 30. (index mismatch)
I20260709 12:18:52.622942 28339 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:52.662096 13253 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:52.616621 28448 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.601729 28273 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.661072 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.676692 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.677358 12638 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:52.677943 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.679028 12644 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 36: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:52.688015 13258 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:52.653450 27340 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:52.682863 28380 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.022s
I20260709 12:18:52.704182 28158 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:52.709388 28060 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.026s
W20260709 12:18:52.711529 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.713112 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:52.720835 12490 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:18:52.727093 12493 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:18:52.727526 28153 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:52.731724 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60242.3us
I20260709 12:18:52.732409 12494 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:52.732827 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:18:52.735402 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.736559 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.643527 28028 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.645208 28440 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.041s
I20260709 12:18:52.646098 28287 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.760686 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 39.
I20260709 12:18:52.765486 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.765269 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:52.767505 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2390) took 1144 ms. Trace:
I20260709 12:18:52.771126 12645 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:18:52.773404 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.772352 12496 rpcz_store.cc:276] 0709 12:18:51.622807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.627865 (+  5058us) service_pool.cc:224] Handling call
0709 12:18:52.767234 (+1139369us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54000f5ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:52.767434 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":845977984}
W20260709 12:18:52.788309 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6078) took 1493 ms (client timeout 1733 ms). Trace:
I20260709 12:18:52.788487 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:52.789218 12491 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:52.789089 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:52.770324 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:52.788870 12489 rpcz_store.cc:269] 0709 12:18:51.295020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.298756 (+  3736us) service_pool.cc:224] Handling call
0709 12:18:52.788002 (+1489246us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54000f73d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:52.788168 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2248156544}
I20260709 12:18:52.796063 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.734959 27340 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
W20260709 12:18:52.769433 12610 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.733s (SENT) [suppressed 11 similar messages]
I20260709 12:18:52.806186 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:52.770478 28333 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.754s	user 0.015s	sys 0.000s
I20260709 12:18:52.682977 28338 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.163s
I20260709 12:18:52.812871 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:52.805441 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.825160 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:52.808717 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43543.6us
I20260709 12:18:52.834350 12639 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:52.834622 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:52.807246 12563 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:52.837733 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:52.788646 28279 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:52.838459 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:52.796303 28111 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:52.847535 12642 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:18:52.853051 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.856477 12644 raft_consensus.cc:2368] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:52.825254 28281 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:18:52.858134 12461 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:52.822036 28060 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:52.859277 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:52.859788 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:52.765784 28339 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.27, Last appended by leader: 27, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:52.832372 28184 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.595s	user 0.013s	sys 0.000s
I20260709 12:18:52.866734 12645 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:18:52.869024 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:52.878837 12642 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 48 because replica has last-logged OpId of term: 47 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 25.
I20260709 12:18:52.871691 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:52.879678 12563 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:18:52.872455 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:52.842969 28443 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:52.843137 28124 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.865804 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:52.849304 28061 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:52.849071 28446 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:18:52.894959 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:52.842370 28279 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:52.888522 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:52.889238 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.901355 28285 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:52.881867 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:52.902076 28285 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:52.903059 12489 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: [50.36-50.36]
W20260709 12:18:52.904430 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62989.8us
I20260709 12:18:52.906028 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:52.906448 28124 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.904675 28285 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:52.907446 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:52.909169 28124 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:52.910202 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44485.7us
I20260709 12:18:52.913231 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:52.905961 28416 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.168s	user 0.005s	sys 0.002s
I20260709 12:18:52.921725 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:52.910326 28285 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:52.931777 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:52.932511 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50752.6us
I20260709 12:18:52.935025 12494 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:52.937244 12644 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:52.934460 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:52.832831 28333 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:52.936874 28415 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:18:52.936208 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:52.899121 28137 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:52.899806 28443 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:52.897507 28296 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:53.046429 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110336us
I20260709 12:18:53.056418 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:52.930909 28396 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:18:52.946144 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:53.046540 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 363231us
I20260709 12:18:53.052428 13262 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
W20260709 12:18:53.057698 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 288285us
I20260709 12:18:52.933215 28285 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:52.939484 27967 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:52.937686 28125 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:52.937017 28452 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:52.942360 28124 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260709 12:18:53.068315 12610 leader_election.cc:400] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51.
I20260709 12:18:53.069264 12610 leader_election.cc:403] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Cancelling election due to peer responding with higher term
I20260709 12:18:52.891577 28184 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.054520 28137 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.081498 12645 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:53.055198 28339 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:53.051596 28281 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:18:53.083837 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 246251us
I20260709 12:18:53.060261 28061 raft_consensus.cc:2764] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:18:53.095724 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:53.057874 28443 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:52.914160 28111 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:52.939630 28134 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.133s
I20260709 12:18:53.058701 28286 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:53.093644 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:53.064550 28425 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:18:53.068037 28285 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:53.068274 27967 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:53.115165 28281 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:53.119592 28124 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
W20260709 12:18:53.124645 28455 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.283s	user 0.000s	sys 0.000s
I20260709 12:18:53.124749 28281 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:53.126665 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4565) took 1073 ms. Trace:
I20260709 12:18:53.087013 28137 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:53.141047 12489 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:53.095337 28338 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:53.147038 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90718.4us
I20260709 12:18:53.148793 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:53.127339 12495 rpcz_store.cc:276] 0709 12:18:52.053164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.406570 (+353406us) service_pool.cc:224] Handling call
0709 12:18:52.406932 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:52.407387 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:52.407419 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:52.408091 (+   672us) log.cc:844] Serialized 26 byte log entry
0709 12:18:52.585294 (+177203us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:52.585344 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:52.585419 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:53.126401 (+540982us) raft_consensus.cc:1685] finished
0709 12:18:53.126424 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:53.126587 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9297,"raft.queue_time_us":3122,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":146,"thread_start_us":176701,"threads_started":1,"wal-append.queue_time_us":346307}
I20260709 12:18:53.080214 28125 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.113296 28060 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.042s
W20260709 12:18:53.139976 28454 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.445s	user 0.000s	sys 0.002s
I20260709 12:18:53.126581 28281 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.29, Last appended by leader: 29, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.164551 28333 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:53.135564 28296 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:18:53.166282 28464 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.243s	user 0.000s	sys 0.000s
I20260709 12:18:53.175683 27779 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:53.176767 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:53.177452 28444 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.833s	user 0.009s	sys 0.007s
I20260709 12:18:53.158373 28448 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:53.179191 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 233094us
I20260709 12:18:53.165403 28055 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:53.121773 27967 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:53.117487 28273 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.155491 28124 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:53.096769 28279 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.147136 28158 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.057s
W20260709 12:18:53.203076 27963 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.060s	user 0.013s	sys 0.003s
I20260709 12:18:53.222098 12495 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:18:53.224782 28279 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:53.155485 28380 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.059s
I20260709 12:18:53.232725 12492 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 51 because replica has last-logged OpId of term: 50 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 28.
W20260709 12:18:53.234257 12637 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.000s
I20260709 12:18:53.234754 12637 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:53.237560 27340 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.209837 28287 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:53.204746 27765 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
W20260709 12:18:53.231356 28460 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.368s	user 0.000s	sys 0.002s
I20260709 12:18:53.211557 28296 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:53.212550 28286 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.212354 27779 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:53.219915 28124 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:53.230458 28333 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:53.254587 12638 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.565s	user 0.013s	sys 0.001s
W20260709 12:18:53.256067 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 324340us
W20260709 12:18:53.254894 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78270.3us
I20260709 12:18:53.286728 12638 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:53.154677 28338 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.249655 28028 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.239761 28452 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election lost for term 50. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51.
I20260709 12:18:53.271394 27340 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.298671 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:53.264653 27765 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:53.231845 28134 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.027s
I20260709 12:18:53.282240 12489 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 46 index: 30. (index mismatch)
I20260709 12:18:53.289340 28124 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:53.275584 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80802.7us
I20260709 12:18:53.303346 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:53.298959 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 98958.1us
I20260709 12:18:53.281129 27779 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:53.327509 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:53.283723 28396 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:18:53.330562 28473 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:18:53.282733 28286 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.331228 28287 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:53.306010 28479 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.325858 28319 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:53.411648 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:53.380458 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52449us
I20260709 12:18:53.387868 28488 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.327351 28184 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:53.387456 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4556) took 1432 ms. Trace:
W20260709 12:18:53.380795 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53419.6us
W20260709 12:18:53.412328 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.328828 27779 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:53.277379 27963 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260709 12:18:53.419633 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:53.300107 28333 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:53.388649 28469 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.119s	user 0.006s	sys 0.001s
I20260709 12:18:53.417016 12488 rpcz_store.cc:276] 0709 12:18:51.955050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.404735 (+449685us) service_pool.cc:224] Handling call
0709 12:18:52.405097 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:52.405391 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:52.405416 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:52.431578 (+ 26162us) log.cc:844] Serialized 26 byte log entry
0709 12:18:52.670176 (+238598us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:52.670240 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:52.670348 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:53.387218 (+716870us) raft_consensus.cc:1685] finished
0709 12:18:53.387242 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:53.387390 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":143023,"threads_started":1,"wal-append.queue_time_us":271493}
W20260709 12:18:53.391005 28476 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:18:53.418951 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:53.398427 28481 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:18:53.412979 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:53.421566 12639 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:53.384693 28448 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:18:53.380808 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73704.5us
I20260709 12:18:53.304328 28028 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.108s
I20260709 12:18:53.432025 28338 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.433225 28491 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:53.391084 28336 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.436311 12460 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:53.437086 28491 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:53.398865 28273 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:53.412719 28319 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:53.433079 27765 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.442307 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:18:53.397974 28484 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.031s
W20260709 12:18:53.459661 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.443626 28125 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:53.457439 28432 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.054s	user 0.004s	sys 0.000s
I20260709 12:18:53.424232 28395 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.039s
W20260709 12:18:53.478444 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65538.1us
I20260709 12:18:53.474110 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:53.523715 12638 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:53.428900 28287 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:53.455871 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:53.454105 28279 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:53.550103 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.447570 28273 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:53.454795 28319 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:53.462613 27833 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.415138 28479 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.426766 28296 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.453217 28448 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:53.420832 28488 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.552759 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:53.554643 12572 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:53.505366 28365 raft_consensus.cc:3055] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 LEADER]: Stepping down as leader of term 53
W20260709 12:18:53.557281 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.557531 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:53.565961 12570 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6"
dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:32884
I20260709 12:18:53.574985 12570 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:18:53.575426 12570 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:53.575786 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.576759 12564 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 43 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 27.
W20260709 12:18:53.528218 28322 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.895s	user 0.009s	sys 0.005s
I20260709 12:18:53.504722 28111 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.560905 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:53.452034 28489 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:53.522161 28336 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:53.463299 28491 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.570763 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.584463 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:53.455183 27967 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.596686 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:53.560014 27765 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:53.562193 27963 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:53.564460 12569 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:53.623090 12569 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:53.606988 12640 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 47 because replica has last-logged OpId of term: 45 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 24.
I20260709 12:18:53.620764 28308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:50712 (request call id 1958) took 5375 ms. Trace:
I20260709 12:18:53.633327 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:53.634136 12567 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:18:53.621570 28478 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
W20260709 12:18:53.646464 28483 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:18:53.521797 28396 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.655088 28488 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.657248 28333 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.660192 28279 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:53.670226 12642 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 45 because replica has last-logged OpId of term: 44 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
I20260709 12:18:53.670795 28479 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:53.672551 12642 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: []
I20260709 12:18:53.624071 12490 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:53.623178 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:53.626165 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:53.674144 12568 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:53.630211 28435 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.681960 28435 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.528954 28443 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.573400 28365 raft_consensus.cc:740] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:53.634053 28336 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:53.638022 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77179.8us
I20260709 12:18:53.639030 12638 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 40. Current term is 41. Ops: [40.27-40.27]
I20260709 12:18:53.640782 28491 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.641979 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.633560 28308 rpcz_store.cc:276] 0709 12:18:48.245586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.245878 (+   292us) service_pool.cc:224] Handling call
0709 12:18:48.917374 (+671496us) ts_tablet_manager.cc:934] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
0709 12:18:51.979787 (+3062413us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:53.166020 (+1186233us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:53.166171 (+   151us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:53.619787 (+453616us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:53.620155 (+   368us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:53.620510 (+   355us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:53.620650 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":150211,"tablet-copy.queue_time_us":434901,"thread_start_us":611577,"threads_started":2}
I20260709 12:18:53.632062 27963 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:53.645002 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.717888 12568 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
W20260709 12:18:53.703888 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 248052us
I20260709 12:18:53.720202 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:18:53.712121 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.721072 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:53.722090 12489 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:53.706977 12569 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.723646 12569 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:53.735296 12565 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:18:53.747416 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.740199 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:53.748430 12564 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
W20260709 12:18:53.753361 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:53.763908 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4569) took 1530 ms. Trace:
I20260709 12:18:53.764657 12491 rpcz_store.cc:276] 0709 12:18:52.233848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.506569 (+272721us) service_pool.cc:224] Handling call
0709 12:18:52.769560 (+262991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:52.795768 (+ 26208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.763495 (+967727us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54000f5ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:53.763787 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":603832576}
I20260709 12:18:53.767664 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:53.773033 12569 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:18:53.719004 27963 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:53.789526 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:53.790793 12491 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:53.792467 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:53.793237 12491 raft_consensus.cc:2376] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50. Re-sending same reply.
I20260709 12:18:53.794023 28482 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.794416 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:53.795058 27951 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:53.795882 28407 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.795717 27951 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.796662 12491 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:18:53.796466 28407 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.797883 28111 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:53.798532 28111 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:53.799692 27951 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.799790 28111 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 45.32, Last appended by leader: 32, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.801229 28407 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.808652 12637 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 30. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:18:53.817124 28407 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.817873 28407 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.818240 28365 consensus_queue.cc:260] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 53.31, Last appended by leader: 31, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.819839 28365 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:53.820353 28407 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.821100 27951 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:53.821695 27951 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:53.824069 27951 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.656131 28448 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 48.28, Last appended by leader: 28, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.702929 27779 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.747777 28308 tablet_copy_client.cc:806] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
W20260709 12:18:53.843024 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 92035.5us
I20260709 12:18:53.805239 28435 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:53.737948 28491 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:18:53.794776 28482 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.841079 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:53.844535 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:53.780933 28055 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:53.860548 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.871023 12536 leader_election.cc:400] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:53.871351 12564 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:53.871673 12536 leader_election.cc:403] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
W20260709 12:18:53.873509 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 55850.3us
I20260709 12:18:53.875119 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:53.879356 28279 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:53.880136 27951 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:53.880914 28055 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:53.881146 28336 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.881428 28124 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.882407 27967 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.882148 28055 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.884074 28124 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.885100 28308 tablet_copy_client.cc:670] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
W20260709 12:18:53.887199 12640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5778) took 2265 ms (client timeout 2280 ms). Trace:
W20260709 12:18:53.887877 12640 rpcz_store.cc:269] 0709 12:18:51.621570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.646157 (+ 24587us) service_pool.cc:224] Handling call
0709 12:18:53.886929 (+2240772us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001f81d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:53.887109 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4438244736}
I20260709 12:18:53.893383 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2247) took 1292 ms. Trace:
I20260709 12:18:53.893916 12641 rpcz_store.cc:276] 0709 12:18:52.601443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.601684 (+   241us) service_pool.cc:224] Handling call
0709 12:18:52.626961 (+ 25277us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:52.650632 (+ 23671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:52.650685 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:52.680693 (+ 30008us) log.cc:844] Serialized 52 byte log entry
0709 12:18:52.825199 (+144506us) raft_consensus.cc:1645] Marking committed up to 35
0709 12:18:52.866953 (+ 41754us) log.cc:844] Serialized 14 byte log entry
0709 12:18:52.885665 (+ 18712us) log.cc:844] Serialized 14 byte log entry
0709 12:18:52.885988 (+   323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:52.886103 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:53.391988 (+505885us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b18003c4180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:53.798744 (+406756us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18003c4180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:53.890343 (+ 91599us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b18003c4180. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:53.893027 (+  2684us) raft_consensus.cc:1685] finished
0709 12:18:53.893071 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:53.893307 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":109288,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":6838,"raft.queue_time_us":22675,"raft.run_cpu_time_us":509,"raft.run_wall_time_us":37402,"spinlock_wait_cycles":46913280,"thread_start_us":116561,"threads_started":1,"wal-append.queue_time_us":348570}
I20260709 12:18:53.898962 28443 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:53.849571 28491 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:53.900652 28491 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:18:53.901736 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 806004us
I20260709 12:18:53.899946 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.904582 12565 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:18:53.878445 28490 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.459s	user 0.005s	sys 0.000s
I20260709 12:18:53.907876 12570 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:53.887277 28482 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:53.950637 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:53.872762 28435 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:53.993768 12568 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:53.955601 28124 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.014055 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:54.004334 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 129321us
W20260709 12:18:54.005230 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54682.8us
W20260709 12:18:54.008738 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4618) took 1327 ms (client timeout 1557 ms). Trace:
W20260709 12:18:54.016588 12643 rpcz_store.cc:269] 0709 12:18:52.680971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.681482 (+   511us) service_pool.cc:224] Handling call
0709 12:18:54.008522 (+1327040us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:54.008671 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1093189376}
I20260709 12:18:54.016815 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.012380 12567 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 46 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 26.
I20260709 12:18:54.010665 12570 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.020351 12489 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:18:54.013875 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2006) took 1365 ms. Trace:
I20260709 12:18:53.946658 28279 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:53.958349 27833 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:18:53.946396 28493 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.133s
I20260709 12:18:54.038578 27779 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.026095 28482 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:54.033337 28338 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:54.095539 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.095995 28338 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:54.095817 28482 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.023720 28124 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.097504 28338 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:54.099951 28124 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.039901 28480 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:54.102015 28480 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.042796 28423 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.030s
I20260709 12:18:54.013934 28425 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:54.043218 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.105984 28425 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:54.106515 28425 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:54.046026 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.107342 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.108412 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:54.047552 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:54.048661 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2234) took 1183 ms. Trace:
I20260709 12:18:54.050678 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.051504 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.111119 12567 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:54.066947 13257 rpcz_store.cc:276] 0709 12:18:52.648839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.649774 (+   935us) service_pool.cc:224] Handling call
0709 12:18:52.671868 (+ 22094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:52.683330 (+ 11462us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:52.858901 (+175571us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.013448 (+1154547us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:54.013504 (+    56us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:54.013571 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.013683 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.013817 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":776944,"prepare.queue_time_us":223074,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":785161,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":17347,"thread_start_us":482148,"threads_started":2,"wal-append.queue_time_us":400709}
I20260709 12:18:54.112223 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.113034 12567 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:18:54.053054 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:53.999979 28435 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.026589 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:54.121841 13256 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:53.994938 28488 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:54.039752 28184 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:54.119984 12571 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:54.129853 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:54.110628 28153 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.363s	user 0.012s	sys 0.004s
W20260709 12:18:54.134195 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.109818 13255 rpcz_store.cc:276] 0709 12:18:52.865898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.866050 (+   152us) service_pool.cc:224] Handling call
0709 12:18:52.871354 (+  5304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:52.871613 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 31
0709 12:18:53.061825 (+190212us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.048274 (+986449us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:54.048339 (+    65us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:18:54.048390 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.048465 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.048602 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":743287,"prepare.queue_time_us":35245,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":743391,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":64,"thread_start_us":400201,"threads_started":2,"wal-append.queue_time_us":414562}
I20260709 12:18:54.133571 12564 raft_consensus.cc:2410] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 47 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 29.
W20260709 12:18:54.127954 28339 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.022s	user 0.012s	sys 0.002s
I20260709 12:18:54.118614 28480 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:54.166508 12571 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:54.128399 28435 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:54.133534 28446 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:54.190706 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6099) took 1422 ms. Trace:
W20260709 12:18:54.193986 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.208787 12495 rpcz_store.cc:276] 0709 12:18:52.768737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.225127 (+456390us) service_pool.cc:224] Handling call
0709 12:18:53.225498 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.225956 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.225984 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:53.226525 (+   541us) log.cc:844] Serialized 26 byte log entry
0709 12:18:53.301357 (+ 74832us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:53.301410 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:53.301482 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:53.801217 (+499735us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:54.167916 (+366699us) raft_consensus.cc:1685] finished
0709 12:18:54.167941 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.190614 (+ 22673us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55099,"raft.queue_time_us":77279,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":22582,"spinlock_wait_cycles":26584320,"thread_start_us":59311,"threads_started":1,"wal-append.queue_time_us":171667}
I20260709 12:18:54.186324 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4572) took 1872 ms. Trace:
I20260709 12:18:54.211858 12494 rpcz_store.cc:276] 0709 12:18:52.315463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.759857 (+444394us) service_pool.cc:224] Handling call
0709 12:18:52.823527 (+ 63670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:52.935525 (+111998us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.186018 (+1250493us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:54.186246 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":255872896}
I20260709 12:18:54.221097 28279 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 41.22, Last appended by leader: 22, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:54.225992 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 458502us
W20260709 12:18:54.214069 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 163609us
W20260709 12:18:54.232014 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.208710 28486 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:18:54.250339 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:54.251246 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 190098us
W20260709 12:18:54.261682 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.252187 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.253818 13256 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:18:54.219348 28285 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.063s	user 0.014s	sys 0.001s
I20260709 12:18:54.268889 12497 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:18:54.189414 28443 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.141894 28184 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.184109 28061 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:54.278126 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.286085 13261 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:54.286738 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:54.291661 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: true dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:54.291931 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.292510 12496 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:54.292748 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.167213 28488 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.298494 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:54.296881 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 394018us
I20260709 12:18:54.300215 12488 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:54.294626 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:54.293968 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.306815 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:54.278448 28281 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:54.294572 28137 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.143s	user 0.012s	sys 0.003s
I20260709 12:18:54.317516 28458 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:54.319432 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:54.311389 12495 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:54.283521 28125 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:54.299973 28482 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.289799 28443 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:54.336048 28448 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.000s	sys 0.019s
I20260709 12:18:54.286080 28285 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:54.359972 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59040.8us
I20260709 12:18:54.305578 28491 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.368343 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6096) took 2065 ms. Trace:
I20260709 12:18:54.370222 28491 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:54.368839 12493 rpcz_store.cc:276] 0709 12:18:52.302914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.729399 (+426485us) service_pool.cc:224] Handling call
0709 12:18:52.740196 (+ 10797us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:52.764875 (+ 24679us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:18:53.085226 (+320351us) log.cc:844] Serialized 14 byte log entry
0709 12:18:53.304752 (+219526us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:54.066648 (+761896us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:54.120397 (+ 53749us) log.cc:844] Serialized 26 byte log entry
0709 12:18:54.120620 (+   223us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:54.120658 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.120736 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.367974 (+247238us) raft_consensus.cc:1685] finished
0709 12:18:54.368108 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.368267 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10604,"prepare.queue_time_us":261919,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":5673,"raft.queue_time_us":87391,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":214,"spinlock_wait_cycles":5500672,"thread_start_us":411830,"threads_started":2,"wal-append.queue_time_us":286670}
I20260709 12:18:54.371153 12494 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:54.373522 12642 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: [46.24-46.24]
I20260709 12:18:54.299587 28061 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.378996 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2254) took 1613 ms. Trace:
I20260709 12:18:54.380317 12645 rpcz_store.cc:276] 0709 12:18:52.765606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.920836 (+155230us) service_pool.cc:224] Handling call
0709 12:18:52.946834 (+ 25998us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:53.082089 (+135255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.378528 (+1296439us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:54.378884 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309203712}
I20260709 12:18:54.358810 28458 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:18:54.324223 28137 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:54.375361 28319 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.805s	user 0.009s	sys 0.006s
I20260709 12:18:54.388957 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:54.376111 28448 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.000s	sys 0.020s
W20260709 12:18:54.393358 13218 connection.cc:256] RPC call timeout handler was delayed by -1.1043784600001345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
I20260709 12:18:54.374804 28285 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:18:54.396239 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.397892 13218 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:54.333830 28281 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:54.416417 12488 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 54. Current term is 55. Ops: []
W20260709 12:18:54.394727 28498 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.174s	user 0.004s	sys 0.000s
I20260709 12:18:54.388677 28480 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:54.366106 28125 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.431234 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.430117 12493 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
I20260709 12:18:54.405843 28430 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:54.457554 12496 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:54.459002 12495 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:18:54.457520 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:54.432505 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:54.401407 28458 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:54.403813 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2008) took 1732 ms. Trace:
I20260709 12:18:54.428916 28491 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:54.486423 12489 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:54.488894 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2246) took 1893 ms. Trace:
W20260709 12:18:54.490295 28495 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:18:54.490423 28061 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.491070 28458 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 27, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.494367 28488 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:54.495085 12565 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 52 index: 30. (index mismatch)
I20260709 12:18:54.487793 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:54.458087 28468 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:54.483464 28476 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:18:54.506074 28503 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:18:54.403656 28319 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:54.505394 28489 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.901s	user 0.010s	sys 0.004s
I20260709 12:18:54.479010 13260 rpcz_store.cc:276] 0709 12:18:52.671269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.672140 (+   871us) service_pool.cc:224] Handling call
0709 12:18:52.688066 (+ 15926us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:52.705894 (+ 17828us) raft_consensus.cc:1534] Early marking committed up to index 36
0709 12:18:52.856958 (+151064us) log.cc:844] Serialized 14 byte log entry
0709 12:18:53.070580 (+213622us) log.cc:844] Serialized 14 byte log entry
0709 12:18:53.900781 (+830201us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b180049f480. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:18:54.389476 (+488695us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.399625 (+ 10149us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:54.399699 (+    74us) raft_consensus.cc:1645] Marking committed up to 36
0709 12:18:54.399791 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.400379 (+   588us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.403635 (+  3256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50999,"prepare.queue_time_us":203444,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":92,"raft.queue_time_us":1820071,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":25812,"spinlock_wait_cycles":37450880,"thread_start_us":778827,"threads_started":3,"wal-append.queue_time_us":321155}
I20260709 12:18:54.506300 28273 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.28, Last appended by leader: 28, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.489065 28285 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.009s
I20260709 12:18:54.522171 13259 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:54.500735 28491 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.549329 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.457736 28480 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.489475 12646 rpcz_store.cc:276] 0709 12:18:52.595162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.595350 (+   188us) service_pool.cc:224] Handling call
0709 12:18:52.622407 (+ 27057us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:52.839033 (+216626us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:52.839070 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:52.865193 (+ 26123us) log.cc:844] Serialized 52 byte log entry
0709 12:18:53.254171 (+388978us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:53.254212 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.478032 (+1223820us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:54.478102 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.478164 (+    62us) raft_consensus.cc:1685] finished
0709 12:18:54.478186 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.488786 (+ 10600us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7035,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":10591,"spinlock_wait_cycles":434082048,"thread_start_us":199802,"threads_started":1,"wal-append.queue_time_us":363855}
I20260709 12:18:54.479578 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:54.517544 27963 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.669s	user 0.013s	sys 0.000s
I20260709 12:18:54.559784 12644 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
W20260709 12:18:54.517087 28500 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:18:54.508003 28493 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.510835 28448 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election won for term 54
W20260709 12:18:54.564507 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4562) took 2563 ms (client timeout 2270 ms). Trace:
I20260709 12:18:54.519938 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:54.524478 28346 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:54.526160 28489 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:18:54.554199 28491 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:18:54.571822 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51985.4us
W20260709 12:18:54.568284 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1332704649998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20260709 12:18:54.577699 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0999704649998421s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 12:18:54.578179 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0963704649998363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20260709 12:18:54.578614 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0963704649998363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20260709 12:18:54.579047 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0792704649998086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
W20260709 12:18:54.579478 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0045704649999152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20260709 12:18:54.564922 28273 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:18:54.570035 12492 rpcz_store.cc:269] 0709 12:18:52.001169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.255948 (+254779us) service_pool.cc:224] Handling call
0709 12:18:54.554079 (+2298131us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54000f55d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:54.564367 (+ 10288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2187145088}
I20260709 12:18:54.584720 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:54.566848 28184 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.567145 28448 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:54.563802 27963 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.570794 28493 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.626379 28448 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:54.632962 28443 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.651257 28137 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.655525 28338 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:18:54.657130 28519 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:18:54.660260 28480 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.633095 28521 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.636446 28285 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
W20260709 12:18:54.662945 28505 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.242s	user 0.000s	sys 0.002s
W20260709 12:18:54.669569 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 271778us
W20260709 12:18:54.689620 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 106470us
I20260709 12:18:54.690992 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:54.672447 28285 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:54.681223 28137 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:54.711355 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117397us
I20260709 12:18:54.699908 28285 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:54.697135 28125 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.723623 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.728600 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.729351 12493 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:54.679581 28521 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.717723 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:54.694845 28184 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.021s
I20260709 12:18:54.711316 27765 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:54.739938 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:54.744637 12495 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:18:54.711742 28319 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:54.701139 28467 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 54.35, Next index: 36, Last known committed idx: 35, Time since last communication: 0.030s
I20260709 12:18:54.752158 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:54.753144 12488 raft_consensus.cc:2368] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43.
I20260709 12:18:54.753464 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5796) took 1345 ms. Trace:
I20260709 12:18:54.746411 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:54.753963 28512 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:18:54.754355 12491 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:18:54.747371 12496 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.23-40.23]
I20260709 12:18:54.707937 28137 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:54.757138 28504 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:18:54.757869 28137 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.706653 28061 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.032s
I20260709 12:18:54.760933 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:54.705372 28468 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.032s
I20260709 12:18:54.762868 28526 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:18:54.763525 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.764675 12493 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 44 index: 28. (index mismatch)
I20260709 12:18:54.767686 12569 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:54.753993 12637 rpcz_store.cc:276] 0709 12:18:53.408008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.832713 (+424705us) service_pool.cc:224] Handling call
0709 12:18:53.854258 (+ 21545us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.906914 (+ 52656us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:18:54.304545 (+397631us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.563004 (+258459us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:54.572561 (+  9557us) log.cc:844] Serialized 26 byte log entry
0709 12:18:54.572714 (+   153us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:54.572747 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.572837 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.753202 (+180365us) raft_consensus.cc:1685] finished
0709 12:18:54.753221 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.753356 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":514674,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":571,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":38157,"thread_start_us":528848,"threads_started":2,"wal-append.queue_time_us":351402}
I20260709 12:18:54.728114 28507 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.766119 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:54.778667 28515 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:18:54.778483 28534 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 50.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:54.719506 28458 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.049s
W20260709 12:18:54.765409 28522 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.107s	user 0.002s	sys 0.000s
W20260709 12:18:54.782137 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64578.8us
I20260709 12:18:54.784277 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:54.786386 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.787503 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5959) took 1234 ms. Trace:
I20260709 12:18:54.787982 12566 rpcz_store.cc:276] 0709 12:18:53.553535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.553709 (+   174us) service_pool.cc:224] Handling call
0709 12:18:53.563751 (+ 10042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.584133 (+ 20382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.584172 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:53.600494 (+ 16322us) log.cc:844] Serialized 26 byte log entry
0709 12:18:53.885291 (+284797us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b18004e3580. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:18:54.223072 (+337781us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:54.223144 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.223259 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.780465 (+557206us) raft_consensus.cc:1685] finished
0709 12:18:54.780492 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.787417 (+  6925us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24270,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":4183,"spinlock_wait_cycles":11229952,"thread_start_us":337480,"threads_started":1,"wal-append.queue_time_us":744286}
I20260709 12:18:54.794255 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5793) took 1387 ms. Trace:
I20260709 12:18:54.794804 12638 rpcz_store.cc:276] 0709 12:18:53.407024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.708658 (+301634us) service_pool.cc:224] Handling call
0709 12:18:53.728938 (+ 20280us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:53.729504 (+   566us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:53.729547 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:53.783962 (+ 54415us) log.cc:844] Serialized 26 byte log entry
0709 12:18:54.349897 (+565935us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:54.714221 (+364324us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.793800 (+ 79579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.793913 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.793976 (+    63us) raft_consensus.cc:1685] finished
0709 12:18:54.793997 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.794181 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2879,"prepare.queue_time_us":151838,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":1794,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":12357,"thread_start_us":875017,"threads_started":3,"wal-append.queue_time_us":793881}
W20260709 12:18:54.808431 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59473.1us
I20260709 12:18:54.812278 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:54.813422 28279 raft_consensus.cc:2764] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
I20260709 12:18:54.814978 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:54.783208 27963 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.797040 28516 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:18:54.811811 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 91425.5us
I20260709 12:18:54.821295 12637 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
W20260709 12:18:54.821126 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43594us
I20260709 12:18:54.807534 28137 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:54.829842 12638 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
I20260709 12:18:54.781013 28507 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.819247 28055 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.836577 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:54.822218 28467 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:54.822635 28516 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:54.844625 28423 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:54.867573 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:54.868130 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.868804 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.869176 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:54.846516 12637 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:54.869899 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:54.837546 28518 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.820573 28443 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.823957 28184 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:54.833034 28319 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:54.883131 28443 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.883579 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.859181 28055 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.853380 28446 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:54.853786 28528 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:18:54.855984 28365 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.036s	user 0.011s	sys 0.005s
W20260709 12:18:54.856197 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:54.858328 12497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4554) took 2929 ms (client timeout 2227 ms). Trace:
I20260709 12:18:54.859743 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:54.831086 28338 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:18:54.858217 28524 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:18:54.870998 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:18:54.873811 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:54.842869 28286 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:54.876024 28458 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:18:54.867347 28423 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:54.848737 28467 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:54.876976 28507 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.877969 28521 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:54.878228 28435 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.627s	user 0.014s	sys 0.001s
W20260709 12:18:54.880237 28425 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.773s	user 0.013s	sys 0.000s
I20260709 12:18:54.866374 28516 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:54.890776 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.891291 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.866413 28396 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:54.893918 12497 rpcz_store.cc:269] 0709 12:18:51.928558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:52.185807 (+257249us) service_pool.cc:224] Handling call
0709 12:18:54.858086 (+2672279us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b540037e954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:54.858242 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4383286016}
W20260709 12:18:54.912966 28531 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:18:54.899044 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.899421 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.899904 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:54.900518 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:54.901696 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:54.904426 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44762.1us
I20260709 12:18:54.858534 28493 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:54.950045 28493 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:54.874480 28184 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:54.951263 28184 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:54.882969 28518 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.953990 28518 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.933561 28541 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.043s
I20260709 12:18:54.956244 28281 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.958283 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:54.959036 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59186us
I20260709 12:18:54.936681 28425 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.960562 13262 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:54.962026 28537 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:18:54.963191 28425 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:54.886703 28446 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:54.882630 28520 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:54.912510 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97682.8us
I20260709 12:18:54.847296 28137 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.925251 28338 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:54.915285 28448 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:54.986752 12565 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 28. Preceding OpId from leader: term: 44 index: 28. (term mismatch)
I20260709 12:18:54.987628 12565 pending_rounds.cc:85] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 27
I20260709 12:18:54.988052 12565 pending_rounds.cc:107] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.28
I20260709 12:18:54.988379 12565 raft_consensus.cc:2889] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:54.919534 28502 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:54.923323 28532 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:54.924320 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:55.006502 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82282.6us
I20260709 12:18:54.931640 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.933060 13255 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:54.999256 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:54.898728 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.009554 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:54.941509 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:54.939641 13253 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:54.937069 28396 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.962724 28423 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.903003 13260 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 54.
I20260709 12:18:54.961314 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:54.890774 28319 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.885555 28443 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:54.918730 28055 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:54.992518 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6294) took 1259 ms. Trace:
W20260709 12:18:54.993821 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 86887us
I20260709 12:18:54.934002 28435 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.035435 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:54.995898 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.997539 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:54.963209 28537 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 50.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.035s
I20260709 12:18:54.978535 28489 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:54.929037 28538 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:54.999217 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:55.020591 28273 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:55.000643 28338 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 50.28, Last appended by leader: 28, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:54.986287 28520 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.014734 27340 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:55.017478 12571 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:55.029570 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.054421 27340 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:55.055084 12569 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:55.056679 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.031006 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:55.045516 28430 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.583s	user 0.014s	sys 0.000s
W20260709 12:18:55.033420 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72208.7us
W20260709 12:18:55.058175 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 116682us
I20260709 12:18:54.942715 28286 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.081s
I20260709 12:18:54.976414 28296 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.064244 12643 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:55.064911 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:55.046752 28055 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:55.045409 12568 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:55.000170 28448 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.036018 12564 rpcz_store.cc:276] 0709 12:18:53.733201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.168035 (+434834us) service_pool.cc:224] Handling call
0709 12:18:54.217387 (+ 49352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:54.398526 (+181139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:54.398562 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:54.399370 (+   808us) log.cc:844] Serialized 26 byte log entry
0709 12:18:54.697780 (+298410us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:54.697838 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.697951 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:54.991701 (+293750us) raft_consensus.cc:1685] finished
0709 12:18:54.992184 (+   483us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:54.992428 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":132580,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":5020,"thread_start_us":361890,"threads_started":2,"wal-append.queue_time_us":262448}
I20260709 12:18:55.076977 12646 raft_consensus.cc:1217] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Deduplicated request from leader. Original: 41.27->[43.28-43.28]   Dedup: 43.28->[]
I20260709 12:18:55.045222 28518 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:54.994484 28548 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:55.065598 12641 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.23-43.23]
I20260709 12:18:55.066319 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:55.074337 27340 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.067739 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:18:55.005686 28502 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.074652 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:55.075515 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.006027 28532 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:55.078133 12646 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.28-43.28]
I20260709 12:18:55.132182 12639 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:18:55.132380 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.053447 28538 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:55.081979 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:55.080197 12638 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
W20260709 12:18:55.081071 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81919us
I20260709 12:18:55.036896 28491 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.131258 27340 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:55.136292 28549 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:55.154204 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:55.062992 28273 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:55.141316 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.064567 28430 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:55.067296 28537 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.153115 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: true dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:55.074819 28055 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:55.158708 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 93869.9us
I20260709 12:18:55.172159 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:55.161392 12641 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 54: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:55.131335 28279 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:18:55.164418 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.174168 12639 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:55.175606 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.176167 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:55.130199 28518 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:55.076556 28296 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.184751 12643 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.28-43.28]
I20260709 12:18:55.165742 28517 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:18:55.137126 28548 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.178114 28055 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:55.175531 28458 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:55.188483 28521 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:18:55.205008 28448 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.210494 12572 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:18:55.225438 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:18:55.214251 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 139679us
I20260709 12:18:55.209769 12566 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:55.205219 28518 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:55.176990 28285 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.146665 28538 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.178573 28537 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:55.249246 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:55.273777 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48465.4us
I20260709 12:18:55.278956 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:55.235184 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:55.284102 12566 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
W20260709 12:18:55.302441 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 234736us
I20260709 12:18:55.212257 28517 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:55.304658 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:55.308032 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72913.3us
I20260709 12:18:55.309984 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:18:55.353888 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 134876us
I20260709 12:18:55.286094 27833 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:18:55.233675 28458 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:55.378765 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:55.379566 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.225487 28532 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 50.21, Last appended by leader: 21, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.314625 28517 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:55.322901 28281 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:18:55.279592 28448 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.361851 28319 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.315219 28396 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:55.389683 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85674.2us
I20260709 12:18:55.329573 28286 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.054s
W20260709 12:18:55.402006 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 123195us
I20260709 12:18:55.368983 28435 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.416780 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:55.416941 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:55.373006 27833 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:55.391074 28539 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:18:55.390599 28281 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:55.454703 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.314514 28273 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 41.29, Last appended by leader: 29, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.401492 28542 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:55.499600 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:55.561447 28538 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.553879 28507 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:55.586827 28507 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.574116 28502 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:55.588611 28458 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:55.426334 28499 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 1.064s	user 0.005s	sys 0.002s
I20260709 12:18:55.586670 28569 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:55.590801 28572 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:55.591482 28572 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:55.586839 28564 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:18:55.589375 28507 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:55.559785 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143091us
I20260709 12:18:55.558790 28548 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.558791 28537 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:55.576117 28554 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.371s	user 0.001s	sys 0.000s
W20260709 12:18:55.576478 28556 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
W20260709 12:18:55.576913 28553 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
W20260709 12:18:55.577948 28561 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:18:55.634788 28539 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:55.578217 28565 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:55.635999 28565 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:18:55.580549 28448 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.637926 28448 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:18:55.638056 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 138601us
I20260709 12:18:55.448660 28532 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:55.639526 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.427315 27833 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:55.427034 28520 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:55.496961 28281 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:55.597329 28286 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:55.565147 28296 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.620296 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:55.625346 28568 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.500837 28560 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:18:55.630235 28285 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.631946 28572 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:55.634038 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74823.2us
I20260709 12:18:55.592093 28569 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:18:55.632661 28518 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.635366 28055 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:55.619347 28516 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.656s	user 0.009s	sys 0.004s
I20260709 12:18:55.574977 28542 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:55.650840 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.652928 12645 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: []
W20260709 12:18:55.629539 28446 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.002s
I20260709 12:18:55.658763 28521 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election won for term 53
W20260709 12:18:55.668179 28184 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.716s	user 0.012s	sys 0.002s
I20260709 12:18:55.654429 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:55.706650 12571 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:55.705330 28446 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.676455 12637 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:55.673568 12639 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:55.710757 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.711596 12645 raft_consensus.cc:2368] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49.
I20260709 12:18:55.711470 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.712302 12639 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:55.712579 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:55.714210 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:55.715032 13253 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:55.713364 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.721819 28576 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.722581 28576 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.723940 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.729028 12639 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:55.733489 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:55.735817 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:55.736781 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2017) took 1212 ms. Trace:
I20260709 12:18:55.740305 28467 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.731961 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:55.658435 28568 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.745806 28568 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.671296 28516 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.747876 28516 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:55.680074 28521 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:55.748834 28521 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:55.754593 28517 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.757019 28517 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:55.636027 28539 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.758109 28184 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.675644 28542 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:55.760526 28184 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.761248 13254 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:55.737174 13260 rpcz_store.cc:276] 0709 12:18:54.524603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.528769 (+  4166us) service_pool.cc:224] Handling call
0709 12:18:55.736593 (+1207824us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:55.736685 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":809962880}
W20260709 12:18:55.697999 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.713366 12645 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:55.766114 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.766844 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:55.698678 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81671.5us
I20260709 12:18:55.768051 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:55.768807 12645 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 47 because replica has last-logged OpId of term: 45 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 24.
I20260709 12:18:55.771733 28571 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:18:55.706169 12569 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47.
W20260709 12:18:55.775828 28506 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.308s	user 0.002s	sys 0.002s
I20260709 12:18:55.779570 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.780938 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:55.725347 28518 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.726881 28055 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:55.742283 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5792) took 2337 ms. Trace:
I20260709 12:18:55.782802 12642 rpcz_store.cc:276] 0709 12:18:53.404480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.712457 (+307977us) service_pool.cc:224] Handling call
0709 12:18:53.732385 (+ 19928us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:54.374023 (+641638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.741963 (+1367940us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b540018c4d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:55.742211 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1473645312}
I20260709 12:18:55.671499 28560 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:55.765488 28446 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:55.785598 12642 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 47 index: 30. (index mismatch)
I20260709 12:18:55.786232 28516 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:55.793018 28567 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:18:55.786517 28560 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:55.794652 28570 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
W20260709 12:18:55.788761 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5789) took 2387 ms (client timeout 2141 ms). Trace:
W20260709 12:18:55.797334 12644 rpcz_store.cc:269] 0709 12:18:53.401805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:53.401971 (+   166us) service_pool.cc:224] Handling call
0709 12:18:55.788518 (+2386547us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:55.788675 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2638335872}
I20260709 12:18:55.793205 28516 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:55.799909 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:55.800284 28516 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.680418 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:55.801837 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.802243 12537 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:55.805428 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:55.805841 13257 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:55.807179 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:55.808697 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2015) took 1323 ms. Trace:
I20260709 12:18:55.809310 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:55.810184 13256 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 32. Preceding OpId from leader: term: 46 index: 33. (index mismatch)
I20260709 12:18:55.812021 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2000) took 1152 ms. Trace:
W20260709 12:18:55.812987 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.813400 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:55.781478 12564 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:55.813984 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:55.814978 28501 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:18:55.815876 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:55.887343 13260 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:55.816488 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.818996 28396 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:55.820188 28308 tablet_copy_client.cc:538] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:18:55.812376 13256 rpcz_store.cc:276] 0709 12:18:54.659905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.661024 (+  1119us) service_pool.cc:224] Handling call
0709 12:18:55.809672 (+1148648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.810727 (+  1055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.811817 (+  1090us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54005875d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:55.811960 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2640416896}
I20260709 12:18:55.820961 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:55.807554 28137 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:55.809113 13259 rpcz_store.cc:276] 0709 12:18:54.485523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.488725 (+  3202us) service_pool.cc:224] Handling call
0709 12:18:54.489052 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:54.535594 (+ 46542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.808402 (+1272808us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:55.808633 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75725184}
I20260709 12:18:55.823396 28468 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Stepping down as leader of term 54
I20260709 12:18:55.769490 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:55.824919 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2400) took 1406 ms (client timeout 1839 ms). Trace:
W20260709 12:18:55.826505 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.827028 28521 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:55.980437 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 159564us
W20260709 12:18:55.679345 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 310476us
I20260709 12:18:55.837337 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2003) took 1155 ms. Trace:
I20260709 12:18:55.981693 13262 rpcz_store.cc:276] 0709 12:18:54.682095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.682357 (+   262us) service_pool.cc:224] Handling call
0709 12:18:54.682707 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:54.960965 (+278258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:55.837084 (+876119us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:55.837265 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":638530944}
I20260709 12:18:55.844199 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.850366 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:55.852929 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53089.7us
I20260709 12:18:55.824946 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:55.854598 28346 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:55.985378 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:55.853245 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51079us
I20260709 12:18:55.773290 28571 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 49.35, Next index: 36, Last known committed idx: 33, Time since last communication: 0.041s
W20260709 12:18:55.860498 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.862663 12639 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:55.864411 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.869241 12571 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:55.888983 12568 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:55.834507 28281 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.982024 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:55.848479 28423 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:55.987571 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:55.990909 12492 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 27. Preceding OpId from leader: term: 41 index: 27. (term mismatch)
W20260709 12:18:55.980119 12489 rpcz_store.cc:269] 0709 12:18:54.418522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.418686 (+   164us) service_pool.cc:224] Handling call
0709 12:18:55.824682 (+1405996us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:55.824852 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45314432}
I20260709 12:18:55.996335 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:56.001575 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.003803 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:55.851063 28573 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 45.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.042s
I20260709 12:18:56.008620 12492 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 26
I20260709 12:18:55.893582 28501 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.009434 12497 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 54. Current term is 55. Ops: []
I20260709 12:18:56.016898 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:56.016999 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:56.009687 12461 connection.cc:256] RPC call timeout handler was delayed by -1.1407133339998836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
I20260709 12:18:56.011070 12492 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 40.27
I20260709 12:18:56.010663 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.019239 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.019515 12492 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:56.019654 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:56.020165 12572 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:55.893049 27765 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:18:56.027506 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:56.030097 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:56.020836 12571 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:56.022498 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.023974 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:56.022591 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.037506 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.036355 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:56.052496 12572 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:56.006683 28493 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:18:55.994195 28346 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:56.055493 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:18:55.979319 28468 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:56.060992 28493 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.062273 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:56.063501 12491 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:56.064710 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:56.009764 28569 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:56.021790 28501 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:56.066390 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2001) took 1406 ms. Trace:
I20260709 12:18:56.066931 13254 rpcz_store.cc:276] 0709 12:18:54.660344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.661046 (+   702us) service_pool.cc:224] Handling call
0709 12:18:54.951588 (+290542us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:18:54.951779 (+   191us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.762291 (+810512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.066111 (+303820us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b540056bdd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:56.066318 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1864255360}
I20260709 12:18:56.016462 28338 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:18:56.072314 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4575) took 1799 ms (client timeout 1906 ms). Trace:
I20260709 12:18:56.062525 12496 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:18:56.066059 28569 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.065585 12491 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 54: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:56.070737 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.078723 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
W20260709 12:18:56.072871 12494 rpcz_store.cc:269] 0709 12:18:54.273203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.273710 (+   507us) service_pool.cc:224] Handling call
0709 12:18:56.072057 (+1798347us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b5400191254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:56.072237 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161405184}
I20260709 12:18:56.073890 12490 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 52. Current term is 53. Ops: []
I20260709 12:18:56.083942 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:56.021240 28137 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:56.088047 13260 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:56.094849 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:56.051198 28124 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:56.109098 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:56.110702 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:56.026074 28489 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:56.027069 27765 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:55.980178 28396 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:55.993525 27833 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.099855 12572 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:18:56.051229 28446 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20260709 12:18:56.143431 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:56.106750 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:56.054948 28482 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:56.101239 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:18:56.287822 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 186654us
I20260709 12:18:56.121470 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:56.068274 28346 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:56.075450 28493 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:56.134310 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:56.292928 28521 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:56.304708 28569 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:56.282537 28125 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:18:56.312258 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4639) took 1244 ms. Trace:
I20260709 12:18:56.312887 12646 rpcz_store.cc:276] 0709 12:18:55.067507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.146152 (+ 78645us) service_pool.cc:224] Handling call
0709 12:18:56.311968 (+1165816us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:56.312152 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1526381952}
I20260709 12:18:56.314208 28573 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:18:56.314926 28573 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:56.315878 28573 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.105134 28517 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.082s
I20260709 12:18:56.163882 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:56.132371 28489 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:56.278684 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:56.339537 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.340133 12568 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 52. Current term is 53. Ops: []
I20260709 12:18:56.108644 28124 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:56.096151 28502 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.078s
I20260709 12:18:56.097766 28279 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 50.36, Next index: 37, Last known committed idx: 36, Time since last communication: 0.090s
I20260709 12:18:56.114737 28137 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.285408 12492 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:56.341181 28124 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:56.344666 12492 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:56.346362 12492 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:56.347643 12492 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:56.348060 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5972) took 2637 ms. Trace:
I20260709 12:18:56.315882 28559 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20260709 12:18:56.139086 27765 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:56.138451 28281 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:56.320479 12571 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:18:56.348562 12567 rpcz_store.cc:276] 0709 12:18:53.710196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.148901 (+438705us) service_pool.cc:224] Handling call
0709 12:18:54.190394 (+ 41493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:54.402004 (+211610us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:54.402058 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:55.756106 (+1354048us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b2c00010828. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:18:55.756671 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.277592 (+520921us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:56.277660 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.277747 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.347684 (+ 69937us) raft_consensus.cc:1685] finished
0709 12:18:56.347711 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.347987 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":138880,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":14347,"spinlock_wait_cycles":9974656,"thread_start_us":646044,"threads_started":2,"wal-append.queue_time_us":589100}
I20260709 12:18:56.350162 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5983) took 1309 ms. Trace:
I20260709 12:18:56.350558 12571 rpcz_store.cc:276] 0709 12:18:55.041114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.317172 (+1276058us) service_pool.cc:224] Handling call
0709 12:18:56.320245 (+  3073us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.349907 (+ 29662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.350102 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.352219 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5987) took 1310 ms. Trace:
I20260709 12:18:56.328575 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:56.352576 12571 rpcz_store.cc:276] 0709 12:18:55.041952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.351330 (+1309378us) service_pool.cc:224] Handling call
0709 12:18:56.351681 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.351875 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.351914 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:56.351956 (+    42us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:18:56.351999 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.352058 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.352162 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.355738 12572 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:18:56.356405 12571 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:56.356920 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5985) took 1315 ms. Trace:
I20260709 12:18:56.358196 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5988) took 1316 ms. Trace:
W20260709 12:18:56.331853 28542 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.571s	user 0.013s	sys 0.006s
I20260709 12:18:56.358196 12568 rpcz_store.cc:276] 0709 12:18:55.041464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.339508 (+1298044us) service_pool.cc:224] Handling call
0709 12:18:56.339890 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.356543 (+ 16653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.356840 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.374409 12564 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:18:56.358741 12571 rpcz_store.cc:276] 0709 12:18:55.042223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.355730 (+1313507us) service_pool.cc:224] Handling call
0709 12:18:56.356159 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.357828 (+  1669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.358122 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.321674 28482 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.388391 12494 pending_rounds.cc:85] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 22
I20260709 12:18:56.389781 12572 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:56.394326 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5991) took 1013 ms. Trace:
I20260709 12:18:56.394863 12572 rpcz_store.cc:276] 0709 12:18:55.380567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.388984 (+1008417us) service_pool.cc:224] Handling call
0709 12:18:56.389498 (+   514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.394005 (+  4507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.394261 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.390267 12564 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
W20260709 12:18:56.373217 28549 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.023s	sys 0.029s
I20260709 12:18:56.399907 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6310) took 1001 ms. Trace:
I20260709 12:18:56.400538 12564 rpcz_store.cc:276] 0709 12:18:55.398644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.389422 (+990778us) service_pool.cc:224] Handling call
0709 12:18:56.390011 (+   589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.399393 (+  9382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.399821 (+   428us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:18:56.390792 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.017s	sys 0.007s
W20260709 12:18:56.388298 28273 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.019s	sys 0.039s
W20260709 12:18:56.401787 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.018s	sys 0.007s
W20260709 12:18:56.404723 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.004s	sys 0.024s
I20260709 12:18:56.393807 12494 pending_rounds.cc:107] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 39.23
W20260709 12:18:56.408998 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.005s	sys 0.024s
I20260709 12:18:56.409551 12494 raft_consensus.cc:2917] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Aborting config change with OpId 39.23: Aborted: Op aborted by new leader
I20260709 12:18:56.396245 28482 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:18:56.416958 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 419847us
W20260709 12:18:56.405570 28273 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.019s	sys 0.039s
I20260709 12:18:56.388609 28542 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:56.403005 28549 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.024s	sys 0.029s
I20260709 12:18:56.434399 12494 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:56.442101 28542 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:56.357949 28468 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 54.35, Last appended by leader: 35, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.425873 12564 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 47 index: 30. (index mismatch)
I20260709 12:18:56.427145 12563 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:56.487308 12563 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:18:56.409782 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 552567us
I20260709 12:18:56.491297 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6316) took 1071 ms. Trace:
I20260709 12:18:56.491925 12564 rpcz_store.cc:276] 0709 12:18:55.420260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.424894 (+1004634us) service_pool.cc:224] Handling call
0709 12:18:56.425370 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.479392 (+ 54022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.491216 (+ 11824us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.499831 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60516 (request call id 7) took 11946 ms. Trace:
I20260709 12:18:56.514878 12494 rpcz_store.cc:276] 0709 12:18:44.540518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:45.528885 (+988367us) service_pool.cc:224] Handling call
0709 12:18:46.528696 (+999811us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b5800725fd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57ee70a 00007ffbf57ef76a 00007ffbf57ee91b 00007ffbf57fa03c 00007ffbf57f96b3 00007ffbfa7b9d29 00007ffbf3d381d4 00007ffbf3d38162
0709 12:18:46.592458 (+ 63762us) log.cc:844] Serialized 506 byte log entry
0709 12:18:46.749143 (+156685us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b18002e5ca0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf574cb0b 00007ffbf57d1b63 00007ffbf57f9752 00007ffbfa7b9d29
0709 12:18:47.177722 (+428579us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b18002e5ca0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706522 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:18:56.453986 (+9276264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27278,"raft.queue_time_us":127486,"raft.run_cpu_time_us":2751,"raft.run_wall_time_us":85114,"spinlock_wait_cycles":2169134976,"thread_start_us":68437,"threads_started":1,"wal-append.queue_time_us":120421,"wal-append.run_cpu_time_us":2684,"wal-append.run_wall_time_us":1027222}
W20260709 12:18:56.520147 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 442912us
I20260709 12:18:56.530270 27833 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:56.530822 28509 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.830s	user 0.003s	sys 0.003s
W20260709 12:18:56.530305 28589 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:18:56.531257 28281 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 53.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:18:56.532838 28281 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:18:56.533285 28281 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:56.534643 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5970) took 2858 ms. Trace:
I20260709 12:18:56.500588 28468 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:18:56.556209 12570 rpcz_store.cc:276] 0709 12:18:53.675947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.038109 (+362162us) service_pool.cc:224] Handling call
0709 12:18:54.038517 (+   408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:54.039030 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:54.039060 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:54.039736 (+   676us) log.cc:844] Serialized 52 byte log entry
0709 12:18:54.391256 (+351520us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:18:54.430287 (+ 39031us) log.cc:844] Serialized 14 byte log entry
0709 12:18:54.472811 (+ 42524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:54.473124 (+   313us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:56.487639 (+2014515us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:18:56.534243 (+ 46604us) raft_consensus.cc:1685] finished
0709 12:18:56.534367 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.534569 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8569,"prepare.queue_time_us":31810,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":25633,"raft.queue_time_us":29860,"raft.run_cpu_time_us":440,"raft.run_wall_time_us":15215,"spinlock_wait_cycles":12152704,"thread_start_us":350958,"threads_started":1,"wal-append.queue_time_us":613448}
W20260709 12:18:56.539937 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:56.560053 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.515388 12563 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 45.32, Last appended by leader: 32, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.611241 13260 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 50 index: 28. Preceding OpId from leader: term: 51 index: 29. (index mismatch)
W20260709 12:18:56.613917 13261 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.013s	sys 0.008s
W20260709 12:18:56.614619 13261 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.013s	sys 0.008s
W20260709 12:18:56.597859 28003 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:18:56.602123 28423 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.591s	user 0.015s	sys 0.000s
W20260709 12:18:56.625670 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.007s	sys 0.015s
W20260709 12:18:56.634222 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.602428 28396 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:56.635871 13253 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:18:56.641741 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.007s	sys 0.015s
W20260709 12:18:56.645684 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 610728us
W20260709 12:18:56.664028 12460 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.019s	sys 0.042s
W20260709 12:18:56.711284 12460 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.019s	sys 0.042s
W20260709 12:18:56.712245 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 603224us
I20260709 12:18:55.207769 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:18:56.638691 28489 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:56.721009 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 733587us
I20260709 12:18:56.718318 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6326) took 1060 ms. Trace:
I20260709 12:18:56.718792 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:56.720245 28516 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:18:56.720896 12536 connection.cc:256] RPC call timeout handler was delayed by -1.67640233599991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3131s for a response
I20260709 12:18:56.743413 12570 rpcz_store.cc:276] 0709 12:18:55.657883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.632764 (+974881us) service_pool.cc:224] Handling call
0709 12:18:56.713212 (+ 80448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.713449 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.713486 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:56.713547 (+    61us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:56.713593 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.717924 (+  4331us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.718124 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.744011 12460 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:18:56.792107 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48214.5us
I20260709 12:18:56.620625 28346 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.624573 28493 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.643553 28430 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:56.789606 28458 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.025s	sys 0.014s
I20260709 12:18:56.708112 27833 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.803771 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:56.807408 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 825372us
W20260709 12:18:56.742996 28501 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.676s	user 0.013s	sys 0.001s
I20260709 12:18:56.641333 28573 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.159s
I20260709 12:18:56.811805 12610 leader_election.cc:400] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43.
I20260709 12:18:56.743517 28559 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:56.807112 28458 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.026s	sys 0.014s
I20260709 12:18:56.857213 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:56.866711 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 336076us
I20260709 12:18:56.867015 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:56.845753 12643 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.023s	sys 0.059s
I20260709 12:18:56.794803 28502 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:56.863509 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:56.867926 12638 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:56.858415 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:56.869938 12643 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.027s	sys 0.059s
W20260709 12:18:56.810395 28541 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.033s	sys 0.018s
I20260709 12:18:56.804553 28601 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:56.803153 28489 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:18:56.882441 12644 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
W20260709 12:18:56.883211 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:56.898511 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:56.898499 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:56.840588 28537 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:56.901645 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:56.910925 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:56.806957 28396 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 24, Time since last communication: 0.077s
W20260709 12:18:56.875444 28575 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.814s	user 0.006s	sys 0.017s
I20260709 12:18:56.711911 28281 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 53.26, Last appended by leader: 26, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.889101 28559 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.918689 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:56.882722 28124 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.000s
W20260709 12:18:56.887013 28541 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.035s	sys 0.019s
I20260709 12:18:56.931394 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:56.805302 28516 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.810400 28430 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:56.949771 12640 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:56.838078 27833 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.950534 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:56.975986 12570 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:56.958031 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:56.960232 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5977) took 2561 ms. Trace:
I20260709 12:18:56.960109 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:56.977732 12645 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:56.977459 12565 rpcz_store.cc:276] 0709 12:18:54.398957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.495925 (+ 96968us) service_pool.cc:224] Handling call
0709 12:18:54.496272 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.016516 (+520244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.959956 (+1943440us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:56.960168 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125798144}
I20260709 12:18:56.978744 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:56.961882 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6327) took 1298 ms. Trace:
I20260709 12:18:56.979874 12571 rpcz_store.cc:276] 0709 12:18:55.663721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.960618 (+1296897us) service_pool.cc:224] Handling call
0709 12:18:56.961118 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.961381 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.961419 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:56.961481 (+    62us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:56.961526 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.961603 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.961794 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.982278 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6329) took 1156 ms. Trace:
I20260709 12:18:56.982717 12571 rpcz_store.cc:276] 0709 12:18:55.825810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.981142 (+1155332us) service_pool.cc:224] Handling call
0709 12:18:56.981569 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.981821 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.981860 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:56.981920 (+    60us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:18:56.981968 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.982045 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.982208 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.984913 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6330) took 1158 ms. Trace:
I20260709 12:18:56.938395 28560 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:56.987593 12644 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 42 index: 23. (index mismatch)
I20260709 12:18:56.869302 28605 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.107s
I20260709 12:18:56.963316 12638 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
W20260709 12:18:56.939011 28575 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.006s	sys 0.018s
I20260709 12:18:56.912993 28537 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:56.861042 28501 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:56.966432 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.006253 12570 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:57.006387 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.007103 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5993) took 1222 ms. Trace:
I20260709 12:18:56.985344 12571 rpcz_store.cc:276] 0709 12:18:55.826148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.983851 (+1157703us) service_pool.cc:224] Handling call
0709 12:18:56.984293 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.984528 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.984567 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:56.984624 (+    57us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:18:56.984671 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.984732 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:56.984844 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.997995 12644 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:57.007587 12570 rpcz_store.cc:276] 0709 12:18:55.785048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.976844 (+1191796us) service_pool.cc:224] Handling call
0709 12:18:57.006031 (+ 29187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.006808 (+   777us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.007036 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:56.999053 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:56.999893 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143229us
I20260709 12:18:56.944519 28580 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:18:56.977010 28526 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.116s	user 0.009s	sys 0.018s
I20260709 12:18:57.009339 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:57.000415 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.012557 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:56.986106 28532 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.004298 28541 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:18:57.040540 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:57.028810 12645 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:57.026317 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.043841 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.043776 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:18:57.046586 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.041550 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:57.027325 28526 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.174s	user 0.010s	sys 0.019s
I20260709 12:18:57.003571 28281 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:57.047251 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.049525 12644 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:18:57.050830 12645 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:56.911587 28601 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.056600 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.058147 28538 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:18:57.062704 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2242) took 1197 ms. Trace:
W20260709 12:18:57.065805 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.068769 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.006434 28537 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:57.054206 28571 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:56.949224 28124 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.075484 28491 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.048096 12492 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 49
I20260709 12:18:57.006387 28549 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.075946 12570 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:18:57.027184 28346 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:57.057150 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 263207us
I20260709 12:18:57.080855 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:57.081135 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.048604 28568 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:18:57.002705 28560 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.054502 28575 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.075451 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.011402 28539 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.072s
I20260709 12:18:57.084489 28055 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.117429 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.146469 12488 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:18:57.118372 12489 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:57.082024 28573 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:18:57.122339 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 263754us
I20260709 12:18:57.055563 28532 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.126039 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.154987 12640 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:57.127755 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47021.3us
I20260709 12:18:57.129331 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.095310 13261 rpcz_store.cc:276] 0709 12:18:55.865150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.009378 (+144228us) service_pool.cc:224] Handling call
0709 12:18:56.051000 (+ 41622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.051349 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 33
0709 12:18:56.625287 (+573938us) log.cc:844] Serialized 14 byte log entry
0709 12:18:57.055348 (+430061us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:57.055702 (+   354us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:18:57.055938 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.062401 (+  6463us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.062611 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":459,"prepare.queue_time_us":447478,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":2101,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":9718,"thread_start_us":967443,"threads_started":2}
I20260709 12:18:57.138892 28569 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:57.151734 12492 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:18:57.055408 28580 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.155927 12645 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:57.160820 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.162629 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.158988 28055 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.188262 28055 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.188578 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4589) took 1114 ms. Trace:
I20260709 12:18:57.189752 28568 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:57.191393 28560 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.191107 28568 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.194247 12644 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 39.
I20260709 12:18:57.116000 28571 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.166364 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.130326 28493 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:57.175174 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:57.199719 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.174996 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.198596 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:18:57.201763 28587 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.470s	user 0.000s	sys 0.001s
I20260709 12:18:57.202180 28493 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:57.203078 28493 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:57.203915 28586 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.382s	user 0.001s	sys 0.001s
I20260709 12:18:57.205545 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5799) took 1566 ms. Trace:
I20260709 12:18:57.207015 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5800) took 1567 ms. Trace:
I20260709 12:18:57.206552 28493 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.207569 12641 rpcz_store.cc:276] 0709 12:18:55.639803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.645353 (+  5550us) service_pool.cc:224] Handling call
0709 12:18:55.645897 (+   544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:55.648987 (+  3090us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:55.649024 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:55.656414 (+  7390us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.113440 (+457026us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:18:56.113513 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.113668 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.206730 (+1093062us) raft_consensus.cc:1685] finished
0709 12:18:57.206760 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.206950 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2501,"raft.queue_time_us":4227,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":71,"thread_start_us":403377,"threads_started":1,"wal-append.queue_time_us":773286}
I20260709 12:18:57.208947 28493 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:57.209733 28563 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:18:57.205725 28596 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.276s	user 0.000s	sys 0.001s
I20260709 12:18:57.210296 28563 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.210211 28493 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:18:57.211373 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5812) took 1515 ms. Trace:
I20260709 12:18:57.211586 28557 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:57.212267 28591 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.443s	user 0.002s	sys 0.000s
W20260709 12:18:57.212634 28497 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 2.730s	user 0.006s	sys 0.025s
I20260709 12:18:57.213366 12492 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:57.170269 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6115) took 2430 ms. Trace:
W20260709 12:18:57.214987 28614 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:18:57.211879 12642 rpcz_store.cc:276] 0709 12:18:55.695871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.791899 (+ 96028us) service_pool.cc:224] Handling call
0709 12:18:55.792266 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:55.821906 (+ 29640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:55.821940 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:55.822729 (+   789us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.413566 (+590837us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:18:56.413676 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.413776 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.211144 (+797368us) raft_consensus.cc:1685] finished
0709 12:18:57.211164 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.211311 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29047,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":582647,"threads_started":1,"wal-append.queue_time_us":795065}
W20260709 12:18:57.216395 28612 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.279s	user 0.002s	sys 0.000s
I20260709 12:18:57.214295 12493 rpcz_store.cc:276] 0709 12:18:54.740177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.740337 (+   160us) service_pool.cc:224] Handling call
0709 12:18:54.740773 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:54.765179 (+ 24406us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.158830 (+2393651us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b5400366d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:57.170165 (+ 11335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53860736}
I20260709 12:18:57.218379 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.198920 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.219185 12493 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:57.220042 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.220400 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.221123 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:57.214881 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:57.212129 28557 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.227511 28557 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.206082 12643 rpcz_store.cc:276] 0709 12:18:55.639438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.640415 (+   977us) service_pool.cc:224] Handling call
0709 12:18:55.640802 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:55.641236 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:55.641267 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:55.641825 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.090489 (+448664us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:18:56.889396 (+798907us) log.cc:844] Serialized 14 byte log entry
0709 12:18:57.036074 (+146678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.036160 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.205320 (+169160us) raft_consensus.cc:1685] finished
0709 12:18:57.205339 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.205489 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":146345,"prepare.queue_time_us":558763,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":168886,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":240,"thread_start_us":1205205,"threads_started":2,"wal-append.queue_time_us":772245}
I20260709 12:18:57.228924 12643 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:18:57.211155 28563 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:57.234102 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.236075 12641 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
I20260709 12:18:57.121654 28549 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:57.243065 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 44438.6us
I20260709 12:18:57.255142 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:18:57.189065 12496 rpcz_store.cc:276] 0709 12:18:56.074671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.335856 (+261185us) service_pool.cc:224] Handling call
0709 12:18:56.336415 (+   559us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:18:56.353818 (+ 17403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.353850 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:18:56.370800 (+ 16950us) log.cc:844] Serialized 52 byte log entry
0709 12:18:56.856810 (+486010us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:18:56.882547 (+ 25737us) log.cc:844] Serialized 14 byte log entry
0709 12:18:57.144793 (+262246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.144904 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.188277 (+ 43373us) raft_consensus.cc:1685] finished
0709 12:18:57.188304 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.188500 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2641,"prepare.queue_time_us":40793,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"raft.queue_time_us":77686,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":32210,"thread_start_us":699828,"threads_started":2,"wal-append.queue_time_us":784100}
I20260709 12:18:57.256807 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:57.174206 28538 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:57.200994 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:57.143536 28575 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.280571 12492 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:57.171018 28346 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.086s
I20260709 12:18:57.252624 28396 consensus_queue.cc:1243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 44.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:18:57.198553 28124 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.171237 28573 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:57.303460 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 297148us
W20260709 12:18:57.304224 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 103297us
I20260709 12:18:57.323086 28571 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.241080 28569 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.052s
I20260709 12:18:57.256847 28273 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.070s
I20260709 12:18:57.324522 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 5986) took 2282 ms. Trace:
I20260709 12:18:57.325950 28575 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.327602 28601 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:57.331768 28581 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.050s	user 0.000s	sys 0.002s
I20260709 12:18:57.340565 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:18:57.347666 12567 rpcz_store.cc:276] 0709 12:18:55.041687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.350677 (+1308990us) service_pool.cc:224] Handling call
0709 12:18:56.351124 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.351398 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.351428 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.351982 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:18:56.710743 (+358761us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:56.710798 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:56.710901 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:57.324156 (+613255us) raft_consensus.cc:1685] finished
0709 12:18:57.324193 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.324427 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":358248,"threads_started":1,"wal-append.queue_time_us":915862}
I20260709 12:18:57.242529 28526 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.104s
I20260709 12:18:57.267598 28629 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:18:57.303067 28559 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.361557 28024 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:57.375181 28620 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:18:57.342326 28627 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:57.364089 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.278650 28491 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.081s
I20260709 12:18:57.341892 28516 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.354210 27833 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.279812 28538 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:57.367878 27765 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.013s	user 0.017s	sys 0.000s
I20260709 12:18:57.406301 28024 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:57.410151 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46156.2us
I20260709 12:18:57.412370 28557 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:18:57.418009 28576 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:57.419102 28573 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.425707 28491 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:18:57.430353 28532 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.430888 28467 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:57.432294 28549 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.386299 28055 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:57.433565 28580 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:57.435169 28611 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
W20260709 12:18:57.442617 28613 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.284s	user 0.001s	sys 0.000s
W20260709 12:18:57.443537 28628 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260709 12:18:57.366183 28501 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.355201 28630 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
W20260709 12:18:57.448038 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s	user 0.018s	sys 0.034s
I20260709 12:18:57.361183 28571 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:57.405092 28627 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:57.452586 27765 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:57.453538 28482 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.019s	user 0.005s	sys 0.010s
I20260709 12:18:57.456883 28308 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:18:57.452456 28557 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.452193 28396 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:18:57.481348 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s	user 0.044s	sys 0.076s
W20260709 12:18:57.536859 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.044s	sys 0.077s
W20260709 12:18:57.499662 28489 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.016s	sys 0.023s
I20260709 12:18:57.466053 28055 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:18:57.501253 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.020s	sys 0.043s
W20260709 12:18:57.542649 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 5990) took 2162 ms (client timeout 1542 ms). Trace:
W20260709 12:18:57.543111 12569 rpcz_store.cc:269] 0709 12:18:55.380240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.639202 (+258962us) service_pool.cc:224] Handling call
0709 12:18:57.540595 (+1901393us) spinlock_profiling.cc:243] Waited 41.1 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:57.542548 (+  1953us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":94490496}
I20260709 12:18:57.546026 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6322) took 1941 ms. Trace:
I20260709 12:18:57.546648 12564 rpcz_store.cc:276] 0709 12:18:55.604976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.493679 (+888703us) service_pool.cc:224] Handling call
0709 12:18:56.494159 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.494457 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:18:56.559074 (+ 64617us) log.cc:844] Serialized 14 byte log entry
0709 12:18:57.545484 (+986410us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:57.545574 (+    90us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:18:57.545680 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.545786 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.545965 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3933,"prepare.queue_time_us":35119,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"thread_start_us":981902,"threads_started":1}
I20260709 12:18:57.553097 28137 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.455686 28560 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.25, Next index: 26, Last known committed idx: 23, Time since last communication: 0.024s
W20260709 12:18:57.471645 28468 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.869s	user 0.010s	sys 0.005s
I20260709 12:18:57.537815 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6323) took 1915 ms. Trace:
I20260709 12:18:57.536511 28557 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:57.539038 12566 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:18:57.556331 12563 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:18:57.558815 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 376086us
I20260709 12:18:57.509321 28580 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.049s
I20260709 12:18:57.479745 28576 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 52.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.057s
W20260709 12:18:57.539584 28489 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.017s	sys 0.023s
W20260709 12:18:57.565099 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 310020us
I20260709 12:18:57.595490 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:57.596029 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:57.506742 12492 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:57.568912 12568 rpcz_store.cc:276] 0709 12:18:55.622098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.542187 (+920089us) service_pool.cc:224] Handling call
0709 12:18:56.542584 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.542839 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:56.543212 (+   373us) log.cc:844] Serialized 14 byte log entry
0709 12:18:57.537323 (+994111us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:57.537406 (+    83us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:57.537470 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.537554 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:57.537744 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2103,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":6967,"raft.queue_time_us":1452,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":16666,"thread_start_us":993718,"threads_started":1}
I20260709 12:18:57.405552 28632 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:18:57.482995 28563 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:57.562407 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 328360us
I20260709 12:18:57.542265 28055 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:57.602270 12461 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:57.603966 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.604926 28137 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:57.605327 28137 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.606006 28137 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:18:57.606364 28650 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:57.607280 28632 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
W20260709 12:18:57.564563 28502 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.005s	sys 0.008s
W20260709 12:18:57.592783 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 89570us
I20260709 12:18:57.532511 28482 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.610224 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.596871 28576 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:18:57.611984 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:18:57.608574 28502 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.006s	sys 0.009s
I20260709 12:18:57.605304 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:57.647620 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:57.575374 28526 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.594966 28636 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:57.609303 28489 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:57.613320 28576 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:57.747546 12491 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:57.718981 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.601497 28563 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:57.633867 28630 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:18:57.753834 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148588us
W20260709 12:18:57.751619 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104109us
I20260709 12:18:57.634647 28468 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:18:57.779587 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.633824 28502 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:57.816570 13258 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
W20260709 12:18:57.752578 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105069us
I20260709 12:18:57.593852 28541 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.154s
I20260709 12:18:57.610015 28184 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.141s
I20260709 12:18:57.763432 28644 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:18:57.764309 28458 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:18:57.836879 13258 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:57.779240 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.839190 12639 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:18:57.788127 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.779486 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.841188 12644 raft_consensus.cc:2368] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46.
I20260709 12:18:57.783437 28630 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:57.791551 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.747830 28526 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.747406 28636 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:18:57.798519 28615 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.604s	user 0.011s	sys 0.000s
I20260709 12:18:57.858129 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:57.802591 28024 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:57.764070 28627 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 46.24, Last appended by leader: 24, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.810282 28542 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:18:57.812438 28482 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:57.814940 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2240) took 2081 ms (client timeout 2166 ms). Trace:
I20260709 12:18:57.883757 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:57.812472 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.884940 12646 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 47
I20260709 12:18:57.761395 12563 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 32. Preceding OpId from leader: term: 46 index: 33. (index mismatch)
W20260709 12:18:57.856698 28537 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.737s	user 0.012s	sys 0.002s
W20260709 12:18:57.883883 13253 rpcz_store.cc:269] 0709 12:18:55.733064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.733222 (+   158us) service_pool.cc:224] Handling call
0709 12:18:57.813554 (+2080332us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54005c4c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:57.814865 (+  1311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2073231744}
I20260709 12:18:57.785038 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:18:57.889756 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 104874us
I20260709 12:18:57.872845 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.821391 13218 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:57.830175 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:57.892706 12639 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:18:57.834808 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:57.765303 28634 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.844111 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:57.844960 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.895339 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:18:57.846825 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:57.896512 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75290.1us
I20260709 12:18:57.774961 28538 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:57.859709 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:57.865203 12611 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 2.166s (SENT) [suppressed 16 similar messages]
I20260709 12:18:57.825888 28644 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.874187 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:57.886211 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6312) took 2483 ms. Trace:
I20260709 12:18:57.901113 12643 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:18:57.906761 12563 rpcz_store.cc:276] 0709 12:18:55.402953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.404215 (+1001262us) service_pool.cc:224] Handling call
0709 12:18:56.426955 (+ 22740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.555635 (+1128680us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b58009956d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf574f0aa 00007ffbf57d1447 00007ffbf57e9fe7 00007ffbf57eb7de 00007ffbf57f5b49 00007ffbf57f6416 00007ffbf57f2d8f 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4
0709 12:18:57.885930 (+330295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:57.886158 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2154618624}
I20260709 12:18:57.908180 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.644656 28650 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:18:57.832108 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:57.893460 12571 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:57.892323 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.846110 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.895381 12642 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:57.850327 28630 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:57.905165 13257 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 38 index: 28. Preceding OpId from leader: term: 41 index: 29. (index mismatch)
I20260709 12:18:57.903888 28024 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:18:57.913728 28644 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:18:57.922444 28645 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:18:57.913777 28489 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 41.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.856618 28468 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.050s
I20260709 12:18:57.929746 28468 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:18:57.930125 28468 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:57.930958 28468 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 46.30, Last appended by leader: 30, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.932245 28468 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:57.935088 28526 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:57.917032 12644 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:57.938807 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.917443 12643 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:18:57.941161 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.917882 28502 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.943439 28636 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.944998 28636 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:57.899680 28569 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:57.946070 28569 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:57.946982 28569 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 50.21, Last appended by leader: 21, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.903738 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.900678 28516 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.049s
W20260709 12:18:57.950138 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46475.4us
I20260709 12:18:57.950460 28538 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:18:57.918870 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.951175 28538 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.951890 28538 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:57.894290 27833 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.059s
I20260709 12:18:57.953465 28646 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:18:57.953933 28646 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:57.954301 28646 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:57.954149 27833 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:18:57.882104 28576 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 52.23, Last appended by leader: 23, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.957150 28576 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:18:57.961998 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:57.962779 12567 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:18:57.904626 28634 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:57.964977 28634 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:57.900310 28573 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.071s
I20260709 12:18:57.966375 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.967582 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.968685 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:57.849749 28580 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.053s
I20260709 12:18:57.875010 28608 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:18:57.952747 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:57.980322 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:18:57.990370 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:57.974893 28281 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.838s	user 0.011s	sys 0.011s
I20260709 12:18:57.994141 12643 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:57.992148 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:58.007489 12645 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45. Re-sending same reply.
I20260709 12:18:57.982977 12571 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:57.979279 28520 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:18:57.977970 28627 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:58.010438 12642 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:18:58.021729 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:58.016609 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:57.919679 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:58.028410 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:57.994481 28601 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:18:57.895779 28537 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:57.981312 28542 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:57.911764 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:58.019536 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4588) took 1959 ms. Trace:
I20260709 12:18:57.994035 28346 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:58.043987 12494 rpcz_store.cc:276] 0709 12:18:56.060436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.291646 (+231210us) service_pool.cc:224] Handling call
0709 12:18:56.292161 (+   515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:56.637098 (+344937us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:56.637152 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:56.641909 (+  4757us) log.cc:844] Serialized 26 byte log entry
0709 12:18:57.014498 (+372589us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:57.014585 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.019106 (+1004521us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x7b540020c1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:58.019161 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.019217 (+    56us) raft_consensus.cc:1685] finished
0709 12:18:58.019230 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.019459 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220621184,"thread_start_us":215570,"threads_started":1,"wal-append.queue_time_us":476638}
I20260709 12:18:58.013124 28520 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.031862 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4628) took 2955 ms. Trace:
I20260709 12:18:58.053897 12637 rpcz_store.cc:276] 0709 12:18:55.062915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.063059 (+   144us) service_pool.cc:224] Handling call
0709 12:18:55.082150 (+ 19091us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:55.710018 (+627868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.017868 (+2307850us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:58.018101 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1355031424}
I20260709 12:18:58.055569 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:58.032509 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2399) took 3614 ms. Trace:
I20260709 12:18:58.057953 12490 rpcz_store.cc:276] 0709 12:18:54.417900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:54.418092 (+   192us) service_pool.cc:224] Handling call
0709 12:18:56.073682 (+1655590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:56.081151 (+  7469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.032276 (+1951125us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400191250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:58.032446 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3770541312}
I20260709 12:18:58.060684 12496 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:58.061149 12490 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 29. Preceding OpId from leader: term: 51 index: 29. (term mismatch)
I20260709 12:18:58.036193 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:58.062079 12644 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:18:58.063294 12645 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:18:58.067540 28346 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:58.068821 12496 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 43 index: 23. (index mismatch)
I20260709 12:18:58.069693 28055 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:58.038089 28601 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:58.070837 28601 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:58.071789 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:58.072525 28055 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:58.073793 28520 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:18:58.048095 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:58.073129 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:58.045076 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4648) took 1158 ms. Trace:
I20260709 12:18:58.052234 28542 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:58.010641 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:58.076689 12646 rpcz_store.cc:276] 0709 12:18:56.886858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.912325 (+ 25467us) service_pool.cc:224] Handling call
0709 12:18:57.812328 (+900003us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:18:58.045015 (+232687us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5062144}
I20260709 12:18:58.077425 12490 pending_rounds.cc:85] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 28
I20260709 12:18:58.091135 12490 pending_rounds.cc:107] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 50.29
I20260709 12:18:58.091696 12490 raft_consensus.cc:2889] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:18:58.094653 12645 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.26-46.26]
I20260709 12:18:58.095948 12645 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
I20260709 12:18:58.097044 12495 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 54.
I20260709 12:18:58.097556 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:58.090121 28538 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:58.100986 28538 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:58.068771 28346 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:58.107889 12639 raft_consensus.cc:2368] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52.
I20260709 12:18:58.109844 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:58.112946 12494 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:18:58.114565 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:58.116221 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6128) took 1013 ms. Trace:
I20260709 12:18:58.116808 12494 rpcz_store.cc:276] 0709 12:18:57.102383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.075755 (+973372us) service_pool.cc:224] Handling call
0709 12:18:58.112714 (+ 36959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.115741 (+  3027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.116053 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:18:58.036824 28571 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.068s
W20260709 12:18:58.146637 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.145159 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:58.182592 12643 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:18:58.115864 28541 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.123474 28646 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:58.234721 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2416) took 2248 ms. Trace:
I20260709 12:18:58.249711 12489 rpcz_store.cc:276] 0709 12:18:55.986247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.051035 (+ 64788us) service_pool.cc:224] Handling call
0709 12:18:56.066837 (+ 15802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.148335 (+1081498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.234411 (+1086076us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540020c6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:58.234643 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2417610112}
W20260709 12:18:58.236081 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4584) took 2204 ms (client timeout 2192 ms). Trace:
W20260709 12:18:58.250988 12491 rpcz_store.cc:269] 0709 12:18:56.031782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.105161 (+ 73379us) service_pool.cc:224] Handling call
0709 12:18:58.235874 (+2130713us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400191c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:58.236007 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3162535680}
I20260709 12:18:58.205766 28541 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:58.257462 28557 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.670s	user 0.012s	sys 0.000s
I20260709 12:18:58.458020 12496 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
W20260709 12:18:58.484438 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.458688 28279 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:58.511092 28279 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:58.421911 28502 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.516393 28502 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:58.457669 28137 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.851s	user 0.010s	sys 0.004s
I20260709 12:18:58.519251 28137 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:18:58.421260 28281 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.522723 28281 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:58.536460 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:58.538697 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:58.567306 12640 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:58.579193 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:58.307672 28055 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.594313 28055 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:58.605538 12569 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:58.637058 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:58.639303 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:58.641050 12569 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:18:58.633375 28569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.113s	sys 0.175s
I20260709 12:18:58.643112 12567 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 46. Ops: []
I20260709 12:18:58.558138 28544 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:58.625728 28518 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Stepping down as leader of term 50
W20260709 12:18:58.602085 28644 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.000s
W20260709 12:18:58.668751 28638 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 1.075s	user 0.003s	sys 0.000s
I20260709 12:18:58.670915 28502 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:58.745306 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:58.559633 28557 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:58.678385 28537 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:58.556464 27833 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:58.682718 12536 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.056s	sys 0.079s
W20260709 12:18:58.776257 12536 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.056s	sys 0.080s
W20260709 12:18:58.776782 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 780167us
I20260709 12:18:58.597364 28055 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:58.751068 12645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.004s	sys 0.000s
I20260709 12:18:58.613343 28279 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:58.755884 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.674183 28542 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:58.714324 28569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.114s	sys 0.175s
I20260709 12:18:58.763749 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:18:58.721530 28346 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.019s	sys 0.055s
W20260709 12:18:58.708114 12646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.003s	sys 0.007s
W20260709 12:18:58.794821 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.024s	sys 0.047s
I20260709 12:18:58.797132 12642 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: [47.30-47.30]
W20260709 12:18:58.799327 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.045s	sys 0.052s
I20260709 12:18:58.727172 28518 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:58.758888 28137 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:58.802182 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.044s	sys 0.061s
W20260709 12:18:58.802500 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.006s	sys 0.000s
W20260709 12:18:58.794948 28636 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.850s	user 0.012s	sys 0.004s
I20260709 12:18:58.756413 28544 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:58.791115 27833 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:58.819299 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
W20260709 12:18:58.821862 12646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.003s	sys 0.007s
W20260709 12:18:58.823108 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.044s	sys 0.062s
I20260709 12:18:58.877130 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6009) took 1043 ms. Trace:
I20260709 12:18:58.877697 12563 rpcz_store.cc:276] 0709 12:18:57.834195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.908273 (+ 74078us) service_pool.cc:224] Handling call
0709 12:18:57.908750 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.909059 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 33
0709 12:18:58.044655 (+135596us) log.cc:844] Serialized 14 byte log entry
0709 12:18:58.876539 (+831884us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:58.876634 (+    95us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:18:58.876721 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.876825 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.877039 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35489,"prepare.queue_time_us":303718,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":2450,"thread_start_us":966483,"threads_started":2}
I20260709 12:18:58.828331 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6000) took 3040 ms. Trace:
I20260709 12:18:58.862958 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6127) took 1779 ms. Trace:
I20260709 12:18:58.889169 12493 rpcz_store.cc:276] 0709 12:18:57.083706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.220901 (+137195us) service_pool.cc:224] Handling call
0709 12:18:58.862703 (+1641802us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:58.862893 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2074581376}
I20260709 12:18:58.881207 12565 rpcz_store.cc:276] 0709 12:18:55.788287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:56.978669 (+1190382us) service_pool.cc:224] Handling call
0709 12:18:57.002982 (+ 24313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:57.027044 (+ 24062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:57.027089 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:57.027664 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:18:57.266851 (+239187us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:57.266908 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.820040 (+1553132us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b54003ad754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:58.820109 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:58.820167 (+    58us) raft_consensus.cc:1685] finished
0709 12:18:58.820181 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.828219 (+  8038us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10914,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":26397440,"thread_start_us":181577,"threads_started":1,"wal-append.queue_time_us":329802,"wal-append.run_cpu_time_us":1723,"wal-append.run_wall_time_us":1001743}
W20260709 12:18:58.901583 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:58.903970 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:58.820976 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.024s	sys 0.048s
W20260709 12:18:58.821775 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.045s	sys 0.052s
I20260709 12:18:58.849690 12569 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:18:58.700150 28468 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:58.907774 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:58.923592 28502 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:58.910341 28630 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.919s	user 0.014s	sys 0.000s
W20260709 12:18:58.910570 12443 auto_rebalancer.cc:629] Failed to schedule move for tablet 3eadf951b63a43fd993cea2c96d5214f: Aborted: Op aborted by new leader
W20260709 12:18:58.872152 28003 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:18:58.813179 28644 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:58.915027 27833 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:58.916167 28670 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:18:58.923933 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2998) took 1079 ms. Trace:
I20260709 12:18:59.046926 12564 rpcz_store.cc:276] 0709 12:18:57.844548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.930859 (+ 86311us) service_pool.cc:224] Handling call
0709 12:18:57.931119 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.931307 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:18:58.204950 (+273643us) log.cc:844] Serialized 14 byte log entry
0709 12:18:58.923454 (+718504us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:58.923527 (+    73us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:18:58.923617 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.923710 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:58.923867 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24930,"prepare.queue_time_us":324475,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":75,"raft.queue_time_us":154,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":4032,"thread_start_us":991167,"threads_started":2}
I20260709 12:18:59.048804 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:58.943892 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:58.992553 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.042363 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.051596 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:59.035166 28627 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.998s	user 0.011s	sys 0.005s
W20260709 12:18:58.822409 28346 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.019s	sys 0.056s
W20260709 12:18:58.833977 28488 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 46 is higher.
W20260709 12:18:58.820024 28467 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.350s	user 0.013s	sys 0.010s
W20260709 12:18:58.841714 28650 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.922s	user 0.015s	sys 0.000s
W20260709 12:18:58.925238 28520 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.843s	user 0.013s	sys 0.003s
I20260709 12:18:58.852839 28279 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:59.101028 28279 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.103283 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5842) took 1280 ms. Trace:
W20260709 12:18:59.020462 28642 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.373s	user 0.006s	sys 0.003s
I20260709 12:18:58.909808 28541 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:59.022751 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5844) took 1199 ms. Trace:
I20260709 12:18:59.104643 12644 rpcz_store.cc:276] 0709 12:18:57.823077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.938514 (+115437us) service_pool.cc:224] Handling call
0709 12:18:59.022471 (+1083957us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:59.022662 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281469312}
I20260709 12:18:59.023190 28575 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:18:59.105398 28644 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:59.105759 28575 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:18:59.060458 28430 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.106665 28575 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.107991 28575 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:18:59.108353 28557 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:59.024319 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6119) took 1984 ms (client timeout 2153 ms). Trace:
I20260709 12:18:59.024436 28534 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:18:59.109925 28534 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:59.110070 28520 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.110934 28534 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:59.026456 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6131) took 1847 ms (client timeout 1870 ms). Trace:
I20260709 12:18:59.112038 28534 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:59.112649 28520 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.029109 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.113534 28650 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:59.114283 12493 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:59.115864 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.116700 12493 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:18:59.030341 12462 connection.cc:256] RPC call timeout handler was delayed by -1.517432627000062s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
I20260709 12:18:59.030742 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:59.118547 12462 connection.cc:256] RPC call timeout handler was delayed by -1.2060326270000132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
I20260709 12:18:59.118942 12490 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:18:59.030836 12566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6319) took 3473 ms (client timeout 1987 ms). Trace:
W20260709 12:18:59.112287 12492 rpcz_store.cc:269] 0709 12:18:57.178890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.214488 (+ 35598us) service_pool.cc:224] Handling call
0709 12:18:59.026199 (+1811711us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:59.026368 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127299456}
W20260709 12:18:59.119434 12566 rpcz_store.cc:269] 0709 12:18:55.557221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.849947 (+292726us) service_pool.cc:224] Handling call
0709 12:18:59.030652 (+3180705us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400181354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:59.030760 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3577609984}
I20260709 12:18:59.120282 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:58.993536 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:59.053330 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.051507 12491 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:18:59.058463 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.122058 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:18:59.038159 28601 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.967s	user 0.017s	sys 0.000s
I20260709 12:18:58.924091 28137 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:59.123082 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.123945 12564 raft_consensus.cc:2410] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 47 because replica has last-logged OpId of term: 44 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 26.
I20260709 12:18:59.121537 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.125952 12492 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 52
I20260709 12:18:59.130577 12642 rpcz_store.cc:276] 0709 12:18:57.822565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.039107 (+216542us) service_pool.cc:224] Handling call
0709 12:18:58.063340 (+ 24233us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:58.821276 (+757936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.102800 (+281524us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54002a58d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:59.103198 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1686555392}
I20260709 12:18:59.036058 28502 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:59.070855 28665 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:18:59.138607 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:59.072183 28055 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:59.139591 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:59.088016 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4663) took 1021 ms. Trace:
I20260709 12:18:59.126196 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:59.138607 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:18:59.089803 28672 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.234s	user 0.000s	sys 0.002s
I20260709 12:18:59.103482 28279 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.019691 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:59.151796 12569 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:59.040020 28630 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:59.109313 12495 rpcz_store.cc:269] 0709 12:18:57.040069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.040213 (+   144us) service_pool.cc:224] Handling call
0709 12:18:59.024051 (+1983838us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:59.024227 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2203112192}
I20260709 12:18:59.154984 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.122244 12497 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 46
I20260709 12:18:59.046113 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.122591 12489 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:59.065869 12563 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:59.158047 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:59.158746 13253 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:18:59.056370 28646 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:59.134857 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:59.136333 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:59.163523 12572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.961s	user 0.054s	sys 0.112s
I20260709 12:18:59.163228 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:18:59.163868 12572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.054s	sys 0.112s
I20260709 12:18:59.141178 12646 rpcz_store.cc:276] 0709 12:18:58.066562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.092622 (+ 26060us) service_pool.cc:224] Handling call
0709 12:18:58.093078 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:58.093574 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:58.093617 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:58.094826 (+  1209us) log.cc:844] Serialized 26 byte log entry
0709 12:18:58.875410 (+780584us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:58.875483 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.875604 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:59.087749 (+212145us) raft_consensus.cc:1685] finished
0709 12:18:59.087772 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.087927 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":147,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":8155,"thread_start_us":735503,"threads_started":1,"wal-append.queue_time_us":915585}
I20260709 12:18:59.150518 12491 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 54: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:59.117926 13255 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:59.165110 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6002) took 1338 ms. Trace:
I20260709 12:18:59.162425 12566 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:59.141196 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:59.153697 12570 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.946s	user 0.121s	sys 0.145s
W20260709 12:18:59.166673 12570 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.121s	sys 0.146s
I20260709 12:18:59.169104 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6007) took 1336 ms. Trace:
I20260709 12:18:58.867537 28518 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 50.36, Last appended by leader: 36, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.169474 12570 rpcz_store.cc:276] 0709 12:18:57.833274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.833465 (+   191us) service_pool.cc:224] Handling call
0709 12:18:57.833949 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.834272 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:58.190431 (+356159us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.168780 (+978349us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:59.168830 (+    50us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:59.168879 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.168950 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.169054 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1457,"prepare.queue_time_us":369126,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"thread_start_us":1295160,"threads_started":2}
I20260709 12:18:59.161069 12492 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:18:59.175215 12569 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:18:59.173805 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.176908 13262 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:59.177049 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:18:59.140614 28542 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.135842 28557 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.150797 28137 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.180620 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.189208 13261 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:18:59.193975 28518 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:59.039494 28468 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.196049 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.203670 28630 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.206111 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.232777 12491 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:18:59.155102 28644 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
W20260709 12:18:59.211473 12488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.015s	user 0.009s	sys 0.016s
I20260709 12:18:59.165467 28507 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.178424 28125 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:59.220386 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.221873 12492 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:18:59.222080 13254 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:18:59.140499 28601 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.234990 12494 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:18:59.139596 28055 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.228708 12489 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:18:59.226513 12497 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:18:59.228080 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:59.234066 12488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.009s	sys 0.017s
I20260709 12:18:59.235967 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.244947 12492 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 55.
I20260709 12:18:59.245358 12495 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:59.246179 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.247059 12492 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:18:59.248176 12569 raft_consensus.cc:2376] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46. Re-sending same reply.
W20260709 12:18:59.238083 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.238814 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.252175 12491 raft_consensus.cc:2393] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:59.239845 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.252954 12494 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:18:59.255098 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.256282 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.258253 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.259835 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.260823 12489 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:18:59.242308 28125 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:18:59.273558 12492 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: [50.36-50.36]
I20260709 12:18:59.234494 28644 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:18:59.250407 28573 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:18:59.274094 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.276002 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.278524 28644 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:59.230652 28557 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.281790 12497 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 41. Current term is 42. Ops: [41.27-41.27]
I20260709 12:18:59.273195 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.229803 28542 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.284919 12491 raft_consensus.cc:2410] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 43 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 22.
I20260709 12:18:59.283938 12495 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:18:59.283077 28573 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.290191 28605 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:59.290726 28605 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:59.274806 28055 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:59.291595 28605 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.293083 28605 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:18:59.250852 28507 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:18:59.283236 28671 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
W20260709 12:18:59.305538 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.291481 28573 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.309471 12494 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:18:59.356000 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.000s	sys 0.027s
W20260709 12:18:59.358240 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.000s	sys 0.027s
I20260709 12:18:59.359725 12493 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 47 because replica has last-logged OpId of term: 46 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 25.
W20260709 12:18:59.365940 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:18:59.368021 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 465880us
I20260709 12:18:59.371217 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.387591 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.356849 28502 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:59.370740 28680 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:18:59.392484 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.369160 28346 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:18:59.411468 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 5834) took 1631 ms (client timeout 1886 ms). Trace:
W20260709 12:18:59.411971 12639 rpcz_store.cc:269] 0709 12:18:57.780265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.919418 (+139153us) service_pool.cc:224] Handling call
0709 12:18:59.411209 (+1491791us) spinlock_profiling.cc:243] Waited 80.4 ms on lock 0x7b5400153954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:18:59.411394 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184953088}
W20260709 12:18:59.411259 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 267815us
I20260709 12:18:59.413410 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6138) took 2093 ms. Trace:
I20260709 12:18:59.413874 12496 rpcz_store.cc:276] 0709 12:18:57.319566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.073804 (+754238us) service_pool.cc:224] Handling call
0709 12:18:58.074092 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.529428 (+455336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.413128 (+883700us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54000f5fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:59.413347 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":808134016}
I20260709 12:18:59.391916 28650 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:18:59.394002 28646 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
W20260709 12:18:59.423651 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 238685us
I20260709 12:18:59.397089 28137 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:59.397322 28468 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.423236 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.390019 28467 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 45.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.029s
I20260709 12:18:59.404459 28601 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:59.390219 28630 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.032s
I20260709 12:18:59.393618 28279 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.029s
I20260709 12:18:59.417439 28430 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.420485 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6130) took 2276 ms. Trace:
I20260709 12:18:59.445652 12488 rpcz_store.cc:276] 0709 12:18:57.144013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.090246 (+946233us) service_pool.cc:224] Handling call
0709 12:18:58.119359 (+ 29113us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.119953 (+   594us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:18:58.195840 (+ 75887us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.419852 (+1224012us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:59.419962 (+   110us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:59.420059 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.420178 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.420382 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":75682,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"spinlock_wait_cycles":5556992,"thread_start_us":1222743,"threads_started":2,"wal-append.queue_time_us":1192920}
I20260709 12:18:59.449958 12638 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 45. Current term is 46. Ops: [45.28-45.28]
I20260709 12:18:59.454141 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.455382 12570 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 46 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
I20260709 12:18:59.432543 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.456236 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.456530 12639 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:18:59.433495 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2265) took 2367 ms. Trace:
I20260709 12:18:59.416105 28542 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:18:59.437119 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.422408 28520 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:18:59.420320 28541 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:59.438642 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.442257 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:18:59.427207 28679 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:18:59.435544 28055 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:18:59.465476 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.466408 12646 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:59.445068 12569 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
I20260709 12:18:59.458123 12641 rpcz_store.cc:276] 0709 12:18:57.066043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.223536 (+157493us) service_pool.cc:224] Handling call
0709 12:18:57.223956 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.255651 (+ 31695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.433241 (+2177590us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5400152054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:59.433432 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26613504}
I20260709 12:18:59.418139 28346 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.430540 28534 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:59.468963 28534 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.471884 28534 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:59.463873 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:18:59.473001 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.474566 12642 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:18:59.476874 12640 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:18:59.482517 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56965.4us
I20260709 12:18:59.483716 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:59.455693 28557 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:59.455570 28683 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:18:59.462571 28646 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:18:59.486265 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60377.4us
W20260709 12:18:59.503082 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:18:59.501956 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:59.508443 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.509464 12569 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 53: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:59.510607 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:59.510867 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.510641 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.512332 12569 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 54 because replica has last-logged OpId of term: 52 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 29.
I20260709 12:18:59.509303 12563 raft_consensus.cc:1217] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Deduplicated request from leader. Original: 39.24->[41.25-45.27]   Dedup: 43.26->[45.27-45.27]
I20260709 12:18:59.513376 12563 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: [41.25-45.27]
I20260709 12:18:59.514783 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.511739 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:18:59.516978 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.518716 12569 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 43 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 21.
I20260709 12:18:59.519853 12563 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:59.521668 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.522341 12571 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:18:59.523515 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.460469 28430 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.527909 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:59.535187 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.522s (SENT)
I20260709 12:18:59.526135 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4666) took 1458 ms. Trace:
I20260709 12:18:59.535826 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.536098 12637 rpcz_store.cc:276] 0709 12:18:58.067413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:58.120717 (+ 53304us) service_pool.cc:224] Handling call
0709 12:18:58.173548 (+ 52831us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:58.174291 (+   743us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:58.174337 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:58.175017 (+   680us) log.cc:844] Serialized 26 byte log entry
0709 12:18:58.906790 (+731773us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:18:58.906839 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:58.906937 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:18:59.525808 (+618871us) raft_consensus.cc:1685] finished
0709 12:18:59.525839 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.526053 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7963,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":16079,"thread_start_us":693182,"threads_started":1,"wal-append.queue_time_us":1036815}
I20260709 12:18:59.542179 12570 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:18:59.479555 28650 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:18:59.498690 28683 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:59.485702 28575 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 26, Time since last communication: 0.033s
I20260709 12:18:59.546721 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:18:59.546892 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.548519 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:18:59.476622 28541 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.554791 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 38 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.562981 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.484494 28055 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.559497 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:18:59.535425 28646 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.518280 28682 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:18:59.560726 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.500818 28686 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 44.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.069s
W20260709 12:18:59.564281 28678 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.171s	user 0.002s	sys 0.000s
I20260709 12:18:59.602942 13259 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:18:59.590658 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:18:59.604588 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:59.595083 12570 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:18:59.603406 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:18:59.593647 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 93612.2us
W20260709 12:18:59.596063 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49161.1us
W20260709 12:18:59.606235 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71122.2us
W20260709 12:18:59.564934 28676 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:18:59.600041 28468 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:18:59.595464 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:18:59.608076 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 40.
I20260709 12:18:59.606981 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:59.611604 13259 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:18:59.573316 27765 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:59.573266 28657 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:18:59.593392 28693 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.033s
I20260709 12:18:59.623592 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:18:59.625300 13218 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.664s (SENT)
I20260709 12:18:59.628175 12565 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:18:59.628298 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:59.598100 28684 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:59.604455 28694 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:59.658325 13254 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 54.
W20260709 12:18:59.661130 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 54063.1us
I20260709 12:18:59.662217 28557 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:59.670905 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:18:59.598256 28646 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.600739 28534 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.041s
I20260709 12:18:59.648258 28544 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:59.676137 28697 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:18:59.676380 28701 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.677769 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:18:59.678628 28685 consensus_queue.cc:1059] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 49.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:18:59.668150 28686 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:18:59.681149 28694 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.167018 12572 rpcz_store.cc:276] 0709 12:18:57.827180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.827341 (+   161us) service_pool.cc:224] Handling call
0709 12:18:57.827710 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:57.828036 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:18:58.197391 (+369355us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.164662 (+967271us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:59.164747 (+    85us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:59.164829 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.164917 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:18:59.165054 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":375092,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":4166,"thread_start_us":1318049,"threads_started":2}
I20260709 12:18:59.684315 12488 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:18:59.685281 28698 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:18:59.632907 28689 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:18:59.629944 28690 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:18:59.688014 12494 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:18:59.688820 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.690074 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.739624 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.740116 12489 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:59.740336 12494 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
W20260709 12:18:59.692094 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54548.6us
W20260709 12:18:59.693429 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98033.4us
I20260709 12:18:59.711081 12565 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:18:59.601215 28682 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.746801 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:18:59.747720 28682 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.748459 12490 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:59.748734 28579 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:18:59.749342 28579 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:59.750491 28579 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.751971 28579 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:18:59.616950 28644 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.755937 28644 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.715170 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.715613 28346 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:18:59.715929 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:18:59.722064 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44625.1us
I20260709 12:18:59.758236 12497 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:18:59.668489 28467 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.680083 28657 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:18:59.733858 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 231968us
W20260709 12:18:59.767201 28696 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:18:59.767884 12461 leader_election.cc:400] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46.
I20260709 12:18:59.768242 12461 leader_election.cc:403] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:18:59.768677 28706 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:18:59.763762 28544 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:18:59.769994 28706 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:18:59.769888 28544 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.770539 28706 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:59.771757 28706 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 50.30, Last appended by leader: 30, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.772706 28467 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.726612 28701 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:18:59.727293 28003 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:18:59.741884 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:18:59.738077 28641 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 2.076s	user 0.011s	sys 0.001s
I20260709 12:18:59.734441 28698 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:18:59.743947 28689 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.760012 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 2999) took 1915 ms. Trace:
I20260709 12:18:59.760257 12496 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52. Ops: []
I20260709 12:18:59.741415 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.739167 12492 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:18:59.762104 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:59.766070 13259 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 52 because replica has last-logged OpId of term: 51 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 28.
I20260709 12:18:59.786324 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:18:59.757215 28601 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Stepping down as leader of term 52
W20260709 12:18:59.876293 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114292us
W20260709 12:18:59.737126 28688 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
W20260709 12:18:59.860965 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 119177us
I20260709 12:18:59.853094 12567 rpcz_store.cc:276] 0709 12:18:57.844837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:57.963513 (+118676us) service_pool.cc:224] Handling call
0709 12:18:57.963852 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:58.715056 (+751204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.759644 (+1044588us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54001b2bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:18:59.759832 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1561556736}
I20260709 12:18:59.873127 12494 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 50 because replica has last-logged OpId of term: 47 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 25.
I20260709 12:18:59.873132 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:18:59.667759 28683 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.28, Last appended by leader: 28, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.896037 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:18:59.882459 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96228.2us
W20260709 12:18:59.882436 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 244677us
I20260709 12:18:59.879617 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:18:59.775592 28467 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:59.875427 28699 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:18:59.779932 28125 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.920213 28699 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:18:59.921617 13259 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
I20260709 12:18:59.782142 28541 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:18:59.852722 28544 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.922484 28708 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:18:59.923115 28699 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:18:59.922009 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:18:59.927999 28542 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:18:59.929797 28684 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:18:59.930894 28684 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:18:59.928844 28542 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.003s
I20260709 12:18:59.933161 28684 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 44.23, Last appended by leader: 23, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.934307 28644 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20260709 12:18:59.936066 28644 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 45. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46.
I20260709 12:18:59.936975 28644 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:18:59.783797 28430 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:18:59.965960 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 92963.5us
I20260709 12:18:59.940366 13218 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:18:59.880290 28537 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:18:59.915215 28690 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:18:59.773514 28706 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:18:59.876569 28601 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:18:59.763934 28693 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:18:59.784487 28055 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:18:59.995862 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:00.004614 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6013) took 1134 ms. Trace:
I20260709 12:19:00.040671 12564 rpcz_store.cc:276] 0709 12:18:58.870270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.124496 (+254226us) service_pool.cc:224] Handling call
0709 12:18:59.155920 (+ 31424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:59.156596 (+   676us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.156620 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:18:59.157101 (+   481us) log.cc:844] Serialized 26 byte log entry
0709 12:18:59.436282 (+279181us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:18:59.436333 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:18:59.436416 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:00.004336 (+567920us) raft_consensus.cc:1685] finished
0709 12:19:00.004367 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:00.004544 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4908,"raft.queue_time_us":1044,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":11380,"thread_start_us":204678,"threads_started":1,"wal-append.queue_time_us":297571}
W20260709 12:18:59.969573 28605 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.676s	user 0.013s	sys 0.004s
I20260709 12:19:00.007299 12494 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 45. Current term is 46. Ops: []
W20260709 12:19:00.004717 28518 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.768s	user 0.007s	sys 0.006s
I20260709 12:18:59.775755 28646 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:18:59.997042 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66153.7us
I20260709 12:19:00.008399 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:18:59.937516 28644 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:00.012073 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71843.4us
I20260709 12:18:59.735735 28694 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:18:59.996605 28683 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:00.020682 28281 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:00.039093 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:00.039408 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:00.060315 12491 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 52: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:19:00.043691 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 275919us
I20260709 12:19:00.046336 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:00.070714 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:00.054006 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:00.055917 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:00.078787 28720 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:19:00.016836 28430 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:00.046938 28682 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:00.047170 28690 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:00.093783 12461 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:00.080844 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:00.081214 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:00.052194 28539 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:19:00.088557 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:00.057901 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 178335us
I20260709 12:19:00.089440 28430 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:00.094436 28467 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 37.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.021s
I20260709 12:19:00.112423 28695 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:00.113062 28695 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:00.113446 28695 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:00.070714 28715 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260709 12:19:00.088815 28541 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:00.111085 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.862s (SENT)
I20260709 12:19:00.086329 28717 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.124816 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:00.113540 28467 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:19:00.132445 28467 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:00.133759 28467 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.27, Last appended by leader: 27, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.135489 28467 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:00.102548 28539 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:00.138620 28707 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:19:00.139148 28539 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.151965 28705 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:00.158541 28701 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:00.122928 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:00.167457 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:00.172501 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49668.2us
I20260709 12:19:00.177826 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:00.112514 28430 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:00.017514 28698 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 44.31, Last appended by leader: 31, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.160163 28557 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:00.094975 28682 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.182581 28715 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:00.185541 28705 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:00.165059 28717 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.202250 12567 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:00.182335 28720 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:00.199811 28690 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 50.28, Last appended by leader: 28, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.185730 28718 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:00.094722 28601 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 52.30, Last appended by leader: 30, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.211758 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:00.207873 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 114332us
I20260709 12:19:00.219986 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:00.221076 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:00.195960 28721 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:00.129165 28644 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.203672 28557 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:00.208407 28705 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:00.214147 28657 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.218736 28698 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:00.215694 28718 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:00.225962 28055 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.234269 28534 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:00.206571 28695 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:00.278277 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58393.4us
I20260709 12:19:00.279373 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:00.263659 28705 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:00.260231 28703 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:19:00.302727 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:00.269124 28721 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:00.270377 28718 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:00.309907 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:00.314628 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:00.284286 28502 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:00.302971 28703 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:00.268042 28720 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.043s
I20260709 12:19:00.318128 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:00.268913 28657 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.282876 28534 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.308226 28725 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:00.277063 28055 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.316583 28727 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:00.321280 28694 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:00.321338 28726 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:00.339169 28695 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:00.325681 28703 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:00.320567 28557 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.305927 28608 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:19:00.318061 28502 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:00.331175 28685 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:00.398408 28502 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:00.399180 28685 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:00.400527 28685 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.28, Last appended by leader: 28, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.405807 28534 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:00.407963 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:00.409093 12566 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:00.376425 28721 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.28, Last appended by leader: 28, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.408084 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:00.416751 12570 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:00.338809 28715 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:00.375635 28720 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:19:00.425000 28720 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:00.425976 28720 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.418218 28682 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:00.427645 28720 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:19:00.366364 28722 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:19:00.435755 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 42 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:00.437032 13262 raft_consensus.cc:2410] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 42 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 26.
I20260709 12:19:00.387144 28726 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:00.450430 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:00.386224 28055 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:00.342940 12461 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:00.474489 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:00.349668 28675 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.932s	user 0.002s	sys 0.000s
W20260709 12:19:00.529033 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:00.526803 13261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2247) took 1402 ms (client timeout 1589 ms). Trace:
W20260709 12:19:00.538611 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 195741us
W20260709 12:19:00.537791 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 63427.9us
I20260709 12:19:00.543006 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.553249 13262 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:00.549403 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:00.550925 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:00.552176 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:00.544771 13261 rpcz_store.cc:269] 0709 12:18:59.124623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.124787 (+   164us) service_pool.cc:224] Handling call
0709 12:19:00.526452 (+1401665us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:00.526703 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99032448}
I20260709 12:19:00.541174 28055 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:00.557737 12637 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:00.558220 12641 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:00.336061 28541 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:00.561800 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:00.329298 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 103352us
I20260709 12:19:00.532951 28534 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:00.560818 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:00.564851 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:00.567368 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:00.567607 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.568375 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:00.570399 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:00.571234 12645 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:00.571393 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:00.571905 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:00.572127 12642 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 36.
I20260709 12:19:00.572535 13261 raft_consensus.cc:2393] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:00.573071 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:00.573982 12645 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:00.574147 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:00.570111 12637 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:19:00.547573 28694 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:00.527865 28705 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.587073 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.600750 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.578027 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:00.557526 28716 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:00.584673 12461 leader_election.cc:400] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52.
I20260709 12:19:00.612968 12461 leader_election.cc:403] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:00.594235 28727 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:00.562775 28055 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:00.570086 28534 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:00.578809 28608 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:00.590114 28725 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:00.630756 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.605371 28544 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:00.632238 28724 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election lost for term 51. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52.
I20260709 12:19:00.568717 28715 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 50.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.059s
I20260709 12:19:00.611922 28694 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:00.632516 28281 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:00.722893 13255 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2022) took 4914 ms (client timeout 2978 ms). Trace:
W20260709 12:19:00.724025 13255 rpcz_store.cc:269] 0709 12:18:55.805073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:55.806899 (+  1826us) service_pool.cc:224] Handling call
0709 12:19:00.719337 (+4912438us) spinlock_profiling.cc:243] Waited 3.31 s on lock 0x7b5400588254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:00.719481 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7611726592}
W20260709 12:19:00.618041 28683 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.551s	user 0.013s	sys 0.000s
I20260709 12:19:00.646380 28608 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:00.648232 28534 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:00.718546 28544 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:00.720234 28688 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:19:00.734972 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.648941 28055 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:00.678960 28701 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.756168 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:00.765385 13255 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:00.727998 28690 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:19:00.733667 28608 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:00.769439 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.732256 28537 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:00.732336 28706 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:00.780917 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2041) took 1108 ms. Trace:
I20260709 12:19:00.786991 13260 rpcz_store.cc:276] 0709 12:18:59.672631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.748771 (+1076140us) service_pool.cc:224] Handling call
0709 12:19:00.780832 (+ 32061us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:00.726876 28725 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:00.785579 13254 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:00.770244 28703 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:00.786371 28733 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:19:00.749126 28544 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Advancing to term 56
W20260709 12:19:00.773144 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 208379us
W20260709 12:19:00.773712 28579 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.019s	user 0.019s	sys 0.000s
I20260709 12:19:00.635183 28716 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.732465 28281 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:00.797000 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:00.815820 13262 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:00.646807 28726 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.785382 28502 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.771672 28690 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:00.803701 12645 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:19:00.804426 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:00.782398 28701 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.791837 28725 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:00.809190 28734 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:19:00.800475 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:00.814271 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 258890us
I20260709 12:19:00.816794 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:00.790534 28706 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.829427 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:00.820719 28718 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.500s	user 0.001s	sys 0.010s
I20260709 12:19:00.793771 28537 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:00.824519 28690 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:19:00.827205 28721 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:00.841565 28281 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:00.841845 28706 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:00.842732 28741 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:00.843376 28488 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:00.843333 28741 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.843900 28488 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:00.844820 28488 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:00.845374 28698 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.000s
I20260709 12:19:00.813973 28694 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.24, Last appended by leader: 24, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.833411 28683 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:00.846331 28701 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:00.835595 28430 consensus_meta.cc:287] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.641s	user 0.005s	sys 0.008s
I20260709 12:19:00.835513 28467 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:00.845618 28741 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:00.838759 28721 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:00.856956 28488 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:00.857126 28430 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.998586 28740 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:00.999953 28740 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.906937 28725 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:00.998649 28720 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:01.011411 12462 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:01.013142 28644 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:19:00.953236 12489 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
W20260709 12:19:00.963258 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.023690 28682 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.024698 28682 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:00.966491 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 165980us
I20260709 12:19:00.866127 28537 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:00.875491 28657 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:00.853811 28683 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.031594 28683 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:01.031872 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:00.878597 28686 raft_consensus.cc:3055] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:01.034178 28686 raft_consensus.cc:740] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:01.035203 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.036391 12564 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:01.035983 28686 consensus_queue.cc:260] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 49.30, Last appended by leader: 30, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.867098 28689 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:00.891589 28690 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.870038 28703 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:00.852003 28705 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.033548 12567 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:19:00.865909 28716 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:00.893527 28717 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:00.904618 28711 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.937s	user 0.003s	sys 0.005s
I20260709 12:19:00.836103 28718 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:00.871137 28714 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.804s	user 0.002s	sys 0.008s
I20260709 12:19:00.906865 28715 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:01.047917 28715 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:01.050053 28715 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 47.30, Last appended by leader: 30, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.053049 28715 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:00.864699 28721 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 50.30, Last appended by leader: 30, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:00.985894 28467 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:01.054482 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.056118 28467 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 26, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.019186 28740 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:01.059231 12495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4597) took 1938 ms (client timeout 1619 ms). Trace:
W20260709 12:19:01.059896 12495 rpcz_store.cc:269] 0709 12:18:59.120908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.154680 (+ 33772us) service_pool.cc:224] Handling call
0709 12:19:01.058608 (+1903928us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:01.058891 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":206272896}
I20260709 12:19:00.880677 28544 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:01.067104 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.072170 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:01.041172 28690 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:01.077940 28682 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.083848 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:01.085953 28644 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:19:00.997457 28430 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:01.037611 28686 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:01.073411 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6012) took 2209 ms. Trace:
I20260709 12:19:01.097709 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:01.099319 28644 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:01.099999 28644 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.100479 28644 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:01.100981 28544 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.102257 13261 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:01.104746 28003 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=1 applied=25 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:19:01.106508 28003 tablet_bootstrap.cc:492] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:19:01.120843 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.122339 28125 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.133353 28125 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.134033 28718 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:01.134392 28725 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.134323 28003 ts_tablet_manager.cc:1403] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 9.226s	user 0.135s	sys 0.060s
I20260709 12:19:01.141480 13256 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:01.146281 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:01.147576 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.148433 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.148766 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.133584 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.153916 12569 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 28. Preceding OpId from leader: term: 52 index: 29. (index mismatch)
I20260709 12:19:01.134274 28125 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.155040 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.155212 12640 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:19:01.135646 28725 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:01.165030 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2011) took 1545 ms. Trace:
I20260709 12:19:01.166714 12571 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:01.167254 12495 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:19:01.168552 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.171173 12571 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
I20260709 12:19:01.171293 12639 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 24. (term mismatch)
I20260709 12:19:01.165459 13256 rpcz_store.cc:276] 0709 12:18:59.612190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.082778 (+1470588us) service_pool.cc:224] Handling call
0709 12:19:01.157213 (+ 74435us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.156347 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.172108 12639 pending_rounds.cc:85] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 23
I20260709 12:19:01.186148 12638 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:19:01.189792 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134763us
I20260709 12:19:01.190033 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.191058 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:01.191964 12642 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:01.192075 28507 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.192942 12646 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:01.194094 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.194872 12644 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:01.195390 12572 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:01.195951 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:01.196213 28720 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:19:01.196640 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:01.197347 13256 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099"
dest_uuid: "9485b0afaf144396adf854fff857414e"
mode: GRACEFUL
new_leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
 from {username='slave'} at 127.0.0.1:50116
I20260709 12:19:01.198324 13256 raft_consensus.cc:606] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Received request to transfer leadership to TS 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:01.200389 13256 raft_consensus.cc:1240] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:01.201035 28752 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.201232 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2036) took 1591 ms. Trace:
I20260709 12:19:01.179701 12639 pending_rounds.cc:107] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 42.24
I20260709 12:19:01.206166 12564 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:01.218142 13256 rpcz_store.cc:276] 0709 12:18:59.609547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.199536 (+1589989us) service_pool.cc:224] Handling call
0709 12:19:01.200028 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:01.200975 (+   947us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.201154 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.207692 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:01.198345 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:01.208489 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:01.209018 28125 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:01.166548 28534 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.218335 12639 raft_consensus.cc:2889] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:01.211580 28539 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:01.212088 28752 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.213014 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2428) took 1202 ms. Trace:
I20260709 12:19:01.214306 12646 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 48 index: 29. (index mismatch)
I20260709 12:19:01.200268 28754 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.214869 13255 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:01.095886 12568 rpcz_store.cc:276] 0709 12:18:58.864145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.098848 (+234703us) service_pool.cc:224] Handling call
0709 12:18:59.099329 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.125022 (+ 25693us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:18:59.136115 (+ 11093us) log.cc:844] Serialized 14 byte log entry
0709 12:18:59.744907 (+608792us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b18004b2260. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:18:59.958561 (+213654us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:59.958657 (+    96us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:18:59.958746 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:00.870418 (+911672us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:01.070629 (+200211us) spinlock_profiling.cc:243] Waited 9.77 ms on lock 0x7b18004b2260. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419
0709 12:19:01.072602 (+  1973us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5400102f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:01.072710 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.073098 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10054,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":149,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":30059,"spinlock_wait_cycles":118742784,"thread_start_us":213145,"threads_started":1,"wal-append.queue_time_us":879545}
I20260709 12:19:01.216876 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:01.235482 12644 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:01.219272 12564 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:01.210824 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:01.237417 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.233804 12489 rpcz_store.cc:276] 0709 12:19:00.010303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.011409 (+  1106us) service_pool.cc:224] Handling call
0709 12:19:01.212744 (+1201335us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b5400192154. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:01.212942 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2051832704}
I20260709 12:19:01.248788 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2040) took 1589 ms. Trace:
I20260709 12:19:01.249430 13255 rpcz_store.cc:276] 0709 12:18:59.660019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.730197 (+1070178us) service_pool.cc:224] Handling call
0709 12:19:01.244646 (+514449us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.250227 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:01.251117 13255 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [41.29-43.30]
I20260709 12:19:01.251899 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2038) took 1616 ms. Trace:
I20260709 12:19:01.252446 13257 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:19:01.252233 13255 rpcz_store.cc:276] 0709 12:18:59.635463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.250302 (+1614839us) service_pool.cc:224] Handling call
0709 12:19:01.250782 (+   480us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:01.251576 (+   794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.251803 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.255271 12645 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:01.260814 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.262933 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:01.264740 28754 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.267624 28754 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:01.268929 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61682.9us
I20260709 12:19:01.270114 12564 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 44.26, Last appended by leader: 26, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.272406 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2039) took 1611 ms. Trace:
I20260709 12:19:01.270577 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:01.260655 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2042) took 1574 ms. Trace:
I20260709 12:19:01.272835 13257 rpcz_store.cc:276] 0709 12:18:59.643819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.251551 (+1607732us) service_pool.cc:224] Handling call
0709 12:19:01.252047 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:01.253064 (+  1017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.255188 (+  2124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.278041 28534 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:01.280916 28760 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:01.235260 28125 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:01.281843 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 66793.1us
I20260709 12:19:01.282313 28125 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.284698 28125 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.285876 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.286871 12563 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:01.242188 28718 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.288038 28718 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.291015 28718 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:01.295655 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:01.304807 13255 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:19:01.305896 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2043) took 1590 ms. Trace:
I20260709 12:19:01.306269 13255 rpcz_store.cc:276] 0709 12:18:59.715166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.303791 (+1588625us) service_pool.cc:224] Handling call
0709 12:19:01.304412 (+   621us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:01.305539 (+  1127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.305829 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.273531 13258 rpcz_store.cc:276] 0709 12:18:59.686203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.257065 (+1570862us) service_pool.cc:224] Handling call
0709 12:19:01.260380 (+  3315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:01.316891 28534 raft_consensus.cc:2764] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election decision vote started in defunct term 44: won
I20260709 12:19:01.321899 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.324659 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4604) took 1639 ms. Trace:
I20260709 12:19:01.325313 12496 rpcz_store.cc:276] 0709 12:18:59.685128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.685376 (+   248us) service_pool.cc:224] Handling call
0709 12:18:59.685816 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.878869 (+193053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.324019 (+1445150us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:01.324430 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170509184}
I20260709 12:19:01.326912 28689 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:01.327800 28689 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:19:01.331521 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4605) took 1645 ms. Trace:
I20260709 12:19:01.332041 12493 rpcz_store.cc:276] 0709 12:18:59.686246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.686384 (+   138us) service_pool.cc:224] Handling call
0709 12:18:59.686681 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:18:59.959422 (+272741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.959460 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:18:59.959517 (+    57us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:18:59.959566 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.330932 (+1371366us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b540012b6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:01.331014 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.331317 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25920,"raft.queue_time_us":109280,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":8616,"spinlock_wait_cycles":588107008}
I20260709 12:19:01.337779 28752 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.342756 28502 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.347136 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.348443 12488 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 52: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:01.351096 13255 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: [46.33-46.33]
I20260709 12:19:01.274995 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:01.367945 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2016) took 1624 ms. Trace:
I20260709 12:19:01.368425 13255 rpcz_store.cc:276] 0709 12:18:59.738498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:01.307946 (+1569448us) service_pool.cc:224] Handling call
0709 12:19:01.308576 (+   630us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:01.362945 (+ 54369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.363241 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:01.372300 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2037) took 1749 ms (client timeout 1958 ms). Trace:
I20260709 12:19:01.372604 12568 raft_consensus.cc:1275] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:19:01.373096 12565 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: [46.33-46.33]
I20260709 12:19:01.373564 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.373672 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:01.372830 13259 rpcz_store.cc:269] 0709 12:18:59.618895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.895650 (+276755us) service_pool.cc:224] Handling call
0709 12:19:01.349384 (+1453734us) spinlock_profiling.cc:243] Waited 4.47 ms on lock 0x7b540058e654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:01.368031 (+ 18647us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10279808}
W20260709 12:19:01.379076 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64688.4us
I20260709 12:19:01.382997 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.381683 12496 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 40. Current term is 41. Ops: [40.23-40.23]
W20260709 12:19:01.376121 13254 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2012) took 1756 ms (client timeout 1601 ms). Trace:
I20260709 12:19:01.378751 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:01.391615 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.393536 12644 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 42 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
W20260709 12:19:01.390394 13254 rpcz_store.cc:269] 0709 12:18:59.619112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.670564 (+ 51452us) service_pool.cc:224] Handling call
0709 12:19:01.368138 (+1697574us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:01.375920 (+  7782us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2472907008}
I20260709 12:19:01.376742 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2013) took 1757 ms. Trace:
W20260709 12:19:01.394047 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123681us
I20260709 12:19:01.395902 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.397883 28704 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:19:01.398272 13261 rpcz_store.cc:276] 0709 12:18:59.619401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.592410 (+973009us) service_pool.cc:224] Handling call
0709 12:19:01.376436 (+784026us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b5400579754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:01.376646 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":866900736}
I20260709 12:19:01.370914 28689 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.376957 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.410689 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.407073 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.410125 12488 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:01.412878 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.412809 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:01.416085 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 191804us
I20260709 12:19:01.421236 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:01.419664 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:01.428557 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:01.428201 28003 raft_consensus.cc:359] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 40 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:01.433557 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:01.434406 28627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.107s	user 0.038s	sys 0.078s
W20260709 12:19:01.435267 28627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.179s	user 0.038s	sys 0.078s
I20260709 12:19:01.435688 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:01.437021 28608 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.649s	user 0.010s	sys 0.010s
I20260709 12:19:01.437916 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.438294 12494 raft_consensus.cc:2410] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 46 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
I20260709 12:19:01.432518 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:01.407289 28507 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.420600 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.448904 12572 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:19:01.450826 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:01.451735 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.451887 12572 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:01.452997 28430 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:01.438607 12493 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:01.454120 28544 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:01.454319 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.455087 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.454968 28544 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.455469 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.456885 12568 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:19:01.426397 28725 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:01.458046 28725 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.458442 28725 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:01.462841 28430 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.466640 13253 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2034) took 1868 ms. Trace:
I20260709 12:19:01.467334 13253 rpcz_store.cc:276] 0709 12:18:59.598103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:59.598433 (+   330us) service_pool.cc:224] Handling call
0709 12:18:59.598809 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:18:59.659896 (+ 61087us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:18:59.659960 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:00.597656 (+937696us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b48000f00d8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:19:01.270630 (+672974us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b2c00007be8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:19:01.271521 (+   891us) log.cc:844] Serialized 26 byte log entry
0709 12:19:01.371809 (+100288us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:01.371855 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:01.385080 (+ 13225us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:01.465938 (+ 80858us) raft_consensus.cc:1685] finished
0709 12:19:01.465967 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:01.466318 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86485,"raft.run_cpu_time_us":22702,"raft.run_wall_time_us":922212,"spinlock_wait_cycles":7604992,"thread_start_us":141862,"threads_started":2,"wal-append.queue_time_us":133763}
I20260709 12:19:01.457034 28601 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:01.470937 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72260.3us
I20260709 12:19:01.449308 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.455569 12492 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:01.438668 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.453413 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:01.411036 28718 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:19:01.488103 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 109556us
I20260709 12:19:01.486642 12488 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
W20260709 12:19:01.489782 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 297751us
I20260709 12:19:01.490557 12563 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:01.443830 12496 raft_consensus.cc:2376] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46. Re-sending same reply.
I20260709 12:19:01.433413 28003 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 40 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:19:01.437417 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:01.497347 28430 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:01.516825 28488 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.535s	user 0.008s	sys 0.011s
I20260709 12:19:01.498432 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.499200 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.499287 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.500139 28507 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.501251 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.521567 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:01.501863 28534 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:01.522318 12569 raft_consensus.cc:3060] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:01.504456 28703 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:01.477865 28608 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.506950 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:01.507548 13260 raft_consensus.cc:2410] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 54 because replica has last-logged OpId of term: 52 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 29.
I20260709 12:19:01.511871 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:01.497973 28705 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:19:01.530517 12565 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:01.520771 12566 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:01.523722 28534 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:01.524238 28721 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:01.525838 28137 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.005s
I20260709 12:19:01.529565 12492 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:19:01.529582 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107741us
I20260709 12:19:01.514448 28627 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:01.503458 28644 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.528985 28703 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.430373 28689 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.552075 28741 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:19:01.554054 12495 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 56: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:01.570555 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:01.572016 12571 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:01.553532 28608 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:01.536262 28534 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:01.577732 12565 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 48 index: 29. (index mismatch)
I20260709 12:19:01.536260 28544 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.575022 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:01.579294 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56176.2us
I20260709 12:19:01.576900 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:01.584160 12564 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:01.558758 28627 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.570067 28703 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:01.585330 12491 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 55: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:01.530040 28715 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.058s
I20260709 12:19:01.587844 28507 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.588526 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:01.583892 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170629us
I20260709 12:19:01.583829 12566 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 51 index: 30. (index mismatch)
I20260709 12:19:01.572067 28741 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:01.589406 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 92473.6us
W20260709 12:19:01.593590 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 202293us
I20260709 12:19:01.589545 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:01.592334 28627 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:01.624795 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:01.589421 28502 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260709 12:19:01.653730 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:01.638280 28488 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:01.640131 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 128475us
I20260709 12:19:01.620049 28741 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:01.634407 12492 raft_consensus.cc:2376] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52. Re-sending same reply.
I20260709 12:19:01.648099 12491 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:19:01.650038 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:19:01.651437 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.652144 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:01.652952 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:01.653687 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:01.654923 12496 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 23. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
W20260709 12:19:01.656253 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 82093.5us
I20260709 12:19:01.658270 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2017) took 1587 ms. Trace:
I20260709 12:19:01.661738 28608 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:01.662917 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72389.5us
I20260709 12:19:01.569984 28003 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:01.694489 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:01.697755 28685 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:01.699656 13254 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:01.663132 28738 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:19:01.707872 28757 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
W20260709 12:19:01.680306 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46928.1us
I20260709 12:19:01.679539 28690 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:01.704404 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 78095.6us
I20260709 12:19:01.689874 13255 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:01.691634 13260 rpcz_store.cc:276] 0709 12:19:00.071082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:00.792252 (+721170us) service_pool.cc:224] Handling call
0709 12:19:01.657945 (+865693us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b5400113fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:01.658182 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1572610816}
W20260709 12:19:01.713048 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 152701us
I20260709 12:19:01.640816 28718 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:01.723961 13253 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:01.746814 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112549us
I20260709 12:19:01.752899 12564 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 45 index: 26. (term mismatch)
I20260709 12:19:01.672979 28715 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
W20260709 12:19:01.752769 28686 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.636s	user 0.001s	sys 0.014s
I20260709 12:19:01.759284 12568 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 53: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:01.762136 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:01.763948 12643 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.005s
I20260709 12:19:01.765470 12461 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:01.772861 12643 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
W20260709 12:19:01.758319 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 187643us
I20260709 12:19:01.766448 12564 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 25
I20260709 12:19:01.774143 12564 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 44.26
I20260709 12:19:01.774530 12564 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:01.762045 28685 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:01.788940 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70595.7us
I20260709 12:19:01.768649 28715 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:01.792912 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:01.788208 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:01.810685 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:01.788069 28690 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:01.770543 28608 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.792253 28685 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:01.812479 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:01.797189 28715 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:01.792936 28718 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:01.825016 28308 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:19:01.827044 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.839620 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:01.839075 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:01.810601 28644 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:01.826640 28683 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.794s	user 0.014s	sys 0.002s
I20260709 12:19:01.821583 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:01.861076 28717 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:01.861682 28717 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:01.862764 28717 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 46.33, Last appended by leader: 33, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.864334 28717 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:01.864221 28534 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.866387 28003 ts_tablet_manager.cc:1434] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.728s	user 0.017s	sys 0.001s
W20260709 12:19:01.866835 28003 ts_tablet_manager.cc:1467] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Tablet startup took 728ms
W20260709 12:19:01.869921 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48499.4us
W20260709 12:19:01.878162 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102150us
I20260709 12:19:01.884804 28718 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:01.885536 28718 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:19:01.886390 28761 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:01.886996 28761 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:01.887960 28761 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.888309 28718 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:19:01.888984 28718 consensus_queue.cc:1243] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:19:01.843546 28690 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:01.893925 28690 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.894358 28690 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:01.896076 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:01.898406 28760 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:01.843981 28601 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:01.915103 28601 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:01.916365 28601 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 45.32, Last appended by leader: 32, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.899398 13253 raft_consensus.cc:2368] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
I20260709 12:19:01.823977 28689 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:01.924342 28741 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:01.926416 12569 raft_consensus.cc:2468] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 55.
I20260709 12:19:01.870617 28683 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.929847 28683 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:01.930678 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 138) took 30144 ms (client timeout 15000 ms). Trace:
W20260709 12:19:01.934360 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 179351us
I20260709 12:19:01.934643 13217 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:01.935104 28686 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:01.935674 28686 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.936064 28686 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:01.936439 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:01.937973 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.708s (SENT)
W20260709 12:19:01.939684 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 150022us
I20260709 12:19:01.940653 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:01.941869 28137 raft_consensus.cc:2764] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election decision vote started in defunct term 46: won
I20260709 12:19:01.942647 28137 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:01.943117 28137 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:01.945619 28468 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:01.946141 28468 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:01.946511 28468 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:01.950516 28137 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.29, Last appended by leader: 29, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:01.952548 28725 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:01.954483 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58505.3us
I20260709 12:19:01.955708 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:01.957056 28724 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:01.958313 28741 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:01.958734 28724 consensus_queue.cc:1048] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:19:01.959120 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 405351us
I20260709 12:19:01.960155 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:01.960457 28724 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20260709 12:19:01.907027 12537 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:01.908521 27833 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:01.897542 28752 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.012s
I20260709 12:19:01.897473 28125 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.012s
I20260709 12:19:01.909339 28760 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:01.930863 13261 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
W20260709 12:19:01.867203 28003 ts_tablet_manager.cc:1469] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Trace:
0709 12:18:40.335425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:40.339993 (+  4568us) service_pool.cc:224] Handling call
0709 12:18:40.687814 (+347821us) ts_tablet_manager.cc:934] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
0709 12:18:45.441051 (+4753237us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:45.964801 (+523750us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:45.964933 (+   132us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:46.185303 (+220370us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:46.202498 (+ 17195us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:46.202920 (+   422us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:46.202997 (+    77us) inbound_call.cc:177] Queueing success response
0709 12:18:51.908127 (+5705130us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:19:01.138167 (+9230040us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:19:01.155318 (+ 17151us) tablet_replica.cc:261] Starting instrumentation
0709 12:19:01.156306 (+   988us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":12801,"raft.queue_time_us":19623,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":10791,"tablet-copy.queue_time_us":250098,"tablet-download-pool.queue_time_us":188,"tablet-download-pool.run_cpu_time_us":2016,"tablet-download-pool.run_wall_time_us":4654053,"thread_start_us":278204,"threads_started":2}
I20260709 12:19:01.908847 28608 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:01.827430 28055 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.038s	user 0.008s	sys 0.010s
W20260709 12:19:01.961930 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55216us
W20260709 12:19:01.964166 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:01.972281 12563 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:01.933048 28703 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:02.046686 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85021.8us
W20260709 12:19:01.984851 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 152234us
W20260709 12:19:02.047217 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 215382us
I20260709 12:19:02.052258 12497 raft_consensus.cc:2368] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48.
I20260709 12:19:01.926416 28534 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:01.968655 28683 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:02.068130 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:02.071794 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 131260us
W20260709 12:19:02.072125 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87450.4us
W20260709 12:19:01.931273 12386 rpcz_store.cc:269] 0709 12:18:31.785774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:32.840998 (+1055224us) service_pool.cc:224] Handling call
0709 12:19:01.930385 (+29089387us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:18:51.621516 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:18:51.622298 (+   782us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:18:51.622351 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:18:51.622368 (+    17us) tablet.cc:662] Decoding operations
0709 12:18:51.624166 (+  1798us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:18:51.624266 (+   100us) write_op.cc:641] Partition lock acquired for write op
0709 12:18:51.624313 (+    47us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:18:51.690309 (+ 65996us) tablet.cc:701] Row locks acquired
0709 12:18:51.690388 (+    79us) write_op.cc:260] PREPARE: finished
0709 12:18:51.690671 (+   283us) write_op.cc:270] Start()
0709 12:18:51.690847 (+   176us) write_op.cc:276] Timestamp: P: 1783599531690575 usec, L: 0
0709 12:18:51.690880 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 12:18:51.691687 (+   807us) log.cc:844] Serialized 6282 byte log entry
0709 12:18:52.891473 (+1199786us) op_driver.cc:464] REPLICATION: finished
0709 12:18:53.385266 (+493793us) write_op.cc:301] APPLY: starting
0709 12:18:53.385368 (+   102us) tablet.cc:1367] starting BulkCheckPresence
0709 12:18:53.649235 (+263867us) tablet.cc:1370] finished BulkCheckPresence
0709 12:18:53.649383 (+   148us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:18:53.664937 (+ 15554us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:18:53.665342 (+   405us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:18:53.665387 (+    45us) write_op.cc:312] APPLY: finished
0709 12:18:53.666240 (+   853us) log.cc:844] Serialized 148 byte log entry
0709 12:18:54.143136 (+476896us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:54.166232 (+ 23096us) write_op.cc:454] Released schema lock
0709 12:18:54.217074 (+ 50842us) write_op.cc:341] FINISH: Updating metrics
0709 12:18:54.323771 (+106697us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:18:54.333176 (+  9405us) write_op.cc:454] Released schema lock
0709 12:18:55.146543 (+813367us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
Metrics: {"mutex_wait_us":4247747,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":401688,"apply.run_cpu_time_us":82112,"apply.run_wall_time_us":904420,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70943,"num_ops":16,"prepare.queue_time_us":45413,"prepare.run_cpu_time_us":30111,"prepare.run_wall_time_us":365698,"raft.queue_time_us":454564,"raft.run_cpu_time_us":39394,"raft.run_wall_time_us":458930,"replication_time_us":1148201,"spinlock_wait_cycles":3664256,"thread_start_us":1078163,"threads_started":5,"wal-append.queue_time_us":880773,"wal-append.run_cpu_time_us":39409,"wal-append.run_wall_time_us":3916737}]]}
I20260709 12:19:01.991477 28741 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:19:01.985652 28308 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.002s	sys 0.001s
I20260709 12:19:02.080513 13260 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 52: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:01.985457 28689 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:02.100492 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:02.105794 28718 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:02.107425 28723 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:02.112079 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.113004 12568 raft_consensus.cc:2393] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 55: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:02.113859 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:02.114367 28774 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.096s	user 0.004s	sys 0.004s
I20260709 12:19:02.114972 28772 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:19:02.111951 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:02.127862 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43627.1us
I20260709 12:19:02.129727 12563 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:19:02.133479 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.139460 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:02.142303 28539 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.905s	user 0.015s	sys 0.001s
I20260709 12:19:02.142547 28773 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:02.143028 28724 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.014s
W20260709 12:19:02.144001 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.130594 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:02.084244 28689 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.153522 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:02.157198 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:02.158962 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 202990us
W20260709 12:19:02.173559 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 108128us
I20260709 12:19:02.150604 28717 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.066057 28055 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.157106 28741 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.158201 28689 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:02.177443 28539 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:02.226711 28703 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.227105 28539 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.205016 12643 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81a9a92616494f46bc97254b53185664"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:19:02.188144 28723 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.230556 28723 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.232880 28723 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:02.171013 28627 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.235281 28627 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.236152 12489 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 54. Current term is 55.
I20260709 12:19:02.206858 28687 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:02.210461 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80202.1us
I20260709 12:19:02.211014 28781 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 40.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:02.192409 28683 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:19:02.212565 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83646.6us
I20260709 12:19:02.209084 28690 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.216641 28685 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.217664 28724 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.218287 28725 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.218520 28125 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:02.219406 28502 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.219628 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:02.220770 28738 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:19:02.187048 28717 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.262880 28724 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.253018 28683 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.225772 28534 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:02.225891 28715 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.237985 28761 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:02.211666 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:02.229410 28539 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.280300 28539 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:02.281807 28761 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:02.281401 12394 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 28 to 43. New cstate: current_term: 43 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:02.282106 28502 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.282931 28761 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.284323 28761 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:02.284588 28502 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.276172 28683 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:02.286103 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 74563.2us
I20260709 12:19:02.286950 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:02.289829 28710 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.290436 28773 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:02.290892 28685 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.283803 12394 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 40 to 44, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 44 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:02.290448 28710 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.294085 12394 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 46 to 47, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 47 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:02.296595 28710 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.296378 12394 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 39 to 40, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:02.297549 28724 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.297956 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:02.299434 28703 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:02.299925 28703 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:02.300325 28703 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:02.301009 13217 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:02.251039 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83094us
I20260709 12:19:02.253608 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:02.298885 12394 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 38 to 42. New cstate: current_term: 42 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:02.302933 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49487.4us
I20260709 12:19:02.304086 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:02.303846 12394 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 35 to 40, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 40 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:02.305939 28720 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:02.213783 28776 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:02.238608 28721 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.678s	user 0.012s	sys 0.000s
I20260709 12:19:02.260778 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:02.311252 28776 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.312227 12493 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:02.289294 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:02.314980 28738 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:02.315407 28738 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:02.315841 28738 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:02.306591 12394 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 34 to 40, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 40 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:02.257488 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:02.260134 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:02.263087 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:02.263083 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59518us
I20260709 12:19:02.193560 28741 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.182472 28468 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.219637 28777 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 51.29, Next index: 30, Last known committed idx: 27, Time since last communication: 0.043s
I20260709 12:19:02.307763 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:02.213475 28605 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.048s
I20260709 12:19:02.263314 28125 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:02.269662 28717 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:02.278546 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 59026.5us
I20260709 12:19:02.279199 28488 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:02.309479 12488 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:02.318537 12491 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:02.245560 28686 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.306921 28720 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:19:02.265789 28687 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.310745 28715 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:02.326314 28776 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.326632 12489 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:19:02.332541 12611 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:19:02.317432 12394 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 36 to 43, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 43 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:02.274484 28055 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.333613 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.338661 12643 raft_consensus.cc:606] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:02.383057 12643 raft_consensus.cc:614] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:02.334981 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:02.350404 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.333905 28430 raft_consensus.cc:2804] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:02.388083 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:02.334975 28780 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:19:02.367388 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.334690 28690 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.334723 28718 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:02.380348 12572 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:02.338725 28721 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:02.336313 28741 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:02.390957 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56111.9us
I20260709 12:19:02.387063 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.408883 12567 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:02.355114 28488 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:02.394583 12565 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:02.336766 28689 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.396277 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:02.425834 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.405273 12493 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:19:02.424311 13260 raft_consensus.cc:1240] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 51. Ops: []
W20260709 12:19:02.426959 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.398203 28776 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.045s
I20260709 12:19:02.392148 28780 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:02.397290 28705 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.407243 28721 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.426295 28488 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:02.497242 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:02.503597 13253 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:02.503114 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.513600 12565 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 46 because replica has last-logged OpId of term: 43 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 24.
I20260709 12:19:02.514681 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.515672 12572 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:02.505980 28715 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.448578 28779 raft_consensus.cc:3055] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:02.456425 28721 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:02.522046 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.522822 12564 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:02.523804 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.455540 28601 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:02.528731 28786 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.528520 28715 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.530952 12460 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:02.540982 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:02.531581 28685 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:19:02.532536 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 224427us
W20260709 12:19:02.541260 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 115207us
I20260709 12:19:02.542968 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:02.548704 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:02.538293 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142176us
I20260709 12:19:02.500710 28781 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:19:02.536763 12394 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 25 to 43. New cstate: current_term: 43 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:02.526021 28687 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.525754 28779 raft_consensus.cc:740] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:02.542595 28605 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:02.570838 28605 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.571285 13261 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 28. Preceding OpId from leader: term: 46 index: 29. (index mismatch)
I20260709 12:19:02.571528 28605 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:02.564232 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:02.573122 13260 raft_consensus.cc:2376] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46. Re-sending same reply.
I20260709 12:19:02.458107 28705 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:02.574616 28787 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:19:02.576262 28705 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.579181 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:02.580096 12493 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:02.597787 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 195865us
I20260709 12:19:02.586243 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:02.577457 28705 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:02.448935 28125 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.30, Last appended by leader: 30, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.587972 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:02.588241 28686 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.590817 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:02.627295 13255 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 53 because replica has last-logged OpId of term: 51 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 28.
I20260709 12:19:02.592448 28752 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:19:02.586938 13257 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 28. Preceding OpId from leader: term: 44 index: 30. (index mismatch)
I20260709 12:19:02.545598 28717 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:02.551630 28468 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.639640 13261 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:02.571808 28724 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:02.559942 28781 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:02.659282 28781 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:19:02.664662 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:02.596621 28780 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.619362 28705 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.669390 12645 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:02.698237 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.701741 12570 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:19:02.692890 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.666137 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.706424 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:02.544986 28786 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:02.624835 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.631589 12537 leader_election.cc:400] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
W20260709 12:19:02.626024 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 287578us
I20260709 12:19:02.639416 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:02.713755 12537 leader_election.cc:403] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Cancelling election due to peer responding with higher term
I20260709 12:19:02.596366 28539 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:02.650291 28796 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:02.650782 13262 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:02.650866 12491 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:02.651376 28797 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 51.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:02.651993 12639 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: []
I20260709 12:19:02.653906 12394 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 32 to 40. New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:02.658461 28724 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:02.641039 28685 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:02.546288 28601 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.627920 28534 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:19:02.705900 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:02.708215 28705 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:02.632190 28752 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:02.734974 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:02.717883 13260 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:02.718181 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 160009us
I20260709 12:19:02.649273 12493 raft_consensus.cc:2468] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:19:02.728521 12646 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:02.681124 28784 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:19:02.669850 28773 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:19:02.747901 12641 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 51 index: 30. (term mismatch)
I20260709 12:19:02.666256 28687 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:02.749186 28539 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.635165 28717 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.750799 28717 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:02.732904 12640 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a1d35de928f546349f88cbea7818041c"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:19:02.712630 28430 raft_consensus.cc:697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:02.743728 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 118226us
I20260709 12:19:02.743157 28784 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:02.748420 12641 pending_rounds.cc:85] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:19:02.733374 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.752156 12640 raft_consensus.cc:606] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:02.756878 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.762822 12641 pending_rounds.cc:107] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 50.30
I20260709 12:19:02.763446 12641 raft_consensus.cc:2889] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:02.742543 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:02.780393 12567 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:02.739182 28710 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:02.785914 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.779295 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:02.788719 12569 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 48
I20260709 12:19:02.790019 28725 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:02.790946 12564 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 32. Preceding OpId from leader: term: 47 index: 33. (index mismatch)
I20260709 12:19:02.793566 13260 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:02.795367 28784 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.796541 28784 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:02.719261 28683 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.799834 28683 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:02.800114 28782 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election lost for term 54. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
I20260709 12:19:02.778246 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:02.802250 28539 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:02.793243 12394 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 37 to 42, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 42 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:02.783149 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:02.816068 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.816985 12563 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:02.805910 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63635.1us
W20260709 12:19:02.765661 12537 connection.cc:256] RPC call timeout handler was delayed by -1.0528395050000654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:19:02.804456 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:02.831022 12536 connection.cc:256] RPC call timeout handler was delayed by -3.1171894760000214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
W20260709 12:19:02.831733 12536 connection.cc:256] RPC call timeout handler was delayed by -3.104589476000001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
W20260709 12:19:02.844322 12536 connection.cc:256] RPC call timeout handler was delayed by -3.0064894760000698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260709 12:19:02.802269 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:02.799764 28780 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:02.847087 12642 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 45 index: 25. (index mismatch)
I20260709 12:19:02.734720 28685 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.847947 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.850981 12567 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:02.842947 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:02.851794 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:02.845316 12641 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:19:02.845907 12645 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 56
I20260709 12:19:02.799678 28689 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:02.853137 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84360.9us
I20260709 12:19:02.853765 12637 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:02.754917 28720 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.073s
I20260709 12:19:02.854104 12638 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 46 index: 29. (index mismatch)
I20260709 12:19:02.667524 28779 consensus_queue.cc:260] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.788506 28710 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:02.730154 28796 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.869560 12563 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:02.857152 12569 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:02.861197 28804 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:02.786208 28534 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.073s
I20260709 12:19:02.882578 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.851315 28686 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:02.883234 12563 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:19:02.871805 12536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.517s (SENT) [suppressed 23 similar messages]
I20260709 12:19:02.884019 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.887471 28125 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:02.890117 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:02.891345 28800 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:19:02.892038 28752 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.884963 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:02.900136 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.914129 12642 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:02.900220 12569 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:02.876562 12394 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 35 to 39, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 39 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:02.891367 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:02.871469 28797 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:02.871196 28710 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:02.910063 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:02.923449 12638 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:02.902341 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43548us
I20260709 12:19:02.918676 28715 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.877177 28773 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:02.925354 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:02.933992 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:02.935436 12394 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 36 to 45. New cstate: current_term: 45 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:02.938005 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.948127 12641 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:02.938392 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:02.939776 28752 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:02.940153 28721 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:02.940845 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:02.941184 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:02.957968 12643 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:19:02.941936 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 85236.3us
I20260709 12:19:02.961093 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:02.962020 28715 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:02.943038 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:02.962960 28715 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:19:02.964048 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:02.965023 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:02.943213 28685 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:02.904663 28804 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:02.967053 28804 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 44.31, Last appended by leader: 31, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.969185 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:02.943730 28430 consensus_queue.cc:237] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 53.31, Last appended by leader: 31, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:02.947469 12567 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:02.985345 12638 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:02.950335 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:19:02.955360 12564 pending_rounds.cc:85] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 21
I20260709 12:19:02.986642 12564 pending_rounds.cc:107] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.22
I20260709 12:19:02.914057 28686 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:02.919963 28797 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:02.988504 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:02.988665 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 116887us
I20260709 12:19:02.987849 12564 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:02.992230 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 166145us
W20260709 12:19:02.951965 28761 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.667s	user 0.012s	sys 0.002s
I20260709 12:19:02.885039 28468 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.068s
I20260709 12:19:02.994681 12610 leader_election.cc:400] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 54. Current term is 55.
I20260709 12:19:03.003247 12610 leader_election.cc:403] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Cancelling election due to peer responding with higher term
I20260709 12:19:02.994436 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:02.999523 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:02.940518 28488 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:02.988073 28800 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:19:03.006019 12571 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:02.992677 28703 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.002s
I20260709 12:19:02.984644 28773 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:03.023758 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:03.044620 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:03.045584 12638 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:03.000150 28797 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:02.945200 28724 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 52.23, Last appended by leader: 23, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.052829 28812 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:03.057818 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 83287.4us
I20260709 12:19:03.060715 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:03.062979 12645 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
I20260709 12:19:03.057449 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:02.991243 28725 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.058s
I20260709 12:19:03.052376 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:03.033913 28810 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election won for term 48
W20260709 12:19:03.084995 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61307.4us
I20260709 12:19:03.035084 28811 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.077838 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:03.078799 12460 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:03.076452 28601 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.075701 12639 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:03.088265 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:03.096695 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:03.097380 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.097954 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.098968 12645 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:03.096731 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:03.100307 12569 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:03.104535 28601 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:03.105230 28601 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:03.125357 12488 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 33. Preceding OpId from leader: term: 47 index: 33. (term mismatch)
I20260709 12:19:03.117264 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:03.127928 12640 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:19:03.128746 12640 raft_consensus.cc:606] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:03.129141 12640 raft_consensus.cc:614] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:03.038362 28703 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.073525 28815 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.133468 12493 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:19:03.130227 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:03.134259 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7679894759999115s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
I20260709 12:19:03.125998 12488 pending_rounds.cc:85] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 32
I20260709 12:19:03.141450 12488 pending_rounds.cc:107] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 46.33
W20260709 12:19:03.141045 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7382894760000909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
I20260709 12:19:03.141775 12488 raft_consensus.cc:2889] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:03.141850 12536 connection.cc:256] RPC call timeout handler was delayed by -2.7202894760000618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2154s for a response
W20260709 12:19:03.142185 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6563894759999584s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
I20260709 12:19:03.069047 28786 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:03.093039 28689 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
W20260709 12:19:03.151362 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:03.142745 12536 connection.cc:256] RPC call timeout handler was delayed by -1.5213294060001772s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20260709 12:19:03.093668 28761 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:03.152079 12536 connection.cc:256] RPC call timeout handler was delayed by -1.4421294060000491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
W20260709 12:19:03.154371 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2747294060000058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 12:19:03.154856 12536 connection.cc:256] RPC call timeout handler was delayed by -1.2513294060001954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
W20260709 12:19:03.155342 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0470294060000924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23450000000000004s for a response
W20260709 12:19:03.155082 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177183us
I20260709 12:19:03.156746 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:03.094162 28685 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:03.203603 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:03.207576 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 251369us
I20260709 12:19:03.211625 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.212539 12639 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:03.215765 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:03.076808 28812 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.226092 12488 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 44. Current term is 45. Ops: [43.25-44.26]
I20260709 12:19:03.146111 28137 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:03.218353 28689 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:03.236799 13218 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:03.155440 28786 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:03.239142 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.158236 28725 consensus_queue.cc:1243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 46.27 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 26
I20260709 12:19:03.154448 28715 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:03.242334 12610 leader_election.cc:400] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48.
I20260709 12:19:03.265844 12610 leader_election.cc:403] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Cancelling election due to peer responding with higher term
I20260709 12:19:03.245271 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.237814 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.266588 12638 raft_consensus.cc:2368] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48.
I20260709 12:19:03.267645 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:03.268471 28803 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:03.269047 28803 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:03.269305 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:03.269855 28308 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:19:03.270139 12639 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:03.270895 28488 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:03.271370 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.272697 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.274642 28468 raft_consensus.cc:2804] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:03.275224 28468 raft_consensus.cc:697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:03.276144 28468 consensus_queue.cc:237] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 49.30, Last appended by leader: 30, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.277897 12497 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 48. Ops: []
I20260709 12:19:03.279294 28703 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.280359 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:03.138582 28686 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 53.26, Last appended by leader: 26, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.283875 28812 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.284186 28796 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:03.281406 28819 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:19:03.247196 12493 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:03.255918 28810 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:03.110844 28811 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.201913 28539 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:03.260326 12489 raft_consensus.cc:2368] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:03.262054 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:03.268496 12638 raft_consensus.cc:2368] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:19:03.289757 12641 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:03.268898 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:03.123167 28601 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 50.36, Last appended by leader: 36, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.333491 12645 raft_consensus.cc:2376] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47. Re-sending same reply.
I20260709 12:19:03.313323 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.315671 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.338510 12646 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:03.201092 28761 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.311522 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:03.334729 12572 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 30. Preceding OpId from leader: term: 44 index: 30. (term mismatch)
I20260709 12:19:03.334934 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:03.342628 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:03.334945 12567 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:03.284819 28125 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:03.335877 12643 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:03.345808 28539 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.346848 28308 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=22 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:19:03.151789 28815 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.149967 28773 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.23, Last appended by leader: 23, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.221695 28685 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.358577 12642 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:03.371452 12489 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 42. Current term is 43. Ops: [42.23-42.23]
W20260709 12:19:03.370434 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116585us
I20260709 12:19:03.358086 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:03.373049 12645 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 39. Current term is 40. Ops: []
W20260709 12:19:03.360853 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 119087us
I20260709 12:19:03.342103 12572 pending_rounds.cc:85] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 29
I20260709 12:19:03.312525 28721 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.360015 12646 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:19:03.369361 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:03.373733 12572 pending_rounds.cc:107] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.30
I20260709 12:19:03.246738 28717 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.374178 12572 raft_consensus.cc:2889] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:03.375631 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155404us
I20260709 12:19:03.270048 28803 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.28, Last appended by leader: 28, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.350289 28125 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:03.349985 28796 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 47. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48.
I20260709 12:19:03.379166 28830 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:03.380751 28738 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.064s	user 0.009s	sys 0.006s
I20260709 12:19:03.414710 28738 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:03.381006 28781 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.721s	user 0.011s	sys 0.001s
I20260709 12:19:03.418015 28781 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.420398 28781 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.381245 28539 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:03.421643 12563 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
I20260709 12:19:03.335448 28812 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 53.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.060s
I20260709 12:19:03.380631 28810 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 46.30, Last appended by leader: 30, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.317991 28715 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:03.383101 12639 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52. Ops: [51.22-51.22]
I20260709 12:19:03.395709 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:03.265105 28786 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.351573 28720 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election lost for term 54. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 54. Current term is 55.
I20260709 12:19:03.402356 28721 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:19:03.481915 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.320447 28785 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.076s
I20260709 12:19:03.405421 28717 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.252750 28137 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.340859 28725 consensus_queue.cc:1059] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.059s
I20260709 12:19:03.417317 28738 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:03.440894 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 362150us
I20260709 12:19:03.444607 12488 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:19:03.445549 12493 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:03.446063 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.447072 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.448175 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.451267 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:03.453490 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:03.517795 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.445588 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:03.467082 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.433596 12646 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "6d4ece942fd5414998215d26edf91001"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:19:03.522078 12646 raft_consensus.cc:606] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Received request to transfer leadership to TS 6d4ece942fd5414998215d26edf91001
I20260709 12:19:03.522624 12646 raft_consensus.cc:614] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:03.433106 12609 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:03.480942 12491 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:03.473908 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.412372 28125 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:03.472316 12567 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 54.
W20260709 12:19:03.437791 28687 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.007s	sys 0.011s
I20260709 12:19:03.487643 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:03.491648 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132367us
I20260709 12:19:03.523430 12491 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:03.523587 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 90578.5us
I20260709 12:19:03.524766 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:03.532670 12490 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:03.538807 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.541055 12570 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 54: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:03.542644 12494 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:03.542896 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:03.544606 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:03.544804 28687 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.007s	sys 0.011s
I20260709 12:19:03.550894 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.551671 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.559217 13262 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 40 LEARNER]: Advancing to term 41
I20260709 12:19:03.551945 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:03.552691 13256 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 53. Current term is 54. Ops: []
I20260709 12:19:03.555265 12493 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 28. Preceding OpId from leader: term: 52 index: 29. (index mismatch)
I20260709 12:19:03.542037 12491 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.473143 28781 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.071s
I20260709 12:19:03.551623 12489 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:03.565826 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:03.567387 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.593497 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.570955 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.570676 12491 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:03.539968 28832 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:03.578312 12568 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 53. Current term is 54. Ops: []
I20260709 12:19:03.578578 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:03.582262 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73931.2us
I20260709 12:19:03.596331 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:03.509482 28785 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.075s
I20260709 12:19:03.551110 28812 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:19:03.567011 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:03.565989 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:03.573982 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.551736 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:03.602744 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:03.574280 28705 consensus_meta.cc:287] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.819s	user 0.013s	sys 0.000s
W20260709 12:19:03.616271 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 142987us
I20260709 12:19:03.626621 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:03.488168 28689 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 46.24, Last appended by leader: 24, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.626960 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.630080 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:03.635468 12493 raft_consensus.cc:2393] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 52: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:03.632200 12571 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.598s	user 0.005s	sys 0.005s
I20260709 12:19:03.578851 28801 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:03.625573 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:03.633798 12491 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 46 index: 27. (term mismatch)
I20260709 12:19:03.578723 28802 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.580067 28781 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:03.662580 12491 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 26
I20260709 12:19:03.664151 12491 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 45.27
I20260709 12:19:03.607236 28832 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:03.556344 28738 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.586598 28137 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.664701 12491 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:03.670176 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.682039 12494 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:03.610627 28812 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:03.586294 28534 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 53.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.046s
I20260709 12:19:03.636317 12571 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:03.688094 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.688467 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.625311 28836 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:03.693468 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63857.1us
I20260709 12:19:03.731403 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:03.695928 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.689287 12488 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:19:03.664469 28811 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:03.736078 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170206us
W20260709 12:19:03.684895 28784 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.888s	user 0.012s	sys 0.004s
I20260709 12:19:03.636967 28703 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.663582 28430 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:03.837790 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:03.789129 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57880.5us
I20260709 12:19:03.667685 28781 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:03.741123 12497 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:03.633307 28725 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:03.668546 28801 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:03.694840 28823 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.332s	user 0.001s	sys 0.001s
W20260709 12:19:03.829596 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:03.637595 28833 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:19:03.733095 28738 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.787631 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.688066 28761 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.843266 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:03.660548 28827 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.131s
I20260709 12:19:03.829977 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:03.694605 28137 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.841143 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:03.686555 28832 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:03.637394 28705 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:03.853886 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 237792us
W20260709 12:19:03.862672 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 295721us
I20260709 12:19:03.671064 28802 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.800882 28796 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:19:03.787228 28488 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:03.876818 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:03.624913 28715 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.860500 28801 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:03.854135 28725 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:03.838272 28430 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:03.860244 28685 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.865556 28833 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:03.895349 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:03.906651 12494 raft_consensus.cc:2368] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:19:03.895177 28830 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:03.838688 28683 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.045s
I20260709 12:19:03.856848 28703 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:03.913954 28605 consensus_meta.cc:287] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.341s	user 0.015s	sys 0.003s
I20260709 12:19:03.884476 28815 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.890854 28786 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.894910 28725 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:03.904244 28430 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:03.903718 28761 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:03.829090 28836 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:03.935019 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68225.6us
W20260709 12:19:03.938292 28821 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.473s	user 0.002s	sys 0.000s
I20260709 12:19:03.938680 28842 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:03.939086 28838 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
W20260709 12:19:03.939549 28820 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.446s	user 0.002s	sys 0.000s
I20260709 12:19:03.939998 28850 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:03.940371 28790 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:03.911677 28833 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:03.948854 28705 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.950805 28703 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:03.903743 28137 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:03.951365 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6375) took 1618 ms. Trace:
I20260709 12:19:03.959127 28738 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:03.959687 28836 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:03.961210 28780 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.106s	user 0.010s	sys 0.003s
I20260709 12:19:03.915162 28830 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:03.962630 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:03.965968 28125 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.872212 28812 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 53.28, Last appended by leader: 28, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.936394 28308 tablet_bootstrap.cc:492] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
W20260709 12:19:03.969019 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 95505.2us
W20260709 12:19:03.936394 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 135934us
I20260709 12:19:03.889950 28488 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:03.974189 28854 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:03.974468 28802 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.101384 12563 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:03.977821 12492 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:03.983522 12571 rpcz_store.cc:276] 0709 12:19:02.332558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.332856 (+   298us) service_pool.cc:224] Handling call
0709 12:19:03.951082 (+1618226us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54001b35d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:03.951241 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1433147776}
I20260709 12:19:03.975728 28838 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:03.988655 28710 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.001s	user 0.013s	sys 0.000s
I20260709 12:19:03.945753 28601 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:03.976455 28539 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Stepping down as leader of term 53
W20260709 12:19:04.054960 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 175055us
I20260709 12:19:03.915140 28781 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:03.987107 28790 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:04.074757 28761 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.083133 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:03.942840 28605 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.056066 28837 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.082s	user 0.003s	sys 0.000s
I20260709 12:19:03.930150 28776 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:04.057643 28850 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:04.107684 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.106642 28838 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:04.098748 28780 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:04.136085 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5880) took 1341 ms. Trace:
W20260709 12:19:04.113348 28835 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.368s	user 0.001s	sys 0.000s
I20260709 12:19:04.108603 28601 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:04.136570 12644 rpcz_store.cc:276] 0709 12:19:02.784370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.893880 (+109510us) service_pool.cc:224] Handling call
0709 12:19:02.894280 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:02.903049 (+  8769us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:02.903087 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:02.906390 (+  3303us) log.cc:844] Serialized 26 byte log entry
0709 12:19:03.095251 (+188861us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:03.095293 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:03.095359 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.125257 (+1029898us) raft_consensus.cc:1685] finished
0709 12:19:04.125284 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.125428 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8264,"raft.queue_time_us":12042,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6903,"thread_start_us":104182,"threads_started":1,"wal-append.queue_time_us":311582}
I20260709 12:19:04.115677 28790 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:04.114466 28125 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.115060 28781 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:04.120352 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6388) took 1407 ms. Trace:
I20260709 12:19:04.154083 12565 rpcz_store.cc:276] 0709 12:19:02.712948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.720492 (+  7544us) service_pool.cc:224] Handling call
0709 12:19:02.720865 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:02.818454 (+ 97589us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:02.818502 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:02.863243 (+ 44741us) log.cc:844] Serialized 26 byte log entry
0709 12:19:03.048997 (+185754us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:03.049035 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:03.049143 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.120099 (+1070956us) raft_consensus.cc:1685] finished
0709 12:19:04.120127 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.120289 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59466,"raft.queue_time_us":47639,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"thread_start_us":96756,"threads_started":1,"wal-append.queue_time_us":313773}
W20260709 12:19:04.099354 28752 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.116s	user 0.010s	sys 0.005s
I20260709 12:19:04.111202 28539 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:04.157912 28539 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 53.31, Last appended by leader: 31, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:04.132944 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170499us
W20260709 12:19:04.141132 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58081.9us
I20260709 12:19:04.160094 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:04.160568 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:04.162178 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:04.138485 12489 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.002s
I20260709 12:19:04.143224 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:04.143849 28601 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:04.144750 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.141222 28780 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:04.165351 12489 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:19:04.145597 28853 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:04.170276 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.168102 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:04.146054 28842 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 36.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:19:04.191927 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:04.165126 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:04.147334 28846 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.148023 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.115799 28802 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.120246 28812 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:04.199090 13261 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:04.195066 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.174394 28798 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.022s
I20260709 12:19:04.071630 28830 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.26, Last appended by leader: 26, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.210798 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.211676 12641 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 56: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:04.110800 28710 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.196904 28839 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.366s	user 0.005s	sys 0.000s
W20260709 12:19:04.183373 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.186731 12496 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:04.104463 28854 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:04.176604 28502 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 46 is higher.
I20260709 12:19:04.193948 28780 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:04.216310 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.169658 28752 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
W20260709 12:19:04.206334 13262 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.637s	user 0.012s	sys 0.000s
I20260709 12:19:04.187091 28125 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:04.190936 28308 ts_tablet_manager.cc:1403] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 6.881s	user 0.098s	sys 0.042s
I20260709 12:19:04.214066 28811 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:04.168895 28797 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.088s	user 0.010s	sys 0.000s
I20260709 12:19:04.249334 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:04.206740 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 326279us
I20260709 12:19:04.268524 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.273710 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.288162 28605 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.288415 28849 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader election won for term 56
I20260709 12:19:04.290840 28488 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:04.295403 28841 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.436s	user 0.001s	sys 0.000s
I20260709 12:19:04.295653 28717 raft_consensus.cc:3055] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:04.296454 28725 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.298159 13262 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 41 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
W20260709 12:19:04.306507 28868 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:19:04.120306 28761 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.177139 28137 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.303892 12495 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:04.164763 28858 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.162s
I20260709 12:19:04.269095 28798 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:19:04.307399 12639 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:19:04.210510 28846 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.309242 12571 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:04.347947 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.349028 12639 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:19:04.309198 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.317997 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:04.322141 12644 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:04.316071 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104668us
I20260709 12:19:04.350697 12493 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:19:04.298087 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.209077 28802 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.301697 28125 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:04.341773 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.354961 12638 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:04.306488 28502 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:04.361974 28502 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:04.362488 28502 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:04.365108 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:04.332662 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 168301us
I20260709 12:19:04.182782 28781 consensus_queue.cc:1059] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.119s
I20260709 12:19:04.344614 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:04.279300 28801 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.370532 28725 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.371117 28849 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:04.371554 28846 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:04.332258 28717 raft_consensus.cc:740] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:04.381399 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:04.400410 12493 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:19:04.224694 28853 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.406188 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54432.2us
W20260709 12:19:04.424984 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43703.5us
I20260709 12:19:04.346143 28858 consensus_queue.cc:1243] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 0a29b5d9f9d44298b02ffcd460811eef log is divergent from this leader: its last log entry 41.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 28
I20260709 12:19:04.433933 28849 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 54.35, Last appended by leader: 35, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.437000 28761 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.352771 28488 consensus_queue.cc:1050] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.085s
I20260709 12:19:04.434048 12644 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
I20260709 12:19:04.345358 28798 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:04.440647 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 272568us
I20260709 12:19:04.438346 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2031) took 1352 ms. Trace:
I20260709 12:19:04.312822 28797 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.322751 28811 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.485484 12489 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 55 index: 32. (index mismatch)
I20260709 12:19:04.406828 28534 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:04.683594 13254 rpcz_store.cc:276] 0709 12:19:03.086379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.086514 (+   135us) service_pool.cc:224] Handling call
0709 12:19:03.103606 (+ 17092us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:03.440607 (+337001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:03.440645 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:03.441312 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:19:03.590760 (+149448us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:03.590818 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:03.835594 (+244776us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b540058d754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:03.835715 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.294281 (+458566us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:04.438087 (+143806us) raft_consensus.cc:1685] finished
0709 12:19:04.438107 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.438275 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18243,"raft.queue_time_us":17965,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":26592,"spinlock_wait_cycles":926528128,"thread_start_us":148846,"threads_started":1,"wal-append.queue_time_us":358536}
W20260709 12:19:04.489904 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76901.9us
I20260709 12:19:04.520958 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.437975 28853 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:04.536185 12638 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:04.691421 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.691727 12569 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 52 because replica has last-logged OpId of term: 51 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 29.
I20260709 12:19:04.538875 12641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357"
dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
mode: GRACEFUL
new_leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
 from {username='slave'} at 127.0.0.1:42064
I20260709 12:19:04.489959 28836 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:04.562307 28861 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:04.563145 28781 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:04.563622 28860 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:04.699698 28781 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.565969 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:04.571789 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:04.569758 28468 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.882s	user 0.019s	sys 0.039s
I20260709 12:19:04.578727 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.345077 28850 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.28, Last appended by leader: 28, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.587625 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.588171 28856 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:04.588297 12655 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-5f402dfd68ae4c8697765a76d9f1c4e6 received from {username='slave'} at 127.0.0.1:35404
W20260709 12:19:04.589421 28871 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:19:04.606532 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:04.593379 28840 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.638s	user 0.000s	sys 0.003s
I20260709 12:19:04.535722 28717 consensus_queue.cc:260] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.571446 28125 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.621435 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.622915 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.620874 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.623319 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.620424 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.607787 28801 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:04.679450 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:04.637372 28430 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.667s	user 0.007s	sys 0.006s
I20260709 12:19:04.613291 28710 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.682581 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3030) took 1330 ms. Trace:
I20260709 12:19:04.726312 28539 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:04.683938 12461 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:04.686731 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6186) took 1014 ms. Trace:
I20260709 12:19:04.687943 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.688009 12571 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:04.692504 12641 raft_consensus.cc:606] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Received request to transfer leadership to TS 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:04.814487 12644 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:19:04.814818 12641 raft_consensus.cc:614] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:19:04.687765 28534 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:04.699826 13257 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:04.700616 28802 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.702625 28308 raft_consensus.cc:359] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [term 40 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.707989 28832 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.804s	user 0.008s	sys 0.006s
I20260709 12:19:04.698820 12489 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 30. Preceding OpId from leader: term: 48 index: 31. (index mismatch)
I20260709 12:19:04.724334 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:04.735602 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.762180 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:04.762224 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.770303 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:04.770373 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:04.721946 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6174) took 1281 ms. Trace:
I20260709 12:19:04.742342 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.774052 12643 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:04.774669 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.730159 28797 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.826448 12638 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:19:04.768039 12639 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:19:04.722021 28833 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.698s	user 0.010s	sys 0.004s
I20260709 12:19:04.765049 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:04.781261 28853 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.040s
W20260709 12:19:04.766485 28838 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
W20260709 12:19:04.722224 28790 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.522s	user 0.003s	sys 0.008s
I20260709 12:19:04.816406 28717 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:04.805405 12645 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41. Re-sending same reply.
I20260709 12:19:04.821187 28534 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:04.806525 12642 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 56: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:19:04.766194 28468 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.019s	sys 0.040s
W20260709 12:19:04.721618 28601 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.002s
I20260709 12:19:04.809873 12637 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:04.614265 28846 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.764266 28855 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.745381 28811 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:04.564103 28863 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:19:04.723433 28860 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:04.817322 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:04.818205 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134400us
I20260709 12:19:04.882375 12640 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:04.810288 12566 rpcz_store.cc:276] 0709 12:19:03.351889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.352960 (+  1071us) service_pool.cc:224] Handling call
0709 12:19:03.382105 (+ 29145us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:03.382589 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:03.382622 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:03.383179 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 12:19:04.196119 (+812940us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:04.681812 (+485693us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b18000f3160. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1
0709 12:19:04.682058 (+   246us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:19:04.682115 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.682222 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.682289 (+    67us) raft_consensus.cc:1685] finished
0709 12:19:04.682306 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.682489 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4956,"raft.queue_time_us":16966,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":25915,"spinlock_wait_cycles":369907840,"thread_start_us":100662,"threads_started":1,"wal-append.queue_time_us":223705}
I20260709 12:19:04.825402 12492 rpcz_store.cc:276] 0709 12:19:03.440526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.445727 (+  5201us) service_pool.cc:224] Handling call
0709 12:19:04.721753 (+1276026us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b5400190fd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:04.721874 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":755228032}
I20260709 12:19:04.828262 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:04.812954 28812 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.587s	user 0.015s	sys 0.000s
I20260709 12:19:04.769945 28781 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.787460 13262 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 53. Current term is 54. Ops: []
I20260709 12:19:04.814136 12494 rpcz_store.cc:276] 0709 12:19:03.672604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.907497 (+234893us) service_pool.cc:224] Handling call
0709 12:19:03.908781 (+  1284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:03.929802 (+ 21021us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:03.929883 (+    81us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:03.983112 (+ 53229us) log.cc:844] Serialized 26 byte log entry
0709 12:19:04.268106 (+284994us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:04.441112 (+173006us) log.cc:844] Serialized 14 byte log entry
0709 12:19:04.489317 (+ 48205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.489446 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.686379 (+196933us) raft_consensus.cc:1685] finished
0709 12:19:04.686411 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.686639 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":163587,"prepare.queue_time_us":162163,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":56,"raft.queue_time_us":86004,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":72398,"thread_start_us":172635,"threads_started":1,"wal-append.queue_time_us":490178}
W20260709 12:19:04.836079 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71137.6us
I20260709 12:19:04.904552 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:04.837080 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.009s	sys 0.007s
I20260709 12:19:04.840153 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:04.840543 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:04.752827 28780 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.805274 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:04.820482 28685 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
W20260709 12:19:04.917559 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 193372us
I20260709 12:19:04.723472 28861 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.918965 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:04.920644 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:04.920720 28861 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.831192 28832 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.859338 28848 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:04.822345 12655 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-5f402dfd68ae4c8697765a76d9f1c4e6 on tablet 5f402dfd68ae4c8697765a76d9f1c4e6 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:19:04.838918 28802 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:04.846290 28752 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:04.838794 28738 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260709 12:19:04.812897 28430 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.944033 28848 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.955610 28430 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:04.956971 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.958312 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:04.960292 12488 raft_consensus.cc:1240] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: [46.24-46.24]
I20260709 12:19:04.789090 28856 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.871898 12493 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:04.963959 28856 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.965370 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.883034 28846 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.967185 28780 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:04.968881 28846 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.969115 28803 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:04.972276 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3034) took 1304 ms. Trace:
I20260709 12:19:04.972750 12563 rpcz_store.cc:276] 0709 12:19:03.668078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.668301 (+   223us) service_pool.cc:224] Handling call
0709 12:19:03.668761 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:04.101885 (+433124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.972039 (+870154us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:04.972212 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":994427264}
I20260709 12:19:04.974006 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:04.975911 28890 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
W20260709 12:19:04.977268 28776 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.853s	user 0.013s	sys 0.000s
W20260709 12:19:04.885406 28502 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.522s	user 0.014s	sys 0.000s
I20260709 12:19:04.900125 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2032) took 1813 ms. Trace:
I20260709 12:19:04.883311 28855 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:19:04.905675 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.010s	sys 0.007s
I20260709 12:19:04.906139 12571 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:04.909135 12639 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:19:04.912181 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72599.3us
I20260709 12:19:05.011864 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.722919 28798 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 41.27, Last appended by leader: 27, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:19:04.922253 13253 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2023) took 2382 ms (client timeout 2762 ms). Trace:
W20260709 12:19:05.014335 13253 rpcz_store.cc:269] 0709 12:19:02.519316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.528191 (+  8875us) service_pool.cc:224] Handling call
0709 12:19:04.886308 (+2358117us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b540019d2d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:04.901115 (+ 14807us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2165817728}
I20260709 12:19:04.924532 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:04.847764 28838 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.957401 28848 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.842365 28833 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.942741 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:04.850642 28790 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:04.949247 28834 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 1.078s	user 0.000s	sys 0.004s
I20260709 12:19:04.950701 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4625) took 1378 ms. Trace:
I20260709 12:19:05.093397 12496 rpcz_store.cc:276] 0709 12:19:03.571981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.578208 (+  6227us) service_pool.cc:224] Handling call
0709 12:19:04.950291 (+1372083us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b540034e254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:04.950598 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1238774016}
I20260709 12:19:04.894778 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:04.980264 28860 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 52.23, Last appended by leader: 23, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:04.941363 28781 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:04.984150 12644 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:19:04.984876 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6072) took 1552 ms. Trace:
I20260709 12:19:04.859917 28601 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:04.942021 28308 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [term 40 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:19:04.935045 28488 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:19:04.941031 28685 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:04.922398 28869 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:04.987913 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67386.6us
I20260709 12:19:04.989778 28687 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:19:04.990625 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.991092 28832 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:04.943363 28861 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:04.989816 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:04.999291 12645 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:05.001210 28854 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:04.998091 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:05.006393 13260 rpcz_store.cc:276] 0709 12:19:03.086809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.086937 (+   128us) service_pool.cc:224] Handling call
0709 12:19:03.113007 (+ 26070us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:03.214406 (+101399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:03.214437 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:03.310646 (+ 96209us) log.cc:844] Serialized 52 byte log entry
0709 12:19:03.591509 (+280863us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:03.591547 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:03.591617 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:03.949202 (+357585us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b48000f00d8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:19:04.592245 (+643043us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:04.886847 (+294602us) raft_consensus.cc:1685] finished
0709 12:19:04.886885 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.900011 (+ 13126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13491,"raft.queue_time_us":106348,"raft.run_cpu_time_us":5476,"raft.run_wall_time_us":713925,"spinlock_wait_cycles":53054976,"thread_start_us":285891,"threads_started":2,"wal-append.queue_time_us":532937}
W20260709 12:19:05.013489 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.199688 13256 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:05.141077 28710 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.013878 28502 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.021687 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:04.934165 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:05.029907 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:05.035279 12569 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:05.014281 28776 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:05.168503 28781 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:05.042075 28828 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.396s	user 0.006s	sys 0.000s
I20260709 12:19:05.046648 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6188) took 1311 ms. Trace:
I20260709 12:19:05.042840 28847 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 49.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.013s
I20260709 12:19:05.014495 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:05.081997 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4620) took 1590 ms. Trace:
I20260709 12:19:05.013079 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:19:05.229135 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 214703us
I20260709 12:19:05.228937 12495 rpcz_store.cc:276] 0709 12:19:03.491667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.500097 (+  8430us) service_pool.cc:224] Handling call
0709 12:19:05.081759 (+1581662us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540037df54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:05.081927 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1720556928}
I20260709 12:19:05.083447 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:05.224659 12488 rpcz_store.cc:276] 0709 12:19:03.735515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.953282 (+217767us) service_pool.cc:224] Handling call
0709 12:19:04.959961 (+1006679us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:04.960782 (+   821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.046195 (+ 85413us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540034e250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:05.046510 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2313456000}
I20260709 12:19:05.232805 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:05.105120 12640 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:19:05.081506 28848 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:05.145116 12567 rpcz_store.cc:276] 0709 12:19:03.432210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:03.528922 (+ 96712us) service_pool.cc:224] Handling call
0709 12:19:03.557744 (+ 28822us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:03.567881 (+ 10137us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:19:04.175594 (+607713us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b18003959e0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:19:04.272659 (+ 97065us) log.cc:844] Serialized 14 byte log entry
0709 12:19:04.810517 (+537858us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:04.813478 (+  2961us) log.cc:844] Serialized 26 byte log entry
0709 12:19:04.813704 (+   226us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:04.813827 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:04.813920 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.984404 (+170484us) raft_consensus.cc:1685] finished
0709 12:19:04.984518 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:04.984799 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9618,"prepare.queue_time_us":760552,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":13084,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":13825,"spinlock_wait_cycles":11374848,"thread_start_us":574280,"threads_started":2,"wal-append.queue_time_us":626110}
W20260709 12:19:05.169257 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.186421 28534 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.113638 28856 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.005832 28802 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:05.265851 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 244245us
I20260709 12:19:05.266597 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:05.267711 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:05.268018 28802 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:05.264740 28488 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:05.268872 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.269107 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:05.270344 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:05.270704 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:05.271046 28880 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.271682 28880 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.036239 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.272233 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:05.272893 12638 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:05.265162 12642 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:05.273983 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:05.101387 12571 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:19:05.214716 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 290227us
I20260709 12:19:05.273026 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:05.276171 12641 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 26.
W20260709 12:19:05.219251 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.201557 12460 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:05.276573 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:05.277079 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75693us
W20260709 12:19:05.216609 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.191627 28687 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:05.264253 28856 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.264636 28534 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:05.265120 28829 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:05.196594 28869 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.265601 28890 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:05.314414 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.316892 12490 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:19:05.283322 28836 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:19:05.275465 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.207332 13254 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:19:05.270527 28784 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:05.281311 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.281968 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 41 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.284314 12610 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:05.271142 12567 raft_consensus.cc:2393] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:05.231187 13257 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 39.
I20260709 12:19:05.274677 12639 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:05.323288 28685 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.283162 28802 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:05.299808 28848 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.006s
W20260709 12:19:05.225867 28859 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.086s	sys 0.121s
I20260709 12:19:05.283715 28830 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.223081 28781 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:19:05.263517 28845 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.344s	user 0.002s	sys 0.013s
I20260709 12:19:05.318795 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.307613 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:05.263906 28832 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:05.328620 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:05.364778 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.385213 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:19:05.235520 28798 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:05.280037 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.283628 28850 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:19:05.344985 28859 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.086s	sys 0.122s
W20260709 12:19:05.344076 13218 leader_election.cc:336] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.580s (SENT)
I20260709 12:19:05.398913 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:05.344434 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60208.5us
I20260709 12:19:05.400104 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:05.345824 12569 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:05.401357 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:05.224843 28710 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.313871 28686 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.353363 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.386050 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:05.357161 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:05.302773 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84692.4us
I20260709 12:19:05.389389 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:05.361099 28717 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.510s	user 0.007s	sys 0.002s
I20260709 12:19:05.396828 12496 raft_consensus.cc:2393] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 43: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:05.407683 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:05.348951 28796 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.433s	user 0.030s	sys 0.034s
I20260709 12:19:05.415782 12572 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:05.294704 28430 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.054s
I20260709 12:19:05.418321 12566 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 53 because replica has last-logged OpId of term: 52 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 28.
I20260709 12:19:05.348721 28721 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:05.443048 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6404) took 1230 ms. Trace:
W20260709 12:19:05.443542 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.443766 12566 rpcz_store.cc:276] 0709 12:19:04.212907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.256632 (+1043725us) service_pool.cc:224] Handling call
0709 12:19:05.442924 (+186292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:05.384737 28832 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:05.406108 28539 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.588s	user 0.008s	sys 0.003s
I20260709 12:19:05.393669 28780 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:05.420679 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76640.4us
I20260709 12:19:05.391681 28855 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:05.454608 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.455811 12571 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:05.300988 28308 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 39.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:05.351752 28830 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.425184 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:05.425660 28796 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.515s	user 0.030s	sys 0.038s
I20260709 12:19:05.360867 28534 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:19:05.433955 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.352268 28829 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:05.439256 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131773us
I20260709 12:19:05.423164 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:05.440447 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54511us
I20260709 12:19:05.422331 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:05.424667 28797 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:05.403344 28836 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:05.460490 12567 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:19:05.394310 28687 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.434158 28779 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:05.433346 28802 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:19:05.488220 28125 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.077s	sys 0.089s
I20260709 12:19:05.486438 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6061) took 2622 ms. Trace:
I20260709 12:19:05.488174 12638 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:05.485755 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 49 index: 35 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.443252 28847 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:05.468369 28796 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:05.497934 12644 raft_consensus.cc:1240] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
W20260709 12:19:05.501420 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 78342.2us
I20260709 12:19:05.532359 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:05.522886 28125 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.077s	sys 0.090s
I20260709 12:19:05.533674 28534 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:19:05.534303 28125 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:19:05.492012 28889 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.033s
I20260709 12:19:05.486378 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:05.535892 12610 leader_election.cc:400] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:19:05.536249 12610 leader_election.cc:403] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Cancelling election due to peer responding with higher term
W20260709 12:19:05.536477 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50198.7us
I20260709 12:19:05.536568 28829 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.27, Last appended by leader: 27, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.537381 12639 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 23. Preceding OpId from leader: term: 46 index: 24. (index mismatch)
I20260709 12:19:05.537947 28786 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:05.538555 28534 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 45. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46.
I20260709 12:19:05.540832 28832 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 49.33, Last appended by leader: 33, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.542706 28832 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:05.512207 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.544651 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:05.497816 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:05.478859 28785 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:05.529896 12568 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 53 because replica has last-logged OpId of term: 50 index: 36, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 35.
I20260709 12:19:05.500818 28717 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.523353 12564 rpcz_store.cc:276] 0709 12:19:02.845871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.954661 (+108790us) service_pool.cc:224] Handling call
0709 12:19:02.955075 (+   414us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:19:03.036207 (+ 81132us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:03.036276 (+    69us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:19:03.046514 (+ 10238us) log.cc:844] Serialized 78 byte log entry
0709 12:19:03.144554 (+ 98040us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:19:03.144581 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:03.144872 (+   291us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:04.980805 (+1835933us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:05.448947 (+468142us) raft_consensus.cc:1685] finished
0709 12:19:05.448984 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:05.467892 (+ 18908us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":166,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":12321,"spinlock_wait_cycles":763345664,"thread_start_us":41672,"threads_started":1,"wal-append.queue_time_us":465364}
I20260709 12:19:05.469482 28812 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:05.546573 28717 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.547206 12642 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 26. Preceding OpId from leader: term: 55 index: 27. (index mismatch)
I20260709 12:19:05.547425 28812 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:05.547739 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6407) took 1266 ms. Trace:
I20260709 12:19:05.548509 28812 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.27, Last appended by leader: 27, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.545828 28687 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:19:05.548216 12564 rpcz_store.cc:276] 0709 12:19:04.281386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.546811 (+1265425us) service_pool.cc:224] Handling call
0709 12:19:05.547209 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.547406 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.547432 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:05.547475 (+    43us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:05.547507 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.547564 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:05.547680 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:05.522614 28855 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:05.476595 28869 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:05.558555 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60879.4us
I20260709 12:19:05.559212 28836 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.560727 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:05.561976 28785 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:05.563195 12566 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 49
I20260709 12:19:05.559639 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:05.500592 28539 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.488145 28790 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:05.605866 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:05.426308 28686 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.555401 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6410) took 1273 ms. Trace:
I20260709 12:19:05.528481 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.609575 12571 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:05.485683 28601 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:05.526530 28893 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.108s	user 0.001s	sys 0.001s
I20260709 12:19:05.608013 12569 rpcz_store.cc:276] 0709 12:19:04.282384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.554250 (+1271866us) service_pool.cc:224] Handling call
0709 12:19:05.554691 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.554947 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.554986 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:05.555045 (+    59us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:05.555096 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.555183 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:05.555317 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:05.609319 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:05.607466 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:05.526769 28838 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:05.621361 28308 ts_tablet_manager.cc:1434] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 1.315s	user 0.008s	sys 0.008s
W20260709 12:19:05.621857 28308 ts_tablet_manager.cc:1467] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Tablet startup took 1320ms
I20260709 12:19:05.474056 28721 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.660862 28830 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:05.660969 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:05.670185 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:05.672086 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 62877.9us
I20260709 12:19:05.673141 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:05.675151 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.613811 28864 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:05.461323 28780 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.650266 28849 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:05.654548 28838 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.040s
I20260709 12:19:05.614096 28855 consensus_queue.cc:1059] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 44.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.069s
W20260709 12:19:05.678284 28752 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.743s	user 0.012s	sys 0.000s
I20260709 12:19:05.668773 28790 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:05.718447 12610 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:05.720337 28862 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:05.721217 28721 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:05.696143 28853 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:19:05.622211 28308 ts_tablet_manager.cc:1469] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Trace:
0709 12:18:48.245586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:48.245878 (+   292us) service_pool.cc:224] Handling call
0709 12:18:48.917374 (+671496us) ts_tablet_manager.cc:934] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
0709 12:18:51.979787 (+3062413us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:18:53.166020 (+1186233us) tablet_metadata.cc:665] Metadata flushed
0709 12:18:53.166171 (+   151us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:18:53.619787 (+453616us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:18:53.620155 (+   368us) tablet_replica.cc:206] Creating consensus instance
0709 12:18:53.620510 (+   355us) ts_tablet_manager.cc:989] Replica registered.
0709 12:18:53.620650 (+   140us) inbound_call.cc:177] Queueing success response
0709 12:18:57.309420 (+3688770us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:19:04.323075 (+7013655us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:19:04.566632 (+243557us) tablet_replica.cc:261] Starting instrumentation
0709 12:19:04.567373 (+   741us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":150211,"mutex_wait_us":942,"raft.queue_time_us":3348,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":15452,"tablet-copy.queue_time_us":434901,"tablet-download-pool.queue_time_us":2136,"tablet-download-pool.run_cpu_time_us":1452,"tablet-download-pool.run_wall_time_us":1932223,"thread_start_us":611577,"threads_started":2}
I20260709 12:19:05.673323 28502 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:05.678103 28830 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.612434 28539 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:05.715593 28893 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:19:05.725584 28895 log.cc:927] Time spent T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: Append to log took a long time: real 0.158s	user 0.002s	sys 0.000s
I20260709 12:19:05.688812 28856 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:05.684056 28902 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:19:05.717197 28468 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.029s	sys 0.021s
I20260709 12:19:05.778282 28802 raft_consensus.cc:3055] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:05.769366 28488 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.775209 28830 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.792713 28880 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:05.782346 28848 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.796061 28717 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:05.797956 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:05.799903 28539 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:05.806977 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 423518us
I20260709 12:19:05.808055 12566 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:05.827880 28724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.976s	user 0.013s	sys 0.026s
I20260709 12:19:05.811267 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:05.688678 28686 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:05.788759 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6408) took 1507 ms. Trace:
I20260709 12:19:05.686302 28849 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:05.815483 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 97131.3us
I20260709 12:19:05.739704 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:05.728027 28790 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:05.801554 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:05.883845 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 82357.8us
W20260709 12:19:05.884011 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.694895 28710 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:05.812044 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6399) took 1621 ms. Trace:
I20260709 12:19:05.885875 12565 rpcz_store.cc:276] 0709 12:19:04.190698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:04.191524 (+   826us) service_pool.cc:224] Handling call
0709 12:19:05.801691 (+1610167us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400181854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:05.811948 (+ 10257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3139142912}
I20260709 12:19:05.888859 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6079) took 1276 ms. Trace:
I20260709 12:19:05.768026 28862 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:05.890431 28862 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.28, Last appended by leader: 28, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.892457 28862 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:05.778762 28854 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.830864 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:05.725893 28752 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:05.775969 28779 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.795323 28780 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:05.826578 28468 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.029s	sys 0.021s
W20260709 12:19:05.813305 28689 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.841s	user 0.015s	sys 0.025s
W20260709 12:19:05.849153 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51286.9us
I20260709 12:19:05.851583 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:05.810034 12646 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:05.855131 28848 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.752013 28861 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.077s
W20260709 12:19:05.874336 28724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.067s	user 0.014s	sys 0.027s
I20260709 12:19:05.871418 12568 rpcz_store.cc:276] 0709 12:19:04.281796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.546824 (+1265028us) service_pool.cc:224] Handling call
0709 12:19:05.787844 (+241020us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26ceeee 000000000036eb39 00007ffbfa7421e0 00007ffbfa80b0c4 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:19:05.788071 (+   227us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.788331 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.788360 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:05.788416 (+    56us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:05.788462 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.788524 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:05.788674 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3547520}
I20260709 12:19:05.880134 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6414) took 1442 ms. Trace:
I20260709 12:19:05.874218 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:05.857865 28776 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:05.694125 28796 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 40.23, Last appended by leader: 23, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.884810 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:05.828347 28802 raft_consensus.cc:740] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:05.889322 12565 rpcz_store.cc:276] 0709 12:19:04.612232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.887264 (+1275032us) service_pool.cc:224] Handling call
0709 12:19:05.887899 (+   635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.888321 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.888357 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:05.888408 (+    51us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:05.888451 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.888529 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:05.888778 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:05.857575 28803 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.888s	user 0.006s	sys 0.005s
I20260709 12:19:05.925719 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:05.928011 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 447621us
W20260709 12:19:05.928900 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:05.957486 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83416.2us
I20260709 12:19:05.956398 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:05.974133 12610 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:05.941685 12563 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:05.924096 28601 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election won for term 50
W20260709 12:19:05.957173 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72499.8us
I20260709 12:19:05.895016 28717 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.941092 12572 rpcz_store.cc:276] 0709 12:19:04.437493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.340526 (+903033us) service_pool.cc:224] Handling call
0709 12:19:05.880033 (+539507us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:05.932354 28689 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.960s	user 0.015s	sys 0.026s
I20260709 12:19:05.927134 28502 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:05.957019 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:05.974362 12611 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:05.981585 12572 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 50 index: 36. Preceding OpId from leader: term: 52 index: 37. (index mismatch)
I20260709 12:19:05.996315 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3037) took 1253 ms. Trace:
I20260709 12:19:05.996946 12572 rpcz_store.cc:276] 0709 12:19:04.743117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.980873 (+1237756us) service_pool.cc:224] Handling call
0709 12:19:05.981214 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.995940 (+ 14726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.996240 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:05.930838 28850 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:05.976082 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50483.6us
I20260709 12:19:05.840384 28488 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:05.990152 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.955951 28780 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:05.989816 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 512359us
I20260709 12:19:05.997767 12565 raft_consensus.cc:1275] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 55 index: 32. (index mismatch)
I20260709 12:19:05.931052 28687 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:06.006242 12568 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:06.007409 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:06.016086 12611 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:06.013823 12493 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:19:05.989838 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:05.894310 28854 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.039315 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:06.042312 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.929247 28779 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.956532 28776 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:06.021606 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3035) took 1279 ms. Trace:
I20260709 12:19:06.021864 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:05.975103 28856 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:19:05.975340 28539 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:06.030685 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:06.032349 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:06.029464 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.038682 12488 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:19:06.039068 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:06.040331 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6080) took 1427 ms. Trace:
I20260709 12:19:06.020052 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:05.983260 28864 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:05.872437 28849 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 43.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:05.983582 28824 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.006011 28534 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:05.994427 28710 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:06.074607 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:06.075343 12489 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:06.067848 12496 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 31. Preceding OpId from leader: term: 46 index: 32. (index mismatch)
I20260709 12:19:06.078090 12563 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:06.068954 13256 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 54: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:06.069226 12494 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:06.071159 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:06.068948 12492 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:06.067210 12568 rpcz_store.cc:276] 0709 12:19:04.612555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.971111 (+1358556us) service_pool.cc:224] Handling call
0709 12:19:06.006022 (+ 34911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.039951 (+ 33929us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.040256 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.067252 12488 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:06.049484 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.571s (SENT)
I20260709 12:19:06.070870 12566 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:06.066879 12493 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
I20260709 12:19:06.069751 12565 rpcz_store.cc:276] 0709 12:19:04.742494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.971994 (+1229500us) service_pool.cc:224] Handling call
0709 12:19:05.980689 (+  8695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.021309 (+ 40620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.021544 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.112283 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.007997 28717 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.006049 28601 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:06.082304 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:06.082520 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.082593 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:06.083981 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:06.027027 28855 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:19:06.023283 28752 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:06.080617 12637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.002s	sys 0.022s
W20260709 12:19:06.066515 28798 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.663s	user 0.013s	sys 0.001s
W20260709 12:19:06.070845 28832 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.528s	user 0.014s	sys 0.000s
I20260709 12:19:06.039776 28883 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:06.084949 12642 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.007s	sys 0.023s
I20260709 12:19:06.069538 12497 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
W20260709 12:19:06.173880 12642 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.007s	sys 0.025s
W20260709 12:19:06.160386 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54269.2us
I20260709 12:19:06.066401 28776 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:06.111229 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.046689 28687 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:06.015215 28502 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:06.074590 28856 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:06.113798 12488 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 53.24, Last appended by leader: 24, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.181291 12488 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:06.182613 12536 leader_election.cc:400] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:06.183218 12536 leader_election.cc:403] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:19:06.114048 28869 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.018s	sys 0.023s
I20260709 12:19:06.014266 28790 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 51.22, Last appended by leader: 22, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.117105 13253 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:06.100869 28829 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:06.122635 12492 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 35. Preceding OpId from leader: term: 52 index: 37. (index mismatch)
I20260709 12:19:06.186199 28856 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:06.172650 12570 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 26.
W20260709 12:19:06.160380 12637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.158s	user 0.002s	sys 0.022s
W20260709 12:19:06.172959 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.163837 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6417) took 1582 ms. Trace:
W20260709 12:19:06.114769 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 495143us
I20260709 12:19:06.112967 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:19:06.114694 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65262.8us
I20260709 12:19:06.045189 28780 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.100785 28802 consensus_queue.cc:260] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 46.24, Last appended by leader: 24, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.022552 28850 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.076730 28864 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.176174 28752 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.162869 28855 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:06.087577 28824 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:06.177587 28836 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.012s	sys 0.027s
I20260709 12:19:06.117128 28830 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:06.111894 28534 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:19:06.192586 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.206208 28853 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:06.200290 28869 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.019s	sys 0.023s
I20260709 12:19:06.205421 12569 rpcz_store.cc:276] 0709 12:19:04.581757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.613909 (+1032152us) service_pool.cc:224] Handling call
0709 12:19:05.614382 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.124319 (+509937us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.124368 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.124441 (+    73us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:06.163532 (+ 39091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.163615 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.163774 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":342025,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7566,"thread_start_us":497753,"threads_started":1}
I20260709 12:19:06.195537 28918 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:06.228471 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.208673 13218 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.989s (SENT)
I20260709 12:19:06.256852 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.254725 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:06.214494 28860 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.007s	sys 0.017s
I20260709 12:19:06.209800 28911 raft_consensus.cc:3060] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 51
I20260709 12:19:06.284065 12568 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:06.226259 28855 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:06.314069 13260 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 28. Preceding OpId from leader: term: 52 index: 29. (index mismatch)
I20260709 12:19:06.310164 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.311125 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:06.324234 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:06.312832 28860 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.008s	sys 0.017s
I20260709 12:19:06.346585 12640 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: []
I20260709 12:19:06.333735 12507 tablet_copy_service.cc:342] P 6d4ece942fd5414998215d26edf91001: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-3eadf951b63a43fd993cea2c96d5214f received from {username='slave'} at 127.0.0.1:54234
I20260709 12:19:06.347900 12507 tablet_copy_service.cc:434] P 6d4ece942fd5414998215d26edf91001: ending tablet copy session 9485b0afaf144396adf854fff857414e-3eadf951b63a43fd993cea2c96d5214f on tablet 3eadf951b63a43fd993cea2c96d5214f with peer 9485b0afaf144396adf854fff857414e
I20260709 12:19:06.317272 28848 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:06.350427 28864 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:06.315239 28918 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:06.335011 28905 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:19:06.335712 28854 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:06.338522 28687 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:06.339556 28850 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:06.340495 28502 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:06.343106 28780 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:06.221992 28883 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.100s
I20260709 12:19:06.211735 28829 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:06.269378 28906 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 41.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.061s
I20260709 12:19:06.371001 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5899) took 1382 ms. Trace:
I20260709 12:19:06.377554 13253 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:06.283551 28534 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:06.399045 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:06.398236 13254 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.033s	sys 0.055s
W20260709 12:19:06.399914 13254 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.033s	sys 0.055s
W20260709 12:19:06.308951 28836 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.013s	sys 0.035s
W20260709 12:19:06.316582 28781 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.922s	user 0.012s	sys 0.012s
I20260709 12:19:06.395589 12637 rpcz_store.cc:276] 0709 12:19:04.988547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.000063 (+ 11516us) service_pool.cc:224] Handling call
0709 12:19:05.000567 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:05.001015 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.001051 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:05.001771 (+   720us) log.cc:844] Serialized 26 byte log entry
0709 12:19:06.204231 (+1202460us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:06.204296 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.204412 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:06.370757 (+166345us) raft_consensus.cc:1685] finished
0709 12:19:06.370782 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.370927 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9600,"raft.queue_time_us":1426,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":6070,"thread_start_us":1201955,"threads_started":1,"wal-append.queue_time_us":1312569}
I20260709 12:19:06.407954 28779 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:06.357995 28908 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
W20260709 12:19:06.418568 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 402538us
W20260709 12:19:06.359627 28903 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.245s	user 0.000s	sys 0.001s
W20260709 12:19:06.359023 28904 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260709 12:19:06.419795 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 211144us
W20260709 12:19:06.379554 28895 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.000s	sys 0.013s
I20260709 12:19:06.366312 28920 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:06.434615 12568 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 30. Preceding OpId from leader: term: 48 index: 31. (index mismatch)
I20260709 12:19:06.353521 28918 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:06.411742 28534 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:06.472898 12494 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:06.398013 28815 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:19:06.427376 28780 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Stepping down as leader of term 42
W20260709 12:19:06.457267 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6411) took 2174 ms (client timeout 2030 ms). Trace:
I20260709 12:19:06.464183 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3039) took 1720 ms. Trace:
I20260709 12:19:06.257901 28601 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 49.33, Last appended by leader: 33, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:06.477927 12637 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:19:06.482739 12568 rpcz_store.cc:276] 0709 12:19:04.743918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.423731 (+1679813us) service_pool.cc:224] Handling call
0709 12:19:06.424141 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.463761 (+ 39620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.464099 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:06.481799 12570 rpcz_store.cc:269] 0709 12:19:04.282644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:04.622980 (+340336us) service_pool.cc:224] Handling call
0709 12:19:06.432572 (+1809592us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:06.457147 (+ 24575us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3146691840}
W20260709 12:19:06.507148 28901 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.849s	user 0.013s	sys 0.013s
I20260709 12:19:06.553855 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5919) took 1064 ms. Trace:
W20260709 12:19:06.528765 12646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2298) took 1941 ms (client timeout 1630 ms). Trace:
I20260709 12:19:06.530469 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:06.555483 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.556039 12638 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:06.531714 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.556545 12568 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:06.554440 12637 rpcz_store.cc:276] 0709 12:19:05.489864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.427057 (+937193us) service_pool.cc:224] Handling call
0709 12:19:06.464610 (+ 37553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.547563 (+ 82953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.553762 (+  6199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.424469 28781 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:06.538581 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:06.562732 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.541129 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:06.527220 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.565551 12639 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:19:06.533032 12563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 3043) took 1564 ms (client timeout 1723 ms). Trace:
W20260709 12:19:06.554666 12646 rpcz_store.cc:269] 0709 12:19:04.605975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:04.606182 (+   207us) service_pool.cc:224] Handling call
0709 12:19:06.528439 (+1922257us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b5400154854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:06.528614 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1575641856}
W20260709 12:19:06.566535 12563 rpcz_store.cc:269] 0709 12:19:04.968179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:04.973735 (+  5556us) service_pool.cc:224] Handling call
0709 12:19:06.532820 (+1559085us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:06.532958 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2223684992}
I20260709 12:19:06.568202 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.568907 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:06.566452 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.569036 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 55.
I20260709 12:19:06.572065 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.573377 12563 raft_consensus.cc:2410] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 49 because replica has last-logged OpId of term: 46 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 25.
I20260709 12:19:06.574548 12568 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:06.575855 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.481594 28780 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:06.578258 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:06.589368 12646 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:19:06.579238 12643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.013s	sys 0.030s
I20260709 12:19:06.582404 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.584138 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5909) took 1591 ms. Trace:
I20260709 12:19:06.591723 12644 rpcz_store.cc:276] 0709 12:19:04.992422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.394131 (+401709us) service_pool.cc:224] Handling call
0709 12:19:05.412593 (+ 18462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.531133 (+118540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.583865 (+1052732us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b54002a5dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:06.584063 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191439616}
W20260709 12:19:06.590508 12643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.014s	sys 0.030s
I20260709 12:19:06.595148 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5918) took 1208 ms. Trace:
W20260709 12:19:06.586228 12564 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.017s	sys 0.013s
I20260709 12:19:06.586963 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.542922 28906 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:06.600771 12564 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.018s	sys 0.013s
I20260709 12:19:06.581306 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:06.595920 12568 raft_consensus.cc:2410] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 53 because replica has last-logged OpId of term: 51 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 28.
I20260709 12:19:06.590500 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:06.591481 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:06.634040 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.599094 12643 rpcz_store.cc:276] 0709 12:19:05.387325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.554335 (+167010us) service_pool.cc:224] Handling call
0709 12:19:05.554671 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.554921 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:05.976694 (+421773us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.594739 (+618045us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.594813 (+    74us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:06.594875 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.594939 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.595078 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":701171,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":110,"thread_start_us":1009282,"threads_started":2}
I20260709 12:19:06.618173 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5905) took 1626 ms. Trace:
I20260709 12:19:06.640414 12638 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:06.641407 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.022s	sys 0.064s
W20260709 12:19:06.641847 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.022s	sys 0.064s
I20260709 12:19:06.635892 12641 rpcz_store.cc:276] 0709 12:19:04.991377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.310291 (+318914us) service_pool.cc:224] Handling call
0709 12:19:05.338983 (+ 28692us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:05.347478 (+  8495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.347541 (+    63us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:05.362258 (+ 14717us) log.cc:844] Serialized 52 byte log entry
0709 12:19:05.678513 (+316255us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:05.678559 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:05.679560 (+  1001us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:06.617785 (+938225us) raft_consensus.cc:1685] finished
0709 12:19:06.617824 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.618090 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8019,"raft.queue_time_us":8154,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":16633,"thread_start_us":236706,"threads_started":1,"wal-append.queue_time_us":553378}
I20260709 12:19:06.643509 12641 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
W20260709 12:19:06.645080 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54712.8us
I20260709 12:19:06.645710 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6412) took 2293 ms. Trace:
I20260709 12:19:06.642220 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.578277 28752 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:06.675695 12463 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.061s	sys 0.070s
W20260709 12:19:06.686995 12463 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.062s	sys 0.070s
I20260709 12:19:06.688024 12568 pending_rounds.cc:85] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 28
I20260709 12:19:06.688936 12568 pending_rounds.cc:107] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.29
I20260709 12:19:06.689312 12568 raft_consensus.cc:2889] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:06.655311 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.017s	sys 0.036s
W20260709 12:19:06.690742 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.019s	sys 0.036s
I20260709 12:19:06.648859 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.692263 12639 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:19:06.646813 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65599.5us
I20260709 12:19:06.679654 12641 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 28. Preceding OpId from leader: term: 52 index: 29. (index mismatch)
I20260709 12:19:06.695633 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.697743 12644 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 44 because replica has last-logged OpId of term: 42 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
I20260709 12:19:06.696291 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:06.698626 12570 raft_consensus.cc:2410] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 46 because replica has last-logged OpId of term: 45 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 24.
I20260709 12:19:06.585808 28861 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:06.694341 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:06.705502 12642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.002s	sys 0.055s
I20260709 12:19:06.700250 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6415) took 2255 ms. Trace:
W20260709 12:19:06.701135 12495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.006s	sys 0.042s
I20260709 12:19:06.562330 28776 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:06.706787 12646 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:06.712366 12641 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 54 index: 35. Preceding OpId from leader: term: 56 index: 36. (index mismatch)
W20260709 12:19:06.707263 12642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.002s	sys 0.055s
I20260709 12:19:06.707499 12567 rpcz_store.cc:276] 0709 12:19:04.437867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.618280 (+1180413us) service_pool.cc:224] Handling call
0709 12:19:05.657370 (+ 39090us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.657710 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:19:06.116257 (+458547us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.693105 (+576848us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.693182 (+    77us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:06.693263 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.693377 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.693576 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":660057,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":6418,"thread_start_us":1030979,"threads_started":2}
W20260709 12:19:06.707846 12495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.006s	sys 0.042s
I20260709 12:19:06.718780 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3041) took 1886 ms. Trace:
I20260709 12:19:06.719337 12563 rpcz_store.cc:276] 0709 12:19:04.831959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.649824 (+1817865us) service_pool.cc:224] Handling call
0709 12:19:06.718184 (+ 68360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.718423 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.718458 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.718508 (+    50us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:06.718537 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.718584 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.718714 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.722604 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.723624 12643 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
W20260709 12:19:06.724920 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 470223us
I20260709 12:19:06.581534 28502 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.094s
W20260709 12:19:06.737082 12491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.006s	sys 0.015s
I20260709 12:19:06.713546 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:06.673454 28724 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:06.744082 28724 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:06.745168 12563 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:06.673887 28836 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:06.705516 28861 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:06.746238 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6083) took 1732 ms. Trace:
I20260709 12:19:06.746491 28861 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:06.746809 12563 rpcz_store.cc:276] 0709 12:19:05.014960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.744610 (+1729650us) service_pool.cc:224] Handling call
0709 12:19:06.744961 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.745900 (+   939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.746167 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.748520 12563 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:06.749243 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6084) took 1517 ms. Trace:
I20260709 12:19:06.716697 28883 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:19:06.745265 28724 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 47.30, Last appended by leader: 30, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:06.742767 12491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.006s	sys 0.015s
I20260709 12:19:06.744095 12567 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:19:06.704814 28832 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:06.613853 28717 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:06.758957 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6082) took 1752 ms. Trace:
I20260709 12:19:06.769217 12567 rpcz_store.cc:276] 0709 12:19:05.006996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.717420 (+1710424us) service_pool.cc:224] Handling call
0709 12:19:06.743914 (+ 26494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.758614 (+ 14700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.758897 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:06.761291 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.614573 28860 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.123s
I20260709 12:19:06.758104 12460 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:06.749614 12563 rpcz_store.cc:276] 0709 12:19:05.232335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.747890 (+1515555us) service_pool.cc:224] Handling call
0709 12:19:06.748302 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.748959 (+   657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.749183 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:06.766999 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 307669us
W20260709 12:19:06.782207 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.041s (SENT)
I20260709 12:19:06.790607 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:06.646272 12564 rpcz_store.cc:276] 0709 12:19:04.352705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.554666 (+1201961us) service_pool.cc:224] Handling call
0709 12:19:05.555000 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.555237 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:05.979127 (+423890us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.645246 (+666119us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.645312 (+    66us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:06.645373 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.645445 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.645602 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43762,"prepare.queue_time_us":671240,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":49258,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":30125,"thread_start_us":1040565,"threads_started":2,"wal-append.queue_time_us":778337}
I20260709 12:19:06.640481 28906 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:06.784792 12567 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 36. Current term is 37. Ops: []
W20260709 12:19:06.751076 28922 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:19:06.771456 28856 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.003s
W20260709 12:19:06.750384 28923 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
I20260709 12:19:06.761849 28832 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:06.767652 28829 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:06.746234 28836 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:06.811273 12563 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:19:06.822202 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6422) took 1337 ms. Trace:
I20260709 12:19:06.822714 12563 rpcz_store.cc:276] 0709 12:19:05.485118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.779943 (+1294825us) service_pool.cc:224] Handling call
0709 12:19:06.810747 (+ 30804us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.821764 (+ 11017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.822116 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.814393 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6421) took 1521 ms. Trace:
I20260709 12:19:06.824201 12567 rpcz_store.cc:276] 0709 12:19:05.293325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.770918 (+1477593us) service_pool.cc:224] Handling call
0709 12:19:06.784503 (+ 13585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.804919 (+ 20416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.814293 (+  9374us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.826018 12567 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:06.826771 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3045) took 1024 ms. Trace:
I20260709 12:19:06.827134 12567 rpcz_store.cc:276] 0709 12:19:05.802725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.825241 (+1022516us) service_pool.cc:224] Handling call
0709 12:19:06.825771 (+   530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.826467 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.826705 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:06.785106 28689 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:19:06.819314 12571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.004s	sys 0.042s
W20260709 12:19:06.828655 12571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.004s	sys 0.043s
I20260709 12:19:06.830899 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6077) took 2364 ms. Trace:
I20260709 12:19:06.831324 12571 rpcz_store.cc:276] 0709 12:19:04.466075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.618212 (+1152137us) service_pool.cc:224] Handling call
0709 12:19:05.671722 (+ 53510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.672052 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:06.197540 (+525488us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.830418 (+632878us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.830510 (+    92us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:06.830584 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.830670 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.830834 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1255,"prepare.queue_time_us":694844,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":1094473,"threads_started":2}
W20260709 12:19:06.844127 12497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.018s	sys 0.029s
W20260709 12:19:06.844580 12497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.018s	sys 0.029s
I20260709 12:19:06.846251 12564 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 26. Preceding OpId from leader: term: 55 index: 27. (index mismatch)
W20260709 12:19:06.846127 12645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s	user 0.001s	sys 0.017s
W20260709 12:19:06.849563 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.971s	user 0.002s	sys 0.017s
I20260709 12:19:06.853536 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5910) took 1860 ms. Trace:
I20260709 12:19:06.858501 12563 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: []
I20260709 12:19:06.769860 28780 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 42.23, Last appended by leader: 23, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:06.847755 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6425) took 1327 ms. Trace:
I20260709 12:19:06.821347 28832 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:06.860343 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6426) took 1329 ms. Trace:
I20260709 12:19:06.856849 12645 rpcz_store.cc:276] 0709 12:19:04.992670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.393161 (+400491us) service_pool.cc:224] Handling call
0709 12:19:05.417221 (+ 24060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.417533 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:19:05.878151 (+460618us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.852771 (+974620us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.853010 (+   239us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.853077 (+    67us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:06.853159 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.853258 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.853442 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":795748,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":128,"thread_start_us":1389782,"threads_started":2}
W20260709 12:19:06.899611 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:06.899677 12563 rpcz_store.cc:276] 0709 12:19:05.531445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.823841 (+1292396us) service_pool.cc:224] Handling call
0709 12:19:06.858221 (+ 34380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.859943 (+  1722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.860269 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:06.885088 28796 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.003s	sys 0.040s
I20260709 12:19:06.886710 12564 rpcz_store.cc:276] 0709 12:19:05.520734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.792309 (+1271575us) service_pool.cc:224] Handling call
0709 12:19:06.818091 (+ 25782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.847406 (+ 29315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.847685 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:06.860227 28855 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.518s	user 0.011s	sys 0.002s
I20260709 12:19:06.832219 28836 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:06.836321 28829 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260709 12:19:06.911365 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:06.917128 12563 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:06.819022 28856 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:06.922657 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:06.956377 12564 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:19:06.884351 28686 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:06.945533 12644 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 31. Preceding OpId from leader: term: 46 index: 32. (index mismatch)
I20260709 12:19:06.898307 28502 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.960645 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:06.967536 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:06.967948 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3042) took 2045 ms. Trace:
W20260709 12:19:06.919507 28796 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.003s	sys 0.040s
I20260709 12:19:06.908787 28781 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:06.985353 12567 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 45 index: 27. (term mismatch)
I20260709 12:19:06.913007 28869 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.934501 28848 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.934823 28927 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.939393 28860 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:06.986018 12567 pending_rounds.cc:85] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 26
I20260709 12:19:06.995754 12567 pending_rounds.cc:107] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 44.27
I20260709 12:19:06.996111 12567 raft_consensus.cc:2889] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:06.997822 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 215619us
W20260709 12:19:06.998950 13218 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.123s (SENT)
I20260709 12:19:06.999521 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:06.934144 28855 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.022075 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:06.990808 28780 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:07.025770 12644 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
I20260709 12:19:07.030316 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5917) took 1817 ms. Trace:
W20260709 12:19:07.033679 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.026434 12567 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:07.034655 12567 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:07.034739 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:07.038383 12645 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:07.031324 12642 rpcz_store.cc:276] 0709 12:19:05.212721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.549240 (+336519us) service_pool.cc:224] Handling call
0709 12:19:05.549541 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:05.572032 (+ 22491us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:06.174375 (+602343us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.744307 (+569932us) log.cc:844] Serialized 14 byte log entry
0709 12:19:07.029863 (+285556us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:07.029928 (+    65us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:07.030012 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.030102 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.030241 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28480,"prepare.queue_time_us":757504,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":79,"raft.queue_time_us":51607,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":1427906,"threads_started":3,"wal-append.queue_time_us":645120}
I20260709 12:19:07.047106 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:06.983263 28502 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:07.066804 28930 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.276s	user 0.001s	sys 0.000s
I20260709 12:19:07.067029 28906 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:07.072127 28936 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:19:07.072333 28488 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.074365 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6207) took 1075 ms. Trace:
I20260709 12:19:07.074813 12495 rpcz_store.cc:276] 0709 12:19:05.999032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.999219 (+   187us) service_pool.cc:224] Handling call
0709 12:19:06.025234 (+ 26015us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:06.047344 (+ 22110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.047375 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:06.074098 (+ 26723us) log.cc:844] Serialized 26 byte log entry
0709 12:19:06.717808 (+643710us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:06.962093 (+244285us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.962710 (+   617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.962919 (+   209us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:07.074099 (+111180us) raft_consensus.cc:1685] finished
0709 12:19:07.074125 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.074293 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27034,"prepare.queue_time_us":355197,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":26513,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":16108,"thread_start_us":779211,"threads_started":2,"wal-append.queue_time_us":693605}
I20260709 12:19:07.028663 28776 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.033268 28854 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:07.001591 28848 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.002760 28860 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.011126 28869 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:07.060963 28929 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:19:07.089229 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.090780 28855 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.092442 28861 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.093544 28776 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
W20260709 12:19:07.094518 28949 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:19:07.094810 28950 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:07.096913 28502 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.006695 28927 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:07.100847 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.065786 12567 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 46.24, Last appended by leader: 24, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.149974 12567 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:07.102268 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.152743 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:07.103084 12644 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54. Re-sending same reply.
I20260709 12:19:07.124909 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6093) took 1125 ms. Trace:
I20260709 12:19:07.143204 28848 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.144770 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2467) took 1034 ms. Trace:
W20260709 12:19:07.154953 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.158929 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.104046 28849 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.157526 12638 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:07.158258 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:07.165505 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130852us
W20260709 12:19:07.161405 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 238950us
I20260709 12:19:07.168395 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:07.159076 12488 rpcz_store.cc:276] 0709 12:19:06.110690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.425481 (+314791us) service_pool.cc:224] Handling call
0709 12:19:06.425887 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:06.459761 (+ 33874us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.459806 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:06.479223 (+ 19417us) log.cc:844] Serialized 26 byte log entry
0709 12:19:06.848224 (+369001us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:06.848286 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.848384 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:07.144434 (+296050us) raft_consensus.cc:1685] finished
0709 12:19:07.144470 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.144688 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5624,"raft.queue_time_us":31627,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":3925,"thread_start_us":282578,"threads_started":1,"wal-append.queue_time_us":499814}
I20260709 12:19:07.145905 28776 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:07.175626 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:07.180233 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:07.181137 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 400598us
W20260709 12:19:07.166057 28937 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.265s	user 0.001s	sys 0.001s
I20260709 12:19:07.152417 28784 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.183671 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:07.189942 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2054) took 1390 ms. Trace:
I20260709 12:19:07.155750 28864 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:07.195267 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6420) took 2212 ms. Trace:
I20260709 12:19:07.197783 12567 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 24. Preceding OpId from leader: term: 47 index: 24. (term mismatch)
I20260709 12:19:07.192481 13254 rpcz_store.cc:276] 0709 12:19:05.799867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.811857 (+ 11990us) service_pool.cc:224] Handling call
0709 12:19:05.834125 (+ 22268us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:05.834585 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:05.834614 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:05.835245 (+   631us) log.cc:844] Serialized 52 byte log entry
0709 12:19:06.400485 (+565240us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:06.812828 (+412343us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.832364 (+ 19536us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.834714 (+  2350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.834829 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:07.189665 (+354836us) raft_consensus.cc:1685] finished
0709 12:19:07.189690 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.189874 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9172,"prepare.queue_time_us":232246,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":10199,"raft.queue_time_us":182423,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":19581,"spinlock_wait_cycles":6080896,"thread_start_us":950088,"threads_started":2,"wal-append.queue_time_us":917062}
I20260709 12:19:07.167868 28848 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:07.185408 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.199790 12567 pending_rounds.cc:85] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 23
I20260709 12:19:07.183429 28776 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:07.232620 12567 pending_rounds.cc:107] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 46.24
I20260709 12:19:07.233037 12567 raft_consensus.cc:2889] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:07.231523 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.154084 28950 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.259984 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:07.261104 12567 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:19:07.191308 28957 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:07.196642 28953 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:07.198190 12570 rpcz_store.cc:276] 0709 12:19:04.983101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.704573 (+1721472us) service_pool.cc:224] Handling call
0709 12:19:06.739187 (+ 34614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.739559 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:19:06.761960 (+ 22401us) log.cc:844] Serialized 14 byte log entry
0709 12:19:07.194839 (+432879us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:07.194909 (+    70us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:07.194968 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.195046 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.195204 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17495,"prepare.queue_time_us":17665,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":5618,"raft.queue_time_us":2167,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":35374,"thread_start_us":432196,"threads_started":1}
I20260709 12:19:07.175100 28849 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.289808 12567 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:19:07.249152 28856 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.245388 28848 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.200232 28784 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.227154 28864 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.297811 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:07.368921 12567 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 26.
I20260709 12:19:07.371554 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:07.372622 12569 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:07.311929 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:07.359724 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:07.353086 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:07.345162 12492 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:19:07.393663 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:07.392570 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:07.436465 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:07.348860 13260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2041) took 1319 ms. Trace:
I20260709 12:19:07.437693 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:07.299711 28954 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:19:07.301007 28940 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
W20260709 12:19:07.297823 28948 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.232s	user 0.001s	sys 0.000s
I20260709 12:19:07.270331 28890 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:19:07.441567 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.451740 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.324735 28869 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.323073 28950 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:07.352120 28943 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
W20260709 12:19:07.300375 28952 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
W20260709 12:19:07.297564 28862 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.405s	user 0.019s	sys 0.001s
I20260709 12:19:07.317301 28833 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:07.462939 28689 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.634s	user 0.013s	sys 0.002s
I20260709 12:19:07.463479 28689 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:19:07.429811 28929 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:19:07.328794 28963 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
W20260709 12:19:07.358314 28955 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:19:07.466285 28963 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:07.466897 28963 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:07.323902 28860 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:07.468739 28534 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.973s	user 0.009s	sys 0.005s
I20260709 12:19:07.469605 28854 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:07.470152 28854 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:07.474404 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5912) took 2457 ms. Trace:
W20260709 12:19:07.456225 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 272655us
I20260709 12:19:07.363390 13218 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:07.338122 28953 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:07.320998 28911 consensus_meta.cc:287] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.993s	user 0.013s	sys 0.000s
I20260709 12:19:07.298715 28796 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:19:07.438117 13260 rpcz_store.cc:276] 0709 12:19:06.029096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.046918 (+ 17822us) service_pool.cc:224] Handling call
0709 12:19:06.083484 (+ 36566us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.314537 (+231053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.348441 (+1033904us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54000d11d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:07.348764 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":526795008}
W20260709 12:19:07.448701 28961 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:19:07.455235 28960 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:07.498391 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51134.7us
I20260709 12:19:07.327868 28849 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.344647 28784 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.509410 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:07.335117 28836 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.330587 28861 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.337617 28848 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:07.346354 28927 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:07.338604 28942 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.256s	user 0.002s	sys 0.017s
I20260709 12:19:07.449507 28959 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.445519 28502 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:07.513957 28502 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:07.513782 28959 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.462520 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 41 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.293009 28957 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.515125 28957 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.515642 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.516491 12496 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:19:07.516729 28959 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:07.483206 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.517581 28957 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.475018 12640 rpcz_store.cc:276] 0709 12:19:05.016915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.459486 (+442571us) service_pool.cc:224] Handling call
0709 12:19:05.497751 (+ 38265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.347093 (+849342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.474016 (+1126923us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54002c4f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:07.474314 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1951416192}
I20260709 12:19:07.517743 28962 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.121s
W20260709 12:19:07.488829 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 125512us
W20260709 12:19:07.519366 28956 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.114s	user 0.001s	sys 0.001s
I20260709 12:19:07.519451 13218 leader_election.cc:400] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
I20260709 12:19:07.520157 13218 leader_election.cc:403] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Cancelling election due to peer responding with higher term
W20260709 12:19:07.363989 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.455780 28864 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.453675 28890 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:07.458652 28969 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.522099 28890 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:07.522569 28969 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.525118 28969 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:07.428864 28785 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:07.469293 28534 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.526922 12643 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41.
I20260709 12:19:07.527901 28534 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.528576 28824 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.528673 28962 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election lost for term 51. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
W20260709 12:19:07.532941 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.493923 28430 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:07.533988 28430 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:07.534310 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2466) took 1429 ms. Trace:
I20260709 12:19:07.534828 12497 rpcz_store.cc:276] 0709 12:19:06.104746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.187451 (+ 82705us) service_pool.cc:224] Handling call
0709 12:19:06.198378 (+ 10927us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:06.198828 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.198873 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:06.206616 (+  7743us) log.cc:844] Serialized 26 byte log entry
0709 12:19:06.844963 (+638347us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:06.845007 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.845087 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:07.534078 (+688991us) raft_consensus.cc:1685] finished
0709 12:19:07.534100 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.534251 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5502,"thread_start_us":573880,"threads_started":1,"wal-append.queue_time_us":858842}
I20260709 12:19:07.536046 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:07.536530 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.499158 28721 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:07.538079 28953 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:07.538296 28869 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:19:07.542421 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:07.543311 12641 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 54: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:07.543614 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:07.506147 12536 connection.cc:256] RPC call timeout handler was delayed by -2.715169792000097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
I20260709 12:19:07.544866 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:07.545475 12536 connection.cc:256] RPC call timeout handler was delayed by -2.6692697920000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260709 12:19:07.545867 12536 connection.cc:256] RPC call timeout handler was delayed by -2.2309697919999962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20260709 12:19:07.546126 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1742697920001319s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21200000000000002s for a response
W20260709 12:19:07.546419 12536 connection.cc:256] RPC call timeout handler was delayed by -2.152669792000097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
W20260709 12:19:07.546663 12536 connection.cc:256] RPC call timeout handler was delayed by -2.1256697920000533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
I20260709 12:19:07.509510 28960 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.514451 28502 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:07.521500 13218 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:07.521618 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.546864 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:07.489084 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.548002 12646 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:19:07.525282 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.549010 12493 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:07.534395 28430 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:07.545042 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.544866 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.551077 28752 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:07.553905 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:07.554148 28752 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.538552 28798 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.538655 28861 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.017s
I20260709 12:19:07.555202 28798 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.556599 28752 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.557735 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:07.555148 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:07.559331 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.560019 12497 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:07.497810 28911 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 50. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:07.550997 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
W20260709 12:19:07.546923 12536 connection.cc:256] RPC call timeout handler was delayed by -1.733269792000101s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.261s for a response
W20260709 12:19:07.566475 12536 connection.cc:256] RPC call timeout handler was delayed by -1.683769792000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2665s for a response
W20260709 12:19:07.566928 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1473697920000632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32610000000000006s for a response
I20260709 12:19:07.567855 12536 leader_election.cc:400] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48.
I20260709 12:19:07.568236 12536 leader_election.cc:403] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:07.552134 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.557173 28752 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:07.588033 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.529305 28824 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 33, Time since last communication: 0.028s
I20260709 12:19:07.588205 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:07.781255 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.030s	sys 0.048s
W20260709 12:19:07.844048 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 299584us
W20260709 12:19:07.551712 28943 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:19:07.844384 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:07.728281 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:07.858070 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:07.858982 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.859159 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.859819 12496 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:19:07.860875 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.860780 12646 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 54: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:07.862216 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.858386 28752 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.863759 28976 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:07.686089 28812 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:07.865008 28752 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.510430 28689 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.355s
I20260709 12:19:07.686201 28790 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:07.866669 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.558838 28849 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.316s
I20260709 12:19:07.867373 12496 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
W20260709 12:19:07.867789 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2052) took 2068 ms (client timeout 1708 ms). Trace:
I20260709 12:19:07.868317 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:07.869971 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:07.843714 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:07.571751 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6418) took 2589 ms. Trace:
W20260709 12:19:07.868227 13259 rpcz_store.cc:269] 0709 12:19:05.799227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:05.802109 (+  2882us) service_pool.cc:224] Handling call
0709 12:19:07.867558 (+2065449us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400655254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:07.867699 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3638923008}
I20260709 12:19:07.870885 12568 rpcz_store.cc:276] 0709 12:19:04.982366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.687143 (+1704777us) service_pool.cc:224] Handling call
0709 12:19:06.687723 (+   580us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:06.709475 (+ 21752us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.709521 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:06.718546 (+  9025us) log.cc:844] Serialized 52 byte log entry
0709 12:19:06.900150 (+181604us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:06.900191 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.900265 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:07.298125 (+397860us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:19:07.552233 (+254108us) raft_consensus.cc:1685] finished
0709 12:19:07.552268 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.571678 (+ 19410us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":227,"raft.run_cpu_time_us":1678,"raft.run_wall_time_us":588632,"spinlock_wait_cycles":4060032,"thread_start_us":154747,"threads_started":1,"wal-append.queue_time_us":434735}
I20260709 12:19:07.873114 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:07.846123 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 278458us
I20260709 12:19:07.869958 12496 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:19:07.874593 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:07.554807 28957 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:07.846470 12644 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:07.846804 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.033s	sys 0.054s
W20260709 12:19:07.737672 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:07.589178 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:07.845170 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:07.541815 28796 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:07.865166 28812 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:07.874877 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 366427us
I20260709 12:19:07.877947 12643 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:19:07.650748 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.856554 28784 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:07.704093 28534 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
W20260709 12:19:08.003551 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 106852us
I20260709 12:19:07.887042 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.021602 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:07.890909 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 353447us
I20260709 12:19:07.649122 28798 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:07.876163 28960 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.057312 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:07.906301 12494 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:07.874454 28686 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:07.910754 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:07.896111 28957 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:07.913839 28970 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260709 12:19:06.972702 12566 rpcz_store.cc:276] 0709 12:19:04.922141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.650126 (+1727985us) service_pool.cc:224] Handling call
0709 12:19:06.717462 (+ 67336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.717695 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 33
0709 12:19:06.754638 (+ 36943us) log.cc:844] Serialized 14 byte log entry
0709 12:19:06.967037 (+212399us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:06.967168 (+   131us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:06.967256 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:06.967672 (+   416us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:06.967857 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":160,"prepare.queue_time_us":36921,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":367,"thread_start_us":211586,"threads_started":1}
I20260709 12:19:07.923343 28812 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:07.894366 28796 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.903580 28849 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.031s
I20260709 12:19:07.903360 28689 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 24, Time since last communication: 0.031s
I20260709 12:19:07.884348 28911 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 47. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48.
I20260709 12:19:07.937268 12497 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:19:07.895023 28918 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
W20260709 12:19:07.910006 28780 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.879s	user 0.014s	sys 0.001s
I20260709 12:19:07.952447 12493 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:19:07.953137 12572 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 53 because replica has last-logged OpId of term: 51 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 21.
I20260709 12:19:07.958443 28488 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:08.064028 28686 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:07.968077 28933 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
W20260709 12:19:07.942023 28832 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.028s	user 0.016s	sys 0.000s
I20260709 12:19:07.994004 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:08.101369 12637 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:07.910209 28784 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:08.002401 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 448928us
I20260709 12:19:07.889565 12488 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.24-46.24]
I20260709 12:19:07.971845 28854 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.008719 28534 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:08.025240 28963 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.555s	user 0.012s	sys 0.002s
W20260709 12:19:08.031910 12610 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.032s	sys 0.029s
W20260709 12:19:08.120216 12610 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.032s	sys 0.029s
I20260709 12:19:08.024922 28902 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:08.041154 13217 connection.cc:256] RPC call timeout handler was delayed by -1.5208959880001203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260709 12:19:08.041880 12462 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.132:33793 timed out after 1.918s (SENT) [suppressed 14 similar messages]
I20260709 12:19:08.043211 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.043486 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:08.120707 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 589222us
I20260709 12:19:08.126375 12568 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:08.026827 28836 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:08.050810 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 193422us
I20260709 12:19:08.020655 28883 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.013s
W20260709 12:19:08.052634 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:08.053797 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166874us
W20260709 12:19:08.052700 28776 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.797s	user 0.013s	sys 0.001s
I20260709 12:19:08.062865 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 37.
I20260709 12:19:08.128214 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:08.051836 28802 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:08.070307 28812 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:19:08.073417 28869 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 51.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.002s
I20260709 12:19:08.077055 28931 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:08.061229 28833 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:19:08.153932 28833 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:08.154438 28833 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:19:08.154641 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.155449 12643 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:19:08.156416 28430 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.000s
I20260709 12:19:08.156936 28430 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.108342 28832 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.080076 28957 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 52.30, Last appended by leader: 30, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:07.963135 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2460) took 1937 ms. Trace:
I20260709 12:19:08.093673 28906 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.165112 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3038) took 3421 ms. Trace:
I20260709 12:19:08.162317 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:08.102962 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.172981 12565 rpcz_store.cc:276] 0709 12:19:04.743526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.356358 (+1612832us) service_pool.cc:224] Handling call
0709 12:19:06.356927 (+   569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.058488 (+1701561us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.058523 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:08.058572 (+    49us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:08.058605 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.164825 (+106220us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003aaf54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:08.164873 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.165044 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76714,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":16328,"spinlock_wait_cycles":3269147008,"thread_start_us":279410,"threads_started":1}
I20260709 12:19:08.168268 12645 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 23. Preceding OpId from leader: term: 47 index: 24. (index mismatch)
W20260709 12:19:08.114483 28890 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.005s
W20260709 12:19:08.115882 13218 leader_election.cc:336] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.630s (SENT)
I20260709 12:19:08.117264 12640 raft_consensus.cc:2393] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:08.094288 28790 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:08.130550 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.023s	sys 0.034s
I20260709 12:19:08.136793 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.100360 28933 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:08.145566 12497 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
W20260709 12:19:08.186486 12572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 6092) took 2177 ms (client timeout 2726 ms). Trace:
I20260709 12:19:08.099838 28488 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.148031 28854 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.056838 28125 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:08.073331 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:08.156430 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.171382 12491 rpcz_store.cc:276] 0709 12:19:06.026052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.026198 (+   146us) service_pool.cc:224] Handling call
0709 12:19:06.026572 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:06.195966 (+169394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.196019 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:06.196573 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:19:06.759630 (+563057us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:06.759693 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.962547 (+1202854us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b540020bcd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:07.962611 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:07.962679 (+    68us) raft_consensus.cc:1685] finished
0709 12:19:07.962698 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.963043 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388765440,"thread_start_us":550111,"threads_started":1,"wal-append.queue_time_us":635980,"wal-append.run_cpu_time_us":2283,"wal-append.run_wall_time_us":1004142}
I20260709 12:19:08.179622 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:08.176681 12642 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 37.
I20260709 12:19:08.198830 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.202000 12643 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:08.181368 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:08.182331 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.028s	sys 0.035s
W20260709 12:19:08.204506 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 132080us
I20260709 12:19:08.205595 12567 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 42 index: 24. (index mismatch)
I20260709 12:19:08.205828 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.181157 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:19:08.207294 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.210543 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.180249 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
I20260709 12:19:08.209044 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.186975 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.214250 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:08.215261 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:08.186954 12572 rpcz_store.cc:269] 0709 12:19:05.998828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.040812 (+ 41984us) service_pool.cc:224] Handling call
0709 12:19:08.176399 (+2135587us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54001b26d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:08.176556 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4222615424}
I20260709 12:19:08.222784 12570 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:08.225222 12641 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56. Ops: [55.27-55.27]
W20260709 12:19:08.226214 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46701us
I20260709 12:19:08.226725 12497 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 41: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:08.227516 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:08.235086 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:08.237233 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:08.170521 28980 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.251096 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:08.251502 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.252003 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.258522 12569 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
W20260709 12:19:08.253530 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 116815us
I20260709 12:19:08.242293 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:08.139511 28902 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.260485 12564 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:08.134207 28963 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.184492 28812 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:08.161517 28979 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:08.262279 12645 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:19:08.262428 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:08.256263 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.138868 28803 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.085s
I20260709 12:19:08.263649 12567 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:08.192721 28780 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.143364 28815 consensus_queue.cc:1048] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.092s
I20260709 12:19:08.162640 28836 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.081s
I20260709 12:19:08.266016 12536 leader_election.cc:400] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:19:08.281241 12536 leader_election.cc:403] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Cancelling election due to peer responding with higher term
I20260709 12:19:08.142889 28968 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.088s
I20260709 12:19:08.267182 12563 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20260709 12:19:08.284307 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.285046 12567 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
W20260709 12:19:08.273721 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.273703 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.230952 28983 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:08.190049 28918 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:08.183219 28931 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.228921 28838 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:08.142417 28776 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:08.253000 28689 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
W20260709 12:19:08.313297 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 271414us
I20260709 12:19:08.288122 12572 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 37: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:08.242753 28854 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:08.290707 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53607.1us
I20260709 12:19:08.190903 28883 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.079s
I20260709 12:19:08.191258 28890 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.278554 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:08.296921 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120932us
I20260709 12:19:08.299463 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.189966 28790 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.266180 28979 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:08.184827 28906 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.309233 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:08.273458 28601 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260709 12:19:08.311810 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:08.311359 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.327318 12570 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:08.332913 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.325525 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.296346 28983 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:08.344319 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.359202 12639 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 43: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:08.361266 12564 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:08.273000 28534 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.323952 28689 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:08.365485 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.282212 28780 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.266603 28980 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.342404 28838 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:08.346101 28918 raft_consensus.cc:2764] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election decision vote started in defunct term 50: won
I20260709 12:19:08.309185 28982 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election lost for term 41. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42.
I20260709 12:19:08.340121 28784 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:08.339134 28854 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.352226 28983 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:08.328277 28979 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:08.401266 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2469) took 1228 ms. Trace:
I20260709 12:19:08.389048 28689 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:08.398234 12494 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
W20260709 12:19:08.398250 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72836.3us
I20260709 12:19:08.429986 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.430958 13259 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 53: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:08.398658 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 282811us
W20260709 12:19:08.407436 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73642us
I20260709 12:19:08.420131 12492 rpcz_store.cc:276] 0709 12:19:07.172641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.172849 (+   208us) service_pool.cc:224] Handling call
0709 12:19:07.173309 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.388854 (+215545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.400955 (+1012101us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:08.401192 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":393988480}
I20260709 12:19:08.432909 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.433540 12492 pending_rounds.cc:85] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 23
I20260709 12:19:08.433947 12492 pending_rounds.cc:107] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 53.24
I20260709 12:19:08.422905 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:08.434238 12492 raft_consensus.cc:2889] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:08.423720 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 188749us
W20260709 12:19:08.435632 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.390s (SENT)
I20260709 12:19:08.397694 28803 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:08.396726 28984 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:08.339120 28933 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.28, Last appended by leader: 28, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:07.153770 12571 rpcz_store.cc:276] 0709 12:19:05.999401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.846115 (+846714us) service_pool.cc:224] Handling call
0709 12:19:06.859862 (+ 13747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:06.860095 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:19:06.884038 (+ 23943us) log.cc:844] Serialized 14 byte log entry
0709 12:19:07.123317 (+239279us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:07.123433 (+   116us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:07.124470 (+  1037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.124586 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:07.124807 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25734,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":52061,"thread_start_us":217519,"threads_started":1}
I20260709 12:19:08.441084 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.526836 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.527077 13259 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:08.662588 12645 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
I20260709 12:19:08.480386 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:08.617345 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 40 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.616565 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.664198 12641 raft_consensus.cc:2368] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41.
I20260709 12:19:08.664443 13256 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:08.443784 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.621147 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4644) took 1078 ms. Trace:
I20260709 12:19:08.455616 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:08.620867 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:08.424917 28970 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:19:08.489637 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:08.617817 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2309) took 1707 ms. Trace:
I20260709 12:19:08.492808 12640 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:19:08.666385 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:08.444638 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.615705 12493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4637) took 1163 ms. Trace:
I20260709 12:19:08.659677 28950 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:08.426702 28812 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:08.400171 28981 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:08.488662 28985 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 12:19:08.513913 28430 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.665458 12488 rpcz_store.cc:276] 0709 12:19:07.543081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.888910 (+345829us) service_pool.cc:224] Handling call
0709 12:19:07.889299 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.108426 (+219127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.552852 (+444426us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b540029b920. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf576adc0 00007ffbf57f6a02 00007ffbf57f2dd6 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:19:08.621053 (+ 68201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70746368}
I20260709 12:19:08.679543 12638 rpcz_store.cc:276] 0709 12:19:06.909964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.911764 (+  1800us) service_pool.cc:224] Handling call
0709 12:19:07.082080 (+170316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:07.162618 (+ 80538us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.569151 (+1406533us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b5400154354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:08.617734 (+ 48583us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172998912}
W20260709 12:19:08.579943 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 47790.2us
I20260709 12:19:08.708806 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.713424 12570 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:08.523545 28854 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.445739 12569 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:08.718180 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:08.447028 28982 raft_consensus.cc:2764] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election decision vote started in defunct term 46: won
I20260709 12:19:08.670782 28125 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:08.667215 12497 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:08.734773 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:08.668670 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.737548 12565 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:08.682392 13257 raft_consensus.cc:2393] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:08.697665 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210773us
I20260709 12:19:08.697983 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:08.450340 28488 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:08.697580 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.682436 28950 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:08.749375 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 3.011s (SENT)
I20260709 12:19:08.749897 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.671648 13261 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 30. Preceding OpId from leader: term: 46 index: 31. (index mismatch)
I20260709 12:19:08.676219 12637 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:08.702072 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.672080 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:08.702492 12645 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 33. Preceding OpId from leader: term: 50 index: 34. (index mismatch)
I20260709 12:19:08.681154 12493 rpcz_store.cc:276] 0709 12:19:07.451766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.681782 (+230016us) service_pool.cc:224] Handling call
0709 12:19:07.843857 (+162075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.091979 (+248122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.565889 (+473910us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:08.614686 (+ 48797us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":249175808}
I20260709 12:19:08.752213 13258 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:08.705276 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.707759 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:08.667145 28935 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.846s	user 0.005s	sys 0.001s
W20260709 12:19:08.671365 28686 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.563s	user 0.014s	sys 0.001s
I20260709 12:19:08.755503 28686 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.710060 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:08.757664 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47683.4us
I20260709 12:19:08.757913 12488 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:08.419795 28796 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:08.761154 28967 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:19:08.762012 28967 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:08.762341 28686 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:08.763583 13256 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:19:08.715610 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.527209 28990 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.765194 13253 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:08.765695 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:08.748265 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 433061us
I20260709 12:19:08.533818 28959 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:08.718366 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:08.666682 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:08.723078 28798 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:08.490185 28869 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:08.703414 28975 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.844s	user 0.009s	sys 0.000s
W20260709 12:19:08.703130 28986 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.302s	user 0.001s	sys 0.002s
W20260709 12:19:08.683642 28502 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.087s	user 0.010s	sys 0.005s
I20260709 12:19:08.555438 28832 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.672358 28933 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:08.680882 12610 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:08.817602 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.618307 28721 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:08.772166 28815 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.033s
I20260709 12:19:08.674590 28980 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:08.671707 28902 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:08.682060 28883 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:08.698232 28724 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.427584 28997 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.291s
I20260709 12:19:08.743605 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:08.798080 28968 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:08.668190 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:08.847076 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
W20260709 12:19:08.746366 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 256852us
I20260709 12:19:08.593485 28833 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.556699 28992 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.190s
I20260709 12:19:08.766455 13254 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 54.
I20260709 12:19:08.856202 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.763304 28967 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 40.28, Last appended by leader: 28, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.718247 28911 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.772921 13259 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 56.
I20260709 12:19:08.669960 28534 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:08.776712 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 58624.8us
I20260709 12:19:08.775846 12570 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:08.872876 12491 raft_consensus.cc:2368] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for earlier term 37. Current term is 38.
I20260709 12:19:08.779361 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:08.742550 28963 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.742367 28931 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:08.743861 28994 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:08.710474 28981 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.790298 28849 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 48.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.012s
I20260709 12:19:08.666122 28984 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.774372 28959 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:08.821466 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 140646us
W20260709 12:19:08.911989 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.747274 28430 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 46.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.051s
I20260709 12:19:08.827083 28721 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:08.783268 28776 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:08.938205 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:08.799153 28890 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:08.769119 28990 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.796380 28812 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 50.30, Last appended by leader: 30, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:08.856144 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:08.859630 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6211) took 1710 ms. Trace:
I20260709 12:19:08.832214 28832 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:08.865547 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:08.793774 28869 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.870025 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:08.816253 28502 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.856950 28906 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:08.893922 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:08.782642 28876 raft_consensus.cc:3060] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:08.913498 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 134226us
I20260709 12:19:08.913873 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:08.882567 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 350063us
W20260709 12:19:09.069996 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:09.070587 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 156780us
I20260709 12:19:08.922325 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:08.929808 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:09.072013 12496 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:08.936199 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:08.745333 12639 raft_consensus.cc:2393] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:08.942258 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:09.073892 12643 raft_consensus.cc:2376] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50. Re-sending same reply.
W20260709 12:19:08.945101 28979 consensus_meta.cc:287] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.504s	user 0.007s	sys 0.002s
I20260709 12:19:08.864291 28488 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.062s
I20260709 12:19:09.085575 28488 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:19:09.086000 28488 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:09.086911 28488 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.088126 28488 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:19:09.058121 28838 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.945792 28833 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:08.882598 28780 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:08.950204 28849 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:08.912240 28994 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.937477 28686 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.098s
I20260709 12:19:08.836829 28883 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.937561 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5936) took 1074 ms. Trace:
I20260709 12:19:08.842422 28724 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:08.953202 13258 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:08.959941 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:08.967070 28832 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:09.111485 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 151633us
I20260709 12:19:08.962533 12495 rpcz_store.cc:276] 0709 12:19:07.149470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.155271 (+  5801us) service_pool.cc:224] Handling call
0709 12:19:07.172153 (+ 16882us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:07.172726 (+   573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:07.172773 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:07.173521 (+   748us) log.cc:844] Serialized 26 byte log entry
0709 12:19:07.494404 (+320883us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:19:07.494445 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:07.494527 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:08.357299 (+862772us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:08.855093 (+497794us) raft_consensus.cc:1685] finished
0709 12:19:08.855119 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.859561 (+  4442us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":166,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":9256,"spinlock_wait_cycles":37025536,"thread_start_us":264883,"threads_started":1,"wal-append.queue_time_us":497340}
I20260709 12:19:09.015820 28960 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:08.959703 12570 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
W20260709 12:19:08.969233 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 99309.8us
I20260709 12:19:08.878150 28911 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:08.946535 28995 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:08.936441 28931 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:08.967837 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 219937us
I20260709 12:19:08.853785 28950 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.076079 12489 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
W20260709 12:19:09.082901 29006 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:19:09.083232 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:09.083683 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.084319 28999 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:19:09.089133 28838 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.091122 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:09.013079 28963 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:09.106666 12644 rpcz_store.cc:276] 0709 12:19:07.862795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.166424 (+303629us) service_pool.cc:224] Handling call
0709 12:19:08.166807 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.188925 (+ 22118us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.188979 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.202241 (+ 13262us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.420374 (+218133us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:08.420441 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.420527 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:08.937293 (+516766us) raft_consensus.cc:1685] finished
0709 12:19:08.937317 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:08.937484 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32003,"raft.queue_time_us":22054,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":34024,"thread_start_us":145617,"threads_started":1,"wal-append.queue_time_us":488428}
I20260709 12:19:09.106683 12496 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:09.166893 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:09.015226 13254 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 54 index: 35. Preceding OpId from leader: term: 56 index: 36. (index mismatch)
I20260709 12:19:09.112380 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.122567 13217 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:09.169315 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46846.7us
I20260709 12:19:09.171309 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.172255 28982 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260709 12:19:09.080390 28969 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:09.173733 28969 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.158887 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:09.176872 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:09.094089 28849 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:09.178805 28849 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 48.29, Last appended by leader: 29, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.180730 28849 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:09.184095 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:09.168745 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56391.2us
I20260709 12:19:09.168762 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:09.187289 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96236.1us
I20260709 12:19:09.121233 28832 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:09.126148 28780 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:09.194749 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.196043 12570 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:19:09.136050 28995 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:09.139483 28853 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:09.201440 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:09.188385 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:09.143384 28796 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.184973 28963 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.194048 12611 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:09.160622 28785 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:09.191763 28776 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:19:09.260509 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72190.8us
I20260709 12:19:09.138000 28960 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:09.268361 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 67073.6us
I20260709 12:19:09.288094 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:09.080593 28959 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.27, Last appended by leader: 27, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.257920 28995 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:09.096158 28979 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:09.307533 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 136241us
W20260709 12:19:09.282639 12537 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.004s	sys 0.010s
I20260709 12:19:09.290500 28776 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:09.274281 28796 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.292114 28833 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20260709 12:19:09.193492 28780 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.296252 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:09.162250 28931 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.305598 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:09.308604 12537 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.906s	user 0.005s	sys 0.010s
W20260709 12:19:09.287415 29003 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:19:09.274827 28963 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:09.320358 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.355304 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:09.322743 12570 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:19:09.202172 28430 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.092s
I20260709 12:19:09.262071 28980 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:19:09.278347 28890 raft_consensus.cc:2804] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:09.285559 28854 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election won for term 55
W20260709 12:19:09.346910 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 944876us
I20260709 12:19:09.292457 28911 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.308379 28999 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:09.349184 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:09.356297 12495 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 54.
I20260709 12:19:09.378937 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:09.359467 12567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.011s	sys 0.007s
I20260709 12:19:09.364665 12637 raft_consensus.cc:2368] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
I20260709 12:19:09.309415 28776 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.346030 28959 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:09.345489 28780 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:09.383541 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 189515us
I20260709 12:19:09.373796 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.386406 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:09.382056 12567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.011s	sys 0.007s
I20260709 12:19:09.269126 28853 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.325165 28998 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.400154 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:09.361109 28982 raft_consensus.cc:2764] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader election decision vote started in defunct term 37: won
I20260709 12:19:09.354480 28992 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:09.364845 29001 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.409s	user 0.001s	sys 0.000s
I20260709 12:19:09.396790 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.362252 28980 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.388288 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:09.382815 29017 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:19:09.363341 28686 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:09.380039 28854 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.415486 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.421126 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:09.391467 28776 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:09.383945 28999 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:09.383814 28890 raft_consensus.cc:697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:09.394119 29000 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:09.397773 28833 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
W20260709 12:19:09.426992 29005 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:19:09.443295 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.443465 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:09.419162 28984 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.430599 28686 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:09.446448 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.031s	sys 0.043s
W20260709 12:19:09.476966 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.031s	sys 0.043s
W20260709 12:19:09.477418 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 694461us
I20260709 12:19:09.478988 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:09.463850 29007 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.327s	user 0.000s	sys 0.002s
W20260709 12:19:09.488384 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 100857us
I20260709 12:19:09.469126 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6227) took 1310 ms. Trace:
I20260709 12:19:09.430080 28502 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.426371 28980 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:09.434916 28854 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:09.409310 28998 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.427896 28981 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:09.474474 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 279819us
I20260709 12:19:09.417534 28911 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.028s
I20260709 12:19:09.597644 29028 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.442198 28999 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:09.451406 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.489545 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.601950 29028 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.604367 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:09.604964 13261 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.31-46.31]
I20260709 12:19:09.606186 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.607256 28833 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:19:09.608387 28994 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.609187 28994 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:19:09.591281 28984 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:09.489428 12492 rpcz_store.cc:276] 0709 12:19:08.158900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.432920 (+274020us) service_pool.cc:224] Handling call
0709 12:19:08.433325 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.434953 (+  1628us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.434990 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.435524 (+   534us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.775521 (+339997us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:08.775583 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.775676 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.468902 (+693226us) raft_consensus.cc:1685] finished
0709 12:19:09.468924 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.469066 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":339582,"threads_started":1,"wal-append.queue_time_us":441705}
I20260709 12:19:09.610634 28984 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.441529 28883 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:09.471684 29009 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 12:19:09.424335 28992 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:09.504961 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 364222us
I20260709 12:19:09.614840 13218 leader_election.cc:400] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 51. Current term is 52.
I20260709 12:19:09.615434 28992 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:09.615871 13218 leader_election.cc:400] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41.
I20260709 12:19:09.601075 28981 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.616279 13218 leader_election.cc:403] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 40 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:09.616508 28981 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.613260 28984 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:09.618091 29016 log.cc:927] Time spent T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 12:19:09.619735 28981 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:09.620533 28981 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:09.620895 28981 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.621275 28981 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:09.612931 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 123529us
I20260709 12:19:09.621364 28883 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:19:09.601791 29039 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:09.622448 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:09.617805 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.622787 28601 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 39 FOLLOWER]: Advancing to term 41
I20260709 12:19:09.624233 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.619655 28994 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:09.609431 28833 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
W20260709 12:19:09.626132 29024 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.332s	user 0.002s	sys 0.000s
I20260709 12:19:09.626468 28833 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:09.624879 28890 consensus_queue.cc:237] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.29, Last appended by leader: 29, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.626996 28833 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:09.630506 29023 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:19:09.626503 28994 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.632897 28686 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.634388 28686 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:09.635461 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:09.637166 28979 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.637847 28984 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:19:09.464267 28869 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.432742 29030 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.073s
I20260709 12:19:09.639987 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:09.689814 13254 raft_consensus.cc:2376] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46. Re-sending same reply.
I20260709 12:19:09.690999 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.640734 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6449) took 1198 ms. Trace:
I20260709 12:19:09.691789 28854 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.434160 28721 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:09.638322 28983 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.201s	user 0.013s	sys 0.000s
I20260709 12:19:09.646221 28931 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.483109 28724 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.508292 28960 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:09.638641 29000 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 53.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.008s
I20260709 12:19:09.646971 28911 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:09.647307 29033 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:09.648053 28796 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.648922 29029 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:19:09.606204 29010 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20260709 12:19:09.809795 29019 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.810621 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6454) took 1366 ms. Trace:
I20260709 12:19:09.810544 29019 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.811120 12568 rpcz_store.cc:276] 0709 12:19:08.443831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.623474 (+179643us) service_pool.cc:224] Handling call
0709 12:19:08.624018 (+   544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.681733 (+ 57715us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.681771 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.714333 (+ 32562us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.016190 (+301857us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:19:09.016437 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.016578 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.810370 (+793792us) raft_consensus.cc:1685] finished
0709 12:19:09.810392 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.810556 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37720,"raft.queue_time_us":57829,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":11319,"thread_start_us":238074,"threads_started":1,"wal-append.queue_time_us":530650}
I20260709 12:19:09.813086 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.650262 28998 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.669736 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:09.814128 29034 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:19:09.814339 13257 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:09.668911 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.487962 28838 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:09.667683 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 318529us
I20260709 12:19:09.674841 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.816447 12570 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 54.
I20260709 12:19:09.816519 28854 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:09.816843 12568 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:19:09.817410 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.817788 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:09.818091 12570 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:09.818328 12568 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 55
I20260709 12:19:09.819123 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.819974 12570 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:09.721799 28911 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:09.821849 28864 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.822552 28864 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.822634 28911 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 46.31, Last appended by leader: 31, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.824156 28911 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:09.825879 28534 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.948s	user 0.014s	sys 0.000s
I20260709 12:19:09.831542 28780 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.831895 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.842043 28864 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.679719 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:09.849870 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.006s	sys 0.027s
W20260709 12:19:09.850426 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.006s	sys 0.027s
I20260709 12:19:09.850750 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:09.851537 28984 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:09.852206 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:09.683144 12640 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 49 index: 31. (index mismatch)
I20260709 12:19:09.851579 28721 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.681128 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:09.686568 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:09.854229 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 167740us
I20260709 12:19:09.692070 12567 rpcz_store.cc:276] 0709 12:19:08.442226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.445418 (+  3192us) service_pool.cc:224] Handling call
0709 12:19:08.508400 (+ 62982us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.619866 (+111466us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:08.619912 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.676549 (+ 56637us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.387960 (+711411us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:09.388029 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.388138 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.640480 (+252342us) raft_consensus.cc:1685] finished
0709 12:19:09.640506 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.640671 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22989,"raft.queue_time_us":110838,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":13260,"thread_start_us":661903,"threads_started":1,"wal-append.queue_time_us":752827}
I20260709 12:19:09.854566 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 38 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:09.683880 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:09.855094 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1879147049999119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260709 12:19:09.855299 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.855778 28853 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.856166 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:09.856559 29008 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:09.857002 28933 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.033s	user 0.000s	sys 0.013s
I20260709 12:19:09.857121 29008 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.857967 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:09.858789 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:09.859397 12640 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:19:09.859889 29008 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:09.860258 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:09.860455 29008 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:09.860982 12640 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 55: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:09.694967 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:09.861788 12645 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
W20260709 12:19:09.862011 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 167177us
I20260709 12:19:09.862967 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:09.699337 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6453) took 1255 ms. Trace:
I20260709 12:19:09.863546 12645 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:09.863780 12564 rpcz_store.cc:276] 0709 12:19:08.443562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.621772 (+178210us) service_pool.cc:224] Handling call
0709 12:19:08.670960 (+ 49188us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.671286 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.671315 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.671894 (+   579us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.881073 (+209179us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:08.881137 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.881243 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.699036 (+817793us) raft_consensus.cc:1685] finished
0709 12:19:09.699066 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.699246 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":207166,"threads_started":1,"wal-append.queue_time_us":438563}
I20260709 12:19:09.863190 28780 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.865353 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:09.723624 28960 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:09.865993 28780 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:09.866256 12564 raft_consensus.cc:2410] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 43 because replica has last-logged OpId of term: 40 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 22.
I20260709 12:19:09.719589 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6455) took 1275 ms. Trace:
W20260709 12:19:09.731559 28488 consensus_meta.cc:287] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.642s	user 0.015s	sys 0.002s
I20260709 12:19:09.778600 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.790988 28855 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:09.792233 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6200) took 3803 ms. Trace:
I20260709 12:19:09.855324 12638 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:19:09.794664 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.803997 28957 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
W20260709 12:19:09.613245 29014 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
I20260709 12:19:09.868513 12490 rpcz_store.cc:276] 0709 12:19:05.988513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:06.006930 (+ 18417us) service_pool.cc:224] Handling call
0709 12:19:06.041045 (+ 34115us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:06.311020 (+269975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:06.311068 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:07.758936 (+1447868us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b18004f1500. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a
0709 12:19:08.481691 (+722755us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b2c000014c8. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf40eca11 00007ffbf40fc372 00007ffbf582ea01 00007ffbf57f343a 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2
0709 12:19:08.574862 (+ 93171us) log.cc:844] Serialized 52 byte log entry
0709 12:19:08.893224 (+318362us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:08.945941 (+ 52717us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.151297 (+205356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.743478 (+592181us) spinlock_profiling.cc:243] Waited 50.6 ms on lock 0x7b18004f1500. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:19:09.791753 (+ 48275us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:09.791816 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.791900 (+    84us) raft_consensus.cc:1685] finished
0709 12:19:09.791920 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.792116 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":140941,"prepare.queue_time_us":58071,"prepare.run_cpu_time_us":9415,"prepare.run_wall_time_us":43359,"raft.queue_time_us":87843,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":19673,"spinlock_wait_cycles":679248768,"thread_start_us":380387,"threads_started":2,"wal-append.queue_time_us":242267}
I20260709 12:19:09.871538 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2476) took 1527 ms. Trace:
I20260709 12:19:09.676803 12644 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 22. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
I20260709 12:19:09.872836 13259 raft_consensus.cc:2393] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:09.873291 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2287) took 1140 ms. Trace:
I20260709 12:19:09.852152 28984 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.854327 28721 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Leader election won for term 38
W20260709 12:19:09.854857 28876 consensus_meta.cc:287] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.761s	user 0.009s	sys 0.003s
W20260709 12:19:09.898820 12566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.014s	sys 0.028s
I20260709 12:19:09.616204 28992 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:19:09.724749 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 257645us
W20260709 12:19:09.706130 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.848115 28833 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.856212 12567 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 48 because replica has last-logged OpId of term: 46 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
I20260709 12:19:09.873533 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.693329 28869 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:09.867739 12571 rpcz_store.cc:276] 0709 12:19:08.444194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.660921 (+216727us) service_pool.cc:224] Handling call
0709 12:19:08.710014 (+ 49093us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:09.056156 (+346142us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:09.056206 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:09.057341 (+  1135us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.158482 (+101141us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:09.158611 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.158817 (+   206us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.719312 (+560495us) raft_consensus.cc:1685] finished
0709 12:19:09.719335 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.719514 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7967,"raft.queue_time_us":13001,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":18010,"thread_start_us":405629,"threads_started":2,"wal-append.queue_time_us":224436}
I20260709 12:19:09.875949 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6461) took 1120 ms. Trace:
W20260709 12:19:09.876288 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60192.5us
I20260709 12:19:09.912848 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.860929 29008 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:09.871948 12497 rpcz_store.cc:276] 0709 12:19:08.344320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.736248 (+391928us) service_pool.cc:224] Handling call
0709 12:19:08.736747 (+   499us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:08.776431 (+ 39684us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.776478 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:08.791389 (+ 14911us) log.cc:844] Serialized 52 byte log entry
0709 12:19:08.964414 (+173025us) raft_consensus.cc:1645] Marking committed up to 36
0709 12:19:09.110835 (+146421us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.394636 (+283801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.394777 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.871297 (+476520us) raft_consensus.cc:1685] finished
0709 12:19:09.871329 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.871474 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77309,"prepare.queue_time_us":114998,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":293930,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":16552,"thread_start_us":474471,"threads_started":3,"wal-append.queue_time_us":573873}
I20260709 12:19:09.879251 12568 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:09.873884 13259 rpcz_store.cc:276] 0709 12:19:08.733289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.867695 (+134406us) service_pool.cc:224] Handling call
0709 12:19:09.778281 (+910586us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:09.873231 (+ 94950us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33422720}
W20260709 12:19:09.899874 12566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.015s	sys 0.028s
I20260709 12:19:09.911800 12564 rpcz_store.cc:276] 0709 12:19:08.755520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.866929 (+1111409us) service_pool.cc:224] Handling call
0709 12:19:09.867268 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.867459 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:09.875612 (+  8153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:09.875659 (+    47us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:09.875717 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.875778 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.875888 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:09.855803 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1753147049998915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20260709 12:19:09.843586 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:09.910354 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:09.919039 12638 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:19:09.908633 12645 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:19:10.081552 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171285us
I20260709 12:19:10.082079 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:09.920693 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6450) took 1478 ms. Trace:
I20260709 12:19:09.958817 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:10.021557 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:09.923233 28854 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.044s
I20260709 12:19:09.708803 28983 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:09.832041 28689 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.391s	user 0.009s	sys 0.005s
I20260709 12:19:09.813112 29019 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:09.831547 29039 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:09.989363 28780 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:09.925523 28721 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:10.020558 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 46 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:09.961486 12490 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: [41.27-43.28]
I20260709 12:19:09.909214 28876 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:09.962916 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.110479 12571 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:09.950740 28855 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:09.968245 28999 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:10.115144 28995 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.791s	user 0.012s	sys 0.001s
I20260709 12:19:10.115684 28995 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.116436 28999 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:09.961486 28833 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:09.950740 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 77265.4us
I20260709 12:19:10.117883 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:09.918205 28869 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:10.118801 29031 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
W20260709 12:19:10.054850 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:09.961016 28601 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election lost for term 40. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for earlier term 40. Current term is 41.
I20260709 12:19:10.118903 28869 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:09.965157 28853 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.123407 28853 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:10.045866 12536 connection.cc:256] RPC call timeout handler was delayed by -1.1078147049997824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 12:19:10.126024 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0961147049997635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
W20260709 12:19:10.126395 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0880147049997504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1628s for a response
W20260709 12:19:10.126668 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0682147049999458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260709 12:19:10.126919 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0160147049998614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260709 12:19:10.127179 12536 connection.cc:256] RPC call timeout handler was delayed by -1.0097147049998512s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
I20260709 12:19:10.128539 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:09.898933 28838 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.105s
I20260709 12:19:09.991042 29000 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:09.966061 28992 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:19:09.950987 12641 raft_consensus.cc:2393] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:09.964324 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51535.6us
I20260709 12:19:10.132030 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:10.050102 13262 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.000s	sys 0.020s
W20260709 12:19:10.133030 13262 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.000s	sys 0.021s
W20260709 12:19:10.080371 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 439330us
I20260709 12:19:10.082713 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:10.138386 12645 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:10.139725 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.083418 13257 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:10.082744 12638 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:10.084306 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.150882 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:10.151855 12638 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:10.153069 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:10.082978 12572 rpcz_store.cc:276] 0709 12:19:08.442599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.445549 (+  2950us) service_pool.cc:224] Handling call
0709 12:19:08.526271 (+ 80722us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.618927 (+ 92656us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.618959 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.619462 (+   503us) log.cc:844] Serialized 26 byte log entry
0709 12:19:08.853919 (+234457us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:08.853988 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:08.854133 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:09.920439 (+1066306us) raft_consensus.cc:1685] finished
0709 12:19:09.920462 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:09.920627 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":104537,"raft.queue_time_us":134029,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":96,"thread_start_us":128145,"threads_started":1,"wal-append.queue_time_us":325965}
I20260709 12:19:10.077607 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:10.159034 12638 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 48 because replica has last-logged OpId of term: 46 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
I20260709 12:19:10.160488 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:10.092420 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:10.094151 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.094177 29045 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:10.113778 28855 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:09.969920 28998 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:10.172459 28853 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:10.172652 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54881.4us
I20260709 12:19:10.173324 29000 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:10.180387 28998 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:10.173595 28992 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:19:10.181182 28998 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:10.181847 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:10.181200 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.174441 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:10.184218 12645 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:10.179368 29028 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:10.162497 12564 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:10.195830 12494 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:10.085714 28796 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:10.117812 29025 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
W20260709 12:19:10.224550 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 84914.5us
I20260709 12:19:10.225564 12536 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:10.200812 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:10.168995 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:10.227452 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 95521.7us
I20260709 12:19:10.228397 12609 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.230454 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 37 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:10.230662 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.231277 12645 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:10.231812 12572 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:10.236493 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.203979 12492 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:10.075924 28981 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:10.253021 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 84120.5us
I20260709 12:19:10.266156 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.267030 12564 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:10.202696 28833 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:10.259799 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.241292 12571 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:10.270262 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:10.285512 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:10.224689 29045 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:10.279347 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:10.241292 12496 raft_consensus.cc:2468] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:19:10.298799 12645 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:10.302053 12572 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 30. Preceding OpId from leader: term: 49 index: 31. (index mismatch)
I20260709 12:19:10.309969 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6462) took 1554 ms. Trace:
I20260709 12:19:10.257459 28853 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:10.308749 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:10.327612 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:19:10.264460 29028 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:10.310464 12572 rpcz_store.cc:276] 0709 12:19:08.755786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.268836 (+1513050us) service_pool.cc:224] Handling call
0709 12:19:10.276002 (+  7166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.309494 (+ 33492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:10.309891 (+   397us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:10.222718 28502 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.066s
W20260709 12:19:10.328286 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91860.4us
I20260709 12:19:10.267813 28992 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:10.277156 28833 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.107465 28689 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.370514 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.243448 28796 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:10.370137 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.626s (SENT)
I20260709 12:19:10.300835 28534 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:10.326411 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:10.291497 28995 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:10.300383 29043 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:10.399760 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:10.417367 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.433645 12572 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:10.327035 28999 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:10.369220 29028 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.431501 12493 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 53: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:10.443704 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3052) took 1770 ms. Trace:
I20260709 12:19:10.455561 12645 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 26. Preceding OpId from leader: term: 47 index: 28. (index mismatch)
I20260709 12:19:10.395359 28960 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.386854 28833 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:10.398618 28534 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.461561 12564 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:10.499696 28534 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:10.454841 12566 rpcz_store.cc:276] 0709 12:19:08.672967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.799554 (+126587us) service_pool.cc:224] Handling call
0709 12:19:08.800134 (+   580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.825861 (+ 25727us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:19:08.964073 (+138212us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.923512 (+959439us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:09.971724 (+ 48212us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.971984 (+   260us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:10.248718 (+276734us) log.cc:844] Serialized 14 byte log entry
0709 12:19:10.441376 (+192658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:10.443285 (+  1909us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:10.443355 (+    70us) raft_consensus.cc:1685] finished
0709 12:19:10.443380 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:10.443609 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9937,"prepare.queue_time_us":709537,"prepare.run_cpu_time_us":142,"prepare.run_wall_time_us":10800,"raft.queue_time_us":114558,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":25062,"thread_start_us":1435498,"threads_started":5,"wal-append.queue_time_us":700978}
W20260709 12:19:10.463238 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170015us
W20260709 12:19:10.501096 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106235us
I20260709 12:19:10.395859 28876 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:10.498198 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:10.386848 29053 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:10.501646 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.502748 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.504575 12566 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:10.505616 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.506453 12564 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:10.515985 28966 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.747s	user 0.007s	sys 0.021s
W20260709 12:19:10.520975 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150206us
I20260709 12:19:10.526105 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.446909 28488 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:10.439761 29043 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:10.540786 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:10.449486 28984 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:10.539414 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.543304 12572 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:10.328709 28721 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 36.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.462642 29028 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:10.509194 29008 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:10.555255 28984 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:10.545750 28853 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:10.557001 28853 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 48.29, Last appended by leader: 29, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.557837 28991 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:10.558415 28991 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:10.559373 28991 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.27, Last appended by leader: 27, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:10.560509 28991 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:10.561570 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:10.562157 28983 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:10.397178 28992 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:10.562362 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:10.533911 28966 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.786s	user 0.008s	sys 0.021s
I20260709 12:19:10.388638 28780 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.28, Last appended by leader: 28, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:10.547457 28849 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.366s	user 0.018s	sys 0.000s
I20260709 12:19:10.573149 29051 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
W20260709 12:19:10.573463 29050 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:19:10.574034 29055 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:10.578399 28902 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:10.570693 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:10.547647 29043 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:10.720018 28981 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:10.584106 28950 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:10.611523 29051 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:10.585283 29045 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.27, Last appended by leader: 27, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:10.722644 29051 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:10.727428 28902 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:10.728106 28902 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:19:10.584472 29056 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260709 12:19:10.527871 28960 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:10.563016 29025 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:19:10.466734 28999 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.589362 12564 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:10.600185 29008 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:10.566432 28911 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.742s	user 0.016s	sys 0.004s
W20260709 12:19:10.585039 29047 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.328s	user 0.001s	sys 0.000s
W20260709 12:19:10.569649 29040 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.482s	user 0.002s	sys 0.000s
I20260709 12:19:10.708307 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.789813 12566 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 55: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:10.791143 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.791805 12566 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
W20260709 12:19:10.738924 28686 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.104s	user 0.010s	sys 0.005s
W20260709 12:19:10.709295 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.528409 28876 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:10.647697 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.746129 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:10.802913 12564 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
W20260709 12:19:10.740554 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 243506us
I20260709 12:19:10.732343 29055 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:10.793044 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:10.658593 28776 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.214s	user 0.009s	sys 0.004s
W20260709 12:19:10.584401 29042 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:19:10.737974 29033 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:10.768009 28980 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.169s	user 0.006s	sys 0.006s
I20260709 12:19:10.819662 28992 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:10.785917 29008 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.738590 29065 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
W20260709 12:19:10.789515 28963 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.420s	user 0.012s	sys 0.003s
I20260709 12:19:10.824743 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:10.825356 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.863087 12564 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 53 because replica has last-logged OpId of term: 51 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 21.
I20260709 12:19:10.832674 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:10.864717 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.866510 12566 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:10.864261 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 97603.8us
I20260709 12:19:10.868356 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:10.869153 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.869346 12564 raft_consensus.cc:2393] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:10.869598 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 308111us
I20260709 12:19:10.871040 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.828896 28911 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
W20260709 12:19:10.862682 28832 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.604s	user 0.014s	sys 0.002s
I20260709 12:19:10.876291 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:10.864161 28812 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.024s	sys 0.026s
I20260709 12:19:10.883894 28689 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:10.888116 28534 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:10.865778 28855 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.007s	sys 0.045s
I20260709 12:19:10.890185 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:10.901930 12489 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4643) took 3398 ms (client timeout 1650 ms). Trace:
I20260709 12:19:10.905421 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2059) took 1827 ms. Trace:
W20260709 12:19:10.909512 12489 rpcz_store.cc:269] 0709 12:19:07.498835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.515351 (+ 16516us) service_pool.cc:224] Handling call
0709 12:19:10.893788 (+3378437us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540034e754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:10.897019 (+  3231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3586183552}
I20260709 12:19:10.910506 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3055) took 2150 ms. Trace:
I20260709 12:19:10.911023 12564 rpcz_store.cc:276] 0709 12:19:08.759787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.905684 (+2145897us) service_pool.cc:224] Handling call
0709 12:19:10.909954 (+  4270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.910166 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:10.910192 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:10.910237 (+    45us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:10.910265 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:10.910321 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:10.910444 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:10.909968 13258 rpcz_store.cc:276] 0709 12:19:09.077981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.126723 (+ 48742us) service_pool.cc:224] Handling call
0709 12:19:09.127143 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:09.326981 (+199838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:09.327040 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:09.343918 (+ 16878us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.470015 (+126097us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:09.470109 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.470244 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:10.905169 (+1434925us) raft_consensus.cc:1685] finished
0709 12:19:10.905196 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:10.905362 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12533,"raft.queue_time_us":14036,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":21378,"thread_start_us":270390,"threads_started":2,"wal-append.queue_time_us":546079}
I20260709 12:19:10.873534 29065 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.929975 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:19:10.934613 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:10.934546 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.840865 28980 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:10.919133 28855 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.007s	sys 0.045s
I20260709 12:19:10.946225 12489 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 52 index: 30. (index mismatch)
I20260709 12:19:10.833346 28776 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:10.947436 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2481) took 1114 ms. Trace:
I20260709 12:19:10.867816 28963 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:10.953347 12489 rpcz_store.cc:276] 0709 12:19:09.834778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.928687 (+1093909us) service_pool.cc:224] Handling call
0709 12:19:10.945669 (+ 16982us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.946985 (+  1316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:10.947347 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:10.931676 28911 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260709 12:19:10.905382 29081 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
W20260709 12:19:10.941723 28959 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.539s	user 0.011s	sys 0.004s
I20260709 12:19:10.957305 12566 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 53 because replica has last-logged OpId of term: 51 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 21.
I20260709 12:19:10.841761 29033 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:10.971271 12489 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:10.956122 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:10.964623 12564 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 33. Preceding OpId from leader: term: 50 index: 34. (index mismatch)
I20260709 12:19:10.969053 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:10.902621 28832 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:10.972347 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2482) took 1135 ms. Trace:
I20260709 12:19:10.974087 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3056) took 2148 ms. Trace:
I20260709 12:19:10.976521 12489 rpcz_store.cc:276] 0709 12:19:09.839523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.954841 (+1115318us) service_pool.cc:224] Handling call
0709 12:19:10.970963 (+ 16122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.971921 (+   958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:10.972259 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:10.977771 12564 rpcz_store.cc:276] 0709 12:19:08.825879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.944216 (+2118337us) service_pool.cc:224] Handling call
0709 12:19:10.964314 (+ 20098us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.973026 (+  8712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:10.974011 (+   985us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:10.956760 29063 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.411s	user 0.000s	sys 0.001s
I20260709 12:19:10.980473 12644 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: [52.29-52.29]
I20260709 12:19:10.954995 29082 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:10.980031 28911 raft_consensus.cc:2764] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election decision vote started in defunct term 42: won
I20260709 12:19:10.941874 29065 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.013666 29087 raft_consensus.cc:2764] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election decision vote started in defunct term 52: won
I20260709 12:19:11.015925 29090 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:11.016939 29033 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:11.017907 28998 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.836s	user 0.007s	sys 0.007s
I20260709 12:19:10.906579 29043 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.018901 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3057) took 2154 ms. Trace:
I20260709 12:19:11.111562 12564 rpcz_store.cc:276] 0709 12:19:08.864688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.017194 (+2152506us) service_pool.cc:224] Handling call
0709 12:19:11.018131 (+   937us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.018374 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.018413 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:11.018506 (+    93us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:11.018569 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.018647 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.018812 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.004698 29046 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:11.003865 29072 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:10.963299 29081 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:11.049039 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 92999.1us
I20260709 12:19:11.067179 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2483) took 1224 ms. Trace:
I20260709 12:19:11.130940 12489 rpcz_store.cc:276] 0709 12:19:09.845427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.978025 (+1132598us) service_pool.cc:224] Handling call
0709 12:19:11.066340 (+ 88315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.066656 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.066694 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:11.066760 (+    66us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:11.066821 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.066901 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.067089 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.131639 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:10.979676 29066 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:19:11.071847 29068 log.cc:927] Time spent T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:19:11.176776 28796 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.082970 29051 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.138444 28999 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:19:11.138835 29063 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:19:11.139416 29055 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.139863 29046 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:11.141203 29043 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.141588 28832 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.068817 29082 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.146709 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2475) took 2802 ms. Trace:
I20260709 12:19:11.150871 12645 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:11.156319 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3051) took 2483 ms. Trace:
I20260709 12:19:11.208041 12488 rpcz_store.cc:276] 0709 12:19:08.344031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.711133 (+367102us) service_pool.cc:224] Handling call
0709 12:19:08.711439 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.758821 (+ 47382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.146268 (+2387447us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b540034dd54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.146610 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106249600}
I20260709 12:19:11.208709 12565 rpcz_store.cc:276] 0709 12:19:08.672594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.738599 (+ 66005us) service_pool.cc:224] Handling call
0709 12:19:08.770578 (+ 31979us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:09.184724 (+414146us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:09.184776 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:09.185477 (+   701us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.784022 (+598545us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b180049bac0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:19:10.076175 (+292153us) raft_consensus.cc:1645] Marking committed up to 36
0709 12:19:10.076459 (+   284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.155805 (+1079346us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.155891 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:11.155962 (+    71us) raft_consensus.cc:1685] finished
0709 12:19:11.155985 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.156229 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16606,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":38422,"spinlock_wait_cycles":998947072,"thread_start_us":291368,"threads_started":1,"wal-append.queue_time_us":1044581}
I20260709 12:19:11.157984 29090 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 40.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:19:11.213440 29090 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:19:11.213619 29051 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.213938 29090 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:11.189097 28689 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:11.161404 28991 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.600s	user 0.010s	sys 0.004s
I20260709 12:19:11.163879 29065 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.164713 29072 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.124625 28911 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:11.166538 28980 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.125864 28534 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.171841 28957 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:11.174226 28876 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.108518 28960 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.177214 29081 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:11.135397 29008 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:11.185933 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54412.6us
W20260709 12:19:11.125075 28853 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.004s	sys 0.014s
I20260709 12:19:11.171913 28776 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.187553 28812 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.160578 28963 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.126327 28998 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.218770 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6463) took 2462 ms. Trace:
I20260709 12:19:11.190762 29066 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:11.223502 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3053) took 2550 ms. Trace:
W20260709 12:19:11.229146 29018 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.922s	user 0.006s	sys 0.003s
I20260709 12:19:11.230264 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5951) took 1547 ms. Trace:
I20260709 12:19:11.297907 12644 rpcz_store.cc:276] 0709 12:19:09.682911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.216264 (+533353us) service_pool.cc:224] Handling call
0709 12:19:10.216893 (+   629us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:11.016827 (+799934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.229987 (+213160us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b5400279054. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.230187 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1755282304}
I20260709 12:19:11.299708 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:11.235010 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.244946 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2320) took 2791 ms. Trace:
I20260709 12:19:11.246492 28724 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Stepping down as leader of term 55
I20260709 12:19:11.248054 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:11.306754 12640 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:11.307343 12644 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:11.249209 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:11.251582 28790 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.253268 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.309051 12639 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 55: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:11.258461 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:11.200206 29064 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:11.202240 29046 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:11.263887 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:11.318819 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54969.1us
I20260709 12:19:11.286753 29008 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.037s
I20260709 12:19:11.324160 29008 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:11.324600 29008 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:11.325932 29064 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.223551 29053 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:11.255972 28957 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:11.274742 28853 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.005s	sys 0.015s
I20260709 12:19:11.291899 29066 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:11.300843 12642 rpcz_store.cc:276] 0709 12:19:08.453735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.624731 (+170996us) service_pool.cc:224] Handling call
0709 12:19:08.670476 (+ 45745us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:09.081096 (+410620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:09.081137 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:09.098775 (+ 17638us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.442590 (+343815us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:09.442677 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.244485 (+1801808us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400278b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.244538 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:11.244590 (+    52us) raft_consensus.cc:1685] finished
0709 12:19:11.244602 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.244847 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5188,"raft.queue_time_us":10512,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":931054848,"thread_start_us":284272,"threads_started":1,"wal-append.queue_time_us":354047,"wal-append.run_cpu_time_us":5411,"wal-append.run_wall_time_us":1341045}
I20260709 12:19:11.226234 28689 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.330413 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:11.335508 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 47 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.339561 12644 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:11.306406 28724 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:11.339515 28869 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.217s	user 0.013s	sys 0.002s
I20260709 12:19:11.338327 28776 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.312640 29055 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.039s
I20260709 12:19:11.290213 28812 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.325428 29008 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.263496 29090 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 40.23, Last appended by leader: 23, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.364378 28869 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.374519 29008 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:11.374610 29090 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:19:11.386711 29094 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.104s	user 0.000s	sys 0.002s
I20260709 12:19:11.389050 29066 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.28, Last appended by leader: 28, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.392289 28998 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.393415 29064 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.394081 29082 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.395272 29072 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:11.398715 29028 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.829s	user 0.009s	sys 0.005s
I20260709 12:19:11.406834 29028 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.410347 29028 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.415446 29081 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.281028 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:19:11.458724 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 177513us
I20260709 12:19:11.256686 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:11.249063 12488 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:11.289963 12563 rpcz_store.cc:276] 0709 12:19:08.673238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.879443 (+206205us) service_pool.cc:224] Handling call
0709 12:19:08.926821 (+ 47378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:08.953418 (+ 26597us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:08.953475 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:08.960376 (+  6901us) log.cc:844] Serialized 26 byte log entry
0709 12:19:09.215312 (+254936us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:09.215380 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:09.215498 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:11.183913 (+1968415us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:19:11.216273 (+ 32360us) raft_consensus.cc:1685] finished
0709 12:19:11.216309 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.223397 (+  7088us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9046,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":2962,"spinlock_wait_cycles":85161216,"thread_start_us":141087,"threads_started":1,"wal-append.queue_time_us":399340}
I20260709 12:19:11.476346 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:11.310109 12565 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:19:11.465384 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 208693us
I20260709 12:19:11.289028 12571 rpcz_store.cc:276] 0709 12:19:08.756080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.274618 (+1518538us) service_pool.cc:224] Handling call
0709 12:19:10.275197 (+   579us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.275557 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:19:10.554091 (+278534us) log.cc:844] Serialized 14 byte log entry
0709 12:19:10.794742 (+240651us) log.cc:844] Serialized 14 byte log entry
0709 12:19:11.218236 (+423494us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:11.218318 (+    82us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:11.218397 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.218515 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.218688 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44787,"prepare.queue_time_us":126301,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":6361,"raft.queue_time_us":223894,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":27736,"spinlock_wait_cycles":15246976,"thread_start_us":763155,"threads_started":3,"wal-append.queue_time_us":270623}
I20260709 12:19:11.308028 12567 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:11.468852 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6457) took 3023 ms. Trace:
W20260709 12:19:11.290503 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.470388 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:11.489771 12642 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:11.477594 12563 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
W20260709 12:19:11.490226 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.491395 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:11.492775 12642 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:11.483740 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:11.487931 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:11.495353 12571 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:11.495566 12567 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:11.480363 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:11.499876 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:11.488232 12569 rpcz_store.cc:276] 0709 12:19:08.444893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.721062 (+276169us) service_pool.cc:224] Handling call
0709 12:19:08.721493 (+   431us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:08.766926 (+ 45433us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:19:08.923332 (+156406us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.121953 (+198621us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.122199 (+   246us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:09.122849 (+   650us) log.cc:844] Serialized 52 byte log entry
0709 12:19:09.123014 (+   165us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:09.123060 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.213245 (+2090185us) spinlock_profiling.cc:243] Waited 7.41 ms on lock 0x7b1800407f20. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f
0709 12:19:11.228704 (+ 15459us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b2400152228. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf274509d 00007ffbf537710e 00007ffbf538e88c 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221
0709 12:19:11.273936 (+ 45232us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b48000f1a58. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:19:11.297662 (+ 23726us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b54001810d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.297714 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:11.467861 (+170147us) raft_consensus.cc:1685] finished
0709 12:19:11.467999 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.468551 (+   552us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":345417,"prepare.run_cpu_time_us":155,"prepare.run_wall_time_us":43291,"raft.queue_time_us":1927,"raft.run_cpu_time_us":6652,"raft.run_wall_time_us":2178386,"spinlock_wait_cycles":203400704,"thread_start_us":292742,"threads_started":2,"wal-append.queue_time_us":221620}
I20260709 12:19:11.513571 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:11.514640 12569 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:11.517129 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.495362 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4708) took 1802 ms. Trace:
I20260709 12:19:11.517894 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:11.493243 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:11.528493 28721 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:11.529994 28812 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.529726 28724 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 55.27, Last appended by leader: 27, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.532217 28876 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:11.531721 28776 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.532708 28689 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:11.377214 28957 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 47.30, Last appended by leader: 30, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.487944 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:19:11.541783 12642 rpcz_store.cc:276] 0709 12:19:09.692858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.491799 (+1798941us) service_pool.cc:224] Handling call
0709 12:19:11.492392 (+   593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.494944 (+  2552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.495242 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.534808 12461 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:11.526623 12563 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:11.565908 28534 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:11.567162 28990 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.551303 28876 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:11.567488 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:11.568073 28876 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:11.568796 12640 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:11.570348 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.572336 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.573413 12642 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:11.576692 12567 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:11.587258 12571 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 55.
I20260709 12:19:11.590595 12638 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 56. Current term is 57. Ops: [56.36-56.36]
I20260709 12:19:11.594995 29046 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.597018 28721 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:11.602689 28721 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 46.32, Last appended by leader: 32, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.598981 28998 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.611660 29081 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.612280 28998 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.612846 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:11.618633 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.620901 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.622143 29019 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:11.622931 29019 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:11.623915 12494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6224) took 3465 ms (client timeout 3053 ms). Trace:
I20260709 12:19:11.623672 29055 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:11.624553 12494 rpcz_store.cc:269] 0709 12:19:08.158112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.426430 (+268318us) service_pool.cc:224] Handling call
0709 12:19:11.623340 (+3196910us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540012cad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:11.623662 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3518201600}
W20260709 12:19:11.625133 12496 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2477) took 2547 ms (client timeout 1760 ms). Trace:
W20260709 12:19:11.625788 12496 rpcz_store.cc:269] 0709 12:19:09.077811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.166615 (+ 88804us) service_pool.cc:224] Handling call
0709 12:19:11.624516 (+2457901us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400191754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:11.624857 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2350632192}
I20260709 12:19:11.626708 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.627415 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:11.630133 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5963) took 1938 ms. Trace:
I20260709 12:19:11.632324 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:11.632579 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.632917 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:11.634025 13259 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:11.635605 13258 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 52 index: 31. (index mismatch)
W20260709 12:19:11.636739 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4706) took 1944 ms (client timeout 1803 ms). Trace:
W20260709 12:19:11.637310 12641 rpcz_store.cc:269] 0709 12:19:09.692300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.855942 (+163642us) service_pool.cc:224] Handling call
0709 12:19:11.636154 (+1780212us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:11.636468 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151526784}
W20260709 12:19:11.638597 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2326) took 2701 ms (client timeout 1626 ms). Trace:
I20260709 12:19:11.639243 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.639981 28957 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:11.640467 12641 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
W20260709 12:19:11.642504 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 108048us
I20260709 12:19:11.643007 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5935) took 3780 ms. Trace:
I20260709 12:19:11.643839 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:11.646798 12489 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 50 index: 28. Preceding OpId from leader: term: 53 index: 30. (index mismatch)
I20260709 12:19:11.638665 28803 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 48.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.010s
W20260709 12:19:11.639206 12637 rpcz_store.cc:269] 0709 12:19:08.937536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.937894 (+   358us) service_pool.cc:224] Handling call
0709 12:19:11.637988 (+2700094us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400152f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:11.638313 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":978088832}
I20260709 12:19:11.650159 28601 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:11.651302 28803 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:11.651130 28601 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.651834 28803 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:11.652017 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:11.654868 29090 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:11.655625 29090 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:11.656098 29090 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:11.656344 28125 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
W20260709 12:19:11.657342 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.657828 29064 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:11.658668 29087 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:11.630714 12643 rpcz_store.cc:276] 0709 12:19:09.691174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.536753 (+845579us) service_pool.cc:224] Handling call
0709 12:19:10.556545 (+ 19792us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:10.791397 (+234852us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:10.791443 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:10.798396 (+  6953us) log.cc:844] Serialized 26 byte log entry
0709 12:19:11.192711 (+394315us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:11.192787 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.629198 (+436411us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54002c5954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.629299 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:11.629400 (+   101us) raft_consensus.cc:1685] finished
0709 12:19:11.629417 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.629857 (+   440us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2163,"raft.queue_time_us":521,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":26175,"spinlock_wait_cycles":538996096,"thread_start_us":327750,"threads_started":1,"wal-append.queue_time_us":447520}
I20260709 12:19:11.659585 29087 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.664312 29087 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.599929 12644 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:11.665508 29057 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.666512 29057 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.667666 29045 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.668499 29045 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.598248 12566 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:11.670382 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:11.671434 29045 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.671757 12571 raft_consensus.cc:2410] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 48 because replica has last-logged OpId of term: 47 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 23.
I20260709 12:19:11.606232 28724 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:11.674880 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:11.607606 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.675590 28803 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 48.31, Last appended by leader: 31, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.678025 28803 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:11.606600 12491 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6222) took 3448 ms (client timeout 2062 ms). Trace:
I20260709 12:19:11.610247 29028 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.643440 12646 rpcz_store.cc:276] 0709 12:19:07.862495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:07.868587 (+  6092us) service_pool.cc:224] Handling call
0709 12:19:07.868878 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.346974 (+478096us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:19:08.797174 (+450200us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.080088 (+282914us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:09.080158 (+    70us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:09.080233 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.642394 (+2562161us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b5400153e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.642494 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.642803 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":355800,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":5635,"raft.queue_time_us":24993,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"spinlock_wait_cycles":1098825088,"thread_start_us":665427,"threads_started":2,"wal-append.queue_time_us":363753,"wal-append.run_cpu_time_us":2746,"wal-append.run_wall_time_us":1255803}
I20260709 12:19:11.682639 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:11.682210 12491 rpcz_store.cc:269] 0709 12:19:08.157607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.210326 (+ 52719us) service_pool.cc:224] Handling call
0709 12:19:11.605928 (+3395602us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b54000f6ed4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:11.606308 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1518956416}
I20260709 12:19:11.623450 29019 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:11.674959 28998 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.656874 28125 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:11.633536 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6473) took 1757 ms. Trace:
I20260709 12:19:11.692399 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.629541 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.691597 12567 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:11.695284 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.694437 12570 rpcz_store.cc:276] 0709 12:19:09.875780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.150843 (+275063us) service_pool.cc:224] Handling call
0709 12:19:11.631829 (+1480986us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b5800004b98. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:11.633437 (+  1608us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79852416}
I20260709 12:19:11.697952 12569 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:11.699632 12566 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:11.656723 28601 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.669476 29057 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.712364 29027 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:11.716154 12641 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:11.721233 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.696631 28125 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:11.722108 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.739521 12642 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:11.724792 12570 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:11.741533 12640 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:11.741926 28710 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
W20260709 12:19:11.742383 28902 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.014s	user 0.009s	sys 0.010s
I20260709 12:19:11.743016 28902 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.698763 13254 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:19:11.749081 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105554us
I20260709 12:19:11.750653 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.751847 13256 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:11.751945 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6117) took 2351 ms. Trace:
I20260709 12:19:11.752534 12570 rpcz_store.cc:276] 0709 12:19:09.400771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.723164 (+2322393us) service_pool.cc:224] Handling call
0709 12:19:11.724413 (+  1249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.740648 (+ 16235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.751850 (+ 11202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.753907 29081 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.754323 12570 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:11.754734 29081 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.755935 29046 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.738279 12496 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:19:11.758198 28833 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.229s	user 0.004s	sys 0.021s
I20260709 12:19:11.758656 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.758963 28833 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.760082 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.761066 12496 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:19:11.698714 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.762095 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.762926 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.764448 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:11.764796 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.765319 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5969) took 1887 ms. Trace:
I20260709 12:19:11.695919 28689 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:11.766783 28812 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:11.767055 29081 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:11.739178 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.771368 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6119) took 2370 ms. Trace:
I20260709 12:19:11.782114 12570 rpcz_store.cc:276] 0709 12:19:09.401387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.753388 (+2352001us) service_pool.cc:224] Handling call
0709 12:19:11.753955 (+   567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.770997 (+ 17042us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.771288 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.771978 12646 rpcz_store.cc:276] 0709 12:19:09.877390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.763499 (+1886109us) service_pool.cc:224] Handling call
0709 12:19:11.764029 (+   530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.765005 (+   976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.765230 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.775782 13259 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:11.778620 29057 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.783082 29028 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.791369 12646 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:11.792222 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2330) took 1655 ms. Trace:
I20260709 12:19:11.792534 12646 rpcz_store.cc:276] 0709 12:19:10.136337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.790596 (+1654259us) service_pool.cc:224] Handling call
0709 12:19:11.791141 (+   545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.791926 (+   785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.792157 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.797868 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6468) took 2198 ms. Trace:
I20260709 12:19:11.798556 12570 rpcz_store.cc:276] 0709 12:19:09.596162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.785550 (+2189388us) service_pool.cc:224] Handling call
0709 12:19:11.788316 (+  2766us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.788746 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.788772 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:11.788856 (+    84us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:11.788915 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.794278 (+  5363us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.794403 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.776798 28710 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:11.805289 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:11.804601 28689 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.803467 29027 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.815109 12569 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:19:11.816221 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6116) took 2415 ms. Trace:
I20260709 12:19:11.815922 29019 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.817734 12565 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:19:11.818563 28710 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 37.26, Last appended by leader: 27, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.819046 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:11.819737 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.819865 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:11.832222 13256 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:11.821408 13257 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 41 LEARNER]: Advancing to term 43
I20260709 12:19:11.826117 12646 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:19:11.826713 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:11.827095 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:11.828193 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.816926 12569 rpcz_store.cc:276] 0709 12:19:09.400473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.726307 (+2325834us) service_pool.cc:224] Handling call
0709 12:19:11.811969 (+ 85662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.815748 (+  3779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.816066 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.840193 13253 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 30. Preceding OpId from leader: term: 47 index: 31. (index mismatch)
I20260709 12:19:11.845114 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.846185 12491 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:11.847057 28689 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.848533 12496 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 55: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:11.817231 28976 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:11.849915 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:11.850850 28833 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.851151 12496 raft_consensus.cc:2376] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53. Re-sending same reply.
I20260709 12:19:11.852497 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.853024 12565 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:11.854030 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6126) took 1572 ms. Trace:
I20260709 12:19:11.854070 13254 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:11.854697 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.858799 29027 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.859280 12489 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2484) took 1601 ms. Trace:
I20260709 12:19:11.854467 12565 rpcz_store.cc:276] 0709 12:19:10.281539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.852032 (+1570493us) service_pool.cc:224] Handling call
0709 12:19:11.852657 (+   625us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.853595 (+   938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.853933 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.859953 12489 rpcz_store.cc:276] 0709 12:19:10.258017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.135236 (+877219us) service_pool.cc:224] Handling call
0709 12:19:11.645861 (+510625us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.647454 (+  1593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.858965 (+211511us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400191750. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:11.859177 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172956416}
I20260709 12:19:11.864476 29027 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 51.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:11.866470 29064 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.26, Next index: 27, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:19:11.850503 28976 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.847805 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:11.871500 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.857256 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5961) took 2166 ms. Trace:
I20260709 12:19:11.872162 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2331) took 1701 ms. Trace:
I20260709 12:19:11.875680 12646 rpcz_store.cc:276] 0709 12:19:10.136633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.793369 (+1656736us) service_pool.cc:224] Handling call
0709 12:19:11.800679 (+  7310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.833145 (+ 32466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.837612 (+  4467us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:11.877743 12494 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:11.875206 12645 rpcz_store.cc:276] 0709 12:19:09.690678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.456480 (+765802us) service_pool.cc:224] Handling call
0709 12:19:10.510389 (+ 53909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.208338 (+697949us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.856414 (+648076us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:11.857020 (+   606us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1472206080}
I20260709 12:19:11.804546 29057 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.876379 29027 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:11.884192 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:11.835000 29045 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:11.889593 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:11.891855 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 37 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.902490 13261 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:11.881664 29064 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:11.904794 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:11.908756 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.900534 28785 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.036s
I20260709 12:19:11.913983 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6477) took 1738 ms. Trace:
I20260709 12:19:11.890729 29027 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:11.914486 12570 rpcz_store.cc:276] 0709 12:19:10.175334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.831015 (+1655681us) service_pool.cc:224] Handling call
0709 12:19:11.874756 (+ 43741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.913561 (+ 38805us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.913596 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:11.913682 (+    86us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:11.913717 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.913790 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:11.913919 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:11.922576 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.924587 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:11.924836 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 225346us
I20260709 12:19:11.925405 12646 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:11.907584 29064 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:11.935508 13256 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
I20260709 12:19:11.938536 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:11.974906 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.975976 12496 raft_consensus.cc:2410] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 52 because replica has last-logged OpId of term: 51 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 30.
W20260709 12:19:11.981037 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:11.985220 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:11.986249 13256 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:19:11.923648 28125 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:11.994179 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:11.998574 12570 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:11.997941 28876 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:11.973310 29081 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:11.960182 29045 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:12.006366 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 153909us
I20260709 12:19:12.007180 29019 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.007086 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:12.007756 29008 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.538s	user 0.003s	sys 0.016s
I20260709 12:19:12.008666 29064 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.009671 28833 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20260709 12:19:12.002220 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.014106 28876 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:11.982555 28601 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:12.014685 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 337204us
I20260709 12:19:12.024006 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6480) took 1574 ms. Trace:
I20260709 12:19:12.029752 12570 rpcz_store.cc:276] 0709 12:19:10.441917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.975914 (+1533997us) service_pool.cc:224] Handling call
0709 12:19:11.998263 (+ 22349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.015970 (+ 17707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.016214 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.027546 13258 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:12.032764 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.037441 28601 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.038334 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.039927 29032 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:12.040452 29032 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:12.040886 29032 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.042852 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:12.043107 28876 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:12.043732 28876 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:12.044041 12563 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 53 because replica has last-logged OpId of term: 51 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 21.
I20260709 12:19:12.044142 28876 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.042922 12570 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:19:12.014981 29028 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.053503 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6481) took 1611 ms. Trace:
I20260709 12:19:12.054057 12570 rpcz_store.cc:276] 0709 12:19:10.442435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.042068 (+1599633us) service_pool.cc:224] Handling call
0709 12:19:12.042532 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.050703 (+  8171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.053438 (+  2735us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.055076 12563 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 40 index: 28. Preceding OpId from leader: term: 42 index: 29. (index mismatch)
I20260709 12:19:12.057224 29066 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:12.057814 29045 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:12.058058 29066 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.062161 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:12.062386 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:12.063997 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:12.063944 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.071497 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:12.065549 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:12.086203 13257 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 43 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:19:12.088126 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3061) took 1323 ms. Trace:
I20260709 12:19:12.089253 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:12.065239 28984 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:12.049585 28996 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.949s	user 0.011s	sys 0.025s
I20260709 12:19:12.097990 13254 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 55.
I20260709 12:19:12.065326 28488 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:12.095834 12563 rpcz_store.cc:276] 0709 12:19:10.764878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.044983 (+1280105us) service_pool.cc:224] Handling call
0709 12:19:12.045473 (+   490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.055527 (+ 10054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.088036 (+ 32509us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.047886 28976 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:12.116338 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 50 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.066025 29008 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:12.065780 28125 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:12.110692 28996 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.149s	user 0.011s	sys 0.027s
I20260709 12:19:12.100661 29045 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election won for term 48
W20260709 12:19:12.119664 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 72242.1us
I20260709 12:19:12.096027 29066 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.071419 28785 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.052s
I20260709 12:19:12.078058 28780 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:12.114156 29046 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election decision vote started in defunct term 45: won
W20260709 12:19:12.134147 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.067330 29081 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.134999 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:12.120689 29028 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:12.143880 12563 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:12.154023 12496 raft_consensus.cc:2410] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 53 because replica has last-logged OpId of term: 52 index: 37, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 36.
W20260709 12:19:12.154428 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83059.1us
I20260709 12:19:12.157754 13261 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:12.138697 28976 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
W20260709 12:19:12.156620 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91148.2us
I20260709 12:19:12.169262 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:12.167232 29090 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.510s	user 0.013s	sys 0.004s
I20260709 12:19:12.126732 28984 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.161433 29057 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.192004 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:12.181177 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.181784 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:12.184686 28976 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.173214 29064 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.186493 28796 raft_consensus.cc:2764] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election decision vote started in defunct term 48: won
I20260709 12:19:12.191213 28780 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.191758 28984 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.160414 29028 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.195370 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:12.205883 28957 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.565s	user 0.015s	sys 0.002s
I20260709 12:19:12.209470 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:12.232267 12488 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:12.209880 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:12.223830 28976 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.28, Last appended by leader: 28, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.204144 28785 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
W20260709 12:19:12.230166 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58995.8us
I20260709 12:19:12.219661 12496 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:12.239610 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:12.249511 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:12.249854 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.239837 29046 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:12.198446 29090 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.243698 28785 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:12.264568 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:12.304225 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.309543 12571 raft_consensus.cc:1240] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:12.284569 13254 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:12.293418 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2480) took 2464 ms. Trace:
W20260709 12:19:12.299378 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.315346 12563 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:12.327674 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 6139) took 1381 ms. Trace:
I20260709 12:19:12.332168 12646 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:12.334131 12566 rpcz_store.cc:276] 0709 12:19:10.946216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.968630 (+ 22414us) service_pool.cc:224] Handling call
0709 12:19:12.307585 (+1338955us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:12.327590 (+ 20005us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1436431232}
W20260709 12:19:12.289891 29044 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.833s	user 0.000s	sys 0.026s
I20260709 12:19:12.339744 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:12.343185 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:12.344246 12566 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
W20260709 12:19:12.335681 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 125944us
I20260709 12:19:12.270748 28710 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:12.263530 28785 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:12.356413 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6245) took 1101 ms. Trace:
I20260709 12:19:12.356967 12494 rpcz_store.cc:276] 0709 12:19:11.247642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.626121 (+378479us) service_pool.cc:224] Handling call
0709 12:19:12.348901 (+722780us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b5400367754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:12.349014 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":573512448}
W20260709 12:19:12.343209 29044 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.915s	user 0.006s	sys 0.029s
I20260709 12:19:12.289124 29107 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:12.307832 28721 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:12.347311 28724 consensus_meta.cc:287] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.674s	user 0.018s	sys 0.000s
I20260709 12:19:12.321980 12497 rpcz_store.cc:276] 0709 12:19:09.830519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.921851 (+ 91332us) service_pool.cc:224] Handling call
0709 12:19:09.922274 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:09.922536 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 32
0709 12:19:10.287329 (+364793us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.292070 (+2004741us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b18004fc120. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:19:12.292467 (+   397us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.292565 (+    98us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:19:12.292767 (+   202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.292929 (+   162us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.293165 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70821,"prepare.queue_time_us":400675,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":7416,"raft.queue_time_us":14948,"spinlock_wait_cycles":24649472,"thread_start_us":299916,"threads_started":2,"wal-append.queue_time_us":584099,"wal-append.run_cpu_time_us":2854,"wal-append.run_wall_time_us":1288101}
I20260709 12:19:12.415970 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:12.415870 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54428us
W20260709 12:19:12.418524 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 354634us
W20260709 12:19:12.420598 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.417605 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:12.354940 29045 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:12.431259 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 106922us
I20260709 12:19:12.287739 29028 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 52.29, Last appended by leader: 29, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.427278 28796 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.433653 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:12.434031 29111 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:19:12.439282 12568 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.27-48.27]
I20260709 12:19:12.441711 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5952) took 2758 ms. Trace:
I20260709 12:19:12.442152 29117 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election won for term 47
W20260709 12:19:12.441975 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 81670.7us
I20260709 12:19:12.433166 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:12.443784 29113 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:19:12.441738 12610 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.446520 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:12.356165 28833 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 52.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.074s
I20260709 12:19:12.356316 28957 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.070s
I20260709 12:19:12.448092 12494 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:19:12.449136 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.454952 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.451165 12638 rpcz_store.cc:276] 0709 12:19:09.683226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:10.267571 (+584345us) service_pool.cc:224] Handling call
0709 12:19:10.268026 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:11.593542 (+1325516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.441220 (+847678us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54002c6d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:12.441601 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2917991296}
I20260709 12:19:12.457441 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6465) took 3389 ms. Trace:
W20260709 12:19:12.495217 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.462538 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.504238 12496 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
W20260709 12:19:12.506695 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88324us
I20260709 12:19:12.445044 28724 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.503734 12568 rpcz_store.cc:276] 0709 12:19:09.067930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.438203 (+3370273us) service_pool.cc:224] Handling call
0709 12:19:12.438793 (+   590us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.457097 (+ 18304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.457362 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.509559 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.527655 13254 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:12.519793 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:12.532243 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.532703 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6479) took 2305 ms. Trace:
I20260709 12:19:12.533284 13255 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:12.533524 13218 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:12.545804 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5965) took 2715 ms. Trace:
I20260709 12:19:12.538000 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:12.547354 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:12.549961 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6140) took 1568 ms. Trace:
I20260709 12:19:12.550446 12566 rpcz_store.cc:276] 0709 12:19:10.965250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.415604 (+1450354us) service_pool.cc:224] Handling call
0709 12:19:12.505988 (+ 90384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.532940 (+ 26952us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:19:12.533035 (+    95us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.533082 (+    47us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:12.533132 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.533203 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.533342 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":556,"raft.queue_time_us":19477}
I20260709 12:19:12.546407 12640 rpcz_store.cc:276] 0709 12:19:09.814766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.743070 (+1928304us) service_pool.cc:224] Handling call
0709 12:19:11.743601 (+   531us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:11.796531 (+ 52930us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.796585 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:11.808273 (+ 11688us) log.cc:844] Serialized 26 byte log entry
0709 12:19:12.123826 (+315553us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:12.123874 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.123971 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:12.530240 (+406269us) raft_consensus.cc:1685] finished
0709 12:19:12.530267 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.530437 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26129,"raft.queue_time_us":40671,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":3468,"thread_start_us":303076,"threads_started":1,"wal-append.queue_time_us":424691}
I20260709 12:19:12.523653 29081 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:12.556763 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6259) took 1071 ms. Trace:
I20260709 12:19:12.560099 12496 rpcz_store.cc:276] 0709 12:19:11.485146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.525927 (+1040781us) service_pool.cc:224] Handling call
0709 12:19:12.556695 (+ 30768us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.512761 29117 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:12.426779 28721 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.557662 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.511727 28876 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.583722 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:12.547950 28984 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.584127 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6128) took 2301 ms. Trace:
I20260709 12:19:12.585219 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.584708 12565 rpcz_store.cc:276] 0709 12:19:10.282410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.861054 (+1578644us) service_pool.cc:224] Handling call
0709 12:19:11.861597 (+   543us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:11.862918 (+  1321us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.862951 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:11.868936 (+  5985us) log.cc:844] Serialized 26 byte log entry
0709 12:19:12.139713 (+270777us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:19:12.139751 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.139828 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:12.583831 (+444003us) raft_consensus.cc:1685] finished
0709 12:19:12.583859 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.584053 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":234,"thread_start_us":270164,"threads_started":1,"wal-append.queue_time_us":393622}
I20260709 12:19:12.456640 29107 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:12.575620 29110 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 12:19:12.576030 28883 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
W20260709 12:19:12.577098 29115 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
W20260709 12:19:12.578501 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146240us
W20260709 12:19:12.580006 29032 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.539s	user 0.014s	sys 0.000s
I20260709 12:19:12.505966 28796 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.572224 12638 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:12.573011 29057 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:12.591074 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6254) took 1125 ms. Trace:
I20260709 12:19:12.620520 12496 rpcz_store.cc:276] 0709 12:19:11.465584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.567295 (+1101711us) service_pool.cc:224] Handling call
0709 12:19:12.590980 (+ 23685us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.592367 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6141) took 1590 ms. Trace:
I20260709 12:19:12.546027 28724 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.621723 12568 rpcz_store.cc:276] 0709 12:19:11.002335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.525884 (+1523549us) service_pool.cc:224] Handling call
0709 12:19:12.589177 (+ 63293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.589516 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.589544 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.589594 (+    50us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:12.589656 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.589713 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.592302 (+  2589us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.604074 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.626365 12494 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 46: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:19:12.622422 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52489.1us
I20260709 12:19:12.586666 29019 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:12.624810 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6138) took 1658 ms. Trace:
I20260709 12:19:12.602301 28883 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:12.635279 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6484) took 1464 ms. Trace:
I20260709 12:19:12.637485 12565 rpcz_store.cc:276] 0709 12:19:11.170618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.588772 (+1418154us) service_pool.cc:224] Handling call
0709 12:19:12.634651 (+ 45879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.634898 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.634926 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.634974 (+    48us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:12.635018 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.635079 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.635208 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.640436 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.641305 12638 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:12.639664 28883 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:12.629899 29045 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.26, Last appended by leader: 26, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:12.646868 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66084.1us
I20260709 12:19:12.647544 28854 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:12.648036 28854 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:12.648521 28854 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:12.651789 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2047) took 4217 ms. Trace:
I20260709 12:19:12.654440 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:12.655408 12494 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:12.655843 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6251) took 1401 ms. Trace:
I20260709 12:19:12.655948 28876 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:12.656212 12494 rpcz_store.cc:276] 0709 12:19:11.262598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.636759 (+1374161us) service_pool.cc:224] Handling call
0709 12:19:12.655779 (+ 19020us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.657349 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:12.658612 12642 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:12.659255 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2329) took 2523 ms. Trace:
I20260709 12:19:12.659785 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.660815 12642 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:19:12.659747 12641 rpcz_store.cc:276] 0709 12:19:10.136033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.785769 (+1649736us) service_pool.cc:224] Handling call
0709 12:19:11.788344 (+  2575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:11.789047 (+   703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.789075 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:11.789748 (+   673us) log.cc:844] Serialized 26 byte log entry
0709 12:19:12.113788 (+324040us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:12.113864 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.113998 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:12.658980 (+544982us) raft_consensus.cc:1685] finished
0709 12:19:12.659007 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.659185 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":375,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":63,"thread_start_us":322547,"threads_started":1,"wal-append.queue_time_us":427217}
I20260709 12:19:12.660991 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6485) took 1490 ms. Trace:
I20260709 12:19:12.652341 13262 rpcz_store.cc:276] 0709 12:19:08.434052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.434196 (+   144us) service_pool.cc:224] Handling call
0709 12:19:08.479969 (+ 45773us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:08.674784 (+194815us) raft_consensus.cc:1534] Early marking committed up to index 30
0709 12:19:09.135474 (+460690us) log.cc:844] Serialized 14 byte log entry
0709 12:19:09.309788 (+174314us) log.cc:844] Serialized 14 byte log entry
0709 12:19:10.133466 (+823678us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:10.133545 (+    79us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:10.133662 (+   117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.651228 (+2517566us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b5400030a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:12.651303 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.651533 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18185,"prepare.queue_time_us":692030,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":85,"raft.queue_time_us":250160,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":37570,"spinlock_wait_cycles":447401984,"thread_start_us":1413521,"threads_started":3,"wal-append.queue_time_us":671482,"wal-append.run_cpu_time_us":3089,"wal-append.run_wall_time_us":1240818}
I20260709 12:19:12.664171 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.661486 12568 rpcz_store.cc:276] 0709 12:19:11.170842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.625844 (+1455002us) service_pool.cc:224] Handling call
0709 12:19:12.660318 (+ 34474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.660598 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.660627 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.660679 (+    52us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:12.660772 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.660832 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.660931 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.643046 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.657311 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.667033 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.668973 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:12.667169 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139296us
W20260709 12:19:12.670161 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 239303us
I20260709 12:19:12.665908 12494 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:12.629873 29019 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:12.646977 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.645576 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.675367 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.686115 12646 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:12.674610 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:12.691416 12496 raft_consensus.cc:2393] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:12.707283 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6261) took 1205 ms. Trace:
I20260709 12:19:12.708026 12496 rpcz_store.cc:276] 0709 12:19:11.502134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.675060 (+1172926us) service_pool.cc:224] Handling call
0709 12:19:12.707151 (+ 32091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.709936 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.712147 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:12.626597 29032 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.713563 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:12.714422 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.712013 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:12.677186 29019 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:12.718403 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6253) took 1254 ms. Trace:
I20260709 12:19:12.722527 12494 rpcz_store.cc:276] 0709 12:19:11.464914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.688223 (+1223309us) service_pool.cc:224] Handling call
0709 12:19:12.718323 (+ 30100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.724733 12642 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 57: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:12.701270 28125 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:19:12.716048 29115 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:19:12.726413 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:12.726672 12640 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:12.714254 29046 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:12.728929 12638 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:12.725952 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85481.3us
I20260709 12:19:12.731648 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:12.717698 29057 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:12.729208 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:12.739846 12641 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 43. Current term is 44. Ops: [41.27-43.28]
I20260709 12:19:12.733070 29046 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:12.657856 29124 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 52.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.094s
I20260709 12:19:12.751859 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:12.752135 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2332) took 2527 ms. Trace:
I20260709 12:19:12.752499 12641 rpcz_store.cc:276] 0709 12:19:10.224598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.728329 (+2503731us) service_pool.cc:224] Handling call
0709 12:19:12.733476 (+  5147us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:12.751777 (+ 18301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.752058 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.753423 12642 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: [46.32-46.32]
I20260709 12:19:12.753847 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.754444 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2333) took 2250 ms. Trace:
I20260709 12:19:12.754781 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6246) took 1506 ms. Trace:
I20260709 12:19:12.755462 12494 rpcz_store.cc:276] 0709 12:19:11.247975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.725832 (+1477857us) service_pool.cc:224] Handling call
0709 12:19:12.754713 (+ 28881us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.755635 29027 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 52.29, Last appended by leader: 29, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.754855 12642 rpcz_store.cc:276] 0709 12:19:10.504162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.731351 (+2227189us) service_pool.cc:224] Handling call
0709 12:19:12.751071 (+ 19720us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.754093 (+  3022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.754353 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.763058 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3059) took 3414 ms. Trace:
I20260709 12:19:12.772063 12646 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:12.753844 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:12.775635 12641 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.759012 12491 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:12.772835 12571 rpcz_store.cc:276] 0709 12:19:09.348417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.725830 (+2377413us) service_pool.cc:224] Handling call
0709 12:19:11.755285 (+ 29455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.316207 (+560922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.762766 (+446559us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b54003ab454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:12.762958 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1273869184}
I20260709 12:19:12.778149 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:12.758846 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6149) took 1518 ms. Trace:
I20260709 12:19:12.772199 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6247) took 1509 ms. Trace:
I20260709 12:19:12.779233 12494 rpcz_store.cc:276] 0709 12:19:11.251651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.756548 (+1504897us) service_pool.cc:224] Handling call
0709 12:19:12.757015 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.757261 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.757303 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.757352 (+    49us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:12.757391 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.757454 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.757582 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.781738 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6250) took 1528 ms. Trace:
I20260709 12:19:12.782146 12496 rpcz_store.cc:276] 0709 12:19:11.253985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.722990 (+1469005us) service_pool.cc:224] Handling call
0709 12:19:12.781668 (+ 58678us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.778921 12564 rpcz_store.cc:276] 0709 12:19:11.240347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.757299 (+1516952us) service_pool.cc:224] Handling call
0709 12:19:12.757810 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.758307 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.758341 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.758388 (+    47us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:12.758439 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.758509 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.758661 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:12.766396 29122 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:19:12.767561 29032 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:12.784245 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6236) took 3865 ms. Trace:
I20260709 12:19:12.767561 29107 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.768211 29057 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:12.786173 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:12.786577 29057 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.28, Last appended by leader: 28, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:12.784312 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49255.6us
I20260709 12:19:12.788391 12461 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:12.790128 29107 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:12.790654 12640 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 40 index: 23. Preceding OpId from leader: term: 41 index: 24. (index mismatch)
I20260709 12:19:12.791136 28833 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 54.35, Next index: 36, Last known committed idx: 35, Time since last communication: 0.001s
I20260709 12:19:12.791563 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5973) took 2283 ms. Trace:
I20260709 12:19:12.792347 28833 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Stepping down as leader of term 56
I20260709 12:19:12.792549 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:12.793962 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6150) took 1327 ms. Trace:
I20260709 12:19:12.794315 28957 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:12.794909 28957 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:12.724928 28710 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.053s
I20260709 12:19:12.795519 28957 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.792021 12640 rpcz_store.cc:276] 0709 12:19:10.508678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.738922 (+2230244us) service_pool.cc:224] Handling call
0709 12:19:12.790064 (+ 51142us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.791234 (+  1170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.791493 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.796554 28710 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:12.796977 28710 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.797358 12640 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [46.26-48.27]
I20260709 12:19:12.797734 29044 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:12.798266 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5977) took 2200 ms. Trace:
I20260709 12:19:12.797931 28710 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.798782 12640 rpcz_store.cc:276] 0709 12:19:10.598210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.796594 (+2198384us) service_pool.cc:224] Handling call
0709 12:19:12.797080 (+   486us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:12.797940 (+   860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.798203 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.799933 28854 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.800515 12640 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:12.783849 28125 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:12.794395 12571 rpcz_store.cc:276] 0709 12:19:11.466850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.778593 (+1311743us) service_pool.cc:224] Handling call
0709 12:19:12.789452 (+ 10859us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.793473 (+  4021us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.793516 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.793566 (+    50us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:12.793613 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.793708 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.793889 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.801251 29124 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:12.799975 28710 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:12.806733 12638 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56. Ops: [55.27-55.27]
I20260709 12:19:12.803084 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:12.807044 28854 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:12.811386 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5974) took 2280 ms. Trace:
I20260709 12:19:12.811774 28838 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:12.784802 12492 rpcz_store.cc:276] 0709 12:19:08.918782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.611461 (+692679us) service_pool.cc:224] Handling call
0709 12:19:09.611856 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:10.241979 (+630123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.750703 (+2508724us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400367254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:12.784004 (+ 33301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1361151872}
I20260709 12:19:12.812327 28838 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:12.812811 28838 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:12.814047 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:12.810137 29124 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:12.816666 29124 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 52.29, Last appended by leader: 29, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.792900 28833 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.804117 29019 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.818251 12461 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:12.819926 28969 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:12.823065 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 110257us
I20260709 12:19:12.823498 28992 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260709 12:19:12.790793 29107 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.801403 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2338) took 1330 ms. Trace:
I20260709 12:19:12.811993 12638 rpcz_store.cc:276] 0709 12:19:10.531005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.751710 (+2220705us) service_pool.cc:224] Handling call
0709 12:19:12.806263 (+ 54553us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:12.811002 (+  4739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.811273 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.533214 12569 rpcz_store.cc:276] 0709 12:19:10.222114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.835316 (+1613202us) service_pool.cc:224] Handling call
0709 12:19:11.863225 (+ 27909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:11.863642 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:12.123550 (+259908us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.527290 (+403740us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.527353 (+    63us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:12.527402 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.527482 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.527627 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8985,"prepare.queue_time_us":326052,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":18416,"thread_start_us":595670,"threads_started":2}
I20260709 12:19:12.824841 28838 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.824705 28785 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.857587 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:12.811460 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:12.800640 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6148) took 1585 ms. Trace:
I20260709 12:19:12.866345 12563 raft_consensus.cc:1275] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 53 index: 30. (index mismatch)
I20260709 12:19:12.757617 29027 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:12.852833 12564 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:19:12.852968 28833 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 56.36, Last appended by leader: 36, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.850637 28876 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:12.855640 28883 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.859280 12638 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 40. Current term is 41. Ops: []
I20260709 12:19:12.769125 28796 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:12.858044 12640 rpcz_store.cc:276] 0709 12:19:11.471603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.799840 (+1328237us) service_pool.cc:224] Handling call
0709 12:19:12.800255 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.801114 (+   859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.801334 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.752815 29046 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:12.841840 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5971) took 2657 ms. Trace:
I20260709 12:19:12.865787 12568 rpcz_store.cc:276] 0709 12:19:11.215377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.682136 (+1466759us) service_pool.cc:224] Handling call
0709 12:19:12.734308 (+ 52172us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.800237 (+ 65929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.800271 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.800320 (+    49us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:12.800369 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.800430 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.800569 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:12.870590 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.864799 12488 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.620s	user 0.013s	sys 0.001s
I20260709 12:19:12.910832 12488 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:12.911671 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6258) took 1427 ms. Trace:
I20260709 12:19:12.894169 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4712) took 1370 ms. Trace:
I20260709 12:19:12.912642 12638 rpcz_store.cc:276] 0709 12:19:11.523763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.858596 (+1334833us) service_pool.cc:224] Handling call
0709 12:19:12.859055 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.893756 (+ 34701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.894063 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.887934 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6492) took 1167 ms. Trace:
I20260709 12:19:12.913549 12564 rpcz_store.cc:276] 0709 12:19:11.720809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.783439 (+1062630us) service_pool.cc:224] Handling call
0709 12:19:12.850921 (+ 67482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.886202 (+ 35281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.887846 (+  1644us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:12.914515 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 202617us
I20260709 12:19:12.912055 12488 rpcz_store.cc:276] 0709 12:19:11.484612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.209067 (+724455us) service_pool.cc:224] Handling call
0709 12:19:12.911591 (+702524us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.859525 29124 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:12.919806 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:12.922658 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:12.922771 12488 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:12.926455 28969 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:12.927450 29136 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:12.928212 29136 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.927227 28969 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:12.858712 28836 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.057s
I20260709 12:19:12.930747 28969 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.935758 12488 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 43 index: 28. (index mismatch)
I20260709 12:19:12.766474 29117 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:12.937999 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6482) took 1768 ms. Trace:
I20260709 12:19:12.924317 28815 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.078s
I20260709 12:19:12.938469 12566 rpcz_store.cc:276] 0709 12:19:11.170091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.574125 (+1404034us) service_pool.cc:224] Handling call
0709 12:19:12.622056 (+ 47931us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.645389 (+ 23333us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:12.706246 (+ 60857us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.937598 (+231352us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.937689 (+    91us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:12.937742 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.937815 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.937932 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15960,"prepare.queue_time_us":34515,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2472,"thread_start_us":177814,"threads_started":1}
I20260709 12:19:12.950870 12489 raft_consensus.cc:1240] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:12.859938 28689 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.065s
I20260709 12:19:12.858749 29107 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 44.28, Last appended by leader: 28, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:12.961930 12488 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:12.962410 29136 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:12.963809 28689 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:12.964277 28689 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.915370 28876 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:12.909552 12645 rpcz_store.cc:276] 0709 12:19:10.184624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.881557 (+1696933us) service_pool.cc:224] Handling call
0709 12:19:11.887121 (+  5564us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:11.983194 (+ 96073us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:12.003857 (+ 20663us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.348632 (+344775us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:12.432527 (+ 83895us) log.cc:844] Serialized 52 byte log entry
0709 12:19:12.432747 (+   220us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:12.447940 (+ 15193us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.448522 (+   582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.448612 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:12.820458 (+371846us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b24003e3738. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf258c9c4 00007ffbf258aacb 00007ffbf2745dc9 00007ffbf2745a55 00007ffbf2744fb0 00007ffbf53bb488 00007ffbf53bbbe8 00007ffbf5393398 00007ffbf538e76a 00007ffbf538dbb6
0709 12:19:12.841504 (+ 21046us) raft_consensus.cc:1685] finished
0709 12:19:12.841545 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.841759 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66245,"prepare.queue_time_us":35121,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":4188,"raft.queue_time_us":9101,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":28018,"spinlock_wait_cycles":71780608,"thread_start_us":301281,"threads_started":1,"wal-append.queue_time_us":501466}
I20260709 12:19:12.956452 12641 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:12.966626 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:12.922247 28796 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:12.970254 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3062) took 1311 ms. Trace:
I20260709 12:19:12.983289 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6486) took 1794 ms. Trace:
I20260709 12:19:12.983108 12563 rpcz_store.cc:276] 0709 12:19:11.659128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.781951 (+1122823us) service_pool.cc:224] Handling call
0709 12:19:12.782333 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.885287 (+102954us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.970178 (+ 84891us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.983876 12565 rpcz_store.cc:276] 0709 12:19:11.188916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.643159 (+1454243us) service_pool.cc:224] Handling call
0709 12:19:12.658318 (+ 15159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.658691 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:19:12.659450 (+   759us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.982880 (+323430us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.982945 (+    65us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:12.983005 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.983073 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.983223 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":221,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":24416,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":20211,"thread_start_us":319416,"threads_started":2,"wal-append.queue_time_us":182734}
W20260709 12:19:12.989192 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:12.992955 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:12.995450 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:12.996589 12638 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 45. Ops: []
I20260709 12:19:12.998562 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5964) took 3184 ms. Trace:
W20260709 12:19:12.999208 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 206622us
I20260709 12:19:13.011363 12645 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:12.999058 12644 rpcz_store.cc:276] 0709 12:19:09.814466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.666649 (+1852183us) service_pool.cc:224] Handling call
0709 12:19:12.315049 (+648400us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:19:12.327405 (+ 12356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.415378 (+ 87973us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:19:12.448387 (+ 33009us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.968193 (+519806us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.968248 (+    55us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:12.998223 (+ 29975us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.998314 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.998486 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21145,"prepare.queue_time_us":31797,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":34297,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":25927,"spinlock_wait_cycles":40326528,"thread_start_us":426307,"threads_started":2,"wal-append.queue_time_us":257078}
I20260709 12:19:13.019273 12646 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:19:13.021483 12646 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 31. Preceding OpId from leader: term: 47 index: 33. (index mismatch)
W20260709 12:19:13.021248 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 250026us
I20260709 12:19:13.031170 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5982) took 1145 ms. Trace:
I20260709 12:19:13.031636 12645 rpcz_store.cc:276] 0709 12:19:11.885310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.982034 (+1096724us) service_pool.cc:224] Handling call
0709 12:19:13.011167 (+ 29133us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.030812 (+ 19645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.031097 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:12.980504 28125 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:13.015888 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:13.004186 29136 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:13.041321 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5981) took 1157 ms. Trace:
I20260709 12:19:13.044996 12640 rpcz_store.cc:276] 0709 12:19:11.883687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.979387 (+1095700us) service_pool.cc:224] Handling call
0709 12:19:13.003752 (+ 24365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.030172 (+ 26420us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:13.030209 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:13.041000 (+ 10791us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:13.041050 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.041123 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.041254 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:13.003925 28992 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:13.003047 28833 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:13.047681 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46780.9us
W20260709 12:19:13.040560 29125 log.cc:927] Time spent T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.234s	user 0.000s	sys 0.002s
I20260709 12:19:13.042188 28883 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.064162 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.065609 12489 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 56.
I20260709 12:19:13.067466 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.053812 28838 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.078151 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.083521 29134 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:13.085389 28957 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.084947 28724 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.087616 29134 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:13.088475 28957 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.088943 29134 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 53.31, Last appended by leader: 31, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.102547 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.103897 28969 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:19:13.106604 29132 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:19:13.073359 29045 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:13.126379 12492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2490) took 1211 ms. Trace:
I20260709 12:19:13.141866 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.164870 28785 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.153899 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:13.163501 28876 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.161895 28125 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.078779 29057 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 46.32, Next index: 33, Last known committed idx: 31, Time since last communication: 0.079s
I20260709 12:19:13.185575 12489 raft_consensus.cc:2368] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
W20260709 12:19:13.192373 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 87092.3us
W20260709 12:19:13.197746 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76894.7us
I20260709 12:19:13.170899 29135 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.127053 12492 rpcz_store.cc:276] 0709 12:19:11.915006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.812699 (+897693us) service_pool.cc:224] Handling call
0709 12:19:12.813232 (+   533us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:12.825910 (+ 12678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.825942 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:12.826749 (+   807us) log.cc:844] Serialized 52 byte log entry
0709 12:19:12.981331 (+154582us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:13.121548 (+140217us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.122722 (+  1174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.122838 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.125986 (+  3148us) raft_consensus.cc:1685] finished
0709 12:19:13.126011 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.126243 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":36234,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":27598,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":14912,"thread_start_us":238567,"threads_started":2,"wal-append.queue_time_us":239515}
I20260709 12:19:13.203536 29135 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.204600 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:13.205587 12492 raft_consensus.cc:2368] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:19:13.205813 29144 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:13.206598 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:13.207309 12492 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:13.206815 29144 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:13.209378 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.061196 28689 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 48.27, Last appended by leader: 27, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:13.203859 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 237372us
I20260709 12:19:13.203570 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:13.216441 28984 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.622s	user 0.009s	sys 0.006s
I20260709 12:19:13.207345 28776 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:13.245260 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.246045 12491 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6262) took 1165 ms. Trace:
I20260709 12:19:13.214181 29144 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:13.237744 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.614s (SENT) [suppressed 16 similar messages]
I20260709 12:19:13.252528 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.253414 12646 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 56: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:13.218977 29057 consensus_queue.cc:1243] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 47.33 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 31
W20260709 12:19:13.244576 28976 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.012s	sys 0.018s
I20260709 12:19:13.259428 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:13.262939 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5986) took 1228 ms. Trace:
I20260709 12:19:13.246249 28984 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:13.267834 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:13.271183 12645 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:13.277419 12492 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
W20260709 12:19:13.279013 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.273739 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:13.264351 28976 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.013s	sys 0.018s
I20260709 12:19:13.284657 12572 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:13.286856 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.289426 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3058) took 3980 ms. Trace:
W20260709 12:19:13.291026 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60986.7us
I20260709 12:19:13.288872 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.292218 12646 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:13.289893 12572 rpcz_store.cc:276] 0709 12:19:09.309070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:13.156683 (+3847613us) service_pool.cc:224] Handling call
0709 12:19:13.184043 (+ 27360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.289046 (+105003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.289338 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:13.292953 28933 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:13.271349 29134 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20260709 12:19:13.246536 12491 rpcz_store.cc:276] 0709 12:19:12.081072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.873298 (+792226us) service_pool.cc:224] Handling call
0709 12:19:12.911680 (+ 38382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.912018 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:19:12.959356 (+ 47338us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.245613 (+286257us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:13.245724 (+   111us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:13.245777 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.245831 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.245951 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1780,"prepare.queue_time_us":93032,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":69,"raft.queue_time_us":2352,"thread_start_us":233829,"threads_started":1}
I20260709 12:19:13.306418 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.288820 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.320109 12567 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 55: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:13.308868 28984 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
W20260709 12:19:13.303354 12493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34482 (request call id 6234) took 4388 ms (client timeout 3226 ms). Trace:
I20260709 12:19:13.321157 28984 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:13.321233 12493 rpcz_store.cc:269] 0709 12:19:08.914614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.914790 (+   176us) service_pool.cc:224] Handling call
0709 12:19:13.302644 (+4387854us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400426654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:13.303018 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3126357248}
I20260709 12:19:13.321851 28984 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:13.309367 13256 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: [52.31-52.31]
I20260709 12:19:13.306030 28902 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:13.330524 29139 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:19:13.310109 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.338836 12563 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:13.305099 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:13.342541 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5991) took 1210 ms. Trace:
I20260709 12:19:13.317454 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.297871 28780 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.345178 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:13.343451 13260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2286) took 4628 ms (client timeout 2298 ms). Trace:
I20260709 12:19:13.346415 12637 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:13.347078 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2069) took 1709 ms. Trace:
I20260709 12:19:13.347694 13258 rpcz_store.cc:276] 0709 12:19:11.638276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.693328 (+ 55052us) service_pool.cc:224] Handling call
0709 12:19:13.346814 (+1653486us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:13.346987 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565682944}
I20260709 12:19:13.293596 28933 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:13.350198 12490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2479) took 3524 ms. Trace:
I20260709 12:19:13.350812 12490 rpcz_store.cc:276] 0709 12:19:09.827214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:09.871379 (+ 44165us) service_pool.cc:224] Handling call
0709 12:19:09.871835 (+   456us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:10.110415 (+238580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.349522 (+3239107us) spinlock_profiling.cc:243] Waited 89.3 ms on lock 0x7b540012a7d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:13.349967 (+   445us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205402496}
I20260709 12:19:13.351389 28933 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 42.23, Last appended by leader: 23, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.308586 29144 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:13.352594 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.352787 29144 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.353981 12489 raft_consensus.cc:2410] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 45 because replica has last-logged OpId of term: 43 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 23.
I20260709 12:19:13.293989 28689 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:13.308905 28869 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.296209 29143 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.357080 29143 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.359755 29143 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.342546 29135 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.345700 28724 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.362231 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:13.361994 28724 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:13.364384 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:13.365116 28883 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:13.365773 28883 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:13.366281 28830 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
W20260709 12:19:13.346351 13260 rpcz_store.cc:269] 0709 12:19:08.715154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:08.860322 (+145168us) service_pool.cc:224] Handling call
0709 12:19:11.645548 (+2785226us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:13.304740 (+1659192us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cee9c 00007ffbf26cea5b 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:13.343341 (+ 38601us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75943552}
I20260709 12:19:13.366899 28883 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.307358 13261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2294) took 1511 ms. Trace:
I20260709 12:19:13.370947 13261 rpcz_store.cc:276] 0709 12:19:11.795725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.795934 (+   209us) service_pool.cc:224] Handling call
0709 12:19:13.307028 (+1511094us) spinlock_profiling.cc:243] Waited 96.2 ms on lock 0x7b54006d1ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:13.307230 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221332224}
W20260709 12:19:13.347112 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4719) took 1736 ms (client timeout 2006 ms). Trace:
I20260709 12:19:13.303757 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.373049 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:19:13.374323 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70685.1us
I20260709 12:19:13.375434 28992 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:13.376214 28992 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:13.362277 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4713) took 1838 ms. Trace:
I20260709 12:19:13.364557 28724 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:13.372503 12637 rpcz_store.cc:269] 0709 12:19:11.610459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.650338 (+ 39879us) service_pool.cc:224] Handling call
0709 12:19:13.344984 (+1694646us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:13.347036 (+  2052us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75397504}
I20260709 12:19:13.377249 12638 rpcz_store.cc:276] 0709 12:19:11.524182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.925743 (+1401561us) service_pool.cc:224] Handling call
0709 12:19:12.996342 (+ 70599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.003004 (+  6662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.361931 (+358927us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54002c6850. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:13.362179 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70338816}
W20260709 12:19:13.377749 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 140038us
I20260709 12:19:13.343282 12644 rpcz_store.cc:276] 0709 12:19:12.132592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:13.017856 (+885264us) service_pool.cc:224] Handling call
0709 12:19:13.018715 (+   859us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:13.019610 (+   895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:13.019668 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:13.026222 (+  6554us) log.cc:844] Serialized 52 byte log entry
0709 12:19:13.213400 (+187178us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:13.213462 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.213564 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.342136 (+128572us) raft_consensus.cc:1685] finished
0709 12:19:13.342174 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.342447 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2785,"raft.queue_time_us":12106,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":10031,"thread_start_us":183827,"threads_started":1,"wal-append.queue_time_us":312194}
W20260709 12:19:13.354017 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4717) took 1744 ms (client timeout 2234 ms). Trace:
I20260709 12:19:13.355919 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.375473 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.303620 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:13.296947 29145 log.cc:927] Time spent T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:19:13.299952 28796 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:19:13.404853 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 153436us
I20260709 12:19:13.407449 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.408255 12646 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
W20260709 12:19:13.407977 12643 rpcz_store.cc:269] 0709 12:19:11.609490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.659881 (+ 50391us) service_pool.cc:224] Handling call
0709 12:19:13.352383 (+1692502us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b5800003398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:13.353916 (+  1533us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53327232}
I20260709 12:19:13.414376 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:13.415287 13261 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42. Re-sending same reply.
W20260709 12:19:13.418583 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 115126us
I20260709 12:19:13.419171 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:13.356117 29144 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.385520 28830 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:13.427201 28849 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.021s
I20260709 12:19:13.427942 29038 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:19:13.430015 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54819.7us
I20260709 12:19:13.430505 28902 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:19:13.430850 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.301916 29046 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.439348 12567 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:13.431550 12609 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:13.406632 28836 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.025s
I20260709 12:19:13.424172 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:13.434109 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 5975) took 2878 ms. Trace:
I20260709 12:19:13.440616 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:13.438207 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:19:13.439188 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 150464us
I20260709 12:19:13.411711 29028 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.021s
I20260709 12:19:13.421509 29143 raft_consensus.cc:2764] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election decision vote started in defunct term 43: won
I20260709 12:19:13.445528 28902 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:13.453504 28721 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.458562 28902 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.461865 28601 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.445865 28776 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:13.445565 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:13.446405 12609 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:13.447257 29028 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:13.447736 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.437952 28796 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:13.448511 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6239) took 2202 ms. Trace:
I20260709 12:19:13.440676 12642 rpcz_store.cc:276] 0709 12:19:10.556041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.781397 (+2225356us) service_pool.cc:224] Handling call
0709 12:19:12.912917 (+131520us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:12.921665 (+  8748us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:12.921698 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:12.926979 (+  5281us) log.cc:844] Serialized 52 byte log entry
0709 12:19:13.005815 (+ 78836us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:13.042617 (+ 36802us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.310191 (+267574us) spinlock_profiling.cc:243] Waited 9.97 ms on lock 0x7b18002d7e40. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:19:13.311214 (+  1023us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.311525 (+   311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.311604 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.433805 (+122201us) raft_consensus.cc:1685] finished
0709 12:19:13.433829 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.434032 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":274122,"prepare.queue_time_us":305144,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":97,"raft.queue_time_us":18959,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":7489,"spinlock_wait_cycles":22942080,"thread_start_us":75436,"threads_started":1,"wal-append.queue_time_us":154333}
I20260709 12:19:13.448557 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:13.450151 12460 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.452616 28830 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.460850 28984 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.464380 28776 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.467195 12495 rpcz_store.cc:276] 0709 12:19:11.245958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:11.249752 (+  3794us) service_pool.cc:224] Handling call
0709 12:19:11.250113 (+   361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:11.250513 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:11.250543 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:11.251212 (+   669us) log.cc:844] Serialized 52 byte log entry
0709 12:19:11.766174 (+514962us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:19:11.766314 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:11.766530 (+   216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:13.431839 (+1665309us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b18004f1500. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:19:13.439040 (+  7201us) raft_consensus.cc:1685] finished
0709 12:19:13.439061 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.448318 (+  9257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":135,"spinlock_wait_cycles":49208192,"thread_start_us":448005,"threads_started":1,"wal-append.queue_time_us":466539}
I20260709 12:19:13.474889 29156 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.475603 29143 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.469375 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:13.480683 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.487574 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.485095 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:13.491210 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43530.4us
I20260709 12:19:13.483992 13253 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:19:13.489389 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:13.492570 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:13.466439 29028 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:13.500805 12637 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:13.502118 12567 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:19:13.499198 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.499292 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.486506 28776 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:13.494954 29124 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:13.508857 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.509610 12563 raft_consensus.cc:2376] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44. Re-sending same reply.
I20260709 12:19:13.510721 12642 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 37 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:19:13.510996 29134 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:13.497761 29038 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:13.512524 29163 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.512864 29038 raft_consensus.cc:697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:13.513273 29163 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.514155 29038 consensus_queue.cc:237] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.23, Last appended by leader: 23, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.506335 29143 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.515949 29163 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.506138 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:13.501483 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.507390 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.519891 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.510812 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.517844 12571 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:13.506799 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.518939 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:13.494727 28721 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20260709 12:19:13.539161 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.540177 12568 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:13.540903 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 153065us
I20260709 12:19:13.463428 29144 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.042s
I20260709 12:19:13.542654 12563 raft_consensus.cc:2376] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55. Re-sending same reply.
I20260709 12:19:13.533255 29163 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.517547 29143 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.544528 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.537851 28125 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:19:13.548907 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47575.2us
I20260709 12:19:13.542475 28780 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:13.556015 12572 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 44: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:13.556706 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.560838 12571 raft_consensus.cc:2410] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 43 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 25.
I20260709 12:19:13.546550 28796 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:13.544055 28776 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.567358 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.564239 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.568825 12563 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:13.538918 29107 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.020s
I20260709 12:19:13.552347 28969 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:13.505448 29156 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.550817 28125 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:13.584367 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.547359 29163 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.566757 29046 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.594980 12563 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:13.582767 28796 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:13.603447 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:13.608088 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46656.1us
I20260709 12:19:13.588672 28125 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:13.617879 28776 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:12.630858 12570 rpcz_store.cc:276] 0709 12:19:10.945918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.097963 (+1152045us) service_pool.cc:224] Handling call
0709 12:19:12.135922 (+ 37959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.147690 (+ 11768us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:12.324729 (+177039us) log.cc:844] Serialized 14 byte log entry
0709 12:19:12.602873 (+278144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:12.603061 (+   188us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:12.603283 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:12.603430 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:12.603757 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24854,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":9757,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":12389,"thread_start_us":324999,"threads_started":2}
W20260709 12:19:13.625545 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:13.636665 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 128871us
I20260709 12:19:13.635387 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.639896 12570 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:13.649004 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.651729 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:13.673027 12572 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:13.696870 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.702484 12536 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:13.703280 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49157.6us
I20260709 12:19:13.613670 29046 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:13.643747 28849 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.055s
W20260709 12:19:13.713438 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61858.5us
W20260709 12:19:13.628396 29140 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.548s	user 0.004s	sys 0.000s
I20260709 12:19:13.717864 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.633569 29163 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.574388 29028 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 52.31, Last appended by leader: 31, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.720544 12568 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:13.627215 29156 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.724066 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:13.726190 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:13.702791 29038 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:13.729034 29165 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:19:13.708630 29117 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.729542 29038 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.732753 29038 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.708567 28601 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.736936 28601 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.738030 29117 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.739269 28601 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.741127 29117 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:13.743019 28992 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.634268 28796 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.29, Last appended by leader: 29, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.744520 29028 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:13.750218 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101720us
I20260709 12:19:13.737203 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.715348 28984 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.750597 28992 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:19:13.754510 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.755265 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:13.757208 12495 raft_consensus.cc:1217] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Deduplicated request from leader. Original: 44.31->[46.32-46.32]   Dedup: 46.32->[]
I20260709 12:19:13.757953 12495 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: [46.32-46.32]
I20260709 12:19:13.759191 13261 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:19:13.727043 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.761601 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.763198 28689 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:13.763952 28689 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:13.764377 28689 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:13.765172 12537 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:13.766710 29138 consensus_queue.cc:1059] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:19:13.765436 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.768180 28984 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:19:13.768849 28984 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:13.769012 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.770181 28830 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:13.770097 28984 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.771263 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.765275 28689 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.772236 29156 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:13.772850 29156 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.751766 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:13.785367 12570 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
W20260709 12:19:13.805887 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 80597us
I20260709 12:19:13.826767 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.831810 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.834164 12570 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:13.802841 29144 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:13.845317 28992 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
W20260709 12:19:13.846628 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75488.2us
I20260709 12:19:13.848532 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:13.821377 28780 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:13.856297 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.855330 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:13.861119 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:13.862792 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 111087us
I20260709 12:19:13.824479 28849 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:13.823666 28933 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:13.781190 29057 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.094s
I20260709 12:19:13.870296 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:19:13.883479 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 155550us
W20260709 12:19:13.882305 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 131000us
I20260709 12:19:13.907091 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.858322 29107 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:13.909063 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.911018 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.863430 29130 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:13.878703 28721 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:13.880158 28992 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.881712 28933 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:13.915544 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.917928 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.919989 12492 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:13.907562 28849 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:13.922753 28849 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 53.28, Last appended by leader: 28, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.926780 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.927865 28125 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.929587 12564 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:13.931948 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.932320 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:13.933215 12490 raft_consensus.cc:3060] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:13.934705 28430 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:13.935223 28430 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:13.936187 12492 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:13.936364 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.937927 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.940502 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:13.941998 29173 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:13.942910 29173 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:13.944269 29090 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.928184 28992 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.945904 29173 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.942263 29044 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:13.946341 29156 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:13.946532 29044 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:13.947770 12490 raft_consensus.cc:2468] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:19:13.947620 29044 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.29, Last appended by leader: 29, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:13.948781 29044 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:13.952951 28125 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:13.953909 28780 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:13.955472 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.919375 12568 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:13.960928 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:13.964329 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:13.965620 12570 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:13.978255 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.916226 29163 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:19:13.994222 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 40 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.000494 28992 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:14.000737 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.001837 12490 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 55: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:13.999141 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 42.
W20260709 12:19:14.005614 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:13.963107 28933 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:14.004799 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:14.005703 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.013883 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:13.992447 28780 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:13.989449 28969 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:14.024066 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.025233 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:14.027346 12570 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
W20260709 12:19:14.030591 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112730us
I20260709 12:19:13.967934 29124 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.055s
I20260709 12:19:13.999956 29180 raft_consensus.cc:3055] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Stepping down as leader of term 54
I20260709 12:19:13.968586 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:13.991569 29130 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:14.024202 29166 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:13.978132 28721 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:13.990345 28902 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.059s
I20260709 12:19:14.044798 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.055513 12568 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.24-43.24]
I20260709 12:19:14.063457 12568 pending_rounds.cc:85] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 30
I20260709 12:19:14.049278 29166 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:14.040287 29180 raft_consensus.cc:740] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:14.064143 12568 pending_rounds.cc:107] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 46.31
I20260709 12:19:14.072808 12568 raft_consensus.cc:2889] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:14.040650 29117 consensus_queue.cc:1048] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.022s
I20260709 12:19:14.078258 29178 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:14.082770 29180 consensus_queue.cc:260] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 54.31, Last appended by leader: 31, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:14.084089 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43957.7us
I20260709 12:19:14.085341 12611 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:14.085879 29130 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.33, Last appended by leader: 33, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.038861 28780 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.088948 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:14.091385 28780 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.091821 28812 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20260709 12:19:14.092338 12563 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 29. Preceding OpId from leader: term: 53 index: 30. (index mismatch)
I20260709 12:19:14.084136 29180 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:14.096441 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:13.936239 28430 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 46.32, Last appended by leader: 32, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.073588 29166 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:14.117532 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 51973.8us
I20260709 12:19:14.112825 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2075) took 1833 ms. Trace:
I20260709 12:19:14.119336 13256 rpcz_store.cc:276] 0709 12:19:12.270528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.270817 (+   289us) service_pool.cc:224] Handling call
0709 12:19:13.309019 (+1038202us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:13.322770 (+ 13751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.103488 (+780718us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54006d1ad0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:14.103777 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2355085056}
I20260709 12:19:14.111828 13261 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:14.118031 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:14.069077 29175 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:14.133988 29175 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:14.134411 28721 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 52.37, Last appended by leader: 37, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.135128 29175 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 50.30, Last appended by leader: 30, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.118681 13258 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 55 index: 32. (index mismatch)
I20260709 12:19:14.122936 12461 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:14.129355 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.132634 29143 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:13.264964 12641 rpcz_store.cc:276] 0709 12:19:12.026251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.982266 (+956015us) service_pool.cc:224] Handling call
0709 12:19:12.982753 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.030553 (+ 47800us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:13.186960 (+156407us) log.cc:844] Serialized 14 byte log entry
0709 12:19:13.254628 (+ 67668us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:13.254689 (+    61us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:13.254750 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:13.254832 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:13.254980 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21698,"raft.queue_time_us":17814,"thread_start_us":130451,"threads_started":2}
I20260709 12:19:14.166025 12495 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 49 index: 27. (term mismatch)
I20260709 12:19:14.166581 12495 pending_rounds.cc:85] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Aborting all ops after (but not including) 26
I20260709 12:19:14.159621 28689 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.007s
I20260709 12:19:14.166843 12495 pending_rounds.cc:107] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Aborting uncommitted NO_OP operation due to leader change: 48.27
I20260709 12:19:14.167234 12495 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:14.155848 29028 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:14.169018 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.169272 29028 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:14.170071 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:14.170657 29028 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.101336 28796 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20260709 12:19:14.145175 29143 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:14.153038 29119 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:14.184458 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55203.1us
I20260709 12:19:14.186496 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.174248 28430 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:14.200898 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:14.212108 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.211290 28780 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:14.222745 28780 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:14.223296 28780 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:14.235509 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.244264 29143 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 46.29, Last appended by leader: 29, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.247314 28984 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:14.247982 28984 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.242240 29124 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:14.291494 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 82139.4us
I20260709 12:19:14.293282 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.294445 12569 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 55: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:14.295671 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.296609 12643 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:14.304859 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.308764 12494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2488) took 2463 ms. Trace:
I20260709 12:19:14.313683 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.314747 12639 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:14.315843 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.316272 12641 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:14.317548 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 36 index: 19 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.318284 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 38.
I20260709 12:19:14.319311 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.320940 29136 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:14.321512 29136 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:14.322003 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.322702 12566 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:14.326329 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:14.326874 13261 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.29-46.29]
I20260709 12:19:14.334096 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4724) took 1084 ms. Trace:
I20260709 12:19:14.338337 12640 rpcz_store.cc:276] 0709 12:19:13.249221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.326031 (+1076810us) service_pool.cc:224] Handling call
0709 12:19:14.333986 (+  7955us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:14.301159 28830 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:14.345101 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:14.346009 13253 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:14.346715 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:14.347636 12642 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:14.348412 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.353363 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.355276 12643 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:19:14.298561 29124 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:14.358117 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.359081 29124 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.341452 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.343047 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.334812 29044 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.359642 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
I20260709 12:19:14.311838 12494 rpcz_store.cc:276] 0709 12:19:11.836961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:12.797195 (+960234us) service_pool.cc:224] Handling call
0709 12:19:12.797663 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:12.797959 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 31
0709 12:19:14.249972 (+1452013us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b1800296080. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf
0709 12:19:14.250237 (+   265us) log.cc:844] Serialized 14 byte log entry
0709 12:19:14.298948 (+ 48711us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:14.299006 (+    58us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:14.299064 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.300053 (+   989us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:14.300204 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54427,"prepare.queue_time_us":115791,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":2581,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":3299,"spinlock_wait_cycles":52114176,"thread_start_us":173176,"threads_started":2}
I20260709 12:19:14.363620 29144 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:14.383507 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48444.9us
I20260709 12:19:14.384616 29144 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.386912 29144 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.387368 29182 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:14.370636 29044 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.387926 29182 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.390234 29044 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:14.390892 29019 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:14.392566 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.393949 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.394671 12488 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:19:14.395869 13260 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:14.396260 13260 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:14.390976 12640 raft_consensus.cc:1240] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:14.372272 29189 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:14.401561 13256 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 32. Preceding OpId from leader: term: 47 index: 33. (index mismatch)
I20260709 12:19:14.433111 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:14.434056 12572 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:19:14.434752 12640 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:14.398591 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:14.436697 13261 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:14.436288 13253 raft_consensus.cc:2468] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:19:14.383097 28984 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.390260 29182 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.463025 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:14.460760 12640 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 28. Preceding OpId from leader: term: 46 index: 29. (index mismatch)
I20260709 12:19:14.448323 13260 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.28, Last appended by leader: 28, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.480528 12490 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:14.487102 29183 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 52.37, Next index: 38, Last known committed idx: 36, Time since last communication: 0.001s
I20260709 12:19:14.487772 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.482087 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:14.444291 29175 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:14.502835 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
W20260709 12:19:14.506525 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.507058 13261 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
W20260709 12:19:14.507249 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:14.508606 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.510044 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:14.510304 12488 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 50: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:14.501503 29175 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:14.513706 29175 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:14.500447 28883 consensus_queue.cc:1059] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.014s
I20260709 12:19:14.515273 28883 raft_consensus.cc:3055] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:14.515647 28883 raft_consensus.cc:740] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:14.516518 28883 consensus_queue.cc:260] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.451947 29044 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 47.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.068s
I20260709 12:19:14.518918 29044 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:14.519377 29044 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:14.519747 29136 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.520373 29044 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.33, Last appended by leader: 33, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.435163 28933 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.523240 29124 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:14.523499 28933 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:14.524871 12637 raft_consensus.cc:2393] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 45: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:14.525043 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48097.8us
I20260709 12:19:14.526651 12462 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:14.519184 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:14.534224 13259 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 28. Preceding OpId from leader: term: 47 index: 29. (index mismatch)
I20260709 12:19:14.536278 13260 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:14.537117 12639 raft_consensus.cc:2393] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 41: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:14.510072 29182 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:14.525832 29117 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:14.542444 29130 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.002s
W20260709 12:19:14.545033 29193 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:19:14.547663 28883 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:14.543211 28812 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.009s
I20260709 12:19:14.551463 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.553421 13254 raft_consensus.cc:1240] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
W20260709 12:19:14.562590 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.539268 29044 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:14.555779 29195 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
W20260709 12:19:14.557761 29197 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:19:14.540769 29019 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
W20260709 12:19:14.601372 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50029.3us
I20260709 12:19:14.543422 29182 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:14.575923 29117 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:14.614988 29117 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.27, Last appended by leader: 27, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.617487 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:14.618794 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 3065) took 1313 ms (client timeout 1589 ms). Trace:
I20260709 12:19:14.619194 13260 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 28. Preceding OpId from leader: term: 44 index: 27. (term mismatch)
I20260709 12:19:14.619607 13260 pending_rounds.cc:85] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 26
I20260709 12:19:14.619860 13260 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 41.27
I20260709 12:19:14.604142 28933 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20260709 12:19:14.620536 13260 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:14.621024 13260 pending_rounds.cc:107] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 43.28
I20260709 12:19:14.621305 13260 raft_consensus.cc:2889] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:14.630472 29182 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 52.37, Last appended by leader: 37, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.632694 29182 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:14.543577 29189 consensus_queue.cc:1059] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 54.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.061s
I20260709 12:19:14.646764 29189 raft_consensus.cc:3055] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 LEADER]: Stepping down as leader of term 54
I20260709 12:19:14.647209 29189 raft_consensus.cc:740] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:14.648141 29189 consensus_queue.cc:260] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 54.24, Last appended by leader: 24, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.536761 29119 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.059s
I20260709 12:19:14.625286 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6511) took 1032 ms. Trace:
I20260709 12:19:14.536085 29064 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 48.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.047s
I20260709 12:19:14.646520 12537 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:14.651072 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6503) took 1152 ms. Trace:
W20260709 12:19:14.619232 12565 rpcz_store.cc:269] 0709 12:19:13.305083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.617116 (+1312033us) service_pool.cc:224] Handling call
0709 12:19:14.618725 (+  1609us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:14.657891 12563 rpcz_store.cc:276] 0709 12:19:13.592644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.053859 (+461215us) service_pool.cc:224] Handling call
0709 12:19:14.066900 (+ 13041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:14.101700 (+ 34800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.624875 (+523175us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:14.625192 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57390464}
I20260709 12:19:14.663380 12642 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 29. Preceding OpId from leader: term: 53 index: 30. (index mismatch)
W20260709 12:19:14.669875 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65311.5us
I20260709 12:19:14.671741 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.674664 12564 rpcz_store.cc:276] 0709 12:19:13.498532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:13.534179 (+ 35647us) service_pool.cc:224] Handling call
0709 12:19:13.546257 (+ 12078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:13.930044 (+383787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.650783 (+720739us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54001a36d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:14.650988 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880965888}
W20260709 12:19:14.683351 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:14.697469 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.662503 29064 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
W20260709 12:19:14.665906 29166 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.519s	user 0.000s	sys 0.012s
I20260709 12:19:14.649597 29189 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:14.652014 28883 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.734505 12572 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
I20260709 12:19:14.672390 29019 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:14.722411 13259 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I20260709 12:19:14.652146 29130 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.664317 29136 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.651376 28721 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.744714 12565 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:14.737874 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:14.655012 29144 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:14.756595 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.705247 28534 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:14.749311 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 52007.6us
I20260709 12:19:14.771909 13217 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:14.774397 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:14.776270 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:14.777154 12642 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:14.682418 29138 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.065s
I20260709 12:19:14.726204 29064 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:14.793557 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.795584 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 3070) took 1332 ms. Trace:
I20260709 12:19:14.794929 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.780633 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:14.796030 12567 rpcz_store.cc:276] 0709 12:19:13.462868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:13.466762 (+  3894us) service_pool.cc:224] Handling call
0709 12:19:14.795373 (+1328611us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:14.795508 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23187328}
I20260709 12:19:14.794692 12496 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:14.781710 12644 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
I20260709 12:19:14.754426 29130 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.770259 28534 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.770025 28996 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.027s
I20260709 12:19:14.810225 29130 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.810719 12566 raft_consensus.cc:1275] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 23. Preceding OpId from leader: term: 47 index: 24. (index mismatch)
I20260709 12:19:14.810945 28534 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:14.811758 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.813108 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.760279 29107 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.047s
I20260709 12:19:14.760442 29183 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:14.735426 29166 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.734503 28780 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.742867 29146 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.058s
I20260709 12:19:14.795261 29120 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:14.749240 28883 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.812625 28430 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:14.829658 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:14.844067 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72231.2us
I20260709 12:19:14.853940 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:14.813243 29163 raft_consensus.cc:2804] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:14.770644 28721 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.847901 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:14.853049 12643 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 41.
I20260709 12:19:14.835433 29136 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.792343 29144 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.865826 29163 raft_consensus.cc:697] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:14.865444 29175 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.846257 29120 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:14.862484 12644 raft_consensus.cc:2393] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:14.864697 12490 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:14.898013 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.904712 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:14.906240 12494 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: []
I20260709 12:19:14.902316 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.867903 29134 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:14.911509 12643 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 57: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:14.915314 12495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:14.915820 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:14.914798 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 268351us
I20260709 12:19:14.880471 29120 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:14.919094 12497 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 48 index: 27. (index mismatch)
I20260709 12:19:14.918529 12495 raft_consensus.cc:2393] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:14.881955 28812 consensus_queue.cc:1059] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.044s
I20260709 12:19:14.926802 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:14.899698 28430 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:14.929714 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.932435 12644 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:14.873507 29136 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.935940 29144 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.936676 29136 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.939735 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:14.940659 12643 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:14.941708 28780 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:14.920409 29064 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 48.25, Last appended by leader: 25, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:14.944567 29064 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:14.919524 29134 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:14.947118 12643 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:14.949011 12643 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:14.954514 29175 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:14.864161 29019 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.31, Last appended by leader: 31, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:14.960918 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6170) took 1026 ms. Trace:
I20260709 12:19:14.961601 12569 rpcz_store.cc:276] 0709 12:19:13.934587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.362652 (+428065us) service_pool.cc:224] Handling call
0709 12:19:14.375471 (+ 12819us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.375879 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:14.375920 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:14.382649 (+  6729us) log.cc:844] Serialized 26 byte log entry
0709 12:19:14.582208 (+199559us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:14.582262 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.582361 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:14.960502 (+378141us) raft_consensus.cc:1685] finished
0709 12:19:14.960538 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:14.960822 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":105582,"threads_started":1,"wal-append.queue_time_us":311234}
I20260709 12:19:14.973973 29136 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:14.936726 29144 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.041486 12642 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:14.973152 28780 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:14.867053 29163 consensus_queue.cc:237] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.051250 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:14.931102 28430 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:15.057050 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.107599 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:15.108502 12569 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:15.094071 28430 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:15.093267 29107 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.059135 28601 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.050s
I20260709 12:19:15.093652 28780 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.063541 29144 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:15.175670 12535 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.014s	sys 0.009s
W20260709 12:19:15.176147 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 238606us
I20260709 12:19:15.188383 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.198530 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:15.200505 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:15.092238 29136 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:15.186439 12535 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.014s	sys 0.009s
I20260709 12:19:15.171116 29219 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:15.260504 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60075.4us
I20260709 12:19:15.202665 28883 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.158766 29107 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.295218 28780 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:15.265020 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.298545 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:15.273347 29136 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:15.276865 29206 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260709 12:19:15.290143 28721 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.237185 29117 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:15.262732 29180 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:19:15.293128 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:15.319907 12569 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:15.321105 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:15.319391 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55100.1us
I20260709 12:19:15.330509 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.330754 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.333288 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.328809 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3080) took 1137 ms. Trace:
W20260709 12:19:15.332007 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.316583 29216 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:19:15.336233 12567 rpcz_store.cc:276] 0709 12:19:14.191830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.806162 (+614332us) service_pool.cc:224] Handling call
0709 12:19:14.806601 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:14.806876 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 31
0709 12:19:15.173065 (+366189us) log.cc:844] Serialized 14 byte log entry
0709 12:19:15.328175 (+155110us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:15.328236 (+    61us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:15.328300 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.328405 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.328701 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22907,"prepare.queue_time_us":502401,"raft.queue_time_us":23768,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":111,"thread_start_us":475136,"threads_started":2}
I20260709 12:19:15.339998 12566 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 47. Current term is 48. Ops: []
W20260709 12:19:15.340283 29044 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.757s	user 0.013s	sys 0.000s
I20260709 12:19:15.341733 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 55 index: 32. (index mismatch)
I20260709 12:19:15.354452 12567 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 42 index: 26. (index mismatch)
I20260709 12:19:15.264588 29219 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.316159 29107 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.360610 12566 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:19:15.326831 29224 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:15.368496 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.347600 29216 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.352136 29136 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:15.373927 12536 leader_election.cc:400] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
I20260709 12:19:15.363227 29229 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:15.305691 28984 consensus_queue.cc:1175] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9485b0afaf144396adf854fff857414e" responder_term: 47 status { last_received { term: 45 index: 28 } last_committed_idx: 28 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: [46.29-46.29]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:19:15.359982 28780 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:15.377501 12536 leader_election.cc:403] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:15.379976 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3077) took 1211 ms. Trace:
I20260709 12:19:15.377904 29229 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.394241 29229 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.396234 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4734) took 1094 ms. Trace:
I20260709 12:19:15.396775 12536 leader_election.cc:400] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:19:15.397142 12536 leader_election.cc:403] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Cancelling election due to peer responding with higher term
I20260709 12:19:15.400071 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:15.396667 12637 rpcz_store.cc:276] 0709 12:19:14.302251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.302433 (+   182us) service_pool.cc:224] Handling call
0709 12:19:15.396012 (+1093579us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400155254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:15.396177 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":470168960}
I20260709 12:19:15.407449 12638 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:15.408437 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.411293 12642 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:19:15.412305 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:15.414285 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:15.378993 29166 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.383801 29144 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.386147 29044 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:15.415107 13259 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 55.
I20260709 12:19:15.377344 29136 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:15.415285 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:15.375509 29216 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:15.389665 12564 rpcz_store.cc:276] 0709 12:19:14.168354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.718308 (+549954us) service_pool.cc:224] Handling call
0709 12:19:14.759166 (+ 40858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:14.770766 (+ 11600us) raft_consensus.cc:1534] Early marking committed up to index 32
0709 12:19:15.117411 (+346645us) log.cc:844] Serialized 14 byte log entry
0709 12:19:15.379593 (+262182us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:15.379666 (+    73us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:19:15.379722 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.379789 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.379918 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65399,"prepare.queue_time_us":462803,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":27010,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":494613,"threads_started":2,"wal-append.queue_time_us":227960}
I20260709 12:19:15.416489 12645 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:15.434940 12645 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:15.404649 28689 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.408107 28984 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election lost for term 46. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47.
I20260709 12:19:15.415421 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:15.375401 29222 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.426477 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3071) took 1288 ms. Trace:
I20260709 12:19:15.433215 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:15.439813 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:15.446252 12644 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:15.447809 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:15.411226 28780 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 54. Reason: Vote denied by peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859) with higher term. Message: Invalid argument: T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55.
I20260709 12:19:15.411123 29229 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20260709 12:19:15.414453 29163 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:15.381156 28430 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.418332 29230 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:15.471624 29230 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:19:15.472460 29230 consensus_queue.cc:1243] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 46.29 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 28
I20260709 12:19:15.449163 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:15.474478 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.373960 29224 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.454120 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:15.475512 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:15.476745 12536 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:15.476982 29224 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.436501 29134 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.477988 29224 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:15.478351 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2078) took 1350 ms. Trace:
I20260709 12:19:15.478941 29044 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.479502 29134 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:15.417968 29064 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.480690 29064 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:15.460013 29220 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 12:19:15.460757 29184 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.271s	user 0.006s	sys 0.007s
I20260709 12:19:15.436412 29144 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:15.483194 29144 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.483625 29144 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:15.485563 12492 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
I20260709 12:19:15.485370 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6010) took 1098 ms. Trace:
I20260709 12:19:15.435246 29136 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:15.435701 12645 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.501564 12645 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:15.447984 12572 rpcz_store.cc:276] 0709 12:19:14.138036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.453185 (+315149us) service_pool.cc:224] Handling call
0709 12:19:14.729354 (+276169us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:14.735680 (+  6326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.422081 (+686401us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54002459d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:15.426382 (+  4301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":558577920}
I20260709 12:19:15.466553 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 58 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:15.503137 12644 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
W20260709 12:19:15.463486 29189 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.717s	user 0.015s	sys 0.000s
I20260709 12:19:15.470654 29219 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.478789 13256 rpcz_store.cc:276] 0709 12:19:14.127679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.152108 (+ 24429us) service_pool.cc:224] Handling call
0709 12:19:14.167742 (+ 15634us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:14.436013 (+268271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.478109 (+1042096us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:15.478291 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527061888}
I20260709 12:19:15.458666 28984 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:15.491101 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:15.492026 12496 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
W20260709 12:19:15.494673 29205 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.713s	user 0.007s	sys 0.000s
W20260709 12:19:15.453336 29182 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.820s	user 0.015s	sys 0.004s
I20260709 12:19:15.488137 12640 rpcz_store.cc:276] 0709 12:19:14.386474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.478450 (+ 91976us) service_pool.cc:224] Handling call
0709 12:19:14.478823 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:14.479080 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:19:14.504378 (+ 25298us) log.cc:844] Serialized 14 byte log entry
0709 12:19:14.756215 (+251837us) log.cc:844] Serialized 14 byte log entry
0709 12:19:15.342632 (+586417us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b18000fa900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26d42a6 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1
0709 12:19:15.344368 (+  1736us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b18000fa900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:19:15.484772 (+140404us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b18000fa900. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:19:15.484940 (+   168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:15.485001 (+    61us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:15.485064 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.485141 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.485299 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":740643,"prepare.queue_time_us":25845,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":1848,"raft.queue_time_us":18403,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":17089,"spinlock_wait_cycles":73891712,"thread_start_us":231526,"threads_started":1,"wal-append.queue_time_us":453474}
W20260709 12:19:15.464010 29225 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:19:15.446655 28689 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:15.530200 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 97077.7us
I20260709 12:19:15.494778 28430 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:15.534860 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54943.3us
I20260709 12:19:15.535768 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:15.536356 12564 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:15.506597 29163 raft_consensus.cc:2804] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:15.536866 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.451953 29222 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.479503 29044 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.542203 29182 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20260709 12:19:15.530221 29234 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.020s
I20260709 12:19:15.533526 29242 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.012s
I20260709 12:19:15.509905 28984 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.544903 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.544955 28984 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:15.517268 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:15.547173 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.593945 29044 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:15.545749 29234 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:15.610821 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63726.5us
I20260709 12:19:15.618068 29163 raft_consensus.cc:697] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:15.620337 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:15.604795 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.604975 29242 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:15.619897 13262 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 42.
I20260709 12:19:15.633739 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3078) took 1465 ms. Trace:
I20260709 12:19:15.623777 29242 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.619397 29163 consensus_queue.cc:237] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 46.24, Last appended by leader: 24, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.635020 29242 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:15.639628 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 858946us
I20260709 12:19:15.640899 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:15.641841 29038 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:15.642344 29038 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:15.642580 29064 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.642781 29038 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:15.644392 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6006) took 1259 ms. Trace:
I20260709 12:19:15.644891 12641 rpcz_store.cc:276] 0709 12:19:14.385330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.397050 (+ 11720us) service_pool.cc:224] Handling call
0709 12:19:14.440682 (+ 43632us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.441079 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:14.441106 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:14.453833 (+ 12727us) log.cc:844] Serialized 26 byte log entry
0709 12:19:14.561391 (+107558us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:19:14.561458 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.561596 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:15.493612 (+932016us) spinlock_profiling.cc:243] Waited 7.46 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:15.631210 (+137598us) raft_consensus.cc:1685] finished
0709 12:19:15.631230 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.644312 (+ 13082us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":555,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":7974,"spinlock_wait_cycles":17160448,"thread_start_us":74546,"threads_started":1,"wal-append.queue_time_us":199576}
W20260709 12:19:15.646344 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.623533 28430 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:15.625006 29231 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.647229 28430 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:15.647715 28430 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:15.647456 29231 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.641274 28689 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:15.654012 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 243569us
W20260709 12:19:15.654070 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:15.656103 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.630414 29241 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.634261 12570 rpcz_store.cc:276] 0709 12:19:14.168798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.729211 (+560413us) service_pool.cc:224] Handling call
0709 12:19:14.757220 (+ 28009us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.779938 (+ 22718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:14.779979 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:14.793369 (+ 13390us) log.cc:844] Serialized 26 byte log entry
0709 12:19:15.159350 (+365981us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:15.159404 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.159487 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:15.633444 (+473957us) raft_consensus.cc:1685] finished
0709 12:19:15.633468 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.633618 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22185,"raft.queue_time_us":184779,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":331298,"threads_started":1,"wal-append.queue_time_us":515950}
I20260709 12:19:15.649946 29045 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:15.649948 29219 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:15.662199 12645 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 24. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
W20260709 12:19:15.658488 29120 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.689s	user 0.008s	sys 0.006s
I20260709 12:19:15.639619 29222 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.660238 29224 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:15.664004 29045 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:15.704011 12564 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:15.663565 28776 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.033s
W20260709 12:19:15.710002 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 161650us
I20260709 12:19:15.677909 29189 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.677557 28969 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.037s
I20260709 12:19:15.664245 29241 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.714740 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:15.717700 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.719785 12564 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:15.707545 29045 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:15.693946 29231 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.710480 29212 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.668066 28984 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.677569 29120 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.727171 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:15.713743 28849 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.721334 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.722860 28830 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.743424 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.744453 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:15.746634 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.726187 29218 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:15.747788 12642 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:15.712410 29224 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.721908 28721 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.018s
I20260709 12:19:15.718545 29189 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.750638 29182 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.770128 29182 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.739279 28849 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.749161 29229 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:19:15.774223 29229 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:15.774647 29229 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:19:15.742359 28689 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:15.777052 29182 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:15.777674 29245 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:19:15.779799 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:15.780949 13256 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Advancing to term 56
W20260709 12:19:15.781522 29223 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.393s	user 0.002s	sys 0.000s
I20260709 12:19:15.748980 28780 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:15.790558 12493 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 41 index: 22. Preceding OpId from leader: term: 44 index: 24. (index mismatch)
I20260709 12:19:15.725271 29219 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.021s
I20260709 12:19:15.795547 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2086) took 1099 ms. Trace:
I20260709 12:19:15.748770 28883 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:15.796015 13255 rpcz_store.cc:276] 0709 12:19:14.696120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.696382 (+   262us) service_pool.cc:224] Handling call
0709 12:19:14.722024 (+ 25642us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.734588 (+ 12564us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:14.734624 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:14.754394 (+ 19770us) log.cc:844] Serialized 26 byte log entry
0709 12:19:15.159842 (+405448us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:15.159894 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.169787 (+  9893us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:15.780653 (+610866us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:15.795207 (+ 14554us) raft_consensus.cc:1685] finished
0709 12:19:15.795248 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.795483 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7239,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":16396,"spinlock_wait_cycles":36348672,"thread_start_us":354255,"threads_started":1,"wal-append.queue_time_us":548954}
I20260709 12:19:15.797595 28883 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.798095 28883 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:15.798619 29032 raft_consensus.cc:3055] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:15.799275 29032 raft_consensus.cc:740] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:15.747845 28830 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.800707 29136 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.749365 29222 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:15.804291 29144 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.807001 29229 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:15.807816 29203 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.013s	user 0.002s	sys 0.015s
I20260709 12:19:15.809269 29229 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.754732 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:15.756050 29120 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:15.760186 29246 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:15.817307 29189 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:15.763046 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:15.818616 29219 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.819253 12570 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:15.819169 29219 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.750360 29216 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.821406 29219 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.810377 29136 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.818097 29241 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.821930 29144 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.823750 29224 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.824397 29216 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.825914 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:15.827065 29229 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:15.767040 28984 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.827569 29229 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.828066 29229 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:15.829149 28984 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:15.830108 28984 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:15.825776 13257 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:19:15.734779 29212 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.800580 29032 consensus_queue.cc:260] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.836545 29212 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.836622 29032 raft_consensus.cc:3060] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:15.753288 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.821961 29246 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:15.838271 29246 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.838328 12564 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
I20260709 12:19:15.838636 29229 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.844233 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:15.845904 29216 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.846707 29216 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:15.853533 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 239) took 56611 ms (client timeout 15000 ms). Trace:
I20260709 12:19:15.853892 29180 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:15.854532 29180 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:15.854625 12568 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:15.855207 12568 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:15.855418 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:15.855813 29180 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 47.30, Last appended by leader: 30, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.856122 12568 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 45.29, Last appended by leader: 29, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.821305 12570 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:15.858879 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.860003 12570 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:15.857528 12568 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:19:15.823678 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69041.1us
I20260709 12:19:15.864037 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:15.833321 28984 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.882431 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:15.899133 13259 raft_consensus.cc:1275] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 44 index: 30. Preceding OpId from leader: term: 45 index: 31. (index mismatch)
I20260709 12:19:15.901437 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:15.902109 13260 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:15.890174 13256 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 56.
I20260709 12:19:15.899219 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.901206 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:15.911145 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:15.905120 12570 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 55.
I20260709 12:19:15.914358 29242 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.881558 29064 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:15.918716 29064 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:15.854159 12394 rpcz_store.cc:269] 0709 12:18:19.241851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:22.389386 (+3147535us) service_pool.cc:224] Handling call
0709 12:19:15.853118 (+53463732us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:04.362175 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:04.362961 (+   786us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:04.363020 (+    59us) write_op.cc:435] Acquired schema lock
0709 12:19:04.363043 (+    23us) tablet.cc:662] Decoding operations
0709 12:19:04.369779 (+  6736us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:04.369873 (+    94us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:04.369906 (+    33us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:19:04.371267 (+  1361us) tablet.cc:701] Row locks acquired
0709 12:19:04.371330 (+    63us) write_op.cc:260] PREPARE: finished
0709 12:19:04.371524 (+   194us) write_op.cc:270] Start()
0709 12:19:04.371697 (+   173us) write_op.cc:276] Timestamp: P: 1783599544371447 usec, L: 0
0709 12:19:04.371737 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:19:04.576356 (+204619us) log.cc:844] Serialized 5179 byte log entry
0709 12:19:05.723759 (+1147403us) spinlock_profiling.cc:243] Waited 7.63 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:06.795200 (+1071441us) op_driver.cc:464] REPLICATION: finished
0709 12:19:07.346888 (+551688us) write_op.cc:301] APPLY: starting
0709 12:19:07.347057 (+   169us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:08.036822 (+689765us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:08.036881 (+    59us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:08.668833 (+631952us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:08.669230 (+   397us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0709 12:19:08.669283 (+    53us) write_op.cc:312] APPLY: finished
0709 12:19:08.717592 (+ 48309us) log.cc:844] Serialized 121 byte log entry
0709 12:19:10.395584 (+1677992us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b
0709 12:19:10.707006 (+311422us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:10.725900 (+ 18894us) write_op.cc:454] Released schema lock
0709 12:19:10.725995 (+    95us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:10.726620 (+   625us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:10.726654 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":48284,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":524478,"apply.run_cpu_time_us":66608,"apply.run_wall_time_us":3379715,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58415,"num_ops":13,"prepare.queue_time_us":640877,"prepare.run_cpu_time_us":216553,"prepare.run_wall_time_us":1097358,"raft.queue_time_us":877519,"raft.run_cpu_time_us":19299,"raft.run_wall_time_us":448970,"replication_time_us":2397111,"spinlock_wait_cycles":35774976,"thread_start_us":3640184,"threads_started":5,"wal-append.queue_time_us":2871739,"wal-append.run_cpu_time_us":29125,"wal-append.run_wall_time_us":3190686}]]}
W20260709 12:19:15.921434 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 240) took 27708 ms (client timeout 15000 ms). Trace:
W20260709 12:19:15.923275 12394 rpcz_store.cc:269] 0709 12:18:48.213280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.921140 (+27707860us) service_pool.cc:211] Skipping call since client already timed out
0709 12:19:15.921340 (+   200us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:19:15.912006 13255 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
I20260709 12:19:15.878321 29216 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.880261 29246 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:15.922561 29242 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.906854 29038 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:15.880846 29229 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.924198 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:15.926146 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.931739 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:15.933169 29038 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:15.936681 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6015) took 1219 ms. Trace:
I20260709 12:19:15.937198 12638 rpcz_store.cc:276] 0709 12:19:14.717769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.393755 (+675986us) service_pool.cc:224] Handling call
0709 12:19:15.399062 (+  5307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:15.416637 (+ 17575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.936447 (+519810us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b5400155250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:15.936606 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11365632}
I20260709 12:19:15.937984 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.939128 28430 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.939363 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.940436 12564 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 42 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
I20260709 12:19:15.931139 13260 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 43.
I20260709 12:19:15.933104 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.923008 29222 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:15.902649 29212 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:15.944146 12567 raft_consensus.cc:2468] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:15.912088 29182 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:15.925304 28984 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:15.952476 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:15.926266 29064 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:15.955515 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 139) took 16753 ms (client timeout 15000 ms). Trace:
W20260709 12:19:15.955998 12394 rpcz_store.cc:269] 0709 12:18:59.195383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.946025 (+16750642us) service_pool.cc:211] Skipping call since client already timed out
0709 12:19:15.948862 (+  2837us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:19:15.954514 12564 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
W20260709 12:19:15.948097 29138 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.127s	user 0.020s	sys 0.012s
I20260709 12:19:15.944787 29216 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:15.963593 12567 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:15.965150 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:15.944195 29242 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:15.946926 29144 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:15.949981 29212 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.959122 29234 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:15.952327 29222 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:15.976787 29239 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.585s	user 0.003s	sys 0.000s
I20260709 12:19:15.968254 29045 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.978611 29222 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.979295 29045 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:15.953800 28849 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:15.981716 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3079) took 1812 ms. Trace:
W20260709 12:19:15.965114 29138 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.146s	user 0.021s	sys 0.012s
I20260709 12:19:15.987767 12565 rpcz_store.cc:276] 0709 12:19:14.169092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.750320 (+581228us) service_pool.cc:224] Handling call
0709 12:19:14.789795 (+ 39475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:15.116528 (+326733us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:15.116576 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:15.164107 (+ 47531us) log.cc:844] Serialized 26 byte log entry
0709 12:19:15.359821 (+195714us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:15.359874 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:15.359953 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:15.958821 (+598868us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:15.981283 (+ 22462us) raft_consensus.cc:1685] finished
0709 12:19:15.981306 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:15.981598 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":152583,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":28865408,"thread_start_us":444936,"threads_started":2,"wal-append.queue_time_us":183746}
W20260709 12:19:15.987973 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58174.1us
I20260709 12:19:15.971865 29216 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:15.980005 29212 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:15.960800 28984 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.009680 12566 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 33. Preceding OpId from leader: term: 48 index: 34. (index mismatch)
I20260709 12:19:15.959506 29182 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:15.997871 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:15.997020 29138 consensus_queue.cc:1243] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 6d4ece942fd5414998215d26edf91001 log is divergent from this leader: its last log entry 48.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:19:15.969612 29064 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.034s
I20260709 12:19:15.992913 29254 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:16.024036 12566 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 26. Preceding OpId from leader: term: 45 index: 26. (term mismatch)
I20260709 12:19:16.003455 29216 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:15.988564 29144 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.018929 28430 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.040099 12565 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 37. Preceding OpId from leader: term: 53 index: 38. (index mismatch)
I20260709 12:19:16.043406 29182 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:16.044746 29213 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.780s	user 0.003s	sys 0.001s
I20260709 12:19:16.047648 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:16.074138 13260 raft_consensus.cc:2368] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:19:16.027886 29117 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.005s
I20260709 12:19:16.056790 12570 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:19:16.057833 12568 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 29. Preceding OpId from leader: term: 46 index: 29. (term mismatch)
I20260709 12:19:15.988369 29242 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.030663 29254 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.026997 12566 pending_rounds.cc:85] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 25
I20260709 12:19:16.063889 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:16.084542 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:16.064316 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66550.8us
I20260709 12:19:16.079406 12568 pending_rounds.cc:85] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 28
I20260709 12:19:16.086732 12568 pending_rounds.cc:107] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 45.29
I20260709 12:19:16.087178 12568 raft_consensus.cc:2889] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:16.088735 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:16.054618 28830 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.063895 28780 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:16.091482 12566 pending_rounds.cc:107] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 43.26
I20260709 12:19:16.092362 12572 raft_consensus.cc:2410] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 50 because replica has last-logged OpId of term: 49 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 29.
I20260709 12:19:16.092108 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:16.092083 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:16.102427 12537 leader_election.cc:400] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:19:16.102891 12537 leader_election.cc:403] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:16.054718 29246 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.096168 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:16.097498 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.099169 12566 raft_consensus.cc:2889] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:16.112836 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6014) took 1557 ms. Trace:
I20260709 12:19:16.113838 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 52 index: 30. (index mismatch)
I20260709 12:19:16.115304 12566 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 44 index: 25. (index mismatch)
I20260709 12:19:16.117249 29260 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:16.117734 29260 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:16.118681 29260 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.120013 29260 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:16.113252 12646 rpcz_store.cc:276] 0709 12:19:14.555133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.647427 (+ 92294us) service_pool.cc:224] Handling call
0709 12:19:14.647838 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:14.648297 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:14.648332 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:14.648899 (+   567us) log.cc:844] Serialized 26 byte log entry
0709 12:19:14.871667 (+222768us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:14.871712 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:14.871781 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:16.044082 (+1172301us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:16.103644 (+ 59562us) raft_consensus.cc:1685] finished
0709 12:19:16.103669 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:16.112758 (+  9089us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1887,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":70,"spinlock_wait_cycles":60937600,"thread_start_us":222281,"threads_started":1,"wal-append.queue_time_us":496926}
I20260709 12:19:16.096010 12568 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 50
I20260709 12:19:16.099895 12565 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:19:16.091629 29241 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader election won for term 56
I20260709 12:19:16.083228 28721 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:16.093649 29136 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election won for term 54
W20260709 12:19:16.140623 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44551.7us
W20260709 12:19:16.094383 29230 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 46 is higher.
I20260709 12:19:16.143015 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:16.069818 29229 consensus_queue.cc:1048] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.032s
I20260709 12:19:16.090354 29254 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:16.149578 12568 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 31. Preceding OpId from leader: term: 50 index: 32. (index mismatch)
I20260709 12:19:16.150839 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:16.151955 13262 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:16.155936 12568 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: [53.30-53.30]
I20260709 12:19:16.158607 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 58 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:16.103570 28780 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election won for term 43
W20260709 12:19:16.159242 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56994.3us
I20260709 12:19:16.159355 13262 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:16.109920 29255 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.162060 29255 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.160830 13255 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:16.164943 13255 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:16.146512 29136 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.168843 13262 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 31. Preceding OpId from leader: term: 50 index: 32. (index mismatch)
I20260709 12:19:16.171947 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:16.167709 13255 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 47.30, Last appended by leader: 30, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.173563 13255 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:16.174288 12496 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.174791 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:16.175215 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 51 index: 22 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.171416 29241 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:16.175817 12492 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
I20260709 12:19:16.176542 29241 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 53.31, Last appended by leader: 31, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.149941 29044 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:16.181083 13258 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:16.138486 28721 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.184533 12490 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [45.29-45.29]
I20260709 12:19:16.167066 29136 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:16.154887 29219 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 49. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50.
I20260709 12:19:16.187057 13262 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 37. Preceding OpId from leader: term: 53 index: 38. (index mismatch)
I20260709 12:19:16.197028 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.188047 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:16.184486 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.164533 29255 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:16.193557 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.194237 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:16.194717 12497 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.195753 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.193105 12490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:16.199308 12492 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:19:16.200845 12493 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:16.201282 12491 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:16.202462 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:16.203514 12488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.208735 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.210215 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.212230 29258 raft_consensus.cc:2804] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:16.213054 29258 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.213565 29258 raft_consensus.cc:3060] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:16.213909 29266 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.001s
W20260709 12:19:16.216360 29256 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:19:16.209766 12497 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:19:16.213275 12489 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:16.222437 29264 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:16.223145 29264 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.202721 28883 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.202590 28601 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:16.211459 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:16.225589 29264 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:16.228009 28883 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.228477 28601 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:16.241675 28601 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:16.234462 12493 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:16.246798 12493 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:16.253486 29268 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:16.254061 29268 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:16.254411 29268 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:16.254613 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:16.260130 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.255347 12570 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:16.239252 12462 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:16.276647 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 604362us
I20260709 12:19:16.278328 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:16.279363 12567 raft_consensus.cc:2393] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 55: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:16.292164 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.296419 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.297506 12494 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:16.237299 28721 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.238222 28984 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:16.300684 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 157742us
I20260709 12:19:16.301810 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.234642 29246 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.238952 29242 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.237874 29144 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:16.309114 12570 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:16.239482 29107 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:16.238464 29266 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:16.313547 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.311044 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:16.295775 28883 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Leader pre-election won for term 58
W20260709 12:19:16.317109 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.317449 12567 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:16.319306 12567 raft_consensus.cc:1240] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 38. Current term is 39. Ops: []
I20260709 12:19:16.344666 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:16.350982 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 73498.5us
I20260709 12:19:16.354591 12565 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 28. Preceding OpId from leader: term: 55 index: 29. (index mismatch)
I20260709 12:19:16.261969 29212 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.356596 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:16.355297 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 32 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.320851 29107 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.361330 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:16.354166 29239 log.cc:927] Time spent T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:19:16.367394 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:16.302027 28812 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.056s
I20260709 12:19:16.321532 28883 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.312559 28780 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:16.378731 13258 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 55. Current term is 56. Ops: [55.32-55.32]
I20260709 12:19:16.379112 28976 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:16.369401 28830 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.010s
I20260709 12:19:16.380565 28534 raft_consensus.cc:1064] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:16.345767 29266 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.353715 29144 raft_consensus.cc:2764] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election decision vote started in defunct term 46: won
I20260709 12:19:16.397348 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:16.350863 29182 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.401708 12490 raft_consensus.cc:2368] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 56. Current term is 57.
I20260709 12:19:16.300251 29264 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.061s
W20260709 12:19:16.414816 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47482us
I20260709 12:19:16.302140 29032 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 50.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.064s
I20260709 12:19:16.402521 12493 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:16.424376 12645 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:16.426532 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 42 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.511945 12640 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:16.429462 12494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.432976 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:16.434965 12638 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 48 index: 28. (index mismatch)
I20260709 12:19:16.367918 29107 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:16.453020 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:16.454015 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.513892 12637 raft_consensus.cc:2376] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48. Re-sending same reply.
I20260709 12:19:16.455731 12492 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001" is_pre_election: true
I20260709 12:19:16.457020 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.460693 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 46 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.464416 12491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
W20260709 12:19:16.514911 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 67551.7us
I20260709 12:19:16.456669 12489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "6d4ece942fd5414998215d26edf91001"
I20260709 12:19:16.412287 28883 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:16.418063 28969 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:16.520855 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88752.6us
I20260709 12:19:16.522387 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:16.435307 29134 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.461710 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.528887 12646 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:16.434507 28976 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:16.482228 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:16.471170 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:16.522709 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 152681us
W20260709 12:19:16.535151 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53005.8us
I20260709 12:19:16.441699 29189 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.516227 13260 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:16.524837 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.537187 12492 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:16.539682 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.540985 12645 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 49: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:16.542122 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.543216 12637 raft_consensus.cc:2393] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:16.544920 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:16.460692 28534 raft_consensus.cc:1068] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:19:16.546568 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.448939 29271 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:16.546877 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:16.416735 29136 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.573364 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.578405 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.541612 28976 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:16.594861 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.535619 29120 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:16.540901 29124 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:16.604602 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.610173 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.610641 13255 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 30. Preceding OpId from leader: term: 48 index: 30. (term mismatch)
I20260709 12:19:16.606290 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.611177 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.551712 29144 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election won for term 54
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:461: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
I20260709 12:19:16.534416 28969 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.609772 28534 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.005s
I20260709 12:19:16.614902 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 40 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.615474 29276 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:16.618906 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
W20260709 12:19:16.612854 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 950664us
I20260709 12:19:16.630120 12638 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:16.601440 12640 raft_consensus.cc:2468] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 42.
I20260709 12:19:16.622597 12639 raft_consensus.cc:2410] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 42 because replica has last-logged OpId of term: 41 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 40 index: 23.
I20260709 12:19:16.522111 29182 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.611232 13255 pending_rounds.cc:85] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 29
I20260709 12:19:16.540874 29134 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.647197 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.647570 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.648882 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.632746 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.651047 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 51 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.627128 29255 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:16.640964 13255 pending_rounds.cc:107] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 47.30
I20260709 12:19:16.682493 13255 raft_consensus.cc:2889] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:16.535629 29254 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.643087 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.551982 29216 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.547919 29189 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:16.655942 12643 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:16.621882 29144 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.626315 29276 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:16.658628 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123091us
I20260709 12:19:16.619889 29136 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:16.690227 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43666.5us
I20260709 12:19:16.695432 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:16.696951 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.697095 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:16.697188 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 135064us
I20260709 12:19:16.711783 12639 raft_consensus.cc:2410] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 50 because replica has last-logged OpId of term: 49 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 30.
W20260709 12:19:16.762362 12536 leader_election.cc:336] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.006s (SENT)
I20260709 12:19:16.778069 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.780126 13254 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:16.787374 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.788417 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.789289 12637 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:16.790796 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88720.5us
I20260709 12:19:16.791365 12640 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
W20260709 12:19:16.795002 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:16.694460 29144 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:16.851567 13258 raft_consensus.cc:2368] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56.
I20260709 12:19:16.700542 29276 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:16.743745 28776 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.107s
I20260709 12:19:16.652050 29120 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.127s
I20260709 12:19:16.871341 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:16.873018 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:16.969810 12536 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.234s (SENT)
I20260709 12:19:16.970477 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:16.980253 12645 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 56: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:16.981855 12637 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: []
I20260709 12:19:16.695420 29255 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.641125 28780 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 40.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:16.858068 29212 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:16.996642 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 205081us
I20260709 12:19:17.065901 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:17.125439 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 428400us
I20260709 12:19:17.009644 29038 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
W20260709 12:19:17.015956 29275 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:19:17.047343 29182 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.010172 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:16.979086 28534 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:16.991351 29218 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.029s
I20260709 12:19:16.983071 29271 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:17.058310 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:17.113811 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 414856us
I20260709 12:19:17.040446 29143 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:17.093645 29278 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.245s	user 0.000s	sys 0.001s
W20260709 12:19:17.116995 29272 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.506s	user 0.000s	sys 0.001s
W20260709 12:19:17.119081 29279 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
I20260709 12:19:17.146705 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.130134 29266 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.144544 28534 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:17.161019 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:17.138624 28601 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.896s	user 0.009s	sys 0.005s
I20260709 12:19:17.159885 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.183555 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.154060 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.184126 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.148792 29038 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:17.196333 12646 raft_consensus.cc:1275] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 24. Preceding OpId from leader: term: 48 index: 25. (index mismatch)
I20260709 12:19:17.194604 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.201042 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:17.200953 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.202147 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:19:17.202528 13262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.011s	sys 0.011s
W20260709 12:19:17.203300 13262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.011s	sys 0.012s
I20260709 12:19:17.203739 12639 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:17.207934 13218 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:17.149885 29124 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.042s
W20260709 12:19:17.193886 29107 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.658s	user 0.013s	sys 0.000s
I20260709 12:19:17.150502 29231 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 42.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.042s
W20260709 12:19:17.214190 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53267.4us
I20260709 12:19:17.218521 12641 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 LEADER]: Stepping down as leader of term 55
I20260709 12:19:17.221973 12641 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:17.188385 29254 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.194530 28969 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.166956 29182 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.034s
I20260709 12:19:17.222856 12641 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 55.32, Last appended by leader: 32, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.238066 12641 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:17.178205 29120 consensus_queue.cc:1048] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.038s
I20260709 12:19:17.195189 28534 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:17.225853 29045 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.205116 29117 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:17.241322 12495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34482 (request call id 6283) took 2324 ms. Trace:
I20260709 12:19:17.220027 29266 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:17.227494 29107 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:17.242961 29290 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:19:17.248965 28969 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.246064 29038 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 48.25, Last appended by leader: 25, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.257732 12495 rpcz_store.cc:276] 0709 12:19:14.916980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.934296 (+ 17316us) service_pool.cc:224] Handling call
0709 12:19:16.187533 (+1253237us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b5800005198. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26cee9c 00007ffbf26cea5b 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7ccfc0 00007ffbfa7b7929 00007ffbfa7b73d2 00007ffbf3d2ebd4
0709 12:19:16.187724 (+   191us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.187964 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.187989 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.194417 (+  6428us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.460629 (+266212us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:16.460690 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.460803 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.226772 (+765969us) raft_consensus.cc:1685] finished
0709 12:19:17.226804 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.241219 (+ 14415us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15155456,"thread_start_us":180160,"threads_started":1,"wal-append.queue_time_us":371275}
I20260709 12:19:17.241658 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.262112 29255 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.259509 29295 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:17.261052 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.008s	sys 0.022s
W20260709 12:19:17.275796 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.008s	sys 0.022s
I20260709 12:19:17.200068 28601 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.265367 12496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34492 (request call id 4694) took 1082 ms. Trace:
I20260709 12:19:17.265904 12643 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:17.281324 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.243798 29182 raft_consensus.cc:2804] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Leader election won for term 42
I20260709 12:19:17.254390 29189 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:17.287595 12638 consensus_meta.cc:287] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.639s	user 0.008s	sys 0.004s
I20260709 12:19:17.276769 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:17.278221 12496 rpcz_store.cc:276] 0709 12:19:16.182921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.183048 (+   127us) service_pool.cc:224] Handling call
0709 12:19:16.183414 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.195035 (+ 11621us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:19:16.253320 (+ 58285us) log.cc:844] Serialized 14 byte log entry
0709 12:19:16.306139 (+ 52819us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.311426 (+  5287us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.311618 (+   192us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:16.311658 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:16.311749 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.265048 (+953299us) raft_consensus.cc:1685] finished
0709 12:19:17.265077 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.265294 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19212,"prepare.queue_time_us":58770,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":15699,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":14979,"thread_start_us":52248,"threads_started":1,"wal-append.queue_time_us":207852}
I20260709 12:19:17.291289 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.296072 12641 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 55 index: 32. Preceding OpId from leader: term: 56 index: 32. (term mismatch)
I20260709 12:19:17.296445 12641 pending_rounds.cc:85] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 31
I20260709 12:19:17.296814 12641 pending_rounds.cc:107] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 55.32
I20260709 12:19:17.296485 12638 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:19:17.297156 12641 raft_consensus.cc:2889] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:17.290851 29295 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.262216 29045 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.321213 29212 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:17.291993 28883 consensus_meta.cc:287] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.755s	user 0.009s	sys 0.003s
W20260709 12:19:17.294754 29284 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:19:17.274394 29294 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:17.339298 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.347779 12642 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:17.306932 29182 raft_consensus.cc:697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:17.339928 28883 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.336648 29212 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.271677 28969 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.355942 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.331617 29120 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.335521 29124 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.303129 29189 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.320845 29295 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.303877 29173 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:17.341747 29166 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:17.372151 29183 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 48 current term: 48 is higher.
I20260709 12:19:17.374102 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:17.374562 29134 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.353571 29294 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:17.374902 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.375700 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.377043 13254 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2094) took 1206 ms. Trace:
W20260709 12:19:17.379431 29268 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.064s	user 0.011s	sys 0.002s
I20260709 12:19:17.387084 13254 rpcz_store.cc:276] 0709 12:19:16.170957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.171113 (+   156us) service_pool.cc:224] Handling call
0709 12:19:16.171507 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:16.780743 (+609236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.376808 (+596065us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b54005875d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:17.376981 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1399117056}
I20260709 12:19:17.380537 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6541) took 1503 ms. Trace:
I20260709 12:19:17.391112 12568 rpcz_store.cc:276] 0709 12:19:15.876863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.150669 (+273806us) service_pool.cc:224] Handling call
0709 12:19:16.151028 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.156616 (+  5588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.380274 (+1223658us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b54002cf6d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:17.380460 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10578176}
I20260709 12:19:17.365784 29120 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:17.385380 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 177523us
I20260709 12:19:17.394369 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:17.388322 28976 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.773s	user 0.015s	sys 0.000s
I20260709 12:19:17.402108 29264 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:17.401986 28976 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.402500 29264 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:17.403483 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 208975us
I20260709 12:19:17.403386 29264 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 50.31, Last appended by leader: 31, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.404265 28976 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.404673 29264 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:17.390652 29173 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:17.405987 29173 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 47.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.406440 28984 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:17.385368 29302 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.410602 29302 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.413173 29302 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:17.418011 29258 consensus_meta.cc:287] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.138s	user 0.013s	sys 0.000s
I20260709 12:19:17.424142 29117 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.421401 12497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4693) took 1238 ms. Trace:
I20260709 12:19:17.426196 12497 rpcz_store.cc:276] 0709 12:19:16.182607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.182731 (+   124us) service_pool.cc:224] Handling call
0709 12:19:17.421159 (+1238428us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400192b54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:17.421294 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24539392}
I20260709 12:19:17.426209 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.427605 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.385053 29287 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.418557 29258 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.430655 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.431794 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:17.440785 12643 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
W20260709 12:19:17.435041 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85790.9us
I20260709 12:19:17.385263 29301 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.442425 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:17.442672 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:17.443403 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:17.444053 12568 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:17.444576 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.445071 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:17.445567 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.364871 29124 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.364884 29189 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.424805 29117 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.458101 13254 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:17.450748 13259 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:17.465015 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:17.472505 12563 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:17.387558 29166 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.428470 29136 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:17.428524 28689 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:17.474870 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:17.484015 12568 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:17.395527 29268 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.457837 29170 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:17.417610 28534 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:17.558677 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113639us
W20260709 12:19:17.563580 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:17.565814 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:17.485867 29107 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.627204 28969 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.458246 28976 consensus_queue.cc:1059] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.048s
I20260709 12:19:17.627707 29107 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:17.467221 29032 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.101s
I20260709 12:19:17.625805 29288 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:17.626359 29295 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.446353 29134 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.043s
I20260709 12:19:17.457728 28721 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 51.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.081s
I20260709 12:19:17.475950 29260 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 23, Time since last communication: 0.141s
I20260709 12:19:17.464015 29296 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 36, Time since last communication: 0.155s
I20260709 12:19:17.537978 29306 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.569242 29230 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.539449 29302 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.474337 29300 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.171s
I20260709 12:19:17.480077 29038 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.137s
I20260709 12:19:17.449553 29287 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:17.663168 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 216242us
I20260709 12:19:17.457525 29301 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.654078 29183 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:17.495064 28689 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.643515 29316 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:19:17.642185 28601 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.629366 29107 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:17.669103 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 219327us
I20260709 12:19:17.688791 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4760) took 1068 ms. Trace:
I20260709 12:19:17.739760 12645 rpcz_store.cc:276] 0709 12:19:16.620429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.624391 (+  3962us) service_pool.cc:224] Handling call
0709 12:19:17.688462 (+1064071us) spinlock_profiling.cc:243] Waited 58 ms on lock 0x7b54003b0bd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:17.688688 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133510400}
I20260709 12:19:17.696282 29216 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.707782 12488 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2507) took 1412 ms. Trace:
I20260709 12:19:17.676100 29286 raft_consensus.cc:3055] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
W20260709 12:19:17.721148 29282 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.873s	user 0.003s	sys 0.003s
I20260709 12:19:17.721463 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4753) took 1255 ms. Trace:
I20260709 12:19:17.677754 29295 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.486181 29182 consensus_queue.cc:237] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 39.24, Last appended by leader: 24, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:17.673704 29143 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:19:17.737326 29163 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.665310 28969 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.077s
I20260709 12:19:17.736982 29212 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.750495 12640 rpcz_store.cc:276] 0709 12:19:16.490227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.650676 (+160449us) service_pool.cc:224] Handling call
0709 12:19:17.721241 (+1070565us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400152a54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:17.721393 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320216320}
I20260709 12:19:17.732247 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:17.719643 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:17.735983 28721 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:17.736089 28780 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:19:17.761186 12571 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 48 because replica has last-logged OpId of term: 46 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 28.
I20260709 12:19:17.742158 12488 rpcz_store.cc:276] 0709 12:19:16.294930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.295165 (+   235us) service_pool.cc:224] Handling call
0709 12:19:16.295702 (+   537us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:16.299064 (+  3362us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:19:16.318887 (+ 19823us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.251987 (+933100us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b180028d200. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:19:17.707018 (+455031us) spinlock_profiling.cc:243] Waited 4.7 ms on lock 0x7b180028d200. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:19:17.707257 (+   239us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:17.707332 (+    75us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:17.707386 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.707488 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.707692 (+   204us) inbound_call.cc:177] Queueing success response
0709 12:19:17.720315 (+ 12623us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
Metrics: {"mutex_wait_us":94935,"prepare.queue_time_us":18726,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":141681,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":37733,"spinlock_wait_cycles":67988352,"thread_start_us":254381,"threads_started":2,"wal-append.queue_time_us":280735}
I20260709 12:19:17.617385 29294 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 43.23, Last appended by leader: 23, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.766669 13256 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:17.678097 29302 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.743556 28883 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 58 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.746060 29286 raft_consensus.cc:740] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:17.674623 29230 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.676251 29306 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.755841 29045 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.757678 29316 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:17.773485 29224 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:17.775926 29298 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.392s	user 0.001s	sys 0.001s
I20260709 12:19:17.752637 28601 consensus_queue.cc:1059] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 50.34, Next index: 35, Last known committed idx: 33, Time since last communication: 0.031s
I20260709 12:19:17.770977 28721 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:17.790146 29314 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.032s
W20260709 12:19:17.818090 29304 log.cc:927] Time spent T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:19:17.819052 29317 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.819720 29317 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:17.822185 29305 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:19:17.763974 29163 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:17.824493 29307 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:19:17.825690 29163 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:17.826102 12638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4750) took 1395 ms (client timeout 1744 ms). Trace:
I20260709 12:19:17.826231 12637 raft_consensus.cc:1240] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: []
W20260709 12:19:17.826596 12638 rpcz_store.cc:269] 0709 12:19:16.431138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.524498 (+ 93360us) service_pool.cc:224] Handling call
0709 12:19:17.825879 (+1301381us) spinlock_profiling.cc:243] Waited 93 ms on lock 0x7b54002a49d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:17.826022 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213830912}
I20260709 12:19:17.832187 28984 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:17.814489 29224 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.835000 29117 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.836040 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4762) took 1197 ms. Trace:
I20260709 12:19:17.836475 12644 rpcz_store.cc:276] 0709 12:19:16.638709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.855053 (+216344us) service_pool.cc:224] Handling call
0709 12:19:16.980110 (+125057us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:17.010824 (+ 30714us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:17.048125 (+ 37301us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:17.048998 (+   873us) log.cc:844] Serialized 52 byte log entry
0709 12:19:17.395513 (+346515us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:17.616655 (+221142us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.735646 (+118991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.736699 (+  1053us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.835756 (+ 99057us) raft_consensus.cc:1685] finished
0709 12:19:17.835781 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.835970 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15838,"prepare.queue_time_us":352887,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":3172,"raft.queue_time_us":98902,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":5396,"thread_start_us":608959,"threads_started":3,"wal-append.queue_time_us":430977}
I20260709 12:19:17.836918 29124 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.840512 28780 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:17.842540 29301 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.844096 29107 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:17.845415 12492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4708) took 1416 ms (client timeout 1878 ms). Trace:
W20260709 12:19:17.846042 12492 rpcz_store.cc:269] 0709 12:19:16.429165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.429278 (+   113us) service_pool.cc:224] Handling call
0709 12:19:17.845147 (+1415869us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b54000f7dd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:17.845322 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50815360}
I20260709 12:19:17.847424 12643 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:17.841391 28780 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.763214 29295 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.854012 13256 raft_consensus.cc:2468] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 54.
I20260709 12:19:17.855425 28780 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.808681 29258 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.868310 29268 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.870083 29302 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.871482 29287 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.872244 28534 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.872922 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:17.873963 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2314) took 1502 ms. Trace:
I20260709 12:19:17.874368 29287 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:17.874918 29287 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:17.875442 29287 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:17.874372 13262 rpcz_store.cc:276] 0709 12:19:16.371372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.410774 (+ 39402us) service_pool.cc:224] Handling call
0709 12:19:16.411171 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.433982 (+ 22811us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.434017 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.451718 (+ 17701us) log.cc:844] Serialized 26 byte log entry
0709 12:19:17.204081 (+752363us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:17.204129 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.204207 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:17.873744 (+669537us) raft_consensus.cc:1685] finished
0709 12:19:17.873770 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:17.873906 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51684,"raft.queue_time_us":29805,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7747,"thread_start_us":654539,"threads_started":1,"wal-append.queue_time_us":829071}
I20260709 12:19:17.876495 29286 consensus_queue.cc:260] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.879032 29306 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:17.880849 29230 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:17.881526 29309 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.362s	user 0.002s	sys 0.000s
I20260709 12:19:17.896246 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:17.883316 29231 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.018s	sys 0.025s
I20260709 12:19:17.860790 29189 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:17.873092 29166 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.902837 28976 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:17.922248 28721 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.28, Last appended by leader: 28, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:17.923593 28721 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:17.927520 29287 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.928583 28689 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:17.936194 12568 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 30. Preceding OpId from leader: term: 49 index: 31. (index mismatch)
I20260709 12:19:17.939989 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:17.917249 29231 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.019s	sys 0.026s
I20260709 12:19:17.956540 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:17.957566 12570 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:17.959210 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:17.960136 12568 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:17.966840 12462 leader_election.cc:400] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56.
I20260709 12:19:17.968377 12462 leader_election.cc:403] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:19:17.963447 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:17.811327 29316 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:17.902462 29258 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.045s
I20260709 12:19:17.975718 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:17.996003 12564 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:17.969575 28812 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:17.976040 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:17.954360 29303 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:17.999303 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 501960us
I20260709 12:19:17.978418 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.003120 12563 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:18.005823 12461 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:18.007928 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:17.974705 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 234416us
I20260709 12:19:17.993395 29317 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:18.030676 29276 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.048s	user 0.008s	sys 0.008s
W20260709 12:19:18.095819 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90162.5us
W20260709 12:19:18.176867 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 491660us
I20260709 12:19:18.090783 13256 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 53 index: 31. (index mismatch)
I20260709 12:19:18.178996 12639 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:17.974325 28689 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.028s
W20260709 12:19:18.045750 11310 thread.cc:527] Waited for 1000ms trying to join with rpc worker (tid 12508)
I20260709 12:19:17.977905 28976 consensus_queue.cc:1059] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.093s
I20260709 12:19:18.117386 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:17.998741 28812 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.095283 12568 raft_consensus.cc:2468] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:17.977591 29280 raft_consensus.cc:3060] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 56
I20260709 12:19:18.145848 12641 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 48 index: 30. (index mismatch)
I20260709 12:19:18.094375 29230 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:18.163560 12536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.011s	sys 0.011s
I20260709 12:19:18.165609 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:18.175690 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:18.027702 29287 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:18.180372 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:18.016781 29303 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.188779 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.190106 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.190820 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.188817 12640 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
W20260709 12:19:18.250114 12536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.019s	sys 0.019s
I20260709 12:19:18.198210 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:18.251374 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 685589us
I20260709 12:19:18.251819 13254 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 51: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:18.253348 12536 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:18.194056 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:18.201764 28812 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:18.205971 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2097) took 1748 ms. Trace:
I20260709 12:19:18.251108 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:18.126866 12564 raft_consensus.cc:2468] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 56.
I20260709 12:19:18.206717 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:18.207715 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6032) took 1002 ms. Trace:
I20260709 12:19:18.262033 12637 rpcz_store.cc:276] 0709 12:19:17.205206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:17.206913 (+  1707us) service_pool.cc:224] Handling call
0709 12:19:17.826034 (+619121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:17.826755 (+   721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:18.207497 (+380742us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b54002a49d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:18.207635 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1422518144}
I20260709 12:19:18.262666 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 58 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:18.263129 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:18.263741 12637 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:18.163008 29268 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.253515 12646 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:18.194104 12563 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:18.254592 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.095894 29231 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
W20260709 12:19:18.245313 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 251233us
I20260709 12:19:18.145257 29107 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:18.263260 12641 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:18.250483 13256 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:18.265127 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:18.281831 12639 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
W20260709 12:19:18.209544 29320 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
W20260709 12:19:18.258105 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67223.2us
I20260709 12:19:18.259343 13257 rpcz_store.cc:276] 0709 12:19:16.457984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.458135 (+   151us) service_pool.cc:224] Handling call
0709 12:19:16.458489 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:16.458943 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:16.458984 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:16.459577 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 12:19:16.694083 (+234506us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:17.083136 (+389053us) log.cc:844] Serialized 14 byte log entry
0709 12:19:17.083421 (+   285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:17.083515 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:18.203537 (+1120022us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b18005088a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf27063f3 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:19:18.205542 (+  2005us) raft_consensus.cc:1685] finished
0709 12:19:18.205696 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:18.205892 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5840,"prepare.queue_time_us":461333,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":8482,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":79131,"spinlock_wait_cycles":24876928,"thread_start_us":537999,"threads_started":2,"wal-append.queue_time_us":638732}
I20260709 12:19:18.273036 12568 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I20260709 12:19:18.275194 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:18.248294 29136 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.756s	user 0.013s	sys 0.000s
I20260709 12:19:18.206640 29295 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:18.287940 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:18.298310 13258 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:18.303824 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.288368 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:18.283213 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 117683us
I20260709 12:19:18.313210 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:18.269366 29317 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:18.188302 29276 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.179855 29166 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.097s
W20260709 12:19:18.320037 12490 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34492 (request call id 4699) took 2132 ms (client timeout 1654 ms). Trace:
W20260709 12:19:18.329321 12490 rpcz_store.cc:269] 0709 12:19:16.187592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.192816 (+  5224us) service_pool.cc:224] Handling call
0709 12:19:18.319775 (+2126959us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540034ec54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:18.319950 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464614144}
W20260709 12:19:18.331220 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.209002 29175 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 47.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.091s
I20260709 12:19:18.281214 29107 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.326493 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:18.278081 29231 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:18.354497 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.284049 28780 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:18.311071 29136 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:19:18.362397 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 74090.6us
I20260709 12:19:18.365355 12641 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 58.
I20260709 12:19:18.308172 29260 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:18.267153 29230 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.370208 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.371083 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:18.376493 12646 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:18.280893 29268 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.338094 29306 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:18.346263 29107 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:18.412263 12643 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
I20260709 12:19:18.358107 29231 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:18.414275 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:18.371472 29287 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.029s
I20260709 12:19:18.406353 29326 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.101s
I20260709 12:19:18.454137 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.342468 28976 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:18.355778 29163 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:19:18.466354 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96362.9us
I20260709 12:19:18.462613 12564 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:18.488169 12563 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 49: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:18.506178 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.505707 12643 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 28. Preceding OpId from leader: term: 55 index: 29. (index mismatch)
I20260709 12:19:18.509661 12566 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:18.434115 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.433383 29306 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.440218 29257 raft_consensus.cc:3055] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
W20260709 12:19:18.442622 29330 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:19:18.447387 29230 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:18.448328 29276 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.367288 29317 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.367161 29325 consensus_queue.cc:1059] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 38.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.095s
W20260709 12:19:18.379966 29144 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.507s	user 0.004s	sys 0.013s
I20260709 12:19:18.454874 12570 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:18.378384 29260 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:18.506330 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.514175 12572 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:18.426749 28780 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:18.377468 29335 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.145s
I20260709 12:19:18.432267 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:18.524894 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.528805 12565 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:18.530311 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.450080 29316 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:18.515013 29306 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:18.544404 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.482848 29175 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.514823 29257 raft_consensus.cc:740] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:18.517554 29294 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:18.482491 29264 consensus_queue.cc:1059] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 51.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.057s
I20260709 12:19:18.527004 28976 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:18.566344 12572 raft_consensus.cc:1240] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:18.538990 29163 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260709 12:19:18.534843 29260 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:18.514101 29136 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.041s
I20260709 12:19:18.542384 29276 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:18.564455 12565 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:18.567561 29166 raft_consensus.cc:3055] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:19:18.574368 12570 raft_consensus.cc:1240] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 44. Current term is 45. Ops: [44.24-44.24]
I20260709 12:19:18.534523 29317 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.539371 29144 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:18.596139 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 249005us
I20260709 12:19:18.578783 29303 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.586028 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
W20260709 12:19:18.584160 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 285522us
W20260709 12:19:18.591019 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 230715us
I20260709 12:19:18.591890 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.584399 12536 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.571971 29175 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.614471 12565 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 54: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:18.608887 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:18.607187 12461 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:18.571399 28780 consensus_queue.cc:1048] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 21, Time since last communication: 0.026s
I20260709 12:19:18.595224 29166 raft_consensus.cc:740] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:18.617875 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 53 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.603849 29264 raft_consensus.cc:3055] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:18.615201 29287 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.018s
I20260709 12:19:18.619278 29276 raft_consensus.cc:2804] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:18.573639 29294 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.622789 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:18.631693 12566 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 57: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:18.644940 12639 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:18.572970 29316 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.030s
I20260709 12:19:18.613363 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.648136 29182 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:18.649169 29325 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:18.651904 29317 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.653469 29144 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.654434 29166 consensus_queue.cc:260] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.683667 29166 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:19:18.646981 13217 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:18.668941 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:18.691540 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.692373 12566 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 41.
I20260709 12:19:18.645087 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:18.634300 29264 raft_consensus.cc:740] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:18.695346 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72662.6us
W20260709 12:19:18.695860 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82755.6us
I20260709 12:19:18.688858 29260 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:18.703171 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56282.8us
I20260709 12:19:18.600697 29257 consensus_queue.cc:260] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 48.34, Last appended by leader: 34, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.706121 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:18.706827 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61854.3us
I20260709 12:19:18.704123 13217 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:18.710394 12566 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 55.
I20260709 12:19:18.710891 12461 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:18.694415 29338 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:19:18.681098 29268 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:18.693718 29182 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.708138 29189 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:18.680485 29276 raft_consensus.cc:697] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:18.693612 29325 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:18.718088 13217 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:18.721939 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.724910 12566 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 55.
I20260709 12:19:18.699295 29218 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:18.726806 13255 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: [53.38-53.38]
I20260709 12:19:18.706357 28780 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:18.729661 29292 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Leader election won for term 58
I20260709 12:19:18.732286 29292 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 58 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:18.734277 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.635449 28976 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 45.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.716208 29343 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:18.750438 29344 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:19:18.754304 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:18.756232 13255 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:18.719732 29346 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:18.717166 29340 raft_consensus.cc:2804] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:18.776170 29346 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.696430 29107 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.723493 29182 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:18.782436 29107 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.783413 29107 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Leader election won for term 57
I20260709 12:19:18.783975 29107 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:18.733810 29189 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.736758 29264 consensus_queue.cc:260] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 51.30, Last appended by leader: 30, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.791626 29264 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:18.733968 29216 raft_consensus.cc:2804] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:18.793972 29216 raft_consensus.cc:697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:18.795157 29216 consensus_queue.cc:237] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 47.31, Last appended by leader: 31, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.726503 29325 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:18.759845 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 58 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:18.800005 13256 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:18.762671 29343 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.764909 29294 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:18.765887 29260 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.757618 28780 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:18.738080 29280 consensus_meta.cc:287] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.536s	user 0.012s	sys 0.000s
I20260709 12:19:18.831704 29343 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:18.770136 29218 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 52.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:18.720930 29335 raft_consensus.cc:2804] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader election won for term 56
I20260709 12:19:18.771155 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:18.773779 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.774330 29175 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:18.775149 29276 consensus_queue.cc:237] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 43.24, Last appended by leader: 24, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:18.776504 29340 raft_consensus.cc:697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:18.779469 29346 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:18.757931 12566 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 46. Current term is 47. Ops: [46.26-46.26]
I20260709 12:19:18.874431 29335 raft_consensus.cc:697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:18.875813 29335 consensus_queue.cc:237] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 54.31, Last appended by leader: 31, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.840288 13256 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 58.
I20260709 12:19:18.772086 28812 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.860704 29230 raft_consensus.cc:3055] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:19:18.785076 29107 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 55.27, Last appended by leader: 27, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:18.880550 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 179646us
W20260709 12:19:18.862989 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 130266us
I20260709 12:19:18.892525 13255 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:18.895637 12460 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:18.897105 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:19:18.869695 29173 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.460s	user 0.004s	sys 0.014s
I20260709 12:19:18.857424 29280 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election lost for term 55. Reason: Vote denied by peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793) with higher term. Message: Invalid argument: T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56.
W20260709 12:19:18.887463 28721 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.955s	user 0.016s	sys 0.000s
I20260709 12:19:18.905490 12537 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:18.906708 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:18.918107 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:18.918083 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:18.910583 29173 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.501s	user 0.005s	sys 0.014s
I20260709 12:19:18.888707 29230 raft_consensus.cc:740] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:18.963153 13260 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 56 index: 32. (index mismatch)
W20260709 12:19:18.962005 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44002.8us
I20260709 12:19:18.956624 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.913929 29136 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:18.899281 29329 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:18.932567 29163 raft_consensus.cc:2804] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:18.978703 28969 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.003s
I20260709 12:19:18.983435 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.991228 13260 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:18.991536 28812 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:18.987039 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:18.971060 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:18.993827 28969 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:18.994256 28969 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:18.994649 28969 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:18.998466 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:18.999192 29319 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:19.001595 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:18.999490 29294 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:18.899184 29144 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.074s
W20260709 12:19:19.008489 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 112955us
W20260709 12:19:19.009385 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 112521us
I20260709 12:19:19.000255 29319 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:19.052587 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:18.899103 29292 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 56.36, Last appended by leader: 36, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:18.979205 29355 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:19.069923 12570 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:19.034853 29356 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:19.093580 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.099105 29374 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.038975 29361 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
W20260709 12:19:19.103183 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 93852.8us
W20260709 12:19:19.102087 29333 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.691s	user 0.003s	sys 0.003s
I20260709 12:19:19.108017 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.115322 12570 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:19.007797 28776 consensus_queue.cc:1059] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.079s
W20260709 12:19:19.126942 12571 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 3075) took 4951 ms (client timeout 1905 ms). Trace:
W20260709 12:19:19.129803 12571 rpcz_store.cc:269] 0709 12:19:14.154595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:14.298700 (+144105us) service_pool.cc:224] Handling call
0709 12:19:19.106235 (+4807535us) spinlock_profiling.cc:243] Waited 3.45 s on lock 0x7b54001a2cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:19.106403 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7944583552}
I20260709 12:19:19.141505 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:19.069406 29363 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.001838 28780 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.28, Last appended by leader: 28, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:19.150738 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 118715us
I20260709 12:19:19.146142 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.160784 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:19:19.054358 29294 consensus_queue.cc:1059] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.055s
I20260709 12:19:19.002452 29329 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:19.102134 29358 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:19:19.091269 29144 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:19.125029 29374 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:19.154641 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:19.176959 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:19.149909 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 144345us
I20260709 12:19:19.249742 12609 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:19.158340 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:19.092722 29355 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:19.171207 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.72473e+06us
I20260709 12:19:19.153483 29356 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:19.167074 29231 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.688s	user 0.009s	sys 0.004s
W20260709 12:19:19.183645 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 79598us
I20260709 12:19:19.169096 29364 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:19.261551 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:19.166743 29363 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:19.187292 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:19.265949 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:19.190380 29359 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 12:19:19.190548 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:19.270112 12565 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:19.196774 29321 raft_consensus.cc:3055] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:19.199674 29120 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:19.204457 29329 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:19.208554 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.213271 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50708 (request call id 2321) took 1199 ms. Trace:
I20260709 12:19:19.277050 13258 rpcz_store.cc:276] 0709 12:19:18.019830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.149464 (+129634us) service_pool.cc:224] Handling call
0709 12:19:19.212974 (+1063510us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b540056a9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:19.213176 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110932864}
I20260709 12:19:19.009857 29340 consensus_queue.cc:237] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.33, Last appended by leader: 33, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:19.215345 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:19.281946 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.03547e+06us
I20260709 12:19:19.219549 29308 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:19:19.221444 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6514) took 4066 ms. Trace:
I20260709 12:19:19.226873 29258 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:19:19.227614 29254 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:19:19.229688 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.293080 12563 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:19.238027 29212 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:19.274361 29321 raft_consensus.cc:740] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:19.302139 12563 raft_consensus.cc:1240] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.34-48.34]
I20260709 12:19:19.240872 29189 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:19.246414 13255 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.25-41.25]
W20260709 12:19:19.238137 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 83625.6us
I20260709 12:19:19.311093 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:19.257989 12610 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 27 similar messages]
W20260709 12:19:19.312666 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54696.8us
I20260709 12:19:19.287599 29308 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:19.185257 29028 consensus_queue.cc:1059] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 52.37, Next index: 38, Last known committed idx: 37, Time since last communication: 0.001s
W20260709 12:19:19.187382 29362 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:19:19.255647 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:19.317456 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:19.257464 29294 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:19.287683 12569 rpcz_store.cc:276] 0709 12:19:15.155116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:15.365175 (+210059us) service_pool.cc:224] Handling call
0709 12:19:19.106542 (+3741367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:19.106920 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:19:19.106960 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:19.107017 (+    57us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:19.107059 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.107136 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:19.221109 (+113973us) spinlock_profiling.cc:243] Waited 3.72 s on lock 0x7b54001a2cd0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:19.221324 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36793,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":26160,"spinlock_wait_cycles":8564512000}
W20260709 12:19:19.289068 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:19.304646 29212 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.321915 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:19.260283 29355 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:19.181188 29144 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.268091 29218 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:19.146754 29319 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 52.30, Last appended by leader: 30, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:19.240255 29374 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:19.273998 29364 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.305944 29321 consensus_queue.cc:260] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 47.31, Last appended by leader: 31, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.263435 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:19.350742 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 87361.2us
I20260709 12:19:19.290697 29258 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:19.292543 29254 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:19.268889 29231 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.287539 12565 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
I20260709 12:19:19.325985 29212 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:19.323307 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:19.268159 29182 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.336694 29283 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:19:19.309137 28969 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:19.319574 29224 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:19.320573 29163 raft_consensus.cc:697] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:19.358834 12565 raft_consensus.cc:1240] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
W20260709 12:19:19.323603 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68496.9us
I20260709 12:19:19.357767 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:19.320684 29032 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:19.393422 12609 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:19.335067 29361 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:19.332002 29266 raft_consensus.cc:3055] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:19:19.363282 29319 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:19.362797 29376 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.412887 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:19.335309 29345 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:19.376938 29308 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 42.26, Last appended by leader: 26, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.357100 29028 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:19.381173 29283 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:19.332943 29294 raft_consensus.cc:697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:19.326109 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:19.402251 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 44594.7us
I20260709 12:19:19.402408 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:19.402698 12643 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 48. Current term is 49. Ops: [48.28-48.28]
I20260709 12:19:19.404155 13259 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: [50.32-50.32]
I20260709 12:19:19.398373 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:19.411002 12637 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:19.414283 12646 raft_consensus.cc:1240] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:19.395231 12639 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 30. Preceding OpId from leader: term: 49 index: 31. (index mismatch)
W20260709 12:19:19.421823 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65576.4us
I20260709 12:19:19.392624 29175 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:19.425997 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:19.407744 29361 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:19.431790 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105788us
W20260709 12:19:19.437829 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:19.439818 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.404968 29124 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:19.454486 29124 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.454972 29124 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:19.407464 29266 raft_consensus.cc:740] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:19.460716 29266 consensus_queue.cc:260] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 53.38, Last appended by leader: 38, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.461892 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:19.462218 29266 raft_consensus.cc:3060] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:19.462625 13261 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:19.418241 29255 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:19.466122 29255 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.466616 29255 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:19.469913 29117 raft_consensus.cc:3055] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:19.470593 29117 raft_consensus.cc:740] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:19.446473 29377 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.028s
I20260709 12:19:19.474892 29254 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.476446 29254 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:19.478346 29182 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:19.484472 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.421679 29136 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.028s
I20260709 12:19:19.431061 29376 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:19.450281 29383 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:19:19.451417 29378 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:19:19.451089 29294 consensus_queue.cc:237] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 51.30, Last appended by leader: 30, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:19.442210 29175 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.442420 29032 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:19.471676 29117 consensus_queue.cc:260] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 50.34, Last appended by leader: 34, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:19.448911 29379 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:19:19.495836 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:19.632984 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 137225us
I20260709 12:19:19.501148 13261 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:19.634428 12645 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 39 index: 24. Preceding OpId from leader: term: 42 index: 25. (index mismatch)
I20260709 12:19:19.420239 29258 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 53.30, Last appended by leader: 30, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:19.436668 29306 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.872s	user 0.012s	sys 0.007s
W20260709 12:19:19.554186 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.436851 29134 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election won for term 56
W20260709 12:19:19.605546 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.444516 29361 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:19.633949 13218 leader_election.cc:336] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:19.641177 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:19.341711 29356 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 42.29, Last appended by leader: 29, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.505942 28976 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:19.647119 13218 leader_election.cc:336] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 49 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:19.648272 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:19.573015 12610 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:19.428097 29028 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:19.618716 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 220439us
I20260709 12:19:19.653208 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:19.424876 29283 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:19.511261 29136 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:19:19.665436 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:19.630170 29218 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:19.667393 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.711643 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:19.720263 29028 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 52.29, Last appended by leader: 29, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.724169 29376 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:19.724558 29144 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:19.724956 29376 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:19.726332 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:19.728327 29231 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:19.642762 29134 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:19.729058 29134 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
W20260709 12:19:19.734427 29381 log.cc:927] Time spent T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:19:19.734933 28969 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:19.736033 29175 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:19.736210 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6049) took 1598 ms. Trace:
I20260709 12:19:19.736665 29175 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:19.736552 12642 rpcz_store.cc:276] 0709 12:19:18.137599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.137735 (+   136us) service_pool.cc:224] Handling call
0709 12:19:18.174781 (+ 37046us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:18.175274 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:18.175314 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:18.182752 (+  7438us) log.cc:844] Serialized 26 byte log entry
0709 12:19:18.891942 (+709190us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b18002c6800. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:19:19.260931 (+368989us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:19.260991 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.261059 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:19.735875 (+474816us) raft_consensus.cc:1685] finished
0709 12:19:19.735898 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:19.736138 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48816,"raft.queue_time_us":3932,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":19750,"spinlock_wait_cycles":8807168,"thread_start_us":368737,"threads_started":1,"wal-append.queue_time_us":1134240}
I20260709 12:19:19.650507 29363 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:19.483733 29163 consensus_queue.cc:237] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 52.37, Last appended by leader: 37, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:19.680032 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 247957us
I20260709 12:19:19.716835 29300 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:19.749321 29364 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:19.603430 29032 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:19.749007 29355 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:19.656922 29258 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:19.646018 29306 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:19.766052 29387 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
W20260709 12:19:19.771905 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.669618 28976 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:19.794487 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:19.786430 29386 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:19:19.806772 12568 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:19.742341 29218 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:19.779647 29032 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:19:19.818074 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57738.7us
I20260709 12:19:19.791968 29300 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:19.807883 29391 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Stepping down as leader of term 46
W20260709 12:19:19.829494 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:19.844226 12610 leader_election.cc:336] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:19.847641 13218 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:19.853806 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:19.859771 12610 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:19.872455 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:19.867030 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 198658us
I20260709 12:19:19.888346 12609 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:19.834142 29391 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:19.894836 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:19.862877 29327 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:19.889613 29386 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
W20260709 12:19:19.919351 29390 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:19:19.927213 12563 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: [52.30-52.30]
I20260709 12:19:19.938292 12563 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: [53.30-53.30]
I20260709 12:19:19.925498 29397 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:19.933861 29277 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:19.964943 29397 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:19.972492 29308 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:19.998762 13262 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.25-41.25]
W20260709 12:19:19.972923 29394 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
W20260709 12:19:19.974961 29124 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.520s	user 0.012s	sys 0.000s
I20260709 12:19:19.979269 12564 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
I20260709 12:19:19.979795 29321 raft_consensus.cc:3060] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:19.982165 28976 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:19.924469 29327 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:19.983767 29218 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:19.985888 29120 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20260709 12:19:19.912734 29300 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 52.31, Last appended by leader: 31, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:20.012966 29266 consensus_meta.cc:287] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.006s
I20260709 12:19:19.962729 29391 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.26, Last appended by leader: 26, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.013392 12569 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 51. Current term is 52. Ops: []
W20260709 12:19:20.003656 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.036453 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 141737us
W20260709 12:19:20.010653 29212 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.640s	user 0.013s	sys 0.000s
I20260709 12:19:20.011910 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:20.025076 29264 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.233s	user 0.007s	sys 0.012s
I20260709 12:19:20.021404 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:20.038305 12563 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:19:20.016206 29255 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.005s
I20260709 12:19:20.023609 12571 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 48 index: 32. (index mismatch)
I20260709 12:19:20.039954 29255 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.042057 29255 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:20.043888 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.047680 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:20.012813 29124 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.017330 29361 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.030362 29391 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:20.028008 29355 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.058670 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.062060 12571 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:20.066550 12638 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 42. Current term is 43. Ops: []
W20260709 12:19:20.067121 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.074707 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:20.075696 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.076277 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.076418 12564 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
W20260709 12:19:20.057021 29254 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.580s	user 0.015s	sys 0.001s
I20260709 12:19:20.077322 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.107887 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.109228 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.081013 29124 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.110158 29124 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.110702 29124 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.063742 29277 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:20.083736 29107 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.117334 13262 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 53. Current term is 54. Ops: [53.31-53.31]
I20260709 12:19:20.031438 28780 consensus_queue.cc:1059] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.035s
I20260709 12:19:20.092665 12563 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:19:20.100832 29325 consensus_meta.cc:287] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.302s	user 0.013s	sys 0.005s
W20260709 12:19:20.111789 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 154148us
W20260709 12:19:20.112025 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.113920 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:20.116459 13261 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:20.084412 29356 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:20.087325 12565 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: [42.26-42.26]
I20260709 12:19:20.046528 29212 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.118319 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:20.119671 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.135159 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:20.138536 12641 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.30-48.30]
I20260709 12:19:20.139724 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.141299 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.142697 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.128528 12564 raft_consensus.cc:1240] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
W20260709 12:19:20.131878 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.132526 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:20.133175 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:20.134214 12563 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 25. Preceding OpId from leader: term: 49 index: 26. (index mismatch)
I20260709 12:19:20.146878 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.146215 12563 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:19:20.164800 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.165547 12641 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:20.148821 12564 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 55 index: 27. Preceding OpId from leader: term: 57 index: 28. (index mismatch)
I20260709 12:19:20.168521 12565 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 45 index: 24. (index mismatch)
W20260709 12:19:20.148561 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:20.177425 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.179073 12642 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 49 because replica has last-logged OpId of term: 46 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
I20260709 12:19:20.181471 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.183251 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.184360 12639 raft_consensus.cc:2410] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 47 because replica has last-logged OpId of term: 46 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 28.
I20260709 12:19:20.131148 28780 consensus_queue.cc:1243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 42.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:19:20.188966 12565 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.30-45.30]
I20260709 12:19:20.200793 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.198199 12639 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 26. Preceding OpId from leader: term: 57 index: 28. (index mismatch)
I20260709 12:19:20.179988 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.203147 12645 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:20.204313 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.129655 29356 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.125182 29277 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:20.205415 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:20.226899 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48902.3us
I20260709 12:19:20.167503 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.162320 29271 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.146464 29189 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.162186 29355 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.221781 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.166065 29175 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.168084 29144 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:20.225236 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.287696 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 51 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.288555 12641 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:20.217074 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:20.239538 29308 raft_consensus.cc:1064] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:20.211469 29136 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:20.134729 29325 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.239557 29397 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.242643 29117 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:20.243595 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.324389 12643 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:20.244037 29257 raft_consensus.cc:3060] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:20.226476 29286 raft_consensus.cc:3060] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:20.245179 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.417121 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:20.419854 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:20.248327 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.421978 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.422196 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 39 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:20.422915 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.423324 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.249576 29400 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
W20260709 12:19:20.254544 29401 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
W20260709 12:19:20.255429 29403 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:19:20.258062 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6564) took 1193 ms. Trace:
I20260709 12:19:20.206579 29173 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:20.125752 29107 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.261539 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.264983 12536 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:20.444283 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 216346us
I20260709 12:19:20.265400 12564 raft_consensus.cc:1275] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 33. Preceding OpId from leader: term: 49 index: 34. (index mismatch)
I20260709 12:19:20.266441 12563 raft_consensus.cc:1275] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 52 index: 30. Preceding OpId from leader: term: 56 index: 32. (index mismatch)
I20260709 12:19:20.447252 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:20.206197 29405 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.262437 29345 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.214401 29356 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:20.212563 28780 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:20.286293 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.421744 28689 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.427175 28776 raft_consensus.cc:3055] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:20.300798 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.306828 12567 raft_consensus.cc:1240] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 53. Current term is 54. Ops: [53.38-53.38]
W20260709 12:19:20.276122 29396 log.cc:927] Time spent T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
W20260709 12:19:20.275981 29395 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.277s	user 0.001s	sys 0.001s
W20260709 12:19:20.228751 29317 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.177s	user 0.016s	sys 0.035s
W20260709 12:19:20.310050 29276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.266s	user 0.013s	sys 0.004s
I20260709 12:19:20.315948 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:20.373865 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.374887 29277 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.276780 29327 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.380131 29182 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:20.383247 29212 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:20.384627 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:20.489877 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 627275us
I20260709 12:19:20.314709 29136 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.305895 29364 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:20.492725 29364 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.494132 29136 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.494877 29364 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.286495 29189 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.498133 29189 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.387156 29124 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.388041 29308 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.28, Last appended by leader: 27, Current term: 43, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.498745 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50696 (request call id 2096) took 1014 ms. Trace:
I20260709 12:19:20.499238 13255 rpcz_store.cc:276] 0709 12:19:19.484884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.485051 (+   167us) service_pool.cc:224] Handling call
0709 12:19:19.485484 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:19.485963 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:19.486001 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:19.606049 (+120048us) log.cc:844] Serialized 26 byte log entry
0709 12:19:19.787016 (+180967us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:19.787068 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.787143 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:20.498479 (+711336us) raft_consensus.cc:1685] finished
0709 12:19:20.498504 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.498677 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17454,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":4366,"thread_start_us":153073,"threads_started":1,"wal-append.queue_time_us":301181}
I20260709 12:19:20.302321 28969 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:20.503280 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2375) took 1089 ms. Trace:
I20260709 12:19:20.286701 29287 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.504197 29287 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.479662 29276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.445s	user 0.014s	sys 0.004s
I20260709 12:19:20.503342 28969 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.508661 28969 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:20.300266 29175 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.395610 29134 consensus_meta.cc:287] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.666s	user 0.010s	sys 0.003s
I20260709 12:19:20.512676 29134 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.514104 29175 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:20.394383 29408 log.cc:927] Time spent T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:19:20.514986 29134 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.275777 29271 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.508541 29287 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.397053 29361 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.295961 29374 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:20.517055 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6569) took 1450 ms. Trace:
I20260709 12:19:20.293745 29144 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:20.398851 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.399442 29254 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260709 12:19:20.460652 29345 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.519021 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6185) took 2151 ms. Trace:
I20260709 12:19:20.519558 29144 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:20.402148 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.428644 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.435645 12569 rpcz_store.cc:276] 0709 12:19:19.064891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.326085 (+261194us) service_pool.cc:224] Handling call
0709 12:19:19.352868 (+ 26783us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.025911 (+673043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.257743 (+231832us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b54003e0654. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:20.257986 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1518259456}
I20260709 12:19:20.410859 12609 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:20.503814 12638 rpcz_store.cc:276] 0709 12:19:19.414326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.414500 (+   174us) service_pool.cc:224] Handling call
0709 12:19:19.414788 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.067155 (+652367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.502993 (+435838us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:20.503208 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1498250112}
I20260709 12:19:20.519433 12568 rpcz_store.cc:276] 0709 12:19:18.367767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.368248 (+   481us) service_pool.cc:224] Handling call
0709 12:19:18.368573 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:19.807458 (+1438885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.518770 (+711312us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400044f54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:20.518960 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3306747264}
I20260709 12:19:20.517618 12566 rpcz_store.cc:276] 0709 12:19:19.066289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.717661 (+651372us) service_pool.cc:224] Handling call
0709 12:19:19.718061 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:19.718680 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:19.718715 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:19.743103 (+ 24388us) log.cc:844] Serialized 26 byte log entry
0709 12:19:19.946353 (+203250us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:19.946397 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:19.946464 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:20.516752 (+570288us) raft_consensus.cc:1685] finished
0709 12:19:20.516779 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.516961 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":124,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":149465,"threads_started":1,"wal-append.queue_time_us":238293}
I20260709 12:19:20.524953 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:20.527874 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:20.528405 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:20.529230 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.537478 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2101) took 1812 ms. Trace:
W20260709 12:19:20.538780 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 110211us
I20260709 12:19:20.459285 28776 raft_consensus.cc:740] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:20.376879 29230 consensus_queue.cc:260] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.25, Last appended by leader: 25, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.548528 13255 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 43 index: 26. (index mismatch)
I20260709 12:19:20.449159 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.550725 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.551537 12563 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:20.453517 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:20.553508 12564 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:20.445708 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:19:20.555835 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 110230us
I20260709 12:19:20.427290 29335 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:20.451260 29259 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.017s
I20260709 12:19:20.458249 29405 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.439558 29173 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.418707 29183 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.038s
I20260709 12:19:20.415827 29231 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 49.29, Next index: 30, Last known committed idx: 27, Time since last communication: 0.037s
W20260709 12:19:20.476891 29317 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.461s	user 0.016s	sys 0.036s
W20260709 12:19:20.492062 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:20.498385 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6559) took 1435 ms. Trace:
I20260709 12:19:20.294687 29355 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.568982 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 76988.8us
I20260709 12:19:20.517663 29374 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.519446 12646 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:20.437183 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:20.577723 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:20.578043 12570 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:20.438133 12641 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 54.
I20260709 12:19:20.542922 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.475167 28689 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.546195 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 135454us
I20260709 12:19:20.537986 13259 rpcz_store.cc:276] 0709 12:19:18.724970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.725739 (+   769us) service_pool.cc:224] Handling call
0709 12:19:18.739800 (+ 14061us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:19.430853 (+691053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.537134 (+1106281us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b54004eeb54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:20.537403 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527091072}
I20260709 12:19:20.548130 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:20.433889 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:20.292069 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 75137.7us
W20260709 12:19:20.515015 29411 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.050s	user 0.002s	sys 0.000s
I20260709 12:19:20.558180 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:20.559111 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2102) took 1831 ms. Trace:
I20260709 12:19:20.569680 12567 rpcz_store.cc:276] 0709 12:19:19.063158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.284395 (+221237us) service_pool.cc:224] Handling call
0709 12:19:19.284961 (+   566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:20.464901 (+1179940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.498062 (+ 33161us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001a22d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:20.498311 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2338529536}
I20260709 12:19:20.601874 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:20.576512 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 50913us
I20260709 12:19:20.466935 28780 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.545842 12536 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:20.579986 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.597330 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54523.4us
W20260709 12:19:20.601367 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 167494us
W20260709 12:19:20.617986 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72198.8us
I20260709 12:19:20.550168 29212 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:20.608614 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.627599 12637 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
W20260709 12:19:20.629390 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.543753 29276 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.605486 13257 rpcz_store.cc:276] 0709 12:19:18.727349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.727552 (+   203us) service_pool.cc:224] Handling call
0709 12:19:19.220179 (+492627us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:19.220485 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 27
0709 12:19:19.320131 (+ 99646us) log.cc:844] Serialized 14 byte log entry
0709 12:19:20.106831 (+786700us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b18002d5440. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a
0709 12:19:20.118192 (+ 11361us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:20.118261 (+    69us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:20.118330 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:20.118424 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:20.558850 (+440426us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b540056a9d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:20.559022 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11015,"prepare.queue_time_us":183124,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":11585,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":9123,"spinlock_wait_cycles":1173104384,"thread_start_us":188652,"threads_started":2,"wal-append.queue_time_us":196242}
W20260709 12:19:20.582306 29319 consensus_meta.cc:287] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.163s	user 0.017s	sys 0.000s
I20260709 12:19:20.637749 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:20.618737 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:20.625095 12460 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:20.567036 29361 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:20.570161 29173 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:20.563251 29175 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:20.657853 29175 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:20.658349 29175 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:20.662038 12568 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 39.
I20260709 12:19:20.579754 29325 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.560760 29136 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260709 12:19:20.639467 12564 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:19:20.677338 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.643788 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:20.644783 29032 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.807s	user 0.009s	sys 0.005s
I20260709 12:19:20.651391 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.651840 29306 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:20.652034 29374 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.654126 28780 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.654649 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:20.634001 29231 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:20.655017 29321 consensus_meta.cc:287] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.639s	user 0.010s	sys 0.003s
I20260709 12:19:20.655961 28721 consensus_queue.cc:1059] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:20.656495 29258 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.869s	user 0.011s	sys 0.004s
I20260709 12:19:20.606388 29218 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:20.680187 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55168.9us
I20260709 12:19:20.664206 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:20.703593 13257 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:20.642961 29391 consensus_meta.cc:287] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.002s
I20260709 12:19:20.634464 29294 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:20.627112 29189 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 50.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.069s
I20260709 12:19:20.627506 29397 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 45.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.069s
W20260709 12:19:20.697389 29283 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.983s	user 0.005s	sys 0.010s
I20260709 12:19:20.722710 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.742107 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 38.
I20260709 12:19:20.692870 29212 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader election won for term 56
W20260709 12:19:20.710903 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67251.6us
I20260709 12:19:20.690972 29120 consensus_queue.cc:1059] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.020s
I20260709 12:19:20.698038 29136 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.008s
I20260709 12:19:20.751060 29120 raft_consensus.cc:3055] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:20.753243 29120 raft_consensus.cc:740] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:20.754238 29120 consensus_queue.cc:260] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.645510 29276 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.737474 12639 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 47 index: 26. (term mismatch)
I20260709 12:19:20.772221 12639 pending_rounds.cc:85] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 25
I20260709 12:19:20.772578 12639 pending_rounds.cc:107] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 46.26
I20260709 12:19:20.772832 12639 raft_consensus.cc:2889] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:20.700846 29374 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.662032 29376 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.045s
I20260709 12:19:20.748869 12570 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:20.750093 29391 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:20.750316 29283 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.751055 29271 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:20.752110 29345 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.693167 29032 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.725116 28689 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.739161 13257 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 49 because replica has last-logged OpId of term: 48 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 29.
I20260709 12:19:20.798066 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.807878 12644 raft_consensus.cc:2468] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
W20260709 12:19:20.808961 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:20.854647 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 56656.5us
I20260709 12:19:20.638397 28776 consensus_queue.cc:260] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 44.24, Last appended by leader: 24, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.845857 13259 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 43 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
I20260709 12:19:20.856240 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.736268 29294 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.726594 29325 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:19:20.703970 29306 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.102s
I20260709 12:19:20.853216 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.752312 29173 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.879253 12569 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:20.707706 29231 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.882354 29134 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:20.884199 29231 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.884964 29231 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:20.794500 29212 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:20.885763 29258 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:20.886572 29258 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.886637 29212 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 54.24, Last appended by leader: 24, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.887130 29258 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.889267 29258 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.869128 28776 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:20.892876 29405 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.869717 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.894740 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.895924 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.897004 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.865161 29294 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:20.809753 29418 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 25, Time since last communication: 0.064s
I20260709 12:19:20.874599 29325 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.903208 29355 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.903548 29325 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.905135 29283 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.910940 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.911760 12646 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 52
I20260709 12:19:20.915234 29032 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.918919 29173 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.854243 29107 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:20.854444 29345 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.920169 29345 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.924129 29345 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.924705 29345 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:20.930284 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.931011 12639 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54. Re-sending same reply.
I20260709 12:19:20.931993 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:20.932145 28969 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.918808 29230 raft_consensus.cc:3060] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:19:20.755307 29356 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:20.936954 29356 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:20.937891 12645 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:20.939932 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.940585 12645 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:20.941762 29325 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:20.941975 12642 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:20.944010 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 41. Current term is 42. Ops: []
I20260709 12:19:20.795418 29391 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:20.957825 12569 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:20.945416 12642 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 54 index: 35. Preceding OpId from leader: term: 58 index: 37. (index mismatch)
I20260709 12:19:20.941413 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:20.885412 29231 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:20.813941 29361 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:20.851585 29120 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:20.976876 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.041s	sys 0.044s
I20260709 12:19:20.858595 29183 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:20.978832 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.041s	sys 0.044s
I20260709 12:19:20.867238 28689 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:20.977407 12646 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:20.794922 29175 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.983108 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.973912 29231 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:20.946879 29266 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:20.989880 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.814703 29374 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:20.980326 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 530915us
I20260709 12:19:20.991886 12642 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 42 index: 26. Preceding OpId from leader: term: 43 index: 26. (term mismatch)
I20260709 12:19:20.992332 12642 pending_rounds.cc:85] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 25
I20260709 12:19:20.992594 12642 pending_rounds.cc:107] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 42.26
I20260709 12:19:20.992867 12642 raft_consensus.cc:2889] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:20.995245 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:21.000707 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:20.966763 29405 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:21.013762 12646 raft_consensus.cc:1240] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 41. Current term is 42. Ops: [41.25-41.25]
I20260709 12:19:20.925185 29345 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:21.011513 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 386673us
I20260709 12:19:21.027796 12639 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:21.029232 13217 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:20.983939 29183 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:20.987792 29144 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.057682 29410 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:21.056129 12646 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 51 index: 28. Preceding OpId from leader: term: 52 index: 29. (index mismatch)
I20260709 12:19:21.009723 29266 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:21.054512 12640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.020s	sys 0.031s
W20260709 12:19:21.140352 12640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.021s	sys 0.031s
W20260709 12:19:21.037853 29257 consensus_meta.cc:287] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.657s	user 0.012s	sys 0.000s
I20260709 12:19:21.033080 28689 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.057963 29355 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:21.077201 29117 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.697s	user 0.009s	sys 0.005s
I20260709 12:19:21.082995 29374 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.058988 12642 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:21.151441 12642 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:21.029414 29405 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.074982 29423 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:21.102636 29276 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.111441 29335 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:21.113384 29308 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:19:21.114790 29345 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.144956 29283 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.027563 29391 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 30, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.120110 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:19:21.188730 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68754.9us
I20260709 12:19:21.190596 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:21.124107 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 267916us
W20260709 12:19:21.098973 29064 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.007s	sys 0.003s
I20260709 12:19:21.094259 28969 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:21.152222 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.155867 12642 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 48.27, Last appended by leader: 27, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.196451 12642 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:21.059814 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 405187us
I20260709 12:19:21.077693 29361 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.27, Last appended by leader: 27, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.148386 29292 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.146603 29254 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:21.160693 29308 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:21.160482 29335 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:21.150754 28689 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.144177 29410 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.179257 29175 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:21.193964 29064 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.007s	sys 0.004s
I20260709 12:19:21.137842 29182 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.040s
I20260709 12:19:21.140393 29294 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.182487 29276 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:21.153981 29355 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.192091 12461 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:21.181176 29405 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.225711 12640 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 48. Current term is 49. Ops: [48.28-48.28]
I20260709 12:19:21.211006 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:21.223913 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:21.212713 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 183569us
I20260709 12:19:21.195564 29163 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
W20260709 12:19:21.221809 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.001s	sys 0.025s
W20260709 12:19:21.200752 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 393832us
W20260709 12:19:21.243516 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.001s	sys 0.029s
W20260709 12:19:21.240803 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48835.9us
I20260709 12:19:21.242462 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 56 index: 36 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:21.250491 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.244326 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 577169us
I20260709 12:19:21.260119 12642 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 48 index: 27. Preceding OpId from leader: term: 49 index: 27. (term mismatch)
I20260709 12:19:21.260648 12642 pending_rounds.cc:85] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 26
I20260709 12:19:21.260962 12642 pending_rounds.cc:107] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 48.27
I20260709 12:19:21.261253 12642 raft_consensus.cc:2889] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:21.265017 29175 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:21.267321 29405 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:19:21.271867 28689 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.272919 28689 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:19:21.274191 29212 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.001s
I20260709 12:19:21.262167 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:21.317834 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55785.5us
I20260709 12:19:21.273481 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:21.318750 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.265492 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:21.320201 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54782.4us
I20260709 12:19:21.323571 13255 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 56 index: 36. Preceding OpId from leader: term: 58 index: 37. (index mismatch)
I20260709 12:19:21.324572 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:21.295835 29355 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.298534 29294 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:21.209849 29292 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.204053 29374 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:21.335595 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 39 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.194536 29283 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:21.370157 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 96756.5us
I20260709 12:19:21.216169 29254 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:21.305797 12641 raft_consensus.cc:1275] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 31. Preceding OpId from leader: term: 48 index: 32. (index mismatch)
W20260709 12:19:21.305284 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2369) took 1918 ms (client timeout 1901 ms). Trace:
W20260709 12:19:21.381304 12644 rpcz_store.cc:269] 0709 12:19:19.386479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:19.386642 (+   163us) service_pool.cc:224] Handling call
0709 12:19:21.305065 (+1918423us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001f77d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:21.305204 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2970535296}
I20260709 12:19:21.301848 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:19:21.269439 29405 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.007s
I20260709 12:19:21.324577 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.240409 29277 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.328186 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.254329 29163 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:21.260823 29258 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:19:21.335218 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.388327 12643 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:21.388870 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:21.350313 12568 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 39: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:21.390995 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.392128 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:21.342010 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:21.327123 29264 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:21.352881 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:21.398303 12568 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 51: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:21.323971 29117 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.356406 12572 raft_consensus.cc:1275] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 41 index: 25. Preceding OpId from leader: term: 42 index: 25. (term mismatch)
I20260709 12:19:21.402822 12572 pending_rounds.cc:85] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 24
I20260709 12:19:21.403123 12572 pending_rounds.cc:107] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 41.25
I20260709 12:19:21.227734 29276 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.358049 29335 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.315543 29308 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 53.31, Last appended by leader: 31, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.360024 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.360476 29410 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.422190 12571 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 54: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:21.324721 29429 consensus_queue.cc:1048] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.037s
I20260709 12:19:21.423982 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.313900 29423 consensus_queue.cc:1048] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.048s
I20260709 12:19:21.326974 29417 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.036s
I20260709 12:19:21.363492 12565 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: [49.26-49.26]
I20260709 12:19:21.348265 29436 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.021s
W20260709 12:19:21.367527 29218 consensus_meta.cc:287] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.661s	user 0.009s	sys 0.002s
I20260709 12:19:21.367996 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:21.376986 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:21.438860 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:21.439775 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:21.441082 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:21.344570 28776 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:21.378463 12640 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56. Ops: [55.29-55.29]
I20260709 12:19:21.337024 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 54 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:21.395555 12563 raft_consensus.cc:1275] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 51 index: 30. Preceding OpId from leader: term: 52 index: 31. (index mismatch)
I20260709 12:19:21.398587 13259 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:21.354699 29231 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.397619 13258 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:21.400440 13253 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 29. Preceding OpId from leader: term: 54 index: 30. (index mismatch)
I20260709 12:19:21.403508 12572 raft_consensus.cc:2889] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:21.404960 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.408789 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 52 index: 37 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.409739 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.411314 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:21.457360 12644 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:21.392524 29322 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:21.392880 29300 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:19:21.392807 29258 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:21.356282 29144 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.397830 28721 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:21.435328 29308 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:21.414494 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:21.482033 12568 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
I20260709 12:19:21.327540 29271 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.074s
I20260709 12:19:21.392966 29163 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:21.301673 29432 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:21.393350 29374 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:21.380017 29296 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.034s
W20260709 12:19:21.422163 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122561us
I20260709 12:19:21.429488 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:21.433164 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.489472 13256 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
W20260709 12:19:21.433246 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113361us
I20260709 12:19:21.401360 29264 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:21.330775 29428 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.093s
I20260709 12:19:21.344163 12570 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 28. Preceding OpId from leader: term: 43 index: 29. (index mismatch)
W20260709 12:19:21.435927 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 107746us
I20260709 12:19:21.499979 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:21.444386 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.445091 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:21.449540 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:21.461251 12566 raft_consensus.cc:2393] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 56: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:21.401737 29277 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.443735 29335 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:21.465178 29322 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:21.484035 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:21.444922 29218 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:21.493670 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 242155us
I20260709 12:19:21.413352 29117 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.470841 29258 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:19:21.522790 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:21.523608 12642 raft_consensus.cc:2376] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48. Re-sending same reply.
W20260709 12:19:21.511963 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.524165 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:21.505457 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:21.471203 29300 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:21.505597 29429 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.485718 28689 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.486811 29432 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:21.526038 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.522471 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.531747 13261 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:21.532878 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:21.500900 29264 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:21.529863 12639 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 58.
I20260709 12:19:21.533753 13261 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:21.532998 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:21.535722 13253 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
W20260709 12:19:21.530711 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.537212 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:21.544168 12610 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:21.545478 12610 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.556977 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:21.564683 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57881.9us
I20260709 12:19:21.511366 29340 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:21.569852 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:21.569685 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:21.572907 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.574715 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.509078 28721 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:21.640995 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66332.1us
I20260709 12:19:21.633985 13259 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 43 index: 30. (index mismatch)
I20260709 12:19:21.637548 12645 raft_consensus.cc:1240] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 48. Current term is 49. Ops: [48.30-48.30]
W20260709 12:19:21.641872 12610 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.533252 29423 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:21.472062 29254 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 45.30, Last appended by leader: 30, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.640000 29218 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:21.572147 28776 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:21.645401 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:21.688836 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:21.689913 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.568897 29405 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:21.581413 29175 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:21.645304 29425 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:21.674256 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 168916us
I20260709 12:19:21.534668 29432 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:21.690850 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:21.711489 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:21.688856 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 188976us
I20260709 12:19:21.532965 29257 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 45.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.133s
I20260709 12:19:21.707406 13253 raft_consensus.cc:2468] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:21.713964 12460 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:21.673130 29437 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:21.662539 29423 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:21.547067 29429 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.546377 28689 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.735483 12461 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:21.638885 29451 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.145s
I20260709 12:19:21.639662 29340 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.790825 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:21.697902 29405 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:21.800984 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65599us
I20260709 12:19:21.708200 29218 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:21.759407 29430 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:21.805302 12461 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:21.806190 12460 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:21.808260 29276 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.716934 28969 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:19:21.808828 29450 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.237s	user 0.000s	sys 0.000s
I20260709 12:19:21.810551 29117 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.810972 29340 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.813190 29391 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:21.815726 29283 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.788898 29437 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.756042 29425 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.884335 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6219) took 1485 ms. Trace:
I20260709 12:19:21.787858 29423 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:21.823110 29438 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.358s	user 0.000s	sys 0.001s
I20260709 12:19:21.832734 29405 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:21.804337 28721 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 50.31, Last appended by leader: 31, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.651201 29374 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 51.22, Last appended by leader: 22, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.725924 29322 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 48.28, Last appended by leader: 28, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.813426 13258 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:21.766206 29355 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:21.758073 29032 consensus_queue.cc:1059] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 53.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.067s
I20260709 12:19:21.765442 28976 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:21.911793 28976 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:21.912787 28976 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 44.24, Last appended by leader: 24, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.913152 29258 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.915328 29258 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.758095 29455 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:21.709734 29300 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.916301 29455 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:21.917053 29436 consensus_queue.cc:1175] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0a29b5d9f9d44298b02ffcd460811eef" responder_term: 41 status { last_received { term: 41 index: 24 } last_committed_idx: 24 last_received_current_leader { term: 41 index: 24 } }
I20260709 12:19:21.850247 29319 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:19:21.855091 29218 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:21.918114 29319 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:21.919169 29319 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 48.25, Last appended by leader: 25, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.858546 28689 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.848269 29464 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:21.859555 29107 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:21.848300 29458 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:21.889912 29231 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.916703 29300 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 41 FOLLOWER]: Advancing to term 42
W20260709 12:19:21.808828 29445 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 12:19:21.930955 29107 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:21.882004 29459 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:21.934366 29458 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.860165 29430 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.927667 29464 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:21.942720 29107 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:21.962051 29429 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.962898 29277 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.967274 29458 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:21.946378 29464 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:21.968886 29425 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:21.975673 29264 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:21.949587 29459 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:21.990200 29457 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:19:21.990852 29423 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:21.997161 29423 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:21.999225 29430 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.002872 29455 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:22.005563 29437 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:22.014643 29405 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.018119 29405 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.010161 29459 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.017807 29432 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.024921 29432 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.030723 29464 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:21.851338 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.074532 12568 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 43.
I20260709 12:19:21.884979 12567 rpcz_store.cc:276] 0709 12:19:20.398908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.605897 (+206989us) service_pool.cc:224] Handling call
0709 12:19:20.606230 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:20.634354 (+ 28124us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:20.807992 (+173638us) log.cc:844] Serialized 14 byte log entry
0709 12:19:21.226880 (+418888us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:21.226942 (+    62us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:21.226986 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.883953 (+656967us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b5400103954. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:21.884026 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:21.884240 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13724,"prepare.queue_time_us":421234,"prepare.run_cpu_time_us":179,"prepare.run_wall_time_us":180,"raft.queue_time_us":436,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":9179,"spinlock_wait_cycles":64071552,"thread_start_us":570712,"threads_started":2,"wal-append.queue_time_us":560504}
W20260709 12:19:22.081103 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:21.857062 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 50977.7us
I20260709 12:19:22.087625 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.088565 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:21.853091 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47882.6us
I20260709 12:19:22.089463 12567 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:22.089947 12566 raft_consensus.cc:2468] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:22.090725 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:22.092070 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:21.898705 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 207843us
I20260709 12:19:22.081156 12638 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:22.096382 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.093412 12565 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
W20260709 12:19:22.163311 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.000s	sys 0.003s
I20260709 12:19:22.098474 12564 pending_rounds.cc:85] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 28
I20260709 12:19:22.144806 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:22.179646 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.001s	sys 0.003s
I20260709 12:19:22.176133 12564 pending_rounds.cc:107] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 46.29
I20260709 12:19:22.182435 12564 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:22.137569 29183 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:22.123556 29335 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.604s	user 0.014s	sys 0.000s
I20260709 12:19:22.155581 29266 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.097616 29064 raft_consensus.cc:3055] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:22.191684 29064 raft_consensus.cc:740] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:22.191845 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:22.105461 29458 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:22.116544 29292 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:22.194252 29458 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:22.194272 29064 consensus_queue.cc:260] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:22.196679 29064 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:22.100255 29455 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:22.198876 29455 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.203073 29455 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.202842 29136 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:22.204701 29458 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:22.206241 12610 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.206053 29136 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.219110 12563 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Advancing to term 43
W20260709 12:19:22.224673 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:22.177265 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65844.6us
I20260709 12:19:22.237905 29423 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:22.238705 29423 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.239152 29423 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:22.243173 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.157455 29405 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:22.251298 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.246580 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.259385 12568 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:22.210779 29136 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.249641 29405 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:22.037359 29464 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.286911 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.283687 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:22.316502 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73529.2us
W20260709 12:19:22.318063 12536 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.318661 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.320531 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2381) took 1109 ms. Trace:
W20260709 12:19:22.321913 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.324430 29455 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:22.325385 29455 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.329909 29136 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:22.331385 12568 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:22.334038 29455 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.340953 29464 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 48.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.003s
I20260709 12:19:22.344630 29464 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:22.339406 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3110) took 3508 ms. Trace:
I20260709 12:19:22.289664 29458 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:22.350708 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:22.350936 29264 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.351868 12566 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:19:22.351778 12572 rpcz_store.cc:276] 0709 12:19:18.831367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:18.936899 (+105532us) service_pool.cc:224] Handling call
0709 12:19:18.937424 (+   525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:21.455003 (+2517579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.338630 (+883627us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54002cf1d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:22.339140 (+   510us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5561841408}
W20260709 12:19:22.330176 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:22.355203 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 65043.3us
W20260709 12:19:22.358857 12610 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.359371 29433 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:22.359648 12610 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:22.360180 29433 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:22.363319 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2377) took 1756 ms. Trace:
I20260709 12:19:22.363891 12641 rpcz_store.cc:276] 0709 12:19:20.606943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:20.607702 (+   759us) service_pool.cc:224] Handling call
0709 12:19:21.305420 (+697718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:21.382703 (+ 77283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.362631 (+979928us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54001f77d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:22.363019 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1603529088}
I20260709 12:19:22.364411 29134 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:19:22.351128 29453 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.703s	user 0.002s	sys 0.001s
I20260709 12:19:22.344872 12387 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 34 to 38. New cstate: current_term: 38 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:22.321280 12637 rpcz_store.cc:276] 0709 12:19:21.210537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.210688 (+   151us) service_pool.cc:224] Handling call
0709 12:19:22.319978 (+1109290us) spinlock_profiling.cc:243] Waited 75 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:22.320231 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172440448}
I20260709 12:19:22.349331 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:22.366084 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.378775 12567 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:22.381323 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:22.382416 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:22.381989 29433 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.383692 13260 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:22.383937 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 59 candidate_status { last_received { term: 54 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:22.385502 13258 raft_consensus.cc:2410] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 59 because replica has last-logged OpId of term: 56 index: 36, which is greater than that of the candidate, which has last-logged OpId of term: 54 index: 35.
I20260709 12:19:22.345167 29464 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:22.385134 29433 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:22.382805 13262 raft_consensus.cc:2410] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 55 because replica has last-logged OpId of term: 52 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 28.
I20260709 12:19:22.386567 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.390170 29319 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.001s
I20260709 12:19:22.391362 29294 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:22.392424 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.394259 29464 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 50.32, Last appended by leader: 32, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:22.401648 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:22.408809 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50040us
I20260709 12:19:22.407155 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.417197 29231 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.418054 29231 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:22.424624 29464 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:22.432204 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 86486.9us
W20260709 12:19:22.457027 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.459302 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:22.461426 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.462062 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.463361 29276 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:22.464187 12536 leader_election.cc:336] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.464653 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.436798 29120 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 48.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.037s
I20260709 12:19:22.425534 29322 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:22.483888 12610 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.473026 28969 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.012s
I20260709 12:19:22.491920 12566 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
I20260709 12:19:22.492257 12565 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:22.510012 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.491501 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:22.513280 29412 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:22.513484 12387 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 40 to 47. New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:22.519589 13256 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2117) took 1145 ms. Trace:
I20260709 12:19:22.492727 29308 consensus_queue.cc:1048] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.043s
I20260709 12:19:22.528375 29459 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.492472 29432 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:22.537499 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2334) took 1021 ms. Trace:
W20260709 12:19:22.538275 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85657.6us
I20260709 12:19:22.520344 13256 rpcz_store.cc:276] 0709 12:19:21.348818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.349473 (+   655us) service_pool.cc:224] Handling call
0709 12:19:21.349925 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:21.489995 (+140070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.488903 (+998908us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:22.494388 (+  5485us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320429312}
I20260709 12:19:22.529312 12387 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 35 to 36, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 36 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:22.540503 29405 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.571076 29405 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:22.538106 13259 rpcz_store.cc:276] 0709 12:19:21.516301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.516432 (+   131us) service_pool.cc:224] Handling call
0709 12:19:21.516771 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:21.642254 (+125483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.537003 (+894749us) spinlock_profiling.cc:243] Waited 69.7 ms on lock 0x7b54005172d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:22.537326 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160209664}
W20260709 12:19:22.544324 29218 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.626s	user 0.010s	sys 0.015s
I20260709 12:19:22.576282 29218 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.546568 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6098) took 1145 ms. Trace:
I20260709 12:19:22.578423 12644 rpcz_store.cc:276] 0709 12:19:21.400766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.481050 (+ 80284us) service_pool.cc:224] Handling call
0709 12:19:21.506193 (+ 25143us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:21.506660 (+   467us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:19:21.883985 (+377325us) log.cc:844] Serialized 14 byte log entry
0709 12:19:22.362065 (+478080us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:22.363450 (+  1385us) log.cc:844] Serialized 52 byte log entry
0709 12:19:22.363704 (+   254us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:22.363758 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.363921 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:22.546067 (+182146us) raft_consensus.cc:1685] finished
0709 12:19:22.546094 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:22.546328 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52166,"prepare.queue_time_us":344995,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":58198,"raft.run_cpu_time_us":573,"raft.run_wall_time_us":29493,"thread_start_us":712389,"threads_started":2,"wal-append.queue_time_us":434479}
W20260709 12:19:22.580680 29489 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:19:22.558465 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6104) took 1156 ms. Trace:
W20260709 12:19:22.559937 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95835us
W20260709 12:19:22.582370 12536 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:22.582444 29488 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.203s	user 0.002s	sys 0.000s
W20260709 12:19:22.516820 29447 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.938s	user 0.000s	sys 0.006s
W20260709 12:19:22.539949 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:22.592988 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53157.1us
I20260709 12:19:22.593297 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:22.594591 29374 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:22.595420 12536 leader_election.cc:336] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:22.595734 12610 leader_election.cc:336] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.596000 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:22.596210 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.597091 29412 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:22.559209 12566 raft_consensus.cc:1240] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: [47.26-47.26]
I20260709 12:19:22.515137 29134 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.523541 29120 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.567816 29306 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:22.581480 12645 rpcz_store.cc:276] 0709 12:19:21.407756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.643100 (+235344us) service_pool.cc:224] Handling call
0709 12:19:22.117850 (+474750us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:22.119056 (+  1206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:22.119102 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:22.120422 (+  1320us) log.cc:844] Serialized 52 byte log entry
0709 12:19:22.251788 (+131366us) raft_consensus.cc:1645] Marking committed up to 36
0709 12:19:22.254782 (+  2994us) log.cc:844] Serialized 14 byte log entry
0709 12:19:22.255164 (+   382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:22.260251 (+  5087us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:22.365694 (+105443us) raft_consensus.cc:1685] finished
0709 12:19:22.365717 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:22.558100 (+192383us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b54002c6d50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:22.558266 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20934,"prepare.queue_time_us":467,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":14622,"raft.run_cpu_time_us":557,"raft.run_wall_time_us":5398,"spinlock_wait_cycles":1090007296,"thread_start_us":130671,"threads_started":1,"wal-append.queue_time_us":221175}
I20260709 12:19:22.561853 29231 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:22.622835 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:22.633070 12536 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.633843 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:22.636479 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75339.7us
I20260709 12:19:22.639175 29296 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:22.641876 29277 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.642803 29277 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.645078 29120 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.559422 29254 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:22.529384 29459 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:22.662379 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:22.662814 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:22.663337 12572 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:22.665464 29230 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:19:22.666538 29436 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:22.667518 29335 consensus_queue.cc:1048] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:19:22.665477 29417 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:22.668869 29417 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.669418 29417 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:22.671437 29218 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.664640 12460 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:22.672082 29277 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:22.667449 29436 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.619341 29306 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:22.569106 12387 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 22 to 46, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 46 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:22.663551 29107 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.689s	user 0.010s	sys 0.012s
I20260709 12:19:22.707316 29107 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.706424 29120 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:22.705749 29254 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:22.629218 29294 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:22.725416 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:22.716053 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 51598.6us
I20260709 12:19:22.709359 29459 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:22.758740 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:22.661096 29423 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:22.704172 29391 consensus_meta.cc:287] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.796s	user 0.016s	sys 0.008s
W20260709 12:19:22.777047 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.705744 29374 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:22.785841 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44054.5us
I20260709 12:19:22.753062 29218 raft_consensus.cc:2804] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:22.785545 29296 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:22.813467 29218 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.813742 29296 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:22.814459 29296 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
W20260709 12:19:22.815125 29300 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.874s	user 0.020s	sys 0.008s
I20260709 12:19:22.807670 29374 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:19:22.822196 29064 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.625s	user 0.018s	sys 0.008s
W20260709 12:19:22.822986 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 227303us
I20260709 12:19:22.822650 29306 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 49.26, Last appended by leader: 26, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:22.824074 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.824577 29374 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.808696 29276 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:22.808992 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:22.828624 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67764.6us
I20260709 12:19:22.815261 13256 raft_consensus.cc:1275] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 52 index: 37. Preceding OpId from leader: term: 54 index: 38. (index mismatch)
I20260709 12:19:22.806905 29335 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:22.813935 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 189018us
W20260709 12:19:22.802155 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 364119us
I20260709 12:19:22.824605 29306 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:22.821094 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:22.821938 29218 raft_consensus.cc:3060] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:22.824621 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:22.817451 29423 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.831485 13259 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:22.813920 29107 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:22.702813 29134 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:22.858741 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.866608 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2379) took 2169 ms. Trace:
I20260709 12:19:22.866861 29435 raft_consensus.cc:3055] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Stepping down as leader of term 55
I20260709 12:19:22.867527 29435 raft_consensus.cc:740] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:22.867208 12646 rpcz_store.cc:276] 0709 12:19:20.676334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.143506 (+467172us) service_pool.cc:224] Handling call
0709 12:19:21.144029 (+   523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:21.145225 (+  1196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:21.145287 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:21.158334 (+ 13047us) log.cc:844] Serialized 26 byte log entry
0709 12:19:21.345843 (+187509us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:21.345908 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:21.346030 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:22.317022 (+970992us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:22.795091 (+478069us) raft_consensus.cc:1685] finished
0709 12:19:22.795130 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:22.845583 (+ 50453us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":625,"raft.queue_time_us":1063,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":11922,"spinlock_wait_cycles":14934400,"thread_start_us":137436,"threads_started":1,"wal-append.queue_time_us":226872}
I20260709 12:19:22.874018 12387 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 35 to 41, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 41 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:22.868993 29435 consensus_queue.cc:260] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 55.29, Last appended by leader: 29, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:22.875564 29134 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:22.843685 29335 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:22.896735 29120 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:22.908223 12536 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:22.898100 12460 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:22.914906 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 90917.7us
W20260709 12:19:22.953085 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:22.926714 12643 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:22.925457 12644 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 54 index: 24. Preceding OpId from leader: term: 56 index: 25. (index mismatch)
W20260709 12:19:22.930339 12563 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.666s	user 0.004s	sys 0.012s
I20260709 12:19:22.980623 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:22.923967 12645 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:22.924793 12646 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 28. Preceding OpId from leader: term: 54 index: 30. (index mismatch)
I20260709 12:19:22.927301 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:22.925711 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:22.993614 12638 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:22.994650 12537 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:22.993247 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.996981 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:22.915742 29163 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.415s	user 0.016s	sys 0.008s
W20260709 12:19:22.995929 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 168025us
W20260709 12:19:23.001490 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 119099us
I20260709 12:19:22.925258 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:23.002815 12641 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:19:22.896147 29391 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:23.006937 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:22.913618 29335 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.009162 12572 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:22.920006 29254 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.007401 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:23.006227 29435 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:22.914305 29120 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.022852 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:23.021435 29107 consensus_queue.cc:1059] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.012s
I20260709 12:19:23.034977 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:23.035944 12567 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:23.037894 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:23.039856 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:23.041929 29459 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:23.042124 29300 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:23.043991 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:23.016558 29064 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:19:23.045876 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.055872 12564 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:23.058715 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.060314 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:23.060354 12645 raft_consensus.cc:2468] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:23.061002 12564 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:23.063771 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:23.018134 29374 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:23.021577 29136 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
W20260709 12:19:23.069546 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 49452.7us
W20260709 12:19:23.075198 12565 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.560s	user 0.013s	sys 0.000s
I20260709 12:19:23.075588 12565 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:23.084581 12640 raft_consensus.cc:2410] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 44 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
I20260709 12:19:23.011570 29163 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:23.082459 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57320.3us
W20260709 12:19:23.072606 29495 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:19:23.096621 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:23.024282 29464 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.066s
W20260709 12:19:23.103929 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 95444.9us
W20260709 12:19:23.108024 29497 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:19:23.108716 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.111224 12563 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 25. Preceding OpId from leader: term: 50 index: 26. (index mismatch)
I20260709 12:19:23.073788 29064 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:23.092463 29436 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.115404 12568 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 54 index: 24. Preceding OpId from leader: term: 56 index: 25. (index mismatch)
I20260709 12:19:23.120026 12644 raft_consensus.cc:1240] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56. Ops: [55.29-55.29]
I20260709 12:19:23.103018 29294 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:23.123934 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:23.125969 12571 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 49. Current term is 50. Ops: []
W20260709 12:19:23.126168 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.127401 12565 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 23. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:19:23.126462 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:23.129563 12460 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:23.126748 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 235582us
W20260709 12:19:23.138064 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.150596 29300 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:23.151403 29300 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:23.153146 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 53744.1us
W20260709 12:19:23.156081 29494 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
W20260709 12:19:23.157526 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 116222us
W20260709 12:19:23.164476 29499 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:19:23.130950 29436 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
W20260709 12:19:23.160328 12610 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:23.177071 12637 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 26. Preceding OpId from leader: term: 50 index: 26. (term mismatch)
W20260709 12:19:23.178684 12536 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:23.178027 12637 pending_rounds.cc:85] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 25
I20260709 12:19:23.183490 12637 pending_rounds.cc:107] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 49.26
I20260709 12:19:23.183996 12637 raft_consensus.cc:2889] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:23.187176 12644 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
W20260709 12:19:23.180496 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.630s (SENT)
W20260709 12:19:23.186060 12610 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:23.175495 29436 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:23.186844 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:23.196620 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.198947 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:23.201500 12563 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 43 index: 30. (index mismatch)
I20260709 12:19:23.207942 29296 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.209563 12645 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:19:23.210695 29296 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.211216 29259 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 47.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.035s
W20260709 12:19:23.203728 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:23.217558 12569 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 53 index: 30. (index mismatch)
W20260709 12:19:23.218036 29498 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.161s	user 0.003s	sys 0.000s
I20260709 12:19:23.219486 12569 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:23.221897 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:23.223312 12572 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.24-45.24]
I20260709 12:19:23.212219 29230 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:23.227078 29230 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:23.227586 29230 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:23.228348 29294 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20260709 12:19:23.212497 29412 consensus_queue.cc:1048] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.024s
I20260709 12:19:23.230151 29120 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:23.230445 12566 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
W20260709 12:19:23.232506 29500 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.176s	user 0.002s	sys 0.000s
I20260709 12:19:23.234167 29259 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:23.236922 12563 raft_consensus.cc:1240] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 42. Current term is 43. Ops: []
W20260709 12:19:23.238919 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 210680us
W20260709 12:19:23.241303 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75113.8us
I20260709 12:19:23.248258 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:23.249558 12645 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:23.254972 29361 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.002s
I20260709 12:19:23.255322 29464 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:19:23.249464 29319 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.256551 29319 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.258791 29319 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.260219 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.260779 12563 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:23.234678 29259 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:23.263316 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89916.7us
I20260709 12:19:23.243170 29322 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:23.259473 29459 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:19:23.264091 29254 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:23.261353 29296 raft_consensus.cc:3055] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:23.254436 29163 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:23.213294 29300 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.288048 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:23.289343 12642 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:23.280148 29136 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.289361 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:23.296839 12638 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:23.276664 29296 raft_consensus.cc:740] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:23.302088 29259 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.33, Last appended by leader: 33, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:23.308624 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.278795 29459 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:23.314366 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3124) took 1890 ms. Trace:
I20260709 12:19:23.314509 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:23.315778 12571 rpcz_store.cc:276] 0709 12:19:21.413602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:21.425392 (+ 11790us) service_pool.cc:224] Handling call
0709 12:19:21.425856 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:23.127377 (+1701521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:23.303285 (+175908us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54003ad254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:23.303616 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3887496064}
I20260709 12:19:23.316803 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:23.327890 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:23.295156 29322 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:23.344345 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:23.280870 29064 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 20, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.370138 12572 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 46 index: 31. (index mismatch)
I20260709 12:19:23.367702 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:23.299758 29136 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.369378 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.303651 29254 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:23.384517 29259 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:23.369935 29405 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.796s	user 0.021s	sys 0.003s
I20260709 12:19:23.398361 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.399415 12572 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:23.322721 29436 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 46.27, Last appended by leader: 27, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:23.402925 29502 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:19:23.405656 29120 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:23.406231 29120 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:23.406648 29120 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:23.405107 29459 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.30, Last appended by leader: 30, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.419643 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:23.420492 29294 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:23.421875 29134 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:19:23.422748 29391 consensus_queue.cc:1059] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:19:23.423497 29277 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20260709 12:19:23.424451 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.827s (SENT)
W20260709 12:19:23.425921 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.321453 29296 consensus_queue.cc:260] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:23.433753 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 180351us
I20260709 12:19:23.449767 12638 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 55 because replica has last-logged OpId of term: 53 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 29.
W20260709 12:19:23.433189 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 154475us
I20260709 12:19:23.441592 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:23.464666 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52838.9us
I20260709 12:19:23.469971 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.476526 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:23.501375 12571 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:23.505975 12572 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:23.421447 29306 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.046s
W20260709 12:19:23.473997 29218 consensus_meta.cc:287] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.585s	user 0.014s	sys 0.000s
W20260709 12:19:23.580554 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.525285 29107 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.529541 29374 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.586248 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 297320us
I20260709 12:19:23.543958 29296 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:23.596830 29255 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.049s
I20260709 12:19:23.601806 29107 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.602214 29335 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:23.593026 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:23.581564 29264 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.625432 12572 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
W20260709 12:19:23.632232 12567 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.596s	user 0.019s	sys 0.005s
I20260709 12:19:23.560587 29218 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.632754 12567 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:23.587811 29423 consensus_queue.cc:1059] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.041s
I20260709 12:19:23.638446 12460 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:23.643045 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 218647us
I20260709 12:19:23.645270 12566 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.29-43.29]
I20260709 12:19:23.638800 12611 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:23.649786 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:23.641029 29432 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.033s	user 0.008s	sys 0.008s
I20260709 12:19:23.652572 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:23.657387 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 237423us
I20260709 12:19:23.627667 29374 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:23.674961 12569 raft_consensus.cc:2393] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:19:23.673539 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:23.673880 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45881.7us
I20260709 12:19:23.677139 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:23.648818 29322 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:23.666293 29433 consensus_meta.cc:287] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.255s	user 0.021s	sys 0.008s
I20260709 12:19:23.643678 29459 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.652436 29319 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.681190 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:23.672803 29432 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:23.648311 29294 consensus_queue.cc:1059] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.033s
W20260709 12:19:23.699496 13218 leader_election.cc:336] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.901s (SENT)
W20260709 12:19:23.700315 12564 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.638s	user 0.010s	sys 0.005s
I20260709 12:19:23.634070 29264 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.701470 12564 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
W20260709 12:19:23.703202 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 120082us
W20260709 12:19:23.698450 29417 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 1.028s	user 0.022s	sys 0.007s
I20260709 12:19:23.706583 12567 raft_consensus.cc:2410] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 49 because replica has last-logged OpId of term: 47 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 23.
I20260709 12:19:23.710794 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:23.711921 29117 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:23.715085 12572 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
W20260709 12:19:23.722206 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55239.7us
I20260709 12:19:23.722668 29306 raft_consensus.cc:2804] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:23.723421 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:23.725148 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:23.723604 29306 raft_consensus.cc:697] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:23.726465 29511 log.cc:927] Time spent T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:19:23.684239 29361 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:23.686609 29522 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:23.738852 12641 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.683s	user 0.014s	sys 0.000s
I20260709 12:19:23.694069 29322 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.707110 29435 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.662s	user 0.012s	sys 0.002s
I20260709 12:19:23.750283 12570 raft_consensus.cc:1240] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:23.753278 12571 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:19:23.756291 12641 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 40.
W20260709 12:19:23.762164 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.739259 29255 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:23.780078 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:23.704137 29517 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
W20260709 12:19:23.761855 29230 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.005s
I20260709 12:19:23.802021 12460 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:23.763187 29064 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:23.707057 29417 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:24.060366 29120 consensus_meta.cc:287] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.653s	user 0.015s	sys 0.012s
I20260709 12:19:23.727962 29319 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.763900 29254 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:23.863468 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 126263us
I20260709 12:19:23.769747 29322 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:23.848573 29136 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.760403 29459 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:23.885656 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 200986us
W20260709 12:19:23.852771 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 127722us
W20260709 12:19:23.892884 29506 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
I20260709 12:19:23.820799 29255 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:23.818739 29361 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.901839 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6607) took 1583 ms. Trace:
W20260709 12:19:23.903095 29504 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.246s	user 0.000s	sys 0.002s
I20260709 12:19:23.852731 29522 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:23.761121 29300 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:24.111651 29300 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:24.111896 29254 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.112632 29254 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.040377 29405 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:24.114022 29405 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:24.114526 29459 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.115051 29405 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 52.30, Last appended by leader: 30, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.117553 29319 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:24.118285 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.118193 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:24.122177 13259 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 54 index: 32. (index mismatch)
I20260709 12:19:24.123138 29335 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:24.040918 29515 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:24.042243 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033409
W20260709 12:19:24.043292 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.112938 29300 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 53.30, Last appended by leader: 30, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:23.891696 29526 raft_consensus.cc:3055] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:19:23.885406 29306 consensus_queue.cc:237] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:24.050158 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.054513 29374 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:24.055794 12645 consensus_meta.cc:287] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.793s	user 0.010s	sys 0.005s
I20260709 12:19:24.068377 13258 raft_consensus.cc:2368] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 51. Current term is 52.
I20260709 12:19:24.069450 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.088408 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:24.060760 29417 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:23.815992 29517 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:24.067932 29120 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.100209 29361 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.092015 12570 rpcz_store.cc:276] 0709 12:19:22.318768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:22.739798 (+421030us) service_pool.cc:224] Handling call
0709 12:19:22.762610 (+ 22812us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:23.801497 (+1038887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:23.901466 (+ 99969us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:23.901746 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244852608}
I20260709 12:19:24.108752 29117 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.115533 29254 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:24.123229 29255 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.176440 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.123994 29335 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:24.124817 29509 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.417s	user 0.004s	sys 0.007s
I20260709 12:19:24.040230 29423 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:24.064905 29277 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:24.046201 29107 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:24.041615 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 362573us
I20260709 12:19:24.124284 29515 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.153515 12565 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:19:24.066718 29064 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:24.154603 12566 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:24.154595 12645 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:24.164995 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.197186 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:24.173961 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.065697 29230 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.182225 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:24.184149 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 85296.9us
I20260709 12:19:24.145495 29218 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:24.148029 29391 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.010s	sys 0.005s
W20260709 12:19:24.226653 29391 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.010s	sys 0.005s
I20260709 12:19:24.191737 29423 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:24.227869 29391 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:24.228025 29423 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 49.31, Last appended by leader: 31, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:24.226233 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.511s (SENT)
I20260709 12:19:24.191570 29277 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:24.229817 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:24.230580 29277 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 49.30, Last appended by leader: 30, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.205687 29515 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:24.232231 29432 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:24.234018 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.234835 29064 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.28, Last appended by leader: 28, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:24.235675 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.236442 12565 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:24.144507 29526 raft_consensus.cc:740] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:24.131891 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.208297 12572 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.30-43.30]
I20260709 12:19:24.207243 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 41.
W20260709 12:19:24.226164 29516 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.443s	user 0.005s	sys 0.006s
I20260709 12:19:24.228646 29391 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:24.192154 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 368115us
I20260709 12:19:24.255605 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:24.245028 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.256357 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
W20260709 12:19:24.233970 12536 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 1.779s (SENT)
I20260709 12:19:24.257054 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:24.262652 12565 raft_consensus.cc:1240] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 53. Current term is 54. Ops: [53.30-53.30]
I20260709 12:19:24.241511 29107 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.281262 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.210218 29361 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.261085 29391 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:24.296139 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.269019 29423 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:24.314759 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:24.298419 13218 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.653s (SENT) [suppressed 94 similar messages]
W20260709 12:19:24.321256 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.254563 29117 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.321657 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:24.327739 12565 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 52
I20260709 12:19:24.319329 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:24.325519 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 48 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.332449 12568 raft_consensus.cc:2410] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 52 because replica has last-logged OpId of term: 50 index: 32, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 31.
I20260709 12:19:24.281092 29218 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:24.352195 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45020.7us
I20260709 12:19:24.295689 29531 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:24.359760 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.359997 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:24.358019 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.330628 29505 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.368996 29505 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.370076 29322 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.348405 29335 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 42.29, Last appended by leader: 29, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.371862 29519 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.372515 29519 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.375023 29519 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:19:24.382105 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.294407 29107 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.386847 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:24.316054 29435 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.046s
I20260709 12:19:24.330440 29523 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:24.358603 29459 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:19:24.423517 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 99337.7us
I20260709 12:19:24.423877 13256 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
W20260709 12:19:24.394722 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 160795us
I20260709 12:19:24.321746 29255 consensus_queue.cc:1059] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 45.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.063s
I20260709 12:19:24.401508 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:24.438984 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:24.460980 12565 raft_consensus.cc:2468] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:24.462643 29259 raft_consensus.cc:3055] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:24.463203 29259 raft_consensus.cc:740] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:24.463922 13253 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 28. Preceding OpId from leader: term: 49 index: 28. (term mismatch)
I20260709 12:19:24.464427 13253 pending_rounds.cc:85] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 27
I20260709 12:19:24.466216 29230 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.464622 29259 consensus_queue.cc:260] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.24, Last appended by leader: 24, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:24.469761 29521 log.cc:927] Time spent T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.395s	user 0.002s	sys 0.000s
I20260709 12:19:24.469915 29259 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:24.348057 29361 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.464723 13253 pending_rounds.cc:107] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 48.28
I20260709 12:19:24.397157 29319 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
W20260709 12:19:24.482403 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 81275.3us
I20260709 12:19:24.484861 13253 raft_consensus.cc:2889] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:24.486663 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 86957.6us
I20260709 12:19:24.425246 29459 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.493355 29459 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:24.438151 29436 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:19:24.495211 29436 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.495627 29436 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:24.504978 13254 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:24.438572 29433 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:24.466260 29512 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:19:24.450325 29523 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.518389 29433 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.504406 29319 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:24.396742 29218 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.498867 29255 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:24.356246 29517 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:24.551566 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58660.6us
I20260709 12:19:24.546455 29120 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.554096 29218 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.546383 29505 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.556529 29531 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:24.535267 29514 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.253s	user 0.001s	sys 0.001s
I20260709 12:19:24.386186 29526 consensus_queue.cc:260] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 42.24, Last appended by leader: 24, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.564407 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.568145 29218 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.574759 29515 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.521409 29546 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.076s
W20260709 12:19:24.582595 29541 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
W20260709 12:19:24.588518 12536 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.156s (SENT)
I20260709 12:19:24.527407 29523 raft_consensus.cc:3060] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:24.546360 29319 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:24.599545 29218 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:24.600975 29529 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
I20260709 12:19:24.548434 29255 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:24.622996 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.643589 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:24.602839 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87592.9us
I20260709 12:19:24.554567 29522 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.562165 29230 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:24.698158 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.699681 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:24.713811 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:24.722877 12645 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 54 index: 32. (index mismatch)
I20260709 12:19:24.725584 12641 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:24.690013 29255 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:24.727532 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:24.732679 12646 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 55
I20260709 12:19:24.714883 12644 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:24.755002 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.763404 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:24.597381 29531 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.716193 29522 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.708940 29539 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:24.764374 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:24.695936 29549 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 42.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.055s
I20260709 12:19:24.728510 29322 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:24.787194 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:24.788218 12638 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:24.790202 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.790712 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.737512 29535 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:24.791230 13254 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:24.789450 29527 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader pre-election won for term 59
I20260709 12:19:24.721751 29230 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:24.816871 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.812103 12569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6627) took 1268 ms. Trace:
I20260709 12:19:24.815665 13262 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:24.737042 29306 consensus_queue.cc:1059] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 51.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.061s
I20260709 12:19:24.777539 29556 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:24.824546 29545 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:24.825142 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 236671us
I20260709 12:19:24.830607 29515 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:24.831343 29107 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.810341 29535 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:24.834777 29548 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
W20260709 12:19:24.854686 29555 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 12:19:24.857872 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 172551us
W20260709 12:19:24.862150 29557 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:19:24.820089 29550 consensus_queue.cc:1059] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 43.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.303s
I20260709 12:19:24.867342 12646 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:24.784124 29322 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:24.776819 29117 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.873250 12568 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 49 index: 28. (index mismatch)
I20260709 12:19:24.817960 12569 rpcz_store.cc:276] 0709 12:19:23.543857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:23.703666 (+159809us) service_pool.cc:224] Handling call
0709 12:19:23.747348 (+ 43682us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:23.764880 (+ 17532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:23.764955 (+    75us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:23.845828 (+ 80873us) log.cc:844] Serialized 26 byte log entry
0709 12:19:24.160580 (+314752us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:24.160845 (+   265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:24.161373 (+   528us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:24.811756 (+650383us) raft_consensus.cc:1685] finished
0709 12:19:24.811797 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:24.812009 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":398,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":6201,"thread_start_us":50306,"threads_started":1,"wal-append.queue_time_us":244869}
I20260709 12:19:24.887010 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.889149 29322 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:24.889178 29533 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:24.890885 29535 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.890576 29533 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.889864 29433 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.891495 29545 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:24.894012 29558 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.894630 29558 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.897073 29558 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.898101 29533 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.898226 29319 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.888758 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.906711 12569 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:24.907477 29319 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:24.908068 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:24.913905 12569 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
W20260709 12:19:24.919128 13258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50696 (request call id 2108) took 2113 ms (client timeout 1598 ms). Trace:
W20260709 12:19:24.919699 13258 rpcz_store.cc:269] 0709 12:19:22.805330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:22.820741 (+ 15411us) service_pool.cc:224] Handling call
0709 12:19:24.918823 (+2098082us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400578ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:24.919006 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2816977664}
I20260709 12:19:24.924398 29134 raft_consensus.cc:3055] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:24.925009 29134 raft_consensus.cc:740] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:24.883427 29556 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.933189 29556 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:24.903276 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.903846 29531 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:24.904702 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6631) took 1194 ms. Trace:
I20260709 12:19:24.935770 29264 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:24.902037 12536 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Timed out: RequestConsensusVote RPC to 127.11.11.129:38859 timed out after 2.163s (SENT)
I20260709 12:19:24.799000 29522 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.944777 12642 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 49 index: 28. (index mismatch)
W20260709 12:19:24.950191 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.966253 12571 rpcz_store.cc:276] 0709 12:19:23.710770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:23.806812 (+ 96042us) service_pool.cc:224] Handling call
0709 12:19:24.056320 (+249508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:24.066429 (+ 10109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:24.066487 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:24.132176 (+ 65689us) log.cc:844] Serialized 26 byte log entry
0709 12:19:24.384588 (+252412us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:24.384752 (+   164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:24.384831 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:24.904440 (+519609us) raft_consensus.cc:1685] finished
0709 12:19:24.904464 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:24.904632 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34569,"raft.queue_time_us":22175,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":22376,"thread_start_us":170635,"threads_started":1,"wal-append.queue_time_us":353487}
I20260709 12:19:24.934271 29322 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:24.977873 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:24.888748 29230 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:24.975001 13218 connection.cc:256] RPC call timeout handler was delayed by -1.2878101609999248s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2544s for a response
I20260709 12:19:24.984652 13262 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 54.
I20260709 12:19:24.987941 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:24.973114 29515 raft_consensus.cc:3055] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Stepping down as leader of term 53
I20260709 12:19:24.871507 29554 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 47.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.109s
W20260709 12:19:24.992049 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:24.972764 29535 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:24.998090 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.006575 12642 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 49 because replica has last-logged OpId of term: 48 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
I20260709 12:19:25.034343 12644 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:19:24.965953 29117 consensus_queue.cc:1048] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.033s
W20260709 12:19:25.034415 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 207613us
I20260709 12:19:24.996850 29515 raft_consensus.cc:740] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:25.074070 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.074669 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:25.037114 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 344984us
I20260709 12:19:25.081418 12642 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 48. Current term is 49. Ops: [48.28-48.28]
W20260709 12:19:25.084079 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.087333 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.032892 29545 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.090682 12645 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 54: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:25.096742 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.002287 29535 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:25.118263 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 216254us
I20260709 12:19:25.116609 12640 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 55 because replica has last-logged OpId of term: 53 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 30.
I20260709 12:19:25.048988 29565 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.138444 12569 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 30. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
W20260709 12:19:25.138973 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61560.2us
I20260709 12:19:25.146898 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.146989 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 53 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.163290 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.090586 29558 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.173553 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.167279 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.168491 12642 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:25.169082 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.175064 12571 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
W20260709 12:19:25.176242 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.086558 29527 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:25.091619 29550 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:25.178329 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.032225 29134 consensus_queue.cc:260] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 45.24, Last appended by leader: 24, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.183262 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 42.
I20260709 12:19:25.186308 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:25.188923 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:25.189721 12571 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 52
W20260709 12:19:25.181331 29391 consensus_meta.cc:287] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.854s	user 0.008s	sys 0.006s
I20260709 12:19:25.207738 29391 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.210361 29391 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.198442 29550 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.214494 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.215623 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:25.216655 29436 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.721s	user 0.011s	sys 0.000s
I20260709 12:19:25.217033 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.204970 29134 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:25.207044 29522 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:25.156774 29433 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.041s
W20260709 12:19:25.257759 29563 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:19:25.258388 12569 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 29. Preceding OpId from leader: term: 55 index: 31. (index mismatch)
I20260709 12:19:25.158870 29565 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.181945 29527 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Advancing to term 59
I20260709 12:19:25.219871 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.262027 12637 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:25.263367 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.264137 12637 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:25.265241 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.196599 29515 consensus_queue.cc:260] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 53.30, Last appended by leader: 30, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.217170 29436 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.288136 29361 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:25.242384 29423 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.916s	user 0.014s	sys 0.000s
W20260709 12:19:25.338795 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.220572 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.021s	user 0.033s	sys 0.015s Metrics: {"bytes_written":4411,"delete_count":0,"lbm_write_time_us":3027,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":166}
I20260709 12:19:25.098021 29554 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.082s
I20260709 12:19:25.182224 29558 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:25.224395 29528 log.cc:927] Time spent T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.960s	user 0.000s	sys 0.006s
W20260709 12:19:25.225354 29459 consensus_meta.cc:287] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.731s	user 0.009s	sys 0.002s
I20260709 12:19:25.226429 12640 raft_consensus.cc:1240] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 52. Current term is 53. Ops: [52.29-52.29]
W20260709 12:19:25.228689 29544 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.519s	user 0.001s	sys 0.005s
I20260709 12:19:25.231035 29550 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.240900 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:25.247097 12571 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:25.444878 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:25.250710 12572 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 40 index: 21. (index mismatch)
I20260709 12:19:25.419842 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.197580 29120 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:25.423026 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.452526 12639 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 46 index: 31. (index mismatch)
I20260709 12:19:25.420830 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:25.452713 29120 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.453507 13253 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 44: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:25.454383 29571 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.008s
I20260709 12:19:25.454649 29527 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.455385 29436 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.455679 29120 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.456861 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:25.457150 29527 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.443094 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:25.419567 29568 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.171s	user 0.000s	sys 0.006s
I20260709 12:19:25.447515 13256 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.30-43.30]
I20260709 12:19:25.448346 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:25.443523 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:25.451195 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:25.442483 29255 consensus_meta.cc:287] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.701s	user 0.013s	sys 0.004s
W20260709 12:19:25.462881 11310 thread.cc:527] Waited for 1000ms trying to join with rpc worker (tid 12468)
W20260709 12:19:25.443871 29523 consensus_meta.cc:287] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.745s	user 0.012s	sys 0.024s
W20260709 12:19:25.464986 29259 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.993s	user 0.011s	sys 0.008s
I20260709 12:19:25.446307 29433 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.020s
I20260709 12:19:25.429563 29546 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:25.466369 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.466835 12565 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:19:25.535506 29546 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:25.536041 12640 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:25.536569 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.445855 12611 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.467590 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:25.468472 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:25.470151 12461 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:25.537700 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 95322.9us
I20260709 12:19:25.470774 29277 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:25.471410 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:25.474706 29558 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.476788 29532 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.449321 29459 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.481459 29218 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.483033 29322 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.435890 29522 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:25.487352 29562 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:25.476774 29543 raft_consensus.cc:3055] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:25.438994 29565 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:25.485026 29517 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:19:25.500360 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 55560.2us
I20260709 12:19:25.434711 29515 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:25.504456 12563 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:25.504566 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.505522 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:25.500860 29296 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:25.513260 12570 raft_consensus.cc:1217] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Deduplicated request from leader. Original: 43.23->[45.24-45.24]   Dedup: 45.24->[]
I20260709 12:19:25.482918 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.436607 12637 raft_consensus.cc:2393] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 56: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:25.514921 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:25.515316 29576 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.515895 29300 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:25.516408 13258 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:19:25.516435 29535 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:25.447515 12536 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:25.526432 13217 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.530002 12642 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
I20260709 12:19:25.533792 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:25.533986 12384 maintenance_manager.cc:419] P 2d10571632e7445d99e8f1c750c96f70: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.192586
I20260709 12:19:25.531575 29255 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.537196 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.539127 13259 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
W20260709 12:19:25.542245 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 72179us
W20260709 12:19:25.547588 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.598178 12570 raft_consensus.cc:1240] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 45. Current term is 46. Ops: [45.24-45.24]
W20260709 12:19:25.548305 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:25.599498 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73169.4us
I20260709 12:19:25.563117 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:25.565902 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:25.600811 12567 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:25.607621 29322 raft_consensus.cc:2804] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:25.608017 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.608323 29322 raft_consensus.cc:697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:25.608928 12563 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:25.609656 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 166390us
I20260709 12:19:25.610260 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.573949 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.611217 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:25.614955 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.577206 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.544207 29391 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:25.579244 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.619879 12641 raft_consensus.cc:2376] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48. Re-sending same reply.
W20260709 12:19:25.584764 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.551681 29464 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:25.620787 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 44 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:25.621804 12569 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:25.537122 29523 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.589476 12460 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:25.625492 12565 raft_consensus.cc:2393] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:25.605970 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.577585 29550 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.577894 29543 raft_consensus.cc:740] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:25.597357 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:25.599001 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 161359us
I20260709 12:19:25.581672 29558 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:25.608903 29255 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.559235 29532 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.630285 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.666074 12641 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 56.
I20260709 12:19:25.588143 29296 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.624246 29259 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 29, Time since last communication: 0.010s
I20260709 12:19:25.536571 29546 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.30, Last appended by leader: 30, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.619642 29391 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:25.647594 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 207113us
I20260709 12:19:25.679968 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:25.648175 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:25.651780 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:25.654593 13257 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2124) took 1446 ms. Trace:
I20260709 12:19:25.655548 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.681290 12572 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:25.657842 29464 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:25.660567 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 73933.5us
I20260709 12:19:25.661211 29459 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.664346 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:25.597108 29569 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:25.598600 29300 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:25.675899 13255 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:25.628989 29322 consensus_queue.cc:237] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 50.34, Last appended by leader: 34, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.674044 12645 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:25.582680 29522 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.680950 12611 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.684815 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.670217 29405 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:25.681299 13257 rpcz_store.cc:276] 0709 12:19:24.204593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:24.280875 (+ 76282us) service_pool.cc:224] Handling call
0709 12:19:25.514586 (+1233711us) spinlock_profiling.cc:243] Waited 6.94 ms on lock 0x7b5800150398. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf26ceb71 00007ffbf26cea22 00007ffbfb5e78d9 00007ffbfa741ee0 00007ffbfa80b0ab 00007ffbfa80b131 00007ffbfa7b90a0 00007ffbfa7b8984 00007ffbf3d336d4 00007ffbf3d33662
0709 12:19:25.650803 (+136217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15965440}
W20260709 12:19:25.702221 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 84149.2us
I20260709 12:19:25.673802 29296 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:25.687410 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:25.689018 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.649473 29064 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:25.685141 29391 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:25.681691 12637 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:25.711238 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:25.712213 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 213237us
W20260709 12:19:25.707685 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:25.717103 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.684365 29306 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 42.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.024s
I20260709 12:19:25.704149 29519 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:25.609522 29576 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.675354 29259 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 53.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:19:25.719918 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:25.690197 29117 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:25.687041 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:25.732139 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:25.696575 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:25.705869 29405 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.709177 29064 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:25.733879 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 67476.4us
I20260709 12:19:25.737926 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.686141 29550 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.739208 12641 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:25.711534 29107 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:25.722644 29558 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:25.746146 12567 raft_consensus.cc:2468] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:25.735982 12460 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:25.747746 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51309.3us
I20260709 12:19:25.739720 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.748194 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:25.722693 29264 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:25.746604 13262 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:25.730597 29306 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 44.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:19:25.731812 29361 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:25.726526 29519 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:25.698365 29569 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.740658 29405 raft_consensus.cc:3060] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:25.723151 29459 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.030s
I20260709 12:19:25.742219 29064 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:25.750214 29558 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.727193 29543 consensus_queue.cc:260] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 43.29, Last appended by leader: 29, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.813287 29543 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:25.766325 29218 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:25.817907 29435 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 47.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:25.818233 29218 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:25.748687 29300 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 47.24, Last appended by leader: 24, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.819474 29218 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 50.32, Last appended by leader: 32, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.710630 29578 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.055s
I20260709 12:19:25.821785 29578 raft_consensus.cc:3055] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
I20260709 12:19:25.822237 29578 raft_consensus.cc:740] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:25.822618 29576 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:25.823361 29578 consensus_queue.cc:260] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 47.29, Last appended by leader: 29, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.725528 29319 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.042s
I20260709 12:19:25.824997 29569 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:25.827301 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.828167 12568 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:25.764776 29436 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:25.829815 13217 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.830904 29576 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:25.831332 29576 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.831737 29576 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:25.763765 29554 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:25.785993 29550 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.741526 29117 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:25.790895 29577 log.cc:927] Time spent T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.594s	user 0.008s	sys 0.000s
W20260709 12:19:25.834343 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 64575.2us
I20260709 12:19:25.836427 29117 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.749157 29535 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
W20260709 12:19:25.788513 29134 consensus_meta.cc:287] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.537s	user 0.013s	sys 0.000s
W20260709 12:19:25.802731 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 152194us
W20260709 12:19:25.846309 12610 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:25.846769 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:25.794548 29523 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:25.818964 29405 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.820113 29300 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:25.836651 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:25.726704 29570 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:19:25.839277 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.843794 12641 raft_consensus.cc:2410] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 48 because replica has last-logged OpId of term: 47 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 25.
I20260709 12:19:25.770408 29519 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:25.858651 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:25.867406 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.868307 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.870036 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.871223 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.871552 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.871963 12642 raft_consensus.cc:2368] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:25.872347 12637 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:25.872820 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 42 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.873142 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.873574 12642 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 42: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:25.867475 12569 raft_consensus.cc:2468] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:25.874882 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.876405 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.877610 12640 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:25.808406 29264 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
W20260709 12:19:25.880692 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.838377 29558 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:25.883723 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.891319 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:25.893033 29569 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
W20260709 12:19:25.894572 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 132189us
I20260709 12:19:25.875542 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:25.795349 29464 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.28, Last appended by leader: 28, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:25.885486 29306 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:25.885946 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 49325.1us
I20260709 12:19:25.888651 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.896054 12637 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:25.897200 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.899564 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:25.898039 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:25.900384 13262 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:25.902101 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.907634 12645 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:25.909462 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:25.911379 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.882266 29550 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
W20260709 12:19:25.917970 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 169853us
I20260709 12:19:25.845188 29107 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 44.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.056s
I20260709 12:19:25.893682 29569 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:25.923848 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:25.887133 29586 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:19:25.912452 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:25.918793 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 187270us
I20260709 12:19:25.921490 12461 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:25.927718 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81451.2us
W20260709 12:19:25.933569 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:25.936836 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.939730 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:25.935813 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:25.927054 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 165781us
I20260709 12:19:25.904335 29306 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:25.946442 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:25.893544 29391 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:25.920562 29134 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:25.931084 29569 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:25.985766 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 99051.2us
I20260709 12:19:25.953235 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:25.995126 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46108.8us
W20260709 12:19:26.012199 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.015381 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.949755 29523 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:26.018348 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:25.981365 12642 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:25.987602 12638 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 30. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:19:26.012152 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:26.004434 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:25.955008 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:25.974275 12644 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:25.947074 29570 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:26.023422 12571 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:26.026587 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:26.027571 29570 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:26.028417 29570 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:26.031688 29405 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:26.007879 29134 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:26.034025 29134 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:26.035612 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:26.036280 12568 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 47
I20260709 12:19:26.014103 29522 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:25.951351 29554 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.076s
I20260709 12:19:26.037238 29296 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:26.039333 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105803us
I20260709 12:19:26.000826 29590 consensus_queue.cc:1059] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 43.26, Next index: 27, Last known committed idx: 25, Time since last communication: 0.052s
I20260709 12:19:25.984925 29459 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.056s
I20260709 12:19:26.042024 29523 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260709 12:19:26.042272 29522 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:19:26.043336 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88410.9us
I20260709 12:19:26.043848 29306 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:26.044237 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:26.040246 12610 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.045094 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:25.990118 29255 consensus_queue.cc:1059] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.055s
W20260709 12:19:26.046403 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:26.046586 12610 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.047152 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:26.047844 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:26.049124 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.050107 29277 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:26.052269 29306 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:25.953341 29545 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:19:26.051210 29277 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:25.955756 29319 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.101s
I20260709 12:19:26.049823 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:26.039758 29296 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:26.063131 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:26.064122 12572 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Advancing to term 43
I20260709 12:19:26.041620 12566 raft_consensus.cc:1240] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 43. Current term is 44. Ops: [43.29-43.29]
I20260709 12:19:26.064914 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.033401 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:26.067149 12569 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:26.061408 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:26.072569 12568 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:26.079579 12637 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:26.083894 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:26.085780 12568 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 52: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:26.091045 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:26.021016 29551 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.094173 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 43 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:26.095033 12637 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:26.086133 12609 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:26.101195 12566 raft_consensus.cc:2393] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:26.101040 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 96734.9us
I20260709 12:19:26.097237 12609 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:26.105496 13262 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 31. Preceding OpId from leader: term: 52 index: 32. (index mismatch)
W20260709 12:19:26.105911 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.108847 29601 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:26.109849 29601 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:26.119349 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:26.081068 29522 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:19:26.121588 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71279.9us
I20260709 12:19:26.085271 29515 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:26.129953 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:26.082969 29535 raft_consensus.cc:2764] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election decision vote started in defunct term 49: won
I20260709 12:19:26.098450 29277 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:26.130373 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:26.139643 29586 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:26.134907 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.162964 12637 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:26.139053 12569 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:26.140604 29550 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20260709 12:19:26.014968 29588 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.094s
I20260709 12:19:26.143565 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:26.124078 29306 raft_consensus.cc:2804] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:26.125535 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:26.145174 13255 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:26.097517 29255 consensus_queue.cc:1243] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 45.28 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 27
I20260709 12:19:26.152515 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:26.134227 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.154986 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:26.096287 29589 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:26.165218 12461 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:26.127970 29517 raft_consensus.cc:2804] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:26.170014 13262 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:26.204298 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:26.208926 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:26.129750 29522 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:26.205346 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 156256us
I20260709 12:19:26.145910 29296 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.219185 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:26.220000 12637 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
W20260709 12:19:26.213611 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 48571.7us
I20260709 12:19:26.222131 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
I20260709 12:19:26.207350 12568 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:26.216485 12610 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.224510 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:26.217571 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50717.8us
I20260709 12:19:26.100857 29551 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:26.273761 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.320850 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 49143.4us
W20260709 12:19:26.323879 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 192804us
I20260709 12:19:26.305509 29255 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:19:26.219009 29522 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:26.346254 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:26.250785 29306 raft_consensus.cc:697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:26.256937 29517 raft_consensus.cc:697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:26.361340 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 206409us
I20260709 12:19:26.251159 29296 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.354054 29545 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:19:26.354516 29255 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:26.393646 12568 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:26.394703 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:26.405913 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:26.413439 29296 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:26.414680 29517 consensus_queue.cc:237] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 44.24, Last appended by leader: 24, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:26.382267 29596 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:19:26.436915 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 220475us
W20260709 12:19:26.438301 12610 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.438872 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.442839 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:19:26.425143 29591 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.409s	user 0.000s	sys 0.002s
I20260709 12:19:26.413308 29255 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 41.24, Last appended by leader: 24, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:26.459141 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:26.381227 29594 consensus_queue.cc:1059] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 51.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.061s
I20260709 12:19:26.427536 29296 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:26.380344 29586 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.432655 29595 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:26.498075 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 59834.6us
W20260709 12:19:26.499063 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:26.486155 29602 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:19:26.471289 29296 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:26.476387 29218 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:26.473507 29545 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
W20260709 12:19:26.510612 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67915us
I20260709 12:19:26.495239 29306 consensus_queue.cc:237] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:26.504599 29596 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:19:26.542117 13219 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.545857 12536 leader_election.cc:400] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:26.546303 12536 leader_election.cc:403] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Cancelling election due to peer responding with higher term
I20260709 12:19:26.516805 29296 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:26.550360 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:26.532256 29569 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.000s
W20260709 12:19:26.555182 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.544924 29596 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:26.550151 29523 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:26.549889 29597 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.299s	user 0.002s	sys 0.000s
I20260709 12:19:26.523530 29616 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:26.572861 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:26.578953 13262 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:26.539878 29218 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:26.586524 29569 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.625155 12643 raft_consensus.cc:2368] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
I20260709 12:19:26.624953 29218 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:26.626557 29134 consensus_meta.cc:287] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.005s
I20260709 12:19:26.627172 29134 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:26.575584 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:26.628152 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52611.9us
W20260709 12:19:26.629982 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.630261 29526 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:19:26.630630 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.631072 29562 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:26.632061 29335 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:26.587939 29605 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election lost for term 50. Reason: Vote denied by peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471) with higher term. Message: Invalid argument: T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51.
W20260709 12:19:26.611253 29436 consensus_meta.cc:287] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.781s	user 0.012s	sys 0.000s
I20260709 12:19:26.585353 29596 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:26.615015 29576 consensus_meta.cc:287] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.783s	user 0.014s	sys 0.000s
W20260709 12:19:26.633064 13218 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.620002 29523 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:26.623814 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51039.8us
I20260709 12:19:26.667045 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:26.627532 29569 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:26.593430 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.635906 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 57330.9us
W20260709 12:19:26.668517 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.583086 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 59289.3us
I20260709 12:19:26.634677 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:26.675714 13262 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:26.640442 12568 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 54 index: 31. (index mismatch)
W20260709 12:19:26.642040 29531 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.886s	user 0.012s	sys 0.027s
I20260709 12:19:26.650444 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:26.664983 12564 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 28. Preceding OpId from leader: term: 44 index: 29. (index mismatch)
W20260709 12:19:26.641966 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142208us
W20260709 12:19:26.679908 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.653s (SENT)
I20260709 12:19:26.681427 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.681715 12568 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 42 index: 29. Preceding OpId from leader: term: 44 index: 30. (index mismatch)
I20260709 12:19:26.682894 12564 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 32. Preceding OpId from leader: term: 52 index: 33. (index mismatch)
I20260709 12:19:26.666956 29436 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:26.687203 12564 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 50 index: 34. Preceding OpId from leader: term: 52 index: 35. (index mismatch)
W20260709 12:19:26.666604 29519 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.784s	user 0.011s	sys 0.004s
W20260709 12:19:26.693162 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.677536 29543 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.860s	user 0.007s	sys 0.008s
W20260709 12:19:26.689551 29531 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.012s	sys 0.028s
I20260709 12:19:26.703465 29519 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.728178 29522 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:26.708257 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58264.5us
W20260709 12:19:26.736495 13255 consensus_meta.cc:287] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.530s	user 0.013s	sys 0.000s
W20260709 12:19:26.701048 12610 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.737953 13255 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 44 index: 29. (term mismatch)
I20260709 12:19:26.711221 29562 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:26.712111 29464 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:26.715862 29619 raft_consensus.cc:3055] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:26.703297 29433 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:26.699128 29613 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:26.632822 29621 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.136s
W20260709 12:19:26.717056 13218 leader_election.cc:336] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.695s (SENT)
I20260709 12:19:26.717324 29322 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:26.720225 29134 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:26.670404 29576 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.722779 12641 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
I20260709 12:19:26.728962 29519 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:26.774565 29064 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.900s	user 0.014s	sys 0.000s
I20260709 12:19:26.740461 29619 raft_consensus.cc:740] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:26.739065 13262 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:26.738372 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.737609 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:26.742360 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:26.699373 29594 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:26.738531 13255 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 28
W20260709 12:19:26.729936 29622 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:19:26.748301 29464 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:26.762415 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:26.764007 29391 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:26.767473 29296 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:26.766882 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:26.840649 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 73795.3us
W20260709 12:19:26.774430 29618 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:19:26.798952 29576 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:26.799108 29630 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.001s
I20260709 12:19:26.705088 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:26.853345 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 148384us
I20260709 12:19:26.854133 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
W20260709 12:19:26.807744 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:26.857277 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:26.811357 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:26.732396 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:26.824592 13253 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:26.861861 29620 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:19:26.862560 29620 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:26.862608 29619 consensus_queue.cc:260] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 52.29, Last appended by leader: 29, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:26.863593 29620 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 49.31, Last appended by leader: 31, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.864872 29619 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:26.868184 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:26.868315 29578 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:26.870724 29549 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:26.872051 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4830) took 1597 ms. Trace:
I20260709 12:19:26.872493 12642 rpcz_store.cc:276] 0709 12:19:25.274570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.923397 (+648827us) service_pool.cc:224] Handling call
0709 12:19:26.871842 (+948445us) spinlock_profiling.cc:243] Waited 56.5 ms on lock 0x7b540018c9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:26.871983 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129873664}
I20260709 12:19:26.875018 29532 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:26.870805 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171076us
I20260709 12:19:26.876660 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:26.819746 29570 consensus_meta.cc:287] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.790s	user 0.015s	sys 0.000s
W20260709 12:19:26.824594 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 87043.8us
W20260709 12:19:26.841430 13218 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.833091 13255 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 43.29
I20260709 12:19:26.880477 13255 raft_consensus.cc:2917] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Aborting config change with OpId 43.29: Aborted: Op aborted by new leader
I20260709 12:19:26.849298 29296 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:26.880669 13218 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.880960 13255 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:26.877060 12610 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.911240 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.911587 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:26.805678 29433 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:26.911525 12536 connection.cc:256] RPC call timeout handler was delayed by -1.3159174739998889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2238s for a response
I20260709 12:19:26.746894 29335 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.094s
I20260709 12:19:26.913352 12642 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:26.908581 13218 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:26.921289 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:26.892840 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 88915.9us
W20260709 12:19:26.928450 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47803.1us
W20260709 12:19:26.931344 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54310.7us
I20260709 12:19:26.932483 29549 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20260709 12:19:26.840981 29218 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:26.928532 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:26.932142 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.962872 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.985189 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:26.985773 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 60425.4us
I20260709 12:19:26.986733 12642 raft_consensus.cc:2410] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 45 because replica has last-logged OpId of term: 44 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 24.
I20260709 12:19:26.987615 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6146) took 1008 ms. Trace:
I20260709 12:19:26.988087 12642 rpcz_store.cc:276] 0709 12:19:25.978963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.984705 (+1005742us) service_pool.cc:224] Handling call
0709 12:19:26.987519 (+  2814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:26.989609 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:26.991257 12642 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:26.990402 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60857.6us
W20260709 12:19:27.025319 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4812) took 1804 ms (client timeout 2238 ms). Trace:
I20260709 12:19:26.912293 29306 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:27.038023 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:27.020864 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:27.025952 12643 rpcz_store.cc:269] 0709 12:19:25.220789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.878713 (+657924us) service_pool.cc:224] Handling call
0709 12:19:27.024996 (+1146283us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b54002e5254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:27.025201 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1671921536}
I20260709 12:19:27.041848 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:27.033227 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45822.5us
I20260709 12:19:27.047781 13257 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:26.919463 29590 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:27.051882 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.052546 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:26.833827 29064 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:27.060179 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 138949us
I20260709 12:19:26.934764 29532 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:26.789000 29613 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:26.844244 29594 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.081297 13256 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 58. Current term is 59. Ops: []
W20260709 12:19:27.083719 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 45822.3us
I20260709 12:19:27.049993 29306 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:27.048115 29605 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
W20260709 12:19:27.091512 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:26.913436 29570 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:27.056355 29576 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:27.064507 29543 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
W20260709 12:19:27.101058 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.103051 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.054574 29588 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.048s
W20260709 12:19:27.115326 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.060282 29436 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.049s
I20260709 12:19:27.099284 29306 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:27.059770 29606 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.047s
W20260709 12:19:27.140416 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.062467 29590 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.094928 29636 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:27.152541 13253 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:27.145136 12643 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:27.169541 13255 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: [52.32-52.32]
I20260709 12:19:27.154610 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:27.159340 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 227205us
W20260709 12:19:27.172163 12610 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:27.153363 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 104515us
I20260709 12:19:27.155339 29621 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:27.188624 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.189323 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:27.191802 12571 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:27.147660 29523 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:27.175105 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 241810us
I20260709 12:19:27.227260 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.195420 12642 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:27.253135 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6161) took 1243 ms. Trace:
I20260709 12:19:27.253716 12642 rpcz_store.cc:276] 0709 12:19:26.009581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.989188 (+979607us) service_pool.cc:224] Handling call
0709 12:19:27.253044 (+263856us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:27.254207 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.255538 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.256469 12642 raft_consensus.cc:2393] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 53: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:27.227598 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:27.105938 29464 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:27.195688 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.259191 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.156623 29319 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:27.194535 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:27.262955 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6167) took 1204 ms. Trace:
I20260709 12:19:27.263634 12645 rpcz_store.cc:276] 0709 12:19:26.059830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.184543 (+1124713us) service_pool.cc:224] Handling call
0709 12:19:27.262853 (+ 78310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:27.158383 29218 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:27.271378 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2403) took 2016 ms. Trace:
I20260709 12:19:27.273779 12642 rpcz_store.cc:276] 0709 12:19:25.255209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.255209 (+2000000us) service_pool.cc:224] Handling call
0709 12:19:27.271297 (+ 16088us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:27.227847 29635 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 12:19:27.256497 29634 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.418s	user 0.002s	sys 0.000s
I20260709 12:19:27.181970 29335 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:27.285246 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.303566 12642 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:27.304422 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6143) took 1326 ms. Trace:
I20260709 12:19:27.304827 12642 rpcz_store.cc:276] 0709 12:19:25.978075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.284847 (+1306772us) service_pool.cc:224] Handling call
0709 12:19:27.304324 (+ 19477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:27.187197 29522 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:27.307009 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:27.308338 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.312265 12642 raft_consensus.cc:2410] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 47 because replica has last-logged OpId of term: 45 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 29.
I20260709 12:19:27.309783 12571 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:27.176803 29587 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.105s
I20260709 12:19:27.266924 29064 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.320456 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34832 (request call id 4823) took 2064 ms. Trace:
I20260709 12:19:27.321241 12642 rpcz_store.cc:276] 0709 12:19:25.255944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.307908 (+2051964us) service_pool.cc:224] Handling call
0709 12:19:27.320344 (+ 12436us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:27.323765 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.327152 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:27.330384 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6166) took 1276 ms. Trace:
I20260709 12:19:27.331959 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.357882 12645 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:27.358982 29590 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.359436 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.360172 29570 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:27.360450 12645 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:27.362211 29633 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:19:27.362392 12568 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 52 index: 29. Preceding OpId from leader: term: 53 index: 29. (term mismatch)
I20260709 12:19:27.363163 12568 pending_rounds.cc:85] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 28
I20260709 12:19:27.363502 12568 pending_rounds.cc:107] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 52.29
I20260709 12:19:27.363886 12568 raft_consensus.cc:2889] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:27.364650 29546 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:27.363894 29632 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.265s	user 0.001s	sys 0.000s
I20260709 12:19:27.370286 29640 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:27.371081 29640 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:27.371542 29640 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:27.351835 29218 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 47.28, Last appended by leader: 28, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.354178 29319 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.375277 29601 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.375592 29319 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.377192 29551 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:27.370397 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63570.3us
I20260709 12:19:27.379434 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:27.379758 29433 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:27.380517 12642 rpcz_store.cc:276] 0709 12:19:26.055872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.323368 (+1267496us) service_pool.cc:224] Handling call
0709 12:19:27.330293 (+  6925us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:27.227080 29621 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.385663 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:27.321221 29523 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:27.381834 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.389837 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 57.
I20260709 12:19:27.394040 29319 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.397881 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 40 candidate_status { last_received { term: 38 index: 20 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.401074 12642 raft_consensus.cc:2393] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 40: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
W20260709 12:19:27.408723 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 222198us
W20260709 12:19:27.422245 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.422904 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:27.403487 29558 consensus_meta.cc:287] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.500s	user 0.014s	sys 0.003s
I20260709 12:19:27.382593 29570 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:27.425388 12645 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:27.439100 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:27.446071 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.407610 29433 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:27.443100 12460 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:27.474100 12563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.000s	sys 0.008s
W20260709 12:19:27.474603 12563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.000s	sys 0.008s
I20260709 12:19:27.475494 12563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6662) took 1458 ms. Trace:
I20260709 12:19:27.440444 29570 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:27.483366 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 61177.3us
W20260709 12:19:27.486790 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.475912 12563 rpcz_store.cc:276] 0709 12:19:26.016857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.017024 (+   167us) service_pool.cc:224] Handling call
0709 12:19:26.062323 (+ 45299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:26.062598 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 26
0709 12:19:26.093170 (+ 30572us) log.cc:844] Serialized 14 byte log entry
0709 12:19:26.875785 (+782615us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b18002dda80. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:19:27.475091 (+599306us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:27.475149 (+    58us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:27.475198 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.475292 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:27.475434 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28135,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":77,"spinlock_wait_cycles":9980160,"thread_start_us":599026,"threads_started":1}
W20260709 12:19:27.463795 29596 consensus_meta.cc:287] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.783s	user 0.008s	sys 0.008s
W20260709 12:19:27.494004 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 432956us
W20260709 12:19:27.503780 12536 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.504323 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.442312 29558 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.454252 29523 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:27.538961 29335 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:19:27.552160 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48422.6us
I20260709 12:19:27.490420 29570 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:27.553030 12536 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:27.550166 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61351.7us
W20260709 12:19:27.557546 12536 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.557801 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:27.558151 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:27.557436 12642 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:19:27.518031 29647 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260709 12:19:27.559228 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.553462 29605 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:27.560855 29626 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:27.561607 29643 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:27.561676 29650 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.071s
I20260709 12:19:27.562362 29643 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:27.563220 29643 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.26, Last appended by leader: 26, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.558704 12563 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
W20260709 12:19:27.571470 12536 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.581209 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.552788 29613 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:27.579784 12536 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.613341 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.517197 29596 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.567584 29594 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.620944 29543 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.617584 29335 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:27.706624 12570 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:27.667734 29522 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:27.717954 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 138200us
W20260709 12:19:27.719136 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.719604 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:27.732380 12536 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.733266 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.714377 29264 raft_consensus.cc:3055] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:27.746500 29264 raft_consensus.cc:740] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:27.715593 29543 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.748055 29264 consensus_queue.cc:260] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 27, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:27.710898 29620 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:27.720595 29391 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.752650 29603 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:27.745858 29613 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:19:27.758230 29653 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:19:27.758107 29644 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:27.758788 29649 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:19:27.758816 29543 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:27.743624 29594 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:27.724288 29522 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:27.770296 29662 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:19:27.795256 29522 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:27.781484 29644 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:27.799854 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.801879 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.802211 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 48 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.803288 12642 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:27.803573 29656 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:19:27.762274 29264 raft_consensus.cc:3060] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:27.804287 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.786118 29558 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.805040 12642 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:27.807173 29626 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:27.807731 29626 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:27.765460 29621 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:27.810527 29659 log.cc:927] Time spent T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:19:27.810834 29596 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.771369 29391 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:27.813094 12461 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.710s (SENT)
W20260709 12:19:27.813859 29526 consensus_meta.cc:287] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 1.183s	user 0.012s	sys 0.004s
I20260709 12:19:27.814458 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.815267 12640 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 56.
I20260709 12:19:27.816368 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.817055 12640 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:27.818032 29391 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:27.818293 12644 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6139) took 2767 ms (client timeout 3536 ms). Trace:
W20260709 12:19:27.818684 12644 rpcz_store.cc:269] 0709 12:19:25.050967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.074366 (+ 23399us) service_pool.cc:224] Handling call
0709 12:19:27.818079 (+2743713us) spinlock_profiling.cc:243] Waited 898 ms on lock 0x7b54000b4ad4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:27.818201 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066048384}
I20260709 12:19:27.819902 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.820741 12644 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:27.823951 12639 raft_consensus.cc:2410] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 44 because replica has last-logged OpId of term: 43 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 29.
W20260709 12:19:27.826776 12641 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2396) took 2573 ms (client timeout 1795 ms). Trace:
W20260709 12:19:27.827443 12641 rpcz_store.cc:269] 0709 12:19:25.253016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.867700 (+614684us) service_pool.cc:224] Handling call
0709 12:19:27.826367 (+1958667us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b540029c554. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:27.826664 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1946390400}
I20260709 12:19:27.801163 29644 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:27.828984 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.814630 29519 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:27.827071 29306 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.002s
I20260709 12:19:27.845343 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.838261 29517 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.846714 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:27.861791 12644 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 44.
W20260709 12:19:27.760037 12536 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.838397 29546 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:27.867411 29519 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.873445 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.876287 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.877547 12644 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 55: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:27.878130 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.882474 12640 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Advancing to term 59
I20260709 12:19:27.858060 29306 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:27.885213 29640 consensus_meta.cc:287] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.004s
I20260709 12:19:27.885799 29640 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.880580 29596 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.887590 29596 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.888250 29640 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.889510 12642 raft_consensus.cc:2468] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:19:27.890705 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.891969 12642 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:27.892760 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 44 candidate_status { last_received { term: 42 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:27.897734 12646 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 42 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:19:27.896322 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:27.898842 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 85786.1us
I20260709 12:19:27.899077 12644 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:27.848037 29517 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:27.910073 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:27.892069 29306 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:27.911391 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 151439us
W20260709 12:19:27.914568 12536 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.896064 29594 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:27.915427 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.890600 29546 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:27.906807 29644 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:27.927145 29600 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:27.931670 29604 raft_consensus.cc:3055] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Stepping down as leader of term 43
W20260709 12:19:27.931804 12536 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:27.932265 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.932776 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.933765 29600 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
W20260709 12:19:27.934439 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.935119 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.932417 29604 raft_consensus.cc:740] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:27.936553 12610 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35404 dropped due to backpressure: service queue is full with 50 items
I20260709 12:19:27.937493 29604 consensus_queue.cc:260] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.26, Last appended by leader: 26, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:27.938925 29604 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:27.940459 29570 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:27.945763 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.948027 29300 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.948814 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6134) took 3072 ms. Trace:
I20260709 12:19:27.948721 29300 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:27.949749 29296 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:27.950381 29296 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:27.951656 12643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6147) took 1972 ms (client timeout 2012 ms). Trace:
W20260709 12:19:27.952157 12643 rpcz_store.cc:269] 0709 12:19:25.979224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.081605 (+1102381us) service_pool.cc:224] Handling call
0709 12:19:27.951357 (+869752us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b54000a82d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:27.951552 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71550336}
W20260709 12:19:27.959224 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:27.960610 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:27.927615 29596 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:27.953366 29296 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:27.963330 13255 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2142) took 1032 ms. Trace:
W20260709 12:19:27.967469 12536 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.973701 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:27.975106 12536 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.975311 12644 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:27.975658 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:27.949358 12638 rpcz_store.cc:276] 0709 12:19:24.876224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:24.876643 (+   419us) service_pool.cc:224] Handling call
0709 12:19:24.877105 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:26.020473 (+1143368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.948441 (+1927968us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003b10d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:27.948709 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2538920320}
I20260709 12:19:27.947913 13259 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:27.978603 12642 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:27.971948 13255 rpcz_store.cc:276] 0709 12:19:26.931545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.019129 (+ 87584us) service_pool.cc:224] Handling call
0709 12:19:27.169236 (+150107us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:27.170340 (+  1104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:27.963120 (+792780us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54005e1350. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:27.963266 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343969664}
I20260709 12:19:27.946215 29546 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:27.936906 12610 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35404 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6177)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6145)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2417)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4806)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4808)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4809)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4810)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4819)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6148)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6149)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6150)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6151)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6152)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6153)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6155)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6164)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6165)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6168)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6169)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6171)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6173)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6174)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6175)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4831)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4834)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6178)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6180)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2404)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2407)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2410)
I20260709 12:19:28.003217 12640 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 59.
I20260709 12:19:27.965173 29613 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:27.961509 29519 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:27.963452 29300 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:27.982169 29594 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:19:27.992295 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:27.978241 29596 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
W20260709 12:19:28.013057 12536 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:27.980792 29626 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.046171 29570 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.052419 29517 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.055127 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:28.055428 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:28.056671 13258 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Advancing to term 55
W20260709 12:19:28.024994 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.903s (SENT)
I20260709 12:19:28.031327 29601 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:28.032553 29603 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:28.032716 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:28.035437 12637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6144) took 2057 ms (client timeout 1710 ms). Trace:
I20260709 12:19:28.011169 29546 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.046766 29626 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:28.022514 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:28.050680 29651 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.467s	user 0.004s	sys 0.000s
I20260709 12:19:28.051623 29600 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
W20260709 12:19:28.076150 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.064508 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:28.077438 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 52496.6us
I20260709 12:19:28.066048 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
W20260709 12:19:28.067075 12637 rpcz_store.cc:269] 0709 12:19:25.978362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.264229 (+285867us) service_pool.cc:224] Handling call
0709 12:19:28.035135 (+1770906us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54002a44d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:28.035332 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1574153088}
I20260709 12:19:28.069293 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:28.070560 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:28.081123 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:28.071842 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:28.084762 12643 raft_consensus.cc:2376] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51. Re-sending same reply.
I20260709 12:19:28.085954 29600 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:28.086019 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:28.087198 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:28.083283 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:28.088541 12637 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 59. Re-sending same reply.
I20260709 12:19:28.073767 29546 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.083855 13256 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:28.090651 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:28.078559 13258 raft_consensus.cc:2410] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 55 because replica has last-logged OpId of term: 54 index: 32, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 31.
I20260709 12:19:28.094226 13262 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:28.094339 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:28.097486 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:28.098784 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:28.090399 29600 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election won for term 58
I20260709 12:19:28.101042 29600 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:28.101606 29600 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:28.103761 29335 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:28.107679 29546 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:28.108445 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:28.109231 13253 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 31. Preceding OpId from leader: term: 52 index: 33. (index mismatch)
I20260709 12:19:28.109964 29603 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.119855 13256 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:28.109892 29322 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:28.071408 29596 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:28.129503 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:28.126324 12536 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:28.132117 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:28.134768 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:28.099551 29576 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:28.134302 13257 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:28.125023 29626 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:28.091017 29601 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.128707 29322 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:28.174798 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:28.182363 29636 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:28.172971 29322 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:28.173506 29644 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:19:28.202746 12565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 6269) took 2177 ms (client timeout 2762 ms). Trace:
W20260709 12:19:28.201751 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:28.140524 29596 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.172724 29630 raft_consensus.cc:2764] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election decision vote started in defunct term 49: won
I20260709 12:19:28.152195 29435 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 58.37, Next index: 38, Last known committed idx: 36, Time since last communication: 0.044s
W20260709 12:19:28.213896 12565 rpcz_store.cc:269] 0709 12:19:26.014865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.015035 (+   170us) service_pool.cc:224] Handling call
0709 12:19:28.191522 (+2176487us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:28.191796 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1772268288}
I20260709 12:19:28.167014 29361 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 50.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.038s
I20260709 12:19:28.234980 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:28.246793 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.249429 12565 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:28.250298 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:28.234476 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 105121us
I20260709 12:19:28.249944 12568 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:28.253069 12462 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:28.253679 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 154961us
I20260709 12:19:28.256669 13218 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:19:28.259933 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:28.227116 29613 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:28.262744 12462 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:28.269169 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.248113 29645 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 58 LEADER]: Stepping down as leader of term 58
I20260709 12:19:28.282841 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:28.258313 29644 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:28.291225 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6135) took 3336 ms. Trace:
I20260709 12:19:28.293111 12646 rpcz_store.cc:276] 0709 12:19:24.955070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:24.955300 (+   230us) service_pool.cc:224] Handling call
0709 12:19:24.955810 (+   510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:27.898194 (+2942384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.290974 (+392780us) spinlock_profiling.cc:243] Waited 2.94 s on lock 0x7b54000a8cd4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:28.291159 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6763216768}
I20260709 12:19:28.295943 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4810) took 3075 ms. Trace:
I20260709 12:19:28.262872 29605 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:28.278280 29645 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:28.311766 12644 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: [43.26-43.26]
I20260709 12:19:28.311590 12565 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:28.287770 29264 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.290928 29652 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:19:28.320644 12565 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
I20260709 12:19:28.321416 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6681) took 1650 ms. Trace:
I20260709 12:19:28.322012 12572 rpcz_store.cc:276] 0709 12:19:26.671092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.671258 (+   166us) service_pool.cc:224] Handling call
0709 12:19:26.687307 (+ 16049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:26.756711 (+ 69404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:26.756768 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:26.807511 (+ 50743us) log.cc:844] Serialized 26 byte log entry
0709 12:19:27.137004 (+329493us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:27.137048 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.307490 (+1170442us) spinlock_profiling.cc:243] Waited 1.63 ms on lock 0x7b54001d7c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:28.307553 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:28.307608 (+    55us) raft_consensus.cc:1685] finished
0709 12:19:28.307633 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.321334 (+ 13701us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42192,"raft.queue_time_us":37528,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":44967,"spinlock_wait_cycles":3759360,"thread_start_us":305401,"threads_started":1,"wal-append.queue_time_us":485302}
I20260709 12:19:28.324905 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4806) took 3099 ms. Trace:
I20260709 12:19:28.326004 12644 rpcz_store.cc:276] 0709 12:19:25.218674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.310996 (+3092322us) service_pool.cc:224] Handling call
0709 12:19:28.311488 (+   492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.317987 (+  6499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.318310 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.327409 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:28.275563 29405 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.034s
I20260709 12:19:28.299233 29594 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:28.334491 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4819) took 3079 ms. Trace:
I20260709 12:19:28.300827 29570 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:28.335640 12644 rpcz_store.cc:276] 0709 12:19:25.254743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.327308 (+3072565us) service_pool.cc:224] Handling call
0709 12:19:28.327724 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.334017 (+  6293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.334062 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:28.334136 (+    74us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:28.334175 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.334244 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.334411 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:28.325085 29675 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
W20260709 12:19:28.325182 29664 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.582s	user 0.005s	sys 0.000s
I20260709 12:19:28.346849 29604 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.328975 29264 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.347759 29604 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.350523 29264 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:28.350807 29601 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.351770 29603 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:28.352108 29676 log.cc:927] Time spent T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:19:28.352684 29645 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 58.37, Last appended by leader: 37, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.337086 13258 raft_consensus.cc:1240] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 43. Current term is 44. Ops: []
I20260709 12:19:28.354125 29645 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Advancing to term 59
I20260709 12:19:28.343389 29652 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:28.337440 13257 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:28.315538 29679 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:28.358757 29652 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.359772 29679 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.336846 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:28.358799 29630 raft_consensus.cc:2804] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:28.362448 29630 raft_consensus.cc:697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:28.363466 29630 consensus_queue.cc:237] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 49.31, Last appended by leader: 31, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.374431 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.374607 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.375437 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:28.376140 12642 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 58. Current term is 59. Ops: []
I20260709 12:19:28.376866 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6149) took 2397 ms. Trace:
I20260709 12:19:28.379652 12642 rpcz_store.cc:276] 0709 12:19:25.979806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.375652 (+2395846us) service_pool.cc:224] Handling call
0709 12:19:28.375958 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.376547 (+   589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.376763 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.377853 29679 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:28.379132 12644 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
I20260709 12:19:28.389808 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6148) took 2410 ms. Trace:
I20260709 12:19:28.390383 12644 rpcz_store.cc:276] 0709 12:19:25.979526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.378438 (+2398912us) service_pool.cc:224] Handling call
0709 12:19:28.378870 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.389367 (+ 10497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.389717 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.392532 12644 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 29. Preceding OpId from leader: term: 44 index: 29. (term mismatch)
I20260709 12:19:28.393067 12644 pending_rounds.cc:85] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 28
I20260709 12:19:28.394479 12644 pending_rounds.cc:107] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 43.29
I20260709 12:19:28.396870 12566 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
I20260709 12:19:28.405941 12572 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:19:28.406435 12644 raft_consensus.cc:2917] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Aborting config change with OpId 43.29: Aborted: Op aborted by new leader
I20260709 12:19:28.402741 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.409226 12644 raft_consensus.cc:2889] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:28.410063 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.305226 29605 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.338145 29596 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.415565 12571 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
W20260709 12:19:28.417990 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.421429 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6151) took 2441 ms. Trace:
I20260709 12:19:28.391347 29570 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:28.413857 29679 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.389214 29681 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:28.405113 29335 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:28.422081 12644 rpcz_store.cc:276] 0709 12:19:25.980255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.391502 (+2411247us) service_pool.cc:224] Handling call
0709 12:19:28.392014 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.421052 (+ 29038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.421338 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.406850 29601 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:28.399017 29361 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.342891 29594 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.435830 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:28.431792 29570 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:28.481954 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 46244.5us
I20260709 12:19:28.499095 29594 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:28.502404 29686 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
W20260709 12:19:28.503453 29688 log.cc:927] Time spent T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:19:28.504531 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4809) took 3284 ms. Trace:
I20260709 12:19:28.502311 29322 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.505333 12643 rpcz_store.cc:276] 0709 12:19:25.219820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.087113 (+2867293us) service_pool.cc:224] Handling call
0709 12:19:28.087519 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.091077 (+  3558us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:19:28.091112 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:28.125928 (+ 34816us) log.cc:844] Serialized 26 byte log entry
0709 12:19:28.246523 (+120595us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:19:28.246582 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.246681 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:28.504044 (+257363us) raft_consensus.cc:1685] finished
0709 12:19:28.504075 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.504442 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4603,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":14549,"thread_start_us":81184,"threads_started":1,"wal-append.queue_time_us":155305}
I20260709 12:19:28.508213 29322 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:28.508598 12643 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
I20260709 12:19:28.509462 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6153) took 2524 ms. Trace:
I20260709 12:19:28.509871 12643 rpcz_store.cc:276] 0709 12:19:25.984738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.507489 (+2522751us) service_pool.cc:224] Handling call
0709 12:19:28.508126 (+   637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.509166 (+  1040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.509397 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.475786 29601 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.511339 12643 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: [49.28-49.28]
I20260709 12:19:28.511209 29600 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.512076 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6155) took 2513 ms. Trace:
I20260709 12:19:28.512678 12563 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 57. Current term is 58. Ops: []
I20260709 12:19:28.487145 29361 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.516136 29361 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.491171 29604 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:28.518505 29600 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:28.524044 29605 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.476387 29681 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.530380 29681 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:28.512432 12643 rpcz_store.cc:276] 0709 12:19:25.998460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.510668 (+2512208us) service_pool.cc:224] Handling call
0709 12:19:28.511095 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.511778 (+   683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.512022 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.534762 29626 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:28.535487 29626 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:28.531548 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45701us
I20260709 12:19:28.539252 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4808) took 3319 ms. Trace:
I20260709 12:19:28.539858 12640 rpcz_store.cc:276] 0709 12:19:25.219512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.009897 (+2790385us) service_pool.cc:224] Handling call
0709 12:19:28.010291 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.010537 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:19:28.235321 (+224784us) log.cc:844] Serialized 14 byte log entry
0709 12:19:28.496224 (+260903us) log.cc:844] Serialized 14 byte log entry
0709 12:19:28.537831 (+ 41607us) log.cc:844] Serialized 14 byte log entry
0709 12:19:28.538205 (+   374us) log.cc:844] Serialized 14 byte log entry
0709 12:19:28.538793 (+   588us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:28.538855 (+    62us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:28.538927 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.539015 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.539175 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":327161,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":5334,"raft.queue_time_us":91,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":108,"thread_start_us":413068,"threads_started":2,"wal-append.queue_time_us":284449}
I20260709 12:19:28.543031 12640 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
W20260709 12:19:28.546928 13259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50712 (request call id 2141) took 1817 ms (client timeout 1663 ms). Trace:
W20260709 12:19:28.547596 13259 rpcz_store.cc:269] 0709 12:19:26.729681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:26.732008 (+  2327us) service_pool.cc:224] Handling call
0709 12:19:28.546619 (+1814611us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400537354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:28.546813 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2504185216}
I20260709 12:19:28.552760 12570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6687) took 1365 ms. Trace:
I20260709 12:19:28.557960 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:28.559190 12641 raft_consensus.cc:2393] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:28.553382 12570 rpcz_store.cc:276] 0709 12:19:27.187277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:27.187399 (+   122us) service_pool.cc:224] Handling call
0709 12:19:27.247748 (+ 60349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:27.711268 (+463520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.552375 (+841107us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54001b17d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:28.552658 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1049473280}
I20260709 12:19:28.531666 12567 raft_consensus.cc:2410] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 54 because replica has last-logged OpId of term: 52 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 30.
I20260709 12:19:28.571636 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6157) took 2569 ms. Trace:
I20260709 12:19:28.573752 12640 rpcz_store.cc:276] 0709 12:19:26.002817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.542466 (+2539649us) service_pool.cc:224] Handling call
0709 12:19:28.542862 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.543391 (+   529us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.571537 (+ 28146us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:28.622490 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.626748 12641 raft_consensus.cc:1240] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: [49.28-49.28]
I20260709 12:19:28.487331 29464 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:28.627766 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6163) took 2616 ms. Trace:
I20260709 12:19:28.632241 12641 rpcz_store.cc:276] 0709 12:19:26.011216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.622408 (+2611192us) service_pool.cc:224] Handling call
0709 12:19:28.626480 (+  4072us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.627424 (+   944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.627690 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:28.638804 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 76243.6us
W20260709 12:19:28.639643 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 102236us
I20260709 12:19:28.641361 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:28.439994 29679 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:28.674199 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.643497 29464 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:28.643502 29620 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:19:28.696848 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.556137 29601 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:28.643090 29517 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:28.708070 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:28.665695 29546 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:28.695582 29694 log.cc:927] Time spent T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:19:28.678334 12460 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:28.700096 29464 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:28.721661 12536 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:28.725905 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47694.1us
I20260709 12:19:28.668988 29554 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.043s
I20260709 12:19:28.728672 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:28.730264 29687 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:19:28.713913 29517 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.717092 29546 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:28.727415 12536 leader_election.cc:336] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:28.679801 29526 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.707618 29644 consensus_queue.cc:1059] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.035s
I20260709 12:19:28.744367 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:28.755713 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6177) took 1494 ms. Trace:
I20260709 12:19:28.756155 12645 rpcz_store.cc:276] 0709 12:19:27.261221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.744081 (+1482860us) service_pool.cc:224] Handling call
0709 12:19:28.755640 (+ 11559us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.744728 29554 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.745069 29546 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:28.679343 29405 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.049s
I20260709 12:19:28.770133 29405 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:28.748035 29517 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:28.741284 12460 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:28.764755 12639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2401) took 3510 ms (client timeout 1695 ms). Trace:
I20260709 12:19:28.717975 29361 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.048s
I20260709 12:19:28.766285 12645 raft_consensus.cc:1217] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Deduplicated request from leader. Original: 48.29->[50.30-54.32]   Dedup: 50.30->[52.31-54.32]
I20260709 12:19:28.774677 12645 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 54. Current term is 55. Ops: [50.30-54.32]
I20260709 12:19:28.775521 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6165) took 2763 ms. Trace:
I20260709 12:19:28.775877 12645 rpcz_store.cc:276] 0709 12:19:26.011949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.757013 (+2745064us) service_pool.cc:224] Handling call
0709 12:19:28.766045 (+  9032us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:19:28.775164 (+  9119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.775448 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:28.767750 29684 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.325s	user 0.002s	sys 0.000s
I20260709 12:19:28.738657 29587 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election won for term 47
W20260709 12:19:28.772369 12639 rpcz_store.cc:269] 0709 12:19:25.254582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:25.579454 (+324872us) service_pool.cc:224] Handling call
0709 12:19:28.755952 (+3176498us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b54003b01d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:28.764628 (+  8676us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4999393152}
I20260709 12:19:28.798499 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:28.806141 13255 raft_consensus.cc:1275] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 45 index: 28. (index mismatch)
I20260709 12:19:28.819029 12645 raft_consensus.cc:1240] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:28.762379 29526 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.822777 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6168) took 2478 ms. Trace:
I20260709 12:19:28.823413 12645 rpcz_store.cc:276] 0709 12:19:26.344160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.776865 (+2432705us) service_pool.cc:224] Handling call
0709 12:19:28.818719 (+ 41854us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.822499 (+  3780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.822717 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.777669 29405 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Leader election won for term 57
I20260709 12:19:28.791462 29587 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:28.832521 29587 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:28.807204 29605 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.780432 29641 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.054s
W20260709 12:19:28.837579 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.009s	sys 0.012s
I20260709 12:19:28.774049 29554 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:28.847242 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.010s	sys 0.012s
I20260709 12:19:28.834754 29405 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:28.872391 12645 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 30. Preceding OpId from leader: term: 55 index: 31. (index mismatch)
I20260709 12:19:28.873540 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6171) took 2051 ms. Trace:
I20260709 12:19:28.874118 12645 rpcz_store.cc:276] 0709 12:19:26.822545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.824482 (+2001937us) service_pool.cc:224] Handling call
0709 12:19:28.865594 (+ 41112us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.873186 (+  7592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.873465 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.876189 12537 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:28.838629 29605 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:28.889182 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6172) took 2066 ms. Trace:
I20260709 12:19:28.901849 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:28.886277 29696 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:28.889873 12645 rpcz_store.cc:276] 0709 12:19:26.822881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.875752 (+2052871us) service_pool.cc:224] Handling call
0709 12:19:28.887394 (+ 11642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.888695 (+  1301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.888737 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:28.888799 (+    62us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:28.888851 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.888938 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.889103 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.944883 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.948611 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.949859 12565 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:28.911343 12611 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:28.955312 12571 raft_consensus.cc:1275] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 42 index: 23. Preceding OpId from leader: term: 46 index: 25. (index mismatch)
W20260709 12:19:28.956328 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 45024.9us
I20260709 12:19:28.956979 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:28.958756 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.978178 12567 raft_consensus.cc:1275] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:19:28.985340 12645 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 30. Preceding OpId from leader: term: 54 index: 31. (index mismatch)
I20260709 12:19:28.986716 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6173) took 2145 ms. Trace:
I20260709 12:19:28.987260 12645 rpcz_store.cc:276] 0709 12:19:26.841664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.921119 (+2079455us) service_pool.cc:224] Handling call
0709 12:19:28.984426 (+ 63307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.986329 (+  1903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.986637 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:28.989562 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:28.990753 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.991707 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.992434 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 42 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.992704 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:28.993654 12570 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 55 because replica has last-logged OpId of term: 54 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 30.
I20260709 12:19:28.910261 29517 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:28.996516 12568 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:28.998991 13218 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.006s	sys 0.018s
I20260709 12:19:28.938442 29546 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:19:29.003147 13218 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.006s	sys 0.018s
I20260709 12:19:28.957708 29600 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:29.023198 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 766560us
I20260709 12:19:29.025192 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:29.030774 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2400) took 3776 ms. Trace:
I20260709 12:19:29.001299 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:29.032006 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.039152 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:28.976174 29681 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.042s
I20260709 12:19:29.047197 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.055559 12571 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:29.033730 12646 rpcz_store.cc:276] 0709 12:19:25.254278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.299073 (+3044795us) service_pool.cc:224] Handling call
0709 12:19:28.303852 (+  4779us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.304152 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:19:28.644382 (+340230us) log.cc:844] Serialized 14 byte log entry
0709 12:19:29.030001 (+385619us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:29.030345 (+   344us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:29.030416 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.030520 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.030696 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107,"prepare.queue_time_us":43722,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"thread_start_us":656728,"threads_started":2}
I20260709 12:19:29.020622 29546 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:28.938942 29405 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 55.29, Last appended by leader: 29, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.063994 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:29.044474 29695 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.169s	user 0.002s	sys 0.000s
W20260709 12:19:29.071183 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 286797us
I20260709 12:19:29.075045 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.090608 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.002990 29517 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.090574 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.068070 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:29.097702 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.098373 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:29.099980 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.020917 29600 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.040035 29575 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20260709 12:19:29.108441 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:29.109139 12645 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:29.119760 29517 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:29.120559 29517 consensus_queue.cc:1048] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.004s
I20260709 12:19:29.080817 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:29.154124 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.094694 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.092483 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:29.071080 29526 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:29.073443 29630 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:29.171408 29630 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:29.171864 29630 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:29.127816 29605 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:29.128293 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:29.130674 29700 log.cc:927] Time spent T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:19:29.131335 29681 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:29.102174 29691 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:29.135563 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.138139 29546 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.29, Last appended by leader: 29, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:28.296537 12637 rpcz_store.cc:276] 0709 12:19:25.220133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.089208 (+2869075us) service_pool.cc:224] Handling call
0709 12:19:28.089551 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.089868 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 32
0709 12:19:28.205288 (+115420us) log.cc:844] Serialized 14 byte log entry
0709 12:19:28.294841 (+ 89553us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:28.294925 (+    84us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:19:28.294990 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.295069 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:28.295867 (+   798us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":829,"thread_start_us":176138,"threads_started":2}
W20260709 12:19:29.148578 29698 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:19:29.130846 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6156) took 3132 ms. Trace:
I20260709 12:19:29.089871 29703 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:29.151801 29600 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:29.106964 29554 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:29.107432 29524 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Leader election won for term 59
I20260709 12:19:29.103999 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.167028 12645 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:29.169904 13262 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:29.170506 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 78103us
I20260709 12:19:29.173959 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.175482 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.178198 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34824 (request call id 6145) took 3199 ms. Trace:
I20260709 12:19:29.194469 12646 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:29.194340 12638 rpcz_store.cc:276] 0709 12:19:25.978734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.135143 (+3156409us) service_pool.cc:224] Handling call
0709 12:19:29.178106 (+ 42963us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:29.183785 11310 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 12522)
I20260709 12:19:29.182251 12643 rpcz_store.cc:276] 0709 12:19:25.998812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.532521 (+2533709us) service_pool.cc:224] Handling call
0709 12:19:28.532915 (+   394us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:28.533620 (+   705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.533673 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:28.534294 (+   621us) log.cc:844] Serialized 52 byte log entry
0709 12:19:28.881918 (+347624us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:28.881965 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.882066 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:29.130523 (+248457us) raft_consensus.cc:1685] finished
0709 12:19:29.130553 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.130765 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":347153,"threads_started":1,"wal-append.queue_time_us":528054}
W20260709 12:19:29.180107 29570 consensus_meta.cc:287] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.686s	user 0.001s	sys 0.012s
W20260709 12:19:29.189136 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 492890us
W20260709 12:19:29.182233 13217 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.922s	user 0.012s	sys 0.032s
W20260709 12:19:29.197665 13217 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.013s	sys 0.033s
I20260709 12:19:29.199306 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:29.177392 29652 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:29.184374 29636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.000s	sys 0.037s
I20260709 12:19:29.213059 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.213963 13258 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:29.215843 13258 raft_consensus.cc:1240] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:29.189108 29526 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.228034 12566 raft_consensus.cc:3055] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:29.234063 12566 raft_consensus.cc:740] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:29.239022 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:29.232223 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:29.238723 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.192647 29703 raft_consensus.cc:697] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:29.211833 29636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.000s	sys 0.038s
I20260709 12:19:29.242718 29605 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:29.243251 29605 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:29.243645 29605 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:29.229091 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:29.235325 12566 consensus_queue.cc:260] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 50.32, Last appended by leader: 32, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.251906 12566 raft_consensus.cc:3060] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:29.225040 29576 raft_consensus.cc:670] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:29.181510 29681 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.254082 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:29.258179 12646 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:29.186154 29464 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.182762 29587 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.230310 29524 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:29.210675 29606 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.047s
I20260709 12:19:29.277783 29641 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:29.279501 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6152) took 3295 ms. Trace:
I20260709 12:19:29.280004 29554 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:29.280963 29626 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.745s	user 0.016s	sys 0.002s
I20260709 12:19:29.249105 29652 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.033s
I20260709 12:19:29.283004 12564 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 25. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:19:29.204080 29107 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.050s
I20260709 12:19:29.205416 29570 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.286813 29300 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:29.287660 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.288961 29681 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:29.289404 12565 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:29.241356 29526 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.290795 29703 consensus_queue.cc:237] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 50.31, Last appended by leader: 31, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.293462 29712 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:19:29.247377 29459 consensus_queue.cc:1059] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.016s
W20260709 12:19:29.294965 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.10739e+06us
W20260709 12:19:29.253520 12642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.008s	sys 0.009s
W20260709 12:19:29.308708 12641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.018s	sys 0.029s
I20260709 12:19:29.311470 12644 rpcz_store.cc:276] 0709 12:19:25.984395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.441346 (+2456951us) service_pool.cc:224] Handling call
0709 12:19:28.441803 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.475304 (+ 33501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.475377 (+    73us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:28.504236 (+ 28859us) log.cc:844] Serialized 26 byte log entry
0709 12:19:28.756253 (+252017us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:19:28.756299 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:28.756375 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:29.279187 (+522812us) raft_consensus.cc:1685] finished
0709 12:19:29.279214 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.279411 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2919,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":9363,"thread_start_us":251528,"threads_started":1,"wal-append.queue_time_us":490654}
I20260709 12:19:29.330041 12646 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:29.344425 12536 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 94 similar messages]
I20260709 12:19:29.379261 13253 raft_consensus.cc:1240] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: [52.32-52.32]
I20260709 12:19:29.378337 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:29.387037 12642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.009s	sys 0.013s
I20260709 12:19:29.388221 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6150) took 3408 ms. Trace:
W20260709 12:19:29.391093 12610 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.391899 12610 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:29.393168 12641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.019s	sys 0.030s
W20260709 12:19:29.394517 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 312103us
W20260709 12:19:29.397439 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.388681 12642 rpcz_store.cc:276] 0709 12:19:25.980005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.380775 (+2400770us) service_pool.cc:224] Handling call
0709 12:19:28.381194 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:28.381488 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:19:28.381990 (+   502us) log.cc:844] Serialized 14 byte log entry
0709 12:19:28.696774 (+314784us) log.cc:844] Serialized 14 byte log entry
0709 12:19:29.387676 (+690902us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:29.387773 (+    97us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:29.387855 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.387957 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.388142 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5736,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":104,"raft.queue_time_us":152135,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":980038,"threads_started":2,"wal-append.queue_time_us":457496}
W20260709 12:19:29.399017 12609 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:29.402383 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 297873us
I20260709 12:19:29.299870 29613 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:29.419358 12610 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.420246 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:29.448235 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45580.9us
I20260709 12:19:29.460556 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:29.346882 29107 consensus_queue.cc:1243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 45.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260709 12:19:29.377589 29526 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:29.466463 12642 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:29.468534 12610 leader_election.cc:336] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.469122 12610 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:29.470611 12610 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:29.463213 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 234253us
I20260709 12:19:29.399780 29708 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:29.487283 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 57 candidate_status { last_received { term: 56 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.488677 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:29.520656 12610 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:29.522235 12610 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:29.566699 12612 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:29.503211 12644 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:29.631590 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.640205 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.384317 29644 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 57.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.081s
I20260709 12:19:29.463105 29255 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.688892 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:29.683820 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.692013 12642 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:29.692153 12646 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:29.341907 29626 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.463811 29606 raft_consensus.cc:3055] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:29.670310 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:29.684366 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 164500us
I20260709 12:19:29.703169 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.504925 29526 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.704887 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 42 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:29.705055 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 182853us
I20260709 12:19:29.456357 12383 maintenance_manager.cc:643] P 2d10571632e7445d99e8f1c750c96f70: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 3.855s	user 0.124s	sys 0.080s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":24757,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":683307,"dirs.run_cpu_time_us":762,"dirs.run_wall_time_us":19277,"lbm_read_time_us":42449,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":21,"lbm_write_time_us":1826,"lbm_writes_lt_1ms":10,"peak_mem_usage":257857,"reinsert_count":0,"thread_start_us":731103,"threads_started":3,"update_count":166,"wal-append.queue_time_us":162120,"wal-append.run_cpu_time_us":912,"wal-append.run_wall_time_us":1050924}
I20260709 12:19:29.705996 29435 raft_consensus.cc:3055] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:29.706909 29601 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Stepping down as leader of term 50
W20260709 12:19:29.716845 29706 log.cc:927] Time spent T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.474s	user 0.011s	sys 0.000s
I20260709 12:19:29.718357 29517 consensus_queue.cc:1048] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:19:29.725334 13253 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
W20260709 12:19:29.742565 12610 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.726507 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:29.743072 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:29.454634 29613 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.498971 29715 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election lost for term 59. Reason: could not achieve majority
I20260709 12:19:29.736286 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.751459 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:29.734943 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.737241 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.682901 29707 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.684082 29641 raft_consensus.cc:3055] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:29.730625 29716 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.765799 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:29.778121 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:29.795723 12646 raft_consensus.cc:2410] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 55 because replica has last-logged OpId of term: 53 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 29.
I20260709 12:19:29.752552 12565 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:29.782058 12642 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 50 because replica has last-logged OpId of term: 49 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 26.
I20260709 12:19:29.736706 29601 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:29.825336 12642 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 50 index: 21. Preceding OpId from leader: term: 54 index: 23. (index mismatch)
I20260709 12:19:29.695634 29606 raft_consensus.cc:740] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:29.826293 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4831) took 2899 ms. Trace:
I20260709 12:19:29.787838 12644 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:29.785245 12645 raft_consensus.cc:1240] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:29.831398 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6174) took 2950 ms. Trace:
I20260709 12:19:29.831944 12645 rpcz_store.cc:276] 0709 12:19:26.881336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.784515 (+2903179us) service_pool.cc:224] Handling call
0709 12:19:29.784986 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.831044 (+ 46058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.831325 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.804495 29708 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:29.750280 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 405843us
I20260709 12:19:29.834462 29600 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:29.835400 12536 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.835162 29600 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:29.836539 12536 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:29.837177 13218 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.186s (SENT)
W20260709 12:19:29.838594 13218 leader_election.cc:336] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.195s (SENT)
I20260709 12:19:29.839104 12644 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 51 index: 28. Preceding OpId from leader: term: 53 index: 29. (index mismatch)
I20260709 12:19:29.839856 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6178) took 2461 ms. Trace:
I20260709 12:19:29.840322 12644 rpcz_store.cc:276] 0709 12:19:27.378891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.838332 (+2459441us) service_pool.cc:224] Handling call
0709 12:19:29.838664 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.839540 (+   876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.839744 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:29.780391 12610 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.507103 29630 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.842485 13256 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:29.844200 29630 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:29.736543 29435 raft_consensus.cc:740] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:29.846483 29435 consensus_queue.cc:260] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 52.32, Last appended by leader: 32, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.848114 29435 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:29.837188 12536 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.854535 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:29.641337 29524 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 58.37, Last appended by leader: 37, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.702438 29255 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.858349 12642 rpcz_store.cc:276] 0709 12:19:26.927373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.801606 (+2874233us) service_pool.cc:224] Handling call
0709 12:19:29.802041 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.825977 (+ 23936us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.826232 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:29.857798 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 77424.1us
I20260709 12:19:29.861171 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6175) took 2946 ms. Trace:
I20260709 12:19:29.741770 29526 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.787360 29707 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.743573 29652 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.061s
I20260709 12:19:29.785365 29641 raft_consensus.cc:740] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:29.865805 12610 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.825531 29703 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:29.902992 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:29.910259 12644 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 43 index: 25. (index mismatch)
I20260709 12:19:29.865761 12646 rpcz_store.cc:276] 0709 12:19:26.914978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.797206 (+2882228us) service_pool.cc:224] Handling call
0709 12:19:29.860393 (+ 63187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.860668 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.860709 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:29.860775 (+    66us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:29.860824 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.860916 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:29.861082 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.911415 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6180) took 2526 ms. Trace:
I20260709 12:19:29.825867 29681 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.915982 12644 rpcz_store.cc:276] 0709 12:19:27.384979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.841358 (+2456379us) service_pool.cc:224] Handling call
0709 12:19:29.859251 (+ 17893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.910987 (+ 51736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.911336 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:19:29.921664 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55875.8us
W20260709 12:19:29.867220 29705 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.409s	user 0.000s	sys 0.001s
W20260709 12:19:29.937645 12610 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:29.856420 29575 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:29.945992 12646 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 43 index: 30. Preceding OpId from leader: term: 44 index: 30. (term mismatch)
I20260709 12:19:29.946452 12646 pending_rounds.cc:85] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:19:29.946708 12646 pending_rounds.cc:107] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 43.30
I20260709 12:19:29.946950 12646 raft_consensus.cc:2889] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:29.913523 29718 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.120s	user 0.001s	sys 0.001s
W20260709 12:19:29.920469 29595 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.016s	sys 0.028s
I20260709 12:19:29.871969 29613 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:29.969600 12462 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.023s	sys 0.022s
W20260709 12:19:29.978516 12462 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.024s	sys 0.022s
I20260709 12:19:29.867262 29570 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:29.901218 29459 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:29.856418 29546 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:29.944689 29652 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 53.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:19:29.902168 29405 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.072s
I20260709 12:19:29.864400 29107 consensus_queue.cc:1059] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 46.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.088s
I20260709 12:19:30.035145 29679 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.035758 29300 raft_consensus.cc:3055] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:30.035938 29679 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:30.036406 29300 raft_consensus.cc:740] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:30.036679 29679 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.952512 29707 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:29.953521 29641 consensus_queue.cc:260] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.28, Last appended by leader: 28, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:30.038290 29600 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.038764 29641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:30.038993 29600 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:30.039691 29600 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.943317 29606 consensus_queue.cc:260] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.28, Last appended by leader: 28, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.954880 29255 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:29.958418 29605 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.714s	user 0.011s	sys 0.000s
I20260709 12:19:29.957986 29575 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.037438 29300 consensus_queue.cc:260] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:30.044781 29652 consensus_queue.cc:1059] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 49.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:30.044979 29605 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.876035 29601 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 50.30, Last appended by leader: 30, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.046044 29300 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:29.939484 29681 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:29.963011 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2407) took 2346 ms. Trace:
I20260709 12:19:30.048156 12646 rpcz_store.cc:276] 0709 12:19:27.617121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.915606 (+2298485us) service_pool.cc:224] Handling call
0709 12:19:29.945621 (+ 30015us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.962750 (+ 17129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.962951 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:29.963963 12644 raft_consensus.cc:1275] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 50 index: 34. Preceding OpId from leader: term: 52 index: 35. (index mismatch)
I20260709 12:19:29.874655 29587 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:29.876268 29724 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.052958 12646 raft_consensus.cc:1240] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:30.053985 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4845) took 1038 ms. Trace:
I20260709 12:19:30.054508 12646 rpcz_store.cc:276] 0709 12:19:29.015545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.049134 (+1033589us) service_pool.cc:224] Handling call
0709 12:19:30.049479 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.053603 (+  4124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.053904 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:30.056537 12646 raft_consensus.cc:1240] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 50. Ops: []
W20260709 12:19:29.913848 29709 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.272s	user 0.000s	sys 0.002s
W20260709 12:19:29.967392 29595 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.017s	sys 0.028s
I20260709 12:19:29.829739 29716 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:19:29.941891 12610 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.018898 29264 consensus_queue.cc:1059] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 54.32, Next index: 33, Last known committed idx: 30, Time since last communication: 0.018s
I20260709 12:19:30.019815 29546 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:29.994731 29613 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:30.021530 29459 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:30.022503 29526 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.024582 12638 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:30.068915 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4835) took 1110 ms. Trace:
I20260709 12:19:30.069356 12638 rpcz_store.cc:276] 0709 12:19:28.958371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.023893 (+1065522us) service_pool.cc:224] Handling call
0709 12:19:30.024322 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.068500 (+ 44178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.068821 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:30.071112 12638 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
I20260709 12:19:30.071830 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6182) took 1025 ms. Trace:
I20260709 12:19:30.072211 12638 rpcz_store.cc:276] 0709 12:19:29.046117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.070317 (+1024200us) service_pool.cc:224] Handling call
0709 12:19:30.070746 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.071475 (+   729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.071761 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:30.028198 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:30.074290 13254 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:30.030268 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.032662 29712 raft_consensus.cc:3055] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Stepping down as leader of term 57
I20260709 12:19:30.079139 29681 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:29.937053 29721 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:29.960287 29626 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:30.058138 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:30.062953 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 121087us
W20260709 12:19:30.098234 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.063215 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4850) took 1022 ms. Trace:
I20260709 12:19:30.099400 12646 rpcz_store.cc:276] 0709 12:19:29.040403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.056003 (+1015600us) service_pool.cc:224] Handling call
0709 12:19:30.056381 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.062850 (+  6469us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.063136 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:30.064026 29606 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:30.046736 29601 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:30.077071 12638 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:30.078557 29711 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.035332 29734 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:30.105250 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 50741.7us
I20260709 12:19:30.111749 12646 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 26. Preceding OpId from leader: term: 49 index: 27. (index mismatch)
I20260709 12:19:30.097548 29636 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.118731 29255 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:30.122931 29636 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.123210 29255 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.110332 29605 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.123723 29575 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:30.079228 29712 raft_consensus.cc:740] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:30.125684 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6162) took 4115 ms. Trace:
I20260709 12:19:30.128991 12640 rpcz_store.cc:276] 0709 12:19:26.009948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.576297 (+2566349us) service_pool.cc:224] Handling call
0709 12:19:28.576689 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.631841 (+ 55152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.631874 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:28.669844 (+ 37970us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.186817 (+516973us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:29.186900 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.187009 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.125343 (+938334us) raft_consensus.cc:1685] finished
0709 12:19:30.125377 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.125578 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48115,"raft.queue_time_us":66234,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":8721,"thread_start_us":474253,"threads_started":1,"wal-append.queue_time_us":660786}
I20260709 12:19:30.099184 29613 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:30.131368 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.134614 12640 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:30.062111 29724 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.098312 29681 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:30.111678 29600 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:30.142163 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 304942us
W20260709 12:19:30.163106 12566 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.003s	sys 0.024s
W20260709 12:19:30.163544 12566 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.003s	sys 0.024s
I20260709 12:19:30.124781 29626 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:30.138515 29526 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:30.178122 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 444088us
I20260709 12:19:30.195343 12638 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 28. Preceding OpId from leader: term: 49 index: 29. (index mismatch)
W20260709 12:19:30.196195 11310 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 12522)
I20260709 12:19:30.207155 12638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6186) took 1114 ms. Trace:
I20260709 12:19:30.207659 12638 rpcz_store.cc:276] 0709 12:19:29.092912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.073126 (+980214us) service_pool.cc:224] Handling call
0709 12:19:30.073543 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.206830 (+133287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.207084 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:30.209602 12640 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
I20260709 12:19:30.210541 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6191) took 1096 ms. Trace:
I20260709 12:19:30.210927 12640 rpcz_store.cc:276] 0709 12:19:29.114595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.130654 (+1016059us) service_pool.cc:224] Handling call
0709 12:19:30.131024 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.210163 (+ 79139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.210466 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:30.211247 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.212091 12638 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:30.212237 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.218163 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.214133 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.124547 29711 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:30.219949 12609 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.017s	sys 0.014s
I20260709 12:19:30.137977 29721 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.243021 29721 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:30.236997 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:30.246039 12640 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:30.237346 12609 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.018s	sys 0.014s
I20260709 12:19:30.241598 29459 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.242136 29711 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.140938 29613 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.245546 29721 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:30.253042 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 778710us
I20260709 12:19:30.256405 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6190) took 1142 ms. Trace:
I20260709 12:19:30.256721 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.257817 29721 consensus_queue.cc:1048] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:19:30.259011 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.238379 12638 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:30.259634 12643 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 55.
I20260709 12:19:30.260366 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:30.261529 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:30.262240 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2410) took 2644 ms. Trace:
I20260709 12:19:30.262583 29645 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.263244 29645 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.264631 13257 raft_consensus.cc:1240] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:30.265195 29739 raft_consensus.cc:3055] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Stepping down as leader of term 54
I20260709 12:19:30.269814 29739 raft_consensus.cc:740] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:30.270979 29714 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.028s	user 0.000s	sys 0.006s
I20260709 12:19:30.270855 29739 consensus_queue.cc:260] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 54.32, Last appended by leader: 32, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.272372 29739 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:30.272769 29255 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.260426 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:30.274350 12643 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:30.274525 29613 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.275291 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.262605 12644 rpcz_store.cc:276] 0709 12:19:27.618208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.918630 (+2300422us) service_pool.cc:224] Handling call
0709 12:19:29.919103 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.049814 (+130711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.261978 (+212164us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b54003c0354. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:30.262182 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102391168}
I20260709 12:19:30.277966 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6164) took 4266 ms. Trace:
I20260709 12:19:30.278331 12641 rpcz_store.cc:276] 0709 12:19:26.011538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.633286 (+2621748us) service_pool.cc:224] Handling call
0709 12:19:28.673229 (+ 39943us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.393819 (+720590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.393870 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.415580 (+ 21710us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.966909 (+551329us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:19:29.966959 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.967055 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.277721 (+310666us) raft_consensus.cc:1685] finished
0709 12:19:30.277742 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.277896 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93494,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":118,"thread_start_us":1012808,"threads_started":2,"wal-append.queue_time_us":679665}
I20260709 12:19:30.278667 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.282342 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:30.268676 29464 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.282609 12644 raft_consensus.cc:2410] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 49 because replica has last-logged OpId of term: 48 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 23.
W20260709 12:19:30.288141 29726 log.cc:927] Time spent T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.384s	user 0.002s	sys 0.000s
I20260709 12:19:30.265597 29645 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.283607 29724 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.276579 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.127619 29734 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.273819 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6169) took 3458 ms. Trace:
I20260709 12:19:30.291353 13259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50708 (request call id 2359) took 1111 ms. Trace:
I20260709 12:19:30.257104 12646 rpcz_store.cc:276] 0709 12:19:29.114334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.100583 (+986249us) service_pool.cc:224] Handling call
0709 12:19:30.101018 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.121671 (+ 20653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.256052 (+134381us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b54002e5c54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:30.256319 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23735680}
I20260709 12:19:30.305531 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:30.308403 13254 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:30.308859 12643 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:30.293053 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.294425 12640 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:30.261050 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:19:30.317341 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.319378 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:30.320144 12646 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:30.322749 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:30.330933 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.332873 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.342424 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.343155 12640 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:30.344321 12640 pending_rounds.cc:85] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 23
I20260709 12:19:30.344617 12640 pending_rounds.cc:107] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 45.24
I20260709 12:19:30.344863 12640 raft_consensus.cc:2889] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:30.164397 29742 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.168s
I20260709 12:19:30.305822 13259 rpcz_store.cc:276] 0709 12:19:29.181127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.181302 (+   175us) service_pool.cc:224] Handling call
0709 12:19:29.181724 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.224975 (+ 43251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.225020 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.238355 (+ 13335us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.673375 (+435020us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:29.673418 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.709257 (+ 35839us) spinlock_profiling.cc:243] Waited 7.71 ms on lock 0x7b540056a9d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:29.709321 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.287381 (+578060us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:30.291121 (+  3740us) raft_consensus.cc:1685] finished
0709 12:19:30.291147 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.291289 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32600,"raft.queue_time_us":35602,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":12377,"spinlock_wait_cycles":30211200,"thread_start_us":434557,"threads_started":1,"wal-append.queue_time_us":663074}
W20260709 12:19:30.318169 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.173979 29681 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.367867 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.368848 29605 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:30.374745 12638 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 55.
I20260709 12:19:30.374974 29605 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:30.373857 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:30.278597 29636 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:30.369112 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.376821 29681 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:30.377076 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 53348.9us
I20260709 12:19:30.370064 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6723) took 1073 ms. Trace:
I20260709 12:19:30.377041 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:30.379136 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 62626.8us
I20260709 12:19:30.380013 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:30.380008 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.304706 12639 rpcz_store.cc:276] 0709 12:19:26.815792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:28.822626 (+2006834us) service_pool.cc:224] Handling call
0709 12:19:28.831671 (+  9045us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:28.836704 (+  5033us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:28.836756 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:28.837436 (+   680us) log.cc:844] Serialized 26 byte log entry
0709 12:19:29.052092 (+214656us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:29.088709 (+ 36617us) log.cc:844] Serialized 14 byte log entry
0709 12:19:29.439028 (+350319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:29.439164 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.270265 (+831101us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:30.273469 (+  3204us) raft_consensus.cc:1685] finished
0709 12:19:30.273577 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.273738 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":36277,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":89001,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":6920,"spinlock_wait_cycles":12383104,"thread_start_us":557262,"threads_started":2,"wal-append.queue_time_us":398258}
I20260709 12:19:30.377923 12568 rpcz_store.cc:276] 0709 12:19:29.296531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.456936 (+160405us) service_pool.cc:224] Handling call
0709 12:19:29.498136 (+ 41200us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.498433 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 33
0709 12:19:29.943216 (+444783us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.369610 (+426394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.369687 (+    77us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:30.369749 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.369826 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.369977 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":218231,"prepare.queue_time_us":376725,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":28,"thread_start_us":649005,"threads_started":2}
I20260709 12:19:30.273926 29679 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.385691 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.388321 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:30.238303 29735 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.157s
I20260709 12:19:30.390069 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:30.312459 29703 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.313864 29708 raft_consensus.cc:2764] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election decision vote started in defunct term 46: won
I20260709 12:19:30.306237 29716 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:30.316110 29721 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:30.312724 29715 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.395067 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:30.397866 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:30.393805 29735 consensus_queue.cc:1243] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 53.30 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:19:30.394239 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.401801 13260 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 30. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:19:30.418994 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:30.217000 29526 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.425390 13259 raft_consensus.cc:1275] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 31. Preceding OpId from leader: term: 53 index: 32. (index mismatch)
I20260709 12:19:30.432433 12646 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 57.
I20260709 12:19:30.405373 12639 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:19:30.399258 29681 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:30.409230 29703 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.410215 29405 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:30.208775 29712 consensus_queue.cc:260] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 57.28, Last appended by leader: 28, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:30.416007 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100634us
I20260709 12:19:30.393961 29679 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:30.418201 12462 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:30.384840 29255 raft_consensus.cc:1064] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:30.420545 29524 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:30.413982 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6722) took 1119 ms. Trace:
I20260709 12:19:30.442623 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.297935 29626 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:30.460938 13218 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.501027 12645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4834) took 3472 ms. Trace:
I20260709 12:19:30.482944 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.501824 12645 rpcz_store.cc:276] 0709 12:19:27.028599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.832947 (+2804348us) service_pool.cc:224] Handling call
0709 12:19:29.833377 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.837400 (+  4023us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:19:30.175752 (+338352us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.481313 (+305561us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.481607 (+   294us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:30.497800 (+ 16193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.497940 (+   140us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.500807 (+  2867us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":572399,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":642404,"threads_started":2}
I20260709 12:19:30.483070 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.503425 13255 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
I20260709 12:19:30.420856 29361 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:30.492511 12564 rpcz_store.cc:276] 0709 12:19:29.295144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.449669 (+154525us) service_pool.cc:224] Handling call
0709 12:19:29.465573 (+ 15904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.468582 (+  3009us) raft_consensus.cc:1534] Early marking committed up to index 28
0709 12:19:30.068279 (+599697us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.413547 (+345268us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.413604 (+    57us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:30.413697 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.413767 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.413918 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20899,"prepare.queue_time_us":406343,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":366,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":19903,"thread_start_us":736223,"threads_started":2}
I20260709 12:19:30.423430 29716 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.498271 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:30.438076 29570 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.022s
I20260709 12:19:30.438691 29264 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:30.511816 12637 raft_consensus.cc:1275] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 31. Preceding OpId from leader: term: 51 index: 32. (index mismatch)
I20260709 12:19:30.439807 29755 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.513520 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.521036 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.521991 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:30.462327 29605 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:30.472795 29546 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.534600 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.540710 13256 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 56. Current term is 57. Ops: []
I20260709 12:19:30.540880 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:30.541448 12564 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 49
W20260709 12:19:30.504312 29255 raft_consensus.cc:1068] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:19:30.486912 29734 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 60 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.498380 29681 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:30.478817 29758 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:30.527707 29524 consensus_queue.cc:1048] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.022s
I20260709 12:19:30.478389 29644 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.064s
I20260709 12:19:30.564261 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:30.494292 29710 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.052s
I20260709 12:19:30.494445 29727 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 54.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.076s
I20260709 12:19:30.440228 29715 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.549521 29681 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:30.530407 29645 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.058s
I20260709 12:19:30.550024 29605 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:19:30.565960 29761 log.cc:927] Time spent T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.155s	user 0.001s	sys 0.001s
I20260709 12:19:30.493556 29435 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.055s
I20260709 12:19:30.530117 29300 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.059s
I20260709 12:19:30.523425 29716 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:30.669922 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.565091 12563 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:30.674080 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 304981us
I20260709 12:19:30.587082 29117 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:30.559893 29704 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:30.630362 29734 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:30.686766 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.697016 12564 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:30.698766 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.681548 12637 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
W20260709 12:19:30.688580 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:30.532682 29755 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.677202 29681 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:30.715277 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.716939 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.718160 12637 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
W20260709 12:19:30.719110 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.679316 29605 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:30.723084 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:30.686419 29546 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:30.727963 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:30.723999 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:30.724496 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.624219 29757 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 50.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.146s
I20260709 12:19:30.676949 29600 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 56.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.051s
I20260709 12:19:30.759133 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:30.761833 12637 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 44.
W20260709 12:19:30.760910 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 340149us
I20260709 12:19:30.766768 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.727252 29605 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:30.779590 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.725997 29734 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:30.787926 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 369771us
I20260709 12:19:30.789506 29546 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:30.694054 29117 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:30.794530 29641 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.755s	user 0.006s	sys 0.009s
I20260709 12:19:30.796810 12564 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 55 index: 29. Preceding OpId from leader: term: 57 index: 30. (index mismatch)
I20260709 12:19:30.798380 12564 raft_consensus.cc:1240] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:30.800361 12564 raft_consensus.cc:1240] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 57. Current term is 58. Ops: []
W20260709 12:19:30.801780 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 508724us
I20260709 12:19:30.803558 12643 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 55.
I20260709 12:19:30.805058 12563 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:30.805490 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2404) took 3191 ms. Trace:
I20260709 12:19:30.805934 12642 rpcz_store.cc:276] 0709 12:19:27.614216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.911974 (+2297758us) service_pool.cc:224] Handling call
0709 12:19:29.912523 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.913332 (+   809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.913377 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.914136 (+   759us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.316993 (+402857us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:19:30.804769 (+487776us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.805154 (+   385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.805241 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:30.805282 (+    41us) raft_consensus.cc:1685] finished
0709 12:19:30.805294 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.805419 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8994,"prepare.queue_time_us":93027,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":4730,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":27571,"thread_start_us":769175,"threads_started":2,"wal-append.queue_time_us":456275}
I20260709 12:19:30.802831 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:30.816805 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.818306 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3189) took 1591 ms. Trace:
W20260709 12:19:30.820833 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 401878us
I20260709 12:19:30.824205 12564 raft_consensus.cc:2410] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 47 because replica has last-logged OpId of term: 46 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 24.
I20260709 12:19:30.703138 29704 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:30.827428 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:30.830016 12563 raft_consensus.cc:2393] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:30.827913 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:30.832674 12536 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:19:30.831363 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:30.818894 12566 rpcz_store.cc:276] 0709 12:19:29.227397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.227546 (+   149us) service_pool.cc:224] Handling call
0709 12:19:29.227893 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:29.455280 (+227387us) raft_consensus.cc:1534] Early marking committed up to index 31
0709 12:19:30.163944 (+708664us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.817853 (+653909us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:30.817936 (+    83us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:30.818007 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.818104 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:30.818232 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5328,"prepare.queue_time_us":419422,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":59202,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":28362,"thread_start_us":1327444,"threads_started":3,"wal-append.queue_time_us":522361}
I20260709 12:19:30.836984 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.801573 29758 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:30.814057 29720 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:30.840845 12564 raft_consensus.cc:2410] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 56 because replica has last-logged OpId of term: 55 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 30.
W20260709 12:19:30.863592 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 95279.1us
I20260709 12:19:30.865221 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.869575 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.827073 29405 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:30.866593 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 60 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:30.864298 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 445185us
I20260709 12:19:30.825549 29526 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:30.837097 29754 log.cc:927] Time spent T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.326s	user 0.001s	sys 0.000s
I20260709 12:19:30.873384 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 43 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:30.875953 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.877791 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:30.881837 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:30.880828 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 170222us
W20260709 12:19:30.864039 29760 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.316s	user 0.000s	sys 0.001s
W20260709 12:19:30.886974 12610 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:30.887995 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.888351 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:30.890837 12460 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:30.836927 29770 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:30.830411 29600 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.049s
W20260709 12:19:30.920867 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 88295.9us
I20260709 12:19:30.873118 29626 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.838672 29641 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.054s
W20260709 12:19:30.936337 12538 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:30.924796 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 56534.2us
I20260709 12:19:30.835506 29751 consensus_queue.cc:1059] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 24, Time since last communication: 0.070s
I20260709 12:19:30.922115 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:30.926205 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 96702us
W20260709 12:19:30.929742 12610 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.934595 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:30.943630 12610 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.944229 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.950524 12460 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.918198 29435 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:19:30.956532 12610 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.902621 29300 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:30.904076 29770 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:30.896977 29783 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
W20260709 12:19:30.953613 29756 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 12:19:30.963819 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:30.985956 12642 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:30.964535 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:30.973425 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:30.963718 29748 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.303s	user 0.000s	sys 0.001s
W20260709 12:19:30.963701 12610 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:30.993690 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.978780 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.932364 29779 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:31.013397 29779 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:31.013898 29779 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:30.987165 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:30.987839 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.943030 29691 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:30.958845 29759 log.cc:927] Time spent T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
W20260709 12:19:30.977972 29766 log.cc:927] Time spent T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:19:30.957047 29626 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:30.958498 29644 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:30.965613 29715 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:30.886149 29734 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 47.29, Last appended by leader: 29, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:31.019196 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55544.1us
I20260709 12:19:30.986719 13260 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 51 index: 30. (index mismatch)
I20260709 12:19:30.876139 29758 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:31.043473 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:30.979468 29770 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:30.991757 29751 raft_consensus.cc:3055] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Stepping down as leader of term 49
W20260709 12:19:30.985235 29749 log.cc:927] Time spent T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.291s	user 0.001s	sys 0.001s
W20260709 12:19:31.015741 29765 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.259s	user 0.000s	sys 0.002s
W20260709 12:19:31.063313 12610 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.098037 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.024649 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:31.099241 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 74665.6us
I20260709 12:19:30.995929 29716 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:31.018214 29645 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.050s
I20260709 12:19:30.994531 29595 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:30.958711 29546 consensus_queue.cc:1048] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.064s
W20260709 12:19:31.070492 12461 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.042s	sys 0.030s
W20260709 12:19:31.103662 12461 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.043s	sys 0.030s
W20260709 12:19:31.104087 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 633046us
I20260709 12:19:31.014277 29779 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:30.993803 29747 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:30.996376 29791 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
I20260709 12:19:31.082638 12642 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:31.091858 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:31.080193 29762 log.cc:927] Time spent T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.268s	user 0.001s	sys 0.001s
I20260709 12:19:31.108168 29595 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.119024 29595 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:31.119055 29772 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
W20260709 12:19:31.122748 29776 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:19:31.118100 29587 consensus_queue.cc:1059] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 57.28, Next index: 29, Last known committed idx: 28, Time since last communication: 0.005s
I20260709 12:19:30.987782 29405 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 47.33, Last appended by leader: 33, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.111239 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:31.113127 29780 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:31.117717 12564 raft_consensus.cc:2410] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d for term 45 because replica has last-logged OpId of term: 44 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 28.
I20260709 12:19:31.076150 29600 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.132494 12567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6724) took 1828 ms. Trace:
I20260709 12:19:31.133603 29772 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:31.138186 29721 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:31.056627 29300 consensus_queue.cc:1059] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.043s
I20260709 12:19:31.146196 13258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50712 (request call id 2157) took 1927 ms. Trace:
W20260709 12:19:31.118770 29764 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:19:31.153268 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:31.099519 29751 raft_consensus.cc:740] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:31.108695 29361 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
W20260709 12:19:31.158695 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 170911us
W20260709 12:19:31.165036 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:31.161435 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:31.166698 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:31.147074 12567 rpcz_store.cc:276] 0709 12:19:29.304710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.685366 (+380656us) service_pool.cc:224] Handling call
0709 12:19:29.685865 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:29.686355 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.686396 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:29.687046 (+   650us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.100742 (+413696us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:30.100790 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.100872 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.132167 (+1031295us) raft_consensus.cc:1685] finished
0709 12:19:31.132200 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.132402 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3499,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":35049,"thread_start_us":312731,"threads_started":1,"wal-append.queue_time_us":795233}
I20260709 12:19:31.168470 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:31.118294 29769 log.cc:927] Time spent T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 12:19:31.154132 13258 rpcz_store.cc:276] 0709 12:19:29.218611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:29.218742 (+   131us) service_pool.cc:224] Handling call
0709 12:19:29.219155 (+   413us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:29.467637 (+248482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:29.467687 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:29.674193 (+206506us) log.cc:844] Serialized 52 byte log entry
0709 12:19:30.128701 (+454508us) raft_consensus.cc:1645] Marking committed up to 32
0709 12:19:30.554429 (+425728us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.554662 (+   233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.554743 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.145917 (+591174us) raft_consensus.cc:1685] finished
0709 12:19:31.145942 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.146124 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11403,"prepare.queue_time_us":623753,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":39522,"raft.run_cpu_time_us":2016,"raft.run_wall_time_us":60232,"thread_start_us":980908,"threads_started":3,"wal-append.queue_time_us":834580}
I20260709 12:19:31.061153 29626 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.055915 29691 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.143846 29117 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.180306 29772 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:31.181871 29758 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.182991 12461 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:31.184162 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:31.186024 29785 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.186551 29785 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:31.188282 29739 consensus_meta.cc:287] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.915s	user 0.014s	sys 0.000s
I20260709 12:19:31.193310 29645 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:31.194211 29645 raft_consensus.cc:3055] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:31.194538 29645 raft_consensus.cc:740] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:31.162338 29554 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:31.195393 29645 consensus_queue.cc:260] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 49.31, Last appended by leader: 31, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.196825 29645 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:31.198063 29751 consensus_queue.cc:260] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.199626 29751 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:31.162943 29708 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:31.202602 29785 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:31.203286 29691 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.203199 29644 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20260709 12:19:31.158838 29600 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:31.206230 11310 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 12522)
I20260709 12:19:31.204279 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:31.206990 29606 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 1.097s	user 0.013s	sys 0.001s
I20260709 12:19:31.207700 29641 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.207894 29600 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:31.208644 29782 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:19:31.208317 29641 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:31.211898 29790 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:19:31.190510 29601 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.078s	user 0.012s	sys 0.001s
I20260709 12:19:31.217365 12569 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 52 because replica has last-logged OpId of term: 50 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 29.
I20260709 12:19:31.192171 29784 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:31.192217 29626 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.192759 29786 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.029s
I20260709 12:19:31.221872 29704 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:31.192912 29300 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:31.222985 29300 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:31.223863 29300 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 45.24, Last appended by leader: 24, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.142138 29716 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:31.228040 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 50 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:31.229205 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 60 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:31.234215 13258 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:31.230468 12646 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 53 index: 28. Preceding OpId from leader: term: 57 index: 30. (index mismatch)
I20260709 12:19:31.234185 12638 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 59.
I20260709 12:19:31.178923 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:31.238121 12641 pending_rounds.cc:85] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:19:31.238744 12641 pending_rounds.cc:107] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 53.30
I20260709 12:19:31.229666 12567 pending_rounds.cc:85] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 31
I20260709 12:19:31.239634 12567 pending_rounds.cc:107] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 50.32
I20260709 12:19:31.143709 29780 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:31.258566 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:31.259546 12563 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:31.261906 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:31.241688 12641 raft_consensus.cc:2889] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:31.240044 12567 raft_consensus.cc:2889] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:31.269546 29758 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:31.192330 29361 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:31.247123 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 68315.3us
I20260709 12:19:31.232633 29587 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.262606 29780 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:31.245820 29679 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 LEADER]: Stepping down as leader of term 56
I20260709 12:19:31.307792 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:31.284421 29748 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:19:31.282879 29742 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election won for term 53
W20260709 12:19:31.327662 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:31.244274 29554 consensus_queue.cc:1059] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 54.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.065s
I20260709 12:19:31.286307 29758 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.260062 29605 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:31.381621 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 318298us
W20260709 12:19:31.382928 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.383692 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:31.387199 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:31.322901 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:31.391499 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69273.7us
I20260709 12:19:31.317934 29679 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:31.388198 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:31.386593 12644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6194) took 1169 ms. Trace:
I20260709 12:19:31.326700 29778 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election won for term 45
W20260709 12:19:31.378291 29765 log.cc:927] Time spent T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:19:31.334399 29742 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:31.410208 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:31.468628 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 208502us
I20260709 12:19:31.464632 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.380259 29758 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:31.512559 12566 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:31.417431 12644 rpcz_store.cc:276] 0709 12:19:30.217159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.284767 (+ 67608us) service_pool.cc:224] Handling call
0709 12:19:30.285130 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:30.285835 (+   705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.285869 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:30.286380 (+   511us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.630201 (+343821us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:30.690881 (+ 60680us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.691111 (+   230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.691202 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.386283 (+695081us) raft_consensus.cc:1685] finished
0709 12:19:31.386313 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.386512 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":60370,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":97233,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":204,"thread_start_us":337720,"threads_started":1,"wal-append.queue_time_us":506180}
I20260709 12:19:31.321161 29587 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:31.468073 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71622us
W20260709 12:19:31.498986 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:31.517385 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 107233us
W20260709 12:19:31.535516 12610 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.536211 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:31.519613 12536 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.541204 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:31.543073 12536 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.543753 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:31.548306 12536 leader_election.cc:336] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.550874 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:31.444787 29778 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:31.582523 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47033.5us
W20260709 12:19:31.583766 12610 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.477006 29742 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:31.542410 29748 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:19:31.559233 29626 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:31.512185 29772 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.537331 29517 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:31.599830 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.593688 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:31.561175 29801 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:31.560727 29601 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:31.598366 29613 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.639986 29613 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.645985 29801 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.647001 29587 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.646714 29801 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.518707 29361 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 47.24, Last appended by leader: 24, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.649075 29801 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.587898 29778 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:31.634060 29797 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20260709 12:19:31.656705 29779 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.509s	user 0.012s	sys 0.000s
I20260709 12:19:31.676226 29779 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.684134 29779 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.684949 29802 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:31.685423 29802 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.685824 29802 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:19:31.708009 12537 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:31.709812 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:31.718204 29707 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Stepping down as leader of term 54
I20260709 12:19:31.718788 29707 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:31.719887 29707 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 54.30, Last appended by leader: 30, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.721910 29712 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Advancing to term 58
W20260709 12:19:31.712812 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 415344us
I20260709 12:19:31.740609 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 60 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:31.616560 29772 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:31.738517 29645 consensus_meta.cc:287] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.541s	user 0.012s	sys 0.000s
I20260709 12:19:31.742512 29613 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.758375 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6727) took 1333 ms. Trace:
I20260709 12:19:31.761996 12568 rpcz_store.cc:276] 0709 12:19:30.425297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.425502 (+   205us) service_pool.cc:224] Handling call
0709 12:19:30.470777 (+ 45275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:30.471032 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:30.516700 (+ 45668us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.726099 (+1209399us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b180025f420. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf539323a 00007ffbf57fda2a
0709 12:19:31.757792 (+ 31693us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:31.757871 (+    79us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:31.757971 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.758092 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.758288 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11702,"prepare.queue_time_us":43150,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":1389,"spinlock_wait_cycles":11822976,"thread_start_us":31270,"threads_started":1}
I20260709 12:19:31.751899 29739 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Leader pre-election won for term 60
I20260709 12:19:31.766215 29605 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:31.761190 29361 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.767364 29811 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.780290 29361 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.781454 12568 raft_consensus.cc:1240] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
W20260709 12:19:31.782848 29595 consensus_meta.cc:287] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.663s	user 0.012s	sys 0.001s
I20260709 12:19:31.780400 29811 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.783562 29361 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:31.789700 29821 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:19:31.783474 29595 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.792637 29595 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.795564 29811 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.749670 29814 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.030s
I20260709 12:19:31.767603 29606 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:31.798156 29606 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.800575 29606 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:31.803126 29802 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.803866 29772 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:19:31.771119 29739 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:31.804677 29739 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Advancing to term 60
I20260709 12:19:31.807694 29802 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.814139 29739 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 60 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.817888 29739 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 60 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:31.779367 29751 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.820127 29751 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.822613 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 2425) took 1522 ms. Trace:
I20260709 12:19:31.823182 12639 rpcz_store.cc:276] 0709 12:19:30.300759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.556692 (+255933us) service_pool.cc:224] Handling call
0709 12:19:30.557287 (+   595us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:30.700177 (+142890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.700258 (+    81us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:30.700913 (+   655us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.889585 (+188672us) raft_consensus.cc:1645] Marking committed up to 31
0709 12:19:30.945142 (+ 55557us) log.cc:844] Serialized 14 byte log entry
0709 12:19:30.945440 (+   298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.822044 (+876604us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54000b45d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:31.822106 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:31.822171 (+    65us) raft_consensus.cc:1685] finished
0709 12:19:31.822309 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:31.822531 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":95356,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":157788,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":37790,"spinlock_wait_cycles":349024000,"thread_start_us":131045,"threads_started":1,"wal-append.queue_time_us":483523}
I20260709 12:19:31.782819 29822 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:31.827414 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:31.830235 29770 consensus_meta.cc:287] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.724s	user 0.010s	sys 0.003s
I20260709 12:19:31.830914 29770 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.831142 29822 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.832437 12642 raft_consensus.cc:1240] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 43. Current term is 44. Ops: [43.25-43.25]
W20260709 12:19:31.837949 29681 consensus_meta.cc:287] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 1.085s	user 0.013s	sys 0.002s
I20260709 12:19:31.838506 29681 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.839668 12570 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:31.841470 12570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32896 (request call id 6731) took 1378 ms (client timeout 1649 ms). Trace:
W20260709 12:19:31.841868 12570 rpcz_store.cc:269] 0709 12:19:30.462799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.463601 (+   802us) service_pool.cc:224] Handling call
0709 12:19:31.841242 (+1377641us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003ac854. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:31.841381 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3058351872}
I20260709 12:19:31.846662 29758 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.847865 12563 raft_consensus.cc:1240] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:31.853318 29606 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.855196 29606 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.773218 29780 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.858618 29361 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.859258 29361 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.859706 29780 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:31.861459 12570 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:31.835485 29822 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:31.875080 29763 log.cc:927] Time spent T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.100s	user 0.005s	sys 0.002s
I20260709 12:19:31.890044 29778 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:31.901051 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:31.884527 29606 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.905407 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 60 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:31.907830 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:31.908768 29783 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:31.907634 29738 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.909436 29783 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:31.908819 13256 raft_consensus.cc:2468] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 53.
I20260709 12:19:31.911453 29681 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.911065 29738 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.912051 29117 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:31.912675 29117 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.891985 29770 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:31.900440 29758 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.923480 29361 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:31.921303 29681 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:31.924619 29742 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:31.925089 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 341319us
W20260709 12:19:31.926273 12610 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.926883 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.928023 29791 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:19:31.928191 12610 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.928773 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:31.929237 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 219578us
I20260709 12:19:31.930290 13258 raft_consensus.cc:1240] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 52. Current term is 53. Ops: []
W20260709 12:19:31.931187 12610 leader_election.cc:336] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.931723 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.926357 29804 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:31.932122 12536 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:31.963174 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:31.959194 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:31.965237 12462 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:31.967187 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.929172 29526 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:19:31.987625 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56460.6us
I20260709 12:19:31.929925 29517 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
W20260709 12:19:32.003902 12610 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.007745 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:32.008746 13218 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.009207 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.009673 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:31.924118 29738 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:32.010850 13256 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:31.997184 12536 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.013571 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.014767 12567 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
I20260709 12:19:32.012238 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:32.017867 29605 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:32.018566 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:32.021450 29605 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.021163 11310 tablet_replica.cc:333] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:32.020071 12536 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.022408 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:32.022099 29605 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.023394 29802 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:32.023851 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.024367 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:32.025466 29802 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader election lost for term 58. Reason: could not achieve majority
I20260709 12:19:32.025339 29691 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:19:32.026705 12536 leader_election.cc:336] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.026145 29605 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.027374 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:32.027313 29691 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.029150 29605 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:32.030727 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.031546 12563 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:31.925248 29681 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:31.986267 29771 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:32.006764 29780 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:32.034556 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.006592 29791 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:32.003027 29644 raft_consensus.cc:2804] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:32.012933 29770 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:32.040364 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.014209 29797 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:32.030613 29805 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.049841 29805 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.050432 29691 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:32.050882 29681 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.052929 29805 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:32.034219 29117 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.043316 29644 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.054180 29644 raft_consensus.cc:3060] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:32.053592 29606 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.058193 29606 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.060806 29606 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:32.040172 29715 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.921s	user 0.015s	sys 0.019s
W20260709 12:19:32.063145 29715 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.015s	sys 0.019s
I20260709 12:19:32.063392 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:32.064218 13254 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:32.065843 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:32.066200 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 59 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.066928 12563 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 58.
I20260709 12:19:32.068116 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.067013 29735 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:32.069922 29735 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.070309 29735 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:32.071048 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.071911 12568 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:32.073160 29742 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:32.074368 29778 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.071718 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:32.048852 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.081157 11310 raft_consensus.cc:2243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:19:32.084451 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:32.073102 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:32.088346 12536 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.088749 13256 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 33. Preceding OpId from leader: term: 49 index: 34. (index mismatch)
W20260709 12:19:32.089977 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.064977 29783 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.090703 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:32.100010 12536 leader_election.cc:336] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.100687 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:32.104730 29681 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:32.105378 12536 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.105876 12569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50142 (request call id 3199) took 1789 ms (client timeout 2231 ms). Trace:
I20260709 12:19:32.106236 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:32.106631 12569 rpcz_store.cc:269] 0709 12:19:30.316891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.317067 (+   176us) service_pool.cc:224] Handling call
0709 12:19:32.105543 (+1788476us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54002459d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:32.105796 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2067432704}
I20260709 12:19:32.108794 13257 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 56. Current term is 57. Ops: []
W20260709 12:19:32.109716 29789 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 1.011s	user 0.005s	sys 0.004s
I20260709 12:19:32.110370 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:32.111241 13261 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:32.111573 12640 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6196) took 1893 ms. Trace:
I20260709 12:19:32.112079 12565 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 51 index: 30. (term mismatch)
I20260709 12:19:32.112562 12565 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 29
I20260709 12:19:32.112955 12565 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 50.30
I20260709 12:19:32.113366 12565 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:32.114169 29758 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 56.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20260709 12:19:32.113476 11310 raft_consensus.cc:2272] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:32.115710 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.115800 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:32.112035 12640 rpcz_store.cc:276] 0709 12:19:30.217810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.343743 (+125933us) service_pool.cc:224] Handling call
0709 12:19:30.344083 (+   340us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:30.679588 (+335505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.679629 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:30.683946 (+  4317us) log.cc:844] Serialized 52 byte log entry
0709 12:19:30.893014 (+209068us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:19:31.050945 (+157931us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.051468 (+   523us) log.cc:844] Serialized 14 byte log entry
0709 12:19:31.159124 (+107656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:31.159206 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.109000 (+949794us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:32.111318 (+  2318us) raft_consensus.cc:1685] finished
0709 12:19:32.111338 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.111498 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42568,"prepare.queue_time_us":282717,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":110,"raft.queue_time_us":366662,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":11421,"spinlock_wait_cycles":54273408,"thread_start_us":717236,"threads_started":4,"wal-append.queue_time_us":308789}
I20260709 12:19:32.102061 29459 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.016s
I20260709 12:19:32.100010 29738 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.143631 29738 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.118083 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:32.118763 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:19:32.122323 12536 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.144965 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:32.104023 29778 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:32.134824 12569 raft_consensus.cc:1240] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:32.151723 13262 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:32.152056 12569 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 47 index: 25. (index mismatch)
I20260709 12:19:32.152359 29605 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:32.153015 29605 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.153448 29605 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:32.107342 29681 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:32.140213 29606 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.134806 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:32.123894 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 112049us
W20260709 12:19:32.168169 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:32.194377 12536 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.195479 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.195879 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:32.198101 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 267389us
I20260709 12:19:32.199225 12460 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:32.209728 12563 raft_consensus.cc:2393] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:32.213268 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:32.214779 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:32.216738 13262 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
W20260709 12:19:32.213193 12536 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.218559 12537 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:32.211375 29789 log.cc:927] Time spent T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:19:32.223228 12536 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.236809 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.209359 29758 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:32.232314 12536 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.212073 29799 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:32.220319 29681 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:32.242208 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:32.230647 29778 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:32.224112 29575 consensus_queue.cc:1048] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.085s
I20260709 12:19:32.310325 29799 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:32.314085 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81788.1us
I20260709 12:19:32.307672 29755 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.230684 29783 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.243643 29822 consensus_queue.cc:1059] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 50.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.068s
I20260709 12:19:32.221756 29691 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:32.222311 29524 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:32.209208 29606 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.223678 29255 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:32.330492 29802 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:19:32.339257 12536 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.363198 12536 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.341024 29691 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.344225 29524 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.264477 29758 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.376837 29691 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:32.382555 29833 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:19:32.385526 29799 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.389479 29641 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.377841 29524 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:32.394354 29735 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.406271 29606 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.409471 29758 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.411574 29735 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.403796 29644 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.413177 29691 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.415330 29691 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:32.463754 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 223986us
I20260709 12:19:32.468737 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:32.470762 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:32.472476 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:32.474191 12460 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:32.480345 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:32.486104 12536 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.487987 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.489171 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.513326 12637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6213) took 1277 ms. Trace:
I20260709 12:19:32.514133 12637 rpcz_store.cc:276] 0709 12:19:31.235868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.600899 (+365031us) service_pool.cc:224] Handling call
0709 12:19:31.601296 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:31.603083 (+  1787us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.603121 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:31.627695 (+ 24574us) log.cc:844] Serialized 26 byte log entry
0709 12:19:32.332871 (+705176us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b1800325300. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf538d20b 00007ffbf5392d23 00007ffbf53930d1 00007ffbf57aea0f
0709 12:19:32.512056 (+179185us) raft_consensus.cc:1645] Marking committed up to 28
0709 12:19:32.512204 (+   148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.512394 (+   190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.512492 (+    98us) raft_consensus.cc:1685] finished
0709 12:19:32.512508 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.512944 (+   436us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23632,"raft.queue_time_us":1457,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"spinlock_wait_cycles":49324928,"thread_start_us":178442,"threads_started":1,"wal-append.queue_time_us":778132}
I20260709 12:19:32.539264 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:32.538996 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:32.546944 29255 consensus_queue.cc:1059] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.29, Next index: 30, Last known committed idx: 28, Time since last communication: 0.006s
I20260709 12:19:32.558240 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:32.559242 29107 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:32.560503 29264 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Leader pre-election won for term 59
I20260709 12:19:32.560591 29107 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.562682 29107 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:32.563320 29606 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:32.564075 29606 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.564602 29606 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:32.570233 29464 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.570794 29570 raft_consensus.cc:3055] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:32.571447 29464 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.561326 29264 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.415159 29644 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.571491 29570 raft_consensus.cc:740] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:32.586295 29641 raft_consensus.cc:3055] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:32.587165 29641 raft_consensus.cc:740] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:32.578739 29264 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 58 FOLLOWER]: Advancing to term 59
W20260709 12:19:32.564273 12564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 6303) took 1742 ms (client timeout 1943 ms). Trace:
I20260709 12:19:32.588196 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32884 (request call id 6306) took 1760 ms. Trace:
I20260709 12:19:32.592280 12566 rpcz_store.cc:276] 0709 12:19:30.822251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.877482 (+ 55231us) service_pool.cc:224] Handling call
0709 12:19:32.565306 (+1687824us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54002477d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:32.582611 (+ 17305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1368549632}
I20260709 12:19:32.597609 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
W20260709 12:19:32.590747 12564 rpcz_store.cc:269] 0709 12:19:30.821149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.873108 (+ 51959us) service_pool.cc:224] Handling call
0709 12:19:32.563598 (+1690490us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400045e54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f72fa 00007ffbfa7b8b17 00007ffbf3d336d4 00007ffbf3d33662 00007ffbf3d33591 00007ffbf3d3350c 00007ffbf3d327c2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:32.563978 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516201856}
I20260709 12:19:32.599740 29558 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:32.603776 29464 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.604707 29641 consensus_queue.cc:260] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.586361 29570 consensus_queue.cc:260] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 50.26, Last appended by leader: 26, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.607424 29641 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:32.608340 29570 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:32.611375 12386 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 38 to 41, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 41 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:32.618686 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.632071 29300 consensus_queue.cc:1059] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 52.33, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
I20260709 12:19:32.633975 29300 raft_consensus.cc:3055] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Stepping down as leader of term 52
I20260709 12:19:32.634929 29300 raft_consensus.cc:740] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:32.616767 12386 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 44 to 49, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 49 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:32.640322 29679 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 56.32, Last appended by leader: 32, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.644634 29707 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
W20260709 12:19:32.656853 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.637312 29300 consensus_queue.cc:260] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 52.33, Last appended by leader: 33, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.682938 29849 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.684041 29849 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.653007 29645 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.651312 29703 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.698625 29786 consensus_queue.cc:1059] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:32.709239 29645 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:32.709880 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.710747 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:32.713125 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.727778 12566 raft_consensus.cc:1240] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 55. Current term is 56. Ops: []
I20260709 12:19:32.740013 29645 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:32.741300 12536 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.743240 12565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6310) took 1700 ms. Trace:
I20260709 12:19:32.745157 29587 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.039s
I20260709 12:19:32.753839 29679 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:32.756062 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 68965.7us
I20260709 12:19:32.757274 29630 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:32.757830 29630 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:32.758527 29630 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:32.760703 29849 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.752753 12639 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6218) took 1497 ms. Trace:
W20260709 12:19:32.768262 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:32.780648 11310 tablet_replica.cc:333] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:32.782841 29770 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.783907 29770 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:32.785740 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.786846 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.789310 29770 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.761188 29755 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:32.790400 29755 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.794391 29755 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:32.753952 12565 rpcz_store.cc:276] 0709 12:19:31.042415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.215384 (+172969us) service_pool.cc:224] Handling call
0709 12:19:32.111690 (+896306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:32.113859 (+  2169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.742632 (+628773us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54002459d0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:32.742998 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060420864}
I20260709 12:19:32.804389 29739 consensus_queue.cc:1059] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 41.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:19:32.806375 29811 raft_consensus.cc:3055] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Stepping down as leader of term 43
I20260709 12:19:32.806846 29811 raft_consensus.cc:740] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:32.807937 29811 consensus_queue.cc:260] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.816363 11310 raft_consensus.cc:2243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:19:32.815019 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.805015 29652 raft_consensus.cc:3055] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
W20260709 12:19:32.823218 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48355.7us
I20260709 12:19:32.767141 29300 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:32.831328 12536 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:32.817503 29792 log.cc:927] Time spent T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 1.657s	user 0.006s	sys 0.000s
I20260709 12:19:32.824877 29652 raft_consensus.cc:740] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:32.838661 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:32.842948 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.848344 29524 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.822785 29264 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.849822 29645 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.858847 29645 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.849903 29739 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.850761 29554 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:32.762481 12639 rpcz_store.cc:276] 0709 12:19:31.256241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.834123 (+577882us) service_pool.cc:224] Handling call
0709 12:19:31.834600 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:31.834877 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:19:31.853950 (+ 19073us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.751235 (+897285us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b180039a660. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287 00007ffbf580e208
0709 12:19:32.751737 (+   502us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:32.751885 (+   148us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:19:32.752035 (+   150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.752198 (+   163us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.752486 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27875,"prepare.queue_time_us":18460,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":734887,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":242,"spinlock_wait_cycles":2820480,"thread_start_us":165439,"threads_started":1,"wal-append.queue_time_us":190877}
I20260709 12:19:32.866384 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.882763 29652 consensus_queue.cc:260] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.27, Last appended by leader: 27, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.884797 29652 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:32.861610 29645 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:32.885704 29751 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.888072 29747 consensus_queue.cc:1059] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 55.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:32.888453 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:32.889698 29747 raft_consensus.cc:3055] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Stepping down as leader of term 55
I20260709 12:19:32.890177 29747 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:32.890494 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 41 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:32.890776 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.891765 29587 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:32.892072 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.886049 13260 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:32.894332 12571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6729) took 2455 ms. Trace:
I20260709 12:19:32.894531 29524 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.895399 12564 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:32.883634 29264 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 59 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.892902 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.900756 12570 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:32.873333 29811 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:32.893468 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.862376 29739 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:32.896463 29778 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.002s	sys 0.028s
I20260709 12:19:32.914227 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 59 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:32.913558 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44284.2us
I20260709 12:19:32.921459 11310 raft_consensus.cc:2272] T 1f92af3771e3446f8c3549eb19f5b2c9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:32.925843 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.934080 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:32.939786 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:32.941044 13262 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:32.894754 12571 rpcz_store.cc:276] 0709 12:19:30.438432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:30.438642 (+   210us) service_pool.cc:224] Handling call
0709 12:19:30.439032 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:30.694421 (+255389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:30.694471 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:30.713076 (+ 18605us) log.cc:844] Serialized 26 byte log entry
0709 12:19:30.890536 (+177460us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:19:30.890680 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:30.891082 (+   402us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.736575 (+1845493us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf538e86e 00007ffbf538dbb6 00007ffbf538d7d8 00007ffbf5397221 00007ffbf53971d9 00007ffbf5397169
0709 12:19:32.893718 (+157143us) raft_consensus.cc:1685] finished
0709 12:19:32.893750 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.894112 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":73022,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":10193,"spinlock_wait_cycles":68449024,"thread_start_us":394605,"threads_started":2,"wal-append.queue_time_us":208311}
I20260709 12:19:32.918076 29107 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:32.944006 12571 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 24. Preceding OpId from leader: term: 49 index: 25. (index mismatch)
I20260709 12:19:32.945786 12570 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Advancing to term 59
W20260709 12:19:32.918401 29778 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.003s	sys 0.029s
I20260709 12:19:32.948273 29778 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:32.925881 29854 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:19:32.949512 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6215) took 1707 ms. Trace:
I20260709 12:19:32.897042 29554 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:32.965090 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57931.6us
I20260709 12:19:32.968933 12460 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:32.975574 12565 raft_consensus.cc:2410] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 52 because replica has last-logged OpId of term: 51 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 27.
W20260709 12:19:32.941016 29605 consensus_meta.cc:287] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.787s	user 0.016s	sys 0.011s
I20260709 12:19:32.983994 29107 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.964874 12646 rpcz_store.cc:276] 0709 12:19:31.241540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.692383 (+450843us) service_pool.cc:224] Handling call
0709 12:19:31.692784 (+   401us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:31.927603 (+234819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:31.927638 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:31.928395 (+   757us) log.cc:844] Serialized 52 byte log entry
0709 12:19:32.157420 (+229025us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:32.190812 (+ 33392us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.869139 (+678327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.901773 (+ 32634us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:32.911901 (+ 10128us) raft_consensus.cc:1685] finished
0709 12:19:32.911921 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.949226 (+ 37305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":797208,"prepare.queue_time_us":33510,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":827136,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":409,"thread_start_us":303018,"threads_started":2,"wal-append.queue_time_us":312670}
I20260709 12:19:32.985858 29605 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:32.988510 29605 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:32.969436 12536 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:32.996284 29738 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:32.996850 12572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32896 (request call id 6747) took 1707 ms. Trace:
I20260709 12:19:32.997372 12572 rpcz_store.cc:276] 0709 12:19:31.289423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:31.651672 (+362249us) service_pool.cc:224] Handling call
0709 12:19:32.582569 (+930897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:32.587563 (+  4994us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:32.695879 (+108316us) log.cc:844] Serialized 14 byte log entry
0709 12:19:32.853314 (+157435us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:32.853388 (+    74us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:32.853465 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:32.863517 (+ 10052us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:32.996498 (+132981us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400045e50. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:32.996648 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11962,"prepare.queue_time_us":133149,"prepare.run_cpu_time_us":191,"prepare.run_wall_time_us":10412,"raft.queue_time_us":37433,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":62369,"spinlock_wait_cycles":2129085696,"thread_start_us":193475,"threads_started":2,"wal-append.queue_time_us":271631}
I20260709 12:19:33.000069 12572 raft_consensus.cc:1240] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:33.000926 12386 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 50 to 53, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 53 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:19:33.008276 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 169695us
I20260709 12:19:33.008594 29517 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:33.009377 29716 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:33.010950 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:33.013496 12637 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 58 index: 37. Preceding OpId from leader: term: 59 index: 38. (index mismatch)
I20260709 12:19:33.015815 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.016825 12638 raft_consensus.cc:2376] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 57. Re-sending same reply.
I20260709 12:19:33.005223 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:33.004391 12386 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 39 to 44. New cstate: current_term: 44 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:33.020793 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.024194 29715 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
W20260709 12:19:33.033123 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53115.7us
W20260709 12:19:33.034157 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.035557 29715 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.024168 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.009505 29517 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.039769 13258 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:33.042872 29715 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:33.068367 12563 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 57.
I20260709 12:19:33.038029 29605 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.041659 11310 tablet_replica.cc:333] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:33.082192 12536 leader_election.cc:336] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:33.091455 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 176057us
I20260709 12:19:33.100057 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.048575 29645 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:33.105206 29645 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.108090 29645 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:33.110380 29630 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.112172 29852 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:33.113447 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79343.5us
W20260709 12:19:33.114341 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.114646 29517 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:33.114840 13218 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.115949 29778 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 43.26, Last appended by leader: 26, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.117978 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.119014 12564 raft_consensus.cc:2393] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 51: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:33.128099 12568 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 51. Current term is 52. Ops: [51.30-51.30]
I20260709 12:19:33.133836 29739 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:33.134624 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:33.141812 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.141619 12536 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.077317 29554 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:33.103171 12565 raft_consensus.cc:1240] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 50. Current term is 51. Ops: []
I20260709 12:19:33.095582 12386 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 42 to 44. New cstate: current_term: 44 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:33.113260 29630 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:33.152786 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:33.156491 13262 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 57.
I20260709 12:19:33.157588 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.153117 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:33.156649 12386 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 42 to 43, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 43 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:33.102813 29606 consensus_meta.cc:287] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.533s	user 0.019s	sys 0.000s
I20260709 12:19:33.149561 29771 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:33.168488 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 58248.2us
W20260709 12:19:33.174959 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.177465 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 76435.1us
I20260709 12:19:33.114650 29747 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 55.31, Last appended by leader: 31, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.186064 12386 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 41 to 45, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:33.187958 12386 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 50. New cstate: current_term: 50 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:33.189983 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 232) took 79477 ms (client timeout 15000 ms). Trace:
I20260709 12:19:33.190182 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.191287 12645 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:33.195971 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.201646 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.202761 12641 raft_consensus.cc:2376] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 9485b0afaf144396adf854fff857414e in term 55. Re-sending same reply.
I20260709 12:19:33.202903 29860 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election won for term 58
I20260709 12:19:33.203714 29860 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.204154 29860 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:33.204690 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.094722 29524 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.083s
I20260709 12:19:33.205797 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.102432 29605 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.190403 29794 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20260709 12:19:33.189755 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:33.197409 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.217487 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:33.150751 29755 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.058s
I20260709 12:19:33.230578 12570 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 59.
I20260709 12:19:33.185518 29747 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:33.166563 29606 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:33.235280 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120998us
I20260709 12:19:33.225176 12644 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:33.215672 29798 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:33.244073 13261 raft_consensus.cc:2393] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:33.249424 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.250267 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.224130 29811 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.038s
I20260709 12:19:33.266844 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:33.267863 29605 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:33.236481 29645 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.245747 29798 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:33.271098 11310 raft_consensus.cc:2243] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:33.239138 29771 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.264988 29526 consensus_queue.cc:1059] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.29, Next index: 30, Last known committed idx: 29, Time since last communication: 0.028s
I20260709 12:19:33.269951 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:33.281087 12641 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 33. Preceding OpId from leader: term: 49 index: 34. (index mismatch)
I20260709 12:19:33.283459 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.287744 12563 raft_consensus.cc:2393] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:33.190438 12387 rpcz_store.cc:269] 0709 12:18:13.712427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:13.712733 (+   306us) service_pool.cc:224] Handling call
0709 12:19:33.183168 (+79470435us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:23.481670 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:23.482861 (+  1191us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:23.482936 (+    75us) write_op.cc:435] Acquired schema lock
0709 12:19:23.482980 (+    44us) tablet.cc:662] Decoding operations
0709 12:19:23.483994 (+  1014us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:23.484140 (+   146us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:23.484181 (+    41us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:19:23.485144 (+   963us) tablet.cc:701] Row locks acquired
0709 12:19:23.485218 (+    74us) write_op.cc:260] PREPARE: finished
0709 12:19:23.485498 (+   280us) write_op.cc:270] Start()
0709 12:19:23.486750 (+  1252us) write_op.cc:276] Timestamp: P: 1783599563485403 usec, L: 0
0709 12:19:23.486807 (+    57us) op_driver.cc:348] REPLICATION: starting
0709 12:19:23.487661 (+   854us) log.cc:844] Serialized 2089 byte log entry
0709 12:19:24.473024 (+985363us) op_driver.cc:464] REPLICATION: finished
0709 12:19:25.050016 (+576992us) write_op.cc:301] APPLY: starting
0709 12:19:25.078069 (+ 28053us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:25.264920 (+186851us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:25.265073 (+   153us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:25.464086 (+199013us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:25.464441 (+   355us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=10,mrs_lookups=0
0709 12:19:25.464498 (+    57us) write_op.cc:312] APPLY: finished
0709 12:19:25.464896 (+   398us) log.cc:844] Serialized 57 byte log entry
0709 12:19:25.516915 (+ 52019us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:25.517042 (+   127us) write_op.cc:454] Released schema lock
0709 12:19:25.517107 (+    65us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:25.517697 (+   590us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:25.517736 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6928,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":495542,"apply.run_cpu_time_us":12400,"apply.run_wall_time_us":483953,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25007,"cfile_init":1,"lbm_read_time_us":488,"lbm_reads_lt_1ms":4,"num_ops":5,"prepare.queue_time_us":251387,"prepare.run_cpu_time_us":18307,"prepare.run_wall_time_us":350230,"raft.queue_time_us":245495,"raft.run_cpu_time_us":27203,"raft.run_wall_time_us":431006,"replication_time_us":909046,"thread_start_us":722712,"threads_started":4,"wal-append.queue_time_us":566380,"wal-append.run_cpu_time_us":16649,"wal-append.run_wall_time_us":2448127}]]}
I20260709 12:19:33.296161 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:33.270190 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:33.304803 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66160us
W20260709 12:19:33.306357 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 238) took 16927 ms (client timeout 15000 ms). Trace:
W20260709 12:19:33.307358 12387 rpcz_store.cc:269] 0709 12:19:16.374289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:33.295366 (+16921077us) service_pool.cc:211] Skipping call since client already timed out
0709 12:19:33.301378 (+  6012us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:19:33.317951 12537 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:33.318488 12643 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 45 index: 24. Preceding OpId from leader: term: 47 index: 25. (index mismatch)
I20260709 12:19:33.269852 29645 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.324769 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:33.333315 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.341876 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:33.336799 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.358209 29691 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:33.355722 12386 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 42 to 44, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 44 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:33.360435 12386 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 39. New cstate: current_term: 39 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:33.361850 29742 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:33.324076 29652 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:33.363409 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.367224 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 126050us
W20260709 12:19:33.368163 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:33.371002 29652 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.373231 12386 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 45 to 50. New cstate: current_term: 50 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:33.346079 29771 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:33.377516 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 60 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.378223 12642 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Advancing to term 60
I20260709 12:19:33.380335 12640 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:33.379742 12460 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:33.347083 29852 consensus_queue.cc:1059] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 44.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.036s
W20260709 12:19:33.407812 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.374781 29652 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:33.418766 29773 raft_consensus.cc:3055] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Stepping down as leader of term 51
I20260709 12:19:33.392740 29570 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:33.395574 29587 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Leader election won for term 59
I20260709 12:19:33.427841 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:33.419968 12386 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 37 to 41. New cstate: current_term: 41 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:33.424916 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:33.430999 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 114892us
I20260709 12:19:33.419516 29773 raft_consensus.cc:740] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:33.440356 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.441968 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.446434 12563 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:33.448804 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 59 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.450066 29645 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:33.450604 29822 consensus_queue.cc:1059] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 50.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.066s
I20260709 12:19:33.453908 29606 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:33.454231 29707 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.454880 29707 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.457700 29783 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:33.458529 29783 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:33.459692 29783 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 30, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.460126 29587 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 59 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:33.461575 29822 raft_consensus.cc:3055] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Stepping down as leader of term 50
I20260709 12:19:33.462289 29822 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:33.463863 29645 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.463471 29822 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 50.31, Last appended by leader: 31, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.464552 29645 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.465075 29822 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:33.458900 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.461709 29587 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 57.28, Last appended by leader: 28, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.457504 29707 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:33.472478 29645 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:33.480505 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.487670 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.487809 29715 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:33.462421 29845 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:19:33.494012 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:33.503528 12644 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 56.
W20260709 12:19:33.504544 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 60124us
I20260709 12:19:33.500906 12641 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:33.504908 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:33.512979 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:33.516736 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.517670 12640 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:33.526918 13261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:33.533710 11310 raft_consensus.cc:2272] T 79b687d20a534e70865ef902fbd0d4a5 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:33.522984 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.537982 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 59 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.538704 12644 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Advancing to term 59
I20260709 12:19:33.545993 29300 raft_consensus.cc:3055] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Stepping down as leader of term 44
I20260709 12:19:33.546700 29300 raft_consensus.cc:740] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:33.547974 29300 consensus_queue.cc:260] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.558773 13262 raft_consensus.cc:1240] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:33.560220 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.522989 12386 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 42 to 43, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 43 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:33.568161 12641 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:33.529947 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.571645 12570 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:33.576756 12564 raft_consensus.cc:2410] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 51 because replica has last-logged OpId of term: 49 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 26.
I20260709 12:19:33.589942 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:33.593047 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.585692 13261 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:33.602913 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:33.583607 13218 leader_election.cc:336] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.786s (SENT)
W20260709 12:19:33.606962 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.609469 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:33.605444 12386 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 43 to 47, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 47 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:33.460196 29773 consensus_queue.cc:260] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.30, Last appended by leader: 30, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.614082 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.582566 29715 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:33.620324 29742 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20260709 12:19:33.621408 29861 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:33.628571 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.631821 12640 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:33.591442 29300 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:33.627913 12567 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
W20260709 12:19:33.641004 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.643471 12571 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 47. Current term is 48. Ops: []
I20260709 12:19:33.641269 12641 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:33.647385 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 392885us
I20260709 12:19:33.605995 29570 consensus_queue.cc:1059] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.033s
W20260709 12:19:33.663960 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 90096.8us
I20260709 12:19:33.667310 29742 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 36, Time since last communication: 0.001s
I20260709 12:19:33.669008 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.674378 12386 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 36 to 45, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 45 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:33.678263 12460 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:33.699119 12460 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:33.699119 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.700006 12570 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
W20260709 12:19:33.701565 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.714715 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:33.714890 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55168.8us
I20260709 12:19:33.726222 12644 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 59.
W20260709 12:19:33.716655 12536 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:33.717851 29867 log.cc:927] Time spent T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:19:33.728086 12386 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 39 to 43. New cstate: current_term: 43 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:33.721215 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:33.696097 29605 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:33.734777 12610 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.722635 12641 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 54.
I20260709 12:19:33.722806 12637 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:33.739439 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.702841 29739 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:33.753907 12637 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:33.775347 12640 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:19:33.776938 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 358513us
I20260709 12:19:33.784941 12642 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 60.
I20260709 12:19:33.735806 29738 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:33.734194 29570 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:33.637543 29861 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.787644 12460 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:33.739464 29641 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:33.802501 29755 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:33.811802 11310 tablet_replica.cc:333] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:33.811165 12386 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 27 to 38. New cstate: current_term: 38 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:33.813315 11310 raft_consensus.cc:2243] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:19:33.813973 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.813122 12386 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 45 to 49, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 49 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:33.803089 29641 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:33.839053 12641 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 24. Preceding OpId from leader: term: 49 index: 25. (index mismatch)
W20260709 12:19:33.842612 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:33.843400 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.786722 29605 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.828329 29738 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:33.855334 12563 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:19:33.856998 12645 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.005s
I20260709 12:19:33.861981 12645 raft_consensus.cc:2468] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
W20260709 12:19:33.862708 12536 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 2.009s (SENT)
I20260709 12:19:33.798621 29606 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 47.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.042s
I20260709 12:19:33.841977 29852 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:33.866158 12460 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:33.842094 12386 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 25, term changed from 37 to 40, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to <none>, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 40 committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:19:33.867254 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 169992us
I20260709 12:19:33.872300 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.863277 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.841343 29641 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:33.882608 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:33.883584 12641 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:33.886221 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.887850 12644 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:33.869490 29652 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:33.889469 29570 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:19:33.930159 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:33.931835 12644 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:33.893738 29852 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:33.903880 29679 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20260709 12:19:33.937772 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:33.947093 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:33.921471 29606 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:33.941784 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 371052us
I20260709 12:19:33.940172 29861 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:33.940411 29570 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:33.921761 29652 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:33.974726 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.997951 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:34.000810 29860 consensus_meta.cc:287] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.796s	user 0.009s	sys 0.008s
W20260709 12:19:34.001663 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:34.002602 29878 log.cc:927] Time spent T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:19:34.001320 29860 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:33.990732 29264 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.008611 29264 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:34.010946 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 171593us
W20260709 12:19:34.016229 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:33.965986 29606 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:33.984670 29517 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:33.994212 29755 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:34.024518 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:34.036621 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 174151us
I20260709 12:19:34.048005 29755 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.052018 29652 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.052623 29755 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:34.040592 13218 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:34.061517 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:34.064042 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 61066.9us
W20260709 12:19:34.069190 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 171226us
I20260709 12:19:34.075893 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:34.071214 13218 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.082808 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.083680 12644 raft_consensus.cc:2368] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for earlier term 47. Current term is 48.
I20260709 12:19:34.080560 29605 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.093958 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:34.094851 12570 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:34.104477 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.084769 29264 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.115160 29605 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:34.115924 29605 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.117000 29606 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.118934 12641 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:34.054299 29852 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.28, Last appended by leader: 28, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:34.116864 29879 log.cc:927] Time spent T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260709 12:19:34.139621 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.150858 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:34.151551 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.126150 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.141613 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:34.154754 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 64198.8us
W20260709 12:19:34.155575 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 72158.8us
I20260709 12:19:34.158849 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:34.161509 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:34.163609 12637 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 56. Current term is 58. Ops: []
I20260709 12:19:34.164749 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:34.166533 29773 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:34.168898 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.169701 12643 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:34.171213 29645 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:34.171661 29645 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:34.171984 29645 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:34.118367 29707 raft_consensus.cc:932] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Unable to handle notification of new term (50): Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:19:34.181273 12643 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:19:34.190663 29645 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.192188 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:34.194525 29885 log.cc:927] Time spent T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:19:34.194995 29605 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:34.105298 29641 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.090245 29570 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 46.29, Last appended by leader: 29, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.194808 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.198208 29784 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Stepping down as leader of term 47
W20260709 12:19:34.196736 12610 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.207372 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:34.208992 13262 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 52 index: 30. (index mismatch)
I20260709 12:19:34.220556 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.225278 11310 pending_rounds.cc:70] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
W20260709 12:19:34.158964 29300 consensus_meta.cc:287] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.517s	user 0.006s	sys 0.007s
I20260709 12:19:34.228421 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.236132 12646 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:34.234704 11310 pending_rounds.cc:77] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 49 index: 30 } timestamp: 7305623850026291200 op_type: NO_OP noop_request { }
I20260709 12:19:34.240245 11310 raft_consensus.cc:2889] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:34.243700 11310 raft_consensus.cc:2272] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:34.243873 29860 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:34.244357 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:34.244980 12564 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:34.245815 12641 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
W20260709 12:19:34.247143 29811 consensus_queue.cc:1512] T 27144fedf0014de09f007ac6cc3f7275 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Unable to notify RaftConsensus of commit index change.: Service unavailable: Thread pool token was shut down
W20260709 12:19:34.247185 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52274.2us
W20260709 12:19:34.252727 12610 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.253335 12610 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.254304 11310 tablet_replica.cc:333] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:34.256206 11310 raft_consensus.cc:2243] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 60 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:34.256994 11310 raft_consensus.cc:2272] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [term 60 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:34.260367 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:34.261821 12566 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:34.242332 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:34.263329 12565 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:34.264433 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.266184 12646 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:34.271934 12639 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 47: Already voted for candidate 6d4ece942fd5414998215d26edf91001 in this term.
I20260709 12:19:34.272078 12570 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 57 index: 28. Preceding OpId from leader: term: 59 index: 29. (index mismatch)
I20260709 12:19:34.232924 29784 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:34.286823 12563 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 44. Current term is 45. Ops: []
I20260709 12:19:34.266541 29860 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:34.273253 29747 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.308981 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.302980 12462 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:34.317458 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.291965 29861 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.029s
I20260709 12:19:34.312040 29798 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.323868 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:34.335599 12639 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
W20260709 12:19:34.342185 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:34.347944 12461 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.790s (SENT) [suppressed 148 similar messages]
W20260709 12:19:34.339536 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:34.347571 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.362725 29898 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:34.363232 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.363962 29798 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:34.364795 29691 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:34.362480 29645 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.363391 29898 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:34.368501 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:34.370467 12565 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 58.
I20260709 12:19:34.370967 12563 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:34.371778 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.372828 12639 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 52 because replica has last-logged OpId of term: 51 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 30.
W20260709 12:19:34.375593 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 122921us
I20260709 12:19:34.360289 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:34.376592 12568 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:34.376344 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:34.348804 29652 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
W20260709 12:19:34.372458 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 77679.6us
I20260709 12:19:34.386085 13261 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 43 index: 26. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:19:34.387995 29783 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:19:34.308036 29747 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.395700 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:34.396507 12610 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.411576 29747 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.377662 29784 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.27, Last appended by leader: 27, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.357337 29300 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.061s
I20260709 12:19:34.402410 12563 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:34.407362 11310 tablet_replica.cc:333] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:34.411579 12610 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:34.415663 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 43500.6us
I20260709 12:19:34.405296 12568 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:34.418118 12460 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:34.419275 29860 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:34.420154 29773 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader election won for term 58
I20260709 12:19:34.420727 29773 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:34.421064 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.421372 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.421728 29641 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.422014 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.423139 12640 raft_consensus.cc:2410] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 52 because replica has last-logged OpId of term: 51 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 29.
I20260709 12:19:34.423172 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.422068 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.424444 12644 raft_consensus.cc:2393] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 51: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:34.421763 29773 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 56.32, Last appended by leader: 32, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.429280 11310 raft_consensus.cc:2243] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Raft consensus shutting down.
W20260709 12:19:34.429811 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.430159 11310 raft_consensus.cc:2272] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:34.373035 29898 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:34.462278 12566 raft_consensus.cc:2468] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:34.444267 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.437973 12460 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:34.479552 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:34.482363 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44490.8us
I20260709 12:19:34.395242 29652 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:34.483738 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:34.441273 29895 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.107s	user 0.002s	sys 0.000s
I20260709 12:19:34.484727 12643 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
W20260709 12:19:34.485558 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.486404 11310 tablet_replica.cc:333] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:34.486735 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.486080 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:34.435211 29641 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:34.484159 12609 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:34.490971 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
I20260709 12:19:34.499217 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.498834 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.497248 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.488353 29652 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:34.496785 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.540139 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.541128 12642 raft_consensus.cc:2376] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45. Re-sending same reply.
I20260709 12:19:34.543009 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.544610 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.545483 12646 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:34.546255 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.546830 11310 raft_consensus.cc:2243] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:34.549682 12640 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 44: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:34.487388 29300 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:34.566910 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.569818 12642 raft_consensus.cc:1240] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 59. Current term is 60. Ops: [59.38-59.38]
I20260709 12:19:34.570636 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.571446 12643 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 40.
I20260709 12:19:34.571722 12638 raft_consensus.cc:2410] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 51 because replica has last-logged OpId of term: 50 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 30.
I20260709 12:19:34.497170 29784 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:34.593154 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.594257 12646 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 49.
W20260709 12:19:34.594898 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:34.593619 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.528211 29679 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:34.528460 29771 raft_consensus.cc:2804] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:34.533913 29777 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:34.604485 29777 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:34.604043 29679 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.605849 29777 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 54.30, Last appended by leader: 30, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.607241 29679 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:34.596787 12610 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.534125 29770 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:34.491528 29747 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.629837 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.549137 29641 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:34.636181 11310 raft_consensus.cc:2272] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:34.639964 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.613263 29860 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:19:34.759812 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:34.760231 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.539063 29778 raft_consensus.cc:493] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:34.749135 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.755421 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 45 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:34.766546 12646 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:34.761215 12644 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 25. Preceding OpId from leader: term: 45 index: 27. (index mismatch)
I20260709 12:19:34.602279 29300 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:34.775224 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:34.784268 12536 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:34.756860 29605 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:34.756490 29771 raft_consensus.cc:697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:34.770887 29587 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.782176 29770 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.817025 29822 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.817446 29707 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.817793 29822 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.818068 29300 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.29, Last appended by leader: 29, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.818243 29707 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.820144 29822 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:34.820526 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 326390us
I20260709 12:19:34.822055 12638 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 51 index: 30. Preceding OpId from leader: term: 52 index: 30. (term mismatch)
I20260709 12:19:34.822726 12638 pending_rounds.cc:85] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 29
I20260709 12:19:34.823028 12638 pending_rounds.cc:107] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 51.30
I20260709 12:19:34.823493 12638 raft_consensus.cc:2889] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:34.821179 29771 consensus_queue.cc:237] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 49.27, Last appended by leader: 27, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.812127 29605 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.824193 29641 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 49.31, Last appended by leader: 31, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.797941 29773 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:34.771246 29747 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.782829 29811 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:34.829538 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.832496 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 50 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:34.838251 12638 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
W20260709 12:19:34.843326 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 249728us
I20260709 12:19:34.843626 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:34.844553 12610 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.844985 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.834676 29773 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.801376 29778 raft_consensus.cc:515] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.848567 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:34.820530 29707 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.817054 29770 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:34.829085 29605 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:34.866732 12610 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.868278 12610 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.867223 13218 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:34.872054 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 320133us
I20260709 12:19:34.860368 29679 raft_consensus.cc:493] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:34.827189 29587 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.867241 29860 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:34.835944 29652 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:34.898897 12610 leader_election.cc:336] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.840149 29811 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:34.905038 12646 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 45.
I20260709 12:19:34.909121 29783 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
W20260709 12:19:34.907016 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:34.894923 29912 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader election won for term 56
I20260709 12:19:34.896873 29770 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
W20260709 12:19:34.920964 12610 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.910065 29783 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.904142 29778 leader_election.cc:290] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:34.927554 12610 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.932303 12610 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:34.936944 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:34.941555 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.933439 29912 raft_consensus.cc:697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 56 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:34.943444 29912 consensus_queue.cc:237] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 54.32, Last appended by leader: 32, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:34.947223 11310 tablet_replica.cc:333] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:34.947456 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:34.951365 29811 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.952952 29747 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:34.955394 29775 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:34.955528 29783 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:34.901784 29679 raft_consensus.cc:515] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:34.958611 29909 log.cc:927] Time spent T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:19:34.967806 13218 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:34.949262 29770 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:34.966145 29798 consensus_meta.cc:287] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.602s	user 0.006s	sys 0.011s
I20260709 12:19:34.971797 29775 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:35.000404 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 35 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.003906 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.023320 11310 raft_consensus.cc:2243] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Raft consensus shutting down.
W20260709 12:19:35.026162 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58476.6us
I20260709 12:19:34.989800 12461 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:34.983390 29770 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:35.040781 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:35.022835 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:34.983472 29914 raft_consensus.cc:2804] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:19:35.042455 12536 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:35.047535 12536 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:35.049464 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:35.044957 29920 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.040417 29679 leader_election.cc:290] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:35.120507 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 71091.7us
W20260709 12:19:35.053047 29898 consensus_meta.cc:287] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.005s
I20260709 12:19:35.050386 29691 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:35.049731 29914 raft_consensus.cc:697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:35.043071 29777 consensus_queue.cc:1059] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 58.37, Next index: 38, Last known committed idx: 36, Time since last communication: 0.066s
I20260709 12:19:35.027233 29798 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.142467 29798 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.128569 29920 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:35.122879 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:35.046695 29770 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.134313 29923 raft_consensus.cc:3060] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:35.145125 29920 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.145571 29770 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.146589 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.148448 29925 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:35.151217 29798 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.157006 29914 consensus_queue.cc:237] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 49.28, Last appended by leader: 28, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.161373 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:35.123768 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 194643us
I20260709 12:19:35.117084 29652 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:35.128224 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 145503us
I20260709 12:19:35.164558 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.161448 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:35.165185 12461 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:35.138232 11310 pending_rounds.cc:70] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
I20260709 12:19:35.166632 11310 pending_rounds.cc:77] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 51 index: 32 } timestamp: 7305623858489901056 op_type: NO_OP noop_request { }
I20260709 12:19:35.166730 13255 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:35.167032 11310 raft_consensus.cc:2889] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:35.167119 12611 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.167560 11310 raft_consensus.cc:2272] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:35.165665 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.169188 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.172125 29517 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:35.175660 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 134973us
I20260709 12:19:35.178247 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:35.181533 12610 leader_election.cc:336] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:35.182991 12461 raft_consensus.cc:2686] T 92ef671b99df423ea5602f8b7ccff307 P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:35.191694 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 57 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.197834 12461 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:35.193369 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:35.198558 12536 leader_election.cc:336] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Timed out: RequestConsensusVote RPC to 127.11.11.131:35471 timed out after 1.587s (SENT)
I20260709 12:19:35.200284 12536 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:19:35.204663 29926 consensus_queue.cc:1175] T 7816534d4a0942f0af150ea0a94392b9 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6d4ece942fd5414998215d26edf91001" status { last_received { term: 51 index: 32 } last_committed_idx: 31 last_received_current_leader { term: 51 index: 32 } }
I20260709 12:19:35.205861 29919 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:35.201277 29777 raft_consensus.cc:3055] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 LEADER]: Stepping down as leader of term 59
I20260709 12:19:35.202018 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 60 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.207698 12638 raft_consensus.cc:2376] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 60. Re-sending same reply.
I20260709 12:19:35.208861 29773 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.201062 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.209543 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 59 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:35.182992 12610 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:35.206570 29777 raft_consensus.cc:740] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:35.211678 29777 consensus_queue.cc:260] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 59.38, Last appended by leader: 38, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.212447 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.213279 12638 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:35.213224 29777 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Advancing to term 60
I20260709 12:19:35.210414 12644 raft_consensus.cc:2376] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 59. Re-sending same reply.
I20260709 12:19:35.201148 29861 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:35.213946 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.219178 12640 raft_consensus.cc:2376] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48. Re-sending same reply.
I20260709 12:19:35.215739 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.218007 12570 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:35.225972 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.226887 12570 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 43.
W20260709 12:19:35.218562 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.185237 29771 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:35.230261 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:35.230515 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:35.233031 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.234439 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:35.236857 12569 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:35.182526 29679 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:35.236379 12570 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 52.
I20260709 12:19:35.236744 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:35.239115 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.239755 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:35.269039 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 70682.9us
I20260709 12:19:35.277594 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.170861 29587 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:35.138386 29898 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:35.281152 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 65514.6us
W20260709 12:19:35.312423 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.281750 29911 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:35.316522 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.282349 29927 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:35.317332 29927 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.317811 29927 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:35.318287 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 57 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.318702 12566 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 56 index: 32. Preceding OpId from leader: term: 58 index: 33. (index mismatch)
I20260709 12:19:35.283205 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.281774 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:35.279199 12564 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 28. Preceding OpId from leader: term: 51 index: 29. (index mismatch)
I20260709 12:19:35.317508 13254 raft_consensus.cc:2393] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:35.280323 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:35.284970 29679 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.286135 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.274240 29739 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:35.335520 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54005us
I20260709 12:19:35.335877 29739 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:35.336269 29739 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.336591 29739 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:35.336835 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:35.337821 29707 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:35.338352 29707 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.338728 29707 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:35.228510 29861 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.341120 29861 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.342284 13257 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:35.344369 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 123566us
I20260709 12:19:35.287242 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.287741 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.346665 12637 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 51
I20260709 12:19:35.287927 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.289613 29923 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:35.228170 29906 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:35.293243 11310 tablet_replica.cc:333] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:35.294124 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:35.359444 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 65390.6us
I20260709 12:19:35.295429 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.296975 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.298949 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:35.183389 29554 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.059s
W20260709 12:19:35.308365 13217 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.182847 29652 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.069s
I20260709 12:19:35.282119 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.321015 12640 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 57 index: 28. Preceding OpId from leader: term: 59 index: 29. (index mismatch)
W20260709 12:19:35.322979 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:35.366528 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.324713 12641 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:35.286525 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.327661 12611 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.329182 29920 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.001s
I20260709 12:19:35.316525 29911 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.287029 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.395035 13255 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 58.
I20260709 12:19:35.395584 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 55.
I20260709 12:19:35.350802 13260 raft_consensus.cc:1240] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 51. Current term is 52. Ops: [51.30-51.30]
I20260709 12:19:35.358357 12645 raft_consensus.cc:2393] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 56: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:35.362661 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 79541.4us
I20260709 12:19:35.368161 12646 raft_consensus.cc:2376] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51. Re-sending same reply.
I20260709 12:19:35.360194 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:35.369570 13262 raft_consensus.cc:2393] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 53: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:35.371035 12611 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.402045 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.403148 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.398988 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.407685 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:35.408528 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.414392 29938 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:35.416188 29938 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:35.400328 29911 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:35.419063 29924 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.283s	user 0.003s	sys 0.000s
I20260709 12:19:35.407519 29641 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.014s
I20260709 12:19:35.370169 29771 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.422561 29923 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:35.422326 29771 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.423000 29923 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.423362 29923 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:35.424983 29771 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.408002 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:35.426510 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 48 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.426716 12460 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:35.427093 12640 raft_consensus.cc:2376] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48. Re-sending same reply.
I20260709 12:19:35.428023 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92ef671b99df423ea5602f8b7ccff307" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.430402 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d03683a6ca224dc89740dc91f644b357" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.431571 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.432212 12645 raft_consensus.cc:2393] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 in current term 51: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:35.432888 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.434173 29300 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:35.434643 29300 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.434978 29300 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:35.435308 12572 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 48 index: 30. (index mismatch)
I20260709 12:19:35.436372 29921 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20260709 12:19:35.437978 29921 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:35.438361 29921 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.438438 29894 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:35.438779 29921 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:35.438905 29894 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.439287 29894 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:35.429126 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.442035 29773 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.442830 29773 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.441929 12569 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:35.447602 12646 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 29. Preceding OpId from leader: term: 48 index: 30. (index mismatch)
I20260709 12:19:35.448069 29933 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:35.449810 29747 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:35.459758 29898 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.465376 29517 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.466323 29517 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.469851 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.369875 29906 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.469347 29517 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.478320 29517 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.479173 29517 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.449647 29916 raft_consensus.cc:2764] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election decision vote started in defunct term 51: won
W20260709 12:19:35.483309 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 75740.4us
I20260709 12:19:35.495256 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 44 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:35.500052 12641 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:35.502991 12637 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
W20260709 12:19:35.514269 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 105830us
I20260709 12:19:35.491533 12460 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:35.516009 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 187794us
I20260709 12:19:35.523514 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.524183 12645 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:35.525470 12461 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.510756 29517 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.505784 29773 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.511267 29898 raft_consensus.cc:3060] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:35.536510 12461 raft_consensus.cc:2686] T a3a0781287514c758f107d79c1f96be4 P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:35.526957 29778 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
I20260709 12:19:35.533828 29652 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
W20260709 12:19:35.599042 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 197039us
I20260709 12:19:35.621412 12645 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 48.
I20260709 12:19:35.584971 29679 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:35.664441 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 123840us
W20260709 12:19:35.667835 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146126us
I20260709 12:19:35.669209 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.674957 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.658406 29775 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:35.706017 12611 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:35.707772 29739 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.709013 11310 raft_consensus.cc:2243] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:35.709952 29943 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.710644 29943 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.693238 29906 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.690274 29775 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.715751 29861 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20260709 12:19:35.717351 29517 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.718221 29517 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.723912 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.710152 29679 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:35.701511 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:35.728432 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.684688 29652 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:35.686873 29771 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:35.684628 29938 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.731402 29822 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:35.739470 29933 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:35.738971 29945 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.738996 29938 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.740281 29771 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.740742 29945 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.742951 29945 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.743199 29938 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.744128 29933 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.746547 29933 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.733345 12571 raft_consensus.cc:2393] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:35.735646 12461 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.726536 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.736804 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.720392 29517 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:35.731617 29860 consensus_meta.cc:287] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.826s	user 0.015s	sys 0.000s
I20260709 12:19:35.727540 29943 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.756907 29860 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:35.717499 29775 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.758031 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:35.754186 11310 pending_rounds.cc:70] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
I20260709 12:19:35.759137 11310 pending_rounds.cc:77] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 49 index: 27 } timestamp: 7305623770295848960 op_type: NO_OP noop_request { }
I20260709 12:19:35.759579 11310 raft_consensus.cc:2889] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:35.760017 11310 raft_consensus.cc:2272] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:35.686031 29707 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.727699 29679 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:35.777272 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53831.1us
I20260709 12:19:35.761600 29898 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:35.742978 29771 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:35.786079 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 93475.1us
I20260709 12:19:35.770759 29933 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.784754 29916 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:35.787283 29300 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.792814 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.797549 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.787220 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 46 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:35.798918 12461 raft_consensus.cc:2686] T 9bbf126b18234b78b4f36bf9708ad96b P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:35.782408 29679 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:35.815274 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.817276 29739 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:35.824332 29927 consensus_meta.cc:287] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.007s
I20260709 12:19:35.805042 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.809118 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.810292 29775 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:35.810837 29860 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:35.811897 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:35.812930 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.834429 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.835129 12563 raft_consensus.cc:2393] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 45: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:35.803544 29771 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.837733 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.838311 29778 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:35.837906 29771 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.840569 29300 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:35.837957 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 102382us
I20260709 12:19:35.843916 29707 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:35.844699 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 185378us
I20260709 12:19:35.845824 29944 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:35.846174 29860 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.846561 29944 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.849114 29860 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.814558 12572 raft_consensus.cc:1275] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:19:35.861987 12536 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.757086 29652 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 50.26, Last appended by leader: 26, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.832329 12645 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 54 index: 30. Preceding OpId from leader: term: 55 index: 31. (index mismatch)
I20260709 12:19:35.836879 29777 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.851866 29911 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.828202 29933 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:35.855623 29935 log.cc:927] Time spent T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.380s	user 0.004s	sys 0.001s
I20260709 12:19:35.874140 29739 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.008s
I20260709 12:19:35.882433 29778 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:35.874272 29919 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:35.798473 29894 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.884292 29919 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:35.884836 29919 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:35.884935 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:35.885819 12570 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:35.886230 29894 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:35.887253 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:35.890417 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.891822 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.892719 29953 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20260709 12:19:35.893519 29953 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260709 12:19:35.894513 29861 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:35.895159 29861 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.895545 29921 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.859077 29775 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.897933 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:35.855919 29956 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
W20260709 12:19:35.855710 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 43883.8us
I20260709 12:19:35.828415 29923 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.814198 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.853374 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:35.849107 29944 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:35.868719 29955 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:35.911350 29937 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.453s	user 0.002s	sys 0.003s
I20260709 12:19:35.879031 29777 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:35.908301 11310 tablet_replica.cc:333] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:35.915453 29777 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:35.917063 11310 raft_consensus.cc:2243] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:19:35.918473 11310 raft_consensus.cc:2272] T f02aa0ab5af34d0e9ef3a4956449c624 P 6d4ece942fd5414998215d26edf91001 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:35.919066 29955 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.925127 11310 tablet_replica.cc:333] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:35.901611 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 48360.5us
I20260709 12:19:35.907341 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.926808 11310 raft_consensus.cc:2243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:19:35.841336 29771 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.910882 29921 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:35.929109 29915 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:35.949064 29915 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:35.949772 29915 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:35.929291 29956 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:35.915364 12461 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:35.918787 29923 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:35.934334 29861 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:35.934708 29955 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.938169 29679 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.918551 29933 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.939452 29775 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:35.993191 11310 raft_consensus.cc:2272] T 8bd1dcd3e8a44986816c86ea6e580df5 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:35.833514 29927 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:35.940400 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 78508.1us
I20260709 12:19:35.940901 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:35.941849 13256 raft_consensus.cc:1240] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
W20260709 12:19:35.902072 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:35.942503 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:35.942718 29911 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:36.001855 12571 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 51 index: 30. (index mismatch)
I20260709 12:19:35.946609 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.003201 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:36.004124 12563 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:35.963246 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.004367 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:35.967561 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.967885 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.967510 29860 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:35.933388 29919 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:35.963743 12570 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
W20260709 12:19:36.007128 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91818.8us
I20260709 12:19:35.980089 13259 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 56 index: 32. Preceding OpId from leader: term: 58 index: 33. (index mismatch)
I20260709 12:19:35.987241 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:36.008498 12569 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:35.992833 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 53 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:35.994262 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:36.012696 12568 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 45
I20260709 12:19:36.015240 12461 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:35.994472 12610 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:35.953015 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 59 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:36.016872 12610 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.017093 13255 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 58.
I20260709 12:19:36.007753 12564 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:36.022210 12567 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
W20260709 12:19:36.024130 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44811.9us
I20260709 12:19:36.018774 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:36.032534 13217 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:36.030673 12610 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.050436 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.052492 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.053478 12638 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 44.
I20260709 12:19:36.057114 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:35.997789 29921 raft_consensus.cc:3060] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:36.062932 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.063917 12638 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:36.064637 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.065786 12640 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:36.067543 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.068589 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 47.
I20260709 12:19:36.066251 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:36.079674 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 47255.3us
I20260709 12:19:36.006824 29778 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:36.083387 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:36.085351 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 54712.3us
I20260709 12:19:36.028347 29771 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:36.088531 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.089655 12639 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 51 index: 29. (index mismatch)
I20260709 12:19:36.092728 12569 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:36.046363 29861 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader pre-election won for term 59
I20260709 12:19:36.092362 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 54 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:36.087970 29605 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.032s	sys 0.069s
I20260709 12:19:36.128136 12642 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:36.100828 12563 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:36.101652 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 203467us
I20260709 12:19:36.058800 29641 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:36.139715 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.140532 12646 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 52
I20260709 12:19:36.143657 12568 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
W20260709 12:19:36.146127 29936 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.039s	sys 0.078s
W20260709 12:19:36.146857 29936 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.039s	sys 0.078s
I20260709 12:19:36.148154 29919 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:36.148974 29953 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.149596 29953 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:36.153882 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 155953us
I20260709 12:19:36.154825 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.029045 29894 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.064s
I20260709 12:19:36.182523 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.166235 12460 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:36.036145 29775 consensus_queue.cc:1048] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.056s
W20260709 12:19:36.154918 12610 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.185539 12610 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.186591 29517 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
W20260709 12:19:36.187232 12610 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.187835 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.188767 29938 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:36.189132 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:36.136188 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.089578 29945 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:36.085772 29784 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.088958 29911 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:36.028268 29860 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.171098 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:36.133987 29605 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.032s	sys 0.070s
I20260709 12:19:36.134507 29652 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:36.230276 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.231644 12639 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 46
I20260709 12:19:36.190292 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.083292 29933 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.133167 29861 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:36.199489 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.136166 29822 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.232883 12646 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
W20260709 12:19:36.233743 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 101607us
I20260709 12:19:36.234097 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.237099 12646 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 49
W20260709 12:19:36.237568 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48460.2us
W20260709 12:19:36.238852 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.239506 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:36.243961 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61489.7us
I20260709 12:19:36.250612 12646 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:36.253190 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.254280 12646 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:36.254974 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.197393 29898 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:36.197613 29948 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:36.198390 29911 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:36.245816 29961 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:36.271850 12646 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 45.
I20260709 12:19:36.280766 12642 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
I20260709 12:19:36.224665 29919 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:19:36.235711 29861 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Advancing to term 59
I20260709 12:19:36.283413 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.246271 29771 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:36.197757 29300 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.074s
I20260709 12:19:36.191196 29955 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.078s
I20260709 12:19:36.267515 29916 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20260709 12:19:36.322172 12645 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 29. Preceding OpId from leader: term: 51 index: 30. (index mismatch)
I20260709 12:19:36.224858 29784 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:36.335098 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98077.2us
I20260709 12:19:36.348522 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.203053 29570 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.080s
I20260709 12:19:36.353693 12646 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 52
I20260709 12:19:36.299810 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 50 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.234190 29652 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.287714 29911 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:36.358377 12642 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:36.244692 29822 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.285495 29898 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:36.251784 29933 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.367386 12461 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:36.285725 29953 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:36.374445 12646 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
W20260709 12:19:36.377990 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 148405us
I20260709 12:19:36.290939 29919 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:36.382356 12460 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:36.386968 12461 raft_consensus.cc:2686] T a16c8cd5242e4ab6840a3a328938b5df P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:36.388582 12461 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:36.393653 12460 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:36.361991 29948 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20260709 12:19:36.290498 29778 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.083s
I20260709 12:19:36.393867 11310 tablet_replica.cc:333] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:36.407706 12461 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.406338 29923 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:36.419034 29784 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:36.422473 29860 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:36.424451 29898 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 50.31, Last appended by leader: 31, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.425522 29652 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:36.429610 29933 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:36.431571 29822 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:36.390394 29919 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:19:36.431914 29962 log.cc:927] Time spent T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:19:36.442981 12642 raft_consensus.cc:2410] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d4ece942fd5414998215d26edf91001 for term 55 because replica has last-logged OpId of term: 54 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 21.
I20260709 12:19:36.437330 12639 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 46.
I20260709 12:19:36.469785 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.471668 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.472647 12565 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
I20260709 12:19:36.491932 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.411286 29641 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:36.498950 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:36.499615 12570 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:36.500492 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:36.490948 29784 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:36.495975 29747 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader election won for term 56
W20260709 12:19:36.511195 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 227621us
I20260709 12:19:36.499984 29300 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:36.506322 12460 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:36.505208 29955 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:36.507423 29641 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:36.551556 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.550319 12638 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 52.
I20260709 12:19:36.499290 29952 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.286s
I20260709 12:19:36.530911 29747 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:36.563333 29915 consensus_meta.cc:287] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.006s
W20260709 12:19:36.578460 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 61543.9us
W20260709 12:19:36.581431 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 173791us
W20260709 12:19:36.579432 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 340585us
I20260709 12:19:36.553985 29300 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.585480 12461 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
W20260709 12:19:36.582882 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:36.567016 29948 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:36.600052 29784 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:36.595667 12610 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.600455 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:36.579865 29587 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:36.558670 29955 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:36.607951 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:36.611961 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.612453 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:36.613744 12610 leader_election.cc:336] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.614151 12610 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:36.617368 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 111144us
I20260709 12:19:36.624275 29771 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:36.625255 29860 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:36.601887 29641 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.27, Last appended by leader: 27, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.635685 29927 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:36.628854 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.639770 29948 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:36.631193 29955 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:36.610731 29587 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:36.642138 29587 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:36.646183 29973 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:19:36.604033 29773 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:36.648586 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:36.650099 29971 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20260709 12:19:36.607026 29978 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.049s
I20260709 12:19:36.577083 29915 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.653474 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:36.641352 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.676616 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 52 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:36.677455 12568 raft_consensus.cc:3060] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 54
I20260709 12:19:36.663257 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:36.684773 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:36.695005 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 81394.1us
W20260709 12:19:36.697068 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.697525 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:36.695202 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 91809.8us
W20260709 12:19:36.657505 29976 log.cc:927] Time spent T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.163s	user 0.003s	sys 0.000s
W20260709 12:19:36.700219 12610 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.653486 29915 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:36.658118 29300 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:36.636515 29927 consensus_queue.cc:1059] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 49.34, Next index: 35, Last known committed idx: 34, Time since last communication: 0.059s
I20260709 12:19:36.690169 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.686893 29982 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.708659 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.693502 12568 raft_consensus.cc:2468] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 54.
I20260709 12:19:36.703522 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.711105 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.719336 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 54 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.722441 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 60 candidate_status { last_received { term: 57 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.713047 13217 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:36.697270 29773 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:36.699813 29955 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:36.728624 29915 raft_consensus.cc:2804] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader election won for term 54
I20260709 12:19:36.694913 29923 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:36.735205 29915 raft_consensus.cc:697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:36.735131 29911 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.735659 29923 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 44.25, Last appended by leader: 25, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.736397 29915 consensus_queue.cc:237] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 52.33, Last appended by leader: 33, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.701298 29916 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:36.724866 29990 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:36.732465 12460 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:36.701463 29945 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:36.686079 29554 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.111s
I20260709 12:19:36.754045 29784 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.649428 29747 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 54.30, Last appended by leader: 30, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.718145 29300 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:36.733065 29707 raft_consensus.cc:3055] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:36.724902 29861 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.730995 29860 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:36.726630 29982 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.739843 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.783629 29679 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:36.783874 12571 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:36.740041 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:36.745388 12563 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 26. Preceding OpId from leader: term: 52 index: 27. (index mismatch)
I20260709 12:19:36.785838 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:36.775125 12564 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 58. Current term is 59. Ops: [58.33-58.33]
I20260709 12:19:36.778308 12566 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:36.780306 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.792591 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:36.780833 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 69805.9us
W20260709 12:19:36.739125 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 97898.7us
I20260709 12:19:36.796036 12536 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.798537 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.781330 29927 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:36.748173 29773 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.803921 29773 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:36.738040 29916 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:36.769945 29990 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Leader election won for term 57
I20260709 12:19:36.779377 29982 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:36.774889 29861 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 59 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:36.807909 29916 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.737587 29911 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:36.809139 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.817212 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.818173 12646 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:19:36.825102 29982 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election won for term 45
I20260709 12:19:36.826031 29982 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:36.826536 29982 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:36.831656 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:36.833995 12571 raft_consensus.cc:2393] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:36.871510 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.872550 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 48.
W20260709 12:19:36.809119 29921 consensus_meta.cc:287] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: Time spent flushing consensus metadata: real 0.723s	user 0.003s	sys 0.010s
I20260709 12:19:36.874954 29921 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:36.876430 12640 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 26. Preceding OpId from leader: term: 52 index: 27. (index mismatch)
I20260709 12:19:36.766156 29707 raft_consensus.cc:740] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:36.880241 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 55 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.880908 12642 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 57
I20260709 12:19:36.888890 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 50 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.889114 11310 raft_consensus.cc:2243] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:19:36.765532 29587 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:36.889076 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 57299.6us
W20260709 12:19:36.900120 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:36.900445 12643 raft_consensus.cc:1275] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 53 index: 31. Preceding OpId from leader: term: 56 index: 33. (index mismatch)
W20260709 12:19:36.891206 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 82416.3us
I20260709 12:19:36.895920 12638 raft_consensus.cc:1275] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 46 index: 27. Preceding OpId from leader: term: 51 index: 29. (index mismatch)
I20260709 12:19:36.904495 12642 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 57.
I20260709 12:19:36.863351 29990 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:36.876183 29861 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election won for term 48
W20260709 12:19:36.908774 12610 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.926374 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.931217 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 45 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.933585 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.935407 12460 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:36.937140 12638 raft_consensus.cc:1240] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 54. Current term is 55. Ops: []
I20260709 12:19:36.945513 12638 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 52. Ops: []
I20260709 12:19:36.946041 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 59 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:36.949091 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.949887 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:36.950174 12645 raft_consensus.cc:3060] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:36.923362 29955 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:36.988190 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 79470.8us
I20260709 12:19:36.907217 29913 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.043s
W20260709 12:19:36.989111 12610 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:36.929373 29861 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:37.003886 29861 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 47.27, Last appended by leader: 27, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:36.992451 12610 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.940989 29707 consensus_queue.cc:260] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 49.34, Last appended by leader: 34, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:36.955482 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:37.031157 12644 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:36.959254 29921 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:37.018949 12610 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.036868 12610 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:36.989768 29770 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:37.014878 29919 consensus_meta.cc:287] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.001s
W20260709 12:19:37.045070 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 166857us
I20260709 12:19:37.005939 29993 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:36.947880 29923 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.069s
I20260709 12:19:37.058182 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 59 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:37.043221 12645 raft_consensus.cc:2468] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:37.016371 29961 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 60 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:36.953346 29898 consensus_queue.cc:1059] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 49.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.086s
I20260709 12:19:37.069011 29707 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:37.065320 29860 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:37.071858 29927 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:37.039404 29630 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.008378 29955 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.050658 29517 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.070681 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:32884 (request call id 6353) took 1006 ms. Trace:
W20260709 12:19:37.151654 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 132746us
I20260709 12:19:37.151943 12564 rpcz_store.cc:276] 0709 12:19:36.064018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.064173 (+   155us) service_pool.cc:224] Handling call
0709 12:19:36.069363 (+  5190us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:36.806620 (+737257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.070338 (+263718us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b5400181d54. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:37.070601 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1622460672}
W20260709 12:19:37.152557 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.152971 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.086009 29961 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 60 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.059459 29919 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:37.163599 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 49 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:37.139801 29679 raft_consensus.cc:2749] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:19:37.173429 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 120431us
I20260709 12:19:37.175602 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 59 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:37.039496 29990 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 55.31, Last appended by leader: 31, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:37.169103 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.178471 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.164055 29919 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:37.168392 29943 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:37.150787 29630 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.197060 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:37.198052 12638 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:37.182602 29587 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:37.193137 13218 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:37.164542 29770 consensus_queue.cc:1059] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 56.32, Next index: 33, Last known committed idx: 32, Time since last communication: 0.025s
W20260709 12:19:37.207433 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:37.160799 29517 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.212694 29961 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 61 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:37.191718 29860 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:37.222504 12638 raft_consensus.cc:2468] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
I20260709 12:19:37.236063 13259 raft_consensus.cc:3055] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 LEADER]: Stepping down as leader of term 55
W20260709 12:19:37.238669 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:19:37.239822 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.244120 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.240449 13259 raft_consensus.cc:740] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:37.224970 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:37.207114 30002 log.cc:927] Time spent T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:19:37.256313 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:37.259186 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 334094us
W20260709 12:19:37.233426 29997 log.cc:927] Time spent T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
W20260709 12:19:37.271986 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 63827.2us
I20260709 12:19:37.271957 13259 consensus_queue.cc:260] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 55.31, Last appended by leader: 31, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.278332 13259 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:37.282723 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 61 candidate_status { last_received { term: 59 index: 38 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:37.283757 12637 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 60.
I20260709 12:19:37.283838 12640 raft_consensus.cc:1240] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 55. Current term is 56. Ops: [55.31-55.31]
W20260709 12:19:37.285033 12536 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.285699 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:37.276387 30012 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:37.284327 29770 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:37.304087 29955 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:37.305958 29630 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:37.307794 29982 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:37.325462 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 47 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:37.326584 12572 raft_consensus.cc:2410] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e for term 51 because replica has last-logged OpId of term: 49 index: 30, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 29.
I20260709 12:19:37.334049 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.344269 29778 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20260709 12:19:37.358373 29570 raft_consensus.cc:3055] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:37.358924 29570 raft_consensus.cc:740] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:37.359930 29570 consensus_queue.cc:260] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 49.25, Last appended by leader: 25, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:37.366786 29517 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:37.368938 11310 pending_rounds.cc:70] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
I20260709 12:19:37.369320 11310 pending_rounds.cc:77] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 49 index: 28 } timestamp: 7305623865903980544 op_type: NO_OP noop_request { }
I20260709 12:19:37.322587 29916 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:37.369717 11310 raft_consensus.cc:2889] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:37.370193 11310 raft_consensus.cc:2272] T 81a9a92616494f46bc97254b53185664 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:37.370980 30015 log.cc:927] Time spent T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:19:37.388022 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 54072.5us
I20260709 12:19:37.389077 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:37.393865 12572 raft_consensus.cc:1275] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 52 index: 33. Preceding OpId from leader: term: 54 index: 34. (index mismatch)
I20260709 12:19:37.395943 29944 consensus_queue.cc:1048] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
W20260709 12:19:37.382732 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.401404 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.403083 29982 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:37.422034 13259 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 55 index: 31. Preceding OpId from leader: term: 56 index: 31. (term mismatch)
I20260709 12:19:37.422590 13259 pending_rounds.cc:85] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 30
I20260709 12:19:37.422832 13259 pending_rounds.cc:107] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 55.31
I20260709 12:19:37.423035 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:37.423298 12567 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 31. Preceding OpId from leader: term: 52 index: 32. (index mismatch)
I20260709 12:19:37.423076 13259 raft_consensus.cc:2889] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:37.431954 13218 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:37.435442 29974 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:19:37.428520 29916 consensus_queue.cc:1059] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 54.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.008s
I20260709 12:19:37.432017 30012 consensus_queue.cc:1048] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.008s
I20260709 12:19:37.442624 12567 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 27. Preceding OpId from leader: term: 49 index: 28. (index mismatch)
I20260709 12:19:37.428757 29948 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
W20260709 12:19:37.456893 13220 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:37.459542 12567 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
W20260709 12:19:37.464344 12536 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.464890 12536 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:37.464946 12463 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:37.469985 12567 raft_consensus.cc:1240] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Rejecting Update request from peer 6d4ece942fd5414998215d26edf91001 for earlier term 49. Current term is 50. Ops: []
I20260709 12:19:37.474457 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 61 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:19:37.480787 12536 leader_election.cc:336] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.484273 29587 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:37.486593 29570 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:37.489265 29953 raft_consensus.cc:493] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.489825 29919 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:37.481348 12536 leader_election.cc:304] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.498585 29778 raft_consensus.cc:3055] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Stepping down as leader of term 54
I20260709 12:19:37.499076 29778 raft_consensus.cc:740] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:37.500034 29778 consensus_queue.cc:260] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 54.23, Last appended by leader: 23, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:37.501338 29778 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
I20260709 12:19:37.489977 29953 raft_consensus.cc:515] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.430696 29915 consensus_queue.cc:1048] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 29, Time since last communication: 0.036s
I20260709 12:19:37.451937 29993 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.041s
I20260709 12:19:37.478439 30012 raft_consensus.cc:2804] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
W20260709 12:19:37.512806 12536 leader_election.cc:336] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.481513 29916 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.519567 12536 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:37.510965 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.522135 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.493734 30017 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 60 FOLLOWER]: Leader pre-election won for term 61
I20260709 12:19:37.511103 29915 consensus_queue.cc:1243] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [LEADER]: Peer 32889a507a6b4c74aec421ffbfd1e25d log is divergent from this leader: its last log entry 53.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 29
I20260709 12:19:37.528617 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:37.528476 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 45 candidate_status { last_received { term: 44 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:37.544857 12569 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Advancing to term 45
I20260709 12:19:37.529480 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:37.530062 29982 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election lost for term 60. Reason: could not achieve majority
I20260709 12:19:37.531177 29919 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:37.511543 29948 raft_consensus.cc:2749] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:37.526320 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 143610us
W20260709 12:19:37.562089 12610 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.553738 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 61 candidate_status { last_received { term: 58 index: 37 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:37.511431 29994 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.037s
I20260709 12:19:37.556372 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:37.543125 11310 tablet_replica.cc:333] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:37.535823 30017 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:37.576140 11310 raft_consensus.cc:2243] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:19:37.556195 30003 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:37.535718 29916 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:37.590601 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:37.593447 11310 raft_consensus.cc:2272] T 1033139a62f44d368f8443d665a2cf22 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:37.570529 30017 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 60 FOLLOWER]: Advancing to term 61
W20260709 12:19:37.601018 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 88229.7us
W20260709 12:19:37.602047 12536 leader_election.cc:336] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:37.555972 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:37.602784 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46873.9us
I20260709 12:19:37.606004 12569 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:37.602609 12536 leader_election.cc:304] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.595288 29953 leader_election.cc:290] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:37.618925 12569 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
W20260709 12:19:37.660005 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57995.6us
I20260709 12:19:37.608124 29993 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:37.661952 12536 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.663228 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:37.666556 30012 raft_consensus.cc:697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
W20260709 12:19:37.672761 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 68381.6us
I20260709 12:19:37.662173 29861 raft_consensus.cc:2749] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:37.681293 29993 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:37.686530 29953 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:37.702183 13218 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.706638 29953 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.713258 29953 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:37.740454 30017 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 61 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.751047 30017 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 61 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:37.754487 13256 raft_consensus.cc:2393] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 61 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 61: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
W20260709 12:19:37.762323 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 100391us
I20260709 12:19:37.763303 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f701a23559945868ce6e3da5866f95c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:37.763394 12536 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:37.764479 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.764868 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.704293 29916 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:37.767289 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:37.769143 12536 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.775141 12536 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:37.777580 12462 leader_election.cc:304] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:37.783603 29587 raft_consensus.cc:2749] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:19:37.779268 29923 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:37.768385 29919 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:37.791090 30029 raft_consensus.cc:2804] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:37.703445 30012 consensus_queue.cc:237] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 44.30, Last appended by leader: 30, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:37.789780 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 61 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.797598 11310 tablet_replica.cc:333] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:37.813844 11310 raft_consensus.cc:2243] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:37.814663 11310 raft_consensus.cc:2272] T 43fecbd2edc14ba2ab065395e5d48d1d P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:37.817812 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.819517 29919 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I20260709 12:19:37.822559 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:37.787616 29923 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.826704 30003 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:19:37.878044 29822 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:19:37.884166 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 109122us
I20260709 12:19:37.884702 13259 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
W20260709 12:19:37.889139 12536 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.882527 11310 tablet_replica.cc:333] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:37.899329 12536 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:37.900163 12536 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:37.902436 12646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34824 (request call id 6305) took 1028 ms. Trace:
I20260709 12:19:37.903074 12646 rpcz_store.cc:276] 0709 12:19:36.878185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.880729 (+  2544us) service_pool.cc:224] Handling call
0709 12:19:36.881228 (+   499us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:36.903897 (+ 22669us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:36.903933 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:36.904739 (+   806us) log.cc:844] Serialized 52 byte log entry
0709 12:19:37.015909 (+111170us) raft_consensus.cc:1645] Marking committed up to 29
0709 12:19:37.048085 (+ 32176us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.484086 (+436001us) spinlock_profiling.cc:243] Waited 3 ms on lock 0x7b4800150258. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfa71caa1 00007ffbfa713b37 00007ffbfa7139ca 00007ffbfa80b26f 00007ffbfa80af37 00007ffbfa81f55c 00007ffbfa81f508 00007ffbfa81f488 00007ffbfa81f443 00007ffbfa81e6ec 00007ffbf9748a96
0709 12:19:37.593674 (+109588us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b180035a220. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf539b4bc 00007ffbf26e948b 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf57eb060 00007ffbf57fd716 00007ffbf580e287
0709 12:19:37.594098 (+   424us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.901941 (+307843us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.902081 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:37.902145 (+    64us) raft_consensus.cc:1685] finished
0709 12:19:37.902163 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:37.902359 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":512211,"prepare.queue_time_us":30614,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":5067,"raft.queue_time_us":57823,"raft.run_cpu_time_us":16362,"raft.run_wall_time_us":564356,"spinlock_wait_cycles":18900224,"thread_start_us":168100,"threads_started":2,"wal-append.queue_time_us":220973}
W20260709 12:19:37.914024 13218 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 61 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:37.911509 29773 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:37.924826 29916 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.034s
W20260709 12:19:37.947867 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:37.948720 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:37.923405 29956 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:37.950359 29956 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.951869 29771 raft_consensus.cc:3055] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Stepping down as leader of term 58
I20260709 12:19:37.952394 29771 raft_consensus.cc:740] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:37.952612 29956 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:37.953310 29771 consensus_queue.cc:260] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 58.33, Last appended by leader: 33, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:37.931782 29517 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:37.956858 29517 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:37.959419 29517 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:37.974105 12646 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 50 index: 31. Preceding OpId from leader: term: 52 index: 32. (index mismatch)
I20260709 12:19:37.976255 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:37.977221 12646 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:37.981850 11310 raft_consensus.cc:2243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 59 LEADER]: Raft consensus shutting down.
I20260709 12:19:37.904194 30029 raft_consensus.cc:697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:37.989926 11310 raft_consensus.cc:2272] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 6d4ece942fd5414998215d26edf91001 [term 59 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:37.965700 29771 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Advancing to term 59
I20260709 12:19:37.994865 12638 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 44 index: 25. Preceding OpId from leader: term: 46 index: 26. (index mismatch)
W20260709 12:19:38.001864 12460 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:38.002682 12646 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
I20260709 12:19:38.004546 12641 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 54 index: 30. Preceding OpId from leader: term: 56 index: 31. (index mismatch)
I20260709 12:19:38.008895 29923 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:38.011121 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:38.016912 12639 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 41 index: 24. Preceding OpId from leader: term: 45 index: 26. (index mismatch)
I20260709 12:19:38.017189 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.017829 12641 raft_consensus.cc:2468] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 56.
I20260709 12:19:38.020153 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.021811 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.022294 12637 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: []
I20260709 12:19:38.022511 12639 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:38.024187 12637 raft_consensus.cc:1240] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 48. Current term is 49. Ops: []
I20260709 12:19:38.028326 12609 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:38.025323 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.025923 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 41 candidate_status { last_received { term: 40 index: 21 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:38.031415 12646 raft_consensus.cc:3060] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:19:38.036340 29916 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 46.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 30
I20260709 12:19:38.041857 12646 raft_consensus.cc:2468] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 41.
I20260709 12:19:38.050688 12569 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 55 index: 31. Preceding OpId from leader: term: 57 index: 32. (index mismatch)
I20260709 12:19:38.055492 29784 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:38.051345 30029 consensus_queue.cc:237] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 51.28, Last appended by leader: 28, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.060263 12611 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.067340 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:38.067580 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:38.060405 29771 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:38.077286 12642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4926) took 1123 ms. Trace:
I20260709 12:19:38.084299 12642 rpcz_store.cc:276] 0709 12:19:36.953674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.953849 (+   175us) service_pool.cc:224] Handling call
0709 12:19:37.032501 (+ 78652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:19:37.032896 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 29
0709 12:19:37.234956 (+202060us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800316f60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf2708a40 00007ffbf27034e4 00007ffbf2703455 00007ffbf91eb3c7 00007ffbf57fd9a7 00007ffbf580d0bf 00007ffbf580d038
0709 12:19:37.235250 (+   294us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.511945 (+276695us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:19:37.514922 (+  2977us) log.cc:844] Serialized 26 byte log entry
0709 12:19:37.515126 (+   204us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:37.516730 (+  1604us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.850625 (+333895us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b1800316f60. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbf96c758f 00007ffbf2706571 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd
0709 12:19:37.862929 (+ 12304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.863025 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:37.863071 (+    46us) raft_consensus.cc:1685] finished
0709 12:19:37.863084 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.077035 (+213951us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b54002e5250. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f28ad 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6 00007ffbf3b2d26f
0709 12:19:38.077195 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":680224,"prepare.queue_time_us":204105,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":97,"raft.queue_time_us":585989,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":29201,"spinlock_wait_cycles":222055168,"thread_start_us":34091,"threads_started":1,"wal-append.queue_time_us":74403}
I20260709 12:19:38.095373 29517 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:38.096375 29517 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:38.097815 29990 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
W20260709 12:19:38.100854 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 72565.7us
I20260709 12:19:38.106503 12536 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:38.123862 12642 raft_consensus.cc:1240] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 49. Current term is 52. Ops: []
I20260709 12:19:38.097033 29517 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.139415 30038 raft_consensus.cc:2804] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:19:38.141490 12461 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:38.125953 30042 raft_consensus.cc:2804] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader election won for term 45
I20260709 12:19:38.128798 30035 consensus_queue.cc:1059] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.021s
I20260709 12:19:38.100111 29916 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:38.160085 12572 raft_consensus.cc:1240] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Rejecting Update request from peer 32889a507a6b4c74aec421ffbfd1e25d for earlier term 58. Current term is 59. Ops: [58.33-58.33]
I20260709 12:19:38.191850 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.192654 30034 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:38.092644 30033 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.084s
I20260709 12:19:38.167460 29916 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:38.168653 30038 raft_consensus.cc:697] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:38.179672 29747 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:38.193449 30034 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.162285 29990 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.032s
I20260709 12:19:38.201902 12461 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.180281 30042 raft_consensus.cc:697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:38.164510 29861 consensus_queue.cc:1059] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 49.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.032s
I20260709 12:19:38.214437 29679 raft_consensus.cc:3055] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Stepping down as leader of term 49
I20260709 12:19:38.221134 29778 raft_consensus.cc:3055] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Stepping down as leader of term 48
I20260709 12:19:38.216849 29517 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:38.211400 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.233904 29770 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:38.215837 29747 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
W20260709 12:19:38.235364 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 672123us
I20260709 12:19:38.236044 30034 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:38.236270 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:38.222728 12643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34832 (request call id 4925) took 1270 ms. Trace:
W20260709 12:19:38.237062 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:38.235694 29747 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.27, Last appended by leader: 27, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.226197 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.199628 29916 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:38.237603 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:38.240798 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.244468 12642 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
I20260709 12:19:38.241263 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 57 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.246459 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.245893 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:38.258345 12640 raft_consensus.cc:1217] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Deduplicated request from leader. Original: 48.29->[50.30-56.33]   Dedup: 50.30->[52.31-56.33]
I20260709 12:19:38.237183 12643 rpcz_store.cc:276] 0709 12:19:36.951857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:36.952000 (+   143us) service_pool.cc:224] Handling call
0709 12:19:36.952370 (+   370us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:37.181213 (+228843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:37.187435 (+  6222us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:37.191110 (+  3675us) log.cc:844] Serialized 52 byte log entry
0709 12:19:37.278764 (+ 87654us) raft_consensus.cc:1645] Marking committed up to 27
0709 12:19:37.323482 (+ 44718us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.525129 (+201647us) log.cc:844] Serialized 14 byte log entry
0709 12:19:37.863588 (+338459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:37.880231 (+ 16643us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b18004da0a0. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2706557 00007ffbf27087c1 00007ffbf2708779 00007ffbf2708709 00007ffbf27086d1 00007ffbf27079cd 0000000000388634 0000000000388479 00007ffbf26e983a
0709 12:19:38.222193 (+341962us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54002c7254. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:38.222307 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:38.222404 (+    97us) raft_consensus.cc:1685] finished
0709 12:19:38.222429 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:38.222646 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":647784,"prepare.queue_time_us":45717,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":2216,"raft.queue_time_us":561238,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":39766,"spinlock_wait_cycles":509468800,"thread_start_us":64661,"threads_started":1,"wal-append.queue_time_us":87390}
I20260709 12:19:38.297972 12640 pending_rounds.cc:85] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Aborting all ops after (but not including) 30
I20260709 12:19:38.298305 12640 pending_rounds.cc:107] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: Aborting uncommitted NO_OP operation due to leader change: 53.31
I20260709 12:19:38.298576 12640 raft_consensus.cc:2889] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:38.251580 12564 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:38.285336 11310 tablet_replica.cc:333] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:38.228874 29778 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:38.302693 29822 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:38.314203 29778 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 48.28, Last appended by leader: 28, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.315943 29778 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:38.318344 30041 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.319051 30041 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.211508 29994 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.081s
I20260709 12:19:38.289896 30032 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:38.322475 29517 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:38.322978 30032 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.323354 29517 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.323861 30038 consensus_queue.cc:237] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 40.21, Last appended by leader: 21, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.199020 29983 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 28, Time since last communication: 0.052s
I20260709 12:19:38.280547 30040 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:38.293779 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 67672.4us
I20260709 12:19:38.293483 29993 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election won for term 54
I20260709 12:19:38.329231 29993 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:38.326678 30040 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.329737 29993 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:38.251467 29860 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:38.231272 29679 raft_consensus.cc:740] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:38.331210 29679 consensus_queue.cc:260] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.331851 30040 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:38.297096 13256 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [45.31-45.31]
I20260709 12:19:38.311000 11310 raft_consensus.cc:2243] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus shutting down.
W20260709 12:19:38.311704 30047 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:19:38.338018 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:38.287925 12642 raft_consensus.cc:1240] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Rejecting Update request from peer 0a29b5d9f9d44298b02ffcd460811eef for earlier term 45. Current term is 46. Ops: [45.31-45.31]
I20260709 12:19:38.232896 29974 raft_consensus.cc:2804] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:38.325563 30032 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:38.345890 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.356808 12637 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 55 index: 31. Preceding OpId from leader: term: 57 index: 32. (index mismatch)
I20260709 12:19:38.357450 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 44 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.349542 12566 raft_consensus.cc:3055] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Stepping down as leader of term 45
I20260709 12:19:38.366186 12566 raft_consensus.cc:740] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:38.353574 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 61 candidate_status { last_received { term: 59 index: 38 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:38.353919 13256 raft_consensus.cc:2393] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 9485b0afaf144396adf854fff857414e in this term.
I20260709 12:19:38.367322 12642 raft_consensus.cc:3060] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Advancing to term 61
I20260709 12:19:38.374758 29517 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:38.374871 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.332849 30041 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:38.376046 12641 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:38.377875 29916 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.382030 11310 raft_consensus.cc:2272] T d03683a6ca224dc89740dc91f644b357 P 6d4ece942fd5414998215d26edf91001 [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:38.389858 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:38.352483 29974 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:19:38.373917 12610 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:38.395470 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:38.406381 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 48896us
I20260709 12:19:38.378846 29822 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:38.407426 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:38.356482 30033 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.047s
I20260709 12:19:38.396003 29974 raft_consensus.cc:3060] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:38.387558 29983 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.032s
I20260709 12:19:38.403883 29679 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 52
I20260709 12:19:38.417467 12642 raft_consensus.cc:2468] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 61.
I20260709 12:19:38.449817 29773 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Leader pre-election lost for term 61. Reason: could not achieve majority
I20260709 12:19:38.451112 12611 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.423050 29587 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.432045 30033 consensus_queue.cc:1243] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 46.31 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 30
I20260709 12:19:38.439407 30040 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.418795 29822 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.328284 30042 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 44.29, Last appended by leader: 29, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:38.464104 29773 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:38.508019 29916 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:38.510612 29822 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:38.521391 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:38.522733 29916 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:38.523186 29916 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:38.523602 29916 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:38.466240 29919 consensus_queue.cc:1059] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: INVALID_TERM, Last received: 44.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.035s
I20260709 12:19:38.466995 29587 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.492594 30040 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.579276 11310 tablet_replica.cc:333] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:38.580804 11310 raft_consensus.cc:2243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:19:38.596536 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:38.582470 29993 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:38.602217 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 139871us
I20260709 12:19:38.605314 29950 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.606038 29950 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.608430 29950 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:38.608969 12536 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:38.597405 12644 raft_consensus.cc:3060] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:38.618129 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.625264 12644 raft_consensus.cc:2468] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:38.616556 12566 consensus_queue.cc:260] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.31, Last appended by leader: 31, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.657306 12566 raft_consensus.cc:3060] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:38.661698 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:38.625847 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 214833us
I20260709 12:19:38.650910 11310 raft_consensus.cc:2272] T 93e029db6a8e4a7db72ad88760a6b7d3 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:38.654254 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 49 candidate_status { last_received { term: 45 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.670047 12566 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 31. Preceding OpId from leader: term: 46 index: 31. (term mismatch)
I20260709 12:19:38.672950 12566 pending_rounds.cc:85] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 30
I20260709 12:19:38.673269 12566 pending_rounds.cc:107] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 45.31
I20260709 12:19:38.673561 12566 raft_consensus.cc:2889] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:19:38.673723 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 51758.2us
I20260709 12:19:38.675235 12536 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:38.689230 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
W20260709 12:19:38.691135 29948 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 46 is higher.
I20260709 12:19:38.690313 29953 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:38.721499 29953 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.724562 29953 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:38.733891 29913 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:19:38.748106 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 52305.6us
I20260709 12:19:38.740950 29916 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.762750 29916 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:38.763056 29994 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.763765 29994 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:38.768493 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 71616.8us
I20260709 12:19:38.777796 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.782248 13218 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:38.796095 12640 raft_consensus.cc:1275] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 44 index: 30. Preceding OpId from leader: term: 46 index: 31. (index mismatch)
I20260709 12:19:38.718469 30033 raft_consensus.cc:2804] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election won for term 52
W20260709 12:19:38.796444 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 101185us
I20260709 12:19:38.803732 12461 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:38.783118 29778 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:38.798919 30065 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 12:19:38.819057 12535 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:38.821390 30033 raft_consensus.cc:697] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:38.822546 30033 consensus_queue.cc:237] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 49.31, Last appended by leader: 31, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.810279 29630 raft_consensus.cc:2804] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 61 FOLLOWER]: Leader election won for term 61
I20260709 12:19:38.854854 29630 raft_consensus.cc:697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 61 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:38.855219 30040 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:38.857293 11310 tablet_replica.cc:333] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:38.858469 11310 raft_consensus.cc:2243] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:38.858857 29994 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:38.859127 29747 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:38.815631 30017 consensus_queue.cc:1048] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.034s
I20260709 12:19:38.862520 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:38.856132 29630 consensus_queue.cc:237] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 59.38, Last appended by leader: 38, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:38.851399 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 47765.3us
I20260709 12:19:38.874169 11310 raft_consensus.cc:2272] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:38.869699 30044 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:19:38.878000 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:38.836390 29778 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.872538 29974 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:38.891284 29974 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:38.896006 13287 heartbeater.cc:646] Failed to heartbeat to 127.11.11.190:35399 (18 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.11.11.190:35399 timed out after 15.000s (SENT)
I20260709 12:19:38.901018 29993 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:38.885020 29587 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:38.920403 12462 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 43759.5us
W20260709 12:19:38.952556 30061 log.cc:927] Time spent T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
W20260709 12:19:38.956416 12536 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:38.962765 30074 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:19:38.964234 12536 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:38.966892 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 74378.4us
I20260709 12:19:38.971801 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:38.972572 12638 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:38.973421 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75679.1us
I20260709 12:19:38.975761 29990 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:38.976429 29990 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:38.988583 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bd1dcd3e8a44986816c86ea6e580df5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:38.988998 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:38.990044 12638 raft_consensus.cc:3060] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:38.991044 12639 raft_consensus.cc:3060] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Advancing to term 54
I20260709 12:19:38.997426 12646 raft_consensus.cc:1275] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 51 index: 28. Preceding OpId from leader: term: 52 index: 29. (index mismatch)
I20260709 12:19:39.000346 12461 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:39.001284 12461 raft_consensus.cc:2686] T 0c56dde469374f4da7440f134140670c P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:39.004495 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.009613 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:38.933813 29974 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:38.998821 30067 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:39.020897 12638 raft_consensus.cc:2468] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:39.044896 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.046170 12564 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:39.053049 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
W20260709 12:19:39.055361 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91618.9us
I20260709 12:19:39.058177 12638 raft_consensus.cc:1275] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 40 index: 21. Preceding OpId from leader: term: 41 index: 22. (index mismatch)
I20260709 12:19:39.033586 30060 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:39.023396 30067 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:39.080814 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:39.081799 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:39.082067 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:39.083359 13259 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 50.
I20260709 12:19:39.076793 29587 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:39.098824 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 57 candidate_status { last_received { term: 54 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:39.099406 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:39.104686 12610 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:39.098879 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 130850us
I20260709 12:19:39.096367 12639 raft_consensus.cc:2468] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 54.
I20260709 12:19:39.107798 12536 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:39.106292 12643 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 49.
W20260709 12:19:39.116396 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.120160 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:39.122282 30051 raft_consensus.cc:2804] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
W20260709 12:19:39.123014 29679 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: Time spent flushing consensus metadata: real 0.692s	user 0.013s	sys 0.000s
I20260709 12:19:39.124143 12572 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:39.098212 29990 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.120338 29778 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.119362 11310 tablet_replica.cc:333] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:39.121304 12536 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:39.136104 12462 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:39.104874 29587 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.146843 11310 raft_consensus.cc:2243] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Raft consensus shutting down.
I20260709 12:19:39.149798 12462 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:39.067402 30060 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.149833 29587 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:39.159756 30060 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.152897 29923 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:39.160840 30012 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:39.161839 30012 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:39.163300 30086 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:19:39.164901 30012 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.134503 29948 consensus_queue.cc:1048] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:19:39.167471 29916 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.168541 11310 raft_consensus.cc:2272] T 256d044fd2e244d4be827104688867ee P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:39.173148 30040 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
W20260709 12:19:39.175000 30094 log.cc:927] Time spent T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:19:39.179968 30067 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:39.155954 30077 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:39.181057 30089 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
W20260709 12:19:39.172047 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.183055 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:39.186887 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.187425 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:39.156659 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:39.123019 30051 raft_consensus.cc:697] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:39.196237 12564 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 48 index: 28. Preceding OpId from leader: term: 49 index: 28. (term mismatch)
I20260709 12:19:39.197065 12564 pending_rounds.cc:85] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting all ops after (but not including) 27
I20260709 12:19:39.197105 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 58 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.197772 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.198288 12564 pending_rounds.cc:107] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: Aborting uncommitted NO_OP operation due to leader change: 48.28
I20260709 12:19:39.201125 12564 raft_consensus.cc:2889] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:39.198661 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:39.202174 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:39.094159 30038 consensus_queue.cc:1048] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.076s
I20260709 12:19:39.205539 12572 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 59: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:39.209168 11310 tablet_replica.cc:333] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:39.209914 30071 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:39.168059 29916 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.220326 29916 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.222399 11310 raft_consensus.cc:2243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:19:39.203190 12462 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:39.172613 30033 raft_consensus.cc:2804] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader election won for term 54
W20260709 12:19:39.205817 12610 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.232962 12610 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:39.196378 30040 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:39.233970 30033 raft_consensus.cc:697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:39.235416 30033 consensus_queue.cc:237] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 53.29, Last appended by leader: 29, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.238456 30071 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:39.239040 30071 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.239408 30071 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:39.242069 30051 consensus_queue.cc:237] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 46.25, Last appended by leader: 25, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.247473 12462 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:39.250300 11310 raft_consensus.cc:2272] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:39.258112 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 52306.3us
I20260709 12:19:39.267398 12637 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 47 index: 27. Preceding OpId from leader: term: 49 index: 28. (index mismatch)
I20260709 12:19:39.268939 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 58 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.270318 12639 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 59.
I20260709 12:19:39.278663 29953 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:39.291970 12610 leader_election.cc:336] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.265712 29983 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.006s
W20260709 12:19:39.314456 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:39.252136 30066 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20260709 12:19:39.272512 30041 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
W20260709 12:19:39.330543 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:39.335561 30042 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.335693 29587 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:39.345243 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 191057us
W20260709 12:19:39.358532 12536 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 147 similar messages]
W20260709 12:19:39.359970 12536 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:39.340209 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 53321.9us
I20260709 12:19:39.368427 11310 tablet_replica.cc:333] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:39.374529 30076 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:39.375172 30076 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.365545 13253 raft_consensus.cc:3060] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: Advancing to term 45
I20260709 12:19:39.380936 29587 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.387326 29861 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:39.363986 30084 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:19:39.393041 13253 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I20260709 12:19:39.369534 11310 raft_consensus.cc:2243] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 57 LEADER]: Raft consensus shutting down.
I20260709 12:19:39.397989 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:39.399080 12566 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
W20260709 12:19:39.395654 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:39.385432 30092 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.255s	user 0.004s	sys 0.000s
I20260709 12:19:39.400315 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 51 candidate_status { last_received { term: 49 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:39.399695 11310 raft_consensus.cc:2272] T 979333b179ba4f49a65315a06d54e27e P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:39.388244 29861 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:39.327126 29971 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.077s
I20260709 12:19:39.406384 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:39.413043 12566 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:39.395236 29989 raft_consensus.cc:493] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.421453 12460 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001; no voters: 
I20260709 12:19:39.388633 30042 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:39.441366 12536 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:39.443573 12610 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:39.435575 29913 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader pre-election won for term 59
I20260709 12:19:39.431267 29747 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:39.462530 12566 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 29. Preceding OpId from leader: term: 45 index: 30. (index mismatch)
I20260709 12:19:39.427199 29989 raft_consensus.cc:515] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.464389 29913 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting leader election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:39.472478 30076 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:39.475281 11310 tablet_replica.cc:333] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:39.477898 12462 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:39.479249 29747 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Becoming Leader. State: Replica: 6d4ece942fd5414998215d26edf91001, State: Running, Role: LEADER
I20260709 12:19:39.496243 29913 raft_consensus.cc:3060] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Advancing to term 59
I20260709 12:19:39.498413 29747 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 49.25, Last appended by leader: 25, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.517167 29770 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:39.518157 29770 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.520768 30042 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:39.522431 29770 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:39.522627 29861 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:39.536466 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:39.542105 11310 raft_consensus.cc:2243] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus shutting down.
W20260709 12:19:39.542798 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:39.544715 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.505947 29587 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:39.568009 12536 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:39.569543 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:39.588006 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 108090us
I20260709 12:19:39.593796 30033 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:39.594436 30033 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:39.598788 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 144769us
W20260709 12:19:39.602128 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:39.610237 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.610919 12645 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 49.
I20260709 12:19:39.612077 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 54 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.614158 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 60 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.614897 12638 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 59.
I20260709 12:19:39.615876 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 60 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.616732 12638 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 60 because replica has last-logged OpId of term: 58 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 56 index: 32.
I20260709 12:19:39.561376 29587 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.618784 29989 leader_election.cc:290] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.619642 29587 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.621052 12643 raft_consensus.cc:1275] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 49 index: 31. Preceding OpId from leader: term: 52 index: 32. (index mismatch)
I20260709 12:19:39.624446 12644 raft_consensus.cc:1275] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 58 index: 37. Preceding OpId from leader: term: 61 index: 39. (index mismatch)
I20260709 12:19:39.612951 12640 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 55.
I20260709 12:19:39.627888 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 60 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.630508 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 60 candidate_status { last_received { term: 56 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:39.628647 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 59.
I20260709 12:19:39.632896 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.636651 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.637357 12572 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:39.638218 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.642146 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.630088 29517 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:39.674960 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 61062.6us
I20260709 12:19:39.676262 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:39.682050 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:39.706604 11310 raft_consensus.cc:2272] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:39.710899 30034 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader pre-election won for term 60
I20260709 12:19:39.697302 30051 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:39.680931 30102 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.061s
I20260709 12:19:39.722461 30033 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:39.724385 30041 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:39.683530 29517 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.748776 30051 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.761376 11310 tablet_replica.cc:333] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:39.750833 30101 log.cc:927] Time spent T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:19:39.736845 30034 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:39.726892 30017 consensus_queue.cc:1048] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 36, Time since last communication: 0.027s
W20260709 12:19:39.760216 30071 consensus_meta.cc:287] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.004s
I20260709 12:19:39.809088 30071 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:39.771090 30040 consensus_meta.cc:287] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: Time spent flushing consensus metadata: real 0.537s	user 0.013s	sys 0.000s
W20260709 12:19:39.772722 30096 log.cc:927] Time spent T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.209s	user 0.000s	sys 0.002s
W20260709 12:19:39.777455 30099 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.215s	user 0.002s	sys 0.000s
I20260709 12:19:39.815075 30040 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:39.815771 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:39.816505 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 50.
I20260709 12:19:39.817804 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:39.818647 29916 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:39.819020 29916 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.789553 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b62b58d64bf4dc2850a3ea4a3cb128a" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 52 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:39.819355 29916 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:39.791759 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feae6096d5e346a3b548be8b5e7488d4" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 45 candidate_status { last_received { term: 43 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:39.803946 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.765395 30051 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:39.805047 30034 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Advancing to term 60
W20260709 12:19:39.846275 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:39.847018 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 146500us
I20260709 12:19:39.847715 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:39.849401 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:39.853930 29916 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:39.853947 29861 consensus_queue.cc:1048] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.002s
I20260709 12:19:39.856925 29916 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:39.861524 12637 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:39.870523 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27144fedf0014de09f007ac6cc3f7275" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 49 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:39.871722 13256 raft_consensus.cc:3060] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:39.870193 30033 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election won for term 56
I20260709 12:19:39.910403 12461 leader_election.cc:304] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:39.911016 11310 raft_consensus.cc:2243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Raft consensus shutting down.
W20260709 12:19:39.874222 30083 raft_consensus.cc:670] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } [suppressed 5 similar messages]
I20260709 12:19:39.884025 30042 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:39.932106 12461 raft_consensus.cc:2686] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:39.935436 12461 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:39.938354 30068 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
I20260709 12:19:39.943389 12637 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4ece942fd5414998215d26edf91001 in term 51.
I20260709 12:19:39.926470 30033 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:39.926633 29913 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:39.951350 29913 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:39.952340 13253 raft_consensus.cc:2410] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef for term 60 because replica has last-logged OpId of term: 58 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 56 index: 32.
I20260709 12:19:39.952773 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 58 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:39.953863 12566 raft_consensus.cc:2393] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate 9485b0afaf144396adf854fff857414e in current term 59: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:39.964017 30034 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:39.966856 30033 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:39.968129 11310 raft_consensus.cc:2272] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:39.982632 13256 raft_consensus.cc:2468] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:39.983256 30040 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:39.982266 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 131953us
I20260709 12:19:39.983577 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:39.988768 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 278709us
W20260709 12:19:39.996083 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:39.997295 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:40.004676 12537 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:40.008186 30111 raft_consensus.cc:2804] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election won for term 51
W20260709 12:19:40.006381 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:40.006156 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:40.010586 12640 raft_consensus.cc:1275] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 25. Preceding OpId from leader: term: 47 index: 26. (index mismatch)
I20260709 12:19:39.944731 30042 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.012431 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:39.997928 30076 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election won for term 60
I20260709 12:19:40.009271 30111 raft_consensus.cc:697] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:40.027791 30034 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.028883 30071 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:40.031548 30042 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:40.021376 30076 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.034513 30076 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Advancing to term 60
I20260709 12:19:40.040094 29517 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:40.044073 30106 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:40.044586 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:40.045351 12568 raft_consensus.cc:3060] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:40.054622 12610 leader_election.cc:336] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.018889 30040 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:40.056893 11310 tablet_replica.cc:333] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:40.057186 30040 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:40.057744 30040 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:40.046005 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:39.982198 30051 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.046880 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 56 candidate_status { last_received { term: 54 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:40.063685 30051 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.047363 12537 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
W20260709 12:19:40.065934 30105 log.cc:927] Time spent T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:19:40.068400 30057 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Leader pre-election lost for term 60. Reason: could not achieve majority
W20260709 12:19:40.070705 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55614.7us
I20260709 12:19:40.074334 12569 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 25. Preceding OpId from leader: term: 51 index: 26. (index mismatch)
I20260709 12:19:40.075655 12568 raft_consensus.cc:2468] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:40.077261 11310 raft_consensus.cc:2243] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 47 LEADER]: Raft consensus shutting down.
I20260709 12:19:40.094448 12609 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
W20260709 12:19:40.088933 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 66364.2us
W20260709 12:19:40.125770 12461 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:19:40.126191 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:40.127260 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 60 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:40.130254 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 59 candidate_status { last_received { term: 58 index: 33 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:19:40.138563 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 44244.7us
I20260709 12:19:40.147346 12643 raft_consensus.cc:1275] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 53 index: 29. Preceding OpId from leader: term: 54 index: 30. (index mismatch)
W20260709 12:19:40.149344 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:40.150187 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:40.150758 11310 raft_consensus.cc:2272] T 0b5f1e67a9754d3090a2610efe47c70f P 6d4ece942fd5414998215d26edf91001 [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:40.140769 30113 raft_consensus.cc:2804] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:40.170302 30034 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.171060 30034 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.176250 30113 raft_consensus.cc:697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:40.177678 30113 consensus_queue.cc:237] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.150135 29953 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.035s
I20260709 12:19:40.191957 12645 raft_consensus.cc:2468] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 59.
W20260709 12:19:40.199720 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 237) took 52625 ms (client timeout 15000 ms). Trace:
W20260709 12:19:40.203238 12536 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.129070 30060 consensus_queue.cc:1048] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.023s
I20260709 12:19:40.210577 30116 consensus_queue.cc:1048] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:40.213258 11310 tablet_replica.cc:333] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:40.214372 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:40.215258 12639 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:40.173498 30034 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.226138 30116 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:40.235308 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:40.237785 11310 raft_consensus.cc:2243] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:40.235096 30116 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:40.240087 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 45361.3us
W20260709 12:19:40.200232 12386 rpcz_store.cc:269] 0709 12:18:47.536597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:02.228008 (+14691411us) service_pool.cc:224] Handling call
0709 12:19:40.162045 (+37934037us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:34.402765 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:34.403619 (+   854us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:34.403681 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:19:34.403699 (+    18us) tablet.cc:662] Decoding operations
0709 12:19:34.405294 (+  1595us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:34.405401 (+   107us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:34.405429 (+    28us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:19:34.414030 (+  8601us) tablet.cc:701] Row locks acquired
0709 12:19:34.414081 (+    51us) write_op.cc:260] PREPARE: finished
0709 12:19:34.414258 (+   177us) write_op.cc:270] Start()
0709 12:19:34.414414 (+   156us) write_op.cc:276] Timestamp: P: 1783599574414200 usec, L: 0
0709 12:19:34.414458 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:19:34.493820 (+ 79362us) log.cc:844] Serialized 7520 byte log entry
0709 12:19:35.474349 (+980529us) op_driver.cc:464] REPLICATION: finished
0709 12:19:35.762405 (+288056us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b4400022b4c. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf258edb1 00007ffbf262cc85 00007ffbf262d6a2 00007ffbf539f1db 00007ffbf539aaaf 00007ffbf26e90ac 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2705c02 00007ffbf9229ef0
0709 12:19:36.190948 (+428543us) write_op.cc:301] APPLY: starting
0709 12:19:36.224973 (+ 34025us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:37.200315 (+975342us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:37.236211 (+ 35896us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:37.557013 (+320802us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:37.558791 (+  1778us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0709 12:19:37.558848 (+    57us) write_op.cc:312] APPLY: finished
0709 12:19:37.574377 (+ 15529us) log.cc:844] Serialized 172 byte log entry
0709 12:19:37.598380 (+ 24003us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:37.598601 (+   221us) write_op.cc:454] Released schema lock
0709 12:19:37.598695 (+    94us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:37.608290 (+  9595us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:37.610219 (+  1929us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1812502,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":615478,"apply.run_cpu_time_us":46483,"apply.run_wall_time_us":1466596,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83471,"cfile_init":1,"lbm_read_time_us":499,"lbm_reads_lt_1ms":4,"num_ops":19,"prepare.queue_time_us":209914,"prepare.run_cpu_time_us":81226,"prepare.run_wall_time_us":361979,"raft.queue_time_us":397470,"raft.run_cpu_time_us":117959,"raft.run_wall_time_us":739342,"replication_time_us":1059679,"spinlock_wait_cycles":82416896,"thread_start_us":1170273,"threads_started":5,"wal-append.queue_time_us":430741,"wal-append.run_cpu_time_us":45092,"wal-append.run_wall_time_us":2346902}]]}
I20260709 12:19:40.272514 12644 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 6d4ece942fd5414998215d26edf91001: Log matching property violated. Preceding OpId in replica: term: 49 index: 25. Preceding OpId from leader: term: 51 index: 26. (index mismatch)
I20260709 12:19:40.206012 30033 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:40.294250 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:40.141721 30111 consensus_queue.cc:237] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 49.30, Last appended by leader: 29, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:40.314354 12536 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:40.315873 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:40.318378 12536 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:40.327533 11310 raft_consensus.cc:2272] T feae6096d5e346a3b548be8b5e7488d4 P 6d4ece942fd5414998215d26edf91001 [term 44 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:40.333588 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.337347 12565 raft_consensus.cc:2393] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 56: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:40.344417 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 140) took 16460 ms (client timeout 15000 ms). Trace:
W20260709 12:19:40.345095 12386 rpcz_store.cc:269] 0709 12:19:24.042715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:40.331023 (+16288308us) service_pool.cc:211] Skipping call since client already timed out
0709 12:19:40.340138 (+  9115us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:19:40.398165 30116 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.416574 30033 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.375464 30076 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.376499 30040 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.436753 30076 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:40.438433 12640 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 60: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:40.445325 30118 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.199s	user 0.000s	sys 0.002s
W20260709 12:19:40.451354 30120 log.cc:927] Time spent T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
W20260709 12:19:40.452991 30121 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:19:40.456388 30129 raft_consensus.cc:1064] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:40.456897 11310 tablet_replica.cc:333] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:40.457006 30129 raft_consensus.cc:1068] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Unable to promote non-voter 9485b0afaf144396adf854fff857414e: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:19:40.482206 11310 raft_consensus.cc:2243] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:40.484210 12642 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:40.503531 30040 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:40.534713 11310 raft_consensus.cc:2272] T b39f4bd5c55347ada5f73766216828ca P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:40.575551 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 176628us
I20260709 12:19:40.576977 11310 tablet_replica.cc:333] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:40.584569 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 244066us
W20260709 12:19:40.585296 12536 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.601616 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:40.602747 12639 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:40.614421 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 54 index: 23 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:40.615070 12646 raft_consensus.cc:3060] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Advancing to term 56
I20260709 12:19:40.627277 13218 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:19:40.637553 12646 raft_consensus.cc:2468] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 56.
I20260709 12:19:40.642776 29911 raft_consensus.cc:2804] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Leader election won for term 59
I20260709 12:19:40.627367 30111 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.650866 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:40.669857 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 46195.8us
I20260709 12:19:40.685766 11310 raft_consensus.cc:2243] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:40.681820 30060 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:40.696517 30060 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.696983 30060 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:40.687824 29911 raft_consensus.cc:697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 59 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:40.699734 29911 consensus_queue.cc:237] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 58.33, Last appended by leader: 33, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:40.702659 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:40.706594 30051 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:40.707127 30051 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.687287 29923 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.020s
I20260709 12:19:40.707558 30051 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:40.685254 30111 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.731165 30111 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.732577 12536 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:40.734673 30117 raft_consensus.cc:2804] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader election won for term 56
I20260709 12:19:40.735411 30117 raft_consensus.cc:697] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:40.738761 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:40.740005 30057 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20260709 12:19:40.741561 12536 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:40.746281 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 56032.4us
I20260709 12:19:40.749292 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:40.752168 12565 raft_consensus.cc:2393] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 50: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:40.749554 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 60 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:40.753553 12564 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 60: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:40.746994 12461 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 58542.7us
W20260709 12:19:40.757366 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 65577.3us
W20260709 12:19:40.770498 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 83761.1us
W20260709 12:19:40.771612 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.773844 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:40.773969 11310 raft_consensus.cc:2272] T 472e786b46074d868d00df67a0985afd P 6d4ece942fd5414998215d26edf91001 [term 57 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:40.780251 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 66442.5us
I20260709 12:19:40.782371 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:40.783165 12642 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 57.
W20260709 12:19:40.793776 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:40.789470 29770 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.799129 29770 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:40.797260 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:40.804265 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.736797 30117 consensus_queue.cc:237] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 54.23, Last appended by leader: 23, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.807024 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:40.808993 30128 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:40.815248 29770 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:40.778739 30116 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:40.833549 29770 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
W20260709 12:19:40.830730 12388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 235) took 196640 ms (client timeout 15000 ms). Trace:
W20260709 12:19:40.838753 12388 rpcz_store.cc:269] 0709 12:16:24.207880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:36.679303 (+12471423us) service_pool.cc:224] Handling call
0709 12:19:40.830452 (+184151149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":182,"reported_tablets":51}
W20260709 12:19:40.839452 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.815253 11310 tablet_replica.cc:333] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:40.870563 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:40.885710 11310 raft_consensus.cc:2243] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 45 LEADER]: Raft consensus shutting down.
W20260709 12:19:40.890380 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:40.896438 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:40.903314 11310 raft_consensus.cc:2272] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:40.894665 30116 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election won for term 58
I20260709 12:19:40.920696 30060 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:40.935761 30060 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:40.941224 11310 tablet_replica.cc:333] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:40.942520 11310 raft_consensus.cc:2243] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:40.943389 11310 pending_rounds.cc:70] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
I20260709 12:19:40.943677 11310 pending_rounds.cc:77] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 54 index: 31 } timestamp: 7305623751475683328 op_type: NO_OP noop_request { }
I20260709 12:19:40.944228 11310 raft_consensus.cc:2889] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:40.944598 11310 raft_consensus.cc:2272] T 6f701a23559945868ce6e3da5866f95c P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:40.946062 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 172757us
I20260709 12:19:40.947422 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:40.925457 30116 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.956931 30060 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:40.965838 30116 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Advancing to term 58
W20260709 12:19:40.989082 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:41.000506 12389 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 6d4ece942fd5414998215d26edf91001 reported cstate change: config changed from index 25 to -1, term changed from 40 to 34, leader changed from <none> to 6d4ece942fd5414998215d26edf91001 (127.11.11.129), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) evicted. New cstate: current_term: 34 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:40.996574 12536 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:41.016647 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 55213.3us
I20260709 12:19:41.018831 30142 raft_consensus.cc:1064] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:40.993541 11310 tablet_replica.cc:333] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:40.978192 30060 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:41.040742 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 93453.3us
W20260709 12:19:41.046552 12610 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:41.078083 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:41.047159 30113 raft_consensus.cc:2749] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20260709 12:19:41.049960 30142 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 45.30, Last appended by leader: 29, Current term: 45, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:41.040403 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45271.5us
W20260709 12:19:41.082679 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:41.083382 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:41.084240 12536 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:41.075341 11310 raft_consensus.cc:2243] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:41.087637 11310 raft_consensus.cc:2272] T bb9177ec2fb345c4bff68c8c8a132af6 P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.090163 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:41.090946 12641 raft_consensus.cc:3060] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:41.091202 11310 tablet_replica.cc:333] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.092212 11310 raft_consensus.cc:2243] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Raft consensus shutting down.
I20260709 12:19:41.101177 30111 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:41.101652 13256 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 45 index: 31. (index mismatch)
I20260709 12:19:41.101895 30111 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:41.103546 30113 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:19:41.104395 30111 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:41.113933 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:41.076165 12537 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 161236us
I20260709 12:19:41.017138 30051 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:41.079437 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:41.119344 12536 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:41.122331 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 57 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:41.123075 12566 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 57.
I20260709 12:19:41.122010 29517 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20260709 12:19:41.130735 30051 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:41.139218 30060 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:41.125432 12641 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 53.
W20260709 12:19:41.143128 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:41.141685 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:41.157994 30113 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader pre-election won for term 58
I20260709 12:19:41.158528 30113 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:41.158998 30113 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:41.175459 30113 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:41.184677 30113 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:41.189280 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:41.200430 30116 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:41.202950 30116 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:41.093204 11310 raft_consensus.cc:2272] T 2dbcfa3bff034f5ab82e2ae802266099 P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.206513 30012 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:41.207235 30012 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:41.209750 11310 tablet_replica.cc:333] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.210350 30012 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:41.210891 11310 raft_consensus.cc:2243] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:41.211567 11310 raft_consensus.cc:2272] T 7069a8d3584b466aab95934840523b8d P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.215139 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:41.216261 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:41.218776 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:41.215163 11310 tablet_replica.cc:333] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.219712 30012 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:41.220687 11310 raft_consensus.cc:2243] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:19:41.220957 30012 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:41.221582 30012 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:41.221992 11310 raft_consensus.cc:2272] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.229856 11310 tablet_replica.cc:333] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.230857 11310 raft_consensus.cc:2243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:41.231503 11310 raft_consensus.cc:2272] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.234193 30012 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:41.236325 12569 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 56 index: 32. Preceding OpId from leader: term: 59 index: 34. (index mismatch)
I20260709 12:19:41.236559 30012 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:41.239106 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:41.239858 12641 raft_consensus.cc:3060] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:41.243896 29913 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20260709 12:19:41.245146 11310 tablet_replica.cc:333] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.247699 11310 raft_consensus.cc:2243] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 51 LEADER]: Raft consensus shutting down.
W20260709 12:19:41.249591 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:41.249024 11310 raft_consensus.cc:2272] T 05ad55cc3c04480abd0546c50226f9c1 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.260663 11310 tablet_replica.cc:333] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.266649 11310 raft_consensus.cc:2243] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:19:41.267715 11310 raft_consensus.cc:2272] T ddf2b3bf3e6344499adb020ce08a3abf P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:41.275307 11310 tablet_replica.cc:333] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:41.281581 11310 raft_consensus.cc:2243] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:41.282337 11310 pending_rounds.cc:70] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Trying to abort 2 pending ops.
I20260709 12:19:41.282622 11310 pending_rounds.cc:77] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 44 index: 23 } timestamp: 7305623658305884160 op_type: NO_OP noop_request { }
I20260709 12:19:41.282963 11310 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:41.283250 11310 pending_rounds.cc:77] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 42 index: 22 } timestamp: 7305623609476083712 op_type: NO_OP noop_request { }
I20260709 12:19:41.283555 11310 raft_consensus.cc:2889] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:41.283866 11310 raft_consensus.cc:2272] T a1d35de928f546349f88cbea7818041c P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:41.261021 13218 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:41.519014 11310 tablet_replica.cc:333] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:42.967557 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 1.70619e+06us
W20260709 12:19:41.493367 12460 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 200595us
I20260709 12:19:41.492620 12536 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:42.722429 12565 raft_consensus.cc:1275] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 53 index: 28. (index mismatch)
I20260709 12:19:42.979889 30174 raft_consensus.cc:493] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:42.742635 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 58 candidate_status { last_received { term: 57 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:42.743609 12569 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 30. Preceding OpId from leader: term: 45 index: 31. (index mismatch)
I20260709 12:19:42.768502 12568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3375) took 1513 ms. Trace:
W20260709 12:19:42.580132 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080019c280 after lost signal to thread 11313
I20260709 12:19:42.994712 30160 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:42.992141 12568 rpcz_store.cc:276] 0709 12:19:41.254788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.254944 (+   156us) service_pool.cc:224] Handling call
0709 12:19:41.255426 (+   482us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:19:41.255860 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:41.255892 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:19:41.256779 (+   887us) log.cc:844] Serialized 52 byte log entry
0709 12:19:41.264285 (+  7506us) raft_consensus.cc:1645] Marking committed up to 33
0709 12:19:41.265045 (+   760us) log.cc:844] Serialized 14 byte log entry
0709 12:19:41.265303 (+   258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:41.265402 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:19:42.767988 (+1502586us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b1800443860. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf2680c0a 00007ffbfbdc0219 00007ffbfb8b9978 00007ffbfb8b9a89 00007ffbfb8b6a5e 00007ffbf5817a62 00007ffbf57f3f34 00007ffbf57f283e 00007ffbfa7b7419
0709 12:19:42.768065 (+    77us) raft_consensus.cc:1685] finished
0709 12:19:42.768180 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:42.768409 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":226217,"prepare.queue_time_us":214,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":1990,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":20068,"spinlock_wait_cycles":10149248,"thread_start_us":6962,"threads_started":1,"wal-append.queue_time_us":5038,"wal-append.run_cpu_time_us":3039,"wal-append.run_wall_time_us":1094094}
W20260709 12:19:42.996186 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800213580 after lost signal to thread 12429
I20260709 12:19:42.760715 29517 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:42.997412 12570 raft_consensus.cc:3060] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:42.990309 30060 raft_consensus.cc:493] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:42.998306 29517 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:42.999507 30060 raft_consensus.cc:515] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.992647 30057 raft_consensus.cc:2804] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:43.001395 30057 raft_consensus.cc:697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:43.001976 30175 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:42.765573 29916 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.003580 29916 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:42.993572 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 1.50023e+06us
I20260709 12:19:42.802142 30017 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.008348 30017 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.003237 30057 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 52.32, Last appended by leader: 32, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.801412 30052 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:42.715332 30071 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:42.782460 30116 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.029855 30071 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.030624 30116 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:42.754290 30148 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:41.836019 30152 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.033272 30071 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.034402 30071 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:42.734229 30156 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.039224 30156 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.039676 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:42.740633 30157 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.042459 30157 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.742790 30159 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:42.766961 30163 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.045861 30168 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.046900 30168 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.773566 30165 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.033609 30148 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.049538 30165 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.049875 30071 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.779662 30166 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.052441 30165 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:42.772037 30167 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.053242 30159 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.054590 30167 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.056241 30159 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:42.746212 30146 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:43.059906 30146 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.173174 30153 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.052383 30071 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.062731 30153 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.064107 30116 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:42.739549 30155 raft_consensus.cc:493] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:42.759137 30161 raft_consensus.cc:493] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.075307 30161 raft_consensus.cc:515] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:42.764850 30162 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.078299 30155 raft_consensus.cc:515] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.079085 30172 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:42.905375 30113 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.079934 30172 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.080106 30113 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.082516 30172 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:42.938315 30028 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.083207 12566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3366) took 1852 ms. Trace:
I20260709 12:19:41.257589 12641 raft_consensus.cc:2468] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
I20260709 12:19:42.942314 30012 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:42.940971 29961 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:42.997431 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800213b80 after lost signal to thread 12521
I20260709 12:19:42.779883 30051 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.093489 30012 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.094260 30051 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:43.098676 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080007f7a0 after lost signal to thread 12596
W20260709 12:19:43.099069 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080038ffe0 after lost signal to thread 12597
I20260709 12:19:43.099237 30146 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.013932 13262 raft_consensus.cc:1275] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 47 index: 29. Preceding OpId from leader: term: 51 index: 31. (index mismatch)
W20260709 12:19:43.100690 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08003bf060 after lost signal to thread 12670
W20260709 12:19:43.101212 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080024bdc0 after lost signal to thread 12671
I20260709 12:19:42.996126 30160 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:43.016950 30017 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.017558 29517 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.102741 29517 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.053484 30166 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.070653 30167 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.017795 29916 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.020068 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 55 candidate_status { last_received { term: 54 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.108353 12570 raft_consensus.cc:2468] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 58.
I20260709 12:19:43.003124 30175 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.109082 30153 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.110411 30113 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.111305 30175 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.033201 30171 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.112198 30175 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.112468 30171 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.113936 30168 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:42.888173 30170 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.029695 30052 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.115525 30170 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.116792 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79b687d20a534e70865ef902fbd0d4a5" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 54 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.117157 30052 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.034617 30152 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.103526 29517 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.118727 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.119712 30152 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.046811 30163 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:42.768347 30164 raft_consensus.cc:493] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.045320 30157 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.052183 30156 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.073849 11310 raft_consensus.cc:2243] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:43.121501 30170 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:42.939440 29913 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.121958 30012 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.122505 29913 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.083706 12566 rpcz_store.cc:276] 0709 12:19:41.231046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.873195 (+1642149us) service_pool.cc:224] Handling call
0709 12:19:42.873975 (+   780us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:43.081944 (+207969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:43.081987 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:43.082087 (+   100us) raft_consensus.cc:1645] Marking committed up to 34
0709 12:19:43.082132 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.082801 (+   669us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54001b21d4. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:43.082850 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.082999 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":477516672}
I20260709 12:19:43.123508 30060 leader_election.cc:290] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.121217 30164 raft_consensus.cc:515] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.126428 30160 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.081447 30155 leader_election.cc:290] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.092675 12641 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35404 (request call id 2515) took 1853 ms. Trace:
I20260709 12:19:43.106806 29961 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.128279 12641 rpcz_store.cc:276] 0709 12:19:41.238637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:41.238810 (+   173us) service_pool.cc:224] Handling call
0709 12:19:43.092289 (+1853479us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:19:43.129048 11310 pending_rounds.cc:70] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
I20260709 12:19:43.129447 11310 pending_rounds.cc:77] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 48 index: 34 } timestamp: 7305623773546700800 op_type: NO_OP noop_request { }
I20260709 12:19:43.130110 11310 raft_consensus.cc:2889] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:43.130424 29961 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.130976 12564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50142 (request call id 3367) took 1899 ms. Trace:
I20260709 12:19:43.131418 12564 rpcz_store.cc:276] 0709 12:19:41.231371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:42.871436 (+1640065us) service_pool.cc:224] Handling call
0709 12:19:42.872178 (+   742us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:19:43.127810 (+255632us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:19:43.127843 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:19:43.127929 (+    86us) raft_consensus.cc:1645] Marking committed up to 30
0709 12:19:43.127970 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:19:43.130532 (+  2562us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400045454. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57f3e01 00007ffbf57f283e 00007ffbfa7b7419 00007ffbf3d2ebd4 00007ffbf3d2eb62 00007ffbf3d2ea91 00007ffbf3d2ea0c 00007ffbf3d2dcc2 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:43.130582 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:19:43.130733 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":586927872}
I20260709 12:19:43.130455 11310 raft_consensus.cc:2272] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:43.037386 30174 raft_consensus.cc:515] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.111258 30113 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.073048 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.134699 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.078578 30162 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.135676 12566 raft_consensus.cc:2468] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:19:43.113018 30175 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.137914 30162 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.139671 30175 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.140472 11310 tablet_replica.cc:333] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:43.141469 30163 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.134807 30113 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:43.142768 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080038cfc0 after lost signal to thread 13286
W20260709 12:19:43.143204 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08003c5c00 after lost signal to thread 13287
W20260709 12:19:43.143529 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08003e0240 after lost signal to thread 29587
W20260709 12:19:43.143784 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800406f80 after lost signal to thread 29747
W20260709 12:19:43.144037 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080009e460 after lost signal to thread 29770
W20260709 12:19:43.144349 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08000433e0 after lost signal to thread 29953
W20260709 12:19:43.144708 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800237ca0 after lost signal to thread 29971
I20260709 12:19:43.144781 30171 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:43.145404 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08003dd060 after lost signal to thread 29983
W20260709 12:19:43.145823 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800163060 after lost signal to thread 29994
I20260709 12:19:43.145982 30164 leader_election.cc:290] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:43.146235 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800318b00 after lost signal to thread 30034
I20260709 12:19:43.141477 30174 leader_election.cc:290] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.144418 30113 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:43.146811 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800230ca0 after lost signal to thread 30040
W20260709 12:19:43.148645 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08003d8d00 after lost signal to thread 30042
W20260709 12:19:43.149245 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08003d0e60 after lost signal to thread 30066
W20260709 12:19:43.149874 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080039be00 after lost signal to thread 30108
I20260709 12:19:43.150329 30166 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.083775 30028 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.133810 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.125072 29913 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.121198 30148 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.120306 29517 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.159130 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.160104 30071 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.160806 30028 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.123438 30161 leader_election.cc:290] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.160902 30071 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.165217 30071 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.166147 30174 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:43.150573 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080030d640 after lost signal to thread 30126
I20260709 12:19:43.166961 30174 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:43.173416 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08001eae80 after lost signal to thread 30128
W20260709 12:19:43.173875 12593 debug-util.cc:398] Leaking SignalData structure 0x7b08001eaf00 after lost signal to thread 30129
W20260709 12:19:43.174248 12593 debug-util.cc:398] Leaking SignalData structure 0x7b080018bfa0 after lost signal to thread 30135
W20260709 12:19:43.174618 12593 debug-util.cc:398] Leaking SignalData structure 0x7b0800331a40 after lost signal to thread 30140
I20260709 12:19:43.171350 30071 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.175573 30174 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.176407 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.175642 30071 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.184669 30071 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.153316 30172 raft_consensus.cc:493] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.186115 30172 raft_consensus.cc:515] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.192406 30051 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.193485 29517 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.194239 29517 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.195765 30174 raft_consensus.cc:493] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.196478 30174 raft_consensus.cc:515] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.198796 30174 leader_election.cc:290] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.199380 30174 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.206043 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.204334 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.207333 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 50 candidate_status { last_received { term: 48 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.204835 11310 raft_consensus.cc:2243] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:19:43.211102 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.214213 13259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.214529 13255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.215441 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.215790 13256 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.217995 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43fecbd2edc14ba2ab065395e5d48d1d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.225028 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.217895 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "472e786b46074d868d00df67a0985afd" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.202035 30174 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.236784 30163 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:43.240949 30155 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.241322 30172 leader_election.cc:290] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.241885 30155 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.243142 11310 raft_consensus.cc:2272] T 6fad7500a4eb45afa94184a247d796b8 P 6d4ece942fd5414998215d26edf91001 [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:43.241026 30161 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.246487 30161 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.247148 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "979333b179ba4f49a65315a06d54e27e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.250281 12645 raft_consensus.cc:2393] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 58: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:43.256407 30163 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.257390 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:43.258448 12643 raft_consensus.cc:3060] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Advancing to term 58
I20260709 12:19:43.260244 30163 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.262295 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:43.263563 29517 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.266216 30172 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.267100 30172 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.267809 30028 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:43.269982 12641 raft_consensus.cc:3060] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:43.278726 30161 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 61 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.268864 30028 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.282706 30028 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.237955 30148 consensus_queue.cc:1048] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.046s
I20260709 12:19:43.245555 30155 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:43.299923 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.299495 30179 raft_consensus.cc:493] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.304607 30176 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.305025 30179 raft_consensus.cc:515] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.305770 30176 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.308184 30179 leader_election.cc:290] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.310424 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.310752 12646 raft_consensus.cc:1275] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 54 index: 23. Preceding OpId from leader: term: 56 index: 24. (index mismatch)
I20260709 12:19:43.311307 12639 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:43.313525 12462 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.304749 30172 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.302070 13254 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 56.
I20260709 12:19:43.319392 30179 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:19:43.311180 11310 tablet_replica.cc:333] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:43.320166 30179 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.322477 29517 raft_consensus.cc:493] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.333969 11310 raft_consensus.cc:2243] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 56 LEADER]: Raft consensus shutting down.
I20260709 12:19:43.333881 29517 raft_consensus.cc:515] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.335542 11310 raft_consensus.cc:2272] T c126dc4ef9114297874fa66b1a0c56b0 P 6d4ece942fd5414998215d26edf91001 [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:43.336519 29517 leader_election.cc:290] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:43.337846 12462 raft_consensus.cc:2686] T 65037753870c4624995dde1173266f49 P 6d4ece942fd5414998215d26edf91001: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:19:43.354522 30179 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.356058 30176 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.359427 30171 raft_consensus.cc:493] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.360416 30171 raft_consensus.cc:515] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.347097 30113 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.363984 30113 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.347266 30172 raft_consensus.cc:493] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.365931 30170 raft_consensus.cc:493] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.366479 30172 raft_consensus.cc:515] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.366967 30170 raft_consensus.cc:515] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.379985 12641 raft_consensus.cc:2468] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 52.
I20260709 12:19:43.369902 30174 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
W20260709 12:19:43.384953 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 91339.6us
I20260709 12:19:43.393971 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.394235 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.394866 12572 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 45.
I20260709 12:19:43.395006 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1d35de928f546349f88cbea7818041c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.396028 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.396463 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.397121 12566 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:43.399843 30171 leader_election.cc:290] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 62 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.397747 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.398183 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7816534d4a0942f0af150ea0a94392b9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.383407 30071 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.401971 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.402117 30071 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.404009 12643 raft_consensus.cc:2468] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 58.
I20260709 12:19:43.405215 30071 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.406648 30152 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.414139 30152 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.406665 30113 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.416873 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.417243 30152 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.385866 30179 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.381173 30176 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.417946 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.418426 30179 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.418817 30176 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.420889 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bb3f87ad7904e7d97f68dbbd8c3ab60" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.369596 30170 leader_election.cc:290] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.421178 30176 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.373811 30172 leader_election.cc:290] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.418406 11310 tablet_replica.cc:333] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:43.421053 30179 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.397018 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.426059 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.427062 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 57 candidate_status { last_received { term: 56 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.435086 12640 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 56.
I20260709 12:19:43.428532 12638 raft_consensus.cc:1275] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 58 index: 33. Preceding OpId from leader: term: 59 index: 34. (index mismatch)
I20260709 12:19:43.429559 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54875fd261ee4c328e3d022d6a9418d8" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 55 candidate_status { last_received { term: 53 index: 38 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.430516 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.438707 12462 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.439236 11310 raft_consensus.cc:2243] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:43.439951 29570 raft_consensus.cc:2706] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Received pre-election callback for term 55 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:19:43.441566 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:43.440785 11310 pending_rounds.cc:70] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Trying to abort 2 pending ops.
I20260709 12:19:43.443857 30179 consensus_queue.cc:1048] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:19:43.432416 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.444671 30113 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
I20260709 12:19:43.442830 11310 pending_rounds.cc:77] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 53 index: 38 } timestamp: 7305623773981499392 op_type: NO_OP noop_request { }
I20260709 12:19:43.445887 11310 raft_consensus.cc:2889] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:43.446307 11310 pending_rounds.cc:77] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 52 index: 37 } timestamp: 7305623729698336768 op_type: NO_OP noop_request { }
I20260709 12:19:43.446337 12609 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.446861 11310 raft_consensus.cc:2889] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:43.447358 11310 raft_consensus.cc:2272] T 54875fd261ee4c328e3d022d6a9418d8 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:43.450119 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.453131 30071 raft_consensus.cc:2804] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader election won for term 58
I20260709 12:19:43.453992 30071 raft_consensus.cc:697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:43.455361 30071 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 57.32, Last appended by leader: 32, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.456157 13262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 60 candidate_status { last_received { term: 58 index: 33 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.453445 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.460912 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.461122 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.462064 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1033139a62f44d368f8443d665a2cf22" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.462015 12568 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:43.464336 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4dc881215d9f4498b162561e79143f2d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.464589 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.465049 11310 tablet_replica.cc:333] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:43.462332 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.465930 12567 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 47.
I20260709 12:19:43.467650 11310 raft_consensus.cc:2243] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 50 LEADER]: Raft consensus shutting down.
I20260709 12:19:43.468894 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9177ec2fb345c4bff68c8c8a132af6" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 54 index: 23 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.469499 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e029db6a8e4a7db72ad88760a6b7d3" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.470443 12565 raft_consensus.cc:2393] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:43.470350 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.471200 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f92af3771e3446f8c3549eb19f5b2c9" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.471645 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.471909 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f02aa0ab5af34d0e9ef3a4956449c624" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 42 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.472711 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.473420 30113 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:43.473870 30113 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.474279 30113 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:43.452610 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 215144us
I20260709 12:19:43.485491 30176 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:19:43.486155 30176 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.486598 30176 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:19:43.486162 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.490558 30170 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:43.491824 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.492316 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "40d0e235c01b4b7f89948ab708403a01" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.493027 30170 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:43.493402 30170 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.493814 30170 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:43.497239 12609 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.502233 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:43.503783 12536 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.509479 12536 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:43.510800 30111 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:43.518810 12461 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.519987 29570 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:43.524523 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.525331 12638 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:43.528363 12638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.529492 29570 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:43.530632 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.530407 29570 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.531549 30172 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:43.532231 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.532805 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0dce81bc984a0690d08335a4937b2c" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.533171 30172 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
I20260709 12:19:43.533732 30172 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.534030 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bbf126b18234b78b4f36bf9708ad96b" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 62 candidate_status { last_received { term: 61 index: 39 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.534291 30172 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:43.534647 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.529579 30174 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:43.538003 13218 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:43.539465 30173 raft_consensus.cc:2804] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:43.540125 30173 raft_consensus.cc:697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:43.541114 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d60fb5ad76741448a5376ca4d3f05df" candidate_uuid: "6d4ece942fd5414998215d26edf91001" candidate_term: 46 candidate_status { last_received { term: 44 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:43.541312 30173 consensus_queue.cc:237] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 49.34, Last appended by leader: 34, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.538411 29570 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:43.545099 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.549186 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.545387 12462 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6d4ece942fd5414998215d26edf91001; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.553828 12609 leader_election.cc:304] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.556802 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.561096 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:43.564709 11310 raft_consensus.cc:2272] T c793ebd4de24492397303f9a6e6bc62a P 6d4ece942fd5414998215d26edf91001 [term 50 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:43.566679 12644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.571902 12641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a30ed0986fa641bd8f88912b4b9ee320" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 43 candidate_status { last_received { term: 42 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.573482 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 55 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.573390 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f402dfd68ae4c8697765a76d9f1c4e6" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 46 candidate_status { last_received { term: 45 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.567812 30179 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:43.560369 30174 raft_consensus.cc:2749] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:43.577224 12640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a16c8cd5242e4ab6840a3a328938b5df" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:19:43.579358 13218 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.579264 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07fe6c41f52649a88327e52e0352b171" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.564880 30153 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:43.577744 12609 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
W20260709 12:19:43.580780 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.587905 12609 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.583348 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.575784 30179 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:43.594213 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 60300.2us
I20260709 12:19:43.594797 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 57 candidate_status { last_received { term: 56 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.593990 30157 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:43.600078 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 61 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.600996 12642 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 60.
W20260709 12:19:43.585937 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.601092 30179 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:43.603389 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.581895 30052 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:43.604573 12645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.602065 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:43.641433 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:43.641745 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 55860.7us
W20260709 12:19:43.642684 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.625022 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 61 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.593864 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:43.616844 30176 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.610299 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1f18763e7844e0994e2f11a6dd89290" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 60 candidate_status { last_received { term: 58 index: 33 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.610294 12567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3c8a7a06a04c369587d6ecb196bb2e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 35 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.603277 30157 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:43.609838 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.651242 12568 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:43.609759 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:43.643455 12572 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 60.
I20260709 12:19:43.647751 12571 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:43.582021 30113 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.643496 12637 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 51.
I20260709 12:19:43.653779 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.656266 12609 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.657547 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:43.652908 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:43.644119 30173 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:19:43.658362 12610 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:43.663902 12536 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.664618 30175 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:43.664897 30163 consensus_queue.cc:1048] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:19:43.665024 30174 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Leader pre-election won for term 61
I20260709 12:19:43.666648 30113 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:43.659740 29517 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:43.667899 12566 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:43.669777 12638 raft_consensus.cc:2393] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 46: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:43.669534 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:43.674566 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dbcfa3bff034f5ab82e2ae802266099" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:43.675184 13260 raft_consensus.cc:3060] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:43.678126 30111 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:43.678658 30111 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:43.679080 30111 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:43.671770 30174 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.682094 30174 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 60 FOLLOWER]: Advancing to term 61
I20260709 12:19:43.688877 30176 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.669833 30175 raft_consensus.cc:493] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.698829 12609 leader_election.cc:304] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.672830 29517 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.706362 12609 leader_election.cc:304] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 60 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.707949 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:43.708635 12566 raft_consensus.cc:3060] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Advancing to term 46
W20260709 12:19:43.709527 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:43.719306 30113 raft_consensus.cc:2749] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Leader pre-election lost for term 60. Reason: could not achieve majority
I20260709 12:19:43.708647 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.672219 30170 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.706791 29517 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:43.726342 30071 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Committing config change with OpId 45.31: config changed from index 25 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:43.706086 30171 raft_consensus.cc:2749] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:43.739207 30170 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.740222 12646 raft_consensus.cc:2393] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:43.740782 12565 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Committing config change with OpId 45.31: config changed from index 25 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:43.698410 30175 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:43.752807 13260 raft_consensus.cc:2468] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 57.
I20260709 12:19:43.752807 12571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:43.760015 12571 raft_consensus.cc:3060] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:43.768112 12611 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:43.769384 30170 raft_consensus.cc:2804] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election won for term 57
W20260709 12:19:43.770737 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 70488us
I20260709 12:19:43.776010 30170 raft_consensus.cc:697] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:43.777396 30170 consensus_queue.cc:237] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 56.31, Last appended by leader: 31, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:43.774667 12536 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.786640 12566 raft_consensus.cc:2468] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 46.
I20260709 12:19:43.788410 12609 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.789428 30171 raft_consensus.cc:2804] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:19:43.789952 30171 raft_consensus.cc:697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:43.790897 30171 consensus_queue.cc:237] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 45.26, Last appended by leader: 26, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.805840 11310 tablet_replica.cc:333] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:43.807683 11310 raft_consensus.cc:2243] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 48 LEADER]: Raft consensus shutting down.
I20260709 12:19:43.766804 30113 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader pre-election won for term 50
W20260709 12:19:43.814568 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 113921us
I20260709 12:19:43.815186 30113 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.815554 12536 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.815825 30113 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:43.816599 30164 raft_consensus.cc:2804] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:43.817174 30164 raft_consensus.cc:697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:43.815006 30148 raft_consensus.cc:2804] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader election won for term 58
I20260709 12:19:43.818197 30148 raft_consensus.cc:697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 58 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:43.818532 30164 consensus_queue.cc:237] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 51.30, Last appended by leader: 30, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.819440 30148 consensus_queue.cc:237] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 57.30, Last appended by leader: 30, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.832120 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.833346 12536 leader_election.cc:304] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.804319 29517 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.834583 30155 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:19:43.842607 30155 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:43.843075 30155 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:19:43.809407 11310 raft_consensus.cc:2272] T 43f1b7080b6a4c329281b7c405f5c998 P 6d4ece942fd5414998215d26edf91001 [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:43.838933 12571 raft_consensus.cc:2468] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:43.837795 30175 raft_consensus.cc:515] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:43.844995 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 57967.9us
I20260709 12:19:43.847965 12609 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.848013 12536 leader_election.cc:304] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.876302 12536 leader_election.cc:304] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e
I20260709 12:19:43.848343 30163 raft_consensus.cc:2749] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
W20260709 12:19:43.933225 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85424.1us
I20260709 12:19:43.844691 30174 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.906050 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 61 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.869424 30172 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.929180 30166 raft_consensus.cc:2749] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:43.899973 30170 raft_consensus.cc:2804] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election won for term 52
I20260709 12:19:43.882216 30057 raft_consensus.cc:2749] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Leader pre-election lost for term 43. Reason: could not achieve majority
I20260709 12:19:43.900198 30179 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:43.968159 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:43.951743 30148 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Leader pre-election won for term 61
I20260709 12:19:43.969060 30148 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:43.969455 30148 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 60 FOLLOWER]: Advancing to term 61
I20260709 12:19:43.970116 12645 raft_consensus.cc:2393] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 50: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:43.970306 30170 raft_consensus.cc:697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:43.971279 30172 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:43.971596 30170 consensus_queue.cc:237] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 51.29, Last appended by leader: 29, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.985788 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:43.991408 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:43.992100 12565 raft_consensus.cc:3060] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:43.975806 30113 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:43.990267 30111 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:44.005116 12389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 231) took 117817 ms (client timeout 15000 ms). Trace:
I20260709 12:19:44.006438 30111 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:44.010556 12567 raft_consensus.cc:1240] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Rejecting Update request from peer 9485b0afaf144396adf854fff857414e for earlier term 46. Current term is 47. Ops: []
I20260709 12:19:44.018925 30161 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election won for term 52
I20260709 12:19:44.026162 30161 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:44.026600 30161 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Advancing to term 52
I20260709 12:19:44.085675 11310 tablet_replica.cc:333] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:44.086835 11310 raft_consensus.cc:2243] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.088366 11310 raft_consensus.cc:2272] T 44b2b570b6dd4e36bc98e56e097afd62 P 6d4ece942fd5414998215d26edf91001 [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.092145 30113 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:44.098412 30175 leader_election.cc:290] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:44.099833 30174 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:44.057703 29517 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:44.009330 12389 rpcz_store.cc:269] 0709 12:17:46.221081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:17:46.239395 (+ 18314us) service_pool.cc:224] Handling call
0709 12:19:44.004788 (+117765393us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:43.281767 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:43.282638 (+   871us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:43.282713 (+    75us) write_op.cc:435] Acquired schema lock
0709 12:19:43.282759 (+    46us) tablet.cc:662] Decoding operations
0709 12:19:43.283273 (+   514us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:43.283405 (+   132us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:43.283451 (+    46us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:19:43.283877 (+   426us) tablet.cc:701] Row locks acquired
0709 12:19:43.283944 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:19:43.284180 (+   236us) write_op.cc:270] Start()
0709 12:19:43.284367 (+   187us) write_op.cc:276] Timestamp: P: 1783599583284095 usec, L: 0
0709 12:19:43.284422 (+    55us) op_driver.cc:348] REPLICATION: starting
0709 12:19:43.285471 (+  1049us) log.cc:844] Serialized 541 byte log entry
0709 12:19:43.549043 (+263572us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b180008ce00. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbfb5ceab1 00007ffbfb5db281 00007ffbf26e8e07 00007ffbfb5439a9 00007ffbf270569b 00007ffbf270393f 00007ffbf2703455 00007ffbf5770190 00007ffbf5769ae7 00007ffbf5768dd9 00007ffbf5773930
0709 12:19:43.656725 (+107682us) op_driver.cc:464] REPLICATION: finished
0709 12:19:43.710715 (+ 53990us) write_op.cc:301] APPLY: starting
0709 12:19:43.710894 (+   179us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:43.712993 (+  2099us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:43.713034 (+    41us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:43.713935 (+   901us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:43.714263 (+   328us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:19:43.714310 (+    47us) write_op.cc:312] APPLY: finished
0709 12:19:43.714658 (+   348us) log.cc:844] Serialized 25 byte log entry
0709 12:19:43.714878 (+   220us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:43.714978 (+   100us) write_op.cc:454] Released schema lock
0709 12:19:43.715040 (+    62us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:43.715739 (+   699us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:43.715784 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4484,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":49080,"apply.run_cpu_time_us":4957,"apply.run_wall_time_us":4960,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8303,"num_ops":1,"prepare.queue_time_us":185551,"prepare.run_cpu_time_us":26144,"prepare.run_wall_time_us":217395,"raft.queue_time_us":135340,"raft.run_cpu_time_us":12107,"raft.run_wall_time_us":57932,"replication_time_us":370684,"spinlock_wait_cycles":66346112,"thread_start_us":453621,"threads_started":4,"wal-append.queue_time_us":208231}]]}
I20260709 12:19:44.118405 12565 raft_consensus.cc:2468] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:44.119740 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:44.081959 30178 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:44.120824 30178 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.121498 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3a0781287514c758f107d79c1f96be4" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:44.123806 30178 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:44.142730 12609 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:44.154513 12610 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 58 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.170439 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 51 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:44.182781 30179 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:44.183800 30155 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.187660 30208 consensus_queue.cc:1059] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: INVALID_TERM, Last received: 46.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:44.188920 30208 raft_consensus.cc:3055] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Stepping down as leader of term 46
I20260709 12:19:44.189335 30208 raft_consensus.cc:740] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:44.190418 30208 consensus_queue.cc:260] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 46.31, Last appended by leader: 31, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:44.163846 30113 raft_consensus.cc:2804] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
W20260709 12:19:44.195233 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 112405us
I20260709 12:19:44.196476 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da4ae1a8feb46d8a6bd6ec5522932d5" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 50 candidate_status { last_received { term: 49 index: 28 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:44.197312 12563 raft_consensus.cc:3060] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Advancing to term 50
I20260709 12:19:44.197540 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81a9a92616494f46bc97254b53185664" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 51 candidate_status { last_received { term: 47 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:44.197839 12571 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 57 index: 32. Preceding OpId from leader: term: 58 index: 33. (index mismatch)
I20260709 12:19:44.198302 12564 raft_consensus.cc:3060] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:44.199242 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 61 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:44.200191 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 52 index: 32 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:44.205879 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:19:44.206629 12568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:44.210098 12568 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 48: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:44.206790 13218 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.216540 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.204444 30113 raft_consensus.cc:697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:44.218458 30113 consensus_queue.cc:237] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.27, Last appended by leader: 27, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.203435 30161 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.212833 30170 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.008s
I20260709 12:19:44.223354 12565 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 52: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:44.224397 30155 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:44.225548 30161 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:44.236615 30012 raft_consensus.cc:2764] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader pre-election decision vote started in defunct term 56: won
I20260709 12:19:44.222746 29913 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:19:44.242609 12610 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.243839 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.244278 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.208020 30208 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:44.245620 12610 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.247694 12610 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.248126 12610 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.245909 30174 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:44.248790 13258 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Committing config change with OpId 45.31: config changed from index 25 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:44.250133 11310 tablet_replica.cc:333] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:44.250326 30174 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:19:44.251260 11310 raft_consensus.cc:2243] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.251932 11310 pending_rounds.cc:70] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Trying to abort 1 pending ops.
I20260709 12:19:44.252177 11310 pending_rounds.cc:77] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001: Aborting op as it isn't in flight: id { term: 52 index: 32 } timestamp: 7305623805562724352 op_type: NO_OP noop_request { }
I20260709 12:19:44.252523 11310 raft_consensus.cc:2889] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:19:44.252861 11310 raft_consensus.cc:2272] T 40d0e235c01b4b7f89948ab708403a01 P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.194440 30148 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.256843 12566 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 61: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
W20260709 12:19:44.256824 12610 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.258292 12610 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:44.266885 29517 raft_consensus.cc:2749] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:19:44.267426 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.268008 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.268227 30148 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 61 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:44.269126 12564 raft_consensus.cc:2468] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 51.
I20260709 12:19:44.271766 13252 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 with delete_type TABLET_DATA_TOMBSTONED (TS 9485b0afaf144396adf854fff857414e not found in new config with opid_index -1) from {username='slave'} at 127.0.0.1:32770
I20260709 12:19:44.273607 12609 leader_election.cc:304] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:44.274837 29517 raft_consensus.cc:2804] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:44.275687 29517 raft_consensus.cc:697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:44.277835 30171 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:44.279317 29517 consensus_queue.cc:237] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 47.27, Last appended by leader: 27, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:44.277140 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.285796 12610 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.287452 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.283001 12563 raft_consensus.cc:2468] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 50.
I20260709 12:19:44.288538 30171 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:19:44.289173 12610 leader_election.cc:336] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.289486 12609 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:44.290496 30171 raft_consensus.cc:2804] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader election won for term 50
I20260709 12:19:44.290791 11310 tablet_replica.cc:333] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:44.291133 30171 raft_consensus.cc:697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:44.291807 11310 raft_consensus.cc:2243] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.292117 30171 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 49.28, Last appended by leader: 28, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:44.293691 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 63081.6us
I20260709 12:19:44.295852 12610 leader_election.cc:304] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:44.296994 30174 raft_consensus.cc:2749] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:44.297574 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.293728 30012 consensus_queue.cc:1048] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.005s
I20260709 12:19:44.298384 12610 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.298498 11310 raft_consensus.cc:2272] T 4d3b77689c31445cbba405c2b232495e P 6d4ece942fd5414998215d26edf91001 [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.300287 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:44.316278 12610 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.318014 12610 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.316605 30215 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:44.319478 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:19:44.321473 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.316687 30174 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:44.322371 30215 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:44.329712 30028 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:19:44.321736 12610 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.332964 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.334076 30174 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:44.344906 11310 tablet_replica.cc:333] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
W20260709 12:19:44.344718 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 191606us
I20260709 12:19:44.353395 13253 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
W20260709 12:19:44.335012 12610 leader_election.cc:336] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.356619 12610 leader_election.cc:304] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:44.356725 12537 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:44.357724 30148 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:44.359390 30148 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:44.358777 30174 raft_consensus.cc:2749] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:44.360857 30148 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 46.31, Last appended by leader: 31, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:44.359552 12610 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 174 similar messages]
I20260709 12:19:44.357594 11310 raft_consensus.cc:2243] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 54 LEADER]: Raft consensus shutting down.
W20260709 12:19:44.391014 12536 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.392050 12536 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.392457 12536 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.397997 12638 raft_consensus.cc:1275] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 49 index: 34. Preceding OpId from leader: term: 51 index: 35. (index mismatch)
I20260709 12:19:44.399688 13253 raft_consensus.cc:1275] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 56 index: 31. Preceding OpId from leader: term: 57 index: 32. (index mismatch)
I20260709 12:19:44.403484 30174 consensus_queue.cc:1048] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20260709 12:19:44.404000 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.406299 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:44.407380 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.407765 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.412809 30170 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:44.424961 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.425599 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.422185 30163 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
I20260709 12:19:44.436969 11310 raft_consensus.cc:2272] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 6d4ece942fd5414998215d26edf91001 [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.438287 30175 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:44.442269 12610 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.443272 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.443701 30170 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:19:44.444535 30179 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:19:44.446256 12610 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.446762 12610 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:44.448235 12610 leader_election.cc:336] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 62 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.448668 12610 leader_election.cc:304] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 62 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:44.449842 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.450467 12564 raft_consensus.cc:1275] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 26. Preceding OpId from leader: term: 46 index: 27. (index mismatch)
W20260709 12:19:44.450773 12610 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.451618 30174 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:19:44.451920 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.452571 30174 raft_consensus.cc:2749] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Leader pre-election lost for term 62. Reason: could not achieve majority
W20260709 12:19:44.453159 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.454310 12610 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.455459 12610 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.456579 12610 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.457461 12610 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.458376 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.458750 12610 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.459636 30171 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
W20260709 12:19:44.460088 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:44.460777 12610 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 50 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.461872 12610 leader_election.cc:336] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.462461 30178 raft_consensus.cc:493] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:44.462754 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.463267 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.463198 30178 raft_consensus.cc:515] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:44.464393 30179 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
W20260709 12:19:44.466616 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.467200 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:44.468364 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.468803 12610 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.468403 30179 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:44.471387 30179 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
W20260709 12:19:44.474840 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:44.485117 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 89882.3us
W20260709 12:19:44.485968 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:44.486905 12536 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.498469 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.515179 11310 tablet_replica.cc:333] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:44.516538 11310 raft_consensus.cc:2243] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Raft consensus shutting down.
W20260709 12:19:44.519830 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:44.521863 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.532058 12646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:44.517514 11310 raft_consensus.cc:2272] T 5d60fb5ad76741448a5376ca4d3f05df P 6d4ece942fd5414998215d26edf91001 [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.547040 12642 raft_consensus.cc:1275] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 57 index: 30. Preceding OpId from leader: term: 58 index: 31. (index mismatch)
I20260709 12:19:44.553102 12646 raft_consensus.cc:2393] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 48: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:44.555117 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 61 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:44.555968 12572 raft_consensus.cc:1275] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 51 index: 29. Preceding OpId from leader: term: 52 index: 30. (index mismatch)
I20260709 12:19:44.557618 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:44.559864 12639 raft_consensus.cc:2393] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 61: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
I20260709 12:19:44.561465 12638 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 52 index: 32. Preceding OpId from leader: term: 53 index: 33. (index mismatch)
I20260709 12:19:44.575033 30171 consensus_queue.cc:1048] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
I20260709 12:19:44.577486 12637 raft_consensus.cc:2393] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a29b5d9f9d44298b02ffcd460811eef in current term 52: Already voted for candidate 32889a507a6b4c74aec421ffbfd1e25d in this term.
W20260709 12:19:44.586333 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.538483 30179 consensus_queue.cc:1048] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.043s
W20260709 12:19:44.591192 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:44.576179 30148 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:44.597245 30148 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.553949 30215 consensus_queue.cc:1048] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.036s
W20260709 12:19:44.593580 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:44.594486 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:44.607247 12610 leader_election.cc:336] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.610421 12391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 232) took 180471 ms (client timeout 15000 ms). Trace:
I20260709 12:19:44.610919 12627 tablet_service.cc:1558] Processing DeleteTablet for tablet 27144fedf0014de09f007ac6cc3f7275 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 7 (current committed config index is 19)) from {username='slave'} at 127.0.0.1:47032
W20260709 12:19:44.611773 12391 rpcz_store.cc:269] 0709 12:16:44.138501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:53.669998 (+9531497us) service_pool.cc:224] Handling call
0709 12:19:44.610080 (+170940082us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":664011,"reported_tablets":51}
I20260709 12:19:44.599581 30148 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:44.630924 13260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c56dde469374f4da7440f134140670c" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 52 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:44.628464 30178 leader_election.cc:290] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:19:44.646531 12381 catalog_manager.cc:5002] TS 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): delete failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 25 but the committed config has opid_index of 31
I20260709 12:19:44.649530 11310 tablet_replica.cc:333] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:44.659900 11310 raft_consensus.cc:2243] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.660822 11310 raft_consensus.cc:2272] T 509f0430baf04e3393879d82fb9d10f9 P 6d4ece942fd5414998215d26edf91001 [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.650067 12611 leader_election.cc:304] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:44.662927 30175 raft_consensus.cc:2749] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:44.718197 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:44.677286 11310 tablet_replica.cc:333] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001: stopping tablet replica
I20260709 12:19:44.731603 12643 raft_consensus.cc:1275] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 51 index: 30. Preceding OpId from leader: term: 52 index: 31. (index mismatch)
I20260709 12:19:44.720145 30111 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.025s
W20260709 12:19:44.746966 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.738021 11310 raft_consensus.cc:2243] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Raft consensus shutting down.
W20260709 12:19:44.748180 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.748706 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:44.749040 11310 raft_consensus.cc:2272] T 59beb90f79b0450ea2fbb2b7e405261d P 6d4ece942fd5414998215d26edf91001 [term 55 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:44.751139 12536 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.751647 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:44.758955 12536 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.759766 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.757289 30211 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
I20260709 12:19:44.714534 30164 consensus_queue.cc:1048] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.053s
W20260709 12:19:44.796221 12536 leader_election.cc:336] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.767915 30155 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:19:44.807961 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:44.806457 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.805181 12389 catalog_manager.cc:5697] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 44 to 48. New cstate: current_term: 48 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:44.817261 12389 catalog_manager.cc:5697] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 38 to 46. New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:44.820566 12536 leader_election.cc:336] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.821233 12536 leader_election.cc:304] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.823318 12571 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 28. Preceding OpId from leader: term: 50 index: 29. (index mismatch)
I20260709 12:19:44.807030 30166 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:19:44.819036 12389 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 41 to 46. New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:44.855494 12571 raft_consensus.cc:1275] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 49 index: 27. Preceding OpId from leader: term: 50 index: 28. (index mismatch)
I20260709 12:19:44.826072 30111 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:44.827889 30211 raft_consensus.cc:2749] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:19:44.860733 12571 raft_consensus.cc:1275] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 27. Preceding OpId from leader: term: 51 index: 28. (index mismatch)
I20260709 12:19:44.855197 30174 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:44.855253 12389 catalog_manager.cc:5697] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 49 to 53. New cstate: current_term: 53 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:44.869076 12389 catalog_manager.cc:5697] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 42 to 51. New cstate: current_term: 51 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:44.874152 30174 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:44.860747 30179 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.013s
W20260709 12:19:44.873801 30233 log.cc:927] Time spent T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:19:44.875157 12389 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 43 to 50. New cstate: current_term: 50 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:44.887737 12389 catalog_manager.cc:5697] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 44 to 51. New cstate: current_term: 51 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:44.898783 12389 catalog_manager.cc:5697] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 37 to 45, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:44.905961 30113 consensus_queue.cc:1048] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20260709 12:19:44.908459 30113 consensus_queue.cc:1048] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.001s
W20260709 12:19:44.925804 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 105260us
W20260709 12:19:44.927445 12536 leader_election.cc:336] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.928174 12536 leader_election.cc:304] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:19:44.928696 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:44.925292 12389 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 45. New cstate: current_term: 45 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:44.935027 12536 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.935781 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.937726 12389 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 42 to 52, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 52 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:44.921734 12380 catalog_manager.cc:5028] TS 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): tablet 27144fedf0014de09f007ac6cc3f7275 (table test-workload [id=f190dd56e4fd4ca6a9b59414b60864a9]) successfully deleted
I20260709 12:19:44.935761 30161 raft_consensus.cc:2749] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:19:44.954885 12389 catalog_manager.cc:5697] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 54, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 54 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:44.960914 12536 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.965735 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.960069 30155 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:19:44.972167 30232 log.cc:927] Time spent T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
W20260709 12:19:44.974246 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.974774 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.976282 30163 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:19:44.977814 12536 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.978339 12536 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.974043 30155 raft_consensus.cc:2749] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:44.979218 30163 raft_consensus.cc:2749] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
W20260709 12:19:44.981055 12536 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.981602 12536 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.982422 30163 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:19:44.983642 12536 leader_election.cc:336] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.984087 12536 leader_election.cc:304] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:44.984905 30163 raft_consensus.cc:2749] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:19:44.986409 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.988305 12536 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.988786 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:44.994221 12536 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.994959 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:44.966403 29517 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:44.996662 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 61 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:44.998929 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:45.001742 12392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 233) took 242100 ms (client timeout 15000 ms). Trace:
W20260709 12:19:45.002343 12392 rpcz_store.cc:269] 0709 12:15:42.855598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:15:54.900351 (+12044753us) service_pool.cc:224] Handling call
0709 12:19:44.951613 (+230051262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":89189,"reported_tablets":51}
I20260709 12:19:45.008642 30174 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:44.997761 30111 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:19:45.010620 29517 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.013451 29517 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:45.024892 30215 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:44.994091 12389 catalog_manager.cc:5697] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 36 to 45, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 45 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:45.025682 30215 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:45.026659 12389 catalog_manager.cc:5697] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 47 to 51. New cstate: current_term: 51 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.028651 12389 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 49 to 56, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 56 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:45.029510 30215 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:45.043557 12389 catalog_manager.cc:5697] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 46 to 56, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 56 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.047817 30242 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:45.048671 30242 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:45.051259 12569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "256d044fd2e244d4be827104688867ee" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 59 candidate_status { last_received { term: 57 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
W20260709 12:19:45.055287 30234 log.cc:927] Time spent T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.200s	user 0.003s	sys 0.001s
I20260709 12:19:45.057422 12389 catalog_manager.cc:5697] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 48 to 56. New cstate: current_term: 56 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.065038 12389 catalog_manager.cc:5697] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 46 to 52. New cstate: current_term: 52 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.071787 12389 catalog_manager.cc:5697] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 46 to 53, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 53 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:45.077214 12389 catalog_manager.cc:5697] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 47 to 55. New cstate: current_term: 55 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.080127 12570 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:45.081008 12570 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 56.
I20260709 12:19:45.057662 30161 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:19:45.091419 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:45.093578 12389 catalog_manager.cc:5697] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 38 to 42, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 42 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.094933 30238 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election won for term 57
I20260709 12:19:45.098333 30238 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.098882 30238 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Advancing to term 57
I20260709 12:19:45.103266 30242 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:45.109575 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:45.167552 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:45.168920 12643 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 50.
I20260709 12:19:45.192091 13253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 47 candidate_status { last_received { term: 46 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:45.193261 13253 raft_consensus.cc:2393] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32889a507a6b4c74aec421ffbfd1e25d in current term 47: Already voted for candidate 0a29b5d9f9d44298b02ffcd460811eef in this term.
I20260709 12:19:45.177197 12389 catalog_manager.cc:5697] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 48 to 61, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 61 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.199723 12638 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:45.234499 12611 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:19:45.241048 13253 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 31. Preceding OpId from leader: term: 47 index: 32. (index mismatch)
I20260709 12:19:45.242735 12638 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 31. Preceding OpId from leader: term: 47 index: 32. (index mismatch)
I20260709 12:19:45.244808 30171 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:19:45.254586 30236 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:45.233525 30238 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.272117 30238 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:45.275095 30218 consensus_queue.cc:1048] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.034s
I20260709 12:19:45.210081 12389 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index -1 to 25, term changed from 34 to 45, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131), NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:45.292634 12389 catalog_manager.cc:5697] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 44 to 54. New cstate: current_term: 54 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:45.285218 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:19:45.304142 30208 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election won for term 51
I20260709 12:19:45.313129 12389 catalog_manager.cc:5697] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 43 to 49. New cstate: current_term: 49 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:45.304890 30208 raft_consensus.cc:493] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.319420 30208 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:45.322268 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 57 candidate_status { last_received { term: 56 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:45.323421 12566 raft_consensus.cc:3060] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Advancing to term 57
W20260709 12:19:45.359395 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 53218.3us
W20260709 12:19:45.369899 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 66186.3us
W20260709 12:19:45.394709 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 143132us
W20260709 12:19:45.401854 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 160888us
W20260709 12:19:45.403329 30252 log.cc:927] Time spent T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:19:45.435132 30250 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:45.438652 30208 raft_consensus.cc:515] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:45.446842 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 47986.8us
W20260709 12:19:45.456528 13218 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:45.458570 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:45.459594 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:45.461550 12566 raft_consensus.cc:2468] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 57.
I20260709 12:19:45.464417 12609 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
W20260709 12:19:45.464905 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:45.465517 30242 raft_consensus.cc:2804] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Leader election won for term 57
I20260709 12:19:45.466264 30242 raft_consensus.cc:697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:45.467195 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:45.467396 30242 consensus_queue.cc:237] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 56.25, Last appended by leader: 25, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:45.483520 12610 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:45.484550 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.484840 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:45.498515 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.498831 30208 leader_election.cc:290] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:45.499280 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:45.489984 30179 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader pre-election lost for term 59. Reason: could not achieve majority
I20260709 12:19:45.501477 30163 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:45.510097 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.505759 30218 raft_consensus.cc:2749] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20260709 12:19:45.522071 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:45.526499 12536 consensus_peers.cc:597] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:45.527383 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 61 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.527848 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:19:45.525215 12610 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.530062 30111 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20260709 12:19:45.526386 30166 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
W20260709 12:19:45.533144 12536 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.533831 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:45.534844 30111 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:19:45.536888 12536 consensus_peers.cc:597] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:45.529372 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:45.537839 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 50997.7us
I20260709 12:19:45.553994 30242 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.555209 30242 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.574245 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:45.559489 30242 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:45.575279 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 54.
I20260709 12:19:45.577247 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:45.578860 30179 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election won for term 55
I20260709 12:19:45.580742 30179 raft_consensus.cc:493] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.529297 30163 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.581449 30179 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Advancing to term 55
W20260709 12:19:45.592938 30256 log.cc:927] Time spent T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 12:19:45.593884 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:45.604363 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c793ebd4de24492397303f9a6e6bc62a" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 51 candidate_status { last_received { term: 50 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:45.605403 12637 raft_consensus.cc:3060] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Advancing to term 51
I20260709 12:19:45.615926 30163 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:45.643419 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:45.644436 12643 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 46.
I20260709 12:19:45.650931 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:45.652446 30218 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election won for term 47
I20260709 12:19:45.652911 30218 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.653309 30218 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Advancing to term 47
W20260709 12:19:45.633816 12610 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.657486 12637 raft_consensus.cc:2468] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 51.
W20260709 12:19:45.670818 12536 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.683262 30179 raft_consensus.cc:515] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.686473 30179 leader_election.cc:290] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:45.693922 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03d0d1b7d2b24046b4dd2cdb4cf7c0af" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 55 candidate_status { last_received { term: 54 index: 31 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:45.694777 12563 raft_consensus.cc:3060] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Advancing to term 55
W20260709 12:19:45.674374 13218 leader_election.cc:336] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.701654 13218 leader_election.cc:304] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:45.703112 30210 raft_consensus.cc:2804] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader election won for term 51
I20260709 12:19:45.703827 30210 raft_consensus.cc:697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:19:45.705001 30210 consensus_queue.cc:237] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 50.31, Last appended by leader: 31, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:45.738492 12563 raft_consensus.cc:2468] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 55.
I20260709 12:19:45.740018 12609 leader_election.cc:304] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:45.741097 30171 raft_consensus.cc:2804] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Leader election won for term 55
I20260709 12:19:45.742190 30171 raft_consensus.cc:697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 55 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:45.744079 30171 consensus_queue.cc:237] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 54.31, Last appended by leader: 31, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:45.748656 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:45.756951 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:45.770030 12610 leader_election.cc:336] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.779965 30238 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.781078 30238 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.783993 30238 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:45.786732 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:45.787619 12563 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 43.
I20260709 12:19:45.797000 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:45.817795 30178 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:19:45.822443 30178 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:45.799144 30218 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.853798 30218 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:45.855927 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44b2b570b6dd4e36bc98e56e097afd62" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:45.857187 12637 raft_consensus.cc:3060] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Advancing to term 47
I20260709 12:19:45.868229 12637 raft_consensus.cc:2468] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 47.
I20260709 12:19:45.847906 30178 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:19:45.874137 12536 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:45.886803 30163 raft_consensus.cc:2804] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader election won for term 47
I20260709 12:19:45.887598 30163 raft_consensus.cc:697] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
I20260709 12:19:45.888787 30163 consensus_queue.cc:237] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 46.26, Last appended by leader: 26, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:45.896008 12536 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:45.897888 30178 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:45.902061 30178 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:45.916934 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d3b77689c31445cbba405c2b232495e" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 44 candidate_status { last_received { term: 43 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:45.917906 12566 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:19:45.918586 12610 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:45.920075 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:46.106817 12566 raft_consensus.cc:2468] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 44.
I20260709 12:19:46.110757 12609 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:46.113183 12564 raft_consensus.cc:1275] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 56 index: 25. Preceding OpId from leader: term: 57 index: 26. (index mismatch)
W20260709 12:19:46.132256 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 201424us
I20260709 12:19:46.141860 30179 raft_consensus.cc:2804] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:19:46.143462 30174 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.174669 30178 consensus_queue.cc:1048] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.028s
W20260709 12:19:46.210536 30267 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:19:46.197692 30179 raft_consensus.cc:697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
W20260709 12:19:46.229482 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:46.232811 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:46.189141 30174 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.247993 30269 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
W20260709 12:19:46.248507 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57540.7us
W20260709 12:19:46.264819 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:46.251241 30269 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.233067 30179 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 43.25, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:46.289572 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:46.301782 12642 raft_consensus.cc:1275] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 50 index: 31. Preceding OpId from leader: term: 51 index: 32. (index mismatch)
W20260709 12:19:46.307953 12610 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 44 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:46.323212 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:46.333297 30269 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:46.343647 12566 raft_consensus.cc:1275] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 54 index: 31. Preceding OpId from leader: term: 55 index: 32. (index mismatch)
I20260709 12:19:46.352850 30174 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:46.366606 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:46.399618 12566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:46.400413 12566 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:46.402777 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:46.403578 12563 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 48.
I20260709 12:19:46.410672 29913 consensus_queue.cc:1048] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:46.404836 30179 consensus_queue.cc:1048] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.032s
I20260709 12:19:46.458837 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:46.460032 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:46.461894 30174 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:46.462530 30174 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.462985 30174 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:46.461618 30273 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election won for term 49
I20260709 12:19:46.466624 30273 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:46.467099 30273 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:46.489351 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:46.506948 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:46.527400 30174 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.541932 30273 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.544493 30273 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:46.547176 12536 consensus_peers.cc:597] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:46.548723 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:46.549747 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:46.550377 12610 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:46.551640 12564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 48 index: 30 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
W20260709 12:19:46.552038 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:46.552367 12564 raft_consensus.cc:3060] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
I20260709 12:19:46.569154 30275 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.569906 30275 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.573871 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:46.581156 12572 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:46.582562 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:46.577762 12638 raft_consensus.cc:1275] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 46 index: 26. Preceding OpId from leader: term: 47 index: 27. (index mismatch)
I20260709 12:19:46.586175 12389 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47684 (request call id 141) took 5778 ms. Trace:
I20260709 12:19:46.573745 30275 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:46.588282 30276 consensus_queue.cc:1048] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:46.586560 12564 raft_consensus.cc:2468] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
I20260709 12:19:46.603950 30174 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:19:46.606117 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:46.591753 30238 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:46.611966 12609 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:46.609297 30238 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.613966 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 49 candidate_status { last_received { term: 47 index: 25 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:46.617237 12572 raft_consensus.cc:3060] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Advancing to term 49
W20260709 12:19:46.612185 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:46.619043 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:46.605594 12389 rpcz_store.cc:276] 0709 12:19:40.808037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:44.108841 (+3300804us) service_pool.cc:224] Handling call
0709 12:19:46.586042 (+2477201us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:45.612993 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:45.613571 (+   578us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:45.613609 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:19:45.613650 (+    41us) tablet.cc:662] Decoding operations
0709 12:19:45.615389 (+  1739us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:45.615455 (+    66us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:45.615483 (+    28us) tablet.cc:685] Acquiring locks for 24 operations
0709 12:19:45.617608 (+  2125us) tablet.cc:701] Row locks acquired
0709 12:19:45.622521 (+  4913us) write_op.cc:260] PREPARE: finished
0709 12:19:45.630899 (+  8378us) write_op.cc:270] Start()
0709 12:19:45.633784 (+  2885us) write_op.cc:276] Timestamp: P: 1783599585627684 usec, L: 0
0709 12:19:45.643187 (+  9403us) op_driver.cc:348] REPLICATION: starting
0709 12:19:45.666093 (+ 22906us) log.cc:844] Serialized 9409 byte log entry
0709 12:19:45.744739 (+ 78646us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575c571 00007ffbf57eee28 00007ffbf9229581 00007ffbf9228d58 00007ffbf922d9f1 00007ffbf922d9a9 00007ffbf922d939 00007ffbf922d901 00007ffbf922cb2d 0000000000388634 0000000000388479
0709 12:19:45.872090 (+127351us) op_driver.cc:464] REPLICATION: finished
0709 12:19:45.941107 (+ 69017us) write_op.cc:301] APPLY: starting
0709 12:19:46.095058 (+153951us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:46.316658 (+221600us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:46.316690 (+    32us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:46.483797 (+167107us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:46.484177 (+   380us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=0,mrs_lookups=0
0709 12:19:46.484225 (+    48us) write_op.cc:312] APPLY: finished
0709 12:19:46.485857 (+  1632us) log.cc:844] Serialized 212 byte log entry
0709 12:19:46.486054 (+   197us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:46.486354 (+   300us) write_op.cc:454] Released schema lock
0709 12:19:46.486418 (+    64us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:46.527278 (+ 40860us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:46.528967 (+  1689us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":152085,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":55934,"apply.run_cpu_time_us":43399,"apply.run_wall_time_us":564814,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104351,"num_ops":24,"prepare.queue_time_us":90407,"prepare.run_cpu_time_us":44696,"prepare.run_wall_time_us":132033,"raft.queue_time_us":98688,"raft.run_cpu_time_us":7120,"raft.run_wall_time_us":55642,"replication_time_us":224503,"spinlock_wait_cycles":28747136,"thread_start_us":241828,"threads_started":4,"wal-append.queue_time_us":60621}]]}
I20260709 12:19:46.622293 30275 raft_consensus.cc:2804] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:46.628329 30275 raft_consensus.cc:697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:46.629420 30275 consensus_queue.cc:237] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 48.30, Last appended by leader: 30, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.619962 30238 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:46.658298 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:46.677242 12610 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:46.678539 12610 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:46.687532 12536 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:46.681067 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:46.702080 12387 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 53 to 56, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 56 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.705861 12572 raft_consensus.cc:2468] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 49.
W20260709 12:19:46.718178 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:46.719692 12609 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:46.727344 30273 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:46.735608 30273 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.760154 30178 raft_consensus.cc:2804] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Leader election won for term 49
I20260709 12:19:46.713438 30238 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.800226 12387 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 40 to 45. New cstate: current_term: 45 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } }
W20260709 12:19:46.808768 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69037us
I20260709 12:19:46.818452 12387 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 40 to 45, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 45 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.832922 30178 raft_consensus.cc:697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:46.834487 30178 consensus_queue.cc:237] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 47.25, Last appended by leader: 25, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:46.855038 12387 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 43 to 47. New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.857220 12387 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 40 to 48. New cstate: current_term: 48 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.926263 30238 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:46.893316 12387 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 47 to 55. New cstate: current_term: 55 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.938187 12387 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 39 to 40, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 40 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:46.945906 30288 log.cc:927] Time spent T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.087s	user 0.004s	sys 0.000s
W20260709 12:19:46.963608 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:46.967252 12564 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 43 index: 25. Preceding OpId from leader: term: 44 index: 26. (index mismatch)
I20260709 12:19:46.967505 30273 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:19:46.985493 12387 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 41 to 47. New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.987290 12387 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 50 to 52. New cstate: current_term: 52 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:46.989825 30286 log.cc:927] Time spent T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.116s	user 0.003s	sys 0.000s
I20260709 12:19:46.988963 12387 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 39 to 47, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.990839 12387 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 49 to 51. New cstate: current_term: 51 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.992650 12387 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 46 to 53, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 53 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.994239 12387 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 42 to 49. New cstate: current_term: 49 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.996420 12387 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 43 to 45. New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:46.998513 12387 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 40 to 45. New cstate: current_term: 45 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.001785 12387 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 49 to 51, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 51 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:46.988852 30287 log.cc:927] Time spent T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.112s	user 0.002s	sys 0.000s
I20260709 12:19:47.013476 12572 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef" is_pre_election: true
I20260709 12:19:47.014338 12572 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:19:47.019001 12565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 51 index: 26 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:47.019678 12565 raft_consensus.cc:3060] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:47.027132 30295 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
W20260709 12:19:47.031288 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:47.037995 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:47.047346 30275 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
I20260709 12:19:47.047853 30275 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (detected failure of leader 6d4ece942fd5414998215d26edf91001)
I20260709 12:19:47.048244 30275 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
W20260709 12:19:47.038826 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:47.068925 12565 raft_consensus.cc:2468] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
W20260709 12:19:47.074591 12610 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:47.075330 12610 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:19:47.076406 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:47.084187 12609 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:47.086608 30269 raft_consensus.cc:2804] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:47.087301 30269 raft_consensus.cc:697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:47.089057 30269 consensus_queue.cc:237] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 51.26, Last appended by leader: 26, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:47.132952 30275 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:47.164355 12563 raft_consensus.cc:1275] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 48 index: 30. Preceding OpId from leader: term: 49 index: 31. (index mismatch)
W20260709 12:19:47.163125 12610 connection.cc:573] client connection to 127.11.11.129:38859 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260709 12:19:47.166762 30295 consensus_queue.cc:1048] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:19:47.191219 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:47.199426 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:47.200116 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:47.227298 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:47.247377 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:47.255770 30275 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859)
I20260709 12:19:47.242626 12563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 27 } } ignore_live_leader: false dest_uuid: "0a29b5d9f9d44298b02ffcd460811eef"
I20260709 12:19:47.257439 12563 raft_consensus.cc:3060] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:19:47.283416 12563 raft_consensus.cc:2468] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:19:47.285707 12609 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:47.290802 30275 raft_consensus.cc:2804] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:19:47.292536 30155 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:47.293289 30155 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:47.295873 30155 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 62 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:47.302210 12643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 62 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:47.303115 12643 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 61.
I20260709 12:19:47.291468 30275 raft_consensus.cc:697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:19:47.315224 30275 consensus_queue.cc:237] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 52.27, Last appended by leader: 27, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:47.333251 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 62 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 
I20260709 12:19:47.334352 30302 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Leader pre-election won for term 62
I20260709 12:19:47.334972 30302 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:47.335996 30302 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 61 FOLLOWER]: Advancing to term 62
W20260709 12:19:47.334281 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:47.349426 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 62 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:19:47.353416 30302 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:47.358928 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:47.360703 12610 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:19:47.360930 12563 raft_consensus.cc:1275] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 47 index: 25. Preceding OpId from leader: term: 49 index: 26. (index mismatch)
W20260709 12:19:47.361663 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:47.364112 30273 consensus_queue.cc:1048] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20260709 12:19:47.367179 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:47.387476 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:47.398056 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:47.411338 30302 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 62 election: Requested vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:47.413056 12637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02f3cd3c4cc54eb9adb6f7c50cea457e" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 62 candidate_status { last_received { term: 59 index: 29 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:19:47.413897 12637 raft_consensus.cc:3060] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Advancing to term 62
W20260709 12:19:47.414845 12536 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 62 election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:19:47.433008 12637 raft_consensus.cc:2468] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a29b5d9f9d44298b02ffcd460811eef in term 62.
I20260709 12:19:47.435945 12536 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 62 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d; no voters: 6d4ece942fd5414998215d26edf91001
I20260709 12:19:47.444566 30302 raft_consensus.cc:2804] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 62 FOLLOWER]: Leader election won for term 62
I20260709 12:19:47.475070 30302 raft_consensus.cc:697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 62 LEADER]: Becoming Leader. State: Replica: 0a29b5d9f9d44298b02ffcd460811eef, State: Running, Role: LEADER
W20260709 12:19:47.495003 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:47.496424 30302 consensus_queue.cc:237] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 59.29, Last appended by leader: 29, Current term: 62, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:47.497407 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:47.627251 12389 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 46 to 47, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 47 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:47.629114 12389 catalog_manager.cc:5697] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 43 to 51, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 51 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:47.638223 12389 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 25 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } attrs { replace: true } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:47.656356 12389 catalog_manager.cc:5697] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 56 to 59, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 59 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:47.662508 12571 raft_consensus.cc:1275] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 51 index: 26. Preceding OpId from leader: term: 53 index: 27. (index mismatch)
I20260709 12:19:47.667512 12389 catalog_manager.cc:5697] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 56 to 57, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 57 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:47.667842 30275 consensus_queue.cc:1048] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20260709 12:19:47.675299 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:47.669322 12389 catalog_manager.cc:5697] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 50 to 51, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 51 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:19:47.702582 12387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 241) took 28708 ms (client timeout 15000 ms). Trace:
I20260709 12:19:47.703580 12389 catalog_manager.cc:5697] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 40 to 51, leader changed from 9485b0afaf144396adf854fff857414e (127.11.11.132) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 51 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:47.709872 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:47.712473 12387 rpcz_store.cc:269] 0709 12:19:18.994408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:33.352138 (+14357730us) service_pool.cc:224] Handling call
0709 12:19:47.702466 (+14350328us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:47.093394 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:47.094026 (+   632us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:47.094069 (+    43us) write_op.cc:435] Acquired schema lock
0709 12:19:47.094087 (+    18us) tablet.cc:662] Decoding operations
0709 12:19:47.095473 (+  1386us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:47.095569 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:47.095598 (+    29us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:19:47.096820 (+  1222us) tablet.cc:701] Row locks acquired
0709 12:19:47.096840 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:19:47.097029 (+   189us) write_op.cc:270] Start()
0709 12:19:47.097166 (+   137us) write_op.cc:276] Timestamp: P: 1783599587096967 usec, L: 0
0709 12:19:47.097197 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:19:47.118139 (+ 20942us) log.cc:844] Serialized 6353 byte log entry
0709 12:19:47.240510 (+122371us) op_driver.cc:464] REPLICATION: finished
0709 12:19:47.318123 (+ 77613us) write_op.cc:301] APPLY: starting
0709 12:19:47.318222 (+    99us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:47.388742 (+ 70520us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:47.388905 (+   163us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:47.593660 (+204755us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:47.593979 (+   319us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:19:47.594025 (+    46us) write_op.cc:312] APPLY: finished
0709 12:19:47.601831 (+  7806us) log.cc:844] Serialized 148 byte log entry
0709 12:19:47.602056 (+   225us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:47.602262 (+   206us) write_op.cc:454] Released schema lock
0709 12:19:47.602335 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:47.614120 (+ 11785us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:47.614157 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":397,"reported_tablets":51,"spinlock_wait_cycles":10624,"child_traces":[["op",{"apply.queue_time_us":77272,"apply.run_cpu_time_us":26561,"apply.run_wall_time_us":295914,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70943,"num_ops":16,"prepare.queue_time_us":73151,"prepare.run_cpu_time_us":5137,"prepare.run_wall_time_us":27807,"replication_time_us":143145,"thread_start_us":88897,"threads_started":2}]]}
I20260709 12:19:47.796305 12571 raft_consensus.cc:1275] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 52 index: 27. Preceding OpId from leader: term: 53 index: 28. (index mismatch)
I20260709 12:19:47.798789 30273 consensus_queue.cc:1048] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20260709 12:19:47.808068 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:47.814368 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:47.817054 12386 catalog_manager.cc:5697] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 41 to 44. New cstate: current_term: 44 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.818830 12386 catalog_manager.cc:5697] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 51 to 52, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 52 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:47.821300 12386 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 38 to 47, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.822973 12386 catalog_manager.cc:5697] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 44 to 48, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 48 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.824474 12386 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 45 to 49. New cstate: current_term: 49 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.826300 12386 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 37 to 46, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 46 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.828130 12386 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 43 to 49, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 49 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.829967 12386 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 45 to 49, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 49 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:47.838375 12536 consensus_peers.cc:597] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:47.855438 12386 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 52 to 55, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 55 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.862681 12386 catalog_manager.cc:5697] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 51, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 51 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:47.858633 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:47.865065 12386 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 48 to 50. New cstate: current_term: 50 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.867197 12386 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 50 to 54, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 54 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.868589 12386 catalog_manager.cc:5697] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 41 to 46. New cstate: current_term: 46 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:47.870446 12386 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 56, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 56 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.873107 12386 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 53 to 57. New cstate: current_term: 57 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.874603 12386 catalog_manager.cc:5697] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 51. New cstate: current_term: 51 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:47.876389 12386 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 49 to 51. New cstate: current_term: 51 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.877794 12386 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 44 to 54, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 54 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.880484 12386 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 38 to 43, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 43 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.882025 12386 catalog_manager.cc:5697] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 45 to 47, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 47 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:47.884109 12386 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 45 to 47, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.885427 12386 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 37 to 44, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 44 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.889599 12386 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 43 to 50, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 50 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.894237 12386 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 35 to 40, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 40 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:47.895993 12386 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 51 to 53. New cstate: current_term: 53 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.015038 12638 raft_consensus.cc:1275] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 59 index: 29. Preceding OpId from leader: term: 62 index: 30. (index mismatch)
W20260709 12:19:48.022224 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:48.021957 30298 consensus_queue.cc:1048] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:48.063716 12536 consensus_peers.cc:597] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:48.071441 12386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42880 (request call id 239) took 12381 ms (client timeout 15000 ms). Trace:
W20260709 12:19:48.081748 12386 rpcz_store.cc:269] 0709 12:19:35.704745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:40.370583 (+4665838us) service_pool.cc:224] Handling call
0709 12:19:48.071323 (+7700740us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:47.921332 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:47.922042 (+   710us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:47.922089 (+    47us) write_op.cc:435] Acquired schema lock
0709 12:19:47.922109 (+    20us) tablet.cc:662] Decoding operations
0709 12:19:47.924444 (+  2335us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:47.924522 (+    78us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:47.924559 (+    37us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:19:47.927012 (+  2453us) tablet.cc:701] Row locks acquired
0709 12:19:47.927037 (+    25us) write_op.cc:260] PREPARE: finished
0709 12:19:47.927276 (+   239us) write_op.cc:270] Start()
0709 12:19:47.927427 (+   151us) write_op.cc:276] Timestamp: P: 1783599587927199 usec, L: 0
0709 12:19:47.927452 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:19:47.928460 (+  1008us) log.cc:844] Serialized 9846 byte log entry
0709 12:19:47.932279 (+  3819us) op_driver.cc:464] REPLICATION: finished
0709 12:19:47.932806 (+   527us) write_op.cc:301] APPLY: starting
0709 12:19:47.932914 (+   108us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:47.958485 (+ 25571us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:47.958621 (+   136us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:47.975579 (+ 16958us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:47.975995 (+   416us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0709 12:19:47.976055 (+    60us) write_op.cc:312] APPLY: finished
0709 12:19:47.976674 (+   619us) log.cc:844] Serialized 220 byte log entry
0709 12:19:47.977716 (+  1042us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:47.978014 (+   298us) write_op.cc:454] Released schema lock
0709 12:19:47.978093 (+    79us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:48.033960 (+ 55867us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.033996 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":114835,"reported_tablets":51,"spinlock_wait_cycles":17280,"child_traces":[["op",{"apply.queue_time_us":291,"apply.run_cpu_time_us":32511,"apply.run_wall_time_us":101063,"cfile_cache_hit":100,"cfile_cache_hit_bytes":104400,"num_ops":25,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":7714,"prepare.run_wall_time_us":27947,"replication_time_us":4628,"spinlock_wait_cycles":18176}]]}
W20260709 12:19:48.081523 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:48.105384 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:48.133524 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:48.148535 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:48.154690 12395 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 36 to 44. New cstate: current_term: 44 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:48.164428 12390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 229) took 191740 ms (client timeout 15000 ms). Trace:
W20260709 12:19:48.164916 12390 rpcz_store.cc:269] 0709 12:16:36.423902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:44.574664 (+8150762us) service_pool.cc:224] Handling call
0709 12:19:48.164177 (+183589513us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6916,"reported_tablets":51}
W20260709 12:19:48.166002 12379 connection.cc:466] server connection from 127.0.0.1:42850 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 229) could send its response
W20260709 12:19:48.184758 11310 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:48.190378 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:48.212122 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:48.229760 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 45264.5us
W20260709 12:19:48.232668 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a30ed0986fa641bd8f88912b4b9ee320: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:19:48.235096 12394 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 40 to 41, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 41 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:48.252213 12395 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 233) took 65035 ms (client timeout 15000 ms). Trace:
W20260709 12:19:48.253856 12395 rpcz_store.cc:269] 0709 12:18:43.216722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:18:51.495853 (+8279131us) service_pool.cc:224] Handling call
0709 12:18:52.414107 (+918254us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400268754. stack: 00007ffbf26d4343 00007ffbf26d4240 00007ffbf20631d4 00007ffbfbda1304 00007ffbfbda1299 00007ffbf575d090 00007ffbf57fbf3a 00007ffbfb5b1e9c 00007ffbfb6b004e 00007ffbf9b712a4 00007ffbf9b71232 00007ffbf9b71161 00007ffbf9b710dc 00007ffbf9b70392 00007ffbf3b2e48c 00007ffbf3b2d8c6
0709 12:19:48.176798 (+55762691us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.176833 (+    35us) write_op.cc:454] Released schema lock
0709 12:19:48.252127 (+ 75294us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:48.167893 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:48.168480 (+   587us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:48.168511 (+    31us) write_op.cc:435] Acquired schema lock
0709 12:19:48.168527 (+    16us) tablet.cc:662] Decoding operations
0709 12:19:48.168886 (+   359us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:48.168951 (+    65us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:48.168977 (+    26us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:19:48.169175 (+   198us) tablet.cc:701] Row locks acquired
0709 12:19:48.169192 (+    17us) write_op.cc:260] PREPARE: finished
0709 12:19:48.169336 (+   144us) write_op.cc:270] Start()
0709 12:19:48.169456 (+   120us) write_op.cc:276] Timestamp: P: 1783599588169277 usec, L: 0
0709 12:19:48.169482 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:19:48.170215 (+   733us) log.cc:844] Serialized 541 byte log entry
0709 12:19:48.172515 (+  2300us) op_driver.cc:464] REPLICATION: finished
0709 12:19:48.173002 (+   487us) write_op.cc:301] APPLY: starting
0709 12:19:48.173080 (+    78us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:48.173987 (+   907us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:48.174081 (+    94us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:48.174838 (+   757us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:48.175004 (+   166us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:19:48.175048 (+    44us) write_op.cc:312] APPLY: finished
0709 12:19:48.175249 (+   201us) log.cc:844] Serialized 25 byte log entry
0709 12:19:48.175431 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.175526 (+    95us) write_op.cc:454] Released schema lock
0709 12:19:48.175571 (+    45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":10902,"reported_tablets":51,"spinlock_wait_cycles":585050496,"child_traces":[["op",{"apply.queue_time_us":253,"apply.run_cpu_time_us":2917,"apply.run_wall_time_us":2917,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4176,"num_ops":1,"prepare.queue_time_us":4110,"prepare.run_cpu_time_us":2641,"prepare.run_wall_time_us":2641,"replication_time_us":2910}]]}
I20260709 12:19:48.246179 12394 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 6d4ece942fd5414998215d26edf91001 reported cstate change: term changed from 44 to 46, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 46 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.261467 12379 connection.cc:466] server connection from 127.0.0.1:42850 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 233) could send its response
W20260709 12:19:48.320724 12536 consensus_peers.cc:597] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:48.320201 12388 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 45 to 48, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 48 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.324649 12388 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 55 to 59. New cstate: current_term: 59 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.330608 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:48.347046 12394 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 234) took 32170 ms (client timeout 15000 ms). Trace:
I20260709 12:19:48.348325 12388 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 41 to 43. New cstate: current_term: 43 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.364511 12394 rpcz_store.cc:269] 0709 12:19:16.177080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:16.177356 (+   276us) service_pool.cc:224] Handling call
0709 12:19:48.274478 (+32097122us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.274510 (+    32us) write_op.cc:454] Released schema lock
0709 12:19:48.346922 (+ 72412us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:48.264084 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:48.264698 (+   614us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:48.264737 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:19:48.264749 (+    12us) tablet.cc:662] Decoding operations
0709 12:19:48.265148 (+   399us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:48.265201 (+    53us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:48.265231 (+    30us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:19:48.265559 (+   328us) tablet.cc:701] Row locks acquired
0709 12:19:48.265572 (+    13us) write_op.cc:260] PREPARE: finished
0709 12:19:48.265736 (+   164us) write_op.cc:270] Start()
0709 12:19:48.265858 (+   122us) write_op.cc:276] Timestamp: P: 1783599588265687 usec, L: 0
0709 12:19:48.265878 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:19:48.266527 (+   649us) log.cc:844] Serialized 929 byte log entry
0709 12:19:48.268865 (+  2338us) op_driver.cc:464] REPLICATION: finished
0709 12:19:48.269339 (+   474us) write_op.cc:301] APPLY: starting
0709 12:19:48.269441 (+   102us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:48.270955 (+  1514us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:48.271056 (+   101us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:48.272414 (+  1358us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:48.272588 (+   174us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:19:48.272634 (+    46us) write_op.cc:312] APPLY: finished
0709 12:19:48.272856 (+   222us) log.cc:844] Serialized 33 byte log entry
0709 12:19:48.273019 (+   163us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.273115 (+    96us) write_op.cc:454] Released schema lock
0709 12:19:48.273160 (+    45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":9504,"reported_tablets":51,"spinlock_wait_cycles":8320,"child_traces":[["op",{"apply.queue_time_us":257,"apply.run_cpu_time_us":4156,"apply.run_wall_time_us":4155,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8352,"num_ops":2,"prepare.queue_time_us":122,"prepare.run_cpu_time_us":2741,"prepare.run_wall_time_us":2742,"replication_time_us":2860}]]}
I20260709 12:19:48.375002 12388 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 45 to 47. New cstate: current_term: 47 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.377460 12388 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: term changed from 46 to 51, leader changed from 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131) to 6d4ece942fd5414998215d26edf91001 (127.11.11.129). New cstate: current_term: 51 leader_uuid: "6d4ece942fd5414998215d26edf91001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.393769 12379 connection.cc:466] server connection from 127.0.0.1:42850 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42850 (request call id 234) could send its response
W20260709 12:19:48.499330 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:48.543504 12388 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 242) took 10167 ms. Trace:
I20260709 12:19:48.563210 12388 rpcz_store.cc:276] 0709 12:19:38.375789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:19:40.849057 (+2473268us) service_pool.cc:224] Handling call
0709 12:19:48.543428 (+7694371us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:19:48.414000 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:19:48.414682 (+   682us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:19:48.414723 (+    41us) write_op.cc:435] Acquired schema lock
0709 12:19:48.414744 (+    21us) tablet.cc:662] Decoding operations
0709 12:19:48.415479 (+   735us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:19:48.415540 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 12:19:48.415571 (+    31us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:19:48.416169 (+   598us) tablet.cc:701] Row locks acquired
0709 12:19:48.416188 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:19:48.416355 (+   167us) write_op.cc:270] Start()
0709 12:19:48.416493 (+   138us) write_op.cc:276] Timestamp: P: 1783599588416298 usec, L: 0
0709 12:19:48.416517 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:19:48.417504 (+   987us) log.cc:844] Serialized 2094 byte log entry
0709 12:19:48.419718 (+  2214us) op_driver.cc:464] REPLICATION: finished
0709 12:19:48.420202 (+   484us) write_op.cc:301] APPLY: starting
0709 12:19:48.420293 (+    91us) tablet.cc:1367] starting BulkCheckPresence
0709 12:19:48.423426 (+  3133us) tablet.cc:1370] finished BulkCheckPresence
0709 12:19:48.423545 (+   119us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:19:48.426921 (+  3376us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:19:48.427149 (+   228us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 12:19:48.427205 (+    56us) write_op.cc:312] APPLY: finished
0709 12:19:48.427506 (+   301us) log.cc:844] Serialized 57 byte log entry
0709 12:19:48.428447 (+   941us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.428581 (+   134us) write_op.cc:454] Released schema lock
0709 12:19:48.428634 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:19:48.429080 (+   446us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:19:48.429118 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22628,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":248,"apply.run_cpu_time_us":8109,"apply.run_wall_time_us":8821,"cfile_cache_hit":20,"cfile_cache_hit_bytes":20880,"num_ops":5,"prepare.queue_time_us":7753,"prepare.run_cpu_time_us":3798,"prepare.run_wall_time_us":3966,"replication_time_us":3065,"spinlock_wait_cycles":28032}]]}
I20260709 12:19:48.618371 12386 catalog_manager.cc:5697] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 51, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 51 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.620294 12386 catalog_manager.cc:5697] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 49 to 50, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 50 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.622963 12386 catalog_manager.cc:5697] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 46 to 47, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 47 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.624763 12386 catalog_manager.cc:5697] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 48 to 49, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 49 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:48.663483 12610 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:48.664757 12386 catalog_manager.cc:5697] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 49 to 52, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 52 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.666947 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:48.666390 12386 catalog_manager.cc:5697] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 49 to 53, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 53 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.668090 12386 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 55 to 58, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 58 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
W20260709 12:19:48.693305 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:48.711221 12386 catalog_manager.cc:5697] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 50 to 53, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 53 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.713562 12386 catalog_manager.cc:5697] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 43 to 46, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 46 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.716332 12386 catalog_manager.cc:5697] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 54 to 56. New cstate: current_term: 56 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.718923 12386 catalog_manager.cc:5697] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 51 to 52, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 52 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.725080 12393 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42864 (request call id 236) took 177403 ms (client timeout 15000 ms). Trace:
W20260709 12:19:48.747310 12393 rpcz_store.cc:269] 0709 12:16:51.321776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:16:59.373617 (+8051841us) service_pool.cc:224] Handling call
0709 12:19:48.724838 (+169351221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":309464,"reported_tablets":51}
W20260709 12:19:48.754446 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:48.747987 12386 catalog_manager.cc:5697] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 54 to 55, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 55 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.758912 12386 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 43 to 44, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 44 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.771003 12386 catalog_manager.cc:5697] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 57 to 58, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 58 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.774976 12386 catalog_manager.cc:5697] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 56 to 57, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 57 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.787382 12386 catalog_manager.cc:5697] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 49, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 49 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.788794 12386 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 50 to 53, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 53 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.802449 12386 catalog_manager.cc:5697] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 51 to 52, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 52 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.804255 12386 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 50, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 50 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.806007 12386 catalog_manager.cc:5697] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 44 to 47, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 47 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.807592 12386 catalog_manager.cc:5697] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 40 to 41, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 41 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.809096 12386 catalog_manager.cc:5697] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 53 to 54, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 54 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
I20260709 12:19:48.820081 12386 catalog_manager.cc:5697] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 53, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 53 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:48.833042 12386 catalog_manager.cc:5697] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 47 to 52, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 52 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.847630 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:48.835824 12386 catalog_manager.cc:5697] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 59 to 62, leader changed from 6d4ece942fd5414998215d26edf91001 (127.11.11.129) to 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130). New cstate: current_term: 62 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } }
W20260709 12:19:48.874500 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:48.952051 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:49.026789 12638 consensus_queue.cc:237] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 45.31, Last appended by leader: 29, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:49.030666 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:49.064879 13260 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 31. Preceding OpId from leader: term: 45 index: 32. (index mismatch)
I20260709 12:19:49.067202 12571 raft_consensus.cc:1275] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 45 index: 31. Preceding OpId from leader: term: 45 index: 32. (index mismatch)
I20260709 12:19:49.067056 30275 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20260709 12:19:49.074052 30275 consensus_queue.cc:1048] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
W20260709 12:19:49.065511 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:49.108378 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:49.125341 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:49.143689 30328 log.cc:927] Time spent T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260709 12:19:49.145956 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:49.146613 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:49.151527 30242 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Committing config change with OpId 45.32: config changed from index 31 to 32, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:49.160195 13260 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Committing config change with OpId 45.32: config changed from index 31 to 32, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:49.185287 11310 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 12518)
I20260709 12:19:49.190856 12393 catalog_manager.cc:5697] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 31 to 32, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New cstate: current_term: 45 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:49.208122 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:49.221101 12571 raft_consensus.cc:2955] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Committing config change with OpId 45.32: config changed from index 31 to 32, VOTER 6d4ece942fd5414998215d26edf91001 (127.11.11.129) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:49.232456 12380 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5f402dfd68ae4c8697765a76d9f1c4e6 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:19:49.241503 12536 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:49.281363 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:19:49.291698 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:49.367483 12380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111) [suppressed 458 similar messages]
W20260709 12:19:49.382735 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:49.385097 12536 consensus_peers.cc:597] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:49.414810 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:49.428572 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:49.435197 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:49.619182 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:49.677701 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:49.729379 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:49.753950 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:49.827987 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:49.832978 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:49.890116 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:49.930366 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:50.033154 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.052557 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.096848 12536 consensus_peers.cc:597] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.185961 11310 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:50.253003 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.264716 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:50.377617 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:50.417040 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:50.449848 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:50.514130 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:50.523900 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.555409 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.586246 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:50.607894 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:50.609316 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:50.640288 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a30ed0986fa641bd8f88912b4b9ee320: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:19:50.683094 12536 consensus_peers.cc:597] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:19:50.713567 12638 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 50.29, Last appended by leader: 28, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:19:50.738999 12611 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3da4ae1a8feb46d8a6bd6ec5522932d5. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:50.759128 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:50.787834 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:50.790756 12571 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 29. Preceding OpId from leader: term: 50 index: 30. (index mismatch)
I20260709 12:19:50.807742 30331 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.001s
W20260709 12:19:50.862516 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 54810.5us
W20260709 12:19:50.891791 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:50.920984 30342 raft_consensus.cc:2955] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Committing config change with OpId 50.30: config changed from index -1 to 30, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:50.923404 12572 raft_consensus.cc:2955] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Committing config change with OpId 50.30: config changed from index -1 to 30, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:19:50.950912 12536 consensus_peers.cc:597] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:50.952328 12386 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 30, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 50 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.045279 12572 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 53.33, Last appended by leader: 32, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
W20260709 12:19:51.053617 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:51.060616 12537 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a8c7204d6b44c9b3d6ea6332e9f824. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:51.065881 12638 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 53 index: 33. Preceding OpId from leader: term: 53 index: 34. (index mismatch)
I20260709 12:19:51.083595 30327 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
W20260709 12:19:51.103608 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:51.129527 30327 raft_consensus.cc:2955] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 LEADER]: Committing config change with OpId 53.34: config changed from index -1 to 34, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.169516 12638 raft_consensus.cc:2955] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Committing config change with OpId 53.34: config changed from index -1 to 34, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:19:51.186697 11310 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:51.205575 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:51.221424 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:51.222864 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:51.241163 12393 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index -1 to 34, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 53 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:19:51.246331 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:51.248032 30354 ts_tablet_manager.cc:933] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:51.264232 12638 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 58.33, Last appended by leader: 32, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:51.272159 30354 tablet_copy_client.cc:323] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
W20260709 12:19:51.277731 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:51.281667 12655 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:19:51.282531 12655 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 3da4ae1a8feb46d8a6bd6ec5522932d5 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-3da4ae1a8feb46d8a6bd6ec5522932d5
W20260709 12:19:51.298348 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 69894us
W20260709 12:19:51.304894 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:51.306807 12611 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 9485b0afaf144396adf854fff857414e (127.11.11.132:33793): Couldn't send request to peer 9485b0afaf144396adf854fff857414e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 979333b179ba4f49a65315a06d54e27e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:51.307201 12563 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 58 index: 33. Preceding OpId from leader: term: 58 index: 34. (index mismatch)
I20260709 12:19:51.305450 12655 tablet_copy_source_session.cc:215] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:19:51.309248 30341 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.001s
W20260709 12:19:51.316007 12536 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:51.318773 30354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da4ae1a8feb46d8a6bd6ec5522932d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:19:51.331439 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:51.343394 30331 raft_consensus.cc:2955] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Committing config change with OpId 58.34: config changed from index -1 to 34, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.365648 12563 raft_consensus.cc:2955] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Committing config change with OpId 58.34: config changed from index -1 to 34, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.386817 12393 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 34, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 58 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.400519 30354 tablet_copy_client.cc:806] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:19:51.401401 30354 tablet_copy_client.cc:670] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:19:51.406265 30354 tablet_copy_client.cc:538] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:19:51.421029 12638 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 44.26, Last appended by leader: 25, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:51.445529 30354 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:19:51.456875 13253 raft_consensus.cc:3060] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 34 LEARNER]: Advancing to term 44
I20260709 12:19:51.462880 12563 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 26. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
W20260709 12:19:51.464421 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:51.469383 30340 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:19:51.489512 13253 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 44 index: 27. (index mismatch)
I20260709 12:19:51.491631 30341 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:19:51.492767 30341 consensus_queue.cc:1243] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Peer 9485b0afaf144396adf854fff857414e log is divergent from this leader: its last log entry 34.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:19:51.519315 13253 pending_rounds.cc:85] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Aborting all ops after (but not including) 18
I20260709 12:19:51.519671 13253 pending_rounds.cc:107] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 32.19
I20260709 12:19:51.519989 13253 raft_consensus.cc:2917] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: Aborting config change with OpId 32.19: Aborted: Op aborted by new leader
I20260709 12:19:51.520408 13253 raft_consensus.cc:2889] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:51.520684 13253 pending_rounds.cc:107] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:19:51.520823 13253 raft_consensus.cc:2889] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 NON_PARTICIPANT]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:19:51.558540 30341 raft_consensus.cc:2955] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Committing config change with OpId 44.27: config changed from index -1 to 27, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.562273 12563 raft_consensus.cc:2955] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Committing config change with OpId 44.27: config changed from index -1 to 27, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
I20260709 12:19:51.620765 12393 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef reported cstate change: config changed from index -1 to 27, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New cstate: current_term: 44 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:19:51.627735 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:51.635396 30363 ts_tablet_manager.cc:933] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:19:51.669795 13253 raft_consensus.cc:2955] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: Committing config change with OpId 44.27: config changed from index -1 to 27, NON_VOTER 9485b0afaf144396adf854fff857414e (127.11.11.132) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } }
W20260709 12:19:51.682614 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:51.713841 30363 tablet_copy_client.cc:323] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.130:32927
W20260709 12:19:51.727675 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:51.735559 12581 tablet_copy_service.cc:140] P 0a29b5d9f9d44298b02ffcd460811eef: Received BeginTabletCopySession request for tablet a7a8c7204d6b44c9b3d6ea6332e9f824 from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:50142)
I20260709 12:19:51.736354 12581 tablet_copy_service.cc:161] P 0a29b5d9f9d44298b02ffcd460811eef: Beginning new tablet copy session on tablet a7a8c7204d6b44c9b3d6ea6332e9f824 from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:50142: session id = 9485b0afaf144396adf854fff857414e-a7a8c7204d6b44c9b3d6ea6332e9f824
I20260709 12:19:51.758919 30242 raft_consensus.cc:1064] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:51.767026 12581 tablet_copy_source_session.cc:215] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:19:51.764254 30242 consensus_queue.cc:237] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 44.27, Last appended by leader: 25, Current term: 44, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:51.782969 30363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a8c7204d6b44c9b3d6ea6332e9f824. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:19:51.782780 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:51.792531 13253 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 44 index: 28. (index mismatch)
I20260709 12:19:51.795111 30340 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:19:51.799692 12563 raft_consensus.cc:1275] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 44 index: 27. Preceding OpId from leader: term: 44 index: 28. (index mismatch)
I20260709 12:19:51.801297 30340 consensus_queue.cc:1048] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20260709 12:19:51.826790 30340 raft_consensus.cc:2955] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Committing config change with OpId 44.28: config changed from index 27 to 28, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:51.830821 13258 raft_consensus.cc:2955] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Committing config change with OpId 44.28: config changed from index 27 to 28, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:51.802213 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:51.879776 12536 consensus_peers.cc:597] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:51.878554 12393 catalog_manager.cc:5697] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 27 to 28, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 44 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:19:51.884438 12563 raft_consensus.cc:2955] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Committing config change with OpId 44.28: config changed from index 27 to 28, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:51.895545 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 142876us
W20260709 12:19:51.898837 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:19:51.919677 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:51.957756 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:51.973268 12611 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:33793, user_credentials={real_user=slave}} blocked reactor thread for 48694.1us
W20260709 12:19:51.975903 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:51.989521 30369 ts_tablet_manager.cc:933] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Initiating tablet copy from peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:19:51.995050 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:52.000797 30363 tablet_copy_client.cc:806] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:19:52.001534 30363 tablet_copy_client.cc:670] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
W20260709 12:19:52.001612 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:52.006405 30363 tablet_copy_client.cc:538] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:19:52.008572 30369 tablet_copy_client.cc:323] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.11.131:35471
I20260709 12:19:52.012297 12655 tablet_copy_service.cc:140] P 32889a507a6b4c74aec421ffbfd1e25d: Received BeginTabletCopySession request for tablet 979333b179ba4f49a65315a06d54e27e from peer 9485b0afaf144396adf854fff857414e ({username='slave'} at 127.0.0.1:35404)
I20260709 12:19:52.012930 12655 tablet_copy_service.cc:161] P 32889a507a6b4c74aec421ffbfd1e25d: Beginning new tablet copy session on tablet 979333b179ba4f49a65315a06d54e27e from peer 9485b0afaf144396adf854fff857414e at {username='slave'} at 127.0.0.1:35404: session id = 9485b0afaf144396adf854fff857414e-979333b179ba4f49a65315a06d54e27e
I20260709 12:19:52.024096 12655 tablet_copy_source_session.cc:215] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:19:52.029659 30369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 979333b179ba4f49a65315a06d54e27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:52.033041 30363 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
I20260709 12:19:52.066552 30369 tablet_copy_client.cc:806] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 0 data blocks...
I20260709 12:19:52.067746 30369 tablet_copy_client.cc:670] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: tablet copy: Starting download of 1 WAL segments...
W20260709 12:19:52.071779 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:52.102871 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:52.103672 30354 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=1 applied=30 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:19:52.111970 30369 tablet_copy_client.cc:538] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:19:52.139009 30354 tablet_bootstrap.cc:492] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:19:52.146657 30354 ts_tablet_manager.cc:1403] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.701s	user 0.083s	sys 0.022s
I20260709 12:19:52.149884 30369 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Bootstrap starting.
W20260709 12:19:52.161734 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:52.161913 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:19:52.166047 30354 raft_consensus.cc:359] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:52.193883 30354 raft_consensus.cc:740] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
W20260709 12:19:52.187403 11310 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 12518)
I20260709 12:19:52.194818 30354 consensus_queue.cc:260] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 50.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:52.196864 30354 ts_tablet_manager.cc:1434] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.047s	user 0.002s	sys 0.003s
I20260709 12:19:52.216763 13258 raft_consensus.cc:1217] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 LEARNER]: Deduplicated request from leader. Original: 50.29->[50.30-50.30]   Dedup: 50.30->[]
I20260709 12:19:52.256379 12655 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-3da4ae1a8feb46d8a6bd6ec5522932d5 received from {username='slave'} at 127.0.0.1:35404
I20260709 12:19:52.256986 12655 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-3da4ae1a8feb46d8a6bd6ec5522932d5 on tablet 3da4ae1a8feb46d8a6bd6ec5522932d5 with peer 9485b0afaf144396adf854fff857414e
W20260709 12:19:52.290458 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:52.398756 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:52.399320 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:52.425673 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:52.444418 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:52.497745 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:52.521976 30369 log.cc:927] Time spent T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.076s	user 0.004s	sys 0.000s
W20260709 12:19:52.563841 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:52.599061 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:52.664297 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:52.678382 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:52.762605 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:52.778749 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:52.821558 30369 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=2 applied=34 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:19:52.835433 30369 tablet_bootstrap.cc:492] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
W20260709 12:19:52.825976 12380 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 69845.3us
W20260709 12:19:52.840276 30363 log.cc:927] Time spent T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Append to log took a long time: real 0.064s	user 0.000s	sys 0.003s
W20260709 12:19:52.836015 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 73843.2us
I20260709 12:19:52.840679 30369 ts_tablet_manager.cc:1403] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.692s	user 0.084s	sys 0.036s
W20260709 12:19:52.872306 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 94409.7us
W20260709 12:19:52.884783 12536 consensus_peers.cc:597] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:52.915345 30363 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Bootstrap replayed 1/1 log segments. Stats: ops{read=39 overwritten=5 applied=34 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:19:52.916646 30363 tablet_bootstrap.cc:492] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Bootstrap complete.
I20260709 12:19:52.917550 30363 ts_tablet_manager.cc:1403] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Time spent bootstrapping tablet: real 0.885s	user 0.108s	sys 0.021s
W20260709 12:19:52.920521 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:52.934355 30363 raft_consensus.cc:359] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:52.937467 30363 raft_consensus.cc:740] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:19:52.939280 30382 raft_consensus.cc:1064] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:52.940502 30369 raft_consensus.cc:359] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:52.941488 30369 raft_consensus.cc:740] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 LEARNER]: Becoming Follower/Learner. State: Replica: 9485b0afaf144396adf854fff857414e, State: Initialized, Role: LEARNER
I20260709 12:19:52.943143 30369 consensus_queue.cc:260] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 58.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:52.944535 30382 consensus_queue.cc:237] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 50.30, Last appended by leader: 28, Current term: 50, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:52.949553 30363 consensus_queue.cc:260] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 53.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } }
I20260709 12:19:52.957192 30369 ts_tablet_manager.cc:1434] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.080s	user 0.008s	sys 0.000s
I20260709 12:19:52.968320 30363 ts_tablet_manager.cc:1434] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 12:19:52.973651 12581 tablet_copy_service.cc:342] P 0a29b5d9f9d44298b02ffcd460811eef: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-a7a8c7204d6b44c9b3d6ea6332e9f824 received from {username='slave'} at 127.0.0.1:50142
I20260709 12:19:52.974211 12581 tablet_copy_service.cc:434] P 0a29b5d9f9d44298b02ffcd460811eef: ending tablet copy session 9485b0afaf144396adf854fff857414e-a7a8c7204d6b44c9b3d6ea6332e9f824 on tablet a7a8c7204d6b44c9b3d6ea6332e9f824 with peer 9485b0afaf144396adf854fff857414e
I20260709 12:19:52.991513 12655 tablet_copy_service.cc:342] P 32889a507a6b4c74aec421ffbfd1e25d: Request end of tablet copy session 9485b0afaf144396adf854fff857414e-979333b179ba4f49a65315a06d54e27e received from {username='slave'} at 127.0.0.1:35404
I20260709 12:19:52.992053 12655 tablet_copy_service.cc:434] P 32889a507a6b4c74aec421ffbfd1e25d: ending tablet copy session 9485b0afaf144396adf854fff857414e-979333b179ba4f49a65315a06d54e27e on tablet 979333b179ba4f49a65315a06d54e27e with peer 9485b0afaf144396adf854fff857414e
W20260709 12:19:53.013146 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:53.016414 13262 raft_consensus.cc:1217] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 LEARNER]: Deduplicated request from leader. Original: 58.33->[58.34-58.34]   Dedup: 58.34->[]
I20260709 12:19:53.033404 12565 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:19:53.035389 30387 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:19:53.039705 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:53.040928 13262 raft_consensus.cc:1275] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 50 index: 30. Preceding OpId from leader: term: 50 index: 31. (index mismatch)
I20260709 12:19:53.047160 30387 consensus_queue.cc:1048] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:19:53.058035 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:53.063694 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:53.196691 11310 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 12518)
I20260709 12:19:53.218271 30342 raft_consensus.cc:2955] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Committing config change with OpId 50.31: config changed from index 30 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:53.241721 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:53.244189 13262 raft_consensus.cc:2955] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Committing config change with OpId 50.31: config changed from index 30 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.262554 13253 raft_consensus.cc:1217] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 LEARNER]: Deduplicated request from leader. Original: 53.33->[53.34-53.34]   Dedup: 53.34->[]
W20260709 12:19:53.339778 12536 consensus_peers.cc:597] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:19:53.369824 12564 raft_consensus.cc:2955] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Committing config change with OpId 50.31: config changed from index 30 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.380292 12393 catalog_manager.cc:5697] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 30 to 31, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 50 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:53.413062 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:53.437810 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:53.499804 12536 consensus_peers.cc:597] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:53.500698 30377 raft_consensus.cc:1064] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:53.508628 30377 consensus_queue.cc:237] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 58.34, Last appended by leader: 32, Current term: 58, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:53.531533 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:53.552973 13262 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 LEARNER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 58 index: 34. Preceding OpId from leader: term: 58 index: 35. (index mismatch)
I20260709 12:19:53.570597 30387 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.001s
W20260709 12:19:53.571545 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:53.580963 12563 raft_consensus.cc:1275] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 58 index: 34. Preceding OpId from leader: term: 58 index: 35. (index mismatch)
I20260709 12:19:53.583796 30340 consensus_queue.cc:1048] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.001s
W20260709 12:19:53.638054 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:53.641829 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:53.654263 30381 raft_consensus.cc:1064] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: attempting to promote NON_VOTER 9485b0afaf144396adf854fff857414e to VOTER
I20260709 12:19:53.664331 30381 consensus_queue.cc:237] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 53.34, Last appended by leader: 32, Current term: 53, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:19:53.675774 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:53.721745 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:53.730074 12536 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:53.732059 13253 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 LEARNER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 53 index: 34. Preceding OpId from leader: term: 53 index: 35. (index mismatch)
W20260709 12:19:53.733997 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:19:53.735642 30356 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.001s
W20260709 12:19:53.740998 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:19:53.741745 12638 raft_consensus.cc:1275] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Refusing update from remote peer 0a29b5d9f9d44298b02ffcd460811eef: Log matching property violated. Preceding OpId in replica: term: 53 index: 34. Preceding OpId from leader: term: 53 index: 35. (index mismatch)
I20260709 12:19:53.744824 30356 consensus_queue.cc:1048] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.001s
I20260709 12:19:53.759991 30342 raft_consensus.cc:2955] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Committing config change with OpId 58.35: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:53.770102 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:53.768424 13262 raft_consensus.cc:2955] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Committing config change with OpId 58.35: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.813913 12565 raft_consensus.cc:2955] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Committing config change with OpId 58.35: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.844592 12386 catalog_manager.cc:5697] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e reported cstate change: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 58 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.894469 30353 raft_consensus.cc:2955] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 LEADER]: Committing config change with OpId 53.35: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.905956 12637 raft_consensus.cc:2955] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Committing config change with OpId 53.35: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:53.939185 13253 raft_consensus.cc:2955] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Committing config change with OpId 53.35: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
I20260709 12:19:54.004379 12388 catalog_manager.cc:5697] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: config changed from index 34 to 35, 9485b0afaf144396adf854fff857414e (127.11.11.132) changed from NON_VOTER to VOTER. New cstate: current_term: 53 leader_uuid: "0a29b5d9f9d44298b02ffcd460811eef" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } } }
W20260709 12:19:54.040149 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:54.051232 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:19:54.197649 11310 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:54.224356 12536 consensus_peers.cc:597] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:54.309878 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:54.379921 13218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111) [suppressed 486 similar messages]
W20260709 12:19:54.469960 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 56511.3us
W20260709 12:19:54.477099 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:54.481994 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:54.566747 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:54.568293 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:54.676245 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:54.725046 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 191096us
W20260709 12:19:54.728783 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:54.729398 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:54.729921 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:54.779805 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:54.802577 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 71067.2us
W20260709 12:19:54.859305 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:54.862922 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:54.865062 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:54.854188 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:54.878629 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 49634.4us
W20260709 12:19:54.881072 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:54.887364 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:54.938776 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:55.090561 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:55.197507 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:55.198237 11310 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:55.301674 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:55.312816 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:55.342525 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:55.363377 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:55.372167 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:55.382261 12536 consensus_peers.cc:597] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:55.406482 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:55.663997 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:55.681892 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800056e20 after lost signal to thread 11313
W20260709 12:19:55.683686 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080038ce40 after lost signal to thread 12429
W20260709 12:19:55.691298 13283 debug-util.cc:398] Leaking SignalData structure 0x7b0800343aa0 after lost signal to thread 12596
W20260709 12:19:55.694002 13283 debug-util.cc:398] Leaking SignalData structure 0x7b080025a6a0 after lost signal to thread 12670
W20260709 12:19:55.695951 13283 debug-util.cc:398] Leaking SignalData structure 0x7b08001c1020 after lost signal to thread 13286
W20260709 12:19:55.801334 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:55.883446 12536 consensus_peers.cc:597] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:55.889246 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:55.890033 12536 consensus_peers.cc:597] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:56.097980 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 205233us
W20260709 12:19:56.101195 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 198294us
W20260709 12:19:56.149407 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 75519.8us
W20260709 12:19:56.168633 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:19:56.182497 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:56.186406 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.187186 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.204123 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:56.199167 11310 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:56.204939 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:56.234398 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:56.249420 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 142644us
W20260709 12:19:56.268783 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 169949us
W20260709 12:19:56.275619 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.338507 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:56.367993 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.382906 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:19:56.404673 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.586110 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:56.659008 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.710564 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:56.808445 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:19:56.853148 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:19:56.909564 12667 debug-util.cc:398] Leaking SignalData structure 0x7b080037b940 after lost signal to thread 11313
W20260709 12:19:56.911125 12667 debug-util.cc:398] Leaking SignalData structure 0x7b080034f680 after lost signal to thread 12429
W20260709 12:19:56.915879 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08001be520 after lost signal to thread 12596
W20260709 12:19:56.918097 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2ce0 after lost signal to thread 12670
W20260709 12:19:56.919299 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08002a0140 after lost signal to thread 13286
W20260709 12:19:56.922056 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:56.948364 12536 consensus_peers.cc:597] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:56.948909 12536 consensus_peers.cc:597] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:56.949319 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:56.995894 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.060281 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a30ed0986fa641bd8f88912b4b9ee320: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:19:57.103046 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:57.147533 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.200143 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.212819 11310 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 12518)
W20260709 12:19:57.304451 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:57.308358 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:57.416826 12536 consensus_peers.cc:597] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.497726 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:19:57.515000 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:57.579908 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:57.596448 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:57.624173 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.707105 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:57.844851 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.848663 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:57.865139 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:57.900425 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:19:57.906369 12536 consensus_peers.cc:597] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:57.945683 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:58.023995 12536 consensus_peers.cc:597] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:58.035595 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:58.047721 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
W20260709 12:19:58.066565 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:58.089344 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:58.187860 12536 consensus_peers.cc:597] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:58.194509 12536 consensus_peers.cc:597] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:58.261646 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:19:58.278404 12536 consensus_peers.cc:597] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:58.346179 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:58.358690 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:58.391520 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 43737.6us
W20260709 12:19:58.544450 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:19:58.550603 12536 consensus_peers.cc:597] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:58.552834 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:58.630851 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:19:58.642503 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:58.702108 12536 consensus_peers.cc:597] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:58.717031 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:19:58.749184 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:58.827749 12536 consensus_peers.cc:597] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:58.908557 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:19:58.918061 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
W20260709 12:19:58.920763 12536 consensus_peers.cc:597] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:19:58.932152 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:58.942209 13217 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:58.962301 12609 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:58.995949 13217 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.031010 12609 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.045287 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:59.030941 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:59.052752 13217 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.063571 12609 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.066392 12536 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 46255.8us
W20260709 12:19:59.066155 12609 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.068163 12609 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.143903 12609 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.194665 13217 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.213508 13217 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.213156 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:19:59.217065 13217 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.217944 13217 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.223057 13217 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.224056 13217 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.294358 12609 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.295143 12609 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.295711 12609 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.299772 13217 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.304674 13217 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.302183 12609 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.326370 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 98827.6us
W20260709 12:19:59.331969 12609 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.336467 12609 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.347530 12609 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.367421 12609 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.384502 12609 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.387050 13218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111) [suppressed 473 similar messages]
W20260709 12:19:59.389111 12536 consensus_peers.cc:597] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:19:59.398393 12609 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.406435 12609 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:59.406447 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:19:59.407872 13217 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.419139 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:19:59.419828 12609 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:19:59.428682 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:19:59.430171 11310 tablet_replica.cc:333] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.431514 11310 raft_consensus.cc:2243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.433441 11310 raft_consensus.cc:2272] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:59.435518 12609 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:59.436851 11310 tablet_replica.cc:333] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.438128 11310 raft_consensus.cc:2243] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.439247 11310 raft_consensus.cc:2272] T 7b3c8a7a06a04c369587d6ecb196bb2e P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.442734 11310 tablet_replica.cc:333] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.443691 11310 raft_consensus.cc:2243] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.450202 11310 raft_consensus.cc:2272] T dd6a5b4068ea44d2ac6f61924dd38c0d P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.453455 11310 tablet_replica.cc:333] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.454574 11310 raft_consensus.cc:2243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.463482 11310 raft_consensus.cc:2272] T 1f92af3771e3446f8c3549eb19f5b2c9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.466851 11310 tablet_replica.cc:333] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
W20260709 12:19:59.467422 12609 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:59.467828 11310 raft_consensus.cc:2243] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.468622 11310 raft_consensus.cc:2272] T e1f18763e7844e0994e2f11a6dd89290 P 0a29b5d9f9d44298b02ffcd460811eef [term 59 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.493880 11310 tablet_replica.cc:333] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.494915 11310 raft_consensus.cc:2243] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.495922 11310 raft_consensus.cc:2272] T 07fe6c41f52649a88327e52e0352b171 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.504307 11310 tablet_replica.cc:333] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.505446 11310 raft_consensus.cc:2243] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.506945 11310 raft_consensus.cc:2272] T 7816534d4a0942f0af150ea0a94392b9 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:59.514436 12536 consensus_peers.cc:597] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:19:59.516738 11310 tablet_replica.cc:333] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.517817 11310 raft_consensus.cc:2243] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.519270 11310 raft_consensus.cc:2272] T 92ef671b99df423ea5602f8b7ccff307 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.530403 11310 tablet_replica.cc:333] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
W20260709 12:19:59.530893 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:19:59.531322 11310 raft_consensus.cc:2243] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.541725 11310 raft_consensus.cc:2272] T 81a9a92616494f46bc97254b53185664 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.545153 11310 tablet_replica.cc:333] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.546136 11310 raft_consensus.cc:2243] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.547539 11310 raft_consensus.cc:2272] T 79b687d20a534e70865ef902fbd0d4a5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.558590 11310 tablet_replica.cc:333] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.559577 11310 raft_consensus.cc:2243] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.560885 11310 raft_consensus.cc:2272] T 27144fedf0014de09f007ac6cc3f7275 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:59.564765 12536 consensus_peers.cc:597] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:19:59.569505 11310 tablet_replica.cc:333] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.570590 11310 raft_consensus.cc:2243] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.571310 11310 raft_consensus.cc:2272] T a16c8cd5242e4ab6840a3a328938b5df P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.575615 11310 tablet_replica.cc:333] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.582379 11310 raft_consensus.cc:2243] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Raft consensus shutting down.
W20260709 12:19:59.584905 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:19:59.585709 11310 raft_consensus.cc:2272] T 1033139a62f44d368f8443d665a2cf22 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.588795 11310 tablet_replica.cc:333] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.594798 11310 raft_consensus.cc:2243] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.596117 11310 raft_consensus.cc:2272] T 43fecbd2edc14ba2ab065395e5d48d1d P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.600446 11310 tablet_replica.cc:333] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.606330 11310 raft_consensus.cc:2243] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.608493 11310 raft_consensus.cc:2272] T a30ed0986fa641bd8f88912b4b9ee320 P 0a29b5d9f9d44298b02ffcd460811eef [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.611716 11310 tablet_replica.cc:333] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.622397 11310 raft_consensus.cc:2243] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.624243 11310 raft_consensus.cc:2272] T f02aa0ab5af34d0e9ef3a4956449c624 P 0a29b5d9f9d44298b02ffcd460811eef [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.627264 11310 tablet_replica.cc:333] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.628190 11310 raft_consensus.cc:2243] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.629426 11310 raft_consensus.cc:2272] T c13f916d8b4d463182a1d1eb50150e31 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.642220 11310 tablet_replica.cc:333] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.644448 11310 raft_consensus.cc:2243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 62 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.646315 11310 raft_consensus.cc:2272] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 0a29b5d9f9d44298b02ffcd460811eef [term 62 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:59.679324 12536 consensus_peers.cc:597] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:19:59.683542 11310 tablet_replica.cc:333] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.691869 11310 raft_consensus.cc:2243] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.699991 11310 raft_consensus.cc:2272] T d03683a6ca224dc89740dc91f644b357 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.703624 11310 tablet_replica.cc:333] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.704591 11310 raft_consensus.cc:2243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.706076 11310 raft_consensus.cc:2272] T 93e029db6a8e4a7db72ad88760a6b7d3 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.724164 11310 tablet_replica.cc:333] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.725317 11310 raft_consensus.cc:2243] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.726367 11310 raft_consensus.cc:2272] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.730525 11310 tablet_replica.cc:333] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.731431 11310 raft_consensus.cc:2243] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.733870 11310 raft_consensus.cc:2272] T 4d3b77689c31445cbba405c2b232495e P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.753947 11310 tablet_replica.cc:333] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.765695 11310 raft_consensus.cc:2243] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 58 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.767232 11310 raft_consensus.cc:2272] T 256d044fd2e244d4be827104688867ee P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.850065 11310 tablet_replica.cc:333] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.851043 11310 raft_consensus.cc:2243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Raft consensus shutting down.
W20260709 12:19:59.852762 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 57962.6us
W20260709 12:19:59.868997 12536 consensus_peers.cc:597] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:19:59.874090 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:19:59.877616 11310 raft_consensus.cc:2272] T 8bd1dcd3e8a44986816c86ea6e580df5 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.881330 11310 tablet_replica.cc:333] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.882426 11310 raft_consensus.cc:2243] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.883847 11310 raft_consensus.cc:2272] T 979333b179ba4f49a65315a06d54e27e P 0a29b5d9f9d44298b02ffcd460811eef [term 58 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.898321 11310 tablet_replica.cc:333] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.899326 11310 raft_consensus.cc:2243] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:59.900120 11310 raft_consensus.cc:2272] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:59.912281 30446 raft_consensus.cc:493] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:19:59.912983 30446 raft_consensus.cc:515] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:19:59.938582 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "509f0430baf04e3393879d82fb9d10f9" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 52 candidate_status { last_received { term: 51 index: 31 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:19:59.940955 30446 leader_election.cc:290] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:19:59.943678 30446 raft_consensus.cc:493] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:59.944440 30446 raft_consensus.cc:515] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:59.946375 12536 leader_election.cc:336] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:19:59.947089 12536 leader_election.cc:304] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:19:59.948122 30444 raft_consensus.cc:2749] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:19:59.948727 30446 leader_election.cc:290] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:59.950589 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3066846eded04c878b1f03ecb43b1c79" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 53 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:19:59.954465 30446 raft_consensus.cc:493] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
W20260709 12:19:59.955157 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:19:59.956761 11310 tablet_replica.cc:333] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:19:59.958328 30444 raft_consensus.cc:493] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:19:59.964246 30444 raft_consensus.cc:515] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:19:59.964895 11310 raft_consensus.cc:2243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:19:59.966620 30444 leader_election.cc:290] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:59.955231 30446 raft_consensus.cc:515] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:19:59.979437 12536 leader_election.cc:336] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:19:59.983860 12536 leader_election.cc:304] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:19:59.985158 30446 leader_election.cc:290] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:19:59.992123 30444 raft_consensus.cc:2749] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:20:00.005148 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eadf951b63a43fd993cea2c96d5214f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.006937 30444 raft_consensus.cc:493] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.007603 30444 raft_consensus.cc:515] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.011682 30444 leader_election.cc:290] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.012663 11310 raft_consensus.cc:2272] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.016460 11310 tablet_replica.cc:333] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.018865 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fad7500a4eb45afa94184a247d796b8" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.027813 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7069a8d3584b466aab95934840523b8d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 48 candidate_status { last_received { term: 47 index: 24 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:20:00.024967 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:20:00.033391 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:00.034525 11310 raft_consensus.cc:2243] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.039614 11310 raft_consensus.cc:2272] T 5d0dce81bc984a0690d08335a4937b2c P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.048699 11310 tablet_replica.cc:333] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.053936 11310 raft_consensus.cc:2243] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:00.043292 12536 leader_election.cc:336] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.054656 12536 leader_election.cc:304] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.055531 30446 raft_consensus.cc:2749] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:20:00.059664 11310 raft_consensus.cc:2272] T 4dc881215d9f4498b162561e79143f2d P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.061421 30446 raft_consensus.cc:493] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:20:00.061988 30446 raft_consensus.cc:515] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:00.065644 30446 leader_election.cc:290] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:20:00.067426 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65037753870c4624995dde1173266f49" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 34 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:00.077665 30446 raft_consensus.cc:493] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.078352 30446 raft_consensus.cc:515] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.080783 30446 leader_election.cc:290] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:00.067409 12536 leader_election.cc:336] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.096364 12536 leader_election.cc:304] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.097337 30444 raft_consensus.cc:2749] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:20:00.099761 12536 leader_election.cc:336] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.100363 12536 leader_election.cc:304] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.101406 30444 raft_consensus.cc:2749] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Leader pre-election lost for term 47. Reason: could not achieve majority
I20260709 12:20:00.106280 11310 tablet_replica.cc:333] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.147253 30446 raft_consensus.cc:493] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.150110 30446 raft_consensus.cc:515] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.152431 30446 leader_election.cc:290] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.146214 30444 raft_consensus.cc:493] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election (detected failure of leader 9485b0afaf144396adf854fff857414e)
I20260709 12:20:00.146561 11310 raft_consensus.cc:2243] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Raft consensus shutting down.
I20260709 12:20:00.166152 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c126dc4ef9114297874fa66b1a0c56b0" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 58 candidate_status { last_received { term: 57 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.170802 11310 raft_consensus.cc:2272] T 44b2b570b6dd4e36bc98e56e097afd62 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.174886 11310 tablet_replica.cc:333] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.175961 11310 raft_consensus.cc:2243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.177390 11310 raft_consensus.cc:2272] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.169301 30444 raft_consensus.cc:515] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:00.179934 30444 leader_election.cc:290] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:20:00.181109 11310 tablet_replica.cc:333] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.190274 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b5f1e67a9754d3090a2610efe47c70f" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 26 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.192564 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59beb90f79b0450ea2fbb2b7e405261d" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 56 candidate_status { last_received { term: 55 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:00.192729 30444 raft_consensus.cc:493] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.193367 30444 raft_consensus.cc:515] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.194420 11310 raft_consensus.cc:2243] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.195632 11310 raft_consensus.cc:2272] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.196208 30444 leader_election.cc:290] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:00.187860 12536 leader_election.cc:336] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.197255 12536 leader_election.cc:304] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
W20260709 12:20:00.198359 12536 leader_election.cc:336] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.199558 11310 tablet_replica.cc:333] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.200754 30446 raft_consensus.cc:2749] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:20:00.200716 11310 raft_consensus.cc:2243] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:20:00.211382 11310 raft_consensus.cc:2272] T feae6096d5e346a3b548be8b5e7488d4 P 0a29b5d9f9d44298b02ffcd460811eef [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.212478 30446 raft_consensus.cc:493] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.213106 30446 raft_consensus.cc:515] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.214986 11310 tablet_replica.cc:333] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.215447 30446 leader_election.cc:290] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.216177 11310 raft_consensus.cc:2243] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.217355 11310 raft_consensus.cc:2272] T 6fad7500a4eb45afa94184a247d796b8 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:00.224500 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:00.226401 11310 tablet_replica.cc:333] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.227499 11310 raft_consensus.cc:2243] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 LEADER]: Raft consensus shutting down.
I20260709 12:20:00.228848 11310 raft_consensus.cc:2272] T e3648047ca0742a2bfaa45ef2af517e7 P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.230521 12536 leader_election.cc:304] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.236867 30446 raft_consensus.cc:2749] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
W20260709 12:20:00.239866 12536 leader_election.cc:336] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.240476 12536 leader_election.cc:304] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:00.241266 12536 raft_consensus.cc:2686] T 0b5f1e67a9754d3090a2610efe47c70f P 0a29b5d9f9d44298b02ffcd460811eef: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:20:00.244237 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f1b7080b6a4c329281b7c405f5c998" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 50 candidate_status { last_received { term: 49 index: 31 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.248497 11310 tablet_replica.cc:333] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.249715 11310 raft_consensus.cc:2243] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:00.250185 12536 leader_election.cc:336] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.250474 11310 raft_consensus.cc:2272] T 7069a8d3584b466aab95934840523b8d P 0a29b5d9f9d44298b02ffcd460811eef [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.250849 12536 leader_election.cc:304] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 56 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 6d4ece942fd5414998215d26edf91001, 9485b0afaf144396adf854fff857414e
I20260709 12:20:00.255975 30446 raft_consensus.cc:2749] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Leader pre-election lost for term 56. Reason: could not achieve majority
W20260709 12:20:00.257262 12536 leader_election.cc:336] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.259227 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05ad55cc3c04480abd0546c50226f9c1" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 54 candidate_status { last_received { term: 53 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.260825 12536 leader_election.cc:304] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 50 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:00.261693 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:20:00.269771 30446 raft_consensus.cc:2749] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Leader pre-election lost for term 50. Reason: could not achieve majority
I20260709 12:20:00.270108 30444 raft_consensus.cc:493] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.270807 30444 raft_consensus.cc:515] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.274418 11310 tablet_replica.cc:333] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.274806 30446 raft_consensus.cc:493] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:00.275578 11310 raft_consensus.cc:2243] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.275504 30446 raft_consensus.cc:515] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.276676 11310 raft_consensus.cc:2272] T 3066846eded04c878b1f03ecb43b1c79 P 0a29b5d9f9d44298b02ffcd460811eef [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.277983 30446 leader_election.cc:290] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.273576 30444 leader_election.cc:290] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.280543 11310 tablet_replica.cc:333] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.285161 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddf2b3bf3e6344499adb020ce08a3abf" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 51 candidate_status { last_received { term: 50 index: 28 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:20:00.285336 12536 leader_election.cc:336] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.289959 12536 leader_election.cc:304] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.291000 30444 raft_consensus.cc:2749] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:20:00.294955 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b39f4bd5c55347ada5f73766216828ca" candidate_uuid: "0a29b5d9f9d44298b02ffcd460811eef" candidate_term: 47 candidate_status { last_received { term: 46 index: 25 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:20:00.295637 12536 leader_election.cc:336] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.297814 11310 raft_consensus.cc:2243] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.298870 12536 leader_election.cc:304] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.299710 11310 raft_consensus.cc:2272] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.303099 30444 raft_consensus.cc:2749] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
I20260709 12:20:00.306481 11310 tablet_replica.cc:333] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.307628 11310 raft_consensus.cc:2243] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 LEADER]: Raft consensus shutting down.
W20260709 12:20:00.308880 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:20:00.314667 12536 leader_election.cc:336] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.315301 12536 leader_election.cc:304] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef [CANDIDATE]: Term 47 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a29b5d9f9d44298b02ffcd460811eef; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:00.316066 12536 raft_consensus.cc:2686] T b39f4bd5c55347ada5f73766216828ca P 0a29b5d9f9d44298b02ffcd460811eef: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:20:00.309454 11310 raft_consensus.cc:2272] T bb9177ec2fb345c4bff68c8c8a132af6 P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.326596 11310 tablet_replica.cc:333] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.327689 11310 raft_consensus.cc:2243] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.328603 11310 raft_consensus.cc:2272] T 3eadf951b63a43fd993cea2c96d5214f P 0a29b5d9f9d44298b02ffcd460811eef [term 46 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:00.334772 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:00.345402 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:20:00.348399 11310 tablet_replica.cc:333] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.356112 11310 raft_consensus.cc:2243] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.357280 11310 raft_consensus.cc:2272] T 6f701a23559945868ce6e3da5866f95c P 0a29b5d9f9d44298b02ffcd460811eef [term 56 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:00.356431 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:20:00.361095 11310 tablet_replica.cc:333] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.362138 11310 raft_consensus.cc:2243] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.362901 11310 raft_consensus.cc:2272] T c126dc4ef9114297874fa66b1a0c56b0 P 0a29b5d9f9d44298b02ffcd460811eef [term 57 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:00.368856 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:00.390559 11310 tablet_replica.cc:333] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.391695 11310 raft_consensus.cc:2243] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.392639 11310 raft_consensus.cc:2272] T 59beb90f79b0450ea2fbb2b7e405261d P 0a29b5d9f9d44298b02ffcd460811eef [term 55 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.396329 11310 tablet_replica.cc:333] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.397290 11310 raft_consensus.cc:2243] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.415103 11310 raft_consensus.cc:2272] T 05ad55cc3c04480abd0546c50226f9c1 P 0a29b5d9f9d44298b02ffcd460811eef [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.418972 11310 tablet_replica.cc:333] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.419975 11310 raft_consensus.cc:2243] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 LEADER]: Raft consensus shutting down.
I20260709 12:20:00.421589 11310 raft_consensus.cc:2272] T 54875fd261ee4c328e3d022d6a9418d8 P 0a29b5d9f9d44298b02ffcd460811eef [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.437430 11310 tablet_replica.cc:333] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.438345 11310 raft_consensus.cc:2243] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.439213 11310 raft_consensus.cc:2272] T ddf2b3bf3e6344499adb020ce08a3abf P 0a29b5d9f9d44298b02ffcd460811eef [term 50 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.458305 11310 tablet_replica.cc:333] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.459169 11310 raft_consensus.cc:2243] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 LEADER]: Raft consensus shutting down.
I20260709 12:20:00.460765 11310 raft_consensus.cc:2272] T a1d35de928f546349f88cbea7818041c P 0a29b5d9f9d44298b02ffcd460811eef [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.472401 11310 tablet_replica.cc:333] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.473258 11310 raft_consensus.cc:2243] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.473992 11310 raft_consensus.cc:2272] T 65037753870c4624995dde1173266f49 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.477121 11310 tablet_replica.cc:333] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.497498 11310 raft_consensus.cc:2243] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:00.516302 30433 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.517138 30433 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:00.518146 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:00.521934 30433 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:20:00.522248 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:00.522608 11310 raft_consensus.cc:2272] T 43f1b7080b6a4c329281b7c405f5c998 P 0a29b5d9f9d44298b02ffcd460811eef [term 49 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:00.524919 12609 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:00.525521 12609 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:20:00.526839 30433 raft_consensus.cc:2749] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:20:00.527231 11310 tablet_replica.cc:333] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.530182 11310 raft_consensus.cc:2243] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:20:00.531594 11310 raft_consensus.cc:2272] T 5d60fb5ad76741448a5376ca4d3f05df P 0a29b5d9f9d44298b02ffcd460811eef [term 45 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:00.544109 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:20:00.559345 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:20:00.569686 30433 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.570421 30433 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.573089 30433 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:00.577103 30451 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.577740 30451 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.580011 30451 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:00.581169 12609 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:00.585665 11310 tablet_replica.cc:333] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef: stopping tablet replica
I20260709 12:20:00.586562 11310 raft_consensus.cc:2243] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:00.601374 12610 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.602056 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:00.606554 12609 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:00.610893 11310 raft_consensus.cc:2272] T 509f0430baf04e3393879d82fb9d10f9 P 0a29b5d9f9d44298b02ffcd460811eef [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:00.623270 30443 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:20:00.626982 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.629748 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.638059 30433 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:20:00.689846 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:20:00.785081 30433 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.785770 30433 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.788178 30433 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:00.792826 12610 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.793285 30433 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:20:00.793903 12609 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:00.794461 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.794107 30433 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.795336 30443 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:20:00.799388 30433 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:00.803346 12609 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:20:00.803979 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.804428 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.805298 30433 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Leader pre-election lost for term 63. Reason: could not achieve majority
I20260709 12:20:00.859128 30448 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.860067 30448 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:00.871390 13217 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:00.875440 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.876266 12642 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 45.
I20260709 12:20:00.878664 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:20:00.880069 30449 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election won for term 46
I20260709 12:20:00.880749 30449 raft_consensus.cc:493] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.881152 30449 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:20:00.874387 30448 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.893575 30457 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.894232 30457 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:00.901008 30457 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.901706 30433 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.902279 30433 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.903908 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 53 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:00.904724 30433 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:00.908350 30443 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.923659 30443 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.926061 30443 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:00.926424 30448 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.929477 30448 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:00.914844 12610 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:00.932322 12609 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:00.934568 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:00.938812 30433 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:20:00.937894 30448 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:00.911697 13217 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:00.912833 30449 raft_consensus.cc:515] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:00.948750 12609 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:00.962389 13218 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:20:00.968633 30448 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:20:00.972493 30449 leader_election.cc:290] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:00.973489 30443 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:00.974112 30443 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:00.976796 30443 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:00.987859 13217 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:00.991446 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.991971 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:00.993199 13217 leader_election.cc:336] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:00.993973 30459 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:20:00.995738 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:00.997954 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:01.002895 12609 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.004540 30433 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:20:01.005007 30443 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.005564 30443 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:01.007867 30443 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:01.013536 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:01.016700 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd6a5b4068ea44d2ac6f61924dd38c0d" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 46 candidate_status { last_received { term: 45 index: 27 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
I20260709 12:20:01.017481 12642 raft_consensus.cc:3060] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Advancing to term 46
I20260709 12:20:01.032691 12642 raft_consensus.cc:2468] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 46.
W20260709 12:20:01.056244 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:01.057540 30443 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.058161 30443 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:01.062587 13218 leader_election.cc:304] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
I20260709 12:20:01.065503 30459 raft_consensus.cc:2804] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Leader election won for term 46
I20260709 12:20:01.066229 30459 raft_consensus.cc:697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:20:01.067309 30459 consensus_queue.cc:237] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 45.27, Last appended by leader: 27, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:01.075794 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:01.076650 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 52.
I20260709 12:20:01.073830 30443 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:20:01.075675 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.094903 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.079896 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:20:01.065371 30452 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.096511 30433 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:20:01.096891 30452 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:01.097313 30433 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election won for term 53
W20260709 12:20:01.085655 12609 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.099967 30433 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.103049 30433 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:20:01.109701 30452 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:01.110172 12609 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:01.105409 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.114475 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.115273 30456 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader pre-election lost for term 59. Reason: could not achieve majority
W20260709 12:20:01.115525 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:01.141300 12609 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.143993 30449 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.144896 30449 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:01.148061 30449 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:01.167166 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.167305 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:01.176614 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:01.178787 13217 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.179994 30443 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:20:01.183856 30433 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:01.188027 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:01.188725 13218 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.189364 13218 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.190992 30449 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:01.192647 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:20:01.201573 30443 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.202155 30443 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:01.204367 30443 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:20:01.206182 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 48 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:01.209076 12388 catalog_manager.cc:5697] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 45 to 46, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 46 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:20:01.214591 12609 leader_election.cc:336] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.215229 12609 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 9485b0afaf144396adf854fff857414e
I20260709 12:20:01.217206 30443 raft_consensus.cc:2749] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:20:01.218314 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13f916d8b4d463182a1d1eb50150e31" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 53 candidate_status { last_received { term: 52 index: 30 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e"
I20260709 12:20:01.218969 13258 raft_consensus.cc:3060] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Advancing to term 53
I20260709 12:20:01.226423 30433 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:20:01.230873 13258 raft_consensus.cc:2468] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:20:01.249600 12611 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
I20260709 12:20:01.251686 30443 raft_consensus.cc:2804] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader election won for term 53
I20260709 12:20:01.253587 30443 raft_consensus.cc:697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Becoming Leader. State: Replica: 32889a507a6b4c74aec421ffbfd1e25d, State: Running, Role: LEADER
I20260709 12:20:01.256270 30443 consensus_queue.cc:237] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 52.30, Last appended by leader: 30, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:01.269299 12609 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:01.283725 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:01.287468 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:01.301095 12386 catalog_manager.cc:5697] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d reported cstate change: term changed from 52 to 53, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131). New cstate: current_term: 53 leader_uuid: "32889a507a6b4c74aec421ffbfd1e25d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:01.340909 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:01.343954 30449 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.344640 30449 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:01.352495 30449 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:01.360682 13217 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:01.365520 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.366153 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.367444 30449 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
W20260709 12:20:01.379391 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:01.409855 30456 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.410634 30456 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:01.427714 12609 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.429406 30443 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.430174 30443 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:01.432461 30449 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.432904 30443 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:01.433256 30449 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:01.436379 13258 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:01.437294 13258 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
I20260709 12:20:01.439594 30456 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
W20260709 12:20:01.440414 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:01.442469 13217 leader_election.cc:336] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.444684 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.453418 30433 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:01.444657 12609 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.455103 30449 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:01.471405 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
W20260709 12:20:01.471668 13217 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.472296 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 47.
W20260709 12:20:01.476486 12609 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.497099 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef
W20260709 12:20:01.496627 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.498436 30449 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Leader pre-election won for term 48
I20260709 12:20:01.499002 30449 raft_consensus.cc:493] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Starting leader election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.499501 30449 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 47 FOLLOWER]: Advancing to term 48
W20260709 12:20:01.503585 13217 consensus_peers.cc:597] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:01.505692 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.506909 30433 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:20:01.514642 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:01.523708 12609 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.527107 30433 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.527705 30433 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:01.530043 30433 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:01.533218 30449 raft_consensus.cc:515] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:01.534931 12609 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.537338 13217 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.541016 12610 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:01.562121 12642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3648047ca0742a2bfaa45ef2af517e7" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 48 candidate_status { last_received { term: 47 index: 32 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d"
W20260709 12:20:01.562757 12609 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.563421 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.562889 12642 raft_consensus.cc:3060] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Advancing to term 48
I20260709 12:20:01.536100 30449 leader_election.cc:290] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Requested vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:01.565760 30433 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:20:01.570663 12609 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.574801 12642 raft_consensus.cc:2468] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 48.
W20260709 12:20:01.585824 13217 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.596690 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:01.598389 12609 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.603212 12609 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.603960 12609 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.606614 13218 leader_election.cc:304] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 
W20260709 12:20:01.608426 13217 leader_election.cc:336] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 48 election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.609889 30459 raft_consensus.cc:2804] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Leader election won for term 48
I20260709 12:20:01.610519 30459 raft_consensus.cc:697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Becoming Leader. State: Replica: 9485b0afaf144396adf854fff857414e, State: Running, Role: LEADER
I20260709 12:20:01.612572 30459 consensus_queue.cc:237] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 47.32, Last appended by leader: 32, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:01.614415 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:01.636076 13217 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.638878 12609 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.664620 12386 catalog_manager.cc:5697] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e reported cstate change: term changed from 47 to 48, leader changed from 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130) to 9485b0afaf144396adf854fff857414e (127.11.11.132). New cstate: current_term: 48 leader_uuid: "9485b0afaf144396adf854fff857414e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:01.669236 30433 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.670078 30433 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:01.676592 30433 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:01.680780 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:01.681399 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:01.683068 12609 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.683576 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.684412 30452 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
I20260709 12:20:01.687714 12642 raft_consensus.cc:1275] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 45 index: 27. Preceding OpId from leader: term: 46 index: 28. (index mismatch)
W20260709 12:20:01.691206 12609 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.692993 30459 consensus_queue.cc:1048] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.002s
W20260709 12:20:01.719987 13217 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.722543 12609 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.743052 12609 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.747367 12609 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.755887 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:01.766003 12609 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.781852 12609 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.790283 12609 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.798808 13217 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.809497 12609 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:01.810279 30448 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:01.811113 30448 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:01.814796 12609 consensus_peers.cc:597] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:01.813892 30448 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:01.815176 13217 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.827262 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:01.828583 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:01.831736 13217 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:01.832357 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:01.840173 30459 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:20:01.841954 13217 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.859417 11310 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 12593)
I20260709 12:20:01.860752 13258 raft_consensus.cc:1275] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Refusing update from remote peer 32889a507a6b4c74aec421ffbfd1e25d: Log matching property violated. Preceding OpId in replica: term: 52 index: 30. Preceding OpId from leader: term: 53 index: 31. (index mismatch)
I20260709 12:20:01.863361 30452 consensus_queue.cc:1048] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.001s
W20260709 12:20:01.878221 12609 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.916613 13217 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:01.964525 13217 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:02.022830 13217 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:02.031355 13217 consensus_peers.cc:597] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:02.060253 13217 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:02.076359 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:02.078715 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:02.111826 12609 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:02.116699 12642 raft_consensus.cc:1275] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Refusing update from remote peer 9485b0afaf144396adf854fff857414e: Log matching property violated. Preceding OpId in replica: term: 47 index: 32. Preceding OpId from leader: term: 48 index: 33. (index mismatch)
I20260709 12:20:02.123077 30449 consensus_queue.cc:1048] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.001s
W20260709 12:20:02.175751 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:02.188934 12609 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:02.327369 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:02.347069 12609 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:02.563292 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:20:02.572014 30443 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:02.572752 30443 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:02.575253 30443 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:02.579996 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:02.586585 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:02.617940 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:02.631594 12609 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:02.632297 12609 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:02.639266 30443 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:20:02.665371 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:20:02.787451 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:02.800228 30443 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:02.800944 30443 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:02.803448 30443 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:02.849974 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44398.7us
W20260709 12:20:02.860141 11310 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 12593)
I20260709 12:20:02.884753 30452 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:02.895115 30452 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:02.897864 30452 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:02.898466 30452 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:02.899281 30452 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:02.902750 30452 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:03.182118 30452 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.182717 30452 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.225893 30475 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.226562 30475 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.230576 30443 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.231230 30443 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.233175 30443 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:03.235100 30475 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:03.260442 30484 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.263304 30484 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:03.267778 30484 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:03.276701 30452 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:03.277561 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 459458us
W20260709 12:20:03.332276 12609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 85862.7us
W20260709 12:20:03.339655 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 59452.8us
I20260709 12:20:03.354163 30449 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.356037 30449 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:03.405241 12609 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.406800 12609 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.407442 12609 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:03.410000 30449 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:03.420538 30452 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.421744 30452 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.425197 30452 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:03.431854 12610 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 44665us
W20260709 12:20:03.433785 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:03.457616 13217 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:03.467849 30452 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.469178 30452 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:03.473225 30452 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 9485b0afaf144396adf854fff857414e (127.11.11.132:33793)
I20260709 12:20:03.479821 13254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "32889a507a6b4c74aec421ffbfd1e25d" candidate_term: 54 candidate_status { last_received { term: 53 index: 35 } } ignore_live_leader: false dest_uuid: "9485b0afaf144396adf854fff857414e" is_pre_election: true
I20260709 12:20:03.481305 13254 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32889a507a6b4c74aec421ffbfd1e25d in term 53.
W20260709 12:20:03.488689 12609 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.489655 12609 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.490185 12610 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.490844 12610 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.492280 30474 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:20:03.493343 30452 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.494275 30452 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.496968 30452 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:03.491333 12609 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.503931 12609 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.498404 13217 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.508682 12609 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.509615 12609 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.516654 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:20:03.527348 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:03.535611 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.537863 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.546221 30490 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.555169 30490 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:03.559667 30490 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:03.563757 30490 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:20:03.563843 13217 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.596246 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:03.618047 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:03.618810 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.622426 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.624642 30490 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
W20260709 12:20:03.633939 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.665843 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.670367 30443 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader pre-election lost for term 59. Reason: could not achieve majority
W20260709 12:20:03.664672 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.681313 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:03.670640 12610 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.682683 30490 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:20:03.686569 12610 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.688021 30443 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:20:03.692440 30443 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.693318 30443 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.696281 30443 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:03.697680 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:20:03.702453 12609 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:03.714762 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:03.729454 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:03.751838 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.766006 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.768122 30443 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Leader pre-election lost for term 63. Reason: could not achieve majority
W20260709 12:20:03.768597 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:03.817999 30452 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.818897 30452 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:03.821894 30452 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:03.813699 12610 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.830365 12610 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.831435 30433 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:20:03.831687 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.832198 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.833096 30433 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:20:03.834021 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:03.841447 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08003a7180 after lost signal to thread 11313
W20260709 12:20:03.849704 12667 debug-util.cc:398] Leaking SignalData structure 0x7b0800118be0 after lost signal to thread 12429
W20260709 12:20:03.852309 12667 debug-util.cc:398] Leaking SignalData structure 0x7b08003011a0 after lost signal to thread 12670
W20260709 12:20:03.854727 12667 debug-util.cc:398] Leaking SignalData structure 0x7b0800343ae0 after lost signal to thread 13286
I20260709 12:20:03.857957 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:03.861115 11310 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 12593)
I20260709 12:20:03.862437 30433 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:20:03.863065 12610 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.863615 12610 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d, 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.864400 30433 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:03.865034 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.870456 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.878455 30452 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:20:03.902602 30475 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:03.903460 30475 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:20:03.914436 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:03.919626 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.920312 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.925978 30475 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:03.922843 12609 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:03.934976 30475 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:20:03.958508 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:03.963968 12610 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:03.971094 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:03.977835 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:03.979089 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:20:03.979415 30443 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:20:03.993896 12609 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:03.994534 12609 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:03.995846 30443 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:20:04.010993 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:04.003753 12609 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.093096 13217 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:20:04.310288 30497 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:04.311267 30497 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:04.314277 30497 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:04.703035 12609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111) [suppressed 701 similar messages]
W20260709 12:20:04.727563 12609 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:04.728087 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:38859, user_credentials={real_user=slave}} blocked reactor thread for 369941us
W20260709 12:20:04.728977 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:04.741976 12609 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.745981 12609 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.746867 12609 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.748176 12609 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.745119 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:04.752621 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:20:04.754652 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:04.761128 13217 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.764447 13217 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.765194 13217 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.765750 13217 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.766420 13217 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.767329 13217 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.764453 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:04.768497 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:04.771469 30443 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:20:04.776441 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:04.777235 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:04.763312 12609 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.778600 12609 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.783242 12609 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.783970 12609 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.790544 13217 consensus_peers.cc:597] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:04.791249 12609 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.802141 12609 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.838156 12609 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.839083 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:04.853479 13217 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.856577 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:04.862044 11310 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 12593)
W20260709 12:20:04.884743 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:20:04.969550 12609 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.983376 13217 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:04.985687 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:05.023054 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:05.070178 12609 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.121966 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:05.138361 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08003b0aa0 after lost signal to thread 11313
W20260709 12:20:05.140048 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08002a8740 after lost signal to thread 12429
W20260709 12:20:05.141957 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08003f8d00 after lost signal to thread 12670
W20260709 12:20:05.143661 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08002367c0 after lost signal to thread 13286
W20260709 12:20:05.165187 13217 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.212715 12609 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.231555 13217 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.245056 12609 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.276296 12609 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.291126 12609 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.293015 13217 consensus_peers.cc:597] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:05.309033 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:05.335420 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a30ed0986fa641bd8f88912b4b9ee320: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:05.338788 13217 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.361768 12609 consensus_peers.cc:597] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:05.380589 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:05.381109 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:20:05.521034 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:20:05.543530 30495 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:05.544255 30495 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:05.546805 30495 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:05.563331 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:05.567771 13217 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:05.568341 13217 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:05.569284 30494 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:20:05.598155 30495 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:05.599155 30495 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:05.609001 13218 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:05.616495 13217 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:05.617249 13217 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:05.618698 30495 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:05.620939 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:05.621869 12639 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
I20260709 12:20:05.623615 30494 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:05.650450 12609 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.671903 12609 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:05.675784 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:05.699947 30475 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:05.700523 30475 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:20:05.708729 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:05.716706 30475 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:05.733387 12609 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:05.734725 12609 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:05.734196 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:20:05.739717 30475 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:20:05.742527 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:05.774580 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:05.856225 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:05.954917 30499 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:05.955755 30499 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:05.961388 30499 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:05.964022 12609 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:05.971076 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:05.971695 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:05.972627 30499 raft_consensus.cc:2749] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
I20260709 12:20:06.025866 30499 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.026556 30499 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.029098 30499 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.030966 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:20:06.039631 12609 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:06.040619 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.041059 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.041967 30499 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
I20260709 12:20:06.096164 30499 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.097031 30499 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.105069 30475 raft_consensus.cc:493] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.105815 30475 raft_consensus.cc:515] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.106624 30499 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:06.108249 30475 leader_election.cc:290] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.125057 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:06.134452 12610 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:06.151122 12609 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:06.151746 12609 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:06.152995 12609 leader_election.cc:336] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:06.154412 12609 leader_election.cc:304] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:06.153934 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:06.157212 30499 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
I20260709 12:20:06.159776 30499 raft_consensus.cc:2749] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:20:06.184398 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:20:06.233584 30499 raft_consensus.cc:493] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.234431 30499 raft_consensus.cc:515] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.236892 30499 leader_election.cc:290] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.244041 12609 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:06.252471 12610 leader_election.cc:336] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.253002 12610 leader_election.cc:304] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.254228 30499 raft_consensus.cc:2749] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Leader pre-election lost for term 59. Reason: could not achieve majority
W20260709 12:20:06.272513 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:20:06.306998 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:20:06.308761 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:06.364996 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:20:06.372231 30495 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.373136 30495 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:06.381578 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.383472 30495 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.392588 13217 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:06.393142 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.394268 30495 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
W20260709 12:20:06.397007 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:06.480811 13217 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.497750 12610 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:06.537906 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.547753 30495 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.548432 30495 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:06.561863 13217 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:06.562796 30495 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.567340 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.568037 13218 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.569518 30495 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:20:06.581480 12610 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:20:06.591976 30483 raft_consensus.cc:493] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.592814 30483 raft_consensus.cc:515] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.595474 30483 leader_election.cc:290] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.600970 12610 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:06.616977 12609 leader_election.cc:336] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:06.617589 12609 leader_election.cc:304] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.618536 30483 raft_consensus.cc:2749] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:20:06.670956 12610 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:20:06.679209 30483 raft_consensus.cc:493] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.679926 30483 raft_consensus.cc:515] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.682484 30483 leader_election.cc:290] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.692260 12609 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:06.693650 12610 leader_election.cc:336] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.694228 12610 leader_election.cc:304] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 63 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.695081 30483 raft_consensus.cc:2749] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Leader pre-election lost for term 63. Reason: could not achieve majority
W20260709 12:20:06.831907 13217 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.840678 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:06.849613 12609 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.849761 13217 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:20:06.851575 30505 raft_consensus.cc:493] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.857980 30505 raft_consensus.cc:515] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:06.860484 30505 leader_election.cc:290] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.862617 13217 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.885609 12609 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:06.887809 30505 raft_consensus.cc:493] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:06.888567 30505 raft_consensus.cc:515] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
W20260709 12:20:06.889009 12610 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:20:06.891520 30505 leader_election.cc:290] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:06.898715 12610 leader_election.cc:336] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.899407 12610 leader_election.cc:304] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.900723 30505 raft_consensus.cc:2749] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:20:06.910671 12609 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:06.921716 12610 leader_election.cc:336] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:06.922267 12610 leader_election.cc:304] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:06.933914 30505 raft_consensus.cc:2749] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
W20260709 12:20:06.942124 12609 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.949959 12610 consensus_peers.cc:597] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:06.954263 12609 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.972600 13217 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:06.976644 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:06.986213 12609 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.093246 13217 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.100703 12610 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:20:07.101977 12609 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.112918 12609 consensus_peers.cc:597] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.127717 12610 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:07.127772 12609 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.132164 12609 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:20:07.138664 30505 raft_consensus.cc:493] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:07.139526 30505 raft_consensus.cc:515] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:07.147982 30505 leader_election.cc:290] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:07.151062 12610 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:07.159572 12609 leader_election.cc:336] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:07.160233 12609 leader_election.cc:304] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:07.161193 30499 raft_consensus.cc:2749] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
W20260709 12:20:07.179283 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:07.182313 12609 consensus_peers.cc:597] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.187469 13217 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.293133 12609 consensus_peers.cc:597] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.298666 12609 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.299719 12610 consensus_peers.cc:597] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:20:07.314881 30495 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:07.315653 30495 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:07.324604 13217 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:07.325856 30495 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
I20260709 12:20:07.329761 12639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a8c7204d6b44c9b3d6ea6332e9f824" candidate_uuid: "9485b0afaf144396adf854fff857414e" candidate_term: 54 candidate_status { last_received { term: 53 index: 35 } } ignore_live_leader: false dest_uuid: "32889a507a6b4c74aec421ffbfd1e25d" is_pre_election: true
I20260709 12:20:07.330648 12639 raft_consensus.cc:2468] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9485b0afaf144396adf854fff857414e in term 53.
W20260709 12:20:07.328958 13218 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:07.335047 13218 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:07.349740 13217 consensus_peers.cc:597] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:20:07.350812 30495 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:07.401903 12609 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.408958 12610 consensus_peers.cc:597] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:20:07.449931 13217 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.452941 12610 consensus_peers.cc:597] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:07.460922 12609 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.490103 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:20:07.517138 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:20:07.530249 12610 consensus_peers.cc:597] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:07.538153 12609 consensus_peers.cc:597] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.574306 13217 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.610347 13217 consensus_peers.cc:597] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:07.626732 13217 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.649217 12609 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:20:07.708120 30483 raft_consensus.cc:493] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:07.708809 30483 raft_consensus.cc:515] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:07.711675 30483 leader_election.cc:290] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:07.719998 13217 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.727324 12609 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:07.744491 30483 raft_consensus.cc:493] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:07.745177 30483 raft_consensus.cc:515] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:07.747605 30483 leader_election.cc:290] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:07.750905 12610 leader_election.cc:336] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:07.757473 12610 leader_election.cc:304] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 48 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:07.758519 30483 raft_consensus.cc:2749] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Leader pre-election lost for term 48. Reason: could not achieve majority
W20260709 12:20:07.780787 12609 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:07.795131 12610 leader_election.cc:336] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:07.800644 12610 leader_election.cc:304] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 57 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:07.814625 30483 raft_consensus.cc:2749] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Leader pre-election lost for term 57. Reason: could not achieve majority
W20260709 12:20:07.813575 12609 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.831969 12609 consensus_peers.cc:597] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.892510 12609 consensus_peers.cc:597] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:20:07.952600 12610 consensus_peers.cc:597] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:07.958560 12609 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.975551 12609 consensus_peers.cc:597] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:07.976855 12610 consensus_peers.cc:597] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:20:07.980273 30483 raft_consensus.cc:493] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:07.981096 30483 raft_consensus.cc:515] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:07.983646 30483 leader_election.cc:290] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:08.013504 12609 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:08.016041 12610 leader_election.cc:336] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:08.018610 12610 leader_election.cc:304] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 49 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:08.020200 30483 raft_consensus.cc:2749] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Leader pre-election lost for term 49. Reason: could not achieve majority
W20260709 12:20:08.025290 12610 consensus_peers.cc:597] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:08.040938 13217 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:08.304240 12609 consensus_peers.cc:597] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:08.342275 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:20:08.382957 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
W20260709 12:20:08.385442 13218 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:20:08.392911 12610 consensus_peers.cc:597] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:08.415591 12610 consensus_peers.cc:597] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:08.540902 12610 consensus_peers.cc:597] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:08.567090 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:20:08.580025 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
W20260709 12:20:08.613555 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:20:08.634124 13218 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:08.658342 12610 consensus_peers.cc:597] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:20:08.690982 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:08.692340 11310 tablet_replica.cc:333] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.693884 11310 raft_consensus.cc:2243] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.694795 11310 raft_consensus.cc:2272] T 7b3c8a7a06a04c369587d6ecb196bb2e P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.697340 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.698964 11310 tablet_replica.cc:333] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.700037 11310 raft_consensus.cc:2243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.701434 11310 raft_consensus.cc:2272] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.705124 11310 tablet_replica.cc:333] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.706240 11310 raft_consensus.cc:2243] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.707404 11310 raft_consensus.cc:2272] T 1f92af3771e3446f8c3549eb19f5b2c9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.710829 11310 tablet_replica.cc:333] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
W20260709 12:20:08.711990 12610 consensus_peers.cc:597] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:20:08.714735 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.716715 11310 raft_consensus.cc:2243] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.718420 11310 raft_consensus.cc:2272] T dd6a5b4068ea44d2ac6f61924dd38c0d P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.721683 30514 raft_consensus.cc:493] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:08.722437 30514 raft_consensus.cc:515] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:08.723251 11310 tablet_replica.cc:333] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.724511 11310 raft_consensus.cc:2243] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.725252 30514 leader_election.cc:290] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:08.725906 11310 raft_consensus.cc:2272] T e1f18763e7844e0994e2f11a6dd89290 P 32889a507a6b4c74aec421ffbfd1e25d [term 59 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.730264 11310 tablet_replica.cc:333] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.731359 11310 raft_consensus.cc:2243] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.733451 11310 raft_consensus.cc:2272] T a16c8cd5242e4ab6840a3a328938b5df P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.736106 12609 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:08.737021 11310 tablet_replica.cc:333] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.741163 11310 raft_consensus.cc:2243] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.742262 11310 raft_consensus.cc:2272] T 4dc881215d9f4498b162561e79143f2d P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.747299 11310 tablet_replica.cc:333] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.748268 11310 raft_consensus.cc:2243] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.749137 11310 raft_consensus.cc:2272] T 5d0dce81bc984a0690d08335a4937b2c P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.751494 13218 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:20:08.752341 11310 tablet_replica.cc:333] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
W20260709 12:20:08.753100 12610 leader_election.cc:336] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:08.753317 11310 raft_consensus.cc:2243] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.755918 12610 leader_election.cc:304] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 53 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:08.756924 11310 raft_consensus.cc:2272] T 9bbf126b18234b78b4f36bf9708ad96b P 32889a507a6b4c74aec421ffbfd1e25d [term 61 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.757222 30514 raft_consensus.cc:2749] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Leader pre-election lost for term 53. Reason: could not achieve majority
I20260709 12:20:08.762411 11310 tablet_replica.cc:333] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.763455 11310 raft_consensus.cc:2243] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.765293 11310 raft_consensus.cc:2272] T a3a0781287514c758f107d79c1f96be4 P 32889a507a6b4c74aec421ffbfd1e25d [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.771936 11310 tablet_replica.cc:333] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.772959 11310 raft_consensus.cc:2243] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.774448 11310 raft_consensus.cc:2272] T 92ef671b99df423ea5602f8b7ccff307 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.775746 12610 consensus_peers.cc:597] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:20:08.778502 11310 tablet_replica.cc:333] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.779628 11310 raft_consensus.cc:2243] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.781185 11310 raft_consensus.cc:2272] T 7816534d4a0942f0af150ea0a94392b9 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.784724 11310 tablet_replica.cc:333] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.786361 11310 raft_consensus.cc:2243] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.787199 11310 raft_consensus.cc:2272] T a30ed0986fa641bd8f88912b4b9ee320 P 32889a507a6b4c74aec421ffbfd1e25d [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.791093 11310 tablet_replica.cc:333] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.793323 11310 raft_consensus.cc:2243] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.794848 11310 raft_consensus.cc:2272] T f02aa0ab5af34d0e9ef3a4956449c624 P 32889a507a6b4c74aec421ffbfd1e25d [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.799261 11310 tablet_replica.cc:333] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.800143 11310 raft_consensus.cc:2243] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.801851 11310 raft_consensus.cc:2272] T 81a9a92616494f46bc97254b53185664 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.806839 11310 tablet_replica.cc:333] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.807926 11310 raft_consensus.cc:2243] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.808893 11310 raft_consensus.cc:2272] T 1033139a62f44d368f8443d665a2cf22 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.813719 11310 tablet_replica.cc:333] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.814585 11310 raft_consensus.cc:2243] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.818112 11310 raft_consensus.cc:2272] T d03683a6ca224dc89740dc91f644b357 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.821717 11310 tablet_replica.cc:333] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.822827 30483 raft_consensus.cc:493] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:08.823423 30483 raft_consensus.cc:515] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } }
I20260709 12:20:08.825739 30483 leader_election.cc:290] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:08.826380 11310 raft_consensus.cc:2243] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.827291 11310 raft_consensus.cc:2272] T 93e029db6a8e4a7db72ad88760a6b7d3 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.831144 11310 tablet_replica.cc:333] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.832000 11310 raft_consensus.cc:2243] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.833854 11310 raft_consensus.cc:2272] T c13f916d8b4d463182a1d1eb50150e31 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.835942 13218 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:08.837814 12609 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:08.843389 11310 tablet_replica.cc:333] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.844440 11310 raft_consensus.cc:2243] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.846027 11310 raft_consensus.cc:2272] T 02f3cd3c4cc54eb9adb6f7c50cea457e P 32889a507a6b4c74aec421ffbfd1e25d [term 62 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.849078 11310 tablet_replica.cc:333] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.850042 11310 raft_consensus.cc:2243] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.850911 11310 raft_consensus.cc:2272] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.857257 11310 tablet_replica.cc:333] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.858220 11310 raft_consensus.cc:2243] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.859553 11310 raft_consensus.cc:2272] T 0c56dde469374f4da7440f134140670c P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.865134 11310 tablet_replica.cc:333] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.866040 11310 raft_consensus.cc:2243] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.866693 11310 raft_consensus.cc:2272] T 256d044fd2e244d4be827104688867ee P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.870004 12610 leader_election.cc:336] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:08.871172 11310 tablet_replica.cc:333] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
W20260709 12:20:08.872022 13218 consensus_peers.cc:597] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:08.873420 13218 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.873823 11310 raft_consensus.cc:2243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:08.874343 13218 consensus_peers.cc:597] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.875375 11310 raft_consensus.cc:2272] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.877041 12610 leader_election.cc:304] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32889a507a6b4c74aec421ffbfd1e25d; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:08.877815 30512 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:20:08.878261 12610 raft_consensus.cc:2686] T 8bd1dcd3e8a44986816c86ea6e580df5 P 32889a507a6b4c74aec421ffbfd1e25d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:20:08.878432 30512 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:08.883522 11310 tablet_replica.cc:333] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.883424 30512 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
I20260709 12:20:08.885097 11310 raft_consensus.cc:2243] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 LEADER]: Raft consensus shutting down.
W20260709 12:20:08.887353 13217 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:08.887698 11310 raft_consensus.cc:2272] T 979333b179ba4f49a65315a06d54e27e P 32889a507a6b4c74aec421ffbfd1e25d [term 58 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.890977 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:08.891692 13218 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:08.892590 30511 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:20:08.891111 11310 tablet_replica.cc:333] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.894770 11310 raft_consensus.cc:2243] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.896548 11310 raft_consensus.cc:2272] T 0b62b58d64bf4dc2850a3ea4a3cb128a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.900899 12610 consensus_peers.cc:597] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:20:08.902534 13218 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:08.903441 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.904783 11310 tablet_replica.cc:333] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.905933 11310 raft_consensus.cc:2243] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.907701 11310 raft_consensus.cc:2272] T 40d0e235c01b4b7f89948ab708403a01 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.911110 11310 tablet_replica.cc:333] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.912031 11310 raft_consensus.cc:2243] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.913142 11310 raft_consensus.cc:2272] T 44b2b570b6dd4e36bc98e56e097afd62 P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.916322 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.925606 11310 tablet_replica.cc:333] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.926841 11310 raft_consensus.cc:2243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.929438 11310 raft_consensus.cc:2272] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.932829 11310 tablet_replica.cc:333] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.934867 11310 raft_consensus.cc:2243] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.936846 11310 raft_consensus.cc:2272] T 0b5f1e67a9754d3090a2610efe47c70f P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.943426 11310 tablet_replica.cc:333] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
W20260709 12:20:08.958706 13218 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.944554 11310 raft_consensus.cc:2243] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.982368 11310 raft_consensus.cc:2272] T feae6096d5e346a3b548be8b5e7488d4 P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.985585 11310 tablet_replica.cc:333] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.986584 11310 raft_consensus.cc:2243] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.987917 11310 raft_consensus.cc:2272] T b39f4bd5c55347ada5f73766216828ca P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:08.988919 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:08.991415 11310 tablet_replica.cc:333] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.992384 11310 raft_consensus.cc:2243] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:08.993587 11310 raft_consensus.cc:2272] T 472e786b46074d868d00df67a0985afd P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:08.996819 11310 tablet_replica.cc:333] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:08.997709 11310 raft_consensus.cc:2243] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:20:08.999161 11310 raft_consensus.cc:2272] T 3eadf951b63a43fd993cea2c96d5214f P 32889a507a6b4c74aec421ffbfd1e25d [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.002323 11310 tablet_replica.cc:333] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.003208 11310 raft_consensus.cc:2243] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:09.004418 11310 raft_consensus.cc:2272] T bb9177ec2fb345c4bff68c8c8a132af6 P 32889a507a6b4c74aec421ffbfd1e25d [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.007949 11310 tablet_replica.cc:333] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.009057 11310 raft_consensus.cc:2243] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.010843 11310 raft_consensus.cc:2272] T 2dbcfa3bff034f5ab82e2ae802266099 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.014108 11310 tablet_replica.cc:333] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.015039 11310 raft_consensus.cc:2243] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.016569 11310 raft_consensus.cc:2272] T 7069a8d3584b466aab95934840523b8d P 32889a507a6b4c74aec421ffbfd1e25d [term 47 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.019796 11310 tablet_replica.cc:333] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.020782 11310 raft_consensus.cc:2243] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.022179 11310 raft_consensus.cc:2272] T 3066846eded04c878b1f03ecb43b1c79 P 32889a507a6b4c74aec421ffbfd1e25d [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.025506 11310 tablet_replica.cc:333] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.026556 11310 raft_consensus.cc:2243] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.028189 11310 raft_consensus.cc:2272] T 6fad7500a4eb45afa94184a247d796b8 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.031558 11310 tablet_replica.cc:333] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.032594 11310 raft_consensus.cc:2243] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:09.033768 11310 raft_consensus.cc:2272] T e3648047ca0742a2bfaa45ef2af517e7 P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.036926 11310 tablet_replica.cc:333] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.037868 11310 raft_consensus.cc:2243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:09.038739 11310 raft_consensus.cc:2272] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.044626 11310 tablet_replica.cc:333] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.045567 11310 raft_consensus.cc:2243] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.047128 11310 raft_consensus.cc:2272] T 05ad55cc3c04480abd0546c50226f9c1 P 32889a507a6b4c74aec421ffbfd1e25d [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.050257 11310 tablet_replica.cc:333] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.051246 11310 raft_consensus.cc:2243] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 50 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.053119 11310 raft_consensus.cc:2272] T ddf2b3bf3e6344499adb020ce08a3abf P 32889a507a6b4c74aec421ffbfd1e25d [term 50 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:09.054859 13218 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:09.056355 11310 tablet_replica.cc:333] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.057323 11310 raft_consensus.cc:2243] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:09.058315 11310 raft_consensus.cc:2272] T a1d35de928f546349f88cbea7818041c P 32889a507a6b4c74aec421ffbfd1e25d [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.062191 11310 tablet_replica.cc:333] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.063185 11310 raft_consensus.cc:2243] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:09.064282 12610 consensus_peers.cc:597] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:20:09.064399 11310 raft_consensus.cc:2272] T 07fe6c41f52649a88327e52e0352b171 P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.067481 11310 tablet_replica.cc:333] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.068336 11310 raft_consensus.cc:2243] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:09.069232 11310 raft_consensus.cc:2272] T c793ebd4de24492397303f9a6e6bc62a P 32889a507a6b4c74aec421ffbfd1e25d [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.072815 11310 tablet_replica.cc:333] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.073889 11310 raft_consensus.cc:2243] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.075397 11310 raft_consensus.cc:2272] T 43f1b7080b6a4c329281b7c405f5c998 P 32889a507a6b4c74aec421ffbfd1e25d [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.079067 11310 tablet_replica.cc:333] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.080084 11310 raft_consensus.cc:2243] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.081609 11310 raft_consensus.cc:2272] T c126dc4ef9114297874fa66b1a0c56b0 P 32889a507a6b4c74aec421ffbfd1e25d [term 57 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.084270 11310 tablet_replica.cc:333] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.084959 11310 raft_consensus.cc:2243] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.087225 11310 raft_consensus.cc:2272] T 4d3b77689c31445cbba405c2b232495e P 32889a507a6b4c74aec421ffbfd1e25d [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:09.090481 11310 tablet_replica.cc:333] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d: stopping tablet replica
I20260709 12:20:09.091317 11310 raft_consensus.cc:2243] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 55 LEADER]: Raft consensus shutting down.
I20260709 12:20:09.092917 11310 raft_consensus.cc:2272] T 03d0d1b7d2b24046b4dd2cdb4cf7c0af P 32889a507a6b4c74aec421ffbfd1e25d [term 55 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:09.116811 13217 consensus_peers.cc:597] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.170876 13217 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.196511 13217 consensus_peers.cc:597] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.206362 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:20:09.210831 12610 connection.cc:466] server connection from 127.0.0.1:35404 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35404 (request call id 3106) could send its response
W20260709 12:20:09.279655 13218 consensus_peers.cc:597] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:20:09.360556 30511 raft_consensus.cc:493] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:09.361285 30511 raft_consensus.cc:515] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:09.364617 30511 leader_election.cc:290] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:09.637898 13217 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:32927, user_credentials={real_user=slave}} blocked reactor thread for 255914us
I20260709 12:20:09.640538 30517 raft_consensus.cc:493] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
W20260709 12:20:09.648552 13217 consensus_peers.cc:597] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:20:09.649034 30517 raft_consensus.cc:515] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:09.654300 30517 leader_election.cc:290] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:09.659514 13217 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:09.661303 13218 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:35471, user_credentials={real_user=slave}} blocked reactor thread for 279700us
W20260709 12:20:09.683899 13218 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:09.696895 13218 leader_election.cc:304] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:20:09.698140 30512 raft_consensus.cc:2749] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:09.706279 13217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111) [suppressed 716 similar messages]
W20260709 12:20:09.710311 13217 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:09.711731 13217 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.714996 13217 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.712962 13218 leader_election.cc:336] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:09.724718 13217 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.737957 13218 leader_election.cc:336] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:09.738727 13218 leader_election.cc:304] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 55 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:09.740016 30512 raft_consensus.cc:2749] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Leader pre-election lost for term 55. Reason: could not achieve majority
W20260709 12:20:09.740259 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:20:09.750274 13218 consensus_peers.cc:597] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:20:09.836258 30516 raft_consensus.cc:493] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:09.837028 30516 raft_consensus.cc:515] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Starting pre-election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:09.845841 30516 leader_election.cc:290] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:09.846941 13217 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:09.858906 13218 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:09.859750 13218 leader_election.cc:304] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:09.860563 30516 raft_consensus.cc:2749] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Leader pre-election lost for term 51. Reason: could not achieve majority
W20260709 12:20:09.861594 13218 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:20:09.862481 13218 leader_election.cc:336] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 51 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
W20260709 12:20:09.891997 13217 consensus_peers.cc:597] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:20:09.898742 30516 raft_consensus.cc:493] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:09.899477 30516 raft_consensus.cc:515] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:09.906077 30516 leader_election.cc:290] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:09.908564 13217 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:09.927570 13217 consensus_peers.cc:597] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:09.937314 13218 leader_election.cc:336] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:09.940501 13218 leader_election.cc:304] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:20:09.941847 30516 raft_consensus.cc:2749] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
W20260709 12:20:09.952345 13217 consensus_peers.cc:597] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:20:10.001415 13218 consensus_peers.cc:597] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:20:10.057796 13217 consensus_peers.cc:597] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:20:10.077150 30512 raft_consensus.cc:493] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:10.077841 30512 raft_consensus.cc:515] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.080263 30512 leader_election.cc:290] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:10.085224 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:10.086112 13218 leader_election.cc:336] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:10.086548 13218 leader_election.cc:304] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:10.087327 30512 raft_consensus.cc:2749] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:20:10.215878 30516 raft_consensus.cc:493] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:10.216734 30516 raft_consensus.cc:515] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Starting pre-election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.224160 30516 leader_election.cc:290] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:10.228845 13217 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
W20260709 12:20:10.236857 13218 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:10.237515 13218 leader_election.cc:304] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:10.238420 30512 raft_consensus.cc:2749] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Leader pre-election lost for term 59. Reason: could not achieve majority
W20260709 12:20:10.238567 13218 leader_election.cc:336] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 59 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:10.241287 30516 raft_consensus.cc:493] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:10.241986 30516 raft_consensus.cc:515] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.244405 30516 leader_election.cc:290] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:10.251854 13218 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
W20260709 12:20:10.258239 13217 leader_election.cc:336] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:10.258841 13217 leader_election.cc:304] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 32889a507a6b4c74aec421ffbfd1e25d
I20260709 12:20:10.259672 30516 raft_consensus.cc:2749] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
I20260709 12:20:10.293253 30512 raft_consensus.cc:493] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:10.293913 30512 raft_consensus.cc:515] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.296350 30512 leader_election.cc:290] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:10.303126 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:10.307497 13218 leader_election.cc:336] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:10.307972 13218 leader_election.cc:304] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 54 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:10.308838 30512 raft_consensus.cc:2749] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Leader pre-election lost for term 54. Reason: could not achieve majority
W20260709 12:20:10.327076 13217 consensus_peers.cc:597] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:20:10.382897 30512 raft_consensus.cc:493] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:10.383690 30512 raft_consensus.cc:515] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Starting pre-election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } attrs { replace: true } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.386840 30512 leader_election.cc:290] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:10.394058 13218 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:10.395367 13217 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:10.395853 13217 leader_election.cc:304] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
W20260709 12:20:10.396694 13218 leader_election.cc:336] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 45 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:10.396875 30512 raft_consensus.cc:2749] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Leader pre-election lost for term 45. Reason: could not achieve majority
W20260709 12:20:10.413450 13217 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:20:10.433661 30512 raft_consensus.cc:493] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:10.434262 30512 raft_consensus.cc:515] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
W20260709 12:20:10.436625 12426 debug-util.cc:398] Leaking SignalData structure 0x7b080000b620 after lost signal to thread 11313
W20260709 12:20:10.437911 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08000fd300 after lost signal to thread 12429
W20260709 12:20:10.439512 12426 debug-util.cc:398] Leaking SignalData structure 0x7b08003a8400 after lost signal to thread 13286
I20260709 12:20:10.443152 30512 leader_election.cc:290] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:10.445818 13218 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:10.452988 13217 leader_election.cc:336] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:10.453850 13217 leader_election.cc:304] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 52 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:10.454926 30512 raft_consensus.cc:2749] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Leader pre-election lost for term 52. Reason: could not achieve majority
I20260709 12:20:10.481230 30512 raft_consensus.cc:493] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Starting pre-election (detected failure of leader 32889a507a6b4c74aec421ffbfd1e25d)
I20260709 12:20:10.482087 30512 raft_consensus.cc:515] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.484978 30512 leader_election.cc:290] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 58 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471)
W20260709 12:20:10.494022 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:10.494956 13218 leader_election.cc:336] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 58 pre-election: RPC error from VoteRequest() call to peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111)
I20260709 12:20:10.495406 13218 leader_election.cc:304] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 58 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 32889a507a6b4c74aec421ffbfd1e25d, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:10.496899 30512 raft_consensus.cc:2749] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Leader pre-election lost for term 58. Reason: could not achieve majority
I20260709 12:20:10.538852 30512 raft_consensus.cc:493] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election (detected failure of leader 0a29b5d9f9d44298b02ffcd460811eef)
I20260709 12:20:10.539438 30512 raft_consensus.cc:515] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: false } }
I20260709 12:20:10.541610 30512 leader_election.cc:290] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Requested pre-vote from peers 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859), 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927)
W20260709 12:20:10.549386 13218 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:10.559098 13217 consensus_peers.cc:597] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e -> Peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Couldn't send request to peer 0a29b5d9f9d44298b02ffcd460811eef. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:20:10.559676 13217 leader_election.cc:336] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: RPC error from VoteRequest() call to peer 0a29b5d9f9d44298b02ffcd460811eef (127.11.11.130:32927): Network error: Client connection negotiation failed: client connection to 127.11.11.130:32927: connect: Connection refused (error 111)
I20260709 12:20:10.560123 13217 leader_election.cc:304] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [CANDIDATE]: Term 46 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9485b0afaf144396adf854fff857414e; no voters: 0a29b5d9f9d44298b02ffcd460811eef, 6d4ece942fd5414998215d26edf91001
I20260709 12:20:10.560915 30512 raft_consensus.cc:2749] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Leader pre-election lost for term 46. Reason: could not achieve majority
I20260709 12:20:10.813167 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
W20260709 12:20:10.819497 13218 consensus_peers.cc:597] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:20:10.969231 11310 tablet_server.cc:179] TabletServer@127.11.11.132:0 shutting down...
W20260709 12:20:11.090140 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:11.099273 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:11.100669 11310 tablet_replica.cc:333] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.102075 11310 raft_consensus.cc:2243] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.103354 11310 raft_consensus.cc:2272] T 979333b179ba4f49a65315a06d54e27e P 9485b0afaf144396adf854fff857414e [term 58 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:11.111967 13218 consensus_peers.cc:597] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:20:11.113962 11310 tablet_replica.cc:333] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.115334 11310 raft_consensus.cc:2243] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.119324 11310 raft_consensus.cc:2272] T a7a8c7204d6b44c9b3d6ea6332e9f824 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:11.123201 13218 consensus_peers.cc:597] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:11.137953 13218 consensus_peers.cc:597] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:11.142976 11310 tablet_replica.cc:333] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.145936 11310 raft_consensus.cc:2243] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.147789 11310 raft_consensus.cc:2272] T 3da4ae1a8feb46d8a6bd6ec5522932d5 P 9485b0afaf144396adf854fff857414e [term 50 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:11.153492 13218 consensus_peers.cc:597] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:11.154389 13218 consensus_peers.cc:597] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e -> Peer 32889a507a6b4c74aec421ffbfd1e25d (127.11.11.131:35471): Couldn't send request to peer 32889a507a6b4c74aec421ffbfd1e25d. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:35471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:11.154451 11310 tablet_replica.cc:333] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.155923 11310 raft_consensus.cc:2243] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.157508 11310 raft_consensus.cc:2272] T 5f402dfd68ae4c8697765a76d9f1c4e6 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.162038 11310 tablet_replica.cc:333] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.163208 11310 raft_consensus.cc:2243] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.164608 11310 raft_consensus.cc:2272] T 4d3b77689c31445cbba405c2b232495e P 9485b0afaf144396adf854fff857414e [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.171134 11310 tablet_replica.cc:333] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.174558 11310 raft_consensus.cc:2243] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.176622 11310 raft_consensus.cc:2272] T a30ed0986fa641bd8f88912b4b9ee320 P 9485b0afaf144396adf854fff857414e [term 42 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.186293 11310 tablet_replica.cc:333] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.187927 11310 raft_consensus.cc:2243] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.189985 11310 raft_consensus.cc:2272] T 27144fedf0014de09f007ac6cc3f7275 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:11.192124 13218 consensus_peers.cc:597] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e -> Peer 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): Couldn't send request to peer 6d4ece942fd5414998215d26edf91001. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:20:11.194380 11310 tablet_replica.cc:333] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.195451 11310 raft_consensus.cc:2243] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.196249 11310 raft_consensus.cc:2272] T 54875fd261ee4c328e3d022d6a9418d8 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.199728 11310 tablet_replica.cc:333] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.200723 11310 raft_consensus.cc:2243] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.202901 11310 raft_consensus.cc:2272] T e3648047ca0742a2bfaa45ef2af517e7 P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.206343 11310 tablet_replica.cc:333] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.207378 11310 raft_consensus.cc:2243] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.208328 11310 raft_consensus.cc:2272] T 5d60fb5ad76741448a5376ca4d3f05df P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.212790 11310 tablet_replica.cc:333] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.213718 11310 raft_consensus.cc:2243] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.214972 11310 raft_consensus.cc:2272] T 509f0430baf04e3393879d82fb9d10f9 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.219094 11310 tablet_replica.cc:333] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.220021 11310 raft_consensus.cc:2243] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.221236 11310 raft_consensus.cc:2272] T 5d0dce81bc984a0690d08335a4937b2c P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.224534 11310 tablet_replica.cc:333] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.225430 11310 raft_consensus.cc:2243] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [term 40 LEARNER]: Raft consensus shutting down.
I20260709 12:20:11.226107 11310 pending_rounds.cc:70] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Trying to abort 1 pending ops.
I20260709 12:20:11.226398 11310 pending_rounds.cc:77] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e: Aborting op as it isn't in flight: id { term: 39 index: 23 } timestamp: 7305623656570089472 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3eadf951b63a43fd993cea2c96d5214f" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } } } new_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6d4ece942fd5414998215d26edf91001" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 38859 } } peers { permanent_uuid: "0a29b5d9f9d44298b02ffcd460811eef" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 32927 } } peers { permanent_uuid: "32889a507a6b4c74aec421ffbfd1e25d" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 35471 } attrs { replace: true } } peers { permanent_uuid: "9485b0afaf144396adf854fff857414e" member_type: NON_VOTER last_known_addr { host: "127.11.11.132" port: 33793 } attrs { promote: true } } } }
I20260709 12:20:11.227805 11310 raft_consensus.cc:2917] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [term 40 LEARNER]: Aborting config change with OpId 39.23: Aborted: Op aborted
I20260709 12:20:11.228281 11310 raft_consensus.cc:2889] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [term 40 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 12:20:11.228628 11310 raft_consensus.cc:2272] T 3eadf951b63a43fd993cea2c96d5214f P 9485b0afaf144396adf854fff857414e [term 40 NON_PARTICIPANT]: Raft consensus is shut down!
I20260709 12:20:11.233031 11310 tablet_replica.cc:333] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.234308 11310 raft_consensus.cc:2243] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 56 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.236017 11310 raft_consensus.cc:2272] T 472e786b46074d868d00df67a0985afd P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.239668 11310 tablet_replica.cc:333] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.240783 11310 raft_consensus.cc:2243] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.242564 11310 raft_consensus.cc:2272] T 6f701a23559945868ce6e3da5866f95c P 9485b0afaf144396adf854fff857414e [term 56 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.248924 11310 tablet_replica.cc:333] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.249919 11310 raft_consensus.cc:2243] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.251441 11310 raft_consensus.cc:2272] T a3a0781287514c758f107d79c1f96be4 P 9485b0afaf144396adf854fff857414e [term 45 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.256173 11310 tablet_replica.cc:333] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.257014 11310 raft_consensus.cc:2243] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 46 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.258443 11310 raft_consensus.cc:2272] T dd6a5b4068ea44d2ac6f61924dd38c0d P 9485b0afaf144396adf854fff857414e [term 46 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.261585 11310 tablet_replica.cc:333] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.262358 11310 raft_consensus.cc:2243] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.263877 11310 raft_consensus.cc:2272] T 0c56dde469374f4da7440f134140670c P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.267138 11310 tablet_replica.cc:333] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.267982 11310 raft_consensus.cc:2243] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.269345 11310 raft_consensus.cc:2272] T c793ebd4de24492397303f9a6e6bc62a P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.272938 11310 tablet_replica.cc:333] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.273769 11310 raft_consensus.cc:2243] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.275048 11310 raft_consensus.cc:2272] T 07fe6c41f52649a88327e52e0352b171 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.279179 11310 tablet_replica.cc:333] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.280325 11310 raft_consensus.cc:2243] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.281255 11310 raft_consensus.cc:2272] T c13f916d8b4d463182a1d1eb50150e31 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.284690 11310 tablet_replica.cc:333] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.285533 11310 raft_consensus.cc:2243] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 59 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.286788 11310 raft_consensus.cc:2272] T e1f18763e7844e0994e2f11a6dd89290 P 9485b0afaf144396adf854fff857414e [term 59 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.290052 11310 tablet_replica.cc:333] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.290925 11310 raft_consensus.cc:2243] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 54 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.292208 11310 raft_consensus.cc:2272] T 79b687d20a534e70865ef902fbd0d4a5 P 9485b0afaf144396adf854fff857414e [term 54 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.295218 11310 tablet_replica.cc:333] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.296079 11310 raft_consensus.cc:2243] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.296875 11310 raft_consensus.cc:2272] T 40d0e235c01b4b7f89948ab708403a01 P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.299986 11310 tablet_replica.cc:333] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.300729 11310 raft_consensus.cc:2243] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 51 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.301975 11310 raft_consensus.cc:2272] T 3bb3f87ad7904e7d97f68dbbd8c3ab60 P 9485b0afaf144396adf854fff857414e [term 51 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.305382 11310 tablet_replica.cc:333] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.306318 11310 raft_consensus.cc:2243] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 61 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.308211 11310 raft_consensus.cc:2272] T 9bbf126b18234b78b4f36bf9708ad96b P 9485b0afaf144396adf854fff857414e [term 61 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.311285 11310 tablet_replica.cc:333] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.312152 11310 raft_consensus.cc:2243] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 48 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.313409 11310 raft_consensus.cc:2272] T 4dc881215d9f4498b162561e79143f2d P 9485b0afaf144396adf854fff857414e [term 48 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.316555 11310 tablet_replica.cc:333] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.317301 11310 raft_consensus.cc:2243] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 49 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.318575 11310 raft_consensus.cc:2272] T 65037753870c4624995dde1173266f49 P 9485b0afaf144396adf854fff857414e [term 49 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.321758 11310 tablet_replica.cc:333] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.322729 11310 raft_consensus.cc:2243] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.324532 11310 raft_consensus.cc:2272] T 43fecbd2edc14ba2ab065395e5d48d1d P 9485b0afaf144396adf854fff857414e [term 53 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.327980 11310 tablet_replica.cc:333] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.328743 11310 raft_consensus.cc:2243] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 52 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.330128 11310 raft_consensus.cc:2272] T 7b3c8a7a06a04c369587d6ecb196bb2e P 9485b0afaf144396adf854fff857414e [term 52 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.333765 11310 tablet_replica.cc:333] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.334529 11310 raft_consensus.cc:2243] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:11.335400 11310 raft_consensus.cc:2272] T 2dbcfa3bff034f5ab82e2ae802266099 P 9485b0afaf144396adf854fff857414e [term 57 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:11.338465 11310 tablet_replica.cc:333] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e: stopping tablet replica
I20260709 12:20:11.339130 11310 raft_consensus.cc:2243] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 LEADER]: Raft consensus shutting down.
I20260709 12:20:11.340408 11310 raft_consensus.cc:2272] T 59beb90f79b0450ea2fbb2b7e405261d P 9485b0afaf144396adf854fff857414e [term 55 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:12.053176 11310 tablet_server.cc:196] TabletServer@127.11.11.132:0 shutdown complete.
I20260709 12:20:12.136161 11310 master.cc:562] Master@127.11.11.190:35399 shutting down...
W20260709 12:20:12.194410 12443 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:20:12.195091 12443 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:20:12.195387 12443 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:20:21.730002 12380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20260709 12:20:21.735251 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a30ed0986fa641bd8f88912b4b9ee320: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
W20260709 12:20:22.935773 12380 catalog_manager.cc:4755] TS 6d4ece942fd5414998215d26edf91001 (127.11.11.129:38859): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f402dfd68ae4c8697765a76d9f1c4e6: Network error: Client connection negotiation failed: client connection to 127.11.11.129:38859: connect: Connection refused (error 111)
I20260709 12:20:23.099645 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:23.101150 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:23.101672 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2d10571632e7445d99e8f1c750c96f70: stopping tablet replica
I20260709 12:20:23.400009 11310 master.cc:584] Master@127.11.11.190:35399 shutdown complete.
I20260709 12:20:23.470945 11310 test_util.cc:182] -----------------------------------------------
I20260709 12:20:23.471273 11310 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783598992211714-11310-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (568708 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:20:23.475571 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:37085
I20260709 12:20:23.476874 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:23.484508 30524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.484495 30525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.485396 30527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:23.487013 11310 server_base.cc:1061] running on GCE node
I20260709 12:20:23.494918 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:23.495222 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:23.495422 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599623495397 us; error 0 us; skew 500 ppm
I20260709 12:20:23.496121 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:23.499684 11310 webserver.cc:533] Webserver started at http://127.11.11.190:42083/ using document root <none> and password file <none>
I20260709 12:20:23.500385 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:23.500612 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:23.500959 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:23.502969 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "906ec57f6a8740bfbcc051926eb49307"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:23.509009 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:20:23.513861 30532 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.514864 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:20:23.515321 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "906ec57f6a8740bfbcc051926eb49307"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:23.515657 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:23.536697 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:23.538318 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:23.588456 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:37085
I20260709 12:20:23.588578 30583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:37085 every 8 connection(s)
I20260709 12:20:23.593480 30584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:23.607234 30584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307: Bootstrap starting.
I20260709 12:20:23.612692 30584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.618590 30584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307: No bootstrap required, opened a new log
I20260709 12:20:23.621359 30584 raft_consensus.cc:359] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906ec57f6a8740bfbcc051926eb49307" member_type: VOTER }
I20260709 12:20:23.621999 30584 raft_consensus.cc:385] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.622262 30584 raft_consensus.cc:740] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906ec57f6a8740bfbcc051926eb49307, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.622992 30584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906ec57f6a8740bfbcc051926eb49307" member_type: VOTER }
I20260709 12:20:23.623579 30584 raft_consensus.cc:399] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:23.623848 30584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:23.624210 30584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.629954 30584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906ec57f6a8740bfbcc051926eb49307" member_type: VOTER }
I20260709 12:20:23.630653 30584 leader_election.cc:304] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 906ec57f6a8740bfbcc051926eb49307; no voters: 
I20260709 12:20:23.632443 30584 leader_election.cc:290] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:23.632943 30587 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.634814 30587 raft_consensus.cc:697] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 1 LEADER]: Becoming Leader. State: Replica: 906ec57f6a8740bfbcc051926eb49307, State: Running, Role: LEADER
I20260709 12:20:23.635699 30587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906ec57f6a8740bfbcc051926eb49307" member_type: VOTER }
I20260709 12:20:23.636394 30584 sys_catalog.cc:565] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:23.644524 30589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 906ec57f6a8740bfbcc051926eb49307. Latest consensus state: current_term: 1 leader_uuid: "906ec57f6a8740bfbcc051926eb49307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906ec57f6a8740bfbcc051926eb49307" member_type: VOTER } }
I20260709 12:20:23.645339 30588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "906ec57f6a8740bfbcc051926eb49307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906ec57f6a8740bfbcc051926eb49307" member_type: VOTER } }
I20260709 12:20:23.645736 30589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:23.646226 30588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:23.648447 30596 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:23.654428 30596 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:23.656394 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:23.665912 30596 catalog_manager.cc:1383] Generated new cluster ID: e1934c788c0640b89ecc78724e7b946b
I20260709 12:20:23.666286 30596 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:23.681512 30596 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:23.683406 30596 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:23.697721 30596 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307: Generated new TSK 0
I20260709 12:20:23.698547 30596 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:23.724396 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:23.732609 30605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.733245 30606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.736039 30608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:23.737802 11310 server_base.cc:1061] running on GCE node
I20260709 12:20:23.742354 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:23.742606 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:23.742866 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599623742822 us; error 0 us; skew 500 ppm
I20260709 12:20:23.743522 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:23.746428 11310 webserver.cc:533] Webserver started at http://127.11.11.129:35215/ using document root <none> and password file <none>
I20260709 12:20:23.746969 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:23.747143 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:23.747488 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:23.748795 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "c18174c596f9488f8be666b85062e329"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:23.754268 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:20:23.758445 30613 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.759526 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:20:23.759914 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "c18174c596f9488f8be666b85062e329"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:23.760355 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:23.778403 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:23.779844 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:23.781590 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:23.784407 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:23.784613 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.784941 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:23.785135 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.839519 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:41991
I20260709 12:20:23.839669 30675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:41991 every 8 connection(s)
I20260709 12:20:23.845964 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:23.856168 30680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.858554 30681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.862417 30683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:23.865551 11310 server_base.cc:1061] running on GCE node
I20260709 12:20:23.866897 30676 heartbeater.cc:344] Connected to a master server at 127.11.11.190:37085
I20260709 12:20:23.867449 30676 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:23.868708 30676 heartbeater.cc:507] Master 127.11.11.190:37085 requested a full tablet report, sending...
I20260709 12:20:23.871984 30549 ts_manager.cc:194] Registered new tserver with Master: c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:23.872557 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:23.872887 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:23.873095 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599623873050 us; error 0 us; skew 500 ppm
I20260709 12:20:23.873883 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:23.874325 30549 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56278
I20260709 12:20:23.877291 11310 webserver.cc:533] Webserver started at http://127.11.11.130:37275/ using document root <none> and password file <none>
I20260709 12:20:23.878154 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:23.878378 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:23.878623 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:23.880108 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "eb6634546938405dac9eabbadd34d2b1"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:23.885712 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:20:23.890084 30688 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.891093 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:20:23.891433 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "eb6634546938405dac9eabbadd34d2b1"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:23.891705 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:23.913341 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:23.915167 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:23.916997 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:23.920046 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:23.920269 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.920626 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:23.920773 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:23.975991 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:37181
I20260709 12:20:23.976140 30750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:37181 every 8 connection(s)
I20260709 12:20:23.982383 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:23.992081 30754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:23.994073 30755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:23.999718 30751 heartbeater.cc:344] Connected to a master server at 127.11.11.190:37085
I20260709 12:20:24.000412 30751 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:24.001386 11310 server_base.cc:1061] running on GCE node
W20260709 12:20:24.001480 30758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:24.001780 30751 heartbeater.cc:507] Master 127.11.11.190:37085 requested a full tablet report, sending...
I20260709 12:20:24.003219 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:24.003527 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:24.003815 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599624003796 us; error 0 us; skew 500 ppm
I20260709 12:20:24.004560 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:24.005077 30549 ts_manager.cc:194] Registered new tserver with Master: eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:24.007220 30549 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56294
I20260709 12:20:24.007879 11310 webserver.cc:533] Webserver started at http://127.11.11.131:37709/ using document root <none> and password file <none>
I20260709 12:20:24.008554 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:24.008873 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:24.009289 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:24.011090 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "c178a628712047a88731a915fc93ed19"
format_stamp: "Formatted at 2026-07-09 12:20:24 on dist-test-slave-4qvp"
I20260709 12:20:24.016486 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:20:24.021363 30762 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:24.022504 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:20:24.022894 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "c178a628712047a88731a915fc93ed19"
format_stamp: "Formatted at 2026-07-09 12:20:24 on dist-test-slave-4qvp"
I20260709 12:20:24.023217 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:24.045197 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:24.046574 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:24.048143 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:24.050838 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:24.051028 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:24.051373 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:24.051539 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:24.102929 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:39529
I20260709 12:20:24.103075 30825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:39529 every 8 connection(s)
I20260709 12:20:24.109277 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:24.119333 30829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:24.121588 30830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:24.126235 30832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:24.126298 11310 server_base.cc:1061] running on GCE node
I20260709 12:20:24.127321 30826 heartbeater.cc:344] Connected to a master server at 127.11.11.190:37085
I20260709 12:20:24.127776 30826 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:24.128217 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:24.128561 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:24.128696 30826 heartbeater.cc:507] Master 127.11.11.190:37085 requested a full tablet report, sending...
I20260709 12:20:24.128780 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599624128762 us; error 0 us; skew 500 ppm
I20260709 12:20:24.129729 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:24.131532 30549 ts_manager.cc:194] Registered new tserver with Master: c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:24.133270 11310 webserver.cc:533] Webserver started at http://127.11.11.132:36563/ using document root <none> and password file <none>
I20260709 12:20:24.133543 30549 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56306
I20260709 12:20:24.134135 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:24.134464 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:24.134853 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:24.136507 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-3-root/instance:
uuid: "23ba075510134f3c8caba2370fff0d40"
format_stamp: "Formatted at 2026-07-09 12:20:24 on dist-test-slave-4qvp"
I20260709 12:20:24.142649 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 12:20:24.147784 30837 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:24.149055 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:20:24.149683 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-3-root
uuid: "23ba075510134f3c8caba2370fff0d40"
format_stamp: "Formatted at 2026-07-09 12:20:24 on dist-test-slave-4qvp"
I20260709 12:20:24.150146 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:24.175380 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:24.177011 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:24.178722 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:24.181941 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:24.182236 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:24.182516 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:24.182708 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:24.240049 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.132:36125
I20260709 12:20:24.240203 30900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.132:36125 every 8 connection(s)
I20260709 12:20:24.257483 30901 heartbeater.cc:344] Connected to a master server at 127.11.11.190:37085
I20260709 12:20:24.258194 30901 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:24.259358 30901 heartbeater.cc:507] Master 127.11.11.190:37085 requested a full tablet report, sending...
I20260709 12:20:24.262310 30549 ts_manager.cc:194] Registered new tserver with Master: 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:24.263195 11310 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.019156586s
I20260709 12:20:24.264133 30549 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56318
I20260709 12:20:24.300595 30549 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56330:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:20:24.585115 30641 tablet_service.cc:1511] Processing CreateTablet for tablet 3db7a62e2e80411ea0c98d2b5282b08f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:24.586937 30641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db7a62e2e80411ea0c98d2b5282b08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.595618 30790 tablet_service.cc:1511] Processing CreateTablet for tablet 61d416be13d449318d3e6f13a14a5d4a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:24.596654 30639 tablet_service.cc:1511] Processing CreateTablet for tablet ef90e8d046fc47fcb7d122ffad59a1f3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:24.598399 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90e8d046fc47fcb7d122ffad59a1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.598023 30712 tablet_service.cc:1511] Processing CreateTablet for tablet 8bc4eca36172433d9134485920115dcc (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:24.599086 30640 tablet_service.cc:1511] Processing CreateTablet for tablet 61d416be13d449318d3e6f13a14a5d4a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:24.599859 30712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc4eca36172433d9134485920115dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.600739 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d416be13d449318d3e6f13a14a5d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.597462 30790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d416be13d449318d3e6f13a14a5d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.603021 30716 tablet_service.cc:1511] Processing CreateTablet for tablet 3db7a62e2e80411ea0c98d2b5282b08f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:24.604676 30716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db7a62e2e80411ea0c98d2b5282b08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.607161 30715 tablet_service.cc:1511] Processing CreateTablet for tablet 61d416be13d449318d3e6f13a14a5d4a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:24.607379 30638 tablet_service.cc:1511] Processing CreateTablet for tablet 7a5651d4117243d69cfd21927e6ce1f9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:24.608834 30715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d416be13d449318d3e6f13a14a5d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.609228 30638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5651d4117243d69cfd21927e6ce1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.609445 30789 tablet_service.cc:1511] Processing CreateTablet for tablet ef90e8d046fc47fcb7d122ffad59a1f3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:24.611194 30789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90e8d046fc47fcb7d122ffad59a1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.611032 30713 tablet_service.cc:1511] Processing CreateTablet for tablet 7a5651d4117243d69cfd21927e6ce1f9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:24.612619 30713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5651d4117243d69cfd21927e6ce1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.614997 30714 tablet_service.cc:1511] Processing CreateTablet for tablet ecdffc08164a4124ad64c3b376d308d5 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:24.619740 30637 tablet_service.cc:1511] Processing CreateTablet for tablet 8bc4eca36172433d9134485920115dcc (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:24.621649 30637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc4eca36172433d9134485920115dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.622156 30714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecdffc08164a4124ad64c3b376d308d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.623499 30788 tablet_service.cc:1511] Processing CreateTablet for tablet ecdffc08164a4124ad64c3b376d308d5 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:24.625458 30788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecdffc08164a4124ad64c3b376d308d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.628818 30787 tablet_service.cc:1511] Processing CreateTablet for tablet 7a5651d4117243d69cfd21927e6ce1f9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:24.630554 30787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5651d4117243d69cfd21927e6ce1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.631691 30711 tablet_service.cc:1511] Processing CreateTablet for tablet db650f2a9f164bcd9c32d847363b1745 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:24.635092 30710 tablet_service.cc:1511] Processing CreateTablet for tablet 1f70173e529b4bdca23bc1fa8fbb2bde (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:24.636798 30711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db650f2a9f164bcd9c32d847363b1745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.638002 30709 tablet_service.cc:1511] Processing CreateTablet for tablet e95ca1a81eea4096a755f54943e95976 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:24.639684 30709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95ca1a81eea4096a755f54943e95976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.639392 30786 tablet_service.cc:1511] Processing CreateTablet for tablet f77972ceb0c74ddca205e16542d12a4d (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:24.641314 30786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77972ceb0c74ddca205e16542d12a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.642377 30710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f70173e529b4bdca23bc1fa8fbb2bde. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:24.651633 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:24.653914 30785 tablet_service.cc:1511] Processing CreateTablet for tablet db650f2a9f164bcd9c32d847363b1745 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:24.655711 30785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db650f2a9f164bcd9c32d847363b1745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.661492 30784 tablet_service.cc:1511] Processing CreateTablet for tablet 1f70173e529b4bdca23bc1fa8fbb2bde (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:24.663630 30784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f70173e529b4bdca23bc1fa8fbb2bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.663182 30708 tablet_service.cc:1511] Processing CreateTablet for tablet 14747c6d552943dbac8ab19f68816ea9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:24.665462 30707 tablet_service.cc:1511] Processing CreateTablet for tablet b4d076571a8e4367836518e952de6f11 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:24.666812 30708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14747c6d552943dbac8ab19f68816ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.667610 30707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d076571a8e4367836518e952de6f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.670536 30636 tablet_service.cc:1511] Processing CreateTablet for tablet f77972ceb0c74ddca205e16542d12a4d (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:24.672432 30636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77972ceb0c74ddca205e16542d12a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.683907 30783 tablet_service.cc:1511] Processing CreateTablet for tablet 3be7907c61d7477da4fc1646f65b2a48 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:24.684095 30782 tablet_service.cc:1511] Processing CreateTablet for tablet 14747c6d552943dbac8ab19f68816ea9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:24.686879 30783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be7907c61d7477da4fc1646f65b2a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.686751 30781 tablet_service.cc:1511] Processing CreateTablet for tablet b4d076571a8e4367836518e952de6f11 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:24.688604 30781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d076571a8e4367836518e952de6f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.690119 30782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14747c6d552943dbac8ab19f68816ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.707073 30632 tablet_service.cc:1511] Processing CreateTablet for tablet b4d076571a8e4367836518e952de6f11 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:24.708818 30632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4d076571a8e4367836518e952de6f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.711102 30634 tablet_service.cc:1511] Processing CreateTablet for tablet e95ca1a81eea4096a755f54943e95976 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:24.712765 30634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95ca1a81eea4096a755f54943e95976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.718832 30635 tablet_service.cc:1511] Processing CreateTablet for tablet 1f70173e529b4bdca23bc1fa8fbb2bde (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:24.720431 30635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f70173e529b4bdca23bc1fa8fbb2bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.722877 30633 tablet_service.cc:1511] Processing CreateTablet for tablet 3be7907c61d7477da4fc1646f65b2a48 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:24.724448 30633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be7907c61d7477da4fc1646f65b2a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.727445 30864 tablet_service.cc:1511] Processing CreateTablet for tablet ef90e8d046fc47fcb7d122ffad59a1f3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:24.731357 30863 tablet_service.cc:1511] Processing CreateTablet for tablet ecdffc08164a4124ad64c3b376d308d5 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:24.734284 30862 tablet_service.cc:1511] Processing CreateTablet for tablet 8bc4eca36172433d9134485920115dcc (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:24.737458 30861 tablet_service.cc:1511] Processing CreateTablet for tablet f77972ceb0c74ddca205e16542d12a4d (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:24.742971 30859 tablet_service.cc:1511] Processing CreateTablet for tablet e95ca1a81eea4096a755f54943e95976 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:24.746234 30858 tablet_service.cc:1511] Processing CreateTablet for tablet 3be7907c61d7477da4fc1646f65b2a48 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:24.748888 30857 tablet_service.cc:1511] Processing CreateTablet for tablet 14747c6d552943dbac8ab19f68816ea9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:24.751674 30856 tablet_service.cc:1511] Processing CreateTablet for tablet 6072fe2fb27a44558a09de34939e4421 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:24.754922 30865 tablet_service.cc:1511] Processing CreateTablet for tablet 3db7a62e2e80411ea0c98d2b5282b08f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:24.741063 30860 tablet_service.cc:1511] Processing CreateTablet for tablet db650f2a9f164bcd9c32d847363b1745 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:24.785351 30864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef90e8d046fc47fcb7d122ffad59a1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.806248 30859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95ca1a81eea4096a755f54943e95976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.805279 30714 tablet_service.cc:1511] Processing CreateTablet for tablet 6072fe2fb27a44558a09de34939e4421 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:24.807876 30714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6072fe2fb27a44558a09de34939e4421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.808736 30865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db7a62e2e80411ea0c98d2b5282b08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.818127 30858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be7907c61d7477da4fc1646f65b2a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.822150 30861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f77972ceb0c74ddca205e16542d12a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.826160 30857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14747c6d552943dbac8ab19f68816ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.830104 30856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6072fe2fb27a44558a09de34939e4421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.830781 30714 tablet_service.cc:1511] Processing CreateTablet for tablet 11bb43acd4e84e5aa5c2731a70542e07 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:24.832422 30714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bb43acd4e84e5aa5c2731a70542e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.834133 30862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc4eca36172433d9134485920115dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.838261 30863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecdffc08164a4124ad64c3b376d308d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.842130 30860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db650f2a9f164bcd9c32d847363b1745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.850371 30786 tablet_service.cc:1511] Processing CreateTablet for tablet 83d271f20d484fc5bce5814a664ac807 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:24.852176 30786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d271f20d484fc5bce5814a664ac807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.858408 30714 tablet_service.cc:1511] Processing CreateTablet for tablet 04fe0c41d3d440498251b09d3d6f5110 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:24.859889 30714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fe0c41d3d440498251b09d3d6f5110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.863516 30638 tablet_service.cc:1511] Processing CreateTablet for tablet 6072fe2fb27a44558a09de34939e4421 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:24.865214 30638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6072fe2fb27a44558a09de34939e4421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.876935 30710 tablet_service.cc:1511] Processing CreateTablet for tablet c1be5a3ea104480399760b60f846245b (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:24.878547 30710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1be5a3ea104480399760b60f846245b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.879850 30707 tablet_service.cc:1511] Processing CreateTablet for tablet e614ae4fde394b488924f185e2808b18 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:24.882184 30707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e614ae4fde394b488924f185e2808b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.882396 30676 heartbeater.cc:499] Master 127.11.11.190:37085 was elected leader, sending a full tablet report...
I20260709 12:20:24.911705 30633 tablet_service.cc:1511] Processing CreateTablet for tablet 88343a7c45e54babbd7fcae9b6376094 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:24.913321 30633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88343a7c45e54babbd7fcae9b6376094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.888127 30639 tablet_service.cc:1511] Processing CreateTablet for tablet 83d271f20d484fc5bce5814a664ac807 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:24.931089 30788 tablet_service.cc:1511] Processing CreateTablet for tablet 04fe0c41d3d440498251b09d3d6f5110 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:24.935468 30637 tablet_service.cc:1511] Processing CreateTablet for tablet 04fe0c41d3d440498251b09d3d6f5110 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:24.941484 30713 tablet_service.cc:1511] Processing CreateTablet for tablet c274ea8c29774701bdc1708a1839fa88 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:24.943220 30713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c274ea8c29774701bdc1708a1839fa88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.901356 30790 tablet_service.cc:1511] Processing CreateTablet for tablet 11bb43acd4e84e5aa5c2731a70542e07 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:24.948482 30715 tablet_service.cc:1511] Processing CreateTablet for tablet 88343a7c45e54babbd7fcae9b6376094 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:24.950439 30715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88343a7c45e54babbd7fcae9b6376094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.952211 30781 tablet_service.cc:1511] Processing CreateTablet for tablet 66db71b232844f1794f53dba1cbd6198 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:24.984575 30711 tablet_service.cc:1511] Processing CreateTablet for tablet b9ed5ce7483340edbbac9dee8cca5f1d (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:24.965797 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d271f20d484fc5bce5814a664ac807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.019829 30637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fe0c41d3d440498251b09d3d6f5110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.933554 30788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fe0c41d3d440498251b09d3d6f5110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.968061 30922 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:24.968873 30789 tablet_service.cc:1511] Processing CreateTablet for tablet c1be5a3ea104480399760b60f846245b (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:25.031800 30789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1be5a3ea104480399760b60f846245b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:25.035792 30534 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 89482.8us
I20260709 12:20:24.972208 30641 tablet_service.cc:1511] Processing CreateTablet for tablet c1be5a3ea104480399760b60f846245b (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:24.957883 30785 tablet_service.cc:1511] Processing CreateTablet for tablet e614ae4fde394b488924f185e2808b18 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:25.011062 30751 heartbeater.cc:499] Master 127.11.11.190:37085 was elected leader, sending a full tablet report...
I20260709 12:20:25.042994 30787 tablet_service.cc:1511] Processing CreateTablet for tablet e5899b18ce3a453ea77be2cdf20e543a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:25.044544 30787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5899b18ce3a453ea77be2cdf20e543a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:24.966322 30921 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:25.056413 30922 tablet_bootstrap.cc:654] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.970216 30923 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:25.066051 30641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1be5a3ea104480399760b60f846245b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.021852 30711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ed5ce7483340edbbac9dee8cca5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.095484 30708 tablet_service.cc:1511] Processing CreateTablet for tablet 7470dd49ce524f4aaf3faf7739fe5bd7 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:25.122954 30716 tablet_service.cc:1511] Processing CreateTablet for tablet a5c48ae242924bcab94096bb803d25d4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:25.129326 30921 tablet_bootstrap.cc:654] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:25.130005 30533 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 158703us
I20260709 12:20:25.137151 30826 heartbeater.cc:499] Master 127.11.11.190:37085 was elected leader, sending a full tablet report...
W20260709 12:20:25.148108 30534 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 61819.8us
I20260709 12:20:25.154951 30716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c48ae242924bcab94096bb803d25d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.157929 30785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e614ae4fde394b488924f185e2808b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.163426 30710 tablet_service.cc:1511] Processing CreateTablet for tablet 2aafa16c29f04448945f5db4aa191223 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:25.193663 30707 tablet_service.cc:1511] Processing CreateTablet for tablet 9873e94bae58472f8de439d5dc1835a6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:25.195227 30864 tablet_service.cc:1511] Processing CreateTablet for tablet 83d271f20d484fc5bce5814a664ac807 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:25.189177 30635 tablet_service.cc:1511] Processing CreateTablet for tablet b9ed5ce7483340edbbac9dee8cca5f1d (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:25.218413 30790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bb43acd4e84e5aa5c2731a70542e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.219419 30864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d271f20d484fc5bce5814a664ac807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.219614 30635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ed5ce7483340edbbac9dee8cca5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.215737 30923 tablet_bootstrap.cc:654] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.237193 30862 tablet_service.cc:1511] Processing CreateTablet for tablet 11bb43acd4e84e5aa5c2731a70542e07 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:25.239244 30862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bb43acd4e84e5aa5c2731a70542e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.238114 30710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aafa16c29f04448945f5db4aa191223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.242198 30707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9873e94bae58472f8de439d5dc1835a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.245844 30781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66db71b232844f1794f53dba1cbd6198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.246132 30861 tablet_service.cc:1511] Processing CreateTablet for tablet 88343a7c45e54babbd7fcae9b6376094 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:25.247700 30861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88343a7c45e54babbd7fcae9b6376094. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.250833 30714 tablet_service.cc:1511] Processing CreateTablet for tablet e5be73f56e004d61ba8762d1afa0cc05 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:25.262400 30638 tablet_service.cc:1511] Processing CreateTablet for tablet c274ea8c29774701bdc1708a1839fa88 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:25.262432 30858 tablet_service.cc:1511] Processing CreateTablet for tablet e614ae4fde394b488924f185e2808b18 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:25.279942 30858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e614ae4fde394b488924f185e2808b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.268983 30901 heartbeater.cc:499] Master 127.11.11.190:37085 was elected leader, sending a full tablet report...
I20260709 12:20:25.274576 30707 tablet_service.cc:1511] Processing CreateTablet for tablet 4ac6fd1105644a1b820d76679057cbe4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:25.282112 30782 tablet_service.cc:1511] Processing CreateTablet for tablet c274ea8c29774701bdc1708a1839fa88 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:25.283722 30782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c274ea8c29774701bdc1708a1839fa88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.283725 30638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c274ea8c29774701bdc1708a1839fa88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.288511 30636 tablet_service.cc:1511] Processing CreateTablet for tablet a5c48ae242924bcab94096bb803d25d4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:25.272871 30784 tablet_service.cc:1511] Processing CreateTablet for tablet b2f67741397b40a0b491cddf4c7e7dea (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:25.311491 30640 tablet_service.cc:1511] Processing CreateTablet for tablet 66db71b232844f1794f53dba1cbd6198 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:25.245369 30788 tablet_service.cc:1511] Processing CreateTablet for tablet 7470dd49ce524f4aaf3faf7739fe5bd7 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:25.331174 30788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7470dd49ce524f4aaf3faf7739fe5bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.322685 30708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7470dd49ce524f4aaf3faf7739fe5bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.333827 30714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5be73f56e004d61ba8762d1afa0cc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.339349 30707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ac6fd1105644a1b820d76679057cbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.329479 30865 tablet_service.cc:1511] Processing CreateTablet for tablet 7470dd49ce524f4aaf3faf7739fe5bd7 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:25.343573 30865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7470dd49ce524f4aaf3faf7739fe5bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.344503 30859 tablet_service.cc:1511] Processing CreateTablet for tablet a5c48ae242924bcab94096bb803d25d4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:25.353809 30859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c48ae242924bcab94096bb803d25d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.357751 30789 tablet_service.cc:1511] Processing CreateTablet for tablet 2d65269b37cc420382672e7ceecd4789 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:25.359994 30789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d65269b37cc420382672e7ceecd4789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.349578 30857 tablet_service.cc:1511] Processing CreateTablet for tablet b9ed5ce7483340edbbac9dee8cca5f1d (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:25.352283 30921 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:25.362291 30921 ts_tablet_manager.cc:1403] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.397s	user 0.019s	sys 0.016s
I20260709 12:20:25.347417 30786 tablet_service.cc:1511] Processing CreateTablet for tablet 9873e94bae58472f8de439d5dc1835a6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:25.363497 30856 tablet_service.cc:1511] Processing CreateTablet for tablet 66db71b232844f1794f53dba1cbd6198 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:25.364341 30922 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:25.365038 30922 ts_tablet_manager.cc:1403] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.398s	user 0.029s	sys 0.006s
I20260709 12:20:25.365428 30856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66db71b232844f1794f53dba1cbd6198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.368795 30922 raft_consensus.cc:359] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:25.367319 30921 raft_consensus.cc:359] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:25.370615 30921 raft_consensus.cc:385] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.372138 30857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ed5ce7483340edbbac9dee8cca5f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.195298 30632 tablet_service.cc:1511] Processing CreateTablet for tablet e5899b18ce3a453ea77be2cdf20e543a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:25.281293 30783 tablet_service.cc:1511] Processing CreateTablet for tablet 2aafa16c29f04448945f5db4aa191223 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:25.381147 30786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9873e94bae58472f8de439d5dc1835a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.395475 30863 tablet_service.cc:1511] Processing CreateTablet for tablet e5899b18ce3a453ea77be2cdf20e543a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:25.399263 30863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5899b18ce3a453ea77be2cdf20e543a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.408914 30641 tablet_service.cc:1511] Processing CreateTablet for tablet b2f67741397b40a0b491cddf4c7e7dea (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:25.409816 30634 tablet_service.cc:1511] Processing CreateTablet for tablet 9873e94bae58472f8de439d5dc1835a6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:25.411243 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66db71b232844f1794f53dba1cbd6198. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.369616 30922 raft_consensus.cc:385] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.433128 30922 raft_consensus.cc:740] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.434136 30634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9873e94bae58472f8de439d5dc1835a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.434492 30922 consensus_queue.cc:260] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:25.379442 30921 raft_consensus.cc:740] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.413548 30637 tablet_service.cc:1511] Processing CreateTablet for tablet e5be73f56e004d61ba8762d1afa0cc05 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:25.439112 30637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5be73f56e004d61ba8762d1afa0cc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.440917 30641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f67741397b40a0b491cddf4c7e7dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.441650 30632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5899b18ce3a453ea77be2cdf20e543a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.418753 30923 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:25.442497 30923 ts_tablet_manager.cc:1403] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.473s	user 0.020s	sys 0.008s
I20260709 12:20:25.450608 30636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c48ae242924bcab94096bb803d25d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.458501 30711 tablet_service.cc:1511] Processing CreateTablet for tablet 6e5f378411424826909f1f1f3aff4409 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
W20260709 12:20:25.459982 30712 consensus_meta.cc:287] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.011s
I20260709 12:20:25.424602 30860 tablet_service.cc:1511] Processing CreateTablet for tablet b2f67741397b40a0b491cddf4c7e7dea (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:25.414732 30925 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:25.417943 30784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f67741397b40a0b491cddf4c7e7dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.465095 30783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aafa16c29f04448945f5db4aa191223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.482308 30862 tablet_service.cc:1511] Processing CreateTablet for tablet 2aafa16c29f04448945f5db4aa191223 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:25.484499 30923 raft_consensus.cc:359] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:25.485489 30923 raft_consensus.cc:385] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.460179 30711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5f378411424826909f1f1f3aff4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.461490 30860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f67741397b40a0b491cddf4c7e7dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.490061 30862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aafa16c29f04448945f5db4aa191223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.491567 30858 tablet_service.cc:1511] Processing CreateTablet for tablet e5be73f56e004d61ba8762d1afa0cc05 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:25.493669 30709 tablet_service.cc:1511] Processing CreateTablet for tablet 87c766be78f1467795131bc36e7766e9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:25.506218 30709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c766be78f1467795131bc36e7766e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.497038 30782 tablet_service.cc:1511] Processing CreateTablet for tablet 6e5f378411424826909f1f1f3aff4409 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:25.510983 30782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5f378411424826909f1f1f3aff4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.518328 30716 tablet_service.cc:1511] Processing CreateTablet for tablet c1243becf9c9421bb2d5083f71c1e292 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:25.520066 30716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1243becf9c9421bb2d5083f71c1e292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.522400 30635 tablet_service.cc:1511] Processing CreateTablet for tablet 2d65269b37cc420382672e7ceecd4789 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:25.523445 30713 tablet_service.cc:1511] Processing CreateTablet for tablet 5dd0a04062174fe1a633ff36362a7523 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:25.525209 30713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd0a04062174fe1a633ff36362a7523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.527551 30787 tablet_service.cc:1511] Processing CreateTablet for tablet 87c766be78f1467795131bc36e7766e9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:25.527983 30633 tablet_service.cc:1511] Processing CreateTablet for tablet c1243becf9c9421bb2d5083f71c1e292 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:25.529385 30787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c766be78f1467795131bc36e7766e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.485862 30923 raft_consensus.cc:740] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.530875 30861 tablet_service.cc:1511] Processing CreateTablet for tablet 2d65269b37cc420382672e7ceecd4789 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:25.436924 30921 consensus_queue.cc:260] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:25.532174 30858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5be73f56e004d61ba8762d1afa0cc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.532802 30785 tablet_service.cc:1511] Processing CreateTablet for tablet c95a27bd5694416aa5dae696482e8ec4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:25.557976 30923 consensus_queue.cc:260] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:25.524104 30635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d65269b37cc420382672e7ceecd4789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.542723 30859 tablet_service.cc:1511] Processing CreateTablet for tablet 4ac6fd1105644a1b820d76679057cbe4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:25.558477 30861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d65269b37cc420382672e7ceecd4789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.581255 30921 ts_tablet_manager.cc:1434] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.218s	user 0.015s	sys 0.005s
I20260709 12:20:25.585879 30640 tablet_service.cc:1511] Processing CreateTablet for tablet c95a27bd5694416aa5dae696482e8ec4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:25.596845 30639 tablet_service.cc:1511] Processing CreateTablet for tablet 6e5f378411424826909f1f1f3aff4409 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:25.606043 30859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ac6fd1105644a1b820d76679057cbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.606209 30925 tablet_bootstrap.cc:654] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.618897 30922 ts_tablet_manager.cc:1434] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.254s	user 0.008s	sys 0.012s
W20260709 12:20:25.619864 30827 tablet.cc:2406] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:25.620903 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e5f378411424826909f1f1f3aff4409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.621409 30785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95a27bd5694416aa5dae696482e8ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.632035 30922 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:25.632020 30790 tablet_service.cc:1511] Processing CreateTablet for tablet 5dd0a04062174fe1a633ff36362a7523 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:25.643318 30790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd0a04062174fe1a633ff36362a7523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.571696 30781 tablet_service.cc:1511] Processing CreateTablet for tablet 4ac6fd1105644a1b820d76679057cbe4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:25.647207 30781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ac6fd1105644a1b820d76679057cbe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.650254 30636 tablet_service.cc:1511] Processing CreateTablet for tablet 5dd0a04062174fe1a633ff36362a7523 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:25.652179 30636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd0a04062174fe1a633ff36362a7523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.661139 30862 tablet_service.cc:1511] Processing CreateTablet for tablet c95a27bd5694416aa5dae696482e8ec4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
W20260709 12:20:25.655197 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:25.654191 30865 tablet_service.cc:1511] Processing CreateTablet for tablet c1243becf9c9421bb2d5083f71c1e292 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:25.682250 30712 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34922 (request call id 4) took 1085 ms. Trace:
I20260709 12:20:25.653504 30922 tablet_bootstrap.cc:654] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.705974 30633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1243becf9c9421bb2d5083f71c1e292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.708693 30923 ts_tablet_manager.cc:1434] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.266s	user 0.033s	sys 0.079s
I20260709 12:20:25.713337 30712 rpcz_store.cc:276] 0709 12:20:24.596583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:24.596755 (+   172us) service_pool.cc:224] Handling call
0709 12:20:24.599154 (+  2399us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:20:24.599294 (+   140us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:20:24.859840 (+260546us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:25.578574 (+718734us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:25.682158 (+103584us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:25.715361 30921 tablet_bootstrap.cc:492] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:25.726580 30923 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:25.737253 30865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1243becf9c9421bb2d5083f71c1e292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.741744 30857 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59004 (request call id 15) took 1006 ms. Trace:
I20260709 12:20:25.742251 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95a27bd5694416aa5dae696482e8ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.756870 30711 tablet_service.cc:1511] Processing CreateTablet for tablet 61996ce5ca7a44f5aa688285da2e65ff (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:25.757895 30857 rpcz_store.cc:276] 0709 12:20:24.735475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:25.304923 (+569448us) service_pool.cc:224] Handling call
0709 12:20:25.362225 (+ 57302us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:20:25.362355 (+   130us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:20:25.565116 (+202761us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:25.719463 (+154347us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:25.741668 (+ 22205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:25.758572 30711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61996ce5ca7a44f5aa688285da2e65ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.744697 30864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59004 (request call id 10) took 1016 ms. Trace:
I20260709 12:20:25.759223 30864 rpcz_store.cc:276] 0709 12:20:24.728396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:25.193947 (+465551us) service_pool.cc:224] Handling call
0709 12:20:25.218750 (+ 24803us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:20:25.218881 (+   131us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:20:25.626650 (+407769us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:25.744132 (+117482us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:25.744627 (+   495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:25.746371 30856 tablet_service.cc:1511] Processing CreateTablet for tablet 87c766be78f1467795131bc36e7766e9 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20260709 12:20:25.710191 30533 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 52136.8us
I20260709 12:20:25.783231 30782 tablet_service.cc:1511] Processing CreateTablet for tablet 50776b4e5bc646a78173a6598eee54c2 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:25.785158 30782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50776b4e5bc646a78173a6598eee54c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.787482 30922 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:25.788582 30922 ts_tablet_manager.cc:1403] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.157s	user 0.013s	sys 0.008s
I20260709 12:20:25.792166 30922 raft_consensus.cc:359] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:25.792985 30922 raft_consensus.cc:385] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.765355 30862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95a27bd5694416aa5dae696482e8ec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.794122 30856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c766be78f1467795131bc36e7766e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.798130 30923 tablet_bootstrap.cc:654] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.801781 30922 raft_consensus.cc:740] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.802835 30863 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59004 (request call id 16) took 1067 ms. Trace:
I20260709 12:20:25.803937 30863 rpcz_store.cc:276] 0709 12:20:24.736241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:25.318427 (+582186us) service_pool.cc:224] Handling call
0709 12:20:25.398647 (+ 80220us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:20:25.398771 (+   124us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:20:25.626516 (+227745us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:25.762097 (+135581us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:25.802756 (+ 40659us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:25.805857 30708 tablet_service.cc:1511] Processing CreateTablet for tablet 6c8dd412d0624072a283e7390f9482b6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:25.803031 30922 consensus_queue.cc:260] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:25.811685 30922 ts_tablet_manager.cc:1434] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260709 12:20:25.812151 30784 tablet_service.cc:1511] Processing CreateTablet for tablet 6c8dd412d0624072a283e7390f9482b6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:25.813045 30922 tablet_bootstrap.cc:492] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:25.813987 30784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8dd412d0624072a283e7390f9482b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.818105 30708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8dd412d0624072a283e7390f9482b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.820351 30639 tablet_service.cc:1511] Processing CreateTablet for tablet 61996ce5ca7a44f5aa688285da2e65ff (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:25.821954 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61996ce5ca7a44f5aa688285da2e65ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.823096 30863 tablet_service.cc:1511] Processing CreateTablet for tablet 61996ce5ca7a44f5aa688285da2e65ff (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:25.824918 30863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61996ce5ca7a44f5aa688285da2e65ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.821281 30864 tablet_service.cc:1511] Processing CreateTablet for tablet 50776b4e5bc646a78173a6598eee54c2 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:25.829288 30864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50776b4e5bc646a78173a6598eee54c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.830120 30925 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:25.830726 30925 ts_tablet_manager.cc:1403] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.424s	user 0.021s	sys 0.003s
I20260709 12:20:25.834071 30929 raft_consensus.cc:493] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.834766 30929 raft_consensus.cc:515] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:25.835371 30713 tablet_service.cc:1511] Processing CreateTablet for tablet 1994c4ce566142fa922312f566387067 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:25.836926 30713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1994c4ce566142fa922312f566387067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.843560 30929 leader_election.cc:290] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:25.843545 30925 raft_consensus.cc:359] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:25.850330 30925 raft_consensus.cc:385] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.841648 30860 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59004 (request call id 19) took 1058 ms. Trace:
I20260709 12:20:25.875682 30922 tablet_bootstrap.cc:654] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.914117 30921 tablet_bootstrap.cc:700] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: Time spent opening tablet: real 0.101s	user 0.004s	sys 0.000s
I20260709 12:20:25.899881 30930 raft_consensus.cc:493] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.938216 30928 raft_consensus.cc:493] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.937986 30930 raft_consensus.cc:515] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:25.938779 30928 raft_consensus.cc:515] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:25.942621 30930 leader_election.cc:290] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:25.859179 30925 raft_consensus.cc:740] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.876209 30632 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33318 (request call id 17) took 1129 ms. Trace:
I20260709 12:20:25.861039 30783 tablet_service.cc:1511] Processing CreateTablet for tablet 1994c4ce566142fa922312f566387067 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:25.943423 30928 leader_election.cc:290] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:25.859565 30860 rpcz_store.cc:276] 0709 12:20:24.781148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:25.423543 (+642395us) service_pool.cc:224] Handling call
0709 12:20:25.460861 (+ 37318us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:20:25.460963 (+   102us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:20:25.630665 (+169702us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:25.733398 (+102733us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:25.839860 (+106462us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:25.962411 30921 tablet_bootstrap.cc:654] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.895467 30861 tablet_service.cc:1511] Processing CreateTablet for tablet 6c8dd412d0624072a283e7390f9482b6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:25.991734 30783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1994c4ce566142fa922312f566387067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.992926 30635 tablet_service.cc:1511] Processing CreateTablet for tablet 50776b4e5bc646a78173a6598eee54c2 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:26.000545 30635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50776b4e5bc646a78173a6598eee54c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.946422 30632 rpcz_store.cc:276] 0709 12:20:24.747258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:25.194105 (+446847us) service_pool.cc:224] Handling call
0709 12:20:25.413731 (+219626us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:20:25.413890 (+   159us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:20:25.744461 (+330571us) tablet_metadata.cc:665] Metadata flushed
0709 12:20:25.875383 (+130922us) tablet_replica.cc:206] Creating consensus instance
0709 12:20:25.876108 (+   725us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:25.991815 30861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c8dd412d0624072a283e7390f9482b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.018390 30709 tablet_service.cc:1511] Processing CreateTablet for tablet 770f228988ee4c8dbe5b6d9eb6313cc4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:26.020174 30709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770f228988ee4c8dbe5b6d9eb6313cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.976922 30925 consensus_queue.cc:260] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.035482 30637 tablet_service.cc:1511] Processing CreateTablet for tablet 1994c4ce566142fa922312f566387067 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:26.037333 30637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1994c4ce566142fa922312f566387067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.039606 30859 tablet_service.cc:1511] Processing CreateTablet for tablet 770f228988ee4c8dbe5b6d9eb6313cc4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:26.041332 30859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770f228988ee4c8dbe5b6d9eb6313cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.043707 30716 tablet_service.cc:1511] Processing CreateTablet for tablet e9041cf65147424a8c1ef0e1d03ab95a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:26.045368 30716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9041cf65147424a8c1ef0e1d03ab95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.048069 30928 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.053013 30928 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.055758 30928 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:26.071508 30923 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:26.071130 30865 tablet_service.cc:1511] Processing CreateTablet for tablet c349e457659c45079da022887088cc7b (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:26.072177 30923 ts_tablet_manager.cc:1403] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.346s	user 0.012s	sys 0.012s
I20260709 12:20:26.072888 30865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c349e457659c45079da022887088cc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.078853 30860 tablet_service.cc:1511] Processing CreateTablet for tablet 58bf19ba51b540ff8dcb14a3af23b8e3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:26.080510 30860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58bf19ba51b540ff8dcb14a3af23b8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.081990 30708 tablet_service.cc:1511] Processing CreateTablet for tablet 58bf19ba51b540ff8dcb14a3af23b8e3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:26.083142 30713 tablet_service.cc:1511] Processing CreateTablet for tablet 9632ab6e7d6d4ac4bdf266d64c523d15 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:26.083930 30708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58bf19ba51b540ff8dcb14a3af23b8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.088534 30922 tablet_bootstrap.cc:492] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:26.121857 30922 ts_tablet_manager.cc:1403] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.309s	user 0.011s	sys 0.008s
I20260709 12:20:26.126169 30712 tablet_service.cc:1511] Processing CreateTablet for tablet e0e79bd22cff4979ba995f0c04f48806 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:26.129834 30713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9632ab6e7d6d4ac4bdf266d64c523d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.136155 30712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e79bd22cff4979ba995f0c04f48806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.137336 30858 tablet_service.cc:1511] Processing CreateTablet for tablet fdae5560db3d45c180b6bed07a6b6197 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:26.139185 30858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdae5560db3d45c180b6bed07a6b6197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.142280 30922 raft_consensus.cc:359] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.143254 30922 raft_consensus.cc:385] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.143605 30922 raft_consensus.cc:740] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.144251 30923 raft_consensus.cc:359] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:26.145251 30923 raft_consensus.cc:385] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.150018 30922 consensus_queue.cc:260] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.158129 30857 tablet_service.cc:1511] Processing CreateTablet for tablet e0e79bd22cff4979ba995f0c04f48806 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:26.160238 30857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e79bd22cff4979ba995f0c04f48806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.162645 30863 tablet_service.cc:1511] Processing CreateTablet for tablet 9632ab6e7d6d4ac4bdf266d64c523d15 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:26.166996 30863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9632ab6e7d6d4ac4bdf266d64c523d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.155680 30923 raft_consensus.cc:740] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.172307 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:26.175536 30690 leader_election.cc:343] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.175881 30710 tablet_service.cc:1511] Processing CreateTablet for tablet 462bf14c43184e17b0f483d0e185993f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:26.190616 30925 ts_tablet_manager.cc:1434] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.360s	user 0.018s	sys 0.000s
I20260709 12:20:26.192508 30925 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:26.196631 30921 tablet_bootstrap.cc:492] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:26.197326 30921 ts_tablet_manager.cc:1403] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.492s	user 0.019s	sys 0.003s
I20260709 12:20:26.199950 30710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462bf14c43184e17b0f483d0e185993f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.198257 30639 tablet_service.cc:1511] Processing CreateTablet for tablet e9041cf65147424a8c1ef0e1d03ab95a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:26.203110 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9041cf65147424a8c1ef0e1d03ab95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.197064 30632 tablet_service.cc:1511] Processing CreateTablet for tablet c349e457659c45079da022887088cc7b (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:26.206805 30632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c349e457659c45079da022887088cc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.207811 30921 raft_consensus.cc:359] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.208631 30921 raft_consensus.cc:385] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.208953 30921 raft_consensus.cc:740] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.199983 30640 tablet_service.cc:1511] Processing CreateTablet for tablet 770f228988ee4c8dbe5b6d9eb6313cc4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:26.212734 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770f228988ee4c8dbe5b6d9eb6313cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.219594 30921 consensus_queue.cc:260] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.231192 30921 ts_tablet_manager.cc:1434] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.001s
I20260709 12:20:26.242679 30921 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:26.243600 30922 ts_tablet_manager.cc:1434] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.121s	user 0.006s	sys 0.003s
I20260709 12:20:26.244935 30922 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:26.228384 30923 consensus_queue.cc:260] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:26.250844 30921 tablet_bootstrap.cc:654] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.250842 30864 tablet_service.cc:1511] Processing CreateTablet for tablet 4f3e88175c47421db220066ad29f61c4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:26.252676 30923 ts_tablet_manager.cc:1434] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.180s	user 0.010s	sys 0.002s
I20260709 12:20:26.256572 30923 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:26.260231 30781 tablet_service.cc:1511] Processing CreateTablet for tablet c349e457659c45079da022887088cc7b (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:26.271597 30781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c349e457659c45079da022887088cc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.271943 30784 tablet_service.cc:1511] Processing CreateTablet for tablet 58bf19ba51b540ff8dcb14a3af23b8e3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:26.274278 30784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58bf19ba51b540ff8dcb14a3af23b8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.266168 30708 tablet_service.cc:1511] Processing CreateTablet for tablet 4f3e88175c47421db220066ad29f61c4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:26.252713 30864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3e88175c47421db220066ad29f61c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.286227 30789 tablet_service.cc:1511] Processing CreateTablet for tablet fdae5560db3d45c180b6bed07a6b6197 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:26.289371 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:26.303629 30789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdae5560db3d45c180b6bed07a6b6197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.303196 30856 tablet_service.cc:1511] Processing CreateTablet for tablet 8203be2a35f743e5ae45a1250189fb1f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:26.290452 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:26.288334 30637 tablet_service.cc:1511] Processing CreateTablet for tablet fdae5560db3d45c180b6bed07a6b6197 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:26.288426 30633 tablet_service.cc:1511] Processing CreateTablet for tablet 4f3e88175c47421db220066ad29f61c4 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:26.285983 30782 tablet_service.cc:1511] Processing CreateTablet for tablet e9041cf65147424a8c1ef0e1d03ab95a (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:26.308985 30782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9041cf65147424a8c1ef0e1d03ab95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.288295 30635 tablet_service.cc:1511] Processing CreateTablet for tablet 462bf14c43184e17b0f483d0e185993f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:26.287204 30787 tablet_service.cc:1511] Processing CreateTablet for tablet e0e79bd22cff4979ba995f0c04f48806 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:26.311254 30716 tablet_service.cc:1511] Processing CreateTablet for tablet 1cea447841c446fea6a32ff304879db6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:26.305037 30875 raft_consensus.cc:2468] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:26.283317 30708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3e88175c47421db220066ad29f61c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.281615 30925 tablet_bootstrap.cc:654] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.318543 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:26.318953 30787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e79bd22cff4979ba995f0c04f48806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.319967 30636 tablet_service.cc:1511] Processing CreateTablet for tablet 8203be2a35f743e5ae45a1250189fb1f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:26.305019 30856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8203be2a35f743e5ae45a1250189fb1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:26.322211 30615 leader_election.cc:343] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.318451 30637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdae5560db3d45c180b6bed07a6b6197. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:26.323067 30689 leader_election.cc:343] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.325462 30636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8203be2a35f743e5ae45a1250189fb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.325824 30689 leader_election.cc:304] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:26.326470 30633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3e88175c47421db220066ad29f61c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.326237 30785 tablet_service.cc:1511] Processing CreateTablet for tablet 462bf14c43184e17b0f483d0e185993f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:26.327121 30929 raft_consensus.cc:2749] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:26.330422 30635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462bf14c43184e17b0f483d0e185993f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.334157 30785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462bf14c43184e17b0f483d0e185993f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.336566 30614 leader_election.cc:304] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:26.338410 30930 raft_consensus.cc:2804] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:26.339098 30930 raft_consensus.cc:493] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.340122 30930 raft_consensus.cc:3060] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.339241 30716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cea447841c446fea6a32ff304879db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.345012 30790 tablet_service.cc:1511] Processing CreateTablet for tablet 8203be2a35f743e5ae45a1250189fb1f (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:26.346712 30786 tablet_service.cc:1511] Processing CreateTablet for tablet 1cea447841c446fea6a32ff304879db6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:26.350206 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:26.354400 30790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8203be2a35f743e5ae45a1250189fb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.350647 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:26.353577 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:26.293319 30922 tablet_bootstrap.cc:654] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.364410 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:26.367362 30786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cea447841c446fea6a32ff304879db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.364265 30640 tablet_service.cc:1511] Processing CreateTablet for tablet 9632ab6e7d6d4ac4bdf266d64c523d15 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:26.377949 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9632ab6e7d6d4ac4bdf266d64c523d15. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:26.384054 30763 leader_election.cc:343] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:26.385244 30763 leader_election.cc:343] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:26.390282 30763 leader_election.cc:343] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:26.385893 30533 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 63751.8us
I20260709 12:20:26.422081 30763 leader_election.cc:304] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:26.423300 30928 raft_consensus.cc:2749] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:26.425001 30763 leader_election.cc:343] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.425697 30763 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
W20260709 12:20:26.385784 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 49581.1us
I20260709 12:20:26.408016 30925 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:26.450314 30634 tablet_service.cc:1511] Processing CreateTablet for tablet 7f25f8eeb23d4d5aa15a052f827302ad (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:26.451069 30925 ts_tablet_manager.cc:1403] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.259s	user 0.014s	sys 0.006s
I20260709 12:20:26.452250 30634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f25f8eeb23d4d5aa15a052f827302ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.415520 30709 tablet_service.cc:1511] Processing CreateTablet for tablet 7f25f8eeb23d4d5aa15a052f827302ad (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:26.453380 30639 tablet_service.cc:1511] Processing CreateTablet for tablet 773dfc99de304153af80953c364a815e (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:26.454573 30709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f25f8eeb23d4d5aa15a052f827302ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.454746 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773dfc99de304153af80953c364a815e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.415818 30712 tablet_service.cc:1511] Processing CreateTablet for tablet 773dfc99de304153af80953c364a815e (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:26.462199 30929 raft_consensus.cc:493] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.462791 30712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773dfc99de304153af80953c364a815e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.462800 30929 raft_consensus.cc:515] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.418298 30923 tablet_bootstrap.cc:654] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.402608 30921 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:26.425994 30858 tablet_service.cc:1511] Processing CreateTablet for tablet 7f25f8eeb23d4d5aa15a052f827302ad (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:26.426767 30860 tablet_service.cc:1511] Processing CreateTablet for tablet 1cea447841c446fea6a32ff304879db6 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:26.474277 30858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f25f8eeb23d4d5aa15a052f827302ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.441594 30922 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:26.475467 30921 ts_tablet_manager.cc:1403] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.233s	user 0.026s	sys 0.026s
I20260709 12:20:26.475963 30922 ts_tablet_manager.cc:1403] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.231s	user 0.016s	sys 0.000s
I20260709 12:20:26.477818 30929 leader_election.cc:290] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:26.441924 30930 raft_consensus.cc:515] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.467689 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:26.478785 30800 raft_consensus.cc:2468] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:26.480517 30690 leader_election.cc:304] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:26.483249 30860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cea447841c446fea6a32ff304879db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.484212 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:26.484965 30875 raft_consensus.cc:3060] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.485906 30929 raft_consensus.cc:2804] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:26.486428 30929 raft_consensus.cc:493] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.486766 30929 raft_consensus.cc:3060] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.490345 30930 leader_election.cc:290] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:26.491113 30930 raft_consensus.cc:493] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.491631 30930 raft_consensus.cc:515] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:26.492079 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
W20260709 12:20:26.493834 30615 leader_election.cc:343] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.489043 30925 raft_consensus.cc:359] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.497893 30925 raft_consensus.cc:385] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.498342 30925 raft_consensus.cc:740] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.499341 30925 consensus_queue.cc:260] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.510993 30948 raft_consensus.cc:493] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.511610 30948 raft_consensus.cc:515] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.513789 30928 raft_consensus.cc:2749] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:26.514075 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 89115.6us
I20260709 12:20:26.540073 30921 raft_consensus.cc:359] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:26.541136 30921 raft_consensus.cc:385] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.546092 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:26.541492 30921 raft_consensus.cc:740] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.550454 30921 consensus_queue.cc:260] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:26.516938 30948 leader_election.cc:290] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:26.550923 30615 leader_election.cc:343] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.550477 30930 leader_election.cc:290] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:26.556535 30929 raft_consensus.cc:515] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.560390 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:26.561265 30800 raft_consensus.cc:3060] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.562711 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:26.565498 30689 leader_election.cc:343] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.574147 30875 raft_consensus.cc:2468] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:26.608476 30856 tablet_service.cc:1511] Processing CreateTablet for tablet 2c9cd28a99d641698ab0d99e20bac569 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:26.611192 30614 leader_election.cc:304] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:26.611196 30856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9cd28a99d641698ab0d99e20bac569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.613897 30930 raft_consensus.cc:2804] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:26.614790 30930 raft_consensus.cc:697] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:26.616209 30930 consensus_queue.cc:237] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.619736 30781 tablet_service.cc:1511] Processing CreateTablet for tablet 2c9cd28a99d641698ab0d99e20bac569 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:26.621515 30781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9cd28a99d641698ab0d99e20bac569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.623347 30633 tablet_service.cc:1511] Processing CreateTablet for tablet 2c9cd28a99d641698ab0d99e20bac569 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:26.625905 30709 tablet_service.cc:1511] Processing CreateTablet for tablet b29c912615fb4627ba5c12d4c0f193d3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:26.628883 30784 tablet_service.cc:1511] Processing CreateTablet for tablet 773dfc99de304153af80953c364a815e (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:26.630597 30784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 773dfc99de304153af80953c364a815e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.631565 30864 tablet_service.cc:1511] Processing CreateTablet for tablet b29c912615fb4627ba5c12d4c0f193d3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:26.585549 30925 ts_tablet_manager.cc:1434] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.134s	user 0.010s	sys 0.000s
I20260709 12:20:26.633927 30709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29c912615fb4627ba5c12d4c0f193d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.648613 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:26.650823 30864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29c912615fb4627ba5c12d4c0f193d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:26.654512 30689 leader_election.cc:343] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.639592 30929 leader_election.cc:290] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:26.651152 30800 raft_consensus.cc:2468] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:26.664505 30690 leader_election.cc:304] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:26.643471 30922 raft_consensus.cc:359] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.665935 30922 raft_consensus.cc:385] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.666347 30922 raft_consensus.cc:740] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.667102 30925 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:26.666561 30634 tablet_service.cc:1511] Processing CreateTablet for tablet 56f37d27937c43f39c083e36508283fd (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:26.667714 30923 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:26.668354 30923 ts_tablet_manager.cc:1403] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.412s	user 0.053s	sys 0.033s
I20260709 12:20:26.664328 30787 tablet_service.cc:1511] Processing CreateTablet for tablet b29c912615fb4627ba5c12d4c0f193d3 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:26.674492 30787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29c912615fb4627ba5c12d4c0f193d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.680567 30921 ts_tablet_manager.cc:1434] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.205s	user 0.017s	sys 0.010s
I20260709 12:20:26.683219 30633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9cd28a99d641698ab0d99e20bac569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.690743 30634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f37d27937c43f39c083e36508283fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.693799 30923 raft_consensus.cc:359] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.699931 30922 consensus_queue.cc:260] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.703287 30922 ts_tablet_manager.cc:1434] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.227s	user 0.010s	sys 0.000s
I20260709 12:20:26.700920 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:26.704569 30712 tablet_service.cc:1511] Processing CreateTablet for tablet 56f37d27937c43f39c083e36508283fd (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:26.723215 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:26.726598 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:26.728672 30549 catalog_manager.cc:5697] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.704823 30789 tablet_service.cc:1511] Processing CreateTablet for tablet 56f37d27937c43f39c083e36508283fd (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:26.748405 30789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f37d27937c43f39c083e36508283fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.733793 30925 tablet_bootstrap.cc:654] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.751279 30921 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:26.695617 30923 raft_consensus.cc:385] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.730401 30858 tablet_service.cc:1511] Processing CreateTablet for tablet 35a132d43ba3447a916ccdfcc6e66c26 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:26.753716 30858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a132d43ba3447a916ccdfcc6e66c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.728142 30712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f37d27937c43f39c083e36508283fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.730970 30640 tablet_service.cc:1511] Processing CreateTablet for tablet 35a132d43ba3447a916ccdfcc6e66c26 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:26.677487 30929 raft_consensus.cc:2804] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:26.761839 30929 raft_consensus.cc:697] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
W20260709 12:20:26.733572 30615 leader_election.cc:343] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.731036 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:26.762784 30615 leader_election.cc:304] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:26.763027 30800 raft_consensus.cc:2468] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:26.732592 30716 tablet_service.cc:1511] Processing CreateTablet for tablet 35a132d43ba3447a916ccdfcc6e66c26 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:26.764315 30930 raft_consensus.cc:2749] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:26.764667 30839 leader_election.cc:304] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:26.764796 30716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a132d43ba3447a916ccdfcc6e66c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.765537 30923 raft_consensus.cc:740] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.765712 30948 raft_consensus.cc:2804] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:26.766724 30948 raft_consensus.cc:493] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.767184 30948 raft_consensus.cc:3060] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.765641 30786 tablet_service.cc:1511] Processing CreateTablet for tablet 4a01bc7b51f24fb5b74aed9349c72e91 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:26.763237 30929 consensus_queue.cc:237] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:26.768702 30786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a01bc7b51f24fb5b74aed9349c72e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.768076 30923 consensus_queue.cc:260] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.762112 30640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a132d43ba3447a916ccdfcc6e66c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.739444 30922 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:26.778131 30650 raft_consensus.cc:2468] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:26.784384 30923 ts_tablet_manager.cc:1434] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.116s	user 0.009s	sys 0.000s
I20260709 12:20:26.787698 30860 tablet_service.cc:1511] Processing CreateTablet for tablet 4a01bc7b51f24fb5b74aed9349c72e91 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:26.792712 30639 tablet_service.cc:1511] Processing CreateTablet for tablet 4a01bc7b51f24fb5b74aed9349c72e91 (DEFAULT_TABLE table=test-workload [id=41e7a00d7cec4439a72574faba3c2e05]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:26.799487 30860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a01bc7b51f24fb5b74aed9349c72e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.791051 30948 raft_consensus.cc:515] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.802618 30925 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:26.803074 30923 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:26.804171 30639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a01bc7b51f24fb5b74aed9349c72e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.805967 30928 raft_consensus.cc:493] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.822211 30928 raft_consensus.cc:515] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.822861 30925 ts_tablet_manager.cc:1403] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.156s	user 0.010s	sys 0.012s
I20260709 12:20:26.826270 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61d416be13d449318d3e6f13a14a5d4a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:26.828629 30763 leader_election.cc:343] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.828368 30925 raft_consensus.cc:359] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.829293 30925 raft_consensus.cc:385] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.829849 30928 leader_election.cc:290] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:26.829859 30925 raft_consensus.cc:740] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.840885 30925 consensus_queue.cc:260] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.857914 30921 tablet_bootstrap.cc:654] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.860594 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:26.861414 30800 raft_consensus.cc:3060] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.875748 30923 tablet_bootstrap.cc:654] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.883019 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:26.884068 30650 raft_consensus.cc:3060] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:26.885903 30922 tablet_bootstrap.cc:654] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.898459 30965 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.899227 30965 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.877290 30948 leader_election.cc:290] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:26.922400 30925 ts_tablet_manager.cc:1434] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.099s	user 0.010s	sys 0.004s
I20260709 12:20:26.924916 30965 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:26.925982 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:26.925987 30929 raft_consensus.cc:493] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.928449 30929 raft_consensus.cc:515] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:26.929136 30838 leader_election.cc:343] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.931306 30929 leader_election.cc:290] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:26.933002 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:26.935035 30690 leader_election.cc:343] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.924479 30925 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:26.948700 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61d416be13d449318d3e6f13a14a5d4a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:26.971027 30800 raft_consensus.cc:2468] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:26.976529 30839 leader_election.cc:304] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:26.977731 30965 raft_consensus.cc:2804] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:26.978302 30965 raft_consensus.cc:697] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:26.979167 30965 consensus_queue.cc:237] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:26.980718 30921 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:26.981986 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:26.982049 30764 leader_election.cc:343] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.982970 30764 leader_election.cc:304] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
W20260709 12:20:26.984045 30689 leader_election.cc:343] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:26.984352 30928 raft_consensus.cc:2749] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:26.984642 30689 leader_election.cc:304] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:26.981457 30921 ts_tablet_manager.cc:1403] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.300s	user 0.006s	sys 0.016s
I20260709 12:20:26.986411 30929 raft_consensus.cc:2749] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:26.990160 30921 raft_consensus.cc:359] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.991218 30921 raft_consensus.cc:385] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:26.991714 30921 raft_consensus.cc:740] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:26.991562 30546 catalog_manager.cc:5697] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.993227 30921 consensus_queue.cc:260] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:26.996766 30925 tablet_bootstrap.cc:654] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:26.997942 30921 ts_tablet_manager.cc:1434] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:20:26.999539 30921 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.005335 30922 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:27.005993 30922 ts_tablet_manager.cc:1403] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.267s	user 0.013s	sys 0.015s
I20260709 12:20:26.957229 30650 raft_consensus.cc:2468] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:27.008208 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:26.980715 30923 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
W20260709 12:20:27.009982 30839 leader_election.cc:343] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.011060 30923 ts_tablet_manager.cc:1403] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.208s	user 0.014s	sys 0.033s
I20260709 12:20:27.011166 30839 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:27.015408 30922 raft_consensus.cc:359] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.017848 30922 raft_consensus.cc:385] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.018251 30922 raft_consensus.cc:740] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.019189 30922 consensus_queue.cc:260] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.019806 30921 tablet_bootstrap.cc:654] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.020291 30965 raft_consensus.cc:2749] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.024183 30922 ts_tablet_manager.cc:1434] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.006s
I20260709 12:20:27.024085 30548 catalog_manager.cc:5697] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.031235 30930 consensus_queue.cc:1048] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:20:27.030777 30923 raft_consensus.cc:359] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.035557 30923 raft_consensus.cc:385] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.035890 30923 raft_consensus.cc:740] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.036813 30923 consensus_queue.cc:260] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.031659 30922 tablet_bootstrap.cc:492] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:27.041067 30923 ts_tablet_manager.cc:1434] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.029s	user 0.002s	sys 0.006s
I20260709 12:20:27.042450 30923 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.037320 30925 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.053759 30925 ts_tablet_manager.cc:1403] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.130s	user 0.013s	sys 0.011s
I20260709 12:20:27.059655 30923 tablet_bootstrap.cc:654] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.076411 30922 tablet_bootstrap.cc:654] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.080842 30925 raft_consensus.cc:359] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.081875 30925 raft_consensus.cc:385] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:20:27.086328 30615 consensus_peers.cc:597] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:27.088006 30921 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.088852 30925 raft_consensus.cc:740] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.089412 30921 ts_tablet_manager.cc:1403] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.090s	user 0.022s	sys 0.029s
I20260709 12:20:27.090010 30925 consensus_queue.cc:260] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.093602 30925 ts_tablet_manager.cc:1434] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.038s	user 0.012s	sys 0.000s
I20260709 12:20:27.093923 30921 raft_consensus.cc:359] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.094841 30921 raft_consensus.cc:385] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.095016 30925 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.095134 30921 raft_consensus.cc:740] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.096424 30923 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.096160 30921 consensus_queue.cc:260] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.097103 30923 ts_tablet_manager.cc:1403] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.055s	user 0.018s	sys 0.009s
I20260709 12:20:27.098984 30921 ts_tablet_manager.cc:1434] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:20:27.100190 30921 tablet_bootstrap.cc:492] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.102615 30923 raft_consensus.cc:359] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.103534 30923 raft_consensus.cc:385] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.104001 30923 raft_consensus.cc:740] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.104228 30925 tablet_bootstrap.cc:654] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.105711 30923 consensus_queue.cc:260] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.108139 30921 tablet_bootstrap.cc:654] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.108724 30923 ts_tablet_manager.cc:1434] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:20:27.110088 30923 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.111760 30922 tablet_bootstrap.cc:492] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:27.113179 30922 ts_tablet_manager.cc:1403] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.081s	user 0.012s	sys 0.006s
I20260709 12:20:27.119499 30952 raft_consensus.cc:493] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.120313 30952 raft_consensus.cc:515] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.123507 30923 tablet_bootstrap.cc:654] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.126295 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.126832 30922 raft_consensus.cc:359] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.127723 30922 raft_consensus.cc:385] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.128078 30922 raft_consensus.cc:740] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
W20260709 12:20:27.129323 30615 leader_election.cc:343] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.130057 30922 consensus_queue.cc:260] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.132797 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:27.138461 30614 leader_election.cc:343] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.139123 30614 leader_election.cc:304] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:27.140900 30952 leader_election.cc:290] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:27.148780 30922 ts_tablet_manager.cc:1434] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.003s
I20260709 12:20:27.152196 30925 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.154024 30952 raft_consensus.cc:2749] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.154346 30925 ts_tablet_manager.cc:1403] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.060s	user 0.015s	sys 0.003s
I20260709 12:20:27.159731 30925 raft_consensus.cc:359] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.161502 30928 raft_consensus.cc:493] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.162439 30928 raft_consensus.cc:515] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.167388 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:27.170266 30764 leader_election.cc:343] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.174011 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:27.176242 30763 leader_election.cc:343] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.176949 30763 leader_election.cc:304] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:27.178591 30928 leader_election.cc:290] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:27.179572 30928 raft_consensus.cc:2749] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.190286 30922 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:27.195945 30928 raft_consensus.cc:493] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.196704 30928 raft_consensus.cc:515] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.198613 30921 tablet_bootstrap.cc:492] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.199610 30921 ts_tablet_manager.cc:1403] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.100s	user 0.022s	sys 0.059s
I20260709 12:20:27.203169 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.204489 30726 raft_consensus.cc:2468] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:27.205271 30921 raft_consensus.cc:359] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.206974 30921 raft_consensus.cc:385] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.206718 30764 leader_election.cc:304] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.207599 30921 raft_consensus.cc:740] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.209144 30928 leader_election.cc:290] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:27.209445 30921 consensus_queue.cc:260] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.213725 30929 raft_consensus.cc:493] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.214181 30923 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.215021 30923 ts_tablet_manager.cc:1403] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.105s	user 0.057s	sys 0.041s
I20260709 12:20:27.214634 30929 raft_consensus.cc:515] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.219817 30929 leader_election.cc:290] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.220206 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61d416be13d449318d3e6f13a14a5d4a" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.219980 30923 raft_consensus.cc:359] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.220731 30921 ts_tablet_manager.cc:1434] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.021s	user 0.013s	sys 0.000s
I20260709 12:20:27.221132 30923 raft_consensus.cc:385] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.221554 30923 raft_consensus.cc:740] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.222157 30921 tablet_bootstrap.cc:492] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.222664 30923 consensus_queue.cc:260] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.223819 30650 raft_consensus.cc:2468] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:27.226142 30689 leader_election.cc:304] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.227263 30929 raft_consensus.cc:2804] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.227751 30929 raft_consensus.cc:493] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.228163 30929 raft_consensus.cc:3060] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.229727 30921 tablet_bootstrap.cc:654] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.235003 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61d416be13d449318d3e6f13a14a5d4a" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.236019 30800 raft_consensus.cc:2468] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
W20260709 12:20:27.236598 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 60400.8us
I20260709 12:20:27.238685 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.239195 30923 ts_tablet_manager.cc:1434] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.024s	user 0.000s	sys 0.008s
W20260709 12:20:27.240479 30763 leader_election.cc:343] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.212975 30928 raft_consensus.cc:2804] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.242926 30923 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.244004 30921 tablet_bootstrap.cc:492] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.244971 30921 ts_tablet_manager.cc:1403] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.011s
I20260709 12:20:27.241961 30928 raft_consensus.cc:493] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.247548 30928 raft_consensus.cc:3060] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.250358 30921 raft_consensus.cc:359] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.251276 30921 raft_consensus.cc:385] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.251623 30921 raft_consensus.cc:740] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.253237 30929 raft_consensus.cc:515] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:27.255090 30689 consensus_peers.cc:597] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:27.256083 30923 tablet_bootstrap.cc:654] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.256780 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61d416be13d449318d3e6f13a14a5d4a" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.257601 30800 raft_consensus.cc:3060] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.160780 30925 raft_consensus.cc:385] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.260012 30925 raft_consensus.cc:740] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.259642 30928 raft_consensus.cc:515] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.261193 30925 consensus_queue.cc:260] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.263377 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
W20260709 12:20:27.265204 30763 leader_election.cc:343] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.267665 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:27.268411 30726 raft_consensus.cc:3060] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.269537 30929 leader_election.cc:290] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.270644 30800 raft_consensus.cc:2468] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:27.271636 30925 ts_tablet_manager.cc:1434] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.117s	user 0.010s	sys 0.000s
I20260709 12:20:27.272248 30923 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.271869 30921 consensus_queue.cc:260] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.272902 30923 ts_tablet_manager.cc:1403] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.030s	user 0.016s	sys 0.005s
I20260709 12:20:27.273033 30925 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.275084 30921 ts_tablet_manager.cc:1434] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260709 12:20:27.276580 30921 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.276985 30923 raft_consensus.cc:359] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.277271 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61d416be13d449318d3e6f13a14a5d4a" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:27.278149 30650 raft_consensus.cc:3060] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.279048 30922 tablet_bootstrap.cc:654] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.284868 30921 tablet_bootstrap.cc:654] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.285830 30923 raft_consensus.cc:385] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.286268 30923 raft_consensus.cc:740] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.290059 30923 consensus_queue.cc:260] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.292774 30925 tablet_bootstrap.cc:654] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.293540 30965 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.294273 30965 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.297243 30929 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.298045 30929 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.298384 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.298583 30965 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:27.299639 30923 ts_tablet_manager.cc:1434] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.002s
I20260709 12:20:27.288651 30650 raft_consensus.cc:2468] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:27.300345 30929 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.301322 30923 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.304442 30689 leader_election.cc:304] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.304677 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:27.305506 30875 raft_consensus.cc:2468] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
W20260709 12:20:27.301229 30838 leader_election.cc:343] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.306903 30921 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.307078 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.307461 30921 ts_tablet_manager.cc:1403] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.031s	user 0.017s	sys 0.004s
I20260709 12:20:27.300509 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:27.308812 30689 leader_election.cc:343] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.309162 30725 raft_consensus.cc:2468] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:27.310765 30923 tablet_bootstrap.cc:654] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.310806 30921 raft_consensus.cc:359] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.311995 30921 raft_consensus.cc:385] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.312463 30921 raft_consensus.cc:740] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.314020 30929 raft_consensus.cc:2804] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.313879 30921 consensus_queue.cc:260] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.315119 30689 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:27.317337 30957 raft_consensus.cc:2804] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.318190 30957 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.318815 30957 raft_consensus.cc:3060] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.324061 30921 ts_tablet_manager.cc:1434] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.006s
I20260709 12:20:27.324779 30925 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.325567 30925 ts_tablet_manager.cc:1403] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.053s	user 0.010s	sys 0.004s
I20260709 12:20:27.325443 30839 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:27.326440 30965 raft_consensus.cc:2804] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.326844 30965 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.327172 30965 raft_consensus.cc:3060] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.315744 30929 raft_consensus.cc:697] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:27.335994 30965 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.337121 30991 consensus_queue.cc:1048] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:20:27.342175 30965 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:27.343106 30930 raft_consensus.cc:493] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.343545 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:27.343689 30930 raft_consensus.cc:515] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.336838 30921 tablet_bootstrap.cc:492] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.346524 30930 leader_election.cc:290] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
W20260709 12:20:27.346916 30838 leader_election.cc:343] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.348423 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:27.348665 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.331364 30929 consensus_queue.cc:237] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:27.350641 30615 leader_election.cc:343] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.352799 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.354293 30724 raft_consensus.cc:2468] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:27.354847 30921 tablet_bootstrap.cc:654] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.356441 30615 leader_election.cc:304] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:27.329559 30925 raft_consensus.cc:359] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.358299 30925 raft_consensus.cc:385] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.358254 30930 raft_consensus.cc:2804] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.359321 30925 raft_consensus.cc:740] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.359417 30930 raft_consensus.cc:493] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.359901 30930 raft_consensus.cc:3060] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.360479 30925 consensus_queue.cc:260] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.363409 30925 ts_tablet_manager.cc:1434] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.004s
I20260709 12:20:27.364666 30925 tablet_bootstrap.cc:492] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.371033 30930 raft_consensus.cc:515] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.373440 30925 tablet_bootstrap.cc:654] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.375941 30921 tablet_bootstrap.cc:492] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.376060 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:27.376600 30921 ts_tablet_manager.cc:1403] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.040s	user 0.009s	sys 0.004s
I20260709 12:20:27.376730 30724 raft_consensus.cc:3060] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.380108 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.380300 30928 leader_election.cc:290] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:27.383502 30615 leader_election.cc:343] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.380514 30921 raft_consensus.cc:359] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.384336 30921 raft_consensus.cc:385] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.384651 30921 raft_consensus.cc:740] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.385440 30921 consensus_queue.cc:260] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.387636 30965 consensus_queue.cc:1048] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:27.388047 30921 ts_tablet_manager.cc:1434] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:20:27.389181 30921 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.393049 30930 leader_election.cc:290] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.399796 30925 tablet_bootstrap.cc:492] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.400434 30925 ts_tablet_manager.cc:1403] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.036s	user 0.006s	sys 0.008s
I20260709 12:20:27.403801 30925 raft_consensus.cc:359] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.404387 30925 raft_consensus.cc:385] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.404732 30925 raft_consensus.cc:740] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.405877 30921 tablet_bootstrap.cc:654] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.405589 30925 consensus_queue.cc:260] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.408489 30925 ts_tablet_manager.cc:1434] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:27.409176 30965 raft_consensus.cc:493] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.409881 30925 tablet_bootstrap.cc:492] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.410264 30965 raft_consensus.cc:515] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.415207 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.415453 30957 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.417531 30725 raft_consensus.cc:2393] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:27.417863 30925 tablet_bootstrap.cc:654] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.422699 30726 raft_consensus.cc:2468] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:27.424690 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:27.425592 30875 raft_consensus.cc:2393] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb6634546938405dac9eabbadd34d2b1 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:27.427764 30839 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:27.438522 30957 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.439869 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:27.452745 30923 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.454284 30923 ts_tablet_manager.cc:1403] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.153s	user 0.023s	sys 0.035s
I20260709 12:20:27.457197 31003 raft_consensus.cc:2749] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.458403 30923 raft_consensus.cc:359] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.459517 30923 raft_consensus.cc:385] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.459887 30923 raft_consensus.cc:740] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.428798 30764 leader_election.cc:304] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:27.443236 30724 raft_consensus.cc:2468] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:27.446247 30957 raft_consensus.cc:493] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.471043 30921 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.472018 30921 ts_tablet_manager.cc:1403] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.083s	user 0.009s	sys 0.022s
I20260709 12:20:27.463531 30965 leader_election.cc:290] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
W20260709 12:20:27.475965 30764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 47262.5us
I20260709 12:20:27.468369 30929 raft_consensus.cc:493] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.469389 30928 raft_consensus.cc:2804] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:20:27.467449 30689 leader_election.cc:343] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.467694 30957 raft_consensus.cc:515] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:27.477236 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 48097.7us
I20260709 12:20:27.475674 30615 leader_election.cc:304] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:27.481930 30928 raft_consensus.cc:697] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:27.486872 30930 raft_consensus.cc:2804] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.488030 30930 raft_consensus.cc:697] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:27.489295 30930 consensus_queue.cc:237] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.489832 30921 raft_consensus.cc:359] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.490658 30921 raft_consensus.cc:385] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.490949 30921 raft_consensus.cc:740] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.491647 30921 consensus_queue.cc:260] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.486768 30928 consensus_queue.cc:237] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.478124 30929 raft_consensus.cc:515] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.499018 30957 leader_election.cc:290] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:27.480118 30689 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:27.501009 30798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.502739 30922 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:27.502974 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:27.503366 30922 ts_tablet_manager.cc:1403] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.313s	user 0.024s	sys 0.051s
I20260709 12:20:27.504690 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14747c6d552943dbac8ab19f68816ea9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.505877 30957 raft_consensus.cc:2749] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:20:27.506842 30690 leader_election.cc:343] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.506482 30929 leader_election.cc:290] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
W20260709 12:20:27.507706 30690 leader_election.cc:343] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.509462 30923 consensus_queue.cc:260] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:27.518407 30689 leader_election.cc:343] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.520036 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14747c6d552943dbac8ab19f68816ea9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:27.522056 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.520167 30689 leader_election.cc:304] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
W20260709 12:20:27.524945 30689 leader_election.cc:343] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.525434 30839 leader_election.cc:304] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:27.526749 31000 consensus_queue.cc:1048] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:27.527740 30998 raft_consensus.cc:2749] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.525936 30929 raft_consensus.cc:2749] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.522323 30921 ts_tablet_manager.cc:1434] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.050s	user 0.006s	sys 0.001s
I20260709 12:20:27.528718 30983 raft_consensus.cc:493] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.529392 30921 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.529326 30983 raft_consensus.cc:515] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.525454 30689 leader_election.cc:304] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:27.535274 30923 ts_tablet_manager.cc:1434] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.081s	user 0.011s	sys 0.003s
I20260709 12:20:27.536355 30923 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.541224 30925 tablet_bootstrap.cc:492] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.542279 30925 ts_tablet_manager.cc:1403] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.133s	user 0.009s	sys 0.010s
I20260709 12:20:27.550712 30952 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.536069 30929 raft_consensus.cc:2749] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.554458 30724 raft_consensus.cc:3060] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.554282 30952 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.555991 30549 catalog_manager.cc:5697] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:27.558867 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:27.559590 30875 raft_consensus.cc:2468] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:27.561085 30614 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:27.562526 30952 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:27.562083 30548 catalog_manager.cc:5697] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:27.563129 30983 leader_election.cc:290] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.563244 30952 raft_consensus.cc:2804] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.564036 30952 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.564441 30952 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.554230 30922 raft_consensus.cc:359] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.568337 30922 raft_consensus.cc:385] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.568666 30922 raft_consensus.cc:740] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.570027 30921 tablet_bootstrap.cc:654] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.569720 30922 consensus_queue.cc:260] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.573405 30923 tablet_bootstrap.cc:654] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.575918 30952 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.566840 30925 raft_consensus.cc:359] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.586228 30925 raft_consensus.cc:385] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.586539 30925 raft_consensus.cc:740] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.587088 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.587387 30925 consensus_queue.cc:260] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.578583 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4d076571a8e4367836518e952de6f11" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.590413 30726 raft_consensus.cc:2393] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:27.574528 30922 ts_tablet_manager.cc:1434] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.071s	user 0.004s	sys 0.006s
I20260709 12:20:27.591881 30925 ts_tablet_manager.cc:1434] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.049s	user 0.005s	sys 0.000s
I20260709 12:20:27.584905 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.592412 30922 tablet_bootstrap.cc:492] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:27.581981 30546 catalog_manager.cc:5697] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.596036 30615 leader_election.cc:304] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:27.593559 30998 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.597577 31009 raft_consensus.cc:3060] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.597743 30998 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.601423 30929 raft_consensus.cc:493] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.602281 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.603605 30922 tablet_bootstrap.cc:654] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:27.604077 30838 leader_election.cc:343] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.602183 30929 raft_consensus.cc:515] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.604876 30998 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.595050 30925 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.607790 30929 leader_election.cc:290] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:27.608696 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.609771 30921 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.610198 30800 raft_consensus.cc:2468] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:27.610492 30921 ts_tablet_manager.cc:1403] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.081s	user 0.013s	sys 0.009s
I20260709 12:20:27.611883 30690 leader_election.cc:304] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.609035 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:27.613806 30929 raft_consensus.cc:2804] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.614259 30929 raft_consensus.cc:493] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:27.614308 30689 leader_election.cc:343] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.614850 30929 raft_consensus.cc:3060] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.614382 30925 tablet_bootstrap.cc:654] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.616750 30921 raft_consensus.cc:359] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.617540 30921 raft_consensus.cc:385] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.617834 30921 raft_consensus.cc:740] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.618829 30921 consensus_queue.cc:260] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:27.619480 30615 leader_election.cc:343] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.627151 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.630194 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:27.631245 30875 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.631138 30929 raft_consensus.cc:515] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.634799 30921 ts_tablet_manager.cc:1434] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260709 12:20:27.635113 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.635838 30800 raft_consensus.cc:3060] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.635879 30921 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.640604 30875 raft_consensus.cc:2468] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:27.642341 30614 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:27.643337 30930 raft_consensus.cc:2804] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.643955 30921 tablet_bootstrap.cc:654] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.643867 30930 raft_consensus.cc:697] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:27.644109 30998 raft_consensus.cc:493] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.645287 30800 raft_consensus.cc:2468] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:27.644800 30998 raft_consensus.cc:515] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.645911 30930 consensus_queue.cc:237] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.647676 30998 leader_election.cc:290] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.649876 30798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.650175 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.650938 30650 raft_consensus.cc:2468] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:27.622298 31009 raft_consensus.cc:2749] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.656458 30690 leader_election.cc:304] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.652417 30838 leader_election.cc:304] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:27.656474 30925 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.658859 30925 ts_tablet_manager.cc:1403] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.064s	user 0.018s	sys 0.002s
W20260709 12:20:27.658092 30839 leader_election.cc:343] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.659940 30957 raft_consensus.cc:2804] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.661458 30957 raft_consensus.cc:697] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:27.662809 30957 consensus_queue.cc:237] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.660534 30998 raft_consensus.cc:2804] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.666070 30998 raft_consensus.cc:493] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.652856 30799 raft_consensus.cc:2468] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:27.666553 30998 raft_consensus.cc:3060] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.668133 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
W20260709 12:20:27.671129 30615 leader_election.cc:343] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.666373 30952 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:27.671882 30925 raft_consensus.cc:359] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.673902 30925 raft_consensus.cc:385] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.674292 30925 raft_consensus.cc:740] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.672849 30839 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:27.676448 30925 consensus_queue.cc:260] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.683980 30929 leader_election.cc:290] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:27.684466 31000 raft_consensus.cc:2804] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.685098 31000 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.685428 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:27.686295 30965 raft_consensus.cc:493] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.686856 30922 tablet_bootstrap.cc:492] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
W20260709 12:20:27.687263 30689 leader_election.cc:343] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.686861 30965 raft_consensus.cc:515] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.687623 30922 ts_tablet_manager.cc:1403] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.095s	user 0.019s	sys 0.001s
I20260709 12:20:27.688635 30925 ts_tablet_manager.cc:1434] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.000s
I20260709 12:20:27.693161 30925 tablet_bootstrap.cc:492] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.685472 31000 raft_consensus.cc:3060] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.691294 30921 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.696847 30921 ts_tablet_manager.cc:1403] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.061s	user 0.019s	sys 0.006s
I20260709 12:20:27.699584 30965 leader_election.cc:290] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.701357 30925 tablet_bootstrap.cc:654] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.703068 30923 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.703735 30923 ts_tablet_manager.cc:1403] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.168s	user 0.016s	sys 0.020s
I20260709 12:20:27.707360 30923 raft_consensus.cc:359] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.708429 30923 raft_consensus.cc:385] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.708757 30923 raft_consensus.cc:740] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.691411 30998 raft_consensus.cc:515] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.710063 30923 consensus_queue.cc:260] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.713200 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14747c6d552943dbac8ab19f68816ea9" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.714257 30725 raft_consensus.cc:2468] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:27.715229 30923 ts_tablet_manager.cc:1434] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.002s
I20260709 12:20:27.716435 30923 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.717773 30921 raft_consensus.cc:359] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.719061 30921 raft_consensus.cc:385] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.719442 30921 raft_consensus.cc:740] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.693500 30922 raft_consensus.cc:359] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.720538 30922 raft_consensus.cc:385] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.720995 30922 raft_consensus.cc:740] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.722160 30922 consensus_queue.cc:260] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.726277 30998 leader_election.cc:290] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.724424 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14747c6d552943dbac8ab19f68816ea9" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.722050 30839 leader_election.cc:304] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.733296 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:27.738799 30650 raft_consensus.cc:3060] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.739990 30925 tablet_bootstrap.cc:492] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.740602 30925 ts_tablet_manager.cc:1403] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.048s	user 0.018s	sys 0.026s
I20260709 12:20:27.734300 30546 catalog_manager.cc:5697] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:27.741374 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:27.737457 30923 tablet_bootstrap.cc:654] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.736526 30957 raft_consensus.cc:493] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.728081 30921 consensus_queue.cc:260] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.748756 30957 raft_consensus.cc:515] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.744100 30925 raft_consensus.cc:359] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.751330 30925 raft_consensus.cc:385] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.751703 30925 raft_consensus.cc:740] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.751879 30922 ts_tablet_manager.cc:1434] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.062s	user 0.009s	sys 0.000s
I20260709 12:20:27.752545 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:27.751185 30921 ts_tablet_manager.cc:1434] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.054s	user 0.007s	sys 0.002s
I20260709 12:20:27.753679 30651 raft_consensus.cc:2468] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:27.745911 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.745532 30547 catalog_manager.cc:5697] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:27.756271 30839 leader_election.cc:343] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.756629 30689 leader_election.cc:304] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.745747 30998 raft_consensus.cc:2804] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.757912 30998 raft_consensus.cc:493] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.753590 30922 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:27.758239 30929 raft_consensus.cc:2804] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.757292 31000 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.758122 30921 tablet_bootstrap.cc:492] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:27.759032 30929 raft_consensus.cc:493] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.759943 30929 raft_consensus.cc:3060] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.752586 30925 consensus_queue.cc:260] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.758383 30998 raft_consensus.cc:3060] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.770030 31002 consensus_queue.cc:1048] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:27.771618 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.771909 31000 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:27.765830 30925 ts_tablet_manager.cc:1434] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.001s
I20260709 12:20:27.772419 30800 raft_consensus.cc:3060] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.773391 30925 tablet_bootstrap.cc:492] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:27.774185 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:27.774873 30651 raft_consensus.cc:3060] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.781279 30921 tablet_bootstrap.cc:654] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.782030 30923 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.782722 30923 ts_tablet_manager.cc:1403] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.067s	user 0.016s	sys 0.013s
I20260709 12:20:27.785753 30925 tablet_bootstrap.cc:654] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.790668 30998 raft_consensus.cc:515] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.792795 31002 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.796859 31002 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.799661 31002 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.801966 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.802878 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:27.805349 30690 leader_election.cc:343] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.806851 30957 leader_election.cc:290] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.746980 30799 raft_consensus.cc:2468] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:27.793466 30922 tablet_bootstrap.cc:654] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.807632 30875 raft_consensus.cc:3060] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.808049 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14747c6d552943dbac8ab19f68816ea9" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.808018 30998 leader_election.cc:290] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.815994 30929 raft_consensus.cc:515] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.819082 30650 raft_consensus.cc:2468] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:27.821045 30838 leader_election.cc:304] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19
W20260709 12:20:27.823930 30689 leader_election.cc:343] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.822754 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14747c6d552943dbac8ab19f68816ea9" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:27.825042 30725 raft_consensus.cc:3060] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.825368 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.825856 30689 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:27.822502 30998 raft_consensus.cc:2804] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.829101 31000 raft_consensus.cc:493] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.832832 30799 raft_consensus.cc:2468] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:27.832719 31000 raft_consensus.cc:515] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.833966 31002 raft_consensus.cc:2749] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.835633 31000 leader_election.cc:290] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.786302 30923 raft_consensus.cc:359] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.832722 30998 raft_consensus.cc:697] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:27.838016 30923 raft_consensus.cc:385] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.838244 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:27.838603 30923 raft_consensus.cc:740] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.838647 30998 consensus_queue.cc:237] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.839443 30799 raft_consensus.cc:2393] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:27.839684 31002 consensus_queue.cc:1048] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:27.840039 30923 consensus_queue.cc:260] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.842551 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db650f2a9f164bcd9c32d847363b1745" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:27.847608 30923 ts_tablet_manager.cc:1434] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.065s	user 0.005s	sys 0.003s
I20260709 12:20:27.848076 30839 leader_election.cc:304] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:27.852572 31000 raft_consensus.cc:3060] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.849961 30922 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:27.853358 30922 ts_tablet_manager.cc:1403] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.100s	user 0.008s	sys 0.006s
I20260709 12:20:27.856980 30922 raft_consensus.cc:359] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.857949 30922 raft_consensus.cc:385] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.858258 30922 raft_consensus.cc:740] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.851979 30725 raft_consensus.cc:2468] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:27.862430 31000 raft_consensus.cc:2749] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.864548 30923 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:27.875334 30922 consensus_queue.cc:260] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.878767 30922 ts_tablet_manager.cc:1434] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 12:20:27.880084 30922 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:27.874975 30549 catalog_manager.cc:5697] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.886369 30923 tablet_bootstrap.cc:654] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.886857 30922 tablet_bootstrap.cc:654] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.892724 30839 leader_election.cc:304] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:27.894421 30998 raft_consensus.cc:2804] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.895018 30998 raft_consensus.cc:697] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:27.896030 30998 consensus_queue.cc:237] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.905121 30923 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:27.910313 30923 ts_tablet_manager.cc:1403] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.046s	user 0.012s	sys 0.000s
I20260709 12:20:27.919518 30922 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:27.920351 30922 ts_tablet_manager.cc:1403] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.041s	user 0.025s	sys 0.003s
I20260709 12:20:27.931468 30923 raft_consensus.cc:359] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.932246 30923 raft_consensus.cc:385] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.932157 30922 raft_consensus.cc:359] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.933109 30922 raft_consensus.cc:385] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.933495 30922 raft_consensus.cc:740] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.932669 30549 catalog_manager.cc:5697] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:27.934486 30922 consensus_queue.cc:260] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.939263 30922 ts_tablet_manager.cc:1434] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:20:27.940619 30922 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:27.943801 30651 raft_consensus.cc:2468] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:27.945765 30838 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:27.947126 30998 raft_consensus.cc:2804] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.948101 30998 raft_consensus.cc:697] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:27.949329 30998 consensus_queue.cc:237] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:27.956969 30928 consensus_queue.cc:1048] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:27.959781 30799 raft_consensus.cc:3060] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.964303 30922 tablet_bootstrap.cc:654] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.968420 30929 leader_election.cc:290] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:27.969509 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:27.970222 30797 raft_consensus.cc:3060] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.968958 30800 raft_consensus.cc:2468] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:27.972429 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:27.973133 30651 raft_consensus.cc:3060] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.983783 30925 tablet_bootstrap.cc:492] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:27.984421 30925 ts_tablet_manager.cc:1403] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.211s	user 0.021s	sys 0.030s
I20260709 12:20:27.984819 30651 raft_consensus.cc:2468] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:27.983959 30549 catalog_manager.cc:5697] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.932670 30923 raft_consensus.cc:740] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.988093 30921 tablet_bootstrap.cc:492] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:27.988945 30921 ts_tablet_manager.cc:1403] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.231s	user 0.031s	sys 0.024s
I20260709 12:20:27.988530 30923 consensus_queue.cc:260] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:27.992297 30689 leader_election.cc:304] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:27.992709 30921 raft_consensus.cc:359] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:27.996977 30921 raft_consensus.cc:385] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.997300 30921 raft_consensus.cc:740] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.998207 30921 consensus_queue.cc:260] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.000433 30921 ts_tablet_manager.cc:1434] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.011s	user 0.001s	sys 0.006s
I20260709 12:20:28.001394 30921 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:28.003243 30923 ts_tablet_manager.cc:1434] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.091s	user 0.005s	sys 0.002s
I20260709 12:20:28.005159 31009 consensus_queue.cc:1048] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.009523 30923 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:28.012636 30797 raft_consensus.cc:2468] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:28.014376 30921 tablet_bootstrap.cc:654] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.018419 30998 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.019249 30998 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:27.993853 30957 raft_consensus.cc:2804] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.995625 30925 raft_consensus.cc:359] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.022810 30925 raft_consensus.cc:385] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.023156 30925 raft_consensus.cc:740] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.024039 30998 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.023996 30925 consensus_queue.cc:260] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.024835 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.025043 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.026129 30957 raft_consensus.cc:697] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:28.026520 30925 ts_tablet_manager.cc:1434] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.001s
W20260709 12:20:28.026602 30839 leader_election.cc:343] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.027833 30925 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
W20260709 12:20:28.028693 30838 leader_election.cc:343] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.029909 30838 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.031582 31009 consensus_queue.cc:1048] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.032414 30998 raft_consensus.cc:2749] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.032752 30957 consensus_queue.cc:237] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.034060 30922 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:28.034683 30922 ts_tablet_manager.cc:1403] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.094s	user 0.020s	sys 0.004s
I20260709 12:20:28.037153 30925 tablet_bootstrap.cc:654] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.047099 30921 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:28.047777 30921 ts_tablet_manager.cc:1403] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.000s
I20260709 12:20:28.037847 30922 raft_consensus.cc:359] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.048856 30922 raft_consensus.cc:385] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.049283 30922 raft_consensus.cc:740] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.050323 30923 tablet_bootstrap.cc:654] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.067366 30929 raft_consensus.cc:493] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.068328 30925 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:28.068104 30929 raft_consensus.cc:515] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.069007 30925 ts_tablet_manager.cc:1403] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.041s	user 0.007s	sys 0.007s
I20260709 12:20:28.075287 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.076297 31042 consensus_queue.cc:1048] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:28.077426 30689 leader_election.cc:343] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.066443 30922 consensus_queue.cc:260] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.062915 30921 raft_consensus.cc:359] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.078644 30921 raft_consensus.cc:385] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.078990 30921 raft_consensus.cc:740] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.079797 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:28.080114 30922 ts_tablet_manager.cc:1434] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.045s	user 0.011s	sys 0.002s
I20260709 12:20:28.080521 31043 raft_consensus.cc:493] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.081502 30922 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:28.081102 31043 raft_consensus.cc:515] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.081727 30929 leader_election.cc:290] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.082342 30800 raft_consensus.cc:2468] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:28.083876 30930 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.084491 30930 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.086926 30930 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.090063 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.090905 30725 raft_consensus.cc:2468] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
W20260709 12:20:28.092450 30615 consensus_peers.cc:597] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:28.092901 31043 leader_election.cc:290] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.093741 30925 raft_consensus.cc:359] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.093986 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.094741 30925 raft_consensus.cc:385] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.094900 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:28.095309 30925 raft_consensus.cc:740] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
W20260709 12:20:28.095973 30763 leader_election.cc:343] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.097083 30922 tablet_bootstrap.cc:654] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.093879 30615 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
W20260709 12:20:28.099062 30615 leader_election.cc:343] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.099627 30930 raft_consensus.cc:2804] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.100713 30930 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.101145 30930 raft_consensus.cc:3060] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.096508 30925 consensus_queue.cc:260] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.103832 30925 ts_tablet_manager.cc:1434] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.002s
I20260709 12:20:28.104957 30923 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:28.105805 30923 ts_tablet_manager.cc:1403] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.097s	user 0.007s	sys 0.019s
I20260709 12:20:28.105700 30690 leader_election.cc:304] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:28.107589 30952 consensus_queue.cc:1048] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.108116 30925 tablet_bootstrap.cc:492] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:28.110468 30923 raft_consensus.cc:359] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.111102 30930 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.111634 30957 raft_consensus.cc:2804] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.112246 30957 raft_consensus.cc:493] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.112663 30957 raft_consensus.cc:3060] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.113893 30930 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.113916 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.111291 30923 raft_consensus.cc:385] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.115355 30923 raft_consensus.cc:740] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.116619 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:28.116505 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:28.117524 30724 raft_consensus.cc:3060] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.121688 30925 tablet_bootstrap.cc:654] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.089740 30921 consensus_queue.cc:260] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.116484 30923 consensus_queue.cc:260] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:28.126170 30615 leader_election.cc:343] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.127030 30724 raft_consensus.cc:2468] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:28.128484 30615 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:28.128897 30923 ts_tablet_manager.cc:1434] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 12:20:28.129696 30921 ts_tablet_manager.cc:1434] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.082s	user 0.011s	sys 0.000s
I20260709 12:20:28.129843 30930 raft_consensus.cc:2804] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.130342 30930 raft_consensus.cc:697] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:28.131649 30921 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:28.131343 30930 consensus_queue.cc:237] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.131835 30922 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:28.133074 30922 ts_tablet_manager.cc:1403] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.052s	user 0.010s	sys 0.005s
I20260709 12:20:28.134585 30923 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:28.147354 30998 raft_consensus.cc:493] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.150341 30998 raft_consensus.cc:515] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.159652 30922 raft_consensus.cc:359] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.161060 30922 raft_consensus.cc:385] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.161478 30922 raft_consensus.cc:740] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.169665 30998 leader_election.cc:290] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.172103 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88343a7c45e54babbd7fcae9b6376094" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.172345 30929 consensus_queue.cc:1048] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.148577 30957 raft_consensus.cc:515] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.172137 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88343a7c45e54babbd7fcae9b6376094" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:28.175504 30838 leader_election.cc:343] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.175937 30725 raft_consensus.cc:2393] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:28.183059 30923 tablet_bootstrap.cc:654] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:28.181814 30839 leader_election.cc:343] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.184779 30839 leader_election.cc:304] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.186885 30925 tablet_bootstrap.cc:492] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:28.186612 30922 consensus_queue.cc:260] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.190608 30925 ts_tablet_manager.cc:1403] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.083s	user 0.017s	sys 0.005s
I20260709 12:20:28.190034 30764 leader_election.cc:304] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.189944 30998 raft_consensus.cc:2749] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.192986 30929 raft_consensus.cc:493] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.193826 30929 raft_consensus.cc:515] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.194268 30957 leader_election.cc:290] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.197203 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
W20260709 12:20:28.199702 30689 leader_election.cc:343] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.201781 31042 raft_consensus.cc:3060] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.201422 30925 raft_consensus.cc:359] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.202368 30925 raft_consensus.cc:385] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.202791 30929 leader_election.cc:290] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.205068 30922 ts_tablet_manager.cc:1434] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.072s	user 0.011s	sys 0.000s
I20260709 12:20:28.206137 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:28.207046 30922 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:28.208315 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:28.211187 30690 leader_election.cc:343] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.211818 30921 tablet_bootstrap.cc:654] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.213028 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.202693 30925 raft_consensus.cc:740] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.214720 30925 consensus_queue.cc:260] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:28.215541 30689 leader_election.cc:343] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.216158 30689 leader_election.cc:304] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:28.217803 30925 ts_tablet_manager.cc:1434] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260709 12:20:28.219125 30925 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:28.219126 30957 raft_consensus.cc:2749] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.221715 31009 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.222627 31009 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.226918 30922 tablet_bootstrap.cc:654] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.228384 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.229692 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.231422 30726 raft_consensus.cc:2393] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:28.229790 30874 raft_consensus.cc:2393] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:28.225477 31009 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.234941 30983 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.248255 30923 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:28.249208 30923 ts_tablet_manager.cc:1403] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.115s	user 0.019s	sys 0.004s
I20260709 12:20:28.250110 30983 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.254081 30983 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.259619 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.260519 31042 raft_consensus.cc:2749] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.260735 30614 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
W20260709 12:20:28.262035 30615 leader_election.cc:343] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.270701 30923 raft_consensus.cc:359] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.267675 31009 raft_consensus.cc:3060] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.274365 30925 tablet_bootstrap.cc:654] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.274091 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.276211 30874 raft_consensus.cc:2468] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:28.275343 30923 raft_consensus.cc:385] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.282742 30923 raft_consensus.cc:740] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.283965 30923 consensus_queue.cc:260] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.288416 30957 consensus_queue.cc:1048] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.288875 30923 ts_tablet_manager.cc:1434] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.039s	user 0.010s	sys 0.000s
I20260709 12:20:28.291433 30923 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:28.310027 30614 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.311249 30983 raft_consensus.cc:2804] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.311964 30983 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.312628 30983 raft_consensus.cc:3060] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.322826 30921 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:28.323798 30921 ts_tablet_manager.cc:1403] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.192s	user 0.029s	sys 0.036s
I20260709 12:20:28.333192 30549 catalog_manager.cc:5697] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.329222 30800 raft_consensus.cc:2468] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:28.342660 30922 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:28.343335 30922 ts_tablet_manager.cc:1403] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.137s	user 0.037s	sys 0.032s
I20260709 12:20:28.342478 30998 consensus_queue.cc:1048] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:28.344578 30965 raft_consensus.cc:493] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.345649 30965 raft_consensus.cc:515] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.353415 31009 raft_consensus.cc:2749] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.342581 30547 catalog_manager.cc:5697] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.356885 30922 raft_consensus.cc:359] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.358280 30922 raft_consensus.cc:385] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.358703 30922 raft_consensus.cc:740] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.359784 30922 consensus_queue.cc:260] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.363571 31002 raft_consensus.cc:493] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.364459 31002 raft_consensus.cc:515] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.362738 30921 raft_consensus.cc:359] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.365823 30921 raft_consensus.cc:385] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.366694 30921 raft_consensus.cc:740] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.367790 30921 consensus_queue.cc:260] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.368300 30965 leader_election.cc:290] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.366283 30922 ts_tablet_manager.cc:1434] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.023s	user 0.011s	sys 0.001s
I20260709 12:20:28.369110 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.366835 30925 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:28.369951 30726 raft_consensus.cc:2468] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:28.370391 30925 ts_tablet_manager.cc:1403] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.152s	user 0.015s	sys 0.010s
I20260709 12:20:28.372409 30839 leader_election.cc:304] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:28.369951 30922 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:28.375293 30799 raft_consensus.cc:3060] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.376111 30983 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.376792 30998 raft_consensus.cc:2804] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.377296 30998 raft_consensus.cc:493] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.377710 30998 raft_consensus.cc:3060] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.378219 30690 leader_election.cc:304] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:28.380299 30923 tablet_bootstrap.cc:654] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.381374 30922 tablet_bootstrap.cc:654] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.382323 31055 raft_consensus.cc:2804] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.385787 31055 raft_consensus.cc:697] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:28.386787 31055 consensus_queue.cc:237] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.382823 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:28.384387 30925 raft_consensus.cc:359] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.385115 30921 ts_tablet_manager.cc:1434] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.061s	user 0.011s	sys 0.000s
I20260709 12:20:28.393105 30925 raft_consensus.cc:385] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.393519 30925 raft_consensus.cc:740] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.394223 30921 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:28.395870 30965 consensus_queue.cc:1048] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:28.396329 30839 leader_election.cc:343] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.397790 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:28.398023 30925 consensus_queue.cc:260] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.398648 30874 raft_consensus.cc:3060] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.402097 30925 ts_tablet_manager.cc:1434] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.004s
I20260709 12:20:28.407469 31002 leader_election.cc:290] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:28.398290 30965 consensus_queue.cc:1048] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:20:28.425541 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e614ae4fde394b488924f185e2808b18" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.433169 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e614ae4fde394b488924f185e2808b18" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:28.433733 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:28.433894 31000 consensus_queue.cc:1048] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.415086 30983 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.439970 30925 tablet_bootstrap.cc:492] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:28.444842 30874 raft_consensus.cc:2468] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:28.453425 30875 raft_consensus.cc:2468] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:28.464483 31054 raft_consensus.cc:493] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.466248 31000 consensus_queue.cc:1048] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260709 12:20:28.465293 31002 raft_consensus.cc:493] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:28.466229 30615 leader_election.cc:343] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.498257 31002 raft_consensus.cc:515] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:28.475168 30690 leader_election.cc:343] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.500418 31042 raft_consensus.cc:493] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.501029 31042 raft_consensus.cc:515] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.465479 31054 raft_consensus.cc:515] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.501223 31002 leader_election.cc:290] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:28.506790 30925 tablet_bootstrap.cc:654] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.502126 30998 raft_consensus.cc:515] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.518025 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.520030 30614 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.522450 30983 raft_consensus.cc:2804] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.523034 30983 raft_consensus.cc:697] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:28.523928 30921 tablet_bootstrap.cc:654] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.509096 31042 leader_election.cc:290] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:28.527983 30923 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:28.528959 30923 ts_tablet_manager.cc:1403] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.238s	user 0.025s	sys 0.021s
I20260709 12:20:28.532485 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:28.534659 30983 consensus_queue.cc:237] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.535617 30998 leader_election.cc:290] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
W20260709 12:20:28.536856 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 68047.4us
I20260709 12:20:28.523149 31054 leader_election.cc:290] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:28.538981 30689 leader_election.cc:304] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:28.539280 30923 raft_consensus.cc:359] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.540134 30923 raft_consensus.cc:385] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.540446 30923 raft_consensus.cc:740] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
W20260709 12:20:28.540706 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 63610.1us
I20260709 12:20:28.541186 30923 consensus_queue.cc:260] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.542940 31000 raft_consensus.cc:493] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.543891 30923 ts_tablet_manager.cc:1434] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:20:28.545387 30923 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:28.543454 31000 raft_consensus.cc:515] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.549091 31002 raft_consensus.cc:2804] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.560693 30800 raft_consensus.cc:2393] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:28.567800 31002 raft_consensus.cc:493] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.573067 31002 raft_consensus.cc:3060] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.575558 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.581099 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3be7907c61d7477da4fc1646f65b2a48" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.584136 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:28.585949 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9ed5ce7483340edbbac9dee8cca5f1d" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.586772 30651 raft_consensus.cc:2468] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:28.590037 31042 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.590500 30615 leader_election.cc:304] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.590711 31042 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:28.592931 30952 raft_consensus.cc:3060] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.600647 31000 leader_election.cc:290] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.603426 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:28.606494 30838 leader_election.cc:343] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.610524 30689 leader_election.cc:304] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
W20260709 12:20:28.611943 30839 leader_election.cc:343] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.615248 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.612138 31055 raft_consensus.cc:2804] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.616806 31055 raft_consensus.cc:493] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.617244 31055 raft_consensus.cc:3060] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.613485 31042 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:28.613013 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.613312 30796 raft_consensus.cc:3060] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.624249 30925 tablet_bootstrap.cc:492] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:28.624958 30925 ts_tablet_manager.cc:1403] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.185s	user 0.012s	sys 0.002s
I20260709 12:20:28.614079 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:28.627445 30726 raft_consensus.cc:3060] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.628671 30925 raft_consensus.cc:359] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.614241 30763 leader_election.cc:304] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:28.615085 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:28.631631 30723 raft_consensus.cc:2393] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:28.633136 31042 raft_consensus.cc:2749] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:28.633421 30764 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:28.637208 30928 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.637910 30925 raft_consensus.cc:385] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.638441 30928 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.638954 30925 raft_consensus.cc:740] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.640568 30925 consensus_queue.cc:260] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.641805 30922 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:28.649190 31042 raft_consensus.cc:3060] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.651966 31002 raft_consensus.cc:515] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.690477 31002 leader_election.cc:290] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:28.640992 30928 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:28.646731 30923 tablet_bootstrap.cc:654] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.648823 30929 consensus_queue.cc:1048] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.647464 30546 catalog_manager.cc:5697] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.668540 30925 ts_tablet_manager.cc:1434] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.005s
I20260709 12:20:28.698115 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e614ae4fde394b488924f185e2808b18" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:28.701542 30726 raft_consensus.cc:2468] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:28.645408 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:28.686582 30921 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:28.712790 30921 ts_tablet_manager.cc:1403] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.319s	user 0.019s	sys 0.006s
W20260709 12:20:28.713796 30763 leader_election.cc:343] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.718740 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9ed5ce7483340edbbac9dee8cca5f1d" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.721908 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e614ae4fde394b488924f185e2808b18" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:28.722687 30874 raft_consensus.cc:3060] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.718897 30921 raft_consensus.cc:359] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.721120 30549 catalog_manager.cc:5697] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.725925 30921 raft_consensus.cc:385] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.698150 30925 tablet_bootstrap.cc:492] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:28.726325 30921 raft_consensus.cc:740] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.727139 30921 consensus_queue.cc:260] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.703114 31042 raft_consensus.cc:2749] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.703665 30952 raft_consensus.cc:2749] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:28.681182 30839 leader_election.cc:343] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.731715 30839 leader_election.cc:304] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
W20260709 12:20:28.732806 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 52028.8us
I20260709 12:20:28.733892 30925 tablet_bootstrap.cc:654] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.704468 31055 raft_consensus.cc:515] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.738019 31069 raft_consensus.cc:2749] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.738830 31069 consensus_queue.cc:1048] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.729610 30921 ts_tablet_manager.cc:1434] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 12:20:28.741106 30921 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:28.741984 30874 raft_consensus.cc:2468] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:28.742695 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.746160 30839 leader_election.cc:304] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:28.752190 30983 consensus_queue.cc:1048] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.752560 31070 raft_consensus.cc:2804] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.753950 31070 raft_consensus.cc:697] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:28.702100 30922 ts_tablet_manager.cc:1403] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.332s	user 0.049s	sys 0.037s
W20260709 12:20:28.702880 30690 leader_election.cc:343] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.755532 31070 consensus_queue.cc:237] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:28.757449 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 54622.3us
I20260709 12:20:28.763952 31055 leader_election.cc:290] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:28.758970 30763 leader_election.cc:343] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.765585 30763 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
W20260709 12:20:28.752743 30689 leader_election.cc:343] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.772239 30921 tablet_bootstrap.cc:654] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.773463 30928 raft_consensus.cc:2749] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.774973 30925 tablet_bootstrap.cc:492] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:28.775722 30925 ts_tablet_manager.cc:1403] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.078s	user 0.012s	sys 0.009s
I20260709 12:20:28.781080 30925 raft_consensus.cc:359] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.781870 30925 raft_consensus.cc:385] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.765616 31055 consensus_queue.cc:1048] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:20:28.782351 30925 raft_consensus.cc:740] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.786594 30925 consensus_queue.cc:260] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.790138 30925 ts_tablet_manager.cc:1434] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.004s
I20260709 12:20:28.791296 30925 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:28.796193 30952 consensus_queue.cc:1048] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.801225 30925 tablet_bootstrap.cc:654] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.808178 30923 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:28.808984 30923 ts_tablet_manager.cc:1403] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.264s	user 0.015s	sys 0.030s
W20260709 12:20:28.792419 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:28.823585 30923 raft_consensus.cc:359] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.824469 30923 raft_consensus.cc:385] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.824774 30923 raft_consensus.cc:740] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.826179 30922 raft_consensus.cc:359] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.829200 30922 raft_consensus.cc:385] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.829844 30922 raft_consensus.cc:740] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.830495 30923 consensus_queue.cc:260] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.834105 30922 consensus_queue.cc:260] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.836110 31054 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.836993 31054 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.838291 30922 ts_tablet_manager.cc:1434] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.082s	user 0.010s	sys 0.000s
I20260709 12:20:28.818115 30689 leader_election.cc:304] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:28.839594 30923 ts_tablet_manager.cc:1434] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.002s
I20260709 12:20:28.841770 30922 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:28.848308 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:28.841938 30991 raft_consensus.cc:2804] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.854489 30991 raft_consensus.cc:697] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
W20260709 12:20:28.854941 30614 leader_election.cc:343] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.856112 30991 consensus_queue.cc:237] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.867570 30923 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:28.842264 31054 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
W20260709 12:20:28.861325 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 67454.6us
I20260709 12:20:28.878165 31070 raft_consensus.cc:493] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.897511 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9ed5ce7483340edbbac9dee8cca5f1d" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:28.901067 31070 raft_consensus.cc:515] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:28.902340 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9ed5ce7483340edbbac9dee8cca5f1d" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:28.908152 30650 raft_consensus.cc:3060] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.910611 30957 consensus_queue.cc:1048] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.921825 30650 raft_consensus.cc:2468] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:28.907490 30923 tablet_bootstrap.cc:654] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:28.947484 31077 log.cc:927] Time spent T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:20:28.894238 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:28.948347 30689 leader_election.cc:343] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:28.950570 30922 tablet_bootstrap.cc:654] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.958920 30925 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:28.966284 30925 ts_tablet_manager.cc:1403] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.175s	user 0.020s	sys 0.004s
I20260709 12:20:28.969208 30925 raft_consensus.cc:359] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.969987 30925 raft_consensus.cc:385] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.970264 30925 raft_consensus.cc:740] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.964576 30800 raft_consensus.cc:2468] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:28.971310 30925 consensus_queue.cc:260] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:28.980332 30689 leader_election.cc:304] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:28.981611 31055 raft_consensus.cc:2804] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.981851 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7470dd49ce524f4aaf3faf7739fe5bd7" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:28.982352 31055 raft_consensus.cc:697] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:28.983315 31055 consensus_queue.cc:237] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:28.981308 30923 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:28.981386 30549 catalog_manager.cc:5697] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:28.986066 30923 ts_tablet_manager.cc:1403] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.119s	user 0.014s	sys 0.004s
I20260709 12:20:28.988083 30925 ts_tablet_manager.cc:1434] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:20:28.989269 30957 consensus_queue.cc:1048] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.989360 30925 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:28.970119 31070 leader_election.cc:290] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
W20260709 12:20:29.000510 30839 leader_election.cc:343] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:29.005620 31084 log.cc:927] Time spent T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:20:29.006068 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7470dd49ce524f4aaf3faf7739fe5bd7" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:29.058369 30928 raft_consensus.cc:493] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.084355 30921 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:29.032838 30923 raft_consensus.cc:359] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.007427 30547 catalog_manager.cc:5697] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:29.085103 30923 raft_consensus.cc:385] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.085692 30923 raft_consensus.cc:740] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.016407 30957 raft_consensus.cc:493] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.048606 30925 tablet_bootstrap.cc:654] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.086776 30923 consensus_queue.cc:260] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:29.087096 30839 leader_election.cc:343] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.089016 30839 leader_election.cc:304] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:29.090224 30957 raft_consensus.cc:515] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.091136 30965 raft_consensus.cc:2749] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:29.082382 30922 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:29.093019 30922 ts_tablet_manager.cc:1403] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.252s	user 0.017s	sys 0.005s
I20260709 12:20:29.085078 30921 ts_tablet_manager.cc:1403] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.344s	user 0.019s	sys 0.011s
I20260709 12:20:29.084184 30928 raft_consensus.cc:515] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.099450 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88343a7c45e54babbd7fcae9b6376094" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.100333 30875 raft_consensus.cc:2468] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:29.101693 30928 leader_election.cc:290] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.099262 30923 ts_tablet_manager.cc:1434] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.113s	user 0.017s	sys 0.014s
W20260709 12:20:29.079185 31085 log.cc:927] Time spent T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:20:29.103583 30923 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.103837 30689 leader_election.cc:304] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:29.105386 30991 raft_consensus.cc:2804] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:29.105893 30991 raft_consensus.cc:493] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.105388 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66db71b232844f1794f53dba1cbd6198" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.106307 30991 raft_consensus.cc:3060] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.106746 30874 raft_consensus.cc:2468] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:29.107364 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66db71b232844f1794f53dba1cbd6198" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.108531 30763 leader_election.cc:304] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:29.109505 30928 raft_consensus.cc:2804] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:29.110002 30928 raft_consensus.cc:493] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.110378 30928 raft_consensus.cc:3060] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:29.112207 30763 leader_election.cc:343] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.035957 30615 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:29.116515 30952 raft_consensus.cc:2804] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:20:29.116664 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 80863.8us
I20260709 12:20:29.117053 30952 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.117534 30952 raft_consensus.cc:3060] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.118371 30923 tablet_bootstrap.cc:654] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:29.121611 30615 consensus_peers.cc:597] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:29.122488 30928 raft_consensus.cc:515] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.100862 30922 raft_consensus.cc:359] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.124791 30922 raft_consensus.cc:385] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.129112 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66db71b232844f1794f53dba1cbd6198" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:29.129837 30874 raft_consensus.cc:3060] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.125267 30922 raft_consensus.cc:740] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.133486 30957 leader_election.cc:290] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.134321 30991 raft_consensus.cc:515] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.134825 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88343a7c45e54babbd7fcae9b6376094" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.135668 30649 raft_consensus.cc:2468] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:29.136229 30983 consensus_queue.cc:1048] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.136773 30922 consensus_queue.cc:260] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.139751 30928 leader_election.cc:290] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.141594 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66db71b232844f1794f53dba1cbd6198" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:29.143642 30922 ts_tablet_manager.cc:1434] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.002s
W20260709 12:20:29.143573 30763 leader_election.cc:343] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.147413 30874 raft_consensus.cc:2468] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:29.150998 30763 leader_election.cc:304] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:29.152603 30928 raft_consensus.cc:2804] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:29.153259 30928 raft_consensus.cc:697] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:29.154103 30952 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.128036 30921 raft_consensus.cc:359] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.149428 30922 tablet_bootstrap.cc:492] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:29.154439 30928 consensus_queue.cc:237] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.157732 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88343a7c45e54babbd7fcae9b6376094" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:29.159683 30921 raft_consensus.cc:385] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.159852 30874 raft_consensus.cc:3060] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.160126 30921 raft_consensus.cc:740] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.158809 30925 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:29.161804 30921 consensus_queue.cc:260] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.165539 30952 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:29.167012 30873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:29.167313 30921 ts_tablet_manager.cc:1434] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.070s	user 0.008s	sys 0.001s
W20260709 12:20:29.171984 30614 leader_election.cc:343] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.172813 30921 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:29.162923 30925 ts_tablet_manager.cc:1403] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.174s	user 0.016s	sys 0.005s
I20260709 12:20:29.167420 30991 leader_election.cc:290] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.168401 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88343a7c45e54babbd7fcae9b6376094" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:29.178608 30649 raft_consensus.cc:3060] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.180300 30874 raft_consensus.cc:2468] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:29.182017 30923 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:29.181839 30689 leader_election.cc:304] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:29.182557 30923 ts_tablet_manager.cc:1403] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.079s	user 0.015s	sys 0.001s
I20260709 12:20:29.182875 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:29.183655 30796 raft_consensus.cc:3060] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.182896 30957 raft_consensus.cc:2804] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:29.186800 30957 raft_consensus.cc:697] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:29.189010 30957 consensus_queue.cc:237] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.191160 30921 tablet_bootstrap.cc:654] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.195145 30796 raft_consensus.cc:2468] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:29.194667 30923 raft_consensus.cc:359] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.198745 30923 raft_consensus.cc:385] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.202243 30615 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:29.203338 30649 raft_consensus.cc:2468] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:29.208173 30922 tablet_bootstrap.cc:654] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.208555 30925 raft_consensus.cc:359] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.209472 30925 raft_consensus.cc:385] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.209843 30925 raft_consensus.cc:740] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.209940 30983 raft_consensus.cc:2804] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:29.210845 30983 raft_consensus.cc:697] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:29.210680 30925 consensus_queue.cc:260] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.211962 30983 consensus_queue.cc:237] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.213243 30925 ts_tablet_manager.cc:1434] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.038s	user 0.005s	sys 0.000s
I20260709 12:20:29.214021 31055 raft_consensus.cc:493] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.216576 30923 raft_consensus.cc:740] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.218235 30923 consensus_queue.cc:260] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.223809 30925 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:29.226948 30923 ts_tablet_manager.cc:1434] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.001s
I20260709 12:20:29.228533 30923 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.216228 31055 raft_consensus.cc:515] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.232555 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.233747 30875 raft_consensus.cc:2468] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:29.241384 30689 leader_election.cc:304] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:29.242678 30991 raft_consensus.cc:2804] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:29.243160 30991 raft_consensus.cc:493] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.243628 30991 raft_consensus.cc:3060] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.242769 30925 tablet_bootstrap.cc:654] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.244709 30546 catalog_manager.cc:5697] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:29.252625 30965 raft_consensus.cc:493] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.253351 30965 raft_consensus.cc:515] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.243284 30549 catalog_manager.cc:5697] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:29.256126 30965 leader_election.cc:290] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:29.234403 31055 leader_election.cc:290] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.237387 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.275292 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:29.286281 30921 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:29.287320 30921 ts_tablet_manager.cc:1403] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.115s	user 0.018s	sys 0.000s
W20260709 12:20:29.293151 30839 leader_election.cc:343] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.295192 30991 raft_consensus.cc:515] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:29.280015 30689 leader_election.cc:343] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.289817 30547 catalog_manager.cc:5697] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:29.282797 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:29.306245 30839 leader_election.cc:343] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.306880 30839 leader_election.cc:304] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:29.308763 31000 raft_consensus.cc:2749] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:29.314851 30925 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:29.315570 30925 ts_tablet_manager.cc:1403] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.092s	user 0.016s	sys 0.009s
I20260709 12:20:29.319069 30922 tablet_bootstrap.cc:492] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
W20260709 12:20:29.322721 30839 consensus_peers.cc:597] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:29.323479 30923 tablet_bootstrap.cc:654] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.326896 30965 raft_consensus.cc:493] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.327638 30965 raft_consensus.cc:515] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.329897 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:29.331022 30875 raft_consensus.cc:3060] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.331272 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9ed5ce7483340edbbac9dee8cca5f1d" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.332420 30649 raft_consensus.cc:2393] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:29.336037 30922 ts_tablet_manager.cc:1403] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.171s	user 0.020s	sys 0.023s
I20260709 12:20:29.340322 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:29.340767 30991 leader_election.cc:290] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:29.342227 30689 leader_election.cc:343] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.339568 30922 raft_consensus.cc:359] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.343050 30922 raft_consensus.cc:385] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.343384 30922 raft_consensus.cc:740] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.337581 30921 raft_consensus.cc:359] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.344314 30922 consensus_queue.cc:260] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.344920 30921 raft_consensus.cc:385] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.345263 30921 raft_consensus.cc:740] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.347038 30922 ts_tablet_manager.cc:1434] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:29.347491 30925 raft_consensus.cc:359] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.348043 30965 leader_election.cc:290] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:29.348281 30925 raft_consensus.cc:385] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.348616 30925 raft_consensus.cc:740] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.349442 30922 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:29.349466 30921 consensus_queue.cc:260] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.349761 30925 consensus_queue.cc:260] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.354494 30925 ts_tablet_manager.cc:1434] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.002s
I20260709 12:20:29.355664 30925 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:29.358680 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9ed5ce7483340edbbac9dee8cca5f1d" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:29.359486 30921 ts_tablet_manager.cc:1434] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.072s	user 0.010s	sys 0.000s
I20260709 12:20:29.359658 30922 tablet_bootstrap.cc:654] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.359762 30923 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:29.361393 30923 ts_tablet_manager.cc:1403] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.133s	user 0.030s	sys 0.010s
I20260709 12:20:29.361531 30921 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:29.365826 30965 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.371374 30923 raft_consensus.cc:359] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.372597 30923 raft_consensus.cc:385] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.372555 30965 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.373343 30923 raft_consensus.cc:740] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.366102 30925 tablet_bootstrap.cc:654] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.374436 30923 consensus_queue.cc:260] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.377614 30796 raft_consensus.cc:3060] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.377928 30923 ts_tablet_manager.cc:1434] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:20:29.380059 30921 tablet_bootstrap.cc:654] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.380268 30923 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.385836 30922 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:29.385949 30839 leader_election.cc:304] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:29.386567 30922 ts_tablet_manager.cc:1403] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.001s
I20260709 12:20:29.388777 30923 tablet_bootstrap.cc:654] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.389257 30548 catalog_manager.cc:5697] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:29.390120 30922 raft_consensus.cc:359] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.390842 30922 raft_consensus.cc:385] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.390486 30991 consensus_queue.cc:1048] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.391162 30922 raft_consensus.cc:740] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.392014 30922 consensus_queue.cc:260] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.395731 31000 raft_consensus.cc:3060] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.397702 30921 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:29.399870 30921 ts_tablet_manager.cc:1403] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.000s
I20260709 12:20:29.398710 30922 ts_tablet_manager.cc:1434] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.003s
I20260709 12:20:29.402143 30922 tablet_bootstrap.cc:492] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:29.402004 30991 consensus_queue.cc:1048] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:29.403998 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.406832 30921 raft_consensus.cc:359] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.410426 30921 raft_consensus.cc:385] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.410745 30922 tablet_bootstrap.cc:654] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.410933 30921 raft_consensus.cc:740] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.412202 30921 consensus_queue.cc:260] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.409348 31055 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.409955 30923 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:29.414106 30923 ts_tablet_manager.cc:1403] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.004s
I20260709 12:20:29.413949 31055 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.417284 30923 raft_consensus.cc:359] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.418053 30923 raft_consensus.cc:385] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.417934 30965 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.418768 30923 raft_consensus.cc:740] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.419622 30923 consensus_queue.cc:260] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.422374 30921 ts_tablet_manager.cc:1434] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:20:29.423278 30873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.423810 30921 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:29.424157 30873 raft_consensus.cc:2468] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:29.424465 30922 tablet_bootstrap.cc:492] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:29.425093 30922 ts_tablet_manager.cc:1403] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.006s
I20260709 12:20:29.427117 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.428025 30649 raft_consensus.cc:2468] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:29.431018 30921 tablet_bootstrap.cc:654] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.430495 30922 raft_consensus.cc:359] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.431520 30922 raft_consensus.cc:385] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.431875 30922 raft_consensus.cc:740] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.416442 31055 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.433065 30922 consensus_queue.cc:260] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.434003 30923 ts_tablet_manager.cc:1434] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:20:29.435580 30923 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.430788 30689 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:29.435953 30922 ts_tablet_manager.cc:1434] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.002s
I20260709 12:20:29.438079 30922 tablet_bootstrap.cc:492] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:29.438359 30929 raft_consensus.cc:2804] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:29.438797 30929 raft_consensus.cc:493] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.439137 30929 raft_consensus.cc:3060] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.442245 30923 tablet_bootstrap.cc:654] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.444960 30922 tablet_bootstrap.cc:654] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.448220 30929 raft_consensus.cc:515] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:29.461227 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 75456.7us
I20260709 12:20:29.462944 31000 raft_consensus.cc:2749] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:29.467299 30965 consensus_queue.cc:1048] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.468097 30875 raft_consensus.cc:2468] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:29.480346 30925 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:29.481047 30925 ts_tablet_manager.cc:1403] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.126s	user 0.015s	sys 0.002s
I20260709 12:20:29.489821 30923 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:29.490491 30923 ts_tablet_manager.cc:1403] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.055s	user 0.010s	sys 0.003s
I20260709 12:20:29.490833 30925 raft_consensus.cc:359] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.491695 30925 raft_consensus.cc:385] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.492139 30925 raft_consensus.cc:740] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.491822 30922 tablet_bootstrap.cc:492] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:29.493076 30922 ts_tablet_manager.cc:1403] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.055s	user 0.015s	sys 0.007s
I20260709 12:20:29.493005 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:29.496802 30796 raft_consensus.cc:2393] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:29.489029 30689 leader_election.cc:304] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:29.499315 31055 raft_consensus.cc:2804] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:29.500106 31055 raft_consensus.cc:697] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:29.497941 30923 raft_consensus.cc:359] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.501209 30923 raft_consensus.cc:385] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.501111 31055 consensus_queue.cc:237] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.501775 30923 raft_consensus.cc:740] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.499409 30925 consensus_queue.cc:260] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.501811 30922 raft_consensus.cc:359] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.503273 30923 consensus_queue.cc:260] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.503541 30957 consensus_queue.cc:1048] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.503824 30965 raft_consensus.cc:493] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.504544 30965 raft_consensus.cc:515] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.503870 30922 raft_consensus.cc:385] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.505568 30922 raft_consensus.cc:740] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.507459 30922 consensus_queue.cc:260] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.510121 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.510988 30649 raft_consensus.cc:2468] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:29.512714 30838 leader_election.cc:304] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:29.514187 31070 raft_consensus.cc:2804] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:29.514662 31070 raft_consensus.cc:493] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.515101 31070 raft_consensus.cc:3060] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.520357 30929 leader_election.cc:290] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.525619 30965 leader_election.cc:290] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.527329 30921 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:29.528070 30921 ts_tablet_manager.cc:1403] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.105s	user 0.021s	sys 0.000s
I20260709 12:20:29.527551 30983 raft_consensus.cc:493] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.510934 30922 ts_tablet_manager.cc:1434] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:20:29.502553 30839 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:29.529771 30983 raft_consensus.cc:515] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.530756 31000 raft_consensus.cc:3060] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.532410 30983 leader_election.cc:290] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:29.530901 30922 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:29.533201 30923 ts_tablet_manager.cc:1434] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.042s	user 0.010s	sys 0.000s
I20260709 12:20:29.534435 30923 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.534904 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:29.535781 30796 raft_consensus.cc:2468] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:29.537115 30615 leader_election.cc:304] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:29.538239 30983 raft_consensus.cc:2804] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:29.538703 30983 raft_consensus.cc:493] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.539067 30873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:29.539796 30873 raft_consensus.cc:3060] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.539067 30983 raft_consensus.cc:3060] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.547519 30921 raft_consensus.cc:359] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.540576 30922 tablet_bootstrap.cc:654] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.571264 30923 tablet_bootstrap.cc:654] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.524915 31070 raft_consensus.cc:515] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.556202 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:29.556728 30873 raft_consensus.cc:2468] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:29.563901 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6072fe2fb27a44558a09de34939e4421" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:29.575675 30649 raft_consensus.cc:3060] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.567414 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:29.578193 30725 raft_consensus.cc:2468] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:29.588491 30921 raft_consensus.cc:385] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.588982 30921 raft_consensus.cc:740] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.591140 30649 raft_consensus.cc:2468] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:29.593146 30923 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:29.593900 30923 ts_tablet_manager.cc:1403] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.004s
I20260709 12:20:29.594427 30689 leader_election.cc:304] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:29.596288 30921 consensus_queue.cc:260] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.597905 30923 raft_consensus.cc:359] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:29.597858 30838 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 85256.3us
I20260709 12:20:29.598829 30923 raft_consensus.cc:385] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.600602 30923 raft_consensus.cc:740] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.584887 30983 raft_consensus.cc:515] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.598834 30957 raft_consensus.cc:2804] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:29.600178 31070 leader_election.cc:290] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.599391 30925 ts_tablet_manager.cc:1434] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.118s	user 0.011s	sys 0.002s
I20260709 12:20:29.601838 30923 consensus_queue.cc:260] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.635375 31000 raft_consensus.cc:2749] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:29.674180 30925 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:29.675410 30922 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:29.676620 30922 ts_tablet_manager.cc:1403] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.146s	user 0.023s	sys 0.013s
I20260709 12:20:29.684137 30921 ts_tablet_manager.cc:1434] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.156s	user 0.013s	sys 0.001s
I20260709 12:20:29.687119 30921 tablet_bootstrap.cc:492] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:29.676229 30957 raft_consensus.cc:697] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:29.692622 30923 ts_tablet_manager.cc:1434] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.098s	user 0.011s	sys 0.000s
W20260709 12:20:29.675223 30839 leader_election.cc:343] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.702374 30957 consensus_queue.cc:237] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.710037 30923 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.713231 30922 raft_consensus.cc:359] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.715824 30922 raft_consensus.cc:385] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.716523 30922 raft_consensus.cc:740] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.720122 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:29.723135 30649 raft_consensus.cc:3060] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.723889 30549 catalog_manager.cc:5697] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:29.725853 30922 consensus_queue.cc:260] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.720611 30929 consensus_queue.cc:1048] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.690892 31054 raft_consensus.cc:493] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.749984 30922 ts_tablet_manager.cc:1434] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.073s	user 0.009s	sys 0.000s
I20260709 12:20:29.755313 30922 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:29.759635 30725 raft_consensus.cc:3060] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:29.742596 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 139377us
I20260709 12:20:29.775372 30928 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.776177 30928 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.779142 31120 consensus_queue.cc:1048] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.780874 31120 raft_consensus.cc:493] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.781469 31120 raft_consensus.cc:515] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.784116 31120 leader_election.cc:290] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:29.739362 31054 raft_consensus.cc:515] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.787282 31054 leader_election.cc:290] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:29.788260 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.789363 30983 leader_election.cc:290] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:29.789535 30875 raft_consensus.cc:2393] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:29.790653 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:29.791394 30796 raft_consensus.cc:3060] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.794243 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:29.795603 30925 tablet_bootstrap.cc:654] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.796973 30800 raft_consensus.cc:3060] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:29.797497 30839 leader_election.cc:343] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.798295 30928 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.798600 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:29.799379 30726 raft_consensus.cc:3060] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.806183 30921 tablet_bootstrap.cc:654] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.789536 30923 tablet_bootstrap.cc:654] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.810456 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66db71b232844f1794f53dba1cbd6198" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.811626 30874 raft_consensus.cc:2393] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:29.813225 30726 raft_consensus.cc:2468] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:29.796991 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.817005 31055 consensus_queue.cc:1048] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.823830 30649 raft_consensus.cc:2468] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:29.825690 30838 leader_election.cc:304] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:29.827279 31121 raft_consensus.cc:2804] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:29.827898 31121 raft_consensus.cc:697] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:29.828822 31121 consensus_queue.cc:237] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.835393 30922 tablet_bootstrap.cc:654] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:29.871915 30763 leader_election.cc:343] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.872910 31055 consensus_queue.cc:1048] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.876778 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66db71b232844f1794f53dba1cbd6198" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:29.880230 30548 catalog_manager.cc:5697] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 2, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:29.886423 30925 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:29.887120 30925 ts_tablet_manager.cc:1403] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.213s	user 0.011s	sys 0.002s
I20260709 12:20:29.890290 30925 raft_consensus.cc:359] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.891040 30925 raft_consensus.cc:385] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.891327 30925 raft_consensus.cc:740] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.899106 30921 tablet_bootstrap.cc:492] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:29.899780 30921 ts_tablet_manager.cc:1403] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.215s	user 0.008s	sys 0.008s
I20260709 12:20:29.903715 30796 raft_consensus.cc:2468] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:29.908214 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5c48ae242924bcab94096bb803d25d4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:29.912344 30921 raft_consensus.cc:359] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.914031 30991 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:29.916756 30991 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:29.919278 30615 leader_election.cc:304] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:29.920356 30991 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.922061 30921 raft_consensus.cc:385] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.922359 30651 raft_consensus.cc:3060] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.923425 30921 raft_consensus.cc:740] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.923909 30925 consensus_queue.cc:260] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.924309 30921 consensus_queue.cc:260] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.922746 31119 consensus_queue.cc:1048] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.924922 31121 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.925598 31121 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.929430 31121 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:29.930260 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:29.932214 30925 ts_tablet_manager.cc:1434] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260709 12:20:29.933086 30925 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:29.934293 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.935967 30649 raft_consensus.cc:3060] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:29.936340 30838 leader_election.cc:343] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:29.943248 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:29.944249 31122 raft_consensus.cc:2749] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:29.944252 30797 raft_consensus.cc:2468] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:29.947626 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.946712 30839 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:29.949514 30925 tablet_bootstrap.cc:654] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.950736 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:29.951337 30921 ts_tablet_manager.cc:1434] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.051s	user 0.005s	sys 0.003s
I20260709 12:20:29.951829 30965 raft_consensus.cc:2804] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:29.952258 30965 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:29.952673 30965 raft_consensus.cc:3060] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:29.955514 30689 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:29.956470 30991 raft_consensus.cc:2749] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:29.958491 30925 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:29.958974 30925 ts_tablet_manager.cc:1403] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.000s
I20260709 12:20:29.961131 30925 raft_consensus.cc:359] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.916265 30922 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:29.961717 30925 raft_consensus.cc:385] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.962136 30925 raft_consensus.cc:740] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.962182 30922 ts_tablet_manager.cc:1403] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.211s	user 0.015s	sys 0.003s
I20260709 12:20:29.963747 30925 consensus_queue.cc:260] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:29.964927 30921 tablet_bootstrap.cc:492] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:29.965224 30923 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:29.965981 30923 ts_tablet_manager.cc:1403] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.272s	user 0.013s	sys 0.009s
I20260709 12:20:29.966295 30925 ts_tablet_manager.cc:1434] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:29.966106 30991 consensus_queue.cc:1048] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:29.967449 30925 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:29.969158 30923 raft_consensus.cc:359] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.970157 30923 raft_consensus.cc:385] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:29.970505 30923 raft_consensus.cc:740] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:29.971304 30923 consensus_queue.cc:260] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.973140 30923 ts_tablet_manager.cc:1434] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:29.974290 30923 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:29.975667 30991 consensus_queue.cc:1048] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:29.976955 30925 tablet_bootstrap.cc:654] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:29.982010 30923 tablet_bootstrap.cc:654] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:29.992475 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 90057.8us
I20260709 12:20:29.944556 30615 leader_election.cc:304] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:29.992667 31130 consensus_queue.cc:1048] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:20:29.993149 31125 raft_consensus.cc:493] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:29.996793 31125 raft_consensus.cc:515] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:29.998359 31130 consensus_queue.cc:1048] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:29.999502 31125 leader_election.cc:290] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:29.991701 30921 tablet_bootstrap.cc:654] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.000900 30952 raft_consensus.cc:2804] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.001227 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7470dd49ce524f4aaf3faf7739fe5bd7" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.001567 30952 raft_consensus.cc:697] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
W20260709 12:20:30.003062 30764 leader_election.cc:343] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.003144 30952 consensus_queue.cc:237] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.004284 30923 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:30.002444 30922 raft_consensus.cc:359] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.005028 30923 ts_tablet_manager.cc:1403] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.014s
I20260709 12:20:30.005265 30922 raft_consensus.cc:385] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.005702 30922 raft_consensus.cc:740] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.006508 30922 consensus_queue.cc:260] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:29.992319 30925 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:30.008904 30925 ts_tablet_manager.cc:1403] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.000s
I20260709 12:20:30.008838 30923 raft_consensus.cc:359] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.009833 30923 raft_consensus.cc:385] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.010186 30923 raft_consensus.cc:740] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.010987 30923 consensus_queue.cc:260] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.014061 30923 ts_tablet_manager.cc:1434] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
I20260709 12:20:30.015362 30923 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:30.024394 30922 ts_tablet_manager.cc:1434] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.062s	user 0.007s	sys 0.000s
I20260709 12:20:30.026721 30923 tablet_bootstrap.cc:654] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.028273 30925 raft_consensus.cc:359] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:30.032888 30763 leader_election.cc:343] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.033723 30763 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:30.036605 31144 raft_consensus.cc:2749] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:30.037289 30873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7470dd49ce524f4aaf3faf7739fe5bd7" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:30.037725 30922 tablet_bootstrap.cc:492] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:30.038067 30873 raft_consensus.cc:2468] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:30.039593 30763 leader_election.cc:304] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.041778 31130 raft_consensus.cc:2804] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.042282 31130 raft_consensus.cc:493] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.042701 31130 raft_consensus.cc:3060] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.047088 30548 catalog_manager.cc:5697] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.059587 31122 raft_consensus.cc:493] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.061249 31122 raft_consensus.cc:515] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.065918 30957 raft_consensus.cc:493] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.066819 30957 raft_consensus.cc:515] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.067711 30923 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:30.068492 30923 ts_tablet_manager.cc:1403] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.053s	user 0.015s	sys 0.005s
I20260709 12:20:30.073055 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:30.073796 31130 raft_consensus.cc:515] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.077757 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7470dd49ce524f4aaf3faf7739fe5bd7" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
W20260709 12:20:30.079514 30764 leader_election.cc:343] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.072767 30923 raft_consensus.cc:359] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.080907 30923 raft_consensus.cc:385] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.081231 30923 raft_consensus.cc:740] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.081444 30957 leader_election.cc:290] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:30.082001 30923 consensus_queue.cc:260] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.084825 31130 leader_election.cc:290] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:30.085544 30873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7470dd49ce524f4aaf3faf7739fe5bd7" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:30.086344 30873 raft_consensus.cc:3060] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.065938 31122 leader_election.cc:290] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:30.089164 30923 ts_tablet_manager.cc:1434] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.000s
I20260709 12:20:30.090445 30923 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:30.029469 30925 raft_consensus.cc:385] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.094134 30925 raft_consensus.cc:740] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.095824 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:30.097599 30689 leader_election.cc:343] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.097864 30923 tablet_bootstrap.cc:654] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.099702 30925 consensus_queue.cc:260] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.102114 30874 raft_consensus.cc:2468] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:30.105067 30873 raft_consensus.cc:2468] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:30.106947 30689 leader_election.cc:304] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:30.108512 30957 raft_consensus.cc:2804] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.108898 30763 leader_election.cc:304] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.109056 31121 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.109876 31130 raft_consensus.cc:2804] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.108922 30957 raft_consensus.cc:493] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.109890 31121 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.110455 30650 raft_consensus.cc:1275] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:30.110771 30957 raft_consensus.cc:3060] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.110553 31130 raft_consensus.cc:697] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:30.112108 30991 consensus_queue.cc:1048] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.112085 31130 consensus_queue.cc:237] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.115520 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:30.117667 30838 leader_election.cc:343] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.118862 31121 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:30.120808 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:30.123306 30839 leader_election.cc:343] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.124058 30839 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:30.127091 31118 raft_consensus.cc:2749] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:30.007686 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 63241.1us
I20260709 12:20:30.132994 30615 leader_election.cc:304] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.141248 30922 tablet_bootstrap.cc:654] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.102535 30925 ts_tablet_manager.cc:1434] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.093s	user 0.010s	sys 0.000s
I20260709 12:20:30.134979 30965 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.153852 30925 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:30.148620 31119 raft_consensus.cc:2749] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:30.141286 30923 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:30.158396 30923 ts_tablet_manager.cc:1403] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.068s	user 0.027s	sys 0.002s
I20260709 12:20:30.118906 30957 raft_consensus.cc:515] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.171742 30925 tablet_bootstrap.cc:654] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.179826 30923 raft_consensus.cc:359] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.181149 30923 raft_consensus.cc:385] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.181469 30923 raft_consensus.cc:740] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.176012 30873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:30.183737 30873 raft_consensus.cc:3060] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.184106 30874 raft_consensus.cc:1275] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:30.186234 31055 consensus_queue.cc:1048] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:30.191291 31124 raft_consensus.cc:493] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.191803 31144 raft_consensus.cc:493] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.191910 31124 raft_consensus.cc:515] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.192423 31144 raft_consensus.cc:515] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.189051 30923 consensus_queue.cc:260] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.198817 30928 raft_consensus.cc:493] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23ba075510134f3c8caba2370fff0d40)
I20260709 12:20:30.200258 30928 raft_consensus.cc:515] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.201495 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.202571 30725 raft_consensus.cc:2468] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:30.204862 30764 leader_election.cc:304] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:30.206138 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:30.207677 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.206161 31125 raft_consensus.cc:2804] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.209331 31125 raft_consensus.cc:493] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.209707 31125 raft_consensus.cc:3060] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.210180 30928 leader_election.cc:290] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:30.215404 30763 leader_election.cc:304] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.216498 31124 leader_election.cc:290] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:30.219825 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5f378411424826909f1f1f3aff4409" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.222913 30965 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:30.223366 30764 leader_election.cc:343] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.225255 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:30.226686 31144 leader_election.cc:290] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:30.226912 30838 leader_election.cc:343] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.227763 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:30.217110 31124 raft_consensus.cc:2749] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:30.239845 30923 ts_tablet_manager.cc:1434] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.081s	user 0.008s	sys 0.000s
I20260709 12:20:30.241145 30923 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:30.242062 30957 leader_election.cc:290] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:30.249948 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:30.237854 30800 raft_consensus.cc:3060] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.261204 31125 raft_consensus.cc:515] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.263201 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5f378411424826909f1f1f3aff4409" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:30.264788 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:30.269188 30689 leader_election.cc:343] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:30.270617 30839 consensus_peers.cc:597] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.274278 30647 raft_consensus.cc:2468] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:30.275827 30651 raft_consensus.cc:2468] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:30.277879 30873 raft_consensus.cc:2468] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:30.279911 31125 leader_election.cc:290] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:30.281713 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:30.295262 30921 tablet_bootstrap.cc:492] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:30.296669 30923 tablet_bootstrap.cc:654] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.299685 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.300180 30922 tablet_bootstrap.cc:492] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:30.301852 30921 ts_tablet_manager.cc:1403] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.337s	user 0.054s	sys 0.064s
I20260709 12:20:30.302177 30922 ts_tablet_manager.cc:1403] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.265s	user 0.031s	sys 0.019s
I20260709 12:20:30.301095 30952 raft_consensus.cc:493] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.303704 30952 raft_consensus.cc:515] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.307296 30723 raft_consensus.cc:3060] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.308617 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:30.309557 30647 raft_consensus.cc:3060] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.311293 30689 leader_election.cc:304] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:30.312906 31055 raft_consensus.cc:2804] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.314065 30983 raft_consensus.cc:493] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.314658 30983 raft_consensus.cc:515] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.307725 30952 leader_election.cc:290] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:30.308039 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:30.320510 30614 leader_election.cc:343] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.321455 30983 leader_election.cc:290] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:30.322080 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9873e94bae58472f8de439d5dc1835a6" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:30.323055 30796 raft_consensus.cc:2393] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:30.324079 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.328038 30763 leader_election.cc:304] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.331135 30647 raft_consensus.cc:2468] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:30.331965 31124 raft_consensus.cc:2804] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.305600 30922 raft_consensus.cc:359] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.332554 31124 raft_consensus.cc:493] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.332762 30922 raft_consensus.cc:385] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.333058 31124 raft_consensus.cc:3060] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.333244 30922 raft_consensus.cc:740] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.334221 30922 consensus_queue.cc:260] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.336508 30925 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:30.335981 30549 catalog_manager.cc:5697] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.337091 30925 ts_tablet_manager.cc:1403] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.184s	user 0.024s	sys 0.007s
I20260709 12:20:30.341151 30763 leader_election.cc:304] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:30.343410 30928 raft_consensus.cc:2804] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.343946 30928 raft_consensus.cc:697] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:30.344856 30928 consensus_queue.cc:237] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.315714 31055 raft_consensus.cc:697] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:30.349298 31055 consensus_queue.cc:237] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.336431 30922 ts_tablet_manager.cc:1434] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.000s
I20260709 12:20:30.354001 30922 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:30.362686 30800 raft_consensus.cc:2468] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:30.366511 30839 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:30.366546 30925 raft_consensus.cc:359] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.374250 30925 raft_consensus.cc:385] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.374707 30925 raft_consensus.cc:740] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.386266 30921 raft_consensus.cc:359] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.387456 30921 raft_consensus.cc:385] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.375854 31002 raft_consensus.cc:493] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.387799 30921 raft_consensus.cc:740] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.406351 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.389370 30965 raft_consensus.cc:2804] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.410356 30965 raft_consensus.cc:697] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:30.411720 30965 consensus_queue.cc:237] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.398677 31002 raft_consensus.cc:515] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.419456 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:30.419981 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:30.423740 30690 leader_election.cc:343] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:30.423928 30615 leader_election.cc:343] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.426748 30615 leader_election.cc:304] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.427896 31119 raft_consensus.cc:2749] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:30.428715 30615 leader_election.cc:343] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.429329 30615 leader_election.cc:304] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.431022 30723 raft_consensus.cc:2468] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:30.433521 31002 leader_election.cc:290] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:30.434865 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:30.436740 30689 leader_election.cc:343] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.438135 30689 leader_election.cc:304] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:30.439391 31002 raft_consensus.cc:2749] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:30.436646 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:30.444815 31121 consensus_queue.cc:1048] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.446871 30922 tablet_bootstrap.cc:654] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.437331 30549 catalog_manager.cc:5697] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:30.400241 30925 consensus_queue.cc:260] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.445331 30921 consensus_queue.cc:260] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:30.472019 30615 leader_election.cc:343] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.472898 30615 leader_election.cc:304] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.474078 31054 raft_consensus.cc:2749] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:30.466084 31119 raft_consensus.cc:2749] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:30.437062 31124 raft_consensus.cc:515] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.485277 30925 ts_tablet_manager.cc:1434] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.148s	user 0.011s	sys 0.000s
I20260709 12:20:30.491479 30925 tablet_bootstrap.cc:492] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:30.490875 30921 ts_tablet_manager.cc:1434] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.189s	user 0.008s	sys 0.001s
I20260709 12:20:30.494477 30921 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:30.494632 30548 catalog_manager.cc:5697] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:30.497189 31124 leader_election.cc:290] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:30.505282 30923 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:30.506018 30923 ts_tablet_manager.cc:1403] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.265s	user 0.067s	sys 0.030s
I20260709 12:20:30.508510 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5f378411424826909f1f1f3aff4409" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:30.509342 30651 raft_consensus.cc:3060] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.501225 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5f378411424826909f1f1f3aff4409" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:30.510735 30923 raft_consensus.cc:359] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.520334 30923 raft_consensus.cc:385] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.520854 30923 raft_consensus.cc:740] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.522835 30921 tablet_bootstrap.cc:654] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.523213 30923 consensus_queue.cc:260] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:30.540910 30764 leader_election.cc:343] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.551095 31164 raft_consensus.cc:493] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.551791 31164 raft_consensus.cc:515] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.556736 30925 tablet_bootstrap.cc:654] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.560523 30928 consensus_queue.cc:1048] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.570766 31125 raft_consensus.cc:493] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.573040 31125 raft_consensus.cc:515] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.574676 31164 leader_election.cc:290] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:30.581348 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1243becf9c9421bb2d5083f71c1e292" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:30.582469 30648 raft_consensus.cc:2468] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:30.585282 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:30.586320 30872 raft_consensus.cc:2468] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:30.587245 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.588128 30725 raft_consensus.cc:2468] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:30.589190 30763 leader_election.cc:304] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:30.590574 30928 raft_consensus.cc:2804] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.591044 30928 raft_consensus.cc:493] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.591454 30928 raft_consensus.cc:3060] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.587505 31125 leader_election.cc:290] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:30.595362 30838 leader_election.cc:304] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:30.596746 31164 raft_consensus.cc:2804] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.597239 31164 raft_consensus.cc:493] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.598299 31164 raft_consensus.cc:3060] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.597088 30549 catalog_manager.cc:5697] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.607998 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1243becf9c9421bb2d5083f71c1e292" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.633122 30922 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:30.634601 30922 ts_tablet_manager.cc:1403] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.281s	user 0.057s	sys 0.053s
I20260709 12:20:30.639752 30923 ts_tablet_manager.cc:1434] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.133s	user 0.007s	sys 0.006s
I20260709 12:20:30.641358 30923 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:30.656426 30651 raft_consensus.cc:2468] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:30.633649 30928 raft_consensus.cc:515] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.659658 30928 leader_election.cc:290] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:30.660693 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:30.661396 30725 raft_consensus.cc:3060] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.661067 31054 consensus_queue.cc:1048] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.661813 30763 leader_election.cc:304] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
W20260709 12:20:30.653820 30839 leader_election.cc:343] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.672837 30725 raft_consensus.cc:2468] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:30.675523 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:30.675526 30764 leader_election.cc:304] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:30.679663 31125 raft_consensus.cc:2804] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.687753 30922 raft_consensus.cc:359] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.688601 30922 raft_consensus.cc:385] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.689044 30922 raft_consensus.cc:740] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.688560 31125 raft_consensus.cc:697] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:30.688892 30549 catalog_manager.cc:5697] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.697368 30921 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:30.698097 30921 ts_tablet_manager.cc:1403] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.204s	user 0.025s	sys 0.040s
I20260709 12:20:30.691468 31054 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.689183 31124 raft_consensus.cc:2804] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.703450 31124 raft_consensus.cc:697] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:30.693673 30872 raft_consensus.cc:3060] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.701342 30922 consensus_queue.cc:260] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.708591 31124 consensus_queue.cc:237] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.709897 31125 consensus_queue.cc:237] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.727454 31164 raft_consensus.cc:515] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.728740 30925 tablet_bootstrap.cc:492] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:30.729446 30925 ts_tablet_manager.cc:1403] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.238s	user 0.019s	sys 0.020s
I20260709 12:20:30.732426 31119 consensus_queue.cc:1048] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.704161 31054 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.741871 30922 ts_tablet_manager.cc:1434] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.107s	user 0.010s	sys 0.000s
I20260709 12:20:30.743638 30922 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:30.756394 30921 raft_consensus.cc:359] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.757614 30921 raft_consensus.cc:385] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.758002 30921 raft_consensus.cc:740] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.759205 30921 consensus_queue.cc:260] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.762641 30921 ts_tablet_manager.cc:1434] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.003s
W20260709 12:20:30.763696 30764 consensus_peers.cc:597] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.764386 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:30.764294 30921 tablet_bootstrap.cc:492] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:30.762847 31054 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:30.779512 30923 tablet_bootstrap.cc:654] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.784075 30925 raft_consensus.cc:359] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.785439 30925 raft_consensus.cc:385] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.786274 30547 catalog_manager.cc:5697] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
W20260709 12:20:30.791182 30689 consensus_peers.cc:597] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:30.798241 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:30.800261 30796 raft_consensus.cc:2393] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:30.786592 30925 raft_consensus.cc:740] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.806982 30925 consensus_queue.cc:260] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.805819 31164 leader_election.cc:290] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:30.809844 30925 ts_tablet_manager.cc:1434] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.080s	user 0.008s	sys 0.000s
I20260709 12:20:30.810189 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1243becf9c9421bb2d5083f71c1e292" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:30.812000 30925 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
W20260709 12:20:30.812539 30839 leader_election.cc:343] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.812844 30922 tablet_bootstrap.cc:654] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.824486 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1243becf9c9421bb2d5083f71c1e292" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:30.836503 30647 raft_consensus.cc:3060] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.847714 30615 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:30.849587 31158 raft_consensus.cc:3060] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.851040 30872 raft_consensus.cc:2468] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:30.863046 30928 consensus_queue.cc:1048] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.866677 30925 tablet_bootstrap.cc:654] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.871554 30923 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:30.860175 30546 catalog_manager.cc:5697] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.881280 30923 ts_tablet_manager.cc:1403] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.240s	user 0.008s	sys 0.017s
I20260709 12:20:30.874032 30647 raft_consensus.cc:2468] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:30.883489 30838 leader_election.cc:304] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:30.884986 31163 raft_consensus.cc:2804] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:30.885605 31163 raft_consensus.cc:697] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:30.886590 31163 consensus_queue.cc:237] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.871582 30921 tablet_bootstrap.cc:654] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.896996 30957 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.898334 30923 raft_consensus.cc:359] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.898392 30957 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:30.899498 30923 raft_consensus.cc:385] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.899866 30923 raft_consensus.cc:740] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.901105 30957 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:30.902755 31158 raft_consensus.cc:2749] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:30.903254 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:30.912076 31175 consensus_queue.cc:1048] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:30.905825 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:30.907240 31164 raft_consensus.cc:493] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:30.915670 31164 raft_consensus.cc:515] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:30.916348 30689 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:30.917814 30957 raft_consensus.cc:2749] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:30.920500 31164 leader_election.cc:290] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:30.923657 30957 consensus_queue.cc:1048] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.931613 30922 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:30.932518 30922 ts_tablet_manager.cc:1403] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.189s	user 0.018s	sys 0.003s
I20260709 12:20:30.923929 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:30.933964 30726 raft_consensus.cc:2468] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:30.935917 30839 leader_election.cc:304] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:30.937065 31164 raft_consensus.cc:2804] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:30.937667 31164 raft_consensus.cc:493] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:30.938047 31164 raft_consensus.cc:3060] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:30.944698 30549 catalog_manager.cc:5697] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:30.948738 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:30.954650 30922 raft_consensus.cc:359] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.951390 30923 consensus_queue.cc:260] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:30.956440 30922 raft_consensus.cc:385] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:30.956895 30922 raft_consensus.cc:740] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:30.960932 30922 consensus_queue.cc:260] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:30.962141 30923 ts_tablet_manager.cc:1434] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.080s	user 0.007s	sys 0.001s
I20260709 12:20:30.963712 30923 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:30.965801 30839 leader_election.cc:343] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:30.966364 30922 ts_tablet_manager.cc:1434] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.002s
I20260709 12:20:30.967612 30922 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:30.977301 30923 tablet_bootstrap.cc:654] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:30.982272 31124 consensus_queue.cc:1048] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:30.989809 31164 raft_consensus.cc:515] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.003157 31164 leader_election.cc:290] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:31.009869 30921 tablet_bootstrap.cc:492] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:31.010550 30921 ts_tablet_manager.cc:1403] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.247s	user 0.018s	sys 0.007s
I20260709 12:20:31.008639 31124 raft_consensus.cc:493] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.017695 31124 raft_consensus.cc:515] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.021072 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:31.023104 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.024008 30648 raft_consensus.cc:2468] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:31.027218 30763 leader_election.cc:304] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:31.028651 31144 raft_consensus.cc:2804] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:31.029060 31144 raft_consensus.cc:493] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.029369 31144 raft_consensus.cc:3060] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:31.030948 30839 leader_election.cc:343] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.033790 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:31.034575 30726 raft_consensus.cc:3060] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.035001 30922 tablet_bootstrap.cc:654] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.035542 31121 consensus_queue.cc:1048] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:20:31.041824 30921 raft_consensus.cc:359] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.044193 30921 raft_consensus.cc:385] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.044508 30921 raft_consensus.cc:740] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.050864 30923 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:31.056342 30925 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:31.057133 30925 ts_tablet_manager.cc:1403] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.245s	user 0.036s	sys 0.019s
I20260709 12:20:31.058661 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:31.061187 30764 leader_election.cc:343] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.063462 30923 ts_tablet_manager.cc:1403] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.100s	user 0.013s	sys 0.024s
I20260709 12:20:31.056748 31124 leader_election.cc:290] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:31.067137 30923 raft_consensus.cc:359] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.069700 30925 raft_consensus.cc:359] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.070564 30925 raft_consensus.cc:385] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.070861 30925 raft_consensus.cc:740] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.071691 30925 consensus_queue.cc:260] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.072336 31144 raft_consensus.cc:515] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.069468 30921 consensus_queue.cc:260] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.074242 30925 ts_tablet_manager.cc:1434] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:20:31.075445 30925 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:31.075496 30921 ts_tablet_manager.cc:1434] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.064s	user 0.006s	sys 0.001s
I20260709 12:20:31.076659 30921 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:31.078611 30923 raft_consensus.cc:385] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.079133 30923 raft_consensus.cc:740] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.080204 30923 consensus_queue.cc:260] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.082747 30923 ts_tablet_manager.cc:1434] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:31.083900 30726 raft_consensus.cc:2468] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:31.086968 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:31.089555 30925 tablet_bootstrap.cc:654] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.083999 30923 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:31.094038 30648 raft_consensus.cc:3060] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.095527 31158 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.096100 31158 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.089859 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
W20260709 12:20:31.099102 30764 leader_election.cc:343] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.100661 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:31.104009 30615 leader_election.cc:343] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.090324 31144 leader_election.cc:290] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:31.104954 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:31.105710 30875 raft_consensus.cc:2468] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:31.107179 30648 raft_consensus.cc:2468] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:31.107743 30723 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.109439 30614 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:31.110664 31054 raft_consensus.cc:2804] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:31.111078 31054 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.111416 31054 raft_consensus.cc:3060] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.112339 30763 leader_election.cc:304] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:31.115058 30839 leader_election.cc:304] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:31.116631 31158 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
W20260709 12:20:31.121248 30764 consensus_peers.cc:597] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:31.092890 30922 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:31.122283 31164 raft_consensus.cc:2804] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:31.122592 30922 ts_tablet_manager.cc:1403] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.155s	user 0.015s	sys 0.000s
I20260709 12:20:31.123251 31164 raft_consensus.cc:697] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:31.124257 31164 consensus_queue.cc:237] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.113673 31144 raft_consensus.cc:2804] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:31.126585 31144 raft_consensus.cc:697] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:31.130309 31144 consensus_queue.cc:237] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.142453 31130 consensus_queue.cc:1048] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:31.144315 31054 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.147274 30921 tablet_bootstrap.cc:654] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.147814 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
W20260709 12:20:31.150828 30615 leader_election.cc:343] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.158802 30922 raft_consensus.cc:359] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.163059 30922 raft_consensus.cc:385] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.163419 30922 raft_consensus.cc:740] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.167156 30923 tablet_bootstrap.cc:654] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.170256 31130 consensus_queue.cc:1048] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:31.172597 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:31.173646 30875 raft_consensus.cc:3060] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.184016 31124 raft_consensus.cc:493] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.184847 31124 raft_consensus.cc:515] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.187970 31124 leader_election.cc:290] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:31.190781 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.202117 30922 consensus_queue.cc:260] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.206291 31054 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:31.207078 30922 ts_tablet_manager.cc:1434] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.084s	user 0.009s	sys 0.001s
I20260709 12:20:31.208814 30922 tablet_bootstrap.cc:492] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:31.212395 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:31.237095 30921 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:31.237856 30921 ts_tablet_manager.cc:1403] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.161s	user 0.026s	sys 0.008s
I20260709 12:20:31.241595 30921 raft_consensus.cc:359] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.242623 30921 raft_consensus.cc:385] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.243001 30921 raft_consensus.cc:740] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.244180 30921 consensus_queue.cc:260] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.247176 30921 ts_tablet_manager.cc:1434] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:20:31.245370 30875 raft_consensus.cc:2468] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:31.248528 30921 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:31.251096 30651 raft_consensus.cc:3060] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.254881 30923 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:31.261730 30923 ts_tablet_manager.cc:1403] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.178s	user 0.017s	sys 0.001s
I20260709 12:20:31.268353 30923 raft_consensus.cc:359] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.258575 30614 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:31.271708 30923 raft_consensus.cc:385] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.272595 30923 raft_consensus.cc:740] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.273484 31119 raft_consensus.cc:2804] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:31.274242 31119 raft_consensus.cc:697] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:31.274540 30923 consensus_queue.cc:260] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.275264 31119 consensus_queue.cc:237] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.279366 30763 leader_election.cc:304] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:31.282325 31130 raft_consensus.cc:2749] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.285715 30800 raft_consensus.cc:3060] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.313606 31130 consensus_queue.cc:1048] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:31.260115 30922 tablet_bootstrap.cc:654] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.324860 30548 catalog_manager.cc:5697] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:31.331168 30923 ts_tablet_manager.cc:1434] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.068s	user 0.009s	sys 0.000s
I20260709 12:20:31.338521 30923 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:31.360910 30923 tablet_bootstrap.cc:654] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.384271 30921 tablet_bootstrap.cc:654] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.384703 30548 catalog_manager.cc:5697] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:31.410812 30923 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:31.411461 30923 ts_tablet_manager.cc:1403] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.076s	user 0.010s	sys 0.008s
I20260709 12:20:31.414816 30923 raft_consensus.cc:359] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.415576 30923 raft_consensus.cc:385] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.415819 30923 raft_consensus.cc:740] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.416504 30923 consensus_queue.cc:260] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.419728 31158 raft_consensus.cc:493] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb6634546938405dac9eabbadd34d2b1)
I20260709 12:20:31.423148 31158 raft_consensus.cc:515] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.426232 30923 ts_tablet_manager.cc:1434] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.006s
I20260709 12:20:31.427376 30923 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:31.432406 30925 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:31.433089 30923 tablet_bootstrap.cc:654] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.433142 30925 ts_tablet_manager.cc:1403] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.358s	user 0.091s	sys 0.120s
I20260709 12:20:31.437103 30922 tablet_bootstrap.cc:492] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:31.437718 30922 ts_tablet_manager.cc:1403] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.229s	user 0.016s	sys 0.006s
I20260709 12:20:31.438174 30925 raft_consensus.cc:359] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.439442 30925 raft_consensus.cc:385] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.439854 30925 raft_consensus.cc:740] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.441033 30925 consensus_queue.cc:260] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.448491 30925 ts_tablet_manager.cc:1434] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.002s
I20260709 12:20:31.445393 30923 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:31.451470 30923 ts_tablet_manager.cc:1403] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.008s
I20260709 12:20:31.451858 30921 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:31.452584 30921 ts_tablet_manager.cc:1403] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.204s	user 0.007s	sys 0.007s
I20260709 12:20:31.454818 31163 raft_consensus.cc:493] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.455549 30925 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:31.455431 31163 raft_consensus.cc:515] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.456275 31158 leader_election.cc:290] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:31.456108 30921 raft_consensus.cc:359] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.457067 30921 raft_consensus.cc:385] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.457410 30921 raft_consensus.cc:740] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.457734 31163 leader_election.cc:290] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:31.440939 30922 raft_consensus.cc:359] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.458469 30922 raft_consensus.cc:385] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.458873 30922 raft_consensus.cc:740] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.459060 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:31.459470 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:31.459719 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.460492 30651 raft_consensus.cc:2468] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:31.460177 30921 consensus_queue.cc:260] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.462162 30838 leader_election.cc:304] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:31.463230 30921 ts_tablet_manager.cc:1434] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.004s
W20260709 12:20:31.463588 30839 leader_election.cc:343] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.464218 31121 raft_consensus.cc:2804] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:31.464568 30921 tablet_bootstrap.cc:492] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:31.454852 30923 raft_consensus.cc:359] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.464777 31121 raft_consensus.cc:493] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.466735 31121 raft_consensus.cc:3060] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.459800 30922 consensus_queue.cc:260] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.462101 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:31.466195 30923 raft_consensus.cc:385] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.471483 30923 raft_consensus.cc:740] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.471740 30615 leader_election.cc:304] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:31.472033 30922 ts_tablet_manager.cc:1434] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.001s
W20260709 12:20:31.475201 30839 consensus_peers.cc:597] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:31.472429 30923 consensus_queue.cc:260] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.476801 30922 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:31.478545 30923 ts_tablet_manager.cc:1434] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 12:20:31.480077 31121 raft_consensus.cc:515] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.483771 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:31.484368 30923 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:31.484601 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:31.485483 30648 raft_consensus.cc:3060] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.486809 31164 consensus_queue.cc:1048] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:31.493173 30921 tablet_bootstrap.cc:654] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.495436 31198 raft_consensus.cc:2749] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:31.485473 30839 leader_election.cc:343] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.503331 30957 raft_consensus.cc:493] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.503659 30923 tablet_bootstrap.cc:654] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.504136 30957 raft_consensus.cc:515] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.506853 30648 raft_consensus.cc:2468] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:31.507464 30957 leader_election.cc:290] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:31.509927 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c766be78f1467795131bc36e7766e9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:31.512255 30689 leader_election.cc:343] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.511564 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c766be78f1467795131bc36e7766e9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:31.515326 30800 raft_consensus.cc:2468] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:31.497340 30922 tablet_bootstrap.cc:654] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.517714 30690 leader_election.cc:304] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:31.519421 30925 tablet_bootstrap.cc:654] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.522697 31121 leader_election.cc:290] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:31.523365 30957 raft_consensus.cc:2804] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:31.473471 30546 catalog_manager.cc:5697] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:31.514366 30838 leader_election.cc:304] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:31.531481 31164 raft_consensus.cc:2804] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:31.532872 31164 raft_consensus.cc:697] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:31.534965 31164 consensus_queue.cc:237] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.524540 30957 raft_consensus.cc:493] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.537953 31130 raft_consensus.cc:493] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.538827 31130 raft_consensus.cc:515] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.543881 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c95a27bd5694416aa5dae696482e8ec4" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.548929 30648 raft_consensus.cc:2468] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:31.546960 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c95a27bd5694416aa5dae696482e8ec4" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:31.550194 30923 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:31.550812 30875 raft_consensus.cc:2468] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:31.551076 30923 ts_tablet_manager.cc:1403] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.067s	user 0.013s	sys 0.011s
I20260709 12:20:31.551076 30763 leader_election.cc:304] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:31.552637 31144 raft_consensus.cc:2804] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:31.552937 31198 raft_consensus.cc:493] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.553838 31198 raft_consensus.cc:515] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.554765 30923 raft_consensus.cc:359] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.555585 30923 raft_consensus.cc:385] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.555886 30923 raft_consensus.cc:740] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.556730 31198 leader_election.cc:290] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:31.553040 31144 raft_consensus.cc:493] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.557914 31144 raft_consensus.cc:3060] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.561259 30925 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:31.561883 30925 ts_tablet_manager.cc:1403] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.107s	user 0.020s	sys 0.000s
I20260709 12:20:31.562067 30922 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:31.541585 31130 leader_election.cc:290] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:31.564823 30921 tablet_bootstrap.cc:492] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:31.565493 30921 ts_tablet_manager.cc:1403] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.101s	user 0.014s	sys 0.000s
I20260709 12:20:31.562784 30922 ts_tablet_manager.cc:1403] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.087s	user 0.011s	sys 0.004s
W20260709 12:20:31.568985 30839 consensus_peers.cc:597] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:31.566282 30925 raft_consensus.cc:359] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.571435 30925 raft_consensus.cc:385] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.571732 30925 raft_consensus.cc:740] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.571812 30921 raft_consensus.cc:359] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.572803 30921 raft_consensus.cc:385] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.572598 30923 consensus_queue.cc:260] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.573201 30921 raft_consensus.cc:740] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.574837 30922 raft_consensus.cc:359] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.575637 30922 raft_consensus.cc:385] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.575919 30922 raft_consensus.cc:740] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.573928 30925 consensus_queue.cc:260] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.576678 30922 consensus_queue.cc:260] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.576294 30923 ts_tablet_manager.cc:1434] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20260709 12:20:31.578184 30921 consensus_queue.cc:260] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.578835 30923 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:31.580586 30922 ts_tablet_manager.cc:1434] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:20:31.581851 30922 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:31.583940 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1994c4ce566142fa922312f566387067" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:31.538097 30957 raft_consensus.cc:3060] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.602638 30921 ts_tablet_manager.cc:1434] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.001s
I20260709 12:20:31.593904 30929 raft_consensus.cc:493] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.612043 30921 tablet_bootstrap.cc:492] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:31.602254 30991 raft_consensus.cc:493] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.605855 31119 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.620599 31119 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.603643 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1994c4ce566142fa922312f566387067" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:31.624048 31119 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:31.624217 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:31.626338 30614 leader_election.cc:343] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.617553 30991 raft_consensus.cc:515] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.631184 31124 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:31.600530 30615 leader_election.cc:343] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.631844 31124 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:31.633608 30615 leader_election.cc:343] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.634261 30615 leader_election.cc:304] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:31.634570 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:31.638055 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:31.637982 31158 raft_consensus.cc:2749] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.637306 30548 catalog_manager.cc:5697] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:31.614775 30929 raft_consensus.cc:515] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.640419 30725 raft_consensus.cc:3060] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.653345 31124 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:31.653646 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.654695 30647 raft_consensus.cc:2393] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6634546938405dac9eabbadd34d2b1 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:31.655150 30991 leader_election.cc:290] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:31.660152 31198 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.660584 31214 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.642652 30929 leader_election.cc:290] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:31.660969 31198 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.666565 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5f378411424826909f1f1f3aff4409" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.675302 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:31.677577 30615 leader_election.cc:343] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.681165 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.681305 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:31.682474 30875 raft_consensus.cc:2393] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:31.683463 30690 leader_election.cc:304] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:31.685066 30923 tablet_bootstrap.cc:654] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.685400 30957 raft_consensus.cc:515] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.688537 30957 leader_election.cc:290] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:31.691524 31124 consensus_queue.cc:1048] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:31.692224 30922 tablet_bootstrap.cc:654] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.692924 30925 ts_tablet_manager.cc:1434] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.131s	user 0.012s	sys 0.001s
I20260709 12:20:31.675833 31214 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.694850 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:31.695947 30921 tablet_bootstrap.cc:654] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.700795 30763 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:31.700810 30872 raft_consensus.cc:2468] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:31.702145 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c766be78f1467795131bc36e7766e9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:31.702889 30799 raft_consensus.cc:3060] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.703401 30614 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:31.704625 31054 raft_consensus.cc:2804] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:31.705034 31054 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:31.705416 31054 raft_consensus.cc:3060] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.705896 30929 raft_consensus.cc:2749] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.630050 31144 raft_consensus.cc:515] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.696019 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e5f378411424826909f1f1f3aff4409" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:31.698741 30925 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:31.713402 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:31.714978 30690 leader_election.cc:304] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:31.718799 30991 raft_consensus.cc:2749] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.725204 31125 consensus_queue.cc:1048] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:31.726003 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:31.697208 31198 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:31.711241 31124 raft_consensus.cc:3060] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.711704 31214 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:31.697471 30615 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:31.750785 30724 raft_consensus.cc:3060] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.751682 30725 raft_consensus.cc:3060] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.744679 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c766be78f1467795131bc36e7766e9" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:31.762830 30925 tablet_bootstrap.cc:654] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.762188 31119 consensus_queue.cc:1048] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20260709 12:20:31.770390 30839 leader_election.cc:343] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:31.778684 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 101182us
W20260709 12:20:31.787690 30689 leader_election.cc:343] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.793005 31144 leader_election.cc:290] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:31.795403 31124 raft_consensus.cc:2749] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.796042 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c95a27bd5694416aa5dae696482e8ec4" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:31.796355 31054 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.798470 30922 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:31.806470 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:31.786129 31198 raft_consensus.cc:2749] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.815798 30647 raft_consensus.cc:2393] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:31.817314 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:31.817552 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:31.818086 30872 raft_consensus.cc:3060] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:31.819272 30615 leader_election.cc:343] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.812863 30923 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:31.826277 30923 ts_tablet_manager.cc:1403] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.248s	user 0.016s	sys 0.016s
I20260709 12:20:31.813110 30922 ts_tablet_manager.cc:1403] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.232s	user 0.020s	sys 0.008s
I20260709 12:20:31.825340 30651 raft_consensus.cc:3060] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.831084 30838 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:31.833811 31054 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:31.841144 31214 consensus_queue.cc:1048] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:31.833335 30615 leader_election.cc:343] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:31.841874 30799 raft_consensus.cc:2468] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:31.844513 31198 consensus_queue.cc:1048] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:31.844805 30690 leader_election.cc:304] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:31.848356 30923 raft_consensus.cc:359] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.850031 30923 raft_consensus.cc:385] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.850436 30923 raft_consensus.cc:740] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.851359 30923 consensus_queue.cc:260] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:31.852875 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c95a27bd5694416aa5dae696482e8ec4" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:31.853928 30874 raft_consensus.cc:3060] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:31.854369 30923 ts_tablet_manager.cc:1434] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.001s
I20260709 12:20:31.855670 30923 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:31.859156 30957 raft_consensus.cc:2804] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:31.869369 30957 raft_consensus.cc:697] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:31.870767 30921 tablet_bootstrap.cc:492] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:31.872190 30921 ts_tablet_manager.cc:1403] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.268s	user 0.012s	sys 0.011s
I20260709 12:20:31.870507 30922 raft_consensus.cc:359] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.873153 30922 raft_consensus.cc:385] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.873672 30922 raft_consensus.cc:740] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.874424 30922 consensus_queue.cc:260] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.870517 30957 consensus_queue.cc:237] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.886111 30921 raft_consensus.cc:359] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.887276 30921 raft_consensus.cc:385] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.890527 30922 ts_tablet_manager.cc:1434] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.000s
I20260709 12:20:31.887655 30921 raft_consensus.cc:740] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.907673 30921 consensus_queue.cc:260] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.910068 30921 ts_tablet_manager.cc:1434] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.002s
I20260709 12:20:31.912479 30922 tablet_bootstrap.cc:492] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:31.915935 30921 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:31.930835 30925 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:31.931773 30925 ts_tablet_manager.cc:1403] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.233s	user 0.011s	sys 0.013s
I20260709 12:20:31.946372 30957 raft_consensus.cc:493] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:31.948515 30957 raft_consensus.cc:515] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:31.964115 30957 leader_election.cc:290] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:31.954159 30872 raft_consensus.cc:2468] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:31.970305 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:31.983772 30925 raft_consensus.cc:359] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:31.995260 30925 raft_consensus.cc:385] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:31.995611 30925 raft_consensus.cc:740] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:31.984577 31217 raft_consensus.cc:2749] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:31.996743 30921 tablet_bootstrap.cc:654] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:31.997328 30923 tablet_bootstrap.cc:654] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:31.991429 30690 leader_election.cc:343] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.001184 30651 raft_consensus.cc:2468] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:32.008020 30763 leader_election.cc:304] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:31.988667 30614 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19
I20260709 12:20:32.009025 31130 raft_consensus.cc:2804] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:32.009821 31130 raft_consensus.cc:697] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:32.010092 31119 raft_consensus.cc:2804] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:32.010728 31119 raft_consensus.cc:697] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:31.991979 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:32.011240 31130 consensus_queue.cc:237] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.011605 31119 consensus_queue.cc:237] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:31.986622 30874 raft_consensus.cc:2468] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:32.029246 30922 tablet_bootstrap.cc:654] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:32.029371 30689 leader_election.cc:343] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.040243 30689 leader_election.cc:304] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:32.042377 31219 raft_consensus.cc:2749] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:32.043869 30921 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:32.045815 30921 ts_tablet_manager.cc:1403] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.134s	user 0.013s	sys 0.004s
I20260709 12:20:32.053973 30923 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:32.054890 30923 ts_tablet_manager.cc:1403] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.199s	user 0.022s	sys 0.007s
I20260709 12:20:32.063403 30921 raft_consensus.cc:359] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.065163 30921 raft_consensus.cc:385] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.065519 30921 raft_consensus.cc:740] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.067552 30922 tablet_bootstrap.cc:492] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:32.068308 30922 ts_tablet_manager.cc:1403] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.156s	user 0.020s	sys 0.000s
I20260709 12:20:32.068406 30921 consensus_queue.cc:260] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.069219 30925 consensus_queue.cc:260] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.071292 30921 ts_tablet_manager.cc:1434] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:20:32.071349 30922 raft_consensus.cc:359] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.072436 30922 raft_consensus.cc:385] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.072690 30922 raft_consensus.cc:740] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.073078 30921 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:32.073385 30922 consensus_queue.cc:260] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.071409 30925 ts_tablet_manager.cc:1434] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.139s	user 0.010s	sys 0.001s
I20260709 12:20:32.076072 30925 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:32.076682 30922 ts_tablet_manager.cc:1434] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:32.078253 30922 tablet_bootstrap.cc:492] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:32.086802 30922 tablet_bootstrap.cc:654] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:32.088258 30839 consensus_peers.cc:597] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:32.094240 30921 tablet_bootstrap.cc:654] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.094424 30922 tablet_bootstrap.cc:492] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:32.095100 30922 ts_tablet_manager.cc:1403] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 12:20:32.097580 30546 catalog_manager.cc:5697] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.110352 31214 consensus_queue.cc:1048] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:32.110901 30921 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:32.111801 30921 ts_tablet_manager.cc:1403] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.005s
I20260709 12:20:32.115268 30921 raft_consensus.cc:359] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.116197 30921 raft_consensus.cc:385] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.098297 30922 raft_consensus.cc:359] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.116896 30921 raft_consensus.cc:740] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.117347 30922 raft_consensus.cc:385] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.118039 31130 raft_consensus.cc:493] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.118906 31130 raft_consensus.cc:515] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.121435 31130 leader_election.cc:290] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:32.122836 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:32.117758 30922 raft_consensus.cc:740] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.123816 30723 raft_consensus.cc:2468] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:32.124246 30922 consensus_queue.cc:260] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.125813 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:32.127475 30764 leader_election.cc:304] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:32.127717 30922 ts_tablet_manager.cc:1434] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
W20260709 12:20:32.128144 30763 leader_election.cc:343] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.129035 31130 raft_consensus.cc:2804] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:32.128831 30921 consensus_queue.cc:260] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.129454 31130 raft_consensus.cc:493] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:32.130069 31130 raft_consensus.cc:3060] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.130903 30922 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:32.131875 30921 ts_tablet_manager.cc:1434] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:20:32.133296 30921 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:32.139446 31130 raft_consensus.cc:515] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.143132 30922 tablet_bootstrap.cc:654] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.147192 30921 tablet_bootstrap.cc:654] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.161418 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:32.162313 30723 raft_consensus.cc:3060] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.058470 30923 raft_consensus.cc:359] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.171435 30923 raft_consensus.cc:385] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.179543 30923 raft_consensus.cc:740] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.182315 30923 consensus_queue.cc:260] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.185021 30923 ts_tablet_manager.cc:1434] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.130s	user 0.007s	sys 0.000s
I20260709 12:20:32.164088 30545 catalog_manager.cc:5697] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:32.188068 30921 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:32.188748 30921 ts_tablet_manager.cc:1403] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.056s	user 0.009s	sys 0.009s
I20260709 12:20:32.189882 31130 leader_election.cc:290] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:32.190236 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
W20260709 12:20:32.192668 30763 leader_election.cc:343] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.195353 30723 raft_consensus.cc:2468] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:32.197546 30764 leader_election.cc:304] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:32.199018 31130 raft_consensus.cc:2804] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:32.199088 30923 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:32.200148 31130 raft_consensus.cc:697] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:32.201705 31130 consensus_queue.cc:237] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.211519 30922 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:32.212177 30922 ts_tablet_manager.cc:1403] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.082s	user 0.027s	sys 0.017s
I20260709 12:20:32.212339 30921 raft_consensus.cc:359] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.213132 30921 raft_consensus.cc:385] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.213439 30921 raft_consensus.cc:740] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.215535 30922 raft_consensus.cc:359] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.216356 30922 raft_consensus.cc:385] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.216660 30922 raft_consensus.cc:740] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.217722 30925 tablet_bootstrap.cc:654] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.217769 30922 consensus_queue.cc:260] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.220191 30922 ts_tablet_manager.cc:1434] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:20:32.220631 30921 consensus_queue.cc:260] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.221375 30922 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:32.220244 30923 tablet_bootstrap.cc:654] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.228538 30921 ts_tablet_manager.cc:1434] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.000s
I20260709 12:20:32.230253 30921 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:32.241149 30922 tablet_bootstrap.cc:654] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.248207 30546 catalog_manager.cc:5697] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:32.249384 30921 tablet_bootstrap.cc:654] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.251053 31144 raft_consensus.cc:493] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.251672 31144 raft_consensus.cc:515] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.255906 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:32.262441 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ac6fd1105644a1b820d76679057cbe4" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:32.264990 30763 leader_election.cc:304] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:32.263170 31144 leader_election.cc:290] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:32.265947 31125 raft_consensus.cc:2749] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:32.267130 30921 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:32.269469 30921 ts_tablet_manager.cc:1403] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.001s
I20260709 12:20:32.272769 30921 raft_consensus.cc:359] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.273537 30921 raft_consensus.cc:385] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.273898 30921 raft_consensus.cc:740] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.275307 31220 raft_consensus.cc:493] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.276026 31220 raft_consensus.cc:515] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.280140 30921 consensus_queue.cc:260] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.285562 30921 ts_tablet_manager.cc:1434] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:20:32.285811 31220 leader_election.cc:290] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:32.286661 30921 tablet_bootstrap.cc:492] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:32.291687 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1243becf9c9421bb2d5083f71c1e292" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:32.304169 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1243becf9c9421bb2d5083f71c1e292" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:32.308243 31158 raft_consensus.cc:493] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.308998 31158 raft_consensus.cc:515] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.311457 31158 leader_election.cc:290] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:32.315757 30689 leader_election.cc:304] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:32.315789 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:32.318071 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:32.318642 30921 tablet_bootstrap.cc:654] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.318881 30957 raft_consensus.cc:2749] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:32.328064 30615 leader_election.cc:343] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:32.329041 30615 leader_election.cc:343] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.329491 30615 leader_election.cc:304] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:32.332544 31119 raft_consensus.cc:2749] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:32.342324 30723 raft_consensus.cc:3060] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.352813 31130 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.353740 31130 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.357584 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:32.358786 30874 raft_consensus.cc:2393] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:32.360459 30921 tablet_bootstrap.cc:492] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:32.363387 31158 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.364403 31158 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.367799 30921 ts_tablet_manager.cc:1403] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.081s	user 0.017s	sys 0.001s
I20260709 12:20:32.369203 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:32.371244 30615 leader_election.cc:343] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.371695 30923 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:32.371814 30921 raft_consensus.cc:359] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.373013 30921 raft_consensus.cc:385] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.373317 30921 raft_consensus.cc:740] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.374353 30921 consensus_queue.cc:260] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.377434 30921 ts_tablet_manager.cc:1434] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.004s
I20260709 12:20:32.379081 31219 raft_consensus.cc:493] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.379911 31219 raft_consensus.cc:515] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.380826 31130 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:32.382596 31158 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:32.362448 30925 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:32.384385 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1994c4ce566142fa922312f566387067" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:32.384311 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:32.384881 30923 ts_tablet_manager.cc:1403] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.186s	user 0.022s	sys 0.082s
I20260709 12:20:32.385361 30651 raft_consensus.cc:2468] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:32.385659 30922 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:32.386205 30922 ts_tablet_manager.cc:1403] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.165s	user 0.023s	sys 0.031s
I20260709 12:20:32.383302 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:32.389035 30763 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:32.389220 30922 raft_consensus.cc:359] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.389983 30922 raft_consensus.cc:385] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.384236 30925 ts_tablet_manager.cc:1403] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.308s	user 0.014s	sys 0.005s
W20260709 12:20:32.386858 30615 leader_election.cc:343] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.386973 30689 leader_election.cc:304] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:32.391558 30957 raft_consensus.cc:2804] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:32.392082 31219 leader_election.cc:290] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:32.390290 30922 raft_consensus.cc:740] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.392656 30957 raft_consensus.cc:493] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:32.393172 30957 raft_consensus.cc:3060] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.393229 30922 consensus_queue.cc:260] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.394388 31125 raft_consensus.cc:3060] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.395648 30922 ts_tablet_manager.cc:1434] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:20:32.396260 30921 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:32.396659 30922 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:32.396668 30548 catalog_manager.cc:5697] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:32.404069 30922 tablet_bootstrap.cc:654] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.405833 30615 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:32.406309 30921 tablet_bootstrap.cc:654] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.395694 30925 raft_consensus.cc:359] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.407053 30925 raft_consensus.cc:385] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.391754 30923 raft_consensus.cc:359] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.407374 30925 raft_consensus.cc:740] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.407577 30923 raft_consensus.cc:385] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.408375 30925 consensus_queue.cc:260] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.409039 30923 raft_consensus.cc:740] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.409618 31119 raft_consensus.cc:2749] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:32.410034 30923 consensus_queue.cc:260] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.412559 30923 ts_tablet_manager.cc:1434] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.027s	user 0.001s	sys 0.005s
I20260709 12:20:32.418138 30923 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:32.426729 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1994c4ce566142fa922312f566387067" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:32.436882 30925 ts_tablet_manager.cc:1434] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.046s	user 0.006s	sys 0.002s
I20260709 12:20:32.438197 30925 tablet_bootstrap.cc:492] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:32.440987 31125 raft_consensus.cc:2749] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:32.438149 30957 raft_consensus.cc:515] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.434851 30799 raft_consensus.cc:2468] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:32.458645 30925 tablet_bootstrap.cc:654] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.469755 30957 leader_election.cc:290] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:32.461908 31218 raft_consensus.cc:493] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.478992 31218 raft_consensus.cc:515] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.481118 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1994c4ce566142fa922312f566387067" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:32.482154 30647 raft_consensus.cc:3060] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.490437 30922 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:32.490562 30796 raft_consensus.cc:3060] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.491123 30922 ts_tablet_manager.cc:1403] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.095s	user 0.009s	sys 0.009s
I20260709 12:20:32.491652 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:32.495391 30923 tablet_bootstrap.cc:654] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:32.493702 30838 leader_election.cc:343] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.502192 30723 raft_consensus.cc:3060] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.502821 30922 raft_consensus.cc:359] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.505698 30922 raft_consensus.cc:385] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.507658 30922 raft_consensus.cc:740] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.509826 30922 consensus_queue.cc:260] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.489822 31218 leader_election.cc:290] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:32.521538 30922 ts_tablet_manager.cc:1434] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.001s
I20260709 12:20:32.521076 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1994c4ce566142fa922312f566387067" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:32.528801 30799 raft_consensus.cc:3060] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.533744 31220 raft_consensus.cc:493] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.534401 31220 raft_consensus.cc:515] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.551393 30921 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:32.556692 30922 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:32.562394 30921 ts_tablet_manager.cc:1403] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.184s	user 0.028s	sys 0.026s
I20260709 12:20:32.542891 31220 leader_election.cc:290] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:32.541779 30725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:32.546064 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:32.544952 31158 consensus_queue.cc:1048] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20260709 12:20:32.579082 31163 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.595835 30725 raft_consensus.cc:2468] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:32.615579 30957 consensus_queue.cc:1048] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:32.593600 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 44499.4us
I20260709 12:20:32.615602 31220 raft_consensus.cc:493] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.619621 31220 raft_consensus.cc:515] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.620904 30839 leader_election.cc:304] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
W20260709 12:20:32.623658 30689 leader_election.cc:343] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.622201 31217 raft_consensus.cc:2804] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:20:32.630483 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 77464us
I20260709 12:20:32.603803 31119 consensus_queue.cc:1048] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260709 12:20:32.626955 31217 raft_consensus.cc:493] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:32.608158 31163 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.658603 30921 raft_consensus.cc:359] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.660022 30921 raft_consensus.cc:385] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.660418 30921 raft_consensus.cc:740] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.667929 30923 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:32.674722 31124 raft_consensus.cc:493] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.674899 30928 consensus_queue.cc:1048] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:32.675448 31124 raft_consensus.cc:515] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.679441 30923 ts_tablet_manager.cc:1403] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.262s	user 0.025s	sys 0.028s
I20260709 12:20:32.647769 31217 raft_consensus.cc:3060] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.685700 30925 tablet_bootstrap.cc:492] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:32.686354 30925 ts_tablet_manager.cc:1403] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.248s	user 0.014s	sys 0.035s
I20260709 12:20:32.688023 30922 tablet_bootstrap.cc:654] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.697012 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:32.701244 30690 leader_election.cc:343] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:32.709134 31220 leader_election.cc:290] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:32.661485 30921 consensus_queue.cc:260] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.715847 30921 ts_tablet_manager.cc:1434] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.153s	user 0.009s	sys 0.000s
I20260709 12:20:32.716266 30690 leader_election.cc:304] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:32.716536 31163 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:32.689363 30925 raft_consensus.cc:359] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.721055 30925 raft_consensus.cc:385] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.721364 30923 raft_consensus.cc:359] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.722342 30923 raft_consensus.cc:385] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.721424 30925 raft_consensus.cc:740] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.722733 30923 raft_consensus.cc:740] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.723702 30923 consensus_queue.cc:260] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.733413 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:32.734305 30724 raft_consensus.cc:2468] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:32.717434 30647 raft_consensus.cc:2468] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
W20260709 12:20:32.738348 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 101401us
I20260709 12:20:32.740007 30925 consensus_queue.cc:260] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:32.743005 31253 raft_consensus.cc:2749] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:32.748123 30925 ts_tablet_manager.cc:1434] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.061s	user 0.009s	sys 0.000s
W20260709 12:20:32.748324 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 47123.1us
I20260709 12:20:32.752959 30925 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:32.756877 30799 raft_consensus.cc:2468] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:32.726095 31124 leader_election.cc:290] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:32.738721 30921 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:32.773438 30923 ts_tablet_manager.cc:1434] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.093s	user 0.008s	sys 0.000s
I20260709 12:20:32.775202 30923 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:32.787879 31217 raft_consensus.cc:515] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:32.802029 30764 leader_election.cc:304] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:32.869326 30925 tablet_bootstrap.cc:654] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:32.809607 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 174674us
I20260709 12:20:32.857004 31124 raft_consensus.cc:2804] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:32.869917 30928 consensus_queue.cc:1048] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:32.870874 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 160381us
I20260709 12:20:32.875777 31124 raft_consensus.cc:493] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:32.882328 31217 leader_election.cc:290] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:32.915406 30645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:32.916343 30645 raft_consensus.cc:2468] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:32.893232 31124 raft_consensus.cc:3060] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.926869 30922 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:32.927553 30922 ts_tablet_manager.cc:1403] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.371s	user 0.009s	sys 0.012s
I20260709 12:20:32.929036 30923 tablet_bootstrap.cc:654] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.921581 31125 consensus_queue.cc:1048] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:20:32.931535 30922 raft_consensus.cc:359] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.932917 30922 raft_consensus.cc:385] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:32.933605 30922 raft_consensus.cc:740] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:32.934605 30922 consensus_queue.cc:260] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:32.941800 31130 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:32.942791 30922 ts_tablet_manager.cc:1434] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.005s
I20260709 12:20:32.942436 31130 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.945029 31130 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:32.954564 30689 leader_election.cc:304] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:32.955758 30957 raft_consensus.cc:2804] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:32.956214 30957 raft_consensus.cc:697] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:32.922315 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:32.956858 30957 consensus_queue.cc:237] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:32.965523 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:32.966228 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:32.975651 30921 tablet_bootstrap.cc:654] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:32.979607 30648 raft_consensus.cc:2468] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:32.983747 30722 raft_consensus.cc:3060] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:32.993450 31119 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.020010 30922 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19: Bootstrap starting.
W20260709 12:20:33.022258 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:33.002404 30838 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
W20260709 12:20:33.044889 30838 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 46268.1us
I20260709 12:20:33.045245 30719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:33.046308 31269 raft_consensus.cc:2804] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:33.046808 31269 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.047245 31269 raft_consensus.cc:3060] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:33.048151 30839 leader_election.cc:343] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.023411 31119 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:33.058710 31265 log.cc:927] Time spent T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:20:33.019644 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.061728 30870 raft_consensus.cc:2393] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6634546938405dac9eabbadd34d2b1 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:33.023640 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.063108 30648 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
W20260709 12:20:33.063201 31267 log.cc:927] Time spent T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:20:33.064280 31119 leader_election.cc:290] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:33.066262 30763 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:33.067221 31125 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:33.053622 31124 raft_consensus.cc:515] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:33.086517 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:33.063321 31271 log.cc:927] Time spent T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:20:33.067649 31125 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.089989 31125 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.092439 31255 raft_consensus.cc:493] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 23ba075510134f3c8caba2370fff0d40)
I20260709 12:20:33.093057 31255 raft_consensus.cc:515] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.095705 31255 leader_election.cc:290] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.083397 31214 raft_consensus.cc:493] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.102402 31214 raft_consensus.cc:515] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:33.104004 31124 leader_election.cc:290] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:33.107600 31125 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:33.110875 30922 tablet_bootstrap.cc:654] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.105407 31214 leader_election.cc:290] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:33.112911 30722 raft_consensus.cc:2468] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
W20260709 12:20:33.069873 30689 consensus_peers.cc:597] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:33.116075 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 98703.4us
W20260709 12:20:33.101020 30615 leader_election.cc:343] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:33.067866 30764 leader_election.cc:343] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:33.114128 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 44428.9us
I20260709 12:20:33.146620 31124 raft_consensus.cc:493] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.147266 31124 raft_consensus.cc:515] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:33.148813 30763 consensus_peers.cc:597] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:33.149865 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:33.151067 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.149601 31124 leader_election.cc:290] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.150656 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:33.183035 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c766be78f1467795131bc36e7766e9" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:33.193403 30923 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:33.205704 30923 ts_tablet_manager.cc:1403] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.431s	user 0.044s	sys 0.021s
I20260709 12:20:33.181288 30647 raft_consensus.cc:3060] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.194161 30648 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.188416 31125 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:33.224476 30764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 157650us
W20260709 12:20:33.227049 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 90880.6us
W20260709 12:20:33.211951 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 60832.7us
I20260709 12:20:33.240013 30839 leader_election.cc:304] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:33.253445 31268 raft_consensus.cc:2804] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:33.250353 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.263072 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.267215 31268 raft_consensus.cc:697] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:33.267589 30925 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:33.268272 30925 ts_tablet_manager.cc:1403] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.519s	user 0.019s	sys 0.011s
I20260709 12:20:33.268358 31268 consensus_queue.cc:237] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.282388 30923 raft_consensus.cc:359] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:33.287933 30689 leader_election.cc:343] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.288286 30923 raft_consensus.cc:385] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.288602 30923 raft_consensus.cc:740] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.288623 30689 leader_election.cc:304] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:33.289497 30923 consensus_queue.cc:260] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.279353 30922 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:33.290819 30922 ts_tablet_manager.cc:1403] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.273s	user 0.010s	sys 0.017s
W20260709 12:20:33.297200 30614 leader_election.cc:343] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.299135 30614 leader_election.cc:304] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:33.295696 30925 raft_consensus.cc:359] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.301667 30925 raft_consensus.cc:385] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.302074 30925 raft_consensus.cc:740] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.302892 30925 consensus_queue.cc:260] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.305140 30922 raft_consensus.cc:359] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.306061 30922 raft_consensus.cc:385] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.306381 30922 raft_consensus.cc:740] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.306445 30923 ts_tablet_manager.cc:1434] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.100s	user 0.008s	sys 0.000s
I20260709 12:20:33.307039 30921 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:33.307199 30922 consensus_queue.cc:260] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.307767 30923 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:33.309556 30922 ts_tablet_manager.cc:1434] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.018s	user 0.002s	sys 0.005s
I20260709 12:20:33.312042 30922 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:33.314450 30647 raft_consensus.cc:2468] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:33.307678 30921 ts_tablet_manager.cc:1403] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.569s	user 0.016s	sys 0.034s
I20260709 12:20:33.319440 30644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:33.296206 31220 raft_consensus.cc:2749] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:33.322547 30922 tablet_bootstrap.cc:654] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.332752 31264 raft_consensus.cc:2749] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:33.335402 30921 raft_consensus.cc:359] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.336346 30921 raft_consensus.cc:385] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.336660 30921 raft_consensus.cc:740] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.322558 30648 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:33.339656 30649 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.340423 30649 raft_consensus.cc:2468] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:33.328862 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.344056 30721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:33.345227 30721 raft_consensus.cc:3060] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.357878 30870 raft_consensus.cc:3060] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:33.344619 31281 log.cc:927] Time spent T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: Append to log took a long time: real 0.164s	user 0.003s	sys 0.000s
I20260709 12:20:33.347700 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:33.278190 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87c766be78f1467795131bc36e7766e9" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:33.346993 30925 ts_tablet_manager.cc:1434] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.078s	user 0.009s	sys 0.000s
I20260709 12:20:33.337460 30921 consensus_queue.cc:260] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.372259 30925 tablet_bootstrap.cc:492] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:33.276669 31269 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.387284 31273 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.388087 31273 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.402102 30839 leader_election.cc:304] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:33.419533 30763 leader_election.cc:304] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:33.421556 31269 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:33.394096 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.433559 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:33.455744 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.456877 30875 raft_consensus.cc:2393] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6634546938405dac9eabbadd34d2b1 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
W20260709 12:20:33.457589 30838 leader_election.cc:343] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.458663 30726 raft_consensus.cc:3060] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.445458 31130 raft_consensus.cc:2804] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:33.467221 31130 raft_consensus.cc:493] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.467747 31130 raft_consensus.cc:3060] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:33.457798 31295 log.cc:927] Time spent T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:20:33.442613 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 46338.5us
I20260709 12:20:33.445191 31273 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.508605 30689 leader_election.cc:304] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:33.509382 30925 tablet_bootstrap.cc:654] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.477016 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.514580 30647 raft_consensus.cc:2468] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:33.517132 31253 raft_consensus.cc:2749] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:33.517252 30764 leader_election.cc:343] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.520728 31125 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.521247 31125 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.523378 31125 leader_election.cc:290] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.528414 30921 ts_tablet_manager.cc:1434] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.201s	user 0.011s	sys 0.000s
I20260709 12:20:33.531991 30545 catalog_manager.cc:5697] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:33.532807 30721 raft_consensus.cc:2468] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:33.529539 30921 tablet_bootstrap.cc:492] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:33.547601 30764 leader_election.cc:304] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:33.547958 30689 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:33.555788 31125 raft_consensus.cc:2804] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:33.557886 30726 raft_consensus.cc:2468] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:33.558491 31125 raft_consensus.cc:697] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:33.559118 31291 consensus_queue.cc:1048] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:20:33.559595 31125 consensus_queue.cc:237] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:33.561650 30957 raft_consensus.cc:2764] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:20:33.560380 31273 consensus_queue.cc:1048] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:20:33.572825 30923 tablet_bootstrap.cc:654] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.569717 30839 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:33.579033 30921 tablet_bootstrap.cc:654] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.580511 30922 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:33.583875 30922 ts_tablet_manager.cc:1403] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.272s	user 0.030s	sys 0.021s
I20260709 12:20:33.575577 31269 raft_consensus.cc:2804] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:33.573294 31130 raft_consensus.cc:515] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.602991 31261 raft_consensus.cc:493] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.603945 31261 raft_consensus.cc:515] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.603168 31130 leader_election.cc:290] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.617414 31269 raft_consensus.cc:697] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:33.618803 30925 tablet_bootstrap.cc:492] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:33.618490 31269 consensus_queue.cc:237] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.621547 30925 ts_tablet_manager.cc:1403] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.254s	user 0.021s	sys 0.010s
I20260709 12:20:33.622651 31214 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.623327 31214 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.642805 30922 raft_consensus.cc:359] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.644052 30922 raft_consensus.cc:385] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.644441 30922 raft_consensus.cc:740] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.646534 31261 leader_election.cc:290] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:33.647912 31214 leader_election.cc:290] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.649614 30921 tablet_bootstrap.cc:492] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:33.650470 30921 ts_tablet_manager.cc:1403] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.121s	user 0.011s	sys 0.007s
I20260709 12:20:33.645447 30922 consensus_queue.cc:260] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.655778 30923 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:33.656997 30923 ts_tablet_manager.cc:1403] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.350s	user 0.033s	sys 0.014s
I20260709 12:20:33.659806 30925 raft_consensus.cc:359] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:33.661715 30925 raft_consensus.cc:385] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.812346 30925 raft_consensus.cc:740] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.727733 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:33.814481 30650 raft_consensus.cc:3060] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.814369 30925 consensus_queue.cc:260] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:33.732218 30763 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: eb6634546938405dac9eabbadd34d2b1
W20260709 12:20:33.819398 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 87859.8us
I20260709 12:20:33.738278 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.839892 30647 raft_consensus.cc:2468] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:33.843065 30838 leader_election.cc:304] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:33.845767 31287 raft_consensus.cc:2804] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:33.846444 31287 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.817433 30925 ts_tablet_manager.cc:1434] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.195s	user 0.008s	sys 0.002s
I20260709 12:20:33.846956 31287 raft_consensus.cc:3060] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.738611 30549 catalog_manager.cc:5697] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:33.740177 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:33.656059 30922 ts_tablet_manager.cc:1434] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.071s	user 0.006s	sys 0.003s
I20260709 12:20:33.741381 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:33.854934 30796 raft_consensus.cc:2468] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:33.855288 30799 raft_consensus.cc:2468] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:33.655521 30921 raft_consensus.cc:359] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:33.856544 30921 raft_consensus.cc:385] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.856892 30921 raft_consensus.cc:740] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:33.758235 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.832626 31217 consensus_queue.cc:1048] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:33.829276 30923 raft_consensus.cc:359] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:33.821120 30763 leader_election.cc:343] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.859511 30923 raft_consensus.cc:385] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:33.859866 30923 raft_consensus.cc:740] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
W20260709 12:20:33.864883 30838 consensus_peers.cc:597] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:33.872550 30922 tablet_bootstrap.cc:492] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
W20260709 12:20:33.886901 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 65901.5us
I20260709 12:20:33.888767 31289 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:33.889585 31289 raft_consensus.cc:697] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:33.890511 30923 consensus_queue.cc:260] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:33.891391 31289 consensus_queue.cc:237] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:33.894309 30923 ts_tablet_manager.cc:1434] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.237s	user 0.011s	sys 0.007s
I20260709 12:20:33.907469 30923 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:33.901165 30921 consensus_queue.cc:260] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:33.877302 30615 leader_election.cc:304] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:33.912261 31264 raft_consensus.cc:2804] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:33.912592 31217 raft_consensus.cc:493] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:33.912954 31264 raft_consensus.cc:493] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:33.913725 31264 raft_consensus.cc:3060] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.913323 31217 raft_consensus.cc:515] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:33.912339 30614 leader_election.cc:343] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.916468 31217 leader_election.cc:290] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:33.901152 30925 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
W20260709 12:20:33.919682 30870 consensus_meta.cc:287] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: Time spent flushing consensus metadata: real 0.559s	user 0.030s	sys 0.008s
I20260709 12:20:33.928467 31264 raft_consensus.cc:515] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.907497 30922 tablet_bootstrap.cc:654] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.932542 30923 tablet_bootstrap.cc:654] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.941545 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:33.942595 30648 raft_consensus.cc:2468] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:33.944701 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:33.953289 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:33.944947 31264 leader_election.cc:290] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:33.916332 30921 ts_tablet_manager.cc:1434] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.265s	user 0.013s	sys 0.001s
I20260709 12:20:33.954434 30796 raft_consensus.cc:3060] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.958534 30921 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:33.954149 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:33.969712 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
W20260709 12:20:33.971350 30614 leader_election.cc:343] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:33.973704 30925 tablet_bootstrap.cc:654] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:33.980471 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:33.984094 30650 raft_consensus.cc:2468] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:33.985085 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:33.970501 31287 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:33.993927 30869 raft_consensus.cc:2393] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:33.998561 31287 leader_election.cc:290] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:33.999087 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:34.000425 30651 raft_consensus.cc:3060] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:33.981923 30648 raft_consensus.cc:3060] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:34.040408 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c8dd412d0624072a283e7390f9482b6" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:34.042975 30545 catalog_manager.cc:5697] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:34.079049 30874 raft_consensus.cc:3060] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:34.106940 30763 leader_election.cc:343] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.108814 31291 raft_consensus.cc:493] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.113569 31264 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.114293 31264 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.122857 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:34.109550 31291 raft_consensus.cc:515] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.124459 30839 leader_election.cc:304] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:34.126706 31291 leader_election.cc:290] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:34.135502 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:34.137506 30925 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:34.129580 30921 tablet_bootstrap.cc:654] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:34.144037 30690 leader_election.cc:343] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.144783 30925 ts_tablet_manager.cc:1403] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.280s	user 0.028s	sys 0.032s
I20260709 12:20:34.117084 31264 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:34.144793 30923 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:34.145956 30923 ts_tablet_manager.cc:1403] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.239s	user 0.023s	sys 0.024s
W20260709 12:20:34.141588 30614 leader_election.cc:343] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.161671 30763 leader_election.cc:304] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:34.162173 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:34.163710 30922 tablet_bootstrap.cc:492] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:34.164319 30922 ts_tablet_manager.cc:1403] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.293s	user 0.026s	sys 0.021s
I20260709 12:20:34.165402 30721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:34.169325 31130 raft_consensus.cc:2804] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:34.169740 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:34.149124 30648 raft_consensus.cc:2468] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
W20260709 12:20:34.169435 30615 leader_election.cc:343] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.173166 31130 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:34.173805 30922 raft_consensus.cc:359] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.179379 30922 raft_consensus.cc:385] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.173897 31130 raft_consensus.cc:3060] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:34.189575 30923 raft_consensus.cc:359] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.190685 30923 raft_consensus.cc:385] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.190979 30923 raft_consensus.cc:740] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.191864 30923 consensus_queue.cc:260] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.194481 30923 ts_tablet_manager.cc:1434] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.001s
I20260709 12:20:34.194585 30796 raft_consensus.cc:2468] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 1.
I20260709 12:20:34.196126 30923 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:34.197935 30689 leader_election.cc:343] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.198639 30689 leader_election.cc:304] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:34.179852 30922 raft_consensus.cc:740] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.174010 30615 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:34.199568 31289 consensus_queue.cc:1048] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.201221 31258 raft_consensus.cc:2749] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.202601 30763 leader_election.cc:304] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:34.185321 30651 raft_consensus.cc:2468] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:34.204385 30928 raft_consensus.cc:2804] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:34.205224 30928 raft_consensus.cc:697] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:34.205833 30838 leader_election.cc:304] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:34.206437 30928 consensus_queue.cc:237] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:34.211162 30615 leader_election.cc:304] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:34.211524 31217 raft_consensus.cc:2804] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:34.212771 31264 raft_consensus.cc:2804] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:34.213443 31264 raft_consensus.cc:697] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:34.211930 30921 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:34.215899 30921 ts_tablet_manager.cc:1403] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.258s	user 0.024s	sys 0.003s
I20260709 12:20:34.215322 31264 consensus_queue.cc:237] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:34.218479 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 43688.1us
I20260709 12:20:34.219528 31217 raft_consensus.cc:697] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:34.216742 31279 raft_consensus.cc:2749] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.223425 31217 consensus_queue.cc:237] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:34.250895 31269 consensus_queue.cc:1048] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.257393 31163 raft_consensus.cc:2749] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.260872 30922 consensus_queue.cc:260] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:34.261590 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 59151.6us
W20260709 12:20:34.312000 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:34.296725 31130 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:34.318792 30925 raft_consensus.cc:359] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:34.322362 30925 raft_consensus.cc:385] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.323030 30921 raft_consensus.cc:359] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:34.323643 30921 raft_consensus.cc:385] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.322643 30925 raft_consensus.cc:740] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.323869 30921 raft_consensus.cc:740] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.334579 30921 consensus_queue.cc:260] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:34.345973 31214 raft_consensus.cc:2749] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.347038 31278 raft_consensus.cc:493] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.347962 31278 raft_consensus.cc:515] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.350842 31278 leader_election.cc:290] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:34.354853 30792 raft_consensus.cc:2393] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:34.357208 30923 tablet_bootstrap.cc:654] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:34.355329 31287 consensus_queue.cc:1048] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:20:34.363718 30921 ts_tablet_manager.cc:1434] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.147s	user 0.004s	sys 0.004s
I20260709 12:20:34.397917 30921 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:34.401005 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:34.402885 30689 leader_election.cc:343] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.409942 31130 leader_election.cc:290] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:34.389201 30721 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:34.418220 30922 ts_tablet_manager.cc:1434] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.254s	user 0.012s	sys 0.002s
I20260709 12:20:34.419891 30922 tablet_bootstrap.cc:492] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
W20260709 12:20:34.423967 31321 log.cc:927] Time spent T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:20:34.438776 30925 consensus_queue.cc:260] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:34.447000 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:34.448745 30925 ts_tablet_manager.cc:1434] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.303s	user 0.010s	sys 0.001s
I20260709 12:20:34.456005 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:34.465021 31130 consensus_queue.cc:1048] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.467703 30800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
W20260709 12:20:34.472882 30690 leader_election.cc:343] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.473523 30690 leader_election.cc:304] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:34.488780 30647 raft_consensus.cc:2393] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:34.488974 30925 tablet_bootstrap.cc:492] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:34.487735 31291 raft_consensus.cc:2749] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.517609 31329 raft_consensus.cc:493] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.517900 30925 tablet_bootstrap.cc:654] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:34.523551 31329 raft_consensus.cc:515] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.531834 30921 tablet_bootstrap.cc:654] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:34.535959 30922 tablet_bootstrap.cc:654] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:34.590469 31262 raft_consensus.cc:493] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.591420 31262 raft_consensus.cc:515] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.596066 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:34.598687 30614 leader_election.cc:343] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.600863 31262 leader_election.cc:290] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:34.616027 30719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:34.609392 30928 consensus_queue.cc:1048] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:20:34.620378 30549 catalog_manager.cc:5697] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:34.624506 30615 leader_election.cc:343] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:34.625177 30615 leader_election.cc:304] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:34.628705 31258 raft_consensus.cc:2749] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.632825 30923 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:34.633669 30923 ts_tablet_manager.cc:1403] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.438s	user 0.023s	sys 0.000s
I20260709 12:20:34.624711 31279 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.634020 31329 leader_election.cc:290] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:34.637646 31279 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:34.640370 31279 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
W20260709 12:20:34.690882 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 126818us
I20260709 12:20:34.698441 30763 leader_election.cc:304] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:34.693307 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:34.703485 30921 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:34.704056 30921 ts_tablet_manager.cc:1403] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.307s	user 0.011s	sys 0.011s
I20260709 12:20:34.709156 30923 raft_consensus.cc:359] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:34.710245 31336 log.cc:927] Time spent T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:20:34.710450 30923 raft_consensus.cc:385] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.710796 30923 raft_consensus.cc:740] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.710600 31125 consensus_queue.cc:1048] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:20:34.713092 30923 consensus_queue.cc:260] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.714386 30651 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:34.718405 31289 raft_consensus.cc:2749] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:34.718290 31264 consensus_queue.cc:1048] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.721385 30923 ts_tablet_manager.cc:1434] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.073s	user 0.008s	sys 0.001s
I20260709 12:20:34.722126 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:34.729460 30925 tablet_bootstrap.cc:492] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:34.730279 30925 ts_tablet_manager.cc:1403] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.246s	user 0.023s	sys 0.031s
I20260709 12:20:34.723829 30689 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
W20260709 12:20:34.736992 30763 leader_election.cc:343] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:34.699747 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 78238.4us
I20260709 12:20:34.771126 30923 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:34.777720 31124 consensus_queue.cc:1048] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.793268 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:34.794251 30723 raft_consensus.cc:2468] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:34.800508 30922 tablet_bootstrap.cc:492] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:34.801194 30922 ts_tablet_manager.cc:1403] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.382s	user 0.027s	sys 0.017s
I20260709 12:20:34.803203 31279 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:20:34.806198 30763 consensus_peers.cc:597] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.821081 30921 raft_consensus.cc:359] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.824586 30921 raft_consensus.cc:385] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.825106 30921 raft_consensus.cc:740] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.835456 30921 consensus_queue.cc:260] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.838711 31288 consensus_queue.cc:1048] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.837811 30925 raft_consensus.cc:359] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.842119 30925 raft_consensus.cc:385] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.842482 30925 raft_consensus.cc:740] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.843248 30925 consensus_queue.cc:260] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.848760 30922 raft_consensus.cc:359] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:34.850473 30922 raft_consensus.cc:385] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:34.846297 31279 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c178a628712047a88731a915fc93ed19)
I20260709 12:20:34.850854 30922 raft_consensus.cc:740] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:34.858740 30925 ts_tablet_manager.cc:1434] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.128s	user 0.009s	sys 0.000s
I20260709 12:20:34.853762 31279 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:34.861316 30923 tablet_bootstrap.cc:654] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:34.863833 30921 ts_tablet_manager.cc:1434] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.160s	user 0.008s	sys 0.002s
I20260709 12:20:34.865345 30921 tablet_bootstrap.cc:492] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:34.866076 30922 consensus_queue.cc:260] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:34.867667 30764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 117960us
I20260709 12:20:34.869374 30922 ts_tablet_manager.cc:1434] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.004s
I20260709 12:20:34.873746 30922 tablet_bootstrap.cc:492] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:34.873557 30874 raft_consensus.cc:3060] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:34.886518 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:34.887115 30764 leader_election.cc:304] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:34.888950 31292 raft_consensus.cc:2804] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:34.889474 31292 raft_consensus.cc:493] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:34.889891 31292 raft_consensus.cc:3060] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:34.898818 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 48909.2us
I20260709 12:20:34.900377 31163 consensus_queue.cc:1048] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:34.903523 31261 raft_consensus.cc:493] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:34.904243 31261 raft_consensus.cc:515] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:34.911190 31261 leader_election.cc:290] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:34.912220 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:34.915382 30921 tablet_bootstrap.cc:654] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:34.922144 30925 tablet_bootstrap.cc:492] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:34.929940 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9632ab6e7d6d4ac4bdf266d64c523d15" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:34.954120 30651 raft_consensus.cc:3060] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:34.963165 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 51462.7us
I20260709 12:20:34.992583 30614 leader_election.cc:304] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:34.993027 31279 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:34.999763 30923 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:35.000628 30923 ts_tablet_manager.cc:1403] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.239s	user 0.028s	sys 0.006s
I20260709 12:20:35.004594 30923 raft_consensus.cc:359] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:34.966713 30922 tablet_bootstrap.cc:654] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.006193 30720 raft_consensus.cc:1240] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Rejecting Update request from peer c178a628712047a88731a915fc93ed19 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:35.048398 31345 consensus_queue.cc:1059] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:35.049721 31345 raft_consensus.cc:3055] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:35.041725 31163 raft_consensus.cc:493] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.063825 31163 raft_consensus.cc:515] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.042264 31259 consensus_queue.cc:1048] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260709 12:20:35.068454 31292 raft_consensus.cc:515] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.050091 31279 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:35.098805 31292 leader_election.cc:290] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:35.100606 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:35.100827 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:35.101235 31258 raft_consensus.cc:2749] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:35.101469 30720 raft_consensus.cc:3060] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:35.102635 30763 leader_election.cc:343] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.037249 30925 tablet_bootstrap.cc:654] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.054136 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:35.107475 31163 leader_election.cc:290] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.138566 30546 catalog_manager.cc:5697] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:35.140602 30546 catalog_manager.cc:5697] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:35.039016 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:35.143692 30689 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:35.110496 30922 tablet_bootstrap.cc:492] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:35.005600 30923 raft_consensus.cc:385] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.145428 30923 raft_consensus.cc:740] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.146226 30923 consensus_queue.cc:260] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.111214 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:35.144994 30922 ts_tablet_manager.cc:1403] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.272s	user 0.016s	sys 0.001s
I20260709 12:20:35.050292 31345 raft_consensus.cc:740] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:35.148694 30923 ts_tablet_manager.cc:1434] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.148s	user 0.007s	sys 0.002s
I20260709 12:20:35.154834 30923 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:35.148968 31345 consensus_queue.cc:260] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.108659 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:35.156322 31345 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:35.116945 31292 raft_consensus.cc:493] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.157044 31292 raft_consensus.cc:515] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.144968 31279 raft_consensus.cc:2749] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:35.160931 31292 leader_election.cc:290] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:35.154906 30922 raft_consensus.cc:359] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.056180 30921 tablet_bootstrap.cc:492] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:35.162447 30839 leader_election.cc:304] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:35.164999 30922 raft_consensus.cc:385] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.165342 30922 raft_consensus.cc:740] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.166364 30922 consensus_queue.cc:260] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.166930 31218 raft_consensus.cc:2749] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:35.163090 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:35.164353 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:35.168989 30726 raft_consensus.cc:2468] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:35.169003 30874 raft_consensus.cc:2468] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:35.170598 30763 leader_election.cc:304] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:35.171286 30921 ts_tablet_manager.cc:1403] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.306s	user 0.022s	sys 0.010s
I20260709 12:20:35.172325 31345 raft_consensus.cc:2804] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.172868 31345 raft_consensus.cc:493] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.173202 31345 raft_consensus.cc:3060] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.175307 30921 raft_consensus.cc:359] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.176362 30921 raft_consensus.cc:385] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.176733 30921 raft_consensus.cc:740] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.181735 30922 ts_tablet_manager.cc:1434] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.033s	user 0.005s	sys 0.003s
I20260709 12:20:35.164686 30923 tablet_bootstrap.cc:654] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.183876 30922 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:35.183810 30545 catalog_manager.cc:5697] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:35.187402 30545 catalog_manager.cc:5697] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 1 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } }
I20260709 12:20:35.192569 30925 tablet_bootstrap.cc:492] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:35.197613 30922 tablet_bootstrap.cc:654] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.198027 30925 ts_tablet_manager.cc:1403] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.276s	user 0.014s	sys 0.002s
I20260709 12:20:35.188354 30720 raft_consensus.cc:2468] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:35.193178 30921 consensus_queue.cc:260] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.212633 30925 raft_consensus.cc:359] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.213311 30921 ts_tablet_manager.cc:1434] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260709 12:20:35.213668 30925 raft_consensus.cc:385] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.214216 30925 raft_consensus.cc:740] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.215723 30764 leader_election.cc:304] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:35.215798 30921 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:35.216754 31292 raft_consensus.cc:2804] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:35.217391 31292 raft_consensus.cc:697] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:35.217715 30922 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:35.218335 30922 ts_tablet_manager.cc:1403] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.035s	user 0.015s	sys 0.002s
I20260709 12:20:35.218528 31292 consensus_queue.cc:237] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.215500 30925 consensus_queue.cc:260] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.196472 31279 raft_consensus.cc:493] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.225561 31279 raft_consensus.cc:515] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.195015 31345 raft_consensus.cc:515] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.227986 31279 leader_election.cc:290] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.229221 30921 tablet_bootstrap.cc:654] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.229786 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:35.230659 30925 ts_tablet_manager.cc:1434] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.001s
I20260709 12:20:35.227892 30923 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
W20260709 12:20:35.231523 30689 leader_election.cc:343] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.232120 30925 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:35.221316 30922 raft_consensus.cc:359] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.236824 31345 leader_election.cc:290] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:35.237206 30922 raft_consensus.cc:385] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.262648 30922 raft_consensus.cc:740] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.242851 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:35.242489 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58bf19ba51b540ff8dcb14a3af23b8e3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:35.264561 30874 raft_consensus.cc:3060] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.264724 30726 raft_consensus.cc:3060] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.232350 30923 ts_tablet_manager.cc:1403] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.078s	user 0.014s	sys 0.004s
I20260709 12:20:35.263466 30922 consensus_queue.cc:260] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.270246 30922 ts_tablet_manager.cc:1434] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.000s
I20260709 12:20:35.270032 30923 raft_consensus.cc:359] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.271000 30923 raft_consensus.cc:385] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.271327 30923 raft_consensus.cc:740] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.271692 30922 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:35.263428 30925 tablet_bootstrap.cc:654] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.272099 30923 consensus_queue.cc:260] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.232971 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:35.274299 30648 raft_consensus.cc:2468] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:35.274710 30923 ts_tablet_manager.cc:1434] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:20:35.275995 30923 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:35.276849 30874 raft_consensus.cc:2468] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:35.278365 30763 leader_election.cc:304] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:35.279654 31345 raft_consensus.cc:2804] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:35.280210 31345 raft_consensus.cc:697] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:35.281149 31345 consensus_queue.cc:237] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.282903 30923 tablet_bootstrap.cc:654] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.289857 30726 raft_consensus.cc:2468] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:35.298318 30922 tablet_bootstrap.cc:654] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.304940 30921 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:35.305984 30921 ts_tablet_manager.cc:1403] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.090s	user 0.033s	sys 0.011s
I20260709 12:20:35.306936 31292 raft_consensus.cc:493] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.307597 31292 raft_consensus.cc:515] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.289067 30689 leader_election.cc:304] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:35.320520 30921 raft_consensus.cc:359] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.330415 30922 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:35.339977 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:35.340116 30922 ts_tablet_manager.cc:1403] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.069s	user 0.012s	sys 0.004s
W20260709 12:20:35.340615 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 51706.8us
I20260709 12:20:35.340787 30651 raft_consensus.cc:2468] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:35.341761 30921 raft_consensus.cc:385] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.342170 30921 raft_consensus.cc:740] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.342291 31292 leader_election.cc:290] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.342517 31220 raft_consensus.cc:2804] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.343237 31220 raft_consensus.cc:493] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.343158 30921 consensus_queue.cc:260] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.344082 31220 raft_consensus.cc:3060] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.345439 30921 ts_tablet_manager.cc:1434] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.001s
I20260709 12:20:35.347018 30921 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:35.344237 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:35.356024 30763 leader_election.cc:343] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:35.458189 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:35.359488 30763 leader_election.cc:304] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:35.367166 30922 raft_consensus.cc:359] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.477067 30922 raft_consensus.cc:385] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:20:35.477386 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 117967us
I20260709 12:20:35.477411 30922 raft_consensus.cc:740] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.478807 30922 consensus_queue.cc:260] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.480034 30921 tablet_bootstrap.cc:700] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: Time spent opening tablet: real 0.127s	user 0.031s	sys 0.056s
I20260709 12:20:35.480966 30921 tablet_bootstrap.cc:654] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.481647 30922 ts_tablet_manager.cc:1434] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.141s	user 0.007s	sys 0.000s
I20260709 12:20:35.482967 30922 tablet_bootstrap.cc:492] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:35.357836 31163 raft_consensus.cc:493] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.485931 31163 raft_consensus.cc:515] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.491967 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:35.494614 30922 tablet_bootstrap.cc:654] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.496546 30921 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:35.497251 30921 ts_tablet_manager.cc:1403] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.150s	user 0.040s	sys 0.062s
I20260709 12:20:35.500855 30921 raft_consensus.cc:359] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.501786 30921 raft_consensus.cc:385] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.502130 30921 raft_consensus.cc:740] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.503240 30921 consensus_queue.cc:260] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.505858 30921 ts_tablet_manager.cc:1434] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.004s
I20260709 12:20:35.507361 30921 tablet_bootstrap.cc:492] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:35.520493 30921 tablet_bootstrap.cc:654] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:35.524710 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 72645.2us
I20260709 12:20:35.539671 30921 tablet_bootstrap.cc:492] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:35.540400 30921 ts_tablet_manager.cc:1403] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.008s
I20260709 12:20:35.477866 31339 raft_consensus.cc:2804] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.540932 31339 raft_consensus.cc:493] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.541399 31339 raft_consensus.cc:3060] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.544018 30921 raft_consensus.cc:359] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.544976 30921 raft_consensus.cc:385] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.545306 30921 raft_consensus.cc:740] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.546336 30921 consensus_queue.cc:260] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.548475 30921 ts_tablet_manager.cc:1434] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:20:35.549708 30921 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:35.552624 31163 leader_election.cc:290] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:35.557679 30921 tablet_bootstrap.cc:654] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.558430 31340 raft_consensus.cc:493] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.559005 31340 raft_consensus.cc:515] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.561480 31340 leader_election.cc:290] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:35.565397 31292 raft_consensus.cc:493] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.566219 31292 raft_consensus.cc:515] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.569998 31292 leader_election.cc:290] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:35.574347 30921 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:35.574985 30921 ts_tablet_manager.cc:1403] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.026s	user 0.018s	sys 0.000s
I20260709 12:20:35.575682 30871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cea447841c446fea6a32ff304879db6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:35.578398 30921 raft_consensus.cc:359] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.579165 30921 raft_consensus.cc:385] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.579439 30921 raft_consensus.cc:740] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.580157 30921 consensus_queue.cc:260] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.582144 30921 ts_tablet_manager.cc:1434] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:35.583318 30921 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1: Bootstrap starting.
I20260709 12:20:35.595281 30921 tablet_bootstrap.cc:654] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:35.619681 30921 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1: No bootstrap required, opened a new log
I20260709 12:20:35.620448 30921 ts_tablet_manager.cc:1403] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.003s
I20260709 12:20:35.624958 31258 raft_consensus.cc:493] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.626230 31258 raft_consensus.cc:515] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:35.626973 30763 leader_election.cc:343] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.631812 31258 leader_election.cc:290] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:35.623915 30921 raft_consensus.cc:359] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.641903 30921 raft_consensus.cc:385] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.642356 30921 raft_consensus.cc:740] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.643211 30921 consensus_queue.cc:260] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.652511 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:35.660482 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cea447841c446fea6a32ff304879db6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:35.661410 30726 raft_consensus.cc:2468] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:35.676851 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:35.677745 30720 raft_consensus.cc:2468] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:35.684962 30925 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:35.685762 30925 ts_tablet_manager.cc:1403] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.454s	user 0.055s	sys 0.095s
I20260709 12:20:35.689504 30925 raft_consensus.cc:359] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.690237 30925 raft_consensus.cc:385] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.690529 30925 raft_consensus.cc:740] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.691238 30925 consensus_queue.cc:260] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.696880 30921 ts_tablet_manager.cc:1434] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1: Time spent starting tablet: real 0.076s	user 0.009s	sys 0.001s
I20260709 12:20:35.698556 31220 raft_consensus.cc:515] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.700963 31220 leader_election.cc:290] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.727154 30922 tablet_bootstrap.cc:492] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:35.727819 30922 ts_tablet_manager.cc:1403] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.245s	user 0.039s	sys 0.059s
I20260709 12:20:35.705181 31339 raft_consensus.cc:515] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:35.706029 30925 ts_tablet_manager.cc:1434] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.001s
W20260709 12:20:35.707391 30690 leader_election.cc:343] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.710023 31292 raft_consensus.cc:493] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.736713 30764 leader_election.cc:304] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:35.712522 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:35.739007 31345 raft_consensus.cc:2804] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.739495 31345 raft_consensus.cc:493] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:35.717990 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 46062.3us
I20260709 12:20:35.739874 31345 raft_consensus.cc:3060] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.725823 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:35.737452 31292 raft_consensus.cc:515] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.742034 30796 raft_consensus.cc:2468] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:35.743153 31292 leader_election.cc:290] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:35.745128 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:35.747202 30925 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:35.748270 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:35.749010 30651 raft_consensus.cc:3060] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.749456 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:35.752787 31339 leader_election.cc:290] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.754500 30875 raft_consensus.cc:3060] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.754997 31130 consensus_queue.cc:1048] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:35.757122 31220 raft_consensus.cc:493] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:35.757618 30763 leader_election.cc:343] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.757879 31220 raft_consensus.cc:515] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.760460 31220 leader_election.cc:290] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:35.762574 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f37d27937c43f39c083e36508283fd" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:35.754635 30922 raft_consensus.cc:359] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
W20260709 12:20:35.766068 30690 leader_election.cc:343] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.747160 30648 raft_consensus.cc:2468] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:35.767555 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:35.768383 30726 raft_consensus.cc:2468] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:35.754534 30923 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:35.770737 30923 ts_tablet_manager.cc:1403] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.495s	user 0.102s	sys 0.143s
I20260709 12:20:35.773609 30651 raft_consensus.cc:2468] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:35.769796 30763 leader_election.cc:304] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:35.773749 30922 raft_consensus.cc:385] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.774863 30922 raft_consensus.cc:740] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:35.775118 30763 leader_election.cc:304] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:35.776252 30615 leader_election.cc:304] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:35.778015 31130 raft_consensus.cc:2804] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.778478 31130 raft_consensus.cc:493] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.778534 31261 raft_consensus.cc:2804] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.778913 31130 raft_consensus.cc:3060] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.779132 31261 raft_consensus.cc:493] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:35.779702 31261 raft_consensus.cc:3060] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:35.821602 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 43535.5us
I20260709 12:20:35.776880 30922 consensus_queue.cc:260] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.823377 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:35.824270 31289 consensus_queue.cc:1048] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:35.824609 31214 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.781977 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0e79bd22cff4979ba995f0c04f48806" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:35.826990 31214 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.827636 31289 consensus_queue.cc:1048] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:35.827035 31130 raft_consensus.cc:515] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:35.751837 31345 raft_consensus.cc:515] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.830689 31130 leader_election.cc:290] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:35.831722 31345 leader_election.cc:290] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:35.782389 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:35.832979 30648 raft_consensus.cc:3060] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:35.833406 31258 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.834048 31258 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:35.845542 30689 leader_election.cc:343] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.813784 31339 raft_consensus.cc:2804] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:35.846431 30689 leader_election.cc:304] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:35.847102 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f37d27937c43f39c083e36508283fd" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:35.824510 31340 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.851362 30925 tablet_bootstrap.cc:654] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:35.856436 30689 leader_election.cc:343] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.860654 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:35.861434 30651 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 0.
I20260709 12:20:35.860102 31351 raft_consensus.cc:493] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.864259 31351 raft_consensus.cc:515] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.865738 31214 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:35.866762 31351 leader_election.cc:290] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.871527 31340 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.868194 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:35.873167 30875 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:35.874194 31340 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:35.875128 30838 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:35.859949 31279 raft_consensus.cc:2749] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:35.878093 30923 raft_consensus.cc:359] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:35.879320 31258 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:35.887800 30923 raft_consensus.cc:385] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:35.888207 30923 raft_consensus.cc:740] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
W20260709 12:20:35.892099 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 70977.3us
I20260709 12:20:35.895129 30614 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:35.896032 30923 consensus_queue.cc:260] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:35.894340 30689 leader_election.cc:343] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:35.889245 30648 raft_consensus.cc:2468] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:35.901947 30689 leader_election.cc:304] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:35.862424 31339 raft_consensus.cc:697] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:35.891814 31163 raft_consensus.cc:2804] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.949891 30923 ts_tablet_manager.cc:1434] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.179s	user 0.011s	sys 0.000s
I20260709 12:20:35.905022 31258 raft_consensus.cc:2804] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:35.975003 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:35.976275 31340 raft_consensus.cc:2749] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:35.979915 30922 ts_tablet_manager.cc:1434] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.252s	user 0.010s	sys 0.017s
I20260709 12:20:35.969571 30549 catalog_manager.cc:5697] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:35.981559 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 53683.1us
I20260709 12:20:35.983328 30689 leader_election.cc:304] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:35.987381 31289 raft_consensus.cc:493] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:35.988029 31289 raft_consensus.cc:515] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:35.983522 30549 catalog_manager.cc:5697] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:35.963980 31258 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.000752 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a132d43ba3447a916ccdfcc6e66c26" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:36.003754 31384 raft_consensus.cc:2804] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:36.004451 31384 raft_consensus.cc:697] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:36.005434 31384 consensus_queue.cc:237] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:36.009789 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.010677 30723 raft_consensus.cc:3060] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.012382 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:36.013458 30874 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
W20260709 12:20:36.014118 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 145034us
I20260709 12:20:36.016136 30839 leader_election.cc:304] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:36.028862 31214 raft_consensus.cc:2749] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:36.029750 30922 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:36.030725 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:36.031556 30720 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 0.
I20260709 12:20:36.035203 30925 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:36.034919 30689 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:36.035904 30925 ts_tablet_manager.cc:1403] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.289s	user 0.009s	sys 0.013s
I20260709 12:20:36.002669 31258 raft_consensus.cc:3060] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.048712 31387 raft_consensus.cc:2804] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:36.063519 30925 raft_consensus.cc:359] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:36.064669 30925 raft_consensus.cc:385] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:36.065062 30925 raft_consensus.cc:740] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:36.071768 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cea447841c446fea6a32ff304879db6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.072784 30726 raft_consensus.cc:3060] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.076233 31261 raft_consensus.cc:515] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.078015 30651 raft_consensus.cc:2393] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c178a628712047a88731a915fc93ed19 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:36.003116 31339 consensus_queue.cc:237] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:36.070379 30925 consensus_queue.cc:260] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:36.054543 31387 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.102391 30925 ts_tablet_manager.cc:1434] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.066s	user 0.008s	sys 0.006s
I20260709 12:20:36.113206 30922 tablet_bootstrap.cc:654] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:36.114418 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
W20260709 12:20:36.116385 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 100331us
I20260709 12:20:36.090353 31387 raft_consensus.cc:3060] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.126518 30923 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:36.115216 30722 raft_consensus.cc:2468] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:36.148600 30875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cea447841c446fea6a32ff304879db6" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:36.172578 30875 raft_consensus.cc:3060] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:36.144733 30838 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 128813us
I20260709 12:20:36.218515 30925 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
W20260709 12:20:36.186991 31309 log.cc:927] Time spent T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
W20260709 12:20:36.260437 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 136885us
I20260709 12:20:36.272466 30645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a132d43ba3447a916ccdfcc6e66c26" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:36.221299 30723 raft_consensus.cc:2468] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:36.335564 30645 raft_consensus.cc:2468] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 0.
I20260709 12:20:36.337278 31261 leader_election.cc:290] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:36.371703 30726 raft_consensus.cc:2468] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:36.392539 30925 tablet_bootstrap.cc:654] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:36.394619 30871 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:36.399307 31163 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:36.377530 30689 leader_election.cc:343] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:36.400239 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 105189us
I20260709 12:20:36.399919 31163 raft_consensus.cc:3060] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.406900 31289 leader_election.cc:290] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
W20260709 12:20:36.423139 30763 leader_election.cc:343] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:36.424490 30763 leader_election.cc:343] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:36.424189 31392 consensus_queue.cc:1048] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:36.431910 30923 tablet_bootstrap.cc:654] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:36.432230 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:36.445525 30689 leader_election.cc:304] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:36.486303 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:36.498281 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:36.499269 30796 raft_consensus.cc:2393] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:36.429191 31163 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:36.494663 31401 log.cc:927] Time spent T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:20:36.519976 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.473482 31383 raft_consensus.cc:2804] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:36.489935 31259 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c178a628712047a88731a915fc93ed19)
W20260709 12:20:36.543962 30689 consensus_peers.cc:597] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:36.564543 30923 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:36.565205 30923 ts_tablet_manager.cc:1403] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.450s	user 0.043s	sys 0.038s
I20260709 12:20:36.567348 30922 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:36.567947 30922 ts_tablet_manager.cc:1403] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.553s	user 0.014s	sys 0.013s
I20260709 12:20:36.573041 30923 raft_consensus.cc:359] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.573902 30923 raft_consensus.cc:385] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:36.574198 30923 raft_consensus.cc:740] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:36.574855 30923 consensus_queue.cc:260] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.577441 30923 ts_tablet_manager.cc:1434] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.003s
I20260709 12:20:36.578612 30923 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:36.550935 31259 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.587810 31259 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:36.593410 30923 tablet_bootstrap.cc:654] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:36.595726 30724 raft_consensus.cc:2393] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
W20260709 12:20:36.597301 31258 consensus_meta.cc:287] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Time spent flushing consensus metadata: real 0.547s	user 0.011s	sys 0.004s
I20260709 12:20:36.533485 31383 raft_consensus.cc:493] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.584887 31387 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:36.608965 30925 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:36.632869 30925 ts_tablet_manager.cc:1403] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.447s	user 0.022s	sys 0.006s
I20260709 12:20:36.612265 30922 raft_consensus.cc:359] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.619590 30764 leader_election.cc:304] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:36.637883 30922 raft_consensus.cc:385] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:36.638394 30922 raft_consensus.cc:740] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:36.627743 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:36.641872 30764 leader_election.cc:304] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:36.646083 31344 raft_consensus.cc:2804] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:36.646978 31344 raft_consensus.cc:697] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:36.649135 31344 consensus_queue.cc:237] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.654073 30764 leader_election.cc:304] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:36.655126 31292 raft_consensus.cc:2804] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:36.655646 31292 raft_consensus.cc:697] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:36.656796 31292 consensus_queue.cc:237] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:36.670826 30923 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: No bootstrap required, opened a new log
I20260709 12:20:36.672629 30923 ts_tablet_manager.cc:1403] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.094s	user 0.023s	sys 0.006s
I20260709 12:20:36.643067 30922 consensus_queue.cc:260] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.675516 31392 raft_consensus.cc:2804] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:36.680600 31392 raft_consensus.cc:493] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.681053 31392 raft_consensus.cc:3060] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.597988 31258 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:36.691205 30726 raft_consensus.cc:2393] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:36.606740 31383 raft_consensus.cc:3060] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:36.696458 30615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 171544us
I20260709 12:20:36.685456 31258 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:36.707664 30922 ts_tablet_manager.cc:1434] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.139s	user 0.008s	sys 0.002s
I20260709 12:20:36.717797 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:36.714605 30647 raft_consensus.cc:2393] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb6634546938405dac9eabbadd34d2b1 in current term 1: Already voted for candidate c18174c596f9488f8be666b85062e329 in this term.
I20260709 12:20:36.729432 30925 raft_consensus.cc:359] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:36.731338 31367 log.cc:927] Time spent T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: Append to log took a long time: real 0.067s	user 0.000s	sys 0.002s
I20260709 12:20:36.715032 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:36.734599 30615 leader_election.cc:304] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:36.735707 30925 raft_consensus.cc:385] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:36.747658 30925 raft_consensus.cc:740] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:36.751592 30721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.751376 30923 raft_consensus.cc:359] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:36.752832 30870 raft_consensus.cc:2393] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:36.753007 30923 raft_consensus.cc:385] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:36.753445 30923 raft_consensus.cc:740] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:36.754233 30923 consensus_queue.cc:260] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:36.755059 30925 consensus_queue.cc:260] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:36.755930 30923 ts_tablet_manager.cc:1434] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.083s	user 0.006s	sys 0.001s
I20260709 12:20:36.756546 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:36.757472 30722 raft_consensus.cc:2393] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 2: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:36.759099 30792 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:36.751663 31387 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:36.759704 31279 consensus_queue.cc:1048] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:36.759281 31388 raft_consensus.cc:2749] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:36.761960 30615 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:36.757509 30925 ts_tablet_manager.cc:1434] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.124s	user 0.007s	sys 0.001s
I20260709 12:20:36.763319 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:36.764142 30870 raft_consensus.cc:2393] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb6634546938405dac9eabbadd34d2b1 in current term 1: Already voted for candidate 23ba075510134f3c8caba2370fff0d40 in this term.
I20260709 12:20:36.765486 31259 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:36.765977 31259 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c178a628712047a88731a915fc93ed19)
I20260709 12:20:36.766348 31259 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:36.757170 31392 raft_consensus.cc:515] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:36.778254 30922 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: Bootstrap starting.
I20260709 12:20:36.779843 31392 leader_election.cc:290] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:36.785126 30643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:36.783902 30614 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:36.831115 30870 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:36.831727 30925 tablet_bootstrap.cc:492] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:36.833604 31259 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.834203 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.834856 30717 raft_consensus.cc:3060] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.836175 31259 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:36.837604 31163 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:36.843457 30763 leader_election.cc:343] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:36.847414 30614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 63600us
I20260709 12:20:36.850617 31261 raft_consensus.cc:2749] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:36.852908 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:36.854591 30722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.848538 30689 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:36.869519 30839 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:36.875675 30925 tablet_bootstrap.cc:654] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:36.876036 31418 raft_consensus.cc:2749] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:36.867069 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:36.879411 31397 raft_consensus.cc:2749] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:36.879587 30549 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56306 (request call id 59) took 1055 ms. Trace:
I20260709 12:20:36.881174 30549 rpcz_store.cc:276] 0709 12:20:35.826287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:35.841181 (+ 14894us) service_pool.cc:224] Handling call
0709 12:20:36.879516 (+1038335us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:36.109734 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:36.110386 (+   652us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:36.110432 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:20:36.110453 (+    21us) tablet.cc:662] Decoding operations
0709 12:20:36.110938 (+   485us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:36.111008 (+    70us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:36.111039 (+    31us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:20:36.111414 (+   375us) tablet.cc:701] Row locks acquired
0709 12:20:36.111436 (+    22us) write_op.cc:260] PREPARE: finished
0709 12:20:36.111616 (+   180us) write_op.cc:270] Start()
0709 12:20:36.111748 (+   132us) write_op.cc:276] Timestamp: P: 1783599636111563 usec, L: 0
0709 12:20:36.111768 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:20:36.112639 (+   871us) log.cc:844] Serialized 928 byte log entry
0709 12:20:36.554876 (+442237us) op_driver.cc:464] REPLICATION: finished
0709 12:20:36.658976 (+104100us) write_op.cc:301] APPLY: starting
0709 12:20:36.659085 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:36.659230 (+   145us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:36.659369 (+   139us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:36.659755 (+   386us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:36.659949 (+   194us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:20:36.659996 (+    47us) write_op.cc:312] APPLY: finished
0709 12:20:36.660240 (+   244us) log.cc:844] Serialized 28 byte log entry
0709 12:20:36.662587 (+  2347us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:36.662699 (+   112us) write_op.cc:454] Released schema lock
0709 12:20:36.662751 (+    52us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:36.664439 (+  1688us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:36.664475 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":98277,"apply.run_cpu_time_us":2145,"apply.run_wall_time_us":5428,"num_ops":2,"prepare.queue_time_us":104586,"prepare.run_cpu_time_us":3406,"prepare.run_wall_time_us":3407,"replication_time_us":413617,"thread_start_us":153411,"threads_started":2}]]}
I20260709 12:20:36.895169 31383 raft_consensus.cc:515] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:36.905529 31130 raft_consensus.cc:493] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:36.927472 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 241349us
I20260709 12:20:36.931098 31130 raft_consensus.cc:515] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.879408 30720 raft_consensus.cc:2393] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c18174c596f9488f8be666b85062e329 in current term 2: Already voted for candidate eb6634546938405dac9eabbadd34d2b1 in this term.
I20260709 12:20:36.906940 31259 raft_consensus.cc:493] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:36.917160 30792 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:36.935359 31259 raft_consensus.cc:515] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:36.938395 30922 tablet_bootstrap.cc:654] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:36.939061 31259 leader_election.cc:290] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:36.939162 31130 leader_election.cc:290] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:36.942054 30647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f37d27937c43f39c083e36508283fd" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:36.949380 30647 raft_consensus.cc:2468] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:36.942711 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:36.950802 30763 leader_election.cc:304] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:36.951929 31289 raft_consensus.cc:2804] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:36.952396 31289 raft_consensus.cc:493] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:36.952767 31289 raft_consensus.cc:3060] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:36.953038 30614 leader_election.cc:343] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:36.951458 30615 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:36.964275 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f37d27937c43f39c083e36508283fd" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:36.965346 31388 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:36.966239 31388 raft_consensus.cc:697] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:36.967818 31388 consensus_queue.cc:237] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:36.965420 30720 raft_consensus.cc:2468] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:36.971951 31383 leader_election.cc:290] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:36.971366 30548 catalog_manager.cc:5697] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:36.976485 30548 catalog_manager.cc:5697] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:36.986325 30651 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a132d43ba3447a916ccdfcc6e66c26" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:36.987099 30651 raft_consensus.cc:3060] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:36.984751 30548 catalog_manager.cc:5697] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.990149 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:36.992182 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a132d43ba3447a916ccdfcc6e66c26" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:36.992765 30717 raft_consensus.cc:2468] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
W20260709 12:20:36.995656 30615 leader_election.cc:343] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:36.996475 30615 leader_election.cc:304] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:36.997332 31262 raft_consensus.cc:2749] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:37.006912 30764 leader_election.cc:304] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:37.016535 31378 raft_consensus.cc:2804] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:37.017194 31378 raft_consensus.cc:697] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
W20260709 12:20:37.019582 30763 consensus_peers.cc:597] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.021966 31377 consensus_queue.cc:1048] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:20:37.025986 31218 raft_consensus.cc:493] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.026552 31218 raft_consensus.cc:515] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:37.028887 31218 leader_election.cc:290] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:37.031255 31289 raft_consensus.cc:515] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:37.034438 30648 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:37.035522 30689 leader_election.cc:343] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:37.042754 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f37d27937c43f39c083e36508283fd" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:37.035068 31289 leader_election.cc:290] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:37.043707 30650 raft_consensus.cc:3060] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.043965 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f37d27937c43f39c083e36508283fd" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:37.044782 30717 raft_consensus.cc:3060] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.053606 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f25f8eeb23d4d5aa15a052f827302ad" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:37.059737 30925 tablet_bootstrap.cc:492] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:37.060456 30925 ts_tablet_manager.cc:1403] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.229s	user 0.016s	sys 0.010s
I20260709 12:20:37.061939 30867 raft_consensus.cc:3060] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.072110 30651 raft_consensus.cc:2468] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:37.044454 31378 consensus_queue.cc:237] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:37.095068 30689 leader_election.cc:304] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:37.105821 30839 leader_election.cc:304] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:37.097995 30717 raft_consensus.cc:2468] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:37.109266 31377 consensus_queue.cc:1048] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.110935 30925 raft_consensus.cc:359] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:37.111721 30925 raft_consensus.cc:385] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:37.112002 30925 raft_consensus.cc:740] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:37.112730 30925 consensus_queue.cc:260] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:37.116588 30922 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: No bootstrap required, opened a new log
I20260709 12:20:37.117237 30922 ts_tablet_manager.cc:1403] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: Time spent bootstrapping tablet: real 0.339s	user 0.008s	sys 0.036s
I20260709 12:20:37.120326 31400 raft_consensus.cc:2804] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:37.121017 31400 raft_consensus.cc:697] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:37.121526 30650 raft_consensus.cc:2468] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:37.122363 31400 consensus_queue.cc:237] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:37.124390 30764 leader_election.cc:304] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:37.125756 31377 raft_consensus.cc:2804] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:37.126343 31377 raft_consensus.cc:697] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:37.128845 30925 ts_tablet_manager.cc:1434] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.068s	user 0.008s	sys 0.000s
I20260709 12:20:37.129340 31377 consensus_queue.cc:237] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:37.142735 30922 raft_consensus.cc:359] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.143541 30922 raft_consensus.cc:385] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:37.143877 30922 raft_consensus.cc:740] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Initialized, Role: FOLLOWER
I20260709 12:20:37.146332 30925 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:37.148238 30922 consensus_queue.cc:260] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.154812 31398 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.163779 31398 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:37.166877 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:37.170775 30922 ts_tablet_manager.cc:1434] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.004s
I20260709 12:20:37.225138 31163 raft_consensus.cc:2749] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:37.239059 30874 raft_consensus.cc:3060] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.243016 31398 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:37.205521 31289 raft_consensus.cc:493] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.246718 30643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:37.247699 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:37.251482 30796 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:37.258409 30690 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:37.261099 30925 tablet_bootstrap.cc:654] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:37.265725 31398 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:37.260587 31289 raft_consensus.cc:515] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.289033 31398 raft_consensus.cc:493] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:37.319298 31392 consensus_queue.cc:1048] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.320179 31289 leader_election.cc:290] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:37.308406 31398 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:37.342108 31436 consensus_queue.cc:1048] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:37.352591 30644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:37.353484 30644 raft_consensus.cc:2468] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 0.
I20260709 12:20:37.356767 30763 leader_election.cc:304] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:37.364087 31378 raft_consensus.cc:2804] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:37.367352 31378 raft_consensus.cc:493] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:37.367950 31378 raft_consensus.cc:3060] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.381307 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
W20260709 12:20:37.391795 30763 leader_election.cc:343] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:37.374624 31398 raft_consensus.cc:515] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:37.419529 30644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:37.421975 31398 leader_election.cc:290] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 3 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:37.426031 31436 consensus_queue.cc:1048] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.433985 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "462bf14c43184e17b0f483d0e185993f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:37.434836 30792 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:37.438067 30925 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:37.438992 30925 ts_tablet_manager.cc:1403] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.293s	user 0.016s	sys 0.017s
I20260709 12:20:37.442775 30925 raft_consensus.cc:359] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.443662 30925 raft_consensus.cc:385] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:37.443887 30546 catalog_manager.cc:5697] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } }
I20260709 12:20:37.445273 31378 raft_consensus.cc:515] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.449484 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:37.453078 30643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:37.454005 30643 raft_consensus.cc:3060] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.449659 30546 catalog_manager.cc:5697] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:37.464943 30763 leader_election.cc:343] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:37.484994 31378 leader_election.cc:290] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:37.485443 30719 raft_consensus.cc:1240] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Rejecting Update request from peer c18174c596f9488f8be666b85062e329 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:37.487527 30643 raft_consensus.cc:2468] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
I20260709 12:20:37.489506 30763 leader_election.cc:304] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:37.491755 31388 consensus_queue.cc:1059] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.493948 31388 raft_consensus.cc:3055] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:37.494369 31388 raft_consensus.cc:740] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:37.495551 31437 raft_consensus.cc:2804] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:37.495389 31388 consensus_queue.cc:260] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:37.496357 31437 raft_consensus.cc:697] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:37.496923 31388 raft_consensus.cc:3060] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:37.497532 31437 consensus_queue.cc:237] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.501363 30792 raft_consensus.cc:2468] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 3.
I20260709 12:20:37.511605 30690 leader_election.cc:304] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:37.513437 31400 raft_consensus.cc:2804] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:37.514068 31400 raft_consensus.cc:697] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 3 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:37.515067 31400 consensus_queue.cc:237] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:37.443975 30925 raft_consensus.cc:740] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:37.524008 30925 consensus_queue.cc:260] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.538678 30925 ts_tablet_manager.cc:1434] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.099s	user 0.007s	sys 0.002s
I20260709 12:20:37.540556 30925 tablet_bootstrap.cc:492] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:37.564597 30548 catalog_manager.cc:5697] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.566345 30925 tablet_bootstrap.cc:654] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:37.567340 30548 catalog_manager.cc:5697] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:37.658452 31437 consensus_queue.cc:1048] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.675087 30549 catalog_manager.cc:5697] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:37.677728 30925 tablet_bootstrap.cc:492] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:37.683239 30925 ts_tablet_manager.cc:1403] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.143s	user 0.029s	sys 0.028s
I20260709 12:20:37.686728 30925 raft_consensus.cc:359] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:37.687623 30925 raft_consensus.cc:385] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:37.687943 30925 raft_consensus.cc:740] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:37.688741 30925 consensus_queue.cc:260] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:37.678201 30549 catalog_manager.cc:5697] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 2 to 3, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 3 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.703945 31345 consensus_queue.cc:1048] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.707944 31400 consensus_queue.cc:1048] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:37.715099 30925 ts_tablet_manager.cc:1434] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 12:20:37.718438 30925 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:37.719197 31395 raft_consensus.cc:493] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:37.719950 30763 consensus_peers.cc:597] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.719794 31395 raft_consensus.cc:515] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:37.722317 31395 leader_election.cc:290] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:37.724570 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:37.725582 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:37.726616 30726 raft_consensus.cc:2393] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23ba075510134f3c8caba2370fff0d40 in current term 1: Already voted for candidate c178a628712047a88731a915fc93ed19 in this term.
I20260709 12:20:37.728154 30839 leader_election.cc:304] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:37.729012 31395 raft_consensus.cc:3060] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.739681 31405 raft_consensus.cc:493] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.740393 31405 raft_consensus.cc:515] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.740594 31437 consensus_queue.cc:1048] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:37.743094 31405 leader_election.cc:290] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:37.745167 30644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
W20260709 12:20:37.747457 30689 consensus_peers.cc:597] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 -> Peer 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125): Couldn't send request to peer 23ba075510134f3c8caba2370fff0d40. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.748453 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:37.749840 31395 raft_consensus.cc:2749] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:37.750324 30839 leader_election.cc:304] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:37.764887 31405 raft_consensus.cc:2749] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:37.756841 30925 tablet_bootstrap.cc:654] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:37.760138 30548 catalog_manager.cc:5697] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:37.784238 30925 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:37.784926 30925 ts_tablet_manager.cc:1403] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.067s	user 0.013s	sys 0.000s
I20260709 12:20:37.788113 30925 raft_consensus.cc:359] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:37.788878 30925 raft_consensus.cc:385] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:37.789170 30925 raft_consensus.cc:740] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:37.789865 30925 consensus_queue.cc:260] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:37.793284 30925 ts_tablet_manager.cc:1434] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:20:37.800357 30925 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40: Bootstrap starting.
I20260709 12:20:37.808537 30925 tablet_bootstrap.cc:654] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:37.821832 30925 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40: No bootstrap required, opened a new log
I20260709 12:20:37.822712 30925 ts_tablet_manager.cc:1403] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.003s
I20260709 12:20:37.825824 30925 raft_consensus.cc:359] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.826620 30925 raft_consensus.cc:385] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:37.826916 30925 raft_consensus.cc:740] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Initialized, Role: FOLLOWER
I20260709 12:20:37.827679 30925 consensus_queue.cc:260] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:37.829936 30925 ts_tablet_manager.cc:1434] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:37.894212 30874 raft_consensus.cc:3060] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:37.939854 30650 raft_consensus.cc:1275] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:20:37.940306 30650 pending_rounds.cc:85] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Aborting all ops after (but not including) 1
I20260709 12:20:37.940632 30650 pending_rounds.cc:107] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:20:37.940996 30650 raft_consensus.cc:2889] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:37.942557 31398 consensus_queue.cc:1048] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.955585 30796 raft_consensus.cc:1275] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 3 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:20:37.959918 31340 consensus_queue.cc:1048] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:37.990612 31163 raft_consensus.cc:493] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:37.991386 31163 raft_consensus.cc:515] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:37.996418 30644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a132d43ba3447a916ccdfcc6e66c26" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:38.002986 31163 leader_election.cc:290] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:38.005435 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a132d43ba3447a916ccdfcc6e66c26" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:38.014314 30839 leader_election.cc:304] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:38.015328 31395 raft_consensus.cc:2749] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:38.082505 31436 consensus_queue.cc:1048] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:38.102341 30874 raft_consensus.cc:3060] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:38.128795 31437 consensus_queue.cc:1048] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:38.235559 30597 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:38.270318 30869 raft_consensus.cc:3060] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:38.507237 31400 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:38.508123 31400 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:38.513257 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:38.514176 30869 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:38.518393 30650 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329" is_pre_election: true
I20260709 12:20:38.519541 30650 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:38.521096 30689 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:38.522295 31398 raft_consensus.cc:2804] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:38.522810 31398 raft_consensus.cc:493] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:38.523172 31398 raft_consensus.cc:3060] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:38.512624 31400 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:38.534019 31398 raft_consensus.cc:515] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:38.540599 31398 leader_election.cc:290] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:38.541601 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:38.542395 30869 raft_consensus.cc:3060] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:38.542791 30644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f3e88175c47421db220066ad29f61c4" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18174c596f9488f8be666b85062e329"
I20260709 12:20:38.543612 30644 raft_consensus.cc:3060] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:38.552424 30869 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:38.553395 30644 raft_consensus.cc:2468] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:38.554198 30689 leader_election.cc:304] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:38.555243 31398 raft_consensus.cc:2804] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:38.555872 31398 raft_consensus.cc:697] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:38.557029 31398 consensus_queue.cc:237] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:38.568964 30547 catalog_manager.cc:5697] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 0 to 2, leader changed from <none> to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:39.001186 30867 raft_consensus.cc:1275] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:39.003727 31340 consensus_queue.cc:1048] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:39.022007 30644 raft_consensus.cc:1275] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:39.023983 31398 consensus_queue.cc:1048] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:39.072278 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
W20260709 12:20:39.092880 30763 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20260709 12:20:39.102003 30763 consensus_peers.cc:597] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.107028 30838 consensus_peers.cc:597] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.133502 30838 consensus_peers.cc:597] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.149417 30689 consensus_peers.cc:597] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.174300 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:39.175637 11310 tablet_replica.cc:333] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.176891 11310 raft_consensus.cc:2243] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.177675 11310 raft_consensus.cc:2272] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.180622 11310 tablet_replica.cc:333] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.181335 11310 raft_consensus.cc:2243] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.181888 11310 raft_consensus.cc:2272] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.184873 11310 tablet_replica.cc:333] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.186272 11310 raft_consensus.cc:2243] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.186919 11310 raft_consensus.cc:2272] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.188182 30689 consensus_peers.cc:597] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.189847 11310 tablet_replica.cc:333] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.191066 11310 raft_consensus.cc:2243] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.192338 11310 raft_consensus.cc:2272] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.195627 30689 consensus_peers.cc:597] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.196573 11310 tablet_replica.cc:333] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.197839 11310 raft_consensus.cc:2243] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.198482 11310 raft_consensus.cc:2272] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.201262 30838 consensus_peers.cc:597] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.201534 11310 tablet_replica.cc:333] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: stopping tablet replica
W20260709 12:20:39.201959 30838 consensus_peers.cc:597] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.202750 11310 raft_consensus.cc:2243] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.203472 11310 raft_consensus.cc:2272] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.205286 30763 consensus_peers.cc:597] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.207371 11310 tablet_replica.cc:333] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.208686 11310 raft_consensus.cc:2243] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.209306 11310 raft_consensus.cc:2272] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.214347 11310 tablet_replica.cc:333] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.215320 11310 raft_consensus.cc:2243] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.216199 11310 raft_consensus.cc:2272] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.219671 11310 tablet_replica.cc:333] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.220803 11310 raft_consensus.cc:2243] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.221421 11310 raft_consensus.cc:2272] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.225066 11310 tablet_replica.cc:333] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.226109 11310 raft_consensus.cc:2243] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.226734 11310 raft_consensus.cc:2272] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.227380 30689 consensus_peers.cc:597] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.230180 11310 tablet_replica.cc:333] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.231307 11310 raft_consensus.cc:2243] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.232364 11310 raft_consensus.cc:2272] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.235222 11310 tablet_replica.cc:333] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.235982 11310 raft_consensus.cc:2243] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.236660 11310 raft_consensus.cc:2272] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.240025 11310 tablet_replica.cc:333] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.241427 11310 raft_consensus.cc:2243] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.242353 11310 raft_consensus.cc:2272] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.246028 11310 tablet_replica.cc:333] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.246865 11310 raft_consensus.cc:2243] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.247977 11310 raft_consensus.cc:2272] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.251598 11310 tablet_replica.cc:333] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.252238 11310 raft_consensus.cc:2243] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.252825 11310 raft_consensus.cc:2272] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.256325 11310 tablet_replica.cc:333] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.257339 11310 raft_consensus.cc:2243] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.257947 11310 raft_consensus.cc:2272] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.260289 30689 consensus_peers.cc:597] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.261260 11310 tablet_replica.cc:333] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.262396 11310 raft_consensus.cc:2243] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.263077 11310 raft_consensus.cc:2272] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.263118 30763 consensus_peers.cc:597] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.263849 30763 consensus_peers.cc:597] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.268361 11310 tablet_replica.cc:333] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: stopping tablet replica
W20260709 12:20:39.269402 30763 consensus_peers.cc:597] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.269603 11310 raft_consensus.cc:2243] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.270323 11310 raft_consensus.cc:2272] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.274235 11310 tablet_replica.cc:333] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: stopping tablet replica
W20260709 12:20:39.274741 30763 consensus_peers.cc:597] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.275221 11310 raft_consensus.cc:2243] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.275977 11310 raft_consensus.cc:2272] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.279999 11310 tablet_replica.cc:333] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.281078 11310 raft_consensus.cc:2243] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.281751 11310 raft_consensus.cc:2272] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.285173 11310 tablet_replica.cc:333] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.286070 11310 raft_consensus.cc:2243] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.287127 11310 raft_consensus.cc:2272] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.289978 11310 tablet_replica.cc:333] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.290635 11310 raft_consensus.cc:2243] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.291112 11310 raft_consensus.cc:2272] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.294292 11310 tablet_replica.cc:333] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.295619 11310 raft_consensus.cc:2243] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.296319 11310 raft_consensus.cc:2272] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.299844 11310 tablet_replica.cc:333] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.300820 11310 raft_consensus.cc:2243] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.301515 11310 raft_consensus.cc:2272] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.305006 11310 tablet_replica.cc:333] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: stopping tablet replica
W20260709 12:20:39.305004 30689 consensus_peers.cc:597] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.305939 11310 raft_consensus.cc:2243] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.306934 11310 raft_consensus.cc:2272] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.310197 11310 tablet_replica.cc:333] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.311172 11310 raft_consensus.cc:2243] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.311836 11310 raft_consensus.cc:2272] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.315683 11310 tablet_replica.cc:333] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: stopping tablet replica
W20260709 12:20:39.316493 30689 consensus_peers.cc:597] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.316690 11310 raft_consensus.cc:2243] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.318166 11310 raft_consensus.cc:2272] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.321589 11310 tablet_replica.cc:333] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.322619 11310 raft_consensus.cc:2243] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.323379 11310 raft_consensus.cc:2272] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.326489 11310 tablet_replica.cc:333] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.327361 11310 raft_consensus.cc:2243] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.328583 11310 raft_consensus.cc:2272] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.330981 11310 tablet_replica.cc:333] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.331881 11310 raft_consensus.cc:2243] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.332523 11310 raft_consensus.cc:2272] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.335393 11310 tablet_replica.cc:333] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: stopping tablet replica
W20260709 12:20:39.335891 30689 consensus_peers.cc:597] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.336129 11310 raft_consensus.cc:2243] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.337762 11310 raft_consensus.cc:2272] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.341849 11310 tablet_replica.cc:333] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.342864 11310 raft_consensus.cc:2243] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:20:39.343407 30838 consensus_peers.cc:597] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.344192 11310 raft_consensus.cc:2272] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.348596 11310 tablet_replica.cc:333] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.349988 11310 raft_consensus.cc:2243] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.350628 11310 raft_consensus.cc:2272] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.354132 11310 tablet_replica.cc:333] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.354877 11310 raft_consensus.cc:2243] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.355468 11310 raft_consensus.cc:2272] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.358927 11310 tablet_replica.cc:333] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.359835 11310 raft_consensus.cc:2243] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:39.360142 30763 consensus_peers.cc:597] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.360569 11310 raft_consensus.cc:2272] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.364567 11310 tablet_replica.cc:333] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.365393 11310 raft_consensus.cc:2243] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.366078 11310 raft_consensus.cc:2272] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.369812 11310 tablet_replica.cc:333] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.370612 11310 raft_consensus.cc:2243] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.371217 11310 raft_consensus.cc:2272] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.373698 30763 consensus_peers.cc:597] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.374373 11310 tablet_replica.cc:333] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.375161 11310 raft_consensus.cc:2243] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.375697 11310 raft_consensus.cc:2272] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.378839 11310 tablet_replica.cc:333] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.379806 11310 raft_consensus.cc:2243] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.380416 11310 raft_consensus.cc:2272] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.383699 11310 tablet_replica.cc:333] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.384326 11310 raft_consensus.cc:2243] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.384845 11310 raft_consensus.cc:2272] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.387786 11310 tablet_replica.cc:333] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.388455 11310 raft_consensus.cc:2243] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.388967 11310 raft_consensus.cc:2272] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.391585 11310 tablet_replica.cc:333] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.392208 11310 raft_consensus.cc:2243] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.393208 11310 raft_consensus.cc:2272] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.396425 11310 tablet_replica.cc:333] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.397229 11310 raft_consensus.cc:2243] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.397869 11310 raft_consensus.cc:2272] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.400820 11310 tablet_replica.cc:333] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.402230 11310 raft_consensus.cc:2243] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.402916 11310 raft_consensus.cc:2272] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.406013 30689 consensus_peers.cc:597] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.406013 11310 tablet_replica.cc:333] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: stopping tablet replica
I20260709 12:20:39.407770 11310 raft_consensus.cc:2243] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:39.409173 11310 raft_consensus.cc:2272] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:39.427780 30689 consensus_peers.cc:597] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.461176 30838 consensus_peers.cc:597] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.488713 30763 consensus_peers.cc:597] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.505687 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
W20260709 12:20:39.516112 30689 consensus_peers.cc:597] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.532217 30838 consensus_peers.cc:597] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.544121 30763 consensus_peers.cc:597] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.571557 30689 consensus_peers.cc:597] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.571949 30763 consensus_peers.cc:597] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.577296 30838 consensus_peers.cc:597] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.585500 30838 consensus_peers.cc:597] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.609736 30763 consensus_peers.cc:597] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.623139 30689 consensus_peers.cc:597] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:39.635946 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:39.636847 30838 consensus_peers.cc:597] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:39.686280 31480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:39.700855 31481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:39.701113 31483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:39.701469 11310 server_base.cc:1061] running on GCE node
I20260709 12:20:39.709547 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:39.709985 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:39.710258 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599639710238 us; error 0 us; skew 500 ppm
I20260709 12:20:39.710996 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:39.714932 11310 webserver.cc:533] Webserver started at http://127.11.11.129:35215/ using document root <none> and password file <none>
I20260709 12:20:39.715648 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:39.715883 11310 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:20:39.721019 11310 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:20:39.732316 31488 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:39.733490 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20260709 12:20:39.733981 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "c18174c596f9488f8be666b85062e329"
format_stamp: "Formatted at 2026-07-09 12:20:23 on dist-test-slave-4qvp"
I20260709 12:20:39.734562 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:39.839308 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:39.841931 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:39.844898 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:39.853009 31497 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:20:40.235213 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:20:40.235630 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.385s	user 0.011s	sys 0.001s
I20260709 12:20:40.236014 11310 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:20:40.243999 31497 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.260385 31497 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.261548 31497 tablet_bootstrap.cc:492] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.262341 31497 ts_tablet_manager.cc:1403] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.019s	user 0.017s	sys 0.000s
I20260709 12:20:40.265338 31497 raft_consensus.cc:359] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.266211 31497 raft_consensus.cc:740] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.267073 31497 consensus_queue.cc:260] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.272981 31497 ts_tablet_manager.cc:1434] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.002s
I20260709 12:20:40.274006 31497 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.290729 31497 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.291802 31497 tablet_bootstrap.cc:492] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.292615 31497 ts_tablet_manager.cc:1403] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.005s
I20260709 12:20:40.295254 31497 raft_consensus.cc:359] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.295943 31497 raft_consensus.cc:740] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.296566 31497 consensus_queue.cc:260] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.298400 31497 ts_tablet_manager.cc:1434] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.004s
I20260709 12:20:40.299225 31497 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.315294 31494 raft_consensus.cc:493] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.316406 31494 raft_consensus.cc:515] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.319609 31494 leader_election.cc:290] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.324889 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:40.351728 31497 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.352795 31497 tablet_bootstrap.cc:492] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.353415 31497 ts_tablet_manager.cc:1403] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.054s	user 0.016s	sys 0.012s
W20260709 12:20:40.356516 30763 leader_election.cc:336] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.357705 30763 leader_election.cc:304] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:40.358706 31494 raft_consensus.cc:2749] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.363170 31497 raft_consensus.cc:359] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.364121 31497 raft_consensus.cc:740] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.365140 31497 consensus_queue.cc:260] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.367148 31497 ts_tablet_manager.cc:1434] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:20:40.367986 31497 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.409451 31497 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.410509 31497 tablet_bootstrap.cc:492] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.411195 31497 ts_tablet_manager.cc:1403] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.043s	user 0.019s	sys 0.011s
I20260709 12:20:40.414278 31497 raft_consensus.cc:359] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.415120 31497 raft_consensus.cc:740] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.416298 31497 consensus_queue.cc:260] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.418380 31497 ts_tablet_manager.cc:1434] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:40.419225 31497 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.445989 31498 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.446684 31498 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.449052 31498 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:40.455096 31497 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.456442 31497 tablet_bootstrap.cc:492] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.457289 31497 ts_tablet_manager.cc:1403] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.038s	user 0.014s	sys 0.019s
W20260709 12:20:40.458043 30838 leader_election.cc:336] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.460214 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:40.460403 31497 raft_consensus.cc:359] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.461992 31398 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.465039 31497 raft_consensus.cc:740] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.465327 31398 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.466135 31497 consensus_queue.cc:260] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.466759 30726 raft_consensus.cc:2468] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:40.468246 31398 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.468997 31497 ts_tablet_manager.cc:1434] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.000s
I20260709 12:20:40.470335 31497 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.470793 30839 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.471519 31163 raft_consensus.cc:493] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.472918 31163 raft_consensus.cc:515] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.475296 31163 leader_election.cc:290] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.477206 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.478436 30796 raft_consensus.cc:2468] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:40.471966 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:40.479584 30867 raft_consensus.cc:2468] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:40.480571 30839 leader_election.cc:304] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:40.481572 31405 raft_consensus.cc:2804] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.482009 31405 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.482388 31405 raft_consensus.cc:3060] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:40.484062 30838 leader_election.cc:336] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.484555 31345 raft_consensus.cc:493] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.486097 31345 raft_consensus.cc:515] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.484728 31498 raft_consensus.cc:2804] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.495029 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:40.497681 31163 raft_consensus.cc:493] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.498230 31163 raft_consensus.cc:515] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.509996 31498 raft_consensus.cc:493] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.511621 31498 raft_consensus.cc:3060] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:40.511088 30838 leader_election.cc:336] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.510540 31345 leader_election.cc:290] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:40.507177 30763 leader_election.cc:336] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.502560 31405 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.514586 30763 leader_election.cc:304] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:40.515976 31345 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.516628 31345 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.517427 31163 leader_election.cc:290] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:40.518584 31405 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:40.514021 31497 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.521392 31497 tablet_bootstrap.cc:492] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.522389 31497 ts_tablet_manager.cc:1403] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.052s	user 0.024s	sys 0.018s
I20260709 12:20:40.522732 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:40.523098 31378 raft_consensus.cc:2749] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.522320 30726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:40.524744 30726 raft_consensus.cc:3060] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.525611 31345 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.526558 31497 raft_consensus.cc:359] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.527494 31497 raft_consensus.cc:740] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.528360 31497 consensus_queue.cc:260] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.522248 30689 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:40.530860 31497 ts_tablet_manager.cc:1434] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:40.531080 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:40.534614 31497 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.536504 31345 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.537235 31345 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:40.537479 30689 leader_election.cc:336] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.541397 31499 raft_consensus.cc:493] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.541203 30839 leader_election.cc:304] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:40.542382 31499 raft_consensus.cc:515] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.543720 31163 raft_consensus.cc:2749] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.544982 31499 leader_election.cc:290] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:40.547135 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.547963 30796 raft_consensus.cc:2468] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:40.552394 31498 raft_consensus.cc:515] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.552951 30690 leader_election.cc:304] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
W20260709 12:20:40.553706 30838 leader_election.cc:336] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.552446 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:40.554416 31499 raft_consensus.cc:2804] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.556186 31499 raft_consensus.cc:493] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.557011 31499 raft_consensus.cc:3060] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.560935 31378 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
W20260709 12:20:40.561161 30689 leader_election.cc:336] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.561586 31378 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.564481 31378 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:40.571082 30838 leader_election.cc:336] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.571844 31405 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.572304 31498 leader_election.cc:290] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.572784 31405 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.571965 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdae5560db3d45c180b6bed07a6b6197" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:40.574095 30792 raft_consensus.cc:3060] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.575130 31345 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:40.576030 30763 leader_election.cc:336] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.579568 31405 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:40.580713 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:40.555059 30726 raft_consensus.cc:2468] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:40.586542 31398 raft_consensus.cc:2764] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:20:40.590199 30792 raft_consensus.cc:2468] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
W20260709 12:20:40.595606 30838 leader_election.cc:336] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.596244 30838 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:40.597836 31405 raft_consensus.cc:2749] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.598409 30839 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.598423 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:40.600998 30839 leader_election.cc:304] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.653187 31163 raft_consensus.cc:2804] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:40.655246 31398 raft_consensus.cc:493] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.655668 31497 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.655941 31398 raft_consensus.cc:515] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.656985 31497 tablet_bootstrap.cc:492] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.657918 31497 ts_tablet_manager.cc:1403] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.124s	user 0.020s	sys 0.037s
I20260709 12:20:40.658665 31405 raft_consensus.cc:2804] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:40.658841 31398 leader_election.cc:290] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:40.662837 30763 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:40.661413 31497 raft_consensus.cc:359] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.667539 31497 raft_consensus.cc:740] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.668919 31497 consensus_queue.cc:260] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.671996 31497 ts_tablet_manager.cc:1434] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:20:40.673044 31497 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.673492 31405 raft_consensus.cc:697] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:40.674940 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.674722 31405 consensus_queue.cc:237] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.676718 31507 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.677309 31507 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.679615 31507 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.673187 31498 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.680670 31498 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.683171 31498 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.686559 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.688565 31499 raft_consensus.cc:515] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.691115 31499 leader_election.cc:290] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:40.693173 31340 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.694196 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.695292 30796 raft_consensus.cc:2468] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:40.694196 31340 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.698431 31497 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:40.699113 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c274ea8c29774701bdc1708a1839fa88" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:40.698938 31340 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:40.699751 31497 tablet_bootstrap.cc:492] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:40.700022 30797 raft_consensus.cc:3060] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.700747 31497 ts_tablet_manager.cc:1403] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.008s
I20260709 12:20:40.700913 31494 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.702184 31494 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.704854 31494 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.668035 31345 raft_consensus.cc:2749] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.708940 30792 raft_consensus.cc:2468] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
I20260709 12:20:40.707264 30839 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:40.715179 31512 raft_consensus.cc:2804] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.715672 31512 raft_consensus.cc:493] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.716053 31512 raft_consensus.cc:3060] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:40.716692 30763 leader_election.cc:336] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.717463 30763 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:40.719650 31494 raft_consensus.cc:2749] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.721306 30797 raft_consensus.cc:2468] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:40.719381 11310 ts_tablet_manager.cc:616] Registered 45 tablets
W20260709 12:20:40.723181 30689 leader_election.cc:336] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.723229 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.487s	user 0.360s	sys 0.024s
W20260709 12:20:40.728530 30838 leader_election.cc:336] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:40.729328 30838 leader_election.cc:336] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:40.729239 30763 leader_election.cc:336] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.723820 30689 leader_election.cc:304] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb6634546938405dac9eabbadd34d2b1; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:40.732167 30763 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:40.731329 31497 raft_consensus.cc:359] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.733340 31345 raft_consensus.cc:2749] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.733268 31497 raft_consensus.cc:740] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:40.736208 31163 raft_consensus.cc:697] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:40.736361 31497 consensus_queue.cc:260] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.743710 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.737567 31163 consensus_queue.cc:237] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.752677 31497 ts_tablet_manager.cc:1434] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.000s
I20260709 12:20:40.757503 31497 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:40.761570 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:40.744822 30796 raft_consensus.cc:2468] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:40.755198 30839 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.767526 31405 raft_consensus.cc:2804] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.768316 31405 raft_consensus.cc:493] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.769212 31405 raft_consensus.cc:3060] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.738392 31398 raft_consensus.cc:2749] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:40.778662 30763 leader_election.cc:336] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.798295 31507 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.799089 31507 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.807340 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:40.808302 30796 raft_consensus.cc:2468] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 1.
W20260709 12:20:40.813989 30689 leader_election.cc:336] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.815466 30763 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
W20260709 12:20:40.816402 30689 leader_election.cc:336] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:40.817507 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 70605.9us
I20260709 12:20:40.819976 30690 leader_election.cc:304] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.817813 31494 raft_consensus.cc:2749] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:40.822706 30690 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.823752 31519 raft_consensus.cc:493] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.823118 31507 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.828943 31499 raft_consensus.cc:2804] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:40.819099 31512 raft_consensus.cc:515] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.831729 31519 raft_consensus.cc:515] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.845867 31520 raft_consensus.cc:2804] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.855420 31345 raft_consensus.cc:493] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.861264 31512 leader_election.cc:290] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.858680 31345 raft_consensus.cc:515] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.866221 31345 leader_election.cc:290] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:40.857878 31499 raft_consensus.cc:697] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:40.868685 31499 consensus_queue.cc:237] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.869319 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:40.870142 30720 raft_consensus.cc:2468] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 1.
W20260709 12:20:40.872262 30838 leader_election.cc:336] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.858393 31520 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.878772 31520 raft_consensus.cc:3060] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.879868 31519 leader_election.cc:290] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:40.884145 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d65269b37cc420382672e7ceecd4789" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:40.884985 30796 raft_consensus.cc:3060] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.887116 30764 leader_election.cc:304] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:40.888484 31520 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.891294 31520 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:40.894675 30546 catalog_manager.cc:5697] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:40.895977 30796 raft_consensus.cc:2468] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:40.893669 30839 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.898248 31378 raft_consensus.cc:2804] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.898814 31378 raft_consensus.cc:493] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.899221 31378 raft_consensus.cc:3060] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.906373 31510 raft_consensus.cc:2804] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:40.907024 31510 raft_consensus.cc:493] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.907398 31510 raft_consensus.cc:3060] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:40.914156 30838 leader_election.cc:336] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.914975 31405 raft_consensus.cc:515] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:40.914206 30763 leader_election.cc:336] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.934664 30839 leader_election.cc:304] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:40.942284 31509 raft_consensus.cc:2804] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:40.943003 31509 raft_consensus.cc:697] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:40.946395 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:40.947304 30867 raft_consensus.cc:2468] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:40.944305 31509 consensus_queue.cc:237] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:40.950863 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 108932us
I20260709 12:20:40.956564 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:40.957362 30799 raft_consensus.cc:3060] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:40.957023 31519 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:40.960459 31519 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:40.962854 31519 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:40.985801 31510 raft_consensus.cc:515] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:40.988950 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83d271f20d484fc5bce5814a664ac807" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:40.997471 30793 raft_consensus.cc:3060] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:40.998608 30838 leader_election.cc:336] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:40.999883 30547 catalog_manager.cc:5697] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.002920 31405 leader_election.cc:290] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:41.003865 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50776b4e5bc646a78173a6598eee54c2" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19"
W20260709 12:20:40.991428 30689 leader_election.cc:336] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.001183 31378 raft_consensus.cc:515] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:40.988204 30799 raft_consensus.cc:2468] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:41.004745 30792 raft_consensus.cc:3060] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:41.009102 31510 leader_election.cc:290] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:41.028314 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:41.029351 30867 raft_consensus.cc:2468] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 1.
I20260709 12:20:41.043107 30792 raft_consensus.cc:2468] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:41.045248 30839 leader_election.cc:304] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:41.053664 30793 raft_consensus.cc:2468] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:41.056162 30839 leader_election.cc:304] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:41.057474 31509 raft_consensus.cc:2804] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:41.058059 31509 raft_consensus.cc:697] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:41.058974 31509 consensus_queue.cc:237] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:41.060539 30838 leader_election.cc:336] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.062017 31510 raft_consensus.cc:2804] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:41.062661 31510 raft_consensus.cc:697] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
W20260709 12:20:41.067271 30763 leader_election.cc:336] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.067090 31510 consensus_queue.cc:237] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:41.069099 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 95784.6us
I20260709 12:20:41.073125 30690 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:41.074043 31522 raft_consensus.cc:2804] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:41.074728 31522 raft_consensus.cc:697] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:41.075075 31378 leader_election.cc:290] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:41.075711 31522 consensus_queue.cc:237] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:41.077035 30719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1f70173e529b4bdca23bc1fa8fbb2bde" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:41.077912 30719 raft_consensus.cc:3060] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:41.080890 31497 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.082012 31497 tablet_bootstrap.cc:492] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:41.082840 31497 ts_tablet_manager.cc:1403] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.326s	user 0.024s	sys 0.009s
I20260709 12:20:41.097486 30689 leader_election.cc:304] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:41.108533 31520 raft_consensus.cc:2804] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:41.109004 31520 raft_consensus.cc:493] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:41.109366 31520 raft_consensus.cc:3060] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:41.110455 31497 raft_consensus.cc:359] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:41.111289 31497 raft_consensus.cc:740] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:41.095862 30719 raft_consensus.cc:2468] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 2.
I20260709 12:20:41.122534 31497 consensus_queue.cc:260] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:41.128389 30764 leader_election.cc:304] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:41.134101 30689 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:41.135169 31497 ts_tablet_manager.cc:1434] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.052s	user 0.008s	sys 0.001s
I20260709 12:20:41.137285 31497 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:41.136200 31494 raft_consensus.cc:2804] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:41.151878 30719 raft_consensus.cc:1275] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.152217 31494 raft_consensus.cc:697] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 2 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:41.139412 31516 raft_consensus.cc:2804] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:41.154060 31516 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:41.154507 31516 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:41.153316 31494 consensus_queue.cc:237] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:41.154372 30689 leader_election.cc:336] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.176680 31405 consensus_queue.cc:1048] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.140489 31520 raft_consensus.cc:515] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.184224 31520 leader_election.cc:290] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
W20260709 12:20:41.207398 30689 leader_election.cc:336] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.214039 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3db7a62e2e80411ea0c98d2b5282b08f" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:41.215142 30869 raft_consensus.cc:3060] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:41.220961 30792 raft_consensus.cc:1275] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.224634 30547 catalog_manager.cc:5697] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.226716 30547 catalog_manager.cc:5697] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.228196 31524 consensus_queue.cc:1048] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.228559 30547 catalog_manager.cc:5697] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.219287 30546 catalog_manager.cc:5697] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.246510 30869 raft_consensus.cc:2468] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:41.238420 30547 catalog_manager.cc:5697] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:41.278208 30689 leader_election.cc:336] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.193023 31516 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.294646 30689 leader_election.cc:304] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
I20260709 12:20:41.295533 31398 raft_consensus.cc:2804] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:41.296077 31398 raft_consensus.cc:697] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:41.296983 31398 consensus_queue.cc:237] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
W20260709 12:20:41.310159 30838 consensus_peers.cc:597] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.315385 30546 catalog_manager.cc:5697] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:41.322750 31516 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:41.325824 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "eb6634546938405dac9eabbadd34d2b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:41.327306 30874 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:41.354171 31497 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.355533 31497 tablet_bootstrap.cc:492] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:41.356411 31497 ts_tablet_manager.cc:1403] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.220s	user 0.020s	sys 0.004s
I20260709 12:20:41.359866 31497 raft_consensus.cc:359] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.360723 31497 raft_consensus.cc:740] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:41.353286 30874 raft_consensus.cc:2468] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6634546938405dac9eabbadd34d2b1 in term 2.
I20260709 12:20:41.362898 31497 consensus_queue.cc:260] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.366525 31497 ts_tablet_manager.cc:1434] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.001s
I20260709 12:20:41.367609 31497 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:41.377714 30548 catalog_manager.cc:5697] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
W20260709 12:20:41.400440 30689 leader_election.cc:336] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:41.419114 30689 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1; no voters: c18174c596f9488f8be666b85062e329
W20260709 12:20:41.422238 30838 consensus_peers.cc:597] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.429306 31499 raft_consensus.cc:2804] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:41.429997 31499 raft_consensus.cc:697] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Leader. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
W20260709 12:20:41.430729 30838 consensus_peers.cc:597] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.430927 31499 consensus_queue.cc:237] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.437376 31501 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.438081 31501 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:41.453042 30793 raft_consensus.cc:1275] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.454458 30792 raft_consensus.cc:1275] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.456512 31501 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:41.501581 30792 raft_consensus.cc:1275] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.510794 31163 consensus_queue.cc:1048] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.457175 31501 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.485888 31512 consensus_queue.cc:1048] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20260709 12:20:41.576807 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 57969.1us
I20260709 12:20:41.522794 31501 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:41.546095 31531 consensus_queue.cc:1048] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:20:41.591908 31501 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
W20260709 12:20:41.598026 30838 consensus_peers.cc:597] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.607030 30763 consensus_peers.cc:597] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.618203 30763 consensus_peers.cc:597] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.618938 30838 consensus_peers.cc:597] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.638572 30717 raft_consensus.cc:1275] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c178a628712047a88731a915fc93ed19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.638569 30546 catalog_manager.cc:5697] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 reported cstate change: term changed from 1 to 2, leader changed from c18174c596f9488f8be666b85062e329 (127.11.11.129) to eb6634546938405dac9eabbadd34d2b1 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "eb6634546938405dac9eabbadd34d2b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:41.648603 30792 raft_consensus.cc:1275] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:20:41.654531 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.132:36125, user_credentials={real_user=slave}} blocked reactor thread for 132793us
I20260709 12:20:41.672246 31522 consensus_queue.cc:1048] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:41.677002 30763 consensus_peers.cc:597] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.690282 31497 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.691434 31497 tablet_bootstrap.cc:492] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
W20260709 12:20:41.691481 30838 consensus_peers.cc:597] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.694962 31345 consensus_queue.cc:1048] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.701143 31497 ts_tablet_manager.cc:1403] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.334s	user 0.036s	sys 0.006s
I20260709 12:20:41.706033 30799 raft_consensus.cc:1275] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.721206 31563 consensus_queue.cc:1048] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:20:41.739686 30763 consensus_peers.cc:597] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.782586 30838 consensus_peers.cc:597] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.761013 31497 raft_consensus.cc:359] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:41.791343 31497 raft_consensus.cc:740] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
W20260709 12:20:41.792524 30763 consensus_peers.cc:597] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.793246 30763 consensus_peers.cc:597] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.801991 30795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:41.809480 30867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:41.811189 30868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:41.814081 31497 consensus_queue.cc:260] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:41.814817 30763 consensus_peers.cc:597] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.805735 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:41.821835 31497 ts_tablet_manager.cc:1434] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.120s	user 0.008s	sys 0.000s
I20260709 12:20:41.823374 31489 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:41.823855 31497 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:41.824889 31490 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:41.825207 31501 raft_consensus.cc:2749] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:41.829891 31501 raft_consensus.cc:2749] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:41.833913 30838 consensus_peers.cc:597] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.855628 31497 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.856837 31497 tablet_bootstrap.cc:492] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:41.857602 31497 ts_tablet_manager.cc:1403] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.034s	user 0.021s	sys 0.000s
W20260709 12:20:41.866765 30838 consensus_peers.cc:597] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.875802 30689 consensus_peers.cc:597] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:41.877285 30689 consensus_peers.cc:597] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.877990 30689 consensus_peers.cc:597] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.882833 30838 consensus_peers.cc:597] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.891124 30763 consensus_peers.cc:597] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.893038 31497 raft_consensus.cc:359] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:41.894057 31497 raft_consensus.cc:740] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:41.895051 31497 consensus_queue.cc:260] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:41.896505 30838 consensus_peers.cc:597] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.900449 31497 ts_tablet_manager.cc:1434] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.002s
I20260709 12:20:41.901685 31497 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:41.908511 30689 consensus_peers.cc:597] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:41.909616 30689 consensus_peers.cc:597] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.916849 30875 raft_consensus.cc:1275] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.934281 31560 consensus_queue.cc:1048] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.937736 31497 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:20:41.938694 30763 consensus_peers.cc:597] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.939267 31497 tablet_bootstrap.cc:492] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:41.940316 31497 ts_tablet_manager.cc:1403] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.039s	user 0.022s	sys 0.005s
W20260709 12:20:41.940508 30689 consensus_peers.cc:597] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.941424 30689 consensus_peers.cc:597] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.942058 30689 consensus_peers.cc:597] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.944396 31497 raft_consensus.cc:359] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.945472 31497 raft_consensus.cc:740] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:41.946695 31497 consensus_queue.cc:260] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:41.949347 31501 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:41.949965 31501 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:41.950758 30689 consensus_peers.cc:597] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.951675 30689 consensus_peers.cc:597] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.953157 30867 raft_consensus.cc:1275] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer eb6634546938405dac9eabbadd34d2b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:41.955889 31560 consensus_queue.cc:1048] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.959247 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:41.961992 31501 leader_election.cc:290] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:41.963259 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:41.965335 31490 leader_election.cc:304] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:41.966022 31497 ts_tablet_manager.cc:1434] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.000s
I20260709 12:20:41.967096 31497 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:41.971810 30689 consensus_peers.cc:597] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:41.972894 31501 raft_consensus.cc:2749] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:41.979249 30689 consensus_peers.cc:597] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:41.979981 30689 consensus_peers.cc:597] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.000777 30689 consensus_peers.cc:597] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.023437 30763 consensus_peers.cc:597] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.024263 30763 consensus_peers.cc:597] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.052309 30838 consensus_peers.cc:597] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:42.054272 31501 raft_consensus.cc:493] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.054979 31501 raft_consensus.cc:515] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:42.059089 31501 leader_election.cc:290] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:42.061136 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:42.066848 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef90e8d046fc47fcb7d122ffad59a1f3" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:42.072906 31501 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.075803 31501 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.076139 31489 leader_election.cc:304] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:42.083388 31501 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:42.084458 31501 raft_consensus.cc:2749] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.098246 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:42.100565 31497 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.101816 31497 tablet_bootstrap.cc:492] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.102645 31497 ts_tablet_manager.cc:1403] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.136s	user 0.043s	sys 0.034s
I20260709 12:20:42.105563 31497 raft_consensus.cc:359] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.106349 31497 raft_consensus.cc:740] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.108091 31497 consensus_queue.cc:260] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.110227 31497 ts_tablet_manager.cc:1434] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:42.111542 31497 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:42.126938 30689 consensus_peers.cc:597] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:42.131266 31607 raft_consensus.cc:493] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.133518 31607 raft_consensus.cc:515] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:42.145069 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:42.147696 31501 raft_consensus.cc:493] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.148919 31490 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:42.149844 31497 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.148646 31501 raft_consensus.cc:515] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.152341 31497 tablet_bootstrap.cc:492] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.153324 31497 ts_tablet_manager.cc:1403] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.042s	user 0.021s	sys 0.012s
I20260709 12:20:42.171635 31501 leader_election.cc:290] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:42.172710 31501 raft_consensus.cc:2749] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.185324 31607 leader_election.cc:290] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:42.183225 31497 raft_consensus.cc:359] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.186447 31497 raft_consensus.cc:740] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.187503 31497 consensus_queue.cc:260] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.192071 31497 ts_tablet_manager.cc:1434] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.000s
W20260709 12:20:42.192373 30838 consensus_peers.cc:597] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:42.193030 31497 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:42.195086 31490 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 46315.2us
I20260709 12:20:42.198460 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:42.199729 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1be5a3ea104480399760b60f846245b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:42.204789 30793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:42.220112 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:42.232191 31490 leader_election.cc:304] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:42.233111 31616 raft_consensus.cc:2749] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.233724 31490 leader_election.cc:304] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19
I20260709 12:20:42.234686 31616 raft_consensus.cc:2749] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.235121 31607 raft_consensus.cc:493] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.235682 31607 raft_consensus.cc:515] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.236545 31497 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.237854 31607 leader_election.cc:290] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:42.238658 31497 tablet_bootstrap.cc:492] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.240008 31497 ts_tablet_manager.cc:1403] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.047s	user 0.024s	sys 0.015s
I20260709 12:20:42.240429 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:42.245960 31497 raft_consensus.cc:359] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.247186 31497 raft_consensus.cc:740] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.252566 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61996ce5ca7a44f5aa688285da2e65ff" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:42.250674 31497 consensus_queue.cc:260] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.256426 31497 ts_tablet_manager.cc:1434] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.016s	user 0.012s	sys 0.000s
I20260709 12:20:42.256608 31490 leader_election.cc:304] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:42.257885 31607 raft_consensus.cc:2749] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.257923 31497 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:42.266861 30763 consensus_peers.cc:597] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.267479 30763 consensus_peers.cc:597] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.274415 30689 consensus_peers.cc:597] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:20:42.278546 30838 consensus_peers.cc:597] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:42.284569 31497 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.286269 31497 tablet_bootstrap.cc:492] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.287448 31497 ts_tablet_manager.cc:1403] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.009s
I20260709 12:20:42.291208 31607 raft_consensus.cc:493] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.292017 31607 raft_consensus.cc:515] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
W20260709 12:20:42.295083 30689 consensus_peers.cc:597] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:42.296653 31497 raft_consensus.cc:359] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.297495 31497 raft_consensus.cc:740] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.298336 31497 consensus_queue.cc:260] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.301070 31497 ts_tablet_manager.cc:1434] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:20:42.302084 31497 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.347548 30719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:42.390583 31607 leader_election.cc:290] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
W20260709 12:20:42.423101 30838 consensus_peers.cc:597] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:42.413082 30795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "773dfc99de304153af80953c364a815e" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c178a628712047a88731a915fc93ed19" is_pre_election: true
I20260709 12:20:42.435729 31497 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.437179 31497 tablet_bootstrap.cc:492] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.436905 31490 leader_election.cc:304] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:42.438737 31497 ts_tablet_manager.cc:1403] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.137s	user 0.033s	sys 0.019s
I20260709 12:20:42.440871 31607 raft_consensus.cc:2749] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.446755 31497 raft_consensus.cc:359] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.452827 31497 raft_consensus.cc:740] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.454035 31497 consensus_queue.cc:260] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.460014 31497 ts_tablet_manager.cc:1434] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 12:20:42.470837 31497 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.546133 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:41991
I20260709 12:20:42.545297 31642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:41991 every 8 connection(s)
I20260709 12:20:42.548699 31497 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.552309 31497 tablet_bootstrap.cc:492] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.553443 31497 ts_tablet_manager.cc:1403] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.089s	user 0.030s	sys 0.008s
I20260709 12:20:42.571442 31497 raft_consensus.cc:359] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.572561 31497 raft_consensus.cc:740] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.573537 31497 consensus_queue.cc:260] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.583678 31497 ts_tablet_manager.cc:1434] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 12:20:42.584647 31497 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.613298 31616 raft_consensus.cc:493] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:42.613917 31616 raft_consensus.cc:515] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.614326 31595 raft_consensus.cc:3060] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:42.616365 31616 leader_election.cc:290] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:42.618597 30868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:42.628516 31497 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.629767 31497 tablet_bootstrap.cc:492] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.630511 31497 ts_tablet_manager.cc:1403] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.046s	user 0.012s	sys 0.008s
I20260709 12:20:42.633279 31497 raft_consensus.cc:359] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.634004 31497 raft_consensus.cc:740] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.634742 31497 consensus_queue.cc:260] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.636286 31497 ts_tablet_manager.cc:1434] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:20:42.638152 30719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e95ca1a81eea4096a755f54943e95976" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb6634546938405dac9eabbadd34d2b1" is_pre_election: true
I20260709 12:20:42.641403 31497 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.643432 31490 leader_election.cc:304] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c18174c596f9488f8be666b85062e329; no voters: 23ba075510134f3c8caba2370fff0d40, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:42.655179 31616 raft_consensus.cc:2749] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:42.705773 31594 raft_consensus.cc:3060] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:42.754640 31497 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.755820 31497 tablet_bootstrap.cc:492] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.756755 31497 ts_tablet_manager.cc:1403] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.120s	user 0.047s	sys 0.056s
I20260709 12:20:42.759708 31497 raft_consensus.cc:359] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.760622 31497 raft_consensus.cc:740] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.761364 31497 consensus_queue.cc:260] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:42.764191 31497 ts_tablet_manager.cc:1434] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:42.765379 31497 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.860211 31595 raft_consensus.cc:3060] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:42.900066 31497 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.901113 31497 tablet_bootstrap.cc:492] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.901854 31497 ts_tablet_manager.cc:1403] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.137s	user 0.031s	sys 0.032s
I20260709 12:20:42.903136 31644 heartbeater.cc:344] Connected to a master server at 127.11.11.190:37085
I20260709 12:20:42.903791 31644 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:42.904613 31497 raft_consensus.cc:359] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.905525 31497 raft_consensus.cc:740] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.906373 31497 consensus_queue.cc:260] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.904948 31644 heartbeater.cc:507] Master 127.11.11.190:37085 requested a full tablet report, sending...
I20260709 12:20:42.908509 31497 ts_tablet_manager.cc:1434] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:42.911003 31497 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.943760 31497 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.944846 31497 tablet_bootstrap.cc:492] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.945612 31497 ts_tablet_manager.cc:1403] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.036s	user 0.006s	sys 0.015s
I20260709 12:20:42.948390 31497 raft_consensus.cc:359] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.949157 31497 raft_consensus.cc:740] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:42.955036 30546 ts_manager.cc:194] Re-registered known tserver with Master: c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:42.959502 31497 consensus_queue.cc:260] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:42.962858 31497 ts_tablet_manager.cc:1434] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
I20260709 12:20:42.964108 31497 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:42.996953 31497 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.998083 31497 tablet_bootstrap.cc:492] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:42.998965 31497 ts_tablet_manager.cc:1403] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.035s	user 0.020s	sys 0.005s
I20260709 12:20:43.001972 31497 raft_consensus.cc:359] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.002837 31497 raft_consensus.cc:740] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.003676 31497 consensus_queue.cc:260] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.006939 31497 ts_tablet_manager.cc:1434] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:43.009069 31497 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.030362 30546 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41940
I20260709 12:20:43.031212 31595 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:43.045213 31497 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.046496 31497 tablet_bootstrap.cc:492] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.047084 31644 heartbeater.cc:499] Master 127.11.11.190:37085 was elected leader, sending a full tablet report...
I20260709 12:20:43.047269 31497 ts_tablet_manager.cc:1403] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.038s	user 0.021s	sys 0.011s
I20260709 12:20:43.050797 31497 raft_consensus.cc:359] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.051636 31497 raft_consensus.cc:740] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.052450 31497 consensus_queue.cc:260] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.055279 31497 ts_tablet_manager.cc:1434] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:20:43.056252 31497 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.076521 31497 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.077687 31497 tablet_bootstrap.cc:492] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.078496 31497 ts_tablet_manager.cc:1403] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.022s	user 0.016s	sys 0.004s
I20260709 12:20:43.081476 31497 raft_consensus.cc:359] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.082401 31497 raft_consensus.cc:740] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.083930 31497 consensus_queue.cc:260] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.087006 31497 ts_tablet_manager.cc:1434] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:20:43.089594 31497 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.105436 31497 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.106526 31497 tablet_bootstrap.cc:492] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.107250 31497 ts_tablet_manager.cc:1403] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.003s
I20260709 12:20:43.111136 31497 raft_consensus.cc:359] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.111914 31497 raft_consensus.cc:740] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.112731 31497 consensus_queue.cc:260] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.115262 31497 ts_tablet_manager.cc:1434] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:43.116175 31497 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.141466 31497 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.142621 31497 tablet_bootstrap.cc:492] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.143280 31497 ts_tablet_manager.cc:1403] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.027s	user 0.017s	sys 0.000s
I20260709 12:20:43.145821 31497 raft_consensus.cc:359] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.146601 31497 raft_consensus.cc:740] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.147908 31497 consensus_queue.cc:260] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.150799 31497 ts_tablet_manager.cc:1434] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:20:43.151829 31497 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.253177 31594 raft_consensus.cc:3060] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:43.310128 31497 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.311185 31497 tablet_bootstrap.cc:492] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.311977 31497 ts_tablet_manager.cc:1403] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.160s	user 0.015s	sys 0.039s
I20260709 12:20:43.319648 31497 raft_consensus.cc:359] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.320611 31497 raft_consensus.cc:740] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.322142 31497 consensus_queue.cc:260] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.324625 31497 ts_tablet_manager.cc:1434] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:20:43.325484 31497 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.372315 31497 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.373551 31497 tablet_bootstrap.cc:492] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.374413 31497 ts_tablet_manager.cc:1403] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.049s	user 0.019s	sys 0.008s
I20260709 12:20:43.378293 31497 raft_consensus.cc:359] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.379102 31497 raft_consensus.cc:740] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.379882 31497 consensus_queue.cc:260] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.382280 31497 ts_tablet_manager.cc:1434] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:43.385103 31497 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.405278 31497 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.406246 31497 tablet_bootstrap.cc:492] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.406970 31497 ts_tablet_manager.cc:1403] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.001s
I20260709 12:20:43.409230 31497 raft_consensus.cc:359] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.409866 31497 raft_consensus.cc:740] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.410574 31497 consensus_queue.cc:260] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.412694 31497 ts_tablet_manager.cc:1434] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:20:43.414292 31497 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.429529 31497 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.430442 31497 tablet_bootstrap.cc:492] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.431047 31497 ts_tablet_manager.cc:1403] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.017s	user 0.015s	sys 0.000s
I20260709 12:20:43.433214 31497 raft_consensus.cc:359] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.433815 31497 raft_consensus.cc:740] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.434689 31497 consensus_queue.cc:260] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.436872 31497 ts_tablet_manager.cc:1434] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.004s
I20260709 12:20:43.441558 31497 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.462445 31497 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.463593 31497 tablet_bootstrap.cc:492] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.464946 31497 ts_tablet_manager.cc:1403] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.027s	user 0.021s	sys 0.000s
I20260709 12:20:43.467536 31497 raft_consensus.cc:359] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.468195 31497 raft_consensus.cc:740] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.468976 31497 consensus_queue.cc:260] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.471030 31497 ts_tablet_manager.cc:1434] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:43.472939 31497 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.480273 31594 raft_consensus.cc:3060] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:43.504171 31497 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.505892 31497 tablet_bootstrap.cc:492] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.506645 31497 ts_tablet_manager.cc:1403] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.008s
I20260709 12:20:43.516850 31497 raft_consensus.cc:359] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.517685 31497 raft_consensus.cc:740] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.518447 31497 consensus_queue.cc:260] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.521000 31497 ts_tablet_manager.cc:1434] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.002s
I20260709 12:20:43.538185 31497 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.573401 31497 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.575217 31497 tablet_bootstrap.cc:492] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.576321 31497 ts_tablet_manager.cc:1403] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.015s
I20260709 12:20:43.585196 31497 raft_consensus.cc:359] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.586427 31497 raft_consensus.cc:740] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.587615 31497 consensus_queue.cc:260] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.589558 31497 ts_tablet_manager.cc:1434] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:20:43.590869 31497 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:43.617913 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 71062.5us
W20260709 12:20:43.670917 30763 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 122483us
I20260709 12:20:43.690205 31497 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.691349 31497 tablet_bootstrap.cc:492] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.692082 31497 ts_tablet_manager.cc:1403] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.102s	user 0.045s	sys 0.043s
I20260709 12:20:43.695573 31497 raft_consensus.cc:359] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.696308 31497 raft_consensus.cc:740] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.697093 31497 consensus_queue.cc:260] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.703987 31497 ts_tablet_manager.cc:1434] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:20:43.704991 31497 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.723610 31497 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.724511 31497 tablet_bootstrap.cc:492] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.726449 31497 ts_tablet_manager.cc:1403] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.022s	user 0.010s	sys 0.007s
I20260709 12:20:43.729193 31497 raft_consensus.cc:359] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.729969 31497 raft_consensus.cc:740] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.730800 31497 consensus_queue.cc:260] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.734177 31497 ts_tablet_manager.cc:1434] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:43.735934 31497 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.756301 31497 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.757249 31497 tablet_bootstrap.cc:492] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.758065 31497 ts_tablet_manager.cc:1403] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.010s
I20260709 12:20:43.760304 31497 raft_consensus.cc:359] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.761022 31497 raft_consensus.cc:740] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.761795 31497 consensus_queue.cc:260] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:43.764354 31497 ts_tablet_manager.cc:1434] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:43.765439 31497 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.781821 31497 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.782696 31497 tablet_bootstrap.cc:492] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.783380 31497 ts_tablet_manager.cc:1403] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.003s
I20260709 12:20:43.785545 31497 raft_consensus.cc:359] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.786245 31497 raft_consensus.cc:740] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.786906 31497 consensus_queue.cc:260] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:43.789144 31497 ts_tablet_manager.cc:1434] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:43.790833 31497 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.812638 31497 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.813848 31497 tablet_bootstrap.cc:492] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.814673 31497 ts_tablet_manager.cc:1403] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.024s	user 0.019s	sys 0.002s
I20260709 12:20:43.817101 31497 raft_consensus.cc:359] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.817914 31497 raft_consensus.cc:740] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:43.818691 31497 consensus_queue.cc:260] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:43.821473 31497 ts_tablet_manager.cc:1434] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:43.823091 31497 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:43.929831 31497 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:43.933866 31497 tablet_bootstrap.cc:492] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:43.935868 31497 ts_tablet_manager.cc:1403] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.113s	user 0.017s	sys 0.006s
I20260709 12:20:43.996706 31592 raft_consensus.cc:3060] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:44.015795 31593 raft_consensus.cc:3060] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:44.011873 31497 raft_consensus.cc:359] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.031164 31497 raft_consensus.cc:740] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:44.041858 31497 consensus_queue.cc:260] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:44.046455 30838 consensus_peers.cc:597] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:44.077811 31497 ts_tablet_manager.cc:1434] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.141s	user 0.015s	sys 0.001s
I20260709 12:20:44.080233 31497 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Bootstrap starting.
W20260709 12:20:44.151921 30838 consensus_peers.cc:597] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:20:44.289553 31497 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=1 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:20:44.293063 30838 consensus_peers.cc:597] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:20:44.302219 31497 tablet_bootstrap.cc:492] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:44.310142 31497 ts_tablet_manager.cc:1403] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.230s	user 0.023s	sys 0.028s
I20260709 12:20:44.316672 31497 raft_consensus.cc:359] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:44.338326 31497 raft_consensus.cc:740] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:44.339321 31497 consensus_queue.cc:260] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:44.341454 31497 ts_tablet_manager.cc:1434] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.000s
I20260709 12:20:44.345786 31497 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:44.445380 31592 raft_consensus.cc:3060] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:44.484217 31497 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:44.485838 31497 tablet_bootstrap.cc:492] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:44.486871 31497 ts_tablet_manager.cc:1403] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.142s	user 0.033s	sys 0.042s
I20260709 12:20:44.491766 31497 raft_consensus.cc:359] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.493237 31497 raft_consensus.cc:740] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:44.498534 31497 consensus_queue.cc:260] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.502635 31497 ts_tablet_manager.cc:1434] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.002s
I20260709 12:20:44.503844 31497 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:44.538513 31497 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:44.539651 31497 tablet_bootstrap.cc:492] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:44.540447 31497 ts_tablet_manager.cc:1403] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.009s
I20260709 12:20:44.546758 31497 raft_consensus.cc:359] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.547534 31497 raft_consensus.cc:740] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:44.548234 31497 consensus_queue.cc:260] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.551512 31497 ts_tablet_manager.cc:1434] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.001s
I20260709 12:20:44.552610 31497 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Bootstrap starting.
I20260709 12:20:44.576323 31497 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:44.577492 31497 tablet_bootstrap.cc:492] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Bootstrap complete.
I20260709 12:20:44.578333 31497 ts_tablet_manager.cc:1403] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Time spent bootstrapping tablet: real 0.026s	user 0.021s	sys 0.001s
I20260709 12:20:44.581394 31497 raft_consensus.cc:359] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.582195 31497 raft_consensus.cc:740] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18174c596f9488f8be666b85062e329, State: Initialized, Role: FOLLOWER
I20260709 12:20:44.583063 31497 consensus_queue.cc:260] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:44.585824 31497 ts_tablet_manager.cc:1434] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:20:44.753790 31592 raft_consensus.cc:3060] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.600541 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:52.667732 30799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:52.668565 30799 raft_consensus.cc:606] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:52.686933 30717 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9"
dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:51634
I20260709 12:20:52.687418 30717 raft_consensus.cc:606] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:52.702477 30874 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:38308
I20260709 12:20:52.703120 30874 raft_consensus.cc:606] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:52.706818 30874 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:38308
I20260709 12:20:52.707414 30874 raft_consensus.cc:606] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:52.711153 30796 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:52.711777 30796 raft_consensus.cc:606] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:52.714722 30874 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:38308
I20260709 12:20:52.715466 30874 raft_consensus.cc:606] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:52.717025 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:20:52.719398 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:52.720710 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:52.726974 31721 raft_consensus.cc:993] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.727587 31721 raft_consensus.cc:1081] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.729382 31592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58998
I20260709 12:20:52.730136 31592 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.730605 31592 raft_consensus.cc:3060] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.740357 31592 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.744609 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:52.745456 30799 raft_consensus.cc:3060] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.745713 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5899b18ce3a453ea77be2cdf20e543a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:52.746423 30874 raft_consensus.cc:3055] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.746788 30874 raft_consensus.cc:740] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:52.747727 30874 consensus_queue.cc:260] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.749167 30874 raft_consensus.cc:3060] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.758725 30799 raft_consensus.cc:2468] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.745451 31592 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:52.760514 31490 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:52.762080 30874 raft_consensus.cc:2468] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.764137 31733 raft_consensus.cc:2804] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.765925 31733 raft_consensus.cc:697] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:52.767067 31733 consensus_queue.cc:237] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.786880 30548 catalog_manager.cc:5697] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from 23ba075510134f3c8caba2370fff0d40 (127.11.11.132) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:52.852627 31717 raft_consensus.cc:993] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.853242 31717 raft_consensus.cc:1081] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.856917 31592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58998
I20260709 12:20:52.857618 31592 raft_consensus.cc:493] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.858073 31592 raft_consensus.cc:3060] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.863101 31728 raft_consensus.cc:993] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.864027 31740 raft_consensus.cc:1081] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.866987 31594 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:59002
I20260709 12:20:52.867693 31594 raft_consensus.cc:493] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.868090 31594 raft_consensus.cc:3060] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.873438 31717 raft_consensus.cc:993] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.869405 31592 raft_consensus.cc:515] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.873994 31717 raft_consensus.cc:1081] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.876714 31592 leader_election.cc:290] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:52.877329 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:52.877709 31591 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58998
I20260709 12:20:52.878289 30874 raft_consensus.cc:3055] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.878755 31591 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:52.878780 30874 raft_consensus.cc:740] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:52.876875 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2f67741397b40a0b491cddf4c7e7dea" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:52.879487 31591 raft_consensus.cc:3060] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.880595 30799 raft_consensus.cc:3060] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.882740 31594 raft_consensus.cc:515] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:52.886132 31594 leader_election.cc:290] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:52.888039 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:52.880829 30874 consensus_queue.cc:260] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.889019 30724 raft_consensus.cc:3060] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.889225 30796 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9041cf65147424a8c1ef0e1d03ab95a" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:52.912434 30796 raft_consensus.cc:3055] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.912784 30796 raft_consensus.cc:740] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:52.913597 30796 consensus_queue.cc:260] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:52.914839 30874 raft_consensus.cc:3060] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.915131 30796 raft_consensus.cc:3060] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.895625 30799 raft_consensus.cc:2468] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.922536 31490 leader_election.cc:304] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:52.924065 30724 raft_consensus.cc:2468] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.908635 31591 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.925558 31733 raft_consensus.cc:2804] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.926396 31733 raft_consensus.cc:697] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:52.927309 31733 consensus_queue.cc:237] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.928879 31591 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:52.929175 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:52.930009 30872 raft_consensus.cc:3055] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:52.930321 30872 raft_consensus.cc:740] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:52.932629 31490 leader_election.cc:304] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:52.933784 31734 raft_consensus.cc:2804] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.939350 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:52.940160 30799 raft_consensus.cc:3060] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.944269 30796 raft_consensus.cc:2468] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.949736 30872 consensus_queue.cc:260] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
W20260709 12:20:52.953310 30841 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.953366 30872 raft_consensus.cc:3060] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.968694 30799 raft_consensus.cc:2468] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.969290 30874 raft_consensus.cc:2468] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.982239 31490 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:52.983879 31733 raft_consensus.cc:2804] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:52.984539 31733 raft_consensus.cc:697] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:52.974924 31734 raft_consensus.cc:697] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:52.993410 31734 consensus_queue.cc:237] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:52.993553 30872 raft_consensus.cc:2468] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:52.993376 31733 consensus_queue.cc:237] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:52.997464 31728 raft_consensus.cc:993] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:52.998104 31728 raft_consensus.cc:1081] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:53.001946 31591 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:59002
I20260709 12:20:53.002908 31591 raft_consensus.cc:493] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:53.003414 31591 raft_consensus.cc:3060] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.036082 31591 raft_consensus.cc:515] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:53.043329 31591 leader_election.cc:290] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:53.044255 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:53.045058 30799 raft_consensus.cc:3055] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:53.045158 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9cd28a99d641698ab0d99e20bac569" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:53.045447 30799 raft_consensus.cc:740] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:53.045866 30872 raft_consensus.cc:3060] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.046434 30799 consensus_queue.cc:260] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:53.048071 30799 raft_consensus.cc:3060] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.058056 30799 raft_consensus.cc:2468] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:53.060021 30872 raft_consensus.cc:2468] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:53.059736 30548 catalog_manager.cc:5697] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from 23ba075510134f3c8caba2370fff0d40 (127.11.11.132) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:53.061160 31490 leader_election.cc:304] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:53.062561 31733 raft_consensus.cc:2804] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.063792 31733 raft_consensus.cc:697] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:53.064862 31733 consensus_queue.cc:237] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:53.061589 30548 catalog_manager.cc:5697] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.068226 30548 catalog_manager.cc:5697] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from 23ba075510134f3c8caba2370fff0d40 (127.11.11.132) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:53.094772 30548 catalog_manager.cc:5697] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:53.136754 31727 raft_consensus.cc:993] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:53.137502 31726 raft_consensus.cc:1081] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:53.139402 31592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58996
I20260709 12:20:53.139982 31592 raft_consensus.cc:493] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:53.140352 31592 raft_consensus.cc:3060] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.149049 31592 raft_consensus.cc:515] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:53.152287 31592 leader_election.cc:290] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:53.154126 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:53.154927 30799 raft_consensus.cc:3060] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.155673 30724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5651d4117243d69cfd21927e6ce1f9" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:53.156369 30724 raft_consensus.cc:3055] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:53.156664 30724 raft_consensus.cc:740] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:53.157660 30724 consensus_queue.cc:260] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:53.159134 30724 raft_consensus.cc:3060] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.163416 30799 raft_consensus.cc:2468] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:53.164809 31490 leader_election.cc:304] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:53.165920 31733 raft_consensus.cc:2804] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.166469 31733 raft_consensus.cc:697] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:53.167446 31733 consensus_queue.cc:237] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:53.172062 30724 raft_consensus.cc:2468] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:53.217942 30548 catalog_manager.cc:5697] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from eb6634546938405dac9eabbadd34d2b1 (127.11.11.130) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:53.222822 30799 raft_consensus.cc:1275] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.238610 31734 consensus_queue.cc:1048] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.274202 30868 raft_consensus.cc:1275] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.276221 31742 consensus_queue.cc:1048] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.430606 30792 raft_consensus.cc:1275] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.432596 31742 consensus_queue.cc:1048] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:53.445490 30872 raft_consensus.cc:1275] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.447728 31760 consensus_queue.cc:1048] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.445490 30868 raft_consensus.cc:1275] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.450680 31760 consensus_queue.cc:1048] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:53.514370 30799 raft_consensus.cc:1275] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.516206 31734 consensus_queue.cc:1048] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:53.539505 30799 raft_consensus.cc:1275] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.544196 30724 raft_consensus.cc:1275] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.544086 31734 consensus_queue.cc:1048] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:53.548346 31734 consensus_queue.cc:1048] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.556362 30872 raft_consensus.cc:1275] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.561478 31734 consensus_queue.cc:1048] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:20:53.576397 30792 raft_consensus.cc:1275] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.583846 31734 consensus_queue.cc:1048] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.585964 30717 raft_consensus.cc:1275] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.594825 31734 consensus_queue.cc:1048] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.675365 30799 raft_consensus.cc:1275] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:53.677848 31734 consensus_queue.cc:1048] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:53.809988 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:53.856408 30799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "23ba075510134f3c8caba2370fff0d40"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:53.857342 30799 raft_consensus.cc:606] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:53.861056 30799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:53.861826 30799 raft_consensus.cc:606] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:53.865226 30799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:53.865814 30799 raft_consensus.cc:606] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:53.869391 30717 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8bc4eca36172433d9134485920115dcc"
dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:51634
I20260709 12:20:53.870173 30717 raft_consensus.cc:606] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:53.873378 30717 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5"
dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
mode: GRACEFUL
new_leader_uuid: "c178a628712047a88731a915fc93ed19"
 from {username='slave'} at 127.0.0.1:51634
I20260709 12:20:53.874099 30717 raft_consensus.cc:606] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Received request to transfer leadership to TS c178a628712047a88731a915fc93ed19
I20260709 12:20:53.875660 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:53.878185 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:53.879110 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:54.166019 31744 raft_consensus.cc:993] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:54.179054 31744 raft_consensus.cc:1081] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:54.233392 31750 raft_consensus.cc:993] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:54.234989 31750 raft_consensus.cc:1081] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:54.235195 31592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:59002
W20260709 12:20:54.235550 30764 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 116322us
I20260709 12:20:54.238587 31592 raft_consensus.cc:493] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:54.240769 31592 raft_consensus.cc:3060] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.255611 31590 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8bc4eca36172433d9134485920115dcc"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58996
I20260709 12:20:54.256737 31590 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:54.257336 31590 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:54.268687 31779 raft_consensus.cc:993] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: : Instructing follower 23ba075510134f3c8caba2370fff0d40 to start an election
I20260709 12:20:54.269690 31743 raft_consensus.cc:1081] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer 23ba075510134f3c8caba2370fff0d40 to start an election
I20260709 12:20:54.271368 31779 raft_consensus.cc:993] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:54.272359 31779 raft_consensus.cc:1081] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:54.274133 30868 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
 from {username='slave'} at 127.0.0.1:59014
I20260709 12:20:54.275068 30868 raft_consensus.cc:493] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:54.275527 30868 raft_consensus.cc:3060] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.275763 31593 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:59002
I20260709 12:20:54.276728 31593 raft_consensus.cc:493] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:54.277204 31593 raft_consensus.cc:3060] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.293870 31592 raft_consensus.cc:515] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:54.321684 31592 leader_election.cc:290] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:54.325590 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:54.328325 31590 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:54.329286 30799 raft_consensus.cc:3055] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:54.333184 30799 raft_consensus.cc:740] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:54.334542 30799 consensus_queue.cc:260] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:54.336954 30799 raft_consensus.cc:3060] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:54.341799 30839 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 52768us
I20260709 12:20:54.348434 31590 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 3 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:54.350390 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:54.354635 30874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4eca36172433d9134485920115dcc" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:54.355285 30720 raft_consensus.cc:3055] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:54.355499 30874 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:54.355674 30720 raft_consensus.cc:740] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:54.350594 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a01bc7b51f24fb5b74aed9349c72e91" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:54.357993 30872 raft_consensus.cc:3060] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.356871 30720 consensus_queue.cc:260] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:54.359349 30720 raft_consensus.cc:3060] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:54.360212 30799 raft_consensus.cc:2468] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:54.370586 31490 leader_election.cc:304] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:54.371732 31760 raft_consensus.cc:2804] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:54.373873 31760 raft_consensus.cc:697] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:54.377483 30720 raft_consensus.cc:2468] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 3.
I20260709 12:20:54.377419 31760 consensus_queue.cc:237] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:54.377851 30868 raft_consensus.cc:515] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:54.395757 31490 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:54.397732 31769 raft_consensus.cc:2804] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:54.398447 31769 raft_consensus.cc:697] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 3 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:54.399682 31769 consensus_queue.cc:237] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:54.402331 30868 leader_election.cc:290] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:54.404992 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:54.405736 30799 raft_consensus.cc:3055] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:54.406051 30799 raft_consensus.cc:740] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:54.407230 30799 consensus_queue.cc:260] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:54.410897 30799 raft_consensus.cc:3060] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.419301 30799 raft_consensus.cc:2468] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:54.425464 30872 raft_consensus.cc:2468] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:54.425293 30874 raft_consensus.cc:2468] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 3.
I20260709 12:20:54.430790 30720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2aafa16c29f04448945f5db4aa191223" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:54.431715 30720 raft_consensus.cc:3060] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.424510 31593 raft_consensus.cc:515] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:54.433743 30839 leader_election.cc:304] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:54.435429 31749 raft_consensus.cc:2804] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:54.436085 31749 raft_consensus.cc:697] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:54.437290 31593 leader_election.cc:290] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:54.438598 30792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:54.439431 30792 raft_consensus.cc:3055] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:54.439787 30792 raft_consensus.cc:740] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:54.438342 31749 consensus_queue.cc:237] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:54.440783 30792 consensus_queue.cc:260] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:54.442448 30792 raft_consensus.cc:3060] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.457374 31750 raft_consensus.cc:993] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: : Instructing follower c178a628712047a88731a915fc93ed19 to start an election
I20260709 12:20:54.458357 31727 raft_consensus.cc:1081] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Signalling peer c178a628712047a88731a915fc93ed19 to start an election
I20260709 12:20:54.460572 30868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8203be2a35f743e5ae45a1250189fb1f" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:54.451400 30720 raft_consensus.cc:2468] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:54.462488 30868 raft_consensus.cc:3060] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.482378 30796 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5"
dest_uuid: "c178a628712047a88731a915fc93ed19"
 from {username='slave'} at 127.0.0.1:55560
I20260709 12:20:54.483436 30796 raft_consensus.cc:493] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:54.483940 30796 raft_consensus.cc:3060] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.482798 30549 catalog_manager.cc:5697] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:54.488709 30792 raft_consensus.cc:2468] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:54.576746 31490 leader_election.cc:304] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:54.572633 30796 raft_consensus.cc:515] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:54.581791 31769 raft_consensus.cc:2804] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:20:54.581801 30689 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:41991, user_credentials={real_user=slave}} blocked reactor thread for 71486us
W20260709 12:20:54.587644 30690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:39529, user_credentials={real_user=slave}} blocked reactor thread for 79670.2us
I20260709 12:20:54.589735 30868 raft_consensus.cc:2468] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:54.623677 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:54.624604 30717 raft_consensus.cc:3055] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:54.624981 30717 raft_consensus.cc:740] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:54.634358 30796 leader_election.cc:290] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:54.632882 31769 raft_consensus.cc:697] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:54.663475 30546 catalog_manager.cc:5697] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:54.691682 30868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecdffc08164a4124ad64c3b376d308d5" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:54.692476 30868 raft_consensus.cc:3060] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.681555 31769 consensus_queue.cc:237] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:54.744408 30868 raft_consensus.cc:2468] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 2.
I20260709 12:20:54.758279 30763 leader_election.cc:304] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:54.764165 31776 raft_consensus.cc:2804] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:54.764814 31776 raft_consensus.cc:697] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 2 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:54.764957 30717 consensus_queue.cc:260] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:54.767000 31776 consensus_queue.cc:237] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:54.767468 30717 raft_consensus.cc:3060] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:54.796452 30797 raft_consensus.cc:1275] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:54.802055 31769 consensus_queue.cc:1048] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:54.819984 30717 raft_consensus.cc:2468] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 2.
I20260709 12:20:54.858621 30868 raft_consensus.cc:1275] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:54.862429 31734 consensus_queue.cc:1048] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:54.861418 30549 catalog_manager.cc:5697] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 1 to 2, leader changed from eb6634546938405dac9eabbadd34d2b1 (127.11.11.130) to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:54.871227 30546 catalog_manager.cc:5697] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 2 to 3, leader changed from eb6634546938405dac9eabbadd34d2b1 (127.11.11.130) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 3 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:54.873178 30546 catalog_manager.cc:5697] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:54.878521 30796 raft_consensus.cc:1275] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:54.881958 31780 consensus_queue.cc:1048] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:54.884567 30717 raft_consensus.cc:1275] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:54.887081 31780 consensus_queue.cc:1048] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:54.911427 30720 raft_consensus.cc:1275] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:54.911799 30868 raft_consensus.cc:1275] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:54.914108 31760 consensus_queue.cc:1048] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:54.919550 31760 consensus_queue.cc:1048] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:55.107228 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:55.215991 31490 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:37181, user_credentials={real_user=slave}} blocked reactor thread for 69189.8us
I20260709 12:20:55.236397 30717 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110"
dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:51634
I20260709 12:20:55.237237 30717 raft_consensus.cc:606] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:55.252468 30869 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
mode: GRACEFUL
new_leader_uuid: "c178a628712047a88731a915fc93ed19"
 from {username='slave'} at 127.0.0.1:38308
I20260709 12:20:55.253146 30869 raft_consensus.cc:606] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Received request to transfer leadership to TS c178a628712047a88731a915fc93ed19
I20260709 12:20:55.259835 30799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "23ba075510134f3c8caba2370fff0d40"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:55.260768 30799 raft_consensus.cc:606] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:55.265602 30723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05"
dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:51634
I20260709 12:20:55.266397 30723 raft_consensus.cc:606] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:55.270344 30723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e614ae4fde394b488924f185e2808b18"
dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
mode: GRACEFUL
new_leader_uuid: "23ba075510134f3c8caba2370fff0d40"
 from {username='slave'} at 127.0.0.1:51634
I20260709 12:20:55.271186 30723 raft_consensus.cc:606] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Received request to transfer leadership to TS 23ba075510134f3c8caba2370fff0d40
I20260709 12:20:55.272842 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:55.276804 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:55.278649 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:55.280592 31749 raft_consensus.cc:993] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: : Instructing follower c178a628712047a88731a915fc93ed19 to start an election
I20260709 12:20:55.281251 31749 raft_consensus.cc:1081] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Signalling peer c178a628712047a88731a915fc93ed19 to start an election
I20260709 12:20:55.288116 30799 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07"
dest_uuid: "c178a628712047a88731a915fc93ed19"
 from {username='slave'} at 127.0.0.1:55568
I20260709 12:20:55.291934 30797 raft_consensus.cc:1275] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.292280 30799 raft_consensus.cc:493] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.292874 30799 raft_consensus.cc:3060] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.297134 30869 raft_consensus.cc:1275] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.298930 31734 consensus_queue.cc:1048] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:55.300625 31769 consensus_queue.cc:1048] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:55.347448 31690 raft_consensus.cc:993] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:55.348188 31726 raft_consensus.cc:1081] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:55.359122 31594 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58996
I20260709 12:20:55.359772 31594 raft_consensus.cc:493] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.360174 31594 raft_consensus.cc:3060] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.361168 30799 raft_consensus.cc:515] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.363888 30799 leader_election.cc:290] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:55.365526 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:55.366533 30723 raft_consensus.cc:3060] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.368873 30868 raft_consensus.cc:1275] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c178a628712047a88731a915fc93ed19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.371294 31775 consensus_queue.cc:1048] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:55.372828 31594 raft_consensus.cc:515] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:55.368155 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11bb43acd4e84e5aa5c2731a70542e07" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:55.380371 30869 raft_consensus.cc:3055] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.380805 30869 raft_consensus.cc:740] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:55.381944 30869 consensus_queue.cc:260] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.383270 31594 leader_election.cc:290] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:55.383446 30869 raft_consensus.cc:3060] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.384543 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:55.385270 30717 raft_consensus.cc:3055] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.385582 30717 raft_consensus.cc:740] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:55.386673 30717 consensus_queue.cc:260] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:55.389619 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04fe0c41d3d440498251b09d3d6f5110" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:55.390489 30797 raft_consensus.cc:3060] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.394701 30723 raft_consensus.cc:2468] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 2.
I20260709 12:20:55.396421 30764 leader_election.cc:304] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:55.397507 31775 raft_consensus.cc:2804] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.401513 30797 raft_consensus.cc:2468] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:55.401520 31775 raft_consensus.cc:697] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 2 LEADER]: Becoming Leader. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:55.404417 31775 consensus_queue.cc:237] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.405570 31490 leader_election.cc:304] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:55.408085 31810 raft_consensus.cc:2804] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.408684 31810 raft_consensus.cc:697] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:55.409577 31810 consensus_queue.cc:237] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:55.412264 30717 raft_consensus.cc:3060] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.430668 31726 raft_consensus.cc:993] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:55.418197 30723 raft_consensus.cc:1275] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c178a628712047a88731a915fc93ed19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.414577 30869 raft_consensus.cc:2468] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c178a628712047a88731a915fc93ed19 in term 2.
I20260709 12:20:55.435708 31811 raft_consensus.cc:1081] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:55.437186 31743 consensus_queue.cc:1048] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:55.438508 30717 raft_consensus.cc:2468] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:55.455672 31594 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58996
I20260709 12:20:55.456420 31594 raft_consensus.cc:493] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.456871 31594 raft_consensus.cc:3060] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.478309 30546 catalog_manager.cc:5697] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from eb6634546938405dac9eabbadd34d2b1 (127.11.11.130) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:55.486353 31594 raft_consensus.cc:515] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:55.487867 30548 catalog_manager.cc:5697] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 reported cstate change: term changed from 1 to 2, leader changed from 23ba075510134f3c8caba2370fff0d40 (127.11.11.132) to c178a628712047a88731a915fc93ed19 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "c178a628712047a88731a915fc93ed19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:55.488940 31594 leader_election.cc:290] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:55.490619 30869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:55.491048 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5be73f56e004d61ba8762d1afa0cc05" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:55.491405 30869 raft_consensus.cc:3060] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.491823 30717 raft_consensus.cc:3055] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.492174 30717 raft_consensus.cc:740] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:55.493078 30717 consensus_queue.cc:260] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:55.494436 30717 raft_consensus.cc:3060] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.598040 30869 raft_consensus.cc:2468] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:55.598907 30717 raft_consensus.cc:2468] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:55.605772 31490 leader_election.cc:304] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:55.607540 31785 raft_consensus.cc:2804] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.608341 31785 raft_consensus.cc:697] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:55.614223 31785 consensus_queue.cc:237] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:55.669327 31744 raft_consensus.cc:993] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: : Instructing follower 23ba075510134f3c8caba2370fff0d40 to start an election
I20260709 12:20:55.669932 31744 raft_consensus.cc:1081] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer 23ba075510134f3c8caba2370fff0d40 to start an election
I20260709 12:20:55.676208 30868 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
 from {username='slave'} at 127.0.0.1:59014
I20260709 12:20:55.676843 30868 raft_consensus.cc:493] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.677237 30868 raft_consensus.cc:3060] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.704497 30868 raft_consensus.cc:515] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.708760 30868 leader_election.cc:290] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:55.711319 30723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:55.711494 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b29c912615fb4627ba5c12d4c0f193d3" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:55.712153 30723 raft_consensus.cc:3060] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.712605 30797 raft_consensus.cc:3055] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.713420 30797 raft_consensus.cc:740] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:55.714776 30797 consensus_queue.cc:260] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.717988 30797 raft_consensus.cc:3060] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.729092 30797 raft_consensus.cc:2468] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:55.730784 30839 leader_election.cc:304] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:55.731946 31774 raft_consensus.cc:2804] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.732776 31774 raft_consensus.cc:697] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:55.731509 30548 catalog_manager.cc:5697] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from eb6634546938405dac9eabbadd34d2b1 (127.11.11.130) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:55.734007 31774 consensus_queue.cc:237] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.738795 30723 raft_consensus.cc:2468] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:55.761461 30548 catalog_manager.cc:5697] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:55.800110 31727 raft_consensus.cc:993] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: : Instructing follower 23ba075510134f3c8caba2370fff0d40 to start an election
I20260709 12:20:55.800750 31727 raft_consensus.cc:1081] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Signalling peer 23ba075510134f3c8caba2370fff0d40 to start an election
I20260709 12:20:55.802826 30868 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e614ae4fde394b488924f185e2808b18"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
 from {username='slave'} at 127.0.0.1:59028
I20260709 12:20:55.803481 30868 raft_consensus.cc:493] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:55.804016 30868 raft_consensus.cc:3060] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.813694 30868 raft_consensus.cc:515] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.816409 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e614ae4fde394b488924f185e2808b18" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:55.816654 30868 leader_election.cc:290] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:55.817167 30717 raft_consensus.cc:3055] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:55.817580 30717 raft_consensus.cc:740] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb6634546938405dac9eabbadd34d2b1, State: Running, Role: LEADER
I20260709 12:20:55.816967 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e614ae4fde394b488924f185e2808b18" candidate_uuid: "23ba075510134f3c8caba2370fff0d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:55.818224 30797 raft_consensus.cc:3060] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.818502 30717 consensus_queue.cc:260] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.819932 30717 raft_consensus.cc:3060] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.828058 30797 raft_consensus.cc:2468] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:55.829514 30717 raft_consensus.cc:2468] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23ba075510134f3c8caba2370fff0d40 in term 2.
I20260709 12:20:55.830533 30839 leader_election.cc:304] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40, c178a628712047a88731a915fc93ed19; no voters: 
I20260709 12:20:55.832374 31751 raft_consensus.cc:2804] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:55.832994 31751 raft_consensus.cc:697] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Becoming Leader. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:55.834547 31751 consensus_queue.cc:237] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } }
I20260709 12:20:55.846395 30720 raft_consensus.cc:1275] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.847692 30548 catalog_manager.cc:5697] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 reported cstate change: term changed from 1 to 2, leader changed from eb6634546938405dac9eabbadd34d2b1 (127.11.11.130) to 23ba075510134f3c8caba2370fff0d40 (127.11.11.132). New cstate: current_term: 2 leader_uuid: "23ba075510134f3c8caba2370fff0d40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:55.848256 31810 consensus_queue.cc:1048] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:55.869504 30797 raft_consensus.cc:1275] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.871717 31810 consensus_queue.cc:1048] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:55.964224 30868 raft_consensus.cc:1275] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c178a628712047a88731a915fc93ed19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.965764 31743 consensus_queue.cc:1048] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:55.978264 30720 raft_consensus.cc:1275] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c178a628712047a88731a915fc93ed19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:55.979846 31743 consensus_queue.cc:1048] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:56.056643 30720 raft_consensus.cc:1275] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.058430 31810 consensus_queue.cc:1048] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:56.064636 30872 raft_consensus.cc:1275] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.066685 31810 consensus_queue.cc:1048] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:56.221040 30720 raft_consensus.cc:1275] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.223186 31751 consensus_queue.cc:1048] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:56.237363 30797 raft_consensus.cc:1275] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.239396 31790 consensus_queue.cc:1048] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:56.255227 30717 raft_consensus.cc:1275] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.258109 31751 consensus_queue.cc:1048] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:56.280234 30797 raft_consensus.cc:1275] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer 23ba075510134f3c8caba2370fff0d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.282326 31751 consensus_queue.cc:1048] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:56.409133 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:56.463683 30797 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523"
dest_uuid: "c178a628712047a88731a915fc93ed19"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:53660
I20260709 12:20:56.464388 30797 raft_consensus.cc:606] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:56.471478 30872 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:38308
I20260709 12:20:56.472218 30872 raft_consensus.cc:606] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:56.479053 30868 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c349e457659c45079da022887088cc7b"
dest_uuid: "23ba075510134f3c8caba2370fff0d40"
mode: GRACEFUL
new_leader_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:38308
I20260709 12:20:56.479786 30868 raft_consensus.cc:606] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Received request to transfer leadership to TS c18174c596f9488f8be666b85062e329
I20260709 12:20:56.484375 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:20:56.486790 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:56.487735 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:56.524292 31751 raft_consensus.cc:993] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:56.525069 31751 raft_consensus.cc:1081] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:56.527139 31592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c349e457659c45079da022887088cc7b"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58998
I20260709 12:20:56.527770 31592 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:56.528106 31592 raft_consensus.cc:3060] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.533828 31592 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:56.536607 31592 leader_election.cc:290] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125)
I20260709 12:20:56.538131 30797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:56.538609 30868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c349e457659c45079da022887088cc7b" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:56.538872 30797 raft_consensus.cc:3060] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.539474 30868 raft_consensus.cc:3055] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:56.539927 30868 raft_consensus.cc:740] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:56.541711 30868 consensus_queue.cc:260] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:56.543550 30868 raft_consensus.cc:3060] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.553854 30797 raft_consensus.cc:2468] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:56.554263 30868 raft_consensus.cc:2468] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:56.555312 31490 leader_election.cc:304] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329; no voters: 
I20260709 12:20:56.556708 31734 raft_consensus.cc:2804] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:56.557372 31734 raft_consensus.cc:697] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:56.559217 31734 consensus_queue.cc:237] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:56.574244 30548 catalog_manager.cc:5697] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from 23ba075510134f3c8caba2370fff0d40 (127.11.11.132) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:56.580451 31837 raft_consensus.cc:993] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:56.581316 31749 raft_consensus.cc:1081] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:56.584328 31594 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:58998
I20260709 12:20:56.585176 31594 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:56.585508 31594 raft_consensus.cc:3060] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.592444 31594 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:56.594697 31594 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
I20260709 12:20:56.598457 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:56.599139 30717 raft_consensus.cc:3060] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.600498 30868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770f228988ee4c8dbe5b6d9eb6313cc4" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "23ba075510134f3c8caba2370fff0d40"
I20260709 12:20:56.601328 30868 raft_consensus.cc:3055] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:56.601797 30868 raft_consensus.cc:740] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 23ba075510134f3c8caba2370fff0d40, State: Running, Role: LEADER
I20260709 12:20:56.603457 30868 consensus_queue.cc:260] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:56.604620 30868 raft_consensus.cc:3060] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.606999 30717 raft_consensus.cc:2468] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:56.608347 31490 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:56.609200 31734 raft_consensus.cc:2804] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:56.609736 31734 raft_consensus.cc:697] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:56.610553 31734 consensus_queue.cc:237] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:56.612834 30868 raft_consensus.cc:2468] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:56.624347 30548 catalog_manager.cc:5697] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from 23ba075510134f3c8caba2370fff0d40 (127.11.11.132) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:56.691448 31778 raft_consensus.cc:993] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: : Instructing follower c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:56.692178 31778 raft_consensus.cc:1081] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Signalling peer c18174c596f9488f8be666b85062e329 to start an election
I20260709 12:20:56.695704 31592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523"
dest_uuid: "c18174c596f9488f8be666b85062e329"
 from {username='slave'} at 127.0.0.1:59002
I20260709 12:20:56.696537 31592 raft_consensus.cc:493] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:56.699769 31592 raft_consensus.cc:3060] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.708205 31592 raft_consensus.cc:515] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:56.710592 31592 leader_election.cc:290] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Requested vote from peers eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181), c178a628712047a88731a915fc93ed19 (127.11.11.131:39529)
I20260709 12:20:56.713089 30799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c178a628712047a88731a915fc93ed19"
I20260709 12:20:56.713898 30799 raft_consensus.cc:3055] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:56.713783 30717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5dd0a04062174fe1a633ff36362a7523" candidate_uuid: "c18174c596f9488f8be666b85062e329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb6634546938405dac9eabbadd34d2b1"
I20260709 12:20:56.714767 30799 raft_consensus.cc:740] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c178a628712047a88731a915fc93ed19, State: Running, Role: LEADER
I20260709 12:20:56.715240 30717 raft_consensus.cc:3060] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.715866 30799 consensus_queue.cc:260] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:56.717238 30799 raft_consensus.cc:3060] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.728509 30717 raft_consensus.cc:2468] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:56.729727 30799 raft_consensus.cc:2468] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18174c596f9488f8be666b85062e329 in term 2.
I20260709 12:20:56.730605 31490 leader_election.cc:304] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1; no voters: 
I20260709 12:20:56.731714 31734 raft_consensus.cc:2804] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:56.732321 31734 raft_consensus.cc:697] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Becoming Leader. State: Replica: c18174c596f9488f8be666b85062e329, State: Running, Role: LEADER
I20260709 12:20:56.733492 31734 consensus_queue.cc:237] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:56.751945 30548 catalog_manager.cc:5697] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 reported cstate change: term changed from 1 to 2, leader changed from c178a628712047a88731a915fc93ed19 (127.11.11.131) to c18174c596f9488f8be666b85062e329 (127.11.11.129). New cstate: current_term: 2 leader_uuid: "c18174c596f9488f8be666b85062e329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:56.996275 30868 raft_consensus.cc:1275] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:56.997989 31734 consensus_queue.cc:1048] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:57.031847 30799 raft_consensus.cc:1275] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:57.033492 31810 consensus_queue.cc:1048] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:57.070427 30717 raft_consensus.cc:1275] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:57.072523 31846 consensus_queue.cc:1048] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:57.085119 30868 raft_consensus.cc:1275] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:57.086833 31846 consensus_queue.cc:1048] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:57.241217 30720 raft_consensus.cc:1275] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:57.242969 31810 consensus_queue.cc:1048] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:57.262917 30797 raft_consensus.cc:1275] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Refusing update from remote peer c18174c596f9488f8be666b85062e329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:57.268430 31846 consensus_queue.cc:1048] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:57.531121 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:57.582036 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:57.583505 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:57.584158 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:57.627249 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:57.674196 11310 tablet_server.cc:179] TabletServer@127.11.11.129:41991 shutting down...
W20260709 12:20:57.704597 30763 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 290 similar messages]
W20260709 12:20:57.711699 30763 consensus_peers.cc:597] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:57.720928 30689 consensus_peers.cc:597] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:57.724233 30689 consensus_peers.cc:597] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.727108 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:57.728597 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.731043 11310 raft_consensus.cc:2243] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.731627 11310 raft_consensus.cc:2272] T c95a27bd5694416aa5dae696482e8ec4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:57.733989 30689 consensus_peers.cc:597] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:57.735668 30689 consensus_peers.cc:597] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.736179 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.737205 11310 raft_consensus.cc:2243] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.738346 11310 raft_consensus.cc:2272] T b2f67741397b40a0b491cddf4c7e7dea P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.742163 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.743441 11310 raft_consensus.cc:2243] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.744100 11310 raft_consensus.cc:2272] T fdae5560db3d45c180b6bed07a6b6197 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.748051 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.748934 11310 raft_consensus.cc:2243] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.749455 11310 raft_consensus.cc:2272] T 6e5f378411424826909f1f1f3aff4409 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.751847 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.752581 11310 raft_consensus.cc:2243] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.753019 11310 raft_consensus.cc:2272] T 6072fe2fb27a44558a09de34939e4421 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.756695 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.757903 11310 raft_consensus.cc:2243] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.759140 11310 raft_consensus.cc:2272] T e5be73f56e004d61ba8762d1afa0cc05 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.762957 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.764155 11310 raft_consensus.cc:2243] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.764735 11310 raft_consensus.cc:2272] T b9ed5ce7483340edbbac9dee8cca5f1d P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.767738 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.768507 11310 raft_consensus.cc:2243] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.769011 11310 raft_consensus.cc:2272] T 1f70173e529b4bdca23bc1fa8fbb2bde P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.771437 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.772068 11310 raft_consensus.cc:2243] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.772553 11310 raft_consensus.cc:2272] T 2d65269b37cc420382672e7ceecd4789 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.774943 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.776010 11310 raft_consensus.cc:2243] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.776599 11310 raft_consensus.cc:2272] T c1243becf9c9421bb2d5083f71c1e292 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.780176 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.782629 11310 raft_consensus.cc:2243] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.783331 11310 raft_consensus.cc:2272] T a5c48ae242924bcab94096bb803d25d4 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.787142 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.787963 11310 raft_consensus.cc:2243] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.788576 11310 raft_consensus.cc:2272] T 50776b4e5bc646a78173a6598eee54c2 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.793121 11310 tablet_replica.cc:333] stopping tablet replica
W20260709 12:20:57.793584 30763 consensus_peers.cc:597] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.794744 11310 raft_consensus.cc:2243] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.795459 11310 raft_consensus.cc:2272] T 1994c4ce566142fa922312f566387067 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.799444 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.800477 11310 raft_consensus.cc:2243] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.801115 11310 raft_consensus.cc:2272] T 462bf14c43184e17b0f483d0e185993f P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.804591 11310 tablet_replica.cc:333] stopping tablet replica
W20260709 12:20:57.804589 30838 consensus_peers.cc:597] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:57.805837 30763 consensus_peers.cc:597] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.806048 11310 raft_consensus.cc:2243] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.806825 11310 raft_consensus.cc:2272] T 83d271f20d484fc5bce5814a664ac807 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:57.808173 30689 consensus_peers.cc:597] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.809971 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.811087 11310 raft_consensus.cc:2243] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.812002 11310 raft_consensus.cc:2272] T 7a5651d4117243d69cfd21927e6ce1f9 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.814904 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.815768 11310 raft_consensus.cc:2243] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.816689 11310 raft_consensus.cc:2272] T 8bc4eca36172433d9134485920115dcc P c18174c596f9488f8be666b85062e329 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.820086 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.820892 11310 raft_consensus.cc:2243] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:57.820896 30689 consensus_peers.cc:597] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.821530 11310 raft_consensus.cc:2272] T 7f25f8eeb23d4d5aa15a052f827302ad P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.824553 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.825524 11310 raft_consensus.cc:2243] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.826129 11310 raft_consensus.cc:2272] T 56f37d27937c43f39c083e36508283fd P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.829452 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.830370 11310 raft_consensus.cc:2243] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.831625 11310 raft_consensus.cc:2272] T e9041cf65147424a8c1ef0e1d03ab95a P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.834806 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.835740 11310 raft_consensus.cc:2243] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.836401 11310 raft_consensus.cc:2272] T 61996ce5ca7a44f5aa688285da2e65ff P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.839756 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.861294 11310 raft_consensus.cc:2243] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.862033 11310 raft_consensus.cc:2272] T 4f3e88175c47421db220066ad29f61c4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.865319 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.866350 11310 raft_consensus.cc:2243] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.867029 11310 raft_consensus.cc:2272] T 3db7a62e2e80411ea0c98d2b5282b08f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.870378 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.871433 11310 raft_consensus.cc:2243] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.872530 11310 raft_consensus.cc:2272] T 8203be2a35f743e5ae45a1250189fb1f P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.875787 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.876711 11310 raft_consensus.cc:2243] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.877785 11310 raft_consensus.cc:2272] T 04fe0c41d3d440498251b09d3d6f5110 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.880861 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.881722 11310 raft_consensus.cc:2243] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.882368 11310 raft_consensus.cc:2272] T 88343a7c45e54babbd7fcae9b6376094 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.885576 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.886489 11310 raft_consensus.cc:2243] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.887537 11310 raft_consensus.cc:2272] T 5dd0a04062174fe1a633ff36362a7523 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.891119 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.891983 11310 raft_consensus.cc:2243] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.892556 11310 raft_consensus.cc:2272] T b4d076571a8e4367836518e952de6f11 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.896031 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.896898 11310 raft_consensus.cc:2243] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.897519 11310 raft_consensus.cc:2272] T 61d416be13d449318d3e6f13a14a5d4a P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.901049 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.901722 11310 raft_consensus.cc:2243] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.902634 11310 raft_consensus.cc:2272] T 770f228988ee4c8dbe5b6d9eb6313cc4 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.906239 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.906942 11310 raft_consensus.cc:2243] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.907449 11310 raft_consensus.cc:2272] T 35a132d43ba3447a916ccdfcc6e66c26 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.910369 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.911001 11310 raft_consensus.cc:2243] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.911598 11310 raft_consensus.cc:2272] T c274ea8c29774701bdc1708a1839fa88 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.914229 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.914858 11310 raft_consensus.cc:2243] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.915742 11310 raft_consensus.cc:2272] T e5899b18ce3a453ea77be2cdf20e543a P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.918334 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.919009 11310 raft_consensus.cc:2243] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.919482 11310 raft_consensus.cc:2272] T 66db71b232844f1794f53dba1cbd6198 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.922461 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.923308 11310 raft_consensus.cc:2243] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.923811 11310 raft_consensus.cc:2272] T 3be7907c61d7477da4fc1646f65b2a48 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.926959 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.927717 11310 raft_consensus.cc:2243] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.928283 11310 raft_consensus.cc:2272] T c1be5a3ea104480399760b60f846245b P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.931334 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.931960 11310 raft_consensus.cc:2243] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.932415 11310 raft_consensus.cc:2272] T 9873e94bae58472f8de439d5dc1835a6 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.934916 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.935559 11310 raft_consensus.cc:2243] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.936458 11310 raft_consensus.cc:2272] T c349e457659c45079da022887088cc7b P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.939848 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.940605 11310 raft_consensus.cc:2243] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:57.940701 30838 consensus_peers.cc:597] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.941220 11310 raft_consensus.cc:2272] T 9632ab6e7d6d4ac4bdf266d64c523d15 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:57.942569 30689 consensus_peers.cc:597] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:57.944504 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.945416 11310 raft_consensus.cc:2243] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.945981 11310 raft_consensus.cc:2272] T e95ca1a81eea4096a755f54943e95976 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.948798 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.949514 11310 raft_consensus.cc:2243] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.950565 11310 raft_consensus.cc:2272] T f77972ceb0c74ddca205e16542d12a4d P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.952839 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.953599 11310 raft_consensus.cc:2243] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.954200 11310 raft_consensus.cc:2272] T 773dfc99de304153af80953c364a815e P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.957782 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.958523 11310 raft_consensus.cc:2243] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:57.959025 11310 raft_consensus.cc:2272] T ef90e8d046fc47fcb7d122ffad59a1f3 P c18174c596f9488f8be666b85062e329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.961432 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.962218 11310 raft_consensus.cc:2243] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.963522 11310 raft_consensus.cc:2272] T 4a01bc7b51f24fb5b74aed9349c72e91 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:57.966838 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:20:57.967445 11310 raft_consensus.cc:2243] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:57.968636 11310 raft_consensus.cc:2272] T 2c9cd28a99d641698ab0d99e20bac569 P c18174c596f9488f8be666b85062e329 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:57.980749 30838 consensus_peers.cc:597] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:57.992642 30838 consensus_peers.cc:597] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:57.995075 30763 consensus_peers.cc:597] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.004868 30689 consensus_peers.cc:597] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.008798 30763 consensus_peers.cc:597] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.034782 30763 consensus_peers.cc:597] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.051772 30689 consensus_peers.cc:597] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.064637 11310 tablet_server.cc:196] TabletServer@127.11.11.129:41991 shutdown complete.
W20260709 12:20:58.070288 30689 consensus_peers.cc:597] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.076076 30689 consensus_peers.cc:597] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.085788 30838 consensus_peers.cc:597] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.091933 30838 consensus_peers.cc:597] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.096067 30689 consensus_peers.cc:597] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.106550 30838 consensus_peers.cc:597] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.123906 30838 consensus_peers.cc:597] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.124460 30838 consensus_peers.cc:597] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.150938 30763 consensus_peers.cc:597] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.188282 30763 consensus_peers.cc:597] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.213112 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
W20260709 12:20:58.216655 30838 consensus_peers.cc:597] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.270220 30764 consensus_peers.cc:597] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.304908 30764 consensus_peers.cc:597] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.307463 30689 consensus_peers.cc:597] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 -> Peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Couldn't send request to peer c18174c596f9488f8be666b85062e329. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.320690 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:58.322222 11310 tablet_replica.cc:333] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.323316 11310 raft_consensus.cc:2243] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.323940 11310 raft_consensus.cc:2272] T b29c912615fb4627ba5c12d4c0f193d3 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.325873 30764 consensus_peers.cc:597] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.327364 11310 tablet_replica.cc:333] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.328590 11310 raft_consensus.cc:2243] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.329586 11310 raft_consensus.cc:2272] T 773dfc99de304153af80953c364a815e P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.332937 11310 tablet_replica.cc:333] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.333866 11310 raft_consensus.cc:2243] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.334854 11310 raft_consensus.cc:2272] T 35a132d43ba3447a916ccdfcc6e66c26 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.337359 11310 tablet_replica.cc:333] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.338330 11310 raft_consensus.cc:2243] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.338888 11310 raft_consensus.cc:2272] T 1cea447841c446fea6a32ff304879db6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.341279 11310 tablet_replica.cc:333] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.342031 11310 raft_consensus.cc:2243] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.342557 11310 raft_consensus.cc:2272] T 770f228988ee4c8dbe5b6d9eb6313cc4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.345608 11310 tablet_replica.cc:333] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.346622 11310 raft_consensus.cc:2243] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.347254 11310 raft_consensus.cc:2272] T 56f37d27937c43f39c083e36508283fd P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.350709 11310 tablet_replica.cc:333] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
W20260709 12:20:58.350975 30764 consensus_peers.cc:597] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.351684 11310 raft_consensus.cc:2243] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.352680 11310 raft_consensus.cc:2272] T 7f25f8eeb23d4d5aa15a052f827302ad P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.355361 11310 tablet_replica.cc:333] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.356019 11310 raft_consensus.cc:2243] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.356549 11310 raft_consensus.cc:2272] T e9041cf65147424a8c1ef0e1d03ab95a P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.359691 11310 tablet_replica.cc:333] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.360657 11310 raft_consensus.cc:2243] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.361234 11310 raft_consensus.cc:2272] T 58bf19ba51b540ff8dcb14a3af23b8e3 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.364331 11310 tablet_replica.cc:333] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.365134 11310 raft_consensus.cc:2243] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.366180 11310 raft_consensus.cc:2272] T 4f3e88175c47421db220066ad29f61c4 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.368731 11310 tablet_replica.cc:333] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.369421 11310 raft_consensus.cc:2243] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.369956 11310 raft_consensus.cc:2272] T 61996ce5ca7a44f5aa688285da2e65ff P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.372253 11310 tablet_replica.cc:333] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.372913 11310 raft_consensus.cc:2243] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.373932 11310 raft_consensus.cc:2272] T 1994c4ce566142fa922312f566387067 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.376565 11310 tablet_replica.cc:333] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.377492 11310 raft_consensus.cc:2243] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.378144 11310 raft_consensus.cc:2272] T c1243becf9c9421bb2d5083f71c1e292 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.379556 30764 consensus_peers.cc:597] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.381085 11310 tablet_replica.cc:333] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.382016 11310 raft_consensus.cc:2243] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.382681 11310 raft_consensus.cc:2272] T 7470dd49ce524f4aaf3faf7739fe5bd7 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.386456 11310 tablet_replica.cc:333] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.387392 11310 raft_consensus.cc:2243] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.388805 11310 raft_consensus.cc:2272] T 87c766be78f1467795131bc36e7766e9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.392565 11310 tablet_replica.cc:333] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.393455 11310 raft_consensus.cc:2243] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.394119 11310 raft_consensus.cc:2272] T 8bc4eca36172433d9134485920115dcc P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.397367 11310 tablet_replica.cc:333] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.398280 11310 raft_consensus.cc:2243] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.398887 11310 raft_consensus.cc:2272] T 9632ab6e7d6d4ac4bdf266d64c523d15 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.400676 30839 consensus_peers.cc:597] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.402228 11310 tablet_replica.cc:333] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.403350 11310 raft_consensus.cc:2243] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.404073 11310 raft_consensus.cc:2272] T e95ca1a81eea4096a755f54943e95976 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.407719 11310 tablet_replica.cc:333] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.408524 11310 raft_consensus.cc:2243] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.409540 11310 raft_consensus.cc:2272] T a5c48ae242924bcab94096bb803d25d4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.412930 11310 tablet_replica.cc:333] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.413887 11310 raft_consensus.cc:2243] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.414500 11310 raft_consensus.cc:2272] T 5dd0a04062174fe1a633ff36362a7523 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.417454 11310 tablet_replica.cc:333] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.418159 11310 raft_consensus.cc:2243] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.419287 11310 raft_consensus.cc:2272] T 88343a7c45e54babbd7fcae9b6376094 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.422219 11310 tablet_replica.cc:333] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.422984 11310 raft_consensus.cc:2243] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.424113 11310 raft_consensus.cc:2272] T b9ed5ce7483340edbbac9dee8cca5f1d P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.427083 11310 tablet_replica.cc:333] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.427779 11310 raft_consensus.cc:2243] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.428313 11310 raft_consensus.cc:2272] T 9873e94bae58472f8de439d5dc1835a6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.430799 11310 tablet_replica.cc:333] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.431496 11310 raft_consensus.cc:2243] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.432219 11310 raft_consensus.cc:2272] T 6e5f378411424826909f1f1f3aff4409 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.435572 11310 tablet_replica.cc:333] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.436472 11310 raft_consensus.cc:2243] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:20:58.436810 30839 consensus_peers.cc:597] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.437088 11310 raft_consensus.cc:2272] T 14747c6d552943dbac8ab19f68816ea9 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.440320 11310 tablet_replica.cc:333] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.441286 11310 raft_consensus.cc:2243] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.441964 11310 raft_consensus.cc:2272] T 4ac6fd1105644a1b820d76679057cbe4 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.445333 11310 tablet_replica.cc:333] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.446300 11310 raft_consensus.cc:2243] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.447628 11310 raft_consensus.cc:2272] T 462bf14c43184e17b0f483d0e185993f P eb6634546938405dac9eabbadd34d2b1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.450968 11310 tablet_replica.cc:333] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.452162 11310 raft_consensus.cc:2243] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.452884 11310 raft_consensus.cc:2272] T 7a5651d4117243d69cfd21927e6ce1f9 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.456843 11310 tablet_replica.cc:333] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.457842 11310 raft_consensus.cc:2243] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.458472 11310 raft_consensus.cc:2272] T e614ae4fde394b488924f185e2808b18 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.461833 11310 tablet_replica.cc:333] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.462679 11310 raft_consensus.cc:2243] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.463323 11310 raft_consensus.cc:2272] T 2aafa16c29f04448945f5db4aa191223 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.463447 30839 consensus_peers.cc:597] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.470063 30764 consensus_peers.cc:597] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.472121 11310 tablet_replica.cc:333] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.473194 11310 raft_consensus.cc:2243] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.473937 11310 raft_consensus.cc:2272] T b4d076571a8e4367836518e952de6f11 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.478092 11310 tablet_replica.cc:333] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.479082 11310 raft_consensus.cc:2243] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.479638 11310 raft_consensus.cc:2272] T ecdffc08164a4124ad64c3b376d308d5 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.482524 11310 tablet_replica.cc:333] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.483151 11310 raft_consensus.cc:2243] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.484265 11310 raft_consensus.cc:2272] T c274ea8c29774701bdc1708a1839fa88 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.487089 11310 tablet_replica.cc:333] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.487936 11310 raft_consensus.cc:2243] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.488556 11310 raft_consensus.cc:2272] T 11bb43acd4e84e5aa5c2731a70542e07 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.491882 11310 tablet_replica.cc:333] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.492740 11310 raft_consensus.cc:2243] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.494019 11310 raft_consensus.cc:2272] T 61d416be13d449318d3e6f13a14a5d4a P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.496675 30764 consensus_peers.cc:597] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.497418 11310 tablet_replica.cc:333] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.498365 11310 raft_consensus.cc:2243] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.498984 11310 raft_consensus.cc:2272] T 6c8dd412d0624072a283e7390f9482b6 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.502821 11310 tablet_replica.cc:333] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
W20260709 12:20:58.503005 30839 consensus_peers.cc:597] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.503724 11310 raft_consensus.cc:2243] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.505306 11310 raft_consensus.cc:2272] T 6072fe2fb27a44558a09de34939e4421 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.507872 30764 consensus_peers.cc:597] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.509033 11310 tablet_replica.cc:333] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.510178 11310 raft_consensus.cc:2243] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.511255 11310 raft_consensus.cc:2272] T db650f2a9f164bcd9c32d847363b1745 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.513548 30764 consensus_peers.cc:597] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.514706 11310 tablet_replica.cc:333] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.515491 11310 raft_consensus.cc:2243] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.515967 11310 raft_consensus.cc:2272] T e5be73f56e004d61ba8762d1afa0cc05 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.518466 11310 tablet_replica.cc:333] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.519179 11310 raft_consensus.cc:2243] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.519636 11310 raft_consensus.cc:2272] T e0e79bd22cff4979ba995f0c04f48806 P eb6634546938405dac9eabbadd34d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.522161 11310 tablet_replica.cc:333] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.522791 11310 raft_consensus.cc:2243] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.523741 11310 raft_consensus.cc:2272] T c1be5a3ea104480399760b60f846245b P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.526381 11310 tablet_replica.cc:333] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.527029 11310 raft_consensus.cc:2243] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.527606 11310 raft_consensus.cc:2272] T 04fe0c41d3d440498251b09d3d6f5110 P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.530754 11310 tablet_replica.cc:333] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.531549 11310 raft_consensus.cc:2243] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:20:58.532311 30764 consensus_peers.cc:597] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.532644 11310 raft_consensus.cc:2272] T 3db7a62e2e80411ea0c98d2b5282b08f P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.535430 11310 tablet_replica.cc:333] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1: stopping tablet replica
I20260709 12:20:58.536082 11310 raft_consensus.cc:2243] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.536594 11310 raft_consensus.cc:2272] T 1f70173e529b4bdca23bc1fa8fbb2bde P eb6634546938405dac9eabbadd34d2b1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.549214 30839 consensus_peers.cc:597] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.563177 30839 consensus_peers.cc:597] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.577282 30839 consensus_peers.cc:597] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.578992 30839 consensus_peers.cc:597] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.619421 30764 consensus_peers.cc:597] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.625778 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
W20260709 12:20:58.629851 30839 consensus_peers.cc:597] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.702904 30764 consensus_peers.cc:597] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.731611 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
W20260709 12:20:58.744668 30764 consensus_peers.cc:597] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 -> Peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Couldn't send request to peer eb6634546938405dac9eabbadd34d2b1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.773983 31775 raft_consensus.cc:493] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:58.774741 31775 raft_consensus.cc:515] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } } peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } }
I20260709 12:20:58.777139 31775 leader_election.cc:290] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:58.783622 30839 consensus_peers.cc:597] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.790900 30764 leader_election.cc:336] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111)
W20260709 12:20:58.796792 30839 consensus_peers.cc:597] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:58.803375 30763 leader_election.cc:336] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:58.804641 30763 leader_election.cc:304] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:58.806437 31778 raft_consensus.cc:2749] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.813122 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:58.814522 11310 tablet_replica.cc:333] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.815582 11310 raft_consensus.cc:2243] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.816265 11310 raft_consensus.cc:2272] T b29c912615fb4627ba5c12d4c0f193d3 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.820226 11310 tablet_replica.cc:333] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.821262 11310 raft_consensus.cc:2243] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.822530 11310 raft_consensus.cc:2272] T 773dfc99de304153af80953c364a815e P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.824369 30839 consensus_peers.cc:597] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.825847 11310 tablet_replica.cc:333] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.826788 11310 raft_consensus.cc:2243] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.827334 11310 raft_consensus.cc:2272] T 4a01bc7b51f24fb5b74aed9349c72e91 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.830452 11310 tablet_replica.cc:333] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.831284 11310 raft_consensus.cc:2243] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.831890 11310 raft_consensus.cc:2272] T 2c9cd28a99d641698ab0d99e20bac569 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.834766 11310 tablet_replica.cc:333] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.835629 11310 raft_consensus.cc:2243] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.836817 11310 raft_consensus.cc:2272] T 1cea447841c446fea6a32ff304879db6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.840026 11310 tablet_replica.cc:333] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.842033 11310 raft_consensus.cc:2243] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.845342 11310 raft_consensus.cc:2272] T 58bf19ba51b540ff8dcb14a3af23b8e3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.848664 11310 tablet_replica.cc:333] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.849566 11310 raft_consensus.cc:2243] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.850262 11310 raft_consensus.cc:2272] T fdae5560db3d45c180b6bed07a6b6197 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.853188 11310 tablet_replica.cc:333] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.854269 11310 raft_consensus.cc:2243] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.855512 11310 raft_consensus.cc:2272] T 56f37d27937c43f39c083e36508283fd P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.859795 11310 tablet_replica.cc:333] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.860819 11310 raft_consensus.cc:2243] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.861778 11310 raft_consensus.cc:2272] T e9041cf65147424a8c1ef0e1d03ab95a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.862466 30839 consensus_peers.cc:597] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.864869 11310 tablet_replica.cc:333] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.866186 11310 raft_consensus.cc:2243] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.867266 11310 raft_consensus.cc:2272] T 6c8dd412d0624072a283e7390f9482b6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.870399 11310 tablet_replica.cc:333] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.871081 11310 raft_consensus.cc:2243] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.871609 11310 raft_consensus.cc:2272] T 1994c4ce566142fa922312f566387067 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.873998 11310 tablet_replica.cc:333] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.874708 11310 raft_consensus.cc:2243] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.875242 11310 raft_consensus.cc:2272] T 50776b4e5bc646a78173a6598eee54c2 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.877511 11310 tablet_replica.cc:333] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.878295 11310 raft_consensus.cc:2243] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.878774 11310 raft_consensus.cc:2272] T 2d65269b37cc420382672e7ceecd4789 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.881546 11310 tablet_replica.cc:333] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.882432 11310 raft_consensus.cc:2243] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.883083 11310 raft_consensus.cc:2272] T 87c766be78f1467795131bc36e7766e9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.886171 11310 tablet_replica.cc:333] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.886971 11310 raft_consensus.cc:2243] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.887506 11310 raft_consensus.cc:2272] T 5dd0a04062174fe1a633ff36362a7523 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.890326 11310 tablet_replica.cc:333] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19: stopping tablet replica
W20260709 12:20:58.891039 30839 consensus_peers.cc:597] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.891167 11310 raft_consensus.cc:2243] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.891888 11310 raft_consensus.cc:2272] T 4ac6fd1105644a1b820d76679057cbe4 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.894146 11310 tablet_replica.cc:333] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.894733 11310 raft_consensus.cc:2243] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.895139 11310 raft_consensus.cc:2272] T 14747c6d552943dbac8ab19f68816ea9 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.897107 11310 tablet_replica.cc:333] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.897799 11310 raft_consensus.cc:2243] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.898614 11310 raft_consensus.cc:2272] T 6e5f378411424826909f1f1f3aff4409 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.900652 11310 tablet_replica.cc:333] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.901209 11310 raft_consensus.cc:2243] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.902146 11310 raft_consensus.cc:2272] T 11bb43acd4e84e5aa5c2731a70542e07 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.904213 11310 tablet_replica.cc:333] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.904763 11310 raft_consensus.cc:2243] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.905165 11310 raft_consensus.cc:2272] T e5899b18ce3a453ea77be2cdf20e543a P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.907282 11310 tablet_replica.cc:333] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.907817 11310 raft_consensus.cc:2243] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.908211 11310 raft_consensus.cc:2272] T c274ea8c29774701bdc1708a1839fa88 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.910247 11310 tablet_replica.cc:333] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.910780 11310 raft_consensus.cc:2243] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.911604 11310 raft_consensus.cc:2272] T 66db71b232844f1794f53dba1cbd6198 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.913420 31858 raft_consensus.cc:493] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:58.914149 11310 tablet_replica.cc:333] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.914286 31858 raft_consensus.cc:515] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:58.915241 11310 raft_consensus.cc:2243] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.916543 11310 raft_consensus.cc:2272] T e0e79bd22cff4979ba995f0c04f48806 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.919802 11310 tablet_replica.cc:333] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.920680 11310 raft_consensus.cc:2243] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.921007 31858 leader_election.cc:290] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:58.921317 11310 raft_consensus.cc:2272] T 3be7907c61d7477da4fc1646f65b2a48 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.924748 11310 tablet_replica.cc:333] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19: stopping tablet replica
W20260709 12:20:58.924868 30838 leader_election.cc:336] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:58.925616 11310 raft_consensus.cc:2243] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.925920 31870 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:58.926386 11310 raft_consensus.cc:2272] T c1be5a3ea104480399760b60f846245b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.926553 31870 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:58.929024 31870 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 23ba075510134f3c8caba2370fff0d40 (127.11.11.132:36125), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:58.929417 11310 tablet_replica.cc:333] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.930485 11310 raft_consensus.cc:2243] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.931216 11310 raft_consensus.cc:2272] T 462bf14c43184e17b0f483d0e185993f P c178a628712047a88731a915fc93ed19 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.931689 30839 leader_election.cc:336] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:58.932166 30839 leader_election.cc:304] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:58.933458 31749 raft_consensus.cc:2749] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.934499 11310 tablet_replica.cc:333] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.935576 11310 raft_consensus.cc:2243] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.936179 11310 raft_consensus.cc:2272] T e614ae4fde394b488924f185e2808b18 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.939765 11310 tablet_replica.cc:333] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19: stopping tablet replica
W20260709 12:20:58.939800 30763 leader_election.cc:336] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:58.940761 11310 raft_consensus.cc:2243] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.941495 11310 raft_consensus.cc:2272] T 7a5651d4117243d69cfd21927e6ce1f9 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.944181 30872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f77972ceb0c74ddca205e16542d12a4d" candidate_uuid: "c178a628712047a88731a915fc93ed19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "23ba075510134f3c8caba2370fff0d40" is_pre_election: true
I20260709 12:20:58.944734 11310 tablet_replica.cc:333] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.945844 11310 raft_consensus.cc:2243] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.946475 11310 raft_consensus.cc:2272] T 83d271f20d484fc5bce5814a664ac807 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.947072 30763 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c178a628712047a88731a915fc93ed19; no voters: 23ba075510134f3c8caba2370fff0d40, c18174c596f9488f8be666b85062e329
I20260709 12:20:58.948103 31870 raft_consensus.cc:2749] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.949430 11310 tablet_replica.cc:333] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.950620 11310 raft_consensus.cc:2243] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.951243 11310 raft_consensus.cc:2272] T 8203be2a35f743e5ae45a1250189fb1f P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.954818 11310 tablet_replica.cc:333] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19: stopping tablet replica
W20260709 12:20:58.956007 30839 consensus_peers.cc:597] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:58.956070 11310 raft_consensus.cc:2243] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.957563 11310 raft_consensus.cc:2272] T 04fe0c41d3d440498251b09d3d6f5110 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.961412 11310 tablet_replica.cc:333] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.962352 11310 raft_consensus.cc:2243] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.963016 11310 raft_consensus.cc:2272] T b2f67741397b40a0b491cddf4c7e7dea P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.965724 31858 raft_consensus.cc:493] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:58.966382 11310 tablet_replica.cc:333] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.966470 31858 raft_consensus.cc:515] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:58.967602 11310 raft_consensus.cc:2243] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.968290 11310 raft_consensus.cc:2272] T 2aafa16c29f04448945f5db4aa191223 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.971163 31858 leader_election.cc:290] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:58.971553 11310 tablet_replica.cc:333] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.972693 11310 raft_consensus.cc:2243] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.973917 11310 raft_consensus.cc:2272] T ecdffc08164a4124ad64c3b376d308d5 P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.977173 11310 tablet_replica.cc:333] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.978101 11310 raft_consensus.cc:2243] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.979485 11310 raft_consensus.cc:2272] T b4d076571a8e4367836518e952de6f11 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.982887 11310 tablet_replica.cc:333] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.983845 11310 raft_consensus.cc:2243] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.985087 11310 raft_consensus.cc:2272] T c95a27bd5694416aa5dae696482e8ec4 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.988734 11310 tablet_replica.cc:333] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.989666 11310 raft_consensus.cc:2243] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:58.990924 11310 raft_consensus.cc:2272] T 7470dd49ce524f4aaf3faf7739fe5bd7 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:58.993251 30838 leader_election.cc:336] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:58.994390 11310 tablet_replica.cc:333] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:58.995359 11310 raft_consensus.cc:2243] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:58.995944 11310 raft_consensus.cc:2272] T 61d416be13d449318d3e6f13a14a5d4a P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:58.999334 11310 tablet_replica.cc:333] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:59.000183 11310 raft_consensus.cc:2243] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.000797 11310 raft_consensus.cc:2272] T db650f2a9f164bcd9c32d847363b1745 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.004117 11310 tablet_replica.cc:333] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:59.005123 11310 raft_consensus.cc:2243] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.006378 11310 raft_consensus.cc:2272] T 1f70173e529b4bdca23bc1fa8fbb2bde P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.010069 11310 tablet_replica.cc:333] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:59.010932 11310 raft_consensus.cc:2243] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.011525 11310 raft_consensus.cc:2272] T c349e457659c45079da022887088cc7b P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:59.012589 30839 leader_election.cc:336] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:59.013193 30839 leader_election.cc:304] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:59.013967 31858 raft_consensus.cc:2749] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.014694 11310 tablet_replica.cc:333] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:59.015574 11310 raft_consensus.cc:2243] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.016803 11310 raft_consensus.cc:2272] T 9873e94bae58472f8de439d5dc1835a6 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:59.017055 30839 consensus_peers.cc:597] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:59.020038 11310 tablet_replica.cc:333] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:59.021396 11310 raft_consensus.cc:2243] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.022127 11310 raft_consensus.cc:2272] T f77972ceb0c74ddca205e16542d12a4d P c178a628712047a88731a915fc93ed19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.025085 11310 tablet_replica.cc:333] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19: stopping tablet replica
I20260709 12:20:59.025825 11310 raft_consensus.cc:2243] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.026384 11310 raft_consensus.cc:2272] T ef90e8d046fc47fcb7d122ffad59a1f3 P c178a628712047a88731a915fc93ed19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.034292 31751 raft_consensus.cc:493] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.034946 31751 raft_consensus.cc:515] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:59.038058 31751 leader_election.cc:290] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:59.046072 30838 leader_election.cc:336] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:59.046948 30839 leader_election.cc:336] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:59.047396 30839 leader_election.cc:304] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:59.048283 31751 raft_consensus.cc:2749] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:20:59.059399 30839 consensus_peers.cc:597] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:59.072981 30839 consensus_peers.cc:597] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:59.125106 31751 raft_consensus.cc:493] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.125739 31751 raft_consensus.cc:515] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:59.128183 31751 leader_election.cc:290] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:59.130916 30838 leader_election.cc:336] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:59.133888 30839 leader_election.cc:336] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111)
I20260709 12:20:59.134492 30839 leader_election.cc:304] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:59.135499 31871 raft_consensus.cc:2749] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:20:59.137236 31871 raft_consensus.cc:493] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.137622 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:20:59.137840 31871 raft_consensus.cc:515] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:59.137980 31858 raft_consensus.cc:493] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.138883 31858 raft_consensus.cc:515] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:59.140492 31871 leader_election.cc:290] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:59.140862 31858 leader_election.cc:290] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:59.146113 30839 leader_election.cc:336] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Network error: Client connection negotiation failed: client connection to 127.11.11.131:39529: connect: Connection refused (error 111)
W20260709 12:20:59.146836 30839 leader_election.cc:336] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111)
W20260709 12:20:59.148458 30838 leader_election.cc:336] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:59.148945 30838 leader_election.cc:304] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:59.149807 31858 raft_consensus.cc:2749] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:20:59.149979 30838 leader_election.cc:336] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:59.150575 30838 leader_election.cc:304] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:59.151285 31858 raft_consensus.cc:2749] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.168376 31871 raft_consensus.cc:493] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.169081 31871 raft_consensus.cc:515] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:59.171881 31871 leader_election.cc:290] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:59.172204 30839 consensus_peers.cc:597] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:39529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:59.177282 30839 consensus_peers.cc:597] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 -> Peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Couldn't send request to peer c178a628712047a88731a915fc93ed19. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.131:39529: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:59.180110 30839 leader_election.cc:336] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111)
W20260709 12:20:59.180686 30838 leader_election.cc:336] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:59.181185 30838 leader_election.cc:304] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:59.182052 31871 raft_consensus.cc:2749] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.250296 11310 tablet_server.cc:179] TabletServer@127.11.11.132:0 shutting down...
I20260709 12:20:59.264693 31858 raft_consensus.cc:493] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eb6634546938405dac9eabbadd34d2b1)
I20260709 12:20:59.265311 31858 raft_consensus.cc:515] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } } peers { permanent_uuid: "eb6634546938405dac9eabbadd34d2b1" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 37181 } }
I20260709 12:20:59.269456 31858 leader_election.cc:290] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c18174c596f9488f8be666b85062e329 (127.11.11.129:41991), eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181)
W20260709 12:20:59.277019 30838 leader_election.cc:336] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:59.278303 30839 leader_election.cc:336] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eb6634546938405dac9eabbadd34d2b1 (127.11.11.130:37181): Network error: Client connection negotiation failed: client connection to 127.11.11.130:37181: connect: Connection refused (error 111)
I20260709 12:20:59.278932 30839 leader_election.cc:304] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c18174c596f9488f8be666b85062e329, eb6634546938405dac9eabbadd34d2b1
I20260709 12:20:59.280066 31858 raft_consensus.cc:2749] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.301294 31858 raft_consensus.cc:493] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.301980 31858 raft_consensus.cc:515] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:59.309643 31858 leader_election.cc:290] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
W20260709 12:20:59.312985 30838 leader_election.cc:336] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
W20260709 12:20:59.315423 30839 leader_election.cc:336] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Network error: Client connection negotiation failed: client connection to 127.11.11.131:39529: connect: Connection refused (error 111)
I20260709 12:20:59.315872 30839 leader_election.cc:304] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:59.316617 31858 raft_consensus.cc:2749] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.324175 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:59.325301 11310 tablet_replica.cc:333] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.326256 11310 raft_consensus.cc:2243] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.327113 11310 raft_consensus.cc:2272] T b29c912615fb4627ba5c12d4c0f193d3 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.329421 11310 tablet_replica.cc:333] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.330138 11310 raft_consensus.cc:2243] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.330665 11310 raft_consensus.cc:2272] T 4a01bc7b51f24fb5b74aed9349c72e91 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.333586 11310 tablet_replica.cc:333] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.334452 11310 raft_consensus.cc:2243] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.335057 11310 raft_consensus.cc:2272] T 2c9cd28a99d641698ab0d99e20bac569 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.337755 11310 tablet_replica.cc:333] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.338688 11310 raft_consensus.cc:2243] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.339363 11310 raft_consensus.cc:2272] T 7f25f8eeb23d4d5aa15a052f827302ad P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.342307 11310 tablet_replica.cc:333] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.342921 11310 raft_consensus.cc:2243] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.343331 11310 raft_consensus.cc:2272] T 58bf19ba51b540ff8dcb14a3af23b8e3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.345393 11310 tablet_replica.cc:333] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.346083 11310 raft_consensus.cc:2243] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.346864 11310 raft_consensus.cc:2272] T fdae5560db3d45c180b6bed07a6b6197 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.349126 11310 tablet_replica.cc:333] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.349778 11310 raft_consensus.cc:2243] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.350287 11310 raft_consensus.cc:2272] T c349e457659c45079da022887088cc7b P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.352391 11310 tablet_replica.cc:333] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.352921 11310 raft_consensus.cc:2243] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.353312 11310 raft_consensus.cc:2272] T 87c766be78f1467795131bc36e7766e9 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.355466 11310 tablet_replica.cc:333] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.356033 11310 raft_consensus.cc:2243] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.356436 11310 raft_consensus.cc:2272] T 35a132d43ba3447a916ccdfcc6e66c26 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.358645 11310 tablet_replica.cc:333] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.359179 11310 raft_consensus.cc:2243] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.359576 11310 raft_consensus.cc:2272] T 1cea447841c446fea6a32ff304879db6 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.361553 11310 tablet_replica.cc:333] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.362211 11310 raft_consensus.cc:2243] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.362627 11310 raft_consensus.cc:2272] T 770f228988ee4c8dbe5b6d9eb6313cc4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.364717 11310 tablet_replica.cc:333] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.365334 11310 raft_consensus.cc:2243] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.366269 11310 raft_consensus.cc:2272] T 50776b4e5bc646a78173a6598eee54c2 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.368619 11310 tablet_replica.cc:333] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.368891 31858 raft_consensus.cc:493] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c18174c596f9488f8be666b85062e329)
I20260709 12:20:59.369292 11310 raft_consensus.cc:2243] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.370019 11310 raft_consensus.cc:2272] T 4f3e88175c47421db220066ad29f61c4 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.369587 31858 raft_consensus.cc:515] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c178a628712047a88731a915fc93ed19" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 39529 } } peers { permanent_uuid: "23ba075510134f3c8caba2370fff0d40" member_type: VOTER last_known_addr { host: "127.11.11.132" port: 36125 } } peers { permanent_uuid: "c18174c596f9488f8be666b85062e329" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 41991 } }
I20260709 12:20:59.372439 31858 leader_election.cc:290] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c178a628712047a88731a915fc93ed19 (127.11.11.131:39529), c18174c596f9488f8be666b85062e329 (127.11.11.129:41991)
I20260709 12:20:59.372821 11310 tablet_replica.cc:333] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.373848 11310 raft_consensus.cc:2243] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.374905 11310 raft_consensus.cc:2272] T 61996ce5ca7a44f5aa688285da2e65ff P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:59.377166 30839 leader_election.cc:336] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c178a628712047a88731a915fc93ed19 (127.11.11.131:39529): Network error: Client connection negotiation failed: client connection to 127.11.11.131:39529: connect: Connection refused (error 111)
I20260709 12:20:59.377985 11310 tablet_replica.cc:333] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
W20260709 12:20:59.377985 30838 leader_election.cc:336] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c18174c596f9488f8be666b85062e329 (127.11.11.129:41991): Network error: Client connection negotiation failed: client connection to 127.11.11.129:41991: connect: Connection refused (error 111)
I20260709 12:20:59.378710 30838 leader_election.cc:304] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23ba075510134f3c8caba2370fff0d40; no voters: c178a628712047a88731a915fc93ed19, c18174c596f9488f8be666b85062e329
I20260709 12:20:59.378885 11310 raft_consensus.cc:2243] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.379942 11310 raft_consensus.cc:2272] T c1243becf9c9421bb2d5083f71c1e292 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.379844 31858 raft_consensus.cc:2749] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.382591 11310 tablet_replica.cc:333] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.383425 11310 raft_consensus.cc:2243] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.383957 11310 raft_consensus.cc:2272] T e5be73f56e004d61ba8762d1afa0cc05 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.386757 11310 tablet_replica.cc:333] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.387404 11310 raft_consensus.cc:2243] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.388443 11310 raft_consensus.cc:2272] T 2aafa16c29f04448945f5db4aa191223 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.390640 11310 tablet_replica.cc:333] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.391216 11310 raft_consensus.cc:2243] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.391649 11310 raft_consensus.cc:2272] T b2f67741397b40a0b491cddf4c7e7dea P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.393846 11310 tablet_replica.cc:333] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.394390 11310 raft_consensus.cc:2243] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.395151 11310 raft_consensus.cc:2272] T e614ae4fde394b488924f185e2808b18 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.397089 11310 tablet_replica.cc:333] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.397687 11310 raft_consensus.cc:2243] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.398679 11310 raft_consensus.cc:2272] T 83d271f20d484fc5bce5814a664ac807 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.400988 11310 tablet_replica.cc:333] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.401520 11310 raft_consensus.cc:2243] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.402072 11310 raft_consensus.cc:2272] T c95a27bd5694416aa5dae696482e8ec4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.404198 11310 tablet_replica.cc:333] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.404749 11310 raft_consensus.cc:2243] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.405150 11310 raft_consensus.cc:2272] T 7470dd49ce524f4aaf3faf7739fe5bd7 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.407344 11310 tablet_replica.cc:333] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.407893 11310 raft_consensus.cc:2243] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.408524 11310 raft_consensus.cc:2272] T a5c48ae242924bcab94096bb803d25d4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.410830 11310 tablet_replica.cc:333] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.411404 11310 raft_consensus.cc:2243] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.411824 11310 raft_consensus.cc:2272] T 88343a7c45e54babbd7fcae9b6376094 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.413909 11310 tablet_replica.cc:333] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.414448 11310 raft_consensus.cc:2243] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.415254 11310 raft_consensus.cc:2272] T 2d65269b37cc420382672e7ceecd4789 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.417193 11310 tablet_replica.cc:333] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.417760 11310 raft_consensus.cc:2243] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.418176 11310 raft_consensus.cc:2272] T ecdffc08164a4124ad64c3b376d308d5 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.420261 11310 tablet_replica.cc:333] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.420811 11310 raft_consensus.cc:2243] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.421684 11310 raft_consensus.cc:2272] T 4ac6fd1105644a1b820d76679057cbe4 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.424201 11310 tablet_replica.cc:333] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.424917 11310 raft_consensus.cc:2243] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.425959 11310 raft_consensus.cc:2272] T 14747c6d552943dbac8ab19f68816ea9 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.428927 11310 tablet_replica.cc:333] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.429827 11310 raft_consensus.cc:2243] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.430387 11310 raft_consensus.cc:2272] T 8203be2a35f743e5ae45a1250189fb1f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.432415 11310 tablet_replica.cc:333] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.432934 11310 raft_consensus.cc:2243] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.433359 11310 raft_consensus.cc:2272] T 3db7a62e2e80411ea0c98d2b5282b08f P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.435560 11310 tablet_replica.cc:333] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.436086 11310 raft_consensus.cc:2243] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.436486 11310 raft_consensus.cc:2272] T b9ed5ce7483340edbbac9dee8cca5f1d P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.438648 11310 tablet_replica.cc:333] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.439172 11310 raft_consensus.cc:2243] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.439601 11310 raft_consensus.cc:2272] T f77972ceb0c74ddca205e16542d12a4d P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.441776 11310 tablet_replica.cc:333] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.442291 11310 raft_consensus.cc:2243] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.442723 11310 raft_consensus.cc:2272] T 6c8dd412d0624072a283e7390f9482b6 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.444854 11310 tablet_replica.cc:333] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.445403 11310 raft_consensus.cc:2243] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.445909 11310 raft_consensus.cc:2272] T 6072fe2fb27a44558a09de34939e4421 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.447913 11310 tablet_replica.cc:333] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.448410 11310 raft_consensus.cc:2243] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.448808 11310 raft_consensus.cc:2272] T e0e79bd22cff4979ba995f0c04f48806 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.451031 11310 tablet_replica.cc:333] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.451666 11310 raft_consensus.cc:2243] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.452541 11310 raft_consensus.cc:2272] T 3be7907c61d7477da4fc1646f65b2a48 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.454627 11310 tablet_replica.cc:333] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.455159 11310 raft_consensus.cc:2243] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.455585 11310 raft_consensus.cc:2272] T db650f2a9f164bcd9c32d847363b1745 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.457790 11310 tablet_replica.cc:333] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.458277 11310 raft_consensus.cc:2243] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.458698 11310 raft_consensus.cc:2272] T 8bc4eca36172433d9134485920115dcc P 23ba075510134f3c8caba2370fff0d40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.460630 11310 tablet_replica.cc:333] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.461110 11310 raft_consensus.cc:2243] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.461511 11310 raft_consensus.cc:2272] T e5899b18ce3a453ea77be2cdf20e543a P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.463496 11310 tablet_replica.cc:333] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.463974 11310 raft_consensus.cc:2243] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.464474 11310 raft_consensus.cc:2272] T 66db71b232844f1794f53dba1cbd6198 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.466697 11310 tablet_replica.cc:333] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.467221 11310 raft_consensus.cc:2243] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:59.467672 11310 raft_consensus.cc:2272] T 11bb43acd4e84e5aa5c2731a70542e07 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.469734 11310 tablet_replica.cc:333] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.470266 11310 raft_consensus.cc:2243] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.471189 11310 raft_consensus.cc:2272] T 9632ab6e7d6d4ac4bdf266d64c523d15 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.473513 11310 tablet_replica.cc:333] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.474099 11310 raft_consensus.cc:2243] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.474902 11310 raft_consensus.cc:2272] T e95ca1a81eea4096a755f54943e95976 P 23ba075510134f3c8caba2370fff0d40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.477468 11310 tablet_replica.cc:333] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40: stopping tablet replica
I20260709 12:20:59.478106 11310 raft_consensus.cc:2243] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.479053 11310 raft_consensus.cc:2272] T ef90e8d046fc47fcb7d122ffad59a1f3 P 23ba075510134f3c8caba2370fff0d40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.560916 11310 tablet_server.cc:196] TabletServer@127.11.11.132:0 shutdown complete.
I20260709 12:20:59.645826 11310 master.cc:562] Master@127.11.11.190:37085 shutting down...
I20260709 12:20:59.680365 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:59.681085 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:59.681448 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 906ec57f6a8740bfbcc051926eb49307: stopping tablet replica
I20260709 12:20:59.705920 11310 master.cc:584] Master@127.11.11.190:37085 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (36341 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:20:59.817859 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:43251
I20260709 12:20:59.819231 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:59.826323 31876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:59.826436 31875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:59.827504 31878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:59.829879 11310 server_base.cc:1061] running on GCE node
I20260709 12:20:59.831352 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:59.831560 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:59.831696 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599659831684 us; error 0 us; skew 500 ppm
I20260709 12:20:59.832234 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:59.835606 11310 webserver.cc:533] Webserver started at http://127.11.11.190:34499/ using document root <none> and password file <none>
I20260709 12:20:59.836153 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:59.836323 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:59.836555 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:59.837946 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "c403a890f9d7443590cebc9e997b6a83"
format_stamp: "Formatted at 2026-07-09 12:20:59 on dist-test-slave-4qvp"
I20260709 12:20:59.843426 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:20:59.848387 31883 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:59.849400 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:20:59.849956 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "c403a890f9d7443590cebc9e997b6a83"
format_stamp: "Formatted at 2026-07-09 12:20:59 on dist-test-slave-4qvp"
I20260709 12:20:59.850335 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:59.877794 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:59.879469 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:59.931860 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:43251
I20260709 12:20:59.932057 31934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:43251 every 8 connection(s)
I20260709 12:20:59.936710 31935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:59.949275 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83: Bootstrap starting.
I20260709 12:20:59.954799 31935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:59.960294 31935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83: No bootstrap required, opened a new log
I20260709 12:20:59.962813 31935 raft_consensus.cc:359] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c403a890f9d7443590cebc9e997b6a83" member_type: VOTER }
I20260709 12:20:59.963229 31935 raft_consensus.cc:385] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:59.963565 31935 raft_consensus.cc:740] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403a890f9d7443590cebc9e997b6a83, State: Initialized, Role: FOLLOWER
I20260709 12:20:59.964195 31935 consensus_queue.cc:260] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c403a890f9d7443590cebc9e997b6a83" member_type: VOTER }
I20260709 12:20:59.964687 31935 raft_consensus.cc:399] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:59.964907 31935 raft_consensus.cc:493] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:59.965217 31935 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:59.970455 31935 raft_consensus.cc:515] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c403a890f9d7443590cebc9e997b6a83" member_type: VOTER }
I20260709 12:20:59.971079 31935 leader_election.cc:304] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c403a890f9d7443590cebc9e997b6a83; no voters: 
I20260709 12:20:59.972430 31935 leader_election.cc:290] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:59.973268 31938 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:59.975291 31938 raft_consensus.cc:697] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 1 LEADER]: Becoming Leader. State: Replica: c403a890f9d7443590cebc9e997b6a83, State: Running, Role: LEADER
I20260709 12:20:59.976158 31935 sys_catalog.cc:565] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:59.976157 31938 consensus_queue.cc:237] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c403a890f9d7443590cebc9e997b6a83" member_type: VOTER }
I20260709 12:20:59.982183 31939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c403a890f9d7443590cebc9e997b6a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c403a890f9d7443590cebc9e997b6a83" member_type: VOTER } }
I20260709 12:20:59.983100 31939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:59.982168 31940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c403a890f9d7443590cebc9e997b6a83. Latest consensus state: current_term: 1 leader_uuid: "c403a890f9d7443590cebc9e997b6a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c403a890f9d7443590cebc9e997b6a83" member_type: VOTER } }
I20260709 12:20:59.984378 31940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:59.987958 31945 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:21:00.002420 31945 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:21:00.004431 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:21:00.018273 31945 catalog_manager.cc:1383] Generated new cluster ID: cc4226e3e12048d38b3289e1293795ba
I20260709 12:21:00.018692 31945 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:21:00.036964 31945 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:21:00.038697 31945 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:21:00.059053 31945 catalog_manager.cc:6070] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83: Generated new TSK 0
I20260709 12:21:00.059978 31945 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:21:00.073457 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:00.081669 31957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.081669 31956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.084538 31959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:00.085083 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:00.086671 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:00.086922 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:00.087265 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599660087243 us; error 0 us; skew 500 ppm
I20260709 12:21:00.088042 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:00.091110 11310 webserver.cc:533] Webserver started at http://127.11.11.129:43297/ using document root <none> and password file <none>
I20260709 12:21:00.091676 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:00.091874 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:00.092142 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:00.093416 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:00.098920 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20260709 12:21:00.103214 31964 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.104516 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20260709 12:21:00.104871 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:00.105268 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:00.125157 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:00.126736 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:00.128594 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:00.131580 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:00.131791 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.132128 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:00.132318 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.187965 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:40845
I20260709 12:21:00.188174 32027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:40845 every 8 connection(s)
I20260709 12:21:00.194584 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:00.203059 32032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.206435 32033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.213032 32035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:00.214145 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:00.215595 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:00.215796 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:00.216029 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599660216009 us; error 0 us; skew 500 ppm
I20260709 12:21:00.216858 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:00.220412 11310 webserver.cc:533] Webserver started at http://127.11.11.130:35975/ using document root <none> and password file <none>
I20260709 12:21:00.220993 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:00.221087 32028 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43251
I20260709 12:21:00.221232 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:00.221951 32028 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:00.222015 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:00.223017 32028 heartbeater.cc:507] Master 127.11.11.190:43251 requested a full tablet report, sending...
I20260709 12:21:00.223647 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "c1e0a103e97c439aac4237ea88e62c1a"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:00.225842 31900 ts_manager.cc:194] Registered new tserver with Master: fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:00.228591 31900 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51288
I20260709 12:21:00.230798 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20260709 12:21:00.235498 32040 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.236740 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.001s
I20260709 12:21:00.237077 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "c1e0a103e97c439aac4237ea88e62c1a"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:00.237404 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:00.260084 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:00.261571 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:00.263513 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:00.266911 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:00.267135 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.267530 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:00.267697 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.320881 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:41757
I20260709 12:21:00.321131 32103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:41757 every 8 connection(s)
I20260709 12:21:00.326802 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:00.335253 32107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:00.336678 32108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:00.340919 32104 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43251
W20260709 12:21:00.341327 32111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:00.341531 32104 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:00.342011 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:00.342969 32104 heartbeater.cc:507] Master 127.11.11.190:43251 requested a full tablet report, sending...
I20260709 12:21:00.343327 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:00.343634 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:00.343853 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599660343834 us; error 0 us; skew 500 ppm
I20260709 12:21:00.344566 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:00.345764 31900 ts_manager.cc:194] Registered new tserver with Master: c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:00.347880 11310 webserver.cc:533] Webserver started at http://127.11.11.131:37541/ using document root <none> and password file <none>
I20260709 12:21:00.347965 31900 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51296
I20260709 12:21:00.348750 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:00.349041 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:00.349390 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:00.350922 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "0cdc7c3888934515815e7f967d05ed7f"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:00.356361 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:21:00.360698 32115 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.361900 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 12:21:00.362313 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "0cdc7c3888934515815e7f967d05ed7f"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:00.362653 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:00.394965 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:00.396580 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:00.398342 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:00.401116 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:00.401386 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.401684 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:00.401873 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:00.452977 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:33905
I20260709 12:21:00.453130 32177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:33905 every 8 connection(s)
I20260709 12:21:00.478451 32178 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43251
I20260709 12:21:00.478880 32178 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:00.479682 32178 heartbeater.cc:507] Master 127.11.11.190:43251 requested a full tablet report, sending...
I20260709 12:21:00.482167 31900 ts_manager.cc:194] Registered new tserver with Master: 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:00.483096 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.026111715s
I20260709 12:21:00.483731 31900 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51304
I20260709 12:21:00.523268 31900 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51316:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:00.526921 31900 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:00.611495 32068 tablet_service.cc:1511] Processing CreateTablet for tablet 9dc970bdd4f642cdb39c7204a4a6dd99 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:21:00.612038 32067 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb7b01cdf344f588152a57c8c516248 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:21:00.613282 32068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc970bdd4f642cdb39c7204a4a6dd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.614257 32067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb7b01cdf344f588152a57c8c516248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.620348 32066 tablet_service.cc:1511] Processing CreateTablet for tablet 6a4a219354da4c82859196f969a75f56 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:21:00.622537 32066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a4a219354da4c82859196f969a75f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.625383 32142 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb7b01cdf344f588152a57c8c516248 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:21:00.626408 32143 tablet_service.cc:1511] Processing CreateTablet for tablet 9dc970bdd4f642cdb39c7204a4a6dd99 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:21:00.627285 32142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb7b01cdf344f588152a57c8c516248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.628194 32143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc970bdd4f642cdb39c7204a4a6dd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.630877 32141 tablet_service.cc:1511] Processing CreateTablet for tablet 6a4a219354da4c82859196f969a75f56 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:21:00.635010 32140 tablet_service.cc:1511] Processing CreateTablet for tablet 25c32b59fa724a38b69a0aa1221f2f8a (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:21:00.636847 32140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25c32b59fa724a38b69a0aa1221f2f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.638518 32141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a4a219354da4c82859196f969a75f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.639032 32139 tablet_service.cc:1511] Processing CreateTablet for tablet ca77a615a159489bb88babe373bdfb23 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:21:00.640688 32139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca77a615a159489bb88babe373bdfb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.643064 32138 tablet_service.cc:1511] Processing CreateTablet for tablet 1e175deb4c034842b8800ba846928c22 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:21:00.644815 32138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e175deb4c034842b8800ba846928c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.647002 32137 tablet_service.cc:1511] Processing CreateTablet for tablet 4428dfd2db04432a919f02ec0870f0a5 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:21:00.648713 32137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4428dfd2db04432a919f02ec0870f0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.657442 32135 tablet_service.cc:1511] Processing CreateTablet for tablet f6ea91a8dcd0496e8c9292943bec8a55 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:21:00.659116 32135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ea91a8dcd0496e8c9292943bec8a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.664178 32136 tablet_service.cc:1511] Processing CreateTablet for tablet 10f6298f890b4c2ebf6862c423be53cc (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:21:00.668215 32134 tablet_service.cc:1511] Processing CreateTablet for tablet d25f5d835b41473c897af0968c384fe4 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:21:00.679705 32065 tablet_service.cc:1511] Processing CreateTablet for tablet 25c32b59fa724a38b69a0aa1221f2f8a (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:21:00.682524 32136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f6298f890b4c2ebf6862c423be53cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.685950 32134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25f5d835b41473c897af0968c384fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.694221 32065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25c32b59fa724a38b69a0aa1221f2f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.733466 32064 tablet_service.cc:1511] Processing CreateTablet for tablet 1e175deb4c034842b8800ba846928c22 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:21:00.735110 32064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e175deb4c034842b8800ba846928c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.740114 32066 tablet_service.cc:1511] Processing CreateTablet for tablet ca77a615a159489bb88babe373bdfb23 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:21:00.740257 32063 tablet_service.cc:1511] Processing CreateTablet for tablet 4428dfd2db04432a919f02ec0870f0a5 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:21:00.741876 32066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca77a615a159489bb88babe373bdfb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.744511 32062 tablet_service.cc:1511] Processing CreateTablet for tablet 10f6298f890b4c2ebf6862c423be53cc (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:21:00.748440 32061 tablet_service.cc:1511] Processing CreateTablet for tablet f6ea91a8dcd0496e8c9292943bec8a55 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:21:00.750922 32063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4428dfd2db04432a919f02ec0870f0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.771075 31992 tablet_service.cc:1511] Processing CreateTablet for tablet 9dc970bdd4f642cdb39c7204a4a6dd99 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:21:00.773000 31992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dc970bdd4f642cdb39c7204a4a6dd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.779301 32195 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:00.781195 32194 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:00.782970 31991 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb7b01cdf344f588152a57c8c516248 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:21:00.784861 31991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb7b01cdf344f588152a57c8c516248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.786423 32062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f6298f890b4c2ebf6862c423be53cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.793843 32061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ea91a8dcd0496e8c9292943bec8a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.788151 31988 tablet_service.cc:1511] Processing CreateTablet for tablet ca77a615a159489bb88babe373bdfb23 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:21:00.795331 31988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca77a615a159489bb88babe373bdfb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.799047 31983 tablet_service.cc:1511] Processing CreateTablet for tablet d25f5d835b41473c897af0968c384fe4 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:21:00.800737 31983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25f5d835b41473c897af0968c384fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.788875 31986 tablet_service.cc:1511] Processing CreateTablet for tablet 4428dfd2db04432a919f02ec0870f0a5 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:21:00.803272 31986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4428dfd2db04432a919f02ec0870f0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.804512 32067 tablet_service.cc:1511] Processing CreateTablet for tablet d25f5d835b41473c897af0968c384fe4 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:21:00.789412 31989 tablet_service.cc:1511] Processing CreateTablet for tablet 25c32b59fa724a38b69a0aa1221f2f8a (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:21:00.791115 31985 tablet_service.cc:1511] Processing CreateTablet for tablet 10f6298f890b4c2ebf6862c423be53cc (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:21:00.810370 31985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f6298f890b4c2ebf6862c423be53cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.792073 31984 tablet_service.cc:1511] Processing CreateTablet for tablet f6ea91a8dcd0496e8c9292943bec8a55 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:21:00.815281 31984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ea91a8dcd0496e8c9292943bec8a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.792678 31990 tablet_service.cc:1511] Processing CreateTablet for tablet 6a4a219354da4c82859196f969a75f56 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:21:00.818847 31990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a4a219354da4c82859196f969a75f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.790755 31987 tablet_service.cc:1511] Processing CreateTablet for tablet 1e175deb4c034842b8800ba846928c22 (DEFAULT_TABLE table=test-workload [id=1ea649f5e32347dfbb8714342b2e7387]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:21:00.822999 31987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e175deb4c034842b8800ba846928c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.824822 31989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25c32b59fa724a38b69a0aa1221f2f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.844062 32067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d25f5d835b41473c897af0968c384fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:00.845520 32194 tablet_bootstrap.cc:654] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:00.891276 32196 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:00.900105 32195 tablet_bootstrap.cc:654] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:00.905939 32194 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:00.906730 32194 ts_tablet_manager.cc:1403] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.126s	user 0.017s	sys 0.002s
I20260709 12:21:00.925827 32194 raft_consensus.cc:359] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:00.934067 32194 raft_consensus.cc:385] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:00.934953 32196 tablet_bootstrap.cc:654] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:00.935138 32194 raft_consensus.cc:740] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:00.936517 32194 consensus_queue.cc:260] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:00.944442 32195 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:00.945276 32195 ts_tablet_manager.cc:1403] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.166s	user 0.009s	sys 0.025s
I20260709 12:21:00.946142 32196 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:00.946731 32196 ts_tablet_manager.cc:1403] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.056s	user 0.019s	sys 0.000s
I20260709 12:21:00.948750 32195 raft_consensus.cc:359] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:00.949846 32196 raft_consensus.cc:359] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:00.951488 32194 ts_tablet_manager.cc:1434] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.038s	user 0.005s	sys 0.003s
I20260709 12:21:00.951721 32195 raft_consensus.cc:385] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:00.951900 32196 raft_consensus.cc:385] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:00.952180 32195 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:00.952771 32178 heartbeater.cc:499] Master 127.11.11.190:43251 was elected leader, sending a full tablet report...
I20260709 12:21:00.953230 32194 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:00.953753 32195 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:00.952379 32196 raft_consensus.cc:740] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:00.956230 32196 consensus_queue.cc:260] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
W20260709 12:21:00.958132 32180 tablet.cc:2406] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:21:00.961026 32194 tablet_bootstrap.cc:654] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:00.989459 32104 heartbeater.cc:499] Master 127.11.11.190:43251 was elected leader, sending a full tablet report...
I20260709 12:21:00.989527 32196 ts_tablet_manager.cc:1434] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.042s	user 0.017s	sys 0.021s
I20260709 12:21:00.989825 32195 ts_tablet_manager.cc:1434] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.044s	user 0.018s	sys 0.023s
I20260709 12:21:00.991199 32196 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:00.991020 32201 raft_consensus.cc:493] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.991312 32195 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:00.991734 32201 raft_consensus.cc:515] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:00.992690 32028 heartbeater.cc:499] Master 127.11.11.190:43251 was elected leader, sending a full tablet report...
I20260709 12:21:00.989665 32194 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:00.996383 32194 ts_tablet_manager.cc:1403] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.043s	user 0.012s	sys 0.025s
I20260709 12:21:01.000185 32194 raft_consensus.cc:359] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:01.001117 32194 raft_consensus.cc:385] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.001570 32194 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.000660 32196 tablet_bootstrap.cc:654] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:01.005002 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:01.007350 32201 leader_election.cc:290] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905), fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:01.010537 32195 tablet_bootstrap.cc:654] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.013288 32194 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:01.022795 32194 ts_tablet_manager.cc:1434] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260709 12:21:01.036778 32194 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.047798 32194 tablet_bootstrap.cc:654] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.048840 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.049790 32153 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 0.
I20260709 12:21:01.053706 32041 leader_election.cc:304] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:01.053933 32195 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.054930 32201 raft_consensus.cc:2804] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.055462 32201 raft_consensus.cc:493] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.056018 32201 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.055091 32195 ts_tablet_manager.cc:1403] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.064s	user 0.019s	sys 0.001s
I20260709 12:21:01.061718 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
W20260709 12:21:01.063969 32042 leader_election.cc:343] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.066568 32195 raft_consensus.cc:359] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.067395 32201 raft_consensus.cc:515] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:01.067560 32195 raft_consensus.cc:385] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.068473 32195 raft_consensus.cc:740] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.069416 32195 consensus_queue.cc:260] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.069553 32196 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.069864 32200 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.072154 32196 ts_tablet_manager.cc:1403] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.081s	user 0.028s	sys 0.012s
I20260709 12:21:01.072124 32200 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.071273 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.077533 32196 raft_consensus.cc:359] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.078447 32196 raft_consensus.cc:385] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.078763 32196 raft_consensus.cc:740] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.078982 32153 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.079646 32196 consensus_queue.cc:260] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.071403 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:01.082105 32194 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.083142 32194 ts_tablet_manager.cc:1403] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.047s	user 0.028s	sys 0.000s
I20260709 12:21:01.083508 32200 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:01.088295 32194 raft_consensus.cc:359] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.089900 32194 raft_consensus.cc:385] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.090431 32194 raft_consensus.cc:740] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.091852 32196 ts_tablet_manager.cc:1434] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
W20260709 12:21:01.092175 32042 leader_election.cc:343] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.091702 32194 consensus_queue.cc:260] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.069942 32201 leader_election.cc:290] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Requested vote from peers 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905), fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:01.100306 32153 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:01.101982 32041 leader_election.cc:304] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:01.103513 32201 raft_consensus.cc:2804] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.104096 32201 raft_consensus.cc:697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:01.105773 32196 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.105018 32201 consensus_queue.cc:237] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:01.115947 32194 ts_tablet_manager.cc:1434] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.003s
I20260709 12:21:01.117316 32196 tablet_bootstrap.cc:654] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.123162 32195 ts_tablet_manager.cc:1434] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.061s	user 0.026s	sys 0.030s
I20260709 12:21:01.124571 32195 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.125682 32194 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.133395 32195 tablet_bootstrap.cc:654] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.136888 31897 catalog_manager.cc:5697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 1 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.142757 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.146257 32195 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.146945 32195 ts_tablet_manager.cc:1403] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.008s
W20260709 12:21:01.147003 32117 leader_election.cc:343] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.147760 32194 tablet_bootstrap.cc:654] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.150128 32196 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.151041 32196 ts_tablet_manager.cc:1403] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.046s	user 0.010s	sys 0.026s
I20260709 12:21:01.150666 32195 raft_consensus.cc:359] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.151680 32195 raft_consensus.cc:385] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.152058 32195 raft_consensus.cc:740] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.153551 32195 consensus_queue.cc:260] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.156662 32195 ts_tablet_manager.cc:1434] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.002s
I20260709 12:21:01.156189 32196 raft_consensus.cc:359] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.157465 32196 raft_consensus.cc:385] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.157932 32196 raft_consensus.cc:740] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.158108 32195 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.159325 32196 consensus_queue.cc:260] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.161269 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
W20260709 12:21:01.163635 32117 leader_election.cc:343] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.164460 32117 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:01.166042 32200 raft_consensus.cc:2749] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:01.168489 32196 ts_tablet_manager.cc:1434] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.004s
I20260709 12:21:01.169330 32195 tablet_bootstrap.cc:654] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.170086 32196 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.180945 32196 tablet_bootstrap.cc:654] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.194252 32196 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.194880 32196 ts_tablet_manager.cc:1403] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.025s	user 0.015s	sys 0.005s
I20260709 12:21:01.198632 32196 raft_consensus.cc:359] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.199484 32196 raft_consensus.cc:385] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.199709 32194 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.200114 32195 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.199816 32196 raft_consensus.cc:740] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.201027 32194 ts_tablet_manager.cc:1403] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.076s	user 0.028s	sys 0.013s
I20260709 12:21:01.200959 32195 ts_tablet_manager.cc:1403] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.023s
I20260709 12:21:01.201651 32196 consensus_queue.cc:260] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.204424 32196 ts_tablet_manager.cc:1434] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:21:01.205804 32196 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.205219 32195 raft_consensus.cc:359] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.206290 32195 raft_consensus.cc:385] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.206634 32195 raft_consensus.cc:740] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.207538 32195 consensus_queue.cc:260] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.211376 32194 raft_consensus.cc:359] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.212272 32195 ts_tablet_manager.cc:1434] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.006s
I20260709 12:21:01.212450 32194 raft_consensus.cc:385] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.213243 32194 raft_consensus.cc:740] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.214028 32195 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.214217 32196 tablet_bootstrap.cc:654] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.214393 32194 consensus_queue.cc:260] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.223472 32195 tablet_bootstrap.cc:654] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.227169 32194 ts_tablet_manager.cc:1434] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.001s
I20260709 12:21:01.228616 32194 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.236080 32194 tablet_bootstrap.cc:654] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.237131 32202 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.237897 32202 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.238878 32195 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.239789 32195 ts_tablet_manager.cc:1403] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.026s	user 0.023s	sys 0.001s
I20260709 12:21:01.243464 32195 raft_consensus.cc:359] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.244339 32195 raft_consensus.cc:385] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.244693 32195 raft_consensus.cc:740] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.244748 32196 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.245502 32196 ts_tablet_manager.cc:1403] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.040s	user 0.014s	sys 0.007s
I20260709 12:21:01.245898 32195 consensus_queue.cc:260] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.248708 32195 ts_tablet_manager.cc:1434] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:21:01.248857 32196 raft_consensus.cc:359] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:01.249701 32196 raft_consensus.cc:385] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.249995 32196 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.250111 32195 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.250834 32196 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:01.259877 32195 tablet_bootstrap.cc:654] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.261351 32196 ts_tablet_manager.cc:1434] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.003s
I20260709 12:21:01.266494 32196 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.275924 32196 tablet_bootstrap.cc:654] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.279160 32224 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.279896 32224 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.282296 32202 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.283509 32224 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.277616 32195 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.285512 32195 ts_tablet_manager.cc:1403] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.005s
I20260709 12:21:01.295920 32194 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.296821 32194 ts_tablet_manager.cc:1403] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.068s	user 0.024s	sys 0.020s
I20260709 12:21:01.297557 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.289353 32195 raft_consensus.cc:359] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.298038 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.298607 32195 raft_consensus.cc:385] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.298548 32078 raft_consensus.cc:2468] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.298861 32077 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.299194 32195 raft_consensus.cc:740] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.300295 32195 consensus_queue.cc:260] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.300576 32194 raft_consensus.cc:359] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.300770 31966 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.301407 32194 raft_consensus.cc:385] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.301791 32194 raft_consensus.cc:740] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.302273 31966 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.303073 32195 ts_tablet_manager.cc:1434] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:21:01.302671 32194 consensus_queue.cc:260] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.304030 32202 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.304356 32195 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.304558 32202 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.305065 32202 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.302405 32224 raft_consensus.cc:2804] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.311139 32194 ts_tablet_manager.cc:1434] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.000s
I20260709 12:21:01.311513 32224 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.312271 32224 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.312706 32194 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.316159 32202 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.326038 32202 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.320979 32194 tablet_bootstrap.cc:654] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.330834 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.327029 32229 raft_consensus.cc:493] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.327988 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:01.332130 32077 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:01.333825 31965 leader_election.cc:343] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.312364 32195 tablet_bootstrap.cc:654] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.320884 32196 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.335528 32196 ts_tablet_manager.cc:1403] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.069s	user 0.017s	sys 0.016s
I20260709 12:21:01.327661 32224 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.329689 32230 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.334510 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.338896 32153 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.338822 32230 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.340444 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.341380 32152 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.343953 32194 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.343699 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:01.344592 32194 ts_tablet_manager.cc:1403] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.004s
I20260709 12:21:01.344658 32078 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.344005 32224 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.343806 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
W20260709 12:21:01.349566 31965 leader_election.cc:343] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.352028 32076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.352352 32229 raft_consensus.cc:515] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.352713 32230 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.352983 32076 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.349211 32196 raft_consensus.cc:359] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.354681 31966 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.353034 32194 raft_consensus.cc:359] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.355962 32194 raft_consensus.cc:385] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.356375 32194 raft_consensus.cc:740] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.356527 32151 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.357738 32229 leader_election.cc:290] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.357518 32078 raft_consensus.cc:2468] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.358037 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.357126 32076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
W20260709 12:21:01.358747 31965 leader_election.cc:343] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.357787 32194 consensus_queue.cc:260] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.359760 32153 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
W20260709 12:21:01.360884 31966 leader_election.cc:343] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.361423 31966 leader_election.cc:304] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:01.362386 32152 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.362718 32229 raft_consensus.cc:2749] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:01.363927 31965 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.364516 32077 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.365356 32224 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.365738 32194 ts_tablet_manager.cc:1434] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.001s
I20260709 12:21:01.358000 32230 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.366060 32224 raft_consensus.cc:697] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Becoming Leader. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Running, Role: LEADER
I20260709 12:21:01.368099 32194 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.367740 32224 consensus_queue.cc:237] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.366755 32230 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.368811 32195 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.369015 32230 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.369426 32195 ts_tablet_manager.cc:1403] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.065s	user 0.012s	sys 0.003s
I20260709 12:21:01.372973 32195 raft_consensus.cc:359] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.373793 32195 raft_consensus.cc:385] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.374135 32195 raft_consensus.cc:740] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.375228 32195 consensus_queue.cc:260] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.379472 31966 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:01.381857 32202 raft_consensus.cc:2804] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.382781 32202 raft_consensus.cc:697] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Becoming Leader. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Running, Role: LEADER
I20260709 12:21:01.383677 32202 consensus_queue.cc:237] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.385586 32230 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.387780 31900 catalog_manager.cc:5697] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.389182 32194 tablet_bootstrap.cc:654] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.390355 32230 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.388007 32195 ts_tablet_manager.cc:1434] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 12:21:01.353762 32196 raft_consensus.cc:385] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.391360 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:01.391644 32196 raft_consensus.cc:740] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.392192 32077 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.391752 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.393286 32152 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.402848 32194 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.403496 32077 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.403723 32194 ts_tablet_manager.cc:1403] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.011s
I20260709 12:21:01.404991 31966 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.414170 32194 raft_consensus.cc:359] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.415391 32194 raft_consensus.cc:385] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.415859 32194 raft_consensus.cc:740] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.392908 32196 consensus_queue.cc:260] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.415490 32195 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.415926 32230 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.418457 32230 raft_consensus.cc:697] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Becoming Leader. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Running, Role: LEADER
I20260709 12:21:01.417119 32194 consensus_queue.cc:260] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.419298 32196 ts_tablet_manager.cc:1434] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.083s	user 0.009s	sys 0.000s
I20260709 12:21:01.425148 32194 ts_tablet_manager.cc:1434] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.021s	user 0.002s	sys 0.005s
I20260709 12:21:01.419497 32230 consensus_queue.cc:237] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.427248 32194 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.421469 32196 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.425894 32195 tablet_bootstrap.cc:654] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.435842 32152 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.444403 32196 tablet_bootstrap.cc:654] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.445123 32194 tablet_bootstrap.cc:654] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.446024 32195 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.446714 32195 ts_tablet_manager.cc:1403] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.032s	user 0.003s	sys 0.014s
I20260709 12:21:01.449985 32195 raft_consensus.cc:359] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.450795 32195 raft_consensus.cc:385] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.451084 32195 raft_consensus.cc:740] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.451849 32195 consensus_queue.cc:260] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.454928 32195 ts_tablet_manager.cc:1434] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:21:01.454550 31900 catalog_manager.cc:5697] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.456578 32195 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.457216 31900 catalog_manager.cc:5697] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.460486 32201 raft_consensus.cc:493] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.461143 32201 raft_consensus.cc:515] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.463734 32201 leader_election.cc:290] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.465389 32195 tablet_bootstrap.cc:654] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.466343 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.467157 32002 raft_consensus.cc:2468] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 0.
I20260709 12:21:01.471347 32195 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.472060 32195 ts_tablet_manager.cc:1403] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.005s
I20260709 12:21:01.473044 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.475284 32195 raft_consensus.cc:359] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
W20260709 12:21:01.476384 32041 leader_election.cc:343] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.476540 32195 raft_consensus.cc:385] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.477366 32195 raft_consensus.cc:740] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.478000 32042 leader_election.cc:304] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:01.478209 32195 consensus_queue.cc:260] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.479490 32209 raft_consensus.cc:2804] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.479918 32209 raft_consensus.cc:493] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.480443 32209 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.481144 32230 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.481382 32195 ts_tablet_manager.cc:1434] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:21:01.481770 32230 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.482760 32195 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: Bootstrap starting.
I20260709 12:21:01.484257 32230 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.486222 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.487033 32077 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.488744 31966 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.479861 32194 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.489760 32194 ts_tablet_manager.cc:1403] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.063s	user 0.019s	sys 0.004s
I20260709 12:21:01.489801 32195 tablet_bootstrap.cc:654] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.491310 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.491848 32196 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.492132 32152 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.492548 32196 ts_tablet_manager.cc:1403] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.072s	user 0.021s	sys 0.000s
I20260709 12:21:01.492945 32230 raft_consensus.cc:2804] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.493669 32230 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.493779 32194 raft_consensus.cc:359] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.494848 32194 raft_consensus.cc:385] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.495467 32194 raft_consensus.cc:740] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.496122 32196 raft_consensus.cc:359] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.496508 32194 consensus_queue.cc:260] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.497000 32196 raft_consensus.cc:385] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.498098 32209 raft_consensus.cc:515] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.494807 32230 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.501726 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:01.502553 32002 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.508255 32194 ts_tablet_manager.cc:1434] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.001s
I20260709 12:21:01.512770 32002 raft_consensus.cc:2468] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:01.514386 32042 leader_election.cc:304] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.515413 32201 raft_consensus.cc:2804] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.515921 32194 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.523402 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
W20260709 12:21:01.525096 32041 leader_election.cc:343] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:01.526343 32209 leader_election.cc:290] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.527034 32194 tablet_bootstrap.cc:654] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.529985 32201 raft_consensus.cc:697] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:01.531273 32201 consensus_queue.cc:237] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.532514 32230 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.497331 32196 raft_consensus.cc:740] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.535863 32196 consensus_queue.cc:260] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.538640 32196 ts_tablet_manager.cc:1434] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.046s	user 0.009s	sys 0.001s
I20260709 12:21:01.539583 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:01.540299 32077 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.540989 32209 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.542412 32195 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: No bootstrap required, opened a new log
I20260709 12:21:01.543040 32195 ts_tablet_manager.cc:1403] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: Time spent bootstrapping tablet: real 0.061s	user 0.014s	sys 0.043s
I20260709 12:21:01.546311 32195 raft_consensus.cc:359] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.547056 32195 raft_consensus.cc:385] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.547319 32195 raft_consensus.cc:740] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.548182 32195 consensus_queue.cc:260] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.550514 32195 ts_tablet_manager.cc:1434] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:01.551797 32196 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.559422 32077 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.558907 31898 catalog_manager.cc:5697] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 1 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.561048 32230 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.561189 31966 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.560371 32196 tablet_bootstrap.cc:654] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.566648 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.567076 32194 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.567392 32153 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.567607 32194 ts_tablet_manager.cc:1403] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.052s	user 0.019s	sys 0.000s
I20260709 12:21:01.579063 32002 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.582866 32194 raft_consensus.cc:359] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.583994 32194 raft_consensus.cc:385] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.584249 32194 raft_consensus.cc:740] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.585222 32194 consensus_queue.cc:260] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.588387 32194 ts_tablet_manager.cc:1434] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.002s
I20260709 12:21:01.590754 32242 raft_consensus.cc:493] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.591359 32242 raft_consensus.cc:515] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.597805 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e175deb4c034842b8800ba846928c22" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.599114 32152 raft_consensus.cc:2468] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 0.
I20260709 12:21:01.600694 32041 leader_election.cc:304] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:01.601804 32209 raft_consensus.cc:2804] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.602254 32209 raft_consensus.cc:493] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.602741 32209 raft_consensus.cc:3060] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.596087 32001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e175deb4c034842b8800ba846928c22" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.606670 32001 raft_consensus.cc:2468] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 0.
I20260709 12:21:01.606789 32242 leader_election.cc:290] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.614248 32242 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.619529 32202 raft_consensus.cc:2804] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.620374 32202 raft_consensus.cc:697] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Becoming Leader. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Running, Role: LEADER
I20260709 12:21:01.621927 32202 consensus_queue.cc:237] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.589957 32194 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: Bootstrap starting.
I20260709 12:21:01.636588 32153 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.641826 32209 raft_consensus.cc:515] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.643347 32196 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.644116 32196 ts_tablet_manager.cc:1403] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.093s	user 0.036s	sys 0.018s
I20260709 12:21:01.645464 32001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e175deb4c034842b8800ba846928c22" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:01.646031 32194 tablet_bootstrap.cc:654] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.646284 32001 raft_consensus.cc:3060] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.648058 32196 raft_consensus.cc:359] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.649326 32196 raft_consensus.cc:385] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.649772 32196 raft_consensus.cc:740] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.650933 32196 consensus_queue.cc:260] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.652138 32209 leader_election.cc:290] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.654688 32196 ts_tablet_manager.cc:1434] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20260709 12:21:01.656245 32196 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.656549 31900 catalog_manager.cc:5697] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.656148 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e175deb4c034842b8800ba846928c22" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.665102 32196 tablet_bootstrap.cc:654] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.665683 32153 raft_consensus.cc:3060] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.678759 32196 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.679747 32196 ts_tablet_manager.cc:1403] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.007s
I20260709 12:21:01.683517 32196 raft_consensus.cc:359] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.684650 32196 raft_consensus.cc:385] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.685015 32196 raft_consensus.cc:740] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.685863 32196 consensus_queue.cc:260] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.688762 32196 ts_tablet_manager.cc:1434] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:21:01.690332 32196 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:01.673748 32001 raft_consensus.cc:2468] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:01.695627 32042 leader_election.cc:304] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.697094 32209 raft_consensus.cc:2804] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.697862 32209 raft_consensus.cc:697] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:01.698028 32196 tablet_bootstrap.cc:654] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:01.699158 32209 consensus_queue.cc:237] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.702528 32153 raft_consensus.cc:2468] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:01.705941 32194 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: No bootstrap required, opened a new log
I20260709 12:21:01.706557 32194 ts_tablet_manager.cc:1403] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: Time spent bootstrapping tablet: real 0.117s	user 0.031s	sys 0.030s
I20260709 12:21:01.708781 32202 raft_consensus.cc:493] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.708900 32194 raft_consensus.cc:359] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.710067 32202 raft_consensus.cc:515] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.710635 32194 raft_consensus.cc:385] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.710975 32194 raft_consensus.cc:740] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.711899 32196 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: No bootstrap required, opened a new log
I20260709 12:21:01.712162 32200 raft_consensus.cc:493] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.712808 32196 ts_tablet_manager.cc:1403] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.010s
I20260709 12:21:01.711907 32194 consensus_queue.cc:260] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.713099 32200 raft_consensus.cc:515] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.714143 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.715252 32077 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.715476 32200 leader_election.cc:290] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:01.717543 31966 leader_election.cc:304] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.719102 32230 raft_consensus.cc:2804] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.719676 32230 raft_consensus.cc:493] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.720149 32230 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.722004 32196 raft_consensus.cc:359] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.722884 32196 raft_consensus.cc:385] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:01.723151 32196 raft_consensus.cc:740] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:01.723865 32196 consensus_queue.cc:260] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.726212 32196 ts_tablet_manager.cc:1434] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:21:01.726365 32200 raft_consensus.cc:493] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.727116 32200 raft_consensus.cc:515] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.727628 32202 leader_election.cc:290] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.735782 32200 leader_election.cc:290] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:01.740006 32201 raft_consensus.cc:493] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.741083 32201 raft_consensus.cc:515] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.746129 32230 raft_consensus.cc:515] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.748327 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.748646 32230 leader_election.cc:290] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.750488 32001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.750744 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.751055 32194 ts_tablet_manager.cc:1434] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: Time spent starting tablet: real 0.044s	user 0.031s	sys 0.010s
I20260709 12:21:01.751304 32153 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 0.
I20260709 12:21:01.752336 32000 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.753006 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.753376 32000 raft_consensus.cc:2393] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cdc7c3888934515815e7f967d05ed7f in current term 1: Already voted for candidate c1e0a103e97c439aac4237ea88e62c1a in this term.
I20260709 12:21:01.753850 32077 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 0.
I20260709 12:21:01.754678 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.755388 32153 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.755880 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:01.756631 32077 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.751665 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.758448 32002 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 0.
I20260709 12:21:01.751253 32001 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 0.
I20260709 12:21:01.760246 32042 leader_election.cc:304] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.761260 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:01.761325 32209 raft_consensus.cc:2804] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.762113 32152 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 0.
I20260709 12:21:01.762655 32209 raft_consensus.cc:493] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.763125 32209 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.764209 32201 leader_election.cc:290] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.765751 32117 leader_election.cc:304] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:01.767127 32117 leader_election.cc:304] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.767800 32257 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.769887 32258 raft_consensus.cc:2804] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:01.770466 32258 raft_consensus.cc:493] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.770874 32258 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.766993 32153 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.768379 31898 catalog_manager.cc:5697] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 1 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.780370 32257 raft_consensus.cc:2749] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:01.780606 32200 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.781134 32200 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.782356 31965 leader_election.cc:304] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.783316 32230 raft_consensus.cc:2804] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.783387 32200 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:01.779625 32258 raft_consensus.cc:515] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.785661 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:01.786399 32077 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in term 1.
I20260709 12:21:01.786752 32078 raft_consensus.cc:2393] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cdc7c3888934515815e7f967d05ed7f in current term 1: Already voted for candidate fcd82f4857ce4c42af9f11d5892f9ce1 in this term.
I20260709 12:21:01.784118 32230 raft_consensus.cc:697] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Becoming Leader. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Running, Role: LEADER
I20260709 12:21:01.787817 32258 leader_election.cc:290] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:01.788147 32002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:01.784660 32001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" is_pre_election: true
I20260709 12:21:01.788944 32002 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.788638 32230 consensus_queue.cc:237] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.783972 32209 raft_consensus.cc:515] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.790956 32117 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:01.791597 32209 leader_election.cc:290] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:01.792255 32258 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.792914 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:01.793735 32153 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.795171 32001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:01.795939 32001 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.799429 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:01.800259 32078 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:01.802877 32258 raft_consensus.cc:2749] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:01.804754 32202 consensus_queue.cc:1048] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.818725 32153 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:01.822866 32041 leader_election.cc:304] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:01.824208 32209 raft_consensus.cc:2804] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.824903 32209 raft_consensus.cc:697] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:01.826547 32209 consensus_queue.cc:237] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.837860 32202 consensus_queue.cc:1048] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.841724 32002 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 1.
I20260709 12:21:01.843777 32117 leader_election.cc:304] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:01.845290 32258 raft_consensus.cc:2804] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:01.845959 32258 raft_consensus.cc:697] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:01.847101 32258 consensus_queue.cc:237] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:01.866211 32001 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:01.874805 31898 catalog_manager.cc:5697] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129). New cstate: current_term: 1 leader_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.873277 32078 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 1.
I20260709 12:21:01.894269 32224 consensus_queue.cc:1048] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.910727 32224 consensus_queue.cc:1048] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.924217 31898 catalog_manager.cc:5697] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 1 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:01.935441 31899 catalog_manager.cc:5697] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 0 to 1, leader changed from <none> to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 1 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.949453 32230 consensus_queue.cc:1048] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:01.966198 32264 consensus_queue.cc:1048] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:02.047675 11310 ts_manager.cc:295] Set tserver state for fcd82f4857ce4c42af9f11d5892f9ce1 to MAINTENANCE_MODE
I20260709 12:21:02.048619 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
I20260709 12:21:02.079263 32264 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:02.081018 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:02.082219 11310 tablet_replica.cc:333] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.083137 11310 raft_consensus.cc:2243] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:02.084220 11310 pending_rounds.cc:70] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Trying to abort 1 pending ops.
I20260709 12:21:02.084553 11310 pending_rounds.cc:77] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305624214704738304 op_type: NO_OP noop_request { }
I20260709 12:21:02.085181 11310 raft_consensus.cc:2889] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:21:02.085587 11310 raft_consensus.cc:2272] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.089265 11310 tablet_replica.cc:333] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.090224 11310 raft_consensus.cc:2243] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:02.090710 11310 raft_consensus.cc:2272] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.093377 11310 tablet_replica.cc:333] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.094360 11310 raft_consensus.cc:2243] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:02.094689 32224 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:02.096174 11310 raft_consensus.cc:2272] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.098861 11310 tablet_replica.cc:333] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.099572 11310 raft_consensus.cc:2243] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:02.100106 11310 raft_consensus.cc:2272] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.103349 11310 tablet_replica.cc:333] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.104202 11310 raft_consensus.cc:2243] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:02.105348 11310 raft_consensus.cc:2272] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.108585 11310 tablet_replica.cc:333] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
W20260709 12:21:02.109200 32042 consensus_peers.cc:597] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:02.109689 11310 raft_consensus.cc:2243] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:02.110984 11310 raft_consensus.cc:2272] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.110908 32209 consensus_queue.cc:1048] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:02.114106 11310 tablet_replica.cc:333] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.114807 11310 raft_consensus.cc:2243] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:02.115265 11310 raft_consensus.cc:2272] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.118149 11310 tablet_replica.cc:333] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.123521 11310 raft_consensus.cc:2243] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:02.124725 11310 raft_consensus.cc:2272] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.128396 11310 tablet_replica.cc:333] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.129225 11310 raft_consensus.cc:2243] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:02.129770 11310 raft_consensus.cc:2272] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:02.132735 11310 tablet_replica.cc:333] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: stopping tablet replica
I20260709 12:21:02.133500 11310 raft_consensus.cc:2243] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:02.134186 11310 raft_consensus.cc:2272] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:02.135462 32042 consensus_peers.cc:597] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:02.158314 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
I20260709 12:21:02.251183 32209 consensus_queue.cc:1048] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:02.272933 32042 consensus_peers.cc:597] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:02.315703 32258 consensus_queue.cc:1048] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:02.334038 32117 consensus_peers.cc:597] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:02.443274 32209 consensus_queue.cc:1048] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:02.583302 32042 consensus_peers.cc:597] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:02.733702 32042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20260709 12:21:03.020551 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:03.397956 32209 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.398872 32209 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.399195 32276 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.399917 32276 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.401314 32209 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.402750 32276 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.404634 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:03.404805 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:03.405481 32153 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:03.405686 32078 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 1.
I20260709 12:21:03.407198 32041 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
W20260709 12:21:03.407789 32042 leader_election.cc:336] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.408030 32209 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:03.407965 32117 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:03.408769 32209 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.409194 32276 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:21:03.409228 32117 leader_election.cc:336] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.409235 32209 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.409670 32276 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.410358 32276 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.410382 32284 raft_consensus.cc:493] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.411118 32284 raft_consensus.cc:515] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.413702 32284 leader_election.cc:290] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.415345 32283 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.416033 32283 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.418829 32283 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.418823 32276 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.420811 32284 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.420956 32153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:03.422294 32276 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.418829 32209 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.422813 32284 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.423116 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:03.423885 32078 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 1.
I20260709 12:21:03.423916 32153 raft_consensus.cc:2468] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:03.424588 32209 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.425134 32284 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.426560 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:03.427506 32152 raft_consensus.cc:2393] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c1e0a103e97c439aac4237ea88e62c1a in current term 2: Already voted for candidate 0cdc7c3888934515815e7f967d05ed7f in this term.
I20260709 12:21:03.427812 32041 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:03.429085 32284 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.429250 32209 raft_consensus.cc:2804] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:03.429811 32284 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.430068 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
W20260709 12:21:03.430114 32042 leader_election.cc:336] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.431169 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
W20260709 12:21:03.431718 32117 leader_election.cc:336] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.431169 32078 raft_consensus.cc:2393] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cdc7c3888934515815e7f967d05ed7f in current term 2: Already voted for candidate c1e0a103e97c439aac4237ea88e62c1a in this term.
I20260709 12:21:03.429824 32209 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.432978 32209 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.433533 32117 leader_election.cc:304] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.432204 32284 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.434553 32276 raft_consensus.cc:2804] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:03.435176 32276 raft_consensus.cc:493] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.436091 32276 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:03.434525 32042 leader_election.cc:336] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.442816 32042 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a; no voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1
W20260709 12:21:03.443836 32117 leader_election.cc:336] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.444191 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:03.444610 32117 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.445567 32284 raft_consensus.cc:2749] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:03.445915 32283 raft_consensus.cc:2749] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:03.445883 32209 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
W20260709 12:21:03.446877 32117 leader_election.cc:336] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.445883 32276 raft_consensus.cc:515] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.447606 32077 raft_consensus.cc:2393] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cdc7c3888934515815e7f967d05ed7f in current term 2: Already voted for candidate c1e0a103e97c439aac4237ea88e62c1a in this term.
I20260709 12:21:03.448923 32209 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.449810 32276 leader_election.cc:290] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.450372 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:03.451046 32152 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.453214 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:03.453953 32077 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.455258 32117 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
W20260709 12:21:03.455611 32042 leader_election.cc:336] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
W20260709 12:21:03.457298 32117 leader_election.cc:336] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.457942 32117 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.460211 32276 raft_consensus.cc:2749] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:03.461071 32117 leader_election.cc:336] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.461267 32152 raft_consensus.cc:2468] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 2.
I20260709 12:21:03.462059 32284 raft_consensus.cc:2749] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:03.463233 32077 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:03.463045 32041 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.464807 32209 raft_consensus.cc:2804] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:03.465104 32284 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.465340 32209 raft_consensus.cc:697] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:03.466817 32117 leader_election.cc:304] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.466712 32284 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.467206 32209 consensus_queue.cc:237] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.467870 32276 raft_consensus.cc:2804] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:03.469197 32284 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.470013 32276 raft_consensus.cc:697] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:03.470993 32283 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.471697 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a" is_pre_election: true
I20260709 12:21:03.471601 32283 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.472971 32077 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 1.
I20260709 12:21:03.471056 32276 consensus_queue.cc:237] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.474179 32283 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.475580 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
W20260709 12:21:03.477233 32117 leader_election.cc:336] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.479856 32152 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:03.479445 31900 catalog_manager.cc:5697] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 1 to 2, leader changed from fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 2 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:03.479074 32042 leader_election.cc:336] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.484127 32117 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.485183 32285 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:03.485819 32285 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.486294 32285 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.488940 31898 catalog_manager.cc:5697] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 1 to 2, leader changed from fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 2 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:03.491605 32041 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.492560 32283 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:03.492944 32283 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.493278 32283 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.496918 32285 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.502751 32283 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.505288 32285 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:03.507428 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
W20260709 12:21:03.507443 32117 leader_election.cc:336] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.508419 32077 raft_consensus.cc:2393] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cdc7c3888934515815e7f967d05ed7f in current term 2: Already voted for candidate c1e0a103e97c439aac4237ea88e62c1a in this term.
I20260709 12:21:03.510429 32283 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.511165 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:03.512161 32152 raft_consensus.cc:2393] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c1e0a103e97c439aac4237ea88e62c1a in current term 2: Already voted for candidate 0cdc7c3888934515815e7f967d05ed7f in this term.
I20260709 12:21:03.510030 32117 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f; no voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.513870 32285 raft_consensus.cc:2749] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:21:03.514550 32042 leader_election.cc:336] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.515058 32042 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a; no voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.515861 32283 raft_consensus.cc:2749] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:03.584414 32283 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.584991 32283 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.586973 32283 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.588249 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:03.588972 32152 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 1.
I20260709 12:21:03.590227 32041 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:03.591197 32283 raft_consensus.cc:2804] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:21:03.591555 32042 leader_election.cc:336] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.591673 32283 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcd82f4857ce4c42af9f11d5892f9ce1)
I20260709 12:21:03.592343 32283 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.599217 32283 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.601236 32283 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:03.602727 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:03.603624 32152 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:03.605679 32042 leader_election.cc:336] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:03.610527 32152 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 2.
I20260709 12:21:03.611994 32041 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:03.612793 32283 raft_consensus.cc:2804] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:03.613296 32283 raft_consensus.cc:697] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:03.614210 32283 consensus_queue.cc:237] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:03.622295 31898 catalog_manager.cc:5697] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 1 to 2, leader changed from fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 2 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:03.867341 32117 consensus_peers.cc:597] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:03.931249 32077 raft_consensus.cc:1275] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:21:03.933247 32285 consensus_queue.cc:1048] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:03.934394 32152 raft_consensus.cc:1275] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:03.936283 32283 consensus_queue.cc:1048] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:03.956004 32042 consensus_peers.cc:597] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:04.021991 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:21:04.117619 32042 consensus_peers.cc:597] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:04.168335 32152 raft_consensus.cc:1275] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:04.170354 32283 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:04.272271 32042 consensus_peers.cc:597] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:04.721210 32042 consensus_peers.cc:597] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:04.743503 32042 consensus_peers.cc:597] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:04.908460 32042 consensus_peers.cc:597] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:05.014855 32117 consensus_peers.cc:597] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:05.023527 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:05.066191 32297 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.067544 32297 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:05.071242 32297 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:05.073212 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:05.074565 32152 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 2.
I20260709 12:21:05.076472 32041 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
W20260709 12:21:05.077121 32042 leader_election.cc:336] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:05.077566 32297 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:05.078114 32297 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:05.078475 32297 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:05.087284 32297 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:05.089289 32297 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:05.090687 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:05.091636 32152 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:05.093775 32042 leader_election.cc:336] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:05.101102 32152 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:05.102880 32041 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:05.103754 32297 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.104097 32303 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:05.104712 32297 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:05.105909 32303 raft_consensus.cc:697] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:05.107261 32297 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:05.108551 32303 consensus_queue.cc:237] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:05.109398 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f" is_pre_election: true
I20260709 12:21:05.110503 32152 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 2.
I20260709 12:21:05.111672 32041 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:05.112574 32297 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:05.112936 32297 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:05.114015 32297 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:05.112926 32042 leader_election.cc:336] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:05.120476 31898 catalog_manager.cc:5697] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 1 to 3, leader changed from fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:05.126106 32297 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:05.128500 32297 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:05.130033 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:05.130851 32152 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:05.133478 32042 leader_election.cc:336] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111)
I20260709 12:21:05.141582 32152 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:05.143024 32041 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: fcd82f4857ce4c42af9f11d5892f9ce1
I20260709 12:21:05.143996 32297 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:05.144557 32297 raft_consensus.cc:697] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:05.145795 32297 consensus_queue.cc:237] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:05.155134 31898 catalog_manager.cc:5697] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 1 to 3, leader changed from fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:05.513679 32042 consensus_peers.cc:597] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:05.539629 32152 raft_consensus.cc:1275] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:05.541536 32297 consensus_queue.cc:1048] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:05.561455 32042 consensus_peers.cc:597] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:05.616528 32152 raft_consensus.cc:1275] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:05.618355 32304 consensus_queue.cc:1048] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:06.025007 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:21:06.240406 32117 consensus_peers.cc:597] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:06.470003 32042 consensus_peers.cc:597] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:06.684170 32042 consensus_peers.cc:597] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:21:06.694659 32042 consensus_peers.cc:597] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:21:07.026376 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:07.180316 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:07.190241 32315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:07.191103 32316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:07.191881 32318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:07.193039 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:07.194398 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:07.194691 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:07.194851 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599667194840 us; error 0 us; skew 500 ppm
I20260709 12:21:07.195523 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:07.199131 11310 webserver.cc:533] Webserver started at http://127.11.11.129:43297/ using document root <none> and password file <none>
I20260709 12:21:07.199697 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:07.199877 11310 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:21:07.206066 11310 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:21:07.211530 32324 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:07.212877 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 12:21:07.213261 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
format_stamp: "Formatted at 2026-07-09 12:21:00 on dist-test-slave-4qvp"
I20260709 12:21:07.213784 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:07.241314 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:07.243265 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:07.245195 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:07.250393 32331 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260709 12:21:07.299259 32042 consensus_peers.cc:597] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:21:07.322460 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:21:07.323068 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.074s	user 0.001s	sys 0.003s
I20260709 12:21:07.323470 11310 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 12:21:07.331977 32331 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
W20260709 12:21:07.336563 32117 consensus_peers.cc:597] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:21:07.337210 32042 consensus_peers.cc:597] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:21:07.347452 32331 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.348407 32331 tablet_bootstrap.cc:492] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.349116 32331 ts_tablet_manager.cc:1403] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 12:21:07.351768 32331 raft_consensus.cc:359] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.352572 32331 raft_consensus.cc:740] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.353415 32331 consensus_queue.cc:260] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.356030 32331 ts_tablet_manager.cc:1434] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:21:07.356858 32331 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.370432 32331 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.371518 32331 tablet_bootstrap.cc:492] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.372256 32331 ts_tablet_manager.cc:1403] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.004s
I20260709 12:21:07.375488 32331 raft_consensus.cc:359] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.376498 32331 raft_consensus.cc:740] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.377331 32331 consensus_queue.cc:260] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.379449 32331 ts_tablet_manager.cc:1434] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 12:21:07.380477 32331 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.387171 11310 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:21:07.387619 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.064s	user 0.055s	sys 0.008s
I20260709 12:21:07.396759 32331 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.397734 32331 tablet_bootstrap.cc:492] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.398370 32331 ts_tablet_manager.cc:1403] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.004s
I20260709 12:21:07.401868 32331 raft_consensus.cc:359] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.402554 32331 raft_consensus.cc:740] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.403337 32331 consensus_queue.cc:260] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.405614 32331 ts_tablet_manager.cc:1434] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:21:07.406709 32331 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.421161 32331 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.422551 32331 tablet_bootstrap.cc:492] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.423686 32331 ts_tablet_manager.cc:1403] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.009s
I20260709 12:21:07.427695 32331 raft_consensus.cc:359] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.428679 32331 raft_consensus.cc:740] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.429986 32331 consensus_queue.cc:260] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.432976 32331 ts_tablet_manager.cc:1434] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:21:07.434195 32331 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.456043 32331 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.457355 32331 tablet_bootstrap.cc:492] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.458262 32331 ts_tablet_manager.cc:1403] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.024s	user 0.023s	sys 0.000s
I20260709 12:21:07.460726 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:40845
I20260709 12:21:07.460860 32393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:40845 every 8 connection(s)
I20260709 12:21:07.461176 32331 raft_consensus.cc:359] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.462071 32331 raft_consensus.cc:740] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.462853 32331 consensus_queue.cc:260] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.464812 32331 ts_tablet_manager.cc:1434] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:21:07.465610 32331 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.489472 32331 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.489562 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:07.490561 32331 tablet_bootstrap.cc:492] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.494258 32331 ts_tablet_manager.cc:1403] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.029s	user 0.017s	sys 0.008s
I20260709 12:21:07.497535 32331 raft_consensus.cc:359] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.498380 32331 raft_consensus.cc:740] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.499749 32331 consensus_queue.cc:260] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.501694 32331 ts_tablet_manager.cc:1434] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:21:07.502509 32331 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.508888 32368 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.522696 32394 heartbeater.cc:344] Connected to a master server at 127.11.11.190:43251
I20260709 12:21:07.523401 32394 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:07.524657 32394 heartbeater.cc:507] Master 127.11.11.190:43251 requested a full tablet report, sending...
I20260709 12:21:07.526726 32331 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.528311 32331 tablet_bootstrap.cc:492] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.529062 32331 ts_tablet_manager.cc:1403] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.027s	user 0.020s	sys 0.004s
I20260709 12:21:07.531826 32331 raft_consensus.cc:359] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.532418 32331 raft_consensus.cc:740] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.533365 32331 consensus_queue.cc:260] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.535312 32331 ts_tablet_manager.cc:1434] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.004s
I20260709 12:21:07.536355 32331 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.544410 32078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:07.545573 32078 raft_consensus.cc:606] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:07.549945 32078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a4a219354da4c82859196f969a75f56"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:07.550616 32078 raft_consensus.cc:606] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:07.554227 32078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:07.554916 32078 raft_consensus.cc:606] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:07.555691 31900 ts_manager.cc:194] Re-registered known tserver with Master: fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:07.558362 32078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:07.559151 32331 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.559194 32078 raft_consensus.cc:606] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:07.560305 32331 tablet_bootstrap.cc:492] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.560948 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:21:07.561273 32331 ts_tablet_manager.cc:1403] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.025s	user 0.017s	sys 0.004s
I20260709 12:21:07.561908 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:07.562991 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:07.564569 32331 raft_consensus.cc:359] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:07.565518 32331 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.566417 32331 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:07.567407 31900 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51322
I20260709 12:21:07.569085 32331 ts_tablet_manager.cc:1434] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:21:07.570225 32331 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.571386 32394 heartbeater.cc:499] Master 127.11.11.190:43251 was elected leader, sending a full tablet report...
I20260709 12:21:07.586128 32331 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:21:07.587229 32331 tablet_bootstrap.cc:492] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.587898 32331 ts_tablet_manager.cc:1403] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.006s
I20260709 12:21:07.590744 32331 raft_consensus.cc:359] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.591480 32331 raft_consensus.cc:740] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.592295 32331 consensus_queue.cc:260] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.594086 32331 ts_tablet_manager.cc:1434] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.005s
I20260709 12:21:07.595099 32331 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap starting.
I20260709 12:21:07.611411 32331 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:07.612260 32331 tablet_bootstrap.cc:492] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Bootstrap complete.
I20260709 12:21:07.612823 32331 ts_tablet_manager.cc:1403] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.000s
I20260709 12:21:07.614959 32331 raft_consensus.cc:359] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.615581 32331 raft_consensus.cc:740] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd82f4857ce4c42af9f11d5892f9ce1, State: Initialized, Role: FOLLOWER
I20260709 12:21:07.616248 32331 consensus_queue.cc:260] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.617967 32331 ts_tablet_manager.cc:1434] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:21:07.633677 32410 raft_consensus.cc:993] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:07.634325 32297 raft_consensus.cc:1081] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:07.636219 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:07.636793 32152 raft_consensus.cc:493] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:07.637081 32152 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.644404 32152 raft_consensus.cc:515] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.647704 32152 leader_election.cc:290] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:07.649722 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:07.650571 32078 raft_consensus.cc:3055] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:07.650981 32078 raft_consensus.cc:740] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:07.651990 32078 consensus_queue.cc:260] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.653609 32078 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.660650 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25c32b59fa724a38b69a0aa1221f2f8a" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:07.661543 32367 raft_consensus.cc:3060] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.663600 32078 raft_consensus.cc:2468] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 3.
I20260709 12:21:07.665499 32117 leader_election.cc:304] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:07.666798 32285 raft_consensus.cc:2804] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:07.668521 32285 raft_consensus.cc:697] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:07.669898 32285 consensus_queue.cc:237] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.670938 32367 raft_consensus.cc:2468] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 3.
I20260709 12:21:07.679224 31898 catalog_manager.cc:5697] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 2 to 3, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 3 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:07.762831 32367 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.808162 32367 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.860920 32297 raft_consensus.cc:993] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:07.861811 32420 raft_consensus.cc:1081] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:07.864343 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a4a219354da4c82859196f969a75f56"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:07.865175 32152 raft_consensus.cc:493] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:07.865545 32152 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.873425 32152 raft_consensus.cc:515] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.876986 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:07.877810 32368 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.878163 32152 leader_election.cc:290] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:07.878418 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a4a219354da4c82859196f969a75f56" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:07.879081 32078 raft_consensus.cc:3055] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:07.879443 32078 raft_consensus.cc:740] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:07.880395 32078 consensus_queue.cc:260] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.881966 32078 raft_consensus.cc:3060] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.886343 32368 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:07.887799 32117 leader_election.cc:304] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:07.888864 32285 raft_consensus.cc:2804] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:07.889442 32285 raft_consensus.cc:697] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:07.890584 32285 consensus_queue.cc:237] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.891238 32078 raft_consensus.cc:2468] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:07.902593 31898 catalog_manager.cc:5697] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 3 to 4, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 4 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:07.951419 32297 raft_consensus.cc:993] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:07.952194 32420 raft_consensus.cc:1081] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:07.954748 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:07.955471 32152 raft_consensus.cc:493] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:07.955806 32152 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.962560 32152 raft_consensus.cc:515] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.965009 32152 leader_election.cc:290] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:07.965925 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:07.966539 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:07.966753 32368 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.967316 32078 raft_consensus.cc:3055] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:07.967690 32078 raft_consensus.cc:740] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:07.968611 32078 consensus_queue.cc:260] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.969918 32078 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.973423 32368 raft_consensus.cc:2468] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:07.975049 32117 leader_election.cc:304] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:07.975975 32285 raft_consensus.cc:2804] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:07.976563 32285 raft_consensus.cc:697] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:07.976691 32078 raft_consensus.cc:2468] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:07.977700 32285 consensus_queue.cc:237] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:07.987277 31900 catalog_manager.cc:5697] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 1 to 2, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 2 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:08.014213 32297 raft_consensus.cc:993] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:08.014992 32420 raft_consensus.cc:1081] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:08.017273 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:08.017949 32152 raft_consensus.cc:493] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:08.018270 32152 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.026840 32152 raft_consensus.cc:515] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
W20260709 12:21:08.027743 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:08.028756 32152 leader_election.cc:290] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:08.029929 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:08.030128 32078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb7b01cdf344f588152a57c8c516248" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:08.030915 32078 raft_consensus.cc:3055] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:08.030951 32368 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.031466 32078 raft_consensus.cc:740] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:08.032532 32078 consensus_queue.cc:260] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:08.033948 32078 raft_consensus.cc:3060] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.039547 32368 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:08.041198 32117 leader_election.cc:304] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:08.041882 32078 raft_consensus.cc:2468] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:08.042243 32285 raft_consensus.cc:2804] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:08.042889 32285 raft_consensus.cc:697] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:08.043958 32285 consensus_queue.cc:237] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:08.051741 32368 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:08.053153 31900 catalog_manager.cc:5697] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 3 to 4, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 4 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:08.059012 32368 pending_rounds.cc:85] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Aborting all ops after (but not including) 0
I20260709 12:21:08.059464 32368 pending_rounds.cc:107] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:21:08.059846 32368 raft_consensus.cc:2889] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:08.067780 32078 raft_consensus.cc:1275] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:08.069412 32285 consensus_queue.cc:1048] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:08.097756 32368 raft_consensus.cc:1275] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:08.099536 32411 consensus_queue.cc:1048] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:08.344666 32368 raft_consensus.cc:1275] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:08.346374 32411 consensus_queue.cc:1048] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:08.348326 32078 raft_consensus.cc:1275] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:08.350216 32411 consensus_queue.cc:1048] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:08.367925 32077 raft_consensus.cc:1275] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:08.371419 32411 consensus_queue.cc:1048] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:08.387143 32368 raft_consensus.cc:1275] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:08.389667 32285 consensus_queue.cc:1048] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:08.518795 32077 raft_consensus.cc:1275] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:08.520438 32285 consensus_queue.cc:1048] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:08.534755 32368 raft_consensus.cc:1275] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:08.538156 32430 consensus_queue.cc:1048] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:08.578186 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:08.598667 32152 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
mode: GRACEFUL
new_leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:60334
I20260709 12:21:08.599263 32152 raft_consensus.cc:606] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 LEADER]: Received request to transfer leadership to TS c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:08.602535 32077 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:08.603267 32077 raft_consensus.cc:606] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:08.606482 32152 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
mode: GRACEFUL
new_leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:60334
I20260709 12:21:08.607080 32152 raft_consensus.cc:606] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Received request to transfer leadership to TS c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:08.608286 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:08.608979 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:08.609738 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:08.915625 32430 raft_consensus.cc:993] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: : Instructing follower c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:08.916337 32411 raft_consensus.cc:1081] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 LEADER]: Signalling peer c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:08.918525 32077 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:45994
I20260709 12:21:08.919212 32077 raft_consensus.cc:493] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:08.919584 32077 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:08.927448 32077 raft_consensus.cc:515] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:08.929710 32077 leader_election.cc:290] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:08.930869 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:08.931257 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:08.931602 32368 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:08.932035 32152 raft_consensus.cc:3055] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:08.932374 32152 raft_consensus.cc:740] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:08.933178 32152 consensus_queue.cc:260] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:08.934398 32152 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:08.938776 32368 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 2.
I20260709 12:21:08.940393 32042 leader_election.cc:304] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:08.941548 32152 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 2.
I20260709 12:21:08.942250 32297 raft_consensus.cc:2804] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:08.942976 32297 raft_consensus.cc:697] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:08.944654 32297 consensus_queue.cc:237] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:08.954185 31898 catalog_manager.cc:5697] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 1 to 2, leader changed from 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 2 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:08.992393 32420 raft_consensus.cc:993] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:08.992982 32297 raft_consensus.cc:1081] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:08.994922 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:08.995478 32152 raft_consensus.cc:493] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:08.995777 32152 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.001695 32152 raft_consensus.cc:515] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.003448 32152 leader_election.cc:290] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:09.004988 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:09.005774 32368 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.005575 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:09.006700 32077 raft_consensus.cc:3055] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:09.007110 32077 raft_consensus.cc:740] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:09.008069 32077 consensus_queue.cc:260] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.009517 32077 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.013305 32368 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:09.015025 32117 leader_election.cc:304] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:09.016013 32430 raft_consensus.cc:2804] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:09.016605 32430 raft_consensus.cc:697] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:09.017315 32077 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:09.017820 32430 consensus_queue.cc:237] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.027113 31898 catalog_manager.cc:5697] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 1 to 2, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 2 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:09.029016 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:09.076504 32411 raft_consensus.cc:993] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: : Instructing follower c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:09.077132 32430 raft_consensus.cc:1081] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Signalling peer c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:09.079136 32077 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:45994
I20260709 12:21:09.079895 32077 raft_consensus.cc:493] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:09.080245 32077 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.087857 32077 raft_consensus.cc:515] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.090083 32077 leader_election.cc:290] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:09.091526 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:09.091848 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:09.092440 32368 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.092638 32152 raft_consensus.cc:3055] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:09.093178 32152 raft_consensus.cc:740] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:09.094251 32152 consensus_queue.cc:260] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.095633 32152 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.102175 32368 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:09.103082 32152 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:09.104058 32042 leader_election.cc:304] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:09.105020 32297 raft_consensus.cc:2804] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:09.105535 32297 raft_consensus.cc:697] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:09.106628 32297 consensus_queue.cc:237] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.114903 31898 catalog_manager.cc:5697] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 2 to 3, leader changed from 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:09.422901 32152 raft_consensus.cc:1275] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:09.424571 32297 consensus_queue.cc:1048] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.437888 32368 raft_consensus.cc:1275] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:09.439630 32297 consensus_queue.cc:1048] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.490114 32368 raft_consensus.cc:1275] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:09.491998 32430 consensus_queue.cc:1048] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.500725 32152 raft_consensus.cc:1275] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:09.503058 32297 consensus_queue.cc:1048] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.504920 32077 raft_consensus.cc:1275] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:09.507141 32430 consensus_queue.cc:1048] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.530898 32368 raft_consensus.cc:1275] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:09.532769 32453 consensus_queue.cc:1048] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.617285 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:09.626971 32152 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
mode: GRACEFUL
new_leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:60334
I20260709 12:21:09.627687 32152 raft_consensus.cc:606] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Received request to transfer leadership to TS c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:09.631217 32077 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:09.631841 32077 raft_consensus.cc:606] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:09.635166 32077 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:09.635958 32077 raft_consensus.cc:606] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:09.637292 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:09.637991 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:09.638639 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:09.683939 32454 raft_consensus.cc:993] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:09.684602 32453 raft_consensus.cc:1081] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:09.686390 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:09.687055 32152 raft_consensus.cc:493] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:09.687444 32152 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.693737 32152 raft_consensus.cc:515] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:09.695636 32152 leader_election.cc:290] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:09.696688 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:09.697171 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:09.697443 32077 raft_consensus.cc:3055] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:09.697846 32368 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.697849 32077 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:09.699023 32077 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:09.700425 32077 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.706494 32368 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:09.708117 32117 leader_election.cc:304] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:09.708307 32077 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 2.
I20260709 12:21:09.709391 32450 raft_consensus.cc:2804] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:09.710044 32450 raft_consensus.cc:697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:09.711051 32450 consensus_queue.cc:237] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:09.719609 31898 catalog_manager.cc:5697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 1 to 2, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 2 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:09.943554 32454 raft_consensus.cc:993] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:09.944206 32453 raft_consensus.cc:1081] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:09.946030 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:09.946511 32152 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:09.946759 32152 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.952049 32152 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.953994 32152 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:09.955401 32368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:09.955549 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:09.956167 32368 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.956270 32077 raft_consensus.cc:3055] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:09.956849 32077 raft_consensus.cc:740] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:09.957702 32077 consensus_queue.cc:260] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.958817 32077 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.963999 32368 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 3.
I20260709 12:21:09.964761 32077 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 3.
I20260709 12:21:09.965714 32117 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:09.966667 32450 raft_consensus.cc:2804] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:09.967181 32450 raft_consensus.cc:697] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:09.968106 32450 consensus_queue.cc:237] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:09.980563 31898 catalog_manager.cc:5697] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 2 to 3, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 3 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:10.030459 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:10.114594 32462 raft_consensus.cc:993] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: : Instructing follower c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:10.115437 32450 raft_consensus.cc:1081] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Signalling peer c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:10.117666 32077 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:45994
I20260709 12:21:10.118405 32077 raft_consensus.cc:493] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:10.118803 32077 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.124289 32077 raft_consensus.cc:515] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:10.126160 32077 leader_election.cc:290] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:10.127395 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:10.127755 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4428dfd2db04432a919f02ec0870f0a5" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:10.128276 32367 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.128499 32152 raft_consensus.cc:3055] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:10.128942 32152 raft_consensus.cc:740] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:10.129712 32152 consensus_queue.cc:260] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:10.130743 32152 raft_consensus.cc:3060] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.136046 32367 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:10.136471 32152 raft_consensus.cc:2468] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:10.137897 32042 leader_election.cc:304] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:10.138907 32454 raft_consensus.cc:2804] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:10.139524 32454 raft_consensus.cc:697] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:10.140553 32454 consensus_queue.cc:237] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:10.148330 31898 catalog_manager.cc:5697] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 2 to 3, leader changed from 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:10.196789 32367 raft_consensus.cc:1275] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:10.198877 32450 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:10.211509 32077 raft_consensus.cc:1275] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:10.213749 32450 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:10.446880 32367 raft_consensus.cc:1275] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:10.449010 32462 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:10.461403 32077 raft_consensus.cc:1275] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:10.464882 32450 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:10.637768 32367 raft_consensus.cc:1275] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:10.639640 32454 consensus_queue.cc:1048] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:10.646534 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:10.656097 32152 raft_consensus.cc:1275] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:10.658599 32474 consensus_queue.cc:1048] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:10.662468 32077 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:10.663213 32077 raft_consensus.cc:606] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:10.671116 32152 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
mode: GRACEFUL
new_leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:60334
I20260709 12:21:10.671878 32152 raft_consensus.cc:606] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Received request to transfer leadership to TS c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:10.675350 32152 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
mode: GRACEFUL
new_leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:60334
I20260709 12:21:10.676079 32152 raft_consensus.cc:606] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Received request to transfer leadership to TS c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:10.677717 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:10.678897 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:10.679854 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:10.703411 32450 raft_consensus.cc:993] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: : Instructing follower c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:10.704161 32462 raft_consensus.cc:1081] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Signalling peer c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:10.706413 32077 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:45994
I20260709 12:21:10.707017 32077 raft_consensus.cc:493] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:10.707429 32077 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.715643 32077 raft_consensus.cc:515] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:10.717566 32077 leader_election.cc:290] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Requested vote from peers 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905), fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:10.718910 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:10.719058 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:10.719573 32152 raft_consensus.cc:3055] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:10.719862 32367 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.720018 32152 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:10.720940 32152 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:10.722097 32152 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.727399 32367 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:10.728118 32152 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:10.729163 32042 leader_election.cc:304] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:10.730314 32474 raft_consensus.cc:2804] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:10.731002 32474 raft_consensus.cc:697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:10.732007 32474 consensus_queue.cc:237] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:10.740691 31900 catalog_manager.cc:5697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 2 to 3, leader changed from 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:10.843150 32473 raft_consensus.cc:993] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:10.843662 32474 raft_consensus.cc:1081] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:10.845479 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:10.845953 32152 raft_consensus.cc:493] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:10.846200 32152 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.851189 32152 raft_consensus.cc:515] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:10.852926 32152 leader_election.cc:290] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:10.854127 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:10.854529 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10f6298f890b4c2ebf6862c423be53cc" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:10.854766 32367 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.855278 32077 raft_consensus.cc:3055] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:10.855590 32077 raft_consensus.cc:740] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:10.856314 32077 consensus_queue.cc:260] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:10.857362 32077 raft_consensus.cc:3060] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.860844 32367 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 3.
I20260709 12:21:10.862399 32117 leader_election.cc:304] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:10.863372 32462 raft_consensus.cc:2804] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:10.863941 32462 raft_consensus.cc:697] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:10.863991 32077 raft_consensus.cc:2468] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 3.
I20260709 12:21:10.865003 32462 consensus_queue.cc:237] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:10.873020 31900 catalog_manager.cc:5697] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 2 to 3, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 3 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:11.031698 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:11.199414 32367 raft_consensus.cc:1275] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.201357 32474 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:11.213049 32152 raft_consensus.cc:1275] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.216571 32474 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:11.276839 32450 raft_consensus.cc:993] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: : Instructing follower c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:11.277508 32462 raft_consensus.cc:1081] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Signalling peer c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:11.279778 32077 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:45994
I20260709 12:21:11.280422 32077 raft_consensus.cc:493] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.280732 32077 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.288102 32077 raft_consensus.cc:515] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.290084 32077 leader_election.cc:290] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:11.291131 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:11.291261 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d25f5d835b41473c897af0968c384fe4" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:11.291942 32367 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.292042 32152 raft_consensus.cc:3055] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:11.292514 32152 raft_consensus.cc:740] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:11.293185 32152 consensus_queue.cc:260] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.294143 32152 raft_consensus.cc:3060] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.298779 32367 raft_consensus.cc:2468] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:11.299292 32152 raft_consensus.cc:2468] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 3.
I20260709 12:21:11.300323 32042 leader_election.cc:304] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1e0a103e97c439aac4237ea88e62c1a, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:11.301292 32473 raft_consensus.cc:2804] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:11.301775 32473 raft_consensus.cc:697] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:11.302532 32473 consensus_queue.cc:237] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.311375 31898 catalog_manager.cc:5697] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 2 to 3, leader changed from 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.358162 32077 raft_consensus.cc:1275] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.359644 32462 consensus_queue.cc:1048] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:11.371904 32367 raft_consensus.cc:1275] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.373548 32285 consensus_queue.cc:1048] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:11.688665 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:11.697959 32077 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:11.698920 32077 raft_consensus.cc:606] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:11.703109 32077 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
mode: GRACEFUL
new_leader_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:46018
I20260709 12:21:11.703944 32077 raft_consensus.cc:606] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Received request to transfer leadership to TS 0cdc7c3888934515815e7f967d05ed7f
I20260709 12:21:11.707691 32152 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
mode: GRACEFUL
new_leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:60334
I20260709 12:21:11.708670 32152 raft_consensus.cc:606] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Received request to transfer leadership to TS c1e0a103e97c439aac4237ea88e62c1a
I20260709 12:21:11.710618 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:21:11.711472 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:11.712345 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:11.739019 32474 raft_consensus.cc:993] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:11.739910 32473 raft_consensus.cc:1081] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:11.743899 32153 raft_consensus.cc:1275] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.742237 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:11.745710 32152 raft_consensus.cc:493] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.746179 32152 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.746043 32454 consensus_queue.cc:1048] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:11.754973 32152 raft_consensus.cc:515] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.757917 32152 leader_election.cc:290] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757)
I20260709 12:21:11.760969 32368 raft_consensus.cc:1275] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:11.762290 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:11.764019 32077 raft_consensus.cc:3055] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:11.764376 32077 raft_consensus.cc:740] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:11.766229 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca77a615a159489bb88babe373bdfb23" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:11.767040 32367 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.769825 32474 consensus_queue.cc:1048] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:11.765527 32077 consensus_queue.cc:260] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.772675 32077 raft_consensus.cc:3060] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.782213 32077 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:11.783812 32117 leader_election.cc:304] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:11.785024 32462 raft_consensus.cc:2804] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.785595 32462 raft_consensus.cc:697] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:11.786754 32462 consensus_queue.cc:237] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.793012 32367 raft_consensus.cc:2468] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:11.797837 31900 catalog_manager.cc:5697] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 3 to 4, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 4 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:11.886955 32285 raft_consensus.cc:993] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: : Instructing follower c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:11.887669 32462 raft_consensus.cc:1081] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Signalling peer c1e0a103e97c439aac4237ea88e62c1a to start an election
I20260709 12:21:11.889998 32077 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55"
dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
 from {username='slave'} at 127.0.0.1:45994
I20260709 12:21:11.890640 32077 raft_consensus.cc:493] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.890969 32077 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.897161 32077 raft_consensus.cc:515] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.899228 32077 leader_election.cc:290] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 4 election: Requested vote from peers fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845), 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131:33905)
I20260709 12:21:11.900650 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:11.900676 32152 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6ea91a8dcd0496e8c9292943bec8a55" candidate_uuid: "c1e0a103e97c439aac4237ea88e62c1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
I20260709 12:21:11.901410 32367 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.901557 32152 raft_consensus.cc:3055] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:11.902149 32152 raft_consensus.cc:740] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:11.902892 32152 consensus_queue.cc:260] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.903890 32152 raft_consensus.cc:3060] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.910245 32152 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 4.
I20260709 12:21:11.910318 32367 raft_consensus.cc:2468] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e0a103e97c439aac4237ea88e62c1a in term 4.
I20260709 12:21:11.911738 32041 leader_election.cc:304] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, c1e0a103e97c439aac4237ea88e62c1a; no voters: 
I20260709 12:21:11.912774 32474 raft_consensus.cc:2804] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.913551 32474 raft_consensus.cc:697] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 4 LEADER]: Becoming Leader. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:11.914546 32474 consensus_queue.cc:237] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } }
I20260709 12:21:11.923842 31900 catalog_manager.cc:5697] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a reported cstate change: term changed from 3 to 4, leader changed from 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131) to c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130). New cstate: current_term: 4 leader_uuid: "c1e0a103e97c439aac4237ea88e62c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:12.032946 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:21:12.220777 32473 raft_consensus.cc:993] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: : Instructing follower 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:12.221513 32474 raft_consensus.cc:1081] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Signalling peer 0cdc7c3888934515815e7f967d05ed7f to start an election
I20260709 12:21:12.223410 32152 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99"
dest_uuid: "0cdc7c3888934515815e7f967d05ed7f"
 from {username='slave'} at 127.0.0.1:60314
I20260709 12:21:12.223956 32152 raft_consensus.cc:493] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:12.224244 32152 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:12.229933 32152 raft_consensus.cc:515] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:12.231488 32152 leader_election.cc:290] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Requested vote from peers c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757), fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845)
I20260709 12:21:12.232862 32077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c1e0a103e97c439aac4237ea88e62c1a"
I20260709 12:21:12.233994 32077 raft_consensus.cc:3055] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:12.233846 32367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dc970bdd4f642cdb39c7204a4a6dd99" candidate_uuid: "0cdc7c3888934515815e7f967d05ed7f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1"
I20260709 12:21:12.234525 32077 raft_consensus.cc:740] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c1e0a103e97c439aac4237ea88e62c1a, State: Running, Role: LEADER
I20260709 12:21:12.234714 32367 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:12.235426 32077 consensus_queue.cc:260] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:12.236763 32077 raft_consensus.cc:3060] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:12.242938 32367 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:12.244625 32077 raft_consensus.cc:2468] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdc7c3888934515815e7f967d05ed7f in term 4.
I20260709 12:21:12.244663 32117 leader_election.cc:304] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdc7c3888934515815e7f967d05ed7f, fcd82f4857ce4c42af9f11d5892f9ce1; no voters: 
I20260709 12:21:12.245893 32462 raft_consensus.cc:2804] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:12.246428 32462 raft_consensus.cc:697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Becoming Leader. State: Replica: 0cdc7c3888934515815e7f967d05ed7f, State: Running, Role: LEADER
I20260709 12:21:12.247123 32462 consensus_queue.cc:237] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } }
I20260709 12:21:12.256114 31900 catalog_manager.cc:5697] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f reported cstate change: term changed from 3 to 4, leader changed from c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130) to 0cdc7c3888934515815e7f967d05ed7f (127.11.11.131). New cstate: current_term: 4 leader_uuid: "0cdc7c3888934515815e7f967d05ed7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:12.263476 32367 raft_consensus.cc:1275] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:12.265199 32462 consensus_queue.cc:1048] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:12.278800 32077 raft_consensus.cc:1275] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:12.280853 32450 consensus_queue.cc:1048] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:12.384928 32367 raft_consensus.cc:1275] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:12.386690 32497 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:12.396986 32152 raft_consensus.cc:1275] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Refusing update from remote peer c1e0a103e97c439aac4237ea88e62c1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:12.401508 32473 consensus_queue.cc:1048] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cdc7c3888934515815e7f967d05ed7f" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:12.719650 32077 raft_consensus.cc:1275] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:12.721480 32450 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e0a103e97c439aac4237ea88e62c1a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:12.723146 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:12.735006 32367 raft_consensus.cc:1275] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Refusing update from remote peer 0cdc7c3888934515815e7f967d05ed7f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:12.740765 32501 consensus_queue.cc:1048] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd82f4857ce4c42af9f11d5892f9ce1" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:12.773604 11310 tablet_server.cc:179] TabletServer@127.11.11.129:40845 shutting down...
I20260709 12:21:12.799226 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:12.800043 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.801050 11310 raft_consensus.cc:2243] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.801757 11310 raft_consensus.cc:2272] T 9dc970bdd4f642cdb39c7204a4a6dd99 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.805111 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.805857 11310 raft_consensus.cc:2243] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.806329 11310 raft_consensus.cc:2272] T 25c32b59fa724a38b69a0aa1221f2f8a P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.808427 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.809095 11310 raft_consensus.cc:2243] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.809617 11310 raft_consensus.cc:2272] T f6ea91a8dcd0496e8c9292943bec8a55 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.812420 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.813100 11310 raft_consensus.cc:2243] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.813614 11310 raft_consensus.cc:2272] T 10f6298f890b4c2ebf6862c423be53cc P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.815864 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.816478 11310 raft_consensus.cc:2243] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.816972 11310 raft_consensus.cc:2272] T d25f5d835b41473c897af0968c384fe4 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.819056 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.819648 11310 raft_consensus.cc:2243] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.820103 11310 raft_consensus.cc:2272] T 7eb7b01cdf344f588152a57c8c516248 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.822232 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.822803 11310 raft_consensus.cc:2243] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.823323 11310 raft_consensus.cc:2272] T 4428dfd2db04432a919f02ec0870f0a5 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.825457 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.826068 11310 raft_consensus.cc:2243] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.826539 11310 raft_consensus.cc:2272] T 6a4a219354da4c82859196f969a75f56 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.828663 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.829321 11310 raft_consensus.cc:2243] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.829864 11310 raft_consensus.cc:2272] T ca77a615a159489bb88babe373bdfb23 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.832851 11310 tablet_replica.cc:333] stopping tablet replica
I20260709 12:21:12.833670 11310 raft_consensus.cc:2243] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.834281 11310 raft_consensus.cc:2272] T 1e175deb4c034842b8800ba846928c22 P fcd82f4857ce4c42af9f11d5892f9ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:12.835767 32042 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 99 similar messages]
W20260709 12:21:12.839741 32042 consensus_peers.cc:597] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:12.846423 32117 consensus_peers.cc:597] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:12.869050 11310 tablet_server.cc:196] TabletServer@127.11.11.129:40845 shutdown complete.
I20260709 12:21:12.893927 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
I20260709 12:21:12.933337 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:12.934425 11310 tablet_replica.cc:333] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.936071 11310 raft_consensus.cc:2243] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:21:12.937284 11310 raft_consensus.cc:2272] T f6ea91a8dcd0496e8c9292943bec8a55 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.941403 11310 tablet_replica.cc:333] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.942435 11310 raft_consensus.cc:2243] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.943080 11310 raft_consensus.cc:2272] T 10f6298f890b4c2ebf6862c423be53cc P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:12.944808 32042 consensus_peers.cc:597] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:12.946640 32117 consensus_peers.cc:597] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:12.946833 11310 tablet_replica.cc:333] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.947877 11310 raft_consensus.cc:2243] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:12.948933 11310 raft_consensus.cc:2272] T d25f5d835b41473c897af0968c384fe4 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.952008 11310 tablet_replica.cc:333] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.953059 11310 raft_consensus.cc:2243] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.954002 11310 raft_consensus.cc:2272] T 25c32b59fa724a38b69a0aa1221f2f8a P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.958170 11310 tablet_replica.cc:333] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.959228 11310 raft_consensus.cc:2243] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 3 LEADER]: Raft consensus shutting down.
W20260709 12:21:12.960049 32117 consensus_peers.cc:597] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:12.960626 11310 raft_consensus.cc:2272] T 4428dfd2db04432a919f02ec0870f0a5 P c1e0a103e97c439aac4237ea88e62c1a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.964003 11310 tablet_replica.cc:333] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.964921 11310 raft_consensus.cc:2243] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.965559 11310 raft_consensus.cc:2272] T 7eb7b01cdf344f588152a57c8c516248 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.968387 11310 tablet_replica.cc:333] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.969136 11310 raft_consensus.cc:2243] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.969698 11310 raft_consensus.cc:2272] T ca77a615a159489bb88babe373bdfb23 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.973851 11310 tablet_replica.cc:333] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.974606 11310 raft_consensus.cc:2243] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:12.975699 11310 raft_consensus.cc:2272] T 1e175deb4c034842b8800ba846928c22 P c1e0a103e97c439aac4237ea88e62c1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.978576 11310 tablet_replica.cc:333] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.979183 11310 raft_consensus.cc:2243] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.979723 11310 raft_consensus.cc:2272] T 6a4a219354da4c82859196f969a75f56 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:12.982229 11310 tablet_replica.cc:333] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a: stopping tablet replica
I20260709 12:21:12.982793 11310 raft_consensus.cc:2243] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:12.983448 11310 raft_consensus.cc:2272] T 9dc970bdd4f642cdb39c7204a4a6dd99 P c1e0a103e97c439aac4237ea88e62c1a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.024199 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
W20260709 12:21:13.024422 32117 consensus_peers.cc:597] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f -> Peer c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757): Couldn't send request to peer c1e0a103e97c439aac4237ea88e62c1a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:41757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:13.034266 31948 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:21:13.053879 32117 consensus_peers.cc:597] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f -> Peer c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757): Couldn't send request to peer c1e0a103e97c439aac4237ea88e62c1a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:41757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:13.060796 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
W20260709 12:21:13.075508 32117 consensus_peers.cc:597] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f -> Peer fcd82f4857ce4c42af9f11d5892f9ce1 (127.11.11.129:40845): Couldn't send request to peer fcd82f4857ce4c42af9f11d5892f9ce1. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:40845: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:13.101055 32117 consensus_peers.cc:597] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f -> Peer c1e0a103e97c439aac4237ea88e62c1a (127.11.11.130:41757): Couldn't send request to peer c1e0a103e97c439aac4237ea88e62c1a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:41757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:13.109015 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:13.110344 11310 tablet_replica.cc:333] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.111378 11310 raft_consensus.cc:2243] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.112066 11310 raft_consensus.cc:2272] T 1e175deb4c034842b8800ba846928c22 P 0cdc7c3888934515815e7f967d05ed7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.114995 11310 tablet_replica.cc:333] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.115765 11310 raft_consensus.cc:2243] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.116694 11310 raft_consensus.cc:2272] T ca77a615a159489bb88babe373bdfb23 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.119801 11310 tablet_replica.cc:333] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.120512 11310 raft_consensus.cc:2243] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.121405 11310 raft_consensus.cc:2272] T 25c32b59fa724a38b69a0aa1221f2f8a P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.123802 11310 tablet_replica.cc:333] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.124478 11310 raft_consensus.cc:2243] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.125375 11310 raft_consensus.cc:2272] T 6a4a219354da4c82859196f969a75f56 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.127763 11310 tablet_replica.cc:333] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.128408 11310 raft_consensus.cc:2243] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.128885 11310 raft_consensus.cc:2272] T f6ea91a8dcd0496e8c9292943bec8a55 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.132368 11310 tablet_replica.cc:333] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.133311 11310 raft_consensus.cc:2243] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.134433 11310 raft_consensus.cc:2272] T 9dc970bdd4f642cdb39c7204a4a6dd99 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.137583 11310 tablet_replica.cc:333] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.138777 11310 raft_consensus.cc:2243] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.139350 11310 raft_consensus.cc:2272] T d25f5d835b41473c897af0968c384fe4 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.141920 11310 tablet_replica.cc:333] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.142536 11310 raft_consensus.cc:2243] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.143499 11310 raft_consensus.cc:2272] T 10f6298f890b4c2ebf6862c423be53cc P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.146149 11310 tablet_replica.cc:333] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.146953 11310 raft_consensus.cc:2243] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:13.147473 11310 raft_consensus.cc:2272] T 4428dfd2db04432a919f02ec0870f0a5 P 0cdc7c3888934515815e7f967d05ed7f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.149717 11310 tablet_replica.cc:333] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f: stopping tablet replica
I20260709 12:21:13.150298 11310 raft_consensus.cc:2243] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.151245 11310 raft_consensus.cc:2272] T 7eb7b01cdf344f588152a57c8c516248 P 0cdc7c3888934515815e7f967d05ed7f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.188030 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:21:13.215624 11310 master.cc:562] Master@127.11.11.190:43251 shutting down...
I20260709 12:21:13.254055 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:13.254743 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:13.255096 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P c403a890f9d7443590cebc9e997b6a83: stopping tablet replica
I20260709 12:21:13.281354 11310 master.cc:584] Master@127.11.11.190:43251 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13521 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:21:13.339598 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:44043
I20260709 12:21:13.341058 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:13.347949 32509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.348138 32510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.349308 32512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:13.350346 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:13.351635 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:13.351835 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:13.352047 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599673352011 us; error 0 us; skew 500 ppm
I20260709 12:21:13.352674 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:13.356186 11310 webserver.cc:533] Webserver started at http://127.11.11.190:45839/ using document root <none> and password file <none>
I20260709 12:21:13.356796 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:13.356976 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:13.357344 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:13.358817 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "8df9789221ec4ce7b41905cff61c0f1c"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.364336 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:21:13.368862 32517 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.369907 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:21:13.370337 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "8df9789221ec4ce7b41905cff61c0f1c"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.370759 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:13.393252 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:13.394877 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:13.443531 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:44043
I20260709 12:21:13.443642 32568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:44043 every 8 connection(s)
I20260709 12:21:13.448448 32569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:13.461146 32569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c: Bootstrap starting.
I20260709 12:21:13.466997 32569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:13.472393 32569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c: No bootstrap required, opened a new log
I20260709 12:21:13.475708 32569 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df9789221ec4ce7b41905cff61c0f1c" member_type: VOTER }
I20260709 12:21:13.476188 32569 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:13.476480 32569 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df9789221ec4ce7b41905cff61c0f1c, State: Initialized, Role: FOLLOWER
I20260709 12:21:13.477190 32569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df9789221ec4ce7b41905cff61c0f1c" member_type: VOTER }
I20260709 12:21:13.477746 32569 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:21:13.477957 32569 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:21:13.478281 32569 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:13.484163 32569 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df9789221ec4ce7b41905cff61c0f1c" member_type: VOTER }
I20260709 12:21:13.484918 32569 leader_election.cc:304] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df9789221ec4ce7b41905cff61c0f1c; no voters: 
I20260709 12:21:13.486328 32569 leader_election.cc:290] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:21:13.486989 32572 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:13.488948 32572 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 1 LEADER]: Becoming Leader. State: Replica: 8df9789221ec4ce7b41905cff61c0f1c, State: Running, Role: LEADER
I20260709 12:21:13.489351 32569 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:21:13.489856 32572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df9789221ec4ce7b41905cff61c0f1c" member_type: VOTER }
I20260709 12:21:13.495630 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df9789221ec4ce7b41905cff61c0f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df9789221ec4ce7b41905cff61c0f1c" member_type: VOTER } }
I20260709 12:21:13.496330 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:13.495718 32574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df9789221ec4ce7b41905cff61c0f1c. Latest consensus state: current_term: 1 leader_uuid: "8df9789221ec4ce7b41905cff61c0f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df9789221ec4ce7b41905cff61c0f1c" member_type: VOTER } }
I20260709 12:21:13.500319 32574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:13.502519 32580 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:21:13.511871 32580 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:21:13.513332 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:21:13.523699 32580 catalog_manager.cc:1383] Generated new cluster ID: b77c2b98082447eeb74f361945dab5ba
I20260709 12:21:13.524155 32580 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:21:13.547348 32580 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:21:13.549052 32580 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:21:13.563450 32580 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c: Generated new TSK 0
I20260709 12:21:13.564570 32580 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:21:13.581112 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:13.589954 32590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.590366 32591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.593461 32593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:13.595336 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:13.596432 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:13.596640 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:13.596856 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599673596838 us; error 0 us; skew 500 ppm
I20260709 12:21:13.597580 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:13.600562 11310 webserver.cc:533] Webserver started at http://127.11.11.129:36961/ using document root <none> and password file <none>
I20260709 12:21:13.601177 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:13.601388 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:13.601768 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:13.603410 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "ddb422aa8cc04d209714d77a41414dba"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.608983 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20260709 12:21:13.614040 32598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.615226 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:21:13.615540 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "ddb422aa8cc04d209714d77a41414dba"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.615823 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:13.634361 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:13.635572 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:13.637161 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:13.639858 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:13.640115 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.640450 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:13.640599 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.691573 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:33851
I20260709 12:21:13.691749 32660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:33851 every 8 connection(s)
I20260709 12:21:13.697937 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:13.708246 32665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.708433 32666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.716624 32668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:13.716816 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:13.719010 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:13.719332 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:13.719552 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599673719525 us; error 0 us; skew 500 ppm
I20260709 12:21:13.720170 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:13.720319 32661 heartbeater.cc:344] Connected to a master server at 127.11.11.190:44043
I20260709 12:21:13.720865 32661 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:13.722132 32661 heartbeater.cc:507] Master 127.11.11.190:44043 requested a full tablet report, sending...
I20260709 12:21:13.723874 11310 webserver.cc:533] Webserver started at http://127.11.11.130:37851/ using document root <none> and password file <none>
I20260709 12:21:13.724865 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:13.725195 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:13.725584 32534 ts_manager.cc:194] Registered new tserver with Master: ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851)
I20260709 12:21:13.725589 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:13.727638 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.728240 32534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33798
I20260709 12:21:13.734622 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20260709 12:21:13.740275 32674 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.741518 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:21:13.742122 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.742540 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:13.771185 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:13.772501 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:13.774259 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:13.776841 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:13.777050 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.777392 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:13.777539 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.828956 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:43685
I20260709 12:21:13.829169 32736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:43685 every 8 connection(s)
I20260709 12:21:13.834447 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:13.843577 32740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:13.845106 32741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:13.852192 32737 heartbeater.cc:344] Connected to a master server at 127.11.11.190:44043
W20260709 12:21:13.852658 32743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:13.852871 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:13.853012 32737 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:13.854357 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:21:13.854485 32737 heartbeater.cc:507] Master 127.11.11.190:44043 requested a full tablet report, sending...
W20260709 12:21:13.854607 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:13.855140 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599673855116 us; error 0 us; skew 500 ppm
I20260709 12:21:13.855923 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:13.857973 32534 ts_manager.cc:194] Registered new tserver with Master: f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685)
I20260709 12:21:13.859328 11310 webserver.cc:533] Webserver started at http://127.11.11.131:33991/ using document root <none> and password file <none>
I20260709 12:21:13.860014 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:13.860246 32534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33804
I20260709 12:21:13.860250 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:13.861089 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:13.862875 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.869138 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:21:13.873939 32748 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.875159 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 12:21:13.875519 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
format_stamp: "Formatted at 2026-07-09 12:21:13 on dist-test-slave-4qvp"
I20260709 12:21:13.875836 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:13.896791 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:13.898334 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:13.900125 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:13.903136 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:13.903414 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.903707 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:13.903896 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:13.955651 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:37453
I20260709 12:21:13.955786   342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:37453 every 8 connection(s)
I20260709 12:21:13.976001   343 heartbeater.cc:344] Connected to a master server at 127.11.11.190:44043
I20260709 12:21:13.976800   343 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:13.977909   343 heartbeater.cc:507] Master 127.11.11.190:44043 requested a full tablet report, sending...
I20260709 12:21:13.981130 32534 ts_manager.cc:194] Registered new tserver with Master: da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:13.981940 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.022466148s
I20260709 12:21:13.983639 32534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33806
I20260709 12:21:14.018456 32534 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:14.021030 32534 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:14.083122   308 tablet_service.cc:1511] Processing CreateTablet for tablet d8a65d323b5d42fb96ea189c9681e0a3 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2d37f794735c414b9e0c2dffe1e35663]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:21:14.084834   308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a65d323b5d42fb96ea189c9681e0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.087162   307 tablet_service.cc:1511] Processing CreateTablet for tablet fd0c9bc1c4d14cf682eddf14127bcce8 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2d37f794735c414b9e0c2dffe1e35663]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:21:14.088845   307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0c9bc1c4d14cf682eddf14127bcce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.091027 32701 tablet_service.cc:1511] Processing CreateTablet for tablet fd0c9bc1c4d14cf682eddf14127bcce8 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2d37f794735c414b9e0c2dffe1e35663]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:21:14.092907 32701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0c9bc1c4d14cf682eddf14127bcce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.094574 32702 tablet_service.cc:1511] Processing CreateTablet for tablet d8a65d323b5d42fb96ea189c9681e0a3 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2d37f794735c414b9e0c2dffe1e35663]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:21:14.096371 32702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a65d323b5d42fb96ea189c9681e0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.116176 32626 tablet_service.cc:1511] Processing CreateTablet for tablet d8a65d323b5d42fb96ea189c9681e0a3 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2d37f794735c414b9e0c2dffe1e35663]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:21:14.116595 32625 tablet_service.cc:1511] Processing CreateTablet for tablet fd0c9bc1c4d14cf682eddf14127bcce8 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=2d37f794735c414b9e0c2dffe1e35663]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:21:14.118257 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a65d323b5d42fb96ea189c9681e0a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.118522   359 tablet_bootstrap.cc:492] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40: Bootstrap starting.
I20260709 12:21:14.119065 32625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0c9bc1c4d14cf682eddf14127bcce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.127952   359 tablet_bootstrap.cc:654] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.143924   362 tablet_bootstrap.cc:492] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a: Bootstrap starting.
I20260709 12:21:14.149015   359 tablet_bootstrap.cc:492] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40: No bootstrap required, opened a new log
I20260709 12:21:14.150144   359 ts_tablet_manager.cc:1403] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40: Time spent bootstrapping tablet: real 0.032s	user 0.021s	sys 0.008s
I20260709 12:21:14.151227   363 tablet_bootstrap.cc:492] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba: Bootstrap starting.
I20260709 12:21:14.155342   359 raft_consensus.cc:359] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.156418   359 raft_consensus.cc:385] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.156800   359 raft_consensus.cc:740] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.157152   362 tablet_bootstrap.cc:654] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.193197   359 consensus_queue.cc:260] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.208127   359 ts_tablet_manager.cc:1434] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40: Time spent starting tablet: real 0.057s	user 0.006s	sys 0.013s
I20260709 12:21:14.210003   362 tablet_bootstrap.cc:492] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a: No bootstrap required, opened a new log
I20260709 12:21:14.211493   362 ts_tablet_manager.cc:1403] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent bootstrapping tablet: real 0.068s	user 0.007s	sys 0.012s
I20260709 12:21:14.212873   343 heartbeater.cc:499] Master 127.11.11.190:44043 was elected leader, sending a full tablet report...
I20260709 12:21:14.213897   359 tablet_bootstrap.cc:492] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40: Bootstrap starting.
I20260709 12:21:14.206317   363 tablet_bootstrap.cc:654] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.219321   362 raft_consensus.cc:359] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.221390   362 raft_consensus.cc:385] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.221972   362 raft_consensus.cc:740] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.223399   362 consensus_queue.cc:260] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.227734 32737 heartbeater.cc:499] Master 127.11.11.190:44043 was elected leader, sending a full tablet report...
I20260709 12:21:14.228283   362 ts_tablet_manager.cc:1434] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent starting tablet: real 0.016s	user 0.016s	sys 0.000s
I20260709 12:21:14.230392   362 tablet_bootstrap.cc:492] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: Bootstrap starting.
I20260709 12:21:14.230722   359 tablet_bootstrap.cc:654] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.239686   359 tablet_bootstrap.cc:492] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40: No bootstrap required, opened a new log
I20260709 12:21:14.240459   359 ts_tablet_manager.cc:1403] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.011s
I20260709 12:21:14.246140   362 tablet_bootstrap.cc:654] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.246013   359 raft_consensus.cc:359] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.249814   359 raft_consensus.cc:385] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.253753   359 raft_consensus.cc:740] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.253130   363 tablet_bootstrap.cc:492] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba: No bootstrap required, opened a new log
I20260709 12:21:14.255277   359 consensus_queue.cc:260] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.256947   363 ts_tablet_manager.cc:1403] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba: Time spent bootstrapping tablet: real 0.106s	user 0.018s	sys 0.017s
I20260709 12:21:14.258641   359 ts_tablet_manager.cc:1434] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.000s
I20260709 12:21:14.259979   362 tablet_bootstrap.cc:492] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: No bootstrap required, opened a new log
I20260709 12:21:14.260646   362 ts_tablet_manager.cc:1403] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.014s
I20260709 12:21:14.263475   363 raft_consensus.cc:359] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.264020   362 raft_consensus.cc:359] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.264802   363 raft_consensus.cc:385] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.265077   362 raft_consensus.cc:385] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.265314   363 raft_consensus.cc:740] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.265522   362 raft_consensus.cc:740] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.266503   363 consensus_queue.cc:260] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.266805   362 consensus_queue.cc:260] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.269726   362 ts_tablet_manager.cc:1434] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.002s
I20260709 12:21:14.269977 32661 heartbeater.cc:499] Master 127.11.11.190:44043 was elected leader, sending a full tablet report...
I20260709 12:21:14.270265   363 ts_tablet_manager.cc:1434] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 12:21:14.271675   363 tablet_bootstrap.cc:492] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba: Bootstrap starting.
I20260709 12:21:14.280079   363 tablet_bootstrap.cc:654] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.287334   363 tablet_bootstrap.cc:492] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba: No bootstrap required, opened a new log
I20260709 12:21:14.287949   363 ts_tablet_manager.cc:1403] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 12:21:14.290562   363 raft_consensus.cc:359] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.291141   363 raft_consensus.cc:385] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.291450   363 raft_consensus.cc:740] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.292594   363 consensus_queue.cc:260] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.294557   363 ts_tablet_manager.cc:1434] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20260709 12:21:14.367563   370 raft_consensus.cc:493] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.368369   370 raft_consensus.cc:515] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.371450   370 leader_election.cc:290] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453), f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685)
I20260709 12:21:14.384410   366 raft_consensus.cc:493] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.385201   366 raft_consensus.cc:515] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.387873   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a65d323b5d42fb96ea189c9681e0a3" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" is_pre_election: true
I20260709 12:21:14.389001   366 leader_election.cc:290] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:14.389457 32712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a65d323b5d42fb96ea189c9681e0a3" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" is_pre_election: true
I20260709 12:21:14.389051   318 raft_consensus.cc:2468] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 0.
I20260709 12:21:14.390470 32712 raft_consensus.cc:2468] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 0.
I20260709 12:21:14.391921 32602 leader_election.cc:304] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, ddb422aa8cc04d209714d77a41414dba; no voters: 
I20260709 12:21:14.393225   370 raft_consensus.cc:2804] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:14.393859   370 raft_consensus.cc:493] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.394232   370 raft_consensus.cc:3060] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.405478 32636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb422aa8cc04d209714d77a41414dba" is_pre_election: true
I20260709 12:21:14.406098   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" is_pre_election: true
I20260709 12:21:14.406903   318 raft_consensus.cc:2468] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 0.
I20260709 12:21:14.406826 32636 raft_consensus.cc:2468] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 0.
I20260709 12:21:14.408622 32678 leader_election.cc:304] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:14.409510   366 raft_consensus.cc:2804] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:14.410012   366 raft_consensus.cc:493] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.410569   366 raft_consensus.cc:3060] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.412163   370 raft_consensus.cc:515] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.415410   370 leader_election.cc:290] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 1 election: Requested vote from peers da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453), f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685)
I20260709 12:21:14.416734   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a65d323b5d42fb96ea189c9681e0a3" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
I20260709 12:21:14.416988 32712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a65d323b5d42fb96ea189c9681e0a3" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
I20260709 12:21:14.417537   318 raft_consensus.cc:3060] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.418025 32712 raft_consensus.cc:3060] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.422632   366 raft_consensus.cc:515] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.425312   366 leader_election.cc:290] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 election: Requested vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:14.426620 32636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
I20260709 12:21:14.427547 32636 raft_consensus.cc:3060] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.427697   317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
I20260709 12:21:14.428586   317 raft_consensus.cc:3060] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.428689   318 raft_consensus.cc:2468] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 1.
I20260709 12:21:14.428820 32712 raft_consensus.cc:2468] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 1.
I20260709 12:21:14.431457 32600 leader_election.cc:304] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddb422aa8cc04d209714d77a41414dba, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:14.432710   370 raft_consensus.cc:2804] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:14.434535   370 raft_consensus.cc:697] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 1 LEADER]: Becoming Leader. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Running, Role: LEADER
I20260709 12:21:14.435827   370 consensus_queue.cc:237] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } }
I20260709 12:21:14.438088   317 raft_consensus.cc:2468] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 1.
I20260709 12:21:14.439477 32678 leader_election.cc:304] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:14.440304   366 raft_consensus.cc:2804] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:14.442238   366 raft_consensus.cc:697] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Becoming Leader. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Running, Role: LEADER
I20260709 12:21:14.442725 32636 raft_consensus.cc:2468] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 1.
I20260709 12:21:14.443524   366 consensus_queue.cc:237] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.450572 32532 catalog_manager.cc:5697] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb422aa8cc04d209714d77a41414dba (127.11.11.129). New cstate: current_term: 1 leader_uuid: "ddb422aa8cc04d209714d77a41414dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.453861 32534 catalog_manager.cc:5697] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a reported cstate change: term changed from 0 to 1, leader changed from <none> to f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130). New cstate: current_term: 1 leader_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:14.584635 32534 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33822:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:14.588230 32534 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:14.620633 32626 tablet_service.cc:1511] Processing CreateTablet for tablet 0c9a18e0ca584d119ca95f2350a56c2a (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=c25a8b51d43e4bce847bfff22306d4db]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:21:14.621770 32701 tablet_service.cc:1511] Processing CreateTablet for tablet 0c9a18e0ca584d119ca95f2350a56c2a (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=c25a8b51d43e4bce847bfff22306d4db]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:21:14.622722 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9a18e0ca584d119ca95f2350a56c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.623629 32701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9a18e0ca584d119ca95f2350a56c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.627319   307 tablet_service.cc:1511] Processing CreateTablet for tablet 0c9a18e0ca584d119ca95f2350a56c2a (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=c25a8b51d43e4bce847bfff22306d4db]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:21:14.628545 32625 tablet_service.cc:1511] Processing CreateTablet for tablet a070cbea5b3a4064b6c1f9b873a05b9d (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=c25a8b51d43e4bce847bfff22306d4db]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:21:14.629894   307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9a18e0ca584d119ca95f2350a56c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.630295 32625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a070cbea5b3a4064b6c1f9b873a05b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.630851 32702 tablet_service.cc:1511] Processing CreateTablet for tablet a070cbea5b3a4064b6c1f9b873a05b9d (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=c25a8b51d43e4bce847bfff22306d4db]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:21:14.632616 32702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a070cbea5b3a4064b6c1f9b873a05b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.634126   308 tablet_service.cc:1511] Processing CreateTablet for tablet a070cbea5b3a4064b6c1f9b873a05b9d (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=c25a8b51d43e4bce847bfff22306d4db]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:21:14.636171   308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a070cbea5b3a4064b6c1f9b873a05b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:14.662932   359 tablet_bootstrap.cc:492] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40: Bootstrap starting.
I20260709 12:21:14.670236   359 tablet_bootstrap.cc:654] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.673653   363 tablet_bootstrap.cc:492] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: Bootstrap starting.
I20260709 12:21:14.674788   362 tablet_bootstrap.cc:492] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: Bootstrap starting.
I20260709 12:21:14.685173   362 tablet_bootstrap.cc:654] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.685405   363 tablet_bootstrap.cc:654] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.691668   359 tablet_bootstrap.cc:492] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40: No bootstrap required, opened a new log
I20260709 12:21:14.692329   359 ts_tablet_manager.cc:1403] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40: Time spent bootstrapping tablet: real 0.030s	user 0.014s	sys 0.009s
I20260709 12:21:14.692646   362 tablet_bootstrap.cc:492] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: No bootstrap required, opened a new log
I20260709 12:21:14.693348   362 ts_tablet_manager.cc:1403] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.003s
I20260709 12:21:14.695123   363 tablet_bootstrap.cc:492] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: No bootstrap required, opened a new log
I20260709 12:21:14.695933   363 ts_tablet_manager.cc:1403] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.005s
I20260709 12:21:14.696249   359 raft_consensus.cc:359] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.696785   362 raft_consensus.cc:359] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.697217   359 raft_consensus.cc:385] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.697681   362 raft_consensus.cc:385] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.697822   359 raft_consensus.cc:740] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.698081   362 raft_consensus.cc:740] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.698916   359 consensus_queue.cc:260] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.699113   362 consensus_queue.cc:260] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.701750   359 ts_tablet_manager.cc:1434] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.003s
I20260709 12:21:14.701920   362 ts_tablet_manager.cc:1434] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:21:14.703181   359 tablet_bootstrap.cc:492] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: Bootstrap starting.
I20260709 12:21:14.699630   363 raft_consensus.cc:359] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.703300   362 tablet_bootstrap.cc:492] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: Bootstrap starting.
I20260709 12:21:14.703763   363 raft_consensus.cc:385] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.704226   363 raft_consensus.cc:740] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.705473   363 consensus_queue.cc:260] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.711390   363 ts_tablet_manager.cc:1434] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:21:14.712185   362 tablet_bootstrap.cc:654] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.712750   363 tablet_bootstrap.cc:492] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba: Bootstrap starting.
I20260709 12:21:14.713129   359 tablet_bootstrap.cc:654] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.723503   363 tablet_bootstrap.cc:654] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:14.726140   362 tablet_bootstrap.cc:492] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: No bootstrap required, opened a new log
I20260709 12:21:14.726284   359 tablet_bootstrap.cc:492] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: No bootstrap required, opened a new log
I20260709 12:21:14.727046   362 ts_tablet_manager.cc:1403] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent bootstrapping tablet: real 0.024s	user 0.019s	sys 0.003s
I20260709 12:21:14.727362   359 ts_tablet_manager.cc:1403] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.005s
I20260709 12:21:14.730938   362 raft_consensus.cc:359] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.732014   362 raft_consensus.cc:385] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.732359   362 raft_consensus.cc:740] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.732234   359 raft_consensus.cc:359] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.733673   359 raft_consensus.cc:385] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.734072   359 raft_consensus.cc:740] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.733932   362 consensus_queue.cc:260] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.735180   359 consensus_queue.cc:260] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.736110   363 tablet_bootstrap.cc:492] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba: No bootstrap required, opened a new log
I20260709 12:21:14.736766   363 ts_tablet_manager.cc:1403] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.008s
I20260709 12:21:14.736917   362 ts_tablet_manager.cc:1434] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:21:14.738262   359 ts_tablet_manager.cc:1434] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:21:14.740103   363 raft_consensus.cc:359] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.741042   363 raft_consensus.cc:385] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:14.741433   363 raft_consensus.cc:740] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Initialized, Role: FOLLOWER
I20260709 12:21:14.742501   363 consensus_queue.cc:260] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.745525   363 ts_tablet_manager.cc:1434] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:21:14.812899   364 raft_consensus.cc:493] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.813766   364 raft_consensus.cc:515] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.816237   364 leader_election.cc:290] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685)
I20260709 12:21:14.831092 32636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb422aa8cc04d209714d77a41414dba" is_pre_election: true
I20260709 12:21:14.831159 32712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" is_pre_election: true
I20260709 12:21:14.832041 32636 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da991b12b9f2471fb308bf6b0bf9ed40 in term 0.
I20260709 12:21:14.832291 32712 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da991b12b9f2471fb308bf6b0bf9ed40 in term 0.
I20260709 12:21:14.833685 32752 leader_election.cc:304] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, ddb422aa8cc04d209714d77a41414dba; no voters: 
I20260709 12:21:14.834781   364 raft_consensus.cc:2804] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:14.835223   364 raft_consensus.cc:493] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.835525   364 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.842255   364 raft_consensus.cc:515] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.844178   364 leader_election.cc:290] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [CANDIDATE]: Term 1 election: Requested vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685)
I20260709 12:21:14.845811 32712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
I20260709 12:21:14.845810 32636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
I20260709 12:21:14.846820 32712 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.846829 32636 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.854959   366 consensus_queue.cc:1048] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:14.855644 32712 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da991b12b9f2471fb308bf6b0bf9ed40 in term 1.
I20260709 12:21:14.855644 32636 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da991b12b9f2471fb308bf6b0bf9ed40 in term 1.
I20260709 12:21:14.857905 32752 leader_election.cc:304] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, ddb422aa8cc04d209714d77a41414dba; no voters: 
I20260709 12:21:14.859508   364 raft_consensus.cc:2804] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:14.860239   364 raft_consensus.cc:697] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 LEADER]: Becoming Leader. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Running, Role: LEADER
I20260709 12:21:14.861456   364 consensus_queue.cc:237] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.881080 32532 catalog_manager.cc:5697] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 reported cstate change: term changed from 0 to 1, leader changed from <none> to da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.892019   378 consensus_queue.cc:1048] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:14.895408   370 consensus_queue.cc:1048] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:14.914827   370 consensus_queue.cc:1048] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:14.922063   366 raft_consensus.cc:493] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.922761   366 raft_consensus.cc:515] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.925112   366 leader_election.cc:290] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:14.927871 32636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb422aa8cc04d209714d77a41414dba" is_pre_election: true
I20260709 12:21:14.928764 32636 raft_consensus.cc:2468] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 0.
I20260709 12:21:14.927174   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" is_pre_election: true
I20260709 12:21:14.930048   318 raft_consensus.cc:2468] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 0.
I20260709 12:21:14.930423 32678 leader_election.cc:304] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddb422aa8cc04d209714d77a41414dba, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:14.931993   378 raft_consensus.cc:2804] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:14.932364   378 raft_consensus.cc:493] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.932682   378 raft_consensus.cc:3060] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.942823   378 raft_consensus.cc:515] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.945554   378 leader_election.cc:290] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 election: Requested vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:14.946663 32636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
I20260709 12:21:14.946895   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
I20260709 12:21:14.947616 32636 raft_consensus.cc:3060] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.947957   318 raft_consensus.cc:3060] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:14.954763 32636 raft_consensus.cc:2468] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 1.
I20260709 12:21:14.956602 32678 leader_election.cc:304] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddb422aa8cc04d209714d77a41414dba, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:14.957804   378 raft_consensus.cc:2804] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:14.957988   318 raft_consensus.cc:2468] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 1.
I20260709 12:21:14.958447   378 raft_consensus.cc:697] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Becoming Leader. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Running, Role: LEADER
I20260709 12:21:14.959558   378 consensus_queue.cc:237] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:14.969067 32530 catalog_manager.cc:5697] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a reported cstate change: term changed from 0 to 1, leader changed from <none> to f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130). New cstate: current_term: 1 leader_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:15.006490 32712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8"
dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
mode: GRACEFUL
new_leader_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
 from {username='slave'} at 127.0.0.1:56092
I20260709 12:21:15.007288 32712 raft_consensus.cc:606] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Received request to transfer leadership to TS da991b12b9f2471fb308bf6b0bf9ed40
I20260709 12:21:15.026374   318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a"
dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
mode: GRACEFUL
new_leader_uuid: "ddb422aa8cc04d209714d77a41414dba"
 from {username='slave'} at 127.0.0.1:54498
I20260709 12:21:15.027067   318 raft_consensus.cc:606] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 LEADER]: Received request to transfer leadership to TS ddb422aa8cc04d209714d77a41414dba
I20260709 12:21:15.034444 32712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d"
dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
mode: GRACEFUL
new_leader_uuid: "ddb422aa8cc04d209714d77a41414dba"
 from {username='slave'} at 127.0.0.1:56092
I20260709 12:21:15.035073 32712 raft_consensus.cc:606] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Received request to transfer leadership to TS ddb422aa8cc04d209714d77a41414dba
I20260709 12:21:15.251837   366 raft_consensus.cc:993] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: : Instructing follower da991b12b9f2471fb308bf6b0bf9ed40 to start an election
I20260709 12:21:15.252434   366 raft_consensus.cc:1081] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Signalling peer da991b12b9f2471fb308bf6b0bf9ed40 to start an election
I20260709 12:21:15.254446   318 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8"
dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
 from {username='slave'} at 127.0.0.1:54486
I20260709 12:21:15.255084   318 raft_consensus.cc:493] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:15.255506   318 raft_consensus.cc:3060] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.256894   394 consensus_queue.cc:1048] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:15.262447   318 raft_consensus.cc:515] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.265048   318 leader_election.cc:290] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [CANDIDATE]: Term 2 election: Requested vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685)
I20260709 12:21:15.266044   364 raft_consensus.cc:993] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: : Instructing follower ddb422aa8cc04d209714d77a41414dba to start an election
I20260709 12:21:15.266182 32635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8" candidate_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
I20260709 12:21:15.267557 32635 raft_consensus.cc:3060] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.268749 32712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0c9bc1c4d14cf682eddf14127bcce8" candidate_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
I20260709 12:21:15.269500 32712 raft_consensus.cc:3055] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:15.270038 32712 raft_consensus.cc:740] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Running, Role: LEADER
I20260709 12:21:15.270900 32712 consensus_queue.cc:260] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.272069 32712 raft_consensus.cc:3060] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.266942   394 raft_consensus.cc:1081] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 LEADER]: Signalling peer ddb422aa8cc04d209714d77a41414dba to start an election
I20260709 12:21:15.278376 32634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a"
dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
 from {username='slave'} at 127.0.0.1:39788
I20260709 12:21:15.282322 32635 raft_consensus.cc:2468] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da991b12b9f2471fb308bf6b0bf9ed40 in term 2.
I20260709 12:21:15.285557 32634 raft_consensus.cc:493] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:15.286221 32634 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.286846   394 consensus_queue.cc:1048] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:15.289270 32752 leader_election.cc:304] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, ddb422aa8cc04d209714d77a41414dba; no voters: 
I20260709 12:21:15.291165   394 raft_consensus.cc:2804] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:15.291831   394 raft_consensus.cc:697] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 LEADER]: Becoming Leader. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Running, Role: LEADER
I20260709 12:21:15.290283 32712 raft_consensus.cc:2468] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da991b12b9f2471fb308bf6b0bf9ed40 in term 2.
I20260709 12:21:15.293601   394 consensus_queue.cc:237] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.296597 32634 raft_consensus.cc:515] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.302217 32532 catalog_manager.cc:5697] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 reported cstate change: term changed from 1 to 2, leader changed from f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130) to da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:15.304407 32634 leader_election.cc:290] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 2 election: Requested vote from peers f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:15.306180   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
I20260709 12:21:15.307025   318 raft_consensus.cc:3055] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:15.306896 32711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
I20260709 12:21:15.307595   318 raft_consensus.cc:740] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: da991b12b9f2471fb308bf6b0bf9ed40, State: Running, Role: LEADER
I20260709 12:21:15.307842 32711 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.313444   318 consensus_queue.cc:260] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.315240   318 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.318470 32711 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 2.
I20260709 12:21:15.320180 32600 leader_election.cc:304] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddb422aa8cc04d209714d77a41414dba, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:15.322084   370 raft_consensus.cc:2804] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:15.322760   370 raft_consensus.cc:697] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Becoming Leader. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Running, Role: LEADER
I20260709 12:21:15.323959   370 consensus_queue.cc:237] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.324391   318 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 2.
I20260709 12:21:15.332643 32534 catalog_manager.cc:5697] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba reported cstate change: term changed from 1 to 2, leader changed from da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131) to ddb422aa8cc04d209714d77a41414dba (127.11.11.129). New cstate: current_term: 2 leader_uuid: "ddb422aa8cc04d209714d77a41414dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:15.380820   366 consensus_queue.cc:1048] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:15.389803   378 raft_consensus.cc:993] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: : Instructing follower ddb422aa8cc04d209714d77a41414dba to start an election
I20260709 12:21:15.390415   378 raft_consensus.cc:1081] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Signalling peer ddb422aa8cc04d209714d77a41414dba to start an election
I20260709 12:21:15.392921 32636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d"
dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
 from {username='slave'} at 127.0.0.1:39772
I20260709 12:21:15.393590 32636 raft_consensus.cc:493] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:15.393991 32636 raft_consensus.cc:3060] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.398159   366 consensus_queue.cc:1048] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:15.403752 32636 raft_consensus.cc:515] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.405790 32634 raft_consensus.cc:1240] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Rejecting Update request from peer f220ee9f1f4946e2bcac0a48d243bc0a for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:15.413254   366 consensus_queue.cc:1059] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:15.415082   366 raft_consensus.cc:3055] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:15.415726   366 raft_consensus.cc:740] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Running, Role: LEADER
I20260709 12:21:15.417976   366 consensus_queue.cc:260] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.421003   366 raft_consensus.cc:3060] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.424539 32711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
I20260709 12:21:15.424502 32636 leader_election.cc:290] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 2 election: Requested vote from peers f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130:43685), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:15.427700   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a070cbea5b3a4064b6c1f9b873a05b9d" candidate_uuid: "ddb422aa8cc04d209714d77a41414dba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
I20260709 12:21:15.428507   318 raft_consensus.cc:3060] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:15.434899   318 raft_consensus.cc:2468] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 2.
I20260709 12:21:15.436255 32602 leader_election.cc:304] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, ddb422aa8cc04d209714d77a41414dba; no voters: 
I20260709 12:21:15.437100   370 raft_consensus.cc:2804] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:15.437619   370 raft_consensus.cc:697] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Becoming Leader. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Running, Role: LEADER
I20260709 12:21:15.438556   370 consensus_queue.cc:237] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.440156 32711 raft_consensus.cc:2468] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb422aa8cc04d209714d77a41414dba in term 2.
I20260709 12:21:15.447896 32534 catalog_manager.cc:5697] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba reported cstate change: term changed from 1 to 2, leader changed from f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130) to ddb422aa8cc04d209714d77a41414dba (127.11.11.129). New cstate: current_term: 2 leader_uuid: "ddb422aa8cc04d209714d77a41414dba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:15.584300 11310 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:21:15.586143 11310 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:21:15.586616 11310 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:21:15.601868 32634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a"
dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
mode: GRACEFUL
new_leader_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
 from {username='slave'} at 127.0.0.1:39796
I20260709 12:21:15.602649 32634 raft_consensus.cc:606] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Received request to transfer leadership to TS f220ee9f1f4946e2bcac0a48d243bc0a
I20260709 12:21:15.604242 11310 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:21:15.676481 32711 raft_consensus.cc:1275] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Refusing update from remote peer da991b12b9f2471fb308bf6b0bf9ed40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:15.678329   394 consensus_queue.cc:1048] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:15.686592 32634 raft_consensus.cc:1275] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Refusing update from remote peer da991b12b9f2471fb308bf6b0bf9ed40: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:15.689059   394 consensus_queue.cc:1048] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:15.907977 32711 raft_consensus.cc:1275] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Refusing update from remote peer ddb422aa8cc04d209714d77a41414dba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:15.909951   370 consensus_queue.cc:1048] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:15.923617   318 raft_consensus.cc:1275] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Refusing update from remote peer ddb422aa8cc04d209714d77a41414dba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:15.925501   376 consensus_queue.cc:1048] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:15.929332   317 raft_consensus.cc:1275] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Refusing update from remote peer ddb422aa8cc04d209714d77a41414dba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:15.931155   376 consensus_queue.cc:1048] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:15.942938 32711 raft_consensus.cc:1275] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Refusing update from remote peer ddb422aa8cc04d209714d77a41414dba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:15.944808   370 consensus_queue.cc:1048] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:15.953064   415 raft_consensus.cc:993] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: : Instructing follower f220ee9f1f4946e2bcac0a48d243bc0a to start an election
I20260709 12:21:15.953716   415 raft_consensus.cc:1081] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Signalling peer f220ee9f1f4946e2bcac0a48d243bc0a to start an election
I20260709 12:21:15.956112 32711 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a"
dest_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a"
 from {username='slave'} at 127.0.0.1:56086
I20260709 12:21:15.956846 32711 raft_consensus.cc:493] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:15.957191 32711 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:15.964577 32711 raft_consensus.cc:515] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.966779 32711 leader_election.cc:290] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 3 election: Requested vote from peers ddb422aa8cc04d209714d77a41414dba (127.11.11.129:33851), da991b12b9f2471fb308bf6b0bf9ed40 (127.11.11.131:37453)
I20260709 12:21:15.968063 32634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ddb422aa8cc04d209714d77a41414dba"
I20260709 12:21:15.968473   318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c9a18e0ca584d119ca95f2350a56c2a" candidate_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "da991b12b9f2471fb308bf6b0bf9ed40"
I20260709 12:21:15.968931 32634 raft_consensus.cc:3055] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:15.969348   318 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:15.969430 32634 raft_consensus.cc:740] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ddb422aa8cc04d209714d77a41414dba, State: Running, Role: LEADER
I20260709 12:21:15.970646 32634 consensus_queue.cc:260] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.971827 32634 raft_consensus.cc:3060] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:15.977769   318 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 3.
I20260709 12:21:15.979400 32678 leader_election.cc:304] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da991b12b9f2471fb308bf6b0bf9ed40, f220ee9f1f4946e2bcac0a48d243bc0a; no voters: 
I20260709 12:21:15.979857 32634 raft_consensus.cc:2468] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f220ee9f1f4946e2bcac0a48d243bc0a in term 3.
I20260709 12:21:15.980232   366 raft_consensus.cc:2804] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:15.982254   366 raft_consensus.cc:697] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 3 LEADER]: Becoming Leader. State: Replica: f220ee9f1f4946e2bcac0a48d243bc0a, State: Running, Role: LEADER
I20260709 12:21:15.983207   366 consensus_queue.cc:237] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } }
I20260709 12:21:15.991484 32534 catalog_manager.cc:5697] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a reported cstate change: term changed from 2 to 3, leader changed from ddb422aa8cc04d209714d77a41414dba (127.11.11.129) to f220ee9f1f4946e2bcac0a48d243bc0a (127.11.11.130). New cstate: current_term: 3 leader_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb422aa8cc04d209714d77a41414dba" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f220ee9f1f4946e2bcac0a48d243bc0a" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da991b12b9f2471fb308bf6b0bf9ed40" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 37453 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:16.261749 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
I20260709 12:21:16.293852 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:16.294786 11310 tablet_replica.cc:333] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba: stopping tablet replica
I20260709 12:21:16.295496 11310 raft_consensus.cc:2243] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.296072 11310 raft_consensus.cc:2272] T 0c9a18e0ca584d119ca95f2350a56c2a P ddb422aa8cc04d209714d77a41414dba [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.299175 11310 tablet_replica.cc:333] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba: stopping tablet replica
I20260709 12:21:16.299834 11310 raft_consensus.cc:2243] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:16.300861 11310 raft_consensus.cc:2272] T a070cbea5b3a4064b6c1f9b873a05b9d P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.304054 11310 tablet_replica.cc:333] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba: stopping tablet replica
I20260709 12:21:16.304713 11310 raft_consensus.cc:2243] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:16.305863 11310 raft_consensus.cc:2272] T d8a65d323b5d42fb96ea189c9681e0a3 P ddb422aa8cc04d209714d77a41414dba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.308275 11310 tablet_replica.cc:333] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba: stopping tablet replica
I20260709 12:21:16.308878 11310 raft_consensus.cc:2243] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.309381 11310 raft_consensus.cc:2272] T fd0c9bc1c4d14cf682eddf14127bcce8 P ddb422aa8cc04d209714d77a41414dba [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.338845 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
I20260709 12:21:16.357034 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
I20260709 12:21:16.388052 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:16.388888 11310 tablet_replica.cc:333] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: stopping tablet replica
I20260709 12:21:16.389676 11310 raft_consensus.cc:2243] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:16.390422 11310 pending_rounds.cc:70] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: Trying to abort 1 pending ops.
I20260709 12:21:16.390626 11310 pending_rounds.cc:77] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7305624272838520832 op_type: NO_OP noop_request { }
I20260709 12:21:16.390950 11310 raft_consensus.cc:2889] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:21:16.391283 11310 raft_consensus.cc:2272] T 0c9a18e0ca584d119ca95f2350a56c2a P f220ee9f1f4946e2bcac0a48d243bc0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.394135 11310 tablet_replica.cc:333] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a: stopping tablet replica
I20260709 12:21:16.394780 11310 raft_consensus.cc:2243] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.395269 11310 raft_consensus.cc:2272] T fd0c9bc1c4d14cf682eddf14127bcce8 P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.397795 11310 tablet_replica.cc:333] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a: stopping tablet replica
I20260709 12:21:16.398375 11310 raft_consensus.cc:2243] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.398866 11310 raft_consensus.cc:2272] T a070cbea5b3a4064b6c1f9b873a05b9d P f220ee9f1f4946e2bcac0a48d243bc0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.401286 11310 tablet_replica.cc:333] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a: stopping tablet replica
I20260709 12:21:16.401930 11310 raft_consensus.cc:2243] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.402423 11310 raft_consensus.cc:2272] T d8a65d323b5d42fb96ea189c9681e0a3 P f220ee9f1f4946e2bcac0a48d243bc0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.417496 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
I20260709 12:21:16.434533 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
I20260709 12:21:16.470218 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:16.471089 11310 tablet_replica.cc:333] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40: stopping tablet replica
I20260709 12:21:16.471962 11310 raft_consensus.cc:2243] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.472533 11310 raft_consensus.cc:2272] T 0c9a18e0ca584d119ca95f2350a56c2a P da991b12b9f2471fb308bf6b0bf9ed40 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.475781 11310 tablet_replica.cc:333] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40: stopping tablet replica
I20260709 12:21:16.476512 11310 raft_consensus.cc:2243] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.477069 11310 raft_consensus.cc:2272] T a070cbea5b3a4064b6c1f9b873a05b9d P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.485363 11310 tablet_replica.cc:333] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40: stopping tablet replica
I20260709 12:21:16.486231 11310 raft_consensus.cc:2243] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:16.486924 11310 raft_consensus.cc:2272] T d8a65d323b5d42fb96ea189c9681e0a3 P da991b12b9f2471fb308bf6b0bf9ed40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.490037 11310 tablet_replica.cc:333] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40: stopping tablet replica
I20260709 12:21:16.490664 11310 raft_consensus.cc:2243] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:16.491766 11310 raft_consensus.cc:2272] T fd0c9bc1c4d14cf682eddf14127bcce8 P da991b12b9f2471fb308bf6b0bf9ed40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.519963 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:21:16.541898 11310 master.cc:562] Master@127.11.11.190:44043 shutting down...
I20260709 12:21:16.571974 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:16.572640 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:16.572943 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8df9789221ec4ce7b41905cff61c0f1c: stopping tablet replica
I20260709 12:21:16.596917 11310 master.cc:584] Master@127.11.11.190:44043 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (3304 ms)
[----------] 6 tests from LeaderRebalancerTest (684251 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:21:16.644111 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:34009
I20260709 12:21:16.645782 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:16.652875   421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:16.653034   420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:16.655210   423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:16.656136 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:16.657281 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:16.657481 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:16.657743 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599676657726 us; error 0 us; skew 500 ppm
I20260709 12:21:16.658356 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:16.661298 11310 webserver.cc:533] Webserver started at http://127.11.11.190:33631/ using document root <none> and password file <none>
I20260709 12:21:16.661943 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:16.662154 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:16.662465 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:16.663714 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "7e6442e6ceb74de68135e8b91ebfda6e"
format_stamp: "Formatted at 2026-07-09 12:21:16 on dist-test-slave-4qvp"
I20260709 12:21:16.669005 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:21:16.673427   428 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:16.674686 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:21:16.675127 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "7e6442e6ceb74de68135e8b91ebfda6e"
format_stamp: "Formatted at 2026-07-09 12:21:16 on dist-test-slave-4qvp"
I20260709 12:21:16.675491 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:16.712230 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:16.713881 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:16.765763 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:34009
I20260709 12:21:16.765897   480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:34009 every 8 connection(s)
I20260709 12:21:16.770764   481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:16.786350   481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e: Bootstrap starting.
I20260709 12:21:16.793010   481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:16.799291   481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e: No bootstrap required, opened a new log
I20260709 12:21:16.802150   481 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" member_type: VOTER }
I20260709 12:21:16.802615   481 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:16.802824   481 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6442e6ceb74de68135e8b91ebfda6e, State: Initialized, Role: FOLLOWER
I20260709 12:21:16.803550   481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" member_type: VOTER }
I20260709 12:21:16.804047   481 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:21:16.804248   481 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:21:16.804474   481 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:16.811058   481 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" member_type: VOTER }
I20260709 12:21:16.811683   481 leader_election.cc:304] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6442e6ceb74de68135e8b91ebfda6e; no voters: 
I20260709 12:21:16.813056   481 leader_election.cc:290] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:21:16.813618   484 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:16.815636   484 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 1 LEADER]: Becoming Leader. State: Replica: 7e6442e6ceb74de68135e8b91ebfda6e, State: Running, Role: LEADER
I20260709 12:21:16.816433   484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" member_type: VOTER }
I20260709 12:21:16.816744   481 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:21:16.822932   485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" member_type: VOTER } }
I20260709 12:21:16.823201   486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6442e6ceb74de68135e8b91ebfda6e. Latest consensus state: current_term: 1 leader_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6442e6ceb74de68135e8b91ebfda6e" member_type: VOTER } }
I20260709 12:21:16.824303   486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:16.824262   485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:16.828500   492 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:21:16.838478   492 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:21:16.839893 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:21:16.850469   492 catalog_manager.cc:1383] Generated new cluster ID: b5c82727e88a4e87bae62eb7e26a35de
I20260709 12:21:16.850891   492 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:21:16.886091   492 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:21:16.887768   492 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:21:16.911325   492 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e: Generated new TSK 0
I20260709 12:21:16.912168   492 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:21:16.972512 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:16.980870   502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:16.982826   503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:16.984628 11310 server_base.cc:1061] running on GCE node
W20260709 12:21:16.984825   505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:16.986147 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:16.986362 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:16.986496 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599676986485 us; error 0 us; skew 500 ppm
I20260709 12:21:16.987022 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:16.989965 11310 webserver.cc:533] Webserver started at http://127.11.11.129:46073/ using document root <none> and password file <none>
I20260709 12:21:16.990511 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:16.990684 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:16.990911 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:16.992080 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "e5c803c9290244dbbc92aaf769dbd54a"
format_stamp: "Formatted at 2026-07-09 12:21:16 on dist-test-slave-4qvp"
I20260709 12:21:16.997509 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:21:17.002111   510 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.003260 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:21:17.003676 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "e5c803c9290244dbbc92aaf769dbd54a"
format_stamp: "Formatted at 2026-07-09 12:21:16 on dist-test-slave-4qvp"
I20260709 12:21:17.004002 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:17.022267 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:17.023634 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:17.025309 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:17.027993 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:17.028286 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.028553 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:17.028745 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.078676 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:44737
I20260709 12:21:17.078907   572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:44737 every 8 connection(s)
I20260709 12:21:17.085373 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:17.094110   577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:17.095314   578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:17.098500 11310 server_base.cc:1061] running on GCE node
W20260709 12:21:17.100060   580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:17.100636   573 heartbeater.cc:344] Connected to a master server at 127.11.11.190:34009
I20260709 12:21:17.101207   573 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:17.101585 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:17.101954 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:17.102150 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599677102129 us; error 0 us; skew 500 ppm
I20260709 12:21:17.102283   573 heartbeater.cc:507] Master 127.11.11.190:34009 requested a full tablet report, sending...
I20260709 12:21:17.102965 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:17.105943   445 ts_manager.cc:194] Registered new tserver with Master: e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737)
I20260709 12:21:17.106513 11310 webserver.cc:533] Webserver started at http://127.11.11.130:42947/ using document root <none> and password file <none>
I20260709 12:21:17.107125 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:17.107299 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:17.107523 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:17.108417   445 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41098
I20260709 12:21:17.108829 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "9ebb903b7402421e9dc6bfb67d537511"
format_stamp: "Formatted at 2026-07-09 12:21:17 on dist-test-slave-4qvp"
I20260709 12:21:17.114835 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.006s
I20260709 12:21:17.119270   585 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.120551 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:21:17.120981 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "9ebb903b7402421e9dc6bfb67d537511"
format_stamp: "Formatted at 2026-07-09 12:21:17 on dist-test-slave-4qvp"
I20260709 12:21:17.121332 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:17.139582 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:17.141028 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:17.143455 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:17.146674 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:17.146929 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.147367 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:17.147567 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.206084 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:42537
I20260709 12:21:17.206194   649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:42537 every 8 connection(s)
I20260709 12:21:17.212255 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:17.221341   653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:17.222256   654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:17.226967   656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:17.228103 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:17.228116   650 heartbeater.cc:344] Connected to a master server at 127.11.11.190:34009
I20260709 12:21:17.228888   650 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:17.229586 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:17.229969 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:17.229979   650 heartbeater.cc:507] Master 127.11.11.190:34009 requested a full tablet report, sending...
I20260709 12:21:17.230302 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599677230278 us; error 0 us; skew 500 ppm
I20260709 12:21:17.231276 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:17.233212   445 ts_manager.cc:194] Registered new tserver with Master: 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:17.234460 11310 webserver.cc:533] Webserver started at http://127.11.11.131:46535/ using document root <none> and password file <none>
I20260709 12:21:17.235158 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:17.235184   445 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41104
I20260709 12:21:17.235383 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:17.235980 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:17.237689 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "27943bf6f68e4f7faeaf374de0336d54"
format_stamp: "Formatted at 2026-07-09 12:21:17 on dist-test-slave-4qvp"
I20260709 12:21:17.244179 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:21:17.249040   661 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.250496 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 12:21:17.251056 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "27943bf6f68e4f7faeaf374de0336d54"
format_stamp: "Formatted at 2026-07-09 12:21:17 on dist-test-slave-4qvp"
I20260709 12:21:17.251422 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:17.279409 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:17.280654 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:17.282261 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:17.284950 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:17.285213 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:17.285501 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:17.285709 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:21:17.342116 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:33749
I20260709 12:21:17.342519   723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:33749 every 8 connection(s)
I20260709 12:21:17.360194   724 heartbeater.cc:344] Connected to a master server at 127.11.11.190:34009
I20260709 12:21:17.360862   724 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:17.361819   724 heartbeater.cc:507] Master 127.11.11.190:34009 requested a full tablet report, sending...
I20260709 12:21:17.364631   445 ts_manager.cc:194] Registered new tserver with Master: 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:17.365576 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018878332s
I20260709 12:21:17.366452   445 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41108
I20260709 12:21:17.396935   445 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41120:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:17.399978   445 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:17.496788   537 tablet_service.cc:1511] Processing CreateTablet for tablet 603f9b51bd29455284aec5af830e344c (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:17.497654   534 tablet_service.cc:1511] Processing CreateTablet for tablet 4c94ee5bad404294983f9198e5d7eb4e (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:17.498481   537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 603f9b51bd29455284aec5af830e344c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.499146   534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c94ee5bad404294983f9198e5d7eb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.502866   535 tablet_service.cc:1511] Processing CreateTablet for tablet 4b4a66fdb4594fd5bb195b5b25c21527 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:17.503460   533 tablet_service.cc:1511] Processing CreateTablet for tablet 9ddd16927bee475eb509866236b26572 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:17.504570   535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4a66fdb4594fd5bb195b5b25c21527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.505703   536 tablet_service.cc:1511] Processing CreateTablet for tablet 243b8823106f4dfcbebb6b643b2c5844 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:17.506311   532 tablet_service.cc:1511] Processing CreateTablet for tablet 762d8bc9a888413c988f106bd58383c1 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:17.507668   536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243b8823106f4dfcbebb6b643b2c5844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.508430   532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762d8bc9a888413c988f106bd58383c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.511004   538 tablet_service.cc:1511] Processing CreateTablet for tablet a32b616ee4b84fae8eb7a251baeb9382 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:17.512779   538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32b616ee4b84fae8eb7a251baeb9382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.514051   533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ddd16927bee475eb509866236b26572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.519649   531 tablet_service.cc:1511] Processing CreateTablet for tablet 57244e68ac9d49629e202a7be0eb0226 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:17.521241   531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57244e68ac9d49629e202a7be0eb0226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.524006   530 tablet_service.cc:1511] Processing CreateTablet for tablet ce78a35bc6404be88f9fcb05fcc88fe1 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:17.525609   530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce78a35bc6404be88f9fcb05fcc88fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.535120   689 tablet_service.cc:1511] Processing CreateTablet for tablet a32b616ee4b84fae8eb7a251baeb9382 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:17.537104   689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32b616ee4b84fae8eb7a251baeb9382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.540196   614 tablet_service.cc:1511] Processing CreateTablet for tablet a32b616ee4b84fae8eb7a251baeb9382 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:17.541811   614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32b616ee4b84fae8eb7a251baeb9382. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.543192   688 tablet_service.cc:1511] Processing CreateTablet for tablet 603f9b51bd29455284aec5af830e344c (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:17.545068   688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 603f9b51bd29455284aec5af830e344c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.545362   610 tablet_service.cc:1511] Processing CreateTablet for tablet 4c94ee5bad404294983f9198e5d7eb4e (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:17.547390   610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c94ee5bad404294983f9198e5d7eb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.547240   613 tablet_service.cc:1511] Processing CreateTablet for tablet 603f9b51bd29455284aec5af830e344c (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:17.548313   687 tablet_service.cc:1511] Processing CreateTablet for tablet 243b8823106f4dfcbebb6b643b2c5844 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:17.549237   613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 603f9b51bd29455284aec5af830e344c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.551069   612 tablet_service.cc:1511] Processing CreateTablet for tablet 243b8823106f4dfcbebb6b643b2c5844 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:17.552487   686 tablet_service.cc:1511] Processing CreateTablet for tablet 4b4a66fdb4594fd5bb195b5b25c21527 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:17.554332   686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4a66fdb4594fd5bb195b5b25c21527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.554108   611 tablet_service.cc:1511] Processing CreateTablet for tablet 4b4a66fdb4594fd5bb195b5b25c21527 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:17.559144   611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4a66fdb4594fd5bb195b5b25c21527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.559224   685 tablet_service.cc:1511] Processing CreateTablet for tablet 4c94ee5bad404294983f9198e5d7eb4e (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:17.561062   685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c94ee5bad404294983f9198e5d7eb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.562155   687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243b8823106f4dfcbebb6b643b2c5844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.566111   612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 243b8823106f4dfcbebb6b643b2c5844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.567181   684 tablet_service.cc:1511] Processing CreateTablet for tablet 9ddd16927bee475eb509866236b26572 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:17.568921   684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ddd16927bee475eb509866236b26572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.576365   683 tablet_service.cc:1511] Processing CreateTablet for tablet 762d8bc9a888413c988f106bd58383c1 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:17.579083   683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762d8bc9a888413c988f106bd58383c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.586174   609 tablet_service.cc:1511] Processing CreateTablet for tablet 9ddd16927bee475eb509866236b26572 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:17.586825   608 tablet_service.cc:1511] Processing CreateTablet for tablet 762d8bc9a888413c988f106bd58383c1 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:17.587939   609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ddd16927bee475eb509866236b26572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.588655   608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 762d8bc9a888413c988f106bd58383c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.589682   607 tablet_service.cc:1511] Processing CreateTablet for tablet 57244e68ac9d49629e202a7be0eb0226 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:17.591300   607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57244e68ac9d49629e202a7be0eb0226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.592156   681 tablet_service.cc:1511] Processing CreateTablet for tablet ce78a35bc6404be88f9fcb05fcc88fe1 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:17.594271   681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce78a35bc6404be88f9fcb05fcc88fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.595028   682 tablet_service.cc:1511] Processing CreateTablet for tablet 57244e68ac9d49629e202a7be0eb0226 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:17.596639   682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57244e68ac9d49629e202a7be0eb0226. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.627205   606 tablet_service.cc:1511] Processing CreateTablet for tablet ce78a35bc6404be88f9fcb05fcc88fe1 (DEFAULT_TABLE table=test-workload [id=ffc0ca1a4ede42e0bd4032376fe48fe7]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:17.628750   606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce78a35bc6404be88f9fcb05fcc88fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:17.653396   742 tablet_bootstrap.cc:492] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:17.684521   743 tablet_bootstrap.cc:492] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:17.690654   742 tablet_bootstrap.cc:654] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.696775   744 tablet_bootstrap.cc:492] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:17.711966   744 tablet_bootstrap.cc:654] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.719072   743 tablet_bootstrap.cc:654] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.733383   744 tablet_bootstrap.cc:492] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:17.733980   744 ts_tablet_manager.cc:1403] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.007s
I20260709 12:21:17.737124   744 raft_consensus.cc:359] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.737953   744 raft_consensus.cc:385] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.738270   744 raft_consensus.cc:740] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.741149   742 tablet_bootstrap.cc:492] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:17.741811   742 ts_tablet_manager.cc:1403] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.089s	user 0.010s	sys 0.010s
I20260709 12:21:17.741792   744 consensus_queue.cc:260] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.742980   743 tablet_bootstrap.cc:492] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:17.743820   743 ts_tablet_manager.cc:1403] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.060s	user 0.008s	sys 0.013s
I20260709 12:21:17.746068   742 raft_consensus.cc:359] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.747393   742 raft_consensus.cc:385] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.747905   742 raft_consensus.cc:740] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.747596   743 raft_consensus.cc:359] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.748984   743 raft_consensus.cc:385] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.749398   743 raft_consensus.cc:740] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.749495   742 consensus_queue.cc:260] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.750501   743 consensus_queue.cc:260] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.757097   724 heartbeater.cc:499] Master 127.11.11.190:34009 was elected leader, sending a full tablet report...
I20260709 12:21:17.757503   650 heartbeater.cc:499] Master 127.11.11.190:34009 was elected leader, sending a full tablet report...
I20260709 12:21:17.757833   744 ts_tablet_manager.cc:1434] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.016s
I20260709 12:21:17.758678   573 heartbeater.cc:499] Master 127.11.11.190:34009 was elected leader, sending a full tablet report...
I20260709 12:21:17.759305   744 tablet_bootstrap.cc:492] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:17.764211   743 ts_tablet_manager.cc:1434] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.003s
I20260709 12:21:17.765901   743 tablet_bootstrap.cc:492] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:17.766647   744 tablet_bootstrap.cc:654] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.774652   742 ts_tablet_manager.cc:1434] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.008s
I20260709 12:21:17.774711   743 tablet_bootstrap.cc:654] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.776072   742 tablet_bootstrap.cc:492] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:17.777282   744 tablet_bootstrap.cc:492] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:17.778244   744 ts_tablet_manager.cc:1403] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.003s
I20260709 12:21:17.783239   744 raft_consensus.cc:359] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.784387   742 tablet_bootstrap.cc:654] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.784646   744 raft_consensus.cc:385] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.785271   744 raft_consensus.cc:740] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.786674   744 consensus_queue.cc:260] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.790736   744 ts_tablet_manager.cc:1434] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.001s
I20260709 12:21:17.792631   744 tablet_bootstrap.cc:492] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:17.803612   744 tablet_bootstrap.cc:654] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.819427   744 tablet_bootstrap.cc:492] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:17.820405   744 ts_tablet_manager.cc:1403] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.007s
I20260709 12:21:17.825954   744 raft_consensus.cc:359] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.827502   744 raft_consensus.cc:385] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.828040   744 raft_consensus.cc:740] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.829600   744 consensus_queue.cc:260] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.832324   748 raft_consensus.cc:493] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:17.832865   494 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:17.833148   748 raft_consensus.cc:515] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.860821   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
W20260709 12:21:17.863652   663 leader_election.cc:343] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:17.871117   748 leader_election.cc:290] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:17.878682   743 tablet_bootstrap.cc:492] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:17.879156   744 ts_tablet_manager.cc:1434] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.058s	user 0.011s	sys 0.000s
I20260709 12:21:17.881079   744 tablet_bootstrap.cc:492] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:17.879379   743 ts_tablet_manager.cc:1403] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.114s	user 0.051s	sys 0.050s
I20260709 12:21:17.881870   742 tablet_bootstrap.cc:492] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:17.882638   742 ts_tablet_manager.cc:1403] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.107s	user 0.042s	sys 0.058s
I20260709 12:21:17.885257   743 raft_consensus.cc:359] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:17.886404   743 raft_consensus.cc:385] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.886132   742 raft_consensus.cc:359] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:17.886879   743 raft_consensus.cc:740] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.887262   742 raft_consensus.cc:385] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.887671   742 raft_consensus.cc:740] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.888190   743 consensus_queue.cc:260] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:17.888684   742 consensus_queue.cc:260] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:17.891235   743 ts_tablet_manager.cc:1434] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:21:17.891237   742 ts_tablet_manager.cc:1434] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:21:17.892421   743 tablet_bootstrap.cc:492] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:17.892663   742 tablet_bootstrap.cc:492] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:17.898684   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:17.899715   742 tablet_bootstrap.cc:654] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.900306   743 tablet_bootstrap.cc:654] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:17.900698   664 leader_election.cc:343] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:17.901283   664 leader_election.cc:304] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54; no voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:17.903014   748 raft_consensus.cc:2749] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:17.905233   744 tablet_bootstrap.cc:654] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.907384   742 tablet_bootstrap.cc:492] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:17.907990   742 ts_tablet_manager.cc:1403] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.000s
I20260709 12:21:17.910676   742 raft_consensus.cc:359] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.911317   742 raft_consensus.cc:385] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.911623   742 raft_consensus.cc:740] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.912731   742 consensus_queue.cc:260] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.914029   743 tablet_bootstrap.cc:492] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:17.914760   743 ts_tablet_manager.cc:1403] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.000s
I20260709 12:21:17.918098   750 raft_consensus.cc:493] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.919080   750 raft_consensus.cc:515] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:17.919257   743 raft_consensus.cc:359] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.920279   743 raft_consensus.cc:385] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.920596   743 raft_consensus.cc:740] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.921758   750 leader_election.cc:290] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:17.921458   743 consensus_queue.cc:260] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.927011   742 ts_tablet_manager.cc:1434] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 12:21:17.929771   743 ts_tablet_manager.cc:1434] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260709 12:21:17.930763   744 tablet_bootstrap.cc:492] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:17.931416   744 ts_tablet_manager.cc:1403] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.051s	user 0.014s	sys 0.006s
I20260709 12:21:17.931602   743 tablet_bootstrap.cc:492] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:17.934705   744 raft_consensus.cc:359] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.935552   744 raft_consensus.cc:385] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.935916   742 tablet_bootstrap.cc:492] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:17.935838   744 raft_consensus.cc:740] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.937072   744 consensus_queue.cc:260] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.943425   742 tablet_bootstrap.cc:654] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.946611   744 ts_tablet_manager.cc:1434] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:21:17.952773   744 tablet_bootstrap.cc:492] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:17.956959   743 tablet_bootstrap.cc:654] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.956604   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:17.958088   548 raft_consensus.cc:2468] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:17.960153   587 leader_election.cc:304] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:17.961338   744 tablet_bootstrap.cc:654] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:17.961577   750 raft_consensus.cc:2804] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:17.962218   750 raft_consensus.cc:493] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.962677   750 raft_consensus.cc:3060] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:17.964320   748 raft_consensus.cc:493] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.965137   748 raft_consensus.cc:515] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.969964   743 tablet_bootstrap.cc:492] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:17.970772   743 ts_tablet_manager.cc:1403] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.040s	user 0.019s	sys 0.002s
I20260709 12:21:17.970666   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:17.971953   548 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 0.
I20260709 12:21:17.973582   663 leader_election.cc:304] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:17.975098   743 raft_consensus.cc:359] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.976197   743 raft_consensus.cc:385] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.976548   743 raft_consensus.cc:740] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.977342   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:17.977942   748 leader_election.cc:290] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:17.978546   624 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 0.
I20260709 12:21:17.978804   748 raft_consensus.cc:2804] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:17.979426   748 raft_consensus.cc:493] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.979852   748 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:17.979758   750 raft_consensus.cc:515] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:17.985026   742 tablet_bootstrap.cc:492] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:17.985716   742 ts_tablet_manager.cc:1403] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.050s	user 0.021s	sys 0.004s
I20260709 12:21:17.984743   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:17.986696   548 raft_consensus.cc:3060] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:17.989552   742 raft_consensus.cc:359] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.990595   742 raft_consensus.cc:385] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:17.990971   742 raft_consensus.cc:740] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:17.990712   750 leader_election.cc:290] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:17.992339   742 consensus_queue.cc:260] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.999209   699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:17.996714   744 tablet_bootstrap.cc:492] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:18.000450   744 ts_tablet_manager.cc:1403] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.048s	user 0.016s	sys 0.016s
I20260709 12:21:18.001544   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.003613   744 raft_consensus.cc:359] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:18.004463   744 raft_consensus.cc:385] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.004750   744 raft_consensus.cc:740] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
W20260709 12:21:18.005052   586 leader_election.cc:343] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.005662   744 consensus_queue.cc:260] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
W20260709 12:21:18.006489   586 leader_election.cc:343] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.007568   548 raft_consensus.cc:2468] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:17.977471   743 consensus_queue.cc:260] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.007551   748 raft_consensus.cc:515] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:17.999536   742 ts_tablet_manager.cc:1434] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.003s
I20260709 12:21:18.009934   742 tablet_bootstrap.cc:492] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:18.010615   748 leader_election.cc:290] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:18.012272   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:18.015590   624 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.015805   743 ts_tablet_manager.cc:1434] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260709 12:21:18.016821   742 tablet_bootstrap.cc:654] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.017102   587 leader_election.cc:304] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:18.008436   744 ts_tablet_manager.cc:1434] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:21:18.018218   750 raft_consensus.cc:2804] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.019903   744 tablet_bootstrap.cc:492] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:18.014547   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:18.022567   547 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.017845   743 tablet_bootstrap.cc:492] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:18.032641   744 tablet_bootstrap.cc:654] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.041839   743 tablet_bootstrap.cc:654] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.063699   743 tablet_bootstrap.cc:492] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:18.067498   743 ts_tablet_manager.cc:1403] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.050s	user 0.005s	sys 0.014s
I20260709 12:21:18.069962   765 raft_consensus.cc:493] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.072032   765 raft_consensus.cc:515] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.073732   743 raft_consensus.cc:359] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.075114   743 raft_consensus.cc:385] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.075529   743 raft_consensus.cc:740] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.065388   547 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 1.
I20260709 12:21:18.068130   750 raft_consensus.cc:697] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:18.078212   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ddd16927bee475eb509866236b26572" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.078786   624 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 1.
I20260709 12:21:18.078176   750 consensus_queue.cc:237] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:18.079293   548 raft_consensus.cc:2468] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:18.080250   664 leader_election.cc:304] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:18.081321   748 raft_consensus.cc:2804] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.081862   748 raft_consensus.cc:697] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:18.082554   587 leader_election.cc:304] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.082804   748 consensus_queue.cc:237] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.076745   743 consensus_queue.cc:260] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.085510   765 leader_election.cc:290] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.089078   765 raft_consensus.cc:2804] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.090278   765 raft_consensus.cc:493] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.091157   765 raft_consensus.cc:3060] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.091583   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ddd16927bee475eb509866236b26572" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
W20260709 12:21:18.095273   586 leader_election.cc:343] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.099570   743 ts_tablet_manager.cc:1434] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.009s
I20260709 12:21:18.107813   743 tablet_bootstrap.cc:492] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:18.115931   443 catalog_manager.cc:5697] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:18.117163   743 tablet_bootstrap.cc:654] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.117377   765 raft_consensus.cc:515] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.117802   445 catalog_manager.cc:5697] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:18.125610   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ddd16927bee475eb509866236b26572" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:18.126346   744 tablet_bootstrap.cc:492] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:18.126379   547 raft_consensus.cc:3060] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.127179   744 ts_tablet_manager.cc:1403] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.108s	user 0.040s	sys 0.047s
I20260709 12:21:18.127691   742 tablet_bootstrap.cc:492] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:18.128681   742 ts_tablet_manager.cc:1403] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.119s	user 0.030s	sys 0.038s
I20260709 12:21:18.130615   744 raft_consensus.cc:359] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.131529   744 raft_consensus.cc:385] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.131817   744 raft_consensus.cc:740] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.132794   744 consensus_queue.cc:260] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.134143   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ddd16927bee475eb509866236b26572" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.135608   744 ts_tablet_manager.cc:1434] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
W20260709 12:21:18.136026   586 leader_election.cc:343] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.136340   765 leader_election.cc:290] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.135504   742 raft_consensus.cc:359] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.137763   744 tablet_bootstrap.cc:492] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:18.138025   742 raft_consensus.cc:385] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.138571   742 raft_consensus.cc:740] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.139690   742 consensus_queue.cc:260] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.140136   547 raft_consensus.cc:2468] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:18.140484   743 tablet_bootstrap.cc:492] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:18.146279   743 ts_tablet_manager.cc:1403] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.039s	user 0.014s	sys 0.015s
I20260709 12:21:18.146955   587 leader_election.cc:304] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:18.147148   744 tablet_bootstrap.cc:654] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.148591   765 raft_consensus.cc:2804] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.149420   765 raft_consensus.cc:697] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:18.150246   743 raft_consensus.cc:359] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.150504   765 consensus_queue.cc:237] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.151202   743 raft_consensus.cc:385] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.151538   743 raft_consensus.cc:740] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.152432   743 consensus_queue.cc:260] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.142904   742 ts_tablet_manager.cc:1434] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.014s	user 0.012s	sys 0.000s
I20260709 12:21:18.155871   742 tablet_bootstrap.cc:492] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:18.155880   743 ts_tablet_manager.cc:1434] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:21:18.157475   743 tablet_bootstrap.cc:492] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:18.172101   743 tablet_bootstrap.cc:654] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.174183   742 tablet_bootstrap.cc:654] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.179422   744 tablet_bootstrap.cc:492] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:18.180164   744 ts_tablet_manager.cc:1403] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.043s	user 0.029s	sys 0.003s
I20260709 12:21:18.183874   744 raft_consensus.cc:359] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.184933   744 raft_consensus.cc:385] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.185313   744 raft_consensus.cc:740] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.186259   744 consensus_queue.cc:260] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.189057   744 ts_tablet_manager.cc:1434] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:21:18.190235   744 tablet_bootstrap.cc:492] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:18.197438   742 tablet_bootstrap.cc:492] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:18.197781   744 tablet_bootstrap.cc:654] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.197469   445 catalog_manager.cc:5697] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.199005   742 ts_tablet_manager.cc:1403] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.044s	user 0.013s	sys 0.010s
I20260709 12:21:18.199503   743 tablet_bootstrap.cc:492] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:18.200187   743 ts_tablet_manager.cc:1403] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.043s	user 0.012s	sys 0.010s
I20260709 12:21:18.203979   743 raft_consensus.cc:359] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.203861   742 raft_consensus.cc:359] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.205114   743 raft_consensus.cc:385] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.205425   742 raft_consensus.cc:385] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.205708   743 raft_consensus.cc:740] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.206231   742 raft_consensus.cc:740] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.207103   743 consensus_queue.cc:260] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.207545   742 consensus_queue.cc:260] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.211861   742 ts_tablet_manager.cc:1434] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:21:18.213714   742 tablet_bootstrap.cc:492] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:18.220902   744 tablet_bootstrap.cc:492] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:18.221705   744 ts_tablet_manager.cc:1403] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.032s	user 0.021s	sys 0.007s
I20260709 12:21:18.221833   742 tablet_bootstrap.cc:654] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.225353   744 raft_consensus.cc:359] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.226293   744 raft_consensus.cc:385] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.226739   744 raft_consensus.cc:740] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.228030   744 consensus_queue.cc:260] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.230485   743 ts_tablet_manager.cc:1434] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260709 12:21:18.231769   743 tablet_bootstrap.cc:492] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:18.235654   742 tablet_bootstrap.cc:492] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:18.235630   744 ts_tablet_manager.cc:1434] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 12:21:18.236591   742 ts_tablet_manager.cc:1403] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.008s
I20260709 12:21:18.237290   744 tablet_bootstrap.cc:492] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:18.237283   765 raft_consensus.cc:493] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.238205   765 raft_consensus.cc:515] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.240543   765 leader_election.cc:290] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.241677   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.241906   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.242700   698 raft_consensus.cc:2468] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:18.244057   749 raft_consensus.cc:493] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.244580   586 leader_election.cc:304] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:18.244513   742 raft_consensus.cc:359] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.246047   765 raft_consensus.cc:2804] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.247371   765 raft_consensus.cc:493] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.244848   749 raft_consensus.cc:515] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.247797   765 raft_consensus.cc:3060] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:18.246305   587 leader_election.cc:343] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.253847   743 tablet_bootstrap.cc:654] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.258944   742 raft_consensus.cc:385] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.262069   742 raft_consensus.cc:740] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.265920   749 leader_election.cc:290] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.267956   749 raft_consensus.cc:493] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.269084   749 raft_consensus.cc:515] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.270256   742 consensus_queue.cc:260] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.272483   749 leader_election.cc:290] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.277750   742 ts_tablet_manager.cc:1434] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.002s
I20260709 12:21:18.278879   744 tablet_bootstrap.cc:654] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.279687   742 tablet_bootstrap.cc:492] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:18.284443   765 raft_consensus.cc:515] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.287900   743 tablet_bootstrap.cc:492] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:18.288072   750 raft_consensus.cc:493] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.288627   743 ts_tablet_manager.cc:1403] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.057s	user 0.011s	sys 0.014s
I20260709 12:21:18.288797   750 raft_consensus.cc:515] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.290366   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:18.291152   742 tablet_bootstrap.cc:654] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:18.292558   587 leader_election.cc:343] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.293037   765 leader_election.cc:290] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.293556   748 raft_consensus.cc:493] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.294260   748 raft_consensus.cc:515] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.294811   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.295854   698 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:18.297581   586 leader_election.cc:304] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:18.297698   750 leader_election.cc:290] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.298781   750 raft_consensus.cc:2804] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.299276   750 raft_consensus.cc:493] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.299718   750 raft_consensus.cc:3060] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.300406   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.297852   699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.302155   699 raft_consensus.cc:3060] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.304997   742 tablet_bootstrap.cc:492] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:18.305365   765 raft_consensus.cc:493] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.305612   742 ts_tablet_manager.cc:1403] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.026s	user 0.020s	sys 0.000s
I20260709 12:21:18.306056   765 raft_consensus.cc:515] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.308307   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.308681   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.309176   547 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 0.
I20260709 12:21:18.305951   748 leader_election.cc:290] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:18.312497   742 raft_consensus.cc:359] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.313473   742 raft_consensus.cc:385] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.313480   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c94ee5bad404294983f9198e5d7eb4e" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.314158   699 raft_consensus.cc:2468] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:18.314427   765 leader_election.cc:290] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.314790   698 raft_consensus.cc:2468] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:18.315256   742 raft_consensus.cc:740] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.316618   663 leader_election.cc:304] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.317292   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c94ee5bad404294983f9198e5d7eb4e" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.321089   750 raft_consensus.cc:515] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.333385   742 consensus_queue.cc:260] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.333932   586 leader_election.cc:304] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.334270   744 tablet_bootstrap.cc:492] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:18.334832   547 raft_consensus.cc:2468] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:18.335345   744 ts_tablet_manager.cc:1403] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.098s	user 0.010s	sys 0.028s
I20260709 12:21:18.335791   742 ts_tablet_manager.cc:1434] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 12:21:18.335700   586 leader_election.cc:304] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:18.329787   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.337034   742 tablet_bootstrap.cc:492] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:18.337044   771 raft_consensus.cc:2804] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.337532   696 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:18.338294   771 raft_consensus.cc:697] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:18.330056   623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.339066   623 raft_consensus.cc:2468] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:18.328442   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.339396   771 consensus_queue.cc:237] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.340576   697 raft_consensus.cc:2468] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:18.331223   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.341914   513 leader_election.cc:304] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.337128   765 raft_consensus.cc:2804] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.334412   624 raft_consensus.cc:2393] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27943bf6f68e4f7faeaf374de0336d54 in current term 1: Already voted for candidate 9ebb903b7402421e9dc6bfb67d537511 in this term.
I20260709 12:21:18.342813   765 raft_consensus.cc:493] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.343340   765 raft_consensus.cc:3060] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.344988   750 leader_election.cc:290] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.345854   749 raft_consensus.cc:2804] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.346243   783 raft_consensus.cc:493] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.346372   749 raft_consensus.cc:493] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.347008   742 tablet_bootstrap.cc:654] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.346833   783 raft_consensus.cc:515] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.348253   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.349048   697 raft_consensus.cc:3060] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.349386   783 leader_election.cc:290] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.346896   749 raft_consensus.cc:3060] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.352950   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:18.353142   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.354189   547 raft_consensus.cc:3060] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.330304   743 raft_consensus.cc:359] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.364159   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.365581   743 raft_consensus.cc:385] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.366097   696 raft_consensus.cc:2468] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:18.364444   511 leader_election.cc:304] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
W20260709 12:21:18.364950   513 leader_election.cc:343] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.366096   743 raft_consensus.cc:740] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.370541   743 consensus_queue.cc:260] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.370833   511 leader_election.cc:304] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:18.379482   743 ts_tablet_manager.cc:1434] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.090s	user 0.009s	sys 0.000s
I20260709 12:21:18.381073   743 tablet_bootstrap.cc:492] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:18.384464   764 raft_consensus.cc:493] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.384265   765 raft_consensus.cc:515] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.372975   744 raft_consensus.cc:359] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.385106   764 raft_consensus.cc:515] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:18.385923   744 raft_consensus.cc:385] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.386281   744 raft_consensus.cc:740] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.387172   744 consensus_queue.cc:260] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.380811   547 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:18.376891   749 raft_consensus.cc:515] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.389793   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.390482   783 raft_consensus.cc:2764] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:21:18.392705   445 catalog_manager.cc:5697] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.394459   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c94ee5bad404294983f9198e5d7eb4e" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.396339   764 leader_election.cc:290] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:18.396870   696 raft_consensus.cc:3060] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.380402   792 raft_consensus.cc:2804] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.394833   587 leader_election.cc:304] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.399785   792 raft_consensus.cc:493] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.399832   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c94ee5bad404294983f9198e5d7eb4e" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:18.400520   771 raft_consensus.cc:2804] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.400904   548 raft_consensus.cc:3060] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.400691   792 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.402948   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.403829   698 raft_consensus.cc:3060] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.405972   771 raft_consensus.cc:697] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:18.407171   771 consensus_queue.cc:237] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.416893   696 raft_consensus.cc:2468] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:18.391419   749 leader_election.cc:290] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.395773   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:18.430219   624 raft_consensus.cc:3060] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.431321   742 tablet_bootstrap.cc:492] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:18.432029   742 ts_tablet_manager.cc:1403] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.095s	user 0.044s	sys 0.014s
I20260709 12:21:18.395100   765 leader_election.cc:290] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.417750   792 raft_consensus.cc:515] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.418205   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.381857   697 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:18.420877   783 raft_consensus.cc:493] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.433543   586 leader_election.cc:304] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:18.434161   783 raft_consensus.cc:515] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.434665   750 raft_consensus.cc:2804] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.435425   750 raft_consensus.cc:697] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:18.436492   750 consensus_queue.cc:237] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.433571   547 raft_consensus.cc:2393] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27943bf6f68e4f7faeaf374de0336d54 in current term 1: Already voted for candidate 9ebb903b7402421e9dc6bfb67d537511 in this term.
I20260709 12:21:18.428473   548 raft_consensus.cc:2468] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:18.438633   743 tablet_bootstrap.cc:654] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:18.438694   783 leader_election.cc:290] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.444329   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b4a66fdb4594fd5bb195b5b25c21527" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.443832   742 raft_consensus.cc:359] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.445147   622 raft_consensus.cc:2468] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:18.445176   742 raft_consensus.cc:385] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.445359   698 raft_consensus.cc:2468] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:18.445987   742 raft_consensus.cc:740] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.439106   748 raft_consensus.cc:2764] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:21:18.447073   511 leader_election.cc:304] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.447997   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b4a66fdb4594fd5bb195b5b25c21527" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.447127   742 consensus_queue.cc:260] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.447149   513 leader_election.cc:304] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.451148   744 ts_tablet_manager.cc:1434] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.115s	user 0.010s	sys 0.003s
I20260709 12:21:18.451622   792 leader_election.cc:290] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.452281   624 raft_consensus.cc:2468] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:18.458423   742 ts_tablet_manager.cc:1434] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.002s
I20260709 12:21:18.453145   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.460101   443 catalog_manager.cc:5697] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.473472   697 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.453886   696 raft_consensus.cc:2468] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:18.484578   748 consensus_queue.cc:1048] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.449707   783 raft_consensus.cc:2804] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.454038   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:18.455458   749 raft_consensus.cc:2804] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:18.486876   749 raft_consensus.cc:493] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.487386   749 raft_consensus.cc:3060] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:18.487920   513 leader_election.cc:343] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:18.490864   783 raft_consensus.cc:697] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:18.491883   697 raft_consensus.cc:2468] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:18.493399   511 leader_election.cc:304] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:18.495146   792 raft_consensus.cc:2804] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.495659   792 raft_consensus.cc:697] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:18.492034   783 consensus_queue.cc:237] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.496383   743 tablet_bootstrap.cc:492] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:18.496511   792 consensus_queue.cc:237] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.500800   743 ts_tablet_manager.cc:1403] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.120s	user 0.013s	sys 0.006s
I20260709 12:21:18.504050   743 raft_consensus.cc:359] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.504788   743 raft_consensus.cc:385] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:18.505053   743 raft_consensus.cc:740] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:18.507671   749 raft_consensus.cc:515] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.508129   663 leader_election.cc:304] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54; no voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.509253   764 raft_consensus.cc:3060] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.509593   743 consensus_queue.cc:260] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.513082   743 ts_tablet_manager.cc:1434] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:21:18.496022   441 catalog_manager.cc:5697] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.518703   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b4a66fdb4594fd5bb195b5b25c21527" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.519819   697 raft_consensus.cc:3060] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.521854   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b4a66fdb4594fd5bb195b5b25c21527" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:18.522671   622 raft_consensus.cc:3060] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.523175   749 leader_election.cc:290] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.523478   443 catalog_manager.cc:5697] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.526017   750 consensus_queue.cc:1048] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.529516   764 raft_consensus.cc:2749] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:18.532363   622 raft_consensus.cc:2468] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:18.532933   697 raft_consensus.cc:2468] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:18.533911   513 leader_election.cc:304] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:18.539089   783 raft_consensus.cc:2804] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:18.540100   783 raft_consensus.cc:697] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:18.541481   783 consensus_queue.cc:237] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.549450   764 consensus_queue.cc:1048] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.569921   769 consensus_queue.cc:1048] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.572307   445 catalog_manager.cc:5697] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.574761   445 catalog_manager.cc:5697] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.585582   783 raft_consensus.cc:493] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.586920   783 raft_consensus.cc:515] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.590648   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.593823   783 leader_election.cc:290] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.595409   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.596406   697 raft_consensus.cc:2393] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5c803c9290244dbbc92aaf769dbd54a in current term 1: Already voted for candidate 9ebb903b7402421e9dc6bfb67d537511 in this term.
I20260709 12:21:18.597543   511 leader_election.cc:304] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5c803c9290244dbbc92aaf769dbd54a; no voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:18.598448   783 raft_consensus.cc:3060] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.606402   783 raft_consensus.cc:2749] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:18.618561   764 raft_consensus.cc:493] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.619372   764 raft_consensus.cc:515] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.622325   764 leader_election.cc:290] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:18.622704   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ddd16927bee475eb509866236b26572" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.622975   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ddd16927bee475eb509866236b26572" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:18.623816   548 raft_consensus.cc:2393] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27943bf6f68e4f7faeaf374de0336d54 in current term 1: Already voted for candidate 9ebb903b7402421e9dc6bfb67d537511 in this term.
I20260709 12:21:18.625115   663 leader_election.cc:304] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54; no voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.626199   764 raft_consensus.cc:3060] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.628093   769 consensus_queue.cc:1048] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:18.636006   764 raft_consensus.cc:2749] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:18.646533   750 consensus_queue.cc:1048] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.649618   622 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
mode: GRACEFUL
new_leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:56200
I20260709 12:21:18.650441   622 raft_consensus.cc:606] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Received request to transfer leadership to TS e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.658674   622 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "603f9b51bd29455284aec5af830e344c"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
mode: GRACEFUL
new_leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:56200
I20260709 12:21:18.659512   622 raft_consensus.cc:606] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Received request to transfer leadership to TS e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.677734   697 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
mode: GRACEFUL
new_leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:58866
I20260709 12:21:18.678476   697 raft_consensus.cc:606] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Received request to transfer leadership to TS e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.707304   548 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
mode: GRACEFUL
new_leader_uuid: "9ebb903b7402421e9dc6bfb67d537511"
 from {username='slave'} at 127.0.0.1:59570
I20260709 12:21:18.708002   548 raft_consensus.cc:606] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Received request to transfer leadership to TS 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:18.715121   548 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
mode: GRACEFUL
new_leader_uuid: "9ebb903b7402421e9dc6bfb67d537511"
 from {username='slave'} at 127.0.0.1:59570
I20260709 12:21:18.715821   548 raft_consensus.cc:606] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Received request to transfer leadership to TS 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:18.722872   622 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
mode: GRACEFUL
new_leader_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:56200
I20260709 12:21:18.723631   622 raft_consensus.cc:606] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Received request to transfer leadership to TS 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:18.871116   750 consensus_queue.cc:1048] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:18.879737   771 raft_consensus.cc:993] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: : Instructing follower e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:18.880383   765 raft_consensus.cc:1081] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Signalling peer e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:18.882429   547 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "603f9b51bd29455284aec5af830e344c"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:59568
I20260709 12:21:18.885730   547 raft_consensus.cc:493] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:18.886319   547 raft_consensus.cc:3060] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:18.887482   765 consensus_queue.cc:1048] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:18.896549   547 raft_consensus.cc:515] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.898032   548 raft_consensus.cc:1240] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Rejecting Update request from peer 9ebb903b7402421e9dc6bfb67d537511 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:18.899413   547 leader_election.cc:290] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.900166   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:18.900967   622 raft_consensus.cc:3055] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:18.901419   622 raft_consensus.cc:740] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:18.901089   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "603f9b51bd29455284aec5af830e344c" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.902194   697 raft_consensus.cc:3060] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:18.901995   765 consensus_queue.cc:1059] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.903571   622 consensus_queue.cc:260] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.905212   622 raft_consensus.cc:3060] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:18.910223   697 raft_consensus.cc:2468] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:18.911533   622 raft_consensus.cc:2468] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:18.911870   511 leader_election.cc:304] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
W20260709 12:21:18.912240   771 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:18.913069   783 raft_consensus.cc:2804] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:18.913763   783 raft_consensus.cc:697] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:18.914762   783 consensus_queue.cc:237] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.922636   445 catalog_manager.cc:5697] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 1 to 2, leader changed from 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130) to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 2 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.936270   783 consensus_queue.cc:1048] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:18.942602   771 consensus_queue.cc:1048] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.946343   749 raft_consensus.cc:993] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: : Instructing follower 9ebb903b7402421e9dc6bfb67d537511 to start an election
I20260709 12:21:18.946940   792 raft_consensus.cc:1081] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Signalling peer 9ebb903b7402421e9dc6bfb67d537511 to start an election
I20260709 12:21:18.947587   771 raft_consensus.cc:493] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.948277   771 raft_consensus.cc:515] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.952111   771 leader_election.cc:290] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.952622   622 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
 from {username='slave'} at 127.0.0.1:56194
I20260709 12:21:18.953579   622 raft_consensus.cc:493] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:18.953697   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:18.954547   783 consensus_queue.cc:1048] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:18.954928   697 raft_consensus.cc:2393] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ebb903b7402421e9dc6bfb67d537511 in current term 1: Already voted for candidate e5c803c9290244dbbc92aaf769dbd54a in this term.
I20260709 12:21:18.955276   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:18.957757   622 raft_consensus.cc:3060] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:18.966310   587 leader_election.cc:304] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511; no voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:18.967512   771 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:18.970984   622 raft_consensus.cc:515] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.972987   769 consensus_queue.cc:1048] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:18.974936   622 leader_election.cc:290] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:18.979509   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:18.978058   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "762d8bc9a888413c988f106bd58383c1" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:18.982775   548 raft_consensus.cc:3055] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:18.983229   548 raft_consensus.cc:740] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:18.984122   548 consensus_queue.cc:260] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:18.985533   548 raft_consensus.cc:3060] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:18.998049   748 raft_consensus.cc:993] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: : Instructing follower e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:18.998786   748 raft_consensus.cc:1081] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Signalling peer e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:18.999836   548 raft_consensus.cc:2468] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 2.
I20260709 12:21:18.998108   765 raft_consensus.cc:993] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: : Instructing follower 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:19.000797   765 raft_consensus.cc:1081] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Signalling peer 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:19.001816   587 leader_election.cc:304] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:19.002549   547 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:59566
I20260709 12:21:19.002781   698 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:58848
I20260709 12:21:19.003288   547 raft_consensus.cc:493] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:19.003558   698 raft_consensus.cc:493] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:19.003803   547 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.004086   765 raft_consensus.cc:2804] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:19.004247   750 consensus_queue.cc:1048] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:19.004985   765 raft_consensus.cc:697] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:19.004079   698 raft_consensus.cc:3060] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.006120   765 consensus_queue.cc:237] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.012241   547 raft_consensus.cc:515] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.013854   783 consensus_queue.cc:1048] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:19.016664   443 catalog_manager.cc:5697] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 1 to 2, leader changed from e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129) to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:19.018702   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:19.019498   622 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.019790   698 raft_consensus.cc:515] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.027834   765 consensus_queue.cc:1048] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:19.029757   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:19.030506   624 raft_consensus.cc:3055] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:19.030838   624 raft_consensus.cc:740] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:19.032027   771 raft_consensus.cc:2749] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:19.031798   624 consensus_queue.cc:260] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.032874   698 leader_election.cc:290] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:19.033123   624 raft_consensus.cc:3060] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.035079   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce78a35bc6404be88f9fcb05fcc88fe1" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:19.035796   546 raft_consensus.cc:3060] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.036468   547 leader_election.cc:290] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:19.038887   622 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:19.043305   624 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:19.044895   664 leader_election.cc:304] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:19.046343   764 raft_consensus.cc:2804] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:19.047032   764 raft_consensus.cc:697] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:19.048221   764 consensus_queue.cc:237] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.062139   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:19.064172   698 raft_consensus.cc:3055] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:19.064617   698 raft_consensus.cc:740] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:19.066114   698 consensus_queue.cc:260] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.068217   698 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.071517   546 raft_consensus.cc:2468] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:19.057447   513 leader_election.cc:304] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:19.084796   792 raft_consensus.cc:2804] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:19.086351   792 raft_consensus.cc:697] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:19.087437   792 consensus_queue.cc:237] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.088642   698 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:19.093806   792 consensus_queue.cc:1048] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:19.100502   443 catalog_manager.cc:5697] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 1 to 2, leader changed from 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130) to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:19.104509   794 consensus_queue.cc:1048] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:19.112682   792 consensus_queue.cc:1048] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:19.115615   445 catalog_manager.cc:5697] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 1 to 2, leader changed from 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131) to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 2 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:19.118582   794 raft_consensus.cc:993] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: : Instructing follower 9ebb903b7402421e9dc6bfb67d537511 to start an election
I20260709 12:21:19.119374   794 raft_consensus.cc:1081] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Signalling peer 9ebb903b7402421e9dc6bfb67d537511 to start an election
I20260709 12:21:19.122100   622 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
 from {username='slave'} at 127.0.0.1:56194
I20260709 12:21:19.123222   622 raft_consensus.cc:493] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:19.123793   622 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.138245   622 raft_consensus.cc:515] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.139355   624 raft_consensus.cc:1240] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Rejecting Update request from peer e5c803c9290244dbbc92aaf769dbd54a for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:19.141901   792 consensus_queue.cc:1059] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:19.143340   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:19.144513   622 leader_election.cc:290] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:19.144996   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:19.145602   696 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.146737   548 raft_consensus.cc:3055] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:19.147262   548 raft_consensus.cc:740] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:19.148224   548 consensus_queue.cc:260] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.151175   548 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.154956   696 raft_consensus.cc:2468] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 2.
I20260709 12:21:19.156412   586 leader_election.cc:304] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:19.157577   771 raft_consensus.cc:2804] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:19.158208   771 raft_consensus.cc:697] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:19.159135   771 consensus_queue.cc:237] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:19.161231   548 raft_consensus.cc:2468] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 2.
W20260709 12:21:19.161919   749 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:19.168751   443 catalog_manager.cc:5697] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 1 to 2, leader changed from e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129) to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:19.183952   765 raft_consensus.cc:993] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: : Instructing follower e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:19.184494   765 raft_consensus.cc:1081] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Signalling peer e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:19.186518   548 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:59568
I20260709 12:21:19.187191   548 raft_consensus.cc:493] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:19.187531   548 raft_consensus.cc:3060] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.193892   548 raft_consensus.cc:515] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:19.196836   548 leader_election.cc:290] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Requested vote from peers 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:19.198020   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:19.198318   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32b616ee4b84fae8eb7a251baeb9382" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:19.198907   696 raft_consensus.cc:3060] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.198958   622 raft_consensus.cc:3055] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:19.199460   622 raft_consensus.cc:740] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:19.200436   622 consensus_queue.cc:260] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:19.201831   622 raft_consensus.cc:3060] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.207397   696 raft_consensus.cc:2468] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:19.209295   511 leader_election.cc:304] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:19.210405   749 raft_consensus.cc:2804] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:19.210717   622 raft_consensus.cc:2468] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:19.211144   749 raft_consensus.cc:697] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:19.212175   749 consensus_queue.cc:237] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } }
I20260709 12:21:19.221141   443 catalog_manager.cc:5697] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 1 to 2, leader changed from 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130) to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 2 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:19.306269   622 raft_consensus.cc:1275] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.308145   749 consensus_queue.cc:1048] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.319840   696 raft_consensus.cc:1275] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.321432   749 consensus_queue.cc:1048] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.464274   696 raft_consensus.cc:3060] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:19.470578   696 raft_consensus.cc:1275] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer 9ebb903b7402421e9dc6bfb67d537511: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.472249   765 consensus_queue.cc:1048] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.482192   548 raft_consensus.cc:1275] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 9ebb903b7402421e9dc6bfb67d537511: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.484090   765 consensus_queue.cc:1048] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.495152   622 raft_consensus.cc:1275] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.497023   749 consensus_queue.cc:1048] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.500917   624 raft_consensus.cc:1275] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.502832   764 consensus_queue.cc:1048] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.508560   696 raft_consensus.cc:1275] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.510370   749 consensus_queue.cc:1048] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:19.515121   548 raft_consensus.cc:1275] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.519873   764 consensus_queue.cc:1048] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.595584   548 raft_consensus.cc:1275] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 9ebb903b7402421e9dc6bfb67d537511: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.597429   765 consensus_queue.cc:1048] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.607995   696 raft_consensus.cc:1275] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer 9ebb903b7402421e9dc6bfb67d537511: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.611068   765 consensus_queue.cc:1048] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.618744   698 raft_consensus.cc:1275] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.621274   749 consensus_queue.cc:1048] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:19.632459   624 raft_consensus.cc:1275] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:19.634078   783 consensus_queue.cc:1048] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:21.761804   443 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59596:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:21.764250   443 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:21.843750   537 tablet_service.cc:1511] Processing CreateTablet for tablet 09c0b37a555e404f910d3c9b5e6224f9 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:21.845366   537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c0b37a555e404f910d3c9b5e6224f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.846382   686 tablet_service.cc:1511] Processing CreateTablet for tablet 09c0b37a555e404f910d3c9b5e6224f9 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:21.846853   534 tablet_service.cc:1511] Processing CreateTablet for tablet 763f2cf7a1fd4ddb83341c75d7579de4 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:21.848551   534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763f2cf7a1fd4ddb83341c75d7579de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.848340   607 tablet_service.cc:1511] Processing CreateTablet for tablet 09c0b37a555e404f910d3c9b5e6224f9 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:21.850418   686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c0b37a555e404f910d3c9b5e6224f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.851387   611 tablet_service.cc:1511] Processing CreateTablet for tablet 763f2cf7a1fd4ddb83341c75d7579de4 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:21.854298   610 tablet_service.cc:1511] Processing CreateTablet for tablet 66069fa7e25942cd95cc23bd7e1c51ff (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:21.854548   685 tablet_service.cc:1511] Processing CreateTablet for tablet 763f2cf7a1fd4ddb83341c75d7579de4 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:21.856199   685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763f2cf7a1fd4ddb83341c75d7579de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.856561   607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c0b37a555e404f910d3c9b5e6224f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.859810   535 tablet_service.cc:1511] Processing CreateTablet for tablet 66069fa7e25942cd95cc23bd7e1c51ff (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:21.862037   610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66069fa7e25942cd95cc23bd7e1c51ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.862933   613 tablet_service.cc:1511] Processing CreateTablet for tablet 03e67e33e2e44833bb0821377ce57149 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:21.864571   613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e67e33e2e44833bb0821377ce57149. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.866164   611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763f2cf7a1fd4ddb83341c75d7579de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.866757   608 tablet_service.cc:1511] Processing CreateTablet for tablet e511c3ec5b5147de9a48e271b24864c6 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:21.869051   608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e511c3ec5b5147de9a48e271b24864c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.872591   684 tablet_service.cc:1511] Processing CreateTablet for tablet 66069fa7e25942cd95cc23bd7e1c51ff (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:21.874493   684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66069fa7e25942cd95cc23bd7e1c51ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.877115   689 tablet_service.cc:1511] Processing CreateTablet for tablet e511c3ec5b5147de9a48e271b24864c6 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:21.879372   689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e511c3ec5b5147de9a48e271b24864c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.881464   609 tablet_service.cc:1511] Processing CreateTablet for tablet c5380597535a4ae2a7e783db3903ec50 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:21.883247   609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5380597535a4ae2a7e783db3903ec50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.883369   688 tablet_service.cc:1511] Processing CreateTablet for tablet 03e67e33e2e44833bb0821377ce57149 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
W20260709 12:21:21.887982   494 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:21.893411   682 tablet_service.cc:1511] Processing CreateTablet for tablet c5380597535a4ae2a7e783db3903ec50 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:21.898378   531 tablet_service.cc:1511] Processing CreateTablet for tablet 03e67e33e2e44833bb0821377ce57149 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:21.903100   532 tablet_service.cc:1511] Processing CreateTablet for tablet e511c3ec5b5147de9a48e271b24864c6 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:21.907639   536 tablet_service.cc:1511] Processing CreateTablet for tablet c5380597535a4ae2a7e783db3903ec50 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:21.861440   535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66069fa7e25942cd95cc23bd7e1c51ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.913803   533 tablet_service.cc:1511] Processing CreateTablet for tablet 59c0a7890a6f4eda9f0c6c4df133f0c2 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:21.915382   533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c0a7890a6f4eda9f0c6c4df133f0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.917691   681 tablet_service.cc:1511] Processing CreateTablet for tablet 59c0a7890a6f4eda9f0c6c4df133f0c2 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:21.919508   606 tablet_service.cc:1511] Processing CreateTablet for tablet 59c0a7890a6f4eda9f0c6c4df133f0c2 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:21.921118   606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c0a7890a6f4eda9f0c6c4df133f0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.926069   612 tablet_service.cc:1511] Processing CreateTablet for tablet f8d474f3a8f2484abca69e66179b2566 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:21.926760   683 tablet_service.cc:1511] Processing CreateTablet for tablet f8d474f3a8f2484abca69e66179b2566 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:21.927650   612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d474f3a8f2484abca69e66179b2566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.930100   536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5380597535a4ae2a7e783db3903ec50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.934216   531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e67e33e2e44833bb0821377ce57149. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.935598   532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e511c3ec5b5147de9a48e271b24864c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.940147   614 tablet_service.cc:1511] Processing CreateTablet for tablet c2a7937ea6a34b53a1c8b339cb02383f (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:21.940469   687 tablet_service.cc:1511] Processing CreateTablet for tablet c2a7937ea6a34b53a1c8b339cb02383f (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:21.942548   614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a7937ea6a34b53a1c8b339cb02383f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.943411   688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e67e33e2e44833bb0821377ce57149. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.944391   687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a7937ea6a34b53a1c8b339cb02383f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.945966   682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5380597535a4ae2a7e783db3903ec50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.947789   681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c0a7890a6f4eda9f0c6c4df133f0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.948542   683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d474f3a8f2484abca69e66179b2566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.982996   533 tablet_service.cc:1511] Processing CreateTablet for tablet f8d474f3a8f2484abca69e66179b2566 (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:21.984532   533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d474f3a8f2484abca69e66179b2566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:21.990404   530 tablet_service.cc:1511] Processing CreateTablet for tablet c2a7937ea6a34b53a1c8b339cb02383f (DEFAULT_TABLE table=soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:21.992000   530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a7937ea6a34b53a1c8b339cb02383f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:22.057675   840 tablet_bootstrap.cc:492] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.092669   839 tablet_bootstrap.cc:492] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.116305   842 tablet_bootstrap.cc:492] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.125020   840 tablet_bootstrap.cc:654] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.134076   842 tablet_bootstrap.cc:654] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.161720   839 tablet_bootstrap.cc:654] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.165495   840 tablet_bootstrap.cc:492] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.166229   840 ts_tablet_manager.cc:1403] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.109s	user 0.020s	sys 0.021s
I20260709 12:21:22.166281   842 tablet_bootstrap.cc:492] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.166891   842 ts_tablet_manager.cc:1403] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.051s	user 0.019s	sys 0.019s
I20260709 12:21:22.167987   839 tablet_bootstrap.cc:492] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.168573   839 ts_tablet_manager.cc:1403] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.076s	user 0.013s	sys 0.000s
I20260709 12:21:22.170284   842 raft_consensus.cc:359] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.170150   840 raft_consensus.cc:359] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.171314   842 raft_consensus.cc:385] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.171438   840 raft_consensus.cc:385] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.171844   842 raft_consensus.cc:740] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.171388   839 raft_consensus.cc:359] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.172299   840 raft_consensus.cc:740] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.172640   839 raft_consensus.cc:385] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.172940   842 consensus_queue.cc:260] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.173116   840 consensus_queue.cc:260] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.176257   840 ts_tablet_manager.cc:1434] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:21:22.173012   839 raft_consensus.cc:740] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.182243   842 ts_tablet_manager.cc:1434] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.007s
I20260709 12:21:22.183203   839 consensus_queue.cc:260] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.183645   842 tablet_bootstrap.cc:492] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.184777   840 tablet_bootstrap.cc:492] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.191816   842 tablet_bootstrap.cc:654] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.193254   840 tablet_bootstrap.cc:654] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.194415   839 ts_tablet_manager.cc:1434] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.026s	user 0.011s	sys 0.000s
I20260709 12:21:22.196039   839 tablet_bootstrap.cc:492] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.214089   839 tablet_bootstrap.cc:654] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.223295   842 tablet_bootstrap.cc:492] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.224110   842 ts_tablet_manager.cc:1403] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.041s	user 0.026s	sys 0.009s
I20260709 12:21:22.225883   840 tablet_bootstrap.cc:492] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.226570   840 ts_tablet_manager.cc:1403] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.042s	user 0.023s	sys 0.017s
I20260709 12:21:22.228018   842 raft_consensus.cc:359] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.229031   842 raft_consensus.cc:385] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.229313   842 raft_consensus.cc:740] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.230124   840 raft_consensus.cc:359] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.230226   842 consensus_queue.cc:260] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.231106   840 raft_consensus.cc:385] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.231447   840 raft_consensus.cc:740] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.232554   840 consensus_queue.cc:260] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.233470   842 ts_tablet_manager.cc:1434] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:21:22.234812   842 tablet_bootstrap.cc:492] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.235126   840 ts_tablet_manager.cc:1434] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:21:22.236331   840 tablet_bootstrap.cc:492] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.243180   842 tablet_bootstrap.cc:654] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.243667   840 tablet_bootstrap.cc:654] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.251582   839 tablet_bootstrap.cc:492] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.252190   839 ts_tablet_manager.cc:1403] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.056s	user 0.009s	sys 0.015s
I20260709 12:21:22.255998   840 tablet_bootstrap.cc:492] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.256351   839 raft_consensus.cc:359] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.256975   840 ts_tablet_manager.cc:1403] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.000s
I20260709 12:21:22.258136   839 raft_consensus.cc:385] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.258590   839 raft_consensus.cc:740] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.259457   839 consensus_queue.cc:260] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.260535   842 tablet_bootstrap.cc:492] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.261183   842 ts_tablet_manager.cc:1403] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.010s
I20260709 12:21:22.262799   841 raft_consensus.cc:493] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.263398   841 raft_consensus.cc:515] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.264604   842 raft_consensus.cc:359] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.262333   839 ts_tablet_manager.cc:1434] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20260709 12:21:22.265417   841 leader_election.cc:290] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.265452   842 raft_consensus.cc:385] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.262393   840 raft_consensus.cc:359] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.266194   842 raft_consensus.cc:740] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.266279   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c0b37a555e404f910d3c9b5e6224f9" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.267436   839 tablet_bootstrap.cc:492] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.267114   842 consensus_queue.cc:260] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
W20260709 12:21:22.268138   513 leader_election.cc:343] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.269306   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c0b37a555e404f910d3c9b5e6224f9" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.269928   842 ts_tablet_manager.cc:1434] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.006s
I20260709 12:21:22.266546   840 raft_consensus.cc:385] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.270007   698 raft_consensus.cc:2468] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:22.270612   840 raft_consensus.cc:740] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.271347   842 tablet_bootstrap.cc:492] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.271842   511 leader_election.cc:304] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:22.273159   841 raft_consensus.cc:2804] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.273578   841 raft_consensus.cc:493] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.274024   841 raft_consensus.cc:3060] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.280042   842 tablet_bootstrap.cc:654] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.287006   793 raft_consensus.cc:493] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.287654   793 raft_consensus.cc:515] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.289984   793 leader_election.cc:290] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:22.289664   840 consensus_queue.cc:260] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.291338   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.291743   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
W20260709 12:21:22.293988   664 leader_election.cc:343] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.294214   840 ts_tablet_manager.cc:1434] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
W20260709 12:21:22.295212   663 leader_election.cc:343] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.295709   663 leader_election.cc:304] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54; no voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:22.296033   840 tablet_bootstrap.cc:492] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.301698   841 raft_consensus.cc:515] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.304929   793 raft_consensus.cc:2749] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:22.306030   839 tablet_bootstrap.cc:654] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.308195   840 tablet_bootstrap.cc:654] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.319278   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c0b37a555e404f910d3c9b5e6224f9" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:22.317977   841 leader_election.cc:290] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
W20260709 12:21:22.321967   513 leader_election.cc:343] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.322088   842 tablet_bootstrap.cc:492] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.324151   842 ts_tablet_manager.cc:1403] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.053s	user 0.028s	sys 0.017s
I20260709 12:21:22.325517   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c0b37a555e404f910d3c9b5e6224f9" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.326293   698 raft_consensus.cc:3060] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.326664   840 tablet_bootstrap.cc:492] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.327231   840 ts_tablet_manager.cc:1403] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.032s	user 0.009s	sys 0.009s
I20260709 12:21:22.329015   842 raft_consensus.cc:359] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.330303   842 raft_consensus.cc:385] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.330709   842 raft_consensus.cc:740] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.330614   840 raft_consensus.cc:359] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.332541   840 raft_consensus.cc:385] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.332027   842 consensus_queue.cc:260] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.332886   840 raft_consensus.cc:740] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.334303   839 tablet_bootstrap.cc:492] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.335861   839 ts_tablet_manager.cc:1403] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.069s	user 0.023s	sys 0.010s
I20260709 12:21:22.335642   840 consensus_queue.cc:260] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.338003   842 ts_tablet_manager.cc:1434] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.004s
I20260709 12:21:22.341904   839 raft_consensus.cc:359] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.342710   842 tablet_bootstrap.cc:492] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.343111   839 raft_consensus.cc:385] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.343636   839 raft_consensus.cc:740] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.344543   839 consensus_queue.cc:260] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.347549   839 ts_tablet_manager.cc:1434] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 12:21:22.338280   698 raft_consensus.cc:2468] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:22.340181   840 ts_tablet_manager.cc:1434] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:21:22.348814   839 tablet_bootstrap.cc:492] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.349565   840 tablet_bootstrap.cc:492] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.349699   511 leader_election.cc:304] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:22.356048   841 raft_consensus.cc:2804] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.356585   841 raft_consensus.cc:697] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:22.356918   839 tablet_bootstrap.cc:654] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.357436   841 consensus_queue.cc:237] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.358332   842 tablet_bootstrap.cc:654] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.360838   840 tablet_bootstrap.cc:654] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.375520   842 tablet_bootstrap.cc:492] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.376375   842 ts_tablet_manager.cc:1403] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.005s
I20260709 12:21:22.381125   842 raft_consensus.cc:359] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.382928   842 raft_consensus.cc:385] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.383276   842 raft_consensus.cc:740] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.384418   839 tablet_bootstrap.cc:492] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.384121   842 consensus_queue.cc:260] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.385142   839 ts_tablet_manager.cc:1403] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.037s	user 0.009s	sys 0.009s
I20260709 12:21:22.386976   842 ts_tablet_manager.cc:1434] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20260709 12:21:22.388314   842 tablet_bootstrap.cc:492] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.388867   839 raft_consensus.cc:359] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.390604   839 raft_consensus.cc:385] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.390926   839 raft_consensus.cc:740] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.391831   839 consensus_queue.cc:260] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.392355   840 tablet_bootstrap.cc:492] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.393106   840 ts_tablet_manager.cc:1403] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.044s	user 0.010s	sys 0.019s
I20260709 12:21:22.394553   839 ts_tablet_manager.cc:1434] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.003s
I20260709 12:21:22.395785   839 tablet_bootstrap.cc:492] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.396591   840 raft_consensus.cc:359] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.397435   840 raft_consensus.cc:385] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.397770   840 raft_consensus.cc:740] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.398762   840 consensus_queue.cc:260] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.403538   840 ts_tablet_manager.cc:1434] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.008s
I20260709 12:21:22.404366   839 tablet_bootstrap.cc:654] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.404891   840 tablet_bootstrap.cc:492] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.405994   842 tablet_bootstrap.cc:654] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.410247   792 raft_consensus.cc:493] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.410962   792 raft_consensus.cc:515] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.414752   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.415129   840 tablet_bootstrap.cc:654] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:22.416771   513 leader_election.cc:343] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.418581   839 tablet_bootstrap.cc:492] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.419520   839 ts_tablet_manager.cc:1403] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.024s	user 0.020s	sys 0.000s
I20260709 12:21:22.426581   792 leader_election.cc:290] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.433461   839 raft_consensus.cc:359] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.434499   839 raft_consensus.cc:385] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.434823   839 raft_consensus.cc:740] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.435644   839 consensus_queue.cc:260] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.441936   445 catalog_manager.cc:5697] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.433581   842 tablet_bootstrap.cc:492] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.444170   842 ts_tablet_manager.cc:1403] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.056s	user 0.023s	sys 0.011s
I20260709 12:21:22.442922   839 ts_tablet_manager.cc:1434] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.004s
I20260709 12:21:22.451399   839 tablet_bootstrap.cc:492] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.448586   842 raft_consensus.cc:359] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.455152   842 raft_consensus.cc:385] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.455509   842 raft_consensus.cc:740] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.455811   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.458128   698 raft_consensus.cc:2468] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:22.462644   839 tablet_bootstrap.cc:654] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.462684   511 leader_election.cc:304] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:22.465122   862 raft_consensus.cc:2804] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.465597   862 raft_consensus.cc:493] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.465967   862 raft_consensus.cc:3060] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.466146   842 consensus_queue.cc:260] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.468899   842 ts_tablet_manager.cc:1434] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260709 12:21:22.470973   842 tablet_bootstrap.cc:492] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.476750   841 raft_consensus.cc:493] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.478673   841 raft_consensus.cc:515] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.480607   842 tablet_bootstrap.cc:654] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.480254   862 raft_consensus.cc:515] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.482007   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.482824   624 raft_consensus.cc:2468] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:22.486313   862 leader_election.cc:290] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.487089   513 leader_election.cc:304] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:22.487890   828 raft_consensus.cc:493] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.488432   828 raft_consensus.cc:515] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.488911   862 raft_consensus.cc:2804] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.489275   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.489750   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:22.490346   698 raft_consensus.cc:3060] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.490983   828 leader_election.cc:290] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.489327   862 raft_consensus.cc:493] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.491864   862 raft_consensus.cc:3060] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.492899   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c0b37a555e404f910d3c9b5e6224f9" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.494415   696 raft_consensus.cc:2393] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ebb903b7402421e9dc6bfb67d537511 in current term 1: Already voted for candidate e5c803c9290244dbbc92aaf769dbd54a in this term.
W20260709 12:21:22.499044   513 leader_election.cc:343] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.500916   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c0b37a555e404f910d3c9b5e6224f9" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:22.501669   841 leader_election.cc:290] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.503273   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.503880   587 leader_election.cc:304] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511; no voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:22.504143   696 raft_consensus.cc:2468] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:22.505275   828 raft_consensus.cc:3060] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.505548   862 raft_consensus.cc:515] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.507665   840 tablet_bootstrap.cc:492] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.509115   840 ts_tablet_manager.cc:1403] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.105s	user 0.031s	sys 0.031s
I20260709 12:21:22.510224   862 leader_election.cc:290] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.511883   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.512905   696 raft_consensus.cc:3060] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.514762   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:22.515491   622 raft_consensus.cc:3060] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.516980   828 raft_consensus.cc:2749] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:22.518898   698 raft_consensus.cc:2468] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:22.520776   511 leader_election.cc:304] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:22.522240   862 raft_consensus.cc:2804] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.524823   842 tablet_bootstrap.cc:492] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.513877   840 raft_consensus.cc:359] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.522874   862 raft_consensus.cc:697] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:22.525727   842 ts_tablet_manager.cc:1403] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.055s	user 0.012s	sys 0.005s
I20260709 12:21:22.524889   622 raft_consensus.cc:2468] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:22.525672   840 raft_consensus.cc:385] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.527661   513 leader_election.cc:304] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:22.527667   862 consensus_queue.cc:237] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.529418   841 raft_consensus.cc:2804] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.530134   840 raft_consensus.cc:740] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.530416   841 raft_consensus.cc:697] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:22.531605   841 consensus_queue.cc:237] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.531574   840 consensus_queue.cc:260] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.533672   696 raft_consensus.cc:2468] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:22.530280   839 tablet_bootstrap.cc:492] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.535053   839 ts_tablet_manager.cc:1403] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.084s	user 0.021s	sys 0.007s
I20260709 12:21:22.553644   863 raft_consensus.cc:493] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.554616   863 raft_consensus.cc:515] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.570446   840 ts_tablet_manager.cc:1434] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.061s	user 0.009s	sys 0.018s
I20260709 12:21:22.570923   828 raft_consensus.cc:493] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.571642   765 raft_consensus.cc:493] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.572201   765 raft_consensus.cc:515] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.572453   863 leader_election.cc:290] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.572660   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.573022   842 raft_consensus.cc:359] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.573971   842 raft_consensus.cc:385] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.574302   842 raft_consensus.cc:740] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
W20260709 12:21:22.575531   513 leader_election.cc:343] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.575289   842 consensus_queue.cc:260] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.576515   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e511c3ec5b5147de9a48e271b24864c6" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:22.576781   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.577373   548 raft_consensus.cc:2468] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:22.578711   765 leader_election.cc:290] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
W20260709 12:21:22.578729   511 leader_election.cc:343] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.576220   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e511c3ec5b5147de9a48e271b24864c6" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.579154   587 leader_election.cc:304] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:22.579627   511 leader_election.cc:304] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5c803c9290244dbbc92aaf769dbd54a; no voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:22.579998   696 raft_consensus.cc:2468] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:22.580788   863 raft_consensus.cc:2749] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:22.580937   840 tablet_bootstrap.cc:492] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.579651   839 raft_consensus.cc:359] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.581866   839 raft_consensus.cc:385] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.582023   828 raft_consensus.cc:515] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.582374   839 raft_consensus.cc:740] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.584059   765 raft_consensus.cc:2804] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.584525   765 raft_consensus.cc:493] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.583737   839 consensus_queue.cc:260] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.584896   765 raft_consensus.cc:3060] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.587900   839 ts_tablet_manager.cc:1434] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.052s	user 0.005s	sys 0.021s
I20260709 12:21:22.589686   839 tablet_bootstrap.cc:492] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.591188   828 leader_election.cc:290] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.592717   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
W20260709 12:21:22.594676   586 leader_election.cc:343] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.603989   840 tablet_bootstrap.cc:654] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.604627   828 raft_consensus.cc:493] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.605186   828 raft_consensus.cc:515] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.611901   828 leader_election.cc:290] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.613277   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.614184   696 raft_consensus.cc:2468] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:22.615474   586 leader_election.cc:304] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:22.616515   839 tablet_bootstrap.cc:654] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.616705   828 raft_consensus.cc:2804] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.617960   828 raft_consensus.cc:493] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.621730   828 raft_consensus.cc:3060] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:22.622663   587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:44737, user_credentials={real_user=slave}} blocked reactor thread for 43593.2us
I20260709 12:21:22.621377   765 raft_consensus.cc:515] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.627569   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:22.630702   765 leader_election.cc:290] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.632176   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e511c3ec5b5147de9a48e271b24864c6" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.635550   842 ts_tablet_manager.cc:1434] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.109s	user 0.016s	sys 0.017s
I20260709 12:21:22.638036   698 raft_consensus.cc:3060] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.641803   842 tablet_bootstrap.cc:492] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.642710   828 raft_consensus.cc:515] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.648088   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:22.651494   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.648149   828 leader_election.cc:290] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.652305   696 raft_consensus.cc:3060] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:22.652890   587 leader_election.cc:343] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.653357   587 leader_election.cc:304] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511; no voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:22.653385   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:22.654735   828 raft_consensus.cc:2749] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:22.656054   839 tablet_bootstrap.cc:492] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.656523   839 ts_tablet_manager.cc:1403] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.067s	user 0.009s	sys 0.024s
I20260709 12:21:22.652969   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e511c3ec5b5147de9a48e271b24864c6" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:22.657972   548 raft_consensus.cc:2468] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:22.658633   546 raft_consensus.cc:3060] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.661918   839 raft_consensus.cc:359] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.662809   839 raft_consensus.cc:385] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.672344   698 raft_consensus.cc:2468] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:22.663131   839 raft_consensus.cc:740] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.675488   842 tablet_bootstrap.cc:654] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.675940   586 leader_election.cc:304] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:22.675848   839 consensus_queue.cc:260] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.676847   840 tablet_bootstrap.cc:492] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.677096   765 raft_consensus.cc:493] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.677685   840 ts_tablet_manager.cc:1403] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.106s	user 0.015s	sys 0.011s
I20260709 12:21:22.677747   765 raft_consensus.cc:515] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.678467   839 ts_tablet_manager.cc:1434] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.005s
I20260709 12:21:22.679754   839 tablet_bootstrap.cc:492] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.680506   765 leader_election.cc:290] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.681133   765 raft_consensus.cc:2804] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.681607   765 raft_consensus.cc:697] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:22.683327   441 catalog_manager.cc:5697] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.685731   441 catalog_manager.cc:5697] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.687222   696 raft_consensus.cc:2468] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:22.688545   546 raft_consensus.cc:2468] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:22.688774   586 leader_election.cc:304] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:22.683334   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5380597535a4ae2a7e783db3903ec50" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.684413   765 consensus_queue.cc:237] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.689775   698 raft_consensus.cc:2468] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:22.690768   873 raft_consensus.cc:2804] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.683087   840 raft_consensus.cc:359] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.692384   873 raft_consensus.cc:697] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:22.692600   840 raft_consensus.cc:385] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.692910   840 raft_consensus.cc:740] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.693074   828 raft_consensus.cc:493] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.693296   873 consensus_queue.cc:237] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.693797   840 consensus_queue.cc:260] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.696344   840 ts_tablet_manager.cc:1434] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.002s
I20260709 12:21:22.693766   828 raft_consensus.cc:515] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.698167   840 tablet_bootstrap.cc:492] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.691545   586 leader_election.cc:304] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:22.699599   828 leader_election.cc:290] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.700881   828 raft_consensus.cc:2804] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.700989   839 tablet_bootstrap.cc:654] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.701278   828 raft_consensus.cc:493] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.701774   828 raft_consensus.cc:3060] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.703410   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5380597535a4ae2a7e783db3903ec50" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:22.710870   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.711759   696 raft_consensus.cc:2393] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ebb903b7402421e9dc6bfb67d537511 in current term 1: Already voted for candidate e5c803c9290244dbbc92aaf769dbd54a in this term.
I20260709 12:21:22.722153   444 catalog_manager.cc:5697] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.725271   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
W20260709 12:21:22.726897   587 leader_election.cc:343] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:22.728756   840 tablet_bootstrap.cc:654] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.730952   839 tablet_bootstrap.cc:492] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.731566   839 ts_tablet_manager.cc:1403] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.052s	user 0.004s	sys 0.024s
I20260709 12:21:22.732072   587 leader_election.cc:304] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511; no voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:22.733050   765 raft_consensus.cc:3060] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.734575   828 raft_consensus.cc:515] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.735845   839 raft_consensus.cc:359] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.746356   839 raft_consensus.cc:385] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.746740   839 raft_consensus.cc:740] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.747428   839 consensus_queue.cc:260] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.750682   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5380597535a4ae2a7e783db3903ec50" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.751418   696 raft_consensus.cc:3060] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.739406   842 tablet_bootstrap.cc:492] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.752168   842 ts_tablet_manager.cc:1403] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.115s	user 0.005s	sys 0.014s
I20260709 12:21:22.754105   441 catalog_manager.cc:5697] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.755187   842 raft_consensus.cc:359] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.756078   842 raft_consensus.cc:385] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.756354   842 raft_consensus.cc:740] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.757094   842 consensus_queue.cc:260] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.741093   840 tablet_bootstrap.cc:492] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.758191   840 ts_tablet_manager.cc:1403] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.005s
I20260709 12:21:22.759397   842 ts_tablet_manager.cc:1434] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:21:22.760890   862 raft_consensus.cc:493] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.761174   842 tablet_bootstrap.cc:492] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: Bootstrap starting.
I20260709 12:21:22.761605   862 raft_consensus.cc:515] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.764045   862 leader_election.cc:290] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.759838   839 ts_tablet_manager.cc:1434] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.003s
I20260709 12:21:22.766249   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:22.766935   698 raft_consensus.cc:2468] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:22.767400   839 tablet_bootstrap.cc:492] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: Bootstrap starting.
I20260709 12:21:22.768476   511 leader_election.cc:304] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:22.765611   840 raft_consensus.cc:359] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.770375   840 raft_consensus.cc:385] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.770762   840 raft_consensus.cc:740] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.764129   696 raft_consensus.cc:2468] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:22.771682   862 raft_consensus.cc:2804] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:22.772212   862 raft_consensus.cc:493] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.772558   862 raft_consensus.cc:3060] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.772859   586 leader_election.cc:304] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:22.774497   839 tablet_bootstrap.cc:654] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.774834   873 raft_consensus.cc:2804] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.775451   873 raft_consensus.cc:697] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:22.747742   828 leader_election.cc:290] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.776408   873 consensus_queue.cc:237] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.777307   842 tablet_bootstrap.cc:654] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.780148   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5380597535a4ae2a7e783db3903ec50" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:22.742516   765 raft_consensus.cc:2749] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:22.781020   546 raft_consensus.cc:3060] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.780933   840 consensus_queue.cc:260] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.787195   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.788106   622 raft_consensus.cc:2468] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 0.
I20260709 12:21:22.791080   546 raft_consensus.cc:2468] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:22.796811   840 ts_tablet_manager.cc:1434] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 12:21:22.799444   862 raft_consensus.cc:515] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.800415   839 tablet_bootstrap.cc:492] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: No bootstrap required, opened a new log
I20260709 12:21:22.801087   839 ts_tablet_manager.cc:1403] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.005s
I20260709 12:21:22.803085   840 tablet_bootstrap.cc:492] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: Bootstrap starting.
I20260709 12:21:22.803925   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:22.804086   839 raft_consensus.cc:359] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.804334   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:22.804575   696 raft_consensus.cc:3060] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.805073   622 raft_consensus.cc:3060] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:22.818117   443 catalog_manager.cc:5697] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.820307   842 tablet_bootstrap.cc:492] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: No bootstrap required, opened a new log
I20260709 12:21:22.820953   842 ts_tablet_manager.cc:1403] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: Time spent bootstrapping tablet: real 0.060s	user 0.013s	sys 0.014s
I20260709 12:21:22.821152   840 tablet_bootstrap.cc:654] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:22.821445   862 leader_election.cc:290] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:22.804746   839 raft_consensus.cc:385] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.822626   839 raft_consensus.cc:740] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.823643   839 consensus_queue.cc:260] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.824535   862 consensus_queue.cc:1048] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:22.827724   622 raft_consensus.cc:2468] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:22.830096   842 raft_consensus.cc:359] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.831271   842 raft_consensus.cc:385] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.831633   842 raft_consensus.cc:740] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.834338   839 ts_tablet_manager.cc:1434] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: Time spent starting tablet: real 0.033s	user 0.004s	sys 0.006s
I20260709 12:21:22.835390   842 consensus_queue.cc:260] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.837960   842 ts_tablet_manager.cc:1434] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.003s
I20260709 12:21:22.834673   696 raft_consensus.cc:2468] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 1.
I20260709 12:21:22.840579   840 tablet_bootstrap.cc:492] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: No bootstrap required, opened a new log
I20260709 12:21:22.840539   511 leader_election.cc:304] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:22.841198   840 ts_tablet_manager.cc:1403] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: Time spent bootstrapping tablet: real 0.038s	user 0.016s	sys 0.008s
I20260709 12:21:22.844776   841 raft_consensus.cc:2804] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:22.844550   840 raft_consensus.cc:359] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.845326   840 raft_consensus.cc:385] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:22.845402   841 raft_consensus.cc:697] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:22.845683   840 raft_consensus.cc:740] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Initialized, Role: FOLLOWER
I20260709 12:21:22.846464   840 consensus_queue.cc:260] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.846413   841 consensus_queue.cc:237] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.848807   840 ts_tablet_manager.cc:1434] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:21:22.875442   444 catalog_manager.cc:5697] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 0 to 1, leader changed from <none> to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 1 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:22.878159   841 consensus_queue.cc:1048] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:22.893181   494 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:22.992548   852 raft_consensus.cc:493] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.992623   792 consensus_queue.cc:1048] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:22.993563   852 raft_consensus.cc:515] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:22.993836   841 consensus_queue.cc:1048] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:22.996735   852 leader_election.cc:290] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:22.997143   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:22.997982   546 raft_consensus.cc:2468] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 0.
I20260709 12:21:22.998191   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:22.998878   622 raft_consensus.cc:2468] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 0.
I20260709 12:21:22.999411   663 leader_election.cc:304] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:23.000605   852 raft_consensus.cc:2804] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:23.002012   852 raft_consensus.cc:493] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.002410   852 raft_consensus.cc:3060] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.014118   852 raft_consensus.cc:515] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.017267   852 leader_election.cc:290] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:23.017943   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:23.018067   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:23.018985   546 raft_consensus.cc:3060] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.019134   624 raft_consensus.cc:3060] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.027464   841 consensus_queue.cc:1048] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.035842   624 raft_consensus.cc:2468] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 1.
I20260709 12:21:23.037134   664 leader_election.cc:304] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:23.038151   852 raft_consensus.cc:2804] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:23.039779   852 raft_consensus.cc:697] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:23.038136   546 raft_consensus.cc:2468] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 1.
I20260709 12:21:23.042923   852 consensus_queue.cc:237] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.043689   873 raft_consensus.cc:493] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.044365   873 raft_consensus.cc:515] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.044534   792 consensus_queue.cc:1048] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.046794   873 leader_election.cc:290] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:23.048473   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54" is_pre_election: true
I20260709 12:21:23.049196   894 raft_consensus.cc:493] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.049939   894 raft_consensus.cc:515] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.050951   698 raft_consensus.cc:2468] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:23.054031   586 leader_election.cc:304] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:23.055223   873 raft_consensus.cc:2804] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:23.055724   873 raft_consensus.cc:493] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.056186   873 raft_consensus.cc:3060] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.055450   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:23.057494   546 raft_consensus.cc:2468] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 0.
I20260709 12:21:23.061787   894 leader_election.cc:290] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:23.065768   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ebb903b7402421e9dc6bfb67d537511" is_pre_election: true
I20260709 12:21:23.065418   873 raft_consensus.cc:515] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.067301   622 raft_consensus.cc:2393] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27943bf6f68e4f7faeaf374de0336d54 in current term 1: Already voted for candidate 9ebb903b7402421e9dc6bfb67d537511 in this term.
I20260709 12:21:23.067459   443 catalog_manager.cc:5697] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:23.069770   698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:23.070185   548 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:23.070616   698 raft_consensus.cc:3060] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.070822   548 raft_consensus.cc:3060] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.066977   546 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a" is_pre_election: true
I20260709 12:21:23.076562   663 leader_election.cc:304] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54; no voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:23.081311   873 leader_election.cc:290] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:23.084013   548 raft_consensus.cc:2468] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:23.085003   698 raft_consensus.cc:2468] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 1.
I20260709 12:21:23.085717   894 raft_consensus.cc:2749] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:23.086251   587 leader_election.cc:304] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:23.090158   765 consensus_queue.cc:1048] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.090865   873 raft_consensus.cc:2804] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:23.091498   873 raft_consensus.cc:697] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:23.091301   546 raft_consensus.cc:3060] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:23.092552   873 consensus_queue.cc:237] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.108378   765 consensus_queue.cc:1048] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.125264   828 consensus_queue.cc:1048] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.124832   444 catalog_manager.cc:5697] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:23.151722   828 consensus_queue.cc:1048] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.172092   622 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
mode: GRACEFUL
new_leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:56200
I20260709 12:21:23.172825   622 raft_consensus.cc:606] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Received request to transfer leadership to TS e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:23.179970   696 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
mode: GRACEFUL
new_leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:58866
I20260709 12:21:23.181064   696 raft_consensus.cc:606] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Received request to transfer leadership to TS e5c803c9290244dbbc92aaf769dbd54a
I20260709 12:21:23.205154   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
mode: GRACEFUL
new_leader_uuid: "9ebb903b7402421e9dc6bfb67d537511"
 from {username='slave'} at 127.0.0.1:59570
I20260709 12:21:23.205863   547 raft_consensus.cc:606] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Received request to transfer leadership to TS 9ebb903b7402421e9dc6bfb67d537511
I20260709 12:21:23.213543   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
mode: GRACEFUL
new_leader_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:59570
I20260709 12:21:23.214282   547 raft_consensus.cc:606] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Received request to transfer leadership to TS 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:23.279709   828 consensus_queue.cc:1048] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.294059   872 consensus_queue.cc:1048] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.301426   792 consensus_queue.cc:1048] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.320712   792 consensus_queue.cc:1048] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.435669   792 raft_consensus.cc:993] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: : Instructing follower 9ebb903b7402421e9dc6bfb67d537511 to start an election
I20260709 12:21:23.436254   841 raft_consensus.cc:1081] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Signalling peer 9ebb903b7402421e9dc6bfb67d537511 to start an election
I20260709 12:21:23.438225   622 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
 from {username='slave'} at 127.0.0.1:56194
I20260709 12:21:23.438841   622 raft_consensus.cc:493] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:23.439266   622 raft_consensus.cc:3060] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.446480   622 raft_consensus.cc:515] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.448477   622 leader_election.cc:290] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:23.449890   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:23.449888   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8d474f3a8f2484abca69e66179b2566" candidate_uuid: "9ebb903b7402421e9dc6bfb67d537511" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:23.450944   696 raft_consensus.cc:3060] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.450991   547 raft_consensus.cc:3055] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:23.451630   547 raft_consensus.cc:740] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:23.452715   547 consensus_queue.cc:260] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.454424   547 raft_consensus.cc:3060] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.458560   696 raft_consensus.cc:2468] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 2.
I20260709 12:21:23.460251   586 leader_election.cc:304] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:23.461146   872 raft_consensus.cc:2804] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:23.461658   872 raft_consensus.cc:697] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Becoming Leader. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:23.462560   872 consensus_queue.cc:237] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.463414   547 raft_consensus.cc:2468] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ebb903b7402421e9dc6bfb67d537511 in term 2.
I20260709 12:21:23.472465   444 catalog_manager.cc:5697] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 reported cstate change: term changed from 1 to 2, leader changed from e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129) to 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "9ebb903b7402421e9dc6bfb67d537511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:23.493607   852 consensus_queue.cc:1048] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.497988   852 consensus_queue.cc:1048] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.503924   895 raft_consensus.cc:993] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: : Instructing follower e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:23.504485   895 raft_consensus.cc:1081] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Signalling peer e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:23.507748   546 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:59566
I20260709 12:21:23.508459   546 raft_consensus.cc:493] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:23.508832   546 raft_consensus.cc:3060] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.519295   546 raft_consensus.cc:515] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.521608   546 leader_election.cc:290] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:23.521545   547 raft_consensus.cc:1240] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Rejecting Update request from peer 27943bf6f68e4f7faeaf374de0336d54 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:23.525060   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:23.525801   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66069fa7e25942cd95cc23bd7e1c51ff" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:23.526546   622 raft_consensus.cc:3060] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.527192   909 consensus_queue.cc:1059] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.525929   696 raft_consensus.cc:3055] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:23.528700   696 raft_consensus.cc:740] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:23.529661   696 consensus_queue.cc:260] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.531255   696 raft_consensus.cc:3060] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.534795   622 raft_consensus.cc:2468] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:23.536123   513 leader_election.cc:304] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:23.537119   792 raft_consensus.cc:2804] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:23.537685   792 raft_consensus.cc:697] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:23.537755   696 raft_consensus.cc:2468] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
W20260709 12:21:23.538561   895 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:23.538658   792 consensus_queue.cc:237] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.547829   443 catalog_manager.cc:5697] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 1 to 2, leader changed from 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131) to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 2 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:23.650723   872 raft_consensus.cc:993] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: : Instructing follower e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:23.651257   872 raft_consensus.cc:1081] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Signalling peer e5c803c9290244dbbc92aaf769dbd54a to start an election
I20260709 12:21:23.653194   547 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
 from {username='slave'} at 127.0.0.1:59568
I20260709 12:21:23.653913   547 raft_consensus.cc:493] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:23.654340   547 raft_consensus.cc:3060] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.656250   841 raft_consensus.cc:993] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: : Instructing follower 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:23.656950   792 raft_consensus.cc:1081] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Signalling peer 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:23.659299   696 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:58862
I20260709 12:21:23.659974   696 raft_consensus.cc:493] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:23.660370   696 raft_consensus.cc:3060] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.663776   547 raft_consensus.cc:515] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.664644   872 consensus_queue.cc:1048] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:23.667284   622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:23.667557   547 leader_election.cc:290] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Requested vote from peers 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537), 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131:33749)
I20260709 12:21:23.669034   622 raft_consensus.cc:3055] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:23.669427   622 raft_consensus.cc:740] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:23.669489   697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "763f2cf7a1fd4ddb83341c75d7579de4" candidate_uuid: "e5c803c9290244dbbc92aaf769dbd54a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
I20260709 12:21:23.670722   697 raft_consensus.cc:3060] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.669440   696 raft_consensus.cc:515] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.670991   622 consensus_queue.cc:260] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.673763   622 raft_consensus.cc:3060] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.674899   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:23.675616   547 raft_consensus.cc:3055] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:23.675961   547 raft_consensus.cc:740] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:23.676906   547 consensus_queue.cc:260] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.681085   828 consensus_queue.cc:1048] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:23.683558   696 leader_election.cc:290] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:23.686403   622 raft_consensus.cc:2468] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:23.681321   547 raft_consensus.cc:3060] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.679028   624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2a7937ea6a34b53a1c8b339cb02383f" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:23.690091   513 leader_election.cc:304] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ebb903b7402421e9dc6bfb67d537511, e5c803c9290244dbbc92aaf769dbd54a; no voters: 
I20260709 12:21:23.690459   624 raft_consensus.cc:3060] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.691107   792 raft_consensus.cc:2804] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:23.691659   792 raft_consensus.cc:697] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Becoming Leader. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:23.694659   792 consensus_queue.cc:237] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.704221   697 raft_consensus.cc:2468] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5c803c9290244dbbc92aaf769dbd54a in term 2.
I20260709 12:21:23.707860   547 raft_consensus.cc:2468] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:23.705475   624 raft_consensus.cc:2468] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:23.711139   664 leader_election.cc:304] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:23.712194   895 raft_consensus.cc:2804] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:23.712723   895 raft_consensus.cc:697] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:23.713913   895 consensus_queue.cc:237] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:23.723467   443 catalog_manager.cc:5697] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 1 to 2, leader changed from e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129) to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:23.724076   444 catalog_manager.cc:5697] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a reported cstate change: term changed from 1 to 2, leader changed from 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130) to e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129). New cstate: current_term: 2 leader_uuid: "e5c803c9290244dbbc92aaf769dbd54a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:23.859613   546 raft_consensus.cc:1275] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 9ebb903b7402421e9dc6bfb67d537511: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:23.861194   873 consensus_queue.cc:1048] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:23.872494   696 raft_consensus.cc:1275] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer 9ebb903b7402421e9dc6bfb67d537511: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:23.875063   873 consensus_queue.cc:1048] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:24.009452   624 raft_consensus.cc:1275] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.011158   841 consensus_queue.cc:1048] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:24.021157   696 raft_consensus.cc:1275] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.025471   863 consensus_queue.cc:1048] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:24.101084   697 raft_consensus.cc:1275] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.102701   863 consensus_queue.cc:1048] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:24.112913   624 raft_consensus.cc:1275] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer e5c803c9290244dbbc92aaf769dbd54a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.114739   863 consensus_queue.cc:1048] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:24.300203   624 raft_consensus.cc:1275] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.306766   895 consensus_queue.cc:1048] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:21:24.332857   546 raft_consensus.cc:1275] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:24.337131   895 consensus_queue.cc:1048] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:26.287381   646 debug-util.cc:398] Leaking SignalData structure 0x7b08001aa7c0 after lost signal to thread 11313
W20260709 12:21:26.288856   646 debug-util.cc:398] Leaking SignalData structure 0x7b08002fc8a0 after lost signal to thread 480
W20260709 12:21:26.290295   646 debug-util.cc:398] Leaking SignalData structure 0x7b08000be6a0 after lost signal to thread 572
W20260709 12:21:26.291497   646 debug-util.cc:398] Leaking SignalData structure 0x7b08001d5a80 after lost signal to thread 649
W20260709 12:21:26.292651   646 debug-util.cc:398] Leaking SignalData structure 0x7b08000acc80 after lost signal to thread 723
I20260709 12:21:26.302712   444 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59596:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:21:26.342243   444 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:59596:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:21:26.400143 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:21:26.485411   937 log.cc:927] Time spent T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: Append to log took a long time: real 0.054s	user 0.002s	sys 0.000s
W20260709 12:21:26.711225   954 log.cc:927] Time spent T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: Append to log took a long time: real 0.098s	user 0.002s	sys 0.000s
W20260709 12:21:26.747582   957 log.cc:927] Time spent T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:21:26.747995   963 tablet.cc:1724] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:26.711835   960 tablet.cc:1724] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:26.752755   966 log.cc:927] Time spent T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:21:26.757848   964 log.cc:927] Time spent T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.064s	user 0.003s	sys 0.000s
I20260709 12:21:26.773025   967 tablet.cc:1724] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:26.814581   973 tablet.cc:1724] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:26.828418   968 log.cc:927] Time spent T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260709 12:21:26.868501   574 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:09c0b37a555e404f910d3c9b5e6224f9)
W20260709 12:21:26.817271   969 log.cc:927] Time spent T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.059s	user 0.000s	sys 0.004s
W20260709 12:21:26.878523   574 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:09c0b37a555e404f910d3c9b5e6224f9)
W20260709 12:21:26.883193   574 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:09c0b37a555e404f910d3c9b5e6224f9)
W20260709 12:21:26.892102   961 log.cc:927] Time spent T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: Append to log took a long time: real 0.057s	user 0.002s	sys 0.000s
I20260709 12:21:26.902594   977 tablet.cc:1724] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:26.903870   976 tablet.cc:1724] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:26.931340   979 tablet.cc:1724] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:26.950299   971 log.cc:927] Time spent T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.195s	user 0.000s	sys 0.003s
I20260709 12:21:26.955154   544 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
mode: GRACEFUL
new_leader_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:45594
I20260709 12:21:26.951512   960 tablet.cc:1724] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:26.956637   544 raft_consensus.cc:606] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Received request to transfer leadership to TS 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:26.981998   981 tablet.cc:1724] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:27.012903   965 log.cc:927] Time spent T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.268s	user 0.000s	sys 0.006s
I20260709 12:21:27.044204   963 tablet.cc:1724] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.072831   986 tablet.cc:1724] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:26.991631   982 tablet.cc:1724] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:27.091002   511 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.131:33749, user_credentials={real_user=slave}} blocked reactor thread for 53216.2us
I20260709 12:21:27.093570   973 tablet.cc:1724] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.102553   987 tablet.cc:1724] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.076306   967 tablet.cc:1724] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.121301   977 tablet.cc:1724] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:27.130887   985 log.cc:927] Time spent T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:21:27.138942   976 tablet.cc:1724] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.172015   981 tablet.cc:1724] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.239374   990 tablet.cc:1724] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:27.179531   984 log.cc:927] Time spent T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: Append to log took a long time: real 0.102s	user 0.000s	sys 0.002s
I20260709 12:21:27.246757   841 raft_consensus.cc:993] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: : Instructing follower 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.247344   841 raft_consensus.cc:1081] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Signalling peer 27943bf6f68e4f7faeaf374de0336d54 to start an election
W20260709 12:21:27.253125   545 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.031s	sys 0.056s
W20260709 12:21:27.253765   545 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.031s	sys 0.056s
I20260709 12:21:27.197415   960 tablet.cc:1724] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.210758   963 tablet.cc:1724] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.195115   979 tablet.cc:1724] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.255859   696 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:58862
I20260709 12:21:27.256801   696 raft_consensus.cc:493] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.257277   696 raft_consensus.cc:3060] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.262142   981 tablet.cc:1724] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.277287   981 tablet.cc:1724] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.279587   696 raft_consensus.cc:515] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.285329   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:27.286201   545 raft_consensus.cc:3055] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:27.286619   545 raft_consensus.cc:740] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:27.286808   696 leader_election.cc:290] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:27.287817   545 consensus_queue.cc:260] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.287912   621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
mode: GRACEFUL
new_leader_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:37630
I20260709 12:21:27.292354   621 raft_consensus.cc:606] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Received request to transfer leadership to TS 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:27.291502   545 raft_consensus.cc:3060] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.307212   623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03e67e33e2e44833bb0821377ce57149" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:27.307989   623 raft_consensus.cc:3060] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.310083   967 tablet.cc:1724] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.315095   986 tablet.cc:1724] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.319759 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 12:21:27.320842 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:27.331974   973 tablet.cc:1724] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:27.352555   872 raft_consensus.cc:993] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: : Instructing follower 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.353478   947 raft_consensus.cc:1081] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Signalling peer 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.354683   623 raft_consensus.cc:2468] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:27.356047   547 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844"
dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
mode: GRACEFUL
new_leader_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:45594
I20260709 12:21:27.370460   547 raft_consensus.cc:606] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Received request to transfer leadership to TS 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:27.371238   664 leader_election.cc:304] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:27.374289   916 raft_consensus.cc:2804] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:27.374866   916 raft_consensus.cc:697] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:27.375262   621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226"
dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
mode: GRACEFUL
new_leader_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:37630
I20260709 12:21:27.376031   621 raft_consensus.cc:606] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Received request to transfer leadership to TS 27943bf6f68e4f7faeaf374de0336d54
I20260709 12:21:27.375900   916 consensus_queue.cc:237] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.377933 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:27.379006 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:27.357079   545 raft_consensus.cc:2468] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:27.368765   696 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:58848
I20260709 12:21:27.380486   696 raft_consensus.cc:493] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.380986   696 raft_consensus.cc:3060] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.396584   696 raft_consensus.cc:515] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.399717   443 catalog_manager.cc:5697] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 1 to 2, leader changed from e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129) to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.400952   696 leader_election.cc:290] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:27.401870   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:27.405002   623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59c0a7890a6f4eda9f0c6c4df133f0c2" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:27.409512   623 raft_consensus.cc:3055] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:27.410214   623 raft_consensus.cc:740] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:27.411461   623 consensus_queue.cc:260] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.413224   623 raft_consensus.cc:3060] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.413915   545 raft_consensus.cc:3060] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.427170   623 raft_consensus.cc:2468] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:27.428802   664 leader_election.cc:304] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:27.430011   916 raft_consensus.cc:2804] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:27.430678   916 raft_consensus.cc:697] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:27.431674   545 raft_consensus.cc:2468] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 2.
I20260709 12:21:27.431861   916 consensus_queue.cc:237] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.447199   443 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e: soft_deleted_table [id=8d43b17f3c3f4bdd8eb5af79315ddd74] alter complete (version 1)
I20260709 12:21:27.462256   441 catalog_manager.cc:5697] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 1 to 2, leader changed from 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130) to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.748853   944 raft_consensus.cc:993] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: : Instructing follower 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.749595   792 raft_consensus.cc:1081] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Signalling peer 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.754535   696 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:58862
I20260709 12:21:27.755302   696 raft_consensus.cc:493] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.755638   696 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.761849   696 raft_consensus.cc:515] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.764122   696 leader_election.cc:290] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 3 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:27.765875   545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:27.765875   623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "243b8823106f4dfcbebb6b643b2c5844" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:27.767066   545 raft_consensus.cc:3055] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:27.767072   623 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.768337   545 raft_consensus.cc:740] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e5c803c9290244dbbc92aaf769dbd54a, State: Running, Role: LEADER
I20260709 12:21:27.770119   545 consensus_queue.cc:260] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.771728   545 raft_consensus.cc:3060] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.777993   623 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 3.
I20260709 12:21:27.780897   545 raft_consensus.cc:2468] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 3.
I20260709 12:21:27.781258   664 leader_election.cc:304] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:27.782393   916 raft_consensus.cc:2804] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:27.782970   916 raft_consensus.cc:697] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:27.783953   916 consensus_queue.cc:237] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.794890   441 catalog_manager.cc:5697] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 2 to 3, leader changed from e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129) to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.819460   545 raft_consensus.cc:1275] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.820377   947 raft_consensus.cc:993] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: : Instructing follower 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.821004   947 raft_consensus.cc:1081] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Signalling peer 27943bf6f68e4f7faeaf374de0336d54 to start an election
I20260709 12:21:27.821561   916 consensus_queue.cc:1048] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.822777   696 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226"
dest_uuid: "27943bf6f68e4f7faeaf374de0336d54"
 from {username='slave'} at 127.0.0.1:58848
I20260709 12:21:27.823442   696 raft_consensus.cc:493] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:27.823889   696 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.845571   696 raft_consensus.cc:515] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.847862   623 raft_consensus.cc:1275] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.856401   696 leader_election.cc:290] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 3 election: Requested vote from peers e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737), 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537)
I20260709 12:21:27.857005   972 consensus_queue.cc:1048] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:27.860550   623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9ebb903b7402421e9dc6bfb67d537511"
I20260709 12:21:27.861264   623 raft_consensus.cc:3055] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:27.861583   623 raft_consensus.cc:740] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9ebb903b7402421e9dc6bfb67d537511, State: Running, Role: LEADER
I20260709 12:21:27.862643   623 consensus_queue.cc:260] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.862847   547 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57244e68ac9d49629e202a7be0eb0226" candidate_uuid: "27943bf6f68e4f7faeaf374de0336d54" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e5c803c9290244dbbc92aaf769dbd54a"
I20260709 12:21:27.863919   547 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.864261   623 raft_consensus.cc:3060] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:27.876806   620 raft_consensus.cc:1275] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.879201   972 consensus_queue.cc:1048] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.879467   623 raft_consensus.cc:2468] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 3.
I20260709 12:21:27.892287   664 leader_election.cc:304] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27943bf6f68e4f7faeaf374de0336d54, 9ebb903b7402421e9dc6bfb67d537511; no voters: 
I20260709 12:21:27.893973   972 raft_consensus.cc:2804] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:27.894577   972 raft_consensus.cc:697] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 LEADER]: Becoming Leader. State: Replica: 27943bf6f68e4f7faeaf374de0336d54, State: Running, Role: LEADER
I20260709 12:21:27.895812   972 consensus_queue.cc:237] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } }
I20260709 12:21:27.904333   545 raft_consensus.cc:1275] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:27.906407   916 consensus_queue.cc:1048] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.917668   441 catalog_manager.cc:5697] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 reported cstate change: term changed from 2 to 3, leader changed from 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130) to 27943bf6f68e4f7faeaf374de0336d54 (127.11.11.131). New cstate: current_term: 3 leader_uuid: "27943bf6f68e4f7faeaf374de0336d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27943bf6f68e4f7faeaf374de0336d54" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.923460   547 raft_consensus.cc:2468] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27943bf6f68e4f7faeaf374de0336d54 in term 3.
I20260709 12:21:28.297676   545 raft_consensus.cc:1275] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:28.299647  1003 consensus_queue.cc:1048] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:28.317902   620 raft_consensus.cc:1275] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:28.319761   916 consensus_queue.cc:1048] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:28.374465   620 raft_consensus.cc:1275] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:28.376209   916 consensus_queue.cc:1048] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ebb903b7402421e9dc6bfb67d537511" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 42537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:28.381212 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:28.396030 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:28.396863 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:28.398147   545 raft_consensus.cc:1275] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Refusing update from remote peer 27943bf6f68e4f7faeaf374de0336d54: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:28.412480  1001 consensus_queue.cc:1048] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c803c9290244dbbc92aaf769dbd54a" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:28.421039 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:28.428190 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:29.429838 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:29.438535 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:29.439152 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:29.447049 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:29.447710 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:30.448920 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:30.456365 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:30.457016 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:30.462990 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:30.463644 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:31.464921 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:31.471040 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:31.471747 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:31.478655 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:31.479420 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:32.480655 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:32.487380 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:32.488054 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:32.494410 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:32.495088 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:33.496320 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:33.502589 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:33.503161 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:33.508595 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:33.509271 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:34.510978 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:34.524001 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:34.524685 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:34.534019 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:34.534662 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:35.536111 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:35.543748 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:35.544431 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:35.550592 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:35.551254 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:36.552644 11310 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:21:36.558609 11310 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:21:36.559245 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:36.565119 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:36.565901 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:37.642460 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
W20260709 12:21:37.689085   587 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 8 similar messages]
W20260709 12:21:37.699133   587 consensus_peers.cc:597] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:37.719779   587 consensus_peers.cc:597] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.720455 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:37.722199 11310 tablet_replica.cc:333] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.723886 11310 raft_consensus.cc:2243] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.725265 11310 raft_consensus.cc:2272] T c5380597535a4ae2a7e783db3903ec50 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:37.726696   663 consensus_peers.cc:597] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.730789 11310 tablet_replica.cc:333] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.732321 11310 raft_consensus.cc:2243] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.733255 11310 raft_consensus.cc:2272] T 03e67e33e2e44833bb0821377ce57149 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.736243 11310 tablet_replica.cc:333] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.736950 11310 raft_consensus.cc:2243] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:37.738050 11310 raft_consensus.cc:2272] T 763f2cf7a1fd4ddb83341c75d7579de4 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.741309 11310 tablet_replica.cc:333] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.742247 11310 raft_consensus.cc:2243] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.743012 11310 raft_consensus.cc:2272] T c2a7937ea6a34b53a1c8b339cb02383f P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.746294 11310 tablet_replica.cc:333] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.747014 11310 raft_consensus.cc:2243] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.747757 11310 raft_consensus.cc:2272] T f8d474f3a8f2484abca69e66179b2566 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.751502 11310 tablet_replica.cc:333] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.752696 11310 raft_consensus.cc:2243] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:37.754398 11310 raft_consensus.cc:2272] T 66069fa7e25942cd95cc23bd7e1c51ff P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.757383 11310 tablet_replica.cc:333] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.758162 11310 raft_consensus.cc:2243] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.758776 11310 raft_consensus.cc:2272] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.761840 11310 tablet_replica.cc:333] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.762584 11310 raft_consensus.cc:2243] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:37.763713 11310 raft_consensus.cc:2272] T 09c0b37a555e404f910d3c9b5e6224f9 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.767098 11310 tablet_replica.cc:333] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.768039 11310 raft_consensus.cc:2243] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.768846 11310 raft_consensus.cc:2272] T 762d8bc9a888413c988f106bd58383c1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.772919 11310 tablet_replica.cc:333] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.773833 11310 raft_consensus.cc:2243] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.774443 11310 raft_consensus.cc:2272] T 4c94ee5bad404294983f9198e5d7eb4e P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.777446 11310 tablet_replica.cc:333] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.778087 11310 raft_consensus.cc:2243] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.778739 11310 raft_consensus.cc:2272] T e511c3ec5b5147de9a48e271b24864c6 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.781242 11310 tablet_replica.cc:333] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.781884 11310 raft_consensus.cc:2243] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:37.782925 11310 raft_consensus.cc:2272] T 4b4a66fdb4594fd5bb195b5b25c21527 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.785776 11310 tablet_replica.cc:333] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.786460 11310 raft_consensus.cc:2243] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.787045 11310 raft_consensus.cc:2272] T ce78a35bc6404be88f9fcb05fcc88fe1 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.791467 11310 tablet_replica.cc:333] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.792114 11310 raft_consensus.cc:2243] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:37.793128 11310 raft_consensus.cc:2272] T a32b616ee4b84fae8eb7a251baeb9382 P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.795920 11310 tablet_replica.cc:333] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.796530 11310 raft_consensus.cc:2243] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.797094 11310 raft_consensus.cc:2272] T 57244e68ac9d49629e202a7be0eb0226 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.799829 11310 tablet_replica.cc:333] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.800444 11310 raft_consensus.cc:2243] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.801011 11310 raft_consensus.cc:2272] T 9ddd16927bee475eb509866236b26572 P e5c803c9290244dbbc92aaf769dbd54a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.803766 11310 tablet_replica.cc:333] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.804409 11310 raft_consensus.cc:2243] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:37.805421 11310 raft_consensus.cc:2272] T 603f9b51bd29455284aec5af830e344c P e5c803c9290244dbbc92aaf769dbd54a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.808910 11310 tablet_replica.cc:333] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a: stopping tablet replica
I20260709 12:21:37.809513 11310 raft_consensus.cc:2243] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.810387 11310 raft_consensus.cc:2272] T 243b8823106f4dfcbebb6b643b2c5844 P e5c803c9290244dbbc92aaf769dbd54a [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:37.812345   663 consensus_peers.cc:597] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.865350 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
W20260709 12:21:37.919274   663 consensus_peers.cc:597] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.925606 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
W20260709 12:21:37.955744   664 consensus_peers.cc:597] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Couldn't send request to peer 9ebb903b7402421e9dc6bfb67d537511. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.976785 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:37.977823 11310 tablet_replica.cc:333] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:37.979168 11310 raft_consensus.cc:2243] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.980033 11310 raft_consensus.cc:2272] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.982859 11310 tablet_replica.cc:333] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:37.983510 11310 raft_consensus.cc:2243] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.984114 11310 raft_consensus.cc:2272] T 66069fa7e25942cd95cc23bd7e1c51ff P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.987099 11310 tablet_replica.cc:333] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:37.987753 11310 raft_consensus.cc:2243] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.988283 11310 raft_consensus.cc:2272] T c2a7937ea6a34b53a1c8b339cb02383f P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.991384 11310 tablet_replica.cc:333] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:37.992624 11310 raft_consensus.cc:2243] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:37.993616 11310 raft_consensus.cc:2272] T 09c0b37a555e404f910d3c9b5e6224f9 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:37.998020 11310 tablet_replica.cc:333] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
W20260709 12:21:37.998564   663 consensus_peers.cc:597] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.998813 11310 raft_consensus.cc:2243] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:21:37.999008   587 consensus_peers.cc:597] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.999854 11310 raft_consensus.cc:2272] T 763f2cf7a1fd4ddb83341c75d7579de4 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.002579 11310 tablet_replica.cc:333] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.003227 11310 raft_consensus.cc:2243] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.003808 11310 raft_consensus.cc:2272] T 03e67e33e2e44833bb0821377ce57149 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.006387 11310 tablet_replica.cc:333] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.006964 11310 raft_consensus.cc:2243] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.007932 11310 raft_consensus.cc:2272] T c5380597535a4ae2a7e783db3903ec50 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.011451 11310 tablet_replica.cc:333] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.012171 11310 raft_consensus.cc:2243] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.013481 11310 raft_consensus.cc:2272] T e511c3ec5b5147de9a48e271b24864c6 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:38.015336   587 consensus_peers.cc:597] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.016705 11310 tablet_replica.cc:333] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.017401 11310 raft_consensus.cc:2243] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.017963 11310 raft_consensus.cc:2272] T 4b4a66fdb4594fd5bb195b5b25c21527 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.020604 11310 tablet_replica.cc:333] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.021243 11310 raft_consensus.cc:2243] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.022282 11310 raft_consensus.cc:2272] T 762d8bc9a888413c988f106bd58383c1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.025411 11310 tablet_replica.cc:333] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.026193 11310 raft_consensus.cc:2243] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.027329 11310 raft_consensus.cc:2272] T 4c94ee5bad404294983f9198e5d7eb4e P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.030571 11310 tablet_replica.cc:333] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.031275 11310 raft_consensus.cc:2243] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.032486 11310 raft_consensus.cc:2272] T f8d474f3a8f2484abca69e66179b2566 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.036489 11310 tablet_replica.cc:333] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.037447 11310 raft_consensus.cc:2243] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.038511 11310 raft_consensus.cc:2272] T ce78a35bc6404be88f9fcb05fcc88fe1 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.043227 11310 tablet_replica.cc:333] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
W20260709 12:21:38.043637   663 consensus_peers.cc:597] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.044062 11310 raft_consensus.cc:2243] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:21:38.044335   663 consensus_peers.cc:597] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer e5c803c9290244dbbc92aaf769dbd54a (127.11.11.129:44737): Couldn't send request to peer e5c803c9290244dbbc92aaf769dbd54a. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:44737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.044963 11310 raft_consensus.cc:2272] T a32b616ee4b84fae8eb7a251baeb9382 P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.047789 11310 tablet_replica.cc:333] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.048449 11310 raft_consensus.cc:2243] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.049067 11310 raft_consensus.cc:2272] T 243b8823106f4dfcbebb6b643b2c5844 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.051783 11310 tablet_replica.cc:333] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.052409 11310 raft_consensus.cc:2243] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.053108 11310 raft_consensus.cc:2272] T 603f9b51bd29455284aec5af830e344c P 9ebb903b7402421e9dc6bfb67d537511 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.055825 11310 tablet_replica.cc:333] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.056401 11310 raft_consensus.cc:2243] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.056880 11310 raft_consensus.cc:2272] T 57244e68ac9d49629e202a7be0eb0226 P 9ebb903b7402421e9dc6bfb67d537511 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.059882 11310 tablet_replica.cc:333] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511: stopping tablet replica
I20260709 12:21:38.060446 11310 raft_consensus.cc:2243] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.061424 11310 raft_consensus.cc:2272] T 9ddd16927bee475eb509866236b26572 P 9ebb903b7402421e9dc6bfb67d537511 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:38.098896   664 consensus_peers.cc:597] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Couldn't send request to peer 9ebb903b7402421e9dc6bfb67d537511. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:42537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.132473 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
W20260709 12:21:38.144223   664 consensus_peers.cc:597] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Couldn't send request to peer 9ebb903b7402421e9dc6bfb67d537511. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:42537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:38.145181   664 consensus_peers.cc:597] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 -> Peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Couldn't send request to peer 9ebb903b7402421e9dc6bfb67d537511. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:42537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.198333 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
I20260709 12:21:38.255956 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:38.256886 11310 tablet_replica.cc:333] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.258219 11310 raft_consensus.cc:2243] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.258981 11310 raft_consensus.cc:2272] T 66069fa7e25942cd95cc23bd7e1c51ff P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.262089 11310 tablet_replica.cc:333] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.262838 11310 raft_consensus.cc:2243] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.263983 11310 raft_consensus.cc:2272] T 59c0a7890a6f4eda9f0c6c4df133f0c2 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.266769 11310 tablet_replica.cc:333] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.267576 11310 raft_consensus.cc:2243] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.268623 11310 raft_consensus.cc:2272] T 03e67e33e2e44833bb0821377ce57149 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.271541 11310 tablet_replica.cc:333] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.272320 11310 raft_consensus.cc:2243] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.272900 11310 raft_consensus.cc:2272] T 763f2cf7a1fd4ddb83341c75d7579de4 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.275696 11310 tablet_replica.cc:333] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.276422 11310 raft_consensus.cc:2243] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.277024 11310 raft_consensus.cc:2272] T c5380597535a4ae2a7e783db3903ec50 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.280506 11310 tablet_replica.cc:333] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.281464 11310 raft_consensus.cc:2243] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.282382 11310 raft_consensus.cc:2272] T f8d474f3a8f2484abca69e66179b2566 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:38.284238   664 consensus_peers.cc:597] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 -> Peer 9ebb903b7402421e9dc6bfb67d537511 (127.11.11.130:42537): Couldn't send request to peer 9ebb903b7402421e9dc6bfb67d537511. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.130:42537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.286314 11310 tablet_replica.cc:333] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.287119 11310 raft_consensus.cc:2243] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.287747 11310 raft_consensus.cc:2272] T 09c0b37a555e404f910d3c9b5e6224f9 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.290717 11310 tablet_replica.cc:333] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.291441 11310 raft_consensus.cc:2243] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.292520 11310 raft_consensus.cc:2272] T c2a7937ea6a34b53a1c8b339cb02383f P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.295403 11310 tablet_replica.cc:333] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.296130 11310 raft_consensus.cc:2243] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.296757 11310 raft_consensus.cc:2272] T e511c3ec5b5147de9a48e271b24864c6 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.299562 11310 tablet_replica.cc:333] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.300262 11310 raft_consensus.cc:2243] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.300855 11310 raft_consensus.cc:2272] T 4b4a66fdb4594fd5bb195b5b25c21527 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.303777 11310 tablet_replica.cc:333] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.304529 11310 raft_consensus.cc:2243] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.305157 11310 raft_consensus.cc:2272] T 762d8bc9a888413c988f106bd58383c1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.307904 11310 tablet_replica.cc:333] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.308561 11310 raft_consensus.cc:2243] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.309084 11310 raft_consensus.cc:2272] T 4c94ee5bad404294983f9198e5d7eb4e P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.312008 11310 tablet_replica.cc:333] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.312747 11310 raft_consensus.cc:2243] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.313467 11310 raft_consensus.cc:2272] T a32b616ee4b84fae8eb7a251baeb9382 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.316412 11310 tablet_replica.cc:333] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.317145 11310 raft_consensus.cc:2243] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.318320 11310 raft_consensus.cc:2272] T ce78a35bc6404be88f9fcb05fcc88fe1 P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.320906 11310 tablet_replica.cc:333] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.321576 11310 raft_consensus.cc:2243] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.322136 11310 raft_consensus.cc:2272] T 9ddd16927bee475eb509866236b26572 P 27943bf6f68e4f7faeaf374de0336d54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.324887 11310 tablet_replica.cc:333] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.325496 11310 raft_consensus.cc:2243] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.326628 11310 raft_consensus.cc:2272] T 57244e68ac9d49629e202a7be0eb0226 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.329602 11310 tablet_replica.cc:333] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.330241 11310 raft_consensus.cc:2243] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:38.330878 11310 raft_consensus.cc:2272] T 603f9b51bd29455284aec5af830e344c P 27943bf6f68e4f7faeaf374de0336d54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.333477 11310 tablet_replica.cc:333] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54: stopping tablet replica
I20260709 12:21:38.334146 11310 raft_consensus.cc:2243] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.335172 11310 raft_consensus.cc:2272] T 243b8823106f4dfcbebb6b643b2c5844 P 27943bf6f68e4f7faeaf374de0336d54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.386698 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:21:38.430974 11310 master.cc:562] Master@127.11.11.190:34009 shutting down...
I20260709 12:21:38.464890 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:38.465878 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:38.466361 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7e6442e6ceb74de68135e8b91ebfda6e: stopping tablet replica
I20260709 12:21:38.495260 11310 master.cc:584] Master@127.11.11.190:34009 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (21918 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:21:38.561414 11310 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.11.11.190:33481
I20260709 12:21:38.562783 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:38.570472  1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:38.570672  1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:38.571542  1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:38.572187 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:38.573608 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:38.573922 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:38.574083 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599698574064 us; error 0 us; skew 500 ppm
I20260709 12:21:38.574747 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:38.578347 11310 webserver.cc:533] Webserver started at http://127.11.11.190:40317/ using document root <none> and password file <none>
I20260709 12:21:38.579062 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:38.579281 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:38.579638 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:38.581122 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/master-0-root/instance:
uuid: "676e1dec53fc4ebf92c58ae165c0ad1b"
format_stamp: "Formatted at 2026-07-09 12:21:38 on dist-test-slave-4qvp"
I20260709 12:21:38.587304 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20260709 12:21:38.592031  1037 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:38.593166 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:21:38.593587 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/master-0-root
uuid: "676e1dec53fc4ebf92c58ae165c0ad1b"
format_stamp: "Formatted at 2026-07-09 12:21:38 on dist-test-slave-4qvp"
I20260709 12:21:38.594034 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:38.609134 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:38.610641 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:38.662045 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.190:33481
I20260709 12:21:38.662189  1088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.190:33481 every 8 connection(s)
I20260709 12:21:38.666994  1089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:38.684266  1089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b: Bootstrap starting.
I20260709 12:21:38.690886  1089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.696620  1089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b: No bootstrap required, opened a new log
I20260709 12:21:38.699213  1089 raft_consensus.cc:359] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" member_type: VOTER }
I20260709 12:21:38.699700  1089 raft_consensus.cc:385] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.699885  1089 raft_consensus.cc:740] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676e1dec53fc4ebf92c58ae165c0ad1b, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.700497  1089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" member_type: VOTER }
I20260709 12:21:38.701022  1089 raft_consensus.cc:399] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:21:38.701498  1089 raft_consensus.cc:493] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:21:38.701866  1089 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.707681  1089 raft_consensus.cc:515] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" member_type: VOTER }
I20260709 12:21:38.708257  1089 leader_election.cc:304] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 676e1dec53fc4ebf92c58ae165c0ad1b; no voters: 
I20260709 12:21:38.709606  1089 leader_election.cc:290] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:21:38.710433  1093 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.712126  1093 raft_consensus.cc:697] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 1 LEADER]: Becoming Leader. State: Replica: 676e1dec53fc4ebf92c58ae165c0ad1b, State: Running, Role: LEADER
I20260709 12:21:38.713006  1093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" member_type: VOTER }
I20260709 12:21:38.713390  1089 sys_catalog.cc:565] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:21:38.719614  1094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" member_type: VOTER } }
I20260709 12:21:38.720767  1094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:38.720257  1095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 676e1dec53fc4ebf92c58ae165c0ad1b. Latest consensus state: current_term: 1 leader_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676e1dec53fc4ebf92c58ae165c0ad1b" member_type: VOTER } }
I20260709 12:21:38.723027  1095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:38.724591  1102 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:21:38.730914  1102 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:21:38.734396 11310 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:21:38.742903  1102 catalog_manager.cc:1383] Generated new cluster ID: 0b4edfe2471e475488d7f5cebd2d9cae
I20260709 12:21:38.743465  1102 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:21:38.772212  1102 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:21:38.774271  1102 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:21:38.791234  1102 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b: Generated new TSK 0
I20260709 12:21:38.792059  1102 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:21:38.803036 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:38.811231  1111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:38.811733  1112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:38.813931  1114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:38.815052 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:38.816350 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:38.816569 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:38.816812 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599698816755 us; error 0 us; skew 500 ppm
I20260709 12:21:38.817493 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:38.820513 11310 webserver.cc:533] Webserver started at http://127.11.11.129:32895/ using document root <none> and password file <none>
I20260709 12:21:38.821074 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:38.821290 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:38.821609 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:38.823072 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-0-root/instance:
uuid: "0ec884978e874b2c8f605373f1a3b31c"
format_stamp: "Formatted at 2026-07-09 12:21:38 on dist-test-slave-4qvp"
I20260709 12:21:38.828781 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 12:21:38.834270  1119 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:38.835711 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:21:38.836251 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-0-root
uuid: "0ec884978e874b2c8f605373f1a3b31c"
format_stamp: "Formatted at 2026-07-09 12:21:38 on dist-test-slave-4qvp"
I20260709 12:21:38.836547 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:38.851917 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:38.853277 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:38.855130 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:38.858184 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:38.858546 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:38.858968 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:38.859165 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:38.911759 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.129:35743
I20260709 12:21:38.912065  1181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.129:35743 every 8 connection(s)
I20260709 12:21:38.917316 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:38.925860  1186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:38.927954  1187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:38.931864  1189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:38.932359 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:38.933837 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:38.934064 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:38.934216 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599698934204 us; error 0 us; skew 500 ppm
I20260709 12:21:38.934702 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:38.938465 11310 webserver.cc:533] Webserver started at http://127.11.11.130:42021/ using document root <none> and password file <none>
I20260709 12:21:38.939149 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:38.939406 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:38.939754 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:38.940004  1182 heartbeater.cc:344] Connected to a master server at 127.11.11.190:33481
I20260709 12:21:38.940498  1182 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:38.941344  1182 heartbeater.cc:507] Master 127.11.11.190:33481 requested a full tablet report, sending...
I20260709 12:21:38.941351 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-1-root/instance:
uuid: "46c1e152ed374a9a9923d464b3a57077"
format_stamp: "Formatted at 2026-07-09 12:21:38 on dist-test-slave-4qvp"
I20260709 12:21:38.944271  1054 ts_manager.cc:194] Registered new tserver with Master: 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743)
I20260709 12:21:38.947198  1054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49824
I20260709 12:21:38.948724 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:21:38.953318  1194 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:38.954449 11310 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:21:38.954867 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-1-root
uuid: "46c1e152ed374a9a9923d464b3a57077"
format_stamp: "Formatted at 2026-07-09 12:21:38 on dist-test-slave-4qvp"
I20260709 12:21:38.955188 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:38.973213 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:38.974637 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:38.976368 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:38.979408 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:38.979699 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:38.980010 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:38.980226 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:39.033538 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.130:45955
I20260709 12:21:39.033929  1257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.130:45955 every 8 connection(s)
I20260709 12:21:39.039805 11310 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:39.050611  1261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:39.051890  1262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:39.056206  1264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:39.057881  1258 heartbeater.cc:344] Connected to a master server at 127.11.11.190:33481
I20260709 12:21:39.058360 11310 server_base.cc:1061] running on GCE node
I20260709 12:21:39.058861  1258 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:39.059914 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:21:39.060127  1258 heartbeater.cc:507] Master 127.11.11.190:33481 requested a full tablet report, sending...
W20260709 12:21:39.060294 11310 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:39.060585 11310 hybrid_clock.cc:648] HybridClock initialized: now 1783599699060561 us; error 0 us; skew 500 ppm
I20260709 12:21:39.061491 11310 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:39.063748  1054 ts_manager.cc:194] Registered new tserver with Master: 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:39.065114 11310 webserver.cc:533] Webserver started at http://127.11.11.131:39559/ using document root <none> and password file <none>
I20260709 12:21:39.065941 11310 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:39.066174 11310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:39.065981  1054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49828
I20260709 12:21:39.066730 11310 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:39.068512 11310 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-2-root/instance:
uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
format_stamp: "Formatted at 2026-07-09 12:21:39 on dist-test-slave-4qvp"
I20260709 12:21:39.074301 11310 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:21:39.079030  1269 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:39.080348 11310 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:21:39.080822 11310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-2-root
uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
format_stamp: "Formatted at 2026-07-09 12:21:39 on dist-test-slave-4qvp"
I20260709 12:21:39.081187 11310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskESCruo/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783598992211714-11310-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:39.108194 11310 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:39.109889 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:39.111536 11310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:39.114351 11310 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:39.114617 11310 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:39.114887 11310 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:39.115113 11310 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:39.173095 11310 rpc_server.cc:307] RPC server started. Bound to: 127.11.11.131:34345
I20260709 12:21:39.173295  1331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.11.131:34345 every 8 connection(s)
I20260709 12:21:39.192113  1332 heartbeater.cc:344] Connected to a master server at 127.11.11.190:33481
I20260709 12:21:39.192636  1332 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:39.193497  1332 heartbeater.cc:507] Master 127.11.11.190:33481 requested a full tablet report, sending...
I20260709 12:21:39.196370  1054 ts_manager.cc:194] Registered new tserver with Master: feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.197105 11310 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020111489s
I20260709 12:21:39.198592  1054 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49830
I20260709 12:21:39.231467  1054 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49846:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:39.234972  1054 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:39.332845  1147 tablet_service.cc:1511] Processing CreateTablet for tablet e18becdfbf98432c8b8d6f246d349918 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:39.334477  1144 tablet_service.cc:1511] Processing CreateTablet for tablet 5f649afba93c4e33bfbd567a618722ed (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:39.335991  1146 tablet_service.cc:1511] Processing CreateTablet for tablet 80786e2a45414023841a360eaeae692a (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:39.337064  1219 tablet_service.cc:1511] Processing CreateTablet for tablet 5f649afba93c4e33bfbd567a618722ed (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:39.337388  1218 tablet_service.cc:1511] Processing CreateTablet for tablet 7b94e35867004e258238795d4c63b63b (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:39.338587  1219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f649afba93c4e33bfbd567a618722ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.340348  1145 tablet_service.cc:1511] Processing CreateTablet for tablet 265e8ad7291f44ea933719b9ed465c7d (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:39.341154  1142 tablet_service.cc:1511] Processing CreateTablet for tablet f1c4dd1b4b284a28a9ac0def27188792 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:39.343079  1222 tablet_service.cc:1511] Processing CreateTablet for tablet e18becdfbf98432c8b8d6f246d349918 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:39.345171  1141 tablet_service.cc:1511] Processing CreateTablet for tablet f6a40443b2994b42b552928f555d04b5 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:39.345942  1143 tablet_service.cc:1511] Processing CreateTablet for tablet 7b94e35867004e258238795d4c63b63b (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:39.334553  1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18becdfbf98432c8b8d6f246d349918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.348595  1221 tablet_service.cc:1511] Processing CreateTablet for tablet 80786e2a45414023841a360eaeae692a (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:39.350083  1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a40443b2994b42b552928f555d04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.350932  1220 tablet_service.cc:1511] Processing CreateTablet for tablet 265e8ad7291f44ea933719b9ed465c7d (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:39.353190  1143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b94e35867004e258238795d4c63b63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.343189  1218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b94e35867004e258238795d4c63b63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.357169  1220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265e8ad7291f44ea933719b9ed465c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.358073  1144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f649afba93c4e33bfbd567a618722ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.359988  1142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c4dd1b4b284a28a9ac0def27188792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.360495  1140 tablet_service.cc:1511] Processing CreateTablet for tablet 224dcfc0396244728eb039ca30e4cf9a (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:39.368001  1140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224dcfc0396244728eb039ca30e4cf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.369779  1347 tablet_bootstrap.cc:492] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:39.370684  1145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265e8ad7291f44ea933719b9ed465c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.374117  1221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80786e2a45414023841a360eaeae692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.377085  1347 tablet_bootstrap.cc:654] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.377897  1222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18becdfbf98432c8b8d6f246d349918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.363742  1139 tablet_service.cc:1511] Processing CreateTablet for tablet c6aba73b5e90440bb2dba4975d845d60 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:39.382396  1296 tablet_service.cc:1511] Processing CreateTablet for tablet 80786e2a45414023841a360eaeae692a (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:39.383383  1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6aba73b5e90440bb2dba4975d845d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.384167  1296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80786e2a45414023841a360eaeae692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.386348  1297 tablet_service.cc:1511] Processing CreateTablet for tablet e18becdfbf98432c8b8d6f246d349918 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:39.387327  1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80786e2a45414023841a360eaeae692a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.387925  1297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18becdfbf98432c8b8d6f246d349918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.389935  1347 tablet_bootstrap.cc:492] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:39.390568  1347 ts_tablet_manager.cc:1403] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.007s
I20260709 12:21:39.391517  1295 tablet_service.cc:1511] Processing CreateTablet for tablet 265e8ad7291f44ea933719b9ed465c7d (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:39.392999  1217 tablet_service.cc:1511] Processing CreateTablet for tablet f6a40443b2994b42b552928f555d04b5 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:39.393725  1295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 265e8ad7291f44ea933719b9ed465c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.394510  1217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a40443b2994b42b552928f555d04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.394131  1294 tablet_service.cc:1511] Processing CreateTablet for tablet 5f649afba93c4e33bfbd567a618722ed (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:39.395692  1294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f649afba93c4e33bfbd567a618722ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.391896  1219 tablet_service.cc:1511] Processing CreateTablet for tablet f1c4dd1b4b284a28a9ac0def27188792 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:39.398662  1219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c4dd1b4b284a28a9ac0def27188792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.398736  1293 tablet_service.cc:1511] Processing CreateTablet for tablet 7b94e35867004e258238795d4c63b63b (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:39.400830  1293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b94e35867004e258238795d4c63b63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.403981  1292 tablet_service.cc:1511] Processing CreateTablet for tablet f1c4dd1b4b284a28a9ac0def27188792 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:39.406591  1292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1c4dd1b4b284a28a9ac0def27188792. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.411631  1291 tablet_service.cc:1511] Processing CreateTablet for tablet f6a40443b2994b42b552928f555d04b5 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:39.413309  1215 tablet_service.cc:1511] Processing CreateTablet for tablet c6aba73b5e90440bb2dba4975d845d60 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:39.414906  1215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6aba73b5e90440bb2dba4975d845d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.415607  1216 tablet_service.cc:1511] Processing CreateTablet for tablet 224dcfc0396244728eb039ca30e4cf9a (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:39.418476  1216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224dcfc0396244728eb039ca30e4cf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.422216  1291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a40443b2994b42b552928f555d04b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.422422  1347 raft_consensus.cc:359] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.424752  1347 raft_consensus.cc:385] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.425102  1347 raft_consensus.cc:740] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.432030  1289 tablet_service.cc:1511] Processing CreateTablet for tablet c6aba73b5e90440bb2dba4975d845d60 (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:39.432030  1290 tablet_service.cc:1511] Processing CreateTablet for tablet 224dcfc0396244728eb039ca30e4cf9a (DEFAULT_TABLE table=test-workload [id=a0c5fd701609403387ba4889d3c9a5e9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:39.434252  1289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6aba73b5e90440bb2dba4975d845d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.440188  1290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224dcfc0396244728eb039ca30e4cf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:39.563172  1349 tablet_bootstrap.cc:492] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:39.554512  1347 consensus_queue.cc:260] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.583755  1350 raft_consensus.cc:493] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.584968  1350 raft_consensus.cc:515] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.609464  1350 leader_election.cc:290] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.610766  1258 heartbeater.cc:499] Master 127.11.11.190:33481 was elected leader, sending a full tablet report...
I20260709 12:21:39.635138  1349 tablet_bootstrap.cc:654] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.636787  1347 ts_tablet_manager.cc:1434] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.221s	user 0.020s	sys 0.052s
I20260709 12:21:39.650192  1351 tablet_bootstrap.cc:492] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:39.663587  1347 tablet_bootstrap.cc:492] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:39.676131  1349 tablet_bootstrap.cc:492] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:39.677003  1349 ts_tablet_manager.cc:1403] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.114s	user 0.010s	sys 0.023s
I20260709 12:21:39.681720  1351 tablet_bootstrap.cc:654] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.685035  1349 raft_consensus.cc:359] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:39.686328  1349 raft_consensus.cc:385] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.686702  1349 raft_consensus.cc:740] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.686921  1347 tablet_bootstrap.cc:654] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.687875  1349 consensus_queue.cc:260] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:39.698874  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:39.703747  1349 ts_tablet_manager.cc:1434] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.008s
W20260709 12:21:39.704098  1198 leader_election.cc:343] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.706547  1182 heartbeater.cc:499] Master 127.11.11.190:33481 was elected leader, sending a full tablet report...
I20260709 12:21:39.706879  1349 tablet_bootstrap.cc:492] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:39.709240  1351 tablet_bootstrap.cc:492] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:39.710021  1351 ts_tablet_manager.cc:1403] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.060s	user 0.016s	sys 0.012s
I20260709 12:21:39.713755  1351 raft_consensus.cc:359] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.714651  1351 raft_consensus.cc:385] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.715066  1349 tablet_bootstrap.cc:654] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.714998  1351 raft_consensus.cc:740] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.716403  1351 consensus_queue.cc:260] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.721695  1347 tablet_bootstrap.cc:492] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:39.722549  1347 ts_tablet_manager.cc:1403] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.059s	user 0.007s	sys 0.013s
I20260709 12:21:39.725486  1332 heartbeater.cc:499] Master 127.11.11.190:33481 was elected leader, sending a full tablet report...
W20260709 12:21:39.728082  1103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:39.734251  1347 raft_consensus.cc:359] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.735321  1347 raft_consensus.cc:385] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.735662  1347 raft_consensus.cc:740] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.735740  1351 ts_tablet_manager.cc:1434] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.025s	user 0.010s	sys 0.004s
I20260709 12:21:39.736701  1347 consensus_queue.cc:260] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.737589  1351 tablet_bootstrap.cc:492] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:39.739955  1347 ts_tablet_manager.cc:1434] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.000s
I20260709 12:21:39.741480  1347 tablet_bootstrap.cc:492] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:39.743227  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:39.746773  1349 tablet_bootstrap.cc:492] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
W20260709 12:21:39.746860  1198 leader_election.cc:343] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.747540  1349 ts_tablet_manager.cc:1403] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.041s	user 0.019s	sys 0.014s
I20260709 12:21:39.748135  1351 tablet_bootstrap.cc:654] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.747918  1198 leader_election.cc:304] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:39.749723  1347 tablet_bootstrap.cc:654] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.749722  1352 raft_consensus.cc:2749] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.751690  1349 raft_consensus.cc:359] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.753123  1349 raft_consensus.cc:385] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.753597  1349 raft_consensus.cc:740] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.755076  1349 consensus_queue.cc:260] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.758282  1349 ts_tablet_manager.cc:1434] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.006s
I20260709 12:21:39.759537  1349 tablet_bootstrap.cc:492] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:39.760466  1351 tablet_bootstrap.cc:492] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:39.761089  1351 ts_tablet_manager.cc:1403] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.003s
I20260709 12:21:39.764398  1351 raft_consensus.cc:359] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.765184  1351 raft_consensus.cc:385] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.765458  1351 raft_consensus.cc:740] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.767005  1349 tablet_bootstrap.cc:654] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.766425  1351 consensus_queue.cc:260] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.769311  1351 ts_tablet_manager.cc:1434] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:21:39.770381  1347 tablet_bootstrap.cc:492] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:39.770550  1351 tablet_bootstrap.cc:492] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:39.771130  1347 ts_tablet_manager.cc:1403] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.030s	user 0.019s	sys 0.007s
I20260709 12:21:39.774910  1347 raft_consensus.cc:359] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.775776  1347 raft_consensus.cc:385] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.776077  1347 raft_consensus.cc:740] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.777076  1347 consensus_queue.cc:260] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.779376  1349 tablet_bootstrap.cc:492] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:39.779732  1351 tablet_bootstrap.cc:654] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.779821  1347 ts_tablet_manager.cc:1434] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.002s
I20260709 12:21:39.780040  1349 ts_tablet_manager.cc:1403] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.005s
I20260709 12:21:39.781278  1347 tablet_bootstrap.cc:492] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:39.784186  1349 raft_consensus.cc:359] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.785257  1349 raft_consensus.cc:385] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.785672  1349 raft_consensus.cc:740] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.786830  1349 consensus_queue.cc:260] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
W20260709 12:21:39.789472  1259 tablet.cc:2406] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:21:39.789969  1347 tablet_bootstrap.cc:654] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.790171  1349 ts_tablet_manager.cc:1434] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:21:39.791400  1349 tablet_bootstrap.cc:492] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:39.792199  1351 tablet_bootstrap.cc:492] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:39.792796  1351 ts_tablet_manager.cc:1403] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.000s
I20260709 12:21:39.796675  1351 raft_consensus.cc:359] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.797497  1351 raft_consensus.cc:385] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.797907  1351 raft_consensus.cc:740] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.798971  1349 tablet_bootstrap.cc:654] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.798872  1351 consensus_queue.cc:260] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.801787  1351 ts_tablet_manager.cc:1434] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:21:39.803176  1347 tablet_bootstrap.cc:492] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:39.803342  1351 tablet_bootstrap.cc:492] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:39.804174  1347 ts_tablet_manager.cc:1403] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.007s
I20260709 12:21:39.808120  1347 raft_consensus.cc:359] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.809016  1347 raft_consensus.cc:385] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.809335  1347 raft_consensus.cc:740] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.810135  1347 consensus_queue.cc:260] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.811345  1349 tablet_bootstrap.cc:492] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:39.812052  1349 ts_tablet_manager.cc:1403] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 12:21:39.812866  1347 ts_tablet_manager.cc:1434] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:21:39.812974  1351 tablet_bootstrap.cc:654] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.814576  1347 tablet_bootstrap.cc:492] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:39.815585  1349 raft_consensus.cc:359] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.817093  1358 raft_consensus.cc:493] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.817845  1358 raft_consensus.cc:515] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.822992  1347 tablet_bootstrap.cc:654] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.827200  1358 leader_election.cc:290] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.816475  1349 raft_consensus.cc:385] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.832746  1349 raft_consensus.cc:740] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.835444  1349 consensus_queue.cc:260] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.839227  1352 raft_consensus.cc:493] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.839378  1351 tablet_bootstrap.cc:492] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:39.840153  1352 raft_consensus.cc:515] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.841032  1351 ts_tablet_manager.cc:1403] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.038s	user 0.018s	sys 0.007s
I20260709 12:21:39.844473  1351 raft_consensus.cc:359] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.844903  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:39.845245  1351 raft_consensus.cc:385] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.845543  1351 raft_consensus.cc:740] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.846306  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
W20260709 12:21:39.846850  1198 leader_election.cc:343] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.847018  1351 consensus_queue.cc:260] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.840304  1349 ts_tablet_manager.cc:1434] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.004s
W20260709 12:21:39.848877  1198 leader_election.cc:343] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.849793  1349 tablet_bootstrap.cc:492] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:39.849751  1198 leader_election.cc:304] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:39.849762  1351 ts_tablet_manager.cc:1434] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:21:39.851012  1350 raft_consensus.cc:2749] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.852890  1351 tablet_bootstrap.cc:492] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:39.859144  1350 raft_consensus.cc:493] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.860281  1351 tablet_bootstrap.cc:654] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.860080  1350 raft_consensus.cc:515] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.864857  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:39.865845  1157 raft_consensus.cc:2468] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:39.866166  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:39.867281  1307 raft_consensus.cc:2468] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:39.867403  1198 leader_election.cc:304] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:39.869211  1350 leader_election.cc:290] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.870250  1350 raft_consensus.cc:2804] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.870831  1350 raft_consensus.cc:493] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.873085  1351 tablet_bootstrap.cc:492] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:39.884591  1351 ts_tablet_manager.cc:1403] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.032s	user 0.014s	sys 0.005s
I20260709 12:21:39.874823  1349 tablet_bootstrap.cc:654] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.846182  1352 leader_election.cc:290] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.879637  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:39.887841  1351 raft_consensus.cc:359] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:39.888751  1351 raft_consensus.cc:385] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.881506  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:39.881601  1359 raft_consensus.cc:493] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.889093  1351 raft_consensus.cc:740] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.889575  1359 raft_consensus.cc:515] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.890054  1351 consensus_queue.cc:260] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:39.890223  1350 raft_consensus.cc:3060] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.902554  1359 leader_election.cc:290] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:39.903656  1349 tablet_bootstrap.cc:492] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:39.905823  1349 ts_tablet_manager.cc:1403] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.056s	user 0.019s	sys 0.004s
I20260709 12:21:39.907160  1350 raft_consensus.cc:515] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.904183  1307 raft_consensus.cc:2468] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 0.
I20260709 12:21:39.910416  1232 raft_consensus.cc:2393] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ec884978e874b2c8f605373f1a3b31c in current term 1: Already voted for candidate 46c1e152ed374a9a9923d464b3a57077 in this term.
I20260709 12:21:39.910856  1123 leader_election.cc:304] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:39.912350  1358 raft_consensus.cc:2804] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.912878  1358 raft_consensus.cc:493] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.913345  1358 raft_consensus.cc:3060] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.910467  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:39.919590  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:39.920445  1306 raft_consensus.cc:3060] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.909001  1349 raft_consensus.cc:359] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.923501  1347 tablet_bootstrap.cc:492] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:39.923933  1349 raft_consensus.cc:385] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.924376  1349 raft_consensus.cc:740] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.924371  1347 ts_tablet_manager.cc:1403] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.110s	user 0.030s	sys 0.052s
I20260709 12:21:39.923601  1358 raft_consensus.cc:515] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.927784  1157 raft_consensus.cc:2393] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 46c1e152ed374a9a9923d464b3a57077 in current term 1: Already voted for candidate 0ec884978e874b2c8f605373f1a3b31c in this term.
I20260709 12:21:39.929443  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:39.929709  1306 raft_consensus.cc:2468] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:39.936450  1232 raft_consensus.cc:2393] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ec884978e874b2c8f605373f1a3b31c in current term 1: Already voted for candidate 46c1e152ed374a9a9923d464b3a57077 in this term.
I20260709 12:21:39.930708  1358 leader_election.cc:290] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.930925  1350 leader_election.cc:290] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:39.936992  1198 leader_election.cc:304] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:39.932014  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1c4dd1b4b284a28a9ac0def27188792" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:39.938316  1307 raft_consensus.cc:2393] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ec884978e874b2c8f605373f1a3b31c in current term 1: Already voted for candidate 46c1e152ed374a9a9923d464b3a57077 in this term.
I20260709 12:21:39.938660  1350 raft_consensus.cc:2804] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.939255  1350 raft_consensus.cc:697] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:39.940436  1123 leader_election.cc:304] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c; no voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:39.932943  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:39.935021  1347 raft_consensus.cc:359] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.940797  1350 consensus_queue.cc:237] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.944262  1157 raft_consensus.cc:2468] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:39.945079  1347 raft_consensus.cc:385] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.946306  1358 raft_consensus.cc:2749] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.946187  1273 leader_election.cc:304] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:39.947314  1347 raft_consensus.cc:740] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.948427  1347 consensus_queue.cc:260] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.945529  1359 raft_consensus.cc:493] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.951166  1359 raft_consensus.cc:515] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.949134  1351 ts_tablet_manager.cc:1434] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.064s	user 0.020s	sys 0.009s
I20260709 12:21:39.949709  1374 raft_consensus.cc:2804] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.953060  1374 raft_consensus.cc:493] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.953459  1374 raft_consensus.cc:3060] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.954020  1359 leader_election.cc:290] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:39.955307  1351 tablet_bootstrap.cc:492] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:39.955853  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6aba73b5e90440bb2dba4975d845d60" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:39.933527  1349 consensus_queue.cc:260] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
W20260709 12:21:39.958988  1273 leader_election.cc:343] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.959276  1349 ts_tablet_manager.cc:1434] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.053s	user 0.007s	sys 0.000s
I20260709 12:21:39.952417  1347 ts_tablet_manager.cc:1434] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.002s
I20260709 12:21:39.962831  1347 tablet_bootstrap.cc:492] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:39.964766  1351 tablet_bootstrap.cc:654] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.967203  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:39.969173  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6aba73b5e90440bb2dba4975d845d60" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
W20260709 12:21:39.969614  1271 leader_election.cc:343] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.970045  1232 raft_consensus.cc:2468] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:39.971884  1351 tablet_bootstrap.cc:492] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:39.972659  1351 ts_tablet_manager.cc:1403] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.005s
I20260709 12:21:39.973171  1271 leader_election.cc:304] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:39.974344  1359 raft_consensus.cc:2804] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.974939  1359 raft_consensus.cc:493] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.975147  1347 tablet_bootstrap.cc:654] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.975360  1359 raft_consensus.cc:3060] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.975459  1349 tablet_bootstrap.cc:492] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:39.976759  1351 raft_consensus.cc:359] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.977718  1351 raft_consensus.cc:385] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.978132  1351 raft_consensus.cc:740] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.979179  1351 consensus_queue.cc:260] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.980181  1374 raft_consensus.cc:515] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:39.983672  1374 leader_election.cc:290] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:39.986155  1349 tablet_bootstrap.cc:654] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.988773  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:39.989611  1157 raft_consensus.cc:3060] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.994563  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.001433  1349 tablet_bootstrap.cc:492] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:40.002707  1349 ts_tablet_manager.cc:1403] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.027s	user 0.017s	sys 0.008s
W20260709 12:21:40.007319  1271 leader_election.cc:343] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.007875  1349 raft_consensus.cc:359] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.009229  1349 raft_consensus.cc:385] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.010157  1349 raft_consensus.cc:740] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.011464  1349 consensus_queue.cc:260] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.012638  1359 raft_consensus.cc:515] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.000286  1054 catalog_manager.cc:5697] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.019939  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6aba73b5e90440bb2dba4975d845d60" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:40.020754  1156 raft_consensus.cc:3060] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.021294  1359 leader_election.cc:290] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.023672  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6aba73b5e90440bb2dba4975d845d60" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.024336  1349 ts_tablet_manager.cc:1434] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.003s
I20260709 12:21:40.024852  1232 raft_consensus.cc:3060] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.026757  1349 tablet_bootstrap.cc:492] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:40.027640  1358 raft_consensus.cc:493] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.028234  1358 raft_consensus.cc:515] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.032652  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.034238  1231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.036296  1349 tablet_bootstrap.cc:654] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:40.035339  1123 leader_election.cc:343] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.035472  1358 leader_election.cc:290] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
W20260709 12:21:40.037930  1121 leader_election.cc:343] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.038828  1121 leader_election.cc:304] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c; no voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.040795  1358 raft_consensus.cc:2749] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:40.043694  1351 ts_tablet_manager.cc:1434] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.071s	user 0.023s	sys 0.020s
I20260709 12:21:40.057034  1347 tablet_bootstrap.cc:492] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:40.057902  1347 ts_tablet_manager.cc:1403] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.095s	user 0.033s	sys 0.027s
I20260709 12:21:40.059759  1351 tablet_bootstrap.cc:492] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:40.049037  1157 raft_consensus.cc:2468] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.046803  1156 raft_consensus.cc:2468] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.061928  1347 raft_consensus.cc:359] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.062882  1347 raft_consensus.cc:385] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.063254  1347 raft_consensus.cc:740] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.064450  1347 consensus_queue.cc:260] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.061084  1232 raft_consensus.cc:2468] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.067057  1347 ts_tablet_manager.cc:1434] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:21:40.068159  1271 leader_election.cc:304] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:40.062978  1273 leader_election.cc:304] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:40.069170  1379 raft_consensus.cc:2804] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.069896  1379 raft_consensus.cc:697] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.070638  1379 consensus_queue.cc:237] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.070996  1359 raft_consensus.cc:2804] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.068719  1347 tablet_bootstrap.cc:492] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:40.078806  1351 tablet_bootstrap.cc:654] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.079768  1359 raft_consensus.cc:697] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.080888  1359 consensus_queue.cc:237] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.083004  1347 tablet_bootstrap.cc:654] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.093901  1351 tablet_bootstrap.cc:492] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:40.089468  1349 tablet_bootstrap.cc:492] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:40.094758  1351 ts_tablet_manager.cc:1403] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.035s	user 0.009s	sys 0.013s
I20260709 12:21:40.094759  1349 ts_tablet_manager.cc:1403] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.068s	user 0.018s	sys 0.019s
I20260709 12:21:40.098980  1351 raft_consensus.cc:359] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.100596  1351 raft_consensus.cc:385] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.101150  1351 raft_consensus.cc:740] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.102546  1351 consensus_queue.cc:260] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.103248  1347 tablet_bootstrap.cc:492] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:40.104012  1347 ts_tablet_manager.cc:1403] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.017s
I20260709 12:21:40.105667  1351 ts_tablet_manager.cc:1434] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 12:21:40.107465  1351 tablet_bootstrap.cc:492] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:40.107259  1347 raft_consensus.cc:359] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.108321  1347 raft_consensus.cc:385] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.108945  1358 raft_consensus.cc:493] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.109512  1358 raft_consensus.cc:515] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.111651  1358 leader_election.cc:290] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.113020  1347 raft_consensus.cc:740] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.114033  1347 consensus_queue.cc:260] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.115411  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.116776  1347 ts_tablet_manager.cc:1434] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
W20260709 12:21:40.117429  1123 leader_election.cc:343] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.117705  1359 raft_consensus.cc:493] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.118160  1347 tablet_bootstrap.cc:492] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:40.118410  1351 tablet_bootstrap.cc:654] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.118415  1359 raft_consensus.cc:515] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.119037  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.119829  1232 raft_consensus.cc:2468] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 0.
I20260709 12:21:40.121323  1121 leader_election.cc:304] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.122579  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:40.122674  1358 raft_consensus.cc:2804] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:40.124873  1358 raft_consensus.cc:493] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.125341  1358 raft_consensus.cc:3060] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.127656  1347 tablet_bootstrap.cc:654] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:40.130820  1273 leader_election.cc:343] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.100921  1349 raft_consensus.cc:359] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.133688  1349 raft_consensus.cc:385] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.134020  1349 raft_consensus.cc:740] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.135075  1349 consensus_queue.cc:260] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.140522  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.141808  1349 ts_tablet_manager.cc:1434] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.044s	user 0.011s	sys 0.000s
I20260709 12:21:40.142087  1232 raft_consensus.cc:2468] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:40.143627  1359 leader_election.cc:290] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.144429  1271 leader_election.cc:304] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.145920  1359 raft_consensus.cc:2804] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:40.146366  1359 raft_consensus.cc:493] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.146762  1359 raft_consensus.cc:3060] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.147063  1358 raft_consensus.cc:515] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.149683  1358 leader_election.cc:290] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.152720  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.154953  1051 catalog_manager.cc:5697] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:40.155999  1123 leader_election.cc:343] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.156960  1359 raft_consensus.cc:515] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.154943  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.158099  1393 raft_consensus.cc:493] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.158375  1232 raft_consensus.cc:3060] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.160887  1351 tablet_bootstrap.cc:492] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:40.160591  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:40.161429  1351 ts_tablet_manager.cc:1403] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.054s	user 0.028s	sys 0.008s
W20260709 12:21:40.162554  1273 leader_election.cc:343] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.164839  1351 raft_consensus.cc:359] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.165257  1231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.165748  1351 raft_consensus.cc:385] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.165964  1231 raft_consensus.cc:3060] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.166114  1351 raft_consensus.cc:740] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.167203  1351 consensus_queue.cc:260] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.170050  1351 ts_tablet_manager.cc:1434] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:21:40.171219  1351 tablet_bootstrap.cc:492] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:40.175854  1359 leader_election.cc:290] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.176745  1393 raft_consensus.cc:515] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:40.184149  1347 tablet_bootstrap.cc:492] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:40.185021  1347 ts_tablet_manager.cc:1403] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.067s	user 0.025s	sys 0.030s
I20260709 12:21:40.186343  1393 leader_election.cc:290] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.157399  1051 catalog_manager.cc:5697] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:40.188077  1230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.188592  1347 raft_consensus.cc:359] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.189375  1347 raft_consensus.cc:385] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.189695  1347 raft_consensus.cc:740] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
W20260709 12:21:40.190018  1121 leader_election.cc:343] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.190564  1347 consensus_queue.cc:260] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.188171  1231 raft_consensus.cc:2468] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.191880  1351 tablet_bootstrap.cc:654] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.192420  1271 leader_election.cc:304] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.192998  1347 ts_tablet_manager.cc:1434] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260709 12:21:40.193801  1359 raft_consensus.cc:2804] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.194382  1359 raft_consensus.cc:697] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.194762  1347 tablet_bootstrap.cc:492] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:40.195720  1379 raft_consensus.cc:493] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.196437  1379 raft_consensus.cc:515] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.197769  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.198675  1307 raft_consensus.cc:2468] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 0.
I20260709 12:21:40.200441  1123 leader_election.cc:304] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:40.201428  1393 raft_consensus.cc:2804] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:40.201915  1393 raft_consensus.cc:493] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.202250  1393 raft_consensus.cc:3060] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.202812  1347 tablet_bootstrap.cc:654] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.188071  1349 tablet_bootstrap.cc:492] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:40.206234  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
W20260709 12:21:40.209237  1273 leader_election.cc:343] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.210253  1359 consensus_queue.cc:237] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.214208  1231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.212651  1393 raft_consensus.cc:515] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:40.215229  1231 raft_consensus.cc:2468] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:40.216804  1271 leader_election.cc:304] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.217882  1385 raft_consensus.cc:2804] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:40.218416  1385 raft_consensus.cc:493] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.218827  1385 raft_consensus.cc:3060] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.218827  1231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
W20260709 12:21:40.221370  1121 leader_election.cc:343] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.222146  1393 leader_election.cc:290] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Requested vote from peers feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.212266  1379 leader_election.cc:290] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.224820  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.225564  1347 tablet_bootstrap.cc:492] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:40.226325  1347 ts_tablet_manager.cc:1403] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.032s	user 0.016s	sys 0.003s
I20260709 12:21:40.225773  1307 raft_consensus.cc:3060] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.229390  1347 raft_consensus.cc:359] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:40.230307  1347 raft_consensus.cc:385] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.229981  1385 raft_consensus.cc:515] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.230659  1347 raft_consensus.cc:740] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.231592  1347 consensus_queue.cc:260] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:40.233088  1232 raft_consensus.cc:2468] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 1.
I20260709 12:21:40.234983  1347 ts_tablet_manager.cc:1434] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:21:40.238412  1349 tablet_bootstrap.cc:654] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.245723  1121 leader_election.cc:304] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.247359  1393 raft_consensus.cc:2804] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.247524  1385 leader_election.cc:290] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.247936  1393 raft_consensus.cc:697] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:40.250123  1393 consensus_queue.cc:237] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.249365  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
W20260709 12:21:40.252646  1273 leader_election.cc:343] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:40.253787  1351 tablet_bootstrap.cc:492] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:40.254438  1351 ts_tablet_manager.cc:1403] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.083s	user 0.022s	sys 0.009s
I20260709 12:21:40.254510  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.256940  1232 raft_consensus.cc:3060] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.257854  1351 raft_consensus.cc:359] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.258697  1351 raft_consensus.cc:385] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.259006  1351 raft_consensus.cc:740] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.260031  1351 consensus_queue.cc:260] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.262759  1351 ts_tablet_manager.cc:1434] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:21:40.250844  1054 catalog_manager.cc:5697] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:40.267475  1232 raft_consensus.cc:2468] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.269798  1307 raft_consensus.cc:2468] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 1.
I20260709 12:21:40.269989  1271 leader_election.cc:304] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.271658  1123 leader_election.cc:304] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:40.271977  1385 raft_consensus.cc:2804] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.272693  1385 raft_consensus.cc:697] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.273062  1393 raft_consensus.cc:2804] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.273973  1393 raft_consensus.cc:697] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:40.273736  1385 consensus_queue.cc:237] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.275014  1393 consensus_queue.cc:237] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:40.279641  1349 tablet_bootstrap.cc:492] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:40.280357  1349 ts_tablet_manager.cc:1403] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.093s	user 0.014s	sys 0.000s
I20260709 12:21:40.283634  1349 raft_consensus.cc:359] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.284411  1349 raft_consensus.cc:385] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.284668  1349 raft_consensus.cc:740] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.284862  1052 catalog_manager.cc:5697] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 1 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.285734  1349 consensus_queue.cc:260] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.288507  1349 ts_tablet_manager.cc:1434] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:21:40.289661  1349 tablet_bootstrap.cc:492] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:40.296470  1349 tablet_bootstrap.cc:654] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:40.302564  1054 catalog_manager.cc:5697] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:40.310094  1349 tablet_bootstrap.cc:492] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:40.310773  1349 ts_tablet_manager.cc:1403] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.010s
I20260709 12:21:40.312867  1052 catalog_manager.cc:5697] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 1 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.314006  1349 raft_consensus.cc:359] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.314854  1349 raft_consensus.cc:385] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:40.315161  1349 raft_consensus.cc:740] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:40.316126  1349 consensus_queue.cc:260] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.318704  1349 ts_tablet_manager.cc:1434] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:21:40.330506  1393 raft_consensus.cc:493] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.331121  1393 raft_consensus.cc:515] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.332571  1385 raft_consensus.cc:493] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.333323  1385 raft_consensus.cc:515] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.333596  1350 raft_consensus.cc:493] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.334498  1350 raft_consensus.cc:515] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.334725  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.334843  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.335726  1385 leader_election.cc:290] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.333339  1393 leader_election.cc:290] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.337038  1350 leader_election.cc:290] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.337415  1231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.337894  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:40.338397  1231 raft_consensus.cc:2393] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in current term 1: Already voted for candidate 0ec884978e874b2c8f605373f1a3b31c in this term.
I20260709 12:21:40.335759  1232 raft_consensus.cc:2393] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ec884978e874b2c8f605373f1a3b31c in current term 1: Already voted for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in this term.
I20260709 12:21:40.339710  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:40.339732  1271 leader_election.cc:304] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:40.340152  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.340453  1121 leader_election.cc:304] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c; no voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.340749  1156 raft_consensus.cc:2468] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:40.340866  1307 raft_consensus.cc:2468] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:40.341408  1385 raft_consensus.cc:3060] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.341961  1198 leader_election.cc:304] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:40.342338  1393 raft_consensus.cc:3060] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.343753  1350 raft_consensus.cc:2804] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:40.344266  1350 raft_consensus.cc:493] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.344568  1350 raft_consensus.cc:3060] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.350229  1385 raft_consensus.cc:2749] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:40.350229  1393 raft_consensus.cc:2749] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:40.351722  1350 raft_consensus.cc:515] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.353924  1350 leader_election.cc:290] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.355099  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.354957  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:40.355834  1307 raft_consensus.cc:3060] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.355911  1156 raft_consensus.cc:3060] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.365037  1156 raft_consensus.cc:2468] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:40.365036  1307 raft_consensus.cc:2468] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:40.366650  1198 leader_election.cc:304] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:40.367534  1350 raft_consensus.cc:2804] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.368156  1350 raft_consensus.cc:697] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:40.369103  1350 consensus_queue.cc:237] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.370723  1352 consensus_queue.cc:1048] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.380707  1052 catalog_manager.cc:5697] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.392380  1405 consensus_queue.cc:1048] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.413870  1385 raft_consensus.cc:493] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.414529  1385 raft_consensus.cc:515] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.414947  1350 raft_consensus.cc:493] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.415478  1350 raft_consensus.cc:515] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:40.417826  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:40.418707  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.418869  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:40.418865  1156 raft_consensus.cc:2468] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:40.419564  1232 raft_consensus.cc:2468] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:40.418099  1350 leader_election.cc:290] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.420892  1385 leader_election.cc:290] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.420746  1273 leader_election.cc:304] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:40.422009  1385 raft_consensus.cc:2804] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:40.422434  1385 raft_consensus.cc:493] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.422750  1385 raft_consensus.cc:3060] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.423099  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.424183  1307 raft_consensus.cc:2393] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46c1e152ed374a9a9923d464b3a57077 in current term 1: Already voted for candidate 0ec884978e874b2c8f605373f1a3b31c in this term.
I20260709 12:21:40.425511  1198 leader_election.cc:304] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.426658  1350 raft_consensus.cc:3060] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.430936  1385 raft_consensus.cc:515] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.433231  1385 leader_election.cc:290] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:40.433328  1350 raft_consensus.cc:2749] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:40.434298  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:40.434719  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.434940  1156 raft_consensus.cc:3060] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.435457  1232 raft_consensus.cc:3060] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.442451  1232 raft_consensus.cc:2468] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.443492  1156 raft_consensus.cc:2468] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:40.443785  1271 leader_election.cc:304] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:40.444880  1385 raft_consensus.cc:2804] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:40.445574  1385 raft_consensus.cc:697] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.446674  1385 consensus_queue.cc:237] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.456863  1054 catalog_manager.cc:5697] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:40.485939  1385 consensus_queue.cc:1048] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.488399  1350 raft_consensus.cc:493] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.489027  1350 raft_consensus.cc:515] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.491344  1350 leader_election.cc:290] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.492774  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.492985  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:40.495304  1198 leader_election.cc:304] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.496558  1350 raft_consensus.cc:2749] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:40.498368  1232 raft_consensus.cc:3060] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.509052  1379 consensus_queue.cc:1048] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.542320  1393 raft_consensus.cc:493] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.543001  1393 raft_consensus.cc:515] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.544919  1393 leader_election.cc:290] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.546161  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:40.546262  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:40.547282  1232 raft_consensus.cc:2393] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ec884978e874b2c8f605373f1a3b31c in current term 1: Already voted for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in this term.
I20260709 12:21:40.548560  1121 leader_election.cc:304] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c; no voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:40.549429  1393 raft_consensus.cc:3060] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:40.557289  1393 raft_consensus.cc:2749] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:40.622330  1379 consensus_queue.cc:1048] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.631407  1307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "80786e2a45414023841a360eaeae692a"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:36894
I20260709 12:21:40.632236  1307 raft_consensus.cc:606] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.639130  1359 consensus_queue.cc:1048] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.640242  1307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:36894
I20260709 12:21:40.641037  1307 raft_consensus.cc:606] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.648708  1307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:36894
I20260709 12:21:40.649588  1307 raft_consensus.cc:606] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:40.668193  1156 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b94e35867004e258238795d4c63b63b"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
mode: GRACEFUL
new_leader_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:51274
I20260709 12:21:40.668931  1156 raft_consensus.cc:606] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Received request to transfer leadership to TS 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:40.671017  1393 consensus_queue.cc:1048] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.680898  1307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
mode: GRACEFUL
new_leader_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:36894
I20260709 12:21:40.681838  1307 raft_consensus.cc:606] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Received request to transfer leadership to TS 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:40.687311  1393 consensus_queue.cc:1048] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.713326  1358 consensus_queue.cc:1048] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.714438  1359 consensus_queue.cc:1048] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.728427  1418 consensus_queue.cc:1048] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.740485  1379 consensus_queue.cc:1048] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.752560  1379 raft_consensus.cc:993] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:40.753135  1385 raft_consensus.cc:1081] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:40.757357  1156 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "80786e2a45414023841a360eaeae692a"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51260
I20260709 12:21:40.758431  1156 raft_consensus.cc:493] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:40.758998  1156 raft_consensus.cc:3060] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.767499  1418 raft_consensus.cc:993] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: : Instructing follower 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:40.769377  1156 raft_consensus.cc:515] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.768292  1393 raft_consensus.cc:1081] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Signalling peer 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:40.772949  1232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b94e35867004e258238795d4c63b63b"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:53690
I20260709 12:21:40.773787  1232 raft_consensus.cc:493] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:40.774371  1231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.774253  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80786e2a45414023841a360eaeae692a" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.774737  1156 leader_election.cc:290] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.775339  1306 raft_consensus.cc:3055] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:40.775722  1306 raft_consensus.cc:740] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.775763  1231 raft_consensus.cc:3060] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.774246  1232 raft_consensus.cc:3060] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.776716  1306 consensus_queue.cc:260] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.778144  1306 raft_consensus.cc:3060] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.786419  1231 raft_consensus.cc:2468] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:40.786924  1232 raft_consensus.cc:515] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.787916  1306 raft_consensus.cc:2468] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:40.788096  1121 leader_election.cc:304] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:40.789041  1393 raft_consensus.cc:2804] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:40.789551  1393 raft_consensus.cc:697] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:40.789997  1232 leader_election.cc:290] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.790454  1393 consensus_queue.cc:237] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.793236  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:40.793902  1156 raft_consensus.cc:3055] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:40.794241  1156 raft_consensus.cc:740] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:40.795045  1156 consensus_queue.cc:260] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.796185  1156 raft_consensus.cc:3060] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.798488  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b94e35867004e258238795d4c63b63b" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.799185  1306 raft_consensus.cc:3060] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.804039  1156 raft_consensus.cc:2468] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:40.804180  1051 catalog_manager.cc:5697] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.805505  1198 leader_election.cc:304] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:40.806581  1350 raft_consensus.cc:2804] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:40.807183  1350 raft_consensus.cc:697] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:40.808034  1350 consensus_queue.cc:237] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.811779  1306 raft_consensus.cc:2468] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:40.817405  1379 consensus_queue.cc:1048] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.819654  1051 catalog_manager.cc:5697] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 1 to 2, leader changed from 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129) to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.833561  1374 consensus_queue.cc:1048] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.845538  1359 raft_consensus.cc:993] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: : Instructing follower 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:40.846148  1374 raft_consensus.cc:1081] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Signalling peer 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:40.848275  1232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:53704
I20260709 12:21:40.848978  1232 raft_consensus.cc:493] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:40.849385  1232 raft_consensus.cc:3060] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.855764  1232 raft_consensus.cc:515] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.857963  1232 leader_election.cc:290] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.859025  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:40.859737  1156 raft_consensus.cc:3060] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.860107  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a40443b2994b42b552928f555d04b5" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.860795  1306 raft_consensus.cc:3055] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:40.861128  1306 raft_consensus.cc:740] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.862063  1306 consensus_queue.cc:260] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.863731  1306 raft_consensus.cc:3060] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.869004  1156 raft_consensus.cc:2468] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:40.870858  1198 leader_election.cc:304] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:40.871865  1350 raft_consensus.cc:2804] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:40.872488  1350 raft_consensus.cc:697] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:40.872386  1306 raft_consensus.cc:2468] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:40.873610  1350 consensus_queue.cc:237] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.881078  1054 catalog_manager.cc:5697] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 1 to 2, leader changed from feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131) to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.895035  1350 consensus_queue.cc:1048] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.897233  1374 consensus_queue.cc:1048] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.909303  1350 consensus_queue.cc:1048] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.924219  1385 consensus_queue.cc:1048] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.938428  1379 raft_consensus.cc:993] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:40.938933  1379 raft_consensus.cc:1081] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:40.940583  1156 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51260
I20260709 12:21:40.941164  1156 raft_consensus.cc:493] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:40.941483  1156 raft_consensus.cc:3060] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.947036  1156 raft_consensus.cc:515] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.949546  1156 leader_election.cc:290] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.950187  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.950774  1232 raft_consensus.cc:3060] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.950747  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f649afba93c4e33bfbd567a618722ed" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.951699  1306 raft_consensus.cc:3055] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:40.952029  1306 raft_consensus.cc:740] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.952906  1306 consensus_queue.cc:260] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.954303  1306 raft_consensus.cc:3060] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.959632  1232 raft_consensus.cc:2468] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:40.961321  1121 leader_election.cc:304] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:40.962386  1393 raft_consensus.cc:2804] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:40.962461  1306 raft_consensus.cc:2468] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:40.963037  1393 raft_consensus.cc:697] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:40.964946  1393 consensus_queue.cc:237] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.967154  1379 raft_consensus.cc:993] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:40.967794  1374 raft_consensus.cc:1081] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:40.969774  1156 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51260
I20260709 12:21:40.970469  1156 raft_consensus.cc:493] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:40.970878  1156 raft_consensus.cc:3060] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.974231  1051 catalog_manager.cc:5697] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:40.978837  1156 raft_consensus.cc:515] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.981154  1156 leader_election.cc:290] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:40.982445  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:40.982533  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "265e8ad7291f44ea933719b9ed465c7d" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:40.983249  1232 raft_consensus.cc:3060] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.983547  1306 raft_consensus.cc:3055] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:40.984047  1306 raft_consensus.cc:740] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:40.984961  1306 consensus_queue.cc:260] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.986302  1306 raft_consensus.cc:3060] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:40.991843  1232 raft_consensus.cc:2468] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:40.993278  1121 leader_election.cc:304] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:40.994218  1393 raft_consensus.cc:2804] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:40.994825  1393 raft_consensus.cc:697] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:40.995782  1393 consensus_queue.cc:237] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:40.997859  1306 raft_consensus.cc:2468] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:41.004354  1051 catalog_manager.cc:5697] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.198931  1232 raft_consensus.cc:1275] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.200706  1393 consensus_queue.cc:1048] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.209996  1306 raft_consensus.cc:1275] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.211653  1393 consensus_queue.cc:1048] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.248950  1306 raft_consensus.cc:1275] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.250654  1350 consensus_queue.cc:1048] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.260497  1156 raft_consensus.cc:1275] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.262346  1431 consensus_queue.cc:1048] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:41.284188  1306 raft_consensus.cc:1275] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.285746  1405 consensus_queue.cc:1048] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.297209  1156 raft_consensus.cc:1275] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.300098  1431 consensus_queue.cc:1048] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:41.392781  1232 raft_consensus.cc:1275] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.394452  1430 consensus_queue.cc:1048] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.403457  1306 raft_consensus.cc:1275] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.405401  1430 consensus_queue.cc:1048] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.419698  1232 raft_consensus.cc:1275] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.421365  1430 consensus_queue.cc:1048] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:41.431703  1306 raft_consensus.cc:1275] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:41.433480  1430 consensus_queue.cc:1048] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:43.734117  1051 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47182:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:43.736748  1051 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260709 12:21:43.794114  1103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:43.799548  1141 tablet_service.cc:1511] Processing CreateTablet for tablet fed323ad0dc84c839c5880be72455a22 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:43.800899  1220 tablet_service.cc:1511] Processing CreateTablet for tablet fed323ad0dc84c839c5880be72455a22 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:43.801367  1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed323ad0dc84c839c5880be72455a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.802470  1220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed323ad0dc84c839c5880be72455a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.802043  1290 tablet_service.cc:1511] Processing CreateTablet for tablet fed323ad0dc84c839c5880be72455a22 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:21:43.803699  1290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed323ad0dc84c839c5880be72455a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.803510  1146 tablet_service.cc:1511] Processing CreateTablet for tablet 139c72a11f16472db5000aee4927b73b (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:43.804937  1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 139c72a11f16472db5000aee4927b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.818608  1222 tablet_service.cc:1511] Processing CreateTablet for tablet 139c72a11f16472db5000aee4927b73b (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:43.820142  1222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 139c72a11f16472db5000aee4927b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.822860  1144 tablet_service.cc:1511] Processing CreateTablet for tablet ea10f3fd9ff14275b04cdb8f01750b64 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:43.824404  1144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea10f3fd9ff14275b04cdb8f01750b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.827316  1216 tablet_service.cc:1511] Processing CreateTablet for tablet ea10f3fd9ff14275b04cdb8f01750b64 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:43.829877  1216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea10f3fd9ff14275b04cdb8f01750b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.830257  1293 tablet_service.cc:1511] Processing CreateTablet for tablet 139c72a11f16472db5000aee4927b73b (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:21:43.831634  1293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 139c72a11f16472db5000aee4927b73b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.834481  1291 tablet_service.cc:1511] Processing CreateTablet for tablet ea10f3fd9ff14275b04cdb8f01750b64 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:21:43.835987  1291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea10f3fd9ff14275b04cdb8f01750b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.859076  1139 tablet_service.cc:1511] Processing CreateTablet for tablet b3459b8c65184dd19c43d655600014e6 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:43.860587  1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3459b8c65184dd19c43d655600014e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.865782  1220 tablet_service.cc:1511] Processing CreateTablet for tablet b3459b8c65184dd19c43d655600014e6 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:43.867290  1220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3459b8c65184dd19c43d655600014e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.872323  1294 tablet_service.cc:1511] Processing CreateTablet for tablet b3459b8c65184dd19c43d655600014e6 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:21:43.873853  1294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3459b8c65184dd19c43d655600014e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.874825  1218 tablet_service.cc:1511] Processing CreateTablet for tablet 0a2c51efcec743d6a228c356ce196160 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:43.876196  1218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2c51efcec743d6a228c356ce196160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.895702  1443 tablet_bootstrap.cc:492] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:43.897814  1444 tablet_bootstrap.cc:492] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:43.900383  1290 tablet_service.cc:1511] Processing CreateTablet for tablet 0a2c51efcec743d6a228c356ce196160 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:43.901206  1146 tablet_service.cc:1511] Processing CreateTablet for tablet 0a2c51efcec743d6a228c356ce196160 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:21:43.911330  1146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2c51efcec743d6a228c356ce196160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.917670  1295 tablet_service.cc:1511] Processing CreateTablet for tablet ffc8eb9b50f24f85b50bf10ec71e8f94 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:43.919198  1295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc8eb9b50f24f85b50bf10ec71e8f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.919250  1147 tablet_service.cc:1511] Processing CreateTablet for tablet cf2df0849c5d499d860c1d21e576c071 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:43.920887  1147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2df0849c5d499d860c1d21e576c071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.921983  1290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2c51efcec743d6a228c356ce196160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.905408  1216 tablet_service.cc:1511] Processing CreateTablet for tablet ffc8eb9b50f24f85b50bf10ec71e8f94 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:43.906653  1141 tablet_service.cc:1511] Processing CreateTablet for tablet ffc8eb9b50f24f85b50bf10ec71e8f94 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:21:43.927650  1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc8eb9b50f24f85b50bf10ec71e8f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.929179  1216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffc8eb9b50f24f85b50bf10ec71e8f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.935714  1215 tablet_service.cc:1511] Processing CreateTablet for tablet cf2df0849c5d499d860c1d21e576c071 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:43.937364  1215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2df0849c5d499d860c1d21e576c071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.910010  1445 tablet_bootstrap.cc:492] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:43.947538  1443 tablet_bootstrap.cc:654] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:43.951581  1444 tablet_bootstrap.cc:654] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:43.952960  1445 tablet_bootstrap.cc:654] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:43.958464  1291 tablet_service.cc:1511] Processing CreateTablet for tablet cf2df0849c5d499d860c1d21e576c071 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:21:43.960155  1291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2df0849c5d499d860c1d21e576c071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.960408  1139 tablet_service.cc:1511] Processing CreateTablet for tablet 2d2b5d560ea741e7b44f121702013971 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:43.962049  1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2b5d560ea741e7b44f121702013971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.973449  1222 tablet_service.cc:1511] Processing CreateTablet for tablet 2d2b5d560ea741e7b44f121702013971 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:43.975068  1222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2b5d560ea741e7b44f121702013971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.987360  1293 tablet_service.cc:1511] Processing CreateTablet for tablet 2d2b5d560ea741e7b44f121702013971 (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:21:43.989362  1293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2b5d560ea741e7b44f121702013971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.989843  1445 tablet_bootstrap.cc:492] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:43.990562  1445 ts_tablet_manager.cc:1403] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.081s	user 0.012s	sys 0.004s
I20260709 12:21:43.994200  1141 tablet_service.cc:1511] Processing CreateTablet for tablet 45977ae92e3a4855b56cbe7ddd8cd98c (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:43.999795  1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45977ae92e3a4855b56cbe7ddd8cd98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:43.997993  1445 raft_consensus.cc:359] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.003450  1445 raft_consensus.cc:385] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.003841  1445 raft_consensus.cc:740] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.012156  1444 tablet_bootstrap.cc:492] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.012908  1444 ts_tablet_manager.cc:1403] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.115s	user 0.008s	sys 0.010s
I20260709 12:21:44.015161  1218 tablet_service.cc:1511] Processing CreateTablet for tablet 45977ae92e3a4855b56cbe7ddd8cd98c (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:44.016678  1218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45977ae92e3a4855b56cbe7ddd8cd98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:44.017256  1443 tablet_bootstrap.cc:492] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.018215  1294 tablet_service.cc:1511] Processing CreateTablet for tablet 45977ae92e3a4855b56cbe7ddd8cd98c (DEFAULT_TABLE table=soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:21:44.018533  1443 ts_tablet_manager.cc:1403] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.123s	user 0.014s	sys 0.003s
I20260709 12:21:44.019774  1294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45977ae92e3a4855b56cbe7ddd8cd98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:44.023957  1444 raft_consensus.cc:359] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.033931  1444 raft_consensus.cc:385] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.034477  1444 raft_consensus.cc:740] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.040228  1444 consensus_queue.cc:260] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.043174  1444 ts_tablet_manager.cc:1434] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260709 12:21:44.054247  1443 raft_consensus.cc:359] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.055302  1443 raft_consensus.cc:385] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.055627  1443 raft_consensus.cc:740] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.054273  1445 consensus_queue.cc:260] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.059723  1443 consensus_queue.cc:260] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.066372  1445 ts_tablet_manager.cc:1434] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.075s	user 0.010s	sys 0.002s
I20260709 12:21:44.067955  1445 tablet_bootstrap.cc:492] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.071379  1444 tablet_bootstrap.cc:492] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.076730  1443 ts_tablet_manager.cc:1434] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.057s	user 0.004s	sys 0.007s
I20260709 12:21:44.080442  1444 tablet_bootstrap.cc:654] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.087893  1443 tablet_bootstrap.cc:492] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.093254  1444 tablet_bootstrap.cc:492] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.094120  1444 ts_tablet_manager.cc:1403] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.001s
I20260709 12:21:44.095454  1443 tablet_bootstrap.cc:654] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.098397  1445 tablet_bootstrap.cc:654] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.097812  1444 raft_consensus.cc:359] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.099144  1444 raft_consensus.cc:385] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.099668  1444 raft_consensus.cc:740] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.100906  1444 consensus_queue.cc:260] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.103940  1405 raft_consensus.cc:493] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.104604  1405 raft_consensus.cc:515] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.107035  1405 leader_election.cc:290] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.110730  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.111668  1306 raft_consensus.cc:2468] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.113557  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.114970  1156 raft_consensus.cc:2468] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.116680  1198 leader_election.cc:304] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:44.123628  1405 raft_consensus.cc:2804] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.124444  1405 raft_consensus.cc:493] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.124864  1405 raft_consensus.cc:3060] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.139853  1405 raft_consensus.cc:515] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.143714  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.144639  1156 raft_consensus.cc:3060] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.150458  1443 tablet_bootstrap.cc:492] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.150899  1445 tablet_bootstrap.cc:492] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.151336  1443 ts_tablet_manager.cc:1403] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.064s	user 0.032s	sys 0.014s
I20260709 12:21:44.151612  1445 ts_tablet_manager.cc:1403] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.084s	user 0.014s	sys 0.003s
I20260709 12:21:44.155594  1445 raft_consensus.cc:359] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.156486  1445 raft_consensus.cc:385] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.156298  1443 raft_consensus.cc:359] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.157253  1445 raft_consensus.cc:740] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.157522  1443 raft_consensus.cc:385] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.158113  1443 raft_consensus.cc:740] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.158532  1445 consensus_queue.cc:260] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.159111  1443 consensus_queue.cc:260] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.163281  1443 ts_tablet_manager.cc:1434] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.011s	user 0.012s	sys 0.000s
I20260709 12:21:44.165302  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:44.165611  1443 tablet_bootstrap.cc:492] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.166340  1306 raft_consensus.cc:3060] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.171362  1445 ts_tablet_manager.cc:1434] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.003s
I20260709 12:21:44.179837  1443 tablet_bootstrap.cc:654] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.182520  1445 tablet_bootstrap.cc:492] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.189580  1405 leader_election.cc:290] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.191483  1445 tablet_bootstrap.cc:654] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.196976  1443 tablet_bootstrap.cc:492] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.197790  1443 ts_tablet_manager.cc:1403] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.008s
I20260709 12:21:44.200052  1156 raft_consensus.cc:2468] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:44.201810  1198 leader_election.cc:304] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:44.202749  1453 raft_consensus.cc:2804] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.202358  1443 raft_consensus.cc:359] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.203595  1443 raft_consensus.cc:385] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.204020  1443 raft_consensus.cc:740] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.209190  1444 ts_tablet_manager.cc:1434] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.115s	user 0.030s	sys 0.075s
I20260709 12:21:44.210492  1444 tablet_bootstrap.cc:492] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.217163  1405 raft_consensus.cc:493] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.217495  1444 tablet_bootstrap.cc:654] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.217962  1405 raft_consensus.cc:515] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.205333  1443 consensus_queue.cc:260] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.223735  1453 raft_consensus.cc:697] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:44.225132  1453 consensus_queue.cc:237] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.228109  1306 raft_consensus.cc:2468] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:44.230504  1443 ts_tablet_manager.cc:1434] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.032s	user 0.010s	sys 0.003s
I20260709 12:21:44.234385  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.234436  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.235272  1307 raft_consensus.cc:2468] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.235816  1443 tablet_bootstrap.cc:492] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
W20260709 12:21:44.236685  1198 leader_election.cc:343] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.237689  1198 leader_election.cc:304] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:44.232175  1405 leader_election.cc:290] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.241583  1444 tablet_bootstrap.cc:492] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.241878  1445 tablet_bootstrap.cc:492] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.242690  1445 ts_tablet_manager.cc:1403] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.060s	user 0.016s	sys 0.013s
I20260709 12:21:44.242745  1444 ts_tablet_manager.cc:1403] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.004s
I20260709 12:21:44.245595  1443 tablet_bootstrap.cc:654] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.246381  1445 raft_consensus.cc:359] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.247285  1445 raft_consensus.cc:385] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.247591  1445 raft_consensus.cc:740] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.248544  1456 raft_consensus.cc:2804] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.248548  1444 raft_consensus.cc:359] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.250159  1444 raft_consensus.cc:385] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.249984  1456 raft_consensus.cc:493] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.250980  1444 raft_consensus.cc:740] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.251171  1456 raft_consensus.cc:3060] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.252075  1444 consensus_queue.cc:260] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.255355  1444 ts_tablet_manager.cc:1434] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:21:44.256899  1444 tablet_bootstrap.cc:492] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.261866  1445 consensus_queue.cc:260] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.264896  1444 tablet_bootstrap.cc:654] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.265105  1379 raft_consensus.cc:493] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.266068  1379 raft_consensus.cc:515] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.271025  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.271878  1156 raft_consensus.cc:2468] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:44.273327  1273 leader_election.cc:304] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:44.276736  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
W20260709 12:21:44.278851  1271 leader_election.cc:343] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.279810  1379 leader_election.cc:290] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:44.280805  1379 raft_consensus.cc:2804] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.282109  1379 raft_consensus.cc:493] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.283416  1379 raft_consensus.cc:3060] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.282629  1052 catalog_manager.cc:5697] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:44.307075  1444 tablet_bootstrap.cc:492] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.308235  1444 ts_tablet_manager.cc:1403] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.052s	user 0.029s	sys 0.008s
I20260709 12:21:44.315770  1444 raft_consensus.cc:359] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.316007  1456 raft_consensus.cc:515] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.316805  1444 raft_consensus.cc:385] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.317215  1444 raft_consensus.cc:740] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.318231  1444 consensus_queue.cc:260] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.320868  1444 ts_tablet_manager.cc:1434] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:21:44.320798  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.321710  1156 raft_consensus.cc:3060] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:44.318241  1273 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.11.11.129:35743, user_credentials={real_user=slave}} blocked reactor thread for 45025.3us
I20260709 12:21:44.324918  1456 leader_election.cc:290] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.324059  1443 tablet_bootstrap.cc:492] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.327667  1379 raft_consensus.cc:515] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.327353  1444 tablet_bootstrap.cc:492] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.332007  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:44.333351  1156 raft_consensus.cc:2468] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
W20260709 12:21:44.333707  1271 leader_election.cc:343] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.334498  1445 ts_tablet_manager.cc:1434] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.091s	user 0.008s	sys 0.005s
I20260709 12:21:44.334522  1379 leader_election.cc:290] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:44.334887  1198 leader_election.cc:304] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:44.336179  1453 raft_consensus.cc:2804] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.336666  1453 raft_consensus.cc:697] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:44.337533  1453 consensus_queue.cc:237] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.343170  1445 tablet_bootstrap.cc:492] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.346011  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.346783  1157 raft_consensus.cc:3060] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.347159  1443 ts_tablet_manager.cc:1403] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.112s	user 0.019s	sys 0.047s
I20260709 12:21:44.353530  1444 tablet_bootstrap.cc:654] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.353248  1443 raft_consensus.cc:359] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.358888  1445 tablet_bootstrap.cc:654] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.379684  1443 raft_consensus.cc:385] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.361500  1453 raft_consensus.cc:493] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.381590  1443 raft_consensus.cc:740] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.381850  1453 raft_consensus.cc:515] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.382871  1443 consensus_queue.cc:260] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.356858  1157 raft_consensus.cc:2468] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:44.365942  1052 catalog_manager.cc:5697] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:44.363101  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:44.385844  1443 ts_tablet_manager.cc:1434] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.007s
I20260709 12:21:44.386008  1306 raft_consensus.cc:3060] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.386659  1444 tablet_bootstrap.cc:492] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.387049  1443 tablet_bootstrap.cc:492] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.387220  1444 ts_tablet_manager.cc:1403] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.004s
I20260709 12:21:44.385216  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.388574  1273 leader_election.cc:304] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:44.389554  1156 raft_consensus.cc:2468] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.390870  1198 leader_election.cc:304] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:44.390931  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.391794  1456 raft_consensus.cc:2804] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.392187  1456 raft_consensus.cc:493] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.392522  1456 raft_consensus.cc:3060] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.396118  1444 raft_consensus.cc:359] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.397353  1444 raft_consensus.cc:385] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.397787  1444 raft_consensus.cc:740] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.398046  1443 tablet_bootstrap.cc:654] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.388803  1453 leader_election.cc:290] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
W20260709 12:21:44.398330  1198 leader_election.cc:343] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.400847  1445 tablet_bootstrap.cc:492] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.401481  1445 ts_tablet_manager.cc:1403] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.059s	user 0.023s	sys 0.013s
I20260709 12:21:44.401921  1462 raft_consensus.cc:2804] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.402801  1462 raft_consensus.cc:697] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:44.403764  1462 consensus_queue.cc:237] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.405002  1445 raft_consensus.cc:359] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.407687  1445 raft_consensus.cc:385] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.408052  1445 raft_consensus.cc:740] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.406558  1306 raft_consensus.cc:2468] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:44.405216  1444 consensus_queue.cc:260] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.408996  1445 consensus_queue.cc:260] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.411011  1443 tablet_bootstrap.cc:492] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.410719  1456 raft_consensus.cc:515] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.411561  1445 ts_tablet_manager.cc:1434] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:21:44.411579  1443 ts_tablet_manager.cc:1403] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.014s
I20260709 12:21:44.411473  1444 ts_tablet_manager.cc:1434] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260709 12:21:44.412997  1445 tablet_bootstrap.cc:492] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.413805  1444 tablet_bootstrap.cc:492] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.413703  1456 leader_election.cc:290] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.415511  1443 raft_consensus.cc:359] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.416313  1443 raft_consensus.cc:385] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.416576  1443 raft_consensus.cc:740] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.417280  1443 consensus_queue.cc:260] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.419433  1443 ts_tablet_manager.cc:1434] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:21:44.420435  1443 tablet_bootstrap.cc:492] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.421962  1445 tablet_bootstrap.cc:654] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.474628  1443 tablet_bootstrap.cc:654] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.476344  1444 tablet_bootstrap.cc:654] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.478856  1453 raft_consensus.cc:493] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:44.482036  1471 log.cc:927] Time spent T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:21:44.484905  1445 tablet_bootstrap.cc:492] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.486472  1445 ts_tablet_manager.cc:1403] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.074s	user 0.001s	sys 0.017s
I20260709 12:21:44.502897  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:44.502974  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.480603  1453 raft_consensus.cc:515] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.506647  1156 raft_consensus.cc:3060] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.510070  1453 leader_election.cc:290] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.516443  1445 raft_consensus.cc:359] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.518038  1445 raft_consensus.cc:385] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.518494  1445 raft_consensus.cc:740] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
W20260709 12:21:44.518997  1198 leader_election.cc:343] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.519706  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.519886  1445 consensus_queue.cc:260] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.524119  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.524472  1051 catalog_manager.cc:5697] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:44.525990  1157 raft_consensus.cc:2393] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46c1e152ed374a9a9923d464b3a57077 in current term 1: Already voted for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in this term.
I20260709 12:21:44.539161  1444 tablet_bootstrap.cc:492] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.540491  1444 ts_tablet_manager.cc:1403] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.127s	user 0.014s	sys 0.016s
I20260709 12:21:44.548338  1444 raft_consensus.cc:359] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.549592  1156 raft_consensus.cc:2468] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:44.549911  1444 raft_consensus.cc:385] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.550781  1444 raft_consensus.cc:740] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.552193  1444 consensus_queue.cc:260] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.559289  1198 leader_election.cc:304] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:44.567186  1445 ts_tablet_manager.cc:1434] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.080s	user 0.010s	sys 0.005s
I20260709 12:21:44.571097  1445 tablet_bootstrap.cc:492] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.562485  1393 raft_consensus.cc:493] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.568656  1198 leader_election.cc:304] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:44.574536  1393 raft_consensus.cc:515] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.575320  1456 raft_consensus.cc:2804] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.576116  1456 raft_consensus.cc:697] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:44.578276  1456 consensus_queue.cc:237] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.581058  1405 raft_consensus.cc:3060] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.581100  1444 ts_tablet_manager.cc:1434] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.040s	user 0.012s	sys 0.004s
I20260709 12:21:44.584300  1462 raft_consensus.cc:493] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.584874  1462 raft_consensus.cc:515] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.589347  1462 leader_election.cc:290] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:44.591493  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.589593  1444 tablet_bootstrap.cc:492] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.592463  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:44.597069  1157 raft_consensus.cc:2393] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in current term 1: Already voted for candidate 46c1e152ed374a9a9923d464b3a57077 in this term.
I20260709 12:21:44.598878  1273 leader_election.cc:304] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:44.601853  1462 raft_consensus.cc:3060] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.603276  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf2df0849c5d499d860c1d21e576c071" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:44.604445  1444 tablet_bootstrap.cc:654] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:44.605680  1121 leader_election.cc:343] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.608315  1393 leader_election.cc:290] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.610634  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf2df0849c5d499d860c1d21e576c071" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
W20260709 12:21:44.618333  1123 leader_election.cc:343] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.619027  1123 leader_election.cc:304] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c; no voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:44.624656  1452 raft_consensus.cc:2749] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:44.636494  1405 raft_consensus.cc:2749] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:44.628829  1443 tablet_bootstrap.cc:492] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.643378  1443 ts_tablet_manager.cc:1403] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.223s	user 0.036s	sys 0.043s
I20260709 12:21:44.645390  1444 tablet_bootstrap.cc:492] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.646098  1444 ts_tablet_manager.cc:1403] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.057s	user 0.028s	sys 0.023s
I20260709 12:21:44.648299  1443 raft_consensus.cc:359] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.650888  1443 raft_consensus.cc:385] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.651206  1443 raft_consensus.cc:740] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.651750  1445 tablet_bootstrap.cc:654] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.652441  1456 raft_consensus.cc:493] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.652261  1443 consensus_queue.cc:260] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.653115  1456 raft_consensus.cc:515] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.647837  1050 catalog_manager.cc:5697] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:44.654642  1443 ts_tablet_manager.cc:1434] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:21:44.655733  1456 leader_election.cc:290] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.656495  1443 tablet_bootstrap.cc:492] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.658499  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.658888  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
W20260709 12:21:44.660409  1198 leader_election.cc:343] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:44.661216  1198 leader_election.cc:343] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.661849  1198 leader_election.cc:304] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:44.663462  1405 raft_consensus.cc:2749] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:44.667625  1444 raft_consensus.cc:359] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.668557  1444 raft_consensus.cc:385] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.668943  1444 raft_consensus.cc:740] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.670722  1444 consensus_queue.cc:260] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.674283  1444 ts_tablet_manager.cc:1434] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.028s	user 0.010s	sys 0.000s
I20260709 12:21:44.675359  1462 raft_consensus.cc:2749] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:44.676461  1443 tablet_bootstrap.cc:654] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.679028  1444 tablet_bootstrap.cc:492] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.683542  1445 tablet_bootstrap.cc:492] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.684445  1445 ts_tablet_manager.cc:1403] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.114s	user 0.014s	sys 0.013s
I20260709 12:21:44.688927  1444 tablet_bootstrap.cc:654] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.689471  1445 raft_consensus.cc:359] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.690577  1445 raft_consensus.cc:385] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.690969  1445 raft_consensus.cc:740] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.692076  1445 consensus_queue.cc:260] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.693859  1379 raft_consensus.cc:493] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.694535  1379 raft_consensus.cc:515] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.696307  1405 consensus_queue.cc:1048] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:44.698329  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.699177  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:44.699218  1157 raft_consensus.cc:2468] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 0.
I20260709 12:21:44.701365  1379 leader_election.cc:290] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
W20260709 12:21:44.701536  1271 leader_election.cc:343] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.703938  1273 leader_election.cc:304] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:44.705307  1379 raft_consensus.cc:2804] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.705829  1379 raft_consensus.cc:493] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.706208  1379 raft_consensus.cc:3060] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.710785  1443 tablet_bootstrap.cc:492] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.711550  1443 ts_tablet_manager.cc:1403] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.056s	user 0.017s	sys 0.011s
I20260709 12:21:44.713910  1444 tablet_bootstrap.cc:492] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.714673  1444 ts_tablet_manager.cc:1403] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.011s
I20260709 12:21:44.704352  1445 ts_tablet_manager.cc:1434] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:21:44.718779  1379 raft_consensus.cc:515] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.718852  1444 raft_consensus.cc:359] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.720012  1444 raft_consensus.cc:385] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.720721  1445 tablet_bootstrap.cc:492] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.722707  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.723587  1157 raft_consensus.cc:3060] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.724082  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
W20260709 12:21:44.726308  1271 leader_election.cc:343] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.727198  1379 leader_election.cc:290] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:44.720257  1443 raft_consensus.cc:359] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.730571  1443 raft_consensus.cc:385] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.731001  1443 raft_consensus.cc:740] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.731936  1443 consensus_queue.cc:260] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.720424  1444 raft_consensus.cc:740] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.734541  1444 consensus_queue.cc:260] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.746686  1452 raft_consensus.cc:493] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.747236  1452 raft_consensus.cc:515] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.752162  1452 leader_election.cc:290] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.752229  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d2b5d560ea741e7b44f121702013971" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:44.753656  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d2b5d560ea741e7b44f121702013971" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
W20260709 12:21:44.754819  1121 leader_election.cc:343] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.757589  1443 ts_tablet_manager.cc:1434] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.001s
I20260709 12:21:44.759071  1445 tablet_bootstrap.cc:654] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.760280  1443 tablet_bootstrap.cc:492] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.769321  1444 ts_tablet_manager.cc:1434] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.054s	user 0.010s	sys 0.000s
I20260709 12:21:44.775175  1444 tablet_bootstrap.cc:492] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: Bootstrap starting.
I20260709 12:21:44.774930  1405 consensus_queue.cc:1048] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:44.780228  1157 raft_consensus.cc:2468] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 1.
I20260709 12:21:44.781985  1273 leader_election.cc:304] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 46c1e152ed374a9a9923d464b3a57077
W20260709 12:21:44.783851  1123 leader_election.cc:343] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.784489  1123 leader_election.cc:304] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c; no voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:44.784844  1379 raft_consensus.cc:2804] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.785540  1379 raft_consensus.cc:697] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:44.786924  1444 tablet_bootstrap.cc:654] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.786741  1379 consensus_queue.cc:237] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.792150  1393 raft_consensus.cc:2749] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:44.808183  1443 tablet_bootstrap.cc:654] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:44.801474  1103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:44.829815  1405 consensus_queue.cc:1048] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:44.828896  1445 tablet_bootstrap.cc:492] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.835529  1445 ts_tablet_manager.cc:1403] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.115s	user 0.026s	sys 0.046s
I20260709 12:21:44.838558  1444 tablet_bootstrap.cc:492] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: No bootstrap required, opened a new log
I20260709 12:21:44.839275  1444 ts_tablet_manager.cc:1403] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: Time spent bootstrapping tablet: real 0.066s	user 0.024s	sys 0.000s
I20260709 12:21:44.839716  1443 tablet_bootstrap.cc:492] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.840272  1443 ts_tablet_manager.cc:1403] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.080s	user 0.017s	sys 0.000s
I20260709 12:21:44.842653  1444 raft_consensus.cc:359] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.843550  1444 raft_consensus.cc:385] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.843396  1443 raft_consensus.cc:359] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.843935  1444 raft_consensus.cc:740] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.844167  1443 raft_consensus.cc:385] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.844493  1443 raft_consensus.cc:740] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.845242  1443 consensus_queue.cc:260] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.845095  1444 consensus_queue.cc:260] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.848212  1444 ts_tablet_manager.cc:1434] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:21:44.849573  1405 raft_consensus.cc:493] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.846673  1051 catalog_manager.cc:5697] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 1 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:44.850308  1405 raft_consensus.cc:515] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.853178  1405 leader_election.cc:290] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.840065  1445 raft_consensus.cc:359] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.856806  1445 raft_consensus.cc:385] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.857260  1445 raft_consensus.cc:740] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.858559  1153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf2df0849c5d499d860c1d21e576c071" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.859050  1458 raft_consensus.cc:493] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.859565  1153 raft_consensus.cc:2468] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.860036  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf2df0849c5d499d860c1d21e576c071" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.859881  1458 raft_consensus.cc:515] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.861985  1379 consensus_queue.cc:1048] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:44.863327  1458 leader_election.cc:290] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.866086  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.866930  1405 raft_consensus.cc:493] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.867075  1307 raft_consensus.cc:2468] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.867480  1198 leader_election.cc:304] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:44.867738  1405 raft_consensus.cc:515] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.869786  1443 ts_tablet_manager.cc:1434] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.000s
I20260709 12:21:44.871531  1456 raft_consensus.cc:2804] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.872115  1456 raft_consensus.cc:493] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.872539  1456 raft_consensus.cc:3060] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.874140  1443 tablet_bootstrap.cc:492] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: Bootstrap starting.
I20260709 12:21:44.874424  1405 leader_election.cc:290] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.879436  1198 leader_election.cc:304] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:44.880401  1405 raft_consensus.cc:2804] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.881693  1443 tablet_bootstrap.cc:654] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.882341  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d2b5d560ea741e7b44f121702013971" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:44.883762  1156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d2b5d560ea741e7b44f121702013971" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.884855  1156 raft_consensus.cc:2468] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.883173  1154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:44.890372  1154 raft_consensus.cc:2468] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.891777  1445 consensus_queue.cc:260] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.892982  1306 raft_consensus.cc:2468] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 0.
I20260709 12:21:44.894845  1445 ts_tablet_manager.cc:1434] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.002s
W20260709 12:21:44.897171  1198 leader_election.cc:343] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:44.898774  1198 leader_election.cc:304] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:44.898933  1456 raft_consensus.cc:515] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.899745  1453 raft_consensus.cc:2804] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:44.900275  1453 raft_consensus.cc:493] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.900694  1453 raft_consensus.cc:3060] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.881722  1405 raft_consensus.cc:493] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.901923  1456 leader_election.cc:290] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.901952  1405 raft_consensus.cc:3060] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.908260  1445 tablet_bootstrap.cc:492] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.916400  1379 consensus_queue.cc:1048] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:44.917433  1456 consensus_queue.cc:1048] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:44.920714  1445 tablet_bootstrap.cc:654] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.921824  1443 tablet_bootstrap.cc:492] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: No bootstrap required, opened a new log
I20260709 12:21:44.922591  1443 ts_tablet_manager.cc:1403] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: Time spent bootstrapping tablet: real 0.049s	user 0.015s	sys 0.000s
I20260709 12:21:44.923840  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf2df0849c5d499d860c1d21e576c071" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:44.924803  1306 raft_consensus.cc:3060] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.927315  1443 raft_consensus.cc:359] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.928308  1443 raft_consensus.cc:385] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.928668  1443 raft_consensus.cc:740] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.929548  1443 consensus_queue.cc:260] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.927894  1154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cf2df0849c5d499d860c1d21e576c071" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.937595  1154 raft_consensus.cc:3060] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.938235  1443 ts_tablet_manager.cc:1434] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260709 12:21:44.933347  1405 raft_consensus.cc:515] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.940425  1445 tablet_bootstrap.cc:492] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:44.941125  1445 ts_tablet_manager.cc:1403] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.033s	user 0.014s	sys 0.002s
I20260709 12:21:44.942293  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:44.943116  1157 raft_consensus.cc:3060] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.947146  1445 raft_consensus.cc:359] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.948184  1445 raft_consensus.cc:385] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:44.948490  1445 raft_consensus.cc:740] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:44.949422  1445 consensus_queue.cc:260] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.955269  1405 leader_election.cc:290] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:44.956516  1157 raft_consensus.cc:2468] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:44.958465  1198 leader_election.cc:304] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:44.959568  1405 raft_consensus.cc:2804] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.960397  1405 raft_consensus.cc:697] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:44.961436  1405 consensus_queue.cc:237] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.966410  1306 raft_consensus.cc:2468] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:44.972395  1445 ts_tablet_manager.cc:1434] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.031s	user 0.009s	sys 0.001s
I20260709 12:21:44.974860  1445 tablet_bootstrap.cc:492] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: Bootstrap starting.
I20260709 12:21:44.978312  1198 leader_election.cc:304] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:44.979566  1456 raft_consensus.cc:2804] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:44.980144  1456 raft_consensus.cc:697] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:44.981128  1456 consensus_queue.cc:237] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:44.982688  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:44.986469  1306 raft_consensus.cc:3060] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:44.998122  1154 raft_consensus.cc:2468] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:45.000329  1445 tablet_bootstrap.cc:654] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:44.997354  1453 raft_consensus.cc:515] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.006054  1306 raft_consensus.cc:2468] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:45.013372  1445 tablet_bootstrap.cc:492] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: No bootstrap required, opened a new log
I20260709 12:21:45.014169  1445 ts_tablet_manager.cc:1403] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent bootstrapping tablet: real 0.040s	user 0.010s	sys 0.012s
I20260709 12:21:45.014434  1154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d2b5d560ea741e7b44f121702013971" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:45.015247  1154 raft_consensus.cc:3060] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:45.016034  1453 leader_election.cc:290] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.015163  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d2b5d560ea741e7b44f121702013971" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.017855  1445 raft_consensus.cc:359] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.017549  1051 catalog_manager.cc:5697] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.019477  1445 raft_consensus.cc:385] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:45.020069  1445 raft_consensus.cc:740] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Initialized, Role: FOLLOWER
I20260709 12:21:45.021270  1445 consensus_queue.cc:260] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.022559  1306 raft_consensus.cc:3060] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:45.028764  1154 raft_consensus.cc:2468] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:45.030444  1198 leader_election.cc:304] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:45.031566  1306 raft_consensus.cc:2468] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 1.
I20260709 12:21:45.031744  1453 raft_consensus.cc:2804] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:45.032859  1453 raft_consensus.cc:697] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.034175  1453 consensus_queue.cc:237] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.040504  1445 ts_tablet_manager.cc:1434] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 12:21:45.054791  1052 catalog_manager.cc:5697] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.056170  1052 catalog_manager.cc:5697] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 1 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.072098  1453 consensus_queue.cc:1048] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:45.085662  1453 raft_consensus.cc:493] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.086236  1453 raft_consensus.cc:515] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.088600  1453 leader_election.cc:290] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.089586  1154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec884978e874b2c8f605373f1a3b31c" is_pre_election: true
I20260709 12:21:45.090560  1154 raft_consensus.cc:2393] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46c1e152ed374a9a9923d464b3a57077 in current term 1: Already voted for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in this term.
I20260709 12:21:45.091276  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:45.091667  1458 consensus_queue.cc:1048] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:45.094533  1198 leader_election.cc:304] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077; no voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:45.095791  1453 raft_consensus.cc:3060] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:45.107053  1453 raft_consensus.cc:2749] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:45.121167  1452 raft_consensus.cc:493] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.122007  1452 raft_consensus.cc:515] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.124465  1452 leader_election.cc:290] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.125803  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077" is_pre_election: true
I20260709 12:21:45.125803  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" is_pre_election: true
I20260709 12:21:45.126940  1306 raft_consensus.cc:2468] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 0.
I20260709 12:21:45.126940  1232 raft_consensus.cc:2468] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 0.
I20260709 12:21:45.128723  1121 leader_election.cc:304] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:45.129735  1452 raft_consensus.cc:2804] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:45.130138  1452 raft_consensus.cc:493] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.130447  1452 raft_consensus.cc:3060] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:45.140488  1452 raft_consensus.cc:515] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.143182  1452 leader_election.cc:290] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.144340  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:45.144632  1306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.145370  1232 raft_consensus.cc:3060] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:45.145422  1306 raft_consensus.cc:3060] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:45.153558  1232 raft_consensus.cc:2468] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 1.
I20260709 12:21:45.153606  1306 raft_consensus.cc:2468] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 1.
I20260709 12:21:45.155251  1121 leader_election.cc:304] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:45.156245  1452 raft_consensus.cc:2804] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:45.156951  1452 raft_consensus.cc:697] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:45.158372  1452 consensus_queue.cc:237] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.169929  1052 catalog_manager.cc:5697] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 1 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.337834  1232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:53710
I20260709 12:21:45.338677  1232 raft_consensus.cc:606] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:45.344347  1379 consensus_queue.cc:1048] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.346383  1307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "139c72a11f16472db5000aee4927b73b"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:36894
I20260709 12:21:45.347374  1307 raft_consensus.cc:606] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:45.355679  1232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:53710
I20260709 12:21:45.356444  1232 raft_consensus.cc:606] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:45.366540  1231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
mode: GRACEFUL
new_leader_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:53710
I20260709 12:21:45.367293  1231 raft_consensus.cc:606] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Received request to transfer leadership to TS 0ec884978e874b2c8f605373f1a3b31c
I20260709 12:21:45.369791  1495 consensus_queue.cc:1048] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.374693  1157 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
mode: GRACEFUL
new_leader_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:51274
I20260709 12:21:45.375382  1157 raft_consensus.cc:606] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Received request to transfer leadership to TS 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:45.377142  1453 consensus_queue.cc:1048] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.386152  1307 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
mode: GRACEFUL
new_leader_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:36894
I20260709 12:21:45.386865  1307 raft_consensus.cc:606] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Received request to transfer leadership to TS 46c1e152ed374a9a9923d464b3a57077
I20260709 12:21:45.394913  1379 raft_consensus.cc:993] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.395630  1462 raft_consensus.cc:1081] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.398816  1495 raft_consensus.cc:993] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: : Instructing follower 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:45.399398  1495 raft_consensus.cc:1081] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Signalling peer 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:45.401901  1232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:53704
I20260709 12:21:45.402667  1232 raft_consensus.cc:493] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.403093  1232 raft_consensus.cc:3060] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.406999  1458 consensus_queue.cc:1048] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.411978  1231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
mode: GRACEFUL
new_leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
 from {username='slave'} at 127.0.0.1:53710
I20260709 12:21:45.412714  1231 raft_consensus.cc:606] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Received request to transfer leadership to TS feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:45.413518  1154 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "139c72a11f16472db5000aee4927b73b"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51260
I20260709 12:21:45.414338  1154 raft_consensus.cc:493] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.414789  1154 raft_consensus.cc:3060] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.422220  1232 raft_consensus.cc:515] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.425318  1232 leader_election.cc:290] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.425143  1154 raft_consensus.cc:515] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.428066  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:45.428794  1154 leader_election.cc:290] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.431175  1307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.431418  1453 raft_consensus.cc:993] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: : Instructing follower feb80d6a43844de4a8c50c7ea4fbe5e9 to start an election
I20260709 12:21:45.432116  1307 raft_consensus.cc:3055] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.432732  1456 raft_consensus.cc:1081] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Signalling peer feb80d6a43844de4a8c50c7ea4fbe5e9 to start an election
I20260709 12:21:45.432847  1307 raft_consensus.cc:740] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:45.434202  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "139c72a11f16472db5000aee4927b73b" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:45.434541  1305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffc8eb9b50f24f85b50bf10ec71e8f94" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.434145  1307 consensus_queue.cc:260] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.435222  1232 raft_consensus.cc:3060] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.435359  1305 raft_consensus.cc:3055] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.435974  1305 raft_consensus.cc:740] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:45.429023  1157 raft_consensus.cc:3060] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.437021  1305 consensus_queue.cc:260] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.438843  1305 raft_consensus.cc:3060] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.441293  1307 raft_consensus.cc:3060] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.439278  1453 consensus_queue.cc:1048] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.445883  1306 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
 from {username='slave'} at 127.0.0.1:36876
I20260709 12:21:45.446708  1306 raft_consensus.cc:493] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.447331  1306 raft_consensus.cc:3060] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.455479  1232 raft_consensus.cc:2468] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.457144  1121 leader_election.cc:304] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:45.458726  1393 raft_consensus.cc:2804] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.459475  1393 raft_consensus.cc:697] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:45.460299  1157 raft_consensus.cc:2468] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:45.460830  1393 consensus_queue.cc:237] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.462167  1198 leader_election.cc:304] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:45.464458  1456 raft_consensus.cc:2804] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.457082  1307 raft_consensus.cc:2468] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.468730  1306 raft_consensus.cc:515] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.471798  1456 raft_consensus.cc:697] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.473658  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:45.474545  1157 raft_consensus.cc:3060] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.474649  1306 leader_election.cc:290] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:45.475581  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45977ae92e3a4855b56cbe7ddd8cd98c" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:45.476397  1232 raft_consensus.cc:3055] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.476740  1232 raft_consensus.cc:740] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.477680  1232 consensus_queue.cc:260] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.473094  1456 consensus_queue.cc:237] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.479197  1232 raft_consensus.cc:3060] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.491302  1052 catalog_manager.cc:5697] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 1 to 2, leader changed from feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131) to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.493773  1232 raft_consensus.cc:2468] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 2.
I20260709 12:21:45.494953  1305 raft_consensus.cc:2468] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:45.495462  1271 leader_election.cc:304] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:45.496722  1456 consensus_queue.cc:1048] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.498000  1495 raft_consensus.cc:2804] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.498916  1050 catalog_manager.cc:5697] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.498920  1495 raft_consensus.cc:697] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:45.501768  1157 raft_consensus.cc:2468] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 2.
I20260709 12:21:45.503149  1495 consensus_queue.cc:237] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.504392  1405 raft_consensus.cc:993] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.505230  1456 raft_consensus.cc:1081] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.507658  1157 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51250
I20260709 12:21:45.508798  1157 raft_consensus.cc:493] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.509444  1157 raft_consensus.cc:3060] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.518989  1052 catalog_manager.cc:5697] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 1 to 2, leader changed from 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130) to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:45.520080  1157 raft_consensus.cc:515] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.523545  1157 leader_election.cc:290] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.523836  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:45.524637  1232 raft_consensus.cc:3055] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.525002  1232 raft_consensus.cc:740] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.526120  1232 consensus_queue.cc:260] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.526791  1305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea10f3fd9ff14275b04cdb8f01750b64" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.527536  1305 raft_consensus.cc:3060] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.528039  1232 raft_consensus.cc:3060] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.538373  1232 raft_consensus.cc:2468] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.538373  1305 raft_consensus.cc:2468] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.540064  1121 leader_election.cc:304] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, 46c1e152ed374a9a9923d464b3a57077; no voters: 
I20260709 12:21:45.541329  1393 raft_consensus.cc:2804] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.541937  1393 raft_consensus.cc:697] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:45.543181  1393 consensus_queue.cc:237] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.546742  1456 consensus_queue.cc:1048] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.556391  1050 catalog_manager.cc:5697] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.566776  1405 consensus_queue.cc:1048] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.641086  1405 raft_consensus.cc:993] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.641608  1458 raft_consensus.cc:1081] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.643576  1157 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51250
I20260709 12:21:45.644160  1157 raft_consensus.cc:493] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.644449  1157 raft_consensus.cc:3060] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.652024  1157 raft_consensus.cc:515] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.654372  1157 leader_election.cc:290] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.655227  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:45.656070  1232 raft_consensus.cc:3055] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.656431  1232 raft_consensus.cc:740] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.656221  1305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed323ad0dc84c839c5880be72455a22" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.657286  1305 raft_consensus.cc:3060] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.657618  1232 consensus_queue.cc:260] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.659067  1232 raft_consensus.cc:3060] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.663928  1305 raft_consensus.cc:2468] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.665421  1123 leader_election.cc:304] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:45.665616  1232 raft_consensus.cc:2468] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.666456  1452 raft_consensus.cc:2804] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.667033  1452 raft_consensus.cc:697] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:45.668069  1452 consensus_queue.cc:237] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.676124  1052 catalog_manager.cc:5697] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.701002  1405 raft_consensus.cc:993] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: : Instructing follower 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.701565  1458 raft_consensus.cc:1081] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Signalling peer 0ec884978e874b2c8f605373f1a3b31c to start an election
I20260709 12:21:45.703301  1157 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
 from {username='slave'} at 127.0.0.1:51250
I20260709 12:21:45.703858  1157 raft_consensus.cc:493] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.704201  1157 raft_consensus.cc:3060] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.710649  1157 raft_consensus.cc:515] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.712478  1157 leader_election.cc:290] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Requested vote from peers 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.713755  1305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.713967  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3459b8c65184dd19c43d655600014e6" candidate_uuid: "0ec884978e874b2c8f605373f1a3b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:45.714589  1305 raft_consensus.cc:3060] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.714720  1232 raft_consensus.cc:3055] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.715197  1232 raft_consensus.cc:740] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.716032  1232 consensus_queue.cc:260] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.717481  1232 raft_consensus.cc:3060] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.722519  1305 raft_consensus.cc:2468] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.724001  1123 leader_election.cc:304] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:45.724843  1452 raft_consensus.cc:2804] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.725411  1452 raft_consensus.cc:697] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Becoming Leader. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:45.726434  1232 raft_consensus.cc:2468] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec884978e874b2c8f605373f1a3b31c in term 2.
I20260709 12:21:45.726488  1452 consensus_queue.cc:237] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.735491  1052 catalog_manager.cc:5697] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c reported cstate change: term changed from 1 to 2, leader changed from 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130) to 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129). New cstate: current_term: 2 leader_uuid: "0ec884978e874b2c8f605373f1a3b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.759234  1393 consensus_queue.cc:1048] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:45.768512  1452 raft_consensus.cc:993] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: : Instructing follower 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:45.769670  1393 raft_consensus.cc:1081] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Signalling peer 46c1e152ed374a9a9923d464b3a57077 to start an election
I20260709 12:21:45.772071  1231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
 from {username='slave'} at 127.0.0.1:53690
I20260709 12:21:45.775244  1231 raft_consensus.cc:493] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:45.775820  1231 raft_consensus.cc:3060] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.778689  1513 consensus_queue.cc:1048] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:45.788627  1231 raft_consensus.cc:515] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.794130  1231 leader_election.cc:290] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131:34345)
I20260709 12:21:45.794929  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:45.795315  1305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a2c51efcec743d6a228c356ce196160" candidate_uuid: "46c1e152ed374a9a9923d464b3a57077" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
I20260709 12:21:45.795809  1157 raft_consensus.cc:3055] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:45.796468  1157 raft_consensus.cc:740] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:45.796501  1305 raft_consensus.cc:3060] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.797497  1157 consensus_queue.cc:260] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.798965  1157 raft_consensus.cc:3060] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:45.804387  1305 raft_consensus.cc:2468] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:45.805979  1157 raft_consensus.cc:2468] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c1e152ed374a9a9923d464b3a57077 in term 2.
I20260709 12:21:45.806056  1198 leader_election.cc:304] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c1e152ed374a9a9923d464b3a57077, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:45.807410  1458 raft_consensus.cc:2804] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:45.808094  1458 raft_consensus.cc:697] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Becoming Leader. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:45.809103  1458 consensus_queue.cc:237] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:45.819238  1052 catalog_manager.cc:5697] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 reported cstate change: term changed from 1 to 2, leader changed from 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129) to 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130). New cstate: current_term: 2 leader_uuid: "46c1e152ed374a9a9923d464b3a57077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.871522  1231 raft_consensus.cc:1275] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:45.873381  1452 consensus_queue.cc:1048] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.883436  1305 raft_consensus.cc:1275] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:45.885532  1452 consensus_queue.cc:1048] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.942077  1305 raft_consensus.cc:1275] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:45.943773  1458 consensus_queue.cc:1048] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:45.958127  1157 raft_consensus.cc:1275] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:45.958467  1231 raft_consensus.cc:1275] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer feb80d6a43844de4a8c50c7ea4fbe5e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:45.960281  1495 consensus_queue.cc:1048] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:45.963801  1458 consensus_queue.cc:1048] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.972174  1154 raft_consensus.cc:1275] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer feb80d6a43844de4a8c50c7ea4fbe5e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:45.978354  1515 consensus_queue.cc:1048] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:46.056538  1231 raft_consensus.cc:1275] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.058115  1452 consensus_queue.cc:1048] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.069063  1305 raft_consensus.cc:1275] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.071270  1513 consensus_queue.cc:1048] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.073202  1305 raft_consensus.cc:1275] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.075217  1513 consensus_queue.cc:1048] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.088366  1231 raft_consensus.cc:1275] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.090587  1393 consensus_queue.cc:1048] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.129485  1231 raft_consensus.cc:1275] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.131094  1452 consensus_queue.cc:1048] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.144683  1305 raft_consensus.cc:1275] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 0ec884978e874b2c8f605373f1a3b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.146464  1513 consensus_queue.cc:1048] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.218050  1157 raft_consensus.cc:1275] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.219646  1458 consensus_queue.cc:1048] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:46.229614  1305 raft_consensus.cc:1275] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Refusing update from remote peer 46c1e152ed374a9a9923d464b3a57077: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:46.232688  1458 consensus_queue.cc:1048] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:48.418911  1051 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47182:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:21:48.445977  1051 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47182:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:21:48.519160 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:48.661295  1553 log.cc:927] Time spent T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: Append to log took a long time: real 0.059s	user 0.002s	sys 0.000s
I20260709 12:21:48.676666  1561 tablet.cc:1724] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:48.714003  1271 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.130:45955, user_credentials={real_user=slave}} blocked reactor thread for 50308.8us
I20260709 12:21:48.763820  1569 tablet.cc:1724] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.805894  1568 tablet.cc:1724] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.817364  1566 tablet.cc:1724] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.870674  1225 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a"
dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
mode: GRACEFUL
new_leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
 from {username='slave'} at 127.0.0.1:53722
I20260709 12:21:48.871537  1225 raft_consensus.cc:606] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Received request to transfer leadership to TS feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:48.881449  1561 tablet.cc:1724] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.885685  1576 tablet.cc:1724] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.889734  1579 tablet.cc:1724] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:21:48.937417  1198 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.11.129:35743, user_credentials={real_user=slave}} blocked reactor thread for 366199us
I20260709 12:21:48.939478  1569 tablet.cc:1724] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.946790  1577 tablet.cc:1724] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:48.987533  1581 tablet.cc:1724] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.019909  1574 raft_consensus.cc:993] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: : Instructing follower feb80d6a43844de4a8c50c7ea4fbe5e9 to start an election
I20260709 12:21:49.021441  1574 raft_consensus.cc:1081] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Signalling peer feb80d6a43844de4a8c50c7ea4fbe5e9 to start an election
W20260709 12:21:49.028571  1582 log.cc:927] Time spent T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:21:49.027812  1566 tablet.cc:1724] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.031505  1583 tablet.cc:1724] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.000806  1568 tablet.cc:1724] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.074797  1586 tablet.cc:1724] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.079262  1561 tablet.cc:1724] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.080188  1590 tablet.cc:1724] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.087442  1302 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
 from {username='slave'} at 127.0.0.1:36876
I20260709 12:21:49.088526  1302 raft_consensus.cc:493] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:49.089039  1302 raft_consensus.cc:3060] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.163632  1577 tablet.cc:1724] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.179883  1590 tablet.cc:1724] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.186916  1594 tablet.cc:1724] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.203178  1157 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918"
dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
mode: GRACEFUL
new_leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
 from {username='slave'} at 127.0.0.1:51288
I20260709 12:21:49.204114  1157 raft_consensus.cc:606] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Received request to transfer leadership to TS feb80d6a43844de4a8c50c7ea4fbe5e9
I20260709 12:21:49.204293  1052 catalog_manager.cc:5789] TS 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955) does not have the latest schema for tablet ffc8eb9b50f24f85b50bf10ec71e8f94 (table soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]). Expected version 1 got 0
I20260709 12:21:49.205742  1052 catalog_manager.cc:5789] TS 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955) does not have the latest schema for tablet cf2df0849c5d499d860c1d21e576c071 (table soft_deleted_table [id=aa76877e19df4d04902872746ba5010d]). Expected version 1 got 0
I20260709 12:21:49.206195 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:49.207201 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:49.208246  1577 tablet.cc:1724] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.211931  1590 tablet.cc:1724] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.223387  1568 tablet.cc:1724] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.226842  1581 tablet.cc:1724] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.245193  1563 raft_consensus.cc:993] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: : Instructing follower feb80d6a43844de4a8c50c7ea4fbe5e9 to start an election
I20260709 12:21:49.233999  1590 tablet.cc:1724] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.246537  1562 raft_consensus.cc:1081] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Signalling peer feb80d6a43844de4a8c50c7ea4fbe5e9 to start an election
I20260709 12:21:49.249840  1304 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918"
dest_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9"
 from {username='slave'} at 127.0.0.1:36892
I20260709 12:21:49.250633  1304 raft_consensus.cc:493] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:49.251037  1304 raft_consensus.cc:3060] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.260290  1569 tablet.cc:1724] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.273751  1579 tablet.cc:1724] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.314085  1569 tablet.cc:1724] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:21:49.316825  1304 raft_consensus.cc:515] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:49.321718  1157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:49.322638  1157 raft_consensus.cc:3055] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:49.322983  1157 raft_consensus.cc:740] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ec884978e874b2c8f605373f1a3b31c, State: Running, Role: LEADER
I20260709 12:21:49.324155  1229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18becdfbf98432c8b8d6f246d349918" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:49.324456  1157 consensus_queue.cc:260] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:49.325109  1229 raft_consensus.cc:3060] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.327368  1157 raft_consensus.cc:3060] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.320950  1302 raft_consensus.cc:515] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:49.327055  1304 leader_election.cc:290] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:49.341346  1157 raft_consensus.cc:2468] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 2.
I20260709 12:21:49.345103  1273 leader_election.cc:304] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:49.346676  1153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0ec884978e874b2c8f605373f1a3b31c"
I20260709 12:21:49.348608  1587 raft_consensus.cc:2804] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:49.348758  1153 raft_consensus.cc:3060] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.349330  1587 raft_consensus.cc:697] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:49.350581  1232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224dcfc0396244728eb039ca30e4cf9a" candidate_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "46c1e152ed374a9a9923d464b3a57077"
I20260709 12:21:49.350507  1587 consensus_queue.cc:237] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } }
I20260709 12:21:49.351403  1232 raft_consensus.cc:3055] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:49.351785  1232 raft_consensus.cc:740] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 46c1e152ed374a9a9923d464b3a57077, State: Running, Role: LEADER
I20260709 12:21:49.352718  1232 consensus_queue.cc:260] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:49.347630  1302 leader_election.cc:290] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 2 election: Requested vote from peers 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743), 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955)
I20260709 12:21:49.354933  1232 raft_consensus.cc:3060] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:49.361588  1229 raft_consensus.cc:2468] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 2.
I20260709 12:21:49.370942  1052 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b: soft_deleted_table [id=aa76877e19df4d04902872746ba5010d] alter complete (version 1)
I20260709 12:21:49.362099  1153 raft_consensus.cc:2468] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 2.
I20260709 12:21:49.385929  1273 leader_election.cc:304] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec884978e874b2c8f605373f1a3b31c, feb80d6a43844de4a8c50c7ea4fbe5e9; no voters: 
I20260709 12:21:49.388120  1587 raft_consensus.cc:2804] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:49.392031  1587 raft_consensus.cc:697] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 LEADER]: Becoming Leader. State: Replica: feb80d6a43844de4a8c50c7ea4fbe5e9, State: Running, Role: LEADER
I20260709 12:21:49.393651  1587 consensus_queue.cc:237] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } }
I20260709 12:21:49.396270  1232 raft_consensus.cc:2468] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb80d6a43844de4a8c50c7ea4fbe5e9 in term 2.
I20260709 12:21:49.408409  1052 catalog_manager.cc:5697] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 1 to 2, leader changed from 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129) to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:49.410305  1052 catalog_manager.cc:5697] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 reported cstate change: term changed from 1 to 2, leader changed from 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130) to feb80d6a43844de4a8c50c7ea4fbe5e9 (127.11.11.131). New cstate: current_term: 2 leader_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb80d6a43844de4a8c50c7ea4fbe5e9" member_type: VOTER last_known_addr { host: "127.11.11.131" port: 34345 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:49.782584  1232 raft_consensus.cc:1275] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer feb80d6a43844de4a8c50c7ea4fbe5e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:49.784608  1587 consensus_queue.cc:1048] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:49.799297  1153 raft_consensus.cc:1275] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer feb80d6a43844de4a8c50c7ea4fbe5e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:49.801705  1587 consensus_queue.cc:1048] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:49.911106  1229 raft_consensus.cc:1275] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Refusing update from remote peer feb80d6a43844de4a8c50c7ea4fbe5e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:49.912707  1587 consensus_queue.cc:1048] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c1e152ed374a9a9923d464b3a57077" member_type: VOTER last_known_addr { host: "127.11.11.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:49.922120  1153 raft_consensus.cc:1275] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Refusing update from remote peer feb80d6a43844de4a8c50c7ea4fbe5e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:49.924356  1587 consensus_queue.cc:1048] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec884978e874b2c8f605373f1a3b31c" member_type: VOTER last_known_addr { host: "127.11.11.129" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:50.208949 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:50.218323 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:50.218966 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:51.220413 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:51.226783 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:51.227587 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:52.260236 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:52.285513 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:52.286824 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:21:53.159615  1085 debug-util.cc:398] Leaking SignalData structure 0x7b0800394680 after lost signal to thread 11313
W20260709 12:21:53.161284  1085 debug-util.cc:398] Leaking SignalData structure 0x7b08002423a0 after lost signal to thread 1088
W20260709 12:21:53.163197  1085 debug-util.cc:398] Leaking SignalData structure 0x7b0800253320 after lost signal to thread 1181
W20260709 12:21:53.164757  1085 debug-util.cc:398] Leaking SignalData structure 0x7b08000f30c0 after lost signal to thread 1257
W20260709 12:21:53.166148  1085 debug-util.cc:398] Leaking SignalData structure 0x7b080038f100 after lost signal to thread 1331
I20260709 12:21:53.288254 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:53.295245 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:53.296157 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:54.297449 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:54.303395 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:54.304114 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:55.305848 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:55.315805 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:55.316514 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:56.318305 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:56.329448 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:56.330190 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:57.331715 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:57.338639 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:57.339471 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:58.340997 11310 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:58.347651 11310 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:58.348289 11310 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:59.425937 11310 tablet_server.cc:179] TabletServer@127.11.11.129:0 shutting down...
W20260709 12:21:59.446099  1198 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 16 similar messages]
W20260709 12:21:59.455322  1198 consensus_peers.cc:597] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:59.478554  1273 consensus_peers.cc:597] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.493503 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:59.494926 11310 tablet_replica.cc:333] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.496678 11310 raft_consensus.cc:2243] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.497923 11310 raft_consensus.cc:2272] T 45977ae92e3a4855b56cbe7ddd8cd98c P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:59.498320  1198 consensus_peers.cc:597] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.501482 11310 tablet_replica.cc:333] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.502987 11310 raft_consensus.cc:2243] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.503971 11310 raft_consensus.cc:2272] T 0a2c51efcec743d6a228c356ce196160 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.507877 11310 tablet_replica.cc:333] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.508813 11310 raft_consensus.cc:2243] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.509564 11310 raft_consensus.cc:2272] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.513396 11310 tablet_replica.cc:333] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.514225 11310 raft_consensus.cc:2243] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.515399 11310 raft_consensus.cc:2272] T ea10f3fd9ff14275b04cdb8f01750b64 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.518622 11310 tablet_replica.cc:333] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.519388 11310 raft_consensus.cc:2243] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.520030 11310 raft_consensus.cc:2272] T f6a40443b2994b42b552928f555d04b5 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.523520 11310 tablet_replica.cc:333] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.524374 11310 raft_consensus.cc:2243] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.525238 11310 raft_consensus.cc:2272] T 7b94e35867004e258238795d4c63b63b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.530803 11310 tablet_replica.cc:333] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
W20260709 12:21:59.531476  1198 consensus_peers.cc:597] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.531847 11310 raft_consensus.cc:2243] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.532727 11310 raft_consensus.cc:2272] T f1c4dd1b4b284a28a9ac0def27188792 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.536166 11310 tablet_replica.cc:333] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.536974 11310 raft_consensus.cc:2243] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.537534 11310 raft_consensus.cc:2272] T 224dcfc0396244728eb039ca30e4cf9a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.540787 11310 tablet_replica.cc:333] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.541491 11310 raft_consensus.cc:2243] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.542682 11310 raft_consensus.cc:2272] T 5f649afba93c4e33bfbd567a618722ed P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.545972 11310 tablet_replica.cc:333] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.546693 11310 raft_consensus.cc:2243] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.547222 11310 raft_consensus.cc:2272] T c6aba73b5e90440bb2dba4975d845d60 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.551810 11310 tablet_replica.cc:333] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.552850 11310 raft_consensus.cc:2243] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.554194 11310 raft_consensus.cc:2272] T 265e8ad7291f44ea933719b9ed465c7d P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.557332 11310 tablet_replica.cc:333] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.558159 11310 raft_consensus.cc:2243] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.558790 11310 raft_consensus.cc:2272] T 2d2b5d560ea741e7b44f121702013971 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.561828 11310 tablet_replica.cc:333] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.562470 11310 raft_consensus.cc:2243] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.563570 11310 raft_consensus.cc:2272] T b3459b8c65184dd19c43d655600014e6 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.567427 11310 tablet_replica.cc:333] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.568040 11310 raft_consensus.cc:2243] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.569114 11310 raft_consensus.cc:2272] T 80786e2a45414023841a360eaeae692a P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.572746 11310 tablet_replica.cc:333] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.573403 11310 raft_consensus.cc:2243] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.574596 11310 raft_consensus.cc:2272] T 139c72a11f16472db5000aee4927b73b P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.577824 11310 tablet_replica.cc:333] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.578626 11310 raft_consensus.cc:2243] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.579316 11310 raft_consensus.cc:2272] T cf2df0849c5d499d860c1d21e576c071 P 0ec884978e874b2c8f605373f1a3b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.582504 11310 tablet_replica.cc:333] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.583285 11310 raft_consensus.cc:2243] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.584208 11310 raft_consensus.cc:2272] T e18becdfbf98432c8b8d6f246d349918 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.587399 11310 tablet_replica.cc:333] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c: stopping tablet replica
I20260709 12:21:59.587991 11310 raft_consensus.cc:2243] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.589376 11310 raft_consensus.cc:2272] T fed323ad0dc84c839c5880be72455a22 P 0ec884978e874b2c8f605373f1a3b31c [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:59.625104  1273 consensus_peers.cc:597] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.652693 11310 tablet_server.cc:196] TabletServer@127.11.11.129:0 shutdown complete.
I20260709 12:21:59.707908 11310 tablet_server.cc:179] TabletServer@127.11.11.130:0 shutting down...
W20260709 12:21:59.739861  1198 consensus_peers.cc:597] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:35743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:59.753991  1198 consensus_peers.cc:597] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:35743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:59.756969  1271 consensus_peers.cc:597] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Couldn't send request to peer 46c1e152ed374a9a9923d464b3a57077. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.762041 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:59.763212 11310 tablet_replica.cc:333] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.764192 11310 raft_consensus.cc:2243] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.765329 11310 raft_consensus.cc:2272] T ffc8eb9b50f24f85b50bf10ec71e8f94 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.768393 11310 tablet_replica.cc:333] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.769063 11310 raft_consensus.cc:2243] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.769692 11310 raft_consensus.cc:2272] T 45977ae92e3a4855b56cbe7ddd8cd98c P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.772913 11310 tablet_replica.cc:333] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.773713 11310 raft_consensus.cc:2243] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.774818 11310 raft_consensus.cc:2272] T 0a2c51efcec743d6a228c356ce196160 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:59.776538  1271 consensus_peers.cc:597] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Couldn't send request to peer 46c1e152ed374a9a9923d464b3a57077. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.779161 11310 tablet_replica.cc:333] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.780455 11310 raft_consensus.cc:2243] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.781255 11310 raft_consensus.cc:2272] T c6aba73b5e90440bb2dba4975d845d60 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:59.782430  1273 consensus_peers.cc:597] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:35743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.785347 11310 tablet_replica.cc:333] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.786388 11310 raft_consensus.cc:2243] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.787015 11310 raft_consensus.cc:2272] T 265e8ad7291f44ea933719b9ed465c7d P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.790151 11310 tablet_replica.cc:333] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.791042 11310 raft_consensus.cc:2243] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.791740 11310 raft_consensus.cc:2272] T 5f649afba93c4e33bfbd567a618722ed P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.795979 11310 tablet_replica.cc:333] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
W20260709 12:21:59.796418  1273 consensus_peers.cc:597] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 0ec884978e874b2c8f605373f1a3b31c (127.11.11.129:35743): Couldn't send request to peer 0ec884978e874b2c8f605373f1a3b31c. Status: Network error: Client connection negotiation failed: client connection to 127.11.11.129:35743: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.796980 11310 raft_consensus.cc:2243] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.798242 11310 raft_consensus.cc:2272] T 7b94e35867004e258238795d4c63b63b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.801167 11310 tablet_replica.cc:333] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.801991 11310 raft_consensus.cc:2243] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.803005 11310 raft_consensus.cc:2272] T f1c4dd1b4b284a28a9ac0def27188792 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.806185 11310 tablet_replica.cc:333] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.806945 11310 raft_consensus.cc:2243] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.807451 11310 raft_consensus.cc:2272] T 224dcfc0396244728eb039ca30e4cf9a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.810298 11310 tablet_replica.cc:333] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.810940 11310 raft_consensus.cc:2243] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.811923 11310 raft_consensus.cc:2272] T f6a40443b2994b42b552928f555d04b5 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.814647 11310 tablet_replica.cc:333] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.815339 11310 raft_consensus.cc:2243] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.815884 11310 raft_consensus.cc:2272] T ea10f3fd9ff14275b04cdb8f01750b64 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.818651 11310 tablet_replica.cc:333] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.819392 11310 raft_consensus.cc:2243] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.820350 11310 raft_consensus.cc:2272] T cf2df0849c5d499d860c1d21e576c071 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.823167 11310 tablet_replica.cc:333] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.823813 11310 raft_consensus.cc:2243] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.824306 11310 raft_consensus.cc:2272] T e18becdfbf98432c8b8d6f246d349918 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.827088 11310 tablet_replica.cc:333] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.827747 11310 raft_consensus.cc:2243] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.828351 11310 raft_consensus.cc:2272] T fed323ad0dc84c839c5880be72455a22 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.831195 11310 tablet_replica.cc:333] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.831991 11310 raft_consensus.cc:2243] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:59.833101 11310 raft_consensus.cc:2272] T 2d2b5d560ea741e7b44f121702013971 P 46c1e152ed374a9a9923d464b3a57077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.836058 11310 tablet_replica.cc:333] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.836694 11310 raft_consensus.cc:2243] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.837311 11310 raft_consensus.cc:2272] T 139c72a11f16472db5000aee4927b73b P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.840221 11310 tablet_replica.cc:333] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.840853 11310 raft_consensus.cc:2243] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.841553 11310 raft_consensus.cc:2272] T b3459b8c65184dd19c43d655600014e6 P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:59.844446 11310 tablet_replica.cc:333] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077: stopping tablet replica
I20260709 12:21:59.845026 11310 raft_consensus.cc:2243] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.845700 11310 raft_consensus.cc:2272] T 80786e2a45414023841a360eaeae692a P 46c1e152ed374a9a9923d464b3a57077 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:59.870847  1271 consensus_peers.cc:597] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 -> Peer 46c1e152ed374a9a9923d464b3a57077 (127.11.11.130:45955): Couldn't send request to peer 46c1e152ed374a9a9923d464b3a57077. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:59.902980 11310 tablet_server.cc:196] TabletServer@127.11.11.130:0 shutdown complete.
I20260709 12:21:59.947350 11310 tablet_server.cc:179] TabletServer@127.11.11.131:0 shutting down...
I20260709 12:21:59.994887 11310 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:59.997124 11310 tablet_replica.cc:333] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:21:59.998657 11310 raft_consensus.cc:2243] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:59.999567 11310 raft_consensus.cc:2272] T 0a2c51efcec743d6a228c356ce196160 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.002671 11310 tablet_replica.cc:333] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.003443 11310 raft_consensus.cc:2243] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:22:00.004812 11310 raft_consensus.cc:2272] T 45977ae92e3a4855b56cbe7ddd8cd98c P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.007892 11310 tablet_replica.cc:333] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.008610 11310 raft_consensus.cc:2243] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.009253 11310 raft_consensus.cc:2272] T ffc8eb9b50f24f85b50bf10ec71e8f94 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.012156 11310 tablet_replica.cc:333] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.012902 11310 raft_consensus.cc:2243] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.013438 11310 raft_consensus.cc:2272] T 7b94e35867004e258238795d4c63b63b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.016142 11310 tablet_replica.cc:333] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.016780 11310 raft_consensus.cc:2243] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.017484 11310 raft_consensus.cc:2272] T f1c4dd1b4b284a28a9ac0def27188792 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.020080 11310 tablet_replica.cc:333] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.020820 11310 raft_consensus.cc:2243] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.021669 11310 raft_consensus.cc:2272] T 5f649afba93c4e33bfbd567a618722ed P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.024199 11310 tablet_replica.cc:333] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.024861 11310 raft_consensus.cc:2243] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:22:00.025939 11310 raft_consensus.cc:2272] T 224dcfc0396244728eb039ca30e4cf9a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.028916 11310 tablet_replica.cc:333] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.029692 11310 raft_consensus.cc:2243] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.030301 11310 raft_consensus.cc:2272] T f6a40443b2994b42b552928f555d04b5 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.032658 11310 tablet_replica.cc:333] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.033300 11310 raft_consensus.cc:2243] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.033941 11310 raft_consensus.cc:2272] T ea10f3fd9ff14275b04cdb8f01750b64 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.036372 11310 tablet_replica.cc:333] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.037004 11310 raft_consensus.cc:2243] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.037578 11310 raft_consensus.cc:2272] T cf2df0849c5d499d860c1d21e576c071 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.040099 11310 tablet_replica.cc:333] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.040704 11310 raft_consensus.cc:2243] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:22:00.041821 11310 raft_consensus.cc:2272] T e18becdfbf98432c8b8d6f246d349918 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.044596 11310 tablet_replica.cc:333] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.045377 11310 raft_consensus.cc:2243] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.046485 11310 raft_consensus.cc:2272] T fed323ad0dc84c839c5880be72455a22 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.049315 11310 tablet_replica.cc:333] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.050030 11310 raft_consensus.cc:2243] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.050637 11310 raft_consensus.cc:2272] T 2d2b5d560ea741e7b44f121702013971 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.053119 11310 tablet_replica.cc:333] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.053797 11310 raft_consensus.cc:2243] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.054451 11310 raft_consensus.cc:2272] T b3459b8c65184dd19c43d655600014e6 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.056881 11310 tablet_replica.cc:333] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.057471 11310 raft_consensus.cc:2243] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.058106 11310 raft_consensus.cc:2272] T 139c72a11f16472db5000aee4927b73b P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.060496 11310 tablet_replica.cc:333] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.061049 11310 raft_consensus.cc:2243] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.061514 11310 raft_consensus.cc:2272] T 80786e2a45414023841a360eaeae692a P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.063961 11310 tablet_replica.cc:333] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.064450 11310 raft_consensus.cc:2243] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:22:00.065299 11310 raft_consensus.cc:2272] T c6aba73b5e90440bb2dba4975d845d60 P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.067665 11310 tablet_replica.cc:333] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9: stopping tablet replica
I20260709 12:22:00.068168 11310 raft_consensus.cc:2243] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.068605 11310 raft_consensus.cc:2272] T 265e8ad7291f44ea933719b9ed465c7d P feb80d6a43844de4a8c50c7ea4fbe5e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.119130 11310 tablet_server.cc:196] TabletServer@127.11.11.131:0 shutdown complete.
I20260709 12:22:00.167869 11310 master.cc:562] Master@127.11.11.190:33481 shutting down...
W20260709 12:22:00.207167  1103 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Catalog manager is not initialized. State: Closing
I20260709 12:22:00.208636 11310 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:22:00.209311 11310 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.209681 11310 tablet_replica.cc:333] T 00000000000000000000000000000000 P 676e1dec53fc4ebf92c58ae165c0ad1b: stopping tablet replica
I20260709 12:22:00.253278 11310 master.cc:584] Master@127.11.11.190:33481 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (21763 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (43681 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (727933 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 12:22:00.329216 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2236 messages since previous log ~33 seconds ago
I20260709 12:22:00.329519 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2235 messages since previous log ~33 seconds ago
I20260709 12:22:00.329767 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2230 messages since previous log ~33 seconds ago
I20260709 12:22:00.330054 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~53 seconds ago
I20260709 12:22:00.330273 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~53 seconds ago
I20260709 12:22:00.330672 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 12 messages since previous log ~0 seconds ago
I20260709 12:22:00.331014 11310 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 53 messages since previous log ~20 seconds ago
